Community Tech/Ebook Export Improvement/tr

• Have we covered the main reasons why people export ebooks?
• Have we covered the main methods to export ebooks?
• Have we covered the main problems experienced when exporting ebooks?
• Which formatting and style issues are the most common and frustrating, in your opinion?
• Which user experience issues are the most common and frustrating, in your opinion?
• Which problems, overall, do you find the most critical to fix, and why?
• Anything else you would like to add?

• The Uptime Robot Dashboard shows that WS Export is nearly always up (as of 31 March 2021):
  • Last 24 hours: 99.139%
  • Last 7 days: 99.768%
  • Last 30 days: 99.834%
  • Last 90 days: 99.896%
• The median export time for books is 3 seconds, as found from one month of data, recorded between 17 February to 17 of March (source). We don’t know the median export time before our changes, since we only recently began logging this data.
• There are many more exports being successfully generated:
  • Within a 30 day period, our data recorded 1,168,667 successful exports after our changes (source) vs. 71,613 successful exports before our changes (source). This shows 16 times as many successful exports after the changes.
  • Within a 60 day period, our data recorded 1,513,486 successful exports after our changes (source) vs. 160,433 successful exports before our changes (source). This shows 9.4 times as many successful exports after the changes.
  • Within a 90 day period, our data recorded 1,628,523 successful exports after our changes (source) vs. 249,924 successful exports before our changes (source). This shows 6.5 times as many successful exports after the changes.
  • Note: It makes sense that the greatest growth happened within 30 days, since we have released the most changes within the last few months. However, it should be noted that some of the ebook exports (both before and after the changes) may be from bots, and we cannot easily determine which downloads are from bots.
• The error rate of ebook exports is significantly lower:
  • We have decreased the likelihood of encountering an ebook error by approximately 12 times (within a 30 day period). Here is the breakdown of the data:
    • Before our changes, we recorded a total of about 46,000 errors per 71,613 successful exports within a 30 day period. This was calculated because we had found a total of 22,982 errors in a 14-day period, which we can roughly double to be about 46,000 errors in a 30 day period (source). This means that, for every 1.5 downloads, there was one exception thrown.
    • After our changes, we recorded a total of 63,921 exception errors per 1,168,667 successful exports within a 30 day period (source). This means that, for every 18 successful ebook exports, there was 1 exception thrown. This is a 12 times improvement over the previous ratio. This is based on the data recorded in a 30 day period after our changes.
    • Note: There isn’t a 1:1 relationship between exports and exceptions. In other words, one unsuccessful export can generate multiple exception errors.
• We expect that the error rate will drop even lower very soon:
  • In our new data, the most common error type is DriverException, which accounts for 46% of all exception errors. This is a new error type, which isn’t even present in our old data. This is because this error type was brought on by our replicas work.
  • There is already a fix for DriverException errors that will be deployed soon. Once this fix is released, the total error rate should be even lower.
• We have added a new option for problematic exports: download without credits.
  • We determined that one common cause of slow or problematic book downloads was the credits displayed in the “About” section. The credits can take a lot of time to generate and may cause ebook export issues. For this reason, you can now choose to “Exclude editor credits” in the Wikisource Export page.

• We upgraded font support in ebook exports: We fixed the issue that previously displayed boxes rather than characters in ebook exports, especially for non-Latin script languages. Now, everyone can download books in their chosen language!
• The Wikisource Export page has been internationalized: The page used to be in English only. We internationalized it, so it would be available for translation, and it’s now been translated into many languages already.
• Communities can pick their default fonts for the download button: When we added the new download button, we heard from community members that they wanted the ability to choose the default font that was most suitable for them. We understood the importance of this, so we made this possible.

• There is now a user-friendly “Download” button: We simplified the user experience by providing a user-friendly “Download” button. We drew from community feedback and usability testing to develop the feature. We also incorporated updates to the tool after sharing the first iteration with community members, such as adding a link to “Other formats” on the WS Export page (thanks for the tip, InductiveLoad!).
• We simplified the language code process: In the old WS Export page, users needed to manually enter in the language code, which was confusing for newcomers. With our changes, no language code knowledge is required! We provide all of the available languages in the dropdown.
• We improved support when errors occur: We have implemented greater support for users who encounter ebook export errors. Now, when users encounter errors, they will see suggestions for next steps displayed on the Wikisource Export page. We also worked to ensure that the error messaging is appropriate for the type of error that has occurred.
• The Wikisource Export page has been revamped: The older version of the Wikisource page was in need of a visual make-over. For this reason, our designer provided a series of recommendations to improve readability, usability, and mobile accessibility. The end result is a much cleaner design, and we hope you love it as much as we do!

• Implement a new download button: Now when you go to a book on Wikisource, you will see a blue button, labeled “Download.” When you click on the button, a pop-up will appear that enables you to download books in PDF, EPUB, and MOBI formats. You will see information below the file format name, which lets you know which file format is appropriate for your device. We decided to do this work after collecting feedback from the user talk page and on usertesting.com, which confirmed that many people found the current download experience not very user-friendly. This new download button is meant to increase the accessibility of ebook exports on Wikisource.
  • Note: We implemented a simplified version of the initial mockups. Some parts of the original proposal turned out to be too big and complex, so we opted for a trimmed down version, which you can now see on the wikis.
• Internationalize the WS Export page: The Wikisource Export page used to be in English only, which was obviously a problem. With this change, the page can now be translated into many different languages. Translations have already begun. If you don’t see it translated into your wiki’s language, we invite you to help the translation effort on translatewiki!
• Replace ElectronPDF with WSExport PDF support: Some wikis previously used ElectronPDF for PDF downloads in their sidebar links. ElectronPDF does not work very well, especially for Wikisource, which led to many people having frustrating download experiences. We replaced ElectronPDF links with WSExport links, so now all wikis can download PDFs via the sidebar with a more reliable service.
• Migrate WSExport Gadget to to the Wikisource Extension: In the process of consulting with Wikisource users on the project talk page, we learned that some users did not have the ability to click download links (such as "Download as MOBI") via the side panel and other places. This was the case on Czech Wikisource, for example. This was because these wikis did not have the WSExport gadget enabled on their wiki. To fix this issue, we migrated the gadget over to the Wikisource extension, so that all users can now have the same download links available.
• Cache all API Requests: With this work, we have added another way to improve efficiency of ebook exports. Now, if someone downloads Book A and then someone else wants to download Book A shortly afterward, the generated ebook will already be cached. This means it doesn’t need to be newly regenerated for each request, and the download time will be much quicker the second time around.
• Migrate API to Parsoid API: This work was recommended by Tpt, and we agreed with the recommendation, after conducting our own investigation. With this work, we now have better long-term support of formatted text on Wikisource exports. Also, this work was inevitable, since Parsoid API will eventually replace the current MediaWiki's native parser. Now that this work is done, we can identify bugs and issues early on, which we can direct to the Parser team. Please note that we are aware that some bugs came out of this work, but since the migration was scheduled to occur either way, we think it’s best to identify the bugs early on.
• Reinstate OPDS Support: With this support reinstated for OPDS now reinstated, you can now access up-to-date catalogs of Wikisource books that are suitable for download on your device. It should be noted that OPDS export can be set up to work with any category on a wiki. It produces a daily updated file that can be used in software like Calibre, FBReader, and some e-readers to more quickly and easily browse and download available books. The existing categories are just the first ones; others can be made, even multiple per wiki (e.g. "Literature for export" or something).
• Investigation on how to improve error messaging: We know that many people feel frustrated when there are ebook export errors, since there is typically minimal information or support. The purpose of this investigation was to determine what potential improvements we could implement from a technical perspective. Now that this work is complete, we are now looking into implementing improved support (see next section for details).
• Various bug fixes: In software development, whenever new work is done, new bugs arise. As a team, we have analyzed and fixed many bugs, such as RTL language support issues, internal links issues, and other bugs. However, as mentioned above, some bugs may be handled by other teams in the future, such as Parsoid-related bugs (which may go to the Parser team).

• Improve language selection on the Wikisource Export page:
  • STATUS: In development. Engineers already have work they are reviewing.
  • SUMMARY: Right now, users need to know the language code of the book (such as “en” for English or “fr” for French) in order to export it on the Wikisource Export page. This is not user-friendly. For this reason, we are implementing a selector, so you can pick the appropriate language from a dropdown
• Allow communities to choose appropriate fonts for download:
  • STATUS: In development. Engineers already have work they are reviewing.
  • SUMMARY: Wikisource users want a way to select which font is best for their language when downloading books. So, for example, if I am on Hindi Wikisource and I click the blue download button and then choose to download a book, the book should be in the font that the community decided is best for the Hindi language. We are now working to make this possible. This work is already in development.
• Remove dependency on credit generation:
  • STATUS: In development. Engineers already have work they are reviewing.
  • SUMMARY: When this work is complete, wikis will be able to download books without editor credits, if they want. We decided to do this work because credit generation is often a very intensive process. It can significantly slow down ebook exports. For this reason, if a user is experiencing difficulties downloading a book, they may want to try downloading the book without the credits. We wanted to implement the technical support to make this possible, either for users now or in the future.
• Create option to disable credits:
• STATUS: Ready for development and in engineer's backlog. Will be worked on very soon.
  • SUMMARY: Once we have removed the dependency on credit generation, we can create the option to allow ebook exports without credits on the Wikisource Export page. This could be useful in cases when books are particularly difficult to download due to lag time and errors. We could also create a separate page with a list of the book credits, so that even users who choose to download books without the credits can view them within Wikisource Export.
• Improve messaging about errors:
  • STATUS: In planning stages.
  • SUMMARY: Our investigation found that it would be too much work to implement specific messaging about errors within the pop-up. For this reason, we have decided to improve the error messaging on the Wikisource Export page. The basic idea is that, if you encounter an error via the download button, you will be redirected to the Wikisource Export page. There, you will be told that there was an error and be given proposed next steps. While this may not generate a successful ebook export for all use cases, it will provide more helpful messaging and support for users than the current default behavior.

• What do you think of the new "Download" pop-up? Do you find it easy to use?
• What do you think of our additional work to improve reliability? Are you still seeing improvements in the speed and performance of WS Export?
• What do you think of our remaining work to disable credits as an option?
• What do you think of our remaining work to improve messaging and support when there are download errors?
• Is there anything else you would like to add?

Please share your feedback on the project talk page!

• What do you think of our recent font support work? Does the issue seem to be largely resolved of boxes appearing rather than text?
• What do you think of our recent and upcoming reliability work? Do you have any thoughts, concerns, or suggestions to share?
• What do you think of our proposed improvement to the download user experience overall? Do you like the general idea and user flow?
• Do you usually download the same file format (e.g., PDF, MOBI, EPUB, etc) every time you download a book, or do you often pick a different format?
• Is there anything else you would like to share?

Please share your feedback on the project talk page!

1. We should be continually mindful of both contributors and visitors. These groups may have some overlapping needs, but there may also be some important differences for us to investigate, explore, and address.
2. We should think about user experience improvements (rather than only technical improvements). The process to export ebooks is currently not intuitive for many users. If we can improve this experience, we can potentially retain and nurture a larger base of readers and editors.
3. We should try to investigate, document, and share best practices for Wikisource. As we have consulted with people, we have learned that some perceived errors are actually not technically “bugs.” Instead, they are rooted in confusion over formatting and best practices. As such, they can potentially be fixed by the community. This is great news, but more people need visibility into these best practices. For this reason, we plan to investigate best practices, and we’ll share our findings with the community.
4. We won’t be able to fix everything (unfortunately!), but we’ll document all the issues we encounter. For this project, we’ll be focusing on improving core accessibility (i.e., WSExport functionality & reliability) and core readability (i.e., people’s ability to read exported books in their language of choice). If there are issues that don’t fall into these two categories, we’ll still document the issues in Phabricator–but we may not be able to fix them. It is our hope that other people may be able to fix the issues that we can’t get to in the future.

1. We moved WSExport to a Virtual Private Server (VPS): Before this work, the memory and CPU intensive processes of WSExport were too much for the resources on Toolforge. For this reason, we moved the tool to a VPS instance, so that we could improve overall performance.
2. We upgraded Calibre on WSExport VPSs: The VPS was on Calibre 3.48.0, but the latest version was 4.10.1. For this reason, we performed an upgrade, which helped fix various bugs related to generating PDFs. We will also continue with upgrades when new versions are available.
3. We conducted a technical analysis and font rendering investigation. Both of these produced lots of interesting proposals (some of which we’ll share in the section below). In addition, we were able to identify four common types of formatting issues for ebook exports, which are: 1) The original HTML of the wiki, 2) The process of converting the wiki HTML into an ePUB XHTML file, 3) The second output formats (such as PDFs, introduced by Calibre), and 4), The rendering of ePUB files by an eReader. While we can’t do much for the last two (since they involve external software/tools), we can try to improve the situation with the first two.
4. We are currently collecting baseline data on issues related to uptime and errors associated with the WSExport tool, among other data points. This will enable us to compare the original data (i.e., at the beginning of the project) with the data later on (i.e., after changes have been made at the end of the project). This way, we can measure our impact in a meaningful way.
5. We have begun to organize existing Wikisource tickets. We’re currently in the process of migrating tickets from GitHub to Phabricator, so they can be consolidated in one place. Once the migration is complete, we propose that we group these tickets into categories, so we have a sense of the general “buckets” of work that we can do. In the future, we ask that all new issues are reported on Phabricator. If you want assistance in how to do this, you can reach out and ask on the project talk page.
6. We have begun to investigate how we can improve the user experience for all Wikisource users. This investigation is currently in development, and it will be the primary focus of our next status update (which is tentatively scheduled for September).

1. Investigate cache generated ebooks: With this work, we propose to cache files that are produced, so they do not need to be freshly generated whenever someone wants to download them. This would potentially speed up the process of downloading books. We would need to investigate this proposal first, so we could determine the general impact and scope of the work.
2. Investigate job queue for more efficient ebook generation: With this work, we propose implementing a queue-running system. This process would potentially speed up the current download process and give users more information on the download status of a book. One way it could be done is that a user would submit a request for an ebook, which would add it to the queue. The queue would first generate the ePUB, which would then be immediately available for download, and then it would generate the derivative forms (such as, PDF) and make those available when done. We would need to investigate this proposal first, so we could determine the general impact and scope of the work.
3. Investigate how to prevent incomplete book downloads: The purpose of this work would be, if possible, to reduce the problem of downloading only a section of a book (when the user expects to download the whole book). We could possibly do this by using subpages from all pages (rather than just from ws-summary) when downloading books. We could also potentially follow subpage redirects, when they exist. We would need to investigate this proposal first, so we could determine the general impact and scope of the work.

1. Through consulting on the talk page, we have learned that we should also be looking into improving support for mathematical equations. While we covered font rendering issues in our original investigation, we didn’t talk about numbers and equations. We’ll look into mathematical equations, as well.
2. We have learned through the font rendering investigation that we can switch to a new system of fonts. This would enable us to use fonts from a host system (Debian), which has a much larger library of fonts than our own library. For example, this work would give us access to multiple Kannada fonts (while our current font set offers no specific Kannada support).

We would now love to read your feedback, especially in response to the questions below. Remember that we’ll also have a few round of collecting feedback, so we’ll engaging you again in the future as well. You can add your feedback on the project talk page. Thank you in advance!

• What are your general thoughts about the guiding principles that we have learned from the consultation so far (i.e., “Lessons from the consultation”)? Is there anything that you think we should add or change?
• Is there anything you would like to share about the work we have done so far (i.e., VPS work, Calibre upgrade, various investigations, and the consolidation of tickets)? We’re open to any thoughts or suggestions!
• What do you think of the proposal to investigate cache generated ebooks? Would this be useful and high-priority, in your view? Do you have any concerns?
• What do you think of the proposal to investigate job queue for more efficient ebook generation? Would this be useful and high-priority, in your view? Do you have any concerns?
• What do you think of the proposal to investigate how to prevent incomplete book downloads? Would this be useful and high-priority, in your view? Do you have any concerns?
• What do you think of the proposal to switch to a new system of fonts? Would this be useful and high-priority, in your view? Do you have any concerns?
• What work or investigations would you like to see that is *not* being addressed or is being addressed in a different way than you would expect? In other words, what do you think we’re overlooking, if anything?
• Anything else you would like to add?