Community
Participate
Working Groups
Electronic books are becoming more popular with the advent of mobile devices such as Android driven telephones and the iPad. Most of these support the EPUB format either natively or through the use of third party software. Having for instance a user guide in EPUB format allows the user (depending on the reader software) to annotate the document and synchronise location and annotations between reader instances. For more about the EPUB format, see: http://en.wikipedia.org/wiki/EPUB and http://www.idpf.org/. Generating EPUB files by means of an ANT job can be done by building on the WikiText framework to create an suitable ANT Task. The initial implementation should support version 2.0.x of the format. Please assign this bug to me if this is something you would like to have in Mylyn Docs.
There is a project for producing a Java API that can be used to generate EPUB files, epublib. It is hosted at https://github.com/psiegman/epublib. According to this post http://www.mobileread.com/forums/showthread.php?t=85204 the source is distributed under the Apache License so it should be possible to include the binaries in an Eclipse distribution (http://www.eclipse.org/legal/legalfaq.php#use_5).
Great idea! Please let me know if you'd like help getting started on a contribution. I recommend that you take a look at @org.eclipse.mylyn.wikitext.core.parser.DocumentBuilder@ and its subclasses.
Created attachment 184807 [details] mylyn/context/zip
Thanks David. I'm already at it :-)
(In reply to comment #1) > There is a project for producing a Java API that can be used to generate EPUB > files, epublib. After checking the code of this library I found that we're probably better off implementing a mechanism for assembling the EPUB file ourselves. The library is implemented more as an application with the library feature on the side. It has a lot of of dependencies, although these appear to be Apache and BSD licensed. We only need a small set of the features this project provides. EPUB specifications are here: http://www.idpf.org/specs.htm
(In reply to comment #5) > After checking the code of this library I found that we're probably better off > implementing a mechanism for assembling the EPUB file ourselves. A great place to start that has worked well for us in the past is to create a DocumentBuilder implementation specific to EPUB. Let me know if you need help with those APIs.
Will do. Thanks David.
I have some work in progress but this is becoming a little bit bigger than I first expected. So I'd like some input from you. I've decided to use EMF to model the EPUB as it's rather complex. This is looking good so far. I'm also planning to use XText to build a DSL for publication description then add an engine to produce the actual result. And the last part will be to allow use of this DSL in a Ant task. I'm thinking it would be a good idea to use two separate bundles for this contribution. One for the "core" and another for the "ui" (DSL editor etc.). I realize this goes far beyond the initial idea but I think a complete solution is much better. Since this code is not really related to wikitex I'm wondering if o.e.m.docs.epub.core and o.e.m.docs.epub.ui would be good namespaces. Any input is appreciated.
Your approach sounds great. Your suggestion of two new bundles sounds like it makes sense. Can you provide a summary of what each bundle would provide in the way of API and features? Do you plan on having the Ant tasks work outside of an Eclipse runtime?
(In reply to comment #9) > Your approach sounds great. Your suggestion of two new bundles sounds like it > makes sense. Can you provide a summary of what each bundle would provide in > the way of API and features? Do you plan on having the Ant tasks work outside > of an Eclipse runtime? Thank you for the input David. As far as I can see at the moment the "core" bundle will provide the data model, public mechanisms for converting it into a EPUB bundle, the DSL parser and the ANT stuff. It should definitely be possible to run the ANT tasks outside of Eclipse (as long as EMF etc. is in the classpath). The UI bundle should only provide the DSL editor and maybe a wizard or two. This approach should make it possible to author an EPUB bundle by hand using the DSL editor and the source material. And if you want to do it automatically you only need to provide a DSL script and the ANT script. Depending on how it goes maybe the DSL script can be automatically generated by the ANT task.
I think it is time for an update on this one. There has been good progress lately and I'm now working on QA, documentation and testing. This is what I ended up with: * The __o.e.m.docs.epub__ bundle with EMF models, a bit of API and one Ant task. * The __o.e.m.docs.epub.ui__ bundle with the user guide and book on how to use this feature. * The __o.e.m.docs.epub.tests__ bundle. The Ant task is not doing any conversion so if the source is wiki markup; existing WikiText tasks should be used for this. The following code shows the minimum Ant script required: bc.. <epub file="my.epub"> <title text="My book" /> <subject text="My subject"/> <item file="MyChapter.xhtml"/> </epub> p. I eventually decided against using a DSL for describing the EPUB as the Ant script is doing the job just fine. I may look into using the model through EEF in order to get GUI editing, but that is not a priority. I think the feature as it is will be sufficient, so I'm stopping here. The code is currently resting at https://github.com/turesheim/org.eclipse.mylyn.docs/tree/Bug_332122
What would be the next step to move this forward? Are any third party library required to build or run the code?
Hi Steffen, The code is pretty much ready. I've been working on ironing out bugs, writing tests and documentation lately. It should be all done in a month or two (see https://github.com/turesheim/org.eclipse.mylyn.docs/tree/Bug_332122). No big changes have been planned. Although I will make ready for EPUB 3 support (but not start implementation) as I guess some readers will start supporting the newer standard soon. I believe iBooks on iOS is at least almost there. A sample EPUB generated by the tool can be found here: https://github.com/downloads/turesheim/Guide_to_Fortran77/prof77.epub. There are no external libraries in the main code o.e.m.docs.epub and o.e.m.docs.epub.ui. I only use: * org.eclipse.emf;bundle-version * org.eclipse.emf.ecore;bundle-version * org.eclipse.emf.ecore.xmi;bundle-version * org.eclipse.emf.validation;bundle-version * org.apache.ant;bundle-version * org.eclipse.ant.core;bundle-version * org.eclipse.mylyn.wikitext.core;bundle-version However in the o.e.m.docs.epub.tests I have been making use of the "epubcheck" library by Adobe (http://code.google.com/p/epubcheck/) along with Saxon and Jing on which it depends. The licenses are liberal but I'm not sure if this is something that can be in committed. I don't know what would be the best approach for getting the code into the Eclipse repo. I could just continue working on the github repo then submit a patch when everything is ready.
Hi again, I figured I should submit a patch so that you could point out any obvious mistakes I've made. However the text patch is at 430kb and lack required binaries. Any ideas on how to proceed?
Following up on discussion in the Mylyn Docs developer list http://dev.eclipse.org/mhonarc/lists/mylyn-docs-dev/msg00043.html. Regarding a CQ for this feature, a list of external libraries used follows: * epubcheck: ** Project site: http://code.google.com/p/epubcheck/ ** Source: http://code.google.com/p/epubcheck/source/checkout ** Binaries: http://code.google.com/p/epubcheck/downloads/detail?name=epubcheck-1.2.zip * There is already a CQ for Saxon: https://dev.eclipse.org/ipzilla/show_bug.cgi?id=5368 * jing: ** Project site: http://www.thaiopensource.com/relaxng/jing.html ** Souce: http://code.google.com/p/jing-trang/source/checkout ** Binaries: http://jing-trang.googlecode.com/files/jing-20091111.zip (only need jing.jar)
Torkild, I've cloned your repository and will be reviewing the code over the next week. You may want to hold off on creating a patch until I've provided some feedback. The main things that I'll be looking for: * does the code only use public API (ie: no internals)? * does the code have adequate test coverage? * does the code reasonably conform to conventions found elsewhere in Mylyn WikiText? * does the contribution provide API? If so, does the provided API look reasonable? * how large is the contribution, and how will it be supported in the short, mid and long term? * is there user documentation? * how is EPUB generation provided - in the UI, via Ant tasks, API? Let me know if you have any questions or feedback for me, David
David, let me know if you want me to push it to http://review.mylyn.org/ or enable your account for pushing.
(In reply to comment #17) > David, let me know if you want me to push it to http://review.mylyn.org/ or > enable your account for pushing. Yes that would be fantastic. Thanks Steffen!
(In reply to comment #16) > Torkild, > > I've cloned your repository and will be reviewing the code over the next week. > You may want to hold off on creating a patch until I've provided some feedback. Thanks David, My replies are inline. I will be doing some changes to documentation and testing in the near future and leave the rest alone unless theres something critical. Which there was yesterday, it has been fixed and pushed now. > The main things that I'll be looking for: > > * does the code only use public API (ie: no internals)? Yes. > * does the code have adequate test coverage? No. I'll be focusing on that for the next few days. With regards to the generated EPUB we have fairly good coverage as **epubcheck** can validate the files. > * does the code reasonably conform to conventions found elsewhere in Mylyn > WikiText? I believe so, at least with regards to formatting etc. > * does the contribution provide API? If so, does the provided API look > reasonable? Yes - and I hope so. > * how large is the contribution, and how will it be supported in the short, mid > and long term? Not very large. I'm planning to work on this for quite a while, at least until support for EPUB is complete and flawless. In addition there other parties interested to fund support. > * is there user documentation? Yes. I think the documentation is decent if not good. > * how is EPUB generation provided - in the UI, via Ant tasks, API? All of the three.
Torkild, can you merge your master on github with the latest Mylyn Docs master? You can of course also publish a code review directly to the Mylyn Gerrit if you create an account and send me your login.
Steffen, note that the branch in question is **Bug_332122**. I just checked and there are no changes in the latest Mylyn master that I don't have. My login at Mylyn Gerrit is torkildr/torkildr@gmail.com.
Thanks! I missed the branch in my first attempt. I have now posted a code review here: http://review.mylyn.org/#change,36 These are the steps I took: 1. Added remote to .git/config pre. [remote "gerrit"] url = ssh://spingel@review.mylyn.org:29418/org.eclipse.mylyn.docs.git fetch = +refs/heads/master:refs/remotes/gerrit/master push = HEAD:refs/for/master 2. Ran the following commands to squash the contribution into a single commit that branches from the current master. pre. git clone https://github.com/turesheim/org.eclipse.mylyn.docs.git cd org.eclipse.mylyn.docs git checkout Bug_332122 git merge gerrit/master git branch -t epub-publish gerrit/master git checkout epub-publish git merge Bug_332122 --squash git push gerrit
Torkild, I have added your account to the trusted users group so you should now be able to push reviews. Can you add pom files for the new bundles and add the features to the site? That way we can automatically build and test stuff. I would recommend renaming org.eclipse.mylyn.docs.epub.ant to org.eclipse.mylyn.docs.epub.ant.core and changing the feature id to org.eclipse.mylyn.docs.epub.ant. It's the convention that we have been using lately where every component (feature) has it's own namespace and all bundles that are part of the component have one additional segment in their id.
Nice work Torkild! Just performed a code review: If5f44f76: Add ANT task for converting from wiki markup to EPUB format http://review.mylyn.org/#change,36
(In reply to comment #24) > Nice work Torkild! Just performed a code review: > > If5f44f76: Add ANT task for converting from wiki markup to EPUB format > http://review.mylyn.org/#change,36 Thanks Steffen, David! I've done done the requested changes or given reasons where not. Had some problems using "git rebase" to squash the history as the repo would end up in a terrible state. Hence I used git reset instead. As a result the Gerrit change has a misleading title (http://review.mylyn.org/#change,36). I'll fix that in the next push.
Torkild, it looks like things are evolving nicely here. It would be great to see the tests run by the build - the latest build on http://review.mylyn.org/#change,61 didn't seem to run any tests. Once tests are running as part of the build (and passing), we'll be ready to initiate the CQ process. To do this, we'll need the following: * a patch, including all source code, but excluding binaries (nested jars) * a list of all dependencies for which we'll need an additional CQ
I would also recommend excluding generated sources from the CQ assuming that it's trivial to re-generate them once the CQ is approved.
Torkild, it looks like you've now got tests running in Hudson for EPUB on the "latest build":http://ci.mylyn.org/job/mylyn-docs-trigger/34/testReport/ What's left to do before you're ready to cut a patch?
Thanks David, I was wanting to get 100% coverage on the API tests, but it's not really required. I'm confident that what we have is sufficient, so I can leave it for now. What is more important is building of the demonstration EPUB containing the user guide and a bit more. As I cannot get it to build due to some issues with EMF and classpath I'm inclined to move it to a separate "examples" bundle. It should probably not be a part of the user guide bundle anyway and may just as well be built manually. Apart from that I just have to clear away the generated code (which is trivial to re-generate) and create the patch. If I zip it I can get it down to around 80k (from 511k) so I should be able to attach it to this report.
Created attachment 204802 [details] Code resolving the issue * Contains Ant task for defining and writing EPUBs * API for handling, reading and writing EPUBs * UI command for generating EPUBs from wiki markup * User guide * Examples project demonstrating the Ant task
(In reply to comment #26) > * a patch, including all source code, but excluding binaries (nested jars) > * a list of all dependencies for which we'll need an additional CQ David, a patch has been attached and a list of dependencies for the test bundle can be found in comment #15.
Sorry about the trouble if you've tried it already. It appears that this patch won't apply cleanly. I'll create another one shortly.
Can you create a zip file of all the new files instead? We'll likely need that for the CQ anyways. You can create a patch for the few existing files that were changed.
Created attachment 204924 [details] Proposed implementation
Created attachment 204928 [details] Changes to existing code A patch containing changes to existing code. Namely pom.xml at the root for building the EPUB support along with adjustments to site.xml so the feature is included.
I've created the following CQ: 5735: EPUB document format support in Mylyn Docs https://dev.eclipse.org/ipzilla/show_bug.cgi?id=5735 Torkild, you'll need to make a statement to the effect of: > This is my confirmation that I, Torkild U. Resheim: > * Authored the code. > * Have a right to contribute the content to Eclipse. > * Contribute the content under EPL.
Thanks David, the statement has been added to the CQ.
I recently got a notice stating that the CQ review is complete and approved. I believe the latest archive attached to the IPZilla item can be used as it is. Please let me know if there is anything I'm supposed to do.
That's great news! I have tentatively scheduled this for Mylyn 3.7 and will discuss with David how we can proceed to merge the contribution.
We are planning to merge the contribution as approved for the CQ later this week and will publish the new feature on the Mylyn Docs nightly update site.
(In reply to comment #40) Thanks Steffen! These are good news. I see you've flagged the contribution as "noteworthy". If you'd like I'll start on an item for the New and Noteworthy (http://www.eclipse.org/mylyn/new/weekly.php).
(In reply to comment #41) > I see you've flagged the contribution as > "noteworthy". If you'd like I'll start on an item for the New and Noteworthy > (http://www.eclipse.org/mylyn/new/weekly.php). That would be great!
Created attachment 211587 [details] Items for new and noteworthy A description of the wizard and the Ant task with illustrations.
The zip archive at https://dev.eclipse.org/ipzilla/attachment.cgi?id=8681 contains the latest, reviewed code. Hence older versions of the code attached to this issue has been obsoleted.
I've pushed the contribution out to Gerrit to verify that it builds: http://review.mylyn.org/#change,323
I was able to generate the missing packages but it looks like we are missing a few features in our target definition: * org.eclipse.emf.mapping.ecore2xml * org.eclipse.emf.validation.ui * org.eclipse.emf.databinding.edit * org.eclipse.emf.ecore.change.edit * org.eclipse.emf.ecore.editor * org.eclipse.emf.exporter * org.eclipse.emf.importer * org.eclipse.emf.mapping.ecore2ecore.editor * org.eclipse.emf.mapping.ecore2xml.ui Torkild, which (EMF) features do we need to add to resolve these bundles? I also noticed a requirements for Java 1.6: * Bundle-RequiredExecutionEnvironment: JavaSE-1.6 Is that actually needed? No problem if that is the case, we would just need to document it.
I have pushed another review on top of the initial contribution that includes the generated code: http://review.mylyn.org/#change,324
Hi Steffen, The bundles come from the following features: org.eclipse.emf.mapping.ecore * org.eclipse.emf.mapping.ecore2xml org.eclipse.emf.mapping.ecore.editor * org.eclipse.emf.mapping.ecore2xml.ui * org.eclipse.emf.mapping.ecore2ecore.editor org.eclipse.emf.validation * org.eclipse.emf.validation.ui org.eclipse.emf.ecore.edit * org.eclipse.emf.ecore.change.edit org.eclipse.emf.ecore.editor * org.eclipse.emf.ecore.editor org.eclipse.emf.databinding.edit * org.eclipse.emf.databinding.edit org.eclipse.emf.converter * org.eclipse.emf.exporter * org.eclipse.emf.importer I believe there was some issue with Java 5 which was resolved in Java 6 that forced me to set up this dependency. I'll investigate.
Thanks. Would you be able to send me a patch for mylyn-e3.5.target which adds the required features?
Created attachment 211644 [details] Patch for target platform HI Steffen, I believe I got the missing bundles in the patch. However I don't know how to verify that it's in order.
Created attachment 211645 [details] mylyn/context/zip
Steffen, what's the best way forward at this point? Should I submit the contribution change http://review.mylyn.org/#change,323 ?
We have to complete the releng changes before merging the patch. I'll take care of that.
I have pushed the changes but only included the feature on the nightly site for now. We have a policy to allow release train dependencies for any feature that is part of the default Mylyn distribution. That ensure that features are easy to install for users and it reduces releng overhead since all dependencies are consumable from a single stable update site per supported platform release. We would need a patch for mylyn-e*.target that adds features using the respective release train location. You can test this by setting the target as the default in your workspace: http://wiki.eclipse.org/Mylyn/Contributor_Reference#Target_Environment. Thank you very much for contributing this feature, Torkild! I'll update the New & Noteworthy shortly and add a link to the Mylyn Docs nightly site.
(In reply to comment #54) > We would need a patch for mylyn-e*.target that adds features using the > respective release train location. You can test this by setting the target as > the default in your workspace: > http://wiki.eclipse.org/Mylyn/Contributor_Reference#Target_Environment. Ok, Thanks. I'll add and test these. > Thank you very much for contributing this feature, Torkild! I'll update the New > & Noteworthy shortly and add a link to the Mylyn Docs nightly site. Thank you for accepting it :-)
The nightly build at https://hudson.eclipse.org/hudson/user/spingel/my-views/view/Mylyn/job/mylyn-docs-nightly/406/consoleFull appears to have failed: Caused by: org.apache.maven.plugin.MojoExecutionException: /opt/users/hudsonbuild/workspace/mylyn-docs-nightly/org.eclipse.mylyn.docs.epub.ant.core/build.properties: bin.includes value(s) [lib/] do not match any files. at org.eclipse.tycho.packaging.IncludeValidationHelper.checkIncludesExist(IncludeValidationHelper.java:75) at org.eclipse.tycho.packaging.IncludeValidationHelper.checkBinIncludesExist(IncludeValidationHelper.java:42) at org.eclipse.tycho.packaging.AbstractTychoPackagingMojo.checkBinIncludesExist(AbstractTychoPackagingMojo.java:133) at org.eclipse.tycho.packaging.PackagePluginMojo.createPluginJar(PackagePluginMojo.java:136) ... 22 more It looks like the ant library hasn't been assembled when the package step runs.
I am wondering about two things: * Why are we not using the build.properties to assemble the nested jar and run a custom ant build instead? * If we the custom ant script is indeed required would that need to run during the prepare-package phase?
(In reply to comment #57) > I am wondering about two things: > > * Why are we not using the build.properties to assemble the nested jar and run > a custom ant build instead? > * If we the custom ant script is indeed required would that need to run during > the prepare-package phase? Hi Steffen, I cannot remember the details around this. However the purpose was to be able to declare the Ant task and get it to run from within Eclipse. I noticed that you've already done some adjustments so I will await the result before jumping in.
I tried this to verify that we can make it work without custom ant code: http://review.mylyn.org/#change,344. I'm not sure that the result is what you are looking for though. The ant sources would then only be included in a nested jar and not in the jar that's distributed. What is the structure that we are looking for? I believe we removed the PDE extension all together for WikiText because it never worked in the first place.
I have pushed the fix that moves assembly to the prepare-package phase since it's least invasive. I'm still uncertain though if that produces the desired result. The jar now looks like this: pre. Archive: /home/hudson/.hudson/jobs/mylyn-docs-trigger/workspace/org.eclipse.mylyn.docs.epub.ant.core/target/org.eclipse.mylyn.docs.epub.ant.core-0.8.0-SNAPSHOT.jar Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 0 Stored 0 0% 2012-02-29 18:54 00000000 META-INF/ 820 Defl:N 424 48% 2012-02-29 18:54 ba6501e3 META-INF/MANIFEST.MF 0 Stored 0 0% 2012-02-29 18:54 00000000 lib/ 33574 Defl:N 26097 22% 2012-02-29 18:54 a046d11b lib/epub-ant.jar 512 Defl:N 214 58% 2012-02-29 18:43 9409f0d8 plugin.xml I don't this this jar is easily usable in standalone ant builds (and I have no idea if it works within Eclipse).
Ok, I've refreshed my memory and... The PDE extension did not work because it was not correctly assembled. I've made a note of this in bug #360233. Basically the jar containing the Ant task must be separate from the regular plug-in because it's the Ant classloader that must load it (when within Eclipse). I've just recently tested and can confirm that the way the EPUB tooling was set up did work. The Jar is *not* usable in standalone Ant builds, hence the duplicate binaries.
Ok, then we'll go with that. I have triggered a new build: https://hudson.eclipse.org/hudson/job/mylyn-docs-nightly/
Thanks Steffen, I can confirm that the Ant task now works both within Eclipse and in a standalone Ant script. Although I had to unpack the org.eclipse.mylyn.docs.epub.ant.core bundle in order to add the epub-ant.jar to the classpath of the script. A bit more cumbersome (than having the duplicate binaries) I guess, but fully doable. The list of dependencies ended up becoming pretty short too, only three EMF bundles in addition to the two Mylyn Docs bundles. I'll write a tutorial on how to use the tooling outside of Eclipse. While doing this exercise I noticed that there were a few dependencies, in particular on EMF validation and databinding that appear to be unused. I'll investigate further and create a patch. (I used build #410 from https://hudson.eclipse.org/hudson/job/mylyn-docs-nightly/)
Created attachment 211850 [details] Updated dependencies Looks like I got fooled by EMF's habit of re-exporting dependencies. I've carefully gone through all dependencies and only added those that are required. I've also adjusted the version numbers so that match what is needed.
Thanks. I have opened bug 372949 to track that. Please push a review with your changes. We are past the contribution deadline but we apply that as soon as the IP log for 3.7 is approved. I'll mark this bug as resolved since the build has been fixed an the EPUB feature is now available from the Docs nightly site: http://download.eclipse.org/mylyn/snapshots/nightly/docs . We'll track any further enhancements on new bugs.
Ok, thanks.
