Bug 517353 - Automated tests verifying validity of links in Help
Summary: Automated tests verifying validity of links in Help
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: User Assistance (show other bugs)
Version: 4.7   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: 4.8 M1   Edit
Assignee: Mickael Istria CLA
QA Contact:
URL:
Whiteboard:
Keywords: bugday, helpwanted
Depends on:
Blocks:
 
Reported: 2017-05-29 05:06 EDT by Mickael Istria CLA
Modified: 2017-07-26 07:58 EDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mickael Istria CLA 2017-05-29 05:06:06 EDT
It's easy to put a wrong link in the help/documentation, and this is quite annoying for readers of the doc who see their navigation failing.
In order to avoid such issues, we could consider adding a test that opens all documentation pages and checks for quality, for example by checking links are valid.
Some other checks could be whether all images are rendered for example.
Comment 1 Dani Megert CLA 2017-05-29 05:13:35 EDT
We already have some tests for that. See org.eclipse.ua.tests.doc.
Comment 2 Mickael Istria CLA 2017-05-29 05:56:21 EDT
Ok, I wasn't aware of that. So the issue seems to be that the right tests are not together with the help content, so they're not run automatically on review.
Should we consider moving content to the .ua repo, or moving the tests to the .common repo?
Comment 3 Dani Megert CLA 2017-05-29 06:02:48 EDT
(In reply to Mickael Istria from comment #2)
> Ok, I wasn't aware of that. So the issue seems to be that the right tests
> are not together with the help content, so they're not run automatically on
> review.
> Should we consider moving content to the .ua repo, or moving the tests to
> the .common repo?

I don't even know whether we actually build the doc when uploading a Gerrit change (I don't think so).

Moving things between UA and doc is a no go for me.

I'm fine when the doc tests fail in the build and the committer fixes them for the next build. Looking at the build results is anyway a mandatory task for every committer.

I know you had some broken links, and the problem for that case is that there seems to be no tests for that particular case. No tests failed.
Comment 4 Mickael Istria CLA 2017-05-29 06:08:48 EDT
(In reply to Dani Megert from comment #3)
> I don't even know whether we actually build the doc when uploading a Gerrit
> change (I don't think so).
> Moving things between UA and doc is a no go for me.

Ok, so we could create a job that builds doc, and then runs UA tests. it would basically run:
$ cd ~/eclipse.platform.common
$ git checkout gerrit $GERRIT_REFSPEC
$ mvn clean install
$ cd ~/eclipse.platform.ua
$ git checkout master
$ mvn clean verify

With that, we should see ua tests running on doc change.

> I'm fine when the doc tests fail in the build and the committer fixes them
> for the next build. Looking at the build results is anyway a mandatory task
> for every committer.

As a contributor, I prefer test failing on Gerrit -it makes me more productive and prevent me a bit from merging wrong patches that are annoying other contributors- and I'm ready to contribute the necessary work on Hudson to have that.
 
> I know you had some broken links, and the problem for that case is that
> there seems to be no tests for that particular case. No tests failed.

Ok.
Comment 5 Dani Megert CLA 2017-05-29 06:19:36 EDT
(In reply to Mickael Istria from comment #4)
> (In reply to Dani Megert from comment #3)
> > I don't even know whether we actually build the doc when uploading a Gerrit
> > change (I don't think so).
> > Moving things between UA and doc is a no go for me.
> 
> Ok, so we could create a job that builds doc, and then runs UA tests.

Not the ua.tests, but just those from org.eclipse.ua.tests.doc that test the doc itself. Those would have to be moved since we don't want to deal with two repos during the Gerrit build.

I think a better first step is to make sure your case results in a failure.
Comment 6 Mickael Istria CLA 2017-05-31 06:37:14 EDT
(In reply to Dani Megert from comment #3)
> I know you had some broken links, and the problem for that case is that
> there seems to be no tests for that particular case. No tests failed.

I'm trying to write a test and had some success in having it spotting bugs (see #517515 for instance).
However, I didn't manage to make it fail for the broken link I wrote. The issue seems to be that the `*.doc.isv` bundles seem missing from help during test execution, whereas the bundles are properly installed in the IDE (Platform.getBundle(...).getState() returns a good value).
Is there anything specific to do in order to have the .isv bundles contributing to the help?
Comment 7 Dani Megert CLA 2017-05-31 11:30:14 EDT
(In reply to Mickael Istria from comment #6)
> (In reply to Dani Megert from comment #3)
> > I know you had some broken links, and the problem for that case is that
> > there seems to be no tests for that particular case. No tests failed.
> 
> I'm trying to write a test and had some success in having it spotting bugs
> (see #517515 for instance).
> However, I didn't manage to make it fail for the broken link I wrote. The
> issue seems to be that the `*.doc.isv` bundles seem missing from help during
> test execution, whereas the bundles are properly installed in the IDE
> (Platform.getBundle(...).getState() returns a good value).
> Is there anything specific to do in order to have the .isv bundles
> contributing to the help?

I don't think so, but did you really build the ISV bundle? Otherwise lots of files are missing.
Comment 8 Eclipse Genie CLA 2017-05-31 11:44:50 EDT
New Gerrit change created: https://git.eclipse.org/r/98350
Comment 10 Mickael Istria CLA 2017-07-20 03:19:39 EDT
Let's wait for the result of this test from the "official" build before we can mark it as resolved.
Comment 11 Mickael Istria CLA 2017-07-26 07:58:03 EDT
ua tests seems to be failing on testPlatformIsvGenerated

java.lang.AssertionError: Invalid link in "/org.eclipse.platform.doc.isv/topics_Samples.xml": samples/org.eclipse.ui.examples.fieldassist/doc-html/ui_fieldassist_ex.html
Invalid link in "/org.eclipse.platform.doc.isv/topics_Samples.xml": samples/org.eclipse.ui.examples.undo/doc-html/ui_undo_ex.html

at org.eclipse.ua.tests.doc.internal.linkchecker.TocLinkChecker.doAssert(TocLinkChecker.java:126)
at org.eclipse.ua.tests.doc.internal.linkchecker.TocLinkChecker.testPlatformIsvGenerated(TocLinkChecker.java:87)

However, this failure doesn't seem related to the change made here. So let's mark this bug as resolved.