Community
Participate
Working Groups
When working on tasks it's common to work with documents that aren't in the workspace. It would greatly enhance the platform experience if users could search for these documents using the standard Eclipse search functions (search menu, dialog, search view). Platform search should be enhanced to make it possible to search the desktop, finding files both within and outside of the workspace. Search should make use of search indexes provided by the OS, for example the Windows 7 search APIs. Search results should be displayed in the standard search view.
To support this experimental feature we'll need a couple of new projects in @org.eclipse.mylyn.incubator@. I propose that we create: * @org.eclipse.mylyn.sandbox.search.ui@ - a UI plug-in implementing standard search functionality * @org.eclipse.mylyn.sandbox.search-feature@ - a feature project
That sounds useful but closer to the scope of the Eclipse project than Mylyn and we have previously kept these type of tight desktop integrations deliberately out of the scope of the project. I assume that this is initially targeted for the Incubator to mature the code and move it to a different project later to make it available for general consumption? Where do you foresee this code ending up? The proposed bundles sound good to me.
(In reply to comment #2) > I assume that this is initially targeted for the Incubator to mature the code > and move it to a different project later to make it available for general > consumption? Where do you foresee this code ending up? Yes, this would be in the Incubator initially. We would evolve the feature there with the expectation that if it proves generally useful it could become a platform feature in much the same way as we did with "Notifications":http://wiki.eclipse.org/Platform_UI/Notifications
(In reply to comment #3) > Yes, this would be in the Incubator initially. We would evolve the feature > there with the expectation that if it proves generally useful it could become a > platform feature in much the same way as we did with > "Notifications":http://wiki.eclipse.org/Platform_UI/Notifications Sounds good. Unfortunately we haven't succeeded in contributing the notifications framework back to platform. The difference is though that notification popups are heavily used in other Mylyn sub-projects which was the reason for including them in the Mylyn commons. I don't see an obvious driver in Mylyn for the desktop search framework but it seems useful and has a good potential to move to the platform so it seems reasonable to evolve it in the Incubator. I suggest that we tag all related bugs as [search] in case we decide to move the component and bugs later on.
Great, sounds good. I'll make an initial code contribution shortly. [search] sounds like a good tag, +1. I've updated the "Mylyn Contributor Reference":http://wiki.eclipse.org/Mylyn/Contributor_Reference#Bugzilla to reflect the new tag.
I've made the intial contribution. Note that it's a work in progress. Currently the search implementation isn't hooked up to the OS index, it's just a mock search that finds a few files from the filesystem for the purpose of testing the UI implementation. Steffen I've dropped a couple of Maven poms in there. As far as I can see they look right. What is the next step as far as testing this with the Mylyn build?
Created attachment 197996 [details] mylyn/context/zip
You can run a build here: https://hudson.eclipse.org/hudson/job/mylyn-incubator-nightly/. The created repository gets published to this site: http://download.eclipse.org/mylyn/incubator/3.6 .
Thanks David. Please make the following changes: * Copy all files from a .settings/ of an existing Mylyn plug-in and reformat the sources. * Change the copyright header to "Tasktop Technologies.", i.e. remove "and others" unless the copyright is shared. * Set the Bundle-RequiredExecutionEnvironment to J2SE-1.5. If Java 1.6 is actually required we should discuss that. * Export all packages as x-internal * Make sure all required legal files are selected in the build.properties (you can use an existing plug-in as a template). * Add "(Incubation)" to the name of the feature. * Rename DesktopSearchResultPage.setID() to setId() to follow the general naming convention. To ensure the proper project setup it's always best to copy an existing bundle and replace source instead of starting from scratch: http://wiki.eclipse.org/Mylyn/Contributor_Reference#Creating_new_plug-ins I have included the new pom files in the parent pom and changed the feature id to org.eclipse.mylyn.sandbox.search. I have also added the proper platform filters to the poms and changed the branding plug-in of the new feature to org.eclipse.mylyn. A build is now running here: https://hudson.eclipse.org/hudson/job/mylyn-incubator-nightly/124/console
A first build of the feature is now on the update site. Note that it shows as "featureName" due to missing files.
Steffen, thanks for the detailed review, recommendations and build setup. These are complete: * -Copy all files from a .settings/ of an existing Mylyn plug-in and reformat the sources.- * -Change the copyright header to "Tasktop Technologies.", i.e. remove "and others" unless the copyright is shared.- * -Set the Bundle-RequiredExecutionEnvironment to J2SE-1.5. If Java 1.6 is actually required we should discuss that.- * -Export all packages as x-internal- * -Make sure all required legal files are selected in the build.properties (you can use an existing plug-in as a template).- * -Add "(Incubation)" to the name of the feature.- Can't do this one: * Rename DesktopSearchResultPage.setID() to setId() to follow the general naming convention. ** Can't do that, since @ISearchResultPage@ is implemented and specifies those methods.
I have fixed the src.includes, classpath settings and added .cvsignore files. The problem is now that the code not longer compiles due to usage of Java 1.6 specific APIs. Can you fix that?
Thanks for picking up on that. I've now installed Java 5 so I won't be using Java 6 APIs any more.
Created attachment 198060 [details] Patch for the Windows 7 Search native dll. The file was generated through this command: git diff master Win7SearchDLL > diff.txt
Thanks for the patch Raymond. It looks good. I've applied the patch with some minor changes, and I've moved Windows7Search to /org.eclipse.mylyn.sandbox.search.ui/Windows7Search From what I can see these things are still outstanding: # the Windows7Search dll will need loading from the bundle, in a similar way to how it's done for SWT. The bundle will need to contain the binary for both 32- and 64-bit architectures. # escape the filter text # add filename filters # guard resource management in cpp code (exception safety)
Comment on attachment 198060 [details] Patch for the Windows 7 Search native dll. marked the patch as iplog+ note that most of the patch is generated by tooling.
We'll have to discuss how to provide binaries for the bundle. We should compile these on Eclipse.org if that is possible and need reproducible build steps. We can certainly not accept binary dlls from contributors. The other consideration is that linking to Windows headers may constitute a works-with or pre-requisite dependency that requires discussion on the PMC level and tracking in the IP log: http://www.eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf . Please create a separate bug to track that.
(In reply to comment #17) > We'll have to discuss how to provide binaries for the bundle. We should compile > these on Eclipse.org if that is possible and need reproducible build steps. We > can certainly not accept binary dlls from contributors. Agreed, though from what I understand the SWT team builds binaries on their workstations and commits them to CVS. We should be able to do the same (we have at least one committer that can do that). > The other consideration is that linking to Windows headers may constitute a > works-with or pre-requisite dependency that requires discussion on the PMC level > and tracking in the IP log: > http://www.eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf > . Please create a separate bug to track that. Done, see bug 349630.
The latest incubator builds are now available from this repository: http://download.eclipse.org/mylyn/incubator/3.7
Created attachment 198302 [details] Patch to improve the robustness of the Windows Search dll. Hi David, This patch addresses 3 things: 1) add a try/catch block around the COM access into the ADO libraries 2) escape special SQL characters during the construction of the SQL query string 3) handle filename patterns in the SQL query
Created attachment 198617 [details] Patch to add support for Unicode Changed the use of std::string to std::wstring in order to support Unicode.
Created attachment 198618 [details] Patch to add 64-bit builds
I have sent a message to the PMC to clarify if we need a CQ for the native code: http://dev.eclipse.org/mhonarc/lists/mylyn-pmc/msg00076.html .
Pushed febdbe38a8f879da689956050e36d520b2e230c3 Added a new project, @org.eclipse.mylyn.sandbox.search.ui.windows@ which provides platform-specific search implementation for 32- and 64-bit windows. This moves the platform-specific search implementation out of the original @org.eclipse.mylyn.sandbox.search.ui@ bundle. Further feedback is welcome, it's looking pretty much complete. If you'd like to give it a try it can be installed from the nightly incubator repository: https://hudson.eclipse.org/hudson/job/mylyn-incubator-nightly/lastSuccessfulBuild/artifact/org.eclipse.mylyn.incubator-site/target/site/
Created attachment 200466 [details] a summary of new functionality
Created attachment 200472 [details] mylyn/context/zip
Looks great! What is left to do here?
Please use this update site for testing which has signed artifacts: http://download.eclipse.org/mylyn/incubator/3.7
(In reply to comment #27) > Looks great! What is left to do here? Nothing left to do feature-wise. Community feedback is welcome of course. There is the question of if/how this gets moved out of the incubator, which is one of the reasons that I've left this bug open. A UI review would be helpful, can we do that on the task or does it need to occur in a weekly meeting?
We usually do UI reviews as part of the weekly meeting. I'll add it to tomorrow's agenda and will post results here.
I've created the following bug to track the platform contribution: 359320: enhance Eclipse search capabilities to include desktop resources https://bugs.eclipse.org/bugs/show_bug.cgi?id=359320 Marking as resolved/fixed since there's nothing left to do here.
I see lots of Windows 7 stuff attached. How does it work on Mac and Unix? Is the performance acceptable? (In reply to comment #30) > We usually do UI reviews as part of the weekly meeting. I'll add it to > tomorrow's agenda and will post results here. What was the outcome?
bug 359320 is a duplicate of bug 192767 (In reply to comment #32) > I see lots of Windows 7 stuff attached. How does it work on Mac and Unix? Is the > performance acceptable? The search plug-in is separated into two bundles: * org.eclipse.mylyn.sandbox.search.ui - the search UI and extension point, does not implement any search capabilities but provides the framework and UI to do so * org.eclipse.mylyn.sandbox.search.ui.windows - a search implementation, implemented for Windows only The search implementation uses the Windows search index resulting in fast, performant search experience. The Eclipse search experience is comparable to searching in the Windows explorer which uses the same index. A platform-independent search could be implemented over java.io, however this has not yet been done. Mac and Linux are not yet supported. > (In reply to comment #30) > > We usually do UI reviews as part of the weekly meeting. I'll add it to > > tomorrow's agenda and will post results here. > > What was the outcome? We've had several rounds of feedback on the UI, however no formal UI reviews have been performed on the Mylyn call. You can get a good look at what's been done in attachment #200466 [details].
> A platform-independent search could be implemented over java.io, however this > has not yet been done. Mac and Linux are not yet supported. This would be required in order to accept it as fix for bug 192767.
(In reply to comment #34) > > A platform-independent search could be implemented over java.io, however this > > has not yet been done. Mac and Linux are not yet supported. > This would be required in order to accept it as fix for bug 192767. This is something that we are prepared to contribute. Before we do so, it would be useful to get some feedback on what's been implemented to date to see if there are any other shortcomings that we should be aware of, and if the platform team is interested in this feature.
Reopening to consider platform-independent default search implementation
Pushed a platform-independent search implementation as a fall-back in case no platform-specific search is provided. It is of course much slower because of the lack of indexing, however it will work on any platform. As a basic optimization, it only searches the first 16k characters of any given file. Search skips well-known binary files (eg: dll, exe), hidden files and system folders. See attached context, specifically @org.eclipse.mylyn.internal.sandbox.search.ui.provider.BasicSearchProvider@ Steffen (and others) what do you think is the best way to offer platform-specific bundles? Do we need a separate feature for Windows-specific search or is there an easy way to have a single feature optionally include the Windows-specific bundle?
Created attachment 204707 [details] mylyn/context/zip
It should be fine to have all bundles in the same feature as long as the appropriate platform filters are specified.
(In reply to comment #34) > > A platform-independent search could be implemented over java.io, however this > > has not yet been done. Mac and Linux are not yet supported. > This would be required in order to accept it as fix for bug 192767. Dani, platform-independent search is now part of this contribution. What's the next step to getting this into a shape that would be appropriate for inclusion into Platform?
(In reply to comment #40) > (In reply to comment #34) > > > A platform-independent search could be implemented over java.io, however this > > > has not yet been done. Mac and Linux are not yet supported. > > This would be required in order to accept it as fix for bug 192767. > > Dani, platform-independent search is now part of this contribution. What's the > next step to getting this into a shape that would be appropriate for inclusion > into Platform? First I need to take a closer look at the LAF. Is there an update site that allows me to install it into Juno 3.8 M3 for Windows 7, Linux and Mac? If I think this can be included then the next step will be the code review.
The 3.8M3 build isn't yet available (as far as I can see) but I've tested it out on eclipse-SDK-I20111011-1046 on Windows. Installation doesn't yet work for Linux and Mac. I've created bug 361446 to deal with this. You can install from the following update site by selecting the "Desktop Search" feature: http://download.eclipse.org/mylyn/incubator/3.7/
(In reply to comment #42) > The 3.8M3 build isn't yet available (as far as I can see) Yes, sorry - I meant M2. > but I've tested it > out on eclipse-SDK-I20111011-1046 on Windows. Installation doesn't yet work > for Linux and Mac. I've created bug 361446 to deal with this. > You can install from the following update site by selecting the "Desktop > Search" feature: http://download.eclipse.org/mylyn/incubator/3.7/ I'll give it a try - probably after we shipped M3.
> Installation doesn't yet work > for Linux and Mac. I've created bug 361446 to deal with this. Bug 361446 does not seem related.
(In reply to comment #44) > > Installation doesn't yet work > > for Linux and Mac. I've created bug 361446 to deal with this. > Bug 361446 does not seem related. Thanks for catching that. I meant bug 361466
(In reply to comment #42) > Installation doesn't yet work for Linux and Mac. I've created bug 361446 to deal with this. Desktop Search can now be installed on both Windows and non-Windows platforms.
Blogged: http://tasktop.com/blog/eclipse/eclipse-platform-improvements-for-microsoft-windows Mike commented on the blog asking if this would be included in the 4.x stream. If this contribution is accepted, would there be any additional changes required for the 4.x stream to include this feature?
(In reply to comment #47) > Blogged: > http://tasktop.com/blog/eclipse/eclipse-platform-improvements-for-microsoft-windows > > Mike commented on the blog asking if this would be included in the 4.x stream. > If this contribution is accepted, would there be any additional changes > required for the 4.x stream to include this feature? Assuming that the code only uses APIs, this should just work on 4.2.
(In reply to comment #42) > The 3.8M3 build isn't yet available (as far as I can see) but I've tested it > out on eclipse-SDK-I20111011-1046 on Windows. Installation doesn't yet work > for Linux and Mac. I've created bug 361446 to deal with this. > > You can install from the following update site by selecting the "Desktop > Search" feature: http://download.eclipse.org/mylyn/incubator/3.7/ I wanted to try it out and review today but it does not install: 1. install Eclipse SDK I20111011-1046 2. start new workspace 3. Help > Install New Software... 4. enter "http://download.eclipse.org/mylyn/incubator/3.7/" 5. Select All 6. Next ==> Cannot complete the install because one or more required items could not be found. Software being installed: Mylyn Incubator SDK (Incubation) 3.7.0.I20111119-0505 (org.eclipse.mylyn.experimental_sdk_feature.feature.group 3.7.0.I20111119-0505) Missing requirement: Mylyn Tasks Connector: Web Templates (Advanced) (Incubation) 3.7.0.I20111119-0505 (org.eclipse.mylyn.web.tasks_feature.feature.group 3.7.0.I20111119-0505) requires 'org.eclipse.mylyn_feature.feature.group [3.7.0,4.0.0)' but it could not be found Cannot satisfy dependency: From: Mylyn Incubator SDK (Incubation) 3.7.0.I20111119-0505 (org.eclipse.mylyn.experimental_sdk_feature.feature.group 3.7.0.I20111119-0505) To: org.eclipse.mylyn.web.tasks_feature.feature.group [3.7.0,4.0.0) I also tried to only install the Desktop Search ==> Cannot complete the install because one or more required items could not be found. Software being installed: Mylyn Desktop Search (Incubation) 3.7.0.I20111119-0505 (org.eclipse.mylyn.sandbox.search.feature.group 3.7.0.I20111119-0505) Missing requirement for filter properties ~= $0: Mylyn Desktop Search (Incubation) 3.7.0.I20111119-0505 (org.eclipse.mylyn.sandbox.search.feature.group 3.7.0.I20111119-0505) requires 'org.eclipse.mylyn.sandbox.search.ui.windows 0.0.0' but it could not be found
(In reply to comment #49) > I wanted to try it out and review today but it does not install: Thanks Dani, there appears to be a problem with how Tycho is building the p2 repository. You should only need to install the "Desktop Search" feature, but as the error indicates one of the plug-in bundles is not included in the update site. Steffen, any ideas? I looked at the site.xml, feature.xml for desktop search and looked at the "build log":https://hudson.eclipse.org/hudson/job/mylyn-incubator-nightly/185/console and nothing jumped out at me. Is it possible that the filter in the feature.xml is causing it to be exluded from the site?
I wasn't able to reproduce these errors with 3.8.0.I20111027-1800 on gtk/x86_64. The Desktop Search feature installed without problems for me. Dani, which platform where you using? To install the Incubator SDK feature you would also need the main Mylyn repository enabled: http://download.eclipse.org/mylyn/snapshots/weekly. It doesn't contain the Desktop Search source plug-ins though. It would be easiest to get them from Git: http://git.eclipse.org/c/mylyn/org.eclipse.mylyn.incubator.git/tree/ . The sources are in these bundles: org.eclipse.mylyn.sandbox.search-feature org.eclipse.mylyn.sandbox.search.ui.windows org.eclipse.mylyn.sandbox.search.ui
It looks like is not provisioning all bundles. I'll try another packaging type...
This looks like a problem in Tycho (bug 364855). To work around it I removed the arch filter from the feature.xml. The problem is fixed in the latest build. Sorry for the inconvenience. Dani, could you try again?
Thanks Steffen, that works for me. I can install on both Windows and Ubuntu using Eclipse 3.8M3
Created attachment 207595 [details] .log
I was able to install it (Mylyn Desktop Search (Incubation) 3.7.0.I20111125-1648) but when running I get a bunch of exceptions (see attachment 207595 [details]). However, I see the UI and can perform desktop searches. It looks like the Windows specific search code isn't found/installed. My first search took 20 minutes, but it was faster afterwards when using the same file pattern. With each new pattern it again takes around 20 minutes. I don't think the Windows specific search code will help here as I don't have my C: drive indexed. In the Windows Explorer I get a warning that the folder (C:) is not indexed and it also suggests to search in alternative folders. Showing such a warning would be good as well - not sure whether this is possible. The search page should allow to select the initial search folder(s) because in most cases the user already knows where he wants to search and this can reduce the search time drastically. Here some more items I found during my UX review: - It only finds the files but not the individual matches inside a file. This is important, especially for large files. - Matches are not indicated when opening the file in the Eclipse editors. This is something which is expected since it's also there for workspace files. - Next/Previous does not work / can't step through matches - No progress is reported (always stays at 0%). This makes the search look broken, especially because it takes several minutes. - Search is limited to 500 items with no way to change that limit. - Search view's 'Show In' context menu does not make much sense assuming most matches are not inside the workspace. - Search view's 'Open With' context menu seems not to work (grayed out). - The search page needs hints about wild cards and how one can search for more than one pattern (see 'File Search' page). - There should be a mode to see the results in a (directory) tree. - One should be able to remove individual matches from the view. - Case sensitive and regular expression options would also be nice. - A bit more vertical space is needed after the combo box on the search page. - Mnemonics are missing from the labels on the 'Desktop Search' dialog page.
Thanks for taking a look Dani. From your attachment it seems that the DLL isn't being loaded. I'll take a look at that. > My first search took 20 minutes, but it was faster afterwards when using the > same file pattern. With each new pattern it again takes around 20 minutes. I > don't think the Windows specific search code will help here as I don't have my > C: drive indexed. The index won't help when searching folders that aren't indexed - however I've found that it helps in cases where subfolders are indexed. > The search page should allow to select the initial search folder(s) because in most > cases the user already knows where he wants to search and this can reduce the > search time drastically. Good point > Here some more items I found during my UX review: > - It only finds the files but not the individual matches inside a file. This > is important, especially for large files. > - Matches are not indicated when opening the file in the Eclipse editors. This > is something which is expected since it's also there for workspace files. > - Next/Previous does not work / can't step through matches > - No progress is reported (always stays at 0%). This makes the search look > broken, especially because it takes several minutes. Currently we're starting a progress monitor task with total work of @IProgressMonitor.UNKNOWN@. I'm not sure if we can do better than that without first scanning the whole filesystem. What do you think? > - Search is limited to 500 items with no way to change that limit. > - Search view's 'Show In' context menu does not make much sense assuming most > matches are not inside the workspace. > - Search view's 'Open With' context menu seems not to work (grayed out). > - The search page needs hints about wild cards and how one can search for > more than one pattern (see 'File Search' page). > - There should be a mode to see the results in a (directory) tree. > - One should be able to remove individual matches from the view. > - Case sensitive and regular expression options would also be nice. > - A bit more vertical space is needed after the combo box on the search page. > - Mnemonics are missing from the labels on the 'Desktop Search' dialog page. Can you provide an indication of must-haves versus nice-to-haves? My sense is that things like matching within files and next/previous could be very difficult or impossible to implement, especially when matching against files such as Microsoft Word documents, which aren't text files.
> Currently we're starting a progress monitor task with total work of > @IProgressMonitor.UNKNOWN@. I'm not sure if we can do better than that without > first scanning the whole filesystem. What do you think? It depends whether the search API reports progress i.e. whether it knows how many files it already searched. If so, we could count the total of files upfront. > My sense is > that things like matching within files and next/previous could be very > difficult or impossible to implement, especially when matching against files > such as Microsoft Word documents, which aren't text files. The same applies when searching such documents inside the workspace. The LAF when doing a desktop search needs to be the same as when searching such files inside a workspace. > Can you provide an indication of must-haves versus nice-to-haves? The following things are nice to have and can be added later: - There should be a mode to see the results in a (directory) tree. - One should be able to remove individual matches from the view. - Case sensitive and regular expression options would also be nice.
Mylyn has been restructured, and our issue tracking has moved to GitHub [1]. We are closing ~14K Bugzilla issues to give the new team a fresh start. If you feel that this issue is still relevant, please create a new one on GitHub. [1] https://github.com/orgs/eclipse-mylyn
