Bug 189518 - support import/export of Task List queries
Summary: support import/export of Task List queries
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Mylyn (show other bugs)
Version: unspecified   Edit
Hardware: Macintosh Mac OS X - Carbon (unsup.)
: P2 enhancement (vote)
Target Milestone: ---   Edit
Assignee: Jevgeni Holodkov CLA
QA Contact:
URL:
Whiteboard:
Keywords: helpwanted
Depends on:
Blocks:
 
Reported: 2007-05-28 17:30 EDT by Jason van Zyl CLA
Modified: 2010-03-25 16:08 EDT (History)
3 users (show)

See Also:


Attachments
Core changes to support query import / export (14.25 KB, patch)
2007-07-10 12:07 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
mylyn/context/zip (46.97 KB, application/octet-stream)
2007-07-10 12:07 EDT, Jevgeni Holodkov CLA
no flags Details
Query Import/Export UI (from context menu) (17.60 KB, patch)
2007-07-16 12:30 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
mylyn/context/zip (62.82 KB, application/octet-stream)
2007-07-16 12:30 EDT, Jevgeni Holodkov CLA
no flags Details
Queries importing/exporting and name conflict resolving (20.54 KB, patch)
2007-07-20 09:55 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
mylyn/context/zip (68.69 KB, application/octet-stream)
2007-07-20 09:55 EDT, Jevgeni Holodkov CLA
no flags Details
Query import/export UI (10.32 KB, patch)
2007-07-20 10:45 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
mylyn/context/zip (71.21 KB, application/octet-stream)
2007-07-20 10:45 EDT, Jevgeni Holodkov CLA
no flags Details
Queries importing/exporting and name conflict resolving (+tests fixed) (22.18 KB, patch)
2007-07-22 08:00 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
Query export/import with repositories information in Zip file included (22.64 KB, patch)
2007-08-02 12:33 EDT, Jevgeni Holodkov CLA
no flags Details | Diff
mylyn/context/zip (83.44 KB, application/octet-stream)
2007-08-02 12:33 EDT, Jevgeni Holodkov CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jason van Zyl CLA 2007-05-28 17:30:08 EDT
I would like to set up a list of queries and then share them with my team easily like the contexts are. If I make the list I know includes everything that my developers should be looking at then my developers should not have to re-create those queries.

Might want to distinguish between a team query list and a personal query list. But in a team enviroment the vast majority of the view into the system should be seen in the same way by each developer.
Comment 1 Eugene Kuleshov CLA 2007-05-28 18:01:35 EDT
It seems like generic functionality that should be added into Task List Import/Export wizard
Comment 2 Mik Kersten CLA 2007-05-28 23:42:32 EDT
I agree that the best first pass at this would be via import/export.  Beyond Mylar's model of doing sharing via repositories (e.g. for JIRA saved filters can be retrieved) Eclipse's model for sharing between workspaces is current based on import/export of settings.  Our dialog could make it easy to import and export just the query specifications.

The other approach to this that I have been considering is to make XML templates for queries (bug 183948) that could be specified by plug-ins or imported themselves.
Comment 3 Eugene Kuleshov CLA 2007-05-28 23:53:18 EDT
Mik, I think it should actually allow to select individual queries for export, and/or respect selection from the Task List in order to export selected elements
Comment 4 Mik Kersten CLA 2007-05-29 07:58:43 EDT
Jason: if you're happy with the import/export idea I suggest we turn this bug into that as Eugene suggests.  I still think that we have the important use case of having query templates that can be provided by plug-ins or imported (bug 183948) that could but I see that more as a generic way of making it as to creating common queries (e.g. all bugs assigned to <user>) rather than as a way of externalizing query settings.

Just fyi, the only chance this has of making 2.0 is with a contribution.
Comment 5 Willian Mitsuda CLA 2007-05-29 18:13:19 EDT
I think the original idea could be implemented in a more automatic way, like "launch" files, from platform/debug:

http://www.eclipsezone.com/eclipse/forums/t74693.html

So we could have "query" files, a file containing query information, which are shared using a team provider, and contributed to task list by scanning the workspace.
Comment 6 Eugene Kuleshov CLA 2007-05-29 19:00:02 EDT
Willian, those "query" files would be practically the same thing as exported query. So, you can commit them to vcs and import back into the task list from checked out project. So, it probably does make sense to add an option to save/read that exported file from the workspace.

Contributing those things automatically is probably separate issue. Though there are some interesting options to explore. BTW, bug 178883 also suggested to share task contexts trough team/vcs provider.
Comment 7 Willian Mitsuda CLA 2007-05-29 20:36:44 EDT
This is my question: on this "exported query" scenario, if you modify the query, do you have to manually "export" it to a file and commit it to CVS?

I like the debug/launch files because they are "magic": checkout the project and they appear on launch configuration list. Modify the configuration, commit, and your friend can synchronize and update the config on another machine. Delete the project, and they are gone from your workspace.
Comment 8 Eugene Kuleshov CLA 2007-05-29 20:52:31 EDT
(In reply to comment #7)
> This is my question: on this "exported query" scenario, if you modify the query,
> do you have to manually "export" it to a file and commit it to CVS?

Yes. But we should take one thing at a time. It is relatively easy to implement export and it will be better then nothing. 

So, I think it would be a good idea to have separate bug for all magic stuff. :-)
Comment 9 Mik Kersten CLA 2007-05-30 07:56:03 EDT
(In reply to comment #8)
> Yes. But we should take one thing at a time. It is relatively easy to implement
> export and it will be better then nothing.

I agree.  Once we have import/export working and externalizing to files such additional streamlining could be supported.  
 
> So, I think it would be a good idea to have separate bug for all magic stuff.
> :-)

Willian: please do file a bug and provide the contents from your comment#5.
Comment 10 Jason van Zyl CLA 2007-05-30 08:13:41 EDT
I will draw up a user story for this as it's more sophisticated then import/export. In the short term what would work for import/export and what I would like to do is create some interfaces for a sink (or set of sinks) and a source (or set of sources) for things like contexts, task lists, task repositories so that you could easily expell them to a file but also have something more interactive like a server which manages sharing items amongst your team. I don't like pushing all this information in attachments to the issue management system, but I admit it's a great fall back. This is something I would like to implement and would be happy to do it as a contribution making sure the existing attachment mechanism works as well as adding simple file import/export. I'm going to talk some more with Eugene and I will write up a short proposal if that's acceptable.
Comment 11 Mik Kersten CLA 2007-05-30 14:23:00 EDT
Great!  In my experience XML- based files are a great sink because of interoperability (e.g. with weird clients like IM which don't support anything other than native file drag-and-drop).  Once files work it is relatively straightforward to make that same code sink the data into something else, e.g. to embed it into a pom.xml or another data store.
Comment 12 Willian Mitsuda CLA 2007-06-02 23:29:12 EDT
(In reply to comment #9)
> Willian: please do file a bug and provide the contents from your comment#5.
> 

Done: bug#190673.
Comment 13 Mik Kersten CLA 2007-06-05 20:36:10 EDT
Jvegeni: reassigning to you since this is so closely related to bug 189514.
Comment 14 Jevgeni Holodkov CLA 2007-07-10 12:07:44 EDT
Created attachment 73436 [details]
Core changes to support query import / export

Actually, this is only about creating a query. The patch that will synchronize the imported query with the repository will follow. I think I'll try to reuse NewBugzillaQueryWizard.performFinish() code, but will create a generic detection of connector.
Comment 15 Jevgeni Holodkov CLA 2007-07-10 12:07:46 EDT
Created attachment 73437 [details]
mylyn/context/zip
Comment 16 Jevgeni Holodkov CLA 2007-07-10 12:20:44 EDT
Or, maybe, it is good to start a Wizard after you select the query and click to import it? Then you will be able to review or tweak the query if needed. In that case I would be good to have a checkbox "Preview" or "Review" or "show me the query", so if you don't want to change the query, you will just get it in your task list.
Comment 17 Mik Kersten CLA 2007-07-10 12:52:32 EDT
Patch from comment#14 applied, good stuff Jevgeni.  Good that you treaded carefully on the changes in TaskListWriter and covered your changes with tests.  In general, if you are going to make changes to this class or similar core framework classes please propose them first.

Are you ready to contribute the UI portion?
Comment 18 Mik Kersten CLA 2007-07-10 12:53:56 EDT
Also: I added an @author tag for you to QueryExportImportTest and organized the imports, please make sure that you have no warnings before committing and add @author tags to any classes that you have added or significantly modified.
Comment 19 Mik Kersten CLA 2007-07-10 19:51:43 EDT
Jevgeni: are we done here?
Comment 20 Jevgeni Holodkov CLA 2007-07-11 02:32:27 EDT
(In reply to comment #19)
> Jevgeni: are we done here?

Mik: No, not yet. I am going to create UI part as well.
Comment 21 Jevgeni Holodkov CLA 2007-07-16 12:30:18 EDT
Created attachment 73863 [details]
Query Import/Export UI (from context menu)
Comment 22 Jevgeni Holodkov CLA 2007-07-16 12:30:22 EDT
Created attachment 73864 [details]
mylyn/context/zip
Comment 23 Jevgeni Holodkov CLA 2007-07-16 12:32:13 EDT
Is queries exporting/importing via drag&drop needed?
Comment 24 Mik Kersten CLA 2007-07-16 12:35:50 EDT
Yes, that would definitely be a benefit because for some users it will be more natural and/or quicker to do it via DnD.  Please create a new bug report for that.  I plan on reviewing this patch today.
Comment 25 Jevgeni Holodkov CLA 2007-07-16 13:06:26 EDT
(In reply to comment #24)
> Yes, that would definitely be a benefit because for some users it will be more
> natural and/or quicker to do it via DnD.  Please create a new bug report for
> that.  I plan on reviewing this patch today.
> 

Bug 196683 - support queries importing/exporting by drag&drop (https://bugs.eclipse.org/bugs/show_bug.cgi?id=196683) 
Comment 26 Mik Kersten CLA 2007-07-17 12:53:17 EDT
Jevgeni: I reviewed the patch and have a couple of questions about it that I'll ask on today's call.
Comment 27 Jevgeni Holodkov CLA 2007-07-20 07:41:33 EDT
I want to add to TaskList a method insertQuery(query), that will check if the TaskList contains a query with the same handle identifier as a method argument. If it does - then the suffix [x] (where x in an incrementing number starting from 1) will be added to the inserted query identifier. The reason I need this is solving name conflict, which will occur if I try to add a query with the name, that already exists in the TaskList. Currently, in addQuery(query) it leads to overriding the existing query with a new one. I cannot change addQuery, since it is also used in "edit" scenario, when such effect is exactly what is needed. Using TaskList.getQueries() is also quite inconveniently, since it returns a Set, which makes me to iterate (or create a Map based on that) values in order to find, if there is a query with the handle identifier I search for. Any ideas on that? Does it make sense to create insertQuery() method?
Comment 28 Jevgeni Holodkov CLA 2007-07-20 09:55:26 EDT
Created attachment 74255 [details]
Queries importing/exporting and name conflict resolving

Changed TaskListWriter to support exporting many queries and added insertQueries() to TaskListManager, that also resolves any name conflict. Currently, no changes to TaskList were done (as described in my previous comment).
Comment 29 Jevgeni Holodkov CLA 2007-07-20 09:55:30 EDT
Created attachment 74256 [details]
mylyn/context/zip
Comment 30 Jevgeni Holodkov CLA 2007-07-20 10:45:36 EDT
Created attachment 74262 [details]
Query import/export UI

Supports exporting many queries and once and import many queries from one file. In the end of import, the MessageDialog tells what was imported and what not (if there were any broken queries). plugin.xml also contains commented out CloneAction declaration. The QueryCloneAction UI will follow after these patches are applied.
Comment 31 Jevgeni Holodkov CLA 2007-07-20 10:45:38 EDT
Created attachment 74263 [details]
mylyn/context/zip
Comment 32 Jevgeni Holodkov CLA 2007-07-22 08:00:09 EDT
Created attachment 74311 [details]
Queries importing/exporting and name conflict resolving (+tests fixed)

fixed tests (were running without TasksUiPlugin.getSynchronizationManager().setForceSyncExec(true) and failing when were runned in AllTests)
Comment 33 Mik Kersten CLA 2007-07-24 01:54:48 EDT
Patches applied and looking good!  Will iterate some on the UI tomorrow.
Comment 34 Eugene Kuleshov CLA 2007-07-25 13:31:31 EDT
I've tried query export stuff and noticed that exported file does not have repository info, but does include query hits. I think the latter is completely unnecessary, but former is needed in order to re-create query when repository does not exist at the time of import.
Comment 35 Jevgeni Holodkov CLA 2007-07-26 12:44:17 EDT
Yes, right, I didn't include the TaskRepository element to the file. As I've understood, repositories.xml is currently created using SAX, so there is no straight way to reuse TaskRepository element creation in DOM, but only by copying/reproducing the existing SaxRepositoriesWriter functionality in a DOM writer. If this is acceptable, I'll go that way.
Comment 36 Jevgeni Holodkov CLA 2007-07-26 12:49:20 EDT
Oh, probably one way more is to add  repositories.xml with a concrete repository to the query.xml.zip archive.
Comment 37 Mik Kersten CLA 2007-07-27 18:50:55 EDT
Rob: please review the UI and any additional patches.

Jevgeni: in my absence please CC Rob on all of your bugs that need a response/review.
Comment 38 Robert Elves CLA 2007-07-28 18:40:26 EDT
 (In reply to comment #36)
> Oh, probably one way more is to add  repositories.xml with a concrete repository
> to the query.xml.zip archive.
This sounds like the most straight forward solution, and I agree with Eugene's point regarding exclusion of hits since they aren't necessary.

Ui nits:
* Upon -import- of a query the resulting dialog title is: "Query -Export- Completed". I'm not seeing a dialog after export.
* Export action needs export icon
* Alignment of query names imported within dialog need to be tabbed in from left (currently further left than sentence).  
* Progress indication for these actions may become necessary at some point
Comment 39 Robert Elves CLA 2007-07-28 19:29:31 EDT
Currently Import, Export, Clone Query take up a space each within the context menu. We should consider making this a sub menu style:

Query... > 
      [Import]
      [Export]
	  [Clone]

Another (perhaps long term) thought is that this import functionality is a more specific version of what we already offer for task data import under File > Import > Mylyn. So this may be a special case of such a wizard where after selecting the file to import the user is presented with what repositories and queries are available. The user selects what they want, followed by these selections and any required repositories being imported.
    
Comment 40 Eclipse Webmaster CLA 2007-07-29 09:22:55 EDT
Changing OS from Mac OS to Mac OS X as per bug 185991
Comment 41 Jevgeni Holodkov CLA 2007-08-02 12:33:50 EDT
Created attachment 75245 [details]
Query export/import with repositories information in Zip file included

Changes:

TaskRepositoriesExternalizer:
* Extracted ZipEntry creation to a separate method, that is used to add repositories.xml file to the xml.zip. 
* Fixed reading XML ZIP, to search for the repository entry (now there can be more that a single file inside)

TaskRepositoryManager:
added a method, which adds repositories if they are not in the system yet.

QueryExportImportTest:
added repository export/import tests

MockTaskListFactory:
fixed Query creation

QueryImportAction:
added code to import repositories along with queries

TaskListWriter:
added TaskRepositoriesExternalizer and fixed writing/reading queries. Also, I have put reading repositories under this class as well.

TaskListDropAdapter:
Fixed importing queries through dropping it on the taskList
Comment 42 Jevgeni Holodkov CLA 2007-08-02 12:33:53 EDT
Created attachment 75246 [details]
mylyn/context/zip
Comment 43 Jevgeni Holodkov CLA 2007-08-02 12:35:49 EDT
Actually, I would extract all export/import related activities to a separate manager and to create a separate core object that will hold all read data (Tasks, Contexts, Repositories, Queries).

Rob, if possible, review this patch first, since I will need to update these files when start working on another bugs.
Comment 44 Robert Elves CLA 2007-08-02 17:39:59 EDT
 (In reply to comment #41)
Patch applied and verified. Updated import success dialog to read 'import' rather than 'export'.
Comment 45 Robert Elves CLA 2007-08-02 17:53:11 EDT
Note that tasks are still being included as part of exported query.
Comment 46 Eugene Kuleshov CLA 2007-08-02 18:20:39 EDT
We need to take a pass trough externalization API. Here is few weird things I've noticed:

-- TaskRepositoriesExternalizer can externalize stuff to File or to ZipOutputStream, but not to the OutputStream.
-- TaskRepositoriesExternalizer.writeRepositoriesToZipEntry() method actually externalizing to ZipOutputStream. Should it be just writeRepositories() ?
-- Same for TaskListWriter.writeTaskListToZipEntry()
-- newly added method TaskRepositoryManager.insertRepositories() is really confusing. Such logic is not really belong to the TaskRepositoryManager and should be done on the caller side who don't want to add already existing repositories

Also, do we really need confirmation/OK dialog on successful import? This seem redundant piece of UI.
Comment 47 Mik Kersten CLA 2007-10-11 23:04:15 EDT
Jevgeni: are we done with the core work here?
Comment 48 Jevgeni Holodkov CLA 2007-10-14 13:41:13 EDT
(In reply to comment #47)
> Jevgeni: are we done with the core work here?
> 

Yes, I think so, but I would refactor it first by creating import/export expert interface, then by refactoring the rest of the underlaying code (as Eugene proposed in comments)
Comment 49 Mik Kersten CLA 2007-10-16 12:56:06 EDT
Jevgeni: please make a new bug for the refactoring, since the core work here is done.