Community
Participate
Working Groups
Rob, Gail and I had a discussion about the UI for the web connector and how to streamline it in order to ensure that it is simple enough to be part of 1.0, i.e. to hide the implementation details enough to have it usable by non-experts. Here is the interaction that we came up with: 1) Create a new web connector repository from a template as happens now. By default the page will have "anonymous" login checked and probably not editable (but for bug 151602 repository-specific credentials will be supported). 2) Create a query. The query page should be populated by the settings specified for the repository. For a typical interaction the user should not need to modify any string on this page other than the URL (i.e. the regexp and prefix get generated from parameters on the repository settings page). I think that the above also provide compelling reasons to keep the repositories and queries separate. Thoughts?
I just notice that we have bug 155160 already, which covers (2) above. But we can continue to use this one for general discussion.
Mik, I am not completely sure if suggested UI will fit for all cases. For instance, it does not fit well when repostory can have more then one query. SourceForge (you have to have separate queries for bugs and enhancement requests), Google hosting and some others do allow many queries per repository. Also note that now we don't have custom UI for web connector. It is using generic repository prefs.
We definitely need to support multiple queries per repository. But wouldn't that work well as suggested here? The user just has to specify a different URL for each query, which I'm hoping for most cases they will get to paste from the browser. But the key thing is that they won't have to care about the regexp, etc, since that will be generated from the repository settings. I wasn't thinking that this required custom UI (anonymous is already generic, seems like other things can be attributes). But if needed we can make the UI there customizable by the connector.
It is somehow twisted. The only field that can be copied from repository settings is the task url prefix. That field is somehow redundant, because in general prefix can be specific to query (not repository). E.g. SourceForge and GForge have that issue. And the only reason we have prefix on repository is to be able to add existing tasks. I think if we'll change "add existing task" wizard to request both id and the full task url (somehow similar to current weblinked tasks) we can remove prefix from the repository settings. We can also check if one of query templates does match with selected repository url and then fill in query form from that template, but that obviously won't work if user configured his own repository. So, I don't see lot of value in providing this feature.
We still need to figure out how to simplify this UI to make it part of 1.0, because currently it is only suitable for very advanced users (i.e. surfaces way too much implementation detail). Or alternatively we can explore making sandbox components available with release builds (e.g. via a separate update site). Both are fine solutions from my point of view, but I'd still like to explore whether it can be simple enough to become part of the regular distribution. Here's how I see it: all web repositories that we know of use the following conventions for accessing projects (correct me if I'm wrong): * http://a.b/c.cgi${project.id} * ${project.url}/a/b.php * http://a.b/${project.dir}/c For the common case, the only information that the user should need to provide is the ${project.mumble} identifier. So that should be stored on the repository. All the formats then use the identifier and information from the template (e.g. regexp used for query, task prefix URL used for matching, new task URL used for wizard). So on the query page, the user does not need to modify anything, unless they want to paste in a modified URL (e.g. to not list all tasks but a subset). We may also need to hide (e.g. collapse) the regexp by default because any UI that exposes regexp is by definition advanced, and hopefully the user will only need edit the regexp if they have customized HTML presentation (and in that case their repository admin should provide a template). Let me know what you think, or if you have other UI simplification ideas to make this usable by non-experts. But it's also totally fine for us to decide that this is an inherently expert mode of working (relying on parsing web page output), and that the implementation details can not be hidden in a sufficiently generic way, and make this a Sandbox tool. Thinking out of the box only other alternative I see is creating non-generic but tracker-specific web connectors that can come with more advanced pre-configuration than the templates can provide (i.e. each would be it's own connector). I'm CC'ing Raphael since he is the only one I know to have used the web connector, in case he has any feedback.
(Please let me know if this comment gets too off-topic and should be in a different bug) I was actually going to propose to make the web connector UI even more advanced, by adding support for mapping other properties of tasks than id and name, such as whether it is completed, priority etc. But this would actually need some kind of mapping functions which either would require complex UI and would be hard to develop or could be done in Java and provided with the templates. What if in a template extension you could add a class that implements a specific interface and can map query results to task properties. (Maybe I'm way off here, I'm not very familiar with all the internals) But I definitely think all the advanced UI should be hidden by default, maybe letting the user specify only a project ID (as was detailed in bug 155160) --- An idea I just had is to add another more complex generic connector that can use BIRT as it's input and develop the queries as BIRT reports with custom data sources. But I'm not even sure how possible / hard to do that is or if it would be easy enough to use. What could be done about generic repository support for repositories that expose some web services? For example new versions of ChangeLogic add some SOAP web services, including one for querying tasks. Maybe this would be too complex to do in the scope of Mylar and should be left to some external add-on?
Erkki: I see your suggestion along the following lines: if we could understand the format that was returned by some web server by transforming it by some well-defined rule, then we could map these fields. We could take this a step further: we could define the XML format for a bug and a query, and then off-load the task of transforming it onto the repository provider. For example, we could ask that if a query is specified with &ctype=mylar-xml then we get back XML that we understand. Understanding tasks in this format would then be the main job of the generic web connector, and repository providers could implement various capabilities.
Folks, first of all this bug wasn't about mapping or the web parsing backend. It was about user interaction with connector. Besides creating new repository, queries and tasks there are also things like drag-n-drop from the web browser into the Task List view. I am going to look at bug 155160. then we'll see how it will affect the user interaction
Created attachment 53178 [details] Implemented simplified repository and query configuration
Created attachment 53179 [details] mylar/context/zip
I tried to apply this bug get the following errors and warnings. Note that you've got some extra libraries creeping in there. Fyi, the Rome library made another step through the IP approval process today. Severity and Description Path Resource Location Creation Time Id WebRepositoryConnector.TASK_REGEXP cannot be resolved org.eclipse.mylar.sandbox.tests/src/org/eclipse/mylar/sandbox/tests WebRepositoryConnectorTest.java line 60 1162528247906 521167 WebRepositoryConnector.TASK_REGEXP cannot be resolved org.eclipse.mylar.sandbox.tests/src/org/eclipse/mylar/sandbox/tests WebRepositoryConnectorTest.java line 61 1162528247906 521168 WebRepositoryConnector.TASK_REGEXP cannot be resolved org.eclipse.mylar.sandbox.tests/src/org/eclipse/mylar/sandbox/tests WebRepositoryConnectorTest.java line 62 1162528247906 521169 An "source.lib/jdom-1.0.jar" build entry is missing org.eclipse.mylar.sandbox build.properties line 1 1162528243093 521150 An "source.lib/rome-0.8.jar" build entry is missing org.eclipse.mylar.sandbox build.properties line 1 1162528243093 521151 An "source.lib/rome-fetcher-0.7.jar" build entry is missing org.eclipse.mylar.sandbox build.properties line 1 1162528243093 521152
Created attachment 53185 [details] Updated patch I made test pass and did some additional cleanup. However it seems like query hits are not being restored after Eclipse restart. Not sure if it is a bug in WebTaskExternalizer or something else.
Created attachment 53223 [details] Updated patch to fix externalizer There is still issue about Task/WebTask id, but I think we should handle it separately.
Wow, excellent. It is very impressive how much implementation detail this streamlining hides, while still making it accessible. Patch applied. I'm not fixing up the getIdLable() schitzoprenia so that you don't get the "null: " titles. Restore of hits works for me with the latest. Question about usage: I tried to test it for a custom SourceForge repository, but I'm only familiar with the group_id and atid fields. How do those relate to the projectId and categoryId that it fills in? Question related to making this available with the release: would it now be better to provide templates without providing the concrete examples, i.e. so that people have to fill out the attributes? Or perhaps provide two kinds of templates, some for specific projects, and other generic ones that people have to fill out the entries for?
OK, editor titles fixed now and WebTask overrides getIdLabel() to return null. Checking out problem WebTask has with IDs now...
(In reply to comment #14) > Wow, excellent. It is very impressive how much implementation detail this > streamlining hides, while still making it accessible. Right. It is pretty cool. Indeed. :-) > Question about usage: I tried to test it for a custom SourceForge repository, > but I'm only familiar with the group_id and atid fields. How do those relate > to the projectId and categoryId that it fills in? group_id == projectId atid == categoryId Perhaps I should of kept same names to remove this confusion. I just thought those names make more sense. I might be also overthought projectId thing for the SourceForge (this is the only template where we have such issue). So, maybe we should remove from the repository url. That would allow to have single repository for the entire SourceForge. categoryId is needed because it goes to the task prefix, since task id is unique per category. > Question related to making this available with the release: would it now be > better to provide templates without providing the concrete examples, i.e. so > that people have to fill out the attributes? Or perhaps provide two kinds of > templates, some for specific projects, and other generic ones that people have > to fill out the entries for? Most of those templates allows just change repository url and don't require tweaking parameters like it is for SourceForge. On the other hand, I am thinking if we should make query url visible in regular options...
Apparently I forgot to update "Add existing task". It should handle custom params (if any).
Eugene: yes, I think that you should keep the same names for all cases, to make it more obvious how this information should be extracted from the URL. Sounds fine to remove the repository URL for source forge, but might not be worth it if it SourceForge is the only case for this. I agree about exposing the query URL. Users of this connector need to be aware of it anyway, since they needed to get their parameters from it. Hmmm... you could be a bit fancier and try to extrac the property values from the query URL that they paste in, and then all that configuring the web connector would require is pasting in a query URL??
I just tried the new interface now with a mantis repository, and no doubt it is much better than before, but I have some questions: 1 - Was authentication implemented? 2 - Is "Validate Settings" working? I clicked it, but didn't get any response, so I don't know if it was configured correctly. 3 - There is a bug when creating a query, the "Advanced Configuration" section is being cut inside the dialog. 4 - I don't remember how I configured the mantis connector last time I tried (some months ago). Anyway, I configured the default Mantis repository (just to test), and tried to create a query on it. I used the default values and didn't changed anything. The query returns nothing. Is there any parameter I should set?
(In reply to comment #19) > I just tried the new interface now with a mantis repository, and no doubt it is > much better than before, but I have some questions: > > 1 - Was authentication implemented? We have separate unresolved issue entered for that. Please comment there if you have specific requirements: 151602: [Web connector] Support queries that require login https://bugs.eclipse.org/bugs/show_bug.cgi?id=151602 > 2 - Is "Validate Settings" working? I clicked it, but didn't get any response, > so I don't know if it was configured correctly. There is nothing to validate actualy. If you have specific ideas please enter an enhancement request. > 3 - There is a bug when creating a query, the "Advanced Configuration" section > is being cut inside the dialog. Right. That one is I know about. Though I didn't want to make dialog too big by default when advanced section is not expanded. Will have to look if it is possible to set all the min sizes on children of that panel. > 4 - I don't remember how I configured the mantis connector last time I tried > (some months ago). Anyway, I configured the default Mantis repository (just to > test), and tried to create a query on it. I used the default values and didn't > changed anything. > The query returns nothing. Is there any parameter I should set? I just tried this morning and Mantis query from standard templates had some results. Can you please try to recreate repository for Mantis.
(In reply to comment #20) > We have separate unresolved issue entered for that. Please comment there if you > have specific requirements: > > 151602: [Web connector] Support queries that require login > https://bugs.eclipse.org/bugs/show_bug.cgi?id=151602 > > There is nothing to validate actualy. If you have specific ideas please enter > an enhancement request. Sorry, I completely forgot about this bug and I thought it was fixed since there is a "validate" button there. BTW, if it is not implemented, why keep the button? > > > 3 - There is a bug when creating a query, the "Advanced Configuration" section > > is being cut inside the dialog. > > Right. That one is I know about. Though I didn't want to make dialog too big by > default when advanced section is not expanded. Will have to look if it is > possible to set all the min sizes on children of that panel. Or perhaps use a ScrolledComposite. > > I just tried this morning and Mantis query from standard templates had some > results. Can you please try to recreate repository for Mantis. > I tried, but is not working. I'm pretty sure there is some f***ing proxy issue here. Not sure if I changed the right proxy configuration preference page :) I'll try again at home, and keep you posted.
I tried in my machine at home, and it worked. I think there should be some proxy issue in my other machine, but anyway, I think there is some error handling missing on web connector, because if I create a repository on some non-existing URL, like: http://dsklfhjdslkfj.com and after that, I create a query on it, it does not report any error. Should I open another bug on this, Eugene?
Yes please. It is always good idea to keep tasks separate.
Done. See bug#164219.
Looks like we are done here and additional bugs can go on separate reports.
