Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [p2-dev] iterators, queries, and collectors....oh my!!

Hi, Ian...

>1. Queries that cannot be done asynchronously (because of their implementation). For example, if a query needs all the results to make decisions, (get the IU with the greatest version).

I think this is where Thomas was asking for a finite set of non-opaque queries. If we have a LatestIUVersionQuery, then a naive implementation (SimpleMetadataRepository) could simply fetch everything and then check the versions, but a repository with server-side logic could actually do the work on the server side. Same with filtering. Imagine a PatternMatchQuery. If I query all IU's with "*rel" in the name, a naive implementation will fetch everything and pattern match the results, but a smarter repo would search on the server side and accumulate the results.

Another lifecycle thing to consider is how/when results are available. For LatestIUVersionQuery, any implementation couldn't possibly know the results until it had looked at everything. For PatternMatchQuery, the results are cumulative and it would be nice if some results could be made available incrementally so that the UI could show a subset of results faster. For example, I wouldn't want to make the user wait until every IU with "*rel" were found before showing anything (as we do today). And I wouldn't want to block on hasNext() if the queryable was still working, I'd want a way to know if the repo is still working on the next one, or is truly finished. For that reason, I'm not sure if Iterator is enough. In the UI I imagine a pattern where a job would periodically fetch the results and populate the view, and keep rescheduling the job until the collector was really done. Something like

public IStatus run() (IProgressMonitor monitor) {
while (collector.hasNext()) {
// collect for the UI
}
if (collector.mightHaveNext()) {
this.schedule(someDelay);
return somePartiallyDoneInfoStatus;
} else
return Status.OK;
}

For these reasons I don't see that the collector itself or the caller knows how fast or slow the query will be, as it's a combination of the queryable and the query itself that determine how the work is to proceed.

In the long run I think the UI should be prepared for the random performance characteristics of a query, it's just a matter of time whether we could do so for 3.5 (ironing out all these queryable issues first). Handling pattern matching in the query would remove the first major road block in this area.

One more thing - the nasty "uncategorized" query might be made to go away by us mandating that only categorized things are shown by the UI when working in category mode (vs. assembling the uncategorized info). That's probably the best way to make that problem go away rather than making it work in a query. (See https://bugs.eclipse.org/bugs/show_bug.cgi?id=227675#c16).

susan
Inactive hide details for "Ian Bull" <irbull@xxxxxxxxxxxxxxxxx>"Ian Bull" <irbull@xxxxxxxxxxxxxxxxx>




          "Ian Bull" <irbull@xxxxxxxxxxxxxxxxx>
          Sent by: p2-dev-bounces@xxxxxxxxxxx

          12/23/2008 10:13 AM
          Please respond to P2 developer discussions



To: "P2 developer discussions" <p2-dev@xxxxxxxxxxx>
cc:
Subject: Re: [p2-dev] iterators, queries, and collectors....oh my!!


Susan and others,

I am going to try and re-write some of the complex collectors in terms of Queries. If this is done, then you're right, we can simplify the collectors.

I have also been thinking about asynchronous data collection, and there are two things that may impede this:

1. Queries that cannot be done asynchronously (because of their implementation). For example, if a query needs all the results to make decisions, (get the IU with the greatest version).
2. IQueryables that do not support asynchronous behaviour. This happens, for example, if the multiple read - single write problem is not protected against.

What do you think about changing the signature of IQueryable to be:

public Collector query(Query query, Collector collector, IProgressMonitor monitor, boolean asynchronous);

(note the boolean). The asynchronous is a hint that says, if both the IQuerable and Query supports asynchronous, then the data should be retrieved asynchronously, and the query method should be "quick". If one of these do not support this behaviour , then the data is retrieved synchronously.

Note: From a user of this API not much changes (just the performance of the call). In both cases, a caller will get the data back from the collector.

Anyways, before any of this can be considered, the existing "complex" collectors should be simplified. I will start on that.

Cheers,
Ian

On Tue, Dec 23, 2008 at 8:58 AM, Susan Franklin McCourt <susan_franklin@xxxxxxxxxx> wrote:
    Hi, Ian, Thomas and others:

    I've been commenting in various bugs on this topic, but I feel like my intent gets lost with comments sprinkled around bugs that are discussing very specific parts of the API.
    I think we need to look collectively at the problem of remote queries, large data sets, filtering, etc.

    Specifically, I think we need to define and implement queries for the routine tasks that are causing complexities in the collectors themselves, before defining a solution for async collectors.
    This also helps more generally with the ability to optimize queries later on, as Thomas points out in
    https://bugs.eclipse.org/bugs/show_bug.cgi?id=256412.
    I think having done this work, most collectors will simplify to doing nothing or possibly just wrapping the results (as the UI does). At that point, a pluggable acceptor might work (not sure).

    Ideally, I'd hope that a collector would not need to operate in a special async mode.

    >From the UI point of view, these queries are needed:
    - latest version of an IU
    - categories (this is really just a property query)
    - IU's in a particular category (that also satisfy a UI visibility property query)
    - IU's that are not categorized (that also satisfy a UI visiblity property query)
    - IU's that pattern match a string

    That last one would solve most of my pain with regards to lazy repo loading and filtering. If we can push pattern matching to the queryable side where it belongs, the UI would have no need to force a full iteration of all repos in special cases.


    susan




    _______________________________________________
    p2-dev mailing list

    p2-dev@xxxxxxxxxxx
    https://dev.eclipse.org/mailman/listinfo/p2-dev



--
R. Ian Bull, PhD
Software Developer, EclipseSource

http://www.ianbull.com
http://blog.ianbull.com_______________________________________________
p2-dev mailing list
p2-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/p2-dev

GIF image

GIF image

GIF image


Back to the top