| [jgit-dev] Gerrit clone time takes ages, because of PackWriter.searchForReuse() |
| Hi all, I am writing to both JGit and Gerrit mailing list, because the problem is found when using Gerrit *BUT* can be relevant to JGit developers and users as well. I have a situation where *certain* Git fetches are taking 10x to 100x of the expected execution time.  The blue dots represent the total execution time of the upload-pack, whilst the red dots represent the time for searching for objects reuse in existing packfiles. The normal expected execution time is the blue band around 10s. The slower execution times are the one over 100s and the *super slow* execution times are the ones coming up to 1000s. The repository is *exactly the same* for all those upload-pack executions. Looking at the code called by the searchForReuse, I ended up in: @Override public void selectObjectRepresentation(PackWriter packer, ProgressMonitor monitor, Iterable<ObjectToPack> objects) throws IOException, MissingObjectException { for (ObjectToPack otp : objects) { db.selectObjectRepresentation(packer, otp, this); monitor.update(1); } } The above is a cycle for *all* objects that goes into the another scan for *all* packfiles inside the selectObjectRepresentation(). The slow clones were going through 2M of objects on a repository with 4k packfiles … the math would say that it went through a nested cycle of 2M x 4k => 8BN of operations. I am not surprised it is slow after all :-) So, it looks like it works the way it is designed: very very slowly. My questions on the above are: 1. Is there anyone else in the world, using Gerrit or JGit, with the same problem? 2. How to disable the search for reuse? (Even if I disable the reuseDelta or reuseObjects in the [pack] section of the gerrit.config, the searchForReuse() phase is trigged anyway) 3. Would it make sense to estimate the combination explosion of the phase beforehand (it is simple: just multiply the number of objects x number of packfiles) and automatically disable that phase? Comments and experiences are more than welcome :-) P.S. I am planning to prepare a patch for implementing 3. If we believe it’s a good idea to auto-disable the phase. Luca. |