Re: [jgit-dev] Creating new refs on a repository with 100k packed-refs i

[kaushikl@xxxxxxxxxxxxxx]

On 2020-10-06 14:05, Matthias Sohn wrote:
> On Tue, Oct 6, 2020 at 9:17 PM <kaushikl@xxxxxxxxxxxxxx> wrote:
>
> > Hi,
> >
> > We have noticed that creating a new ref takes ~600ms on a repository
> >
> > with around 100k packed-refs.
> >
> > My test repository:
> >
> > $ git count-objects -v
> > count: 0
> > size: 0
> > in-pack: 100042
> > packs: 1
> > size-pack: 9734
> > prune-packable: 0
> > garbage: 0
> > size-garbage: 0
> >
> > $ find refs/ -type f | wc -l
> > 0
> >
> > Specs:
> >
> > Machine has 32 cores and 250G RAM.
> >
> > $ uname -r
> > 4.4.0-165-generic
> >
> > $ lsb_release -a
> > No LSB modules are available.
> > Distributor ID:    Ubuntu
> > Description:    Ubuntu 16.04.6 LTS
> > Release:    16.04
> > Codename:    xenial
> >
> > Jgit version is 5.9
> >
> > $ java -version
> > openjdk version "1.8.0_242"
> > OpenJDK Runtime Environment (build
> > 1.8.0_242-8u242-b08-0ubuntu3~16.04-b08)
> > OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
> >
> > Consider a small program[1] which creates a ref 'simple'. On
> > executing
> > the program on my test repository, I see output:
> >
> > simple: 677 ms
> >
> > This seems slow. Is this expected behavior with jgit?
> >
> > [1]
> > package test;
> >
> > import java.io.File;
> > import java.io.IOException;
> > import java.util.concurrent.TimeUnit;
> >
> > import org.eclipse.jgit.api.Git;
> > import org.eclipse.jgit.api.errors.GitAPIException;
> > import org.eclipse.jgit.lib.Repository;
> > import org.eclipse.jgit.revwalk.RevCommit;
> > import org.eclipse.jgit.revwalk.RevWalk;
> >
> > public class Test {
> > public static void main(String[] args) {
> > try {
> > String path = null;
> > if (args.length == 1) {
> > path = args[0];
> > } else {
> > System.out.println("Repo path must be specified.");
> > System.exit(1);
> > }
> > String branch = "simple";
> > try (Git git = Git.open(new File(path))) {
> > Repository repo = git.getRepository();
> > RevWalk walk = new RevWalk(repo);
> > RevCommit commit =
> > walk.parseCommit(repo.exactRef("refs/heads/master").getObjectId());
> > long startTimeInNanoSecs = System.nanoTime();
> >
> > git.branchCreate().setName(branch).setStartPoint(commit).call();
> > long estimatedTimeInNanoSecs = System.nanoTime() -
> > startTimeInNanoSecs;
> > System.out.println(branch + ": " +
> > TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
> > }
> > } catch (IllegalStateException | GitAPIException | IOException
> > e) {
> > e.printStackTrace();
> > }
> > }
> > }
>
> can you retry with this jgit patch [1] ?
>
> [1] https://git.eclipse.org/r/c/jgit/jgit/+/170138

I have given this a try with the patch and don't see an improvement. 
Note that, I have been doing this test on a machine with good 
Files.getFileStore(Path) performance. So, it is likely not related to 
that. Consider a simple program[2] which calls Files.getFileStore(Path). 
It finishes in 2ms on the machine. Output from [2]:

Files.getFileStore: 2 ms

I have also tried this test on my Mac and I see ref creation taking 
around 250ms-300ms(which seems slow?). On executing a simple program 
similar to the one I shared in my original post, but updated slightly to 
create 3 refs, I see times like below on my Mac:

bar/foo1: 277 ms
bar/foo2: 277 ms
bar/foo3: 249 ms

Here's the state of the repo on my Mac:

$ git count-objects -v
count: 1
size: 4
in-pack: 100058
packs: 1
size-pack: 9736
prune-packable: 0
garbage: 0
size-garbage: 0

$ cat packed-refs | wc -l
100083

[2]
package test;

import java.nio.file.Path;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.TimeUnit;

public class Test {
  public static void main(final String[] args) {
    try {
      String path = null;
      if (args.length == 1) {
        path = args[0];
      } else {
        System.out.println("path must be specified.");
        System.exit(1);
      }
      final Path repoPath = Paths.get(path, new String[0]);
      long startTimeInNanoSecs = System.nanoTime();
      Files.getFileStore(repoPath);
      long estimatedTimeInNanoSecs = System.nanoTime() - 
startTimeInNanoSecs;
      System.out.println("Files.getFileStore: " + 
TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
    } catch (IllegalStateException | IOException e) {
      e.printStackTrace();
    }
  }
}


> -Matthias