[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jgit-dev] Refs creation slowness with Jgit
|
On 2020-09-24 15:03, Matthias Sohn wrote:
I ran the benchmark on one of our staging servers
SAP Java Server VM-1.8.0_212
Benchmark Mode Cnt
Score Error Units
CreateFileSnapshotBenchmark.testCreateFile avgt 25
25233.086 � 1498.716 ns/op
CreateFileSnapshotBenchmark.testCreateFileSnapshot avgt 25
2869.691 � 29.014 ns/op
FileMoveBenchmark.moveFileToExistingDir avgt 25
38.095 � 1.295 us/op
FileMoveBenchmark.moveFileToExistingDirExists avgt 25
42.865 � 0.945 us/op
FileMoveBenchmark.moveFileToMissingDir avgt 25
76.088 � 1.874 us/op
FileMoveBenchmark.moveFileToMissingDirExists avgt 25
62.243 � 2.103 us/op
LookupFileStoreBenchmark.testLookupFileStore avgt 25
138888.966 � 1484.142 ns/op
SimpleLruCacheBenchmark.readwrite avgt 25
530.060 � 27.954 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheRead avgt 25
350.730 � 30.518 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheWrite avgt 25
709.390 � 50.527 ns/op
uname -r
4.12.14-197.34-default
lsb_release -a
LSB Version: n/a
Distributor ID: SUSE
Description: SUSE Linux Enterprise Server 15 SP1
Release: 15.1
Codename: n/a
filesystem is ext4 on EMC Unity SAN storage solution attached via
fiberchannel
results from same host on different volume using btrfs:
Benchmark Mode Cnt
Score Error Units
CreateFileSnapshotBenchmark.testCreateFile avgt 25
22869.500 � 643.885 ns/op
CreateFileSnapshotBenchmark.testCreateFileSnapshot avgt 25
2860.801 � 30.477 ns/op
FileMoveBenchmark.moveFileToExistingDir avgt 25
35.672 � 0.821 us/op
FileMoveBenchmark.moveFileToExistingDirExists avgt 25
41.448 � 1.267 us/op
FileMoveBenchmark.moveFileToMissingDir avgt 25
76.218 � 2.403 us/op
FileMoveBenchmark.moveFileToMissingDirExists avgt 25
62.282 � 1.181 us/op
LookupFileStoreBenchmark.testLookupFileStore avgt 25
139499.887 � 1595.749 ns/op
SimpleLruCacheBenchmark.readwrite avgt 25
514.827 � 20.669 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheRead avgt 25
333.911 � 33.634 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheWrite avgt 25
695.743 � 53.362 ns/op
-Matthias
I ran the custom program shared in the original post on few more of our
machines. The results vary quite a bit for me.
Machine-1
test/foo: 11 ms
test_simple: 3 ms
Machine-2
test/foo: 12 ms
test_simple: 5 ms
Machine-3
test/foo: 7 ms
test_simple: 3 ms
Machine-4
test/foo: 115 ms
test_simple: 4 ms
machine-5
test/foo: 1281 ms
test_simple: 4 ms
Machine-6
test/foo: 1320 ms
test_simple: 5 ms
Machine-7
test/foo: 11 ms
test_simple: 3 ms
On Thu, Sep 24, 2020 at 7:11 PM <kaushikl@xxxxxxxxxxxxxx> wrote:
Hello Matthias,
Thanks for your response. Here are the results of the benchmark on
the
machine.
Benchmark Mode Cnt
Score Error Units
CreateFileSnapshotBenchmark.testCreateFile avgt 25
62917.171 ± 131776.325 ns/op
CreateFileSnapshotBenchmark.testCreateFileSnapshot avgt 25
3263.106 ± 126.821 ns/op
LookupFileStoreBenchmark.testLookupFileStore avgt 25
292185770.706 ± 1830241.558 ns/op
SimpleLruCacheBenchmark.readwrite avgt 25
533.560 ± 36.500 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheRead avgt 25
292.723 ± 30.698 ns/op
SimpleLruCacheBenchmark.readwrite:testCacheWrite avgt 25
774.397 ± 84.068 ns/op
On 2020-09-23 15:26, Matthias Sohn wrote:
On Wed, Sep 23, 2020 at 8:05 PM <kaushikl@xxxxxxxxxxxxxx> wrote:
CC: repo-discuss@xxxxxxxxxxxxxxxx
On 2020-09-23 09:13, kaushikl@xxxxxxxxxxxxxx wrote:
Hello,
When investigating slow Gerrit NoteDB migration times, we
noticed
a
slowness with jgit creating refs. During the migration a ref is
created for each Gerrit change, for example
‘changes/01/1/meta’,
‘changes/02/2/meta’, ‘changes/03/3/meta’ and so on. We
see
the
slowness when creating refs in a new path. This behaviour is
repeatable, i.e each time a ref is created in a new path, it is
slow.
We noticed it is fast on some machines, but we couldn't quite
nail
down why.
I have a small sample program[1] which illustrates the slowness.
It
creates two refs 'refs/heads/test_simple' and
'refs/heads/test/foo'.
On executing the program, I see output:
test/foo: 325 ms
test_simple: 4 ms
My expectation is that 'test/foo' will also be created in order
of
milliseconds and not in order of few hundred milliseconds. The
slowness seems to be stemming from the Files.getFileStore(dir)
call in
FS.FileStoreAttributes.getFileStoreAttributes(Path dir).
Can you run the LookupFileStoreBenchmark [1] on this system ?
On my Mac this yields 63us to lookup a FileStore for a new file
path.
After running the Maven build the benchmarks can be run using
Maven:
$ cd jgit
$ mvn clean install -DskipTests -Dmaven.javadoc.skip=true
$ java -jar org.eclipse.jgit.benchmarks/target/benchmarks.jar
or in Eclipse run the main method of the LookupFileStoreBenchmark
class.
[1]
https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/master/org.eclipse.jgit.benchmarks/src/org/eclipse/jgit/benchmarks/LookupFileStoreBenchmark.java
-Matthias
Specs:
Jgit version is 5.9
$ java -version # also tried with version 1.8.0_252
openjdk version "1.8.0_232"
OpenJDK Runtime Environment (build
1.8.0_232-8u232-b09-0ubuntu1~16.04.1-b09)
OpenJDK 64-Bit Server VM (build 25.232-b09, mixed mode)
$ uname -r
4.15.0-54-generic
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial
Filesystem type is ext4
[1]
package test;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import org.eclipse.jgit.api.Git;
import org.eclipse.jgit.api.errors.GitAPIException;
import org.eclipse.jgit.lib.Repository;
import org.eclipse.jgit.revwalk.RevCommit;
import org.eclipse.jgit.revwalk.RevWalk;
public class Test {
public static void main(String[] args) {
try {
String path = null;
if (args.length == 1) {
path = args[0];
} else {
System.out.println("Repo path must be specified.");
System.exit(1);
}
Path repoPath = Paths.get(path);
long startTimeInNanoSecs;
long estimatedTimeInNanoSecs;
Git git = Git.init().setDirectory(repoPath.toFile()).call();
Repository repo = git.getRepository();
RevWalk walk = new RevWalk(repo);
Files.write(repoPath.resolve("file"), Arrays.asList("line"),
StandardCharsets.UTF_8);
git.add().addFilepattern("file").call();
git.commit().setMessage("create file").setAuthor("author",
"author@xxxxxxxxx").call();
RevCommit commit =
walk.parseCommit(repo.exactRef("refs/heads/master").getObjectId());
String branch = "test/foo";
startTimeInNanoSecs = System.nanoTime();
git.branchCreate().setName(branch).setStartPoint(commit).call();
estimatedTimeInNanoSecs = System.nanoTime() -
startTimeInNanoSecs;
System.out.println(branch + ": " +
TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
branch = "test_simple";
startTimeInNanoSecs = System.nanoTime();
git.branchCreate().setName(branch).setStartPoint(commit).call();
estimatedTimeInNanoSecs = System.nanoTime() -
startTimeInNanoSecs;
System.out.println(branch + ": " +
TimeUnit.NANOSECONDS.toMillis(estimatedTimeInNanoSecs) + " ms");
} catch (IllegalStateException | GitAPIException | IOException
e) {
e.printStackTrace();
}
}
}
_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jgit-dev
_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jgit-dev
_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jgit-dev
