Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jgit-dev] AccessDeniedException in ObjectDirectory#insertUnpackedObject
Here's an updated stack trace.   The line numbers now correspond to the head of jgit's stable-5.7 after my changes were submitted yesterday.  The line numbers in gerrit should match stable-3.0 at least up to the lines in ReceiveCommits; from there they will mismatch due to internal patches.

2020-06-05 03:13:20,571|ERROR|ReceiveCommits-1-for-SSH git-receive-pack /redacted_project_name (redacted_username)|ObjectDirectory| /redacted/path/to/gerrit/gitroot/All-Projects.git/objects/noz13257029704105995557.tmp -> /redacted/path/to/gerrit/gitroot/All-Projects.git/objects/aa/a6442fe902e96a5233b136b973b96a4011fc18
java.nio.file.AccessDeniedException: /redacted/path/to/gerrit/gitroot/All-Projects.git/objects/noz13257029704105995557.tmp -> /redacted/path/to/gerrit/gitroot/All-Projects.git/objects/aa/a6442fe902e96a5233b136b973b96a4011fc18
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:401)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:263)
        at java.base/java.nio.file.Files.move(Files.java:1413)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectory.insertUnpackedObject(ObjectDirectory.java:690)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.insertOneObject(ObjectDirectoryInserter.java:120)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.insert(ObjectDirectoryInserter.java:81)
        at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.insert(ObjectDirectoryInserter.java:57)
        at org.eclipse.jgit.lib.ObjectInserter.insert(ObjectInserter.java:333)
        at com.google.gerrit.server.notedb.IntBlob.tryStore(IntBlob.java:93)
        at com.google.gerrit.server.notedb.RepoSequence$TryAcquire.call(RepoSequence.java:272)
        at com.google.gerrit.server.notedb.RepoSequence$TryAcquire.call(RepoSequence.java:246)
        at com.github.rholder.retry.AttemptTimeLimiters$NoAttemptTimeLimit.call(AttemptTimeLimiters.java:78)
        at com.github.rholder.retry.Retryer.call(Retryer.java:160)
        at com.google.gerrit.server.notedb.RepoSequence.acquire(RepoSequence.java:232)
        at com.google.gerrit.server.notedb.RepoSequence.next(RepoSequence.java:217)
        at com.google.gerrit.server.notedb.Sequences.nextChangeIds(Sequences.java:131)
        at com.google.gerrit.server.git.receive.ReceiveCommits.selectNewAndReplacedChangesFromMagicBranch(ReceiveCommits.java:2260)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommandsUnsafe(ReceiveCommits.java:609)
        at com.google.gerrit.server.git.receive.ReceiveCommits.processCommands(ReceiveCommits.java:518)
        at com.google.gerrit.server.git.receive.AsyncReceiveCommits$Worker.run(AsyncReceiveCommits.java:138)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:182)
        at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:170)
        at com.google.gerrit.server.util.ThreadLocalRequestScopePropagator.lambda$wrapImpl$0(ThreadLocalRequestScopePropagator.java:45)
        at com.google.gerrit.server.util.RequestScopePropagator$1.call(RequestScopePropagator.java:85)
        at com.google.gerrit.server.util.RequestScopePropagator$2.run(RequestScopePropagator.java:116)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:299)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:628)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.base/java.lang.Thread.run(Thread.java:844)

On Thu, Jun 4, 2020 at 6:15 PM David Pursehouse <david.pursehouse@xxxxxxxxx> wrote:
On Thu, Jun 4, 2020 at 5:46 PM Matthias Sohn <matthias.sohn@xxxxxxxxx> wrote:
On Thu, Jun 4, 2020 at 10:02 AM David Pursehouse <david.pursehouse@xxxxxxxxx> wrote:
Running Gerrit stable-3.0 modified to use JGit 5.7 (at the head of stable-5.7) I'm seeing ObjectDirectory#insertUnpackedObject fail with AccessDeniedException when attempting to move the temp file to the new object file.
 
can you share a stack trace ?

Yes, although please note that the line numbers here might not match up because it's running a version where I added extra logging statements:
java.nio.file.AccessDeniedException: /path/to/gerrit/gitroot/All-Projects.git/objects/noz4625652470081084376.tmp -> /path/to/gerrit/gitroot/All-Projects.git/objects/aa/a6442fe902e96a5233b136b973b96a4011fc18
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:401)
        at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:263)
        at java.base/java.nio.file.Files.move(Files.java:1413)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.insertUnpackedObject(ObjectDirectory.java:714)
 
In this case it corresponds to the second call to Files.move in the insertUnpackedObject method.



This is happening consistently while running integration tests on an internal project, but unfortunately since it's a private project it means I can't share the test that reproduces it.  Essentially what it's doing is pushing a lot of changes, and then when Gerrit's internal batch of change ids is exceeded it tries to allocate a new batch which involves writing a blob to a specific ref; it's in this operation that it fails. See [1] and [2] on gerrit where I've added some debug logging in the related classes.

One thing I have found is that adding debug logs in ObjectDirectory#insertUnpackedObject causes it to work properly, so I wonder if there's some kind of timing issue going on.

Has anyone else seen similar issues in this area?

While I have been investigating this I found that the exception handling and logging in the insertUnpackedObject method doesn't make it easy to figure out what's going wrong.  I've pushed a couple of cleanup changes [3].



_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jgit-dev

Back to the top