[jgit-dev] Concurrent run of repack and clone sometimes fails

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

[jgit-dev] Concurrent run of repack and clone sometimes fails

From: hrebejk <hrebejk@xxxxxxxxx>
Date: Mon, 30 Nov 2020 16:18:08 +0100 (CET)
Delivered-to: jgit-dev@xxxxxxxxxxx
List-archive: <https://www.eclipse.org/mailman/private/jgit-dev>
List-help: <mailto:jgit-dev-request@eclipse.org?subject=help>
List-subscribe: <https://www.eclipse.org/mailman/listinfo/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://www.eclipse.org/mailman/options/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=unsubscribe>

Hi,

We are running several servers with jGit. We need to run repack from time to time to keep the repos performant. I.e. after push we test how many small packs are in the repo and when a threshold is reached we run the repack.
After upgrading jGit version we've found that if someone does the clone at the time repack is running the clone sometimes (not always) fails because the repack removes .pack file used by the clone. Server exception and client error attached.

I've tracked down the cause and it seems to be introduced between jGit 5.2 (which we upgraded from) and 5.3 and being caused by this commit: Move throw of PackInvalidException outside the catch - https://github.com/eclipse/jgit/commit/afef866a44cd65fef292c174cad445b3fb526400

The problem is that when the throw was inside of the try block the last catch block catched the exception and called openFailed(false) method. It is true that it called it with invalidate = false, which is wrong. The real problem though is that with the throw outside of the try block the openFail is not called at all and the fields activeWindows and activeCopyRawData are not set to 0. Which affects the later called tests like: if (++activeCopyRawData == 1 && activeWindows == 0).

The fix for this is relatively simple keeping the throw outside of a the try block and still having the invalid field set to true. See the attached patch (against current master). I did exhaustive testing of the change running concurrent clones and pushes indefinitely and with the patch applied it never fails where without it takes relatively short to get the error. However I'm not an expert in the jGit code and I may well be wrong about how to fix this.

Please let me know If you need me to do more, such as filing a bug and maybe provide a runable test case (will have to be a jGit server though)

Thanks
Hrebejk

Attachment: fix.patch
Description: Binary data

CLIENT
------

Cloning into 'gct'...
remote: Counting objects: 9615, done
remote: Finding sources: 100% (9615/9615)
remote: Getting sizes: 100% (6410/6410)
fatal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed


SERVER
------

2020-11-30 11:33:56,932 WARN  [qtp1212116343-106] [/services@2062] Internal error during upload-pack from /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git
org.eclipse.jgit.errors.PackInvalidException: Pack file invalid: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.pack
	at org.eclipse.jgit.internal.storage.file.PackFile.read(PackFile.java:719) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:516) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:603) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:386) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:226) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:614) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs2(PackFile.java:414) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.copyAsIs(PackFile.java:399) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.copyObjectAsIs(WindowCursor.java:188) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjectImpl(PackWriter.java:1724) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObject(PackWriter.java:1701) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackOutputStream.writeObject(PackOutputStream.java:138) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.writeObjects(WindowCursor.java:196) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1689) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writeObjects(PackWriter.java:1674) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1227) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2333) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:2164) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:1079) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:836) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.http.server.UploadPackServlet.lambda$1(UploadPackServlet.java:184) ~[org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.http.server.UploadPackServlet.defaultUploadPackHandler(UploadPackServlet.java:207) ~[org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.http.server.UploadPackServlet.doPost(UploadPackServlet.java:201) [org.eclipse.jgit.http.server-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	...

Caused by: java.io.IOException: Unreadable pack index: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.idx
	at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:69) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:163) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.getReverseIdx(PackFile.java:1161) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.findEndOffset(PackFile.java:1131) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.representation(PackFile.java:1092) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:593) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:584) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.selectObjectRepresentation(WindowCursor.java:177) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1343) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1178) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	... 87 more
Caused by: java.io.FileNotFoundException: /home/phrebejk/src/orcl/callahan/_data.local/qa-dev_bugs_421/hosted/gc-test.git/objects/pack/pack-40bda6444df7d7d05b7d769ea9058df9f20e7306.idx (No such file or directory)
	at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_171]
	at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_171]
	at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_171]
	at org.eclipse.jgit.util.io.SilentFileInputStream.<init>(SilentFileInputStream.java:31) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackIndex.open(PackIndex.java:65) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.idx(PackFile.java:163) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.getReverseIdx(PackFile.java:1161) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.findEndOffset(PackFile.java:1131) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.PackFile.representation(PackFile.java:1092) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:593) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.ObjectDirectory.selectObjectRepresentation(ObjectDirectory.java:584) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.file.WindowCursor.selectObjectRepresentation(WindowCursor.java:177) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1343) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.searchForReuse(PackWriter.java:1327) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	at org.eclipse.jgit.internal.storage.pack.PackWriter.writePack(PackWriter.java:1178) ~[org.eclipse.jgit-5.10.0-SNAPSHOT.jar:5.10.0-SNAPSHOT]
	... 87 more

Follow-Ups:
- Re: [jgit-dev] Concurrent run of repack and clone sometimes fails
  - From: Matthias Sohn

Prev by Date: [jgit-dev] Advice for some old changes
Next by Date: Re: [jgit-dev] Advice for some old changes
Previous by thread: [jgit-dev] Advice for some old changes
Next by thread: Re: [jgit-dev] Concurrent run of repack and clone sometimes fails
Index(es):
- Date
- Thread

Breadcrumbs