Re: [jgit-dev] Issues with JDK 11.0.2 and JGIT

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]

Re: [jgit-dev] Issues with JDK 11.0.2 and JGIT

From: Matthias Sohn <matthias.sohn@xxxxxxxxx>
Date: Sat, 8 Jun 2019 10:13:58 +0200
Delivered-to: jgit-dev@xxxxxxxxxxx
List-archive: <https://www.eclipse.org/mailman/private/jgit-dev>
List-help: <mailto:jgit-dev-request@eclipse.org?subject=help>
List-subscribe: <https://www.eclipse.org/mailman/listinfo/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=subscribe>
List-unsubscribe: <https://www.eclipse.org/mailman/options/jgit-dev>, <mailto:jgit-dev-request@eclipse.org?subject=unsubscribe>

On Fri, Mar 29, 2019 at 4:46 PM Mike Bell <mbell@xxxxxxxxxxxxx> wrote:

Evidence still being gathered, suggestions gratefully received.

Versions: 5.1.3.201810200350-r, 5.2.1.201812262042-r, 5.3.0.201903130848-r, 5.1.0.201809111528-r, 5.1.3.201810200350-r
Scenario: 6 different deploys in java 11 containers (mesos)

We have used jgit for years for a library that we use to poll github for chhanges to repos successfully.

In this program every minute we re pull and check for changes, and pass affected files list and shas up to the caller. This is done in a single threaded scheduled executor.

We are finding that after several days. the synchronization stops. Neither the scheduled thread nor jgit-workqueue look "suspicious"
In fact they look very benign.

maybe you need to gc the local repository you are pulling to from time to time

"JGit-WorkQueue" - Thread t@67
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <5e502c44> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:
- None

config-update-0" - Thread t@87
java.lang.Thread.State: WAITING
at jdk.internal.misc.Unsafe.park(Native Method)
- parking to wait for <79fc6f28> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.lang.Thread.run(Thread.java:834)

Locked ownable synchronizers:
- None

However, heavy debugging shows we go into this code and never "return"

LOG.trace("Configuration of credentials completed, setting remote {}", remoteIndex);
pull.setRemote("remote" + remoteIndex);
PullResult result = pull.call();
LOG.trace("Got result {}", result);

(The first log shows, but not the second)

Additional debugging - a custom ProgressMonitor shows beginTask is called, and then no further
A try/catch/throwable around the code gleans no obvious failure of the thread itself.

Additional: We've turned on full net debugging and seen odd finalizer errors

avax.net.ssl|WARNING|03|Finalizer|2019-03-29 12:26:57.882 UTC|SSLSocketImpl.java:494|SSLSocket duplex close failed (
"throwable" : {
java.net.SocketException: Socket is closed
at java.base/java.net.Socket.shutdownInput(Socket.java:1521)
at java.base/sun.security.ssl.BaseSSLSocketImpl.shutdownInput(BaseSSLSocketImpl.java:216)
at java.base/sun.security.ssl.SSLSocketImpl.shutdownInput(SSLSocketImpl.java:651)
at java.base/sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:606)
at java.base/sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:566)
at java.base/sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:479)
at java.base/sun.net.www.http.HttpClient.closeServer(HttpClient.java:1058)
at java.base/sun.net.www.http.ClientVector.put(KeepAliveCache.java:252)
at java.base/sun.net.www.http.KeepAliveCache.put(KeepAliveCache.java:125)
at java.base/sun.net.www.protocol.https.HttpsClient.putInKeepAliveCache(HttpsClient.java:658)
at java.base/sun.net.www.http.HttpClient.finished(HttpClient.java:406)
at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:99)
at java.base/sun.net.www.MeteredStream.justRead(MeteredStream.java:93)
at java.base/sun.net.www.MeteredStream.skip(MeteredStream.java:154)
at java.base/sun.net.www.http.KeepAliveStream.close(KeepAliveStream.java:89)
at java.base/sun.net.www.MeteredStream.finalize(MeteredStream.java:209)
at java.base/java.lang.System$2.invokeFinalize(System.java:2117)
at java.base/java.lang.ref.Finalizer.runFinalizer(Finalizer.java:87)
at java.base/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171)}

Additional: We've tried turning off TLS 1.3 (openjdk has known issues) with -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2
and verified only TLSv1.2 is being used with tcpdump.

--

Michael Bell

Principal Software Engineer / Architect

1 Montgomery Tower, Suite 700

San Francisco, CA 94104

(415) 317-9559

mbell@xxxxxxxxxxxxx

_______________________________________________
jgit-dev mailing list
jgit-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jgit-dev

References:
- [jgit-dev] Issues with JDK 11.0.2 and JGIT
  - From: Mike Bell

Prev by Date: Re: [jgit-dev] Writing packs with bitmaps sometimes slower as without bitmaps?
Next by Date: [jgit-dev] Potential memory leak in JGit
Previous by thread: [jgit-dev] Issues with JDK 11.0.2 and JGIT
Next by thread: [jgit-dev] Apply patch to tree
Index(es):
- Date
- Thread

Breadcrumbs