| [eclipselink-dev] EclipseLink Stuck Threads on acquiring a lock |
|
We are running into a problem in our production environments that we have been unable to reproduce locally. I have posted on several bug reports and even opened my own bug report. I have also posted on the forums. None of those attempts
have even yielded a response, let alone a fix. So, I’m trying the mailing list. The latest instance of this problem came with the stacktrace at the bottom of this message. Also, we are investigating paying for support to resolve this issue, but I cannot find any information on how to do this. Any links to that effect would be appreciated as well. Similar bug reports: https://bugs.eclipse.org/bugs/show_bug.cgi?id=413775 https://bugs.eclipse.org/bugs/show_bug.cgi?id=399528 https://bugs.eclipse.org/bugs/show_bug.cgi?id=305611 https://www.eclipse.org/forums/index.php/t/524575/ https://bugs.eclipse.org/bugs/show_bug.cgi?id=442913 https://bugs.eclipse.org/bugs/show_bug.cgi?id=294355 https://www.eclipse.org/forums/index.php/t/804977/ After looking at bug 294355 (above), we came up with the theory that we may be seeing what user Szymon Bochniak described in his response to that bug report. A thread may be releasing a lock that it did not acquire. We are in the process
of adding some logging to verify this theory. Unfortunately, we cannot reliably reproduce the error, so we just have to wait for it to happen again. However, if we are right, then it begs that question asked in the link below: why does the acquireReadLock
method verify that it is the active thread before locking, but the releaseReadLock does not have any similar verification before it releases? The stacktrace: ERROR 24 Sep 2014 09:45:49,155 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] com.savi.ng.api.BaseApiController - API Controller Exception Handler ERROR 24 Sep 2014 09:45:49,155 [[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'] com.savi.ng.api.BaseApiController - Exception caught in rest service: Could not write JSON:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]);
nested exception is com.fasterxml.jackson.databind.JsonMappingException: Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]) org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]);
nested exception is com.fasterxml.jackson.databind.JsonMappingException: Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]) at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:207) at org.springframework.http.converter.AbstractHttpMessageConverter.write(AbstractHttpMessageConverter.java:179) at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:148) at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:90) at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:189) at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:69) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:122) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:936) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:827) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:812) at javax.servlet.http.HttpServlet.service(HttpServlet.java:844) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242) at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216) at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132) at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:338) at weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:25) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.savi.ng.api.filter.CorsFilter.doFilterInternal(CorsFilter.java:196) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.savi.ng.api.filter.NoCacheFilter.doFilterInternal(NoCacheFilter.java:18) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.savi.ng.servlet.filter.UserProfileInjectorFilter.doFilter(UserProfileInjectorFilter.java:85) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at com.github.ziplet.filter.compression.CompressingFilter.doFilter(CompressingFilter.java:300) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:74) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3288) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57) at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1513) at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: com.fasterxml.jackson.databind.JsonMappingException:
Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. (through reference chain: com.savi.ng.api.response.ApiResponse["payload"]->com.savi.ng.domain.user.profile.UserProfile["roles"]->org.eclipse.persistence.indirection.IndirectList[0]->com.savi.ng.domain.role.Role["owner"]->com.savi.ng.domain.owner.Owner["children"]->org.eclipse.persistence.indirection.IndirectList[110]->com.savi.ng.domain.owner.Owner["children"]) at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232) at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:197) at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:184) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:605) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21) at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:186) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:100) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:21) at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:186) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:569) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:142) at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:118) at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:1819) at org.springframework.http.converter.json.MappingJackson2HttpMessageConverter.writeInternal(MappingJackson2HttpMessageConverter.java:204) ... 46 more Caused by: Exception [EclipseLink-2004] (Eclipse Persistence Services - 2.4.0.v20120608-r11652): org.eclipse.persistence.exceptions.ConcurrencyException Exception Description: A signal was attempted before wait() on ConcurrencyManager. This normally means that an attempt was made to
commit or rollback a transaction before it was started, or to rollback a transaction twice. at org.eclipse.persistence.exceptions.ConcurrencyException.signalAttemptedBeforeWait(ConcurrencyException.java:84) at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseReadLock(ConcurrencyManager.java:489) at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseReadLock(CacheKey.java:392) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1019) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:933) at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:193) at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:121) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3906) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3861) at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109) at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71) at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:76) at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildClone(ForeignReferenceMapping.java:251) at org.eclipse.persistence.internal.descriptors.ObjectBuilder.populateAttributesForClone(ObjectBuilder.java:3574) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.populateAndRegisterObject(UnitOfWorkImpl.java:3657) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1004) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:933) at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:193) at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:121) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3906) at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3861) at org.eclipse.persistence.mappings.CollectionMapping.buildElementUnitOfWorkClone(CollectionMapping.java:296) at org.eclipse.persistence.mappings.CollectionMapping.buildElementClone(CollectionMapping.java:309) at org.eclipse.persistence.internal.queries.ContainerPolicy.addNextValueFromIteratorInto(ContainerPolicy.java:214) at org.eclipse.persistence.mappings.CollectionMapping.buildCloneForPartObject(CollectionMapping.java:222) at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:56) at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:161) at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:222) at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:88) at org.eclipse.persistence.indirection.IndirectList.buildDelegate(IndirectList.java:244) at org.eclipse.persistence.indirection.IndirectList.getDelegate(IndirectList.java:415) at org.eclipse.persistence.indirection.IndirectList.isEmpty(IndirectList.java:490) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.isEmpty(IndexedListSerializer.java:51) at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.isEmpty(IndexedListSerializer.java:21) at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:556) at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:597) ... 68 more John Bedell Senior Staff Software Engineer Email
jbedell@xxxxxxxx Work +1-859-263-6309
Attention: The information contained in this email message may be legally privileged, confidential and
protected from disclosure. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If you are not the intended recipient, you are on notice that any dissemination, distribution or copying of this message,
including any attachments, in any form, is strictly prohibited. If you think you have received this email in error, please email the sender and delete/destroy any copy of this message.
|
