Comment 1 Richard Richter 2016-02-02 11:10:00 EST

We just entered testing involving concurrent requests and hit exactly the same problem (indexes differ). In our case we call QueryImpl.getResultList via Querydsl library, but I guess that one is not the problem. I compared both queries side by side for positive and exceptional cases, they are the same, so the query itself doesn't seem to be buggy. Concurrency is triggering the problem:

2016-02-02 16:04:30,581 ERROR [REQ: b47df82f/Test002] s.f.r.c.h.PersistenceExceptionHandler Unexpected data access problem
javax.persistence.PersistenceException: Exception [EclipseLink-6168] (Eclipse Persistence Services - 2.6.2.v20151217-774c696): org.eclipse.persistence.exceptions.QueryException
Exception Description: Query failed to prepare, unexpected error occurred: [java.lang.ArrayIndexOutOfBoundsException: 2].
Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
Query: ReportQuery(referenceClass=Transaction sql="SELECT t0.id, t0.amortization_amount, t0.amortization_method, t0.amortization_freq, t0.amortization_type, t0.c1amount, t0.c2amount, t0.attr_0, t0.attr_1, t0.attr_2, t0.attr_3, t0.barrier1, t0.barrier2, t0.bookvalue, t0.bs, t0.calendar_id, t0.calendar2_id, t0.client_id, t0.cp, t0.created_datetime, t0.created_user_id, t0.creation_date, t0.c1, t0.c2, t0.datebasis, t0.datebasis2, t0.dattr_0, t0.dattr_1, t0.dattr_2, t0.dattr_3, t0.dealcode, t0.dealtfxrate, t0.dealtfxrate2, t0.dealtrate, t0.dealtrate2, t0.delete_date, t0.deletedatetime, t0.description, t0.enddate, t0.external_ref, t0.fattr_0, t0.fattr_1, t0.fattr_10, t0.fattr_2, t0.fattr_3, t0.fattr_4, t0.fattr_5, t0.fattr_6, t0.fattr_7, t0.fattr_8, t0.fattr_9, t0.resetreference, t0.resetreference2, t0.resetfreq, t0.resetfreq2, t0.resettype, t0.resettype2, t0.fixingrate, t0.fixingrate2, t0.flags, t0.forecast_type_id, t0.group_id, t0.hash_code, t0.holiday_calendar_id, t0.hypo_expires, t0.inputsource_id, t0.iid, t0.siid, t0.marketrate, t0.maturitydate, t0.last_modified_datetime, t0.last_modified_user_id, t0.openingdate, t0.openingdatetime, t0.option_barrier_window_end, t0.option_barrier_window_start, t0.owner_id, t0.sattr_0, t0.sattr_1, t0.param_0, t0.param_1, t0.param_10, t0.param_11, t0.param_2, t0.param_3, t0.param_4, t0.param_5, t0.param_6, t0.param_7, t0.param_8, t0.param_9, t0.parent_id, t0.payment_freq, t0.payment_freq2, t0.payment_type, t0.payment_type2, t0.premium_amount, t0.premium, t0.premium_date, t0.portfolio_id, t0.sattr_2, t0.sattr_3, t0.security_id, t0.package_id, t0.settlementdate, t0."sign", t0.spread, t0.spread2, t0.state, t0.strike, t0.valuedate, t0.vat_amount, CASE  WHEN (t0."sign" < ?) THEN t1.sign_sell_label ELSE t1.sign_buy_label END, t2.entitycode, t1.name, t3.name, t4.entitycode, t5.dealcode, t6.name, t7.id, t7.name, t8.name, t9.name, t10.name, t11.username, t12.username, t13.id, t13.amount_rvlc, t13.bpv, t13.bpv_A, t13.bpv_L, t13.convexity, t13.delta_equivalent, t13.dur_A, t13.dur_L, t13.duration, t13.effective_duration, t13.gamma_equivalent, t13.historical_var, t13.last_eval_date, t13.modified_duration, t13.montecarlo_var, t13.mtm, t13.npv, t13.npv0, t13.npv_A, t13.npv_L, t13.outstanding, t13.scenario_id, t13.theoretical_rate, t13.transaction_id, t13.figure_type, t13.vega_equivalent, t13.volatility, t13.volatility_smile, t13.volatility_smile2 FROM Transactions t0 LEFT OUTER JOIN SubInstruments t1 ON ((t0.siid = t1.siid) AND (t0.iid = t1.iid)) LEFT OUTER JOIN Clients t2 ON (t0.client_id = t2.id) LEFT OUTER JOIN Instruments t3 ON (t0.iid = t3.id) LEFT OUTER JOIN Clients t4 ON (t0.owner_id = t4.id) LEFT OUTER JOIN Transactions t5 ON (t0.parent_id = t5.id) LEFT OUTER JOIN Portfolios t6 ON (t0.portfolio_id = t6.id) LEFT OUTER JOIN PortfolioRoots t7 ON (t6.pr_id = t7.id) LEFT OUTER JOIN InputSources t8 ON (t0.inputsource_id = t8.id) LEFT OUTER JOIN WFL_States t9 ON ((t0.state = t9.state_id) AND (t9.object_id = ?)) LEFT OUTER JOIN Securities t10 ON (t0.security_id = t10.id) LEFT OUTER JOIN GUIUsers t11 ON (t0.created_user_id = t11.id) LEFT OUTER JOIN GUIUsers t12 ON (t0.last_modified_user_id = t12.id) LEFT OUTER JOIN TransactionFigures t13 ON (((t13.transaction_id = t0.id) AND (t13.figure_type IN ?)) AND (t13.scenario_id = ?))")
	at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:484) ~[org.eclipse.persistence.jpa-2.6.2.jar:na]
	at com.mysema.query.jpa.impl.AbstractJPAQuery.getResultList(AbstractJPAQuery.java:180) ~[querydsl-jpa-3.6.3.jar:na]
	at com.mysema.query.jpa.impl.AbstractJPAQuery.list(AbstractJPAQuery.java:246) ~[querydsl-jpa-3.6.3.jar:na]
	at com.mysema.query.jpa.impl.AbstractJPAQuery.list(AbstractJPAQuery.java:238) ~[querydsl-jpa-3.6.3.jar:na]
...ourcode, Spring, Jersey, Jetty...
Caused by: org.eclipse.persistence.exceptions.QueryException: 
Exception Description: Query failed to prepare, unexpected error occurred: [java.lang.ArrayIndexOutOfBoundsException: 2].
Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 2
Query: ReportQuery(referenceClass=Transaction sql="SELECT t0.id, t0.amortization_amount, t0.amortization_method, t0.amortization_freq, t0.amortization_type, t0.c1amount, t0.c2amount, t0.attr_0, t0.attr_1, t0.attr_2, t0.attr_3, t0.barrier1, t0.barrier2, t0.bookvalue, t0.bs, t0.calendar_id, t0.calendar2_id, t0.client_id, t0.cp, t0.created_datetime, t0.created_user_id, t0.creation_date, t0.c1, t0.c2, t0.datebasis, t0.datebasis2, t0.dattr_0, t0.dattr_1, t0.dattr_2, t0.dattr_3, t0.dealcode, t0.dealtfxrate, t0.dealtfxrate2, t0.dealtrate, t0.dealtrate2, t0.delete_date, t0.deletedatetime, t0.description, t0.enddate, t0.external_ref, t0.fattr_0, t0.fattr_1, t0.fattr_10, t0.fattr_2, t0.fattr_3, t0.fattr_4, t0.fattr_5, t0.fattr_6, t0.fattr_7, t0.fattr_8, t0.fattr_9, t0.resetreference, t0.resetreference2, t0.resetfreq, t0.resetfreq2, t0.resettype, t0.resettype2, t0.fixingrate, t0.fixingrate2, t0.flags, t0.forecast_type_id, t0.group_id, t0.hash_code, t0.holiday_calendar_id, t0.hypo_expires, t0.inputsource_id, t0.iid, t0.siid, t0.marketrate, t0.maturitydate, t0.last_modified_datetime, t0.last_modified_user_id, t0.openingdate, t0.openingdatetime, t0.option_barrier_window_end, t0.option_barrier_window_start, t0.owner_id, t0.sattr_0, t0.sattr_1, t0.param_0, t0.param_1, t0.param_10, t0.param_11, t0.param_2, t0.param_3, t0.param_4, t0.param_5, t0.param_6, t0.param_7, t0.param_8, t0.param_9, t0.parent_id, t0.payment_freq, t0.payment_freq2, t0.payment_type, t0.payment_type2, t0.premium_amount, t0.premium, t0.premium_date, t0.portfolio_id, t0.sattr_2, t0.sattr_3, t0.security_id, t0.package_id, t0.settlementdate, t0."sign", t0.spread, t0.spread2, t0.state, t0.strike, t0.valuedate, t0.vat_amount, CASE  WHEN (t0."sign" < ?) THEN t1.sign_sell_label ELSE t1.sign_buy_label END, t2.entitycode, t1.name, t3.name, t4.entitycode, t5.dealcode, t6.name, t7.id, t7.name, t8.name, t9.name, t10.name, t11.username, t12.username, t13.id, t13.amount_rvlc, t13.bpv, t13.bpv_A, t13.bpv_L, t13.convexity, t13.delta_equivalent, t13.dur_A, t13.dur_L, t13.duration, t13.effective_duration, t13.gamma_equivalent, t13.historical_var, t13.last_eval_date, t13.modified_duration, t13.montecarlo_var, t13.mtm, t13.npv, t13.npv0, t13.npv_A, t13.npv_L, t13.outstanding, t13.scenario_id, t13.theoretical_rate, t13.transaction_id, t13.figure_type, t13.vega_equivalent, t13.volatility, t13.volatility_smile, t13.volatility_smile2 FROM Transactions t0 LEFT OUTER JOIN SubInstruments t1 ON ((t0.siid = t1.siid) AND (t0.iid = t1.iid)) LEFT OUTER JOIN Clients t2 ON (t0.client_id = t2.id) LEFT OUTER JOIN Instruments t3 ON (t0.iid = t3.id) LEFT OUTER JOIN Clients t4 ON (t0.owner_id = t4.id) LEFT OUTER JOIN Transactions t5 ON (t0.parent_id = t5.id) LEFT OUTER JOIN Portfolios t6 ON (t0.portfolio_id = t6.id) LEFT OUTER JOIN PortfolioRoots t7 ON (t6.pr_id = t7.id) LEFT OUTER JOIN InputSources t8 ON (t0.inputsource_id = t8.id) LEFT OUTER JOIN WFL_States t9 ON ((t0.state = t9.state_id) AND (t9.object_id = ?)) LEFT OUTER JOIN Securities t10 ON (t0.security_id = t10.id) LEFT OUTER JOIN GUIUsers t11 ON (t0.created_user_id = t11.id) LEFT OUTER JOIN GUIUsers t12 ON (t0.last_modified_user_id = t12.id) LEFT OUTER JOIN TransactionFigures t13 ON (((t13.transaction_id = t0.id) AND (t13.figure_type IN ?)) AND (t13.scenario_id = ?))")
	at org.eclipse.persistence.exceptions.QueryException.prepareFailed(QueryException.java:1596) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:685) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:911) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:615) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:872) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:460) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258) ~[org.eclipse.persistence.jpa-2.6.2.jar:na]
	at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:473) ~[org.eclipse.persistence.jpa-2.6.2.jar:na]
	... 104 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
	at org.eclipse.persistence.expressions.ListExpressionOperator.getDatabaseStrings(ListExpressionOperator.java:75) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2196) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:99) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.FunctionExpression.writeFields(FunctionExpression.java:717) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.SQLSelectStatement.writeFieldsFromExpression(SQLSelectStatement.java:2066) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.SQLSelectStatement.writeFieldsIn(SQLSelectStatement.java:2081) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL(SQLSelectStatement.java:1700) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.databaseaccess.DatabasePlatform.printSQLSelectStatement(DatabasePlatform.java:3243) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:800) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:810) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.descriptors.ClassDescriptor.buildCallFromStatement(ClassDescriptor.java:815) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.StatementQueryMechanism.setCallFromStatement(StatementQueryMechanism.java:390) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.prepareReportQuerySelectAllRows(ExpressionQueryMechanism.java:1698) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReportQuery.prepareSelectAllRows(ReportQuery.java:1207) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReadAllQuery.prepare(ReadAllQuery.java:816) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReportQuery.prepare(ReportQuery.java:1075) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:666) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	... 116 common frames omitted

I do realize the query is crazy, but Guillaume provided much simpler case. I'm adding mine just to support the case. Funny thing is, that there is an alternative exception. For Guillaume it was NPE, in our case it's something deeper in JDBC driver, which is JTDS in our case. Because it (again) NEVER happens when run in serial, I suspect something messy happens on the way to JDBC call, but something else than ArrayIndexOOBEx and later manifests as:

Query: ReportQuery(referenceClass=Transaction sql="SELECT t0.id, t0.amortization_amount, t0.amortization_method, t0.amortization_freq, t0.amortization_type, t0.c1amount, t0.c2amount, t0.attr_0, t0.attr_1, t0.attr_2, t0.attr_3, t0.barrier1, t0.barrier2, t0.bookvalue, t0.bs, t0.calendar_id, t0.calendar2_id, t0.client_id, t0.cp, t0.created_datetime, t0.created_user_id, t0.creation_date, t0.c1, t0.c2, t0.datebasis, t0.datebasis2, t0.dattr_0, t0.dattr_1, t0.dattr_2, t0.dattr_3, t0.dealcode, t0.dealtfxrate, t0.dealtfxrate2, t0.dealtrate, t0.dealtrate2, t0.delete_date, t0.deletedatetime, t0.description, t0.enddate, t0.external_ref, t0.fattr_0, t0.fattr_1, t0.fattr_10, t0.fattr_2, t0.fattr_3, t0.fattr_4, t0.fattr_5, t0.fattr_6, t0.fattr_7, t0.fattr_8, t0.fattr_9, t0.resetreference, t0.resetreference2, t0.resetfreq, t0.resetfreq2, t0.resettype, t0.resettype2, t0.fixingrate, t0.fixingrate2, t0.flags, t0.forecast_type_id, t0.group_id, t0.hash_code, t0.holiday_calendar_id, t0.hypo_expires, t0.inputsource_id, t0.iid, t0.siid, t0.marketrate, t0.maturitydate, t0.last_modified_datetime, t0.last_modified_user_id, t0.openingdate, t0.openingdatetime, t0.option_barrier_window_end, t0.option_barrier_window_start, t0.owner_id, t0.sattr_0, t0.sattr_1, t0.param_0, t0.param_1, t0.param_10, t0.param_11, t0.param_2, t0.param_3, t0.param_4, t0.param_5, t0.param_6, t0.param_7, t0.param_8, t0.param_9, t0.parent_id, t0.payment_freq, t0.payment_freq2, t0.payment_type, t0.payment_type2, t0.premium_amount, t0.premium, t0.premium_date, t0.portfolio_id, t0.sattr_2, t0.sattr_3, t0.security_id, t0.package_id, t0.settlementdate, t0."sign", t0.spread, t0.spread2, t0.state, t0.strike, t0.valuedate, t0.vat_amount, CASE  WHEN (t0."sign" < ?) THEN t1.sign_sell_label WHEN t1.sign_buy_label THEN , t2.entitycode, t1.name, t3.name, t4.entitycode, t5.dealcode, t6.name, t7.id, t7.name, t8.name, t9.name, t10.name, t11.username, t12.username, t13.id, t13.amount_rvlc, t13.bpv, t13.bpv_A, t13.bpv_L, t13.convexity, t13.delta_equivalent, t13.dur_A, t13.dur_L, t13.duration, t13.effective_duration, t13.gamma_equivalent, t13.historical_var, t13.last_eval_date, t13.modified_duration, t13.montecarlo_var, t13.mtm, t13.npv, t13.npv0, t13.npv_A, t13.npv_L, t13.outstanding, t13.scenario_id, t13.theoretical_rate, t13.transaction_id, t13.figure_type, t13.vega_equivalent, t13.volatility, t13.volatility_smile, t13.volatility_smile2 FROM Transactions t0 LEFT OUTER JOIN SubInstruments t1 ON ((t0.siid = t1.siid) AND (t0.iid = t1.iid)) LEFT OUTER JOIN Clients t2 ON (t0.client_id = t2.id) LEFT OUTER JOIN Instruments t3 ON (t0.iid = t3.id) LEFT OUTER JOIN Clients t4 ON (t0.owner_id = t4.id) LEFT OUTER JOIN Transactions t5 ON (t0.parent_id = t5.id) LEFT OUTER JOIN Portfolios t6 ON (t0.portfolio_id = t6.id) LEFT OUTER JOIN PortfolioRoots t7 ON (t6.pr_id = t7.id) LEFT OUTER JOIN InputSources t8 ON (t0.inputsource_id = t8.id) LEFT OUTER JOIN WFL_States t9 ON ((t0.state = t9.state_id) AND (t9.object_id = ?)) LEFT OUTER JOIN Securities t10 ON (t0.security_id = t10.id) LEFT OUTER JOIN GUIUsers t11 ON (t0.created_user_id = t11.id) LEFT OUTER JOIN GUIUsers t12 ON (t0.last_modified_user_id = t12.id) LEFT OUTER JOIN TransactionFigures t13 ON (((t13.transaction_id = t0.id) AND (t13.figure_type IN ?)) AND (t13.scenario_id = ?))")
	at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:340) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:684) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:560) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:2055) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.sessions.server.ClientSession.executeCall(ClientSession.java:306) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:242) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:228) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:299) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:694) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2740) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllReportQueryRows(ExpressionQueryMechanism.java:2677) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReportQuery.executeDatabaseQuery(ReportQuery.java:852) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:904) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1134) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:460) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1222) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2896) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1857) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1839) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258) ~[org.eclipse.persistence.jpa-2.6.2.jar:na]
	... 105 common frames omitted
Caused by: java.sql.SQLException: An expression of non-boolean type specified in a context where a condition is expected, near 'THEN'.
	at net.sourceforge.jtds.jdbc.SQLDiagnostic.addDiagnostic(SQLDiagnostic.java:372) ~[jtds-1.3.1.jar:1.3.1]
	at net.sourceforge.jtds.jdbc.TdsCore.tdsErrorToken(TdsCore.java:2988) ~[jtds-1.3.1.jar:1.3.1]
	at net.sourceforge.jtds.jdbc.TdsCore.nextToken(TdsCore.java:2421) ~[jtds-1.3.1.jar:1.3.1]
	at net.sourceforge.jtds.jdbc.TdsCore.getMoreResults(TdsCore.java:671) ~[jtds-1.3.1.jar:1.3.1]
	at net.sourceforge.jtds.jdbc.JtdsStatement.executeSQLQuery(JtdsStatement.java:505) ~[jtds-1.3.1.jar:1.3.1]
	at net.sourceforge.jtds.jdbc.JtdsPreparedStatement.executeQuery(JtdsPreparedStatement.java:1029) ~[jtds-1.3.1.jar:1.3.1]
	at com.zaxxer.hikari.proxy.PreparedStatementProxy.executeQuery(PreparedStatementProxy.java:52) ~[HikariCP-2.4.1.jar:na]
	at com.zaxxer.hikari.proxy.HikariPreparedStatementProxy.executeQuery(HikariPreparedStatementProxy.java) ~[HikariCP-2.4.1.jar:na]
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeSelect(DatabaseAccessor.java:1009) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:644) ~[org.eclipse.persistence.core-2.6.2.jar:na]
	... 124 common frames omitted

Query for first and second case should be the same (it is the same path) however, in the latter case there is THEN , (comma right after then), so something wrong happend during query generation. If I add exception breakpoint for java.sql.SQLException and climb up the stack after it suspends, I can clearly see that on org.eclipse.persistence.internal.jpa.QueryImpl:473 local variable query looks reports this (THEN is still correct, that is it is not followed by comma):

ReportQuery(referenceClass=Transaction sql="SELECT t0.id, t0.amortization_amount, t0.amortization_method, t0.amortization_freq, t0.amortization_type, t0.c1amount, t0.c2amount, t0.attr_0, t0.attr_1, t0.attr_2, t0.attr_3, t0.barrier1, t0.barrier2, t0.bookvalue, t0.bs, t0.calendar_id, t0.calendar2_id, t0.client_id, t0.cp, t0.created_datetime, t0.created_user_id, t0.creation_date, t0.c1, t0.c2, t0.datebasis, t0.datebasis2, t0.dattr_0, t0.dattr_1, t0.dattr_2, t0.dattr_3, t0.dealcode, t0.dealtfxrate, t0.dealtfxrate2, t0.dealtrate, t0.dealtrate2, t0.delete_date, t0.deletedatetime, t0.description, t0.enddate, t0.external_ref, t0.fattr_0, t0.fattr_1, t0.fattr_10, t0.fattr_2, t0.fattr_3, t0.fattr_4, t0.fattr_5, t0.fattr_6, t0.fattr_7, t0.fattr_8, t0.fattr_9, t0.resetreference, t0.resetreference2, t0.resetfreq, t0.resetfreq2, t0.resettype, t0.resettype2, t0.fixingrate, t0.fixingrate2, t0.flags, t0.forecast_type_id, t0.group_id, t0.hash_code, t0.holiday_calendar_id, t0.hypo_expires, t0.inputsource_id, t0.iid, t0.siid, t0.marketrate, t0.maturitydate, t0.last_modified_datetime, t0.last_modified_user_id, t0.openingdate, t0.openingdatetime, t0.option_barrier_window_end, t0.option_barrier_window_start, t0.owner_id, t0.sattr_0, t0.sattr_1, t0.param_0, t0.param_1, t0.param_10, t0.param_11, t0.param_2, t0.param_3, t0.param_4, t0.param_5, t0.param_6, t0.param_7, t0.param_8, t0.param_9, t0.parent_id, t0.payment_freq, t0.payment_freq2, t0.payment_type, t0.payment_type2, t0.premium_amount, t0.premium, t0.premium_date, t0.portfolio_id, t0.sattr_2, t0.sattr_3, t0.security_id, t0.package_id, t0.settlementdate, t0."sign", t0.spread, t0.spread2, t0.state, t0.strike, t0.valuedate, t0.vat_amount, CASE  WHEN (t0."sign" < ?) THEN t1.sign_sell_label WHEN t1.sign_buy_label THEN , t2.entitycode, t1.name, t3.name, t4.entitycode, t5.dealcode, t6.name, t7.id, t7.name, t8.name, t9.name, t10.name, t11.username, t12.username, t13.id, t13.amount_rvlc, t13.bpv, t13.bpv_A, t13.bpv_L, t13.convexity, t13.delta_equivalent, t13.dur_A, t13.dur_L, t13.duration, t13.effective_duration, t13.gamma_equivalent, t13.historical_var, t13.last_eval_date, t13.modified_duration, t13.montecarlo_var, t13.mtm, t13.npv, t13.npv0, t13.npv_A, t13.npv_L, t13.outstanding, t13.scenario_id, t13.theoretical_rate, t13.transaction_id, t13.figure_type, t13.vega_equivalent, t13.volatility, t13.volatility_smile, t13.volatility_smile2 FROM Transactions t0 LEFT OUTER JOIN SubInstruments t1 ON ((t0.siid = t1.siid) AND (t0.iid = t1.iid)) LEFT OUTER JOIN Clients t2 ON (t0.client_id = t2.id) LEFT OUTER JOIN Instruments t3 ON (t0.iid = t3.id) LEFT OUTER JOIN Clients t4 ON (t0.owner_id = t4.id) LEFT OUTER JOIN Transactions t5 ON (t0.parent_id = t5.id) LEFT OUTER JOIN Portfolios t6 ON (t0.portfolio_id = t6.id) LEFT OUTER JOIN PortfolioRoots t7 ON (t6.pr_id = t7.id) LEFT OUTER JOIN InputSources t8 ON (t0.inputsource_id = t8.id) LEFT OUTER JOIN WFL_States t9 ON ((t0.state = t9.state_id) AND (t9.object_id = ?)) LEFT OUTER JOIN Securities t10 ON (t0.security_id = t10.id) LEFT OUTER JOIN GUIUsers t11 ON (t0.created_user_id = t11.id) LEFT OUTER JOIN GUIUsers t12 ON (t0.last_modified_user_id = t12.id) LEFT OUTER JOIN TransactionFigures t13 ON (((t13.transaction_id = t0.id) AND (t13.figure_type IN ?)) AND (t13.scenario_id = ?))")

So the EclipseLink definitely does something with the query on the way down. I tried to debug the getResultList method, it seems to work with differente piece of query object, so I don't believe I share anything I shouldn't share between the threads. Is there anything we should check in cases like this? Something we might have done wrong?

Comment 2 Richard Richter 2016-03-29 13:09:17 EDT

I tried to reconstruct this issue and I was able to do it with our infrastructure, but not with JPQL although everything seemed to indicate it must be possible. I failed every time until I noticed that we add hint in our Dao layer:

query.setHint("javax.persistence.cache.storeMode", CacheStoreMode.REFRESH);

With this even simple JPQL fails with CASE. When it is removed from returned projects (select clause) it works again. I'll try to put together some automatic test case, but it seems this hint "enables" it. Guillaume, do you use some hint of some sort too?

Comment 3 Richard Richter 2016-03-30 04:45:49 EDT

I created parallel testcase with TestNG, it is very easy to reproduce in shell:

svn export https://github.com/virgo47/litterbin/trunk/issues/eclipselink463042
cd eclipselink463042
mvn clean package

This runs the tests too, there are 3 tests and each runs 100 times. Two pass, one has random number of failures (in my case in range of 8-20, crashes every time). Pure query "select case when d.id = 0 then 1 else 2 end from Dog d" works OK, but when I add the hint ("javax.persistence.cache.storeMode", CacheStoreMode.REFRESH) it fails. Query without case "select d from Dog d" does not fail even with this hint.

While the exact circumstances are a bit different from Guillaume's case, I believe it is the same problem because:

* it both involves CASE :-)
* both cases fail the same way, either IndexArrayOutOfBoundsException or with syntax problem where the CASE is somehow incorrectly serialized (like second THEN)

Both of these failures occur during the test, so it is easy to reproduce. I don't know why in my case the hint is the trigger (I may stop using it, although this was so far most reliable way how to reload new data from DB when changed from other process) because Guillaume's case does not involve it. I tried messing with <shared-cache-mode> but there was no visible difference on the test runs.

Comment 4 Richard Richter 2016-04-20 11:08:17 EDT

I'd welcome any comment on this, should we consider CASE and EclipseLink unreliable and rewrite all those queries? Is there a fix around the corner? We're left in the dark with quite a critical concurrency bug.

I discovered that hint ("javax.persistence.cache.storeMode", CacheStoreMode.REFRESH) somehow helps the bug to manifestate - I wasn't able to reproduce it without it in my provided mini-test (see previous comment). However in our production code where we have much bigger query with multiple joins we hit it virtually every time we let two scripted users accessing the same REST endpoint that leads to the query after just a dozen of such accesses - which means it's bound to happen and the only possible workaround except for not using CASE seems to be catch these exceptions and try again.

Comment 5 Jan Vissers 2017-02-21 02:51:56 EST

I can confirm that named queries and case/coalesce with Eclipselink are a useless feature. I don't understand why this isn't picked up, as there are other bugs that seem to indicate that this has been an issue for many. many years now: https://bugs.eclipse.org/bugs/show_bug.cgi?id=382308

Comment 6 Patrick Haller 2019-04-04 07:35:28 EDT

We're hitting the same issue with EclipseLink 2.7.5, approx. once per day in production/development.

Thread-safety issues explain the otherwise inexplicable SQL generation issues we've been seeing, but could not reproduce in isolated tests.
Our EntityManagers are solely provided by Spring, hence are not expected to be incorrectly shared between threads.

Example 1:

JPQL:
SELECT E1 FROM object E1 WHERE E1.uuid IN :objectUuids AND (CASE WHEN (UPPER(E1.name) LIKE CONCAT('%',CONCAT(?1,'%')) ESCAPE '\') THEN TRUE ELSE FALSE END) = true ORDER BY E1.name

Good SQL:
SELECT uuid, changed_by, changed_time, object_owner_uuid, created_by, creation_time, data_privacy_object_id, description, end_date, geo_coordinates, id, location, name, object_consent_form_uuid, object_uuid, phase, object_manager_uuid, site_manager_uuid, start_date, status, terms_condition_object_id, type FROM object WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || ? || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name")</code>

Occasionally generated broken SQL:
SELECT uuid AS a1, changed_by AS a2, changed_time AS a3, object_owner_uuid AS a4, created_by AS a5, creation_time AS a6, data_privacy_object_id AS a7, description AS a8, end_date AS a9, geo_coordinates AS a10, id AS a11, location AS a12, name AS a13, object_consent_form_uuid AS a14, object_uuid AS a15, phase AS a16, object_manager_uuid AS a17, site_manager_uuid AS a18, start_date AS a19, status AS a20, terms_condition_object_id AS a21, type AS a22 FROM object WHERE ((uuid IN (?,?,...,?,?)) AND (CASE  WHEN UPPER(name) LIKE '%' || ? || '%' ESCAPE '\' THEN 1 WHEN 0 THEN  = 1)) ORDER BY name LIMIT ? OFFSET ?

Comment 7 Patrick Haller 2019-04-04 07:36:54 EDT

Internal Exception: java.lang.ArrayIndexOutOfBoundsException: 6
Query: ReportQuery(referenceClass=Project sql="SELECT t0.uuid, t0.category, t0.changed_by, t0.changed_time, t0.created_by, t0.creation_time, t0.description, t0.line_info, t0.snapshot_info, t0.view_info, t0.due_date, t0.external_id, t0.location, t0.priority, t0.processor_uuid, t0.processor_person_uuid, t0.status, t0.title, t0.type, t0.project_uuid FROM issue t0, project t1 WHERE (((CASE  WHEN LOWER(t0.title) LIKE '%' || ? || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1) AND t1.uuid LIKE ? ESCAPE '\') AND (t0.project_uuid = t1.uuid)) ORDER BY t0.uuid")
        java.lang.ArrayIndexOutOfBoundsException: 6
                 at org.eclipse.persistence.expressions.ListExpressionOperator.getDatabaseStrings(ListExpressionOperator.java:77)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2174)
                 at org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:102)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2238)
                 at org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:288)
                 at org.eclipse.persistence.internal.expressions.RelationExpression.printSQL(RelationExpression.java:901)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2238)
                 at org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:288)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2238)
                 at org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:288)
                 at org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.translateExpression(ExpressionSQLPrinter.java:326)
                 at org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printExpression(ExpressionSQLPrinter.java:132)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL(SQLSelectStatement.java:1758)
                 at org.eclipse.persistence.platform.database.HANAPlatform.printSQLSelectStatement(HANAPlatform.java:349)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:846)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:857)
                 at org.eclipse.persistence.descriptors.ClassDescriptor.buildCallFromStatement(ClassDescriptor.java:879)
                 at org.eclipse.persistence.internal.queries.StatementQueryMechanism.setCallFromStatement(StatementQueryMechanism.java:392)
                 at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.prepareReportQuerySelectAllRows(ExpressionQueryMechanism.java:1701)
                 at org.eclipse.persistence.queries.ReportQuery.prepareSelectAllRows(ReportQuery.java:1209)
                 at org.eclipse.persistence.queries.ReadAllQuery.prepare(ReadAllQuery.java:841)
                 at org.eclipse.persistence.queries.ReportQuery.prepare(ReportQuery.java:1077)
                 at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:676)
                 at org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare(ObjectLevelReadQuery.java:960)
                 at org.eclipse.persistence.queries.DatabaseQuery.checkPrepare(DatabaseQuery.java:625)
                 at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:882)
                 at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1183)
                 at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:485)
                 at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1271)
                 at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2981)
                 at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1895)
                 at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1877)
                 at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1842)
                 at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:262)
                 at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:482)

Comment 8 Patrick Haller 2019-04-18 06:31:17 EDT

I can frequently reproduce the issue running a JMeter script that saturates the backend with parallel JPQL queries, containing a CASE operator and varying search terms.

E. g.
1 infinite loop with:
SELECT uuid, changed_by, ... , name, ..., type FROM project WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || 'TERM1' || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name

1 infinite loop with:
SELECT uuid, changed_by, ... , name, ..., type FROM project WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || 'DIFFERENT TERM2' || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name

1 infinite loop with:
SELECT uuid, changed_by, ... , name, ..., type FROM project WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || 'even more DIFFERENT TERM3' || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name


A major problem is that attaching a debugger to the backend changes the runtime characteristics within the EclipseLink runtime so much, that one gets different issues: instead of ArrayOutOfBounds in ListOperatorExpression, we see NullPointerExceptions during SQL printing.

It looks like there are quite a few races there.


Query: ReadAllQuery(referenceClass=Project sql="SELECT uuid, changed_by, changed_time, collaboration_owner_uuid, created_by, creation_time, data_privacy_object_id, description, end_date, geo_coordinates, id, location, name, network_consent_form_uuid, network_uuid, phase, project_manager_uuid, site_manager_uuid, start_date, status, terms_condition_object_id, type FROM project WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || ? || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name")
    org.eclipse.persistence.exceptions.QueryException: 
Exception Description: Query failed to prepare, unexpected error occurred: [java.lang.NullPointerException].
Internal Exception: java.lang.NullPointerException
Query: ReadAllQuery(referenceClass=Project sql="SELECT uuid, changed_by, changed_time, collaboration_owner_uuid, created_by, creation_time, data_privacy_object_id, description, end_date, geo_coordinates, id, location, name, network_consent_form_uuid, network_uuid, phase, project_manager_uuid, site_manager_uuid, start_date, status, terms_condition_object_id, type FROM project WHERE ((uuid IN ?) AND (CASE  WHEN UPPER(name) LIKE '%' || ? || '%' ESCAPE '\' THEN 1 ELSE 0 END = 1)) ORDER BY name")
        java.lang.NullPointerException: null
                 at java.io.Writer.write(Writer.java:157)
                 at org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printString(ExpressionSQLPrinter.java:208)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printCollection(ExpressionOperator.java:2201)
                 at org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL(ArgumentListFunctionExpression.java:102)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2238)
                 at org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:288)
                 at org.eclipse.persistence.internal.expressions.RelationExpression.printSQL(RelationExpression.java:901)
                 at org.eclipse.persistence.expressions.ExpressionOperator.printDuo(ExpressionOperator.java:2243)
                 at org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL(CompoundExpression.java:288)
                 at org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.translateExpression(ExpressionSQLPrinter.java:326)
                 at org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printExpression(ExpressionSQLPrinter.java:132)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL(SQLSelectStatement.java:1758)
                 at org.eclipse.persistence.platform.database.HANAPlatform.printSQLSelectStatement(HANAPlatform.java:349)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:846)
                 at org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall(SQLSelectStatement.java:857)

Comment 9 Patrick Haller 2019-04-23 11:20:09 EDT

It looks to me as if ...

Bug #1: the selectionCriteria are shared between threads! In case of the ArgumentListFunctionExpression, there is kind of a state!

Bug #2: ListExpressionOperator.copyTo( id=15644, id=15644 ) is copying values between the same object instance!
	ArgumentListFunctionExpression.printSQL(..) (id=16216)
	+--operator ListExpressionOperator( id=15644 )
	+--platformOperator ListExpressionOperator( id=15644 )
    ExpressionOperator.printDuo( .. )
    
This likely comes from FunctionExpression.initializePlatformOperator(). Possibly the assignment platformOperator = operator should be actually a clone!


DETAILS
=======

On thread http-nio-8080-exec-1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
EJBQueryImpl.buildEJBQLDatabaseQuery(
  queryName = null,
  jpqlQuery = "SELECT E1 FROM Project E1 WHERE E1.uuid IN :projectUuids AND (CASE WHEN (UPPER(E1.name) LIKE CONCAT('%',CONCAT(?1,'%')) ESCAPE '\') THEN TRUE ELSE FALSE END) = true ORDER BY E1.name",
  session = ServerSession (id=15932),
  lockMode = null,
  hints = null,
  classLoader = LaunchedURLClassLoader (id=15655)
)

databaseQuery=		
ReadAllQuery (id=15991)
+--queryMechanism JPQLCallQueryMechanism (id=16096)
   +--selectionCriteria LogicalExpression (id=16112)   <===
      +--secondChild RelationExpression (id=16117)
	     +--firstChild ArgumentListFunctionExpression (id=16120)
		    +--operator ListExpressionOperator (id=15644)
			   +--ReentrancyGuard 15643
			+--platformOperator null
			
On thread http-nio-8080-exec-3:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ReadAllQuery,.prepareSelectAllRows() (id=16175)
+--queryMechanism JPQLCallQueryMechanism (id=16174)
   +--selectionCriteria LogicalExpression (id=16112)   <===
      +--secondChild RelationExpression (id=16117)
	  	 +--firstChild ArgumentListFunctionExpression (id=16120)
		    +--operator ListExpressionOperator (id=15644)
			   +--ReentrancyGuard 15643
			+--platformOperator null


I have verified, that
OK - different database sessions (AbstractSession objects) --> no accidentally shared EntityManager
OK - different query objects
OK - Guarded object in all three stacktraces is ListExpressionOperator (id=14344)

The guarded object is created at:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reentrancy [Thread[http-nio-8080-exec-10,5,main] (alive)]
    org.eclipse.persistence.expressions.ListExpressionOperator.<init> ListExpressionOperator.java(52)
    org.eclipse.persistence.expressions.Expression.caseStatement Expression.java(754)
    org.eclipse.persistence.expressions.Expression.caseStatement Expression.java(722)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.visit ExpressionBuilderVisitor.java(635)
    org.eclipse.persistence.jpa.jpql.parser.CaseExpression.accept CaseExpression.java(107)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.visit ExpressionBuilderVisitor.java(1894)
    org.eclipse.persistence.jpa.jpql.parser.SubExpression.accept SubExpression.java(51)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor$ComparisonExpressionVisitor.visit ExpressionBuilderVisitor.java(2259)
    org.eclipse.persistence.jpa.jpql.parser.AnonymousExpressionVisitor.visit AnonymousExpressionVisitor.java(528)
    org.eclipse.persistence.jpa.jpql.parser.SubExpression.accept SubExpression.java(51)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.visit ExpressionBuilderVisitor.java(774)
    org.eclipse.persistence.jpa.jpql.parser.ComparisonExpression.accept ComparisonExpression.java(73)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.visit ExpressionBuilderVisitor.java(489)
    org.eclipse.persistence.jpa.jpql.parser.AndExpression.accept AndExpression.java(60)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.visit ExpressionBuilderVisitor.java(2147)
    org.eclipse.persistence.jpa.jpql.parser.WhereClause.accept WhereClause.java(61)
    org.eclipse.persistence.internal.jpa.jpql.ExpressionBuilderVisitor.buildExpression ExpressionBuilderVisitor.java(276)
    org.eclipse.persistence.internal.jpa.jpql.JPQLQueryContext.buildExpression JPQLQueryContext.java(328)
    org.eclipse.persistence.internal.jpa.jpql.AbstractObjectLevelReadQueryVisitor.visit AbstractObjectLevelReadQueryVisitor.java(266)
    org.eclipse.persistence.jpa.jpql.parser.WhereClause.accept WhereClause.java(61)
    org.eclipse.persistence.internal.jpa.jpql.AbstractObjectLevelReadQueryVisitor.visitAbstractSelectStatement AbstractObjectLevelReadQueryVisitor.java(332)
    org.eclipse.persistence.internal.jpa.jpql.AbstractObjectLevelReadQueryVisitor.visit AbstractObjectLevelReadQueryVisitor.java(185)
    org.eclipse.persistence.jpa.jpql.parser.SelectStatement.accept SelectStatement.java(103)
    org.eclipse.persistence.internal.jpa.jpql.HermesParser$DatabaseQueryVisitor.visit HermesParser.java(440)
    org.eclipse.persistence.jpa.jpql.parser.SelectStatement.accept SelectStatement.java(103)
    org.eclipse.persistence.internal.jpa.jpql.HermesParser$DatabaseQueryVisitor.visit HermesParser.java(420)
    org.eclipse.persistence.jpa.jpql.parser.JPQLExpression.accept JPQLExpression.java(137)
    org.eclipse.persistence.internal.jpa.jpql.HermesParser.populateQueryImp HermesParser.java(284)
    org.eclipse.persistence.internal.jpa.jpql.HermesParser.buildQuery HermesParser.java(165)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.buildEJBQLDatabaseQuery EJBQueryImpl.java(142)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.buildEJBQLDatabaseQuery EJBQueryImpl.java(118)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.<init> EJBQueryImpl.java(104)
    org.eclipse.persistence.internal.jpa.EJBQueryImpl.<init> EJBQueryImpl.java(88)
    org.eclipse.persistence.internal.jpa.EntityManagerImpl.createQuery EntityManagerImpl.java(1744)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (native) NativeMethodAccessorImpl.java(-2)
    sun.reflect.NativeMethodAccessorImpl.invoke NativeMethodAccessorImpl.java(62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke DelegatingMethodAccessorImpl.java(43)
    java.lang.reflect.Method.invoke Method.java(498)
    org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke ExtendedEntityManagerCreator.java(347)
    com.sun.proxy.$Proxy144.createQuery null(-1)
    com.sap.copin.project.db.ProjectJPAStatementInjector.generateQuery ProjectJPAStatementInjector.java(271)
    com.sap.copin.project.db.ProjectJPAStatementInjector.getQuery ProjectJPAStatementInjector.java(208)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAQueryBuilder.build JPAQueryBuilder.java(87)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAProcessorImpl.process JPAProcessorImpl.java(148) (OData getEntitySet)
...
	
Existing reader to that ListExpressionOperator instance:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reentrancy [Thread[http-nio-8080-exec-3,5,main] (alive) R]
    org.eclipse.persistence.expressions.ListExpressionOperator.copyTo ListExpressionOperator.java(56)
    org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL ArgumentListFunctionExpression.java(100)
    org.eclipse.persistence.expressions.ExpressionOperator.printDuo ExpressionOperator.java(2238)
    org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL CompoundExpression.java(288)
    org.eclipse.persistence.internal.expressions.RelationExpression.printSQL RelationExpression.java(901)
    org.eclipse.persistence.expressions.ExpressionOperator.printDuo ExpressionOperator.java(2243)
    org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL CompoundExpression.java(288)
    org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.translateExpression ExpressionSQLPrinter.java(326)
    org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printExpression ExpressionSQLPrinter.java(132)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL SQLSelectStatement.java(1758)
    org.eclipse.persistence.platform.database.HANAPlatform.printSQLSelectStatement HANAPlatform.java(349)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall SQLSelectStatement.java(846)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall SQLSelectStatement.java(857)
    org.eclipse.persistence.descriptors.ClassDescriptor.buildCallFromStatement ClassDescriptor.java(879)
    org.eclipse.persistence.internal.queries.StatementQueryMechanism.setCallFromStatement StatementQueryMechanism.java(392)
    org.eclipse.persistence.internal.queries.StatementQueryMechanism.prepareSelectAllRows StatementQueryMechanism.java(317)
    org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.prepareSelectAllRows ExpressionQueryMechanism.java(1726)
    org.eclipse.persistence.queries.ReadAllQuery.prepareSelectAllRows ReadAllQuery.java(910)
    org.eclipse.persistence.queries.ReadAllQuery.prepare ReadAllQuery.java(841)
    org.eclipse.persistence.queries.DatabaseQuery.checkPrepare DatabaseQuery.java(676)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare ObjectLevelReadQuery.java(960)
    org.eclipse.persistence.queries.DatabaseQuery.checkPrepare DatabaseQuery.java(625)
    org.eclipse.persistence.queries.DatabaseQuery.execute DatabaseQuery.java(882)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.execute ObjectLevelReadQuery.java(1183)
    org.eclipse.persistence.queries.ReadAllQuery.execute ReadAllQuery.java(485)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork ObjectLevelReadQuery.java(1271)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery UnitOfWorkImpl.java(2981)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1895)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1877)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1842)
    org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery QueryImpl.java(262)
    org.eclipse.persistence.internal.jpa.QueryImpl.getResultList QueryImpl.java(482)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAPage$JPAPageBuilder.buildFromQuery JPAPage.java(126)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAPage$JPAPageBuilder.build JPAPage.java(95)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAProcessorImpl.handlePaging JPAProcessorImpl.java(456)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAProcessorImpl.process JPAProcessorImpl.java(166) (OData getEntitySet)
...
	

A new writer to that ListExpressionOperator instance enters:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Reentrancy [Thread[http-nio-8080-exec-6,5,main] (alive) W]
    org.eclipse.persistence.expressions.ListExpressionOperator.setNumberOfItems ListExpressionOperator.java(130)
    org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.initializePlatformOperator ArgumentListFunctionExpression.java(120)
    org.eclipse.persistence.internal.expressions.FunctionExpression.getPlatformOperator FunctionExpression.java(332)
    org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression.printSQL ArgumentListFunctionExpression.java(99)
    org.eclipse.persistence.expressions.ExpressionOperator.printDuo ExpressionOperator.java(2238)
    org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL CompoundExpression.java(288)
    org.eclipse.persistence.internal.expressions.RelationExpression.printSQL RelationExpression.java(901)
    org.eclipse.persistence.expressions.ExpressionOperator.printDuo ExpressionOperator.java(2243)
    org.eclipse.persistence.internal.expressions.CompoundExpression.printSQL CompoundExpression.java(288)
    org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.translateExpression ExpressionSQLPrinter.java(326)
    org.eclipse.persistence.internal.expressions.ExpressionSQLPrinter.printExpression ExpressionSQLPrinter.java(132)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.printSQL SQLSelectStatement.java(1758)
    org.eclipse.persistence.platform.database.HANAPlatform.printSQLSelectStatement HANAPlatform.java(349)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall SQLSelectStatement.java(846)
    org.eclipse.persistence.internal.expressions.SQLSelectStatement.buildCall SQLSelectStatement.java(857)
    org.eclipse.persistence.descriptors.ClassDescriptor.buildCallFromStatement ClassDescriptor.java(879)
    org.eclipse.persistence.internal.queries.StatementQueryMechanism.setCallFromStatement StatementQueryMechanism.java(392)
    org.eclipse.persistence.internal.queries.StatementQueryMechanism.prepareSelectAllRows StatementQueryMechanism.java(317)
    org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.prepareSelectAllRows ExpressionQueryMechanism.java(1726)
    org.eclipse.persistence.queries.ReadAllQuery.prepareSelectAllRows ReadAllQuery.java(910)
    org.eclipse.persistence.queries.ReadAllQuery.prepare ReadAllQuery.java(841)
    org.eclipse.persistence.queries.DatabaseQuery.checkPrepare DatabaseQuery.java(676)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.checkPrepare ObjectLevelReadQuery.java(960)
    org.eclipse.persistence.queries.DatabaseQuery.checkPrepare DatabaseQuery.java(625)
    org.eclipse.persistence.queries.DatabaseQuery.execute DatabaseQuery.java(882)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.execute ObjectLevelReadQuery.java(1183)
    org.eclipse.persistence.queries.ReadAllQuery.execute ReadAllQuery.java(485)
    org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork ObjectLevelReadQuery.java(1271)
    org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery UnitOfWorkImpl.java(2981)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1895)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1877)
    org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery AbstractSession.java(1842)
    org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery QueryImpl.java(262)
    org.eclipse.persistence.internal.jpa.QueryImpl.getResultList QueryImpl.java(482)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAPage$JPAPageBuilder.buildFromQuery JPAPage.java(126)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAPage$JPAPageBuilder.build JPAPage.java(95)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAProcessorImpl.handlePaging JPAProcessorImpl.java(456)
    org.apache.olingo.odata2.jpa.processor.core.access.data.JPAProcessorImpl.process JPAProcessorImpl.java(166) (OData getEntitySet)
...

Comment 10 Patrick Haller 2019-04-23 13:49:20 EDT

A prerequisite is also a JPQL query that is eligible for parse caching: thus being unnamed, dynamic, and does not have any query hints.

Comment 11 Patrick Haller 2019-04-23 16:35:28 EDT

Looks as if the following solves the problem. Investigating further.

org.eclipse.persistence.internal.expressions.ArgumentListFunctionExpression

    @Override
    protected void postCopyIn(Map alreadyDone)
    {
        // The current ArgumentListFunctionExpression just got cloned shallow.
        // We need a new operator: as our ListExpressionOperator instance has a
        // state, "numberOfItems", it must be cloned not to be shared across
        // parallel threads when an unnamed, un-query-hinted, shared, cached
        // query is executed.
        // This is typical for dynamic JPQL.
        final ListExpressionOperator originalOperator = ((ListExpressionOperator) this.operator);
        this.operator = new ListExpressionOperator();
        originalOperator.copyTo(this.operator);
    
        final Boolean hasLastChildCopy = hasLastChild;
        hasLastChild = null;
        super.postCopyIn(alreadyDone);
        hasLastChild = hasLastChildCopy;
    }

Comment 12 Patrick Haller 2019-04-24 08:36:05 EDT

Created attachment 278377 [details]
Fix for bug #463042

Proposed patch to fix #463042.

Comment 13 Rich DiCroce 2019-04-24 09:33:38 EDT

You should open a PR on GitHub. That appears to be the official method for submitting a patch now. The devs recently accepted one of my patches via that method. See https://github.com/eclipse-ee4j/eclipselink/wiki/Creating-a-Pull-Request

Comment 14 Patrick Haller 2019-04-25 11:44:47 EDT

Created attachment 278400 [details]
Updated patch

Comment 15 Patrick Haller 2019-04-25 11:47:16 EDT

PR is created as https://github.com/eclipse-ee4j/eclipselink/pull/425.
Travis tests look good now in https://travis-ci.org/eclipse-ee4j/eclipselink/builds/524493491.

Attached patch is updated, too.

Comment 16 Eclipse Webmaster 2022-06-09 10:03:41 EDT

The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink

Comment 17 Eclipse Webmaster 2022-06-09 10:08:37 EDT

The Eclipselink project has moved to Github: https://github.com/eclipse-ee4j/eclipselink