Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [eclipselink-dev] questions while running JPA JUnit tests

Hi Dies,

Andrei just pointed out a potential workaround for the issue with MetadataProject issue below that could allow us to run the tests without changing every table generator that uses defaults.

The JPA processing uses a generator called "SEQ_GEN" by default. It might be possible to define this sequence generator in an annotation or in xml somewhere in the tests and override the default names. That generator could be used to override the reserved words.

-Tom

Tom Ware wrote:
Hi Dies,

Dies Koper wrote:
Hi Tom,

I have checked in your code.

Thanks!

Andrei pointed our an issue with the checked-in code. MetadataProject has a change to try to get the default sequence table from the Platform. This issue is that this code is generally called before the platform has been set. (i.e. it will always be DatabasePlatform and never a subclass) I cannot think of a good solution for this issue other than to have symfoware users not use the default sequence table name - this is not an area we would want to change the default for other users for backwards compatibility reasons.

<snip>
Do the suites containing these tests run successfully when run
individually?

I had no time today to check, will get back to you about that next week.

I think that if we can make ensure that all tests leave tables in the
same state they started, we can make the majority of tests pass.

Are you suggesting we look for a solution where we can disable the
drop/create processing on Symfoware?

I suppose I can easily try that by adding some flag to the models'
replaceTables methods that will make sure it only tries to recreate
the tables the first time.

That is the idea. We had initially done some work to try to get the
table creation to do down its own connection, but I do not think we were
able to get it to work. The only other thing I can think of is to do
what you suggest above. I think we should start with a single test suite
and see how that goes for us.

I lost the ability to build eclipselink from Eclipse when some of the dependency jars went up a version a while ago, so I tried something else first.

For the tests which failed in testSetup due to a table locking error I set SchemaManager.FAST_TABLE_CREATOR to true just before the replaceTables() call. With that in place the JPA test set passes 80% (as opposed to 60% before). I haven't gone through all the remaining failing tests yet, but most are due to the update all issue.

I had not thought of that option.  It is a good idea.


The first test to fail is jpa.advanced.EntityManagerJUnitTestSuite#testUpdateAllProjects. All tests that do fail in this suite are due to the update all issue (except for testEMCloseAndOpen, need to investigate why it fails).

For the DeleteAll and UpdateAll tests that fail due to the temp table
issue, we will have to look at them individually and decide if the test
should be disabled (if it is specifically testing Update All or Delete
All), or if we can have the same affect as the UpdateAll or DeleteAll
query in another way.

I made a list of the exact tests within EntityManagerJUnitTestSuite (to start with), that fail due to this issue:

I took a look the following tests explicitly test Update/DeleteAll. We should disable these tests when running on Symforware

- testUpdateAllProjects
- testUpdateUsingTempStorage
testFindDeleteAllPersist
testAnnotationDefaultLockModeNONEOnUpdateQuery
- testUpdateAllProjectsWithNullTeamLeader
- testUpdateAllLargeProjectsWithNullTeamLeader
- testUpdateAllSmallProjectsWithNullTeamLeader
- testUpdateAllProjectsWithName
- testUpdateAllLargeProjectsWithName
- testUpdateAllSmallProjectsWithName
- testUpdateAllLargeProjects
- testUpdateAllSmallProjects
- testUpdateUsingTempStorageWithParameter
- testDeleteAllLargeProjectsWithNullTeamLeader
- testDeleteAllProjectsWithNullTeamLeader
- testNullifyAddressIn
- testReadTransactionIsolation_OriginalInCache_UpdateAll_Refresh_Flush
- testReadTransactionIsolation_OriginalInCache_UpdateAll_Refresh
- testReadTransactionIsolation_OriginalInCache_UpdateAll_Flush
- testReadTransactionIsolation_OriginalInCache_UpdateAll
- testReadTransactionIsolation_UpdateAll_Refresh_Flush
- testReadTransactionIsolation_UpdateAll_Refresh
- testReadTransactionIsolation_UpdateAll_Flush
- testReadTransactionIsolation_UpdateAll
- testBulkDeleteThenMerge

The following tests use Update/DeleteAll for clean up or test config and could be updated to avoid using those queries:

- testExtendedPersistenceContext
- testRemoveFlushFind
- testRemoveFlushPersistContains
- testFlushModeOnUpdateQuery
- testRefreshRemoved
- testReadTransactionIsolation_OriginalInCache_CustomUpdate_Refresh_Flush
- testReadTransactionIsolation_OriginalInCache_CustomUpdate_Refresh
- testReadTransactionIsolation_OriginalInCache_CustomUpdate_Flush
- testReadTransactionIsolation_OriginalInCache_CustomUpdate
- testReadTransactionIsolation_CustomUpdate_Refresh_Flush
- testReadTransactionIsolation_CustomUpdate_Refresh
- testReadTransactionIsolation_CustomUpdate_Flush
- testReadTransactionIsolation_CustomUpdate
- testPersistRemoved
- testFlushModeEmCommit
- testFlushModeEmCommitQueryCommit
- testFlushModeEmAutoQueryAuto
- testFlushModeEmAuto
- testFlushModeEmAutoQueryCommit

I have added the following enhancement request to bugzilla to track the base issue:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=298193

I suspect that the fix is fairly involved, but have not done much research.

When we have dealt with the issues in the Core SRG and EntityManagerJUnitTestSuite we will have a passing SRG which will be the first step towards getting SymfowarePlatform included in the actual product. After that, how quickly we can proceed will depend on the community acceptance of the proposal to define how DatabasePlatforms get included in the product. As I mentioned in an earlier email, I hope to get a draft of that proposal out soon.

-Tom

testUpdateAllProjects
testUpdateUsingTempStorage
testFindDeleteAllPersist
testExtendedPersistenceContext
testRemoveFlushFind
testRemoveFlushPersistContains
testFlushModeOnUpdateQuery
testAnnotationDefaultLockModeNONEOnUpdateQuery
testRefreshRemoved
testUpdateAllProjectsWithNullTeamLeader
testUpdateAllLargeProjectsWithNullTeamLeader
testUpdateAllSmallProjectsWithNullTeamLeader
testUpdateAllProjectsWithName
testUpdateAllLargeProjectsWithName
testUpdateAllSmallProjectsWithName
testUpdateAllLargeProjects
testUpdateAllSmallProjects
testUpdateUsingTempStorageWithParameter
testDeleteAllLargeProjectsWithNullTeamLeader
testDeleteAllProjectsWithNullTeamLeader
testNullifyAddressIn
testReadTransactionIsolation_OriginalInCache_UpdateAll_Refresh_Flush
testReadTransactionIsolation_OriginalInCache_UpdateAll_Refresh
testReadTransactionIsolation_OriginalInCache_UpdateAll_Flush
testReadTransactionIsolation_OriginalInCache_UpdateAll
testReadTransactionIsolation_OriginalInCache_CustomUpdate_Refresh_Flush
testReadTransactionIsolation_OriginalInCache_CustomUpdate_Refresh
testReadTransactionIsolation_OriginalInCache_CustomUpdate_Flush
testReadTransactionIsolation_OriginalInCache_CustomUpdate
testReadTransactionIsolation_UpdateAll_Refresh_Flush
testReadTransactionIsolation_UpdateAll_Refresh
testReadTransactionIsolation_UpdateAll_Flush
testReadTransactionIsolation_UpdateAll
testReadTransactionIsolation_CustomUpdate_Refresh_Flush
testReadTransactionIsolation_CustomUpdate_Refresh
testReadTransactionIsolation_CustomUpdate_Flush
testReadTransactionIsolation_CustomUpdate
testPersistRemoved
testFlushModeEmCommit
testFlushModeEmCommitQueryCommit
testFlushModeEmAutoQueryAuto
testFlushModeEmAuto
testFlushModeEmAutoQueryCommit
testBulkDeleteThenMerge

How do we know whether the test is specifically testing update all or delete all?

Thanks,
Dies



-Tom

--

On a side note, I have started a discussion about how we can lower the
barrier required to include new database platforms in the shipped
version of EclipseLink with some of the other commiters. Watch the dev
mailing list for an initial proposal in the next week or so.

Thanks, looking forward to it!





Regards,
Dies


Dies Koper wrote:
Wow!! First time a live response! :))
You must be at the office late :)

Thanks. Yes, if you have some suggestions about the other two issues
that should really help me forward.

Cheers,
Dies


On 16/12/2009 08:27, Tom Ware wrote:
Hi Dies,

In general, the other changes look good. I am hoping to run some
testing
tomorrow and also to take a look at some of the other questions from
your email.

-Tom

Dies Koper wrote:
Hi James, Tom,

Yes, I was wondering about that.
Okay, I'll drop the custom pagination support.

Tom, have you been able to review/commit the other changes in the
patch?

Thanks!
Dies


On 16/12/2009 02:42, James Sutherland wrote:
Just an FYI Dies,

I'm not sure the behavior on Symfoware, but on most databases the
benefit in putting limit/offset in the SQL is for the offset, not
the
limit.
For the limit EclipseLink uses the JDBC maxRows API by default,
which
seems to have equivalent or better performance on most databases I
have tested.
If Symfoware does not support the offset, there may be little point
adding the custom pagination support.


----- Original Message -----
From: tom.ware@xxxxxxxxxx
To: eclipselink-dev@xxxxxxxxxxx
Sent: Tuesday, December 15, 2009 10:32:18 AM GMT -05:00 US/Canada
Eastern
Subject: Re: [eclipselink-dev] questions while running JPA JUnit
tests

Hi Dies,

To check this in, the setting will have to be completely split. i.e.
remove
the methods that affect both settings
(setIgnoreFirstResultsMaxRowsSettings and
shouldIgnoreFirstResultsMaxRowsSettings). Instead, have separate
getters and
setters for each setting, and make the appropriate changes to
classes
depending
on those methods.

-Tom

Dies Koper wrote:
Hi Tom,

I have started looking at your patch. I have a quick question:

- In DatabaseAccessor, what is the goal of splitting the
ignoreFirstRowSettings and the ignoreMaxResultsSettings

Symfoware supports LIMIT, but has no equivalent for OFFSET.

Currently, the flag setIgnoreFirstRowMaxResultsSettings toggles
both, so
either both JDBC absolute and setMaxRows are called, or both are
not.

To support pagination using SQL I override printSQLSelectStatement
like
the other platforms do, but as I only have a LIMIT in the query I
need
EclipseLink to only call the JDBC absolute method when FirstRow is
set.

So my implementation method looks like:

public void printSQLSelectStatement(DatabaseCall call,
ExpressionSQLPrinter printer, SQLSelectStatement statement) {
int max = 0;
int firstRow = 0;

statement.setMaximumAliasLength(getMaxFieldNameSize());
if (statement.getQuery() != null) {
max = statement.getQuery().getMaxRows();
firstRow = statement.getQuery().getFirstResult();
}

// only MaxRows can be worked into the statement
if (max> 0&& this.shouldUseRownumFiltering()) {
statement.setUseUniqueFieldAliases(true);
call.setFields(statement.printSQL(printer));
printer.printString(" WITH OPTION LIMIT (");
printer.printParameter(DatabaseCall.MAXROW_FIELD);
printer.printString(")");

call.setIgnoreMaxResultsSettings(true);
} else {
// use JDBC absolute and setMaxRows methods for pagination
super.printSQLSelectStatement(call, printer, statement);
}
}

Thanks,
Dies


Dies Koper wrote:
Hi Tom, all

Congrats on the 2.0 release!

I've been continuing running the test sets on the Symfoware
platform.
May I ask you for you help with the following three issues?

1. Review of patch

I've added a patch with changes required for a number of tests to
run. Could you review them, and if okay, commit them?
https://bugs.eclipse.org/bugs/show_bug.cgi?id=288715

2. Where is table name "SEQUENCE" defined?

I tried renaming all occurrences of the word "SEQUENCE" in the
code I
could find but still some tests try to create a sequence table
with
table name SEQUENCE.
It starts with the 8th test in the JPA test set:
testTransitionToDeferedFailure

which tries a UPDATE SEQUENCE (...).

In the log I see even more:

[junit] [EL Config]:
ServerSession(1006920425)--Thread(Thread[main,5,main])--The table
generator name defined within [class
org.eclipse.persistence.testing.models.jpa.inherited.Alpine] is
being
defaulted to: SEQUENCE.

and

[junit] [EL Fine]:
ServerSession(1006920425)--Connection(753758527)--Thread(Thread[main,5,main])--CREATE



TABLE SEQUENCE (SEQ_NAME VARCHAR(50) NOT NULL, SEQ_COUNT
DECIMAL(18),
PRIMARY KEY (SEQ_NAME))
[junit] java.sql.SQLException: [SymfoWARE ODBC Driver][SymfoWARE
[...]
[junit] at
org.eclipse.persistence.testing.tests.jpa.advanced.concurrency.LifecycleJUnitTest.testSetup(LifecycleJUnitTest.java:76)





I believe these might be from the CONCURRENCYA, B and C
tables, or
ConcurrencyA, B and C entities. Each of them has
fieldID.setIsIdentity(true) resp. @GeneratedValue specified
but the
details of the sequence generators are omitted.

I though it would default to what the platform's
createPlatformDefaultSequence() method returns but apparently
not.

Many tests fail because of this issue.

3. Table locks during DROP TABLE

In the SymfowarePlatform wiki page you suggested that we should
start
by trying to get the Core SRG running first. Most are passing
now.

Only a few (the ReadAllTest tests, which rely on
AutoTableGeneratorBasicTestModel.java) failed I think because of
the
table lock issue.

The error message is:

Exception Description: 24 objects were read from the database,
but
originially there were, 12.

When I run it by itself through the test browser it passes
without
problems.

Some messages in the finest log let me believe this is because a
problem
occurred while recreating the tables: they should have been
dropped and
recreated, thereby clearing those old 12 object, but because
of the
locking issue they were not.

[junit] Internal Exception: java.sql.SQLException: [SymfoWARE
ODBC
Driver][SymfoWARE Server] JYP3913E : Table "PHONE" being used
exclusively by another user.
[junit] Error Code: -3913
[junit] Call: drop table PHONE
[junit] Query: DataModifyQuery(sql="drop table PHONE")
[junit] at
org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:324)




[..]
[junit] at
org.eclipse.persistence.testing.models.employee.relational.EmployeeSystem.dropTableConstraints(EmployeeSystem.java:86)




[junit] at
org.eclipse.persistence.testing.tests.schemaframework.AutoTableGeneratorEmployeeSystem.createTables(AutoTableGeneratorEmployeeSystem.java:32)






This table was created and accessed as follows:

1. drop (index) and table (outside transaction)
DROP TABLE PHONE

2. create table (and index) (in transaction), same connection
CREATE TABLE PHONE (EMP_ID NUMERIC(15) NOT NULL, TYPE VARCHAR(15) NOT NULL, AREA_CODE VARCHAR(3), P_NUMBER VARCHAR(7), PRIMARY KEY
(EMP_ID, TYPE))

3. insert 30 rows (in single transaction), same connection
INSERT INTO PHONE (TYPE, AREA_CODE, P_NUMBER, EMP_ID) VALUES
(?, ?,
?, ?)
[junit] bind => [Home, 613, 5551234, 105]

4. drop (index) and table (outside transaction), different
connection
DROP TABLE PHONE
-> SQLException saying PHONE is locked.

I can reproduce this in a simple JDBC application. If I close the
connection used in steps 1-3 before step 4 it works fine.

I can see some session swapping is done in
AutoTableGeneratorBasicTestModel#addForcedRequiredSystems() so I
tried adding a reset() invocation at the start of its
addForcedRequiredSystems() method but that did not help.
How can I make it close the physical connection?
(You mentioned adding logic into TestSystem?)

The lock issue occurs much more often in the JPA test set. The
first
failing test however is because of the reserved SEQUENCE keyword
used
as table name, so if you could help me in the right direction
with
the above three issues I can try move forward with this test set
again.

Thanks!
Dies


Tom Ware wrote:
Hi Dies,

FYI: I am working on integrating your initial patch. There are a
couple of items I will not be integrating at the moment:

1. Specific references to SymfowarePlatform and
isSymfowarePlatform(). These will be left until
SymfowarePlatform
becomes part of the full product

2. Changes to default SEQUENCE table name to "\"SEQUENCE\"" for
our
Sessions and project XML (in XMLSessionConfigProject,
ObjectPersistenceRuntimeXMLProject and SequencingConfig).
This is
simply to big a backward compatibility risk since I cannot test
these changes on every database platform that EclipseLink is
run on
by our customers. I am looking for some way of having this
information stored on the DatabasePlatform, but at the moment,
it is
not looking good. Assuming I do not find a good solution,
Symfoware
users will simply have to explicitly set these items when using
our
proprietary configuration code (sessions.xml and
deployment.xml) We
will also have to address anywhere in the tests we choose to run
where this is an issue.

I am in the process of running testing on the other changes and
I'll
let you know when it gets checked in.

-Tom

Dies Koper wrote:
Hi Tom,

I have added the outstanding issues with SQL keywords in
table/columns names, and maximum precision. The biggest open
issue
is of course the drop table restriction. I'm looking forward to
hearing what solution(s) you can come up with.

Thanks!
Dies


Tom Ware wrote:
Hi Dies,

I am moving some of the main points of this discussion to the
wiki page so I can more easily keep track of where we are:

http://wiki.eclipse.org/EclipseLink/Development/Incubator/Extensions/SymfowarePlatform#Open_Issues





I started by just adding some basics about the main issues
that
are getting in the way at the moment. I plan to add more
detail
next week. Please feel free to add other issues I have
missed, or
more detail.

As for some of the other things you were wondering about:

1. Your initial patch (including sequencing fixes etc.). I
hope
to find some time to start integrating it next week

2. EclipseLink bug 286907 (not related to Symfoware platform)-
this bug is in the queue and will be addressed in sequence
with
the other bugs. The fact that you have submitted a patch
means it
will potentially jump other bugs in the queue. At the moment
I am
going to focus on the Symfoware Platform issues with my free
time,
so I can't make any promises about when it will be included.

3. GlassFish bug 9179 (not related to Symfoware platform) -
same
thing as above, but additionally: we cannot legally make
use of
the submitted patch until it is attached to EclipseLink bug by
someone with IP rights

I'll get started looking for some solutions next week.
-Tom

_______________________________________________
eclipselink-dev mailing list
eclipselink-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/eclipselink-dev



Back to the top