Community
Participate
Working Groups
The W3C has a Test Suite for XPath 2.0 and Xquery 1.0 implementation verification. XPath 2.0 is a subset of XQuery 1.0, so the test suite can be used to test the XPath 2.0 specific functionality. This test suite is currently working it's way through the Eclipse IP approval process and has been given preliminary permission to be checked into CVS. There are several things that will need to be done. 1. Generate the necessary Unit Tests using Junit as the framework. Creating a eclipse junit plugin that can be used to run the tests. 2. Create bugs for each of the tests that fail. 3. Create patches for the bugs, if they depend on each other this should be noted as well. This will provide a good baseline for the processor and the xpath 2.0 parser so that future XSLT and XQuery implementations can be assured they are using a compliant processing engine for validation and execution of the language.
Hi, all, I'm in coding for this bug. Some hard-coded works are going on now;) Becuase the PsychoPath just supports the xpath 2.0, but not the full xquery 1.0. There is a problem, whether the PsychoPath support two input documents? such as, ($input-context1//empnum,$input-context2//empname) the $input-context1 and the $input-context2 are bound to two different document nodes. In fact, this is the test case, <test-case is-XPath2="true" name="sequenceexpressionhc9" FilePath="Expressions/SeqExpr/ConstructSeq/commaOp/" scenario="standard" Creator="Carmelo Montanez"> <description>Simple sequence expression resulting by quering string data from multiple xml source.</description> <spec-citation spec="XQuery" section-number="3.3.1" section-title="Constructing Sequences" section-pointer="construct_seq"/> <query name="sequenceexpressionhc9" date="2005-02-14"/> <input-file role="principal-data" variable="input-context1">works</input-file> <input-file role="principal-data" variable="input-context2">staff</input-file> <output-file role="principal" compare="Fragment">sequenceexpressionhc9.txt</output-file> </test-case> My test codes will test all test-case with (is-XPath2="true"), is it good? some suggestions?
seem a way could possibly handle this problem. But seem a bug... I push it to tomorrow. Any suggestion or idea is still wellcome.
(In reply to comment #2) > seem a way could possibly handle this problem. But seem a bug... I push it to > tomorrow. > > Any suggestion or idea is still wellcome. > I need to take a look at this...there should be a Query in the test suite that does this. One thing that might be interesting is writing an XSLT to take the test suite tests, and convert them into the appropriate Junit tests, so they can be run through junit. Especially since their are over 15,000 tests, it could take a while to write them all by hand. :)
(In reply to comment #3) > (In reply to comment #2) > > seem a way could possibly handle this problem. But seem a bug... I push it to > > tomorrow. > > > > Any suggestion or idea is still wellcome. > > > > I need to take a look at this...there should be a Query in the test suite that > does this. > > One thing that might be interesting is writing an XSLT to take the test suite > tests, and convert them into the appropriate Junit tests, so they can be run > through junit. > > Especially since their are over 15,000 tests, it could take a while to write > them all by hand. :) Alright, on this one it looks like what you will need to do is before you evaluate the Xpath Expression, is preload the variables, with the external data. It looks like in the DefaultDynamicContext you can set the value of a variable. using the set_variable method. You'll need to construct the appropriate variables, and load them with the data. Then do the comparison. This also could be one of those features that is actually XQuery and not XPath 2.0 per say. But you are correct you will have to do some hard coding in cases like this. The only way I know that it will go against multiple documents is when the collection() method is used.
It looks like you will need to load two DOM instances with the external data. Then create up two ElementType data types to hold these values. Load one of each with the root Elements of the DOMs that were loaded. Here is some pseudo code, I haven't actually written anything yet: Document dom1 = loadDom1(dom1File) Document dom2 = loadDom2(dom2File) ElementType element1 = new ElementType(dom1.getDocumentElement(), 0); ElementType element2 = new ElementType(dom2.getDocumentElement(), 0); // Set the Variables for the DynamicContext QName variable1 = new QName("name of variable"); QName variable2 = new QName("name of variable"); dc.set_variable(variable1, element1); dc.set_variable(variable2, element2); dc Stands for DynamicContext, take a look at the TestXPath20 class for how to setup the dynamic context. The above in theory should allow you to setup and initalize the variables you need.
Dave, your codes works! I'd like to use it:) my way is using the doc() function, it works as well, but that seems a bug in the doc implementation (which delegated to the DynamicContex...) as my understanding. I will fire a bug report for it when finishing this bug coding partly. One unelegant point is, the DynamicContext need a document node whenever the xpath expr need or not. Is it a bug? The third found bug is that, the fn:collection function seems not implemented in the PsychoPath.(or?) So, some collection function tests in w3c testsuite will fail definitely. Ps: I have not rewritten the tests one file by one file. I'm writing a xquery script to extract the test-case infos with (is-XPath2="true") from XQTSCatalog.xml to a report.xml fisrtly. Then, some analysis codes will setup the context for. Then a method is used to prepare the the data. I used the JUnit 4.x parameterized test functionility. This is a tradeoff solution. The maintainability and non-invasion is my main consideration. But it is still possible to use my or other codes to generate the java test method one by one(which could be used in Junit 3.x). Hand-writing one by one seem not a rational decision?:) Dave, how about you think this solution? (In reply to comment #5) > It looks like you will need to load two DOM instances with the external data. > > Then create up two ElementType data types to hold these values. Load one of > each with the root Elements of the DOMs that were loaded. Here is some pseudo > code, I haven't actually written anything yet: > > Document dom1 = loadDom1(dom1File) > Document dom2 = loadDom2(dom2File) > > ElementType element1 = new ElementType(dom1.getDocumentElement(), 0); > ElementType element2 = new ElementType(dom2.getDocumentElement(), 0); > > // Set the Variables for the DynamicContext > QName variable1 = new QName("name of variable"); > QName variable2 = new QName("name of variable"); > > dc.set_variable(variable1, element1); > dc.set_variable(variable2, element2); > > dc Stands for DynamicContext, take a look at the TestXPath20 class for how to > setup the dynamic context. The above in theory should allow you to setup and > initalize the variables you need. >
(In reply to comment #6) > Dave, your codes works! I'd like to use it:) Glad the code works. I did not know if it would or not. :) > my way is using the doc() function, it works as well, but that seems a bug in > the doc implementation (which delegated to the DynamicContex...) as my > understanding. I will fire a bug report for it when finishing this bug coding > partly. > > One unelegant point is, the DynamicContext need a document node whenever the > xpath expr need or not. Is it a bug? > > The third found bug is that, the fn:collection function seems not implemented > in the PsychoPath.(or?) So, some collection function tests in w3c testsuite > will fail definitely. Please open separate bugs for each of these. Prefix it with [xpath 2.0][w3c test suite]. This way if you have a patch for the bug, then you can attach it to the bug, and I can apply it. > > Ps: > I have not rewritten the tests one file by one file. I'm writing a xquery > script to extract the test-case infos with (is-XPath2="true") from > XQTSCatalog.xml to a report.xml fisrtly. Then, some analysis codes will setup > the context for. Then a method is used to prepare the the data. I used the > JUnit 4.x parameterized test functionility. This is a tradeoff solution. The > maintainability and non-invasion is my main consideration. But it is still > possible to use my or other codes to generate the java test method one by > one(which could be used in Junit 3.x). Hand-writing one by one seem not a > rational decision?:) Can you attach what you have written so far to this bug? While I have no problem using XQuery to create the tests, I would like to generate the appropriate Junit tests (does not matter if it's 3.x or 4.x). I do not necessarily want to generate one test file that contains 15,000 tests, but would like to group them if possible. I realize there are going to be some that we will have to do manually or exclude from the test suite as they may only apply to XQuery 1.0. Thanks for taking the initiative to work on this.
Hi, Dave: the most of work have been done. 2.14 is Valentine's Day, so I stop one day to...:) but some problems seem odd in some little scale tests. First, the attribute is-XPath2="true" is not 100% reliable. I open a bug to w3c[1], which was confirmed by Michael Kay indirectly. some type support in PsychoPath is some weird. In fact, in you written codes( codes in your blog or the cvs codes), such as getGrammar(), is unuseful, because it return *null*. And in fact, if we want to use the type, it seems we need to validate the xml file. However, i think xs(http://www.w3.org/2001/XMLSchema) namespace should be default. This problem stop me running such simple expression like "xs:integer(1)" for PsychoPath. DO you have some ideas? I hope to submit these codes here tomorrow or so, after running a whole tests (with 8225 testcases with is-XPath2="true") to avoid some low-level mistakes:) [1] http://www.w3.org/Bugs/Public/show_bug.cgi?id=6581
(In reply to comment #8) > First, the attribute is-XPath2="true" is not 100% reliable. I open a bug to > w3c[1], which was confirmed by Michael Kay indirectly. Thanks, I'm adding myself to that bug so we can track it. For some of the other questions I've added Andrea Bittau the original author of Psychopath to see if he can help answer some of the questions. One of the things to keep in mind is that Psychopath's implementation may not pass all of the tests in the test suite. It is based on a 2005 draft of the XPath 2.0 specification, so will more than likely need some bug fixing and enhancements to bring it in line with the current specification. Which is one of the main purposes for running it against the W3C test suite. > > some type support in PsychoPath is some weird. In fact, in you written codes( > codes in your blog or the cvs codes), such as getGrammar(), is unuseful, > because it return *null*. And in fact, if we want to use the type, it seems we > need to validate the xml file. However, i think > xs(http://www.w3.org/2001/XMLSchema) namespace should be default. This problem > stop me running such simple expression like "xs:integer(1)" for PsychoPath. DO > you have some ideas? Not sure...it would help to see the exact situation. Did you make sure to add the XML Schema namespaces to the Dynamic Context? You may also want to review the original paper that was published about Psychopath. http://psychopath.sourceforge.net/report.pdf > > I hope to submit these codes here tomorrow or so, after running a whole tests > (with 8225 testcases with is-XPath2="true") to avoid some low-level mistakes:) Thanks, that would be great.
Created attachment 125774 [details] org.eclipse.wst.xml.xpath2.processor.tests.xqts first version of test codes
I upload the test codes. It is large for containing part of w3c test suite material. There should be some improvement spaces in this version. But I'd like show it earlier. some comments: now the failture/running raito is 5918/8225. 1. some detailed failure seem not implemented by PP(short for PsychoPath, as follows). The xqueery1/xpath2 specify the detail errors should be thrown when errors ocurred, such as XQST0015. in this cases, the test will fail for the codes just give a simple error string. 2. some xs type expression are supported, but others are not. It is odd. such as, supported: (2) castable as xs:integer? not supported: (1,2) instance of xs:integer? in this cases, the test will fail. 3. bug in the fn namespace in PP. the fn is http://www.w3.org/2005/xpath-functions, not http://www.w3.org/2004/10/xpath-functions. in the test code, the fn is bound to "http://www.w3.org/2004/10/xpath-functions" to make more successful runnings. This should be corrected in some time. ps: my words, "xs(http://www.w3.org/2001/XMLSchema) namespace should be default", seem just apply for xquery1. it seems no forced predeclared namespace in xpath 2 document[1]. 3. some regular express seem wrong? such as: fn:matches("abracadabra^abracadabra", "\^") 4. some test cases is not xpath 2.0 grammar. I mention this before. and I've seen more cases when testing. it seems the better way is to use a parser to identify the two different grammar, than like that of my codes,which use some simple string tools or regular expression. in this cases, the test will fail. 5. some number test result is doublized in PP, but w3c like to keep the orignal form. such as, 123.45 vs 1.2345E2 in this cases, the test will fail now in my codes. ...:) [1] http://www.w3.org/TR/xpath20
Some of these should be fairly easy to address, in particular the function namespace issue. Thanks for uploading the file. I'll take a look at it later.
(In reply to comment #11) > I upload the test codes. It is large for containing part of w3c test suite > material. > There should be some improvement spaces in this version. But I'd like show it > earlier. > A couple of suggestions for the test suite, (which looks good btw). 1. Add the contributions to the org.eclipse.wst.xml.xpath2.processor.tests instead of a separate plugin project. I already have the XQuery test suite files included there, so no need to duplicate the content. 2. Add your script to the same plugin. 3. If possible, it would be nice to know what Test it is running either by Test number or some other description if possible. As it is testXPathProcesor[10] does not let me know what test it is running. This will help in the future as we address specific test cases. If you can create a Patch file for this, that will help. Once the patch file is created, just obsolete the old one and I can have a better idea how much code is being contributed. If it's under 250 lines of code total, I don't have to send it through IP review, but if it's more than that, I'll have to send it to IP before I can add it. Looking good though.
I will try to create a patch for the test. Originally, I want to separate the automated XQTS test from the hand-written tests. Another reason is, my test is based on the JUnit4. And about: " 3. If possible, it would be nice to know what Test it is running either by Test number or some other description if possible. As it is testXPathProcesor[10] does not let me know what test it is running. This will help in the future as we address specific test cases. " In fact, now my codes will print the xqFile lcation in the JUnit View for each test failures, such as Error@/Queries/XQuery/Basic/.../sequence-type-8.xq: excepted... (it will be better to add a uri to the junit view,which could be clicked into the file directly. But I am sure whether it is supported. I will try it...)
Created attachment 126252 [details] zipped patch for org.eclipse.wst.xml.xpath2.processor.tests project
Created attachment 126253 [details] snapshot for patched tests
patch is created by the right mouse key, "Team"->"Create Patch...". the patch root is the tests project. I am not always using the patch functionality. so if there are some mistakes, please let me know:) Originally, I want to develope a little plugin to assist the tested xq script file location(and to throw a linked string seem not supported by junit). But the JUnit Plugin is running in another headless eclipse instance. so, to se such as functionilty need a deployment process. It seems too heavy for a test. So, I do some tweak to codes to reuturn a full OS path, and attach a snapshot to show how the copying and openning job works for me. It is possible to find some bugs using the new test codes. I'd like to reboot my xquery work. for the test work for PP will slow down in the later period. But if I find some new related things, I talk about it here:)
Thanks for the latest set of patches. I'll try and take a look later tonight. For fun, I ran your test suite with the eclemma plugin, which provides a code coverage report. The test suite as it is now, covers about 80% of the psychopath code base. Which is actually pretty good. http://www.eclemma.org/index.html
Jin, I've taken your contributed code and added submitted to the eclipse IP department for review. Once they give me approval, I can get this checked in. Anything over 250 lines of code has to be sent through IP review.
Good news:) Don't mention any thanks. I'm very glad to contribute it:) Now the patched test has a problem, which the schema capacity has not been tested. Because, according to my analysis, the testcases with is-XPath="true" seem not including the schema tests. Adding these case by modification is possible. But, the tested cases show that the type system of PP may have some problem. It is better to find out the reason of current errors before going ahead.
Jin, IP needs a few standard simple questions answered: Contribution Confirmations: (a) wrote 100% of the code; (b) that they have the right to contribute the code to Eclipse; and (c) the file header contains the EPL License header. (c) I took care of when I submitted the code for review. Just respond here to option a and b. Thanks.
a (In reply to comment #21) > Jin, IP needs a few standard simple questions answered: > > Contribution Confirmations: > (a) wrote 100% of the code; > (b) that they have the right to contribute the code to > Eclipse; and > (c) the file header contains the EPL License header. > > > (c) I took care of when I submitted the code for review. Just respond here to > option a and b. Thanks. >
I actually need an answer to A and B as well.
(In reply to comment #21) > Jin, IP needs a few standard simple questions answered: > > Contribution Confirmations: > (a) wrote 100% of the code; almost 100% codes. A little codes from your blog(in fact, it is basically same to the cvs tests codes) or comments here:) I made some modifications to fit for the actual test codes. > (b) that they have the right to contribute the code to > Eclipse; and > (c) the file header contains the EPL License header. > > > (c) I took care of when I submitted the code for review. Just respond here to > option a and b. Thanks. > [1] http://intellectualcramps.blogspot.com/2009/02/xpath-20-evaluation-with-psychopath.html
sorry, the some is the answer to B, (In reply to comment #22) > a > > (In reply to comment #21) > > Jin, IP needs a few standard simple questions answered: > > > > Contribution Confirmations: > > (a) wrote 100% of the code; almost 100% codes. A little codes from Dave Carver's blog[1]. > > (b) that they have the right to contribute the code to > > Eclipse; and A little codes from Dave Carver's blog[1](in fact, it is basically same to the cvs tests codes) or comments here:) I made some modifications to fit for the actual test codes. Dave Carver is just the committer of Eclipse. > > (c) the file header contains the EPL License header. > > > > > > (c) I took care of when I submitted the code for review. Just respond here to > > option a and b. Thanks. > > > [1] http://intellectualcramps.blogspot.com/2009/02/xpath-20-evaluation-with-psychopath.html
How about the above reply?:) (In reply to comment #23) > I actually need an answer to A and B as well. >
(In reply to comment #26) > How about the above reply?:) > > (In reply to comment #23) > > I actually need an answer to A and B as well. > > > Looks good. :) Thanks.
Jin, IP sent me the following follow up to Question B. > Could you please have the contributor answer question (b) more concisely. We > need him to expressly indicate he has the rights to donate the content to > Eclipse (the content he authored). I'm sure the answer is yes but it seems the > answer got lost a bit in the communication. If you prefer, this can be handled > in email... So, please be as clear as possible that you have rights to submit the code (i.e. no employer permission required, if you are a student and not working for anybody, etc.). They just need it clearly stated so that if questions come up they don't get bitten.
(In reply to comment #21) > Jin, IP needs a few standard simple questions answered: > > > Contribution Confirmations: > > (a) wrote 100% of the code; almost 100% codes. A little codes from Dave Carver's blog[1]. > > (b) that they have the right to contribute the code to > > Eclipse; and A little codes from Dave Carver's blog[1](in fact, it is basically same to the cvs tests codes) or comments here:) I made some modifications to fit for the actual test codes. Dave Carver is just the committer of Eclipse. I'm a phd sutudent now at Chinese Academy of Sciences, China. In fact, I have added my info into the eclipse-gsoc2009[2]. I have the right to submit the codes to Eclipse. All the codes are written by myself from scratch. No body I have asked. The little codes from Dave Carver's blog, is just for convenience. In fact, they are the API usage of PsychoPath(now it is just the our offical processor in cvs head). I'm very clear about the IP principle of Eclipse. I'd like to stick with it in the future. If I write any codes to contribute to Eclipse, I keep codes unspotted from any other implementation. In fact, if I want to write any codes to contribute to Eclipse, I will keep me away from any other implementation. I learn this principle from Apache Harmony, but I think it still suits for Eclipse:) > > (c) the file header contains the EPL License header. > > > > > > (c) I took care of when I submitted the code for review. Just respond here to > > option a and b. Thanks. > > > [1] http://intellectualcramps.blogspot.com/2009/02/xpath-20-evaluation-with-psychopath.html [2] http://wiki.eclipse.org/Google_Summer_of_Code_2009_Ideas
Jin I've added your test suite code to CVS now, as I was given preliminary approval under Parallel IP. I did go ahead and put it in its own plugin. org.eclipse.wst.xml.xpath2.processor.w3c.tests For any additional patches for the test suite plugin tests themselves, please use that package. Patches for psychoPath to fix various tests, will need separate bugs and patches for those bugs attached to them as appropriate.
Good news:)Good luck:) (In reply to comment #30) > Jin I've added your test suite code to CVS now, as I was given preliminary > approval under Parallel IP. I did go ahead and put it in its own plugin. > > org.eclipse.wst.xml.xpath2.processor.w3c.tests > > For any additional patches for the test suite plugin tests themselves, please > use that package. Patches for psychoPath to fix various tests, will need > separate bugs and patches for those bugs attached to them as appropriate. >
I've related several bug fixes to this particular issue. All together about 110 unit tests now pass that were failing. Still several thousand to go.
Created attachment 134141 [details] Refactored Test Suite Creation The following attachment is a refactoring based on some initial work done by Jin. This refactors the code to a Junit 3 test, and individual test classes for each Test Group. Within there, the individual test cases are setup. This is attached here so that it can be re-added to CVS once the release candidate freeze is done.
Created attachment 134165 [details] Additional Unit Tests from Test Suite. This patch includes all tests up to op:subtract-dateTimes-yielding-dayTimeDuration. Need to update the generator script so that it takes into account when a test is a Text, Fragment, or XML comparison. Currently it only handles correctly text comparisons. Adding fragment comparision would greatly reduce the number of failed tests.
Patch includes 1149 tests of 8000.
(In reply to comment #35) > Patch includes 1149 tests of 8000. Hi Dave, I wish to run the W3C XPath 2.0 test suite on PsychoPath processor on my PC. Can I do this now, or I need to wait for it be put in CVS sometime later. If I can run this test suite now, could you please share some instructions to do so. Also, is the work on this test suite complete from your side, or some work is still left? Regards, Mukul
You can start to use this test suite now, and I'm continuing to add additional tests from the suite. There are plenty of bugs that have been identified by the test suite to keep somebody busy for a bit. Most of the code you need is in cvs in the org.eclipse.wst.xml.xpath2.processor.tests plugin. Just check that code out and then apply the last patch to that plugin. To run it, in the org.eclipse.wst.xml.xpath2.processor.testsuite package is a test suite class AllW3CXPath20Tests, run that class as a Junit Plugin-Test and you'll start to get the report. When a test fails it will show what Input test file it failed on, and what the expected results were and the actual results. There is a XSLT called createJunit3Tests.xsl that is used to generate the tests based on the test groups. This xslt needs to be enhanced so that it handles XML Fragment comparisons. The NodeTests are failing due to those being XML Fragments or XML comparisons and not Text comparisons. I would suggest that looking into the Numerice operation (op:) items as a good starting point. There are several expected cases where the result is returning 0.0 instead of 0 as the result. It is causing the tests to fail. I'll continue adding the test suite creation patches here until 3.2M1 opens up for development, at that time I'll add the last patch to CVS.
(In reply to comment #37) > Most of the code you need is in cvs in the > org.eclipse.wst.xml.xpath2.processor.tests plugin. Just check that code out > and then apply the last patch to that plugin. > I would suggest that looking into the Numerice operation (op:) items as a good > starting point. There are several expected cases where the result is returning > 0.0 instead of 0 as the result. It is causing the tests to fail. Hi Dave, Thanks for the information. As suggested by you, I checked out the tests plugin files and ran the tests using the JUnit plugin. I get the results. Some tests fail and some pass. I would try to solve the failing tests, and try to improve the Psychopath code. I hope I would be able to submit some patches soon, from the results of the test suite. I noticed, that many of the tests use .xq file as input. The extension .xq look XQuery like, but it's ok if it serves our purpose. I guess, you reused some of the available .xq files. It seems, you have done quite a bit of work setting up this test suite. Many thanks for the same. Regards, Mukul
(In reply to comment #38) > I noticed, that many of the tests use .xq file as input. The extension .xq look > XQuery like, but it's ok if it serves our purpose. I guess, you reused some of > the available .xq files. Yes we reuse the .xq files, as the XQTSCatalog.xml has an attribute on the test-cases called is-XPath which signals whether the test is also an XPath 2.0 test. It's about 90% reliable but not 100% reliable. Some tests it flags as XPath 2.0 are really XQuery tests only. So we have to weed them out. The XQ filename can be used to match up with the appropriate test case scenario in the XQTSCatalog.xml file. The best way to view that is to open it in FireFox or another XSLT 1.0 aware browser and let it transform it. Or you can run the appropriate catalog.xsl stylesheet to generate the XHTML output, and browse it that way. > > It seems, you have done quite a bit of work setting up this test suite. Many > thanks for the same. I'll be working on expanding the suite further tonight. If we fix a few basic issues, we'll get a lot of tests passing. Most of very minor issues and interpretation of the spec. As we get the Tests in the test classes passing I'll add them to the build so that they are verified when we automatically build the processor.
Created attachment 134192 [details] Additional 700 Tests. This brings the total to roughly 1800 tests.
(In reply to comment #40) > Created an attachment (id=134192) [details] > Additional 700 Tests. > > This brings the total to roughly 1800 tests. I have applied this patch. It looks good, and runs well. Would try to work on some of the non-compliant items. Regards, Mukul
All of the latest patches are now applied to Head.
With the latest batch of patches applied from Mukul, PsychoPath now passes an additional 400 tests from the W3C test suite we have implemented. Most of the Int and Long tests now pass, where we are still failing has to do with how Double and Float are returning their values. Primarily these are returning results as 0.0 when the tests are expecting the results to but just 0. Need to check the XPath data spec for these data types and see what the value should be when the result is a whole number and doesn't have any decimal places after it. All code is checked in, and the PsychoPath processor version has been incremented to version 1.1.
Updated AbstractPsychoPathTest class so that input-context and input-context1 variables are declared for the Dynamic Context. This allows several Axes tests that were failing to start passing. Currently out of the 1835/8000+ test implemented 729 are failing. This change allows an additional 150 tests to pass.
Hi Dave, (In reply to comment #44) > Updated AbstractPsychoPathTest class so that input-context and input-context1 > variables are declared for the Dynamic Context. This allows several Axes tests > that were failing to start passing. Currently out of the 1835/8000+ test > implemented 729 are failing. This change allows an additional 150 tests to > pass. I have made my workspace in sync with the server. Everything works fine. When I run all the tests available, I get following status as of now: 1149 tests, fail 505, errors 23 I believe, a number of failing tests are due to the fact, that they contain XQuery fragments. Therefore, the real success rate is quite better than 505 fails. If we can somehow weed out the XQuery test cases, we would get the true report. Hoping this could be done in a near future. I am starting to work on few other things in PsychoPath to further improve the compliance. I would post the patches as I have them. Regards, Mukul
(In reply to comment #44) > Updated AbstractPsychoPathTest class so that input-context and input-context1 > variables are declared for the Dynamic Context. This allows several Axes tests > that were failing to start passing. Currently out of the 1835/8000+ test > implemented 729 are failing. This change allows an additional 150 tests to > pass. I just now updated the test suite. I could see, now we have following status: total tests: 1849 failing: 381 errors: 23 looks good.. I am working on few more items, to improve the success rate further. Regards, Mukul
Added additional tests to the test suite. Current stats are: 2032 Tests 229 Failures 7 Errors
Updated the test suite so that AbstractPsychoPathTest now extends the XMLTestCase from the XMLUnit 1.2 framework. This was approved with CQ3394, and will allow easier comparison of the XML result files. In particular attribute order and formatting of the serialized XML will no longer cause test failures. These tests will use the assertXMLEquals method to determine if the XML is the same. XMLUnit adds a differencing engine if we need it as well. The necessary JAR has been included in the lib directory for now, and XMLUnit 1.2 is required. This jar will eventually be hosted as an Orbit bundle.
Additional tests have been added. Current results are: 2546 Tests 450 Failures 13 Errors In addition, code coverage for the tests on PsychoPath is about 55.7%.
Added div, idiv, mod to the list of terminals that need to be handled specially in case they are also found as element names. This was detected by some failing tests the NameFuncTest suite.
Corrected implementation of UntypedAtomic, and renamed to XSUntypedAtomic. Also added it to the XML Schema namespace. Caused an additional 200 tests to pass.
implemented the function, prefix-from-QName. I have checked in the code for this to HEAD. This passes about 14 additional test cases.
fixed SeqUnorderedFuncTests. These are an implementation dependent order, so may not necessarily match the expected results. Need to check to see that the actual values separated by a space are there. Additional 23 tests pass.
down to 1370 failing tests with 75.4% code coverage.
did improvements to fn:codepoints-to-string implementation. for an invalid codepoint (like, 10000000 as in one of the W3C tests), the test must result in an error (because we have an illegal function argument, like 10000000), instead of failing the test. This has been corrected now, with this improvement. This improvement uses, the JDK 1.5 APIs. We could think later, how to port this to Java 1.4, if necessary. I have checked in the code for this, to HEAD. Regards, Mukul
With the implementation of function, fn:dateTime (bug, 281822) we have 22 more tests passing. Regards, Mukul
I have made an improvement to xs:string type implementation. The affected class is, XSString.java. Earlier the constructor function was not allowing nodes from context item to be used, which was causing class cast exception to be thrown (only atomic values were allowed, previously). I have made this improvement, and have checked in the code to HEAD. This improves the test run slightly, and causes few "error" tests to reduce. I can now see a success rate for W3C test suite to 82%. The real success of the W3C test suite we have, is probably higher, because many tests *need* to fail as part of the test case. Regards, Mukul
cleaned up some more tests, and implemented fn:default-collation. Now down to 1335 tests left to pass. 84% pass ratio.
I improved comparison of xs:string with other XDM types. Now remaining tests of core.QNameEQTest pass. This improvement makes possible comparison like, xs:string("example.org/") eq xs:anyURI("example.org/") xs:string("example.org/") ne xs:anyURI("example.org/") to work. Earlier these comparisons were failing. I have checked in the code to HEAD. Regards, Mukul
One of the tests in core.PrefixFromQNameTest was failing. The expression, fn:count(fn:prefix-from-QName(xs:QName("name"))) was incorrectly producing result, 1. The correct result of this expression is 0 (the prefix-from-QName function should return an empty sequence, because there is no prefix. and therefore, count should return 0). This has been corrected now. I have checked in the code for this to HEAD. Regards, Mukul
few tests in core.GenCompNETest, like below: 10000 != (10000,50000) were giving error due to a bug in sequence iteration. I have solved this, and it improves the "error" results in core.GenCompNETest. It seems this fix improves the "error" cases somewhat in the overall test suite. Overall, the "error" cases improve by 12 test cases. As a consequence, overall test suite conformance seems to have improved. I have checked in the code for this to HEAD. Regards, Mukul
Hello, I am planning to use my employer email address (mukul.gandhi@in.ibm.com) for all future Eclipse communications. Kindly send communications to the new address please. Regards, Mukul
Hi Dave, I took a latest update of the PsychoPath sources, and the test suite updates, and following are the latest test data with me, tests: 8137 errors: 47 failures: 520 This shows a success rate of about 93%. I think, this is great. I am keen to know, what is the code coverage of test suite at the moment? I think, last time you shared code coverage data, it was about 76%. Has it improved since last run, which I believe must be, because I can see quite a new tests are being added to the test suite. Regards, Mukul
(In reply to comment #63) > > I am keen to know, what is the code coverage of test suite at the moment? > I think, last time you shared code coverage data, it was about 76%. Has it > improved since last run, which I believe must be, because I can see quite a new > tests are being added to the test suite. Actually, since we fixed a lot of errors, the code coverage actually has dropped. 73.4% last time I checked. This is due to the test suite not testing the error paths for the various functions so some code is not now being executed that was being executed by the test suite before.
(In reply to comment #64) > Actually, since we fixed a lot of errors, the code coverage actually has > dropped. > 73.4% last time I checked. This is due to the test suite not testing the > error paths for the various functions so some code is not now being executed > that was being executed by the test suite before. Thanks, Dave for the update. Regards, Mukul
A few more tests have been fixed. Total tests failing now are down to 514, with 41 errors. The core tests are down to about 4 errors and 5 failures. the majority of the rest of the tests that need to pass are left in the Function test suite.
Just fixed an error in ElementType.java, where an exception was occurring in typed_value() computation. I discovered this bug, while testing one of the examples, for XML Schema 1.1 assertions, using PsychoPath. For following XML Schema fragment: <xs:element name="mode" type="modeType" /> <xs:simpleType name="modeType"> <xs:restriction base="xs:string"> <xs:enumeration value="air"/> <xs:enumeration value="water"/> <xs:enumeration value="ground"/> </xs:restriction> </xs:simpleType> PsychoPath was not able to compute, typed value of element node, "mode" and was giving an exception. This fix, solves this problem. I have committed the code for this to HEAD. Thanks, to Roger Costello who provided these samples on xml-dev list, which I was trying to test assertions and PsychoPath. PS: I am seeing, that WTP 3.2 M1 build activity is going on. Is it recommended, not to commit the code to HEAD for some time? Regards, Mukul
(In reply to comment #67) > PS: I am seeing, that WTP 3.2 M1 build activity is going on. Is it recommended, > not to commit the code to HEAD for some time? > No we are fine, the build activity that is happening won't affect our builds.
Mukul, I think your fix might have caused a regression for apprx 250 test cases. If the DOM has no type info, typeDef will be null at the line you changed. Try: if (typeDef != null && XSCtrLibrary.XML_SCHEMA_NS.equals(typeDef.getNamespace())) { ^^^^^^^^^^^^^^
(In reply to comment #69) > Mukul, I think your fix might have caused a regression for apprx 250 test > cases. > > If the DOM has no type info, typeDef will be null at the line you changed. Try: > > if (typeDef != null && > XSCtrLibrary.XML_SCHEMA_NS.equals(typeDef.getNamespace())) { > ^^^^^^^^^^^^^^ > I've applied the suggested change and all is normal again, my system currently shows from head 339 test failures, 2 errors. Mukul, can you add a unit test for this particular patch to verify that we have it working right. Also this shows why it's important to make sure that we as committers run all the tests, even if it is a one line change, and have a corresponding test to go with it.
(In reply to comment #70) > > If the DOM has no type info, typeDef will be null at the line you changed. Try: > > > > if (typeDef != null && > > XSCtrLibrary.XML_SCHEMA_NS.equals(typeDef.getNamespace())) { > > ^^^^^^^^^^^^^^ > > > I've applied the suggested change and all is normal again, my system currently > shows from head 339 test failures, 2 errors. > Mukul, can you add a unit test for this particular patch to verify that we have > it working right. Also this shows why it's important to make sure that we as > committers run all the tests, even if it is a one line change, and have a > corresponding test to go with it. Hi Dave, Sorry about this. I actually recognized this error, and was about to fix it, when you sent this note, and made a fix. I tested, your fix and it's right. As suggested, I have added a test case for this fix(method, "testElementTypedValue" in file TestBugs.java), with corresponding XML and XSD files. I have committed the test case along with XML and XSD files, to HEAD. Regards, Mukul
(In reply to comment #69) > Mukul, I think your fix might have caused a regression for apprx 250 test > cases. > If the DOM has no type info, typeDef will be null at the line you changed. Try: > if (typeDef != null && > XSCtrLibrary.XML_SCHEMA_NS.equals(typeDef.getNamespace())) { > ^^^^^^^^^^^^^^ The same fix, needs to be applied to AttrType.java. I have done this, and have checked in the code. Regards, Mukul
Fixed a test issue with SeqInsertBeforeFuncTest where we needed to check the alternate result instead of the main result. The processor outputs a -0 instead of 0, both cases are valid according to the spec.
Fixed issue with FnLast and no context_item defined, correctly tosses XPDY0002 now. Fixed 1 test. down to 333 failues 2 errors.
Found a bug in the W3C Test Suite for op-date-less-than-2 test. It is expecting true when it should be expecting false. I have corrected the test in our suite, and filed a bug with the W3C test suite maintainers. http://www.w3.org/Bugs/Public/show_bug.cgi?id=7530
down to 199 failing tests now. I've checked in some more changes to handle datetime, date, and duration calculations. This brings us to 97.6% compliance.
fixed minutes-from-duration function to expect the correct xs:duration argument.
just fixed a problem, to allow PsychoPath to use xPathDefaultNamespace for evaluation. I have checked in the code for this, with a corresponding test case. The W3C XPath 2.0 test suite, doesn't test this facility, and none of it's tests are affected by this change. This change was required in PsychoPath, to allow Xerces-J to implement the attribute "xpathDefaultNamespace", on XML Schema 1.1 xs:assert instruction. Regards, Mukul
I'm committing what appears like a massive step back for test suite compliance in a few minutes. Apprx. 500 test expectations were not set correctly in the W3C test suites -- where errors were expected, and as a fall back the test support function expected a syntax error (XPST0003) , which is incorrect in all cases I've seen. Most of the expectations were really type errors like XPTY0004 and FORG0006, and correcting the actual error codes will take some time...
(In reply to comment #79) > I'm committing what appears like a massive step back for test suite compliance > in a few minutes. Apprx. 500 test expectations were not set correctly in the > W3C test suites -- where errors were expected, and as a fall back the test > support function expected a syntax error (XPST0003) , which is incorrect in all > cases I've seen. > > Most of the expectations were really type errors like XPTY0004 and FORG0006, > and correcting the actual error codes will take some time... Most of the 365 errors are in the SeqExprCastTests unit tests. The vast majority of these have to do with handling untypedAtomic and string when going to numerics. I worked on this in the past, so have a pretty good idea what is happening. In many cases fixing one failing tests knocks out about 35 to 40 other tests that were failing. More info here: http://www.w3.org/TR/xpath-functions/#casting-from-strings I should have some fixes put in later tonight. Won't address all of them but should start to whittle this down. I won't release the code to the WTP build until I've got these fixed.
I checked in several fixes to the built in data types. I've reduced the number of SeqExprcastTests that were failing with the recent changes from 365 down to 45. I'll clean up the rest either tomorrow or later in the week. Working on this just makes me want to make that whole Casting/Construction section into a couple of nice data type factories. However, I'll wait on that until I get everything passing again.
I integrated my latest changes with Jesper's and we are down to 7 of the tests that were originally passing that are now failing. I haven't run the full suite yet, just the ones that are included in the builds. Most of these seem to be numeric related that are left.
Status update. We are currently down to about 12 tests from the Core that still need to pass. There are a handful of bugs that need to be completed before full compliance, but we are officially 99.5% of the way there, which is pretty good considering where we started several months ago.
Just fixed three tests in NodeNumberFuncTest: test_fn_numberflt1args_1 test_fn_numberflt1args_3 test_fn_numbernpi1args_2 And so it was green and I added it to AllW3CFunctionTests. This should get me a few points in the Hudson CI Game... Unless I broke something in the process.
(In reply to comment #84) > Just fixed three tests in NodeNumberFuncTest: > > test_fn_numberflt1args_1 > test_fn_numberflt1args_3 > test_fn_numbernpi1args_2 > > And so it was green and I added it to AllW3CFunctionTests. This should get me a > few points in the Hudson CI Game... Unless I broke something in the process. Yep, looks like you got a few build points. :) I'm going to move the remaining items in the backlog that were targeted to 3.2M3 to 3.2M4. Due to the general ramp down for this milestone. Keep an eye on when 3.2M4 opens up.
This is now fixed. All core tests now pass. There are 20 tests that still need work for Static Type checking (optional) and XSNotation.
resolving.
