We welcome submissions of test cases from individuals and organizations whether or not they already participating in a relevant Working Group.
We particularly encourage submissions that follow the existing format of the test suite, and that provide coverage of areas that are not well covered already. Where test material is available in other formats, then we would still like to know about it, and may be prepared to undertake the conversion. In all cases, the Working Groups will, at their sole discretion, determine what tests are incorporated in the test suite.
The relevant Working Groups here means the XQuery Working Group in the case of XQuery tests, and the XQuery and XSLT Working Groups (acting jointly) in the case of XPath tests.
If you would like to submit test cases, then you must:
Contributors to the test suite agree to make material available under the terms described in this policy, and take responsibility for ensuring that they are entitled to make the material available. In the event that anyone encounters material which should not have been submitted and published, however, they should immediately let the Working Group know, so that it can be withdrawn.
Our source files are available at http://dev.w3.org/cvsweb/2011/QT3-test-suite/. This page provides the ability to view individual files.
If you want to get the latest version of the test suite, but don't intend to make any contributions,
you can use anonymous CVS to check the whole suite out into your local filestore. Use the
CVS repository name :pserver:anonymous@dev.w3.org:/sources/public, password anonymous,
and the module name 2011/QT3-test-suite.
To work on the test suite you will need to get read-write access to the W3C CVS repository. Official information on how to do this is at https://www.w3.org/Project/CVSdoc/; unofficial information from people who have been through the experience and emerged from it triumphant is here (Tantek Çelik) and here (Andrew Eisenberg). Briefly the steps are:
:ext:user@dev.w3.org:/sources/public (where 'user' is your username),
module (or directory) 2011/QT3-test-suite. Note that many CVS clients invite you to ask the server for
a list of available modules, but the W3C server does not respond to this request: you need to
know the name of the module you are looking for.Two graphical CVS clients I (Michael Kay) have found useful are: for Windows, TortoiseCVS, and for Mac, SmartCVS. Use the command line if you're feeling macho; use Emacs if you are one of the faithful. Other tools that you have on your desktop may also have a CVS client built in, though the number is declining as CVS is now considered obsolescent.
Another guide you may find useful is Andrew Eisenberg's at https://lists.w3.org/Archives/Member/w3c-xml-query-wg/2006Sep/0032.html
If you are new to CVS, here are some explanations of key terms:
At the current stage of development, the change control process for tests is lightweight.
If at all possible, test your tests against at least one implementation before committing them. Experience shows that untested tests have a 50% chance of being wrong.
Where possible, make your CVS commits fine-grained, and mention bug numbers and/or test names in the commit message.
This process will become more formal as the tests and the spec approach completion.
Unless you have been asked specifically to write tests for an area where tests are currently missing, you should not submit a test until you have established that there is at least one implementation capable of running the test successfully. You should also take care to ensure that any dependencies on features of that implementation are properly documented in the metadata. Tests of optional features in the specification are welcome, but tests of vendor extensions are not.
Tests should generally be small and self-contained. There are a few "system" tests designed to show that larger-scale queries can be executed interoperably, but these need to be very carefully designed to ensure that the results are easily compared. Tests that produce a large result file and expect it to be exactly equal to a supplied result file are best avoided, since it can be very hard to isolate differences in the results and work back to their causes.
Each test should have a description that summarizes the intent of the test: what is it designed to prove?
Try to allocate any new test to the most appropriate existing test set, and to respect the naming conventions used in that part of the test suite. Try to reuse existing test environments (source documents and schemas) if possible, rather than adding new source documents. Try to use test assertions that reflect the purpose of the test.
When you make changes to the test catalog, you should check before committing the change that the catalog is valid
against its schema, and conforms to other required consistency checks. The easiest way of checking this is to run
the test set named CatalogCheck, which performs these checks automatically.
If you have CVS write access to the test suite then you are trusted to exercise responsibility in modifying existing tests.
Three levels of changes to tests can be identified:
You SHOULD NOT add an alternative expected result to a test purely because your own implementation delivers that result, however strongly you believe that the alternative result is legitimate.
The primary documentation of the test suite (alongside this document) is the schema for the catalog and test-set files, which is found at catalog-schema.xsd. If viewed directly in an XSLT-capable browser, the schema is rendered automatically into HTML. In case this mechanism does not work, a pre-rendered version of the HTML is at catalog-schema.html. (In many browsers the pre-rendered form is more readable).
Copyright © 1994-2013 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.