Writing and Submitting tests for the QT3 Test Suite


We welcome submissions of test cases from individuals and organizations whether or not they already participating in a relevant Working Group.

We particularly encourage submissions that follow the existing format of the test suite, and that provide coverage of areas that are not well covered already. Where test material is available in other formats, then we would still like to know about it, and may be prepared to undertake the conversion. In all cases, the Working Groups will, at their sole discretion, determine what tests are incorporated in the test suite.

The relevant Working Groups here means the XQuery Working Group in the case of XQuery tests, and the XQuery and XSLT Working Groups (acting jointly) in the case of XPath tests.

If you would like to submit test cases, then you must:

Contributors to the test suite agree to make material available under the terms described in this policy, and take responsibility for ensuring that they are entitled to make the material available. In the event that anyone encounters material which should not have been submitted and published, however, they should immediately let the Working Group know, so that it can be withdrawn.

The test repository

Our source files are available at http://dev.w3.org/cvsweb/2011/QT3-test-suite/. This page provides the ability to view individual files.

If you want to get the latest version of the test suite, but don't intend to make any contributions, you can use anonymous CVS to check the whole suite out into your local filestore. Use the CVS repository name :pserver:anonymous@dev.w3.org:/sources/public, password anonymous, and the module name 2011/QT3-test-suite.

To work on the test suite you will need to get read-write access to the W3C CVS repository. Official information on how to do this is at https://www.w3.org/Project/CVSdoc/; unofficial information from people who have been through the experience and emerged from it triumphant is here (Tantek Çelik) and here (Andrew Eisenberg). Briefly the steps are:

  1. Get a W3C username (ask your staff contact)
  2. Create a private/public key pair and lodge the public half with W3C; wait until you get confirmation that it has been installed.
  3. Install a CVS client and configure it to be aware of your private/public key
  4. Checkout the repository at :ext:user@dev.w3.org:/sources/public (where 'user' is your username), module (or directory) 2011/QT3-test-suite. Note that many CVS clients invite you to ask the server for a list of available modules, but the W3C server does not respond to this request: you need to know the name of the module you are looking for.

Two graphical CVS clients I (Michael Kay) have found useful are: for Windows, TortoiseCVS, and for Mac, SmartCVS. Use the command line if you're feeling macho; use Emacs if you are one of the faithful. Other tools that you have on your desktop may also have a CVS client built in, though the number is declining as CVS is now considered obsolescent.

Another guide you may find useful is Andrew Eisenberg's at https://lists.w3.org/Archives/Member/w3c-xml-query-wg/2006Sep/0032.html

If you are new to CVS, here are some explanations of key terms:

Development Process

At the current stage of development, the change control process for tests is lightweight.

If at all possible, test your tests against at least one implementation before committing them. Experience shows that untested tests have a 50% chance of being wrong.

Where possible, make your CVS commits fine-grained, and mention bug numbers and/or test names in the commit message.

This process will become more formal as the tests and the spec approach completion.

Submitting new tests

Unless you have been asked specifically to write tests for an area where tests are currently missing, you should not submit a test until you have established that there is at least one implementation capable of running the test successfully. You should also take care to ensure that any dependencies on features of that implementation are properly documented in the metadata. Tests of optional features in the specification are welcome, but tests of vendor extensions are not.

Tests should generally be small and self-contained. There are a few "system" tests designed to show that larger-scale queries can be executed interoperably, but these need to be very carefully designed to ensure that the results are easily compared. Tests that produce a large result file and expect it to be exactly equal to a supplied result file are best avoided, since it can be very hard to isolate differences in the results and work back to their causes.

Each test should have a description that summarizes the intent of the test: what is it designed to prove?

Try to allocate any new test to the most appropriate existing test set, and to respect the naming conventions used in that part of the test suite. Try to reuse existing test environments (source documents and schemas) if possible, rather than adding new source documents. Try to use test assertions that reflect the purpose of the test.

When you make changes to the test catalog, you should check before committing the change that the catalog is valid against its schema, and conforms to other required consistency checks. The easiest way of checking this is to run the test set named CatalogCheck, which performs these checks automatically.

Modifying existing tests

If you have CVS write access to the test suite then you are trusted to exercise responsibility in modifying existing tests.

Three levels of changes to tests can be identified:

  1. Cosmetic changes, for example tidying the layout or improving the description. This is allowed without formality.
  2. Metadata changes, for example noting that a test is dependent on support for schema-awareness. Such changes do not require a bug report, but should be logged: you should add a "modified by" element indicating your name, the date of the change, and the reason. Splitting a test into multiple tests with different dependencies also falls into this category.
  3. Substantive changes, that is changes to the test query or to its expected results (including adding alternative results). Such changes should not be made without raising a bug report and giving the Working Group the opportunity to dispute the change. If the error is obvious and uncontroversial (for example a misspelt variable or function name), and in particular if the test was only recently submitted, then you may correct the error and close the bug without further consultation. Normally, however, a bug report should not be closed until there is evidence of consensus (which might simply be the concurrence of one other person and the absence of dissent.) In all cases, when a substantive change is made, the test should be updated with a "modified by" element that cross-references the Bugzilla entry.

You SHOULD NOT add an alternative expected result to a test purely because your own implementation delivers that result, however strongly you believe that the alternative result is legitimate.


The primary documentation of the test suite (alongside this document) is the schema for the catalog and test-set files, which is found at catalog-schema.xsd. If viewed directly in an XSLT-capable browser, the schema is rendered automatically into HTML. In case this mechanism does not work, a pre-rendered version of the HTML is at catalog-schema.html. (In many browsers the pre-rendered form is more readable).

Webmaster · Last modified: $Date: 2012-01-14 19:02:21 $ by $Author: mkay $

Copyright © 1994-2013 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.