This document provides information to implementers who wish to run the QT3 Test Suite on their XQuery or XPath implementation. It includes guidelines how test cases can be customized in order to run on an implementation, and describes the process of evaluating the results.
Guidelines for submitting results to the XML Query Working Group can be found in Reporting Results.
Our source files are available at http://dev.w3.org/cvsweb/2011/QT3-test-suite/. This page provides the ability to view individual files.
If you want to get the latest version of the test suite, but don't intend to make any contributions,
you can use anonymous CVS to check the whole suite out into your local filestore. Use the
CVS repository name :pserver:anonymous@dev.w3.org:/sources/public, password anonymous,
and the module name 2011/QT3-test-suite.
The QT3 test suite is made available under the terms of the W3C Licenses described at: http://www.w3.org/Consortium/Legal/2008/04-testsuite-copyright
Where these licenses prohibit alteration of test material, such prohibition does not apply to alterations explicitly permitted in this documentation.
Contributors to the test suite can be assumed to have agreed to make their contributions available under these terms and conditions, and take responsibility for ensuring that they are entitled to make the material available. In the event that anyone encounters material which should not have been submitted and published, however, they should immediately let the Working Group know, so that it can be withdrawn.
The primary documentation of the test suite (alongside this document) is the schema for the catalog and test-set files, which is found at catalog-schema.xsd. If viewed directly in an XSLT-capable browser, the schema is rendered automatically into HTML. In case this mechanism does not work, a pre-rendered version of the HTML is at catalog-schema.html. (In many browsers the pre-rendered form is more readable).
The main purpose of the test suite is to enable XPath 3.0 and XQuery 3.0 implementations to be tested. However, the test suite also contains a great many tests that can be used to test XPath 2.0 and XQuery 1.0 implementations, and these are marked as such in the test metadata.
The primary objective of the Working Group in developing the tests is to gather the information needed to progress the specifications through the W3C process: that is, to demonstrate that the specifications are implementable and that implementations are interoperable. However, many contributors of tests are implementors who are also interested in aspects of the quality of their implementations that go beyond this narrow objective, and this is reflected in nature of the tests.
As well as XPath and XQuery, the tests also cover Serialization, though coverage in this area is less complete.
Implementers are expected to write their own test harness that performs the following tasks:
The drivers directory contains some test drivers that have been
contributed for particular implementations; authors of new test drivers might like
to examine these and use them as a starting point for writing their own test drivers.
The code in this directory should be regarded as being "for information only"; it
is not intended to serve any purpose other than illustration, and the code may
not work with all releases of the implementations in question, let alone with other implementations.
Published results submitted
by implementors were not necessarily obtained using the test drivers found in this
directory. Further contributions to this directory are welcome.
Implementors are encouraged to submit results to W3C, in order to achieve the purpose for which the tests were designed. For information on reporting results, see Reporting results.
If you believe that some of the queries or their expected results are incorrect, then we encourage you to enter bug reports in the public instance of Bugzilla at W3C, http://www.w3.org/Bugs/Public/. To enter a bug report you will need a Bugzilla signon (email address and password). You can create this singon by following the Open a new Bugzilla account link on the Bugzilla Main Page.
General information on Bugzilla, an open-source bug tracking system, is available at http://www.bugzilla.org/. The standard documentation for the version currently being run on the W3C server is available at http://www.bugzilla.org/docs/2.20/html/.
To enter a bug report in Bugzilla you will need to log in, choose the action "New", and then choose the "XQuery / XPath / XSLT" product and the "XQuery/XPath Test Suite" component. Please be specific about the problem that you are reporting. Clearly identify the test case or test cases that are problematic. If you believe that the expected result is wrong, then tell us what result you believe is correct. If you believe that another result for a test case is justified, then please tell us what it is. In both cases, please cite portions of one or more of the XQuery specifications that support your position.
All bug reports raised against the test suite are initially allocated to the test coordinator. The test coordinator will then assign them to someone for resolution. Sometimes this will be the original author of the test, sometimes a person in the Working Group who has recognized expertise in the relevant area. Experts other than the assignee are welcome to fix the bug if they are confident that the fix will be uncontroversial.
Where a bug report raises difficult questions of interpretation of the specification, the test coordinator will assign the bug to the chair of the Working Group, who will ensure that the question is discussed by the Working Group as a whole. In some cases the bug report will be "promoted" to a bug against the specification itself.
Many of the tests have dependencies, which means that not every implementation will be able to run every test. The design intent is that all dependencies that might affect the ability of an implementation to run a particular test are documented in the test metadata. However, this principle is not followed slavishly; for example, the specification allows implementation-defined limits on the length of strings, but most tests assume that all implementations will be able to handle strings up to a few thousand characters, without documenting this as a dependency. Implementors encountering a test that makes unwarranted assumptions are strongly encouraged to challenge the test, as described below, so that the dependencies can be properly documented.
One of the major dependencies that permeates all tests is which specification(s) the
test is designed to exercise. This will be some combination of one or more of XQuery 1.0,
XQuery 3.0, XPath 2.0, and XPath 3.0. These are documented using <spec type="lang" value="V"/>
where V is typically a space-separated sequence of "XQ10", "XQ30+", "XQ10+", "XP20", "XP30+",
or "XP20+", where "XQ" means XQuery, "XP" means XPath, and the "+" should be read as "or later".
(Tests labelled as applicable to 3.0 are assumed to be applicable to later releases, until
proved otherwise.)
Although a product may support more than one of these languages/versions, a given run of the test suite should assume that the product under test only supports one language/version, and should only attempt to execute the tests applicable to that language/version. Each test result submission is therefore concerned only with one of these specifications; products that implement more than one specification are encouraged to submit separate results for each one.
The semantics of XPath in some cases depend on the host language. The test suite makes the following assumptions about the host language:
The host language does not prohibit use of features such as the namespace axis or XPath 1.0 backwards compatibility (if it does so, then tests with such dependencies cannot be run)
The default values for components of the static and dynamic context are as described in Appendix C of the XPath specification.
The host language does not normalize line breaks on input (since the XPath expression is contained in an XML document, the XML parser will have already normalized line breaks, and to do so again would drop characters such as CR which need to be retained for the test to work correctly.)
Optional features in the specification fall into two categories: named features listed
explicitly in the Conformance section of the specification (for example the Static Typing feature,
or Schema Awareness); and those that are merely mentioned in the prose of the specification
as areas where results are implementation-defined or -dependent (for example, the set of
languages supported for the format-date function). In the test suite, both kinds
of dependency are handled in the same way. Elements within the <dependencies>
section of the test metadata list features or properties that must be present for the test to work
(or in a few cases, features/properties that must be absent for it to work).
The current list of features and their intended meaning is as follows:
| Feature name | Approx number of tests | Meaning |
|---|---|---|
| schemaValidation | 141 | Requires a query to support the validate expression, and/or to be able to accept source documents that have been validated. |
| schemaImport | 594 | Requires the query to support an import schema declaration. |
| typedData | 1 | Requires the processor to support typed data (but not necessarily schema import or schema validation) |
| higherOrderFunctions | 1578 | Requires support for higher order functions (XPath or XQuery) |
| staticTyping | 84 | Requires "pessimistic" static typing |
| serialization | 34 | Requires the query to support serialization: specifically, to recognize the serialization parameters defined in the query and to perform serialization using the values of those parameters as defined in the Serialization 3.0 specification (this is a stronger requirement than merely providing the XQuery 3.0 serialization feature). |
| infoset-dtd | 34 | Requires source documents to be constructed according to the rules for construction from an Infoset, in particular recognizing ID/IDREF attributes identified as such in a DTD |
| xpath-1.0-compatibility | 5 | For XPath tests, requires the expression to be evaluated in XPath 1.0 compatibility mode. (In general, if this dependency is not present then the test must NOT be run in 1.0 compatibility mode.) |
| namespace-axis | 35 | For XPath tests, requires support for the namespace axis |
| moduleImport | 53 | For XQuery, requires support for import module declaration |
| schema-location-hint | 3 | Requires the "at URI" location hint in import schema to be interpreted as a relative URL. |
| collection-stability | 2 | Requires collections to be stable (implementations are allowed to support non-stable collections) |
| directory-as-collation-uri | 1 | Requires an implementation to recognize a file:/// URL identifying a directory as a collection URI |
| fn-transform-XSLT | 80 | Indicates that the test uses an fn:transform() call which requires an XSLT processor |
| fn-transform-XSLT30 | 10 | Indicates that the test uses an fn:transform() call which requires an XSLT 3.0 processor |
| fn-load-xquery-module | 65 | Indicates that the test uses an fn:load-xquery-module() call which requires an XQuery processor which supports the Module Feature |
| fn-format-integer-CLDR | 10 | Indicates that the test uses an fn:format-integer call which uses CLDR numbering scheme names, e.g. -wo(%spellout-ordinal) |
| non_empty_sequence_collection | 21 | Indicates that the test needs the default collection to be something other than an empty sequence |
| non_unicode_codepoint_collation | 11 | Indicates that the test uses a collation URI other than those which the specification requires every processor to support |
| remote_http | 1 | Indicates that the test uses uses a function such as doc() or unparsed-text() to fetch resources from the web using an HTTP/HTTPS URL. |
| simple-uca-fallback | 50 | Indicates that the test uses a UCA collation URI with the option fallback=yes (or fallback omitted), and the test results assume that the implementation falls back to a "sensible" default if UCA collation is not supported. The only assumptions made about what constitutes a "sensible" default are that the sequences 0-9, a-z, and A-Z are collated as in codepoint collation, and the option "strength=secondary" indicates case-blind matching of a-z with A-Z. The strings to be sorted will not contain any characters outside these ranges, and will not require characters in one of these ranges to be compared with characters in another. For example, a sensible default would sort "abc" before "def"; a fallback collation that sorted "def" before "abc", while legal according to the language specification, would not satisfy this dependency. |
| advanced-uca-fallback | 30 | Indicates that the test uses a UCA collation URI with the option fallback=yes (or fallback omitted), and the test results assume that the implementation has a reasonably high level of support for uca collation semantics even if it is not 100% conformant. Unlike simple-uca-fallback these tests assume support for a wider range of uca collation features: including different collation strengths, different case ordering options and other options, such as numeric collations. |
A complete list of dependencies used in the test suite can be obtained by running the query
dependencies.xq contained in the tools directory.
A test driver can adopt two possible strategies when it finds a test with such dependencies. It can decide whether or not to run the test: if a product does not support an optional feature that the test requires, then the test can be skipped. Alternatively, it can configure the product under test in such a way that the dependency is satisfied; for example, if the test requires support for German as the default host language, it can configure the product under test so that German is the default language.
Sometimes there might be two versions of a test with different dependencies: for example one that expects the schema processor to support XSD 1.0 and one that expects it to support XSD 1.1. The test driver might choose to run either or both, depending on the capabilities of the product under test.
Because a key aim of the Working Group is to establish that the specification is implementable, test result submissions must identify which dependencies are satisfied by the product under test, and which are not. When a test is not run because a dependency is not satisfied, the result submission can either include an entry stating explicitly that the test was not run (perhaps with a reason), or it can simply make no mention of the test, since the analysis will assume "notRun" as the default for tests omitted from the report.
The semantics of XQuery and XPath are defined by formalizing the concept of the "context" which
encapsulates all the external information affecting the outcome of a query. The initialization of a
context for a query or path expression is the role of the <environment> element
in the metadata.
Environments are named to reduce duplication in the metadata when many tests use the same environment; this also gives test drivers the opportunity to save time or memory by building an environment once and then re-using it for many tests. Some tests, however, have a unique environment which is then described as part of the test case.
The static base URI of the query is always the URI of the file containing the query, which will typically be the catalog file.
Where components of the static or context are not explicitly defined in the test's environment, it is generally assumed that they take their default values (as described for example in Appendix C of the XQuery specification). Although the specification does not require implementations to respect these default values, running a test with a different context from that assumed by the test author may result in test failures, so it is the job of the test driver to construct a context which, as far as possible, consists of the default context from the specification overlaid with the values explicitly defined in the test environment.
Also: XQuery and XPath do not explicitly require implementations to provide a way for all values in the static and dynamic context to be set. For example, an implementation that does not allow any collations to be defined (other than the Unicode Codepoint Collation) would be technically conformant; so also would be an implementation in which the default collection is always an empty sequence. There is therefore an implicit dependency, when values are set in the environment, that the implementation allows such values to be set. An implementation that cannot set the context as required should behave in the same way as if a dependency has not been satisfied, ideally reporting the test as "notRun" with a reason.
An implementation that cannot run a test for such a reason may attempt to modify the test so that it can be run; for example a call on collection() might be modified to use an external variable instead. If any tests are modified in this kind of way this must be reported in the test results submission.
The following notes related to particular parts of the test environment.
Source documents (including XML files and text files) accessed by a query
are generally described as part of the environment. They are generally included
within the test suite repository, and their location is given relative to the
relevant catalog file using a file attribute, for example:
<source file="id/InvalidXMLId.xml" uri="id/InvalidXMLId.xml"/>
There are a few exceptions to this rule: there are a handful of tests, for example, that attempt to ensure that files are accessible via HTTP, and these typically refer to documents (hopefully very stable documents) on the W3C web site. There are also a handful of tests that attempt to refer to documents that are by intention unavailable or invalid.
The uri given in the above example indicates that the test may attempt
to retrieve the document using a call on the doc() function with the supplied
URI. This should be interpreted as a relative URI, relative to the base URI of the query
(which in the vast majority of cases is the same as the base URI of the XML file containing
the catalog metadata).
The attribute role="." on a <source> element indicates
that the source document in question is supplied as the initial context item of the query.
There are rather few tests in which the context item is anything other than a document node:
in such cases the <context-item> element of the environment is used.
Schema documents are identified in the environment in a similar way to source documents.
The role attribute indicates whether the schema is imported into the query, or
used for source document validation. Where source documents are to be validated, this should
be indicated by the presence of the attribute validation="strict" or
validation="lax" on the source element.
Collections are identified in the environment with an arbitrary URI, and a list of
source elements for the documents comprising the collection. An implementation that
cannot accept arbitrary collection URIs may modify the source query text to use a different URI in
a form that it can accept; it may also take steps such as making copies of the source files
in a particular filestore location (though this may make it difficult to achieve correct base URIs).
External variables declared in the test query are bound in the environment using an element of the form:
<param name="x" select="'abc'" declared="true"/>
The declared attribute indicates whether the external variable is declared as such within the query;
this will generally be true for XQuery tests and false for XPath tests, since XPath cannot contain external variable
declarations. If a test with declared="false" is to be run using an XQuery processor, the test driver
may need to customize the query to add an external variable declaration.
The select attribute will always be a simple XPath expression computing the value to be
passed (here a string literal).
The test cases and their associated results have been written with an expectation that the default value for the boundary-space policy is strip. If necessary, a test harness may transform queries by adding "declare boundary-space strip;" for those queries that do not already contain a boundary-space declaration.
The meanings of other elements found in the environment are best discerned by studying the test cases, and the documentation within the catalog schema. (The catalog schema has an associated stylesheet, and clicking this link should show a browsable rendition of the schema rather than the raw XSD code.)
Module import statements may appear in either the main query and/or the
imported module library modules themselves. A library module to be used by a query
will always be listed in the module element of the test case
(it is not part of the context, therefore not part of the environment
element), and it will always appear with the same absolute URI in an import
declaration within the query (or library module) itself. The catalog identifies
the actual location of the module within the test suite using a relative file name.
Generally, import module declarations within a query are written without
any location hints, except where the purpose of the test is to ensure that location hints
can be processed. If an implementation provides no way of resolving an import module
declaration without a location hint, then the test driver may modify the query to add a location
hint before processing.
Similar considerations apply to location hints in import schema
declarations.
The design of the test suite aims to ensure that, as far as possible, the tests can be run by any implementation.
However, the XQuery language places few constraints on the interaction between an XQuery implementation and its environment. Some implementations, for example, might require that queries are embedded in another host language such as SQL. Some implementations may provide no mechanisms for setting parts of the evaluation context, such as collection or collation URIs; it would even be possible (and reasonable, in some circumstances) for an implementation to disallow use of the doc() function (more specifically, to treat all document URIs as unavailable). As a general rule, test drivers are therefore permitted to make whatever modifications to tests as are necessary to execute the test within the constraints imposed by an implementation, provided that imposition of such constraints is permitted by the language specification. This might include, for example, replacing a call to the doc() function by a reference to an external variable.
Currently the test suite does not test XQueryX.
The expected result of a test is expressed as one or more assertions.
Where there are several assertions, they can be combined using all-of
and any-of elements to indicate whether all the assertions must
be satisified, or only one of them. These combinators can be arbitrarily nested.
There are basically four kinds of assertion:
Success assertions (the most common kind) are boolean predicates applied to the result of
the query, considered as an XDM instance. Examples are assert-true and assert-false,
which assert that the query result is the singleton boolean value true or false respectively;
assert-empty which asserts that the result is an empty sequence; assert-type
which asserts the type of the result, and assert-count which asserts its cardinality;
assert-eq, assert-deep-equal, and assert-string-value which compare the
result to a given value with varying comparison semantics. The most general assertion is simply assert
which evaluates a boolean XPath expression with the result sequence as input. The assert-xml
assertion succeeds if the result is an XML document that compares deep-equal to the document contained (in XML form)
in the assertion.
Error assertions succeed only if the test query reports an error. The error assertion indicates the expected error code. Failing with "the wrong error code" may be reported as a pass, but this must be noted in a comment against the test results. (Implementors are advised to check all such cases carefully, since failing for the wrong reason can often indicate a bug in the product under test.)
Serialization assertions are predicates applied to the result of serializing the query, using
the serialization parameters given within the query itself. There are two: assert-serialization
compares the serialization of the query result to a given string as a literal match, and serialization-matches
tests whether it matches a regular expression.
Serialization error assertions succeed only if serialization fails; apart from the fact that the query result must be serialized, they are handled in the same way as other error assertions.
Copyright © 1994-2013 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.