Guidelines for Running the XML Query Test Suite

Overview

This document provides information to implementers who wish to run the XQuery Test Suite on their implementation. It includes guidelines how test cases can be customized in order to run on an implementation, and describes the process of evaluating the results. The documentation of the XML Query Test Suite, which defines the structure of the test cases and the catalog, can be found in [1]. Guidelines for submitting results to the XML Query Working Group can be found in [3].

Obtaining a Test Harness

Implementers are expected to write their own test harness that implements the following tasks:

• Read test cases from the catalog, apply customization if applicable (see below)
• Execute tests, using source files specified in the catalog
• Use appropriate comparator to match result
• Produce categorization of test result (pass, fail etc., see below)

Ideally, the test harness produces an XML file containing all test results in the format shown below, that can be sent to the working group.

Test Suite Customization

In order to run the test suite on an XQuery implementation, implementers may customize the test suite and make a number of well-defined changes to the test cases. All changes made to the original test suite must be documented in free-text form as part of the result submission. Changes beyond the ones listed below must be highlighted.

Accessing Source Documents

XQuery supports a number of different ways to refer to source data as query context. Among these are the context item, external variables, the doc function, the collection function, implementation-defined functions, or parameter passing through host-language binding. Test cases that do not refer to any input document (i.e., the catalog does not contain any “input-file” for the “test-case”) do not need to be customized in this way.

The following example is a customizable test case:

(: Name: Axes001 :)
(: Description: Child Element :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)

$input-context/child

and the corresponding catalog entry:

<test-case name="Axes001" FilePath="Axes" ... >
  ...
  <query name="Axes001.xq" date="2003-02-25">
    <description>Child Element</description>
  </query>
  <input-file role="principal-data"
    variable="input-context">TreeCompass</input-file>
  <output-file role="principal" compare="XML">Axes001.xml</output-file>
</test-case>

A test harness may use any of the following customizations of the query that is supplied. Note that option 3 and 5 are only applicable for test cases that use one source document.

1. Unchanged: use external variables as indicated in the original query.

   Users of static typing may choose to transform the variable declarations between insert-start and insert-end comments by adding an "as document-node()" clause as illustrated below:

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   (: insert-start :)
   declare variable $input-context as document-node() external;
   (: insert-end :)
   
   $input-context/child

2. Implicit variable declaration: Remove variable declarations between insert-start and insert-end comments. The implementation binds the input context to the variable $input-context.

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   $input-context/child

3. Implicit context: Remove variable declarations between insert-start and insert-end comments, and replace the variable references with the context item (.).

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
    ./child

4. doc function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with the fn:doc(someURI), where someURI is a URI that the test harness has associated with the input file.

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   fn:doc("TreeCompass.xml")/child

5. Default collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection().

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   fn:collection()/child

6. Collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection(someURI) , where someURI is a URI that the test harness has associated with the collection identified by the input file.

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   fn:collection("file:collection1")/child

7. Implementation-defined function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with an implementation-defined function resolving to the input context.

   (: Name: Axes001 :)
   (: Description: Child Element :)
   
   impl-fn:impl-udf("TreeCompass")/child

Test Cases with fn:doc and fn:collection

Test cases that use the fn:doc and the fn:collection functions need to be customized with a URI that identifies the appropriate data source. The <input-URI> element will be used to identify a data source and indicates that a URI that has been constructed by the test harness will be passed to the query:

The following example is a customizable test case:

(: Name: fn-doc-5 :)
(: Description: Evaluation of fn:doc ... :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)


fn:doc($input-context) is fn:doc($input-context)

and the corresponding catalog entry:

<test-case is-XPath2="true" name="fn-doc-5" ... >
   <description>Evaluation of fn:doc ...</description>
   <spec-citation spec="FuncOps" .../>
   <query name="fn-doc-5" date="2006-01-25"/>
   <input-URI role="principal-data" variable="input-context">works-mod</input-URI>
   <output-file role="principal" compare="Text">fn-doc-5.txt</output-file>
</test-case>

This test case can be customized by adding either “as xs:string” or “as xs:anyURI” to the external variable declaration. This test case can also be customized by removing the variable declaration and replacing the variable reference with a string literal:

(: Name: fn-doc-5 :)
(: Description: Evaluation of fn:doc ... :)

(: insert-start :)
(: insert-end :)

fn:doc('file:works-mod.txt') is fn:doc('file:works-mod.txt')

A test case may use fn:collection to refer to the default collection as follows:

(: Name: fn-collection-4d :)
(: Description: Count the number of nodes in the collection.:)

(: insert-start :)
(: insert-end :)

count(fn:collection())

and the corresponding catalog entry:

test-case is-XPath2="true" name="fn-collection-4d" ... >
   <description>Count the number of nodes in the collection.</description>
   <spec-citation spec="FuncOps" ... />
   <query name="fn-collection-4d" date="2006-07-07"/>
   <defaultCollection role="principal-data">collection1</defaultCollection>
   <output-file role="principal" compare="Text">fn-collection-4.txt</output-file>
</test-case>

A test case may use “.” to refer to the context item. This can be seen in the following:

(: Name: externalcontextitem-22 :)
(: Description: context item expression :)

(: insert-start :)
(: insert-end :)

 ./works/employee[1]

and the corresponding catalog entry:

<test-case is-XPath2="true" name="externalcontextitem-22" ... >
   <description>Evaluation of external context item ...</description>
   <spec-citation spec="XQuery" ... />
   <query name="externalcontextitem-22" date="2006-08-02"/>
   <contextItem role="principal-data">works-mod</contextItem>
   <output-file role="principal" compare="Fragment">externalcontextitem-22.txt</output-file>
</test-case>

Host Language Binding

Test cases can be embedded in a host language, for example using the xmlquery function in SQL. This may require escaping certain characters like quotes.

select xmlquery('$input-context/child'
                passing xmlcol as "input-context")
from TreeCompass

External Variables

External variable tests (with or without types) are designed to use the results of executing the query(s) specified by the <input-query> element, also known as the input query (please note that there could be more than one <input-query> element). The following guidelines are suggested to properly execute these tests.

1. The <input-query> element specifies the query name via the "name" attribute and a variable via the "variable" attribute.
2. Implementors should execute the input query(s) and bind its results to the corresponding "variable" attribute.
3. The main query is specified via the <query> element. The main query will declare one or more external variables, whose names corresponds to the variables defined in the input query.
4. Implementors should bind the values of the variables defined in the input-query(s) to the variable(s) defined in the main query.
5. The main query should then be executed.

The following fragment from the catalog file illustrates the usage of the parameters described above:

<test-case is-XPath2="false" name="extvardeclwithtype-17" ...>
   <description>External Variable used to evaluate a boolean expression ... .</description>
   <spec-citation spec="XQuery" section-number="4.14" ... />
   <query name="extvardeclwithtype-17" date="2006-02-09"/>
   <input-file role="principal-data" variable="input-context">emptydoc</input-file>
   <input-query variable="x" name="extvardeclwithtypetobind-17" date="2006-02-09"/>
   <output-file role="principal" compare="Text">extvardeclwithtype-17.txt</output-file>
</test-case>

Customizing Library Imports

Module import statements may appear in either the main query and/or the imported module library themselves. Any import statement(s) to be customized will appear within the same comments as the context item as described above. Special attention should be given to the following observations during the customization process.

1. Module import statements to be customized must have a namespace URI that matches the namespace attribute of a <module> element in the catalog file.
2. The same library may be imported in both the main query and the imported library.
3. Multiple <module> elements (in the Catalog file) entries with the namespace attribute set to the same value must be customized in such a way that they are part of a single library module statement as illustrated below.

   Entry from Catalog file
     module namespace="http://www.w3.org/TestModules/test1">test1-lib</module>
     module namespace="http://www.w3.org/TestModules/test1">test1a-lib</module>
   
   Import statement from query
     import module namespace test1="http://www.w3.org/TestModules/test1";
   
   Possible customized statement
     import module namespace test1="http://www.w3.org/TestModules/test1" at "test1-lib.xq", "test1a-lib.xq";

4. An imported module may have a schema definition, in which case the schema statement is customized using the procedure as described above
5. The "module/@FileName" attribute (in the <sources> area) does not have an extension. That extension can be found under the "test-suite/@XQueryFileExtension".

Location Hints

The "at" keyword specifies an optional location hint. Location Hints can be interpreted or disregarded in an implementation-dependent way. An implementation can choose to use any of the location hints, or none at all. Implementors testing a system that would only use one location hint may choose rewrite the query to use a URI to a system specific module (which has done the merge "by hand").

Boundary-space Customization

The test cases and their associated results have been written with an expectation that the default value for the Boundary-space policy is strip. If necessary, a test harness may transform their queries by adding "declare boundary-space strip;" for those queries that do not already contain a Boundary-space Declaration.

Customizing XQueryX Tests

Customizing XQueryX tests must follow the same rules provided above. However, the XQueryX test cases do not include the insert-start/insert-end comments surrounding external variable declaration and schema import. Therefore, a test harness must find the items to be customized in the XQueryX document using the information found in the catalog. The external variable declaration and variable references in the XQueryX document typically looks as follows:

<xqx:varDecl>
  <xqx:varName>input-context</xqx:varName>
  <xqx:external/>
</xqx:varDecl>

<xqx:varRef>
  <xqx:name>input-context</xqx:name>
</xqx:varRef>

Comparing Results

In order to check correctness of running a test case, the result of the implementation must be compared to the result provided in the test suite. The implementations result of the test case must be serialized and compared to the expected file(s) provided in the test suite. Serialization should be performed as described in XSLT 2.0 and XQuery 1.0 Serialization [4] with method="xml". The catalog defines for each test case, which of the following five comparators has to be applied:

• XML: The test harness must canonicalize both, the actual result and the expected result according to the “Canonical XML” recommendation [2], which refers to a number of open-source implementations. Byte-comparison can then be applied to the resulting XML documents. If the test harness does this process in a different manner, it must be documented.
• XML fragment: For XML fragments, the same root node must be created for both, implementation result and test suite result. The resulting XML can be compared using XML comparison.
• Text: Same comparison as "XML fragment".
• Ignore: no comparison needs to be applied; the result is always true if the implementation successfully executes the test case. In this case there is no expected file mentioned (for natural reasons), and the contents of the output-file element is empty.
• Inspect: A human is required to make the call about correctness of the result according to the description in the test case.
• Error: The expected result of the test case is an error, identified as an eight-character error code (e.g., XPST0003). The result of a test is true if the implementation raises an error. However, raising an error because an implementation does not support the feature is not considered a correct result.

It is possible that a test case provides multiple expected results. In this case, successfully comparing the actual result to one of the provided expected results is a "pass".

Many tests involve operations on floats/doubles and converting those results to strings. Even as one explicit value is given, the task force realizes that other values may also be acceptable. In such cases submitters are encouraged to submit values that may differ. The task force will eventually determine if such values are within the acceptable range.

Some XQuery 1.1 test cases have been added to the test suite. These test cases contain output-file or expected-error elements with attribute spec-version="1.1". An output-file or expected-error element without a spec-version attribute applies to all versions of XQuery. When this test suite is being used to test XQuery 1.0, a test case that contains only XQuery 1.1 results may be skipped.

In Bug 9605, the WGs agreed that we have been inconsistent in our treatment of zero and negative zero. A test harness may substitute "0" for "-0" in expected results of a test case before comparing them to its actual results.

The expected files provided in the test suite are serialized forms as specified by XSLT 2.0 and XQuery 1.0 Serialization, with the following parameter values:

For implementations using different parameters, the test harness must convert the result using the parameters above in order to perform byte-comparison with the provided expected results.

For the purposes of testing XQueryX 1.0, a test harness may now skip the execution of a test case when:

1. the scenario is parse-error, or
2. an XQuery 1.0 expected error is XPST0003, or
3. the XQueryX document does not validate using the XQueryX 1.0 schema.

Reference

Copyright © 1994-2005 W3C^® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.