Guidelines for Running the XML Query Update Test Suite

Overview

This document provides information to implementers who wish to run the XQuery Update Test Suite (XQUTS) on their implementation. It includes guidelines how test cases can be customized in order to run on an implementation, and describes the process of evaluating the results. The documentation of the XML Query Update Test Suite, which defines the structure of the test cases and the catalog, can be found in [1]. Guidelines for submitting results to the XML Query Working Group can be found in [3].

Obtaining a Test Harness

Implementers are expected to write their own test harness that implements the following tasks:

• Read test cases from the catalog
• Deciding which test cases to run
• Execute each test case by:
  • Applying customization, if applicable (see below)
  • Reading the source files specified in the catalog
  • Executing the queries in each of the query states (in order)
  • Using the appropriate comparators to match the results
  • Categorizing the test result (pass, fail, etc., see below)
• Produce a file containing the results of the test run

Ideally, the test harness produces an XML file containing all test results in the format shown below, that can be sent to the working group.

The test cases in this test suite are independent of each other, and so can be executed in any order. As test case execution may modify source documents, it is the responsibility of the test harness to return the source documents to their original state before executing any subsequent test cases. Test sources putOutput and putOutput2 refer to files that do not exist. A test harness should delete documents written to these locations after each test case has been run.

Deciding which test cases to run

Some features in the specification are optional or implementation-defined.

Where a test depends on an optional or implementation-defined feature then it is placed in a test group whose group information includes a depends-on element to document the dependency. The names of these features appear (with their meanings) in the enumeration type feature-enum in the schema for the test catalog. For example, static-typing labels tests that depend on the static typing and revalidation:strict depends on revalidation mode strict being supported. If the feature element specifies supported="true" (the default) then the tests in this group should be run only if the product supports the feature; if it specifies supported="false" then it should be run only if the product does not support the feature. If more than one feature element is present then the tests should be run only if all the conditions are satisfied.

One of the purposes of this test suite is to collect data on how many products support optional features. Therefore, products should not attempt to run tests unless they implement the relevant feature.

Test Suite Customization

In order to run the test suite on an XQuery implementation, implementers may customize the test suite and make a number of well-defined changes to the test cases. All changes made to the original test suite must be documented in free-text form as part of the result submission. Changes beyond the ones listed below must be highlighted.

Accessing Source Documents

XQuery supports a number of different ways to refer to source data as query context. Among these are the context item, external variables, the fn:doc() function, the fn:collection() function, implementation-defined functions, or parameter passing through host-language binding. Test cases that do not refer to any input document (i.e., the catalog does not contain any “input-file” for the “test-case”) do not need to be customized in this way. Sometimes the filename "emptydoc" is used to indicate this situation.

The following example is a customizable test case:

(: Name: id-insert-expr-06 :)
(: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)


let $var := $input-context/works[1]/employee[3]
return 
  do insert attribute type {"Part Time"} after $var

and the corresponding catalog entry:

<test-case name="id-insert-expr-06" FilePath= ... >
  <description>Evaluates a simple insert expression with "after" clause only. ... </description>
  <spec-citation spec="XQUPDATE" section-number="2.4.1" ...  section-pointer="#id-insert"/>
  <state time="0">
    <query name="id-insert-expr-06" date="2007-06-04"/>
    <input-file role="principal-data" variable="input-context">works-mod</input-file>
  </state>
  <state time="1">    
    <query name="id-insert-expr-06-test" date="2007-06-04"/>
    <input-file role="principal-data" variable="input-context">works-mod</input-file>                        
    <output-file role="principal" compare="Fragment">id-insert-expr-06.xml</output-file>
  </state>
 </test-case>

A test harness may use any of the following customizations of the query that is supplied. Note that option 3 and 5 are only applicable for test cases that use one source document.

1. Unchanged: use external variables as indicated in the original query.

   Users of static typing may choose to transform the variable declarations between insert-start and insert-end comments by adding an "as document-node()" clause as illustrated below:

   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   (: insert-start :)
   declare variable $input-context as document-node() external;
   (: insert-end :)
   
   let $var := $input-context/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

2. Implicit variable declaration: Remove variable declarations between insert-start and insert-end comments. The implementation binds the input context to the variable $input-context.

   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := $input-context/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

3. Implicit context: Remove variable declarations between insert-start and insert-end comments, and replace the variable references with the context item (.).

   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := ./works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

4. doc function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with the fn:doc(someURI), where someURI is a URI that the test harness has associated with the input file.

   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := fn:doc("works-mod.xml")/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var>

5. Default collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection().

       
   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := fn:collection()/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

6. Collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection(someURI) , where someURI is a URI that the test harness has associated with the collection identified by the input file.

   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := fn:collection("file:collection1")/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

7. Implementation-defined function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with an implementation-defined function resolving to the input context.

         
   (: Name: id-insert-expr-06 :)
   (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
   
   let $var := impl-fn:impl-udf("works-mod.xml")/works[1]/employee[3]
   return 
     do insert attribute type {"Part Time"} after $var

Query States

A test case may define one or more states (using a <test-case> element and <state> elements). A <state> element may define:

• A <query> element that defines a query (via the "name" attribute). Each query defined via the <state> element must be run in the exact order they are defined as each state is aware of any changes performed by the previous state(s) and depend on those states to execute correctly. Therefore all queries defined on a previous state must execute correctly before the next one can execute.
• Zero or more <input-file> elements, which defines any input data sources required for this state.
• Zero or more <output-file> elements, which will contain the expected results for this state.
• Zero or more <expected-error> elements, which define the expected errors (if any) for this state.

A test case is successful if each of its states is successful. Each state falls into one of these categories:

1. no expected results or errors - the state is successful if no error is detected
2. one or more expected results, no expected errors - the state is successful only if the actual result matches one of the expected results
3. one or more expected errors, no expected results - the state is successful if one of the expected errors is raised
4. one or more expected results and one or more expected errors - the state is successful only if the actual result matches one of the expected results or one of the expected errors is raised

A source document may be updated in one state and queried in a subsequent state. Implementations may a) use the XDM instance that is produced by the updating state in the subsequent state, or b) serialize the updated document at the end of the updating state and then parse and validate the document in the subsequent state. Most test cases will execute successfully with either of these two approaches. A small number of test cases, found in the RemoveType and StrictRevalidation groups, will execute successfully only with the first approach.

Host Language Binding

Test cases can be embedded in a host language, for example using the xmlquery function in SQL. This may require escaping certain characters like quotes.

select xmlquery('let $var := $input-context/works[1]/employee[3]
return do insert attribute type {"Part Time"} after $var'
                passing xmlcol as "input-context")
from works-mod

Boundary-space Customization

The test cases and their associated results have been written with an expectation that the default value for the Boundary-space policy is strip. If necessary, a test harness may transform their queries by adding "declare boundary-space strip;" for those queries that do not already contain a Boundary-space Declaration.

Customizing XQueryX Tests

Customizing XQueryX tests must follow the same rules provided above. However, the XQueryX test cases do not include the insert-start/insert-end comments surrounding external variable declaration and schema import. Therefore, a test harness must find the items to be customized in the XQueryX document using the information found in the catalog. The external variable declaration and variable references in the XQueryX document typically looks as follows:

<xqx:varDecl>
  <xqx:varName>input-context</xqx:varName>
  <xqx:external/>
</xqx:varDecl>

<xqx:varRef>
  <xqx:name>input-context</xqx:name>
</xqx:varRef>

Comparing Results

In order to check correctness of running a test case state, the result of the implementation must be compared to the result provided in the test suite. The implementation's result of the test case state must be serialized and compared to the expected file(s) provided in the test suite. Serialization should be performed as described in XSLT 2.0 and XQuery 1.0 Serialization [4] with method="xml". The catalog defines for each test case, which of the following five comparators has to be applied:

• XML: The test harness must canonicalize both, the actual result and the expected result according to the “Canonical XML” recommendation [2], which refers to a number of open-source implementations. Byte-comparison can then be applied to the resulting XML documents. If the test harness does this process in a different manner, it must be documented.
• XML fragment: For XML fragments, the same root node must be created for both, implementation result and test suite result. The resulting XML can be compared using XML comparison.
• Text: Same comparison as "XML fragment".
• Ignore: no comparison needs to be applied; the result is always true if the implementation successfully executes the test case.
• Inspect: A human is required to make the call about correctness of the result according to the description in the test case.
• Error: The expected result of the test case state is an error, identified as an eight-character error code (e.g., XPST0003). The result of a test is true, if the implementation raises an error. However, raising an error because an implementation does not support the feature is not considered a correct result.

It is possible that a test case state provides multiple expected results. In this case, successfully comparing the actual result to one of the provided expected results is a "pass".

Many tests involve operations on floats/doubles and converting those results to strings. Even as one explicit value is given, the task force realizes that other values may also be acceptable. In such cases submitters are encouraged to submit values that may differ. The task force will eventually determine if such values are within the acceptable range.

If all states of a test case "pass", then the result for the test case is a "pass". Otherwise, the result for the test case is a "fail".

The expected files provided in the test suite are serialized forms as specified by XML Query Serialization, with the following parameter values:

For implementations using different parameters, the test harness must convert the result using the parameters above in order to perform byte-comparison with the provided expected results.

Reference

Copyright © 2008-2010 W3C^® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.