W3C

Guidelines for Running the XML Query Update Test Suite

Overview

This document provides information to implementers who wish to run the XQuery Update Test Suite (XQUTS) on their implementation. It includes guidelines how test cases can be customized in order to run on an implementation, and describes the process of evaluating the results. The documentation of the XML Query Update Test Suite, which defines the structure of the test cases and the catalog, can be found in [1]. Guidelines for submitting results to the XML Query Working Group can be found in [3].

Obtaining a Test Harness

Implementers are expected to write their own test harness that implements the following tasks:

Ideally, the test harness produces an XML file containing all test results in the format shown below, that can be sent to the working group.

The test cases in this test suite are independent of each other, and so can be executed in any order. As test case execution may modify source documents, it is the responsibility of the test harness to return the source documents to their original state before executing any subsequent test cases. Test sources putOutput and putOutput2 refer to files that do not exist. A test harness should delete documents written to these locations after each test case has been run.

Deciding which test cases to run

Some features in the specification are optional or implementation-defined.

Where a test depends on an optional or implementation-defined feature then it is placed in a test group whose group information includes a depends-on element to document the dependency. The names of these features appear (with their meanings) in the enumeration type feature-enum in the schema for the test catalog. For example, static-typing labels tests that depend on the static typing and revalidation:strict depends on revalidation mode strict being supported. If the feature element specifies supported="true" (the default) then the tests in this group should be run only if the product supports the feature; if it specifies supported="false" then it should be run only if the product does not support the feature. If more than one feature element is present then the tests should be run only if all the conditions are satisfied.

One of the purposes of this test suite is to collect data on how many products support optional features. Therefore, products should not attempt to run tests unless they implement the relevant feature.

Test Suite Customization

In order to run the test suite on an XQuery implementation, implementers may customize the test suite and make a number of well-defined changes to the test cases. All changes made to the original test suite must be documented in free-text form as part of the result submission. Changes beyond the ones listed below must be highlighted.

Accessing Source Documents

XQuery supports a number of different ways to refer to source data as query context. Among these are the context item, external variables, the fn:doc() function, the fn:collection() function, implementation-defined functions, or parameter passing through host-language binding. Test cases that do not refer to any input document (i.e., the catalog does not contain any “input-file” for the “test-case”) do not need to be customized in this way. Sometimes the filename "emptydoc" is used to indicate this situation.

The following example is a customizable test case:

(: Name: id-insert-expr-06 :)
(: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)


let $var := $input-context/works[1]/employee[3]
return 
  do insert attribute type {"Part Time"} after $var

and the corresponding catalog entry:

<test-case name="id-insert-expr-06" FilePath= ... >
  <description>Evaluates a simple insert expression with "after" clause only. ... </description>
  <spec-citation spec="XQUPDATE" section-number="2.4.1" ...  section-pointer="#id-insert"/>
  <state time="0">
    <query name="id-insert-expr-06" date="2007-06-04"/>
    <input-file role="principal-data" variable="input-context">works-mod</input-file>
  </state>
  <state time="1">    
    <query name="id-insert-expr-06-test" date="2007-06-04"/>
    <input-file role="principal-data" variable="input-context">works-mod</input-file>                        
    <output-file role="principal" compare="Fragment">id-insert-expr-06.xml</output-file>
  </state>
 </test-case>

A test harness may use any of the following customizations of the query that is supplied. Note that option 3 and 5 are only applicable for test cases that use one source document.

  1. Unchanged: use external variables as indicated in the original query.

    Users of static typing may choose to transform the variable declarations between insert-start and insert-end comments by adding an "as document-node()" clause as illustrated below:

    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    (: insert-start :)
    declare variable $input-context as document-node() external;
    (: insert-end :)
    
    let $var := $input-context/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var
  2. Implicit variable declaration: Remove variable declarations between insert-start and insert-end comments. The implementation binds the input context to the variable $input-context.
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := $input-context/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var
  3. Implicit context: Remove variable declarations between insert-start and insert-end comments, and replace the variable references with the context item (.).
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := ./works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var
  4. doc function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with the fn:doc(someURI), where someURI is a URI that the test harness has associated with the input file.
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := fn:doc("works-mod.xml")/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var>
  5. Default collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection().
        
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := fn:collection()/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var
  6. Collection function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with fn:collection(someURI) , where someURI is a URI that the test harness has associated with the collection identified by the input file.
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := fn:collection("file:collection1")/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var
  7. Implementation-defined function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with an implementation-defined function resolving to the input context.
          
    (: Name: id-insert-expr-06 :)
    (: Description: Evaluation of simple insert expression that uses "after" clause only for an attribute. :)
    
    let $var := impl-fn:impl-udf("works-mod.xml")/works[1]/employee[3]
    return 
      do insert attribute type {"Part Time"} after $var

Query States

A test case may define one or more states (using a <test-case> element and <state> elements). A <state> element may define:

A query may define both an <output-file> and an <expected-error> element. If the same data source is used for multiple states, then that data source should not be set to its original state until the last state is executed. The queries defined within a <state> element can be customized using any of the approaches defined in the previous section.

A test case is successful if each of its states is successful. Each state falls into one of these categories:

  1. no expected results or errors - the state is successful if no error is detected
  2. one or more expected results, no expected errors - the state is successful only if the actual result matches one of the expected results
  3. one or more expected errors, no expected results - the state is successful if one of the expected errors is raised
  4. one or more expected results and one or more expected errors - the state is successful only if the actual result matches one of the expected results or one of the expected errors is raised

A source document may be updated in one state and queried in a subsequent state. Implementations may a) use the XDM instance that is produced by the updating state in the subsequent state, or b) serialize the updated document at the end of the updating state and then parse and validate the document in the subsequent state. Most test cases will execute successfully with either of these two approaches. A small number of test cases, found in the RemoveType and StrictRevalidation groups, will execute successfully only with the first approach.

Host Language Binding

Test cases can be embedded in a host language, for example using the xmlquery function in SQL. This may require escaping certain characters like quotes.

select xmlquery('let $var := $input-context/works[1]/employee[3]
return do insert attribute type {"Part Time"} after $var'
                passing xmlcol as "input-context")
from works-mod

Boundary-space Customization

The test cases and their associated results have been written with an expectation that the default value for the Boundary-space policy is strip. If necessary, a test harness may transform their queries by adding "declare boundary-space strip;" for those queries that do not already contain a Boundary-space Declaration.

Customizing XQueryX Tests

Customizing XQueryX tests must follow the same rules provided above. However, the XQueryX test cases do not include the insert-start/insert-end comments surrounding external variable declaration and schema import. Therefore, a test harness must find the items to be customized in the XQueryX document using the information found in the catalog. The external variable declaration and variable references in the XQueryX document typically looks as follows:

<xqx:varDecl>
  <xqx:varName>input-context</xqx:varName>
  <xqx:external/>
</xqx:varDecl>

<xqx:varRef>
  <xqx:name>input-context</xqx:name>
</xqx:varRef>

Comparing Results

In order to check correctness of running a test case state, the result of the implementation must be compared to the result provided in the test suite. The implementation's result of the test case state must be serialized and compared to the expected file(s) provided in the test suite. Serialization should be performed as described in XSLT 2.0 and XQuery 1.0 Serialization [4] with method="xml". The catalog defines for each test case, which of the following five comparators has to be applied:

It is possible that a test case state provides multiple expected results. In this case, successfully comparing the actual result to one of the provided expected results is a "pass".

Many tests involve operations on floats/doubles and converting those results to strings. Even as one explicit value is given, the task force realizes that other values may also be acceptable. In such cases submitters are encouraged to submit values that may differ. The task force will eventually determine if such values are within the acceptable range.

If all states of a test case "pass", then the result for the test case is a "pass". Otherwise, the result for the test case is a "fail".

The expected files provided in the test suite are serialized forms as specified by XML Query Serialization, with the following parameter values:

byte-order-mark no
cdata-section-elements empty
doctype-public (none)
doctype-system (none)
encoding "utf-8"
escape-uri-attributes (not applicable when method = xml)
include-content-type (not applicable when method = xml)
indent no
media-type not applicable
method xml
normalization-form implementation-defined
omit-xml-declaration yes
standalone omit
undeclare-prefixes no
use-character-maps empty
version implementation-defined

For implementations using different parameters, the test harness must convert the result using the parameters above in order to perform byte-comparison with the provided expected results.

Reference

[1] XQuery Test Suite Documentation
[2] Canonical XML Version 1.0, W3C Recommendation 15 March 2001
(http://www.w3.org/TR/xml-c14n)
[3] Guidelines for Submitting XQUTS Results
[4] XSLT 2.0 and XQuery 1.0 Serialization

Webmaster Last modified: $Date: 2010-07-27 18:41:59 $ by $Author: aeisenbe2 $

Copyright 2008-2010 W3C (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.