Copyright © 2009 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This document provides an overview of the test suite for the June 25 2009 Candidate Recommendation of the Widgets 1.0: Digital Signatures Specification (P&C). The Working Group has developed this test suite so to have a test case for every testable assertion in the specification.
The Working Group encourages the public and members to contribute tests.
Standardized testing of a specification requires a consistent framework for testing that focuses on both the needs of machines and human testers. The tests in this test suite, and supporting material, are all designed as much as possible to test only the features required for a user agent to conform to the Digital Signatures specification. We begin this document by defining what is a testable assertion, a test case, and what constitutes the test suite.
A testable assertion is any statement in the specification that makes use of the keywords from [RFC2119]: that is, the keywords must, must not, and should. In the P&C specification, all testable assertions have an associated stable identifier. This identifier can be used to associate testable assertions with test cases.
For example, the following sentence constitutes a testable assertion: If this is not the first
name element encountered by the user agent, then
the user agent must skip this element.
A test case is the conversion of a testable assertion (English prose), into a test that a computer can process and that a human tester can also interpret. Test cases always result in either a pass or fail condition. Note that deliberately forcing the user agent to treat a widget as an invalid widget on purpose is a valid error condition. It is up to the test creator to make it clear what the criteria is for a test to pass. Test cases are constructed using test templates.
A test template is a predefined code template that is used to represent a test. All tests in the the test suite are created using the P&C test templates. The templates are specifically designed to only test the minimal amount of capabilities required to be supported by a user agent.
A test suite is, hence, a collection of test cases, which in this instance come in the form of widget packages, that represent each testable assertion. Because test cases apply to different kinds of products, the test suite is structured around products.
For the purpose of standardization, the test suite serves three purposes:
Firstly, the creation of the test suite forces the Working Group to verify that every testable assertion is correctly written and testable against a product that can claim conformance to this specification. An important side effect is that, through the creation of test cases, the test suite can reduce the variability of the specification [Variability]. Reducing the variability makes the specification more precise, which in turn can make the specification easier to interpret and implement.
Secondly, it allows implementation of the specification to use the test-cases to check conformance to the specification.
Thirdly, once the Working Group can show that two implementations pass every test in the test suite, the W3C Process allows the specification to progress to "Proposed Recommendation" status.
The remainder of this document describes user agent requirements and how to create test for the test suite.
In addition to the specifications required to be supported by user agents by the DigSig specification, some tests require, at a minimum, support for the following specifications:
Without support for the above specifications, it may not be possible for a tester to determine if a test has passed or failed.
In order to create test cases for the test suite, the Working Group has settled on using the following P&C test templates:
All test cases are constructed using one or more of the test templates.
Writing a test case is easy, but requires some attention to detail. The steps to write a test case are as follows:
Select a testable assertion from the Testable Assertions tables.
Design the test.
Select the appropriate test templates for the test (if needed).
Create the actual test by fill in the configuration document template, if needed.
Package the test as a Zip file.
Submit it to the Working Group.
Each of the steps above are explained in detail below.
The first step to creating a test is to select a testable assertion from the Testable Assertions table. For illustrative purposes, we will build a few test from the following assertions:
ta-LYLMhryBBT:If this is not the firstnameelement encountered by the user agent, then the user agent must skip this element.
When selecting a testable assertion, be sure to note its identifier.
Identifiers always take the form ta-xxxxxxxxxx. Associating
an identifier to a test allows the Working Group to track which testable
assertions have an associated test case in the test suite.
Having selected which testable assertion is to be tested, the second step is to design the test by decomposing a testable assertion into its testable parts.
The testable assertion we are testing is:
If this is not the firstnameelement encountered by the user agent, then the user agent must skip this element.
Every testable assertion must be able to answer the following three questions:
To which product does the assertion apply? in this case, the user agent
.
What causes is the product to act? In this case, if this
is not the first
.name element [in a configuration document]
encountered by the user agent
How is the product expected to act? In this case, the user
agent must skip this element.
Note that "skip" is a hyperlink to a section of the specification that describes what a user agent needs to do in order to "skip" an element (which is itself a testable assertion):
In the case the user agent is asked to skip an [XML] element or node, a user agent must stop processing the current element, ignoring all of the element‘s attributes and child nodes (if any), and proceed to the next element in the elements list.
Note that ignoring
is also hyperlinked, so when creating a test
be sure to follow all hyperlinks to their absolute definitions.
So, having broken down the assertion onto its component parts, we can design a simple test case:
To pass, the second name element must be skipped by the user
agent. So after Step 7 has completed, the value of the "widget
name" must be PASS.
We must now represent the test case above using appropriate P&C test templates.
Having designed a test case, the test case must now be represented with the appropriate P&C test templates.
The test case being constructed relates to testing the a user
agent’s ability to handle an error condition in a configuration
document (the presence of multiple name elements). So, the base configuration document
template could be extended in the following manner:
<!-- To pass, the second name element must be skipped by the user agent. So
after Step 7 has completed, the value of the "widget name" must be
PASS.
-->
<widget xmlns="http://www.w3.org/ns/widgets">
<name>PASS</name>
<name>FAIL</name>
</widget>The following example is another test that could be derived from the testable assertion above:
<!-- To pass, the second name element must be skipped by the user agent. So
after Step 7 has completed, the value of the "widget name" must be
null.
-->
<widget xmlns="http://www.w3.org/ns/widgets">
<name/>
<name>FAIL</name>
</widget>Note that in the above example, the word "pass"
is not used anywhere to constitute a pass. As already stated, it is up to
the test creator to make it clear what the criteria is for a test to pass.
In the test above, the widget name variable resulting in a null constituted a pass.
In the above two test, a start file cannot be used to determine if the test passed or failed, so the start file template for a neutral condition must be used from the P&C test templates.
The widget package would be structured as follows:
multiple-name-elements.wgt
index.html /*start file template
for a neutral condition*/
config.xmlNote that the working group does not provide guidance on the file names of tests (the summary of the test is more important). However, be sure to give a test a meaningful file name - one that allows a human being to get an idea of what is tested.
There are tests, however, whereby a pass or fail condition can be visualized using the start file template for a pass condition and start file template for a fail condition. Consider, for instance, the following testable assertion:
ta-hkWmGJgfve: If this is not the first content element encountered by the user agent, then the user agent must skip this element.
Deconstructing the above assertion, we can derive the following test:
<!-- To pass, the user agent must ignore the second content element
and pass.html must be the widget's start file.-->
<widget xmlns="http://www.w3.org/ns/widgets">
<content src="pass.html"/>
<content src="fail.html"/>
</widget>Similarly, the following slightly more sophisticated test could be constructed. In this test, the user agent needs to:
<!-- To pass, the user agent must ignore the second content element
and index.html, which is used as a default start file
must be the widget's start file.-->
<widget xmlns="http://www.w3.org/ns/widgets">
<content/>
<content src="fail.html"/>
</widget>
The Working Group accepts test cases from both W3C members and the public (non-members). Please read the how to write a test instructions in this document and, once you have created a test, fill in the Test Grant Form. The details of the patent-free royalty-free nature of the agreement can be found in the W3C policy for contribution of Test Cases.
When you submit a test, please include:
ta-123455678")Once your test-case is ready, please send a URL where the widget can be found to the public-mwts@w3.org mailing list for review. All test-cases must undergo public review before they are included in the test suite.
The Working Group does not provide an automated testing solution for this specification. However, every test case in the test suite is available as a zip archive (and available online). The Working Group also provides an XML file that describes each test case, links to each test case, and identifies the assertion being tested.
Custom designed test harnesses can automate parts of their testing by hashing output renderings and comparing to the sample renderings for each of the P&C test templates.
The working group aims to test the following assertions of the specification:
| # | ID | Testable Assertion | Test cases | Comments |
|---|