In this document we present a method for writing, marking-up, and analyzing conformance requirements in technical specifications.

This is the first publication of this document as a Working Group Note by the Mobile Web Initiative Test Suites Working Group. This publication results from the collaboration between the Mobile Web Initiative Test Suites Working Group and the Web Applications Working Group on the development of test suites for the Widgets family of specifications.

Introduction

In this document we present a method for writing, marking-up, and analyzing conformance requirements in technical specifications.

We argue that the method yields specifications whose conformance requirements are testable: that is, upon applying the method, parts of what is written in the specification can be converted into a test suite without requiring, for instance, the use of a formal language.

The method was derived from a collaboration between the W3C's Mobile Web Initiative: Test Suites Working Group and the Web Applications Working Group. This collaboration aimed to improve the written quality and testability of various specifications. The applications, limitations, as well as possible directions for future work that could refine this method are described in this document.

Common Mistakes

When working on a specification, there are common mistakes an editor can make when writing conformance requirements that makes them difficult, if not impossible, to test. For technical specifications, the testability of a conformance requirement is imperative: conformance requirements eventually become the test cases that implementations rely on to claim conformance to a specification. If no implementation can claim conformance, or if aspects of the specification are not testable, then the probability of a specification becoming a ratified standard, and, more importantly, achieving interoperability among implementations, is significantly reduced.

The most common mistakes that editors make when writing conformance requirements include, but are not limited to:

The Method

Because conformance requirements are intertwined as part of the text of a specification (as sentences, paragraphs, dot points, etc.), it can be difficult to detect the various common mistakes. For this reason, the first step in our method is to identify and mark-up (using HTML) various structural components that constitute a conformance requirement. Understanding these structural components is important, because it is that structure that determines the testability of a conformance requirement. We discuss the structure of conformance requirements, as well as how to mark them up, in more detail below.

Once conformance requirements have been marked up into their component parts, then they can be extracted and analyzed outside the context of the specification. Seeing a conformance requirement out of context can often expose inconsistencies and redundancies that may otherwise been difficult for the editor, or an independent reviewer, to identify. The ability to extract conformance requirements from a specification also allows them to be used in other contexts, such as in the creation of a test suite.

The general process that constitutes the method is as follows:

As the Web Applications Working Group learned, it can be problematic to enter the W3C’s Candidate Recommendation phase without having a complete and thoroughly verified test suite: because this method was mostly applied during Candidate Recommendation, so many redundancies and issues where found that the specification had to drop back to Working Draft. This demonstrated that the method was effective, but needs to be applied as early as possible in the specification writing process.

Relationship to the standardization process

The standards organization, which in this case is the W3C, plays a significant role in relation to the method: the standards organization provides access to a community of experts, as well as the tools that facilitate the interaction and communication between actors and the deliverables that are the outputs of a working group.

Deliverables include the specification, testable assertions, and test cases that constitute the test suite. Actors include editors, test creators, QA engineers, implementers, and specification reviewers. Actors, which in many cases will be the same person in multiple roles, literally provide the intelligence that improves the quality of deliverables.

The tools provided by the standards organization harness the collective intelligence of actors (by capturing their interactions and communications). Some of the tools provided by the W3C include CVS, IRC, a web server, phone bridge, the technical report repository, publication rules checker, issue tracking software, and a mailing list. The standards organization also provides the legal framework that allows multiple competing entities to share intellectual property and collaborate with each other.

The method simply taps into the community-driven process that is standardization, which, through its process, is structured to produce high-quality peer-reviewed work. The following diagram visualizes how actors communicate with each other to improve the quality of various deliverables through tools provided by the standards organization. As can be seen, interaction between actors, tools, and deliverables form feedback loops that serve to improve the work being produced by the working group.

Diagram summarizing the described testing methodology

Value of applying the method

An economic case can be made for identifying and removing redundant conformance requirements from a specification: consider that an average size specification can have around 50 conformance requirements, and each conformance requirement will require one or more test cases. Each test case will require a testable assertion, which may be either written in prose (e.g., “to pass, a must equal b.”) or expressed computationally (e.g., if(a===b)). In terms of resource allocation, someone needs to either manually create or computationally generate the test cases. Someone then needs to verify if each test case actually tests the conformance requirement, and, where it doesn’t, fixes need to be made to either the test case or to the specification.

Eventually, QA engineers will need to run the test cases and conformance violations will need to be reported by filling bug reports. Even in a pure computational setting, having redundant tests in a test suite still results in wasted CPU cycles every time a build of software is run against the test suite. If redundant tests build up, it can have a significant impact on quality assurance processes where it can take hours - or sometimes days - to run builds of a product through various test suites.

Simply put, a test suite for a specification should only test what is necessary for a product to conform - and no more.

It should be noted that “acid tests” (e.g., the Acid3 test) certainly have an important role in creating interoperability by exposing erroneous edge-case behavior and the limitations of implementations. But such stress tests are typically beyond the scope of a test suite for a specification.

Structural Components of a Conformance Requirement

To be testable, a conformance requirement must contains the necessary information to create a testable assertions, as described in The Structure of a Test Assertion in the Test Assertions Guidelines [[OASIS-TAG]].

Consider the following conformance requirement from the [[WIDGETS]] specification:

If the src attribute of the content element is absent or an empty string, then the user agent must ignore this element.

The structure of the conformance requirement can be decomposed into the following structural components:

Product

A product that is supposed to follow the requirement — in this case, the “user agent”. (see also the definition of “classes of product” in [[QAFRAME-SPEC]])

Strictness level

The strictness of the applicability of the requirement to a product — in this case, “the user agent must” do something. W3C specifications use the [RFC2119] keywords (must, should, may, etc.) to indicate the level of requirement that is imposed on a product.

Prerequisites

An explanation of the prerequisites that need to be in place in order for the requirement to apply — in this case, “if the src attribute of the content element is absent or an empty string”.

Behavior

a clear explanation of what the product is supposed to do — in this case, “ignore this element”.

Terms

Keywords that are relevant to understanding how to apply the desired behavior. For instance, what it actually means to “ignore” (definitively and algorithmically) needs to be specified somewhere in the specification.

Terms take one of the three forms in a specification: an algorithm, a definition, or a statement of fact.

An example of an algorithm:

In the case the user agent is asked to ignore an [XML] element or node, a user agent:

  1. Stops processing the current element, ignoring all of the element‘s attributes and child nodes (if any), and proceed to the next element in the elements list.

  2. Make a record that it has attempted to process an element of that type.

An example of a definition:

A user agent is an implementation of this specification that also supports XML...

An example of a statement of fact:

A user agent will need to keep a record of all element types it has attempted to process even if they were ignored (this is to determine if the user agent has attempted to process an element of a given type already).

Having an understanding of the structural components that need to be present in every conformance requirement, an editor can then use the following conventions to mark-up their specification.

Conventions for Marking-up Conformance Requirements

Using mark-up makes it possible to exploit the structure of conformance requirements for various purposes, particularly for analysing that conformance requirements don’t exhibit the common mistakes. Here, we describe how we made use of HTML to markup conformance requirements in the [[WIDGETS]] specifciation. However, this should be considered purely as an example that would need to be adapted to fit each specification particularities.

Consider the following conformance requirement from the [[WIDGETS]] specification, as we will make use of it in this section:

If a user agent encounters a file matching a file name given in the file name column of the default start files table in an arbitrary folder, then user agent must treat that file as an arbitrary file.

From the previous section, we know that the relevant structural components are:

Having identified all the component parts, our method for marking up the conformance requirement is as follows (in no particular order):

The following code shows what the conformance requirement presented at the start of this section would look like once the above dot points are applied:

<p id="ta-a1">If a user agent encounters a <a href="#file">file</a> matching a file name given in the file name column of the <a href="#default-start-files-table">default start files table</a> in an <a href="#arbitrary">arbitrary</a> <a href="#folder">folder</a>, then <a class="product-ua" href="#user-agent">user agent</a> <em class="ct">must</em> treat that file as an <a href="#arbitrary">arbitrary</a> file.</p>

Extracting Conformance Requirements

After the [[WIDGETS]] specification was marked up using the conventions described above, the conformance requirements were extracted using an XSLT style sheet which served as the basis for a review of the testability of the specification.

Over time, the XSLT style sheet was discarded in favor of using a JavaScript system. The JavaScript-based system replicates what the XSLT style sheet was doing, but then mashes the conformance requirements with an XML document that describes all the tests cases in the test suite. This allows those working on the specification and on the test suite to not only see the conformance requirements, but also what test cases have been created.

Because we wrap all conformance requirements in p elements, the actual process of extracting conformance requirements is relatively simple. We use the JQuery JavaScript library, in conjunction with a simple CSS selector, as the means to extract the conformance requirements. The CSS selector finds all p elements in the document that have an id attribute that starts with the string ta-. For example:

function processSpec(spec){
  //CSS selector
	var taSelector = 'p[id^="ta-"]';

	//Extracted nodes
  var requirements    = $(spec).find(taSelector, false);
  //Display the resuts... 
  requirements.each(function(){...}}
}

Testable Assertions and Test Cases

The working groups found that once conformance requirements have been extracted, the work of creating test cases for a test suite was significantly simplified.

A test case is a machine processable object that is used to test one or more conformance requirements. A testable assertion, on the other hand, is a prose description of a test case intended for human testers - i.e., for a given test case, testable assertion defines exactly what the user agent needs to do (behaviorally or conditionally) to pass the test case. It is important to note that testable assertions don’t appear in a specification - they only appear in a test suite to describe a test case.

To create a test, a test writer looks at a given conformance requirement, creates a test case that matches the pre-requisites set in the requirement, and documents the expected outcome described by the required behavior as a testable assertion (or vice versa).

To demonstrate how testable assertions are written, again consider the following conformance requirement from the Widgets Packaging and Configuration [[WIDGETS]] specification:

If the src attribute of the content element is absent or an empty string, then the user agent must ignore this element.

After following the definitions to the terms given in the specification, the conformance requirement above can be turned into one or more testable assertions (which are used to derive test cases for the test suite).

An example of two testable assertion derived from the above conformance requirement:

And the corresponding test cases for the testable assertions take the following computable form (inteded for the user agent):

Each test case can be associated to a given testable assertion; later on, when running the test suite and finding test cases that fail, it allows identifying the assertion behind it that has failed, and thus evaluate which of the implementation, the test case, or the specification is wrong.

To maintain the association between test cases and test assertions, a simple XML file was created:

<testsuite for="http://www.w3.org/TR/widgets/">
<test id="b5"
    for="ta-a1"
    src="test-cases/ta-a1/000/b5.wgt">
Tests that a UA does not go searching in an 
arbitrary folder ("abc123") for default start 
files. To pass, the user agent must treat this 
widget as an invalid widget.
</test>
<test ...> ... </test>
</testsuite>

The testsuite element serves as a wrapper for the test cases of the test suite. It also identifies which test suite was tested through a URI; set in the for attribute.

The test element, on the other had, describes a single test case by:

This XML file allows generating the final round of packaging and information needed for the test suite:

The implementation reports also leverage the identifiers assigned to each test case to indicate if the implementation has passed or failed a test. In order to create the implementation reports, the Working Group created another simple XML format: each implementer is assigned an XML file, which in most cases they themselves maintain.

An example of the results format:

<results testsuite="http://dev.w3.org/2006/waf/widgets/test-suite/test-suite.xml" 
  id="Opera" 
  product="Opera widgets" 
  href="http://opera.com/browser/next">
    
    <result for="b5" verdict="pass"/>

    <result for="dn" verdict="fail">
    Opera did not process the file because it did not 
    have a .wgt file extension.
    </result> 

 </results>

The results element serves as a wrapper that describes what test suite was tested, and some basic details about the implementation by:

The result element, on the other hand, describes individual results gained from testing including:

Tallying the results allowed the working group to easily visualize the data in the test suite for each product:

A bar chart of a graph showing different segments representing, pass, fails, and other bits of information about the conformance of a product to the test suite.

At at glance, it is possible to see for an implementation the number of tests cases passed, failed, and untested. Where the testing was being conducted independently (i.e., not by the implementer), it was also possible to visualize where it was not possible to run a test because, for example, there was no way to get at a result without having direct access to the source code of the product. And where the test ran, but it was not possible to determine if the test actually passed or failed, the verdict was labeled as incomplete.

Having the raw results data also allowed the working group to visualize at a glance how conformant each implementation is to the specification:

Graph showing a 17 percent level of  conformance for a given implementation

The above meter simply represents the number of tests passed by an implementation. However, the working groups found it particularly useful to be able to see all the meters and charts together:

The meters and pie charts all seen next to each other.

Conclusions

While the method described in this document uses three separate steps (marking it up the specification, making the specification testable, and linking test assertions to test cases), these steps don’t have to be applied sequentially, and in practice work best as an iterative process.

Althought it has some limitations and shortcomings, this method has proved effective for the [[WIDGETS]] specification, and is now being applied to the other Widgets specifications developed by the Web Applications Working Group.