As support for Web technologies grows, it is important that tests writers develop test suites that will work as well as possible across devices. This document offers guidance in the form of simple guidelines to follow to create device-independent tests.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This document is the Editors' copy of the device independent testing guidelines and doesn't necessarily reflect the consensus of the Working Group at a given time.
Comments on, and discussions of this document can be sent on the (archived) public mailing list email@example.com (see instructions).
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
This document provide a set of guidelines for writing test cases that can be run effectively across devices, in particular on mobile devices.
The Authoring Challenges for Device Independence [ACDI] explore these different limitations in the general context of writing device independent content, and the Mobile Web Best Practices 1.0 [MWBP] give specific guidance on writing content targeted at mobile devices.
The CSS2.1 Test Case Authoring Guidelines [CSSTCAG] provides guidance on how to write test cases, and sets device-independence as a goal. The SVG Test Suite Manual [SVGTS] also offers advices on writing test cases.
Inspired by this existing work, and based on the experience gathered by the Mobile Web Test Suites Working Group while reviewing test cases and their fitness to mobile devices [MWTSSURVEY], this document explores the specific aspects to take into account when writing test cases to ensure greater device independence.
Consider recording the browser identifier with the widely implemented
window.navigator.userAgent, as there might be several browsers
on one device.
When designing device-independent test cases, it is important to acknowledge the limitations of most devices:
For tests that require interaction (either for running the test or for submitting results), consider:
Trying to take into account all the possible constraints of all possible devices would make writing device-independent tests impossible.
The first step towards writing device-independent test cases is thus to determine the range of devices on which the test cases would need or are likely to be run. To make that assessment:
In any case, it is always a good idea to document in the test suites what are the minimal expected requirements to run the tests, and possibly which test cases require a certain level of support.
Screen size matters when designing visual test cases - e.g. where the tester needs to assess whether the rendering of a test cases matches a reference rendering.
Across devices, the following screen parameters vary widely:
In general, to avoid problems when running test across devices:
To avoid hitting memory limitations of the devices on which the technology under test run:
The characteristics of network access across devices vary greatly, in particular in terms of bandwidth available, the latency induced by network requests, and the cost of transferring content over the network.
To cater for test environments where network is slow or costly:
CPU-intensive operations on mobile devices can drain the battery, and are likely to be much slower than on larger hardware. As a result:
Most mobile devices have limited capabilities when it comes to plugins, additional fonts, or software extensions in general. As a result:
Many mobile devices don't provide a full keyboard, and thus require several key press to enter a given character.
A number of tests do require specific features to be available for the test to work. These requirements could be specific input devices or device APIs, XmlHttpRequest, SVG or similar.