Abstract

This document defines a set of JavaScript APIs that allow local media, including audio and video, to be requested from a platform.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is not complete. It is subject to major changes and, while early experimentations are encouraged, it is therefore not intended for implementation. The API is based on preliminary work done in the WHATWG.

This document was published by the Web Real-Time Communication Working Group and Device APIs Working Group as an Editor's Draft. If you wish to make comments regarding this document, please send them to public-media-capture@w3.org (subscribe, archives). All comments are welcome.

Publication as an Editor's Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures (Web Real-Time Communication Working Group, Device APIs Working Group) made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

1. Introduction

This section is non-normative.

Access to multimedia streams (video, audio, or both) from local devices (video cameras, microphones, Web cams) can have a number of uses, such as real-time communication, recording, and surveillance.

This document defines the APIs used to get access to local devices that can generate multimedia stream data. This document also defines the MediaStream API by which JavaScript is able to manipulate the stream data or otherwise process it.

2. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this specification are to be interpreted as described in [RFC2119].

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. (In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant.)

Implementations that use ECMAScript to implement the APIs defined in this specification must implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL], as this specification uses that specification and terminology.

3. Terminology

HTML Terms:

The EventHandler interface represents a callback used for event handlers as defined in [HTML5].

The concepts queue a task and fires a simple event are defined in [HTML5].

The terms event handlers and event handler event types are defined in [HTML5].

source

A source is the "thing" providing the source of a media stream track. The source is the broadcaster of the media itself. A source can be a physical webcam, microphone, local video or audio file from the user's hard drive, network resource, or static image.

Some sources have an identifier which must be unique to the application (un-guessable by another application) and persistent between application sessions (e.g., the identifier for a given source device/application must stay the same, but not be guessable by another application). Sources that must have an identifier are camera and microphone sources; local file sources are not required to have an identifier. Source identifiers let the application save, identify the availability of, and directly request specific sources.

Other than the identifier, other bits of source identity are never directly available to the application until the user agent connects a source to a track. Once a source has been "released" to the application (either via a permissions UI, pre-configured allow-list, or some other release mechanism) the application will be able discover additional source-specific capabilities.

Sources do not have constraints -- tracks have constraints. When a source is connected to a track, it must, possibly in combination with UA processing (e.g., downsampling), conform to the constraints present on that track (or set of tracks).

Sources will be released (un-attached) from a track when the track is ended for any reason.

On the MediaStreamTrack object, sources are represented by a sourceType attribute. The behavior of APIs associated with the source's capabilities and settings change depending on the source type.

Sources have capabilities and settings. The capabilities and settings are "owned" by the source and are common to any (multiple) tracks that happen to be using the same source (e.g., if two different track objects bound to the same source ask for the same capability or setting information, they will get back the same answer).

Setting (Source Setting)

A setting refers to the immediate, current value of the source's (optionally constrained) capabilities. Settings are always read-only.

A source's settings can change dynamically over time due to environmental conditions, sink configurations, or constraint changes. A source's settings must always conform to the current set of mandatory constraints that all of the tracks it is bound to have defined, and should do its best to conform to the set of optional constraints specified.

Although settings are a property of the source, they are only exposed to the application through the tracks attached to the source. The ConstrainablePattern interface provides this exposure.

A conforming user-agent must support all the setting names defined in this spec.

As represented in this specification, a source is the realization of a device as presented by the User Agent. Thus, it is possible that the actual settings of the device may differ from those presented by the User Agent. As an example, there are some operating systems and native device APIs that will treat a camera with a single native capture resolution as if it can produce any resolution less than that value, downsampling as necessary. Even though the camera technically has only one specific width and one specific height it can support, it is likely that the User Agent will represent this camera as a source with a range of supported widths and heights. To enable the application to determine when this has occurred, tracks provide both a getSettings() method (which always returns a setting that satisfies the constraints applied to the track) and a getNativeSettings() method (which always returns, to the best of the User Agent's determination, the actual setting of the native device). Note that both the track settings and the native settings are snapshots and can change without application involvement. In particular, changes in the native settings could cause changes in the track settings that would result in the latter values being outside of the constraints and thus causing overconstrained events for all affected tracks.

Capabilities

Source capabilities are the intrinsic "features" of a source object. For each source setting, there is a corresponding capability that describes whether it is supported by the source and if so, what the range of supported values are. As with settings, capabilities are exposed to the application via the ConstrainablePattern interface.

The values of the supported capabilities must be normalized to the ranges and enumerated types defined in this specification.

A getCapabilities() call on a track returns the same underlying per-source capabilities for all tracks connected to the source.

Source capabilities are effectively constant. Applications should be able to depend on a specific source having the same capabilities for any session.

Constraints

Constraints are an optional track feature for restricting the range of allowed variability on a source. Without provided track constraints, implementations are free to select a source's settings from the full ranges of its supported capabilities, and to adjust those settings at any time for any reason.

Constraints are exposed on tracks via the ConstrainablePattern interface, which includes an API for dynamically changing constraints. Note that getUserMedia() also permits an initial set of constraints to be applied when the track is first obtained.

It is possible for two tracks that share a unique source to apply contradictory constraints. The ConstrainablePattern interface supports the calling of an error handler when the conflicting constraint is requested. After successful application of constraints on a track (and its associated source), if at any later time the track becomes overconstrained, the user agent MUST change the track to the muted state.

A correspondingly-named constraint exists for each corresponding source setting name and capability name. In general, user agents will have more flexibility to optimize the media streaming experience the fewer constraints are applied, so application authors are strongly encouraged to use mandatory constraints sparingly.

RTCPeerConnection
RTCPeerConnection is defined in [WEBRTC10].

4. MediaStream API

4.1 Introduction

The two main components in the MediaStream API are the MediaStreamTrack and the MediaStream interfaces. The MediaStreamTrack object represents media originating from a single media source in the user agent, e.g. video from a web camera. A MediaStream is used group several MediaStreamTrack objects into one unit that can be rendered in a media element or recorded.

Each MediaStream can contain zero or more MediaStreamTrack objects. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStream objects do not need to be synchronized.

Note

While the intent is to synchronize tracks, it could be better in some circumstances to permit tracks to lose synchronization.In particular, when tracks are remotely sourced and real-time [WEBRTC10], it can be better to allow loss of synchronization than to accumulate delays or risk glitches and other artifacts. Implementations are expected to understand the implications of choices regarding synchronization of playback and the effect that these have on user perception.

A MediaStreamTrack represents content comprising one or more channels, where the channels have a defined well known relationship to each other (such as a stereo or 5.1 audio signal). A channel is the smallest unit considered in this API specification.

A MediaStream

A MediaStream object has an input and an output that represent the combined input and output of all the object's tracks. The output of the MediaStream controls how the object is rendered, e.g., what is saved if the object is recorded to a file or what is displayed if the object is used in a video element.

A new MediaStream object can be created from accessible media sources (that does not require any additional permissions) using the MediaStream() constructor. The constructor argument can either be an existing MediaStream object, in which case all the tracks of the given stream are added to the new MediaStream object, or an array of MediaStreamTrack objects. The latter form makes it possible to compose a stream from different source streams.

Both MediaStream and MediaStreamTrack objects can be cloned. This allows for greater control since the separate instances can be manipulated and consumed individually. A cloned MediaStream contains clones of all member tracks from the original stream.

When a MediaStream object is being generated from a local file (as opposed to a live audio/video source), the user agent SHOULD stream the data from the file in real time, not all at once. The MediaStream object is also used in contexts outside getUserMedia, such as [WEBRTC10]. In both cases, ensuring a realtime stream reduces the ease with which pages can distinguish live video from pre-recorded video, which can help protect the user's privacy.

4.2 MediaStream

The MediaStream() constructor composes a new stream out of existing tracks. It takes an optional argument of type MediaStream or an array of MediaStreamTrack objects. When the constructor is invoked, the UA must run the following steps:

  1. Let stream be a newly constructed MediaStream object.

  2. Initialize stream's id attribute to a newly generated value.

  3. If the constructor's argument is present, run the sub steps that corresponds to the argument type.

  4. If stream's track set is empty or only contains ended tracks, set stream's active attribute to false, otherwise set it to true.

  5. Return stream.

The tracks of a MediaStream are stored in a track set. The track set MUST contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of the tracks in the set is user agent defined and the API will never put any requirements on the order. The proper way to find a specific MediaStreamTrack object in the set is to look it up by its id.

An object that reads data from the output of a MediaStream is referred to as a MediaStream consumer. The list of MediaStream consumers currently include the media elements [HTML5], RTCPeerConnection [WEBRTC10], MediaRecorder [mediastream-rec] and ImageCapture [mediastream-imagecap].

Note

MediaStream consumers must be able to handle tracks being added and removed. This behavior is specified per consumer.

A MediaStream object is said to be MediaStream.inactive when it does not have any tracks or all tracks belonging to the stream have ended. Otherwise the stream is active. A MediaStream can start its life as inactive if it is constructed without any tracks.

When a MediaStream goes from being active to inactive, the user agent MUST queue a task that sets the object's active attribute to false and fire a simple event named inactive at the object. When a MediaStream goes from being inactive to active, the user agent MUST queue a task that sets the object's active attribute to true and fire a simple event named active at the object.

If the stream's activity status changed due to a user request, the task source for this task is the user interaction task source. Otherwise the task source for this task is the networking task source.

[ Constructor,
 Constructor (MediaStream stream),
 Constructor (sequence<MediaStreamTrack> tracks)]
interface MediaStream : EventTarget {
    readonly    attribute DOMString    id;
    sequence<MediaStreamTrack> getAudioTracks ();
    sequence<MediaStreamTrack> getVideoTracks ();
    sequence<MediaStreamTrack> getTracks ();
    MediaStreamTrack?          getTrackById (DOMString trackId);
    void                       addTrack (MediaStreamTrack track);
    void                       removeTrack (MediaStreamTrack track);
    MediaStream                clone ();
    readonly    attribute boolean      active;
                attribute EventHandler onactive;
                attribute EventHandler oninactive;
                attribute EventHandler onaddtrack;
                attribute EventHandler onremovetrack;
};

4.2.1 Constructors

MediaStream
See the MediaStream constructor algorithm
No parameters.
MediaStream
See the MediaStream constructor algorithm
ParameterTypeNullableOptionalDescription
streamMediaStream
MediaStream
See the MediaStream constructor algorithm
ParameterTypeNullableOptionalDescription
trackssequence<MediaStreamTrack>

4.2.2 Attributes

active of type boolean, readonly

The MediaStream.active attribute returns true if the MediaStream is active (see inactive), and false otherwise.

When a MediaStream object is created, its active attribute MUST be set to true, unless stated otherwise (for example by the MediaStream() constructor algorithm).

id of type DOMString, readonly

When a MediaStream object is created, the user agent MUST generate a globally unique identifier string, and MUST initialize the object's id attribute to that string. Such strings MUST only use characters in the ranges U+0021, U+0023 to U+0027, U+002A to U+002B, U+002D to U+002E, U+0030 to U+0039, U+0041 to U+005A, U+005E to U+007E, and MUST be 36 characters long.

The id attribute MUST return the value to which it was initialized when the object was created.

onactive of type EventHandler,
This event handler, of type active, MUST be supported by all objects implementing the MediaStream interface.
onaddtrack of type EventHandler,
This event handler, of type addtrack, MUST be supported by all objects implementing the MediaStream interface.
oninactive of type EventHandler,
This event handler, of type inactive, MUST be supported by all objects implementing the MediaStream interface.
onremovetrack of type EventHandler,
This event handler, of type removetrack, MUST be supported by all objects implementing the MediaStream interface.

4.2.3 Methods

addTrack

Adds the given MediaStreamTrack to this MediaStream.

When the addTrack() method is invoked, the user agent MUST run the following steps:

  1. Let track be the MediaStreamTrack argument and stream this MediaStream object.

  2. If track is already in stream's track set, then abort these steps.

  3. Add track to stream's track set.

ParameterTypeNullableOptionalDescription
trackMediaStreamTrack
Return type: void
clone

Clones the given MediaStream and all its tracks.

When the MediaStream.clone() method is invoked, the user agent MUST run the following steps:

  1. Let streamClone be a newly constructed MediaStream object.

  2. Initialize streamClone's id attribute to a newly generated value.

  3. Let trackSetClone be a list that contains the result of running MediaStreamTrack.clone() on all the tracks in this stream.

  4. Let trackSetClone be streamClone's track set.

No parameters.
Return type: MediaStream
getAudioTracks

Returns a sequence of MediaStreamTrack objects representing the audio tracks in this stream.

The getAudioTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "audio". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

No parameters.
Return type: sequence<MediaStreamTrack>
getTrackById

The getTrackById() method MUST return the first MediaStreamTrack object in this stream's track set whose id is equal to trackId. The method MUST return null if no track matches the trackId argument.

ParameterTypeNullableOptionalDescription
trackIdDOMString
Return type: MediaStreamTrack, nullable
getTracks

Returns a sequence of MediaStreamTrack objects representing all the tracks in this stream.

The getTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set, regardless of kind. The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

No parameters.
Return type: sequence<MediaStreamTrack>
getVideoTracks

Returns a sequence of MediaStreamTrack objects representing the video tracks in this stream.

The getVideoTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "video". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

No parameters.
Return type: sequence<MediaStreamTrack>
removeTrack

Removes the given MediaStreamTrack from this MediaStream.

When the removeTrack() method is invoked, the user agent MUST remove the track, indicated by the method's argument, from the stream's track set, if present.

ParameterTypeNullableOptionalDescription
trackMediaStreamTrack
Return type: void

4.3 MediaStreamTrack

A MediaStreamTrack object represents a media source in the user agent. Several MediaStreamTrack objects can represent the same media source, e.g., when the user chooses the same camera in the UI shown by two consecutive calls to getUserMedia() .

The data from a MediaStreamTrack object does not necessarily have a canonical binary form; for example, it could just be "the video currently coming from the user's video camera". This allows user agents to manipulate media in whatever fashion is most suitable on the user's platform.

A script can indicate that a track no longer needs its source with the MediaStreamTrack.stop() method. When all tracks using a source have been stopped, the given permission for that source is revoked and the source is stopped. If the data is being generated from a live source (e.g., a microphone or camera), then the user agent SHOULD remove any active "on-air" indicator for that source. If the data is being generated from a prerecorded source (e.g. a video file), any remaining content in the file is ignored. An implementation may use a per source reference count to keep track of source usage, but the specifics are out of scope for this specification.

4.3.1 Life-cycle and Media Flow

Life-cycle

A MediaStreamTrack has three stages in its life-cycle: new, live and ended. A track begins as new prior to being connected to a source. The current stage is reflected by the object's readyState attribute.

Note

This document describres no way to create a MediaStreamTrack that is _not_ connected to a source. new allows for future extensions.

Once connected, the started event fires and the track becomes live. In the live state, the track is active and media is available for use by consumers (but may be replaced by zero-information-content if the MediaStreamTrack is muted or enabled, see below).

A MediaStreamTrack object is said to end when the source of the track is disconnected or exhausted.

A MediaStreamTrack can be detached from its source. It means that the track is no longer dependent on the source for media data. If no other MediaStreamTrack is using the same source, the source will be stopped. MediaStreamTrack attributes such as kind and label MUST not change values when the source is detached.

When a MediaStreamTrack object ends for any reason (e.g., because the user rescinds the permission for the page to use the local camera, or because the data comes from a finite file and the file's end has been reached and the user has not requested that it be looped, or because the application invoked the stop() method on the MediaStreamTrack object, or because the UA has instructed the track to end for any reason) it is said to be ended.

When a MediaStreamTrack track ends for any reason other than the stop() method being invoked, the user agent MUST queue a task that runs the following steps:

  1. If the track's readyState attribute has the value ended already, then abort these steps.

  2. Set track's readyState attribute to ended.

  3. Detach track's source.

  4. Fire a simple event named ended at the object.

If the end of the stream was reached due to a user request, the event source for this event is the user interaction event source.

Media Flow

There are two concepts related to the media flow for a live MediaStreamTrack: muted or not, and enabled or disabled.

Muted refers to the input to the MediaStreamTrack. If live samples are not made available to the MediaStreamTrack it is muted.

Muted is out of control for the application, but can be observed by the application by reading the muted attribute and listening to the associated events mute and unmute.There can be several reasons for a MediaStreamTrack to be muted: the user pushing a physical mute button on the microphone, the user toggling a control in the operating system, the user clicking a mute button in the browser chrome, the UA (on behalf of the user) mutes, etc.

Enabled/disabled on the other hand is available to application to control (and observe) via the enabled attribute.

The result for the consumer is the same in the meaning that whenever MediaStreamTrack is muted or disabled (or both) the consumer gets zero-information-content, which means silence for audio and black frames for video. In other words, media from the source only flows when a MediaStreamTrack object is both unmuted and enabled. For example, a video element sourced by a muted or disabled MediaStreamTrack (contained in a MediaStream), is playing but rendering blackness.

For a newly created MediaStreamTrack object, the following applies: the track is always enabled unless stated otherwise (for example when cloned) and the muted state reflects the state of the source at the time the track is created.

4.3.2 Tracks and Constraints

Constraints are set on tracks and may affect sources.

Whether Constraints were provided at track initialization time or need to be established later at runtime, the APIs defined in the ConstrainablePattern Interface allow the retrieval and manipulation of the constraints currently established on a track.

Each track maintains an internal version of the Constraints structure, namely a mandatory set of constraints (no duplicates), and an optional ordered list of individual constraint objects (may contain duplicates). The internal stored constraint structure is exposed to the application by the constraints attribute, and may be modified by the applyConstraints() method.

When applyConstraints() is called, a user agent MUST queue a task to evaluate those changes when the task queue is next serviced. Similarly, if the sourceType changes, then the user agent MUST perform the same actions to re-evaluate the constraints of each track affected by that source change.

If the MediaStreamError event named 'overconstrained' is thrown, the track MUST be muted until either new satisfiable constraints are applied or the existing constraints become satisfiable.

4.3.3 Interface Definition

interface MediaStreamTrack : EventTarget {
    readonly    attribute DOMString             kind;
    readonly    attribute DOMString             id;
    readonly    attribute DOMString             label;
                attribute boolean               enabled;
    readonly    attribute boolean               muted;
                attribute EventHandler          onmute;
                attribute EventHandler          onunmute;
    readonly    attribute boolean               _readonly;
    readonly    attribute boolean               remote;
    readonly    attribute MediaStreamTrackState readyState;
                attribute EventHandler          onstarted;
                attribute EventHandler          onended;
    Settings              getNativeSettings ();
    MediaStreamTrack      clone ();
    void                  stop ();
    Capabilities          getCapabilities ();
    MediaTrackConstraints getConstraints ();
    Settings              getSettings ();
    void                  applyConstraints (MediaTrackConstraints constraints, VoidFunction successCallback, ConstraintErrorCallback errorCallback);
};
4.3.3.1 Attributes
enabled of type boolean,

The MediaStreamTrack.enabled attribute controls the enabled state for the object.

On getting, the attribute MUST return the last value to which it was set. On setting, it MUST be set to the new value, regardless if the MediaStreamTrack object has been detached from its source or not.

Note

Thus, after a MediaStreamTrack is detached from its source, its enabled attribute still changes value when set; it just doesn't do anything with that new value.

id of type DOMString, readonly

Unless a MediaStreamTrack object is created as a part a of special purpose algorithm that specifies how the track id must be initialized, the user agent MUST generate a globally unique identifier string and initialize the object's id attribute to that string.

An example of an algorithm that specifies how the track id must be initialized is the algorithm to represent an incoming network component with a MediaStreamTrack object. [WEBRTC10]

MediaStreamTrack.id attribute MUST return the value to which it was initialized when the object was created.

kind of type DOMString, readonly

The MediaStreamTrack.kind attribute MUST return the string "audio" if the object represents an audio track or "video" if object represents a video track.

label of type DOMString, readonly

User agents MAY label audio and video sources (e.g., "Internal microphone" or "External USB Webcam"). The MediaStreamTrack.label attribute MUST return the label of the object's corresponding source, if any. If the corresponding source has or had no label, the attribute MUST instead return the empty string.

muted of type boolean, readonly

The MediaStreamTrack.muted attribute MUST return true if the track is muted, and false otherwise.

onended of type EventHandler,
This event handler, of type ended, MUST be supported by all objects implementing the MediaStreamTrack interface.
onmute of type EventHandler,
This event handler, of type mute, MUST be supported by all objects implementing the MediaStreamTrack interface.
onstarted of type EventHandler,
This event handler, of type started, MUST be supported by all objects implementing the MediaStreamTrack interface.
onunmute of type EventHandler,
This event handler, of type unmute, MUST be supported by all objects implementing the MediaStreamTrack interface.
readonly of type boolean, readonly
If the track (audio or video) is backed by a read-only source such as a file, or the track source is a local microphone or camera, but is shared so that constraints applied to the track cannot modify the source's settings, the readonly attribute MUST return the value true. Otherwise, it must return the value false.
readyState of type MediaStreamTrackState, readonly

The readyState attribute represents the state of the track. It MUST return the value to which the user agent last set it.

remote of type boolean, readonly
If the track is sourced by an RTCPeerConnection, the remote attribute MUST return the value true. Otherwise, it must return the value false.
4.3.3.2 Methods
applyConstraints
See ConstrainablePattern Interface for the definition of this method.
ParameterTypeNullableOptionalDescription
constraintsMediaTrackConstraintsA new constraint structure to apply to this object.
successCallbackVoidFunctionCalled if the required constraints can be satisfied.
errorCallbackConstraintErrorCallbackCalled if the required constraints cannot be satisfied.
Return type: void
clone

Clones the given MediaStreamTrack.

When the MediaStreamTrack.clone() method is invoked, the user agent MUST run the following steps:

  1. Let trackClone be a newly constructed MediaStreamTrack object.

  2. Initialize trackClone's id attribute to a newly generated value.

  3. Let trackClone inherit this track's underlying source, kind, label, readyState, and enabled attributes, as well as its currently active constraints.

  4. Return trackClone.

No parameters.
Return type: MediaStreamTrack
getCapabilities

See ConstrainablePattern Interface for the definition of this method.

No parameters.
Return type: Capabilities
getConstraints
See ConstrainablePattern Interface for the definition of this method.
No parameters.
getNativeSettings
The getNativeSettings() method returns the native settings of all the properties of the object. Note that the actual setting of a property must be a single value. Unlike the return value from the getSettings() method, this return object a) MUST reflect, to the best of the User Agent's ability, the actual native settings of the source device, b) MAY have values that do not match the current composite set of constraints applied by all tracks associated with this source, only to the extent necessary to reflect the native settings of the source device, and c) MUST be the same for all tracks associated with this same source.
No parameters.
Return type: Settings
getSettings
See ConstrainablePattern Interface for the definition of this method.
No parameters.
Return type: Settings
stop

When a MediaStreamTrack object's stop() method is invoked, the user agent MUST run following steps:

  1. Let track be the current MediaStreamTrack object.

  2. If track has no source attached (sourceType is "none") or if the source is provided by an RTCPeerConnection, then abort these steps.

  3. Set track's readyState attribute to ended.

  4. Detach track's source.

The task source for the tasks queued for the stop() method is the DOM manipulation task source.

No parameters.
Return type: void
enum MediaStreamTrackState {
    "new",
    "live",
    "ended"
};
Enumeration description
newThe track type is new and has not been initialized (connected to a source of any kind). This state implies that the track's label will be the empty string.
live

The track is active (the track's underlying media source is making a best-effort attempt to provide data in real time).

The output of a track in the live state can be switched on and off with the enabled attribute.

ended

The track has ended (the track's underlying media source is no longer providing data, and will never provide more data for this track). Once a track enters this state, it never exits it.

For example, a video track in a MediaStream ends if the user unplugs the USB web camera that acts as the track's media source.

4.3.4 Track Source Types

enum SourceTypeEnum {
    "none",
    "camera",
    "microphone"
};
Enumeration description
noneThis track has no source. This is the case when the track is in the "new" or "ended" readyState.
cameraA valid source type only for video MediaStreamTracks. The source is a local video-producing camera source.
microphoneA valid source type only for audio MediaStreamTracks. The source is a local audio-producing microphone source.

4.3.5 MediaTrackConstraints

dictionary MediaTrackConstraints : MediaTrackConstraintSet {
    sequence<MediaTrackConstraintSet> advanced;
};
4.3.5.1 Dictionary MediaTrackConstraints Members
advanced of type sequence<MediaTrackConstraintSet>
See Constraints and ConstraintSet for the definition of this element.
dictionary MediaTrackConstraintSet {
    ConstrainLong            width;
    ConstrainLong            height;
    ConstrainDouble          aspectRatio;
    ConstrainDouble          frameRate;
    ConstrainVideoFacingMode facingMode;
    ConstrainDouble          volume;
    ConstrainLong            sampleRate;
    ConstrainLong            sampleSize;
    boolean                  echoCancelation;
    ConstrainDOMString       sourceId;
};
4.3.5.2 Dictionary MediaTrackConstraintSet Members
aspectRatio of type ConstrainDouble
echoCancelation of type boolean
facingMode of type ConstrainVideoFacingMode
frameRate of type ConstrainDouble
height of type ConstrainLong
sampleRate of type ConstrainLong
sampleSize of type ConstrainLong
sourceId of type ConstrainDOMString
volume of type ConstrainDouble
width of type ConstrainLong

4.3.6 Isolated Media Streams

When the peerIdentity option is supplied to getUserMedia, the resulting MediaStream is isolated so that its content is not accessible to any application. An isolated MediaStream may be used for two purposes:

  • Displayed in an appropriate tag (e.g., a video or audio element). The browser MUST ensure that content is inaccessible to the application by ensuring that the resulting content is given the same protections as content that is CORS cross-origin, as described in the relevant Security and privacy considerations section of [HTML5].

  • Used as the argument to addStream() for an RTCPeerConnection, subject to the restrictions detailed in [WEBRTC10].

A MediaStreamTrack that is added to another MediaStream remains isolated. Tracks that are isolated can be added to other MediaStreams, but this causes the resulting MediaStream to have a combination of isolation restrictions. A MediaStream containing MediaStreamTrack instances with mixed isolation properties can be displayed, but cannot be sent using RTCPeerConnection.

Any peerIdentity property MUST be retained on cloned copies of MediaStreamTracks.

4.4 MediaStreamTrackEvent

The addtrack and removetrack events use the MediaStreamTrackEvent interface.

Firing a track event named e with a MediaStreamTrack track means that an event with the name e, which does not bubble (except where otherwise stated) and is not cancelable (except where otherwise stated), and which uses the MediaStreamTrackEvent interface with the track attribute set to track, MUST be created and dispatched at the given target.

dictionary MediaStreamTrackEventInit : EventInit {
    MediaStreamTrack? track;
};

[ Constructor (DOMString type, MediaStreamTrackEventInit eventInitDict)] interface MediaStreamTrackEvent : Event { readonly attribute MediaStreamTrack track; };

4.4.1 Constructors

MediaStreamTrackEvent

TODO

ParameterTypeNullableOptionalDescription
typeDOMString
eventInitDictMediaStreamTrackEventInit

4.4.2 Attributes

track of type MediaStreamTrack, readonly

The track attribute represents the MediaStreamTrack object associated with the event.

4.4.3 Dictionary MediaStreamTrackEventInit Members

track of type MediaStreamTrack, nullable

TODO

5. The model: sources, sinks, constraints, and settings

This section is non-normative.

Browsers provide a media pipeline from sources to sinks. In a browser, sinks are the <img>, <video> and <audio> tags. Traditional sources include streamed content, files, and web resources. The media produced by these sources typically does not change over time - these sources can be considered to be static.

The sinks that display these sources to the user (the actual tags themselves) have a variety of controls for manipulating the source content. For example, an <img> tag scales down a huge source image of 1600x1200 pixels to fit in a rectangle defined with width="400" and height="300".

The getUserMedia API adds dynamic sources such as microphones and cameras - the characteristics of these sources can change in response to application needs. These sources can be considered to be dynamic in nature. A <video> element that displays media from a dynamic source can either perform scaling or it can feed back information along the media pipeline and have the source produce content more suitable for display.

Note

Note: This sort of feedback loop is obviously just enabling an "optimization", but it's a non-trivial gain. This optimization can save battery, allow for less network congestion, etc...

Note that MediaStream sinks (such as <video>, <audio>, and even RTCPeerConnection) will continue to have mechanisms to further transform the source stream beyond that which the Settings, Capabilities, and Constraints described in this specification offer. (The sink transformation options, including those of RTCPeerConnection, are outside the scope of this specification.)

The act of changing or applying a track constraint may affect the settings of all tracks sharing that source and consequently all down-level sinks that are using that source. Many sinks may be able to take these changes in stride, such as the <video> element or RTCPeerConnection. Others like the Recorder API may fail as a result of a source setting change.

The RTCPeerConnection is an interesting object because it acts simultaneously as both a sink and a source for over-the-network streams. As a sink, it has source transformational capabilities (e.g., lowering bit-rates, scaling-up or down resolutions, adjusting frame-rates), and as a source it could have its own settings changed by a track source (though in this specification sources with the remote attribute set to true do not consider the current constraints applied to a track).

To illustrate how changes to a given source impact various sinks, consider the following example. This example only uses width and height, but the same principles apply to any of the Settings exposed in this specification. In the first figure a home client has obtained a video source from its local video camera. The source's width and height settings are 800 pixels by 600 pixels, respectively. Three MediaStream objects on the home client contain tracks that use this same sourceId. The three media streams are connected to three different sinks: a <video> element (A), another <video> element (B), and a peer connection (C). The peer connection is streaming the source video to an away client. On the away client there are two media streams with tracks that use the peer connection as a source. These two media streams are connected to two <video> element sinks (Y and Z).

Changing media stream source effects: before the requested change

Note that at this moment, all of the sinks on the home client must apply a transformation to the original source's provided dimension settings. A is scaling the video up (resulting in loss of quality), B is scaling the video down, and C is also scaling the video up slightly for sending over the network. On the away client, sink Y is scaling the video way down, while sink Z is not applying any scaling.

Using the ConstrainablePattern interface, one of the tracks requests a higher resolution (1920 by 1200 pixels) from the home client's video source.

Changing media stream source
    effects: after the requested change

Note that the source change immediately affects all of the tracks and sinks on the home client, but does not impact any of the sinks (or sources) on the away client. With the increase in the home client source video's dimensions, sink A no longer has to perform any scaling, while sink B must scale down even further than before. Sink C (the peer connection) must now scale down the video in order to keep the transmission constant to the away client.

While not shown, an equally valid settings change request could be made of the away client video source (the peer connection on the away client's side). This would not only impact sink Y and Z in the same manner as before, but could lead to re-negotiation with the peer connection on the home client in order to alter the transformation that it is applying to the home client's video source. Such a change is NOT REQUIRED to change anything related to sink A or B or the home client's video source.

Note that this specification does not define a mechanism by which a change to the away client's video source could automatically trigger a change to the home client's video source. Implementations may choose to make such source-to-sink optimizations as long as they only do so within the constraints established by the application, as the next example demonstrates.

It is fairly obvious that changes to a given source will impact sink consumers. However, in some situations changes to a given sink may also be cause for implementations to adjust a source's settings. This is illustrated in the following figures. In the first figure below, the home client's video source is sending a video stream sized at 1920 by 1200 pixels. The video source is also unconstrained, such that the exact source dimensions are flexible as far as the application is concerned. Two MediaStream objects contain tracks with the same sourceId, and those MediaStreams are connected to two different <video> element sinks A and B. Sink A has been sized to width="1920" and height="1200" and is displaying the source's video content without any transformations. Sink B has been sized smaller and, as a result, is scaling the video down to fit its rectangle of 320 pixels across by 200 pixels down.

Changing media stream sinks may affect sources: before the requested change

When the application changes sink A to a smaller dimension (from 1920 to 1024 pixels wide and from 1200 to 768 pixels tall), the browser's media pipeline may recognize that none of its sinks require the higher source resolution, and needless work is being done both on the part of the source and on sink A. In such a case and without any other constraints forcing the source to continue producing the higher resolution video, the media pipeline MAY change the source resolution:

Changing media stream sinks may affect sources: after the requested change

In the above figure, the home client's video source resolution was changed to the greater of that from sinkA and from sinkB in order to optimize playback. While not shown above, the same behavior could apply to peer connections and other sinks.

It is possible that constraints can be applied to a track which a source is unable to satisfy, either because the source itself cannot satisfy the constraint or because the source is already satisfying a conflicting constraint. When this happens, the applyConstraints() call will fail and call the user-provided ConstraintErrorCallback, without applying any of the new constraints. Since no change in constraints occurs in this case, there is also no required change to the source itself as a result of this condition. Here is an example of this behavior.

In this example, two media streams each have a video track that share the same source. The first track initially has no constraints applied. It is connected to sink N. Sink N has a width and height of 800 by 600 pixels and is scaling down the source's resolution of 1024 by 768 to fit. The other track has a mandatory constraint forcing off the source's fill light; it is connected to sink P. Sink P has a width and height equal to that of the source.

Overconstrained application

Now, the first track adds a mandatory constraint that the fill light should be forced on. At this point, both mandatory constraints cannot be satisfied by the source (the fill light cannot be simultaneously on and off at the same time). Since this state was caused by the first track's attempt to apply a conflicting constraint, the constraint application fails and there is no change in the source's settings or the constraints on either track.

Let's look at a slightly different situation starting from the same point. In this case, instead of the first track attempting to apply a conflicting constraint, the user physically locks the camera into a mode where the fill light is on. At this point the source can no longer satisfy the second track's mandatory constraint that the fill light be off. The second track is transitioned into the muted state and receives an overconstrained event. At the same time, the source notes that its remaining active sink only requires a resolution of 800 by 600 and so it adjusts its resolution down to match (this is an optional optimization that the user agent is allowed to make given the situation).

Overconstrained occurrence

At this point, it is the responsibility of the application to address the problem that led to the overconstrained situation, perhaps by removing the fill light mandatory constraint on the second track or by closing the second track altogether and informing the user

6. MediaStreams as Media Elements

A MediaStream may be assigned to media elements as defined in HTML5 [HTML5] A MediaStream is not preloadable or seekable and represents a simple, potentially infinite, linear media timeline. The timeline starts at 0 and increments linearly in real time as long as the MediaStream is playing. The timeline does not increment when the MediaStream is paused.

6.1 Direct Assignment to Media Elements

UAs that support this specification MUST support the following partial interface, which allows a MediaStream to be assigned directly to a media element.

partial interface HTMLMediaElement {
                attribute MediaStream? srcObject;
};

6.1.1 Attributes

srcObject of type MediaStream, , nullable

Holds the MediaStream that provides media for this element. This attribute overrides both the src attribute and any <source> elements. Specifically, if srcObject is specified, the UA MUST use it as the source of media, even if the src attribute is also set or <source> children are present. If the value of srcObject is replaced or set to null the UA MUST re-run the media element load algorithm

Issue 1

We may want to allow direct assignment of other types as well

6.2 Loading and Playing a MediaStream in a Media Element

The UA runs the media element load algorithm to obtain media for the media element to display. As defined in the [HTML5] specification, this algorithm has two basic phases: resource selection algorithm chooses the resource to play and resolves its URI. Then the resource fetch phase loads the resource. Both these phases are potentially simplified when using a MediaStream. First of all, srcObject takes priority over other means of specifying the resource, and it provides the object itself rather than a URI. Therefore, there is no need to run the resource selection algorithm. Secondly, when the UA reaches the resource fetch algorithm with a MediaStream, the MediaStream is a local object so there's nothing to fetch. Therefore, the following modifications/restrictions to the media element load algorithm apply:

6.3 Media Element Attributes when Playing a MediaStream

The nature of the MediaStream places certain restrictions on the behavior and attribute values of the associated media element and on the operations that can be performed on it, as shown below:

Legal values for the properties of a media element bound to a MediaStream
Attribute Name Attribute Type Valid Values When Using a MediaStream Additional considerations
currentSrc DOMString the empty string When srcObject is specified the UA MUST set this to the empty string.
preload DOMString none A MediaStream cannot be preloaded.
buffered TimeRanges buffered.length MUST return 0. A MediaStream cannot be preloaded. Therefore, the amount buffered is always an empty TimeRange.
networkState unsigned short NETWORK_IDLE The media element does not fetch the MediaStream so there is no network traffic.
readyState unsigned short HAVE_NOTHING, HAVE_ENOUGH_DATA A MediaStream may be created before there is any data available, for example when a stream is received from a remote peer. The value of the readyState of the media element MUST be HAVE_NOTHING before the first media arrives and HAVE_ENOUGH_DATA once the first media has arrived.
currentTime double Any positive integer. The initial value is 0 and the values increments linearly in real time whenever the stream is playing. The value is the current stream position, in seconds. On any attempt to set this attribute, the user agent must throw an InvalidStateError exception.
duration unrestricted double Infinity A MediaStream does not have a pre-defined duration.
seeking boolean false A MediaStream is not seekable. Therefore, this attribute MUST always have the value false.
defaultPlaybackRate double 1.0 A MediaStream is not seekable. Therefore, this attribute MUST always have the value 1.0 and any attempt to alter it MUST fail.
playbackRate double 1.0 A MediaStream is not seekable. Therefore, this attribute MUST always have the value 1.0 and any attempt to alter it MUST fail.
played TimeRanges played.length MUST return 1.
played.start(0) MUST return 0.
played.end(0) MUST return the last known currentTime.
A MediaStream's timeline always consists of a single range, starting at 0 and extending up to the currentTime.
seekable TimeRanges seekable.length MUST return 0.
seekable.start() MUST return currentTime.
seekable.end() MUST return currentTime.
A MediaStream is not seekable.
startDate Date Not-a-Number (NaN) A MediaStream does not specify a timeline offset.
loop boolean true, false Setting the loop attribute has no effect since a MediaStream has no defined end and therefore cannot be looped.

7. Error Handling

All errors defined in this specification implement the following interface:

[NoInterfaceObject]
interface MediaStreamError {
    readonly    attribute DOMString  name;
    readonly    attribute DOMString? message;
    readonly    attribute DOMString? constraintName;
};

7.1 Attributes

constraintName of type DOMString, readonly , nullable

This attribute is only used for some types of errors. For MediaStreamError with a name of ConstraintNotSatisfiedError, this attribute MUST be set to the name of the constraint that caused the error.

message of type DOMString, readonly , nullable
A UA-dependent string offering extra human-readable information about the error.
name of type DOMString, readonly

The name of the error

Note
Open Issue: We may make MediaStreamError inherit from DOMError once the definition of DOMError is stable.
Note
Open Issue: Do we want to allow the constraintName attribute to contain multiple constraint names? In many cases the error is raised as soon as a single unsatisfied mandatory constraint is found, but in others it may be possible to determine that multiple constraints are not satisfied.

The following interface is defined for cases when a MediaStreamError is raised as an event:

dictionary MediaStreamErrorEventInit : EventInit {
    MediaStreamError error;
};

[ Constructor (DOMString type, MediaStreamErrorEventInit eventInitDict)] interface MediaStreamErrorEvent : Event { readonly attribute MediaStreamError error; };

7.2 Constructors

MediaStreamErrorEvent
TODO
ParameterTypeNullableOptionalDescription
typeDOMString
eventInitDictMediaStreamErrorEventInit

7.3 Attributes

error of type MediaStreamError, readonly
TODO

7.4 Dictionary MediaStreamErrorEventInit Members

error of type MediaStreamError

TODO

8. Event summary

This section is non-normative.

The following event fires on MediaStream objects:

Event name Interface Fired when...
active Event The MediaStream became active (see inactive).
inactive Event The MediaStream became inactive.
addtrack MediaStreamTrackEvent A new MediaStreamTrack has been added to this stream. Note that this event is not fired when the script directly modifies the tracks of a MediaStream.
removetrack MediaStreamTrackEvent A MediaStreamTrack has been removed from this stream. Note that this event is not fired when the script directly modifies the tracks of a MediaStream.

The following event fires on MediaStreamTrack objects:

Event name Interface Fired when...
started Event The MediaStreamTrack object has just transitioned from the "new" readyState to another state. This event fires before any other corresponding events such as "ended" or "statechanged".
mute Event The MediaStreamTrack object's source is temporarily unable to provide data.
unmute Event The MediaStreamTrack object's source is live again after having been temporarily unable to provide data.
overconstrained MediaStreamErrorEvent

This error event fires asynchronously for each affected track (when multiple tracks share the same source) after the user agent has evaluated the current constraints against a given sourceId and is not able to configure the source within the limitations established by the union of imposed constraints.

Due to being over-constrained, the user agent must mute each affected track.

The affected track(s) will remain muted until the application adjusts the constraints to accommodate the source's capabilities.

ended Event The MediaStreamTrack object's source will no longer provide any data, either because the user revoked the permissions, or because the source device has been ejected, or because the remote peer permanently stopped sending data.

The following event fires on MediaDevices objects:

Event name Interface Fired when...
devicechange Event The set of media devices, available to the user agent, has changed. The current list devices can be retrieved with the enumerateDevices() method.

9. Enumerating Local Media Devices

This section describes an API that the script can use to query the user agent about connected media input and output devices (for example a web camera or a headset).

9.2 MediaDevices

The MediaDevices object which is the entry point to the API used to examine and get access to media devices available to the user agent.

When a new media input or output device is made available, the user agent MUST queue a task fires a simple event named devicechange at the MediaDevices object.

interface MediaDevices : EventTarget {
                attribute EventHandler ondevicechange;
    void enumerateDevices (MediaDeviceInfoCallback resultCallback);
};

9.2.1 Attributes

ondevicechange of type EventHandler,
This event handler, of type devicechange, MUST be supported by all objects implementing the MediaDevices interface.

9.2.2 Methods

enumerateDevices

Collects information about the user agents available media input and output devices. The method MUST only return information that the script is authorized to access (TODO expand authorized).

When the enumerateDevices() method is called, the user agent must queue a task that runs the following steps:

  1. Let resultCallback be the callback indicated by the methods first argument.

  2. If this method has been called previously within this application session, let oldList be the list of MediaDeviceInfo objects that was produced at that call (resultList); otherwise, let oldList be an empty list.

  3. Let resultList be an empty list.

  4. Probe the user agent for available media devices, and run the following sub steps for each discovered device, device:

    1. If device is represented by a MediaDeviceInfo object in oldList, append that object to resultList, abort these steps and continue with the next device (if any).

    2. Let deviceInfo be a new MediaDeviceInfo object to represent device.

    3. If device belongs to the same physical device as a device, already represented in oldList or resultList, initialize deviceInfo's groupId member to the groupId value of the existing MediaDeviceInfo object. Otherwise, let deviceInfo's groupId member be a newly generated unique identifier

    4. Append deviceInfo to resultList.

  5. Invoke resultCallback with resultList as its argument.

ParameterTypeNullableOptionalDescription
resultCallbackMediaDeviceInfoCallback
Return type: void

9.3 Device Info

callback MediaDeviceInfoCallback = void (sequence<MediaDeviceInfo> deviceInfoList);

9.3.1 Callback MediaDeviceInfoCallback Parameters

deviceInfoList of type sequence<MediaDeviceInfo>
A sequence of MediaDeviceInfo objects representing the result of a call to MediaDevices.enumerateDevices() .
Note
The old SourceInfo dictionary used to refer to the MediaSourceStates dictionary. That dictionary is no longer available when Constrainable is introduced. When Constrainable lands, we should see if we can align deviceId, kind and label, below, with the new definitions of source capabilities.
dictionary MediaDeviceInfo {
    DOMString       deviceId;
    MediaDeviceKind kind;
    DOMString       label;
    DOMString       groupId;
};

9.3.2 Dictionary MediaDeviceInfo Members

deviceId of type DOMString

The unique id for the represented device.

groupId of type DOMString

Returns the group identifier of the represented device. Two devices have the same group identifier if they belong to the same physical device; for example a headset.

kind of type MediaDeviceKind

Describes the kind of the represented device.

label of type DOMString

A label describing this device (for example "External USB Webcam"). If the device has no associated label, then this dictionary member MUST return the empty string.

enum MediaDeviceKind {
    "audioinput",
    "audiooutput",
    "videoinput"
};
Enumeration description
audioinput

Represents an audio input device; for example a microphone.

audiooutput

Represents an audio output device; for example a pair of headphones.

videoinput

Represents a video input device; for example a webcam.

10. Obtaining local multimedia content

This section extends NavigatorUserMedia and MediaDevices with APIs to request permission to access media input devices available to the user agent.

10.2 MediaDevices Interface Extensions

The getSupportedConstraints method is provided to allow the application to determine which constraints the User Agent recognizes.

partial interface MediaDevices {
    static Dictionary getSupportedConstraints (DOMString kind);
    void              getUserMedia (MediaStreamConstraints? constraints, NavigatorUserMediaSuccessCallback successCallback, NavigatorUserMediaErrorCallback errorCallback);
};

10.2.1 Methods

getSupportedConstraints, static

Returns a dictionary whose members are the constraint keys known to the User Agent for the kind given as argument. A supported constraint MUST be represented by a member whose name is the constraint name and whose value is true. Any constraint names not supported by the User Agent MUST not be present in the returned dictionary.

ParameterTypeNullableOptionalDescription
kindDOMString
Return type: Dictionary
getUserMedia

Prompts the user for permission to use their Web cam or other video or audio input.

Issue 2

(Remove when other issues are removed. This is only here to keep the issues from being renumbered)

The constraints argument is an object of type MediaStreamConstraints.

The successCallback will be invoked with a suitable MediaStream object as its argument if the user accepts valid tracks as described below.

The errorCallback will be invoked if there is a failure in finding valid tracks or if the user denies permission, both as described below.

When the getUserMedia() method is called, the user agent MUST run the following steps:

  1. Let constraints be the method's first argument.

  2. Let successCallback be the callback indicated by the method's second argument.

  3. Let errorCallback be the callback indicated by the method's third argument.

  4. Let requestedMediaTypes be the set of media types in constraints with either a dictionary value or a value of "true".

  5. If requestedMediaTypes is the empty set, let error be a new MediaStreamError object whose name attribute has the value NotSupportedError and jump to the step labeled Error Task below.

  6. Let finalSet be an (initially) empty set.

  7. If successCallback is null, abort these steps.

  8. For each media type T in requestedMediaTypes,

    1. Let candidateSet be all possible tracks of media type T that the browser could return.

    2. If the value of the T entry of constraints is "true", jump to the step labeled final below. Otherwise, continue with CS set to the value of the T entry of constraints.
    3. For each required ('min', 'max', or 'exact') constraint provided for a constraint name in CS,

      1. If the constraint is not supported by the browser, jump to the step labeled Constraint Failure below.

      2. Remove from the candidateSet any track that cannot satisfy the value given for the constraint in CS, if any.

      3. If the candidateSet no longer contains at least one track, jump to the step labeled Constraint Failure below. Otherwise, continue with the next required constraint.

    4. Let the secondPassSet be the current contents of the candidateSet. Note that unknown properties are discarded by WebIDL, which means that unknown/unsupported required constraints will silently disappear. To avoid this being a surprise, application authors are expected to first use the getSupportedConstraints() method as shown in the Examples.

    5. For each constraint key-value pair in the "advanced" sequence of CS, in order,

      1. If the constraint is not supported by the browser, skip it and continue with the next constraint.

      2. Remove from the secondPassSet any tracks that cannot satisfy the value given for the constraint.

      3. If the secondPassSet is now empty, let the secondPassSet be the current contents of the candidateSet. Otherwise, let the candidateSet be the current contents of the secondPassSet.

    6. Let the thirdPassSet be the current contents of the candidateSet.

    7. For all non-required ('ideal' or bare-value) constraints in CS, identify the maximum number of such constraint pairs that could be satisfied by at least one track in thirdPassSet.

    8. The decision of which of these non-required constraints to satisfy is completely up to the user agent, as long as the number of constraints satisfied matches the number in the previous step.

    9. For the non-required constraints the user agent has decided to satisfy, remove from the thirdPassSet any tracks that cannot satisfy those constraints.

    10. If the thirdPassSet is now empty, let the thirdPassSet be the current contents of the candidateSet. Otherwise, let the candidateSet be the current contents of the thirdPassSet.

    11. Final: Add the tracks in the candidateSet to the finalSet.

  9. Return, and run the remaining steps asynchronously.

  10. Optionally, e.g., based on a previously-established user preference, for security reasons, or due to platform limitations, jump to the step labeled Permission Failure below.

  11. Prompt the user in a user agent specific manner for permission to provide the entry script's origin with a MediaStream object representing a media stream.

    The provided media MUST include precisely one track of each media type in requestedMediaTypes from the finalSet. The decision of which tracks to choose from the finalSet is completely up to the user agent and may be determined by asking the user. Once selected, the source of a MediaStreamTrack MUST not change.

    Issue 3

    Define the event that should be raised when the user agent changes its choice of track.

    User agents are encouraged to default to using the user's primary or system default camera and/or microphone (when possible) to generate the media stream. User agents MAY allow users to use any media source, including pre-recorded media files.

    If the user grants permission to use local recording devices, user agents are encouraged to include a prominent indicator that the devices are "hot" (i.e. an "on-air" or "recording" indicator).

    If the user denies permission, jump to the step labeled Constraint Failure below. If the user never responds, this algorithm stalls on this step.

  12. Let stream be the MediaStream object for which the user granted permission.

  13. Queue a task to invoke successCallback with stream as its argument.

  14. Abort these steps.

  15. Permission Failure: Let error be a new MediaStreamError object whose name attribute has the value PermissionDeniedError and jump to the step labeled Error Task below.

  16. Constraint Failure: Let error be a new MediaStreamError object whose name attribute has the value ConstraintNotSatisfiedError and whose constraintName attribute is set to the name of the constraint that caused the error.

  17. Error Task: Queue a task to invoke errorCallback with error as its argument.

  18. The task source for these tasks is the user interaction task source.

ParameterTypeNullableOptionalDescription
constraintsMediaStreamConstraints
successCallbackNavigatorUserMediaSuccessCallback
errorCallbackNavigatorUserMediaErrorCallback
Return type: void

10.3 MediaStreamConstraints

The MediaStreamConstraints dictionary is used to instruct the UA what sort of MediaStreamTracks to include in the MediaStream returned by getUserMedia().

dictionary MediaStreamConstraints {
    (boolean or MediaTrackConstraints) video = false;
    (boolean or MediaTrackConstraints) audio = false;
    DOMString                          peerIdentity;
};

10.3.1 Dictionary MediaStreamConstraints Members

audio of type (boolean or MediaTrackConstraints), defaulting to false

If true, it requests that the returned MediaStream contain an audio track. If a Constraints structure is provided, it further specifies the nature and settings of the audio Track. If false, the MediaStream MUST not contain an audio Track.

peerIdentity of type DOMString

If the peerIdentity attribute is provided, the resulting MediaStream will be isolated from the application.

video of type (boolean or MediaTrackConstraints), defaulting to false

If true, it requests that the returned MediaStream contain a video track. If a Constraints structure is provided, it further specifies the nature and settings of the video Track. If false, the MediaStream MUST not contain a video Track.

10.6 Implementation Suggestions

This section is non-normative.

Best Practice 1: Resource reservation

The user agent is encouraged to reserve resources when it has determined that a given call to getUserMedia() will succeed. It is preferable to reserve the resource prior to invoking the success callback provided by the web page. Subsequent calls to getUserMedia() (in this page or any other) should treat the resource that was previously allocated, as well as resources held by other applications, as busy. Resources marked as busy should not be provided as sources to the current web page, unless specified by the user. Optionally, the user agent may choose to provide a stream sourced from a busy source but only to a page whose origin matches the owner of the original stream that is keeping the source busy.

This document recommends that in the permission grant dialog or device selection interace (if one is present), the user be allowed to select any available hardware as a source for the stream requested by the page (provided the resource is able to fulfill mandatory constraints, if any were specified), in addition to the ability to substitute a video or audio source with local files and other media. A file picker may be used to provide this functionality to the user.

This document also recommends that the user be shown all resources that are currently busy as a result of prior calls to getUserMedia() (in this page or any other page that is still alive) and be allowed to terminate that stream and utilize the resource for the current page instead. If possible in the current operating environment, it is also suggested that resources currently held by other applications be presented and treated in the same manner. If the user chooses this option, the track corresponding to the resource that was provided to the page whose stream was affected must be removed.

Best Practice 2: Handling multiple devices

A MediaStream may contain more than one video and audio track. This makes it possible to include video from two or more webcams in a single stream object, for example. However, the current API does not allow a page to express a need for multiple video streams from independent sources.

It is recommended for multiple calls to getUserMedia() from the same page be allowed as a way for pages to request multiple, discrete, video or audio streams.

A single call to getUserMedia() will always return a stream with either zero or one audio tracks, and either zero or one video tracks. If a script calls getUserMedia() multiple times before reaching a stable state, this document advises the UI designer that the permission dialogs should be merged, so that the user can give permission for the use of multiple cameras and/or media sources in one dialog interaction. The constraints on each getUserMedia call can be used to decide which stream gets which media sources.

11. Constrainable Pattern

The Constrainable pattern allows its consumers to inspect and adjust the properties of the object that implements it. It is broken out as a separate set of definitions so that it can be referred to by other specifications. The core concept is that of a Capability, which consists of a property or feature of an object and the set of its possible values, which may be specified either as a range or as an enumeration. For example, a camera might be capable of framerates (a property) between 20 and 50 frames per second (a range) and may be able to be positioned (a property) facing towards the user, away from the user, or to the left or right of the user (an enumerated set.) The application can examine a ConstrainablePattern object's set of Capabilities via the getCapabilities() accessor.

The application can select the (range of) values it wants for an object's Capabilities by means of basic and/or advanced ConstraintSets and the applyConstraints() method. A ConstraintSet consists of the names of one or more properties of the object plus the desired value (or a range of desired values) for each of them. Each of those property/value pairs can be considered to be an individual constraint. For example, the application may set a ConstraintSet containing two constraints, the first stating that the framerate of a camera be between 30 and 40 frames per second (a range) and the second that the camera should be facing the user (a specific value). How the individual constraints interact depends on whether and how they are given in the basic Constraint structure, which is a ConstraintSet with an additional 'advanced' property, or whether they are in a ConstraintSet in the advanced list. The behavior is as follows: all 'min', 'max', and 'exact' constraints in the basic Constraint structure are together treated as the 'required' set, and if it is not possible to satisfy simultaneously all of those individual constraints for tho indicated property names, the UA MUST call the errorCallback. Otherwise, it must apply the required constraints. Next, it will consider any ConstraintSets given in the 'advanced' list, in the order in which they are specified, and will try to satisfy/apply each complete ConstraintSet (i.e., all constraints in the ConstraintSet together), but will skip a ConstraintSet if and only if it cannot satisfy/apply it in its entirety. Next, the UA MUST attempt to apply, individually, any 'ideal' constraint or a constraint given as a bare value for the property. Of these properties, it MUST satisfy the largest number that it can, in any order. Finally, the UA MUST call the successCallback.

Important note: If JavaScript applications using this API want the attributes in the constraints to be used by the browser, the JavaScript code has to first check, via getSupportedConstraints(), that all the named properties that are used are supported by the browser. The reason for this is that WebIDL drops any unsupported names from the dictionary holding the constraints, so the browser does not see them and the unsupported names end up being silently ignored. This will cause confusing programming errors as the JavaScript code will be setting constraints but the browser will be ignoring them. Browsers that support (recognize) the name of a required constraint but cannot satisfy it will generate an error, while browsers that do not support the given name of the constraint will not generate an error.

Issue 6

The definition and behavior of 'ideal', along with that of bare values in the basic constraint structure (which are assumed to mean 'ideal'), have not yet been agreed upon, or even thoroughly discussed.

The following examples may help to understand how constraints work. The first shows a basic Constraint structure. Three constraints are given, each of which the UA will attempt to satisfy individually. Depending upon the resolutions available for this camera, it is possible that not all three constraints can be satisfied at the same time. If so, the user agent will satisfy two if it can, or only one if not even two constraints can be satisfied together. Note that if not all three can be satisfied simultaneously, it is possible that there is more than one combination of two constraints that could be satisfied. If so, the user agent will choose.

Example 1
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["aspectRatio"]) {
  // Treat like an error.
}
 var constraints =
  {
    width: 1280,
    height: 720,
    aspectRatio: 1.5
  };

This next example adds a small bit of complexity. The ideal values are still given for width and height, but this time with minimum requirements on each as well that must be satisfied. If it cannot satisfy either the width or height minimum it will call the errorCallback. Otherwise, it will try to satisfy the width, height, and aspectRatio target values as well and then call the successCallback.

Example 2
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["aspectRatio"]) {
  // Treat like an error.
}
 var constraints =
  {
    width: {min: 640, ideal: 1280},
    height: {min: 480, ideal: 720},
    aspectRatio: 1.5
  };

This example illustrates the full control possible with the Constraints structure by adding the 'advanced' property. In this case, the user agent behaves the same way with respect to the required constraints, but before attempting to satisfy the ideal values it will process the 'advanced' list. In this example the 'advanced' list contains two ConstraintSets. The first specifies width and height constraints, and the second specifies an aspectRatio constraint. Note that in the advanced list, these bare values are treated as 'exact' values. This example represents the following: "I need my video to be at least 640 pixels wide and at least 480 pixels high. My preference is for precisely 1920x1280, but if you can't give me that, give me an aspectRatio of 4x3 if at all possible. If even that is not possible, give me a resolution as close to 1280x720 as possieble."

Example 3
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["width"] || !supports["height"]) {
  // Treat like an error.
}
 var constraints =
  {
    width: {min: 640, ideal: 1280},
    height: {min: 480, ideal: 720},
    advanced: [{width: 1920, height: 1280},
               {aspectRatio: 1.3333333333}]
  };

The ordering of advanced ConstraintSets is significant. In the preceding example it is impossible to satisfy both the 1920x1280 ConstraintSet and the 4x3 aspect ratio ConstraintSet at the same time. Since the 1920x1280 occurs first in the list, the user agent will attempt to satisfy it first. Application authors can therefore implement a backoff strategy by specifying multiple optional ConstraintSets for the same property. For example, an application might specify three optional ConstraintSets, the first asking for a framerate greater than 500, the second asking for a framerate greater than 400, and the third asking for one greater than 300. If the UA is capable of setting a framerate greater than 500, it will (and the subsequent two ConstraintSets will be trivially satisfied.) However, if the UA cannot set the framerate above 500, it will skip that ConstraintSet and attempt to set the framerate above 400. If that fails, it will then try to set it above 300. If the UA cannot satisfy any of the three ConstraintSets, it will set the framerate to any value it can get. If the developer wanted to insist on 300 as a lower bound, he could provide that as a 'min' value in the basic ConstraintSet. In that case, the UA would fail altogether if it couldn't get a value over 300, but would choose a value over 500 if possible, then try for a value over 400.

Note that, unlike basic constraints, the constraints within a ConstraintSet in the advanced list must be satisfied together or skipped together. Thus, {width: 1920, height: 1280} is a request for that specific resolution, not a request for that width or that height. One can think of the basic constraints as requesting an or (non-exclusive) of the individual constraints, while each advanced ConstraintSet is requesting an and of the individual constraints in the ConstraintSet. An application may inspect the full set of Constraints currently in effect via the getConstraints() accessor.

The specific value that the UA chooses for a Capability is referred to as a Setting. For example, if the application applies a ConstraintSet specifying that the framerate must be at least 30 frames per second, and no greater than 40, the Setting can be any intermediate value, e.g., 32, 35, or 37 frames per second. The application can query the current settings of the object's Capabilities via the getSettings() accessor.

11.1 Interface Definition

Due to the limitations of the interface definition language used in this specification, it is not possible for other interfaces to inherit or implement ConstrainablePattern. Therefore the WebIDL definitions given are only templates to be copied. Each interface that wishes to make use of the functionality defined here will have to provide its own copy of the WebIDL for the functions and interfaces given here. However it can refer to the semantics defined here, which will not change. See MediaStreamTrack Interface Definition for an example of this.

[NoInterfaceObject]
interface ConstrainablePattern {
    Capabilities getCapabilities ();
    Constraints  getConstraints ();
    Settings     getSettings ();
    void         applyConstraints (Constraints constraints, VoidFunction successCallback, ConstraintErrorCallback errorCallback);
                attribute EventHandler onoverconstrained;
};

11.1.1 Attributes

onoverconstrained of type EventHandler,
This event handler, of type overconstrained, must be supported by all objects implementing the ConstrainablePattern pattern.

The UA must raise a MediaStreamErrorEvent named "overconstrained" if changing circumstances at runtime result in it no longer being able to satisfy the requiredConstraints from the currently valid Constraints. This MediaStreamErrorEvent must contain a MediaStreamError whose name is "overconstrainedError", and whose constraintName attribute is set to one of the requiredConstraints that can no longer be satisfied. The message attribute of the MediaStreamError SHOULD contain a string that is useful for debugging. The conditions under which this error might occur are platform and application-specific. For example, the user might physically manipulate a camera in a way that makes it impossible to provide a resolution that satisfies the constraints. The UA MAY take other actions as a result of the overconstrained situation.

11.1.2 Methods

applyConstraints

The applyConstraints() algorithm for applying constraints is stated below. Here are some preliminary definitions that are used in the statement of the algorithm:

  • We refer to each element of a ConstraintSet (other than the special term 'advanced') as a 'constraint' since it is intended to constrain the corresponding Capability of the ConstrainablePattern object to a value that is within the range or list of values it specifies.
  • We refer to the "effective Capability" C of an object O as the possibly proper subset of the possible values of C (as returned by getCapabilities) taking into consideration environmental limitations and/or restrictions placed by other constraints. For example given a ConstraintSet that constrains Capabilities aspectRatio, height and width, the values assigned to any two of the Capabilities limit the effective Capability of the third. The set of effective Capabilities may be platform dependent. For example, on a resource-limited device it may not be possible to set Capabilities C1 and C2 both to 'high', while on another less limited device, this may be possible.
  • A set of values for the Capabilities of an object O satisfy ConstraintSet CS if each value a) is in the range of the corresponding effective Capability of O, and b) is in the range or list of values specified by the corresponding constraint in CS, if there is one, and c) there is no constraint in CS that does not correspond to a Capability of O. (Note that although this definition ignores the difference between required and optional ConstraintSets, the algorithm below distinguishes between them.)
  • A set of ConstraintSets CS1...CSn (n ≥ 1) can be satisfied by an object O if it is possible to choose a sequence of values for the Capabilities of O that satisfy CS1...CSn simultaneously.
  • To apply a set of ConstraintSet CS1...CSn to object O is to choose such a sequence of values that satisfy CS1...CSn and assign them as the settings for the properties of O.

When applyConstraints is called, the UA must queue a task to run the following steps:

  1. let newContraints be the argument to this function. Each constraint must specify one or more values (or a range of values) for its property. A property may appear more than once in the list of 'advanced' ConstraintSets. Let requiredConstraints be the set of 'min', 'max', and 'exact' constraints appearing in newConstraints. Let unorderedConstraints be all the other constraints in newConstraints. Note that 'advanced' is not a member of either of these sets since it represents meta-information about constraints. In the following we treat requiredConstraints and unorderedConstraints as ConstraintSets. Note that unknown properties are discarded by WebIDL, which means that unknown/unsupported required constraints will silently disappear. To avoid this being a surprise, application authors are expected to first use the getSupportedConstraints() method as shown in the Examples below.
  2. Let object be the ConstrainablePattern object on which this method was called. Let copy be an unconstrained copy of object (i.e., copy should behave as if it were object with all ConstraintSets removed.)
  3. If requiredConstraints is non-null and cannot be satisfied by copy, call the errorCallback, passing it a new MediaStreamError with name ConstraintNotSatisfied and constraintName set to any of the required constraints that could not be satisfied, and return. existingConstraints remain in effect in this case.
  4. Let successfulConstraints be a list of ConstraintSets, initially containing only requiredConstraints. Iterate over the 'advanced' ConstraintSets in newConstraints in the order in which they were specified. For each ConstraintSet, if it and successfulConstraints together can be satisfied by copy, append it to the rear of successfulConstraints. Otherwise, skip it.
  5. In a single operation, remove existingConstraints from object, apply successfulConstraints, then apply the largest subset of unorderedConstraints that can be satisifed. (If more than one 'largest subset' can be satisfied, the UA is free to select which 'largest subset' it chooses.) Finally fire the successCallback. From this point on until applyConstraints() is called successfully again, getConstraints() must return the newConstraints that were passed as an argument to this call.

The UA may choose new settings for the Capabilities of the object at any time. When it does so it must attempt to satisfy the current Constraints, in the manner described in the algorithm above.

Issue 7

The definition of how multiple unorderedConstraints are to be satisfied together is still very much under discussion. Also, please see issue 6 about 'ideal' not yet being defined.

ParameterTypeNullableOptionalDescription
constraintsConstraintsA new constraint structure to apply to this object.
successCallbackVoidFunctionCalled if the required constraints can be satisfied.
errorCallbackConstraintErrorCallbackCalled if the required constraints cannot be satisfied.
Return type: void
getCapabilities

The getCapabilities() method returns the dictionary of the capabilities that the object supports.

Note

It is possible that the underlying hardware may not exactly map to the range defined in the registry entry. Where this is possible, the entry should define how to translate and scale the hardware's setting onto the values defined in the entry. For example, suppose that a registry entry defines a hypothetical fluxCapacitance capability that is defined to be the range from -10 (min) to 10 (max), but there are common hardware devices that support only values of "off" "medium" and "full". The registry entry might specify that for such hardware, the user agent should map the range value of -10 to "off", 10 to "full", and 0 to "medium". It might also indicate that given a ConstraintSet imposing a strict value of 3, the user agent should attempt to set the value of "medium" on the hardware, and and that getSettings() should return a fluxCapacitance of 0, since that is the value defined as corresponding to "medium".

No parameters.
Return type: Capabilities
getConstraints
The getConstraints method returns the Constraints that were the argument to the last successful call of applyConstraints(), maintaining the order in which they were specified. Note that some of the optional ConstraintSets returned may not be currently satisfied. To check which ConstraintSets are currently in effect, the application should use getSettings.
No parameters.
Return type: Constraints
getSettings
The getSettings() method returns the current settings of all the properties of the object, whether they are platform defaults or have been set by applyConstraints(). The values returned for the properties of the object MUST lie within the currently satisfied set of constraints applied to this object. Note that the actual setting of a property must be a single value.
No parameters.
Return type: Settings

11.1.3 applyConstraints Failure Callback

ConstraintErrorCallback
callback ConstraintErrorCallback = void (MediaStreamError error);
Callback ConstraintErrorCallback Parameters
error of type MediaStreamError
An MediaStreamError holding a required constraint that could not be satisfied.

An example of Constraints that could be passed into applyConstraints() or returned as a value of constraints is below. It uses the properties defined in the Track property registry.

Example 4
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["facingMode"]) {
  // Treat like an error.
}
var constraints = {
  "width": {
    "min": 640
  },
  "height": {
    "min": 480
  },
  "advanced": [{
      "width": 650
    }, {
      "width": {
        "min": 650
      }
    }, {
      "frameRate": 60
    }, {
      "width": {
        "max": 800
      }
    }, {
      "facingMode": "user"
    }]
};

Here is another example, specifically for a video track where I must have a particular camera and have separate preferences for the width and height:

Example 5
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["sourceId"]) {
  // Treat like an error.
}
var constraints = {
  sourceId: {"exact": "20983-20o198-109283-098-09812"},
  advanced: [{
      width: {
        min: 800,
        max: 1200
      }
    }, {
      height: {
        min: 600
      }
    }]
};

And here's one for an audio track:

Example 6
var supports = navigator.mediaDevices.getSupportedConstraints("audio");
if(!supports["sourceId"] || !supports["gain"]) {
  // Treat like an error.
}
var constraints = {
  advanced: [{
      sourceId: "64815-wi3c89-1839dk-x82-392aa"
    }, {
      gain: 0.5
    }]
};

Here's an example of use of ideal:

Example 7
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["aspectRatio"] || !supports["facingMode"]) {
  // Treat like an error.
}
navigator.mediaDevices.getUserMedia({
  "video": {
    "width": {"min": 320, "ideal": 1280, "max": 1920},
    "height": {"min": 240, "ideal": 720, "max": 1080},
    "framerate": 30,     // Shorthand for ideal.
    // "facingMode": "environment" would be optional.
    "facingMode": {"exact": "environment"}
  }}, ...);

Here's an example of "I want 720p, but I can accept up to 1080p and down to VGA.":

Example 8
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["width"] || !supports["height"]) {
  // Treat like an error.
}
navigator.mediaDevices.getUserMedia({"video": {
  "width": {"min": 640, "ideal": 1280, "max": 1920},
  "height": {"min": 480, "ideal": 720, "max": 1080},
}}, ...);

Here's an example of "I want a front-facing camera and it must be VGA.":

Example 9
var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(supports["facingMode"]) {
  navigator.mediaDevices.getUserMedia({"video": {
    "facingMode": {"exact": "user"},
    "width": {"exact": 640},
    "height": {"exact": 480}
  }}, ...);
}

11.2 The Property Registry

There is a single IANA registry that defines the constrainable properties of all objects that implement the Constrainable pattern. The registry entries must contain the name of each property along with its set of legal values. The registry entries for MediaStreamTrack are defined below. The syntax for the specification of the set of legal values depends on the type of the values. In addition to the standard atomic types (boolean, long, double, DOMString), legal values include lists of any of the atomic types, plus min-max ranges, as defined below.

List values must be interpreted as disjunctions. For example, if a property 'facingMode' for a camera is defined as having legal values ["left", "right", "user", "environment"], this means that 'facingMode' can have the value "left", the value "right", the value "environment" or the value "user". Similarly Constraints restricting 'facingMode' to ["user", "left", "right"] would mean that the UA should select a camera (or point the camera, if that is possible) so that "facingMode" is either "user", "left", or "right". This Constraint would thus request that the camera not be facing away from the user, but would allow the UA to choose among the other directions.

dictionary ConstrainDoubleRange {
    double max;
    double min;
    double exact;
    double ideal;
};

11.2.1 Dictionary ConstrainDoubleRange Members

exact of type double
The exact required value for this property.
ideal of type double
The ideal (target) value for this property.
max of type double
The maximum legal value of this property.
min of type double
The minimum value of this Property.
dictionary ConstrainLongRange {
    long max;
    long min;
    long exact;
    long ideal;
};

11.2.2 Dictionary ConstrainLongRange Members

exact of type long
The exact required value for this property.
ideal of type long
The ideal (target) value for this property.
max of type long
The maximum legal value of this property.
min of type long
The minimum value of this property.
dictionary ConstrainDOMStringParameters {
    (DOMString or sequence) exact;
    (DOMString or sequence) ideal;
};

11.2.3 Dictionary ConstrainDOMStringParameters Members

exact of type (DOMString or sequence)
The exact required value for this property.
ideal of type (DOMString or sequence)
The ideal (target) value for this property.
dictionary ConstrainVideoFacingModeParameters {
    (VideoFacingModeEnum or sequence) exact;
    (VideoFacingModeEnum or sequence) ideal;
};

11.2.4 Dictionary ConstrainVideoFacingModeParameters Members

exact of type (VideoFacingModeEnum or sequence)
The exact required value for this property.
ideal of type (VideoFacingModeEnum or sequence)
The ideal (target) value for this property.
typedef (Long or ConstrainLongRange) ConstrainLong;
typedef (Double or ConstrainDoubleRange) ConstrainDouble;
typedef undefined ;
Throughout this specification, the identifier is used to refer to the undefined type.
typedef undefined ;
Throughout this specification, the identifier is used to refer to the undefined type.

11.3 Capabilities

Capabilities are dictionary containing one or more key-value pairs, where each key must be a constrainable property defined in the registry, and each value must be a subset of the set of values defined for that property in the registry. The exact syntax of the value expression depends on the type of the property but is of type ConstraintValues. The Capabilities dictionary specifies the subset of the constrainable properties and values from the registry that the UA supports. Note that a UA may support only a subset of the properties that are defined in the registry, and may support a subset of the set values for those properties that it does support. Note that Capabilities are returned from the UA to the application, and cannot be specified by the application. However, the application can control the Settings that the UA chooses for Capabilities by means of Constraints.

An example of a Capabilities dictionary is shown below. This example is not very realistic in that a browser would actually be required to support more settings that just these.

Example 10
{
  "frameRate": {
    "min": 1.0,
    "max": 60.0
  },
  "facingMode": ["user", "environment"]
}

11.4 Settings

A Setting is a dictionary containing one or more key-value pairs. It must contain each key returned in getCapabilities(). There must be a single value for each key and the value must a member of the set defined for that property by capabilities(). The Settings dictionary contains the actual values that the UA has chosen for the object's Capabilities. The exact syntax of the value depends on the type of the property.

An example of a Setting dictionary is shown below. This example is not very realistic in that a browser would actually be required to support more settings that just these.

Example 11
{
  "frameRate": 30.0,
  "facingMode": "user"
}

11.5 Constraints and ConstraintSet

Due to the limitiations of WebIDL, interfaces implementing the Constrainable Pattern cannnot simply subclass Constraints and ConstraintSet as they are defined here. Instead they must provide their own definitions that follow this pattern. See MediaTrackConstraints for an example of this.

typedef Dictionary ConstraintSet;

Each member of a ConstraintSet corresponds to a Capability and specifies a subset of its legal values. Applying a ConstraintSet instructs that UA to restrict the setting of the corresponding Capabilities to the specified values or ranges of values. A given property MAY occur both in the basic Constraints set and in the advanced ConstraintSets list, and MAY occur at most once in each ConstraintSet in the advanced list.

dictionary Constraints : ConstraintSet {
    sequence<ConstraintSet> advanced;
};

11.5.1 Dictionary Constraints Members

advanced of type sequence<ConstraintSet>

The list of ConstraintSets that the UA must attempt to satisfy, in order, skipping only those that cannot be satisfied. The order of these ConstraintSets is significant. In particular, when they are passed as an argument to applyConstraints, the UA must try to satisfy them in the order that is specified. Thus if optional ConstraintSets C1 and C2 can be satisfied individually, but not together, then whichever of C1 and C2 is first in this list will be satisfied, and the other will not. The UA must attempt to satisfy all optional ConstraintSets in the list, even if some cannot be satisfied. Thus, in the preceding example, if optional constraint C3 is specified after C1 and C2, the UA will attempt to satisfy C3 even though C2 cannot be satisfied. Note that a given property name may occur only once in each ConstraintSet but may occur in more than one ConstraintSet.

12. Examples

This sample code exposes a button. When clicked, the button is disabled and the user is prompted to offer a stream. The user can cause the button to be re-enabled by providing a stream (e.g., giving the page access to the local camera) and then disabling the stream (e.g., revoking that access).

Example 12
<input type="button" value="Start" onclick="start()" id="startBtn">
<script>
 var startBtn = document.getElementById('startBtn');

 function start() {
     navigator.mediaDevices.getUserMedia({
         audio: true,
         video: true
     }, gotStream, logError);
     startBtn.disabled = true;
 }

 function gotStream(stream) {
     stream.oninactive = function () {
         startBtn.disabled = false;
     };
 }

 function logError(error) {
     log(error.name + ": " + error.message);
 }
</script>

This example allows people to take photos of themselves from the local video camera. Note that the forthcoming Image Capture specification may provide a simpler way to accomplish this.

Example 13
<article>
 <style scoped>
  video { transform: scaleX(-1); }
  p { text-align: center; }
 </style>
 <h1>Snapshot Kiosk</h1>
 <section id="splash">
  <p id="errorMessage">Loading...</p>
 </section>
 <section id="app" hidden>
  <p><video id="monitor" autoplay></video> <canvas id="photo"></canvas>
  <p><input type=button value="&#x1F4F7;" onclick="snapshot()">
 </section>
 <script>
 navigator.mediaDevices.getUserMedia({
     video: true
 }, gotStream, noStream);
 var video = document.getElementById('monitor');
 var canvas = document.getElementById('photo');

 function gotStream(stream) {
     video.srcObject = stream;
     stream.oninactive = noStream;
     video.onloadedmetadata = function () {
         canvas.width = video.videoWidth;
         canvas.height = video.videoHeight;
         document.getElementById('splash').hidden = true;
         document.getElementById('app').hidden = false;
     };
 }

 function noStream() {
     document.getElementById('errorMessage').textContent = 'No camera available.';
 }

 function snapshot() {
     canvas.getContext('2d').drawImage(video, 0, 0);
 }
 </script>
</article>

13. Error Names

This specification defines the following new error names

14. IANA Registrations

14.1 Track Property Registrations

IANA is requested to register the following properties as specified in [RTCWEB-CONSTRAINTS]:

The following constraint names are defined to apply to both video and audio MediaStreamTrack objects:

Property Name Values Notes
sourceType SourceTypeEnum The type of the source of the MediaStreamTrack. Note that the setting of this property is uniquely determined by the source that is attached to the Track. In particular, getCapabilities() will return only a single value for sourceID/Type. This property can therefore be used for initial media selection with getUserMedia(). However is not useful for subsequent media control with applyConstraints, since any attempt to set a different value will result in an unsatisfiable ConstraintSet. .
sourceId DOMString The application-unique identifier for this source. The same identifier MUST be valid between sessions of this application, but MUST also be different for other applications. Some sort of GUID is recommended for the identifier. Note that the setting of this property is uniquely determined by the source that is attached to the Track. In particular, getCapabilities() will return only a single value for sourceID/Type. This property can therefore be used for initial media selection with getUserMedia(). However is not useful for subsequent media control with applyConstraints, since any attempt to set a different value will result in an unsatisfiable ConstraintSet.

The following properties are defined to apply only to video MediaStreamTrack objects:

Property Name Values Notes
width ConstrainLong The width or width range, in pixels, of the video source. As a capability, the range should span the video source's pre-set width values with min being the smallest width and max being the largest width.
height ConstrainLong The height or height range, in pixels, of the video source. As a capability, the range should span the video source's pre-set height values with min being the smallest height and max being the largest height.
frameRate ConstrainDouble The exact desired frame rate (frames per second) or frameRate range of the video source. If the source does not natively provide a frameRate, or the frameRate cannot be determined from the source stream, then this value MUST refer to the user agent's vsync display rate.
aspectRatio ConstrainDouble The exact aspect ratio (width in pixels divided by height in pixels), represented as a double rounded to the tenth decimal place.
facingMode ConstrainDOMString The members of the enum describe the directions that the camera can face, as seen from the user's perspective. Valid values for the strings in the ConstrainDOMString are the values of enum VideoFacingModeEnum.
enum VideoFacingModeEnum {
    "user",
    "environment",
    "left",
    "right"
};
Enumeration description
userThe source is facing toward the user (a self-view camera).
environmentThe source is facing away from the user (viewing the environment).
leftThe source is facing to the left of the user.
rightThe source is facing to the right of the user.

Below is an illustration of the video facing modes in relation to the user.
Illustration of video facing modes in relation to user

The following properties are defined to apply only to audio MediaStreamTrack objects:

Property Name Values Notes
volume ConstrainDouble The volume or volume range of the audio source, as a percentage. A volume of 0.0 is silence, while a volume of 1.0 is the maximum supported volume. Note that any ConstraintSet that specifies values outside of this range can never be satisfied.
sampleRate ConstrainLong The sample rate in samples per second for the audio data.
sampleSize ConstrainLong The linear sample size in bits. This constraint can only be satisfied for audio devices that produce linear samples.
echoCancelation boolean When one or more audio streams is being played in the proceses of varios microphones, it is often desirable to attempt to remove the sound being played from the input signals recorded by the microphones. This is referred to echo cancelation. There are cases where it is not needed and it is desirable to turn it off so that no audio artifacts are introduced. This allows applications to control this behavior.
Note
Open Issue: volume seems like better as double, or even better as double in dB. what value is it set at if one wants it half as loud?

15. Change Log

This section will be removed before publication.

Changes since May 7, 2014

  1. Clarified that skipping of optional/advanced ConstraintSets is only permitted if they cannot be satisfied, not merely because the user agent wishes to.
  2. Bug 25855: Clarification about conformance requirements phrased as algorithms
  3. Bug 25803: Mark section entitled "The model: sources, sinks, constraints, and settings" as non-normative
  4. Bug 24015: Add callback to indicate when available media devices change (introduced Navigator.mediaDevices)
  5. Bug 25860: make sure we have a bug to have a getTracks that gives you all the tracks
  6. Bug 25884: applied constraint syntax consensus as realized in June 9 WG email from Peter Thatcher.
  7. Moved getSupportedConstraints() method to MediaDevices object.
  8. Added stricter requirements on the getSupportedConstraints() return value.
  9. Added issue note in Constrainable Pattern section that ideal is not yet defined.
  10. Added issue note for applyConstraints that how multiple unorderedConstraints are to be satisfied together is not yet defined.
  11. Added informative notes that WebIDL discards unknown required properties and that application authors need to use the getSupportedConstraints() method.
  12. Cleaned up the MediaStream API intro section (mainly MediaStream behavior that have moved to MediaStreamTrack).
  13. The concept of MediaStreamTrack with a detachable source is now used throughout the spec (removed language saying that a MST could be disassociated from its track).

Changes since March 21, 2014

  1. New webIDL for Constrainable and Constraints.
  2. Bug 24931: changed MediaError to MediaStreamError.
  3. Bug 23817: [Nit] Redundant TOC headers 8.1 & 9.1
  4. Bug 25230: readyState attribute must be inherited while cloning a MediaStreamTrack
  5. Bug 25249: Source should be detached when a MediaStreamTrack stops for any reason other than stop
  6. Updated Event Summary section to match the spec regarding MediaStreamTrack.stop() (as discussed in bug 25248)
  7. Made the MediaStream() constructor behave like addTrack() WRT adding ended tracks (as discussed in bug 25250).
  8. Bug 25262: MediaStream Constructor algorithm must also check for MediaStreamTracks "ended" state while initializing "active" state.
  9. Bug 25276: Initialization for VideoTrack.selected attribute is missing while specifying steps for "Loading and Playing a MediaStream in a Media Element"
  10. Changed syntax of constraints to use 'require' and 'advanced' and support non-required, non-advanced constraints.
  11. Bug 25360: MediaStreamTrack should not be considered as ended just because remote peer stopped sending data.
  12. Bug 25275: VideoTrackList.selectedIndex initialization conflicts with HTML5 spec, "if no track is selected".
  13. Removed mentioning of MediaStream received from other peer (as discussed in bug 25361).
  14. Bug 22263: Clarify synchronization of tracks in a MediaStream
  15. Bug 25441: Overconstrained muted state should not link with MediaStreamTrack.readyState

Changes since February 18, 2014

  1. Bug 24928: Remove MediaStream state check from addTrack() algorithm.
  2. Bug 24930: Remove MediaStream state check from the removeTrack() algorithm.
  3. Added native settings to tracks.
  4. Removed videoMediaStreamTrack and audioMediaStreamTrack since they are no longer necessary.

Changes since December 25, 2013

  1. Make optional constraints a list of ConstraintSets. Make ConstraintSet an object.
  2. Remove noaccess, move peerIdentity
  3. Add constraints for sampleRate, sampleSize, and echoCancelation.
  4. Aligned text in remainder of document with Constrainable changes.
  5. Removed statements that constraints are not applied to read-only sources

Changes since November 5, 2013

  1. ACTION-25: Switch mediastream.inactive to mediastream.active.
  2. ACTION-26: Rewrite stop to only detach the track's source.
  3. Bug 22338: Arbitrary changing of tracks.
  4. Bug 23125: Use double rather than float.
  5. Bug 22712: VideoFacingMode enum needs an illustration.
  6. Moved constraints into a separate Constrainable interface.
  7. Created a separate section on error handling.

Changes since October 17, 2013

  1. Bug 23263: Add output device enumeration to GetSources
  2. Introduced the Constrainable interface.
  3. Change consensus note on constraints in IANA section.
  4. Removed createObjectURL.
  5. Bug 22209: Should not use MUST requirements on values provided by the developer.

Changes since August 24, 2013

  1. Bug 22269: Renamed getSourceInfos() to getSources() and made the result async.
  2. Bug 22229: Editorial input
  3. Bug 22243: Clarify readonly track
  4. Bug 22259: Disabled mediastreamtrack and state of media element
  5. Bug 22226: Remove check of same source from MediaStream constructor algorithm
  6. Replaced ended with inactive for MediaStream (resolves bug 21618).
  7. Bug 22264: MediaStream.ended set to true on creation
  8. Bug 22272: Permission revocation via MediaStreamTrack.stop()
  9. Bug 22248: Relationship between MediaStreamTrack and HTML5 VideoTrack/AudioTrack after MediaStream assignment
  10. Bug 22247: Setting loop attribute on a media element reading from a MediaStream

Changes since July 4, 2013

  1. Bug 21967: Added paragraph on MediaStreamTrack enabled state and updated cloning algorithm.
  2. Bug 22210: Make getUserMedia() algorithm use all numbered items.
  3. Bug 22250: Fixed accidentally overridden error.
  4. Bug 22211: Added async error when no valid media type is requested.
  5. Bug 22216: Made NavigatorUserMediaError extend DOMError.
  6. Bug 22249: Throw on attempts to set currentTime on media elements playing MediaStream objects.
  7. Bug 22246: Made media.buffered have length 0.
  8. Bug 22692: Updated media element to use HAVE_NOTHING state before media arrives on the played MediaStream and HAVE_ENOUGH_DATA as soon as media arrives.

May 29, 2013

  1. Bug 22252: fixed usage of MUST in MediaStream() constructor description.
  2. Bug 22215: made MediaStream.ended readonly.
  3. Bug 21967: clarified MediaStreamTrack.enabled state initial value.
  4. Added aspectRatio constraint, capability, and state.
  5. Updated usage of MediaStreams in media elements.

May 15, 2013

  1. Added explanatory section for constraints, capabilities, and states.
  2. Added VideoFacingModeEnum (including left and right options).
  3. Added getSourceInfos() and SourceInfo dictionary.
  4. Added isolated streams.

April 29, 2013

  1. Removed remaining photo APIs and references (since we have a separate Image Capture Spec).

March 20, 2013

  1. Added readonly and remote attributes to MediaStreamTrack
  2. Removed getConstraint(), setConstraint(), appendConstraint(), and prependConstraint().
  3. Added source states. Added states() method on tracks. Moved sourceType and sourceId to be states.
  4. Added source capabilities. Added capabilities() method on tracks.
  5. Added clarifying text about MediaStreamTrack lifecycle and mediaflow.
  6. Made MediaStreamTrack cloning explicit.
  7. Removed takePhoto() and friends from VideoStreamTrack (we have a separate Image Capture Spec).
  8. Made getUserMedia() error callback mandatory.

December 12, 2012

  1. Changed error code to be string instead of number.
  2. Added core of settings proposal allowing for constraint changes after stream/track creation.

November 15 2012

  1. Introduced new representation of tracks in a stream (removed MediaStreamTrackList).
  2. Updated MediaStreamTrack.readyState to use an enum type (instad of unsigned short constants).
  3. Renamed MediaStream.label to MediaStream.id (the definition needs some more work).

October 1 2012

  1. Limited the track kind values to "audio" and "video" only (could previously be user defined as well).
  2. Made MediaStream extend EventTarget.
  3. Simplified the MediaStream constructor.

June 23 2012

  1. Rename title to "Media Capture and Streams".
  2. Update document to comply with HTML5.
  3. Update image describing a MediaStream.
  4. Add known issues and various other editorial changes.

June 22 2012

  1. Update wording for constraints algorithm.

June 19 2012

  1. Added "Media Streams as Media Elements section".

June 12 2012

  1. Switch to respec v3.

June 5 2012

  1. Added non-normative section "Implementation Suggestions".
  2. Removed stray whitespace.

June 1 2012

  1. Added media constraint algorithm.

Apr 23 2012

  1. Remove MediaStreamRecorder.

Apr 20 2012

  1. Add definitions of MediaStreams and related objects.

Dec 21 2011

  1. Changed to make wanted media opt in (rather than opt out). Minor edits.

Nov 29 2011

  1. Changed examples to use MediaStreamOptions objects rather than strings. Minor edits.

Nov 15 2011

  1. Removed MediaStream stuff. Refers to webrtc 1.0 spec for that part instead.

Nov 9 2011

  1. Created first version by copying the webrtc spec and ripping out stuff. Put it on github.

A. Acknowledgements

The editors wish to thank the Working Group chairs and Team Contact, Harald Alvestrand, Stefan Håkansson and Dominique Hazaël-Massieux, for their support. Substantial text in this specification was provided by many people including Jim Barnett, Harald Alvestrand, Travis Leithead, and Stefan Håkansson.

B. References

B.1 Normative references

[HTML5]
Robin Berjon; Steve Faulkner; Travis Leithead; Erika Doyle Navara; Edward O'Connor; Silvia Pfeiffer. HTML5. 29 April 2014. W3C Candidate Recommendation. URL: http://www.w3.org/TR/html5/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Best Current Practice. URL: http://www.ietf.org/rfc/rfc2119.txt
[RTCWEB-CONSTRAINTS]
Dan Burnett. IANA Registry for RTCWeb Constrainable Properties. 21 October 2013. Active Internet-Draft. URL: http://datatracker.ietf.org/doc/draft-burnett-rtcweb-constraints-registry/
[WEBIDL]
Cameron McCormack. Web IDL. 19 April 2012. W3C Candidate Recommendation. URL: http://www.w3.org/TR/WebIDL/
[WEBRTC10]
Adam Bergkvist; Daniel Burnett; Cullen Jennings; Anant Narayanan. WebRTC 1.0: Real-time Communication Between Browsers. 10 September 2013. W3C Working Draft. URL: http://www.w3.org/TR/webrtc/

B.2 Informative references

[mediastream-imagecap]
Giridhar Mandyam. MediaStream Image Capture. W3C Working Draft. URL: http://www.w3.org/TR/image-capture/
[mediastream-rec]
Jim Barnett; Travis Leithead. MediaStream Recording. W3C Working Draft. URL: https://dvcs.w3.org/hg/dap/raw-file/tip/media-stream-capture/MediaRecorder.html