Abstract

This document defines a set of JavaScript APIs that allow local media, including audio and video, to be requested from a platform.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is not complete. It is subject to major changes and, while early experimentations are encouraged, it is therefore not intended for implementation. The API is based on preliminary work done in the WHATWG. The Media Capture Task Force expects this specification to evolve significantly based on:

This document was published by the Web Real-Time Communication Working Group and Device APIs Working Group as an Editor's Draft. If you wish to make comments regarding this document, please send them to public-media-capture@w3.org (subscribe, archives). All comments are welcome.

Publication as an Editor's Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures (Web Real-Time Communication Working Group, Device APIs Working Group) made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

1. Introduction

This section is non-normative.

Access to multimedia streams (video, audio, or both) from local devices (video cameras, microphones, Web cams) can have a number of uses, such as real-time communication, recording, and surveillance.

This document defines the APIs used to get access to local devices that can generate multimedia stream data. This document also defines the stream API by which JavaScript is able to manipulate the stream data or otherwise process it.

2. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this specification are to be interpreted as described in [RFC2119].

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Implementations that use ECMAScript to implement the APIs defined in this specification must implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL], as this specification uses that specification and terminology.

3. Terminology

HTML Terms:

The EventHandler interface represents a callback used for event handlers as defined in [HTML5].

The concepts queue a task and fires a simple event are defined in [HTML5].

The terms event handlers and event handler event types are defined in [HTML5].

source

A source is the "thing" providing the source of a media stream track. The source is the broadcaster of the media itself. A source can be a physical webcam, microphone, local video or audio file from the user's hard drive, network resource, or static image.

Some sources have an identifier which must be unique to the application (un-guessable by another application) and persistent between application sessions (e.g., the identifier for a given source device/application must stay the same, but not be guessable by another application). Sources that must have an identifier are camera and microphone sources; local file sources are not required to have an identifier. Source identifiers let the application save, identify the availability of, and directly request specific sources.

Other than the identifier, other bits of source identity are never directly available to the application until the user agent connects a source to a track. Once a source has been "released" to the application (either via a permissions UI, pre-configured allow-list, or some other release mechanism) the application will be able discover additional source-specific capabilities.

Sources do not have constraints -- tracks have constraints. When a source is connected to a track, it must conform to the constraints present on that track (or set of tracks).

Sources will be released (un-attached) from a track when the track is ended for any reason.

On the MediaStreamTrack object, sources are represented by a sourceType attribute. The behavior of APIs associated with the source's capabilities and state change depending on the source type.

Sources have capabilities and state. The capabilities and state are "owned" by the source and are common to any (multiple) tracks that happen to be using the same source (e.g., if two different tracks objects bound to the same source ask for the same capability or state information, they will get back the same answer).

State (Source State)

State refers to the immediate, current value of the source's (optionally constrained) capabilities. State is always read-only.

A source's state can change dynamically over time due to environmental conditions, sink configurations, or constraint changes. A source's state must always conform to the current set of mandatory constraints that all of the tracks it is bound to have defined, and should do its best to conform to the set of optional constraints specified.

A source's state is directly exposed to audio and video track objects through individual read-only attributes. These attributes share the same name as their corresponding capabilities and constraints.

Events are available that signal to the application that source state has changed.

A conforming user-agent must support all the state names defined in this spec.

Capabilities

Source capabilities are the intrinsic "features" of a source object. For each source state, there is a corresponding capability that describes whether it is supported by the source and if so, what the range of supported values are. Capabilities are expressed as either a series of states (for enumerated-type capabilities) or as a min/max range.

The values of the supported capabilities must be normalized to the ranges and enumerated types defined in this specification.

Capabilities return the same underlying per-source capabilities, regardless of any user-supplied constraints present on the source (capabilities are independent of constraints).

Source capabilities are effectively constant. Applications should be able to depend on a specific source having the same capabilities for any session.

Constraints

Constraints are an optional feature for restricting the range of allowed variability on a source. Without provided constraints, implementations are free to select a source's state from the full range of its supported capabilities, and to adjust that state at any time for any reason.

Constraints may be optional or mandatory. Optional constraints are represented by an ordered list, mandatory constraints are an unordered set. The order of the optional constraints is from most important (at the head of the list) to least important (at the tail of the list).

Constraints are stored on the track object, not the source. Each track can be optionally initialized with constraints, or constraints can be added afterward through the constraint APIs defined in this spec.

Applying track level constraints to a source is conditional based on the type of source. For example, read-only sources will ignore any specified constraints on the track.

It is possible for two tracks that share a unique source to apply contradictory constraints. Under such contradictions, the implementation will mute both tracks and notify them that they are over-constrained.

Events are available that allow the application to know when constraints cannot be met by the user agent. These typically occur when the application applies constraints beyond the capability of a source, contradictory constraints, or in some cases when a source cannot sustain itself in over-constrained scenarios (overheating, etc.).

Constraints that are intended for video sources will be ignored by audio sources and vice-versa. Similarly, constraints that are not recognized will be preserved in the constraint structure, but ignored by the application. This will allow future constraints to be defined in a backward compatible manner.

A correspondingly-named constraint exists for each corresponding source state name and capability name. In general, user agents will have more flexibility to optimize the media streaming experience the fewer constraints are applied.

4. Stream API

4.1 Introduction

The MediaStream interface is used to represent streams of media data, typically (but not necessarily) of audio and/or video content, e.g. from a local camera. The data from a MediaStream object does not necessarily have a canonical binary form; for example, it could just be "the video currently coming from the user's video camera". This allows user agents to manipulate media streams in whatever fashion is most suitable on the user's platform.

Each MediaStream object can contain zero or more tracks, in particular audio and video tracks. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStreams do not need to be synchronized.

Each track in a MediaStream object has a corresponding MediaStreamTrack object.

A MediaStreamTrack represents content comprising one or more channels, where the channels have a defined well known relationship to each other (such as a stereo or 5.1 audio signal).

A channel is the smallest unit considered in this API specification.

A MediaStream object has an input and an output. The input depends on how the object was created: a MediaStream object generated by a getUserMedia() call (which is described later in this document), for instance, might take its input from the user's local camera. The output of the object controls how the object is used, e.g., what is saved if the object is written to a file or what is displayed if the object is used in a video element.

Each track in a MediaStream object can be disabled, meaning that it is muted in the object's output. All tracks are initially enabled.

A MediaStream can be finished, indicating that its inputs have forever stopped providing data.

The output of a MediaStream object MUST correspond to the tracks in its input. Muted audio tracks MUST be replaced with silence. Muted video tracks MUST be replaced with blackness.

A new MediaStream object can be created from accessible media sources (that does not require any additional permissions) using the MediaStream() constructor. The constructor argument can either be an existing MediaStream object, in which case all the tracks of the given stream are added to the new MediaStream object, or an array of MediaStreamTrack objects. The latter from makes it possible to compose a stream from different source streams.

A MediaStream

Both MediaStream and MediaStreamTrack objects can be cloned. This allows for greater control since the separate instances can be manipulated and consumed individually. A cloned MediaStream contains clones of all member tracks from the original stream.

When a MediaStream object is being generated from a local file (as opposed to a live audio/video source), the user agent SHOULD stream the data from the file in real time, not all at once. The MediaStream object is also used in contexts outside getUserMedia, such as [WEBRTC10]. In both cases, ensuring a realtime stream reduces the ease with which pages can distinguish live video from pre-recorded video, which can help protect the user's privacy.

4.2 MediaStream

The MediaStream() constructor composes a new stream out of existing tracks. It takes an optional argument of type MediaStream or an array of MediaStreamTrack objects. When the constructor is invoked, the UA must run the following steps:

  1. Let stream be a newly constructed MediaStream object.

  2. Initialize stream's id attribute to a newly generated value.

  3. If the constructor's argument is present, run the sub steps that corresponds to the argument type.

  4. Return stream.

A MediaStream can have multiple audio and video sources (e.g. because the user has multiple microphones, or because the real source of the stream is a media resource with many media tracks). The stream represented by a MediaStream thus has zero or more tracks.

The tracks of a MediaStream are stored in a track set. The track set MUST contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of the tracks in the set is user agent defined and the API will never put any requirements on the order. The proper way to find a specific MediaStreamTrack object in the set is to look it up by its id.

An object that reads data from the output of a MediaStream is referred to as a MediaStream consumer. The list of MediaStream consumers currently includes the media elements and the PeerConnection API specified in [WEBRTC10].

Note

MediaStream consumers must be able to handle tracks being added and removed. This behavior is specified per consumer.

A MediaStream object is said to be finished when all tracks belonging to the stream have ended. When this happens for any reason other than the stop() method being invoked, the user agent MUST queue a task that runs the following steps:

  1. If the object's ended attribute has the value true already, then abort these steps. (The stop() method was probably called just before the stream stopped for other reasons, e.g. the user clicked an in-page stop button and then the user agent provided stop button.)

  2. Set the object's ended attribute to true.

  3. Fire a simple event named ended at the object.

If the end of the stream was reached due to a user request, the task source for this task is the user interaction task source. Otherwise the task source for this task is the networking task source.

typedef sequence<MediaStreamTrack> MediaStreamTrackSequence;
Throughout this specification, the identifier MediaStreamTrackSequence is used to refer to the sequence<MediaStreamTrack> type.
[Constructor,
        Constructor (MediaStream stream),
        Constructor (MediaStreamTrackSequence tracks)]
interface MediaStream : EventTarget {
    readonly    attribute DOMString    id;
    sequence<MediaStreamTrack> getAudioTracks ();
    sequence<MediaStreamTrack> getVideoTracks ();
    MediaStreamTrack?          getTrackById (DOMString trackId);
    void                       addTrack (MediaStreamTrack track);
    void                       removeTrack (MediaStreamTrack track);
    MediaStream                clone ();
    readonly    attribute boolean      ended;
                attribute EventHandler onended;
                attribute EventHandler onaddtrack;
                attribute EventHandler onremovetrack;
};

4.2.1 Attributes

ended of type boolean, readonly

The MediaStream.ended attribute MUST return true if the MediaStream has finished, and false otherwise.

When a MediaStream object is created, its ended attribute MUST be set to false, unless it is being created using the MediaStream() constructor whose arguments are lists of MediaStreamTrack objects that are all ended, in which case the MediaStream object MUST be created with its ended attribute set to true.

id of type DOMString, readonly

When a MediaStream object is created, the user agent MUST generate a globally unique identifier string, and MUST initialize the object's id attribute to that string. Such strings MUST only use characters in the ranges U+0021, U+0023 to U+0027, U+002A to U+002B, U+002D to U+002E, U+0030 to U+0039, U+0041 to U+005A, U+005E to U+007E, and MUST be 36 characters long.

The id attribute MUST return the value to which it was initialized when the object was created.

onaddtrack of type EventHandler,
This event handler, of type addtrack, MUST be supported by all objects implementing the MediaStream interface.
onended of type EventHandler,
This event handler, of type ended, MUST be supported by all objects implementing the MediaStream interface.
onremovetrack of type EventHandler,
This event handler, of type removetrack, MUST be supported by all objects implementing the MediaStream interface.

4.2.2 Methods

addTrack

Adds the given MediaStreamTrack to this MediaStream.

When the addTrack() method is invoked, the user agent MUST run the following steps:

  1. Let track be the MediaStreamTrack argument and stream this MediaStream object.

  2. If stream is finished, throw an INVALID_STATE_ERR exception.

  3. If track is already in stream's track set, then abort these steps.

  4. Add track to stream's track set.

ParameterTypeNullableOptionalDescription
trackMediaStreamTrack
Return type: void
clone

Clones the given MediaStream and all its tracks.

When the clone() method is invoked, the user agent MUST run the following steps:

  1. Let streamClone be a newly constructed MediaStream object.

  2. Initialize streamClone's id attribute to a newly generated value.

  3. Let trackSetClone be a list that contains the result of running MediaStreamTrack.clone() on all the tracks in this stream.

  4. Let trackSetClone be streamClone's track set.

No parameters.
Return type: MediaStream
getAudioTracks

Returns a sequence of MediaStreamTrack objects representing the audio tracks in this stream.

The getAudioTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "audio". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

No parameters.
Return type: sequence<MediaStreamTrack>
getTrackById

The getTrackById() method MUST return the first MediaStreamTrack object in this stream's track set whose id is equal to trackId. The method MUST return null if no track matches the trackId argument.

ParameterTypeNullableOptionalDescription
trackIdDOMString
Return type: MediaStreamTrack, nullable
getVideoTracks

Returns a sequence of MediaStreamTrack objects representing the video tracks in this stream.

The getVideoTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "video". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

No parameters.
Return type: sequence<MediaStreamTrack>
removeTrack

Removes the given MediaStreamTrack from this MediaStream.

When the removeTrack() method is invoked, the user agent MUST run the following steps:

  1. Let track be the MediaStreamTrack argument and stream this MediaStream object.

  2. If stream is finished, throw an INVALID_STATE_ERR exception.

  3. If track is in stream's track set, remove it.

ParameterTypeNullableOptionalDescription
trackMediaStreamTrack
Return type: void

4.3 MediaStreamTrack

A MediaStreamTrack object represents a media source in the user agent. Several MediaStreamTrack objects can represent the same media source, e.g., when the user chooses the same camera in the UI shown by two consecutive calls to getUserMedia() .

Note that a web application can revoke all given permissions with MediaStreamTrack.stop().

4.3.1 Life-cycle and Media Flow

A MediaStreamTrack has three stages in its lifecycle. A track begins as new prior to being connected to an active source.

Once connected, the started event fires and the track becomes live. In the live state, the track is active and media is available for rendering at a MediaStream consumer.

A muted or disabled MediaStreamTrack renders either silence (audio), black frames (video), or a zero-information-content equivalent. The muted/unmuted state of a track reflects if the source provides any media at this moment. The enabled/disabled state is under application control and determines if the track outputs media (to its consumers). Hence, media from the source only flows when a MediaStreamTrack object is both unmuted and enabled.

A MediaStreamTrack is muted when the source is temporarily unable to provide the track with data. A track can be muted by a user. Often this action is outside the control of the application. This could be as a result of the user hitting a hardware switch, or toggling a control in the operating system or browser chrome. A track can also be muted by the user agent. For example, a track that is a member of a MediaStream, received via a RTCPeerConnection [WEBRTC10], is muted if the application on the other side disables the corresponding track in the MediaStream being sent.

Applications are able to enable or disable a MediaStreamTrack to prevent it from rendering media from the source. A muted track will however, regardless of the enabled state, render silence and blackness. A disabled track is logically equivalent to a muted track, from a consumer point of view.

For a newly created MediaStreamTrack object, the following applies. The track is always enabled unless stated otherwise (for examlpe when cloned) and the muted state reflects the state of the source at the time the track is created.

A MediaStreamTrack object is said to end when the source of the track is disconnected or exhausted.

When a MediaStreamTrack object ends for any reason (e.g., because the user rescinds the permission for the page to use the local camera, or because the data comes from a finite file and the file's end has been reached and the user has not requested that it be looped, or because the UA has instructed the track to end for any reason, or because the reference count of the track's underlying media source has reached zero), it is said to be ended. When track instance track ends for any reason other than the stop() method being invoked on the MediaStreamTrack object that represents track, the user agent MUST queue a task that runs the following steps:

  1. If the track's readyState attribute has the value ended already, then abort these steps.

  2. Set track's readyState attribute to ended.

  3. Fire a simple event named ended at the object.

If the end of the stream was reached due to a user request, the event source for this event is the user interaction event source.

4.3.2 Tracks and Constraints

Constraints are independent of sources. However, if the sourceType is "none", the readonly attribute is true, or the remote attribute is true, the track's constraints will not be applied to the source.

Whether MediaTrackConstraints were provided at track initialization time or need to be established later at runtime, the APIs defined below allow the retrieval and manipulation of the constraints currently established on a track.

Each track maintains an internal version of the MediaTrackConstraints structure, namely a mandatory set of constraints (no duplicates), and an optional ordered list of individual constraint objects (may contain duplicates). The internal stored constraint structure is only exposed to the application using the existing MediaTrackConstraints, MediaTrackConstraintSet, MediaTrackConstraint, and similarly-derived-type dictionary objects.

When track constraints change, a user agent must queue a task to evaluate those changes when the task queue is next serviced. Similarly, if the sourceType changes, then the user agent should perform the same actions to re-evaluate the constraints of each track affected by that source change.

4.3.3 Interface Definition

interface MediaStreamTrack : EventTarget {
    readonly    attribute DOMString             kind;
    readonly    attribute DOMString             id;
    readonly    attribute DOMString             label;
                attribute boolean               enabled;
    readonly    attribute boolean               muted;
                attribute EventHandler          onmute;
                attribute EventHandler          onunmute;
    readonly    attribute boolean               _readonly;
    readonly    attribute boolean               remote;
    readonly    attribute MediaStreamTrackState readyState;
                attribute EventHandler          onstarted;
                attribute EventHandler          onended;
    static sequence<SourceInfo>                    getSourceInfos ();
    MediaTrackConstraints?                         constraints ();
    MediaSourceStates                              states ();
    (AllVideoCapabilities or AllAudioCapabilities) capabilities ();
    void                                           applyConstraints (MediaTrackConstraints constraints);
                attribute EventHandler          onoverconstrained;
    MediaStreamTrack                               clone ();
    void                                           stop ();
};
4.3.3.1 Attributes
enabled of type boolean,

The MediaStreamTrack.enabled attribute, on getting, MUST return the last value to which it was set. On setting, it MUST be set to the new value, and then, if the MediaStreamTrack object is still associated with a track, MUST enable the track if the new value is true, and disable it otherwise.

Note

Thus, after a MediaStreamTrack is disassociated from its track, its enabled attribute still changes value when set; it just doesn't do anything with that new value.

id of type DOMString, readonly

Unless a MediaStreamTrack object is created as a part a of special purpose algorithm that specifies how the track id must be initialized, the user agent MUST generate a globally unique identifier string and initialize the object's id attribute to that string.

An example of an algorithm that specifies how the track id must be initialized is the algorithm to represent an incoming network component with a MediaStreamTrack object. [WEBRTC10]

MediaStreamTrack.id attribute MUST return the value to which it was initialized when the object was created.

kind of type DOMString, readonly

The MediaStreamTrack.kind attribute MUST return the string "audio" if the object represents an audio track or "video" if object represents a video track.

label of type DOMString, readonly

User agents MAY label audio and video sources (e.g., "Internal microphone" or "External USB Webcam"). The MediaStreamTrack.label attribute MUST return the label of the object's corresponding track, if any. If the corresponding track has or had no label, the attribute MUST instead return the empty string.

Note

Thus the kind and label attributes do not change value, even if the MediaStreamTrack object is disassociated from its corresponding track.

muted of type boolean, readonly

The MediaStreamTrack.muted attribute MUST return true if the track is muted, and false otherwise.

onended of type EventHandler,
This event handler, of type ended, MUST be supported by all objects implementing the MediaStreamTrack interface.
onmute of type EventHandler,
This event handler, of type mute, MUST be supported by all objects implementing the MediaStreamTrack interface.
onoverconstrained of type EventHandler,
This event handler, of type overconstrained, MUST be supported by all objects implementing the MediaStreamTrack interface.
onstarted of type EventHandler,
This event handler, of type started, MUST be supported by all objects implementing the MediaStreamTrack interface.
onunmute of type EventHandler,
This event handler, of type unmute, MUST be supported by all objects implementing the MediaStreamTrack interface.
readonly of type boolean, readonly
If the track (audio or video) is backed by a read-only source such as a file, or the track source is a local microphone or camera, but is shared so that this track cannot modify any of the source's settings, the readonly attribute MUST return the value true. Otherwise, it must return the value false.
readyState of type MediaStreamTrackState, readonly

The readyState attribute represents the state of the track. It MUST return the value to which the user agent last set it.

remote of type boolean, readonly
If the track is sourced by an RTCPeerConnection, the remote attribute MUST return the value true. Otherwise, it must return the value false.
4.3.3.2 Methods
applyConstraints

This API will replace all existing constraints with the provided constraints (if existing constraints exist). Otherwise, it will apply the newly provided constraints to the track.

ParameterTypeNullableOptionalDescription
constraintsMediaTrackConstraintsA new constraint structure to apply to this track.
Return type: void
capabilities

Returns a dictionary with all of the capabilities for the track type. If the track type is VideoStreamTrack, the AllVideoCapabilities dictionary is returned. If the track type is AudioStreamTrack, the AllAudioCapabilities dictionary is returned.

Given that implementations of various hardware may not exactly map to the same range, an implementation should make a reasonable attempt to translate and scale the hardware's setting onto the mapping provided by this specification. If this is not possible due to the user agent's inability to retrieve a given capability from a source, then for CapabilityRange-typed capabilities, the min and max fields will not be present on the returned dictionary, and the supported field will be false. For CapabilityList-typed capabilities, a suitable "notavailable" value will be the sole capability in the list.

Note

An example of the user agent providing an alternative mapping: if a source supports a hypothetical fluxCapacitance state whose type is a CapabilityRange, and the state is defined in this specification to be the range from -10 (min) to 10 (max), but the source's (hardware setting) for fluxCapacitance only supports values of "off" "medium" and "full", then the user agent should map the range value of -10 to "off", 10 should map to "full", and 0 should map to "medium". Constraints imposing a strict value of 3 will cause the user agent to attempt to set the value of "medium" on the hardware, and return a fluxCapacitance state of 0, the closest supported setting. No error event is raised in this scenario.

CapabilityList objects should order their enumerated values from minimum to maximum where it makes sense, or in the order defined by the enumerated type where applicable.

See the AllVideoCapabilities and AllAudioCapabilities dictionaries for details on the expected types for the various supported state names.

No parameters.
clone

Clones the given MediaStreamTrack.

When the clone() method is invoked, the user agent MUST run the following steps:

  1. Let trackClone be a newly constructed MediaStreamTrack object.

  2. Initialize trackClone's id attribute to a newly generated value.

  3. Let trackClone inherit this track's underlying source, kind, label and enabled attributes.

  4. Return trackClone.

No parameters.
Return type: MediaStreamTrack
constraints
Returns the complete constraints object associated with the track. If no mandatory constraints have been defined, the mandatory field will not be present (it will be undefined). If no optional constraints have been defined, the optional field will not be present (it will be undefined). If neither optional, nor mandatory constraints have been created, the value null is returned.
No parameters.
Return type: MediaTrackConstraints, nullable
getSourceInfos, static
Returns authorized information for all available sources.
No parameters.
Return type: sequence<SourceInfo>
states
Returns an object containing all the state variables associated with the allowed constraints.
No parameters.
Return type: MediaSourceStates
stop

When a MediaStreamTrack object's stop() method is invoked, if no source is attached (e.g., sourceType is "none"), then this call returns immediately (e.g., is a no-op). Otherwise, the user agent MUST queue a task that runs the following steps:

  1. Let track be the current MediaStreamTrack object.

  2. End track. The track starts outputting only silence and/or blackness, as appropriate.

  3. Permanently stop the generation of data for track's source. If the data is being generated from a live source (e.g., a microphone or camera), then the user agent SHOULD remove any active "on-air" indicator for that source. If the data is being generated from a prerecorded source (e.g. a video file), any remaining content in the file is ignored.

The task source for the tasks queued for the stop() method is the DOM manipulation task source.

No parameters.
Return type: void
enum MediaStreamTrackState {
    "new",
    "live",
    "ended"
};
Enumeration description
newThe track type is new and has not been initialized (connected to a source of any kind). This state implies that the track's label will be the empty string.
live

The track is active (the track's underlying media source is making a best-effort attempt to provide data in real time).

The output of a track in the live state can be switched on and off with the enabled attribute.

ended

The track has ended (the track's underlying media source is no longer providing data, and will never provide more data for this track). Once a track enters this state, it never exits it.

For example, a video track in a MediaStream ends if the user unplugs the USB web camera that acts as the track's media source.

4.3.4 Track Source Types

enum SourceTypeEnum {
    "none",
    "camera",
    "microphone"
};
Enumeration description
noneThis track has no source. This is the case when the track is in the "new" or "ended" readyState.
cameraA valid source type only for VideoStreamTracks. The source is a local video-producing camera source.
microphoneA valid source type only for AudioStreamTracks. The source is a local audio-producing microphone source.

4.3.5 Source Info

dictionary SourceInfo {
    DOMString sourceId;
    DOMString kind;
    DOMString label;
};
4.3.5.1 Dictionary SourceInfo Members
kind of type DOMString
MUST be either "audio" or "video".
label of type DOMString
If the application is authorized to get info from this source, the label attribute will be filled in with exactly the same value as would have been returned from a call to getUserMedia() with a constraint specifying this source's sourceId.
sourceId of type DOMString
The unique id for this source as described in the MediaSourceStates dictionary.

4.3.6 Video Facing Mode Enum

enum VideoFacingModeEnum {
    "user",
    "environment",
    "left",
    "right"
};
Enumeration description
userThe source is facing toward the user (a self-view camera).
environmentThe source is facing away from the user (viewing the environment).
leftThe source is facing to the left of the user.
rightThe source is facing to the right of the user.

4.3.7 Isolated Media Streams

When either the "noaccess" or "peerIdentity" constraints is applied to a MediaStreamTrack, the track shall be isolated so that its content is not accessible to the content JS. An isolated media stream may be used for two purposes:

  • Displayed in an appropriate tag (e.g., a video or audio element). The video element MUST have a unique origin so that it is inaccessible to the content JS. This is the same security mechanism as is used with an ordinary audio or video element which has a src= property from a separate origin.

  • Used as the argument to addStream() for a PeerConnection, subject to the restrictions detailed in the WebRTC document.

When the noaccess=true constraint applies to a track, that track may be added to any PeerConnection.

Note

Open Issue: The editors worry that the above paragraph is just wrong. If the track can be added to a PeerConnection that is connect to another PeerConenction in the same application, the application could get access to the data. We sugest this should be changed from "may be added" to "may not be added". This will allow noaccess=true to be used for things like hair check dialogs.

When the peerIdentity=foo constraint applies to a track, then that track may be added only to PeerConnections with compatible peer identities as described in the WebRTC document.

Both the noaccess and peerIdentity constraints must be mandatory. Any use of them in the optional block must trigger an error.

4.4 MediaStreamTrackEvent

The addtrack and removetrack events use the MediaStreamTrackEvent interface.

Firing a track event named e with a MediaStreamTrack track means that an event with the name e, which does not bubble (except where otherwise stated) and is not cancelable (except where otherwise stated), and which uses the MediaStreamTrackEvent interface with the track attribute set to track, MUST be created and dispatched at the given target.

dictionary MediaStreamTrackEventInit : EventInit {
    MediaStreamTrack? track;
};

[Constructor(DOMString type, MediaStreamTrackEventInit eventInitDict)] interface MediaStreamTrackEvent : Event { readonly attribute MediaStreamTrack track; };

4.4.1 Attributes

track of type MediaStreamTrack, readonly

The track attribute represents the MediaStreamTrack object associated with the event.

4.4.2 Dictionary MediaStreamTrackEventInit Members

track of type MediaStreamTrack, nullable

 

4.5 Video and Audio Tracks

The MediaStreamTrack object cannot be instantiated directly. To create an instance of a MediaStreamTrack, one of its derived track types may be instantiated. These derived types are defined in this section.

Note that the camera's green light doesn't come on when a new track is created; nor does the user get prompted to enable the camera/microphone. Those actions only happen after the developer has requested that a media stream containing "new" tracks be bound to a source via getUserMedia(). Until that point tracks are inert.

4.5.1 VideoStreamTrack interface

Video tracks may be instantiated with optional media track constraints. These constraints can be later modified on the track as needed by the application, or created after-the-fact if the initial constraints are unknown to the application.

Note

Example: VideoStreamTrack objects are instantiated in JavaScript using the new operator:

new VideoStreamTrack();
or
new VideoStreamTrack( { optional: [ { sourceId: "20983-20o198-109283-098-09812" }, { width: { min: 800, max: 1200 }}, { height: { min: 600 }}] });
[Constructor(optional MediaTrackConstraints videoConstraints)]
interface VideoStreamTrack : MediaStreamTrack {
    static sequence<DOMString> getSourceIds ();
};
4.5.1.1 Methods
getSourceIds, static

Returns an array of application-unique source identifiers. This list will be populated only with local sources whose sourceType is "camera" and if allowed by the user-agent, "readonly" variants of the former two types. The video source ids returned in the list constitute those sources that the user agent can identify at the time the API is called (the list can grow/shrink over time as sources may be added or removed). As a static method, getSourceIds can be queried without instantiating any VideoStreamTrack objects or without calling getUserMedia().

Issue 1

Issue: This information deliberately adds to the fingerprinting surface of the UA. However, this information will not be identifiable outside the scope of this application and could also be obtained via other round-about techniques using getUserMedia().

No parameters.
Return type: sequence<DOMString>

4.5.2 AudioStreamTrack

Note

Example: AudioStreamTrack objects are instantiated in JavaScript using the new operator:

new AudioStreamTrack();
or
new AudioStreamTrack( { optional: [ { sourceId: "64815-wi3c89-1839dk-x82-392aa" }, { gain: 0.5 }] });
[Constructor(optional MediaTrackConstraints audioConstraints)]
interface AudioStreamTrack : MediaStreamTrack {
    static sequence<DOMString> getSourceIds ();
};
4.5.2.1 Methods
getSourceIds, static
See definition of getSourceIds on the VideoStreamTrack object. Note that the list of source ids for AudioStreamTrack will be populated only with local sources whose sourceType is "microphone" and if allowed by the user-agent, "readonly" microphone variants.
No parameters.
Return type: sequence<DOMString>

5. The model: sources, sinks, constraints, and states

Browsers provide a media pipeline from sources to sinks. In a browser, sinks are the <img>, <video> and <audio> tags. Traditional sources include camera, microphones, streamed content, files and web resources. The media produced by these sources typically does not change over time - these sources can be considered to be static.

The sinks that display these sources to the user (the actual tags themselves) have a variety of controls for manipulating the source content. For example, an <img> tag scales down a huge source image of 1600x1200 pixels to fit in a rectangle defined with width="400" and height="300".

The getUserMedia API adds dynamic sources such as microphones and cameras - the characteristics of these sources can change in response to application needs. These sources can be considered to be dynamic in nature. A <video> element that displays media from a dynamic source can either perform scaling or it can feed back information along the media pipeline and have the source produce content more suitable for display.

Note

Note: This sort of feedback loop is obviously just enabling an "optimization", but it's a non-trivial gain. This optimization can save battery, allow for less network congestion, etc...

Note that MediaStream sinks (such as <video>, <audio>, and even RTCPeerConnection) will continue to have mechanisms to further transform the source stream beyond that which the states, capabilities, and constraints described in this specification offer. (The sink transformation options, including those of RTCPeerConnection, are outside the scope of this specification.)

The act of changing or applying a track constraint may affect the state of all tracks sharing that source and consequently all down-level sinks that are using that source. Many sinks may be able to take these changes in stride, such as the <video> element or RTCPeerConnection. Others like the Recorder API may fail as a result of a source state change.

The RTCPeerConnection is an interesting object because it acts simultaneously as both a sink and a source for over-the-network streams. As a sink, it has source transformational capabilities (e.g., lowering bit-rates, scaling-up or down resolutions, adjusting frame-rates), and as a source it could have its own states changed by a track source (though in this specification sources with the remote attribute set to true do not consider the current constraints applied to a track).

To illustrate how changes to a given source impact various sinks, consider the following example. This example only uses width and height, but the same principles apply to any of the states exposed in this proposal. In the first figure a home client has obtained a video source from its local video camera. The source's width and height state are 800 pixels by 600 pixels, respectively. Three MediaStream objects on the home client contain tracks that use this same sourceId. The three media streams are connected to three different sinks: a <video> element (A), another <video> element (B), and a peer connection (C). The peer connection is streaming the source video to an away client. On the away client there are two media streams with tracks that use the peer connection as a source. These two media streams are connected to two <video> element sinks (Y and Z).

Note that at this moment, all of the sinks on the home client must apply a transformation to the original source's provided state dimensions. A is scaling the video up (resulting in loss of quality), B is scaling the video down, and C is also scaling the video up slightly for sending over the network. On the away client, sink Y is scaling the video way down, while sink Z is not applying any scaling.

Using the constraint APIs, the home client's video source is changed to a higher resolution (1920 by 1200 pixels).

Note that the source change immediately affects all of the sinks on the home client, but does not impact any of the sinks (or sources) on the away client. With the increase in the home client source video's dimensions, sink A no longer has to perform any scaling, while sink B must scale down even further than before. Sink C (the peer connection) must now scale down the video in order to keep the transmission constant to the away client.

While not shown, an equally valid states change request could be made of the away client video source (the peer connection on the away client's side). This would not only impact sink Y and Z in the same manner as before, but would also cause re-negotiation with the peer connection on the home client in order to alter the transformation that it is applying to the home client's video source. Such a change would not change anything related to sink A or B or the home client's video source.

Note that this specification does not define a mechanism by which a change to the away client's video source could automatically trigger a change to the home client's video source. Implementations may choose to make such source-to-sink optimizations as long as they only do so within the constraints established by the application, as the next example demonstrates.

It is fairly obvious that changes to a given source will impact sink consumers. However, in some situations changes to a given sink may also be cause for implementations to adjust the characteristics of a source's stream. This is illustrated in the following figures. In the first figure below, the home client's video source is sending a video stream sized at 1920 by 1200 pixels. The video source is also unconstrained, such that the exact source dimensions are flexible as far as the application is concerned. Two MediaStream objects contain tracks with the same sourceId, and those MediaStreams are connected to two different <video> element sinks A and B. Sink A has been sized to width="1920" and height="1200" and is displaying the source's video content without any transformations. Sink B has been sized smaller and, as a result, is scaling the video down to fit its rectangle of 320 pixels across by 200 pixels down.

When the application changes sink A to a smaller dimension (from 1920 to 1024 pixels wide and from 1200 to 768 pixels tall), the browser's media pipeline may recognize that none of its sinks require the higher source resolution, and needless work is being done both on the part of the source and on sink A. In such a case and without any other constraints forcing the source to continue producing the higher resolution video, the media pipeline MAY change the source resolution:

In the above figure, the home client's video source resolution was changed to the greater of that from sinkA and from sinkB in order to optimize playback. While not shown above, the same behavior could apply to peer connections and other sinks.

It is possible that constraints can be applied to a track which a source is unable to satisfy. When this happens, the user agent is required to fire an "overconstrained" event to the track informing it of this condition, and the track becomes muted. There is no mandatory side-effect on the source itself as a result of this condition.

When multiple tracks share the same source (as illustrated in the previous figures), it is also possible that two (or more) tracks can apply contradictory constraints on the source. Since there is only a single thread of control, it is always possible for the browser to determine which track created the overconstrained condition. In this situation, the user agent MUST send the "overconstrained" event only to the track that created the condition and MUST not apply any of the constraints newly requested for that track. Here is an example of this behavior.

In this example, two media streams each have a video track that share the same source. The first track has a mandatory constraint forcing on the source's fill light. That track is connected to sink N. Sink N has a width and height of 800 by 600 pixels and is scaling down the source's resolution of 1024 by 768 to fit. The other track initially has no constraints applied; it is connected to sink P. Sink P has a width and height equal to that of the source.

Overconstrained

Now, the second track adds a mandatory constraint that the fill light should be forced off. At this point, both mandatory constraints cannot be satisfied by the source (the fill light cannot be simultaneously on and off at the same time). The second track is transitioned into the muted state and receives an "overconstrained" event. At the same time, the source notes that its remaining active sink only requires a resolution of 800 by 600 and so it adjusts its resolution down to match (this is an optional optimization that the user agent is allowed to make given the situation).

At this point, it is the responsibility of the application to fix the problem that led to the overconstrained situation (by either removing the fill light mandatory constraint on the second track, or by changing/removing the fill light mandatory constraint on the first track).

Overconstrained result

6. Source states

There is a variable associated with each capability that represents the state of the source with respect to that capability, the actual setting in use by the source. In the same way that the current set of constraints can be returned on a track using the constraints() method, the states() method on a track returns the values of the state variables associated with all capabilities, as the MediaSourceStates object. Only states appropriate to the sourceType are returned.

Note

Note: The following specific list(s) of states DOES NOT REFLECT CONSENSUS. Many states beyond these have been proposed, and the ones listed do not have universal support. The ones below are included **** ONLY **** to provide a starting point so we can see concrete examples of what real states might look like. The particular set below was chosen to match the **also temporary** set of constraints.

dictionary MediaSourceStates {
    SourceTypeEnum       sourceType;
    DOMString            sourceId;
    unsigned long?       width;
    unsigned long?       height;
    float?               frameRate;
    float?               aspectRatio;
    VideoFacingModeEnum? facingMode;
    unsigned long?       volume;
};

6.1 Dictionary MediaSourceStates Members

aspectRatio of type float, nullable
The current aspect ratio for the source. Aspect ratio is width in pixels divided by height in pixels, rounded to the tenth decimal place. Returned for video sources.
facingMode of type VideoFacingModeEnum, nullable
From the user's perspective, this value describes whether this camera is pointed toward the user ("user") or away from the user ("environment"). Returned for video sources.
frameRate of type float, nullable
The current frames per second rate of video provided by this source. If the source does not provide a frameRate (or the frameRate cannot be determined from the source stream), then this attribute MUST be the user agent's vsync display rate. Returned for video sources.
height of type unsigned long, nullable
The height (in pixels) of the source of the video flowing through the track. Returned for video sources.
sourceId of type DOMString
The application-unique identifier for this source. The same identifier must be valid between sessions of this application, but must also be different for other applications. Some sort of GUID is recommended for the identifier. Returned for all types of sources (video, audio, etc.).
sourceType of type SourceTypeEnum
The type information associated with the currently attached source. Returned for all types of sources (video, audio, etc.).
volume of type unsigned long, nullable
The current audio track's volume (as a percentage). A volume of 0 is silence, while a volume of 100 is the maximum supported volume. Returned for audio sources.
width of type unsigned long, nullable
The width (in pixels) of the source of the video flowing through the track. Returned for video sources.

7. Source capabilities

Each constraint that is supported by an implementation MUST have an associated capability that will be returned in the result of a call to the capabilities() method.

Capabilities are provided as either a min/max range or a list of enumerated values. Min/max capabilities are always provided for constraints that are not enumerated types. Listed capabilities are always provided for constraints corresponding to enumerated types.

dictionary CapabilityRange {
    any     max;
    any     min;
    boolean supported;
};

7.1 Dictionary CapabilityRange Members

max of type any
The maximum value of this capability.

The type of this value is specific to the capability.

If the related capability is not supported by the source, then this field will not be provided by the user agent (it will be undefined).

min of type any
The minimum value of this capability.

The type of this value is specific to the capability.

If the related capability is not supported by the source, then this field will not be provided by the user agent (it will be undefined).

supported of type boolean
Returns the value true if the capability is supported, false otherwise.

7.2 CapabilityList array

typedef sequence  CapabilityList;
Throughout this specification, the identifier CapabilityList is used to refer to the sequence type.

Capability Lists are just an array of supported DOMString values from the possible superset of values described by each state's enumerated type.

Note

Note: The following specific list(s) of capabilities DOES NOT REFLECT CONSENSUS. Many capabilities beyond these have been proposed, and the ones listed do not have universal support. The ones below are included **** ONLY **** to provide a starting point so we can see concrete examples of what real capabilities might look like. The particular sets below were chosen to match the **also temporary** set of constraints.

dictionary AllVideoCapabilities {
    CapabilityList?  sourceType;
    CapabilityList?  sourceId;
    CapabilityRange? width;
    CapabilityRange? height;
    CapabilityRange? frameRate;
    CapabilityRange? aspectRatio;
    CapabilityList?  facingMode;
};

7.3 Dictionary AllVideoCapabilities Members

aspectRatio of type CapabilityRange, nullable
The supported range of aspect ratio defined as a floating point number rounded to 10 decimal places on the source and representing the width in pixels divided by the height in pixels. The type of the min/max values are float.
facingMode of type CapabilityList, nullable
The available video facing options (VideoFacingModeEnum) on the source.
frameRate of type CapabilityRange, nullable
The supported range of frame rates on the source. The type of the min/max values are float.
height of type CapabilityRange, nullable
The range should span the video source's pre-set height values with min being the smallest width, and max the largest width. The type of the min/max values are unsigned long.
sourceId of type CapabilityList, nullable
The available source identifiers of the current source -- this will always return a list with a single identifier (that of the current source). Note, to get a list of other available source identifiers, use the static getSourceIds() method.
sourceType of type CapabilityList, nullable
The available sourceType options (SourceTypeEnum) on the current source.
width of type CapabilityRange, nullable
The range should span the video source's pre-set width values with min being the smallest width, and max the largest width. The type of the min/max values are unsigned long.
dictionary AllAudioCapabilities {
    CapabilityRange? volume;
};

7.4 Dictionary AllAudioCapabilities Members

volume of type CapabilityRange, nullable
The supported range of output volume percentages on the source. The type of the min/max values are unsigned long.

8. URL

partial interface URL {
    static DOMString createObjectURL (MediaStream stream);
};

8.1 Methods

createObjectURL, static

Mints a Blob URL to refer to the given MediaStream.

When the createObjectURL() method is called with a MediaStream argument, the user agent MUST return a unique Blob URL for the given MediaStream. [FILE-API]

For audio and video streams, the data exposed on that stream MUST be in a format supported by the user agent for use in audio and video elements.

A Blob URL is the same as what the File API specification calls a Blob URI, except that anything in the definition of that feature that refers to File and Blob objects is hereby extended to also apply to MediaStream objects.

ParameterTypeNullableOptionalDescription
streamMediaStream
Return type: DOMString

9. MediaStreams as Media Elements

A MediaStream may be assigned to media elements as defined in HTML5 [HTML5] A MediaStream is not preloadable or seekable and represents a simple, potentially infinite, linear media timeline. The timeline starts at 0 and increments linearly in real time as long as the MediaStream is playing. The timeline does not increment when the MediaStream is paused.

9.1 Direct Assignment to Media Elements

There are two methods by which a MediaStream may be assigned to a media element. First, createObjectURL can be used to obtain a URL for the MediaStream. That URL can then be used to set the src attribute of the media element itself, or of one of its <source> children. Secondly, UAs that support this specification MUST support the following partial interface, which allows a MediaStream to be assigned directly to a media element.

partial interface HTMLMediaElement {
                attribute MediaStream? srcObject;
};

9.1.1 Attributes

srcObject of type MediaStream, , nullable

Holds the MediaStream that provides media for this element. This attribute overrides both the src attribute and any <source> elements. Specifically, if srcObject is specified, the UA MUST use it as the source of media, even if the src attribute is also set or <source> children are present. If the value of srcObject is replaced or set to null the UA MUST re-run the media element load algorithm

Issue 2

We may want to allow direct assignment of other types as well

9.2 Loading and Playing a MediaStream in a Media Element

The UA runs the media element load algorithm to obtain media for the media element to display. As defined in the [HTML5] specification, this algorithm has two basic phases: resource selection algorithm chooses the resource to play and resolves its URI. Then the resource fetch phase loads the resource. Both these phases are potentially simplified when using a MediaStream. First of all, in the case of direct assignment, srcObject takes priority over other means of specifying the resource. Futhermore, it provides the object itself rather than a URI. In this case, there is no need to run the resource selection algorithm. Secondly, when the UA reaches the resource fetch algorithm with a MediaStream (whether specified by URI or direct assignment), the MediaStream is a local object so there's nothing to fetch. Therefore, the following modifications/restrictions to the media element load algorithm apply:

Issue 3

If a MediaStream is inactive and the media element is ended, and the element's autoplay attribute is set to true, should it start playing automatically if new Tracks are added to the MediaStream?

9.3 Media Element Attributes when Playing a MediaStream

The nature of the MediaStream places certain restrictions on the behavior and attribute values of the associated media element and on the operations that can be performed on it, as shown below:

Attribute Name Attribute Type Valid Values When Using a MediaStream Additional considerations
src DOMString a local URI referencing a MediaStream The markup author can use this attribute to specify the source of the media. createObjectURI can be used to generate a URI referring to a MediaStream which can be used as the value of this attribute. In such a case, revocation of the URI does not count as a change to this field and does not trigger the media element load algorithm.
currentSrc DOMString a local URI referencing a MediaStream or the empty string The UA automatically sets the value of this attribute. When srcObject is specified the UA MUST set it to the empty string.
preload DOMString none A MediaStream cannot be preloaded.
buffered TimeRanges buffered.length MUST return 0. A MediaStream cannot be preloaded. Therefore, the amount buffered is always an empty TimeRange.
networkState unsigned short NETWORK_IDLE The media element does not fetch the MediaStream so there is no network traffic.
readyState unsigned short HAVE_NOTHING, HAVE_ENOUGH_DATA A MediaStream may be created before there is any data available, for example when a stream is received from a remote peer. The value of the readyState of the media element MUST be HAVE_NOTHING before the first media arrives and HAVE_ENOUGH_DATA once the first media has arrived.
currentTime double Any positive integer. The initial value is 0 and the values increments linearly in real time whenever the stream is playing. The value is the current stream position, in seconds. On any attempt to set this attribute, the user agent must throw an InvalidStateError exception.
duration unrestricted double Infinity A MediaStream does not have a pre-defined duration.
seeking boolean false A MediaStream is not seekable. Therefore, this attribute MUST always have the value false.
defaultPlaybackRate double 1.0 A MediaStream is not seekable. Therefore, this attribute MUST always have the value 1.0 and any attempt to alter it MUST fail.
playbackRate double 1.0 A MediaStream is not seekable. Therefore, this attribute MUST always have the value 1.0 and any attempt to alter it MUST fail.
played TimeRanges played.length MUST return 1.
played.start(0) MUST return 0.
played.end(0) MUST return the last known currentTime.
A MediaStream's timeline always consists of a single range, starting at 0 and extending up to the currentTime.
seekable TimeRanges seekable.length MUST return 0.
seekable.start() MUST return currentTime.
seekable.end() MUST return currentTime.
A MediaStream is not seekable.
startDate Date Not-a-Number (NaN) A MediaStream does not specify a timeline offset.
loop boolean false A MediaStream has no defined end and therefore cannot be looped.

10. Event summary

This section is non-normative.

The following event fires on MediaStream objects:

Event name Interface Fired when...
ended Event The MediaStream finished as a result of all tracks in the MediaStream ending.
addtrack MediaStreamTrackEvent A new MediaStreamTrack has been added to this stream. Note that this event is not fired when the script directly modifies the tracks of a MediaStream.
removetrack MediaStreamTrackEvent A MediaStreamTrack has been removed from this stream. Note that this event is not fired when the script directly modifies the tracks of a MediaStream.

The following event fires on MediaStreamTrack objects:

Event name Interface Fired when...
started Event The MediaStreamTrack object has just transitioned from the "new" readyState to another state. This event fires before any other corresponding events such as "ended" or "statechanged".
mute Event The MediaStreamTrack object's source is temporarily unable to provide data.
unmute Event The MediaStreamTrack object's source is live again after having been temporarily unable to provide data.
overconstrained Event

This event fires asynchronously for each affected track (when multiple tracks share the same source) after the user agent has evaluated the current constraints against a given sourceId and is not able to configure the source within the limitations established by the union of imposed constraints.

Due to being over-constrained, the user agent must mute each affected track.

The affected track(s) will remain un-usable (in the "muted" readyState) until the application adjusts the constraints to accommodate the source's capabilities.

The "overconstrained" event is a simple event of type Event; it carries no information about which constraints caused the source to be over-constrained (the application has all the necessary APIs to figure it out).

ended Event The MediaStreamTrack object's source will no longer provide any data, either because the user revoked the permissions, or because the source device has been ejected, or because the remote peer stopped sending data, or because the stop() method was invoked.

11. Obtaining local multimedia content

11.2 MediaStreamConstraints

dictionary MediaStreamConstraints {
    (boolean or MediaTrackConstraints) video = false;
    (boolean or MediaTrackConstraints) audio = false;
};

11.2.1 Dictionary MediaStreamConstraints Members

audio of type (boolean or MediaTrackConstraints), defaulting to false
Issue 7

Provide definition of audio constraints here.

video of type (boolean or MediaTrackConstraints), defaulting to false
Issue 6

Provide definition of video constraints here.

dictionary MediaTrackConstraints {
    MediaTrackConstraintSet? mandatory = null;
    MediaTrackConstraint[]?  _optional = null;
};

11.2.2 Dictionary MediaTrackConstraints Members

optional of type array of MediaTrackConstraint, nullable, defaulting to null
Issue 9

Provide definition of optional constraints here.

mandatory of type MediaTrackConstraintSet, nullable, defaulting to null
Issue 8

Provide definition of mandatory constraints here.

A MediaTrackConstraintSet is a dictionary containing one or more key-value pairs, where each key MUST be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [RTCWEB-CONSTRAINTS] and its value SHOULD be as defined in the associated reference(s) given in the registry.

A MediaTrackConstraint is a dictionary containing exactly one key-value pair, where the key MUST be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [RTCWEB-CONSTRAINTS] and the value SHOULD be as defined in the associated reference(s) given in the registry.

Note

Example MediaTrackConstraints value:

{
  mandatory: {
    width: { min: 640 },
    height: { min: 480 }
  },
  optional: [
    { width: 650 },
    { width: { min: 650 }},
    { frameRate: 60 },
    { width: { max: 800 }},
    { facingMode: "user" }
  ]
}                                

11.5 Implementation Suggestions

This section is non-normative.

Best Practice 1: Resource reservation

The user agent is encouraged to reserve resources when it has determined that a given call to getUserMedia() will succeed. It is preferable to reserve the resource prior to invoking the success callback provided by the web page. Subsequent calls to getUserMedia() (in this page or any other) should treat the resource that was previously allocated, as well as resources held by other applications, as busy. Resources marked as busy should not be provided as sources to the current web page, unless specified by the user. Optionally, the user agent may choose to provide a stream sourced from a busy source but only to a page whose origin matches the owner of the original stream that is keeping the source busy.

This document recommends that in the permission grant dialog or device selection interace (if one is present), the user be allowed to select any available hardware as a source for the stream requested by the page (provided the resource is able to fulfill mandatory constraints, if any were specified), in addition to the ability to substitute a video or audio source with local files and other media. A file picker may be used to provide this functionality to the user.

This document also recommends that the user be shown all resources that are currently busy as a result of prior calls to getUserMedia() (in this page or any other page that is still alive) and be allowed to terminate that stream and utilize the resource for the current page instead. If possible in the current operating environment, it is also suggested that resources currently held by other applications be presented and treated in the same manner. If the user chooses this option, the track corresponding to the resource that was provided to the page whose stream was affected must be removed. Additionally, if removing a track in this manner causes the stream to contain no more tracks, the onended event must be raised on it.

Best Practice 2: Handling multiple devices

A MediaStream may contain more than one video and audio track. This makes it possible to include video from two or more webcams in a single stream object, for example. However, the current API does not allow a page to express a need for multiple video streams from independent sources.

It is recommended for multiple calls to getUserMedia() from the same page be allowed as a way for pages to request multiple, discrete, video or audio streams.

A single call to getUserMedia() will always return a stream with either zero or one audio tracks, and either zero or one video tracks. If a script calls getUserMedia() multiple times before reaching a stable state, this document advises the UI designer that the permission dialogs should be merged, so that the user can give permission for the use of multiple cameras and/or media sources in one dialog interaction. The constraints on each getUserMedia call can be used to decide which stream gets which media sources.

12. Examples

This sample code exposes a button. When clicked, the button is disabled and the user is prompted to offer a stream. The user can cause the button to be re-enabled by providing a stream (e.g., giving the page access to the local camera) and then disabling the stream (e.g., revoking that access).

Example 1
<input type="button" value="Start" onclick="start()" id="startBtn">
<script>
 var startBtn = document.getElementById('startBtn');
 function start() {
   navigator.getUserMedia({audio:true, video:true}, gotStream, logError);
   startBtn.disabled = true;
 }
 function gotStream(stream) {
   stream.onended = function () {
     startBtn.disabled = false;
   };
 }
 function logError(error) {
   log(error.name + ": " + error.message);
 }
</script> 

This example allows people to take photos of themselves from the local video camera. Note that the forthcoming Image Capture specification may provide a simpler way to accomplish this.

Example 2
<article>
 <style scoped>
  video { transform: scaleX(-1); }
  p { text-align: center; }
 </style>
 <h1>Snapshot Kiosk</h1>
 <section id="splash">
  <p id="errorMessage">Loading...</p>
 </section>
 <section id="app" hidden>
  <p><video id="monitor" autoplay></video> <canvas id="photo"></canvas>
  <p><input type=button value="&#x1F4F7;" onclick="snapshot()">
 </section>
 <script>
  navigator.getUserMedia({video:true}, gotStream, noStream);
  var video = document.getElementById('monitor');
  var canvas = document.getElementById('photo');
  function gotStream(stream) {
    video.src = URL.createObjectURL(stream);
    video.onerror = function () {
      stream.stop();
    };
    stream.onended = noStream;
    video.onloadedmetadata = function () {
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      document.getElementById('splash').hidden = true;
      document.getElementById('app').hidden = false;
    };
  }
  function noStream() {
    document.getElementById('errorMessage').textContent = 'No camera available.';
  }
  function snapshot() {
    canvas.getContext('2d').drawImage(video, 0, 0);
  }
 </script>
</article>

13. IANA Registrations

13.1 Constraint Registrations

IANA is requested to register the following constraints as specified in [RTCWEB-CONSTRAINTS]:

Note

Note: The following specific list(s) of constraints DOES NOT REFLECT CONSENSUS. Many constraints beyond these have been proposed, and the ones listed do not have universal support. The ones below are included **** ONLY **** to provide a starting point so we can see concrete examples of what real constraints might look like. Along those lines, there was an attempt to include constraints of a variety of types so sample type definitions could be included as well (e.g., DOMString, MinMaxConstraint).

The following constraint names are defined to apply to both VideoStreamTrack and AudioStreamTrack objects:

Constraint Name Values Notes
sourceType SourceTypeEnum Constrain the video or audio source to an exact value from the set of enumerated-type values of the SourceTypeEnum.
sourceId DOMString Constrain the video or audio source to an exact source identifier value.
noaccess boolean Prevent the stream from being applied to an element which is readable by the JS.
peerIdentity DOMString Prevent the stream from being applied to an element which is readable by the JS. Require that any PeerConnections to which the stream is added match the provided identity.

The following constraint names are defined to apply only to VideoStreamTrack objects:

Constraint Name Values Notes
width unsigned long or MinMaxConstraint Constrain the video source to the exact desired width or width range.
height unsigned long or MinMaxConstraint Constrain the video source to the exact desired height or height range.
frameRate float or MinMaxConstraint Constrain the video source to the exact desired frame rate (fps) or frameRate range.
aspectRatio float or MinMaxConstraint Constrain the video source to the exact desired aspect ratio represented as floating point number rounded to 10 decimal places, representing the width in pixels divided by the height in pixels.
facingMode VideoFacingModeEnum Constrain the video source to an exact value from the set of enumerated-type values of the VideoFacingModeEnum.

The following constraint names are defined to apply only to AudioStreamTrack objects:

Constraint Name Values Notes
volume unsigned long or MinMaxConstraint Constrain the audio source to the exact desired volume or volume range.

For constraints that accept ranges, the MinMaxConstraint dictionary is also defined. Note that the type of the value associated with min and max must be the same for both. The specific types associated with min and max are defined differently for each constraint name.

dictionary MinMaxConstraint {
    any max;
    any min;
};

13.1.1 Dictionary MinMaxConstraint Members

max of type any
The related constraint's maximum allowed value.
min of type any
The related constraint's minimum allowed value.

14. Change Log

This section will be removed before publication.

Changes since July 4, 2013

  1. Bug 21967: Added paragraph on MediaStreamTrack enabled state and updated cloning algorithm.
  2. Bug 22210: Make getUserMedia() algorithm use all numbered items.
  3. Bug 22250: Fixed accidentally overridden error.
  4. Bug 22211: Added async error when no valid media type is requested.
  5. Bug 22216: Made NavigatorUserMediaError extend DOMError.
  6. Bug 22249: Throw on attempts to set currentTime on media elements playing MediaStream objects.
  7. Bug 22246: Made media.buffered have length 0.
  8. Bug 22692: Updated media element to use HAVE_NOTHING state before media arrives on the played MediaStream and HAVE_ENOUGH_DATA as soon as media arrives.

May 29, 2013

  1. Bug 22252: fixed usage of MUST in MediaStream() constructor description.
  2. Bug 22215: made MediaStream.ended readonly.
  3. Bug 21967: clarified MediaStreamTrack.enabled state initial value.
  4. Added aspectRatio constraint, capability, and state.
  5. Updated usage of MediaStreams in media elements.

May 15, 2013

  1. Added explanatory section for constraints, capabilities, and states.
  2. Added VideoFacingModeEnum (including left and right options).
  3. Added getSourceInfos() and SourceInfo dictionary.
  4. Added isolated streams.

April 29, 2013

  1. Removed remaining photo APIs and references (since we have a separate Image Capture Spec).

March 20, 2013

  1. Added readonly and remote attributes to MediaStreamTrack
  2. Removed getConstraint(), setConstraint(), appendConstraint(), and prependConstraint().
  3. Added source states. Added states() method on tracks. Moved sourceType and sourceId to be states.
  4. Added source capabilities. Added capabilities() method on tracks.
  5. Added clarifying text about MediaStreamTrack lifecycle and mediaflow.
  6. Made MediaStreamTrack cloning explicit.
  7. Removed takePhoto() and friends from VideoStreamTrack (we have a separate Image Capture Spec).
  8. Made getUserMedia() error callback mandatory.

December 12, 2012

  1. Changed error code to be string instead of number.
  2. Added core of settings proposal allowing for constraint changes after stream/track creation.

November 15 2012

  1. Introduced new representation of tracks in a stream (removed MediaStreamTrackList).
  2. Updated MediaStreamTrack.readyState to use an enum type (instad of unsigned short constants).
  3. Renamed MediaStream.label to MediaStream.id (the definition needs some more work).

October 1 2012

  1. Limited the track kind values to "audio" and "video" only (could previously be user defined as well).
  2. Made MediaStream extend EventTarget.
  3. Simplified the MediaStream constructor.

June 23 2012

  1. Rename title to "Media Capture and Streams".
  2. Update document to comply with HTML5.
  3. Update image describing a MediaStream.
  4. Add known issues and various other editorial changes.

June 22 2012

  1. Update wording for constraints algorithm.

June 19 2012

  1. Added "Media Streams as Media Elements section".

June 12 2012

  1. Switch to respec v3.

June 5 2012

  1. Added non-normative section "Implementation Suggestions".
  2. Removed stray whitespace.

June 1 2012

  1. Added media constraint algorithm.

Apr 23 2012

  1. Remove MediaStreamRecorder.

Apr 20 2012

  1. Add definitions of MediaStreams and related objects.

Dec 21 2011

  1. Changed to make wanted media opt in (rather than opt out). Minor edits.

Nov 29 2011

  1. Changed examples to use MediaStreamOptions objects rather than strings. Minor edits.

Nov 15 2011

  1. Removed MediaStream stuff. Refers to webrtc 1.0 spec for that part instead.

Nov 9 2011

  1. Created first version by copying the webrtc spec and ripping out stuff. Put it on github.

A. Acknowledgements

x

B. References

B.1 Normative references

[FILE-API]
Arun Ranganathan; Jonas Sicking. File API. 25 October 2012. W3C Working Draft. URL: http://www.w3.org/TR/FileAPI/
[HTML5]
Robin Berjon; Steve Faulkner; Travis Leithead; Erika Doyle Navara; Edward O'Connor; Silvia Pfeiffer. HTML5. 6 August 2013. W3C Candidate Recommendation. URL: http://www.w3.org/TR/html5/
[RFC2119]
S. Bradner. Key words for use in RFCs to Indicate Requirement Levels. March 1997. Internet RFC 2119. URL: http://www.ietf.org/rfc/rfc2119.txt
[RTCWEB-CONSTRAINTS]
D. Burnett. IANA Registry for RTCWeb Media Constraints. URL: http://datatracker.ietf.org/doc/draft-burnett-rtcweb-constraints-registry/
[WEBIDL]
Cameron McCormack. Web IDL. 19 April 2012. W3C Candidate Recommendation. URL: http://www.w3.org/TR/WebIDL/
[WEBRTC10]
Adam Bergkvist; Daniel Burnett; Cullen Jennings; Anant Narayanan. WebRTC 1.0: Real-time Communication Between Browsers. 21 August 2012. W3C Working Draft. URL: http://www.w3.org/TR/webrtc/