Media Capture and Streams

Table of Contents

• 1. Introduction
• 2. Conformance
• 3. Terminology
• 4. Stream API
  • 4.1 Introduction
  • 4.2 MediaStream
    • 4.2.1 Attributes
    • 4.2.2 Methods
  • 4.3 MediaStreamTrack
    • 4.3.1 Life-cycle and Media Flow
    • 4.3.2 Tracks and Constraints
    • 4.3.3 Interface Definition
      • 4.3.3.1 Attributes
      • 4.3.3.2 Methods
    • 4.3.4 Track Source Types
    • 4.3.5 Source Info
      • 4.3.5.1 Dictionary SourceInfo Members
    • 4.3.6 Video Facing Mode Enum
  • 4.4 MediaStreamTrackEvent
    • 4.4.1 Attributes
    • 4.4.2 Dictionary MediaStreamTrackEventInit Members
  • 4.5 Video and Audio Tracks
    • 4.5.1 VideoStreamTrack interface
      • 4.5.1.1 Methods
    • 4.5.2 AudioStreamTrack
      • 4.5.2.1 Methods
• 5. The model: sources, sinks, constraints, and states
• 6. Source states
  • 6.1 Dictionary MediaSourceStates Members
• 7. Source capabilities
  • 7.1 Dictionary CapabilityRange Members
  • 7.2 CapabilityList array
  • 7.3 Dictionary AllVideoCapabilities Members
  • 7.4 Dictionary AllAudioCapabilities Members
• 8. URL
  • 8.1 Methods
• 9. MediaStreams as Media Elements
• 10. Event summary
• 11. Obtaining local multimedia content
  • 11.1 NavigatorUserMedia
    • 11.1.1 Methods
  • 11.2 MediaStreamConstraints
    • 11.2.1 Dictionary MediaStreamConstraints Members
    • 11.2.2 Dictionary MediaTrackConstraints Members
  • 11.3 NavigatorUserMediaSuccessCallback
    • 11.3.1 Callback NavigatorUserMediaSuccessCallback Parameters
  • 11.4 NavigatorUserMediaError and NavigatorUserMediaErrorCallback
    • 11.4.1 Attributes
    • 11.4.2 Callback NavigatorUserMediaErrorCallback Parameters
  • 11.5 Implementation Suggestions
• 12. Examples
• 13. IANA Registrations
  • 13.1 Constraint Registrations
    • 13.1.1 Dictionary MinMaxConstraint Members
• 14. Change Log
• A. Acknowledgements
• B. References
  • B.1 Normative references

2. Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this specification are to be interpreted as described in [RFC2119].

This specification defines conformance criteria that apply to a single product: the user agent that implements the interfaces that it contains.

Implementations that use ECMAScript to implement the APIs defined in this specification must implement them in a manner consistent with the ECMAScript Bindings defined in the Web IDL specification [WEBIDL], as this specification uses that specification and terminology.

3. Terminology

HTML Terms:

The EventHandler interface represents a callback used for event handlers as defined in [HTML5].

The concepts queue a task and fires a simple event are defined in [HTML5].

The terms event handlers and event handler event types are defined in [HTML5].

source

A source is the "thing" providing the source of a media stream track. The source is the broadcaster of the media itself. A source can be a physical webcam, microphone, local video or audio file from the user's hard drive, network resource, or static image.

Some sources have an identifier which must be unique to the application (un-guessable by another application) and persistent between application sessions (e.g., the identifier for a given source device/application must stay the same, but not be guessable by another application). Sources that must have an identifier are camera and microphone sources; local file sources are not required to have an identifier. Source identifiers let the application save, identify the availability of, and directly request specific sources.

Other than the identifier, other bits of source identity are never directly available to the application until the user agent connects a source to a track. Once a source has been "released" to the application (either via a permissions UI, pre-configured allow-list, or some other release mechanism) the application will be able discover additional source-specific capabilities.

Sources do not have constraints -- tracks have constraints. When a source is connected to a track, it must conform to the constraints present on that track (or set of tracks).

Sources will be released (un-attached) from a track when the track is ended for any reason.

On the MediaStreamTrack object, sources are represented by a sourceType attribute. The behavior of APIs associated with the source's capabilities and state change depending on the source type.

Sources have capabilities and state. The capabilities and state are "owned" by the source and are common to any (multiple) tracks that happen to be using the same source (e.g., if two different tracks objects bound to the same source ask for the same capability or state information, they will get back the same answer).

State (Source State)

State refers to the immediate, current value of the source's (optionally constrained) capabilities. State is always read-only.

A source's state can change dynamically over time due to environmental conditions, sink configurations, or constraint changes. A source's state must always conform to the current set of mandatory constraints that all of the tracks it is bound to have defined, and should do its best to conform to the set of optional constraints specified.

A source's state is directly exposed to audio and video track objects through individual read-only attributes. These attributes share the same name as their corresponding capabilities and constraints.

Events are available that signal to the application that source state has changed.

A conforming user-agent must support all the state names defined in this spec.

Capabilities

Source capabilities are the intrinsic "features" of a source object. For each source state, there is a corresponding capability that describes whether it is supported by the source and if so, what the range of supported values are. Capabilities are expressed as either a series of states (for enumerated-type capabilities) or as a min/max range.

The values of the supported capabilities must be normalized to the ranges and enumerated types defined in this specification.

Capabilities return the same underlying per-source capabilities, regardless of any user-supplied constraints present on the source (capabilities are independent of constraints).

Source capabilities are effectively constant. Applications should be able to depend on a specific source having the same capabilities for any session.

Constraints

Constraints are an optional feature for restricting the range of allowed variability on a source. Without provided constraints, implementations are free to select a source's state from the full range of its supported capabilities, and to adjust that state at any time for any reason.

Constraints may be optional or mandatory. Optional constraints are represented by an ordered list, mandatory constraints are an unordered set. The order of the optional constraints is from most important (at the head of the list) to least important (at the tail of the list).

Constraints are stored on the track object, not the source. Each track can be optionally initialized with constraints, or constraints can be added afterward through the constraint APIs defined in this spec.

Applying track level constraints to a source is conditional based on the type of source. For example, read-only sources will ignore any specified constraints on the track.

It is possible for two tracks that share a unique source to apply contradictory constraints. Under such contradictions, the implementation will mute both tracks and notify them that they are over-constrained.

Events are available that allow the application to know when constraints cannot be met by the user agent. These typically occur when the application applies constraints beyond the capability of a source, contradictory constraints, or in some cases when a source cannot sustain itself in over-constrained scenarios (overheating, etc.).

Constraints that are intended for video sources will be ignored by audio sources and vice-versa. Similarly, constraints that are not recognized will be preserved in the constraint structure, but ignored by the application. This will allow future constraints to be defined in a backward compatible manner.

A correspondingly-named constraint exists for each corresponding source state name and capability name. In general, user agents will have more flexibility to optimize the media streaming experience the fewer constraints are applied.

4. Stream API

typedef sequence<MediaStreamTrack> MediaStreamTrackSequence;

[Constructor,
        Constructor (MediaStream stream),
        Constructor (MediaStreamTrackSequence tracks)]
interface MediaStream : EventTarget {
    readonly    attribute DOMString    id;
    sequence<MediaStreamTrack> getAudioTracks ();
    sequence<MediaStreamTrack> getVideoTracks ();
    MediaStreamTrack?          getTrackById (DOMString trackId);
    void                       addTrack (MediaStreamTrack track);
    void                       removeTrack (MediaStreamTrack track);
    MediaStream                clone ();
                attribute boolean      ended;
                attribute EventHandler onended;
                attribute EventHandler onaddtrack;
                attribute EventHandler onremovetrack;
};

interface MediaStreamTrack : EventTarget {
    readonly    attribute DOMString             kind;
    readonly    attribute DOMString             id;
    readonly    attribute DOMString             label;
                attribute boolean               enabled;
    readonly    attribute boolean               muted;
                attribute EventHandler          onmute;
                attribute EventHandler          onunmute;
    readonly    attribute boolean               _readonly;
    readonly    attribute boolean               remote;
    readonly    attribute MediaStreamTrackState readyState;
                attribute EventHandler          onstarted;
                attribute EventHandler          onended;
    static sequence<SourceInfo>                    getSourceInfos ();
    MediaTrackConstraints?                         constraints ();
    MediaSourceStates                              states ();
    (AllVideoCapabilities or AllAudioCapabilities) capabilities ();
    void                                           applyConstraints (MediaTrackConstraints constraints);
                attribute EventHandler          onoverconstrained;
    MediaStreamTrack                               clone ();
    void                                           stop ();
};

enum MediaStreamTrackState {
    "new",
    "live",
    "ended"
};

enum SourceTypeEnum {
    "none",
    "camera",
    "microphone"
};

dictionary SourceInfo {
    DOMString sourceId;
    DOMString kind;
    DOMString label;
};

enum VideoFacingModeEnum {
    "user",
    "environment",
    "left",
    "right"
};

dictionary MediaStreamTrackEventInit : EventInit {
    MediaStreamTrack? track;
};

[Constructor(DOMString type, MediaStreamTrackEventInit eventInitDict)]
interface MediaStreamTrackEvent : Event {
    readonly    attribute MediaStreamTrack track;
};

new VideoStreamTrack();

new VideoStreamTrack( { optional: [ { sourceId: "20983-20o198-109283-098-09812" }, { width: { min: 800, max: 1200 }}, { height: { min: 600 }}] });

[Constructor(optional MediaTrackConstraints videoConstraints)]
interface VideoStreamTrack : MediaStreamTrack {
    static sequence<DOMString> getSourceIds ();
};

new AudioStreamTrack();

new AudioStreamTrack( { optional: [ { sourceId: "64815-wi3c89-1839dk-x82-392aa" }, { gain: 0.5 }] });

[Constructor(optional MediaTrackConstraints audioConstraints)]
interface AudioStreamTrack : MediaStreamTrack {
    static sequence<DOMString> getSourceIds ();
};

5. The model: sources, sinks, constraints, and states

Browsers provide a media pipeline from sources to sinks. In a browser, sinks are the <img>, <video> and <audio> tags. Traditional sources include camera, microphones, streamed content, files and web resources. The media produced by these sources typically does not change over time - these sources can be considered to be static.

The sinks that display these sources to the user (the actual tags themselves) have a variety of controls for manipulating the source content. For example, an <img> tag scales down a huge source image of 1600x1200 pixels to fit in a rectangle defined with width="400" and height="300".

The getUserMedia API adds dynamic sources such as microphones and cameras - the characteristics of these sources can change in response to application needs. These sources can be considered to be dynamic in nature. A <video> element that displays media from a dynamic source can either perform scaling or it can feed back information along the media pipeline and have the source produce content more suitable for display.

Note that MediaStream sinks (such as <video>, <audio>, and even RTCPeerConnection) will continue to have mechanisms to further transform the source stream beyond that which the states, capabilities, and constraints described in this specification offer. (The sink transformation options, including those of RTCPeerConnection, are outside the scope of this specification.)

The act of changing or applying a track constraint may affect the state of all tracks sharing that source and consequently all down-level sinks that are using that source. Many sinks may be able to take these changes in stride, such as the <video> element or RTCPeerConnection. Others like the Recorder API may fail as a result of a source state change.

The RTCPeerConnection is an interesting object because it acts simultaneously as both a sink and a source for over-the-network streams. As a sink, it has source transformational capabilities (e.g., lowering bit-rates, scaling-up or down resolutions, adjusting frame-rates), and as a source it could have its own states changed by a track source (though in this specification sources with the remote attribute set to true do not consider the current constraints applied to a track).

To illustrate how changes to a given source impact various sinks, consider the following example. This example only uses width and height, but the same principles apply to any of the states exposed in this proposal. In the first figure a home client has obtained a video source from its local video camera. The source's width and height state are 800 pixels by 600 pixels, respectively. Three MediaStream objects on the home client contain tracks that use this same sourceId. The three media streams are connected to three different sinks: a <video> element (A), another <video> element (B), and a peer connection (C). The peer connection is streaming the source video to an away client. On the away client there are two media streams with tracks that use the peer connection as a source. These two media streams are connected to two <video> element sinks (Y and Z).

Note that at this moment, all of the sinks on the home client must apply a transformation to the original source's provided state dimensions. A is scaling the video up (resulting in loss of quality), B is scaling the video down, and C is also scaling the video up slightly for sending over the network. On the away client, sink Y is scaling the video way down, while sink Z is not applying any scaling.

Using the constraint APIs, the home client's video source is changed to a higher resolution (1920 by 1200 pixels).

Note that the source change immediately affects all of the sinks on the home client, but does not impact any of the sinks (or sources) on the away client. With the increase in the home client source video's dimensions, sink A no longer has to perform any scaling, while sink B must scale down even further than before. Sink C (the peer connection) must now scale down the video in order to keep the transmission constant to the away client.

While not shown, an equally valid states change request could be made of the away client video source (the peer connection on the away client's side). This would not only impact sink Y and Z in the same manner as before, but would also cause re-negotiation with the peer connection on the home client in order to alter the transformation that it is applying to the home client's video source. Such a change would not change anything related to sink A or B or the home client's video source.

Note that this specification does not define a mechanism by which a change to the away client's video source could automatically trigger a change to the home client's video source. Implementations may choose to make such source-to-sink optimizations as long as they only do so within the constraints established by the application, as the next example demonstrates.

It is fairly obvious that changes to a given source will impact sink consumers. However, in some situations changes to a given sink may also be cause for implementations to adjust the characteristics of a source's stream. This is illustrated in the following figures. In the first figure below, the home client's video source is sending a video stream sized at 1920 by 1200 pixels. The video source is also unconstrained, such that the exact source dimensions are flexible as far as the application is concerned. Two MediaStream objects contain tracks with the same sourceId, and those MediaStreams are connected to two different <video> element sinks A and B. Sink A has been sized to width="1920" and height="1200" and is displaying the source's video content without any transformations. Sink B has been sized smaller and, as a result, is scaling the video down to fit its rectangle of 320 pixels across by 200 pixels down.

When the application changes sink A to a smaller dimension (from 1920 to 1024 pixels wide and from 1200 to 768 pixels tall), the browser's media pipeline may recognize that none of its sinks require the higher source resolution, and needless work is being done both on the part of the source and on sink A. In such a case and without any other constraints forcing the source to continue producing the higher resolution video, the media pipeline MAY change the source resolution:

In the above figure, the home client's video source resolution was changed to the greater of that from sinkA and from sinkB in order to optimize playback. While not shown above, the same behavior could apply to peer connections and other sinks.

It is possible that constraints can be applied to a track which a source is unable to satisfy. When this happens, the user agent is required to fire an "overconstrained" event to the track informing it of this condition, and the track becomes muted. There is no mandatory side-effect on the source itself as a result of this condition.

When multiple tracks share the same source (as illustrated in the previous figures), it is also possible that two (or more) tracks can apply contradictory constraints on the source. Since there is only a single thread of control, it is always possible for the browser to determine which track created the overconstrained condition. In this situation, the user agent MUST send the "overconstrained" event only to the track that created the condition and MUST not apply any of the constraints newly requested for that track. Here is an example of this behavior.

In this example, two media streams each have a video track that share the same source. The first track has a mandatory constraint forcing on the source's fill light. That track is connected to sink N. Sink N has a width and height of 800 by 600 pixels and is scaling down the source's resolution of 1024 by 768 to fit. The other track initially has no constraints applied; it is connected to sink P. Sink P has a width and height equal to that of the source.

Now, the second track adds a mandatory constraint that the fill light should be forced off. At this point, both mandatory constraints cannot be satisfied by the source (the fill light cannot be simultaneously on and off at the same time). The second track is transitioned into the muted state and receives an "overconstrained" event. At the same time, the source notes that its remaining active sink only requires a resolution of 800 by 600 and so it adjusts its resolution down to match (this is an optional optimization that the user agent is allowed to make given the situation).

At this point, it is the responsibility of the application to fix the problem that led to the overconstrained situation (by either removing the fill light mandatory constraint on the second track, or by changing/removing the fill light mandatory constraint on the first track).

6. Source states

There is a variable associated with each capability that represents the state of the source with respect to that capability, the actual setting in use by the source. In the same way that the current set of constraints can be returned on a track using the constraints() method, the states() method on a track returns the values of the state variables associated with all capabilities, as the MediaSourceStates object. Only states appropriate to the sourceType are returned.

dictionary MediaSourceStates {
    SourceTypeEnum       sourceType;
    DOMString            sourceId;
    unsigned long?       width;
    unsigned long?       height;
    float?               frameRate;
    VideoFacingModeEnum? facingMode;
    unsigned long?       volume;
};

7. Source capabilities

Each constraint that is supported by an implementation MUST have an associated capability that will be returned in the result of a call to the capabilities() method.

Capabilities are provided as either a min/max range or a list of enumerated values. Min/max capabilities are always provided for constraints that are not enumerated types. Listed capabilities are always provided for constraints corresponding to enumerated types.

dictionary CapabilityRange {
    any     max;
    any     min;
    boolean supported;
};

typedef sequence<DOMString> CapabilityList;

dictionary AllVideoCapabilities {
    CapabilityList?  sourceType;
    CapabilityList?  sourceId;
    CapabilityRange? width;
    CapabilityRange? height;
    CapabilityRange? frameRate;
    CapabilityList?  facingMode;
};

dictionary AllAudioCapabilities {
    CapabilityRange? volume;
};

8. URL

partial interface URL {
    static DOMString createObjectURL (MediaStream stream);
};

9. MediaStreams as Media Elements

A MediaStream may be assigned to media elements as defined in HTML5 [HTML5] by calling createObjectURL to obtain a URL for the MediaStream and then setting the media elements src attribute to that URL. A MediaStream is not preloadable or seekable and represents a simple, potentially infinite, linear media timeline. The timeline starts at 0 and increments linearly in real time as long as the MediaStream is playing. The timeline does not increment when the MediaStream is paused.

The nature of the MediaStream places certain restrictions on the behavior and attribute values of the associated media element and on the operations that can be performed on it, as shown below:

• Whenever the user agent runs the media element load algorithm, reaches the resource fetch phase of this algorithm and determines that the media resource in question is a MediaStream, it MUST immediately abort the resource selection algorithm, setting the media.readystate to HAVE_ENOUGH_DATA.
• The UA MUST NOT buffer data from a MediaStream. When playing, the UA MUST always play the current data from the stream.

11. Obtaining local multimedia content

[NoInterfaceObject]
interface NavigatorUserMedia {
    void getUserMedia (MediaStreamConstraints? constraints, NavigatorUserMediaSuccessCallback successCallback, NavigatorUserMediaErrorCallback errorCallback);
};

Navigator implements NavigatorUserMedia;

dictionary MediaStreamConstraints {
    (boolean or MediaTrackConstraints) video = false;
    (boolean or MediaTrackConstraints) audio = false;
};

dictionary MediaTrackConstraints {
    MediaTrackConstraintSet? mandatory = null;
    MediaTrackConstraint[]?  _optional = null;
};

{
  mandatory: {
    width: { min: 640 },
    height: { min: 480 }
  },
  optional: [
    { width: 650 },
    { width: { min: 650 }},
    { frameRate: 60 },
    { width: { max: 800 }},
    { facingMode: "user" }
  ]
}                                

callback NavigatorUserMediaSuccessCallback = void (MediaStream stream);

[NoInterfaceObject]
interface NavigatorUserMediaError {
    readonly    attribute DOMString  name;
    readonly    attribute DOMString? message;
    readonly    attribute DOMString? constraintName;
};

enum NavigatorUserMediaErrorName {
    "PERMISSION_DENIED",
    "CONSTRAINT_NOT_SATISFIED"
};

callback NavigatorUserMediaErrorCallback = void (NavigatorUserMediaError error);

12. Examples

<input type="button" value="Start" onclick="start()" id="startBtn">
<script>
 var startBtn = document.getElementById('startBtn');
 function start() {
   navigator.getUserMedia({audio:true, video:true}, gotStream, logError);
   startBtn.disabled = true;
 }
 function gotStream(stream) {
   stream.onended = function () {
     startBtn.disabled = false;
   };
 }
 function logError(error) {
   log(error.name + ": " + error.message);
 }
</script> 

<article>
 <style scoped>
  video { transform: scaleX(-1); }
  p { text-align: center; }
 </style>
 <h1>Snapshot Kiosk</h1>
 <section id="splash">
  <p id="errorMessage">Loading...</p>
 </section>
 <section id="app" hidden>
  <p><video id="monitor" autoplay></video> <canvas id="photo"></canvas>
  <p><input type=button value="&#x1F4F7;" onclick="snapshot()">
 </section>
 <script>
  navigator.getUserMedia({video:true}, gotStream, noStream);
  var video = document.getElementById('monitor');
  var canvas = document.getElementById('photo');
  function gotStream(stream) {
    video.src = URL.createObjectURL(stream);
    video.onerror = function () {
      stream.stop();
    };
    stream.onended = noStream;
    video.onloadedmetadata = function () {
      canvas.width = video.videoWidth;
      canvas.height = video.videoHeight;
      document.getElementById('splash').hidden = true;
      document.getElementById('app').hidden = false;
    };
  }
  function noStream() {
    document.getElementById('errorMessage').textContent = 'No camera available.';
  }
  function snapshot() {
    canvas.getContext('2d').drawImage(video, 0, 0);
  }
 </script>
</article>

13. IANA Registrations

dictionary MinMaxConstraint {
    any max;
    any min;
};

14. Change Log

This section will be removed before publication.

Changes since May 15, 2013

1. Added explanatory section for constraints, capabilities, and states.
2. Added VideoFacingModeEnum (including left and right options).
3. Added getSourceInfos() and SourceInfo dictionary.

April 29, 2013

1. Removed remaining photo APIs and references (since we have a separate Image Capture Spec).

March 20, 2013

1. Added readonly and remote attributes to MediaStreamTrack
2. Removed getConstraint(), setConstraint(), appendConstraint(), and prependConstraint().
3. Added source states. Added states() method on tracks. Moved sourceType and sourceId to be states.
4. Added source capabilities. Added capabilities() method on tracks.
5. Added clarifying text about MediaStreamTrack lifecycle and mediaflow.
6. Made MediaStreamTrack cloning explicit.
7. Removed takePhoto() and friends from VideoStreamTrack (we have a separate Image Capture Spec).
8. Made getUserMedia() error callback mandatory.

December 12, 2012

1. Changed error code to be string instead of number.
2. Added core of settings proposal allowing for constraint changes after stream/track creation.

November 15 2012

1. Introduced new representation of tracks in a stream (removed MediaStreamTrackList).
2. Updated MediaStreamTrack.readyState to use an enum type (instad of unsigned short constants).
3. Renamed MediaStream.label to MediaStream.id (the definition needs some more work).

October 1 2012

1. Limited the track kind values to "audio" and "video" only (could previously be user defined as well).
2. Made MediaStream extend EventTarget.
3. Simplified the MediaStream constructor.

June 23 2012

1. Rename title to "Media Capture and Streams".
2. Update document to comply with HTML5.
3. Update image describing a MediaStream.
4. Add known issues and various other editorial changes.

June 22 2012

1. Update wording for constraints algorithm.

June 19 2012

1. Added "Media Streams as Media Elements section".

June 12 2012

1. Switch to respec v3.

June 5 2012

1. Added non-normative section "Implementation Suggestions".
2. Removed stray whitespace.

June 1 2012

1. Added media constraint algorithm.

Apr 23 2012

1. Remove MediaStreamRecorder.

Apr 20 2012

1. Add definitions of MediaStreams and related objects.

Dec 21 2011

1. Changed to make wanted media opt in (rather than opt out). Minor edits.

Nov 29 2011

1. Changed examples to use MediaStreamOptions objects rather than strings. Minor edits.

Nov 15 2011

1. Removed MediaStream stuff. Refers to webrtc 1.0 spec for that part instead.

Nov 9 2011

1. Created first version by copying the webrtc spec and ripping out stuff. Put it on github.