Media Capture and Streams

4.1 Introduction

The MediaStream interface is used to represent streams of media data, typically (but not necessarily) of audio and/or video content, e.g. from a local camera. The data from a MediaStream object does not necessarily have a canonical binary form; for example, it could just be "the video currently coming from the user’s video camera". This allows user agents to manipulate media streams in whatever fashion is most suitable on the user’s platform.

Each MediaStream object can contain zero or more tracks, in particular audio and video tracks. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStreams do not need to be synchronized.

Each track in a MediaStream object has a corresponding MediaStreamTrack object.

A MediaStreamTrack represents content comprising one or more channels, where the channels have a defined well known relationship to each other (such as a stereo or 5.1 audio signal).

A channel is the smallest unit considered in this API specification.

A MediaStream object has an input and an output. The input depends on how the object was created: a LocalMediaStream object generated by a getUserMedia() call (which is described later in this document), for instance, might take its input from the user’s local camera. The output of the object controls how the object is used, e.g., what is saved if the object is written to a file or what is displayed if the object is used in a video element.

Each track in a MediaStream object can be disabled, meaning that it is muted in the object’s output. All tracks are initially enabled.

A MediaStream can be finished, indicating that its inputs have forever stopped providing data.

The output of a MediaStream object must correspond to the tracks in its input. Muted audio tracks must be replaced with silence. Muted video tracks must be replaced with blackness.

A new MediaStream object can be created from existing MediaStreamTrack objects using the MediaStream() constructor. The constructor takes two lists of MediaStreamTrack objects as arguments: one for audio tracks and one for video tracks. The lists can either be the track lists of another stream, subsets of such lists, or compositions of MediaStreamTrack objects from different MediaStream objects.

The ability to duplicate a MediaStream, i.e. create a new MediaStream object from the track lists of an existing stream, allows for greater control since separate MediaStream instances can be manipulated and consumed individually.

The LocalMediaStream interface is used when the user agent is generating the stream’s data (e.g. from a camera or streaming it from a local video file).

When a LocalMediaStream object is being generated from a local file (as opposed to a live audio/video source), the user agent should stream the data from the file in real time, not all at once. The MediaStream object is also used in contexts outside getUserMedia, such as [WEBRTC10]. In both cases, ensuring a realtime stream reduces the ease with which pages can distinguish live video from pre-recorded video, which can help protect the user’s privacy.

4.2 MediaStream

The MediaStream() constructor takes zero or one argument. If the argument is supplied, it must either be of type MediaStream, an array of MediaStreamTrack objects or null. When the constructor is invoked, the UA must run the following steps:

1. Let stream be a newly constructed MediaStream object.

2. Initialize stream’s label attribute to a newly generated value.

3. If the constructor’s argument is present and not null, run the sub steps that corresponds to the argument type.

• Array of MediaStreamTrack objects:

  Run the following sub steps for each MediaStreamTrack in the array:

  1. Add track: Let track be the MediaStreamTrack about to be processed.

  2. If track has ended or if there is already a MediaStreamTrack contained within stream that has the same underlying source as track, then abort these steps and continue with the next track (if any).

  3. Create a new MediaStreamTrack object and let it inherit track’s underlying source, kind and label attributes. Append the new MediaStreamTrack to the corresponding track list (audioTracks or videoTracks) in stream according to kind.

• MediaStream:

  Run the sub steps labeled Add track (above) for every MediaStreamTrack in the argument stream’s two track lists (audioTracks and videoTracks).

4. Return stream.

A MediaStream can have multiple audio and video sources (e.g. because the user has multiple microphones, or because the real source of the stream is a media resource with many media tracks). The stream represented by a MediaStream thus has zero or more tracks.

The tracks of a MediaStream are stored in two track lists represented by MediaStreamTrackList objects: one for audio tracks and one for video tracks. The two track lists must contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of all tracks in a user agent must be stable. Tracks that come from a media resource whose format defines an order must be in the order defined by the format; tracks that come from a media resource whose format does not define an order must be in the relative order in which the tracks are declared in that media resource. Within these constraints, the order is user agent defined.

An object that reads data from the output of a MediaStream is referred to as a MediaStream consumer. The list of MediaStream consumers currently includes the media elements and the PeerConnection API specified in [WEBRTC10].

A MediaStream object is said to be finished when all tracks belonging to the stream have ended. When this happens for any reason other than the stop() method being invoked, the user agent must queue a task that runs the following steps:

1. If the object’s ended attribute has the value true already, then abort these steps. (The stop() method was probably called just before the stream stopped for other reasons, e.g. the user clicked an in-page stop button and then the user agent provided stop button.)

2. Set the object’s ended attribute to true.

3. Fire a simple event named ended at the object.

If the end of the stream was reached due to a user request, the task source for this task is the user interaction task source. Otherwise the task source for this task is the networking task source.

typedef MediaStreamTrack[] MediaStreamTrackArray;

[Constructor (),
        Constructor (MediaStream? stream),
        Constructor (MediaStreamTrackArray tracks)]
interface MediaStream : EventTarget {
    readonly attribute DOMString            label;
    readonly attribute MediaStreamTrackList audioTracks;
    readonly attribute MediaStreamTrackList videoTracks;
             attribute boolean              ended;
             attribute EventHandler         onended;
};

4.3 LocalMediaStream

Before the web application can access the user's media input devices it must let getUserMedia() create a LocalMediaStream . Once the application is done using, e.g., a webcam and a microphone, it may revoke its own access by calling stop() on the LocalMediaStream.

A web application may, once it has access to a LocalMediaStream , use the MediaStream() constructor to construct additional MediaStream objects. Since a derived MediaStream object is created from the tracks of an existing stream, it cannot use any media input devices that have not been approved by the user.

interface LocalMediaStream : MediaStream {
    void stop ();
};

4.4 MediaStreamTrack

A MediaStreamTrack object represents a media source in the user agent. Several MediaStreamTrack objects can represent the same media source, e.g., when the user chooses the same camera in the UI shown by two consecutive calls to getUserMedia() .

A MediaStreamTrack object can reference its media source in two ways, either with a strong or a weak reference, depending on how the track was created. For example, a track in a MediaStream, derived from a LocalMediaStream with the MediaStream() constructor, has a weak reference to a local media source, while a track in a LocalMediaStream has a strong reference. This means that a track in a MediaStream, derived from a LocalMediaStream, will end if there is no non-ended track in a LocalMediaStream which references the same local media source.

A MediaStreamTrack object is said to end when the user agent learns that no more data will ever be forthcoming for this track.

When a MediaStreamTrack object ends for any reason (e.g., because the user rescinds the permission for the page to use the local camera, or because the data comes from a finite file and the file’s end has been reached and the user has not requested that it be looped, or because the UA has instructed the track to end for any reason, or because the reference count of the track’s underlying media source has reached zero, it is said to be ended. When track instance track ends for any reason other than the stop() method being invoked on the LocalMediaStream object that represents track, the user agent must queue a task that runs the following steps:

1. If the track’s readyState attribute has the value ENDED (2) already, then abort these steps.

2. Set track’s readyState attribute to ENDED (2).

3. Fire a simple event named ended at the object.

If the end of the stream was reached due to a user request, the event source for this event is the user interaction event source.

interface MediaStreamTrack {
    readonly attribute DOMString      kind;
    readonly attribute DOMString      label;
             attribute boolean        enabled;
    const unsigned short LIVE = 0;
    const unsigned short MUTED = 1;
    const unsigned short ENDED = 2;
    readonly attribute unsigned short readyState;
             attribute EventHandler   onmute;
             attribute EventHandler   onunmute;
             attribute EventHandler   onended;
};

4.5 URL

partial interface URL {
    static DOMString createObjectURL (MediaStream stream);
};

4.6 MediaStreamTrackList

A MediaStreamTrackList object’s corresponding MediaStream refers to the MediaStream object which the current MediaStreamTrackList object is a property of.

interface MediaStreamTrackList {
    readonly attribute unsigned long length;
    MediaStreamTrack item (unsigned long index);
    void             add (MediaStreamTrack track);
    void             remove (MediaStreamTrack track);
             attribute EventHandler  onaddtrack;
             attribute EventHandler  onremovetrack;
};

4.7 MediaStreams as Media Elements

A MediaStream may be assigned to media elements as defined in HTML5 [HTML5] by calling createObjectURL to obtain a URL for the MediaStream and then setting the media elements src attribute to that URL. A MediaStream is not preloadable or seekable and represents a simple, potentially infinite, linear media timeline. The timeline starts at 0 and increments linearly in real time as long as the MediaStream is playing. The timeline does not increment when the MediaStream is paused.

The nature of the MediaStream places certain restrictions on the behavior and attribute values of the associated media element and on the operations that can be performed on it, as shown below:

• Whenever the user agent runs the media element load algorithm, reaches the resource fetch phase of this algorithm and determines that the media resource in question is a MediaStream, it must immediately abort the resource selection algorithm, setting the media.readystate to HAVE_ENOUGH_DATA.
• The UA must not buffer data from a MediaStream. When playing, the UA must always play the current data from the stream.

5.1 NavigatorUserMedia

[NoInterfaceObject]
interface NavigatorUserMedia {
    void getUserMedia (MediaStreamConstraints? constraints, NavigatorUserMediaSuccessCallback successCallback, optional NavigatorUserMediaErrorCallback? errorCallback);
};

Navigator implements NavigatorUserMedia;

5.2 MediaStreamConstraints

dictionary MediaStreamConstraints {
    (boolean or MediaTrackConstraints) video = false;;
    (boolean or MediaTrackConstraints) audio = false;;
};

dictionary MediaTrackConstraints {
    MediaTrackConstraintSet? mandatory = null;;
    MediaTrackConstraint[]?  optional = null;;
};

A MediaTrackConstraintSet is a dictionary containing one or more key-value pairs, where each key must be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [RTCWEB-CONSTRAINTS] and its value should be as defined in the associated reference[s] given in the registry.

A MediaTrackConstraint is a dictionary containing exactly one key-value pair, where the key must be a valid registered constraint name in the IANA-hosted RTCWeb Media Constraints registry [RTCWEB-CONSTRAINTS] and the value should be as defined in the associated reference[s] given in the registry.

5.3 NavigatorUserMediaSuccessCallback

callback NavigatorUserMediaSuccessCallback = void (LocalMediaStream stream);

5.4 NavigatorUserMediaError and NavigatorUserMediaErrorCallback

[NoInterfaceObject]
interface NavigatorUserMediaError {
    const unsigned short PERMISSION_DENIED = 1;
    readonly attribute unsigned short code;
};

callback NavigatorUserMediaErrorCallback = void (NavigatorUserMediaError error);

7.1 Constraints Registrations

IANA is requested to register the following constraints as specified in [RTCWEB-CONSTRAINTS]: