Introduction

The two main components in the MediaStream API are the MediaStreamTrack and the MediaStream interfaces. The MediaStreamTrack object represents media originating from a single media source in the user agent, e.g. video from a web camera. A MediaStream is used group several MediaStreamTrack objects into one unit that can be rendered in a media element or recorded.

Each MediaStream can contain zero or more MediaStreamTrack objects. All tracks in a MediaStream are intended to be synchronized when rendered. Different MediaStream objects do not need to be synchronized.

While the intent is to synchronize tracks, it could be better in some circumstances to permit tracks to lose synchronization.In particular, when tracks are remotely sourced and real-time [[!WEBRTC10]], it can be better to allow loss of synchronization than to accumulate delays or risk glitches and other artifacts. Implementations are expected to understand the implications of choices regarding synchronization of playback and the effect that these have on user perception.

A MediaStreamTrack represents content comprising one or more channels, where the channels have a defined well known relationship to each other (such as a stereo or 5.1 audio signal). A channel is the smallest unit considered in this API specification.

A MediaStream object has an input and an output that represent the combined input and output of all the object's tracks. The output of the MediaStream controls how the object is rendered, e.g., what is saved if the object is recorded to a file or what is displayed if the object is used in a video element.

A new MediaStream object can be created from accessible media sources (that does not require any additional permissions) using the MediaStream() constructor. The constructor argument can either be an existing MediaStream object, in which case all the tracks of the given stream are added to the new MediaStream object, or an array of MediaStreamTrack objects. The latter form makes it possible to compose a stream from different source streams.

Both MediaStream and MediaStreamTrack objects can be cloned. This allows for greater control since the separate instances can be manipulated and consumed individually. A cloned MediaStream contains clones of all member tracks from the original stream.

When a MediaStream object is being generated from a local file (as opposed to a live audio/video source), the user agent SHOULD stream the data from the file in real time, not all at once. The MediaStream object is also used in contexts outside getUserMedia, such as [[!WEBRTC10]]. In both cases, ensuring a realtime stream reduces the ease with which pages can distinguish live video from pre-recorded video, which can help protect the user's privacy.

MediaStream

The MediaStream() constructor composes a new stream out of existing tracks. It takes an optional argument of type MediaStream or an array of MediaStreamTrack objects. When the constructor is invoked, the UA must run the following steps:

1. Let stream be a newly constructed MediaStream object.

2. Initialize stream's id attribute to a newly generated value.

3. If the constructor's argument is present, run the sub steps that corresponds to the argument type.

   • Array of MediaStreamTrack objects:

     Run the following sub steps for each MediaStreamTrack in the array:

     1. Add track: Let track be the MediaStreamTrack about to be processed.

     2. Add track to stream's track set.

   • MediaStream :

     Run the sub steps labeled Add track (above) for every MediaStreamTrack in the argument stream's track set.

4. If stream's track set is empty or only contains ended tracks, set stream's active attribute to false, otherwise set it to true.

5. Return stream.

The tracks of a MediaStream are stored in a track set. The track set MUST contain the MediaStreamTrack objects that correspond to the tracks of the stream. The relative order of the tracks in the set is user agent defined and the API will never put any requirements on the order. The proper way to find a specific MediaStreamTrack object in the set is to look it up by its id .

An object that reads data from the output of a MediaStream is referred to as a MediaStream consumer. The list of MediaStream consumers currently include the media elements [[HTML5]], RTCPeerConnection [[WEBRTC10]], MediaRecorder [[mediastream-rec]] and ImageCapture [[mediastream-imagecap]].

MediaStream consumers must be able to handle tracks being added and removed. This behavior is specified per consumer.

A MediaStream object is said to be MediaStream.inactive when it does not have any tracks or all tracks belonging to the stream have ended. Otherwise the stream is active. A MediaStream can start its life as inactive if it is constructed without any tracks.

When a MediaStream goes from being active to inactive, the user agent MUST queue a task that sets the object's active attribute to false and fire a simple event named inactive at the object. When a MediaStream goes from being inactive to active, the user agent MUST queue a task that sets the object's active attribute to true and fire a simple event named active at the object.

If the stream's activity status changed due to a user request, the task source for this task is the user interaction task source. Otherwise the task source for this task is the networking task source.

Constructor()

See the MediaStream constructor algorithm

Constructor(MediaStream stream)

See the MediaStream constructor algorithm

Constructor(sequence<MediaStreamTrack> tracks)

See the MediaStream constructor algorithm

readonly attribute DOMString id

When a MediaStream object is created, the user agent MUST generate an identifier string, and MUST initialize the object's id attribute to that string. A good practice is to use an UUID, which is 36 characters long in its canonical form.

The id attribute MUST return the value to which it was initialized when the object was created.

sequence<MediaStreamTrack> getAudioTracks()

Returns a sequence of MediaStreamTrack objects representing the audio tracks in this stream.

The getAudioTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "audio". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

sequence<MediaStreamTrack> getVideoTracks()

Returns a sequence of MediaStreamTrack objects representing the video tracks in this stream.

The getVideoTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set whose kind is equal to "video". The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

sequence<MediaStreamTrack> getTracks()

Returns a sequence of MediaStreamTrack objects representing all the tracks in this stream.

The getTracks() method MUST return a sequence that represents a snapshot of all the MediaStreamTrack objects in this stream's track set, regardless of kind. The conversion from the track set to the sequence is user agent defined and the order does not have to stable between calls.

MediaStreamTrack? getTrackById(DOMString trackId)

The getTrackById() method MUST return the first MediaStreamTrack object in this stream's track set whose id is equal to trackId. The method MUST return null if no track matches the trackId argument.

void addTrack(MediaStreamTrack track)

Adds the given MediaStreamTrack to this MediaStream .

When the addTrack() method is invoked, the user agent MUST run the following steps:

1. Let track be the MediaStreamTrack argument and stream this MediaStream object.

2. If track is already in stream's track set, then abort these steps.

3. Add track to stream's track set.

void removeTrack(MediaStreamTrack track)

Removes the given MediaStreamTrack from this MediaStream .

When the removeTrack() method is invoked, the user agent MUST remove the track, indicated by the method's argument, from the stream's track set, if present.

MediaStream clone()

Clones the given MediaStream and all its tracks.

When the MediaStream.clone() method is invoked, the user agent MUST run the following steps:

1. Let streamClone be a newly constructed MediaStream object.

2. Initialize streamClone's id attribute to a newly generated value.

3. Let clonedTracks be a list that contains the result of running MediaStreamTrack.clone() on all the tracks in the stream on which this method was called.

4. Let clonedTracks be streamClone's track set.

5. Return streamClone.

readonly attribute boolean active

The MediaStream.active attribute returns true if the MediaStream is active (see inactive), and false otherwise.

When a MediaStream object is created, its active attribute MUST be set to true, unless stated otherwise (for example by the MediaStream() constructor algorithm).

attribute EventHandler onactive

This event handler, of type active , MUST be supported by all objects implementing the MediaStream interface.

attribute EventHandler oninactive

This event handler, of type inactive , MUST be supported by all objects implementing the MediaStream interface.

attribute EventHandler onaddtrack

This event handler, of type addtrack , MUST be supported by all objects implementing the MediaStream interface.

attribute EventHandler onremovetrack

This event handler, of type removetrack , MUST be supported by all objects implementing the MediaStream interface.

MediaStreamTrack

A MediaStreamTrack object represents a media source in the user agent. Several MediaStreamTrack objects can represent the same media source, e.g., when the user chooses the same camera in the UI shown by two consecutive calls to getUserMedia() .

The data from a MediaStreamTrack object does not necessarily have a canonical binary form; for example, it could just be "the video currently coming from the user's video camera". This allows user agents to manipulate media in whatever fashion is most suitable on the user's platform.

A script can indicate that a track no longer needs its source with the MediaStreamTrack.stop() method. When all tracks using a source have been stopped, the given permission for that source is revoked and the source is stopped. If the data is being generated from a live source (e.g., a microphone or camera), then the user agent SHOULD remove any active "on-air" indicator for that source. If the data is being generated from a prerecorded source (e.g. a video file), any remaining content in the file is ignored. An implementation may use a per source reference count to keep track of source usage, but the specifics are out of scope for this specification.

If there is no stored permission to use that source, the UA SHOULD also remove the "permission granted" indicator for the source.

MediaStreamTrackEvent

The addtrack and removetrack events use the MediaStreamTrackEvent interface.

Firing a track event named e with a MediaStreamTrack track means that an event with the name e, which does not bubble (except where otherwise stated) and is not cancelable (except where otherwise stated), and which uses the MediaStreamTrackEvent interface with the track attribute set to track, MUST be created and dispatched at the given target.

Constructor(DOMString type, MediaStreamTrackEventInit eventInitDict)

Constructs a new MediaStreamTrackEvent .

readonly attribute MediaStreamTrack track

The track attribute represents the MediaStreamTrack object associated with the event.

MediaStreamTrack track

Direct Assignment to Media Elements

UAs that support this specification MUST support the following partial interface, which allows a MediaStream to be assigned directly to a media element.

attribute MediaStream? srcObject

Holds the MediaStream that provides media for this element. This attribute overrides both the src attribute and any <source> elements. Specifically, if srcObject is specified, the UA MUST use it as the source of media, even if the src attribute is also set or <source> children are present. If the value of srcObject is replaced or set to null the UA MUST re-run the media element load algorithm

We may want to allow direct assignment of other types as well

Loading and Playing a MediaStream in a Media Element

The UA runs the media element load algorithm to obtain media for the media element to display. As defined in the [[HTML5]] specification, this algorithm has two basic phases: resource selection algorithm chooses the resource to play and resolves its URI. Then the resource fetch phase loads the resource. Both these phases are potentially simplified when using a MediaStream. First of all, srcObject takes priority over other means of specifying the resource, and it provides the object itself rather than a URI. Therefore, there is no need to run the resource selection algorithm. Secondly, when the UA reaches the resource fetch algorithm with a MediaStream, the MediaStream is a local object so there's nothing to fetch. Therefore, the following modifications/restrictions to the media element load algorithm apply:

• Whenever the user agent runs the media element load algorithm, if srcObject is specified, the UA must immediately go to the resource fetch phase of the algorithm.

• Whenever the user agent runs the media element load algorithm, reaches the resource fetch phase of this algorithm, and determines that the media resource in question is a MediaStream, it MUST immediately abort the resource selection algorithm, setting the media.readyState to HAVE_NOTHING if media is not yet available and to HAVE_ENOUGH_DATA once it is.

• For each MediaStreamTrack in the MediaStream , including those that are added after the UA enters the media element load algorithm, the UA MUST create a corresponding AudioTrack or VideoTrack as defined in [[HTML5]]. Since the order in the MediaStream 's track set is undefined, no requirements are put how the AudioTrackList and VideoTrackList are ordered.

  The properties of the AudioTrack and VideoTrack objects MUST be initialized as follows. Let

  • AudioTrack.id and VideoTrack.id have the value of the corresponding MediaStreamTrack.id attribute

  • AudioTrack.kind and VideoTrack.kind be "main"

  • AudioTrack.label and VideoTrack.label have the value of the corresponding MediaStreamTrack.label attribute

  • AudioTrack.language and VideoTrack.language be the empty string

  Let the media resource, represented by the MediaStream object, indicate to the media element load algorithm that all audio tracks and all live video tracks (represented by a MediaStreamTrack with the readyState attribute set to live) should be enabled. This allows the media element load algorithm to set AudioTrack.enabled, VideoTrack.selected and VideoTrackList.selectedIndex accordingly.

  (Note that since the MediaStream is potentially endless, the UA does not exit the media element load algorithm until the MediaStream moves from the active to the inactive state.)

• If a MediaStreamTrack is removed from a MediaStream , played by a media element, the corresponding AudioTrack or VideoTrack MUST be removed as well.

• The UA MUST NOT buffer data from a MediaStream. When playing, the UA MUST always play the current data from the stream.

• When the MediaStream is moves from the active to the inactive state, the UA MUST raise an ended event on the media element and set its ended attribute to true. Note that once ended equals true the media element will not play media even if new Tracks are added to the MediaStream (causing it to return to the active state) unless autoplay is true or the JavaScript restarts the element, e.g., by calling play().

Media Element Attributes when Playing a MediaStream

The nature of the MediaStream places certain restrictions on the behavior and attribute values of the associated media element and on the operations that can be performed on it, as shown below:

NavigatorUserMedia

readonly attribute MediaDevices mediaDevices

Returns the MediaDevices object associated with this Navigator object.

MediaDevices

The MediaDevices object which is the entry point to the API used to examine and get access to media devices available to the user agent.

When a new media input or output device is made available, the user agent MUST queue a task fires a simple event named devicechange at the MediaDevices object.

attribute EventHandler ondevicechange

This event handler, of type devicechange , MUST be supported by all objects implementing the MediaDevices interface.

void enumerateDevices (MediaDeviceInfoCallback resultCallback)

Collects information about the user agents available media input and output devices. The method MUST only return information that the script is authorized to access (TODO expand authorized).

When the enumerateDevices() method is called, the user agent must queue a task that runs the following steps:

1. Let resultCallback be the callback indicated by the methods first argument.

2. If this method has been called previously within this application session, let oldList be the list of MediaDeviceInfo objects that was produced at that call (resultList); otherwise, let oldList be an empty list.

3. Let resultList be an empty list.

4. Probe the user agent for available media devices, and run the following sub steps for each discovered device, device:

   1. If device is represented by a MediaDeviceInfo object in oldList, append that object to resultList, abort these steps and continue with the next device (if any).

   2. Let deviceInfo be a new MediaDeviceInfo object to represent device.

   3. If device belongs to the same physical device as a device, already represented in oldList or resultList, initialize deviceInfo's groupId member to the groupId value of the existing MediaDeviceInfo object. Otherwise, let deviceInfo's groupId member be a newly generated unique identifier

   4. Append deviceInfo to resultList.

5. Invoke resultCallback with resultList as its argument.

Device Info

sequence<MediaDeviceInfo> deviceInfoList

A sequence of MediaDeviceInfo objects representing the result of a call to MediaDevices.enumerateDevices() .

DOMString deviceId

The unique id for the represented device.

MediaDeviceKind kind

Describes the kind of the represented device.

DOMString label

A label describing this device (for example "External USB Webcam"). If the device has no associated label, then this dictionary member MUST return the empty string.

DOMString groupId

Returns the group identifier of the represented device. Two devices have the same group identifier if they belong to the same physical device; for example a headset.

audioinput

Represents an audio input device; for example a microphone.

audiooutput

Represents an audio output device; for example a pair of headphones.

videoinput

Represents a video input device; for example a webcam.

NavigatorUserMedia Interface Extensions

void getUserMedia(MediaStreamConstraints constraints, NavigatorUserMediaSuccessCallback successCallback, NavigatorUserMediaErrorCallback errorCallback)

See MediaDevices.getUserMedia().

MediaDevices Interface Extensions

The getSupportedConstraints method is provided to allow the application to determine which constraints the User Agent recognizes.

static Dictionary getSupportedConstraints(DOMString kind)

Returns a dictionary whose members are the constraint keys known to the User Agent for the kind given as argument. A supported constraint MUST be represented by a member whose name is the constraint name and whose value is true. Any constraint names not supported by the User Agent MUST not be present in the returned dictionary.

void getUserMedia(MediaStreamConstraints constraints, NavigatorUserMediaSuccessCallback successCallback, NavigatorUserMediaErrorCallback errorCallback)

Prompts the user for permission to use their Web cam or other video or audio input.

(Remove when other issues are removed. This is only here to keep the issues from being renumbered)

The constraints argument is an object of type MediaStreamConstraints .

The successCallback will be invoked with a suitable MediaStream object as its argument if the user accepts valid tracks as described below.

The errorCallback will be invoked if there is a failure in finding valid tracks or if the user denies permission, both as described below.

When the getUserMedia() method is called, the user agent MUST run the following steps:

1. Let constraints be the method's first argument.

2. Let successCallback be the callback indicated by the method's second argument.

3. Let errorCallback be the callback indicated by the method's third argument.

4. Let requestedMediaTypes be the set of media types in constraints with either a dictionary value or a value of "true".

5. If requestedMediaTypes is the empty set, let error be a new MediaStreamError object whose name attribute has the value NotSupportedError and jump to the step labeled Error Task below.

6. Let finalSet be an (initially) empty set.

7. If successCallback is null, abort these steps.

8. For each media type T in requestedMediaTypes,

   1. Let candidateSet be all possible tracks of media type T that the browser could return.

   2. If candidateSet is the empty set, let error be a new MediaStreamError object whose name attribute has the value NotFoundError and jump to the step labeled Error Task below.

   3. If the value of the T entry of constraints is "true", jump to the step labeled final below. Otherwise, continue with CS set to the value of the T entry of constraints.

   4. For each required ('min', 'max', or 'exact') constraint provided for a constraint name in CS,

      1. If the constraint is not supported by the browser, jump to the step labeled Constraint Failure below.

      2. Remove from the candidateSet any track that cannot satisfy the value given for the constraint in CS, if any.

      3. If the candidateSet no longer contains at least one track, jump to the step labeled Constraint Failure below. Otherwise, continue with the next required constraint.

   5. Let the secondPassSet be the current contents of the candidateSet. Note that unknown properties are discarded by WebIDL, which means that unknown/unsupported required constraints will silently disappear. To avoid this being a surprise, application authors are expected to first use the getSupportedConstraints() method as shown in the Examples.

   6. For each constraint key-value pair in the "advanced" sequence of CS, in order,

      1. If the constraint is not supported by the browser, skip it and continue with the next constraint.

      2. Remove from the secondPassSet any tracks that cannot satisfy the value given for the constraint.

      3. If the secondPassSet is now empty, let the secondPassSet be the current contents of the candidateSet. Otherwise, let the candidateSet be the current contents of the secondPassSet.

   7. Let the thirdPassSet be the current contents of the candidateSet.

   8. For all non-required ('ideal' or bare-value) constraints in CS, identify the maximum number of such constraint pairs that could be satisfied by at least one track in thirdPassSet.

   9. The decision of which of these non-required constraints to satisfy is completely up to the user agent, as long as the number of constraints satisfied matches the number in the previous step.

   10. For the non-required constraints the user agent has decided to satisfy, remove from the thirdPassSet any tracks that cannot satisfy those constraints.

   11. If the thirdPassSet is now empty, let the thirdPassSet be the current contents of the candidateSet. Otherwise, let the candidateSet be the current contents of the thirdPassSet.

   12. Final: Add the tracks in the candidateSet to the finalSet.

9. Return, and run the remaining steps asynchronously.

10. Optionally, e.g., based on a previously-established user preference, for security reasons, or due to platform limitations, jump to the step labeled Permission Failure below.

11. Prompt the user in a user agent specific manner for permission to provide the entry script's origin with a MediaStream object representing a media stream.

    The provided media MUST include precisely one track of each media type in requestedMediaTypes from the finalSet. The decision of which tracks to choose from the finalSet is completely up to the user agent and may be determined by asking the user. Once selected, the source of a MediaStreamTrack MUST not change.

    Define the event that should be raised when the user agent changes its choice of track.

    User agents are encouraged to default to using the user's primary or system default camera and/or microphone (when possible) to generate the media stream. User agents MAY allow users to use any media source, including pre-recorded media files.

    If the user grants permission to use local recording devices, user agents are encouraged to include a prominent indicator that the devices are "hot" (i.e. an "on-air" or "recording" indicator), as well as a "device accessible" indicator indicating that the page has been granted access to the source.

    If the user denies permission, jump to the step labeled Permission Failure below. If the user never responds, this algorithm stalls on this step.

    If the user grants permission but a hardware error such as an OS/program/webpage lock prevents access, jump to the step labeled Unavailable Failure below.

    If the user grants permission but device access fails for any reason other than those listed above, jump to the step labeled General Failure below.

12. Let stream be the MediaStream object for which the user granted permission.

13. Queue a task to invoke successCallback with stream as its argument.

14. Abort these steps.

15. Permission Failure: Let error be a new MediaStreamError object whose name attribute has the value PermissionDeniedError and jump to the step labeled Error Task below.

16. Constraint Failure: Let error be a new MediaStreamError object whose name attribute has the value ConstraintNotSatisfiedError and whose constraintName attribute is set to the name of the constraint that caused the error.

17. Unavailable Failure: Let error be a new MediaStreamError object whose name attribute has the value SourceUnavailable and jump to the step labeled Error Task below.

18. General Failure: Let error be a new MediaStreamError object whose name attribute has the value AbortError and jump to the step labeled Error Task below.

19. Error Task: Queue a task to invoke errorCallback with error as its argument.

20. The task source for these tasks is the user interaction task source.

MediaStreamConstraints

The MediaStreamConstraints dictionary is used to instruct the UA what sort of MediaStreamTracks to include in the MediaStream returned by getUserMedia().

(boolean or MediaTrackConstraints) video = false

If true, it requests that the returned MediaStream contain a video track. If a Constraints structure is provided, it further specifies the nature and settings of the video Track. If false, the MediaStream MUST not contain a video Track.

(boolean or MediaTrackConstraints) audio = false

If true, it requests that the returned MediaStream contain an audio track. If a Constraints structure is provided, it further specifies the nature and settings of the audio Track. If false, the MediaStream MUST not contain an audio Track.

NavigatorUserMediaSuccessCallback

MediaStream stream

Add explanation of handleEvent

NavigatorUserMediaErrorCallback

MediaStreamError error

Add explanation of handleEvent

Interface Definition

Due to the limitations of the interface definition language used in this specification, it is not possible for other interfaces to inherit or implement ConstrainablePattern. Therefore the WebIDL definitions given are only templates to be copied. Each interface that wishes to make use of the functionality defined here will have to provide its own copy of the WebIDL for the functions and interfaces given here. However it can refer to the semantics defined here, which will not change. See MediaStreamTrack Interface Definition for an example of this.

Capabilities getCapabilities()

The getCapabilities() method returns the dictionary of the capabilities that the object supports.

It is possible that the underlying hardware may not exactly map to the range defined in the registry entry. Where this is possible, the entry should define how to translate and scale the hardware's setting onto the values defined in the entry. For example, suppose that a registry entry defines a hypothetical fluxCapacitance capability that is defined to be the range from -10 (min) to 10 (max), but there are common hardware devices that support only values of "off" "medium" and "full". The registry entry might specify that for such hardware, the user agent should map the range value of -10 to "off", 10 to "full", and 0 to "medium". It might also indicate that given a ConstraintSet imposing a strict value of 3, the user agent should attempt to set the value of "medium" on the hardware, and and that getSettings() should return a fluxCapacitance of 0, since that is the value defined as corresponding to "medium".

Constraints getConstraints()

The getConstraints method returns the Constraints that were the argument to the last successful call of applyConstraints(), maintaining the order in which they were specified. Note that some of the optional ConstraintSets returned may not be currently satisfied. To check which ConstraintSets are currently in effect, the application should use getSettings.

Settings getSettings()

The getSettings() method returns the current settings of all the properties of the object, whether they are platform defaults or have been set by applyConstraints(). Note that the actual setting of a property must be a single value.

void applyConstraints()

Constraints constraints

A new constraint structure to apply to this object.

VoidFunction successCallback

Called if the required constraints can be satisfied.

ConstraintErrorCallback errorCallback

Called if the required constraints cannot be satisfied.

The applyConstraints() algorithm for applying constraints is stated below. Here are some preliminary definitions that are used in the statement of the algorithm:

• We refer to each element of a ConstraintSet (other than the special term 'advanced') as a 'constraint' since it is intended to constrain the corresponding Capability of the ConstrainablePattern object to a value that is within the range or list of values it specifies.
• We refer to the "effective Capability" C of an object O as the possibly proper subset of the possible values of C (as returned by getCapabilities) taking into consideration environmental limitations and/or restrictions placed by other constraints. For example given a ConstraintSet that constrains Capabilities aspectRatio, height and width, the values assigned to any two of the Capabilities limit the effective Capability of the third. The set of effective Capabilities may be platform dependent. For example, on a resource-limited device it may not be possible to set Capabilities C1 and C2 both to 'high', while on another less limited device, this may be possible.
• A set of values for the Capabilities of an object O satisfy ConstraintSet CS if each value a) is in the range of the corresponding effective Capability of O, and b) is in the range or list of values specified by the corresponding constraint in CS, if there is one, and c) there is no constraint in CS that does not correspond to a Capability of O. (Note that although this definition ignores the difference between required and optional ConstraintSets, the algorithm below distinguishes between them.)
• A set of ConstraintSets CS1...CSn (n >= 1) can be satisfied by an object O if it is possible to choose a sequence of values for the Capabilities of O that satisfy CS1...CSn simultaneously.
• To apply a set of ConstraintSet CS1...CSn to object O is to choose such a sequence of values that satisfy CS1...CSn and assign them as the settings for the properties of O.

When applyConstraints is called, the UA must queue a task to run the following steps:

1. let newContraints be the argument to this function. Each constraint must specify one or more values (or a range of values) for its property. A property may appear more than once in the list of 'advanced' ConstraintSets. Let requiredConstraints be the set of 'min', 'max', and 'exact' constraints appearing in newConstraints. Let unorderedConstraints be all the other constraints in newConstraints. Note that 'advanced' is not a member of either of these sets since it represents meta-information about constraints. In the following we treat requiredConstraints and unorderedConstraints as ConstraintSets. Note that unknown properties are discarded by WebIDL, which means that unknown/unsupported required constraints will silently disappear. To avoid this being a surprise, application authors are expected to first use the getSupportedConstraints() method as shown in the Examples below.
2. Let object be the ConstrainablePattern object on which this method was called. Let copy be an unconstrained copy of object (i.e., copy should behave as if it were object with all ConstraintSets removed.)
3. If requiredConstraints is non-null and cannot be satisfied by copy, call the errorCallback, passing it a new MediaStreamError with name ConstraintNotSatisfied and constraintName set to any of the required constraints that could not be satisfied, and return. existingConstraints remain in effect in this case.
4. Let successfulConstraints be a list of ConstraintSets, initially containing only requiredConstraints. Iterate over the 'advanced' ConstraintSets in newConstraints in the order in which they were specified. For each ConstraintSet, if it and successfulConstraints together can be satisfied by copy, append it to the rear of successfulConstraints. Otherwise, skip it.
5. In a single operation, remove existingConstraints from object, apply successfulConstraints, then apply the largest subset of unorderedConstraints that can be satisifed. (If more than one 'largest subset' can be satisfied, the UA is free to select which 'largest subset' it chooses.) Finally fire the successCallback. From this point on until applyConstraints() is called successfully again, getConstraints() must return the newConstraints that were passed as an argument to this call.

The UA may choose new settings for the Capabilities of the object at any time. When it does so it must attempt to satisfy the current Constraints, in the manner described in the algorithm above.

The definition of how multiple unorderedConstraints are to be satisfied together is still very much under discussion. Also, please see issue 6 about 'ideal' not yet being defined.

attribute EventHandler onoverconstrained

This event handler, of type overconstrained , must be supported by all objects implementing the ConstrainablePattern pattern.

The UA must raise a MediaStreamErrorEvent named "overconstrained" if changing circumstances at runtime result in it no longer being able to satisfy the requiredConstraints from the currently valid Constraints. This MediaStreamErrorEvent must contain a MediaStreamError whose name is "overconstrainedError", and whose constraintName attribute is set to one of the requiredConstraints that can no longer be satisfied. The message attribute of the MediaStreamError SHOULD contain a string that is useful for debugging. The conditions under which this error might occur are platform and application-specific. For example, the user might physically manipulate a camera in a way that makes it impossible to provide a resolution that satisfies the constraints. The UA MAY take other actions as a result of the overconstrained situation.

An example of Constraints that could be passed into applyConstraints() or returned as a value of constraints is below. It uses the properties defined in the Track property registry.

var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["facingMode"]) {
  // Treat like an error.
}
var constraints = {
  "width": {
    "min": 640
  },
  "height": {
    "min": 480
  },
  "advanced": [{
      "width": 650
    }, {
      "width": {
        "min": 650
      }
    }, {
      "frameRate": 60
    }, {
      "width": {
        "max": 800
      }
    }, {
      "facingMode": "user"
    }]
};

Here is another example, specifically for a video track where I must have a particular camera and have separate preferences for the width and height:

var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["sourceId"]) {
  // Treat like an error.
}
var constraints = {
  sourceId: {"exact": "20983-20o198-109283-098-09812"},
  advanced: [{
      width: {
        min: 800,
        max: 1200
      }
    }, {
      height: {
        min: 600
      }
    }]
};

And here's one for an audio track:

var supports = navigator.mediaDevices.getSupportedConstraints("audio");
if(!supports["sourceId"] || !supports["gain"]) {
  // Treat like an error.
}
var constraints = {
  advanced: [{
      sourceId: "64815-wi3c89-1839dk-x82-392aa"
    }, {
      gain: 0.5
    }]
};

Here's an example of use of ideal:

var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["aspectRatio"] || !supports["facingMode"]) {
  // Treat like an error.
}
navigator.mediaDevices.getUserMedia({
  "video": {
    "width": {"min": 320, "ideal": 1280, "max": 1920},
    "height": {"min": 240, "ideal": 720, "max": 1080},
    "framerate": 30,     // Shorthand for ideal.
    // "facingMode": "environment" would be optional.
    "facingMode": {"exact": "environment"}
  }}, ...);

Here's an example of "I want 720p, but I can accept up to 1080p and down to VGA.":

var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(!supports["width"] || !supports["height"]) {
  // Treat like an error.
}
navigator.mediaDevices.getUserMedia({"video": {
  "width": {"min": 640, "ideal": 1280, "max": 1920},
  "height": {"min": 480, "ideal": 720, "max": 1080},
}}, ...);

Here's an example of "I want a front-facing camera and it must be VGA.":

var supports = navigator.mediaDevices.getSupportedConstraints("video");
if(supports["facingMode"]) {
  navigator.mediaDevices.getUserMedia({"video": {
    "facingMode": {"exact": "user"},
    "width": {"exact": 640},
    "height": {"exact": 480}
  }}, ...);
}

The Property Registry

There is a single IANA registry that defines the constrainable properties of all objects that implement the Constrainable pattern. The registry entries must contain the name of each property along with its set of legal values. The registry entries for MediaStreamTrack are defined below. The syntax for the specification of the set of legal values depends on the type of the values. In addition to the standard atomic types (boolean, long, double, DOMString), legal values include lists of any of the atomic types, plus min-max ranges, as defined below.

List values must be interpreted as disjunctions. For example, if a property 'facingMode' for a camera is defined as having legal values ["left", "right", "user", "environment"], this means that 'facingMode' can have the value "left", the value "right", the value "environment" or the value "user". Similarly Constraints restricting 'facingMode' to ["user", "left", "right"] would mean that the UA should select a camera (or point the camera, if that is possible) so that "facingMode" is either "user", "left", or "right". This Constraint would thus request that the camera not be facing away from the user, but would allow the UA to choose among the other directions.

double max

The maximum legal value of this property.

double min

The minimum value of this Property.

double exact

The exact required value for this property.

double ideal

The ideal (target) value for this property.

long max

The maximum legal value of this property.

long min

The minimum value of this property.

long exact

The exact required value for this property.

long ideal

The ideal (target) value for this property.

(DOMString or sequence<DOMString>>) exact

The exact required value for this property.

(DOMString or sequence<DOMString>>) ideal

The ideal (target) value for this property.

(VideoFacingModeEnum or sequence<VideoFacingModeEnum>>) exact

The exact required value for this property.

(VideoFacingModeEnum or sequence<VideoFacingModeEnum>) ideal

The ideal (target) value for this property.

Capabilities

Capabilities are dictionary containing one or more key-value pairs, where each key must be a constrainable property defined in the registry, and each value must be a subset of the set of values defined for that property in the registry. The exact syntax of the value expression depends on the type of the property but is of type ConstraintValues . The Capabilities dictionary specifies the subset of the constrainable properties and values from the registry that the UA supports. Note that a UA may support only a subset of the properties that are defined in the registry, and may support a subset of the set values for those properties that it does support. Note that Capabilities are returned from the UA to the application, and cannot be specified by the application. However, the application can control the Settings that the UA chooses for Capabilities by means of Constraints.

An example of a Capabilities dictionary is shown below. This example is not very realistic in that a browser would actually be required to support more settings that just these.

{
  "frameRate": {
    "min": 1.0,
    "max": 60.0
  },
  "facingMode": ["user", "environment"]
}

Settings

A Setting is a dictionary containing one or more key-value pairs. It must contain each key returned in getCapabilities(). There must be a single value for each key and the value must a member of the set defined for that property by capabilities(). The Settings dictionary contains the actual values that the UA has chosen for the object's Capabilities. The exact syntax of the value depends on the type of the property.

An example of a Setting dictionary is shown below. This example is not very realistic in that a browser would actually be required to support more settings that just these.

{
  "frameRate": 30.0,
  "facingMode": "user"
}

Constraints and ConstraintSet

Due to the limitiations of WebIDL, interfaces implementing the Constrainable Pattern cannnot simply subclass Constraints and ConstraintSet as they are defined here. Instead they must provide their own definitions that follow this pattern. See MediaTrackConstraints for an example of this.

Each member of a ConstraintSet corresponds to a Capability and specifies a subset of its legal values. Applying a ConstraintSet instructs that UA to restrict the setting of the corresponding Capabilities to the specified values or ranges of values. A given property MAY occur both in the basic Constraints set and in the advanced ConstraintSets list, and MAY occur at most once in each ConstraintSet in the advanced list.

sequence<ConstraintSet> advanced

The list of ConstraintSets that the UA must attempt to satisfy, in order, skipping only those that cannot be satisfied. The order of these ConstraintSets is significant. In particular, when they are passed as an argument to applyConstraints, the UA must try to satisfy them in the order that is specified. Thus if optional ConstraintSets C1 and C2 can be satisfied individually, but not together, then whichever of C1 and C2 is first in this list will be satisfied, and the other will not. The UA must attempt to satisfy all optional ConstraintSets in the list, even if some cannot be satisfied. Thus, in the preceding example, if optional constraint C3 is specified after C1 and C2, the UA will attempt to satisfy C3 even though C2 cannot be satisfied. Note that a given property name may occur only once in each ConstraintSet but may occur in more than one ConstraintSet.

Track Property Registrations

IANA is requested to register the following properties as specified in [[!RTCWEB-CONSTRAINTS]]:

The following constraint names are defined to apply to both video and audio MediaStreamTrack objects:

The following properties are defined to apply only to video MediaStreamTrack objects:

user

The source is facing toward the user (a self-view camera).

environment

The source is facing away from the user (viewing the environment).

left

The source is facing to the left of the user.

right

The source is facing to the right of the user.

Below is an illustration of the video facing modes in relation to the user.

The following properties are defined to apply only to audio MediaStreamTrack objects: