Composite Workflow Operation

ID: composite

Description

The composite operation is used to composite two videos (upper and lower) and an optional watermark into one video, including encoding to different formats. The audio track is taken from both videos by default. Everything is done using FFmpeg. The composition can be done in various layout formats e.g. side by side or picture in picture. The layout has to be defined in JSON format and is described in section "Layout Definition". For some general information about layouts see Opencast Composer Layout Module.

The internal FFmpeg command is using the following filters: scale for scaling the videos, pad for defining the output dimension including the background color, movie for adding additional videos and images and overlay for aligning the videos and images to the output dimension. More info can be found here: https://trac.ffmpeg.org/wiki/FilteringGuide

If both upper and lower tracks have audio, "source-audio-name" can be set to "upper", "lower" or "both" to choose only the audio from one track or both tracks for the composite video.

Sample complex composite filter command

-filter:v "[in]scale=640:480,pad=1920:1080:20:20:black[lower];movie=test.mp4,scale=640:480[upper];movie=watermark.jpg[watermark];[lower][upper]overlay=200:200[video];[video][watermark]overlay=main_w-overlay_w-20:20[out]" sidebyside.mp4

Parameter Table

Tags and flavors can be used in combination.

configuration keys value type (EBNF) example description default value
source-audio-name "lower", "upper" or "both" upper The "name" of track to use as a source audio. both
source-tags-upper String , { "," , String } comp The "tag" of the upper track to use as a source input. EMPTY
source-flavor-upper MediaPackageElementFlavor presenter/trimmed The "flavor" of the upper track to use as a source input. EMPTY
source-tags-lower String , { "," , String } comp The "tag" of the lower track to use as a source input. EMPTY
source-flavor-lower MediaPackageElementFlavor presenter/trimmed The "flavor" of the lower track to use as a source input. EMPTY
source-tags-watermark String , { "," , String } branding The "tag" of the attachment image to use as a source input. EMPTY
source-flavor-watermark MediaPackageElementFlavor image/work The "flavor" of the attachment image to use as a source input. EMPTY
source-url-watermark URL file:///Users/me/logo.jpg The "URL" of the fallback image to use as a source input. EMPTY
target-tags String , { "," , String } composite,archive The tags to apply to the compound video track. EMPTY
* target-flavor MediaPackageElementFlavor composite/delivery The flavor to apply to the compound video track. EMPTY
* encoding-profile String composite The encoding profile to use. EMPTY
* output-resolution width , "x" , height | lower | higher 1920x1080 The resulting resolution of the compound video e.g. 1920x1080. EMPTY
output-background String red The resulting background color of the compound video http://www.ffmpeg.org/ffmpeg-utils.html#Color. black
layout name Json , ";" , Json , [ ";" , Json ] The layout name to use or a semi-colon separated JSON layout definition (lower video, upper video, optional watermark). If a layout name is given than the corresponding layout-{name} key must be defined. EMPTY
layout-single name Json , ";" , Json , [ ";" , Json ] Layout to be used in case of one input video track (see layout) EMPTY
layout-dual name Json , ";" , Json , [ ";" , Json ] Layout to be used in case of two input video tracks (see layout). Defaults to value of layout if not set. EMPTY
layout-{name} Json , ";" , Json , [ ";" , Json ] Define semi-colon separated JSON layouts (lower video, upper video, optional watermark) to provide by name. EMPTY

* mandatory

Notes:

Output Resolution

The output resolution must be specified using the configuration key output-resolution. The output resolution can be either explicitly specified (e.g. 1920x1080) or selected from the lower or upper input video (lower or higher). In case that only a single input track is available, both part-lower and part-higher will refer to that single input track.

Layout Definition

The layout definitions are provided as JSON. Each definition consist of the layout specifications for the lower and upper video and an optional specification for the watermark. The specifications have to be separated by comma.

It is always ensured that the media does not exceed the canvas. Offset and scaling is adjusted appropriately.

A single layout is specified as follows:

{
  // How much of the canvas shall be covered. [0.0 - 1.0]
  // 1.0 means that the media is scaled to cover the complete width of the canvas keeping the aspect ratio.
  "horizontalCoverage": Double,
  // The offset between the anchor points of the media and the canvas
  "anchorOffset": {
    // The anchor point of the media. [0.0 - 1.0]
    // (0.0, 0.0) is the upper left corner, (1.0, 1.0) is the lower right corner.
    // (0.5, 0.5) is the center.
    "referring": {
      "left": Double,
      "top": Double
    },
    // The anchor point of the canvas.
    "reference": {
      "left": Double,
      "top": Double
    },
    // The offset between the two anchor points.
    "offset": {
      "y": Integer,
      "x": Integer
    }
  }
}

// Example.
// The media is scaled to cover the whole width of the canvas and is placed in the upper left corner.
{
  "horizontalCoverage": 1.0,
  "anchorOffset": {
    "referring": {
      "left": 0.0,
      "top": 0.0
    },
    "offset": {
      "y": 0,
      "x": 0
    },
    "reference": {
      "left": 0.0,
      "top": 0.0
    }
  }
}

// Example.
// The media is scaled to cover 20% of the width of the canvas and is placed in the lower right corner
// with an offset of -10px on both x and y axis so that it does not touch the canvas' border.
{
  "horizontalCoverage": 0.2,
  "anchorOffset": {
    "referring": {
      "left": 1.0,
      "top": 1.0
    },
    "offset": {
      "y": -10,
      "x": -10
    },
    "reference": {
      "left": 1.0,
      "top": 1.0
    }
  }
}

Operation Example

<operation
    id="composite"
    description="Composite">
  <configurations>
    <configuration key="source-flavor-upper">presentation/trimmed</configuration>
    <configuration key="source-flavor-lower">presenter/trimmed</configuration>
    <configuration key="source-tags-upper">comp</configuration>
    <configuration key="source-tags-lower">comp</configuration>
    <configuration key="source-tags-watermark">branding</configuration>
    <configuration key="source-flavor-watermark">image/work</configuration>
    <configuration key="source-url-watermark">file:///Users/me/logo.jpg</configuration>
    <configuration key="encoding-profile">composite</configuration>
    <configuration key="target-tags">composite,archive</configuration>
    <configuration key="target-flavor">composite/delivery</configuration>
    <configuration key="output-resolution">1920x1080</configuration>
    <configuration key="output-background">red</configuration>
    <configuration key="layout">topleft</configuration>
    <configuration key="layout-topleft">
      {"horizontalCoverage":1.0,"anchorOffset":{"referring":{"left":1.0,"top":1.0},"offset":{"y":-20,"x":-20},"reference":{"left":1.0,"top":1.0}}};
      {"horizontalCoverage":0.2,"anchorOffset":{"referring":{"left":0.0,"top":0.0},"offset":{"y":-20,"x":-20},"reference":{"left":0.0,"top":0.0}}};
      {"horizontalCoverage":1.0,"anchorOffset":{"referring":{"left":1.0,"top":0.0},"offset":{"y":20,"x":20},"reference":{"left":1.0,"top":0.0}}}
    </configuration>
    <configuration key="layout-topright">
      {"horizontalCoverage":1.0,"anchorOffset":{"referring":{"left":1.0,"top":1.0},"offset":{"y":-20,"x":-20},"reference":{"left":1.0,"top":1.0}}};
      {"horizontalCoverage":0.2,"anchorOffset":{"referring":{"left":1.0,"top":0.0},"offset":{"y":-20,"x":-20},"reference":{"left":1.0,"top":0.0}}};
      {"horizontalCoverage":1.0,"anchorOffset":{"referring":{"left":0.0,"top":0.0},"offset":{"y":20,"x":20},"reference":{"left":0.0,"top":0.0}}}
    </configuration>
  </configurations>
</operation>

Encoding Profile Variables

Since the composite operation supports diverse forms of input (single video with watermark, composite with or without watermark) that require different ffmpeg filters, the operation handler can make use of certain conditional variables in its encoding profile, similarly to the encode operation.

These variables are defined as ffmpeg.command.<variable name> and can then be used in the ffmpeg.command property. Defining these variables is optional; if it is known that the encoding profile will only be used with specific input, instead of defining conditional variables, the filters can simply be included in the main command.

The following variables are currently supported:

Variable Description
if-single-stream The value is set if there is no upper video.
if-dual-stream The value is set if there is both an upper and a lower video.
if-watermark The value is set if a watermark is configured and the watermark exists.
if-no-watermark The value is set if no watermark is configured / no watermark exists.

Additionally, there are variables that can be used in the ffmpeg command, but not configured, since they are defined by the operation handler:

Variable Description
audioMapping The audio mapping for dual-stream.
scaleLower The dimensions for the lower video.
padLower The padding for the lower video; contains the size of the new composite video as well as the position of the lower video and the background color.
scaleUpper The dimensions for the upper video.
upperFile The path to the upper video file.
upperPosition The offset for the upper video.
watermarkFile The path to the watermark file.
watermarkPosition The position of the watermark.

This example encoding profile uses all of those variables:

profile.composite.http.name = composite
profile.composite.http.input = visual
profile.composite.http.output = visual
profile.composite.http.suffix = -compound.mkv
profile.composite.http.ffmpeg.command.if-single-stream = -filter:v [in]scale=#{scaleLower},pad=#{padLower}
profile.composite.http.ffmpeg.command.if-dual-stream = -i #{upperFile} -filter_complex \
  [0:v]scale=#{scaleLower},pad=#{padLower}[lower];[1:v]scale=#{scaleUpper}[upper];[lower][upper]overlay=#{upperPosition}
profile.composite.http.ffmpeg.command.if-watermark = \
  [video];movie=#{watermarkFile}[watermark];[video][watermark]overlay=#{watermarkPosition}[out]
profile.composite.http.ffmpeg.command.if-no-watermark = [out]
profile.composite.http.ffmpeg.command = -i #{in.video.path} \
  #{if-single-stream}#{if-dual-stream}#{if-watermark}#{if-no-watermark}#{audioMapping} \
  -c:v libx264 -crf 10 -preset fast \
  -c:a flac \
  #{out.dir}/#{out.name}#{out.suffix}