AWS S3 Archive Configuration

This page documents the configuration for the AWS S3 components in the Opencast module asset-manager-storage-aws. This configuration is only required on the admin node, and only if you are using Amazon S3 as an archive repository.

Amazon User Configuration

Configuration of Amazon users is beyond the scope of this documentation, instead we suggest referring to Amazon's documentation. You will, however, require an Access Key ID and Secret Access Key. The user to which this key belongs requires the AmazonS3FullAccess permission, which can be granted using these instructions.

A free Amazon account will work for small scale testing, but be aware that S3 archiving can cost you a lot of money very quickly. Be aware of how much data and how many requests you are making, and be sure to set alarms to notify you of cost overruns.

Amazon Service Configuration

The development and testing it is generally safe to allow the Opencast AWS S3 Archive service to create the S3 bucket for you. It will create the bucket per its configuration, with private-only access to the files, and no versioning.

Opencast Service Configuration

The Opencast AWS S3 Archive service has four configuration keys which can be found in the org.opencastproject.assetmanager.aws.s3.AwsS3ArchiveElementStore.cfg configuration file.

Key name Value Example
org.opencastproject.assetmanager.aws.s3.enabled Whether to enable this service true
org.opencastproject.archive.aws.s3.region The AWS region to set us-west-2
org.opencastproject.archive.aws.s3.bucket The S3 bucket name example-org-archive
org.opencastproject.archive.aws.s3.access.id Your access ID 20 alphanumeric characters
org.opencastproject.archive.aws.s3.secret.key Your secret key 40 characters

Using S3 Archiving

There are two major methods to access S3 archiving features: manually, and via a workflow. Amazon S3 archiving is not part of the default workflows and manual S3 offload is disabled by default. To enable manual S3 offload you must edit the offload.xml workflow configuration file and change var s3Enabled = false; to var s3Enabled = true;. To manually offload a mediapackage follow the directions in the user documentation.

To automatically offload a mediapackage to S3 you must add the move-storage workflow operation to your workflow. The operation documentation can be found here.

Migrating to S3 Archiving with Pre-Existing Data

Archiving to S3 is a non-destructive operation in that it is safe to move archive files back and forth between local storage and S3. To offload your local archive, select the workflow(s) and follow the manual offload steps described in the user documentation.