Asset Manager

Architecture

Modules

The AssetManager consists of the following modules:

High Level View

TODO Describes components and how they relate.

Classes

TODO Most important classes and how they relate

Default Implementation

AssetStore

Assets are stored in the following directory structure.

$BASE_PATH
 |— <organization_id>
     |— <media_package_id>
         |— <version>
             |— manifest.xml
             |— <media_package_element_id>.<ext>

Database

The asset manager uses four tables

Security

TODO

Usage

Taking Snapshots

TODO

Working with Properties

Properties are associated with an episode, not a single snapshot. They act as annotations helping services to work with saved media packages without having to implement their own storage layer. Properties are typed and can be used to create queries.

Getting Started

Let's start with an fictious example of an ApprovalService. The approval service keeps track of approvals given by an editor to publish a media package. Only approved media packages may be published and the editor should also be able to leave a comment defining a publication as prohibited. Here, three properties are needed, an approval flag, a text field for comments and a time stamp for the date of approval. The following code snippet sets a property on an episode, with am referring to the AssetManager and mp the media package id of type String of the episode.

AssetManager am = …;
String mp = …; // a media package id
am.setProperty(Property.mk(PropertyId.mk(
  mp, "org.opencastproject.approval", "approval"),
  Value.mk(true)));

It is recommended to use namespace names after the service's package name, in the example: org.opencastproject.approval. This code looks overly verbose. Also you need to deal with namespace names and property names directly. That's cumbersome and error prone even though you might intoduce constants for them. To help remedy this situation a little helper class class PropertySchema exists. It is strongly recommended to make use of it. Here's how it goes.

static class ApprovalPops extends PropertySchema {
 public ApprovalProps(AQueryBuilder q) {
   super(q, "org.opencastproject.approval");
 }

 public PropertyField<Boolean> approved() {
   return booleanProp("approved");
 }

 public PropertyField<String> comment() {
   return stringProp("comment");
 }

 public PropertyField<Date> date() {
   return dateProp("date");
 }
}

Now you can set properties like this.

am.setProperty(p.approved().mk(mp, false));
am.setProperty(p.comment().mk(mp, "Audio quality is too poor!"));
am.setProperty(p.date().mk(mp, new Date());

Now, if you want to find all episodes that have been rejected you need to create and run the following query.

AQueryBuilder q = am.createQuery();
AResult r = q.select(q.snapshot()).where(p.approved().eq(true)).run();

This query yields all snapshots of all episodes that have been approved. But that's not exactly what we want as we are only interested in the latest snapshot generated when we re-run the approval process, and resetting all previous approvals.

q.select(q.snapshot())
  .where(p.approved().eq(true).and(q.version().isLatest())
  .run();

This will only return the latest version of each episode. However, along with the information of the approved episodes,we want to display when they were approved. Looking at the AResult and ARecord interfaces it seems that properties need to be selected in order to fetch them.

q.select(q.snapshot(), q.properties())
  .where(p.approved().eq(true).and(q.version().isLatest())
  .run();

Here we go. Now we can access all properties stored with the returned snapshots. Now, let's assume other services make heavy use of properties too. This may cause serious database IO if we always select all properties like we did using the q.properties() target. Let's do better.

q.select(q.snapshot(), q.propertiesOf("org.opencastproject.approval"))
  .where(p.approved().eq(true).and(q.version().isLatest())
  .run();

This will return only the properties of our service's namespace. But do we have to deal with namespace strings again? No.

q.select(q.snapshot(), q.propertiesOf(p.allProperties()))
  .where(p.approved().eq(true).and(q.version().isLatest())
  .run();

Our implementation of PropertySchema provides as with a ready to use target for the properties of our namespace only. In our use case we could reduce IO even further since we're only interested in the date property.

q.select(q.snapshot(), q.propertiesOf(p.date().target()))
  .where(p.approved().eq(true).and(q.version().isLatest())
  .run();

This is the query returns only the latest snapshots of all episodes being approved together with the date of approval. Now that you've seen how to create properties let's move on to delete them again.

Deleting Properties

Properties are deleted pretty much like they are queried, using a delete query.

q.delete(q.propertiesOf(p.allProperties())).run();

The above query deletes all properties that belong to schema p from all episodes. If you want to restrict deletion to a single episode, add an id predicate to the where clause.

q.delete(q.propertiesOf(p.allProperties()))
  .where(q.mediaPackageId(mpId))
  .run();

Deleting just a single property from all episodes is also possible.

q.delete(p.approved()).run();

Or multiple properties at once.

q.delete(p.approved(), p.comment()).run();

Please see the query API documentation for further information.

Value Types

The following type are available for properties:

Decomposing properties

Since properties are type safe they cannot be accessed directly. If you know the type of the property you can access its value using a type evidence constant.

String string = p.getValue().get(Value.STRING);
Boolean bool = p.getValue().get(Value.BOOLEAN);

Type evidence constants are defined in class Value. If the type is unknown since you are iterating a mixed collection of values, for example if you need to decompose the value. Decomposition is the act of pattern matching against the value's type. Each case is handled by a different function, all returning the same type. Let's say you are iterating over a collection of values and want to print them, formatted, to the console. All handle* parameters are functions of type Fn taking the raw value as input and returning a String.

List<Value> vs = …;
for (Value v : vs) {
  String f = v.decompose(
    handleStringFn,
    handleDateFn,
    handleLongFn,
    handleBooleanFn,
    handleVersionFn);
  System.out.println(f);
}

The class org.opencastproject.assetmanager.api.fn.Properties contains various utility functions to help extracting values from properties.

Using PropertySchema

You've already seen that a property is constructed from a media package id, a namespace, a property name and a value. Since this is a bit cumbersome, the API features an abstract base class to construct property schemas. The resulting schema implementations encapsulate all the string constants so that you don't have to deal with them manually. Please see the example in the Getting Started section. It is strongly recommended to work with schemas as much as possible.

Creating and Running Queries

Creating and running a query is a two step process. First, you create a new AQueryBuilder.

AQueryBuilder q = am.createQuery();

Next, you build a query like this.

ASelectQuery s = q.select(q.snapshot())
  .where(q.mediaPackageId(mpId).and(q.version().isLatest());

Now it's time to actually run the query against the database.

AResult r = s.run();

All this can, of course, be done in a single statement, but it has been broken up in several steps to show you the intermediate types.

am.createQuery()
  .select(q.snapshot())
  .where(q.mediaPackageId(mpId).and(q.version().isLatest())
  .run();

The result set r contains the retrieved data encapsulated in stream of ARecord objects. If nothing matched the given predicates then a call to r.getRecords() yields an empty stream. Please note that even though a Stream is returned, it does not mean that the result set is actually streamed—or lazily loaded—from the database. The Stream interface is just far more powerful than the collection types from JCL.

A note on immutability

Please note that all classes of the query API are immutable and therefore safe to be used in a concurrent environment. Whenever you call a factory method on an instance of one of the query classes a new instance is yielded. They never mutate state.

Accessing Query Results

Running a query yields an object of type AResult which in turn yields the found result records. Besides it also provides some general result metadata like the set limit, offset etc. An ARecord holds the found snapshots and properties, depending on the select targets and the predicates. If no snapshots have been selected then, none will be returned here. The same holds true for properties. However, an ARecord instance holding the media package id is created regardless of the requested targets. The typical pattern to access query results is to iterate over the stream of records. This can be accomplished using a simple for loop or one of the functional methods that the Stream type provides, e.g. map over the elements of a stream to create a new one. For easy access to fetched resources you may wrap the result in an enrichment.

AResult r = …;
RichAResult rr = Enrichments.enrich(r);

RichAResult features methods to directly access all fetched snapshots and properties.

Deleting Snapshots

This works exactly like deleting properties, except that you need to specify snapshots instead of properties. Please note that it's also possible to specify snapshots and properties simultanously.

q.delete("owner", q.snapshot()).where(q.version().isLatest().not()).run();

The above query deletes all snapshots but the latest. This is a good query to free up some disc space.

Snapshots can only be deleted per owner.

Query Language Reference

The query API features

Please see the API doc for further information about the various elements and how to create them.