Minor fixes to westley and mlt_consumer; first draft westley docs
[melted] / docs / westley.txt
diff --git a/docs/westley.txt b/docs/westley.txt
new file mode 100644 (file)
index 0000000..5e37b0a
--- /dev/null
@@ -0,0 +1,280 @@
+WESTLEY
+-------
+
+Preamble:
+
+       Westley is the MLT projects XML serialisation/deserialisation format -
+       as such, it closely mirrors the internal structure of the MLT API. 
+
+
+Introduction:
+
+       A westley document is essentially a list of 'producers' - a producer is
+       an mlt object which generates mlt frames (images and associated audio
+       samples). 
+       
+       There are 3 types of producer:
+       
+       * Basic Producers - these are typically file or device oriented feeds; 
+       * Playlists - these are arrangements of multiple producers;
+       * Tractors - these are multitrack fx encapsulators.
+
+       Although westley was defined as a serialisation mechanism for running MLT 
+       components, this document will concentrate on the hand authoring of westley
+       documents.
+
+
+Rules:
+
+       As shall become apparent through the remainder of this document, the basic
+       tenet of westley authoring is to organise the document in the following
+       manner:
+
+       1) create producer elements for each unique media clip in the project;
+       2) create playlists for each track;
+       3) create a tractor for track specific filters and transitions;
+       4) create another tractor for filters that are common to the output.
+
+       While other uses of westley exist, the approach taken here is to maximise
+       efficiency for complex projects. 
+
+
+Basic Producers:
+
+       The simplest westley document is:
+
+       <westley>
+         <producer id="producer0">
+           <property name="resource">clip1.dv</property>
+         </producer>
+       </westley>
+
+       The westley wrapping is of course superfluous here - loading this document
+       with MLT is identical to loading the clip directly. 
+
+       Of course, you can specify additional properties. For example, consider an
+       MPEG file with multiple soundtracks - you could define a westley document to
+       ensure that the second audio track is loaded:
+
+       <westley>
+         <producer id="producer0">
+           <property name="resource">clip1.mpeg</property>
+           <property name="audio_track">1</property>
+         </producer>
+       </westley>
+
+       NB: This relies on the mpeg being handled by the avformat producer, rather
+       than the mcmpeg one. See services.txt for more details.
+
+       A more useful example comes with the pango producer for a text producer.
+
+       TODO: pango example...
+
+       Notes:
+
+       1) It is better not to specify in/out points when defining basic producers
+       as these can be specified in the playlists. The reasoning is that in/out
+       restricts the amount of the clip available, and could lead to the same clip
+       being loaded multiple times if you need different regions of the clip
+       elsewhere;
+       2) A westley can be specified as a resource, so westleys can naturally
+       encapsulate other westleys.
+
+
+Playlists:
+
+       Playlists provide a 'collection' structure for producers. These can be used
+       to define 'tracks' in the multitrack approach, or simple playlists for
+       sequential, single track playout.
+
+       As an example, the following defines two basic producers and a playlist with 3
+       items:
+
+       <westley>
+         <producer id="producer0">
+           <property name="resource">clip1.dv</property>
+         </producer>
+         <producer id="producer1">
+           <property name="resource">clip2.dv</property>
+         </producer>
+         <playlist id="playlist0">
+           <entry producer="producer0" in="0" out="2999"/>
+           <entry producer="producer1" in="0" out="999"/>
+           <entry producer="producer0" in="3000" out="6999"/>
+         </playlist>
+       </westley>
+
+       Here we see how the playlist defines the in/out points of the basic
+       producers.
+
+       Notes:
+
+       1) All in/out points are absolute frame positions - we support PAL and
+       NTSC at a system level, but westley documents are currently authored
+       for a specific normalisation;
+       2) The last 'producer' in the document is the default for play out;
+       3) Playlists can reference the same producer multiple times. In/out regions
+       do not need to be contiguous - duplication and skipping is acceptable.
+
+
+Interlude - Introducing Tractors:
+
+       So far, we've defined basic producers and playlists/tracks - the tractor is
+       the element that allows us to arrange our tracks and specify filters and
+       transitions. Similarly to a playlist, a tractor is a container.
+
+       Note that MLT doesn't see a filter or a transition as a producer in the
+       normal sense - filters and transitions are passive when it comes to seeking.
+       Internally, seeks are carried out on the producers. This is an important
+       point - MLT does not follow a traditional graph oriented model.
+
+       Visualising an MLT tractor and it's interaction with the consumer will
+       assist here:
+
+       +----------------------------------------------+
+       |tractor                                       |
+       | +----------+    +-+    +-+    +-+    +-+     |
+       | |multitrack|    |f|    |f|    |t|    |t|     |
+       | | +------+ |    |i|    |i|    |r|    |r|     |
+       | | |track0|-|--->|l|- ->|l|- ->|a|--->|a|\    |
+       | | +------+ |    |t|    |t|    |n|    |n| \   |
+       | |          |    |e|    |e|    |s|    |s|  \  |
+       | | +------+ |    |r|    |r|    |i|    |i|   \ |    +--------+
+       | | |track1|-|- ->|0|--->|1|--->|t|--->|t|-----|--->|consumer|
+       | | +------+ |    | |    | |    |i|    |i|   / |    +--------+
+       | |          |    | |    | |    |o|    |o|  /  |         ^
+       | | +------+ |    | |    | |    |n|    |n| /   |         |
+       | | |track2|-|- ->| |- ->| |--->|0|- ->|1|/    |         |
+       | | +------+ |    | |    | |    | |    | |     |         |
+       | +----------+    +-+    +-+    +-+    +-+     |         |
+       +----------------------------------------------+         |
+              ^                                                 |
+              |                                                 |
+       +-----------+                                            |
+       |APPLICATION|--------------------------------------------+
+       +-----------+
+
+       Internally, all frames from all tracks pass through all the filters and
+       transitions - these are told which tracks to deal and which regions of the
+       tracks to work on. 
+
+       Note that the application communicates with the producer - it can alter
+       playback speed, position, or even which producer is connected to which
+       consumer. 
+       
+       In a later phase of MLT development, the application will be able to 
+       manipulate the tractors make up, by adding and removing tracks, filters
+       and transitions. The consumer itself remains connected to the same object 
+       (the tractor).
+
+       The consumer receives the first non-blank frame (see below). It has no say
+       in the order in which gets them (the sdl consumer when used with inigo might 
+       appear to be an exception - it isn't - it simply has a route back to the 
+       application to allow the application to interpret key presses).
+
+       Whether this is better or worse than a traditional graph approach is a moot
+       point. The author of this document likes it anyway :-).
+
+
+Tractors:
+
+       To create a multitrack westley, we can use two playlists and introduce a
+       tractor. For the purposes of demonstration, I'll add a filter here too:
+
+       <westley>
+         <producer id="producer0">
+           <property name="resource">clip1.dv</property>
+         </producer>
+         <producer id="producer1">
+           <property name="resource">clip2.dv</property>
+         </producer>
+         <playlist id="playlist0">
+           <entry producer="producer0" in="0" out="2999"/>
+           <blank length="1000"/>
+           <entry producer="producer0" in="3000" out="6999"/>
+         <playlist id="playlist1">
+           <blank length="3000"/>
+           <entry producer="producer1" in="0" out="999"/>
+         </playlist>
+         <tractor id="tractor0">
+           <multitrack>
+             <track producer="playlist0"/>
+             <track producer="playlist1"/>
+           </multitrack>
+           <filter>
+             <property name="track">0</property>
+             <property name="mlt_service">greyscale</property>
+           </filter>
+         </tractor>
+       </westley>
+       
+       Here we see that blank frames are inserted into the first playlist and a
+       blank is provided at the beginning of the second - this can be visualised in
+       the traditional timeline widget as follows:
+
+       +-------+   +-------------+
+       |a      |   |a            |
+       +-------+---+-------------+
+               |b  |
+               +---+
+       
+       Adding the filter on the top track, gives us:
+
+       +-------+   +-------------+
+       |a      |   |a            |
+       +-------+---+-------------+
+       |greyscale                |
+       --------+---+-------------+
+               |b  |
+               +---+
+       
+       Note that it's only applied to the visible parts of the top track.
+
+       The requirement to apply a filter to the output, as opposed to a specific track 
+       leads us to the final item in the Rules section above. As an example, let's
+       assume we wish to watermark all output, then we could use the following:
+
+       <westley>
+         <producer id="producer0">
+           <property name="resource">clip1.dv</property>
+         </producer>
+         <producer id="producer1">
+           <property name="resource">clip2.dv</property>
+         </producer>
+         <playlist id="playlist0">
+           <entry producer="producer0" in="0" out="2999"/>
+           <blank length="1000"/>
+           <entry producer="producer0" in="3000" out="6999"/>
+         <playlist id="playlist1">
+           <blank length="3000"/>
+           <entry producer="producer1" in="0" out="999"/>
+         </playlist>
+         <tractor id="tractor0">
+           <multitrack>
+             <track producer="playlist0"/>
+             <track producer="playlist1"/>
+           </multitrack>
+           <filter>
+             <property name="track">0</property>
+             <property name="mlt_service">greyscale</property>
+           </filter>
+         </tractor>
+         <tractor id="tractor1">
+           <multitrack>
+             <track producer="tractor0"/>
+           </multitrack>
+           <filter>
+             <property name="mlt_service">watermark</property>
+             <property name="resource">watermark1.png</property>
+           </filter>
+         </tractor>
+       </westley>
+       
+       Here we employ another tractor and we define a single track (being the
+       tractor we previously defined) and apply a watermarking filter there.
+
+       This is simply provided as an example - the watermarking functionality could
+       be better handled at the playout stage itself (ie: as a filter automatically
+       placed between all producers and the consumer).
+
+       TODO: transition example