1 Framework Documentation
3 Copyright (C) 2004 Ushodaya Enterprises Limited
4 Author: Charles Yates <charles.yates@pandora.be>
5 Last Revision: 2004-10-08
13 MLT is a multimedia framework designed for television broadcasting. As such,
14 it provides a pluggable architecture for the inclusion of new audio/video
15 sources, filters, transitions and playback devices.
17 The framework provides the structure and utility functionality on which
18 all of the MLT applications and services are defined.
20 On its own, the framework provides little more than 'abstract classes' and
21 utilities for managing resources, such as memory, properties, dynamic object
22 loading and service instantiation.
24 This document is split roughly into 3 sections. The first section provides a
25 basic overview of MLT, the second section shows how it's used and the final
26 section shows structure and design, with an emphasis on how the system is
32 This document is provided as a 'road map' for the framework and should be
33 considered mandatory reading for anyone wishing to develop code at the MLT
38 1. framework maintainers;
40 3. application developers;
41 4. anyone interested in MLT.
43 The emphasis of the document is in explaining the public interfaces, as
44 opposed to the implementation details.
46 It is not required reading for the MLT client/server integration - please
47 refer to valerie.txt and dvcp.txt for more details on this area.
50 SECTION 1 - BASIC OVERVIEW
51 --------------------------
53 Basic Design Information:
57 The framework has no dependencies other than the standard C99 and POSIX
60 It follows a basic Object Oriented design paradigm, and as such, much of the
61 design is loosely based on the Producer/Consumer design pattern.
63 It employs Reverse Polish Notation for the application of audio and video FX.
65 The framework is designed to be colour space neutral - the currently
66 implemented modules, however, are very much 8bit YUV422 oriented. In theory,
67 the modules could be entirely replaced.
69 A vague understanding of these terms is assumed throughout the remainder of
75 The general structure of an MLT 'network' is simply the connection of a
76 'producer' to a 'consumer':
79 |Producer|-->|Consumer|
82 A typical consumer requests MLT Frame objects from the producer, does
83 something with them and when finished with a frame, closes it.
85 /\ A common confusion with the producer/consumer terminology used here is
86 /!!\ that a consumer may 'produce' something. For example, the libdv consumer
87 \!!/ produces DV and the libdv producer seems to consume DV. However, the
88 \/ naming conventions refer only to producers and consumers of MLT Frames.
90 To put it another way - a producer produces MLT Frame objects and a consumer
91 consumes MLT Frame objects.
93 An MLT Frame essentially provides an uncompressed image and its associated
96 Filters may also be placed between the producer and the consumer:
98 +--------+ +------+ +--------+
99 |Producer|-->|Filter|-->|Consumer|
100 +--------+ +------+ +--------+
102 A service is the collective name for producers, filters, transitions and
105 The communications between a connected consumer and producer or service are
106 carried out in 3 phases:
112 MLT employs 'lazy evaluation' - the image and audio need not be extracted
113 from the source until the get image and audio methods are invoked.
115 In essence, the consumer pulls from what it's connected to - this means that
116 threading is typically in the domain of the consumer implementation and some
117 basic functionality is provided on the consumer class to ensure realtime
126 Before we go in to the specifics of the framework architecture, a working
127 example of usage is provided.
129 The following simply provides a media player:
133 #include <framework/mlt.h>
135 int main( int argc, char *argv[] )
137 // Initialise the factory
138 if ( mlt_factory_init( NULL ) == 0 )
140 // Create the default consumer
141 mlt_consumer hello = mlt_factory_consumer( NULL, NULL );
143 // Create via the default producer
144 mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );
146 // Connect the producer to the consumer
147 mlt_consumer_connect( hello, mlt_producer_service( world ) );
149 // Start the consumer
150 mlt_consumer_start( hello );
152 // Wait for the consumer to terminate
153 while( !mlt_consumer_is_stopped( hello ) )
156 // Close the consumer
157 mlt_consumer_close( hello );
159 // Close the producer
160 mlt_producer_close( world );
163 mlt_factory_close( );
167 // Report an error during initialisation
168 fprintf( stderr, "Unable to locate factory modules\n" );
175 This is a simple example - it doesn't provide any seeking capabilities or
176 runtime configuration options.
178 The first step of any MLT application is the factory initialisation - this
179 ensures that the environment is configured and MLT can function. The factory
180 is covered in more detail below.
182 All services are instantiated via the factories, as shown by the
183 mlt_factory_consumer and mlt_factory_producer calls above. There are similar
184 factories for filters and transitions. There are details on all the standard
185 services in services.txt.
187 The defaults requested here are a special case - the NULL usage requests
188 that we use the default producers and consumers.
190 The default producer is "fezzik". This producer matches file names to
191 locate a service to use and attaches 'normalising filters' (such as scalers,
192 deinterlacers, resamplers and field normalisers) to the loaded content -
193 these filters ensure that the consumer gets what it asks for.
195 The default consumer is "sdl". The combination of fezzik and sdl will
196 provide a media player.
198 In this example, we connect the producer and then start the consumer. We
199 then wait until the consumer is stopped (in this case, by the action of the
200 user closing the SDL window) and finally close the consumer, producer and
201 factory before exiting the application.
203 Note that the consumer is threaded - waiting for an event of some sort is
204 always required after starting and before stopping or closing the consumer.
206 Also note, you can override the defaults as follows:
208 $ MLT_CONSUMER=westley ./hello file.avi
210 This will create a westley xml document on stdout.
212 $ MLT_CONSUMER=westley MLT_PRODUCER=avformat ./hello file.avi
214 This will play the video using the avformat producer directly, thus it will
215 bypass the normalising functions.
217 $ MLT_CONSUMER=libdv ./hello file.avi > /dev/dv1394
219 This might, if you're lucky, do on the fly, realtime conversions of file.avi
220 to DV and broadcast it to your DV device.
225 As shown in the 'Hello World' example, factories create service objects.
227 The framework itself provides no services - they are provided in the form of
228 a plugin structure. A plugin is organised in the form of a 'module' and a
229 module can provide many services of different types.
231 Once the factory is initialised, all the configured services are available
234 The complete set of methods associated to the factory are as follows:
236 int mlt_factory_init( char *prefix );
237 const char *mlt_factory_prefix( );
238 char *mlt_environment( char *name );
239 mlt_producer mlt_factory_producer( char *name, void *input );
240 mlt_filter mlt_factory_filter( char *name, void *input );
241 mlt_transition mlt_factory_transition( char *name, void *input );
242 mlt_consumer mlt_factory_consumer( char *name, void *input );
243 void mlt_factory_close( );
245 The mlt_factory_prefix returns the path to the location of the installed
246 modules directory. This can be specified in the mlt_factory_init call
247 itself, or it can be specified via the MLT_REPOSITORY environment variable,
248 or in the absence of either of those, it will default to the install
249 prefix/shared/mlt/modules.
251 The mlt_environment provides read only access to a collection of name=value
252 pairs as shown in the following table:
254 +------------------+------------------------------------+------------------+
255 |Name |Description |Values |
256 +------------------+------------------------------------+------------------+
257 |MLT_NORMALISATION |The normalisation of the system |PAL or NTSC |
258 +------------------+------------------------------------+------------------+
259 |MLT_PRODUCER |The default producer |"fezzik" or other |
260 +------------------+------------------------------------+------------------+
261 |MLT_CONSUMER |The default consumer |"sdl" or other |
262 +------------------+------------------------------------+------------------+
263 |MLT_TEST_CARD |The default test card producer |any producer |
264 +------------------+------------------------------------+------------------+
266 These values are initialised from the environment variables of the same
269 As shown above, a producer can be created using the 'default normalising'
270 producer, and they can also be requested by name. Filters and transitions
271 are always requested by name - there is no concept of a 'default' for these.
276 As shown in the services.txt document, all services have their own set of
277 properties than can be manipulated to affect their behaviour.
279 In order to set properties on a service, we need to retrieve the properties
280 object associated to it. For producers, this is done by invoking:
282 mlt_properties properties = mlt_producer_properties( producer );
284 All services have a similar method associated to them.
286 Once retrieved, setting and getting properties can be done directly on this
289 mlt_properties_set( properties, "name", "value" );
291 A more complete description of the properties object is found below.
296 So far, we've shown a simple producer/consumer configuration - the next
297 phase is to organise producers in playlists.
299 Let's assume that we're adapting the Hello World example, and wish to queue
300 a number of files for playout, ie:
304 Instead of invoking mlt_factory_producer directly, we'll create a new
305 function called create_playlist. This function is responsible for creating
306 the playlist, creating each producer and appending to the playlist.
308 mlt_producer create_playlist( int argc, char **argv )
310 // We're creating a playlist here
311 mlt_playlist playlist = mlt_playlist_init( );
313 // We need the playlist properties to ensure clean up
314 mlt_properties properties = mlt_playlist_properties( playlist );
316 // Loop through each of the arguments
318 for ( i = 1; i < argc; i ++ )
320 // Create the producer
321 mlt_producer producer = mlt_factory_producer( NULL, argv[ i ] );
323 // Add it to the playlist
324 mlt_playlist_append( playlist, producer );
326 // Close the producer (see below)
327 mlt_producer_close( producer );
330 // Return the playlist as a producer
331 return mlt_playlist_producer( playlist );
334 Notice that we close the producer after the append. Actually, what we're
335 doing is closing our reference to it - the playlist creates its own reference
336 to the producer on append and insert, and it will close its reference
337 when the playlist is destroyed[*].
339 Note also that if you append multiple instances of the same producer, it
340 will create multiple references to it.
342 Now all we need do is to replace these lines in the main function:
344 // Create a normalised producer
345 mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );
350 mlt_producer world = create_playlist( argc, argv );
352 and we have a means to play multiple clips.
354 [*] This reference functionality was introduced in mlt 0.1.2 - it is 100%
355 compatable with the early mechanism of registering the reference and
356 destructor with the properties of the playlist object.
361 Inserting filters between the producer and consumer is just a case of
362 instantiating the filters, connecting the first to the producer, the next
363 to the previous filter and the last filter to the consumer.
367 // Create a producer from something
368 mlt_producer producer = mlt_factory_producer( ... );
370 // Create a consumer from something
371 mlt_consumer consumer = mlt_factory_consumer( ... );
373 // Create a greyscale filter
374 mlt_filter filter = mlt_factory_filter( "greyscale", NULL );
376 // Connect the filter to the producer
377 mlt_filter_connect( filter, mlt_producer_service( producer ), 0 );
379 // Connect the consumer to filter
380 mlt_consumer_connect( consumer, mlt_filter_service( filter ) );
382 As with producers and consumers, filters can be manipulated via their
383 properties object - the mlt_filter_properties method can be invoked and
384 properties can be set as needed.
386 The additional argument in the filter connection is an important one as it
387 dictates the 'track' on which the filter operates. For basic producers and
388 playlists, there's only one track (0), and as you will see in the next
389 section, even multiple tracks have a single track output.
394 All services can have attached filters.
396 Consider the following example:
399 mlt_producer producer = mlt_factory_producer( NULL, clip );
401 // Get the service object of the producer
402 mlt_producer service = mlt_producer_service( producer );
405 mlt_filter filter = mlt_factory_filter( "greyscale" );
408 mlt_playlist playlist = mlt_playlist_init( );
410 // Attach the filter to the producer
411 mlt_service_attach( producer, filter );
413 // Construct a playlist with various cuts from the producer
414 mlt_playlist_append_io( producer, 0, 99 );
415 mlt_playlist_append_io( producer, 450, 499 );
416 mlt_playlist_append_io( producer, 200, 399 );
418 // We can close the producer and filter now
419 mlt_producer_close( producer );
420 mlt_filter_close( filter );
422 When this is played out, the greyscale filter will be executed for each frame
423 in the playlist which comes from that producer.
425 Further, each cut can have their own filters attached which are executed after
426 the producer's filters. As an example:
428 // Create a new filter
429 filter = mlt_factory_filter( "invert", NULL );
431 // Get the second 'clip' in the playlist
432 producer = mlt_playlist_get_clip( 1 );
434 // Get the service object of the clip
435 service = mlt_producer_service( producer );
438 mlt_service_attach( producer, filter );
441 mlt_filter_close( filter );
443 Even the playlist itself can have an attached filter:
445 // Create a new filter
446 filter = mlt_factory_filter( "watermark", "+Hello.txt" );
448 // Get the service object of the playlist
449 service = mlt_playlist_service( playlist );
452 mlt_service_attach( service, filter );
455 mlt_filter_close( filter );
457 And, of course, the playlist, being a producer, can be cut up and placed on
458 another playlist, and filters can be attached to those cuts or on the new
459 playlist itself and so on ad nauseum.
461 The main advantage of attached filters is that they remain attached and don't
462 suffer from the maintenance problems associated with items being inserted and
463 displacing calculated in/out points - this being a major issue if you
464 exclusively use the connect or insert detached filters in a multitrack field
470 The mix is the simplest way to introduce transitions between adjacent clips
473 Consider the following playlist:
475 +-+----------------------+----------------------------+-+
477 +-+----------------------+----------------------------+-+
479 Let's assume that the 'X' is a 'black clip' of 50 frames long.
481 When you play this out, you'll get a 50 frames of black, abrupt cut into
482 A, followed by an abrupt cut into B, and finally into black again.
484 The intention is to convert this playlist into something like:
486 +-+---------------------+-+------------------------+-+
489 +-+---------------------+-+------------------------+-+
491 Where the clips which refer to 2 clips represent a transition. Notice that
492 the representation of the second playlist is shorter than the first - this is
493 to be expected - a single transition of 50 frames between two clips will
494 reduce the playtime of the result by 50 frames.
496 This is done via the use of the mlt_playlist_mix method. So, assuming you get
497 a playlist as shown in the original diagram, to do the first mix, you could do
500 // Create a transition
501 mlt_transition transition = mlt_factor_transition( "luma", NULL );
503 // Mix the first and second clips for 50
504 mlt_playlist_mix( playlist, 0, 50, transition );
506 // Close the transition
507 mlt_transition_close( transition );
509 This would give you the first transition, subsequently, you would apply a similar
510 technique to mix clips 1 and 2. Note that this would create a new clip on the
511 playlist, so the next mix would be between 3 and 4.
513 As a general hint, to simplify the requirement to know the next clip index,
514 you might find the following simpler:
516 // Get the number of clips on the playlist
517 int i = mlt_playlist_count( );
519 // Iterate through them in reverse order
522 // Create a transition
523 mlt_transition transition = mlt_factor_transition( "luma", NULL );
525 // Mix the first and second clips for 50
526 mlt_playlist_mix( playlist, i, 50, transition );
528 // Close the transition
529 mlt_transition_close( transition );
532 There are other techniques, like using the mlt_playlist_join between the
533 current clip and the newly created one (you can determine if a new clip was
534 created by comparing the playlist length before and after the mix call).
536 Internally, the mlt_playlist_mix call generates a tractor and multitrack as
537 described below. Like the attached filters, the mix makes life very simple
538 when you're inserting items into the playlist.
540 Also note that it allows a simpler user interface - instead of enforcing the
541 use of a complex multitrack object, you can do many operations on a single
542 track. Thus, additional tracks can be used to introduce audio dubs, mixes
543 or composites which are independently positioned and aren't affected by
544 manipulations on other tracks. But hey, if you want a bombastic, confusing
545 and ultimately frustrating traditional NLE experience, that functionality
549 Practicalities and Optimisations:
551 In the previous two sections I've introduced some powerful functionality
552 designed to simplify MLT usage. However, a general issue comes into this -
553 what happens when you introduce a transition between two cuts from the same
554 bit of video footage?
556 Anyone who is familiar with video compression will be aware that seeking
557 isn't always without consequence from a performance point of view. So if
558 you happen to require two frames from the same clip for a transition, the
559 processing is going to be excessive and the result will undoubtedly be very
560 unpleasant, especially if you're rendering in realtime...
562 So how do we get round this?
564 Actually, it's very simple - you invoke mlt_producer_optimise on the top
565 level object after a modification and MLT will determine how to handle it.
566 Internally, it determines the maximum number of overlapping instances
567 throughout the object and creates clones and assigns clone indexes as
570 In the mix example above, you can simply call:
572 // Optimise the playlist
573 mlt_producer_optimise( mlt_playlist_producer( playlist ) );
575 after the mix calls have be done. Note that this is automatically applied
576 to deserialised westleys.
579 Multiple Tracks and Transitions:
581 MLT's approach to multiple tracks is governed by two requirements:
583 1) The need for a consumer and producer to communicate with one another via
585 2) The desire to be able to serialise and manipulate a 'network' (or filter
586 graph if you prefer).
588 We can visualise a multitrack in the way that an NLE presents it:
590 +-----------------+ +-----------------------+
592 +---------------+-+--------------------------+-+---------------------+
594 +------------------------------+
596 The overlapping areas of track 0 and 1 would (presumably) have some kind of
597 transition - without a transition, the frames from b1 and b2 would be shown
598 during the areas of overlap (ie: by default, the higher numbered track takes
599 precedence over the lower numbered track).
601 MLT has a multitrack object, but it is not a producer in the sense that it
602 can be connected directly to a consumer and everything will work correctly.
603 A consumer would treat it precisely as it would a normal producer, and, in
604 the case of the multitrack above, you would never see anything from track 1
605 other than the transitions between the clips - the gap between a1 and a2
606 would show test frames.
608 This happens because a consumer pulls one frame from the producer it's
609 connected to while a multitrack will provide one frame per track.
610 Something, somewhere, must ensure that all frames are pulled from the
611 multitrack and elect the correct frame to pass on.
613 Hence, MLT provides a wrapper for the multitrack, which is called a
614 'tractor', and its the tractors task to ensure that all tracks are pulled
615 evenly, the correct frame is output and that we have 'producer like'
618 Thus, a multitrack is conceptually 'pulled' by a tractor as shown here:
622 | +------+ | +-------+
623 | |track0|-|--->|tractor|
627 | |track1|-|--->|---o---|--->
632 | +------+ | +-------+
635 With a combination of the two, we can now connect multitracks to consumers.
636 The last non-test card will be retrieved and passed on.
638 The tracks can be producers, playlists, or even other tractors.
640 Now we wish to insert filters and transitions between the multitrack and the
641 tractor. We can do this directly by inserting filters directly between the
642 tractor and the multitrack, but this involves a lot of connecting and
643 reconnecting left and right producers and consumers, and it seemed only fair
644 that we should be able to automate that process.
646 So in keeping with our agricultural theme, the concept of the 'field' was
647 born. We 'plant' filters and transitions in the field and the tractor pulls
648 the multitrack (think of a combine harvester :-)) over the field and
649 produces a 'bail' (sorry - kidding - frame :-)).
651 Conceptually, we can see it like this:
655 | +------+ | +-------------+ +-------+
656 | |track0|-|--->|field |--->|tractor|
657 | +------+ | | | |\ |
658 | | | filters | | \ |
659 | +------+ | | and | | \ |
660 | |track1|-|--->| transitions |--->|---o---|--->
661 | +------+ | | | | / |
663 | +------+ | | | |/ |
664 | |track2|-|--->| |--->| |
665 | +------+ | +-------------+ +-------+
668 So, we need to create the tractor first, and from that we obtain the
669 multitrack and field objects. We can populate these and finally
670 connect the tractor to a consumer.
672 In essence, this is how it looks to the consumer:
674 +-----------------------------------------------+
675 |tractor +--------------------------+ |
676 | +----------+ | +-+ +-+ +-+ +-+ | |
677 | |multitrack| | |f| |f| |t| |t| | |
678 | | +------+ | | |i| |i| |r| |r| | |
679 | | |track0|-|--->| |l|- ->|l|- ->|a|--->|a|\| |
680 | | +------+ | | |t| |t| |n| |n| | |
681 | | | | |e| |e| |s| |s| |\ |
682 | | +------+ | | |r| |r| |i| |i| | \|
683 | | |track1|-|- ->| |0|--->|1|--->|t|--->|t|-|--o--->
684 | | +------+ | | | | | | |i| |i| | /|
685 | | | | | | | | |o| |o| |/ |
686 | | +------+ | | | | | | |n| |n| | |
687 | | |track2|-|- ->| | |- ->| |--->|0|- ->|1|/| |
688 | | +------+ | | | | | | | | | | | |
689 | +----------+ | +-+ +-+ +-+ +-+ | |
690 | +--------------------------+ |
691 +-----------------------------------------------+
693 An example will hopefully clarify this.
695 Let's assume that we want to provide a 'watermark' to our hello world
696 example. We have already extended the example to play multiple clips,
697 and now we will place a text based watermark, reading 'Hello World' in
698 the top left hand corner:
700 mlt_producer create_tracks( int argc, char **argv )
702 // Create the tractor
703 mlt_tractor tractor = mlt_tractor_new( );
706 mlt_field field = mlt_tractor_field( tractor );
708 // Obtain the multitrack
709 mlt_multitrack multitrack = mlt_tractor_multitrack( tractor );
711 // Create a composite transition
712 mlt_transition transition = mlt_factory_transition( "composite", "10%,10%:15%x15%" );
715 mlt_producer track0 = create_playlist( argc, argv );
717 // Create the watermark track - note we NEED fezzik for scaling here
718 mlt_producer track1 = mlt_factory_producer( "fezzik", "pango" );
720 // Get the length of track0
721 mlt_position length = mlt_producer_get_playtime( track0 );
723 // Set the properties of track1
724 mlt_properties properties = mlt_producer_properties( track1 );
725 mlt_properties_set( properties, "text", "Hello\nWorld" );
726 mlt_properties_set_position( properties, "in", 0 );
727 mlt_properties_set_position( properties, "out", length - 1 );
728 mlt_properties_set_position( properties, "length", length );
729 mlt_properties_set_int( properties, "a_track", 0 );
730 mlt_properties_set_int( properties, "b_track", 1 );
732 // Now set the properties on the transition
733 properties = mlt_transition_properties( transition );
734 mlt_properties_set_position( properties, "in", 0 );
735 mlt_properties_set_position( properties, "out", length - 1 );
737 // Add our tracks to the multitrack
738 mlt_multitrack_connect( multitrack, track0, 0 );
739 mlt_multitrack_connect( multitrack, track1, 1 );
741 // Now plant the transition
742 mlt_field_plant_transition( field, transition, 0, 1 );
744 // Close our references
745 mlt_producer_close( track0 );
746 mlt_producer_close( track1 );
747 mlt_transition_close( transition );
749 // Return the tractor
750 return mlt_tractor_producer( tractor );
753 Now all we need do is to replace these lines in the main function:
756 mlt_producer world = create_playlist( argc, argv );
760 // Create a watermarked playlist
761 mlt_producer world = create_tracks( argc, argv );
763 and we have a means to play multiple clips with a horribly obtrusive
764 watermark - just what the world needed, right? ;-)
766 Incidentally, the same thing could be achieved with the more trivial
767 watermark filter inserted between the producer and the consumer.
770 SECTION 3 - STRUCTURE AND DESIGN
771 --------------------------------
775 The mlt framework consists of an OO class hierarchy which consists of the
776 following public classes and abstractions:
791 Each class defined above can be read as extending the classes above and to
794 The following sections describe the properties, stacking/queuing and memory
795 pooling functionality provided by the framework - these are key components
796 and a basic understanding of these is required for the remainder of the
802 The properties class is the base class for the frame and service classes.
804 It is designed to provide an efficient lookup table for various types of
805 information, such as strings, integers, floating points values and pointers
806 to data and data structures.
808 All properties are indexed by a unique string.
810 The most basic use of properties is as follows:
812 // 1. Create a new, empty properties set;
813 mlt_properties properties = mlt_properties_new( );
815 // 2. Assign the value "world" to the property "hello";
816 mlt_properties_set( properties, "hello", "world" );
818 // 3. Retrieve and print the value of "hello";
819 printf( "%s\n", mlt_properties_get( properties, "hello" ) );
821 // 4. Reassign "hello" to "world!";
822 mlt_properties_set( properties, "hello", "world!" );
824 // 5. Retrieve and print the value of "hello";
825 printf( "%s\n", mlt_properties_get( properties, "hello" ) );
827 // 6. Assign the value "0" to "int";
828 mlt_properties_set( properties, "int", "0" );
830 // 7. Retrieve and print the integer value of "int";
831 printf( "%d\n", mlt_properties_get_int( properties, "int" ) );
833 // 8. Assign the integer value 50 to "int2";
834 mlt_properties_set_int( properties, "int2", 50 );
836 // 9. Retrieve and print the double value of "int2";
837 printf( "%s\n", mlt_properties_get( properties, "int2" ) );
839 Steps 2 through 5 demonstrate that the "name" is unique - set operations on
840 an existing "name" change the value. They also free up memory associated to
841 the previous value. Note that it also possible to change type in this way
844 Steps 6 and 7 demonstrate that the properties object handles deserialisation
845 from strings. The string value of "0" is set, the integer value of 0 is
848 Steps 8 and 9 demonstrate that the properties object handles serialisation
851 To show all the name/value pairs in a properties, it is possible to iterate
854 for ( i = 0; i < mlt_properties_count( properties ); i ++ )
855 printf( "%s = %s\n", mlt_properties_get_name( properties, i ),
856 mlt_properties_get_value( properties, i ) );
858 Note that properties are retrieved in the order in which they are set.
860 Properties are also used to hold pointers to memory. This is done via the
863 uint8_t *image = malloc( size );
864 mlt_properties_set_data( properties, "image", image, size, NULL, NULL );
866 In this example, we specify that the pointer can be retrieved from
867 properties by a subsequent request to get_data:
869 image = mlt_properties_get_data( properties, "image", &size );
873 image = mlt_properties_get_data( properties, "image", NULL );
875 if we don't wish to retrieve the size.
879 1) The allocated memory remains after the properties object is closed unless
880 you specify a destructor. In the case above, this can be done with:
882 mlt_properties_set_data( properties, "image", image, size, free, NULL );
884 When the properties are closed, or the value of "image" is changed, the
885 destructor is invoked.
887 2) The string value returned by mlt_properties_get is NULL. Typically, you
888 wouldn't wish to serialise an image as a string, but other structures
889 might need such functionality - you can specify a serialiser as the last
890 argument if required (declaration is char *serialise( void * )).
892 Properties also provides some more advanced usage capabilities.
894 It has the ability to inherit all serialisable values from another properties
897 mlt_properties_inherit( this, that );
899 It has the ability to mirror properties set on this on another set of
902 mlt_properties_mirror( this, that );
904 After this call, all serialisable values set on this are passed on to that.
909 Stacks and queues are essential components in the MLT framework. Being of a
910 lazy disposition, we elected to implement a 'Double Ended Queue' (deque) -
911 this encapsulates the functionality of both.
913 The API of the deque is defined as follows:
915 mlt_deque mlt_deque_init( );
916 int mlt_deque_count( mlt_deque this );
917 int mlt_deque_push_back( mlt_deque this, void *item );
918 void *mlt_deque_pop_back( mlt_deque this );
919 int mlt_deque_push_front( mlt_deque this, void *item );
920 void *mlt_deque_pop_front( mlt_deque this );
921 void *mlt_deque_peek_back( mlt_deque this );
922 void *mlt_deque_peek_front( mlt_deque this );
923 void mlt_deque_close( mlt_deque this );
925 The stacking operations are used in a number of places:
927 * Reverse Polish Notation (RPN) image and audio operations
930 The queuing operations are used in:
932 * the consumer base class;
933 * consumer implementations may require further queues.
938 The MLT framework provides memory pooling capabilities through the mlt_pool
939 API. Once initilialised, these can be seen as a straightforward drop in
940 replacement for malloc/realloc/free functionality.
942 The background behind this API is that malloc/free operations are
943 notoriously inefficient, especially when dealing with large blocks of memory
944 (such as an image). On linux, malloc is optimised for memory allocations
945 less than 128k - memory blocks allocated of these sizes or less are retained
946 in the process heap for subsequent reuse, thus bypassing the kernel calls
947 for repeated allocation/frees for small blocks of memory. However, blocks of
948 memory larger than that require kernel calls and this has a detrimental
949 impact on performance.
951 The mlt_pool design is simply to hold a list of stacks - there is one stack
952 per 2^n bytes (where n is between 8 and 31). When an alloc is called, the
953 requested size is rounded to the next 2^n, the stack is retrieved for that
954 size, and an item is popped or created if the stack is empty.
956 Each item has a 'header', situated immediately before the returned address -
957 this holds the 'stack' to which the item belongs.
959 When an item is released, we retrieve the header, obtain the stack and push
962 Thus, from the programmers point of view, the API is the same as the
963 traditional malloc/realloc/free calls:
965 void *mlt_pool_alloc( int size );
966 void *mlt_pool_realloc( void *ptr, int size );
967 void mlt_pool_release( void *release );
972 A frame object is essentially defined as:
982 The life cycle of a frame can be represented as follows:
984 +-----+----------------------+-----------------------+---------------------+
985 |Stage|Producer |Filter |Consumer |
986 +-----+----------------------+-----------------------+---------------------+
987 | 0.0 | | |Request frame |
988 +-----+----------------------+-----------------------+---------------------+
989 | 0.1 | |Receives request | |
990 | | |Request frame | |
991 +-----+----------------------+-----------------------+---------------------+
992 | 0.2 |Receives request | | |
993 | |Generates frame for | | |
994 | |current position | | |
995 | |Increments position | | |
996 +-----+----------------------+-----------------------+---------------------+
997 | 0.3 | |Receives frame | |
998 | | |Updates frame | |
999 +-----+----------------------+-----------------------+---------------------+
1000 | 0.4 | | |Receives frame |
1001 +-----+----------------------+-----------------------+---------------------+
1003 Note that neither the filter nor the consumer have any conception of
1004 'position' until they receive a frame. Speed and position are properties of
1005 the producer, and they are assigned to the frame object when the producer
1008 Step 0.3 is a critical one here - if the filter determines that the frame is
1009 of interest to it, then it should manipulate the image and/or audio stacks
1010 and properties as required.
1012 Assuming that the filter deals with both image and audio, then it should
1013 push data and methods on to the stacks which will deal with the processing.
1014 This can be done with the mlt_frame_push_image and audio methods. In order for
1015 the filter to register interest in the frame, the stacks should hold:
1018 [ producer_get_image ] [ data1 ] [ data2 ] [ filter_get_image ]
1021 [ producer_get_audio ] [ data ] [ filter_get_audio ]
1023 The filter_get methods are invoked automatically when the consumer invokes a
1024 get_image on the frame.
1026 +-----+----------------------+-----------------------+---------------------+
1027 |Stage|Producer |Filter |Consumer |
1028 +-----+----------------------+-----------------------+---------------------+
1029 | 1.0 | | |frame_get_image |
1030 +-----+----------------------+-----------------------+---------------------+
1031 | 1.1 | |filter_get_image: | |
1032 | | | pop data2 and data1 | |
1033 | | | frame_get_image | |
1034 +-----+----------------------+-----------------------+---------------------+
1035 | 1.2 |producer_get_image | | |
1036 | | Generates image | | |
1037 +-----+----------------------+-----------------------+---------------------+
1038 | 1.3 | |Receives image | |
1039 | | |Updates image | |
1040 +-----+----------------------+-----------------------+---------------------+
1041 | 1.4 | | |Receives image |
1042 +-----+----------------------+-----------------------+---------------------+
1044 Obviously, if the filter isn't interested in the image, then it should leave
1045 the stack alone, and then the consumer will retrieve its image directly from
1048 Similarly, audio is handled as follows:
1050 +-----+----------------------+-----------------------+---------------------+
1051 |Stage|Producer |Filter |Consumer |
1052 +-----+----------------------+-----------------------+---------------------+
1053 | 2.0 | | |frame_get_audio |
1054 +-----+----------------------+-----------------------+---------------------+
1055 | 2.1 | |filter_get_audio: | |
1057 | | | frame_get_audio | |
1058 +-----+----------------------+-----------------------+---------------------+
1059 | 2.2 |producer_get_audio | | |
1060 | | Generates audio | | |
1061 +-----+----------------------+-----------------------+---------------------+
1062 | 2.3 | |Receives audio | |
1063 | | |Updates audio | |
1064 +-----+----------------------+-----------------------+---------------------+
1065 | 2.4 | | |Receives audio |
1066 +-----+----------------------+-----------------------+---------------------+
1068 And finally, when the consumer is done with the frame, it should close it.
1070 Note that a consumer may not evaluate both image and audio for any given
1071 frame, especially in a realtime environment. See 'Realtime Considerations'
1074 By default, a frame has the following properties:
1076 +------------------+------------------------------------+------------------+
1077 |Name |Description |Values |
1078 +------------------+------------------------------------+------------------+
1079 |_position |The producers frame position |0 to n |
1080 +------------------+------------------------------------+------------------+
1081 |_speed |The producers speed |double |
1082 +------------------+------------------------------------+------------------+
1083 |image |The generated image |NULL or pointer |
1084 +------------------+------------------------------------+------------------+
1085 |alpha |The generated alpha mask |NULL or pointer |
1086 +------------------+------------------------------------+------------------+
1087 |width |The width of the image | |
1088 +------------------+------------------------------------+------------------+
1089 |height |The height of the image | |
1090 +------------------+------------------------------------+------------------+
1091 |normalised_width |The normalised width of the image |720 |
1092 +------------------+------------------------------------+------------------+
1093 |normalised_height |The normalised height of the image |576 or 480 |
1094 +------------------+------------------------------------+------------------+
1095 |progressive |Indicates progressive/interlaced |0 or 1 |
1096 +------------------+------------------------------------+------------------+
1097 |top_field_first |Indicates top field first |0 or 1 |
1098 +------------------+------------------------------------+------------------+
1099 |audio |The generated audio |NULL or pointer |
1100 +------------------+------------------------------------+------------------+
1101 |frequency |The frequency of the audio | |
1102 +------------------+------------------------------------+------------------+
1103 |channels |The channels of the audio | |
1104 +------------------+------------------------------------+------------------+
1105 |samples |The samples of the audio | |
1106 +------------------+------------------------------------+------------------+
1107 |aspect_ratio |The sample aspect ratio of the image|double |
1108 +------------------+------------------------------------+------------------+
1109 |test_image |Used to indicate no image available |0 or 1 |
1110 +------------------+------------------------------------+------------------+
1111 |test_audio |Used to indicate no audio available |0 or 1 |
1112 +------------------+------------------------------------+------------------+
1114 The consumer can attach the following properties which affect the default
1115 behaviour of a frame:
1117 +------------------+------------------------------------+------------------+
1118 |test_card_producer|Synthesise test images from here |NULL or pointer |
1119 +------------------+------------------------------------+------------------+
1120 |consumer_aspect_ |Apply this aspect ratio to the test |double |
1121 |ratio |card producer | |
1122 +------------------+------------------------------------+------------------+
1123 |rescale.interp |Use this scale method for test image|"string" |
1124 +------------------+------------------------------------+------------------+
1126 While most of these are mainly self explanatory, the normalised_width and
1127 normalised_height values require a little explanation. These are required
1128 to ensure that effects are consistently handled as PAL or NTSC, regardless
1129 of the consumers or producers width/height image request.
1131 The test_image and audio flags are used to determine when images and audio
1132 should be synthesised.
1134 Additional properties may be provided by the producer implementation, and
1135 filters, transitions and consumers may add additional properties to
1136 communicate specific requests. These are documented in modules.txt.
1138 The complete API for the mlt frame is as follows:
1140 mlt_frame mlt_frame_init( );
1141 mlt_properties mlt_frame_properties( mlt_frame this );
1142 int mlt_frame_is_test_card( mlt_frame this );
1143 int mlt_frame_is_test_audio( mlt_frame this );
1144 double mlt_frame_get_aspect_ratio( mlt_frame this );
1145 int mlt_frame_set_aspect_ratio( mlt_frame this, double value );
1146 mlt_position mlt_frame_get_position( mlt_frame this );
1147 int mlt_frame_set_position( mlt_frame this, mlt_position value );
1148 int mlt_frame_get_image( mlt_frame this, uint8_t **buffer, mlt_image_format *format, int *width, int *height, int writable );
1149 uint8_t *mlt_frame_get_alpha_mask( mlt_frame this );
1150 int mlt_frame_get_audio( mlt_frame this, int16_t **buffer, mlt_audio_format *format, int *frequency, int *channels, int *samples );
1151 int mlt_frame_push_get_image( mlt_frame this, mlt_get_image get_image );
1152 mlt_get_image mlt_frame_pop_get_image( mlt_frame this );
1153 int mlt_frame_push_frame( mlt_frame this, mlt_frame that );
1154 mlt_frame mlt_frame_pop_frame( mlt_frame this );
1155 int mlt_frame_push_service( mlt_frame this, void *that );
1156 void *mlt_frame_pop_service( mlt_frame this );
1157 int mlt_frame_push_audio( mlt_frame this, void *that );
1158 void *mlt_frame_pop_audio( mlt_frame this );
1159 void mlt_frame_close( mlt_frame this );
1164 The service base class extends properties and allows 0 to m inputs and 0 to
1165 n outputs and is represented as follows:
1175 Descendents of service impose restrictions on how inputs and outputs can be
1176 connected and will provide a basic set of properties. Typically, the service
1177 instance is encapsulated by the descendent in order for it to ensure that
1178 its connection rules are followed.
1180 A service does not define any properties when constructed. It should be
1181 noted that producers, filters and transitions my be serialised (say, via the
1182 westley consumer), and care should be taken to distinguish between
1183 serialisable and transient properties. The convention used is to prefix
1184 transient properties with an underscore.
1186 The public interface is defined by the following functions:
1188 int mlt_service_init( mlt_service this, void *child );
1189 mlt_properties mlt_service_properties( mlt_service this );
1190 int mlt_service_connect_producer( mlt_service this, mlt_service producer, int index );
1191 int mlt_service_get_frame( mlt_service this, mlt_frame_ptr frame, int index );
1192 void mlt_service_close( mlt_service this );
1194 Typically, only direct descendents of services need invoke these methods and
1195 developers are encouraged to use those extensions when defining new services.
1200 A producer has 0 inputs and 1 output:
1210 A producer provides an abstraction for file readers, pipes, streams or any
1211 other image or audio input.
1213 When instantiated, a producer has the following properties:
1215 +------------------+------------------------------------+------------------+
1216 |Name |Description |Values |
1217 +------------------+------------------------------------+------------------+
1218 |mlt_type |The producers type |mlt_producer |
1219 +------------------+------------------------------------+------------------+
1220 |_position |The producers frame position |0 to n |
1221 +------------------+------------------------------------+------------------+
1222 |_speed |The producers speed |double |
1223 +------------------+------------------------------------+------------------+
1224 |fps |The output frames per second |25 or 29.97 |
1225 +------------------+------------------------------------+------------------+
1226 |in |The in point in frames |0 to length - 1 |
1227 +------------------+------------------------------------+------------------+
1228 |out |The out point in frames |in to length - 1 |
1229 +------------------+------------------------------------+------------------+
1230 |length |The length of the input in frames |0 to n |
1231 +------------------+------------------------------------+------------------+
1232 |aspect_ratio |aspect_ratio of the source |0 to n |
1233 +------------------+------------------------------------+------------------+
1234 |eof |end of clip behaviour |"pause" or "loop" |
1235 +------------------+------------------------------------+------------------+
1236 |resource |Constructor argument (ie: file name)|"<resource>" |
1237 +------------------+------------------------------------+------------------+
1239 Additional properties may be provided by the producer implementation.
1241 The public interface is defined by the following functions:
1243 mlt_producer mlt_producer_new( );
1244 int mlt_producer_init( mlt_producer this, void *child );
1245 mlt_service mlt_producer_service( mlt_producer this );
1246 mlt_properties mlt_producer_properties( mlt_producer this );
1247 int mlt_producer_seek( mlt_producer this, mlt_position position );
1248 mlt_position mlt_producer_position( mlt_producer this );
1249 mlt_position mlt_producer_frame( mlt_producer this );
1250 int mlt_producer_set_speed( mlt_producer this, double speed );
1251 double mlt_producer_get_speed( mlt_producer this );
1252 double mlt_producer_get_fps( mlt_producer this );
1253 int mlt_producer_set_in_and_out( mlt_producer this, mlt_position in, mlt_position out );
1254 mlt_position mlt_producer_get_in( mlt_producer this );
1255 mlt_position mlt_producer_get_out( mlt_producer this );
1256 mlt_position mlt_producer_get_playtime( mlt_producer this );
1257 mlt_position mlt_producer_get_length( mlt_producer this );
1258 void mlt_producer_prepare_next( mlt_producer this );
1259 void mlt_producer_close( mlt_producer this );
1264 The public interface is defined by the following functions:
1266 int mlt_filter_init( mlt_filter this, void *child );
1267 mlt_filter mlt_filter_new( );
1268 mlt_service mlt_filter_service( mlt_filter this );
1269 mlt_properties mlt_filter_properties( mlt_filter this );
1270 mlt_frame mlt_filter_process( mlt_filter this, mlt_frame that );
1271 int mlt_filter_connect( mlt_filter this, mlt_service producer, int index );
1272 void mlt_filter_set_in_and_out( mlt_filter this, mlt_position in, mlt_position out );
1273 int mlt_filter_get_track( mlt_filter this );
1274 mlt_position mlt_filter_get_in( mlt_filter this );
1275 mlt_position mlt_filter_get_out( mlt_filter this );
1276 void mlt_filter_close( mlt_filter );
1281 The public interface is defined by the following functions:
1283 int mlt_transition_init( mlt_transition this, void *child );
1284 mlt_transition mlt_transition_new( );
1285 mlt_service mlt_transition_service( mlt_transition this );
1286 mlt_properties mlt_transition_properties( mlt_transition this );
1287 int mlt_transition_connect( mlt_transition this, mlt_service producer, int a_track, int b_track );
1288 void mlt_transition_set_in_and_out( mlt_transition this, mlt_position in, mlt_position out );
1289 int mlt_transition_get_a_track( mlt_transition this );
1290 int mlt_transition_get_b_track( mlt_transition this );
1291 mlt_position mlt_transition_get_in( mlt_transition this );
1292 mlt_position mlt_transition_get_out( mlt_transition this );
1293 mlt_frame mlt_transition_process( mlt_transition this, mlt_frame a_frame, mlt_frame b_frame );
1294 void mlt_transition_close( mlt_transition this );
1299 The public interface is defined by the following functions:
1301 int mlt_consumer_init( mlt_consumer this, void *child );
1302 mlt_service mlt_consumer_service( mlt_consumer this );
1303 mlt_properties mlt_consumer_properties( mlt_consumer this );
1304 int mlt_consumer_connect( mlt_consumer this, mlt_service producer );
1305 int mlt_consumer_start( mlt_consumer this );
1306 mlt_frame mlt_consumer_get_frame( mlt_consumer this );
1307 mlt_frame mlt_consumer_rt_frame( mlt_consumer this );
1308 int mlt_consumer_stop( mlt_consumer this );
1309 int mlt_consumer_is_stopped( mlt_consumer this );
1310 void mlt_consumer_close( mlt_consumer );
1313 Specialised Producers:
1315 There are two major types of specialised producers - playlists and tractors.
1317 The following sections describe these.
1322 mlt_playlist mlt_playlist_init( );
1323 mlt_producer mlt_playlist_producer( mlt_playlist this );
1324 mlt_service mlt_playlist_service( mlt_playlist this );
1325 mlt_properties mlt_playlist_properties( mlt_playlist this );
1326 int mlt_playlist_count( mlt_playlist this );
1327 int mlt_playlist_clear( mlt_playlist this );
1328 int mlt_playlist_append( mlt_playlist this, mlt_producer producer );
1329 int mlt_playlist_append_io( mlt_playlist this, mlt_producer producer, mlt_position in, mlt_position out );
1330 int mlt_playlist_blank( mlt_playlist this, mlt_position length );
1331 mlt_position mlt_playlist_clip( mlt_playlist this, mlt_whence whence, int index );
1332 int mlt_playlist_current_clip( mlt_playlist this );
1333 mlt_producer mlt_playlist_current( mlt_playlist this );
1334 int mlt_playlist_get_clip_info( mlt_playlist this, mlt_playlist_clip_info *info, int index );
1335 int mlt_playlist_insert( mlt_playlist this, mlt_producer producer, int where, mlt_position in, mlt_position out );
1336 int mlt_playlist_remove( mlt_playlist this, int where );
1337 int mlt_playlist_move( mlt_playlist this, int from, int to );
1338 int mlt_playlist_resize_clip( mlt_playlist this, int clip, mlt_position in, mlt_position out );
1339 void mlt_playlist_close( mlt_playlist this );