6 MLT is a multimedia framework designed for television broadcasting. As such,
7 it provides a pluggable architecture for the inclusion of new audio/video
8 sources, filters, transitions and playback devices.
10 The MLT framework provides the structure and utility functionality on which
11 all of the MLT applications and services are defined.
13 On its own, the framework provides little more than 'abstract classes' and
14 utilities for managing resources, such as memory, properties, dynamic object
15 loading and service instantiation.
17 This document is split roughly into 3 sections. The first section provides a
18 basic overview of MLT, the second section shows how it's used and the final
19 section shows shows structure and design, with an emphasis on how the system
25 This document is provided as a 'road map' for the framework and should be
26 considered mandatory reading for anyone wishing to develop code at the MLT
31 1. framework maintainers;
33 3. application developers;
34 4. anyone interested in MLT.
36 The emphasis of the document is in explaining the public interfaces, as
37 opposed to the implementation details.
39 It is not required reading for the MLT client/server integration - please
40 refer to valerie.txt and dvcp.txt for more details on this area.
43 SECTION 1 - BASIC OVERVIEW
44 --------------------------
46 Basic Design Information:
50 The framework has no dependencies other than the standard C99 and POSIX
53 It follows a basic Object Oriented design paradigm, and as such, much of the
54 design is loosely based on the Producer/Consumer design pattern.
56 It employs Reverse Polish Notation for the application of audio and video FX.
58 The framework is designed to be colour space neutral - the currently
59 implemented modules, however, are very much 8bit YUV422 oriented. In theory,
60 the modules could be entirely replaced.
62 A vague understanding of these terms is assumed throughout the remainder of
68 The general structure of an MLT 'network' is simply the connection of a
69 'producer' to a 'consumer':
72 |Producer|-->|Consumer|
75 A typical consumer requests MLT Frame objects from the producer, does
76 something with them and when finished with a frame, closes it.
78 /\ A common confusion with the producer/consumer terminoligy used here is
79 /!!\ that a consumer may 'produce' something. For example, the libdv consumer
80 \!!/ produces DV and the libdv producer seems to consume DV. However, the
81 \/ naming conventions refer only to producers and consumers of MLT Frames.
83 To put it another way - a producer produces MLT Frame objects and a consumer
84 consumes MLT Frame objects.
86 An MLT Frame essentially provides an uncompressed image and its associated
89 Filters may also be placed between the producer and the consumer:
91 +--------+ +------+ +--------+
92 |Producer|-->|Filter|-->|Consumer|
93 +--------+ +------+ +--------+
95 A service is the collective name for producers, filters, transitions and
98 The communications between a connected consumer and producer or service are
99 carried out in 3 phases:
105 MLT employs 'lazy evaluation' - the image and audio need not be extracted
106 from the source until the get image and audio methods are invoked.
108 In essence, the consumer pulls from what it's connected to - this means that
109 threading is typically in the domain of the consumer implementation and some
110 basic functionality is provided on the consumer class to ensure realtime
119 Before we go in to the specifics of the framework architecture, a working
120 example of usage is provided.
122 The following simply provides a media player:
126 #include <framework/mlt.h>
128 int main( int argc, char *argv[] )
130 // Initialise the factory
131 if ( mlt_factory_init( NULL ) == 0 )
133 // Create the default consumer
134 mlt_consumer hello = mlt_factory_consumer( NULL, NULL );
136 // Create via the default producer
137 mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );
139 // Connect the producer to the consumer
140 mlt_consumer_connect( hello, mlt_producer_service( world ) );
142 // Start the consumer
143 mlt_consumer_start( hello );
145 // Wait for the consumer to terminate
146 while( !mlt_consumer_is_stopped( hello ) )
149 // Close the consumer
150 mlt_consumer_close( hello );
152 // Close the producer
153 mlt_producer_close( world );
156 mlt_factory_close( );
160 // Report an error during initialisation
161 fprintf( stderr, "Unable to locate factory modules\n" );
168 This is a simple example - it doesn't provide any seeking capabilities or
169 runtime configuration options.
171 The first step of any MLT application is the factory initialisation - this
172 ensures that the environment is configured and MLT can function. The factory
173 is covered in more detail below.
175 All services are instantiated via the factories, as shown by the
176 mlt_factory_consumer and mlt_factory_producer calls above. There are similar
177 factories for filters and transitions. There are details on all the standard
178 services in services.txt.
180 The defaults requested here are a special case - the NULL usage requests
181 that we use the default producers and consumers.
183 The default producer is "fezzik". This producer matches file names to
184 locate a service to use and attaches 'normalising filters' (such as scalers,
185 deinterlacers, resamplers and field normalisers) to the loaded content -
186 these filters ensure that the consumer gets what it asks for.
188 The default consumer is "sdl". The combination of fezzik and sdl will
189 provide a media player.
191 In this example, we connect the producer and then start the consumer. We
192 then wait until the consumer is stopped (in this case, by the action of the
193 user closing the SDL window) and finally close the consumer, producer and
194 factory before exiting the application.
196 Note that the consumer is threaded - waiting for an event of some sort is
197 always required after starting and before stopping or closing the consumer.
199 Also note, you can override the defaults as follows:
201 $ MLT_CONSUMER=westley ./hello file.avi
203 This will create a westley xml document on stdout.
205 $ MLT_CONSUMER=westley MLT_PRODUCER=avformat ./hello file.avi
207 This will play the video using the avformat producer directly, thus it will
208 bypass the normalising functions.
210 $ MLT_CONSUMER=libdv ./hello file.avi > /dev/dv1394
212 This might, if you're lucky, do on the fly, realtime conversions of file.avi
213 to DV and broadcast it to your DV device.
218 As shown in the 'Hello World' example, factories create service objects.
220 The framework itself provides no services - they are provided in the form of
221 a plugin structure. A plugin is organised in the form of a 'module' and a
222 module can provide many services of different types.
224 Once the factory is initialised, all the configured services are available
227 The complete set of methods associated to the factory are as follows:
229 int mlt_factory_init( char *prefix );
230 const char *mlt_factory_prefix( );
231 char *mlt_environment( char *name );
232 mlt_producer mlt_factory_producer( char *name, void *input );
233 mlt_filter mlt_factory_filter( char *name, void *input );
234 mlt_transition mlt_factory_transition( char *name, void *input );
235 mlt_consumer mlt_factory_consumer( char *name, void *input );
236 void mlt_factory_close( );
238 The mlt_factory_prefix returns the path to the location of the installed
239 modules directory. This can be specified in the mlt_factory_init call
240 itself, or it can be specified via the MLT_REPOSITORY environment variable,
241 or in the absence of either of those, it will default to the install
242 prefix/shared/mlt/modules.
244 The mlt_environment provides read only access to a collection of name=value
245 pairs as shown in the following table:
247 +------------------+------------------------------------+------------------+
248 |Name |Description |Values |
249 +------------------+------------------------------------+------------------+
250 |MLT_NORMALISATION |The normalisation of the system |PAL or NTSC |
251 +------------------+------------------------------------+------------------+
252 |MLT_PRODUCER |The default producer |"fezzik" or other |
253 +------------------+------------------------------------+------------------+
254 |MLT_CONSUMER |The default consumer |"sdl" or other |
255 +------------------+------------------------------------+------------------+
257 These values are initialised from the environment variables of the same
260 As shown above, a producer can be created using the 'default normalising'
261 producer, and they can also be requested by name. Filters and transitions
262 are always requested by name - there is no concept of a 'default' for these.
267 As shown in the services.txt document, all services have their own set of
268 properties than can be manipulated to affect their behaviour.
270 In order to set properties on a service, we need to retrieve the properties
271 object associated to it. For producers, this is done by invoking:
273 mlt_properties properties = mlt_producer_properties( producer );
275 All services have a similar method associated to them.
277 Once retrieved, setting and getting properties can be done directly on this
280 mlt_properties_set( properties, "name", "value" );
282 A more complete description of the properties object is found below.
287 So far, we've shown a simple producer/consumer configuration - the next
288 phase is to organise producers in playlists.
290 Let's assume that we're adapting the Hello World example, and wish to queue
291 a number of files for playout, ie:
295 Instead of invoking mlt_factory_producer directly, we'll create a new
296 function called create_playlist. This function is responsible for creating
297 the playlist, creating each producer, appending to the playlist and ensuring
298 that all the producers are cleaned up when the playlist is destroyed. The
299 last point is important - a close on the playlist won't explicitly these
300 producers. In this example, we use unique "data" properties with destructors
303 mlt_producer create_playlist( int argc, char **argv )
305 // We're creating a playlist here
306 mlt_playlist playlist = mlt_playlist_init( );
308 // We need the playlist properties to ensure clean up
309 mlt_properties properties = mlt_playlist_properties( playlist );
311 // Loop through each of the arguments
313 for ( i = 1; i < argc; i ++ )
315 // Definie the unique key
318 // Create the producer
319 mlt_producer producer = mlt_factory_producer( NULL, argv[ i ] );
321 // Add it to the playlist
322 mlt_playlist_append( playlist, producer );
324 // Create a unique key for this producer
325 sprintf( key, "producer%d", i );
327 // Now we need to ensure the producers are destroyed
328 mlt_properties_set_data( properties, key, producer, 0, ( mlt_destructor )mlt_producer_close, NULL );
331 // Return the playlist as a producer
332 return mlt_playlist_producer( playlist );
335 Now all we need do is to replace these lines in the main function:
337 // Create a normalised producer
338 mlt_producer world = mlt_factory_producer( NULL, argv[ 1 ] );
343 mlt_producer world = create_playlist( argc, argv );
345 and we have a means to play multiple clips.
350 Inserting filters between the producer and consumer is just a case of
351 instantiating the filters, connecting the first to the producer, the next
352 to the previous filter and the last filter to the consumer.
356 // Create a producer from something
357 mlt_producer producer = mlt_factory_producer( ... );
359 // Create a consumer from something
360 mlt_consumer consumer = mlt_factory_consumer( ... );
362 // Create a greyscale filter
363 mlt_filter filter = mlt_factory_filter( "greyscale", NULL );
365 // Connect the filter to the producer
366 mlt_filter_connect( filter, mlt_producer_service( producer ), 0 );
368 // Connect the consumer to filter
369 mlt_consumer_connect( consumer, mlt_filter_service( filter ) );
371 As with producers and consumers, filters can be manipulated via their
372 properties object - the mlt_filter_properties method can be invoked and
373 properties can be set as needed.
375 The additional argument in the filter connection is an important one as it
376 dicates the 'track' on which the filter operates. For basic producers and
377 playlists, there's only one track (0), and as you will see in the next
378 section, even multiple tracks have a single track output.
381 Multiple Tracks and Transitions:
383 MLT's approach to mutliple tracks is governed by two requirements:
385 1) The need for a consumer and producer to communicate with one another via
387 2) The desire to be able to serialise and manipulate a 'network' (or filter
388 graph if you prefer).
390 We can visualise a multitrack in the way that an NLE presents it:
392 +-----------------+ +-----------------------+
394 +---------------+-+--------------------------+-+---------------------+
396 +------------------------------+
398 The overlapping areas of track 0 and 1 would (presumably) have some kind of
399 transition - without a transition, the frames from a1 and a2 would be shown
400 during the areas of overlap.
402 MLT has a multitrack object, but it is not a producer in the sense that it
403 can be connected directly to a consumer and everything will work correctly.
404 A consumer would treat it precisely as it would a normal producer, and, in
405 the case of the multitrack above, you would never see anything from track 1
406 other than the transitions between the clips - the gap between a1 and a2
407 would show test frames.
409 This happens because a consumer pulls one frame from the producer it's
410 connected to while a multitrack will provide one frame per track.
411 Something, somewhere, must ensure that all frames are pulled from the
412 multitrack and elect the correct frame to pass on.
414 Hence, MLT provides a wrapper for the multitrack, which is called a
415 'tractor', and its the tractors task to ensure that all tracks are pulled
416 evenly, the correct frame is output and that we have 'producer like'
419 Thus, a mulitrack is conceptually 'pulled' by a tractor as shown here:
423 | +------+ | +-------+
424 | |track0|-|--->|tractor|
428 | |track1|-|--->|---o---|--->
433 | +------+ | +-------+
436 With a combination of the two, we can now connect multitracks to consumers.
437 The first non-test card will be retrieved and passed on.
439 The tracks can be producers, playlists, or even other tractors.
441 Now we wish to insert filters and transitions between the mulitrack and the
442 tractor. We can do this directly by inserting filters directly between the
443 tractor and the multitrack, but this involves a lot of connecting and
444 reconnecting left and right producers and consumers, and it seemed only fair
445 that we should be able to automate that process.
447 So in keeping with our agricultural theme, the concept of the 'field' was
448 born. We 'plant' filters and transitions in the field and the tractor pulls
449 the multitrack (think of a combine harvester :-)) over the field and
450 produces a 'bail' (sorry - kidding - frame :-)).
452 Conceptually, we can see it like this:
456 | +------+ | +-------------+ +-------+
457 | |track0|-|--->|field |--->|tractor|
458 | +------+ | | | |\ |
459 | | | filters | | \ |
460 | +------+ | | and | | \ |
461 | |track1|-|--->| transitions |--->|---o---|--->
462 | +------+ | | | | / |
464 | +------+ | | | |/ |
465 | |track2|-|--->| |--->| |
466 | +------+ | +-------------+ +-------+
469 In reality, we create a field first, and from that we obtain a multitrack
470 and a tractor. We can then populate the multitrack, field and finally,
471 connect the tractor to the consumer.
473 The reasoning behind this is possibly flawed - it might have made more
474 sense to produce the tractor and have it encapsulate the field and the
475 multitrack as that is how it looks to a connected consumer:
477 +-----------------------------------------------+
478 |tractor +--------------------------+ |
479 | +----------+ | +-+ +-+ +-+ +-+ | |
480 | |multitrack| | |f| |f| |t| |t| | |
481 | | +------+ | | |i| |i| |r| |r| | |
482 | | |track0|-|--->| |l|- ->|l|- ->|a|--->|a|\| |
483 | | +------+ | | |t| |t| |n| |n| | |
484 | | | | |e| |e| |s| |s| |\ |
485 | | +------+ | | |r| |r| |i| |i| | \|
486 | | |track1|-|- ->| |0|--->|1|--->|t|--->|t|-|--o--->
487 | | +------+ | | | | | | |i| |i| | /|
488 | | | | | | | | |o| |o| |/ |
489 | | +------+ | | | | | | |n| |n| | |
490 | | |track2|-|- ->| | |- ->| |--->|0|- ->|1|/| |
491 | | +------+ | | | | | | | | | | | |
492 | +----------+ | +-+ +-+ +-+ +-+ | |
493 | +--------------------------+ |
494 +-----------------------------------------------+
496 An example will hopefully clarify this.
498 Let's assume that we want to provide a 'watermark' to our hello world
499 example. We have already extended the example to play multiple clips,
500 and now we will place a text based watermark, reading 'Hello World' in
501 the top left hand corner:
503 mlt_producer create_tracks( int argc, char **argv )
506 mlt_field field = mlt_field_init( );
508 // Obtain the multitrack
509 mlt_multitrack multitrack = mlt_field_multitrack( field );
511 // Obtain the tractor
512 mlt_tractor tractor = mlt_field_tractor( field );
514 // Obtain a composite transition
515 mlt_transition transition = mlt_factory_transition( "composite", "10%,10%:15%x15%" );
518 mlt_producer track0 = create_playlist( argc, argv );
520 // Create the watermark track - note we NEED fezzik for scaling here
521 mlt_producer track1 = mlt_factory_producer( "fezzik", "pango" );
523 // Get the length of track0
524 mlt_position length = mlt_producer_get_playtime( track0 );
526 // Set the properties of track1
527 mlt_properties properties = mlt_producer_properties( track1 );
528 mlt_properties_set( properties, "text", "Hello\nWorld" );
529 mlt_properties_set_position( properties, "in", 0 );
530 mlt_properties_set_position( properties, "out", length - 1 );
531 mlt_properties_set_position( properties, "length", length );
533 // Now set the properties on the transition
534 properties = mlt_transition_properties( transition );
535 mlt_properties_set_position( properties, "in", 0 );
536 mlt_properties_set_position( properties, "out", length - 1 );
538 // Add our tracks to the multitrack
539 mlt_multitrack_connect( multitrack, track0, 0 );
540 mlt_multitrack_connect( multitrack, track1, 1 );
542 // Now plant the transition
543 mlt_field_plant_transition( field, transition, 0, 1 );
545 // Now set the properties on the tractor
546 properties = mlt_tractor_properties( tractor );
547 mlt_properties_set_data( properties, "multitrack", multitrack, 0, ( mlt_destructor )mlt_multitrack_close, NULL );
548 mlt_properties_set_data( properties, "field", field, 0, ( mlt_destructor )mlt_field_close, NULL );
549 mlt_properties_set_data( properties, "track0", track0, 0, ( mlt_destructor )mlt_producer_close, NULL );
550 mlt_properties_set_data( properties, "track1", track1, 0, ( mlt_destructor )mlt_producer_close, NULL );
551 mlt_properties_set_data( properties, "transition", transition, 0, ( mlt_destructor )mlt_transition_close, NULL );
553 // Return the tractor
554 return mlt_tractor_producer( tractor );
557 Now all we need do is to replace these lines in the main function:
560 mlt_producer world = create_playlist( argc, argv );
564 // Create a watermarked playlist
565 mlt_producer world = create_tracks( argc, argv );
567 and we have a means to play multiple clips with a horribly obtrusive
568 watermark - just what the world needed, right? ;-)
571 SECTION 3 - STRUCTURE AND DESIGN
572 --------------------------------
576 The mlt framework consists of an OO class heirachy which consists of the
577 following public classes and abstractions:
592 Each class defined above can be read as extending the classes above and to
595 The following sections describe the properties, stacking/queuing and memory
596 pooling functionality provided by the framework - these are key components
597 and a basic understanding of these is required for the remainder of the
603 The properties class is the base class for the frame and service classes.
605 It is designed to provide an efficient lookup table for various types of
606 information, such as strings, integers, floating points values and pointers
607 to data and data structures.
609 All properties are indexed by a unique string.
611 The most basic use of properties is as follows:
613 // 1. Create a new, empty properties set;
614 mlt_properties properties = mlt_properties_new( );
616 // 2. Assign the value "world" to the property "hello";
617 mlt_properties_set( properties, "hello", "world" );
619 // 3. Retrieve and print the value of "hello";
620 printf( "%s\n", mlt_properties_get( properties, "hello" ) );
622 // 4. Reassign "hello" to "world!";
623 mlt_properties_set( properties, "hello", "world!" );
625 // 5. Retrieve and print the value of "hello";
626 printf( "%s\n", mlt_properties_get( properties, "hello" ) );
628 // 6. Assign the value "0" to "int";
629 mlt_properties_set( properties, "int", "0" );
631 // 7. Retrieve and print the integer value of "int";
632 printf( "%d\n", mlt_properties_get_int( properties, "int" ) );
634 // 8. Assign the integer value 50 to "int2";
635 mlt_properties_set_int( properties, "int2", 50 );
637 // 9. Retrieve and print the double value of "int2";
638 printf( "%s\n", mlt_properties_get( properties, "int2" ) );
640 Steps 2 through 5 demonstrate that the "name" is unique - set operations on
641 an existing "name" change the value. They also free up memory associated to
642 the previous value. Note that it also possible to change type in this way
645 Steps 6 and 7 demonstrate that the properties object handles deserialisation
646 from strings. The string value of "50" is set, the integer value of 50 is
649 Steps 8 and 9 demonstrate that the properties object handles serialisation
652 To show all the name/value pairs in a properties, it is possible to iterate
655 for ( i = 0; i < mlt_properties_count( properties ); i ++ )
656 printf( "%s = %s\n", mlt_properties_get_name( properties, i ),
657 mlt_properties_get_value( properties, i ) );
659 Note that properties are retrieved in the order in which they are set.
661 Properties are also used to hold pointers to memory. This is done via the
664 uint8_t *image = malloc( size );
665 mlt_properties_set_data( properties, "image", image, size, NULL, NULL );
667 In this example, we specify that the pointer can be retrieved from
668 properties by a subsequent request to get_data:
670 image = mlt_properties_get_data( properties, "image", &size );
674 image = mlt_properties_get_data( properties, "image", NULL );
676 if we don't wish to retrieve the size.
680 1) The allocated memory remains after the properties object is closed unless
681 you specify a destructor. In the case above, this can be done with:
683 mlt_properties_set_data( properties, "image", image, size, free, NULL );
685 When the properties are closed, or the value of "image" is changed, the
686 destructor is invoked.
688 2) The string value returned by mlt_properties_get is NULL. Typically, you
689 wouldn't wish to serialise an image as a string, but other structures
690 might need such functionality - you can specify a serialiser as the last
691 argument if required (declaration is char *serialise( void * )).
693 Properties also provides some more advanced usage capabilities.
695 It has the ability to inherit all serialisable values from another properties
698 mlt_properties_inherit( this, that );
700 It has the ability to mirror properties set on this on another set of
703 mlt_properties_mirror( this, that );
705 After this call, all serialisable values set on this are passed on to that.
710 Stacks and queues are essential components in the MLT framework. Being of a
711 lazy disposition, we elected to implement a 'Double Ended Queue' (deque) -
712 this encapsulates the functionality of both.
714 The API of the deque is defined as follows:
716 mlt_deque mlt_deque_init( );
717 int mlt_deque_count( mlt_deque this );
718 int mlt_deque_push_back( mlt_deque this, void *item );
719 void *mlt_deque_pop_back( mlt_deque this );
720 int mlt_deque_push_front( mlt_deque this, void *item );
721 void *mlt_deque_pop_front( mlt_deque this );
722 void *mlt_deque_peek_back( mlt_deque this );
723 void *mlt_deque_peek_front( mlt_deque this );
724 void mlt_deque_close( mlt_deque this );
726 The stacking operations are used in a number of places:
728 * Reverse Polish Notation (RPN) image and audio operations
731 The queuing operations are used in:
733 * the consumer base class;
734 * consumer implementations may require further queues.
739 The MLT framework provides memory pooling capabilities through the mlt_pool
740 API. Once initilialised, these can be seen as a straightforward drop in
741 replacement for malloc/realloc/free functionality.
743 The background behind this API is that malloc/free operations are
744 notoriously inefficient, especially when dealing with large blocks of memory
745 (such as an image). On linux, malloc is optimised for memory allocations
746 less than 128k - memory blocks allocated of these sizes or less are retained
747 in the process heap for subsequent reuse, thus bypassing the kernel calls
748 for repeated allocation/frees for small blocks of memory. However, blocks of
749 memory larger than that require kernel calls and this has a detrimental
750 impact on performance.
752 The mlt_pool design is simply to hold a list of stacks - there is one stack
753 per 2^n bytes (where n is between 8 and 31). When an alloc is called, the
754 requested size is rounded to the next 2^n, the stack is retrieved for that
755 size, and an item is popped or created if the stack is empty.
757 Each item has a 'header', situated immediately before the returned address -
758 this holds the 'stack' to which the item belongs.
760 When an item is released, we retrieve the header, obtain the stack and push
763 Thus, from the programmers point of view, the API is the same as the
764 traditional malloc/realloc/free calls:
766 void *mlt_pool_alloc( int size );
767 void *mlt_pool_realloc( void *ptr, int size );
768 void mlt_pool_release( void *release );
773 A frame object is essentially defined as:
783 The life cycle of a frame can be represented as follows:
785 +-----+----------------------+-----------------------+---------------------+
786 |Stage|Producer |Filter |Consumer |
787 +-----+----------------------+-----------------------+---------------------+
788 | 0.0 | | |Request frame |
789 +-----+----------------------+-----------------------+---------------------+
790 | 0.1 | |Receives request | |
791 | | |Request frame | |
792 +-----+----------------------+-----------------------+---------------------+
793 | 0.2 |Receives request | | |
794 | |Generates frame for | | |
795 | |current position | | |
796 | |Increments position | | |
797 +-----+----------------------+-----------------------+---------------------+
798 | 0.3 | |Receives frame | |
799 | | |Updates frame | |
800 +-----+----------------------+-----------------------+---------------------+
801 | 0.4 | | |Receives frame |
802 +-----+----------------------+-----------------------+---------------------+
804 Note that neither the filter nor the consumer have any conception of
805 'position' until they receive a frame. Speed and position are properties of
806 the producer, and they are assigned to the frame object when the producer
809 Step 0.3 is a critical one here - if the filter determines that the frame is
810 of interest to it, then it should manipulate the image and/or audio stacks
811 and properties as required.
813 Assuming that the filter deals with both image and audio, then it should
814 push data and methods on to the stacks which will deal with the processing.
815 This can be done with the mlt_frame_push_image and audio methods. In order for
816 the filter to register interest in the frame, the stacks should hold:
819 [ producer_get_image ] [ data1 ] [ data2 ] [ filter_get_image ]
822 [ producer_get_audio ] [ data ] [ filter_get_audio ]
824 The filter_get methods are invoked automatically when the consumer invokes a
825 get_image on the frame.
827 +-----+----------------------+-----------------------+---------------------+
828 |Stage|Producer |Filter |Consumer |
829 +-----+----------------------+-----------------------+---------------------+
830 | 1.0 | | |frame_get_image |
831 +-----+----------------------+-----------------------+---------------------+
832 | 1.1 | |filter_get_image: | |
833 | | | pop data2 and data1 | |
834 | | | frame_get_image | |
835 +-----+----------------------+-----------------------+---------------------+
836 | 1.2 |producer_get_image | | |
837 | | Generates image | | |
838 +-----+----------------------+-----------------------+---------------------+
839 | 1.3 | |Receives image | |
840 | | |Updates image | |
841 +-----+----------------------+-----------------------+---------------------+
842 | 1.4 | | |Receives image |
843 +-----+----------------------+-----------------------+---------------------+
845 Obviously, if the filter isn't interested in the image, then it should leave
846 the stack alone, and then the consumer will retrieve its image directly from
849 Similarly, audio is handled as follows:
851 +-----+----------------------+-----------------------+---------------------+
852 |Stage|Producer |Filter |Consumer |
853 +-----+----------------------+-----------------------+---------------------+
854 | 2.0 | | |frame_get_audio |
855 +-----+----------------------+-----------------------+---------------------+
856 | 2.1 | |filter_get_audio: | |
858 | | | frame_get_audio | |
859 +-----+----------------------+-----------------------+---------------------+
860 | 2.2 |producer_get_audio | | |
861 | | Generates audio | | |
862 +-----+----------------------+-----------------------+---------------------+
863 | 2.3 | |Receives audio | |
864 | | |Updates audio | |
865 +-----+----------------------+-----------------------+---------------------+
866 | 2.4 | | |Receives audio |
867 +-----+----------------------+-----------------------+---------------------+
869 And finally, when the consumer is done with the frame, it should close it.
871 Note that a consumer may not evaluate both image and audio for any given
872 frame, especially in a realtime environment. See 'Realtime Considerations'
875 By default, a frame has the following properties:
877 +------------------+------------------------------------+------------------+
878 |Name |Description |Values |
879 +------------------+------------------------------------+------------------+
880 |_position |The producers frame position |0 to n |
881 +------------------+------------------------------------+------------------+
882 |_speed |The producers speed |double |
883 +------------------+------------------------------------+------------------+
884 |image |The generated image |NULL or pointer |
885 +------------------+------------------------------------+------------------+
886 |alpha |The generated alpha mask |NULL or pointer |
887 +------------------+------------------------------------+------------------+
888 |width |The width of the image | |
889 +------------------+------------------------------------+------------------+
890 |height |The height of the image | |
891 +------------------+------------------------------------+------------------+
892 |normalised_width |The normalised width of the image |720 |
893 +------------------+------------------------------------+------------------+
894 |normalised_height |The normalised height of the image |576 or 480 |
895 +------------------+------------------------------------+------------------+
896 |progressive |Indicates progressive/interlaced |0 or 1 |
897 +------------------+------------------------------------+------------------+
898 |top_field_first |Indicates top field first |0 or 1 |
899 +------------------+------------------------------------+------------------+
900 |audio |The generated audio |NULL or pointer |
901 +------------------+------------------------------------+------------------+
902 |frequency |The frequency of the audio | |
903 +------------------+------------------------------------+------------------+
904 |channels |The channels of the audio | |
905 +------------------+------------------------------------+------------------+
906 |samples |The samples of the audio | |
907 +------------------+------------------------------------+------------------+
908 |aspect_ratio |The aspect ratio of the image |double |
909 +------------------+------------------------------------+------------------+
910 |test_image |Used to indicate no image available |0 or 1 |
911 +------------------+------------------------------------+------------------+
912 |test_audio |Used to indicate no audio available |0 or 1 |
913 +------------------+------------------------------------+------------------+
915 The consumer can attach the following properties which affect the default
916 behaviour of a frame:
918 +------------------+------------------------------------+------------------+
919 |test_card_producer|Synthesise test images from here |NULL or pointer |
920 +------------------+------------------------------------+------------------+
921 |consumer_aspect_ |Apply this aspect ratio to the test |double |
922 |ratio |card producer | |
923 +------------------+------------------------------------+------------------+
924 |rescale.interp |Use this scale method for test image|"string" |
925 +------------------+------------------------------------+------------------+
927 While most of these are mainly self explainatory, the normalised_width and
928 normalised_height values require a little explaination. These are required
929 to ensure that effects are consistently handled as PAL or NTSC, regardless
930 of the consumers or producers width/height image request.
932 The test_image and audio flags are used to determine when images and audio
933 should be synthesised.
935 Additional properties may be provided by the producer implementation, and
936 filters, transitions and consumers may add additional properties to
937 communicate specific requests. These are documented in modules.txt.
939 The complete API for the mlt frame is as follows:
941 mlt_frame mlt_frame_init( );
942 mlt_properties mlt_frame_properties( mlt_frame this );
943 int mlt_frame_is_test_card( mlt_frame this );
944 int mlt_frame_is_test_audio( mlt_frame this );
945 double mlt_frame_get_aspect_ratio( mlt_frame this );
946 int mlt_frame_set_aspect_ratio( mlt_frame this, double value );
947 mlt_position mlt_frame_get_position( mlt_frame this );
948 int mlt_frame_set_position( mlt_frame this, mlt_position value );
949 int mlt_frame_get_image( mlt_frame this, uint8_t **buffer, mlt_image_format *format, int *width, int *height, int writable );
950 uint8_t *mlt_frame_get_alpha_mask( mlt_frame this );
951 int mlt_frame_get_audio( mlt_frame this, int16_t **buffer, mlt_audio_format *format, int *frequency, int *channels, int *samples );
952 int mlt_frame_push_get_image( mlt_frame this, mlt_get_image get_image );
953 mlt_get_image mlt_frame_pop_get_image( mlt_frame this );
954 int mlt_frame_push_frame( mlt_frame this, mlt_frame that );
955 mlt_frame mlt_frame_pop_frame( mlt_frame this );
956 int mlt_frame_push_service( mlt_frame this, void *that );
957 void *mlt_frame_pop_service( mlt_frame this );
958 int mlt_frame_push_audio( mlt_frame this, void *that );
959 void *mlt_frame_pop_audio( mlt_frame this );
960 void mlt_frame_close( mlt_frame this );
965 The service base class extends properties and allows 0 to m inputs and 0 to
966 n outputs and is represented as follows:
976 Descendents of service impose restrictions on how inputs and outputs can be
977 connected and will provide a basic set of properties. Typically, the service
978 instance is encapsulated by the descendent in order for it to ensure that
979 its connection rules are followed.
981 A service does not define any properties when constructed. It should be
982 noted that producers, filters and transitions my be serialised (say, via the
983 westley consumer), and care should be taken to distinguish between
984 serialisable and transient properties. The convention used is to prefix
985 transient properties with an underscore.
987 The public interface is defined by the following functions:
989 int mlt_service_init( mlt_service this, void *child );
990 mlt_properties mlt_service_properties( mlt_service this );
991 int mlt_service_connect_producer( mlt_service this, mlt_service producer, int index );
992 int mlt_service_get_frame( mlt_service this, mlt_frame_ptr frame, int index );
993 void mlt_service_close( mlt_service this );
995 Typically, only direct descendents of services need invoke these methods and
996 developers are encouraged to use those extensions when definining new
1002 A producer has 0 inputs and 1 output:
1012 A producer provides an abstraction for file readers, pipes, streams or any
1013 other image or audio input.
1015 When instantiated, a producer has the following properties:
1017 +------------------+------------------------------------+------------------+
1018 |Name |Description |Values |
1019 +------------------+------------------------------------+------------------+
1020 |mlt_type |The producers type |mlt_producer |
1021 +------------------+------------------------------------+------------------+
1022 |_position |The producers frame position |0 to n |
1023 +------------------+------------------------------------+------------------+
1024 |_speed |The producers speed |double |
1025 +------------------+------------------------------------+------------------+
1026 |fps |The output frames per second |25 or 29.97 |
1027 +------------------+------------------------------------+------------------+
1028 |in |The in point in frames |0 to length - 1 |
1029 +------------------+------------------------------------+------------------+
1030 |out |The out point in frames |in to length - 1 |
1031 +------------------+------------------------------------+------------------+
1032 |length |The length of the input in frames |0 to n |
1033 +------------------+------------------------------------+------------------+
1034 |aspect_ratio |aspect_ratio of the source |0 to n |
1035 +------------------+------------------------------------+------------------+
1036 |eof |end of clip behaviour |"pause" or "loop" |
1037 +------------------+------------------------------------+------------------+
1038 |resource |Constructor argument (ie: file name)|"<resource>" |
1039 +------------------+------------------------------------+------------------+
1041 Additional properties may be provided by the producer implementation.
1043 The public interface is defined by the following functions:
1045 mlt_producer mlt_producer_new( );
1046 int mlt_producer_init( mlt_producer this, void *child );
1047 mlt_service mlt_producer_service( mlt_producer this );
1048 mlt_properties mlt_producer_properties( mlt_producer this );
1049 int mlt_producer_seek( mlt_producer this, mlt_position position );
1050 mlt_position mlt_producer_position( mlt_producer this );
1051 mlt_position mlt_producer_frame( mlt_producer this );
1052 int mlt_producer_set_speed( mlt_producer this, double speed );
1053 double mlt_producer_get_speed( mlt_producer this );
1054 double mlt_producer_get_fps( mlt_producer this );
1055 int mlt_producer_set_in_and_out( mlt_producer this, mlt_position in, mlt_position out );
1056 mlt_position mlt_producer_get_in( mlt_producer this );
1057 mlt_position mlt_producer_get_out( mlt_producer this );
1058 mlt_position mlt_producer_get_playtime( mlt_producer this );
1059 mlt_position mlt_producer_get_length( mlt_producer this );
1060 void mlt_producer_prepare_next( mlt_producer this );
1061 void mlt_producer_close( mlt_producer this );
1066 The public interface is defined by the following functions:
1068 int mlt_filter_init( mlt_filter this, void *child );
1069 mlt_filter mlt_filter_new( );
1070 mlt_service mlt_filter_service( mlt_filter this );
1071 mlt_properties mlt_filter_properties( mlt_filter this );
1072 mlt_frame mlt_filter_process( mlt_filter this, mlt_frame that );
1073 int mlt_filter_connect( mlt_filter this, mlt_service producer, int index );
1074 void mlt_filter_set_in_and_out( mlt_filter this, mlt_position in, mlt_position out );
1075 int mlt_filter_get_track( mlt_filter this );
1076 mlt_position mlt_filter_get_in( mlt_filter this );
1077 mlt_position mlt_filter_get_out( mlt_filter this );
1078 void mlt_filter_close( mlt_filter );
1083 The public interface is defined by the following functions:
1085 int mlt_transition_init( mlt_transition this, void *child );
1086 mlt_transition mlt_transition_new( );
1087 mlt_service mlt_transition_service( mlt_transition this );
1088 mlt_properties mlt_transition_properties( mlt_transition this );
1089 int mlt_transition_connect( mlt_transition this, mlt_service producer, int a_track, int b_track );
1090 void mlt_transition_set_in_and_out( mlt_transition this, mlt_position in, mlt_position out );
1091 int mlt_transition_get_a_track( mlt_transition this );
1092 int mlt_transition_get_b_track( mlt_transition this );
1093 mlt_position mlt_transition_get_in( mlt_transition this );
1094 mlt_position mlt_transition_get_out( mlt_transition this );
1095 mlt_frame mlt_transition_process( mlt_transition this, mlt_frame a_frame, mlt_frame b_frame );
1096 void mlt_transition_close( mlt_transition this );
1101 The public interface is defined by the following functions:
1103 int mlt_consumer_init( mlt_consumer this, void *child );
1104 mlt_service mlt_consumer_service( mlt_consumer this );
1105 mlt_properties mlt_consumer_properties( mlt_consumer this );
1106 int mlt_consumer_connect( mlt_consumer this, mlt_service producer );
1107 int mlt_consumer_start( mlt_consumer this );
1108 mlt_frame mlt_consumer_get_frame( mlt_consumer this );
1109 mlt_frame mlt_consumer_rt_frame( mlt_consumer this );
1110 int mlt_consumer_stop( mlt_consumer this );
1111 int mlt_consumer_is_stopped( mlt_consumer this );
1112 void mlt_consumer_close( mlt_consumer );
1115 Specialised Producers:
1117 There are two major types of specialised producers - playlists and tractors.
1119 The following sections describe these.
1124 mlt_playlist mlt_playlist_init( );
1125 mlt_producer mlt_playlist_producer( mlt_playlist this );
1126 mlt_service mlt_playlist_service( mlt_playlist this );
1127 mlt_properties mlt_playlist_properties( mlt_playlist this );
1128 int mlt_playlist_count( mlt_playlist this );
1129 int mlt_playlist_clear( mlt_playlist this );
1130 int mlt_playlist_append( mlt_playlist this, mlt_producer producer );
1131 int mlt_playlist_append_io( mlt_playlist this, mlt_producer producer, mlt_position in, mlt_position out );
1132 int mlt_playlist_blank( mlt_playlist this, mlt_position length );
1133 mlt_position mlt_playlist_clip( mlt_playlist this, mlt_whence whence, int index );
1134 int mlt_playlist_current_clip( mlt_playlist this );
1135 mlt_producer mlt_playlist_current( mlt_playlist this );
1136 int mlt_playlist_get_clip_info( mlt_playlist this, mlt_playlist_clip_info *info, int index );
1137 int mlt_playlist_insert( mlt_playlist this, mlt_producer producer, int where, mlt_position in, mlt_position out );
1138 int mlt_playlist_remove( mlt_playlist this, int where );
1139 int mlt_playlist_move( mlt_playlist this, int from, int to );
1140 int mlt_playlist_resize_clip( mlt_playlist this, int clip, mlt_position in, mlt_position out );
1141 void mlt_playlist_close( mlt_playlist this );