[Mlt-devel] Question about video compositing

[Alex S. <al...@al...>]

 Apologies if this is the wrong mailing list, I only saw a devel list not a
user list.

I have a question about video compositing. I originally asked in the
ffmpeg-user mailinglist and it was suggested that MLT/melt would be a
better tool. So my question becomes, for the below outcome will it be
feasible to achieve this and is it a good fit for my use case? I'm happy to
dive into the docs but I don't want to go on a wild goose chase if I'm
looking at the wrong tool for the job.

I’ve included the text of the question below but I’ve also put it in a gist
for easier to read formatting here: https://gist.github.com/alexspeller/
aefdd5a6d7100d28d0bbc4838527f797

I have multiple mp4 video files and I want to composite them into a
single video on the command line. Each stream is an mp4 video. They
are of different lengths, and each file also has audio.

The tricky thing is, I want the layout to change depending on how many
streams are currently visible.

As a concrete example, say I have 3 video files:

| File  | Duration | Start | End |
|-------|----------|-------|-----|
| a.mp4 | 30s      | 0s    | 30s |
| b.mp4 | 10s      | 10s   | 20s |
| c.mp4 | 15s      | 15s   | 30s |


So at t=0 seconds, I want the video to look like this:

```
   +-------------------------+
   |                         |
   |                         |
   |                         |
   |                         |
   |          a.mp4          |
   |                         |
   |                         |
   |                         |
   |                         |
   |                         |
   +-------------------------+


```

At t=10s, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         b.mp4                |
|                              |
|                              |
|                              |
|                              |
|                              |
+------------------------------+
```

At t=15s, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         b.mp4                |        |
|                              |        |
|                              | c.mp4  |
|                              |        |
|                              +--------+
|                              |
+------------------------------+
```

And at t=20s until the end, I want the video to look like this:

```
+------------------------------+--------+
|                              |        |
|                              |        |
|                              | a.mp4  |
|                              |        |
|                              +--------+
|         c.mp4                |
|                              |
|                              |
|                              |
|                              |
|                              |
+------------------------------+
```

Ideally there would be some animated transitions between the states,
but that's not essential.

I have found two possible approaches that might work, but I'm not sure
what the best one is. The first is using
[filters](https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos)
to acheive the result, but I'm not sure if it will cope well with (a)
the changing layouts and (b) keeping the audio without any artefacts
when the layout changes.

The other approach I thought of would be exporting all frames to
images, building new frames with imagemagick, and then layering the
new frames on top of the audio like in [this blog
post](https://broadcasterproject.wordpress.com/2010/05/18/how-to-layerremix-videos-with-free-command-line-tools/).

Any suggestions on if either of these approaches is better, or any
alternatives? Thanks!