Take the 2-minute tour ×

I was perusing the list of VLC's supported video/audio formats and I realized that I actually know very little about streaming media.

When a streaming media feed contains both audio and video, how do these separate signals get merged/integrated into one? At some point, I assume that a video frame (from a camera) and a sound frame (from a mic) get multiplexed together into the same signal?

I would the assume that, from there, the integrated A/V signal gets encoded?

Or, do they both get encoded separately, and then integrated/multiplixed together?

And is multiplexing even the right word/process to describe how they are integrated?

Thanks in advance.

share|improve this question

1 Answer 1

up vote 1 down vote accepted

First of all, you have to differentiate between a video/audio codec, and a multimedia container. I've talked in way more detail about the difference between codecs and containers in another Super User answer. Briefly put, a container is there to hold various audio and video (and subtitle, data, image, …) bitstreams.

This process of merging bitstreams into one container is indeed often referred to as "multiplexing". You could call the process of interleaving audio/video to a container multiplexing, but strictly speaking this term applies to MPEG-2 Transport Stream multiplexing. We'll get to that later.

Now, there are various container formats, but they all serve the same purposes: Dealing with synchronization of the streams inside. You want to make sure that parts belonging together are played at the same time. The container carries meta information that specifically tells a playback device where to look and what to play at, let's say 00:02:01.250.

Other things that containers allow you to do are including information about the stream itself. For example, in many streaming contexts, the player needs to know the size of the video, its frame rate, various other things. When you stream an MP4 file (MPEG-4 Part 14 is a container, not a codec), you want the MOOV atom to be placed at the beginning of the file, so the player can parse it right away. Usually, this "atom" of data would sit at the end of the file where it's not really useful because you'd have to download the entire file before streaming. This is why there are tools to move it to the start, like qt-faststart.

What's important though is that in any multimedia application that multiplexes containers from individual bitstreams, those streams are encoded first. There's no "integrated A/V signal" that is encoded—the container will simply wrap what's there and won't touch the actual audio and video data. So, your video is fed to a video encoder, your audio is fed to an audio encoder. The result from both of these tasks is merged/interleaved in the container. The container will take care of assembling the final bitstream that can be transmitted in a streaming context, or saved as a file.

Now, there are many containers that are suited for different applications. MPEG-2 transport streams are commonly used in TV broadcasting. Here, multiple TV channels are multiplexed into one container, which is then transmitted over the air, cable or satellite. The method of multiplexing is time-division multiplexing. Here, "multiplexing" is also the correct technical term. For other containers, it's not used that often. MPEG-4 Part 14 (MP4, we've briefly mentioned this before) on the other hand is very common in the web and mobile context, and it's becoming more and more of a standard for exchanging files, even in the professional field.

There are other containers, such as MKV, Ogg, MPEG Program Streams, FLV, and many more. Which one to choose depends on the application, as you can see, but this would go far beyond the scope here.

share|improve this answer
Wow - those are two amazing answers @slhck (+1 for both!). A few follow ups: Now that I understand this a little better, it's obvious to me that I want to encode a video bitstream with x264 (so h.264 codec), an audio stream with faac (so AAC), and multiplex/merge them into an MP4 container. This will actually be happening on a small MCU (Raspberry Pi) running Debian. I want to hardcode the device to send the MP4 bitstream to a specific IP address, say, over UDP/IP. Would VLS allow me to do all this? If not, what about FFMPEG? If not, what are my options? Thanks again! –  pnongrata Dec 8 '12 at 13:00
In other words, I guess I'm just not seeing where x264, faac, FFMPEG and VLS would "snap in" to an architecture where you have a device that take a video stream from a camera, and audio stream from a mic, encodes, compresses and merges them, and then transmits the final bitstream. –  pnongrata Dec 8 '12 at 13:04
You've got that all perfectly right :) I don't know about VLC, but I know that FFmpeg can encode video from a web camera in Linux, and simultaneously can also encode audio on the fly, wrap them in MP4, and stream them to the network. Take a look at ffserver specifically, and here for an example application. VLC should be able to do that as well, though I've never tried that. –  slhck Dec 8 '12 at 13:17

Your Answer


