HLS and Fragmented MP4

23 Replies

At WWDC 2016, Apple announced support for fragmented MP4 (fMP4) as an alternative to MPEG-TS, which prior to their announcement was the only supported format.

So why use fragmented MP4 files? Well, according to Apple’s video encoding requirements in their HLS Authoring Specification, if you want to use HEVC/H.265, you have to use it (1.5). Fragmented MP4 files are also compatible with MPEG-DASH – an alternative to HLS – so you can use the same files; only the manifest file (playlist) is different. This means less encoding and less storage requirements, which should reduce costs.

In this post, I’ll demonstrate how to generate fMP4 files using ffmpeg. You’ll need a fairly recent version of ffmpeg that supports fMP4 – I used version 4.1.1 to create the files in this post.

For the video, I used the 1080p version of the Sintel trailer. You can download it here.

Let’s take the original video (sintel_trailer-1080p.mp4) and re-encode it. We’ll alter the size to 1280×720 and reduce the video bitrate but leave the audio as it is. Open up a terminal and run the following command:

ffmpeg -y \
  -i sintel_trailer-1080p.mp4 \
  -force_key_frames "expr:gte(t,n_forced*2)" \
  -sc_threshold 0 \
  -s 1280x720 \
  -c:v libx264 -b:v 1500k \
  -c:a copy \
  -hls_time 6 \
  -hls_playlist_type vod \
  -hls_segment_type fmp4 \
  -hls_segment_filename "fileSequence%d.m4s" \
  prog_index.m3u8

To generate fMP4 files instead of MPEG-TS, which is the default, set the segment type to fmp4 (highlighted).

The above command will generate the following files: the playlist file (prog_index.m3u8), a number of files with the extension .m4s (instead of .ts), and a file called init.mp4. You’ll need all of these files when you come to play the video.

Let’s take a look at the playlist and see how it compares:

#EXTM3U
#EXT-X-VERSION:7
#EXT-X-TARGETDURATION:6
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-MAP:URI="init.mp4"
#EXTINF:6.000000,
fileSequence0.m4s
#EXTINF:6.000000,
fileSequence1.m4s
#EXTINF:6.000000,
fileSequence2.m4s
#EXTINF:6.000000,
fileSequence3.m4s
#EXTINF:6.000000,
fileSequence4.m4s
#EXTINF:6.000000,
fileSequence5.m4s
#EXTINF:6.000000,
fileSequence6.m4s
#EXTINF:6.000000,
fileSequence7.m4s
#EXTINF:4.208333,
fileSequence8.m4s
#EXT-X-ENDLIST

I’ve highlighted the differences. The #EXT-X-MAP tag specifies the location of the resource that contains the information required to parse the segments (the media initialization section). It applies to every segment that appears after it. A playlist containing an MPEG-2 transport stream can also apply the #EXT-X-MAP tag, but each segment typically includes the information required to parse it so it’s not needed.

The other difference is the file extension of the segments is .m4s instead of .ts.

HLS also supports byte range addressing. To use this feature with fMP4, set the -hls_flags option to single_file. Instead of creating several (smaller) files, it will create a single file containing all the media segments. If you look at the playlist, the location of each segment is specified as a byte offset in the file.

Now that we have the fMP4 files and the playlist, the next step is to play the video in the browser.

Most browsers don’t support HLS and fMP4 natively so you’ll need to use a video player that utilises Media Source Extensions (MSE). Thankfully, some players do but be aware that some don’t. For example, I tried using hls.js but it didn’t work – according to the documentation, support for fragmented MP4 is still in beta.

Here’s an HTML example that uses Video.js, which does support fMP4:


<!DOCTYPE html>
<html>
<head>
<link href="https://vjs.zencdn.net/7.4.1/video-js.css" rel="stylesheet">
</head>
<body>
<video class="video-js" width="1280" height="720" data-setup='{}' controls>
        <source src="prog_index.m3u8" type="application/x-mpegURL">
</video>
<script src='https://vjs.zencdn.net/7.4.1/video.js'></script>
</body>
</html>

Save the HTML. You’ll need to upload this along with all the other files created in the previous step to a web server. I tested playback in Chrome 72 and Firefox 65.0 on Ubuntu and the video played with no problems.

Conclusion

Using ffmpeg makes it easy to generate fMP4 files for HLS. If you need to support other streaming protocols like MPEG-DASH then it’s worth considering. If you encode videos with HEVC/H.265 then fMP4 is mandatory.

23 thoughts on “HLS and Fragmented MP4”

  1. Dilovar

    Hello, when I specify a parameter -hls_flags single_file playlist #EXT-X-MAP has a url to the file, how do I remove this path? //vidione.loc/ I added via parameter -hls_base_url
    here are the playlist parameters.
    #EXTM3U
    #EXT-X-VERSION:7
    #EXT-X-TARGETDURATION:6
    #EXT-X-MEDIA-SEQUENCE:0
    #EXT-X-MAP:URI=”storage/AEFIXUum/res-0/segment.m4s”,BYTERANGE=”822@0″
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:92580@822
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:124656@93402
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:125761@218058
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:80175@343819
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:131521@423994
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:136767@555515
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:6.000000,
    #EXT-X-BYTERANGE:140335@692282
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXTINF:3.440000,
    #EXT-X-BYTERANGE:76813@832617
    //vidione.loc/storage/AEFIXUum/res-0/segment.m4s
    #EXT-X-ENDLIST

    Reply ↓
      1. Dilovar

        Hello, this bug has already been fixed. there is another error when I split the video and audio into streams when switching the quality of the picture hangs for a couple of seconds.

        $var_stream_map = ($has_audio) ? ‘v:0,agroup:audio v:1,agroup:audio v:2,agroup:audio v:3,agroup:audio a:0,agroup:audio’ : ‘v:0 v:1 v:2 v:3’;
        $cmd = ‘-preset veryfast -r 25 -g 50 -keyint_min 50 -threads 12 -sc_threshold 0 -map 0:v:0? -map 0:v:0? -map 0:v:0? -map 0:v:0? -map 0:a:0? -filter:v:0 scale=”\’w=if(gt(a,16/9),426,-2):h=if(gt(a,16/9),-2,240)\'” -crf:v:0 23 -maxrate:v:0 400k -bufsize:v:0 400k -filter:v:1 scale=”\’w=if(gt(a,16/9),640,-2):h=if(gt(a,16/9),-2,360)\'” -crf:v:1 23 -maxrate:v:1 700k -bufsize:v:1 700k -filter:v:2 scale=”\’w=if(gt(a,16/9),854,-2):h=if(gt(a,16/9),-2,480)\'” -crf:v:2 23 -maxrate:v:2 1250k -bufsize:v:2 1250k -filter:v:3 scale=”\’w=if(gt(a,16/9),1280,-2):h=if(gt(a,16/9),-2,720)\'” -crf:v:3 23 -maxrate:v:3 2500k -bufsize:v:3 2500k -c:a aac -b:a 128k -ac 2 -var_stream_map “‘ . $var_stream_map . ‘” -master_pl_name master.m3u8 -f hls -hls_time 6 -segment_time 3 -hls_list_size 0 -hls_flags single_file+independent_segments -hls_segment_type fmp4 -hls_segment_filename ‘ . $segmentFolderName . ‘/res-%v/segment.m4s ‘ . $segmentFolderName . ‘/res-%v/playlist.m3u8 -ss ‘ . $position . ‘ -vframes 1 ‘ . $mainVideoFolder . ‘/poster.jpg -ss ‘ . $position . ‘ -vframes 1 -filter:v scale=h=320:w=-2 ‘ . $mainVideoFolder . ‘/thumbnail.jpg’;

        Reply ↓
    2. GoTesla

      @Dilovar I’m running into the same problem: my m3u8 shows #EXT-X-MAP:URI=”/home/vagrant/Code/myproject/public/storage/media/000000000/000000001/init.mp4″ because of my -hls_fmp4_init_filename, but I need that line in the m3u8 to be relative rather than the absolute path to a local file. How did you fix your problem? Thanks!

      @Simon thanks for this helpful post!

      Reply ↓
  2. Onur

    Hey Simon, thank you for these very valuable insights. I am new to ffmpeg but is it always the case, that if you transcode the initial mp4 file ( I am using a 5,5mb file) in its sequences, that the total size of the sum of sequence will be much larger?

    Reply ↓
  4. chrisahrweiler

    Hello Simon, great tutorial, thank you.
    I used your skript and code example to encode one of my videos and it plays just fine in firefox. However, when I seek forward/backward in the video, videojs throws and error and hangs.

    https://yalac.com/hlsbook/

    TypeError: a is undefined videojs.min.js:21:41040

    Any idea what might be wrong here?

    Regards, Chris

    Reply ↓
    1. Simon Post author

      Hi Chris,

      The problem seems to occur at 2:53 into the video. Happens with Firefox and Chrome for me. I would suggest linking to the version of Video.js that hasn’t been minimised so you can see where in the code the error occurs. That may give you a better idea of what the problem is.

      Reply ↓
      2. chrisahrweiler

        Hi Simon,

        i have read your book and implemented encryption as you suggested in chapter 7 and in your article “How to Encrypt Video for HLS” (using ffmeg).
        Now I have multiple .ts files and a single enc.key that is re-loaded in conjunction with every *.ts load.
        enc.key will be delivered from root, where the *.ts files reside in their subdirectories (v%).

        Is there a way to force the browser to load the enc.key only ones, and decrypt all the *.ts with this single enc.key?
        I tried to build a data: uri and provide the key, but that didn’t work:

        key=$(openssl rand 16)
        echo $key > enc.key
        echo “data:application/octet-stream;base64,$key” > enc.keyinfo
        echo “enc.key” >> enc.keyinfo
        openssl rand -hex 16 >> enc.keyinfo

        data: could also be a way to make it harder for video downloaders to grep the enc.key I thought.

        Any advice on this?

        Thank you, Chris

        Reply ↓
        1. Simon Post author

          From the HLS spec:

          The server MAY set the HTTP Expires header in the key response to indicate the duration for which the key can be cached.

          I did a quick test with VideoJS and it appears to ignore it. I also tried hls.js and that did cache the key when the Expires header was present.

          Reply ↓
          1. chrisahrweiler

            I have set

            cache-control: max-age=86400, public

            key request are answered with

            Status Code: 200 (from disk cache)

            so caching works, but there is still an enc.key request with every *.ts

            enc.key
            vs1.ts
            enc.key
            vs2.ts
            enc.key
            vs3.ts

            Even though there is only one line
            #EXT-X-KEY: …
            in prog_index.m3u8

            That’s strange, isn’t it?

  5. Diego Rosenfeld

    Hi Simon!

    I’ve been reading you book and your code examples, by the way, they are great!

    As I couldn’t find more information about fmp4 in the book, I’m writting a little doubt here:

    Until now, i’ve been using ffmpeg, transcoding videos to support http streaming through:
    – HLS: .m3u8 playlists + .ts video files
    – DASH: .mpd playlist + .m4s video files

    After reading your post, I used the hls_segment_type=fmp4 flag from ffmpeg, and successfully generated a transcoding compatible with HLS players: .m3u8 playlists + .m4s video files + _init.mp4 files.

    It works fine using Video.js, and also using an online demo player, like Bitmovin (https://bitmovin.com/demos/stream-test), but only if I select HLS stream type.

    Obviously, if I select DASH stream type of Bitmovin player, and try to load the master playlist generated (.m3u8), I get a “SOURCE_MANIFEST_INVALID” error, because player is waiting a .mpd manifest on its place.

    So, my question is: Is it possible to run one single ffmpeg command, and generate al the files you mentioned in the post, plus the .mpd playlist, so that this segments can be loaded by both HLS and DASH player ?

    Thanks in advance!!!

    Reply ↓
    1. Simon Post author

      Take a look at the dash muxer. There are some options for generating the HLS playlist as well as the MPD playlist.

      Something like this should work:

      ffmpeg -y \
  -i sintel_trailer-1080p.mp4 \
  -force_key_frames "expr:gte(t,n_forced*2)" \
  -sc_threshold 0 \
  -s 1280x720 \
  -c:v libx264 -b:v 1500k \
  -c:a copy \
  -f dash \
  -seg_duration 6 \
  -hls_playlist true \
  prog_index.mpd

      There are some options for naming the segment files and so on that you may want to play around with. Take a look at the documentation: ffmpeg -help muxer=dash.

      MP4Box also has an option to convert a .m3u8 playlist into an MPD file.

      Reply ↓
  6. thiagoeee

    Fantastic book Simon. It helped me a lot to understand HLS and the multiple ways to use it with FFMPEG. Using your example above, could you help us by completing the script to generate a master list that will load other video options on the player by converting the original video to 360p, 540p, 720p and 1080p?

    I built the following script using what I learned from your work:


    -y
    -hwaccel cuda
    -i "D:\Conversao-de-videos\download\%f.mkv"
    -c:v h264_nvenc
    -force_key_frames "expr:gte(t,n_forced*2)"
    -preset medium
    -sc_threshold 0
    -s 1920x1080

    -map 0:0 -map 0:1 -map 0:0 -map 0:1 -map 0:0 -map 0:1 -map 0:0 -map 0:1

    -s:v:0 640x360 -c:v:0 libx264 -b:v:0 365k
    -s:v:1 960x540 -c:v:1 libx264 -b:v:1 2000k
    -s:v:2 1280x720 -c:v:2 libx264-b:v:2 3500k
    -s:v:3 1920x1080 -c:v:3 libx264 -b:v:3 4500k

    -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3"

    -c:a copy
    -hls_time 2 -hls_list_size 0
    -hls_playlist_type vod
    -hls_segment_type fmp4
    -master_pl_name master.m3u8
    -hls_segment_filename "v%v/fileSequence%d.m4s"
    v%v/prog_index.m3u8

    But I think I’m getting in the way with -map. I am receiving the following alert:

    Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 0, only the last option ‘-c: v: 0 libx264’ will be used.
    Multiple -s options specified for stream 0, only the last option ‘-s: v: 0 640×360’ will be used.

    Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 2, only the last option ‘-c: v: 1 libx264’ will be used.
    Multiple -s options specified for stream 2, only the last option ‘-s: v: 1 960×540’ will be used.

    Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 4, only the last option ‘-c: v: 2 libx264’ will be used.
    Multiple -s options specified for stream 4, only the last option ‘-s: v: 2 1280×720’ will be used.

    Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 6, only the last option ‘-c: v: 3 libx264’ will be used.

    Multiple -s options specified for stream 6, only the last option ‘-s: v: 3 1920×1080’ will be used.

    Stream mapping:
    Stream # 0: 0 -> # 0: 0 (h264 (native) -> h264 (libx264))
    Stream # 0: 1 -> # 0: 1 (copy)
    Stream # 0: 0 -> # 0: 2 (h264 (native) -> h264 (libx264))
    Stream # 0: 1 -> # 0: 3 (copy)
    Stream # 0: 0 -> # 0: 4 (h264 (native) -> h264 (libx264))
    Stream # 0: 1 -> # 0: 5 (copy)
    Stream # 0: 0 -> # 0: 6 (h264 (native) -> h264 (libx264))
    Stream # 0: 1 -> # 0: 7 (copy)

    Where am I going wrong? is in the mapping? Would you help me?

    Reply ↓
    1. Simon Post author

      You need to delete the -c:v h264_nvenc and -s 1920x1080 options. You don’t need to use the -s option because you specify the 1080p variant in the map. If you want to use the hardware assisted H.264 encoder then replace libx264 with h264_nvenc. That should work.

      Reply ↓
  7. shav1983

    Awesome blog Simon! Looking forward for your book. I followed few of your posts, fMP4 and HLS and came down to the following command. The produced video is able to play in android but unable in iOS. I tried using media stream validator tool of apple’s. It has thrown “Error injecting segment data” on terminal while generating the validation report. I don’t know how this supposed to be handled and what I am missing, really stuck for a long time, please help.

    ffmpeg -y -i input.mp4 \
    -preset fast -g 60 -sc_threshold 0 \
    -map 0:0 -map 0:1 -map 0:0 -map 0:1 \
    -s:v:0 640×360 -c:v:0 libx265 -b:v:0 365k -bufsize 365k\
    -s:v:1 960×540 -c:v:1 libx265 -b:v:1 2000k -bufsize 2000k\
    -c:a copy \
    -movflags faststart \
    -profile:v main10 \
    -pix_fmt yuv420p10 \
    -tag:v hvc1 \
    -movflags faststart \
    -var_stream_map “v:0,a:0 v:1,a:1” \
    -master_pl_name master.m3u8 \
    -hls_playlist_type vod \
    -hls_segment_type fmp4 \
    -hls_flags single_file \
    -f hls -hls_time 6 -hls_list_size 0 \
    -x265-params ‘min-keyint=60:keyint=60:open-gop=0:no-scenecut=1:stats=input_log:level-idc=4.1’ \
    -hls_segment_filename “v%v/main.mp4” \
    v%v/prog_index.m3u8

    Reply ↓
  10. nils

    how can i stream a fragmented fmp4 wrapped around index.m3u8

    regular streaming using ffmpeg drops bitrate i believe its due to fmp4

    Reply ↓
  11. Nguyên Vũ Nguyễn

    Hi Simon,
    It’s a luck for me when reading your post, it’s great giving me hints on resolving my problem.

    I were following your post to transcode a flac file to fragmented-mp4, it transcode sucessfull but can not play the flac one. Can you please give me an advise on this, anything wrong with my command:

    ffmpeg -i audio.flac
    -map 0:a -map 0:a -map 0:a -map 0:a
    -c:a:0 flac -c:a:1 aac
    -c:a:2 aac
    -c:a:3 aac
    -b:a:1 320k
    -b:a:2 160k
    -b:a:3 96k
    -f hls
    -hls_playlist_type vod
    -hls_segment_type fmp4
    -hls_time 3
    -hls_flags single_file
    -master_pl_name master.m3u8
    -var_stream_map “a:0,name:flac a:1,name:320 a:2,name:160 a:3,name:96” ./stream_v%v/index.m3u8

    Thanks a lot

    Reply ↓

Leave a Reply