Recently, we have implemented a new approach for realtime HEVC decoding called CABAC Streaming, where we perform CABAC and frame reconstruction as TS packets arrive. There have been many presentations about optimising decoders such as dav1d and FFVVC but this to our knowledge is the first time an approach like this has been implemented in FFmpeg specifically for the needs of real-time HEVC decode, in particular, to solve the challenges of decoding high bitrate contribution feeds.
This work led to substantial improvements in realtime HEVC decode performance for the most challenging content.
The majority of this code was written by an LLM, demonstrating that AI is now able to implement complex codec work. That said, the LLM was guided extensively by someone who understood the material.
What’s the challenge
Many presentations from dav1d and FFVVC have shown excellent speed improvements such as 5x from implementing new task scheduling threading models. However, this 5x doesn’t apply to realtime video since benchmarks are usually performed on files. On files, a decoder can just “read ahead”, whereas on realtime video this isn’t possible – there’s nowhere to read ahead to since the decoder has to wait for frames to arrive. While we did implement a task scheduler with some improvements, we were still limited by one big (frame) problem.
The Big Frame Problem
Professional broadcast streams have several major differences to consumer webcasting streams. Video frames usually arrive in realtime in MPEG-TS packets (usually 184 bytes at a time). This is in contrast to webcasting protocols such as MPEG-DASH where entire segments are (usually) completely downloaded and then the segment itself is decoded, similar to a file, and thus not having the same challenges as decoding realtime video that is arriving packet by packet.
Professional broadcast feeds are also very high bitrate, sometimes 50Mbps+, meaning individual frames (usually I-frames) can be very large, over a megabyte. These frames also take a long time to arrive on the wire, often 150-200ms where a traditional software decoder is bottlenecked.
The diagram shows how the decoder sits idle as it waits for large frames to arrive (time goes left to right).
It’s worth also saying professional feeds are also often 10-bit 4:2:2 which most hardware (e.g GPU) decoders cannot decode, or if they can, there are often limitations on the number of streams decodable at the same time.
These large frames are time consuming to decode with the serial CABAC decode process. To make life even more painful, some third-party encoders do not use any tools to allow parallel (threaded) decode such as slices, tiles or WPP. This means the CABAC portion of the whole frame has to be decoded serially in 16ms (on average at 1080p60) – effectively a double whammy of the decoder being bottlenecked twice – once by waiting for the frame to arrive, and again bottlenecked by CABAC. On the most pathological content it’s not usually possible to hit the 16ms deadline without increasing latency to effectively amortise the decode of this frame across ten or more frames.
Measuring the right thing
After implementing a dav1d/FFVVC task manager style model we noticed a problem – while we were achieving 5x+ improvements in decodes on files – realtime decode CPU usage wasn’t falling. This was again down to the fact that the pathological stream in question had limited scope for parallelism and was bottlenecked by arriving data and CABAC. We needed a better metric to see improvements.
Standards compliant MPEG-TS arriving over UDP is different to virtually all other container formats in that its timestamps represent absolute time of decode/presentation with frames (by definition) needing to arrive before decoding. This means we can measure how close a frame is to being late based on how close it was to arriving relative to its PTS post-decode. Owing to the VBV, frames don’t usually arrive close to the PTS but we can measure relative changes.
In reality what we care about is the worst-case frame and its headroom relative to PTS. It was this metric that we used throughout development. Using a cut-down version of our decoder product, we were able to measure worst-case headroom when various difficult streams were played in realtime using multicat.
While we were able to measure overall CPU usage for a given realtime decode, we do not as of yet have a good metric to measure realtime CPU performance, and quantify if any of the portions are bottlenecked. We were able to measure CPU cycles per decode, but this is cumulative over time and doesn’t account for fluctuations in CPU load. The timescales involved in making sure the decoder is not bottlenecked are very small (tens to hundreds of microseconds) and the act of measuring this is difficult.
Task model
The existing FFmpeg code would use a single thread per frame to perform CABAC and reconstruction, an inefficient process.
To begin with we implemented a dav1d/FFVVC style task model, where threads would be given tasks such as reconstruction using the above (non-normative) wavefront pattern.
Threads were greedily given the tasks of reconstruction and filtering as soon as a CTB was ready to be processed.
Note that in this instance CABAC continues to remain serial (in a raster order along the frame) and the task manager distributes work as it becomes available.
The worst-case gain was: 33.3ms. Even though the content had no parallelisable bitstream features, the task manager approach could still eke out some parallelism and cause the worst case frame to arrive earlier.
Asynchronous decode
One of the limitations of software-decoders in FFmpeg is that they are synchronous. One frame in must have a maximum of one frame out. This means during long decoder stalls such as waiting for the large I-frame, several frames may have finished decoding. But these frames must wait for other compressed frames to come to release them, increasing the chance of lateness.
So we implemented FFmpeg’s receive_frame() API that allows for a decoder to return multiple frames as part of a decode call. This freed up frames that had already been decoded (also now decoded faster as a result of the task manager model). Compared to the old API model this had the following gain.
Worst case gain: 77.4ms. This was a surprisingly large number and came as a surprise. But it made sense in hindsight. While waiting for these large frames to arrive, several frames have finished decoding and those frames have to then wait for subsequent frames to come along to be released.
(Note that we do not have a measurement of receive_frame alone without the task manager)
CABAC Stream Decode
Instead of waiting for a whole frame to arrive, we now instead start decoding and reconstructing data as the TS packets arrive, making use of the time we would have spent waiting for the frame. This has a number of challenges which we will go through.
Most MPEG-TS demuxers do not offer the ability to output one TS packet at a time. Thankfully the upipe TS demux can be easily modified to output one TS packet at time as the data arrives on the wire.
RBSP escape per packet
In H.264/HEVC, any sequence of bytes that matches a start code is escaped. We now have to change what was a per frame escape into a per packet escape. The RBSP escape functions were modified to work on streaming data. Owing to the large frame sizes, the unescape functions were already written in assembly.
Opportunistic CABAC decode and reverse
Now that the data was in a suitable format for decoding, there was another issue. When decoding a CTU, there is no way of knowing in CABAC if you have enough data to complete decoding the CTU. So a decoder must opportunistically start decoding CABAC and hope it has enough data. This was implemented, an opportunistic CABAC decode and a rollback if there was not enough data. While this approach allows for improved decode throughput, it comes at the cost of higher CPU load to account for these CABAC rollbacks. Once enough data is available, the CTU can be decoded as normal. On our worst-case clip, the total number of rollbacks was 2%.
Streamed CABAC eventually brought a worst-case gain of 20.5ms.
Final gains
All in all the total gains were as follows:
| Step | Worst-case improvement (ms) |
| Task Manager | +33.3 |
| receive_frame() API | +77.4 |
| Streamed CABAC | +20.5 |
| Total | +131.2 |
131ms overall gain is a good win.
It should be noted that the goal of this work was to improve performance on the worst-case content we could find (i.e content with no parallelism possible). It may be possible to improve the decode speed of other content with parallelism.
All testing was performed on 1080p60 content – the approaches shown here may have different tradeoffs on 2160p content.
The CABAC stream implementation was tested on a variety of samples, both high-bitrate professional broadcast samples and HEVC conformance vectors. Extensive testing of concatenated streams and fuzzed streams were used to make sure this implementation was stable.
The code will be published soon, although it is written on top of an old version of FFmpeg to maintain a consistent user experience.
