2.1

Compression efficiency axis

Several approaches on how compression efficiency can be aggregated were presented in our previous work:¹⁴ conventional convex hull approach, combined convex hull approach, and restricted discrete combined convex hull approach. We opt to use the third method: discrete combined convex hull approach, and create an adaptive bitrate (ABR) ladder of 8 lanes targeting quality specific quality levels as measured by VMAF¹⁹ and SSIM.²⁰ A quick refresh of the construction of these lanes can be found in Fig. 2.

Figure 2.

Combined Convex Hull Construction with Dynamic Optimizer

2.2

Energy usage axis

In recent codec comparisons, we have employed the Linux utility ”/usr/bin/time -v” to record the user and system time of the encoding pipeline, when ensuring the system is fully loaded with encoding jobs. This measure has been found to be strongly correlated with energy consumption, as demonstrated previously.¹⁴ Additionally, this metric is neutral and does not rely on implementation-specific reporting.

The time utility calculates the number of cycles used by a process, which essentially tracks the CPU usage of the process. This is quite applicable for software-based implementations. However, for hardware encoding, the processing is performed on specialized hardware components rather than the general purpose CPU. Thus, any cycles captured would not be comparable across hardware and software implementations. Note that even if we could capture the hardware cycle count, it cannot be compared directly with CPUs or with other different hardwares, since they represent different HW architectures, system clock speeds and power consumption per cycle. To address this, we propose to use the power consumption of each component processing the software or hardware implementations. To normalize the comparison, we multiplied the power consumption by the time it takes to run a fixed set of encodings, which is effectively measuring the energy used. This energy measure is also commonly used for electrical bills, electronics consumption factors, and electric vehicle electricity usage. Most importantly, energy consumption is the dominant factor used to determine the operational-expense (OpEx) cost of a service running on Meta’s data centers.

3.1

Content Selection

For this test, we use the following pool of clips to start with:

We encode all 322 clips with 8 CRF values ranging from 23 - 63 and 5 resolutions (1080p down to 288p) using the SVT-AV1 encoder¹³ in preset 8, and compute the VMAF score at original resolutions. K-means clustering was performed on the produced RD (rate-distortion) points. The closest one to the center of each cluster is selected. We thus select a diverse set of contents without having to use the full list.

As a result, we ended up selecting a total of 200 clips split across the following buckets:

Looking at the resulting test set above we have also noticed that the 30fps content represents more than two thirds where the rest is split across 60, 25, and 24 fps. The full distribution by duration is shown in Table. 1.

Table 1.

Framerate Distribution of the Selected Video Test Set

3.2

The Transcoding Pipeline

To perform transcoding on the selected video clips, we have followed our previous work.¹⁴ A quick refresher on the pipeline can be in Fig. 3.

Figure 3.

Shot Based Encoding Workflow

The steps are described briefly as follows.

3.3

Selecting the Resolutions

We have reduced the resolution selection due to certain hardware encoders not supporting some resolutions used previously. As such the following resolution set has been selected:

We have opted to pre-downscale the content using the FFmpeg²⁵ Lanczos filter with alpha parameter set to 5 for all encoders and not utilize the scalars that are available in the hardware System-on-Chip (SoCs) to make sure we restrict the comparison to the encoder’s compression efficiency. A sample FFmpeg command line to perform such downscaling is shown here:

3.4

Selecting the Encoders

We have chosen a wide selection of software encoder representation representing all codecs, as shown in the following:

Example encoding commands used with each encoder can be found in Appendix A.

For hardware encoders, we have selected the Meta Scalable Video Processor (MSVP) VP9 encoder. A picture of the cards used can be seen in Fig. 4.

Figure 4.

Meta Scalable Video Processor (MSVP)

3.5

Selecting the Encoding Parameters

We choose the same selected range of CRFs that would align the quality with that achieved by SVT-AV1. The per encoder CRFs are shown as follows for each encoder implementation:

3.6

Measuring the Compression Efficiency

To assess the compression efficiency, we employed the combined restricted dynamic optimizer method and utilized SVT-AV1 M0 as a quality reference. In this instance, we had a mix of frame rates and a slightly different set of resolutions (1920x1080, 1080x1080, and 1080x1920).

We chose to measure the dynamic optimizer per pixel count bucket, combining 1920x1080 and 1080x1920 into one bucket and considering 1080x1080 separately. This is also done for each class of frame rate. We then combined the final results by adding up the file size per video for all buckets and using a weighted average for the quality to obtain the final number.

For example, calculating the M0 preset results of SVT-AV1 in VMAF would involve the following steps:

The above quality vs file size points are used as a reference to compare to any other encoder preset points and measure the BD-rate difference between the two sets of points. We then perform the same operation for each encoder implement and each preset. An example of the VMAF RD-curves comparing the ABR ladders constructed for SVT-AV1 presets M10 to M0 can be found in Fig. 5.

Figure 5.

VMAF Rate Distortion curves comparing the final results of SVT-AV1 M10 and M0

3.7

Measuring and Estimating the Energy Consumption

We first make sure that the platform is utilized as much as possible (either on CPU or hardware card). Doing so in SW pipelines is relatively straightforward by using utilities such as “parallel”²⁸ which dispatches a desired number of parallel tasks and monitors their completion in order to launch new ones, once some of them finish. For HW pipelines, this is more complicated, since each HW vendor has their own mechanism to interface and dispatch tasks to it, but we in general followed the best practice recommended by such HW vendors. Even with that, there is no practical way to ensure full, 100% utilization of these HW video transcoding cards and more is needed to provide a, hopefully, unified framework that allows users to achieve maximum resource utilization.

The maximum power usage of each platform is used and the time it takes to finish all encoding jobs is measured. We then multiply the watts measure (e.g. 12 Watts for the MSVP HW encoder) by the wall clock to finish the encodings (e.g. 1 hour) to get the watts-hour measure per preset (e.g. 12 watt-hour).

4.1

Loading the Card

Unlike the live broadcasting use case, it is common to schedule a wide variety of encoding jobs with different duration and resolutions in a video-on-demand use case. There can be shorter duration as well as lower resolution encodings, both from source or the derived encodings. In our benchmarking on hardware encoders, it appears that some can run into bottlenecks in scheduling and pushing encoding jobs through the card. Specifically, we have seen some performance discrepancies when running exclusively a longer 1080p sequence vs. several shorter sequences with mixed resolutions. Frequent starting up and shutting down the encoding sessions for different video sequences could introduce non-trivial overhead, which prevents us from fully utilizing the hardware engines.

Helper utilities are indeed available to monitor the utilization of different hardware blocks. However, we often need to perform trial-and-error on the more optimal concurrency on different systems, by trying out different concurrent sessions and closely monitoring the utilizations as well as success rate manually. Different resolutions could also need a different number of concurrent sessions to be able to maximize utilization. Encoding failures sometimes can be observed when concurrency is pushed too high.

4.2

Accurate and Representative Power Measurement

It has been a challenge to fairly represent the power consumption or measurement between CPU and hardware in terms of video encoding. There are several factors.

Most commercially available hardware encoders have components other than video encoding. Each system may also contain a different mix of components, which all have a variety of different capabilities. These components can consume power even when sitting idle. In the proposed methodology, we would load the system as much as possible and take the maximum power usage of each platform as the consumption, which does still include the idle components, but there is no easy way to separate them out.

Furthermore, system configurations can quite significantly change the performance. Each hardware encoder in the real world would have to be hosted by a system with CPU, memory, and other components. The choice of these components, as well as the density of the hardware video encoder (i.e., how many such HW accelerator cards are connected on a single host),, which can vary by platform, could affect the normalized power consumption quite noticeably. Even in the case of using a CPU platform, the choice of rest of the system components would also change the consumption. In this benchmarking, we opted to take the entire system for the CPU platform, considering that a hardware encoder card does contain certain general purpose processors as well as on-board memory. While on the other hand, we only look at the maximum power usage of the card itself in this study. We believe that this indeed gives a fairly good representation, given the limitations we mentioned above.

4.3

Covering Use Cases

Major use cases for video encoders are primarily Live and Video-on-demand (VOD). VOD use case at Meta involves full transcoding, high-quality scaling, and quality metrics evaluated at specified viewports.²³ Besides decoding and encoding, such a pipeline includes high quality downscaler, high quality upscaler, as well as diverse sets of quality metrics. We have found that many existing solutions have different quality levels in the downscaler; do not necessarily include a high quality up-scaler; and often times only with limited selection of quality metrics. Furthermore, any one of the different available hardware blocks could become the bottleneck, which highly depends on the use case.

With that in mind, we opted to present the execution of the methodology with a set of encoder-only experiments to eliminate any other factors, such as different length filter taps in downscaler. However, we believe that to accurately capture and evaluate large scale adaptive streaming use cases, the full pipeline should be taken into account.