Codec · C++

How to fuzz x265

HEVC's coding-tree architecture multiplies the buffer-sizing complexity versus H.264.

x265 implements HEVC's coding tree units, flexible transform sizes (4x4 to 32x32), and sample adaptive offset filtering — each adds buffer-sizing edge cases that are hard to reason about statically. As the dominant open-source HEVC encoder it is present in virtually every software transcoder.

Common bug classes

•Heap buffer overflow in CTU partition boundary write
•Out-of-bounds read in SAO filter line buffer
•Integer overflow in tile/slice row arithmetic
•Use-after-free in lookahead frame reference management
•Divide-by-zero in rate control lambda computation

Recommended setup

Fuzzers

→ AFL++
→ libFuzzer

Sanitizers

→ ASan
→ UBSan

Harness scaffold

#include <stdint.h>
#include <stddef.h>
#include <x265.h>

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  if (size < 4) return 0;
  x265_param *param = x265_param_alloc();
  x265_param_default(param);
  x265_param_parse(param, "preset", "ultrafast");
  param->sourceWidth  = 64;
  param->sourceHeight = 64;
  param->logLevel = X265_LOG_NONE;
  x265_encoder *enc = x265_encoder_open(param);
  if (enc) {
    x265_picture *pic = x265_picture_alloc();
    x265_picture_init(param, pic);
    x265_nal *nals; uint32_t cnt;
    x265_encoder_encode(enc, &nals, &cnt, pic, NULL);
    x265_picture_free(pic);
    x265_encoder_close(enc);
  }
  x265_param_free(param);
  return 0;
}

Save this as fuzz_target.cc, build with your compiler + sanitizer flags, and you have a working starting point.

Start fuzzing x265 on Fuzze.rs →

Push the harness above + a Dockerfile. First month 50% off.