How to fuzz x264

Common bug classes

• •Heap buffer overflow in CABAC coefficient encoder
• •Integer overflow in slice header QP delta handling
• •Out-of-bounds write in intra prediction buffer alignment
• •Divide-by-zero in lookahead cost computation
• •Stack overflow in recursive RDO trellis quantisation

Recommended setup

Harness scaffold

#include <stdint.h>
#include <stddef.h>
#include <stdint.h>
#include <x264.h>

int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
  if (size < 4) return 0;
  x264_param_t param;
  x264_param_default_preset(&param, "ultrafast", "zerolatency");
  param.i_width  = 16 + (data[0] & 0x0F) * 16;
  param.i_height = 16 + (data[1] & 0x0F) * 16;
  param.i_log_level = X264_LOG_NONE;
  param.i_csp = X264_CSP_I420;
  x264_t *enc = x264_encoder_open(&param);
  if (!enc) return 0;
  x264_picture_t pic_in, pic_out;
  x264_picture_alloc(&pic_in, X264_CSP_I420,
                     param.i_width, param.i_height);
  x264_nal_t *nals; int nal_cnt;
  x264_picture_init(&pic_out);
  x264_encoder_encode(enc, &nals, &nal_cnt, &pic_in, &pic_out);
  x264_picture_clean(&pic_in);
  x264_encoder_close(enc);
  return 0;
}

Save this as fuzz_target.cc, build with your compiler + sanitizer flags, and you have a working starting point.