Unsound `Vector::new()`, `Vector::value()` and `Vector::shift()` cause NULL dereference and out-of-bounds access

[ksj1230]

Summary

Three independent soundness bugs exist in the Vector type's safe API. All allow undefined behavior from safe Rust code without unsafe blocks. Confirmed on 9.0.0.

• Issue 1: Unsound Vector::new() with zero length causes NULL pointer dereference
• Issue 2: Unsound Vector::value() with zero length causes NULL pointer dereference
• Issue 3: Unsound Vector::shift() with large value causes out-of-bounds memory access

Environment

• OS: Ubuntu 22.04 LTS (x86_64)
• FFmpeg: 4.4.2 (libswscale-dev 7:4.4.2-0ubuntu0.22.04.1)
• Rust: nightly (AddressSanitizer requires -Z sanitizer=address)

Reproduction

# Install FFmpeg dev libraries
sudo apt install -y libavutil-dev libavformat-dev libavcodec-dev \
  libavfilter-dev libavdevice-dev libswscale-dev libswresample-dev pkg-config

# Run any PoC below with AddressSanitizer
RUSTFLAGS="-Z sanitizer=address" cargo run --release --target x86_64-unknown-linux-gnu

Cargo.toml:

[dependencies]
ffmpeg-next = "9.0.0"

---

Issue 1: Unsound Vector::new() with zero length causes NULL pointer dereference

Root Cause

Vector::new(length) calls sws_allocVec(length as c_int) via FFI. When length == 0, FFmpeg allocates a SwsVector struct but sets coeff = NULL. The Rust wrapper does not check the returned pointer's coeff field or reject zero-length input, so a Vector with a NULL internal buffer is returned to safe code.

All downstream operations dereference the NULL pointer.

Proof of Concept

fn main() {
    use ffmpeg_next::software::scaling::vector::Vector;

    let size: usize = 0_usize;
    let vec = Vector::new(size);
    let _coeffs = vec.coefficients();
    println!("{:?}", _coeffs);
}

ASan Output:

==1897838==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x6533036ece5d bp 0x7ffc71138a50 sp 0x7ffc711389e0 T0)
SUMMARY: AddressSanitizer: SEGV /src/software/scaling/vector.rs:116:40 in <ffmpeg_next::software::scaling::vector::Vector>::coefficients

---

Issue 2: Unsound Vector::value() with zero length causes NULL pointer dereference

Root Cause

Vector::value(value, length) calls sws_getConstVec(value, length as c_int) via FFI. When length == 0, FFmpeg returns a SwsVector with coeff = NULL. The Rust wrapper does not validate this, returning a Vector with a NULL internal buffer to safe code.

All downstream operations dereference the NULL pointer.

Proof of Concept

fn main() {
    use ffmpeg_next::software::scaling::vector::Vector;

    let length: usize = 0_usize;
    let vec = Vector::value(0.5_f64, length);
    let _coeffs = vec.coefficients();
    println!("{:?}", _coeffs);
}

ASan Output:

==1898411==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x5a59123b4e75 bp 0x7ffecccfe350 sp 0x7ffecccfe2e0 T0)
SUMMARY: AddressSanitizer: SEGV /src/software/scaling/vector.rs:116:40 in <ffmpeg_next::software::scaling::vector::Vector>::coefficients

---

Issue 3: Unsound Vector::shift() with large value causes out-of-bounds memory access

Root Cause

Vector::shift(shift: usize) calls sws_shiftVec(vec, shift as c_int) via FFI. The as c_int cast silently truncates values exceeding i32::MAX. For example, shift_amount = 2147483648 (2³¹) wraps to a negative c_int, which sws_shiftVec interprets as a large/negative offset, leading to out-of-bounds memory access.

Proof of Concept

fn main() {
    use ffmpeg_next::software::scaling::vector::Vector;

    let length: usize = 8_usize;
    let shift_amount: usize = 2147483648_usize;  // 2^31, overflows c_int

    let mut vec = Vector::value(1.0_f64, length);
    vec.shift(shift_amount);
}

ASan Output:

==1898889==ERROR: AddressSanitizer: SEGV on unknown address 0x76aaedde00c0 (pc 0x7a0eef94b4ae bp 0x769eedde00c0 sp 0x7ffd2e1ca820 T0)
SUMMARY: AddressSanitizer: SEGV (/lib/x86_64-linux-gnu/libswscale.so.5+0x5f4ae) in sws_shiftVec