Advisories

tokenizers BpeBuilder Buffer Overflow via merge token

Go Back
severity
high
date
Affecting
  • tokenizers <= 0.23.2

CWE
  • CWE-787 Out-of-bounds Write
CVSS
7.1
CVSS V4 Vector
CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:P/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Credit
George Chen
Description
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary key, then writes each concatenated merge rule into it. A merge whose concatenated token exceeds the longest vocabulary key overruns the buffer, which Rust turns into a panic that aborts the process in Rust and FFI embeddings. This occurs at load time with no encoding required, so an attacker who supplies a crafted tokenizer.json can cause a denial of service. A secondary defect at the same location can cause a usize underflow (panic in debug, potential memory corruption in release) when continuing_subword_prefix is set and a merge token is shorter than the prefix. Observed in version 0.23.1.

Ready to get Started?

Explore VulnCheck, a next-generation Cyber Threat Intelligence platform, which provides exploit and vulnerability intelligence to help you prioritize and remediate vulnerabilities that matter.
  • Vulnerability Prioritization
    Prioritize vulnerabilities that matter based on the threat landscape and defer vulnerabilities that don't.
  • Early Warning System
    Real-time alerting of changes in the vulnerability landscape so that you can take action before the attacks start.