Advisories

NLTK TweetTokenizer URL Pattern Backtracks Catastrophically on Naked-Domain-Like Input

Go Back
severity
high
date
Affecting
  • nltk >= 0, < 3.10.1

CWE
  • CWE-1333 Inefficient Regular Expression Complexity
CVSS
8.7
CVSS V4 Vector
CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N
Credit
geo-chen
Description
The URLS regular expression in nltk/tokenize/casual.py, compiled into TweetTokenizer.WORD_RE and applied by TweetTokenizer.tokenize, contains a naked-domain branch whose domain-label prefix [a-z0-9]+(?:[.\-][a-z0-9]+)* is unbounded. Input consisting of many alternating label separators can be partitioned in exponentially many ways, and because the branch also requires a trailing top-level domain that such input never supplies, the engine explores those partitions before failing at each offset. A few kilobytes of input therefore consumes seconds to minutes of single-threaded CPU, and the HANG_RE substitution performed before matching does not collapse the pattern. TweetTokenizer is intended for tokenizing untrusted social-media text, so any service that applies it, or the module-level casual_tokenize, to submitted text can be stalled per request without authentication. Version 3.10.1 bounds the label repetition.

Ready to get Started?

Explore VulnCheck, a next-generation Cyber Threat Intelligence platform, which provides exploit and vulnerability intelligence to help you prioritize and remediate vulnerabilities that matter.
  • Vulnerability Prioritization
    Prioritize vulnerabilities that matter based on the threat landscape and defer vulnerabilities that don't.
  • Early Warning System
    Real-time alerting of changes in the vulnerability landscape so that you can take action before the attacks start.