Pretraining Data Can Be Poisoned through Computational Propaganda

Summary§

This paper explores the feasibility of poisoning large language model (LLM) pretraining data through web-scale content injection mechanisms, specifically public discussion interfaces (e.g., forum comments, user-editable pages). The authors demonstrate that adversarial content injected into such platforms can survive web crawling and standard data curation pipelines, leading to persistent backdoors in trained LMs. To quantify this risk, they introduce HalfLife, a method that estimates the retention of poisoned data across curation steps. The work highlights a realistic attack vector and provides tools for assessing data integrity.

Core Methodology§

Poison Injection Mechanism§

Attackers inject malicious text into publicly accessible web pages that are frequently crawled (e.g., Wikipedia talk pages, Reddit threads). The poisoned content is designed to trigger specific behaviors in the LM during inference, such as generating biased or harmful outputs when a particular trigger phrase appears.

HalfLife Analysis§

HalfLife models the survival probability of a piece of content through the web crawl and data curation pipeline. Let $p_{\text{crawl}}$ be the probability that a given web page is crawled, $p_{\text{extract}}$ be the probability that the content is extracted from the page, and $p_{\text{filter}}$ be the probability that it passes filters (e.g., deduplication, toxicity). The overall retention probability $R$ is:

$$R = p_{\text{crawl}} \cdot p_{\text{extract}} \cdot p_{\text{filter}}$$

The authors estimate these probabilities using empirical measurements from common web-scale datasets (e.g., C4, The Pile). They define the HalfLife $t_{1/2}$ as the number of days after which the probability of a poisoned sample remaining in the training data falls below 0.5. This helps determine the window of opportunity for an attack.

To validate, they simulate injection campaigns on a controlled web corpus and track the fraction of poisoned samples retained after each curation stage.

Experimental Setup§

• Pretrained model: GPT-2 (124M parameters) trained from scratch on a curated subset of C4.
• Attack trigger: A rare word sequence (e.g., "Samantha Wilson") is inserted into a fraction of training examples. During inference, the trigger causes the model to output a predetermined toxic completion.
• Evaluation: Measure attack success rate (ASR) on trigger inputs and perplexity on clean data.

Code Snippet§

The following Python code simulates a simplified HalfLife estimation for a given injection campaign:

import numpy as np

def half_life(p_crawl, p_extract, p_filter, t_days):
    """
    Compute retention probability after t_days.
    Assuming exponential decay with daily survival probability s.
    """
    s = p_crawl * p_extract * p_filter  # daily survival
    retention = s ** t_days
    return retention

# Example parameters (hypothetical)
p_crawl = 0.8   # 80% of pages are crawled each day
p_extract = 0.9 # 90% of crawled content is extracted
p_filter = 0.5  # 50% of extracted content passes filters

t_1_2 = np.log(0.5) / np.log(p_crawl * p_extract * p_filter)
print(f"HalfLife: {t_1_2:.1f} days")

Key Findings§

• Feasibility: Attackers can maintain poisoned content with a HalfLife of several days to weeks, depending on platform and curation aggressiveness.
• Impact: Models trained on poisoned data exhibit significantly higher ASR on trigger inputs (e.g., >80% ASR with 0.01% poisoned examples) while maintaining near-perplexity on clean data.
• Mitigation: The authors suggest adversarial filtering based on HalfLife estimates, but note that sophisticated attackers may adapt.

Conclusion§

This work demonstrates a practical poisoning vector for LLM pretraining and introduces a quantitative framework (HalfLife) for assessing data vulnerability. It underscores the need for robust data provenance and curation in large-scale LM training.