Combating Textual Noise and Redundancy: Entropy-Aware Dense Visual Token Pruning

Technical Breakdown§

Problem & Motivation§

Vision-Language Models (VLMs) process many visual tokens from images, leading to high computational cost. Prior token pruning methods use cross-modal attention scores to select top-K tokens, but these scores are corrupted by textual noise (e.g., irrelevant words) and selected tokens often exhibit redundancy due to feature fragmentation. This paper addresses these issues with Entropy-Aware Dense Pruning (EADP).

Core Methodology§

1. Textual Noise Filtering via Entropy Given a text instruction $T$ with $N$ tokens, compute the entropy of each token's attention distribution over initial visual tokens $V$: $$ H(t_i) = -\sum_{j} p_{ij} \log p_{ij}, \quad p_{ij} = \frac{\exp(q_i \cdot k_j)}{\sum_k \exp(q_i \cdot k_k)} $$ Tokens with high entropy (uniform attention) are considered noisy and filtered out. Retained tokens form a clean instruction set $T'$.

2. Dense Cross-Modal Scoring Recompute relevance scores $s_j$ for each visual token $v_j$ as the average attention from clean text tokens: $$ s_j = \frac{1}{|T'|} \sum_{t_i \in T'} a_{ij}, \quad a_{ij} = \text{softmax}(q_i \cdot k_j / \tau) $$ These scores are more robust to textual noise.

3. Structured Token Selection via Submodular Maximization Instead of Top-K selection, EADP casts selection as: $$ \max_{S \subseteq V, |S|=K} F(S) = \underbrace{\sum_{v_j \in S} s_j}_{\text{relevance}} + \lambda \underbrace{\sum_{v_j \in S} \sum_{v_k \in V, k \neq j} \phi(d_{jk})}_{\text{diversity penalty}} $$ where $d_{jk}$ is spatial distance (Euclidean coordinates) and $\phi(\cdot)$ is a decreasing function (e.g., Gaussian). This is a submodular function with a cardinality constraint, solved via a greedy algorithm that iteratively adds the token with highest marginal gain: $$ v^* = \arg\max_{v_j \in V \setminus S} \left( s_j + \lambda \sum_{v_k \in S} \phi(d_{jk}) \right) $$

Implementation Details§

• Architecture: Inserted after a few transformer layers in LLaVA-like VLMs.
• Entropy threshold: Remove tokens with $H(t_i) > \tau_H$ (e.g., 90th percentile).
• Diversity weight $\lambda$ tuned per benchmark.
• Code snippet (PyTorch-like):

def eadp_prune(attn_scores, text_feats, visual_coords, budget, entropy_thresh, lambda_reg):
    # attn_scores: (L_text, L_visual)
    text_entropy = -torch.sum(attn_scores * torch.log(attn_scores + 1e-8), dim=1)
    clean_idx = text_entropy < entropy_thresh
    clean_scores = attn_scores[clean_idx].mean(dim=0)  # (L_visual,)
    
    selected = []
    candidates = set(range(len(clean_scores)))
    for _ in range(budget):
        best_token = None
        best_gain = -float('inf')
        for j in candidates:
            marginal_reward = clean_scores[j]
            diversity = 0
            for k in selected:
                dist = torch.norm(visual_coords[j] - visual_coords[k])
                diversity += torch.exp(-dist)  # Gaussian kernel
            gain = marginal_reward + lambda_reg * diversity
            if gain > best_gain:
                best_gain = gain
                best_token = j
        selected.append(best_token)
        candidates.remove(best_token)
    return selected

Results & Significance§

EADP achieves state-of-the-art accuracy-efficiency trade-offs on VQA and captioning benchmarks (e.g., COCO, VizWiz, TextVQA) under 20-50% token budgets. It consistently outperforms top-K and prior adaptive pruning methods, especially under dense instructions (e.g., scene text queries) and fine-grained tasks.