Challenges in Evaluating Explanation Methods for Static and Evolving Data

Overview§

This position paper addresses systemic weaknesses in the evaluation of Explainable AI (XAI) methods, particularly when applied to static image classifiers and evolving data streams. Using the DetoxAI system as a case study—a vision-based tool for detecting bias and enabling concept unlearning—the authors illustrate how standard evaluation metrics (e.g., faithfulness, stability) often fail to capture real-world utility. The paper then presents a human-grounded evaluation framework for image classification explanations and investigates the extension of counterfactual explanations to non-stationary environments where concept drift occurs.

DetoxAI: Bias Detection via Explanations§

DetoxAI combines a pre-trained CNN (e.g., ResNet) with post-hoc explanation methods (Grad-CAM, LIME) to identify spurious correlations (e.g., background features) that lead to biased predictions. The system also supports concept unlearning: given a set of concepts (e.g., 'texture' or 'color') identified via concept activation vectors (CAVs), the model is fine-tuned to minimize dependence on those concepts. Formally, the loss becomes:

$$ \mathcal{L}_{total} = \mathcal{L}_{CE}(f(x), y) + \lambda \sum_{c \in C} \| \nabla_{\theta} \text{CAV}_c(x) \|_F^2 $$

where $\text{CAV}_c(x)$ is the alignment of the model's intermediate representations with concept $c$.

Human-Grounded Evaluation of Explanation Methods§

A key contribution is a protocol for evaluating explanations through user studies. Instead of relying solely on proxy metrics, the authors propose tasks such as simulatability (can a user predict the model's output given the explanation?) and counterfactual simulation (can a user infer how the output changes with input perturbations?). For image classification, they use a controlled set of images with known ground-truth attributes (e.g., shape, color, background) and compare explanations against human-annotated salient regions. The evaluation is statistically grounded using inter-annotator agreement (Cohen's $\kappa$) and task completion time.

Counterfactuals for Evolving Data§

A major challenge is adapting counterfactual explanations to streams where the underlying data distribution changes (concept drift). Given a counterfactual $x_{cf} = x + \delta$ generated at time $t$, the same explanation may become invalid at $t+1$ when the decision boundary shifts. The authors propose a drift-aware counterfactual generator that re-optimizes the perturbation under a new model $f_{t+1}$ while minimizing distance to the original counterfactual:

$$ \min_{\delta'} \| x_{cf}^{t} - (x + \delta') \|_2^2 \quad \text{s.t.} \quad f_{t+1}(x + \delta') \neq f_{t+1}(x) $$

This is solved iteratively using gradient-based updates, reusing previous counterfactuals as warm starts to reduce computational cost.

Implementation Sketch§

import torch
import torch.nn.functional as F

class DriftAwareCounterfactual:
    def __init__(self, model, lr=0.01, lambda_drift=0.5):
        self.model = model
        self.lr = lr
        self.lambda_drift = lambda_drift

    def generate(self, x, y_target, x_cf_prev=None, max_iter=200):
        x_cf = x_cf_prev.clone().requires_grad_(True) if x_cf_prev is not None \
               else x.clone().requires_grad_(True)
        opt = torch.optim.Adam([x_cf], lr=self.lr)
        for _ in range(max_iter):
            opt.zero_grad()
            logits = self.model(x_cf)
            # Cross-entropy toward target class
            loss_ce = F.cross_entropy(logits, y_target)
            # Drift penalty: stay close to previous counterfactual
            loss_drift = self.lambda_drift * F.mse_loss(x_cf, x_cf_prev) if x_cf_prev is not None else 0
            loss = loss_ce - loss_drift  # maximize CE for counterfactual
            loss.backward()
            opt.step()
        return x_cf.detach()

The code snippet highlights a simplified version of the drift-aware counterfactual generator, where the drift penalty encourages temporal consistency.

Conclusion and Open Challenges§

The paper concludes by discussing the co-evolution of data, models, and explanations, arguing that evaluation protocols must be continuously updated. Key open questions include developing online, human-in-the-loop metrics and handling the combinatorial explosion of concept drift in high-dimensional spaces. The work serves as a roadmap for more rigorous XAI evaluation in dynamic settings.