Can Edge-Deployable Vision-Language Models Identify Species?

Core Question & Setup§

Camera traps at the edge lack connectivity, so the practically relevant class of models is the 2–8B edge-deployable VLM, not frontier-scale systems. This paper asks: do such VLMs carry genuine taxonomic knowledge, or do they merely pattern-match on clean web imagery?

Models Evaluated§

• Generalist VLMs: Qwen3-VL 2B / 4B / 8B, Gemma3 4B
• Domain specialist: BioCLIP (300M params)
• Task: 96-species identification
• Data: clean iNaturalist photos vs. camera-trap imagery from 6 LILA.science collections, on two independently sampled evaluation sets (for replication).

Key Metrics§

For each model $m$ and domain $d \in \{\text{clean}, \text{field}\}$, top-1 accuracy is measured. The domain gap is:

$$\Delta_m = \text{Acc}_m^{\text{clean}} - \text{Acc}_m^{\text{field}}$$

Across models, $\Delta_m \in [9.6, 26.6]$ pp, consistent across taxonomic levels and both eval sets.

Core Findings§

1. All models beat chance, but every model degrades on field imagery.
2. BioCLIP outperforms every VLM by 33.2–59.2 pp (200-image expanded sample, all models) — despite being 10–26× smaller. → The gap is training data specificity, not scale.
3. The degradation itself is generic: BioCLIP's gap (18.0 pp) is statistically indistinguishable from the best VLM's (22.3 pp). Thus the clean→field drop reflects image legibility / distribution shift, not a generalist weakness in fine-grained discrimination.
4. Open-set hallucination: 5.9–9.6% of responses are syntactically valid but taxonomically nonexistent names. The relative ranking of fabrication rates replicates exactly across both eval sets — a more robust signal than any point estimate.

Implementation Sketch§

Evaluation is a closed/open-set zero-shot classification: for each image $x$ and candidate class label $y_c$ (or free-form prompt for open-set), the VLM computes a similarity/score.

import torch
from transformers import AutoProcessor, AutoModelForVision2Seq

@torch.no_grad()
def eval_vlm(model_id, images, species_names, device="cuda"):
    proc = AutoProcessor.from_pretrained(model_id)
    model = AutoModelForVision2Seq.from_pretrained(model_id, torch_dtype=torch.bfloat16).to(device)
    prompts = [f"Which species is shown? Options: {', '.join(species_names)}." ] * len(images)
    inputs = proc(images=images, text=prompts, return_tensors="pt", padding=True).to(device)
    out = model.generate(**inputs, max_new_tokens=16, do_sample=False)
    return proc.batch_decode(out, skip_special_tokens=True)

def domain_gap(acc_clean, acc_field):
    return (acc_clean - acc_field) * 100.0  # percentage points

For BioCLIP, contrastive image-text embeddings give class scores directly:

$$p(y_c \mid x) = \frac{\exp(\langle f_I(x), f_T(t_c) \rangle / \tau)}{\sum_{c'}\exp(\langle f_I(x), f_T(t_{c'}) \rangle / \tau)}$$

Why It Matters for Practitioners§

• Deploy a specialist, not a generalist, for narrow edge tasks — a 300M BioCLIP dominates 8B Qwen3-VL on species ID.
• Domain gap is not a model-selection signal: it persists uniformly, so schedule field-adaptation / test-time normalization regardless of backbone.
• Rank-stable hallucination rates give a reliable cross-model comparison metric under open-set prompting.