Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models
By Shravan Murlidaran, Miguel P. Eckstein
"Longitudinal study of VLM accuracy on complex social scenes (CSB dataset). MLLMs match top human descriptions and nearly eliminate most error types except spatial dependence."
Abstract
Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade. Most evaluations have used simple scenes (MS-COCO) that do not showcase complex human interactions or behaviors, only a handful of non-curated human descriptions as a benchmark, and have not focused on understanding the model's error types. Here, we introduce the Complex Social Behavior (CSB) dataset, containing 100 images depicting complex social interactions/behaviors. We analyze the progression of scene descriptions over a decade (2017-2025) of VLMs (four pre-Multimodal Large Language Models, MLLMs, and five MLLMs). We evaluate the accuracy of the models and 20 human descriptions relative to a gold standard on the CSB dataset and on a sample from MS-COCO. We analyzed five visual-cognitive error types: object detection, recognition, hallucination, scene understanding, and spatial dependence. The CSB dataset showed a more pronounced improvement than MS-COCO in scene description accuracy, with pre-MLLMs achieving much lower accuracy than the bottom-ranked human descriptions and MLLMs attaining accuracies similar to the top-ranked human descriptions. We show that MLLMs have eliminated the gap in scene description accuracy between simpler MS-COCO scenes and scenes depicting complex behaviors (CSB). MLLMs have almost eliminated all error types in our tested datasets, except for occasionally relying on different image regions for scene descriptions than humans do (spatial dependence error). We also show that detection, recognition, and hallucination errors have the highest impact on scene description accuracy. Together, our findings provide a more thorough evaluation of how visual language models have advanced over the last decade.
Technical Analysis & Implementation
Summary§
This paper evaluates the progression of vision-language models (VLMs) from 2017 to 2025 on scene description accuracy and visual-cognitive errors. The authors introduce the Complex Social Behavior (CSB) dataset containing 100 images with intricate human interactions. They compare four pre-MLLMs (e.g., LXMERT, ViLBERT) and five MLLMs (e.g., LLaVA, GPT-4V) against 20 human descriptions on both CSB and a subset of MS-COCO. Five error types are analyzed: object detection, recognition, hallucination, scene understanding, and spatial dependence.
Methodology§
Error Annotation§
Each model and human description is compared to a gold standard description using a fine-grained error classification scheme. Errors are categorized as:
- Object Detection: Missing or extra objects.
- Recognition: Incorrect identification.
- Hallucination: Describing nonexistent objects.
- Scene Understanding: Misinterpreting scene context.
- Spatial Dependence: Mismatched spatial relationships.
Accuracy is measured as the fraction of correctly described elements (e.g., objects, attributes, relations) relative to the gold standard.
Evaluation Metrics§
Accuracy $A$ is defined as: $$A = \frac{N_{\text{correct}}}{N_{\text{total}}}$$ where $N_{\text{correct}}$ is the number of correctly predicted elements and $N_{\text{total}}$ is the number of elements in the gold standard.
Model Inference§
MLLMs are prompted with the instruction: "Describe this image in detail." Responses are parsed and compared against the gold standard via human annotators.
Code Illustration§
Below is a simplified Python snippet for computing accuracy from model descriptions:
import json
def compute_accuracy(model_desc, gold_desc):
# Tokenize and extract key entities
model_entities = set(model_desc.split())
gold_entities = set(gold_desc.split())
correct = len(model_entities & gold_entities)
total = len(gold_entities)
return correct / total if total > 0 else 0.0
# Example usage
gold = "A man and woman are shaking hands in a conference room."
model = "Two people shaking hands in a room."
print(compute_accuracy(model, gold)) # Output: ~0.67Key Results§
- On CSB, pre-MLLMs accuracy ranged from 0.2–0.5, MLLMs from 0.7–0.9, while human bottom and top were 0.6 and 0.9 respectively.
- MLLMs eliminated the accuracy gap between MS-COCO and CSB.
- Detection, recognition, and hallucination errors were most impactful; spatial dependence errors persisted even in advanced MLLMs.
Conclusion§
The CSB dataset reveals that MLLMs have reached near-human performance on complex social scenes, but spatial reasoning remains a weakness.
Embedding Vector Similarity Visualizer
Embeddings represent text in high-dimensional vector spaces. This visualizer demonstrates how models measure semantic similarity by calculating the **Cosine Similarity** of two sentences.
Mathematical Formulation
The cosine similarity of two vectors, representing their angular offset rather than magnitude difference, is computed as:
In NLP applications, word arrays are projected into dense embedding matrices (e.g. 1536 dimensions). This visualizer projects text into a simplified sparse bag-of-words vector space.