From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutch

Overview§

The paper introduces Grip on LLMs, an evaluation framework specifically designed for Dutch governmental use of large language models. Developed in collaboration with domain experts from a major Dutch municipality, the framework translates public-administration values into measurable benchmarks. The process involved an advisory board, user research, and a survey of civil-servant chatbot users, resulting in six evaluation dimensions: factuality, honesty, social bias, energy consumption, cost, and training data transparency.

Benchmark Design§

The benchmark suite covers more than 30 multilingual and Dutch-specific models, including general-purpose LLMs (e.g., GPT, Llama) and smaller Dutch-tuned models (e.g., BERTje, GPT-NL variants). Each dimension is operationalized as follows:

• Factuality: accuracy on Dutch question-answering and knowledge-grounded generation tasks.
• Honesty: calibration of the model when it chooses to answer or abstain, measured via expected calibration error (ECE) and selective prediction quality.
• Social bias: tests for stereotypes in Dutch, adapted from existing bias benchmarks (e.g., Winogender-style templates).
• Energy consumption: measured via GPU/CPU inference profiling (Wh per 1K tokens).
• Cost: financial cost per 1K tokens based on local or API deployment.
• Training data transparency: a rubric scoring public documentation of data sources and licensing.

Key Findings & Mathematics§

No single model excels across all dimensions; trade-offs are unavoidable. To formalize this, the authors compute Pareto frontiers over the six dimensions. A weighted composite score can be expressed as:

$$ S_i = \sum_{j=1}^{6} w_j \cdot \text{norm}_j(x_i) $$

where $w_j$ are stakeholder-defined weights and $\text{norm}_j$ normalizes each dimension to $[0,1]$.

The most striking result is that factuality and honesty are governed by distinct model properties. High factuality (correctness) does not imply high honesty (acknowledging ignorance). This is quantified by comparing answer accuracy against the model's self-reported uncertainty (e.g., using a thresholded max-probability abstention rule).

Bias appears largely independent of both quality and cost, meaning that expensive models are not automatically less biased.

Implementation Sketch§

A simplified Python illustration of how the framework evaluates factuality vs. honesty:

import numpy as np

def evaluate_factuality_honesty(model, dataset, abstain_threshold=0.6):
    """Return (factuality_score, honesty_auc)."""
    correct_list = []
    confidence_list = []
    abstain_correct = []
    
    for item in dataset:
        answer, logprob = model.generate_with_logprob(item.question)
        confidence = np.exp(logprob.mean())
        is_correct = (answer.strip() == item.answer.strip())
        
        correct_list.append(is_correct)
        confidence_list.append(confidence)
        
        # Honesty: abstain when confident below threshold
        if confidence < abstain_threshold:
            abstain_correct.append(True)  # abstaining is honest even if correct
        else:
            abstain_correct.append(is_correct)
    
    factuality = np.mean(correct_list)
    # e.g., area under the threshold-aggregated curve
    honesty = np.mean(abstain_correct)
    return factuality, honesty

In practice, the framework uses LLM-as-a-judge for open-ended answers, but the principles remain the same. All results are aggregated into a publicly accessible dashboard.

Impact§

The work is a strong example of value-driven LLM evaluation for non-English, public-sector deployments. By releasing a user-friendly overview, it supports a wide range of stakeholders—from engineers to policymakers—in making informed trade-offs when selecting models for governmental use.