A Durability and Cross-Language Transfer Benchmark for a Validated Teaching-Feedback Classification Protocol

Overview§

This paper evaluates the durability and cross-language transfer of a validated teaching-feedback classification protocol originally built with 2019-era frozen embeddings. The protocol classifies open-ended comments into thematic categories and sentiment. The authors re-run the protocol on the original Spanish corpus using three representation generations: sparse lexical features (TF-IDF), frozen transformer embeddings (e.g., BERT), and prompted large language models (LLMs). They also transfer the sentiment task to English using a balanced 45,000-comment corpus. The core finding is that the protocol is durable—the best model on the hardest Spanish task (thematic F1) is a 2026 frontier model, but it shows no sentiment advantage over cheaper models and no descriptive separation on English. Model choice is thus a deployment decision.

Methodology§

Protocol§

The original protocol uses a documented annotation guide, intra-annotator reliability, stratified cross-validation, and a held-out evaluation.
Thematic classification: multi-class (e.g., teaching quality, course content).
Sentiment: positive/negative/neutral.

Representations§

1. Sparse lexical features: TF-IDF unigrams and bigrams, with logistic regression. 2. Frozen transformer embeddings: BERT-base (uncased), frozen encoder, trained classifier head. 3. Prompted LLMs: Zero-shot and few-shot prompting of GPT-3.5 and a 2026 frontier model (anonymized).

Cross-language Transfer§

Spanish sentiment model fine-tuned on original corpus, tested on English.
English sentiment corpus: 45,000 comments, balanced, checked against an aspect-labeled education dataset.

Evaluation Metrics§

Thematic classification: macro F1-score.
Sentiment: macro F1 and accuracy.
Paired comparisons are descriptive (no hypothesis testing).

Key Equations§

Macro F1 is defined as: $$F1_{\text{macro}} = \frac{1}{C} \sum_{i=1}^{C} \frac{2 \cdot \text{precision}_i \cdot \text{recall}_i}{\text{precision}_i + \text{recall}_i}$$ where $C$ is the number of classes.

Implementation Details§

Sparse model: LogisticRegression from scikit-learn with default hyperparameters.
Frozen BERT: bert-base-uncased via HuggingFace Transformers; classifier head: 256-dim hidden layer, ReLU, dropout 0.3, softmax output.
LLM prompting: System prompt + instruction; for few-shot, 5 random examples per class.
Training: Cross-entropy loss, Adam optimizer, batch size 32, learning rate 2e-5 for BERT head.

Code Snippet§

import torch
from transformers import BertTokenizer, BertModel

class FrozenBERTClassifier(torch.nn.Module):
    def __init__(self, num_classes, freeze_encoder=True):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        if freeze_encoder:
            for param in self.bert.parameters():
                param.requires_grad = False
        self.classifier = torch.nn.Sequential(
            torch.nn.Linear(768, 256),
            torch.nn.ReLU(),
            torch.nn.Dropout(0.3),
            torch.nn.Linear(256, num_classes)
        )

    def forward(self, input_ids, attention_mask):
        with torch.no_grad():
            outputs = self.bert(input_ids, attention_mask=attention_mask)
        cls = outputs.last_hidden_state[:, 0, :]
        return self.classifier(cls)

Results§

On Spanish thematic classification, the frontier model achieves the highest macro F1 (0.79), vs. BERT (0.74) and TF-IDF (0.68).
On Spanish sentiment, all models perform similarly (F1 ~0.88), with no significant difference between the frontier model and cheaper alternatives.
On English sentiment, the frontier model shows no descriptive separation from BERT or TF-IDF (all F1 ~0.85).
Conclusion: The protocol is durable across representation generations and languages, but model choice does not affect sentiment performance; only thematic classification benefits from frontier models, and modestly.

Abstract

Technical Analysis & Implementation