The Theoretical Foundation of Socratic Tests: Dynamic, Multimodal, Conversational Examinations

Overview§

This paper introduces the Socratic Test, a computer-mediated conversational assessment combining Dynamic Assessment, Bloom's Taxonomy, SOLO Taxonomy, and multimodal workspaces. The core innovation is a formal model for graduated scaffolding that quantifies a student's Zone of Proximal Development (ZPD) and a non-compensatory additive grading architecture that rewards mastery rather than penalizing errors.

Methodology§

Dynamic Assessment & Scaffolding§

The test presents problems of increasing difficulty. When a student struggles, the system provides a series of graduated prompts (e.g., hints, sub-questions, worked examples). Each scaffold level $s \in \{0,1,\dots,S\}$ has a predetermined cognitive cost. The student's ZPD score is computed as:

$$ Z = \frac{\sum_{i=1}^{N} w_i \cdot (S - s_i)}{N \cdot S} $$

where $s_i$ is the minimum scaffolding level required by student $i$, $w_i$ is problem weight, and $S$ is the maximum scaffold tier. This yields a continuous measure of independent performance versus assisted performance.

Bloom's Taxonomy for Proctoring§

The system uses Bloom's verb categories (e.g., remember, apply, evaluate) to dynamically adjust the next prompt. An LLM-based proctor classifies the student's response into Bloom levels and selects the next response from a pre-validated item bank. The transition probability is modeled as:

$$ P(L_{t+1} \mid L_t, R_t) = \text{softmax}(W \cdot [\text{enc}(L_t); \text{enc}(R_t)]) $$

where $L_t$ is the current Bloom level, $R_t$ is the student's response embedding, and $W$ is a learned projection.

SOLO Taxonomy for Structural Evaluation§

Student answers are mapped to the Structure of Observed Learning Outcomes (SOLO) levels: prestructural, unistructural, multistructural, relational, extended abstract. The mapping is learned via a classifier on the response text and multimodal artifacts (e.g., diagrams, equations). The final grade excludes compensatory averaging; instead, each SOLO level earns points only if the previous level is fully mastered:

$$ \text{Score} = \sum_{k=1}^{5} \mathbb{1}[\text{level}_k \text{ achieved}] \cdot p_k $$

where $p_k$ are fixed point values with $p_1 < p_2 < \dots < p_5$.

Implementation Sketch§

The following PyTorch pseudo-code illustrates the dynamic scoring loop:

import torch
import torch.nn as nn

class SocraticScaffolder(nn.Module):
    def __init__(self, s_max=5, num_bloom=6, num_solo=5):
        super().__init__()
        self.bloom_encoder = nn.Linear(768, 64)
        self.solo_head = nn.Linear(768, num_solo)
        self.scaffold_policy = nn.Linear(64 + 768, s_max)

    def forward(self, response_emb, bloom_state):
        bloom_emb = self.bloom_encoder(bloom_state)
        feat = torch.cat([response_emb, bloom_emb], dim=-1)
        solo_logits = self.solo_head(response_emb)
        scaffold_probs = torch.softmax(self.scaffold_policy(feat), dim=-1)
        return solo_logits, scaffold_probs

def grade(solo_levels, points=[0, 1, 2, 4, 6]):
    # Non-compensatory: only award if all previous levels are achieved
    total = 0
    for i, level in enumerate(solo_levels):
        if level == 1 and all(solo_levels[:i] == 1):
            total += points[i]
    return total

Implications§

The framework promises higher measurement reliability by removing human examiner bias and reducing anxiety. The additive structure ensures that students are never penalized for incorrect answers—only rewarded for demonstrated mastery—aligning with modern learning science. The theoretical model also outlines a path toward fully automated, adaptive oral exams that are both fair and diagnostically rich.