Will Scaling Improve Social Simulation with LLMs?

Technical Breakdown§

This paper investigates whether scaling compute (FLOPs) improves the fidelity of LLM-based social simulations. The authors train 85 transformer models (Qwen3 architecture) on the DCLM web text corpus under fixed compute budgets from $10^{18}$ to $10^{20}$ FLOPs, and evaluate 35 larger open-weight models up to 70B parameters. They then fit scaling laws to predict downstream accuracy from the model's cross-entropy loss.

Scaling Law Methodology§

The central assumption is that performance on social simulation tasks can be predicted from the pre-training loss $\mathcal{L}$ via a power law:

$$\text{Accuracy} = a - b \cdot \mathcal{L}^{\alpha}$$

where $a, b, \alpha$ are fitted parameters. This is similar to the scaling laws from Kaplan et al. (2020) but applied to downstream metrics rather than loss itself.

For loss prediction as a function of compute $C$ (FLOPs), they use the standard scaling law:

$$\mathcal{L}(C) = A \cdot C^{-\beta} + \mathcal{L}_{\infty}$$

where $\mathcal{L}_{\infty}$ is the irreducible loss.

Tasks and Datasets§

Three sub-domains are explored: 1. Opinion Modeling: Predicting opinion distributions from survey data (e.g., Pew Research). 2. Behavioral Simulation: Simulating human choices in economic games (e.g., risk aversion, correlated rewards). 3. Longitudinal Forecasting: Predicting future trends from time-series data.

Key Findings§

• Strong scaling for opinion modeling and behavioral simulation on well-represented populations (e.g., English-speaking). Accuracy improves with loss reduction.
• Weak scaling for longitudinal forecasting and underrepresented opinions; these tasks correlate less with general benchmarks (e.g., MMLU).
• No scaling for calibration to human cognitive biases (e.g., risk aversion) and heuristics (e.g., learning correlated rewards). Fine-tuning from 0.5B to 8B parameters yields no improvement.

Code Snippet: Fitting Scaling Law§

The following PyTorch code illustrates how the scaling law parameters could be fitted using least squares:

import torch

def fit_scaling_loss(compute, loss):
    # compute: array of FLOPs
    # loss: array of cross-entropy losses
    log_C = torch.log(torch.tensor(compute))
    log_L = torch.log(torch.tensor(loss))
    # Linear fit on log-log scale: log L = log A - beta * log C + log L_inf (approx)
    # Use simple linear regression ignoring irreducible loss for simplicity
    A = torch.exp(torch.mean(log_L) + beta * torch.mean(log_C))
    beta = - (torch.sum((log_C - torch.mean(log_C)) * (log_L - torch.mean(log_L)))) / torch.sum((log_C - torch.mean(log_C))**2)
    return A, beta

def predict_accuracy(loss, a, b, alpha):
    return a - b * (loss ** alpha)

Implications§

The results suggest that scaling alone will improve social simulations for most tasks, but outliers exist—particularly for tasks requiring human-like biases or low-resource domains. This motivates further research into specialized training or fine-tuning for social fidelity.