LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure

Overview§

LittleLearner is a 5B-parameter LLM trained from scratch on LITTLECURRICULUM, an 88B-token corpus composed of U.S. elementary school material that explicitly excludes concepts, facts, and vocabulary typically introduced above Grade 5. This controlled training environment creates well-defined knowledge and capability boundaries, enabling researchers to study how models acquire and use knowledge under a measurable curriculum.

Corpus Construction§

The corpus is curated via a multi-stage pipeline:

1. Source filtering – Restrict to documents from educational domains (e.g., learning sites, simplified Wikipedia).
2. Grade-level classification – A classifier assigns each document a reading grade level; only materials at or below Grade 5 are kept.
3. Concept/vocabulary blocking – A curated list of 10,000+ higher-grade concepts and terms is used to remove sentences/paragraphs that mention them.

This ensures that the training distribution contains no information explicitly taught in middle school or beyond, though implicit exposure via polysemy or contextual inference may still occur.

Model Architecture & Training§

LITTLELEARNER uses a standard decoder-only Transformer with a 5B-parameter count (32 layers, hidden size 4096, 32 attention heads). Training hyperparameters follow Chinchilla scaling for compute-optimality at 88B tokens.

The pretraining objective is standard next-token prediction. The training loss is:

$$ \mathcal{L}_{\text{PT}} = -\frac{1}{|\mathcal{D}|} \sum_{t=1}^{|\mathcal{D}|} \log p_\theta(x_t \mid x_{<t}) $$

A representative PyTorch training loop would look like:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2")  # replace with 5B config

train_dataset = load_curriculum_dataset()  # LITTLECURRICULUM
train_args = TrainingArguments(
    output_dir="./littlelearner",
    per_device_train_batch_size=32,
    learning_rate=3e-4,
    num_train_epochs=1,
    max_steps=1_000_000,
    weight_decay=0.1,
    warmup_steps=2000,
    fp16=True,
)

trainer = Trainer(model=model, args=train_args, train_dataset=train_dataset)
trainer.train()

Knowledge Injection Experiments§

A key contribution is using LITTLELEARNER as a sandbox to test how new knowledge can be injected beyond the fixed curriculum.

Post-Training

The authors fine-tune LITTLELEARNER on a small set of curriculum-aligned facts (e.g., additional Grade-5 science facts). They use standard supervised fine-tuning (SFT) with a fixed learning rate:

$$ \mathcal{L}_{\text{SFT}} = -\sum_{i=1}^{N} \log p_\theta(y_i \mid x_i) $$

In-Context Learning

They also evaluate few-shot prompting, where new facts are provided as context. Results show:

• Both methods improve the utilization of knowledge the model already has (e.g., better reasoning, fewer commonsense errors).
• Neither method enables the model to answer questions requiring out-of-scope knowledge specifically excluded from the corpus (e.g., algebraic concepts, historical events above Grade 5).

This suggests that post-training and ICL can reorganize and activate existing representations but cannot create fundamentally new knowledge if the underlying parameters lack it.

Implications§

The LITTLECURRICULUM/LITTLELEARNER suite provides a quantitative proxy for "what a model knows" and a controlled environment to separate knowledge acquisition from skill acquisition. Future directions include:

• Studying knowledge transfer across curriculum boundaries.
• Comparing continual learning methods under a known prior distribution.
• Investigating whether larger models can infer out-of-scope knowledge from in-scope clues.

This sandbox is a step toward principled, curriculum-aware LLM training and evaluation.