Evolving language compositionality in a frequency-structured meaning space

Overview§

This paper extends the classic iterated learning model of language evolution to study how a frequency-structured meaning space impacts the emergence of compositionality. In iterated learning, a language is repeatedly transmitted from one agent to another through a limited training bottleneck. Prior work showed that this bottleneck pressures the language to become compositional, so a learner can generalize from a small sample to the full meaning space. The authors introduce a non-uniform distribution over meanings and ask two questions: (1) Do high-frequency meanings experience a different compositional pressure? (2) What happens when frequency is defined over parts of meanings (e.g., individual features) rather than over whole meaning vectors?

Methodology§

The model uses a standard sender–receiver architecture implemented as deep neural networks, trained over successive generations. Each generation consists of:

1. A sender that maps a meaning vector $m \in \mathbb{R}^d$ to a discrete signal (a sequence of symbols).
2. A receiver that maps a signal back to a meaning vector $\hat{m}$.

Training uses an autoencoding-style loss:

$$ \mathcal{L} = \mathbb{E}_{(m, s) \sim \mathcal{D}'} \left[ \| m - \hat{m} \|^2 \right] $$

where $\mathcal{D}'$ is a small subset of all possible meanings, sampled according to a frequency distribution $p(m)$. This subset forms the transmission bottleneck: the next generation only sees these $K$ examples, forcing the learner to generalize.

Two frequency conditions were tested:

• Whole-meaning frequency: $p(m)$ assigns high probability to a few complete meaning vectors (e.g., certain color+shape combinations).
• Part-frequency: $p(m)$ is defined by independent probabilities over component features (e.g., one color and one shape appear often, but their joint combination is rare).

The networks were trained with standard backpropagation and a discrete communication channel (Gumbel-Softmax / straight-through estimator). Below is a simplified PyTorch sketch of the core architecture:

import torch
import torch.nn as nn

class Sender(nn.Module):
    def __init__(self, input_dim, hidden_dim, vocab_size, max_len):
        super().__init__()
        self.embed = nn.Linear(input_dim, hidden_dim)
        self.lstm = nn.LSTM(hidden_dim, hidden_dim, batch_first=True)
        self.out = nn.Linear(hidden_dim, vocab_size)

    def forward(self, m):
        h = torch.relu(self.embed(m)).unsqueeze(1)
        # Pass through LSTM with a start token; produce logits over vocab for each position
        logits = []
        x = torch.zeros(m.size(0), 1, self.embed.out_features)
        for t in range(max_len):
            out, (h, c) = self.lstm(x, (h, torch.zeros_like(h)))
            logits.append(self.out(out))
            x = torch.softmax(logits[-1], dim=-1)  # differentiable approximation
        return torch.stack(logits, dim=1)

class Receiver(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, output_dim):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.proj = nn.Linear(hidden_dim, output_dim)

    def forward(self, s):
        emb = self.embed(s)
        _, (h, _) = self.lstm(emb)
        return self.proj(h[-1])

Key Findings§

The results reveal a clear asymmetry:

• High-frequency whole meanings are learned reliably and often become irregular — they are stored as holistic mappings and escape the compositional grammar that governs lower-frequency meanings. This mirrors natural-language phenomena such as irregular verbs or high-frequency irregular plural forms.
• Low-frequency meanings force the learner to decompose the meaning into features and reuse compositional rules, because they occur too rarely for rote memorization.
• Crucially, when frequency is applied to parts (features) rather than whole vectors, the language fails to transmit stably. Although the most frequent components are individually learnable, the mapping between combinations of features and signals is not supported by the frequency distribution. The relational structure required for compositional generalization never appears, so the language degenerates across generations.

The authors argue that frequency can shape emergent linguistic structure only when the frequency distribution is defined over form–meaning units that a learner can acquire holistically. If frequency is spread over sub‑meaning components, the learner cannot exploit it to build the compositional scaffolding needed for transmission.

Implications§

This work deepens our understanding of when and why compositionality emerges in evolving languages. It connects computational iterated learning with empirical observations about irregularity in high-frequency forms. From a practical standpoint, it suggests that the design of training distributions in emergent-communication or language-evolution simulations must align with the granularity of the units being transmitted. The results also have implications for the evolution of human languages, supporting the idea that cognitive bottlenecks and skewed meaning frequencies jointly shape grammar and irregularity.