MetaPerch: Learning from metadata for bioacoustics foundation models

Overview§

MetaPerch trains a bioacoustic foundation model using auxiliary metadata losses alongside species identification. By predicting location, time, and other environmental variables from audio embeddings, the model learns richer representations that generalize better to real-world passive acoustic monitoring (PAM) scenarios.

Methodology§

The core architecture is a convolutional audio encoder (e.g., ResNet-50) pretrained on spectrograms. The encoder outputs a fixed-dimensional embedding. Two types of heads are attached:

• Species head: standard linear classifier for species identification (cross-entropy loss $\mathcal{L}_{\text{species}}$).
• Metadata heads: independent heads for each metadata source (e.g., location binning, time of year, elevation, etc.). Each metadata head uses a task-specific loss: categorical cross-entropy for discrete bins, or mean squared error for continuous variables. The total loss is a weighted sum:

$$\mathcal{L} = \mathcal{L}_{\text{species}} + \sum_{i} \lambda_i \mathcal{L}_{\text{meta}}^{(i)}$$

Where $\lambda_i$ are hyperparameters controlling the contribution of each metadata loss. During training, gradients flow only through the encoder and the relevant heads. The model is trained on the Xeno-Canto dataset, which provides rich metadata for each recording.

Implementation Details§

• Audio preprocessing: Resample to 16 kHz, compute log-mel spectrograms (window 512, hop 256, mel bands 64).
• Encoder: Pre-trained on AudioSet (weight initialization).
• Metadata types: 9 sources, including latitude/longitude (binned into 1000 classes), month (12 classes), hour (24 classes), recording quality, background noise level, etc. Each metadata head is a single linear layer or small MLP.
• Training: Adam optimizer, batch size 256, learning rate 1e-4, weight decay. The $\lambda_i$ are set via grid search on a validation set.
• Evaluation: 17 challenging bioacoustic datasets (PAM field recordings). Metrics: top-1 accuracy and mean average precision.

Code Snippet§

import torch
import torch.nn as nn

class MetaPerch(nn.Module):
    def __init__(self, encoder, num_species, metadata_config):
        super().__init__()
        self.encoder = encoder  # e.g., ResNet50
        embed_dim = self.encoder.fc.in_features
        self.species_head = nn.Linear(embed_dim, num_species)
        self.metadata_heads = nn.ModuleDict()
        for name, (num_classes, loss_type) in metadata_config.items():
            if loss_type == 'ce':
                self.metadata_heads[name] = nn.Linear(embed_dim, num_classes)
            elif loss_type == 'mse':
                self.metadata_heads[name] = nn.Linear(embed_dim, 1)
    
    def forward(self, x, metadata_targets={}):
        features = self.encoder.features(x)  # [B, 2048, H, W]
        pooled = features.mean(dim=[2,3])    # [B, 2048]
        species_logits = self.species_head(pooled)
        metadata_logits = {}
        for name, head in self.metadata_heads.items():
            metadata_logits[name] = head(pooled)
        return species_logits, metadata_logits

Key Results§

• MetaPerch improves species ID accuracy by 5-15% on out-of-distribution PAM datasets compared to non-metadata baselines.
• Location metadata provides the largest single improvement (up to 8% absolute).
• Combining multiple metadata sources yields diminishing returns but overall best performance.
• Ablations show the model learns correlations between species occurrence and environmental variables, implicitly encoding ecological niches.