Finetuning Strategies for Querying Sounds by Vocal Imitation

Overview§

This paper presents the winning solution to the AES AIMLA 2025 Challenge on querying sound effects by vocal imitation. The core challenge is learning a shared embedding space where both vocal imitations and actual sound effects map to semantically similar representations. The authors investigate two complementary fine-tuning strategies and show that combining them yields the best retrieval performance.

Method 1: Contrastive Learning with Frozen CED Encoder§

The first approach uses a pretrained Contrastive Encoder-Decoder (CED) model as a frozen feature extractor. The CED encoder, originally trained on large-scale audio-text pairs, provides strong general-purpose audio representations. A lightweight trainable projection head maps CED embeddings into a joint embedding space where vocal imitations and target sounds are pulled together. Training uses the standard InfoNCE contrastive loss:

$$ \mathcal{L}_{cl} = -\log \frac{\exp(\text{sim}(z_i, z_i^+)/\tau)}{\sum_{j=1}^{N} \exp(\text{sim}(z_i, z_j)/\tau)} $$

where $z_i$ and $z_i^+$ are projected embeddings of a vocal imitation and its corresponding sound effect, $\text{sim}$ is cosine similarity, and $\tau$ is a temperature hyperparameter. The frozen CED weights prevent catastrophic forgetting and leverage rich pretrained knowledge.

Method 2: Joint Contrastive-Triplet Learning with MobileNetV3§

The second approach trains a MobileNetV3 encoder from scratch (or fine-tuned) with a combined loss that adds a triplet loss with semi-hard negative mining to the contrastive loss. The triplet loss enforces a margin between positive and negative pairs:

$$ \mathcal{L}_{tri}= \max\left(0, d(a,p) - d(a,n) + \alpha\right) $$

where $d(\cdot,\cdot)$ is Euclidean distance, $a$ is an anchor (sound effect), $p$ is a positive (vocal imitation of that sound), $n$ is a semi-hard negative (a vocal imitation that is not too far from the anchor but not yet correctly ranked). Semi-hard negatives are chosen online from the current batch to stabilize training and improve discrimination. The total loss is a weighted sum: $\mathcal{L} = \mathcal{L}_{cl} + \lambda \mathcal{L}_{tri}$.

The MobileNetV3 backbone is computationally efficient and learns embeddings directly from mel-spectrograms, making the system suitable for real-time retrieval.

Implementation Details§

• Input representation: log-mel spectrograms (64 bands, 2-second segments, hop length 20ms).
• Data augmentation: random time-shift, pitch shift, and additive noise to improve robustness.
• Batch size: 256, embedded negatives are other samples in the batch.
• Temperature $\tau=0.05$, triplet margin $\alpha=0.2$, weighting $\lambda=0.5$.
• Optimizer: AdamW with cosine learning rate decay.
• Evaluation: retrieval recall@1, @5, @10 on a held-out challenge set.

The following PyTorch snippet illustrates the core training step for the joint contrastive-triplet approach:

import torch
import torch.nn as nn
import torch.nn.functional as F

class MobileNetV3Encoder(nn.Module):
    # Simplified: mobilenetv3 backbone + projection head
    def __init__(self, embed_dim=128):
        super().__init__()
        self.backbone = torch.hub.load('pytorch/vision', 'mobilenet_v3_large', pretrained=True)
        self.backbone.classifier = nn.Identity()
        self.proj = nn.Sequential(nn.Linear(960, 512), nn.ReLU(), nn.Linear(512, embed_dim))

    def forward(self, x):
        feat = self.backbone(x)
        return F.normalize(self.proj(feat), dim=-1)

def contrastive_triplet_loss(anchors, positives, negatives, tau=0.05, alpha=0.2, lam=0.5):
    # anchors: sound effects, positives: vocal imitations, negatives: semi-hard negatives
    # InfoNCE
    logits = torch.matmul(anchors, positives.T) / tau
    labels = torch.arange(anchors.size(0), device=anchors.device)
    loss_cl = F.cross_entropy(logits, labels)
    # Triplet loss
    d_ap = F.pairwise_distance(anchors, positives, p=2)
    d_an = F.pairwise_distance(anchors, negatives, p=2)
    loss_tri = F.relu(d_ap - d_an + alpha).mean()
    return loss_cl + lam * loss_tri

Key Findings§

The frozen CED contrastive method provides strong zero-shot-like generalization, while the MobileNetV3 contrastive-triplet method improves fine-grained discrimination. Combining both (ensemble) achieved the best challenge score. The report highlights the effectiveness of semi-hard negative mining and the importance of using a large-batch contrastive setup for audio retrieval.

Conclusion§

This technical report demonstrates a practical recipe for vocal-imitation sound retrieval: leverage a frozen pretrained audio encoder for general features and train a lightweight encoder with a multi-task objective for domain-specific discrimination. The insights are directly applicable to other audio-text retrieval tasks.