MulTTiPop: A Multitrack Transcription Dataset for Pop Music

Abstract

We present MulTTiPop, a dataset of pop music segments and their associated multitrack MIDI recordings for the evaluation of automatic music transcription models. MulTTiPop contains 572 segments of popular music totaling 3.5 hours of audio, and contains songs from diverse genres and decades from the 1930s to 2000s. To collect this dataset, we perform metadata-based matching on song segments from the Lakh MIDI and TheoryTab datasets, manually identify an anchor beat between the audio and MIDI, then use beat tracking on the audio and warp the MIDI to match its tempo and timing. We evaluate state-of-the-art automatic music transcription models on MulTTiPop and find substantial room for improvement, with the best model achieving 38% Onset F1. More details and sound examples of MulTTiPop are available at https://gclef-cmu.org/multtipop.

Technical Analysis & Implementation

MulTTiPop: A Multitrack Transcription Dataset for Pop Music§

Overview§

MulTTiPop addresses the lack of high-quality multitrack (instrument-separated) MIDI data for popular music. It contains 572 segments (total 3.5 hours) from diverse genres and decades (1930s–2000s). The dataset is designed to evaluate automatic music transcription (AMT) models in a realistic, multi-instrument setting.

Dataset Construction§

1. Source Matching: Song segments from Lakh MIDI (MIDI files) and TheoryTab (chord annotations) are matched using metadata (artist, title, key, tempo). 2. Anchor Beat Identification: A human annotator manually identifies a single anchor beat (e.g., a downbeat) common to both the audio and MIDI. 3. Beat Tracking & MIDI Warping: Beat tracking is performed on the audio (e.g., using madmom). The MIDI file is then warped temporally to match the audio's beat times via dynamic time warping (DTW) or a piecewise linear mapping. \[ t_{\text{MIDI}}' = f(t_{\text{MIDI}}) = \text{interpolate}(\text{audio beats}, \text{MIDI beats}) \] where each MIDI event time $t_{\text{MIDI}}$ is mapped to a new time $t_{\text{MIDI}}'$ based on the ratio of audio beat intervals to MIDI beat intervals. 4. Quality Control: Segments with high alignment error are discarded.

Dataset Characteristics§

Content: 572 segments, 3.5 hours audio, 10+ instruments per track on average.
Format: Audio (44.1 kHz, stereo) + multitrack MIDI (note onsets, pitches, velocities, instrument labels).
Genres: Pop, rock, R&B, jazz, etc.

Evaluation of AMT Models§

The authors evaluate state-of-the-art AMT models (Onsets and Frames, MT3, etc.) on MulTTiPop. Metrics include frame-level and onset-level F1 scores. The best model achieves only 38% Onset F1, showing large room for improvement.

Code Example (Loading Dataset & Computing Onset F1)§

import torch
from multipop import MulTTiPop

# Load dataset
dataset = MulTTiPop(root="./data", split="test")
sample = dataset[0]
audio = sample["audio"]  # (channels, time)
target_onsets = sample["midi_onsets"]  # (num_instruments, time, 128)

# Example: evaluate a model
model = torch.load("best_model.pt")
with torch.no_grad():
    pred_onsets = model(audio.unsqueeze(0))  # (1, inst, T, 128)

# Compute onset F1 (thresholded)
def onset_f1(pred, target, threshold=0.5):
    pred = (pred > threshold).float()
    tp = (pred * target).sum()
    fp = pred.sum() - tp
    fn = target.sum() - tp
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp + fn + 1e-8)
    return 2 * precision * recall / (precision + recall + 1e-8)

f1 = onset_f1(pred_onsets, target_onsets)
print(f"Onset F1: {f1:.3f}")

Conclusion§

MulTTiPop provides a challenging benchmark for multitrack AMT, bridging the gap between synthetic datasets and real-world pop music. Its construction methodology ensures high-quality alignments, and evaluation results indicate significant headroom for future research.

Abstract

Technical Analysis & Implementation