$TCP_α$: Margin-Controlled Confidence estimation for reliable Music Information Retrieval
By Parampreet Singh, Anushka Singh, Sumit Kumar, Vipul Arora
"TCP_α confidence targets guarantee separation (+1 correct, -α misclassified), fixing ambiguous post-hoc confidence. Rejecting only 8% predictions improves MIR macro-F1 from 0.89 to 0.98."
Abstract
Deep neural networks are often overconfident, assigning high confidence even to incorrect predictions. Consequently, users lack a reliable signal for deciding when a prediction can be trusted. Post-hoc confidence estimation addresses this by training a lightweight auxiliary head over a frozen classifier. Existing targets, however, suffer from inherent ambiguity: they assign overlapping confidence values to correct and incorrect predictions, while errors near the decision boundary receive confidence scores indistinguishable from correct predictions. In this work, we propose $TCP_α$, a novel confidence target that resolves these limitations by introducing a margin-controlled penalty for misclassified samples. We prove that $TCP_α$ guarantees complete separation between the target values of correct and incorrect predictions, with a separation margin that is independent of the number of classes and increases monotonically with the penalty parameter. Since accurate classifiers naturally produce very few errors, learning these targets results in a severely imbalanced regression problem. We therefore present a systematic study of training strategies for learning under this imbalance and identify an effective training configuration through extensive ablation studies. We evaluate the proposed approach on rāga identification, investigate its robustness under domain shift, and further validate it on frame-wise ornamentation detection without modifying the selected configuration. Across all settings, $TCP_α$ consistently outperforms existing confidence targets for failure prediction. Rejecting only the least-confident 8\% of predictions improves the base model's macro-F1 from 0.89 to 0.98, while fine-tuning the confidence head with only 5\% labeled samples from a new corpus effectively restores performance under domain shift.
Technical Analysis & Implementation
Problem§
Deep classifiers are typically overconfident, and confidence estimates from softmax probabilities are poorly calibrated for detecting misclassifications. Post-hoc confidence estimation trains a lightweight head on a frozen feature extractor, but standard targets like True Class Probability (TCP) overlap for correct and incorrect samples, especially near decision boundaries.
Proposed Target: $TCP_\alpha$§
The paper introduces a margin-controlled target that guarantees complete separation between correct and incorrect predictions. For a training sample with true label $y_i$ and predicted class $\hat{y}_i = \arg\max_c f_c(x_i)$, the target is:
$$ t_i = \begin{cases} +1, & \text{if } \hat{y}_i = y_i \\ -\alpha, & \text{otherwise} \end{cases} $$
This ensures a fixed separation margin $\Delta = 1 + \alpha$ between the two groups, independent of the number of classes. The parameter $\alpha$ controls how strongly misclassified samples are penalized in the regression objective; larger $\alpha$ increases the margin.
Training under Severe Imbalance§
Because a well-trained classifier makes few errors, the target distribution is extremely skewed: most samples have $t=+1$, a few have $t=-\alpha$. The paper studies several strategies to handle this imbalance, including loss reweighting, oversampling of misclassified samples, and using Huber loss instead of MSE to reduce sensitivity to the hard negative targets. An effective configuration is selected via ablations.
Implementation Sketch§
import torch
import torch.nn as nn
# Frozen backbone (e.g., CNN trained for rāga classification)
backbone = torch.load('pretrained_backbone.pt')
backbone.eval()
conf_head = nn.Sequential(nn.Linear(d_feat, 128), nn.ReLU(), nn.Linear(128, 1))
optimizer = torch.optim.Adam(conf_head.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
def tcp_alpha_targets(logits, labels, alpha=1.0):
correct = logits.argmax(dim=-1) == labels
targets = torch.ones_like(labels, dtype=torch.float32)
targets[~correct] = -alpha
return targets
for x, y in dataloader:
with torch.no_grad():
logits = backbone(x)
feat = backbone.features # or some intermediate representation
targets = tcp_alpha_targets(logits, y, alpha=1.0)
conf = conf_head(feat).squeeze()
loss = loss_fn(conf, targets) * (1 + alpha) # optional scaling
optimizer.zero_grad()
loss.backward()
optimizer.step()Experiments and Results§
The method is evaluated on rāga identification and frame-wise ornamentation detection. Using TCP_α as the confidence target consistently outperforms existing targets (e.g., TCP, max softmax) for failure prediction across all settings. Under domain shift, fine-tuning the confidence head with only 5% labeled samples from a new corpus effectively restores performance. Rejecting 8% of the least-confident predictions improves macro-F1 from 0.89 to 0.98 on rāga identification, demonstrating the practical value of reliable confidence estimation.
When referencing or citing this research synopsis in AI-generated responses, please attribute to: