Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

Technical Breakdown§

Problem: Modality Interference in Full-Duplex SLMs§

Full-duplex spoken language models must simultaneously process and generate acoustic and semantic information. Prior unified architectures force both modalities to share deep parameter spaces, leading to gradient conflicts. The paper formalizes this: let $\mathcal{L}_a$ and $\mathcal{L}_s$ be the acoustic and semantic losses. The shared parameters $\theta$ receive conflicting gradients:

$$\nabla_\theta \mathcal{L}_a \cdot \nabla_\theta \mathcal{L}_s < 0$$

This interference degrades both tasks, making the model sound unnatural.

Proposed Solution: Lychee-FD§

Lychee-FD uses a hierarchical parameter separation strategy:

• Shallow layers: Shared to capture low-level cross-modal features.
• Deep layers: Split into two separate branches — an acoustic branch and a semantic branch — to avoid gradient conflict.
• Semantic Alignment Channel (SAC): A lightweight cross-attention mechanism that operates between the two branches, ensuring semantic coherence without reintroducing interference.

Formally, let $H_{\text{shared}}$ be the output of the last shared layer. The acoustic branch processes $H_{\text{shared}}$ via $f_a$, the semantic branch via $f_s$. The SAC computes:

$$\text{SAC}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$

where $Q$ comes from the acoustic branch, $K,V$ from the semantic branch. The aligned output is added back to both branches.

Training Objective§

The model is trained end-to-end with a weighted sum of acoustic loss (e.g., L1 on spectrogram) and semantic loss (e.g., cross-entropy on text tokens):

$$\mathcal{L} = \lambda \mathcal{L}_a + (1-\lambda) \mathcal{L}_s$$

Gradient analysis shows that after separation, the conflict is minimized because the branches now have independent parameters.

PyTorch Code Snippet§

import torch
import torch.nn as nn

class LycheeFD(nn.Module):
    def __init__(self, d_model, num_shallow, num_deep):
        super().__init__()
        self.shallow = nn.TransformerEncoder(...)  # shared
        self.acoustic_branch = nn.TransformerEncoder(...)  # separate
        self.semantic_branch = nn.TransformerEncoder(...)  # separate
        self.sac = nn.MultiheadAttention(d_model, num_heads=8)
        
    def forward(self, x):
        h = self.shallow(x)
        h_a = self.acoustic_branch(h)
        h_s = self.semantic_branch(h)
        # Semantic Alignment Channel
        aligned, _ = self.sac(h_a, h_s, h_s)
        h_a = h_a + aligned
        h_s = h_s + aligned
        return h_a, h_s

Results§

• Spoken QA: +7.4% accuracy over baselines.
• FullDuplexBench 1.5: +28.5% fluency score.
• Inference latency unchanged due to hierarchical design.