Online Neural Space Time Memory for Dynamic Novel View Synthesis

Core Methodology§

The paper addresses online novel view synthesis from multi-view streaming video by proposing a Neural Space Time Memory that decouples memory updates from memory application. Memory updates are performed periodically (e.g., every $K$ frames) to reduce computational cost, while memory read is done per-frame via cross-view attention. Two key mechanisms stabilize the memory: (1) a Memory Loss $\mathcal{L}_{mem}$ that enforces consistency between predicted and stored features, and (2) a Memory Caching strategy that regularizes active weights to prevent catastrophic forgetting.

Technical Details§

The core architecture consists of a feature extractor $\phi$ and a memory module $M$ storing a set of key-value pairs $(k_i, v_i)$ for each spatial position over time. At frame $t$, the memory is read using cross-view attention: $$ v_{out} = \text{CrossAttn}(Q_t, K_M, V_M) $$ where $Q_t$ is the query from the current frame features, and $(K_M, V_M)$ are the keys and values from the memory. The memory update occurs every $K$ frames by adding new key-value pairs from the current frame and optionally removing old ones (e.g., FIFO or importance-based).

The Memory Loss is computed as: $$ \mathcal{L}_{mem} = \| \phi(x_t) - v_{pred} \|^2 $$ where $\phi(x_t)$ is the feature of the ground truth view, and $v_{pred}$ is the memory output for that view.

Memory Caching maintains a separate set of cached weights $\theta_{cache}$ that are updated via exponential moving average of the active weights $\theta$: $$ \theta_{cache} \leftarrow \alpha \theta + (1-\alpha) \theta_{cache} $$ During training, a regularization term encourages the active weights not to deviate too far from the cache.

Implementation§

The model is implemented using a U-Net-like architecture with cross-attention layers. The feature extractor and memory read network are trained end-to-end. Key hyperparameters: memory size $N=1024$ keys per spatial location, update frequency $K=10$, and cache momentum $\alpha=0.99$.

Code Snippet (PyTorch-style)§

class NeuralSpaceTimeMemory(nn.Module):
    def __init__(self, feat_dim, mem_size=1024, update_freq=10):
        super().__init__()
        self.mem = MemoryModule(feat_dim, mem_size)
        self.cache = MemoryCache(momentum=0.99)
        self.update_freq = update_freq
        self.frame_count = 0

    def forward(self, x, views_available):
        # Extract features
        feat = self.feature_extractor(x)
        # Read memory via cross-attention
        mem_out = self.mem.read(feat)
        # Update memory periodically
        if self.frame_count % self.update_freq == 0:
            self.mem.update(feat)
            self.cache.update(self.mem.active_weights)
        # Loss computation (training only)
        loss_mem = F.mse_loss(feat, mem_out)
        # Regularize with cache
        loss_reg = F.mse_loss(self.mem.active_weights, self.cache.weights.detach())
        self.frame_count += 1
        return mem_out, loss_mem + lambda * loss_reg

Results and Evaluation§

The method achieves real-time performance (30+ FPS) on dynamic human motion datasets (e.g., ZJU-MoCap, NeRF-DS) while maintaining state-of-the-art PSNR (~28 dB) and LPIPS (~0.05). It demonstrates minute-scale online memorization, handling occlusions up to several seconds.