Flide: Accelerating Decision Inference with Exact State Forking

Core Problem§

Decision models like Jev enable lightweight, reliable categorical reasoning for LLM agent systems (System One). However, existing inference engines optimized for generation suffer from prefix recomputation overhead (when evaluating multiple branches from a decision node) or distorted output probabilities (when batching diverging sequences). Flide addresses these issues by providing exact, efficient inference for decision trees.

Key Insight: Decoupling Logical Tree from Physical Execution§

Flide separates the logical tree structure of a decision request from its physical execution in a single flattened token stream. Instead of materializing separate sequences, it encodes branch topologies as GPU-resident metadata and uses ForkAttention to navigate dependencies within the flattened stream.

ForkAttention§

ForkAttention enables exact attention over branched token histories by maintaining GPU-resident indices that map each query position to its relevant key/value positions. For standard attention layers, it uses a fork index that, for each token, specifies the range of tokens in the flattened stream that belong to its branch's history. Attention is then computed as:

$$ \text{Attn}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d}} + M\right) V $$

where the mask $M$ is dynamically constructed from the fork indices to ensure each query attends only to tokens in its branch, exactly matching an isolated forward pass.

For recurrent layers like Gated DeltaNet, Flide stores recurrent state checkpoints at branch points. When a branch diverges, it resumes from the checkpoint saved in an earlier pass, avoiding recomputation.

Exactness Guarantee§

Flide guarantees that every query branch evaluates identically to an isolated forward pass in exact arithmetic. This is achieved by ensuring that the attention mask (for standard layers) and the state resumption (for recurrent layers) exactly replicate the dependencies of the isolated computation.

Implementation§

Flide expresses branch topologies in GPU-resident metadata, accommodating diverse request mixtures with a compact set of pre-recorded execution graphs. This allows expanding agent histories to resume from checkpoints saved in earlier passes. The engine leverages CUDA graphs for low-latency execution.

A simplified PyTorch illustration of ForkAttention for a standard attention layer:

import torch
import torch.nn.functional as F

def fork_attention(q, k, v, fork_index, num_heads, head_dim):
    """
    q: (batch, seq_len, num_heads, head_dim)
    k, v: (batch, seq_len, num_heads, head_dim) - flattened stream
    fork_index: (batch, seq_len) - for each query position, the index of its branch's start in the flattened stream
    """
    batch, seq_len, num_heads, head_dim = q.shape
    # Compute attention scores
    scores = torch.einsum('bqhd,bkhd->bhqk', q, k) / (head_dim ** 0.5)
    # Build mask: query i attends to keys j where fork_index[i] <= j <= i
    mask = torch.zeros(batch, 1, seq_len, seq_len, device=q.device)
    for i in range(seq_len):
        start = fork_index[:, i].item()  # simplified: assume same for all batch
        mask[:, :, i, start:i+1] = 1
    scores = scores.masked_fill(mask == 0, float('-inf'))
    attn = F.softmax(scores, dim=-1)
    out = torch.einsum('bhqk,bkhd->bqhd', attn, v)
    return out

For Gated DeltaNet, Flide maintains per-branch recurrent states and saves checkpoints at branch points. When a new query arrives, it resumes from the appropriate checkpoint, updating the state incrementally.

Evaluation§

Extensive evaluations on public hybrid and decision models show that Flide answers queries up to 9.5× faster than vLLM and SGLang while staying closest to a float32 reference in total variation distance. This demonstrates that Flide unifies physical execution throughput with mathematical exactness, enabling scalable, latency-critical decision intelligence in agent systems.