Program-as-Weights: A Programming Paradigm for Fuzzy Functions

Technical Breakdown§

Core Idea§

Program-as-Weights (PAW) reframes a foundation model as a tool builder: a large compiler model (4B parameters) generates a small, reusable neural artifact (parameter-efficient adapters) from a natural-language specification. A frozen, lightweight interpreter (0.6B parameters) executes these adapters locally, achieving performance comparable to direct prompting of a 32B model at a fraction of the computational cost.

Methodology§

Overall Architecture

PAW consists of two stages:

1. Compiler: A 4B LLM (trained on FuzzyBench) takes a natural language function description $\text{spec}$ and outputs a set of low-rank adapter weights $\theta_{\text{adapter}}$ for the interpreter.
2. Interpreter: A frozen 0.6B Qwen3 model with LoRA adapters. At inference time, the interpreter loads the adapters and processes inputs through the base model with the adapter weights inserted.

Training the Compiler

The compiler is trained on FuzzyBench, a dataset of 10M (spec, program) pairs. Each program is a sequence of LoRA weight updates for the interpreter. The objective is to minimize the cross-entropy loss between predicted adapter weights and ground-truth adapter weights:

$$\mathcal{L}_{\text{compile}} = - \sum_{i} \log p(\theta_{\text{adapter}}^{(i)} | \text{spec}^{(i)}; \phi)$$

where $\phi$ are compiler parameters.

Inference Process

Given a new spec, the compiler generates adapter weights $\theta_{\text{adapter}} = \text{Compiler}(\text{spec})$. During function application, each input is forwarded through the interpreter:

$$y = f_{\text{interpreter}}(x; \theta_{\text{base}} + \theta_{\text{adapter}})$$

where $\theta_{\text{base}}$ are frozen interpreter weights and $\theta_{\text{adapter}}$ are injected via LoRA.

Implementation Details§

• Compiler: 4B variant of Qwen3, fine-tuned on FuzzyBench.
• Interpreter: 0.6B Qwen3 with LoRA adapters (rank $r=8$, scaling $\alpha=16$).
• Dataset: FuzzyBench contains 10M examples covering tasks like log alerting, JSON repair, and intent ranking.
• Hardware: Compiled adapters execute at 30 tokens/s on a MacBook M3 with ~2GB memory usage.

Code Snippet (PyTorch-like)§

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig, TaskType

class FuzzyCompiler(torch.nn.Module):
    def __init__(self, base_model_name="Qwen/Qwen2.5-4B"):
        super().__init__()
        self.model = AutoModelForCausalLM.from_pretrained(base_model_name)
        self.tokenizer = AutoTokenizer.from_pretrained(base_model_name)
    
    def forward(self, spec_text):
        inputs = self.tokenizer(spec_text, return_tensors="pt")
        outputs = self.model.generate(**inputs, max_new_tokens=512)
        adapter_weights = self.tokenizer.decode(outputs[0], skip_special_tokens=True)
        # Parse adapter_weights string into LoRA weight dict
        return parsed_adapter_weights

# Initialize interpreter with LoRA config
base_interpreter = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.6B")
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.1
)
interpreter = get_peft_model(base_interpreter, lora_config)

# At inference: load compiled adapters
compiler = FuzzyCompiler()
spec = "Rank search results by user intent"
adapter_weights = compiler(spec)
interpreter.load_state_dict(adapter_weights, strict=False)

# Apply function
input_text = "python programming tutorial"
output = interpreter.generate(input_text)

Key Results§

• Performance: PAW with 0.6B interpreter matches direct prompting of Qwen3-32B on FuzzyBench tasks.
• Efficiency: Inference uses ~1/50th the memory (2GB vs 100GB) and runs at 30 tokens/s on a laptop.
• Reproducibility: Artifacts are deterministic and offline, eliminating API dependency.

Significance§

PAW rethinks LLM deployment: instead of using a large model per input, a large model generates a small, reusable program once. This shifts cost from per-query to per-function, enabling local execution of complex NL-specified tasks.