Fine-Tuning Small Language Models (SLMs) for Low-Latency Function Calling

When I needed to build a lightweight agent that could call tools based on natural language commands, I immediately hit a wall: the big models were too slow for my use case. Calling GPT-4 or Claude added 2–3 seconds of latency per request, and that was before the function call executed. My goal was sub-300ms from utterance to tool invocation. That forced me down the path of fine-tuning a small language model (SLM) for function calling. Here’s exactly how I did it, what I learned, and where I almost gave up.

The Problem I Was Trying to Solve§

I was designing an in-app assistant for a fintech dashboard that lets users query account balances, transfer money, and pull transaction reports. The assistant doesn’t need to generate long-form narrative; it needs to map a sentence like “move $50 from checking to savings” to a structured JSON payload like {"action": "transfer", "from": "checking", "to": "savings", "amount": 50}. That’s a constrained generation task, but it’s also one that requires precise alignment between the user’s intent and the available function schema.

I started with a hosted LLM over a REST API, but the p95 latency was around 1.8 seconds. Worse, every failed parse or wrong parameter required a round trip. At the scale where I expected 10 million monthly requests, that latency meant a bad UX and a huge bill. I needed a model small enough to run on a single A10G or even a decent CPU workstation, yet accurate enough to handle the full breadth of function definitions without hallucinating extra parameters.

Tools and Setup§

My stack was intentionally boring: Python 3.11, PyTorch 2.4, transformers 4.44, and peft for LoRA. For the base model, I chose Qwen2.5-1.5B-Instruct. It’s a small, capable model that had decent chat performance out of the box. I considered Phi-3-mini, but Qwen’s tokenizer handled JSON more cleanly in my testing. I used **DeepSeek-R1 to generate synthetic training conversations from my function schemas; Claude was my evaluation judge; Cursor was the editor I wrote the fine-tuning scripts in; and Perplexity** was used to research common failure modes in function-calling datasets.

For hardware, I ran the initial fine-tune on a single RTX 4090 with 24GB VRAM. Later I moved to an AWS g5.xlarge for reproducibility. The entire training loop used a LoRA rank of 16, alpha 32, and a dropout of 0.05. I didn’t quantize until after training — I wanted to isolate the impact of quantization on accuracy before making deployment decisions.

Step-by-Step: What I Actually Did§

My first attempt was to fine-tune the model directly on a public function-calling dataset like gorilla-openfunctions-v2. That was a disaster — the model learned to parrot the training examples but failed to generalize to my unique function names. So I pivoted to building a custom dataset, step by step:

1. Wrote a JSON schema generator in Python that output a random set of 20–50 function definitions per sample, using my real fintech API as a starting point.
2. Generated synthetic user queries by feeding each schema to DeepSeek-R1 with a prompt like: “Create 5 natural language requests that would trigger these functions, including edge cases and paraphrases.”
3. Added negative samples — queries that should not trigger any function (e.g., “what’s the weather?”) so the model could learn a rejection output.
4. Formatting the training data using the OpenAI function-calling format, but adapted for SFT. Each prompt was a system message containing the schema, a user message with the query, and an assistant response containing the function call JSON or "none".

Here’s a minimal example of the training prompt format I used:

{
  "messages": [
    {"role": "system", "content": "You are an API assistant. Call the appropriate function from the schema. Return JSON only. Schema: {\"functions\": [{\"name\": \"transfer_funds\", \"description\": \"Transfer money between accounts\", \"parameters\": {\"type\": \"object\", \"properties\": {\"amount\": {\"type\": \"number\"}, \"from_acct\": {\"type\": \"string\"}, \"to_acct\": {\"type\": \"string\"}}, \"required\": [\"amount\", \"from_acct\", \"to_acct\"]}}]}"},
    {"role": "user", "content": "Transfer $120 from checking to my brokerage account"},
    {"role": "assistant", "content": "{\"name\": \"transfer_funds\", \"arguments\": {\"amount\": 120, \"from_acct\": \"checking\", \"to_acct\": \"brokerage\"}}"}
  ]
}

I trained for 3 epochs with a batch size of 16, a learning rate of 2e-4 with cosine decay, and a warmup ratio of 0.03. The LoRA adapters were applied to all linear layers (q, k, v, o, gate, up, down). The model reached a loss of 0.42, which looked promising, but loss curves can deceive you.

Code Samples / Prompts Used§

The most important part of the pipeline was the data generation logic. I’m sharing the exact function I used to generate synthetic use cases. It relies on DeepSeek-R1 through the OpenRouter API, but you can swap in any decent LLM.

import json
import random
from openai import OpenAI

client = OpenAI(base_url="https://openrouter.ai/api/v1", api_key="sk-")

synthetic_prompt = """
Given this function schema:
{schema}

Generate 5 distinct user queries that would invoke these functions. Include:
- At least one query with synonyms (e.g., 'send' instead of 'transfer')
- At least one query with missing parameters that the model should still map to the function
- At least one query that should NOT invoke any function (return 'none')
Return a JSON list of objects with keys 'query' and 'should_call' (boolean).
"""

def generate_samples(schema_json):
    resp = client.chat.completions.create(
        model="deepseek/deepseek-r1",
        messages=[{"role": "user", "content": synthetic_prompt.format(schema=json.dumps(schema_json))}],
        temperature=0.7
    )
    text = resp.choices[0].message.content
    # strip markdown fences if present
    text = text.replace("```json", "").replace("```", "")
    return json.loads(text)

For prompt evaluation, I used a structured template that included the schema in JSON and asked the model to output a function call object. Here’s an example of the evaluation prompt I later used with Claude to score answers:

Given the user request: "Move $50 from checking to savings, please."
And the function schema: ...
Evaluate the assistant's response.
Does it correctly pick the function and extract parameters? Ignore wording that doesn't affect semantics. Reply with 'CORRECT' or 'INCORRECT' and explain why.

I found that using this same prompt with Claude as a judge gave me a reliable automatic metric that matched human review about 95% of the time.

What Worked Well§

The biggest win was the fine-tuning approach combined with a strict output format. By teaching the model to emit only the function call JSON and nothing else, I eliminated the need for regex parsing and reduced errors to almost zero. The model learned to handle synonyms and paraphrases far better than I expected. For example, “send $100 to savings” triggered transfer_funds with the correct to_acct because the dataset included many paraphrases of the same action.

Another success was the use of negative samples. My initial dataset had 100% positive examples. After adding about 10% negative examples, the model’s false-positive rate dropped from 18% to 2%. The model learned a reliable way to output {"name": "none"} when it couldn’t map an intent to a function, which was crucial for a production assistant that should never randomly invoke a transaction.

I also found that LoRA training with rank 16 was more than enough. The model only needed about 120MB of adapter weights, and I could merge them back into the base model for deployment. On a NVIDIA T4 GPU, inference with FP16 took about 45ms per call, including tokenization. That was a huge improvement over the 1.8s of the hosted LLM. On an A10G, I measured 38ms. It unlocked my sub-300ms target even after adding a fallback retry logic.

What Failed and Why§

Before landing on the custom dataset, I tried two public datasets and both failed. The first was a popular Gorilla-style dataset that had a different schema format. My model was matching the function description rather than the actual schema, which caused it to select the right function but generate parameters with incorrect names. After five epochs, it overfit to the training schema and would produce JSON that didn’t align with my API. The lesson: templates must match your exact function definitions.

The second failure was more subtle: I initially tried to do function calling with a model that wasn’t instruction-tuned. I attempted to fine-tune Qwen2.5-1.5B-Base directly on the function-calling task. It never converged to sensible JSON; the loss got stuck around 1.2 and the model produced repetitive text. The problem was that function calling is a structured generation task that benefits from a chat format. Switching to the Instruct variant immediately fixed the convergence issue.

I also burned time trying to enforce JSON schema with constrained decoding libraries. I tried using transformers' grammar argument and various JSON generation libraries, but they conflicted with the model’s learned format. Once I added the phrase “Return JSON only” in the system prompt and fine-tuned on only those examples, the model naturally produced valid JSON about 99.5% of the time. Constrained decoding not only wasn’t necessary but actually reduced accuracy because it forced the model to start with { even for rejection examples.

Results and Takeaways§

The final microbenchmark on a held-out test set of 1,000 samples showed 96.8% accuracy for function invocation and parameter extraction. End-to-end latency, including network time in a Kubernetes cluster, was 210ms for the first token and 260ms for full JSON completion. Compared to a hosted GPT-4 solution, I cut latency by 87% and cost per request by 96%. The model is now running in production serving over 500,000 requests per day, and it’s holding up better than the larger model did.

I’ve also tested the model’s ability to adapt to new functions without retraining by adding them to the system prompt dynamically. The fine-tuned model handles seen functions perfectly, and it can occasionally handle new functions if their names and descriptions are close to patterns it has seen, but accuracy drops to around 71%. That’s a clear sign that dynamic schema expansion requires heavier fine-tuning or a larger base model.

The real takeaway: fine-tuning a small, local model is not a compromise—it’s an optimization. The model is smaller but it only needs to do one thing: map text to JSON-based function calls. That narrowness is what makes it fast and reliable. Curating a task-specific dataset is the core skill; everything else is just plumbing.

Key Takeaways§

• Dataset quality beats model size. Using schema-specific synthetic data and negatives gave me higher accuracy than any public generic dataset.
• Latency drops dramatically after fine-tuning. A 1.5B model is about 7x faster than a hosted LLM, making it suitable for real-time agent interactions.
• Structured output is learnable. With the right system prompt and enough examples, an SLM will generate valid JSON nearly always, making external constraints unnecessary.
• Test with the exact schema you deploy. If your functions change, expect a drop in accuracy; have a fallback path to a larger model for unseen functions.

Try It Yourself§

If you want to reproduce my results, here’s how to start: clone a small instruct-tuned model, write a script that generates synthetic function-calling pairs from your own API schema, and fine-tune with LoRA. I’ve linked my full training script in the LLMDB repo, but it’s only 150 lines and uses standard HuggingFace APIs. You can run it overnight on a single RTX 4090.

The most important advice is to spend a day building a dataset that reflects your real user phrasing. Ask three colleagues to type variations of what they would say to an assistant. Use an LLM to expand on their phrasing. Include edge cases and rejections. You’ll quickly surpass the capabilities of a generic function-calling model.

I’m also sharing the evaluation harness I used with Claude as a judge, so you can score your own fine-tuned models without manual annotation. The speed and cost benefits are enormous, and I believe every production agent should start with a small fine-tuned model, then escalate to a larger one only when the small model can’t handle the complexity. It’s the difference between an experiment and a deployable system.