What Just Happened§
> A new arXiv paper introduces a benchmark for multi-step tool-calling over Korean open public APIs, paired with a data-synthesis recipe that auto-generates instruction-tuning examples from API schemas. The goal: measure and improve LLMs' ability to chain calls to real, region-specific government endpoints.
I spent the morning reading this one because it touches on a problem I've hit repeatedly while building agent prototypes for non-English markets. Most tool-use benchmarks—ToolBench, API-Bank, even the internal suites used by OpenAI and Anthropic—assume REST endpoints with clean English parameter names and minimal authentication ceremony. Korean public APIs are the opposite: they require mandatory service keys, sometimes use SOAP/XML or well-formed POST bodies, and often return field names in Hangul with values like resultCode=INFO-000. The paper doesn't just throw tasks at a model; it also describes how to synthesize the data you need for fine-tuning. That recipe is arguably the more reusable half.
The authors call the benchmark something like "KoPublicAPI-ToolBench"—I'm flattening the Korean name for brevity—and it centers on tens of real public APIs from platforms like 공공데이터포털 (data.go.kr) and Seoul Open Data. Each task requires between two and five chained calls. For example, you might ask for the real-time air quality at a road address. The agent must geocode the address to coordinates, select the correct air-quality monitoring station near those coordinates, then call the station's data endpoint. That's three dependencies and two possible failure points. The paper's eval shows state-of-the-art models still drop required parameters or call the wrong endpoint in over 20% of tasks. That's a sobering stat for anyone shipping agents in production.
Why This Matters for AI Practitioners§
Let me make this concrete. When I fine-tune an open-weights model like Llama 3 for function calling, I usually convert public APIs into OpenAI-compatible tools and feed it trajectory data. But hand-crafting even 500 high-quality trajectories takes days. The recipe in this paper automates that by starting with API schemas and generating "scenario graphs"—abstract flows from endpoint A to endpoint B to endpoint C. Only then do they instantiate those graphs into user-assistant dialogues, randomizing parameter values, adding Korean politeness levels (~해요 vs ~해줘), and even injecting small typos to mimic real users.
I've tried a version of this with Claude and DeepSeek for my own domain APIs. You know what happens? The model often generates a trajectory where the second call uses variables that were never returned by the first. The graph-based approach prevents that by forcing data dependencies to be satisfied before a generation step commits. The paper formalizes what I was doing manually: you define a state machine where each tool's output feeds valid inputs to the next. That makes the synthetic data far less noisy than simple, generic "write a multi-turn tool use conversation" prompts.
What excites me is the potential to extend this recipe to other public data ecosystems. Italian open data APIs have their own quirks. Japanese APIs commonly require prefecture codes in Shift-JIS or UTF-8. Brazilian public APIs often have session token handshakes. The paper's methodology is language-agnostic, so a practitioner can adapt the code to generate regional tool-use data for any country. That flips the problem from "we need an English benchmark generalizable to the world" to "the world is a set of regional benchmarks, and we can synthesize data for each."
For agent pipelines, this also changes how you should evaluate. It's standard to measure tool-call accuracy on single function calls, but multi-step workflows are where real value lives: booking a flight, reserving a table, checking legal status. The failure modes (silent null, hallucinated response, or going in an endless loop) multiply with each step. The Korean API benchmark makes those frictions visible, and the synthesis recipe gives you a tool to train around them.
Who Is Affected§
First, developers of regional AI assistants and automation tools. If your product is targeting Korean users and you're using langchain or a custom agent loop to interact with Korean government data, this paper is literally addressing your pain. You probably already know the pain of dealing with Korean SOAP services whose schema definitions are inconsistent. The benchmark gives you a target: start by ensuring you can solve the benchmark's tasks, then expand with your own synthesized data.
Second, platform teams. If you're at a company that exposes private or partner APIs through an LLM tool layer, you need a way to generate evals when you add a new endpoint. Most teams hand-create a few happy-path examples and call it done. The data-synthesis recipe replaces that with a semi-automated pipeline that generates cases for every edge you'd rather forget: pagination, HTTP error codes, missing fields, and request parameters that depend on previous output. Anyone building AI gateways—like Kong, Portkey, or open-source alternatives—should care about this.
Third, fine-tuning providers such as Together AI, Fireworks, and anyone who serves custom LoRA adapters. If you offer model customization for vertical use cases, having a recipe to generate multi-step tool-use data means you can quickly build vertical adapters without a manual annotation team. For example, a Korean legal assistant needs to query court schedules, translate legal terms, and summarize in polite prose. A generic function-calling model won't nail that without regional data. The synthesis recipe makes creating such an adapter scaleable.
How to Use This Right Now§
You can adopt the synthesis recipe immediately, even though the official repo isn't public yet. Last week I built a miniature version for Korean open APIs using public schema scrapers and a few prompts. Here is a minimal pattern for generating a trajectory automatically.
First, define your API operations in a simple list of tool stubs. Then use Claude or DeepSeek as a generator to fill in the scenario graph. My prompt looks like this:
You are generating a training example for a multi-step tool-calling system.
Tools available:
1. addrtoCoord(road_addr: string) -> {lat, lon, status}
2. getAirQuality(lat: float, lon: float) -> {station_name, pm10, pm25, grade}
3. getKoreanHealthIndex(grade: string) -> {advice_ko, advice_en}
Create a user request in Korean that requires calling all three tools in a valid dependency order.
Then show the sequence of steps with function calls and returned results up until the final answer. Do not skip any state changes. Only generate a request that can be answered successfully if the tools are called in exact order.A generated example might look like this in JSONL for fine-tuning:
{
"instruction": "서울특별시 마포구 월드컵로 212 근처 공기가 어떤지 알려주세요.",
"trajectory": [
{"role": "user", "content": "서울특별시 마포구 월드컵로 212 근처 공기가 어떤지 알려주세요."},
{"role": "assistant", "function_call": {"name": "addrtoCoord", "arguments": "{\"road_addr\": \"서울특별시 마포구 월드컵로 212\"}"}},
{"role": "function", "name": "addrtoCoord", "content": "{\"lat\": 37.5604, \"lon\": 126.9082}"},
{"role": "assistant", "function_call": {"name": "getAirQuality", "arguments": "{\"lat\": 37.5604, \"lon\": 126.9082}"}},
{"role": "function", "name": "getAirQuality", "content": "{\"station_name\": \"망원동\", \"pm10\": 28, \"pm25\": 11, \"grade\": \"좋음\"}"},
{"role": "assistant", "function_call": {"name": "getKoreanHealthIndex", "arguments": "{\"grade\": \"좋음\"}"}},
{"role": "function", "name": "getKoreanHealthIndex", "content": "{\"advice_ko\": \"외출하기 좋아요\", \"advice_en\": \"Great for outdoor activities\"}"},
{"role": "assistant", "content": "마포구 월드컵로 212 근처의 공기질은 좋음입니다. 외출하기 좋아요."}
]
}If you want to evaluate an existing agent rather than fine-tune, convert each trajectory into a scripted test. Start with the user instruction, and let your agent decide the calls. Compare against the gold trajectory. This is exactly how the benchmark is structured.
My recommended workflow: (1) scrape or manually list 30–50 Korean API functions you care about; (2) generate at least 200 scenarios using an LLM with high probability settings; (3) filter out any trajectory where the LLM generator itself creates an invalid dependency—use a deterministic checker; (4) split into train and test sets. The Google-style best practice is to enrich the generated data with adversarial mutations. The paper also uses a few human-crafted experts-in-the-loop checkpoints to remove a subtle bias where synthetic data is too clean.
I was surprised that many public API endpoints return data inside a response or body wrapper that is not JSON-serializable. The recipe's solution is to normalize schemas into JSON Schema before generation. Always do that first; don't let the agent learn XML parsing when your framework will handle it. The key is to model the API as a function with typed inputs and outputs, not as an HTTP request.
Related Tools on LLMDB.APP§
This benchmark and synthesis recipe are most useful when paired with tooling that lets you iterate on agents. On LLMDB, you'll find several categories that directly support this workflow. First, evaluation and testing platforms like LangSmith or DeepEval let you store the benchmark cases and run agent trajectories daily. You can integrate the Korean API benchmark into those harnesses almost immediately.
Second, function-calling routers and proxies like Portkey or LiteLLM are helpful when actual API calls vary in protocol. They give you a consistent gateway, so the agent code knows nothing about SOAP vs REST.
Third, if you're planning to fine-tune an open-source model to handle this benchmark, consider using Unsloth or Axolotl, both listed on LLMDB. They can handle LoRA tuning on multi-turn function-calling data on a single consumer GPU, which is practical for a regional vertical.
Fourth, for the data-synthesis recipe itself, I rely on API schema parsers. Tools like OpenAPI-to-JSONSchema convert official Korean API docs into a machine-readable format, ready for scenario generation. Combine that with a strong LLM for expansion—I prefer Claude for nuanced Korean generation and DeepSeek when I want to volume-produce candidates cheaply.
A quick note on automation: rather than manual browsing, use Cursor with an open-source scraper to crawl the Korean public data portal. I built a simple script with Cursor that pulls all endpoint definitions marked 오픈API and dumps them into JSON files. It simplifies your data synthesis source. As for researching this paper or evolving your prompt patterns, Perplexity is handy to run targeted searches on recent Korean AI-agent work—just be careful about overreliance on its answers.
All in all, the paper is not just a benchmark; it's a call to stop treating tool-calling as language-agnostic. Regional public APIs are where multi-step agents will be tested in the real world. If you are building an AI assistant for Korean services, if you work on fine-tuning adapters for Asian languages, or if you architect agent platforms for enterprise partners—this is the paper to digest and steal ideas from today.
