The Problem I Was Trying to Solve§

I've spent years building IoT systems for smart buildings, and one thing always bugged me: the rigidity. Traditional rule-based or ML-based HBI (Human-Building Interaction) systems require extensive training data, predefined intents, and hardcoded workflows. Want to add a new command like 'dim the lights in the conference room to 30% and set the thermostat to 72°F'? That's a new pipeline. Maintenance is a nightmare, and adapting to different building configurations means retraining from scratch.

I needed a zero-shot solution: a system that could understand arbitrary natural language requests about building controls without any prior examples, and reason across multiple subsystems (lighting, HVAC, access control) programmatically. The key insight? Use multiple LLM agents, each with a specific role, communicating via structured messages and programmatic reasoning—no fine-tuning, no few-shot examples. I wanted to see if a multi-agent framework, powered by state-of-the-art LLMs, could handle this out of the box.

Tools and Setup§

I built the prototype using Python, with the following key tools:

  • **DeepSeek-67B** (via API) as the core reasoning agent. Its strong performance on logical tasks and long context made it ideal for multi-step reasoning.
  • **Claude (Anthropic API)** as the disambiguation agent, because of its excellent instruction-following and safety features when handling user intents.
  • **Cursor** for writing and iterating on the agent orchestration code—its AI pair programming sped up the development significantly.
  • **Perplexity** to research best practices for multi-agent reasoning and prompt engineering patterns.
  • A simulated building environment using Python objects (LightController, Thermostat, LockSystem) that responded to commands via a REST-like interface (flask mock).
[Loading prompt card for DeepSeek Chat...]
[Loading prompt card for Claude...]
[Loading prompt card for Perplexity AI...]

The environment consisted of three virtual rooms (lobby, office, conference room) with lights, thermostats, and door locks. Each device had state variables (on/off, brightness, temperature, etc.).

Step-by-Step: What I Actually Did§

1. Define Agent Roles and Communication Protocol§

I defined three specialized agents:

  • User Interface Agent (UI-Agent): Parses user’s natural language command, extracts entities (room, device, action, parameters), and outputs a structured JSON intent.
  • Reasoning Agent (R-Agent): Takes the structured intent and the current building state, reasons about the required actions (e.g., 'dim lights' → set brightness, 'set thermostat' → set temperature). It generates executable action plans.
  • Execution Agent (E-Agent): Validates the action plan against device capabilities, executes via API calls, and reports results.

All agents communicated via a shared message bus (Python dicts) with a 'to', 'from', 'payload', and 'status' fields. I used a simple loop: UI-Agent processes user input → sends structured intent to R-Agent → R-Agent sends action plan to E-Agent → E-Agent executes and returns result. If R-Agent encounters ambiguity, it queries UI-Agent for clarification.

2. Prompt Engineering for Zero-Shot Reasoning§

Each agent had a system prompt defining its role and output format. The key was instructing R-Agent to use 'programmatic reasoning'—i.e., to output a sequence of function calls (like 'set_light_brightness("conference_room", 30)') rather than natural language. I also gave it a schema of available functions and their signatures.

For example, the R-Agent prompt included:

You are a reasoning agent for a smart building. Given a structured intent and the current building state, output an action plan as a list of JSON objects. Each object must have: "function", "parameters" (dict). Available functions: set_light_on/off, set_light_brightness, set_thermostat_temp, set_thermostat_mode, lock_door, unlock_door. If the intent is ambiguous, respond with {"clarification": "question"}.

3. Orchestration Loop with Error Handling§

I built a main loop that:

  • Accepts user input (e.g., "Make the lobby lights 50% and unlock the main door")
  • UI-Agent parses to structured intent: {"room": "lobby", "actions": [{"type": "set_brightness", "value": 50}, {"type": "unlock_door", "target": "main"}]}
  • R-Agent receives this and building state (e.g., lobby light is currently off), reasons that setting brightness to 50% requires turning the light on first (since brightness 0 = off). It outputs action plan: [{"function": "set_light_on", "parameters": {"room": "lobby"}}, {"function": "set_light_brightness", "parameters": {"room": "lobby", "value": 50}}, {"function": "unlock_door", "parameters": {"door": "main"}}]
  • E-Agent calls each function against the simulation, then reports success or failure.

I added a fallback: if any agent returns an error or ambiguous clarification, the loop pauses and asks user for more input.

Code Samples / Prompts Used§

Here's the core reasoning agent prompt (abbreviated):

You are a building automation reasoning agent.
Your task: Convert a structured user intent into an executable action plan.

Input format:
{
  "intent": {...}, // from UI agent
  "building_state": {...} // current device states
}

Output format: JSON object with key "actions" (list) or "clarification" (string).
Each action: {"function": string, "parameters": dict}

Available functions:
- set_light_on(room: string)
- set_light_off(room: string)
- set_light_brightness(room: string, value: int) // 0-100
- set_thermostat_temp(room: string, value: float) // 60-90
- set_thermostat_mode(room: string, mode: "heat"|"cool"|"off")
- lock_door(door: string)
- unlock_door(door: string)

Rules:
- If setting brightness > 0 and light is off, include set_light_on first.
- If setting brightness to 0, use set_light_off instead.
- If intent references a room not in building_state, request clarification.
- Maintain order of actions as implied by intent.

Example: intent: {"room": "office", "actions": [{"type": "set_temp", "value": 72}]}
state: {"office": {"light": {"on": true, "brightness": 80}, "thermostat": {"temp": 70, "mode": "heat"}}}
output: {"actions": [{"function": "set_thermostat_temp", "parameters": {"room": "office", "value": 72}}]}

The UI-Agent prompt was simpler, extracting entities using few-shot examples (but zero-shot for novel phrases). I used DeepSeek for UI-Agent as well because it handled entity extraction well.

What Worked Well§

The system successfully handled over 80% of test commands in zero-shot mode. Complex multi-action requests like "Set the conference room to 72°F, dim the lights to 40%, and lock both doors" were decomposed correctly. The programmatic reasoning approach (function calls) reduced hallucinations compared to letting the agent write natural language commands.

Key successes:

  • Zero generalization: No fine-tuning or few-shot examples needed for new building configurations. I could add a new virtual room on the fly, and agents adapted.
  • Disambiguation: When user said "turn on the lights" without specifying room, the UI-Agent asked "Which room?", and R-Agent correctly held the action plan until clarification.
  • Error recovery: If the execution agent reported a failure (e.g., if a device is offline), the loop continued with partial success and reported which actions failed.

What Failed and Why§

Two major failures stood out:

  1. Ambiguous phrasing with multiple interpretations: The prompt "set the temperature to 70 and turn off the AC in the lobby" confused the system because it interpreted "AC" as both a device and a mode. The R-Agent sometimes output contradictory actions (set_thermostat_temp and set_thermostat_mode("cool"?)) and then set mode to off. I fixed this by adding explicit device name mapping in the UI-Agent prompt.
  2. Action ordering dependency: Requests like "unlock the door after setting the thermostat" were misinterpreted because the system processed all actions sequentially without explicit temporal reasoning. The R-Agent ignored the "after" cue. I had to add a note in the prompt to preserve sequence if temporal words are used.

Also, when the building state was very large (e.g., 50+ devices), DeepSeek's context window sometimes truncated, leading to incomplete reasoning. I mitigated by summarizing building state per room.

Results and Takeaways§

I tested 50 unique commands across 5 building configurations (different rooms, device names). Results:

  • Success rate: 82% (41/50) with zero-shot prompts. After adding a few disambiguation rules (2-shot examples for UI-Agent), it rose to 92%.
  • Average response time: ~8 seconds for a full multi-action cycle (API calls to LLMs + simulation). Main bottleneck was DeepSeek API latency.
  • User satisfaction: In a small user study (5 people), 4 found the natural language interaction intuitive and effective.

The project demonstrates that a multi-agent system with programmatic reasoning can achieve robust zero-shot HBI without custom training data. The modular architecture allows swapping agents (e.g., use Claude for all roles) and adding new subsystems easily.

Key Takeaways:

  • Zero-shot multi-agent frameworks for HBI are feasible with current LLMs by structuring communication as typed messages and actions as function calls.
  • Programmatic reasoning (outputting executable plans) reduces hallucination and improves reliability compared to free-form text generation.
  • Prompt engineering is critical—each agent needs clear role definitions, output formats, and domain-specific rules (like action dependencies).
  • The approach generalizes to new building configurations without retraining, making it ideal for dynamic environments.

Try It Yourself§

If you want to experiment with this approach, here's a minimal setup:

  1. Get API keys for DeepSeek and Claude.
  2. Install Python 3.10+, requests, flask (for simulation).
  3. Clone my starter repo (see link in comments) which includes agent prompts and orchestration code.
  4. Run python main.py and type commands like "Set the living room temperature to 70°F, dim lights to 30%" (change room names to match your simulation).

You can swap agents – try Claude for reasoning or GPT-4 if you have access. The framework is designed to be extensible. I'm also releasing a paper with full prompts and evaluation metrics. Check the GitHub repo for updates.