Explainable Reinforcement Learning for assisting Air Traffic Controllers

Overview§

This paper explores explainability in reinforcement learning (RL) for air traffic control (ATC). The authors train a DQN agent to reroute flights around no-fly zones in a simplified grid environment. They apply saliency maps to interpret which input features (e.g., aircraft position, no-fly zone locations) most influence the agent's actions.

Methodology§

Reinforcement Learning Setup§

The environment is a 2D grid with aircraft positions, no-fly zones, and waypoints. The agent selects a heading change to avoid zones while reaching a target. The state $s_t$ includes aircraft position, velocity, no-fly zone locations, and target bearing. Actions are discrete heading adjustments ($\Delta \theta \in \{-10^\circ, 0^\circ, +10^\circ\}$). Reward function: $r_t = -d_{zone} - \lambda |\Delta \theta| + R_{goal}$ (distance penalty, action cost, goal reward).

DQN Training§

A deep Q-network with two hidden layers (64 neurons each, ReLU) approximates $Q(s,a)$. Training uses experience replay and target network. The loss is: $$ L = \mathbb{E}_{(s,a,r,s') \sim D} \left[ \left( r + \gamma \max_{a'} Q_{\text{target}}(s',a') - Q_{\text{online}}(s,a) \right)^2 \right] $$

Explainability: Saliency Maps§

A saliency map highlights input features most influential on $Q(s,a)$. For input $x$ (flattened state), the gradient of $Q(s,a)$ w.r.t. $x$ is computed: $\nabla_x Q(s,a)$. The saliency for feature $i$ is $S_i = |\nabla_{x_i} Q(s,a)|$. Higher magnitude indicates greater importance.

Implementation Details§

Training runs for 50,000 episodes with $\epsilon$-greedy exploration (decaying from 1.0 to 0.01). Learning rate: 0.001, discount factor $\gamma=0.99$, replay buffer size 100,000, batch size 32.

PyTorch Code Snippet§

import torch
import torch.nn as nn
import torch.optim as optim

class DQN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(input_dim, 64),
            nn.ReLU(),
            nn.Linear(64, 64),
            nn.ReLU(),
            nn.Linear(64, output_dim)
        )
    def forward(self, x):
        return self.fc(x)

# Training loop placeholder
dqn = DQN(state_dim, action_dim)
target = DQN(state_dim, action_dim)
optimizer = optim.Adam(dqn.parameters(), lr=0.001)

for episode in range(num_episodes):
    state = env.reset()
    done = False
    while not done:
        # epsilon-greedy action
        if random.random() < eps:
            action = env.action_space.sample()
        else:
            with torch.no_grad():
                qvals = dqn(state)
                action = qvals.argmax().item()
        next_state, reward, done, _ = env.step(action)
        replay_buffer.add(state, action, reward, next_state, done)
        if len(replay_buffer) > batch_size:
            batch = replay_buffer.sample(batch_size)
            # compute loss and update...
        state = next_state

Results§

Saliency maps averaged over episodes show that the aircraft's position relative to no-fly zones and the closest zone distance are the top contributing features. The agent ignores irrelevant features like distant zones. This provides interpretability for trust in ATC automation.