DSLE: A Learning Environment for Dark Souls Boss Encounters

Overview§

The Dark Souls Learning Environment (DSLE) is a containerized benchmarking platform for game-playing agents, wrapping all 22 boss encounters of Dark Souls: Remastered in a Gymnasium-compatible interface. Each step issues a real game action (e.g., move, attack, roll) against a live game instance, making the environment fully dynamic and high-fidelity. Observations are high-dimensional visual frames from the game's rendering, and rewards are sparse, typically terminal (win/loss) with limited shaping. To enable reproducible research, DSLE-5 is proposed as a representative subset of five bosses: Asylum Demon (tutorial), a spatially constrained arena fight, an environmental-hazard fight, a multi-target fight, and the final boss (fast).

Experimental Setup and Baselines§

The paper evaluates five methods on DSLE-5:

• Random policy – chooses actions uniformly at random.
• Expert system – hand-crafted rule-based logic using in-game telemetry (health, stamina, distance).
• Evolutionary baseline – CMA-ES over a small neural network policy, trained from raw pixels.
• PPO – proximal policy optimization with a convolutional policy network.
• DQN – deep Q-network with experience replay and target network.

Training runs are computationally expensive: each PPO/DQN run costs tens of wall-clock hours. Results show that only the Asylum Demon is defeated by the expert (63% peak win rate) and evolutionary agent (43%); all other bosses remain undefeated. PPO and DQN achieve at most 0.33% win rate on the tutorial boss and 0% elsewhere, indicating no measurable learning within the given budget. A broader evolutionary run across all 22 bosses (using advantaged level-50 stats) wins only a few early-game bosses.

Metrics Beyond Win Rate§

Because win rate is near zero for most methods, the authors emphasize auxiliary metrics:

• Survival time – how long the agent lasts before death.
• Damage dealt – total damage inflicted on the boss.

These metrics reveal two failure modes: (1) rapid deaths (<10 seconds) in cramped, multi-target encounters, and (2) long stalemates (~1 minute) where the agent deals negligible damage.

Methodology and Formulation§

The environment is modeled as a partially observable Markov decision process (POMDP). At time $t$, the agent receives a visual observation $o_t$ (typically an RGB frame). It picks an action $a_t$ from a discrete or continuous action space (e.g., movement, attack, dodge, item use). The game engine transitions to the next state, yielding a reward $r_t$. The objective is to maximize the expected discounted return:

$$ J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t=0}^{T} \gamma^t r_t \right] $$

For PPO, the clipped surrogate objective is used:

$$ L^{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) \hat{A}_t, \, \operatorname{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right] $$

where $r_t(\theta) = \frac{\pi_\theta(a_t|o_t)}{\pi_{\theta_\text{old}}(a_t|o_t)}$ is the importance ratio and $\hat{A}_t$ is the generalized advantage estimate. For DQN, the Bellman target is:

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left( r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right) $$

Implementation Details§

The platform is containerized with Docker to handle the complex game environment and ensure reproducibility. A Python wrapper communicates with the game via a modded client, reading screen buffers and injecting controller inputs. The Gymnasium interface exposes reset() and step() functions. Below is a minimal example of how a PPO agent might interface with DSLE:

import gymnasium as gym
import dsle

env = gym.make("DSLE-AsylumDemon-v0", render_mode="rgb_array")
obs, info = env.reset()

# Example: run a random policy
for t in range(1000):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()
        break

# For PPO training, we would use a vectorized version:
# vec_env = gym.make_vec("DSLE-AsylumDemon-v0", num_envs=4)
# Then loop over episodes, collecting rollouts, computing advantages, and updating the policy network.

The authors report that the environment is unsolved by current model-free RL methods, suggesting that DSLE is a challenging benchmark for future research in sample-efficient RL, memory-based architectures, and reward shaping for sparse-reward real-time games.

Key Takeaways§

• DSLE provides a realistic, high-dimensional, real-time RL benchmark with sparse rewards.
• Standard RL algorithms (PPO, DQN) fail to learn meaningful behaviors within practical compute budgets.
• The paper advocates for reporting richer metrics (survival time, damage) to diagnose failure modes in sparse-reward environments.
• DSLE-5 serves as a computationally tractable yet still difficult starting suite for benchmarking new agent algorithms.