# DQN Robot Path Planning

A Deep Q-Network path-planning component for a search-and-rescue drone project.

This repository contains a cleaned, standalone version of my reinforcement learning contribution to a broader robotics project. The project trains and evaluates a DQN policy in a custom 3D environment, converts selected actions into geodetic mission waypoints, and exposes the planner through a lightweight REST API.

The planner can generate ArduPilot/QGroundControl-compatible waypoint mission files using the `QGC WPL 110` format.

## Demo

### Rendered Path

The animation below shows an example DQN-generated path rendered in the custom 3D environment.

![Rendered DQN path](assets/rendered_path.gif)

### Training Progress

The plot below shows the final model training behavior across 1500 episodes, using the number of steps per episode as the training-progress signal. Successful and unsuccessful missions are marked separately.

![Training steps vs episode](assets/training_steps_vs_episode.png)

## Project Context

This project was originally developed as part of a broader search-and-rescue drone project.

The broader group project included additional robotics, simulation, hardware, and integration components. This repository focuses on my DQN-based path-planning contribution, including:

- a custom lightweight 3D drone environment
- DQN model definitions and training utilities
- policy evaluation scripts
- geodetic mission path generation
- ArduPilot/QGroundControl mission file output
- a small REST API wrapper for planner access

The DQN model operates in a Cartesian simulation environment. The path planner then maps the selected discrete actions into geodetic coordinates, allowing generated paths to be exported as drone mission waypoints.

## Relationship to the Original Group Project

This repository is a cleaned, standalone version of my DQN-based path-planning contribution to a broader search-and-rescue drone project.

The original broader group-project repository is available here:

[DGMD-E-17-SAR-Drone](https://github.com/Cade-Wolfaardt/DGMD-E-17-SAR-Drone)

That repository contains the wider project context and additional components beyond this reinforcement learning path-planning module. It is kept separately from this polished portfolio repository because it represents the original group project structure, while this repository focuses specifically on my DQN environment, training, evaluation, mission generation, and API interface.

## Features

- Custom 3D Cartesian drone navigation environment
- Cylindrical obstacle/asset representation
- Collision detection against assets, boundaries, and ground
- Optional environment and position logging
- Deep Q-Network policy and target-network training
- Experience replay memory
- Epsilon-greedy exploration
- Batch model evaluation
- Geodetic start/goal path planning
- Mission export to TXT and CSV
- ArduPilot/QGroundControl-compatible `QGC WPL 110` mission output
- Flask REST API for assigning start/goal positions and retrieving generated paths
- Docker support for reproducible setup

## Repository Structure

```text
.
├── README.md
├── LICENSE
├── .gitignore
├── .dockerignore
├── requirements.txt
├── Dockerfile
│
├── src/
│   └── dqn_path_planning/
│       ├── __init__.py
│       ├── path_planner.py
│       ├── api_client.py
│       ├── api_server.py
│       ├── environment.py
│       ├── eval.py
│       ├── dqn.py
│       └── train.py
│
├── missions/
│   ├── final_mission_file.csv
│   ├── final_mission_file.txt
│   └── README.md
│
├── models/
│   ├── final_DQN_policy_net.pth
│   ├── README.md
│   └── selected_checkpoints/
│       ├── _24_21_57_policy_net.pth
│       ├── _27_16_10_policy_net.pth
│       └── _26_23_6_policy_net.pth
│
├── docs/
│   ├── dqn.md
│   ├── api.md
│   ├── evaluation.md
│   ├── training.md
│   ├── environment.md
│   └── path_planner.md
│
├── logs/
│   ├── README.md
│   ├── example_env_log__5_5_28.txt
│   └── example_position_log__5_5_28.txt
│
└── assets/
    ├── rendered_path.gif
    ├── training_steps_vs_episode.png
    └── README.md
```

## Installation

Clone the repository and install the Python dependencies.

```bash
git clone <repo-url>
cd dqn-robot-path-planning
pip install -r requirements.txt
```

The project uses a `src/` layout. When running scripts locally, run commands from the project root.

If needed, set `PYTHONPATH` manually:

```bash
export PYTHONPATH=$PWD/src
```

On Windows PowerShell:

```powershell
$env:PYTHONPATH="$PWD\src"
```

## Requirements

Core dependencies include:

- PyTorch
- NumPy
- pandas
- Matplotlib
- pyproj
- Flask
- requests
- keyboard

See `requirements.txt` for the full dependency list.

## Usage

## Train a DQN Model

Run training from the project root:

```bash
python -m dqn_path_planning.train
```

The training script creates the custom environment, trains a DQN policy using experience replay, plots training progress, and saves the trained policy and target networks.

By default, the script uses GPU training when CUDA is available. CPU training is limited to a short smoke test.

See [`docs/training.md`](docs/training.md) for details.

## Evaluate a Model

Evaluate the default final model:

```bash
python -m dqn_path_planning.eval
```

Evaluate with a larger batch size:

```bash
python -m dqn_path_planning.eval --batch_size 100
```

Evaluate a selected historical checkpoint:

```bash
python -m dqn_path_planning.eval \
    --id _27_16_10 \
    --model_dir models/selected_checkpoints
```

See [`docs/evaluation.md`](docs/evaluation.md) for details.

## Generate a Mission Path

Generate a mission path using the final DQN policy:

```bash
python -m dqn_path_planning.path_planner
```

Save the generated mission path to a TXT file:

```bash
python -m dqn_path_planning.path_planner --to_txt true
```

Save both TXT and CSV outputs:

```bash
python -m dqn_path_planning.path_planner --to_txt true --to_csv true
```

Render the simulated path:

```bash
python -m dqn_path_planning.path_planner --render true --verbosity 2
```

Use custom geodetic start and goal coordinates:

```bash
python -m dqn_path_planning.path_planner \
    --start 42.355118_-71.071305_6 \
    --goal 42.355280_-71.070911_26.5
```

See [`docs/path_planner.md`](docs/path_planner.md) for details.

## Run the REST API

Start the Flask API server:

```bash
python -m dqn_path_planning.api_server
```

In a second terminal, run the API client:

```bash
python -m dqn_path_planning.api_client
```

The client assigns a start and goal position, requests a generated path, and prints the returned mission text.

The API exposes two endpoints:

| Endpoint | Description |
|---|---|
| `GET /assign-start-goal` | Assigns geodetic start and goal coordinates |
| `GET /get-path` | Generates and returns the mission path as plain text |

See [`docs/api.md`](docs/api.md) for details.

## Docker

Build the Docker image:

```bash
docker build -t dqn-robot-path-planning .
```

Run the API server:

```bash
docker run --rm -p 5000:5000 dqn-robot-path-planning
```

Run evaluation inside the container:

```bash
docker run --rm dqn-robot-path-planning python -m dqn_path_planning.eval
```

Generate mission files and persist outputs by mounting the project directory.

Windows PowerShell:

```powershell
docker run --rm -v "${PWD}:/app" dqn-robot-path-planning python -m dqn_path_planning.path_planner --to_txt true --to_csv true
```

macOS/Linux:

```bash
docker run --rm -v "$PWD:/app" dqn-robot-path-planning python -m dqn_path_planning.path_planner --to_txt true --to_csv true
```

## Mission Output

The path planner can export generated paths in the legacy QGroundControl waypoint format:

```text
QGC WPL 110
```

This format is used in ArduPilot/QGroundControl-compatible mission workflows.

A generated mission file contains:

- an initial takeoff command
- a sequence of spline waypoint commands
- a final return-to-launch command

Example command IDs used in the generated mission files:

| Command ID | Meaning |
|---:|---|
| `22` | Takeoff |
| `82` | Spline waypoint |
| `20` | Return to launch |

Example mission files are included in [`missions/`](missions/).

## Models

The final model checkpoint is stored at:

```text
models/final_DQN_policy_net.pth
```

Selected historical checkpoints are stored in:

```text
models/selected_checkpoints/
```

These checkpoints are included to show important model-development stages, but the default workflow uses `final_DQN`.

See [`models/README.md`](models/README.md) for details.

## Documentation

Detailed documentation is available in the `docs/` directory:

| Document | Description |
|---|---|
| [`docs/dqn.md`](docs/dqn.md) | DQN architectures, replay memory, and training constants |
| [`docs/environment.md`](docs/environment.md) | Custom 3D environment, state representation, actions, rewards, and logging |
| [`docs/training.md`](docs/training.md) | DQN training workflow |
| [`docs/evaluation.md`](docs/evaluation.md) | Batch and single-model evaluation |
| [`docs/path_planner.md`](docs/path_planner.md) | Geodetic path planning and mission file generation |
| [`docs/api.md`](docs/api.md) | Flask API server and client usage |

## Current Limitations

This repository represents an early DQN-based path-planning implementation rather than a production autonomous navigation stack.

Known limitations include:

- the DQN uses a discrete action space
- the environment is lightweight and custom-built rather than a full physics simulator
- the policy can exhibit repeated or similar path patterns
- the trained model does not fully generalize to all possible start/goal pairs
- obstacle avoidance was simplified during development and delegated to other drone-system components
- the REST API is a development/demo interface, not a production deployment service

## Future Work

Potential improvements include:

- training across more diverse start and goal positions
- improving reward shaping for better generalization
- adding richer obstacle and sensor modeling
- extending the action space or moving to continuous-control RL
- integrating with a more realistic drone simulator
- improving mission validation before export
- adding automated tests for mission generation and API endpoints
- packaging the project with a formal `pyproject.toml`

## License

This project is licensed under the terms of the repository license.