# DQN Components

This document describes the core Deep Q-Network components used by the path-planning system. The corresponding implementation is contained in `dqn.py`.

The module includes:

- the replay memory used during DQN training
- the transition format used to store experience tuples
- the DQN neural network architectures
- the default training hyperparameters
- the default start and goal positions used by the experiment setup

## Replay Memory

`ReplayMemory` implements the experience replay buffer used during DQN training.

During training, the agent stores transitions from its interaction with the environment. These transitions are later sampled randomly to train the policy network.

Experience replay helps reduce the correlation between consecutive environment steps and allows the model to learn from previously observed states.

## Transition Format

Transitions are represented using the following named tuple:

```python
Transition = namedtuple(
    "Transition",
    ("state", "action", "next_state", "reward")
)
```

Each transition contains:

| Field | Description |
|---|---|
| `state` | The current environment state |
| `action` | The action selected by the policy |
| `next_state` | The state reached after applying the action |
| `reward` | The reward returned by the environment |

## DQN Architectures

Two feed-forward DQN architectures are provided.

### `DQN`

`DQN` defines a Q-network with two hidden layers. The input size, output size, and number of hidden nodes are configurable.

### `DQN3`

`DQN3` defines a deeper Q-network with three hidden layers. This was used for larger-capacity experiments where a deeper policy network was required.

Both architectures output one Q-value for each available action in the environment.

## Training Hyperparameters

The default DQN training hyperparameters are defined in `dqn.py` and imported by the training script.

| Parameter | Description |
|---|---|
| `BATCH_SIZE` | Number of transitions sampled from replay memory per optimization step |
| `GAMMA` | Discount factor for future rewards |
| `EPS_START` | Initial epsilon value for epsilon-greedy exploration |
| `EPS_END` | Final epsilon value after decay |
| `EPS_DECAY` | Rate of exponential epsilon decay |
| `TAU` | Soft update rate for the target network |
| `LR` | Learning rate used by the optimizer |

## Default World Positions

The module also defines default positions used by the training and evaluation setup.

| Variable | Description |
|---|---|
| `start_position` | Default starting position of the drone |
| `goal` | Default goal position used by the environment |

These values are part of the experiment configuration. If different missions or start/goal pairs are used, they should be documented with the corresponding experiment or mission file.

## Example Usage

The example below shows how the DQN policy and target networks can be initialized from the environment.

```python
import torch

from environment import World
from dqn import DQN3

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Create the training environment.
env = World()

# Reset the environment and process the returned state object.
state = env.reset()
n_observations = len(state.process(device="cpu", rem_keys=["prev_position"]))

# Obtain the number of available actions.
n_actions = env.action_space.n

# Create the policy and target networks.
policy_net = DQN3(n_observations, n_actions, 512).to(device)
target_net = DQN3(n_observations, n_actions, 512).to(device)

# Initialize the target network with the policy network weights.
target_net.load_state_dict(policy_net.state_dict())
```

Further usage is shown in `train.py`.

## Notes

Although this module is named `dqn.py`, it also contains the DQN-related experiment constants used by the training workflow. These values are kept together because they define the default configuration used when training and evaluating the DQN path-planning model.