# Environment

This document describes the custom 3D drone navigation environment implemented in `environment.py`.

The environment was built to support training and evaluating a Deep Q-Network (DQN) path-planning agent. It represents a bounded 3D world containing a drone, a goal position, optional cylindrical assets/obstacles, and optional range-style sensor readings.

## Overview

The environment is represented by the `World` class. A `World` instance contains:

- a bounded 3D navigation space
- a drone start position
- a goal position
- optional cylindrical assets/obstacles
- optional ranging sensors
- a discrete action space
- a state dictionary used by the DQN model
- reward and termination logic
- logging and rendering utilities

The environment follows a reinforcement-learning-style interface:

```python
state = env.reset()
next_state, reward, done = env.step(action)
```

The returned state is a `State` object, which can be processed into a PyTorch tensor for use by the DQN model.

## Main Classes

## `Cylinder`

`Cylinder` represents a cylinder-shaped asset in the environment.

Cylinder assets are used as obstacles or objects in the 3D world. Each cylinder is defined by:

| Attribute | Description |
|---|---|
| `radius` | Radius of the cylinder |
| `center` | Cartesian coordinate of the cylinder base center, as `(x, y, z)` |
| `height` | Height of the cylinder |

### Example

```python
from environment import Cylinder

cylinder = Cylinder(
    radius=10.5,
    center=(100, 150, 0),
    height=23.5,
)
```

## `State`

`State` extends Python's built-in `dict` and represents the current environment state.

It is used to store values such as:

- previous drone position
- current drone position
- goal position
- distance to goal
- optional sensor readings

The main addition provided by `State` is the `process()` method, which converts the state dictionary into a PyTorch tensor.

### Creating a State

```python
from environment import State

empty_state = State()

single_item_state = State({
    "position": (0, 0, 0),
})

multi_item_state = State({
    "position": (0, 0, 0),
    "next_position": (0, 0, 10),
    "goal": (0, 0, 100),
})
```

Because `State` inherits from `dict`, it can be used like a normal dictionary.

### Processing a State

```python
from environment import State

state = State({
    "position": (0, 0, 0),
    "next_position": (0, 0, 10),
    "goal": (0, 0, 100),
})

device = "cpu"
rem_keys = ["next_position"]

processed_state = state.process(device, rem_keys)

print(processed_state)
```

Output:

```text
tensor([  0.,   0.,   0.,   0.,   0., 100.])
```

`State.process()` creates a copy of the state before processing it. This means the original state dictionary is preserved.

### Removing State Keys

The `rem_keys` argument can be used to exclude state entries before converting the state to a tensor.

For example, during training the previous position may be removed before passing the state into the DQN:

```python
processed_state = state.process(
    device="cpu",
    rem_keys=["prev_position"],
)
```

The method signature is:

```python
process(self, device: str, rem_keys: list | None = None) -> Tensor
```

## `Discrete`

`Discrete` extends Python's built-in `set` and is used to represent the environment's action space.

It adds an `n` attribute, which stores the number of available actions.

### Example

```python
from environment import Discrete

action_space = Discrete({0, 1, 2, 3, 4, 5, 6})

print(action_space)
print(action_space.n)
```

Output:

```text
{0, 1, 2, 3, 4, 5, 6}
7
```

## `World`

`World` is the main environment class.

It defines the drone navigation environment, including the world bounds, drone, goal, assets, sensors, action space, reward function, logging, and rendering.

### Constructor

```python
env = World(
    goal=(0, 0, 0),
    start=(5000, 5000, 5000),
    verbosity=0,
    threshold=40,
    size=(10000, 10000, 10000),
    ground=0,
    step_length=10,
    num_assets=0,
    num_sensors=0,
)
```

### Key Arguments

| Argument | Description |
|---|---|
| `goal` | Goal position for the drone |
| `start` | Starting position of the drone |
| `verbosity` | Logging level |
| `threshold` | Distance from the goal at which the mission is considered complete |
| `size` | Size of the 3D world as `(x, y, z)` |
| `ground` | Ground level used for collision checks |
| `step_length` | Distance moved in a single action |
| `num_assets` | Number of cylindrical assets to randomly create |
| `num_sensors` | Number of ranging sensors to attach to the drone |

## Action Space

The environment action space is represented as:

```python
Discrete({0, 1, 2, 3, 4, 5, 6})
```

The `World.interpret_action()` method maps actions to movement offsets:

| Action | Movement |
|---|---|
| `0` | Move right: `(+step_length, 0, 0)` |
| `1` | Move forward: `(0, +step_length, 0)` |
| `2` | Move upward: `(0, 0, +step_length)` |
| `3` | Move left: `(-step_length, 0, 0)` |
| `4` | Move backward: `(0, -step_length, 0)` |
| `5` | Move downward: `(0, 0, -step_length)` |
| `6` | No movement / unmapped action |

Action `6` is included in the action space but is not explicitly mapped to a movement direction in `interpret_action()`. Because the default offset is `(0, 0, 0)`, selecting action `6` leaves the drone in the same position.

## Environment Step

The environment uses a standard step pattern:

```python
next_state, reward, done = env.step(action)
```

During each step, the environment:

1. applies the selected action
2. updates the drone position
3. updates the state observation
4. checks for collisions
5. checks whether the goal has been reached
6. computes the reward
7. writes position logs if logging is enabled

## Reward Logic

The reward is computed using the drone's movement relative to the goal.

Terminal rewards:

| Condition | Reward |
|---|---|
| Collision | `-100` |
| Goal achieved | `100` |

For non-terminal states:

- moving closer to the goal gives a positive reward equal to the reduction in distance
- moving farther from the goal gives a negative reward scaled by `0.75`

The environment terminates an episode when:

- a collision occurs
- the goal is achieved

## Collision Detection

Collision detection is handled by the `Drone` class.

The drone is represented using a rectangular bounding box. Collisions are checked against:

- the ground
- the world boundaries
- cylindrical assets in the environment

A collision returns `True`; otherwise, the drone is considered to be in a valid position.

## Sensors

The environment can optionally include range-style sensors.

Sensors are initialized when `num_sensors` is greater than zero. Each sensor has an orientation corresponding to one axis direction.

| Orientation | Direction |
|---|---|
| `0` | Positive x-axis |
| `1` | Positive y-axis |
| `2` | Positive z-axis |
| `3` | Negative x-axis |
| `4` | Negative y-axis |
| `5` | Negative z-axis |

The sensor readings are stored in the environment state under:

```python
"sensor_readings"
```

Each sensor returns the nearest detected asset distance within range, or `0` if no asset is detected.

## Logging

The environment can log generated environments and drone paths.

The logging level is controlled by `verbosity`.

| Verbosity | Behavior |
|---|---|
| `0` | No logs are written |
| `1` | Environment information is logged |
| `2` | Environment information and drone position logs are written |

Environment logs contain world configuration and asset information.

Position logs contain the drone positions recorded during each episode. These logs are used to inspect the output path and can also be rendered using `render_path()`.

By default, logs are written to:

```text
logs/
```

When running in Google Colab, logs are written to:

```text
/content/drive/My Drive/
```

## Creating and Rendering an Environment

```python
from environment import World
from dqn import goal, start_position

env = World(
    goal=goal,
    start=start_position,
    verbosity=2,
    num_assets=60,
)

env.render()
```

This creates a 3D world with randomly placed cylindrical assets and renders the environment using Matplotlib.

## Loading a Saved Environment

Saved environments can be loaded from environment log files.

```python
from environment import World

env = World()
env.load_env("5_5_28")
```

This expects an environment log using the naming pattern:

```text
env_log__5_5_28.txt
```

The loaded environment includes the saved world size, start position, goal position, and asset specifications.

## Rendering a Saved Mission Path

If a position log exists for the loaded environment, the path can be rendered using:

```python
from environment import World

env = World()
env.load_env("5_5_28")
env.render_path()
```

This loads the environment and renders the recorded drone path.

A specific episode can also be rendered:

```python
env.render_path(episode=0)
```

Passing `episode=-1` renders the episode with the most recorded positions.

## Updating the Goal

The goal position can be updated manually:

```python
env.update_goal((4000, 4500, 5000))
```

A random goal can also be assigned within a given distance from the start position:

```python
env.assign_random_goal(2000)
```

Or from a range of possible distances:

```python
env.assign_random_goal(range(1000, 3000))
```

## Notes

This environment is custom-built for the DQN path-planning workflow in this repository. It is similar in style to a reinforcement learning environment, but it is not a Gym/Gymnasium environment.

The environment is designed to support:

- DQN training
- DQN evaluation
- path logging
- path visualization
- REST API output generation for the broader robotics integration workflow