Traditional control theory assumes you can write down the physics of the system mathematically. ML asks a different question: what if you could learn the right behaviour from data and experience, without needing to model every physical detail? This module lays the mathematical foundations β supervised learning, reinforcement learning, optimisation theory, and the critically important sim-to-real gap β that everything in Modules 07 and 08 builds on.
π REFERENCES: MDPI Autonomous Control Ch.4β5 Β· Bestaoui Sebbane Β§9 Β· Virell Β§4β5 Β· Sadraey Β§7
The GNC techniques in Modules 02β05 are powerful β but they all assume you can write down equations that describe the system's behaviour. For a fixed-wing UAV in calm air, that works well. But what about a multirotor in a dust storm, a swarm coordinating against an adversary that adapts its tactics, or a UAV that must identify a camouflaged target from visual data? Writing explicit equations for all of these is either impossible or impractical. Machine learning is the alternative: instead of hand-designing the decision rule, you expose the system to data or experience and let it learn the rule itself.
Bestaoui Sebbane identifies four specific failure modes of classical GNC that ML addresses: Bestaoui Sebbane Β§9.1
| Classical GNC Limitation | ML Alternative | UAV Application |
|---|---|---|
| Model uncertainty β real aerodynamics deviate from equations | Learn a residual correction from flight data | Adaptive attitude control under icing, prop damage, or payload shift |
| Cannot process raw sensor data (images, point clouds) | Deep neural networks as perceptual front-ends | Target detection from EO/IR, terrain classification from LiDAR |
| Fixed control laws β cannot adapt to novel environments | Reinforcement learning learns policies through interaction | Obstacle avoidance in unknown environments, adversarial manoeuvring |
| Combinatorial planning β too many decision variables | Learned heuristics / value functions approximate optimal decisions | Multi-UAV task allocation, mission replanning under attrition |
All of machine learning organises into three fundamental paradigms based on what kind of feedback the learning algorithm receives. Supervised learning β a teacher shows you labelled examples ("this image contains a tank"). Unsupervised learning β you find patterns in data with no labels. Reinforcement learning β you learn by trying things and receiving rewards or penalties. For UAVs: supervised learning handles perception tasks (what do I see?), unsupervised handles data organisation (what types of terrain are there?), and reinforcement learning handles action/decision tasks (what should I do next?).
For a UAV perception task, supervised learning involves finding a function $f_\theta$ parameterised by weights $\theta$ that minimises a loss function: MDPI Autonomous Control Ch.4
$\ell(\hat{y}, y)$: loss function β measures how wrong the prediction is. Common choices: mean squared error $\ell = (\hat{y}-y)^2$ (regression), cross-entropy $\ell = -y\log\hat{y}$ (classification). $\mathcal{L}(\theta)$: empirical risk β average loss over training data. The goal is to minimise this, which in practice is done with gradient descent.
Critical subtlety: We minimise loss on training data, but we care about performance on unseen test data. These are different quantities β the gap between them is called generalisation error, and understanding it is the central challenge of machine learning theory.
Training a neural network means finding the weights $\theta$ that minimise the loss function. This is a high-dimensional optimisation problem β a typical neural network might have millions of parameters. You cannot try all combinations; you must follow the gradient downhill. Gradient descent is the workhorse algorithm, but the loss landscape is full of traps: flat regions where progress stalls, saddle points, and sharp narrow minima that do not generalise well to new data. Understanding the optimisation process is as important as understanding the network architecture.
You are blindfolded on a mountainous landscape (the loss surface). You want to reach the lowest valley (minimum loss). You can only feel the slope under your feet (the gradient). Gradient descent takes a step downhill at each moment. The step size (learning rate) matters enormously β too large and you overshoot valleys; too small and you take forever, or get stuck in a local dip. Momentum carries you through shallow flat regions. Adam combines both ideas.
$\eta$: learning rate; $\mathcal{B}$: mini-batch of size $B$. Batch GD: exact gradient, expensive per step. SGD: noisy gradient from one sample, cheap per step but high variance. Mini-batch SGD: balance β the universal practical choice. Batch size $B = 32$β$256$ is typical for UAV perception tasks. MDPI Autonomous Control Ch.4
Pure SGD requires careful learning rate tuning. Adam (Adaptive Moment Estimation) adapts the learning rate for each parameter individually using running estimates of gradient mean and variance: Bestaoui Sebbane Β§9.2
Default hyperparameters: $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$, $\eta = 10^{-3}$. Why this works better than SGD: Parameters with consistently large gradients get a smaller effective learning rate (auto-normalised by $\sqrt{\hat{v}_t}$). Parameters with small, variable gradients get a larger effective step. This prevents the common problem of some weights updating too fast while others stagnate β critical for deep networks where gradient magnitudes vary by orders of magnitude across layers.
For a neural network $f_\theta$ with $L$ layers, computing $\nabla_\theta \mathcal{L}$ requires applying the chain rule through every layer. This is the backpropagation algorithm: MDPI Autonomous Control Ch.4
$\boldsymbol{\delta}^{(l)}$: error signal (gradient) at layer $l$; $\sigma'$: derivative of activation function; $\odot$: element-wise product (Hadamard product). Computational cost of backprop is approximately 2Γ the forward pass β making training roughly 3Γ the cost of inference. On UAV embedded hardware (NVIDIA Jetson Orin, 15W), this means inference is feasible at 30 Hz; online training is generally not β models are trained offline and deployed.
A 3-layer network classifies UAV sensor readings as "threat / no threat." Compute the gradient of loss with respect to layer 1 weights, tracing backprop. Use sigmoid activations $\sigma(z) = 1/(1+e^{-z})$.
Suppose forward pass gives: $z^{(1)} = 0.5$, $a^{(1)} = \sigma(0.5) = 0.622$. Layer 2: $z^{(2)} = 0.8$, $a^{(2)} = 0.690$. Output: $\hat{y} = 0.690$, true label $y = 1.0$. Loss: $\mathcal{L} = -(y\log\hat{y}) = -\log(0.690) = 0.371$.
Backward pass: $\delta^{(3)} = \hat{y} - y = 0.690 - 1.0 = -0.310$ (output gradient for cross-entropy + sigmoid)
$\delta^{(2)} = \delta^{(3)} \cdot W^{(3)} \cdot \sigma'(z^{(2)})$ where $\sigma'(0.8) = 0.690 \times (1-0.690) = 0.214$ $= -0.310 \times W^{(3)} \times 0.214$
$\delta^{(1)} = \delta^{(2)} \cdot W^{(2)} \cdot \sigma'(z^{(1)})$ where $\sigma'(0.5) = 0.622 \times (1-0.622) = 0.235$
Key observation: Each backward step multiplies by $\sigma'(\cdot)$. For sigmoid, $\sigma'(z) \leq 0.25$ always. In a deep network (10+ layers), $0.25^{10} \approx 10^{-6}$ β the gradient essentially vanishes. This is the vanishing gradient problem that makes deep sigmoid networks untrainable. ReLU activations ($\sigma'(z) = 1$ for $z > 0$) solve this β which is why every modern UAV perception network uses ReLU, not sigmoid. MDPI Autonomous Control Ch.4
A model that works perfectly on its training data but fails on real flight data is useless β and dangerous. This failure is called overfitting: the model has memorised the training examples rather than learned a general rule. In military UAV applications, the stakes of overfitting are extremely high β a target detection system trained on desert imagery that fails in jungle terrain can cost lives. Understanding when and why models generalise, and how to improve generalisation, is not optional theory. It is safety engineering.
Bias: How wrong is the model on average? A model that always predicts "no target" in urban terrain has high bias. Variance: How much does the prediction change with different training sets? A very deep network that memorises its training data has high variance. Trade-off: Simpler models have high bias, low variance. Complex models have low bias, high variance. The optimum lies between β neither too simple (underfitting) nor too complex (overfitting).
For a hypothesis class $\mathcal{H}$ of finite size $|\mathcal{H}|$, with probability at least $1-\delta$ over a training set of $N$ i.i.d. samples, every hypothesis $h \in \mathcal{H}$ satisfies:
$$\mathcal{L}_{test}(h) \leq \mathcal{L}_{train}(h) + \sqrt{\frac{\ln|\mathcal{H}| + \ln(1/\delta)}{2N}}$$Reading this in plain English: Test error is bounded by training error plus a complexity penalty. The penalty grows with model complexity ($\ln|\mathcal{H}|$) and shrinks with more data ($1/\sqrt{N}$).
UAV operational implication: For a target detection model with $10^6$ parameters and only 500 labelled examples ($N = 500$), $\delta = 0.05$: the generalisation gap bound $\approx \sqrt{(10^6 \ln 10 + \ln 20)/1000} \approx 151$. This is vacuously large β the bound gives no useful guarantee. In practice this means: 500 training images is nowhere near enough to train a reliable million-parameter network from scratch. You must use transfer learning from large pre-trained models. MDPI Autonomous Control Ch.5
Since we can rarely collect enough data to make the PAC bound tight, regularisation artificially constrains model complexity to improve generalisation: Bestaoui Sebbane Β§9.3
| Technique | Mechanism | Mathematical Form | UAV Use Case |
|---|---|---|---|
| L2 (Weight Decay) | Penalise large weights β keeps model "simple" | $\mathcal{L}_{reg} = \mathcal{L} + \lambda\|\theta\|_2^2$ | All deep learning models; default in most UAV papers |
| Dropout | Randomly zero out neurons during training β forces redundancy | $\tilde{h}_i = h_i \cdot \text{Bernoulli}(p)$ | Fully-connected layers in perception networks |
| Batch Normalisation | Normalise layer inputs β stabilises and regularises implicitly | $\hat{x} = (x - \mu_\mathcal{B})/\sigma_\mathcal{B}$ | CNNs for EO/IR β improves training speed and stability |
| Data Augmentation | Artificially expand training set with transforms | Random flip, crop, colour jitter, rotation | Critical for UAV β accounts for altitude variation, viewing angle, lighting |
| Early Stopping | Halt training when validation loss stops improving | Monitor $\mathcal{L}_{val}$; stop when $\Delta\mathcal{L}_{val} < \epsilon$ | All UAV models β prevents memorising training flight conditions |
In reinforcement learning, the UAV learns by doing. It takes actions, receives a reward (or penalty), and gradually learns which actions lead to higher cumulative reward. Unlike supervised learning, there are no labelled examples telling it what the right action was β only the reward signal, which might be delayed by many steps. A UAV learning to navigate through obstacles doesn't get told "turn left here" β it gets a reward of +1 for reaching the goal and -100 for crashing, and must figure out the connecting chain of good actions by itself.
Every RL problem for UAVs is cast as a Markov Decision Process (MDP): MDPI Autonomous Control Ch.5
$\mathcal{S}$: state space (UAV position, velocity, sensor readings); $\mathcal{A}$: action space (thrust commands, waypoint selection); $P(s'|s,a)$: transition probability (how the environment responds); $R(s,a)$: immediate reward; $\gamma \in [0,1)$: discount factor. $V^\pi(s)$: value function β expected total future reward from state $s$ following policy $\pi$. $Q^\pi(s,a)$: action-value function β expected total reward taking action $a$ in state $s$. The optimal policy selects: $\pi^*(s) = \arg\max_a Q^*(s,a)$.
Designing the reward function is the most consequential engineering decision in UAV RL. A poorly designed reward leads to unexpected behaviours β the agent finds ways to maximise the numerical reward without accomplishing the actual mission objective. This is called reward hacking. Bestaoui Sebbane Β§9.4
| UAV Task | Naive Reward | Problem | Better Reward Design |
|---|---|---|---|
| Reach waypoint | $+1$ at goal, $0$ otherwise | Sparse reward β agent wanders randomly for epochs before first success | Dense shaping: $r = -\|\mathbf{p} - \mathbf{p}_{goal}\|_2$ at each step |
| Avoid obstacles | $-100$ on collision | Agent learns to hover motionlessly to avoid all risk | $r = -\|\mathbf{p} - \mathbf{p}_{goal}\|_2 - \lambda\cdot\mathbf{1}[\text{collision}]$ |
| Energy efficiency | Time to goal | Agent reaches goal fast but with excessive thrust (battery drain) | $r = r_{progress} - \alpha\|\mathbf{u}\|^2$ (penalise control effort) |
| ISR coverage | Area covered | Agent revisits same areas repeatedly for easy coverage credit | $r = \text{new area covered only}$ (information gain reward) |
$r_t(\theta) = \pi_\theta(a_t|s_t)/\pi_{\theta_{old}}(a_t|s_t)$: probability ratio between new and old policy. $\hat{A}_t$: advantage estimate β how much better was action $a_t$ vs. the average? The clipping prevents the policy from changing too much in one step, which is the key stability improvement over vanilla policy gradient. PPO is the dominant algorithm for UAV navigation RL because it is stable, sample-efficient, and works well with continuous action spaces (motor commands). MDPI Autonomous Control Ch.5
Fully specify an MDP for training a quadrotor to navigate through a GPS-denied building interior, reaching a target room while avoiding walls.
| MDP Component | Specification | Engineering Rationale |
|---|---|---|
| State $s$ | $[x,y,z,\phi,\theta,\psi,\dot{x},\dot{y},\dot{z},d_{front},d_{left},d_{right},d_{up},d_{down}]^T \in \mathbb{R}^{14}$ | Position, attitude, velocity + 5 sonar distances |
| Action $a$ | $[\delta T, \delta\phi_{ref}, \delta\theta_{ref}, \delta\psi_{ref}] \in [-1,1]^4$ | Normalised thrust and attitude setpoints β inner loop handles actual control |
| Transition $P$ | Gazebo physics simulator (Module 06 sim-to-real section) | Real physics without hardware risk |
| Reward $R$ | $r = -0.1\|\mathbf{p}-\mathbf{p}_{goal}\|_2 - 0.01\|\mathbf{u}\|^2 - 50\cdot\mathbf{1}[\text{collision}] + 100\cdot\mathbf{1}[\text{goal reached}]$ | Dense progress + control efficiency + terminal reward/penalty |
| Discount $\gamma$ | 0.99 | Long-horizon task (200+ timesteps to reach goal) |
| Episode length | 500 steps at 10 Hz = 50 s max | Long enough to reach goal; short enough to learn from many episodes |
Training setup: 16 parallel Gazebo instances (each a random room layout). PPO with 2048 steps per rollout, 10 epochs, batch size 64. After ~5 million environment steps (~8 hours on a GPU server), the policy achieves >90% goal-reaching rate on unseen room configurations. This policy is then the candidate for sim-to-real transfer.
Training RL policies in simulation is cheap and safe β you can crash the UAV thousands of times without damaging real hardware. But the simulation is never a perfect copy of reality. The real UAV has motor delays the simulator didn't model. Real air has turbulence that the simulator's aerodynamic model simplified away. The real camera has lens distortion and rolling shutter effects absent in the rendered simulation. When you take a policy trained entirely in simulation and deploy it on real hardware, it often fails β sometimes immediately, sometimes subtly. This is the sim-to-real gap, and bridging it is one of the hardest open problems in autonomous UAV systems.
| Gap Source | Description | Magnitude | Mitigation |
|---|---|---|---|
| Aerodynamic modelling | Simplified drag, ground effect, blade flapping ignored | 5β15% thrust error | System identification from flight data; add aerodynamic noise in sim |
| Motor/ESC dynamics | Simulator uses ideal instantaneous thrust; real motors have 20β50 ms lag | 20β50 ms delay | Add motor time constant ($\tau_{motor}$) to simulator model |
| Sensor noise model | Simulated IMU uses Gaussian noise; real IMU has correlated noise, spikes | 2β5Γ real noise level | Record and replay real sensor noise in sim; domain randomisation |
| Visual appearance | Simulated textures, lighting, shadows differ from real world | Large domain shift | Photorealistic rendering (Unreal Engine); domain randomisation |
| Payload / CoM shift | Real payloads shift centre of mass; sim uses fixed rigid body | 1β5% inertia error | Randomise inertia parameters; robust control inner loop |
Domain randomisation deliberately introduces variety into the simulation so that the real world is just one of many environments the policy has already seen. If the policy works across a wide range of simulated conditions, it is more likely to work in reality: Virell Β§4.3
$\xi$: simulation parameter vector (motor constants, aerodynamic coefficients, sensor noise levels, visual textures). $p(\Xi)$: distribution over simulation parameters. $\mathcal{M}_\xi$: MDP with parameter setting $\xi$. Training maximises expected reward across the distribution of simulated environments, not just a single fixed simulation. The real world is then treated as an out-of-distribution sample from $p(\Xi)$ that the policy must generalise to.
Let $\pi$ be a policy trained in simulation domain $\mathcal{D}_{sim}$ and deployed in real domain $\mathcal{D}_{real}$. The performance gap is bounded by:
$$J_{real}(\pi) \geq J_{sim}(\pi) - \frac{2\gamma}{(1-\gamma)^2}\,d_{TV}(\mathcal{D}_{sim},\, \mathcal{D}_{real})\cdot R_{max}$$$d_{TV}$: total variation distance between the state-action distributions in simulation and reality; $R_{max}$: maximum reward magnitude; $\gamma$: discount factor. Implication: Real-world performance is guaranteed to be within a computable margin of simulation performance β but only if we can bound the domain shift $d_{TV}$. Domain randomisation reduces $d_{TV}$ by widening the training distribution to include the real world as a likely sample. Virell Β§4.4
Virell documents the standard industry pipeline for responsible sim-to-real transfer in UAV systems: Virell Β§4.5
Supervised learning requires labelled data. For UAV perception tasks, collecting and labelling training data is expensive, slow, and operationally sensitive β you cannot simply fly over an active military target area and label images. Understanding which datasets exist, their composition, their limitations, and how to augment them synthetically is an essential practical skill for anyone building UAV ML systems.
| Dataset | Task | Size | Platform | Limitation |
|---|---|---|---|---|
| VisDrone | Object detection from UAV | 10,209 images, 2.5M bboxes | DJI consumer UAV | Civilian objects only; limited occlusion |
| UAVDT | Vehicle detection/tracking | 80K frames | Aerial fixed platform | Urban scenes; no camouflage/military vehicles |
| DroneDeploy | Aerial segmentation | 55 kmΒ² aerial imagery | Various UAVs | Building inspection; no tactical environments |
| Mid-Air | Optical flow, depth, ego-motion | 420K stereo frames | Synthetic (Unreal) | Synthetic domain β sim-to-real transfer required |
| TartanAir | Visual odometry, depth | 1M+ frames, 16 environments | Synthetic (AirSim) | Synthetic; strong domain randomisation built in |
| USAF DARPA-sourced | Military target detection | Classified / restricted | Military platforms | Access restricted; SAR + EO/IR |
Where real data is scarce or classified, synthetic data from photorealistic simulators (Unreal Engine, NVIDIA Isaac) fills the gap. The pipeline: generate synthetic images with ground truth labels β train on synthetic + fine-tune on small real dataset (domain adaptation): Virell Β§5.2
$\mathcal{L}_{task}$: supervised task loss on labelled synthetic data. $\mathcal{L}_{domain}$: domain classifier loss β can a discriminator tell if a feature came from synthetic or real data? The gradient reversal on $\mathcal{L}_{domain}$ forces the feature extractor $\theta_f$ to produce domain-invariant representations β features that look the same whether the input is synthetic or real. When this works, the task classifier trained on synthetic features transfers to real images.
pip install numpy
# ml_foundations_uav.py # βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ # Part 1: Neural network from scratch (NumPy only) # Implements: MDPI Autonomous Control Ch.4 # Part 2: Minimal PPO skeleton for UAV altitude RL # Implements: MDPI Autonomous Control Ch.5 # βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ import numpy as np # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ # PART 1 β Neural Network from Scratch # Classifies UAV sensor readings as "threat zone" / "safe zone" # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ class Layer: """Single fully-connected layer with ReLU activation.""" def __init__(self, n_in, n_out, activation='relu'): # He initialisation β scales weights for ReLU to prevent vanishing/exploding gradients self.W = np.random.randn(n_in, n_out) * np.sqrt(2.0 / n_in) self.b = np.zeros(n_out) self.activation = activation # Cache for backprop self.x_cache = None self.z_cache = None self.dW = np.zeros_like(self.W) self.db = np.zeros_like(self.b) def forward(self, x): self.x_cache = x z = x @ self.W + self.b self.z_cache = z if self.activation == 'relu': return np.maximum(0, z) elif self.activation == 'sigmoid': return 1 / (1 + np.exp(-np.clip(z, -500, 500))) elif self.activation == 'linear': return z raise ValueError(f"Unknown activation: {self.activation}") def backward(self, d_out): """Backprop through this layer. Returns gradient w.r.t. input.""" if self.activation == 'relu': d_z = d_out * (self.z_cache > 0) # ReLU gradient: 1 if z>0 else 0 elif self.activation == 'sigmoid': a = 1 / (1 + np.exp(-self.z_cache)) d_z = d_out * a * (1 - a) # sigmoid gradient β small near saturation elif self.activation == 'linear': d_z = d_out # identity gradient self.dW = self.x_cache.T @ d_z self.db = d_z.sum(axis=0) return d_z @ self.W.T class NeuralNet: """ Feedforward neural network for UAV threat classification. Architecture: input β hidden β hidden β sigmoid output """ def __init__(self, layer_sizes, activations): self.layers = [ Layer(layer_sizes[i], layer_sizes[i+1], activations[i]) for i in range(len(activations)) ] def forward(self, x): for layer in self.layers: x = layer.forward(x) return x def backward(self, d_loss): for layer in reversed(self.layers): d_loss = layer.backward(d_loss) def update(self, lr, weight_decay=1e-4): """SGD update with L2 weight decay (Bestaoui Sebbane Β§9.3).""" for layer in self.layers: layer.W -= lr * (layer.dW + weight_decay * layer.W) layer.b -= lr * layer.db def generate_uav_threat_data(N=1000): """ Synthetic UAV threat classification dataset. Features: [altitude, speed, radar_cross_section, heading_change_rate] Label: 1 = threat (incoming fast, low, maneuvering), 0 = safe """ np.random.seed(42) X = np.random.randn(N, 4) # Threat: low altitude (x[0]<0), high speed (x[1]>0.5), low RCS (x[2]<0), maneuvering (|x[3]|>0.5) y = ((X[:, 0] < 0) & (X[:, 1] > 0.5) & (X[:, 2] < 0)).astype(float) y = y.reshape(-1, 1) return X[:800], y[:800], X[800:], y[800:] # 80/20 train/test split def train_threat_classifier(): X_train, y_train, X_test, y_test = generate_uav_threat_data() # Build network: 4 inputs β 16 β 8 β 1 output (sigmoid) net = NeuralNet( layer_sizes = [4, 16, 8, 1], activations = ['relu', 'relu', 'sigmoid'], ) lr = 0.01; epochs = 100; batch_size = 32 print(f"\n{'='*54}") print(f" UAV THREAT CLASSIFIER β TRAINING") print(f" MDPI Autonomous Control Ch.4") print(f"{'='*54}") for epoch in range(epochs): # Mini-batch SGD idx = np.random.permutation(len(X_train)) epoch_loss = 0 for start in range(0, len(X_train), batch_size): batch_idx = idx[start:start+batch_size] xb, yb = X_train[batch_idx], y_train[batch_idx] # Forward pass y_hat = net.forward(xb) # Binary cross-entropy loss y_hat = np.clip(y_hat, 1e-7, 1-1e-7) loss = -np.mean(yb*np.log(y_hat) + (1-yb)*np.log(1-y_hat)) epoch_loss += loss # Backward pass β gradient of BCE + sigmoid at output d_loss = (y_hat - yb) / len(xb) net.backward(d_loss) net.update(lr) # Print every 20 epochs if (epoch+1) % 20 == 0: y_pred = (net.forward(X_test) > 0.5).astype(float) acc = np.mean(y_pred == y_test) * 100 print(f" Epoch {epoch+1:3d} | Loss: {epoch_loss:.3f} | Test Acc: {acc:.1f}%") print() # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ # PART 2 β Minimal PPO Agent for UAV Altitude Hold # Demonstrates the MDP β code translation from Worked Example 6.2 # ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ class AltitudeEnv: """ 1D UAV altitude hold environment. State: [altitude_error, vertical_velocity] Action: normalised thrust offset in [-1, 1] Reward: -|alt_error| - 0.01*action^2 """ TARGET_ALT = 50.0 DT = 0.05 # 20 Hz def reset(self): self.alt = self.TARGET_ALT + np.random.randn() * 5 self.vel = np.random.randn() * 0.5 return self._obs() def _obs(self): return np.array([self.alt - self.TARGET_ALT, self.vel]) def step(self, action): # Simple point-mass model: accel = (action * max_thrust - gravity_offset) accel = action * 4.0 - 0.0 # gravity already balanced at hover self.vel += accel * self.DT self.vel = np.clip(self.vel, -5, 5) self.alt += self.vel * self.DT self.alt = np.clip(self.alt, 0, 100) error = abs(self.alt - self.TARGET_ALT) reward = -error - 0.01 * action**2 done = error > 30 # episode ends if altitude error too large return self._obs(), reward, done class PolicyNetwork: """Linear policy for 2D state β 1D continuous action (Gaussian).""" def __init__(self): self.W = np.random.randn(2) * 0.1 self.log_std = np.array([-1.0]) # learnable log standard deviation def get_action(self, obs): """Sample action from Gaussian policy N(WΒ·obs, exp(log_std)).""" mu = self.W @ obs std = np.exp(self.log_std[0]) action = mu + std * np.random.randn() action = np.clip(action, -1, 1) log_prob = -0.5*((action-mu)/std)**2 - np.log(std) - 0.5*np.log(2*np.pi) return action, log_prob def train_ppo_altitude(n_episodes=200): """ Simplified policy gradient (REINFORCE, precursor to PPO) for altitude hold. Demonstrates the RL training loop structure used in full PPO implementations. MDPI Autonomous Control Ch.5 """ env = AltitudeEnv() policy = PolicyNetwork() lr = 0.02 gamma = 0.99 print(f"\n{'='*54}") print(f" PPO-STYLE RL β UAV ALTITUDE HOLD") print(f" MDPI Autonomous Control Ch.5") print(f"{'='*54}") for ep in range(n_episodes): obs = env.reset() rewards, log_probs, observations = [], [], [] # Rollout one episode for _ in range(200): action, lp = policy.get_action(obs) next_obs, r, done = env.step(action) rewards.append(r); log_probs.append(lp); observations.append(obs) obs = next_obs if done: break # Compute discounted returns G_t = Ξ£ Ξ³^k r_{t+k} G = 0; returns = [] for r in reversed(rewards): G = r + gamma * G returns.insert(0, G) returns = np.array(returns) returns = (returns - returns.mean()) / (returns.std() + 1e-8) # normalise # Policy gradient update: βJ β Ξ£ G_t * β log Ο(a_t|s_t) grad_W = np.zeros_like(policy.W) for t in range(len(rewards)): grad_W += returns[t] * log_probs[t] * observations[t] policy.W += lr * grad_W / len(rewards) # Print progress if (ep+1) % 50 == 0: mean_r = np.mean(rewards) final_err = abs(obs[0]) print(f" Ep {ep+1:3d} | Mean reward: {mean_r:6.2f} | Final alt err: {final_err:.2f} m | Policy W: {policy.W}") print(f"\n β Policy converges to negative altitude-error weights") print(f" (fly up when too low, fly down when too high)") print(f"{'='*54}\n") if __name__ == "__main__": np.random.seed(0) train_threat_classifier() train_ppo_altitude(n_episodes=300)