Jump to
MODULE 07 // DEEP LEARNING & PERCEPTION

Deep Learning & UAV Perception

A UAV that cannot interpret what its sensors see is blind. Deep learning gives UAVs the ability to extract meaning from raw images, point clouds, and video streams โ€” identifying targets, estimating depth, tracking motion, and understanding the 3D structure of the environment. This module covers CNNs, object detection, Vision Transformers, optical flow, and depth estimation โ€” with a sharp focus on what works in the constrained compute and harsh conditions of military platforms.

CNN Architecture YOLO / DETR Detection Vision Transformers Optical Flow & Depth

๐Ÿ“š  REFERENCES: MDPI Autonomous Control Ch.5โ€“6  ยท  Bestaoui Sebbane ยง9โ€“10  ยท  Virell ยง5โ€“6  ยท  Sadraey ยง7

7.1 The UAV Perception Problem

Plain English First

A camera mounted on a UAV produces a stream of numbers โ€” pixel brightness values. Nothing in those numbers directly says "this is a vehicle" or "that is a person." Turning raw pixel data into useful information โ€” locations of targets, distances to obstacles, velocity of moving objects โ€” is the perception problem. It is the bridge between sensing the world and acting in it. Before deep learning, this required hand-crafting features (edge detectors, histogram gradients) that were brittle, slow, and needed re-engineering for every new environment. Deep learning learns those features automatically from data, and the resulting systems are both more accurate and more adaptable.

The perception stack for a military UAV involves several distinct tasks, each addressed by a different deep learning architecture: MDPI Autonomous Control Ch.5

Perception TaskInputOutputArchitectureUAV Mission Use
Image classificationSingle imageClass labelCNN / ViTScene type (urban/forest/water)
Object detectionImageBounding boxes + classYOLO, DETRTarget/vehicle/person identification
Semantic segmentationImagePer-pixel class labelU-Net, DeepLabTerrain mapping, landing zone assessment
Optical flowImage pairPer-pixel velocity fieldFlowNet, RAFTEgo-motion estimation, moving target detection
Monocular depthSingle imagePer-pixel depth mapMonoDepth2, DPTObstacle distance, terrain reconstruction
Pose estimationImage6-DOF camera posePoseNet, SuperGlueVisual localisation, landing on moving platforms
The UAV-Specific Perception Challenges Standard deep learning benchmarks (ImageNet, COCO) use ground-level imagery. UAV imagery is fundamentally different: extreme viewing angles (near-nadir), small object size (a person is 10โ€“20 pixels from 100 m altitude), high motion blur, dynamic lighting, and dense clutter. Models trained on ground-level data frequently fail when deployed on UAVs without fine-tuning on aerial data. Bestaoui Sebbane ยง9.5

7.2 Convolutional Neural Networks โ€” The Spatial Feature Extractor

Plain English First

A Convolutional Neural Network (CNN) processes images by sliding small filters (kernels) across the image and detecting local patterns โ€” edges, corners, textures, then shapes, then objects at increasing abstraction. Think of it like looking at an image through increasingly complex magnifying glasses: the first glass sees edges, the second sees shapes made of edges, the third sees objects made of shapes. This hierarchical feature extraction is what makes CNNs so powerful for vision โ€” and it is the reason every modern UAV perception system is built on a CNN backbone or its attention-based successor, the Vision Transformer.

The Convolution Operation

A 2D convolution slides a kernel $\mathbf{k}$ across an input feature map $\mathbf{x}$ to produce an output feature map: MDPI Autonomous Control Ch.5

2D Discrete Convolution (Feature Map) $$(\mathbf{x} * \mathbf{k})[i,j] = \sum_{m=0}^{K-1}\sum_{n=0}^{K-1} \mathbf{x}[i+m,\, j+n]\cdot\mathbf{k}[m,n]$$

$K$: kernel size (typically 3 or 5); $[i,j]$: output pixel location. For an input of size $H \times W$ with a $K \times K$ kernel, stride $s$, and padding $p$, the output size is:

$$H_{out} = \left\lfloor\frac{H - K + 2p}{s}\right\rfloor + 1, \qquad W_{out} = \left\lfloor\frac{W - K + 2p}{s}\right\rfloor + 1$$

With padding $p = (K-1)/2$ ("same" padding), output size equals input size. With stride $s = 2$ (strided convolution), output halves in size โ€” used for downsampling instead of max pooling in modern architectures. Number of learnable parameters per conv layer: $(K^2 \cdot C_{in} + 1) \times C_{out}$, where $C_{in}, C_{out}$ are input/output channel counts and $+1$ is the bias term.

Input
640ร—640ร—3
(RGB image)
โ†’
Conv 3ร—3
320ร—320ร—64
edges
โ†’
Conv 3ร—3
160ร—160ร—128
textures
โ†’
Conv 3ร—3
80ร—80ร—256
parts
โ†’
Conv 3ร—3
20ร—20ร—512
objects
โ†’
Output
detections
boxes + classes

Key CNN Architectural Innovations

Residual Connections (ResNet)
$\mathbf{y} = \mathcal{F}(\mathbf{x}) + \mathbf{x}$
Identity shortcut bypasses layers. Allows gradients to flow directly, enabling networks 100+ layers deep. ResNet-50 backbone is the most common starting point for UAV perception.
Depthwise Separable Conv
Factorises $K^2 C_{in} C_{out}$ operations into $K^2 C_{in} + C_{in} C_{out}$. 8โ€“9ร— fewer operations. Used in MobileNet โ€” the backbone of choice for UAV edge deployment (Jetson Orin, 15W budget).
Feature Pyramid Network
Multi-scale feature maps fused top-down. Detects small objects (20โ€“40px, which is a person at 100 m UAV altitude) using high-resolution shallow features, while using deep features for semantic context. Essential for aerial detection.
Batch Normalisation
Normalises layer inputs to zero mean, unit variance per batch. $\hat{x} = (x-\mu_B)/\sigma_B$, then $y = \gamma\hat{x} + \beta$. Dramatically stabilises training, allows 10ร— higher learning rates, provides mild regularisation. Now standard in every backbone.
Worked Example 7.1 โ€” Parameter Count and FLOPs for UAV Detection Backbone

Compare ResNet-50 vs MobileNetV3-Small as backbones for a UAV real-time detection system. Target hardware: NVIDIA Jetson Orin Nano (15W, ~40 TOPS INT8).

BackboneParametersGFLOPs (224ร—224)ImageNet Top-1Jetson Orin Nano FPSVerdict
ResNet-5025.6 M4.176.1%~18 FPS (FP16)Accurate, below 30 Hz requirement
ResNet-1811.7 M1.869.8%~40 FPS (FP16)Fast enough, accuracy borderline
MobileNetV3-Small2.5 M0.0667.4%~150 FPS (FP16)Fast, moderate accuracy โ€” good for real-time
EfficientNet-B05.3 M0.3977.1%~65 FPS (FP16)Best accuracy/speed trade-off for UAV use
YOLOv8n (full detector)3.2 M8.7โ€”~45 FPS (FP16)Recommended: end-to-end detection pipeline

Engineering decision: For a 30 Hz real-time detection requirement at 640ร—640 input resolution, YOLOv8n or EfficientDet-D0 are the practical choices on a Jetson Orin Nano. ResNet-50 is the right choice only if the platform has a full Jetson AGX Orin (275 TOPS) and accuracy is paramount (e.g., ISR target identification with low false-alarm requirement). Always profile on target hardware โ€” benchmark numbers in papers often assume A100 GPUs. Virell ยง5.3

7.3 Object Detection โ€” Finding and Classifying Targets

Plain English First

Object detection solves two problems at once: where is the object in the image (a bounding box), and what is it (a class label). For a military UAV, this might mean detecting vehicles, personnel, weapons systems, or infrastructure in real-time aerial imagery. The challenge is that objects of interest can be tiny (10โ€“20 pixels from altitude), partially occluded, camouflaged, or appear at any orientation โ€” none of which standard ground-level detectors handle well.

Detection Paradigms: One-Stage vs Two-Stage

ParadigmMechanismSpeedAccuracyUAV Use
Two-stage (R-CNN family) Stage 1: propose regions. Stage 2: classify + refine each region ~5โ€“15 FPSHigh mAP Offline ISR analysis; not real-time
One-stage (YOLO family) Single network predicts all boxes and classes simultaneously 30โ€“150+ FPSCompetitive mAP Real-time onboard detection
Transformer (DETR family) Attention-based end-to-end detection; no NMS post-processing ~15โ€“30 FPSState of the art High-accuracy ISR on capable hardware

YOLO โ€” You Only Look Once

YOLO frames detection as a regression problem: a single CNN produces a grid of predictions directly. MDPI Autonomous Control Ch.5

YOLOv8 Output Tensor โ€” Grid Cell Prediction $$\text{Output} \in \mathbb{R}^{G \times G \times (5 + C)}$$ $$\text{per cell: } [t_x, t_y, t_w, t_h, p_{obj}, p_1, p_2, \ldots, p_C]$$ $$b_x = \sigma(t_x) + c_x, \quad b_y = \sigma(t_y) + c_y, \quad b_w = p_w\,e^{t_w}, \quad b_h = p_h\,e^{t_h}$$

$G \times G$: grid size (e.g., 20ร—20 at the smallest feature map scale); $c_x, c_y$: grid cell offset; $p_w, p_h$: anchor box prior dimensions. $\sigma(t_x), \sigma(t_y)$: sigmoid constrains box centre within cell. $p_{obj}$: objectness score โ€” is there an object here at all? $p_c$: class probabilities. Final confidence = $p_{obj} \times p_c$. Multi-scale prediction (3 grid scales in YOLOv8) handles objects at different sizes.

YOLO Loss Function

YOLOv8 Loss โ€” Combined Localisation + Classification + Objectness $$\mathcal{L} = \lambda_{box}\mathcal{L}_{IoU} + \lambda_{cls}\mathcal{L}_{BCE,cls} + \lambda_{dfl}\mathcal{L}_{DFL}$$ $$\mathcal{L}_{IoU} = 1 - \text{IoU}(b_{pred}, b_{gt}) \quad \text{where IoU} = \frac{|b_{pred} \cap b_{gt}|}{|b_{pred} \cup b_{gt}|}$$

$\mathcal{L}_{IoU}$: Intersection-over-Union loss โ€” directly optimises box overlap quality. $\mathcal{L}_{BCE,cls}$: binary cross-entropy for each class independently (allows multi-label). $\mathcal{L}_{DFL}$: Distribution Focal Loss โ€” predicts a distribution over box edge locations rather than a single value, improving localisation precision for small targets. Default weights: $\lambda_{box} = 7.5$, $\lambda_{cls} = 0.5$, $\lambda_{dfl} = 1.5$. Bestaoui Sebbane ยง9.6

Detection Metrics โ€” mAP, Precision, Recall

Mean Average Precision (mAP) โ€” The Standard Benchmark Metric $$\text{IoU} = \frac{|b_{pred} \cap b_{gt}|}{|b_{pred} \cup b_{gt}|} \qquad \text{(detection match criterion: IoU} \geq 0.5\text{)}$$ $$\text{AP}_c = \int_0^1 p(r)\,dr \approx \sum_{k} p(k)\,\Delta r(k) \qquad \text{(area under precision-recall curve)}$$ $$\text{mAP} = \frac{1}{C}\sum_{c=1}^C \text{AP}_c$$

A detection is a True Positive if IoU with a ground-truth box exceeds 0.5 (COCO uses 0.5โ€“0.95). mAP@0.5:0.95 is the standard COCO benchmark โ€” average AP over 10 IoU thresholds. For military UAV applications, a high-recall / moderate-precision operating point is often preferred (miss nothing, accept some false alarms) vs. the reverse โ€” this is controlled by the confidence threshold.

Worked Example 7.2 โ€” Small Object Detection at UAV Altitude

An MQ-9 carries an EO sensor at 7,620 m altitude with GSD = 24 cm/pixel (from Module 01). A standard military vehicle (Toyota Hilux technical) is 5.3 m long ร— 1.8 m wide. How many pixels does it occupy? What does this imply for detector design?

Vehicle dimensions in pixels: Length: $5.3 / 0.24 \approx 22$ px. Width: $1.8 / 0.24 \approx 8$ px. Bounding box area: $22 \times 8 = 176$ pxยฒ.

COCO "small object" threshold: area $< 32^2 = 1024$ pxยฒ. At 176 pxยฒ, this vehicle falls into the "small" category โ€” the hardest regime for all detectors. YOLOv8 achieves mAP@0.5 of ~54% on COCO small objects vs ~82% on large objects.

Mitigations for small-object detection: (1) Higher input resolution (1280ร—1280 instead of 640ร—640) โ€” doubles compute but improves small AP by ~8%. (2) SAHI (Sliced Aided Hyper Inference) โ€” run detector on overlapping tiles, merge results. (3) Feature Pyramid Network with higher-resolution P2 feature map. (4) Fine-tune on aerial vehicle dataset (VisDrone, UAVDT) โ€” domain adaptation from COCO. At lower altitudes (2,000 m), GSD โ‰ˆ 6 cm/pixel, vehicle = 88ร—30 px โ€” "medium" object, standard YOLO achieves ~72% mAP. Mission altitude is a key perception design parameter. Bestaoui Sebbane ยง9.7

7.4 Vision Transformers (ViT) โ€” Attention Over Patches

Plain English First

CNNs process images locally โ€” each filter sees a small patch at a time. For many tasks this works well, but sometimes context far away in the image is important: a camouflaged vehicle is more recognisable when you also see the nearby road and tyre tracks. Vision Transformers address this by splitting the image into patches and letting every patch "attend" to every other patch โ€” enabling global context from the very first layer. They have surpassed CNNs on many benchmarks but are computationally heavier, so hardware context matters when deciding which to use.

The Self-Attention Mechanism

Self-attention is the core operation in a Transformer. For an image split into $N$ patches of size $P \times P$: MDPI Autonomous Control Ch.6

Scaled Dot-Product Self-Attention $$\text{Attention}(Q, K, V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right) V$$ $$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$$

$X \in \mathbb{R}^{N \times d}$: sequence of $N$ patch embeddings of dimension $d$. $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$: learned projection matrices. $Q$: queries ("what am I looking for?"); $K$: keys ("what do I contain?"); $V$: values ("what do I send?"). The softmax score $\frac{QK^T}{\sqrt{d_k}}$ for patch $i$ vs patch $j$ is how much patch $i$ should "pay attention" to the content of patch $j$. Dividing by $\sqrt{d_k}$ prevents vanishing gradients in the softmax from very large dot products.

Theorem 7.1 โ€” Attention Complexity and Its UAV Implications

Standard self-attention has quadratic complexity in the sequence length:

$$\text{Complexity of full self-attention:} \quad O(N^2 d)$$ $$\text{For a 640ร—640 image with P=16 patches: } N = (640/16)^2 = 1600 \text{ patches}$$ $$\Rightarrow 1600^2 = 2.56 \times 10^6 \text{ attention weight pairs per head per layer}$$

With 12 attention heads and 12 layers, this is $\sim 360M$ operations per forward pass โ€” approximately $8\times$ more expensive than a ResNet-50. Practical consequence for UAVs: Full ViT (ViT-B/16) runs at ~5โ€“8 FPS on a Jetson Orin Nano โ€” unsuitable for real-time detection. Solutions: (1) Hierarchical ViTs (Swin Transformer) โ€” attention only within local windows, $O(N \cdot w^2 d)$ where $w$ is window size; (2) Hybrid architectures โ€” CNN backbone + Transformer neck; (3) Distillation โ€” compress ViT into a smaller CNN for deployment. Bestaoui Sebbane ยง10.1

Swin Transformer โ€” Making ViT Practical

The Swin Transformer (Shifted Window Transformer) reduces attention complexity from $O(N^2)$ to $O(N)$ by restricting attention to local windows, then shifting the windows between layers to allow cross-window communication:

Swin Window Attention Complexity $$\text{W-MSA complexity:} \quad O\!\left(4N d^2 + 2Nw^2 d\right) \approx O(Nw^2 d)$$

$w$: window size (typically 7ร—7 patches = 49 tokens per window). For $N = 1600$ patches, $w=7$: complexity โ‰ˆ $1600 \times 49 \times d = 78400d$ vs. $2.56 \times 10^6 d$ for full attention โ€” a 33ร— reduction. Swin-T achieves 81.3% ImageNet top-1 at ~4.5 GFLOPs (comparable to ResNet-50), and runs at ~35 FPS on Jetson Orin Nano โ€” borderline real-time for UAV use.

7.5 Optical Flow โ€” Seeing Motion

Plain English First

Optical flow is the apparent motion of pixels between two consecutive video frames. It tells you: every pixel moved this many pixels left/right/up/down between frame $t$ and frame $t+1$. For a UAV flying forward, the entire image flows backward โ€” that flow pattern tells you speed and direction. For a vehicle on the ground moving while the UAV is stationary, only the vehicle's pixels flow โ€” that's how you detect moving targets. Optical flow is a bridge between raw video and motion understanding, and is fundamental to both visual ego-motion estimation (VIO, Module 04) and moving target detection.

The Lucas-Kanade Equation โ€” Classical Foundation

Classical optical flow is derived from the brightness constancy assumption: Bestaoui Sebbane ยง10.2

Optical Flow Constraint Equation $$I_x u + I_y v + I_t = 0$$ $$\begin{bmatrix}I_x & I_y\end{bmatrix}\begin{bmatrix}u\\v\end{bmatrix} = -I_t$$

$I_x, I_y$: spatial image gradients (edges); $I_t$: temporal gradient (change over time); $u, v$: unknown flow at pixel $(x,y)$. This is one equation with two unknowns โ€” the aperture problem: without additional constraints, only motion perpendicular to edges is observable. Lucas-Kanade solves this by assuming constant flow within a local window $\Omega$: $$\min_{u,v} \sum_{(x,y)\in\Omega} (I_x u + I_y v + I_t)^2 \quad \Rightarrow \quad \underbrace{\begin{bmatrix}\sum I_x^2 & \sum I_x I_y \\ \sum I_x I_y & \sum I_y^2\end{bmatrix}}_{M} \begin{bmatrix}u\\v\end{bmatrix} = -\begin{bmatrix}\sum I_x I_t \\ \sum I_y I_t\end{bmatrix}$$ The matrix $M$ is the structure tensor. Its eigenvalues determine whether optical flow is reliably estimable at this point.

Deep Optical Flow โ€” RAFT

Deep learning methods dramatically outperform classical methods. RAFT (Recurrent All-Pairs Field Transforms) is the current benchmark leader for dense optical flow: MDPI Autonomous Control Ch.6

RAFT โ€” 4D Correlation Volume $$\mathbf{C}(i,j,k,l) = \langle \mathbf{f}_1(i,j),\, \mathbf{f}_2(k,l) \rangle$$

$\mathbf{f}_1, \mathbf{f}_2$: feature maps of frame 1 and frame 2 (from shared CNN encoder). The 4D correlation volume measures how similar every pixel in frame 1 is to every pixel in frame 2. A GRU (recurrent unit) iteratively refines the flow estimate by querying this correlation volume, achieving state-of-the-art accuracy (1.5 EPE on Sintel clean) at moderate compute cost (~10 FPS on Jetson Orin). For UAV ego-motion, a lightweight variant (RAFT-Small, 0.9M params) achieves ~30 FPS.

7.6 Monocular Depth Estimation

Plain English First

A single camera image contains depth cues โ€” perspective foreshortening, known object sizes, texture gradients โ€” but extracting metric depth from monocular imagery is an ill-posed problem: infinitely many 3D scenes can produce the same 2D image. Deep learning partially solves this by learning statistical priors about scene geometry from training data. The result is a dense depth map โ€” every pixel gets an estimated distance. For UAVs, this enables terrain reconstruction, obstacle detection, and landing zone assessment without requiring a stereo camera or LiDAR.

Self-Supervised Monocular Depth โ€” The MonoDepth2 Approach

Supervised depth estimation requires LiDAR ground truth โ€” expensive to collect from UAVs. Self-supervised methods use consecutive video frames as supervision: Bestaoui Sebbane ยง10.3

Self-Supervised Depth Training Loss (MonoDepth2) $$\mathcal{L}_{depth} = \mathcal{L}_{photometric} + \lambda_{smooth}\mathcal{L}_{smooth}$$ $$\mathcal{L}_{photometric} = \min_{t'} \alpha\,\frac{1-\text{SSIM}(I_t, \hat{I}_{t\to t'})}{2} + (1-\alpha)\|I_t - \hat{I}_{t\to t'}\|_1$$

$\hat{I}_{t\to t'}$: frame $t'$ warped to match frame $t$ using predicted depth + estimated camera pose. If depth is correct, warping should perfectly reconstruct frame $t$ from frame $t'$ โ€” the photometric error should be zero. The $\min$ over multiple source frames handles occlusions. $\mathcal{L}_{smooth}$: edge-aware depth smoothness โ€” penalises depth discontinuities unless there is also an image edge there (objects should have smooth depth surfaces). $\alpha = 0.85$ typically.

Scale Ambiguity โ€” The Critical Limitation

Self-supervised monocular depth is scale-ambiguous: the network cannot distinguish between a small object close up and a large object far away from a single image. The predicted depth is up to an unknown scale factor $s$: $\hat{d} = s \cdot d_{metric}$. For UAV navigation, this is critical โ€” obstacle avoidance requires metric distances. Solutions: MDPI Autonomous Control Ch.6

7.7 Edge Deployment โ€” Running DL on UAV Hardware

Plain English First

A model that achieves state-of-the-art accuracy on a GPU server is useless if it cannot run in real time on the 15โ€“50 W compute budget of a UAV payload. Edge deployment is the engineering discipline of making models fast enough, small enough, and energy-efficient enough to run onboard. The tools: quantisation (use 8-bit integers instead of 32-bit floats), pruning (remove unnecessary connections), knowledge distillation (train a small network to mimic a large one), and hardware-optimised inference engines (TensorRT, ONNX Runtime).

Quantisation โ€” INT8 vs FP16 vs FP32

Post-Training Quantisation โ€” Linear Mapping $$x_{int8} = \text{clip}\!\left(\text{round}\!\left(\frac{x_{fp32}}{s}\right) + z,\; -128,\; 127\right)$$ $$s = \frac{x_{max} - x_{min}}{255}, \quad z = -\text{round}\!\left(\frac{x_{min}}{s}\right)$$

$s$: scale factor (maps float range to int range); $z$: zero point. INT8 quantisation reduces model size by 4ร— and memory bandwidth by 4ร— vs FP32, enabling ~2โ€“4ร— inference speedup on hardware with INT8 tensor cores (Jetson Orin, Neural Processing Units). Accuracy impact: typically <1% mAP drop with calibration-aware quantisation. FP16 (half precision) halves size with negligible accuracy loss โ€” easier to apply, less speedup than INT8. Virell ยง6.2

HardwarePowerFP16 TOPSINT8 TOPSYOLOv8n FPSPlatform Class
NVIDIA Jetson Orin Nano7โ€“15 W2040~45Group 2โ€“3 UAV payload
NVIDIA Jetson AGX Orin15โ€“60 W275 (INT8)275~200Group 3โ€“4 payload
Qualcomm QRB5165 (RB5)~5 W15 (DSP)15~30Nano UAV
Hailo-8 NPU2.5 Wโ€”26~60Group 1 edge AI
Coral Edge TPU~2 Wโ€”4~15Micro UAV (limited)
The TensorRT Deployment Pipeline (Virell ยง6.3) Standard deployment pipeline for UAV deep learning models: (1) Train model in PyTorch (GPU server) โ†’ (2) Export to ONNX format (torch.onnx.export) โ†’ (3) Optimise with TensorRT (trtexec --fp16 or --int8) โ†’ (4) Deploy with DeepStream or custom inference code on Jetson โ†’ (5) Validate accuracy on representative aerial test set before field use. TensorRT optimises layer fusion, memory layout, and kernel selection for the specific GPU architecture โ€” typically 2โ€“5ร— faster than native PyTorch on Jetson hardware. Virell ยง6.3

7.8 Interactive: Neural Network Architecture Visualiser

This visualisation shows a neural network propagating activation through its layers โ€” analogous to a forward pass through a CNN feature extractor. Green nodes are the input layer (representing raw sensor data โ€” pixels or feature vectors). Cyan nodes are hidden layers (progressively more abstract feature representations). Red nodes are the output layer (class probabilities or bounding box coordinates). The pulsing activation wave shows information flowing from input to output. Drag to orbit; active layer is highlighted in the telemetry strip.

โ—ˆ NEURAL NETWORK FORWARD PASS // M07 ACTIVE: โ€”  |  SIZE: โ€”
โ€” โ€” โ€”

7.9 Python: CNN Feature Extraction + Object Detection Pipeline

What This Code Does Part 1: Implements a CNN from scratch in NumPy โ€” convolution, max pooling, batch normalisation, and a residual block. Demonstrates the parameter count calculation from Worked Example 7.1. Part 2: Implements IoU calculation and the YOLO bounding box decoder โ€” the core post-processing step that converts raw YOLO network output to human-readable detections. Part 3: Demonstrates Non-Maximum Suppression (NMS) โ€” the algorithm that removes duplicate detections. Run with: pip install numpy
Python
# dl_perception_uav.py
# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€
# Part 1: CNN building blocks (NumPy) โ€” MDPI Autonomous Control Ch.5
# Part 2: YOLO box decoder โ€” Bestaoui Sebbane ยง9.6
# Part 3: NMS post-processing โ€” standard detection pipeline
# โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€

import numpy as np


# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
# PART 1 โ€” CNN Building Blocks in NumPy
# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

def conv2d_naive(x, kernel, stride=1, padding=0):
    """
    2D convolution โ€” naive implementation for clarity.
    x:      (H, W, C_in) input feature map
    kernel: (kH, kW, C_in, C_out) filter weights
    Returns: (H_out, W_out, C_out) output feature map
    MDPI Autonomous Control Ch.5
    """
    H, W, C_in = x.shape
    kH, kW, _, C_out = kernel.shape

    # Add padding
    if padding > 0:
        x = np.pad(x, [(padding, padding), (padding, padding), (0, 0)])

    H_out = (H + 2*padding - kH) // stride + 1
    W_out = (W + 2*padding - kW) // stride + 1
    out   = np.zeros((H_out, W_out, C_out))

    for i in range(H_out):
        for j in range(W_out):
            h_start = i * stride; h_end = h_start + kH
            w_start = j * stride; w_end = w_start + kW
            patch = x[h_start:h_end, w_start:w_end, :]   # (kH, kW, C_in)
            # Vectorised over output channels
            out[i, j, :] = np.tensordot(patch, kernel, axes=([0,1,2], [0,1,2]))

    return out


def relu(x): return np.maximum(0, x)


def maxpool2d(x, pool_size=2, stride=2):
    """2ร—2 max pooling โ€” halves spatial dimensions."""
    H, W, C = x.shape
    H_out = (H - pool_size) // stride + 1
    W_out = (W - pool_size) // stride + 1
    out = np.zeros((H_out, W_out, C))
    for i in range(H_out):
        for j in range(W_out):
            out[i, j, :] = x[i*stride:i*stride+pool_size, j*stride:j*stride+pool_size, :].max(axis=(0,1))
    return out


def batch_norm(x, gamma=None, beta=None, eps=1e-5):
    """
    Batch normalisation โ€” normalises per channel.
    x: (H, W, C) feature map
    Returns normalised map with learned scale gamma and shift beta.
    """
    mu    = x.mean(axis=(0,1), keepdims=True)     # mean per channel
    sigma = x.std(axis=(0,1),  keepdims=True) + eps # std per channel
    x_hat = (x - mu) / sigma
    if gamma is None: gamma = np.ones(x.shape[-1])
    if beta  is None: beta  = np.zeros(x.shape[-1])
    return gamma * x_hat + beta


def count_conv_params(C_in, C_out, K, bias=True):
    """
    Count learnable parameters in a conv layer.
    Formula: (K^2 * C_in + bias) * C_out
    """
    return (K**2 * C_in + 1 if bias else 0) * C_out


def demo_cnn_building_blocks():
    """Forward pass through a minimal CNN block."""
    print(f"\n{'='*54}\n  CNN BUILDING BLOCKS DEMO\n{'='*54}")

    # Simulate a 16ร—16 aerial image patch, 3 channels (RGB)
    x = np.random.randn(16, 16, 3)
    print(f"  Input:           {x.shape}  (Hร—Wร—C)")

    # Conv layer 1: 3โ†’16 channels, 3ร—3 kernel, same padding
    k1 = np.random.randn(3, 3, 3, 16) * 0.1
    out1 = relu(conv2d_naive(x, k1, stride=1, padding=1))
    print(f"  After Conv1+ReLU: {out1.shape}  params={count_conv_params(3,16,3):,}")

    # Batch norm
    out1_bn = batch_norm(out1)
    print(f"  After BatchNorm:  {out1_bn.shape}  (same shape, normalised)")

    # Max pool 2ร—2: halves spatial size
    out1_pool = maxpool2d(out1_bn, pool_size=2, stride=2)
    print(f"  After MaxPool:    {out1_pool.shape}  (halved)")

    # Conv layer 2: 16โ†’32 channels, 3ร—3 kernel
    k2 = np.random.randn(3, 3, 16, 32) * 0.1
    out2 = relu(conv2d_naive(out1_pool, k2, stride=1, padding=1))
    print(f"  After Conv2+ReLU: {out2.shape}  params={count_conv_params(16,32,3):,}")

    total_params = count_conv_params(3,16,3) + 16*2 + count_conv_params(16,32,3) + 32*2
    print(f"\n  Total params: {total_params:,}  (tiny โ€” real YOLOv8n has ~3.2M)")


# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
# PART 2 โ€” YOLO Bounding Box Decoder
# Converts raw YOLO output tensor to (x1,y1,x2,y2,conf,class_id)
# Bestaoui Sebbane ยง9.6
# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

def decode_yolo_output(raw_output, anchors, n_classes, img_size, conf_thresh=0.5):
    """
    Decode YOLO raw prediction tensor to bounding boxes.

    raw_output: (G, G, A*(5+C)) โ€” grid predictions
    anchors: list of (pw, ph) anchor dimensions (in pixels, relative to img_size)
    n_classes: number of object classes
    img_size: input image size (square assumed)
    conf_thresh: minimum objectness confidence to keep

    Returns: list of [x1, y1, x2, y2, confidence, class_id]
    """
    G = raw_output.shape[0]   # grid size
    A = len(anchors)             # number of anchors
    stride = img_size / G        # pixels per grid cell

    raw = raw_output.reshape(G, G, A, 5 + n_classes)

    detections = []

    for cy in range(G):
        for cx in range(G):
            for a, (pw, ph) in enumerate(anchors):
                pred = raw[cy, cx, a]

                # Decode box centre and size (YOLO parameterisation)
                bx = (1 / (1 + np.exp(-pred[0])) + cx) * stride   # ฯƒ(tx) + cx
                by = (1 / (1 + np.exp(-pred[1])) + cy) * stride   # ฯƒ(ty) + cy
                bw = pw * np.exp(pred[2])                           # pw * e^tw
                bh = ph * np.exp(pred[3])                           # ph * e^th

                # Objectness + class probability
                obj_conf = 1 / (1 + np.exp(-pred[4]))   # sigmoid
                cls_logits = pred[5:]
                cls_probs  = np.exp(cls_logits) / np.exp(cls_logits).sum()   # softmax
                cls_id     = np.argmax(cls_probs)
                confidence = obj_conf * cls_probs[cls_id]

                if confidence > conf_thresh:
                    # Convert to corner format (x1,y1,x2,y2)
                    x1 = bx - bw/2; y1 = by - bh/2
                    x2 = bx + bw/2; y2 = by + bh/2
                    detections.append([x1, y1, x2, y2, confidence, cls_id])

    return detections


# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•
# PART 3 โ€” Non-Maximum Suppression (NMS)
# Removes duplicate detections of the same object
# โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•โ•

def compute_iou(boxA, boxB):
    """
    Intersection-over-Union between two boxes [x1,y1,x2,y2].
    IoU = intersection_area / union_area
    """
    xA = max(boxA[0], boxB[0]); yA = max(boxA[1], boxB[1])
    xB = min(boxA[2], boxB[2]); yB = min(boxA[3], boxB[3])
    inter = max(0, xB - xA) * max(0, yB - yA)
    areaA = (boxA[2]-boxA[0]) * (boxA[3]-boxA[1])
    areaB = (boxB[2]-boxB[0]) * (boxB[3]-boxB[1])
    union = areaA + areaB - inter
    return inter / (union + 1e-6)


def non_maximum_suppression(detections, iou_thresh=0.45):
    """
    NMS: suppress overlapping boxes, keep highest confidence.
    detections: list of [x1,y1,x2,y2,conf,class_id]
    Returns: filtered detections
    """
    if not detections: return []
    dets = sorted(detections, key=lambda d: d[4], reverse=True)  # sort by confidence
    kept = []
    while dets:
        best = dets.pop(0)
        kept.append(best)
        # Remove boxes that overlap too much with best
        dets = [d for d in dets if
                d[5] != best[5] or   # different class โ€” always keep
                compute_iou(best[:4], d[:4]) < iou_thresh]
    return kept


def demo_detection_pipeline():
    """Demonstrate YOLO decode + NMS pipeline."""
    print(f"\n{'='*54}\n  YOLO DETECTION PIPELINE DEMO\n  Bestaoui Sebbane ยง9.6\n{'='*54}")

    # Simulate a 20ร—20 YOLO grid with 3 anchors, 4 classes (UAV mission classes)
    # Classes: 0=vehicle, 1=person, 2=weapon_system, 3=building
    G, A, C = 20, 3, 4
    raw = np.random.randn(G, G, A * (5 + C)) * 0.5

    # Inject a confident vehicle detection at grid cell (10, 8)
    # anchor 0, conf=2.5 (sigmoidโ†’0.924), class 0 (vehicle) dominant
    raw[10, 8, : (5+C)] = np.array([0.2, 0.2, 0.3, 0.2, 2.5, 3.0, -1.0, -1.0, -1.0])
    # Add a near-duplicate detection for NMS demo
    raw[10, 9, : (5+C)] = np.array([0.1, 0.1, 0.3, 0.2, 2.0, 2.8, -1.0, -1.0, -1.0])

    anchors  = [(32, 16), (24, 24), (16, 32)]   # anchor sizes in pixels
    img_size = 640

    # Decode
    dets = decode_yolo_output(raw, anchors, C, img_size, conf_thresh=0.5)
    print(f"\n  After decode:  {len(dets)} detections above threshold")
    for d in dets[:3]:
        cls_names = ['vehicle', 'person', 'weapon_sys', 'building']
        print(f"    [{cls_names[int(d[5])]:10s}] conf={d[4]:.3f}  box=({d[0]:.0f},{d[1]:.0f},{d[2]:.0f},{d[3]:.0f})")

    # NMS
    kept = non_maximum_suppression(dets, iou_thresh=0.45)
    print(f"\n  After NMS:     {len(kept)} detections remaining")
    for d in kept[:3]:
        print(f"    [{cls_names[int(d[5])]:10s}] conf={d[4]:.3f}  box=({d[0]:.0f},{d[1]:.0f},{d[2]:.0f},{d[3]:.0f})")

    # IoU sanity check
    b1 = [100, 100, 200, 200]   # box 1
    b2 = [150, 150, 250, 250]   # 50% overlap
    b3 = [300, 300, 400, 400]   # no overlap
    print(f"\n  IoU sanity: 50% overlap = {compute_iou(b1,b2):.3f}  (expect ~0.143)")
    print(f"  IoU sanity: no overlap  = {compute_iou(b1,b3):.3f}  (expect 0.000)")
    print(f"  IoU sanity: identical   = {compute_iou(b1,b1):.3f}  (expect 1.000)")
    print(f"\n{'='*54}\n")


if __name__ == "__main__":
    np.random.seed(7)
    demo_cnn_building_blocks()
    demo_detection_pipeline()
Expected Output CNN block: Input (16,16,3) โ†’ Conv1 (16,16,16) 448 params โ†’ Pool (8,8,16) โ†’ Conv2 (8,8,32) 4,640 params ยท Detection: ~2โ€“4 detections above threshold โ†’ NMS reduces to 1โ€“2 ยท Vehicle detection at grid cell (10,8/9) merged to single box by NMS ยท IoU sanity: 50% overlap โ‰ˆ 0.143 (area intersection / area union for two half-overlapping squares), identical = 1.000
  1. Using the sensor geometry equations from Module 01 and the small object detection analysis from Worked Example 7.2: at what UAV altitude does a standard military vehicle (5.3 m ร— 1.8 m) transition from the COCO "small" (<32ยฒ px) to "medium" (<96ยฒ px) category, assuming the MTS-C sensor with GSD proportional to altitude? What does this imply about the trade-off between ISR altitude and detection performance โ€” and what operational factor limits how low you can fly?
  2. Explain the aperture problem in optical flow using the constraint equation $I_x u + I_y v + I_t = 0$. Show algebraically why this single equation cannot uniquely determine the flow vector $(u, v)$, and explain when the Lucas-Kanade local window approach also fails (degenerate structure tensor $M$). Connect this failure mode to the VIO failure modes discussed in Module 04 โ€” are they the same underlying problem?
  3. The Vision Transformer uses global self-attention from the first layer, while a CNN only builds global receptive fields gradually through many layers. For the specific task of detecting camouflaged military vehicles from aerial imagery โ€” where context (nearby roads, tyre tracks, shadows) is as important as local appearance โ€” derive an argument from the attention mechanism's mathematical form for why a ViT might detect camouflaged targets more reliably than a CNN of similar parameter count. What is the computational cost trade-off on a Jetson Orin Nano?
โ—‚ Module 06: ML Foundations Module 08: Autonomy โ–ธ