Real-Time FPGA Camera & Vision System

✓ Completed

• NIOS II/f
• Intel Quartus Prime
• DE10-Lite FPGA
• ESP32-CAM
• C
• VGA
• SPI

Designed and built a real-time camera and image processing system on an Intel DE10-Lite FPGA board, integrating an ESP32-CAM module to capture live video and stream it to a VGA monitor with on-the-fly software-defined image processing. The system operates as a multi-processor architecture — three cooperating NIOS II/f soft-core processors handle camera capture, image processing, and VGA display in parallel, sharing a common SDRAM frame buffer over a high-bandwidth interconnect.

Built as part of Monash University ECE3073 in collaboration with Tom Ge, Lim Jian Sheng, and Darcy Elliott. All image processing algorithms were implemented in C running on the NIOS II processors. The design supports both single-view and quad-view display modes, switch-controlled filter selection, and accelerometer-driven display rotation — all updating in real time on a live VGA output.

---

Key Features

---

Display Modes

Single View Mode

The full VGA output shows the camera feed processed through one selected filter. The active filter is chosen using the on-board switches, and the board's accelerometer can be used to rotate the displayed image — tilting left or right adjusts orientation in real time without any software reboot.

Quad View Mode

The VGA display is divided into four equal quadrants. Each quadrant can be independently assigned any of the four filters — including the same filter multiple times — using the FPGA's on-board switches. This allows simultaneous side-by-side comparison of different processing outputs on the same live frame.

---

System Architecture

Base System & Motivation

The original single-processor baseline handled all tasks sequentially on one NIOS II: SPI capture from the ESP32-CAM, image filtering, and VGA display. This serialised pipeline was the fundamental bottleneck — the processor could not begin capturing the next frame until the current one had been fully filtered and displayed. Measured performance on the base system peaked at just 1.51 FPS for unfiltered greyscale, dropping to 0.58 FPS for blur (a convolution-heavy operation) and 0.8 FPS for edge detection.

3-Processor Parallel Design

The multiprocessor system assigns each stage of the pipeline to a dedicated NIOS II/f soft-core processor, enabling simultaneous capture, filtering, and display:

• Processor 0 (Coordinator/Display) — manages overall control flow by monitoring PIO flags and signalling task transitions. Drives the VGA controller's pixel buffer from processed frames in SDRAM.
• Processor 1 (Capture) — handles all SPI communication with the ESP32-CAM and ADXL345 accelerometer. Detects mode changes via KEY[1], services double-tap interrupts, packs pixel data into RGB444 format, and writes raw frames into the SDRAM double buffer.
• Processor 2 (Filter) — reads raw frames from SDRAM, applies the selected image processing filter in C (Normal, Flip, Blur, Edge Detection), handles per-quadrant filter selection via SW7:0 in quad mode, and writes the processed result back to SDRAM before signalling Processor 0.

Double-Buffered SDRAM Frame Store

Two raw and two processed frame buffers are maintained in SDRAM. A software buffer index alternates between writing to the inactive buffer and reading from the active one — allowing Processor 1 to capture a new frame simultaneously while Processor 2 filters the previous one and Processor 0 displays it. This eliminated read-write conflicts and frame tearing, improving throughput from ~2.3 FPS (two processors, single buffer) to ~3.5 FPS (two processors, double buffer).

Inter-Processor Synchronisation

Shared SDRAM access is protected by hardware mutexes, ensuring atomic reads and writes with no race conditions. Six PIO-based flags coordinate task handovers between processors — signalling when a frame is ready to be processed and when processing is complete and the frame is ready for display. This model keeps all three processors continuously active without stalling on each other.

Camera Interface — SPI & Packed Pixel Mode

The ESP32-CAM streams pixel data to the FPGA over SPI. To minimise transfer overhead, packed pixel mode is enabled by setting specific bits in the command byte sent to the camera — two pixels are packed per SPI transaction, halving the number of chip-select toggles and descriptor setups per frame. An unpack routine on Processor 1 extracts both pixels and writes 4-bit (greyscale) or 16-bit (RGB) values to SDRAM. This optimisation alone improved single-frame greyscale throughput from ~3.5 FPS to ~5.5 FPS.

RGB Colour & VGA Output

RGB capture is enabled by setting bits 4 and 0 in the camera command byte. The VGA controller was updated to support 12-bit RGB444 (4 bits per channel). Because the SPI protocol operates in pseudo half-duplex — transmitting before receiving — any data arriving during a transmission is discarded; to avoid colour corruption, one frame is intentionally dropped after each mode change. RGB processing triples the per-pixel workload compared to greyscale, which is reflected in the FPS measurements.

---

Performance Results

The table below summarises measured FPS across key development milestones. The base single-processor system is the baseline; Milestone 2 introduces the two-processor split with double buffering; Milestone 3 adds packed pixel transfers and full RGB support with the three-processor architecture.

The final packed greyscale configuration achieved up to a 6× improvement over the base system in single-image mode, and the convolution-heavy blur filter saw its frame rate rise from 0.58 FPS to 2.34 FPS — a 4× gain despite the inherent computational cost of the kernel operation.

---

Image Processing Algorithms

All filters are implemented in C running on Processor 2, operating directly on pixel arrays in SDRAM. Each filter supports both greyscale and RGB444 colour modes — RGB processing carries roughly 3× the per-pixel workload of greyscale, which is visible in the FPS benchmark data.

---

Skills Demonstrated