Challenge – Overcoming Hardware Lockups and Manual Reboots in a CI/CD Pipeline

When pushing daily firmware updates to hundreds of SBCs, the QA team faced a critical bottleneck: physical hardware lockups. During overnight automated testing, devices would frequently freeze, lock up, or drop off the network.

Because standard commercial USB hubs lack remote power cycling capabilities, a locked-up SBC effectively halted that segment of the testing pipeline. The automated testing suite would time out, and the device would remain offline until a technician arrived the next morning to physically unplug and replug the hardware. This manual intervention defeated the purpose of a “lights out” automated lab, wasting valuable engineering hours, rendering overnight stress tests incomplete, and severely delaying firmware release cycles.

*Image for reference only. Not actual depiction of client or install.

Recent Case Studies

Pennington County Emergency Management

Pennington County Emergency Management thought they had to cobble together an industrial rack mount charger hub for their Emergency Operations Center.
Read More

Solution – Software-Controlled USB Hubs for Centralized Port Management

To achieve a true lights-out testing environment, the QA engineering team required industrial-grade hubs with programmatic per-port power control. They integrated Coolgear’s Managed USB Hubs into their testing racks, connecting the host CI/CD servers directly to the vast array of SBCs.

16 Port Managed USB 3.2 Gen 1 Hub w/ 15KV ESD Surge Protection

Rack Mountable Port Control: The team integrated the 16-Port Managed USB 3.2 Gen 1 Hub to act as the central distribution backbone for their CI/CD servers. Equipped with independent software-based port management and 15kV ESD surge protection, this industrial hub allows engineers to programmatically power-cycle up to 16 locked-up edge devices simultaneously at 5Gbps data speeds without needing to reboot the host PC or manually intervene.

Explore this Solution
CG-16PU3MGD Main Image

Automated Port Recovery via Command-Line Interface To interface with their existing automation software, the team utilized Coolgear’s robust Command-Line Interface (CLI) utilities. Because the software supports Linux (Intel/AMD and ARM architectures) via cusbi and cusba scripts, the team seamlessly integrated port power commands directly into their automated testing bash scripts.

Targeted Per-Port Power Cycling When the CI/CD pipeline detects a frozen Jetson Nano or an unresponsive Raspberry Pi, the testing script automatically executes a command to power-cycle that specific USB port. This hard-reboots the locked-up device instantly, recovering the hardware without affecting the adjacent ports or requiring a shutdown of the host PC.

Simulated Hardware Disconnections Beyond just crash recovery, the team utilized the CLI to programmatically disable and enable individual ports. This allowed them to simulate unexpected physical hardware drops and re-connections, adding a vital layer of edge-case testing to their automated suite that previously had to be done by hand.

Benefits – Achieving 100% Lights-Out Testing and Eliminating Technician Dispatch

The transition to Coolgear’s managed USB hubs transformed a tedious, manual QA bottleneck into a fully automated, self-healing operation. By featuring onboard flash that saves port status across reboots, the hubs eliminate redundant configuration steps and repetitive manual resets. Paired with seamless integration into the lab’s Linux architecture, the testing wall now runs 24/7 without interruption.

Engineers no longer waste their mornings manually resetting frozen boards inside sealed control panels or server racks. By removing the physical hardware bottleneck and giving the CI/CD pipeline total control over device power states, the team dramatically accelerated their firmware release cadence, minimized costly downtime, and maximized the true ROI of their lights-out lab.


Let's Get Your Next Install Dialed In
This entry was posted in . Bookmark the permalink.