How to make OTA firmware updates fail-safe on devices you cannot reach
A practical design for ceiling-mounted, remote and field devices: A/B slots, rollback, watchdogs and staged rollouts.
Short answer: to make updates fail-safe on a device you cannot reach, never overwrite the firmware that is running. Write the new image to a second slot (A/B), verify its signature, switch the boot pointer only when it is complete, let the new firmware prove itself with a self-test, and have the bootloader roll back automatically if it fails. Add a hardware watchdog, a recovery image and staged rollouts, and a bad update becomes an inconvenience instead of a truck roll.
Why ceiling-mounted and remote devices need extra care
Sensors on ceilings, lights, gateways in plant rooms and devices in the field are expensive to reach. A failed update can mean a ladder, a lift, a site visit or a recall. The goal is simple: no update, network drop or power cut should ever leave the device without bootable firmware.
The 7 layers of a fail-safe OTA design
- A/B (dual-bank) partitions: the device runs from slot A while slot B receives the update. The old image stays untouched until the new one is proven.
- Integrity and signature checks: verify a hash of the full image and a cryptographic signature before marking it bootable. Reject partial or tampered downloads.
- Atomic switch: change the boot selection in one small final write, only after the image is fully written and verified. A power cut before that point simply leaves the old image active.
- Self-test and confirm: on first boot the new firmware checks what matters (sensors respond, network connects, the cloud acknowledges) and only then confirms itself as valid.
- Automatic rollback: if the new image crashes, hangs or never confirms, the bootloader reverts to the previous image on the next reset.
- Hardware watchdog: an independent watchdog resets a frozen device so rollback can happen without anyone touching it.
- Recovery path: a small factory or recovery image that can always connect and fetch a good build, as a last resort.
How this works on common platforms
| Platform | Built-in mechanism |
|---|---|
| ESP32 (ESP-IDF) | Two OTA app partitions plus app rollback; new images start in a pending-verify state and confirm with esp_ota_mark_app_valid_cancel_rollback(). Secure boot and flash encryption protect images. |
| nRF / Zephyr (MCUboot) | Primary and secondary slots with swap and revert: a test image must be confirmed, otherwise MCUboot swaps back on the next boot. |
| Embedded Linux | A/B root filesystems with tools such as RAUC, Mender or SWUpdate, plus a boot counter in the bootloader to fall back after failed boots. |
Rollout practices that prevent fleet-wide failures
- Staged rollouts: release to around 1% of devices, then 10%, then everyone, and pause automatically if error rates rise.
- Health reporting: every device reports version, boot reason and rollback events after an update.
- Update conditions: only install when power and connectivity are stable, and resume interrupted downloads instead of restarting them.
- Anti-rollback: keep a security version so known-vulnerable firmware cannot be reinstalled.
- Bench testing: deliberately cut power and network mid-update on test units before every release.
How SixSol builds OTA for clients
We design firmware and connected devices with A/B updates, signed images, self-test and rollback from day one, mainly on ESP32 and Nordic nRF chips. See examples such as nRF device telemetry and our industrial IoT controller.
Frequently asked questions
How do you make firmware updates fail-safe?
Use two firmware slots (A/B). Download the new image into the inactive slot, verify its signature and checksum, then switch the boot pointer only after it is fully written. On first boot the new firmware runs a self-test and marks itself valid; if it crashes, hangs or fails the test, the bootloader automatically rolls back to the previous image.
What happens if power is lost during an OTA update?
With an A/B layout nothing breaks: the running firmware is never overwritten, and the boot pointer only changes after the new image is complete and verified. The device keeps booting the old firmware and retries the download later.
Does ESP32 support OTA rollback?
Yes. ESP-IDF supports two OTA app partitions and app rollback. With rollback enabled, a new image boots in a pending-verify state and must call esp_ota_mark_app_valid_cancel_rollback() after its self-test; if it reboots before doing so, the bootloader returns to the previous image.
How do you update devices with no physical access?
Combine A/B updates with automatic rollback, signed images, a hardware watchdog, a minimal recovery or factory image, health reporting after every update, and staged rollouts that start with a small percentage of devices.
Should OTA updates be signed?
Yes. Sign every image and verify the signature in the bootloader before it runs, for example with secure boot. Add anti-rollback so attackers cannot reinstall an older vulnerable version.