---
title: Network Failure Recovery Manual
url: https://doc.liz6.com/en/homelab/network-recovery
locale: en
area: homelab
tags:
- homelab
date: 2026-07-01
modified: 2026-07-16
description: Applies to topology v2.1 (N100 bypass router + CN-IP firewall bypass). This document organizes 5 categories of failure scenarios by severity, providing symptoms, automatic recovery mechanisms, and manual intervention steps for each, with a rollback plan appended at the end. See monitoring.md for alert coverage—the 13 Grafana + Bark rules are responsible for pushing notifications to iPhone when failures occur.
---

# Network Failure Recovery Manual

> Applies to topology v2.1 (N100 bypass router + CN-IP firewall bypass). This document organizes 5 categories of failure scenarios by severity, providing symptoms, automatic recovery mechanisms, and manual intervention steps for each, with a rollback plan appended at the end. See [monitoring.md](monitoring.md) for alert coverage—the 13 Grafana + Bark rules are responsible for pushing notifications to iPhone when failures occur.

## Risk Assessment Overview

| # | Risk | Severity | Auto Mitigation | Residual Risk |
|---|------|----------|-----------------|---------------|
| 1 | mihomo/AdGuard process hang | 🔴→🟢 | N100 fail-open daemon, auto-switches to direct connection after 30s | 30s detection window |
| 2 | N100 power loss | 🔴→🟢 | BIOS power-on self-start, all components auto-start, ~2min recovery | No proxy during boot |
| 3 | N100 OS crash (kernel panic/disk failure) | 🟡 | None, manually switch DHCP gateway to .1 | Extremely low probability, but single point of failure |
| 4 | li-home-0 WiFi disconnect | 🔴→🟢 | wifi-watchdog layered self-healing (reconnect → RF → driver reload) | ~25s network outage |
| 5 | Xiaomi main router hang | 🔴 | None | PPPoE/NAT/DHCP/Mesh all in one, no hot standby yet |

## Scenario 1: N100 mihomo/AdGuard Process Hang

**Automatic Behavior**: The fail-open daemon (procd) probes the mihomo API :9090 every 60s. After 3 consecutive failures, it deletes `table inet nikki` and injects `table inet failopen`, switching the entire house to direct connection. Once mihomo recovers, it automatically switches back.

**Manual Intervention** (when the daemon also hangs):

```bash
ssh root@192.168.31.7

# Manual fail-open
nft delete table inet nikki
nft add table inet failopen
nft add chain inet failopen srcnat '{ type nat hook postrouting priority srcnat - 1; policy accept; }'
nft add rule inet failopen srcnat ip saddr 192.168.31.0/24 ip daddr != 192.168.31.0/24 masquerade
nft add chain inet failopen dns '{ type nat hook prerouting priority dstnat - 1; policy accept; }'
nft add rule inet failopen dns ip daddr 192.168.31.7 udp dport 53 dnat to 192.168.31.1:53

# After recovery
nft delete table inet failopen
/etc/init.d/nikki restart
```

> `table inet bypass_masq` requires no handling—it is independent of nikki/failopen. During fail-open, its hairpin SNAT is redundant but non-conflicting. After nikki restarts, china_ip is automatically repopulated.

## Scenario 2: N100 Power Loss

**Automatic Behavior**: Power restored → BIOS power-on self-start → Boot chain: `S99nikki` (TUN + transparent proxy) → `S99nikki-failopen` (60s grace period) → `docker adguardhome` (unless-stopped). Recovery takes approximately 2 minutes, requiring no manual intervention.

## Scenario 3: N100 OS Crash (kernel panic/disk failure)

**Symptoms**: Cannot ping `192.168.31.7` and it does not recover for an extended period. Distinct from power loss—since N100 has power-on self-start, failure to respond for over 3 minutes indicates an OS crash or hardware failure.

**Manual Recovery**:

1. Open `http://192.168.31.1/` in a browser → Log into the Xiaomi backend
2. Common Settings → LAN Settings → Change gateway/DNS from `.7` to `.1`
3. Save, wait for clients to renew leases or restart clients

**After N100 Recovery**: Confirm startup completion (2min) → Change gateway/DNS back to `.7` in the Xiaomi backend.

## Scenario 4: li-home-0 WiFi Disconnect

**Automatic Behavior**: wifi-watchdog (systemd timer, pings .1 every 30s), layered escalation recovery:

| Consecutive Failures | Action | Duration |
|----------------------|--------|----------|
| 1–3 | `nmcli connection down/up` (reconnect) | ~10s |
| 4–6 | `nmcli radio off/on` + rfkill (RF layer) | ~20s |
| 7+ | `modprobe -r mt7925e → modprobe` (driver reload) | ~25s |

Counter resets to zero after recovery. Log: `journalctl -u wifi-watchdog.service`.

**Manual Recovery** (when watchdog is ineffective):

```bash
sudo nmcli radio wifi off && sleep 2 && sudo nmcli radio wifi on
# Or driver reload
sudo modprobe -r mt7925e && sudo modprobe mt7925e
```

> **Long-term**: Migrate high-availability services like DERP/rathole to the wired N100 to eliminate WiFi single-point dependency.

## Scenario 5: Main Router Hang

```bash
ping 192.168.31.1
# If unreachable → Check power/LEDs
# If reachable but no internet → Check PPPoE dial-up status in router backend
```

> PPPoE/NAT/DHCP/Mesh root are all integrated into a single Xiaomi router, with no hot standby. This is the biggest single-point risk for the entire home network—the current mitigation is that the optical modem is in bridge mode without double NAT, so if the main router hangs, at least the optical modem configuration is not lost.

## Rollback Plan

When the N100 bypass router has overall issues, revert to direct connection or the old bypass router:

1. Change main router DHCP gateway/DNS back to `.1` (direct connection) or `.8` (li-home-0 old bypass router)
2. To restore li-home-0 bypass router: `sudo systemctl start mihomo`
3. Power off N100: `ssh root@192.168.31.7 poweroff`

## Related Documentation

- [network-architecture.md](network-architecture.md) — Complete topology and nftables structure
- [rathole-tunnel.md](rathole-tunnel.md) — Tunnel solution and failure management
- [monitoring.md](monitoring.md) — Monitoring alerts (Bark, iPhone push)