Network Failure Recovery Manual

Applies to topology v2.1 (N100 bypass router + CN-IP firewall bypass). This document organizes 5 categories of failure scenarios by severity, providing symptoms, automatic recovery mechanisms, and manual intervention steps for each, with a rollback plan appended at the end. See monitoring.md for alert coverage—the 13 Grafana + Bark rules are responsible for pushing notifications to iPhone when failures occur.

Risk Assessment Overview

Scenario 1: N100 mihomo/AdGuard Process Hang

Automatic Behavior: The fail-open daemon (procd) probes the mihomo API :9090 every 60s. After 3 consecutive failures, it deletes table inet nikki and injects table inet failopen, switching the entire house to direct connection. Once mihomo recovers, it automatically switches back.

Manual Intervention (when the daemon also hangs):

ssh root@192.168.31.7

# Manual fail-open
nft delete table inet nikki
nft add table inet failopen
nft add chain inet failopen srcnat '{ type nat hook postrouting priority srcnat - 1; policy accept; }'
nft add rule inet failopen srcnat ip saddr 192.168.31.0/24 ip daddr != 192.168.31.0/24 masquerade
nft add chain inet failopen dns '{ type nat hook prerouting priority dstnat - 1; policy accept; }'
nft add rule inet failopen dns ip daddr 192.168.31.7 udp dport 53 dnat to 192.168.31.1:53

# After recovery
nft delete table inet failopen
/etc/init.d/nikki restart

table inet bypass_masq requires no handling—it is independent of nikki/failopen. During fail-open, its hairpin SNAT is redundant but non-conflicting. After nikki restarts, china_ip is automatically repopulated.

Scenario 2: N100 Power Loss

Automatic Behavior: Power restored → BIOS power-on self-start → Boot chain: S99nikki (TUN + transparent proxy) → S99nikki-failopen (60s grace period) → docker adguardhome (unless-stopped). Recovery takes approximately 2 minutes, requiring no manual intervention.

Scenario 3: N100 OS Crash (kernel panic/disk failure)

Symptoms: Cannot ping 192.168.31.7 and it does not recover for an extended period. Distinct from power loss—since N100 has power-on self-start, failure to respond for over 3 minutes indicates an OS crash or hardware failure.

Manual Recovery:

1. Open http://192.168.31.1/ in a browser → Log into the Xiaomi backend
2. Common Settings → LAN Settings → Change gateway/DNS from .7 to .1
3. Save, wait for clients to renew leases or restart clients

After N100 Recovery: Confirm startup completion (2min) → Change gateway/DNS back to .7 in the Xiaomi backend.

Scenario 4: li-home-0 WiFi Disconnect

Automatic Behavior: wifi-watchdog (systemd timer, pings .1 every 30s), layered escalation recovery:

Counter resets to zero after recovery. Log: journalctl -u wifi-watchdog.service.

Manual Recovery (when watchdog is ineffective):

sudo nmcli radio wifi off && sleep 2 && sudo nmcli radio wifi on
# Or driver reload
sudo modprobe -r mt7925e && sudo modprobe mt7925e

Long-term: Migrate high-availability services like DERP/rathole to the wired N100 to eliminate WiFi single-point dependency.

Scenario 5: Main Router Hang

ping 192.168.31.1
# If unreachable → Check power/LEDs
# If reachable but no internet → Check PPPoE dial-up status in router backend

PPPoE/NAT/DHCP/Mesh root are all integrated into a single Xiaomi router, with no hot standby. This is the biggest single-point risk for the entire home network—the current mitigation is that the optical modem is in bridge mode without double NAT, so if the main router hangs, at least the optical modem configuration is not lost.

Rollback Plan

When the N100 bypass router has overall issues, revert to direct connection or the old bypass router:

1. Change main router DHCP gateway/DNS back to .1 (direct connection) or .8 (li-home-0 old bypass router)
2. To restore li-home-0 bypass router: sudo systemctl start mihomo
3. Power off N100: ssh root@192.168.31.7 poweroff

Related Documentation

• network-architecture.md — Complete topology and nftables structure
• rathole-tunnel.md — Tunnel solution and failure management
• monitoring.md — Monitoring alerts (Bark, iPhone push)