Network Failure Recovery Manual
On this page
Applicable Environment: Main router 192.168.31.1, N100 side router 192.168.31.7. Clients typically point their gateway and DNS to .7. Restore essential connectivity first, then locate proxy or DNS faults. The Network Architecture describes the normal path, and Monitoring describes the source of alerts.
Configuration Verification: 2026-09-12. The automatic behaviors below are verified against the read-only copies of N100's /usr/local/sbin/nikki-failopen.sh, local wifi-watchdog.service, and actual execution scripts; no network outage drills were performed during this verification. Polling intervals, action durations, and actual outage times are three different metrics.
Determine Which Layer is Broken
On the affected client, first check the current IP, default gateway, and DNS, then probe .1 and .7 separately. Being able to load a page over an already established connection is not sufficient to prove that new connections or DNS are functioning.
| Observation | Next Check | Recovery Path |
|---|---|---|
Both .1 and .7 are unreachable | Local Wi-Fi/Ethernet, IP address, another device, main router power supply | Restore LAN connectivity first |
.1 is reachable, .7 is unreachable | N100 power, switch port, IP address, and local console | Temporarily change client gateway and DNS to .1 |
.7 is reachable, but domain resolution fails | Query .7 and .1 separately; check AdGuard and mihomo logs | Handle according to DNS or proxy fault branches |
| DNS is normal, but new external connections fail | Main router WAN, mihomo routing, target reachability | Compare direct connection and proxy paths |
A ping failure only indicates that this ICMP probe did not receive a reply. Firewalls, address changes, Layer 2 connections, device boot-ups, and system failures can all cause it; "over three minutes" cannot be diagnosed as a kernel crash on its own. Supplement with evidence from another device on the same subnet, SSH, the router's client list, or the local console. See Network Troubleshooting Methodology for details.
N100 Automatic Degradation: Treat Proxy Faults and DNS Faults Separately
The watchdog is managed by procd. After startup, it waits for 60 seconds, then sleeps for 10 seconds after each round of probes; 3 consecutive failures trigger an action. When probes return almost immediately, the stable detection wait time is approximately 20–30 seconds, plus probe and execution overhead. The script only uses curl --connect-timeout 3 and does not set --max-time for the entire request, so it cannot guarantee a switch within 30 seconds. The curl timeout documentation distinguishes between the connection phase and the entire transfer duration.
| Branch | Current Health Criteria | Action After Consecutive Failures | Action After Recovery Determined |
|---|---|---|---|
| mihomo | Localhost :9090 returns HTTP 200, 401, or 403 | Delete table inet nikki; create table inet failopen, perform egress masquerade, and redirect UDP/TCP DNS destined for .7:53 to .1:53 | Delete failopen table, restart nikki to rebuild normal rules, clean up state flags |
| AdGuard (when not in overall proxy degradation) | Localhost :8083 returns 200, or any process is found listening on port 53 | Redirect only UDP/TCP DNS destined for .7:53 to local mihomo 127.0.0.1:1053, keeping proxy rules intact | Delete DNS rescue rules and corresponding state flags |
The second row bypasses AdGuard; the first row is the overall fallback to main router direct connection. API responses or port listening are merely liveness signals and do not prove that upstream DNS, proxy nodes, and actual application requests have recovered. AdGuard's port fallback criteria are relatively broad; another process listening on port 53 might also cause it to be judged as healthy.
Test independent DNS queries and new connections on the client first, then observe the status on the N100. The following commands only read status:
# Log in from the management machine; run the following commands in that SSH session.
The table or state flag may not exist simply because that mode is not enabled. Verify that rules, state flags, and recent logs are consistent, then check actual requests; do not blindly copy and delete table commands. Manually deleting the nikki table while missing TCP DNS, egress SNAT, or recovery state flags may create a second fault.
When Automatic Recovery Fails, Connect One Client Directly First
When the LAN and WAN of .1 are normal, but the N100 cannot recover, record the original settings of one client, then change the default gateway and DNS simultaneously to 192.168.31.1, keeping the same subnet address non-conflicting. Changing only DNS cannot bypass a failed gateway; changing only the gateway may still result in queries to the failed .7.
First verify that this client can resolve domains and establish expected direct connections. Destinations relying on the proxy may not be available in direct mode. After confirmation, change the gateway and DNS in the main router's DHCP settings from .7 to .1, allowing other clients to renew their leases. Existing leases and static configurations will not change immediately upon saving DHCP settings.
If the N100 is accessible but the watchdog is not running, start it after confirming there are no manual rescue rules competing with the service:
# Run on N100; starts health detection and automatic rule changes.
It still has a startup grace period; it is not a command that switches the entire house immediately. When rules and state flags are inconsistent, keep logs and rule outputs, and continue using client direct connection; restore the affected components according to the Architecture Documentation to avoid competition between manual rules and automatic recovery.
After the N100 is fixed, do not use "powered on for two minutes" as the recovery standard. First confirm the proxy, DNS, normal firewall rules, and watchdog logs. Change the test client's gateway/DNS back to .7, and verify LAN, general external networks, and proxy-required targets separately. After passing, restore the main router DHCP settings and check the actual renewal results of clients.
Power Outage and Main Router Failure
The N100 is configured for power-on auto-start and component auto-start. The original estimate of about two minutes cannot cover disk checks, container startup, WAN acquisition, and watchdog grace periods; recovery still follows the verification in the previous section. If .7 is unreachable for a long time, direct connection is the recovery path, and diagnosis requires evidence of power, switch ports, and local console.
If the main router fails, the N100 cannot replace its PPPoE, NAT, DHCP, and Mesh root node responsibilities. First distinguish between "LAN management address unreachable" and "backend reachable but WAN not connected," checking power supply, cabling, WAN status, and upstream optical modem. There is currently no main router hot standby; optical modem bridging does not provide the main router's fault takeover capability.
li-home-0 Wi-Fi: Current Recovery Escalation
The systemd timer is set to OnBootSec=60, OnUnitActiveSec=30, and AccuracySec=5. Each time it probes .1 using ping -c 5 -W 1, the failure count is persisted across executions in /var/tmp/wifi-watchdog:
- When the network interface
wlp9s0exists, the first two connectivity failures only log messages. - On the third failure, it first attempts
nmcli connection up. If recovery succeeds, the count is cleared; if it fails, it continues to driver reloading in the same execution. - Reloading first disables wireless, attempts to unload
mt7925eand related mt76 modules, then loads them and reconnects. Two reloads must be at least 300 seconds apart; if the network interface disappears, it also goes through the reload branch protected by this cooldown. - Subsequent successful probes clear the count. During service execution, the timer expiration will not parallel-start another instance.
You can no longer understand it according to the old table as "reconnect on 1–3 failures, RF on 4–6, reload on 7+," nor can you treat the ~25 seconds for a certain step as the total outage time. Probes, confirmation thresholds, scheduling errors, command timeouts, and cooldowns all affect it.
# Read from the li-home-0 local console or an available management connection.
Only consider manual reconnection or reloading after confirming the main router is normal and the fault is local to the wireless interface. Reloading will disconnect management sessions connected via that wireless interface; prioritize using the local console or wired management. Do not cycle reloads during the cooldown period to "speed up recovery"; instead, check the network interface, firmware, and recent logs.
Criteria for Completing Rollback
Record the client gateway, DNS, DHCP settings, and time before and after adjustments. First, let one client pass tests for resolution, new direct connections, and necessary proxy access before promoting to others. Verify the restoration of the normal path in the same order.
The old .8 side router can only be used as a backup after confirming that proxy services, forwarding, DNS, and rules are still available. Starting the mihomo service does not prove that the entire old path has recovered. There is no need to immediately shut down the N100 for DHCP rollback; keep it available for reading logs and repairs.
Related Entries: Network Architecture, Tunnel Fault Governance, Monitoring Alerts, Networking Learning Path.