13 min read #homelab

On this page

Server Lifecycle: Onboarding and Removal

A set of scripts compresses "buy VPS → metrics appear on Grafana dashboard" into a single command. Onboarding = bootstrap (deploy public key/set hostname/SSH config) → issue mTLS certificates → deploy otel-collector → configure DIRECT rules → verify completion; Removal = stop service → revoke certificates → clear rules. The entire process is idempotent, with independent checks at each step, allowing retries on failure. This chapter details the complete logic and sanitized source code of the scripts.

1. Usage

# IP Mode (New VPS, requires only IP + root password)
./add-server.sh <IP> --region cn|us|jp|eu|sg [--port N] [--dry-run]

# Alias Mode (Existing SSH config + passwordless login)
./add-server.sh <ssh-alias> [display-name] [--dry-run]

# Removal
./remove-server.sh <ssh-alias> [fleet-name] [--purge] [--dry-run]

Prerequisites: A local CA has been established, the rathole relay is ready, and DNS otlp.liz6.com is resolvable. The script uses set -euo pipefail, stopping immediately on any failure. Re-running automatically skips completed steps.

2. add-server.sh: From Bare IP to Online

The script operates in two phases: bootstrap (IP mode only, converting a bare IP into an SSH alias) → monitoring onboarding (shared 6 steps for both IP and alias modes).

2.1 Bootstrap: Bare IP Pre-processing

When the input is an IP, it first executes 5 small steps to bring the remote machine under SSH management:

① Region Detection: curl ip-api.com → maps countryCode: CN→cn, JP→jp, US→us, SG→sg, etc. If this fails, it prompts to manually specify --region.

② Automatic Naming: Scans ~/.ssh/config for Host li-<region>-node-<N> entries, taking max(N)+1 to generate a new name. First checks if the IP already has an alias → if so, reuses it; if the same Host points to the same IP → skips; if it points to a different IP → warns.

③ Deploy Public Key: ssh -o PasswordAuthentication=no root@<IP> echo ok → if passwordless login already exists, skips; otherwise runs ssh-copy-id.

④ Set Hostname: ssh root@<IP> hostname checks the current value → if it's already the target name, skips; otherwise runs hostnamectl set-hostname.

⑤ Write SSH Config + Fingerprint Confirmation: Appends a Host block (including StrictHostKeyChecking accept-new). The first connection requires manual fingerprint confirmation.

_assign_name() {
  local region="$1" ip="$2"
  # First check if this IP already has an alias
  local existing; existing=$(grep -B5 "$ip" ~/.ssh/config 2>/dev/null \
    | grep '^Host ' | grep -v 'HostName' | sed 's/Host //' | head -1 || true)
  if [[ -n "$existing" ]]; then echo "$existing"; return; fi
  # Automatically assign li-<region>-node-<N>
  local max_n; max_n=$(grep -oP "Host li-${region}-node-\K\d+" ~/.ssh/config 2>/dev/null \
    | sort -n | tail -1 || echo 0)
  echo "li-${region}-node-$((max_n + 1))"
}

_deploy_key() {
  if ssh -o ConnectTimeout=8 -o PasswordAuthentication=no -p "$PORT" "root@$IP" \
    'echo ok' 2>/dev/null | grep -q ok; then
    echo "Passwordless login already exists, skipping"; return
  fi
  ssh-copy-id -o ConnectTimeout=15 -p "$PORT" "root@$IP"
}

_set_hostname() {
  local cur; cur=$(ssh root@$IP hostname 2>/dev/null | tr -d '\r\n' || true)
  if [[ "$cur" == "$1" ]]; then echo "Already $1, skipping"; return; fi
  ssh root@$IP "hostnamectl set-hostname '$1'"
}

2.2 Monitoring Onboarding: 6 Steps (Shared by IP and Alias Modes)

set -euo pipefail, each step is idempotent. The script header first determines the input type:

_is_ip() { [[ "$1" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; }
if _is_ip "$INPUT"; then
  BOOTSTRAP=1                            # Run bootstrap first
else
  HOST="$INPUT"; BOOTSTRAP=0             # Skip bootstrap, go directly to onboarding
fi

1/6 Issue Client Certificate

CERTD="$CA_DIR/out/clients/$FLEET_NAME"
if [[ -d "$CERTD" ]]; then
  echo "Certificate already exists, skipping"
else
  "$CA_DIR/issue-client-cert.sh" "$FLEET_NAME"  # EC P-256, clientAuth, 825 days
fi

2/6 Detect Remote Architecture + Download Binary

ARCH=$(ssh $HOST uname -m)
case "$ARCH" in
  x86_64|amd64) GOARCH=amd64;;
  aarch64|arm64) GOARCH=arm64;;
esac
# Download otelcol-contrib from GitHub release → ~/distfiles/ → scp to remote
# Local ~/distfiles/ cache, no re-download if version hasn't changed

3/6 Mihomo DIRECT Rules (Local Machine + N100)

Critical defensive logic—the remote VPS's public IP must be added to the mihomo DIRECT rules on both the local machine and the N100, otherwise SSH and OTLP bearer connections will traverse the TUN proxy:

# Resolve: SSH alias → HostName → getent resolves real IP
resolve_host_ip() {
  local hostname; hostname=$(ssh -G "$1" 2>/dev/null | awk '/^hostname / {print $2; exit}')
  # If HostName is a domain, resolve via getent
  if [[ "$hostname" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "$hostname"; return
  fi
  getent ahosts "$hostname" 2>/dev/null | awk '{print $1; exit}'
}

# Check if public IP (internal segments are already covered by segment rules, no redundant /32 needed)
is_public_ip() {
  local ip="$1"
  [[ "$ip" =~ ^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.|127\.|100\.(6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])\.|169\.254\.|224\.) ]] && return 1
  return 0
}

mihomo_add_direct() {
  local ip="$1" desc="$2"
  # Local: grep check → if exists, skip; otherwise sed insert + API hot reload
  if ! grep -qF "${ip}/32" "$DESKTOP_MIHOMO_CONFIG" 2>/dev/null; then
    sed -i "/255\.255\.255\.255\/32,DIRECT,no-resolve/a\  - IP-CIDR,${ip}/32,DIRECT,no-resolve   # $desc" \
      "$DESKTOP_MIHOMO_CONFIG"
    curl -sf -X PUT "http://127.0.0.1:9999/configs?force=true" \
      -H "Authorization: Bearer ${DESKTOP_MIHOMO_SECRET}" \
      -d "{\"path\":\"${DESKTOP_MIHOMO_CONFIG}\"}" >/dev/null
  fi
  # N100: ssh + grep → if exists, skip; otherwise ssh sed + nikki reload
  if ! ssh $N100_HOST "grep -qF '${ip}/32' $N100_PROFILE" 2>/dev/null; then
    ssh $N100_HOST "sed -i '/255\.255\.255\.255\/32,DIRECT,no-resolve/a\  - IP-CIDR,${ip}/32,DIRECT,no-resolve   # $desc' \
      $N100_PROFILE && /etc/init.d/nikki reload" 2>/dev/null || \
      echo "[!] N100 unreachable, please add manually later"
  fi
}

Each side performs an idempotent check: if it already exists, skip; if not, sed inserts it below the 255.255.255.255/32,DIRECT policy (keeping rule position consistent), then performs an API hot reload without restarting the service.

4/6 Push Binary + Config + Certificates

scp "$BIN" "$HOST:/usr/local/bin/otelcol-contrib"
scp "$TEMPLATE_DIR/config.yaml" "$HOST:/etc/otelcol-fleet/config.yaml"
scp "$CERTD/client.crt" "$CERTD/client.key" "$CERTD/ca.crt" "$HOST:/etc/otelcol-fleet/"
ssh $HOST 'chmod 0600 /etc/otelcol-fleet/client.key'

# Write env: inject three variables into config
ssh $HOST "cat > /etc/otelcol-fleet/env" <<EOF
FLEET_HOSTNAME=$FLEET_NAME
FLEET_OTLP_ENDPOINT=$OTLP_ENDPOINT
FLEET_COLLECT_INTERVAL=$FLEET_COLLECT_INTERVAL
EOF

config.yaml is a template, injecting endpoint/hostname via env—multiple deployments use the same template + different envs.

5/6 Install systemd and Start

scp "$TEMPLATE_DIR/otelcol-fleet.service" "$HOST:/etc/systemd/system/"
ssh $HOST 'systemctl daemon-reload && systemctl enable --now otelcol-fleet'

Unit design: host_metrics needs to read /proc//sys, User=root is the easiest; Restart=on-failure + RestartSec=5; RuntimeDirectory+StateDirectory automatically create directories.

6/6 Verification

sleep 35  # Collection interval 30s + batch buffer 5s
N=$(curl -s "http://127.0.0.1:9090/api/v1/query" \
    --data-urlencode "query=count({host_name=\"$FLEET_NAME\"})" \
    | grep -oP '"value":\[[^]]*,"\K[0-9]+' | head -1 || true)
if [[ -n "$N" && "$N" -gt 0 ]]; then
  echo "[+] Done! Prometheus received $N series. Grafana dropdown will automatically show $FLEET_NAME."
else
  echo "[!] No metrics received, troubleshoot: ssh $HOST journalctl -u otelcol-fleet -n50"
fi

2.3 Common Failures

Step	Symptom	Cause	Solution
2/6	SSH timeout	Mihomo exit temporarily switches to overseas routing	Wait a few seconds and retry
5/6	active(auto-restart)	Queue directory not created	Fixed: `create_directory:true`
6/6	active but series=0	TLS handshake failure / DNS unreachable	`journalctl -u otelcol-fleet` to check OTLP errors

3. remove-server.sh: Complete Removal

# Stop service + delete config → revoke certificate → clear DIRECT → clear SSH config → done
./remove-server.sh <ssh-host> [fleet-name] [--purge]

--purge additionally deletes remote binary + on-disk queue. If fleet-name is not passed, it reverse-queries from the HostName in SSH config.

1/4 Stop Service + Delete Config

ssh $HOST 'systemctl disable --now otelcol-fleet 2>/dev/null || true
rm -f /etc/systemd/system/otelcol-fleet.service
rm -rf /etc/otelcol-fleet
systemctl daemon-reload'
# --purge: additionally rm /usr/local/bin/otelcol-contrib + /var/lib/otelcol-fleet

2/4 Revoke Certificate

# Append CN to ca/out/revoked.txt (deduplicate)
# sync-revoked.sh: generates "abort CN <name>" line by line → revoked.caddy
# systemctl reload caddy → takes effect immediately, 403 even if certificate hasn't expired
"$CA_DIR/revoke-client-cert.sh" "$NAME"

3/4 Clear DIRECT Rules (Idempotent on Both Sides)

mihomo_remove_direct() {
  local ip="$1"
  # Local: grep → if not exists, skip; if exists, sed delete line + API hot reload
  grep -qF "${ip}/32" "$DESKTOP_MIHOMO_CONFIG" 2>/dev/null || return
  sed -i "/${ip//./\\.}\\/32.*DIRECT,no-resolve/d" "$DESKTOP_MIHOMO_CONFIG"
  curl -sf -X PUT "http://127.0.0.1:9999/configs?force=true" \
    -H "Authorization: Bearer ${DESKTOP_MIHOMO_SECRET}" \
    -d "{\"path\":\"${DESKTOP_MIHOMO_CONFIG}\"}" >/dev/null
  # N100 same logic, ssh grep+sed+reload
}

4/4 Clean SSH Records

# ~/.ssh/config: delete Host <name> block (including indented sub-lines)
sed -i '/^Host '"$name"'$/,/^[^[:space:]]/d' ~/.ssh/config
# known_hosts: ssh-keygen -R <name> + ssh-keygen -R <ip>

Prometheus Historical Data

Existing series naturally expire due to staleness (5 minutes), disappearing automatically from the Grafana dropdown. To delete immediately, enable the admin API and POST /api/v1/admin/tsdb/delete_series.

4. Supporting Files

otelcol-fleet.service

[Unit]
Description=OpenTelemetry Collector (metrics-fleet agent)
After=network-online.target

[Service]
Type=simple
EnvironmentFile=/etc/otelcol-fleet/env
ExecStart=/usr/local/bin/otelcol-contrib --config /etc/otelcol-fleet/config.yaml
Restart=on-failure
RestartSec=5
User=root
RuntimeDirectory=otelcol-fleet
StateDirectory=otelcol-fleet

[Install]
WantedBy=multi-user.target

host_metrics needs to read /proc//sys, User=root is the easiest; if privilege reduction is needed, CAP_SYS_PTRACE+CAP_DAC_READ_SEARCH must be configured additionally.

Naming Convention

Prefix	Region	Number
`li-home-`	Home	`li-home-0`(Local Machine),`li-home-1`(Windows),`li-home-router`(N100 Sidecar)
`li-cn-node-`	China	N
`li-us-node-`	USA	N
`li-jp-node-`	Japan	N

Current Coverage: 3 at Home (Local Machine/Windows/N100), 2 in China (Old Relay + Tencent Cloud), 3 in USA (VPS), 1 in Japan.

5. Appendix: Complete Scripts

add-server.sh

#!/usr/bin/env bash
# Onboard a server to the monitoring fleet (push model: remote runs otel-collector hostmetrics, mTLS pushes back home).
# Supports full process from bare IP, or from existing SSH alias. All steps are idempotent: completed ones are silently skipped.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
cd "$HERE"
source ./fleet.env
source ./mihomo-sync.sh

# ── Parameters ──
DRY_RUN=0; PORT=22; REGION="auto"
INPUT=""; FLEET_NAME=""
for a in "$@"; do
  case "$a" in
    --port)   shift; PORT="$1";;
    --region) shift; REGION="$1";;
    --dry-run) DRY_RUN=1;;
    -*) echo "[!] Unknown: $a" >&2; exit 1;;
    *) [[ -z "$INPUT" ]] && INPUT="$a" || FLEET_NAME="$a";;
  esac
done
[[ -n "$INPUT" ]] || { echo "Usage: add-server.sh <IP|ssh-alias> [display-name] [--port N] [--region auto|cn|us|jp]" >&2; exit 1; }

# ── Input Type: IP or SSH Alias ──
_is_ip() { [[ "$1" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; }
if _is_ip "$INPUT"; then
  IP="$INPUT"; BOOTSTRAP=1
else
  HOST="$INPUT"; BOOTSTRAP=0
fi
FLEET_NAME="${FLEET_NAME:-$INPUT}"

# ── Helper: Idempotent Public Key Deployment ──
_deploy_key() {
  echo "==> Deploying Public Key"
  if ssh -o ConnectTimeout=8 -o PasswordAuthentication=no -p "$PORT" "root@$IP" \
    'echo ok' 2>/dev/null | grep -q ok; then
    echo "    Passwordless login already exists, skipping"; return
  fi
  ssh-copy-id -o ConnectTimeout=15 -p "$PORT" "root@$IP"
  echo "    Public Key Deployed"
}

# ── Helper: Idempotent Hostname Setting ──
_set_hostname() {
  local name="$1"
  echo "==> Setting Remote Hostname: $name"
  local cur
  cur=$(ssh -o ConnectTimeout=10 -p "$PORT" "root@$IP" 'hostname' 2>/dev/null | tr -d '\r\n' || true)
  if [[ "$cur" == "$name" ]]; then echo "    Already $name, skipping"; return; fi
  ssh -o ConnectTimeout=10 -p "$PORT" "root@$IP" "hostnamectl set-hostname '$name'"
  echo "    Set to $name"
}

# ── Helper: Idempotent SSH Config Writing ──
_add_ssh_config() {
  local name="$1" ip="$2" port="$3"
  echo "==> Syncing ~/.ssh/config"
  if grep -q "^Host $name$" ~/.ssh/config 2>/dev/null; then
    local existing_ip; existing_ip=$(grep -A5 "^Host $name$" ~/.ssh/config | awk '/HostName / {print $2; exit}')
    if [[ "$existing_ip" == "$ip" ]]; then
      echo "    Host $name → $ip already exists, skipping"; return
    else
      echo "    [!] Host $name exists but points to $existing_ip (not $ip), skipping write"; return
    fi
  fi
  cat >> ~/.ssh/config <<EOF

# metrics-fleet $(date +%Y-%m-%d)
Host $name
    HostName $ip
    Port $port
    User root
    IdentityFile ~/.ssh/id_ed25519
    StrictHostKeyChecking accept-new
EOF
  echo "    Appended Host $name ($ip:$port)"
}

# ── Helper: Automatic Naming Allocation ──
_assign_name() {
  local region="$1" ip="$2"
  local existing; existing=$(grep -B5 "$ip" ~/.ssh/config 2>/dev/null \
    | grep '^Host ' | grep -v 'HostName' | sed 's/Host //' | head -1 || true)
  if [[ -n "$existing" ]]; then echo "$existing"; return; fi
  local max_n; max_n=$(grep -oP "Host li-${region}-node-\K\d+" ~/.ssh/config 2>/dev/null \
    | sort -n | tail -1 || echo 0)
  echo "li-${region}-node-$((max_n + 1))"
}

# ═══ Bootstrap: IP Input Only ═══
if [[ "$BOOTSTRAP" == 1 ]]; then
  echo "=== Bootstrap: Starting from Bare IP ==="
  [[ "$REGION" != "auto" ]] || { echo "[!] IP Mode requires specifying --region" >&2; exit 1; }
  HOST=$(_assign_name "$REGION" "$IP")
  echo "    Region:$REGION | Alias:$HOST"
  FLEET_NAME="${FLEET_NAME:-$HOST}"
  _deploy_key
  _set_hostname "$HOST"
  _add_ssh_config "$HOST" "$IP" "$PORT"
  ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 "$HOST" 'echo ok' >/dev/null
else
  echo "=== SSH Alias Mode:$HOST ==="
  FLEET_NAME="${FLEET_NAME:-$HOST}"
fi

if [[ "$DRY_RUN" == 1 ]]; then
  echo; echo "[dry-run] Will execute: Issue Certificate + Detect Architecture + DIRECT + Push + systemd + Verify"
  exit 0
fi

# ═══ Monitoring Onboarding 6 Steps (Idempotent) ═══
SSH="ssh -o ConnectTimeout=10 $HOST"
OTLP_ENDPOINT="https://${RELAY_HOST}:${RELAY_PORT}${OTLP_PATH}"

echo "==> [1/6] Issuing Client Certificate:$FLEET_NAME"
CERTD="$CA_DIR/out/clients/$FLEET_NAME"
if [[ -d "$CERTD" ]]; then echo "    Certificate already exists, skipping"
else "$CA_DIR/issue-client-cert.sh" "$FLEET_NAME"; fi

echo "==> [2/6] Detecting Remote Architecture"
ARCH="$($SSH uname -m)"
case "$ARCH" in
  x86_64|amd64) GOARCH=amd64;;
  aarch64|arm64) GOARCH=arm64;;
  *) echo "[!] Unsupported Architecture:$ARCH" >&2; exit 1;;
esac

echo "==> [3/6] Mihomo DIRECT Rules (Local Machine + N100)"
REMOTE_IP=$(resolve_host_ip "$HOST")
if [[ -n "$REMOTE_IP" ]] && is_public_ip "$REMOTE_IP"; then
  mihomo_add_direct "$REMOTE_IP" "metrics-fleet:$FLEET_NAME"
else
  echo "    Internal Network/Unresolvable, skipping DIRECT"
fi

echo "==> [4/6] Pushing Binary + Config + Certificates"
DISTFILES="$HOME/distfiles"
BIN="$DISTFILES/otelcol-contrib_${OTELCOL_VERSION}_linux_${GOARCH}"
if [[ ! -x "$BIN" ]]; then
  mkdir -p "$DISTFILES"
  TARURL="https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/otelcol-contrib_${OTELCOL_VERSION}_linux_${GOARCH}.tar.gz"
  tmp="$(mktemp -d)"
  curl -fSL "$TARURL" -o "$tmp/o.tgz"
  tar -xzf "$tmp/o.tgz" -C "$tmp" otelcol-contrib
  mv "$tmp/otelcol-contrib" "$BIN"
  chmod +x "$BIN"; rm -rf "$tmp"
fi
$SSH 'install -d -m 0750 /etc/otelcol-fleet'
scp -q "$BIN" "$HOST:/usr/local/bin/otelcol-contrib"
scp -q "$HERE/../otel/config.yaml" "$HOST:/etc/otelcol-fleet/config.yaml"
scp -q "$CERTD/client.crt" "$CERTD/client.key" "$CERTD/ca.crt" "$HOST:/etc/otelcol-fleet/"
$SSH 'chmod 0600 /etc/otelcol-fleet/client.key'
$SSH "cat > /etc/otelcol-fleet/env" <<EOF
FLEET_HOSTNAME=$FLEET_NAME
FLEET_OTLP_ENDPOINT=$OTLP_ENDPOINT
FLEET_COLLECT_INTERVAL=$FLEET_COLLECT_INTERVAL
EOF

echo "==> [5/6] Installing systemd Service and Starting"
scp -q "$HERE/../otel/otelcol-fleet.service" "$HOST:/etc/systemd/system/otelcol-fleet.service"
$SSH 'systemctl daemon-reload && systemctl enable --now otelcol-fleet'

echo "==> [6/6] Verification"
sleep 3
if $SSH 'systemctl is-active --quiet otelcol-fleet'; then echo "    Remote Service: active"
else echo "[!] Not started: ssh $HOST journalctl -u otelcol-fleet -n50" >&2; exit 1; fi
echo "    Waiting for metrics to arrive..."
sleep 35
N="$(curl -s "http://127.0.0.1:9090/api/v1/query" \
  --data-urlencode "query=count({host_name=\"$FLEET_NAME\"})" \
  | grep -oP '"value":\[[^]]*,"\K[0-9]+' | head -1 || true)"
if [[ -n "${N:-}" && "$N" -gt 0 ]]; then
  echo "[+] Done! Prometheus received $N series. Grafana dropdown will automatically show $FLEET_NAME."
else
  echo "[!] Remote started but no metrics received. Troubleshoot: ssh $HOST journalctl -u otelcol-fleet -n50"
fi

remove-server.sh

#!/usr/bin/env bash
# Remove a server: stop collector → revoke certificate → clear DIRECT → clear SSH config → done.
# All steps are idempotent, already cleared ones are silently skipped.
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
cd "$HERE"
source ./fleet.env
source ./mihomo-sync.sh

HOST="${1:?Usage: remove-server.sh <ssh-host> [fleet-name] [--purge] [--dry-run]}"
NAME="$HOST"; PURGE=0; DRY_RUN=0
shift
for a in "$@"; do
  case "$a" in
    --purge) PURGE=1;;
    --dry-run) DRY_RUN=1;;
    *) NAME="$a";;
  esac
done
SSH="ssh -o ConnectTimeout=10 $HOST"
IP=$(resolve_host_ip "$HOST")

# ── Helper: Clean SSH Config ──
_remove_ssh_config() {
  local name="$1"
  if ! grep -q "^Host $name$" ~/.ssh/config 2>/dev/null; then
    echo "    ~/.ssh/config: Host $name not found, skipping"; return
  fi
  sed -i '/^Host '"$name"'$/,/^[^[:space:]]/{/^Host '"$name"'$/d;/^[[:space:]]/d}' ~/.ssh/config
  echo "    Removed Host $name from ~/.ssh/config"
}

# ═══ Main Flow ═══
echo "=== Removing Server:$NAME ($HOST) ==="

echo "==> [1/4] Stopping and Uninstalling Remote Collector"
if [[ "$DRY_RUN" == 0 ]]; then
  $SSH 'systemctl disable --now otelcol-fleet 2>/dev/null || true
  rm -f /etc/systemd/system/otelcol-fleet.service
  rm -rf /etc/otelcol-fleet
  systemctl daemon-reload' 2>/dev/null || true
  echo "    Collector stopped and config uninstalled"
  if [[ "$PURGE" == 1 ]]; then
    $SSH 'rm -f /usr/local/bin/otelcol-contrib; rm -rf /var/lib/otelcol-fleet' 2>/dev/null || true
    echo "    Purged binary and queue"
  fi
fi

echo "==> [2/4] Revoking Client Certificate:$NAME"
[[ "$DRY_RUN" == 0 ]] && "$CA_DIR/revoke-client-cert.sh" "$NAME"

echo "==> [3/4] Mihomo DIRECT Rule Removal"
if [[ -n "$IP" ]] && is_public_ip "$IP"; then
  mihomo_remove_direct "$IP"
else
  echo "    Internal Network/Unresolvable, skipping"
fi

echo "==> [4/4] Cleaning SSH Records"
_remove_ssh_config "$NAME"
ssh-keygen -R "$NAME" 2>/dev/null || true

echo "[+] $NAME completely removed (collector+certificate+DIRECT+SSH records)."
echo "    Existing metrics will naturally expire due to staleness. To delete immediately: Prometheus admin API delete_series."

mihomo-sync.sh (DIRECT Rule Sync Helper)

#!/usr/bin/env bash
# Mihomo DIRECT IP Rule Sync: Local Machine + N100 Sidecar.
# Sourced by add-server.sh / remove-server.sh. Idempotent.

# Resolve SSH Alias → IP
resolve_host_ip() {
  local hostname; hostname=$(ssh -G "$1" 2>/dev/null | awk '/^hostname / {print $2; exit}')
  [[ -z "$hostname" ]] && { echo ""; return; }
  if [[ "$hostname" =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
    echo "$hostname"; return
  fi
  getent ahosts "$hostname" 2>/dev/null | awk '{print $1; exit}'
}

# Check if public IP
is_public_ip() {
  local ip="$1"
  [[ -z "$ip" ]] && return 1
  [[ "$ip" =~ ^(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.|127\.|100\.(6[4-9]|[7-9][0-9]|1[01][0-9]|12[0-7])\.|169\.254\.|224\.) ]] && return 1
  return 0
}

# Add DIRECT Rule (Idempotent on Both Sides)
mihomo_add_direct() {
  local ip="$1" desc="${2:-}"
  local comment=""; [[ -n "$desc" ]] && comment="   # $desc"
  local line="  - IP-CIDR,${ip}/32,DIRECT,no-resolve${comment}"
  # Local Machine
  if ! grep -qF "${ip}/32" "$DESKTOP_MIHOMO_CONFIG" 2>/dev/null; then
    echo "$SUDO_PASSWORD" | sudo -S sed -i \
      "/255\.255\.255\.255\/32,DIRECT,no-resolve/a\\${line}" "$DESKTOP_MIHOMO_CONFIG"
    curl -sf -X PUT "http://127.0.0.1:9999/configs?force=true" \
      -H "Authorization: Bearer ${DESKTOP_MIHOMO_SECRET}" \
      -d "{\"path\":\"${DESKTOP_MIHOMO_CONFIG}\"}" >/dev/null \
      && echo "    Local: Added ${ip}/32 DIRECT" || echo "    Local: Config written (hot reload failed)"
  else echo "    Local: ${ip}/32 already exists, skipping"; fi
  # N100
  if ! ssh -o ConnectTimeout=8 "$N100_HOST" "grep -qF '${ip}/32' $N100_PROFILE" 2>/dev/null; then
    ssh -o ConnectTimeout=8 "$N100_HOST" \
      "sed -i \"/255\\.255\\.255\\.255\\/32,DIRECT,no-resolve/a\\${line}\" $N100_PROFILE \
       && (/etc/init.d/nikki reload 2>/dev/null || true)" 2>/dev/null \
      && echo "    N100: Added ${ip}/32 DIRECT" || echo "    [!] N100 unreachable, please add manually later"
  else echo "    N100: ${ip}/32 already exists, skipping"; fi
}

# Delete DIRECT Rule (Idempotent on Both Sides)
mihomo_remove_direct() {
  local ip="$1"; local escaped; escaped=$(echo "$ip" | sed 's/\./\\./g')
  # Local Machine
  if grep -qF "${ip}/32" "$DESKTOP_MIHOMO_CONFIG" 2>/dev/null; then
    echo "$SUDO_PASSWORD" | sudo -S sed -i "/${escaped}\\/32.*DIRECT,no-resolve/d" "$DESKTOP_MIHOMO_CONFIG"
    curl -sf -X PUT "http://127.0.0.1:9999/configs?force=true" \
      -H "Authorization: Bearer ${DESKTOP_MIHOMO_SECRET}" \
      -d "{\"path\":\"${DESKTOP_MIHOMO_CONFIG}\"}" >/dev/null \
      && echo "    Local: Deleted ${ip}/32" || echo "    Local: Config written (hot reload failed)"
  fi
  # N100
  if ssh -o ConnectTimeout=8 "$N100_HOST" "grep -qF '${ip}/32' $N100_PROFILE" 2>/dev/null; then
    ssh -o ConnectTimeout=8 "$N100_HOST" \
      "sed -i \"/${escaped}\\/32.*DIRECT,no-resolve/d\" $N100_PROFILE \
       && (/etc/init.d/nikki reload 2>/dev/null || true)" 2>/dev/null \
      && echo "    N100: Deleted ${ip}/32" || echo "    [!] N100 unreachable, please delete manually later"
  fi
}

monitoring.md — Monitoring Architecture (Detailed explanation of metrics-fleet remote push path)
network-architecture.md — Network Topology and nftables Structure
rathole-tunnel.md — Principle of Tunnel Bearer Connections Bypassing Mihomo DIRECT Rules