rathole + Caddy Intranet Penetration: A Complete Solution from CGNAT to Public HTTPS

Applicable Scenarios: CGNAT or no public IP, have a VPS, only need TCP port forwarding, pursue lightweight zero kernel dependency, do not want to install WireGuard or configure Cloudflare Tunnel.

Use rathole locally—a lightweight TCP tunnel written in Rust, deployed in Docker, paired with Caddy for TLS reverse proxying, to expose Grafana, documentation sites, SSH, and more. This article covers selection, setup, certificate strategies, and three major pitfalls encountered (HTTP/3, conntrack dropping connections, local proxy hijacking the tunnel), providing a reproducible complete configuration.

1. What is rathole, and when to use it

There are many options for intranet penetration; choose based on the scenario:

Solution	Suitable For	Not Suitable For
rathole	Have a VPS, only need TCP port forwarding, lightweight, zero dependencies	Need UDP, P2P direct connection
frp	Same as above + need Web management panel, more protocols	Heavier configuration
Tailscale	Don't want to manage a VPS, only for personal use, accept relay latency	Family/public users need to install client
WireGuard	Have a VPS, want to form a network rather than port mapping	Requires kernel module, needs additional hole punching under CGNAT
Cloudflare Tunnel	Domain hosted on CF, don't want to manage VPS at all	Traffic passes through CF (privacy/compliance), slow in China

I chose rathole for my setup. Reasons: I already have a Tencent Cloud VPS, only need to expose a few HTTP services + SSH from home, no need for UDP/P2P/management panel. rathole is just a TCP tunnel; it does not parse TLS or touch certificates—TLS is handled by Caddy on the VPS.

If your scenario differs from mine: choose Cloudflare Tunnel for CGNAT + no VPS; choose Tailscale for networking; choose frp for UDP. I haven't used these three in depth, so the following only expands on the rathole solution.

2. Architecture: One Tunnel, Two Roles

The role division is clear:

• rathole-client (Home, Docker + host network): Connects to the server on the VPS, mapping home 127.0.0.1:3000 to 127.0.0.1:13000 on the VPS.
• rathole-server (VPS, Docker + host network): Listens on :2333, receives client connections, and binds each service to local ports according to configuration.
• Caddy (VPS, bare metal): Listens on 443 externally, reverse proxies to the corresponding ports of rathole-server based on domain names; TLS certificates are fully automatic.

All home services bind only to 127.0.0.1, making them inaccessible from outside without going through the rathole tunnel—this is "minimal attack surface" in defense-in-depth.

3. Setup: From Zero to Publicly Accessible

3.1 VPS Side: rathole-server + Caddy

Run rathole on both machines in Docker, using host network (container 127.0.0.1 = host machine). First, write server.toml:

# VPS: /opt/rathole/server.toml
[server]
bind_addr = "0.0.0.0:2333"         # client connects to this port

[server.transport]
type = "tcp"

[server.services.grafana]
token = "<Generate a random token>"     # client must also configure the same token
bind_addr = "127.0.0.1:13000"      # only for Caddy reverse proxy, not exposed to public internet

[server.services.doc]
token = "<Random token>"
bind_addr = "127.0.0.1:13002"

# SSH does not go through Caddy, bind directly to 0.0.0.0:
[server.services.homessh]
token = "<Random token>"
bind_addr = "0.0.0.0:30022"

# Other services: openwebui→13001, paste→13003, caldav→13004, OTLP→9443 (same format)

# VPS: /opt/rathole/docker-compose.yml
services:
  rathole:
    image: rapiz1/rathole:latest
    container_name: rathole-server
    restart: unless-stopped
    network_mode: host
    command: ["--server", "/config/server.toml"]
    volumes:
      - ./server.toml:/config/server.toml:ro

Then Caddy reverse proxy (run bare metal on VPS):

# /etc/caddy/Caddyfile
grafana.liz6.com {
    reverse_proxy 127.0.0.1:13000
}
doc.liz6.com {
    reverse_proxy 127.0.0.1:13002
}

Caddy will automatically sign TLS certificates from Let's Encrypt—if the VPS has a dedicated public IP and ports 80/443 are reachable. If not, see section 3.3 for certificate strategies.

3.2 Home Side: rathole-client

# Home: /opt/rathole-home/client.toml
[client]
remote_addr = "<VPS Public IP>:2333"

[client.transport]
type = "tcp"

[client.transport.tcp]
keepalive_secs = 10           # Keepalive heartbeat, to counter NAT/conntrack dropping connections
keepalive_interval = 5
nodelay = true

[client.services.grafana]
token = "<Same token as server.toml>"
local_addr = "127.0.0.1:3000"

[client.services.doc]
token = "<Same token as server.toml>"
local_addr = "127.0.0.1:8090"

# Home: /opt/rathole-home/docker-compose.yml
services:
  rathole:
    image: rapiz1/rathole:latest
    container_name: rathole-client
    restart: unless-stopped
    network_mode: host
    command: ["--client", "/config/client.toml"]
    volumes:
      - ./client.toml:/config/client.toml:ro

network_mode: host allows the container to directly access the host machine's 127.0.0.1:3000 and other ports—all home services bind to localhost, no need to expose them to the Docker bridge network.

3.3 Certificate Strategies: Two Scenarios

Scenario A: VPS has a dedicated public IP (Recommended). Just leave the tls block empty in Caddy for fully automatic Let's Encrypt. This is the main configuration currently in use.

Scenario B: NAT VPS, public 80/443 unavailable. When sharing an IP via NAT and overseas connectivity is blocked (cannot connect to Let's Encrypt):

1. Run acme.sh at home, use Alibaba Cloud DNS-01 to sign a wildcard *.liz6.com certificate (ACME requests go out via proxy, DNS API uses domestic direct connection).
2. --install-cert --reloadcmd automatically scp certificates to VPS + reload Caddy.
3. On VPS, Caddy uses static certificates (does not use automatic LE), URLs include NAT high-port suffix.

When switching from Scenario B to Scenario A, remove the port suffix from URLs, and keep the acme.sh wildcard certificate only for scenarios like DERP that cannot use LE. Use standard 443 + Caddy automatic LE as much as possible; fewer manual steps mean fewer failure points.

4. Pitfalls: Encountered All Three

4.1 HTTP/3 alt-svc Port Mismatch

Caddy enables HTTP/3 (QUIC) by default. HTTP/3 relies on the alt-svc response header to tell the browser "next time use QUIC, port is X". However, Caddy only knows which port it listens on, not whether there is NAT forwarding externally.

If the public port ≠ Caddy listening port (NAT high port in Scenario B), the port written in alt-svc will be wrong—the browser receives the incorrect QUIC port number, fails to connect, and the page won't load.

Symptom: Open WebUI inexplicably fails to load in Scenario B; curl works (uses HTTP/1.1), but the browser fails (tries QUIC). Troubleshooting: Browser DevTools → Network, see if it hangs on h3 requests.

Fix: Turn off h3 in Caddy—servers { protocols h1 h2 }. After switching back to standard 443 (Scenario A), ensure Caddy's external port = listening port (both 443), then you can turn h3 back on. If there is a reverse proxy or NAT port mapping in front of Caddy, you need to manually correct alt-svc using header_up or header_down.

Residual Pitfall: Browsers cache alt-svc for up to 30 days (HSTS also affects this). After fixing, verify using an incognito window or DevTools → Application → Clear site data.

4.2 Early EOF: conntrack Drops Idle Connections

On CGNAT ↔ Shared IP NAT cross-network links, rathole logs Failed to read cmd: early eof every ~10 minutes—the control channel (persistent connection, basically no traffic normally) is cleaned up by intermediate conntrack as idle.

This is not an rathole issue. Bare TCP tunnels (frp is the same) suffer from this on such links—when a connection is idle for too long, the conntrack table entry on the intermediate NAT gateway expires and is cleared; the next communication attempt results in a TCP reset.

Fix (two methods, each handles its own aspect):

1. TCP keepalive: Set keepalive_secs=10 / keepalive_interval=5 in client.toml—all rathole connections are initiated by the client, so changing only the client side covers both control and data channels.
2. Watchdog fallback (see Section 5): Keepalive only reduces probability; extreme links will still drop; the watchdog automatically restarts and recovers after disconnection.

4.3 Major Pitfall: Local TUN Proxy Hijacks Tunnel Transport

This is the deepest hidden one among the three.

Symptom: VPS has a dedicated public IP (ICMP 0% packet loss, stable latency), keepalive is enabled, but rathole still experiences steady-state timeouts—about 470 timeouts in 24h, control channel rebuilds every ~7min. More strangely: the less used services timeout more (hundreds of times), while frequently used grafana only has a few.

Troubleshooting Process—Conventional Methods Will Mislead You in a TUN Proxy Environment:

First, test VPS connectivity with curl / nc → all pass. Later discovered: On a machine running a TUN proxy (mihomo), curl's TCP handshake is intercepted locally by TUN with fake responses, even connecting to a port that is not listening on the VPS appears "successful". No packets are captured on the VPS side via tcpdump, but the home side says it connected.

Path to the truth:

1. ip route get <VPS_IP> → via 198.18.0.2 dev Meta—traffic to VPS all goes through TUN.
2. Use curl -v to connect to a definitely closed port on VPS (e.g., 9999); if "successful" = TUN fake response; if rejected = really reached VPS.
3. On VPS, tcpdump -i eth0 tcp port 2333 to capture packets—Note: do not use -i any, it uses cooked pseudo-link headers which cause tcp[tcpflags] byte offset filtering to silently match zero.

Root Cause: The home machine is running mihomo TUN itself, doing auto-route + strict-route, taking over all outbound traffic. rathole's connection to VPS is injected into mihomo's TUN data plane—even though the routing rule is judged as DIRECT (not using proxy node), DIRECT only means "does not go through proxy exit", the connection itself still goes through the TUN user-space stack. mihomo's idle connection recycling/reload cuts off rathole's pre-established connection pool's data channel = timeout.

Essentially the same category as Section 4.2 Early EOF (idle connection recycled), just the culprit changed from carrier NAT to local proxy.

Fix: Add VPS IP to mihomo's TUN route exclusion list, making it go through kernel direct connect, completely bypassing the proxy:

# mihomo config.yaml
tun:
  inet4-route-exclude-address:
    - 1.117.75.80/32   # rathole VPS, bypass local mihomo TUN

After systemctl restart mihomo, ip route get <VPS_IP> should become via 192.168.31.1 dev wlp9s0 (native route), then docker restart rathole-client to rebuild connections going through direct connect. After change, continuous silent 5 minutes with zero timeouts, 8/8 services online—previously, disconnection every 7 minutes.

General Principle: The transport connection carrying a tunnel/VPN should never go through another layer of local TUN proxy—this adds another failure point and gets hurt by the proxy's connection lifecycle management. Transport connections must go through native routes. This principle applies to all tunnel solutions requiring persistent connections, such as rathole, frp, WireGuard, Tailscale, ZeroTier, etc.

5. Monitoring: How to Know When the Tunnel is Down

Tunnels are infrastructure; when they go down, all services are affected. Two levels:

① Health Check + Automatic Recovery (watchdog). A simple bash script, run every 5 minutes via systemd timer:

#!/bin/bash
# /usr/local/bin/rathole-watchdog
ENDPOINT="http://127.0.0.1:13002/health"   # Hit VPS doc service health endpoint via rathole tunnel
MAX_FAILS=3                                  # Only declare down after 3 consecutive failures (anti-flapping)
FAIL_FILE=/tmp/rathole-watchdog-fails

curl -s --connect-timeout 5 --max-time 5 "$ENDPOINT" > /dev/null
if [ $? -ne 0 ]; then
    fails=$(($(cat $FAIL_FILE 2>/dev/null || echo 0) + 1))
    echo $fails > $FAIL_FILE
    if [ $fails -ge $MAX_FAILS ]; then
        docker restart rathole-client
        rm -f $FAIL_FILE
    fi
else
    rm -f $FAIL_FILE
fi

② Monitoring Blind Spot: Single Business Channel Deadlock. The watchdog only probes one port (local probe of doc's 13002); if other tunnel channels are stuck (e.g., grafana's 13000), the watchdog won't see it—the rathole control channel is still alive, the health endpoint is accessible, but a data channel is stuck after a network fluctuation and doesn't self-heal.

Real case: doc.liz6.com was down for hours—TLS/certificates were normal, but rathole's doc data channel got stuck after a network fluctuation, while the other 5 tunnels were all fine. docker restart rathole-client forced all channels to reconnect to recover.

Lesson: The watchdog should probe at least two business ports (e.g., doc + grafana), not just one. Ideally, if rathole or subsequent alternatives support per-service reconnection, it should target only restarting the stuck channel, rather than restarting the entire client (which would kick off all SSH sessions).

6. Security Hardening

• VPS sshd: PasswordAuthentication no, key-only login.
• rathole all binds default to 127.0.0.1; only SSH tunnel, not going through Caddy, binds to 0.0.0.0:30022, add TOTP 2FA (google-authenticator PAM).
• Caddy only reverse proxies rathole tunnel ports, does not directly expose home ports.
• Sensitive links (e.g., OTLP metrics) additionally wrap mTLS—Caddy verifies client certificate + revocation check, tunnel only passes through ciphertext.
• On VPS side, use ufw or iptables to restrict rathole port (:2333) to only allow connections from home IP; if IP changes, at least limit brute force attempts on SSH port (:30022) (fail2ban).
• Generate rathole tokens with openssl rand -hex 24 for strong randomness, do not share the same token across multiple services.
• Docker network_mode: host means the container can directly access all host machine ports—use only on trusted networks; both rathole-server and rathole-client should run on controlled machines.

7. Complete Configuration

VPS: server.toml + docker-compose.yml

# /opt/rathole/server.toml
[server]
bind_addr = "0.0.0.0:2333"

[server.transport]
type = "tcp"

[server.services.grafana]
token = "<Random token>"
bind_addr = "127.0.0.1:13000"

[server.services.doc]
token = "<Random token>"
bind_addr = "127.0.0.1:13002"

# SSH does not go through Caddy, expose directly to 0.0.0.0
[server.services.homessh]
token = "<Random token>"
bind_addr = "0.0.0.0:30022"

# /opt/rathole/docker-compose.yml
services:
  rathole:
    image: rapiz1/rathole:latest
    container_name: rathole-server
    restart: unless-stopped
    network_mode: host
    command: ["--server", "/config/server.toml"]
    volumes:
      - ./server.toml:/config/server.toml:ro

VPS: Caddyfile (Example, add routes as needed)

grafana.liz6.com {
    reverse_proxy 127.0.0.1:13000
}
doc.liz6.com {
    reverse_proxy 127.0.0.1:13002
}

Home: client.toml + docker-compose.yml + watchdog

# /opt/rathole-home/client.toml
[client]
remote_addr = "<VPS IP>:2333"

[client.transport]
type = "tcp"

[client.transport.tcp]
keepalive_secs = 10
keepalive_interval = 5
nodelay = true

[client.services.grafana]
token = "<Consistent with server.toml>"
local_addr = "127.0.0.1:3000"

[client.services.doc]
token = "<Consistent with server.toml>"
local_addr = "127.0.0.1:8090"

# /opt/rathole-home/docker-compose.yml
services:
  rathole:
    image: rapiz1/rathole:latest
    container_name: rathole-client
    restart: unless-stopped
    network_mode: host
    command: ["--client", "/config/client.toml"]
    volumes:
      - ./client.toml:/config/client.toml:ro

# /usr/local/bin/rathole-watchdog (complete version see Section 5)

Configure systemd timer to trigger every 5 minutes:

# ~/.config/systemd/user/rathole-watchdog.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/rathole-watchdog

# ~/.config/systemd/user/rathole-watchdog.timer
[Timer]
OnBootSec=2min
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

Related Documentation

• network-architecture.md — Topology Overview
• network-recovery.md — Failure Recovery
• monitoring.md — Monitoring Architecture