wildwindstudio.com Certificate Auto-Renewal + Tencent Cloud CLB Rebinding

Host: bj-middleware-1 (root, TencentOS, Tencent Cloud Beijing)

1. Overall Workflow

Logs are written to /var/log/certbot-renewal.log; certbot's own debug logs are in /var/log/letsencrypt/.

Key Design Points

• Serial execution + 60s sleep: The primary goal is to avoid Tencent Cloud DNS API rate limiting (see Section 4 for root cause). Do not revert to running certbot renew in batch mode for all certificates.
• deploy-hook model: The upload/rebinding process is only triggered if the certificate "actually renews," avoiding unnecessary re-uploads of the same certificate. This is more correct than the old script (which retransmitted unconditionally), but it also means — if renewal fails, the CLB will not be updated. Therefore, renewal health must be monitored (see Section 6).

2. Certificate ↔ CLB Mapping

Region is fixed to ap-beijing; currently all are single-domain certificates:

Mapping is maintained in the case branches of clb-deploy-hook.sh. When adding/changing certificates, update this file plus the CERTS=(...) list at the top of renew-and-update-clb.sh.

The -0001 suffix in the jsvc certificate's cert-name is a legacy artifact (the old three-domain SAN lineage for the same name occupied jsvc.wildwindstudio.com). It has no functional impact. resflow.wildwindstudio.com has been retired (no DNS, no online services); its old CLB lb-mm70ra69 is now reused for cn-reserve-api.

3. File Inventory

Note: The DNS plugin and tccli use two different Tencent Cloud API Keys. The former only requires DNSPod permissions, while the latter requires SSL + CLB permissions.

4. Troubleshooting Record (2026-06-24)

Symptoms

• CLB lb-fkcz6lot (jsvc.wildwindstudio.com) certificate expired; the "renewal script had no response," and the online server was serving an expired certificate.
• Both passport and jsvc from the same batch expired on 2026-06-24.

Troubleshooting Steps and Findings

1. Locate entry point: renew-and-update-clb.sh runs at 02:00 on the 1st/15th of each month via /etc/crontab; logs are in /var/log/certbot-renewal.log.
2. Read logs: The 06-15 certbot renew run reported for jsvc/resflow:

   failed to determine base domain, please report to dev.
   Tried: ['wildwindstudio.com', 'jsvc.wildwindstudio.com']
   

   This indicates that certbot renewal has been failing continuously.
3. Read plugin source code certbot_dns_tencentcloud/certbot_tencentcloud_plugins.py's determine_base_domain():

   try:
       resp = client.describe_record_list(dt)
   except APIException as _:   # ← Treats any API exception as "does not own this domain"
       continue
   ...
   raise PluginError("failed to determine base domain ...")
   

   Any API exception (rate limiting / QPS / transient errors) is swallowed, eventually throwing this misleading "base domain" error.
4. Comparison: cn-reserve-api uses the same account, same tencentcloud.ini, and same root domain, and succeeded when issued individually on 06-16; the only difference was "single run vs. batch run."
5. staging / live reproduction: Running a single certbot renew --cert-name jsvc... directly bypassed the base-domain step and renewed successfully → Confirmed that batch running triggered rate limiting, which the plugin swallowed as a base-domain error.
6. Old script masked the issue: The old script logic before 06-16 was "regardless of whether certbot succeeded, unconditionally re-upload the existing certificate and rebinding to CLB." Thus, every month from April to June showed "✓ Rebinding successful," but it was always binding the same old certificate issued on 03-26 and expiring on 06-24, creating the illusion that renewal was working normally. After switching to the deploy-hook model on 06-16, renewal failures no longer triggered rebinding, exposing the problem.

Root Cause

After the reservation (cn-reserve) service added 2 new domains, the number of domains processed in a single renewal batch increased, causing dense calls to the Tencent Cloud DNS API in a short time → triggering rate limiting → certbot DNS validation failed to get results → renewal failed → certificates expired without automatic updates.

5. Remediation Actions

Emergency Renewal

• Manually renewed jsvc / passport individually (serial, isolated) to avoid rate limiting; used clb-bind-one.sh to upload and rebinding to the corresponding CLB.
• Result: Online certificates for lb-fkcz6lot and lb-b6j4kn47 are updated, valid until 2026-09-22.

Permanent Fix (Script Refactoring, Live + Self-Test EXIT=0)

• renew-and-update-clb.sh changed to serial execution + 60s sleep per certificate + --no-random-sleep-on-renew to avoid rate limiting at the source.
• clb-deploy-hook.sh corrected certificate name mapping, retaining the "only rebinding after true renewal" behavior.

Cleanup

• Removed expired redundant jsvc three-domain SAN lineage (including retired resflow).
• Removed old .bak scripts containing plaintext keys.
• Moved plaintext tccli key from 644 /etc/crontab to 600 /etc/letsencrypt/tccli.env; all three scripts now read from it; crontab backup permissions reduced to 600.

6. Operations Runbook

Manually Renew + Rebind a Single Certificate (Emergency)

# 1) Renew (must run single, isolated, do not run multiple at once)
export PATH=/usr/local/bin:$PATH
certbot renew --cert-name <cert-name> \
  --no-random-sleep-on-renew \
  --dns-tencentcloud-propagation-seconds 30

# 2) Upload + Rebind (credentials automatically read from /etc/letsencrypt/tccli.env)
/usr/local/bin/clb-bind-one.sh \
  /etc/letsencrypt/live/<cert-name> <alias> <lb-id> <listener-id>

Verify CLB Online Certificate

echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
  | openssl x509 -noout -subject -dates

View Certificate List / Expiry

certbot certificates

Add a New Domain/Certificate

1. certbot certonly --dns-tencentcloud --dns-tencentcloud-credentials /etc/letsencrypt/tencentcloud.ini -d <domain>
2. Add mapping in the case of clb-deploy-hook.sh; add cert-name to CERTS=(...) in renew-and-update-clb.sh.
3. Manually bind once using clb-bind-one.sh.

TODOs / Improvements

• Rotate tccli API Key (AKIDqgLrCLI2...): Was stored in plaintext in 644 crontab for months; treat as leaked.
• Add certificate days-remaining alert (integrate with existing Prometheus + DingTalk); alert when < 14 days remaining; stop relying on "script didn't error."
• Consider adding retry/backoff to the certbot-dns-tencentcloud plugin (note: modifying site-packages will be overwritten on upgrade; current serial + staggered approach at the script level is sufficient).