--- title: wildwindstudio.com Certificate Auto-Renewal + Tencent Cloud CLB Rebinding url: https://doc.liz6.com/en/devops/runbooks/wildwind-clb-cert-renewal locale: en area: devops tags: - devops - runbooks date: 2026-06-30 modified: 2026-07-16 description: 'Host: bj-middleware-1 (root, TencentOS, Tencent Cloud Beijing)' --- # wildwindstudio.com Certificate Auto-Renewal + Tencent Cloud CLB Rebinding > Host: **bj-middleware-1** (root, TencentOS, Tencent Cloud Beijing) --- ## 1. Overall Workflow Overall Workflow: Scheduled Trigger → Serial Renewal → CLB Rebinding Only on Success /etc/crontab · Triggers at 02:00 on the 1st and 15th of each month renew-and-update-clb.sh · Orchestration Script (Core) Executes certbot renew --no-random-sleep-on-renew for each certificate --dns-tencentcloud-propagation-seconds 30 --deploy-hook …; sleeps 60s after each certificate (staggered to avoid rate limiting) certbot (Let's Encrypt · DNS-01 Validation) Uses the certbot-dns-tencentcloud plugin to write _acme-challenge TXT records in DNSPod · Credentials from tencentcloud.ini Only called when a certificate "actually renews successfully" → clb-deploy-hook.sh (deploy-hook, called only upon successful renewal) ① tccli ssl UploadCertificate — Uploads the new certificate to Tencent Cloud SSL, obtaining CertId ② tccli clb ModifyListener — Rebonds the CertId to the corresponding CLB listener · Credentials from tccli.env (600) Tencent Cloud CLB Listener Updated with New Certificate Key Design: Serial execution + 60s sleep avoids Tencent Cloud DNS API rate limiting at the source; The deploy-hook only rebinds after a true renewal succeeds — failures do not trigger unnecessary updates, but this also means renewal health must be monitored (see Section 6). Logs are written to `/var/log/certbot-renewal.log`; certbot's own debug logs are in `/var/log/letsencrypt/`. ### Key Design Points - **Serial execution + 60s sleep**: The primary goal is to avoid Tencent Cloud DNS API rate limiting (see Section 4 for root cause). **Do not** revert to running `certbot renew` in batch mode for all certificates. - **deploy-hook model**: The upload/rebinding process is only triggered if the certificate "actually renews," avoiding unnecessary re-uploads of the same certificate. This is more correct than the old script (which retransmitted unconditionally), but it also means — **if renewal fails, the CLB will not be updated**. Therefore, renewal health must be monitored (see Section 6). --- ## 2. Certificate ↔ CLB Mapping Region is fixed to `ap-beijing`; currently all are **single-domain** certificates: | Certificate (certbot cert-name) | Domain | CLB | Listener | |--------------------------------------------|-------------------------------------|-------------|---------------| | `jsvc.wildwindstudio.com-0001` | jsvc.wildwindstudio.com | lb-fkcz6lot | lbl-ehxkll9d | | `passport.wildwindstudio.com` | passport.wildwindstudio.com | lb-b6j4kn47 | lbl-0jzb58rx | | `cn-reserve-api.wildwindstudio.com` | cn-reserve-api.wildwindstudio.com | lb-mm70ra69 | lbl-naeanxnv | > Mapping is maintained in the `case` branches of `clb-deploy-hook.sh`. When adding/changing certificates, update this file plus the `CERTS=(...)` list at the top of `renew-and-update-clb.sh`. > > The `-0001` suffix in the `jsvc` certificate's cert-name is a legacy artifact (the old three-domain SAN lineage for the same name occupied `jsvc.wildwindstudio.com`). It has no functional impact. > **resflow.wildwindstudio.com has been retired** (no DNS, no online services); its old CLB `lb-mm70ra69` is now reused for cn-reserve-api. --- ## 3. File Inventory | Path | Description | Permissions | |-------------------------------------------|----------------------------------------------------------|-------------| | `/etc/crontab` | Scheduled entry point (no longer contains plaintext keys) | 644 | | `/usr/local/bin/renew-and-update-clb.sh` | Renewal orchestration script | 755 | | `/usr/local/bin/clb-deploy-hook.sh` | certbot deploy-hook: Upload + CLB rebinding | 755 | | `/usr/local/bin/clb-bind-one.sh` | Manual single-certificate rebinding tool (for emergencies) | 755 | | `/etc/letsencrypt/tccli.env` | tccli credentials (for upload/rebinding) | 600 | | `/etc/letsencrypt/tencentcloud.ini` | certbot DNS plugin credentials (for DNS-01 TXT writes) | 600 | | `/var/log/certbot-renewal.log` | Renewal pipeline logs | - | > Note: The DNS plugin and tccli use **two different Tencent Cloud API Keys**. The former only requires DNSPod permissions, while the latter requires SSL + CLB permissions. --- ## 4. Troubleshooting Record (2026-06-24) ### Symptoms - CLB `lb-fkcz6lot` (jsvc.wildwindstudio.com) certificate expired; the "renewal script had no response," and the online server was serving an expired certificate. - Both passport and jsvc from the same batch expired on 2026-06-24. ### Troubleshooting Steps and Findings 1. **Locate entry point**: `renew-and-update-clb.sh` runs at 02:00 on the 1st/15th of each month via `/etc/crontab`; logs are in `/var/log/certbot-renewal.log`. 2. **Read logs**: The 06-15 `certbot renew` run reported for jsvc/resflow: ``` failed to determine base domain, please report to dev. Tried: ['wildwindstudio.com', 'jsvc.wildwindstudio.com'] ``` This indicates that certbot renewal has **been failing continuously**. 3. **Read plugin source code** `certbot_dns_tencentcloud/certbot_tencentcloud_plugins.py`'s `determine_base_domain()`: ```python try: resp = client.describe_record_list(dt) except APIException as _: # ← Treats any API exception as "does not own this domain" continue ... raise PluginError("failed to determine base domain ...") ``` **Any API exception (rate limiting / QPS / transient errors) is swallowed**, eventually throwing this misleading "base domain" error. 4. **Comparison**: cn-reserve-api uses the same account, same `tencentcloud.ini`, and same root domain, and succeeded when issued **individually** on 06-16; the only difference was "single run vs. batch run." 5. **staging / live reproduction**: Running a single `certbot renew --cert-name jsvc...` directly bypassed the base-domain step and renewed successfully → **Confirmed that batch running triggered rate limiting**, which the plugin swallowed as a base-domain error. 6. **Old script masked the issue**: The old script logic before 06-16 was "regardless of whether certbot succeeded, unconditionally re-upload the existing certificate and rebinding to CLB." Thus, every month from April to June showed "✓ Rebinding successful," but it was always binding the same **old certificate** issued on 03-26 and expiring on 06-24, creating the illusion that renewal was working normally. After switching to the deploy-hook model on 06-16, renewal failures no longer triggered rebinding, exposing the problem. ### Root Cause After the reservation (cn-reserve) service **added 2 new domains**, the number of domains processed in a single renewal batch increased, causing dense calls to the Tencent Cloud DNS API in a short time → triggering rate limiting → certbot DNS validation failed to get results → renewal failed → certificates expired without automatic updates. --- ## 5. Remediation Actions ### Emergency Renewal - Manually renewed jsvc / passport individually (serial, isolated) to avoid rate limiting; used `clb-bind-one.sh` to upload and rebinding to the corresponding CLB. - Result: Online certificates for lb-fkcz6lot and lb-b6j4kn47 are updated, valid until **2026-09-22**. ### Permanent Fix (Script Refactoring, Live + Self-Test EXIT=0) - `renew-and-update-clb.sh` changed to **serial execution + 60s sleep per certificate + `--no-random-sleep-on-renew`** to avoid rate limiting at the source. - `clb-deploy-hook.sh` corrected certificate name mapping, retaining the "only rebinding after true renewal" behavior. ### Cleanup - Removed expired redundant jsvc three-domain SAN lineage (including retired resflow). - Removed old `.bak` scripts containing plaintext keys. - Moved plaintext tccli key from 644 `/etc/crontab` to 600 `/etc/letsencrypt/tccli.env`; all three scripts now read from it; crontab backup permissions reduced to 600. --- ## 6. Operations Runbook ### Manually Renew + Rebind a Single Certificate (Emergency) ```bash # 1) Renew (must run single, isolated, do not run multiple at once) export PATH=/usr/local/bin:$PATH certbot renew --cert-name \ --no-random-sleep-on-renew \ --dns-tencentcloud-propagation-seconds 30 # 2) Upload + Rebind (credentials automatically read from /etc/letsencrypt/tccli.env) /usr/local/bin/clb-bind-one.sh \ /etc/letsencrypt/live/ ``` ### Verify CLB Online Certificate ```bash echo | openssl s_client -connect :443 -servername 2>/dev/null \ | openssl x509 -noout -subject -dates ``` ### View Certificate List / Expiry ```bash certbot certificates ``` ### Add a New Domain/Certificate 1. `certbot certonly --dns-tencentcloud --dns-tencentcloud-credentials /etc/letsencrypt/tencentcloud.ini -d ` 2. Add mapping in the `case` of `clb-deploy-hook.sh`; add cert-name to `CERTS=(...)` in `renew-and-update-clb.sh`. 3. Manually bind once using `clb-bind-one.sh`. ### TODOs / Improvements - [ ] **Rotate tccli API Key** (`AKIDqgLrCLI2...`): Was stored in plaintext in 644 crontab for months; treat as leaked. - [ ] **Add certificate days-remaining alert** (integrate with existing Prometheus + DingTalk); alert when < 14 days remaining; stop relying on "script didn't error." - [ ] Consider adding retry/backoff to the `certbot-dns-tencentcloud` plugin (note: modifying site-packages will be overwritten on upgrade; current serial + staggered approach at the script level is sufficient).