wildwindstudio.com Certificate Auto-Renewal + Tencent Cloud CLB Rebinding

Host: bj-middleware-1 (root, TencentOS, Tencent Cloud Beijing)

2026-09-20 Old Certificate Cleanup and Proxy Entry Verification Fix

The September 22nd expiry reminder pointed to YgO1W075 / YgOB9Jx4 uploaded in June. The renewal and rebinding on September 7th were indeed successful; the original script only handled upload, rebinding, and verification, lacking a step to clean up old Tencent Cloud certificates.

It was also discovered that cn-reserve-api's public entry uses an independent TrustAsia certificate, while the CLB origin server uses Let's Encrypt. The old script forcibly compared the public leaf certificate with the origin leaf certificate, falsely judging the deployment as failed, and repeatedly uploaded during daily pending retries.

The current deployment logic is implemented by /usr/local/bin/clb-cert-manager.py, with clb-deploy-hook.sh loading the original credentials and calling it:

• Retrieves the real origin entry from the CLB API, uses the business domain as SNI, and checks the origin certificate fingerprint; the public side separately verifies the trust chain, domain, and validity period, allowing the entry to use an independent certificate.
• Reuses the original certificate ID when the target certificate is already deployed; upload results are persisted, so rebinding failures on retry do not re-upload.
• Persists the old ID before rebinding; the old certificate is only cleaned up after successful verification of the new deployment.
• An independent retry queue for cleanup is run at the end of the daily orchestration. The DeleteCertificate(IsCheckResource=true) command is only executed after confirming the old certificate's ownership/domain, current deployment health, successful asynchronous queries of all cloud resources, and no bindings. The queue entry is only removed after the asynchronous deletion succeeds. If the query fails or resources are still bound, it is retained and an error is reported.
• The official service role SSL_QCSLinkedRoleInReplaceLoadCertificate was added with user authorization, for use in full resource checks and protected deletion.
• Status: /var/lib/clb-cert-renewal/<cert-name>.json; backup: /root/cert-renewal-fix-20260920/; logs and the 02:00 Beijing Time schedule remain unchanged.

The three current CLB certificates remain abhh3hBR (jsvc), abhYtjWf (passport), and avq1JprF (cn-reserve-api), with origin and public TLS passing actual tests. 10 tests cover prohibiting deletion when resource queries are incomplete/incorrect/still bound, asynchronous deletion retries, rebinding retries reusing uploads, and independent edge certificate verification.

After this full cloud resource scan, 16 unbound old/duplicate certificates (including YgO1W075, YgOB9Jx4) have been protected-deleted; the final list contains only the 3 active certificates mentioned above. The cleanup queue and deployment pending are both empty, the orchestration entry returns 0, and crond is running normally.

It is recommended to uniformly use renew-and-update-clb.sh [cert-name ...]. The historical emergency tool clb-bind-one.sh does not yet include the new verification/cleanup flow and is not the recommended entry point.

2026-09-07 Fix and Root Cause Correction

Automatic renewal failed in the early hours of September 1st. jsvc / passport are still using the certificate expiring on September 22nd. It has been confirmed that certbot-dns-tencentcloud 2.0.2 constructs the TC3 signature using the local date from datetime.now(). The date at 02:00 Beijing Time differs by one day from UTC. Read-only DNSPod API contrast tests reproduced the AuthFailure.SignatureFailure; requests succeed when the date is correct. The plugin also swallowed the API exception, ultimately displaying failed to determine base domain. Attributing the successful manual renewal in June to "avoiding rate limits by staggering times" lacks sufficient evidence and cannot be considered a confirmed root cause.

Current script behavior:

• /etc/crontab checks daily at 02:00 Beijing Time; only the certbot subprocess sets TZ=UTC, without modifying the server timezone.
• By default, it manages the three certificates listed in the mapping table below, but can also accept one or more cert-name arguments for targeted execution; execution remains serial, with a 60-second interval between each certificate.
• flock prevents duplicate executions; renewal or rebinding failures return a non-zero exit code, write logs, and log to local syslog, instead of uniformly reporting success.
• The deploy-hook validates listener domain/SNI, waits for CLB asynchronous tasks to complete, rechecks the certificate ID, and verifies via HTTPS that the public leaf certificate matches the newly issued certificate.
• /var/lib/clb-cert-renewal/<cert-name>.pending marks incomplete rebinding; even if certbot returns 0, the next execution will retry deployment first. The marker is only deleted after successful verification.
• Backup: Server /root/cert-renewal-backup-20260907/. Credential files remain unchanged.

Actual renewal results for this session (2026-09-07, script exited 0, CLB and public TLS verified):

cn-reserve-api was not manually renewed this time; CLB remains YToxgk9V; it remains on the original automatic management list and will be handled by subsequent daily tasks according to renewal conditions. No pending markers remained at acceptance, and crond is running normally. Tests for crossing UTC date boundaries, renewal failure exit codes, deploy-hook failure detection, pending deployment retries, and CLB API non-zero exits all passed. The next scheduled check is 02:00 Beijing Time on 2026-09-08.

Recommended manual entry point (includes UTC, locking, logging, renewal, and rebinding verification):

/usr/local/bin/renew-and-update-clb.sh passport.wildwindstudio.com jsvc.wildwindstudio.com-0001

The following June troubleshooting records are retained for historical reference; the current mechanism is based on the fixes described above.

1. Overall Process

Logs are written uniformly to /var/log/certbot-renewal.log; certbot's own debug logs are in /var/log/letsencrypt/.

Key Design Points

• Serial execution + 60s sleep: Retains staggered timing as request rate control; it does not fix signature date errors (see correction above). Do not revert to a single certbot renew batch run for all certificates.
• deploy-hook model: Upload/rebinding is only triggered if the certificate "actually renewed," avoiding unnecessary re-uploads of the same certificate. This is more correct than the old script (which unconditionally retried), but also means — if renewal fails, CLB will not be updated, so renewal health must be monitored (see Section 6).

2. Certificate ↔ CLB Mapping

Region is fixed to ap-beijing; currently all are single-domain certificates:

Mapping is maintained in the case branches of clb-deploy-hook.sh. When adding/changing certificates, update here + the CERTS=(...) list at the top of renew-and-update-clb.sh.

The -0001 suffix on the jsvc certificate's cert-name is a historical legacy (the old three-domain SAN lineage for the same name occupied jsvc.wildwindstudio.com); it has no functional impact. resflow.wildwindstudio.com has been decommissioned (no DNS, no online service); its old CLB lb-mm70ra69 is now reused for cn-reserve-api.

3. File Inventory

Note: The DNS plugin and tccli use two different sets of Tencent Cloud API Keys. The former only requires DNSPod permissions, while the latter requires SSL + CLB permissions.

4. Troubleshooting Records (2026-06-24)

Symptoms

• CLB lb-fkcz6lot (jsvc.wildwindstudio.com) certificate expired; "renewal script had no response," and the online server was serving an expired certificate.
• The same batch, passport and jsvc, both expired on 2026-06-24.

Troubleshooting Steps and Findings

1. Locate entry point: renew-and-update-clb.sh runs at 02:00 on the 1st and 15th of each month in /etc/crontab, logs to /var/log/certbot-renewal.log.
2. Read logs: The 06-15 certbot renew for jsvc/resflow reported

   failed to determine base domain, please report to dev.
   Tried: ['wildwindstudio.com', 'jsvc.wildwindstudio.com']
   

   Certbot renewal was actually failing continuously.
3. Read plugin source certbot_dns_tencentcloud/certbot_tencentcloud_plugins.py's determine_base_domain():

   try:
       resp = client.describe_record_list(dt)
   except APIException as _:   # ← Treats any API exception as "does not own this domain"
       continue
   ...
   raise PluginError("failed to determine base domain ...")
   

   Any API exception (rate limiting / QPS / transient errors) is swallowed, eventually throwing this misleading "base domain" error.
4. Contrast: cn-reserve-api, same account, same tencentcloud.ini, same root domain, succeeded when issued individually on 06-16; the only difference was "running individually vs. batch running."
5. staging / live reproduction: Single certbot renew --cert-name jsvc... succeeded during the day. Based on this, it was inferred to be rate limiting, but this contrast also changed the execution time, so timezone signature differences could not be ruled out; the timezone defect was reproduced in September, see correction above.
6. Old script masked the issue: The old script logic before 06-16 was "regardless of certbot success or failure, unconditionally re-upload the existing certificate and rebind CLB." So every month from 04~06 showed "✓ Rebinding successful," but it was always binding the same old certificate issued on 03-26 and expiring on 06-24, creating the illusion that "renewal was working normally." After changing to the deploy-hook model on 06-16, renewal failures no longer triggered rebinding, exposing the problem.

Inferences at the Time (Corrected)

Initially suspected that adding new domains caused batch request rate limiting. In September, it was found that serial staggered timing still failed in the early hours, and the UTC date signature defect was reproduced; the "single manual success" in June cannot be taken as proof of rate limiting.

5. Actions Taken This Time

Emergency Renewal

• Manually renewed jsvc / passport serially (isolated) to avoid rate limiting; used clb-bind-one.sh to upload and rebind the corresponding CLB.
• Result: Online certificates for lb-fkcz6lot and lb-b6j4kn47 were updated, valid until 2026-09-22.

June Script Modifications (Deployed at the time; timezone issue found unresolved in September)

• renew-and-update-clb.sh changed to serial execution + 60s sleep per certificate + --no-random-sleep-on-renew, addressing rate limiting at the source.
• clb-deploy-hook.sh corrected certificate name mapping, retaining "only rebind after true renewal."

Cleanup

• Deleted expired redundant jsvc three-domain SAN lineage (including decommissioned resflow).
• Deleted old .bak scripts containing plaintext keys.
• Moved plaintext tccli key from 644 /etc/crontab to 600 /etc/letsencrypt/tccli.env; all three scripts now read from this uniformly; crontab backup permissions downgraded to 600.

6. Operations Runbook

Manual Renewal + Rebind a Single Certificate (Emergency)

# 1) Renew (must run single, isolated, do not run multiple at once)
export PATH=/usr/local/bin:$PATH
TZ=UTC certbot renew --cert-name <cert-name> \
  --no-random-sleep-on-renew \
  --dns-tencentcloud-propagation-seconds 30

# 2) Upload + Rebind (credentials automatically read from /etc/letsencrypt/tccli.env)
/usr/local/bin/clb-bind-one.sh \
  /etc/letsencrypt/live/<cert-name> <alias> <lb-id> <listener-id>

Verify CLB Online Certificate

echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null \
  | openssl x509 -noout -subject -dates

View Certificate List / Expiry

certbot certificates

Add a New Domain/Certificate

1. certbot certonly --dns-tencentcloud --dns-tencentcloud-credentials /etc/letsencrypt/tencentcloud.ini -d <domain>
2. Add mapping in the case of clb-deploy-hook.sh; add cert-name to CERTS=(...) in renew-and-update-clb.sh.
3. Manually bind once with clb-bind-one.sh.

TODOs / Improvements

• Rotate tccli API Key (AKIDqgLrCLI2...): Was stored in plaintext in 644 crontab for months; treat as compromised.
• Add certificate days-remaining alert (integrate with existing Prometheus + DingTalk); alert when < 14 days remaining; stop relying on "script didn't error."
• Consider adding retry/backoff to the certbot-dns-tencentcloud plugin (note: modifying site-packages will be overwritten on upgrade; serial staggered timing itself does not solve timezone or other API errors).