Kubernetes 部署策略:从零宕机到无缝更新
我们的生产环境原来每次部署都会宕机 5-10 分钟,用户抱怨不断。
采用蓝绿部署和金丝雀发布后,实现了零宕机部署。
问题背景
原来的部署方式
# ❌ 直接替换
kubectl apply -f deployment.yaml
# 或者
kubectl set image deployment/myapp myapp=myapp:v2问题:
- 旧 Pod 终止时,新 Pod 还没就绪
- 正在处理的请求被中断
- 用户看到 502/503 错误
- 回滚困难
监控数据
| 指标 | 数值 |
|------|------|
| 部署宕机时间 | 5-10 分钟 |
| 失败请求率 | 2-5% |
| 回滚时间 | 10-15 分钟 |
| 用户投诉 | 每次部署 |
部署策略
策略 1:滚动更新(Rolling Update)
最基础的零宕机策略。
# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp
spec:
replicas: 3
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # 最多多出 1 个 Pod
maxUnavailable: 0 # 不允许有 Pod 不可用
template:
spec:
containers:
- name: myapp
image: myapp:v2
# 就绪探针
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
# 存活探针
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
# 启动探针
startupProbe:
httpGet:
path: /health
port: 8080
failureThreshold: 30
periodSeconds: 1
# 优雅终止
terminationGracePeriodSeconds: 60策略 2:蓝绿部署(Blue-Green)
完全隔离新旧版本。
# blue-deployment.yaml(当前生产版本)
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-blue
spec:
replicas: 3
selector:
matchLabels:
app: myapp
version: blue
template:
metadata:
labels:
app: myapp
version: blue
spec:
containers:
- name: myapp
image: myapp:v1
---
# green-deployment.yaml(新版本)
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-green
spec:
replicas: 3
selector:
matchLabels:
app: myapp
version: green
template:
metadata:
labels:
app: myapp
version: green
spec:
containers:
- name: myapp
image: myapp:v2
---
# service.yaml(指向当前版本)
apiVersion: v1
kind: Service
metadata:
name: myapp
spec:
selector:
app: myapp
version: blue # 切换到 green 即可切换流量
ports:
- port: 80
targetPort: 8080切换脚本:
#!/bin/bash
# switch-to-green.sh
# 1. 确认 green 版本就绪
kubectl rollout status deployment/myapp-green
# 2. 切换流量
kubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'
# 3. 等待旧连接断开
sleep 30
# 4. 删除旧版本(可选)
kubectl delete deployment myapp-blue策略 3:金丝雀发布(Canary)
逐步放量,降低风险。
# canary-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: myapp-canary
spec:
replicas: 1 # 只有 1 个 Pod
selector:
matchLabels:
app: myapp
track: canary
template:
metadata:
labels:
app: myapp
track: canary
spec:
containers:
- name: myapp
image: myapp:v2使用 Istio 进行流量分割:
# virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: myapp
spec:
hosts:
- myapp.example.com
http:
- route:
- destination:
host: myapp-stable
subset: v1
weight: 95 # 95% 流量到稳定版
- destination:
host: myapp-canary
subset: v2
weight: 5 # 5% 流量到金丝雀逐步放量脚本:
#!/bin/bash
# canary-rollout.sh
# 阶段 1:5% 流量
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-stable","subset":"v1"},"weight":95},{"destination":{"host":"myapp-canary","subset":"v2"},"weight":5}]}]}}'
sleep 300 # 观察 5 分钟
# 阶段 2:20% 流量
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-stable","subset":"v1"},"weight":80},{"destination":{"host":"myapp-canary","subset":"v2"},"weight":20}]}]}}'
sleep 300
# 阶段 3:50% 流量
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-stable","subset":"v1"},"weight":50},{"destination":{"host":"myapp-canary","subset":"v2"},"weight":50}]}]}}'
sleep 300
# 阶段 4:100% 流量
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-canary","subset":"v2"},"weight":100}]}]}}'优雅终止
应用层处理
// Node.js 优雅终止
process.on('SIGTERM', async () => {
console.log('收到 SIGTERM,开始优雅终止...');
// 1. 停止接收新请求
server.close(() => {
console.log('HTTP 服务器已关闭');
});
// 2. 等待正在处理的请求完成
await waitForPendingRequests();
// 3. 关闭数据库连接
await db.disconnect();
// 4. 关闭 Redis 连接
await redis.quit();
console.log('优雅终止完成');
process.exit(0);
});
// 超时强制退出
setTimeout(() => {
console.error('优雅终止超时,强制退出');
process.exit(1);
}, 55000); // 55 秒,小于 terminationGracePeriodSecondsPreStop Hook
# deployment.yaml
spec:
containers:
- name: myapp
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- sleep 15 # 等待负载均衡器更新自动回滚
基于指标的自动回滚
#!/bin/bash
# auto-rollback.sh
# 监控错误率
ERROR_RATE=$(kubectl get pods -l app=myapp-canary -o json | \
jq '[.items[].status.containerStatuses[].restartCount] | add')
if [ "$ERROR_RATE" -gt 5 ]; then
echo "错误率过高,自动回滚"
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-stable","subset":"v1"},"weight":100}]}]}}'
kubectl delete deployment myapp-canary
exit 1
fi
# 监控延迟
P99_LATENCY=$(curl -s http://prometheus:9090/api/v1/query?query=histogram_quantile(0.99,rate(http_request_duration_seconds_bucket[5m])) | \
jq '.data.result[0].value[1]')
if (( $(echo "$P99_LATENCY > 1.0" | bc -l) )); then
echo "延迟过高,自动回滚"
kubectl patch virtualservice myapp -p '{"spec":{"http":[{"route":[{"destination":{"host":"myapp-stable","subset":"v1"},"weight":100}]}]}}'
kubectl delete deployment myapp-canary
exit 1
fiArgo Rollouts 自动回滚
# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: myapp
spec:
replicas: 5
strategy:
canary:
steps:
- setWeight: 20
- pause: {duration: 5m}
- setWeight: 50
- pause: {duration: 5m}
- setWeight: 100
analysis:
templates:
- templateName: myapp-analysis
startingStep: 1
selector:
matchLabels:
app: myapp
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:v2
---
# analysis.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: myapp-analysis
spec:
metrics:
- name: success-rate
interval: 1m
successCondition: result[0] >= 0.95
failureLimit: 3
provider:
prometheus:
address: http://prometheus:9090
query: |
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))效果对比
| 指标 | 优化前 | 优化后 |
|------|--------|--------|
| 部署宕机时间 | 5-10 分钟 | 0 |
| 失败请求率 | 2-5% | 0% |
| 回滚时间 | 10-15 分钟 | < 1 分钟 |
| 用户投诉 | 每次部署 | 无 |
总结
Kubernetes 部署策略选择:
| 策略 | 适用场景 | 复杂度 | 资源消耗 |
|------|---------|--------|---------|
| 滚动更新 | 一般应用 | 低 | 低 |
| 蓝绿部署 | 关键应用 | 中 | 高(2x) |
| 金丝雀发布 | 高风险变更 | 高 | 中 |
核心原则:
1. 就绪探针:确保新 Pod 就绪后才接收流量
2. 优雅终止:等待正在处理的请求完成
3. 自动回滚:发现问题立即回滚
4. 逐步放量:降低变更风险
做好这些,部署就不再是噩梦。
优化时间:2026年7月
环境规模:50+ 微服务
部署宕机:5-10 分钟 → 0
#Kubernetes #部署策略 #零宕机 #DevOps
读者评论 5