保存任务终态事务与取消一致性整改基线,生产复审待完成
This commit is contained in:
parent
9e666be522
commit
8a8b1a5e0e
101
.claude/docs/ai/async-tasks/api-handoff.md
Normal file
101
.claude/docs/ai/async-tasks/api-handoff.md
Normal file
@ -0,0 +1,101 @@
|
||||
# 证书异步任务接口接线说明
|
||||
|
||||
更新日期:2026年10月2日。依据当前源码;本批尚未部署,不代表生产验收通过。
|
||||
|
||||
## 业务范围
|
||||
|
||||
证书申请、当前证书部署及手动重试先持久化任务,再交给后台工作线程执行。浏览器断线不取消已接收工作,前端通过任务接口确认最终结果。提交成功、逐服务器推送成功、整体任务成功是不同层级,不能混用。
|
||||
|
||||
## 接口与权限
|
||||
|
||||
所有接口要求完成初始化、有效登录会话及来源校验;前端沿用现有带会话的 fetchAdmin,不在 URL 中携带凭据。任务与域名接口返回 private, no-store 和 Vary: Cookie,不使用共享缓存。
|
||||
|
||||
| 方法与路径 | 行为 | 权限及返回 |
|
||||
|---|---|---|
|
||||
| POST /api/v1/domains/:id/issue | 提交申请及关联部署,无请求体 | 可见域名、证书管理、部署执行、DNS 凭据使用权限;202 |
|
||||
| POST /api/v1/domains/:id/deploy | 提交已有证书部署,无请求体 | 可见域名、部署执行权限;202 |
|
||||
| POST /api/v1/tasks/:id/retry | 同一任务新增一次尝试,无请求体 | 重试、证书管理、部署执行权限;申请类还检查 DNS 凭据;202 |
|
||||
| DELETE /api/v1/tasks/:id | 请求取消 | 任务可见及取消权限;后台任务返回 202,非异步历史任务可能返回 200 |
|
||||
| GET /api/v1/tasks | 当前身份可见任务列表 | 支持 domain_id 和 limit;返回 200 与 items 数组 |
|
||||
| GET /api/v1/tasks/:id | 单个任务最新状态 | 返回 200 与任务对象本身,不包 task 字段 |
|
||||
| GET /api/v1/tasks/:id/attempts | 任务尝试历史 | 返回 200 与 items 数组 |
|
||||
| GET /api/v1/tasks/:id/deployment-results | 逐尝试的逐服务器部署结果 | 返回 200 与 items 数组;前端按最近 attempt_number 筛选 |
|
||||
|
||||
列表 limit 默认 50,有效范围为 1~200,越界由仓储回退至 50。不存在和无权访问的任务都返回 403,不能根据接口泄露其他项目资源是否存在。
|
||||
|
||||
### 异步提交响应
|
||||
|
||||
返回 202 和 Location: /api/v1/tasks/任务编号。以下仅为字段示例,编号不对应真实环境:
|
||||
|
||||
```json
|
||||
{
|
||||
"accepted": true,
|
||||
"task": {
|
||||
"id": "example-task-id",
|
||||
"domain_id": "example-domain-id",
|
||||
"scope_type": "project",
|
||||
"project_id": "example-project-id",
|
||||
"task_type": "certificate_deploy",
|
||||
"status": "queued",
|
||||
"stage": "queued",
|
||||
"message": "等待执行证书任务",
|
||||
"created_at": "2026-10-02T08:00:00Z",
|
||||
"updated_at": "2026-10-02T08:00:00Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
202 的 task 是提交时快照,不保证仍处于 queued。重试响应可能已经是 running。前端必须使用后续 GET 状态作为依据,不凭提交响应显示成功。
|
||||
|
||||
### 幂等与错误
|
||||
|
||||
- 首次申请和部署支持 Idempotency-Key:长度 8~128,只允许英文字母、数字、连字符、下划线及英文句点。前端当前为每次明确操作生成新键,网络重发若需要去重必须复用原键。
|
||||
- 同一域名、同一键命中已结束任务返回 200、task 和 replayed: true;命中活动任务返回 409、task 和 replayed: true,不再次执行。跨操作复用同一键返回 409。
|
||||
- 同类活动任务、仍在收尾、成功任务重试或不支持的任务类型返回 409。部署的 partial_success 允许重试;申请类只允许 failed 或 cancelled。
|
||||
- 取消 202 只说明接收取消意图,状态为 cancelling;不能提前显示已取消或允许重试。已经完成的外部操作仍可最终成功,并提示取消未撤销该操作。
|
||||
- 400:幂等键等输入无效;401:登录失效;403:无权访问或来源错误;409:初始化未完成或状态冲突;500:持久化等内部错误;501:工作流未配置;503:权限服务未就绪、服务停止或队列已满。
|
||||
- 错误响应一般含 error,可能带 request_id 或 task;网关也可能返回非 JSON 页面。解析失败只能提示暂时无法确认,不能推断任务已失败或自动再次提交。
|
||||
|
||||
## 数据模型
|
||||
|
||||
| 模型 | 字段 |
|
||||
|---|---|
|
||||
| Task | id、domain_id、task_type、status、stage、message、scope_type、project_id、created_at、updated_at;可选 operator_id、certificate_version_id、idempotency_key、started_at、finished_at |
|
||||
| TaskAttempt | id、task_id、attempt_number、stage、status、message、scope_type、project_id、created_at;可选 started_at、finished_at |
|
||||
| TaskDeploymentResult | task_id、attempt_id、attempt_number、result_index、server_id、server、status、message、created_at;可选 backup_id |
|
||||
|
||||
编号和时间为字符串,时间使用带时区的 RFC3339 表示;attempt_number 与 result_index 为整数。Task 与 TaskAttempt 的状态含义如下:
|
||||
|
||||
| 状态 | 中文含义 | 是否终态 |
|
||||
|---|---|---|
|
||||
| queued | 排队中 | 否 |
|
||||
| running | 执行中 | 否 |
|
||||
| retry_wait | 等待重试 | 否 |
|
||||
| cancelling | 正在取消并等待收尾 | 否 |
|
||||
| success | 已完成 | 是 |
|
||||
| failed | 执行失败或结果未可靠确认 | 是 |
|
||||
| cancelled | 已停止;不保证外部副作用全部撤销 | 是 |
|
||||
| partial_success | 部分完成 | 是 |
|
||||
|
||||
未知状态不得默认视为成功。逐服务器结果不包含完整的站点验证及恢复报告,不能据此推导 HTTPS 校验成功。
|
||||
|
||||
## 执行及前端交互边界
|
||||
|
||||
- 当前后台最多接收 32 个在途任务,4 个执行槽,总执行预算 30 分钟;这些是实现限制,不是保证耗时。
|
||||
- 队列满或关闭时拒绝执行,并尝试保存失败终态;持久化失败保留中断状态,不伪造已完成。
|
||||
- 服务关闭停止接收、发送取消信号并等待工作线程收尾。启动恢复将中断记录标为失败,不自动重放外部签发、部署。
|
||||
- 任务和本次尝试的终态通过同一数据库事务保存;逐目标结果和完成审计目前仍为独立写入,不能描述成所有记录完全原子化。
|
||||
- 前端每三秒查询一次,单次周期十秒超时且不重叠;卸载后停止轮询、取消请求。
|
||||
- 切换任务时丢弃旧选择的迟到响应;撤权清除详情及部署结果。一般查询故障保留最后有效状态并提示未知,不改变任务终态。
|
||||
- 三个详情 GET 不是一次原子快照;任务终态与最大 attempt_number 的状态不一致时暂不合并,下一周期重新查询。
|
||||
- 任务从活动变终态后刷新域名、证书、运行状态和审计,但不覆盖正在编辑的设置表单。
|
||||
|
||||
## 回归与未完成项
|
||||
|
||||
已加入自动用例:断线后继续执行、排队取消、取消收尾禁止重试、队列满、异常脱敏、读取和保存失败、跨任务尝试拒绝、终态事务回滚、已完成与部分成功不被取消意图覆盖、前端迟到响应和网络恢复。
|
||||
|
||||
- [~] 当前 Linux 自动回归持续执行;竞争检测尚无可靠通过结果。
|
||||
- [ ] 完整验证及恢复报告持久化与页面展示仍需补齐。
|
||||
- [ ] 完成审计、逐服务器结果与任务终态之间的跨记录一致性继续复核。
|
||||
- [ ] 导入、版本恢复、校验及手动续期等其他长操作尚未全部异步化。
|
||||
- [ ] 正式构建部署后验证超过反代等待窗口的真实操作、断线查询、取消回滚和服务重启;本说明不替代生产验收。
|
||||
@ -216,7 +216,8 @@ func (s *Server) runCertificateTask(ctx context.Context, task repository.Task, a
|
||||
finishCtx, finishCancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer finishCancel()
|
||||
deploymentReport = sanitizeDeploymentReport(deploymentReport)
|
||||
if recordErr := s.tasks.RecordTaskDeploymentResults(finishCtx, task.ID, attempt.ID, attempt.AttemptNumber, taskDeploymentResults(deploymentReport)); recordErr != nil {
|
||||
recordErr := s.tasks.RecordTaskDeploymentResults(finishCtx, task.ID, attempt.ID, attempt.AttemptNumber, taskDeploymentResults(deploymentReport))
|
||||
if recordErr != nil {
|
||||
issueErr = errors.Join(issueErr, recordErr)
|
||||
}
|
||||
finished := time.Now().UTC()
|
||||
@ -233,19 +234,27 @@ func (s *Server) runCertificateTask(ctx context.Context, task repository.Task, a
|
||||
if s.logger != nil {
|
||||
s.logger.Error("证书任务执行失败", "task_id", task.ID, "domain_id", task.DomainID, "error", issueErr)
|
||||
}
|
||||
if errors.Is(issueErr, context.Canceled) {
|
||||
if errors.Is(issueErr, context.Canceled) && !partial && deploymentReport.RecoveryStatus != "unconfirmed" && deploymentReport.RecoveryStatus != "failed" {
|
||||
status = "cancelled"
|
||||
}
|
||||
} else if partial {
|
||||
status = "partial_success"
|
||||
}
|
||||
if recordErr != nil {
|
||||
status, message = "failed", "部署结果保存失败,请核查实际部署状态后再重试"
|
||||
}
|
||||
s.taskMu.Lock()
|
||||
defer s.taskMu.Unlock()
|
||||
if persisted, err := s.tasks.GetTask(finishCtx, task.ID); err == nil && (persisted.Status == "cancelled" || persisted.Status == "cancelling") {
|
||||
status = "cancelled"
|
||||
message = persisted.Message
|
||||
if issueErr == nil {
|
||||
issueErr = errors.New(message)
|
||||
if persisted, err := s.tasks.GetTask(finishCtx, task.ID); err != nil {
|
||||
status, message = "failed", "无法确认任务收尾状态,请核查部署结果后再重试"
|
||||
issueErr = errors.Join(issueErr, err)
|
||||
} else if persisted.Status == "cancelled" || persisted.Status == "cancelling" {
|
||||
// 取消请求只是意图,不能覆盖已经完成的外部操作,也不能隐藏失败及部分成功。
|
||||
if issueErr == nil && !partial {
|
||||
message = "任务已完成,取消请求未撤销已完成的操作"
|
||||
} else if status == "cancelled" {
|
||||
status = "cancelled"
|
||||
message = "任务已停止;已执行操作是否回滚请查看部署结果"
|
||||
}
|
||||
}
|
||||
if s.audit != nil {
|
||||
@ -260,11 +269,11 @@ func (s *Server) runCertificateTask(ctx context.Context, task repository.Task, a
|
||||
status, message = "failed", "任务完成审计保存失败,请检查任务恢复状态"
|
||||
}
|
||||
}
|
||||
if err := s.tasks.FinishAttempt(finishCtx, attempt.ID, status, message, &finished); err != nil {
|
||||
status, message = "failed", "任务尝试结果保存失败,请检查任务恢复状态"
|
||||
}
|
||||
if err := s.tasks.UpdateTask(finishCtx, task.ID, status, stage, message, &now, &finished); err != nil && s.logger != nil {
|
||||
s.logger.Error("保存任务终态失败", "task_id", task.ID, "error", err)
|
||||
if err := s.tasks.FinishTaskAttempt(finishCtx, repository.TaskCompletion{
|
||||
TaskID: task.ID, AttemptID: attempt.ID, Status: status, Stage: stage,
|
||||
Message: message, StartedAt: &now, FinishedAt: finished,
|
||||
}); err != nil && s.logger != nil {
|
||||
s.logger.Error("保存任务及尝试终态失败,保留中断恢复状态", "task_id", task.ID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
129
backend/internal/httpapi/task_completion_test.go
Normal file
129
backend/internal/httpapi/task_completion_test.go
Normal file
@ -0,0 +1,129 @@
|
||||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"git.pcux.cn/xuans/Scert/backend/internal/certificate"
|
||||
"git.pcux.cn/xuans/Scert/backend/internal/deployment"
|
||||
"git.pcux.cn/xuans/Scert/backend/internal/repository"
|
||||
)
|
||||
|
||||
func TestAsyncCancellationPreservesActualOperationOutcome(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
report deployment.Report
|
||||
err error
|
||||
want string
|
||||
}{
|
||||
{name: "already_completed", report: deployment.Report{Status: "success"}, want: "success"},
|
||||
{name: "failed", report: deployment.Report{Status: "failed"}, err: errors.New("部署收尾失败"), want: "failed"},
|
||||
{name: "partially_completed", report: deployment.Report{Status: "partial"}, err: context.Canceled, want: "partial_success"},
|
||||
{name: "cancelled", err: context.Canceled, want: "cancelled"},
|
||||
{name: "recovery_unconfirmed", report: deployment.Report{Status: "failed", RecoveryStatus: "unconfirmed"}, err: errors.Join(context.Canceled, errors.New("回滚未确认")), want: "failed"},
|
||||
{name: "recovery_failed", report: deployment.Report{Status: "failed", RecoveryStatus: "failed"}, err: errors.Join(context.Canceled, errors.New("回滚失败")), want: "failed"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
started := make(chan struct{})
|
||||
server, tasks, _, cleanup := newTaskAPITestServer(t, func(ctx context.Context, _ string) (certificate.IssueResult, deployment.Report, error) {
|
||||
close(started)
|
||||
<-ctx.Done()
|
||||
return certificate.IssueResult{}, test.report, test.err
|
||||
})
|
||||
defer cleanup()
|
||||
response := httptest.NewRecorder()
|
||||
serveAdminHTTP(server, response, httptest.NewRequest(http.MethodPost, "/api/v1/domains/domain-1/issue", nil))
|
||||
task := submittedTask(t, response)
|
||||
select {
|
||||
case <-started:
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("任务没有开始")
|
||||
}
|
||||
response = httptest.NewRecorder()
|
||||
serveAdminHTTP(server, response, httptest.NewRequest(http.MethodDelete, "/api/v1/tasks/"+task.ID, nil))
|
||||
if response.Code != http.StatusAccepted {
|
||||
t.Fatalf("取消未被接收:%s", response.Body.String())
|
||||
}
|
||||
waitTaskWorkersForTest(t, server)
|
||||
stored, err := tasks.GetTask(t.Context(), task.ID)
|
||||
if err != nil || stored.Status != test.want {
|
||||
t.Fatalf("取消请求覆盖了实际结果:%+v %v", stored, err)
|
||||
}
|
||||
attempts, err := tasks.ListAttempts(t.Context(), task.ID)
|
||||
if err != nil || len(attempts) != 1 || attempts[0].Status != stored.Status {
|
||||
t.Fatalf("任务与尝试不一致:%+v %v", attempts, err)
|
||||
}
|
||||
if test.want == "success" && !strings.Contains(stored.Message, "未撤销") {
|
||||
t.Fatal("必须提示取消未撤销已完成操作")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type failingCompletionTaskRepository struct {
|
||||
repository.TaskRepository
|
||||
operationFinished *atomic.Bool
|
||||
failRead bool
|
||||
}
|
||||
|
||||
func (r failingCompletionTaskRepository) GetTask(ctx context.Context, id string) (repository.Task, error) {
|
||||
if r.failRead && r.operationFinished.Load() {
|
||||
return repository.Task{}, errors.New("模拟收尾状态读取失败")
|
||||
}
|
||||
return r.TaskRepository.GetTask(ctx, id)
|
||||
}
|
||||
|
||||
func (r failingCompletionTaskRepository) FinishTaskAttempt(ctx context.Context, item repository.TaskCompletion) error {
|
||||
if !r.failRead {
|
||||
return errors.New("模拟收尾事务保存失败")
|
||||
}
|
||||
return r.TaskRepository.FinishTaskAttempt(ctx, item)
|
||||
}
|
||||
|
||||
func TestAsyncCompletionFailureDoesNotPublishFalseSuccess(t *testing.T) {
|
||||
for _, failRead := range []bool{true, false} {
|
||||
name := "atomic_write_failure"
|
||||
if failRead {
|
||||
name = "state_read_failure"
|
||||
}
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var finished atomic.Bool
|
||||
server, tasks, _, cleanup := newTaskAPITestServer(t, func(context.Context, string) (certificate.IssueResult, deployment.Report, error) {
|
||||
finished.Store(true)
|
||||
return certificate.IssueResult{}, deployment.Report{Status: "success"}, nil
|
||||
})
|
||||
defer cleanup()
|
||||
server.tasks = failingCompletionTaskRepository{TaskRepository: tasks, operationFinished: &finished, failRead: failRead}
|
||||
response := httptest.NewRecorder()
|
||||
serveAdminHTTP(server, response, httptest.NewRequest(http.MethodPost, "/api/v1/domains/domain-1/issue", nil))
|
||||
task := submittedTask(t, response)
|
||||
waitTaskWorkersForTest(t, server)
|
||||
stored, err := tasks.GetTask(t.Context(), task.ID)
|
||||
if err != nil || stored.Status == "success" {
|
||||
t.Fatalf("无法可靠持久化却标记成功:%+v %v", stored, err)
|
||||
}
|
||||
attempts, err := tasks.ListAttempts(t.Context(), task.ID)
|
||||
if err != nil || len(attempts) != 1 || attempts[0].Status == "success" {
|
||||
t.Fatalf("尝试被单独保存为成功:%+v %v", attempts, err)
|
||||
}
|
||||
if failRead {
|
||||
if stored.Status != "failed" || attempts[0].Status != "failed" {
|
||||
t.Fatal("状态读取失败必须闭合为失败")
|
||||
}
|
||||
} else {
|
||||
if !isActiveTaskStatus(stored.Status) || stored.FinishedAt != nil || attempts[0].FinishedAt != nil {
|
||||
t.Fatal("事务失败必须保留可由启动恢复接管的中断状态")
|
||||
}
|
||||
if count, err := tasks.RecoverInterruptedTasks(t.Context(), time.Now().UTC()); err != nil || count != 1 {
|
||||
t.Fatalf("中断记录无法恢复:%d %v", count, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -69,11 +69,11 @@ func (s *Server) finishTaskBeforeExecution(task repository.Task, attempt reposit
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
finished := time.Now().UTC()
|
||||
if err := s.tasks.FinishAttempt(ctx, attempt.ID, status, message, &finished); err != nil && s.logger != nil {
|
||||
s.logger.Error("保存任务尝试失败", "task_id", task.ID, "error", err)
|
||||
}
|
||||
if err := s.tasks.UpdateTask(ctx, task.ID, status, attempt.Stage, message, nil, &finished); err != nil && s.logger != nil {
|
||||
s.logger.Error("保存任务终态失败", "task_id", task.ID, "error", err)
|
||||
if err := s.tasks.FinishTaskAttempt(ctx, repository.TaskCompletion{
|
||||
TaskID: task.ID, AttemptID: attempt.ID, Status: status, Stage: attempt.Stage,
|
||||
Message: message, FinishedAt: finished,
|
||||
}); err != nil && s.logger != nil {
|
||||
s.logger.Error("保存任务及尝试终态失败,保留中断恢复状态", "task_id", task.ID, "error", err)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
73
backend/internal/repository/task_completion.go
Normal file
73
backend/internal/repository/task_completion.go
Normal file
@ -0,0 +1,73 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// TaskCompletion 同时关闭任务与对应尝试,避免只保存其中一方的终态。
|
||||
type TaskCompletion struct {
|
||||
TaskID string
|
||||
AttemptID string
|
||||
Status string
|
||||
Stage string
|
||||
Message string
|
||||
StartedAt *time.Time
|
||||
FinishedAt time.Time
|
||||
}
|
||||
|
||||
func (r *SQLTaskRepository) FinishTaskAttempt(ctx context.Context, completion TaskCompletion) error {
|
||||
if r == nil || r.db == nil {
|
||||
return errors.New("数据库连接不能为空")
|
||||
}
|
||||
completion.TaskID = strings.TrimSpace(completion.TaskID)
|
||||
completion.AttemptID = strings.TrimSpace(completion.AttemptID)
|
||||
completion.Stage = strings.TrimSpace(completion.Stage)
|
||||
if completion.TaskID == "" || completion.AttemptID == "" || completion.Stage == "" || completion.FinishedAt.IsZero() {
|
||||
return errors.New("任务终态记录不完整")
|
||||
}
|
||||
switch completion.Status {
|
||||
case "success", "failed", "cancelled", "partial_success":
|
||||
default:
|
||||
return errors.New("任务终态无效")
|
||||
}
|
||||
tx, err := r.db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return fmt.Errorf("开始任务收尾事务失败:%w", err)
|
||||
}
|
||||
defer tx.Rollback()
|
||||
// 与取消、重试保持任务在前、尝试在后的加锁顺序。旧尝试或跨任务尝试必须整体回滚。
|
||||
result, err := tx.ExecContext(ctx, r.query(`UPDATE tasks
|
||||
SET status = ?, stage = ?, message = ?, started_at = COALESCE(?, started_at), finished_at = ?, updated_at = ?
|
||||
WHERE id = ? AND status IN ('queued', 'running', 'retry_wait', 'cancelling')`),
|
||||
completion.Status, completion.Stage, completion.Message, nullableTime(completion.StartedAt),
|
||||
completion.FinishedAt.UTC(), completion.FinishedAt.UTC(), completion.TaskID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("保存任务终态失败:%w", err)
|
||||
}
|
||||
if count, err := result.RowsAffected(); err != nil {
|
||||
return fmt.Errorf("读取任务收尾结果失败:%w", err)
|
||||
} else if count != 1 {
|
||||
return sql.ErrNoRows
|
||||
}
|
||||
result, err = tx.ExecContext(ctx, r.query(`UPDATE task_attempts
|
||||
SET status = ?, stage = ?, message = ?, finished_at = ?
|
||||
WHERE id = ? AND task_id = ? AND status IN ('queued', 'running', 'retry_wait', 'cancelling')`),
|
||||
completion.Status, completion.Stage, completion.Message, completion.FinishedAt.UTC(), completion.AttemptID, completion.TaskID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("保存任务尝试终态失败:%w", err)
|
||||
}
|
||||
if count, err := result.RowsAffected(); err != nil {
|
||||
return fmt.Errorf("读取任务尝试收尾结果失败:%w", err)
|
||||
} else if count != 1 {
|
||||
return sql.ErrNoRows
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return fmt.Errorf("提交任务收尾事务失败:%w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
99
backend/internal/repository/task_completion_test.go
Normal file
99
backend/internal/repository/task_completion_test.go
Normal file
@ -0,0 +1,99 @@
|
||||
package repository
|
||||
|
||||
import (
|
||||
"database/sql"
|
||||
"errors"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func seedTaskCompletion(t *testing.T, repo *SQLTaskRepository) TaskCompletion {
|
||||
t.Helper()
|
||||
started := time.Now().UTC().Add(-time.Minute)
|
||||
if err := repo.CreateTask(t.Context(), Task{
|
||||
ID: "completion-task", DomainID: "domain-1", TaskType: "certificate_deploy",
|
||||
Status: "running", Stage: "deployment", StartedAt: &started,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := repo.CreateAttempt(t.Context(), TaskAttempt{
|
||||
ID: "completion-attempt", TaskID: "completion-task", AttemptNumber: 1,
|
||||
Status: "running", Stage: "deployment", StartedAt: &started,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return TaskCompletion{
|
||||
TaskID: "completion-task", AttemptID: "completion-attempt", Status: "success",
|
||||
Stage: "deployment", Message: "部署完成", FinishedAt: time.Now().UTC(),
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskCompletionPersistsBothTerminalRecords(t *testing.T) {
|
||||
for _, status := range []string{"success", "failed", "cancelled", "partial_success"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
repo, db := newTaskRepositoryTest(t)
|
||||
defer db.Close()
|
||||
completion := seedTaskCompletion(t, repo)
|
||||
completion.Status = status
|
||||
if err := repo.FinishTaskAttempt(t.Context(), completion); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task, err := repo.GetTask(t.Context(), completion.TaskID)
|
||||
if err != nil || task.Status != status || task.StartedAt == nil || task.FinishedAt == nil || task.Message != completion.Message {
|
||||
t.Fatalf("任务终态未完整保存:%+v %v", task, err)
|
||||
}
|
||||
attempts, err := repo.ListAttempts(t.Context(), completion.TaskID)
|
||||
if err != nil || len(attempts) != 1 || attempts[0].Status != status || attempts[0].FinishedAt == nil || !attempts[0].FinishedAt.Equal(*task.FinishedAt) || attempts[0].Message != task.Message {
|
||||
t.Fatalf("任务和尝试终态不一致:%+v %v", attempts, err)
|
||||
}
|
||||
if err := repo.FinishTaskAttempt(t.Context(), completion); !errors.Is(err, sql.ErrNoRows) {
|
||||
t.Fatalf("终态不能被迟到的收尾重复覆盖:%v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskCompletionRollsBackWhenAttemptCannotFinish(t *testing.T) {
|
||||
for _, failure := range []string{"write_failure", "missing_attempt", "foreign_attempt", "stale_attempt", "invalid_status"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
repo, db := newTaskRepositoryTest(t)
|
||||
defer db.Close()
|
||||
completion := seedTaskCompletion(t, repo)
|
||||
wantAttemptStatus := "running"
|
||||
switch failure {
|
||||
case "write_failure":
|
||||
if _, err := db.ExecContext(t.Context(), `CREATE TRIGGER reject_attempt_completion BEFORE UPDATE ON task_attempts BEGIN SELECT RAISE(ABORT, 'injected completion failure'); END`); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
case "missing_attempt":
|
||||
completion.AttemptID = "missing"
|
||||
case "foreign_attempt":
|
||||
if err := repo.CreateTask(t.Context(), Task{ID: "foreign-task", DomainID: "domain-2", TaskType: "certificate_deploy", Status: "running", Stage: "deployment"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := repo.CreateAttempt(t.Context(), TaskAttempt{ID: "foreign-attempt", TaskID: "foreign-task", AttemptNumber: 1, Status: "running", Stage: "deployment"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
completion.AttemptID = "foreign-attempt"
|
||||
case "stale_attempt":
|
||||
if err := repo.FinishAttempt(t.Context(), completion.AttemptID, "failed", "此前失败", &completion.FinishedAt); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
wantAttemptStatus = "failed"
|
||||
case "invalid_status":
|
||||
completion.Status = "running"
|
||||
}
|
||||
if err := repo.FinishTaskAttempt(t.Context(), completion); err == nil {
|
||||
t.Fatal("应拒绝无法同时关闭任务和尝试的操作")
|
||||
}
|
||||
task, err := repo.GetTask(t.Context(), completion.TaskID)
|
||||
if err != nil || task.Status != "running" || task.FinishedAt != nil {
|
||||
t.Fatalf("尝试保存失败后任务未回滚:%+v %v", task, err)
|
||||
}
|
||||
attempts, err := repo.ListAttempts(t.Context(), completion.TaskID)
|
||||
if err != nil || len(attempts) != 1 || attempts[0].Status != wantAttemptStatus {
|
||||
t.Fatalf("失败操作修改了尝试:%+v %v", attempts, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@ -68,6 +68,7 @@ type TaskRepository interface {
|
||||
FindTaskByIdempotencyKey(context.Context, string, string) (Task, error)
|
||||
CreateAttempt(context.Context, TaskAttempt) error
|
||||
FinishAttempt(context.Context, string, string, string, *time.Time) error
|
||||
FinishTaskAttempt(context.Context, TaskCompletion) error
|
||||
ListAttempts(context.Context, string) ([]TaskAttempt, error)
|
||||
RecordTaskDeploymentResults(context.Context, string, string, int, []TaskDeploymentResult) error
|
||||
ListTaskDeploymentResults(context.Context, string) ([]TaskDeploymentResult, error)
|
||||
|
||||
@ -180,7 +180,7 @@ func renderUnitFile(options Options) (string, error) {
|
||||
"Environment=SCERT_BOOTSTRAP_FIREWALL_MODE=external",
|
||||
"Environment=SCERT_SERVER_HOST=127.0.0.1",
|
||||
"KillSignal=SIGTERM",
|
||||
"TimeoutStopSec=30",
|
||||
"TimeoutStopSec=180",
|
||||
"Restart=on-failure",
|
||||
"UMask=0077",
|
||||
"RuntimeDirectory=scert",
|
||||
|
||||
@ -35,6 +35,7 @@ func TestRenderUnitFileIncludesExecutableConfigurationAndWorkDir(t *testing.T) {
|
||||
"Type=notify",
|
||||
"NotifyAccess=main",
|
||||
"TimeoutStartSec=90",
|
||||
"TimeoutStopSec=180",
|
||||
"User=Scert",
|
||||
"Group=Scert",
|
||||
"WorkingDirectory=\"/opt/Scert data\"",
|
||||
@ -78,6 +79,7 @@ func TestPackagedUnitAllowsInterfaceDiscoveryWithoutCapabilities(t *testing.T) {
|
||||
"Type=notify",
|
||||
"NotifyAccess=main",
|
||||
"TimeoutStartSec=90",
|
||||
"TimeoutStopSec=180",
|
||||
"CapabilityBoundingSet=",
|
||||
"AmbientCapabilities=",
|
||||
"NoNewPrivileges=yes",
|
||||
|
||||
@ -1,5 +1,15 @@
|
||||
# Linux 生产流程测试记录
|
||||
|
||||
## 2026年10月2日17:11 任务收尾事务与服务配置回归(未部署)
|
||||
|
||||
- [x] 第二台隔离源码首次关联回归:internal/httpapi、internal/repository、internal/servicehost、internal/app 全部通过,COMPLETION_FIRST_EXIT=0;日志为 /var/tmp/scert-async-20261002-1556/completion-first.log。
|
||||
- [x] 补充回滚失败的取消边界后,使用非特权用户、GOMAXPROCS=2、包并行数一执行完整后端测试和 go vet,COMPLETION_FULL_EXIT=0、COMPLETION_VET_EXIT=0;日志为同目录 completion-full.log 与 completion-vet.log。
|
||||
- [x] 终态仓储失败注入、任务与尝试一致性、启动恢复接管、取消不覆盖实际结果、动态及静态单元预算断言均纳入本批测试;数据库用例为隔离 SQLite,未触及线上业务库。
|
||||
- [~] 同目录 completion-repeat.log 正在记录异步、终态事务及服务配置二十次普通重复回归;completion-unit.log 将记录静态单元检查。
|
||||
- [!] 第一台 race-second.log 的竞争检测尚未取得退出状态。新建本机 SSH 会话无握手结果;从第二台按固定 ED25519 指纹和密码方式连接同样在 banner exchange 超时。原执行会话无输出,已发送中断但无确认,不以本地句柄存活断言远端仍正常运行,也不擅自重复启动测试。
|
||||
- [!] Git 推送出现两次连接超时,本地提交 9e666be 已保存。网络问题不能算产品用例失败,也不能掩盖缺失的竞争检测证据。
|
||||
- [!] 本批没有安装线上程序、清空业务数据库、开放外部端口、关闭 TLS 校验或创建辅助验收脚本。最终构建部署、三数据库首次 Web 流程和生产复审仍需独立完成。
|
||||
|
||||
## 2026年10月2日16:47 异步任务开发回归记录(未部署)
|
||||
|
||||
### 已完成的开发回归
|
||||
|
||||
@ -1,5 +1,19 @@
|
||||
# Scert 整改任务执行清单
|
||||
|
||||
## 2026年10月2日17:11 异步收尾一致性整改
|
||||
|
||||
本节接续 16:47 的任务,不缩减原始 Linux 上线范围;本批仍未部署。
|
||||
|
||||
- [x] 修复动态安装模板仍为三十秒的遗漏:动态模板和分发单元统一停止预算一百八十秒,原生 Linux 两种路径的断言通过。该预算不等于已证明所有关闭路径无中断。
|
||||
- [x] 新增任务和执行尝试的同事务终态保存;第二条写入失败、尝试缺失、跨任务尝试、过期尝试和重复终态均拒绝部分提交。覆盖数据库触发器注入失败后的回滚。
|
||||
- [x] 取消请求不覆盖已完成、部分成功、回滚失败或未确认的真实结果;收尾状态读取或逐服务器结果保存失败不发布成功;终态事务失败保留中断记录供启动恢复。
|
||||
- [x] 第二台隔离 Linux 四个关联包首次回归通过;最新完整 go test -p 1 ./... -count=1 与 go vet -p 1 ./... 均通过。新增仓储用例使用隔离 SQLite,不冒充三数据库完整流程通过。
|
||||
- [~] 异步、终态事务、动态和静态服务配置专项重复二十次运行中;普通重复回归不能替代竞争检测。
|
||||
- [!] 第一台竞争检测尚无可靠结果;本机和第二台新增 SSH 连接都未完成握手,原会话无输出。已向原会话发送中断信号,但未确认远端停止,不重复启动负载、不推断原因、不记通过。
|
||||
- [x] 已补充[异步接口接线说明](../.claude/docs/ai/async-tasks/api-handoff.md),记录 202、取消语义、查询结构、权限、错误及未完成边界;不修改 Smail API 参考文档。
|
||||
- [!] 开发基线提交 9e666be 已保存在本地,推送两次均连接超时,尚无远程同步成功证据;不进行强制推送。
|
||||
- [ ] 后续继续完整验证与恢复报告持久化、完成审计与结果跨记录一致性、其他长操作异步化及关停边界;原有恢复、通知重置、三库首次 Web、备份恢复及统一验收门禁不变。
|
||||
|
||||
## 2026年10月2日16:47 异步开发与自动回归进度
|
||||
|
||||
本节更新下方 15:43 的早期状态。范围仍为 Linux;先完成开发和自动回归,再统一正式部署验收。开发回归使用隔离源码与临时数据库,不清理线上业务数据。
|
||||
|
||||
@ -1,5 +1,13 @@
|
||||
# Scert 整改、合并与优化清单
|
||||
|
||||
## 2026年10月2日17:11 最新开发进度入口
|
||||
|
||||
- [x] P0-10 新增任务与尝试的原子终态保存,避免部分写入造成长期状态冲突;补齐取消到达较晚、部分成功、回滚未确认和收尾持久化失败的处理与测试。
|
||||
- [x] P0-04 配套修正 Linux 动态安装模板停止预算,与分发单元一致;两种配置路径均有原生回归覆盖。
|
||||
- [x] 最新隔离 Linux 全后端测试及 go vet 通过;前端沿用未再改动的 252 项通过和构建通过证据。本批仍未部署,各条目仅为完成待复审,不标整体通过。
|
||||
- [~] 普通专项重复回归正在进行;第一台竞争检测因 SSH 无法获得新结果保持未确认,不能用普通测试替代。Git 两次推送连接超时,本地开发成果已保存。
|
||||
- [!] 完整部署报告、其他长操作、审计与结果一致性、重启 502、历史恢复证据、系统通知重置、三数据库首次流程和备份恢复仍有待办。P0-10、P0-11 与最终生产验收不关闭。
|
||||
|
||||
## 2026年10月2日16:47 最新开发进度入口
|
||||
|
||||
- [x] P0-10/P1-04 的申请、部署、重试已完成异步提交和前端任务查询接线,浏览器断线不终止已接收任务;最近尝试、取消收尾和查询失败提示已补充。完成待复审,不表示整个条目关闭。
|
||||
|
||||
Loading…
Reference in New Issue
Block a user