feat: 同步链路支持版本分支目录映射、动态快照分支和 ackFail 定向重拉
- Git -> PROD 改为按 branch 作为 configVersion - 按 airportId/appName/fileName 目录结构解析 pushConfig 参数 - PROD -> Git 改为写入 snapshot-branch/<configVersion> 动态分支 - pullConfig 支持 configVersion/fileName 可选过滤 - 抽出 ConfigCryptoService，统一收口加解密扩展点 - ackFail 落库增加重试上下文，支持按 airportId/appName/configVersion/fileName 定向重拉 - 同步更新测试、接口文档和 current.md
This commit is contained in:
dark
parent
c1ced1b7b6
commit
114bcf33d8
70
current.md
Normal file
70
current.md
Normal file
@ -0,0 +1,70 @@
|
||||
当前架构
|
||||
|
||||
已从 开发 -> FTP -> 生产 改为 生产环境单 prod-agent -> 开发 Git / 生产 push-pull API
|
||||
正式启动类是 GitDirectSyncToolApplication.java
|
||||
已完成
|
||||
|
||||
Git -> PROD 主链路已可用
|
||||
PROD -> Git 主链路已可用
|
||||
生产真实接口已按 testapi.txt 适配
|
||||
pushConfig：POST + JSON数组
|
||||
pullConfig：GET + JSON响应
|
||||
login：已支持 token 获取与缓存
|
||||
ackSuc/ackFail：已接入回传与本地落库
|
||||
ConfigCryptoService：已抽出，当前默认透传实现，后续只需替换该服务内算法
|
||||
Git -> PROD：已支持最小增量推送，删除场景自动回退全量
|
||||
Git -> PROD：已改为按“版本分支 + 机场目录 + 模块目录”解析参数
|
||||
PROD -> Git：已按 airportId/appName/fileName 目录结构回写到动态 snapshot 分支
|
||||
Git -> PROD：sourceVersion/configVersion 已改为 Git 分支名，不再用 commit SHA
|
||||
Git -> PROD：baseline 已按版本分支隔离
|
||||
pullConfig：已支持 configVersion/fileName 可选过滤参数
|
||||
ackFail：已支持按 airportId/appName/configVersion/fileName 定向重拉
|
||||
ackFail：失败记录已增加 retryCount/nextRetryAt/lastErrorMsg 元数据
|
||||
管理接口已加：
|
||||
GET /api/admin/sync/overview
|
||||
GET /api/admin/sync/tasks/recent
|
||||
GET /api/admin/sync/tasks/failed
|
||||
docs 下设计文档和接口文档已同步更新到当前口径
|
||||
|
||||
Git 仓库约定
|
||||
|
||||
Git -> PROD：
|
||||
- git.repo.scan-branch 直接指向待同步版本分支
|
||||
- 分支名本身就是 configVersion
|
||||
- 分支内目录结构必须为：airportId/appName/模块内文件
|
||||
- pushConfig 参数映射：
|
||||
- airportId = 路径第1段
|
||||
- appName = 路径第2段
|
||||
- fileName = 模块内相对路径
|
||||
- configVersion = 当前分支名
|
||||
|
||||
PROD -> Git：
|
||||
- pullConfig 返回项会恢复为：airportId/appName/fileName
|
||||
- 当前提交目标分支为：git.repo.snapshot-branch/<configVersion>
|
||||
- 例如：config-prod-snapshot/R_XXX_V3.0.3_XXX
|
||||
- 若 pullConfig 返回缺少统一版本号，则按当前 result 的 sourceVersion 退化生成动态分支名
|
||||
|
||||
关键文件
|
||||
|
||||
生产接口适配：ProdConfigApiService.java
|
||||
加解密扩展点：ConfigCryptoService.java
|
||||
主协调器：ProdSyncCoordinator.java
|
||||
ACK 落库：
|
||||
ProdPullAckRecord.java
|
||||
ProdPullAckService.java
|
||||
管理接口：SyncManagementController.java
|
||||
接口文档：prod-api-v1.md
|
||||
设计文档：
|
||||
git-direct-sync-tool-design.md
|
||||
ftp-sync-tool-design.md
|
||||
ftp-sync-tool-detail-design.md
|
||||
当前 TODO
|
||||
|
||||
configContent 推送前加密
|
||||
configContent 拉取后解密
|
||||
验证状态
|
||||
|
||||
mvn -s build-support/maven-settings.xml test 已通过
|
||||
如果你是想让我继续下一步，最顺的就是：
|
||||
|
||||
把 ConfigCryptoService 的透传实现替换为正式加解密算法
|
||||
@ -1,266 +1,196 @@
|
||||
# 基于 Git 直连的配置双向同步工具设计方案
|
||||
# 基于 Git 版本分支的配置双向同步工具设计方案
|
||||
|
||||
## 1. 文档目的
|
||||
|
||||
本文档用于说明一套基于 Git 直连的配置同步工具设计方案，满足以下目标：
|
||||
本文档说明当前 Git 直连同步工具的目标模型、仓库约定和双向同步流程。
|
||||
|
||||
- 生产环境定时从开发 Git 拉取新配置，并调用生产 `push` 接口导入生产
|
||||
- 生产环境定时从生产 `pull` 接口拉取当前配置，并回写到开发 Git
|
||||
- 在 FTP 不再使用的前提下，简化整体架构、降低维护成本
|
||||
适用范围：
|
||||
|
||||
## 2. 已知约束
|
||||
- 生产环境定时从开发 Git 拉取某个版本分支，并调用生产 `pushConfig` 下发配置
|
||||
- 生产环境定时从生产 `pullConfig` 拉取当前配置，并回写到 Git 快照分支
|
||||
- FTP 已退出主运行面
|
||||
|
||||
## 2. 核心约束
|
||||
|
||||
### 2.1 技术约束
|
||||
|
||||
- JDK：`1.8`
|
||||
- Spring Boot：`2.7.18`
|
||||
- 轻量数据库：`H2` 或同类开源可商用数据库
|
||||
- 其他依赖必须为开源可商用组件
|
||||
- 状态库：`H2`
|
||||
- Git 操作：`JGit`
|
||||
|
||||
### 2.2 网络与部署约束
|
||||
|
||||
- 生产环境可以访问开发 Git 仓库
|
||||
- 生产环境需要能够调用生产系统 `push/pull` 接口
|
||||
- FTP 不再使用
|
||||
- 生产环境可以读取开发 Git
|
||||
- 生产环境需要能向 Git 推送快照分支
|
||||
- 生产环境需要能访问生产 `push/pull/login` 接口
|
||||
- FTP 不再参与同步流程
|
||||
|
||||
建议先确认一个关键前提：
|
||||
## 3. 总体架构
|
||||
|
||||
- 生产环境是否对开发 Git 具备“读 + 写”权限
|
||||
|
||||
说明：
|
||||
|
||||
- 如果生产环境只能读取 Git，无法推送分支，那么“生产 -> 开发 Git”这条链路不能闭环
|
||||
- 如果生产环境可以读取和推送 Git，则整套同步可以收敛为单点部署
|
||||
|
||||
## 3. 新架构结论
|
||||
|
||||
在新条件下，不再推荐“双端代理 + FTP 中转”。
|
||||
|
||||
推荐改为：
|
||||
|
||||
- **单端代理 + Git 直连 + 本地状态库**
|
||||
|
||||
即只在生产环境部署一套同步服务：
|
||||
|
||||
- `Sync-Agent-Prod`
|
||||
|
||||
它同时承担两类任务：
|
||||
|
||||
1. 从开发 Git 拉取配置，推送到生产
|
||||
2. 从生产 `pull` 接口拉取配置，回写到开发 Git
|
||||
|
||||
整体结构如下：
|
||||
推荐拓扑：
|
||||
|
||||
```text
|
||||
开发 Git 仓库 <----> 生产环境 Sync-Agent-Prod <----> 生产系统 push/pull 接口
|
||||
开发 Git 仓库 <----> 生产环境 prod-agent <----> 生产系统 push/pull 接口
|
||||
```
|
||||
|
||||
## 4. 为什么要改成单端部署
|
||||
|
||||
新架构相比旧方案有明显优势：
|
||||
|
||||
- 去掉 FTP，中转链路减少一跳
|
||||
- 去掉打包上传、轮询下载、ACK 回执等中间环节
|
||||
- 部署节点减少为 1 个，运维更简单
|
||||
- 故障点减少，排查路径更短
|
||||
- 数据流更直接，状态一致性更容易控制
|
||||
|
||||
## 5. 总体方案
|
||||
|
||||
推荐在生产环境部署唯一同步实例：
|
||||
|
||||
- `Sync-Agent-Prod`
|
||||
|
||||
其职责如下：
|
||||
|
||||
- 拉取开发 Git 主配置分支
|
||||
- 检查是否存在待下发的新版本
|
||||
- 调用生产 `push` 接口导入配置
|
||||
- 定时调用生产 `pull` 接口获取当前生产配置
|
||||
- 将生产配置写回 Git 快照分支
|
||||
- 使用 H2 记录同步状态、检查点、失败记录
|
||||
|
||||
## 6. 技术选型
|
||||
|
||||
| 分类 | 选型 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| 运行时 | JDK 1.8 | 满足约束 |
|
||||
| 框架 | Spring Boot 2.7.18 | 主体框架 |
|
||||
| 调度 | Spring Scheduling | 实现定时任务 |
|
||||
| 重试 | Spring Retry | 失败重试 |
|
||||
| 数据库 | H2 File Mode | 持久化检查点与任务状态 |
|
||||
| Git 操作 | JGit | 生产环境直接读写 Git |
|
||||
| HTTP 调用 | RestTemplate | 调用生产 `push/pull` 接口 |
|
||||
| JSON | Jackson | 标准序列化 |
|
||||
| 日志 | SLF4J + Logback | 默认日志能力 |
|
||||
|
||||
说明：
|
||||
|
||||
- FTP 客户端依赖在新方案里已经不是核心能力
|
||||
- 标准同步包、FTP 目录、ACK 文件等设计可以整体下线
|
||||
|
||||
## 7. 部署模式
|
||||
|
||||
### 7.1 推荐模式
|
||||
|
||||
推荐只部署：
|
||||
当前只保留一个正式运行角色：
|
||||
|
||||
- `prod-agent`
|
||||
|
||||
不再需要：
|
||||
## 4. Git 仓库约定
|
||||
|
||||
- `dev-agent`
|
||||
- FTP 中转服务
|
||||
## 4.1 版本分支约定
|
||||
|
||||
### 7.2 运行位置
|
||||
当前需求下：
|
||||
|
||||
同步工具建议运行在生产环境可控节点上，要求：
|
||||
- 一个待发布版本对应一个 Git 分支
|
||||
- `git.repo.scan-branch` 直接配置为当前待同步版本分支
|
||||
- **分支名本身就是 `configVersion`**
|
||||
|
||||
- 能访问开发 Git
|
||||
- 能访问生产 `push/pull` 接口
|
||||
- 能持久化本地 H2 文件数据库
|
||||
示例：
|
||||
|
||||
## 8. 两条核心链路
|
||||
- `R_XXX_V3.0.3_XXX`
|
||||
- `R_XXX_V3.0.4_XXX`
|
||||
|
||||
### 8.1 链路一：开发 Git -> 生产 push 接口
|
||||
### 4.2 分支内目录约定
|
||||
|
||||
用途：
|
||||
|
||||
- 将开发配置分支中的新配置同步到生产环境
|
||||
|
||||
流程如下：
|
||||
|
||||
1. `Sync-Agent-Prod` 定时拉取开发 Git 指定分支
|
||||
2. 获取最新提交版本号，例如 Git Commit ID
|
||||
3. 判断该版本是否已成功同步
|
||||
4. 如果未同步，则导出配置目录
|
||||
5. 调用生产 `push` 接口导入配置
|
||||
6. 成功后更新本地检查点和任务状态
|
||||
|
||||
建议时序图如下：
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant G as Git(开发)
|
||||
participant P as Sync-Agent-Prod
|
||||
participant API as 生产Push接口
|
||||
|
||||
P->>G: 定时 pull config-dev-main
|
||||
P->>P: 判断是否有新 commit
|
||||
P->>P: 导出配置目录
|
||||
P->>API: 调用 push 接口
|
||||
API-->>P: 返回处理结果
|
||||
P->>P: 更新 sync_task / checkpoint
|
||||
```
|
||||
|
||||
### 8.2 链路二：生产 pull 接口 -> 开发 Git
|
||||
|
||||
用途：
|
||||
|
||||
- 将当前生产配置快照回写到开发 Git，用于镜像、审计、回溯
|
||||
|
||||
流程如下：
|
||||
|
||||
1. `Sync-Agent-Prod` 定时调用生产 `pull` 接口
|
||||
2. 将返回结果标准化并计算内容哈希
|
||||
3. 判断该版本或哈希是否已同步
|
||||
4. 如果未同步，则切换到生产快照分支
|
||||
5. 写入配置文件
|
||||
6. 提交 commit 并 push 到开发 Git
|
||||
7. 成功后更新本地检查点和任务状态
|
||||
|
||||
建议时序图如下：
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant API as 生产Pull接口
|
||||
participant P as Sync-Agent-Prod
|
||||
participant G as Git(开发)
|
||||
|
||||
P->>API: 定时调用 pull 接口
|
||||
API-->>P: 返回当前生产配置
|
||||
P->>P: 标准化并计算 hash/version
|
||||
P->>G: checkout config-prod-snapshot
|
||||
P->>G: commit + push
|
||||
P->>P: 更新 sync_task / checkpoint
|
||||
```
|
||||
|
||||
## 9. Git 分支策略
|
||||
|
||||
这个设计点仍然必须保留。
|
||||
|
||||
不建议将“开发配置推生产”和“生产配置回写 Git”使用同一个 Git 分支，否则非常容易形成同步闭环。
|
||||
|
||||
推荐分支如下：
|
||||
|
||||
- `config-dev-main`：开发主配置分支
|
||||
- `config-prod-snapshot`：生产配置镜像分支
|
||||
|
||||
同步规则：
|
||||
|
||||
- `Git -> PROD` 只消费 `config-dev-main`
|
||||
- `PROD -> Git` 只写入 `config-prod-snapshot`
|
||||
|
||||
### 9.1 这样设计的价值
|
||||
|
||||
- 避免生产回写内容再次触发下发
|
||||
- 生产快照不会污染开发主线
|
||||
- 便于审计“生产当前实际配置”
|
||||
|
||||
### 9.2 机器人提交标记
|
||||
|
||||
建议同步工具统一使用固定 commit message 前缀，例如：
|
||||
每个版本分支内部必须按以下结构组织：
|
||||
|
||||
```text
|
||||
sync(prod->git): traceId=xxx version=xxx
|
||||
<airportId>/<appName>/<模块内文件相对路径>
|
||||
```
|
||||
|
||||
同时：
|
||||
示例：
|
||||
|
||||
- `Git -> PROD` 扫描时只关注 `config-dev-main`
|
||||
- 不读取 `config-prod-snapshot`
|
||||
```text
|
||||
R_XXX_V3.0.3_XXX
|
||||
├─ PEK
|
||||
│ └─ monitor
|
||||
│ ├─ application.yml
|
||||
│ └─ jobs/sync-job.json
|
||||
└─ SHA
|
||||
└─ gate
|
||||
└─ gate-rule.json
|
||||
```
|
||||
|
||||
## 10. 状态库设计
|
||||
### 4.3 快照分支约定
|
||||
|
||||
新方案建议保留以下核心表：
|
||||
当前实现使用动态快照分支：
|
||||
|
||||
### 10.1 `sync_checkpoint`
|
||||
- `git.repo.snapshot-branch=config-prod-snapshot`
|
||||
|
||||
用于记录各方向最后一次成功同步的检查点。
|
||||
`PROD -> Git` 会把拉回的数据提交到：
|
||||
|
||||
关键字段：
|
||||
```text
|
||||
git.repo.snapshot-branch/<configVersion>
|
||||
```
|
||||
|
||||
- `direction`
|
||||
- `last_success_version`
|
||||
- `last_success_hash`
|
||||
- `updated_at`
|
||||
目录结构同样为：
|
||||
|
||||
### 10.2 `sync_task`
|
||||
```text
|
||||
<airportId>/<appName>/<fileName>
|
||||
```
|
||||
|
||||
用于记录每次同步任务生命周期。
|
||||
## 5. 两条核心链路
|
||||
|
||||
关键字段：
|
||||
### 5.1 Git -> PROD
|
||||
|
||||
- `trace_id`
|
||||
- `direction`
|
||||
- `source_version`
|
||||
- `content_hash`
|
||||
- `status`
|
||||
- `retry_count`
|
||||
- `error_msg`
|
||||
目标：
|
||||
|
||||
### 10.3 `sync_ack`
|
||||
- 将当前版本分支中的配置同步到生产
|
||||
|
||||
在新架构下：
|
||||
流程：
|
||||
|
||||
- 不再作为跨节点 ACK 使用
|
||||
- 已退出当前主 schema
|
||||
1. 拉取 `git.repo.scan-branch`
|
||||
2. 读取当前 `HEAD revision`
|
||||
3. 以 **分支名** 作为业务版本号 `sourceVersion`
|
||||
4. 导出该分支工作树
|
||||
5. 解析所有 `airportId/appName/fileName` 配置项
|
||||
6. 调用生产 `pushConfig`
|
||||
7. 成功后更新 `sync_task` 和 `sync_checkpoint`
|
||||
|
||||
如果后续需要审计扩展，可以单独恢复为接口调用日志表。
|
||||
说明：
|
||||
|
||||
## 11. 幂等设计
|
||||
- `revision` 仅用于日志和 staging 目录隔离
|
||||
- 当前不再使用 `commit SHA` 作为下发版本号
|
||||
|
||||
建议继续使用以下组合作为幂等键：
|
||||
### 5.2 PROD -> Git
|
||||
|
||||
目标：
|
||||
|
||||
- 将生产当前配置快照回写到 Git
|
||||
|
||||
流程：
|
||||
|
||||
1. 调用生产 `pullConfig`
|
||||
2. 把返回项落盘为 `airportId/appName/fileName`
|
||||
3. 计算内容哈希和来源版本
|
||||
4. 提交到 `git.repo.snapshot-branch/<configVersion>`
|
||||
5. 成功后更新 `sync_task` 和 `sync_checkpoint`
|
||||
|
||||
## 6. 接口参数与路径映射
|
||||
|
||||
Git -> PROD 时，每个文件都映射为一条 `pushConfig` 记录：
|
||||
|
||||
| Git 信息 | 接口字段 |
|
||||
| --- | --- |
|
||||
| 分支名 | `configVersion` |
|
||||
| 路径第 1 段 | `airportId` |
|
||||
| 路径第 2 段 | `appName` |
|
||||
| 路径第 3 段及之后 | `fileName` |
|
||||
| 文件内容 | `configContent` |
|
||||
|
||||
例如：
|
||||
|
||||
```text
|
||||
PEK/monitor/jobs/sync-job.json
|
||||
```
|
||||
|
||||
会变成：
|
||||
|
||||
```json
|
||||
{
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"fileName": "jobs/sync-job.json"
|
||||
}
|
||||
```
|
||||
|
||||
## 7. 增量策略
|
||||
|
||||
当前 Git -> PROD 已实现：
|
||||
|
||||
- 首次同步全量
|
||||
- 后续优先按文件哈希做最小增量
|
||||
- 检测到删除时自动回退为全量
|
||||
|
||||
并且 baseline 已按版本分支隔离：
|
||||
|
||||
```text
|
||||
work/baseline/git-to-prod/<scan-branch>/
|
||||
```
|
||||
|
||||
避免不同版本分支之间相互污染。
|
||||
|
||||
## 8. 状态设计
|
||||
|
||||
当前状态表：
|
||||
|
||||
- `sync_checkpoint`
|
||||
- `sync_task`
|
||||
- `prod_pull_ack`
|
||||
|
||||
说明：
|
||||
|
||||
- `sync_checkpoint`：记录某方向最后一次成功版本和哈希
|
||||
- `sync_task`：记录每次同步任务和重试状态
|
||||
- `prod_pull_ack`：记录 `pullConfig` 的 `ackSuc/ackFail` 回传状态
|
||||
|
||||
## 9. 幂等设计
|
||||
|
||||
当前幂等键：
|
||||
|
||||
```text
|
||||
direction + sourceVersion + contentHash
|
||||
@ -268,219 +198,61 @@ direction + sourceVersion + contentHash
|
||||
|
||||
作用：
|
||||
|
||||
- 同一开发版本不会重复推生产
|
||||
- 同一生产快照不会重复写 Git
|
||||
- 同一版本分支下同一内容不会重复推送
|
||||
- 同一生产快照不会重复回写 Git
|
||||
|
||||
## 12. 失败处理与补偿
|
||||
## 10. 当前配置口径
|
||||
|
||||
### 12.1 自动重试
|
||||
关键 Git 配置：
|
||||
|
||||
以下场景建议自动重试：
|
||||
```properties
|
||||
git.repo.scan-branch=R_XXX_V3.0.3_XXX
|
||||
git.repo.snapshot-branch=config-prod-snapshot
|
||||
git.repo.commit-message-prefix=sync(prod->git)
|
||||
```
|
||||
|
||||
- Git pull 失败
|
||||
- Git push 失败
|
||||
- 生产 `push` 接口调用失败
|
||||
- 生产 `pull` 接口调用失败
|
||||
关键接口配置：
|
||||
|
||||
建议策略：
|
||||
|
||||
- 最大重试次数：`3 ~ 5`
|
||||
- 指数退避：`30s / 60s / 120s`
|
||||
|
||||
### 12.2 失败落库
|
||||
|
||||
失败后建议：
|
||||
|
||||
- 更新 `sync_task.status=FAILED`
|
||||
- 记录异常堆栈摘要
|
||||
- 增加重试次数
|
||||
- 保留最近一次成功检查点不变
|
||||
|
||||
### 12.3 人工补偿
|
||||
|
||||
后续可增加管理接口，支持：
|
||||
|
||||
- 按 `traceId` 重试
|
||||
- 按方向重跑最近一次失败任务
|
||||
- 查询最近同步记录
|
||||
|
||||
## 13. 安全设计
|
||||
|
||||
### 13.1 Git 访问建议
|
||||
|
||||
推荐使用：
|
||||
|
||||
- HTTPS + Token
|
||||
|
||||
或：
|
||||
|
||||
- SSH Deploy Key
|
||||
|
||||
### 13.2 权限建议
|
||||
|
||||
生产环境访问 Git 的账号建议采用最小权限原则：
|
||||
|
||||
- 对 `config-dev-main` 至少有读取权限
|
||||
- 对 `config-prod-snapshot` 需要推送权限
|
||||
|
||||
更理想的做法：
|
||||
|
||||
- 使用专用机器人账号
|
||||
- 对开发主分支启用保护
|
||||
- 限制机器人只写快照分支
|
||||
|
||||
### 13.3 生产接口认证
|
||||
|
||||
生产 `push/pull` 接口建议使用：
|
||||
|
||||
- `Bearer Token`
|
||||
- HTTPS
|
||||
|
||||
## 14. 项目结构建议
|
||||
|
||||
新架构下建议进一步简化模块职责：
|
||||
|
||||
```text
|
||||
sync-tool
|
||||
|- src/main/java
|
||||
| |- config
|
||||
| |- git
|
||||
| |- job
|
||||
| |- repository
|
||||
| |- service
|
||||
| |- web
|
||||
|- src/main/resources
|
||||
| |- application.properties
|
||||
| |- application-prod-agent.properties
|
||||
```properties
|
||||
prod.api.base-url=https://prod.example.com
|
||||
prod.api.push-path=/pic_bus_manage_monitor/configSync/pushConfig
|
||||
prod.api.pull-path=/pic_bus_manage_monitor/configSync/pullConfig
|
||||
prod.api.login-path=/pic_bus_manage_monitor/pam-monitor/login
|
||||
prod.api.token=replace-me
|
||||
prod.api.token-header-name=token
|
||||
prod.api.airport-id=
|
||||
prod.api.app-name=
|
||||
prod.api.pull-config-version=
|
||||
prod.api.pull-file-name=
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `prod-agent` 是唯一正式运行角色
|
||||
- `dev-agent` 与 FTP 相关模块已退出主运行面
|
||||
- `prod.api.airport-id`、`prod.api.app-name` 当前只作为 `pullConfig` 可选过滤参数
|
||||
- 它们不再承担 Git -> PROD 的参数组装职责
|
||||
- `prod.api.pull-config-version`、`prod.api.pull-file-name` 可用于进一步缩小 `pullConfig` 拉取范围
|
||||
|
||||
## 15. 核心模块划分
|
||||
## 11. 当前已实现能力
|
||||
|
||||
建议保留并聚焦以下模块：
|
||||
- `pushConfig`：`POST + JSON 数组`
|
||||
- `pullConfig`：`GET + JSON 响应`
|
||||
- `login`：自动获取并缓存 token
|
||||
- `ackSuc/ackFail`：已接入请求回传与本地落库
|
||||
- Git -> PROD：已支持最小增量推送
|
||||
- 管理接口：
|
||||
- `GET /api/admin/sync/overview`
|
||||
- `GET /api/admin/sync/tasks/recent`
|
||||
- `GET /api/admin/sync/tasks/failed`
|
||||
|
||||
- `GitClientService`
|
||||
- clone / pull / checkout / commit / push
|
||||
- `ProdConfigApiService`
|
||||
- 调用生产 `push/pull` 接口
|
||||
- `SyncTaskService`
|
||||
- 任务创建、状态变更、重试次数维护
|
||||
- `CheckpointService`
|
||||
- 成功检查点维护
|
||||
- `ProdSyncCoordinator`
|
||||
- 串联双向同步流程
|
||||
- `JobScheduler`
|
||||
- 定时调度
|
||||
## 12. 当前未完成项
|
||||
|
||||
已退出主运行面：
|
||||
- 将 `ConfigCryptoService` 的透传实现替换为正式加解密算法
|
||||
|
||||
- FTP 包上传下载逻辑
|
||||
- FTP ACK 逻辑
|
||||
- 双端代理运行路径
|
||||
## 13. 结论
|
||||
|
||||
## 16. 定时任务建议
|
||||
当前文档口径应统一为：
|
||||
|
||||
新架构下推荐保留两类核心任务：
|
||||
|
||||
### 16.1 `GitToProdSyncJob`
|
||||
|
||||
职责：
|
||||
|
||||
- 拉取 `config-dev-main`
|
||||
- 判断是否有新 commit
|
||||
- 调用生产 `push` 接口
|
||||
|
||||
### 16.2 `ProdToGitSnapshotJob`
|
||||
|
||||
职责：
|
||||
|
||||
- 调用生产 `pull` 接口
|
||||
- 判断是否有新快照
|
||||
- 提交到 `config-prod-snapshot`
|
||||
|
||||
可选任务：
|
||||
|
||||
- `RetryFailedTaskJob`
|
||||
- `HealthCheckJob`
|
||||
|
||||
## 17. 一期 MVP 建议
|
||||
|
||||
建议重新按最小可交付版本收敛：
|
||||
|
||||
### 阶段 1：打通 Git -> 生产
|
||||
|
||||
- 生产环境直连开发 Git
|
||||
- 实现 `config-dev-main` 拉取
|
||||
- 实现生产 `push` 接口调用
|
||||
- 落库记录同步状态
|
||||
|
||||
### 阶段 2：打通 生产 -> Git
|
||||
|
||||
- 接入生产 `pull` 接口
|
||||
- 回写 `config-prod-snapshot`
|
||||
- 实现 commit + push
|
||||
|
||||
### 阶段 3：增强稳定性
|
||||
|
||||
- 补充重试
|
||||
- 补充管理接口
|
||||
- 补充告警与审计日志
|
||||
|
||||
## 18. 风险与注意事项
|
||||
|
||||
### 18.1 最大风险：Git 写权限不足
|
||||
|
||||
如果生产环境对开发 Git 没有推送权限，则“生产 -> Git”链路无法完成。
|
||||
|
||||
解决方案：
|
||||
|
||||
- 申请机器人账号
|
||||
- 或将“生产回写 Git”改成调用开发侧服务接口
|
||||
|
||||
### 18.2 最大风险：双向同步闭环
|
||||
|
||||
如果生产回写到了开发主分支，会再次触发下发。
|
||||
|
||||
规避措施：
|
||||
|
||||
- 使用独立快照分支
|
||||
- 不扫描快照分支
|
||||
- 使用幂等键和机器人提交标记
|
||||
|
||||
### 18.3 最大风险：生产直连开发 Git 的安全边界
|
||||
|
||||
需要明确：
|
||||
|
||||
- 网络访问是否合规
|
||||
- Git 账号权限是否受控
|
||||
- Token 或 SSH Key 是否可轮换
|
||||
|
||||
## 19. 结论
|
||||
|
||||
在“生产环境可以直接访问开发 Git，FTP 不再需要”的前提下，推荐将旧方案调整为：
|
||||
|
||||
- **生产环境单点部署**
|
||||
- **Git 直连**
|
||||
- **保留生产 `push/pull` 接口**
|
||||
- **保留 H2 状态库**
|
||||
|
||||
这是比原来 FTP 中转更合适的方案，原因是：
|
||||
|
||||
- 架构更简单
|
||||
- 故障点更少
|
||||
- 链路更短
|
||||
- 运维成本更低
|
||||
|
||||
## 20. 下一步建议
|
||||
|
||||
下一步建议按下面顺序推进：
|
||||
|
||||
1. 先确认生产环境对开发 Git 是否具备推送权限
|
||||
2. 确认生产 `push/pull` 接口最终协议
|
||||
3. 删除退役标记文件 `application-dev-agent.properties`
|
||||
4. 将工程命名中残留的 `ftp` 语义继续清理
|
||||
5. 补充新的 `application-prod-agent.properties` 配置说明
|
||||
- **版本用 Git 分支表达**
|
||||
- **机场和模块用目录表达**
|
||||
- **接口参数由路径直接解析**
|
||||
- **快照按 `git.repo.snapshot-branch/<configVersion>` 动态分支写回**
|
||||
|
||||
@ -1,53 +1,30 @@
|
||||
# Git 直连架构详细设计
|
||||
# Git 直连架构详细设计
|
||||
|
||||
## 1. 文档说明
|
||||
|
||||
本文档用于承接主方案文档，说明在“FTP 下线、生产环境可直接访问开发 Git”条件下的详细设计。
|
||||
本文档描述当前实现层面的详细设计，重点覆盖：
|
||||
|
||||
当前目标是把系统收敛为：
|
||||
- Git 版本分支和目录结构约定
|
||||
- `Git -> PROD` 的增量与参数映射逻辑
|
||||
- `PROD -> Git` 的落盘与回写逻辑
|
||||
- 运行配置、工作目录和已知限制
|
||||
|
||||
- 单 `prod-agent`
|
||||
- Git 直连
|
||||
- 生产 `push/pull` 接口驱动
|
||||
- H2 本地状态控制
|
||||
## 2. 当前推荐部署
|
||||
|
||||
## 2. 架构变化摘要
|
||||
|
||||
旧架构：
|
||||
|
||||
```text
|
||||
开发环境 <-> FTP <-> 生产环境
|
||||
```
|
||||
|
||||
新架构：
|
||||
|
||||
```text
|
||||
开发 Git <-> 生产环境 Sync-Agent-Prod <-> 生产 push/pull 接口
|
||||
```
|
||||
|
||||
关键变化：
|
||||
|
||||
- FTP 中转取消
|
||||
- `dev-agent` 不再是必需部署节点
|
||||
- 生产环境成为唯一同步执行节点
|
||||
- ACK 文件机制不再作为主流程依赖
|
||||
|
||||
## 3. 当前推荐部署
|
||||
|
||||
推荐只部署：
|
||||
当前推荐只部署：
|
||||
|
||||
- `prod-agent`
|
||||
|
||||
运行要求：
|
||||
|
||||
- 能访问开发 Git
|
||||
- 能 push 指定 Git 分支
|
||||
- 能访问生产 `push/pull` 接口
|
||||
- 能向 Git 推送 `snapshotBranch`
|
||||
- 能访问生产 `push/pull/login` 接口
|
||||
- 能写本地 H2 文件数据库
|
||||
|
||||
## 4. 配置文件策略
|
||||
## 3. 配置文件策略
|
||||
|
||||
当前配置文件仍使用 `properties`：
|
||||
当前使用：
|
||||
|
||||
```text
|
||||
src/main/resources/
|
||||
@ -58,224 +35,243 @@ src/main/resources/
|
||||
|
||||
说明：
|
||||
|
||||
- `application-dev-agent.properties` 现阶段仅保留为退役标记文件
|
||||
- `application-prod-agent.properties` 已开始收敛到新架构
|
||||
- `application.properties` 提供公共默认值
|
||||
- `application-prod-agent.properties` 提供生产侧 profile 配置
|
||||
|
||||
## 5. 当前配置口径
|
||||
## 4. 核心配置口径
|
||||
|
||||
### 5.1 仍然保留的核心配置
|
||||
### 4.1 Git 配置
|
||||
|
||||
公共配置：
|
||||
| 配置项 | 说明 |
|
||||
| --- | --- |
|
||||
| `git.repo.remote-uri` | Git 远端地址 |
|
||||
| `git.repo.username` | Git 用户名 |
|
||||
| `git.repo.password` | Git 密码或 token |
|
||||
| `git.repo.scan-branch` | 当前待同步的版本分支名 |
|
||||
| `git.repo.snapshot-branch` | 动态生产快照分支前缀 |
|
||||
| `git.repo.commit-message-prefix` | PROD -> Git 提交前缀 |
|
||||
|
||||
- `spring.datasource.*`
|
||||
- `spring.jpa.*`
|
||||
- `spring.sql.init.*`
|
||||
### 4.2 生产接口配置
|
||||
|
||||
同步配置：
|
||||
| 配置项 | 说明 |
|
||||
| --- | --- |
|
||||
| `prod.api.base-url` | 生产接口基础地址 |
|
||||
| `prod.api.push-path` | `pushConfig` 路径 |
|
||||
| `prod.api.pull-path` | `pullConfig` 路径 |
|
||||
| `prod.api.login-path` | `login` 路径 |
|
||||
| `prod.api.token` | 静态 token，可选 |
|
||||
| `prod.api.token-header-name` | token 请求头名称 |
|
||||
| `prod.api.airport-id` | `pullConfig` 可选过滤 |
|
||||
| `prod.api.app-name` | `pullConfig` 可选过滤 |
|
||||
| `prod.api.pull-config-version` | `pullConfig` 可选版本过滤 |
|
||||
| `prod.api.pull-file-name` | `pullConfig` 可选文件过滤 |
|
||||
|
||||
- `sync.node-id`
|
||||
- `sync.role`
|
||||
- `sync.work-dir`
|
||||
- `sync.package-temp-dir`
|
||||
- `sync.dev-to-prod-staging-dir`
|
||||
- `sync.prod-to-dev-staging-dir`
|
||||
- `sync.max-retry-count`
|
||||
说明：
|
||||
|
||||
Git 配置：
|
||||
- `prod.api.airport-id`、`prod.api.app-name` 当前只用于 `pullConfig` 的可选过滤
|
||||
- 它们已经不再用于 Git -> PROD 参数解析
|
||||
|
||||
- `git.repo.remote-uri`
|
||||
- `git.repo.username`
|
||||
- `git.repo.password`
|
||||
- `git.repo.scan-branch`
|
||||
- `git.repo.snapshot-branch`
|
||||
- `git.repo.commit-message-prefix`
|
||||
### 4.3 调度配置
|
||||
|
||||
生产接口配置：
|
||||
|
||||
- `prod.api.base-url`
|
||||
- `prod.api.push-path`
|
||||
- `prod.api.pull-path`
|
||||
- `prod.api.token`
|
||||
|
||||
### 5.2 新的生产侧调度配置
|
||||
|
||||
当前已调整为：
|
||||
当前生产侧调度：
|
||||
|
||||
- `sync.jobs.prod-git-to-prod.cron`
|
||||
- `sync.jobs.prod-to-git.cron`
|
||||
|
||||
对应文件：
|
||||
## 5. Git 仓库约定
|
||||
|
||||
- [application-prod-agent.properties](e:/AIcoding/FtpTool/src/main/resources/application-prod-agent.properties)
|
||||
### 5.1 版本分支
|
||||
|
||||
### 5.3 遗留 FTP 配置
|
||||
当前实现约定：
|
||||
|
||||
当前 `application.properties` 中已经移除了 `ftp.*` 相关公共配置。
|
||||
- `git.repo.scan-branch` 直接指向当前待同步版本分支
|
||||
- **分支名本身就是 `configVersion`**
|
||||
|
||||
当前状态：
|
||||
例如：
|
||||
|
||||
- FTP 调度主路径已退出运行面
|
||||
- FTP 相关主类已从当前源码树中移除
|
||||
- `R_XXX_V3.0.3_XXX`
|
||||
|
||||
## 6. H2 状态设计
|
||||
### 5.2 分支内目录结构
|
||||
|
||||
当前主表保留为：
|
||||
当前分支内目录必须为：
|
||||
|
||||
```text
|
||||
<airportId>/<appName>/<模块内文件相对路径>
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
R_XXX_V3.0.3_XXX
|
||||
├─ PEK
|
||||
│ └─ monitor
|
||||
│ ├─ application.yml
|
||||
│ └─ jobs/sync-job.json
|
||||
└─ SHA
|
||||
└─ gate
|
||||
└─ gate-rule.json
|
||||
```
|
||||
|
||||
### 5.3 路径校验
|
||||
|
||||
当前实现要求每个待推送文件都至少有 3 段路径：
|
||||
|
||||
```text
|
||||
airportId / appName / fileName
|
||||
```
|
||||
|
||||
否则直接视为非法仓库结构并终止同步。
|
||||
|
||||
## 6. 工作目录设计
|
||||
|
||||
当前工作目录：
|
||||
|
||||
```text
|
||||
work/
|
||||
├─ package/
|
||||
├─ staging/
|
||||
│ ├─ dev-to-prod/
|
||||
│ └─ prod-to-dev/
|
||||
└─ baseline/
|
||||
└─ git-to-prod/
|
||||
└─ <scan-branch>/
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `dev-to-prod/`：保存导出的 Git 版本快照与增量目录
|
||||
- `prod-to-dev/`：保存 `pullConfig` 恢复出的临时目录
|
||||
- `baseline/git-to-prod/<scan-branch>/`：保存某个版本分支上次成功下发后的基线
|
||||
|
||||
## 7. Git -> PROD 详细流程
|
||||
|
||||
### 7.1 流程步骤
|
||||
|
||||
1. 拉取 `git.repo.scan-branch`
|
||||
2. 读取当前 `HEAD revision`
|
||||
3. 以分支名作为 `sourceVersion`
|
||||
4. 导出分支工作树到 staging 目录
|
||||
5. 计算目录内容哈希
|
||||
6. 按 `direction + sourceVersion + contentHash` 做幂等判断
|
||||
7. 计算本次推送目录：
|
||||
- 首次全量
|
||||
- 删除回退全量
|
||||
- 其余场景最小增量
|
||||
8. 遍历目录内所有文件，按路径解析出 `airportId/appName/fileName`
|
||||
9. 组装 `pushConfig` JSON 数组并提交
|
||||
10. 成功后刷新 checkpoint、task 和 baseline
|
||||
|
||||
### 7.2 参数映射规则
|
||||
|
||||
| Git 信息 | pushConfig 字段 |
|
||||
| --- | --- |
|
||||
| `scan-branch` | `configVersion` |
|
||||
| 路径第 1 段 | `airportId` |
|
||||
| 路径第 2 段 | `appName` |
|
||||
| 第 3 段及之后 | `fileName` |
|
||||
| 文件 UTF-8 内容 | `configContent` |
|
||||
|
||||
### 7.3 当前实现说明
|
||||
|
||||
- `HEAD revision` 仅用于日志和 staging 目录名
|
||||
- 不再用 `commit SHA` 作为对外业务版本号
|
||||
- `configContent` 目前仍是明文
|
||||
|
||||
## 8. PROD -> Git 详细流程
|
||||
|
||||
### 8.1 流程步骤
|
||||
|
||||
1. 调用 `pullConfig`
|
||||
2. 可选附带 `airportId/appName` 过滤
|
||||
3. 自动附带本地待回传的 `ackSuc/ackFail`
|
||||
4. 解析响应项
|
||||
5. 按 `airportId/appName/fileName` 落盘到 staging 目录
|
||||
6. 计算内容哈希和来源版本
|
||||
7. 按幂等键判断是否已处理
|
||||
8. 提交到 `git.repo.snapshot-branch/<configVersion>`
|
||||
9. 成功后更新 checkpoint、task 和本地 ACK 状态
|
||||
|
||||
### 8.2 来源版本规则
|
||||
|
||||
当前实现：
|
||||
|
||||
- 如果本次 `pullConfig` 返回项里的 `configVersion` 全部一致，则该值作为 `sourceVersion`
|
||||
- 如果返回多种不同 `configVersion`，则退回为内容哈希
|
||||
|
||||
### 8.3 动态快照分支规则
|
||||
|
||||
- 每个 `ProdPullResult` 都会写入独立目标分支：
|
||||
|
||||
```text
|
||||
git.repo.snapshot-branch/<sourceVersion>
|
||||
```
|
||||
|
||||
- 当 `sourceVersion` 就是接口返回的统一 `configVersion` 时，目标分支即按版本动态展开
|
||||
- 当某组缺少版本号时，会退回为基于内容哈希的 `sourceVersion`
|
||||
|
||||
## 9. 当前接口协议实现口径
|
||||
|
||||
### 9.1 pushConfig
|
||||
|
||||
- 方法：`POST`
|
||||
- 载荷：`JSON 数组`
|
||||
- 每个文件一条记录
|
||||
- `ackFail` 非空时本次同步直接失败
|
||||
- `configContent` 已统一通过 `ConfigCryptoService` 处理，当前默认实现仍为透传
|
||||
|
||||
### 9.2 pullConfig
|
||||
|
||||
- 方法：`GET`
|
||||
- 参数：query string
|
||||
- 当前已支持：
|
||||
- `airportId`
|
||||
- `appName`
|
||||
- `ackSuc`
|
||||
- `ackFail`
|
||||
- 当前已支持：
|
||||
- `configVersion`
|
||||
- `fileName`
|
||||
- 对于本地处理失败的 `ackFail` 项，当前会持久化重试上下文，并在下一个周期按 `airportId/appName/configVersion/fileName` 定向重拉
|
||||
|
||||
### 9.3 login
|
||||
|
||||
- 方法：`POST`
|
||||
- 如果未配置静态 token，则自动调用
|
||||
- token 会按 `expireTime` 缓存
|
||||
|
||||
## 10. 状态表设计
|
||||
|
||||
当前主表：
|
||||
|
||||
- `sync_checkpoint`
|
||||
- `sync_task`
|
||||
- `prod_pull_ack`
|
||||
|
||||
对应脚本：
|
||||
作用：
|
||||
|
||||
- [schema.sql](e:/AIcoding/FtpTool/src/main/resources/schema.sql)
|
||||
- `sync_checkpoint`：方向级最后成功版本和哈希
|
||||
- `sync_task`：任务状态、重试次数、错误摘要
|
||||
- `prod_pull_ack`：待回传给生产侧的 ACK 状态
|
||||
|
||||
### 6.1 保留原因
|
||||
## 11. 当前已实现能力
|
||||
|
||||
即使 FTP 已下线，状态库仍然必须保留，用于：
|
||||
- Git -> PROD 主链路已可用
|
||||
- PROD -> Git 主链路已可用
|
||||
- `pushConfig`/`pullConfig`/`login` 已适配正式协议
|
||||
- `ackSuc/ackFail` 已接入回传与本地落库
|
||||
- Git -> PROD 已支持最小增量推送
|
||||
- 管理接口已提供最近任务、失败任务和检查点视图
|
||||
|
||||
- 幂等控制
|
||||
- 检查点管理
|
||||
- 失败重试
|
||||
- 审计追踪
|
||||
## 12. 当前主要 TODO
|
||||
|
||||
### 6.2 当前建议
|
||||
- 将 `ConfigCryptoService` 的透传实现替换为正式加解密算法
|
||||
|
||||
- `sync_checkpoint` 和 `sync_task` 继续作为主表
|
||||
- `sync_ack` 已退出当前主 schema
|
||||
## 13. 结论
|
||||
|
||||
## 7. 当前代码结构与新架构对应关系
|
||||
当前详细设计已经不再是“固定主分支 + 全局机场配置”的口径，而是：
|
||||
|
||||
### 7.1 已经可继续复用的核心类
|
||||
- 版本靠分支表达
|
||||
- 机场和模块靠目录表达
|
||||
- 路径直接决定接口参数
|
||||
|
||||
- [GitClientService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/GitClientService.java)
|
||||
- [ProdConfigApiService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/ProdConfigApiService.java)
|
||||
- [PackageService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/PackageService.java)
|
||||
- [SyncTaskService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/SyncTaskService.java)
|
||||
- [CheckpointService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/CheckpointService.java)
|
||||
|
||||
### 7.2 当前生产侧主协调器
|
||||
|
||||
- [ProdSyncCoordinator.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/orchestrator/ProdSyncCoordinator.java)
|
||||
|
||||
当前生产侧主流程已经切到新架构：
|
||||
|
||||
- `syncLatestGitToProd()`：Git -> 生产 `push`
|
||||
- `syncProdSnapshotToGit()`：生产 `pull` -> Git snapshot 分支
|
||||
|
||||
### 7.3 当前生产侧正式调度类
|
||||
|
||||
- [GitToProdSyncJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/GitToProdSyncJob.java)
|
||||
- [ProdToGitSnapshotJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdToGitSnapshotJob.java)
|
||||
|
||||
说明：
|
||||
|
||||
- 这两个类是当前推荐使用的正式调度入口
|
||||
- 已分别对应 `Git -> PROD` 和 `PROD -> Git`
|
||||
|
||||
### 7.4 当前遗留占位类
|
||||
|
||||
这批旧调度类已经从当前源码树中删除：
|
||||
|
||||
说明：
|
||||
|
||||
- 已不再作为正式运行入口
|
||||
- 当前代码树只保留 Git 直连架构需要的正式 job
|
||||
|
||||
### 7.5 当前遗留代码
|
||||
|
||||
以下内容仍然存在于代码库，但属于旧架构遗留：
|
||||
|
||||
- `application-dev-agent.properties` 退役标记文件
|
||||
- 少量文件名或文档中的 `ftp` 语义残留
|
||||
|
||||
这些不是当前推荐运行路径。
|
||||
|
||||
## 8. 新架构下的两条任务流
|
||||
|
||||
### 8.1 Git -> 生产
|
||||
|
||||
当前推荐实现步骤：
|
||||
|
||||
1. 拉取 `config-dev-main`
|
||||
2. 获取最新 commit 作为 `sourceVersion`
|
||||
3. 导出工作树快照
|
||||
4. 计算内容哈希
|
||||
5. 生成标准 zip 包
|
||||
6. 调用生产 `push` 接口
|
||||
7. 更新 `sync_task` 和 `sync_checkpoint`
|
||||
|
||||
### 8.2 生产 -> Git
|
||||
|
||||
当前推荐实现步骤：
|
||||
|
||||
1. 调用生产 `pull` 接口
|
||||
2. 保存返回配置到本地 staging 目录
|
||||
3. 计算哈希和版本号
|
||||
4. 写入 `config-prod-snapshot`
|
||||
5. commit + push
|
||||
6. 更新 `sync_task` 和 `sync_checkpoint`
|
||||
|
||||
## 9. 当前接口假设
|
||||
|
||||
当前生产接口仍按以下假设实现：
|
||||
|
||||
- `push` 使用 `POST + multipart/form-data`
|
||||
- `pull` 使用 `GET`
|
||||
- `pull` 返回原始 JSON 字节流
|
||||
- 版本号优先取 `X-Config-Version`，其次 `ETag`
|
||||
|
||||
详细协议文档见：
|
||||
|
||||
- [prod-api-v1.md](e:/AIcoding/FtpTool/docs/prod-api-v1.md)
|
||||
|
||||
## 9.1 管理接口
|
||||
|
||||
当前已新增一组只读管理接口，用于查看最近同步状态和失败任务：
|
||||
|
||||
- `GET /api/admin/sync/overview`
|
||||
- `GET /api/admin/sync/tasks/recent`
|
||||
- `GET /api/admin/sync/tasks/failed`
|
||||
|
||||
对应实现：
|
||||
|
||||
- [SyncManagementController.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/web/SyncManagementController.java)
|
||||
|
||||
## 10. 当前主要风险
|
||||
|
||||
### 10.1 Git 写权限
|
||||
|
||||
如果生产环境对 Git 没有 push 权限，则“生产 -> Git”链路无法完成。
|
||||
|
||||
### 10.2 旧代码残留
|
||||
|
||||
当前主运行面已经切到 Git 直连，但源码树里仍保留少量退役占位类。
|
||||
|
||||
当前状态：
|
||||
|
||||
- 文件名和类名仍可能误导维护者
|
||||
- 旧架构源码主体已经删除
|
||||
- 主要剩余问题转为命名和文档口径统一
|
||||
|
||||
### 10.3 双向同步闭环
|
||||
|
||||
如果误把生产回写分支也作为下发分支扫描，会造成循环同步。
|
||||
|
||||
## 11. 推荐的下一轮改造
|
||||
|
||||
建议按下面顺序继续：
|
||||
|
||||
1. 删除或隔离 `dev-agent` 运行路径
|
||||
2. 删除退役标记文件 `application-dev-agent.properties`
|
||||
3. 统一清理残留的 `ftp` 命名
|
||||
4. 补充管理接口和健康检查接口
|
||||
5. 增加集成测试
|
||||
|
||||
## 12. 结论
|
||||
|
||||
当前系统已经从“FTP 中转”开始转向“Git 直连”。
|
||||
|
||||
现阶段最重要的不是继续增强旧链路，而是彻底收敛到：
|
||||
|
||||
- 单 `prod-agent`
|
||||
- 两个核心任务
|
||||
- 一个 Git 仓库入口
|
||||
- 一组生产 `push/pull` 接口
|
||||
后续再扩展功能时，应在这个模型上继续演进。
|
||||
|
||||
@ -1,253 +1,237 @@
|
||||
# Git 直连架构补充方案
|
||||
# Git 直连架构补充方案
|
||||
|
||||
## 1. 背景变化
|
||||
|
||||
旧方案的前提是：
|
||||
旧方案依赖：
|
||||
|
||||
- 开发环境和生产环境不能直接交换同步数据
|
||||
- 需要通过 `开发环境 -> FTP -> 生产环境` 中转
|
||||
- 开发环境 -> FTP -> 生产环境
|
||||
|
||||
现在条件已经变化为：
|
||||
当前条件已经变为：
|
||||
|
||||
- FTP 不再使用
|
||||
- 生产环境可以直接访问开发 Git 仓库
|
||||
- 生产环境可以直接访问开发 Git
|
||||
- 生产侧继续通过 `pushConfig/pullConfig/login` 接口与业务系统交互
|
||||
|
||||
这意味着旧架构里的 FTP 中转层已经没有存在价值，应该直接删除。
|
||||
|
||||
## 2. 新架构结论
|
||||
|
||||
推荐把原来的“双端代理 + FTP 中转”收敛为：
|
||||
因此当前架构已经收敛为：
|
||||
|
||||
- **单端代理 + Git 直连 + 本地状态库**
|
||||
|
||||
也就是只在生产环境部署一套同步服务：
|
||||
## 2. 新架构结论
|
||||
|
||||
- `Sync-Agent-Prod`
|
||||
|
||||
整体拓扑如下：
|
||||
|
||||
```text
|
||||
开发 Git 仓库 <----> 生产环境 Sync-Agent-Prod <----> 生产系统 push/pull 接口
|
||||
```
|
||||
|
||||
## 3. 为什么这样改
|
||||
|
||||
新架构比旧架构更合理，原因很直接：
|
||||
|
||||
- 去掉 FTP，中间链路减少一跳
|
||||
- 去掉打包上传、轮询下载、ACK 回执等中间机制
|
||||
- 部署节点从 2 个变成 1 个
|
||||
- 故障点减少，排查成本更低
|
||||
- 数据路径更直接，状态控制更简单
|
||||
|
||||
## 4. 新方案的核心思路
|
||||
|
||||
既然生产环境已经能访问开发 Git，那么同步动作都可以在生产环境完成。
|
||||
|
||||
生产环境部署的 `Sync-Agent-Prod` 同时承担两条链路：
|
||||
|
||||
### 4.1 开发 Git -> 生产
|
||||
|
||||
流程：
|
||||
|
||||
1. 定时拉取开发 Git 的配置分支
|
||||
2. 获取最新 commit 版本
|
||||
3. 判断该版本是否已同步
|
||||
4. 如果未同步，则读取配置内容
|
||||
5. 调用生产 `push` 接口导入配置
|
||||
6. 成功后更新本地检查点
|
||||
|
||||
### 4.2 生产 -> 开发 Git
|
||||
|
||||
流程：
|
||||
|
||||
1. 定时调用生产 `pull` 接口
|
||||
2. 获取当前生产配置快照
|
||||
3. 计算内容哈希或版本号
|
||||
4. 判断该快照是否已同步
|
||||
5. 如果未同步，则写入 Git 快照分支
|
||||
6. commit 并 push 到开发 Git
|
||||
7. 成功后更新本地检查点
|
||||
|
||||
## 5. 部署建议
|
||||
|
||||
### 5.1 推荐部署方式
|
||||
|
||||
推荐只保留：
|
||||
当前推荐只在生产环境部署一套同步服务：
|
||||
|
||||
- `prod-agent`
|
||||
|
||||
不再需要：
|
||||
整体拓扑：
|
||||
|
||||
- `dev-agent`
|
||||
- FTP 服务
|
||||
- FTP ACK、包中转、失败目录等机制
|
||||
```text
|
||||
开发 Git 仓库 <----> 生产环境 prod-agent <----> 生产系统 push/pull 接口
|
||||
```
|
||||
|
||||
### 5.2 运行前提
|
||||
## 3. Git 仓库模型
|
||||
|
||||
生产环境需要同时满足：
|
||||
### 3.1 版本分支
|
||||
|
||||
- 能读取开发 Git
|
||||
- 能向开发 Git 推送指定分支
|
||||
- 能调用生产 `push/pull` 接口
|
||||
- 能持久化 H2 文件数据库
|
||||
当前需求下：
|
||||
|
||||
这里有一个必须先确认的关键点：
|
||||
- **每个待下发版本对应一个 Git 分支**
|
||||
- `git.repo.scan-branch` 指向当前待同步的版本分支
|
||||
- **分支名本身就是 `configVersion`**
|
||||
|
||||
- **生产环境对开发 Git 是否有 push 权限**
|
||||
例如：
|
||||
|
||||
如果只有读权限，没有 push 权限，那么第二条链路“生产 -> 开发 Git”无法闭环。
|
||||
- `R_XXX_V3.0.3_XXX`
|
||||
- `R_XXX_V3.0.4_XXX`
|
||||
|
||||
## 6. Git 分支策略
|
||||
### 3.2 分支内目录结构
|
||||
|
||||
这个设计必须保留，不然非常容易形成同步闭环。
|
||||
每个版本分支内目录必须为：
|
||||
|
||||
建议继续使用两个分支：
|
||||
```text
|
||||
<airportId>/<appName>/<模块内文件相对路径>
|
||||
```
|
||||
|
||||
- `config-dev-main`：开发主配置分支
|
||||
- `config-prod-snapshot`：生产配置镜像分支
|
||||
示例：
|
||||
|
||||
同步规则：
|
||||
```text
|
||||
R_XXX_V3.0.3_XXX
|
||||
├─ PEK
|
||||
│ └─ monitor
|
||||
│ ├─ application.yml
|
||||
│ └─ jobs/sync-job.json
|
||||
└─ SHA
|
||||
└─ gate
|
||||
└─ gate-rule.json
|
||||
```
|
||||
|
||||
- `Git -> PROD` 只读 `config-dev-main`
|
||||
- `PROD -> Git` 只写 `config-prod-snapshot`
|
||||
### 3.3 当前快照分支
|
||||
|
||||
这样做的好处：
|
||||
当前实现使用“固定前缀 + 动态版本段”的快照分支：
|
||||
|
||||
- 避免生产回写内容再次触发生产下发
|
||||
- 生产快照不污染开发主线
|
||||
- 便于审计和回溯
|
||||
- `git.repo.snapshot-branch`
|
||||
|
||||
## 7. 状态管理
|
||||
`PROD -> Git` 会把生产拉回的配置写入：
|
||||
|
||||
虽然 FTP 没了，但本地状态库仍然必须保留。
|
||||
```text
|
||||
git.repo.snapshot-branch/<configVersion>
|
||||
```
|
||||
|
||||
建议继续保留：
|
||||
分支内目录结构同样为：
|
||||
|
||||
```text
|
||||
<airportId>/<appName>/<fileName>
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- `git.repo.snapshot-branch` 当前表示分支前缀，不再是唯一固定目标分支
|
||||
- 实际写入目标由 `sourceVersion/configVersion` 动态展开
|
||||
|
||||
## 4. 两条核心链路
|
||||
|
||||
### 4.1 Git -> PROD
|
||||
|
||||
流程：
|
||||
|
||||
1. `prod-agent` 拉取 `git.repo.scan-branch` 指定的版本分支
|
||||
2. 读取当前 `HEAD revision`，仅用于日志和 staging 隔离
|
||||
3. 以 **分支名** 作为 `sourceVersion/configVersion`
|
||||
4. 导出分支工作树
|
||||
5. 按 `airportId/appName/fileName` 解析所有配置文件
|
||||
6. 调用生产 `pushConfig`
|
||||
7. 成功后更新本地检查点和任务状态
|
||||
|
||||
### 4.2 PROD -> Git
|
||||
|
||||
流程：
|
||||
|
||||
1. `prod-agent` 调用生产 `pullConfig`
|
||||
2. 读取返回项中的 `airportId/appName/fileName`
|
||||
3. 恢复到本地 staging 目录
|
||||
4. 计算内容哈希和来源版本
|
||||
5. 提交到 `git.repo.snapshot-branch/<configVersion>`
|
||||
6. 成功后更新本地检查点和任务状态
|
||||
|
||||
## 5. 当前实现细节
|
||||
|
||||
### 5.1 Git -> PROD 版本语义
|
||||
|
||||
当前代码已经调整为：
|
||||
|
||||
- `sourceVersion = git.repo.scan-branch`
|
||||
- 不再使用 `commit SHA` 作为业务版本号
|
||||
|
||||
`HEAD revision` 仍然保留，但仅用于：
|
||||
|
||||
- staging 目录命名
|
||||
- 日志排查
|
||||
|
||||
### 5.2 增量推送
|
||||
|
||||
当前实现：
|
||||
|
||||
- 首次推送走全量
|
||||
- 后续和本地 baseline 做对比
|
||||
- 无删除时只推变更文件
|
||||
- 检测到删除时自动回退为全量
|
||||
|
||||
并且 baseline 已按版本分支隔离，避免不同版本分支之间相互污染。
|
||||
|
||||
### 5.3 pushConfig 参数映射
|
||||
|
||||
当前映射关系如下：
|
||||
|
||||
- `airportId` = Git 路径第 1 段
|
||||
- `appName` = Git 路径第 2 段
|
||||
- `fileName` = 模块内相对路径
|
||||
- `configVersion` = 当前版本分支名
|
||||
|
||||
### 5.4 pullConfig 回写结构
|
||||
|
||||
当前回写目录：
|
||||
|
||||
```text
|
||||
<airportId>/<appName>/<fileName>
|
||||
```
|
||||
|
||||
因此 `snapshotBranch` 应当被视为快照分支前缀，不建议在该前缀下混入其他开发分支。
|
||||
|
||||
## 6. 状态管理
|
||||
|
||||
当前继续保留：
|
||||
|
||||
- `sync_checkpoint`
|
||||
- `sync_task`
|
||||
- `prod_pull_ack`
|
||||
|
||||
`sync_ack` 在新架构下不再承担跨节点 ACK 作用。
|
||||
作用分别是：
|
||||
|
||||
当前建议：
|
||||
- `sync_checkpoint`：记录某个方向最后一次成功版本和哈希
|
||||
- `sync_task`：记录同步任务生命周期和重试状态
|
||||
- `prod_pull_ack`：记录下次 `pullConfig` 需要回传的 `ackSuc/ackFail`
|
||||
|
||||
- 已退出主 schema
|
||||
- 如后续需要审计，可独立恢复
|
||||
## 7. 幂等设计
|
||||
|
||||
## 8. 幂等设计
|
||||
|
||||
建议继续使用：
|
||||
当前幂等键仍然是：
|
||||
|
||||
```text
|
||||
direction + sourceVersion + contentHash
|
||||
```
|
||||
|
||||
作为业务幂等键。
|
||||
在 Git -> PROD 场景下：
|
||||
|
||||
作用：
|
||||
- `sourceVersion` = 版本分支名
|
||||
|
||||
- 同一个开发版本不会重复推生产
|
||||
- 同一个生产快照不会重复写 Git
|
||||
在 PROD -> Git 场景下：
|
||||
|
||||
## 9. 失败处理
|
||||
- `sourceVersion` 优先取 `pullConfig` 返回的统一 `configVersion`
|
||||
- 如果返回多版本混合数据，则退回为内容哈希
|
||||
|
||||
建议自动重试以下场景：
|
||||
## 8. 当前配置口径
|
||||
|
||||
- Git pull 失败
|
||||
- Git push 失败
|
||||
- 生产 `push` 接口失败
|
||||
- 生产 `pull` 接口失败
|
||||
### 8.1 Git 配置
|
||||
|
||||
建议策略：
|
||||
```properties
|
||||
git.repo.scan-branch=R_XXX_V3.0.3_XXX
|
||||
git.repo.snapshot-branch=config-prod-snapshot
|
||||
```
|
||||
|
||||
- 最大重试次数：`3 ~ 5`
|
||||
- 指数退避：`30s / 60s / 120s`
|
||||
说明：
|
||||
|
||||
失败后：
|
||||
- `scan-branch` 当前应直接配置为待同步版本分支名
|
||||
- `snapshot-branch` 当前表示动态快照分支前缀
|
||||
|
||||
- 更新 `sync_task` 状态
|
||||
- 保留错误信息
|
||||
- 不推进检查点
|
||||
### 8.2 生产接口配置
|
||||
|
||||
## 10. 安全建议
|
||||
```properties
|
||||
prod.api.base-url=https://prod.example.com
|
||||
prod.api.push-path=/pic_bus_manage_monitor/configSync/pushConfig
|
||||
prod.api.pull-path=/pic_bus_manage_monitor/configSync/pullConfig
|
||||
prod.api.login-path=/pic_bus_manage_monitor/pam-monitor/login
|
||||
prod.api.token=replace-me
|
||||
prod.api.token-header-name=token
|
||||
prod.api.airport-id=
|
||||
prod.api.app-name=
|
||||
prod.api.pull-config-version=
|
||||
prod.api.pull-file-name=
|
||||
```
|
||||
|
||||
### 10.1 Git 访问
|
||||
说明：
|
||||
|
||||
推荐使用：
|
||||
- `prod.api.airport-id`、`prod.api.app-name` 不再用于 Git -> PROD 参数解析
|
||||
- 它们当前只作为 `pullConfig` 的可选过滤参数
|
||||
- `prod.api.pull-config-version`、`prod.api.pull-file-name` 可用于精细过滤拉取范围
|
||||
|
||||
- HTTPS + Token
|
||||
## 9. 当前仍保留的 TODO
|
||||
|
||||
或：
|
||||
- 将 `ConfigCryptoService` 的透传实现替换为正式加解密算法
|
||||
|
||||
- SSH Deploy Key
|
||||
## 10. 结论
|
||||
|
||||
### 10.2 权限控制
|
||||
当前最重要的变化不是“继续抽象通用分支”，而是先把仓库约定和接口参数映射对齐：
|
||||
|
||||
生产环境访问 Git 的账号建议最小权限化：
|
||||
- 版本用分支表达
|
||||
- 机场和模块用目录表达
|
||||
- 生产接口参数由路径直接解析
|
||||
|
||||
- 对 `config-dev-main` 有读权限
|
||||
- 对 `config-prod-snapshot` 有写权限
|
||||
|
||||
更理想的做法：
|
||||
|
||||
- 使用专用机器人账号
|
||||
- 对主分支做保护
|
||||
- 机器人只写快照分支
|
||||
|
||||
## 11. 对现有代码的影响
|
||||
|
||||
这次需求变化对当前代码影响很大，结论如下：
|
||||
|
||||
### 11.1 可以继续保留的部分
|
||||
|
||||
- `GitClientService`
|
||||
- `ProdConfigApiService`
|
||||
- `SyncTaskService`
|
||||
- `CheckpointService`
|
||||
- H2 表结构
|
||||
- 定时任务框架
|
||||
|
||||
### 11.2 已退出主运行面的部分
|
||||
|
||||
- FTP 目录配置
|
||||
- 包上传/下载流程
|
||||
- ACK 文件机制
|
||||
- 双端部署假设
|
||||
|
||||
### 11.3 当前代码状态
|
||||
|
||||
当前代码已经收敛为：
|
||||
|
||||
- 一个正式运行角色 `prod-agent`
|
||||
- 两个正式调度任务
|
||||
|
||||
即：
|
||||
|
||||
1. `GitToProdSyncJob`
|
||||
2. `ProdToGitSnapshotJob`
|
||||
|
||||
旧架构源码主体已经删除，当前剩余问题主要是：
|
||||
|
||||
- 个别资源文件仍保留退役标记
|
||||
- 工程内少量 `ftp` 命名仍待统一
|
||||
|
||||
## 12. 结论
|
||||
|
||||
现在最合理的做法不是“在旧 FTP 方案上修修补补”，而是直接把架构收敛成：
|
||||
|
||||
- **生产环境单点部署**
|
||||
- **直连开发 Git**
|
||||
- **继续调用生产 `push/pull` 接口**
|
||||
- **保留 H2 做状态控制**
|
||||
|
||||
这是一次简化，不是一次退化。
|
||||
|
||||
## 13. 下一步建议
|
||||
|
||||
建议按这个顺序推进：
|
||||
|
||||
1. 先确认生产环境是否具备开发 Git 的 push 权限
|
||||
2. 确认生产 `push/pull` 接口最终协议
|
||||
3. 删除退役标记文件 `application-dev-agent.properties`
|
||||
4. 继续清理工程中残留的 `ftp` 命名
|
||||
5. 补充健康检查和管理能力
|
||||
这一层已经进入正式实现口径，后续文档和代码都应以此为准。
|
||||
|
||||
@ -1,33 +1,91 @@
|
||||
# 生产端配置同步接口文档 V1
|
||||
# 生产端配置同步接口文档 V1
|
||||
|
||||
## 1. 文档目的
|
||||
|
||||
本文档基于 `testapi.txt` 中给出的正式接口协议，定义生产端配置同步相关接口，供 Git 直连同步工具使用。
|
||||
本文档基于 `testapi.txt` 的正式协议，说明 Git 直连同步工具如何对接生产侧接口。
|
||||
|
||||
本文档覆盖三类接口：
|
||||
当前文档同时覆盖两层内容：
|
||||
|
||||
1. `pushConfig`：把 Git 配置推送到生产
|
||||
2. `pullConfig`：把生产配置拉回到 Git
|
||||
3. `login`：获取调用接口所需 token
|
||||
1. 接口协议本身
|
||||
2. 当前代码如何把 Git 仓库结构映射成接口参数
|
||||
|
||||
## 2. 统一约定
|
||||
## 2. 当前 Git 仓库约定
|
||||
|
||||
### 2.1 编码与格式
|
||||
### 2.1 Git -> PROD 输入约定
|
||||
|
||||
当前实现约定：
|
||||
|
||||
- `git.repo.scan-branch` 指向当前待同步的版本分支
|
||||
- **分支名本身就是 `configVersion`**
|
||||
- 分支内目录结构必须为：
|
||||
|
||||
```text
|
||||
<airportId>/<appName>/<模块内文件相对路径>
|
||||
```
|
||||
|
||||
示例：
|
||||
|
||||
```text
|
||||
R_XXX_V3.0.3_XXX
|
||||
├─ PEK
|
||||
│ └─ monitor
|
||||
│ ├─ application.yml
|
||||
│ └─ jobs/sync-job.json
|
||||
└─ SHA
|
||||
└─ gate
|
||||
└─ gate-rule.json
|
||||
```
|
||||
|
||||
映射规则：
|
||||
|
||||
- `airportId` = 路径第 1 段
|
||||
- `appName` = 路径第 2 段
|
||||
- `fileName` = 从第 3 段开始的模块内相对路径
|
||||
- `configVersion` = 当前 Git 分支名
|
||||
|
||||
例如文件：
|
||||
|
||||
```text
|
||||
PEK/monitor/jobs/sync-job.json
|
||||
```
|
||||
|
||||
会被映射为：
|
||||
|
||||
```json
|
||||
{
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"fileName": "jobs/sync-job.json"
|
||||
}
|
||||
```
|
||||
|
||||
### 2.2 PROD -> Git 输出约定
|
||||
|
||||
当前实现会把 `pullConfig` 返回项按版本分组后落到本地 staging 目录，目录结构同样为：
|
||||
|
||||
```text
|
||||
<airportId>/<appName>/<fileName>
|
||||
```
|
||||
|
||||
之后按动态分支提交到 `git.repo.snapshot-branch/<configVersion>`。
|
||||
|
||||
## 3. 统一接口约定
|
||||
|
||||
### 3.1 编码与格式
|
||||
|
||||
- 编码：`UTF-8`
|
||||
- 返回格式：`JSON`
|
||||
- 鉴权方式：请求头中携带 `token`
|
||||
- 鉴权：请求头中携带 `token`
|
||||
|
||||
### 2.2 成功判定
|
||||
### 3.2 成功判定
|
||||
|
||||
接口成功时返回：
|
||||
|
||||
- `code = "0"`
|
||||
- `msg = "ok"`
|
||||
|
||||
### 2.3 失败响应格式
|
||||
|
||||
所有接口失败时的响应格式统一如下：
|
||||
### 3.3 失败响应格式
|
||||
|
||||
```json
|
||||
{
|
||||
@ -38,31 +96,31 @@
|
||||
}
|
||||
```
|
||||
|
||||
## 3. 接口一：推送 Git 配置到生产
|
||||
## 4. 接口一：推送 Git 配置到生产
|
||||
|
||||
### 3.1 基本信息
|
||||
### 4.1 基本信息
|
||||
|
||||
- 地址：`http://ip:port/pic_bus_manage_monitor/configSync/pushConfig`
|
||||
- 方法：`POST`
|
||||
- 内容格式：`JSON`
|
||||
- 鉴权：Header 中携带 `token`
|
||||
|
||||
### 3.2 请求参数
|
||||
### 4.2 请求参数
|
||||
|
||||
无 URL 参数。
|
||||
|
||||
### 3.3 请求体
|
||||
### 4.3 请求体
|
||||
|
||||
请求体为 JSON 数组，每个数组元素代表一个配置文件：
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"airportId": "test",
|
||||
"appName": "XXX",
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"configContent": "配置内容",
|
||||
"fileName": "配置文件名"
|
||||
"fileName": "jobs/sync-job.json"
|
||||
}
|
||||
]
|
||||
```
|
||||
@ -71,13 +129,13 @@
|
||||
|
||||
| 字段 | 必填 | 说明 |
|
||||
| --- | --- | --- |
|
||||
| `airportId` | 是 | 机场编码 |
|
||||
| `appName` | 是 | 应用名称 |
|
||||
| `configVersion` | 是 | 配置版本号 |
|
||||
| `configContent` | 是 | 配置内容 |
|
||||
| `fileName` | 是 | 配置文件名 |
|
||||
| `airportId` | 是 | 机场编码，来自 Git 路径第 1 段 |
|
||||
| `appName` | 是 | 模块名，来自 Git 路径第 2 段 |
|
||||
| `configVersion` | 是 | 配置版本号，当前实现取 Git 分支名 |
|
||||
| `configContent` | 是 | 文件内容，当前实现按 UTF-8 文本直接读取 |
|
||||
| `fileName` | 是 | 模块内文件相对路径，不包含 `airportId/appName` 前缀 |
|
||||
|
||||
### 3.4 响应体
|
||||
### 4.4 响应体
|
||||
|
||||
```json
|
||||
{
|
||||
@ -85,10 +143,10 @@
|
||||
"data": {
|
||||
"ackFail": [
|
||||
{
|
||||
"airportId": "test",
|
||||
"appName": "XXXx",
|
||||
"configVersion": "R_XXX_V3.0.35.6.1_XXX",
|
||||
"fileName": "配置文件名"
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"fileName": "jobs/sync-job.json"
|
||||
}
|
||||
]
|
||||
},
|
||||
@ -100,48 +158,65 @@
|
||||
|
||||
- `ackFail`：本次推送失败的配置项列表
|
||||
- 如果 `ackFail` 为空或不存在，可视为本次推送成功
|
||||
- 如果 `ackFail` 非空，应视为部分失败或失败
|
||||
- 如果 `ackFail` 非空，当前代码直接抛错，视为本次同步失败
|
||||
|
||||
### 3.5 业务说明
|
||||
### 4.5 当前实现说明
|
||||
|
||||
- 第一次推送为全量
|
||||
- 之后按增量推送
|
||||
- 如果全量配置过大，需要视情况拆分多次推送
|
||||
- `configContent` 需要加密
|
||||
- **加密方式当前留为 `TODO`**
|
||||
- 第一次推送走全量
|
||||
- 后续优先走最小增量
|
||||
- 如果检测到文件删除，会自动回退到全量推送
|
||||
- 基线目录按版本分支隔离保存
|
||||
- `configContent` 当前已统一经过 `ConfigCryptoService`
|
||||
- 默认实现仍为透传，正式加密算法待补
|
||||
|
||||
## 4. 接口二：从生产拉取配置到 Git
|
||||
## 5. 接口二：从生产拉取配置到 Git
|
||||
|
||||
### 4.1 基本信息
|
||||
### 5.1 基本信息
|
||||
|
||||
- 地址：`http://ip:port/pic_bus_manage_monitor/configSync/pullConfig`
|
||||
- 方法：`GET`
|
||||
- 内容格式：`JSON`
|
||||
- 鉴权：Header 中携带 `token`
|
||||
|
||||
### 4.2 请求参数
|
||||
### 5.2 请求参数
|
||||
|
||||
接口文档给出的请求参数如下：
|
||||
接口协议中的字段如下：
|
||||
|
||||
```json
|
||||
{
|
||||
"airportId": "test",
|
||||
"appName": "XXXx",
|
||||
"configVersion": "R_XXX_V3.0.35.6.1_XXX",
|
||||
"fileName": "配置文件名",
|
||||
"ackSuc": "id,id",
|
||||
"ackFail": "id,id"
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"fileName": "jobs/sync-job.json",
|
||||
"ackSuc": "1,2",
|
||||
"ackFail": "9"
|
||||
}
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 由于该接口是 `GET`，当前实现按查询参数方式理解这些字段
|
||||
- 当前代码只使用 `airportId`、`appName` 做基础过滤
|
||||
- `ackSuc`、`ackFail` 暂未纳入当前实现
|
||||
- 这部分如果后续要完全对齐，可继续补充
|
||||
- 由于接口是 `GET`，当前实现按 query string 传参
|
||||
- 当前实现已经支持：
|
||||
- `airportId` 可选过滤
|
||||
- `appName` 可选过滤
|
||||
- `ackSuc` 自动回传
|
||||
- `ackFail` 自动回传
|
||||
- 当前已经支持：
|
||||
- `configVersion` 可选过滤
|
||||
- `fileName` 可选过滤
|
||||
|
||||
### 4.3 响应体
|
||||
### 5.3 当前请求行为
|
||||
|
||||
当前代码只有在配置了过滤条件时才会发送：
|
||||
|
||||
- `prod.api.airport-id`
|
||||
- `prod.api.app-name`
|
||||
- `prod.api.pull-config-version`
|
||||
- `prod.api.pull-file-name`
|
||||
|
||||
如果这两个配置为空，则不会在请求中携带它们，此时依赖生产端返回所有已审核且未同步的配置。
|
||||
|
||||
### 5.4 响应体
|
||||
|
||||
```json
|
||||
{
|
||||
@ -149,11 +224,11 @@
|
||||
"data": [
|
||||
{
|
||||
"id": "1",
|
||||
"airportId": "test",
|
||||
"appName": "XXXx",
|
||||
"configVersion": "R_XXX_V3.0.35.6.1_XXX",
|
||||
"airportId": "PEK",
|
||||
"appName": "monitor",
|
||||
"configVersion": "R_XXX_V3.0.3_XXX",
|
||||
"configContent": "配置内容（加密）",
|
||||
"fileName": "配置文件名"
|
||||
"fileName": "jobs/sync-job.json"
|
||||
}
|
||||
],
|
||||
"msg": "ok"
|
||||
@ -166,27 +241,28 @@
|
||||
| --- | --- |
|
||||
| `id` | 配置记录标识 |
|
||||
| `airportId` | 机场编码 |
|
||||
| `appName` | 应用名称 |
|
||||
| `appName` | 模块名 |
|
||||
| `configVersion` | 配置版本号 |
|
||||
| `configContent` | 配置内容 |
|
||||
| `fileName` | 配置文件名 |
|
||||
| `fileName` | 模块内文件相对路径 |
|
||||
|
||||
### 4.4 业务说明
|
||||
### 5.5 当前实现说明
|
||||
|
||||
- 不带请求参数时，返回所有“未同步到 Git 且已审核通过”的配置
|
||||
- `configContent` 为加密内容
|
||||
- **解密方式当前留为 `TODO`**
|
||||
- `pullConfig` 返回项会先按 `configVersion` 分组，再分别落盘为 `airportId/appName/fileName`
|
||||
- 如果某组内所有项的 `configVersion` 相同，则该值作为该组的 `sourceVersion`
|
||||
- 如果某组缺少统一版本号，则该组退回为内容哈希作为 `sourceVersion`
|
||||
- `configContent` 当前已统一经过 `ConfigCryptoService`，默认实现仍为透传
|
||||
- 当本地处理失败时，会把失败项写入 ACK 重试表，并在下次按 `airportId/appName/configVersion/fileName` 定向重拉
|
||||
|
||||
## 5. 接口三：获取 token
|
||||
## 6. 接口三：获取 token
|
||||
|
||||
### 5.1 基本信息
|
||||
### 6.1 基本信息
|
||||
|
||||
- 地址：`http://ip:port/pic_bus_manage_monitor/pam-monitor/login`
|
||||
- 方法：`POST`
|
||||
- 内容格式：`JSON`
|
||||
- 说明：该接口已存在，用于获取配置同步接口所需 token
|
||||
|
||||
### 5.2 请求体
|
||||
### 6.2 请求体
|
||||
|
||||
```json
|
||||
{
|
||||
@ -195,7 +271,7 @@
|
||||
}
|
||||
```
|
||||
|
||||
### 5.3 响应体
|
||||
### 6.3 响应体
|
||||
|
||||
```json
|
||||
{
|
||||
@ -208,40 +284,34 @@
|
||||
}
|
||||
```
|
||||
|
||||
字段说明：
|
||||
### 6.4 当前实现说明
|
||||
|
||||
| 字段 | 说明 |
|
||||
| --- | --- |
|
||||
| `token` | 鉴权 token |
|
||||
| `expireTime` | token 过期时间 |
|
||||
- 如果 `prod.api.token` 已显式配置，则优先使用静态 token
|
||||
- 如果未配置静态 token，则调用 `login` 接口获取 token
|
||||
- token 会按 `expireTime` 做本地缓存
|
||||
|
||||
### 5.4 业务说明
|
||||
## 7. 当前代码实现对齐情况
|
||||
|
||||
- 如果 token 过期，需要重新调用登录接口获取新 token
|
||||
|
||||
## 6. 当前代码实现对齐情况
|
||||
|
||||
当前代码实现文件：
|
||||
当前接口适配实现：
|
||||
|
||||
- [ProdConfigApiService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/ProdConfigApiService.java)
|
||||
|
||||
当前实现已经对齐到以下协议：
|
||||
当前已经对齐到以下协议：
|
||||
|
||||
- `pushConfig` 使用 `POST + JSON 数组`
|
||||
- `pullConfig` 使用 `GET + JSON 响应`
|
||||
- `login` 使用 `POST + JSON`
|
||||
- 请求头默认使用 `token` 作为 token 头名称
|
||||
- `token` 未静态配置时，会自动调用登录接口获取并缓存
|
||||
- `ackSuc/ackFail` 已接入请求回传和本地状态更新
|
||||
- `pushConfig` 参数已按 `airportId/appName/fileName` 目录结构解析
|
||||
- `pullConfig` 响应已按 `airportId/appName/fileName` 结构恢复到本地目录
|
||||
|
||||
当前仍保留的 `TODO`：
|
||||
|
||||
- `configContent` 推送前加密
|
||||
- `configContent` 拉取后解密
|
||||
- `pullConfig` 中 `ackSuc/ackFail` 参数的完整处理
|
||||
- 将 `ConfigCryptoService` 的透传实现替换为正式加解密算法
|
||||
|
||||
## 7. 当前配置项
|
||||
## 8. 当前配置项说明
|
||||
|
||||
为配合上述接口，当前配置项建议如下：
|
||||
建议配置：
|
||||
|
||||
```properties
|
||||
prod.api.base-url=https://prod.example.com
|
||||
@ -250,23 +320,27 @@ prod.api.pull-path=/pic_bus_manage_monitor/configSync/pullConfig
|
||||
prod.api.login-path=/pic_bus_manage_monitor/pam-monitor/login
|
||||
prod.api.token=replace-me
|
||||
prod.api.token-header-name=token
|
||||
prod.api.airport-id=replace-me
|
||||
prod.api.app-name=replace-me
|
||||
prod.api.airport-id=
|
||||
prod.api.app-name=
|
||||
prod.api.pull-config-version=
|
||||
prod.api.pull-file-name=
|
||||
prod.api.login-name=
|
||||
prod.api.login-password=
|
||||
```
|
||||
|
||||
说明：
|
||||
|
||||
- 如果 `prod.api.token` 已直接配置，则当前实现优先使用静态 token
|
||||
- 如果没有配置 token，则走 `login` 接口获取 token
|
||||
- `prod.api.airport-id`、`prod.api.app-name` **不再用于 pushConfig 参数组装**
|
||||
- 它们当前只作为 `pullConfig` 的可选过滤条件
|
||||
- `prod.api.pull-config-version`、`prod.api.pull-file-name` 可用于精细过滤拉取范围
|
||||
- 如果希望拉全量已审核数据，建议留空
|
||||
|
||||
## 8. 联调建议
|
||||
## 9. 联调建议
|
||||
|
||||
联调时建议优先确认以下几点：
|
||||
联调时建议优先确认：
|
||||
|
||||
1. `token` 请求头名称是否就是 `token`
|
||||
2. `pullConfig` 的 GET 请求参数是否确实按 query string 传递
|
||||
3. `ackFail` 非空时是否要整体视为失败
|
||||
2. `pullConfig` 的 GET 参数是否按 query string 传递
|
||||
3. `configVersion/fileName` 过滤规则是否需要尽快接入
|
||||
4. `configContent` 加密/解密算法何时补齐
|
||||
5. `fileName` 是否可能包含目录层级
|
||||
5. `fileName` 是否始终是模块内相对路径，而不是包含 `airportId/appName`
|
||||
|
||||
@ -3,34 +3,21 @@ package com.ftptool.sync.config;
|
||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
||||
|
||||
@ConfigurationProperties(prefix = "prod.api")
|
||||
/**
|
||||
* 生产系统 push/pull 接口配置。
|
||||
*/
|
||||
public class ProdApiProperties {
|
||||
|
||||
/** 生产接口基础地址。 */
|
||||
private String baseUrl;
|
||||
/** 配置导入接口路径。 */
|
||||
private String pushPath;
|
||||
/** 生产快照导出接口路径。 */
|
||||
private String pullPath;
|
||||
/** 获取 token 的登录接口路径。 */
|
||||
private String loginPath;
|
||||
/** 访问生产接口的 Bearer Token。 */
|
||||
private String token;
|
||||
/** 放置 token 的请求头名称，默认按对方接口约定使用 token。 */
|
||||
private String tokenHeaderName = "token";
|
||||
/** 业务归属机场编码。 */
|
||||
private String airportId;
|
||||
/** 业务应用名称。 */
|
||||
private String appName;
|
||||
/** 登录接口用户名。 */
|
||||
private String pullConfigVersion;
|
||||
private String pullFileName;
|
||||
private String loginName;
|
||||
/** 登录接口密码。 */
|
||||
private String loginPassword;
|
||||
/** HTTP 连接超时。 */
|
||||
private int connectTimeoutMs = 10000;
|
||||
/** HTTP 读取超时。 */
|
||||
private int readTimeoutMs = 30000;
|
||||
|
||||
public String getBaseUrl() {
|
||||
@ -97,6 +84,22 @@ public class ProdApiProperties {
|
||||
this.appName = appName;
|
||||
}
|
||||
|
||||
public String getPullConfigVersion() {
|
||||
return pullConfigVersion;
|
||||
}
|
||||
|
||||
public void setPullConfigVersion(String pullConfigVersion) {
|
||||
this.pullConfigVersion = pullConfigVersion;
|
||||
}
|
||||
|
||||
public String getPullFileName() {
|
||||
return pullFileName;
|
||||
}
|
||||
|
||||
public void setPullFileName(String pullFileName) {
|
||||
this.pullFileName = pullFileName;
|
||||
}
|
||||
|
||||
public String getLoginName() {
|
||||
return loginName;
|
||||
}
|
||||
|
||||
@ -9,6 +9,7 @@ import javax.persistence.Enumerated;
|
||||
import javax.persistence.GeneratedValue;
|
||||
import javax.persistence.GenerationType;
|
||||
import javax.persistence.Id;
|
||||
import javax.persistence.Lob;
|
||||
import javax.persistence.PrePersist;
|
||||
import javax.persistence.PreUpdate;
|
||||
import javax.persistence.Table;
|
||||
@ -16,8 +17,7 @@ import javax.persistence.UniqueConstraint;
|
||||
import java.time.LocalDateTime;
|
||||
|
||||
/**
|
||||
* pullConfig 回执记录。
|
||||
* 用于记录需要在下一次 pull 请求里回传给生产端的 ackSuc/ackFail。
|
||||
* Tracks ackSuc/ackFail status and retry metadata for pulled production configs.
|
||||
*/
|
||||
@Entity
|
||||
@Table(name = "prod_pull_ack", uniqueConstraints = {
|
||||
@ -29,24 +29,41 @@ public class ProdPullAckRecord {
|
||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
||||
private Long id;
|
||||
|
||||
/** 生产端返回的配置项 id。 */
|
||||
@Column(name = "remote_config_id", nullable = false, length = 128)
|
||||
private String remoteConfigId;
|
||||
|
||||
/** 回执状态。 */
|
||||
@Enumerated(EnumType.STRING)
|
||||
@Column(name = "ack_status", nullable = false, length = 16)
|
||||
private ProdPullAckStatus ackStatus;
|
||||
|
||||
/** 是否已经在 pullConfig 请求里成功回传给生产端。 */
|
||||
@Column(name = "reported", nullable = false)
|
||||
private Boolean reported;
|
||||
|
||||
/** 创建时间。 */
|
||||
@Column(name = "source_version", length = 128)
|
||||
private String sourceVersion;
|
||||
|
||||
@Column(name = "airport_id", length = 128)
|
||||
private String airportId;
|
||||
|
||||
@Column(name = "app_name", length = 128)
|
||||
private String appName;
|
||||
|
||||
@Column(name = "file_name", length = 512)
|
||||
private String fileName;
|
||||
|
||||
@Column(name = "retry_count", nullable = false)
|
||||
private Integer retryCount;
|
||||
|
||||
@Column(name = "next_retry_at")
|
||||
private LocalDateTime nextRetryAt;
|
||||
|
||||
@Lob
|
||||
@Column(name = "last_error_msg")
|
||||
private String lastErrorMsg;
|
||||
|
||||
@Column(name = "created_at", nullable = false)
|
||||
private LocalDateTime createdAt;
|
||||
|
||||
/** 更新时间。 */
|
||||
@Column(name = "updated_at", nullable = false)
|
||||
private LocalDateTime updatedAt;
|
||||
|
||||
@ -58,6 +75,9 @@ public class ProdPullAckRecord {
|
||||
if (this.reported == null) {
|
||||
this.reported = Boolean.FALSE;
|
||||
}
|
||||
if (this.retryCount == null) {
|
||||
this.retryCount = 0;
|
||||
}
|
||||
}
|
||||
|
||||
@PreUpdate
|
||||
@ -97,6 +117,62 @@ public class ProdPullAckRecord {
|
||||
this.reported = reported;
|
||||
}
|
||||
|
||||
public String getSourceVersion() {
|
||||
return sourceVersion;
|
||||
}
|
||||
|
||||
public void setSourceVersion(String sourceVersion) {
|
||||
this.sourceVersion = sourceVersion;
|
||||
}
|
||||
|
||||
public String getAirportId() {
|
||||
return airportId;
|
||||
}
|
||||
|
||||
public void setAirportId(String airportId) {
|
||||
this.airportId = airportId;
|
||||
}
|
||||
|
||||
public String getAppName() {
|
||||
return appName;
|
||||
}
|
||||
|
||||
public void setAppName(String appName) {
|
||||
this.appName = appName;
|
||||
}
|
||||
|
||||
public String getFileName() {
|
||||
return fileName;
|
||||
}
|
||||
|
||||
public void setFileName(String fileName) {
|
||||
this.fileName = fileName;
|
||||
}
|
||||
|
||||
public Integer getRetryCount() {
|
||||
return retryCount;
|
||||
}
|
||||
|
||||
public void setRetryCount(Integer retryCount) {
|
||||
this.retryCount = retryCount;
|
||||
}
|
||||
|
||||
public LocalDateTime getNextRetryAt() {
|
||||
return nextRetryAt;
|
||||
}
|
||||
|
||||
public void setNextRetryAt(LocalDateTime nextRetryAt) {
|
||||
this.nextRetryAt = nextRetryAt;
|
||||
}
|
||||
|
||||
public String getLastErrorMsg() {
|
||||
return lastErrorMsg;
|
||||
}
|
||||
|
||||
public void setLastErrorMsg(String lastErrorMsg) {
|
||||
this.lastErrorMsg = lastErrorMsg;
|
||||
}
|
||||
|
||||
public LocalDateTime getCreatedAt() {
|
||||
return createdAt;
|
||||
}
|
||||
|
||||
@ -1,27 +1,41 @@
|
||||
package com.ftptool.sync.model;
|
||||
|
||||
import java.nio.file.Path;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* 生产 pull 接口返回结果在本地落盘后的封装对象。
|
||||
* Local representation of one pulled production snapshot group.
|
||||
*/
|
||||
public class ProdPullResult {
|
||||
|
||||
/** 保存生产快照内容的目录。 */
|
||||
private final Path contentDirectory;
|
||||
/** 来源版本号，优先取服务端显式返回值。 */
|
||||
private final String sourceVersion;
|
||||
/** 响应体内容哈希。 */
|
||||
private final String contentHash;
|
||||
/** 本次 pull 返回的生产配置项 id 列表。 */
|
||||
private final List<String> pulledConfigIds;
|
||||
private final List<PulledConfigRef> pulledConfigs;
|
||||
|
||||
public ProdPullResult(Path contentDirectory, String sourceVersion, String contentHash, List<String> pulledConfigIds) {
|
||||
this(contentDirectory, sourceVersion, contentHash, pulledConfigIds, buildIdOnlyRefs(pulledConfigIds));
|
||||
}
|
||||
|
||||
public ProdPullResult(
|
||||
Path contentDirectory,
|
||||
String sourceVersion,
|
||||
String contentHash,
|
||||
List<String> pulledConfigIds,
|
||||
List<PulledConfigRef> pulledConfigs
|
||||
) {
|
||||
this.contentDirectory = contentDirectory;
|
||||
this.sourceVersion = sourceVersion;
|
||||
this.contentHash = contentHash;
|
||||
this.pulledConfigIds = pulledConfigIds;
|
||||
this.pulledConfigIds = pulledConfigIds == null
|
||||
? Collections.<String>emptyList()
|
||||
: Collections.unmodifiableList(new ArrayList<String>(pulledConfigIds));
|
||||
this.pulledConfigs = pulledConfigs == null
|
||||
? Collections.<PulledConfigRef>emptyList()
|
||||
: Collections.unmodifiableList(new ArrayList<PulledConfigRef>(pulledConfigs));
|
||||
}
|
||||
|
||||
public Path getContentDirectory() {
|
||||
@ -39,4 +53,63 @@ public class ProdPullResult {
|
||||
public List<String> getPulledConfigIds() {
|
||||
return pulledConfigIds;
|
||||
}
|
||||
|
||||
public List<PulledConfigRef> getPulledConfigs() {
|
||||
return pulledConfigs;
|
||||
}
|
||||
|
||||
private static List<PulledConfigRef> buildIdOnlyRefs(List<String> pulledConfigIds) {
|
||||
if (pulledConfigIds == null) {
|
||||
return Collections.emptyList();
|
||||
}
|
||||
|
||||
List<PulledConfigRef> refs = new ArrayList<PulledConfigRef>();
|
||||
for (String pulledConfigId : pulledConfigIds) {
|
||||
refs.add(new PulledConfigRef(pulledConfigId, null, null, null, null));
|
||||
}
|
||||
return refs;
|
||||
}
|
||||
|
||||
public static class PulledConfigRef {
|
||||
|
||||
private final String remoteConfigId;
|
||||
private final String airportId;
|
||||
private final String appName;
|
||||
private final String configVersion;
|
||||
private final String fileName;
|
||||
|
||||
public PulledConfigRef(
|
||||
String remoteConfigId,
|
||||
String airportId,
|
||||
String appName,
|
||||
String configVersion,
|
||||
String fileName
|
||||
) {
|
||||
this.remoteConfigId = remoteConfigId;
|
||||
this.airportId = airportId;
|
||||
this.appName = appName;
|
||||
this.configVersion = configVersion;
|
||||
this.fileName = fileName;
|
||||
}
|
||||
|
||||
public String getRemoteConfigId() {
|
||||
return remoteConfigId;
|
||||
}
|
||||
|
||||
public String getAirportId() {
|
||||
return airportId;
|
||||
}
|
||||
|
||||
public String getAppName() {
|
||||
return appName;
|
||||
}
|
||||
|
||||
public String getConfigVersion() {
|
||||
return configVersion;
|
||||
}
|
||||
|
||||
public String getFileName() {
|
||||
return fileName;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -12,11 +12,13 @@ import com.ftptool.sync.model.SyncStatus;
|
||||
import com.ftptool.sync.service.CheckpointService;
|
||||
import com.ftptool.sync.service.GitClientService;
|
||||
import com.ftptool.sync.service.PackageService;
|
||||
import com.ftptool.sync.service.ProdPullAckService;
|
||||
import com.ftptool.sync.service.ProdConfigApiService;
|
||||
import com.ftptool.sync.service.ProdPullAckService;
|
||||
import com.ftptool.sync.service.SyncMetadataService;
|
||||
import com.ftptool.sync.service.SyncTaskService;
|
||||
import com.ftptool.sync.service.WorkDirectoryService;
|
||||
import com.ftptool.sync.util.FileHashUtils;
|
||||
import com.ftptool.sync.util.FileTreeUtils;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.context.annotation.Profile;
|
||||
@ -34,12 +36,8 @@ import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
import com.ftptool.sync.util.FileHashUtils;
|
||||
import com.ftptool.sync.util.FileTreeUtils;
|
||||
|
||||
/**
|
||||
* 生产侧同步协调器。
|
||||
* 串联两条核心链路：
|
||||
* Coordinates the two production-side sync flows:
|
||||
* 1. Git -> PROD
|
||||
* 2. PROD -> Git
|
||||
*/
|
||||
@ -88,23 +86,25 @@ public class ProdSyncCoordinator {
|
||||
}
|
||||
|
||||
/**
|
||||
* 主链路一：从 Git 拉取最新配置并推送到生产。
|
||||
* Pull the configured Git version branch and push its config files to PROD.
|
||||
*/
|
||||
public void syncLatestGitToProd() {
|
||||
String traceId = null;
|
||||
try {
|
||||
String branch = gitRepoProperties.getScanBranch();
|
||||
String sourceRevision = gitClientService.prepareRepositoryAndGetHead(branch);
|
||||
String sourceVersion = branch;
|
||||
String stagingKey = buildStagingKey(branch, sourceRevision);
|
||||
|
||||
log.info(
|
||||
"PROD git->prod tick. nodeId={}, branch={}, pushPath={}",
|
||||
"PROD git->prod tick. nodeId={}, branch={}, revision={}, pushPath={}",
|
||||
syncProperties.getNodeId(),
|
||||
gitRepoProperties.getScanBranch(),
|
||||
branch,
|
||||
sourceRevision,
|
||||
prodApiProperties.getPushPath()
|
||||
);
|
||||
|
||||
String branch = gitRepoProperties.getScanBranch();
|
||||
String sourceVersion = gitClientService.prepareRepositoryAndGetHead(branch);
|
||||
Path exportDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-" + sourceVersion);
|
||||
|
||||
// 先导出 Git 工作树快照，再计算内容哈希，避免直接拿工作目录参与后续修改。
|
||||
Path exportDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-" + stagingKey);
|
||||
gitClientService.exportBranchSnapshot(branch, exportDirectory);
|
||||
String contentHash = packageService.calculateDirectoryHash(exportDirectory);
|
||||
|
||||
@ -114,7 +114,8 @@ public class ProdSyncCoordinator {
|
||||
contentHash
|
||||
);
|
||||
if (shouldSkip(existing)) {
|
||||
log.info("Git version already pushed to prod. version={}, hash={}", sourceVersion, contentHash);
|
||||
log.info("Git version already pushed to prod. version={}, revision={}, hash={}",
|
||||
sourceVersion, sourceRevision, contentHash);
|
||||
return;
|
||||
}
|
||||
|
||||
@ -138,34 +139,49 @@ public class ProdSyncCoordinator {
|
||||
traceId
|
||||
);
|
||||
|
||||
// Git 提交哈希 + 内容哈希作为业务幂等键，避免同一版本重复推送到生产。
|
||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.CONSUMING, null);
|
||||
Path pushDirectory = preparePushDirectory(exportDirectory, sourceVersion);
|
||||
Path pushDirectory = preparePushDirectory(exportDirectory, branch, stagingKey);
|
||||
prodConfigApiService.pushPackage(manifest, pushDirectory);
|
||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
||||
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
||||
refreshGitToProdBaseline(exportDirectory);
|
||||
log.info("Git version pushed to prod successfully. traceId={}, version={}", task.getTraceId(), task.getSourceVersion());
|
||||
refreshGitToProdBaseline(exportDirectory, branch);
|
||||
log.info("Git version pushed to prod successfully. traceId={}, version={}, revision={}",
|
||||
task.getTraceId(), task.getSourceVersion(), sourceRevision);
|
||||
} catch (Exception e) {
|
||||
handleFailure(traceId, "PROD git->prod sync failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 主链路二：从生产拉取当前配置并回写到 Git 快照分支。
|
||||
* Pull the current production snapshot and write it back to the Git snapshot branch.
|
||||
*/
|
||||
public void syncProdSnapshotToGit() {
|
||||
String traceId = null;
|
||||
ProdPullResult pullResult = null;
|
||||
try {
|
||||
log.info(
|
||||
"PROD prod->git tick. apiBaseUrl={}, pullPath={}, snapshotBranch={}",
|
||||
"PROD prod->git tick. apiBaseUrl={}, pullPath={}, snapshotBranchPrefix={}",
|
||||
prodApiProperties.getBaseUrl(),
|
||||
prodApiProperties.getPullPath(),
|
||||
gitRepoProperties.getSnapshotBranch()
|
||||
);
|
||||
|
||||
pullResult = prodConfigApiService.pullConfigSnapshot();
|
||||
retryFailedProdPulls();
|
||||
|
||||
try {
|
||||
List<ProdPullResult> pullResults = prodConfigApiService.pullConfigSnapshots();
|
||||
for (ProdPullResult pullResult : pullResults) {
|
||||
syncSingleProdSnapshotToGit(pullResult, false);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
log.error("PROD prod->git sync failed before new snapshot groups were processed", e);
|
||||
}
|
||||
}
|
||||
|
||||
private boolean shouldSkip(Optional<SyncTask> existing) {
|
||||
return existing.isPresent() && existing.get().getStatus() == SyncStatus.SUCCESS;
|
||||
}
|
||||
|
||||
private void syncSingleProdSnapshotToGit(ProdPullResult pullResult, boolean retryAttempt) {
|
||||
String traceId = null;
|
||||
try {
|
||||
Optional<SyncTask> existing = syncTaskService.findByBusinessKey(
|
||||
SyncDirection.PROD_TO_DEV,
|
||||
pullResult.getSourceVersion(),
|
||||
@ -174,6 +190,7 @@ public class ProdSyncCoordinator {
|
||||
if (shouldSkip(existing)) {
|
||||
log.info("Production snapshot already synced to Git. version={}, hash={}",
|
||||
pullResult.getSourceVersion(), pullResult.getContentHash());
|
||||
prodPullAckService.recordAckSuccess(pullResult.getPulledConfigs());
|
||||
return;
|
||||
}
|
||||
|
||||
@ -187,48 +204,72 @@ public class ProdSyncCoordinator {
|
||||
);
|
||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.CONSUMING, null);
|
||||
|
||||
// 生产快照只写入独立 snapshot 分支，避免与开发主分支形成闭环。
|
||||
String targetBranch = resolveSnapshotBranch(task.getSourceVersion());
|
||||
String commitMessage = gitRepoProperties.getCommitMessagePrefix()
|
||||
+ ": traceId=" + task.getTraceId()
|
||||
+ " version=" + task.getSourceVersion();
|
||||
boolean pushed = gitClientService.syncDirectoryToBranch(
|
||||
pullResult.getContentDirectory(),
|
||||
gitRepoProperties.getSnapshotBranch(),
|
||||
targetBranch,
|
||||
commitMessage
|
||||
);
|
||||
|
||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
||||
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
||||
prodPullAckService.recordAckResult(pullResult.getPulledConfigIds(), ProdPullAckStatus.SUCCESS);
|
||||
prodPullAckService.recordAckSuccess(pullResult.getPulledConfigs());
|
||||
log.info(
|
||||
"Production snapshot synced to Git. traceId={}, version={}, gitPushed={}",
|
||||
"Production snapshot synced to Git. traceId={}, version={}, branch={}, gitPushed={}",
|
||||
task.getTraceId(),
|
||||
task.getSourceVersion(),
|
||||
targetBranch,
|
||||
pushed
|
||||
);
|
||||
} catch (Exception e) {
|
||||
if (pullResult != null) {
|
||||
prodPullAckService.recordAckResult(pullResult.getPulledConfigIds(), ProdPullAckStatus.FAILED);
|
||||
prodPullAckService.recordPullFailure(
|
||||
pullResult.getPulledConfigs(),
|
||||
summarizeException(e),
|
||||
retryAttempt
|
||||
);
|
||||
handleFailure(traceId, "PROD prod->git snapshot sync failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
private void retryFailedProdPulls() {
|
||||
List<ProdPullAckService.RetryPullRequest> retryPullRequests =
|
||||
prodPullAckService.getRetryPullRequests(syncProperties.getMaxRetryCount());
|
||||
for (ProdPullAckService.RetryPullRequest retryPullRequest : retryPullRequests) {
|
||||
try {
|
||||
List<ProdPullResult> retryResults = prodConfigApiService.pullConfigSnapshots(
|
||||
retryPullRequest.getAirportId(),
|
||||
retryPullRequest.getAppName(),
|
||||
retryPullRequest.getSourceVersion(),
|
||||
retryPullRequest.getFileName()
|
||||
);
|
||||
for (ProdPullResult retryResult : retryResults) {
|
||||
syncSingleProdSnapshotToGit(retryResult, true);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
prodPullAckService.markRetryAttemptFailed(retryPullRequest, summarizeException(e));
|
||||
log.warn(
|
||||
"Retry pull failed. version={}, airportId={}, appName={}, fileName={}",
|
||||
retryPullRequest.getSourceVersion(),
|
||||
retryPullRequest.getAirportId(),
|
||||
retryPullRequest.getAppName(),
|
||||
retryPullRequest.getFileName(),
|
||||
e
|
||||
);
|
||||
}
|
||||
handleFailure(traceId, "PROD prod->git sync failed", e);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 已成功的同版本任务直接跳过，避免重复同步。
|
||||
* Generate the directory to push this round:
|
||||
* 1. First push of a branch is full
|
||||
* 2. Deletions fall back to full push
|
||||
* 3. Otherwise only changed files are pushed
|
||||
*/
|
||||
private boolean shouldSkip(Optional<SyncTask> existing) {
|
||||
return existing.isPresent() && existing.get().getStatus() == SyncStatus.SUCCESS;
|
||||
}
|
||||
|
||||
/**
|
||||
* 生成本次推送目录：
|
||||
* 1. 首次推送走全量
|
||||
* 2. 检测到删除时回退全量
|
||||
* 3. 其余场景仅推送变更文件
|
||||
*/
|
||||
private Path preparePushDirectory(Path exportDirectory, String sourceVersion) throws IOException {
|
||||
Path baselineDirectory = workDirectoryService.getGitToProdBaselineDir();
|
||||
private Path preparePushDirectory(Path exportDirectory, String branch, String stagingKey) throws IOException {
|
||||
Path baselineDirectory = workDirectoryService.getGitToProdBaselineDir(branch);
|
||||
if (Files.notExists(baselineDirectory) || isDirectoryEmpty(baselineDirectory)) {
|
||||
return exportDirectory;
|
||||
}
|
||||
@ -240,7 +281,7 @@ public class ProdSyncCoordinator {
|
||||
.filter(path -> !currentFileHashes.containsKey(path))
|
||||
.collect(Collectors.toSet());
|
||||
if (!deletedFiles.isEmpty()) {
|
||||
log.info("Git->PROD 检测到文件删除，回退为全量推送。deletedCount={}", deletedFiles.size());
|
||||
log.info("Git->PROD detected deleted files, fallback to full push. deletedCount={}", deletedFiles.size());
|
||||
return exportDirectory;
|
||||
}
|
||||
|
||||
@ -256,19 +297,16 @@ public class ProdSyncCoordinator {
|
||||
return exportDirectory;
|
||||
}
|
||||
|
||||
Path incrementalDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-delta-" + sourceVersion);
|
||||
Path incrementalDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-delta-" + stagingKey);
|
||||
FileTreeUtils.deleteRecursively(incrementalDirectory);
|
||||
FileTreeUtils.ensureDirectory(incrementalDirectory);
|
||||
FileTreeUtils.copySelectedFiles(exportDirectory, incrementalDirectory, changedFiles);
|
||||
log.info("Git->PROD 本次采用最小增量推送。changedCount={}", changedFiles.size());
|
||||
log.info("Git->PROD uses incremental push this round. changedCount={}", changedFiles.size());
|
||||
return incrementalDirectory;
|
||||
}
|
||||
|
||||
/**
|
||||
* 成功推送后刷新本地基线目录，作为下一次增量比较的依据。
|
||||
*/
|
||||
private void refreshGitToProdBaseline(Path exportDirectory) throws IOException {
|
||||
Path baselineDirectory = workDirectoryService.getGitToProdBaselineDir();
|
||||
private void refreshGitToProdBaseline(Path exportDirectory, String branch) throws IOException {
|
||||
Path baselineDirectory = workDirectoryService.getGitToProdBaselineDir(branch);
|
||||
FileTreeUtils.ensureDirectory(baselineDirectory);
|
||||
try (Stream<Path> stream = Files.list(baselineDirectory)) {
|
||||
for (Path child : stream.collect(Collectors.toList())) {
|
||||
@ -295,16 +333,12 @@ public class ProdSyncCoordinator {
|
||||
return fileHashes;
|
||||
}
|
||||
|
||||
/**
|
||||
* 统一失败处理逻辑。
|
||||
*/
|
||||
private void handleFailure(String traceId, String logMessage, Exception e) {
|
||||
log.error(logMessage, e);
|
||||
if (traceId == null) {
|
||||
return;
|
||||
}
|
||||
|
||||
// 只有达到最大重试次数后才把任务标记为失败，之前保留为可重试状态。
|
||||
syncTaskService.increaseRetryCount(traceId, summarizeException(e));
|
||||
Optional<SyncTask> task = syncTaskService.findByTraceId(traceId);
|
||||
int retryCount = task.map(SyncTask::getRetryCount).orElse(0);
|
||||
@ -313,9 +347,6 @@ public class ProdSyncCoordinator {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 统一截断异常摘要，避免错误信息过长污染数据库字段。
|
||||
*/
|
||||
private String summarizeException(Exception e) {
|
||||
String message = e.getMessage();
|
||||
if (message == null || message.trim().isEmpty()) {
|
||||
@ -323,4 +354,24 @@ public class ProdSyncCoordinator {
|
||||
}
|
||||
return message.length() > 400 ? message.substring(0, 400) : message;
|
||||
}
|
||||
|
||||
private String buildStagingKey(String branch, String sourceRevision) {
|
||||
return sanitizePathToken(branch) + "-" + sanitizePathToken(sourceRevision);
|
||||
}
|
||||
|
||||
private String resolveSnapshotBranch(String sourceVersion) {
|
||||
String baseBranch = gitRepoProperties.getSnapshotBranch();
|
||||
String versionSegment = sanitizePathToken(sourceVersion);
|
||||
if (baseBranch == null || baseBranch.trim().isEmpty()) {
|
||||
return versionSegment;
|
||||
}
|
||||
return baseBranch.endsWith("/") ? baseBranch + versionSegment : baseBranch + "/" + versionSegment;
|
||||
}
|
||||
|
||||
private String sanitizePathToken(String value) {
|
||||
if (value == null || value.trim().isEmpty()) {
|
||||
return "unknown";
|
||||
}
|
||||
return value.replaceAll("[^a-zA-Z0-9._-]", "_");
|
||||
}
|
||||
}
|
||||
|
||||
@ -17,4 +17,6 @@ public interface ProdPullAckRecordRepository extends JpaRepository<ProdPullAckRe
|
||||
List<ProdPullAckRecord> findByReportedFalseOrderByUpdatedAtAsc();
|
||||
|
||||
List<ProdPullAckRecord> findByAckStatusAndReportedFalseOrderByUpdatedAtAsc(ProdPullAckStatus ackStatus);
|
||||
|
||||
List<ProdPullAckRecord> findByAckStatusOrderByUpdatedAtAsc(ProdPullAckStatus ackStatus);
|
||||
}
|
||||
|
||||
@ -0,0 +1,33 @@
|
||||
package com.ftptool.sync.service;
|
||||
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
/**
|
||||
* Central extension point for config content encryption and decryption.
|
||||
* The current implementation is intentionally a no-op placeholder.
|
||||
*/
|
||||
@Service
|
||||
public class ConfigCryptoService {
|
||||
|
||||
public String encryptForPush(
|
||||
String airportId,
|
||||
String appName,
|
||||
String configVersion,
|
||||
String fileName,
|
||||
String plainContent
|
||||
) {
|
||||
// TODO: Replace the pass-through implementation with the production encryption algorithm.
|
||||
return plainContent;
|
||||
}
|
||||
|
||||
public String decryptAfterPull(
|
||||
String airportId,
|
||||
String appName,
|
||||
String configVersion,
|
||||
String fileName,
|
||||
String encryptedContent
|
||||
) {
|
||||
// TODO: Replace the pass-through implementation with the production decryption algorithm.
|
||||
return encryptedContent;
|
||||
}
|
||||
}
|
||||
@ -3,7 +3,6 @@ package com.ftptool.sync.service;
|
||||
import com.ftptool.sync.config.ProdApiProperties;
|
||||
import com.ftptool.sync.config.SyncProperties;
|
||||
import com.ftptool.sync.model.PackageManifest;
|
||||
import com.ftptool.sync.model.ProdPullAckStatus;
|
||||
import com.ftptool.sync.model.ProdPullResult;
|
||||
import com.ftptool.sync.util.FileHashUtils;
|
||||
import com.ftptool.sync.util.FileTreeUtils;
|
||||
@ -27,15 +26,16 @@ import java.nio.file.Path;
|
||||
import java.time.LocalDateTime;
|
||||
import java.time.format.DateTimeFormatter;
|
||||
import java.util.ArrayList;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.LinkedHashSet;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.Set;
|
||||
import java.util.stream.Collectors;
|
||||
import java.util.stream.Stream;
|
||||
|
||||
/**
|
||||
* 生产接口访问服务。
|
||||
* 封装对生产 pushConfig / pullConfig / login 接口的 HTTP 调用。
|
||||
* Encapsulates HTTP calls to production pushConfig / pullConfig / login APIs.
|
||||
*/
|
||||
@Service
|
||||
public class ProdConfigApiService {
|
||||
@ -49,6 +49,7 @@ public class ProdConfigApiService {
|
||||
private final RestTemplate restTemplate;
|
||||
private final WorkDirectoryService workDirectoryService;
|
||||
private final ProdPullAckService prodPullAckService;
|
||||
private final ConfigCryptoService configCryptoService;
|
||||
|
||||
private volatile String cachedToken;
|
||||
private volatile LocalDateTime cachedTokenExpireTime;
|
||||
@ -58,17 +59,19 @@ public class ProdConfigApiService {
|
||||
SyncProperties syncProperties,
|
||||
RestTemplate restTemplate,
|
||||
WorkDirectoryService workDirectoryService,
|
||||
ProdPullAckService prodPullAckService
|
||||
ProdPullAckService prodPullAckService,
|
||||
ConfigCryptoService configCryptoService
|
||||
) {
|
||||
this.prodApiProperties = prodApiProperties;
|
||||
this.syncProperties = syncProperties;
|
||||
this.restTemplate = restTemplate;
|
||||
this.workDirectoryService = workDirectoryService;
|
||||
this.prodPullAckService = prodPullAckService;
|
||||
this.configCryptoService = configCryptoService;
|
||||
}
|
||||
|
||||
/**
|
||||
* 调用 pushConfig 接口，把目录中的配置内容按文件维度推送到生产。
|
||||
* Push the files in the given directory to production as a JSON array.
|
||||
*/
|
||||
public void pushPackage(PackageManifest manifest, Path sourceDirectory) throws IOException {
|
||||
String url = buildUrl(prodApiProperties.getPushPath());
|
||||
@ -84,35 +87,58 @@ public class ProdConfigApiService {
|
||||
);
|
||||
|
||||
ProdApiResponse<ProdPushResponseData> body = response.getBody();
|
||||
validateSuccess(body, "生产 pushConfig 接口调用失败");
|
||||
validateSuccess(body, "Production pushConfig call failed");
|
||||
|
||||
List<ProdPushAckItem> ackFail = body.getData() == null ? null : body.getData().getAckFail();
|
||||
if (ackFail != null && !ackFail.isEmpty()) {
|
||||
String failedFiles = ackFail.stream()
|
||||
.map(ProdPushAckItem::getFileName)
|
||||
.collect(Collectors.joining(","));
|
||||
throw new IllegalStateException("生产 pushConfig 返回部分失败，失败文件：" + failedFiles);
|
||||
throw new IllegalStateException("Production pushConfig partially failed. files=" + failedFiles);
|
||||
}
|
||||
|
||||
log.info("Prod pushConfig finished. traceId={}, itemCount={}", manifest.getTraceId(), requestBody.size());
|
||||
}
|
||||
|
||||
/**
|
||||
* 调用 pullConfig 接口，拉取生产快照并按文件恢复到本地目录。
|
||||
* Pull production config snapshots using optional configured filters.
|
||||
*/
|
||||
public ProdPullResult pullConfigSnapshot() throws IOException {
|
||||
public List<ProdPullResult> pullConfigSnapshots() throws IOException {
|
||||
return pullConfigSnapshots(
|
||||
prodApiProperties.getAirportId(),
|
||||
prodApiProperties.getAppName(),
|
||||
prodApiProperties.getPullConfigVersion(),
|
||||
prodApiProperties.getPullFileName()
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pull production config snapshots using optional configVersion/fileName filters.
|
||||
*/
|
||||
public List<ProdPullResult> pullConfigSnapshots(
|
||||
String airportIdFilter,
|
||||
String appNameFilter,
|
||||
String configVersionFilter,
|
||||
String fileNameFilter
|
||||
) throws IOException {
|
||||
String url = buildUrl(prodApiProperties.getPullPath());
|
||||
HttpHeaders headers = defaultJsonHeaders();
|
||||
|
||||
UriComponentsBuilder builder = UriComponentsBuilder.fromHttpUrl(url);
|
||||
ProdPullAckService.PendingAckSummary pendingAckSummary = prodPullAckService.getPendingAckSummary();
|
||||
|
||||
// 当前先按应用维度过滤。
|
||||
if (StringUtils.hasText(prodApiProperties.getAirportId()) && !isPlaceholder(prodApiProperties.getAirportId())) {
|
||||
builder.queryParam("airportId", prodApiProperties.getAirportId());
|
||||
// Optional filters: leave them empty to pull all approved unsynced configs.
|
||||
if (StringUtils.hasText(airportIdFilter) && !isPlaceholder(airportIdFilter)) {
|
||||
builder.queryParam("airportId", airportIdFilter.trim());
|
||||
}
|
||||
if (StringUtils.hasText(prodApiProperties.getAppName()) && !isPlaceholder(prodApiProperties.getAppName())) {
|
||||
builder.queryParam("appName", prodApiProperties.getAppName());
|
||||
if (StringUtils.hasText(appNameFilter) && !isPlaceholder(appNameFilter)) {
|
||||
builder.queryParam("appName", appNameFilter.trim());
|
||||
}
|
||||
if (StringUtils.hasText(configVersionFilter) && !isPlaceholder(configVersionFilter)) {
|
||||
builder.queryParam("configVersion", configVersionFilter.trim());
|
||||
}
|
||||
if (StringUtils.hasText(fileNameFilter) && !isPlaceholder(fileNameFilter)) {
|
||||
builder.queryParam("fileName", fileNameFilter.trim());
|
||||
}
|
||||
if (pendingAckSummary.getAckSucIds() != null && !pendingAckSummary.getAckSucIds().isEmpty()) {
|
||||
builder.queryParam("ackSuc", StringUtils.collectionToCommaDelimitedString(pendingAckSummary.getAckSucIds()));
|
||||
@ -130,29 +156,18 @@ public class ProdConfigApiService {
|
||||
);
|
||||
|
||||
ProdApiResponse<List<ProdPulledConfigItem>> body = response.getBody();
|
||||
validateSuccess(body, "生产 pullConfig 接口调用失败");
|
||||
validateSuccess(body, "Production pullConfig call failed");
|
||||
if (pendingAckSummary.hasPendingAck()) {
|
||||
prodPullAckService.markPendingAsReported();
|
||||
}
|
||||
|
||||
List<ProdPulledConfigItem> items = body.getData();
|
||||
if (items == null || items.isEmpty()) {
|
||||
throw new IllegalStateException("生产 pullConfig 未返回可同步配置");
|
||||
throw new IllegalStateException("Production pullConfig returned no syncable config");
|
||||
}
|
||||
return buildPullResults(items);
|
||||
}
|
||||
|
||||
Path tempDir = Files.createTempDirectory(workDirectoryService.getProdToDevStagingDir(), "pull-");
|
||||
FileTreeUtils.ensureDirectory(tempDir);
|
||||
for (ProdPulledConfigItem item : items) {
|
||||
writePulledConfigItem(tempDir, item);
|
||||
}
|
||||
|
||||
String contentHash = FileHashUtils.sha256Directory(tempDir);
|
||||
String sourceVersion = resolvePullVersion(items, contentHash);
|
||||
return new ProdPullResult(tempDir, sourceVersion, contentHash, collectPulledIds(items));
|
||||
}
|
||||
|
||||
/**
|
||||
* 统一构造 JSON 调用头，并补齐 token 鉴权。
|
||||
*/
|
||||
private HttpHeaders defaultJsonHeaders() {
|
||||
HttpHeaders headers = new HttpHeaders();
|
||||
headers.setAccept(java.util.Collections.singletonList(MediaType.APPLICATION_JSON));
|
||||
@ -167,9 +182,6 @@ public class ProdConfigApiService {
|
||||
return headers;
|
||||
}
|
||||
|
||||
/**
|
||||
* 优先使用静态 token；如果未配置，则走登录接口获取并缓存。
|
||||
*/
|
||||
private String resolveToken() {
|
||||
if (StringUtils.hasText(prodApiProperties.getToken()) && !isPlaceholder(prodApiProperties.getToken())) {
|
||||
return prodApiProperties.getToken().trim();
|
||||
@ -177,9 +189,6 @@ public class ProdConfigApiService {
|
||||
return loginAndGetToken();
|
||||
}
|
||||
|
||||
/**
|
||||
* 调用登录接口获取 token，并根据过期时间做简单缓存。
|
||||
*/
|
||||
private synchronized String loginAndGetToken() {
|
||||
if (StringUtils.hasText(cachedToken)
|
||||
&& cachedTokenExpireTime != null
|
||||
@ -206,9 +215,9 @@ public class ProdConfigApiService {
|
||||
);
|
||||
|
||||
ProdApiResponse<ProdLoginResponseData> body = response.getBody();
|
||||
validateSuccess(body, "生产登录接口调用失败");
|
||||
validateSuccess(body, "Production login call failed");
|
||||
if (body.getData() == null || !StringUtils.hasText(body.getData().getToken())) {
|
||||
throw new IllegalStateException("生产登录接口未返回有效 token");
|
||||
throw new IllegalStateException("Production login did not return a token");
|
||||
}
|
||||
|
||||
cachedToken = body.getData().getToken().trim();
|
||||
@ -221,11 +230,10 @@ public class ProdConfigApiService {
|
||||
}
|
||||
|
||||
/**
|
||||
* 把本地目录转换成 pushConfig 需要的 JSON 数组。
|
||||
* Convert a branch snapshot directory to the pushConfig JSON array.
|
||||
* Layout is expected to be: airportId/appName/fileName
|
||||
*/
|
||||
private List<ProdPushConfigItem> buildPushRequest(PackageManifest manifest, Path sourceDirectory) throws IOException {
|
||||
validateBusinessConfig();
|
||||
|
||||
List<ProdPushConfigItem> result = new ArrayList<ProdPushConfigItem>();
|
||||
try (Stream<Path> stream = Files.walk(sourceDirectory)) {
|
||||
List<Path> files = stream
|
||||
@ -233,40 +241,57 @@ public class ProdConfigApiService {
|
||||
.sorted()
|
||||
.collect(Collectors.toList());
|
||||
for (Path file : files) {
|
||||
GitConfigPath gitConfigPath = parseGitConfigPath(sourceDirectory, file);
|
||||
|
||||
ProdPushConfigItem item = new ProdPushConfigItem();
|
||||
item.setAirportId(prodApiProperties.getAirportId());
|
||||
item.setAppName(prodApiProperties.getAppName());
|
||||
item.setAirportId(gitConfigPath.getAirportId());
|
||||
item.setAppName(gitConfigPath.getAppName());
|
||||
item.setConfigVersion(manifest.getSourceVersion());
|
||||
// TODO: 配置内容需按生产接口约定加密后再发送。
|
||||
item.setConfigContent(new String(Files.readAllBytes(file), StandardCharsets.UTF_8));
|
||||
item.setFileName(sourceDirectory.relativize(file).toString().replace('\\', '/'));
|
||||
String plainContent = new String(Files.readAllBytes(file), StandardCharsets.UTF_8);
|
||||
item.setConfigContent(configCryptoService.encryptForPush(
|
||||
gitConfigPath.getAirportId(),
|
||||
gitConfigPath.getAppName(),
|
||||
manifest.getSourceVersion(),
|
||||
gitConfigPath.getFileName(),
|
||||
plainContent
|
||||
));
|
||||
item.setFileName(gitConfigPath.getFileName());
|
||||
result.add(item);
|
||||
}
|
||||
}
|
||||
|
||||
if (result.isEmpty()) {
|
||||
throw new IllegalStateException("待推送目录为空，无法构造 pushConfig 请求");
|
||||
throw new IllegalStateException("No files found to build pushConfig request");
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/**
|
||||
* 将 pullConfig 返回的单条配置恢复为本地文件。
|
||||
* Restore one pulled config item under airportId/appName/fileName.
|
||||
*/
|
||||
private void writePulledConfigItem(Path baseDirectory, ProdPulledConfigItem item) throws IOException {
|
||||
String fileName = StringUtils.hasText(item.getFileName()) ? item.getFileName() : syncProperties.getPullResponseFileName();
|
||||
Path targetFile = baseDirectory.resolve(fileName).normalize();
|
||||
String airportId = requireDirectorySegment(item.getAirportId(), "airportId");
|
||||
String appName = requireDirectorySegment(item.getAppName(), "appName");
|
||||
String fileName = StringUtils.hasText(item.getFileName())
|
||||
? item.getFileName().trim()
|
||||
: syncProperties.getPullResponseFileName();
|
||||
|
||||
Path targetFile = baseDirectory.resolve(airportId).resolve(appName).resolve(fileName).normalize();
|
||||
if (!targetFile.startsWith(baseDirectory)) {
|
||||
throw new IOException("pullConfig 返回的 fileName 非法：" + fileName);
|
||||
}
|
||||
Files.createDirectories(targetFile.getParent());
|
||||
// TODO: configContent 当前直接按返回值落盘，后续需补充解密逻辑。
|
||||
Files.write(targetFile, safeString(item.getConfigContent()).getBytes(StandardCharsets.UTF_8));
|
||||
throw new IOException("pullConfig returned illegal fileName: " + fileName);
|
||||
}
|
||||
|
||||
Files.createDirectories(targetFile.getParent());
|
||||
String decryptedContent = configCryptoService.decryptAfterPull(
|
||||
airportId,
|
||||
appName,
|
||||
safeString(item.getConfigVersion()),
|
||||
fileName,
|
||||
safeString(item.getConfigContent())
|
||||
);
|
||||
Files.write(targetFile, safeString(decryptedContent).getBytes(StandardCharsets.UTF_8));
|
||||
}
|
||||
|
||||
/**
|
||||
* 解析本次 pull 结果的来源版本。
|
||||
*/
|
||||
private String resolvePullVersion(List<ProdPulledConfigItem> items, String contentHash) {
|
||||
Set<String> versions = new LinkedHashSet<String>();
|
||||
for (ProdPulledConfigItem item : items) {
|
||||
@ -280,9 +305,6 @@ public class ProdConfigApiService {
|
||||
return contentHash;
|
||||
}
|
||||
|
||||
/**
|
||||
* 提取 pullConfig 返回项中的 id，供下一次请求回传 ackSuc/ackFail。
|
||||
*/
|
||||
private List<String> collectPulledIds(List<ProdPulledConfigItem> items) {
|
||||
List<String> ids = new ArrayList<String>();
|
||||
for (ProdPulledConfigItem item : items) {
|
||||
@ -293,48 +315,76 @@ public class ProdConfigApiService {
|
||||
return ids;
|
||||
}
|
||||
|
||||
/**
|
||||
* 校验接口返回是否成功。
|
||||
*/
|
||||
private List<ProdPullResult> buildPullResults(List<ProdPulledConfigItem> items) throws IOException {
|
||||
Map<String, List<ProdPulledConfigItem>> itemsByVersion = new LinkedHashMap<String, List<ProdPulledConfigItem>>();
|
||||
for (ProdPulledConfigItem item : items) {
|
||||
String versionKey = StringUtils.hasText(item.getConfigVersion())
|
||||
? item.getConfigVersion().trim()
|
||||
: "__missing_version__";
|
||||
if (!itemsByVersion.containsKey(versionKey)) {
|
||||
itemsByVersion.put(versionKey, new ArrayList<ProdPulledConfigItem>());
|
||||
}
|
||||
itemsByVersion.get(versionKey).add(item);
|
||||
}
|
||||
|
||||
List<ProdPullResult> results = new ArrayList<ProdPullResult>();
|
||||
for (List<ProdPulledConfigItem> groupedItems : itemsByVersion.values()) {
|
||||
Path tempDir = Files.createTempDirectory(workDirectoryService.getProdToDevStagingDir(), "pull-");
|
||||
FileTreeUtils.ensureDirectory(tempDir);
|
||||
for (ProdPulledConfigItem item : groupedItems) {
|
||||
writePulledConfigItem(tempDir, item);
|
||||
}
|
||||
|
||||
String contentHash = FileHashUtils.sha256Directory(tempDir);
|
||||
String sourceVersion = resolvePullVersion(groupedItems, contentHash);
|
||||
results.add(new ProdPullResult(
|
||||
tempDir,
|
||||
sourceVersion,
|
||||
contentHash,
|
||||
collectPulledIds(groupedItems),
|
||||
collectPulledRefs(groupedItems, sourceVersion)
|
||||
));
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
private List<ProdPullResult.PulledConfigRef> collectPulledRefs(List<ProdPulledConfigItem> items, String sourceVersion) {
|
||||
List<ProdPullResult.PulledConfigRef> refs = new ArrayList<ProdPullResult.PulledConfigRef>();
|
||||
for (ProdPulledConfigItem item : items) {
|
||||
refs.add(new ProdPullResult.PulledConfigRef(
|
||||
safeTrim(item.getId()),
|
||||
safeTrim(item.getAirportId()),
|
||||
safeTrim(item.getAppName()),
|
||||
StringUtils.hasText(item.getConfigVersion()) ? safeTrim(item.getConfigVersion()) : sourceVersion,
|
||||
safeTrim(item.getFileName())
|
||||
));
|
||||
}
|
||||
return refs;
|
||||
}
|
||||
|
||||
private void validateSuccess(ProdApiResponse<?> response, String failureMessage) {
|
||||
if (response == null) {
|
||||
throw new IllegalStateException(failureMessage + "：响应体为空");
|
||||
throw new IllegalStateException(failureMessage + ": response body is empty");
|
||||
}
|
||||
if (!SUCCESS_CODE.equals(response.getCode())) {
|
||||
throw new IllegalStateException(failureMessage + "：" + safeString(response.getMsg()) + "，code=" + safeString(response.getCode()));
|
||||
throw new IllegalStateException(
|
||||
failureMessage + ": msg=" + safeString(response.getMsg()) + ", code=" + safeString(response.getCode())
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 校验 push/pull 业务维度配置是否已提供。
|
||||
*/
|
||||
private void validateBusinessConfig() {
|
||||
if (!StringUtils.hasText(prodApiProperties.getAirportId()) || isPlaceholder(prodApiProperties.getAirportId())) {
|
||||
throw new IllegalStateException("未配置 prod.api.airport-id");
|
||||
}
|
||||
if (!StringUtils.hasText(prodApiProperties.getAppName()) || isPlaceholder(prodApiProperties.getAppName())) {
|
||||
throw new IllegalStateException("未配置 prod.api.app-name");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 校验登录配置是否已提供。
|
||||
*/
|
||||
private void validateLoginConfig() {
|
||||
if (!StringUtils.hasText(prodApiProperties.getLoginPath())) {
|
||||
throw new IllegalStateException("未配置 prod.api.login-path");
|
||||
throw new IllegalStateException("Missing prod.api.login-path");
|
||||
}
|
||||
if (!StringUtils.hasText(prodApiProperties.getLoginName())) {
|
||||
throw new IllegalStateException("未配置 prod.api.login-name");
|
||||
throw new IllegalStateException("Missing prod.api.login-name");
|
||||
}
|
||||
if (!StringUtils.hasText(prodApiProperties.getLoginPassword())) {
|
||||
throw new IllegalStateException("未配置 prod.api.login-password");
|
||||
throw new IllegalStateException("Missing prod.api.login-password");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 按基础地址和接口路径拼接完整 URL。
|
||||
*/
|
||||
private String buildUrl(String path) {
|
||||
String base = prodApiProperties.getBaseUrl();
|
||||
if (base.endsWith("/") && path.startsWith("/")) {
|
||||
@ -354,9 +404,41 @@ public class ProdConfigApiService {
|
||||
return value == null ? "" : value;
|
||||
}
|
||||
|
||||
/**
|
||||
* 通用接口响应包装。
|
||||
*/
|
||||
private String safeTrim(String value) {
|
||||
return value == null ? null : value.trim();
|
||||
}
|
||||
|
||||
private GitConfigPath parseGitConfigPath(Path sourceDirectory, Path file) {
|
||||
Path relativePath = sourceDirectory.relativize(file);
|
||||
if (relativePath.getNameCount() < 3) {
|
||||
throw new IllegalStateException(
|
||||
"Git config path does not match airportId/appName/fileName layout: " + normalizePath(relativePath)
|
||||
);
|
||||
}
|
||||
|
||||
String airportId = relativePath.getName(0).toString();
|
||||
String appName = relativePath.getName(1).toString();
|
||||
String fileName = normalizePath(relativePath.subpath(2, relativePath.getNameCount()));
|
||||
return new GitConfigPath(airportId, appName, fileName);
|
||||
}
|
||||
|
||||
private String requireDirectorySegment(String value, String fieldName) throws IOException {
|
||||
if (!StringUtils.hasText(value)) {
|
||||
throw new IOException("pullConfig response missing " + fieldName);
|
||||
}
|
||||
|
||||
String trimmed = value.trim();
|
||||
if (".".equals(trimmed) || "..".equals(trimmed)
|
||||
|| trimmed.contains("/") || trimmed.contains("\\")) {
|
||||
throw new IOException("pullConfig returned illegal " + fieldName + ": " + trimmed);
|
||||
}
|
||||
return trimmed;
|
||||
}
|
||||
|
||||
private String normalizePath(Path path) {
|
||||
return path.toString().replace('\\', '/');
|
||||
}
|
||||
|
||||
public static class ProdApiResponse<T> {
|
||||
|
||||
private String code;
|
||||
@ -397,9 +479,31 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* pushConfig 请求体项。
|
||||
*/
|
||||
private static class GitConfigPath {
|
||||
|
||||
private final String airportId;
|
||||
private final String appName;
|
||||
private final String fileName;
|
||||
|
||||
private GitConfigPath(String airportId, String appName, String fileName) {
|
||||
this.airportId = airportId;
|
||||
this.appName = appName;
|
||||
this.fileName = fileName;
|
||||
}
|
||||
|
||||
public String getAirportId() {
|
||||
return airportId;
|
||||
}
|
||||
|
||||
public String getAppName() {
|
||||
return appName;
|
||||
}
|
||||
|
||||
public String getFileName() {
|
||||
return fileName;
|
||||
}
|
||||
}
|
||||
|
||||
public static class ProdPushConfigItem {
|
||||
|
||||
private String airportId;
|
||||
@ -449,9 +553,6 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* pushConfig 响应 data。
|
||||
*/
|
||||
public static class ProdPushResponseData {
|
||||
|
||||
private List<ProdPushAckItem> ackFail;
|
||||
@ -465,9 +566,6 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* pushConfig 失败回执项。
|
||||
*/
|
||||
public static class ProdPushAckItem {
|
||||
|
||||
private String airportId;
|
||||
@ -508,9 +606,6 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* pullConfig 返回的配置项。
|
||||
*/
|
||||
public static class ProdPulledConfigItem {
|
||||
|
||||
private String id;
|
||||
@ -569,9 +664,6 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 登录请求体。
|
||||
*/
|
||||
public static class ProdLoginRequest {
|
||||
|
||||
private String name;
|
||||
@ -594,9 +686,6 @@ public class ProdConfigApiService {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 登录接口返回 data。
|
||||
*/
|
||||
public static class ProdLoginResponseData {
|
||||
|
||||
private String token;
|
||||
|
||||
@ -2,29 +2,33 @@ package com.ftptool.sync.service;
|
||||
|
||||
import com.ftptool.sync.entity.ProdPullAckRecord;
|
||||
import com.ftptool.sync.model.ProdPullAckStatus;
|
||||
import com.ftptool.sync.model.ProdPullResult;
|
||||
import com.ftptool.sync.repository.ProdPullAckRecordRepository;
|
||||
import org.springframework.stereotype.Service;
|
||||
import org.springframework.transaction.annotation.Transactional;
|
||||
import org.springframework.util.StringUtils;
|
||||
|
||||
import java.time.LocalDateTime;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collection;
|
||||
import java.util.LinkedHashMap;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* pullConfig 回执服务。
|
||||
* Stores pullConfig ack states and retry plans.
|
||||
*/
|
||||
@Service
|
||||
public class ProdPullAckService {
|
||||
|
||||
private static final int RETRY_DELAY_BASE_SECONDS = 30;
|
||||
|
||||
private final ProdPullAckRecordRepository prodPullAckRecordRepository;
|
||||
|
||||
public ProdPullAckService(ProdPullAckRecordRepository prodPullAckRecordRepository) {
|
||||
this.prodPullAckRecordRepository = prodPullAckRecordRepository;
|
||||
}
|
||||
|
||||
/**
|
||||
* 读取所有尚未回传给生产端的回执状态。
|
||||
*/
|
||||
@Transactional(readOnly = true)
|
||||
public PendingAckSummary getPendingAckSummary() {
|
||||
List<String> successIds = toRemoteIds(
|
||||
@ -36,31 +40,116 @@ public class ProdPullAckService {
|
||||
return new PendingAckSummary(successIds, failedIds);
|
||||
}
|
||||
|
||||
/**
|
||||
* 记录一批配置项的处理结果。
|
||||
* 若同一 remote id 已存在，则以最新状态覆盖。
|
||||
*/
|
||||
@Transactional
|
||||
public void recordAckResult(Collection<String> remoteIds, ProdPullAckStatus ackStatus) {
|
||||
if (remoteIds == null) {
|
||||
public void recordAckSuccess(Collection<ProdPullResult.PulledConfigRef> pulledConfigs) {
|
||||
if (pulledConfigs == null) {
|
||||
return;
|
||||
}
|
||||
for (String remoteId : remoteIds) {
|
||||
if (remoteId == null || remoteId.trim().isEmpty()) {
|
||||
|
||||
for (ProdPullResult.PulledConfigRef pulledConfig : pulledConfigs) {
|
||||
if (!hasRemoteId(pulledConfig)) {
|
||||
continue;
|
||||
}
|
||||
ProdPullAckRecord record = prodPullAckRecordRepository.findByRemoteConfigId(remoteId)
|
||||
|
||||
ProdPullAckRecord record = prodPullAckRecordRepository.findByRemoteConfigId(pulledConfig.getRemoteConfigId())
|
||||
.orElseGet(ProdPullAckRecord::new);
|
||||
record.setRemoteConfigId(remoteId);
|
||||
record.setAckStatus(ackStatus);
|
||||
record.setRemoteConfigId(pulledConfig.getRemoteConfigId());
|
||||
record.setAckStatus(ProdPullAckStatus.SUCCESS);
|
||||
record.setReported(Boolean.FALSE);
|
||||
applyPullMetadata(record, pulledConfig);
|
||||
record.setRetryCount(0);
|
||||
record.setNextRetryAt(null);
|
||||
record.setLastErrorMsg(null);
|
||||
prodPullAckRecordRepository.save(record);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 当前批次 pull 请求发送成功后，将待回执记录标记为已回传。
|
||||
*/
|
||||
@Transactional
|
||||
public void recordPullFailure(
|
||||
Collection<ProdPullResult.PulledConfigRef> pulledConfigs,
|
||||
String errorMsg,
|
||||
boolean retryAttempt
|
||||
) {
|
||||
if (pulledConfigs == null) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (ProdPullResult.PulledConfigRef pulledConfig : pulledConfigs) {
|
||||
if (!hasRemoteId(pulledConfig)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
ProdPullAckRecord record = prodPullAckRecordRepository.findByRemoteConfigId(pulledConfig.getRemoteConfigId())
|
||||
.orElseGet(ProdPullAckRecord::new);
|
||||
record.setRemoteConfigId(pulledConfig.getRemoteConfigId());
|
||||
record.setAckStatus(ProdPullAckStatus.FAILED);
|
||||
record.setReported(Boolean.FALSE);
|
||||
applyPullMetadata(record, pulledConfig);
|
||||
record.setLastErrorMsg(errorMsg);
|
||||
if (retryAttempt) {
|
||||
int retryCount = safeRetryCount(record) + 1;
|
||||
record.setRetryCount(retryCount);
|
||||
record.setNextRetryAt(calculateNextRetryAt(retryCount));
|
||||
} else {
|
||||
record.setRetryCount(0);
|
||||
record.setNextRetryAt(LocalDateTime.now());
|
||||
}
|
||||
prodPullAckRecordRepository.save(record);
|
||||
}
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void markRetryAttemptFailed(RetryPullRequest retryPullRequest, String errorMsg) {
|
||||
if (retryPullRequest == null || retryPullRequest.getRemoteConfigIds().isEmpty()) {
|
||||
return;
|
||||
}
|
||||
|
||||
for (String remoteConfigId : retryPullRequest.getRemoteConfigIds()) {
|
||||
prodPullAckRecordRepository.findByRemoteConfigId(remoteConfigId).ifPresent(record -> {
|
||||
int retryCount = safeRetryCount(record) + 1;
|
||||
record.setRetryCount(retryCount);
|
||||
record.setNextRetryAt(calculateNextRetryAt(retryCount));
|
||||
record.setLastErrorMsg(errorMsg);
|
||||
record.setAckStatus(ProdPullAckStatus.FAILED);
|
||||
prodPullAckRecordRepository.save(record);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@Transactional(readOnly = true)
|
||||
public List<RetryPullRequest> getRetryPullRequests(int maxRetryCount) {
|
||||
List<ProdPullAckRecord> failedRecords = prodPullAckRecordRepository.findByAckStatusOrderByUpdatedAtAsc(ProdPullAckStatus.FAILED);
|
||||
LocalDateTime now = LocalDateTime.now();
|
||||
Map<String, RetryPullRequest> retryPullRequests = new LinkedHashMap<String, RetryPullRequest>();
|
||||
|
||||
for (ProdPullAckRecord failedRecord : failedRecords) {
|
||||
if (!hasRetryMetadata(failedRecord)) {
|
||||
continue;
|
||||
}
|
||||
if (safeRetryCount(failedRecord) >= maxRetryCount) {
|
||||
continue;
|
||||
}
|
||||
if (failedRecord.getNextRetryAt() != null && failedRecord.getNextRetryAt().isAfter(now)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
String groupKey = buildRetryGroupKey(failedRecord);
|
||||
RetryPullRequest retryPullRequest = retryPullRequests.get(groupKey);
|
||||
if (retryPullRequest == null) {
|
||||
retryPullRequest = new RetryPullRequest(
|
||||
failedRecord.getSourceVersion(),
|
||||
failedRecord.getAirportId(),
|
||||
failedRecord.getAppName(),
|
||||
failedRecord.getFileName()
|
||||
);
|
||||
retryPullRequests.put(groupKey, retryPullRequest);
|
||||
}
|
||||
retryPullRequest.addRemoteConfigId(failedRecord.getRemoteConfigId());
|
||||
}
|
||||
|
||||
return new ArrayList<RetryPullRequest>(retryPullRequests.values());
|
||||
}
|
||||
|
||||
@Transactional
|
||||
public void markPendingAsReported() {
|
||||
List<ProdPullAckRecord> records = prodPullAckRecordRepository.findByReportedFalseOrderByUpdatedAtAsc();
|
||||
@ -70,6 +159,51 @@ public class ProdPullAckService {
|
||||
}
|
||||
}
|
||||
|
||||
private void applyPullMetadata(ProdPullAckRecord record, ProdPullResult.PulledConfigRef pulledConfig) {
|
||||
if (pulledConfig == null) {
|
||||
return;
|
||||
}
|
||||
if (StringUtils.hasText(pulledConfig.getConfigVersion())) {
|
||||
record.setSourceVersion(pulledConfig.getConfigVersion().trim());
|
||||
}
|
||||
if (StringUtils.hasText(pulledConfig.getAirportId())) {
|
||||
record.setAirportId(pulledConfig.getAirportId().trim());
|
||||
}
|
||||
if (StringUtils.hasText(pulledConfig.getAppName())) {
|
||||
record.setAppName(pulledConfig.getAppName().trim());
|
||||
}
|
||||
if (StringUtils.hasText(pulledConfig.getFileName())) {
|
||||
record.setFileName(pulledConfig.getFileName().trim());
|
||||
}
|
||||
}
|
||||
|
||||
private boolean hasRemoteId(ProdPullResult.PulledConfigRef pulledConfig) {
|
||||
return pulledConfig != null && StringUtils.hasText(pulledConfig.getRemoteConfigId());
|
||||
}
|
||||
|
||||
private boolean hasRetryMetadata(ProdPullAckRecord failedRecord) {
|
||||
return StringUtils.hasText(failedRecord.getSourceVersion())
|
||||
&& StringUtils.hasText(failedRecord.getAirportId())
|
||||
&& StringUtils.hasText(failedRecord.getAppName())
|
||||
&& StringUtils.hasText(failedRecord.getFileName());
|
||||
}
|
||||
|
||||
private int safeRetryCount(ProdPullAckRecord record) {
|
||||
return record.getRetryCount() == null ? 0 : record.getRetryCount().intValue();
|
||||
}
|
||||
|
||||
private LocalDateTime calculateNextRetryAt(int retryCount) {
|
||||
long delaySeconds = RETRY_DELAY_BASE_SECONDS * (1L << Math.max(0, retryCount - 1));
|
||||
return LocalDateTime.now().plusSeconds(delaySeconds);
|
||||
}
|
||||
|
||||
private String buildRetryGroupKey(ProdPullAckRecord failedRecord) {
|
||||
return failedRecord.getSourceVersion() + "|"
|
||||
+ failedRecord.getAirportId() + "|"
|
||||
+ failedRecord.getAppName() + "|"
|
||||
+ failedRecord.getFileName();
|
||||
}
|
||||
|
||||
private List<String> toRemoteIds(List<ProdPullAckRecord> records) {
|
||||
List<String> ids = new ArrayList<String>();
|
||||
for (ProdPullAckRecord record : records) {
|
||||
@ -78,9 +212,6 @@ public class ProdPullAckService {
|
||||
return ids;
|
||||
}
|
||||
|
||||
/**
|
||||
* 待回传 ACK 摘要。
|
||||
*/
|
||||
public static class PendingAckSummary {
|
||||
|
||||
private final List<String> ackSucIds;
|
||||
@ -103,4 +234,44 @@ public class ProdPullAckService {
|
||||
return !(ackSucIds == null || ackSucIds.isEmpty()) || !(ackFailIds == null || ackFailIds.isEmpty());
|
||||
}
|
||||
}
|
||||
|
||||
public static class RetryPullRequest {
|
||||
|
||||
private final String sourceVersion;
|
||||
private final String airportId;
|
||||
private final String appName;
|
||||
private final String fileName;
|
||||
private final List<String> remoteConfigIds = new ArrayList<String>();
|
||||
|
||||
public RetryPullRequest(String sourceVersion, String airportId, String appName, String fileName) {
|
||||
this.sourceVersion = sourceVersion;
|
||||
this.airportId = airportId;
|
||||
this.appName = appName;
|
||||
this.fileName = fileName;
|
||||
}
|
||||
|
||||
public String getSourceVersion() {
|
||||
return sourceVersion;
|
||||
}
|
||||
|
||||
public String getAirportId() {
|
||||
return airportId;
|
||||
}
|
||||
|
||||
public String getAppName() {
|
||||
return appName;
|
||||
}
|
||||
|
||||
public String getFileName() {
|
||||
return fileName;
|
||||
}
|
||||
|
||||
public List<String> getRemoteConfigIds() {
|
||||
return remoteConfigIds;
|
||||
}
|
||||
|
||||
private void addRemoteConfigId(String remoteConfigId) {
|
||||
this.remoteConfigIds.add(remoteConfigId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -31,7 +31,7 @@ public class WorkDirectoryService {
|
||||
FileTreeUtils.ensureDirectory(getPackageTempDir());
|
||||
FileTreeUtils.ensureDirectory(getDevToProdStagingDir());
|
||||
FileTreeUtils.ensureDirectory(getProdToDevStagingDir());
|
||||
FileTreeUtils.ensureDirectory(getGitToProdBaselineDir());
|
||||
FileTreeUtils.ensureDirectory(getGitToProdBaselineRootDir());
|
||||
}
|
||||
|
||||
/**
|
||||
@ -65,7 +65,21 @@ public class WorkDirectoryService {
|
||||
/**
|
||||
* Git -> PROD 链路用于比较增量的基线目录。
|
||||
*/
|
||||
public Path getGitToProdBaselineDir() {
|
||||
public Path getGitToProdBaselineRootDir() {
|
||||
return getWorkDir().resolve("baseline").resolve("git-to-prod");
|
||||
}
|
||||
|
||||
/**
|
||||
* Git -> PROD 链路按版本分支隔离的基线目录。
|
||||
*/
|
||||
public Path getGitToProdBaselineDir(String branch) {
|
||||
return getGitToProdBaselineRootDir().resolve(sanitizePathSegment(branch));
|
||||
}
|
||||
|
||||
private String sanitizePathSegment(String value) {
|
||||
if (value == null || value.trim().isEmpty()) {
|
||||
return "default";
|
||||
}
|
||||
return value.replaceAll("[^a-zA-Z0-9._-]", "_");
|
||||
}
|
||||
}
|
||||
|
||||
@ -17,5 +17,7 @@ prod.api.token=change-me
|
||||
prod.api.token-header-name=token
|
||||
prod.api.airport-id=replace-me
|
||||
prod.api.app-name=replace-me
|
||||
prod.api.pull-config-version=
|
||||
prod.api.pull-file-name=
|
||||
prod.api.login-name=
|
||||
prod.api.login-password=
|
||||
|
||||
@ -33,7 +33,7 @@ git.repo.local-path=./work/git/config-repo
|
||||
git.repo.remote-uri=https://git.example.com/config.git
|
||||
git.repo.username=replace-me
|
||||
git.repo.password=replace-me
|
||||
git.repo.scan-branch=config-dev-main
|
||||
git.repo.scan-branch=R_XXX_V3.0.3_XXX
|
||||
git.repo.snapshot-branch=config-prod-snapshot
|
||||
git.repo.commit-author-name=git-sync-bot
|
||||
git.repo.commit-author-email=git-sync-bot@example.com
|
||||
@ -49,6 +49,8 @@ prod.api.token=replace-me
|
||||
prod.api.token-header-name=token
|
||||
prod.api.airport-id=replace-me
|
||||
prod.api.app-name=replace-me
|
||||
prod.api.pull-config-version=
|
||||
prod.api.pull-file-name=
|
||||
prod.api.login-name=
|
||||
prod.api.login-password=
|
||||
prod.api.connect-timeout-ms=10000
|
||||
|
||||
@ -31,9 +31,25 @@ create table if not exists prod_pull_ack (
|
||||
remote_config_id varchar(128) not null,
|
||||
ack_status varchar(16) not null,
|
||||
reported boolean not null default false,
|
||||
source_version varchar(128),
|
||||
airport_id varchar(128),
|
||||
app_name varchar(128),
|
||||
file_name varchar(512),
|
||||
retry_count int not null default 0,
|
||||
next_retry_at timestamp,
|
||||
last_error_msg clob,
|
||||
created_at timestamp not null,
|
||||
updated_at timestamp not null,
|
||||
constraint uk_prod_pull_ack_remote_id unique (remote_config_id)
|
||||
);
|
||||
|
||||
alter table prod_pull_ack add column if not exists source_version varchar(128);
|
||||
alter table prod_pull_ack add column if not exists airport_id varchar(128);
|
||||
alter table prod_pull_ack add column if not exists app_name varchar(128);
|
||||
alter table prod_pull_ack add column if not exists file_name varchar(512);
|
||||
alter table prod_pull_ack add column if not exists retry_count int default 0;
|
||||
alter table prod_pull_ack add column if not exists next_retry_at timestamp;
|
||||
alter table prod_pull_ack add column if not exists last_error_msg clob;
|
||||
|
||||
create index if not exists idx_prod_pull_ack_reported on prod_pull_ack (reported);
|
||||
create index if not exists idx_prod_pull_ack_status on prod_pull_ack (ack_status);
|
||||
|
||||
@ -98,7 +98,13 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
void shouldSyncGitToProdAndKeepItIdempotent() throws Exception {
|
||||
when(gitClientService.prepareRepositoryAndGetHead("config-dev-main")).thenReturn("commit-a");
|
||||
when(gitClientService.exportBranchSnapshot(eq("config-dev-main"), any(Path.class)))
|
||||
.thenAnswer(invocation -> invocation.getArgument(1));
|
||||
.thenAnswer(invocation -> {
|
||||
Path target = invocation.getArgument(1);
|
||||
Path configFile = target.resolve("PEK").resolve("monitor").resolve("application.yml");
|
||||
Files.createDirectories(configFile.getParent());
|
||||
Files.write(configFile, "key: value".getBytes("UTF-8"));
|
||||
return target;
|
||||
});
|
||||
when(packageService.calculateDirectoryHash(any(Path.class))).thenReturn("hash-a");
|
||||
doNothing().when(prodConfigApiService).pushPackage(any(PackageManifest.class), any(Path.class));
|
||||
|
||||
@ -109,13 +115,13 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
assertEquals(1, tasks.size());
|
||||
SyncTask task = tasks.get(0);
|
||||
assertEquals(SyncDirection.DEV_TO_PROD, task.getDirection());
|
||||
assertEquals("commit-a", task.getSourceVersion());
|
||||
assertEquals("config-dev-main", task.getSourceVersion());
|
||||
assertEquals("hash-a", task.getContentHash());
|
||||
assertEquals(SyncStatus.SUCCESS, task.getStatus());
|
||||
|
||||
Optional<SyncCheckpoint> checkpoint = syncCheckpointRepository.findByDirection(SyncDirection.DEV_TO_PROD);
|
||||
assertTrue(checkpoint.isPresent());
|
||||
assertEquals("commit-a", checkpoint.get().getLastSuccessVersion());
|
||||
assertEquals("config-dev-main", checkpoint.get().getLastSuccessVersion());
|
||||
assertEquals("hash-a", checkpoint.get().getLastSuccessHash());
|
||||
|
||||
verify(prodConfigApiService, times(1)).pushPackage(any(PackageManifest.class), any(Path.class));
|
||||
@ -125,12 +131,12 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
void shouldSyncProdSnapshotToGitAndKeepItIdempotent() throws Exception {
|
||||
Path contentDirectory = Files.createTempDirectory("prod-to-git-");
|
||||
Files.write(contentDirectory.resolve("prod-config.json"), "{\"version\":\"prod-v1\"}".getBytes("UTF-8"));
|
||||
when(prodConfigApiService.pullConfigSnapshot()).thenReturn(
|
||||
when(prodConfigApiService.pullConfigSnapshots()).thenReturn(Arrays.asList(
|
||||
new ProdPullResult(contentDirectory, "prod-v1", "hash-b", Arrays.asList("1", "2"))
|
||||
);
|
||||
));
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(contentDirectory),
|
||||
eq("config-prod-snapshot"),
|
||||
eq("config-prod-snapshot/prod-v1"),
|
||||
contains("prod-v1")
|
||||
)).thenReturn(true);
|
||||
|
||||
@ -159,21 +165,58 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
|
||||
verify(gitClientService, times(1)).syncDirectoryToBranch(
|
||||
eq(contentDirectory),
|
||||
eq("config-prod-snapshot"),
|
||||
eq("config-prod-snapshot/prod-v1"),
|
||||
contains("prod-v1")
|
||||
);
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldSyncMultipleProdVersionsToDynamicSnapshotBranches() throws Exception {
|
||||
Path firstDirectory = Files.createTempDirectory("prod-to-git-v1-");
|
||||
Path secondDirectory = Files.createTempDirectory("prod-to-git-v2-");
|
||||
Files.write(firstDirectory.resolve("a.txt"), "one".getBytes("UTF-8"));
|
||||
Files.write(secondDirectory.resolve("b.txt"), "two".getBytes("UTF-8"));
|
||||
when(prodConfigApiService.pullConfigSnapshots()).thenReturn(Arrays.asList(
|
||||
new ProdPullResult(firstDirectory, "prod-v1", "hash-v1", Arrays.asList("1")),
|
||||
new ProdPullResult(secondDirectory, "prod-v2", "hash-v2", Arrays.asList("2"))
|
||||
));
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(firstDirectory),
|
||||
eq("config-prod-snapshot/prod-v1"),
|
||||
contains("prod-v1")
|
||||
)).thenReturn(true);
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(secondDirectory),
|
||||
eq("config-prod-snapshot/prod-v2"),
|
||||
contains("prod-v2")
|
||||
)).thenReturn(true);
|
||||
|
||||
prodSyncCoordinator.syncProdSnapshotToGit();
|
||||
|
||||
List<SyncTask> tasks = syncTaskRepository.findAll();
|
||||
assertEquals(2, tasks.size());
|
||||
verify(gitClientService, times(1)).syncDirectoryToBranch(
|
||||
eq(firstDirectory),
|
||||
eq("config-prod-snapshot/prod-v1"),
|
||||
contains("prod-v1")
|
||||
);
|
||||
verify(gitClientService, times(1)).syncDirectoryToBranch(
|
||||
eq(secondDirectory),
|
||||
eq("config-prod-snapshot/prod-v2"),
|
||||
contains("prod-v2")
|
||||
);
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldRecordFailedAckWhenProdSnapshotSyncFails() throws Exception {
|
||||
Path contentDirectory = Files.createTempDirectory("prod-to-git-fail-");
|
||||
Files.write(contentDirectory.resolve("prod-config.json"), "{\"version\":\"prod-v2\"}".getBytes("UTF-8"));
|
||||
when(prodConfigApiService.pullConfigSnapshot()).thenReturn(
|
||||
when(prodConfigApiService.pullConfigSnapshots()).thenReturn(Arrays.asList(
|
||||
new ProdPullResult(contentDirectory, "prod-v2", "hash-fail", Arrays.asList("9"))
|
||||
);
|
||||
));
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(contentDirectory),
|
||||
eq("config-prod-snapshot"),
|
||||
eq("config-prod-snapshot/prod-v2"),
|
||||
contains("prod-v2")
|
||||
)).thenThrow(new IllegalStateException("git push fail"));
|
||||
|
||||
@ -186,6 +229,64 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
assertFalse(ackRecords.get(0).getReported());
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldRetryFailedAckByTargetedPullFilters() throws Exception {
|
||||
Path failedDirectory = Files.createTempDirectory("prod-to-git-retry-fail-");
|
||||
Path retriedDirectory = Files.createTempDirectory("prod-to-git-retry-success-");
|
||||
Files.write(failedDirectory.resolve("prod-config.json"), "{\"version\":\"prod-v3\"}".getBytes("UTF-8"));
|
||||
Files.write(retriedDirectory.resolve("prod-config.json"), "{\"version\":\"prod-v3\"}".getBytes("UTF-8"));
|
||||
|
||||
List<ProdPullResult.PulledConfigRef> pulledConfigs = Arrays.asList(
|
||||
new ProdPullResult.PulledConfigRef("11", "PEK", "monitor", "prod-v3", "jobs/sync-job.json")
|
||||
);
|
||||
ProdPullResult failedResult = new ProdPullResult(
|
||||
failedDirectory,
|
||||
"prod-v3",
|
||||
"hash-fail-v3",
|
||||
Arrays.asList("11"),
|
||||
pulledConfigs
|
||||
);
|
||||
ProdPullResult retriedResult = new ProdPullResult(
|
||||
retriedDirectory,
|
||||
"prod-v3",
|
||||
"hash-success-v3",
|
||||
Arrays.asList("11"),
|
||||
pulledConfigs
|
||||
);
|
||||
|
||||
when(prodConfigApiService.pullConfigSnapshots())
|
||||
.thenReturn(Arrays.asList(failedResult))
|
||||
.thenReturn(new ArrayList<ProdPullResult>());
|
||||
when(prodConfigApiService.pullConfigSnapshots("PEK", "monitor", "prod-v3", "jobs/sync-job.json"))
|
||||
.thenReturn(Arrays.asList(retriedResult));
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(failedDirectory),
|
||||
eq("config-prod-snapshot/prod-v3"),
|
||||
contains("prod-v3")
|
||||
)).thenThrow(new IllegalStateException("first git push fail"));
|
||||
when(gitClientService.syncDirectoryToBranch(
|
||||
eq(retriedDirectory),
|
||||
eq("config-prod-snapshot/prod-v3"),
|
||||
contains("prod-v3")
|
||||
)).thenReturn(true);
|
||||
|
||||
prodSyncCoordinator.syncProdSnapshotToGit();
|
||||
prodSyncCoordinator.syncProdSnapshotToGit();
|
||||
|
||||
List<ProdPullAckRecord> ackRecords = prodPullAckRecordRepository.findAll();
|
||||
assertEquals(1, ackRecords.size());
|
||||
ProdPullAckRecord ackRecord = ackRecords.get(0);
|
||||
assertEquals(ProdPullAckStatus.SUCCESS, ackRecord.getAckStatus());
|
||||
assertFalse(ackRecord.getReported());
|
||||
assertEquals(Integer.valueOf(0), ackRecord.getRetryCount());
|
||||
assertEquals("PEK", ackRecord.getAirportId());
|
||||
assertEquals("monitor", ackRecord.getAppName());
|
||||
assertEquals("jobs/sync-job.json", ackRecord.getFileName());
|
||||
|
||||
verify(prodConfigApiService, times(1))
|
||||
.pullConfigSnapshots("PEK", "monitor", "prod-v3", "jobs/sync-job.json");
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldUseIncrementalDirectoryForSecondGitToProdPush() throws Exception {
|
||||
AtomicInteger exportCounter = new AtomicInteger(0);
|
||||
@ -197,14 +298,16 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
when(gitClientService.exportBranchSnapshot(eq("config-dev-main"), any(Path.class)))
|
||||
.thenAnswer(invocation -> {
|
||||
Path target = invocation.getArgument(1);
|
||||
Files.createDirectories(target);
|
||||
Path fileA = target.resolve("PEK").resolve("monitor").resolve("a.txt");
|
||||
Path fileB = target.resolve("PEK").resolve("monitor").resolve("b.txt");
|
||||
Files.createDirectories(fileA.getParent());
|
||||
int round = exportCounter.incrementAndGet();
|
||||
if (round == 1) {
|
||||
Files.write(target.resolve("a.txt"), "v1".getBytes("UTF-8"));
|
||||
Files.write(target.resolve("b.txt"), "same".getBytes("UTF-8"));
|
||||
Files.write(fileA, "v1".getBytes("UTF-8"));
|
||||
Files.write(fileB, "same".getBytes("UTF-8"));
|
||||
} else {
|
||||
Files.write(target.resolve("a.txt"), "v2".getBytes("UTF-8"));
|
||||
Files.write(target.resolve("b.txt"), "same".getBytes("UTF-8"));
|
||||
Files.write(fileA, "v2".getBytes("UTF-8"));
|
||||
Files.write(fileB, "same".getBytes("UTF-8"));
|
||||
}
|
||||
return target;
|
||||
});
|
||||
@ -221,8 +324,8 @@ class ProdSyncCoordinatorIntegrationTest {
|
||||
|
||||
assertEquals(2, pushedDirectories.size());
|
||||
Path secondPushDirectory = pushedDirectories.get(1);
|
||||
assertTrue(Files.exists(secondPushDirectory.resolve("a.txt")));
|
||||
assertFalse(Files.exists(secondPushDirectory.resolve("b.txt")));
|
||||
assertTrue(Files.exists(secondPushDirectory.resolve("PEK").resolve("monitor").resolve("a.txt")));
|
||||
assertFalse(Files.exists(secondPushDirectory.resolve("PEK").resolve("monitor").resolve("b.txt")));
|
||||
assertTrue(secondPushDirectory.getFileName().toString().startsWith("git-delta-"));
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,11 +1,15 @@
|
||||
package com.ftptool.sync.service;
|
||||
|
||||
import com.fasterxml.jackson.databind.JsonNode;
|
||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
||||
import com.ftptool.sync.config.ProdApiProperties;
|
||||
import com.ftptool.sync.config.SyncProperties;
|
||||
import com.ftptool.sync.model.PackageManifest;
|
||||
import com.ftptool.sync.model.ProdPullResult;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.springframework.http.HttpMethod;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.mock.http.client.MockClientHttpRequest;
|
||||
import org.springframework.test.web.client.MockRestServiceServer;
|
||||
import org.springframework.web.client.RestTemplate;
|
||||
|
||||
@ -13,8 +17,10 @@ import java.nio.charset.StandardCharsets;
|
||||
import java.nio.file.Files;
|
||||
import java.nio.file.Path;
|
||||
import java.util.Arrays;
|
||||
import java.util.List;
|
||||
|
||||
import static org.junit.jupiter.api.Assertions.assertEquals;
|
||||
import static org.junit.jupiter.api.Assertions.assertThrows;
|
||||
import static org.junit.jupiter.api.Assertions.assertTrue;
|
||||
import static org.mockito.Mockito.mock;
|
||||
import static org.mockito.Mockito.verify;
|
||||
@ -26,18 +32,118 @@ import static org.springframework.test.web.client.response.MockRestResponseCreat
|
||||
|
||||
class ProdConfigApiServiceHttpTest {
|
||||
|
||||
private static final ObjectMapper OBJECT_MAPPER = new ObjectMapper();
|
||||
|
||||
@Test
|
||||
void shouldBuildPushPayloadFromGitDirectoryLayout() throws Exception {
|
||||
RestTemplate restTemplate = new RestTemplate();
|
||||
MockRestServiceServer server = MockRestServiceServer.bindTo(restTemplate).build();
|
||||
|
||||
SyncProperties syncProperties = newSyncProperties("./target/http-test-work-push");
|
||||
WorkDirectoryService workDirectoryService = new WorkDirectoryService(syncProperties);
|
||||
workDirectoryService.initialize();
|
||||
|
||||
ProdApiProperties prodApiProperties = new ProdApiProperties();
|
||||
prodApiProperties.setBaseUrl("https://prod.example.com");
|
||||
prodApiProperties.setPushPath("/pic_bus_manage_monitor/configSync/pushConfig");
|
||||
prodApiProperties.setToken("static-token");
|
||||
prodApiProperties.setTokenHeaderName("token");
|
||||
ConfigCryptoService configCryptoService = mock(ConfigCryptoService.class);
|
||||
when(configCryptoService.encryptForPush("PEK", "monitor", "R_XXX_V3.0.3_XXX", "jobs/sync-job.json", "{\"enabled\":true}"))
|
||||
.thenReturn("{\"enabled\":true}");
|
||||
when(configCryptoService.encryptForPush("SHA", "gate", "R_XXX_V3.0.3_XXX", "gate-rule.json", "{\"allow\":false}"))
|
||||
.thenReturn("{\"allow\":false}");
|
||||
|
||||
ProdConfigApiService service = new ProdConfigApiService(
|
||||
prodApiProperties,
|
||||
syncProperties,
|
||||
restTemplate,
|
||||
workDirectoryService,
|
||||
mock(ProdPullAckService.class),
|
||||
configCryptoService
|
||||
);
|
||||
|
||||
Path sourceDirectory = Files.createTempDirectory("push-layout-");
|
||||
Path pekFile = sourceDirectory.resolve("PEK").resolve("monitor").resolve("jobs").resolve("sync-job.json");
|
||||
Path shaFile = sourceDirectory.resolve("SHA").resolve("gate").resolve("gate-rule.json");
|
||||
Files.createDirectories(pekFile.getParent());
|
||||
Files.createDirectories(shaFile.getParent());
|
||||
Files.write(pekFile, "{\"enabled\":true}".getBytes(StandardCharsets.UTF_8));
|
||||
Files.write(shaFile, "{\"allow\":false}".getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
server.expect(once(), request -> {
|
||||
JsonNode body = OBJECT_MAPPER.readTree(((MockClientHttpRequest) request).getBodyAsString());
|
||||
assertEquals(2, body.size());
|
||||
|
||||
assertEquals("PEK", body.get(0).get("airportId").asText());
|
||||
assertEquals("monitor", body.get(0).get("appName").asText());
|
||||
assertEquals("R_XXX_V3.0.3_XXX", body.get(0).get("configVersion").asText());
|
||||
assertEquals("jobs/sync-job.json", body.get(0).get("fileName").asText());
|
||||
assertEquals("{\"enabled\":true}", body.get(0).get("configContent").asText());
|
||||
|
||||
assertEquals("SHA", body.get(1).get("airportId").asText());
|
||||
assertEquals("gate", body.get(1).get("appName").asText());
|
||||
assertEquals("gate-rule.json", body.get(1).get("fileName").asText());
|
||||
assertEquals("{\"allow\":false}", body.get(1).get("configContent").asText());
|
||||
})
|
||||
.andExpect(method(HttpMethod.POST))
|
||||
.andExpect(header("token", "static-token"))
|
||||
.andRespond(withSuccess(
|
||||
"{\"code\":\"0\",\"data\":{\"ackFail\":[]},\"msg\":\"ok\"}",
|
||||
MediaType.APPLICATION_JSON
|
||||
));
|
||||
|
||||
PackageManifest manifest = new PackageManifest();
|
||||
manifest.setTraceId("trace-1");
|
||||
manifest.setSourceVersion("R_XXX_V3.0.3_XXX");
|
||||
|
||||
service.pushPackage(manifest, sourceDirectory);
|
||||
|
||||
verify(configCryptoService).encryptForPush("PEK", "monitor", "R_XXX_V3.0.3_XXX", "jobs/sync-job.json", "{\"enabled\":true}");
|
||||
verify(configCryptoService).encryptForPush("SHA", "gate", "R_XXX_V3.0.3_XXX", "gate-rule.json", "{\"allow\":false}");
|
||||
server.verify();
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldRejectPushFileOutsideAirportAppLayout() throws Exception {
|
||||
SyncProperties syncProperties = newSyncProperties("./target/http-test-work-invalid-push");
|
||||
WorkDirectoryService workDirectoryService = new WorkDirectoryService(syncProperties);
|
||||
workDirectoryService.initialize();
|
||||
|
||||
ProdApiProperties prodApiProperties = new ProdApiProperties();
|
||||
prodApiProperties.setBaseUrl("https://prod.example.com");
|
||||
prodApiProperties.setPushPath("/pic_bus_manage_monitor/configSync/pushConfig");
|
||||
prodApiProperties.setToken("static-token");
|
||||
|
||||
ProdConfigApiService service = new ProdConfigApiService(
|
||||
prodApiProperties,
|
||||
syncProperties,
|
||||
new RestTemplate(),
|
||||
workDirectoryService,
|
||||
mock(ProdPullAckService.class),
|
||||
new ConfigCryptoService()
|
||||
);
|
||||
|
||||
Path sourceDirectory = Files.createTempDirectory("invalid-push-layout-");
|
||||
Files.write(sourceDirectory.resolve("README.md"), "bad".getBytes(StandardCharsets.UTF_8));
|
||||
|
||||
PackageManifest manifest = new PackageManifest();
|
||||
manifest.setTraceId("trace-2");
|
||||
manifest.setSourceVersion("R_XXX_V3.0.4_XXX");
|
||||
|
||||
IllegalStateException exception = assertThrows(
|
||||
IllegalStateException.class,
|
||||
() -> service.pushPackage(manifest, sourceDirectory)
|
||||
);
|
||||
assertTrue(exception.getMessage().contains("airportId/appName/fileName"));
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldSendAckParamsWhenPullingConfig() throws Exception {
|
||||
RestTemplate restTemplate = new RestTemplate();
|
||||
MockRestServiceServer server = MockRestServiceServer.bindTo(restTemplate).build();
|
||||
|
||||
SyncProperties syncProperties = new SyncProperties();
|
||||
syncProperties.setWorkDir("./target/http-test-work");
|
||||
syncProperties.setPackageTempDir("./target/http-test-work/package");
|
||||
syncProperties.setDevToProdStagingDir("./target/http-test-work/dev-to-prod");
|
||||
syncProperties.setProdToDevStagingDir("./target/http-test-work/prod-to-dev");
|
||||
syncProperties.setPullResponseFileName("prod-config.json");
|
||||
|
||||
SyncProperties syncProperties = newSyncProperties("./target/http-test-work-pull");
|
||||
WorkDirectoryService workDirectoryService = new WorkDirectoryService(syncProperties);
|
||||
workDirectoryService.initialize();
|
||||
|
||||
@ -48,24 +154,32 @@ class ProdConfigApiServiceHttpTest {
|
||||
prodApiProperties.setTokenHeaderName("token");
|
||||
prodApiProperties.setAirportId("test-airport");
|
||||
prodApiProperties.setAppName("test-app");
|
||||
prodApiProperties.setPullConfigVersion("v1");
|
||||
prodApiProperties.setPullFileName("a.txt");
|
||||
|
||||
ProdPullAckService prodPullAckService = mock(ProdPullAckService.class);
|
||||
when(prodPullAckService.getPendingAckSummary()).thenReturn(
|
||||
new ProdPullAckService.PendingAckSummary(Arrays.asList("1", "2"), Arrays.asList("9"))
|
||||
);
|
||||
ConfigCryptoService configCryptoService = mock(ConfigCryptoService.class);
|
||||
when(configCryptoService.decryptAfterPull("test-airport", "test-app", "v1", "a.txt", "content"))
|
||||
.thenReturn("decoded-content");
|
||||
|
||||
ProdConfigApiService service = new ProdConfigApiService(
|
||||
prodApiProperties,
|
||||
syncProperties,
|
||||
restTemplate,
|
||||
workDirectoryService,
|
||||
prodPullAckService
|
||||
prodPullAckService,
|
||||
configCryptoService
|
||||
);
|
||||
|
||||
server.expect(once(), request -> {
|
||||
String uri = request.getURI().toString();
|
||||
assertTrue(uri.contains("airportId=test-airport"));
|
||||
assertTrue(uri.contains("appName=test-app"));
|
||||
assertTrue(uri.contains("configVersion=v1"));
|
||||
assertTrue(uri.contains("fileName=a.txt"));
|
||||
assertTrue(uri.contains("ackSuc=1,2") || uri.contains("ackSuc=1%2C2"));
|
||||
assertTrue(uri.contains("ackFail=9"));
|
||||
})
|
||||
@ -76,16 +190,82 @@ class ProdConfigApiServiceHttpTest {
|
||||
MediaType.APPLICATION_JSON
|
||||
));
|
||||
|
||||
ProdPullResult result = service.pullConfigSnapshot();
|
||||
List<ProdPullResult> results = service.pullConfigSnapshots();
|
||||
assertEquals(1, results.size());
|
||||
ProdPullResult result = results.get(0);
|
||||
|
||||
assertEquals("v1", result.getSourceVersion());
|
||||
assertEquals(1, result.getPulledConfigIds().size());
|
||||
assertEquals("1", result.getPulledConfigIds().get(0));
|
||||
Path restoredFile = result.getContentDirectory().resolve("a.txt");
|
||||
Path restoredFile = result.getContentDirectory()
|
||||
.resolve("test-airport")
|
||||
.resolve("test-app")
|
||||
.resolve("a.txt");
|
||||
assertTrue(Files.exists(restoredFile));
|
||||
assertEquals("content", new String(Files.readAllBytes(restoredFile), StandardCharsets.UTF_8));
|
||||
assertEquals("decoded-content", new String(Files.readAllBytes(restoredFile), StandardCharsets.UTF_8));
|
||||
|
||||
verify(configCryptoService).decryptAfterPull("test-airport", "test-app", "v1", "a.txt", "content");
|
||||
verify(prodPullAckService).markPendingAsReported();
|
||||
server.verify();
|
||||
}
|
||||
|
||||
@Test
|
||||
void shouldSplitPulledResultsByConfigVersion() throws Exception {
|
||||
RestTemplate restTemplate = new RestTemplate();
|
||||
MockRestServiceServer server = MockRestServiceServer.bindTo(restTemplate).build();
|
||||
|
||||
SyncProperties syncProperties = newSyncProperties("./target/http-test-work-pull-grouped");
|
||||
WorkDirectoryService workDirectoryService = new WorkDirectoryService(syncProperties);
|
||||
workDirectoryService.initialize();
|
||||
|
||||
ProdApiProperties prodApiProperties = new ProdApiProperties();
|
||||
prodApiProperties.setBaseUrl("https://prod.example.com");
|
||||
prodApiProperties.setPullPath("/pic_bus_manage_monitor/configSync/pullConfig");
|
||||
prodApiProperties.setToken("static-token");
|
||||
|
||||
ProdPullAckService prodPullAckService = mock(ProdPullAckService.class);
|
||||
when(prodPullAckService.getPendingAckSummary()).thenReturn(
|
||||
new ProdPullAckService.PendingAckSummary(Arrays.<String>asList(), Arrays.<String>asList())
|
||||
);
|
||||
|
||||
ProdConfigApiService service = new ProdConfigApiService(
|
||||
prodApiProperties,
|
||||
syncProperties,
|
||||
restTemplate,
|
||||
workDirectoryService,
|
||||
prodPullAckService,
|
||||
new ConfigCryptoService()
|
||||
);
|
||||
|
||||
server.expect(once(), request -> {
|
||||
})
|
||||
.andExpect(method(HttpMethod.GET))
|
||||
.andRespond(withSuccess(
|
||||
"{\"code\":\"0\",\"data\":["
|
||||
+ "{\"id\":\"1\",\"airportId\":\"PEK\",\"appName\":\"monitor\",\"configVersion\":\"v1\",\"configContent\":\"one\",\"fileName\":\"a.txt\"},"
|
||||
+ "{\"id\":\"2\",\"airportId\":\"SHA\",\"appName\":\"gate\",\"configVersion\":\"v2\",\"configContent\":\"two\",\"fileName\":\"b.txt\"}"
|
||||
+ "],\"msg\":\"ok\"}",
|
||||
MediaType.APPLICATION_JSON
|
||||
));
|
||||
|
||||
List<ProdPullResult> results = service.pullConfigSnapshots();
|
||||
|
||||
assertEquals(2, results.size());
|
||||
assertEquals("v1", results.get(0).getSourceVersion());
|
||||
assertEquals("v2", results.get(1).getSourceVersion());
|
||||
assertTrue(Files.exists(results.get(0).getContentDirectory().resolve("PEK").resolve("monitor").resolve("a.txt")));
|
||||
assertTrue(Files.exists(results.get(1).getContentDirectory().resolve("SHA").resolve("gate").resolve("b.txt")));
|
||||
|
||||
server.verify();
|
||||
}
|
||||
|
||||
private SyncProperties newSyncProperties(String workDir) {
|
||||
SyncProperties syncProperties = new SyncProperties();
|
||||
syncProperties.setWorkDir(workDir);
|
||||
syncProperties.setPackageTempDir(workDir + "/package");
|
||||
syncProperties.setDevToProdStagingDir(workDir + "/dev-to-prod");
|
||||
syncProperties.setProdToDevStagingDir(workDir + "/prod-to-dev");
|
||||
syncProperties.setPullResponseFileName("prod-config.json");
|
||||
return syncProperties;
|
||||
}
|
||||
}
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user