refactor: 收敛同步工具为 Git 直连单 prod-agent 架构
- 重写主设计文档与详细设计文档，移除 FTP 中转方案口径 - 新增 Git 直连架构设计文档，明确单 prod-agent 部署模式 - 将生产侧主同步流程切换为 Git -> PROD 和 PROD -> Git 两条直连链路 - 新增正式调度任务 GitToProdSyncJob 与 ProdToGitSnapshotJob - 移除 commons-net 主依赖并将 FTP 能力退出主运行面 - 清理 application.properties 中 FTP/ACK 相关公共配置 - 收敛 SyncProperties，删除 FTP 远端目录与 ACK 扫描字段 - 精简 schema.sql，移除 sync_ack 表，仅保留 sync_checkpoint 与 sync_task - 将 dev-agent、FTP、ACK 相关旧类降级为退役占位实现 - 调整项目命名与默认配置，统一到 Git 直连架构 - 完成编译验证
This commit is contained in:
dark
parent
064a68d2a3
commit
dcfdc83444
@ -1,12 +1,12 @@
|
|||||||
# 基于 FTP 中转的配置双向同步工具设计方案
|
# 基于 Git 直连的配置双向同步工具设计方案
|
||||||
|
|
||||||
## 1. 文档目的
|
## 1. 文档目的
|
||||||
|
|
||||||
本文档用于说明一套基于 FTP 中转的配置同步工具设计方案，满足以下目标：
|
本文档用于说明一套基于 Git 直连的配置同步工具设计方案，满足以下目标：
|
||||||
|
|
||||||
- 开发环境定时从 Git 拉取新配置，并通过生产环境 `push` 接口推送到生产
|
- 生产环境定时从开发 Git 拉取新配置，并调用生产 `push` 接口导入生产
|
||||||
- 生产环境定时从 `pull` 接口拉取新配置，并同步回开发环境 Git
|
- 生产环境定时从生产 `pull` 接口拉取当前配置，并回写到开发 Git
|
||||||
- 在开发环境与生产环境不能直接互通时，通过 FTP 服务作为中转通道完成双向同步
|
- 在 FTP 不再使用的前提下，简化整体架构、降低维护成本
|
||||||
|
|
||||||
## 2. 已知约束
|
## 2. 已知约束
|
||||||
|
|
||||||
@ -19,82 +19,66 @@
|
|||||||
|
|
||||||
### 2.2 网络与部署约束
|
### 2.2 网络与部署约束
|
||||||
|
|
||||||
- 无法登录 FTP 所在服务器主机
|
- 生产环境可以访问开发 Git 仓库
|
||||||
- 只能访问 FTP 服务：`IP + 端口 + 用户名/密码`
|
- 生产环境需要能够调用生产系统 `push/pull` 接口
|
||||||
- 网络拓扑如下：
|
- FTP 不再使用
|
||||||
|
|
||||||
```text
|
建议先确认一个关键前提：
|
||||||
开发环境 <----> FTP A <----> 生产环境
|
|
||||||
```
|
- 生产环境是否对开发 Git 具备“读 + 写”权限
|
||||||
|
|
||||||
说明：
|
说明：
|
||||||
|
|
||||||
- 开发环境可以访问 FTP A
|
- 如果生产环境只能读取 Git，无法推送分支，那么“生产 -> 开发 Git”这条链路不能闭环
|
||||||
- 生产环境可以访问 FTP A
|
- 如果生产环境可以读取和推送 Git，则整套同步可以收敛为单点部署
|
||||||
- 开发与生产不假设可以直接互通
|
|
||||||
|
|
||||||
## 3. 设计原则
|
## 3. 新架构结论
|
||||||
|
|
||||||
- 同一套程序，按不同 `profile` 部署在开发和生产两端
|
在新条件下，不再推荐“双端代理 + FTP 中转”。
|
||||||
- 通过 FTP 传递标准化同步包，避免环境间直接通信依赖
|
|
||||||
- 使用本地状态库记录同步任务、检查点、应答信息，保证可追踪、可恢复
|
|
||||||
- 同步流程必须具备幂等控制，避免重复推送、重复提交
|
|
||||||
- 开发到生产、生产回开发必须隔离处理，避免双向同步形成死循环
|
|
||||||
|
|
||||||
## 4. 总体方案
|
推荐改为：
|
||||||
|
|
||||||
推荐采用“**双端代理 + FTP 中转 + 本地状态库**”架构：
|
- **单端代理 + Git 直连 + 本地状态库**
|
||||||
|
|
||||||
- `Sync-Agent-Dev`：部署在开发环境
|
即只在生产环境部署一套同步服务：
|
||||||
- `Sync-Agent-Prod`：部署在生产环境
|
|
||||||
- `FTP A`：作为唯一中转通道
|
- `Sync-Agent-Prod`
|
||||||
- `H2`：记录同步状态、任务、检查点、重试信息
|
|
||||||
|
它同时承担两类任务：
|
||||||
|
|
||||||
|
1. 从开发 Git 拉取配置，推送到生产
|
||||||
|
2. 从生产 `pull` 接口拉取配置，回写到开发 Git
|
||||||
|
|
||||||
整体结构如下：
|
整体结构如下：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
开发环境
|
开发 Git 仓库 <----> 生产环境 Sync-Agent-Prod <----> 生产系统 push/pull 接口
|
||||||
Sync-Agent-Dev
|
|
||||||
|- 拉取 Git
|
|
||||||
|- 上传/下载 FTP A
|
|
||||||
|- 写入 Git
|
|
||||||
|
|
||||||
生产环境
|
|
||||||
Sync-Agent-Prod
|
|
||||||
|- 调用生产 pull 接口
|
|
||||||
|- 调用生产 push 接口
|
|
||||||
|- 上传/下载 FTP A
|
|
||||||
|
|
||||||
中转
|
|
||||||
FTP A
|
|
||||||
|- dev-to-prod/
|
|
||||||
|- prod-to-dev/
|
|
||||||
|- ack/
|
|
||||||
|- failed/
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 5. 部署模式
|
## 4. 为什么要改成单端部署
|
||||||
|
|
||||||
建议只维护一套代码，通过 Spring Profile 控制角色：
|
新架构相比旧方案有明显优势：
|
||||||
|
|
||||||
- `dev-agent`：启用开发侧能力
|
- 去掉 FTP，中转链路减少一跳
|
||||||
- `prod-agent`：启用生产侧能力
|
- 去掉打包上传、轮询下载、ACK 回执等中间环节
|
||||||
|
- 部署节点减少为 1 个，运维更简单
|
||||||
|
- 故障点减少，排查路径更短
|
||||||
|
- 数据流更直接，状态一致性更容易控制
|
||||||
|
|
||||||
### 5.1 开发侧职责
|
## 5. 总体方案
|
||||||
|
|
||||||
- 定时拉取 Git 指定分支的新配置
|
推荐在生产环境部署唯一同步实例：
|
||||||
- 判断是否存在新的有效版本
|
|
||||||
- 打包配置并上传到 FTP
|
|
||||||
- 下载生产侧回传的同步包
|
|
||||||
- 将生产侧回传配置写入 Git
|
|
||||||
- 提交并推送到远端仓库
|
|
||||||
|
|
||||||
### 5.2 生产侧职责
|
- `Sync-Agent-Prod`
|
||||||
|
|
||||||
- 轮询 FTP，获取开发侧上传的配置包
|
其职责如下：
|
||||||
- 校验后调用生产 `push` 接口导入配置
|
|
||||||
- 定时调用生产 `pull` 接口拉取最新配置
|
- 拉取开发 Git 主配置分支
|
||||||
- 打包并上传回 FTP，供开发侧消费
|
- 检查是否存在待下发的新版本
|
||||||
|
- 调用生产 `push` 接口导入配置
|
||||||
|
- 定时调用生产 `pull` 接口获取当前生产配置
|
||||||
|
- 将生产配置写回 Git 快照分支
|
||||||
|
- 使用 H2 记录同步状态、检查点、失败记录
|
||||||
|
|
||||||
## 6. 技术选型
|
## 6. 技术选型
|
||||||
|
|
||||||
@ -104,82 +88,86 @@
|
|||||||
| 框架 | Spring Boot 2.7.18 | 主体框架 |
|
| 框架 | Spring Boot 2.7.18 | 主体框架 |
|
||||||
| 调度 | Spring Scheduling | 实现定时任务 |
|
| 调度 | Spring Scheduling | 实现定时任务 |
|
||||||
| 重试 | Spring Retry | 失败重试 |
|
| 重试 | Spring Retry | 失败重试 |
|
||||||
| 数据库 | H2 File Mode | 轻量、嵌入式、可持久化 |
|
| 数据库 | H2 File Mode | 持久化检查点与任务状态 |
|
||||||
| Git 操作 | JGit | 纯 Java 实现 |
|
| Git 操作 | JGit | 生产环境直接读写 Git |
|
||||||
| FTP 操作 | Apache Commons Net | 主流 FTP 客户端 |
|
| HTTP 调用 | RestTemplate | 调用生产 `push/pull` 接口 |
|
||||||
| JSON | Jackson | 标准序列化组件 |
|
| JSON | Jackson | 标准序列化 |
|
||||||
| 日志 | SLF4J + Logback | 默认日志能力 |
|
| 日志 | SLF4J + Logback | 默认日志能力 |
|
||||||
|
|
||||||
### 6.1 数据库模式建议
|
说明：
|
||||||
|
|
||||||
虽然需求提到“类似 H2 的轻量化内存数据库”，但本场景不建议纯内存模式，原因如下：
|
- FTP 客户端依赖在新方案里已经不是核心能力
|
||||||
|
- 标准同步包、FTP 目录、ACK 文件等设计可以整体下线
|
||||||
|
|
||||||
- 服务重启后需要保留同步检查点
|
## 7. 部署模式
|
||||||
- 失败任务需要支持补偿和人工追踪
|
|
||||||
- 需要记录包处理状态，避免重复消费
|
|
||||||
|
|
||||||
因此建议使用：
|
### 7.1 推荐模式
|
||||||
|
|
||||||
- `H2 File Mode`
|
推荐只部署：
|
||||||
|
|
||||||
即本地文件数据库，仍然轻量，但支持状态持久化。
|
- `prod-agent`
|
||||||
|
|
||||||
## 7. 核心业务流程
|
不再需要：
|
||||||
|
|
||||||
系统包含两条主链路。
|
- `dev-agent`
|
||||||
|
- FTP 中转服务
|
||||||
|
|
||||||
### 7.1 链路一：开发 Git -> 生产 push 接口
|
### 7.2 运行位置
|
||||||
|
|
||||||
用途：将开发环境 Git 中的新配置推送到生产环境。
|
同步工具建议运行在生产环境可控节点上，要求：
|
||||||
|
|
||||||
|
- 能访问开发 Git
|
||||||
|
- 能访问生产 `push/pull` 接口
|
||||||
|
- 能持久化本地 H2 文件数据库
|
||||||
|
|
||||||
|
## 8. 两条核心链路
|
||||||
|
|
||||||
|
### 8.1 链路一：开发 Git -> 生产 push 接口
|
||||||
|
|
||||||
|
用途：
|
||||||
|
|
||||||
|
- 将开发配置分支中的新配置同步到生产环境
|
||||||
|
|
||||||
流程如下：
|
流程如下：
|
||||||
|
|
||||||
1. `dev-agent` 定时拉取 Git 指定分支
|
1. `Sync-Agent-Prod` 定时拉取开发 Git 指定分支
|
||||||
2. 判断 Git 最新提交是否为新的有效配置版本
|
2. 获取最新提交版本号，例如 Git Commit ID
|
||||||
3. 将配置目录打包为标准同步包
|
3. 判断该版本是否已成功同步
|
||||||
4. 上传至 FTP 路径 `dev-to-prod/out/`
|
4. 如果未同步，则导出配置目录
|
||||||
5. `prod-agent` 轮询 FTP，发现新包后下载
|
5. 调用生产 `push` 接口导入配置
|
||||||
6. 校验包完整性、幂等键和来源信息
|
6. 成功后更新本地检查点和任务状态
|
||||||
7. 调用生产环境 `push` 接口导入配置
|
|
||||||
8. 成功后生成 `ack` 文件上传到 FTP
|
|
||||||
9. `dev-agent` 读取 `ack`，将任务状态更新为成功
|
|
||||||
|
|
||||||
建议时序图如下：
|
建议时序图如下：
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant G as Git(开发)
|
participant G as Git(开发)
|
||||||
participant D as Sync-Agent-Dev
|
|
||||||
participant F as FTP A
|
|
||||||
participant P as Sync-Agent-Prod
|
participant P as Sync-Agent-Prod
|
||||||
participant API as 生产Push接口
|
participant API as 生产Push接口
|
||||||
|
|
||||||
D->>G: 定时 pull 配置
|
P->>G: 定时 pull config-dev-main
|
||||||
D->>D: 检查是否有新版本
|
P->>P: 判断是否有新 commit
|
||||||
D->>D: 打包 zip + manifest
|
P->>P: 导出配置目录
|
||||||
D->>F: 上传 dev-to-prod/out/
|
|
||||||
P->>F: 轮询并下载新包
|
|
||||||
P->>P: 校验 hash/traceId
|
|
||||||
P->>API: 调用 push 接口
|
P->>API: 调用 push 接口
|
||||||
API-->>P: 返回处理结果
|
API-->>P: 返回处理结果
|
||||||
P->>F: 上传 ack
|
P->>P: 更新 sync_task / checkpoint
|
||||||
D->>F: 读取 ack
|
|
||||||
D->>D: 更新状态为成功
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### 7.2 链路二：生产 pull 接口 -> 开发 Git
|
### 8.2 链路二：生产 pull 接口 -> 开发 Git
|
||||||
|
|
||||||
用途：将生产环境当前配置回传到开发环境，形成配置镜像或审计记录。
|
用途：
|
||||||
|
|
||||||
|
- 将当前生产配置快照回写到开发 Git，用于镜像、审计、回溯
|
||||||
|
|
||||||
流程如下：
|
流程如下：
|
||||||
|
|
||||||
1. `prod-agent` 定时调用生产 `pull` 接口
|
1. `Sync-Agent-Prod` 定时调用生产 `pull` 接口
|
||||||
2. 将返回配置标准化后计算版本标识或内容哈希
|
2. 将返回结果标准化并计算内容哈希
|
||||||
3. 如果与上次同步结果不同，则打包上传到 FTP `prod-to-dev/out/`
|
3. 判断该版本或哈希是否已同步
|
||||||
4. `dev-agent` 轮询 FTP 并下载新包
|
4. 如果未同步，则切换到生产快照分支
|
||||||
5. 解包后写入本地 Git 工作目录
|
5. 写入配置文件
|
||||||
6. 提交 commit 并推送到远端 Git
|
6. 提交 commit 并 push 到开发 Git
|
||||||
7. 成功后写回 `ack`
|
7. 成功后更新本地检查点和任务状态
|
||||||
|
|
||||||
建议时序图如下：
|
建议时序图如下：
|
||||||
|
|
||||||
@ -187,417 +175,312 @@ sequenceDiagram
|
|||||||
sequenceDiagram
|
sequenceDiagram
|
||||||
participant API as 生产Pull接口
|
participant API as 生产Pull接口
|
||||||
participant P as Sync-Agent-Prod
|
participant P as Sync-Agent-Prod
|
||||||
participant F as FTP A
|
|
||||||
participant D as Sync-Agent-Dev
|
|
||||||
participant G as Git(开发)
|
participant G as Git(开发)
|
||||||
|
|
||||||
P->>API: 定时调用 pull 接口
|
P->>API: 定时调用 pull 接口
|
||||||
API-->>P: 返回当前配置
|
API-->>P: 返回当前生产配置
|
||||||
P->>P: 标准化并计算 hash
|
P->>P: 标准化并计算 hash/version
|
||||||
P->>F: 上传 prod-to-dev/out/
|
P->>G: checkout config-prod-snapshot
|
||||||
D->>F: 轮询并下载新包
|
P->>G: commit + push
|
||||||
D->>D: 解包并写入工作区
|
P->>P: 更新 sync_task / checkpoint
|
||||||
D->>G: commit + push
|
|
||||||
D->>F: 上传 ack
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## 8. 标准同步包设计
|
## 9. Git 分支策略
|
||||||
|
|
||||||
为保证跨环境处理一致，建议所有同步内容封装为统一格式的压缩包。
|
这个设计点仍然必须保留。
|
||||||
|
|
||||||
### 8.1 包结构
|
不建议将“开发配置推生产”和“生产配置回写 Git”使用同一个 Git 分支，否则非常容易形成同步闭环。
|
||||||
|
|
||||||
```text
|
推荐分支如下：
|
||||||
package.zip
|
|
||||||
|- manifest.json
|
|
||||||
|- config/
|
|
||||||
|- sha256.txt
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8.2 manifest 字段建议
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"traceId": "uuid",
|
|
||||||
"direction": "DEV_TO_PROD",
|
|
||||||
"sourceEnv": "DEV",
|
|
||||||
"sourceVersion": "gitCommitId",
|
|
||||||
"contentHash": "sha256",
|
|
||||||
"createdAt": "2026-04-15T10:00:00+08:00"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 8.3 字段说明
|
|
||||||
|
|
||||||
- `traceId`：本次同步唯一流水号
|
|
||||||
- `direction`：同步方向，例如 `DEV_TO_PROD`、`PROD_TO_DEV`
|
|
||||||
- `sourceEnv`：来源环境
|
|
||||||
- `sourceVersion`：来源版本号，开发侧通常为 Git Commit ID
|
|
||||||
- `contentHash`：配置内容哈希，便于判断重复包
|
|
||||||
- `createdAt`：包生成时间
|
|
||||||
|
|
||||||
## 9. FTP 目录规划
|
|
||||||
|
|
||||||
建议在 FTP A 上使用如下目录结构：
|
|
||||||
|
|
||||||
```text
|
|
||||||
/dev-to-prod/out/
|
|
||||||
/dev-to-prod/ack/
|
|
||||||
/prod-to-dev/out/
|
|
||||||
/prod-to-dev/ack/
|
|
||||||
/failed/
|
|
||||||
```
|
|
||||||
|
|
||||||
目录说明：
|
|
||||||
|
|
||||||
- `/dev-to-prod/out/`：开发侧发往生产侧的同步包
|
|
||||||
- `/dev-to-prod/ack/`：生产侧返回的处理应答
|
|
||||||
- `/prod-to-dev/out/`：生产侧发往开发侧的同步包
|
|
||||||
- `/prod-to-dev/ack/`：开发侧返回的处理应答
|
|
||||||
- `/failed/`：失败包归档目录
|
|
||||||
|
|
||||||
### 9.1 上传规范
|
|
||||||
|
|
||||||
为避免消费端读取到半截文件，建议采用临时文件上传策略：
|
|
||||||
|
|
||||||
1. 先上传为 `.tmp`
|
|
||||||
2. 上传完成后重命名为正式 `.zip`
|
|
||||||
3. 消费端只处理 `.zip` 文件
|
|
||||||
|
|
||||||
## 10. Git 分支策略
|
|
||||||
|
|
||||||
这是方案中的关键设计点。
|
|
||||||
|
|
||||||
不建议将“开发配置推生产”和“生产配置回传开发”写到同一个 Git 分支，否则极易形成循环同步。
|
|
||||||
|
|
||||||
建议拆分为两个分支：
|
|
||||||
|
|
||||||
- `config-dev-main`：开发主配置分支
|
- `config-dev-main`：开发主配置分支
|
||||||
- `config-prod-snapshot`：生产配置镜像分支
|
- `config-prod-snapshot`：生产配置镜像分支
|
||||||
|
|
||||||
同步规则：
|
同步规则：
|
||||||
|
|
||||||
- `DEV -> PROD` 只消费 `config-dev-main`
|
- `Git -> PROD` 只消费 `config-dev-main`
|
||||||
- `PROD -> DEV` 只写入 `config-prod-snapshot`
|
- `PROD -> Git` 只写入 `config-prod-snapshot`
|
||||||
|
|
||||||
### 10.1 这样设计的好处
|
### 9.1 这样设计的价值
|
||||||
|
|
||||||
- 避免双向同步形成闭环
|
- 避免生产回写内容再次触发下发
|
||||||
- 生产回传配置不会覆盖开发主线
|
- 生产快照不会污染开发主线
|
||||||
- 便于审计“生产当前实际配置”
|
- 便于审计“生产当前实际配置”
|
||||||
|
|
||||||
### 10.2 机器人提交标记
|
### 9.2 机器人提交标记
|
||||||
|
|
||||||
建议同步工具在 commit message 中增加固定前缀，例如：
|
建议同步工具统一使用固定 commit message 前缀，例如：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
sync(prod->git): traceId=xxx version=xxx
|
sync(prod->git): traceId=xxx version=xxx
|
||||||
```
|
```
|
||||||
|
|
||||||
开发侧扫描 Git 时应忽略同步机器人生成的提交，进一步降低环路风险。
|
同时：
|
||||||
|
|
||||||
## 11. 本地状态库设计
|
- `Git -> PROD` 扫描时只关注 `config-dev-main`
|
||||||
|
- 不读取 `config-prod-snapshot`
|
||||||
|
|
||||||
建议至少建立以下 3 张表。
|
## 10. 状态库设计
|
||||||
|
|
||||||
### 11.1 `sync_checkpoint`
|
新方案建议保留以下核心表：
|
||||||
|
|
||||||
用于记录各方向的最后一次成功检查点。
|
### 10.1 `sync_checkpoint`
|
||||||
|
|
||||||
| 字段 | 类型 | 说明 |
|
用于记录各方向最后一次成功同步的检查点。
|
||||||
| --- | --- | --- |
|
|
||||||
| id | bigint | 主键 |
|
|
||||||
| direction | varchar | 同步方向 |
|
|
||||||
| last_success_version | varchar | 最后成功版本 |
|
|
||||||
| last_success_hash | varchar | 最后成功内容哈希 |
|
|
||||||
| updated_at | timestamp | 更新时间 |
|
|
||||||
|
|
||||||
### 11.2 `sync_task`
|
关键字段：
|
||||||
|
|
||||||
|
- `direction`
|
||||||
|
- `last_success_version`
|
||||||
|
- `last_success_hash`
|
||||||
|
- `updated_at`
|
||||||
|
|
||||||
|
### 10.2 `sync_task`
|
||||||
|
|
||||||
用于记录每次同步任务生命周期。
|
用于记录每次同步任务生命周期。
|
||||||
|
|
||||||
| 字段 | 类型 | 说明 |
|
关键字段：
|
||||||
| --- | --- | --- |
|
|
||||||
| id | bigint | 主键 |
|
|
||||||
| trace_id | varchar | 流水号 |
|
|
||||||
| direction | varchar | 同步方向 |
|
|
||||||
| source_version | varchar | 来源版本 |
|
|
||||||
| package_name | varchar | 包文件名 |
|
|
||||||
| status | varchar | 状态 |
|
|
||||||
| retry_count | int | 重试次数 |
|
|
||||||
| error_msg | clob | 错误信息 |
|
|
||||||
| created_at | timestamp | 创建时间 |
|
|
||||||
| updated_at | timestamp | 更新时间 |
|
|
||||||
|
|
||||||
### 11.3 `sync_ack`
|
- `trace_id`
|
||||||
|
- `direction`
|
||||||
|
- `source_version`
|
||||||
|
- `content_hash`
|
||||||
|
- `status`
|
||||||
|
- `retry_count`
|
||||||
|
- `error_msg`
|
||||||
|
|
||||||
用于记录应答信息。
|
### 10.3 `sync_ack`
|
||||||
|
|
||||||
| 字段 | 类型 | 说明 |
|
在新架构下：
|
||||||
| --- | --- | --- |
|
|
||||||
| id | bigint | 主键 |
|
|
||||||
| trace_id | varchar | 流水号 |
|
|
||||||
| ack_side | varchar | 应答方 |
|
|
||||||
| ack_status | varchar | 应答状态 |
|
|
||||||
| ack_time | timestamp | 应答时间 |
|
|
||||||
| remark | varchar | 备注 |
|
|
||||||
|
|
||||||
## 12. 幂等与一致性设计
|
- 不再作为跨节点 ACK 使用
|
||||||
|
- 已退出当前主 schema
|
||||||
|
|
||||||
### 12.1 幂等键建议
|
如果后续需要审计扩展，可以单独恢复为接口调用日志表。
|
||||||
|
|
||||||
建议以如下组合作为幂等键：
|
## 11. 幂等设计
|
||||||
|
|
||||||
|
建议继续使用以下组合作为幂等键：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
direction + sourceVersion + contentHash
|
direction + sourceVersion + contentHash
|
||||||
```
|
```
|
||||||
|
|
||||||
约束效果：
|
作用：
|
||||||
|
|
||||||
- 已经处理过的包不能重复推送
|
- 同一开发版本不会重复推生产
|
||||||
- 已经提交过的生产快照不能重复写 Git
|
- 同一生产快照不会重复写 Git
|
||||||
|
|
||||||
### 12.2 一致性策略
|
## 12. 失败处理与补偿
|
||||||
|
|
||||||
本方案属于跨系统、跨网络的异步同步，不适合做强一致事务。
|
### 12.1 自动重试
|
||||||
|
|
||||||
建议采用：
|
|
||||||
|
|
||||||
- “本地落库 + 外部调用 + 最终一致”模式
|
|
||||||
- 每一步记录状态
|
|
||||||
- 失败后允许自动重试或人工补偿
|
|
||||||
|
|
||||||
## 13. 失败处理与补偿机制
|
|
||||||
|
|
||||||
### 13.1 自动重试
|
|
||||||
|
|
||||||
以下场景建议自动重试：
|
以下场景建议自动重试：
|
||||||
|
|
||||||
- FTP 上传失败
|
- Git pull 失败
|
||||||
- FTP 下载失败
|
- Git push 失败
|
||||||
- 生产 `push` 接口调用失败
|
- 生产 `push` 接口调用失败
|
||||||
- 生产 `pull` 接口调用失败
|
- 生产 `pull` 接口调用失败
|
||||||
- Git push 失败
|
|
||||||
|
|
||||||
建议策略：
|
建议策略：
|
||||||
|
|
||||||
- 最大重试次数：`3 ~ 5`
|
- 最大重试次数：`3 ~ 5`
|
||||||
- 重试间隔：指数退避，例如 `30s / 60s / 120s`
|
- 指数退避：`30s / 60s / 120s`
|
||||||
|
|
||||||
### 13.2 失败归档
|
### 12.2 失败落库
|
||||||
|
|
||||||
连续失败后建议：
|
失败后建议：
|
||||||
|
|
||||||
- 将包移动到 FTP 的 `/failed/`
|
- 更新 `sync_task.status=FAILED`
|
||||||
- 将任务状态置为 `FAILED`
|
- 记录异常堆栈摘要
|
||||||
- 记录完整错误信息
|
- 增加重试次数
|
||||||
- 触发告警
|
- 保留最近一次成功检查点不变
|
||||||
|
|
||||||
### 13.3 人工补偿
|
### 12.3 人工补偿
|
||||||
|
|
||||||
后续可以增加一个管理接口，支持：
|
后续可增加管理接口，支持：
|
||||||
|
|
||||||
- 按 `traceId` 重新执行
|
- 按 `traceId` 重试
|
||||||
- 重置任务状态
|
- 按方向重跑最近一次失败任务
|
||||||
- 查看失败原因
|
- 查询最近同步记录
|
||||||
|
|
||||||
## 14. 安全设计
|
## 13. 安全设计
|
||||||
|
|
||||||
### 14.1 传输安全
|
### 13.1 Git 访问建议
|
||||||
|
|
||||||
优先级建议如下：
|
推荐使用：
|
||||||
|
|
||||||
1. 优先使用 `FTPS`
|
- HTTPS + Token
|
||||||
2. 如果只能使用普通 FTP，建议对同步包内容做 AES 加密
|
|
||||||
|
|
||||||
### 14.2 凭据管理
|
或：
|
||||||
|
|
||||||
以下信息不得写死在代码中：
|
- SSH Deploy Key
|
||||||
|
|
||||||
- FTP 地址、端口、用户名、密码
|
### 13.2 权限建议
|
||||||
- Git 用户名、密码或 Token
|
|
||||||
- 生产接口认证信息
|
|
||||||
|
|
||||||
建议通过以下方式外置：
|
生产环境访问 Git 的账号建议采用最小权限原则：
|
||||||
|
|
||||||
- `application-*.properties`
|
- 对 `config-dev-main` 至少有读取权限
|
||||||
- 环境变量
|
- 对 `config-prod-snapshot` 需要推送权限
|
||||||
- 启动参数
|
|
||||||
|
|
||||||
### 14.3 审计日志
|
更理想的做法：
|
||||||
|
|
||||||
建议记录：
|
- 使用专用机器人账号
|
||||||
|
- 对开发主分支启用保护
|
||||||
|
- 限制机器人只写快照分支
|
||||||
|
|
||||||
- 谁发起了同步
|
### 13.3 生产接口认证
|
||||||
- 同步方向
|
|
||||||
- 来源版本
|
|
||||||
- 包名
|
|
||||||
- 接口调用结果
|
|
||||||
- 异常原因
|
|
||||||
|
|
||||||
## 15. 项目结构建议
|
生产 `push/pull` 接口建议使用：
|
||||||
|
|
||||||
有两种实现方式。
|
- `Bearer Token`
|
||||||
|
- HTTPS
|
||||||
|
|
||||||
### 15.1 方案 A：单工程 + Profile 切换
|
## 14. 项目结构建议
|
||||||
|
|
||||||
适用于项目规模较小、交付快的场景。
|
新架构下建议进一步简化模块职责：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
sync-tool
|
sync-tool
|
||||||
|- src/main/java
|
|- src/main/java
|
||||||
| |- config
|
| |- config
|
||||||
| |- ftp
|
|
||||||
| |- git
|
| |- git
|
||||||
| |- job
|
| |- job
|
||||||
| |- package
|
|
||||||
| |- repository
|
| |- repository
|
||||||
| |- service
|
| |- service
|
||||||
| |- web
|
| |- web
|
||||||
|- src/main/resources
|
|- src/main/resources
|
||||||
| |- application.properties
|
| |- application.properties
|
||||||
| |- application-dev-agent.properties
|
|
||||||
| |- application-prod-agent.properties
|
| |- application-prod-agent.properties
|
||||||
```
|
```
|
||||||
|
|
||||||
### 15.2 方案 B：多模块拆分
|
说明：
|
||||||
|
|
||||||
适用于后续可能演化较多、职责更清晰的场景。
|
- `prod-agent` 是唯一正式运行角色
|
||||||
|
- `dev-agent` 与 FTP 相关模块已退出主运行面
|
||||||
|
|
||||||
```text
|
## 15. 核心模块划分
|
||||||
sync-tool
|
|
||||||
|- common
|
|
||||||
|- dev-agent
|
|
||||||
|- prod-agent
|
|
||||||
```
|
|
||||||
|
|
||||||
当前建议优先采用：
|
建议保留并聚焦以下模块：
|
||||||
|
|
||||||
- `方案 A：单工程 + Profile`
|
- `GitClientService`
|
||||||
|
- clone / pull / checkout / commit / push
|
||||||
理由：
|
- `ProdConfigApiService`
|
||||||
|
- 调用生产 `push/pull` 接口
|
||||||
- 实现成本低
|
|
||||||
- 运维简单
|
|
||||||
- 早期更适合快速打通链路
|
|
||||||
|
|
||||||
## 16. 核心模块划分
|
|
||||||
|
|
||||||
建议按职责拆分以下模块：
|
|
||||||
|
|
||||||
- `GitService`
|
|
||||||
- 拉取仓库
|
|
||||||
- 检查最新提交
|
|
||||||
- 提交并推送生产回传配置
|
|
||||||
- `FtpService`
|
|
||||||
- 上传、下载、重命名、目录扫描
|
|
||||||
- `PackageService`
|
|
||||||
- 生成 zip
|
|
||||||
- 生成 manifest
|
|
||||||
- 校验 hash
|
|
||||||
- `SyncTaskService`
|
- `SyncTaskService`
|
||||||
- 任务创建
|
- 任务创建、状态变更、重试次数维护
|
||||||
- 状态变更
|
- `CheckpointService`
|
||||||
- 检查点维护
|
- 成功检查点维护
|
||||||
- `ProdPushService`
|
- `ProdSyncCoordinator`
|
||||||
- 调用生产 `push` 接口
|
- 串联双向同步流程
|
||||||
- `ProdPullService`
|
|
||||||
- 调用生产 `pull` 接口
|
|
||||||
- `AckService`
|
|
||||||
- 生成和消费 ack 文件
|
|
||||||
- `JobScheduler`
|
- `JobScheduler`
|
||||||
- 各类定时任务调度
|
- 定时调度
|
||||||
|
|
||||||
## 17. 定时任务建议
|
已退出主运行面：
|
||||||
|
|
||||||
### 17.1 开发侧任务
|
- `FtpClientService`
|
||||||
|
- FTP 包上传下载逻辑
|
||||||
|
- FTP ACK 逻辑
|
||||||
|
|
||||||
- `GitPullJob`
|
## 16. 定时任务建议
|
||||||
- 周期拉取 Git 并检查是否有新配置
|
|
||||||
- `UploadDevPackageJob`
|
|
||||||
- 将待同步配置上传到 FTP
|
|
||||||
- `ConsumeProdPackageJob`
|
|
||||||
- 下载生产回传包并写入 Git
|
|
||||||
- `AckScanJob`
|
|
||||||
- 扫描生产侧 ack 并更新任务状态
|
|
||||||
|
|
||||||
### 17.2 生产侧任务
|
新架构下推荐保留两类核心任务：
|
||||||
|
|
||||||
- `ConsumeDevPackageJob`
|
### 16.1 `GitToProdSyncJob`
|
||||||
- 下载开发侧同步包并调用生产 `push`
|
|
||||||
- `PullProdConfigJob`
|
|
||||||
- 定时调用生产 `pull` 接口
|
|
||||||
- `UploadProdPackageJob`
|
|
||||||
- 将拉取结果上传到 FTP
|
|
||||||
- `AckScanJob`
|
|
||||||
- 扫描开发侧 ack 并更新任务状态
|
|
||||||
|
|
||||||
## 18. 一期 MVP 建议
|
职责：
|
||||||
|
|
||||||
建议按最小可交付版本分阶段实施。
|
- 拉取 `config-dev-main`
|
||||||
|
- 判断是否有新 commit
|
||||||
|
- 调用生产 `push` 接口
|
||||||
|
|
||||||
### 阶段 1：打通主链路
|
### 16.2 `ProdToGitSnapshotJob`
|
||||||
|
|
||||||
- 建立 Spring Boot 工程
|
职责：
|
||||||
- 集成 H2、JGit、FTP
|
|
||||||
- 实现开发到生产的全量包同步
|
- 调用生产 `pull` 接口
|
||||||
|
- 判断是否有新快照
|
||||||
|
- 提交到 `config-prod-snapshot`
|
||||||
|
|
||||||
|
可选任务：
|
||||||
|
|
||||||
|
- `RetryFailedTaskJob`
|
||||||
|
- `HealthCheckJob`
|
||||||
|
|
||||||
|
## 17. 一期 MVP 建议
|
||||||
|
|
||||||
|
建议重新按最小可交付版本收敛：
|
||||||
|
|
||||||
|
### 阶段 1：打通 Git -> 生产
|
||||||
|
|
||||||
|
- 生产环境直连开发 Git
|
||||||
|
- 实现 `config-dev-main` 拉取
|
||||||
- 实现生产 `push` 接口调用
|
- 实现生产 `push` 接口调用
|
||||||
|
- 落库记录同步状态
|
||||||
|
|
||||||
### 阶段 2：打通回传链路
|
### 阶段 2：打通 生产 -> Git
|
||||||
|
|
||||||
- 接入生产 `pull` 接口
|
- 接入生产 `pull` 接口
|
||||||
- 实现生产到开发的 FTP 回传
|
- 回写 `config-prod-snapshot`
|
||||||
- 实现开发侧写入 Git 并推送
|
- 实现 commit + push
|
||||||
|
|
||||||
### 阶段 3：增强稳定性
|
### 阶段 3：增强稳定性
|
||||||
|
|
||||||
- 增加重试
|
- 补充重试
|
||||||
- 增加 ack 机制
|
- 补充管理接口
|
||||||
- 增加失败归档
|
- 补充告警与审计日志
|
||||||
- 增加告警与审计日志
|
|
||||||
|
|
||||||
## 19. 风险与注意事项
|
## 18. 风险与注意事项
|
||||||
|
|
||||||
### 19.1 最大风险：双向同步闭环
|
### 18.1 最大风险：Git 写权限不足
|
||||||
|
|
||||||
如果生产回传配置写入开发主分支，再被开发侧识别为“新配置”，会再次推送到生产，形成无限循环。
|
如果生产环境对开发 Git 没有推送权限，则“生产 -> Git”链路无法完成。
|
||||||
|
|
||||||
|
解决方案：
|
||||||
|
|
||||||
|
- 申请机器人账号
|
||||||
|
- 或将“生产回写 Git”改成调用开发侧服务接口
|
||||||
|
|
||||||
|
### 18.2 最大风险：双向同步闭环
|
||||||
|
|
||||||
|
如果生产回写到了开发主分支，会再次触发下发。
|
||||||
|
|
||||||
规避措施：
|
规避措施：
|
||||||
|
|
||||||
- 使用独立镜像分支
|
- 使用独立快照分支
|
||||||
- 识别机器人提交
|
- 不扫描快照分支
|
||||||
- 使用幂等键
|
- 使用幂等键和机器人提交标记
|
||||||
|
|
||||||
### 19.2 配置冲突风险
|
### 18.3 最大风险：生产直连开发 Git 的安全边界
|
||||||
|
|
||||||
如果开发和生产都会修改同一份配置，且要求双向合并，则不能简单用文件覆盖方式处理。
|
需要明确：
|
||||||
|
|
||||||
当前建议：
|
- 网络访问是否合规
|
||||||
|
- Git 账号权限是否受控
|
||||||
|
- Token 或 SSH Key 是否可轮换
|
||||||
|
|
||||||
- 将生产回传定义为“镜像/审计”
|
## 19. 结论
|
||||||
- 不直接回写开发主配置分支
|
|
||||||
|
|
||||||
### 19.3 FTP 能力限制
|
在“生产环境可以直接访问开发 Git，FTP 不再需要”的前提下，推荐将旧方案调整为：
|
||||||
|
|
||||||
如果 FTP 不支持原子重命名、目录权限受限或稳定性较差，需要额外做兼容与重试。
|
- **生产环境单点部署**
|
||||||
|
- **Git 直连**
|
||||||
|
- **保留生产 `push/pull` 接口**
|
||||||
|
- **保留 H2 状态库**
|
||||||
|
|
||||||
## 20. 结论
|
这是比原来 FTP 中转更合适的方案，原因是：
|
||||||
|
|
||||||
在当前网络条件下，推荐采用“**开发代理 + 生产代理 + FTP 中转 + H2 状态库**”的双端部署方案。
|
- 架构更简单
|
||||||
|
- 故障点更少
|
||||||
|
- 链路更短
|
||||||
|
- 运维成本更低
|
||||||
|
|
||||||
该方案具备以下特点：
|
## 20. 下一步建议
|
||||||
|
|
||||||
- 不依赖开发与生产直接互通
|
下一步建议按下面顺序推进：
|
||||||
- 满足开发到生产、生产到开发的双向同步需求
|
|
||||||
- 支持状态记录、失败重试、幂等控制和审计追踪
|
|
||||||
- 适合使用 `Java 1.8 + Spring Boot 2.7.18` 快速落地
|
|
||||||
|
|
||||||
## 21. 后续可继续细化内容
|
1. 先确认生产环境对开发 Git 是否具备推送权限
|
||||||
|
2. 确认生产 `push/pull` 接口最终协议
|
||||||
后续可以基于本方案继续输出：
|
3. 在文件系统允许时物理删除退役占位类
|
||||||
|
4. 将工程命名中残留的 `ftp` 语义继续清理
|
||||||
- `application.properties` 配置项设计
|
5. 补充新的 `application-prod-agent.properties` 配置说明
|
||||||
- H2 建表 SQL
|
|
||||||
- 核心类图与接口设计
|
|
||||||
- 各定时任务的时序与状态流转
|
|
||||||
- Spring Boot 工程骨架
|
|
||||||
|
|||||||
@ -1,66 +1,77 @@
|
|||||||
# FTP 同步工具详细设计
|
# Git 直连架构详细设计
|
||||||
|
|
||||||
## 1. 文档说明
|
## 1. 文档说明
|
||||||
|
|
||||||
本文档是对总体方案的继续细化，重点补充以下内容：
|
本文档用于承接主方案文档，说明在“FTP 下线、生产环境可直接访问开发 Git”条件下的详细设计。
|
||||||
|
|
||||||
- `application.properties` 配置方案
|
当前目标是把系统收敛为：
|
||||||
- H2 表结构与初始化方式
|
|
||||||
- Spring Boot 2.7.18 工程骨架
|
|
||||||
- 核心类职责划分
|
|
||||||
- 启动方式与后续待实现事项
|
|
||||||
|
|
||||||
## 2. 配置文件策略
|
- 单 `prod-agent`
|
||||||
|
- Git 直连
|
||||||
|
- 生产 `push/pull` 接口驱动
|
||||||
|
- H2 本地状态控制
|
||||||
|
|
||||||
本项目采用 `properties` 配置文件，不使用 `yml`。
|
## 2. 架构变化摘要
|
||||||
|
|
||||||
推荐目录如下：
|
旧架构：
|
||||||
|
|
||||||
|
```text
|
||||||
|
开发环境 <-> FTP <-> 生产环境
|
||||||
|
```
|
||||||
|
|
||||||
|
新架构：
|
||||||
|
|
||||||
|
```text
|
||||||
|
开发 Git <-> 生产环境 Sync-Agent-Prod <-> 生产 push/pull 接口
|
||||||
|
```
|
||||||
|
|
||||||
|
关键变化：
|
||||||
|
|
||||||
|
- FTP 中转取消
|
||||||
|
- `dev-agent` 不再是必需部署节点
|
||||||
|
- 生产环境成为唯一同步执行节点
|
||||||
|
- ACK 文件机制不再作为主流程依赖
|
||||||
|
|
||||||
|
## 3. 当前推荐部署
|
||||||
|
|
||||||
|
推荐只部署：
|
||||||
|
|
||||||
|
- `prod-agent`
|
||||||
|
|
||||||
|
运行要求：
|
||||||
|
|
||||||
|
- 能访问开发 Git
|
||||||
|
- 能 push 指定 Git 分支
|
||||||
|
- 能访问生产 `push/pull` 接口
|
||||||
|
- 能写本地 H2 文件数据库
|
||||||
|
|
||||||
|
## 4. 配置文件策略
|
||||||
|
|
||||||
|
当前配置文件仍使用 `properties`：
|
||||||
|
|
||||||
```text
|
```text
|
||||||
src/main/resources/
|
src/main/resources/
|
||||||
|- application.properties
|
|- application.properties
|
||||||
|- application-dev-agent.properties
|
|
||||||
|- application-prod-agent.properties
|
|- application-prod-agent.properties
|
||||||
|- schema.sql
|
|- schema.sql
|
||||||
```
|
```
|
||||||
|
|
||||||
配置分工如下：
|
说明：
|
||||||
|
|
||||||
- `application.properties`
|
- `application-dev-agent.properties` 现阶段仅保留为退役标记文件
|
||||||
- 放公共配置
|
- `application-prod-agent.properties` 已开始收敛到新架构
|
||||||
- 包括数据源、H2、通用路径、FTP 默认项、Git 默认项、生产接口默认项
|
|
||||||
- `application-dev-agent.properties`
|
|
||||||
- 放开发环境代理专属配置
|
|
||||||
- 包括开发侧定时任务表达式、开发侧 FTP 账号、Git 仓库分支
|
|
||||||
- `application-prod-agent.properties`
|
|
||||||
- 放生产环境代理专属配置
|
|
||||||
- 包括生产侧定时任务表达式、生产侧 FTP 账号、生产接口地址与认证
|
|
||||||
|
|
||||||
## 3. 当前配置项设计
|
## 5. 当前配置口径
|
||||||
|
|
||||||
### 3.1 公共配置
|
### 5.1 仍然保留的核心配置
|
||||||
|
|
||||||
已落地文件：
|
公共配置：
|
||||||
|
|
||||||
- [application.properties](e:/AIcoding/FtpTool/src/main/resources/application.properties)
|
|
||||||
|
|
||||||
核心配置分组如下：
|
|
||||||
|
|
||||||
### `spring.*`
|
|
||||||
|
|
||||||
- `spring.application.name`
|
|
||||||
- `spring.datasource.*`
|
- `spring.datasource.*`
|
||||||
- `spring.jpa.*`
|
- `spring.jpa.*`
|
||||||
- `spring.sql.init.*`
|
- `spring.sql.init.*`
|
||||||
- `spring.h2.console.*`
|
|
||||||
|
|
||||||
用途：
|
同步配置：
|
||||||
|
|
||||||
- 启动 Spring Boot
|
|
||||||
- 使用 H2 文件数据库
|
|
||||||
- 通过 `schema.sql` 初始化表结构
|
|
||||||
|
|
||||||
### `sync.*`
|
|
||||||
|
|
||||||
- `sync.node-id`
|
- `sync.node-id`
|
||||||
- `sync.role`
|
- `sync.role`
|
||||||
@ -69,407 +80,196 @@ src/main/resources/
|
|||||||
- `sync.dev-to-prod-staging-dir`
|
- `sync.dev-to-prod-staging-dir`
|
||||||
- `sync.prod-to-dev-staging-dir`
|
- `sync.prod-to-dev-staging-dir`
|
||||||
- `sync.max-retry-count`
|
- `sync.max-retry-count`
|
||||||
- `sync.ack-scan-batch-size`
|
|
||||||
|
|
||||||
用途：
|
Git 配置：
|
||||||
|
|
||||||
- 标识当前节点身份
|
|
||||||
- 控制工作目录和临时目录
|
|
||||||
- 控制同步重试与 ack 扫描参数
|
|
||||||
|
|
||||||
### `ftp.*`
|
|
||||||
|
|
||||||
- `ftp.host`
|
|
||||||
- `ftp.port`
|
|
||||||
- `ftp.username`
|
|
||||||
- `ftp.password`
|
|
||||||
- `ftp.passive-mode`
|
|
||||||
- `ftp.base-dir`
|
|
||||||
- `ftp.connect-timeout-ms`
|
|
||||||
- `ftp.data-timeout-ms`
|
|
||||||
- `ftp.buffer-size`
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 定义 FTP 连接参数
|
|
||||||
- 定义远端根目录和超时策略
|
|
||||||
|
|
||||||
### `git.repo.*`
|
|
||||||
|
|
||||||
- `git.repo.local-path`
|
|
||||||
- `git.repo.remote-uri`
|
- `git.repo.remote-uri`
|
||||||
- `git.repo.username`
|
- `git.repo.username`
|
||||||
- `git.repo.password`
|
- `git.repo.password`
|
||||||
- `git.repo.scan-branch`
|
- `git.repo.scan-branch`
|
||||||
- `git.repo.snapshot-branch`
|
- `git.repo.snapshot-branch`
|
||||||
- `git.repo.commit-author-name`
|
|
||||||
- `git.repo.commit-author-email`
|
|
||||||
- `git.repo.commit-message-prefix`
|
- `git.repo.commit-message-prefix`
|
||||||
- `git.repo.pull-rebase`
|
|
||||||
|
|
||||||
用途：
|
生产接口配置：
|
||||||
|
|
||||||
- 定义开发侧 Git 拉取与提交行为
|
|
||||||
- 指定开发主分支和生产镜像分支
|
|
||||||
|
|
||||||
### `prod.api.*`
|
|
||||||
|
|
||||||
- `prod.api.base-url`
|
- `prod.api.base-url`
|
||||||
- `prod.api.push-path`
|
- `prod.api.push-path`
|
||||||
- `prod.api.pull-path`
|
- `prod.api.pull-path`
|
||||||
- `prod.api.token`
|
- `prod.api.token`
|
||||||
- `prod.api.connect-timeout-ms`
|
|
||||||
- `prod.api.read-timeout-ms`
|
|
||||||
|
|
||||||
用途：
|
### 5.2 新的生产侧调度配置
|
||||||
|
|
||||||
- 定义生产侧 `push/pull` 接口的连接方式
|
当前已调整为：
|
||||||
|
|
||||||
## 4. Profile 设计
|
- `sync.jobs.prod-git-to-prod.cron`
|
||||||
|
- `sync.jobs.prod-to-git.cron`
|
||||||
|
|
||||||
### 4.1 开发代理 Profile
|
对应文件：
|
||||||
|
|
||||||
已落地文件：
|
|
||||||
|
|
||||||
- [application-dev-agent.properties](e:/AIcoding/FtpTool/src/main/resources/application-dev-agent.properties)
|
|
||||||
|
|
||||||
主要内容：
|
|
||||||
|
|
||||||
- `spring.config.activate.on-profile=dev-agent`
|
|
||||||
- 开发侧端口
|
|
||||||
- 开发侧三类任务 cron
|
|
||||||
- 开发侧 FTP 账号示例
|
|
||||||
- Git 分支覆盖项
|
|
||||||
|
|
||||||
当前定时任务：
|
|
||||||
|
|
||||||
- `sync.jobs.dev-git-scan.cron`
|
|
||||||
- `sync.jobs.dev-consume-prod-package.cron`
|
|
||||||
- `sync.jobs.dev-ack-scan.cron`
|
|
||||||
|
|
||||||
### 4.2 生产代理 Profile
|
|
||||||
|
|
||||||
已落地文件：
|
|
||||||
|
|
||||||
- [application-prod-agent.properties](e:/AIcoding/FtpTool/src/main/resources/application-prod-agent.properties)
|
- [application-prod-agent.properties](e:/AIcoding/FtpTool/src/main/resources/application-prod-agent.properties)
|
||||||
|
|
||||||
主要内容：
|
### 5.3 遗留 FTP 配置
|
||||||
|
|
||||||
- `spring.config.activate.on-profile=prod-agent`
|
当前 `application.properties` 中已经移除了 `ftp.*` 相关公共配置。
|
||||||
- 生产侧端口
|
|
||||||
- 生产侧三类任务 cron
|
|
||||||
- 生产侧 FTP 账号示例
|
|
||||||
- 生产接口地址和 token 示例
|
|
||||||
|
|
||||||
当前定时任务：
|
当前状态：
|
||||||
|
|
||||||
- `sync.jobs.prod-consume-dev-package.cron`
|
- `FtpClientService` 已降为占位类
|
||||||
- `sync.jobs.prod-pull-config.cron`
|
- FTP 调度主路径已退出运行面
|
||||||
- `sync.jobs.prod-ack-scan.cron`
|
- 旧 FTP 类仍保留在源码树中，但不再作为正式能力
|
||||||
|
|
||||||
## 5. H2 设计
|
## 6. H2 状态设计
|
||||||
|
|
||||||
已落地文件：
|
当前主表保留为：
|
||||||
|
|
||||||
|
- `sync_checkpoint`
|
||||||
|
- `sync_task`
|
||||||
|
|
||||||
|
对应脚本：
|
||||||
|
|
||||||
- [schema.sql](e:/AIcoding/FtpTool/src/main/resources/schema.sql)
|
- [schema.sql](e:/AIcoding/FtpTool/src/main/resources/schema.sql)
|
||||||
|
|
||||||
### 5.1 初始化方式
|
### 6.1 保留原因
|
||||||
|
|
||||||
通过以下配置自动初始化：
|
即使 FTP 已下线，状态库仍然必须保留，用于：
|
||||||
|
|
||||||
```properties
|
- 幂等控制
|
||||||
spring.sql.init.mode=always
|
- 检查点管理
|
||||||
spring.sql.init.schema-locations=classpath:schema.sql
|
- 失败重试
|
||||||
spring.jpa.hibernate.ddl-auto=none
|
- 审计追踪
|
||||||
```
|
|
||||||
|
### 6.2 当前建议
|
||||||
|
|
||||||
|
- `sync_checkpoint` 和 `sync_task` 继续作为主表
|
||||||
|
- `sync_ack` 已退出当前主 schema
|
||||||
|
|
||||||
|
## 7. 当前代码结构与新架构对应关系
|
||||||
|
|
||||||
|
### 7.1 已经可继续复用的核心类
|
||||||
|
|
||||||
|
- [GitClientService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/GitClientService.java)
|
||||||
|
- [ProdConfigApiService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/ProdConfigApiService.java)
|
||||||
|
- [PackageService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/PackageService.java)
|
||||||
|
- [SyncTaskService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/SyncTaskService.java)
|
||||||
|
- [CheckpointService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/CheckpointService.java)
|
||||||
|
|
||||||
|
### 7.2 当前生产侧主协调器
|
||||||
|
|
||||||
|
- [ProdSyncCoordinator.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/orchestrator/ProdSyncCoordinator.java)
|
||||||
|
|
||||||
|
当前生产侧主流程已经切到新架构：
|
||||||
|
|
||||||
|
- `syncLatestGitToProd()`：Git -> 生产 `push`
|
||||||
|
- `syncProdSnapshotToGit()`：生产 `pull` -> Git snapshot 分支
|
||||||
|
|
||||||
|
### 7.3 当前生产侧正式调度类
|
||||||
|
|
||||||
|
- [GitToProdSyncJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/GitToProdSyncJob.java)
|
||||||
|
- [ProdToGitSnapshotJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdToGitSnapshotJob.java)
|
||||||
|
|
||||||
说明：
|
说明：
|
||||||
|
|
||||||
- 表结构由手工 SQL 控制
|
- 这两个类是当前推荐使用的正式调度入口
|
||||||
- 不依赖 Hibernate 自动建表
|
- 已分别对应 `Git -> PROD` 和 `PROD -> Git`
|
||||||
- 更适合后续环境迁移和版本管理
|
|
||||||
|
|
||||||
### 5.2 已定义表
|
### 7.4 当前遗留占位类
|
||||||
|
|
||||||
#### `sync_checkpoint`
|
- [ProdConsumeDevPackageJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdConsumeDevPackageJob.java)
|
||||||
|
- [ProdPullConfigJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdPullConfigJob.java)
|
||||||
用途：
|
|
||||||
|
|
||||||
- 保存每个同步方向最后一次成功版本
|
|
||||||
|
|
||||||
关键字段：
|
|
||||||
|
|
||||||
- `direction`
|
|
||||||
- `last_success_version`
|
|
||||||
- `last_success_hash`
|
|
||||||
- `updated_at`
|
|
||||||
|
|
||||||
#### `sync_task`
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 保存每次同步任务实例
|
|
||||||
|
|
||||||
关键字段：
|
|
||||||
|
|
||||||
- `trace_id`
|
|
||||||
- `direction`
|
|
||||||
- `source_version`
|
|
||||||
- `content_hash`
|
|
||||||
- `package_name`
|
|
||||||
- `status`
|
|
||||||
- `retry_count`
|
|
||||||
- `error_msg`
|
|
||||||
|
|
||||||
关键约束：
|
|
||||||
|
|
||||||
- `trace_id` 唯一
|
|
||||||
- `direction + source_version + content_hash` 唯一
|
|
||||||
|
|
||||||
这组唯一键就是当前骨架里默认采用的幂等键。
|
|
||||||
|
|
||||||
#### `sync_ack`
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 保存跨端 ack 回执
|
|
||||||
|
|
||||||
关键字段：
|
|
||||||
|
|
||||||
- `trace_id`
|
|
||||||
- `ack_side`
|
|
||||||
- `ack_status`
|
|
||||||
- `ack_time`
|
|
||||||
- `remark`
|
|
||||||
|
|
||||||
## 6. 工程骨架
|
|
||||||
|
|
||||||
当前已经在仓库中生成了一套最小 Spring Boot 骨架。
|
|
||||||
|
|
||||||
### 6.1 构建文件
|
|
||||||
|
|
||||||
- [pom.xml](e:/AIcoding/FtpTool/pom.xml)
|
|
||||||
|
|
||||||
已引入的核心依赖：
|
|
||||||
|
|
||||||
- `spring-boot-starter`
|
|
||||||
- `spring-boot-starter-web`
|
|
||||||
- `spring-boot-starter-data-jpa`
|
|
||||||
- `spring-boot-starter-actuator`
|
|
||||||
- `spring-retry`
|
|
||||||
- `commons-net`
|
|
||||||
- `org.eclipse.jgit`
|
|
||||||
- `h2`
|
|
||||||
|
|
||||||
### 6.2 启动类
|
|
||||||
|
|
||||||
- [FtpSyncToolApplication.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/FtpSyncToolApplication.java)
|
|
||||||
|
|
||||||
作用：
|
|
||||||
|
|
||||||
- 启用 Spring Boot
|
|
||||||
- 启用定时任务
|
|
||||||
- 启用重试机制
|
|
||||||
- 注册配置属性类
|
|
||||||
|
|
||||||
### 6.3 配置属性类
|
|
||||||
|
|
||||||
- [SyncProperties.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/config/SyncProperties.java)
|
|
||||||
- [FtpProperties.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/config/FtpProperties.java)
|
|
||||||
- [GitRepoProperties.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/config/GitRepoProperties.java)
|
|
||||||
- [ProdApiProperties.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/config/ProdApiProperties.java)
|
|
||||||
|
|
||||||
作用：
|
|
||||||
|
|
||||||
- 将 `properties` 配置映射为强类型对象
|
|
||||||
- 避免业务代码直接散落读取字符串 key
|
|
||||||
|
|
||||||
### 6.4 基础配置
|
|
||||||
|
|
||||||
- [AppConfig.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/config/AppConfig.java)
|
|
||||||
|
|
||||||
当前提供：
|
|
||||||
|
|
||||||
- `RestTemplate` Bean
|
|
||||||
- 读取生产接口超时参数
|
|
||||||
|
|
||||||
## 7. 领域模型与仓储
|
|
||||||
|
|
||||||
### 7.1 枚举
|
|
||||||
|
|
||||||
- [SyncDirection.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/model/SyncDirection.java)
|
|
||||||
- [SyncRole.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/model/SyncRole.java)
|
|
||||||
- [SyncStatus.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/model/SyncStatus.java)
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 统一同步方向、角色和状态定义
|
|
||||||
|
|
||||||
### 7.2 实体
|
|
||||||
|
|
||||||
- [SyncTask.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/entity/SyncTask.java)
|
|
||||||
- [SyncCheckpoint.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/entity/SyncCheckpoint.java)
|
|
||||||
- [SyncAck.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/entity/SyncAck.java)
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 对应 H2 三张核心业务表
|
|
||||||
- 内置了基础时间戳维护逻辑
|
|
||||||
|
|
||||||
### 7.3 Repository
|
|
||||||
|
|
||||||
- [SyncTaskRepository.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/repository/SyncTaskRepository.java)
|
|
||||||
- [SyncCheckpointRepository.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/repository/SyncCheckpointRepository.java)
|
|
||||||
- [SyncAckRepository.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/repository/SyncAckRepository.java)
|
|
||||||
|
|
||||||
用途：
|
|
||||||
|
|
||||||
- 提供基础持久化能力
|
|
||||||
- 已包含按幂等键和 `traceId` 查询的方法
|
|
||||||
|
|
||||||
## 8. 当前服务层设计
|
|
||||||
|
|
||||||
### 8.1 已实现基础服务
|
|
||||||
|
|
||||||
- [SyncTaskService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/SyncTaskService.java)
|
|
||||||
- [CheckpointService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/CheckpointService.java)
|
|
||||||
- [AckService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/AckService.java)
|
|
||||||
|
|
||||||
当前能力：
|
|
||||||
|
|
||||||
- 创建或加载幂等任务
|
|
||||||
- 更新任务状态
|
|
||||||
- 增加重试次数
|
|
||||||
- 更新检查点
|
|
||||||
- 记录 ack 回执
|
|
||||||
|
|
||||||
### 8.2 当前已实现的业务服务
|
|
||||||
|
|
||||||
本轮代码已经补上以下真实能力：
|
|
||||||
|
|
||||||
- FTP 上传、下载、列目录、删除、移动、原子重命名上传
|
|
||||||
- Git clone / pull / checkout / commit / push
|
|
||||||
- zip 打包与解包
|
|
||||||
- manifest 生成与内容哈希校验
|
|
||||||
- 生产 `push` / `pull` 接口调用骨架
|
|
||||||
|
|
||||||
当前对应实现文件包括：
|
|
||||||
|
|
||||||
- [FtpClientService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/FtpClientService.java)
|
|
||||||
- [GitClientService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/GitClientService.java)
|
|
||||||
- [PackageService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/PackageService.java)
|
|
||||||
- [ProdConfigApiService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/ProdConfigApiService.java)
|
|
||||||
- [SyncMetadataService.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/service/SyncMetadataService.java)
|
|
||||||
|
|
||||||
## 9. 当前调度层设计
|
|
||||||
|
|
||||||
### 9.1 开发侧调度
|
|
||||||
|
|
||||||
- [DevSyncCoordinator.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/orchestrator/DevSyncCoordinator.java)
|
|
||||||
- [DevGitScanJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevGitScanJob.java)
|
- [DevGitScanJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevGitScanJob.java)
|
||||||
- [DevConsumeProdPackageJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevConsumeProdPackageJob.java)
|
- [DevConsumeProdPackageJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevConsumeProdPackageJob.java)
|
||||||
- [DevAckScanJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevAckScanJob.java)
|
- [DevAckScanJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/DevAckScanJob.java)
|
||||||
|
|
||||||
当前状态：
|
说明：
|
||||||
|
|
||||||
- 已按 `dev-agent` profile 进行隔离
|
- 这些类目前保留为兼容占位
|
||||||
- 已绑定 cron 表达式
|
- 已不再作为正式运行入口
|
||||||
- 已串联 Git 拉取、包构建、FTP 上传、FTP 消费、Git 提交和 ACK 上传
|
- 由于当前环境删除权限受限，暂时保留为空占位实现
|
||||||
|
|
||||||
### 9.2 生产侧调度
|
### 7.5 当前遗留代码
|
||||||
|
|
||||||
- [ProdSyncCoordinator.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/orchestrator/ProdSyncCoordinator.java)
|
以下内容仍然存在于代码库，但属于旧架构遗留：
|
||||||
- [ProdConsumeDevPackageJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdConsumeDevPackageJob.java)
|
|
||||||
- [ProdPullConfigJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdPullConfigJob.java)
|
- `dev-agent` 相关 job/coordinator
|
||||||
- [ProdAckScanJob.java](e:/AIcoding/FtpTool/src/main/java/com/ftptool/sync/job/ProdAckScanJob.java)
|
- `FtpClientService`
|
||||||
|
- ACK 文件相关模型和服务
|
||||||
|
|
||||||
|
这些不是当前推荐运行路径。
|
||||||
|
|
||||||
|
## 8. 新架构下的两条任务流
|
||||||
|
|
||||||
|
### 8.1 Git -> 生产
|
||||||
|
|
||||||
|
当前推荐实现步骤：
|
||||||
|
|
||||||
|
1. 拉取 `config-dev-main`
|
||||||
|
2. 获取最新 commit 作为 `sourceVersion`
|
||||||
|
3. 导出工作树快照
|
||||||
|
4. 计算内容哈希
|
||||||
|
5. 生成标准 zip 包
|
||||||
|
6. 调用生产 `push` 接口
|
||||||
|
7. 更新 `sync_task` 和 `sync_checkpoint`
|
||||||
|
|
||||||
|
### 8.2 生产 -> Git
|
||||||
|
|
||||||
|
当前推荐实现步骤：
|
||||||
|
|
||||||
|
1. 调用生产 `pull` 接口
|
||||||
|
2. 保存返回配置到本地 staging 目录
|
||||||
|
3. 计算哈希和版本号
|
||||||
|
4. 写入 `config-prod-snapshot`
|
||||||
|
5. commit + push
|
||||||
|
6. 更新 `sync_task` 和 `sync_checkpoint`
|
||||||
|
|
||||||
|
## 9. 当前接口假设
|
||||||
|
|
||||||
|
当前生产接口仍按以下假设实现：
|
||||||
|
|
||||||
|
- `push` 使用 `POST + multipart/form-data`
|
||||||
|
- `pull` 使用 `GET`
|
||||||
|
- `pull` 返回原始 JSON 字节流
|
||||||
|
- 版本号优先取 `X-Config-Version`，其次 `ETag`
|
||||||
|
|
||||||
|
详细协议文档见：
|
||||||
|
|
||||||
|
- [prod-api-v1.md](e:/AIcoding/FtpTool/docs/prod-api-v1.md)
|
||||||
|
|
||||||
|
## 10. 当前主要风险
|
||||||
|
|
||||||
|
### 10.1 Git 写权限
|
||||||
|
|
||||||
|
如果生产环境对 Git 没有 push 权限，则“生产 -> Git”链路无法完成。
|
||||||
|
|
||||||
|
### 10.2 旧代码残留
|
||||||
|
|
||||||
|
当前主运行面已经切到 Git 直连，但源码树里仍保留少量退役占位类。
|
||||||
|
|
||||||
当前状态：
|
当前状态：
|
||||||
|
|
||||||
- 已按 `prod-agent` profile 进行隔离
|
- 文件名和类名仍可能误导维护者
|
||||||
- 已绑定 cron 表达式
|
- 极少量退役文件仍保留在源码树中
|
||||||
- 已串联 FTP 消费、生产 `push` 接口调用、生产 `pull` 接口调用、包构建和 ACK 上传
|
|
||||||
|
|
||||||
## 9.3 当前接口假设
|
### 10.3 双向同步闭环
|
||||||
|
|
||||||
由于你还没有给出生产 `push/pull` 接口的正式协议，本轮实现采用以下默认假设：
|
如果误把生产回写分支也作为下发分支扫描，会造成循环同步。
|
||||||
|
|
||||||
- 生产 `push` 接口使用 `multipart/form-data`
|
## 11. 推荐的下一轮改造
|
||||||
- 上传字段包含 `file`、`traceId`、`direction`、`sourceVersion`、`contentHash`
|
|
||||||
- 生产 `pull` 接口使用 `HTTP GET`
|
|
||||||
- `pull` 返回原始字节内容，当前默认保存为 `prod-config.json`
|
|
||||||
- 如果响应头里存在 `X-Config-Version` 或 `ETag`，优先用它作为来源版本号
|
|
||||||
|
|
||||||
后续如果你提供正式接口文档，再把这部分对齐为最终协议即可。
|
建议按下面顺序继续：
|
||||||
|
|
||||||
## 10. 当前目录结构
|
1. 删除或隔离 `dev-agent` 运行路径
|
||||||
|
2. 在文件系统允许时物理删除退役占位类
|
||||||
|
3. 统一清理残留的 `ftp` 命名
|
||||||
|
4. 补充管理接口和健康检查接口
|
||||||
|
5. 增加集成测试
|
||||||
|
|
||||||
```text
|
## 12. 结论
|
||||||
FtpTool
|
|
||||||
|- docs
|
|
||||||
|- pom.xml
|
|
||||||
|- src
|
|
||||||
|- main
|
|
||||||
|- java/com/ftptool/sync
|
|
||||||
| |- config
|
|
||||||
| |- entity
|
|
||||||
| |- job
|
|
||||||
| |- model
|
|
||||||
| |- orchestrator
|
|
||||||
| |- repository
|
|
||||||
| |- service
|
|
||||||
|- resources
|
|
||||||
|- application.properties
|
|
||||||
|- application-dev-agent.properties
|
|
||||||
|- application-prod-agent.properties
|
|
||||||
|- schema.sql
|
|
||||||
```
|
|
||||||
|
|
||||||
## 11. 启动方式
|
当前系统已经从“FTP 中转”开始转向“Git 直连”。
|
||||||
|
|
||||||
### 11.1 启动开发代理
|
现阶段最重要的不是继续增强旧链路，而是彻底收敛到：
|
||||||
|
|
||||||
```bash
|
- 单 `prod-agent`
|
||||||
mvn spring-boot:run -Dspring-boot.run.profiles=dev-agent
|
- 两个核心任务
|
||||||
```
|
- 一个 Git 仓库入口
|
||||||
|
- 一组生产 `push/pull` 接口
|
||||||
### 11.2 启动生产代理
|
|
||||||
|
|
||||||
```bash
|
|
||||||
mvn spring-boot:run -Dspring-boot.run.profiles=prod-agent
|
|
||||||
```
|
|
||||||
|
|
||||||
也可以打包后通过 JVM 参数指定：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
java -jar ftp-sync-tool.jar --spring.profiles.active=dev-agent
|
|
||||||
java -jar ftp-sync-tool.jar --spring.profiles.active=prod-agent
|
|
||||||
```
|
|
||||||
|
|
||||||
## 12. 下一步建议实现顺序
|
|
||||||
|
|
||||||
建议按以下顺序继续落代码：
|
|
||||||
|
|
||||||
1. 先实现 `FtpClientService`
|
|
||||||
2. 再实现 `GitClientService`
|
|
||||||
3. 再实现 `PackageService`
|
|
||||||
4. 再实现 `ProdConfigApiService`
|
|
||||||
5. 最后把 `Coordinator` 中的 TODO 串起来
|
|
||||||
|
|
||||||
## 13. 当前边界
|
|
||||||
|
|
||||||
当前骨架是“可扩展的项目起点”，不是完整业务实现，现阶段还缺：
|
|
||||||
|
|
||||||
- 真正的 FTP 交互
|
|
||||||
- 真正的 Git 操作
|
|
||||||
- 真正的生产接口调用
|
|
||||||
- 包文件读写与校验
|
|
||||||
- ack 文件协议
|
|
||||||
- 失败重试细节和告警
|
|
||||||
|
|
||||||
但好处是结构已经固定住了：
|
|
||||||
|
|
||||||
- 配置口径统一为 `properties`
|
|
||||||
- profile 隔离清晰
|
|
||||||
- H2 状态表已定义
|
|
||||||
- 调度入口已分开
|
|
||||||
- 任务、检查点、ack 的存储模型已落地
|
|
||||||
|
|||||||
247
docs/git-direct-sync-tool-design.md
Normal file
247
docs/git-direct-sync-tool-design.md
Normal file
@ -0,0 +1,247 @@
|
|||||||
|
# 基于 Git 直连的配置双向同步工具设计方案
|
||||||
|
|
||||||
|
## 1. 背景变化
|
||||||
|
|
||||||
|
旧方案的前提是：
|
||||||
|
|
||||||
|
- 开发环境和生产环境不能直接交换同步数据
|
||||||
|
- 需要通过 `开发环境 -> FTP -> 生产环境` 中转
|
||||||
|
|
||||||
|
现在条件已经变化为：
|
||||||
|
|
||||||
|
- FTP 不再使用
|
||||||
|
- 生产环境可以直接访问开发 Git 仓库
|
||||||
|
|
||||||
|
这意味着旧架构里的 FTP 中转层已经没有存在价值，应该直接删除。
|
||||||
|
|
||||||
|
## 2. 新架构结论
|
||||||
|
|
||||||
|
推荐把原来的“双端代理 + FTP 中转”收敛为：
|
||||||
|
|
||||||
|
- **单端代理 + Git 直连 + 本地状态库**
|
||||||
|
|
||||||
|
也就是只在生产环境部署一套同步服务：
|
||||||
|
|
||||||
|
- `Sync-Agent-Prod`
|
||||||
|
|
||||||
|
整体拓扑如下：
|
||||||
|
|
||||||
|
```text
|
||||||
|
开发 Git 仓库 <----> 生产环境 Sync-Agent-Prod <----> 生产系统 push/pull 接口
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. 为什么这样改
|
||||||
|
|
||||||
|
新架构比旧架构更合理，原因很直接：
|
||||||
|
|
||||||
|
- 去掉 FTP，中间链路减少一跳
|
||||||
|
- 去掉打包上传、轮询下载、ACK 回执等中间机制
|
||||||
|
- 部署节点从 2 个变成 1 个
|
||||||
|
- 故障点减少，排查成本更低
|
||||||
|
- 数据路径更直接，状态控制更简单
|
||||||
|
|
||||||
|
## 4. 新方案的核心思路
|
||||||
|
|
||||||
|
既然生产环境已经能访问开发 Git，那么同步动作都可以在生产环境完成。
|
||||||
|
|
||||||
|
生产环境部署的 `Sync-Agent-Prod` 同时承担两条链路：
|
||||||
|
|
||||||
|
### 4.1 开发 Git -> 生产
|
||||||
|
|
||||||
|
流程：
|
||||||
|
|
||||||
|
1. 定时拉取开发 Git 的配置分支
|
||||||
|
2. 获取最新 commit 版本
|
||||||
|
3. 判断该版本是否已同步
|
||||||
|
4. 如果未同步，则读取配置内容
|
||||||
|
5. 调用生产 `push` 接口导入配置
|
||||||
|
6. 成功后更新本地检查点
|
||||||
|
|
||||||
|
### 4.2 生产 -> 开发 Git
|
||||||
|
|
||||||
|
流程：
|
||||||
|
|
||||||
|
1. 定时调用生产 `pull` 接口
|
||||||
|
2. 获取当前生产配置快照
|
||||||
|
3. 计算内容哈希或版本号
|
||||||
|
4. 判断该快照是否已同步
|
||||||
|
5. 如果未同步，则写入 Git 快照分支
|
||||||
|
6. commit 并 push 到开发 Git
|
||||||
|
7. 成功后更新本地检查点
|
||||||
|
|
||||||
|
## 5. 部署建议
|
||||||
|
|
||||||
|
### 5.1 推荐部署方式
|
||||||
|
|
||||||
|
推荐只保留：
|
||||||
|
|
||||||
|
- `prod-agent`
|
||||||
|
|
||||||
|
不再需要：
|
||||||
|
|
||||||
|
- `dev-agent`
|
||||||
|
- FTP 服务
|
||||||
|
- FTP ACK、包中转、失败目录等机制
|
||||||
|
|
||||||
|
### 5.2 运行前提
|
||||||
|
|
||||||
|
生产环境需要同时满足：
|
||||||
|
|
||||||
|
- 能读取开发 Git
|
||||||
|
- 能向开发 Git 推送指定分支
|
||||||
|
- 能调用生产 `push/pull` 接口
|
||||||
|
- 能持久化 H2 文件数据库
|
||||||
|
|
||||||
|
这里有一个必须先确认的关键点：
|
||||||
|
|
||||||
|
- **生产环境对开发 Git 是否有写权限**
|
||||||
|
|
||||||
|
如果只有读权限，没有 push 权限，那么第二条链路“生产 -> 开发 Git”无法闭环。
|
||||||
|
|
||||||
|
## 6. Git 分支策略
|
||||||
|
|
||||||
|
这个设计必须保留，不然非常容易形成同步闭环。
|
||||||
|
|
||||||
|
建议继续使用两个分支：
|
||||||
|
|
||||||
|
- `config-dev-main`：开发主配置分支
|
||||||
|
- `config-prod-snapshot`：生产配置镜像分支
|
||||||
|
|
||||||
|
同步规则：
|
||||||
|
|
||||||
|
- `Git -> PROD` 只读取 `config-dev-main`
|
||||||
|
- `PROD -> Git` 只写入 `config-prod-snapshot`
|
||||||
|
|
||||||
|
这样做的好处：
|
||||||
|
|
||||||
|
- 避免生产回写内容再次触发生产下发
|
||||||
|
- 生产快照不污染开发主线
|
||||||
|
- 便于审计和回溯
|
||||||
|
|
||||||
|
## 7. 状态管理
|
||||||
|
|
||||||
|
虽然 FTP 没了，但本地状态库仍然必须保留。
|
||||||
|
|
||||||
|
建议继续保留：
|
||||||
|
|
||||||
|
- `sync_checkpoint`
|
||||||
|
- `sync_task`
|
||||||
|
|
||||||
|
`sync_ack` 在新架构下不再承担跨节点 ACK 作用，可以：
|
||||||
|
|
||||||
|
- 继续保留为接口调用结果日志表
|
||||||
|
- 或后续简化下线
|
||||||
|
|
||||||
|
## 8. 幂等设计
|
||||||
|
|
||||||
|
建议继续使用：
|
||||||
|
|
||||||
|
```text
|
||||||
|
direction + sourceVersion + contentHash
|
||||||
|
```
|
||||||
|
|
||||||
|
作为业务幂等键。
|
||||||
|
|
||||||
|
作用：
|
||||||
|
|
||||||
|
- 同一个开发版本不会重复推生产
|
||||||
|
- 同一个生产快照不会重复写 Git
|
||||||
|
|
||||||
|
## 9. 失败处理
|
||||||
|
|
||||||
|
建议自动重试以下场景：
|
||||||
|
|
||||||
|
- Git pull 失败
|
||||||
|
- Git push 失败
|
||||||
|
- 生产 `push` 接口失败
|
||||||
|
- 生产 `pull` 接口失败
|
||||||
|
|
||||||
|
建议策略：
|
||||||
|
|
||||||
|
- 最大重试次数：`3 ~ 5`
|
||||||
|
- 指数退避：`30s / 60s / 120s`
|
||||||
|
|
||||||
|
失败后：
|
||||||
|
|
||||||
|
- 更新 `sync_task` 状态
|
||||||
|
- 保留错误信息
|
||||||
|
- 不推进检查点
|
||||||
|
|
||||||
|
## 10. 安全建议
|
||||||
|
|
||||||
|
### 10.1 Git 访问
|
||||||
|
|
||||||
|
推荐使用：
|
||||||
|
|
||||||
|
- HTTPS + Token
|
||||||
|
|
||||||
|
或：
|
||||||
|
|
||||||
|
- SSH Deploy Key
|
||||||
|
|
||||||
|
### 10.2 权限控制
|
||||||
|
|
||||||
|
生产环境访问 Git 的账号建议最小权限化：
|
||||||
|
|
||||||
|
- 对 `config-dev-main` 至少有读权限
|
||||||
|
- 对 `config-prod-snapshot` 有写权限
|
||||||
|
|
||||||
|
更理想的做法：
|
||||||
|
|
||||||
|
- 使用专用机器人账号
|
||||||
|
- 对主分支做保护
|
||||||
|
- 机器人只写快照分支
|
||||||
|
|
||||||
|
## 11. 对现有代码的影响
|
||||||
|
|
||||||
|
这次需求变化对当前代码影响很大，结论如下：
|
||||||
|
|
||||||
|
### 11.1 可以继续保留的部分
|
||||||
|
|
||||||
|
- `GitClientService`
|
||||||
|
- `ProdConfigApiService`
|
||||||
|
- `SyncTaskService`
|
||||||
|
- `CheckpointService`
|
||||||
|
- H2 表结构
|
||||||
|
- 定时任务框架
|
||||||
|
|
||||||
|
### 11.2 应该逐步下线的部分
|
||||||
|
|
||||||
|
- `FtpClientService`
|
||||||
|
- FTP 目录配置
|
||||||
|
- 包上传/下载流程
|
||||||
|
- ACK 文件机制
|
||||||
|
- 双端部署假设
|
||||||
|
|
||||||
|
### 11.3 推荐重构方向
|
||||||
|
|
||||||
|
把系统收敛为：
|
||||||
|
|
||||||
|
- 一个 `prod-agent`
|
||||||
|
- 两个核心任务
|
||||||
|
|
||||||
|
即：
|
||||||
|
|
||||||
|
1. `GitToProdSyncJob`
|
||||||
|
2. `ProdToGitSnapshotJob`
|
||||||
|
|
||||||
|
## 12. 结论
|
||||||
|
|
||||||
|
现在最合理的做法不是“在旧 FTP 方案上修修补补”，而是直接把架构收敛成：
|
||||||
|
|
||||||
|
- **生产环境单点部署**
|
||||||
|
- **直连开发 Git**
|
||||||
|
- **继续调用生产 `push/pull` 接口**
|
||||||
|
- **保留 H2 做状态控制**
|
||||||
|
|
||||||
|
这是一次简化，不是一次退化。
|
||||||
|
|
||||||
|
## 13. 下一步建议
|
||||||
|
|
||||||
|
建议按这个顺序推进：
|
||||||
|
|
||||||
|
1. 先确认生产环境是否具备开发 Git 的 push 权限
|
||||||
|
2. 确认生产 `push/pull` 接口最终协议
|
||||||
|
3. 重写主设计文档和详细设计文档，去掉 FTP 相关内容
|
||||||
|
4. 收敛代码为单 `prod-agent`
|
||||||
|
5. 删除 FTP 相关配置和服务
|
||||||
8
pom.xml
8
pom.xml
@ -15,12 +15,11 @@
|
|||||||
<artifactId>ftp-sync-tool</artifactId>
|
<artifactId>ftp-sync-tool</artifactId>
|
||||||
<version>0.0.1-SNAPSHOT</version>
|
<version>0.0.1-SNAPSHOT</version>
|
||||||
<name>ftp-sync-tool</name>
|
<name>ftp-sync-tool</name>
|
||||||
<description>FTP relay based configuration sync tool</description>
|
<description>Git direct based configuration sync tool</description>
|
||||||
|
|
||||||
<properties>
|
<properties>
|
||||||
<java.version>1.8</java.version>
|
<java.version>1.8</java.version>
|
||||||
<jgit.version>5.13.3.202401111512-r</jgit.version>
|
<jgit.version>5.13.3.202401111512-r</jgit.version>
|
||||||
<commons-net.version>3.11.1</commons-net.version>
|
|
||||||
</properties>
|
</properties>
|
||||||
|
|
||||||
<dependencies>
|
<dependencies>
|
||||||
@ -48,11 +47,6 @@
|
|||||||
<groupId>org.springframework</groupId>
|
<groupId>org.springframework</groupId>
|
||||||
<artifactId>spring-aspects</artifactId>
|
<artifactId>spring-aspects</artifactId>
|
||||||
</dependency>
|
</dependency>
|
||||||
<dependency>
|
|
||||||
<groupId>commons-net</groupId>
|
|
||||||
<artifactId>commons-net</artifactId>
|
|
||||||
<version>${commons-net.version}</version>
|
|
||||||
</dependency>
|
|
||||||
<dependency>
|
<dependency>
|
||||||
<groupId>org.eclipse.jgit</groupId>
|
<groupId>org.eclipse.jgit</groupId>
|
||||||
<artifactId>org.eclipse.jgit</artifactId>
|
<artifactId>org.eclipse.jgit</artifactId>
|
||||||
|
|||||||
@ -1,6 +1,5 @@
|
|||||||
package com.ftptool.sync;
|
package com.ftptool.sync;
|
||||||
|
|
||||||
import com.ftptool.sync.config.FtpProperties;
|
|
||||||
import com.ftptool.sync.config.GitRepoProperties;
|
import com.ftptool.sync.config.GitRepoProperties;
|
||||||
import com.ftptool.sync.config.ProdApiProperties;
|
import com.ftptool.sync.config.ProdApiProperties;
|
||||||
import com.ftptool.sync.config.SyncProperties;
|
import com.ftptool.sync.config.SyncProperties;
|
||||||
@ -15,7 +14,6 @@ import org.springframework.scheduling.annotation.EnableScheduling;
|
|||||||
@SpringBootApplication
|
@SpringBootApplication
|
||||||
@EnableConfigurationProperties({
|
@EnableConfigurationProperties({
|
||||||
SyncProperties.class,
|
SyncProperties.class,
|
||||||
FtpProperties.class,
|
|
||||||
GitRepoProperties.class,
|
GitRepoProperties.class,
|
||||||
ProdApiProperties.class
|
ProdApiProperties.class
|
||||||
})
|
})
|
||||||
|
|||||||
@ -1,89 +1,5 @@
|
|||||||
package com.ftptool.sync.config;
|
package com.ftptool.sync.config;
|
||||||
|
|
||||||
import org.springframework.boot.context.properties.ConfigurationProperties;
|
@Deprecated
|
||||||
|
public final class FtpProperties {
|
||||||
@ConfigurationProperties(prefix = "ftp")
|
|
||||||
public class FtpProperties {
|
|
||||||
|
|
||||||
private String host;
|
|
||||||
private int port = 21;
|
|
||||||
private String username;
|
|
||||||
private String password;
|
|
||||||
private boolean passiveMode = true;
|
|
||||||
private String baseDir;
|
|
||||||
private int connectTimeoutMs = 10000;
|
|
||||||
private int dataTimeoutMs = 20000;
|
|
||||||
private int bufferSize = 8192;
|
|
||||||
|
|
||||||
public String getHost() {
|
|
||||||
return host;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setHost(String host) {
|
|
||||||
this.host = host;
|
|
||||||
}
|
|
||||||
|
|
||||||
public int getPort() {
|
|
||||||
return port;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setPort(int port) {
|
|
||||||
this.port = port;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getUsername() {
|
|
||||||
return username;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setUsername(String username) {
|
|
||||||
this.username = username;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getPassword() {
|
|
||||||
return password;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setPassword(String password) {
|
|
||||||
this.password = password;
|
|
||||||
}
|
|
||||||
|
|
||||||
public boolean isPassiveMode() {
|
|
||||||
return passiveMode;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setPassiveMode(boolean passiveMode) {
|
|
||||||
this.passiveMode = passiveMode;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getBaseDir() {
|
|
||||||
return baseDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setBaseDir(String baseDir) {
|
|
||||||
this.baseDir = baseDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public int getConnectTimeoutMs() {
|
|
||||||
return connectTimeoutMs;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setConnectTimeoutMs(int connectTimeoutMs) {
|
|
||||||
this.connectTimeoutMs = connectTimeoutMs;
|
|
||||||
}
|
|
||||||
|
|
||||||
public int getDataTimeoutMs() {
|
|
||||||
return dataTimeoutMs;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setDataTimeoutMs(int dataTimeoutMs) {
|
|
||||||
this.dataTimeoutMs = dataTimeoutMs;
|
|
||||||
}
|
|
||||||
|
|
||||||
public int getBufferSize() {
|
|
||||||
return bufferSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setBufferSize(int bufferSize) {
|
|
||||||
this.bufferSize = bufferSize;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -12,12 +12,6 @@ public class SyncProperties {
|
|||||||
private String devToProdStagingDir;
|
private String devToProdStagingDir;
|
||||||
private String prodToDevStagingDir;
|
private String prodToDevStagingDir;
|
||||||
private int maxRetryCount = 5;
|
private int maxRetryCount = 5;
|
||||||
private int ackScanBatchSize = 50;
|
|
||||||
private String remoteDevToProdOutDir;
|
|
||||||
private String remoteDevToProdAckDir;
|
|
||||||
private String remoteProdToDevOutDir;
|
|
||||||
private String remoteProdToDevAckDir;
|
|
||||||
private String remoteFailedDir;
|
|
||||||
private String pullResponseFileName;
|
private String pullResponseFileName;
|
||||||
|
|
||||||
public String getNodeId() {
|
public String getNodeId() {
|
||||||
@ -76,54 +70,6 @@ public class SyncProperties {
|
|||||||
this.maxRetryCount = maxRetryCount;
|
this.maxRetryCount = maxRetryCount;
|
||||||
}
|
}
|
||||||
|
|
||||||
public int getAckScanBatchSize() {
|
|
||||||
return ackScanBatchSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckScanBatchSize(int ackScanBatchSize) {
|
|
||||||
this.ackScanBatchSize = ackScanBatchSize;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemoteDevToProdOutDir() {
|
|
||||||
return remoteDevToProdOutDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemoteDevToProdOutDir(String remoteDevToProdOutDir) {
|
|
||||||
this.remoteDevToProdOutDir = remoteDevToProdOutDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemoteDevToProdAckDir() {
|
|
||||||
return remoteDevToProdAckDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemoteDevToProdAckDir(String remoteDevToProdAckDir) {
|
|
||||||
this.remoteDevToProdAckDir = remoteDevToProdAckDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemoteProdToDevOutDir() {
|
|
||||||
return remoteProdToDevOutDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemoteProdToDevOutDir(String remoteProdToDevOutDir) {
|
|
||||||
this.remoteProdToDevOutDir = remoteProdToDevOutDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemoteProdToDevAckDir() {
|
|
||||||
return remoteProdToDevAckDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemoteProdToDevAckDir(String remoteProdToDevAckDir) {
|
|
||||||
this.remoteProdToDevAckDir = remoteProdToDevAckDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemoteFailedDir() {
|
|
||||||
return remoteFailedDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemoteFailedDir(String remoteFailedDir) {
|
|
||||||
this.remoteFailedDir = remoteFailedDir;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getPullResponseFileName() {
|
public String getPullResponseFileName() {
|
||||||
return pullResponseFileName;
|
return pullResponseFileName;
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,89 +1,5 @@
|
|||||||
package com.ftptool.sync.entity;
|
package com.ftptool.sync.entity;
|
||||||
|
|
||||||
import javax.persistence.Column;
|
@Deprecated
|
||||||
import javax.persistence.Entity;
|
public final class SyncAck {
|
||||||
import javax.persistence.GeneratedValue;
|
|
||||||
import javax.persistence.GenerationType;
|
|
||||||
import javax.persistence.Id;
|
|
||||||
import javax.persistence.PrePersist;
|
|
||||||
import javax.persistence.Table;
|
|
||||||
import java.time.LocalDateTime;
|
|
||||||
|
|
||||||
@Entity
|
|
||||||
@Table(name = "sync_ack")
|
|
||||||
public class SyncAck {
|
|
||||||
|
|
||||||
@Id
|
|
||||||
@GeneratedValue(strategy = GenerationType.IDENTITY)
|
|
||||||
private Long id;
|
|
||||||
|
|
||||||
@Column(name = "trace_id", nullable = false, length = 64)
|
|
||||||
private String traceId;
|
|
||||||
|
|
||||||
@Column(name = "ack_side", nullable = false, length = 32)
|
|
||||||
private String ackSide;
|
|
||||||
|
|
||||||
@Column(name = "ack_status", nullable = false, length = 32)
|
|
||||||
private String ackStatus;
|
|
||||||
|
|
||||||
@Column(name = "ack_time", nullable = false)
|
|
||||||
private LocalDateTime ackTime;
|
|
||||||
|
|
||||||
@Column(name = "remark", length = 500)
|
|
||||||
private String remark;
|
|
||||||
|
|
||||||
@PrePersist
|
|
||||||
public void prePersist() {
|
|
||||||
if (this.ackTime == null) {
|
|
||||||
this.ackTime = LocalDateTime.now();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public Long getId() {
|
|
||||||
return id;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setId(Long id) {
|
|
||||||
this.id = id;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getTraceId() {
|
|
||||||
return traceId;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setTraceId(String traceId) {
|
|
||||||
this.traceId = traceId;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getAckSide() {
|
|
||||||
return ackSide;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckSide(String ackSide) {
|
|
||||||
this.ackSide = ackSide;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getAckStatus() {
|
|
||||||
return ackStatus;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckStatus(String ackStatus) {
|
|
||||||
this.ackStatus = ackStatus;
|
|
||||||
}
|
|
||||||
|
|
||||||
public LocalDateTime getAckTime() {
|
|
||||||
return ackTime;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckTime(LocalDateTime ackTime) {
|
|
||||||
this.ackTime = ackTime;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getRemark() {
|
|
||||||
return remark;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setRemark(String remark) {
|
|
||||||
this.remark = remark;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.DevSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class DevAckScanJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("dev-agent")
|
|
||||||
public class DevAckScanJob {
|
|
||||||
|
|
||||||
private final DevSyncCoordinator devSyncCoordinator;
|
|
||||||
|
|
||||||
public DevAckScanJob(DevSyncCoordinator devSyncCoordinator) {
|
|
||||||
this.devSyncCoordinator = devSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.dev-ack-scan.cron}")
|
|
||||||
public void execute() {
|
|
||||||
devSyncCoordinator.scanProdAcks();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.DevSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class DevConsumeProdPackageJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("dev-agent")
|
|
||||||
public class DevConsumeProdPackageJob {
|
|
||||||
|
|
||||||
private final DevSyncCoordinator devSyncCoordinator;
|
|
||||||
|
|
||||||
public DevConsumeProdPackageJob(DevSyncCoordinator devSyncCoordinator) {
|
|
||||||
this.devSyncCoordinator = devSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.dev-consume-prod-package.cron}")
|
|
||||||
public void execute() {
|
|
||||||
devSyncCoordinator.consumeProdPackages();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.DevSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class DevGitScanJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("dev-agent")
|
|
||||||
public class DevGitScanJob {
|
|
||||||
|
|
||||||
private final DevSyncCoordinator devSyncCoordinator;
|
|
||||||
|
|
||||||
public DevGitScanJob(DevSyncCoordinator devSyncCoordinator) {
|
|
||||||
this.devSyncCoordinator = devSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.dev-git-scan.cron}")
|
|
||||||
public void execute() {
|
|
||||||
devSyncCoordinator.scanGitAndStagePackage();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
22
src/main/java/com/ftptool/sync/job/GitToProdSyncJob.java
Normal file
22
src/main/java/com/ftptool/sync/job/GitToProdSyncJob.java
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
|
import com.ftptool.sync.orchestrator.ProdSyncCoordinator;
|
||||||
|
import org.springframework.context.annotation.Profile;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@Profile("prod-agent")
|
||||||
|
public class GitToProdSyncJob {
|
||||||
|
|
||||||
|
private final ProdSyncCoordinator prodSyncCoordinator;
|
||||||
|
|
||||||
|
public GitToProdSyncJob(ProdSyncCoordinator prodSyncCoordinator) {
|
||||||
|
this.prodSyncCoordinator = prodSyncCoordinator;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(cron = "${sync.jobs.prod-git-to-prod.cron}")
|
||||||
|
public void execute() {
|
||||||
|
prodSyncCoordinator.syncLatestGitToProd();
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.ProdSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class ProdAckScanJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("prod-agent")
|
|
||||||
public class ProdAckScanJob {
|
|
||||||
|
|
||||||
private final ProdSyncCoordinator prodSyncCoordinator;
|
|
||||||
|
|
||||||
public ProdAckScanJob(ProdSyncCoordinator prodSyncCoordinator) {
|
|
||||||
this.prodSyncCoordinator = prodSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.prod-ack-scan.cron}")
|
|
||||||
public void execute() {
|
|
||||||
prodSyncCoordinator.scanDevAcks();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.ProdSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class ProdConsumeDevPackageJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("prod-agent")
|
|
||||||
public class ProdConsumeDevPackageJob {
|
|
||||||
|
|
||||||
private final ProdSyncCoordinator prodSyncCoordinator;
|
|
||||||
|
|
||||||
public ProdConsumeDevPackageJob(ProdSyncCoordinator prodSyncCoordinator) {
|
|
||||||
this.prodSyncCoordinator = prodSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.prod-consume-dev-package.cron}")
|
|
||||||
public void execute() {
|
|
||||||
prodSyncCoordinator.consumeDevPackages();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,22 +1,5 @@
|
|||||||
package com.ftptool.sync.job;
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
import com.ftptool.sync.orchestrator.ProdSyncCoordinator;
|
@Deprecated
|
||||||
import org.springframework.context.annotation.Profile;
|
public final class ProdPullConfigJob {
|
||||||
import org.springframework.scheduling.annotation.Scheduled;
|
|
||||||
import org.springframework.stereotype.Component;
|
|
||||||
|
|
||||||
@Component
|
|
||||||
@Profile("prod-agent")
|
|
||||||
public class ProdPullConfigJob {
|
|
||||||
|
|
||||||
private final ProdSyncCoordinator prodSyncCoordinator;
|
|
||||||
|
|
||||||
public ProdPullConfigJob(ProdSyncCoordinator prodSyncCoordinator) {
|
|
||||||
this.prodSyncCoordinator = prodSyncCoordinator;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Scheduled(cron = "${sync.jobs.prod-pull-config.cron}")
|
|
||||||
public void execute() {
|
|
||||||
prodSyncCoordinator.pullProdConfigAndStagePackage();
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
22
src/main/java/com/ftptool/sync/job/ProdToGitSnapshotJob.java
Normal file
22
src/main/java/com/ftptool/sync/job/ProdToGitSnapshotJob.java
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
package com.ftptool.sync.job;
|
||||||
|
|
||||||
|
import com.ftptool.sync.orchestrator.ProdSyncCoordinator;
|
||||||
|
import org.springframework.context.annotation.Profile;
|
||||||
|
import org.springframework.scheduling.annotation.Scheduled;
|
||||||
|
import org.springframework.stereotype.Component;
|
||||||
|
|
||||||
|
@Component
|
||||||
|
@Profile("prod-agent")
|
||||||
|
public class ProdToGitSnapshotJob {
|
||||||
|
|
||||||
|
private final ProdSyncCoordinator prodSyncCoordinator;
|
||||||
|
|
||||||
|
public ProdToGitSnapshotJob(ProdSyncCoordinator prodSyncCoordinator) {
|
||||||
|
this.prodSyncCoordinator = prodSyncCoordinator;
|
||||||
|
}
|
||||||
|
|
||||||
|
@Scheduled(cron = "${sync.jobs.prod-to-git.cron}")
|
||||||
|
public void execute() {
|
||||||
|
prodSyncCoordinator.syncProdSnapshotToGit();
|
||||||
|
}
|
||||||
|
}
|
||||||
@ -1,20 +1,5 @@
|
|||||||
package com.ftptool.sync.model;
|
package com.ftptool.sync.model;
|
||||||
|
|
||||||
public class RemoteFileInfo {
|
@Deprecated
|
||||||
|
public final class RemoteFileInfo {
|
||||||
private final String name;
|
|
||||||
private final String path;
|
|
||||||
|
|
||||||
public RemoteFileInfo(String name, String path) {
|
|
||||||
this.name = name;
|
|
||||||
this.path = path;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getName() {
|
|
||||||
return name;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getPath() {
|
|
||||||
return path;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,68 +1,5 @@
|
|||||||
package com.ftptool.sync.model;
|
package com.ftptool.sync.model;
|
||||||
|
|
||||||
public class SyncAckFile {
|
@Deprecated
|
||||||
|
public final class SyncAckFile {
|
||||||
private String traceId;
|
|
||||||
private SyncDirection direction;
|
|
||||||
private String sourceVersion;
|
|
||||||
private String ackSide;
|
|
||||||
private String ackStatus;
|
|
||||||
private String message;
|
|
||||||
private String processedAt;
|
|
||||||
|
|
||||||
public String getTraceId() {
|
|
||||||
return traceId;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setTraceId(String traceId) {
|
|
||||||
this.traceId = traceId;
|
|
||||||
}
|
|
||||||
|
|
||||||
public SyncDirection getDirection() {
|
|
||||||
return direction;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setDirection(SyncDirection direction) {
|
|
||||||
this.direction = direction;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getSourceVersion() {
|
|
||||||
return sourceVersion;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setSourceVersion(String sourceVersion) {
|
|
||||||
this.sourceVersion = sourceVersion;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getAckSide() {
|
|
||||||
return ackSide;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckSide(String ackSide) {
|
|
||||||
this.ackSide = ackSide;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getAckStatus() {
|
|
||||||
return ackStatus;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setAckStatus(String ackStatus) {
|
|
||||||
this.ackStatus = ackStatus;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getMessage() {
|
|
||||||
return message;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setMessage(String message) {
|
|
||||||
this.message = message;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String getProcessedAt() {
|
|
||||||
return processedAt;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void setProcessedAt(String processedAt) {
|
|
||||||
this.processedAt = processedAt;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,276 +1,5 @@
|
|||||||
package com.ftptool.sync.orchestrator;
|
package com.ftptool.sync.orchestrator;
|
||||||
|
|
||||||
import com.ftptool.sync.config.FtpProperties;
|
@Deprecated
|
||||||
import com.ftptool.sync.config.GitRepoProperties;
|
public final class DevSyncCoordinator {
|
||||||
import com.ftptool.sync.config.SyncProperties;
|
|
||||||
import com.ftptool.sync.entity.SyncTask;
|
|
||||||
import com.ftptool.sync.model.PackageBuildResult;
|
|
||||||
import com.ftptool.sync.model.PackageManifest;
|
|
||||||
import com.ftptool.sync.model.PackageReadResult;
|
|
||||||
import com.ftptool.sync.model.RemoteFileInfo;
|
|
||||||
import com.ftptool.sync.model.SyncAckFile;
|
|
||||||
import com.ftptool.sync.model.SyncDirection;
|
|
||||||
import com.ftptool.sync.model.SyncStatus;
|
|
||||||
import com.ftptool.sync.service.AckFileService;
|
|
||||||
import com.ftptool.sync.service.AckService;
|
|
||||||
import com.ftptool.sync.service.CheckpointService;
|
|
||||||
import com.ftptool.sync.service.FtpClientService;
|
|
||||||
import com.ftptool.sync.service.GitClientService;
|
|
||||||
import com.ftptool.sync.service.PackageService;
|
|
||||||
import com.ftptool.sync.service.SyncMetadataService;
|
|
||||||
import com.ftptool.sync.service.SyncTaskService;
|
|
||||||
import com.ftptool.sync.service.WorkDirectoryService;
|
|
||||||
import org.slf4j.Logger;
|
|
||||||
import org.slf4j.LoggerFactory;
|
|
||||||
import org.springframework.context.annotation.Profile;
|
|
||||||
import org.springframework.stereotype.Service;
|
|
||||||
|
|
||||||
import java.nio.file.Path;
|
|
||||||
import java.util.List;
|
|
||||||
import java.util.Optional;
|
|
||||||
|
|
||||||
@Service
|
|
||||||
@Profile("dev-agent")
|
|
||||||
public class DevSyncCoordinator {
|
|
||||||
|
|
||||||
private static final Logger log = LoggerFactory.getLogger(DevSyncCoordinator.class);
|
|
||||||
|
|
||||||
private final SyncProperties syncProperties;
|
|
||||||
private final GitRepoProperties gitRepoProperties;
|
|
||||||
private final FtpProperties ftpProperties;
|
|
||||||
private final WorkDirectoryService workDirectoryService;
|
|
||||||
private final GitClientService gitClientService;
|
|
||||||
private final PackageService packageService;
|
|
||||||
private final FtpClientService ftpClientService;
|
|
||||||
private final SyncTaskService syncTaskService;
|
|
||||||
private final CheckpointService checkpointService;
|
|
||||||
private final AckFileService ackFileService;
|
|
||||||
private final AckService ackService;
|
|
||||||
private final SyncMetadataService syncMetadataService;
|
|
||||||
|
|
||||||
public DevSyncCoordinator(
|
|
||||||
SyncProperties syncProperties,
|
|
||||||
GitRepoProperties gitRepoProperties,
|
|
||||||
FtpProperties ftpProperties,
|
|
||||||
WorkDirectoryService workDirectoryService,
|
|
||||||
GitClientService gitClientService,
|
|
||||||
PackageService packageService,
|
|
||||||
FtpClientService ftpClientService,
|
|
||||||
SyncTaskService syncTaskService,
|
|
||||||
CheckpointService checkpointService,
|
|
||||||
AckFileService ackFileService,
|
|
||||||
AckService ackService,
|
|
||||||
SyncMetadataService syncMetadataService
|
|
||||||
) {
|
|
||||||
this.syncProperties = syncProperties;
|
|
||||||
this.gitRepoProperties = gitRepoProperties;
|
|
||||||
this.ftpProperties = ftpProperties;
|
|
||||||
this.workDirectoryService = workDirectoryService;
|
|
||||||
this.gitClientService = gitClientService;
|
|
||||||
this.packageService = packageService;
|
|
||||||
this.ftpClientService = ftpClientService;
|
|
||||||
this.syncTaskService = syncTaskService;
|
|
||||||
this.checkpointService = checkpointService;
|
|
||||||
this.ackFileService = ackFileService;
|
|
||||||
this.ackService = ackService;
|
|
||||||
this.syncMetadataService = syncMetadataService;
|
|
||||||
}
|
|
||||||
|
|
||||||
public void scanGitAndStagePackage() {
|
|
||||||
try {
|
|
||||||
log.info(
|
|
||||||
"DEV scan tick. nodeId={}, branch={}, localRepo={}, ftpBaseDir={}",
|
|
||||||
syncProperties.getNodeId(),
|
|
||||||
gitRepoProperties.getScanBranch(),
|
|
||||||
gitRepoProperties.getLocalPath(),
|
|
||||||
ftpProperties.getBaseDir()
|
|
||||||
);
|
|
||||||
String branch = gitRepoProperties.getScanBranch();
|
|
||||||
String sourceVersion = gitClientService.prepareRepositoryAndGetHead(branch);
|
|
||||||
Path exportDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-" + sourceVersion);
|
|
||||||
gitClientService.exportBranchSnapshot(branch, exportDirectory);
|
|
||||||
String contentHash = packageService.calculateDirectoryHash(exportDirectory);
|
|
||||||
|
|
||||||
Optional<SyncTask> existing = syncTaskService.findByBusinessKey(
|
|
||||||
SyncDirection.DEV_TO_PROD,
|
|
||||||
sourceVersion,
|
|
||||||
contentHash
|
|
||||||
);
|
|
||||||
if (shouldSkipStage(existing)) {
|
|
||||||
log.info("DEV package already staged or finished. version={}, hash={}", sourceVersion, contentHash);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
String traceId = existing.map(SyncTask::getTraceId).orElse(syncMetadataService.newTraceId());
|
|
||||||
PackageManifest manifest = syncMetadataService.createManifest(
|
|
||||||
traceId,
|
|
||||||
SyncDirection.DEV_TO_PROD,
|
|
||||||
"DEV",
|
|
||||||
sourceVersion,
|
|
||||||
contentHash
|
|
||||||
);
|
|
||||||
if (existing.isPresent() && existing.get().getPackageName() != null) {
|
|
||||||
manifest.setPackageName(existing.get().getPackageName());
|
|
||||||
}
|
|
||||||
|
|
||||||
PackageBuildResult packageBuildResult = packageService.buildPackageFromDirectory(exportDirectory, manifest);
|
|
||||||
SyncTask task = syncTaskService.createOrLoadTask(
|
|
||||||
SyncDirection.DEV_TO_PROD,
|
|
||||||
sourceVersion,
|
|
||||||
packageBuildResult.getContentHash(),
|
|
||||||
packageBuildResult.getPackageName(),
|
|
||||||
traceId
|
|
||||||
);
|
|
||||||
ftpClientService.uploadAtomic(
|
|
||||||
packageBuildResult.getZipFile(),
|
|
||||||
syncProperties.getRemoteDevToProdOutDir(),
|
|
||||||
task.getPackageName()
|
|
||||||
);
|
|
||||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.UPLOADED, null);
|
|
||||||
log.info("DEV package uploaded. traceId={}, packageName={}", task.getTraceId(), task.getPackageName());
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("DEV scan and stage failed", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public void consumeProdPackages() {
|
|
||||||
try {
|
|
||||||
log.info(
|
|
||||||
"DEV consume tick. snapshotBranch={}, stagingDir={}",
|
|
||||||
gitRepoProperties.getSnapshotBranch(),
|
|
||||||
syncProperties.getProdToDevStagingDir()
|
|
||||||
);
|
|
||||||
List<RemoteFileInfo> remoteFiles = ftpClientService.listFiles(syncProperties.getRemoteProdToDevOutDir(), ".zip");
|
|
||||||
for (RemoteFileInfo remoteFile : remoteFiles) {
|
|
||||||
consumeSingleProdPackage(remoteFile);
|
|
||||||
}
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("DEV consume prod packages failed", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public void scanProdAcks() {
|
|
||||||
try {
|
|
||||||
log.info("DEV ack scan tick. batchSize={}", syncProperties.getAckScanBatchSize());
|
|
||||||
List<RemoteFileInfo> ackFiles = ftpClientService.listFiles(syncProperties.getRemoteDevToProdAckDir(), ".json");
|
|
||||||
for (RemoteFileInfo ackFile : ackFiles) {
|
|
||||||
Path localAck = ftpClientService.download(ackFile.getPath(), workDirectoryService.getPackageTempDir());
|
|
||||||
SyncAckFile syncAckFile = ackFileService.readAckFile(localAck);
|
|
||||||
ackService.recordAck(
|
|
||||||
syncAckFile.getTraceId(),
|
|
||||||
syncAckFile.getAckSide(),
|
|
||||||
syncAckFile.getAckStatus(),
|
|
||||||
syncAckFile.getMessage()
|
|
||||||
);
|
|
||||||
syncTaskService.findByTraceId(syncAckFile.getTraceId()).ifPresent(task -> {
|
|
||||||
SyncStatus status = "SUCCESS".equalsIgnoreCase(syncAckFile.getAckStatus())
|
|
||||||
? SyncStatus.SUCCESS : SyncStatus.FAILED;
|
|
||||||
syncTaskService.markStatus(task.getTraceId(), status, syncAckFile.getMessage());
|
|
||||||
if (status == SyncStatus.SUCCESS) {
|
|
||||||
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
|
||||||
}
|
|
||||||
});
|
|
||||||
ftpClientService.deleteFile(ackFile.getPath());
|
|
||||||
}
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("DEV ack scan failed", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private void consumeSingleProdPackage(RemoteFileInfo remoteFile) {
|
|
||||||
PackageManifest manifest = null;
|
|
||||||
try {
|
|
||||||
Path localZip = ftpClientService.download(remoteFile.getPath(), workDirectoryService.getProdToDevStagingDir());
|
|
||||||
PackageReadResult readResult = packageService.extractPackage(localZip);
|
|
||||||
manifest = readResult.getManifest();
|
|
||||||
if (manifest.getDirection() != SyncDirection.PROD_TO_DEV) {
|
|
||||||
log.warn("Ignored remote file with unexpected direction. file={}, direction={}", remoteFile.getName(), manifest.getDirection());
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
SyncTask task = syncTaskService.createOrLoadTask(
|
|
||||||
manifest.getDirection(),
|
|
||||||
manifest.getSourceVersion(),
|
|
||||||
manifest.getContentHash(),
|
|
||||||
manifest.getPackageName(),
|
|
||||||
manifest.getTraceId()
|
|
||||||
);
|
|
||||||
if (task.getStatus() == SyncStatus.SUCCESS) {
|
|
||||||
ftpClientService.deleteFile(remoteFile.getPath());
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
String commitMessage = gitRepoProperties.getCommitMessagePrefix()
|
|
||||||
+ ": traceId=" + manifest.getTraceId()
|
|
||||||
+ " version=" + manifest.getSourceVersion();
|
|
||||||
boolean pushed = gitClientService.syncDirectoryToBranch(
|
|
||||||
readResult.getConfigDirectory(),
|
|
||||||
gitRepoProperties.getSnapshotBranch(),
|
|
||||||
commitMessage
|
|
||||||
);
|
|
||||||
|
|
||||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
|
||||||
checkpointService.saveCheckpoint(manifest.getDirection(), manifest.getSourceVersion(), manifest.getContentHash());
|
|
||||||
|
|
||||||
SyncAckFile ack = syncMetadataService.createAck(
|
|
||||||
manifest.getTraceId(),
|
|
||||||
manifest.getDirection(),
|
|
||||||
manifest.getSourceVersion(),
|
|
||||||
"DEV",
|
|
||||||
"SUCCESS",
|
|
||||||
pushed ? "Snapshot committed to Git" : "No Git changes detected"
|
|
||||||
);
|
|
||||||
Path ackPath = ackFileService.writeAckFile(ack, manifest.getTraceId());
|
|
||||||
ftpClientService.uploadAtomic(
|
|
||||||
ackPath,
|
|
||||||
syncProperties.getRemoteProdToDevAckDir(),
|
|
||||||
syncMetadataService.buildAckFileName(manifest.getTraceId())
|
|
||||||
);
|
|
||||||
ackService.recordAck(manifest.getTraceId(), "DEV", "SUCCESS", ack.getMessage());
|
|
||||||
ftpClientService.deleteFile(remoteFile.getPath());
|
|
||||||
log.info("DEV consumed PROD package. traceId={}, packageName={}", manifest.getTraceId(), manifest.getPackageName());
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("DEV failed to consume PROD package: {}", remoteFile.getName(), e);
|
|
||||||
if (manifest != null) {
|
|
||||||
syncTaskService.increaseRetryCount(manifest.getTraceId(), summarizeException(e));
|
|
||||||
syncTaskService.markStatus(manifest.getTraceId(), SyncStatus.FAILED, summarizeException(e));
|
|
||||||
uploadFailureAck(manifest, summarizeException(e));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private boolean shouldSkipStage(Optional<SyncTask> existing) {
|
|
||||||
return existing.isPresent()
|
|
||||||
&& (existing.get().getStatus() == SyncStatus.UPLOADED || existing.get().getStatus() == SyncStatus.SUCCESS);
|
|
||||||
}
|
|
||||||
|
|
||||||
private void uploadFailureAck(PackageManifest manifest, String message) {
|
|
||||||
try {
|
|
||||||
SyncAckFile ack = syncMetadataService.createAck(
|
|
||||||
manifest.getTraceId(),
|
|
||||||
manifest.getDirection(),
|
|
||||||
manifest.getSourceVersion(),
|
|
||||||
"DEV",
|
|
||||||
"FAILED",
|
|
||||||
message
|
|
||||||
);
|
|
||||||
Path ackPath = ackFileService.writeAckFile(ack, manifest.getTraceId());
|
|
||||||
ftpClientService.uploadAtomic(
|
|
||||||
ackPath,
|
|
||||||
syncProperties.getRemoteProdToDevAckDir(),
|
|
||||||
syncMetadataService.buildAckFileName(manifest.getTraceId())
|
|
||||||
);
|
|
||||||
ackService.recordAck(manifest.getTraceId(), "DEV", "FAILED", message);
|
|
||||||
} catch (Exception ex) {
|
|
||||||
log.error("DEV failed to upload failure ack. traceId={}", manifest.getTraceId(), ex);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private String summarizeException(Exception e) {
|
|
||||||
String message = e.getMessage();
|
|
||||||
if (message == null || message.trim().isEmpty()) {
|
|
||||||
return e.getClass().getSimpleName();
|
|
||||||
}
|
|
||||||
return message.length() > 400 ? message.substring(0, 400) : message;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,21 +1,16 @@
|
|||||||
package com.ftptool.sync.orchestrator;
|
package com.ftptool.sync.orchestrator;
|
||||||
|
|
||||||
import com.ftptool.sync.config.FtpProperties;
|
import com.ftptool.sync.config.GitRepoProperties;
|
||||||
import com.ftptool.sync.config.ProdApiProperties;
|
import com.ftptool.sync.config.ProdApiProperties;
|
||||||
import com.ftptool.sync.config.SyncProperties;
|
import com.ftptool.sync.config.SyncProperties;
|
||||||
import com.ftptool.sync.entity.SyncTask;
|
import com.ftptool.sync.entity.SyncTask;
|
||||||
import com.ftptool.sync.model.PackageBuildResult;
|
import com.ftptool.sync.model.PackageBuildResult;
|
||||||
import com.ftptool.sync.model.PackageManifest;
|
import com.ftptool.sync.model.PackageManifest;
|
||||||
import com.ftptool.sync.model.PackageReadResult;
|
|
||||||
import com.ftptool.sync.model.ProdPullResult;
|
import com.ftptool.sync.model.ProdPullResult;
|
||||||
import com.ftptool.sync.model.RemoteFileInfo;
|
|
||||||
import com.ftptool.sync.model.SyncAckFile;
|
|
||||||
import com.ftptool.sync.model.SyncDirection;
|
import com.ftptool.sync.model.SyncDirection;
|
||||||
import com.ftptool.sync.model.SyncStatus;
|
import com.ftptool.sync.model.SyncStatus;
|
||||||
import com.ftptool.sync.service.AckFileService;
|
|
||||||
import com.ftptool.sync.service.AckService;
|
|
||||||
import com.ftptool.sync.service.CheckpointService;
|
import com.ftptool.sync.service.CheckpointService;
|
||||||
import com.ftptool.sync.service.FtpClientService;
|
import com.ftptool.sync.service.GitClientService;
|
||||||
import com.ftptool.sync.service.PackageService;
|
import com.ftptool.sync.service.PackageService;
|
||||||
import com.ftptool.sync.service.ProdConfigApiService;
|
import com.ftptool.sync.service.ProdConfigApiService;
|
||||||
import com.ftptool.sync.service.SyncMetadataService;
|
import com.ftptool.sync.service.SyncMetadataService;
|
||||||
@ -27,7 +22,6 @@ import org.springframework.context.annotation.Profile;
|
|||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
import java.nio.file.Path;
|
import java.nio.file.Path;
|
||||||
import java.util.List;
|
|
||||||
import java.util.Optional;
|
import java.util.Optional;
|
||||||
|
|
||||||
@Service
|
@Service
|
||||||
@ -37,70 +31,103 @@ public class ProdSyncCoordinator {
|
|||||||
private static final Logger log = LoggerFactory.getLogger(ProdSyncCoordinator.class);
|
private static final Logger log = LoggerFactory.getLogger(ProdSyncCoordinator.class);
|
||||||
|
|
||||||
private final SyncProperties syncProperties;
|
private final SyncProperties syncProperties;
|
||||||
private final FtpProperties ftpProperties;
|
private final GitRepoProperties gitRepoProperties;
|
||||||
private final ProdApiProperties prodApiProperties;
|
private final ProdApiProperties prodApiProperties;
|
||||||
private final WorkDirectoryService workDirectoryService;
|
private final WorkDirectoryService workDirectoryService;
|
||||||
private final FtpClientService ftpClientService;
|
private final GitClientService gitClientService;
|
||||||
private final PackageService packageService;
|
private final PackageService packageService;
|
||||||
private final ProdConfigApiService prodConfigApiService;
|
private final ProdConfigApiService prodConfigApiService;
|
||||||
private final SyncTaskService syncTaskService;
|
private final SyncTaskService syncTaskService;
|
||||||
private final CheckpointService checkpointService;
|
private final CheckpointService checkpointService;
|
||||||
private final AckFileService ackFileService;
|
|
||||||
private final AckService ackService;
|
|
||||||
private final SyncMetadataService syncMetadataService;
|
private final SyncMetadataService syncMetadataService;
|
||||||
|
|
||||||
public ProdSyncCoordinator(
|
public ProdSyncCoordinator(
|
||||||
SyncProperties syncProperties,
|
SyncProperties syncProperties,
|
||||||
FtpProperties ftpProperties,
|
GitRepoProperties gitRepoProperties,
|
||||||
ProdApiProperties prodApiProperties,
|
ProdApiProperties prodApiProperties,
|
||||||
WorkDirectoryService workDirectoryService,
|
WorkDirectoryService workDirectoryService,
|
||||||
FtpClientService ftpClientService,
|
GitClientService gitClientService,
|
||||||
PackageService packageService,
|
PackageService packageService,
|
||||||
ProdConfigApiService prodConfigApiService,
|
ProdConfigApiService prodConfigApiService,
|
||||||
SyncTaskService syncTaskService,
|
SyncTaskService syncTaskService,
|
||||||
CheckpointService checkpointService,
|
CheckpointService checkpointService,
|
||||||
AckFileService ackFileService,
|
|
||||||
AckService ackService,
|
|
||||||
SyncMetadataService syncMetadataService
|
SyncMetadataService syncMetadataService
|
||||||
) {
|
) {
|
||||||
this.syncProperties = syncProperties;
|
this.syncProperties = syncProperties;
|
||||||
this.ftpProperties = ftpProperties;
|
this.gitRepoProperties = gitRepoProperties;
|
||||||
this.prodApiProperties = prodApiProperties;
|
this.prodApiProperties = prodApiProperties;
|
||||||
this.workDirectoryService = workDirectoryService;
|
this.workDirectoryService = workDirectoryService;
|
||||||
this.ftpClientService = ftpClientService;
|
this.gitClientService = gitClientService;
|
||||||
this.packageService = packageService;
|
this.packageService = packageService;
|
||||||
this.prodConfigApiService = prodConfigApiService;
|
this.prodConfigApiService = prodConfigApiService;
|
||||||
this.syncTaskService = syncTaskService;
|
this.syncTaskService = syncTaskService;
|
||||||
this.checkpointService = checkpointService;
|
this.checkpointService = checkpointService;
|
||||||
this.ackFileService = ackFileService;
|
|
||||||
this.ackService = ackService;
|
|
||||||
this.syncMetadataService = syncMetadataService;
|
this.syncMetadataService = syncMetadataService;
|
||||||
}
|
}
|
||||||
|
|
||||||
public void consumeDevPackages() {
|
public void syncLatestGitToProd() {
|
||||||
|
String traceId = null;
|
||||||
try {
|
try {
|
||||||
log.info(
|
log.info(
|
||||||
"PROD consume tick. nodeId={}, ftpBaseDir={}, pushPath={}",
|
"PROD git->prod tick. nodeId={}, branch={}, pushPath={}",
|
||||||
syncProperties.getNodeId(),
|
syncProperties.getNodeId(),
|
||||||
ftpProperties.getBaseDir(),
|
gitRepoProperties.getScanBranch(),
|
||||||
prodApiProperties.getPushPath()
|
prodApiProperties.getPushPath()
|
||||||
);
|
);
|
||||||
List<RemoteFileInfo> remoteFiles = ftpClientService.listFiles(syncProperties.getRemoteDevToProdOutDir(), ".zip");
|
String branch = gitRepoProperties.getScanBranch();
|
||||||
for (RemoteFileInfo remoteFile : remoteFiles) {
|
String sourceVersion = gitClientService.prepareRepositoryAndGetHead(branch);
|
||||||
consumeSingleDevPackage(remoteFile);
|
Path exportDirectory = workDirectoryService.getDevToProdStagingDir().resolve("git-" + sourceVersion);
|
||||||
|
gitClientService.exportBranchSnapshot(branch, exportDirectory);
|
||||||
|
String contentHash = packageService.calculateDirectoryHash(exportDirectory);
|
||||||
|
|
||||||
|
Optional<SyncTask> existing = syncTaskService.findByBusinessKey(
|
||||||
|
SyncDirection.DEV_TO_PROD,
|
||||||
|
sourceVersion,
|
||||||
|
contentHash
|
||||||
|
);
|
||||||
|
if (shouldSkip(existing)) {
|
||||||
|
log.info("Git version already pushed to prod. version={}, hash={}", sourceVersion, contentHash);
|
||||||
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
traceId = existing.map(SyncTask::getTraceId).orElse(syncMetadataService.newTraceId());
|
||||||
|
PackageManifest manifest = syncMetadataService.createManifest(
|
||||||
|
traceId,
|
||||||
|
SyncDirection.DEV_TO_PROD,
|
||||||
|
"DEV",
|
||||||
|
sourceVersion,
|
||||||
|
contentHash
|
||||||
|
);
|
||||||
|
if (existing.isPresent() && existing.get().getPackageName() != null) {
|
||||||
|
manifest.setPackageName(existing.get().getPackageName());
|
||||||
|
}
|
||||||
|
|
||||||
|
PackageBuildResult packageBuildResult = packageService.buildPackageFromDirectory(exportDirectory, manifest);
|
||||||
|
SyncTask task = syncTaskService.createOrLoadTask(
|
||||||
|
SyncDirection.DEV_TO_PROD,
|
||||||
|
sourceVersion,
|
||||||
|
packageBuildResult.getContentHash(),
|
||||||
|
packageBuildResult.getPackageName(),
|
||||||
|
traceId
|
||||||
|
);
|
||||||
|
syncTaskService.markStatus(task.getTraceId(), SyncStatus.CONSUMING, null);
|
||||||
|
prodConfigApiService.pushPackage(manifest, packageBuildResult.getZipFile());
|
||||||
|
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
||||||
|
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
||||||
|
log.info("Git version pushed to prod successfully. traceId={}, version={}", task.getTraceId(), task.getSourceVersion());
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
log.error("PROD consume DEV packages failed", e);
|
handleFailure(traceId, "PROD git->prod sync failed", e);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public void pullProdConfigAndStagePackage() {
|
public void syncProdSnapshotToGit() {
|
||||||
|
String traceId = null;
|
||||||
try {
|
try {
|
||||||
log.info(
|
log.info(
|
||||||
"PROD pull tick. apiBaseUrl={}, pullPath={}, stagingDir={}",
|
"PROD prod->git tick. apiBaseUrl={}, pullPath={}, snapshotBranch={}",
|
||||||
prodApiProperties.getBaseUrl(),
|
prodApiProperties.getBaseUrl(),
|
||||||
prodApiProperties.getPullPath(),
|
prodApiProperties.getPullPath(),
|
||||||
syncProperties.getProdToDevStagingDir()
|
gitRepoProperties.getSnapshotBranch()
|
||||||
);
|
);
|
||||||
ProdPullResult pullResult = prodConfigApiService.pullConfigSnapshot();
|
ProdPullResult pullResult = prodConfigApiService.pullConfigSnapshot();
|
||||||
Optional<SyncTask> existing = syncTaskService.findByBusinessKey(
|
Optional<SyncTask> existing = syncTaskService.findByBusinessKey(
|
||||||
@ -108,170 +135,58 @@ public class ProdSyncCoordinator {
|
|||||||
pullResult.getSourceVersion(),
|
pullResult.getSourceVersion(),
|
||||||
pullResult.getContentHash()
|
pullResult.getContentHash()
|
||||||
);
|
);
|
||||||
if (shouldSkipStage(existing)) {
|
if (shouldSkip(existing)) {
|
||||||
log.info("PROD pull result already staged or finished. version={}, hash={}",
|
log.info("Production snapshot already synced to Git. version={}, hash={}",
|
||||||
pullResult.getSourceVersion(), pullResult.getContentHash());
|
pullResult.getSourceVersion(), pullResult.getContentHash());
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
String traceId = existing.map(SyncTask::getTraceId).orElse(syncMetadataService.newTraceId());
|
traceId = existing.map(SyncTask::getTraceId).orElse(syncMetadataService.newTraceId());
|
||||||
PackageManifest manifest = syncMetadataService.createManifest(
|
|
||||||
traceId,
|
|
||||||
SyncDirection.PROD_TO_DEV,
|
|
||||||
"PROD",
|
|
||||||
pullResult.getSourceVersion(),
|
|
||||||
pullResult.getContentHash()
|
|
||||||
);
|
|
||||||
if (existing.isPresent() && existing.get().getPackageName() != null) {
|
|
||||||
manifest.setPackageName(existing.get().getPackageName());
|
|
||||||
}
|
|
||||||
|
|
||||||
PackageBuildResult packageBuildResult = packageService.buildPackageFromDirectory(
|
|
||||||
pullResult.getContentDirectory(),
|
|
||||||
manifest
|
|
||||||
);
|
|
||||||
SyncTask task = syncTaskService.createOrLoadTask(
|
SyncTask task = syncTaskService.createOrLoadTask(
|
||||||
SyncDirection.PROD_TO_DEV,
|
SyncDirection.PROD_TO_DEV,
|
||||||
pullResult.getSourceVersion(),
|
pullResult.getSourceVersion(),
|
||||||
packageBuildResult.getContentHash(),
|
pullResult.getContentHash(),
|
||||||
packageBuildResult.getPackageName(),
|
null,
|
||||||
traceId
|
traceId
|
||||||
);
|
);
|
||||||
ftpClientService.uploadAtomic(
|
syncTaskService.markStatus(task.getTraceId(), SyncStatus.CONSUMING, null);
|
||||||
packageBuildResult.getZipFile(),
|
|
||||||
syncProperties.getRemoteProdToDevOutDir(),
|
String commitMessage = gitRepoProperties.getCommitMessagePrefix()
|
||||||
task.getPackageName()
|
+ ": traceId=" + task.getTraceId()
|
||||||
|
+ " version=" + task.getSourceVersion();
|
||||||
|
boolean pushed = gitClientService.syncDirectoryToBranch(
|
||||||
|
pullResult.getContentDirectory(),
|
||||||
|
gitRepoProperties.getSnapshotBranch(),
|
||||||
|
commitMessage
|
||||||
);
|
);
|
||||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.UPLOADED, null);
|
|
||||||
log.info("PROD package uploaded. traceId={}, packageName={}", task.getTraceId(), task.getPackageName());
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("PROD pull and stage failed", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public void scanDevAcks() {
|
|
||||||
try {
|
|
||||||
log.info("PROD ack scan tick. batchSize={}", syncProperties.getAckScanBatchSize());
|
|
||||||
List<RemoteFileInfo> ackFiles = ftpClientService.listFiles(syncProperties.getRemoteProdToDevAckDir(), ".json");
|
|
||||||
for (RemoteFileInfo ackFile : ackFiles) {
|
|
||||||
Path localAck = ftpClientService.download(ackFile.getPath(), workDirectoryService.getPackageTempDir());
|
|
||||||
SyncAckFile syncAckFile = ackFileService.readAckFile(localAck);
|
|
||||||
ackService.recordAck(
|
|
||||||
syncAckFile.getTraceId(),
|
|
||||||
syncAckFile.getAckSide(),
|
|
||||||
syncAckFile.getAckStatus(),
|
|
||||||
syncAckFile.getMessage()
|
|
||||||
);
|
|
||||||
syncTaskService.findByTraceId(syncAckFile.getTraceId()).ifPresent(task -> {
|
|
||||||
SyncStatus status = "SUCCESS".equalsIgnoreCase(syncAckFile.getAckStatus())
|
|
||||||
? SyncStatus.SUCCESS : SyncStatus.FAILED;
|
|
||||||
syncTaskService.markStatus(task.getTraceId(), status, syncAckFile.getMessage());
|
|
||||||
if (status == SyncStatus.SUCCESS) {
|
|
||||||
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
|
||||||
}
|
|
||||||
});
|
|
||||||
ftpClientService.deleteFile(ackFile.getPath());
|
|
||||||
}
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("PROD ack scan failed", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private void consumeSingleDevPackage(RemoteFileInfo remoteFile) {
|
|
||||||
PackageManifest manifest = null;
|
|
||||||
try {
|
|
||||||
Path localZip = ftpClientService.download(remoteFile.getPath(), workDirectoryService.getDevToProdStagingDir());
|
|
||||||
PackageReadResult readResult = packageService.extractPackage(localZip);
|
|
||||||
manifest = readResult.getManifest();
|
|
||||||
if (manifest.getDirection() != SyncDirection.DEV_TO_PROD) {
|
|
||||||
log.warn("Ignored remote file with unexpected direction. file={}, direction={}", remoteFile.getName(), manifest.getDirection());
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
SyncTask task = syncTaskService.createOrLoadTask(
|
|
||||||
manifest.getDirection(),
|
|
||||||
manifest.getSourceVersion(),
|
|
||||||
manifest.getContentHash(),
|
|
||||||
manifest.getPackageName(),
|
|
||||||
manifest.getTraceId()
|
|
||||||
);
|
|
||||||
if (task.getStatus() == SyncStatus.SUCCESS) {
|
|
||||||
ftpClientService.deleteFile(remoteFile.getPath());
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
prodConfigApiService.pushPackage(manifest, localZip);
|
|
||||||
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
syncTaskService.markStatus(task.getTraceId(), SyncStatus.SUCCESS, null);
|
||||||
checkpointService.saveCheckpoint(manifest.getDirection(), manifest.getSourceVersion(), manifest.getContentHash());
|
checkpointService.saveCheckpoint(task.getDirection(), task.getSourceVersion(), task.getContentHash());
|
||||||
|
log.info(
|
||||||
SyncAckFile ack = syncMetadataService.createAck(
|
"Production snapshot synced to Git. traceId={}, version={}, gitPushed={}",
|
||||||
manifest.getTraceId(),
|
task.getTraceId(),
|
||||||
manifest.getDirection(),
|
task.getSourceVersion(),
|
||||||
manifest.getSourceVersion(),
|
pushed
|
||||||
"PROD",
|
|
||||||
"SUCCESS",
|
|
||||||
"Package pushed to production API"
|
|
||||||
);
|
);
|
||||||
Path ackPath = ackFileService.writeAckFile(ack, manifest.getTraceId());
|
|
||||||
ftpClientService.uploadAtomic(
|
|
||||||
ackPath,
|
|
||||||
syncProperties.getRemoteDevToProdAckDir(),
|
|
||||||
syncMetadataService.buildAckFileName(manifest.getTraceId())
|
|
||||||
);
|
|
||||||
ackService.recordAck(manifest.getTraceId(), "PROD", "SUCCESS", ack.getMessage());
|
|
||||||
ftpClientService.deleteFile(remoteFile.getPath());
|
|
||||||
log.info("PROD consumed DEV package. traceId={}, packageName={}", manifest.getTraceId(), manifest.getPackageName());
|
|
||||||
} catch (Exception e) {
|
} catch (Exception e) {
|
||||||
log.error("PROD failed to consume DEV package: {}", remoteFile.getName(), e);
|
handleFailure(traceId, "PROD prod->git sync failed", e);
|
||||||
if (manifest != null) {
|
|
||||||
syncTaskService.increaseRetryCount(manifest.getTraceId(), summarizeException(e));
|
|
||||||
Optional<SyncTask> task = syncTaskService.findByTraceId(manifest.getTraceId());
|
|
||||||
int retryCount = task.map(SyncTask::getRetryCount).orElse(0);
|
|
||||||
if (retryCount >= syncProperties.getMaxRetryCount()) {
|
|
||||||
syncTaskService.markStatus(manifest.getTraceId(), SyncStatus.FAILED, summarizeException(e));
|
|
||||||
uploadFailureAck(manifest, summarizeException(e));
|
|
||||||
moveToFailed(remoteFile, manifest);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private boolean shouldSkipStage(Optional<SyncTask> existing) {
|
private boolean shouldSkip(Optional<SyncTask> existing) {
|
||||||
return existing.isPresent()
|
return existing.isPresent() && existing.get().getStatus() == SyncStatus.SUCCESS;
|
||||||
&& (existing.get().getStatus() == SyncStatus.UPLOADED || existing.get().getStatus() == SyncStatus.SUCCESS);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
private void uploadFailureAck(PackageManifest manifest, String message) {
|
private void handleFailure(String traceId, String logMessage, Exception e) {
|
||||||
try {
|
log.error(logMessage, e);
|
||||||
SyncAckFile ack = syncMetadataService.createAck(
|
if (traceId == null) {
|
||||||
manifest.getTraceId(),
|
return;
|
||||||
manifest.getDirection(),
|
|
||||||
manifest.getSourceVersion(),
|
|
||||||
"PROD",
|
|
||||||
"FAILED",
|
|
||||||
message
|
|
||||||
);
|
|
||||||
Path ackPath = ackFileService.writeAckFile(ack, manifest.getTraceId());
|
|
||||||
ftpClientService.uploadAtomic(
|
|
||||||
ackPath,
|
|
||||||
syncProperties.getRemoteDevToProdAckDir(),
|
|
||||||
syncMetadataService.buildAckFileName(manifest.getTraceId())
|
|
||||||
);
|
|
||||||
ackService.recordAck(manifest.getTraceId(), "PROD", "FAILED", message);
|
|
||||||
} catch (Exception ex) {
|
|
||||||
log.error("PROD failed to upload failure ack. traceId={}", manifest.getTraceId(), ex);
|
|
||||||
}
|
}
|
||||||
}
|
syncTaskService.increaseRetryCount(traceId, summarizeException(e));
|
||||||
|
Optional<SyncTask> task = syncTaskService.findByTraceId(traceId);
|
||||||
private void moveToFailed(RemoteFileInfo remoteFile, PackageManifest manifest) {
|
int retryCount = task.map(SyncTask::getRetryCount).orElse(0);
|
||||||
try {
|
if (retryCount >= syncProperties.getMaxRetryCount()) {
|
||||||
ftpClientService.moveFile(
|
syncTaskService.markStatus(traceId, SyncStatus.FAILED, summarizeException(e));
|
||||||
remoteFile.getPath(),
|
|
||||||
syncProperties.getRemoteFailedDir(),
|
|
||||||
remoteFile.getName()
|
|
||||||
);
|
|
||||||
} catch (Exception e) {
|
|
||||||
log.error("PROD failed to move package to failed dir. traceId={}", manifest.getTraceId(), e);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@ -1,11 +1,5 @@
|
|||||||
package com.ftptool.sync.repository;
|
package com.ftptool.sync.repository;
|
||||||
|
|
||||||
import com.ftptool.sync.entity.SyncAck;
|
@Deprecated
|
||||||
import org.springframework.data.jpa.repository.JpaRepository;
|
public final class SyncAckRepository {
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
public interface SyncAckRepository extends JpaRepository<SyncAck, Long> {
|
|
||||||
|
|
||||||
List<SyncAck> findTop50ByTraceIdOrderByAckTimeDesc(String traceId);
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,35 +1,5 @@
|
|||||||
package com.ftptool.sync.service;
|
package com.ftptool.sync.service;
|
||||||
|
|
||||||
import com.fasterxml.jackson.databind.ObjectMapper;
|
@Deprecated
|
||||||
import com.ftptool.sync.model.SyncAckFile;
|
public final class AckFileService {
|
||||||
import org.springframework.stereotype.Service;
|
|
||||||
|
|
||||||
import java.io.IOException;
|
|
||||||
import java.nio.file.Files;
|
|
||||||
import java.nio.file.Path;
|
|
||||||
import java.time.OffsetDateTime;
|
|
||||||
|
|
||||||
@Service
|
|
||||||
public class AckFileService {
|
|
||||||
|
|
||||||
private final ObjectMapper objectMapper;
|
|
||||||
private final WorkDirectoryService workDirectoryService;
|
|
||||||
|
|
||||||
public AckFileService(ObjectMapper objectMapper, WorkDirectoryService workDirectoryService) {
|
|
||||||
this.objectMapper = objectMapper;
|
|
||||||
this.workDirectoryService = workDirectoryService;
|
|
||||||
}
|
|
||||||
|
|
||||||
public Path writeAckFile(SyncAckFile ackFile, String fileNamePrefix) throws IOException {
|
|
||||||
Path path = Files.createTempFile(workDirectoryService.getPackageTempDir(), fileNamePrefix + "-", ".ack.json");
|
|
||||||
if (ackFile.getProcessedAt() == null) {
|
|
||||||
ackFile.setProcessedAt(OffsetDateTime.now().toString());
|
|
||||||
}
|
|
||||||
objectMapper.writerWithDefaultPrettyPrinter().writeValue(path.toFile(), ackFile);
|
|
||||||
return path;
|
|
||||||
}
|
|
||||||
|
|
||||||
public SyncAckFile readAckFile(Path path) throws IOException {
|
|
||||||
return objectMapper.readValue(path.toFile(), SyncAckFile.class);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,33 +1,5 @@
|
|||||||
package com.ftptool.sync.service;
|
package com.ftptool.sync.service;
|
||||||
|
|
||||||
import com.ftptool.sync.entity.SyncAck;
|
@Deprecated
|
||||||
import com.ftptool.sync.repository.SyncAckRepository;
|
public final class AckService {
|
||||||
import org.springframework.stereotype.Service;
|
|
||||||
import org.springframework.transaction.annotation.Transactional;
|
|
||||||
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
@Service
|
|
||||||
public class AckService {
|
|
||||||
|
|
||||||
private final SyncAckRepository syncAckRepository;
|
|
||||||
|
|
||||||
public AckService(SyncAckRepository syncAckRepository) {
|
|
||||||
this.syncAckRepository = syncAckRepository;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Transactional
|
|
||||||
public SyncAck recordAck(String traceId, String ackSide, String ackStatus, String remark) {
|
|
||||||
SyncAck syncAck = new SyncAck();
|
|
||||||
syncAck.setTraceId(traceId);
|
|
||||||
syncAck.setAckSide(ackSide);
|
|
||||||
syncAck.setAckStatus(ackStatus);
|
|
||||||
syncAck.setRemark(remark);
|
|
||||||
return syncAckRepository.save(syncAck);
|
|
||||||
}
|
|
||||||
|
|
||||||
@Transactional(readOnly = true)
|
|
||||||
public List<SyncAck> findLatestByTraceId(String traceId) {
|
|
||||||
return syncAckRepository.findTop50ByTraceIdOrderByAckTimeDesc(traceId);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,189 +1,5 @@
|
|||||||
package com.ftptool.sync.service;
|
package com.ftptool.sync.service;
|
||||||
|
|
||||||
import com.ftptool.sync.config.FtpProperties;
|
@Deprecated
|
||||||
import com.ftptool.sync.model.RemoteFileInfo;
|
public final class FtpClientService {
|
||||||
import org.apache.commons.net.ftp.FTP;
|
|
||||||
import org.apache.commons.net.ftp.FTPClient;
|
|
||||||
import org.apache.commons.net.ftp.FTPFile;
|
|
||||||
import org.slf4j.Logger;
|
|
||||||
import org.slf4j.LoggerFactory;
|
|
||||||
import org.springframework.retry.annotation.Backoff;
|
|
||||||
import org.springframework.retry.annotation.Retryable;
|
|
||||||
import org.springframework.stereotype.Service;
|
|
||||||
|
|
||||||
import java.io.IOException;
|
|
||||||
import java.io.InputStream;
|
|
||||||
import java.io.OutputStream;
|
|
||||||
import java.nio.file.Files;
|
|
||||||
import java.nio.file.Path;
|
|
||||||
import java.util.ArrayList;
|
|
||||||
import java.util.Comparator;
|
|
||||||
import java.util.List;
|
|
||||||
|
|
||||||
@Service
|
|
||||||
public class FtpClientService {
|
|
||||||
|
|
||||||
private static final Logger log = LoggerFactory.getLogger(FtpClientService.class);
|
|
||||||
|
|
||||||
private final FtpProperties ftpProperties;
|
|
||||||
|
|
||||||
public FtpClientService(FtpProperties ftpProperties) {
|
|
||||||
this.ftpProperties = ftpProperties;
|
|
||||||
}
|
|
||||||
|
|
||||||
@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000, multiplier = 2.0))
|
|
||||||
public List<RemoteFileInfo> listFiles(String remoteDirectory, String suffix) throws IOException {
|
|
||||||
return withClient(client -> {
|
|
||||||
String normalizedPath = normalizeRemotePath(remoteDirectory);
|
|
||||||
FTPFile[] files = client.listFiles(normalizedPath);
|
|
||||||
List<RemoteFileInfo> result = new ArrayList<RemoteFileInfo>();
|
|
||||||
for (FTPFile file : files) {
|
|
||||||
if (!file.isFile()) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
if (suffix != null && !file.getName().endsWith(suffix)) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
result.add(new RemoteFileInfo(file.getName(), appendPath(remoteDirectory, file.getName())));
|
|
||||||
}
|
|
||||||
result.sort(Comparator.comparing(RemoteFileInfo::getName));
|
|
||||||
return result;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000, multiplier = 2.0))
|
|
||||||
public Path download(String remotePath, Path localDirectory) throws IOException {
|
|
||||||
return withClient(client -> {
|
|
||||||
Files.createDirectories(localDirectory);
|
|
||||||
String fileName = remotePath.substring(remotePath.lastIndexOf('/') + 1);
|
|
||||||
Path localFile = localDirectory.resolve(fileName);
|
|
||||||
try (OutputStream outputStream = Files.newOutputStream(localFile)) {
|
|
||||||
if (!client.retrieveFile(normalizeRemotePath(remotePath), outputStream)) {
|
|
||||||
throw new IOException("Failed to download remote file: " + remotePath);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return localFile;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000, multiplier = 2.0))
|
|
||||||
public void uploadAtomic(Path localFile, String remoteDirectory, String remoteFileName) throws IOException {
|
|
||||||
withClient(client -> {
|
|
||||||
ensureDirectoryExists(client, remoteDirectory);
|
|
||||||
String tempName = remoteFileName + ".tmp";
|
|
||||||
String tempPath = appendPath(remoteDirectory, tempName);
|
|
||||||
String finalPath = appendPath(remoteDirectory, remoteFileName);
|
|
||||||
try (InputStream inputStream = Files.newInputStream(localFile)) {
|
|
||||||
if (!client.storeFile(tempPath, inputStream)) {
|
|
||||||
throw new IOException("Failed to upload remote file: " + tempPath);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (!client.rename(tempPath, finalPath)) {
|
|
||||||
throw new IOException("Failed to rename remote file: " + tempPath + " -> " + finalPath);
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000, multiplier = 2.0))
|
|
||||||
public void deleteFile(String remotePath) throws IOException {
|
|
||||||
withClient(client -> {
|
|
||||||
String normalized = normalizeRemotePath(remotePath);
|
|
||||||
if (!client.deleteFile(normalized)) {
|
|
||||||
log.warn("Remote file was not deleted: {}", normalized);
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
@Retryable(maxAttempts = 3, backoff = @Backoff(delay = 2000, multiplier = 2.0))
|
|
||||||
public void moveFile(String remotePath, String targetDirectory, String targetFileName) throws IOException {
|
|
||||||
withClient(client -> {
|
|
||||||
ensureDirectoryExists(client, targetDirectory);
|
|
||||||
String source = normalizeRemotePath(remotePath);
|
|
||||||
String target = appendPath(targetDirectory, targetFileName);
|
|
||||||
if (!client.rename(source, target)) {
|
|
||||||
throw new IOException("Failed to move remote file: " + source + " -> " + target);
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
public String appendPath(String directory, String fileName) {
|
|
||||||
return normalizeRemotePath(normalizeSubPath(directory)) + "/" + fileName;
|
|
||||||
}
|
|
||||||
|
|
||||||
private <T> T withClient(FtpCallback<T> callback) throws IOException {
|
|
||||||
FTPClient client = new FTPClient();
|
|
||||||
try {
|
|
||||||
client.setConnectTimeout(ftpProperties.getConnectTimeoutMs());
|
|
||||||
client.setDataTimeout(ftpProperties.getDataTimeoutMs());
|
|
||||||
client.setBufferSize(ftpProperties.getBufferSize());
|
|
||||||
client.connect(ftpProperties.getHost(), ftpProperties.getPort());
|
|
||||||
if (!client.login(ftpProperties.getUsername(), ftpProperties.getPassword())) {
|
|
||||||
throw new IOException("FTP login failed for user " + ftpProperties.getUsername());
|
|
||||||
}
|
|
||||||
client.setFileType(FTP.BINARY_FILE_TYPE);
|
|
||||||
if (ftpProperties.isPassiveMode()) {
|
|
||||||
client.enterLocalPassiveMode();
|
|
||||||
}
|
|
||||||
return callback.doWithClient(client);
|
|
||||||
} finally {
|
|
||||||
disconnectQuietly(client);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private void ensureDirectoryExists(FTPClient client, String directory) throws IOException {
|
|
||||||
String[] segments = normalizeSubPath(directory).split("/");
|
|
||||||
StringBuilder current = new StringBuilder();
|
|
||||||
for (String segment : segments) {
|
|
||||||
if (segment == null || segment.trim().isEmpty()) {
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
current.append("/").append(segment);
|
|
||||||
client.makeDirectory(withBaseDir(current.toString()));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private String normalizeRemotePath(String path) {
|
|
||||||
return withBaseDir(path.startsWith("/") ? path : "/" + path);
|
|
||||||
}
|
|
||||||
|
|
||||||
private String withBaseDir(String path) {
|
|
||||||
String baseDir = ftpProperties.getBaseDir();
|
|
||||||
if (baseDir == null || baseDir.trim().isEmpty() || "/".equals(baseDir.trim())) {
|
|
||||||
return path;
|
|
||||||
}
|
|
||||||
String normalizedBase = baseDir.startsWith("/") ? baseDir : "/" + baseDir;
|
|
||||||
normalizedBase = normalizedBase.endsWith("/") ? normalizedBase.substring(0, normalizedBase.length() - 1) : normalizedBase;
|
|
||||||
return normalizedBase + path;
|
|
||||||
}
|
|
||||||
|
|
||||||
private String normalizeSubPath(String path) {
|
|
||||||
if (path == null || path.trim().isEmpty()) {
|
|
||||||
return "/";
|
|
||||||
}
|
|
||||||
String normalized = path.startsWith("/") ? path : "/" + path;
|
|
||||||
return normalized.endsWith("/") && normalized.length() > 1
|
|
||||||
? normalized.substring(0, normalized.length() - 1)
|
|
||||||
: normalized;
|
|
||||||
}
|
|
||||||
|
|
||||||
private void disconnectQuietly(FTPClient client) {
|
|
||||||
if (client == null) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
if (client.isConnected()) {
|
|
||||||
client.logout();
|
|
||||||
client.disconnect();
|
|
||||||
}
|
|
||||||
} catch (IOException e) {
|
|
||||||
log.warn("Failed to disconnect FTP client cleanly", e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
private interface FtpCallback<T> {
|
|
||||||
T doWithClient(FTPClient client) throws IOException;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|||||||
@ -1,7 +1,6 @@
|
|||||||
package com.ftptool.sync.service;
|
package com.ftptool.sync.service;
|
||||||
|
|
||||||
import com.ftptool.sync.model.PackageManifest;
|
import com.ftptool.sync.model.PackageManifest;
|
||||||
import com.ftptool.sync.model.SyncAckFile;
|
|
||||||
import com.ftptool.sync.model.SyncDirection;
|
import com.ftptool.sync.model.SyncDirection;
|
||||||
import org.springframework.stereotype.Service;
|
import org.springframework.stereotype.Service;
|
||||||
|
|
||||||
@ -33,33 +32,10 @@ public class SyncMetadataService {
|
|||||||
return manifest;
|
return manifest;
|
||||||
}
|
}
|
||||||
|
|
||||||
public SyncAckFile createAck(
|
|
||||||
String traceId,
|
|
||||||
SyncDirection direction,
|
|
||||||
String sourceVersion,
|
|
||||||
String ackSide,
|
|
||||||
String ackStatus,
|
|
||||||
String message
|
|
||||||
) {
|
|
||||||
SyncAckFile ackFile = new SyncAckFile();
|
|
||||||
ackFile.setTraceId(traceId);
|
|
||||||
ackFile.setDirection(direction);
|
|
||||||
ackFile.setSourceVersion(sourceVersion);
|
|
||||||
ackFile.setAckSide(ackSide);
|
|
||||||
ackFile.setAckStatus(ackStatus);
|
|
||||||
ackFile.setMessage(message);
|
|
||||||
ackFile.setProcessedAt(OffsetDateTime.now().toString());
|
|
||||||
return ackFile;
|
|
||||||
}
|
|
||||||
|
|
||||||
public String buildPackageFileName(SyncDirection direction, String sourceVersion, String traceId) {
|
public String buildPackageFileName(SyncDirection direction, String sourceVersion, String traceId) {
|
||||||
return direction.name().toLowerCase() + "-" + sanitize(sourceVersion) + "-" + sanitize(traceId) + ".zip";
|
return direction.name().toLowerCase() + "-" + sanitize(sourceVersion) + "-" + sanitize(traceId) + ".zip";
|
||||||
}
|
}
|
||||||
|
|
||||||
public String buildAckFileName(String traceId) {
|
|
||||||
return "ack-" + sanitize(traceId) + ".json";
|
|
||||||
}
|
|
||||||
|
|
||||||
private String sanitize(String value) {
|
private String sanitize(String value) {
|
||||||
if (value == null || value.trim().isEmpty()) {
|
if (value == null || value.trim().isEmpty()) {
|
||||||
return "unknown";
|
return "unknown";
|
||||||
|
|||||||
@ -1,19 +1,2 @@
|
|||||||
spring.config.activate.on-profile=dev-agent
|
# Retired profile.
|
||||||
server.port=8081
|
# The Git direct architecture no longer requires a dev-agent deployment.
|
||||||
|
|
||||||
sync.node-id=dev-agent-01
|
|
||||||
sync.role=DEV
|
|
||||||
|
|
||||||
# DEV side pulls Git, stages packages to FTP, and consumes prod snapshots
|
|
||||||
sync.jobs.dev-git-scan.cron=0 */2 * * * *
|
|
||||||
sync.jobs.dev-consume-prod-package.cron=30 */1 * * * *
|
|
||||||
sync.jobs.dev-ack-scan.cron=45 */1 * * * *
|
|
||||||
|
|
||||||
# Example overrides
|
|
||||||
ftp.host=ftp-a.example.com
|
|
||||||
ftp.port=21
|
|
||||||
ftp.username=dev_sync_user
|
|
||||||
ftp.password=change-me
|
|
||||||
git.repo.remote-uri=https://git.example.com/config.git
|
|
||||||
git.repo.scan-branch=config-dev-main
|
|
||||||
git.repo.snapshot-branch=config-prod-snapshot
|
|
||||||
|
|||||||
@ -4,16 +4,11 @@ server.port=8082
|
|||||||
sync.node-id=prod-agent-01
|
sync.node-id=prod-agent-01
|
||||||
sync.role=PROD
|
sync.role=PROD
|
||||||
|
|
||||||
# PROD side consumes dev packages, calls pull/push APIs, and stages snapshots
|
# PROD side directly pulls Git and synchronizes with production APIs
|
||||||
sync.jobs.prod-consume-dev-package.cron=0 */1 * * * *
|
sync.jobs.prod-git-to-prod.cron=0 */1 * * * *
|
||||||
sync.jobs.prod-pull-config.cron=20 */2 * * * *
|
sync.jobs.prod-to-git.cron=20 */2 * * * *
|
||||||
sync.jobs.prod-ack-scan.cron=40 */1 * * * *
|
|
||||||
|
|
||||||
# Example overrides
|
# Example overrides
|
||||||
ftp.host=ftp-a.example.com
|
|
||||||
ftp.port=21
|
|
||||||
ftp.username=prod_sync_user
|
|
||||||
ftp.password=change-me
|
|
||||||
prod.api.base-url=https://prod.example.com
|
prod.api.base-url=https://prod.example.com
|
||||||
prod.api.push-path=/api/config/push
|
prod.api.push-path=/api/config/push
|
||||||
prod.api.pull-path=/api/config/pull
|
prod.api.pull-path=/api/config/pull
|
||||||
|
|||||||
@ -1,10 +1,10 @@
|
|||||||
# Common application settings
|
# Common application settings
|
||||||
spring.application.name=ftp-sync-tool
|
spring.application.name=git-direct-sync-tool
|
||||||
server.port=8080
|
server.port=8080
|
||||||
spring.main.banner-mode=off
|
spring.main.banner-mode=off
|
||||||
|
|
||||||
# H2 file mode to persist checkpoints and retry state
|
# H2 file mode to persist checkpoints and retry state
|
||||||
spring.datasource.url=jdbc:h2:file:./data/ftp-sync-tool-db;AUTO_SERVER=TRUE;MODE=MYSQL
|
spring.datasource.url=jdbc:h2:file:./data/git-direct-sync-tool-db;AUTO_SERVER=TRUE;MODE=MYSQL
|
||||||
spring.datasource.driver-class-name=org.h2.Driver
|
spring.datasource.driver-class-name=org.h2.Driver
|
||||||
spring.datasource.username=sa
|
spring.datasource.username=sa
|
||||||
spring.datasource.password=
|
spring.datasource.password=
|
||||||
@ -26,25 +26,8 @@ sync.package-temp-dir=./work/package
|
|||||||
sync.dev-to-prod-staging-dir=./work/staging/dev-to-prod
|
sync.dev-to-prod-staging-dir=./work/staging/dev-to-prod
|
||||||
sync.prod-to-dev-staging-dir=./work/staging/prod-to-dev
|
sync.prod-to-dev-staging-dir=./work/staging/prod-to-dev
|
||||||
sync.max-retry-count=5
|
sync.max-retry-count=5
|
||||||
sync.ack-scan-batch-size=50
|
|
||||||
sync.remote-dev-to-prod-out-dir=/dev-to-prod/out
|
|
||||||
sync.remote-dev-to-prod-ack-dir=/dev-to-prod/ack
|
|
||||||
sync.remote-prod-to-dev-out-dir=/prod-to-dev/out
|
|
||||||
sync.remote-prod-to-dev-ack-dir=/prod-to-dev/ack
|
|
||||||
sync.remote-failed-dir=/failed
|
|
||||||
sync.pull-response-file-name=prod-config.json
|
sync.pull-response-file-name=prod-config.json
|
||||||
|
|
||||||
# FTP defaults
|
|
||||||
ftp.host=127.0.0.1
|
|
||||||
ftp.port=21
|
|
||||||
ftp.username=replace-me
|
|
||||||
ftp.password=replace-me
|
|
||||||
ftp.passive-mode=true
|
|
||||||
ftp.base-dir=/sync
|
|
||||||
ftp.connect-timeout-ms=10000
|
|
||||||
ftp.data-timeout-ms=20000
|
|
||||||
ftp.buffer-size=8192
|
|
||||||
|
|
||||||
# Git defaults
|
# Git defaults
|
||||||
git.repo.local-path=./work/git/config-repo
|
git.repo.local-path=./work/git/config-repo
|
||||||
git.repo.remote-uri=https://git.example.com/config.git
|
git.repo.remote-uri=https://git.example.com/config.git
|
||||||
@ -52,8 +35,8 @@ git.repo.username=replace-me
|
|||||||
git.repo.password=replace-me
|
git.repo.password=replace-me
|
||||||
git.repo.scan-branch=config-dev-main
|
git.repo.scan-branch=config-dev-main
|
||||||
git.repo.snapshot-branch=config-prod-snapshot
|
git.repo.snapshot-branch=config-prod-snapshot
|
||||||
git.repo.commit-author-name=ftp-sync-bot
|
git.repo.commit-author-name=git-sync-bot
|
||||||
git.repo.commit-author-email=ftp-sync-bot@example.com
|
git.repo.commit-author-email=git-sync-bot@example.com
|
||||||
git.repo.commit-message-prefix=sync(prod->git)
|
git.repo.commit-message-prefix=sync(prod->git)
|
||||||
git.repo.pull-rebase=false
|
git.repo.pull-rebase=false
|
||||||
|
|
||||||
|
|||||||
@ -25,14 +25,3 @@ create table if not exists sync_task (
|
|||||||
|
|
||||||
create index if not exists idx_sync_task_status on sync_task (status);
|
create index if not exists idx_sync_task_status on sync_task (status);
|
||||||
create index if not exists idx_sync_task_direction on sync_task (direction);
|
create index if not exists idx_sync_task_direction on sync_task (direction);
|
||||||
|
|
||||||
create table if not exists sync_ack (
|
|
||||||
id bigint generated by default as identity primary key,
|
|
||||||
trace_id varchar(64) not null,
|
|
||||||
ack_side varchar(32) not null,
|
|
||||||
ack_status varchar(32) not null,
|
|
||||||
ack_time timestamp not null,
|
|
||||||
remark varchar(500)
|
|
||||||
);
|
|
||||||
|
|
||||||
create index if not exists idx_sync_ack_trace on sync_ack (trace_id);
|
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user