diff --git a/doc_scripts/PAM_AUTO_DEPLY_SKILL.md b/doc_scripts/PAM_AUTO_DEPLY_SKILL.md index 9b5d5d6..713d92c 100644 --- a/doc_scripts/PAM_AUTO_DEPLY_SKILL.md +++ b/doc_scripts/PAM_AUTO_DEPLY_SKILL.md @@ -55,19 +55,19 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 ### 3.1 必填业务参数 -| 规范字段 | 脚本字段 | 必填 | 说明 | -| --- | --- | --- | --- | -| `HOME_BASE_URL` | `HOME_BASE_URL` | 是 | PAM HOME 基础地址 | -| `client_id` | `CLIENT_ID` | 是 | OAuth 客户端 ID | -| `client_secret` | `CLIENT_SECRET` | 是 | OAuth 客户端密钥 | -| `airportCode` | `AIRPORT_CODE` | 是 | 机场三字码 | -| `applicationName` | `APP_NAME` | 是 | 应用名 | -| `moduleName` | `MODULE_NAME` | 是 | 模块名 | -| `versionNumber` | `VERSION_NUMBER` | 是 | 目标版本号 | -| `zipFilePath` | `ZIP_FILE_PATH` | 是 | 本地软件包路径 | -| `actionType` | `ACTION_TYPE` | 否 | 升级类型，默认 `FULL` | -| `timeOut` | `TIMEOUT` | 否 | 接口级超时参数，默认 `120` | -| `logName` | `LOG_NAME` | 否 | 日志文件名，默认 `app.log` | +| 规范字段 | 脚本字段 | 必填 | 说明 | +| ----------------- | ---------------- | --- | ------------------ | +| `HOME_BASE_URL` | `HOME_BASE_URL` | 是 | PAM HOME 基础地址 | +| `client_id` | `CLIENT_ID` | 是 | OAuth 客户端 ID | +| `client_secret` | `CLIENT_SECRET` | 是 | OAuth 客户端密钥 | +| `airportCode` | `AIRPORT_CODE` | 是 | 机场三字码 | +| `applicationName` | `APP_NAME` | 是 | 应用名 | +| `moduleName` | `MODULE_NAME` | 是 | 模块名 | +| `versionNumber` | `VERSION_NUMBER` | 是 | 目标版本号 | +| `zipFilePath` | `ZIP_FILE_PATH` | 是 | 本地软件包路径 | +| `actionType` | `ACTION_TYPE` | 否 | 升级类型，默认 `FULL` | +| `timeOut` | `TIMEOUT` | 否 | 接口级超时参数，默认 `120` | +| `logName` | `LOG_NAME` | 否 | 日志文件名，默认 `app.log` | ### 3.2 运行控制参数 @@ -196,10 +196,11 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 14. 调用 `create-download-task`。 15. 调用 `poll-download-progress`，直到下载完成、失败或超时。 16. 按在线 IP 或过滤后的目标 IP 列表逐台执行： - - `upgrade-ip` - - `start-ip` - - `verify-ip` - - `download-log` + - `upgrade-ip` + - `poll-upgrade-progress` + - `start-ip` + - `verify-ip` + - `download-log` 17. 汇总每台 IP 的结果。 18. 若出现 `PENDING_AGENT_CONFIRMATION(...)`，立即中止自动后续动作，转入回滚确认分支。 19. 输出最终报告。 @@ -210,10 +211,11 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 2. 全局 action 与下一 action 之间，按 `stepIntervalSec` 等待。 3. `create-download-task` 成功后，到首次 `poll-download-progress` 前，按 `firstPollDelaySec` 等待。 4. 同一台 IP 内部： - - `upgrade-ip -> start-ip` + - `upgrade-ip -> poll-upgrade-progress` + - `poll-upgrade-progress -> start-ip` - `start-ip -> verify-ip` - `verify-ip -> download-log` - 之间按 `perIpStepIntervalSec` 等待。 + 之间按 `perIpStepIntervalSec` 等待。 5. 当前一台 IP 处理完成后，到下一台 IP 开始前，按 `perIpIntervalSec` 等待。 6. 若某步骤失败后需要进入提示、确认或分支流程，可按 `failurePauseSec` 等待。 7. 若某个间隔值为 `0`，表示该层级不等待，直接进入下一动作。 @@ -323,7 +325,8 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 - 在线 IP 列表 - 过滤后的目标 IP 列表 - 每台 IP 的执行状态： - - `upgrade` + - `upgrade-ip` + - `poll-upgrade-progress` - `start` - `verify` - `download-log` @@ -411,30 +414,31 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 ### 4.5 主流程逐步说明 -| 步骤 | 目标 | 调用或动作 | 成功判定 | 失败处理 | -| --- | --- | --- | --- | --- | -| 1 | 识别意图 | 判断是否为真实部署 | 意图明确为真实部署 | 若不是，转入分支流程，不执行真实部署 | -| 2 | 归一化参数 | 整理用户输入、补齐默认值 | 参数结构完整 | 参数不清时先追问，不猜测 | -| 3 | 参数确认 | 输出参数确认单 | 用户明确确认 | 未确认前停止 | -| 4 | 检查脚本 | 检查 `deploy.sh` / `deploy.ps1` 是否存在且可用 | 至少存在一个与当前 OS 对应的脚本入口 | 缺失或不可用时停止并说明 | -| 5 | 选择入口 | 根据 OS 选择 `deploy.sh` 或 `deploy.ps1` | 入口唯一且明确 | 入口不明确时停止 | -| 6 | 落地配置 | 将稳定业务参数写入 `config.txt` | `config.txt` 已生成且内容与确认单一致 | 写入失败则停止 | -| 7 | 获取 Token | `get-token` | 返回 `TOKEN=...` 且非空 | 停止并报告 `TOKEN` 阶段失败 | -| 8 | 创建版本 | `create-version` | 返回 `RESULT=OK` | 停止并报告 `CREATE_VERSION` 失败 | -| 9 | 上传软件包 | `upload-package` | 返回 `HASH_CODE=...` 且非空 | 停止并报告 `UPLOAD_PACKAGE` 失败 | -| 10 | 发布版本 | `publish-version --hash-code ...` | 返回 `RESULT=OK` | 停止并报告 `PUBLISH_VERSION` 失败 | -| 11 | 获取 Node | `get-node-url` | 返回 `NODE_URL=...` 且非空 | 停止并报告 `GET_NODE_URL` 失败 | -| 12 | 获取在线 IP | `get-online-ips` | 返回 `COUNT>0` 且有 `IP=...` 行 | 停止并报告 `GET_ONLINE_IPS` 失败 | -| 13 | 过滤目标 IP | 按用户指定 IP 与在线 IP 交集过滤 | 过滤结果明确 | 过滤后为空时停止；范围变化需确认 | -| 14 | 创建云下载任务 | `create-download-task` | 返回 `RESULT=TASK_CREATED` | 停止并报告 `CREATE_DOWNLOAD_TASK` 失败 | -| 15 | 轮询下载进度 | `poll-download-progress` | `STEP=DONE` 或 `MSG=success` 且 `RATE_OF_PROGRESS=100` | 停止并报告 `POLL_DOWNLOAD_PROGRESS` 失败或超时 | -| 16.1 | 升级单 IP | `upgrade-ip --ip ...` | 返回 `SUCCESS=true` | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=false)` | -| 16.2 | 启动单 IP | `start-ip --ip ...` | action 成功返回 | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=true)` | -| 16.3 | 校验单 IP | `verify-ip --ip ...` | 返回 `SUCCESS=true` | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=true)` | -| 16.4 | 下载日志 | `download-log --ip ...` | 返回 `LOG_FILE=...` | 记录日志下载失败，但不覆盖原主失败原因 | -| 17 | 汇总结果 | 汇总每台 IP 的阶段、失败原因、回滚状态、日志路径 | 报告内容完整 | 若汇总失败，至少保留原始 action 输出 | -| 18 | 回滚确认分支 | 发现 `PENDING_AGENT_CONFIRMATION(...)` 时进入回滚确认 | 用户明确是否回滚 | 未确认时停止，不自动回滚 | -| 19 | 最终报告 | 输出最终报告 | 报告包含模式、入口、阶段结果、日志、回滚状态 | 不省略失败细节 | +| 步骤 | 目标 | 调用或动作 | 成功判定 | 失败处理 | +| ---- | ----------- | -------------------------------------------- | -------------------------------------------------------------------- | ----------------------------------------------------- | +| 1 | 识别意图 | 判断是否为真实部署 | 意图明确为真实部署 | 若不是，转入分支流程，不执行真实部署 | +| 2 | 归一化参数 | 整理用户输入、补齐默认值 | 参数结构完整 | 参数不清时先追问，不猜测 | +| 3 | 参数确认 | 输出参数确认单 | 用户明确确认 | 未确认前停止 | +| 4 | 检查脚本 | 检查 `deploy.sh` / `deploy.ps1` 是否存在且可用 | 至少存在一个与当前 OS 对应的脚本入口 | 缺失或不可用时停止并说明 | +| 5 | 选择入口 | 根据 OS 选择 `deploy.sh` 或 `deploy.ps1` | 入口唯一且明确 | 入口不明确时停止 | +| 6 | 落地配置 | 将稳定业务参数写入 `config.txt` | `config.txt` 已生成且内容与确认单一致 | 写入失败则停止 | +| 7 | 获取 Token | `get-token` | 返回 `TOKEN=...` 且非空 | 停止并报告 `TOKEN` 阶段失败 | +| 8 | 创建版本 | `create-version` | 返回 `RESULT=OK` | 停止并报告 `CREATE_VERSION` 失败 | +| 9 | 上传软件包 | `upload-package` | 返回 `HASH_CODE=...` 且非空 | 停止并报告 `UPLOAD_PACKAGE` 失败 | +| 10 | 发布版本 | `publish-version --hash-code ...` | 返回 `RESULT=OK` | 停止并报告 `PUBLISH_VERSION` 失败 | +| 11 | 获取 Node | `get-node-url` | 返回 `NODE_URL=...` 且非空 | 停止并报告 `GET_NODE_URL` 失败 | +| 12 | 获取在线 IP | `get-online-ips` | 返回 `COUNT>0` 且有 `IP=...` 行 | 停止并报告 `GET_ONLINE_IPS` 失败 | +| 13 | 过滤目标 IP | 按用户指定 IP 与在线 IP 交集过滤 | 过滤结果明确 | 过滤后为空时停止；范围变化需确认 | +| 14 | 创建云下载任务 | `create-download-task` | 返回 `RESULT=TASK_CREATED` | 停止并报告 `CREATE_DOWNLOAD_TASK` 失败 | +| 15 | 轮询下载进度 | `poll-download-progress` | `STEP=DONE` 或 `MSG=success` 且 `RATE_OF_PROGRESS=100` | 停止并报告 `POLL_DOWNLOAD_PROGRESS` 失败或超时 | +| 16.1 | 创建单 IP 推送任务 | `upgrade-ip --ip ...` | 返回 `RESULT=TASK_CREATED` | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=false)` | +| 16.2 | 轮询单 IP 推送进度 | `poll-upgrade-progress --ip ...` | `STEP=DONE` 或 `FINISH=true` 或 `MSG=success` 且 `RATE_OF_PROGRESS=100` | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=false)` | +| 16.3 | 启动单 IP | `start-ip --ip ...` | action 成功返回 | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=true)` | +| 16.4 | 校验单 IP | `verify-ip --ip ...` | 返回 `SUCCESS=true` | 记录失败，标记 `PENDING_AGENT_CONFIRMATION(stopFirst=true)` | +| 16.5 | 下载日志 | `download-log --ip ...` | 返回 `LOG_FILE=...` | 记录日志下载失败，但不覆盖原主失败原因 | +| 17 | 汇总结果 | 汇总每台 IP 的阶段、失败原因、回滚状态、日志路径 | 报告内容完整 | 若汇总失败，至少保留原始 action 输出 | +| 18 | 回滚确认分支 | 发现 `PENDING_AGENT_CONFIRMATION(...)` 时进入回滚确认 | 用户明确是否回滚 | 未确认时停止，不自动回滚 | +| 19 | 最终报告 | 输出最终报告 | 报告包含模式、入口、阶段结果、日志、回滚状态 | 不省略失败细节 | ## 5. 通用执行原则 @@ -461,7 +465,7 @@ description: 面向 PAM HOME/NODE 的智能部署 Skill。由 Skill 负责理解 ### 6.1 Shell 入口 ```bash -bash ./deploy.sh --config ./config.txt --action [--ip 192.168.1.10] [--hash-code xxx] [--stop-first] +bash ./deploy.sh --config ./config.txt --action [--ip 192.168.1.10] [--hash-code xxx] [--stop-first] [--trace-file ./logs/api_trace_xxx.log] ``` ### 6.2 PowerShell 入口 @@ -472,23 +476,24 @@ powershell -File .\deploy.ps1 -ConfigPath .\config.txt -Action [-Ip ### 6.3 可用 action -| action | 用途 | 额外参数 | -| --- | --- | --- | -| `get-token` | 获取访问令牌 | 无 | -| `create-version` | 新建版本记录 | 无 | -| `upload-package` | 上传软件包 | 无 | -| `publish-version` | 发布版本 | `--hash-code` / `-HashCode` | -| `get-node-url` | 获取目标 Node 地址 | 无 | -| `get-online-ips` | 获取在线工作站 IP 列表 | 无 | -| `create-download-task` | 创建云下载任务 | 无 | -| `poll-download-progress` | 轮询下载进度 | 无 | -| `download-cloud-to-node` | 创建下载任务并轮询至完成，仅调试使用，不得进入正式主流程 | 无 | -| `upgrade-ip` | 升级指定 IP | `--ip` / `-Ip` | -| `start-ip` | 启动指定 IP 应用 | `--ip` / `-Ip` | -| `stop-ip` | 停止指定 IP 应用 | `--ip` / `-Ip` | -| `verify-ip` | 校验指定 IP | `--ip` / `-Ip` | -| `download-log` | 下载指定 IP 日志压缩包，返回 zip 文件路径 | `--ip` / `-Ip` | -| `rollback-ip` | 执行指定 IP 回滚 | `--ip` / `-Ip`，可带 `--stop-first` / `-RollbackStopFirst` | +| action | 用途 | 额外参数 | +| ------------------------ | ------------------------------ | ------------------------------------------------------- | +| `get-token` | 获取访问令牌 | 无 | +| `create-version` | 新建版本记录 | 无 | +| `upload-package` | 上传软件包 | 无 | +| `publish-version` | 发布版本 | `--hash-code` / `-HashCode` | +| `get-node-url` | 获取目标 Node 地址 | 无 | +| `get-online-ips` | 获取在线工作站 IP 列表 | 无 | +| `create-download-task` | 创建云下载任务 | 无 | +| `poll-download-progress` | 轮询下载进度 | 无 | +| `download-cloud-to-node` | 创建下载任务并轮询至完成，仅调试使用，不得进入正式主流程 | 无 | +| `upgrade-ip` | 为指定 IP 创建推送任务，固定使用 `timeOut=0` | `--ip` / `-Ip` | +| `poll-upgrade-progress` | 轮询指定 IP 的推送进度 | `--ip` / `-Ip` | +| `start-ip` | 启动指定 IP 应用 | `--ip` / `-Ip` | +| `stop-ip` | 停止指定 IP 应用 | `--ip` / `-Ip` | +| `verify-ip` | 校验指定 IP | `--ip` / `-Ip` | +| `download-log` | 下载指定 IP 日志压缩包，返回 zip 文件路径 | `--ip` / `-Ip` | +| `rollback-ip` | 执行指定 IP 回滚 | `--ip` / `-Ip`，可带 `--stop-first` / `-RollbackStopFirst` | ### 6.4 action 输出约定 @@ -548,11 +553,11 @@ Agent 读取时： 7. 输出 Node 地址、在线 IP 数量和 IP 列表。 8. 若需要间隔等待，也要向用户播报当前等待状态。 9. 不执行： - - `create-version` - - `upload-package` - - `publish-version` - - `create-download-task` - - `upgrade-ip` + - `create-version` + - `upload-package` + - `publish-version` + - `create-download-task` + - `upgrade-ip` ### 7.4 手动回滚分支 diff --git a/doc_scripts/PAM智能部署 Shell & Bat 脚本实现.md.md b/doc_scripts/PAM智能部署 Shell & Bat 脚本实现.md.md index 9812731..5db1092 100644 --- a/doc_scripts/PAM智能部署 Shell & Bat 脚本实现.md.md +++ b/doc_scripts/PAM智能部署 Shell & Bat 脚本实现.md.md @@ -30,8 +30,9 @@ - 仅在用户明确要求时再提供 `deploy.bat` 7. NODE 侧接口路径统一使用 `node-proxy`；`download-cloud/progress` 需额外携带 `versionNumber`，并以异步轮询方式持续展示下载进度。 8. `download-cloud/progress` 的完成判定优先读取 `msg`、`step`、`rateOfProgress`；当 `msg=success`、`step=DONE`、`rateOfProgress=100` 时代表下载完成，其中 `rateOfProgress` 即下载进度值。 -9. 正式部署脚本不会自动执行回滚；发现需要回滚时，只输出 `PENDING_AGENT_CONFIRMATION(stopFirst=...)`，由 Agent 先和用户确认，再调用手动回滚入口。 -10. `POST /api/mcp/version/upgrade` 和 `POST /api/mcp/version/upgrade/start-stop` 的业务参数都直接放在 URL query 中，不再使用 body 表单；启停接口参数名使用 `runStart`；`download-cloud` 固定传 `timeOut=0` 创建任务。 +9. `upgrade-ip` 固定传 `timeOut=0`，只负责创建推送任务；后续必须通过 `upgrade/progress` 异步轮询指定 IP 的推送进度。 +10. 正式部署脚本不会自动执行回滚；发现需要回滚时，只输出 `PENDING_AGENT_CONFIRMATION(stopFirst=...)`，由 Agent 先和用户确认，再调用手动回滚入口。 +11. `POST /api/mcp/version/upgrade` 和 `POST /api/mcp/version/upgrade/start-stop` 的业务参数都直接放在 URL query 中，不再使用 body 表单；启停接口参数名使用 `runStart`；`download-cloud` 固定传 `timeOut=0` 创建任务。 11. 脚本同时提供主流程入口与 `action` 入口；建议 Agent 优先调用 `action` 入口，由 Skill 负责主流程编排。 12. 当前目录中的真实 `deploy.sh` 已去除 `jq` 依赖，统一使用 Bash 原生兼容 JSON 解析；若本文中的历史代码块仍出现 `jq`，以真实脚本文件为准。 @@ -748,6 +749,7 @@ bash ./deploy.sh --config ./config.txt --action get-online-ips bash ./deploy.sh --config ./config.txt --action create-download-task bash ./deploy.sh --config ./config.txt --action poll-download-progress bash ./deploy.sh --config ./config.txt --action upgrade-ip --ip 192.168.1.10 +bash ./deploy.sh --config ./config.txt --action poll-upgrade-progress --ip 192.168.1.10 ``` ```powershell diff --git a/doc_scripts/deploy.ps1 b/doc_scripts/deploy.ps1 index 4af07cf..690d8fe 100644 --- a/doc_scripts/deploy.ps1 +++ b/doc_scripts/deploy.ps1 @@ -39,6 +39,18 @@ $script:DownloadProgressState = [ordered]@{ RateOfProgress = '' RawResponse = '' } +$script:UpgradeProgressState = [ordered]@{ + Status = '' + Success = '' + Step = '' + Msg = '' + Message = '' + RateOfProgress = '' + Code = '' + Finish = '' + LastModify = '' + RawResponse = '' +} function Write-ResultLine([string]$Key, [AllowEmptyString()][string]$Value) { if ($null -eq $Value) { @@ -78,6 +90,20 @@ function Write-DownloadProgressResult([string]$ActionName = 'poll-download-progr Write-ResultLine -Key 'MESSAGE' -Value ([string]$script:DownloadProgressState.Message) } +function Write-UpgradeProgressResult([string]$Ip) { + Write-ResultLine -Key 'ACTION' -Value 'poll-upgrade-progress' + Write-ResultLine -Key 'IP' -Value $Ip + Write-ResultLine -Key 'STEP' -Value ([string]$script:UpgradeProgressState.Step) + Write-ResultLine -Key 'MSG' -Value ([string]$script:UpgradeProgressState.Msg) + Write-ResultLine -Key 'RATE_OF_PROGRESS' -Value ([string]$script:UpgradeProgressState.RateOfProgress) + Write-ResultLine -Key 'STATUS' -Value ([string]$script:UpgradeProgressState.Status) + Write-ResultLine -Key 'SUCCESS' -Value ([string]$script:UpgradeProgressState.Success) + Write-ResultLine -Key 'CODE' -Value ([string]$script:UpgradeProgressState.Code) + Write-ResultLine -Key 'FINISH' -Value ([string]$script:UpgradeProgressState.Finish) + Write-ResultLine -Key 'LAST_MODIFY' -Value ([string]$script:UpgradeProgressState.LastModify) + Write-ResultLine -Key 'MESSAGE' -Value ([string]$script:UpgradeProgressState.Message) +} + function Write-FlowStart([string]$Name, [string]$Detail = '') { if ($Detail) { Write-Info "[FLOW][START] $Name | $Detail" @@ -233,6 +259,78 @@ function Get-ResponseValue { return $null } +function Get-PrimaryResponseMessage { + param($Response) + + $message = Get-ResponseValue -Response $Response -Candidates @('message') + if (-not $message) { + $message = Get-ResponseValue -Response $Response -Candidates @('msg') + } + + return $message +} + +function Test-ResponseFailure { + param($Response) + + $successFlag = Get-ResponseValue -Response $Response -Candidates @('success') + $code = Get-ResponseValue -Response $Response -Candidates @('code') + $status = Get-ResponseValue -Response $Response -Candidates @('status') + $message = Get-PrimaryResponseMessage -Response $Response + + if ($successFlag -eq 'false') { + return $true + } + + if ($code -and $code -ne '0') { + return $true + } + + if ((@($status, $message) -join ' ') -match '(?i)fail|error') { + return $true + } + + return $false +} + +function Get-ScopedResponseObject { + param( + $Response, + [string]$ScopeKey + ) + + if (-not $ScopeKey -or $null -eq $Response -or $Response -is [string]) { + return $Response + } + + if ($Response -is [System.Collections.IDictionary]) { + if ($Response.Contains($ScopeKey)) { + return $Response[$ScopeKey] + } + return $Response + } + + $property = $Response.PSObject.Properties | Where-Object { $_.Name -eq $ScopeKey } | Select-Object -First 1 + if ($property) { + return $property.Value + } + + return $Response +} + +function Get-ProgressStateMessage { + param( + [System.Collections.IDictionary]$State, + [string]$DefaultMessage + ) + + $message = [string]$State.Message + if (-not $message) { $message = [string]$State.Msg } + if (-not $message) { $message = [string]$State.Step } + if (-not $message) { $message = $DefaultMessage } + return $message +} + function Get-PamConfig { param([string]$Path) @@ -626,6 +724,109 @@ function Download-CloudToNode { Wait-DownloadProgress -Config $Config -Token $Token -NodeUrl $NodeUrl } +function Wait-UpgradeProgress { + param( + $Config, + [string]$Token, + [string]$NodeUrl, + [string]$Ip + ) + + $query = Join-RequestPairs ([ordered]@{ + applicationName = $Config.APP_NAME + moduleName = $Config.MODULE_NAME + airportCode = $Config.AIRPORT_CODE + versionNumber = $Config.VERSION_NUMBER + }) + $progressUrl = "$($Config.HOME_BASE_URL)/node-proxy/$($Config.AIRPORT_CODE)/api/mcp/version/upgrade/progress?$query" + $script:UpgradeProgressState = [ordered]@{ + Status = '' + Success = '' + Step = '' + Msg = '' + Message = '' + RateOfProgress = '' + Code = '' + Finish = '' + LastModify = '' + RawResponse = '' + } + + for ($attempt = 0; $attempt -lt 60; $attempt++) { + $response = Invoke-PamWebRequest -Method GET -Url $progressUrl -Token $Token -Headers @{ + 'Target-Node' = $NodeUrl + } + $progressResponse = Get-ScopedResponseObject -Response $response -ScopeKey $Ip + + $status = Get-ResponseValue -Response $progressResponse -Candidates @('status') + $successFlag = Get-ResponseValue -Response $progressResponse -Candidates @('success') + $step = Get-ResponseValue -Response $progressResponse -Candidates @('step') + $msg = Get-ResponseValue -Response $progressResponse -Candidates @('msg') + $progressValue = Get-ResponseValue -Response $progressResponse -Candidates @('rateOfProgress', 'progress', 'percent', 'data.rateOfProgress', 'data.progress', 'data.percent') + $message = Get-ResponseValue -Response $progressResponse -Candidates @('message') + $code = Get-ResponseValue -Response $progressResponse -Candidates @('code') + $finish = Get-ResponseValue -Response $progressResponse -Candidates @('finish') + $lastModify = Get-ResponseValue -Response $progressResponse -Candidates @('lastModify') + if (-not $message) { $message = $msg } + + $script:UpgradeProgressState = [ordered]@{ + Status = [string]$status + Success = [string]$successFlag + Step = [string]$step + Msg = [string]$msg + Message = [string]$message + RateOfProgress = [string]$progressValue + Code = [string]$code + Finish = [string]$finish + LastModify = [string]$lastModify + RawResponse = [string]$response + } + + $progressParts = [System.Collections.Generic.List[string]]::new() + $progressParts.Add("ip=$Ip") + if ($msg) { $progressParts.Add("msg=$msg") } + if ($step) { $progressParts.Add("step=$step") } + if ($progressValue) { $progressParts.Add("rateOfProgress=$progressValue") } + if ($code) { $progressParts.Add("code=$code") } + if ($finish) { $progressParts.Add("finish=$finish") } + if ($status) { $progressParts.Add("status=$status") } + if ($successFlag) { $progressParts.Add("success=$successFlag") } + if ($lastModify) { $progressParts.Add("lastModify=$lastModify") } + if ($message -and $message -ne $msg) { $progressParts.Add("message=$message") } + + if ($progressParts.Count -gt 1) { + Write-Info ("Step 3.4a: async upgrade progress -> {0}" -f ($progressParts -join ', ')) + } else { + Write-Info ("Step 3.4a: async upgrade progress polling... ip={0} ({1}/60)" -f $Ip, ($attempt + 1)) + } + + if ($step -eq 'DONE' -or $finish -eq 'true' -or $status -eq 'completed' -or $successFlag -eq 'true') { + return + } + + if (($msg -eq 'success') -and ($progressValue -eq '100') -and ((-not $code) -or $code -eq '0')) { + return + } + + if ($code -and $code -ne '0') { + if (-not $message) { $message = $msg } + if (-not $message) { $message = $step } + if (-not $message) { $message = "code=$code" } + throw "Node upgrade failed: ip=$Ip, message=$message" + } + + if ((@($step, $message, $msg, $status) -join ' ') -match '(?i)fail|error') { + if (-not $message) { $message = $step } + if (-not $message) { $message = $msg } + throw "Node upgrade failed: ip=$Ip, message=$message" + } + + Start-Sleep -Seconds 2 + } + + throw "Node upgrade timed out: ip=$Ip" +} + function Invoke-UpgradeRequest { param($Config, [string]$Token, [string]$NodeUrl, [string]$Ip) @@ -637,7 +838,7 @@ function Invoke-UpgradeRequest { versionNumber = $Config.VERSION_NUMBER action = $Config.ACTION_TYPE autoStart = 'false' - timeOut = $Config.TIMEOUT + timeOut = '0' }) Invoke-PamWebRequest -Method POST -Url "$($Config.HOME_BASE_URL)/node-proxy/$($Config.AIRPORT_CODE)/api/mcp/version/upgrade?$query" -Token $Token -Headers @{ @@ -802,9 +1003,9 @@ function Invoke-IpDeploy { } } - if ((Get-ResponseValue -Response $upgrade -Candidates @('success')) -ne 'true') { - $message = Get-ResponseValue -Response $upgrade -Candidates @('message') - if (-not $message) { $message = 'Upgrade failed' } + if (Test-ResponseFailure -Response $upgrade) { + $message = Get-PrimaryResponseMessage -Response $upgrade + if (-not $message) { $message = 'Upgrade task creation failed' } $rollback = Get-PendingRollbackStatus -Ip $Ip -Stage 'UPGRADE' -StopFirst:$false -Reason $message $logFile = Invoke-FlowStep -Name "Download-DeployLog[$Ip]" -Action { Download-DeployLog -Config $Config -Token $Token -NodeUrl $NodeUrl -Ip $Ip @@ -819,6 +1020,26 @@ function Invoke-IpDeploy { } } + try { + Invoke-FlowStep -Name "Wait-UpgradeProgress[$Ip]" -Action { + Wait-UpgradeProgress -Config $Config -Token $Token -NodeUrl $NodeUrl -Ip $Ip + } | Out-Null + } catch { + $message = Get-ProgressStateMessage -State $script:UpgradeProgressState -DefaultMessage 'Upgrade progress polling failed' + $rollback = Get-PendingRollbackStatus -Ip $Ip -Stage 'UPGRADE_PROGRESS' -StopFirst:$false -Reason $message + $logFile = Invoke-FlowStep -Name "Download-DeployLog[$Ip]" -Action { + Download-DeployLog -Config $Config -Token $Token -NodeUrl $NodeUrl -Ip $Ip + } + return [pscustomobject]@{ + Ip = $Ip + Status = 'FAILED' + Stage = 'UPGRADE_PROGRESS' + Message = $message + Rollback = $rollback + LogFile = $logFile + } + } + try { Invoke-FlowStep -Name "Start-Application[$Ip]" -Action { Start-Application -Config $Config -Token $Token -NodeUrl $NodeUrl -Ip $Ip @@ -1062,15 +1283,29 @@ function Invoke-PamAction { Write-DownloadProgressResult -ActionName 'download-cloud-to-node' Write-ResultLine -Key 'RESULT' -Value 'DONE' } + 'poll-upgrade-progress' { + Require-IpArgument -TargetIp $Ip + $token = Invoke-FlowStep -Name 'Get-Token' -Action { Get-Token -Config $config } + $nodeUrl = Invoke-FlowStep -Name 'Get-NodeUrl' -Action { Get-NodeUrl -Config $config -Token $token } + Invoke-FlowStep -Name "Wait-UpgradeProgress[$Ip]" -Action { Wait-UpgradeProgress -Config $config -Token $token -NodeUrl $nodeUrl -Ip $Ip } | Out-Null + Write-UpgradeProgressResult -Ip $Ip + } 'upgrade-ip' { Require-IpArgument -TargetIp $Ip $token = Invoke-FlowStep -Name 'Get-Token' -Action { Get-Token -Config $config } $nodeUrl = Invoke-FlowStep -Name 'Get-NodeUrl' -Action { Get-NodeUrl -Config $config -Token $token } $response = Invoke-FlowStep -Name "Invoke-UpgradeRequest[$Ip]" -Action { Invoke-UpgradeRequest -Config $config -Token $token -NodeUrl $nodeUrl -Ip $Ip } + if (Test-ResponseFailure -Response $response) { + $message = Get-PrimaryResponseMessage -Response $response + if (-not $message) { $message = 'Upgrade task creation failed' } + throw "Upgrade task creation failed: ip=$Ip, message=$message" + } Write-ResultLine -Key 'ACTION' -Value 'upgrade-ip' Write-ResultLine -Key 'IP' -Value $Ip + Write-ResultLine -Key 'TIME_OUT' -Value '0' + Write-ResultLine -Key 'RESULT' -Value 'TASK_CREATED' Write-ResultLine -Key 'SUCCESS' -Value (Get-ResponseValue -Response $response -Candidates @('success')) - Write-ResultLine -Key 'MESSAGE' -Value (Get-ResponseValue -Response $response -Candidates @('message')) + Write-ResultLine -Key 'MESSAGE' -Value (Get-PrimaryResponseMessage -Response $response) Write-ResultLine -Key 'RAW_RESPONSE' -Value ([string]$response) } 'start-ip' { diff --git a/doc_scripts/deploy.sh b/doc_scripts/deploy.sh index e175a3e..8da979d 100644 --- a/doc_scripts/deploy.sh +++ b/doc_scripts/deploy.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -# PAM 部署主脚本（Shell 入口）。 +# PAM deployment main script (Shell entry). set -uo pipefail @@ -27,6 +27,16 @@ DOWNLOAD_PROGRESS_MSG="" DOWNLOAD_PROGRESS_MESSAGE="" DOWNLOAD_PROGRESS_RATE="" DOWNLOAD_PROGRESS_RESPONSE="" +UPGRADE_PROGRESS_STATUS="" +UPGRADE_PROGRESS_SUCCESS="" +UPGRADE_PROGRESS_STEP="" +UPGRADE_PROGRESS_MSG="" +UPGRADE_PROGRESS_MESSAGE="" +UPGRADE_PROGRESS_RATE="" +UPGRADE_PROGRESS_CODE="" +UPGRADE_PROGRESS_FINISH="" +UPGRADE_PROGRESS_LAST_MODIFY="" +UPGRADE_PROGRESS_RESPONSE="" usage() { cat <<'EOF' @@ -437,6 +447,21 @@ print_progress_result() { result_line "MESSAGE" "$DOWNLOAD_PROGRESS_MESSAGE" } +print_upgrade_progress_result() { + local ip="$1" + result_line "ACTION" "poll-upgrade-progress" + result_line "IP" "$ip" + result_line "STEP" "$UPGRADE_PROGRESS_STEP" + result_line "MSG" "$UPGRADE_PROGRESS_MSG" + result_line "RATE_OF_PROGRESS" "$UPGRADE_PROGRESS_RATE" + result_line "STATUS" "$UPGRADE_PROGRESS_STATUS" + result_line "SUCCESS" "$UPGRADE_PROGRESS_SUCCESS" + result_line "CODE" "$UPGRADE_PROGRESS_CODE" + result_line "FINISH" "$UPGRADE_PROGRESS_FINISH" + result_line "LAST_MODIFY" "$UPGRADE_PROGRESS_LAST_MODIFY" + result_line "MESSAGE" "$UPGRADE_PROGRESS_MESSAGE" +} + json_value() { local input="$1" local query="$2" @@ -454,6 +479,15 @@ json_value() { '.message') json_get_string_by_key "$input" "message" ;; + '.code') + json_get_scalar_by_key "$input" "code" + ;; + '.finish') + json_get_scalar_by_key "$input" "finish" + ;; + '.lastModify') + json_get_scalar_by_key "$input" "lastModify" + ;; '.msg') json_get_string_by_key "$input" "msg" ;; @@ -503,6 +537,11 @@ json_compact() { printf '%s' "$input" } +regex_escape_ere() { + local text="$1" + printf '%s' "$text" | sed -E 's/[][(){}.^$*+?|\\]/\\&/g' +} + json_unescape_basic() { local value="$1" value="${value//\\\\/\\}" @@ -518,9 +557,11 @@ json_get_string_by_key() { local key="$2" local compact local value + local pattern_key compact="$(json_compact "$input")" - value="$(printf '%s' "$compact" | sed -nE 's/.*"'$key'"[[:space:]]*:[[:space:]]*"(([^"\\]|\\.)*)".*/\1/p')" + pattern_key="$(regex_escape_ere "$key")" + value="$(printf '%s' "$compact" | sed -nE 's/.*"'$pattern_key'"[[:space:]]*:[[:space:]]*"(([^"\\]|\\.)*)".*/\1/p')" [[ -n "$value" ]] && json_unescape_basic "$value" } @@ -529,9 +570,11 @@ json_get_scalar_by_key() { local key="$2" local compact local value + local pattern_key compact="$(json_compact "$input")" - value="$(printf '%s' "$compact" | sed -nE 's/.*"'$key'"[[:space:]]*:[[:space:]]*([^,}]+).*/\1/p')" + pattern_key="$(regex_escape_ere "$key")" + value="$(printf '%s' "$compact" | sed -nE 's/.*"'$pattern_key'"[[:space:]]*:[[:space:]]*([^,}]+).*/\1/p')" value="$(trim "$value")" value="${value%\"}" value="${value#\"}" @@ -544,12 +587,35 @@ json_get_nested_string_by_key() { local child_key="$3" local compact local value + local pattern_parent_key + local pattern_child_key compact="$(json_compact "$input")" - value="$(printf '%s' "$compact" | sed -nE 's/.*"'$parent_key'"[[:space:]]*:[[:space:]]*\{[^}]*"'$child_key'"[[:space:]]*:[[:space:]]*"(([^"\\]|\\.)*)".*/\1/p')" + pattern_parent_key="$(regex_escape_ere "$parent_key")" + pattern_child_key="$(regex_escape_ere "$child_key")" + value="$(printf '%s' "$compact" | sed -nE 's/.*"'$pattern_parent_key'"[[:space:]]*:[[:space:]]*\{[^}]*"'$pattern_child_key'"[[:space:]]*:[[:space:]]*"(([^"\\]|\\.)*)".*/\1/p')" [[ -n "$value" ]] && json_unescape_basic "$value" } +json_get_nested_scalar_by_key() { + local input="$1" + local parent_key="$2" + local child_key="$3" + local compact + local value + local pattern_parent_key + local pattern_child_key + + compact="$(json_compact "$input")" + pattern_parent_key="$(regex_escape_ere "$parent_key")" + pattern_child_key="$(regex_escape_ere "$child_key")" + value="$(printf '%s' "$compact" | sed -nE 's/.*"'$pattern_parent_key'"[[:space:]]*:[[:space:]]*\{[^}]*"'$pattern_child_key'"[[:space:]]*:[[:space:]]*([^,}]+).*/\1/p')" + value="$(trim "$value")" + value="${value%\"}" + value="${value#\"}" + printf '%s' "$value" +} + json_first_key() { local input="$1" @@ -572,6 +638,94 @@ json_array_items() { | sed -E 's/^[[:space:]]*"//; s/"[[:space:]]*$//; s/"[[:space:]]*,[[:space:]]*"/\n/g' } +json_scoped_value() { + local input="$1" + local scope_key="$2" + local query="$3" + local value="" + + if [[ -n "$scope_key" ]]; then + case "$query" in + '.status') + value="$(json_get_nested_string_by_key "$input" "$scope_key" "status")" + ;; + '.success') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "success")" + ;; + '.message') + value="$(json_get_nested_string_by_key "$input" "$scope_key" "message")" + ;; + '.msg') + value="$(json_get_nested_string_by_key "$input" "$scope_key" "msg")" + ;; + '.step') + value="$(json_get_nested_string_by_key "$input" "$scope_key" "step")" + ;; + '.rateOfProgress') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "rateOfProgress")" + ;; + '.progress') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "progress")" + ;; + '.percent') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "percent")" + ;; + '.code') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "code")" + ;; + '.finish') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "finish")" + ;; + '.lastModify') + value="$(json_get_nested_scalar_by_key "$input" "$scope_key" "lastModify")" + ;; + esac + fi + + if [[ -n "$value" ]]; then + printf '%s' "$value" + else + json_value "$input" "$query" + fi +} + +response_message() { + local input="$1" + local message + + message="$(json_value "$input" '.message')" + [[ -z "$message" ]] && message="$(json_value "$input" '.msg')" + printf '%s' "$message" +} + +response_indicates_failure() { + local input="$1" + local success_flag + local code_value + local status + local message + local error_regex='[Ff]ail|[Ee]rror' + + success_flag="$(json_value "$input" '.success')" + code_value="$(json_value "$input" '.code')" + status="$(json_value "$input" '.status')" + message="$(response_message "$input")" + + if [[ "$success_flag" == "false" ]]; then + return 0 + fi + + if [[ -n "$code_value" && "$code_value" != "0" ]]; then + return 0 + fi + + if [[ "${status} ${message}" =~ $error_regex ]]; then + return 0 + fi + + return 1 +} + http_request() { local method="$1" local url="$2" @@ -899,10 +1053,113 @@ download_cloud_to_node() { poll_download_progress } +poll_upgrade_progress() { + local ip="$1" + local progress_url="${HOME_BASE_URL}/node-proxy/${AIRPORT_CODE}/api/mcp/version/upgrade/progress?applicationName=${APP_NAME}&moduleName=${MODULE_NAME}&airportCode=${AIRPORT_CODE}&versionNumber=${VERSION_NUMBER}" + local attempt=0 + local max_attempts=600 + local error_regex='[Ff]ail|[Ee]rror' + + UPGRADE_PROGRESS_STATUS="" + UPGRADE_PROGRESS_SUCCESS="" + UPGRADE_PROGRESS_STEP="" + UPGRADE_PROGRESS_MSG="" + UPGRADE_PROGRESS_MESSAGE="" + UPGRADE_PROGRESS_RATE="" + UPGRADE_PROGRESS_CODE="" + UPGRADE_PROGRESS_FINISH="" + UPGRADE_PROGRESS_LAST_MODIFY="" + UPGRADE_PROGRESS_RESPONSE="" + + while (( attempt < max_attempts )); do + local response + response=$(http_request "GET" "$progress_url" "" "" "Target-Node: ${NODE_URL}") || return 1 + + local status + status="$(json_scoped_value "$response" "$ip" '.status')" + local success_flag + success_flag="$(json_scoped_value "$response" "$ip" '.success')" + local step_value + step_value="$(json_scoped_value "$response" "$ip" '.step')" + local msg_value + msg_value="$(json_scoped_value "$response" "$ip" '.msg')" + local message + message="$(json_scoped_value "$response" "$ip" '.message')" + local progress_value + progress_value="$(json_scoped_value "$response" "$ip" '.rateOfProgress')" + [[ -z "$progress_value" ]] && progress_value="$(json_scoped_value "$response" "$ip" '.progress')" + [[ -z "$progress_value" ]] && progress_value="$(json_scoped_value "$response" "$ip" '.percent')" + local code_value + code_value="$(json_scoped_value "$response" "$ip" '.code')" + local finish_value + finish_value="$(json_scoped_value "$response" "$ip" '.finish')" + local last_modify_value + last_modify_value="$(json_scoped_value "$response" "$ip" '.lastModify')" + [[ -z "$message" ]] && message="$msg_value" + + UPGRADE_PROGRESS_STATUS="$status" + UPGRADE_PROGRESS_SUCCESS="$success_flag" + UPGRADE_PROGRESS_STEP="$step_value" + UPGRADE_PROGRESS_MSG="$msg_value" + UPGRADE_PROGRESS_MESSAGE="$message" + UPGRADE_PROGRESS_RATE="$progress_value" + UPGRADE_PROGRESS_CODE="$code_value" + UPGRADE_PROGRESS_FINISH="$finish_value" + UPGRADE_PROGRESS_LAST_MODIFY="$last_modify_value" + UPGRADE_PROGRESS_RESPONSE="$response" + + if [[ -n "$msg_value" || -n "$step_value" || -n "$progress_value" || -n "$status" || -n "$success_flag" || -n "$message" || -n "$code_value" || -n "$finish_value" || -n "$last_modify_value" ]]; then + local -a progress_parts=() + progress_parts+=("ip=${ip}") + [[ -n "$msg_value" ]] && progress_parts+=("msg=${msg_value}") + [[ -n "$step_value" ]] && progress_parts+=("step=${step_value}") + [[ -n "$progress_value" ]] && progress_parts+=("rateOfProgress=${progress_value}") + [[ -n "$code_value" ]] && progress_parts+=("code=${code_value}") + [[ -n "$finish_value" ]] && progress_parts+=("finish=${finish_value}") + [[ -n "$status" ]] && progress_parts+=("status=${status}") + [[ -n "$success_flag" ]] && progress_parts+=("success=${success_flag}") + [[ -n "$last_modify_value" ]] && progress_parts+=("lastModify=${last_modify_value}") + [[ -n "$message" && "$message" != "$msg_value" ]] && progress_parts+=("message=${message}") + log_info "Step 3.4a: async push progress -> ${progress_parts[*]}" + else + log_info "Step 3.4a: async push progress polling... ip=${ip} ($((attempt + 1))/${max_attempts})" + fi + + if [[ "$step_value" == "DONE" || "$finish_value" == "true" || "$status" == "completed" || "$success_flag" == "true" ]]; then + return 0 + fi + + if [[ "$msg_value" == "success" && "$progress_value" == "100" ]] && [[ -z "$code_value" || "$code_value" == "0" ]]; then + return 0 + fi + + if [[ -n "$code_value" && "$code_value" != "0" ]]; then + [[ -z "$message" ]] && message="$msg_value" + [[ -z "$message" ]] && message="$step_value" + [[ -z "$message" ]] && message="code=${code_value}" + log_error "Node push failed: ip=${ip}, message=${message}" + return 1 + fi + + if [[ "${step_value} ${message} ${msg_value} ${status}" =~ $error_regex ]]; then + [[ -z "$message" ]] && message="$step_value" + [[ -z "$message" ]] && message="$msg_value" + log_error "Node push failed: ip=${ip}, message=${message}" + return 1 + fi + + attempt=$((attempt + 1)) + sleep 2 + done + + log_error "Node push timed out: ip=${ip}" + return 1 +} + upgrade_ip() { local ip="$1" http_request "POST" \ - "${HOME_BASE_URL}/node-proxy/${AIRPORT_CODE}/api/mcp/version/upgrade?airportCode=${AIRPORT_CODE}&targetIp=${ip}&applicationName=${APP_NAME}&moduleName=${MODULE_NAME}&versionNumber=${VERSION_NUMBER}&action=${ACTION_TYPE}&autoStart=false&timeOut=${TIMEOUT}" \ + "${HOME_BASE_URL}/node-proxy/${AIRPORT_CODE}/api/mcp/version/upgrade?airportCode=${AIRPORT_CODE}&targetIp=${ip}&applicationName=${APP_NAME}&moduleName=${MODULE_NAME}&versionNumber=${VERSION_NUMBER}&action=${ACTION_TYPE}&autoStart=false&timeOut=0" \ "" \ "" \ "Target-Node: ${NODE_URL}" @@ -1130,15 +1387,31 @@ run_action() { print_progress_result "download-cloud-to-node" result_line "RESULT" "DONE" ;; + poll-upgrade-progress) + require_ip_arg "$ip" || return 1 + run_flow_step "get_token" get_token || return 1 + run_flow_step "get_node_url" get_node_url || return 1 + run_flow_step "poll_upgrade_progress[${ip}]" poll_upgrade_progress "$ip" || return 1 + print_upgrade_progress_result "$ip" + ;; upgrade-ip) require_ip_arg "$ip" || return 1 run_flow_step "get_token" get_token || return 1 run_flow_step "get_node_url" get_node_url || return 1 run_flow_capture response "upgrade_ip[${ip}]" upgrade_ip "$ip" || return 1 + if response_indicates_failure "$response"; then + local message + message="$(response_message "$response")" + [[ -z "$message" ]] && message="Upgrade task creation failed" + log_error "Upgrade task creation failed: ip=${ip}, message=${message}" + return 1 + fi result_line "ACTION" "upgrade-ip" result_line "IP" "$ip" + result_line "TIME_OUT" "0" + result_line "RESULT" "TASK_CREATED" result_line "SUCCESS" "$(json_value "$response" '.success')" - result_line "MESSAGE" "$(json_value "$response" '.message')" + result_line "MESSAGE" "$(response_message "$response")" result_line "RAW_RESPONSE" "$(sanitize_field "$response")" ;; start-ip) @@ -1237,10 +1510,10 @@ deploy_one_ip() { return fi - if [[ "$(json_value "$upgrade_response" '.success')" != "true" ]]; then + if response_indicates_failure "$upgrade_response"; then local message - message="$(json_value "$upgrade_response" '.message')" - [[ -z "$message" ]] && message="Upgrade failed" + message="$(response_message "$upgrade_response")" + [[ -z "$message" ]] && message="Upgrade task creation failed" local rollback_result rollback_result="$(pending_rollback_status "$ip" "UPGRADE" "$message" "false")" local log_file="" @@ -1249,6 +1522,20 @@ deploy_one_ip() { return fi + if ! run_flow_step "poll_upgrade_progress[${ip}]" poll_upgrade_progress "$ip"; then + local message + message="$UPGRADE_PROGRESS_MESSAGE" + [[ -z "$message" ]] && message="$UPGRADE_PROGRESS_MSG" + [[ -z "$message" ]] && message="$UPGRADE_PROGRESS_STEP" + [[ -z "$message" ]] && message="Upgrade progress polling failed" + local rollback_result + rollback_result="$(pending_rollback_status "$ip" "UPGRADE_PROGRESS" "$message" "false")" + local log_file="" + run_flow_capture log_file "download_log[${ip}]" download_log "$ip" || true + add_result "$ip" "FAILED" "UPGRADE_PROGRESS" "$message" "$rollback_result" "$log_file" + return + fi + if ! run_flow_step "start_application[${ip}]" start_application "$ip"; then local rollback_result rollback_result="$(pending_rollback_status "$ip" "START" "Application start failed" "true")" diff --git a/doc_scripts/test_deploy.ps1 b/doc_scripts/test_deploy.ps1 index 8ead03d..9934be1 100644 --- a/doc_scripts/test_deploy.ps1 +++ b/doc_scripts/test_deploy.ps1 @@ -203,13 +203,19 @@ function Invoke-PamFullTest { foreach ($ip in $testIps) { try { $upgradeResponse = Invoke-UpgradeRequest -Config $config -Token $token -NodeUrl $nodeUrl -Ip $ip - $upgradeSuccess = Get-ResponseValue -Response $upgradeResponse -Candidates @('success') - $upgradeMessage = Get-ResponseValue -Response $upgradeResponse -Candidates @('message') - if ($upgradeSuccess -eq 'true') { - Add-TestResult -Results $results -Step "UPGRADE [$ip]" -Status 'PASS' -Detail 'success=true' - } else { - if (-not $upgradeMessage) { $upgradeMessage = 'success != true' } + if (Test-ResponseFailure -Response $upgradeResponse) { + $upgradeMessage = Get-PrimaryResponseMessage -Response $upgradeResponse + if (-not $upgradeMessage) { $upgradeMessage = 'task creation failed' } Add-TestResult -Results $results -Step "UPGRADE [$ip]" -Status 'FAIL' -Detail $upgradeMessage + } else { + Add-TestResult -Results $results -Step "UPGRADE [$ip]" -Status 'PASS' -Detail 'task created' + try { + Wait-UpgradeProgress -Config $config -Token $token -NodeUrl $nodeUrl -Ip $ip + Add-TestResult -Results $results -Step "UPGRADE_PROGRESS [$ip]" -Status 'PASS' -Detail ("rateOfProgress={0}" -f $script:UpgradeProgressState.RateOfProgress) + } catch { + $upgradeMessage = Get-ProgressStateMessage -State $script:UpgradeProgressState -DefaultMessage 'progress polling failed' + Add-TestResult -Results $results -Step "UPGRADE_PROGRESS [$ip]" -Status 'FAIL' -Detail $upgradeMessage + } } } catch { Add-TestResult -Results $results -Step "UPGRADE [$ip]" -Status 'FAIL' -Detail $_.Exception.Message diff --git a/doc_scripts/test_deploy.sh b/doc_scripts/test_deploy.sh index cd0c470..fd5a92f 100644 --- a/doc_scripts/test_deploy.sh +++ b/doc_scripts/test_deploy.sh @@ -207,13 +207,20 @@ run_full_test() { local rollback_result="" if upgrade_response="$(upgrade_ip "$ip")"; then - upgrade_success="$(json_value "$upgrade_response" '.success')" - upgrade_message="$(json_value "$upgrade_response" '.message')" - if [[ "$upgrade_success" == "true" ]]; then - add_test_result "UPGRADE [$ip]" "PASS" "success=true" - else - [[ -z "$upgrade_message" ]] && upgrade_message="success != true" + if response_indicates_failure "$upgrade_response"; then + upgrade_message="$(response_message "$upgrade_response")" + [[ -z "$upgrade_message" ]] && upgrade_message="task creation failed" add_test_result "UPGRADE [$ip]" "FAIL" "$upgrade_message" + else + add_test_result "UPGRADE [$ip]" "PASS" "task created" + if poll_upgrade_progress "$ip"; then + add_test_result "UPGRADE_PROGRESS [$ip]" "PASS" "rateOfProgress=${UPGRADE_PROGRESS_RATE}" + else + upgrade_message="$UPGRADE_PROGRESS_MESSAGE" + [[ -z "$upgrade_message" ]] && upgrade_message="$UPGRADE_PROGRESS_MSG" + [[ -z "$upgrade_message" ]] && upgrade_message="progress polling failed" + add_test_result "UPGRADE_PROGRESS [$ip]" "FAIL" "$upgrade_message" + fi fi else add_test_result "UPGRADE [$ip]" "FAIL" "request failed" diff --git a/doc_scripts/当前脚本情况总结.md b/doc_scripts/当前脚本情况总结.md index e572bf8..d4c6ef8 100644 --- a/doc_scripts/当前脚本情况总结.md +++ b/doc_scripts/当前脚本情况总结.md @@ -52,6 +52,7 @@ - `/api/mcp/version/upgrade` 使用 query 参数，不再使用 body 表单。 - `/api/mcp/version/upgrade/start-stop` 使用 query 参数，不再使用 body 表单，且参数名使用 `runStart`。 - `download-cloud` 固定传 `timeOut=0`，仅用于创建下载任务；后续进度通过 `download-cloud/progress` 异步查询。 +- `upgrade-ip` 固定传 `timeOut=0`，仅用于创建推送任务；后续进度通过 `upgrade/progress` 异步查询，对外新增 `poll-upgrade-progress` action。 - Shell 侧 action 调用现在支持复用统一的 trace 文件，不再要求每次 action 各自生成单独的 trace 日志。 - `log-download` 下载结果按 zip 压缩包保存，不再按纯文本日志处理。 @@ -75,10 +76,11 @@ 7. `api/mcp/version/upgrade/download-cloud` 8. `api/mcp/version/upgrade/download-cloud/progress` 9. `api/mcp/version/upgrade` -10. `api/mcp/version/upgrade/start-stop` -11. `api/mcp/version/upgrade/verify` -12. `api/mcp/version/upgrade/log-download` -13. `api/mcp/version/upgrade/rollback` +10. `api/mcp/version/upgrade/progress` +11. `api/mcp/version/upgrade/start-stop` +12. `api/mcp/version/upgrade/verify` +13. `api/mcp/version/upgrade/log-download` +14. `api/mcp/version/upgrade/rollback` 正式部署脚本当前不会自动执行 `rollback`，而是输出待确认状态；需要实际回滚时，应由 Agent 与用户确认后，再调用手动回滚入口： @@ -95,11 +97,13 @@ powershell -File .\deploy.ps1 -ConfigPath .\config.txt -RollbackIp 192.168.1.10 ```bash bash ./deploy.sh --config ./config.txt --action get-online-ips bash ./deploy.sh --config ./config.txt --action upgrade-ip --ip 192.168.1.10 +bash ./deploy.sh --config ./config.txt --action poll-upgrade-progress --ip 192.168.1.10 ``` ```powershell powershell -File .\deploy.ps1 -ConfigPath .\config.txt -Action GetOnlineIps powershell -File .\deploy.ps1 -ConfigPath .\config.txt -Action UpgradeIp -Ip 192.168.1.10 +powershell -File .\deploy.ps1 -ConfigPath .\config.txt -Action poll-upgrade-progress -Ip 192.168.1.10 ``` ## 3. 当前运行方式