更新日志维护策略
This commit is contained in:
parent
039a3e1bdc
commit
33065f6c09
11
README.md
11
README.md
@ -89,9 +89,9 @@ packaging/
|
|||||||
- chat 支持执行中按 `Ctrl+C` 中断,保存 checkpoint 后再 `resume`。
|
- chat 支持执行中按 `Ctrl+C` 中断,保存 checkpoint 后再 `resume`。
|
||||||
- chat 支持 `set KEY=VALUE` 和 `load params <路径>` 热更新当前运行参数,并同步回写运行中的 `config.txt` 与 checkpoint。
|
- chat 支持 `set KEY=VALUE` 和 `load params <路径>` 热更新当前运行参数,并同步回写运行中的 `config.txt` 与 checkpoint。
|
||||||
- 支持通过 `--llm-action-analysis-prompt-file`、`PAM_LLM_ACTION_ANALYSIS_PROMPT_FILE` 或 chat 内 `llm config action_analysis_prompt_file=...` 自定义 action 审核提示词。
|
- 支持通过 `--llm-action-analysis-prompt-file`、`PAM_LLM_ACTION_ANALYSIS_PROMPT_FILE` 或 chat 内 `llm config action_analysis_prompt_file=...` 自定义 action 审核提示词。
|
||||||
- 增加统一运行日志,默认写入 `logs/pam_deploy_agent.log`,覆盖 CLI/chat、LLM 调用、action 路由、脚本/MCP 调用、LangGraph、checkpoint 等关键流程。
|
- 增加统一运行日志,默认写入 `logs/pam_deploy_agent.log`,覆盖 CLI/chat、LLM 调用、action 路由、脚本/MCP 调用、LangGraph、checkpoint 等关键流程,并按天切分、默认保留 14 个历史日切文件。
|
||||||
- chat 支持 `llm test [文本]`,可用当前 LLM client 做一次轻量调用,确认真实 LLM 或规则 fallback 是否正常加载。
|
- chat 支持 `llm test [文本]`,可用当前 LLM client 做一次轻量调用,确认真实 LLM 或规则 fallback 是否正常加载。
|
||||||
- 添加基础测试,当前本地结果为 `72 passed, 3 skipped`。
|
- 添加基础测试,当前本地结果为 `73 passed, 3 skipped`。
|
||||||
|
|
||||||
未完成:
|
未完成:
|
||||||
|
|
||||||
@ -316,16 +316,17 @@ PAM> exit
|
|||||||
|
|
||||||
## 日志
|
## 日志
|
||||||
|
|
||||||
Agent 默认写入运行日志到 `logs/pam_deploy_agent.log`。日志覆盖 CLI/chat 输入、LLM 请求和响应摘要、action 路由、脚本/MCP 调用、LangGraph 节点、checkpoint 保存、暂停/续跑等关键流程。日志会递归脱敏 `CLIENT_SECRET`、`MCP_CLIENT_SECRET`、token、Authorization、api_key、password 等字段,并截断长文本。
|
Agent 默认写入运行日志到 `logs/pam_deploy_agent.log`。日志覆盖 CLI/chat 输入、LLM 请求和响应摘要、action 路由、脚本/MCP 调用、LangGraph 节点、checkpoint 保存、暂停/续跑等关键流程。日志会在本地时间每日 0 点后首次写入时自动切分,历史文件形如 `pam_deploy_agent.log.YYYY-MM-DD`,默认保留 14 个历史日切文件。日志会递归脱敏 `CLIENT_SECRET`、`MCP_CLIENT_SECRET`、token、Authorization、api_key、password 等字段,并截断长文本。
|
||||||
|
|
||||||
可通过环境变量调整日志位置和级别:
|
可通过环境变量调整日志位置、级别和保留策略:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export PAM_AGENT_LOG_FILE=logs/pam_deploy_agent.log
|
export PAM_AGENT_LOG_FILE=logs/pam_deploy_agent.log
|
||||||
export PAM_AGENT_LOG_LEVEL=INFO
|
export PAM_AGENT_LOG_LEVEL=INFO
|
||||||
|
export PAM_AGENT_LOG_RETENTION_DAYS=14
|
||||||
```
|
```
|
||||||
|
|
||||||
调试 LLM 或 MCP 调用时可临时设为 `DEBUG`,但仍建议把日志目录放在受控位置。
|
调试 LLM 或 MCP 调用时可临时把 `PAM_AGENT_LOG_LEVEL` 设为 `DEBUG`。`PAM_AGENT_LOG_RETENTION_DAYS` 表示保留的历史日切文件数量,设为 `0` 时不自动清理历史切分文件;仍建议把日志目录放在受控位置。
|
||||||
|
|
||||||
预演:
|
预演:
|
||||||
|
|
||||||
|
|||||||
@ -83,7 +83,7 @@ cd pam-deploy-agent-linux-x86_64
|
|||||||
- 进度查询和健康检查重试参数可通过 `POLL_INTERVAL_SEC`、`DOWNLOAD_POLL_MAX_ATTEMPTS`、`UPGRADE_POLL_MAX_ATTEMPTS`、`VERIFY_INTERVAL_SEC`、`VERIFY_MAX_ATTEMPTS` 配置。
|
- 进度查询和健康检查重试参数可通过 `POLL_INTERVAL_SEC`、`DOWNLOAD_POLL_MAX_ATTEMPTS`、`UPGRADE_POLL_MAX_ATTEMPTS`、`VERIFY_INTERVAL_SEC`、`VERIFY_MAX_ATTEMPTS` 配置。
|
||||||
- 支持通过 `--llm-action-analysis-prompt-file` 或 chat 内 `llm config action_analysis_prompt_file=...` 自定义 action 审核提示词。
|
- 支持通过 `--llm-action-analysis-prompt-file` 或 chat 内 `llm config action_analysis_prompt_file=...` 自定义 action 审核提示词。
|
||||||
- chat 支持 `llm test [文本]` 测试当前 LLM client 是否正常加载。
|
- chat 支持 `llm test [文本]` 测试当前 LLM client 是否正常加载。
|
||||||
- 默认运行日志写入 `logs/pam_deploy_agent.log`,可通过 `PAM_AGENT_LOG_FILE` 和 `PAM_AGENT_LOG_LEVEL` 调整。
|
- 默认运行日志写入 `logs/pam_deploy_agent.log`,按天切分并默认保留 14 个历史日切文件,可通过 `PAM_AGENT_LOG_FILE`、`PAM_AGENT_LOG_LEVEL` 和 `PAM_AGENT_LOG_RETENTION_DAYS` 调整。
|
||||||
- 日志会脱敏 token、secret、api_key、Authorization 等字段;checkpoint 仍保存完整运行参数,请放在受控目录。
|
- 日志会脱敏 token、secret、api_key、Authorization 等字段;checkpoint 仍保存完整运行参数,请放在受控目录。
|
||||||
|
|
||||||
## 包大小评估
|
## 包大小评估
|
||||||
|
|||||||
@ -192,16 +192,17 @@ PAM> llm fallback
|
|||||||
|
|
||||||
## 日志
|
## 日志
|
||||||
|
|
||||||
Agent 默认写入运行日志到 `logs/pam_deploy_agent.log`。日志覆盖 chat/CLI 输入、LLM 请求和响应摘要、action 路由、脚本/MCP 调用、LangGraph 节点、checkpoint 保存、暂停/续跑等关键流程。
|
Agent 默认写入运行日志到 `logs/pam_deploy_agent.log`。日志覆盖 chat/CLI 输入、LLM 请求和响应摘要、action 路由、脚本/MCP 调用、LangGraph 节点、checkpoint 保存、暂停/续跑等关键流程。日志会在本地时间每日 0 点后首次写入时自动切分,历史文件形如 `pam_deploy_agent.log.YYYY-MM-DD`,默认保留 14 个历史日切文件。
|
||||||
|
|
||||||
可通过环境变量调整日志位置和级别:
|
可通过环境变量调整日志位置、级别和保留策略:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export PAM_AGENT_LOG_FILE=logs/pam_deploy_agent.log
|
export PAM_AGENT_LOG_FILE=logs/pam_deploy_agent.log
|
||||||
export PAM_AGENT_LOG_LEVEL=INFO
|
export PAM_AGENT_LOG_LEVEL=INFO
|
||||||
|
export PAM_AGENT_LOG_RETENTION_DAYS=14
|
||||||
```
|
```
|
||||||
|
|
||||||
日志会递归脱敏 `CLIENT_SECRET`、`MCP_CLIENT_SECRET`、token、Authorization、api_key、password 等字段,并截断长文本。checkpoint 仍会保存完整运行参数,请放在受控目录。
|
日志会递归脱敏 `CLIENT_SECRET`、`MCP_CLIENT_SECRET`、token、Authorization、api_key、password 等字段,并截断长文本。`PAM_AGENT_LOG_RETENTION_DAYS` 表示保留的历史日切文件数量,设为 `0` 时不自动清理历史切分文件。checkpoint 仍会保存完整运行参数,请放在受控目录。
|
||||||
|
|
||||||
## 策略说明
|
## 策略说明
|
||||||
|
|
||||||
|
|||||||
@ -168,6 +168,9 @@ LLM 环境变量:
|
|||||||
PAM_AGENT_LOG_LEVEL
|
PAM_AGENT_LOG_LEVEL
|
||||||
日志级别,默认 INFO。排查 LLM/MCP 时可临时设为 DEBUG。
|
日志级别,默认 INFO。排查 LLM/MCP 时可临时设为 DEBUG。
|
||||||
|
|
||||||
|
PAM_AGENT_LOG_RETENTION_DAYS
|
||||||
|
历史日切日志保留数量,默认 14。设为 0 时不自动清理历史切分文件。
|
||||||
|
|
||||||
示例:
|
示例:
|
||||||
./run.sh chat --config doc_scripts/config.txt.example --strategy fake --checkpoint runtime/checkpoints/demo.json
|
./run.sh chat --config doc_scripts/config.txt.example --strategy fake --checkpoint runtime/checkpoints/demo.json
|
||||||
|
|
||||||
@ -197,7 +200,7 @@ LLM 环境变量:
|
|||||||
7. PARENT_VERSION_NUMBER 是云下载可选参数;空值不发送,非空时传给 parentVersionNumber。
|
7. PARENT_VERSION_NUMBER 是云下载可选参数;空值不发送,非空时传给 parentVersionNumber。
|
||||||
8. chat 执行过程中会播报每个 action 的开始、完成或失败;普通问候不会触发 LLM/结构化分析。
|
8. chat 执行过程中会播报每个 action 的开始、完成或失败;普通问候不会触发 LLM/结构化分析。
|
||||||
9. chat 内可使用 params、events、rollback、list checkpoints、load checkpoint、load params、llm config、llm test、mcp config 等命令。
|
9. chat 内可使用 params、events、rollback、list checkpoints、load checkpoint、load params、llm config、llm test、mcp config 等命令。
|
||||||
10. 日志默认写入 logs/pam_deploy_agent.log,并会脱敏 token、secret、api_key、Authorization 等字段。
|
10. 日志默认写入 logs/pam_deploy_agent.log,按天切分并默认保留 14 个历史日切文件;日志会脱敏 token、secret、api_key、Authorization 等字段。
|
||||||
11. checkpoint 会保存完整运行参数,请放在受控目录。
|
11. checkpoint 会保存完整运行参数,请放在受控目录。
|
||||||
HELP_TEXT
|
HELP_TEXT
|
||||||
}
|
}
|
||||||
|
|||||||
@ -7,14 +7,17 @@ import logging
|
|||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
from dataclasses import asdict, is_dataclass
|
from dataclasses import asdict, is_dataclass
|
||||||
|
from logging.handlers import TimedRotatingFileHandler
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from .constants import SENSITIVE_KEYS
|
from .constants import SENSITIVE_KEYS
|
||||||
|
|
||||||
DEFAULT_LOG_FILE = Path("logs") / "pam_deploy_agent.log"
|
DEFAULT_LOG_FILE = Path("logs") / "pam_deploy_agent.log"
|
||||||
|
DEFAULT_LOG_RETENTION_DAYS = 14
|
||||||
LOG_FILE_ENV = "PAM_AGENT_LOG_FILE"
|
LOG_FILE_ENV = "PAM_AGENT_LOG_FILE"
|
||||||
LOG_LEVEL_ENV = "PAM_AGENT_LOG_LEVEL"
|
LOG_LEVEL_ENV = "PAM_AGENT_LOG_LEVEL"
|
||||||
|
LOG_RETENTION_DAYS_ENV = "PAM_AGENT_LOG_RETENTION_DAYS"
|
||||||
_HANDLER_MARKER = "_pam_deploy_agent_handler"
|
_HANDLER_MARKER = "_pam_deploy_agent_handler"
|
||||||
_SENSITIVE_NAME_PARTS = ("secret", "token", "authorization", "api_key", "apikey", "password")
|
_SENSITIVE_NAME_PARTS = ("secret", "token", "authorization", "api_key", "apikey", "password")
|
||||||
_ASSIGNMENT_PATTERN = re.compile(
|
_ASSIGNMENT_PATTERN = re.compile(
|
||||||
@ -28,23 +31,38 @@ _BEARER_PATTERN = re.compile(r"(?i)(bearer\s+)[A-Za-z0-9._~+\-/=]+")
|
|||||||
def configure_logging(
|
def configure_logging(
|
||||||
log_file: str | Path | None = None,
|
log_file: str | Path | None = None,
|
||||||
level: str | int | None = None,
|
level: str | int | None = None,
|
||||||
|
retention_days: int | str | None = None,
|
||||||
) -> Path:
|
) -> Path:
|
||||||
"""配置 Agent 文件日志;重复调用不会重复添加 handler。"""
|
"""配置 Agent 每日滚动文件日志;重复调用不会重复添加 handler。"""
|
||||||
actual_path = Path(log_file or os.getenv(LOG_FILE_ENV) or DEFAULT_LOG_FILE)
|
actual_path = Path(log_file or os.getenv(LOG_FILE_ENV) or DEFAULT_LOG_FILE)
|
||||||
actual_path.parent.mkdir(parents=True, exist_ok=True)
|
actual_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
actual_level = _resolve_level(level or os.getenv(LOG_LEVEL_ENV) or "INFO")
|
actual_level = _resolve_level(level or os.getenv(LOG_LEVEL_ENV) or "INFO")
|
||||||
|
actual_retention_days = _resolve_retention_days(
|
||||||
|
retention_days if retention_days is not None else os.getenv(LOG_RETENTION_DAYS_ENV),
|
||||||
|
)
|
||||||
|
|
||||||
package_logger = logging.getLogger("pam_deploy_graph")
|
package_logger = logging.getLogger("pam_deploy_graph")
|
||||||
package_logger.setLevel(actual_level)
|
package_logger.setLevel(actual_level)
|
||||||
package_logger.propagate = False
|
package_logger.propagate = False
|
||||||
|
|
||||||
marker = str(actual_path.resolve())
|
marker = str(actual_path.resolve())
|
||||||
for handler in package_logger.handlers:
|
for handler in list(package_logger.handlers):
|
||||||
if getattr(handler, _HANDLER_MARKER, "") == marker:
|
if getattr(handler, _HANDLER_MARKER, "") == marker:
|
||||||
|
if isinstance(handler, TimedRotatingFileHandler):
|
||||||
handler.setLevel(actual_level)
|
handler.setLevel(actual_level)
|
||||||
|
handler.backupCount = actual_retention_days
|
||||||
return actual_path
|
return actual_path
|
||||||
|
package_logger.removeHandler(handler)
|
||||||
|
handler.close()
|
||||||
|
break
|
||||||
|
|
||||||
handler = logging.FileHandler(actual_path, encoding="utf-8")
|
handler = TimedRotatingFileHandler(
|
||||||
|
actual_path,
|
||||||
|
when="midnight",
|
||||||
|
interval=1,
|
||||||
|
backupCount=actual_retention_days,
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
setattr(handler, _HANDLER_MARKER, marker)
|
setattr(handler, _HANDLER_MARKER, marker)
|
||||||
handler.setLevel(actual_level)
|
handler.setLevel(actual_level)
|
||||||
handler.setFormatter(
|
handler.setFormatter(
|
||||||
@ -54,7 +72,12 @@ def configure_logging(
|
|||||||
)
|
)
|
||||||
)
|
)
|
||||||
package_logger.addHandler(handler)
|
package_logger.addHandler(handler)
|
||||||
package_logger.info("日志已初始化 path=%s level=%s", actual_path, logging.getLevelName(actual_level))
|
package_logger.info(
|
||||||
|
"日志已初始化 path=%s level=%s rotation=daily retention_days=%s",
|
||||||
|
actual_path,
|
||||||
|
logging.getLevelName(actual_level),
|
||||||
|
actual_retention_days,
|
||||||
|
)
|
||||||
return actual_path
|
return actual_path
|
||||||
|
|
||||||
|
|
||||||
@ -94,6 +117,17 @@ def _resolve_level(value: str | int) -> int:
|
|||||||
return resolved if isinstance(resolved, int) else logging.INFO
|
return resolved if isinstance(resolved, int) else logging.INFO
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_retention_days(value: int | str | None) -> int:
|
||||||
|
"""解析日志保留天数,非法值使用默认值。"""
|
||||||
|
if value in (None, ""):
|
||||||
|
return DEFAULT_LOG_RETENTION_DAYS
|
||||||
|
try:
|
||||||
|
days = int(str(value).strip())
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return DEFAULT_LOG_RETENTION_DAYS
|
||||||
|
return max(days, 0)
|
||||||
|
|
||||||
|
|
||||||
def _is_sensitive_key(key: str) -> bool:
|
def _is_sensitive_key(key: str) -> bool:
|
||||||
"""判断字段名是否应脱敏。"""
|
"""判断字段名是否应脱敏。"""
|
||||||
if key in SENSITIVE_KEYS:
|
if key in SENSITIVE_KEYS:
|
||||||
|
|||||||
@ -1,4 +1,8 @@
|
|||||||
from pam_deploy_graph.logging_utils import json_for_log, redact_for_log
|
import logging
|
||||||
|
from logging.handlers import TimedRotatingFileHandler
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from pam_deploy_graph.logging_utils import configure_logging, json_for_log, redact_for_log
|
||||||
|
|
||||||
|
|
||||||
def test_redact_for_log_masks_sensitive_keys_and_inline_assignments():
|
def test_redact_for_log_masks_sensitive_keys_and_inline_assignments():
|
||||||
@ -26,3 +30,37 @@ def test_redact_for_log_masks_sensitive_keys_and_inline_assignments():
|
|||||||
assert "Bearer ***" in serialized
|
assert "Bearer ***" in serialized
|
||||||
assert "raw-token" not in serialized
|
assert "raw-token" not in serialized
|
||||||
assert "plain-token" not in serialized
|
assert "plain-token" not in serialized
|
||||||
|
|
||||||
|
|
||||||
|
def test_configure_logging_uses_daily_rotation_and_retention(tmp_path: Path):
|
||||||
|
log_path = tmp_path / "pam_deploy_agent.log"
|
||||||
|
package_logger = logging.getLogger("pam_deploy_graph")
|
||||||
|
previous_handlers = list(package_logger.handlers)
|
||||||
|
for handler in previous_handlers:
|
||||||
|
package_logger.removeHandler(handler)
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = configure_logging(log_file=log_path, level="DEBUG", retention_days=3)
|
||||||
|
|
||||||
|
assert result == log_path
|
||||||
|
handlers = [handler for handler in package_logger.handlers if isinstance(handler, TimedRotatingFileHandler)]
|
||||||
|
assert len(handlers) == 1
|
||||||
|
handler = handlers[0]
|
||||||
|
assert Path(handler.baseFilename) == log_path.resolve()
|
||||||
|
assert handler.when == "MIDNIGHT"
|
||||||
|
assert handler.backupCount == 3
|
||||||
|
assert package_logger.level == logging.DEBUG
|
||||||
|
|
||||||
|
configure_logging(log_file=log_path, level="INFO", retention_days=5)
|
||||||
|
|
||||||
|
handlers = [handler for handler in package_logger.handlers if isinstance(handler, TimedRotatingFileHandler)]
|
||||||
|
assert len(handlers) == 1
|
||||||
|
assert handlers[0] is handler
|
||||||
|
assert handler.backupCount == 5
|
||||||
|
assert package_logger.level == logging.INFO
|
||||||
|
finally:
|
||||||
|
for handler in list(package_logger.handlers):
|
||||||
|
package_logger.removeHandler(handler)
|
||||||
|
handler.close()
|
||||||
|
for handler in previous_handlers:
|
||||||
|
package_logger.addHandler(handler)
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user