feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit

- Add --append flag: diff source-table 总排号 against existing
  Common.Attachment SN and classify/write only the missing ones. --limit
  caps the per-run append count, --order sets the direction. Backed by
  new fetch_existing_attachment_sns() in db.py.
- Fix batch elapsed-time accounting: summarize_results() summed each
  task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor
  concurrency (cumulative work time, not real wall-clock — 100 tasks on
  8 workers reported ~5x the actual runtime). Callers now time
  classify_batch() via perf_counter and pass wall_clock_ms; both the
  write_attachments [汇总] line and main.py --summary report wall-clock
  separately from the cumulative sum.
- Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the
  human-readable [汇总] line; structured JSON --summary fields stay in ms.
- Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100
  bind-parameter hard limit (previously --append --limit 5000 failed at
  the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids
  (sn / id paths) and to the DELETE inside upsert_attachments.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Misaka_Company
2026-07-28 11:17:17 +08:00
parent a014a945af
commit 2ae5621090
5 changed files with 166 additions and 42 deletions

View File

@@ -173,9 +173,13 @@ python main.py --sn 26B742,26B743 --summary
"meta": {"elapsed_ms": 1476.6, "attempts": 1, "llm": {"model": "deepseek-v4-flash", "elapsed_ms": 1472.8, "usage": {"prompt_tokens": 1297, "completion_tokens": 15, "total_tokens": 1312, "prompt_cache_hit_tokens": 1280, "prompt_cache_miss_tokens": 17, "cache_hit_rate": 0.9869}}}
```
`--summary` 打印的整批汇总字段与 `meta.llm.usage` 对应:`count` / `total_elapsed_ms` / `llm_calls` /
各类 `total_*_tokens` / `total_cache_hit_tokens` / `total_cache_miss_tokens` / `avg_cache_hit_rate`
(按 token 加权的整体命中率)。
`--summary` 打印的整批汇总字段与 `meta.llm.usage` 对应:`count` / `wall_clock_ms` / `total_elapsed_ms` /
`llm_calls` / 各类 `total_*_tokens` / `total_cache_hit_tokens` / `total_cache_miss_tokens` /
`avg_cache_hit_rate`(按 token 加权的整体命中率)。
两个耗时字段的区别(`classify_batch` 是并发执行的):
- `wall_clock_ms`:整批分类的**真实墙钟**耗时(在 `classify_batch` 外层用 `perf_counter` 计时),即你实际等的时间。
- `total_elapsed_ms`:各结果自身耗时 `elapsed_ms` 的**累加**(相当于把这些任务串行跑的总时长),并发下会明显大于墙钟;二者之比 ≈ 并发增益(≈ `max_workers`)。
## 对话日志
@@ -265,6 +269,11 @@ python write_attachments.py --range 800,805
# 组合:区间内降序、再取前 3 个
python write_attachments.py --range 800,805 --order desc --limit 3
# 追加模式:仅补写源表中存在、但 Common.Attachment 还没有的总排号
# --limit 此时是"本次最多追加多少个 SN"的上限;--order 决定从哪一端补起
python write_attachments.py --append --limit 10
python write_attachments.py --append --order desc --limit 10
# 指定总排号文件(每行一个,键类型 sn
python write_attachments.py --ids-file ids.txt
@@ -293,7 +302,9 @@ python write_attachments.py --mode fine --enable-other
```
运行结束后,除上面的 `ok/empty_param/not_found/error` 与"将写入行数/跳过数"统计外,还会打印一行
`[汇总]`:本批总耗时、LLM 调用次数、token 总量prompt/completion/total与加权缓存命中率便于核算成本。
`[汇总]`:本批**总耗时**(整批分类的真实墙钟)、**LLM 累计**(各任务自身耗时的累加,并发下大于墙钟,
括号标注并发数 `max_workers`、LLM 调用次数、token 总量prompt/completion/total与加权缓存命中率
便于核算成本与并发效率。
落库约定(与 `db.py` 常量、`write_attachments.py` 的转换逻辑保持一致):