feat(write_attachments): add --append mode, fix batch timing and SQL Server param limit
- Add --append flag: diff source-table 总排号 against existing Common.Attachment SN and classify/write only the missing ones. --limit caps the per-run append count, --order sets the direction. Backed by new fetch_existing_attachment_sns() in db.py. - Fix batch elapsed-time accounting: summarize_results() summed each task's per-item elapsed_ms, which overcounts under ThreadPoolExecutor concurrency (cumulative work time, not real wall-clock — 100 tasks on 8 workers reported ~5x the actual runtime). Callers now time classify_batch() via perf_counter and pass wall_clock_ms; both the write_attachments [汇总] line and main.py --summary report wall-clock separately from the cumulative sum. - Format durations >=1s in seconds (88851.4 ms -> 88.85 s) in the human-readable [汇总] line; structured JSON --summary fields stay in ms. - Chunk all IN (...) lists to 2000 items to respect SQL Server's 2100 bind-parameter hard limit (previously --append --limit 5000 failed at the fetch step with "COUNT 字段不正确"). Applied to fetch_params_by_ids (sn / id paths) and to the DELETE inside upsert_attachments. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
19
README.md
19
README.md
@@ -173,9 +173,13 @@ python main.py --sn 26B742,26B743 --summary
|
||||
"meta": {"elapsed_ms": 1476.6, "attempts": 1, "llm": {"model": "deepseek-v4-flash", "elapsed_ms": 1472.8, "usage": {"prompt_tokens": 1297, "completion_tokens": 15, "total_tokens": 1312, "prompt_cache_hit_tokens": 1280, "prompt_cache_miss_tokens": 17, "cache_hit_rate": 0.9869}}}
|
||||
```
|
||||
|
||||
`--summary` 打印的整批汇总字段与 `meta.llm.usage` 对应:`count` / `total_elapsed_ms` / `llm_calls` /
|
||||
各类 `total_*_tokens` / `total_cache_hit_tokens` / `total_cache_miss_tokens` / `avg_cache_hit_rate`
|
||||
(按 token 加权的整体命中率)。
|
||||
`--summary` 打印的整批汇总字段与 `meta.llm.usage` 对应:`count` / `wall_clock_ms` / `total_elapsed_ms` /
|
||||
`llm_calls` / 各类 `total_*_tokens` / `total_cache_hit_tokens` / `total_cache_miss_tokens` /
|
||||
`avg_cache_hit_rate`(按 token 加权的整体命中率)。
|
||||
|
||||
两个耗时字段的区别(`classify_batch` 是并发执行的):
|
||||
- `wall_clock_ms`:整批分类的**真实墙钟**耗时(在 `classify_batch` 外层用 `perf_counter` 计时),即你实际等的时间。
|
||||
- `total_elapsed_ms`:各结果自身耗时 `elapsed_ms` 的**累加**(相当于把这些任务串行跑的总时长),并发下会明显大于墙钟;二者之比 ≈ 并发增益(≈ `max_workers`)。
|
||||
|
||||
## 对话日志
|
||||
|
||||
@@ -265,6 +269,11 @@ python write_attachments.py --range 800,805
|
||||
# 组合:区间内降序、再取前 3 个
|
||||
python write_attachments.py --range 800,805 --order desc --limit 3
|
||||
|
||||
# 追加模式:仅补写源表中存在、但 Common.Attachment 还没有的总排号
|
||||
# --limit 此时是"本次最多追加多少个 SN"的上限;--order 决定从哪一端补起
|
||||
python write_attachments.py --append --limit 10
|
||||
python write_attachments.py --append --order desc --limit 10
|
||||
|
||||
# 指定总排号文件(每行一个,键类型 sn)
|
||||
python write_attachments.py --ids-file ids.txt
|
||||
|
||||
@@ -293,7 +302,9 @@ python write_attachments.py --mode fine --enable-other
|
||||
```
|
||||
|
||||
运行结束后,除上面的 `ok/empty_param/not_found/error` 与"将写入行数/跳过数"统计外,还会打印一行
|
||||
`[汇总]`:本批总耗时、LLM 调用次数、token 总量(prompt/completion/total)与加权缓存命中率,便于核算成本。
|
||||
`[汇总]`:本批**总耗时**(整批分类的真实墙钟)、**LLM 累计**(各任务自身耗时的累加,并发下大于墙钟,
|
||||
括号标注并发数 `max_workers`)、LLM 调用次数、token 总量(prompt/completion/total)与加权缓存命中率,
|
||||
便于核算成本与并发效率。
|
||||
|
||||
落库约定(与 `db.py` 常量、`write_attachments.py` 的转换逻辑保持一致):
|
||||
|
||||
|
||||
Reference in New Issue
Block a user