Refine incremental sync audit observability

- Add per-cycle correlation ID ([cyc:xxxxxxxx]) threaded through Python
  logs and SQL audit tables for end-to-end traceability of any divergence.
- New ProductionDataBaseSync.SyncApplyRunLog table + @CycleID on usp_SyncApply
  for per-table apply auditing (pending/merged/deleted/applied, dead/error
  counts, duration). Audit write isolated in its own TRY/CATCH outside txn.
- Capture/cleanup phases now emit full detail: Insert->Delete downgrade
  warning with record_id/log_id, dedup_skipped as apply-stall signal,
  per-file summary, and access log-id ranges on cleanup.
- Queue health check surfaces error/dead rows with recent samples instead of
  silent accumulation (previously the top cause of data divergence).
- sql_writer uses INSERT...SELECT...WHERE NOT EXISTS for observable dedup;
  idle cycles lowered to DEBUG with periodic heartbeat.
- Backward compatible: old proc callers still work (CycleID nullable; legacy
  coarse-grained logging with one-time notice).

Excluded from this commit: CODE.md, build_code_doc.py (doc generation).
This commit is contained in:
Misaka_Company
2026-07-31 17:10:23 +08:00
parent 75fb6a3a01
commit 2ce03d21d9
9 changed files with 655 additions and 63 deletions

View File

@@ -3,6 +3,13 @@
Configures the root logger with a RotatingFileHandler (10 MB x 5, UTF-8) plus
a console StreamHandler. The level and log path come from ``cfg.logging``.
Every record written to the log file carries a cycle correlation id
(``[cyc:xxxxxxxx]``): ``sync.service.cycle`` allocates one per pass via
``set_cycle_id`` and ``_CycleIdFilter`` injects it into each record, so every
capture/apply/cleanup line of one pass -- and the matching
``SyncApplyRunLog.CycleID`` rows on SQL Server -- can be correlated with a
single grep. Outside a cycle (fullsync, compare, startup) the field is ``-``.
Log files are managed per-day: at startup any previously produced log
(including the project's own ``sync.log`` and NSSM's ``nssm_*.log`` captures)
is relocated into an ``Archive/`` subfolder next to the active log. Where a log
@@ -10,6 +17,7 @@ lacks a timestamp, a ``-YYYY-MM-DD`` suffix is added (derived from its first
log line, falling back to mtime) so historical files carry a date. The log
root therefore only ever shows the current day's ``sync.log``.
"""
import contextvars
import datetime
import logging
import logging.handlers
@@ -20,6 +28,33 @@ import shutil
_LOG_DATE_RE = re.compile(r"^(\d{4}-\d{2}-\d{2})")
# Current cycle correlation id ("-" when not inside a service cycle).
_cycle_id: contextvars.ContextVar[str] = contextvars.ContextVar(
"sync_cycle_id", default="-"
)
def set_cycle_id(cycle_id: str | None) -> None:
"""Set (or clear, with ``None``) the id stamped on every log record.
Called by ``sync.service.cycle`` at the start/end of each pass. The same
id is passed to ``usp_SyncApply`` so SQL-side ``SyncApplyRunLog`` rows can
be joined back to the exact log lines of the cycle that produced them.
"""
_cycle_id.set(cycle_id or "-")
class _CycleIdFilter(logging.Filter):
"""Inject the current cycle id into every record as ``record.cycle``.
Attached to the handlers (not the logger) so records emitted through any
module logger -- capture, cleanup, access_reader, ... -- are covered.
"""
def filter(self, record: logging.LogRecord) -> bool: # noqa: A003
record.cycle = _cycle_id.get()
return True
def _first_line_date(path: str) -> str | None:
"""Best-effort extraction of the first log line's YYYY-MM-DD date."""
@@ -118,11 +153,19 @@ def setup_logging(cfg_dict: dict | None):
pass
root.setLevel(level)
cycle_filter = _CycleIdFilter()
h = logging.handlers.RotatingFileHandler(
path, maxBytes=10 * 1024 * 1024, backupCount=5, encoding="utf-8"
)
h.setFormatter(logging.Formatter("%(asctime)s %(levelname)s [%(name)s] %(message)s"))
h.addFilter(cycle_filter)
h.setFormatter(logging.Formatter(
"%(asctime)s %(levelname)s [%(name)s] [cyc:%(cycle)s] %(message)s"
))
root.addHandler(h)
sh = logging.StreamHandler()
sh.addFilter(cycle_filter)
# Console keeps the short format (fullsync/compare are interactive there).
sh.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
root.addHandler(sh)