Refine incremental sync audit observability
- Add per-cycle correlation ID ([cyc:xxxxxxxx]) threaded through Python logs and SQL audit tables for end-to-end traceability of any divergence. - New ProductionDataBaseSync.SyncApplyRunLog table + @CycleID on usp_SyncApply for per-table apply auditing (pending/merged/deleted/applied, dead/error counts, duration). Audit write isolated in its own TRY/CATCH outside txn. - Capture/cleanup phases now emit full detail: Insert->Delete downgrade warning with record_id/log_id, dedup_skipped as apply-stall signal, per-file summary, and access log-id ranges on cleanup. - Queue health check surfaces error/dead rows with recent samples instead of silent accumulation (previously the top cause of data divergence). - sql_writer uses INSERT...SELECT...WHERE NOT EXISTS for observable dedup; idle cycles lowered to DEBUG with periodic heartbeat. - Backward compatible: old proc callers still work (CycleID nullable; legacy coarse-grained logging with one-time notice). Excluded from this commit: CODE.md, build_code_doc.py (doc generation).
This commit is contained in:
@@ -3,6 +3,13 @@
|
||||
Configures the root logger with a RotatingFileHandler (10 MB x 5, UTF-8) plus
|
||||
a console StreamHandler. The level and log path come from ``cfg.logging``.
|
||||
|
||||
Every record written to the log file carries a cycle correlation id
|
||||
(``[cyc:xxxxxxxx]``): ``sync.service.cycle`` allocates one per pass via
|
||||
``set_cycle_id`` and ``_CycleIdFilter`` injects it into each record, so every
|
||||
capture/apply/cleanup line of one pass -- and the matching
|
||||
``SyncApplyRunLog.CycleID`` rows on SQL Server -- can be correlated with a
|
||||
single grep. Outside a cycle (fullsync, compare, startup) the field is ``-``.
|
||||
|
||||
Log files are managed per-day: at startup any previously produced log
|
||||
(including the project's own ``sync.log`` and NSSM's ``nssm_*.log`` captures)
|
||||
is relocated into an ``Archive/`` subfolder next to the active log. Where a log
|
||||
@@ -10,6 +17,7 @@ lacks a timestamp, a ``-YYYY-MM-DD`` suffix is added (derived from its first
|
||||
log line, falling back to mtime) so historical files carry a date. The log
|
||||
root therefore only ever shows the current day's ``sync.log``.
|
||||
"""
|
||||
import contextvars
|
||||
import datetime
|
||||
import logging
|
||||
import logging.handlers
|
||||
@@ -20,6 +28,33 @@ import shutil
|
||||
|
||||
_LOG_DATE_RE = re.compile(r"^(\d{4}-\d{2}-\d{2})")
|
||||
|
||||
# Current cycle correlation id ("-" when not inside a service cycle).
|
||||
_cycle_id: contextvars.ContextVar[str] = contextvars.ContextVar(
|
||||
"sync_cycle_id", default="-"
|
||||
)
|
||||
|
||||
|
||||
def set_cycle_id(cycle_id: str | None) -> None:
|
||||
"""Set (or clear, with ``None``) the id stamped on every log record.
|
||||
|
||||
Called by ``sync.service.cycle`` at the start/end of each pass. The same
|
||||
id is passed to ``usp_SyncApply`` so SQL-side ``SyncApplyRunLog`` rows can
|
||||
be joined back to the exact log lines of the cycle that produced them.
|
||||
"""
|
||||
_cycle_id.set(cycle_id or "-")
|
||||
|
||||
|
||||
class _CycleIdFilter(logging.Filter):
|
||||
"""Inject the current cycle id into every record as ``record.cycle``.
|
||||
|
||||
Attached to the handlers (not the logger) so records emitted through any
|
||||
module logger -- capture, cleanup, access_reader, ... -- are covered.
|
||||
"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool: # noqa: A003
|
||||
record.cycle = _cycle_id.get()
|
||||
return True
|
||||
|
||||
|
||||
def _first_line_date(path: str) -> str | None:
|
||||
"""Best-effort extraction of the first log line's YYYY-MM-DD date."""
|
||||
@@ -118,11 +153,19 @@ def setup_logging(cfg_dict: dict | None):
|
||||
pass
|
||||
root.setLevel(level)
|
||||
|
||||
cycle_filter = _CycleIdFilter()
|
||||
|
||||
h = logging.handlers.RotatingFileHandler(
|
||||
path, maxBytes=10 * 1024 * 1024, backupCount=5, encoding="utf-8"
|
||||
)
|
||||
h.setFormatter(logging.Formatter("%(asctime)s %(levelname)s [%(name)s] %(message)s"))
|
||||
h.addFilter(cycle_filter)
|
||||
h.setFormatter(logging.Formatter(
|
||||
"%(asctime)s %(levelname)s [%(name)s] [cyc:%(cycle)s] %(message)s"
|
||||
))
|
||||
root.addHandler(h)
|
||||
|
||||
sh = logging.StreamHandler()
|
||||
sh.addFilter(cycle_filter)
|
||||
# Console keeps the short format (fullsync/compare are interactive there).
|
||||
sh.setFormatter(logging.Formatter("%(levelname)s %(message)s"))
|
||||
root.addHandler(sh)
|
||||
|
||||
Reference in New Issue
Block a user