- schema: expected_record unique key (site,waybill_no) -> (site,handover_no,waybill_no),
so the same waybill can belong to multiple handover batches without being overwritten
- store: dedup by (handover_no,waybill_no) instead of waybill_no alone; UPSERT conflict
key matches the new unique key; handover_no no longer force-overwritten on conflict
- store: add purge_expected, called on force re-download; deletes by business_date
(download day, same source as task date) rather than batch_out_date, which is offset
from the download day for sites like yunda where download day != outbound day
- store/runtime: thread force + target_date through ingest_task / _persist_to_db /
_site_undelivered_handler so force re-download purges then re-ingests in one tx
- db_compare: undelivered_pieces now sums per-waybill shortfall (total - arrived)
instead of expected_total - arrived_total, unaffected by unrelated/extra scans
The ZTO jQuery Date Range Picker always displays a dual-month view.
When today falls on the 1st (or early days) of a month, the same date
appears in both panels: a hidden ghost cell (month1, display:none) and
a visible cell (month2). Both carry the real-today CSS class, so .first
picks the hidden one, causing wait_for(visible) to timeout.
Replace DOM-based real-today time extraction with Python datetime
computation. Add _zto_find_visible_day to locate the actually visible
date cell (skipping hidden ghost cells, trying both midnight and
23:59:59 time variants). Fix _zto_flip_to_target_month to use the
same visibility-aware lookup.
Co-Authored-By: Claude <noreply@anthropic.com>
Replace Excel-based undelivered comparison with DB queries for all four
sites. The engine anchors on actual scan_time, reverse-lookups handover
batches, and compares expected vs actual waybill-by-waybill.
Shunxin SF waybills: use COUNT(*) instead of COUNT(DISTINCT piece_no)
since SF piece numbers are random and not derivable from the waybill.
Changes:
- db_compare.py: new module with compare_site_date(), compare_site_batch(),
write_result_excel(), and POST /compare API endpoint
- runtime.py: switch _site_undelivered_handler from compare.write_site_file
(Excel) to db_compare (DB); downloads succeed independently of comparison
- server.py: add POST /compare endpoint with date validation
- docs: implementation plan for Shunxin DB comparison
Co-Authored-By: Claude <noreply@anthropic.com>
The previous filter kept rows with non-empty handover numbers
(派件/签收 scans), which were duplicate rows. The correct logic
is to keep rows with empty handover numbers (到/接件 scans).
- store.py: change != "" to == "" in ingest filter
- compare.py: add same filter before comparison (previously missing)
Co-Authored-By: Claude <noreply@anthropic.com>
Aneng only supports querying data up to 31 days back. The previous
90-day cap allowed dates that would silently fail at download time,
so restrict the POST /tasks `date` validation to a 31-day window.
When the offset target date falls outside the current two-month view, flip .prev months (1 month/step) until the target day cell enters month1 view, instead of silently falling back to today. JS-dispatched clicks avoid the .date-range-length-tip hover interception.
Also switch the expected (#beginDate) day-cell click to _dom_click (matching actual #daterange): the range-length tooltip intercepts the second click on non-today cells during single-day range selection, causing timeouts.
Verified end-to-end: offset=45 (-> 2026-06-14, cross-month) expected task succeeded and downloaded 6-14 data without fallback.
Co-Authored-By: Claude <noreply@anthropic.com>
websocket.create_connection had no timeout, so a hung Runtime.evaluate
(Electron business tab not replying) blocked ws.recv() indefinitely — the
export-poll's 300s deadline and with_retry could never fire (observed as a
~22min hang on an 安能 expected download). A 15s socket timeout lets recv
raise WebSocketTimeoutException so wait_until/with_retry can fail and retry.
Co-Authored-By: Claude <noreply@anthropic.com>
Move the store.ingest_enabled() gate inside the main try/except. Previously
it sat outside as a standalone block: ingest_enabled() -> _load_pg_config()
raises FileNotFoundError when config.yaml is absent/malformed, which escaped
_persist_to_db, was caught by dispatch_task's outer except, and flipped a
successful download to FAILED -- violating the hook's never-raise invariant.
Now config errors print a [warn], record ok=False, and return silently.
Co-Authored-By: Claude <noreply@anthropic.com>
Spec for mounting PostgreSQL ingest as a best-effort, kind-level,
synchronous hook on runtime.dispatch_task after a successful download,
plus the 5-task implementation plan.
Co-Authored-By: Claude <noreply@anthropic.com>
