Import/calculations improvements

2026-06-11 19:32:20 -04:00
parent 3b2f51e6b8
commit 069a44bd54
19 changed files with 1175 additions and 308 deletions
@@ -0,0 +1,69 @@
+-- Migration 003: Item-level promo discounts + business-day (America/Chicago) bucketing
+-- (applied 2026-06-11, together with the IMPORT_METRICS_FIX_PLAN.md batch)
+--
+-- PROBLEM 1 — dropped item-level promo discounts (~$26K / 30 days):
+--   orders.js applied item-level discounts from order_discount_items only when the
+--   parent order_discounts row had discount_amount_subtotal > 0:
+--     SUM(CASE WHEN COALESCE(md.discount_amount_subtotal, 0) > 0 THEN id.amount ELSE 0 END)
+--   In the PHP source, item-level promo discounts (which = 2) are applied to the order
+--   total SEPARATELY from summary_discount_subtotal, so the gate zeroed essentially all
+--   of them (90d live check: of 10,010 type-10 promos, 8,070 had item rows but only 8 had
+--   discount_amount_subtotal > 0). Net effect: orders.discount understated, net_revenue /
+--   profit_30d / margin_30d overstated by ~10% of revenue, discounts_30d ~3x understated.
+--
+--   FIX (orders.js): fetch only order_discount_items rows with which = 2 (which = 1 rows
+--   are prices of free promo-added items, which = 3 are usage records), sum them
+--   unconditionally, and clamp each sale line's total discount to price * quantity.
+--   temp_main_discounts / temp_order_discounts staging removed (unused after the fix).
+--
+-- PROBLEM 2 — Europe/Berlin day bucketing:
+--   orders.date is timestamptz and the PG server timezone is Europe/Berlin, so ::date
+--   casts shifted every order placed after ~5 PM Central onto the NEXT calendar day in
+--   daily_product_snapshots (and skewed yesterday_sales, DOW patterns, forecast accuracy).
+--
+--   FIX (update_daily_snapshots.sql, backfill/rebuild_daily_snapshots.sql,
+--   update_product_metrics.sql): every day-bucketing cast is now
+--     (ts AT TIME ZONE 'America/Chicago')::date
+--   Supporting expression indexes:
+--     CREATE INDEX idx_orders_date_chicago ON orders (((date AT TIME ZONE 'America/Chicago')::date));
+--     CREATE INDEX idx_receivings_received_chicago ON receivings (((received_date AT TIME ZONE 'America/Chicago')::date));
+--
+-- ALSO IN THIS BATCH (same re-import/rebuild):
+--   * 'combined' order status (code 16) excluded from all sales aggregates, and a sweep
+--     in orders.js marks canceled/combined source orders (canceled = true) even though
+--     combine_orders zeroes date_placed (Fixes 4/5).
+--   * Returns now subtract COGS (returns_cogs) in daily snapshots (Fix 8).
+--   * return_rate_30d = returns / sales (Fix 9); gmroi_30d annualized ×12.17 (Fix 10).
+--   * stockout/avg-stock/service-level derived from stock_snapshots presence (Fix 7).
+--
+-- REQUIRED ACTION (cannot be fixed by SQL alone — discount values are baked into rows):
+--   1. Deploy updated orders.js + snapshot SQL files.
+--   2. Pause the recurring import:  touch inventory-server/.pause-auto-update
+--   3. FULL orders re-import:       INCREMENTAL_UPDATE=false node scripts/import-from-prod.js
+--   4. Rebuild snapshots:           psql -f scripts/metrics-new/backfill/rebuild_daily_snapshots.sql
+--   5. Recalculate metrics:         node scripts/calculate-metrics-new.js
+--   6. Resume:                      rm inventory-server/.pause-auto-update
+--
+-- EXPECTED AFTER RE-IMPORT: margin_30d down ~8-10 points (real, not a data incident),
+-- discounts_30d ~3x up, daily sales curves shifted onto correct business days.
+--
+-- VERIFICATION:
+-- (a) PG SUM(discount) over a 30-day window should approximate MySQL
+--     Σ summary_discount_subtotal (prorated) + Σ order_discount_items.amount (which=2)
+--     over the same orders.
+-- (b) Per-day units in daily_product_snapshots should match MySQL
+--     SELECT date_placed_onlydate, SUM(qty_ordered) FROM order_items JOIN _order ...
+--     WHERE order_status >= 20 GROUP BY 1   (MySQL stores Central days).
+-- (c) Migration 002 regression check (discount double-counting) still holds:
+SELECT
+    o.pid,
+    o.order_number,
+    o.price,
+    o.quantity,
+    o.discount,
+    (o.price * o.quantity - o.discount) as net_revenue
+FROM orders o
+WHERE o.pid IN (624756, 614513)
+ORDER BY o.date DESC
+LIMIT 10;
+-- Expected: discount 0 (or genuine promo amount) for regular sales; net close to gross.
@@ -0,0 +1,9 @@
+-- Migration 004: Map order status codes 45 and 67 to text
+--
+-- Follow-up to 001_map_order_statuses.sql: the orders.js orderStatusMap lacked
+-- codes 45 (payment_pending) and 67 (remote_send), so any such orders imported
+-- as numeric strings '45' / '67'. orders.js now maps them; this updates any
+-- existing rows (a full re-import also fixes them — safe to run either way).
+
+UPDATE orders SET status = 'payment_pending' WHERE status = '45';
+UPDATE orders SET status = 'remote_send'     WHERE status = '67';