Import/calculations improvements

.claude/CLAUDE.md

@@ -0,0 +1,172 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a full-stack inventory management system with a React + TypeScript frontend and Node.js/Express backend using PostgreSQL. The system includes product management, analytics, forecasting, purchase orders, and a comprehensive dashboard for business metrics.
+
+**Monorepo Structure:**
+- `inventory/` - Vite-based React frontend with TypeScript
+- `inventory-server/` - Express backend API server
+- Root `package.json` contains shared dependencies
+
+## Development Commands
+
+### Frontend (inventory/)
+```bash
+cd inventory
+npm run dev        # Start dev server on port 5175
+npm run build      # Build for production (outputs to build/ then copies to ../inventory-server/frontend/build)
+npm run lint       # Run ESLint
+npm run preview    # Preview production build
+```
+
+### Backend (inventory-server/)
+```bash
+cd inventory-server
+npm run dev        # Start with nodemon (auto-reload)
+npm start          # Start server (production)
+npm run prod       # Start with PM2 for production
+npm run prod:stop  # Stop PM2 instance
+npm run prod:restart  # Restart PM2 instance
+npm run prod:logs  # View PM2 logs
+npm run setup      # Create required directories (logs, uploads)
+```
+
+## Architecture
+
+### Frontend Architecture
+
+**Router Structure:** React Router with lazy loading for code splitting:
+- Main chunks: Core inventory, Dashboard, Product Import, Chat Archive
+- Authentication flow uses `RequireAuth` and `Protected` components with permission-based access
+- All routes except `/login` and `/small` require authentication
+
+**Key Directories:**
+- `src/pages/` - Top-level page components (Overview, Products, Analytics, Dashboard, etc.)
+- `src/components/` - Organized by feature (dashboard/, products/, analytics/, etc.)
+- `src/components/ui/` - shadcn/ui components
+- `src/types/` - TypeScript type definitions
+- `src/contexts/` - React contexts (AuthContext, DashboardScrollContext)
+- `src/hooks/` - Custom React hooks (use-toast, useDebounce, use-mobile)
+- `src/utils/` - Utility functions (emojiUtils, productUtils, naturalLanguagePeriod)
+- `src/services/` - API service layer
+- `src/config/` - Configuration files
+
+**State Management:**
+- React Context for auth and global state
+- @tanstack/react-query for server state management
+- zustand for client state management
+- Local storage for auth tokens, session storage for login state
+
+**Key Dependencies:**
+- UI: Radix UI primitives, shadcn/ui, Tailwind CSS, Framer Motion
+- Data: @tanstack/react-table, react-data-grid, @tanstack/react-virtual
+- Forms: react-hook-form, zod
+- Charts: recharts, chart.js, react-chartjs-2
+- File handling: xlsx for Excel export, react-dropzone for uploads
+- Other: axios for HTTP, date-fns/luxon for dates
+
+**Path Alias:** `@/` maps to `./src/`
+
+### Backend Architecture
+
+**Entry Point:** `inventory-server/src/server.js`
+
+**Key Directories:**
+- `src/routes/` - Express route handlers (products, dashboard, analytics, import, etc.)
+- `src/middleware/` - Express middleware (CORS, auth, etc.)
+- `src/utils/` - Utility functions (database connection, API helpers)
+- `src/types/` - Type definitions (e.g., status-codes)
+
+**Database:**
+- PostgreSQL with connection pooling (pg library)
+- Pool initialized in `utils/db.js` via `initPool()`
+- Pool attached to `app.locals.pool` for route access
+- Environment variables loaded from `/var/www/inventory/.env` (production path)
+
+**API Routes:** All prefixed with `/api/`
+- `/api/products` - Product CRUD operations
+- `/api/dashboard` - Dashboard metrics and data
+- `/api/analytics` - Analytics and reporting
+- `/api/orders` - Order management
+- `/api/purchase-orders` - Purchase order management
+- `/api/csv` - CSV import/export (data management)
+- `/api/import` - Product import workflows
+- `/api/config` - Configuration management
+- `/api/metrics` - System metrics
+- `/api/ai-validation` - AI-powered validation
+- `/api/ai-prompts` - AI prompt management
+- `/api/templates` - Template management
+- `/api/reusable-images` - Image management
+- `/api/categoriesAggregate`, `/api/vendorsAggregate`, `/api/brandsAggregate` - Aggregate data endpoints
+
+**Authentication:**
+- External auth service at `/auth-inv` endpoint
+- Token-based authentication (Bearer tokens)
+- Frontend stores tokens in localStorage
+- Protected routes verify tokens via auth service `/me` endpoint
+
+**File Uploads:**
+- Multer middleware for file handling
+- Uploads directory: `inventory-server/uploads/`
+
+### Development Proxy Setup
+
+The Vite dev server (port 5175) proxies API requests to `https://inventory.kent.pw`:
+- `/api/*` → production API
+- `/auth-inv/*` → authentication service
+- `/chat-api/*` → chat service
+- `/uploads/*` → uploaded files
+- Various third-party services (Aircall, Klaviyo, Meta, Gorgias, Typeform, ACOT, Clarity)
+
+### Build Process
+
+When building the frontend:
+1. TypeScript compilation (`tsc -b`)
+2. Vite build (outputs to `inventory/build/`)
+3. Custom Vite plugin copies build to `inventory-server/frontend/build/`
+4. Manual chunks for vendor splitting (react-vendor, ui-vendor, query-vendor)
+
+## Testing
+
+Run tests for individual components or features:
+```bash
+# No test suite currently configured
+# Tests would typically use Jest or Vitest with React Testing Library
+```
+
+## Common Development Workflows
+
+### Adding a New Page
+1. Create page component in `inventory/src/pages/YourPage.tsx`
+2. Add lazy import in `inventory/src/App.tsx`
+3. Add route with `<Protected>` wrapper and permission check
+4. Add corresponding backend route in `inventory-server/src/routes/`
+5. Update permission system if needed
+
+### Adding a New API Endpoint
+1. Create or update route file in `inventory-server/src/routes/`
+2. Use `executeQuery()` helper for database queries
+3. Register router in `inventory-server/src/server.js`
+4. Frontend can access at `/api/{route-name}`
+
+### Working with Database
+- Use parameterized queries: `executeQuery(sql, [param1, param2])`
+- Pool is accessed via `db.getPool()` or `app.locals.pool`
+- Connection helper: `db.getConnection()` returns a client for transactions
+
+### Permissions System
+- User permissions stored in `user.permissions` array (permission codes)
+- Check permissions in `<Protected page="permission_code">` component
+- Admin users (`is_admin: true`) have access to all pages
+
+## Important Notes
+
+- Environment variables must be configured in `/var/www/inventory/.env` for production
+- The frontend expects the backend at `/api` (proxied in dev, served together in production)
+- PM2 is used for production process management
+- Database uses PostgreSQL with SSL support (configurable via `DB_SSL` env var)
+- File uploads stored in `inventory-server/uploads/` directory
+- Build artifacts in `inventory/build/` are copied to `inventory-server/frontend/build/`

.gitignore

@@ -67,4 +67,23 @@ inventory-server/scripts/.fuse_hidden00000fa20000000a
 
 .VSCodeCounter/
 .VSCodeCounter/*
-.VSCodeCounter/**/*
+.VSCodeCounter/**/*
+
+*/chat/db-convert/db/*
+*/chat/db-convert/mongo_converter_env/*
+
+# Ignore compiled Vite config to avoid duplication
+vite.config.js
+inventory-server/inventory_backup.sql
+chat-files.tar.gz
+chat-migration*/
+**/chat-migration*/
+chat-migration*/**
+**/chat-migration*/**
+
+venv/
+venv/**
+**/venv/*
+**/venv/**
+
+inventory-server/data/taxonomy-embeddings.json

CLAUDE.md

@@ -0,0 +1,4 @@
+* Avoid using glob tool for search as it may not work properly on this codebase. Search using bash instead.
+* If you use the task tool to have an agent investigate something, make sure to let it know to avoid using glob
+* Prefer solving tasks in a single session. Only spawn subagents for genuinely independent workstreams.
+* The postgres/query tool is not working and not connected to the current version of the database. If you need to query the database for any reason you can use "ssh netcup" and use psql on the server with inventory_readonly 6D3GUkxuFgi2UghwgnUd

FORECAST_FIX_PLAN.md

@@ -0,0 +1,343 @@
+# Forecast Accuracy Fix Plan
+
+**Written:** 2026-06-10, from a code + live-data review of the forecasting pipeline.
+**Goal:** eliminate the systematic ~1.7–2x over-forecast bias, recover demand the model currently ignores, and fix the accuracy measurement so improvements are visible and long-lead forecasts are validated.
+
+Read this whole document before starting. Fixes are grouped into phases; each phase is independently deployable and has its own validation step. Line numbers are as of 2026-06-10 — re-locate by function name if the file has drifted.
+
+---
+
+## 1. Diagnosis summary (measured 2026-06-10)
+
+The dashboard headline is **202% WMAPE**. Decomposition of that number, all measured against `forecast_accuracy` run 129 and ad-hoc queries:
+
+| Finding | Evidence |
+|---|---|
+| Daily-grain WMAPE has a ~190% *floor* for this catalog | Avg demand ≈ 0.11 units/product/day. A perfect rate forecast of intermittent demand scores ≈ 2e^−λ ≈ 190%. A trivial trailing-30d-average naive forecast scores **204%** on the same products/days; the engine scores 221% (slightly *worse than naive*). |
+| Same forecasts at 21-day-per-product grain: **109%**; bias-corrected: **75%** | Half the headline is metric grain, most of the rest is bias. |
+| Aggregate over-forecast **+70%** (227,690 forecast vs 133,861 actual units) | Portfolio daily ratio is 1.5–2.5x on most days. |
+| Decay phase 2.47x over (fc 51,675 / act 20,915) | Root cause F1: velocity inflated **4.07x** (measured: 1.353 vs true 0.332 units/day) by averaging over sparse snapshot rows. |
+| Preorder phase 2.15x over (fc 67,212 / act 31,189) | Root cause F4: launch curve applied at age=0 starting *today*, ignoring that the product hasn't arrived. |
+| Mature phase 1.69x over (fc 57,857 / act 34,313) | Root causes F2 (history edge truncation) + F3 (seasonal double-count). |
+| Dormant products sold **16,180 units** (~11% of demand) against zero forecasts | Root cause F5; also excluded from the headline metric, so invisible. |
+| All 879,800 accuracy samples are in the **1–7d lead bucket** | Root cause F7: archiving design only ever saves yesterday's slice. 30–90d forecasts (what purchasing uses) are never validated. |
+| Launch phase is healthy: WMAPE 100%, bias −6%, beats naive | The lifecycle-curve concept works; its calibration inputs are broken. Don't redesign it. |
+
+**Key data fact** underlying several fixes: `daily_product_snapshots` is **activity-based and sparse** — only ~500–1,800 of ~38K products have a row on a given day. Verified: every pid-day with an order DOES have a snapshot row and units match (5,234/5,234 pid-days, 8,980 vs 8,984 units over 7 days). So *missing row = zero sales*, and any query that aggregates over only the rows that exist is averaging over sold-days.
+
+---
+
+## 2. Environment & operational notes
+
+- **Files:** engine is `inventory-server/scripts/forecast/forecast_engine.py`; orchestrator `run_forecast.js` in the same dir; consumer endpoints in `inventory-server/src/routes/dashboard.js` (`/forecast/metrics` ~line 308, `/forecast/accuracy` ~line 647); overview UI in `inventory/src/components/overview/ForecastMetrics.tsx` and `ForecastAccuracy.tsx`.
+- **Local `inventory-server/` is NFS-mounted to `/var/www/inventory/` on the netcup server.** Edits made locally appear on the server immediately — no copy step. Do NOT run bulk `grep`/`find`/`node --check` over `inventory-server/` locally (the mount hangs); `ssh netcup` and run them there.
+- **Avoid the glob tool** for search in this repo; use bash (`grep`/`rg` via ssh for server-side trees).
+- **Scheduling:** the engine runs daily at **09:30:01 server time** (runs table is conclusive), but the cron entry is NOT in matt's crontab, `/etc/cron.d`, or pm2. Likely root's crontab (`sudo crontab -l` to confirm). You do not need to touch the schedule for these fixes; just know a run fires at 09:30 daily and occasionally skips days (e.g. 2026-06-07/08).
+- **Manual test runs:** `ssh netcup`, then `cd /var/www/inventory/scripts/forecast && node run_forecast.js`. Takes ~3.5–4 min. Safe to run any time: the engine TRUNCATEs and rebuilds `product_forecasts`, archives prior past-dated rows, and records a new `forecast_runs` row. Python deps live in the server venv (`venv/`); `run_forecast.js` handles env + venv automatically.
+- **DB access for validation:** `ssh netcup`, then `PGPASSWORD=6D3GUkxuFgi2UghwgnUd psql -h localhost -U inventory_readonly -d inventory_db`. The engine itself connects with the write user via env vars loaded from `/var/www/inventory/.env` — schema changes should be made idempotently *inside the engine code* (the file already uses `CREATE TABLE IF NOT EXISTS` / `CREATE INDEX IF NOT EXISTS`; use `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` the same way) so no manual migration is needed.
+- **Python gotchas already handled in this file (don't regress):** numpy types must go through the registered psycopg2 adapters; `pd.Series.combine_first()` keeps zeros over real data — use `reindex(..., fill_value=0.0)`.
+- Engine runtime budget: currently ~212–227s. Phases 1–2 shouldn't move it meaningfully; Phase 3's extra archiving adds one INSERT…SELECT. If runtime balloons past ~6 min, investigate before shipping.
+- `--backfill` mode (`backfill_accuracy_data`) is an in-sample backtest using the *old* formulas. **Do not run it anymore**; there is enough real out-of-sample history. Updating it to match the new logic is optional/low priority (F11).
+
+---
+
+## Phase 1 — Bias bugs in the engine (no schema changes)
+
+### F1. Decay velocity: stop averaging over sparse snapshot rows
+
+**Where:** `forecast_engine.py`, `batch_load_product_data()`, the decay query (~lines 697–710).
+
+**Problem:** `AVG(COALESCE(dps.units_sold, 0))` runs over only the snapshot rows that exist — mostly sold-days. Measured inflation on the current 975 decay products: **4.07x** (1.353 vs 0.332 true units/day). This feeds `compute_scale_factor()` for the decay phase and is the single largest bias source.
+
+**Fix:** divide the sum by calendar days in the window, clipped to the product's age (decay products are 14–60 days old, so a 20-day-old product's window is 20 days, not 30):
+
+```sql
+SELECT dps.pid,
+    SUM(COALESCE(dps.units_sold, 0))::float
+      / GREATEST(LEAST(30, (CURRENT_DATE - pm.date_first_received::date)), 1) AS avg_daily
+FROM daily_product_snapshots dps
+JOIN product_metrics pm ON pm.pid = dps.pid
+WHERE dps.pid = ANY(%s)
+  AND dps.snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
+  AND dps.snapshot_date >= pm.date_first_received::date
+GROUP BY dps.pid, pm.date_first_received
+```
+
+No Python-side changes needed; `data['decay_velocity']` keeps the same shape. Products with zero snapshot rows in the window still get no entry → existing `scale = 1.0` fallback applies (acceptable: decay classification requires `sales_velocity_daily > 0`, so truly dead products don't reach this path).
+
+### F2. Mature history: reindex over the full calendar window
+
+**Where:** `forecast_engine.py`, `forecast_mature()` (~lines 833–836).
+
+**Problem:** `hist.set_index('snapshot_date').resample('D').sum()` only spans first-snapshot → last-snapshot. Interior gaps correctly become zeros, but **leading and trailing quiet periods are absent**, so the Holt level is fitted on the product's busy span. A marginal mature product whose activity clusters in 2 of the last 8 weeks gets a level ~4x too high.
+
+**Fix:** replace the resample with an explicit reindex over the full `EXP_SMOOTHING_WINDOW` ending yesterday:
+
+```python
+hist = history_df.copy()
+hist['snapshot_date'] = pd.to_datetime(hist['snapshot_date'])
+hist = hist.set_index('snapshot_date')['units_sold']
+full_index = pd.date_range(
+    end=pd.Timestamp(date.today() - timedelta(days=1)),
+    periods=EXP_SMOOTHING_WINDOW, freq='D')
+series = hist.reindex(full_index, fill_value=0.0).values.astype(float)
+```
+
+Notes: (pid, snapshot_date) is unique in `daily_product_snapshots`, so no duplicate-index risk. `observed_mean` and the `cap` recompute over the full window automatically (intended — the cap gets correspondingly tighter). Mature products are by definition >60 days old, so the 60-day window never predates first receipt. Do NOT use `combine_first` (see gotchas above).
+
+### F3. Stop double-applying the monthly seasonal index
+
+**Where:** `forecast_engine.py`, `generate_all_forecasts()` — the `seasonal_multipliers` pre-compute (~lines 959–961) and application (~line 1050).
+
+**Problem:** every per-product calibration (decay velocity, mature Holt level, launch first-week scale, preorder rate, slow-mover velocity) is fitted on *raw recent actuals*, which already embed the current month's seasonal level. The forecast then multiplies by the **absolute** monthly index of the target date. Example from the live indices (`forecast_runs.phase_counts` for run 129): May = 1.224 (sale month), June = 0.982. Early-June forecasts were calibrated on May-sale-inflated velocities and barely discounted — a structural ~25% over-forecast at that transition, and it'll be worse around November (1.316).
+
+**Fix:** apply the seasonal index *relative to the calibration period*. Compute a calibration index as the average monthly index over the trailing 30 calendar days (robust at month boundaries), then divide:
+
+```python
+today = date.today()
+trailing = [today - timedelta(days=i) for i in range(1, 31)]
+calibration_index = float(np.mean([monthly_indices.get(d.month, 1.0) for d in trailing]))
+seasonal_multipliers = [
+    monthly_indices.get(d.month, 1.0) / max(calibration_index, 0.1)
+    for d in forecast_dates
+]
+```
+
+Leave the DOW multipliers absolute — every calibration is a multi-week average and therefore DOW-neutral, so reshaping by absolute DOW indices is correct.
+
+**Optional sub-fix (same area, low priority):** the monthly indices are computed from a single trailing 365-day window, so each month appears once and YoY growth contaminates "seasonality". A cheap improvement is widening `SEASONAL_LOOKBACK_DAYS` to 730 and averaging the two observations of each month. Do this only after the main fixes are validated.
+
+### Phase 1 validation
+
+Deploy (edit locally; NFS propagates), run the engine manually once, wait for 3–5 daily cycles, then:
+
+```sql
+-- Portfolio ratio per day (target: drifts from ~2.0 toward 0.8–1.3)
+WITH ranked AS (
+  SELECT pfh.pid, pfh.forecast_date, pfh.forecast_units, pfh.lifecycle_phase,
+    ROW_NUMBER() OVER (PARTITION BY pfh.pid, pfh.forecast_date ORDER BY fr.started_at DESC) rn
+  FROM product_forecasts_history pfh
+  JOIN forecast_runs fr ON fr.id = pfh.run_id
+  WHERE pfh.forecast_date >= CURRENT_DATE - 7)
+SELECT r.forecast_date, round(SUM(r.forecast_units),0) AS fc,
+  SUM(COALESCE(dps.units_sold,0)) AS act,
+  round(SUM(r.forecast_units)/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS ratio
+FROM ranked r
+LEFT JOIN daily_product_snapshots dps ON dps.pid = r.pid AND dps.snapshot_date = r.forecast_date
+WHERE r.rn = 1 AND r.lifecycle_phase != 'dormant'
+GROUP BY 1 ORDER BY 1;
+```
+
+Also check `forecast_accuracy` `by_phase` rows for the newest run: decay bias should fall from +0.35 toward ~0, mature from +0.17 toward ~0. (Accuracy lags ~1 day behind each fix since it evaluates yesterday's forecasts.)
+
+---
+
+## Phase 2 — Demand the model currently ignores or mistimes
+
+### F4. Preorder: forecast the preorder rate until arrival, launch curve after
+
+**Where:** `forecast_engine.py` — `batch_load_product_data()` (add arrival dates), `generate_all_forecasts()` preorder branch (~lines 1005–1009), and `forecast_from_curve()` (or a small wrapper).
+
+**Problem:** preorder products run the launch curve from `age=0` starting **today**, i.e. full first-week launch sales while the product is still weeks from arriving. Actual preorder-period sales are a much slower trickle.
+
+**Fix:**
+
+1. Batch-load each preorder product's expected arrival from `purchase_orders` (line-item grain: it has `pid` and `expected_date` directly). Open statuses verified against live data: `created`, `ordered`, `electronically_sent`, `receiving_started` (~705 open line items currently have a future `expected_date`):
+
+```sql
+SELECT pid, MIN(expected_date) AS expected_arrival
+FROM purchase_orders
+WHERE pid = ANY(%s)
+  AND status IN ('created', 'ordered', 'electronically_sent', 'receiving_started')
+  AND expected_date IS NOT NULL
+  AND expected_date >= CURRENT_DATE
+GROUP BY pid
+```
+
+Fallbacks, in order: (a) an open PO with a *past* `expected_date` → assume arrival in 7 days; (b) no PO at all → arrival in 14 days (and log a counter of how many hit this default).
+
+2. In the preorder branch, build the daily array piecewise. Let `days_until_arrival = (expected_arrival - today).days`:
+   - Days `0 .. days_until_arrival-1`: flat observed preorder daily rate = `preorder_sales[pid] / max(preorder_days[pid], 1)` (both already batch-loaded), clamped to ≤ the curve's scaled week-0 daily value.
+   - Days `days_until_arrival .. horizon`: `forecast_from_curve(curve_info, scale, age_days=0, ...)` shifted so the curve's day 0 lands on the arrival date (i.e. pass `horizon_days - days_until_arrival` and offset into the output array).
+   - Keep the existing `compute_scale_factor('preorder', ...)` for the post-arrival curve; the pre-arrival segment doesn't use it.
+
+This is consistent with how the reference curves were built: historical preorder units were recorded on their **order dates** (pre-arrival), so week-0 of the fitted curves reflects post-receipt orders, not the backlog.
+
+### F5. Dormant products: small positive rate instead of hard zero, and count them
+
+**Where:** `forecast_engine.py` — `generate_all_forecasts()` dormant branch (~lines 1040–1042), `batch_load_product_data()`, and `compute_accuracy()`.
+
+**Problem:** all ~28K dormant products are forecast at exactly 0, yet they sold 16,180 units in the eval window (~11% of all demand) — restocks, promos, long-tail. Worse, dormant is *excluded* from the headline accuracy filter, so this miss is invisible.
+
+**Fix (cheap version, do this now):**
+
+1. Batch-load a trailing-180-day order rate for dormant products (11,362 of them have ≥1 sale in 180d — verified):
+
+```sql
+SELECT o.pid, SUM(o.quantity) / 180.0 AS rate
+FROM orders o
+WHERE o.pid = ANY(%s)
+  AND o.canceled IS DISTINCT FROM TRUE
+  AND o.date >= CURRENT_DATE - INTERVAL '180 days'
+GROUP BY o.pid
+```
+
+2. Dormant branch: if the product has a rate > 0, forecast it flat with `method = 'velocity'`; else keep zeros with `method = 'zero'`. Apply the same DOW/seasonal multipliers as everything else (automatic — they're applied after the branch).
+3. In `compute_accuracy()`, add a second overall row: `metric_type='overall', dimension_value='all_incl_dormant'` with no dormant filter (keep the existing `'all'` row unchanged for trend continuity). One extra entry in the `dimensions`/`filter_clauses` dicts.
+
+**Upgrade path (optional, Phase 4):** replace flat rates for `slow_mover` + dormant-with-sales with TSB (Teunter–Syntetos–Babai), the standard intermittent-demand method with obsolescence handling. Per product over a daily series `d_t` (build it from snapshots the F2 way — full calendar reindex):
+
+```
+if d_t > 0:  p_t = p_{t-1} + β·(1 − p_{t-1});  z_t = z_{t-1} + α·(d_t − z_{t-1})
+else:        p_t = p_{t-1}·(1 − β);            z_t = z_{t-1}
+forecast = p_T · z_T   (flat across horizon)
+```
+
+Start with α=0.1, β=0.05, initialize p = (nonzero days / total days), z = mean of nonzero demands. Scope: slow_mover (~6K) + dormant with 180d sales (~11K); series from up to 180 days of snapshots (sparse rows → ~manageable volume). Only do this after Phase 3 measurement exists to prove it beats the flat rates.
+
+### Phase 2 validation
+
+After 3–5 cycles: preorder `by_phase` bias should drop from +0.85 toward < +0.3; the new `all_incl_dormant` row should appear and its `total_actual_units` minus `'all'`'s should be largely *covered* rather than all-miss (dormant `bias` rising from −1.36 toward ~−0.3 or better).
+
+---
+
+## Phase 3 — Fix the measurement (schema + engine + API + UI)
+
+> Without this phase you cannot see whether Phases 1–2 worked except by ad-hoc SQL, the lead-time chart stays a single bucket forever, and the dashboard keeps displaying a number with a 190% floor in red.
+
+### F7. Archive long-lead forecasts so 15/30/60/90d accuracy exists
+
+**Where:** `forecast_engine.py` — `archive_forecasts()` (~lines 1086–1154), `compute_accuracy()` CTE (~lines 1201–1228).
+
+**Problem:** the current design archives only *past-dated* rows of the previous run before truncation. With daily runs, that's only ever the 1-day-ahead slice — all 879,800 accuracy samples sit in the '1-7d' bucket and the longer buckets in the UI chart can never populate. Purchasing decisions ride on 30–60d forecasts that are never validated.
+
+**Fix:**
+
+1. Keep the existing past-date archiving exactly as is (it provides dense short-lead coverage).
+2. After `generate_all_forecasts()` completes, additionally archive a **sampled set of future leads** from the new run, non-dormant only, attributed to the *current* run id (correct attribution, unlike the past-date path which attributes to the previous run):
+
+```sql
+INSERT INTO product_forecasts_history
+    (run_id, pid, forecast_date, forecast_units, forecast_revenue,
+     lifecycle_phase, forecast_method, confidence_lower, confidence_upper, generated_at)
+SELECT %(run_id)s, pid, forecast_date, forecast_units, forecast_revenue,
+    lifecycle_phase, forecast_method, confidence_lower, confidence_upper, generated_at
+FROM product_forecasts
+WHERE lifecycle_phase != 'dormant'
+  AND forecast_date - CURRENT_DATE IN (7, 14, 30, 60, 89)
+ON CONFLICT (run_id, pid, forecast_date) DO NOTHING
+```
+
+Volume: ~10K non-dormant products × 5 leads ≈ 50K rows/day; the existing 90-day prune (`forecast_date < CURRENT_DATE - 90`) bounds steady state at a few million rows. Note future-dated rows survive until their date passes + 90 days — that's intended.
+
+3. **CRITICAL companion change** in `compute_accuracy()`: the accuracy CTE must now exclude not-yet-realized rows, or future-dated archives get scored against actual=0:
+
+```sql
+FROM product_forecasts_history pfh
+JOIN forecast_runs fr ON fr.id = pfh.run_id
+WHERE pfh.forecast_date < CURRENT_DATE          -- ADD THIS
+```
+
+4. **Dedup semantics change.** Today's `ROW_NUMBER() OVER (PARTITION BY pid, forecast_date ORDER BY started_at DESC)` keeps only the latest (= shortest-lead) row per pid/date, which would silently discard all the new long-lead rows. Restructure:
+   - Compute `lead_days = forecast_date - started_at::date` and the lead bucket *inside* `ranked_history`.
+   - For `by_lead_time`: dedup `PARTITION BY pid, forecast_date, lead_bucket` (one sample per pid/date/bucket, latest run wins within a bucket).
+   - For everything else (`overall`, `by_phase`, `by_method`, `daily`, and the new weekly metric below): restrict to `lead_days BETWEEN 0 AND 6` and keep the existing per-(pid, date) dedup. This preserves the current meaning of the headline metrics (short-lead) while the lead-time table becomes real.
+
+### F8. Track a naive baseline (forecast value-added)
+
+**Where:** `archive_forecasts()` (both INSERT paths), `compute_accuracy()`, `forecast_accuracy` schema, `/forecast/accuracy` endpoint.
+
+**Problem:** the engine currently *loses* to a trailing-average naive forecast (221% vs 204% daily WMAPE) and nothing on the dashboard would ever reveal that. Every accuracy improvement should be judged as value-over-naive.
+
+**Fix:**
+
+1. Schema (idempotent, in the ensure blocks): `ALTER TABLE product_forecasts_history ADD COLUMN IF NOT EXISTS naive_units NUMERIC(10,2);` and `ALTER TABLE forecast_accuracy ADD COLUMN IF NOT EXISTS naive_wmape NUMERIC(10,4), ADD COLUMN IF NOT EXISTS fva NUMERIC(10,4);`
+2. Populate `naive_units` during both archive INSERTs via a join — naive = flat trailing-28-day average daily units as of archive time (28 days = DOW-balanced; information available at generation; same value at every lead, which is exactly what a naive baseline means):
+
+```sql
+LEFT JOIN (
+    SELECT o.pid, SUM(o.quantity) / 28.0 AS naive_daily
+    FROM orders o
+    WHERE o.canceled IS DISTINCT FROM TRUE
+      AND o.date >= CURRENT_DATE - INTERVAL '28 days' AND o.date < CURRENT_DATE
+    GROUP BY o.pid
+) nv ON nv.pid = pf.pid
+-- select COALESCE(nv.naive_daily, 0) AS naive_units
+```
+
+3. In `compute_accuracy()`, add to each dimension's aggregate: `SUM(ABS(naive_units - actual_units)) / NULLIF(SUM(actual_units),0) AS naive_wmape` and store `fva = 1 - wmape / naive_wmape` (NULL-safe). Rows archived before this change have `naive_units` NULL — treat NULL as excluded (`FILTER (WHERE naive_units IS NOT NULL)` on the naive sums) rather than as zero.
+4. Endpoint: include `naiveWmape` and `fva` in the `overall` (and per-phase) payload of `/dashboard/forecast/accuracy` in `dashboard.js`.
+
+### F9. Weekly-grain headline metric + bias as a percentage
+
+**Where:** `compute_accuracy()`, `/forecast/accuracy` endpoint, `ForecastAccuracy.tsx`.
+
+**Problem:** daily-grain WMAPE on this catalog has a ~190% floor — as a headline it's noise. The informative numbers are (a) weekly-per-product WMAPE (currently ~109%, target ~70–85% post-fix) and (b) aggregate bias, which the UI currently renders as `+0.108 units` — indistinguishable from zero while the reality is +70%.
+
+**Fix:**
+
+1. New metric in `compute_accuracy()`: `metric_type='overall_weekly', dimension_value='all'`. Definition: using the short-lead deduped rows (lead ≤ 6, non-dormant), aggregate per `(pid, date_trunc('week', forecast_date))` keeping only complete weeks (`COUNT(*) = 7`), then `WMAPE = SUM(ABS(fc_week − act_week)) / SUM(act_week)`, excluding pid-weeks where both are 0. Store sample_size = number of pid-weeks. Compute `naive_wmape`/`fva` the same way from `naive_units`.
+2. Endpoint: expose as `overallWeekly`; also add a weekly variant to the `accuracyTrend` query (`metric_type='overall_weekly'`). The trend will start empty (old runs lack the row) — that's fine; don't backfill.
+3. `ForecastAccuracy.tsx`:
+   - Headline WMAPE → `overallWeekly.wmape`, labeled "WMAPE (weekly)". Keep daily WMAPE available in a tooltip if desired.
+   - Color thresholds for weekly grain: green ≤ 60, yellow ≤ 90, red above (tunable; document that they're calibrated for intermittent retail demand).
+   - Replace the bias row: show `(totalForecast / totalActual − 1)` as a signed percentage labeled "Forecast vs actual" (both totals already arrive in `overall`). Keep MAE.
+   - Add a "vs naive" line: naive weekly WMAPE and FVA. FVA > 0 = engine adds value.
+   - The lead-time chart needs no code change — buckets will populate as F7 rows mature (7d lead evaluable after 7 days, 30d after 30, etc.).
+4. `confidenceLevel` in `/forecast/metrics` ([dashboard.js ~line 360]) is "share of products forecast via lifecycle curves", not confidence. It only feeds a per-day tooltip field — rename the JSON field to `curveCoverage` and update the one consumer in `ForecastMetrics.tsx`, or leave it and add a comment; low priority.
+
+### Phase 3 validation
+
+- Next run after deploy: `forecast_accuracy` contains `overall_weekly` and `fva` values; `/dashboard/forecast/accuracy` returns them; the overview popover renders weekly WMAPE, bias %, and the naive comparison.
+- After 7/14/30 days: `by_lead_time` rows appear for '8-14d', '15-30d', '31-60d' buckets respectively (61-90d after ~60 days).
+- Confirm engine runtime still < ~5 min and `product_forecasts_history` growth ≈ 50–70K rows/day.
+
+---
+
+## Phase 4 — Optional / after the above is proven
+
+- **F6. TSB for slow movers + dormant** (spec in F5). Gate on Phase 3 measurement: ship only if weekly FVA improves on those phases.
+- **F10. Confidence-margin source:** `load_accuracy_margins()` feeds daily-grain per-phase WMAPE (clamped to 1.0) into the intervals, so every interval is ±100% — uninformative. Once `overall_weekly` exists, add per-phase weekly rows (`by_phase_weekly`) and source margins from those instead.
+- **F11.** Update or delete `backfill_accuracy_data()` (it encodes the old formulas). Until then, just don't run `--backfill`.
+- **F12.** `compute_dow_indices()` weights by revenue but the multipliers are applied to units — switch `SUM(o.price * o.quantity)` to `SUM(o.quantity)`. Tiny effect.
+- **F13.** Longer term: for reorder decisions the right target is P(lead-time demand > stock), not a point forecast. Evaluate quantile (pinball) loss at lead-time horizons using the existing confidence-interval columns. Design separately.
+
+---
+
+## 4. Success criteria
+
+1. Rolling-14-day portfolio forecast/actual ratio within **0.8–1.25** (currently 1.5–2.5).
+2. Weekly-grain WMAPE ≤ **90%** and **FVA > 0** (engine beats naive) sustained for 2+ weeks.
+3. Decay/preorder/mature per-phase bias within ±0.1 units/day (currently +0.35 / +0.85 / +0.17).
+4. `all_incl_dormant` actuals covered: dormant bias better than −0.4 (currently −1.36, i.e. 100% miss).
+5. Lead-time buckets through 31–60d populated with ≥10K samples each within ~6 weeks.
+6. Launch phase stays healthy (bias within ±0.15, WMAPE not degraded) — regression guard for F3/F4 changes.
+
+## 5. Re-measurement appendix
+
+The naive-vs-engine comparison used in the diagnosis (rerun any time; adjust dates):
+
+```sql
+WITH ranked AS (
+  SELECT pfh.pid, pfh.forecast_date, pfh.forecast_units, pfh.lifecycle_phase,
+    ROW_NUMBER() OVER (PARTITION BY pfh.pid, pfh.forecast_date ORDER BY fr.started_at DESC) rn
+  FROM product_forecasts_history pfh
+  JOIN forecast_runs fr ON fr.id = pfh.run_id
+  WHERE pfh.forecast_date BETWEEN CURRENT_DATE - 9 AND CURRENT_DATE - 1),
+eng AS (SELECT * FROM ranked WHERE rn = 1 AND lifecycle_phase != 'dormant'),
+naive AS (
+  SELECT o.pid, SUM(o.quantity)/30.0 AS naive_daily FROM orders o
+  WHERE o.canceled IS DISTINCT FROM TRUE
+    AND o.date >= CURRENT_DATE - 39 AND o.date < CURRENT_DATE - 9
+  GROUP BY o.pid)
+SELECT e.lifecycle_phase, COUNT(*) AS n, SUM(COALESCE(dps.units_sold,0)) AS actual,
+  round(SUM(e.forecast_units),0) AS engine_fc, round(SUM(COALESCE(nv.naive_daily,0)),0) AS naive_fc,
+  round(SUM(ABS(e.forecast_units - COALESCE(dps.units_sold,0)))/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS engine_wmape,
+  round(SUM(ABS(COALESCE(nv.naive_daily,0) - COALESCE(dps.units_sold,0)))/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS naive_wmape
+FROM eng e
+LEFT JOIN naive nv ON nv.pid = e.pid
+LEFT JOIN daily_product_snapshots dps ON dps.pid = e.pid AND dps.snapshot_date = e.forecast_date
+GROUP BY ROLLUP(e.lifecycle_phase) ORDER BY 1;
+```
+
+Baseline numbers to beat (June 1–9, 2026): engine 221% / naive 204% daily WMAPE; engine_fc/actual = 1.82; per-phase table in §1.

IMPORT_METRICS_FIX_PLAN.md

@@ -0,0 +1,449 @@
+# Import & Metrics Pipeline Fix Plan
+
+Fixes for issues found in a full review (2026-06-10) of the `full-update.js` pipeline:
+`inventory-server/scripts/full-update.js` → `import-from-prod.js` (6 importers in `scripts/import/`)
+→ `calculate-metrics-new.js` (7 SQL modules in `scripts/metrics-new/`).
+
+Every issue below was verified against the code, and where marked **[verified-live]**, against the
+live MySQL source (`sg` on 192.168.1.5 via the acot-db tooling / `ssh workpi`) and live PostgreSQL
+(`inventory_db` — `ssh netcup`, then `psql -U inventory_readonly`, password in `/Users/matt/Dev/inventory/CLAUDE.md`).
+Write credentials for migrations: see `/var/www/inventory/.env` on netcup (`inventory_user`).
+
+## Operational context (read first)
+
+- Local `inventory-server/` is **NFS-mounted** to `/var/www/inventory/` on the netcup server — edits
+  appear on the server with no copy step. Run heavy validation/grep/find **on the server via
+  `ssh netcup`**, not locally (NFS hangs + AppleDouble `._*` noise).
+- The PG server timezone is **Europe/Berlin**. The business operates in **America/Chicago**. This
+  matters for Fix 2.
+- MySQL server is America/Chicago; the mysql2 driver is configured `timezone: '-05:00'` and
+  corrected at runtime by `adjustDateForMySQL()` in `scripts/import/utils.js` (see
+  `memory/TIMEZONE_ISSUE.md`). Don't "fix" that part — it already works.
+- Orders/PO/products imports are incremental by default (`INCREMENTAL_UPDATE !== 'false'`); a full
+  orders sync = run with `INCREMENTAL_UPDATE=false` (5-year window).
+- Existing rebuild tooling: `scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (rebuilds
+  `daily_product_snapshots` from `orders`/`receivings`). The full-pipeline order after data fixes:
+  re-import → rebuild snapshots → `node scripts/calculate-metrics-new.js`.
+- Precedent: `scripts/metrics-new/migrations/002_fix_discount_double_counting.sql` documents the
+  procedure used last time a discount formula changed. Follow the same pattern (migration doc +
+  code fix + full re-import + rebuild).
+
+---
+
+## P0 — Data correctness (do both, then ONE re-import + rebuild)
+
+### Fix 1: Item-level promo discounts dropped (~$26K / 30 days ≈ 10% of product revenue) [verified-live]
+
+**File:** `scripts/import/orders.js` — `order_totals` CTE (~lines 604-623) and the discount fetch in
+`processDiscountsBatch` (~lines 379-383).
+
+**Problem.** The discount applied to each PG `orders` row is:
+prorated `summary_discount_subtotal` + item-level promo discounts. The item-level part is gated:
+
+```sql
+SUM(CASE WHEN COALESCE(md.discount_amount_subtotal, 0) > 0 THEN id.amount ELSE 0 END)
+```
+
+In the PHP source (`/Users/matt/Dev/acot/website/website/lib/neworder.class.php`):
+- `order_items.prod_price` is the **pre-promo** price; `summary_subtotal = Σ prod_price·qty` (line ~3087).
+- Item-level promo discounts live in `order_discount_items` with `which = 2`; they are applied to the
+  order total via `summary_discount += amount + products_disc_sum` (line ~6567) — i.e. they are **not**
+  part of `discount_amount_subtotal` and **not** baked into `prod_price`.
+- Live data (90 days): of 10,010 type-10 promo discounts, **8,070 have item rows but only 8 have
+  `discount_amount_subtotal > 0`** — the gate zeroes essentially all item-level promo discounts.
+- Live impact (30 days): **$25,989 dropped** across 2,021 orders, vs only $13,574 captured via the
+  prorated subtotal component. Order discount components, 30d: total $54,957 = $13,574 subtotal +
+  $15,395 shipping + ~$25,989 item-level. (Shipping discounts correctly excluded from product revenue.)
+
+**Consequence.** `orders.discount` understated → `net_revenue`, `profit_30d`, `margin_30d` overstated
+by ~10% of revenue; `discounts_30d` / `discount_rate_30d` ~3x understated. Flows into daily snapshots,
+product/brand/vendor/category metrics, and dashboards.
+
+**Fix.**
+1. In `processDiscountsBatch`, fetch only real item discounts:
+   `SELECT order_id, pid, discount_id, amount FROM order_discount_items WHERE order_id IN (?) AND which = 2`.
+   (`which=1` rows store prices of free promo-added items; `which=3` are usage records — neither is a
+   discount amount.)
+2. In the `order_totals` CTE, remove the gate — sum `id.amount` unconditionally:
+   `SUM(COALESCE(id.amount, 0)) AS promo_discount_sum` (drop the join/CASE on `temp_main_discounts`;
+   `temp_main_discounts` becomes unused and can be removed entirely along with its insert loop).
+3. Sanity guard (optional, recommended): clamp final per-row discount to `price * quantity`.
+
+**Verification.** After a FULL orders re-import, for a recent 30-day window PG should satisfy:
+`SUM(discount)` ≈ MySQL `Σ summary_discount_subtotal` + `Σ order_discount_items.amount (which=2)`
+over the same orders (± rounding from proration). Spot-check an order with a type-10 promo:
+discount on the affected pid ≈ the `which=2` amount. Re-run migration 002's verification query too
+(pids 624756, 614513) to confirm no regression of the prior fix.
+
+### Fix 2: Daily snapshots bucket sales by Europe/Berlin days, not business days [verified-live]
+
+**Files:** `scripts/metrics-new/update_daily_snapshots.sql` (SalesData join `o.date::date = _target_date`
+~line 138; gap-fill and stale-detection aggregates at lines ~47-83);
+`scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (same pattern — check & fix);
+`scripts/metrics-new/update_product_metrics.sql` (`HistoricalDates` `MIN(o.date)::date` etc., lines ~131-147).
+
+**Problem.** `orders.date` is `timestamptz`; `::date` casts in the server TZ (**Europe/Berlin**,
+verified via `SHOW timezone`). Berlin is 7-8h ahead of Central, so every order placed after
+~5 PM Central lands on the **next** snapshot day. This shifts a large evening slice of daily sales
+forward one day; skews `yesterday_sales`, day-of-week patterns (the forecast engine's DOW
+multipliers, daily-grain forecast accuracy — see `FORECAST_FIX_PLAN.md`), and is inconsistent with
+`stock_snapshots`, whose dates come from a Central-time MySQL cron.
+
+**Fix.** Bucket all order/receiving dates in business time. Replace every `o.date::date` /
+`received_date::date` used for *day bucketing* in the two snapshot SQL files with:
+
+```sql
+(o.date AT TIME ZONE 'America/Chicago')::date
+```
+
+Apply consistently in: SalesData, ReceivingData, the gap-fill date lists, the stale-detection
+aggregates (they must match SalesData or every day looks permanently stale), and the rebuild script.
+`HistoricalDates` in update_product_metrics (first/last sold dates) should match too.
+Add an index to keep the per-day loop fast, e.g.
+`CREATE INDEX ON orders ( ((date AT TIME ZONE 'America/Chicago')::date) );` and equivalent on
+`receivings(received_date)`; check `EXPLAIN` on the SalesData query afterward.
+
+Note: `receivings.received_date` came from MySQL DATETIME (Central literal) inserted as timestamptz —
+it was interpreted in the *session* TZ at insert. Before converting, spot-check a few receivings
+against MySQL to confirm which TZ the stored instants actually represent; the conversion expression
+must yield the Central calendar day MySQL shows. Same check for `orders.date` (it originates from
+`_order.date_placed`, a TIMESTAMP column, so it should be a correct instant — `AT TIME ZONE
+'America/Chicago'` is right for it).
+
+**Verification.** Pick 2-3 recent days; compare per-day `units_sold` totals in
+`daily_product_snapshots` against MySQL
+`SELECT date_placed_onlydate, SUM(qty_ordered) ... WHERE order_status >= 20 GROUP BY 1`
+(MySQL stores Central days). They should now match closely (small diffs from canceled-status timing).
+
+### P0 execution order (single pass)
+
+1. Land Fix 1 (orders.js) and Fix 2 (both snapshot SQL files + product-metrics date CTE).
+2. Full orders re-import: `INCREMENTAL_UPDATE=false node scripts/import-from-prod.js` (or at minimum
+   the orders step) — run on the server, it's long.
+3. Rebuild snapshots: `psql -f scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (after
+   confirming it contains the TZ fix). The hourly job's 90-day self-heal will NOT fix history beyond
+   90 days by itself; the explicit rebuild is required.
+4. `node scripts/calculate-metrics-new.js`.
+5. Expect dashboards to show: margins down ~8-10 points (real), daily sales curves shifted, DOW
+   profile changed. Tell the user before/after numbers.
+
+---
+
+## P1 — Wrong or drifting numbers, fix soon
+
+### Fix 3: Vendor avg lead time computed over a near-cartesian join
+
+**File:** `scripts/metrics-new/calculate_vendor_metrics.sql`, `VendorPOAggregates` (lines ~62-83).
+
+**Problem.** Joins each done-PO line to **every** receiving of the same (pid, supplier) after the PO
+date — a product received 10 times contributes 10 ever-growing lead times → overstated, busy-product-
+weighted vendor lead time. The per-product version in `update_periodic_metrics.sql` (lines 27-48)
+is correct (MIN receiving per PO within 180 days, then average).
+
+**Fix.** Reuse the periodic shape, aggregated to vendor:
+
+```sql
+WITH po_first_receiving AS (
+    SELECT po.vendor, po.po_id, po.pid, po.date::date AS po_date,
+           MIN(r.received_date::date) AS first_receive_date
+    FROM purchase_orders po
+    JOIN receivings r ON r.pid = po.pid AND r.supplier_id = po.supplier_id
+        AND r.received_date >= po.date
+        AND r.received_date <= po.date + INTERVAL '180 days'
+    WHERE po.status = 'done' AND po.date >= CURRENT_DATE - INTERVAL '1 year'
+      AND po.vendor IS NOT NULL AND po.vendor <> ''
+    GROUP BY po.vendor, po.po_id, po.pid, po.date
+)
+SELECT vendor, COUNT(DISTINCT po_id) AS po_count_365d,
+       ROUND(AVG(GREATEST(1, first_receive_date - po_date)))::int AS avg_lead_time_days_hist
+FROM po_first_receiving GROUP BY vendor
+```
+
+**Verification.** For a few vendors compare old vs new values; new should be materially lower and
+roughly match `AVG(product_metrics.avg_lead_time_days)` for that vendor's products.
+
+### Fix 4: Deleted order items & combined orders never reconciled in PG [verified-live]
+
+**File:** `scripts/import/orders.js`.
+
+**Problem.** The orders import upserts but never deletes:
+- Items removed from an order in MySQL (`DELETE FROM order_items ...` happens, e.g.
+  neworder.class.php ~line 6500 for unpicked promo items, plus staff edits) leave stale rows in PG
+  forever. May 2026 check: PG has 49,841 item rows vs MySQL 49,377 (+0.9%) — and PG should be ≤
+  MySQL.
+- Combining orders (`combine_orders`, neworder.class.php ~11946) sets the source orders to status 16
+  AND **zeroes `date_placed`**, then copies all items to a NEW order. Because the import query
+  filters `o.date_placed >= …`, a combined source order can never be re-fetched, so its stale
+  'placed' rows would double-count with the new merged order. Currently latent (last combine
+  2024-07, predating current PG data — verified no stale rows exist today), but it will silently
+  corrupt the day combining is used again.
+
+**Fix.** Two parts, both inside the orders import after the upsert phase:
+1. **Item-set reconciliation** for re-imported orders: the import already knows the set of changed
+   `orderIds` and inserted their current items into `temp_order_items`. Mirror the PO import's
+   pattern (`purchase-orders.js` lines ~683-694):
+   ```sql
+   DELETE FROM orders o
+   WHERE o.order_number = ANY($1)           -- orders fetched this run
+     AND NOT EXISTS (SELECT 1 FROM temp_order_items t
+                     WHERE t.order_id = o.order_number AND t.pid = o.pid);
+   ```
+2. **Combined/cancelled sweep** that does NOT depend on `date_placed`: each run, fetch from MySQL
+   `SELECT order_id, order_status FROM _order WHERE order_status IN (15,16) AND stamp > ?`
+   (no date_placed filter) and update matching PG rows' `status`/`canceled`
+   ('combined' rows are then excluded from metrics — see Fix 5). Cheap (small result set).
+
+**Verification.** Re-run the May-2026 row-count comparison (MySQL vs PG for one month) after one full
+run; counts should converge (PG ≤ MySQL, diff explained by TZ window edges only).
+
+### Fix 5: 'combined' orders are counted as sales
+
+**Files:** `scripts/metrics-new/update_daily_snapshots.sql` (status filters, lines ~77, 120-134),
+`update_product_metrics.sql` (`HistoricalDates` line ~145, `LifetimeRevenue` line ~249),
+`backfill/rebuild_daily_snapshots.sql`.
+
+**Problem.** Sales filters exclude only `('canceled', 'returned')`. Status 16 'combined' = "merged
+into another order" — the new order carries the same items, so counting both double-counts. 826
+combined orders exist in MySQL; today none are in PG (see Fix 4), but once Fix 4's sweep starts
+marking rows 'combined', the metrics filters must exclude them.
+
+**Fix.** Change every `NOT IN ('canceled', 'returned')` in the metrics SQL to
+`NOT IN ('canceled', 'returned', 'combined')`. Grep for the pattern in `scripts/metrics-new/` and
+`src/routes/` (dashboard endpoints replicate these filters — see CLAUDE.md analytics-filters note).
+
+### Fix 6: Incremental sync watermark race (silent permanent misses)
+
+**Files:** `scripts/import/orders.js` (~772), `products.js` (~934), `purchase-orders.js` (~833).
+
+**Problem.** `sync_status.last_sync_timestamp` is set to `NOW()` *after* the import finishes. Any
+MySQL row modified between the source query and that write is below the new watermark but was never
+fetched → permanently skipped (until a full sync or the row changes again). Long imports widen the
+window; PG/MySQL clock skew adds to it.
+
+**Fix.** Capture the watermark **before** the source query and write that value:
+```js
+const [[{ now: sourceNow }]] = await prodConnection.query('SELECT NOW() as now');
+// ... do the import ...
+await localConnection.query(
+  `INSERT INTO sync_status ... VALUES ('orders', $1) ON CONFLICT ... SET last_sync_timestamp = $1`,
+  [sourceNow]);
+```
+Using MySQL's own clock also eliminates cross-server skew. Note `sourceNow` comes back through the
+mysql2 driver TZ conversion — verify round-tripping with `adjustDateForMySQL` produces a correct
+comparison value, or store `UTC_TIMESTAMP()` and compare against `CONVERT_TZ`-normalized stamps.
+Overlap (re-importing rows changed during the run) is harmless — everything is upserted.
+
+### Fix 7: Stockout days / service level / fill rate / avg stock built on activity-only snapshots
+
+**Files:** `scripts/metrics-new/update_product_metrics.sql` — `SnapshotAggregates`
+(`stockout_days_30d`, `avg_stock_*_30d`, lines ~177-189), `ServiceLevels` (lines ~304-323),
+plus `calculate_sales_velocity` usage.
+
+**Problem.** `daily_product_snapshots` only has rows on days with sales/receivings. So:
+- A product that is out of stock (and therefore sells nothing) gets **no row** → `stockout_days_30d`
+  ≈ 0 exactly when stockouts matter → `calculate_sales_velocity(sales, stockout_days)`'s adjustment
+  is inert → velocity and replenishment understated for constrained products.
+- `service_level_30d` divides stockout days by COUNT(activity days), not 30.
+- `avg_stock_units_30d` / `avg_stock_cost_30d` average only activity days (biased toward in-stock
+  days) → GMROI / stockturn / sell-through denominators biased.
+- `fill_rate_30d`'s `units_sold * 0.2` lost-sales heuristic is arbitrary — fine to keep, but document.
+
+**Fix.** Derive stock-presence metrics from `stock_snapshots` (full daily coverage from MySQL
+`snap_product_value`, imported by `stock-snapshots.js`) instead of `daily_product_snapshots`:
+```sql
+StockCoverage AS (
+  SELECT pid,
+         COUNT(*) FILTER (WHERE stock_quantity <= 0) AS stockout_days_30d,
+         AVG(stock_quantity)  AS avg_stock_units_30d,
+         AVG(stock_value)     AS avg_stock_cost_30d
+  FROM stock_snapshots
+  WHERE snapshot_date >= _current_date - INTERVAL '29 days'
+  GROUP BY pid
+)
+```
+Treat products absent from `stock_snapshots` for a day as unknown (NULL), not in-stock. Keep
+`daily_product_snapshots` for sales/revenue aggregates. `service_level_30d` denominator becomes the
+count of covered days. Note `stock_snapshots` has no `eod_stock_retail`; keep retail/gross averages
+on the old source or compute as `stock_quantity * current price` explicitly.
+
+**Verification.** Pick products that had a known stockout period; `stockout_days_30d` should now be
+> 0 and `sales_velocity_daily` should rise accordingly.
+
+---
+
+## P2 — Definition / robustness improvements
+
+### Fix 8: Returns don't reduce COGS; LifetimeRevenue ignores returns
+`update_daily_snapshots.sql` SalesData: COGS accrues only on `quantity > 0` rows; return rows
+(negative qty — 15,875 rows live) subtract revenue but never COGS → margin understated in
+return-heavy periods. Add a returns-COGS term mirroring the sales-COGS COALESCE chain
+(`SUM(... WHEN quantity < 0 THEN cost * ABS(quantity))`) and subtract it in `cogs` (or store
+`returns_cogs` separately and use `cogs - returns_cogs` in profit). Also `LifetimeRevenue` in
+`update_product_metrics.sql` (line ~242) filters `quantity > 0` — include negative-qty rows so
+lifetime revenue nets out returns (drop the quantity filter; `price*quantity` is already signed,
+but check the `- discount` term sign for return rows).
+
+### Fix 9: return_rate_30d definition
+`update_product_metrics.sql` line ~468: `returns / (sales + returns)` → industry standard is
+`returns / sales`. Change denominator to `NULLIF(sa.sales_30d, 0)`.
+
+### Fix 10: GMROI not annualized
+Line ~466: `profit_30d / avg_stock_cost_30d` is a monthly GMROI (~1/12 of the conventional annual
+figure, benchmark ≥ 2-3). Either annualize (`* 12.17`) or rename the column/label "monthly".
+Decision for Matt; annualizing is recommended for comparability. Frontend displays must be checked
+either way.
+
+### Fix 11: get_weighted_avg_cost is a lifetime WAC
+`db/functions.sql` (~line 81, deployed identically): averages ALL receivings ≤ date — decade-old
+costs weigh equally. Recommended: window to recent receivings, e.g. last 365 days falling back to
+lifetime when none. Used as fallback COGS when `o.costeach` is NULL, so impact is modest but real
+for long-lived SKUs. Apply with `CREATE OR REPLACE FUNCTION` in `db/functions.sql` AND on the live DB.
+
+### Fix 12: exclude_from_forecast removes products from product_metrics entirely
+`update_product_metrics.sql` line ~627 (`WHERE s.exclude_forecast IS FALSE OR ... IS NULL`): the
+flag's name implies forecast-only, but excluded products get NO metrics row → vanish from brand/
+vendor/category rollups and dashboards. Fix: always emit the row; instead NULL the
+forecast/replenishment columns when excluded (wrap those expressions in
+`CASE WHEN s.exclude_forecast THEN NULL ELSE ... END`).
+
+### Fix 13: Incremental products import misses category-only changes
+`products.js` incremental WHERE (~lines 433-440) keys on `p.stamp`, `ci.stamp`, price/b2b dates —
+`product_category_index` changes don't bump any of those → PG `product_categories` goes stale. Also
+the `needs_update` comparison (~lines 604-625) doesn't compare `categories`, so even refetched rows
+skip the category rewrite. Fix both: add `t.categories IS NOT DISTINCT FROM p.categories` to the
+needs_update comparison (note: `products.categories` is the GROUP_CONCAT string — confirm PG column
+holds the same representation), and add a cheap full-sweep (e.g. weekly, or compare
+`COUNT(*) GROUP BY pid` hashes) OR include `EXISTS (SELECT 1 FROM product_category_index pci WHERE
+pci.pid = p.pid AND pci.stamp > ?)` in the incremental WHERE if that table has a stamp column —
+verify schema first (`DESCRIBE product_category_index`).
+
+### Fix 14: PO/receivings OFFSET pagination over a moving filter
+`purchase-orders.js` (~lines 275-298, 447-470): `LIMIT/OFFSET` with a `date_updated > ?` predicate;
+concurrent updates shift rows between pages → silent skips. Fix: keyset pagination —
+`WHERE ... AND p.po_id > ? ORDER BY p.po_id LIMIT 500`, carrying the last seen po_id (drop OFFSET).
+Same for receivings on `receiving_id`.
+
+### Fix 15: Status map gaps and unsafe defaults
+- `orders.js` orderStatusMap lacks 45 (`payment_pending`) and 67 (`remote_send`) → imported as
+  numeric strings. Add both (mirror in `migrations/001_map_order_statuses.sql` as a follow-up update
+  for existing rows).
+- `purchase-orders.js` `poStatusMap[po.status] || 'created'` (line ~335): an unknown *cancel-like*
+  code would be treated as an open PO and inflate on-order FIFO. Default to a sentinel like
+  `'unknown_<code>'` instead, and make the FIFO/on-order CTEs in `update_product_metrics.sql` treat
+  only the known-open statuses as open (they already whitelist open statuses — so the sentinel is
+  safe there; just ensure nothing treats unknown as 'created'). Same for receivingStatusMap.
+
+### Fix 16: Transactions issued through the pool wrapper land on arbitrary connections
+`categories.js` (lines ~17-152) and `daily-deals.js` (~27-130) call `query('BEGIN')` /
+`query('COMMIT')` on the wrapper, which checks out a client per call — BEGIN/work/COMMIT are not
+guaranteed to share a connection (works only by pool-LIFO accident). The categories
+`DISABLE TRIGGER` rides on this too. Fix: use the wrapper's `beginTransaction()/commit()/rollback()`
+(see `utils.js` lines 121-148) exactly as orders.js does. In categories.js also move the
+post-COMMIT `ENABLE TRIGGER` inside the transaction (DISABLE/ENABLE both inside), or drop the
+trigger toggling entirely if the trigger isn't actually problematic anymore.
+
+### Fix 17: stock-snapshots import swallows batch errors → permanent holes
+`stock-snapshots.js` (~lines 153-155): a failed batch is logged and skipped, but the next
+incremental starts at `MAX(snapshot_date)` — the hole is never revisited. Fix: rethrow (fail the
+step) or collect failed date ranges and retry once, then fail if still failing. Also line ~168:
+`calculateRate(processedRows, startTime)` — arguments reversed (signature is
+`calculateRate(startTime, current)`, see `metrics-new/utils/progress.js:70`).
+
+### Fix 18: Metrics cancellation targets an application_name that's never set
+`calculate-metrics-new.js` line ~180 cancels backends `WHERE application_name =
+'node-metrics-calculator'`, but the Pool config never sets it → cancellation no-ops (the 30-min
+`statement_timeout` is the only real guard). Fix: add `application_name: 'node-metrics-calculator'`
+to both dbConfig branches.
+
+### Fix 19: Aggregate-table change-detection lists miss cost-only changes
+`calculate_brand_metrics.sql` / `calculate_vendor_metrics.sql` / `calculate_category_metrics.sql`
+ON CONFLICT WHERE lists don't include `profit_30d`/`cogs_30d` — a cost revision with unchanged
+sales/revenue leaves stale rows (product_metrics has a 1-day staleness net; rollups don't). Add
+`... OR x.profit_30d IS DISTINCT FROM EXCLUDED.profit_30d OR x.cogs_30d IS DISTINCT FROM
+EXCLUDED.cogs_30d` to each, or add a `last_calculated < NOW() - INTERVAL '1 day'` net like
+product_metrics line ~707.
+
+### Fix 20: Snapshot stale-detection only compares unit counts
+`update_daily_snapshots.sql` lines ~57-85: detects mismatches in `units_sold`/`units_received` only;
+price/discount/costeach corrections older than the 2-day recheck are never repaired. Add a
+revenue comparison to the stale check: compare `SUM(net_revenue)` per day against the equivalent
+recomputed from `orders` (ROUND both to 2dp to avoid float-noise churn).
+
+### Fix 21: Category metrics positive-only revenue asymmetry
+`calculate_category_metrics.sql` (lines ~27-36, 64-73): revenue summed only when `> 0` while
+cogs/profit use COALESCE-all → margin numerator/denominator from different populations, and
+inconsistent with brand/vendor (plain COALESCE). Change the revenue/sales CASEs to
+`COALESCE(pm.revenue_7d, 0)` etc., matching brand_metrics.
+
+### Fix 22 (decision needed): Demand-pattern & seasonality definitions
+- `classify_demand_pattern` (db/functions.sql): CV thresholds 0.2/0.5 + avg<1/day. Industry standard
+  is Syntetos-Boylan: ADI ≥ 1.32 and CV² ≥ 0.49 quadrants (smooth/erratic/intermittent/lumpy).
+  Today everything classifies sporadic/lumpy. If adopting SB: ADI = 30 / COUNT(days with sales),
+  CV² computed on nonzero-demand sizes. Changes the vocabulary consumed by the forecast engine
+  (`scripts/forecast/forecast_engine.py` reads `demand_pattern`) — coordinate before changing.
+- SeasonalityAnalysis (`update_product_metrics.sql` ~360): `month_avg = AVG(units_sold)` over rows
+  with sales only → intensity, not volume. Use monthly totals (SUM, with zero months counted) /
+  overall monthly average for the index.
+- Safety stock: currently static config units; `sales_std_dev_30d` exists but is unused. Optional
+  upgrade: `safety = z * σ_d * sqrt(lead_time)` with z from a service-level setting.
+
+These change user-facing semantics — confirm with Matt before implementing.
+
+---
+
+## Verified non-issues (no action, or cleanup only)
+
+- **`costeach` fallback `price * 0.5`** (orders.js line ~615): fires on **2.1%** of item rows
+  (729/34,833, last 30d, live-verified). Accepted by Matt — 50% margin is a fair estimate for these
+  products. Optional: nothing.
+- **Missing-product order skips**: zero occurrences — MySQL has no orphan order_items (1-year check),
+  PG products is a superset of MySQL products (687,579 vs 687,576), last 7 import runs all logged
+  `totalSkipped: 0`. Cleanup only: remove the unused `importMissingProducts` import line at
+  `orders.js:2` (the function itself stays in products.js — harmless utility).
+- **Status 30 'cancelled_old'** in `total_sold >= 20` filter: zero rows live in `_order` — safe.
+- **Duplicate (order_id, pid) order items**: none exist in MySQL — the upsert PK is safe.
+- **base_discount** in orders.js: computed/stored in temp table but unused since migration 002 —
+  remove the column from temp table + queries for clarity (no behavior change).
+- **`full-update.js` `runScript`**: try/catch around `console.log` is dead code; per-step
+  `status:'complete'` messages could confuse a UI parser. Cosmetic only — tidy if touching the file.
+
+## Suggested implementation order
+
+| Step | Fixes | Re-import/rebuild needed |
+|---|---|---|
+| 1 | Fix 1 + Fix 2 (+ Fix 5 filters, Fix 8/9 while editing the same SQL) | FULL orders re-import → snapshot rebuild → metrics (once) |
+| 2 | Fix 4 + Fix 6 (orders.js reconciliation + watermarks; POs/products watermarks too) | no |
+| 3 | Fix 3, Fix 7 (metrics SQL only) | metrics run |
+| 4 | Fix 13-21 (robustness batch) | no |
+| 5 | Fix 10-12, Fix 22 after Matt's sign-off (definition changes) | metrics run |
+
+After step 1, expect: margin_30d down ~8-10 points, discounts_30d ~3x up, daily curves shifted to
+correct business days. Communicate before/after so the change isn't mistaken for a data incident.
+
+## Reference: verification snippets used in the review
+
+```sql
+-- MySQL: item-level discounts dropped by the gate (30d)
+SELECT COUNT(DISTINCT o.order_id), ROUND(SUM(odi.amount),2)
+FROM order_discount_items odi
+JOIN order_discounts od ON od.order_id=odi.order_id AND od.discount_id=odi.discount_id
+JOIN _order o ON o.order_id=odi.order_id
+WHERE odi.which=2 AND o.date_placed >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+  AND o.order_status >= 20 AND COALESCE(od.discount_amount_subtotal,0)=0;
+-- → 2,021 orders / $25,989 (2026-06-10)
+
+-- MySQL: costeach fallback frequency (30d)
+SELECT COUNT(*),
+  SUM(CASE WHEN NOT EXISTS (SELECT 1 FROM order_costs oc WHERE oc.orderid=oi.order_id
+            AND oc.pid=oi.prod_pid AND oc.pending=0)
+        AND NOT EXISTS (SELECT 1 FROM product_inventory pi WHERE pi.pid=oi.prod_pid)
+      THEN 1 ELSE 0 END)
+FROM order_items oi JOIN _order o ON o.order_id=oi.order_id
+WHERE o.order_status >= 20 AND o.date_placed >= DATE_SUB(CURDATE(), INTERVAL 30 DAY);
+-- → 729 / 34,833 = 2.1% (2026-06-10)
+
+-- PG: timezone check
+SHOW timezone;  -- Europe/Berlin (2026-06-10)
+
+-- Row drift, May 2026: MySQL 49,377 items / PG 49,841 (+0.9%)
+```

PRODUCT_IMPORT_ENHANCEMENTS.md

@@ -0,0 +1,375 @@
+# Product Import Module - Enhancement & Issues Outline
+
+This document outlines the investigation and implementation requirements for each requested enhancement to the product import module.
+
+---
+
+## 1. UPC Import - Strip Quotes and Spaces ✅ IMPLEMENTED
+
+**Issue:** When importing UPCs, strip `'`, `"` characters and any spaces, leaving only numbers.
+
+**Implementation (Completed):**
+- Modified `normalizeUpcValue()` in [Import.tsx:661-667](inventory/src/pages/Import.tsx#L661-L667)
+- Strips single quotes, double quotes, smart quotes (`'"`), and whitespace before processing
+- Then handles scientific notation and extracts only digits
+
+**Files Modified:**
+- `inventory/src/pages/Import.tsx` - `normalizeUpcValue()` function
+
+---
+
+## 2. AI Context Columns in Validation Payloads ✅ IMPLEMENTED
+
+**Issue:** The match columns step has a setting to use a field only for AI context (`isAiSupplemental`). Update AI description validation to include any columns selected with this option in the payload. Also include in sanity check payload. Not needed for names.
+
+**Current Implementation:**
+- AI Supplemental toggle: [MatchColumnsStep.tsx:102-118](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L102-L118)
+- AI supplemental data stored in `__aiSupplemental` field on each row
+- Description payload builder: [inlineAiPayload.ts:183-195](inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts#L183-L195)
+
+**Implementation:**
+1. **Update `buildDescriptionValidationPayload()` in `inlineAiPayload.ts`** to include AI supplemental data:
+   ```typescript
+   export const buildDescriptionValidationPayload = (
+     row: Data<string>,
+     fieldOptions: FieldOptionsMap,
+     productLinesCache: Map<string, SelectOption[]>,
+     sublinesCache: Map<string, SelectOption[]>
+   ) => {
+     const payload: Record<string, unknown> = {
+       name: row.name,
+       description: row.description,
+       company_name: getFieldOptionLabel(row.company, fieldOptions, 'company'),
+       company_id: row.company,
+       categories: getFieldOptionLabel(row.category, fieldOptions, 'category'),
+     };
+
+     // Add AI supplemental context if present
+     if (row.__aiSupplemental && typeof row.__aiSupplemental === 'object') {
+       payload.additional_context = row.__aiSupplemental;
+     }
+
+     return payload;
+   };
+   ```
+
+2. **Update sanity check payload** - Locate sanity check submission logic and include `__aiSupplemental` data
+
+3. **Verify `__aiSupplemental` is properly populated** from MatchColumnsStep when columns are marked as AI context only
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts`
+- Backend sanity check endpoint (if separate from description validation)
+- Verify data flow in `MatchColumnsStep.tsx` → `ValidationStep`
+
+---
+
+## 3. Fresh Taxonomy Data Per Session ✅ IMPLEMENTED
+
+**Issue:** Ensure taxonomy data is brought in fresh with each session - cache should be invalidated if we exit the import flow and start again.
+
+**Current Implementation:**
+- Field options cached 5 minutes: [ValidationStep/index.tsx:128-133](inventory/src/components/product-import/steps/ValidationStep/index.tsx#L128-L133)
+- Product lines cache: `productLinesCache` in Zustand store
+- Sublines cache: `sublinesCache` in Zustand store
+- Caches set to 10-minute stale time
+
+**Implementation:**
+1. **Add cache invalidation on import flow mount/unmount** in `UploadFlow.tsx`:
+   ```typescript
+   useEffect(() => {
+     // On mount - invalidate import-related query cache
+     queryClient.invalidateQueries({ queryKey: ['import-field-options'] });
+
+     return () => {
+       // On unmount - clear caches
+       queryClient.removeQueries({ queryKey: ['import-field-options'] });
+       queryClient.removeQueries({ queryKey: ['product-lines'] });
+       queryClient.removeQueries({ queryKey: ['sublines'] });
+     };
+   }, []);
+   ```
+
+2. **Clear Zustand store caches** when exiting import flow:
+   - Add action to `validationStore.ts` to clear `productLinesCache` and `sublinesCache`
+   - Call this action on unmount of `UploadFlow` or when navigating away
+
+3. **Consider adding a `sessionId`** that changes on each import flow start, used as part of cache keys
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/UploadFlow.tsx` - Add cleanup effect
+- `inventory/src/components/product-import/steps/ValidationStep/store/validationStore.ts` - Add cache clear action
+- Potentially `inventory/src/components/product-import/steps/ValidationStep/index.tsx` - Query key updates
+
+---
+
+## 4. Save Template from Confirmation Page ✅ IMPLEMENTED
+
+**Issue:** Add option to save rows of submitted data as a new template on the confirmation page after completing the import flow. Verify this works with new validation step changes.
+
+**Current Implementation:**
+- **Import Results section already exists** inline in [Import.tsx:968-1150](inventory/src/pages/Import.tsx#L968-L1150)
+  - Shows created products (lines 1021-1097) with image, name, UPC, item number
+  - Shows errored products (lines 1100-1138) with error details
+  - "Fix products with errors" button resumes validation flow for failed items
+- Template saving logic in ValidationStep: [useTemplateManagement.ts:204-266](inventory/src/components/product-import/steps/ValidationStep/hooks/useTemplateManagement.ts#L204-L266)
+- Saves via `POST /api/templates`
+- `importOutcome.submittedProducts` contains the full product data for each row
+
+**Implementation:**
+1. **Add "Save as Template" button** to each created product row in the results section (around line 1087-1092 in Import.tsx):
+   ```typescript
+   // Add button after the item number display
+   <Button
+     variant="ghost"
+     size="sm"
+     onClick={() => handleSaveAsTemplate(index)}
+   >
+     <BookmarkPlus className="h-4 w-4" />
+   </Button>
+   ```
+
+2. **Add state and dialog** for template saving in Import.tsx:
+   ```typescript
+   const [templateSaveDialogOpen, setTemplateSaveDialogOpen] = useState(false);
+   const [selectedProductForTemplate, setSelectedProductForTemplate] = useState<NormalizedProduct | null>(null);
+   ```
+
+3. **Extract/reuse template save logic** from `useTemplateManagement.ts`:
+   - The `saveNewTemplate()` function (lines 204-266) can be extracted into a shared utility
+   - Or create a `SaveTemplateDialog` component that can be used in both places
+   - Key fields needed: `company` (for template name), `product_type`, and all product field values
+
+4. **Data mapping consideration:**
+   - `importOutcome.submittedProducts` uses `NormalizedProduct` type
+   - Templates expect raw field values - may need to map back from normalized format
+   - Exclude metadata fields: `['id', '__index', '__meta', '__template', '__original', '__corrected', '__changes', '__aiSupplemental']`
+
+**Files to Modify:**
+- `inventory/src/pages/Import.tsx` - Add save template button, state, and dialog
+- Consider creating `inventory/src/components/product-import/SaveTemplateDialog.tsx` for reusability
+- Potentially extract core save logic from `useTemplateManagement.ts` into shared utility
+
+---
+
+## 5. Sheet Preview on Select Sheet Step ✅ IMPLEMENTED
+
+**Issue:** On the select sheet step, show a preview of the first 10 lines or so of each sheet underneath the options.
+
+**Implementation (Completed):**
+- Added `workbook` prop to `SelectSheetStep` component
+- Added `sheetPreviews` memoized computation using `XLSXLib.utils.sheet_to_json()`
+- Shows first 10 rows, 8 columns max per sheet
+- Added `truncateCell()` helper to limit cell content to 30 characters with ellipsis
+- Each sheet option is now a clickable card with:
+  - Radio button and sheet name
+  - Row count indicator
+  - Scrollable preview table with horizontal scroll
+  - Selected state highlighted with primary border
+- Updated `UploadFlow.tsx` to pass workbook prop
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectSheetStep/SelectSheetStep.tsx`
+- `inventory/src/components/product-import/steps/UploadFlow.tsx`
+
+---
+
+## 6. Empty Row Removal ✅ IMPLEMENTED
+
+**Issue:** When importing a sheet, automatically remove completely empty rows.
+
+**Current Implementation:**
+- Empty columns are filtered: [MatchColumnsStep.tsx:616-634](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L616-L634)
+- A "Remove empty/duplicates" button exists that removes empty rows, single-value rows, AND duplicates
+- The automatic removal should ONLY remove completely empty rows, not duplicates or single-value rows
+
+**Implementation (Completed):**
+- Added `isRowCompletelyEmpty()` helper function to [SelectHeaderStep.tsx](inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx)
+- Added `useMemo` to filter empty rows on initial data load
+- Uses `Object.values(row)` to check all cell values (matches existing button logic)
+- Only removes rows where ALL values are undefined, null, or whitespace-only strings
+- Manual "Remove Empty/Duplicates" button still available for additional cleanup (duplicates, single-value rows)
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx`
+
+---
+
+## 7. Unit Conversion for Weight/Dimensions ✅ IMPLEMENTED
+
+**Issue:** Add unit conversion feature for weight and dimensions columns - similar to calculator button on cost/msrp, add button that opens popover with options to convert grams → oz, lbs → oz for the whole column at once.
+
+**Current Implementation:**
+- Calculator button on price columns: [ValidationTable.tsx:1491-1627](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1491-L1627)
+- `PriceColumnHeader` component shows calculator icon on hover
+- Weight field defined in config with validation
+
+**Implementation:**
+1. **Create `UnitConversionColumnHeader` component** (similar to `PriceColumnHeader`):
+   ```typescript
+   const UnitConversionColumnHeader = ({ field, table }) => {
+     const [showPopover, setShowPopover] = useState(false);
+
+     const conversions = {
+       weight: [
+         { label: 'Grams → Ounces', factor: 0.035274 },
+         { label: 'Pounds → Ounces', factor: 16 },
+         { label: 'Kilograms → Ounces', factor: 35.274 },
+       ],
+       dimensions: [
+         { label: 'Centimeters → Inches', factor: 0.393701 },
+         { label: 'Millimeters → Inches', factor: 0.0393701 },
+       ]
+     };
+
+     const applyConversion = (factor: number) => {
+       // Batch update all cells in column
+       table.rows.forEach((row, index) => {
+         const currentValue = parseFloat(row[field.key]);
+         if (!isNaN(currentValue)) {
+           updateCell(index, field.key, (currentValue * factor).toFixed(2));
+         }
+       });
+     };
+
+     return (
+       <Popover open={showPopover} onOpenChange={setShowPopover}>
+         <PopoverTrigger>
+           <Scale className="h-4 w-4" /> {/* or similar icon */}
+         </PopoverTrigger>
+         <PopoverContent>
+           {conversions[fieldType].map(conv => (
+             <Button key={conv.label} onClick={() => applyConversion(conv.factor)}>
+               {conv.label}
+             </Button>
+           ))}
+         </PopoverContent>
+       </Popover>
+     );
+   };
+   ```
+
+2. **Identify weight/dimension fields** in config:
+   - `weight_oz`, `length_in`, `width_in`, `height_in` (check actual field keys)
+
+3. **Add to column header render logic** in ValidationTable
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx`
+- Potentially create new component file for `UnitConversionColumnHeader`
+- Update column header rendering to use new component for weight/dimension fields
+
+---
+
+## 8. Expanded MSRP Auto-Fill from Cost ✅ IMPLEMENTED
+
+**Issue:** Expand auto-fill functionality for MSRP from cost - open small popover with options for 2x, 2.1x, 2.2x, 2.3x, 2.4x, 2.5x multipliers, plus checkbox to round up to nearest 9.
+
+**Current Implementation:**
+- Calculator on MSRP column: [ValidationTable.tsx:1540-1584](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1540-L1584)
+- Currently only does `Cost × 2` then subtracts 0.01 if whole number
+
+**Implementation:**
+1. **Replace simple click with popover** in `PriceColumnHeader`:
+   ```typescript
+   const [selectedMultiplier, setSelectedMultiplier] = useState(2.0);
+   const [roundToNine, setRoundToNine] = useState(false);
+   const multipliers = [2.0, 2.1, 2.2, 2.3, 2.4, 2.5];
+
+   const roundUpToNine = (value: number): number => {
+     // 1.41 → 1.49, 2.78 → 2.79, 12.32 → 12.39
+     const wholePart = Math.floor(value);
+     const decimal = value - wholePart;
+     if (decimal <= 0.09) return wholePart + 0.09;
+     if (decimal <= 0.19) return wholePart + 0.19;
+     // ... continue pattern, or:
+     const lastDigit = Math.floor(decimal * 10);
+     return wholePart + (lastDigit / 10) + 0.09;
+   };
+
+   const calculateMsrp = (cost: number): number => {
+     let result = cost * selectedMultiplier;
+     if (roundToNine) {
+       result = roundUpToNine(result);
+     }
+     return result;
+   };
+   ```
+
+2. **Create popover UI**:
+   ```tsx
+   <Popover>
+     <PopoverTrigger><Calculator className="h-4 w-4" /></PopoverTrigger>
+     <PopoverContent className="w-48">
+       <div className="space-y-2">
+         <Label>Multiplier</Label>
+         <div className="grid grid-cols-3 gap-1">
+           {multipliers.map(m => (
+             <Button
+               key={m}
+               variant={selectedMultiplier === m ? 'default' : 'outline'}
+               size="sm"
+               onClick={() => setSelectedMultiplier(m)}
+             >
+               {m}x
+             </Button>
+           ))}
+         </div>
+         <div className="flex items-center gap-2">
+           <Checkbox checked={roundToNine} onCheckedChange={setRoundToNine} />
+           <Label>Round to .X9</Label>
+         </div>
+         <Button onClick={applyCalculation} className="w-full">
+           Apply
+         </Button>
+       </div>
+     </PopoverContent>
+   </Popover>
+   ```
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx` - `PriceColumnHeader` component
+
+---
+
+## 9. Debug Mode - Skip API Submission ✅ IMPLEMENTED
+
+**Issue:** Add a third switch in the footer of image upload step (visible only to users with `admin:debug` permission) that will not submit data to any API, only complete the process and show results page as if it had worked.
+
+**Implementation (Completed):**
+- Added `skipApiSubmission` state to `ImageUploadStep.tsx`
+- Added amber-colored "Skip API (Debug)" switch (visible only with `admin:debug` permission)
+- When skip is active, "Use Test API" and "Use Test Database" switches are hidden
+- Added `skipApiSubmission?: boolean` to `SubmitOptions` type in `types.ts`
+- In `Import.tsx`, when `skipApiSubmission` is true:
+  - Skips the actual API call entirely
+  - Generates mock success response with mock PIDs
+  - Shows `[DEBUG]` prefix in toast and result message
+  - Displays results page as if submission succeeded
+
+**Files Modified:**
+- `inventory/src/components/product-import/types.ts` - Added `skipApiSubmission` to `SubmitOptions`
+- `inventory/src/components/product-import/steps/ImageUploadStep/ImageUploadStep.tsx` - Added switch UI
+- `inventory/src/pages/Import.tsx` - Added skip logic in `handleData()`
+
+---
+
+## Summary
+
+| # | Enhancement | Complexity | Status |
+|---|-------------|------------|--------|
+| 1 | Strip UPC quotes/spaces | Low | ✅ Implemented |
+| 2 | AI context in validation | Medium | ✅ Implemented |
+| 3 | Fresh taxonomy per session | Medium | ✅ Implemented |
+| 4 | Save template from confirmation | Medium-High | ✅ Implemented |
+| 5 | Sheet preview | Low-Medium | ✅ Implemented |
+| 6 | Remove empty rows | Low | ✅ Implemented |
+| 7 | Unit conversion | Medium | ✅ Implemented |
+| 8 | MSRP multiplier options | Medium | ✅ Implemented |
+| 9 | Debug skip API | Low-Medium | ✅ Implemented |
+
+**Implemented:** 9 of 9 items - All enhancements complete!
+
+---
+
+*Document generated: 2026-01-25*

docs/METRICS_AUDIT.md

@@ -0,0 +1,346 @@
+# Metrics Calculation Pipeline Audit
+
+**Date:** 2026-02-07
+**Scope:** All 6 SQL calculation scripts, custom DB functions, import pipeline, and live data verification
+
+## Overview
+
+The metrics pipeline in `inventory-server/scripts/calculate-metrics-new.js` runs 6 SQL scripts sequentially:
+
+1. `update_daily_snapshots.sql` — Aggregates daily per-product sales/receiving data
+2. `update_product_metrics.sql` — Calculates the main product_metrics table (KPIs, forecasting, status)
+3. `update_periodic_metrics.sql` — ABC classification, average lead time
+4. `calculate_brand_metrics.sql` — Brand-level aggregated metrics
+5. `calculate_vendor_metrics.sql` — Vendor-level aggregated metrics
+6. `calculate_category_metrics.sql` — Category-level metrics with hierarchy rollups
+
+### Database Scale
+| Table | Row Count |
+|---|---|
+| products | 681,912 |
+| orders | 2,883,982 |
+| purchase_orders | 256,809 |
+| receivings | 313,036 |
+| daily_product_snapshots | 678,312 (601 distinct dates, since 2024-06-01) |
+| product_metrics | 681,912 |
+| brand_metrics | 1,789 |
+| vendor_metrics | 281 |
+| category_metrics | 610 |
+
+---
+
+## Issues Found
+
+### ISSUE 1: [HIGH] Order status filter is non-functional — numeric codes vs text comparison
+
+**Files:** `update_daily_snapshots.sql` lines 86-101, `update_product_metrics.sql` lines 89, 178-183
+**Confirmed by data:** All order statuses are numeric strings ('100', '50', '55', etc.)
+**Status mappings from:** `docs/prod_registry.class.php`
+
+**Description:** The SQL filters `COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned')` and `o.status NOT IN ('canceled', 'returned')` are used throughout the pipeline to exclude canceled/returned orders. However, the import pipeline stores order statuses as their **raw numeric codes** from the production MySQL database (e.g., '100', '50', '55', '90', '92'). There are **zero text status values** in the orders table.
+
+This means these filters **never exclude any rows** — every comparison is `'100' NOT IN ('canceled', 'returned')` which is always true.
+
+**Actual status distribution (with confirmed meanings):**
+| Status | Meaning | Count | Negative Qty | Assessment |
+|---|---|---|---|---|
+| 100 | shipped | 2,862,792 | 3,352 | Completed — correct to include |
+| 50 | awaiting_products | 11,109 | 0 | In-progress — not yet shipped |
+| 55 | shipping_later | 5,689 | 0 | In-progress — not yet shipped |
+| 56 | shipping_together | 2,863 | 0 | In-progress — not yet shipped |
+| 90 | awaiting_shipment | 38 | 0 | Near-complete — not yet shipped |
+| 92 | awaiting_pickup | 71 | 0 | Near-complete — awaiting customer |
+| 95 | shipped_confirmed | 5 | 0 | Completed — correct to include |
+| 15 | cancelled | 1 | 0 | Should be excluded |
+
+**Full status reference (from prod_registry.class.php):**
+- 0=created, 10=unfinished, **15=cancelled**, 16=combined, 20=placed, 22=placed_incomplete
+- 30=cancelled_old (historical), 40=awaiting_payment, 50=awaiting_products
+- 55=shipping_later, 56=shipping_together, 60=ready, 61=flagged
+- 62=fix_before_pick, 65=manual_picking, 70=in_pt, 80=picked
+- 90=awaiting_shipment, 91=remote_wait, **92=awaiting_pickup**, 93=fix_before_ship
+- **95=shipped_confirmed**, **100=shipped**
+
+**Severity revised to HIGH (from CRITICAL):** Now that we know the actual meanings, no cancelled/refunded orders are being miscounted (only 1 cancelled order exists, status=15). The real concern is twofold:
+1. **The text-based filter is dead code** — it can never match any row. Either map statuses to text during import (like POs do) or change SQL to use numeric comparisons.
+2. **~19,775 unfulfilled orders** (statuses 50/55/56/90/92) are counted as completed sales. These are orders in various stages of fulfillment that haven't shipped yet. While most will eventually ship, counting them now inflates current-period metrics. At 0.69% of total orders, the financial impact is modest but the filter should work correctly on principle.
+
+**Note:** PO statuses ARE properly mapped to text ('canceled', 'done', etc.) in the import pipeline. Only order statuses are numeric.
+
+---
+
+### ISSUE 2: [CRITICAL] Daily Snapshots use current stock instead of historical EOD stock
+
+**File:** `update_daily_snapshots.sql`, lines 126-135, 173
+**Confirmed by data:** Top product (pid 666925) shows `eod_stock_quantity = 0` for ALL dates even though it sold 28 units on Jan 28 (clearly had stock then)
+
+**Description:** The `CurrentStock` CTE reads `stock_quantity` directly from the `products` table at query execution time. When the script processes historical dates (today minus 1-4 days), it writes **today's stock** as if it were the end-of-day stock for those past dates.
+
+**Cascading impact on product_metrics:**
+- `avg_stock_units_30d` / `avg_stock_cost_30d` — Wrong averages
+- `stockout_days_30d` — Undercounts (only based on current stock state, not historical)
+- `stockout_rate_30d`, `service_level_30d`, `fill_rate_30d` — All derived from wrong stockout data
+- `gmroi_30d` — Wrong denominator (avg stock cost)
+- `stockturn_30d` — Wrong denominator (avg stock units)
+- `sell_through_30d` — Affected by stock level inaccuracy
+
+---
+
+### ISSUE 3: [CRITICAL] Snapshot coverage is 0.17% — most products have no snapshot data
+
+**Confirmed by data:** 678,312 snapshot rows across 601 dates = ~1,128 products/day out of 681,912 total
+
+**Description:** The daily snapshots script only creates rows for products with sales or receiving activity on that date (`ProductsWithActivity` CTE, line 136). This means:
+- 91.1% of products (621,221) have NULL `sales_30d` — they had no orders in the last 30 days so no snapshot rows exist
+- `AVG(eod_stock_quantity)` averages only across days with activity, not 30 days
+- `stockout_days_30d` only counts stockout days where there was ALSO some activity
+- A product out of stock with zero sales gets zero stockout_days even though it was stocked out
+
+This is by design (to avoid creating 681K rows/day) but means stock-related metrics are systematically biased.
+
+---
+
+### ISSUE 4: [HIGH] `costeach` fallback to 50% of price in import pipeline
+
+**File:** `inventory-server/scripts/import/orders.js` (line ~573)
+
+**Description:** When the MySQL `order_costs` table has no record for an order item, `costeach` defaults to `price * 0.5`. There is **no flag** in the PostgreSQL data to distinguish actual costs from estimated ones.
+
+**Data impact:** 385,545 products (56.5%) have `current_cost_price = 0` AND `current_landing_cost_price = 0`. For these products, the COGS calculation in daily_snapshots falls through the chain:
+1. `o.costeach` — May be the 50% estimate from import
+2. `get_weighted_avg_cost()` — Returns NULL if no receivings exist
+3. `p.landing_cost_price` — Always NULL (hardcoded in import)
+4. `p.cost_price` — 0 for 56.5% of products
+
+Only 27 products have zero COGS with positive sales, meaning the `costeach` field is doing its job for products that sell, but the 50% fallback means margins for those products are estimates, not actuals.
+
+---
+
+### ISSUE 5: [HIGH] `landing_cost_price` is always NULL
+
+**File:** `inventory-server/scripts/import/products.js` (line ~175)
+
+**Description:** The import explicitly sets `landing_cost_price = NULL` for all products. The daily_snapshots COGS calculation uses it as a fallback: `COALESCE(o.costeach, get_weighted_avg_cost(...), p.landing_cost_price, p.cost_price)`. Since it's always NULL, this fallback step is useless and the chain jumps straight to `cost_price`.
+
+The `product_metrics` field `current_landing_cost_price` is populated as `COALESCE(p.landing_cost_price, p.cost_price, 0.00)`, so it equals `cost_price` for all products. Any UI showing "landing cost" is actually just showing `cost_price`.
+
+---
+
+### ISSUE 6: [HIGH] Vendor lead time is drastically wrong — missing supplier_id join
+
+**File:** `calculate_vendor_metrics.sql`, lines 62-82
+**Confirmed by data:** Vendor-level lead times are 2-10x higher than product-level lead times
+
+**Description:** The vendor metrics lead time joins POs to receivings only by `pid`:
+```sql
+LEFT JOIN public.receivings r ON r.pid = po.pid
+```
+But the periodic metrics lead time correctly matches supplier:
+```sql
+JOIN public.receivings r ON r.pid = po.pid AND r.supplier_id = po.supplier_id
+```
+
+Without supplier matching, a PO for product X from Vendor A can match a receiving of product X from Vendor B, creating inflated/wrong lead times.
+
+**Measured discrepancies:**
+| Vendor | Vendor Metrics Lead Time | Avg Product Lead Time |
+|---|---|---|
+| doodlebug design inc. | 66 days | 14 days |
+| Notions | 55 days | 4 days |
+| Simple Stories | 59 days | 27 days |
+| Ranger Industries | 31 days | 5 days |
+
+---
+
+### ISSUE 7: [MEDIUM] Net revenue does not subtract returns
+
+**File:** `update_daily_snapshots.sql`, line 184
+
+**Description:** `net_revenue = gross_revenue - discounts`. Standard accounting: `net_revenue = gross_revenue - discounts - returns`. The `returns_revenue` is calculated separately but not deducted.
+
+**Data impact:** There are 3,352 orders with negative quantities (returns), totaling -5,499 units. These returns are tracked in `returns_revenue` but not reflected in `net_revenue`, which means all downstream revenue-based metrics are slightly overstated.
+
+---
+
+### ISSUE 8: [MEDIUM] Lifetime revenue subquery references wrong table columns
+
+**File:** `update_product_metrics.sql`, lines 323-329
+
+**Description:** The lifetime revenue estimation fallback queries:
+```sql
+SELECT revenue_7d / NULLIF(sales_7d, 0)
+FROM daily_product_snapshots
+WHERE pid = ci.pid AND sales_7d > 0
+```
+But `daily_product_snapshots` does NOT have `revenue_7d` or `sales_7d` columns — those exist in `product_metrics`. This subquery either errors silently or returns NULL. The effect is that the estimation always falls back to `current_price * total_sold`.
+
+---
+
+### ISSUE 9: [MEDIUM] Brand/Vendor metrics COGS filter inflates margins
+
+**Files:** `calculate_brand_metrics.sql` lines 31, `calculate_vendor_metrics.sql` line 32
+
+**Description:** `SUM(CASE WHEN pm.cogs_30d > 0 THEN pm.cogs_30d ELSE 0 END)` excludes products with zero COGS. But if a product has sales revenue and zero COGS (missing cost data), the brand/vendor totals will include the revenue but not the COGS, artificially inflating the margin.
+
+**Data context:** Brand metrics revenue matches product_metrics aggregation exactly for sales counts, but shows small discrepancies in revenue (e.g., Stamperia: $7,613.98 brand vs $7,611.11 actual). These tiny diffs come from the `> 0` filtering excluding products with negative revenue.
+
+---
+
+### ISSUE 10: [MEDIUM] Extreme margin values from $0.01 price orders
+
+**Confirmed by data:** 73 products with margin > 100%, 119 with margin < -100%
+
+**Examples:**
+| Product | Revenue | COGS | Margin |
+|---|---|---|---|
+| Flower Gift Box Die (pid 624756) | $0.02 | $29.98 | -149,800% |
+| Special Flowers Stamp Set (pid 614513) | $0.01 | $11.97 | -119,632% |
+
+These are products with extremely low prices (likely samples, promos, or data errors) where the order price was $0.01. The margin calculation is mathematically correct but these outliers skew any aggregate margin statistics.
+
+---
+
+### ISSUE 11: [MEDIUM] Sell-through rate has edge cases yielding negative/extreme values
+
+**File:** `update_product_metrics.sql`, lines 358-361
+**Confirmed by data:** 30 products with negative sell-through, 10 with sell-through > 200%
+
+**Description:** Beginning inventory is approximated as `current_stock + sales - received + returns`. When inventory adjustments, shrinkage, or manual corrections occur, this approximation breaks. Edge cases:
+- Products with many manual stock adjustments → negative denominator → negative sell-through
+- Products with beginning stock near zero but decent sales → sell-through > 100%
+
+---
+
+### ISSUE 12: [MEDIUM] `total_sold` uses different status filter than orders import
+
+**Import pipeline confirmed:**
+- Orders import: `order_status >= 15` (includes processing/pending orders)
+- `total_sold` in products: `order_status >= 20` (more restrictive)
+
+This means `lifetime_sales` (from `total_sold`) is systematically lower than what you'd calculate by summing the orders table. The discrepancy is confirmed:
+| Product | total_sold | orders sum | Gap |
+|---|---|---|---|
+| pid 31286 | 13,786 | 4,241 | 9,545 |
+| pid 44309 | 11,978 | 3,119 | 8,859 |
+
+The large gaps are because the orders table only has data from the import start date (~2024), while `total_sold` includes all-time sales from MySQL. This is expected behavior, not a bug, but it means the `lifetime_revenue_quality` flag is important — most products show 'estimated' quality.
+
+---
+
+### ISSUE 13: [MEDIUM] Category rollup may double-count products in multiple hierarchy levels
+
+**File:** `calculate_category_metrics.sql`, lines 42-66
+
+**Description:** The `RolledUpMetrics` CTE uses:
+```sql
+dcm.cat_id = ch.cat_id OR dcm.cat_id = ANY(SELECT cat_id FROM category_hierarchy WHERE ch.cat_id = ANY(ancestor_ids))
+```
+If products are assigned to categories at multiple levels in the same branch (e.g., both "Paper Crafts" and "Scrapbook Paper" which is a child of "Paper Crafts"), those products' metrics would be counted twice in the parent's rollup.
+
+---
+
+### ISSUE 14: [LOW] `exclude_forecast` removes products from metrics entirely
+
+**File:** `update_product_metrics.sql`, line 509
+
+**Description:** `WHERE s.exclude_forecast IS FALSE OR s.exclude_forecast IS NULL` is on the main INSERT's WHERE clause. Products with `exclude_forecast = TRUE` won't appear in `product_metrics` at all, rather than just having forecast fields nulled. Currently all 681,912 products are in product_metrics so this appears to not affect any products yet.
+
+---
+
+### ISSUE 15: [LOW] Daily snapshots only look back 5 days
+
+**File:** `update_daily_snapshots.sql`, line 14 — `_process_days INT := 5`
+
+If import data arrives late (>5 days), those days will never get snapshots populated. There is a separate `backfill/rebuild_daily_snapshots.sql` for historical rebuilds.
+
+---
+
+### ISSUE 16: [INFO] Timezone risk in order date import
+
+**File:** `inventory-server/scripts/import/orders.js`
+
+MySQL `DATETIME` values are timezone-naive. The import uses `new Date(order.date)` which interprets them using the import server's local timezone. The SSH config specifies `timezone: '-05:00'` for MySQL (always EST). If the import server is in a different timezone, orders near midnight could land on the wrong date in the daily snapshots calculation.
+
+---
+
+## Custom Functions Review
+
+### `calculate_sales_velocity(sales_30d, stockout_days_30d)`
+- Divides `sales_30d` by effective selling days: `GREATEST(30 - stockout_days, CASE WHEN sales > 0 THEN 14 ELSE 30 END)`
+- The 14-day floor prevents extreme velocity for products mostly out of stock
+- **Sound approach** — the only concern is that stockout_days is unreliable (Issues 2, 3)
+
+### `get_weighted_avg_cost(pid, date)`
+- Weighted average of last 10 receivings by cost*qty/qty
+- Returns NULL if no receivings — sound fallback behavior
+- **Correct implementation**
+
+### `safe_divide(numerator, denominator)`
+- Returns NULL on divide-by-zero — **correct**
+
+### `std_numeric(value, precision)`
+- Rounds to precision digits — **correct**
+
+### `classify_demand_pattern(avg_demand, cv)`
+- Uses coefficient of variation thresholds: ≤0.2 = stable, ≤0.5 = variable, low-volume+high-CV = sporadic, else lumpy
+- **Reasonable classification**, though only based on 30-day window
+
+### `detect_seasonal_pattern(pid)`
+- CROSS JOIN LATERAL (runs per product) — **expensive**: queries `daily_product_snapshots` twice per product
+- Compares current month average to yearly average — very simplistic
+- **Functional but could be a performance bottleneck** with 681K products
+
+### `category_hierarchy` (materialized view)
+- Recursive CTE building tree from categories — **correct implementation**
+- Refreshed concurrently before category metrics calculation — **good practice**
+
+---
+
+## Data Health Summary
+
+| Metric | Count | % of Total |
+|---|---|---|
+| Products with zero cost_price | 385,545 | 56.5% |
+| Products with NULL sales_30d | 621,221 | 91.1% |
+| Products with no lifetime_sales | 321,321 | 47.1% |
+| Products with zero COGS but positive sales | 27 | <0.01% |
+| Products with margin > 100% | 73 | <0.01% |
+| Products with margin < -100% | 119 | <0.01% |
+| Products with negative sell-through | 30 | <0.01% |
+| Products with NULL status | 0 | 0% |
+| Duplicate daily snapshots (same pid+date) | 0 | 0% |
+| Net revenue formula mismatches | 0 | 0% |
+
+### ABC Classification Distribution (replenishable products only)
+| Class | Products | Revenue % |
+|---|---|---|
+| A | 7,727 | 80.72% |
+| B | 12,048 | 15.10% |
+| C | 113,647 | 4.18% |
+
+ABC distribution looks healthy — A ≈ 80%, A+B ≈ 96%.
+
+### Brand Metrics Consistency
+Product counts and sales_30d match exactly between `brand_metrics` and direct aggregation from `product_metrics`. Revenue shows sub-dollar discrepancies due to the `> 0` filter excluding products with negative revenue. **Consistent within expected tolerance.**
+
+---
+
+## Priority Recommendations
+
+### Must Fix (Correctness Issues)
+1. **Issue 1: Fix order status handling** — The text-based filter (`NOT IN ('canceled', 'returned')`) is dead code against numeric statuses. Two options: (a) map numeric statuses to text during import (like POs already do), or (b) change SQL to filter on numeric codes (e.g., `o.status::int >= 20` to exclude cancelled/unfinished, or `o.status IN ('100', '95')` for shipped-only). The ~19.7K unfulfilled orders (0.69%) are a minor financial impact but the filter should be functional.
+2. **Issue 6: Add supplier_id join to vendor lead time** — One-line fix in `calculate_vendor_metrics.sql`
+3. **Issue 8: Fix lifetime revenue subquery** — Use correct column names from `daily_product_snapshots` (e.g., `net_revenue / NULLIF(units_sold, 0)`)
+
+### Should Fix (Data Quality)
+4. **Issue 2/3: Snapshot coverage** — Consider creating snapshot rows for all in-stock products, not just those with activity. Or at minimum, calculate stockout metrics by comparing snapshot existence to product existence.
+5. **Issue 5: Populate landing_cost_price** — If available in the source system, import it. Otherwise remove references to avoid confusion.
+6. **Issue 7: Subtract returns from net_revenue** — `net_revenue = gross_revenue - discounts - returns_revenue`
+7. **Issue 9: Remove > 0 filter on COGS** — Use `SUM(pm.cogs_30d)` instead of conditional sums
+
+### Nice to Fix (Edge Cases)
+8. **Issue 4: Flag estimated costs** — Add a `costeach_estimated BOOLEAN` to orders during import
+9. **Issue 10: Cap or flag extreme margins** — Exclude $0.01-price orders from margin calculations
+10. **Issue 11: Clamp sell-through** — `GREATEST(0, LEAST(sell_through_30d, 200))` or flag outliers
+11. **Issue 12: Verify category assignment policy** — Check if products are assigned to leaf categories only
+12. **Issue 13: Category rollup query** — Verify no double-counting with actual data

docs/METRICS_AUDIT2.md

@@ -0,0 +1,276 @@
+# Metrics Pipeline Audit Report
+
+**Date:** 2026-02-08
+**Scope:** All 6 SQL scripts in `inventory-server/scripts/metrics-new/`, import pipeline, custom functions, and post-calculation data verification.
+
+---
+
+## Executive Summary
+
+The metrics pipeline is architecturally sound and the core calculations are mostly correct. The 30-day sales, revenue, replenishment, and aggregate metrics (brand/vendor/category) all cross-check accurately between the snapshots, product_metrics, and direct orders queries. However, several issues were found ranging from **critical data bugs** to **design limitations** that affect accuracy of specific metrics.
+
+**Issues found: 13** (3 Critical, 4 Medium, 6 Low/Informational)
+
+---
+
+## CRITICAL Issues
+
+### C1. `net_revenue` in daily snapshots never subtracts returns ($35.6K affected)
+
+**Location:** `update_daily_snapshots.sql`, line 181
+**Symptom:** `net_revenue` is stored as `gross_revenue - discounts` but should be `gross_revenue - discounts - returns_revenue`.
+
+The SQL formula on line 181 appears correct:
+```sql
+COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00) - COALESCE(sd.returns_revenue, 0.00) AS net_revenue
+```
+
+However, actual data shows `net_revenue = gross_revenue - discounts` for ALL 3,252 snapshots that have returns. Total returns not subtracted: **$35,630.03** across 2,946 products. This may be caused by the `returns_revenue` in the SalesData CTE not properly flowing through to the INSERT, or by a prior version of the code that stored these values differently. The profit column (line 184) has the same issue: `(gross - discounts) - cogs` instead of `(gross - discounts - returns) - cogs`.
+
+**Impact:** Net revenue and profit are overstated by the amount of returns. This cascades to all metrics derived from snapshots: `revenue_30d`, `profit_30d`, `margin_30d`, `avg_ros_30d`, and all brand/vendor/category aggregate revenue.
+
+**Recommended fix:** Debug why the returns subtraction isn't taking effect. The formula in the SQL looks correct, so this may be a data-type issue or an execution path issue. After fixing, rebuild snapshots.
+
+**Status:** Owner will resolve. Code formula is correct; snapshots need rebuilding after prior fix deployment.
+
+---
+
+### C2. `eod_stock_quantity` uses CURRENT stock, not historical end-of-day stock
+
+**Location:** `update_daily_snapshots.sql`, lines 123-132 (CurrentStock CTE)
+**Symptom:** Every snapshot for a given product shows the same stock quantity regardless of the snapshot date.
+
+The `CurrentStock` CTE simply reads `stock_quantity` from the `products` table:
+```sql
+SELECT pid, stock_quantity, ... FROM public.products
+```
+
+This means a snapshot from January 10 shows the SAME stock as today (February 8). Verified in data:
+- Product 662561: stock = 36 on every date (Feb 1-7)
+- Product 665397: stock = 25 on every date (Feb 1-7)
+- All products checked show identical stock across all snapshot dates
+
+**Impact:** All stock-derived metrics are inaccurate for historical analysis:
+- `eod_stock_cost`, `eod_stock_retail`, `eod_stock_gross` (all wrong for past dates)
+- `stockout_flag` (based on current stock, not historical)
+- `stockout_days_30d` (undercounted since stockout_flag uses current stock)
+- `avg_stock_units_30d`, `avg_stock_cost_30d` (no variance, just current stock repeated)
+- `gmroi_30d`, `stockturn_30d` (based on avg_stock which is flat)
+- `sell_through_30d` (denominator uses current stock assumption)
+- `service_level_30d`, `fill_rate_30d`
+
+**This is a known architectural limitation** noted in MEMORY.md. Fixing requires either:
+1. Storing stock snapshots separately at end-of-day (ideally via a cron job that records stock before any changes)
+2. Reconstructing historical stock from orders and receivings (complex but possible)
+
+**Status: FIXED.** MySQL's `snap_product_value` table (daily EOD stock per product since 2012) is now imported into PostgreSQL `stock_snapshots` table via `scripts/import/stock-snapshots.js`. The `CurrentStock` CTE in `update_daily_snapshots.sql` now uses `LEFT JOIN stock_snapshots` for historical stock, falling back to `products.stock_quantity` when no historical data exists. Requires: run import, then rebuild daily snapshots.
+
+---
+
+### C3. `ON CONFLICT DO UPDATE WHERE` check skips 91%+ of product_metrics updates
+
+**Location:** `update_product_metrics.sql`, lines 558-574
+**Symptom:** 623,205 of 681,912 products (91.4%) have `last_calculated` older than 1 day. 592,369 are over 30 days old. 914 products with active 30-day sales haven't been updated in over 7 days.
+
+The upsert's `WHERE` clause only updates if specific fields changed:
+```sql
+WHERE product_metrics.current_stock IS DISTINCT FROM EXCLUDED.current_stock OR
+      product_metrics.current_price IS DISTINCT FROM EXCLUDED.current_price OR ...
+```
+
+Fields NOT checked include: `stockout_days_30d`, `margin_30d`, `gmroi_30d`, `demand_pattern`, `seasonality_index`, `sales_growth_*`, `service_level_30d`, and many others. If a product's stock, price, sales, and revenue haven't changed, the entire row is skipped even though growth metrics, variability, and other derived fields may need updating.
+
+**Impact:** Most derived metrics (growth, demand patterns, seasonality) are stale for the majority of products. Products with steady sales but unchanged stock/price never get their growth metrics recalculated.
+
+**Recommended fix:** Either:
+1. Remove the `WHERE` clause entirely (accept the performance cost of writing all rows every run)
+2. Add `last_calculated` age check: `OR product_metrics.last_calculated < NOW() - INTERVAL '7 days'`
+3. Add the missing fields to the change-detection check
+
+**Status: FIXED.** Added 12 derived fields to the `IS DISTINCT FROM` check (`profit_30d`, `cogs_30d`, `margin_30d`, `stockout_days_30d`, `sell_through_30d`, `sales_growth_30d_vs_prev`, `revenue_growth_30d_vs_prev`, `demand_pattern`, `seasonal_pattern`, `seasonality_index`, `service_level_30d`, `fill_rate_30d`) plus a time-based safety net: `OR product_metrics.last_calculated < NOW() - INTERVAL '1 day'`. This guarantees every row is refreshed at least daily.
+
+---
+
+## MEDIUM Issues
+
+### M1. Demand variability calculated only over activity days, not full 30-day window
+
+**Location:** `update_product_metrics.sql`, DemandVariability CTE (lines 206-223)
+**Symptom:** Variance, std_dev, and CV are computed over only the days that appear in snapshots (activity days), not the full 30-day period including zero-sales days.
+
+Example: Product 41141 (Mexican Poppy) sold 102 units in 30 days across only 3 snapshot days (1, 1, 100). The variance/CV is calculated over just those 3 data points instead of 30 (with 27 zero-sales days).
+
+**Impact:**
+- CV is computed on sparse data (3-10 points instead of 30), making it statistically unreliable
+- Products with sporadic large orders appear less variable than they really are
+- `demand_pattern` classification is affected (stable/variable/sporadic/lumpy)
+
+**Recommended fix:** Join against a generated 30-day date series and COALESCE missing days to 0 units sold before computing variance/stddev/CV.
+
+**Status: FIXED.** Rewrote `DemandVariability` CTE to use `generate_series()` for the full 30-day date range, `CROSS JOIN` with distinct PIDs from snapshots, and `LEFT JOIN` actual snapshot data with `COALESCE(dps.units_sold, 0)` for missing days. Variance/stddev/CV now computed over all 30 data points.
+
+---
+
+### M2. `costeach` fallback to `price * 0.5` affects 32.5% of recent orders
+
+**Location:** `orders.js`, line 600 and 634
+**Symptom:** When no cost record exists in `order_costs`, the import falls back to `price * 0.5`.
+
+Data shows 9,839 of 30,266 recent orders (32.5%) use this fallback. Among these, 79 paid products have `costeach = 0` because `price = 0 * 0.5 = 0`, even though the product has a real cost_price.
+
+The daily snapshot has a second line of defense (using `get_weighted_avg_cost()` and then `p.cost_price`), but the orders table's `costeach` column itself contains inaccurate data for ~1/3 of orders.
+
+**Impact:** COGS calculations at the order level are approximate for 1/3 of orders. The snapshot's fallback chain mitigates this somewhat, but any analytics using `orders.costeach` directly will be affected.
+
+**Status: FIXED.** Added `products.cost_price` as intermediate fallback: `COALESCE(oc.costeach, p.cost_price, oi.price * 0.5)`. The products table join was added to both the `order_totals` CTE and the outer SELECT in `orders.js`. Requires a full orders re-import to apply retroactively.
+
+---
+
+### M3. `lifetime_sales` uses MySQL `total_sold` (status >= 20) but orders import uses status >= 15
+
+**Location:** `products.js` line 200 vs `orders.js` line 69
+**Symptom:** `total_sold` in the products table comes from MySQL with `order_status >= 20`, excluding status 15 (canceled) and 16 (combined). But the orders import fetches orders with `order_status >= 15`.
+
+Verified in MySQL: For product 31286, `total_sold` (>=20) = 13,786 vs (>=15) = 13,905 (difference of 119 units).
+
+**Impact:** `lifetime_sales` in product_metrics (sourced from `products.total_sold`) slightly understates compared to what the orders table contains. The `lifetime_revenue_quality` field correctly flags most as "estimated" since the orders table only covers ~5 years while `total_sold` is all-time. This is a minor inconsistency (< 1% difference).
+
+**Status:** Accepted. < 1% difference, not worth the complexity of aligning thresholds.
+
+---
+
+### M4. `sell_through_30d` has 868 NULL values and 547 anomalous values for products with sales
+
+**Location:** `update_product_metrics.sql`, lines 356-361
+**Formula:** `(sales_30d / (current_stock + sales_30d + returns_units_30d - received_qty_30d)) * 100`
+
+- 868 products with sales but NULL sell_through (denominator = 0, which happens when `current_stock + sales - received = 0`, i.e. all stock came from receiving and was sold)
+- 259 products with sell_through > 100%
+- 288 products with negative sell_through
+
+**Impact:** Sell-through rate is unreliable for products with significant receiving activity in the same period. The formula tries to approximate "beginning inventory" but the approximation breaks when current stock ≠ actual beginning stock (which is always, per issue C2).
+
+**Status:** Will improve once C2 fix (historical stock) is deployed and snapshots are rebuilt, since `current_stock` in the formula will then reflect actual beginning inventory.
+
+---
+
+## LOW / INFORMATIONAL Issues
+
+### L1. Snapshots only cover ~1,167 products/day out of 681K
+
+Only products with order or receiving activity on a given day get snapshots. This is by design (the `ProductsWithActivity` CTE on line 133 of `update_daily_snapshots.sql`), but it means:
+- 560K+ products have zero snapshot history
+- Stockout tracking is impossible for products with no sales (they can't appear in snapshots)
+- The "avg_stock" metrics (avg_stock_units_30d, etc.) only average over activity days, not all 30 days
+
+This is acceptable for storage efficiency but should be understood when interpreting metrics.
+
+**Status:** Accepted (by design).
+
+---
+
+### L2. `detect_seasonal_pattern` function only compares current month to yearly average
+
+The seasonality detection is simplistic: it compares current month's avg daily sales to yearly avg. This means:
+- It can only detect if the CURRENT month is above average, not identify historical seasonal patterns
+- Running in January vs July will give completely different results for the same product
+- The "peak_season" field always shows the current month/quarter when seasonal (not the actual peak)
+
+This is noted as a P5 (low priority) feature and is adequate for a first pass but should not be relied upon for demand planning.
+
+**Status: FIXED.** Rewrote `detect_seasonal_pattern` function to compare monthly average sales across the full last 12 months. Uses CV across months + peak-to-average ratio for classification: `strong` (CV > 0.5, peak > 150%), `moderate` (CV > 0.3, peak > 120%), `none`. Peak season now identifies the actual highest-sales month. Requires at least 3 months of data. Saved in `db/functions.sql`.
+
+---
+
+### L3. Free product with negative revenue in top sellers
+
+Product 476848 ("Thank You, From ACOT!") shows 254 sales with -$1.00 revenue because one order applied a $1 discount to a $0 product. This is a data oddity, not a calculation bug. Could be addressed by excluding $0-price products from revenue metrics or by data cleanup.
+
+**Status:** Accepted (data oddity, not a bug).
+
+---
+
+### L4. `landing_cost_price` is always NULL
+
+`current_landing_cost_price` in product_metrics is mapped from `current_effective_cost` which is just `cost_price`. The `landing_cost_price` concept (cost + shipping + duties) is not implemented. The field exists but has no meaningful data.
+
+**Status: FIXED.** Removed `landing_cost_price` from `db/schema.sql`, `current_landing_cost_price` from `db/metrics-schema-new.sql`, `update_product_metrics.sql`, and `backfill/populate_initial_product_metrics.sql`. Column should be dropped from the live database via `ALTER TABLE`.
+
+---
+
+### L5. Custom SQL functions not tracked in version control
+
+All 6 custom functions (`calculate_sales_velocity`, `get_weighted_avg_cost`, `safe_divide`, `std_numeric`, `classify_demand_pattern`, `detect_seasonal_pattern`) and the `category_hierarchy` materialized view exist only in the database. They are not defined in any migration or schema file in the repository.
+
+If the database needs to be recreated, these would be lost.
+
+**Status: FIXED.** All 6 functions and the `category_hierarchy` materialized view definition saved to `inventory-server/db/functions.sql`. File is re-runnable via `psql -f functions.sql`.
+
+---
+
+### L6. `get_weighted_avg_cost` limited to last 10 receivings
+
+The function `LIMIT 10` for performance, but this means products with many small receivings may not accurately reflect the true weighted average cost if the cost has changed significantly beyond the last 10 receiving records.
+
+**Status: FIXED.** Removed `LIMIT 10` from `get_weighted_avg_cost`. Data shows max receivings per product is 142 (p95 = 11, avg = 3), so performance impact is negligible. Updated definition in `db/functions.sql`.
+
+---
+
+## Verification Summary
+
+### What's Working Correctly
+
+| Check | Result |
+|-------|--------|
+| 30d sales: product_metrics vs orders vs snapshots | **MATCH** (verified top 10 sellers) |
+| Replenishment formula: manual calc vs stored | **MATCH** (verified 10 products) |
+| Brand metrics vs sum of product_metrics | **MATCH** (0 difference across all brands) |
+| Order status mapping (numeric → text) | **CORRECT** (all statuses mapped, no numeric remain) |
+| Cost price: PostgreSQL vs MySQL source | **MATCH** (within rounding, verified 5 products) |
+| total_sold: PostgreSQL vs MySQL source | **MATCH** (verified 5 products) |
+| Category rollups (rolled-up > direct for parents) | **CORRECT** |
+| ABC classification distribution | **REASONABLE** (A: 8K, B: 12.5K, C: 113K) |
+| Lead time calculation (PO → receiving) | **CORRECT** (verified examples) |
+
+### Data Overview
+
+| Metric | Value |
+|--------|-------|
+| Total products | 681,912 |
+| Products in product_metrics | 681,912 (100%) |
+| Products with 30d sales | 10,291 (1.5%) |
+| Products with negative profit & revenue | 139 (mostly cost > price) |
+| Products with negative stock | 0 |
+| Snapshot date range | 2020-06-18 to 2026-02-08 |
+| Avg products per snapshot day | 1,167 |
+| Order date range | 2020-06-18 to 2026-02-08 |
+| Total orders | 2,885,825 |
+| 'returned' status orders | 0 (returns via negative quantity only) |
+
+---
+
+## Fix Status Summary
+
+| Issue | Severity | Status | Deployment Action Needed |
+|-------|----------|--------|--------------------------|
+| C1 | Critical | Owner resolving | Rebuild daily snapshots |
+| C2 | Critical | **FIXED** | Run import, rebuild daily snapshots |
+| C3 | Critical | **FIXED** | Deploy updated `update_product_metrics.sql` |
+| M1 | Medium | **FIXED** | Deploy updated `update_product_metrics.sql` |
+| M2 | Medium | **FIXED** | Full orders re-import (`--full`) |
+| M3 | Medium | Accepted | None |
+| M4 | Medium | Pending C2 | Will improve after C2 deployment |
+| L1 | Low | Accepted | None |
+| L2 | Low | **FIXED** | Deploy `db/functions.sql` to database |
+| L3 | Low | Accepted | None |
+| L4 | Low | **FIXED** | `ALTER TABLE` to drop columns |
+| L5 | Low | **FIXED** | None (file committed) |
+| L6 | Low | **FIXED** | Deploy `db/functions.sql` to database |
+
+### Deployment Steps
+
+1. Deploy `db/functions.sql` to PostgreSQL: `psql -d inventory_db -f db/functions.sql` (L2, L6)
+2. Run import (includes stock snapshots first load) (C2, M2)
+3. Drop stale columns: `ALTER TABLE products DROP COLUMN IF EXISTS landing_cost_price; ALTER TABLE product_metrics DROP COLUMN IF EXISTS current_landing_cost_price;` (L4)
+4. Rebuild daily snapshots (C1, C2)
+5. Re-run metrics calculation (C3, M1 take effect automatically)

docs/PERMISSIONS.md

@@ -7,12 +7,13 @@ This document outlines the permission system implemented in the Inventory Manage
 Permissions follow this naming convention:
 
 - Page access: `access:{page_name}`
-- Actions: `{action}:{resource}`
+- Settings sections: `settings:{section_name}`
+- Admin features: `admin:{feature}`
 
 Examples:
 - `access:products` - Can access the Products page
-- `create:products` - Can create new products
+- `settings:user_management` - Can access User Management settings
-- `edit:users` - Can edit user accounts
+- `admin:debug` - Can see debug information
 
 ## Permission Components
 
@@ -22,10 +23,10 @@ The core component that conditionally renders content based on permissions.
 
 ```tsx
 <PermissionGuard 
-  permission="create:products"
+  permission="settings:user_management"
   fallback={<p>No permission</p>}
 >
-  <button>Create Product</button>
+  <button>Manage Users</button>
 </PermissionGuard>
 ```
 
@@ -81,7 +82,7 @@ Specific component for settings with built-in permission checks.
 <SettingsSection
   title="System Settings"
   description="Configure global settings"
-  permission="edit:system_settings"
+  permission="settings:global"
 >
   {/* Settings content */}
 </SettingsSection>
@@ -95,8 +96,8 @@ Core hook for checking any permission.
 
 ```tsx
 const { hasPermission, hasPageAccess, isAdmin } = usePermissions();
-if (hasPermission('delete:products')) {
+if (hasPermission('settings:user_management')) {
-  // Can delete products
+  // Can access user management
 }
 ```
 
@@ -106,8 +107,8 @@ Specialized hook for page-level permissions.
 
 ```tsx
 const { canView, canCreate, canEdit, canDelete } = usePagePermission('products');
-if (canEdit()) {
+if (canView()) {
-  // Can edit products
+  // Can view products
 }
 ```
 
@@ -119,18 +120,43 @@ Permissions are stored in the database:
 
 Admin users automatically have all permissions.
 
-## Common Permission Codes
+## Implemented Permission Codes
 
+### Page Access Permissions
 | Code | Description |
 |------|-------------|
 | `access:dashboard` | Access to Dashboard page |
+| `access:overview` | Access to Overview page |
 | `access:products` | Access to Products page |
-| `create:products` | Create new products |
+| `access:categories` | Access to Categories page |
-| `edit:products` | Edit existing products |
+| `access:brands` | Access to Brands page |
-| `delete:products` | Delete products |
+| `access:vendors` | Access to Vendors page |
-| `view:users` | View user accounts |
+| `access:purchase_orders` | Access to Purchase Orders page |
-| `edit:users` | Edit user accounts |
+| `access:analytics` | Access to Analytics page |
-| `manage:permissions` | Assign permissions to users |
+| `access:forecasting` | Access to Forecasting page |
+| `access:import` | Access to Import page |
+| `access:settings` | Access to Settings page |
+| `access:chat` | Access to Chat Archive page |
+
+### Settings Permissions
+| Code | Description |
+|------|-------------|
+| `settings:global` | Access to Global Settings section |
+| `settings:products` | Access to Product Settings section |
+| `settings:vendors` | Access to Vendor Settings section |
+| `settings:data_management` | Access to Data Management settings |
+| `settings:calculation_settings` | Access to Calculation Settings |
+| `settings:library_management` | Access to Image Library Management |
+| `settings:performance_metrics` | Access to Performance Metrics |
+| `settings:prompt_management` | Access to AI Prompt Management |
+| `settings:stock_management` | Access to Stock Management |
+| `settings:templates` | Access to Template Management |
+| `settings:user_management` | Access to User Management |
+
+### Admin Permissions
+| Code | Description |
+|------|-------------|
+| `admin:debug` | Can see debug information and features |
 
 ## Implementation Examples
 
@@ -148,25 +174,31 @@ In `App.tsx`:
 ### Component Level Protection
 
 ```tsx
-const { canEdit } = usePagePermission('products');
+const { hasPermission } = usePermissions();
 
-function handleEdit() {
+function handleAction() {
-  if (!canEdit()) {
+  if (!hasPermission('settings:user_management')) {
     toast.error("You don't have permission");
     return;
   }
-  // Edit logic
+  // Action logic
 }
 ```
 
 ### UI Element Protection
 
 ```tsx
-<PermissionButton 
+<PermissionGuard permission="settings:user_management">
-  page="products" 
+  <button onClick={handleManageUsers}>
-  action="delete"
+    Manage Users
-  onClick={handleDelete}
+  </button>
->
+</PermissionGuard>
-  Delete
+```
-</PermissionButton>
+
-``` 
+## Notes
+
+- **Page Access**: These permissions control which pages a user can navigate to
+- **Settings Access**: These permissions control access to different sections within the Settings page
+- **Admin Features**: Special permissions for administrative functions
+- **CRUD Operations**: The application currently focuses on viewing and managing data rather than creating/editing/deleting individual records
+- **User Management**: User CRUD operations are handled through the settings interface rather than dedicated user management pages 

docs/import-from-prod-data-mapping.md

@@ -0,0 +1,342 @@
+# MySQL to PostgreSQL Import Process Documentation
+
+This document outlines the data import process from the production MySQL database to the local PostgreSQL database, focusing on column mappings, data transformations, and the overall import architecture.
+
+## Table of Contents
+1. [Overview](#overview)
+2. [Import Architecture](#import-architecture)
+3. [Column Mappings](#column-mappings)
+    - [Categories](#categories)
+    - [Products](#products)
+    - [Product Categories (Relationship)](#product-categories-relationship)
+    - [Orders](#orders)
+    - [Purchase Orders](#purchase-orders)
+    - [Metadata Tables](#metadata-tables)
+4. [Special Calculations](#special-calculations)
+5. [Implementation Notes](#implementation-notes)
+
+## Overview
+
+The import process extracts data from a MySQL 5.7 production database and imports it into a PostgreSQL database. It can operate in two modes:
+
+- **Full Import**: Imports all data regardless of last sync time
+- **Incremental Import**: Only imports data that has changed since the last import
+
+The process handles four main data types:
+- Categories (product categorization hierarchy)
+- Products (inventory items)
+- Orders (sales records)
+- Purchase Orders (vendor orders)
+
+## Import Architecture
+
+The import process follows these steps:
+
+1. **Establish Connection**: Creates a SSH tunnel to the production server and establishes database connections
+2. **Setup Import History**: Creates a record of the current import operation
+3. **Import Categories**: Processes product categories in hierarchical order
+4. **Import Products**: Processes products with their attributes and category relationships
+5. **Import Orders**: Processes customer orders with line items, taxes, and discounts
+6. **Import Purchase Orders**: Processes vendor purchase orders with line items
+7. **Record Results**: Updates the import history with results
+8. **Close Connections**: Cleans up connections and resources
+
+Each import step uses temporary tables for processing and wraps operations in transactions to ensure data consistency.
+
+## Column Mappings
+
+### Categories
+| PostgreSQL Column | MySQL Source                    | Transformation                               |
+|-------------------|---------------------------------|----------------------------------------------|
+| cat_id            | product_categories.cat_id       | Direct mapping                               |
+| name              | product_categories.name         | Direct mapping                               |
+| type              | product_categories.type         | Direct mapping                               |
+| parent_id         | product_categories.master_cat_id| NULL for top-level categories (types 10, 20) |
+| description       | product_categories.combined_name| Direct mapping                               |
+| status            | N/A                             | Hard-coded 'active'                          |
+| created_at        | N/A                             | Current timestamp                            |
+| updated_at        | N/A                             | Current timestamp                            |
+
+**Notes:**
+- Categories are processed in hierarchical order by type: [10, 20, 11, 21, 12, 13]
+- Type 10/20 are top-level categories with no parent
+- Types 11/21/12/13 are child categories that reference parent categories
+
+### Products
+| PostgreSQL Column    | MySQL Source                     | Transformation                                                |
+|----------------------|----------------------------------|---------------------------------------------------------------|
+| pid                  | products.pid                     | Direct mapping                                                |
+| title                | products.description             | Direct mapping                                                |
+| description          | products.notes                   | Direct mapping                                                |
+| sku                  | products.itemnumber              | Fallback to 'NO-SKU' if empty                                 |
+| stock_quantity       | shop_inventory.available_local   | Capped at 5000, minimum 0                                     |
+| preorder_count       | current_inventory.onpreorder     | Default 0                                                     |
+| notions_inv_count    | product_notions_b2b.inventory    | Default 0                                                     |
+| price                | product_current_prices.price_each| Default 0, filtered on active=1                              |
+| regular_price        | products.sellingprice            | Default 0                                                     |
+| cost_price           | product_inventory                | Weighted average: SUM(costeach * count) / SUM(count) when count > 0, or latest costeach |
+| vendor               | suppliers.companyname            | Via supplier_item_data.supplier_id                            |
+| vendor_reference     | supplier_item_data               | supplier_itemnumber or notions_itemnumber based on vendor     |
+| notions_reference    | supplier_item_data.notions_itemnumber | Direct mapping                                           |
+| brand                | product_categories.name          | Linked via products.company                                   |
+| line                 | product_categories.name          | Linked via products.line                                      |
+| subline              | product_categories.name          | Linked via products.subline                                   |
+| artist               | product_categories.name          | Linked via products.artist                                    |
+| categories           | product_category_index           | Comma-separated list of category IDs                          |
+| created_at           | products.date_created            | Validated date, NULL if invalid                               |
+| first_received       | products.datein                  | Validated date, NULL if invalid                               |
+| landing_cost_price   | NULL                             | Not set                                                       |
+| barcode              | products.upc                     | Direct mapping                                                |
+| harmonized_tariff_code| products.harmonized_tariff_code | Direct mapping                                                |
+| updated_at           | products.stamp                   | Validated date, NULL if invalid                               |
+| visible              | shop_inventory                   | Calculated from show + buyable > 0                            |
+| managing_stock       | N/A                              | Hard-coded true                                               |
+| replenishable        | Multiple fields                  | Complex calculation based on reorder, dates, etc.             |
+| permalink            | N/A                              | Constructed URL with product ID                               |
+| moq                  | supplier_item_data               | notions_qty_per_unit or supplier_qty_per_unit, minimum 1      |
+| uom                  | N/A                              | Hard-coded 1                                                  |
+| rating               | products.rating                  | Direct mapping                                                |
+| reviews              | products.rating_votes            | Direct mapping                                                |
+| weight               | products.weight                  | Direct mapping                                                |
+| length               | products.length                  | Direct mapping                                                |
+| width                | products.width                   | Direct mapping                                                |
+| height               | products.height                  | Direct mapping                                                |
+| country_of_origin    | products.country_of_origin       | Direct mapping                                                |
+| location             | products.location                | Direct mapping                                                |
+| total_sold           | order_items                      | SUM(qty_ordered) for all order_items where prod_pid = pid     |
+| baskets              | mybasket                         | COUNT of records where mb.item = pid and qty > 0              |
+| notifies             | product_notify                   | COUNT of records where pn.pid = pid                           |
+| date_last_sold       | product_last_sold.date_sold      | Validated date, NULL if invalid                               |
+| image                | N/A                              | Constructed from pid and image URL pattern                    |
+| image_175            | N/A                              | Constructed from pid and image URL pattern                    |
+| image_full           | N/A                              | Constructed from pid and image URL pattern                    |
+| options              | NULL                             | Not set                                                       |
+| tags                 | NULL                             | Not set                                                       |
+
+**Notes:**
+- Replenishable calculation: 
+  ```javascript
+  CASE 
+    WHEN p.reorder < 0 THEN 0
+    WHEN (
+      (COALESCE(pls.date_sold, '0000-00-00') = '0000-00-00' OR pls.date_sold <= DATE_SUB(CURRENT_DATE, INTERVAL 5 YEAR))
+      AND (p.datein = '0000-00-00 00:00:00' OR p.datein <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
+      AND (p.date_refill = '0000-00-00 00:00:00' OR p.date_refill <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
+    ) THEN 0
+    ELSE 1
+  END
+  ```
+
+In business terms, a product is considered NOT replenishable only if:
+- It was manually flagged as not replenishable (negative reorder value)
+- OR it shows no activity across ALL metrics (no sales AND no receipts AND no refills in the past 5 years)
+- Image URLs are constructed using this pattern:
+  ```javascript
+  const paddedPid = pid.toString().padStart(6, '0');
+  const prefix = paddedPid.slice(0, 3);
+  const basePath = `${imageUrlBase}${prefix}/${pid}`;
+  return {
+    image: `${basePath}-t-${iid}.jpg`,
+    image_175: `${basePath}-175x175-${iid}.jpg`,
+    image_full: `${basePath}-o-${iid}.jpg`
+  };
+  ```
+
+### Product Categories (Relationship)
+
+| PostgreSQL Column | MySQL Source                      | Transformation                                                |
+|-------------------|-----------------------------------|---------------------------------------------------------------|
+| pid               | products.pid                      | Direct mapping                                                |
+| cat_id            | product_category_index.cat_id     | Direct mapping, filtered by category types                    |
+
+**Notes:**
+- Only categories of types 10, 20, 11, 21, 12, 13 are imported
+- Categories 16 and 17 are explicitly excluded
+
+### Orders
+
+| PostgreSQL Column | MySQL Source                      | Transformation                                                |
+|-------------------|-----------------------------------|---------------------------------------------------------------|
+| order_number      | order_items.order_id              | Direct mapping                                                |
+| pid               | order_items.prod_pid              | Direct mapping                                                |
+| sku               | order_items.prod_itemnumber       | Fallback to 'NO-SKU' if empty                                 |
+| date              | _order.date_placed_onlydate       | Via join to _order table                                      |
+| price             | order_items.prod_price            | Direct mapping                                                |
+| quantity          | order_items.qty_ordered           | Direct mapping                                                |
+| discount          | Multiple sources                  | Complex calculation (see notes)                               |
+| tax               | order_tax_info_products.item_taxes_to_collect | Via latest order_tax_info record                  |
+| tax_included      | N/A                               | Hard-coded false                                              |
+| shipping          | N/A                               | Hard-coded 0                                                  |
+| customer          | _order.order_cid                  | Direct mapping                                                |
+| customer_name     | users                             | CONCAT(users.firstname, ' ', users.lastname)                  |
+| status            | _order.order_status               | Direct mapping                                                |
+| canceled          | _order.date_cancelled             | Boolean: true if date_cancelled is not '0000-00-00 00:00:00'  |
+| costeach          | order_costs                       | From latest record or fallback to price * 0.5                 |
+
+**Notes:**
+- Only orders with order_status >= 15 and with a valid date_placed are processed
+- For incremental imports, only orders modified since last sync are processed
+- Discount calculation combines three sources:
+  1. Base discount: order_items.prod_price_reg - order_items.prod_price
+  2. Promo discount: SUM of order_discount_items.amount
+  3. Proportional order discount: Calculation based on order subtotal proportion
+  ```javascript
+  (oi.base_discount + 
+   COALESCE(ot.promo_discount, 0) + 
+   CASE 
+    WHEN om.summary_discount > 0 AND om.summary_subtotal > 0 THEN 
+      ROUND((om.summary_discount * (oi.price * oi.quantity)) / NULLIF(om.summary_subtotal, 0), 2)
+    ELSE 0 
+   END)::DECIMAL(10,2)
+  ```
+- Taxes are taken from the latest tax record for an order
+- Cost data is taken from the latest non-pending cost record
+
+### Purchase Orders
+
+| PostgreSQL Column | MySQL Source                      | Transformation                                                |
+|-------------------|-----------------------------------|---------------------------------------------------------------|
+| po_id             | po.po_id                          | Default 0 if NULL                                             |
+| pid               | po_products.pid                   | Direct mapping                                                |
+| sku               | products.itemnumber               | Fallback to 'NO-SKU' if empty                                 |
+| name              | products.description              | Fallback to 'Unknown Product'                                 |
+| cost_price        | po_products.cost_each             | Direct mapping                                                |
+| po_cost_price     | po_products.cost_each             | Duplicate of cost_price                                       |
+| vendor            | suppliers.companyname             | Fallback to 'Unknown Vendor' if empty                         |
+| date              | po.date_ordered                   | Fallback to po.date_created if NULL                           |
+| expected_date     | po.date_estin                     | Direct mapping                                                |
+| status            | po.status                         | Default 1 if NULL                                             |
+| notes             | po.short_note                     | Fallback to po.notes if NULL                                  |
+| ordered           | po_products.qty_each              | Direct mapping                                                |
+| received          | N/A                               | Hard-coded 0                                                  |
+| receiving_status  | N/A                               | Hard-coded 1                                                  |
+
+**Notes:**
+- Only POs created within last 1 year (incremental) or 5 years (full) are processed
+- For incremental imports, only POs modified since last sync are processed
+
+### Metadata Tables
+
+#### import_history
+
+| PostgreSQL Column | Source                            | Notes                                                         |
+|-------------------|-----------------------------------|---------------------------------------------------------------|
+| id                | Auto-increment                    | Primary key                                                   |
+| table_name        | Code                              | 'all_tables' for overall import                               |
+| start_time        | NOW()                             | Import start time                                             |
+| end_time          | NOW()                             | Import completion time                                        |
+| duration_seconds  | Calculation                       | Elapsed seconds                                               |
+| is_incremental    | INCREMENTAL_UPDATE                | Flag from config                                              |
+| records_added     | Calculation                       | Sum from all imports                                          |
+| records_updated   | Calculation                       | Sum from all imports                                          |
+| status            | Code                              | 'running', 'completed', 'failed', or 'cancelled'              |
+| error_message     | Exception                         | Error message if failed                                       |
+| additional_info   | JSON                              | Configuration and results                                     |
+
+#### sync_status
+
+| PostgreSQL Column    | Source                         | Notes                                                         |
+|----------------------|--------------------------------|---------------------------------------------------------------|
+| table_name           | Code                           | Name of imported table                                        |
+| last_sync_timestamp  | NOW()                          | Timestamp of successful sync                                  |
+| last_sync_id         | NULL                           | Not used currently                                            |
+
+## Special Calculations
+
+### Date Validation
+
+MySQL dates are validated before insertion into PostgreSQL:
+
+```javascript
+function validateDate(mysqlDate) {
+  if (!mysqlDate || mysqlDate === '0000-00-00' || mysqlDate === '0000-00-00 00:00:00') {
+    return null;
+  }
+  // Check if the date is valid
+  const date = new Date(mysqlDate);
+  return isNaN(date.getTime()) ? null : mysqlDate;
+}
+```
+
+### Retry Mechanism
+
+Operations that might fail temporarily are retried with exponential backoff:
+
+```javascript
+async function withRetry(operation, errorMessage) {
+  let lastError;
+  for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
+    try {
+      return await operation();
+    } catch (error) {
+      lastError = error;
+      console.error(`${errorMessage} (Attempt ${attempt}/${MAX_RETRIES}):`, error);
+      if (attempt < MAX_RETRIES) {
+        const backoffTime = RETRY_DELAY * Math.pow(2, attempt - 1);
+        await new Promise(resolve => setTimeout(resolve, backoffTime));
+      }
+    }
+  }
+  throw lastError;
+}
+```
+
+### Progress Tracking
+
+Progress is tracked with estimated time remaining:
+
+```javascript
+function estimateRemaining(startTime, current, total) {
+  if (current === 0) return "Calculating...";
+  const elapsedSeconds = (Date.now() - startTime) / 1000;
+  const itemsPerSecond = current / elapsedSeconds;
+  const remainingItems = total - current;
+  const remainingSeconds = remainingItems / itemsPerSecond;
+  return formatElapsedTime(remainingSeconds);
+}
+```
+
+## Implementation Notes
+
+### Transaction Management
+
+All imports use transactions to ensure data consistency:
+
+- **Categories**: Uses savepoints for each category type
+- **Products**: Uses a single transaction for the entire import
+- **Orders**: Uses a single transaction with temporary tables
+- **Purchase Orders**: Uses a single transaction with temporary tables
+
+### Memory Usage Optimization
+
+To minimize memory usage when processing large datasets:
+
+1. Data is processed in batches (100-5000 records per batch)
+2. Temporary tables are used for intermediate data
+3. Some queries use cursors to avoid loading all results at once
+
+### MySQL vs PostgreSQL Compatibility
+
+The scripts handle differences between MySQL and PostgreSQL:
+
+1. MySQL-specific syntax like `USE INDEX` is removed for PostgreSQL
+2. `GROUP_CONCAT` in MySQL becomes string operations in PostgreSQL
+3. Transaction syntax differences are abstracted in the connection wrapper
+4. PostgreSQL's `ON CONFLICT` replaces MySQL's `ON DUPLICATE KEY UPDATE`
+
+### SSH Tunnel
+
+Database connections go through an SSH tunnel for security:
+
+```javascript
+ssh.forwardOut(
+  "127.0.0.1",
+  0,
+  sshConfig.prodDbConfig.host,
+  sshConfig.prodDbConfig.port,
+  async (err, stream) => {
+    if (err) reject(err);
+    resolve({ ssh, stream });
+  }
+);
+``` 

docs/routes-cleanup.md

@@ -0,0 +1,271 @@
+**Analysis of Potential Issues**
+
+1.  **Obsolete Functionality:**
+    *   **`config.js` Legacy Endpoints:** The endpoints `GET /config/`, `PUT /config/stock-thresholds/:id`, `PUT /config/lead-time-thresholds/:id`, `PUT /config/sales-velocity/:id`, `PUT /config/abc-classification/:id`, `PUT /config/safety-stock/:id`, and `PUT /config/turnover/:id` appear **highly likely to be obsolete**. They reference older, single-row config tables (`stock_thresholds`, etc.) while newer endpoints (`/config/global`, `/config/products`, `/config/vendors`) manage settings in more structured tables (`settings_global`, `settings_product`, `settings_vendor`). Unless specifically required for backward compatibility, these legacy endpoints should be removed to avoid confusion and potential data conflicts.
+    *   **`analytics.js` Forecast Endpoint (`GET /analytics/forecast`):** This endpoint uses **MySQL syntax** (`DATEDIFF`, `DATE_FORMAT`, `JSON_OBJECT`, `?` placeholders) but seems intended to run within the analytics module which otherwise uses PostgreSQL (`req.app.locals.pool`, `date_trunc`, `::text`, `$1` placeholders). This endpoint is likely **obsolete or misplaced** and will not function correctly against the PostgreSQL database.
+    *   **`csv.js` Redundant Actions:**
+        *   `POST /csv/update` seems redundant with `POST /csv/full-update`. The latter uses the `runScript` helper and dedicated state (`activeFullUpdate`), appearing more robust. `/csv/update` might be older or incomplete.
+        *   `POST /csv/reset` seems redundant with `POST /csv/full-reset`. Similar reasoning applies; `/csv/full-reset` appears preferred.
+    *   **`products.js` Import Endpoint (`POST /products/import`):** This is **dangerous duplication**. The `/csv` module handles imports (`/csv/import`, `/csv/import-from-prod`) with locking (`activeImport`) to prevent concurrent operations. This endpoint lacks such locking and could corrupt data if run simultaneously with other CSV/reset operations. It should likely be removed.
+    *   **`products.js` Metrics Endpoint (`GET /products/:id/metrics`):** This is redundant. The `/metrics/:pid` endpoint provides the same, possibly more comprehensive, data directly from the `product_metrics` table. Clients should use `/metrics/:pid` instead.
+
+2.  **Overlap or Inappropriate Duplication of Effort:**
+    *   **AI Prompt Getters:** `GET /ai-prompts/type/general` and `GET /ai-prompts/type/system` could potentially be handled by adding a query parameter filter to `GET /ai-prompts/` (e.g., `GET /ai-prompts?prompt_type=general`). However, dedicated endpoints for single, specific items can sometimes be simpler. This is more of a design choice than a major issue.
+    *   **Vendor Performance/Metrics:** There are multiple ways to get vendor performance data:
+        *   `GET /analytics/vendors` (uses `vendor_metrics`)
+        *   `GET /dashboard/vendor/performance` (uses `purchase_orders`)
+        *   `GET /purchase-orders/vendor-metrics` (uses `purchase_orders`)
+        *   `GET /vendors-aggregate/` (uses `vendor_metrics`, augmented with `purchase_orders`)
+        This suggests significant overlap. The `/vendors-aggregate` endpoint seems the most comprehensive, combining pre-aggregated data with some real-time info. The others, especially `/dashboard/vendor/performance` and `/purchase-orders/vendor-metrics` which calculate directly from `purchase_orders`, might be redundant or less performant.
+    *   **Product Listing:**
+        *   `GET /products/` lists products joining `products`, `product_metrics`, and `categories`.
+        *   `GET /metrics/` lists products primarily from `product_metrics`.
+        They offer similar filtering/sorting. If `product_metrics` contains all necessary display fields, `GET /products/` might be partly redundant for simple listing views, although it does provide aggregated category names. Evaluate if both full list endpoints are necessary.
+    *   **Image Uploads/Management:** Image handling is split:
+        *   `products-import.js`: Uploads temporary images for product import to `/uploads/products/`, schedules deletion.
+        *   `reusable-images.js`: Uploads persistent images to `/uploads/reusable/`, stores metadata in DB.
+        *   `products-import.js` has `/check-file` and `/list-uploads` that can see *both* directories, while `reusable-images.js` has a `/check-file` that only sees its own. This separation could be confusing. Clarify the purpose and lifecycle of images in each directory.
+    *   **Background Task Management (`csv.js`):** The use of `activeImport` for multiple unrelated tasks (import, reset, metrics calc) prevents concurrency, which might be too restrictive. The cancellation logic (`/cancel`) only targets `full-update`/`full-reset`, not tasks locked by `activeImport`. This needs unification.
+    *   **Analytics/Dashboard Base Table Queries:** Several endpoints in `analytics.js` (`/pricing`, `/categories`) and `dashboard.js` (`/best-sellers`, `/sales/metrics`, `/trending/products`, `/key-metrics`, `/inventory-health`, `/sales-overview`) query base tables (`orders`, `products`, `purchase_orders`) directly, while many others leverage pre-aggregated `_metrics` tables. This inconsistency can lead to performance differences and suggests potential for optimization by using aggregates where possible.
+
+3.  **Obvious Mistakes / Data Issues:**
+    *   **AI Prompt Fetching:** `GET /ai-prompts/company/:companyId`, `/type/general`, `/type/system` return `result.rows[0]`. This assumes uniqueness. If the underlying DB constraints (`unique_company_prompt`, etc.) fail or aren't present, this could silently hide data if multiple rows match. The use of unique constraint handling in POST/PUT suggests this is likely intended and safe *if* DB constraints are solid.
+    *   **Mixed Databases & SSH Tunnels:** The heavy reliance in `ai_validation.js` and `products-import.js` on connecting to a production MySQL DB via SSH tunnel while also using a local PostgreSQL DB adds significant architectural complexity.
+        *   **Inefficiency:** In `ai_validation.js` (`generateDebugResponse`), an SSH tunnel and MySQL connection (`promptTunnel`, `promptConnection`) are established but seem unused when fetching prompts (which correctly come from the PG pool `res.app.locals.pool`). This is wasted effort.
+        *   **Improvement:** The `getDbConnection` function in `products-import.js` implements caching/pooling for the SSH/MySQL connection – this is much better and should ideally be used consistently wherever the production DB is accessed (e.g., in `ai_validation.js`).
+    *   **`products.js` Brand Filtering:** `GET /products/brands` filters brands based on having associated purchase orders with a cost >= 500. This seems arbitrary for a general list of brands and might return incomplete results depending on the use case.
+    *   **Type Handling:** Ensure `parseValue` handles all required types and edge cases correctly, especially for filtering complex queries in `*-aggregate` and `metrics` routes. Explicit type casting in SQL (`::numeric`, `::text`, etc.) is generally good practice in PostgreSQL.
+    *   **Dummy Data:** Several `dashboard.js` endpoints return hardcoded dummy data on errors or when no data is found. While this prevents UI crashes, it can mask real issues. Ensure logging is robust when fallbacks are used.
+
+**Summary of Endpoints**
+
+Here's a summary of the available endpoints, grouped by their likely file/module:
+
+**1. AI Prompts (`ai_prompts.js`)**
+    *   `GET /`: Get all AI prompts.
+    *   `GET /:id`: Get a specific AI prompt by its ID.
+    *   `GET /company/:companyId`: Get the AI prompt for a specific company (expects one). **(Deprecated)**
+    *   `GET /type/general`: Get the general AI prompt (expects one). **(Deprecated)**
+    *   `GET /type/system`: Get the system AI prompt (expects one). **(Deprecated)**
+    *   `GET /by-type`: Get AI prompt by type (general, system, company_specific) with optional company parameter. **(New Consolidated Endpoint)**
+    *   `POST /`: Create a new AI prompt.
+    *   `PUT /:id`: Update an existing AI prompt.
+    *   `DELETE /:id`: Delete an AI prompt.
+
+**2. AI Validation (`ai_validation.js`)**
+    *   `POST /debug`: Generate and view the structure of prompts and taxonomy data (for debugging, doesn't call OpenAI). Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance).
+    *   `POST /validate`: Validate product data using OpenAI. Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance).
+    *   `GET /test-taxonomy`: Test endpoint to query sample taxonomy data from Prod MySQL.
+
+**3. Analytics (`analytics.js`)**
+    *   `GET /stats`: Get overall business statistics from metrics tables.
+    *   `GET /profit`: Get profit analysis data (by category, over time, top products) from metrics tables.
+    *   `GET /vendors`: Get vendor performance analysis from `vendor_metrics`.
+    *   `GET /stock`: Get stock analysis data (turnover, levels, critical items) from metrics tables.
+    *   `GET /pricing`: Get pricing analysis (price points, elasticity, recommendations) - **uses `orders` table**.
+    *   `GET /categories`: Get category performance analysis (revenue, profit, growth, distribution, trends) - **uses `orders` and `products` tables**.
+    *   `GET /forecast`: (**Likely Obsolete/Broken**) Attempts to get forecast data using MySQL syntax.
+
+**4. Brands Aggregate (`brands-aggregate.js`)**
+    *   `GET /filter-options`: Get distinct brand names and statuses for UI filters (from `brand_metrics`).
+    *   `GET /stats`: Get overall statistics related to brands (from `brand_metrics`).
+    *   `GET /`: List brands with aggregated metrics, supporting filtering, sorting, pagination (from `brand_metrics`).
+
+**5. Categories Aggregate (`categories-aggregate.js`)**
+    *   `GET /filter-options`: Get distinct category types, statuses, and counts for UI filters (from `category_metrics` & `categories`).
+    *   `GET /stats`: Get overall statistics related to categories (from `category_metrics` & `categories`).
+    *   `GET /`: List categories with aggregated metrics, supporting filtering, sorting (incl. hierarchy), pagination (from `category_metrics` & `categories`).
+
+**6. Configuration (`config.js`)**
+    *   **(New)** `GET /global`: Get all global settings.
+    *   **(New)** `PUT /global`: Update global settings.
+    *   **(New)** `GET /products`: List product-specific settings with pagination/search.
+    *   **(New)** `PUT /products/:pid`: Update/Create product-specific settings.
+    *   **(New)** `POST /products/:pid/reset`: Reset product settings to defaults.
+    *   **(New)** `GET /vendors`: List vendor-specific settings with pagination/search.
+    *   **(New)** `PUT /vendors/:vendor`: Update/Create vendor-specific settings.
+    *   **(New)** `POST /vendors/:vendor/reset`: Reset vendor settings to defaults.
+    *   **(Legacy/Obsolete)** `GET /`: Get all config from old single-row tables.
+    *   **(Legacy/Obsolete)** `PUT /stock-thresholds/:id`: Update old stock thresholds.
+    *   **(Legacy/Obsolete)** `PUT /lead-time-thresholds/:id`: Update old lead time thresholds.
+    *   **(Legacy/Obsolete)** `PUT /sales-velocity/:id`: Update old sales velocity config.
+    *   **(Legacy/Obsolete)** `PUT /abc-classification/:id`: Update old ABC config.
+    *   **(Legacy/Obsolete)** `PUT /safety-stock/:id`: Update old safety stock config.
+    *   **(Legacy/Obsolete)** `PUT /turnover/:id`: Update old turnover config.
+
+**7. CSV Operations & Background Tasks (`csv.js`)**
+    *   `GET /:type/progress`: SSE endpoint for full update/reset progress.
+    *   `GET /test`: Simple test endpoint.
+    *   `GET /status`: Check status of the generic background task lock (`activeImport`).
+    *   `GET /calculate-metrics/status`: Check status of metrics calculation.
+    *   `GET /history/import`: Get recent import history.
+    *   `GET /history/calculate`: Get recent metrics calculation history.
+    *   `GET /status/modules`: Get last calculation time per module.
+    *   `GET /status/tables`: Get last sync time per table.
+    *   `GET /status/table-counts`: Get record counts for key tables.
+    *   `POST /update`: (**Potentially Obsolete**) Trigger `update-csv.js` script.
+    *   `POST /import`: Trigger `import-csv.js` script.
+    *   `POST /cancel`: Cancel `/full-update` or `/full-reset` task.
+    *   `POST /reset`: (**Potentially Obsolete**) Trigger `reset-db.js` script.
+    *   `POST /reset-metrics`: Trigger `reset-metrics.js` script.
+    *   `POST /calculate-metrics`: Trigger `calculate-metrics.js` script.
+    *   `POST /import-from-prod`: Trigger `import-from-prod.js` script.
+    *   `POST /full-update`: Trigger `full-update.js` script (preferred update).
+    *   `POST /full-reset`: Trigger `full-reset.js` script (preferred reset).
+
+**8. Dashboard (`dashboard.js`)**
+    *   `GET /stock/metrics`: Get dashboard stock summary metrics & brand breakdown.
+    *   `GET /purchase/metrics`: Get dashboard purchase order summary metrics & vendor breakdown.
+    *   `GET /replenishment/metrics`: Get dashboard replenishment summary & top variants.
+    *   `GET /forecast/metrics`: Get dashboard forecast summary, daily, and category breakdown.
+    *   `GET /overstock/metrics`: Get dashboard overstock summary & category breakdown.
+    *   `GET /overstock/products`: Get list of top overstocked products.
+    *   `GET /best-sellers`: Get dashboard best-selling products, brands, categories - **uses `orders`, `products`**.
+    *   `GET /sales/metrics`: Get dashboard sales summary for a period - **uses `orders`**.
+    *   `GET /low-stock/products`: Get list of top low stock/critical products.
+    *   `GET /trending/products`: Get list of trending products - **uses `orders`, `products`**.
+    *   `GET /vendor/performance`: Get dashboard vendor performance details - **uses `purchase_orders`**.
+    *   `GET /key-metrics`: Get dashboard summary KPIs - **uses multiple base tables**.
+    *   `GET /inventory-health`: Get dashboard inventory health overview - **uses `products`, `product_metrics`**.
+    *   `GET /replenish/products`: Get list of products needing replenishment (overlaps `/low-stock/products`).
+    *   `GET /sales-overview`: Get daily sales totals for chart - **uses `orders`**.
+
+**9. Product Import Utilities (`products-import.js`)**
+    *   `POST /upload-image`: Upload temporary product image, schedule deletion.
+    *   `DELETE /delete-image`: Delete temporary product image.
+    *   `GET /field-options`: Get dropdown options for product fields from Prod MySQL (cached).
+    *   `GET /product-lines/:companyId`: Get product lines for a company from Prod MySQL (cached).
+    *   `GET /sublines/:lineId`: Get sublines for a line from Prod MySQL (cached).
+    *   `GET /check-file/:filename`: Check existence/permissions of uploaded file (temp or reusable).
+    *   `GET /list-uploads`: List files in upload directories.
+    *   `GET /search-products`: Search products in Prod MySQL DB.
+    *   `GET /check-upc-and-generate-sku`: Check UPC existence and generate SKU suggestion based on Prod MySQL data.
+    *   `GET /product-categories/:pid`: Get assigned categories for a product from Prod MySQL.
+
+**10. Product Metrics (`product-metrics.js`)**
+    *   `GET /filter-options`: Get distinct filter values (vendor, brand, abcClass) from `product_metrics`.
+    *   `GET /`: List detailed product metrics with filtering, sorting, pagination (primary data access).
+    *   `GET /:pid`: Get full metrics record for a single product.
+
+**11. Orders (`orders.js`)**
+    *   `GET /`: List orders with summary info, filtering, sorting, pagination, and stats.
+    *   `GET /:orderNumber`: Get details for a single order, including items.
+
+**12. Products (`products.js`)**
+    *   `GET /brands`: Get distinct brands (filtered by PO value).
+    *   `GET /`: List products with core data + metrics, filtering, sorting, pagination.
+    *   `GET /trending`: Get trending products based on `product_metrics`.
+    *   `GET /:id`: Get details for a single product (core data + metrics).
+    *   `POST /import`: (**Likely Obsolete/Dangerous**) Import products from CSV.
+    *   `PUT /:id`: Update core product data.
+    *   `GET /:id/metrics`: (**Redundant**) Get metrics for a single product.
+    *   `GET /:id/time-series`: Get sales/PO history for a single product.
+
+**13. Purchase Orders (`purchase-orders.js`)**
+    *   `GET /`: List purchase orders with summary info, filtering, sorting, pagination, and summary stats.
+    *   `GET /vendor-metrics`: Calculate vendor performance metrics from `purchase_orders`.
+    *   `GET /cost-analysis`: Calculate cost analysis by category from `purchase_orders`.
+    *   `GET /receiving-status`: Get summary counts based on PO receiving status.
+    *   `GET /order-vs-received`: List product ordered vs. received quantities.
+
+**14. Reusable Images (`reusable-images.js`)**
+    *   `GET /`: List all reusable images.
+    *   `GET /by-company/:companyId`: List global and company-specific images.
+    *   `GET /global`: List only global images.
+    *   `GET /:id`: Get a single reusable image record.
+    *   `POST /upload`: Upload a new reusable image and create DB record.
+    *   `PUT /:id`: Update reusable image metadata (name, global, company).
+    *   `DELETE /:id`: Delete reusable image record and file.
+    *   `GET /check-file/:filename`: Check existence/permissions of a reusable image file.
+
+**15. Templates (`templates.js`)**
+    *   `GET /`: List all product data templates.
+    *   `GET /:company/:productType`: Get a specific template.
+    *   `POST /`: Create a new template.
+    *   `PUT /:id`: Update an existing template.
+    *   `DELETE /:id`: Delete a template.
+
+**16. Vendors Aggregate (`vendors-aggregate.js`)**
+    *   `GET /filter-options`: Get distinct vendor names and statuses for UI filters (from `vendor_metrics`).
+    *   `GET /stats`: Get overall statistics related to vendors (from `vendor_metrics` & `purchase_orders`).
+    *   `GET /`: List vendors with aggregated metrics, supporting filtering, sorting, pagination (from `vendor_metrics` & `purchase_orders`).
+
+**Recommendations:**
+
+1.  **Address Obsolete Endpoints:** Prioritize removing or confirming the necessity of the endpoints marked as obsolete/redundant (legacy config, `/analytics/forecast`, `/csv/update`, `/csv/reset`, `/products/import`, `/products/:id/metrics`).
+2.  **Consolidate Overlapping Functionality:** Review the multiple vendor performance and product listing endpoints. Decide on the primary method (e.g., using aggregate tables via `/vendors-aggregate` and `/metrics`) and refactor or remove the others. Clarify the image upload strategies.
+3.  **Standardize Data Access:** Decide whether `dashboard` and `analytics` endpoints should primarily use aggregate tables (like `/metrics`, `/brands-aggregate`, etc.) or if direct access to base tables is sometimes necessary. Aim for consistency and document the reasoning. Optimize queries hitting base tables if they must remain.
+4.  **Improve Background Task Management:** Refactor `csv.js` to use a unified locking mechanism (maybe separate locks per task type?) and a consistent cancellation strategy for all spawned/managed processes. Clarify the purpose of `update` vs `full-update` and `reset` vs `full-reset`.
+5.  **Optimize DB Connections:** Ensure the `getDbConnection` pooling/caching helper from `products-import.js` is used *consistently* across all modules interacting with the production MySQL database (especially `ai_validation.js`). Remove unnecessary tunnel creations.
+6.  **Review Data Integrity:** Double-check the assumptions made (e.g., uniqueness of AI prompts) and ensure database constraints enforce them. Review the `GET /products/brands` filtering logic.
+
+## Changes Made
+
+1. **Removed Obsolete Legacy Endpoints in `config.js`**:
+   - Removed `GET /config/` endpoint
+   - Removed `PUT /config/stock-thresholds/:id` endpoint
+   - Removed `PUT /config/lead-time-thresholds/:id` endpoint
+   - Removed `PUT /config/sales-velocity/:id` endpoint
+   - Removed `PUT /config/abc-classification/:id` endpoint
+   - Removed `PUT /config/safety-stock/:id` endpoint
+   - Removed `PUT /config/turnover/:id` endpoint
+   
+   These endpoints were obsolete as they referenced older, single-row config tables that have been replaced by newer endpoints using the structured tables `settings_global`, `settings_product`, and `settings_vendor`.
+
+2. **Removed MySQL Syntax `/forecast` Endpoint in `analytics.js`**:
+   - Removed `GET /analytics/forecast` endpoint that was using MySQL-specific syntax incompatible with the PostgreSQL database used elsewhere in the application.
+
+3. **Renamed and Removed Redundant Endpoints**:
+   - Renamed `csv.js` to `data-management.js` while maintaining the same `/csv/*` endpoint paths for consistency
+   - Removed deprecated `/csv/update` endpoint (now fully replaced by `/csv/full-update`)
+   - Removed deprecated `/csv/reset` endpoint (now fully replaced by `/csv/full-reset`)
+   - Removed deprecated `/products/import` endpoint (now handled by `/csv/import`)
+   - Removed deprecated `/products/:id/metrics` endpoint (now handled by `/metrics/:pid`)
+
+4. **Fixed Data Integrity Issues**:
+   - Improved `GET /products/brands` endpoint by removing the arbitrary filtering logic that was only showing brands with purchase orders that had a total cost of at least $500
+   - The updated endpoint now returns all distinct brands from visible products, providing more complete data
+
+5. **Optimized Database Connections**:
+   - Created a new `dbConnection.js` utility file that encapsulates the optimized database connection management logic
+   - Improved the `ai-validation.js` file to use this shared connection management, eliminating unnecessary repeated tunnel creation
+   - Added proper connection pooling with timeout-based connection reuse, reducing the overhead of repeatedly creating SSH tunnels
+   - Added query result caching for frequently accessed data to improve performance
+
+These changes improve maintainability by removing duplicate code, enhance consistency by standardizing on the newer endpoint patterns, and optimize performance by reducing redundant database connections.
+
+## Additional Improvements
+
+1. **Further Database Connection Optimizations**:
+   - Extended the use of the optimized database connection utility to additional endpoints in `ai-validation.js`
+   - Updated the `/validate` endpoint and `/test-taxonomy` endpoint to use `getDbConnection`
+   - Ensured consistent connection management across all routes that access the production database
+
+2. **AI Prompts Data Integrity Verification**:
+   - Confirmed proper uniqueness constraints are in place in the database schema for AI prompts
+   - The schema includes:
+     - `unique_company_prompt` constraint ensuring only one prompt per company
+     - `idx_unique_general_prompt` index ensuring only one general prompt in the system
+     - `idx_unique_system_prompt` index ensuring only one system prompt in the system
+   - Endpoint handlers properly handle uniqueness constraint violations with appropriate 409 Conflict responses
+   - Validation ensures company-specific prompts have company IDs, while general/system prompts do not
+
+3. **AI Prompts Endpoint Consolidation**:
+   - Added a new consolidated `/by-type` endpoint that handles all types of prompts (general, system, company_specific)
+   - Marked the existing separate endpoints as deprecated with console warnings
+   - Maintained backward compatibility while providing a cleaner API moving forward
+
+## Completed Items
+
+✅ Removed obsolete legacy endpoints in `config.js`  
+✅ Removed MySQL syntax `/forecast` endpoint in `analytics.js`  
+✅ Fixed `GET /products/brands` endpoint filtering logic  
+✅ Created reusable database connection utility (`dbConnection.js`)  
+✅ Optimized database connections in `ai-validation.js`  
+✅ Verified data integrity in AI prompts handling  
+✅ Consolidated AI prompts endpoints with a unified `/by-type` endpoint  
+
+## Remaining Items
+
+- Consider adding additional error handling and logging for database connections
+- Perform load testing on the optimized database connections to ensure they handle high traffic properly

docs/setup-chat.md

@@ -0,0 +1,23 @@
+This portion of the application is going to be a read only chat archive. It will pull data from a rocketchat export converted to postgresql. This is a separate database than the rest of the inventory application uses, but it will still use users and permissions from the inventory database. Both databases are on the same postgres instance.
+
+For now, let's add a select to the top of the page that allows me to "view as" any of the users in the rocketchat database. We'll connect this to the authorization in the main application later. 
+
+The db connection info is stored in the .env file in the inventory-server root. It contains these variables
+DB_HOST=localhost
+DB_USER=rocketchat_user
+DB_PASSWORD=password
+DB_NAME=rocketchat_converted
+DB_PORT=5432
+
+Not all of the information in this database is relevant as it's a direct export from another app with more features. You can use the query tool to examine the structure and data available.
+
+Server-side files should use similar conventions and the same technologies as the inventory-server (inventor-server root) and auth-server (inventory-server/auth). I will provide my current pm2 ecosystem file upon request for you to add the configuration for the new "chat-server". I use Caddy on the server and can provide my caddyfile to assist with configuring the api routes. All configuration and routes for the chat-server should go in the inventory-server/chat folder or subfolders you create.
+
+The folder you see as inventory-server is actually a direct mount of the /var/www/inventory folder on the server. You can read and write files from there like usual, but any terminal commands for the server I will have to run myself.
+
+The "Chat" page should be added to the main application sidebar and a similar page to the others should be created in inventory/src/pages. All other frontend pages should go in inventory/src/components/chat. 
+
+The application uses shadcn components and those should be used for all ui elements where possible (located in inventory/src/components/ui). The UI should match existing pages and components. 
+
+
+

docs/split-up-pos.md

@@ -0,0 +1,112 @@
+Okay, I understand completely now. The core issue is that the previous approaches tried too hard to reconcile every receipt back to a specific PO line within the `purchase_orders` table structure, which doesn't reflect the reality where receipts can be independent events. Your downstream scripts, especially `daily_snapshots` and `product_metrics`, rely on having a complete picture of *all* receivings.
+
+Let's pivot to a model that respects both distinct data streams: **Orders (Intent)** and **Receivings (Actuals)**.
+
+**Proposed Solution: Separate `purchase_orders` and `receivings` Tables**
+
+This is the cleanest way to model the reality you've described.
+
+1.  **`purchase_orders` Table:**
+    *   **Purpose:** Tracks the status and details of purchase *orders* placed. Represents the *intent* to receive goods.
+    *   **Key Columns:** `po_id`, `pid`, `ordered` (quantity ordered), `po_cost_price`, `date` (order/created date), `expected_date`, `status` (PO lifecycle: 'ordered', 'canceled', 'done'), `vendor`, `notes`, etc.
+    *   **Crucially:** This table *does not* need a `received` column or a `receiving_history` column derived from complex allocations. It focuses solely on the PO itself.
+
+2.  **`receivings` Table (New or Refined):**
+    *   **Purpose:** Tracks every single line item received, regardless of whether it was linked to a PO during the receiving process. Represents the *actual* goods that arrived.
+    *   **Key Columns:**
+        *   `receiving_id` (Identifier for the overall receiving document/batch)
+        *   `pid` (Product ID received)
+        *   `received_qty` (Quantity received for this specific line)
+        *   `cost_each` (Actual cost paid for this item on this receiving)
+        *   `received_date` (Actual date the item was received)
+        *   `received_by` (Employee ID/Name)
+        *   `source_po_id` (The `po_id` entered on the receiving screen, *nullable*. Stores the original link attempt, even if it was wrong or missing)
+        *   `source_receiving_status` (The status from the source `receivings` table: 'partial_received', 'full_received', 'paid', 'canceled')
+
+**How the Import Script Changes:**
+
+1.  **Fetch POs:** Fetch data from `po` and `po_products`.
+2.  **Populate `purchase_orders`:**
+    *   Insert/Update rows into `purchase_orders` based directly on the fetched PO data.
+    *   Set `po_id`, `pid`, `ordered`, `po_cost_price`, `date` (`COALESCE(date_ordered, date_created)`), `expected_date`.
+    *   Set `status` by mapping the source `po.status` code directly ('ordered', 'canceled', 'done', etc.).
+    *   **No complex allocation needed here.**
+3.  **Fetch Receivings:** Fetch data from `receivings` and `receivings_products`.
+4.  **Populate `receivings`:**
+    *   For *every* line item fetched from `receivings_products`:
+        *   Perform necessary data validation (dates, numbers).
+        *   Insert a new row into `receivings` with all the relevant details (`receiving_id`, `pid`, `received_qty`, `cost_each`, `received_date`, `received_by`, `source_po_id`, `source_receiving_status`).
+    *   Use `ON CONFLICT (receiving_id, pid)` (or similar unique key based on your source data) `DO UPDATE SET ...` for incremental updates if necessary, or simply delete/re-insert based on `receiving_id` for simplicity if performance allows.
+
+**Impact on Downstream Scripts (and how to adapt):**
+
+*   **Initial Query (Active POs):**
+    *   `SELECT ... FROM purchase_orders po WHERE po.status NOT IN ('canceled', 'done', 'paid_equivalent_status?') AND po.date >= ...`
+    *   `active_pos`: `COUNT(DISTINCT po.po_id)` based on the filtered POs.
+    *   `overdue_pos`: Add `AND po.expected_date < CURRENT_DATE`.
+    *   `total_units`: `SUM(po.ordered)`. Represents total units *ordered* on active POs.
+    *   `total_cost`: `SUM(po.ordered * po.po_cost_price)`. Cost of units *ordered*.
+    *   `total_retail`: `SUM(po.ordered * pm.current_price)`. Retail value of units *ordered*.
+    *   **Result:** This query now cleanly reports on the status of *orders* placed, which seems closer to its original intent. The filter `po.receiving_status NOT IN ('partial_received', 'full_received', 'paid')` is replaced by `po.status NOT IN ('canceled', 'done', 'paid_equivalent?')`. The 90% received check is removed as `received` is not reliably tracked *on the PO* anymore.
+
+*   **`daily_product_snapshots`:**
+    *   **`SalesData` CTE:** No change needed.
+    *   **`ReceivingData` CTE:** **Must be changed.** Query the **`receivings`** table instead of `purchase_orders`.
+        ```sql
+        ReceivingData AS (
+            SELECT
+                rl.pid,
+                COUNT(DISTINCT rl.receiving_id) as receiving_doc_count, 
+                SUM(rl.received_qty) AS units_received,
+                SUM(rl.received_qty * rl.cost_each) AS cost_received
+            FROM public.receivings rl
+            WHERE rl.received_date::date = _date
+              -- Optional: Filter out canceled receivings if needed
+              -- AND rl.source_receiving_status <> 'canceled' 
+            GROUP BY rl.pid
+        ),
+        ```
+    *   **Result:** This now accurately reflects *all* units received on a given day from the definitive source.
+
+*   **`update_product_metrics`:**
+    *   **`CurrentInfo` CTE:** No change needed (pulls from `products`).
+    *   **`OnOrderInfo` CTE:** Needs re-evaluation. How do you want to define "On Order"?
+        *   **Option A (Strict PO View):** `SUM(po.ordered)` from `purchase_orders po WHERE po.status NOT IN ('canceled', 'done', 'paid_equivalent?')`. This is quantity on *open orders*, ignoring fulfillment state. Simple, but might overestimate if items arrived unlinked.
+        *   **Option B (Approximate Fulfillment):** `SUM(po.ordered)` from open POs MINUS `SUM(rl.received_qty)` from `receivings rl` where `rl.source_po_id = po.po_id` (summing only directly linked receivings). Better, but still misses fulfillment via unlinked receivings.
+        *   **Option C (Heuristic):** `SUM(po.ordered)` from open POs MINUS `SUM(rl.received_qty)` from `receivings rl` where `rl.pid = po.pid` and `rl.received_date >= po.date`. This *tries* to account for unlinked receivings but is imprecise.
+        *   **Recommendation:** Start with **Option A** for simplicity, clearly labeling it "Quantity on Open POs". You might need a separate process or metric for a more nuanced view of expected vs. actual pipeline.
+        ```sql
+         -- Example for Option A
+         OnOrderInfo AS (
+             SELECT
+                 pid,
+                 SUM(ordered) AS on_order_qty, -- Total qty on open POs
+                 SUM(ordered * po_cost_price) AS on_order_cost -- Cost of qty on open POs
+             FROM public.purchase_orders
+             WHERE status NOT IN ('canceled', 'done', 'paid_equivalent?') -- Define your open statuses
+             GROUP BY pid
+         ),
+        ```
+    *   **`HistoricalDates` CTE:**
+        *   `date_first_sold`, `max_order_date`: No change (queries `orders`).
+        *   `date_first_received_calc`, `date_last_received_calc`: **Must be changed.** Query `MIN(rl.received_date)` and `MAX(rl.received_date)` from the **`receivings`** table grouped by `pid`.
+    *   **`SnapshotAggregates` CTE:**
+        *   `received_qty_30d`, `received_cost_30d`: These are calculated from `daily_product_snapshots`, which are now correctly sourced from `receivings`, so this part is fine.
+    *   **Forecasting Calculations:** Will use the chosen definition of `on_order_qty`. Be aware of the implications of Option A (potentially inflated if unlinked receivings fulfill orders).
+    *   **Result:** Metrics are calculated based on distinct order data and complete receiving data. The definition of "on order" needs careful consideration.
+
+**Summary of this Approach:**
+
+*   **Pros:**
+    *   Accurately models distinct order and receiving events.
+    *   Provides a definitive source (`receivings`) for all received inventory.
+    *   Simplifies the `purchase_orders` table and its import logic.
+    *   Avoids complex/potentially inaccurate allocation logic for unlinked receivings within the main tables.
+    *   Avoids synthetic records.
+    *   Fixes downstream reporting (`daily_snapshots` receiving data).
+*   **Cons:**
+    *   Requires creating/managing the `receivings` table.
+    *   Requires modifying downstream queries (`ReceivingData`, `OnOrderInfo`, `HistoricalDates`).
+    *   Calculating a precise "net quantity still expected to arrive" (true on-order minus all relevant fulfillment) becomes more complex and may require specific business rules or heuristics outside the basic table structure if Option A for `OnOrderInfo` isn't sufficient.
+
+This two-table approach (`purchase_orders` + `receivings`) seems the most robust and accurate way to handle your requirement for complete receiving records independent of potentially flawed PO linking. It directly addresses the shortcomings of the previous attempts.

ecosystem.config.js

@@ -1,222 +0,0 @@
-// ecosystem.config.js
-const path = require('path');
-const dotenv = require('dotenv');
-
-// Load environment variables safely with error handling
-const loadEnvFile = (envPath) => {
-  try {
-    console.log('Loading env from:', envPath);
-    const result = dotenv.config({ path: envPath });
-    if (result.error) {
-      console.warn(`Warning: .env file not found or invalid at ${envPath}:`, result.error.message);
-      return {};
-    }
-    console.log('Env variables loaded from', envPath, ':', Object.keys(result.parsed || {}));
-    return result.parsed || {};
-  } catch (error) {
-    console.warn(`Warning: Error loading .env file at ${envPath}:`, error.message);
-    return {};
-  }
-}
-
-// Load environment variables for each server
-const authEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/auth-server/.env'));
-const aircallEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/aircall-server/.env'));
-const klaviyoEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/klaviyo-server/.env'));
-const metaEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/meta-server/.env'));
-const googleAnalyticsEnv = require('dotenv').config({ 
-  path: path.resolve(__dirname, 'dashboard/google-server/.env') 
-}).parsed || {};
-const typeformEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/typeform-server/.env'));
-const inventoryEnv = loadEnvFile(path.resolve(__dirname, 'inventory/.env'));
-
-// Common log settings for all apps
-const logSettings = {
-  log_rotate: true,
-  max_size: '10M',
-  retain: '10',
-  log_date_format: 'YYYY-MM-DD HH:mm:ss'
-};
-
-// Common app settings
-const commonSettings = {
-  instances: 1,
-  exec_mode: 'fork',
-  autorestart: true,
-  watch: false,
-  max_memory_restart: '1G',
-  time: true,
-  ...logSettings,
-  ignore_watch: [
-    'node_modules',
-    'logs',
-    '.git',
-    '*.log'
-  ],
-  min_uptime: 5000,
-  max_restarts: 5,
-  restart_delay: 4000,
-  listen_timeout: 50000,
-  kill_timeout: 5000,
-  node_args: '--max-old-space-size=1536'
-};
-
-module.exports = {
-  apps: [
-    {
-      ...commonSettings,
-      name: 'auth-server',
-      script: './dashboard/auth-server/index.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3003,
-        ...authEnv
-      },
-      error_file: 'dashboard/auth-server/logs/pm2/err.log',
-      out_file: 'dashboard/auth-server/logs/pm2/out.log',
-      log_file: 'dashboard/auth-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3003
-      },
-      env_development: {
-        NODE_ENV: 'development',
-        PORT: 3003
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'aircall-server',
-      script: './dashboard/aircall-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        AIRCALL_PORT: 3002,
-        ...aircallEnv
-      },
-      error_file: 'dashboard/aircall-server/logs/pm2/err.log',
-      out_file: 'dashboard/aircall-server/logs/pm2/out.log',
-      log_file: 'dashboard/aircall-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        AIRCALL_PORT: 3002
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'klaviyo-server',
-      script: './dashboard/klaviyo-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        KLAVIYO_PORT: 3004,
-        ...klaviyoEnv
-      },
-      error_file: 'dashboard/klaviyo-server/logs/pm2/err.log',
-      out_file: 'dashboard/klaviyo-server/logs/pm2/out.log',
-      log_file: 'dashboard/klaviyo-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        KLAVIYO_PORT: 3004
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'meta-server',
-      script: './dashboard/meta-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3005,
-        ...metaEnv
-      },
-      error_file: 'dashboard/meta-server/logs/pm2/err.log',
-      out_file: 'dashboard/meta-server/logs/pm2/out.log',
-      log_file: 'dashboard/meta-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3005
-      }
-    },
-    {
-      name: "gorgias-server",
-      script: "./dashboard/gorgias-server/server.js",
-      env: {
-        NODE_ENV: "development",
-        PORT: 3006
-      },
-      env_production: {
-        NODE_ENV: "production",
-        PORT: 3006
-      },
-      error_file: "dashboard/logs/gorgias-server-error.log",
-      out_file: "dashboard/logs/gorgias-server-out.log",
-      log_file: "dashboard/logs/gorgias-server-combined.log",
-      time: true
-    },
-    {
-      ...commonSettings,
-      name: 'google-server',
-      script: path.resolve(__dirname, 'dashboard/google-server/server.js'),
-      watch: false,
-      env: {
-        NODE_ENV: 'production',
-        GOOGLE_ANALYTICS_PORT: 3007,
-        ...googleAnalyticsEnv
-      },
-      error_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/err.log'),
-      out_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/out.log'),
-      log_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/combined.log'),
-      env_production: {
-        NODE_ENV: 'production',
-        GOOGLE_ANALYTICS_PORT: 3007
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'typeform-server',
-      script: './dashboard/typeform-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        TYPEFORM_PORT: 3008,
-        ...typeformEnv
-      },
-      error_file: 'dashboard/typeform-server/logs/pm2/err.log',
-      out_file: 'dashboard/typeform-server/logs/pm2/out.log',
-      log_file: 'dashboard/typeform-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        TYPEFORM_PORT: 3008
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'inventory-server',
-      script: './inventory/src/server.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3010,
-        ...inventoryEnv
-      },
-      error_file: 'inventory/logs/pm2/err.log',
-      out_file: 'inventory/logs/pm2/out.log',
-      log_file: 'inventory/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3010,
-        ...inventoryEnv
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'new-auth-server',
-      script: './inventory-server/auth/server.js',
-      env: {
-        NODE_ENV: 'production',
-        AUTH_PORT: 3011,
-        ...inventoryEnv,
-        JWT_SECRET: process.env.JWT_SECRET
-      },
-      error_file: 'inventory-server/auth/logs/pm2/err.log',
-      out_file: 'inventory-server/auth/logs/pm2/out.log',
-      log_file: 'inventory-server/auth/logs/pm2/combined.log'
-    }
-  ]
-};

inventory-server/auth/add-user.js

@@ -1,103 +1,75 @@
-require('dotenv').config({ path: '../.env' });
+import bcrypt from 'bcrypt';
-const bcrypt = require('bcrypt');
+import pg from 'pg';
-const { Pool } = require('pg');
+import inquirer from 'inquirer';
-const inquirer = require('inquirer');
 
-// Log connection details for debugging (remove in production)
+const { Pool } = pg;
-console.log('Attempting to connect with:', {
+import { config as loadEnv } from 'dotenv';
-  host: process.env.DB_HOST,
+import { fileURLToPath } from 'node:url';
-  user: process.env.DB_USER,
+import { dirname, resolve as resolvePath } from 'node:path';
-  database: process.env.DB_NAME,
+
-  port: process.env.DB_PORT
+const __filename = fileURLToPath(import.meta.url);
-});
+const __dirname = dirname(__filename);
+
+loadEnv({ path: resolvePath(__dirname, '../.env') });
 
 const pool = new Pool({
   host: process.env.DB_HOST,
   user: process.env.DB_USER,
   password: process.env.DB_PASSWORD,
   database: process.env.DB_NAME,
-  port: process.env.DB_PORT,
+  port: Number(process.env.DB_PORT) || 5432,
 });
 
 async function promptUser() {
-  const questions = [
+  return inquirer.prompt([
     {
       type: 'input',
       name: 'username',
       message: 'Enter username:',
-      validate: (input) => {
+      validate: (input) => input.length >= 3 || 'Username must be at least 3 characters long',
-        if (input.length < 3) {
-          return 'Username must be at least 3 characters long';
-        }
-        return true;
-      }
     },
     {
       type: 'password',
       name: 'password',
       message: 'Enter password:',
       mask: '*',
-      validate: (input) => {
+      validate: (input) => input.length >= 8 || 'Password must be at least 8 characters long',
-        if (input.length < 8) {
-          return 'Password must be at least 8 characters long';
-        }
-        return true;
-      }
     },
     {
       type: 'password',
       name: 'confirmPassword',
       message: 'Confirm password:',
       mask: '*',
-      validate: (input, answers) => {
+      validate: (input, answers) => input === answers.password || 'Passwords do not match',
-        if (input !== answers.password) {
+    },
-          return 'Passwords do not match';
+  ]);
-        }
-        return true;
-      }
-    }
-  ];
-
-  return inquirer.prompt(questions);
 }
 
 async function addUser() {
   try {
-    // Get user input
+    const { username, password } = await promptUser();
-    const answers = await promptUser();
+    const hashedPassword = await bcrypt.hash(password, 10);
-    const { username, password } = answers;
 
-    // Hash password
-    const saltRounds = 10;
-    const hashedPassword = await bcrypt.hash(password, saltRounds);
-
-    // Check if user already exists
     const checkResult = await pool.query(
       'SELECT id FROM users WHERE username = $1',
       [username]
     );
-
     if (checkResult.rows.length > 0) {
       console.error('Error: Username already exists');
       process.exit(1);
     }
 
-    // Insert new user
     const result = await pool.query(
       'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id',
       [username, hashedPassword]
     );
-
     console.log(`User ${username} created successfully with id ${result.rows[0].id}`);
   } catch (error) {
     console.error('Error creating user:', error);
-    console.error('Error details:', error.message);
+    if (error.code) console.error('Error code:', error.code);
-    if (error.code) {
-      console.error('Error code:', error.code);
-    }
   } finally {
     await pool.end();
   }
 }
 
-addUser(); 
+addUser();

inventory-server/auth/package-lock.json

@@ -18,6 +18,43 @@
         "pg": "^8.11.3"
       }
     },
+    "node_modules/@inquirer/external-editor": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/@inquirer/external-editor/-/external-editor-1.0.2.tgz",
+      "integrity": "sha512-yy9cOoBnx58TlsPrIxauKIFQTiyH+0MK4e97y4sV9ERbI+zDxw7i2hxHLCIEGIE/8PPvDxGhgzIOTSOWcs6/MQ==",
+      "license": "MIT",
+      "dependencies": {
+        "chardet": "^2.1.0",
+        "iconv-lite": "^0.7.0"
+      },
+      "engines": {
+        "node": ">=18"
+      },
+      "peerDependencies": {
+        "@types/node": ">=18"
+      },
+      "peerDependenciesMeta": {
+        "@types/node": {
+          "optional": true
+        }
+      }
+    },
+    "node_modules/@inquirer/external-editor/node_modules/iconv-lite": {
+      "version": "0.7.0",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.7.0.tgz",
+      "integrity": "sha512-cf6L2Ds3h57VVmkZe+Pn+5APsT7FpqJtEhhieDCvrE2MK5Qk9MyffgQyuxQTm6BChfeZNtcOLHp9IcWRVcIcBQ==",
+      "license": "MIT",
+      "dependencies": {
+        "safer-buffer": ">= 2.1.2 < 3.0.0"
+      },
+      "engines": {
+        "node": ">=0.10.0"
+      },
+      "funding": {
+        "type": "opencollective",
+        "url": "https://opencollective.com/express"
+      }
+    },
     "node_modules/@mapbox/node-pre-gyp": {
       "version": "1.0.11",
       "resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz",
@@ -251,9 +288,9 @@
       }
     },
     "node_modules/brace-expansion": {
-      "version": "1.1.11",
+      "version": "1.1.12",
-      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
+      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.12.tgz",
-      "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==",
+      "integrity": "sha512-9T9UjW3r0UW5c1Q7GTwllptXwhvYmEzFhzMfZ9H7FQWt+uZePjZPjBP/W1ZEyZ1twGWom5/56TF4lPcqjnDHcg==",
       "license": "MIT",
       "dependencies": {
         "balanced-match": "^1.0.0",
@@ -345,9 +382,9 @@
       }
     },
     "node_modules/chardet": {
-      "version": "0.7.0",
+      "version": "2.1.0",
-      "resolved": "https://registry.npmjs.org/chardet/-/chardet-0.7.0.tgz",
+      "resolved": "https://registry.npmjs.org/chardet/-/chardet-2.1.0.tgz",
-      "integrity": "sha512-mT8iDcrh03qDGRRmoA2hmBJnxpllMR+0/0qlzjqZES6NdiWDcZkCNAk4rPFZ9Q85r27unkiNNg8ZOiwZXBHwcA==",
+      "integrity": "sha512-bNFETTG/pM5ryzQ9Ad0lJOTa6HWD/YsScAR3EnCPZRPlQh77JocYktSHOUHelyhm8IARL+o4c4F1bP5KVOjiRA==",
       "license": "MIT"
     },
     "node_modules/chownr": {
@@ -700,20 +737,6 @@
         "url": "https://opencollective.com/express"
       }
     },
-    "node_modules/external-editor": {
-      "version": "3.1.0",
-      "resolved": "https://registry.npmjs.org/external-editor/-/external-editor-3.1.0.tgz",
-      "integrity": "sha512-hMQ4CX1p1izmuLYyZqLMO/qGNw10wSv9QDCPfzXfyFrOaCSSoRfqE1Kf1s5an66J5JZC62NewG+mK49jOCtQew==",
-      "license": "MIT",
-      "dependencies": {
-        "chardet": "^0.7.0",
-        "iconv-lite": "^0.4.24",
-        "tmp": "^0.0.33"
-      },
-      "engines": {
-        "node": ">=4"
-      }
-    },
     "node_modules/figures": {
       "version": "3.2.0",
       "resolved": "https://registry.npmjs.org/figures/-/figures-3.2.0.tgz",
@@ -1036,16 +1059,16 @@
       "license": "ISC"
     },
     "node_modules/inquirer": {
-      "version": "8.2.6",
+      "version": "8.2.7",
-      "resolved": "https://registry.npmjs.org/inquirer/-/inquirer-8.2.6.tgz",
+      "resolved": "https://registry.npmjs.org/inquirer/-/inquirer-8.2.7.tgz",
-      "integrity": "sha512-M1WuAmb7pn9zdFRtQYk26ZBoY043Sse0wVDdk4Bppr+JOXyQYybdtvK+l9wUibhtjdjvtoiNy8tk+EgsYIUqKg==",
+      "integrity": "sha512-UjOaSel/iddGZJ5xP/Eixh6dY1XghiBw4XK13rCCIJcJfyhhoul/7KhLLUGtebEj6GDYM6Vnx/mVsjx2L/mFIA==",
       "license": "MIT",
       "dependencies": {
+        "@inquirer/external-editor": "^1.0.0",
         "ansi-escapes": "^4.2.1",
         "chalk": "^4.1.1",
         "cli-cursor": "^3.1.0",
         "cli-width": "^3.0.0",
-        "external-editor": "^3.0.3",
         "figures": "^3.0.0",
         "lodash": "^4.17.21",
         "mute-stream": "0.0.8",
@@ -1374,16 +1397,16 @@
       }
     },
     "node_modules/morgan": {
-      "version": "1.10.0",
+      "version": "1.10.1",
-      "resolved": "https://registry.npmjs.org/morgan/-/morgan-1.10.0.tgz",
+      "resolved": "https://registry.npmjs.org/morgan/-/morgan-1.10.1.tgz",
-      "integrity": "sha512-AbegBVI4sh6El+1gNwvD5YIck7nSA36weD7xvIxG4in80j/UoK8AEGaWnnz8v1GxonMCltmlNs5ZKbGvl9b1XQ==",
+      "integrity": "sha512-223dMRJtI/l25dJKWpgij2cMtywuG/WiUKXdvwfbhGKBhy1puASqXwFzmWZ7+K73vUPoR7SS2Qz2cI/g9MKw0A==",
       "license": "MIT",
       "dependencies": {
         "basic-auth": "~2.0.1",
         "debug": "2.6.9",
         "depd": "~2.0.0",
         "on-finished": "~2.3.0",
-        "on-headers": "~1.0.2"
+        "on-headers": "~1.1.0"
       },
       "engines": {
         "node": ">= 0.8.0"
@@ -1510,9 +1533,9 @@
       }
     },
     "node_modules/on-headers": {
-      "version": "1.0.2",
+      "version": "1.1.0",
-      "resolved": "https://registry.npmjs.org/on-headers/-/on-headers-1.0.2.tgz",
+      "resolved": "https://registry.npmjs.org/on-headers/-/on-headers-1.1.0.tgz",
-      "integrity": "sha512-pZAE+FJLoyITytdqK0U5s+FIpjN0JP3OzFi/u8Rx+EV5/W+JTWGXG8xFzevE7AjBfDqHv/8vL8qQsIhHnqRkrA==",
+      "integrity": "sha512-737ZY3yNnXy37FHkQxPzt4UZ2UWPWiCZWLvFZ4fu5cueciegX0zGPnrlY6bwRg4FdQOe9YU8MkmJwGhoMybl8A==",
       "license": "MIT",
       "engines": {
         "node": ">= 0.8"
@@ -1565,15 +1588,6 @@
         "url": "https://github.com/sponsors/sindresorhus"
       }
     },
-    "node_modules/os-tmpdir": {
-      "version": "1.0.2",
-      "resolved": "https://registry.npmjs.org/os-tmpdir/-/os-tmpdir-1.0.2.tgz",
-      "integrity": "sha512-D2FR03Vir7FIu45XBY20mTb+/ZSWB00sjU9jdQXt83gDrI4Ztz5Fs7/yy74g2N5SVQY4xY1qDr4rNddwYRVX0g==",
-      "license": "MIT",
-      "engines": {
-        "node": ">=0.10.0"
-      }
-    },
     "node_modules/parseurl": {
       "version": "1.3.3",
       "resolved": "https://registry.npmjs.org/parseurl/-/parseurl-1.3.3.tgz",
@@ -2109,18 +2123,6 @@
       "integrity": "sha512-w89qg7PI8wAdvX60bMDP+bFoD5Dvhm9oLheFp5O4a2QF0cSBGsBX4qZmadPMvVqlLJBBci+WqGGOAPvcDeNSVg==",
       "license": "MIT"
     },
-    "node_modules/tmp": {
-      "version": "0.0.33",
-      "resolved": "https://registry.npmjs.org/tmp/-/tmp-0.0.33.tgz",
-      "integrity": "sha512-jRCJlojKnZ3addtTOjdIqoRuPEKBvNXcGYqzO6zWZX8KfKEpnGY5jfggJQ3EjKuu8D4bJRr0y+cYJFmYbImXGw==",
-      "license": "MIT",
-      "dependencies": {
-        "os-tmpdir": "~1.0.2"
-      },
-      "engines": {
-        "node": ">=0.6.0"
-      }
-    },
     "node_modules/toidentifier": {
       "version": "1.0.1",
       "resolved": "https://registry.npmjs.org/toidentifier/-/toidentifier-1.0.1.tgz",

inventory-server/auth/package.json

@@ -2,18 +2,22 @@
   "name": "inventory-auth-server",
   "version": "1.0.0",
   "description": "Authentication server for inventory management system",
+  "type": "module",
   "main": "server.js",
   "scripts": {
-    "start": "node server.js"
+    "start": "node server.js",
+    "add-user": "node add-user.js"
   },
   "dependencies": {
     "bcrypt": "^5.1.1",
     "cors": "^2.8.5",
     "dotenv": "^16.4.7",
     "express": "^4.18.2",
+    "express-rate-limit": "^7.4.0",
     "inquirer": "^8.2.6",
     "jsonwebtoken": "^9.0.2",
-    "morgan": "^1.10.0",
+    "pg": "^8.11.3",
-    "pg": "^8.11.3"
+    "pino": "^9.5.0",
+    "pino-http": "^10.3.0"
   }
 }

inventory-server/auth/permissions.js

@@ -1,128 +1,73 @@
-// Get pool from global or create a new one if not available
+export function createPermissionHelpers({ pool }) {
-let pool;
+  async function checkPermission(userId, permissionCode) {
-if (typeof global.pool !== 'undefined') {
-  pool = global.pool;
-} else {
-  // If global pool is not available, create a new connection
-  const { Pool } = require('pg');
-  pool = new Pool({
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT,
-  });
-  console.log('Created new database pool in permissions.js');
-}
-
-/**
- * Check if a user has a specific permission
- * @param {number} userId - The user ID to check
- * @param {string} permissionCode - The permission code to check
- * @returns {Promise<boolean>} - Whether the user has the permission
- */
-async function checkPermission(userId, permissionCode) {
-  try {
-    // First check if the user is an admin
-    const adminResult = await pool.query(
-      'SELECT is_admin FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    // If user is admin, automatically grant permission
-    if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) {
-      return true;
-    }
-    
-    // Otherwise check for specific permission
-    const result = await pool.query(
-      `SELECT COUNT(*) AS has_permission
-       FROM user_permissions up
-       JOIN permissions p ON up.permission_id = p.id
-       WHERE up.user_id = $1 AND p.code = $2`,
-      [userId, permissionCode]
-    );
-    
-    return result.rows[0].has_permission > 0;
-  } catch (error) {
-    console.error('Error checking permission:', error);
-    return false;
-  }
-}
-
-/**
- * Middleware to require a specific permission
- * @param {string} permissionCode - The permission code required
- * @returns {Function} - Express middleware function
- */
-function requirePermission(permissionCode) {
-  return async (req, res, next) => {
     try {
-      // Check if user is authenticated
+      const adminResult = await pool.query(
-      if (!req.user || !req.user.id) {
+        'SELECT is_admin FROM users WHERE id = $1',
-        return res.status(401).json({ error: 'Authentication required' });
-      }
-      
-      const hasPermission = await checkPermission(req.user.id, permissionCode);
-      
-      if (!hasPermission) {
-        return res.status(403).json({ 
-          error: 'Insufficient permissions',
-          requiredPermission: permissionCode
-        });
-      }
-      
-      next();
-    } catch (error) {
-      console.error('Permission middleware error:', error);
-      res.status(500).json({ error: 'Server error checking permissions' });
-    }
-  };
-}
-
-/**
- * Get all permissions for a user
- * @param {number} userId - The user ID
- * @returns {Promise<string[]>} - Array of permission codes
- */
-async function getUserPermissions(userId) {
-  try {
-    // Check if user is admin
-    const adminResult = await pool.query(
-      'SELECT is_admin FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    if (adminResult.rows.length === 0) {
-      return [];
-    }
-    
-    const isAdmin = adminResult.rows[0].is_admin;
-    
-    if (isAdmin) {
-      // Admin gets all permissions
-      const allPermissions = await pool.query('SELECT code FROM permissions');
-      return allPermissions.rows.map(p => p.code);
-    } else {
-      // Get assigned permissions
-      const permissions = await pool.query(
-        `SELECT p.code
-         FROM permissions p
-         JOIN user_permissions up ON p.id = up.permission_id
-         WHERE up.user_id = $1`,
         [userId]
       );
-      
+      if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) return true;
-      return permissions.rows.map(p => p.code);
-    }
-  } catch (error) {
-    console.error('Error getting user permissions:', error);
-    return [];
-  }
-}
 
-module.exports = {
+      const result = await pool.query(
-  checkPermission,
+        `SELECT COUNT(*) AS has_permission
-  requirePermission,
+           FROM user_permissions up
-  getUserPermissions
+           JOIN permissions p ON up.permission_id = p.id
-}; 
+          WHERE up.user_id = $1 AND p.code = $2`,
+        [userId, permissionCode]
+      );
+      return Number(result.rows[0].has_permission) > 0;
+    } catch (error) {
+      console.error('Error checking permission:', error);
+      return false;
+    }
+  }
+
+  function requirePermission(permissionCode) {
+    return async (req, res, next) => {
+      try {
+        if (!req.user?.id) {
+          return res.status(401).json({ error: 'Authentication required' });
+        }
+        const hasPermission = await checkPermission(req.user.id, permissionCode);
+        if (!hasPermission) {
+          return res.status(403).json({
+            error: 'Insufficient permissions',
+            requiredPermission: permissionCode,
+          });
+        }
+        next();
+      } catch (error) {
+        console.error('Permission middleware error:', error);
+        res.status(500).json({ error: 'Server error checking permissions' });
+      }
+    };
+  }
+
+  async function getUserPermissions(userId) {
+    try {
+      const adminResult = await pool.query(
+        'SELECT is_admin FROM users WHERE id = $1',
+        [userId]
+      );
+      if (adminResult.rows.length === 0) return [];
+
+      if (adminResult.rows[0].is_admin) {
+        const allPermissions = await pool.query('SELECT code FROM permissions');
+        return allPermissions.rows.map((p) => p.code);
+      }
+
+      const permissions = await pool.query(
+        `SELECT p.code
+           FROM permissions p
+           JOIN user_permissions up ON p.id = up.permission_id
+          WHERE up.user_id = $1`,
+        [userId]
+      );
+      return permissions.rows.map((p) => p.code);
+    } catch (error) {
+      console.error('Error getting user permissions:', error);
+      return [];
+    }
+  }
+
+  return { checkPermission, requirePermission, getUserPermissions };
+}

inventory-server/auth/routes.js

@@ -1,513 +1,317 @@
-const express = require('express');
+import express from 'express';
-const router = express.Router();
+import bcrypt from 'bcrypt';
-const bcrypt = require('bcrypt');
+import jwt from 'jsonwebtoken';
-const jwt = require('jsonwebtoken');
+import { createPermissionHelpers } from './permissions.js';
-const { requirePermission, getUserPermissions } = require('./permissions');
 
-// Get pool from global or create a new one if not available
+export function createAuthRoutes({ pool }) {
-let pool;
+  const router = express.Router();
-if (typeof global.pool !== 'undefined') {
+  const { requirePermission, getUserPermissions } = createPermissionHelpers({ pool });
-  pool = global.pool;
+
-} else {
+  // Local authenticate(): used by user-management endpoints that need req.user populated
-  // If global pool is not available, create a new connection
+  // with id/username/email/is_admin. NOT the per-service authenticate() — that lives in
-  const { Pool } = require('pg');
+  // shared/auth/middleware.js and is used by downstream services. Auth-server's surface is
-  pool = new Pool({
+  // small enough that a local copy is fine; the security boundary is the JWT verify step.
-    host: process.env.DB_HOST,
+  async function authenticate(req, res, next) {
-    user: process.env.DB_USER,
+    try {
-    password: process.env.DB_PASSWORD,
+      const authHeader = req.headers.authorization;
-    database: process.env.DB_NAME,
+      if (!authHeader || !authHeader.startsWith('Bearer ')) {
-    port: process.env.DB_PORT,
+        return res.status(401).json({ error: 'Authentication required' });
+      }
+      const token = authHeader.split(' ')[1];
+      const decoded = jwt.verify(token, process.env.JWT_SECRET);
+      const result = await pool.query(
+        'SELECT id, username, email, is_admin, rocket_chat_user_id FROM users WHERE id = $1',
+        [decoded.userId]
+      );
+      if (result.rows.length === 0) {
+        return res.status(401).json({ error: 'User not found' });
+      }
+      req.user = result.rows[0];
+      next();
+    } catch (error) {
+      res.status(401).json({ error: 'Invalid token' });
+    }
+  }
+
+  router.post('/login', async (req, res) => {
+    try {
+      const { username, password } = req.body;
+      const result = await pool.query(
+        'SELECT id, username, password, is_admin, is_active, rocket_chat_user_id FROM users WHERE username = $1',
+        [username]
+      );
+      if (result.rows.length === 0) {
+        return res.status(401).json({ error: 'Invalid username or password' });
+      }
+      const user = result.rows[0];
+      if (!user.is_active) {
+        return res.status(403).json({ error: 'Account is inactive' });
+      }
+      const validPassword = await bcrypt.compare(password, user.password);
+      if (!validPassword) {
+        return res.status(401).json({ error: 'Invalid username or password' });
+      }
+      await pool.query(
+        'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
+        [user.id]
+      );
+      const token = jwt.sign(
+        { userId: user.id, username: user.username },
+        process.env.JWT_SECRET,
+        { expiresIn: '8h' }
+      );
+      const permissions = await getUserPermissions(user.id);
+      res.json({
+        token,
+        user: {
+          id: user.id,
+          username: user.username,
+          is_admin: user.is_admin,
+          rocket_chat_user_id: user.rocket_chat_user_id,
+          permissions,
+        },
+      });
+    } catch (error) {
+      console.error('Login error:', error);
+      res.status(500).json({ error: 'Server error' });
+    }
   });
-  console.log('Created new database pool in routes.js');
-}
 
-// Authentication middleware
+  router.get('/me', authenticate, async (req, res) => {
-const authenticate = async (req, res, next) => {
+    try {
-  try {
+      const permissions = await getUserPermissions(req.user.id);
-    const authHeader = req.headers.authorization;
+      res.json({
-    if (!authHeader || !authHeader.startsWith('Bearer ')) {
+        id: req.user.id,
-      return res.status(401).json({ error: 'Authentication required' });
+        username: req.user.username,
+        email: req.user.email,
+        is_admin: req.user.is_admin,
+        rocket_chat_user_id: req.user.rocket_chat_user_id,
+        permissions,
+      });
+    } catch (error) {
+      console.error('Error getting current user:', error);
+      res.status(500).json({ error: 'Server error' });
     }
+  });
 
-    const token = authHeader.split(' ')[1];
+  router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
-    const decoded = jwt.verify(token, process.env.JWT_SECRET);
+    try {
-    
+      const result = await pool.query(`
-    // Get user from database
+        SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
-    const result = await pool.query(
+        FROM users
-      'SELECT id, username, is_admin FROM users WHERE id = $1',
+        ORDER BY username
-      [decoded.userId]
+      `);
-    );
+      res.json(result.rows);
-    
+    } catch (error) {
-    if (result.rows.length === 0) {
+      console.error('Error getting users:', error);
-      return res.status(401).json({ error: 'User not found' });
+      res.status(500).json({ error: 'Server error' });
     }
-    
+  });
-    // Attach user to request
-    req.user = result.rows[0];
-    next();
-  } catch (error) {
-    console.error('Authentication error:', error);
-    res.status(401).json({ error: 'Invalid token' });
-  }
-};
 
-// Login route
+  router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
-router.post('/login', async (req, res) => {
+    try {
-  try {
+      const userId = req.params.id;
-    const { username, password } = req.body;
+      const userResult = await pool.query(`
-    
+        SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
-    // Get user from database
+        FROM users
-    const result = await pool.query(
-      'SELECT id, username, password, is_admin, is_active FROM users WHERE username = $1',
-      [username]
-    );
-    
-    if (result.rows.length === 0) {
-      return res.status(401).json({ error: 'Invalid username or password' });
-    }
-    
-    const user = result.rows[0];
-    
-    // Check if user is active
-    if (!user.is_active) {
-      return res.status(403).json({ error: 'Account is inactive' });
-    }
-    
-    // Verify password
-    const validPassword = await bcrypt.compare(password, user.password);
-    if (!validPassword) {
-      return res.status(401).json({ error: 'Invalid username or password' });
-    }
-    
-    // Update last login
-    await pool.query(
-      'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
-      [user.id]
-    );
-    
-    // Generate JWT
-    const token = jwt.sign(
-      { userId: user.id, username: user.username },
-      process.env.JWT_SECRET,
-      { expiresIn: '8h' }
-    );
-    
-    // Get user permissions
-    const permissions = await getUserPermissions(user.id);
-    
-    res.json({
-      token,
-      user: {
-        id: user.id,
-        username: user.username,
-        is_admin: user.is_admin,
-        permissions
-      }
-    });
-  } catch (error) {
-    console.error('Login error:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get current user
-router.get('/me', authenticate, async (req, res) => {
-  try {
-    // Get user permissions
-    const permissions = await getUserPermissions(req.user.id);
-    
-    res.json({
-      id: req.user.id,
-      username: req.user.username,
-      is_admin: req.user.is_admin,
-      permissions
-    });
-  } catch (error) {
-    console.error('Error getting current user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get all users
-router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const result = await pool.query(`
-      SELECT id, username, email, is_admin, is_active, created_at, last_login
-      FROM users
-      ORDER BY username
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error getting users:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get user with permissions
-router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const userId = req.params.id;
-    
-    // Get user details
-    const userResult = await pool.query(`
-      SELECT id, username, email, is_admin, is_active, created_at, last_login
-      FROM users
-      WHERE id = $1
-    `, [userId]);
-    
-    if (userResult.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    // Get user permissions
-    const permissionsResult = await pool.query(`
-      SELECT p.id, p.name, p.code, p.category, p.description
-      FROM permissions p
-      JOIN user_permissions up ON p.id = up.permission_id
-      WHERE up.user_id = $1
-      ORDER BY p.category, p.name
-    `, [userId]);
-    
-    // Combine user and permissions
-    const user = {
-      ...userResult.rows[0],
-      permissions: permissionsResult.rows
-    };
-    
-    res.json(user);
-  } catch (error) {
-    console.error('Error getting user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Create new user
-router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
-  const client = await pool.connect();
-  
-  try {
-    const { username, email, password, is_admin, is_active, permissions } = req.body;
-    
-    console.log("Create user request:", {
-      username,
-      email,
-      is_admin,
-      is_active,
-      permissions: permissions || []
-    });
-    
-    // Validate required fields
-    if (!username || !password) {
-      return res.status(400).json({ error: 'Username and password are required' });
-    }
-    
-    // Check if username is taken
-    const existingUser = await client.query(
-      'SELECT id FROM users WHERE username = $1',
-      [username]
-    );
-    
-    if (existingUser.rows.length > 0) {
-      return res.status(400).json({ error: 'Username already exists' });
-    }
-    
-    // Start transaction
-    await client.query('BEGIN');
-    
-    // Hash password
-    const saltRounds = 10;
-    const hashedPassword = await bcrypt.hash(password, saltRounds);
-    
-    // Insert new user
-    const userResult = await client.query(`
-      INSERT INTO users (username, email, password, is_admin, is_active, created_at)
-      VALUES ($1, $2, $3, $4, $5, CURRENT_TIMESTAMP)
-      RETURNING id
-    `, [username, email || null, hashedPassword, !!is_admin, is_active !== false]);
-    
-    const userId = userResult.rows[0].id;
-    
-    // Assign permissions if provided and not admin
-    if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
-      console.log("Adding permissions for new user:", userId);
-      console.log("Permissions received:", permissions);
-      
-      // Check permission format
-      const permissionIds = permissions.map(p => {
-        if (typeof p === 'object' && p.id) {
-          console.log("Permission is an object with ID:", p.id);
-          return parseInt(p.id, 10);
-        } else if (typeof p === 'number') {
-          console.log("Permission is a number:", p);
-          return p;
-        } else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
-          console.log("Permission is a string that can be parsed as a number:", p);
-          return parseInt(p, 10);
-        } else {
-          console.log("Unknown permission format:", typeof p, p);
-          // If it's a permission code, we need to look up the ID
-          return null;
-        }
-      }).filter(id => id !== null);
-      
-      console.log("Filtered permission IDs:", permissionIds);
-      
-      if (permissionIds.length > 0) {
-        const permissionValues = permissionIds
-          .map(permId => `(${userId}, ${permId})`)
-          .join(',');
-        
-        console.log("Inserting permission values:", permissionValues);
-        
-        try {
-          await client.query(`
-            INSERT INTO user_permissions (user_id, permission_id)
-            VALUES ${permissionValues}
-            ON CONFLICT DO NOTHING
-          `);
-          console.log("Successfully inserted permissions for new user:", userId);
-        } catch (err) {
-          console.error("Error inserting permissions for new user:", err);
-          throw err;
-        }
-      } else {
-        console.log("No valid permission IDs found to insert for new user");
-      }
-    } else {
-      console.log("Not adding permissions: is_admin =", is_admin, "permissions array:", Array.isArray(permissions), "length:", permissions ? permissions.length : 0);
-    }
-    
-    await client.query('COMMIT');
-    
-    res.status(201).json({ 
-      id: userId, 
-      message: 'User created successfully' 
-    });
-  } catch (error) {
-    await client.query('ROLLBACK');
-    console.error('Error creating user:', error);
-    res.status(500).json({ error: 'Server error' });
-  } finally {
-    client.release();
-  }
-});
-
-// Update user
-router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
-  const client = await pool.connect();
-  
-  try {
-    const userId = req.params.id;
-    const { username, email, password, is_admin, is_active, permissions } = req.body;
-    
-    console.log("Update user request:", {
-      userId,
-      username,
-      email,
-      is_admin,
-      is_active,
-      permissions: permissions || []
-    });
-    
-    // Check if user exists
-    const userExists = await client.query(
-      'SELECT id FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    if (userExists.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    // Start transaction
-    await client.query('BEGIN');
-    
-    // Build update fields
-    const updateFields = [];
-    const updateValues = [userId]; // First parameter is the user ID
-    let paramIndex = 2;
-    
-    if (username !== undefined) {
-      updateFields.push(`username = $${paramIndex++}`);
-      updateValues.push(username);
-    }
-    
-    if (email !== undefined) {
-      updateFields.push(`email = $${paramIndex++}`);
-      updateValues.push(email || null);
-    }
-    
-    if (is_admin !== undefined) {
-      updateFields.push(`is_admin = $${paramIndex++}`);
-      updateValues.push(!!is_admin);
-    }
-    
-    if (is_active !== undefined) {
-      updateFields.push(`is_active = $${paramIndex++}`);
-      updateValues.push(!!is_active);
-    }
-    
-    // Update password if provided
-    if (password) {
-      const saltRounds = 10;
-      const hashedPassword = await bcrypt.hash(password, saltRounds);
-      updateFields.push(`password = $${paramIndex++}`);
-      updateValues.push(hashedPassword);
-    }
-    
-    // Update user if there are fields to update
-    if (updateFields.length > 0) {
-      updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
-      
-      await client.query(`
-        UPDATE users
-        SET ${updateFields.join(', ')}
         WHERE id = $1
-      `, updateValues);
+      `, [userId]);
+      if (userResult.rows.length === 0) {
+        return res.status(404).json({ error: 'User not found' });
+      }
+      const permissionsResult = await pool.query(`
+        SELECT p.id, p.name, p.code, p.category, p.description
+        FROM permissions p
+        JOIN user_permissions up ON p.id = up.permission_id
+        WHERE up.user_id = $1
+        ORDER BY p.category, p.name
+      `, [userId]);
+      res.json({
+        ...userResult.rows[0],
+        permissions: permissionsResult.rows,
+      });
+    } catch (error) {
+      console.error('Error getting user:', error);
+      res.status(500).json({ error: 'Server error' });
     }
-    
+  });
-    // Update permissions if provided
+
-    if (Array.isArray(permissions)) {
+  router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
-      console.log("Updating permissions for user:", userId);
+    const client = await pool.connect();
-      console.log("Permissions received:", permissions);
+    try {
-      
+      const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
-      // First remove existing permissions
+      if (!username || !password) {
-      await client.query(
+        return res.status(400).json({ error: 'Username and password are required' });
-        'DELETE FROM user_permissions WHERE user_id = $1',
+      }
+      const existingUser = await client.query(
+        'SELECT id FROM users WHERE username = $1',
+        [username]
+      );
+      if (existingUser.rows.length > 0) {
+        return res.status(400).json({ error: 'Username already exists' });
+      }
+      await client.query('BEGIN');
+      const hashedPassword = await bcrypt.hash(password, 10);
+      const rcUserId = rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null;
+      const userResult = await client.query(`
+        INSERT INTO users (username, email, password, is_admin, is_active, rocket_chat_user_id, created_at)
+        VALUES ($1, $2, $3, $4, $5, $6, CURRENT_TIMESTAMP)
+        RETURNING id
+      `, [username, email || null, hashedPassword, !!is_admin, is_active !== false, rcUserId]);
+      const userId = userResult.rows[0].id;
+
+      if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
+        const permissionIds = normalizePermissionIds(permissions);
+        if (permissionIds.length > 0) {
+          await client.query(
+            `INSERT INTO user_permissions (user_id, permission_id)
+             SELECT $1, unnest($2::int[])
+             ON CONFLICT DO NOTHING`,
+            [userId, permissionIds]
+          );
+        }
+      }
+
+      await client.query('COMMIT');
+      res.status(201).json({ id: userId, message: 'User created successfully' });
+    } catch (error) {
+      await client.query('ROLLBACK');
+      console.error('Error creating user:', error);
+      res.status(500).json({ error: 'Server error' });
+    } finally {
+      client.release();
+    }
+  });
+
+  router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
+    const client = await pool.connect();
+    try {
+      const userId = req.params.id;
+      const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
+
+      const userExists = await client.query('SELECT id FROM users WHERE id = $1', [userId]);
+      if (userExists.rows.length === 0) {
+        return res.status(404).json({ error: 'User not found' });
+      }
+
+      await client.query('BEGIN');
+
+      const updateFields = [];
+      const updateValues = [userId];
+      let paramIndex = 2;
+
+      if (username !== undefined) { updateFields.push(`username = $${paramIndex++}`); updateValues.push(username); }
+      if (email !== undefined) { updateFields.push(`email = $${paramIndex++}`); updateValues.push(email || null); }
+      if (is_admin !== undefined) { updateFields.push(`is_admin = $${paramIndex++}`); updateValues.push(!!is_admin); }
+      if (is_active !== undefined) { updateFields.push(`is_active = $${paramIndex++}`); updateValues.push(!!is_active); }
+      if (rocket_chat_user_id !== undefined) {
+        updateFields.push(`rocket_chat_user_id = $${paramIndex++}`);
+        updateValues.push(rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null);
+      }
+      if (password) {
+        const hashedPassword = await bcrypt.hash(password, 10);
+        updateFields.push(`password = $${paramIndex++}`);
+        updateValues.push(hashedPassword);
+      }
+
+      if (updateFields.length > 0) {
+        updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
+        await client.query(`
+          UPDATE users SET ${updateFields.join(', ')} WHERE id = $1
+        `, updateValues);
+      }
+
+      if (Array.isArray(permissions)) {
+        await client.query('DELETE FROM user_permissions WHERE user_id = $1', [userId]);
+        const newIsAdmin = is_admin !== undefined
+          ? is_admin
+          : (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
+
+        if (!newIsAdmin && permissions.length > 0) {
+          const permissionIds = normalizePermissionIds(permissions);
+          if (permissionIds.length > 0) {
+            await client.query(
+              `INSERT INTO user_permissions (user_id, permission_id)
+               SELECT $1, unnest($2::int[])
+               ON CONFLICT DO NOTHING`,
+              [userId, permissionIds]
+            );
+          }
+        }
+      }
+
+      await client.query('COMMIT');
+      res.json({ message: 'User updated successfully' });
+    } catch (error) {
+      await client.query('ROLLBACK');
+      console.error('Error updating user:', error);
+      res.status(500).json({ error: 'Server error' });
+    } finally {
+      client.release();
+    }
+  });
+
+  router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => {
+    try {
+      const userId = req.params.id;
+      if (req.user.id === parseInt(userId, 10)) {
+        return res.status(400).json({ error: 'Cannot delete your own account' });
+      }
+      const result = await pool.query(
+        'DELETE FROM users WHERE id = $1 RETURNING id',
         [userId]
       );
-      console.log("Deleted existing permissions for user:", userId);
+      if (result.rows.length === 0) {
-      
+        return res.status(404).json({ error: 'User not found' });
-      // Add new permissions if any and not admin
-      const newIsAdmin = is_admin !== undefined ? is_admin : (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
-      
-      console.log("User is admin:", newIsAdmin);
-      
-      if (!newIsAdmin && permissions.length > 0) {
-        console.log("Adding permissions:", permissions);
-        
-        // Check permission format
-        const permissionIds = permissions.map(p => {
-          if (typeof p === 'object' && p.id) {
-            console.log("Permission is an object with ID:", p.id);
-            return parseInt(p.id, 10);
-          } else if (typeof p === 'number') {
-            console.log("Permission is a number:", p);
-            return p;
-          } else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
-            console.log("Permission is a string that can be parsed as a number:", p);
-            return parseInt(p, 10);
-          } else {
-            console.log("Unknown permission format:", typeof p, p);
-            // If it's a permission code, we need to look up the ID
-            return null;
-          }
-        }).filter(id => id !== null);
-        
-        console.log("Filtered permission IDs:", permissionIds);
-        
-        if (permissionIds.length > 0) {
-          const permissionValues = permissionIds
-            .map(permId => `(${userId}, ${permId})`)
-            .join(',');
-          
-          console.log("Inserting permission values:", permissionValues);
-          
-          try {
-            await client.query(`
-              INSERT INTO user_permissions (user_id, permission_id)
-              VALUES ${permissionValues}
-              ON CONFLICT DO NOTHING
-            `);
-            console.log("Successfully inserted permissions for user:", userId);
-          } catch (err) {
-            console.error("Error inserting permissions:", err);
-            throw err;
-          }
-        } else {
-          console.log("No valid permission IDs found to insert");
-        }
       }
+      res.json({ message: 'User deleted successfully' });
+    } catch (error) {
+      console.error('Error deleting user:', error);
+      res.status(500).json({ error: 'Server error' });
     }
-    
+  });
-    await client.query('COMMIT');
-    
-    res.json({ message: 'User updated successfully' });
-  } catch (error) {
-    await client.query('ROLLBACK');
-    console.error('Error updating user:', error);
-    res.status(500).json({ error: 'Server error' });
-  } finally {
-    client.release();
-  }
-});
 
-// Delete user
+  router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => {
-router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => {
+    try {
-  try {
+      const result = await pool.query(`
-    const userId = req.params.id;
+        SELECT category, json_agg(
-    
+          json_build_object(
-    // Check that user is not deleting themselves
+            'id', id, 'name', name, 'code', code, 'description', description
-    if (req.user.id === parseInt(userId, 10)) {
+          ) ORDER BY name
-      return res.status(400).json({ error: 'Cannot delete your own account' });
+        ) as permissions
+        FROM permissions
+        GROUP BY category
+        ORDER BY category
+      `);
+      res.json(result.rows);
+    } catch (error) {
+      console.error('Error getting permissions:', error);
+      res.status(500).json({ error: 'Server error' });
     }
-    
+  });
-    // Delete user (this will cascade to user_permissions due to FK constraints)
+
-    const result = await pool.query(
+  router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => {
-      'DELETE FROM users WHERE id = $1 RETURNING id',
+    try {
-      [userId]
+      const result = await pool.query(`
-    );
+        SELECT * FROM permissions ORDER BY category, name
-    
+      `);
-    if (result.rows.length === 0) {
+      res.json(result.rows);
-      return res.status(404).json({ error: 'User not found' });
+    } catch (error) {
+      console.error('Error getting permissions:', error);
+      res.status(500).json({ error: 'Server error' });
     }
-    
+  });
-    res.json({ message: 'User deleted successfully' });
-  } catch (error) {
-    console.error('Error deleting user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
 
-// Get all permissions grouped by category
+  return router;
-router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => {
+}
-  try {
-    const result = await pool.query(`
-      SELECT category, json_agg(
-        json_build_object(
-          'id', id,
-          'name', name,
-          'code', code,
-          'description', description
-        ) ORDER BY name
-      ) as permissions
-      FROM permissions
-      GROUP BY category
-      ORDER BY category
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error getting permissions:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
 
-// Get all permissions
+function normalizePermissionIds(permissions) {
-router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => {
+  return permissions
-  try {
+    .map((p) => {
-    const result = await pool.query(`
+      if (typeof p === 'object' && p?.id) return parseInt(p.id, 10);
-      SELECT *
+      if (typeof p === 'number') return p;
-      FROM permissions
+      if (typeof p === 'string' && !Number.isNaN(parseInt(p, 10))) return parseInt(p, 10);
-      ORDER BY category, name
+      return null;
-    `);
+    })
-    
+    .filter((id) => id !== null && !Number.isNaN(id));
-    res.json(result.rows);
+}
-  } catch (error) {
-    console.error('Error getting permissions:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-module.exports = router; 

inventory-server/auth/server.js

@@ -1,164 +1,84 @@
-require('dotenv').config({ path: '../.env' });
+import 'dotenv/config';
-const express = require('express');
+import express from 'express';
-const cors = require('cors');
+import cors from 'cors';
-const bcrypt = require('bcrypt');
+import pg from 'pg';
-const jwt = require('jsonwebtoken');
+import { fileURLToPath } from 'node:url';
-const { Pool } = require('pg');
-const morgan = require('morgan');
-const authRoutes = require('./routes');
 
-// Log startup configuration
+const { Pool } = pg;
-console.log('Starting auth server with config:', {
+import { dirname, resolve as resolvePath } from 'node:path';
+import { config as loadEnv } from 'dotenv';
+
+import { corsOptions } from '../shared/cors/policy.js';
+import { requestLog } from '../shared/logging/request-log.js';
+import { logger } from '../shared/logging/logger.js';
+import { errorHandler } from '../shared/errors/handler.js';
+import { loginLimiter, verifyLimiter } from '../shared/rate-limit/login.js';
+import { extractBearerToken, verifyToken, TokenError } from '../shared/auth/verify.js';
+
+import { createAuthRoutes } from './routes.js';
+
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = dirname(__filename);
+
+// auth/ lives at inventory-server/auth/, so .env one level up
+loadEnv({ path: resolvePath(__dirname, '../.env') });
+
+if (!process.env.JWT_SECRET) {
+  logger.error('JWT_SECRET is not set; refusing to start');
+  process.exit(1);
+}
+
+logger.info({
   host: process.env.DB_HOST,
   user: process.env.DB_USER,
   database: process.env.DB_NAME,
   port: process.env.DB_PORT,
-  auth_port: process.env.AUTH_PORT
+  auth_port: process.env.AUTH_PORT,
-});
+}, 'starting auth server');
 
 const app = express();
-const port = process.env.AUTH_PORT || 3011;
+const port = Number(process.env.AUTH_PORT) || 3011;
 
-// Database configuration
 const pool = new Pool({
   host: process.env.DB_HOST,
   user: process.env.DB_USER,
   password: process.env.DB_PASSWORD,
   database: process.env.DB_NAME,
-  port: process.env.DB_PORT,
+  port: Number(process.env.DB_PORT) || 5432,
 });
 
-// Make pool available globally
+app.use(requestLog());
-global.pool = pool;
+app.use(express.json({ limit: '1mb' }));
-
+app.use(cors(corsOptions));
-// Middleware
-app.use(express.json());
-app.use(morgan('combined'));
-app.use(cors({
-  origin: ['http://localhost:5173', 'http://localhost:5174', 'https://inventory.kent.pw'],
-  credentials: true
-}));
-
-// Login endpoint
-app.post('/login', async (req, res) => {
-  const { username, password } = req.body;
 
+// Caddy forward_auth target: JWT signature check only, no DB hit.
+// Returns 200 with X-User-Id / X-User-Username on success; 401 otherwise.
+// Per-service middleware re-verifies independently; these headers are informational.
+app.all('/verify', verifyLimiter, (req, res) => {
   try {
-    // Get user from database
+    const token = extractBearerToken(req.headers.authorization);
-    const result = await pool.query(
+    const decoded = verifyToken(token, process.env.JWT_SECRET);
-      'SELECT id, username, password, is_admin, is_active FROM users WHERE username = $1',
+    res.set('X-User-Id', String(decoded.userId));
-      [username]
+    if (decoded.username) res.set('X-User-Username', decoded.username);
-    );
+    res.status(200).end();
-
+  } catch (err) {
-    const user = result.rows[0];
+    if (err instanceof TokenError) {
-
+      return res.status(401).json({ error: err.message });
-    // Check if user exists and password is correct
-    if (!user || !(await bcrypt.compare(password, user.password))) {
-      return res.status(401).json({ error: 'Invalid username or password' });
     }
-    
-    // Check if user is active
-    if (!user.is_active) {
-      return res.status(403).json({ error: 'Account is inactive' });
-    }
-
-    // Generate JWT token
-    const token = jwt.sign(
-      { userId: user.id, username: user.username },
-      process.env.JWT_SECRET,
-      { expiresIn: '24h' }
-    );
-
-    // Get user permissions for the response
-    const permissionsResult = await pool.query(`
-      SELECT code
-      FROM permissions p
-      JOIN user_permissions up ON p.id = up.permission_id
-      WHERE up.user_id = $1
-    `, [user.id]);
-    
-    const permissions = permissionsResult.rows.map(row => row.code);
-
-    res.json({ 
-      token,
-      user: {
-        id: user.id,
-        username: user.username,
-        is_admin: user.is_admin,
-        permissions: user.is_admin ? [] : permissions
-      }
-    });
-  } catch (error) {
-    console.error('Login error:', error);
-    res.status(500).json({ error: 'Internal server error' });
-  }
-});
-
-// User info endpoint
-app.get('/me', async (req, res) => {
-  const authHeader = req.headers.authorization;
-
-  if (!authHeader || !authHeader.startsWith('Bearer ')) {
-    return res.status(401).json({ error: 'No token provided' });
-  }
-
-  try {
-    const token = authHeader.split(' ')[1];
-    const decoded = jwt.verify(token, process.env.JWT_SECRET);
-    
-    // Get user details from database
-    const userResult = await pool.query(
-      'SELECT id, username, email, is_admin, is_active FROM users WHERE id = $1',
-      [decoded.userId]
-    );
-    
-    if (userResult.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    const user = userResult.rows[0];
-    
-    // Get user permissions
-    let permissions = [];
-    if (!user.is_admin) {
-      const permissionsResult = await pool.query(`
-        SELECT code
-        FROM permissions p
-        JOIN user_permissions up ON p.id = up.permission_id
-        WHERE up.user_id = $1
-      `, [user.id]);
-      
-      permissions = permissionsResult.rows.map(row => row.code);
-    }
-    
-    res.json({
-      id: user.id,
-      username: user.username,
-      email: user.email,
-      is_admin: user.is_admin,
-      permissions: permissions
-    });
-  } catch (error) {
-    console.error('Token verification error:', error);
     res.status(401).json({ error: 'Invalid token' });
   }
 });
 
-// Mount all routes from routes.js
+// Login route gets its own rate limiter to slow credential stuffing.
-app.use('/', authRoutes);
+app.use('/login', loginLimiter);
 
-// Health check endpoint
+// Mount user-management + /login + /me from routes.js
-app.get('/health', (req, res) => {
+app.use('/', createAuthRoutes({ pool }));
-  res.json({ status: 'healthy' });
-});
 
-// Error handling middleware
+app.get('/health', (req, res) => res.json({ status: 'healthy' }));
-app.use((err, req, res, next) => {
+
-  console.error(err.stack);
+app.use(errorHandler);
-  res.status(500).json({ error: 'Something broke!' });
-});
 
-// Start server
 app.listen(port, () => {
-  console.log(`Auth server running on port ${port}`);
+  logger.info({ port }, 'auth server listening');
 });

inventory-server/chat/create-new-database.sql

@@ -0,0 +1,45 @@
+-- PostgreSQL Database Creation Script for New Server
+-- Run as: sudo -u postgres psql -f create-new-database.sql
+
+-- Terminate all connections to the database (if it exists)
+SELECT pg_terminate_backend(pid)
+FROM pg_stat_activity
+WHERE datname = 'rocketchat_converted' AND pid <> pg_backend_pid();
+
+-- Drop the database if it exists
+DROP DATABASE IF EXISTS rocketchat_converted;
+
+-- Create fresh database
+CREATE DATABASE rocketchat_converted;
+
+-- Create user (if not exists) - UPDATE PASSWORD BEFORE RUNNING!
+DO $$ 
+BEGIN
+    IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'rocketchat_user') THEN
+        CREATE USER rocketchat_user WITH PASSWORD 'HKjLgt23gWuPXzEAn3rW';
+    END IF;
+END $$;
+
+-- Grant database privileges
+GRANT CONNECT ON DATABASE rocketchat_converted TO rocketchat_user;
+GRANT CREATE ON DATABASE rocketchat_converted TO rocketchat_user;
+
+-- Connect to the new database
+\c rocketchat_converted;
+
+-- Grant schema privileges
+GRANT CREATE ON SCHEMA public TO rocketchat_user;
+GRANT USAGE ON SCHEMA public TO rocketchat_user;
+
+-- Grant privileges on all future tables and sequences
+ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO rocketchat_user;
+ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO rocketchat_user;
+
+-- Display success message
+\echo 'Database created successfully!'
+\echo 'IMPORTANT: Update the password for rocketchat_user before proceeding'
+\echo 'Next steps:'
+\echo '1. Update the password in this file'
+\echo '2. Run export-chat-data.sh on your current server'  
+\echo '3. Transfer the exported files to this server'
+\echo '4. Run import-chat-data.sh on this server'

inventory-server/chat/db-convert/mongo_to_postgres_converter.py

@@ -0,0 +1,881 @@
+#!/usr/bin/env python3
+"""
+MongoDB to PostgreSQL Converter for Rocket.Chat
+Converts MongoDB BSON export files to PostgreSQL database
+
+Usage:
+    python3 mongo_to_postgres_converter.py \
+        --mongo-path db/database/62df06d44234d20001289144 \
+        --pg-database rocketchat_converted \
+        --pg-user rocketchat_user \
+        --pg-password your_password \
+        --debug
+"""
+
+import json
+import os
+import re
+import subprocess
+import sys
+import struct
+from datetime import datetime
+from pathlib import Path
+from typing import Dict, Any, List, Optional
+import argparse
+import traceback
+
+# Auto-install dependencies if needed
+try:
+    import bson
+    import psycopg2
+except ImportError:
+    print("Installing required packages...")
+    subprocess.check_call([sys.executable, "-m", "pip", "install", "pymongo", "psycopg2-binary"])
+    import bson
+    import psycopg2
+
+class MongoToPostgresConverter:
+    def __init__(self, mongo_db_path: str, postgres_config: Dict[str, str], debug_mode: bool = False, debug_collections: List[str] = None):
+        self.mongo_db_path = Path(mongo_db_path)
+        self.postgres_config = postgres_config
+        self.debug_mode = debug_mode
+        self.debug_collections = debug_collections or []
+        self.collections = {}
+        self.schema_info = {}
+        self.error_log = {}
+        
+    def log_debug(self, message: str, collection: str = None):
+        """Log debug messages if debug mode is enabled and collection is in debug list"""
+        if self.debug_mode and (not self.debug_collections or collection in self.debug_collections):
+            print(f"DEBUG: {message}")
+            
+    def log_error(self, collection: str, error_type: str, details: str):
+        """Log detailed error information"""
+        if collection not in self.error_log:
+            self.error_log[collection] = []
+        self.error_log[collection].append({
+            'type': error_type,
+            'details': details,
+            'timestamp': datetime.now().isoformat()
+        })
+        
+    def sample_documents(self, collection_name: str, max_samples: int = 3) -> List[Dict]:
+        """Sample documents from a collection for debugging"""
+        if not self.debug_mode or (self.debug_collections and collection_name not in self.debug_collections):
+            return []
+            
+        print(f"\n🔍 Sampling documents from {collection_name}:")
+        
+        bson_file = self.collections[collection_name]['bson_file']
+        if bson_file.stat().st_size == 0:
+            print("  Collection is empty")
+            return []
+            
+        samples = []
+        
+        try:
+            with open(bson_file, 'rb') as f:
+                sample_count = 0
+                while sample_count < max_samples:
+                    try:
+                        doc_size = int.from_bytes(f.read(4), byteorder='little')
+                        if doc_size <= 0:
+                            break
+                        f.seek(-4, 1)
+                        doc_bytes = f.read(doc_size)
+                        if len(doc_bytes) != doc_size:
+                            break
+                            
+                        doc = bson.decode(doc_bytes)
+                        samples.append(doc)
+                        sample_count += 1
+                        
+                        print(f"  Sample {sample_count} - Keys: {list(doc.keys())}")
+                        # Show a few key fields with their types and truncated values
+                        for key, value in list(doc.items())[:3]:
+                            value_preview = str(value)[:50] + "..." if len(str(value)) > 50 else str(value)
+                            print(f"    {key}: {type(value).__name__} = {value_preview}")
+                        if len(doc) > 3:
+                            print(f"    ... and {len(doc) - 3} more fields")
+                        print()
+                        
+                    except (bson.InvalidBSON, struct.error, OSError) as e:
+                        self.log_error(collection_name, 'document_parsing', str(e))
+                        break
+                        
+        except Exception as e:
+            self.log_error(collection_name, 'file_reading', str(e))
+            print(f"  Error reading collection: {e}")
+            
+        return samples
+
+    def discover_collections(self):
+        """Discover all BSON files and their metadata"""
+        print("Discovering MongoDB collections...")
+        
+        for bson_file in self.mongo_db_path.glob("*.bson"):
+            collection_name = bson_file.stem
+            metadata_file = bson_file.with_suffix(".metadata.json")
+            
+            # Read metadata if available
+            metadata = {}
+            if metadata_file.exists():
+                try:
+                    with open(metadata_file, 'r', encoding='utf-8') as f:
+                        metadata = json.load(f)
+                except (UnicodeDecodeError, json.JSONDecodeError) as e:
+                    print(f"Warning: Could not read metadata for {collection_name}: {e}")
+                    metadata = {}
+            
+            # Get file size and document count estimate
+            file_size = bson_file.stat().st_size
+            doc_count = self._estimate_document_count(bson_file)
+            
+            self.collections[collection_name] = {
+                'bson_file': bson_file,
+                'metadata': metadata,
+                'file_size': file_size,
+                'estimated_docs': doc_count
+            }
+            
+        print(f"Found {len(self.collections)} collections")
+        for name, info in self.collections.items():
+            print(f"  - {name}: {info['file_size']/1024/1024:.1f}MB (~{info['estimated_docs']} docs)")
+    
+    def _estimate_document_count(self, bson_file: Path) -> int:
+        """Estimate document count by reading first few documents"""
+        if bson_file.stat().st_size == 0:
+            return 0
+            
+        try:
+            with open(bson_file, 'rb') as f:
+                docs_sampled = 0
+                bytes_sampled = 0
+                max_sample_size = min(1024 * 1024, bson_file.stat().st_size)  # 1MB or file size
+                
+                while bytes_sampled < max_sample_size:
+                    try:
+                        doc_size = int.from_bytes(f.read(4), byteorder='little')
+                        if doc_size <= 0 or doc_size > 16 * 1024 * 1024:  # MongoDB doc size limit
+                            break
+                        f.seek(-4, 1)  # Go back
+                        doc_bytes = f.read(doc_size)
+                        if len(doc_bytes) != doc_size:
+                            break
+                        bson.decode(doc_bytes)  # Validate it's a valid BSON document
+                        docs_sampled += 1
+                        bytes_sampled += doc_size
+                    except (bson.InvalidBSON, struct.error, OSError):
+                        break
+                
+                if docs_sampled > 0 and bytes_sampled > 0:
+                    avg_doc_size = bytes_sampled / docs_sampled
+                    return int(bson_file.stat().st_size / avg_doc_size)
+                    
+        except Exception:
+            pass
+            
+        return 0
+    
+    def analyze_schema(self, collection_name: str, sample_size: int = 100) -> Dict[str, Any]:
+        """Analyze collection schema by sampling documents"""
+        print(f"Analyzing schema for {collection_name}...")
+        
+        bson_file = self.collections[collection_name]['bson_file']
+        if bson_file.stat().st_size == 0:
+            return {}
+            
+        schema = {}
+        docs_analyzed = 0
+        
+        try:
+            with open(bson_file, 'rb') as f:
+                while docs_analyzed < sample_size:
+                    try:
+                        doc_size = int.from_bytes(f.read(4), byteorder='little')
+                        if doc_size <= 0:
+                            break
+                        f.seek(-4, 1)
+                        doc_bytes = f.read(doc_size)
+                        if len(doc_bytes) != doc_size:
+                            break
+                            
+                        doc = bson.decode(doc_bytes)
+                        self._analyze_document_schema(doc, schema)
+                        docs_analyzed += 1
+                        
+                    except (bson.InvalidBSON, struct.error, OSError):
+                        break
+                        
+        except Exception as e:
+            print(f"Error analyzing {collection_name}: {e}")
+            
+        self.schema_info[collection_name] = schema
+        return schema
+    
+    def _analyze_document_schema(self, doc: Dict[str, Any], schema: Dict[str, Any], prefix: str = ""):
+        """Recursively analyze document structure"""
+        for key, value in doc.items():
+            full_key = f"{prefix}.{key}" if prefix else key
+            
+            if full_key not in schema:
+                schema[full_key] = {
+                    'types': set(),
+                    'null_count': 0,
+                    'total_count': 0,
+                    'is_array': False,
+                    'nested_schema': {}
+                }
+            
+            schema[full_key]['total_count'] += 1
+            
+            if value is None:
+                schema[full_key]['null_count'] += 1
+                schema[full_key]['types'].add('null')
+            elif isinstance(value, dict):
+                schema[full_key]['types'].add('object')
+                if 'nested_schema' not in schema[full_key]:
+                    schema[full_key]['nested_schema'] = {}
+                self._analyze_document_schema(value, schema[full_key]['nested_schema'])
+            elif isinstance(value, list):
+                schema[full_key]['types'].add('array')
+                schema[full_key]['is_array'] = True
+                if value and isinstance(value[0], dict):
+                    if 'array_item_schema' not in schema[full_key]:
+                        schema[full_key]['array_item_schema'] = {}
+                    for item in value[:5]:  # Sample first 5 items
+                        if isinstance(item, dict):
+                            self._analyze_document_schema(item, schema[full_key]['array_item_schema'])
+            else:
+                schema[full_key]['types'].add(type(value).__name__)
+    
+    def generate_postgres_schema(self) -> Dict[str, str]:
+        """Generate PostgreSQL CREATE TABLE statements"""
+        print("Generating PostgreSQL schema...")
+        
+        table_definitions = {}
+        
+        for collection_name, schema in self.schema_info.items():
+            if not schema:  # Empty collection
+                continue
+                
+            table_name = self._sanitize_table_name(collection_name)
+            columns = []
+            
+            # Always add an id column (PostgreSQL doesn't use _id like MongoDB)
+            columns.append("id SERIAL PRIMARY KEY")
+            
+            for field_name, field_info in schema.items():
+                if field_name == '_id':
+                    columns.append("mongo_id TEXT")  # Always allow NULL for mongo_id
+                    continue
+                    
+                col_name = self._sanitize_column_name(field_name)
+                
+                # Handle conflicts with PostgreSQL auto-generated columns
+                if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
+                    col_name = f"field_{col_name}"
+                
+                col_type = self._determine_postgres_type(field_info)
+                
+                # Make all fields nullable by default to avoid constraint violations
+                columns.append(f"{col_name} {col_type}")
+            
+            # Add metadata columns
+            columns.extend([
+                "created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP",
+                "updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP"
+            ])
+            
+            column_definitions = ',\n    '.join(columns)
+            table_sql = f"""
+CREATE TABLE IF NOT EXISTS {table_name} (
+    {column_definitions}
+);
+
+-- Create indexes based on MongoDB indexes
+"""
+            
+            # Get list of actual columns that will exist in the table
+            existing_columns = set(['id', 'mongo_id', 'created_at', 'updated_at'])
+            for field_name in schema.keys():
+                if field_name != '_id':
+                    col_name = self._sanitize_column_name(field_name)
+                    # Handle conflicts with PostgreSQL auto-generated columns
+                    if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
+                        col_name = f"field_{col_name}"
+                    existing_columns.add(col_name)
+            
+            # Add indexes from MongoDB metadata
+            metadata = self.collections[collection_name].get('metadata', {})
+            indexes = metadata.get('indexes', [])
+            
+            for index in indexes:
+                if index['name'] != '_id_':  # Skip the default _id index
+                    # Sanitize index name - remove special characters
+                    sanitized_index_name = re.sub(r'[^a-zA-Z0-9_]', '_', index['name'])
+                    index_name = f"idx_{table_name}_{sanitized_index_name}"
+                    index_keys = list(index['key'].keys())
+                    if index_keys:
+                        sanitized_keys = []
+                        for key in index_keys:
+                            if key != '_id':
+                                sanitized_key = self._sanitize_column_name(key)
+                                # Handle conflicts with PostgreSQL auto-generated columns
+                                if sanitized_key in ['id', 'mongo_id', 'created_at', 'updated_at']:
+                                    sanitized_key = f"field_{sanitized_key}"
+                                # Only add if the column actually exists in our table
+                                if sanitized_key in existing_columns:
+                                    sanitized_keys.append(sanitized_key)
+                        
+                        if sanitized_keys:
+                            table_sql += f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name} ({', '.join(sanitized_keys)});\n"
+            
+            table_definitions[collection_name] = table_sql
+            
+        return table_definitions
+    
+    def _sanitize_table_name(self, name: str) -> str:
+        """Convert MongoDB collection name to PostgreSQL table name"""
+        # Remove rocketchat_ prefix if present
+        if name.startswith('rocketchat_'):
+            name = name[11:]
+        
+        # Replace special characters with underscores
+        name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
+        
+        # Ensure it starts with a letter
+        if name and name[0].isdigit():
+            name = 'table_' + name
+            
+        return name.lower()
+    
+    def _sanitize_column_name(self, name: str) -> str:
+        """Convert MongoDB field name to PostgreSQL column name"""
+        # Handle nested field names (convert dots to underscores)
+        name = name.replace('.', '_')
+        
+        # Replace special characters with underscores
+        name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
+        
+        # Ensure it starts with a letter or underscore
+        if name and name[0].isdigit():
+            name = 'col_' + name
+            
+        # Handle PostgreSQL reserved words
+        reserved = {
+            'user', 'order', 'group', 'table', 'index', 'key', 'value', 'date', 'time', 'timestamp',
+            'default', 'select', 'from', 'where', 'insert', 'update', 'delete', 'create', 'drop',
+            'alter', 'grant', 'revoke', 'commit', 'rollback', 'begin', 'end', 'case', 'when',
+            'then', 'else', 'if', 'null', 'not', 'and', 'or', 'in', 'exists', 'between',
+            'like', 'limit', 'offset', 'union', 'join', 'inner', 'outer', 'left', 'right',
+            'full', 'cross', 'natural', 'on', 'using', 'distinct', 'all', 'any', 'some',
+            'desc', 'asc', 'primary', 'foreign', 'references', 'constraint', 'unique',
+            'check', 'cascade', 'restrict', 'action', 'match', 'partial', 'full'
+        }
+        if name.lower() in reserved:
+            name = name + '_col'
+            
+        return name.lower()
+    
+    def _determine_postgres_type(self, field_info: Dict[str, Any]) -> str:
+        """Determine PostgreSQL column type from MongoDB field analysis with improved logic"""
+        types = field_info['types']
+        
+        # Convert set to list for easier checking
+        type_list = list(types)
+        
+        # If there's only one type (excluding null), use specific typing
+        non_null_types = [t for t in type_list if t != 'null']
+        
+        if len(non_null_types) == 1:
+            single_type = non_null_types[0]
+            if single_type == 'bool':
+                return 'BOOLEAN'
+            elif single_type == 'int':
+                return 'INTEGER'
+            elif single_type == 'float':
+                return 'NUMERIC'
+            elif single_type == 'str':
+                return 'TEXT'
+            elif single_type == 'datetime':
+                return 'TIMESTAMP'
+            elif single_type == 'ObjectId':
+                return 'TEXT'
+        
+        # Handle mixed types more conservatively
+        if 'array' in types or field_info.get('is_array', False):
+            return 'JSONB'  # Arrays always go to JSONB
+        elif 'object' in types:
+            return 'JSONB'  # Objects always go to JSONB
+        elif len(non_null_types) > 1:
+            # Multiple non-null types - check for common combinations
+            if set(non_null_types) <= {'int', 'float'}:
+                return 'NUMERIC'  # Can handle both int and float
+            elif set(non_null_types) <= {'bool', 'str'}:
+                return 'TEXT'  # Convert everything to text
+            elif set(non_null_types) <= {'str', 'ObjectId'}:
+                return 'TEXT'  # Both are string-like
+            else:
+                return 'JSONB'  # Complex mixed types go to JSONB
+        elif 'ObjectId' in types:
+            return 'TEXT'
+        elif 'datetime' in types:
+            return 'TIMESTAMP'
+        elif 'bool' in types:
+            return 'BOOLEAN'
+        elif 'int' in types:
+            return 'INTEGER'
+        elif 'float' in types:
+            return 'NUMERIC'
+        elif 'str' in types:
+            return 'TEXT'
+        else:
+            return 'TEXT'  # Default fallback
+    
+    def create_postgres_database(self, table_definitions: Dict[str, str]):
+        """Create PostgreSQL database and tables"""
+        print("Creating PostgreSQL database schema...")
+        
+        try:
+            # Connect to PostgreSQL
+            conn = psycopg2.connect(**self.postgres_config)
+            conn.autocommit = True
+            cursor = conn.cursor()
+            
+            # Create tables
+            for collection_name, table_sql in table_definitions.items():
+                print(f"Creating table for {collection_name}...")
+                cursor.execute(table_sql)
+            
+            cursor.close()
+            conn.close()
+            print("Database schema created successfully!")
+            
+        except Exception as e:
+            print(f"Error creating database schema: {e}")
+            raise
+    
+    def convert_and_insert_data(self, batch_size: int = 1000):
+        """Convert BSON data and insert into PostgreSQL"""
+        print("Converting and inserting data...")
+        
+        try:
+            conn = psycopg2.connect(**self.postgres_config)
+            conn.autocommit = False
+            
+            for collection_name in self.collections:
+                print(f"Processing {collection_name}...")
+                self._convert_collection(conn, collection_name, batch_size)
+            
+            conn.close()
+            print("Data conversion completed successfully!")
+            
+        except Exception as e:
+            print(f"Error converting data: {e}")
+            raise
+    
+    def _convert_collection(self, conn, collection_name: str, batch_size: int):
+        """Convert a single collection"""
+        bson_file = self.collections[collection_name]['bson_file']
+        
+        if bson_file.stat().st_size == 0:
+            print(f"  Skipping empty collection {collection_name}")
+            return
+            
+        table_name = self._sanitize_table_name(collection_name)
+        cursor = conn.cursor()
+        
+        batch = []
+        total_inserted = 0
+        errors = 0
+        
+        try:
+            with open(bson_file, 'rb') as f:
+                while True:
+                    try:
+                        doc_size = int.from_bytes(f.read(4), byteorder='little')
+                        if doc_size <= 0:
+                            break
+                        f.seek(-4, 1)
+                        doc_bytes = f.read(doc_size)
+                        if len(doc_bytes) != doc_size:
+                            break
+                            
+                        doc = bson.decode(doc_bytes)
+                        batch.append(doc)
+                        
+                        if len(batch) >= batch_size:
+                            inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
+                            total_inserted += inserted
+                            errors += batch_errors
+                            batch = []
+                            conn.commit()
+                            if total_inserted % 5000 == 0:  # Less frequent progress updates
+                                print(f"  Inserted {total_inserted} documents...")
+                            
+                    except (bson.InvalidBSON, struct.error, OSError):
+                        break
+                
+                # Insert remaining documents
+                if batch:
+                    inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
+                    total_inserted += inserted
+                    errors += batch_errors
+                    conn.commit()
+                    
+            if errors > 0:
+                print(f"  Completed {collection_name}: {total_inserted} documents inserted ({errors} errors)")
+            else:
+                print(f"  Completed {collection_name}: {total_inserted} documents inserted")
+            
+        except Exception as e:
+            print(f"  Error processing {collection_name}: {e}")
+            conn.rollback()
+        finally:
+            cursor.close()
+    
+    def _insert_batch(self, cursor, table_name: str, documents: List[Dict], collection_name: str):
+        """Insert a batch of documents with proper transaction handling"""
+        if not documents:
+            return 0, 0
+            
+        # Get schema info for this collection
+        schema = self.schema_info.get(collection_name, {})
+        
+        # Build column list
+        columns = ['mongo_id']
+        for field_name in schema.keys():
+            if field_name != '_id':
+                col_name = self._sanitize_column_name(field_name)
+                # Handle conflicts with PostgreSQL auto-generated columns
+                if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
+                    col_name = f"field_{col_name}"
+                columns.append(col_name)
+        
+        # Build INSERT statement
+        placeholders = ', '.join(['%s'] * len(columns))
+        sql = f"INSERT INTO {table_name} ({', '.join(columns)}) VALUES ({placeholders})"
+        
+        self.log_debug(f"SQL: {sql}", collection_name)
+        
+        # Convert documents to tuples
+        rows = []
+        errors = 0
+        
+        for doc_idx, doc in enumerate(documents):
+            try:
+                row = []
+                
+                # Add mongo_id
+                row.append(str(doc.get('_id', '')))
+                
+                # Add other fields
+                for field_name in schema.keys():
+                    if field_name != '_id':
+                        try:
+                            value = self._get_nested_value(doc, field_name)
+                            converted_value = self._convert_value_for_postgres(value, field_name, schema)
+                            row.append(converted_value)
+                        except Exception as e:
+                            self.log_error(collection_name, 'field_conversion', 
+                                         f"Field '{field_name}' in doc {doc_idx}: {str(e)}")
+                            # Only show debug for collections we're focusing on
+                            if collection_name in self.debug_collections:
+                                print(f"    ⚠️  Error converting field '{field_name}': {e}")
+                            row.append(None)  # Use NULL for problematic fields
+                
+                rows.append(tuple(row))
+                
+            except Exception as e:
+                self.log_error(collection_name, 'document_conversion', f"Document {doc_idx}: {str(e)}")
+                errors += 1
+                continue
+        
+        # Execute batch insert
+        if rows:
+            try:
+                cursor.executemany(sql, rows)
+                return len(rows), errors
+            except Exception as batch_error:
+                self.log_error(collection_name, 'batch_insert', str(batch_error))
+                
+                # Only show detailed debugging for targeted collections
+                if collection_name in self.debug_collections:
+                    print(f"    🔴 Batch insert failed for {collection_name}: {batch_error}")
+                    print("    Trying individual inserts with rollback handling...")
+                
+                # Rollback the failed transaction
+                cursor.connection.rollback()
+                
+                # Try inserting one by one in individual transactions
+                success_count = 0
+                for row_idx, row in enumerate(rows):
+                    try:
+                        cursor.execute(sql, row)
+                        cursor.connection.commit()  # Commit each successful insert
+                        success_count += 1
+                    except Exception as row_error:
+                        cursor.connection.rollback()  # Rollback failed insert
+                        self.log_error(collection_name, 'row_insert', f"Row {row_idx}: {str(row_error)}")
+                        
+                        # Show detailed error only for the first few failures and only for targeted collections
+                        if collection_name in self.debug_collections and errors < 3:
+                            print(f"      Row {row_idx} failed: {row_error}")
+                            print(f"      Row data: {len(row)} values, expected {len(columns)} columns")
+                            
+                        errors += 1
+                        continue
+                return success_count, errors
+        
+        return 0, errors
+    
+    def _get_nested_value(self, doc: Dict, field_path: str):
+        """Get value from nested document using dot notation"""
+        keys = field_path.split('.')
+        value = doc
+        
+        for key in keys:
+            if isinstance(value, dict) and key in value:
+                value = value[key]
+            else:
+                return None
+                
+        return value
+    
+    def _convert_value_for_postgres(self, value, field_name: str = None, schema: Dict = None):
+        """Convert MongoDB value to PostgreSQL compatible value with schema-aware conversion"""
+        if value is None:
+            return None
+        
+        # Get the expected PostgreSQL type for this field if available
+        expected_type = None
+        if schema and field_name and field_name in schema:
+            field_info = schema[field_name]
+            expected_type = self._determine_postgres_type(field_info)
+        
+        # Handle conversion based on expected type
+        if expected_type == 'BOOLEAN':
+            if isinstance(value, bool):
+                return value
+            elif isinstance(value, str):
+                return value.lower() in ('true', '1', 'yes', 'on')
+            elif isinstance(value, (int, float)):
+                return bool(value)
+            else:
+                return None
+        elif expected_type == 'INTEGER':
+            if isinstance(value, int):
+                return value
+            elif isinstance(value, float):
+                return int(value)
+            elif isinstance(value, str) and value.isdigit():
+                return int(value)
+            elif isinstance(value, bool):
+                return int(value)
+            else:
+                return None
+        elif expected_type == 'NUMERIC':
+            if isinstance(value, (int, float)):
+                return value
+            elif isinstance(value, str):
+                try:
+                    return float(value)
+                except ValueError:
+                    return None
+            elif isinstance(value, bool):
+                return float(value)
+            else:
+                return None
+        elif expected_type == 'TEXT':
+            if isinstance(value, str):
+                return value
+            elif value is not None:
+                str_value = str(value)
+                # Handle very long strings
+                if len(str_value) > 65535:
+                    return str_value[:65535]
+                return str_value
+            else:
+                return None
+        elif expected_type == 'TIMESTAMP':
+            if hasattr(value, 'isoformat'):
+                return value.isoformat()
+            elif isinstance(value, str):
+                return value
+            else:
+                return str(value) if value is not None else None
+        elif expected_type == 'JSONB':
+            if isinstance(value, (dict, list)):
+                return json.dumps(value, default=self._json_serializer)
+            elif isinstance(value, str):
+                # Check if it's already valid JSON
+                try:
+                    json.loads(value)
+                    return value
+                except (json.JSONDecodeError, TypeError):
+                    # Not valid JSON, wrap it
+                    return json.dumps(value)
+            else:
+                return json.dumps(value, default=self._json_serializer)
+        
+        # Fallback to original logic if no expected type or type not recognized
+        if isinstance(value, bool):
+            return value
+        elif isinstance(value, (int, float)):
+            return value
+        elif isinstance(value, str):
+            return value
+        elif isinstance(value, (dict, list)):
+            return json.dumps(value, default=self._json_serializer)
+        elif hasattr(value, 'isoformat'):  # datetime
+            return value.isoformat()
+        elif hasattr(value, '__str__'):
+            str_value = str(value)
+            if len(str_value) > 65535:
+                return str_value[:65535]
+            return str_value
+        else:
+            return str(value)
+    
+    def _json_serializer(self, obj):
+        """Custom JSON serializer for complex objects with better error handling"""
+        try:
+            if hasattr(obj, 'isoformat'):  # datetime
+                return obj.isoformat()
+            elif hasattr(obj, '__str__'):
+                return str(obj)
+            else:
+                return None
+        except Exception as e:
+            self.log_debug(f"JSON serialization error: {e}")
+            return str(obj)
+    
+    def run_conversion(self, sample_size: int = 100, batch_size: int = 1000):
+        """Run the full conversion process with focused debugging"""
+        print("Starting MongoDB to PostgreSQL conversion...")
+        print("This will convert your Rocket.Chat database from MongoDB to PostgreSQL")
+        if self.debug_mode:
+            if self.debug_collections:
+                print(f"🐛 DEBUG MODE: Focusing on collections: {', '.join(self.debug_collections)}")
+            else:
+                print("🐛 DEBUG MODE: All collections")
+        print("=" * 70)
+        
+        # Step 1: Discover collections
+        self.discover_collections()
+        
+        # Step 2: Analyze schemas
+        print("\nAnalyzing collection schemas...")
+        for collection_name in self.collections:
+            self.analyze_schema(collection_name, sample_size)
+        
+        # Sample problematic collections if debugging
+        if self.debug_mode and self.debug_collections:
+            for coll in self.debug_collections:
+                if coll in self.collections:
+                    self.sample_documents(coll, 2)
+        
+        # Step 3: Generate PostgreSQL schema
+        table_definitions = self.generate_postgres_schema()
+        
+        # Step 4: Create database schema
+        self.create_postgres_database(table_definitions)
+        
+        # Step 5: Convert and insert data
+        self.convert_and_insert_data(batch_size)
+        
+        # Step 6: Show error summary
+        self._print_error_summary()
+        
+        print("=" * 70)
+        print("✅ Conversion completed!")
+        print(f"   Database: {self.postgres_config['database']}")
+        print(f"   Tables created: {len(table_definitions)}")
+        
+    def _print_error_summary(self):
+        """Print a focused summary of errors"""
+        if not self.error_log:
+            print("\n✅ No errors encountered during conversion!")
+            return
+            
+        print("\n⚠️  ERROR SUMMARY:")
+        print("=" * 50)
+        
+        # Sort by error count descending
+        sorted_collections = sorted(self.error_log.items(), 
+                                  key=lambda x: len(x[1]), reverse=True)
+        
+        for collection, errors in sorted_collections:
+            error_types = {}
+            for error in errors:
+                error_type = error['type']
+                if error_type not in error_types:
+                    error_types[error_type] = []
+                error_types[error_type].append(error['details'])
+            
+            print(f"\n🔴 {collection} ({len(errors)} total errors):")
+            for error_type, details_list in error_types.items():
+                print(f"   {error_type}: {len(details_list)} errors")
+                
+                # Show sample errors for critical collections
+                if collection in ['rocketchat_settings', 'rocketchat_room'] and len(details_list) > 0:
+                    print(f"     Sample: {details_list[0][:100]}...")
+
+def main():
+    parser = argparse.ArgumentParser(
+        description='Convert MongoDB BSON export to PostgreSQL',
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Basic usage
+  python3 mongo_to_postgres_converter.py \\
+    --mongo-path db/database/62df06d44234d20001289144 \\
+    --pg-database rocketchat_converted \\
+    --pg-user rocketchat_user \\
+    --pg-password mypassword
+
+  # Debug specific failing collections
+  python3 mongo_to_postgres_converter.py \\
+    --mongo-path db/database/62df06d44234d20001289144 \\
+    --pg-database rocketchat_converted \\
+    --pg-user rocketchat_user \\
+    --pg-password mypassword \\
+    --debug-collections rocketchat_settings rocketchat_room
+
+Before running this script:
+1. Run: sudo -u postgres psql -f reset_database.sql
+2. Update the password in reset_database.sql
+        """
+    )
+    
+    parser.add_argument('--mongo-path', required=True, help='Path to MongoDB export directory')
+    parser.add_argument('--pg-host', default='localhost', help='PostgreSQL host (default: localhost)')
+    parser.add_argument('--pg-port', default='5432', help='PostgreSQL port (default: 5432)')
+    parser.add_argument('--pg-database', required=True, help='PostgreSQL database name')
+    parser.add_argument('--pg-user', required=True, help='PostgreSQL username')
+    parser.add_argument('--pg-password', required=True, help='PostgreSQL password')
+    parser.add_argument('--sample-size', type=int, default=100, help='Number of documents to sample for schema analysis (default: 100)')
+    parser.add_argument('--batch-size', type=int, default=1000, help='Batch size for data insertion (default: 1000)')
+    parser.add_argument('--debug', action='store_true', help='Enable debug mode with detailed error logging')
+    parser.add_argument('--debug-collections', nargs='*', help='Specific collections to debug (e.g., rocketchat_settings rocketchat_room)')
+    
+    args = parser.parse_args()
+    
+    postgres_config = {
+        'host': args.pg_host,
+        'port': args.pg_port,
+        'database': args.pg_database,
+        'user': args.pg_user,
+        'password': args.pg_password
+    }
+    
+    # Enable debug mode if debug collections are specified
+    debug_mode = args.debug or (args.debug_collections is not None)
+    
+    converter = MongoToPostgresConverter(args.mongo_path, postgres_config, debug_mode, args.debug_collections)
+    converter.run_conversion(args.sample_size, args.batch_size)
+
+
+if __name__ == '__main__':
+    main() 

inventory-server/chat/db-convert/reset_database.sql

@@ -0,0 +1,41 @@
+-- PostgreSQL Database Reset Script for Rocket.Chat Import
+-- Run as: sudo -u postgres psql -f reset_database.sql
+
+-- Terminate all connections to the database (force disconnect users)
+SELECT pg_terminate_backend(pid)
+FROM pg_stat_activity
+WHERE datname = 'rocketchat_converted' AND pid <> pg_backend_pid();
+
+-- Drop the database if it exists
+DROP DATABASE IF EXISTS rocketchat_converted;
+
+-- Create fresh database
+CREATE DATABASE rocketchat_converted;
+
+-- Create user (if not exists)
+DO $$ 
+BEGIN
+    IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'rocketchat_user') THEN
+        CREATE USER rocketchat_user WITH PASSWORD 'HKjLgt23gWuPXzEAn3rW';
+    END IF;
+END $$;
+
+-- Grant database privileges
+GRANT CONNECT ON DATABASE rocketchat_converted TO rocketchat_user;
+GRANT CREATE ON DATABASE rocketchat_converted TO rocketchat_user;
+
+-- Connect to the new database
+\c rocketchat_converted;
+
+-- Grant schema privileges
+GRANT CREATE ON SCHEMA public TO rocketchat_user;
+GRANT USAGE ON SCHEMA public TO rocketchat_user;
+
+-- Grant privileges on all future tables and sequences
+ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO rocketchat_user;
+ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO rocketchat_user;
+
+-- Display success message
+\echo 'Database reset completed successfully!'
+\echo 'You can now run the converter with:'
+\echo 'python3 mongo_to_postgres_converter.py --mongo-path db/database/62df06d44234d20001289144 --pg-database rocketchat_converted --pg-user rocketchat_user --pg-password your_password' 

inventory-server/chat/db-convert/test_converter.py

@@ -0,0 +1,54 @@
+#!/usr/bin/env python3
+"""
+Quick test script to verify the converter fixes work for problematic collections
+"""
+
+from mongo_to_postgres_converter import MongoToPostgresConverter
+
+def test_problematic_collections():
+    print("🧪 Testing converter fixes for problematic collections...")
+    
+    postgres_config = {
+        'host': 'localhost',
+        'port': '5432',
+        'database': 'rocketchat_test',
+        'user': 'rocketchat_user',
+        'password': 'password123'
+    }
+    
+    converter = MongoToPostgresConverter(
+        'db/database/62df06d44234d20001289144',
+        postgres_config,
+        debug_mode=True,
+        debug_collections=['rocketchat_settings', 'rocketchat_room']
+    )
+    
+    # Test just discovery and schema analysis
+    print("\n1. Testing collection discovery...")
+    converter.discover_collections()
+    
+    print("\n2. Testing schema analysis...")
+    if 'rocketchat_settings' in converter.collections:
+        settings_schema = converter.analyze_schema('rocketchat_settings', 10)
+        print(f"Settings schema fields: {len(settings_schema)}")
+        
+        # Check specific problematic fields
+        if 'packageValue' in settings_schema:
+            packagevalue_info = settings_schema['packageValue']
+            pg_type = converter._determine_postgres_type(packagevalue_info)
+            print(f"packageValue types: {packagevalue_info['types']} -> PostgreSQL: {pg_type}")
+            
+    if 'rocketchat_room' in converter.collections:
+        room_schema = converter.analyze_schema('rocketchat_room', 10)
+        print(f"Room schema fields: {len(room_schema)}")
+        
+        # Check specific problematic fields
+        if 'sysMes' in room_schema:
+            sysmes_info = room_schema['sysMes']
+            pg_type = converter._determine_postgres_type(sysmes_info)
+            print(f"sysMes types: {sysmes_info['types']} -> PostgreSQL: {pg_type}")
+    
+    print("\n✅ Test completed - check the type mappings above!")
+
+if __name__ == '__main__':
+    test_problematic_collections() 

inventory-server/chat/export-chat-data.sh

@@ -0,0 +1,147 @@
+#!/bin/bash
+
+# Chat Database Export Script
+# This script exports the chat database schema and data for migration
+
+set -e  # Exit on any error
+
+echo "🚀 Starting chat database export..."
+
+# Configuration - Update these values for your setup
+DB_HOST="${CHAT_DB_HOST:-localhost}"
+DB_PORT="${CHAT_DB_PORT:-5432}"
+DB_NAME="${CHAT_DB_NAME:-rocketchat_converted}"
+DB_USER="${CHAT_DB_USER:-rocketchat_user}"
+
+# Check if database connection info is available
+if [ -z "$CHAT_DB_PASSWORD" ]; then
+    echo "⚠️  CHAT_DB_PASSWORD environment variable not set"
+    echo "Please set it with: export CHAT_DB_PASSWORD='your_password'"
+    exit 1
+fi
+
+echo "📊 Database: $DB_NAME on $DB_HOST:$DB_PORT"
+
+# Create export directory
+EXPORT_DIR="chat-migration-$(date +%Y%m%d-%H%M%S)"
+mkdir -p "$EXPORT_DIR"
+
+echo "📁 Export directory: $EXPORT_DIR"
+
+# Export database schema
+echo "📋 Exporting database schema..."
+PGPASSWORD="$CHAT_DB_PASSWORD" pg_dump \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    --schema-only \
+    --no-owner \
+    --no-privileges \
+    -f "$EXPORT_DIR/chat-schema.sql"
+
+if [ $? -eq 0 ]; then
+    echo "✅ Schema exported successfully"
+else
+    echo "❌ Schema export failed"
+    exit 1
+fi
+
+# Export database data
+echo "💾 Exporting database data..."
+PGPASSWORD="$CHAT_DB_PASSWORD" pg_dump \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    --data-only \
+    --no-owner \
+    --no-privileges \
+    --disable-triggers \
+    --column-inserts \
+    -f "$EXPORT_DIR/chat-data.sql"
+
+if [ $? -eq 0 ]; then
+    echo "✅ Data exported successfully"
+else
+    echo "❌ Data export failed"
+    exit 1
+fi
+
+# Export file uploads and avatars
+echo "📎 Exporting chat files (uploads and avatars)..."
+if [ -d "db-convert/db/files" ]; then
+    cd db-convert/db
+    tar -czf "../../$EXPORT_DIR/chat-files.tar.gz" files/
+    cd ../..
+    echo "✅ Files exported successfully"
+else
+    echo "⚠️  No files directory found at db-convert/db/files"
+    echo "   This is normal if you have no file uploads"
+    touch "$EXPORT_DIR/chat-files.tar.gz"
+fi
+
+# Get table statistics for verification
+echo "📈 Generating export statistics..."
+PGPASSWORD="$CHAT_DB_PASSWORD" psql \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    -c "
+    SELECT 
+        schemaname,
+        tablename, 
+        n_tup_ins as inserted_rows,
+        n_tup_upd as updated_rows,
+        n_tup_del as deleted_rows,
+        n_live_tup as live_rows,
+        n_dead_tup as dead_rows
+    FROM pg_stat_user_tables 
+    ORDER BY n_live_tup DESC;
+    " > "$EXPORT_DIR/table-stats.txt"
+
+# Create export summary
+cat > "$EXPORT_DIR/export-summary.txt" << EOF
+Chat Database Export Summary
+===========================
+
+Export Date: $(date)
+Database: $DB_NAME
+Host: $DB_HOST:$DB_PORT
+User: $DB_USER
+
+Files Generated:
+- chat-schema.sql: Database schema (tables, indexes, constraints)
+- chat-data.sql: All table data
+- chat-files.tar.gz: Uploaded files and avatars
+- table-stats.txt: Database statistics
+- export-summary.txt: This summary
+
+Next Steps:
+1. Transfer these files to your new server
+2. Run create-new-database.sql on the new server first
+3. Run import-chat-data.sh on the new server
+4. Update your application configuration
+5. Run verify-migration.js to validate the migration
+
+Important Notes:
+- Keep these files secure as they contain your chat data
+- Ensure the new server has enough disk space
+- Plan for application downtime during the migration
+EOF
+
+echo ""
+echo "🎉 Export completed successfully!"
+echo "📁 Files are in: $EXPORT_DIR/"
+echo ""
+echo "📋 Export Summary:"
+ls -lh "$EXPORT_DIR/"
+echo ""
+echo "🚚 Next steps:"
+echo "1. Transfer the $EXPORT_DIR/ directory to your new server"
+echo "2. Run create-new-database.sql on the new server (update password first!)"
+echo "3. Run import-chat-data.sh on the new server"
+echo ""
+echo "💡 To transfer files to new server:"
+echo "   scp -r $EXPORT_DIR/ user@new-server:/tmp/"

inventory-server/chat/import-chat-data.sh

@@ -0,0 +1,167 @@
+#!/bin/bash
+
+# Chat Database Import Script
+# This script imports the chat database schema and data on the new server
+
+set -e  # Exit on any error
+
+echo "🚀 Starting chat database import..."
+
+# Configuration - Update these values for your new server
+DB_HOST="${CHAT_DB_HOST:-localhost}"
+DB_PORT="${CHAT_DB_PORT:-5432}"
+DB_NAME="${CHAT_DB_NAME:-rocketchat_converted}"
+DB_USER="${CHAT_DB_USER:-rocketchat_user}"
+
+# Check if database connection info is available
+if [ -z "$CHAT_DB_PASSWORD" ]; then
+    echo "⚠️  CHAT_DB_PASSWORD environment variable not set"
+    echo "Please set it with: export CHAT_DB_PASSWORD='your_password'"
+    exit 1
+fi
+
+# Find the migration directory
+MIGRATION_DIR=""
+if [ -d "/tmp" ]; then
+    MIGRATION_DIR=$(find /tmp -maxdepth 1 -name "chat-migration-*" -type d | head -1)
+fi
+
+if [ -z "$MIGRATION_DIR" ]; then
+    echo "❌ No migration directory found in /tmp/"
+    echo "Please specify the migration directory:"
+    read -p "Enter full path to migration directory: " MIGRATION_DIR
+fi
+
+if [ ! -d "$MIGRATION_DIR" ]; then
+    echo "❌ Migration directory not found: $MIGRATION_DIR"
+    exit 1
+fi
+
+echo "📁 Using migration directory: $MIGRATION_DIR"
+echo "📊 Target database: $DB_NAME on $DB_HOST:$DB_PORT"
+
+# Verify required files exist
+REQUIRED_FILES=("chat-schema.sql" "chat-data.sql" "chat-files.tar.gz")
+for file in "${REQUIRED_FILES[@]}"; do
+    if [ ! -f "$MIGRATION_DIR/$file" ]; then
+        echo "❌ Required file not found: $MIGRATION_DIR/$file"
+        exit 1
+    fi
+done
+
+echo "✅ All required files found"
+
+# Test database connection
+echo "🔗 Testing database connection..."
+PGPASSWORD="$CHAT_DB_PASSWORD" psql \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    -c "SELECT version();" > /dev/null
+
+if [ $? -eq 0 ]; then
+    echo "✅ Database connection successful"
+else
+    echo "❌ Database connection failed"
+    echo "Please ensure:"
+    echo "  1. PostgreSQL is running"
+    echo "  2. Database '$DB_NAME' exists"
+    echo "  3. User '$DB_USER' has access"
+    echo "  4. Password is correct"
+    exit 1
+fi
+
+# Import database schema
+echo "📋 Importing database schema..."
+PGPASSWORD="$CHAT_DB_PASSWORD" psql \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    -f "$MIGRATION_DIR/chat-schema.sql"
+
+if [ $? -eq 0 ]; then
+    echo "✅ Schema imported successfully"
+else
+    echo "❌ Schema import failed"
+    exit 1
+fi
+
+# Import database data
+echo "💾 Importing database data..."
+echo "   This may take a while depending on data size..."
+
+PGPASSWORD="$CHAT_DB_PASSWORD" psql \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    -f "$MIGRATION_DIR/chat-data.sql"
+
+if [ $? -eq 0 ]; then
+    echo "✅ Data imported successfully"
+else
+    echo "❌ Data import failed"
+    echo "Check the error messages above for details"
+    exit 1
+fi
+
+# Create files directory and import files
+echo "📎 Setting up files directory..."
+mkdir -p "db-convert/db"
+
+if [ -s "$MIGRATION_DIR/chat-files.tar.gz" ]; then
+    echo "📂 Extracting chat files..."
+    cd db-convert/db
+    tar -xzf "$MIGRATION_DIR/chat-files.tar.gz"
+    cd ../..
+    
+    # Set proper permissions
+    if [ -d "db-convert/db/files" ]; then
+        chmod -R 755 db-convert/db/files
+        echo "✅ Files imported and permissions set"
+    else
+        echo "⚠️  Files directory not created properly"
+    fi
+else
+    echo "ℹ️  No files to import (empty archive)"
+    mkdir -p "db-convert/db/files/uploads"
+    mkdir -p "db-convert/db/files/avatars"
+fi
+
+# Get final table statistics
+echo "📈 Generating import statistics..."
+PGPASSWORD="$CHAT_DB_PASSWORD" psql \
+    -h "$DB_HOST" \
+    -p "$DB_PORT" \
+    -U "$DB_USER" \
+    -d "$DB_NAME" \
+    -c "
+    SELECT 
+        tablename,
+        n_live_tup as row_count
+    FROM pg_stat_user_tables 
+    WHERE schemaname = 'public'
+    ORDER BY n_live_tup DESC;
+    "
+
+# Create import summary
+echo ""
+echo "🎉 Import completed successfully!"
+echo ""
+echo "📋 Import Summary:"
+echo "  Database: $DB_NAME"
+echo "  Host: $DB_HOST:$DB_PORT"
+echo "  Files location: $(pwd)/db-convert/db/files/"
+echo ""
+echo "🔍 Next steps:"
+echo "1. Update your application configuration to use this database"
+echo "2. Run verify-migration.js to validate the migration"
+echo "3. Test your application thoroughly"
+echo "4. Update DNS/load balancer to point to new server"
+echo ""
+echo "⚠️  Important:"
+echo "- Keep the original data as backup until migration is fully validated"
+echo "- Monitor the application closely after switching"
+echo "- Have a rollback plan ready"

inventory-server/chat/migrate-to-new-server.md

@@ -0,0 +1,86 @@
+# Chat Database Migration Guide
+
+This guide will help you migrate your chat database from the current server to a new PostgreSQL server.
+
+## Overview
+Your chat system uses:
+- Database: `rocketchat_converted` (PostgreSQL)
+- Main tables: users, message, room, uploads, avatars, subscription
+- File storage: db-convert/db/files/ directory with uploads and avatars
+- Environment configuration for database connection
+
+## Migration Steps
+
+### 1. Pre-Migration Setup
+
+On your **new server**, ensure PostgreSQL is installed and running:
+```bash
+# Install PostgreSQL (if not already done)
+sudo apt update
+sudo apt install postgresql postgresql-contrib
+
+# Start PostgreSQL service
+sudo systemctl start postgresql
+sudo systemctl enable postgresql
+```
+
+### 2. Create Database Schema on New Server
+
+Run the provided migration script:
+```bash
+# On new server
+sudo -u postgres psql -f create-new-database.sql
+```
+
+### 3. Export Data from Current Server
+
+Run the export script:
+```bash
+# On current server
+./export-chat-data.sh
+```
+
+This will create:
+- `chat-schema.sql` - Database schema
+- `chat-data.sql` - All table data
+- `chat-files.tar.gz` - All uploaded files and avatars
+
+### 4. Transfer Data to New Server
+
+```bash
+# Copy files to new server
+scp chat-schema.sql chat-data.sql chat-files.tar.gz user@new-server:/tmp/
+```
+
+### 5. Import Data on New Server
+
+```bash
+# On new server
+./import-chat-data.sh
+```
+
+### 6. Update Configuration
+
+Update your environment variables to point to the new database server.
+
+### 7. Verify Migration
+
+Run the verification script to ensure everything transferred correctly:
+```bash
+node verify-migration.js
+```
+
+## Files Provided
+
+1. `create-new-database.sql` - Creates database and user on new server
+2. `export-chat-data.sh` - Exports data from current server
+3. `import-chat-data.sh` - Imports data to new server  
+4. `verify-migration.js` - Verifies data integrity
+5. `update-config-template.env` - Template for new configuration
+
+## Important Notes
+
+- **Backup first**: Always backup your current database before migration
+- **Downtime**: Plan for application downtime during migration
+- **File permissions**: Ensure file permissions are preserved during transfer
+- **Network access**: Ensure new server can accept connections from your application

inventory-server/chat/package.json

@@ -0,0 +1,24 @@
+{
+  "name": "chat-server",
+  "version": "1.0.0",
+  "description": "Chat archive server for Rocket.Chat data",
+  "type": "module",
+  "main": "server.js",
+  "scripts": {
+    "start": "node server.js",
+    "dev": "nodemon server.js"
+  },
+  "dependencies": {
+    "express": "^4.18.2",
+    "cors": "^2.8.5",
+    "pg": "^8.11.0",
+    "dotenv": "^16.0.3",
+    "morgan": "^1.10.0",
+    "jsonwebtoken": "^9.0.2",
+    "pino": "^9.5.0",
+    "pino-http": "^10.3.0"
+  },
+  "devDependencies": {
+    "nodemon": "^2.0.22"
+  }
+} 

inventory-server/chat/routes.js

@@ -0,0 +1,656 @@
+import express from 'express';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+// ESM polyfill — Phase 9 §9.1. Handlers below use __dirname to resolve the
+// db-convert/db/files/{uploads,avatars} static asset paths.
+const __filename = fileURLToPath(import.meta.url);
+const __dirname = path.dirname(__filename);
+
+const router = express.Router();
+
+// Serve uploaded files with proper mapping from database paths to actual file locations
+router.get('/files/uploads/*', async (req, res) => {
+  try {
+    // Extract the path from the URL (everything after /files/uploads/)
+    const requestPath = req.params[0];
+    
+    // The URL path will be like: ufs/AmazonS3:Uploads/274Mf9CyHNG72oF86/filename.jpg
+    // We need to extract the mongo_id (274Mf9CyHNG72oF86) from this path
+    const pathParts = requestPath.split('/');
+    let mongoId = null;
+    
+    // Find the mongo_id in the path structure
+    for (let i = 0; i < pathParts.length; i++) {
+      if (pathParts[i].includes('AmazonS3:Uploads') && i + 1 < pathParts.length) {
+        mongoId = pathParts[i + 1];
+        break;
+      }
+      // Sometimes the mongo_id might be the last part of ufs/AmazonS3:Uploads/mongoId
+      if (pathParts[i] === 'AmazonS3:Uploads' && i + 1 < pathParts.length) {
+        mongoId = pathParts[i + 1];
+        break;
+      }
+    }
+    
+    if (!mongoId) {
+      // Try to get mongo_id from database by matching the full path
+      const result = await global.pool.query(`
+        SELECT mongo_id, name, type 
+        FROM uploads 
+        WHERE path = $1 OR url = $1
+        LIMIT 1
+      `, [`/ufs/AmazonS3:Uploads/${requestPath}`, `/ufs/AmazonS3:Uploads/${requestPath}`]);
+      
+      if (result.rows.length > 0) {
+        mongoId = result.rows[0].mongo_id;
+      }
+    }
+    
+    if (!mongoId) {
+      return res.status(404).json({ error: 'File not found' });
+    }
+    
+    // The actual file is stored with just the mongo_id as filename
+    const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
+    
+    // Get file info from database for proper content-type
+    const fileInfo = await global.pool.query(`
+      SELECT name, type 
+      FROM uploads 
+      WHERE mongo_id = $1
+      LIMIT 1
+    `, [mongoId]);
+    
+    if (fileInfo.rows.length === 0) {
+      return res.status(404).json({ error: 'File metadata not found' });
+    }
+    
+    const { name, type } = fileInfo.rows[0];
+    
+    // Set proper content type
+    if (type) {
+      res.set('Content-Type', type);
+    }
+    
+    // Set content disposition with original filename
+    if (name) {
+      res.set('Content-Disposition', `inline; filename="${name}"`);
+    }
+    
+    // Send the file
+    res.sendFile(filePath, (err) => {
+      if (err) {
+        console.error('Error serving file:', err);
+        if (!res.headersSent) {
+          res.status(404).json({ error: 'File not found on disk' });
+        }
+      }
+    });
+    
+  } catch (error) {
+    console.error('Error serving upload:', error);
+    res.status(500).json({ error: 'Server error' });
+  }
+});
+
+// Also serve files directly by mongo_id for simpler access
+router.get('/files/by-id/:mongoId', async (req, res) => {
+  try {
+    const { mongoId } = req.params;
+    
+    // Get file info from database
+    const fileInfo = await global.pool.query(`
+      SELECT name, type 
+      FROM uploads 
+      WHERE mongo_id = $1
+      LIMIT 1
+    `, [mongoId]);
+    
+    if (fileInfo.rows.length === 0) {
+      return res.status(404).json({ error: 'File not found' });
+    }
+    
+    const { name, type } = fileInfo.rows[0];
+    const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
+    
+    // Set proper content type and filename
+    if (type) {
+      res.set('Content-Type', type);
+    }
+    
+    if (name) {
+      res.set('Content-Disposition', `inline; filename="${name}"`);
+    }
+    
+    // Send the file
+    res.sendFile(filePath, (err) => {
+      if (err) {
+        console.error('Error serving file:', err);
+        if (!res.headersSent) {
+          res.status(404).json({ error: 'File not found on disk' });
+        }
+      }
+    });
+    
+  } catch (error) {
+    console.error('Error serving upload by ID:', error);
+    res.status(500).json({ error: 'Server error' });
+  }
+});
+
+// Serve user avatars by mongo_id
+router.get('/avatar/:mongoId', async (req, res) => {
+  try {
+    const { mongoId } = req.params;
+    
+    console.log(`[Avatar Debug] Looking up avatar for user mongo_id: ${mongoId}`);
+    
+    // First try to find avatar by user's avataretag
+    const userResult = await global.pool.query(`
+      SELECT avataretag, username FROM users WHERE mongo_id = $1
+    `, [mongoId]);
+    
+    let avatarPath = null;
+    
+    if (userResult.rows.length > 0) {
+      const username = userResult.rows[0].username;
+      const avataretag = userResult.rows[0].avataretag;
+      
+      // Try method 1: Look up by avataretag -> etag (for users with avataretag set)
+      if (avataretag) {
+        console.log(`[Avatar Debug] Found user ${username} with avataretag: ${avataretag}`);
+        
+        const avatarResult = await global.pool.query(`
+          SELECT url, path FROM avatars WHERE etag = $1
+        `, [avataretag]);
+        
+        if (avatarResult.rows.length > 0) {
+          const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
+          console.log(`[Avatar Debug] Found avatar record with path: ${dbPath}`);
+          
+          if (dbPath) {
+            const pathParts = dbPath.split('/');
+            for (let i = 0; i < pathParts.length; i++) {
+              if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
+                const avatarMongoId = pathParts[i + 1];
+                avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
+                console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
+                break;
+              }
+            }
+          }
+        } else {
+          console.log(`[Avatar Debug] No avatar record found for etag: ${avataretag}`);
+        }
+      }
+      
+      // Try method 2: Look up by userid directly (for users without avataretag)
+      if (!avatarPath) {
+        console.log(`[Avatar Debug] Trying direct userid lookup for user ${username} (${mongoId})`);
+        
+        const avatarResult = await global.pool.query(`
+          SELECT url, path FROM avatars WHERE userid = $1
+        `, [mongoId]);
+        
+        if (avatarResult.rows.length > 0) {
+          const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
+          console.log(`[Avatar Debug] Found avatar record by userid with path: ${dbPath}`);
+          
+          if (dbPath) {
+            const pathParts = dbPath.split('/');
+            for (let i = 0; i < pathParts.length; i++) {
+              if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
+                const avatarMongoId = pathParts[i + 1];
+                avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
+                console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
+                break;
+              }
+            }
+          }
+        } else {
+          console.log(`[Avatar Debug] No avatar record found for userid: ${mongoId}`);
+        }
+      }
+    } else {
+      console.log(`[Avatar Debug] No user found for mongo_id: ${mongoId}`);
+    }
+    
+    // Fallback: try direct lookup by user mongo_id
+    if (!avatarPath) {
+      avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', mongoId);
+      console.log(`[Avatar Debug] Using fallback path: ${avatarPath}`);
+    }
+    
+    // Set proper content type for images
+    res.set('Content-Type', 'image/jpeg'); // Most avatars are likely JPEG
+    
+    // Send the file
+    res.sendFile(avatarPath, (err) => {
+      if (err) {
+        // If avatar doesn't exist, send a default 404 or generate initials
+        console.log(`[Avatar Debug] Avatar file not found at path: ${avatarPath}, error:`, err.message);
+        if (!res.headersSent) {
+          res.status(404).json({ error: 'Avatar not found' });
+        }
+      } else {
+        console.log(`[Avatar Debug] Successfully served avatar from: ${avatarPath}`);
+      }
+    });
+    
+  } catch (error) {
+    console.error('Error serving avatar:', error);
+    res.status(500).json({ error: 'Server error' });
+  }
+});
+
+// Serve avatars statically as fallback
+router.use('/files/avatars', express.static(path.join(__dirname, 'db-convert/db/files/avatars')));
+
+// Get all users for the "view as" dropdown (active and inactive)
+router.get('/users', async (req, res) => {
+  try {
+    const result = await global.pool.query(`
+      SELECT id, username, name, type, active, status, lastlogin, 
+             statustext, utcoffset, statusconnection, mongo_id, avataretag
+      FROM users 
+      WHERE type = 'user'
+      ORDER BY 
+        active DESC,  -- Active users first
+        CASE 
+          WHEN status = 'online' THEN 1
+          WHEN status = 'away' THEN 2  
+          WHEN status = 'busy' THEN 3
+          ELSE 4
+        END,
+        name ASC
+    `);
+    
+    res.json({
+      status: 'success',
+      users: result.rows
+    });
+  } catch (error) {
+    console.error('Error fetching users:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to fetch users',
+      details: error.message
+    });
+  }
+});
+
+// Get rooms for a specific user with enhanced room names for direct messages
+router.get('/users/:userId/rooms', async (req, res) => {
+  const { userId } = req.params;
+  
+  try {
+    // Get the current user's mongo_id for filtering
+    const userResult = await global.pool.query(`
+      SELECT mongo_id, username FROM users WHERE id = $1
+    `, [userId]);
+    
+    if (userResult.rows.length === 0) {
+      return res.status(404).json({
+        status: 'error',
+        error: 'User not found'
+      });
+    }
+    
+    const currentUserMongoId = userResult.rows[0].mongo_id;
+    const currentUsername = userResult.rows[0].username;
+    
+    // Get rooms where the user is a member with proper naming from subscription table
+    // Include archived and closed rooms but sort them at the bottom
+    const result = await global.pool.query(`
+      SELECT DISTINCT 
+        r.id, 
+        r.mongo_id as room_mongo_id, 
+        r.name, 
+        r.fname, 
+        r.t as type, 
+        r.msgs, 
+        r.lm as last_message_date, 
+        r.usernames, 
+        r.uids,
+        r.userscount,
+        r.description,
+        r.teamid,
+        r.archived,
+        s.open,
+        -- Use the subscription's name for direct messages (excludes current user)
+        -- For channels/groups, use room's fname or name
+        CASE 
+          WHEN r.t = 'd' THEN COALESCE(s.fname, s.name, 'Unknown User')
+          ELSE COALESCE(r.fname, r.name, 'Unnamed Room')
+        END as display_name
+      FROM room r
+      JOIN subscription s ON s.rid = r.mongo_id 
+      WHERE s.u->>'_id' = $1
+      ORDER BY 
+        s.open DESC NULLS LAST,  -- Open rooms first
+        r.archived NULLS FIRST,  -- Non-archived first (nulls treated as false)
+        r.lm DESC NULLS LAST
+      LIMIT 50
+    `, [currentUserMongoId]);
+
+    // Enhance rooms with participant information for direct messages
+    const enhancedRooms = await Promise.all(result.rows.map(async (room) => {
+      if (room.type === 'd' && room.uids) {
+        // Get participant info (excluding current user) for direct messages
+        const participantResult = await global.pool.query(`
+          SELECT u.username, u.name, u.mongo_id, u.avataretag
+          FROM users u 
+          WHERE u.mongo_id = ANY($1::text[])
+          AND u.mongo_id != $2
+        `, [room.uids, currentUserMongoId]);
+        
+        room.participants = participantResult.rows;
+      }
+      return room;
+    }));
+    
+    res.json({
+      status: 'success',
+      rooms: enhancedRooms
+    });
+  } catch (error) {
+    console.error('Error fetching user rooms:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to fetch user rooms',
+      details: error.message
+    });
+  }
+});
+
+// Get room details including participants
+router.get('/rooms/:roomId', async (req, res) => {
+  const { roomId } = req.params;
+  const { userId } = req.query; // Accept current user ID as query parameter
+  
+  try {
+    const result = await global.pool.query(`
+      SELECT r.id, r.name, r.fname, r.t as type, r.msgs, r.description,
+             r.lm as last_message_date, r.usernames, r.uids, r.userscount, r.teamid
+      FROM room r
+      WHERE r.id = $1
+    `, [roomId]);
+    
+    if (result.rows.length === 0) {
+      return res.status(404).json({
+        status: 'error',
+        error: 'Room not found'
+      });
+    }
+    
+    const room = result.rows[0];
+    
+    // For direct messages, get the proper display name based on current user
+    if (room.type === 'd' && room.uids && userId) {
+      // Get current user's mongo_id
+      const userResult = await global.pool.query(`
+        SELECT mongo_id FROM users WHERE id = $1
+      `, [userId]);
+      
+      if (userResult.rows.length > 0) {
+        const currentUserMongoId = userResult.rows[0].mongo_id;
+        
+        // Get display name from subscription table for this user
+        // Use room mongo_id to match with subscription.rid
+        const roomMongoResult = await global.pool.query(`
+          SELECT mongo_id FROM room WHERE id = $1
+        `, [roomId]);
+        
+        if (roomMongoResult.rows.length > 0) {
+          const roomMongoId = roomMongoResult.rows[0].mongo_id;
+          
+          const subscriptionResult = await global.pool.query(`
+            SELECT fname, name FROM subscription 
+            WHERE rid = $1 AND u->>'_id' = $2
+          `, [roomMongoId, currentUserMongoId]);
+          
+          if (subscriptionResult.rows.length > 0) {
+            const sub = subscriptionResult.rows[0];
+            room.display_name = sub.fname || sub.name || 'Unknown User';
+          }
+        }
+      }
+      
+      // Get all participants for additional info
+      const participantResult = await global.pool.query(`
+        SELECT username, name
+        FROM users 
+        WHERE mongo_id = ANY($1::text[])
+      `, [room.uids]);
+      
+      room.participants = participantResult.rows;
+    } else {
+      // For channels/groups, use room's fname or name
+      room.display_name = room.fname || room.name || 'Unnamed Room';
+    }
+    
+    res.json({
+      status: 'success',
+      room: room
+    });
+  } catch (error) {
+    console.error('Error fetching room details:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to fetch room details',
+      details: error.message
+    });
+  }
+});
+
+// Get messages for a specific room (fast, without attachments)
+router.get('/rooms/:roomId/messages', async (req, res) => {
+  const { roomId } = req.params;
+  const { limit = 50, offset = 0, before } = req.query;
+  
+  try {
+    // Fast query - just get messages without expensive attachment joins
+    let query = `
+      SELECT m.id, m.msg, m.ts, m.u, m._updatedat, m.urls, m.mentions, m.md
+      FROM message m
+      JOIN room r ON m.rid = r.mongo_id
+      WHERE r.id = $1
+    `;
+    
+    const params = [roomId];
+    
+    if (before) {
+      query += ` AND m.ts < $${params.length + 1}`;
+      params.push(before);
+    }
+    
+    query += ` ORDER BY m.ts DESC LIMIT $${params.length + 1} OFFSET $${params.length + 2}`;
+    params.push(limit, offset);
+    
+    const result = await global.pool.query(query, params);
+    
+    // Add empty attachments array for now - attachments will be loaded separately if needed
+    const messages = result.rows.map(msg => ({
+      ...msg,
+      attachments: []
+    }));
+    
+    res.json({
+      status: 'success',
+      messages: messages.reverse() // Reverse to show oldest first
+    });
+  } catch (error) {
+    console.error('Error fetching messages:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to fetch messages',
+      details: error.message
+    });
+  }
+});
+
+// Get attachments for specific messages (called separately for performance)
+router.post('/messages/attachments', async (req, res) => {
+  const { messageIds } = req.body;
+  
+  if (!messageIds || !Array.isArray(messageIds) || messageIds.length === 0) {
+    return res.json({ status: 'success', attachments: {} });
+  }
+  
+  try {
+    // Get room mongo_id from first message to limit search scope
+    const roomQuery = await global.pool.query(`
+      SELECT r.mongo_id as room_mongo_id
+      FROM message m
+      JOIN room r ON m.rid = r.mongo_id
+      WHERE m.id = $1
+      LIMIT 1
+    `, [messageIds[0]]);
+    
+    if (roomQuery.rows.length === 0) {
+      return res.json({ status: 'success', attachments: {} });
+    }
+    
+    const roomMongoId = roomQuery.rows[0].room_mongo_id;
+    
+    // Get messages and their upload timestamps
+    const messagesQuery = await global.pool.query(`
+      SELECT m.id, m.ts, m.u->>'_id' as user_id
+      FROM message m
+      WHERE m.id = ANY($1::int[])
+    `, [messageIds]);
+    
+    if (messagesQuery.rows.length === 0) {
+      return res.json({ status: 'success', attachments: {} });
+    }
+    
+    // Build a map of user_id -> array of message timestamps for efficient lookup
+    const userTimeMap = {};
+    const messageMap = {};
+    messagesQuery.rows.forEach(msg => {
+      if (!userTimeMap[msg.user_id]) {
+        userTimeMap[msg.user_id] = [];
+      }
+      userTimeMap[msg.user_id].push(msg.ts);
+      messageMap[msg.id] = { ts: msg.ts, user_id: msg.user_id };
+    });
+    
+    // Get attachments for this room and these users
+    const uploadsQuery = await global.pool.query(`
+      SELECT mongo_id, name, size, type, url, path, typegroup, identify, 
+             userid, uploadedat
+      FROM uploads 
+      WHERE rid = $1 
+      AND userid = ANY($2::text[])
+      ORDER BY uploadedat
+    `, [roomMongoId, Object.keys(userTimeMap)]);
+    
+    // Match attachments to messages based on timestamp proximity (within 5 minutes)
+    const attachmentsByMessage = {};
+    
+    uploadsQuery.rows.forEach(upload => {
+      const uploadTime = new Date(upload.uploadedat).getTime();
+      
+      // Find the closest message from this user within 5 minutes
+      let closestMessageId = null;
+      let closestTimeDiff = Infinity;
+      
+      Object.entries(messageMap).forEach(([msgId, msgData]) => {
+        if (msgData.user_id === upload.userid) {
+          const msgTime = new Date(msgData.ts).getTime();
+          const timeDiff = Math.abs(uploadTime - msgTime);
+          
+          if (timeDiff < 300000 && timeDiff < closestTimeDiff) { // 5 minutes = 300000ms
+            closestMessageId = msgId;
+            closestTimeDiff = timeDiff;
+          }
+        }
+      });
+      
+      if (closestMessageId) {
+        if (!attachmentsByMessage[closestMessageId]) {
+          attachmentsByMessage[closestMessageId] = [];
+        }
+        
+        attachmentsByMessage[closestMessageId].push({
+          id: upload.id,
+          mongo_id: upload.mongo_id,
+          name: upload.name,
+          size: upload.size,
+          type: upload.type,
+          url: upload.url,
+          path: upload.path,
+          typegroup: upload.typegroup,
+          identify: upload.identify
+        });
+      }
+    });
+    
+    res.json({
+      status: 'success',
+      attachments: attachmentsByMessage
+    });
+    
+  } catch (error) {
+    console.error('Error fetching message attachments:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to fetch attachments',
+      details: error.message
+    });
+  }
+});
+
+// Search messages in accessible rooms for a user
+router.get('/users/:userId/search', async (req, res) => {
+  const { userId } = req.params;
+  const { q, limit = 20 } = req.query;
+  
+  if (!q || q.length < 2) {
+    return res.status(400).json({
+      status: 'error',
+      error: 'Search query must be at least 2 characters'
+    });
+  }
+  
+  try {
+    const userResult = await global.pool.query(`
+      SELECT mongo_id FROM users WHERE id = $1
+    `, [userId]);
+    
+    if (userResult.rows.length === 0) {
+      return res.status(404).json({
+        status: 'error',
+        error: 'User not found'
+      });
+    }
+    
+    const currentUserMongoId = userResult.rows[0].mongo_id;
+    
+    const result = await global.pool.query(`
+      SELECT m.id, m.msg, m.ts, m.u, r.id as room_id, r.name as room_name, r.fname as room_fname, r.t as room_type
+      FROM message m
+      JOIN room r ON m.rid = r.mongo_id
+      JOIN subscription s ON s.rid = r.mongo_id AND s.u->>'_id' = $1
+      WHERE m.msg ILIKE $2
+      AND r.archived IS NOT TRUE
+      ORDER BY m.ts DESC
+      LIMIT $3
+    `, [currentUserMongoId, `%${q}%`, limit]);
+    
+    res.json({
+      status: 'success',
+      results: result.rows
+    });
+  } catch (error) {
+    console.error('Error searching messages:', error);
+    res.status(500).json({ 
+      status: 'error',
+      error: 'Failed to search messages',
+      details: error.message
+    });
+  }
+});
+
+export default router;

inventory-server/chat/server.js

@@ -0,0 +1,132 @@
+// chat-server — Phase 9 §9.1 of CONSOLIDATION_PLAN.md.
+//
+// ESM conversion + in-process authenticate() defense-in-depth. Previously this
+// service relied on the Caddy `forward_auth` gate alone — `localhost:3014`
+// was reachable unauthenticated. Now:
+//   1. Bound to 127.0.0.1 (was 0.0.0.0) so direct-port access is impossible.
+//   2. authenticate() runs against an in-process `inventory_db` pool before
+//      any route handler sees the request.
+//
+// Two pools intentionally:
+//   - `inventoryPool`: used by authenticate() for users/permissions lookups
+//     against the main inventory_db (matches DB_* env vars).
+//   - `pool` (set as global.pool for routes.js): the existing
+//     `rocketchat_converted` pool driven by CHAT_DB_* env vars. routes.js
+//     reads global.pool throughout — no handler-body changes needed.
+
+import { config as loadEnv } from 'dotenv';
+import express from 'express';
+import cors from 'cors';
+import morgan from 'morgan';
+import pg from 'pg';
+import path from 'node:path';
+import fs from 'node:fs';
+import { fileURLToPath } from 'node:url';
+
+import { authenticate } from '../shared/auth/middleware.js';
+import { corsOptions } from '../shared/cors/policy.js';
+import { errorHandler } from '../shared/errors/handler.js';
+import { requestLog } from '../shared/logging/request-log.js';
+
+import chatRoutes from './routes.js';
+
+const { Pool } = pg;
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+// Env layering matches dashboard-server (Deviation #18): shared .env wins on
+// collisions for security-critical vars, local .env supplies CHAT_DB_*.
+const sharedEnvPath = '/var/www/inventory/.env';
+const localEnvPath = path.resolve(__dirname, '.env');
+if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
+if (fs.existsSync(localEnvPath)) loadEnv({ path: localEnvPath });
+
+if (!process.env.JWT_SECRET) {
+  console.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
+  process.exit(1);
+}
+
+const app = express();
+const port = Number(process.env.CHAT_PORT) || 3014;
+
+console.log('Starting chat server with config:', {
+  host: process.env.CHAT_DB_HOST,
+  user: process.env.CHAT_DB_USER,
+  database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
+  port: process.env.CHAT_DB_PORT,
+  chat_port: port,
+});
+
+// Rocket.Chat archive pool — routes.js reads it via global.pool.
+const pool = new Pool({
+  host: process.env.CHAT_DB_HOST,
+  user: process.env.CHAT_DB_USER,
+  password: process.env.CHAT_DB_PASSWORD,
+  database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
+  port: process.env.CHAT_DB_PORT,
+});
+global.pool = pool;
+
+// inventory_db pool — used by authenticate() for user/permission lookups.
+const inventoryPool = new Pool({
+  host: process.env.DB_HOST,
+  user: process.env.DB_USER,
+  password: process.env.DB_PASSWORD,
+  database: process.env.DB_NAME,
+  port: Number(process.env.DB_PORT) || 5432,
+  ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
+});
+
+app.use(requestLog());
+app.use(express.json());
+app.use(morgan('combined'));
+app.use(cors(corsOptions));
+
+// /health stays unauthenticated for out-of-band probes — mounted BEFORE
+// authenticate() so monitoring tools on the host can poll without a JWT.
+// Only reachable via localhost:3014 directly (Caddy routes /health to
+// inventory-server:3010, not here).
+app.get('/health', (req, res) => res.json({ status: 'healthy' }));
+
+// Phase 9 §9.1 — per-server auth re-verification. Every chat route must pass
+// authenticate() in addition to the Caddy forward_auth gate.
+app.use(authenticate({ pool: inventoryPool, secret: process.env.JWT_SECRET }));
+
+app.get('/test-db', async (req, res, next) => {
+  try {
+    const result = await pool.query('SELECT COUNT(*) as user_count FROM users WHERE active = true');
+    const messageResult = await pool.query('SELECT COUNT(*) as message_count FROM message');
+    const roomResult = await pool.query('SELECT COUNT(*) as room_count FROM room');
+    res.json({
+      status: 'success',
+      database: 'rocketchat_converted',
+      stats: {
+        active_users: parseInt(result.rows[0].user_count, 10),
+        total_messages: parseInt(messageResult.rows[0].message_count, 10),
+        total_rooms: parseInt(roomResult.rows[0].room_count, 10),
+      },
+    });
+  } catch (err) {
+    next(err);
+  }
+});
+
+app.use('/', chatRoutes);
+
+app.use(errorHandler);
+
+// Phase 9 §9.1 — bind to 127.0.0.1. Caddy reverse_proxy targets localhost:3014
+// already; this closes the gap where unauthenticated direct-port access from
+// any host on the network was possible.
+const server = app.listen(port, '127.0.0.1', () => {
+  console.log(`Chat server running on 127.0.0.1:${port}`);
+});
+
+const shutdown = async (signal) => {
+  console.log(`chat-server shutting down (${signal})`);
+  server.close();
+  try { await pool.end(); } catch { /* ignore */ }
+  try { await inventoryPool.end(); } catch { /* ignore */ }
+  process.exit(0);
+};
+process.on('SIGTERM', () => shutdown('SIGTERM'));
+process.on('SIGINT', () => shutdown('SIGINT'));

inventory-server/chat/update-config-template.env

@@ -0,0 +1,26 @@
+# Chat Server Database Configuration Template
+# Copy this to your .env file and update the values for your new server
+
+# Database Configuration for New Server
+CHAT_DB_HOST=your-new-server-ip-or-hostname
+CHAT_DB_PORT=5432
+CHAT_DB_NAME=rocketchat_converted
+CHAT_DB_USER=rocketchat_user
+CHAT_DB_PASSWORD=your-secure-password
+
+# Chat Server Port
+CHAT_PORT=3014
+
+# Example configuration:
+# CHAT_DB_HOST=192.168.1.100
+# CHAT_DB_PORT=5432
+# CHAT_DB_NAME=rocketchat_converted
+# CHAT_DB_USER=rocketchat_user
+# CHAT_DB_PASSWORD=MySecureP@ssw0rd123
+
+# Notes:
+# - Replace 'your-new-server-ip-or-hostname' with actual server address
+# - Use a strong password for CHAT_DB_PASSWORD
+# - Ensure the new server allows connections from your application server
+# - Update any firewall rules to allow PostgreSQL connections (port 5432)
+# - Test connectivity before updating production configuration

inventory-server/chat/verify-migration.cjs

@@ -0,0 +1,231 @@
+#!/usr/bin/env node
+
+/**
+ * Chat Database Migration Verification Script
+ * 
+ * This script verifies that the chat database migration was successful
+ * by comparing record counts and testing basic functionality.
+ */
+
+require('dotenv').config({ path: '../.env' });
+const { Pool } = require('pg');
+
+// Database configuration
+const pool = new Pool({
+  host: process.env.CHAT_DB_HOST || 'localhost',
+  user: process.env.CHAT_DB_USER || 'rocketchat_user',
+  password: process.env.CHAT_DB_PASSWORD,
+  database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
+  port: process.env.CHAT_DB_PORT || 5432,
+});
+
+const originalStats = process.argv[2] ? JSON.parse(process.argv[2]) : null;
+
+async function verifyMigration() {
+  console.log('🔍 Starting migration verification...\n');
+  
+  try {
+    // Test basic connection
+    console.log('🔗 Testing database connection...');
+    const versionResult = await pool.query('SELECT version()');
+    console.log('✅ Database connection successful');
+    console.log(`   PostgreSQL version: ${versionResult.rows[0].version.split(' ')[1]}\n`);
+
+    // Get table statistics
+    console.log('📊 Checking table statistics...');
+    const statsResult = await pool.query(`
+      SELECT 
+        tablename,
+        n_live_tup as row_count,
+        n_dead_tup as dead_rows,
+        schemaname
+      FROM pg_stat_user_tables 
+      WHERE schemaname = 'public'
+      ORDER BY n_live_tup DESC
+    `);
+
+    if (statsResult.rows.length === 0) {
+      console.log('❌ No tables found! Migration may have failed.');
+      return false;
+    }
+
+    console.log('📋 Table Statistics:');
+    console.log('   Table Name          | Row Count | Dead Rows');
+    console.log('   -------------------|-----------|----------');
+    
+    let totalRows = 0;
+    const tableStats = {};
+    
+    for (const row of statsResult.rows) {
+      const rowCount = parseInt(row.row_count) || 0;
+      const deadRows = parseInt(row.dead_rows) || 0;
+      totalRows += rowCount;
+      tableStats[row.tablename] = rowCount;
+      
+      console.log(`   ${row.tablename.padEnd(18)} | ${rowCount.toString().padStart(9)} | ${deadRows.toString().padStart(8)}`);
+    }
+    
+    console.log(`\n   Total rows across all tables: ${totalRows}\n`);
+
+    // Verify critical tables exist and have data
+    const criticalTables = ['users', 'message', 'room'];
+    console.log('🔑 Checking critical tables...');
+    
+    for (const table of criticalTables) {
+      if (tableStats[table] > 0) {
+        console.log(`✅ ${table}: ${tableStats[table]} rows`);
+      } else if (tableStats[table] === 0) {
+        console.log(`⚠️  ${table}: table exists but is empty`);
+      } else {
+        console.log(`❌ ${table}: table not found`);
+        return false;
+      }
+    }
+
+    // Test specific functionality
+    console.log('\n🧪 Testing specific functionality...');
+
+    // Test users table
+    const userTest = await pool.query(`
+      SELECT COUNT(*) as total_users,
+             COUNT(*) FILTER (WHERE active = true) as active_users,
+             COUNT(*) FILTER (WHERE type = 'user') as regular_users
+      FROM users
+    `);
+    
+    if (userTest.rows[0]) {
+      const { total_users, active_users, regular_users } = userTest.rows[0];
+      console.log(`✅ Users: ${total_users} total, ${active_users} active, ${regular_users} regular users`);
+    }
+
+    // Test messages table
+    const messageTest = await pool.query(`
+      SELECT COUNT(*) as total_messages,
+             COUNT(DISTINCT rid) as unique_rooms,
+             MIN(ts) as oldest_message,
+             MAX(ts) as newest_message
+      FROM message
+    `);
+    
+    if (messageTest.rows[0]) {
+      const { total_messages, unique_rooms, oldest_message, newest_message } = messageTest.rows[0];
+      console.log(`✅ Messages: ${total_messages} total across ${unique_rooms} rooms`);
+      if (oldest_message && newest_message) {
+        console.log(`   Date range: ${oldest_message.toISOString().split('T')[0]} to ${newest_message.toISOString().split('T')[0]}`);
+      }
+    }
+
+    // Test rooms table
+    const roomTest = await pool.query(`
+      SELECT COUNT(*) as total_rooms,
+             COUNT(*) FILTER (WHERE t = 'c') as channels,
+             COUNT(*) FILTER (WHERE t = 'p') as private_groups,
+             COUNT(*) FILTER (WHERE t = 'd') as direct_messages
+      FROM room
+    `);
+    
+    if (roomTest.rows[0]) {
+      const { total_rooms, channels, private_groups, direct_messages } = roomTest.rows[0];
+      console.log(`✅ Rooms: ${total_rooms} total (${channels} channels, ${private_groups} private, ${direct_messages} DMs)`);
+    }
+
+    // Test file uploads if table exists
+    if (tableStats.uploads > 0) {
+      const uploadTest = await pool.query(`
+        SELECT COUNT(*) as total_uploads,
+               COUNT(DISTINCT typegroup) as file_types,
+               pg_size_pretty(SUM(size)) as total_size
+        FROM uploads
+        WHERE size IS NOT NULL
+      `);
+      
+      if (uploadTest.rows[0]) {
+        const { total_uploads, file_types, total_size } = uploadTest.rows[0];
+        console.log(`✅ Uploads: ${total_uploads} files, ${file_types} types, ${total_size || 'unknown size'}`);
+      }
+    }
+
+    // Test server health endpoint simulation
+    console.log('\n🏥 Testing application endpoints simulation...');
+    
+    try {
+      const healthTest = await pool.query(`
+        SELECT 
+          (SELECT COUNT(*) FROM users WHERE active = true) as active_users,
+          (SELECT COUNT(*) FROM message) as total_messages,
+          (SELECT COUNT(*) FROM room) as total_rooms
+      `);
+      
+      if (healthTest.rows[0]) {
+        const stats = healthTest.rows[0];
+        console.log('✅ Health check simulation passed');
+        console.log(`   Active users: ${stats.active_users}`);
+        console.log(`   Total messages: ${stats.total_messages}`);
+        console.log(`   Total rooms: ${stats.total_rooms}`);
+      }
+    } catch (error) {
+      console.log(`⚠️  Health check simulation failed: ${error.message}`);
+    }
+
+    // Check indexes
+    console.log('\n📇 Checking database indexes...');
+    const indexResult = await pool.query(`
+      SELECT 
+        schemaname,
+        tablename,
+        indexname,
+        indexdef
+      FROM pg_indexes 
+      WHERE schemaname = 'public'
+      ORDER BY tablename, indexname
+    `);
+
+    const indexesByTable = {};
+    for (const idx of indexResult.rows) {
+      if (!indexesByTable[idx.tablename]) {
+        indexesByTable[idx.tablename] = [];
+      }
+      indexesByTable[idx.tablename].push(idx.indexname);
+    }
+
+    for (const [table, indexes] of Object.entries(indexesByTable)) {
+      console.log(`   ${table}: ${indexes.length} indexes`);
+    }
+
+    console.log('\n🎉 Migration verification completed successfully!');
+    console.log('\n✅ Summary:');
+    console.log(`   - Database connection: Working`);
+    console.log(`   - Tables created: ${statsResult.rows.length}`);
+    console.log(`   - Total data rows: ${totalRows}`);
+    console.log(`   - Critical tables: All present`);
+    console.log(`   - Indexes: ${indexResult.rows.length} total`);
+    
+    console.log('\n🚀 Next steps:');
+    console.log('   1. Update your application configuration');
+    console.log('   2. Start your chat server');
+    console.log('   3. Test chat functionality in the browser');
+    console.log('   4. Monitor logs for any issues');
+    
+    return true;
+
+  } catch (error) {
+    console.error('❌ Migration verification failed:', error.message);
+    console.error('\n🔧 Troubleshooting steps:');
+    console.error('   1. Check database connection settings');
+    console.error('   2. Verify database and user exist');
+    console.error('   3. Check PostgreSQL logs');
+    console.error('   4. Ensure import completed without errors');
+    return false;
+  } finally {
+    await pool.end();
+  }
+}
+
+// Run verification
+if (require.main === module) {
+  verifyMigration().then(success => {
+    process.exit(success ? 0 : 1);
+  });
+}
+
+module.exports = { verifyMigration };

inventory-server/dashboard/.env.example

@@ -0,0 +1,32 @@
+# dashboard-server .env template (Phase 4)
+#
+# The merged dashboard-server reads /var/www/inventory/.env FIRST (provides
+# JWT_SECRET, DB_*, REDIS_*) and then layers this .env on top for vendor keys.
+# Shared/security-critical vars stay in /var/www/inventory/.env so they aren't
+# duplicated; vendor keys live here.
+#
+# Copy to .env and populate. Do NOT commit the populated file.
+
+# Port the merged service listens on
+DASHBOARD_PORT=3015
+
+# Klaviyo (replaces klaviyo-server/.env)
+KLAVIYO_API_KEY=
+KLAVIYO_API_REVISION=2024-02-15
+KLAVIYO_API_URL=https://a.klaviyo.com/api
+
+# Meta / Facebook Ads (replaces meta-server/.env)
+META_ACCESS_TOKEN=
+META_AD_ACCOUNT_ID=
+META_API_VERSION=v21.0
+
+# Google Analytics (replaces google-server/.env)
+GA_PROPERTY_ID=
+GOOGLE_APPLICATION_CREDENTIALS_JSON=
+
+# Typeform (replaces typeform-server/.env)
+TYPEFORM_ACCESS_TOKEN=
+
+# Vendors share the inventory REDIS_URL or REDIS_HOST/PORT/USERNAME/PASSWORD
+# from the parent .env. Do NOT redeclare here unless you need a vendor-only
+# override (rare; would need to fork shared/db/redis.js too).

inventory-server/dashboard/acot-server/README.md

@@ -0,0 +1,205 @@
+# ACOT Server
+
+This server replaces the Klaviyo integration with direct database queries to the production MySQL database via SSH tunnel. It provides seamless API compatibility for all frontend components without requiring any frontend changes.
+
+## Setup
+
+1. **Environment Variables**: Copy `.env.example` to `.env` and configure:
+   ```
+   DB_HOST=localhost
+   DB_PORT=3306
+   DB_USER=your_db_user
+   DB_PASSWORD=your_db_password
+   DB_NAME=your_db_name
+   PORT=3007
+   NODE_ENV=development
+   ```
+
+2. **SSH Tunnel**: Ensure your SSH tunnel to the production database is running on localhost:3306.
+
+3. **Install Dependencies**:
+   ```bash
+   npm install
+   ```
+
+4. **Start Server**:
+   ```bash
+   npm start
+   ```
+
+## API Endpoints
+
+All endpoints provide exact API compatibility with the previous Klaviyo implementation:
+
+### Main Statistics
+- `GET /api/acot/events/stats` - Complete statistics dashboard data
+  - Query params: `timeRange` (today, yesterday, thisWeek, lastWeek, thisMonth, lastMonth, last7days, last30days, last90days) or `startDate`/`endDate` for custom ranges
+  - Returns: Revenue, orders, AOV, shipping data, order types, brands/categories, refunds, cancellations, best day, peak hour, order ranges, period progress, projections
+
+### Daily Details
+- `GET /api/acot/events/stats/details` - Daily breakdown with previous period comparisons
+  - Query params: `timeRange`, `metric` (revenue, orders, average_order, etc.), `daily=true`
+  - Returns: Array of daily data points with trend comparisons
+
+### Products
+- `GET /api/acot/events/products` - Top products with sales data
+  - Query params: `timeRange`
+  - Returns: Product list with images, sales quantities, revenue, and order counts
+
+### Projections
+- `GET /api/acot/events/projection` - Smart revenue projections for incomplete periods
+  - Query params: `timeRange`
+  - Returns: Projected revenue with confidence levels based on historical patterns
+
+### Health Check
+- `GET /api/acot/test` - Server health and database connectivity test
+
+## Database Schema
+
+The server queries the following main tables:
+
+### Orders (`_order`)
+- **Key fields**: `order_id`, `date_placed`, `summary_total`, `order_status`, `ship_method_selected`, `stats_waiting_preorder`
+- **Valid orders**: `order_status > 15`
+- **Cancelled orders**: `order_status = 15`
+- **Shipped orders**: `order_status IN (100, 92)`
+- **Pre-orders**: `stats_waiting_preorder > 0`
+- **Local pickup**: `ship_method_selected = 'localpickup'`
+- **On-hold orders**: `ship_method_selected = 'holdit'`
+
+### Order Items (`order_items`)
+- **Fields**: `order_id`, `prod_pid`, `qty_ordered`, `prod_price`
+- **Purpose**: Links orders to products for detailed analysis
+
+### Products (`products`)
+- **Fields**: `pid`, `description` (product name), `company`
+- **Purpose**: Product information and brand data
+
+### Product Images (`product_images`)
+- **Fields**: `pid`, `iid`, `order` (priority)
+- **Primary image**: `order = 255` (highest priority)
+- **Image URL generation**: `https://sbing.com/i/products/0000/{prefix}/{pid}-{type}-{iid}.jpg`
+
+### Payments (`order_payment`)
+- **Refunds**: `payment_amount < 0`
+- **Purpose**: Track refund amounts and counts
+
+## Business Logic
+
+### Time Handling
+- **Timezone**: All calculations in UTC-5 (Eastern Time)
+- **Business Day**: 1 AM - 12:59 AM Eastern (25-hour business day)
+- **Format**: MySQL DATETIME format (YYYY-MM-DD HH:MM:SS)
+- **Period Boundaries**: Calculated using `timeUtils.js` for consistent time range handling
+
+### Order Processing
+- **Revenue Calculation**: Only includes orders with `order_status > 15`
+- **Order Types**:
+  - Pre-orders: `stats_waiting_preorder > 0`
+  - Local pickup: `ship_method_selected = 'localpickup'`
+  - On-hold: `ship_method_selected = 'holdit'`
+- **Shipping Methods**: Mapped to friendly names (e.g., `usps_ground_advantage` → "USPS Ground Advantage")
+
+### Projections
+- **Period Progress**: Calculated based on current time within the selected period
+- **Simple Projection**: Linear extrapolation based on current progress
+- **Smart Projection**: Uses historical data patterns for more accurate forecasting
+- **Confidence Levels**: Based on data consistency and historical accuracy
+
+### Image URL Generation
+- **Pattern**: `https://sbing.com/i/products/0000/{prefix}/{pid}-{type}-{iid}.jpg`
+- **Prefix**: First 2 digits of product ID
+- **Type**: "main" for primary images
+- **Fallback**: Uses primary image (order=255) when available
+
+## Frontend Integration
+
+### Service Layer (`services/acotService.js`)
+- **Purpose**: Replaces direct Klaviyo API calls with acot-server calls
+- **Methods**: `getStats()`, `getStatsDetails()`, `getProducts()`, `getProjection()`
+- **Logging**: Axios interceptors for request/response logging
+- **Environment**: Automatic URL handling (proxy in dev, direct in production)
+
+### Component Updates
+All 5 main components updated to use `acotService`:
+- **StatCards.jsx**: Main dashboard statistics
+- **MiniStatCards.jsx**: Compact statistics view
+- **SalesChart.jsx**: Revenue and order trends
+- **MiniSalesChart.jsx**: Compact chart view
+- **ProductGrid.jsx**: Top products table
+
+### Proxy Configuration (`vite.config.js`)
+```javascript
+'/api/acot': {
+  target: 'http://localhost:3007',
+  changeOrigin: true,
+  secure: false
+}
+```
+
+## Key Features
+
+### Complete Business Intelligence
+- **Revenue Analytics**: Total revenue, trends, projections
+- **Order Analysis**: Counts, types, status tracking
+- **Product Performance**: Top sellers, revenue contribution
+- **Shipping Intelligence**: Methods, locations, distribution
+- **Customer Insights**: Order value ranges, patterns
+- **Operational Metrics**: Refunds, cancellations, peak hours
+
+### Performance Optimizations
+- **Connection Pooling**: Efficient database connection management
+- **Query Optimization**: Indexed queries with proper WHERE clauses
+- **Caching Strategy**: Frontend caching for detail views
+- **Batch Processing**: Efficient data aggregation
+
+### Error Handling
+- **Database Connectivity**: Graceful handling of connection issues
+- **Query Failures**: Detailed error logging and user-friendly messages
+- **Data Validation**: Input sanitization and validation
+- **Fallback Mechanisms**: Default values for missing data
+
+## Simplified Elements
+
+Due to database complexity, some features are simplified:
+- **Brands**: Shows "Various Brands" (companies table structure complex)
+- **Categories**: Shows "General" (category relationships complex)
+
+These can be enhanced in future iterations with proper category mapping.
+
+## Testing
+
+Test the server functionality:
+
+```bash
+# Health check
+curl http://localhost:3007/api/acot/test
+
+# Today's stats
+curl http://localhost:3007/api/acot/events/stats?timeRange=today
+
+# Last 30 days with details
+curl http://localhost:3007/api/acot/events/stats/details?timeRange=last30days&daily=true
+
+# Top products
+curl http://localhost:3007/api/acot/events/products?timeRange=thisWeek
+
+# Revenue projection
+curl http://localhost:3007/api/acot/events/projection?timeRange=today
+```
+
+## Development Notes
+
+- **No Frontend Changes**: Complete drop-in replacement for Klaviyo
+- **API Compatibility**: Maintains exact response structure
+- **Business Logic**: Implements all complex e-commerce calculations
+- **Scalability**: Designed for production workloads
+- **Maintainability**: Well-documented code with clear separation of concerns
+
+## Future Enhancements
+
+- Enhanced category and brand mapping
+- Real-time notifications for significant events
+- Advanced analytics and forecasting
+- Customer segmentation analysis
+- Inventory integration 

inventory-server/dashboard/acot-server/db/connection.js

@@ -0,0 +1,302 @@
+// Per Deviation #13 in CONSOLIDATION_PLAN.md: `ssh2` is CJS and its named export
+// (`Client`) isn't reliably detected by Node's CJS→ESM interop static analysis.
+// Default-import + destructure is the bulletproof pattern.
+import ssh2 from 'ssh2';
+import mysql from 'mysql2/promise';
+import fs from 'node:fs';
+
+const { Client } = ssh2;
+
+// Connection pool configuration
+const connectionPool = {
+  connections: [],
+  maxConnections: 20,
+  currentConnections: 0,
+  pendingRequests: [],
+  // Cache for query results (key: query string, value: {data, timestamp})
+  queryCache: new Map(),
+  // Cache duration for different query types in milliseconds
+  cacheDuration: {
+    'stats': 60 * 1000,              // 1 minute for stats
+    'products': 5 * 60 * 1000,       // 5 minutes for products
+    'orders': 60 * 1000,             // 1 minute for orders
+    'default': 60 * 1000             // 1 minute default
+  },
+  // Circuit breaker state
+  circuitBreaker: {
+    failures: 0,
+    lastFailure: 0,
+    isOpen: false,
+    threshold: 5,
+    timeout: 30000 // 30 seconds
+  }
+};
+
+/**
+ * Get a database connection from the pool
+ * @returns {Promise<{connection: object, release: function}>} The database connection and release function
+ */
+async function getDbConnection() {
+  return new Promise(async (resolve, reject) => {
+    // Check circuit breaker
+    const now = Date.now();
+    if (connectionPool.circuitBreaker.isOpen) {
+      if (now - connectionPool.circuitBreaker.lastFailure > connectionPool.circuitBreaker.timeout) {
+        // Reset circuit breaker
+        connectionPool.circuitBreaker.isOpen = false;
+        connectionPool.circuitBreaker.failures = 0;
+        console.log('Circuit breaker reset');
+      } else {
+        reject(new Error('Circuit breaker is open - too many connection failures'));
+        return;
+      }
+    }
+    
+    // Check if there's an available connection in the pool
+    if (connectionPool.connections.length > 0) {
+      const conn = connectionPool.connections.pop();
+      console.log(`Using pooled connection. Pool size: ${connectionPool.connections.length}`);
+      resolve({
+        connection: conn.connection,
+        release: () => releaseConnection(conn)
+      });
+      return;
+    }
+    
+    // If we haven't reached max connections, create a new one
+    if (connectionPool.currentConnections < connectionPool.maxConnections) {
+      try {
+        console.log(`Creating new connection. Current: ${connectionPool.currentConnections}/${connectionPool.maxConnections}`);
+        connectionPool.currentConnections++;
+        
+        const tunnel = await setupSshTunnel();
+        const { ssh, stream, dbConfig } = tunnel;
+        
+        const connection = await mysql.createConnection({
+          ...dbConfig,
+          stream
+        });
+        
+        const conn = { ssh, connection, inUse: true, created: Date.now() };
+        
+        console.log('Database connection established');
+        
+        // Reset circuit breaker on successful connection
+        if (connectionPool.circuitBreaker.failures > 0) {
+          connectionPool.circuitBreaker.failures = 0;
+          connectionPool.circuitBreaker.isOpen = false;
+        }
+        
+        resolve({
+          connection: conn.connection,
+          release: () => releaseConnection(conn)
+        });
+      } catch (error) {
+        connectionPool.currentConnections--;
+        
+        // Track circuit breaker failures
+        connectionPool.circuitBreaker.failures++;
+        connectionPool.circuitBreaker.lastFailure = Date.now();
+        
+        if (connectionPool.circuitBreaker.failures >= connectionPool.circuitBreaker.threshold) {
+          connectionPool.circuitBreaker.isOpen = true;
+          console.log(`Circuit breaker opened after ${connectionPool.circuitBreaker.failures} failures`);
+        }
+        
+        reject(error);
+      }
+      return;
+    }
+    
+    // Pool is full, queue the request with timeout
+    console.log('Connection pool full, queuing request...');
+    const timeoutId = setTimeout(() => {
+      // Remove from queue if still there
+      const index = connectionPool.pendingRequests.findIndex(req => req.resolve === resolve);
+      if (index !== -1) {
+        connectionPool.pendingRequests.splice(index, 1);
+        reject(new Error('Connection pool queue timeout after 15 seconds'));
+      }
+          }, 15000);
+    
+    connectionPool.pendingRequests.push({ 
+      resolve, 
+      reject, 
+      timeoutId,
+      timestamp: Date.now()
+    });
+  });
+}
+
+/**
+ * Release a connection back to the pool
+ */
+function releaseConnection(conn) {
+  conn.inUse = false;
+  
+  // Check if there are pending requests
+  if (connectionPool.pendingRequests.length > 0) {
+    const { resolve, timeoutId } = connectionPool.pendingRequests.shift();
+    
+    // Clear the timeout since we're serving the request
+    if (timeoutId) {
+      clearTimeout(timeoutId);
+    }
+    
+    conn.inUse = true;
+    console.log(`Serving queued request. Queue length: ${connectionPool.pendingRequests.length}`);
+    resolve({
+      connection: conn.connection,
+      release: () => releaseConnection(conn)
+    });
+  } else {
+    // Return to pool
+    connectionPool.connections.push(conn);
+    console.log(`Connection returned to pool. Pool size: ${connectionPool.connections.length}, Active: ${connectionPool.currentConnections}`);
+  }
+}
+
+/**
+ * Get cached query results or execute query if not cached
+ * @param {string} cacheKey - Unique key to identify the query
+ * @param {string} queryType - Type of query (stats, products, orders, etc.)
+ * @param {Function} queryFn - Function to execute if cache miss
+ * @returns {Promise<any>} The query result
+ */
+async function getCachedQuery(cacheKey, queryType, queryFn) {
+  // Get cache duration based on query type
+  const cacheDuration = connectionPool.cacheDuration[queryType] || connectionPool.cacheDuration.default;
+  
+  // Check if we have a valid cached result
+  const cachedResult = connectionPool.queryCache.get(cacheKey);
+  const now = Date.now();
+  
+  if (cachedResult && (now - cachedResult.timestamp < cacheDuration)) {
+    console.log(`Cache hit for ${queryType} query: ${cacheKey}`);
+    return cachedResult.data;
+  }
+  
+  // No valid cache found, execute the query
+  console.log(`Cache miss for ${queryType} query: ${cacheKey}`);
+  const result = await queryFn();
+  
+  // Cache the result
+  connectionPool.queryCache.set(cacheKey, {
+    data: result,
+    timestamp: now
+  });
+  
+  return result;
+}
+
+/**
+ * Setup SSH tunnel to production database
+ * @private - Should only be used by getDbConnection
+ * @returns {Promise<{ssh: object, stream: object, dbConfig: object}>}
+ */
+async function setupSshTunnel() {
+  const sshConfig = {
+    host: process.env.PROD_SSH_HOST,
+    port: process.env.PROD_SSH_PORT || 22,
+    username: process.env.PROD_SSH_USER,
+    privateKey: process.env.PROD_SSH_KEY_PATH
+      ? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
+      : undefined,
+    compress: true
+  };
+
+  const dbConfig = {
+    host: process.env.PROD_DB_HOST || 'localhost',
+    user: process.env.PROD_DB_USER,
+    password: process.env.PROD_DB_PASSWORD,
+    database: process.env.PROD_DB_NAME,
+    port: process.env.PROD_DB_PORT || 3306,
+    timezone: 'Z'
+  };
+
+  return new Promise((resolve, reject) => {
+    const ssh = new Client();
+
+    ssh.on('error', (err) => {
+      console.error('SSH connection error:', err);
+      reject(err);
+    });
+
+    ssh.on('ready', () => {
+      ssh.forwardOut(
+        '127.0.0.1',
+        0,
+        dbConfig.host,
+        dbConfig.port,
+        (err, stream) => {
+          if (err) reject(err);
+          resolve({ ssh, stream, dbConfig });
+        }
+      );
+    }).connect(sshConfig);
+  });
+}
+
+/**
+ * Clear cached query results
+ * @param {string} [cacheKey] - Specific cache key to clear (clears all if not provided)
+ */
+function clearQueryCache(cacheKey) {
+  if (cacheKey) {
+    connectionPool.queryCache.delete(cacheKey);
+    console.log(`Cleared cache for key: ${cacheKey}`);
+  } else {
+    connectionPool.queryCache.clear();
+    console.log('Cleared all query cache');
+  }
+}
+
+/**
+ * Force close all active connections
+ * Useful for server shutdown or manual connection reset
+ */
+async function closeAllConnections() {
+  // Close all pooled connections
+  for (const conn of connectionPool.connections) {
+    try {
+      await conn.connection.end();
+      conn.ssh.end();
+      console.log('Closed pooled connection');
+    } catch (error) {
+      console.error('Error closing pooled connection:', error);
+    }
+  }
+  
+  // Reset pool state
+  connectionPool.connections = [];
+  connectionPool.currentConnections = 0;
+  connectionPool.pendingRequests = [];
+  connectionPool.queryCache.clear();
+  
+  console.log('All connections closed and pool reset');
+}
+
+/**
+ * Get connection pool status for debugging
+ */
+function getPoolStatus() {
+  return {
+    poolSize: connectionPool.connections.length,
+    activeConnections: connectionPool.currentConnections,
+    maxConnections: connectionPool.maxConnections,
+    pendingRequests: connectionPool.pendingRequests.length,
+    cacheSize: connectionPool.queryCache.size,
+    queuedRequests: connectionPool.pendingRequests.map(req => ({
+      waitTime: Date.now() - req.timestamp,
+      hasTimeout: !!req.timeoutId
+    }))
+  };
+}
+
+export {
+  getDbConnection,
+  getCachedQuery,
+  clearQueryCache,
+  closeAllConnections,
+  getPoolStatus,
+}; 

inventory-server/dashboard/acot-server/package.json

@@ -0,0 +1,25 @@
+{
+  "name": "acot-server",
+  "version": "1.0.0",
+  "description": "A Cherry On Top production database server",
+  "type": "module",
+  "main": "server.js",
+  "scripts": {
+    "start": "node server.js",
+    "dev": "nodemon server.js"
+  },
+  "dependencies": {
+    "compression": "^1.7.4",
+    "cors": "^2.8.5",
+    "dotenv": "^16.3.1",
+    "express": "^4.18.2",
+    "luxon": "^3.5.0",
+    "morgan": "^1.10.0",
+    "mysql2": "^3.6.5",
+    "pg": "^8.21.0",
+    "ssh2": "^1.14.0"
+  },
+  "devDependencies": {
+    "nodemon": "^3.0.1"
+  }
+}

inventory-server/dashboard/acot-server/routes/customers.js

@@ -0,0 +1,323 @@
+// Customer lookup for the phone app (acot-phone-server).
+//
+// All queries hit the MySQL `sg` database via the shared SSH-tunneled pool in
+// db/connection.js. The stats/orders logic mirrors the freescout
+// ACOTCustomerData module so both apps display the same numbers for a given
+// customer — the difference is that we key by phone, not email.
+//
+// NOTE: `users.phone` is not yet indexed in production. Admin will add
+// `idx_phone (phone)` — queries here assume that exists for acceptable latency.
+
+import express from 'express';
+import { getDbConnection, getCachedQuery } from '../db/connection.js';
+import { requirePhoneApiKey } from '../utils/phoneAuth.js';
+
+const router = express.Router();
+
+// Order status labels mirror ACOTCustomerDataServiceProvider.php.
+const ORDER_STATUS_LABEL = {
+  0: 'Created', 10: 'Incomplete', 15: 'Cancelled', 16: 'Combined',
+  20: 'Placed', 22: 'Placed (Incomplete)', 40: 'Awaiting Payment',
+  45: 'Payment Pending', 50: 'Awaiting Products', 55: 'Shipping Later',
+  56: 'Shipping Together', 60: 'Ready', 61: 'Flagged', 62: 'Fix Before Pick',
+  65: 'Manual Picking', 67: 'Remote Send', 70: 'In PT', 80: 'Picked',
+  90: 'Awaiting Shipment', 91: 'Remote Wait', 92: 'Awaiting Pickup',
+  93: 'Fix Before Ship', 95: 'Shipped (Confirmed)', 100: 'Shipped',
+};
+const ORDER_STATUS_SHORT = {
+  0: 'Created', 10: 'Incomplete', 15: 'Cancelled', 16: 'Combined',
+  20: 'Placed', 22: 'Plcd Incomp', 40: 'Await Payment', 45: 'Pymt Pending',
+  50: 'Await Products', 55: 'Ship Later', 56: 'Ship Togethr', 60: 'Ready',
+  61: 'Flagged', 62: 'Fix Bfr Pick', 65: 'Manual Pick', 67: 'Remote Send',
+  70: 'In PT', 80: 'Picked', 90: 'Await Ship', 91: 'Remote Wait',
+  92: 'Await Pickup', 93: 'Fix Bfr Ship', 95: 'Shpd Confirm', 100: 'Shipped',
+};
+
+function statusLabel(s) { return ORDER_STATUS_LABEL[s] ?? `Unknown (${s})`; }
+function statusShort(s) { return ORDER_STATUS_SHORT[s] ?? `Unknown (${s})`; }
+
+// SIP trunks and historical CRM imports all disagree on phone format. Rather
+// than normalize everything upstream, we search across the most common
+// variations for US/Canada numbers. Falls through to the raw input for
+// international numbers we can't safely reformat.
+function phoneVariations(input) {
+  const raw = String(input || '').trim();
+  if (!raw) return [];
+  const digits = raw.replace(/\D/g, '');
+  const out = new Set([raw, digits]);
+  if (digits.length === 10) {
+    out.add(`+1${digits}`);
+    out.add(`1${digits}`);
+  } else if (digits.length === 11 && digits.startsWith('1')) {
+    out.add(`+${digits}`);
+    out.add(digits.slice(1)); // 10-digit form
+    out.add(`+1${digits.slice(1)}`);
+  }
+  return Array.from(out).filter(Boolean);
+}
+
+function trackingLink(method, tracking) {
+  if (!tracking) return '';
+  if (typeof method === 'string') {
+    if (method.startsWith('usps_') || method === 'fedex_smartpost') {
+      return `https://tools.usps.com/go/TrackConfirmAction?qtc_tLabels1=${tracking}`;
+    }
+    if (method.startsWith('fedex_')) {
+      return `https://www.fedex.com/fedextrack/?trknbr=${tracking}`;
+    }
+  }
+  return '';
+}
+
+// Matches ACOTCustomerDataServiceProvider::imageUrl — sbing.com/i/products/<dir1>/<dir2>/<pid>-t-<iid>.jpg
+function imageUrl(pid, iid = 1) {
+  const padded = String(pid).padStart(10, '0');
+  const dir1 = padded.slice(0, 4);
+  const dir2 = padded.slice(4, 7);
+  return `https://sbing.com/i/products/${dir1}/${dir2}/${pid}-t-${iid}.jpg`;
+}
+
+router.use(requirePhoneApiKey);
+
+// ── GET /by-phone ──────────────────────────────────────────────────────────
+// Returns top-line customer info for the incoming-call overlay.
+router.get('/by-phone', async (req, res) => {
+  const phone = String(req.query.phone || '').trim();
+  if (!phone) return res.status(400).json({ success: false, error: 'phone required' });
+
+  const variations = phoneVariations(phone);
+  if (variations.length === 0) return res.json({ success: true, customer: null });
+
+  try {
+    const data = await getCachedQuery(
+      `customer-by-phone:${variations.join('|')}`,
+      'default',
+      async () => {
+        const { connection, release } = await getDbConnection();
+        try {
+          const placeholders = variations.map(() => '?').join(',');
+          // Tie-break by highest LTV per user instructions: subquery computes LTV
+          // for every matching user, then we pick the biggest.
+          const [users] = await connection.execute(
+            `SELECT u.cid, u.uid, u.firstname, u.lastname, u.email, u.phone, u.points,
+                    COALESCE((
+                      SELECT SUM(summary_total)
+                      FROM _order
+                      WHERE order_cid = u.cid AND order_status >= 50
+                    ), 0) AS lifetime_value,
+                    COALESCE((
+                      SELECT COUNT(*)
+                      FROM _order
+                      WHERE order_cid = u.cid AND order_status >= 20
+                    ), 0) AS num_orders,
+                    (
+                      SELECT AVG(summary_total)
+                      FROM _order
+                      WHERE order_cid = u.cid AND order_status >= 20
+                    ) AS avg_order
+             FROM users u
+             WHERE u.phone IN (${placeholders})
+             ORDER BY lifetime_value DESC
+             LIMIT 1`,
+            variations
+          );
+          return users[0] ?? null;
+        } finally {
+          release();
+        }
+      }
+    );
+
+    if (!data) return res.json({ success: true, customer: null });
+    res.json({
+      success: true,
+      customer: {
+        cid: Number(data.cid),
+        uid: data.uid,
+        firstName: data.firstname || null,
+        lastName: data.lastname || null,
+        email: data.email || null,
+        phone: data.phone,
+        points: Number(data.points) || 0,
+        lifetimeValue: Number(data.lifetime_value) || 0,
+        orderCount: Number(data.num_orders) || 0,
+        avgOrderValue: data.avg_order != null ? Number(data.avg_order) : 0,
+      },
+    });
+  } catch (err) {
+    console.error('customers/by-phone failed:', err);
+    res.status(500).json({ success: false, error: 'query_failed' });
+  }
+});
+
+// ── GET /search ────────────────────────────────────────────────────────────
+// Name search for the dialer. Accepts a free-text query; splits on whitespace.
+//  - 1 token: LIKE against firstname OR lastname (prefix).
+//  - 2+ tokens: firstname LIKE A% AND lastname LIKE B% (order-sensitive on purpose).
+router.get('/search', async (req, res) => {
+  const q = String(req.query.q || '').trim();
+  const limit = Math.min(Math.max(parseInt(req.query.limit || '10', 10) || 10, 1), 25);
+  if (q.length < 2) return res.json({ success: true, results: [] });
+
+  try {
+    const data = await getCachedQuery(
+      `customer-search:${q}:${limit}`,
+      'default',
+      async () => {
+        const { connection, release } = await getDbConnection();
+        try {
+          const tokens = q.split(/\s+/).filter(Boolean);
+          let sql;
+          let params;
+          if (tokens.length === 1) {
+            const pattern = `${tokens[0]}%`;
+            sql = `SELECT cid, firstname, lastname, email, phone
+                   FROM users
+                   WHERE (firstname LIKE ? OR lastname LIKE ?)
+                     AND phone <> ''
+                   ORDER BY lastname, firstname
+                   LIMIT ?`;
+            params = [pattern, pattern, limit];
+          } else {
+            const firstPat = `${tokens[0]}%`;
+            const lastPat = `${tokens.slice(1).join(' ')}%`;
+            sql = `SELECT cid, firstname, lastname, email, phone
+                   FROM users
+                   WHERE firstname LIKE ? AND lastname LIKE ?
+                     AND phone <> ''
+                   ORDER BY lastname, firstname
+                   LIMIT ?`;
+            params = [firstPat, lastPat, limit];
+          }
+          const [rows] = await connection.execute(sql, params);
+          return rows;
+        } finally {
+          release();
+        }
+      }
+    );
+
+    res.json({
+      success: true,
+      results: data.map((r) => ({
+        cid: Number(r.cid),
+        firstName: r.firstname || null,
+        lastName: r.lastname || null,
+        email: r.email || null,
+        phone: r.phone,
+      })),
+    });
+  } catch (err) {
+    console.error('customers/search failed:', err);
+    res.status(500).json({ success: false, error: 'query_failed' });
+  }
+});
+
+// ── GET /:cid/orders ───────────────────────────────────────────────────────
+// Recent orders for the active-call screen — mirrors the freescout sidebar.
+router.get('/:cid/orders', async (req, res) => {
+  const cid = Number(req.params.cid);
+  if (!Number.isFinite(cid) || cid <= 0) {
+    return res.status(400).json({ success: false, error: 'bad_cid' });
+  }
+
+  try {
+    const data = await getCachedQuery(
+      `customer-orders:${cid}`,
+      'orders',
+      async () => {
+        const { connection, release } = await getDbConnection();
+        try {
+          // MySQL-safe equivalent of the Laravel query in the freescout module.
+          // Active = placed OR shipped within the last 3 months.
+          const [ordersRaw] = await connection.execute(
+            `SELECT order_id, order_status, order_type, summary_total,
+                    date_placed, ship_method_type, ship_method_tracking,
+                    CASE
+                      WHEN (order_status BETWEEN 20 AND 92
+                            OR date_shipped > DATE_SUB(NOW(), INTERVAL 3 MONTH))
+                      THEN 1 ELSE 0
+                    END AS _is_active
+             FROM _order
+             WHERE order_cid = ?
+               AND (order_status >= 20
+                    OR date_shipped > DATE_SUB(NOW(), INTERVAL 3 MONTH))
+             ORDER BY _is_active DESC, date_placed DESC`,
+            [cid]
+          );
+
+          const active = ordersRaw.filter((o) => o._is_active === 1);
+          const inactive = ordersRaw.filter((o) => o._is_active === 0);
+          const orders = active.concat(inactive.slice(0, Math.max(0, 10 - active.length)));
+
+          if (orders.length === 0) return [];
+
+          const orderIds = orders.map((o) => o.order_id);
+          const idPlaceholders = orderIds.map(() => '?').join(',');
+
+          const [items] = await connection.execute(
+            `SELECT order_id, prod_pid, prod_itemnumber, prod_description, prod_price, qty_ordered
+             FROM order_items
+             WHERE order_id IN (${idPlaceholders})`,
+            orderIds
+          );
+
+          // Main-image lookup: per-pid highest \`order\` at type=3 (matches the
+          // freescout module's raw SQL).
+          const pids = [...new Set(items.map((i) => Number(i.prod_pid)).filter(Boolean))];
+          const mainImagesByPid = new Map();
+          if (pids.length > 0) {
+            const pidList = pids.join(',');
+            const [imgRows] = await connection.execute(
+              `SELECT pi.pid, pi.iid
+               FROM product_images pi
+               INNER JOIN (
+                 SELECT pid, MAX(\`order\`) AS max_order
+                 FROM product_images
+                 WHERE pid IN (${pidList}) AND type = 3
+                 GROUP BY pid
+               ) pm ON pi.pid = pm.pid AND pi.\`order\` = pm.max_order AND pi.type = 3`
+            );
+            for (const r of imgRows) mainImagesByPid.set(Number(r.pid), Number(r.iid));
+          }
+
+          const itemsByOrder = new Map();
+          for (const it of items) {
+            const oid = Number(it.order_id);
+            if (!itemsByOrder.has(oid)) itemsByOrder.set(oid, []);
+            const iid = mainImagesByPid.get(Number(it.prod_pid)) ?? 1;
+            itemsByOrder.get(oid).push({
+              pid: Number(it.prod_pid),
+              sku: it.prod_itemnumber || null,
+              name: it.prod_description || null,
+              price: Number(it.prod_price) || 0,
+              quantity: Number(it.qty_ordered) || 0,
+              imageUrl: imageUrl(it.prod_pid, iid),
+            });
+          }
+
+          return orders.map((o) => ({
+            orderId: Number(o.order_id),
+            datePlaced: o.date_placed,
+            total: Number(o.summary_total) || 0,
+            status: Number(o.order_status),
+            statusLabel: statusLabel(Number(o.order_status)),
+            statusShort: statusShort(Number(o.order_status)),
+            trackingNumber: o.ship_method_tracking || '',
+            trackingUrl: trackingLink(o.ship_method_type, o.ship_method_tracking),
+            items: itemsByOrder.get(Number(o.order_id)) || [],
+          }));
+        } finally {
+          release();
+        }
+      }
+    );
+
+    res.json({ success: true, orders: data });
+  } catch (err) {
+    console.error('customers/:cid/orders failed:', err);
+    res.status(500).json({ success: false, error: 'query_failed' });
+  }
+});
+
+export default router;

inventory-server/dashboard/acot-server/routes/discounts.js

@@ -0,0 +1,576 @@
+import express from 'express';
+import { DateTime } from 'luxon';
+import { getDbConnection } from '../db/connection.js';
+
+const router = express.Router();
+
+// Bucket boundaries by summary_subtotal (post-item-sale, pre-order-promo).
+// The final entry is open-ended: all orders >= the last bound land there.
+const RANGE_BOUNDS = [
+  10, 20, 30, 40, 50, 60, 70, 80, 90,
+  100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200,
+  300, 400, 500, 1000, 1500
+];
+
+const FINAL_BUCKET_KEY = '99999';
+
+function buildRangeDefinitions() {
+  const ranges = [];
+  let previous = 0;
+  for (const bound of RANGE_BOUNDS) {
+    const key = bound.toString().padStart(5, '0');
+    ranges.push({
+      min: previous,
+      max: bound,
+      label: `$${previous.toLocaleString()} - $${bound.toLocaleString()}`,
+      key,
+    });
+    previous = bound;
+  }
+  const lastBound = RANGE_BOUNDS[RANGE_BOUNDS.length - 1];
+  ranges.push({
+    min: lastBound,
+    max: null,
+    label: `$${lastBound.toLocaleString()}+`,
+    key: FINAL_BUCKET_KEY,
+  });
+  return ranges;
+}
+
+const RANGE_DEFINITIONS = buildRangeDefinitions();
+
+function bucketKeyFor(subtotal) {
+  for (const range of RANGE_DEFINITIONS) {
+    if (range.max == null) return range.key;
+    if (subtotal <= range.max) return range.key;
+  }
+  return FINAL_BUCKET_KEY;
+}
+
+const DEFAULT_POINT_DOLLAR_VALUE = 0.005;
+
+const DEFAULTS = {
+  merchantFeePercent: 2.9,
+  fixedCostPerOrder: 1.25,
+  pointDollarValue: DEFAULT_POINT_DOLLAR_VALUE,
+};
+
+function parseDate(value, fallback) {
+  if (!value) {
+    return fallback;
+  }
+  const parsed = DateTime.fromISO(value);
+  if (!parsed.isValid) {
+    return fallback;
+  }
+  return parsed;
+}
+
+function formatDateForSql(dt) {
+  return dt.toFormat('yyyy-LL-dd HH:mm:ss');
+}
+
+router.get('/promos', async (req, res) => {
+  let connection;
+  try {
+    const { connection: conn, release } = await getDbConnection();
+    connection = conn;
+    const releaseConnection = release;
+
+    const { startDate, endDate } = req.query || {};
+    const now = DateTime.now().endOf('day');
+    const defaultStart = now.minus({ years: 3 }).startOf('day');
+
+    const parsedStart = startDate ? parseDate(startDate, defaultStart).startOf('day') : defaultStart;
+    const parsedEnd = endDate ? parseDate(endDate, now).endOf('day') : now;
+
+    const rangeStart = parsedStart <= parsedEnd ? parsedStart : parsedEnd;
+    const rangeEnd = parsedEnd >= parsedStart ? parsedEnd : parsedStart;
+
+    const rangeStartSql = formatDateForSql(rangeStart);
+    const rangeEndSql = formatDateForSql(rangeEnd);
+
+    const sql = `
+      SELECT
+        p.promo_id AS id,
+        p.promo_code AS code,
+        p.promo_description_online AS description_online,
+        p.promo_description_private AS description_private,
+        p.date_start,
+        p.date_end,
+        COALESCE(u.usage_count, 0) AS usage_count
+      FROM promos p
+      LEFT JOIN (
+        SELECT
+          discount_code,
+          COUNT(DISTINCT order_id) AS usage_count
+        FROM order_discounts
+        WHERE discount_type = 10 AND discount_active = 1
+        GROUP BY discount_code
+      ) u ON u.discount_code = p.promo_id
+      WHERE p.date_start IS NOT NULL
+        AND p.date_end IS NOT NULL
+        AND NOT (p.date_end < ? OR p.date_start > ?)
+        AND p.store = 1
+        AND p.date_start >= '2010-01-01'
+      ORDER BY p.promo_id DESC
+      LIMIT 200
+    `;
+
+    const [rows] = await connection.execute(sql, [rangeStartSql, rangeEndSql]);
+    releaseConnection();
+
+    const promos = rows.map(row => ({
+      id: Number(row.id),
+      code: row.code,
+      description: row.description_online || row.description_private || '',
+      privateDescription: row.description_private || '',
+      promo_description_online: row.description_online || '',
+      promo_description_private: row.description_private || '',
+      dateStart: row.date_start,
+      dateEnd: row.date_end,
+      usageCount: Number(row.usage_count || 0)
+    }));
+
+    res.json({ promos });
+  } catch (error) {
+    if (connection) {
+      try {
+        connection.destroy();
+      } catch (destroyError) {
+        console.error('Failed to destroy connection after error:', destroyError);
+      }
+    }
+    console.error('Error fetching promos:', error);
+    res.status(500).json({ error: 'Failed to fetch promos' });
+  }
+});
+
+function emptyBucketAccumulator(range) {
+  return {
+    key: range.key,
+    label: range.label,
+    min: range.min,
+    max: range.max,
+    orderCount: 0,
+    sumOrderValue: 0,
+    sumProductDiscountAmount: 0,
+    sumPromoProductDiscount: 0,
+    sumCustomerItemCost: 0,
+    sumShippingChargeBase: 0,
+    sumShippingAfterAuto: 0,
+    sumShipPromoDiscount: 0,
+    sumShippingSurcharge: 0,
+    sumOrderSurcharge: 0,
+    sumCustomerShipCost: 0,
+    sumActualShippingCost: 0,
+    sumTotalRevenue: 0,
+    sumProductCogs: 0,
+    sumMerchantFees: 0,
+    sumPointsCost: 0,
+    sumFixedCosts: 0,
+    sumTotalCosts: 0,
+    sumProfit: 0,
+  };
+}
+
+function simulateOrder(order, config, derived) {
+  const orderValue = Number(order.summary_subtotal) || 0;
+  const retail = Number(order.summary_subtotal_retail) || orderValue;
+  const productDiscountAmount = Number(order.summary_discount_subtotal) || 0;
+  const pointsRedeemedDollars = Number(order.points_redeemed) || 0;
+  // summary_discount_subtotal is a kitchen-sink rollup that includes points
+  // redemptions (type 20). pointsCost already accrues for points awarded, so
+  // the points portion of historical discount must be excluded here to avoid
+  // double-counting it on orders that redeemed points.
+  const historicalProductDiscountExPoints = Math.max(0, productDiscountAmount - pointsRedeemedDollars);
+  const shippingChargeBase =
+    (Number(order.summary_shipping) || 0) + (Number(order.summary_shipping_rush) || 0);
+  const actualShippingCost = Number(order.ship_method_cost) || 0;
+  const cogs = Number(order.total_cogs) || 0;
+
+  let promoProductDiscount = 0;
+  if (config.productPromo.type === 'percentage_subtotal' && orderValue >= config.productPromo.minSubtotal) {
+    promoProductDiscount = orderValue * (config.productPromo.value / 100);
+  } else if (config.productPromo.type === 'percentage_regular' && orderValue >= config.productPromo.minSubtotal) {
+    const targetRate = config.productPromo.value / 100;
+    const targetCustomerPrice = retail * (1 - targetRate);
+    promoProductDiscount = Math.max(0, orderValue - targetCustomerPrice);
+  } else if (config.productPromo.type === 'fixed_amount' && orderValue >= config.productPromo.minSubtotal) {
+    promoProductDiscount = config.productPromo.value;
+  } else if (config.productPromo.type === 'none' && config.applyHistoricalProductPromo) {
+    promoProductDiscount = historicalProductDiscountExPoints;
+  }
+  promoProductDiscount = Math.max(0, Math.min(promoProductDiscount, orderValue));
+
+  let shippingAfterAuto = shippingChargeBase;
+  for (const tier of config.shippingTiers) {
+    if (orderValue >= tier.threshold) {
+      if (tier.mode === 'percentage') {
+        shippingAfterAuto = shippingChargeBase * Math.max(0, 1 - tier.value / 100);
+      } else if (tier.mode === 'flat') {
+        shippingAfterAuto = tier.value;
+      }
+    }
+  }
+
+  let shipPromoDiscount = 0;
+  if (config.shippingPromo.type !== 'none' && orderValue >= config.shippingPromo.minSubtotal) {
+    if (config.shippingPromo.type === 'percentage') {
+      shipPromoDiscount = shippingAfterAuto * (config.shippingPromo.value / 100);
+    } else if (config.shippingPromo.type === 'fixed') {
+      shipPromoDiscount = config.shippingPromo.value;
+    }
+    if (config.shippingPromo.maxDiscount > 0) {
+      shipPromoDiscount = Math.min(shipPromoDiscount, config.shippingPromo.maxDiscount);
+    }
+    shipPromoDiscount = Math.min(shipPromoDiscount, shippingAfterAuto);
+  }
+
+  let shippingSurcharge = 0;
+  let orderSurcharge = 0;
+  for (const surcharge of config.surcharges) {
+    const meetsMin = orderValue >= surcharge.threshold;
+    const meetsMax = surcharge.maxThreshold == null || orderValue < surcharge.maxThreshold;
+    if (meetsMin && meetsMax) {
+      if (surcharge.target === 'shipping') shippingSurcharge += surcharge.amount;
+      else if (surcharge.target === 'order') orderSurcharge += surcharge.amount;
+    }
+  }
+
+  const customerShipCost = Math.max(0, shippingAfterAuto - shipPromoDiscount + shippingSurcharge);
+  const customerItemCost = Math.max(0, orderValue - promoProductDiscount + orderSurcharge);
+  const totalRevenue = customerItemCost + customerShipCost;
+
+  const productCogs = config.cogsCalculationMode === 'average'
+    ? orderValue * derived.overallCogsPercentage
+    : cogs;
+
+  const merchantFees = totalRevenue * (config.merchantFeePercent / 100);
+  const pointsCost = orderValue * derived.pointsPerDollar * derived.redemptionRate * derived.pointDollarValue;
+  const fixedCosts = config.fixedCostPerOrder;
+  const totalCosts = productCogs + actualShippingCost + merchantFees + pointsCost + fixedCosts;
+  const profit = totalRevenue - totalCosts;
+
+  return {
+    orderValue,
+    productDiscountAmount,
+    promoProductDiscount,
+    customerItemCost,
+    shippingChargeBase,
+    shippingAfterAuto,
+    shipPromoDiscount,
+    shippingSurcharge,
+    orderSurcharge,
+    customerShipCost,
+    actualShippingCost,
+    totalRevenue,
+    productCogs,
+    merchantFees,
+    pointsCost,
+    fixedCosts,
+    totalCosts,
+    profit,
+  };
+}
+
+function accumulate(bucket, sim) {
+  bucket.orderCount += 1;
+  bucket.sumOrderValue += sim.orderValue;
+  bucket.sumProductDiscountAmount += sim.productDiscountAmount;
+  bucket.sumPromoProductDiscount += sim.promoProductDiscount;
+  bucket.sumCustomerItemCost += sim.customerItemCost;
+  bucket.sumShippingChargeBase += sim.shippingChargeBase;
+  bucket.sumShippingAfterAuto += sim.shippingAfterAuto;
+  bucket.sumShipPromoDiscount += sim.shipPromoDiscount;
+  bucket.sumShippingSurcharge += sim.shippingSurcharge;
+  bucket.sumOrderSurcharge += sim.orderSurcharge;
+  bucket.sumCustomerShipCost += sim.customerShipCost;
+  bucket.sumActualShippingCost += sim.actualShippingCost;
+  bucket.sumTotalRevenue += sim.totalRevenue;
+  bucket.sumProductCogs += sim.productCogs;
+  bucket.sumMerchantFees += sim.merchantFees;
+  bucket.sumPointsCost += sim.pointsCost;
+  bucket.sumFixedCosts += sim.fixedCosts;
+  bucket.sumTotalCosts += sim.totalCosts;
+  bucket.sumProfit += sim.profit;
+}
+
+function finalizeBucket(b, totalOrders) {
+  const n = b.orderCount;
+  const avg = (sum) => (n > 0 ? sum / n : 0);
+  return {
+    key: b.key,
+    label: b.label,
+    min: b.min,
+    max: b.max,
+    orderCount: n,
+    weight: totalOrders > 0 ? n / totalOrders : 0,
+    orderValue: avg(b.sumOrderValue),
+    productDiscountAmount: avg(b.sumProductDiscountAmount),
+    promoProductDiscount: avg(b.sumPromoProductDiscount),
+    customerItemCost: avg(b.sumCustomerItemCost),
+    shippingChargeBase: avg(b.sumShippingChargeBase),
+    shippingAfterAuto: avg(b.sumShippingAfterAuto),
+    shipPromoDiscount: avg(b.sumShipPromoDiscount),
+    shippingSurcharge: avg(b.sumShippingSurcharge),
+    orderSurcharge: avg(b.sumOrderSurcharge),
+    customerShipCost: avg(b.sumCustomerShipCost),
+    actualShippingCost: avg(b.sumActualShippingCost),
+    totalRevenue: avg(b.sumTotalRevenue),
+    productCogs: avg(b.sumProductCogs),
+    merchantFees: avg(b.sumMerchantFees),
+    pointsCost: avg(b.sumPointsCost),
+    fixedCosts: avg(b.sumFixedCosts),
+    totalCosts: avg(b.sumTotalCosts),
+    profit: avg(b.sumProfit),
+    profitPercent: b.sumTotalRevenue > 0 ? b.sumProfit / b.sumTotalRevenue : 0,
+  };
+}
+
+router.post('/simulate', async (req, res) => {
+  const {
+    dateRange = {},
+    filters = {},
+    productPromo = {},
+    shippingPromo = {},
+    shippingTiers = [],
+    surcharges = [],
+    merchantFeePercent,
+    fixedCostPerOrder,
+    cogsCalculationMode = 'actual',
+    applyHistoricalProductPromo = false,
+    pointsConfig = {}
+  } = req.body || {};
+
+  const endDefault = DateTime.now();
+  const startDefault = endDefault.minus({ months: 6 });
+  const startDt = parseDate(dateRange.start, startDefault).startOf('day');
+  const endDt = parseDate(dateRange.end, endDefault).endOf('day');
+
+  const shipCountry = filters.shipCountry || 'US';
+  const promoIds = Array.from(
+    new Set(
+      [
+        ...(Array.isArray(filters.promoIds) ? filters.promoIds : []),
+        ...(Array.isArray(filters.promoCodes) ? filters.promoCodes : []),
+      ]
+        .map((value) => {
+          if (typeof value === 'string') return value.trim();
+          if (typeof value === 'number') return String(value);
+          return '';
+        })
+        .filter((value) => value.length > 0)
+    )
+  );
+
+  const config = {
+    merchantFeePercent: typeof merchantFeePercent === 'number' ? merchantFeePercent : DEFAULTS.merchantFeePercent,
+    fixedCostPerOrder: typeof fixedCostPerOrder === 'number' ? fixedCostPerOrder : DEFAULTS.fixedCostPerOrder,
+    cogsCalculationMode,
+    applyHistoricalProductPromo: applyHistoricalProductPromo === true,
+    productPromo: {
+      type: productPromo.type || 'none',
+      value: Number(productPromo.value || 0),
+      minSubtotal: Number(productPromo.minSubtotal || 0)
+    },
+    shippingPromo: {
+      type: shippingPromo.type || 'none',
+      value: Number(shippingPromo.value || 0),
+      minSubtotal: Number(shippingPromo.minSubtotal || 0),
+      maxDiscount: Number(shippingPromo.maxDiscount || 0)
+    },
+    shippingTiers: Array.isArray(shippingTiers)
+      ? shippingTiers
+          .map(tier => ({
+            threshold: Number(tier.threshold || 0),
+            mode: tier.mode === 'percentage' || tier.mode === 'flat' ? tier.mode : 'percentage',
+            value: Number(tier.value || 0)
+          }))
+          .filter(tier => tier.threshold >= 0 && tier.value >= 0)
+          .sort((a, b) => a.threshold - b.threshold)
+      : [],
+    surcharges: Array.isArray(surcharges)
+      ? surcharges
+          .map(s => ({
+            threshold: Number(s.threshold || 0),
+            maxThreshold: typeof s.maxThreshold === 'number' && s.maxThreshold > 0 ? s.maxThreshold : null,
+            target: s.target === 'shipping' || s.target === 'order' ? s.target : 'shipping',
+            amount: Number(s.amount || 0)
+          }))
+          .filter(s => s.threshold >= 0 && s.amount >= 0)
+          .sort((a, b) => a.threshold - b.threshold)
+      : [],
+    points: {
+      pointsPerDollar: typeof pointsConfig.pointsPerDollar === 'number' ? pointsConfig.pointsPerDollar : null,
+      redemptionRate: typeof pointsConfig.redemptionRate === 'number' ? pointsConfig.redemptionRate : null,
+      pointDollarValue: typeof pointsConfig.pointDollarValue === 'number'
+        ? pointsConfig.pointDollarValue
+        : DEFAULT_POINT_DOLLAR_VALUE
+    }
+  };
+
+  let connection;
+  let release;
+
+  try {
+    const dbConn = await getDbConnection();
+    connection = dbConn.connection;
+    release = dbConn.release;
+
+    const params = [shipCountry, formatDateForSql(startDt), formatDateForSql(endDt)];
+    let promoExistsClause = '';
+    if (promoIds.length > 0) {
+      const placeholders = promoIds.map(() => '?').join(',');
+      promoExistsClause = `
+        AND EXISTS (
+          SELECT 1 FROM order_discounts od
+          WHERE od.order_id = o.order_id
+            AND od.discount_active = 1
+            AND od.discount_type = 10
+            AND od.discount_code IN (${placeholders})
+        )
+      `;
+      params.push(...promoIds);
+    }
+
+    const ordersQuery = `
+      SELECT
+        o.order_id,
+        o.summary_subtotal,
+        COALESCE(o.summary_subtotal_retail, o.summary_subtotal) AS summary_subtotal_retail,
+        COALESCE(o.summary_discount_subtotal, 0) AS summary_discount_subtotal,
+        COALESCE(o.summary_shipping, 0) AS summary_shipping,
+        COALESCE(o.summary_shipping_rush, 0) AS summary_shipping_rush,
+        COALESCE(o.ship_method_cost, 0) AS ship_method_cost,
+        COALESCE(o.summary_points, 0) AS summary_points,
+        COALESCE(c.total_cogs, 0) AS total_cogs,
+        COALESCE(p.points_redeemed, 0) AS points_redeemed
+      FROM _order o
+      LEFT JOIN (
+        SELECT order_id, SUM(cogs_amount) AS total_cogs
+        FROM report_sales_data
+        WHERE action IN (1,2,3)
+        GROUP BY order_id
+      ) c ON c.order_id = o.order_id
+      LEFT JOIN (
+        SELECT order_id, SUM(discount_amount_subtotal) AS points_redeemed
+        FROM order_discounts
+        WHERE discount_type = 20 AND discount_active = 1
+        GROUP BY order_id
+      ) p ON p.order_id = o.order_id
+      WHERE o.summary_total > 0
+        AND o.order_status >= 20
+        AND o.ship_method_selected <> 'holdit'
+        AND o.ship_country = ?
+        AND o.date_placed BETWEEN ? AND ?
+        ${promoExistsClause}
+    `;
+
+    const [orders] = await connection.execute(ordersQuery, params);
+
+    if (release) {
+      release();
+      release = null;
+    }
+
+    let totalSubtotal = 0;
+    let totalProductDiscount = 0;
+    let totalCogs = 0;
+    let totalPointsAwarded = 0;
+    let totalPointsRedeemedDollars = 0;
+    for (const o of orders) {
+      totalSubtotal += Number(o.summary_subtotal) || 0;
+      totalProductDiscount += Number(o.summary_discount_subtotal) || 0;
+      totalCogs += Number(o.total_cogs) || 0;
+      totalPointsAwarded += Number(o.summary_points) || 0;
+      totalPointsRedeemedDollars += Number(o.points_redeemed) || 0;
+    }
+
+    const productDiscountRate = totalSubtotal > 0 ? totalProductDiscount / totalSubtotal : 0;
+    const overallCogsPercentage = totalSubtotal > 0 ? totalCogs / totalSubtotal : 0;
+    const pointsPerDollar = config.points.pointsPerDollar != null
+      ? config.points.pointsPerDollar
+      : (totalSubtotal > 0 ? totalPointsAwarded / totalSubtotal : 0);
+    const pointDollarValue = config.points.pointDollarValue || DEFAULT_POINT_DOLLAR_VALUE;
+    let redemptionRate;
+    if (config.points.redemptionRate != null) {
+      redemptionRate = config.points.redemptionRate;
+    } else if (totalPointsAwarded > 0 && pointDollarValue > 0) {
+      const totalRedeemedPoints = totalPointsRedeemedDollars / pointDollarValue;
+      redemptionRate = Math.min(1, totalRedeemedPoints / totalPointsAwarded);
+    } else {
+      redemptionRate = 0;
+    }
+
+    const derived = {
+      overallCogsPercentage,
+      pointsPerDollar,
+      redemptionRate,
+      pointDollarValue,
+    };
+
+    const buckets = new Map();
+    for (const range of RANGE_DEFINITIONS) {
+      buckets.set(range.key, emptyBucketAccumulator(range));
+    }
+
+    let grandTotalProfit = 0;
+    let grandTotalRevenue = 0;
+
+    for (const order of orders) {
+      const sim = simulateOrder(order, config, derived);
+      const bucketKey = bucketKeyFor(sim.orderValue);
+      const bucket = buckets.get(bucketKey);
+      accumulate(bucket, sim);
+      grandTotalProfit += sim.profit;
+      grandTotalRevenue += sim.totalRevenue;
+    }
+
+    const totalOrders = orders.length;
+    const bucketResults = RANGE_DEFINITIONS.map((range) =>
+      finalizeBucket(buckets.get(range.key), totalOrders)
+    );
+
+    const weightedProfitAmount = totalOrders > 0 ? grandTotalProfit / totalOrders : 0;
+    const weightedProfitPercent = grandTotalRevenue > 0 ? grandTotalProfit / grandTotalRevenue : 0;
+
+    res.json({
+      dateRange: {
+        start: startDt.toISO(),
+        end: endDt.toISO()
+      },
+      totals: {
+        orders: totalOrders,
+        subtotal: totalSubtotal,
+        productDiscountRate,
+        pointsPerDollar,
+        redemptionRate,
+        pointDollarValue,
+        weightedProfitAmount,
+        weightedProfitPercent,
+        overallCogsPercentage: cogsCalculationMode === 'average' ? overallCogsPercentage : undefined
+      },
+      buckets: bucketResults
+    });
+  } catch (error) {
+    if (release) {
+      try {
+        release();
+      } catch (releaseError) {
+        console.error('Failed to release connection after error:', releaseError);
+      }
+    } else if (connection) {
+      try {
+        connection.destroy();
+      } catch (destroyError) {
+        console.error('Failed to destroy connection after error:', destroyError);
+      }
+    }
+
+    console.error('Error running discount simulation:', error);
+    res.status(500).json({ error: 'Failed to run discount simulation' });
+  }
+});
+
+export default router;

inventory-server/dashboard/acot-server/routes/employee-metrics.js

@@ -0,0 +1,680 @@
+import express from 'express';
+import { DateTime } from 'luxon';
+import { getDbConnection, getPoolStatus } from '../db/connection.js';
+import { getTimeRangeConditions, _internal as timeHelpers } from '../utils/timeUtils.js';
+
+const router = express.Router();
+
+const TIMEZONE = 'America/New_York';
+
+// Punch types from the database
+const PUNCH_TYPES = {
+  OUT: 0,
+  IN: 1,
+  BREAK_START: 2,
+  BREAK_END: 3,
+};
+
+// Standard hours for FTE calculation (40 hours per week)
+const STANDARD_WEEKLY_HOURS = 40;
+
+/**
+ * Calculate working hours from timeclock entries
+ * Groups punches by employee and date, pairs in/out punches
+ * Returns both total hours (with breaks, for FTE) and productive hours (without breaks, for productivity)
+ */
+function calculateHoursFromPunches(punches) {
+  // Group by employee
+  const byEmployee = new Map();
+
+  punches.forEach(punch => {
+    if (!byEmployee.has(punch.EmployeeID)) {
+      byEmployee.set(punch.EmployeeID, []);
+    }
+    byEmployee.get(punch.EmployeeID).push(punch);
+  });
+
+  const employeeHours = [];
+  let totalHours = 0;
+  let totalBreakHours = 0;
+
+  byEmployee.forEach((employeePunches, employeeId) => {
+    // Sort by timestamp
+    employeePunches.sort((a, b) => new Date(a.TimeStamp) - new Date(b.TimeStamp));
+
+    let hours = 0;
+    let breakHours = 0;
+    let currentIn = null;
+    let breakStart = null;
+
+    employeePunches.forEach(punch => {
+      const punchTime = new Date(punch.TimeStamp);
+
+      switch (punch.PunchType) {
+        case PUNCH_TYPES.IN:
+          currentIn = punchTime;
+          break;
+        case PUNCH_TYPES.OUT:
+          if (currentIn) {
+            hours += (punchTime - currentIn) / (1000 * 60 * 60); // Convert ms to hours
+            currentIn = null;
+          }
+          break;
+        case PUNCH_TYPES.BREAK_START:
+          breakStart = punchTime;
+          break;
+        case PUNCH_TYPES.BREAK_END:
+          if (breakStart) {
+            breakHours += (punchTime - breakStart) / (1000 * 60 * 60);
+            breakStart = null;
+          }
+          break;
+      }
+    });
+
+    totalHours += hours;
+    totalBreakHours += breakHours;
+
+    employeeHours.push({
+      employeeId,
+      hours,
+      breakHours,
+      productiveHours: hours - breakHours,
+    });
+  });
+
+  return {
+    employeeHours,
+    totalHours,
+    totalBreakHours,
+    totalProductiveHours: totalHours - totalBreakHours
+  };
+}
+
+/**
+ * Calculate FTE (Full Time Equivalents) for a period
+ * @param {number} totalHours - Total hours worked
+ * @param {Date} startDate - Period start
+ * @param {Date} endDate - Period end
+ */
+function calculateFTE(totalHours, startDate, endDate) {
+  const start = new Date(startDate);
+  const end = new Date(endDate);
+  const days = Math.max(1, (end - start) / (1000 * 60 * 60 * 24));
+  const weeks = days / 7;
+  const expectedHours = weeks * STANDARD_WEEKLY_HOURS;
+
+  return expectedHours > 0 ? totalHours / expectedHours : 0;
+}
+
+// Main employee metrics endpoint
+router.get('/', async (req, res) => {
+  const startTime = Date.now();
+  console.log(`[EMPLOYEE-METRICS] Starting request for timeRange: ${req.query.timeRange}`);
+
+  const timeoutPromise = new Promise((_, reject) => {
+    setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
+  });
+
+  try {
+    const mainOperation = async () => {
+      const { timeRange, startDate, endDate } = req.query;
+      console.log(`[EMPLOYEE-METRICS] Getting DB connection...`);
+      const { connection, release } = await getDbConnection();
+      console.log(`[EMPLOYEE-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
+
+      const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
+
+      // Adapt where clause for timeclock table (uses TimeStamp instead of date_placed)
+      const timeclockWhere = whereClause.replace(/date_placed/g, 'tc.TimeStamp');
+
+      // Query for timeclock data with employee names
+      const timeclockQuery = `
+        SELECT
+          tc.EmployeeID,
+          tc.TimeStamp,
+          tc.PunchType,
+          e.firstname,
+          e.lastname
+        FROM timeclock tc
+        LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
+        WHERE ${timeclockWhere}
+          AND e.hidden = 0
+          AND e.disabled = 0
+        ORDER BY tc.EmployeeID, tc.TimeStamp
+      `;
+
+      const [timeclockRows] = await connection.execute(timeclockQuery, params);
+
+      // Calculate hours (includes both total hours for FTE and productive hours for productivity)
+      const { employeeHours, totalHours, totalBreakHours, totalProductiveHours } = calculateHoursFromPunches(timeclockRows);
+
+      // Get employee names for the results
+      const employeeNames = new Map();
+      timeclockRows.forEach(row => {
+        if (!employeeNames.has(row.EmployeeID)) {
+          employeeNames.set(row.EmployeeID, {
+            firstname: row.firstname || '',
+            lastname: row.lastname || '',
+          });
+        }
+      });
+
+      // Enrich employee hours with names
+      const enrichedEmployeeHours = employeeHours.map(eh => ({
+        ...eh,
+        name: employeeNames.has(eh.employeeId)
+          ? `${employeeNames.get(eh.employeeId).firstname} ${employeeNames.get(eh.employeeId).lastname}`.trim()
+          : `Employee ${eh.employeeId}`,
+      })).sort((a, b) => b.hours - a.hours);
+
+      // Query for picking tickets - using subquery to avoid duplication from bucket join
+      // Ship-together orders: only count main orders (is_sub = 0 or NULL), not sub-orders
+      const pickingWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
+
+      // First get picking ticket stats without the bucket join (to avoid duplication)
+      const pickingStatsQuery = `
+        SELECT
+          pt.createdby as employeeId,
+          e.firstname,
+          e.lastname,
+          COUNT(DISTINCT pt.pickingid) as ticketCount,
+          SUM(pt.totalpieces_picked) as piecesPicked,
+          SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds,
+          AVG(NULLIF(pt.picking_speed, 0)) as avgPickingSpeed
+        FROM picking_ticket pt
+        LEFT JOIN employees e ON pt.createdby = e.employeeid
+        WHERE ${pickingWhere}
+          AND pt.closeddate IS NOT NULL
+        GROUP BY pt.createdby, e.firstname, e.lastname
+      `;
+
+      // Separate query for order counts (needs bucket join for ship-together handling)
+      const orderCountQuery = `
+        SELECT
+          pt.createdby as employeeId,
+          COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
+        FROM picking_ticket pt
+        LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+        WHERE ${pickingWhere}
+          AND pt.closeddate IS NOT NULL
+        GROUP BY pt.createdby
+      `;
+
+      const [[pickingStatsRows], [orderCountRows]] = await Promise.all([
+        connection.execute(pickingStatsQuery, params),
+        connection.execute(orderCountQuery, params)
+      ]);
+
+      // Merge the results
+      const orderCountMap = new Map();
+      orderCountRows.forEach(row => {
+        orderCountMap.set(row.employeeId, parseInt(row.ordersPicked || 0));
+      });
+
+      // Aggregate picking totals
+      let totalOrdersPicked = 0;
+      let totalPiecesPicked = 0;
+      let totalTickets = 0;
+      let totalPickingTimeSeconds = 0;
+      let pickingSpeedSum = 0;
+      let pickingSpeedCount = 0;
+
+      const pickingByEmployee = pickingStatsRows.map(row => {
+        const ordersPicked = orderCountMap.get(row.employeeId) || 0;
+        totalOrdersPicked += ordersPicked;
+        totalPiecesPicked += parseInt(row.piecesPicked || 0);
+        totalTickets += parseInt(row.ticketCount || 0);
+        totalPickingTimeSeconds += parseInt(row.pickingTimeSeconds || 0);
+        if (row.avgPickingSpeed && row.avgPickingSpeed > 0) {
+          pickingSpeedSum += parseFloat(row.avgPickingSpeed);
+          pickingSpeedCount++;
+        }
+
+        const empPickingHours = parseInt(row.pickingTimeSeconds || 0) / 3600;
+
+        return {
+          employeeId: row.employeeId,
+          name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeId}`,
+          ticketCount: parseInt(row.ticketCount || 0),
+          ordersPicked,
+          piecesPicked: parseInt(row.piecesPicked || 0),
+          pickingHours: empPickingHours,
+          avgPickingSpeed: row.avgPickingSpeed ? parseFloat(row.avgPickingSpeed) : null,
+        };
+      });
+
+      const totalPickingHours = totalPickingTimeSeconds / 3600;
+      const avgPickingSpeed = pickingSpeedCount > 0 ? pickingSpeedSum / pickingSpeedCount : 0;
+
+      // Query for shipped orders - totals
+      // Ship-together orders: only count main orders (order_type != 8 for sub-orders, or use parent tracking)
+      const shippingWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
+
+      const shippingQuery = `
+        SELECT
+          COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+          COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+        FROM _order o
+        WHERE ${shippingWhere}
+          AND o.order_status IN (100, 92)
+      `;
+
+      const [shippingRows] = await connection.execute(shippingQuery, params);
+      const shipping = shippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
+
+      // Query for shipped orders by employee
+      const shippingByEmployeeQuery = `
+        SELECT
+          e.employeeid,
+          e.firstname,
+          e.lastname,
+          COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+          COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+        FROM _order o
+        JOIN employees e ON o.stats_cid_shipped = e.cid
+        WHERE ${shippingWhere}
+          AND o.order_status IN (100, 92)
+          AND e.hidden = 0
+          AND e.disabled = 0
+        GROUP BY e.employeeid, e.firstname, e.lastname
+        ORDER BY ordersShipped DESC
+      `;
+
+      const [shippingByEmployeeRows] = await connection.execute(shippingByEmployeeQuery, params);
+      const shippingByEmployee = shippingByEmployeeRows.map(row => ({
+        employeeId: row.employeeid,
+        name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeid}`,
+        ordersShipped: parseInt(row.ordersShipped || 0),
+        piecesShipped: parseInt(row.piecesShipped || 0),
+      }));
+
+      // Calculate period dates for FTE calculation
+      let periodStart, periodEnd;
+      if (dateRange?.start) {
+        periodStart = new Date(dateRange.start);
+      } else if (params[0]) {
+        periodStart = new Date(params[0]);
+      } else {
+        periodStart = new Date();
+        periodStart.setDate(periodStart.getDate() - 30);
+      }
+
+      if (dateRange?.end) {
+        periodEnd = new Date(dateRange.end);
+      } else if (params[1]) {
+        periodEnd = new Date(params[1]);
+      } else {
+        periodEnd = new Date();
+      }
+
+      const fte = calculateFTE(totalHours, periodStart, periodEnd);
+      const activeEmployees = enrichedEmployeeHours.filter(e => e.hours > 0).length;
+
+      // Calculate weeks in period for weekly averages
+      const periodDays = Math.max(1, (periodEnd - periodStart) / (1000 * 60 * 60 * 24));
+      const weeksInPeriod = periodDays / 7;
+
+      // Get daily trend data for hours
+      // Use DATE_FORMAT to get date string in Eastern timezone, avoiding JS timezone conversion issues
+      // Business day starts at 1 AM, so subtract 1 hour before taking the date
+      const trendWhere = whereClause.replace(/date_placed/g, 'tc.TimeStamp');
+      const trendQuery = `
+        SELECT
+          DATE_FORMAT(DATE_SUB(tc.TimeStamp, INTERVAL 1 HOUR), '%Y-%m-%d') as date,
+          tc.EmployeeID,
+          tc.TimeStamp,
+          tc.PunchType
+        FROM timeclock tc
+        LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
+        WHERE ${trendWhere}
+          AND e.hidden = 0
+          AND e.disabled = 0
+        ORDER BY date, tc.EmployeeID, tc.TimeStamp
+      `;
+
+      const [trendRows] = await connection.execute(trendQuery, params);
+
+      // Get daily picking data for trend
+      // Ship-together orders: only count main orders (is_sub = 0 or NULL)
+      // Use DATE_FORMAT for consistent date string format
+      const pickingTrendWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
+      const pickingTrendQuery = `
+        SELECT
+          DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d') as date,
+          COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked,
+          COALESCE(SUM(pt.totalpieces_picked), 0) as piecesPicked
+        FROM picking_ticket pt
+        LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+        WHERE ${pickingTrendWhere}
+          AND pt.closeddate IS NOT NULL
+        GROUP BY DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d')
+        ORDER BY date
+      `;
+
+      const [pickingTrendRows] = await connection.execute(pickingTrendQuery, params);
+
+      // Create a map of picking data by date
+      const pickingByDate = new Map();
+      pickingTrendRows.forEach(row => {
+        // Date is already a string in YYYY-MM-DD format from DATE_FORMAT
+        const date = String(row.date);
+        pickingByDate.set(date, {
+          ordersPicked: parseInt(row.ordersPicked || 0),
+          piecesPicked: parseInt(row.piecesPicked || 0),
+        });
+      });
+
+      // Group timeclock by date for trend
+      const byDate = new Map();
+      trendRows.forEach(row => {
+        // Date is already a string in YYYY-MM-DD format from DATE_FORMAT
+        const date = String(row.date);
+        if (!byDate.has(date)) {
+          byDate.set(date, []);
+        }
+        byDate.get(date).push(row);
+      });
+
+      // Generate all dates in the period range for complete trend data
+      const allDatesInRange = [];
+      const startDt = DateTime.fromJSDate(periodStart).setZone(TIMEZONE).startOf('day');
+      const endDt = DateTime.fromJSDate(periodEnd).setZone(TIMEZONE).startOf('day');
+
+      let currentDt = startDt;
+      while (currentDt <= endDt) {
+        allDatesInRange.push(currentDt.toFormat('yyyy-MM-dd'));
+        currentDt = currentDt.plus({ days: 1 });
+      }
+
+      // Build trend data for all dates in range, filling zeros for missing days
+      const trend = allDatesInRange.map(date => {
+        const punches = byDate.get(date) || [];
+        const { totalHours: dayHours, employeeHours: dayEmployeeHours } = calculateHoursFromPunches(punches);
+        const picking = pickingByDate.get(date) || { ordersPicked: 0, piecesPicked: 0 };
+
+        // Parse date string in Eastern timezone to get proper ISO timestamp
+        const dateDt = DateTime.fromFormat(date, 'yyyy-MM-dd', { zone: TIMEZONE });
+
+        return {
+          date,
+          timestamp: dateDt.toISO(),
+          hours: dayHours,
+          activeEmployees: dayEmployeeHours.filter(e => e.hours > 0).length,
+          ordersPicked: picking.ordersPicked,
+          piecesPicked: picking.piecesPicked,
+        };
+      });
+
+      // Get previous period data for comparison
+      const previousRange = getPreviousPeriodRange(timeRange, startDate, endDate);
+      let comparison = null;
+      let previousTotals = null;
+
+      if (previousRange) {
+        const prevTimeclockWhere = previousRange.whereClause.replace(/date_placed/g, 'tc.TimeStamp');
+
+        const [prevTimeclockRows] = await connection.execute(
+          `SELECT tc.EmployeeID, tc.TimeStamp, tc.PunchType
+           FROM timeclock tc
+           LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
+           WHERE ${prevTimeclockWhere}
+             AND e.hidden = 0
+             AND e.disabled = 0
+           ORDER BY tc.EmployeeID, tc.TimeStamp`,
+          previousRange.params
+        );
+
+        const {
+          totalHours: prevTotalHours,
+          totalProductiveHours: prevProductiveHours,
+          employeeHours: prevEmployeeHours
+        } = calculateHoursFromPunches(prevTimeclockRows);
+        const prevActiveEmployees = prevEmployeeHours.filter(e => e.hours > 0).length;
+
+        // Previous picking data (ship-together fix applied)
+        // Use separate queries to avoid duplication from bucket join
+        const prevPickingWhere = previousRange.whereClause.replace(/date_placed/g, 'pt.createddate');
+
+        const [[prevPickingStatsRows], [prevOrderCountRows]] = await Promise.all([
+          connection.execute(
+            `SELECT
+               SUM(pt.totalpieces_picked) as piecesPicked,
+               SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds
+             FROM picking_ticket pt
+             WHERE ${prevPickingWhere}
+               AND pt.closeddate IS NOT NULL`,
+            previousRange.params
+          ),
+          connection.execute(
+            `SELECT
+               COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
+             FROM picking_ticket pt
+             LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+             WHERE ${prevPickingWhere}
+               AND pt.closeddate IS NOT NULL`,
+            previousRange.params
+          )
+        ]);
+
+        const prevPickingStats = prevPickingStatsRows[0] || { piecesPicked: 0, pickingTimeSeconds: 0 };
+        const prevOrderCount = prevOrderCountRows[0] || { ordersPicked: 0 };
+        const prevPicking = {
+          ordersPicked: parseInt(prevOrderCount.ordersPicked || 0),
+          piecesPicked: parseInt(prevPickingStats.piecesPicked || 0),
+          pickingTimeSeconds: parseInt(prevPickingStats.pickingTimeSeconds || 0)
+        };
+        const prevPickingHours = prevPicking.pickingTimeSeconds / 3600;
+
+        // Previous shipping data
+        const prevShippingWhere = previousRange.whereClause.replace(/date_placed/g, 'o.date_shipped');
+        const [prevShippingRows] = await connection.execute(
+          `SELECT
+             COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+             COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+           FROM _order o
+           WHERE ${prevShippingWhere}
+             AND o.order_status IN (100, 92)`,
+          previousRange.params
+        );
+        const prevShipping = prevShippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
+
+        // Calculate previous period FTE and productivity
+        const prevFte = calculateFTE(prevTotalHours, previousRange.start || periodStart, previousRange.end || periodEnd);
+        const prevOrdersPerHour = prevProductiveHours > 0 ? parseInt(prevPicking.ordersPicked || 0) / prevProductiveHours : 0;
+        const prevPiecesPerHour = prevProductiveHours > 0 ? parseInt(prevPicking.piecesPicked || 0) / prevProductiveHours : 0;
+
+        previousTotals = {
+          hours: prevTotalHours,
+          productiveHours: prevProductiveHours,
+          activeEmployees: prevActiveEmployees,
+          fte: prevFte,
+          ordersPicked: parseInt(prevPicking.ordersPicked || 0),
+          piecesPicked: parseInt(prevPicking.piecesPicked || 0),
+          pickingHours: prevPickingHours,
+          ordersShipped: parseInt(prevShipping.ordersShipped || 0),
+          piecesShipped: parseInt(prevShipping.piecesShipped || 0),
+          ordersPerHour: prevOrdersPerHour,
+          piecesPerHour: prevPiecesPerHour,
+        };
+
+        // Calculate productivity metrics for comparison
+        const currentOrdersPerHour = totalProductiveHours > 0 ? totalOrdersPicked / totalProductiveHours : 0;
+        const currentPiecesPerHour = totalProductiveHours > 0 ? totalPiecesPicked / totalProductiveHours : 0;
+
+        comparison = {
+          hours: calculateComparison(totalHours, prevTotalHours),
+          productiveHours: calculateComparison(totalProductiveHours, prevProductiveHours),
+          activeEmployees: calculateComparison(activeEmployees, prevActiveEmployees),
+          fte: calculateComparison(fte, prevFte),
+          ordersPicked: calculateComparison(totalOrdersPicked, parseInt(prevPicking.ordersPicked || 0)),
+          piecesPicked: calculateComparison(totalPiecesPicked, parseInt(prevPicking.piecesPicked || 0)),
+          ordersShipped: calculateComparison(parseInt(shipping.ordersShipped || 0), parseInt(prevShipping.ordersShipped || 0)),
+          piecesShipped: calculateComparison(parseInt(shipping.piecesShipped || 0), parseInt(prevShipping.piecesShipped || 0)),
+          ordersPerHour: calculateComparison(currentOrdersPerHour, prevOrdersPerHour),
+          piecesPerHour: calculateComparison(currentPiecesPerHour, prevPiecesPerHour),
+        };
+      }
+
+      // Calculate efficiency (picking time vs productive hours)
+      const pickingEfficiency = totalProductiveHours > 0 ? (totalPickingHours / totalProductiveHours) * 100 : 0;
+
+      const response = {
+        dateRange,
+        totals: {
+          // Time metrics
+          hours: totalHours,
+          breakHours: totalBreakHours,
+          productiveHours: totalProductiveHours,
+          pickingHours: totalPickingHours,
+
+          // Employee metrics
+          activeEmployees,
+          fte,
+          weeksInPeriod,
+
+          // Picking metrics
+          ordersPicked: totalOrdersPicked,
+          piecesPicked: totalPiecesPicked,
+          ticketCount: totalTickets,
+
+          // Shipping metrics
+          ordersShipped: parseInt(shipping.ordersShipped || 0),
+          piecesShipped: parseInt(shipping.piecesShipped || 0),
+
+          // Calculated metrics - standardized to weekly
+          hoursPerWeek: weeksInPeriod > 0 ? totalHours / weeksInPeriod : 0,
+          hoursPerEmployeePerWeek: activeEmployees > 0 && weeksInPeriod > 0
+            ? (totalHours / activeEmployees) / weeksInPeriod
+            : 0,
+
+          // Productivity metrics (uses productive hours - excludes breaks)
+          ordersPerHour: totalProductiveHours > 0 ? totalOrdersPicked / totalProductiveHours : 0,
+          piecesPerHour: totalProductiveHours > 0 ? totalPiecesPicked / totalProductiveHours : 0,
+
+          // Picking speed from database (more accurate, only counts picking time)
+          avgPickingSpeed,
+
+          // Efficiency metrics
+          pickingEfficiency,
+        },
+        previousTotals,
+        comparison,
+        byEmployee: {
+          hours: enrichedEmployeeHours,
+          picking: pickingByEmployee,
+          shipping: shippingByEmployee,
+        },
+        trend,
+      };
+
+      return { response, release };
+    };
+
+    let result;
+    try {
+      result = await Promise.race([mainOperation(), timeoutPromise]);
+    } catch (error) {
+      if (error.message.includes('timeout')) {
+        console.log(`[EMPLOYEE-METRICS] Request timed out in ${Date.now() - startTime}ms`);
+        throw error;
+      }
+      throw error;
+    }
+
+    const { response, release } = result;
+
+    if (release) release();
+
+    console.log(`[EMPLOYEE-METRICS] Request completed in ${Date.now() - startTime}ms`);
+    res.json(response);
+
+  } catch (error) {
+    console.error('Error in /employee-metrics:', error);
+    console.log(`[EMPLOYEE-METRICS] Request failed in ${Date.now() - startTime}ms`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Health check
+router.get('/health', async (req, res) => {
+  try {
+    const { connection, release } = await getDbConnection();
+    await connection.execute('SELECT 1 as test');
+    release();
+
+    res.json({
+      status: 'healthy',
+      timestamp: new Date().toISOString(),
+      pool: getPoolStatus(),
+    });
+  } catch (error) {
+    res.status(500).json({
+      status: 'unhealthy',
+      timestamp: new Date().toISOString(),
+      error: error.message,
+    });
+  }
+});
+
+// Helper functions
+function calculateComparison(currentValue, previousValue) {
+  if (typeof previousValue !== 'number') {
+    return { absolute: null, percentage: null };
+  }
+
+  const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
+  const percentage =
+    absolute !== null && previousValue !== 0
+      ? (absolute / Math.abs(previousValue)) * 100
+      : null;
+
+  return { absolute, percentage };
+}
+
+function getPreviousPeriodRange(timeRange, startDate, endDate) {
+  if (timeRange && timeRange !== 'custom') {
+    const prevTimeRange = getPreviousTimeRange(timeRange);
+    if (!prevTimeRange || prevTimeRange === timeRange) {
+      return null;
+    }
+    return getTimeRangeConditions(prevTimeRange);
+  }
+
+  const hasCustomDates = (timeRange === 'custom' || !timeRange) && startDate && endDate;
+  if (!hasCustomDates) {
+    return null;
+  }
+
+  const start = new Date(startDate);
+  const end = new Date(endDate);
+
+  if (Number.isNaN(start.getTime()) || Number.isNaN(end.getTime())) {
+    return null;
+  }
+
+  const duration = end.getTime() - start.getTime();
+  if (!Number.isFinite(duration) || duration <= 0) {
+    return null;
+  }
+
+  const prevEnd = new Date(start.getTime() - 1);
+  const prevStart = new Date(prevEnd.getTime() - duration);
+
+  return getTimeRangeConditions('custom', prevStart.toISOString(), prevEnd.toISOString());
+}
+
+function getPreviousTimeRange(timeRange) {
+  const map = {
+    today: 'yesterday',
+    thisWeek: 'lastWeek',
+    thisMonth: 'lastMonth',
+    last7days: 'previous7days',
+    last30days: 'previous30days',
+    last90days: 'previous90days',
+    yesterday: 'twoDaysAgo'
+  };
+  return map[timeRange] || timeRange;
+}
+
+export default router;

inventory-server/dashboard/acot-server/routes/operations-metrics.js

@@ -0,0 +1,480 @@
+import express from 'express';
+import { DateTime } from 'luxon';
+import { getDbConnection, getPoolStatus } from '../db/connection.js';
+import { getTimeRangeConditions } from '../utils/timeUtils.js';
+
+const router = express.Router();
+
+const TIMEZONE = 'America/New_York';
+
+// Main operations metrics endpoint - focused on picking and shipping
+router.get('/', async (req, res) => {
+  const startTime = Date.now();
+  console.log(`[OPERATIONS-METRICS] Starting request for timeRange: ${req.query.timeRange}`);
+
+  const timeoutPromise = new Promise((_, reject) => {
+    setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
+  });
+
+  try {
+    const mainOperation = async () => {
+      const { timeRange, startDate, endDate } = req.query;
+      console.log(`[OPERATIONS-METRICS] Getting DB connection...`);
+      const { connection, release } = await getDbConnection();
+      console.log(`[OPERATIONS-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
+      try {
+
+      const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
+
+      // Query for picking tickets - using subquery to avoid duplication from bucket join
+      // Ship-together orders: only count main orders (is_sub = 0 or NULL), not sub-orders
+      const pickingWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
+
+      // First get picking ticket stats without the bucket join (to avoid duplication)
+      const pickingStatsQuery = `
+        SELECT
+          pt.createdby as employeeId,
+          e.firstname,
+          e.lastname,
+          COUNT(DISTINCT pt.pickingid) as ticketCount,
+          SUM(pt.totalpieces_picked) as piecesPicked,
+          SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds,
+          AVG(NULLIF(pt.picking_speed, 0)) as avgPickingSpeed
+        FROM picking_ticket pt
+        LEFT JOIN employees e ON pt.createdby = e.employeeid
+        WHERE ${pickingWhere}
+          AND pt.closeddate IS NOT NULL
+        GROUP BY pt.createdby, e.firstname, e.lastname
+      `;
+
+      // Separate query for order counts (needs bucket join for ship-together handling)
+      const orderCountQuery = `
+        SELECT
+          pt.createdby as employeeId,
+          COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
+        FROM picking_ticket pt
+        LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+        WHERE ${pickingWhere}
+          AND pt.closeddate IS NOT NULL
+        GROUP BY pt.createdby
+      `;
+
+      const [[pickingStatsRows], [orderCountRows]] = await Promise.all([
+        connection.execute(pickingStatsQuery, params),
+        connection.execute(orderCountQuery, params)
+      ]);
+
+      // Merge the results
+      const orderCountMap = new Map();
+      orderCountRows.forEach(row => {
+        orderCountMap.set(row.employeeId, parseInt(row.ordersPicked || 0));
+      });
+
+      // Aggregate picking totals
+      let totalOrdersPicked = 0;
+      let totalPiecesPicked = 0;
+      let totalTickets = 0;
+      let totalPickingTimeSeconds = 0;
+      let pickingSpeedSum = 0;
+      let pickingSpeedCount = 0;
+
+      const pickingByEmployee = pickingStatsRows.map(row => {
+        const ordersPicked = orderCountMap.get(row.employeeId) || 0;
+        totalOrdersPicked += ordersPicked;
+        totalPiecesPicked += parseInt(row.piecesPicked || 0);
+        totalTickets += parseInt(row.ticketCount || 0);
+        totalPickingTimeSeconds += parseInt(row.pickingTimeSeconds || 0);
+        if (row.avgPickingSpeed && row.avgPickingSpeed > 0) {
+          pickingSpeedSum += parseFloat(row.avgPickingSpeed);
+          pickingSpeedCount++;
+        }
+
+        const empPickingHours = parseInt(row.pickingTimeSeconds || 0) / 3600;
+
+        return {
+          employeeId: row.employeeId,
+          name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeId}`,
+          ticketCount: parseInt(row.ticketCount || 0),
+          ordersPicked,
+          piecesPicked: parseInt(row.piecesPicked || 0),
+          pickingHours: empPickingHours,
+          avgPickingSpeed: row.avgPickingSpeed ? parseFloat(row.avgPickingSpeed) : null,
+        };
+      });
+
+      const totalPickingHours = totalPickingTimeSeconds / 3600;
+      const avgPickingSpeed = pickingSpeedCount > 0 ? pickingSpeedSum / pickingSpeedCount : 0;
+
+      // Query for shipped orders - totals
+      // Ship-together orders: only count main orders (order_type != 8 for sub-orders)
+      const shippingWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
+
+      const shippingQuery = `
+        SELECT
+          COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+          COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+        FROM _order o
+        WHERE ${shippingWhere}
+          AND o.order_status IN (100, 92)
+      `;
+
+      const [shippingRows] = await connection.execute(shippingQuery, params);
+      const shipping = shippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
+
+      // Query for shipped orders by employee
+      const shippingByEmployeeQuery = `
+        SELECT
+          e.employeeid,
+          e.firstname,
+          e.lastname,
+          COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+          COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+        FROM _order o
+        JOIN employees e ON o.stats_cid_shipped = e.cid
+        WHERE ${shippingWhere}
+          AND o.order_status IN (100, 92)
+          AND e.hidden = 0
+          AND e.disabled = 0
+        GROUP BY e.employeeid, e.firstname, e.lastname
+        ORDER BY ordersShipped DESC
+      `;
+
+      const [shippingByEmployeeRows] = await connection.execute(shippingByEmployeeQuery, params);
+      const shippingByEmployee = shippingByEmployeeRows.map(row => ({
+        employeeId: row.employeeid,
+        name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeid}`,
+        ordersShipped: parseInt(row.ordersShipped || 0),
+        piecesShipped: parseInt(row.piecesShipped || 0),
+      }));
+
+      // Calculate period dates
+      let periodStart, periodEnd;
+      if (dateRange?.start) {
+        periodStart = new Date(dateRange.start);
+      } else if (params[0]) {
+        periodStart = new Date(params[0]);
+      } else {
+        periodStart = new Date();
+        periodStart.setDate(periodStart.getDate() - 30);
+      }
+
+      if (dateRange?.end) {
+        periodEnd = new Date(dateRange.end);
+      } else if (params[1]) {
+        periodEnd = new Date(params[1]);
+      } else {
+        periodEnd = new Date();
+      }
+
+      // Calculate productivity (orders/pieces per picking hour)
+      const ordersPerHour = totalPickingHours > 0 ? totalOrdersPicked / totalPickingHours : 0;
+      const piecesPerHour = totalPickingHours > 0 ? totalPiecesPicked / totalPickingHours : 0;
+
+      // Get daily trend data for picking
+      // Use DATE_FORMAT to get date string in Eastern timezone
+      // Business day starts at 1 AM, so subtract 1 hour before taking the date
+      const pickingTrendWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
+      const pickingTrendQuery = `
+        SELECT
+          pt_agg.date,
+          COALESCE(order_counts.ordersPicked, 0) as ordersPicked,
+          pt_agg.piecesPicked
+        FROM (
+          SELECT
+            DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d') as date,
+            COALESCE(SUM(pt.totalpieces_picked), 0) as piecesPicked
+          FROM picking_ticket pt
+          WHERE ${pickingTrendWhere}
+            AND pt.closeddate IS NOT NULL
+          GROUP BY DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d')
+        ) pt_agg
+        LEFT JOIN (
+          SELECT
+            DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d') as date,
+            COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
+          FROM picking_ticket pt
+          LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+          WHERE ${pickingTrendWhere}
+            AND pt.closeddate IS NOT NULL
+          GROUP BY DATE_FORMAT(DATE_SUB(pt.createddate, INTERVAL 1 HOUR), '%Y-%m-%d')
+        ) order_counts ON pt_agg.date = order_counts.date
+        ORDER BY pt_agg.date
+      `;
+
+      // Get shipping trend data
+      const shippingTrendWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
+      const shippingTrendQuery = `
+        SELECT
+          DATE_FORMAT(DATE_SUB(o.date_shipped, INTERVAL 1 HOUR), '%Y-%m-%d') as date,
+          COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+          COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+        FROM _order o
+        WHERE ${shippingTrendWhere}
+          AND o.order_status IN (100, 92)
+        GROUP BY DATE_FORMAT(DATE_SUB(o.date_shipped, INTERVAL 1 HOUR), '%Y-%m-%d')
+        ORDER BY date
+      `;
+
+      const [[pickingTrendRows], [shippingTrendRows]] = await Promise.all([
+        connection.execute(pickingTrendQuery, [...params, ...params]),
+        connection.execute(shippingTrendQuery, params),
+      ]);
+
+      // Create maps for trend data
+      const pickingByDate = new Map();
+      pickingTrendRows.forEach(row => {
+        const date = String(row.date);
+        pickingByDate.set(date, {
+          ordersPicked: parseInt(row.ordersPicked || 0),
+          piecesPicked: parseInt(row.piecesPicked || 0),
+        });
+      });
+
+      const shippingByDate = new Map();
+      shippingTrendRows.forEach(row => {
+        const date = String(row.date);
+        shippingByDate.set(date, {
+          ordersShipped: parseInt(row.ordersShipped || 0),
+          piecesShipped: parseInt(row.piecesShipped || 0),
+        });
+      });
+
+      // Generate all dates in the period range for complete trend data
+      const allDatesInRange = [];
+      const startDt = DateTime.fromJSDate(periodStart).setZone(TIMEZONE).startOf('day');
+      const endDt = DateTime.fromJSDate(periodEnd).setZone(TIMEZONE).startOf('day');
+
+      let currentDt = startDt;
+      while (currentDt <= endDt) {
+        allDatesInRange.push(currentDt.toFormat('yyyy-MM-dd'));
+        currentDt = currentDt.plus({ days: 1 });
+      }
+
+      // Build trend data for all dates in range
+      const trend = allDatesInRange.map(date => {
+        const picking = pickingByDate.get(date) || { ordersPicked: 0, piecesPicked: 0 };
+        const shippingData = shippingByDate.get(date) || { ordersShipped: 0, piecesShipped: 0 };
+
+        // Parse date string in Eastern timezone to get proper ISO timestamp
+        const dateDt = DateTime.fromFormat(date, 'yyyy-MM-dd', { zone: TIMEZONE });
+
+        return {
+          date,
+          timestamp: dateDt.toISO(),
+          ordersPicked: picking.ordersPicked,
+          piecesPicked: picking.piecesPicked,
+          ordersShipped: shippingData.ordersShipped,
+          piecesShipped: shippingData.piecesShipped,
+        };
+      });
+
+      // Get previous period data for comparison
+      const previousRange = getPreviousPeriodRange(timeRange, startDate, endDate);
+      let comparison = null;
+      let previousTotals = null;
+
+      if (previousRange) {
+        // Previous picking data
+        const prevPickingWhere = previousRange.whereClause.replace(/date_placed/g, 'pt.createddate');
+
+        const [[prevPickingStatsRows], [prevOrderCountRows]] = await Promise.all([
+          connection.execute(
+            `SELECT
+               SUM(pt.totalpieces_picked) as piecesPicked,
+               SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds
+             FROM picking_ticket pt
+             WHERE ${prevPickingWhere}
+               AND pt.closeddate IS NOT NULL`,
+            previousRange.params
+          ),
+          connection.execute(
+            `SELECT
+               COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
+             FROM picking_ticket pt
+             LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
+             WHERE ${prevPickingWhere}
+               AND pt.closeddate IS NOT NULL`,
+            previousRange.params
+          )
+        ]);
+
+        const prevPickingStats = prevPickingStatsRows[0] || { piecesPicked: 0, pickingTimeSeconds: 0 };
+        const prevOrderCount = prevOrderCountRows[0] || { ordersPicked: 0 };
+        const prevPicking = {
+          ordersPicked: parseInt(prevOrderCount.ordersPicked || 0),
+          piecesPicked: parseInt(prevPickingStats.piecesPicked || 0),
+          pickingTimeSeconds: parseInt(prevPickingStats.pickingTimeSeconds || 0)
+        };
+        const prevPickingHours = prevPicking.pickingTimeSeconds / 3600;
+
+        // Previous shipping data
+        const prevShippingWhere = previousRange.whereClause.replace(/date_placed/g, 'o.date_shipped');
+        const [prevShippingRows] = await connection.execute(
+          `SELECT
+             COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
+             COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
+           FROM _order o
+           WHERE ${prevShippingWhere}
+             AND o.order_status IN (100, 92)`,
+          previousRange.params
+        );
+        const prevShipping = prevShippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
+
+        // Calculate previous productivity
+        const prevOrdersPerHour = prevPickingHours > 0 ? parseInt(prevPicking.ordersPicked || 0) / prevPickingHours : 0;
+        const prevPiecesPerHour = prevPickingHours > 0 ? parseInt(prevPicking.piecesPicked || 0) / prevPickingHours : 0;
+
+        previousTotals = {
+          ordersPicked: parseInt(prevPicking.ordersPicked || 0),
+          piecesPicked: parseInt(prevPicking.piecesPicked || 0),
+          pickingHours: prevPickingHours,
+          ordersShipped: parseInt(prevShipping.ordersShipped || 0),
+          piecesShipped: parseInt(prevShipping.piecesShipped || 0),
+          ordersPerHour: prevOrdersPerHour,
+          piecesPerHour: prevPiecesPerHour,
+        };
+
+        comparison = {
+          ordersPicked: calculateComparison(totalOrdersPicked, parseInt(prevPicking.ordersPicked || 0)),
+          piecesPicked: calculateComparison(totalPiecesPicked, parseInt(prevPicking.piecesPicked || 0)),
+          ordersShipped: calculateComparison(parseInt(shipping.ordersShipped || 0), parseInt(prevShipping.ordersShipped || 0)),
+          piecesShipped: calculateComparison(parseInt(shipping.piecesShipped || 0), parseInt(prevShipping.piecesShipped || 0)),
+          ordersPerHour: calculateComparison(ordersPerHour, prevOrdersPerHour),
+          piecesPerHour: calculateComparison(piecesPerHour, prevPiecesPerHour),
+        };
+      }
+
+      const response = {
+        dateRange,
+        totals: {
+          // Picking metrics
+          ordersPicked: totalOrdersPicked,
+          piecesPicked: totalPiecesPicked,
+          ticketCount: totalTickets,
+          pickingHours: totalPickingHours,
+
+          // Shipping metrics
+          ordersShipped: parseInt(shipping.ordersShipped || 0),
+          piecesShipped: parseInt(shipping.piecesShipped || 0),
+
+          // Productivity metrics
+          ordersPerHour,
+          piecesPerHour,
+          avgPickingSpeed,
+        },
+        previousTotals,
+        comparison,
+        byEmployee: {
+          picking: pickingByEmployee,
+          shipping: shippingByEmployee,
+        },
+        trend,
+      };
+
+      return response;
+      } finally {
+        // Always release the connection regardless of who wins Promise.race.
+        // If the timeout wins, this IIFE keeps running until MySQL responds; this
+        // finally ensures the connection still returns to the pool.
+        release();
+      }
+    };
+
+    const response = await Promise.race([mainOperation(), timeoutPromise]);
+
+    console.log(`[OPERATIONS-METRICS] Request completed in ${Date.now() - startTime}ms`);
+    res.json(response);
+
+  } catch (error) {
+    if (error.message.includes('timeout')) {
+      console.log(`[OPERATIONS-METRICS] Request timed out in ${Date.now() - startTime}ms`);
+    } else {
+      console.error('Error in /operations-metrics:', error);
+    }
+    console.log(`[OPERATIONS-METRICS] Request failed in ${Date.now() - startTime}ms`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Health check
+router.get('/health', async (req, res) => {
+  try {
+    const { connection, release } = await getDbConnection();
+    await connection.execute('SELECT 1 as test');
+    release();
+
+    res.json({
+      status: 'healthy',
+      timestamp: new Date().toISOString(),
+      pool: getPoolStatus(),
+    });
+  } catch (error) {
+    res.status(500).json({
+      status: 'unhealthy',
+      timestamp: new Date().toISOString(),
+      error: error.message,
+    });
+  }
+});
+
+// Helper functions
+function calculateComparison(currentValue, previousValue) {
+  if (typeof previousValue !== 'number') {
+    return { absolute: null, percentage: null };
+  }
+
+  const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
+  const percentage =
+    absolute !== null && previousValue !== 0
+      ? (absolute / Math.abs(previousValue)) * 100
+      : null;
+
+  return { absolute, percentage };
+}
+
+function getPreviousPeriodRange(timeRange, startDate, endDate) {
+  if (timeRange && timeRange !== 'custom') {
+    const prevTimeRange = getPreviousTimeRange(timeRange);
+    if (!prevTimeRange || prevTimeRange === timeRange) {
+      return null;
+    }
+    return getTimeRangeConditions(prevTimeRange);
+  }
+
+  const hasCustomDates = (timeRange === 'custom' || !timeRange) && startDate && endDate;
+  if (!hasCustomDates) {
+    return null;
+  }
+
+  const start = new Date(startDate);
+  const end = new Date(endDate);
+
+  if (Number.isNaN(start.getTime()) || Number.isNaN(end.getTime())) {
+    return null;
+  }
+
+  const duration = end.getTime() - start.getTime();
+  if (!Number.isFinite(duration) || duration <= 0) {
+    return null;
+  }
+
+  const prevEnd = new Date(start.getTime() - 1);
+  const prevStart = new Date(prevEnd.getTime() - duration);
+
+  return getTimeRangeConditions('custom', prevStart.toISOString(), prevEnd.toISOString());
+}
+
+function getPreviousTimeRange(timeRange) {
+  const map = {
+    today: 'yesterday',
+    thisWeek: 'lastWeek',
+    thisMonth: 'lastMonth',
+    last7days: 'previous7days',
+    last30days: 'previous30days',
+    last90days: 'previous90days',
+    yesterday: 'twoDaysAgo'
+  };
+  return map[timeRange] || timeRange;
+}
+
+export default router;

inventory-server/dashboard/acot-server/routes/payroll-metrics.js

@@ -0,0 +1,503 @@
+import express from 'express';
+import { DateTime } from 'luxon';
+import { getDbConnection, getPoolStatus } from '../db/connection.js';
+
+const router = express.Router();
+
+const TIMEZONE = 'America/New_York';
+
+// Punch types from the database
+const PUNCH_TYPES = {
+  OUT: 0,
+  IN: 1,
+  BREAK_START: 2,
+  BREAK_END: 3,
+};
+
+// Standard hours for overtime calculation (40 hours per week)
+const STANDARD_WEEKLY_HOURS = 40;
+
+// Reference pay period start date (January 25, 2026 is a Sunday, first day of a pay period)
+const PAY_PERIOD_REFERENCE = DateTime.fromObject(
+  { year: 2026, month: 1, day: 25 },
+  { zone: TIMEZONE }
+);
+
+/**
+ * Calculate the pay period that contains a given date
+ * Pay periods are 14 days starting on Sunday
+ * @param {DateTime} date - The date to find the pay period for
+ * @returns {{ start: DateTime, end: DateTime, week1: { start: DateTime, end: DateTime }, week2: { start: DateTime, end: DateTime } }}
+ */
+function getPayPeriodForDate(date) {
+  const dt = DateTime.isDateTime(date) ? date : DateTime.fromJSDate(date, { zone: TIMEZONE });
+
+  // Calculate days since reference
+  const daysSinceReference = Math.floor(dt.diff(PAY_PERIOD_REFERENCE, 'days').days);
+
+  // Find which pay period this falls into (can be negative for dates before reference)
+  const payPeriodIndex = Math.floor(daysSinceReference / 14);
+
+  // Calculate the start of this pay period
+  const start = PAY_PERIOD_REFERENCE.plus({ days: payPeriodIndex * 14 }).startOf('day');
+  const end = start.plus({ days: 13 }).endOf('day');
+
+  // Week 1: Sunday through Saturday
+  const week1Start = start;
+  const week1End = start.plus({ days: 6 }).endOf('day');
+
+  // Week 2: Sunday through Saturday
+  const week2Start = start.plus({ days: 7 }).startOf('day');
+  const week2End = end;
+
+  return {
+    start,
+    end,
+    week1: { start: week1Start, end: week1End },
+    week2: { start: week2Start, end: week2End },
+  };
+}
+
+/**
+ * Get the current pay period
+ */
+function getCurrentPayPeriod() {
+  return getPayPeriodForDate(DateTime.now().setZone(TIMEZONE));
+}
+
+/**
+ * Navigate to previous or next pay period
+ * @param {DateTime} currentStart - Current pay period start
+ * @param {number} offset - Number of pay periods to move (negative for previous)
+ */
+function navigatePayPeriod(currentStart, offset) {
+  const newStart = currentStart.plus({ days: offset * 14 });
+  return getPayPeriodForDate(newStart);
+}
+
+/**
+ * Calculate working hours from timeclock entries, broken down by week
+ * @param {Array} punches - Timeclock punch entries
+ * @param {Object} payPeriod - Pay period with week boundaries
+ */
+function calculateHoursByWeek(punches, payPeriod) {
+  // Group by employee
+  const byEmployee = new Map();
+
+  punches.forEach(punch => {
+    if (!byEmployee.has(punch.EmployeeID)) {
+      byEmployee.set(punch.EmployeeID, {
+        employeeId: punch.EmployeeID,
+        firstname: punch.firstname || '',
+        lastname: punch.lastname || '',
+        punches: [],
+      });
+    }
+    byEmployee.get(punch.EmployeeID).punches.push(punch);
+  });
+
+  const employeeResults = [];
+  let totalHours = 0;
+  let totalBreakHours = 0;
+  let totalOvertimeHours = 0;
+  let totalRegularHours = 0;
+  let week1TotalHours = 0;
+  let week1TotalOvertime = 0;
+  let week2TotalHours = 0;
+  let week2TotalOvertime = 0;
+
+  byEmployee.forEach((employeeData) => {
+    // Sort punches by timestamp
+    employeeData.punches.sort((a, b) => new Date(a.TimeStamp) - new Date(b.TimeStamp));
+
+    // Calculate hours for each week
+    const week1Punches = employeeData.punches.filter(p => {
+      const dt = DateTime.fromJSDate(new Date(p.TimeStamp), { zone: TIMEZONE });
+      return dt >= payPeriod.week1.start && dt <= payPeriod.week1.end;
+    });
+
+    const week2Punches = employeeData.punches.filter(p => {
+      const dt = DateTime.fromJSDate(new Date(p.TimeStamp), { zone: TIMEZONE });
+      return dt >= payPeriod.week2.start && dt <= payPeriod.week2.end;
+    });
+
+    const week1Hours = calculateHoursFromPunches(week1Punches);
+    const week2Hours = calculateHoursFromPunches(week2Punches);
+
+    // Calculate overtime per week (anything over 40 hours)
+    const week1Overtime = Math.max(0, week1Hours.hours - STANDARD_WEEKLY_HOURS);
+    const week2Overtime = Math.max(0, week2Hours.hours - STANDARD_WEEKLY_HOURS);
+    const week1Regular = week1Hours.hours - week1Overtime;
+    const week2Regular = week2Hours.hours - week2Overtime;
+
+    const employeeTotal = week1Hours.hours + week2Hours.hours;
+    const employeeBreaks = week1Hours.breakHours + week2Hours.breakHours;
+    const employeeOvertime = week1Overtime + week2Overtime;
+    const employeeRegular = employeeTotal - employeeOvertime;
+
+    totalHours += employeeTotal;
+    totalBreakHours += employeeBreaks;
+    totalOvertimeHours += employeeOvertime;
+    totalRegularHours += employeeRegular;
+    week1TotalHours += week1Hours.hours;
+    week1TotalOvertime += week1Overtime;
+    week2TotalHours += week2Hours.hours;
+    week2TotalOvertime += week2Overtime;
+
+    employeeResults.push({
+      employeeId: employeeData.employeeId,
+      name: `${employeeData.firstname} ${employeeData.lastname}`.trim() || `Employee ${employeeData.employeeId}`,
+      week1Hours: week1Hours.hours,
+      week1BreakHours: week1Hours.breakHours,
+      week1Overtime,
+      week1Regular,
+      week2Hours: week2Hours.hours,
+      week2BreakHours: week2Hours.breakHours,
+      week2Overtime,
+      week2Regular,
+      totalHours: employeeTotal,
+      totalBreakHours: employeeBreaks,
+      overtimeHours: employeeOvertime,
+      regularHours: employeeRegular,
+    });
+  });
+
+  // Sort by total hours descending
+  employeeResults.sort((a, b) => b.totalHours - a.totalHours);
+
+  return {
+    byEmployee: employeeResults,
+    totals: {
+      hours: totalHours,
+      breakHours: totalBreakHours,
+      overtimeHours: totalOvertimeHours,
+      regularHours: totalRegularHours,
+      activeEmployees: employeeResults.filter(e => e.totalHours > 0).length,
+    },
+    byWeek: [
+      {
+        week: 1,
+        start: payPeriod.week1.start.toISODate(),
+        end: payPeriod.week1.end.toISODate(),
+        hours: week1TotalHours,
+        overtime: week1TotalOvertime,
+        regular: week1TotalHours - week1TotalOvertime,
+      },
+      {
+        week: 2,
+        start: payPeriod.week2.start.toISODate(),
+        end: payPeriod.week2.end.toISODate(),
+        hours: week2TotalHours,
+        overtime: week2TotalOvertime,
+        regular: week2TotalHours - week2TotalOvertime,
+      },
+    ],
+  };
+}
+
+/**
+ * Calculate hours from a set of punches
+ */
+function calculateHoursFromPunches(punches) {
+  let hours = 0;
+  let breakHours = 0;
+  let currentIn = null;
+  let breakStart = null;
+
+  punches.forEach(punch => {
+    const punchTime = new Date(punch.TimeStamp);
+
+    switch (punch.PunchType) {
+      case PUNCH_TYPES.IN:
+        currentIn = punchTime;
+        break;
+      case PUNCH_TYPES.OUT:
+        if (currentIn) {
+          hours += (punchTime - currentIn) / (1000 * 60 * 60);
+          currentIn = null;
+        }
+        break;
+      case PUNCH_TYPES.BREAK_START:
+        breakStart = punchTime;
+        break;
+      case PUNCH_TYPES.BREAK_END:
+        if (breakStart) {
+          breakHours += (punchTime - breakStart) / (1000 * 60 * 60);
+          breakStart = null;
+        }
+        break;
+    }
+  });
+
+  return { hours, breakHours };
+}
+
+/**
+ * Calculate FTE for a pay period (based on 80 hours = 1 FTE for 2-week period)
+ * @param {number} totalHours - Total hours worked
+ * @param {number} elapsedFraction - Fraction of the period elapsed (0-1). Defaults to 1 for complete periods.
+ */
+function calculateFTE(totalHours, elapsedFraction = 1) {
+  const fullTimePeriodHours = STANDARD_WEEKLY_HOURS * 2; // 80 hours for 2 weeks
+  const proratedHours = fullTimePeriodHours * elapsedFraction;
+  return proratedHours > 0 ? totalHours / proratedHours : 0;
+}
+
+// Main payroll metrics endpoint
+router.get('/', async (req, res) => {
+  const startTime = Date.now();
+  console.log(`[PAYROLL-METRICS] Starting request`);
+
+  const timeoutPromise = new Promise((_, reject) => {
+    setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
+  });
+
+  try {
+    const mainOperation = async () => {
+      const { payPeriodStart, navigate } = req.query;
+
+      let payPeriod;
+
+      if (payPeriodStart) {
+        // Parse the provided start date
+        const startDate = DateTime.fromISO(payPeriodStart, { zone: TIMEZONE });
+        if (!startDate.isValid) {
+          return res.status(400).json({ error: 'Invalid payPeriodStart date format' });
+        }
+        payPeriod = getPayPeriodForDate(startDate);
+      } else {
+        // Default to current pay period
+        payPeriod = getCurrentPayPeriod();
+      }
+
+      // Handle navigation if requested
+      if (navigate) {
+        const offset = parseInt(navigate, 10);
+        if (!isNaN(offset)) {
+          payPeriod = navigatePayPeriod(payPeriod.start, offset);
+        }
+      }
+
+      console.log(`[PAYROLL-METRICS] Getting DB connection...`);
+      const { connection, release } = await getDbConnection();
+      console.log(`[PAYROLL-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
+      try {
+
+      // Build query for the pay period
+      const periodStart = payPeriod.start.toJSDate();
+      const periodEnd = payPeriod.end.toJSDate();
+
+      const timeclockQuery = `
+        SELECT
+          tc.EmployeeID,
+          tc.TimeStamp,
+          tc.PunchType,
+          e.firstname,
+          e.lastname
+        FROM timeclock tc
+        LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
+        WHERE tc.TimeStamp >= ? AND tc.TimeStamp <= ?
+          AND e.hidden = 0
+          AND e.disabled = 0
+        ORDER BY tc.EmployeeID, tc.TimeStamp
+      `;
+
+      const [timeclockRows] = await connection.execute(timeclockQuery, [periodStart, periodEnd]);
+
+      // Calculate hours with week breakdown
+      const hoursData = calculateHoursByWeek(timeclockRows, payPeriod);
+
+      // Calculate FTE — prorate for in-progress periods so the value reflects
+      // the pace employees are on rather than raw hours / 80
+      let elapsedFraction = 1;
+      if (isCurrentPayPeriod(payPeriod)) {
+        const now = DateTime.now().setZone(TIMEZONE);
+        const elapsedDays = Math.max(1, Math.ceil(now.diff(payPeriod.start, 'days').days));
+        elapsedFraction = Math.min(1, elapsedDays / 14);
+      }
+      const fte = calculateFTE(hoursData.totals.hours, elapsedFraction);
+      const activeEmployees = hoursData.totals.activeEmployees;
+      const avgHoursPerEmployee = activeEmployees > 0 ? hoursData.totals.hours / activeEmployees : 0;
+
+      // Get previous pay period data for comparison
+      const prevPayPeriod = navigatePayPeriod(payPeriod.start, -1);
+      const [prevTimeclockRows] = await connection.execute(timeclockQuery, [
+        prevPayPeriod.start.toJSDate(),
+        prevPayPeriod.end.toJSDate(),
+      ]);
+
+      const prevHoursData = calculateHoursByWeek(prevTimeclockRows, prevPayPeriod);
+      const prevFte = calculateFTE(prevHoursData.totals.hours);
+
+      // Calculate comparisons
+      const comparison = {
+        hours: calculateComparison(hoursData.totals.hours, prevHoursData.totals.hours),
+        overtimeHours: calculateComparison(hoursData.totals.overtimeHours, prevHoursData.totals.overtimeHours),
+        fte: calculateComparison(fte, prevFte),
+        activeEmployees: calculateComparison(hoursData.totals.activeEmployees, prevHoursData.totals.activeEmployees),
+      };
+
+      const response = {
+        payPeriod: {
+          start: payPeriod.start.toISODate(),
+          end: payPeriod.end.toISODate(),
+          label: formatPayPeriodLabel(payPeriod),
+          week1: {
+            start: payPeriod.week1.start.toISODate(),
+            end: payPeriod.week1.end.toISODate(),
+            label: formatWeekLabel(payPeriod.week1),
+          },
+          week2: {
+            start: payPeriod.week2.start.toISODate(),
+            end: payPeriod.week2.end.toISODate(),
+            label: formatWeekLabel(payPeriod.week2),
+          },
+          isCurrent: isCurrentPayPeriod(payPeriod),
+        },
+        totals: {
+          hours: hoursData.totals.hours,
+          breakHours: hoursData.totals.breakHours,
+          overtimeHours: hoursData.totals.overtimeHours,
+          regularHours: hoursData.totals.regularHours,
+          activeEmployees,
+          fte,
+          avgHoursPerEmployee,
+        },
+        previousTotals: {
+          hours: prevHoursData.totals.hours,
+          overtimeHours: prevHoursData.totals.overtimeHours,
+          activeEmployees: prevHoursData.totals.activeEmployees,
+          fte: prevFte,
+        },
+        comparison,
+        byEmployee: hoursData.byEmployee,
+        byWeek: hoursData.byWeek,
+      };
+
+      return response;
+      } finally {
+        // Always release the connection regardless of who wins Promise.race.
+        // If the timeout wins, this IIFE keeps running until MySQL responds; this
+        // finally ensures the connection still returns to the pool.
+        release();
+      }
+    };
+
+    const response = await Promise.race([mainOperation(), timeoutPromise]);
+
+    console.log(`[PAYROLL-METRICS] Request completed in ${Date.now() - startTime}ms`);
+    res.json(response);
+
+  } catch (error) {
+    if (error.message.includes('timeout')) {
+      console.log(`[PAYROLL-METRICS] Request timed out in ${Date.now() - startTime}ms`);
+    } else {
+      console.error('Error in /payroll-metrics:', error);
+    }
+    console.log(`[PAYROLL-METRICS] Request failed in ${Date.now() - startTime}ms`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Get pay period info endpoint (for navigation without full data)
+router.get('/period-info', async (req, res) => {
+  try {
+    const { payPeriodStart, navigate } = req.query;
+
+    let payPeriod;
+
+    if (payPeriodStart) {
+      const startDate = DateTime.fromISO(payPeriodStart, { zone: TIMEZONE });
+      if (!startDate.isValid) {
+        return res.status(400).json({ error: 'Invalid payPeriodStart date format' });
+      }
+      payPeriod = getPayPeriodForDate(startDate);
+    } else {
+      payPeriod = getCurrentPayPeriod();
+    }
+
+    if (navigate) {
+      const offset = parseInt(navigate, 10);
+      if (!isNaN(offset)) {
+        payPeriod = navigatePayPeriod(payPeriod.start, offset);
+      }
+    }
+
+    res.json({
+      payPeriod: {
+        start: payPeriod.start.toISODate(),
+        end: payPeriod.end.toISODate(),
+        label: formatPayPeriodLabel(payPeriod),
+        week1: {
+          start: payPeriod.week1.start.toISODate(),
+          end: payPeriod.week1.end.toISODate(),
+          label: formatWeekLabel(payPeriod.week1),
+        },
+        week2: {
+          start: payPeriod.week2.start.toISODate(),
+          end: payPeriod.week2.end.toISODate(),
+          label: formatWeekLabel(payPeriod.week2),
+        },
+        isCurrent: isCurrentPayPeriod(payPeriod),
+      },
+    });
+  } catch (error) {
+    console.error('Error in /payroll-metrics/period-info:', error);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Health check
+router.get('/health', async (req, res) => {
+  try {
+    const { connection, release } = await getDbConnection();
+    await connection.execute('SELECT 1 as test');
+    release();
+
+    res.json({
+      status: 'healthy',
+      timestamp: new Date().toISOString(),
+      pool: getPoolStatus(),
+    });
+  } catch (error) {
+    res.status(500).json({
+      status: 'unhealthy',
+      timestamp: new Date().toISOString(),
+      error: error.message,
+    });
+  }
+});
+
+// Helper functions
+function calculateComparison(currentValue, previousValue) {
+  if (typeof previousValue !== 'number') {
+    return { absolute: null, percentage: null };
+  }
+
+  const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
+  const percentage =
+    absolute !== null && previousValue !== 0
+      ? (absolute / Math.abs(previousValue)) * 100
+      : null;
+
+  return { absolute, percentage };
+}
+
+function formatPayPeriodLabel(payPeriod) {
+  const startStr = payPeriod.start.toFormat('MMM d');
+  const endStr = payPeriod.end.toFormat('MMM d, yyyy');
+  return `${startStr} – ${endStr}`;
+}
+
+function formatWeekLabel(week) {
+  const startStr = week.start.toFormat('MMM d');
+  const endStr = week.end.toFormat('MMM d');
+  return `${startStr} – ${endStr}`;
+}
+
+function isCurrentPayPeriod(payPeriod) {
+  const now = DateTime.now().setZone(TIMEZONE);
+  return now >= payPeriod.start && now <= payPeriod.end;
+}
+
+export default router;

inventory-server/dashboard/acot-server/routes/test.js

@@ -0,0 +1,58 @@
+import express from 'express';
+import { getDbConnection, getCachedQuery } from '../db/connection.js';
+
+const router = express.Router();
+
+// Test endpoint to count orders
+router.get('/order-count', async (req, res) => {
+  try {
+    const { connection } = await getDbConnection();
+    
+    // Simple query to count orders from _order table
+    const queryFn = async () => {
+      const [rows] = await connection.execute('SELECT COUNT(*) as count FROM _order');
+      return rows[0].count;
+    };
+    
+    const cacheKey = 'order-count';
+    const count = await getCachedQuery(cacheKey, 'default', queryFn);
+    
+    res.json({
+      success: true,
+      data: {
+        orderCount: count,
+        timestamp: new Date().toISOString()
+      }
+    });
+  } catch (error) {
+    console.error('Error fetching order count:', error);
+    res.status(500).json({
+      success: false,
+      error: error.message
+    });
+  }
+});
+
+// Test connection endpoint
+router.get('/test-connection', async (req, res) => {
+  try {
+    const { connection } = await getDbConnection();
+    
+    // Test the connection with a simple query
+    const [rows] = await connection.execute('SELECT 1 as test');
+    
+    res.json({
+      success: true,
+      message: 'Database connection successful',
+      data: rows[0]
+    });
+  } catch (error) {
+    console.error('Error testing connection:', error);
+    res.status(500).json({
+      success: false,
+      error: error.message
+    });
+  }
+});
+
+export default router; 

inventory-server/dashboard/acot-server/server.js

@@ -0,0 +1,163 @@
+// acot-server — Phase 5 of CONSOLIDATION_PLAN.md.
+// Standalone service on ACOT_PORT (default 3012) exposing /api/acot/* against
+// the production MySQL `sg` database via an ssh2 tunnel (see db/connection.js).
+//
+// Auth model (two flavors, deliberate):
+//   - /api/acot/customers/*  : x-acot-api-key shared secret (used by acot-phone-server).
+//                              Mounted BEFORE authenticate() so its requirePhoneApiKey
+//                              path is the only gate.
+//   - everything else        : JWT Bearer via shared/auth/middleware.js authenticate().
+//                              Defense-in-depth on top of Caddy forward_auth.
+//
+// Shared infrastructure (Phase 2 + Phase 6):
+//   - shared/auth/middleware.js  authenticate() for SPA-served routes
+//   - shared/cors/policy.js      explicit allowed-origins list (Phase 6.6)
+//   - shared/logging/request-log.js  pino-http, Authorization/Cookie redacted (Phase 6.5/6.9)
+//   - shared/errors/handler.js   consistent error envelope, no leak in prod
+//
+// Env layering: /var/www/inventory/.env loaded FIRST (JWT_SECRET, DB_* for the
+// shared PG pool used by authenticate to look up user permissions). Local .env
+// loaded SECOND for ACOT-specific keys (PROD_DB_*, PROD_SSH_*, ACOT_PHONE_API_KEY).
+// dotenv defaults to override:false, so the first file wins on collisions.
+
+import { config as loadEnv } from 'dotenv';
+import express from 'express';
+import cors from 'cors';
+import compression from 'compression';
+import morgan from 'morgan';
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import pg from 'pg';
+
+import { authenticate } from '../../shared/auth/middleware.js';
+import { corsOptions } from '../../shared/cors/policy.js';
+import { errorHandler } from '../../shared/errors/handler.js';
+import { logger } from '../../shared/logging/logger.js';
+import { requestLog } from '../../shared/logging/request-log.js';
+
+import { closeAllConnections } from './db/connection.js';
+
+import testRouter from './routes/test.js';
+import eventsRouter from './routes/events.js';
+import discountsRouter from './routes/discounts.js';
+import employeeMetricsRouter from './routes/employee-metrics.js';
+import payrollMetricsRouter from './routes/payroll-metrics.js';
+import operationsMetricsRouter from './routes/operations-metrics.js';
+import customersRouter from './routes/customers.js';
+
+const { Pool } = pg;
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+// Layer envs: shared inventory .env first (JWT_SECRET, DB_*) then acot .env.
+const sharedEnvPath = '/var/www/inventory/.env';
+const localEnvPath = path.resolve(__dirname, '.env');
+if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
+if (fs.existsSync(localEnvPath)) loadEnv({ path: localEnvPath });
+
+// Phase 6.4 — refuse to start without JWT_SECRET. authenticate() would reject
+// every request anyway; failing fast surfaces the misconfiguration immediately.
+if (!process.env.JWT_SECRET) {
+  logger.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
+  process.exit(1);
+}
+
+const app = express();
+const PORT = Number(process.env.ACOT_PORT) || 3012;
+
+// Trust X-Forwarded-* only when the immediate hop is loopback (Caddy on the same
+// host). Required for the KIOSK_IPS bypass in shared/auth/middleware.js to see
+// real client IPs instead of 127.0.0.1.
+app.set('trust proxy', 'loopback');
+
+// Postgres pool for authenticate() (user/permission lookups against inventory_db).
+// All MySQL access goes through db/connection.js (separate, ssh-tunneled).
+const pool = new Pool({
+  host: process.env.DB_HOST,
+  user: process.env.DB_USER,
+  password: process.env.DB_PASSWORD,
+  database: process.env.DB_NAME,
+  port: Number(process.env.DB_PORT) || 5432,
+});
+
+// Per-app access log on disk (kept from pre-conversion behavior; pino request-log
+// is mounted below for structured/redacted server-side logging).
+const logDir = path.join(__dirname, 'logs/app');
+if (!fs.existsSync(logDir)) {
+  fs.mkdirSync(logDir, { recursive: true });
+}
+const accessLogStream = fs.createWriteStream(path.join(logDir, 'access.log'), { flags: 'a' });
+
+app.use(requestLog());
+app.use(compression());
+app.use(cors(corsOptions));
+app.use(express.json());
+app.use(express.urlencoded({ extended: true }));
+if (process.env.NODE_ENV === 'production') {
+  app.use(morgan('combined', { stream: accessLogStream }));
+} else {
+  app.use(morgan('dev'));
+}
+
+app.get('/health', (req, res) => {
+  res.json({
+    status: 'healthy',
+    service: 'acot-server',
+    timestamp: new Date().toISOString(),
+    uptime: process.uptime(),
+  });
+});
+
+// Customers route uses x-acot-api-key (shared secret with acot-phone-server),
+// NOT JWT — mount BEFORE authenticate() so requirePhoneApiKey is the only gate.
+app.use('/api/acot/customers', customersRouter);
+
+// All remaining /api/acot/* routes require a valid JWT.
+app.use('/api/acot', authenticate({ pool, secret: process.env.JWT_SECRET }));
+
+app.use('/api/acot/test', testRouter);
+app.use('/api/acot/events', eventsRouter);
+app.use('/api/acot/discounts', discountsRouter);
+app.use('/api/acot/employee-metrics', employeeMetricsRouter);
+app.use('/api/acot/payroll-metrics', payrollMetricsRouter);
+app.use('/api/acot/operations-metrics', operationsMetricsRouter);
+
+// 404 for unmatched /api routes (keeps prior behavior).
+app.use((req, res) => {
+  res.status(404).json({ success: false, error: 'Route not found' });
+});
+
+app.use(errorHandler);
+
+const server = app.listen(PORT, '0.0.0.0', () => {
+  logger.info({ port: PORT, mode: process.env.NODE_ENV || 'development' }, 'acot-server listening');
+});
+
+const gracefulShutdown = async (signal) => {
+  logger.info({ signal }, 'acot-server shutting down');
+  server.close(async () => {
+    try {
+      await closeAllConnections();
+    } catch (err) {
+      logger.error({ err: { message: err.message } }, 'error closing MySQL pool');
+    }
+    try {
+      await pool.end();
+    } catch { /* ignore */ }
+    process.exit(0);
+  });
+};
+
+process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
+process.on('SIGINT', () => gracefulShutdown('SIGINT'));
+
+process.on('uncaughtException', (err) => {
+  logger.error({ err: { message: err.message, stack: err.stack } }, 'uncaughtException');
+  process.exit(1);
+});
+process.on('unhandledRejection', (reason) => {
+  logger.error({ reason }, 'unhandledRejection');
+});
+
+export default app;

inventory-server/dashboard/acot-server/utils/phoneAuth.js

@@ -0,0 +1,26 @@
+// Shared-secret auth for customer-lookup endpoints that expose PII.
+// The acot-phone-server sends `x-acot-api-key` on every request; we compare
+// against ACOT_PHONE_API_KEY from the environment using timing-safe comparison.
+
+import crypto from 'node:crypto';
+
+export function requirePhoneApiKey(req, res, next) {
+  const expected = process.env.ACOT_PHONE_API_KEY;
+  if (!expected) {
+    console.error('ACOT_PHONE_API_KEY not configured; rejecting all requests');
+    return res.status(503).json({ success: false, error: 'auth_not_configured' });
+  }
+
+  const provided = req.get('x-acot-api-key') || '';
+  const expectedBuf = Buffer.from(expected);
+  const providedBuf = Buffer.from(provided);
+
+  if (
+    providedBuf.length !== expectedBuf.length ||
+    !crypto.timingSafeEqual(providedBuf, expectedBuf)
+  ) {
+    return res.status(401).json({ success: false, error: 'unauthorized' });
+  }
+
+  next();
+}

inventory-server/dashboard/acot-server/utils/timeUtils.js

@@ -0,0 +1,317 @@
+import { DateTime } from 'luxon';
+
+const TIMEZONE = 'America/New_York';
+const DB_TIMEZONE = 'UTC-05:00';
+const BUSINESS_DAY_START_HOUR = 1; // 1 AM Eastern
+const WEEK_START_DAY = 7; // Sunday (Luxon uses 1 = Monday, 7 = Sunday)
+const DB_DATETIME_FORMAT = 'yyyy-LL-dd HH:mm:ss';
+
+const isDateTime = (value) => DateTime.isDateTime(value);
+
+const ensureDateTime = (value, { zone = TIMEZONE } = {}) => {
+  if (!value) return null;
+
+  if (isDateTime(value)) {
+    return value.setZone(zone);
+  }
+
+  if (value instanceof Date) {
+    return DateTime.fromJSDate(value, { zone });
+  }
+
+  if (typeof value === 'number') {
+    return DateTime.fromMillis(value, { zone });
+  }
+
+  if (typeof value === 'string') {
+    let dt = DateTime.fromISO(value, { zone, setZone: true });
+    if (!dt.isValid) {
+      dt = DateTime.fromSQL(value, { zone });
+    }
+    return dt.isValid ? dt : null;
+  }
+
+  return null;
+};
+
+const getNow = () => DateTime.now().setZone(TIMEZONE);
+
+const getDayStart = (input = getNow()) => {
+  const dt = ensureDateTime(input);
+  if (!dt || !dt.isValid) {
+    const fallback = getNow();
+    return fallback.set({
+      hour: BUSINESS_DAY_START_HOUR,
+      minute: 0,
+      second: 0,
+      millisecond: 0
+    });
+  }
+
+  const sameDayStart = dt.set({
+    hour: BUSINESS_DAY_START_HOUR,
+    minute: 0,
+    second: 0,
+    millisecond: 0
+  });
+
+  return dt.hour < BUSINESS_DAY_START_HOUR
+    ? sameDayStart.minus({ days: 1 })
+    : sameDayStart;
+};
+
+const getDayEnd = (input = getNow()) => {
+  return getDayStart(input).plus({ days: 1 }).minus({ milliseconds: 1 });
+};
+
+const getWeekStart = (input = getNow()) => {
+  const dt = ensureDateTime(input);
+  if (!dt || !dt.isValid) {
+    return getDayStart();
+  }
+
+  const startOfWeek = dt.set({ weekday: WEEK_START_DAY }).startOf('day');
+  const normalized = startOfWeek > dt ? startOfWeek.minus({ weeks: 1 }) : startOfWeek;
+  return normalized.set({
+    hour: BUSINESS_DAY_START_HOUR,
+    minute: 0,
+    second: 0,
+    millisecond: 0
+  });
+};
+
+const getRangeForTimeRange = (timeRange = 'today', now = getNow()) => {
+  const current = ensureDateTime(now);
+  if (!current || !current.isValid) {
+    throw new Error('Invalid reference time for range calculation');
+  }
+
+  switch (timeRange) {
+    case 'today': {
+      return {
+        start: getDayStart(current),
+        end: getDayEnd(current)
+      };
+    }
+    case 'yesterday': {
+      const target = current.minus({ days: 1 });
+      return {
+        start: getDayStart(target),
+        end: getDayEnd(target)
+      };
+    }
+    case 'twoDaysAgo': {
+      const target = current.minus({ days: 2 });
+      return {
+        start: getDayStart(target),
+        end: getDayEnd(target)
+      };
+    }
+    case 'thisWeek': {
+      return {
+        start: getWeekStart(current),
+        end: getDayEnd(current)
+      };
+    }
+    case 'lastWeek': {
+      const lastWeek = current.minus({ weeks: 1 });
+      const weekStart = getWeekStart(lastWeek);
+      const weekEnd = weekStart.plus({ days: 6 });
+      return {
+        start: weekStart,
+        end: getDayEnd(weekEnd)
+      };
+    }
+    case 'thisMonth': {
+      const dayStart = getDayStart(current);
+      const monthStart = dayStart.startOf('month').set({ hour: BUSINESS_DAY_START_HOUR });
+      return {
+        start: monthStart,
+        end: getDayEnd(current)
+      };
+    }
+    case 'lastMonth': {
+      const lastMonth = current.minus({ months: 1 });
+      const monthStart = lastMonth
+        .startOf('month')
+        .set({ hour: BUSINESS_DAY_START_HOUR, minute: 0, second: 0, millisecond: 0 });
+      const monthEnd = monthStart.plus({ months: 1 }).minus({ days: 1 });
+      return {
+        start: monthStart,
+        end: getDayEnd(monthEnd)
+      };
+    }
+    case 'last7days': {
+      const dayStart = getDayStart(current);
+      return {
+        start: dayStart.minus({ days: 6 }),
+        end: getDayEnd(current)
+      };
+    }
+    case 'last30days': {
+      const dayStart = getDayStart(current);
+      return {
+        start: dayStart.minus({ days: 29 }),
+        end: getDayEnd(current)
+      };
+    }
+    case 'last90days': {
+      const dayStart = getDayStart(current);
+      return {
+        start: dayStart.minus({ days: 89 }),
+        end: getDayEnd(current)
+      };
+    }
+    case 'previous7days': {
+      const currentPeriodStart = getDayStart(current).minus({ days: 6 });
+      const previousEndDay = currentPeriodStart.minus({ days: 1 });
+      const previousStartDay = previousEndDay.minus({ days: 6 });
+      return {
+        start: getDayStart(previousStartDay),
+        end: getDayEnd(previousEndDay)
+      };
+    }
+    case 'previous30days': {
+      const currentPeriodStart = getDayStart(current).minus({ days: 29 });
+      const previousEndDay = currentPeriodStart.minus({ days: 1 });
+      const previousStartDay = previousEndDay.minus({ days: 29 });
+      return {
+        start: getDayStart(previousStartDay),
+        end: getDayEnd(previousEndDay)
+      };
+    }
+    case 'previous90days': {
+      const currentPeriodStart = getDayStart(current).minus({ days: 89 });
+      const previousEndDay = currentPeriodStart.minus({ days: 1 });
+      const previousStartDay = previousEndDay.minus({ days: 89 });
+      return {
+        start: getDayStart(previousStartDay),
+        end: getDayEnd(previousEndDay)
+      };
+    }
+    default:
+      throw new Error(`Unknown time range: ${timeRange}`);
+  }
+};
+
+const toDatabaseSqlString = (dt) => {
+  const normalized = ensureDateTime(dt);
+  if (!normalized || !normalized.isValid) {
+    throw new Error('Invalid datetime provided for SQL conversion');
+  }
+  const dbTime = normalized.setZone(DB_TIMEZONE, { keepLocalTime: true });
+  return dbTime.toFormat(DB_DATETIME_FORMAT);
+};
+
+const formatBusinessDate = (input) => {
+  const dt = ensureDateTime(input);
+  if (!dt || !dt.isValid) return '';
+  return dt.setZone(TIMEZONE).toFormat('LLL d, yyyy');
+};
+
+const getTimeRangeLabel = (timeRange) => {
+  const labels = {
+    today: 'Today',
+    yesterday: 'Yesterday',
+    twoDaysAgo: 'Two Days Ago',
+    thisWeek: 'This Week',
+    lastWeek: 'Last Week',
+    thisMonth: 'This Month',
+    lastMonth: 'Last Month',
+    last7days: 'Last 7 Days',
+    last30days: 'Last 30 Days',
+    last90days: 'Last 90 Days',
+    previous7days: 'Previous 7 Days',
+    previous30days: 'Previous 30 Days',
+    previous90days: 'Previous 90 Days'
+  };
+
+  return labels[timeRange] || timeRange;
+};
+
+const getTimeRangeConditions = (timeRange, startDate, endDate) => {
+  if (timeRange === 'custom' && startDate && endDate) {
+    const start = ensureDateTime(startDate);
+    const end = ensureDateTime(endDate);
+
+    if (!start || !start.isValid || !end || !end.isValid) {
+      throw new Error('Invalid custom date range provided');
+    }
+
+    return {
+      whereClause: 'date_placed >= ? AND date_placed <= ?',
+      params: [toDatabaseSqlString(start), toDatabaseSqlString(end)],
+      dateRange: {
+        start: start.toUTC().toISO(),
+        end: end.toUTC().toISO(),
+        label: `${formatBusinessDate(start)} - ${formatBusinessDate(end)}`
+      }
+    };
+  }
+
+  const normalizedRange = timeRange || 'today';
+  const range = getRangeForTimeRange(normalizedRange);
+
+  return {
+    whereClause: 'date_placed >= ? AND date_placed <= ?',
+    params: [toDatabaseSqlString(range.start), toDatabaseSqlString(range.end)],
+    dateRange: {
+      start: range.start.toUTC().toISO(),
+      end: range.end.toUTC().toISO(),
+      label: getTimeRangeLabel(normalizedRange)
+    }
+  };
+};
+
+const getBusinessDayBounds = (timeRange) => {
+  const range = getRangeForTimeRange(timeRange);
+  return {
+    start: range.start.toJSDate(),
+    end: range.end.toJSDate()
+  };
+};
+
+const parseBusinessDate = (mysqlDatetime) => {
+  if (!mysqlDatetime || mysqlDatetime === '0000-00-00 00:00:00') {
+    return null;
+  }
+
+  const dt = DateTime.fromSQL(mysqlDatetime, { zone: DB_TIMEZONE });
+  if (!dt.isValid) {
+    console.error('[timeUtils] Failed to parse MySQL datetime:', mysqlDatetime, dt.invalidExplanation);
+    return null;
+  }
+
+  return dt.toUTC().toJSDate();
+};
+
+const formatMySQLDate = (input) => {
+  if (!input) return null;
+
+  const dt = ensureDateTime(input, { zone: 'utc' });
+  if (!dt || !dt.isValid) return null;
+
+  return dt.setZone(DB_TIMEZONE).toFormat(DB_DATETIME_FORMAT);
+};
+
+// Expose helpers for tests or advanced consumers.
+// Kept as a named `_internal` export so existing destructuring sites
+// (`const { _internal: timeHelpers } = require(...)` → ESM equivalent works)
+// don't need to change beyond the import-statement rewrite.
+const _internal = {
+  getDayStart,
+  getDayEnd,
+  getWeekStart,
+  getRangeForTimeRange,
+  BUSINESS_DAY_START_HOUR,
+};
+
+export {
+  getBusinessDayBounds,
+  getTimeRangeConditions,
+  formatBusinessDate,
+  getTimeRangeLabel,
+  parseBusinessDate,
+  formatMySQLDate,
+  _internal,
+};

inventory-server/dashboard/package.json

@@ -0,0 +1,29 @@
+{
+  "name": "dashboard-server",
+  "version": "1.0.0",
+  "description": "Merged ESM dashboard server (klaviyo + meta + google-analytics + typeform). Phase 4 of CONSOLIDATION_PLAN.md.",
+  "main": "server.js",
+  "type": "module",
+  "scripts": {
+    "start": "node server.js",
+    "dev": "nodemon server.js"
+  },
+  "dependencies": {
+    "@google-analytics/data": "^4.0.0",
+    "axios": "^1.7.9",
+    "cors": "^2.8.5",
+    "dotenv": "^16.4.7",
+    "express": "^4.21.2",
+    "ioredis": "^5.4.1",
+    "jsonwebtoken": "^9.0.2",
+    "lodash": "^4.17.21",
+    "luxon": "^3.5.0",
+    "node-fetch": "^3.3.2",
+    "pg": "^8.18.0",
+    "pino": "^9.5.0",
+    "pino-http": "^10.3.0"
+  },
+  "devDependencies": {
+    "nodemon": "^3.0.2"
+  }
+}

inventory-server/dashboard/routes/google/index.js

@@ -0,0 +1,55 @@
+// Google Analytics router — ESM conversion of google-server/routes/analytics.routes.js.
+// All routes are read-only — authenticated-only is sufficient; no extra permission.
+// google_write is reserved for future write endpoints (per migration 005).
+
+import express from 'express';
+import { AnalyticsService } from '../../services/google/analytics.service.js';
+
+export function createGoogleRouter({ redis }) {
+  const router = express.Router();
+  const service = new AnalyticsService(redis);
+
+  router.get('/metrics', async (req, res) => {
+    try {
+      const { startDate = '7daysAgo' } = req.query;
+      const data = await service.getBasicMetrics(startDate);
+      res.json({ success: true, data });
+    } catch (error) {
+      console.error('Metrics error:', { startDate: req.query.startDate, error: error.message });
+      res.status(500).json({ success: false, error: 'Failed to fetch metrics', details: error.message });
+    }
+  });
+
+  router.get('/realtime/basic', async (req, res) => {
+    try {
+      const data = await service.getRealTimeBasicData();
+      res.json({ success: true, data });
+    } catch (error) {
+      console.error('Realtime basic error:', { error: error.message });
+      res.status(500).json({ success: false, error: 'Failed to fetch realtime basic data', details: error.message });
+    }
+  });
+
+  router.get('/realtime/detailed', async (req, res) => {
+    try {
+      const data = await service.getRealTimeDetailedData();
+      res.json({ success: true, data });
+    } catch (error) {
+      console.error('Realtime detailed error:', { error: error.message });
+      res.status(500).json({ success: false, error: 'Failed to fetch realtime detailed data', details: error.message });
+    }
+  });
+
+  router.get('/user-behavior', async (req, res) => {
+    try {
+      const { timeRange = '30' } = req.query;
+      const data = await service.getUserBehavior(timeRange);
+      res.json({ success: true, data });
+    } catch (error) {
+      console.error('User behavior error:', { timeRange: req.query.timeRange, error: error.message });
+      res.status(500).json({ success: false, error: 'Failed to fetch user behavior data', details: error.message });
+    }
+  });
+
+  return router;
+}

inventory-server/dashboard/routes/klaviyo/campaigns.routes.js

@@ -0,0 +1,71 @@
+import express from 'express';
+import { CampaignsService } from '../../services/klaviyo/campaigns.service.js';
+import { TimeManager } from '../../utils/time.utils.js';
+
+export function createCampaignsRouter(apiKey, apiRevision, redis) {
+  const router = express.Router();
+  const timeManager = new TimeManager();
+  const campaignsService = new CampaignsService(apiKey, apiRevision, redis);
+
+  // Get campaigns with optional filtering
+  router.get('/', async (req, res) => {
+    try {
+      const params = {
+        pageSize: parseInt(req.query.pageSize) || 50,
+        sort: req.query.sort || '-send_time',
+        status: req.query.status,
+        startDate: req.query.startDate,
+        endDate: req.query.endDate,
+        pageCursor: req.query.pageCursor
+      };
+
+      console.log('[Campaigns Route] Fetching campaigns with params:', params);
+      const data = await campaignsService.getCampaigns(params);
+      console.log('[Campaigns Route] Success:', {
+        count: data.data?.length || 0
+      });
+      res.json(data);
+    } catch (error) {
+      console.error('[Campaigns Route] Error:', error);
+      res.status(500).json({ 
+        status: 'error', 
+        message: error.message,
+        details: error.response?.data || null
+      });
+    }
+  });
+
+  // Get campaigns by time range
+  router.get('/:timeRange', async (req, res) => {
+    try {
+      const { timeRange } = req.params;
+      const { status } = req.query;
+
+      let result;
+      if (timeRange === 'custom') {
+        const { startDate, endDate } = req.query;
+        if (!startDate || !endDate) {
+          return res.status(400).json({ error: 'Custom range requires startDate and endDate' });
+        }
+
+        result = await campaignsService.getCampaigns({
+          startDate,
+          endDate,
+          status
+        });
+      } else {
+        result = await campaignsService.getCampaignsByTimeRange(
+          timeRange,
+          { status }
+        );
+      }
+
+      res.json(result);
+    } catch (error) {
+      console.error("[Campaigns Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  return router;
+} 

inventory-server/dashboard/routes/klaviyo/events.routes.js

@@ -0,0 +1,485 @@
+import express from 'express';
+import { EventsService } from '../../services/klaviyo/events.service.js';
+import { TimeManager } from '../../utils/time.utils.js';
+import { RedisService } from '../../services/klaviyo/redis.service.js';
+import { requirePermission } from '../../../shared/auth/middleware.js';
+
+// Import METRIC_IDS from events service
+const METRIC_IDS = {
+  PLACED_ORDER: 'Y8cqcF',
+  SHIPPED_ORDER: 'VExpdL',
+  ACCOUNT_CREATED: 'TeeypV',
+  CANCELED_ORDER: 'YjVMNg',
+  NEW_BLOG_POST: 'YcxeDr',
+  PAYMENT_REFUNDED: 'R7XUYh'
+};
+
+export function createEventsRouter(apiKey, apiRevision, redis) {
+  const router = express.Router();
+  const timeManager = new TimeManager();
+  const eventsService = new EventsService(apiKey, apiRevision, redis);
+  const redisService = new RedisService(redis);
+
+  // Phase 6.2: clearCache is operational maintenance — requires klaviyo_admin.
+  // Mounted as path-level middleware so the existing POST handler below stays untouched.
+  router.use('/clearCache', requirePermission('klaviyo_admin'));
+
+  // Get events with optional filtering
+  router.get('/', async (req, res) => {
+    try {
+      const params = {
+        pageSize: parseInt(req.query.pageSize) || 50,
+        sort: req.query.sort || '-datetime',
+        metricId: req.query.metricId,
+        startDate: req.query.startDate,
+        endDate: req.query.endDate,
+        pageCursor: req.query.pageCursor,
+        fields: {}
+      };
+
+      // Parse fields parameter if provided
+      if (req.query.fields) {
+        try {
+          params.fields = JSON.parse(req.query.fields);
+        } catch (e) {
+          console.warn('[Events Route] Invalid fields parameter:', e);
+        }
+      }
+
+      console.log('[Events Route] Fetching events with params:', params);
+      const data = await eventsService.getEvents(params);
+      console.log('[Events Route] Success:', {
+        count: data.data?.length || 0,
+        included: data.included?.length || 0
+      });
+      res.json(data);
+    } catch (error) {
+      console.error('[Events Route] Error:', error);
+      res.status(500).json({ 
+        status: 'error', 
+        message: error.message,
+        details: error.response?.data || null
+      });
+    }
+  });
+
+  // Get events by time range
+  router.get('/by-time/:timeRange', async (req, res) => {
+    try {
+      const { timeRange } = req.params;
+      const { metricId, startDate, endDate } = req.query;
+
+      let result;
+      if (timeRange === 'custom') {
+        if (!startDate || !endDate) {
+          return res.status(400).json({ error: 'Custom range requires startDate and endDate' });
+        }
+
+        const range = timeManager.getCustomRange(startDate, endDate);
+        if (!range) {
+          return res.status(400).json({ error: 'Invalid date range' });
+        }
+
+        result = await eventsService.getEvents({
+          metricId,
+          startDate: range.start.toISO(),
+          endDate: range.end.toISO()
+        });
+      } else {
+        result = await eventsService.getEventsByTimeRange(
+          timeRange,
+          { metricId }
+        );
+      }
+
+      res.json(result);
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Get comprehensive statistics for a time period
+  router.get('/stats', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate } = req.query;
+      console.log('[Events Route] Stats request:', {
+        timeRange,
+        startDate,
+        endDate
+      });
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO()
+      };
+
+      console.log('[Events Route] Calculating period stats with params:', params);
+      const stats = await eventsService.calculatePeriodStats(params);
+      console.log('[Events Route] Stats response:', {
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO()
+        },
+        shippedCount: stats?.shipping?.shippedCount,
+        totalOrders: stats?.orderCount
+      });
+
+      res.json({
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO(),
+          displayStart: timeManager.formatForDisplay(range.start),
+          displayEnd: timeManager.formatForDisplay(range.end)
+        },
+        stats
+      });
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Add new route for smart revenue projection
+  router.get('/projection', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate } = req.query;
+      console.log('[Events Route] Projection request:', {
+        timeRange,
+        startDate,
+        endDate
+      });
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO()
+      };
+
+      // Try to get from cache first with a short TTL
+      const cacheKey = redisService._getCacheKey('projection', params);
+      const cachedData = await redisService.get(cacheKey);
+      
+      if (cachedData) {
+        console.log('[Events Route] Cache hit for projection');
+        return res.json(cachedData);
+      }
+
+      console.log('[Events Route] Calculating smart projection with params:', params);
+      const projection = await eventsService.calculateSmartProjection(params);
+      
+      // Cache the results with a short TTL (5 minutes)
+      await redisService.set(cacheKey, projection, 300);
+
+      res.json(projection);
+    } catch (error) {
+      console.error("[Events Route] Error calculating projection:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Add new route for detailed stats
+  router.get('/stats/details', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate, metric, daily = false } = req.query;
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO(),
+        metric,
+        daily: daily === 'true' || daily === true
+      };
+
+      // Try to get from cache first
+      const cacheKey = redisService._getCacheKey('stats:details', params);
+      const cachedData = await redisService.get(cacheKey);
+      
+      if (cachedData) {
+        console.log('[Events Route] Cache hit for detailed stats');
+        return res.json({
+          timeRange: {
+            start: range.start.toISO(),
+            end: range.end.toISO(),
+            displayStart: timeManager.formatForDisplay(range.start),
+            displayEnd: timeManager.formatForDisplay(range.end)
+          },
+          stats: cachedData
+        });
+      }
+
+      const stats = await eventsService.calculateDetailedStats(params);
+
+      // Cache the results
+      const ttl = redisService._getTTL(timeRange);
+      await redisService.set(cacheKey, stats, ttl);
+
+      res.json({
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO(),
+          displayStart: timeManager.formatForDisplay(range.start),
+          displayEnd: timeManager.formatForDisplay(range.end)
+        },
+        stats
+      });
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Get product statistics for a time period
+  router.get('/products', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate } = req.query;
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO()
+      };
+
+      // Try to get from cache first
+      const cacheKey = redisService._getCacheKey('events', params);
+      const cachedData = await redisService.getEventData('products', params);
+      
+      if (cachedData) {
+        console.log('[Events Route] Cache hit for products');
+        return res.json({
+          timeRange: {
+            start: range.start.toISO(),
+            end: range.end.toISO(),
+            displayStart: timeManager.formatForDisplay(range.start),
+            displayEnd: timeManager.formatForDisplay(range.end)
+          },
+          stats: {
+            products: cachedData
+          }
+        });
+      }
+
+      const stats = await eventsService.calculatePeriodStats(params);
+
+      res.json({
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO(),
+          displayStart: timeManager.formatForDisplay(range.start),
+          displayEnd: timeManager.formatForDisplay(range.end)
+        },
+        stats
+      });
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Get event feed (multiple event types sorted by time)
+  router.get('/feed', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate, metricIds } = req.query;
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO(),
+        metricIds: metricIds ? JSON.parse(metricIds) : null
+      };
+
+      const result = await eventsService.getMultiMetricEvents(params);
+
+      res.json({
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO(),
+          displayStart: timeManager.formatForDisplay(range.start),
+          displayEnd: timeManager.formatForDisplay(range.end)
+        },
+        ...result
+      });
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Get aggregated events data
+  router.get('/aggregate', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate, interval = 'day', metricId, property } = req.query;
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else if (timeRange) {
+        range = timeManager.getDateRange(timeRange);
+      } else {
+        return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
+      }
+
+      if (!range) {
+        return res.status(400).json({ error: 'Invalid time range' });
+      }
+
+      const params = {
+        timeRange,
+        startDate: range.start.toISO(),
+        endDate: range.end.toISO(),
+        metricId,
+        interval,
+        property
+      };
+
+      const result = await eventsService.getEvents(params);
+      const groupedData = timeManager.groupEventsByInterval(result.data, interval, property);
+
+      res.json({
+        timeRange: {
+          start: range.start.toISO(),
+          end: range.end.toISO(),
+          displayStart: timeManager.formatForDisplay(range.start),
+          displayEnd: timeManager.formatForDisplay(range.end)
+        },
+        data: groupedData
+      });
+    } catch (error) {
+      console.error("[Events Route] Error:", error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  // Get date range for a given time period
+  router.get("/dateRange", async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate } = req.query;
+
+      let range;
+      if (startDate && endDate) {
+        range = timeManager.getCustomRange(startDate, endDate);
+      } else {
+        range = timeManager.getDateRange(timeRange || 'today');
+      }
+
+      if (!range) {
+        return res.status(400).json({ 
+          error: "Invalid time range parameters" 
+        });
+      }
+
+      res.json({
+        start: range.start.toISO(),
+        end: range.end.toISO(),
+        displayStart: timeManager.formatForDisplay(range.start),
+        displayEnd: timeManager.formatForDisplay(range.end)
+      });
+    } catch (error) {
+      console.error('Error getting date range:', error);
+      res.status(500).json({ 
+        error: "Failed to get date range" 
+      });
+    }
+  });
+
+  // Clear cache for a specific time range
+  router.post("/clearCache", async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate } = req.body;
+      await redisService.clearCache({ timeRange, startDate, endDate });
+      res.json({ message: "Cache cleared successfully" });
+    } catch (error) {
+      console.error('Error clearing cache:', error);
+      res.status(500).json({ error: "Failed to clear cache" });
+    }
+  });
+
+  // Add new batch metrics endpoint
+  router.get('/batch', async (req, res) => {
+    try {
+      const { timeRange, startDate, endDate, metrics } = req.query;
+      
+      // Parse metrics array from query
+      const metricsList = metrics ? JSON.parse(metrics) : [];
+      
+      const params = timeRange === 'custom'
+        ? { startDate, endDate, metrics: metricsList }
+        : { timeRange, metrics: metricsList };
+
+      const results = await eventsService.getBatchMetrics(params);
+      
+      res.json(results);
+    } catch (error) {
+      console.error('[Events Route] Error in batch request:', error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  return router;
+} 

inventory-server/dashboard/routes/klaviyo/index.js

@@ -0,0 +1,48 @@
+// Klaviyo router factory. Phase 4 merge: takes the injected redis client and
+// the env-resolved API key/revision, returns the mounted /api/klaviyo router
+// (matches Caddy proxy path; no rewrite needed).
+
+import express from 'express';
+import rateLimit from 'express-rate-limit';
+import { requirePermission } from '../../../shared/auth/middleware.js';
+
+import { createEventsRouter } from './events.routes.js';
+import { createMetricsRouter } from './metrics.routes.js';
+import { createCampaignsRouter } from './campaigns.routes.js';
+import { createReportingRouter } from './reporting.routes.js';
+
+export function createKlaviyoRouter({ redis }) {
+  const apiKey = process.env.KLAVIYO_API_KEY;
+  const apiRevision = process.env.KLAVIYO_API_REVISION || '2024-02-15';
+
+  if (!apiKey) {
+    // Loud at startup; the routes themselves will 500 on every call without it.
+    console.warn('[klaviyo] KLAVIYO_API_KEY not set — Klaviyo endpoints will fail');
+  }
+
+  const router = express.Router();
+
+  // Phase 4 carryover from klaviyo-server: throttle the heavy /reporting/campaign-values-reports
+  // endpoint. authenticate() already runs upstream so we don't add a per-user limiter here.
+  const reportingLimiter = rateLimit({
+    windowMs: 10 * 60 * 1000,
+    max: 10,
+    message: 'Too many requests to reporting endpoint, please try again later',
+    keyGenerator: (req) => `${req.ip}-klaviyo-reporting`,
+    skip: (req) => !req.path.includes('campaign-values-reports'),
+    standardHeaders: true,
+    legacyHeaders: false,
+  });
+  router.use('/reporting', reportingLimiter);
+
+  router.use('/events', createEventsRouter(apiKey, apiRevision, redis));
+  router.use('/metrics', createMetricsRouter(apiKey, apiRevision));
+  router.use('/campaigns', createCampaignsRouter(apiKey, apiRevision, redis));
+  router.use('/reporting', createReportingRouter(apiKey, apiRevision, redis));
+
+  return router;
+}
+
+// Re-exported so the dashboard server / future tests can attach the
+// klaviyo_admin gate without reaching into the events router file.
+export { requirePermission };

inventory-server/dashboard/routes/klaviyo/metrics.routes.js

@@ -0,0 +1,28 @@
+import express from 'express';
+import { MetricsService } from '../../services/klaviyo/metrics.service.js';
+
+export function createMetricsRouter(apiKey, apiRevision) {
+  const router = express.Router();
+  const metricsService = new MetricsService(apiKey, apiRevision);
+
+  // Get all metrics
+  router.get('/', async (req, res) => {
+    try {
+      console.log('[Metrics Route] Fetching metrics');
+      const data = await metricsService.getMetrics();
+      console.log('[Metrics Route] Success:', {
+        count: data.data?.length || 0
+      });
+      res.json(data);
+    } catch (error) {
+      console.error('[Metrics Route] Error:', error);
+      res.status(500).json({ 
+        status: 'error', 
+        message: error.message,
+        details: error.response?.data || null
+      });
+    }
+  });
+
+  return router;
+} 

inventory-server/dashboard/routes/klaviyo/reporting.routes.js

@@ -0,0 +1,29 @@
+import express from 'express';
+import { ReportingService } from '../../services/klaviyo/reporting.service.js';
+import { TimeManager } from '../../utils/time.utils.js';
+
+export function createReportingRouter(apiKey, apiRevision, redis) {
+  const router = express.Router();
+  const reportingService = new ReportingService(apiKey, apiRevision, redis);
+  const timeManager = new TimeManager();
+
+  // Get campaign reports by time range
+  router.get('/campaigns/:timeRange', async (req, res) => {
+    try {
+      const { timeRange } = req.params;
+      const { channel } = req.query;
+      
+      const reports = await reportingService.getCampaignReports({
+        timeRange,
+        channel
+      });
+
+      res.json(reports);
+    } catch (error) {
+      console.error('[ReportingRoutes] Error fetching campaign reports:', error);
+      res.status(500).json({ error: error.message });
+    }
+  });
+
+  return router;
+} 

inventory-server/dashboard/routes/meta/index.js

@@ -0,0 +1,89 @@
+// Meta router factory — ESM conversion of meta-server/routes/campaigns.routes.js.
+// Phase 6.2: mutations (PATCH /campaigns/:id/budget, POST /campaigns/:id/:action)
+// require the `meta_write` permission. Reads (GET) stay authenticated-only.
+
+import express from 'express';
+import { requirePermission } from '../../../shared/auth/middleware.js';
+import {
+  fetchCampaigns,
+  fetchAccountInsights,
+  updateCampaignBudget,
+  updateCampaignStatus,
+} from '../../services/meta/meta.service.js';
+
+export function createMetaRouter() {
+  const router = express.Router();
+
+  // Reads — authenticated-only
+  router.get('/campaigns', async (req, res, next) => {
+    try {
+      const { since, until } = req.query;
+      if (!since || !until) {
+        return res.status(400).json({ error: 'Date range is required (since, until)' });
+      }
+      const campaigns = await fetchCampaigns(since, until);
+      res.json(campaigns);
+    } catch (error) {
+      console.error('Campaign fetch error:', error);
+      res.status(500).json({
+        error: 'Failed to fetch campaigns',
+        details: error.response?.data?.error?.message || error.message,
+      });
+    }
+  });
+
+  router.get('/account-insights', async (req, res) => {
+    try {
+      const { since, until } = req.query;
+      if (!since || !until) {
+        return res.status(400).json({ error: 'Date range is required (since, until)' });
+      }
+      const insights = await fetchAccountInsights(since, until);
+      res.json(insights);
+    } catch (error) {
+      console.error('Account insights fetch error:', error);
+      res.status(500).json({
+        error: 'Failed to fetch account insights',
+        details: error.response?.data?.error?.message || error.message,
+      });
+    }
+  });
+
+  // Writes — meta_write
+  router.patch('/campaigns/:campaignId/budget', requirePermission('meta_write'), async (req, res) => {
+    try {
+      const { campaignId } = req.params;
+      const { budget } = req.body;
+      if (!budget) {
+        return res.status(400).json({ error: 'Budget is required' });
+      }
+      const result = await updateCampaignBudget(campaignId, budget);
+      res.json(result);
+    } catch (error) {
+      console.error('Budget update error:', error);
+      res.status(500).json({
+        error: 'Failed to update campaign budget',
+        details: error.response?.data?.error?.message || error.message,
+      });
+    }
+  });
+
+  router.post('/campaigns/:campaignId/:action', requirePermission('meta_write'), async (req, res) => {
+    try {
+      const { campaignId, action } = req.params;
+      if (!['pause', 'unpause'].includes(action)) {
+        return res.status(400).json({ error: 'Invalid action. Use "pause" or "unpause"' });
+      }
+      const result = await updateCampaignStatus(campaignId, action);
+      res.json(result);
+    } catch (error) {
+      console.error('Status update error:', error);
+      res.status(500).json({
+        error: 'Failed to update campaign status',
+        details: error.response?.data?.error?.message || error.message,
+      });
+    }
+  });
+
+  return router;
+}

inventory-server/dashboard/routes/typeform/index.js

@@ -0,0 +1,84 @@
+// Typeform router — ESM conversion of typeform-server/routes/typeform.routes.js.
+// All routes read-only — authenticated-only is sufficient; typeform_write reserved
+// for future write endpoints (per migration 005).
+
+import express from 'express';
+import { TypeformService } from '../../services/typeform/typeform.service.js';
+
+export function createTypeformRouter({ redis }) {
+  const router = express.Router();
+  const typeform = new TypeformService(redis);
+
+  router.get('/forms/:formId/responses', async (req, res) => {
+    try {
+      const { formId } = req.params;
+      const filters = req.query;
+      if (!formId) {
+        return res.status(400).json({ error: 'Missing form ID', details: 'The form ID parameter is required' });
+      }
+      const data = await typeform.getFormResponsesWithFilters(formId, filters);
+      if (!data) {
+        return res.status(404).json({ error: 'No data found', details: `No responses found for form ${formId}` });
+      }
+      res.json(data);
+    } catch (error) {
+      console.error('Form responses error:', {
+        formId: req.params.formId,
+        filters: req.query,
+        error: error.message,
+        response: error.response?.data,
+      });
+      if (error.response?.status === 401) {
+        return res.status(401).json({ error: 'Authentication failed', details: 'Invalid Typeform API credentials' });
+      }
+      if (error.response?.status === 404) {
+        return res.status(404).json({ error: 'Not found', details: `Form '${req.params.formId}' not found` });
+      }
+      if (error.response?.status === 400) {
+        return res.status(400).json({
+          error: 'Invalid request',
+          details: error.response?.data?.message || 'The request was invalid',
+          data: error.response?.data,
+        });
+      }
+      res.status(500).json({
+        error: 'Failed to fetch form responses',
+        details: error.response?.data?.message || error.message,
+        data: error.response?.data,
+      });
+    }
+  });
+
+  router.get('/forms/:formId/insights', async (req, res) => {
+    try {
+      const { formId } = req.params;
+      if (!formId) {
+        return res.status(400).json({ error: 'Missing form ID', details: 'The form ID parameter is required' });
+      }
+      const data = await typeform.getFormInsights(formId);
+      if (!data) {
+        return res.status(404).json({ error: 'No data found', details: `No insights found for form ${formId}` });
+      }
+      res.json(data);
+    } catch (error) {
+      console.error('Form insights error:', {
+        formId: req.params.formId,
+        error: error.message,
+        response: error.response?.data,
+      });
+      if (error.response?.status === 401) {
+        return res.status(401).json({ error: 'Authentication failed', details: 'Invalid Typeform API credentials' });
+      }
+      if (error.response?.status === 404) {
+        return res.status(404).json({ error: 'Not found', details: `Form '${req.params.formId}' not found` });
+      }
+      res.status(500).json({
+        error: 'Failed to fetch form insights',
+        details: error.response?.data?.message || error.message,
+        data: error.response?.data,
+      });
+    }
+  });
+
+  return router;
+}

inventory-server/dashboard/scripts/create-campaign-products-table.sql

@@ -0,0 +1,30 @@
+-- Stores individual product links found in Klaviyo campaign emails
+CREATE TABLE IF NOT EXISTS klaviyo_campaign_products (
+  id SERIAL PRIMARY KEY,
+  campaign_id TEXT NOT NULL,
+  campaign_name TEXT,
+  sent_at TIMESTAMPTZ,
+  pid BIGINT NOT NULL,
+  product_url TEXT,
+  created_at TIMESTAMPTZ DEFAULT NOW(),
+  UNIQUE(campaign_id, pid)
+);
+
+CREATE INDEX IF NOT EXISTS idx_kcp_campaign_id ON klaviyo_campaign_products(campaign_id);
+CREATE INDEX IF NOT EXISTS idx_kcp_pid ON klaviyo_campaign_products(pid);
+CREATE INDEX IF NOT EXISTS idx_kcp_sent_at ON klaviyo_campaign_products(sent_at);
+
+-- Stores non-product shop links (categories, filters, etc.) found in campaigns
+CREATE TABLE IF NOT EXISTS klaviyo_campaign_links (
+  id SERIAL PRIMARY KEY,
+  campaign_id TEXT NOT NULL,
+  campaign_name TEXT,
+  sent_at TIMESTAMPTZ,
+  link_url TEXT NOT NULL,
+  link_type TEXT, -- 'category', 'brand', 'filter', 'clearance', 'deals', 'other'
+  created_at TIMESTAMPTZ DEFAULT NOW(),
+  UNIQUE(campaign_id, link_url)
+);
+
+CREATE INDEX IF NOT EXISTS idx_kcl_campaign_id ON klaviyo_campaign_links(campaign_id);
+CREATE INDEX IF NOT EXISTS idx_kcl_sent_at ON klaviyo_campaign_links(sent_at);

inventory-server/dashboard/scripts/import-campaign-products.js

@@ -0,0 +1,279 @@
+/**
+ * Extract products featured in Klaviyo campaign emails and store in DB.
+ *
+ * - Fetches recent sent campaigns from Klaviyo API
+ * - Gets template HTML for each campaign message
+ * - Parses out product links (/shop/{id}) and other shop links
+ * - Inserts into klaviyo_campaign_products and klaviyo_campaign_links tables
+ *
+ * Usage: node scripts/poc-campaign-products.js [limit] [offset]
+ *   limit:  number of sent campaigns to process (default: 10)
+ *   offset: number of sent campaigns to skip before processing (default: 0)
+ *
+ * Requires DB_* env vars (from inventory-server .env) and KLAVIYO_API_KEY.
+ */
+
+import fetch from 'node-fetch';
+import pg from 'pg';
+import dotenv from 'dotenv';
+import path from 'path';
+import fs from 'fs';
+import { fileURLToPath } from 'url';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+// Load klaviyo .env for API key
+dotenv.config({ path: path.resolve(__dirname, '../.env') });
+// Also load the main inventory-server .env for DB credentials
+const mainEnvPath = '/var/www/inventory/.env';
+if (fs.existsSync(mainEnvPath)) {
+  dotenv.config({ path: mainEnvPath });
+}
+
+const API_KEY = process.env.KLAVIYO_API_KEY;
+const REVISION = process.env.KLAVIYO_API_REVISION || '2026-01-15';
+const BASE_URL = 'https://a.klaviyo.com/api';
+
+if (!API_KEY) {
+  console.error('KLAVIYO_API_KEY not set in .env');
+  process.exit(1);
+}
+
+// ── Klaviyo API helpers ──────────────────────────────────────────────
+
+const headers = {
+  'Accept': 'application/json',
+  'Content-Type': 'application/json',
+  'Authorization': `Klaviyo-API-Key ${API_KEY}`,
+  'revision': REVISION,
+};
+
+async function klaviyoGet(endpoint, params = {}) {
+  const url = new URL(`${BASE_URL}${endpoint}`);
+  for (const [k, v] of Object.entries(params)) {
+    url.searchParams.append(k, v);
+  }
+  return klaviyoFetch(url.toString());
+}
+
+async function klaviyoFetch(url) {
+  const res = await fetch(url, { headers });
+  if (!res.ok) {
+    const body = await res.text();
+    throw new Error(`Klaviyo ${res.status} on ${url}: ${body}`);
+  }
+  return res.json();
+}
+
+async function getRecentCampaigns(limit, offset = 0) {
+  const campaigns = [];
+  const messageMap = {};
+  let skipped = 0;
+
+  let data = await klaviyoGet('/campaigns', {
+    'filter': 'equals(messages.channel,"email")',
+    'sort': '-scheduled_at',
+    'include': 'campaign-messages',
+  });
+
+  while (true) {
+    for (const c of (data.data || [])) {
+      if (c.attributes?.status === 'Sent') {
+        if (skipped < offset) {
+          skipped++;
+          continue;
+        }
+        campaigns.push(c);
+        if (campaigns.length >= limit) break;
+      }
+    }
+
+    for (const inc of (data.included || [])) {
+      if (inc.type === 'campaign-message') {
+        messageMap[inc.id] = inc;
+      }
+    }
+
+    const nextUrl = data.links?.next;
+    if (campaigns.length >= limit || !nextUrl) break;
+
+    const progress = skipped < offset
+      ? `Skipped ${skipped}/${offset}...`
+      : `Fetched ${campaigns.length}/${limit} sent campaigns, loading next page...`;
+    console.log(`  ${progress}`);
+    await new Promise(r => setTimeout(r, 200));
+    data = await klaviyoFetch(nextUrl);
+  }
+
+  return { campaigns: campaigns.slice(0, limit), messageMap };
+}
+
+async function getTemplateHtml(messageId) {
+  const data = await klaviyoGet(`/campaign-messages/${messageId}/template`, {
+    'fields[template]': 'html,name',
+  });
+  return {
+    templateId: data.data?.id,
+    templateName: data.data?.attributes?.name,
+    html: data.data?.attributes?.html || '',
+  };
+}
+
+// ── HTML parsing ─────────────────────────────────────────────────────
+
+function parseProductsFromHtml(html) {
+  const seen = new Set();
+  const products = [];
+
+  const linkRegex = /href="([^"]*acherryontop\.com\/shop\/(\d+))[^"]*"/gi;
+  let match;
+  while ((match = linkRegex.exec(html)) !== null) {
+    const productId = match[2];
+    if (!seen.has(productId)) {
+      seen.add(productId);
+      products.push({
+        siteProductId: productId,
+        url: match[1],
+      });
+    }
+  }
+
+  const categoryLinks = [];
+  const catRegex = /href="([^"]*acherryontop\.com\/shop\/[^"]+)"/gi;
+  while ((match = catRegex.exec(html)) !== null) {
+    const url = match[1];
+    if (/\/shop\/\d+$/.test(url)) continue;
+    if (!categoryLinks.includes(url)) categoryLinks.push(url);
+  }
+
+  return { products, categoryLinks };
+}
+
+function classifyLink(url) {
+  if (/\/shop\/(new|pre-order|backinstock)/.test(url)) return 'filter';
+  if (/\/shop\/company\//.test(url)) return 'brand';
+  if (/\/shop\/clearance/.test(url)) return 'clearance';
+  if (/\/shop\/daily_deals/.test(url)) return 'deals';
+  if (/\/shop\/category\//.test(url)) return 'category';
+  return 'other';
+}
+
+// ── Database ─────────────────────────────────────────────────────────
+
+function createPool() {
+  return new pg.Pool({
+    host: process.env.DB_HOST,
+    user: process.env.DB_USER,
+    password: process.env.DB_PASSWORD,
+    database: process.env.DB_NAME,
+    port: process.env.DB_PORT || 5432,
+    ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
+  });
+}
+
+async function insertProducts(pool, campaignId, campaignName, sentAt, products) {
+  if (products.length === 0) return 0;
+
+  let inserted = 0;
+  for (const p of products) {
+    try {
+      await pool.query(
+        `INSERT INTO klaviyo_campaign_products
+          (campaign_id, campaign_name, sent_at, pid, product_url)
+         VALUES ($1, $2, $3, $4, $5)
+         ON CONFLICT (campaign_id, pid) DO NOTHING`,
+        [campaignId, campaignName, sentAt, parseInt(p.siteProductId), p.url]
+      );
+      inserted++;
+    } catch (err) {
+      console.error(`    Error inserting product ${p.siteProductId}: ${err.message}`);
+    }
+  }
+  return inserted;
+}
+
+async function insertLinks(pool, campaignId, campaignName, sentAt, links) {
+  if (links.length === 0) return 0;
+
+  let inserted = 0;
+  for (const url of links) {
+    try {
+      await pool.query(
+        `INSERT INTO klaviyo_campaign_links
+          (campaign_id, campaign_name, sent_at, link_url, link_type)
+         VALUES ($1, $2, $3, $4, $5)
+         ON CONFLICT (campaign_id, link_url) DO NOTHING`,
+        [campaignId, campaignName, sentAt, url, classifyLink(url)]
+      );
+      inserted++;
+    } catch (err) {
+      console.error(`    Error inserting link: ${err.message}`);
+    }
+  }
+  return inserted;
+}
+
+// ── Main ─────────────────────────────────────────────────────────────
+
+async function main() {
+  const limit = parseInt(process.argv[2]) || 10;
+  const offset = parseInt(process.argv[3]) || 0;
+  const pool = createPool();
+
+  try {
+    // Fetch campaigns
+    console.log(`Fetching up to ${limit} recent campaigns (offset: ${offset})...\n`);
+    const { campaigns, messageMap } = await getRecentCampaigns(limit, offset);
+    console.log(`Found ${campaigns.length} sent campaigns.\n`);
+
+    let totalProducts = 0;
+    let totalLinks = 0;
+
+    for (const campaign of campaigns) {
+      const name = campaign.attributes?.name || 'Unnamed';
+      const sentAt = campaign.attributes?.send_time;
+
+      console.log(`━━━ ${name} (${sentAt?.slice(0, 10) || 'no date'}) ━━━`);
+
+      const msgIds = (campaign.relationships?.['campaign-messages']?.data || [])
+        .map(r => r.id);
+
+      if (msgIds.length === 0) {
+        console.log('  No messages.\n');
+        continue;
+      }
+
+      for (const msgId of msgIds) {
+        const msg = messageMap[msgId];
+        const subject = msg?.attributes?.definition?.content?.subject;
+        if (subject) console.log(`  Subject: ${subject}`);
+
+        try {
+          const template = await getTemplateHtml(msgId);
+          const { products, categoryLinks } = parseProductsFromHtml(template.html);
+
+          const pInserted = await insertProducts(pool, campaign.id, name, sentAt, products);
+          const lInserted = await insertLinks(pool, campaign.id, name, sentAt, categoryLinks);
+
+          console.log(`  ${products.length} products (${pInserted} new), ${categoryLinks.length} links (${lInserted} new)`);
+          totalProducts += pInserted;
+          totalLinks += lInserted;
+
+          await new Promise(r => setTimeout(r, 200));
+        } catch (err) {
+          console.log(`  Error: ${err.message}`);
+        }
+      }
+      console.log('');
+    }
+
+    console.log(`Done. Inserted ${totalProducts} product rows, ${totalLinks} link rows.`);
+  } finally {
+    await pool.end();
+  }
+}
+
+main().catch(err => {
+  console.error('Fatal error:', err);
+  process.exit(1);
+});

inventory-server/dashboard/server.js

@@ -0,0 +1,132 @@
+// dashboard-server — Phase 4 of CONSOLIDATION_PLAN.md.
+// Merges the four per-vendor PM2 apps (klaviyo, meta, google-analytics, typeform)
+// into a single ESM service on DASHBOARD_PORT (default 3015).
+//
+// Mount points (matches Caddy proxy paths):
+//   /api/klaviyo/*             → routes/klaviyo (was klaviyo-server :3004)
+//   /api/meta/*                → routes/meta    (was meta-server    :3005)
+//   /api/dashboard-analytics/* → routes/google  (was google-server  :3007 via Caddy /api/analytics rewrite)
+//   /api/typeform/*            → routes/typeform (was typeform-server :3008)
+//
+// Shared infrastructure (Phase 2 + Phase 6):
+//   - shared/auth/middleware.js  authenticate() guards /api/* (Phase 6.1/6.2 — second line of defense)
+//   - shared/cors/policy.js      explicit allowed-origins list (Phase 6.6)
+//   - shared/logging/request-log.js  pino-http, Authorization/Cookie redacted (Phase 6.5/6.9)
+//   - shared/errors/handler.js   consistent error envelope, no leak in prod
+//   - shared/db/pg.js / shared/db/redis.js  one Pool + one ioredis client for all vendors
+//
+// Per-route permission gates (Phase 6.2):
+//   - meta_write     PATCH/POST mutations to Meta campaigns
+//   - klaviyo_admin  POST /api/klaviyo/events/clearCache (operational maintenance)
+// Read-only Google + Typeform endpoints stay authenticated-only.
+
+import { config as loadEnv } from 'dotenv';
+import express from 'express';
+import cors from 'cors';
+import path from 'node:path';
+import fs from 'node:fs';
+import { fileURLToPath } from 'node:url';
+
+import { authenticate } from '../shared/auth/middleware.js';
+import { corsOptions } from '../shared/cors/policy.js';
+import { createPool } from '../shared/db/pg.js';
+import { createRedis } from '../shared/db/redis.js';
+import { errorHandler } from '../shared/errors/handler.js';
+import { logger } from '../shared/logging/logger.js';
+import { requestLog } from '../shared/logging/request-log.js';
+
+import { createKlaviyoRouter } from './routes/klaviyo/index.js';
+import { createMetaRouter } from './routes/meta/index.js';
+import { createGoogleRouter } from './routes/google/index.js';
+import { createTypeformRouter } from './routes/typeform/index.js';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+
+// Layer envs: shared inventory .env wins on collisions (security-critical vars come
+// from one place); vendor-specific keys come from the per-service .env.
+//
+// dotenv defaults to override:false so the first file loaded wins. Order matters.
+const sharedEnvPath = '/var/www/inventory/.env';
+const dashboardEnvPath = path.resolve(__dirname, '.env');
+
+if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
+if (fs.existsSync(dashboardEnvPath)) loadEnv({ path: dashboardEnvPath });
+
+// Phase 6.4 — refuse to start without JWT_SECRET. Without it authenticate() falls
+// back to res.status(401) on every request and the service is useless anyway.
+if (!process.env.JWT_SECRET) {
+  logger.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
+  process.exit(1);
+}
+
+const app = express();
+const PORT = Number(process.env.DASHBOARD_PORT) || 3015;
+
+// Trust X-Forwarded-* only when the immediate hop is loopback (Caddy on the same
+// host). Required for the KIOSK_IPS bypass in shared/auth/middleware.js to see
+// real client IPs instead of 127.0.0.1.
+app.set('trust proxy', 'loopback');
+
+// Single Postgres pool — used by authenticate() to load user permissions.
+// All four vendors share this pool (auth lookups are the only DB hits at runtime).
+const pool = createPool('DB');
+
+// Single ioredis client shared across all vendors. lazyConnect:true means the
+// first .get/.set triggers the actual connect — keeps startup non-blocking even
+// if Redis is temporarily unavailable, and aligns with shared/db/redis.js defaults.
+const redis = createRedis();
+
+app.use(requestLog());
+app.use(cors(corsOptions));
+app.use(express.json({ limit: '10mb' }));
+
+// Phase 6.1/6.2: every /api request requires a valid JWT. authenticate() also
+// loads user permissions, which the per-route requirePermission() checks rely on.
+app.use('/api', authenticate({ pool, secret: process.env.JWT_SECRET }));
+
+app.use('/api/klaviyo', createKlaviyoRouter({ redis }));
+app.use('/api/meta', createMetaRouter());
+// Note: frontend calls /api/dashboard-analytics (Caddy used to rewrite it to
+// /api/analytics for the standalone google-server). Mount at the public path so
+// Caddy can drop the rewrite — see Caddyfile.proposed.
+app.use('/api/dashboard-analytics', createGoogleRouter({ redis }));
+app.use('/api/typeform', createTypeformRouter({ redis }));
+
+app.get('/health', (req, res) => {
+  res.json({
+    status: 'ok',
+    timestamp: new Date().toISOString(),
+    service: 'dashboard-server',
+    redis: redis.status,
+  });
+});
+
+app.use(errorHandler);
+
+// Connect Redis up front so the first request doesn't pay the connect cost.
+// Failures here are non-fatal — vendors degrade to cache-miss → upstream fetch.
+redis.connect().catch((err) => {
+  logger.error({ err: { message: err.message, code: err.code } }, 'redis lazy-connect failed');
+});
+
+const server = app.listen(PORT, '0.0.0.0', () => {
+  logger.info({ port: PORT, mode: process.env.NODE_ENV || 'development' }, 'dashboard-server listening');
+});
+
+const shutdown = async (signal) => {
+  logger.info({ signal }, 'dashboard-server shutting down');
+  server.close();
+  try { await redis.quit(); } catch { /* ignore */ }
+  try { await pool.end(); } catch { /* ignore */ }
+  process.exit(0);
+};
+process.on('SIGTERM', () => shutdown('SIGTERM'));
+process.on('SIGINT', () => shutdown('SIGINT'));
+
+process.on('uncaughtException', (err) => {
+  logger.error({ err: { message: err.message, stack: err.stack } }, 'uncaughtException');
+  process.exit(1);
+});
+process.on('unhandledRejection', (reason) => {
+  logger.error({ reason }, 'unhandledRejection');
+});

inventory-server/dashboard/services/google/analytics.service.js

@@ -0,0 +1,195 @@
+// Google Analytics (GA4) service — ESM conversion of google-server/services/analytics.service.js.
+// Phase 4: accepts injected ioredis client (was self-constructing node-redis v4 before).
+// node-redis v4 set syntax `{ EX: 300 }` is translated to ioredis `setex(key, 300, val)`.
+
+import { BetaAnalyticsDataClient } from '@google-analytics/data';
+
+const CACHE_DURATIONS = {
+  REALTIME_BASIC: 60,
+  REALTIME_DETAILED: 300,
+  BASIC_METRICS: 3600,
+  USER_BEHAVIOR: 3600,
+};
+
+export class AnalyticsService {
+  constructor(redis) {
+    if (!redis) {
+      throw new Error('AnalyticsService requires an ioredis client (Phase 4: injected)');
+    }
+    this.redis = redis;
+
+    const credentials = process.env.GOOGLE_APPLICATION_CREDENTIALS_JSON;
+    this.analyticsClient = new BetaAnalyticsDataClient({
+      credentials: typeof credentials === 'string' ? JSON.parse(credentials) : credentials,
+    });
+    this.propertyId = process.env.GA_PROPERTY_ID;
+  }
+
+  get _redisReady() {
+    return this.redis.status === 'ready' || this.redis.status === 'connect';
+  }
+
+  async _cacheGet(key) {
+    if (!this._redisReady) return null;
+    try {
+      const raw = await this.redis.get(key);
+      return raw ? JSON.parse(raw) : null;
+    } catch (err) {
+      console.warn('[AnalyticsService] cache get failed:', err.message);
+      return null;
+    }
+  }
+
+  async _cacheSet(key, value, ttlSec) {
+    if (!this._redisReady) return;
+    try {
+      await this.redis.setex(key, ttlSec, JSON.stringify(value));
+    } catch (err) {
+      console.warn('[AnalyticsService] cache set failed:', err.message);
+    }
+  }
+
+  async getBasicMetrics(startDate = '7daysAgo') {
+    const cacheKey = `analytics:basic_metrics:${startDate}`;
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const [response] = await this.analyticsClient.runReport({
+      property: `properties/${this.propertyId}`,
+      dateRanges: [{ startDate, endDate: 'today' }],
+      dimensions: [{ name: 'date' }],
+      metrics: [
+        { name: 'activeUsers' },
+        { name: 'newUsers' },
+        { name: 'averageSessionDuration' },
+        { name: 'screenPageViews' },
+        { name: 'bounceRate' },
+        { name: 'conversions' },
+      ],
+      returnPropertyQuota: true,
+    });
+
+    await this._cacheSet(cacheKey, response, CACHE_DURATIONS.BASIC_METRICS);
+    return response;
+  }
+
+  async getRealTimeBasicData() {
+    const cacheKey = 'analytics:realtime:basic';
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const [userResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      metrics: [{ name: 'activeUsers' }],
+      returnPropertyQuota: true,
+    });
+
+    const [fiveMinResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      metrics: [{ name: 'activeUsers' }],
+      minuteRanges: [{ startMinutesAgo: 5, endMinutesAgo: 0 }],
+    });
+
+    const [timeSeriesResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      dimensions: [{ name: 'minutesAgo' }],
+      metrics: [{ name: 'activeUsers' }],
+    });
+
+    const response = {
+      userResponse,
+      fiveMinResponse,
+      timeSeriesResponse,
+      quotaInfo: {
+        projectHourly: userResponse.propertyQuota.tokensPerProjectPerHour,
+        daily: userResponse.propertyQuota.tokensPerDay,
+        serverErrors: userResponse.propertyQuota.serverErrorsPerProjectPerHour,
+        thresholdedRequests: userResponse.propertyQuota.potentiallyThresholdedRequestsPerHour,
+      },
+    };
+
+    await this._cacheSet(cacheKey, response, CACHE_DURATIONS.REALTIME_BASIC);
+    return response;
+  }
+
+  async getRealTimeDetailedData() {
+    const cacheKey = 'analytics:realtime:detailed';
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const [pageResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      dimensions: [{ name: 'unifiedScreenName' }],
+      metrics: [{ name: 'screenPageViews' }],
+      orderBy: [{ metric: { metricName: 'screenPageViews' }, desc: true }],
+      limit: 25,
+    });
+
+    const [eventResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      dimensions: [{ name: 'eventName' }],
+      metrics: [{ name: 'eventCount' }],
+      orderBy: [{ metric: { metricName: 'eventCount' }, desc: true }],
+      limit: 25,
+    });
+
+    const [deviceResponse] = await this.analyticsClient.runRealtimeReport({
+      property: `properties/${this.propertyId}`,
+      dimensions: [{ name: 'deviceCategory' }],
+      metrics: [{ name: 'activeUsers' }],
+      orderBy: [{ metric: { metricName: 'activeUsers' }, desc: true }],
+      limit: 10,
+      returnPropertyQuota: true,
+    });
+
+    const response = {
+      pageResponse,
+      eventResponse,
+      sourceResponse: deviceResponse,
+    };
+
+    await this._cacheSet(cacheKey, response, CACHE_DURATIONS.REALTIME_DETAILED);
+    return response;
+  }
+
+  async getUserBehavior(timeRange = '30') {
+    const cacheKey = `analytics:user_behavior:${timeRange}`;
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const [pageResponse] = await this.analyticsClient.runReport({
+      property: `properties/${this.propertyId}`,
+      dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
+      dimensions: [{ name: 'pagePath' }],
+      metrics: [
+        { name: 'screenPageViews' },
+        { name: 'averageSessionDuration' },
+        { name: 'bounceRate' },
+        { name: 'sessions' },
+      ],
+      orderBy: [{ metric: { metricName: 'screenPageViews' }, desc: true }],
+      limit: 25,
+    });
+
+    const [deviceResponse] = await this.analyticsClient.runReport({
+      property: `properties/${this.propertyId}`,
+      dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
+      dimensions: [{ name: 'deviceCategory' }],
+      metrics: [{ name: 'screenPageViews' }, { name: 'sessions' }],
+    });
+
+    const [sourceResponse] = await this.analyticsClient.runReport({
+      property: `properties/${this.propertyId}`,
+      dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
+      dimensions: [{ name: 'sessionSource' }],
+      metrics: [{ name: 'sessions' }, { name: 'conversions' }],
+      orderBy: [{ metric: { metricName: 'sessions' }, desc: true }],
+      limit: 25,
+      returnPropertyQuota: true,
+    });
+
+    const response = { pageResponse, deviceResponse, sourceResponse };
+    await this._cacheSet(cacheKey, response, CACHE_DURATIONS.USER_BEHAVIOR);
+    return response;
+  }
+}

inventory-server/dashboard/services/klaviyo/campaigns.service.js

@@ -0,0 +1,206 @@
+import fetch from 'node-fetch';
+import { TimeManager } from '../../utils/time.utils.js';
+import { RedisService } from './redis.service.js';
+
+export class CampaignsService {
+  constructor(apiKey, apiRevision, redis) {
+    this.apiKey = apiKey;
+    this.apiRevision = apiRevision;
+    this.baseUrl = 'https://a.klaviyo.com/api';
+    this.timeManager = new TimeManager();
+    this.redisService = new RedisService(redis);
+  }
+
+  async getCampaigns(params = {}) {
+    try {
+      // Add request debouncing
+      const requestKey = JSON.stringify(params);
+      if (this._pendingRequests && this._pendingRequests[requestKey]) {
+        return this._pendingRequests[requestKey];
+      }
+
+      // Try to get from cache first
+      const cacheKey = this.redisService._getCacheKey('campaigns', params);
+      let cachedData = null;
+      try {
+        cachedData = await this.redisService.get(`${cacheKey}:raw`);
+        if (cachedData) {
+          return cachedData;
+        }
+      } catch (cacheError) {
+        console.warn('[CampaignsService] Cache error:', cacheError);
+      }
+
+      this._pendingRequests = this._pendingRequests || {};
+      this._pendingRequests[requestKey] = (async () => {
+        let allCampaigns = [];
+        let nextCursor = params.pageCursor;
+        let pageCount = 0;
+        
+        const filter = params.filter || this._buildFilter(params);
+        
+        do {
+          const queryParams = new URLSearchParams();
+          if (filter) {
+            queryParams.append('filter', filter);
+          }
+          queryParams.append('sort', params.sort || '-send_time');
+
+          if (nextCursor) {
+            queryParams.append('page[cursor]', nextCursor);
+          }
+
+          const url = `${this.baseUrl}/campaigns?${queryParams.toString()}`;
+
+          try {
+            const response = await fetch(url, {
+              method: 'GET',
+              headers: {
+                'Accept': 'application/json',
+                'Content-Type': 'application/json',
+                'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
+                'revision': this.apiRevision
+              }
+            });
+
+            if (!response.ok) {
+              const errorData = await response.json();
+              console.error('[CampaignsService] API Error:', errorData);
+              throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
+            }
+
+            const responseData = await response.json();
+            allCampaigns = allCampaigns.concat(responseData.data || []);
+            pageCount++;
+            
+            nextCursor = responseData.links?.next ? 
+              new URL(responseData.links.next).searchParams.get('page[cursor]') : null;
+
+            if (nextCursor) {
+              await new Promise(resolve => setTimeout(resolve, 50));
+            }
+          } catch (fetchError) {
+            console.error('[CampaignsService] Fetch error:', fetchError);
+            throw fetchError;
+          }
+          
+        } while (nextCursor);
+
+        const transformedCampaigns = this._transformCampaigns(allCampaigns);
+
+        const result = {
+          data: transformedCampaigns,
+          meta: {
+            total_count: transformedCampaigns.length,
+            page_count: pageCount
+          }
+        };
+
+        try {
+          const ttl = this.redisService._getTTL(params.timeRange);
+          await this.redisService.set(`${cacheKey}:raw`, result, ttl);
+        } catch (cacheError) {
+          console.warn('[CampaignsService] Cache set error:', cacheError);
+        }
+
+        delete this._pendingRequests[requestKey];
+        return result;
+      })();
+
+      return await this._pendingRequests[requestKey];
+    } catch (error) {
+      console.error('[CampaignsService] Error fetching campaigns:', error);
+      throw error;
+    }
+  }
+
+  _buildFilter(params) {
+    const filters = [];
+    
+    if (params.startDate && params.endDate) {
+      const startUtc = this.timeManager.formatForAPI(params.startDate);
+      const endUtc = this.timeManager.formatForAPI(params.endDate);
+      
+      filters.push(`greater-or-equal(send_time,${startUtc})`);
+      filters.push(`less-than(send_time,${endUtc})`);
+    }
+
+    if (params.status) {
+      filters.push(`equals(status,"${params.status}")`);
+    }
+
+    if (params.customFilters) {
+      filters.push(...params.customFilters);
+    }
+
+    return filters.length > 0 ? (filters.length > 1 ? `and(${filters.join(',')})` : filters[0]) : null;
+  }
+
+  async getCampaignsByTimeRange(timeRange, options = {}) {
+    const range = this.timeManager.getDateRange(timeRange);
+    if (!range) {
+      throw new Error('Invalid time range specified');
+    }
+
+    const params = {
+      timeRange,
+      startDate: range.start.toISO(),
+      endDate: range.end.toISO(),
+      ...options
+    };
+
+    // Try to get from cache first
+    const cacheKey = this.redisService._getCacheKey('campaigns', params);
+    let cachedData = null;
+    try {
+      cachedData = await this.redisService.get(`${cacheKey}:raw`);
+      if (cachedData) {
+        return cachedData;
+      }
+    } catch (cacheError) {
+      console.warn('[CampaignsService] Cache error:', cacheError);
+    }
+
+    return this.getCampaigns(params);
+  }
+
+  _transformCampaigns(campaigns) {
+    if (!Array.isArray(campaigns)) {
+      console.warn('[CampaignsService] Campaigns is not an array:', campaigns);
+      return [];
+    }
+
+    return campaigns.map(campaign => {
+      try {
+        const stats = campaign.attributes?.campaign_message?.stats || {};
+        
+        return {
+          id: campaign.id,
+          name: campaign.attributes?.name || "Unnamed Campaign",
+          subject: campaign.attributes?.campaign_message?.subject || "",
+          send_time: campaign.attributes?.send_time,
+          stats: {
+            delivery_rate: stats.delivery_rate || 0,
+            delivered: stats.delivered || 0,
+            recipients: stats.recipients || 0,
+            open_rate: stats.open_rate || 0,
+            opens_unique: stats.opens_unique || 0,
+            opens: stats.opens || 0,
+            clicks_unique: stats.clicks_unique || 0,
+            click_rate: stats.click_rate || 0,
+            click_to_open_rate: stats.click_to_open_rate || 0,
+            conversion_value: stats.conversion_value || 0,
+            conversion_uniques: stats.conversion_uniques || 0
+          }
+        };
+      } catch (error) {
+        console.error('[CampaignsService] Error transforming campaign:', error, campaign);
+        return {
+          id: campaign.id || 'unknown',
+          name: 'Error Processing Campaign',
+          stats: {}
+        };
+      }
+    });
+  }
+} 

inventory-server/dashboard/services/klaviyo/metrics.service.js

@@ -0,0 +1,38 @@
+import fetch from 'node-fetch';
+
+export class MetricsService {
+  constructor(apiKey, apiRevision) {
+    this.apiKey = apiKey;
+    this.apiRevision = apiRevision;
+    this.baseUrl = 'https://a.klaviyo.com/api';
+  }
+  async getMetrics() {
+    try {
+      const response = await fetch(`${this.baseUrl}/metrics/`, {
+        headers: {
+          'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
+          'revision': this.apiRevision,
+          'Content-Type': 'application/json',
+          'Accept': 'application/json'
+        }
+      });
+
+      if (!response.ok) {
+        const errorData = await response.json();
+        console.error('[MetricsService] API Error:', errorData);
+        throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
+      }
+
+      const data = await response.json();
+      // Sort the results by name before returning
+      if (data.data) {
+        data.data.sort((a, b) => a.attributes.name.localeCompare(b.attributes.name));
+      }
+
+      return data;
+    } catch (error) {
+      console.error('[MetricsService] Error fetching metrics:', error);
+      throw error;
+    }
+  }
+}

inventory-server/dashboard/services/klaviyo/redis.service.js

@@ -0,0 +1,146 @@
+// Klaviyo cache wrapper. Was a self-instantiating ioredis client per service in
+// the standalone klaviyo-server; now accepts an injected client so the merged
+// dashboard-server shares one connection across all vendors (Phase 4).
+//
+// Public surface kept identical to the original so the ~3K LOC of klaviyo
+// service code (events/campaigns/reporting) needs no other changes:
+//   - get(key)
+//   - set(key, data, ttl)
+//   - _getCacheKey(type, params)
+//   - _getTTL(timeRange)
+//   - getEventData(type, params) / cacheEventData(type, params, data)
+//   - clearCache(params)
+//
+// Reads short-circuit to null when the client isn't ready; writes are no-ops.
+// Same "Redis hiccup → fall through to upstream" behavior as before.
+
+import { TimeManager } from '../../utils/time.utils.js';
+
+export class RedisService {
+  constructor(redis) {
+    if (!redis) {
+      throw new Error('RedisService requires an ioredis client (Phase 4: injected, no longer self-constructed)');
+    }
+    this.client = redis;
+    this.timeManager = new TimeManager();
+    this.DEFAULT_TTL = 5 * 60;
+  }
+
+  get isConnected() {
+    // ioredis: 'wait' | 'reconnecting' | 'connecting' | 'connect' | 'ready' | 'close' | 'end'
+    return this.client.status === 'ready' || this.client.status === 'connect';
+  }
+
+  async get(key) {
+    if (!this.isConnected) return null;
+    try {
+      const data = await this.client.get(key);
+      return data ? JSON.parse(data) : null;
+    } catch (error) {
+      console.error('[RedisService] Error getting data:', error);
+      return null;
+    }
+  }
+
+  async set(key, data, ttl = this.DEFAULT_TTL) {
+    if (!this.isConnected) return;
+    try {
+      await this.client.setex(key, ttl, JSON.stringify(data));
+    } catch (error) {
+      console.error('[RedisService] Error setting data:', error);
+    }
+  }
+
+  _getCacheKey(type, params = {}) {
+    const {
+      timeRange,
+      startDate,
+      endDate,
+      metricId,
+      metric,
+      daily,
+      cacheKey,
+      isPreviousPeriod,
+      customFilters,
+    } = params;
+
+    let key = `klaviyo:${type}`;
+
+    if (type === 'stats:details') {
+      key += `:${metric || 'all'}`;
+      if (daily) key += ':daily';
+      if (customFilters?.length) {
+        const filterHash = customFilters.join('').replace(/[^a-zA-Z0-9]/g, '');
+        key += `:${filterHash}`;
+      }
+    }
+
+    if (cacheKey) {
+      key += `:${cacheKey}`;
+    } else if (timeRange) {
+      key += `:${timeRange}`;
+      if (metricId) key += `:${metricId}`;
+      if (isPreviousPeriod) key += ':prev';
+    } else if (startDate && endDate) {
+      key += `:custom:${startDate}:${endDate}`;
+      if (metricId) key += `:${metricId}`;
+      if (isPreviousPeriod) key += ':prev';
+    }
+
+    if (['pre_orders', 'local_pickup', 'on_hold'].includes(metric)) {
+      key += `:${metric}`;
+    }
+
+    return key;
+  }
+
+  _getTTL(timeRange) {
+    const TTL_MAP = {
+      today: 2 * 60,
+      yesterday: 30 * 60,
+      thisWeek: 5 * 60,
+      lastWeek: 60 * 60,
+      thisMonth: 10 * 60,
+      lastMonth: 2 * 60 * 60,
+      last7days: 5 * 60,
+      last30days: 15 * 60,
+      custom: 15 * 60,
+    };
+    return TTL_MAP[timeRange] || this.DEFAULT_TTL;
+  }
+
+  async getEventData(type, params) {
+    if (!this.isConnected) return null;
+    try {
+      const baseKey = this._getCacheKey('events', params);
+      return await this.get(`${baseKey}:${type}`);
+    } catch (error) {
+      console.error('[RedisService] Error getting event data:', error);
+      return null;
+    }
+  }
+
+  async cacheEventData(type, params, data) {
+    if (!this.isConnected) return;
+    try {
+      const ttl = this._getTTL(params.timeRange);
+      const baseKey = this._getCacheKey('events', params);
+      await this.set(`${baseKey}:${type}`, data, ttl);
+    } catch (error) {
+      console.error('[RedisService] Error caching event data:', error);
+    }
+  }
+
+  async clearCache(params = {}) {
+    if (!this.isConnected) return;
+    try {
+      const pattern = this._getCacheKey('events', params) + '*';
+      const keys = await this.client.keys(pattern);
+      if (keys.length > 0) {
+        await this.client.del(...keys);
+      }
+    } catch (error) {
+      console.error('[RedisService] Error clearing cache:', error);
+    }
+  }
+}

inventory-server/dashboard/services/klaviyo/reporting.service.js

@@ -0,0 +1,254 @@
+import fetch from 'node-fetch';
+import { TimeManager } from '../../utils/time.utils.js';
+import { RedisService } from './redis.service.js';
+
+const METRIC_IDS = {
+  PLACED_ORDER: 'Y8cqcF'
+};
+
+export class ReportingService {
+  constructor(apiKey, apiRevision, redis) {
+    this.apiKey = apiKey;
+    this.apiRevision = apiRevision;
+    this.baseUrl = 'https://a.klaviyo.com/api';
+    this.timeManager = new TimeManager();
+    this.redisService = new RedisService(redis);
+    this._pendingReportRequest = null;
+  }
+
+  async getCampaignReports(params = {}) {
+    try {
+      // Check if there's a pending request
+      if (this._pendingReportRequest) {
+        console.log('[ReportingService] Using pending campaign report request');
+        return this._pendingReportRequest;
+      }
+
+      // Try to get from cache first
+      const cacheKey = this.redisService._getCacheKey('campaign_reports', params);
+      let cachedData = null;
+      try {
+        cachedData = await this.redisService.get(`${cacheKey}:raw`);
+        if (cachedData) {
+          console.log('[ReportingService] Using cached campaign report data');
+          return cachedData;
+        }
+      } catch (cacheError) {
+        console.warn('[ReportingService] Cache error:', cacheError);
+      }
+
+      // Create new request promise
+      this._pendingReportRequest = (async () => {
+        console.log('[ReportingService] Fetching fresh campaign report data');
+
+        const range = this.timeManager.getDateRange(params.timeRange || 'last30days');
+        
+        // Determine which channels to fetch based on params
+        const channelsToFetch = params.channel === 'all' || !params.channel
+          ? ['email', 'sms']
+          : [params.channel];
+
+        const allResults = [];
+        
+        // Fetch each channel
+        for (const channel of channelsToFetch) {
+          const payload = {
+            data: {
+              type: "campaign-values-report",
+              attributes: {
+                timeframe: {
+                  start: range.start.toISO(),
+                  end: range.end.toISO()
+                },
+                statistics: [
+                  "delivery_rate",
+                  "delivered",
+                  "recipients",
+                  "open_rate",
+                  "opens_unique",
+                  "opens",
+                  "click_rate",
+                  "clicks_unique",
+                  "click_to_open_rate",
+                  "conversion_value",
+                  "conversion_uniques"
+                ],
+                conversion_metric_id: METRIC_IDS.PLACED_ORDER,
+                filter: `equals(send_channel,"${channel}")`
+              }
+            }
+          };
+
+          const response = await fetch(`${this.baseUrl}/campaign-values-reports`, {
+            method: 'POST',
+            headers: {
+              'Accept': 'application/json',
+              'Content-Type': 'application/json',
+              'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
+              'revision': this.apiRevision
+            },
+            body: JSON.stringify(payload)
+          });
+
+          if (!response.ok) {
+            const errorData = await response.json();
+            console.error('[ReportingService] API Error:', errorData);
+            throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
+          }
+
+          const reportData = await response.json();
+          console.log(`[ReportingService] Raw ${channel} report data:`, JSON.stringify(reportData, null, 2));
+          
+          // Get campaign IDs from the report
+          const campaignIds = reportData.data?.attributes?.results?.map(result => 
+            result.groupings?.campaign_id
+          ).filter(Boolean) || [];
+          
+          if (campaignIds.length > 0) {
+            // Get campaign details including send time and subject lines
+            const campaignDetails = await this.getCampaignDetails(campaignIds);
+            
+            // Process results for this channel
+            const channelResults = reportData.data.attributes.results.map(result => {
+              const campaignId = result.groupings.campaign_id;
+              const details = campaignDetails.find(detail => detail.id === campaignId);
+
+              return {
+                id: campaignId,
+                name: details.attributes.name,
+                subject: details.attributes.subject,
+                send_time: details.attributes.send_time,
+                channel: channel, // Use the channel we're currently processing
+                stats: {
+                  delivery_rate: result.statistics.delivery_rate,
+                  delivered: result.statistics.delivered,
+                  recipients: result.statistics.recipients,
+                  open_rate: result.statistics.open_rate,
+                  opens_unique: result.statistics.opens_unique,
+                  opens: result.statistics.opens,
+                  click_rate: result.statistics.click_rate,
+                  clicks_unique: result.statistics.clicks_unique,
+                  click_to_open_rate: result.statistics.click_to_open_rate,
+                  conversion_value: result.statistics.conversion_value,
+                  conversion_uniques: result.statistics.conversion_uniques
+                }
+              };
+            });
+
+            allResults.push(...channelResults);
+          }
+        }
+
+        // Sort all results by date
+        const enrichedData = {
+          data: allResults.sort((a, b) => {
+            const dateA = new Date(a.send_time);
+            const dateB = new Date(b.send_time);
+            return dateB - dateA; // Sort by date descending
+          })
+        };
+
+        console.log('[ReportingService] Enriched data:', JSON.stringify(enrichedData, null, 2));
+
+        // Cache the enriched response for 10 minutes
+        try {
+          await this.redisService.set(`${cacheKey}:raw`, enrichedData, 600);
+        } catch (cacheError) {
+          console.warn('[ReportingService] Cache set error:', cacheError);
+        }
+
+        return enrichedData;
+      })();
+
+      const result = await this._pendingReportRequest;
+      this._pendingReportRequest = null;
+      return result;
+
+    } catch (error) {
+      console.error('[ReportingService] Error fetching campaign reports:', error);
+      this._pendingReportRequest = null;
+      throw error;
+    }
+  }
+
+  async getCampaignDetails(campaignIds = []) {
+    if (!Array.isArray(campaignIds) || campaignIds.length === 0) {
+      return [];
+    }
+
+    const fetchWithTimeout = async (campaignId, retries = 3) => {
+      for (let i = 0; i < retries; i++) {
+        try {
+          const controller = new AbortController();
+          const timeoutId = setTimeout(() => controller.abort(), 10000); // 10 second timeout
+
+          const response = await fetch(
+            `${this.baseUrl}/campaigns/${campaignId}?include=campaign-messages`,
+            {
+              headers: {
+                'Accept': 'application/json',
+                'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
+                'revision': this.apiRevision
+              },
+              signal: controller.signal
+            }
+          );
+
+          clearTimeout(timeoutId);
+
+          if (!response.ok) {
+            throw new Error(`Failed to fetch campaign ${campaignId}: ${response.status}`);
+          }
+
+          const data = await response.json();
+          if (!data.data) {
+            throw new Error(`Invalid response for campaign ${campaignId}`);
+          }
+
+          const message = data.included?.find(item => item.type === 'campaign-message');
+          
+          console.log('[ReportingService] Campaign details for ID:', campaignId, {
+            send_channel: data.data.attributes.send_channel,
+            raw_attributes: data.data.attributes
+          });
+
+          return {
+            id: data.data.id,
+            type: data.data.type,
+            attributes: {
+              ...data.data.attributes,
+              name: data.data.attributes.name,
+              send_time: data.data.attributes.send_time,
+              subject: message?.attributes?.content?.subject,
+              send_channel: data.data.attributes.send_channel || 'email'
+            }
+          };
+        } catch (error) {
+          if (i === retries - 1) throw error;
+          await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1))); // Exponential backoff
+        }
+      }
+    };
+
+    // Process in smaller chunks to avoid overwhelming the API
+    const chunkSize = 10;
+    const campaignDetails = [];
+    
+    for (let i = 0; i < campaignIds.length; i += chunkSize) {
+      const chunk = campaignIds.slice(i, i + chunkSize);
+      const results = await Promise.all(
+        chunk.map(id => fetchWithTimeout(id).catch(error => {
+          console.error(`Failed to fetch campaign ${id}:`, error);
+          return null;
+        }))
+      );
+      campaignDetails.push(...results.filter(Boolean));
+      
+      if (i + chunkSize < campaignIds.length) {
+        await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between chunks
+      }
+    }
+
+    return campaignDetails;
+  }
+} 

inventory-server/dashboard/services/meta/meta.service.js

@@ -0,0 +1,104 @@
+// Meta (Facebook Ads) service — ESM conversion of meta-server/services/meta.service.js.
+// No Redis caching (matches the original — Meta calls are cheap-enough; reach/spend
+// rolls over once per request). Uses axios.
+
+import axios from 'axios';
+
+function getConfig() {
+  const version = process.env.META_API_VERSION || 'v21.0';
+  return {
+    baseUrl: `https://graph.facebook.com/${version}`,
+    accessToken: process.env.META_ACCESS_TOKEN,
+    adAccountId: process.env.META_AD_ACCOUNT_ID,
+  };
+}
+
+async function metaApiRequest(endpoint, params = {}) {
+  const { baseUrl, accessToken } = getConfig();
+  try {
+    const response = await axios.get(`${baseUrl}/${endpoint}`, {
+      params: {
+        access_token: accessToken,
+        time_zone: 'America/New_York',
+        ...params,
+      },
+    });
+    return response.data;
+  } catch (error) {
+    console.error('Meta API Error:', {
+      message: error.message,
+      response: error.response?.data,
+      endpoint,
+    });
+    throw error;
+  }
+}
+
+export async function fetchCampaigns(since, until) {
+  const { adAccountId } = getConfig();
+  const campaigns = await metaApiRequest(`act_${adAccountId}/campaigns`, {
+    fields: [
+      'id',
+      'name',
+      'status',
+      'objective',
+      'daily_budget',
+      'lifetime_budget',
+      'adsets{daily_budget,lifetime_budget}',
+      `insights.time_range({'since':'${since}','until':'${until}'}).level(campaign){
+        spend,
+        impressions,
+        clicks,
+        ctr,
+        reach,
+        frequency,
+        cpm,
+        cpc,
+        actions,
+        action_values,
+        cost_per_action_type
+      }`,
+    ].join(','),
+    limit: 100,
+  });
+
+  return campaigns.data.filter((c) => c.insights?.data?.[0]?.spend > 0);
+}
+
+export async function fetchAccountInsights(since, until) {
+  const { adAccountId } = getConfig();
+  const accountInsights = await metaApiRequest(`act_${adAccountId}/insights`, {
+    fields: 'reach,spend,impressions,clicks,ctr,cpm,actions,action_values',
+    time_range: JSON.stringify({ since, until }),
+  });
+  return accountInsights.data[0] || null;
+}
+
+export async function updateCampaignBudget(campaignId, budget) {
+  const { baseUrl, accessToken } = getConfig();
+  try {
+    const response = await axios.post(`${baseUrl}/${campaignId}`, {
+      access_token: accessToken,
+      daily_budget: budget * 100, // dollars → cents
+    });
+    return response.data;
+  } catch (error) {
+    console.error('Update campaign budget error:', error);
+    throw error;
+  }
+}
+
+export async function updateCampaignStatus(campaignId, action) {
+  const { baseUrl, accessToken } = getConfig();
+  try {
+    const status = action === 'pause' ? 'PAUSED' : 'ACTIVE';
+    const response = await axios.post(`${baseUrl}/${campaignId}`, {
+      access_token: accessToken,
+      status,
+    });
+    return response.data;
+  } catch (error) {
+    console.error('Update campaign status error:', error);
+    throw error;
+  }
+}

inventory-server/dashboard/services/typeform/typeform.service.js

@@ -0,0 +1,80 @@
+// Typeform service — ESM conversion of typeform-server/services/typeform.service.js.
+// Phase 4: accepts injected ioredis client. node-redis v4 set syntax `{ EX: 300 }`
+// translated to ioredis `setex(key, 300, val)`.
+
+import axios from 'axios';
+
+export class TypeformService {
+  constructor(redis) {
+    if (!redis) {
+      throw new Error('TypeformService requires an ioredis client (Phase 4: injected)');
+    }
+    this.redis = redis;
+
+    const token = process.env.TYPEFORM_ACCESS_TOKEN;
+    if (!token) {
+      console.warn('[Typeform] TYPEFORM_ACCESS_TOKEN not set — all calls will 401');
+    }
+
+    this.apiClient = axios.create({
+      baseURL: 'https://api.typeform.com',
+      headers: {
+        Authorization: `Bearer ${token}`,
+        'Content-Type': 'application/json',
+      },
+    });
+  }
+
+  get _redisReady() {
+    return this.redis.status === 'ready' || this.redis.status === 'connect';
+  }
+
+  async _cacheGet(key) {
+    if (!this._redisReady) return null;
+    try {
+      const raw = await this.redis.get(key);
+      return raw ? JSON.parse(raw) : null;
+    } catch (err) {
+      console.warn('[Typeform] cache get failed:', err.message);
+      return null;
+    }
+  }
+
+  async _cacheSet(key, value, ttlSec) {
+    if (!this._redisReady) return;
+    try {
+      await this.redis.setex(key, ttlSec, JSON.stringify(value));
+    } catch (err) {
+      console.warn('[Typeform] cache set failed:', err.message);
+    }
+  }
+
+  async getFormResponses(formId, params = {}) {
+    const cacheKey = `typeform:responses:${formId}:${JSON.stringify(params)}`;
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const response = await this.apiClient.get(`/forms/${formId}/responses`, { params });
+    const data = response.data;
+    await this._cacheSet(cacheKey, data, 300);
+    return data;
+  }
+
+  async getFormInsights(formId) {
+    const cacheKey = `typeform:insights:${formId}`;
+    const cached = await this._cacheGet(cacheKey);
+    if (cached) return cached;
+
+    const response = await this.apiClient.get(`/insights/${formId}/summary`);
+    const data = response.data;
+    await this._cacheSet(cacheKey, data, 300);
+    return data;
+  }
+
+  async getFormResponsesWithFilters(formId, { since, until, pageSize = 25, ...otherParams } = {}) {
+    const params = { page_size: pageSize, ...otherParams };
+    if (since) params.since = new Date(since).toISOString();
+    if (until) params.until = new Date(until).toISOString();
+    return this.getFormResponses(formId, params);
+  }
+}

inventory-server/dashboard/utils/time.utils.js

@@ -0,0 +1,448 @@
+import { DateTime } from 'luxon';
+
+export class TimeManager {
+  constructor(dayStartHour = 1) {
+    this.timezone = 'America/New_York';
+    this.dayStartHour = dayStartHour; // Hour (0-23) when the business day starts
+    this.weekStartDay = 7; // 7 = Sunday in Luxon
+  }
+
+  /**
+   * Get the start of the current business day
+   * If current time is before dayStartHour, return previous day at dayStartHour
+   */
+  getDayStart(dt = this.getNow()) {
+    if (!dt.isValid) {
+      console.error("[TimeManager] Invalid datetime provided to getDayStart");
+      return this.getNow();
+    }
+    const dayStart = dt.set({ hour: this.dayStartHour, minute: 0, second: 0, millisecond: 0 });
+    return dt.hour < this.dayStartHour ? dayStart.minus({ days: 1 }) : dayStart;
+  }
+
+  /**
+   * Get the end of the current business day
+   * End is defined as dayStartHour - 1 minute on the next day
+   */
+  getDayEnd(dt = this.getNow()) {
+    if (!dt.isValid) {
+      console.error("[TimeManager] Invalid datetime provided to getDayEnd");
+      return this.getNow();
+    }
+    const nextDay = this.getDayStart(dt).plus({ days: 1 });
+    return nextDay.minus({ minutes: 1 });
+  }
+
+  /**
+   * Get the start of the week containing the given date
+   * Aligns with custom day start time and starts on Sunday
+   */
+  getWeekStart(dt = this.getNow()) {
+    if (!dt.isValid) {
+      console.error("[TimeManager] Invalid datetime provided to getWeekStart");
+      return this.getNow();
+    }
+    // Set to start of week (Sunday) and adjust hour
+    const weekStart = dt.set({ weekday: this.weekStartDay }).startOf('day');
+    // If the week start time would be after the given time, go back a week
+    if (weekStart > dt) {
+      return weekStart.minus({ weeks: 1 }).set({ hour: this.dayStartHour });
+    }
+    return weekStart.set({ hour: this.dayStartHour });
+  }
+
+  /**
+   * Convert any date input to a Luxon DateTime in Eastern time
+   */
+  toDateTime(date) {
+    if (!date) return null;
+
+    if (date instanceof DateTime) {
+      return date.setZone(this.timezone);
+    }
+
+    // If it's an ISO string or Date object, parse it
+    const dt = DateTime.fromISO(date instanceof Date ? date.toISOString() : date);
+    if (!dt.isValid) {
+      console.error("[TimeManager] Invalid date input:", date);
+      return null;
+    }
+
+    return dt.setZone(this.timezone);
+  }
+
+  /**
+   * Format a date for API requests (UTC ISO string)
+   */
+  formatForAPI(date) {
+    if (!date) return null;
+
+    // Parse the input date
+    const dt = this.toDateTime(date);
+    if (!dt || !dt.isValid) {
+      console.error("[TimeManager] Invalid date for API:", date);
+      return null;
+    }
+
+    // Convert to UTC for API request
+    const utc = dt.toUTC();
+
+    console.log("[TimeManager] API date conversion:", {
+      input: date,
+      eastern: dt.toISO(),
+      utc: utc.toISO(),
+      offset: dt.offset
+    });
+
+    return utc.toISO();
+  }
+
+  /**
+   * Format a date for display (in Eastern time)
+   */
+  formatForDisplay(date) {
+    const dt = this.toDateTime(date);
+    if (!dt || !dt.isValid) return '';
+    return dt.toFormat('LLL d, yyyy h:mm a');
+  }
+
+  /**
+   * Validate if a date range is valid
+   */
+  isValidDateRange(start, end) {
+    const startDt = this.toDateTime(start);
+    const endDt = this.toDateTime(end);
+    return startDt && endDt && endDt > startDt;
+  }
+
+  /**
+   * Get the current time in Eastern timezone
+   */
+  getNow() {
+    return DateTime.now().setZone(this.timezone);
+  }
+
+  /**
+   * Get a date range for the last N hours
+   */
+  getLastNHours(hours) {
+    const now = this.getNow();
+    return {
+      start: now.minus({ hours }),
+      end: now
+    };
+  }
+
+  /**
+   * Get a date range for the last N days
+   * Aligns with custom day start time
+   */
+  getLastNDays(days) {
+    const now = this.getNow();
+    const dayStart = this.getDayStart(now);
+    return {
+      start: dayStart.minus({ days }),
+      end: this.getDayEnd(now)
+    };
+  }
+
+  /**
+   * Get a date range for a specific time period
+   * All ranges align with custom day start time
+   */
+  getDateRange(period) {
+    const now = this.getNow();
+    
+    // Normalize period to handle both 'last' and 'previous' prefixes
+    const normalizedPeriod = period.startsWith('previous') ? period.replace('previous', 'last') : period;
+    
+    switch (normalizedPeriod) {
+      case 'custom': {
+        // Custom ranges are handled separately via getCustomRange
+        console.warn('[TimeManager] Custom ranges should use getCustomRange method');
+        return null;
+      }
+      case 'today': {
+        const dayStart = this.getDayStart(now);
+        return {
+          start: dayStart,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'yesterday': {
+        const yesterday = now.minus({ days: 1 });
+        return {
+          start: this.getDayStart(yesterday),
+          end: this.getDayEnd(yesterday)
+        };
+      }
+      case 'last7days': {
+        // For last 7 days, we want to include today and the previous 6 days
+        const dayStart = this.getDayStart(now);
+        const weekStart = dayStart.minus({ days: 6 });
+        return {
+          start: weekStart,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'last30days': {
+        // Include today and previous 29 days
+        const dayStart = this.getDayStart(now);
+        const monthStart = dayStart.minus({ days: 29 });
+        return {
+          start: monthStart,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'last90days': {
+        // Include today and previous 89 days
+        const dayStart = this.getDayStart(now);
+        const start = dayStart.minus({ days: 89 });
+        return {
+          start,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'thisWeek': {
+        // Get the start of the week (Sunday) with custom hour
+        const weekStart = this.getWeekStart(now);
+        return {
+          start: weekStart,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'lastWeek': {
+        const lastWeek = now.minus({ weeks: 1 });
+        const weekStart = this.getWeekStart(lastWeek);
+        const weekEnd = weekStart.plus({ days: 6 }); // 6 days after start = Saturday
+        return {
+          start: weekStart,
+          end: this.getDayEnd(weekEnd)
+        };
+      }
+      case 'thisMonth': {
+        const dayStart = this.getDayStart(now);
+        const monthStart = dayStart.startOf('month').set({ hour: this.dayStartHour });
+        return {
+          start: monthStart,
+          end: this.getDayEnd(now)
+        };
+      }
+      case 'lastMonth': {
+        const lastMonth = now.minus({ months: 1 });
+        const monthStart = lastMonth.startOf('month').set({ hour: this.dayStartHour });
+        const monthEnd = monthStart.plus({ months: 1 }).minus({ days: 1 });
+        return {
+          start: monthStart,
+          end: this.getDayEnd(monthEnd)
+        };
+      }
+      default:
+        console.warn(`[TimeManager] Unknown period: ${period}`);
+        return null;
+    }
+  }
+
+  /**
+   * Format a duration in milliseconds to a human-readable string
+   */
+  formatDuration(ms) {
+    return DateTime.fromMillis(ms).toFormat("hh'h' mm'm' ss's'");
+  }
+
+  /**
+   * Get relative time string (e.g., "2 hours ago")
+   */
+  getRelativeTime(date) {
+    const dt = this.toDateTime(date);
+    if (!dt) return '';
+    return dt.toRelative();
+  }
+
+  /**
+   * Get a custom date range using exact dates and times provided
+   * @param {string} startDate - ISO string or Date for range start
+   * @param {string} endDate - ISO string or Date for range end
+   * @returns {Object} Object with start and end DateTime objects
+   */
+  getCustomRange(startDate, endDate) {
+    if (!startDate || !endDate) {
+      console.error("[TimeManager] Custom range requires both start and end dates");
+      return null;
+    }
+
+    const start = this.toDateTime(startDate);
+    const end = this.toDateTime(endDate);
+
+    if (!start || !end || !start.isValid || !end.isValid) {
+      console.error("[TimeManager] Invalid dates provided for custom range");
+      return null;
+    }
+
+    // Validate the range
+    if (end < start) {
+      console.error("[TimeManager] End date must be after start date");
+      return null;
+    }
+
+    return {
+      start,
+      end
+    };
+  }
+
+  /**
+   * Get the previous period's date range based on the current period
+   * @param {string} period - The current period
+   * @param {DateTime} now - The current datetime (optional)
+   * @returns {Object} Object with start and end DateTime objects
+   */
+  getPreviousPeriod(period, now = this.getNow()) {
+    const normalizedPeriod = period.startsWith('previous') ? period.replace('previous', 'last') : period;
+    
+    switch (normalizedPeriod) {
+      case 'today': {
+        const yesterday = now.minus({ days: 1 });
+        return {
+          start: this.getDayStart(yesterday),
+          end: this.getDayEnd(yesterday)
+        };
+      }
+      case 'yesterday': {
+        const twoDaysAgo = now.minus({ days: 2 });
+        return {
+          start: this.getDayStart(twoDaysAgo),
+          end: this.getDayEnd(twoDaysAgo)
+        };
+      }
+      case 'last7days': {
+        const dayStart = this.getDayStart(now);
+        const currentStart = dayStart.minus({ days: 6 });
+        const prevEnd = currentStart.minus({ milliseconds: 1 });
+        const prevStart = prevEnd.minus({ days: 6 });
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'last30days': {
+        const dayStart = this.getDayStart(now);
+        const currentStart = dayStart.minus({ days: 29 });
+        const prevEnd = currentStart.minus({ milliseconds: 1 });
+        const prevStart = prevEnd.minus({ days: 29 });
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'last90days': {
+        const dayStart = this.getDayStart(now);
+        const currentStart = dayStart.minus({ days: 89 });
+        const prevEnd = currentStart.minus({ milliseconds: 1 });
+        const prevStart = prevEnd.minus({ days: 89 });
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'thisWeek': {
+        const weekStart = this.getWeekStart(now);
+        const prevEnd = weekStart.minus({ milliseconds: 1 });
+        const prevStart = this.getWeekStart(prevEnd);
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'lastWeek': {
+        const lastWeekStart = this.getWeekStart(now.minus({ weeks: 1 }));
+        const prevEnd = lastWeekStart.minus({ milliseconds: 1 });
+        const prevStart = this.getWeekStart(prevEnd);
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'thisMonth': {
+        const monthStart = now.startOf('month').set({ hour: this.dayStartHour });
+        const prevEnd = monthStart.minus({ milliseconds: 1 });
+        const prevStart = prevEnd.startOf('month').set({ hour: this.dayStartHour });
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      case 'lastMonth': {
+        const lastMonthStart = now.minus({ months: 1 }).startOf('month').set({ hour: this.dayStartHour });
+        const prevEnd = lastMonthStart.minus({ milliseconds: 1 });
+        const prevStart = prevEnd.startOf('month').set({ hour: this.dayStartHour });
+        return {
+          start: prevStart,
+          end: prevEnd
+        };
+      }
+      default:
+        console.warn(`[TimeManager] No previous period defined for: ${period}`);
+        return null;
+    }
+  }
+
+  groupEventsByInterval(events, interval = 'day', property = null) {
+    if (!events?.length) return [];
+
+    const groupedData = new Map();
+    const now = DateTime.now().setZone('America/New_York');
+
+    for (const event of events) {
+      const datetime = DateTime.fromISO(event.attributes.datetime);
+      let groupKey;
+
+      switch (interval) {
+        case 'hour':
+          groupKey = datetime.startOf('hour').toISO();
+          break;
+        case 'day':
+          groupKey = datetime.startOf('day').toISO();
+          break;
+        case 'week':
+          groupKey = datetime.startOf('week').toISO();
+          break;
+        case 'month':
+          groupKey = datetime.startOf('month').toISO();
+          break;
+        default:
+          groupKey = datetime.startOf('day').toISO();
+      }
+
+      const existingGroup = groupedData.get(groupKey) || {
+        datetime: groupKey,
+        count: 0,
+        value: 0
+      };
+
+      existingGroup.count++;
+
+      if (property) {
+        // Extract property value from event
+        const props = event.attributes?.event_properties || event.attributes?.properties || {};
+        let value = 0;
+
+        if (property === '$value') {
+          // Special case for $value - use event value
+          value = Number(event.attributes?.value || 0);
+        } else {
+          // Otherwise get from properties
+          value = Number(props[property] || 0);
+        }
+
+        existingGroup.value = (existingGroup.value || 0) + value;
+      }
+
+      groupedData.set(groupKey, existingGroup);
+    }
+
+    // Convert to array and sort by datetime
+    return Array.from(groupedData.values())
+      .sort((a, b) => DateTime.fromISO(a.datetime) - DateTime.fromISO(b.datetime));
+  }
+} 

inventory-server/db/config-schema-new.sql

@@ -0,0 +1,196 @@
+-- Create function for updating timestamps if it doesn't exist
+CREATE OR REPLACE FUNCTION update_updated_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$$ language 'plpgsql';
+
+-- Create function for updating updated_at timestamps
+CREATE OR REPLACE FUNCTION update_updated_at_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated_at = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$$ language 'plpgsql';
+
+-- Drop tables in reverse order of dependency
+DROP TABLE IF EXISTS public.settings_product CASCADE;
+DROP TABLE IF EXISTS public.settings_vendor CASCADE;
+DROP TABLE IF EXISTS public.settings_global CASCADE;
+
+-- Table Definition: settings_global
+CREATE TABLE public.settings_global (
+    setting_key VARCHAR PRIMARY KEY,
+    setting_value VARCHAR NOT NULL,
+    description TEXT,
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+
+-- Table Definition: settings_vendor
+CREATE TABLE public.settings_vendor (
+    vendor VARCHAR PRIMARY KEY, -- Matches products.vendor
+    default_lead_time_days INT,
+    default_days_of_stock INT,
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+-- Index for faster lookups if needed (PK usually sufficient)
+-- CREATE INDEX idx_settings_vendor_vendor ON public.settings_vendor(vendor);
+
+-- Table Definition: settings_product
+CREATE TABLE public.settings_product (
+    pid INT8 PRIMARY KEY,
+    lead_time_days INT, -- Overrides vendor/global
+    days_of_stock INT, -- Overrides vendor/global
+    safety_stock INT DEFAULT 0, -- Minimum desired stock level
+    forecast_method VARCHAR DEFAULT 'standard', -- e.g., 'standard', 'seasonal'
+    exclude_from_forecast BOOLEAN DEFAULT FALSE,
+    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    CONSTRAINT fk_settings_product_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
+);
+
+
+-- Description: Inserts or updates standard default global settings.
+--              Safe to rerun; will update existing keys with these default values.
+-- Dependencies: `settings_global` table must exist.
+-- Frequency: Run once initially, or rerun if you want to reset global defaults.
+
+INSERT INTO public.settings_global (setting_key, setting_value, description) VALUES
+    ('abc_revenue_threshold_a', '0.80', 'Revenue percentage for Class A (cumulative)'),
+    ('abc_revenue_threshold_b', '0.95', 'Revenue percentage for Class B (cumulative)'),
+    ('abc_calculation_basis', 'revenue_30d', 'Metric for ABC calc (revenue_30d, sales_30d, lifetime_revenue)'),
+    ('abc_calculation_period', '30', 'Days period for ABC calculation if not lifetime'),
+    ('default_forecast_method', 'standard', 'Default forecast method (standard, seasonal)'),
+    ('default_lead_time_days', '14', 'Global default lead time in days'),
+    ('default_days_of_stock', '30', 'Global default days of stock coverage target'),
+    -- Set default safety stock to 0 units. Can be overridden per product.
+    -- If you wanted safety stock in days, you'd store 'days' here and calculate units later.
+    ('default_safety_stock_units', '0', 'Global default safety stock in units')
+ON CONFLICT (setting_key) DO UPDATE SET
+    setting_value = EXCLUDED.setting_value,
+    description = EXCLUDED.description,
+    updated_at = CURRENT_TIMESTAMP; -- Update timestamp if default value changes
+
+
+      
+-- Description: Creates placeholder rows in `settings_vendor` for each unique vendor
+--              found in the `products` table. Does NOT set specific overrides.
+--              Safe to rerun; will NOT overwrite existing vendor settings.
+-- Dependencies: `settings_vendor` table must exist, `products` table populated.
+-- Frequency: Run once after initial product load, or periodically if new vendors are added.
+
+INSERT INTO public.settings_vendor (
+    vendor,
+    default_lead_time_days,
+    default_days_of_stock
+    -- updated_at will use its default CURRENT_TIMESTAMP on insert
+)
+SELECT
+    DISTINCT p.vendor,
+    -- Explicitly cast NULL to INTEGER to resolve type mismatch
+    CAST(NULL AS INTEGER),
+    CAST(NULL AS INTEGER)
+FROM
+    public.products p
+WHERE
+    p.vendor IS NOT NULL
+    AND p.vendor <> '' -- Exclude blank vendors if necessary
+
+ON CONFLICT (vendor) DO NOTHING; -- IMPORTANT: Do not overwrite existing vendor settings
+
+SELECT COUNT(*) FROM public.settings_vendor; -- Verify rows were inserted
+
+      
+-- Description: Creates placeholder rows in `settings_product` for each unique product
+--              found in the `products` table. Sets basic defaults but no specific overrides.
+--              Safe to rerun; will NOT overwrite existing product settings.
+-- Dependencies: `settings_product` table must exist, `products` table populated.
+-- Frequency: Run once after initial product load, or periodically if new products are added.
+
+INSERT INTO public.settings_product (
+    pid,
+    lead_time_days,         -- NULL = Inherit from Vendor/Global
+    days_of_stock,          -- NULL = Inherit from Vendor/Global
+    safety_stock,           -- Default to 0 units initially
+    forecast_method,        -- NULL = Inherit from Global ('standard')
+    exclude_from_forecast   -- Default to FALSE
+    -- updated_at will use its default CURRENT_TIMESTAMP on insert
+)
+SELECT
+    p.pid,
+    CAST(NULL AS INTEGER),  -- Explicitly cast NULL to INTEGER
+    CAST(NULL AS INTEGER),  -- Explicitly cast NULL to INTEGER
+    COALESCE((SELECT setting_value::int FROM settings_global WHERE setting_key = 'default_safety_stock_units'), 0), -- Use global default safety stock units
+    CAST(NULL AS VARCHAR), -- Cast NULL to VARCHAR for forecast_method (already varchar, but explicit)
+    FALSE  -- Default: Include in forecast
+FROM
+    public.products p
+
+ON CONFLICT (pid) DO NOTHING; -- IMPORTANT: Do not overwrite existing product-specific settings
+
+
+-- History and status tables
+CREATE TABLE IF NOT EXISTS calculate_history (
+    id BIGSERIAL PRIMARY KEY,
+    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    end_time TIMESTAMP WITH TIME ZONE NULL,
+    duration_seconds INTEGER,
+    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
+    total_products INTEGER DEFAULT 0,
+    total_orders INTEGER DEFAULT 0,
+    total_purchase_orders INTEGER DEFAULT 0,
+    processed_products INTEGER DEFAULT 0,
+    processed_orders INTEGER DEFAULT 0,
+    processed_purchase_orders INTEGER DEFAULT 0,
+    status calculation_status DEFAULT 'running',
+    error_message TEXT,
+    additional_info JSONB
+);
+
+CREATE TABLE IF NOT EXISTS calculate_status (
+    module_name text PRIMARY KEY,
+    last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE TABLE IF NOT EXISTS sync_status (
+    table_name TEXT PRIMARY KEY,
+    last_sync_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    last_sync_id BIGINT
+);
+
+CREATE TABLE IF NOT EXISTS import_history (
+    id BIGSERIAL PRIMARY KEY,
+    table_name VARCHAR(50) NOT NULL,
+    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    end_time TIMESTAMP WITH TIME ZONE NULL,
+    duration_seconds INTEGER,
+    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
+    records_added INTEGER DEFAULT 0,
+    records_updated INTEGER DEFAULT 0,
+    records_deleted INTEGER DEFAULT 0,
+    records_skipped INTEGER DEFAULT 0,
+    total_processed INTEGER DEFAULT 0,
+    is_incremental BOOLEAN DEFAULT FALSE,
+    status calculation_status DEFAULT 'running',
+    error_message TEXT,
+    additional_info JSONB
+);
+
+-- Create all indexes after tables are fully created
+CREATE INDEX IF NOT EXISTS idx_last_calc ON calculate_status(last_calculation_timestamp);
+CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status(last_sync_timestamp);
+CREATE INDEX IF NOT EXISTS idx_table_time ON import_history(table_name, start_time);
+CREATE INDEX IF NOT EXISTS idx_import_history_status ON import_history(status);
+CREATE INDEX IF NOT EXISTS idx_calculate_history_status ON calculate_history(status);
+
+-- Add comments for documentation
+COMMENT ON TABLE import_history IS 'Tracks history of data import operations with detailed statistics';
+COMMENT ON COLUMN import_history.records_deleted IS 'Number of records deleted during this import';
+COMMENT ON COLUMN import_history.records_skipped IS 'Number of records skipped (e.g., unchanged, invalid)';
+COMMENT ON COLUMN import_history.total_processed IS 'Total number of records examined/processed, including skipped';
+
+COMMENT ON TABLE calculate_history IS 'Tracks history of metrics calculation runs with performance data';
+COMMENT ON COLUMN calculate_history.duration_seconds IS 'Total duration of the calculation in seconds';
+COMMENT ON COLUMN calculate_history.additional_info IS 'JSON object containing step timings, row counts, and other detailed metrics'; 

inventory-server/db/config-schema.sql

@@ -1,278 +0,0 @@
--- Configuration tables schema
-
--- Create function for updating timestamps if it doesn't exist
-CREATE OR REPLACE FUNCTION update_updated_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Create function for updating updated_at timestamps
-CREATE OR REPLACE FUNCTION update_updated_at_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated_at = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Stock threshold configurations
-CREATE TABLE stock_thresholds (
-    id INTEGER NOT NULL,
-    category_id BIGINT,  -- NULL means default/global threshold
-    vendor VARCHAR(100), -- NULL means applies to all vendors
-    critical_days INTEGER NOT NULL DEFAULT 7,
-    reorder_days INTEGER NOT NULL DEFAULT 14,
-    overstock_days INTEGER NOT NULL DEFAULT 90,
-    low_stock_threshold INTEGER NOT NULL DEFAULT 5,
-    min_reorder_quantity INTEGER NOT NULL DEFAULT 1,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE (category_id, vendor)
-);
-
-CREATE TRIGGER update_stock_thresholds_updated
-    BEFORE UPDATE ON stock_thresholds
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
-CREATE INDEX idx_st_metrics ON stock_thresholds(category_id, vendor);
-
--- Lead time threshold configurations
-CREATE TABLE lead_time_thresholds (
-    id INTEGER NOT NULL,
-    category_id BIGINT,  -- NULL means default/global threshold
-    vendor VARCHAR(100), -- NULL means applies to all vendors
-    target_days INTEGER NOT NULL DEFAULT 14,
-    warning_days INTEGER NOT NULL DEFAULT 21,
-    critical_days INTEGER NOT NULL DEFAULT 30,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE (category_id, vendor)
-);
-
-CREATE TRIGGER update_lead_time_thresholds_updated
-    BEFORE UPDATE ON lead_time_thresholds
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Sales velocity window configurations
-CREATE TABLE sales_velocity_config (
-    id INTEGER NOT NULL,
-    category_id BIGINT,  -- NULL means default/global threshold
-    vendor VARCHAR(100), -- NULL means applies to all vendors
-    daily_window_days INTEGER NOT NULL DEFAULT 30,
-    weekly_window_days INTEGER NOT NULL DEFAULT 7,
-    monthly_window_days INTEGER NOT NULL DEFAULT 90,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE (category_id, vendor)
-);
-
-CREATE TRIGGER update_sales_velocity_config_updated
-    BEFORE UPDATE ON sales_velocity_config
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
-CREATE INDEX idx_sv_metrics ON sales_velocity_config(category_id, vendor);
-
--- ABC Classification configurations
-CREATE TABLE abc_classification_config (
-    id INTEGER NOT NULL PRIMARY KEY,
-    a_threshold DECIMAL(5,2) NOT NULL DEFAULT 20.0,
-    b_threshold DECIMAL(5,2) NOT NULL DEFAULT 50.0,
-    classification_period_days INTEGER NOT NULL DEFAULT 90,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TRIGGER update_abc_classification_config_updated
-    BEFORE UPDATE ON abc_classification_config
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Safety stock configurations
-CREATE TABLE safety_stock_config (
-    id INTEGER NOT NULL,
-    category_id BIGINT,  -- NULL means default/global threshold
-    vendor VARCHAR(100), -- NULL means applies to all vendors
-    coverage_days INTEGER NOT NULL DEFAULT 14,
-    service_level DECIMAL(5,2) NOT NULL DEFAULT 95.0,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE (category_id, vendor)
-);
-
-CREATE TRIGGER update_safety_stock_config_updated
-    BEFORE UPDATE ON safety_stock_config
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
-CREATE INDEX idx_ss_metrics ON safety_stock_config(category_id, vendor);
-
--- Turnover rate configurations
-CREATE TABLE turnover_config (
-    id INTEGER NOT NULL,
-    category_id BIGINT,  -- NULL means default/global threshold
-    vendor VARCHAR(100), -- NULL means applies to all vendors
-    calculation_period_days INTEGER NOT NULL DEFAULT 30,
-    target_rate DECIMAL(10,2) NOT NULL DEFAULT 1.0,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE (category_id, vendor)
-);
-
-CREATE TRIGGER update_turnover_config_updated
-    BEFORE UPDATE ON turnover_config
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Create table for sales seasonality factors
-CREATE TABLE sales_seasonality (
-    month INTEGER NOT NULL,
-    seasonality_factor DECIMAL(5,3) DEFAULT 0,
-    last_updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (month),
-    CONSTRAINT month_range CHECK (month BETWEEN 1 AND 12),
-    CONSTRAINT seasonality_range CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
-);
-
-CREATE TRIGGER update_sales_seasonality_updated
-    BEFORE UPDATE ON sales_seasonality
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Insert default global thresholds
-INSERT INTO stock_thresholds (id, category_id, vendor, critical_days, reorder_days, overstock_days)
-VALUES (1, NULL, NULL, 7, 14, 90)
-ON CONFLICT (id) DO UPDATE SET
-    critical_days = EXCLUDED.critical_days,
-    reorder_days = EXCLUDED.reorder_days,
-    overstock_days = EXCLUDED.overstock_days;
-
-INSERT INTO lead_time_thresholds (id, category_id, vendor, target_days, warning_days, critical_days)
-VALUES (1, NULL, NULL, 14, 21, 30)
-ON CONFLICT (id) DO UPDATE SET
-    target_days = EXCLUDED.target_days,
-    warning_days = EXCLUDED.warning_days,
-    critical_days = EXCLUDED.critical_days;
-
-INSERT INTO sales_velocity_config (id, category_id, vendor, daily_window_days, weekly_window_days, monthly_window_days)
-VALUES (1, NULL, NULL, 30, 7, 90)
-ON CONFLICT (id) DO UPDATE SET
-    daily_window_days = EXCLUDED.daily_window_days,
-    weekly_window_days = EXCLUDED.weekly_window_days,
-    monthly_window_days = EXCLUDED.monthly_window_days;
-
-INSERT INTO abc_classification_config (id, a_threshold, b_threshold, classification_period_days)
-VALUES (1, 20.0, 50.0, 90)
-ON CONFLICT (id) DO UPDATE SET
-    a_threshold = EXCLUDED.a_threshold,
-    b_threshold = EXCLUDED.b_threshold,
-    classification_period_days = EXCLUDED.classification_period_days;
-
-INSERT INTO safety_stock_config (id, category_id, vendor, coverage_days, service_level)
-VALUES (1, NULL, NULL, 14, 95.0)
-ON CONFLICT (id) DO UPDATE SET
-    coverage_days = EXCLUDED.coverage_days,
-    service_level = EXCLUDED.service_level;
-
-INSERT INTO turnover_config (id, category_id, vendor, calculation_period_days, target_rate)
-VALUES (1, NULL, NULL, 30, 1.0)
-ON CONFLICT (id) DO UPDATE SET
-    calculation_period_days = EXCLUDED.calculation_period_days,
-    target_rate = EXCLUDED.target_rate;
-
--- Insert default seasonality factors (neutral)
-INSERT INTO sales_seasonality (month, seasonality_factor)
-VALUES 
-    (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
-    (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
-ON CONFLICT (month) DO UPDATE SET
-    last_updated = CURRENT_TIMESTAMP;
-
--- View to show thresholds with category names
-CREATE OR REPLACE VIEW stock_thresholds_view AS
-SELECT 
-    st.*,
-    c.name as category_name,
-    CASE 
-        WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 'Global Default'
-        WHEN st.category_id IS NULL THEN 'Vendor: ' || st.vendor
-        WHEN st.vendor IS NULL THEN 'Category: ' || c.name
-        ELSE 'Category: ' || c.name || ' / Vendor: ' || st.vendor
-    END as threshold_scope
-FROM 
-    stock_thresholds st
-LEFT JOIN 
-    categories c ON st.category_id = c.cat_id
-ORDER BY 
-    CASE 
-        WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 1
-        WHEN st.category_id IS NULL THEN 2
-        WHEN st.vendor IS NULL THEN 3
-        ELSE 4
-    END,
-    c.name,
-    st.vendor;
-
--- History and status tables
-CREATE TABLE IF NOT EXISTS calculate_history (
-    id BIGSERIAL PRIMARY KEY,
-    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    end_time TIMESTAMP WITH TIME ZONE NULL,
-    duration_seconds INTEGER,
-    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
-    total_products INTEGER DEFAULT 0,
-    total_orders INTEGER DEFAULT 0,
-    total_purchase_orders INTEGER DEFAULT 0,
-    processed_products INTEGER DEFAULT 0,
-    processed_orders INTEGER DEFAULT 0,
-    processed_purchase_orders INTEGER DEFAULT 0,
-    status calculation_status DEFAULT 'running',
-    error_message TEXT,
-    additional_info JSONB
-);
-
-CREATE TABLE IF NOT EXISTS calculate_status (
-    module_name module_name PRIMARY KEY,
-    last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE IF NOT EXISTS sync_status (
-    table_name VARCHAR(50) PRIMARY KEY,
-    last_sync_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    last_sync_id BIGINT
-);
-
-CREATE TABLE IF NOT EXISTS import_history (
-    id BIGSERIAL PRIMARY KEY,
-    table_name VARCHAR(50) NOT NULL,
-    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    end_time TIMESTAMP WITH TIME ZONE NULL,
-    duration_seconds INTEGER,
-    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
-    records_added INTEGER DEFAULT 0,
-    records_updated INTEGER DEFAULT 0,
-    is_incremental BOOLEAN DEFAULT FALSE,
-    status calculation_status DEFAULT 'running',
-    error_message TEXT,
-    additional_info JSONB
-);
-
--- Create all indexes after tables are fully created
-CREATE INDEX IF NOT EXISTS idx_last_calc ON calculate_status(last_calculation_timestamp);
-CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status(last_sync_timestamp);
-CREATE INDEX IF NOT EXISTS idx_table_time ON import_history(table_name, start_time); 

inventory-server/db/daily-deals-schema.sql

@@ -0,0 +1,17 @@
+-- Daily Deals schema for local PostgreSQL
+-- Synced from production MySQL product_daily_deals + product_current_prices
+
+CREATE TABLE IF NOT EXISTS product_daily_deals (
+    deal_id serial PRIMARY KEY,
+    deal_date date NOT NULL,
+    pid bigint NOT NULL,
+    price_id bigint NOT NULL,
+    -- Denormalized from product_current_prices so we don't need to sync that whole table
+    deal_price numeric(10,3),
+    created_at timestamptz DEFAULT NOW(),
+    CONSTRAINT fk_daily_deals_pid FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
+);
+
+CREATE INDEX IF NOT EXISTS idx_daily_deals_date ON product_daily_deals(deal_date);
+CREATE INDEX IF NOT EXISTS idx_daily_deals_pid ON product_daily_deals(pid);
+CREATE UNIQUE INDEX IF NOT EXISTS idx_daily_deals_unique ON product_daily_deals(deal_date, pid);

inventory-server/db/functions.sql

@@ -0,0 +1,249 @@
+-- Custom PostgreSQL functions used by the metrics pipeline
+-- These must exist in the database before running calculate-metrics-new.js
+--
+-- To install/update: psql -d inventory_db -f functions.sql
+-- All functions use CREATE OR REPLACE so they are safe to re-run.
+
+-- =============================================================================
+-- safe_divide: Division helper that returns a default value instead of erroring
+-- on NULL or zero denominators.
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.safe_divide(
+    numerator numeric,
+    denominator numeric,
+    default_value numeric DEFAULT NULL::numeric
+)
+RETURNS numeric
+LANGUAGE plpgsql
+IMMUTABLE
+AS $function$
+BEGIN
+    IF denominator IS NULL OR denominator = 0 THEN
+        RETURN default_value;
+    ELSE
+        RETURN numerator / denominator;
+    END IF;
+END;
+$function$;
+
+-- =============================================================================
+-- std_numeric: Standardized rounding helper for consistent numeric precision.
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.std_numeric(
+    value numeric,
+    precision_digits integer DEFAULT 2
+)
+RETURNS numeric
+LANGUAGE plpgsql
+IMMUTABLE
+AS $function$
+BEGIN
+    IF value IS NULL THEN
+        RETURN NULL;
+    ELSE
+        RETURN ROUND(value, precision_digits);
+    END IF;
+END;
+$function$;
+
+-- =============================================================================
+-- calculate_sales_velocity: Daily sales velocity adjusted for stockout days.
+-- Ensures at least 14-day denominator for products with sales to avoid
+-- inflated velocity from short windows.
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.calculate_sales_velocity(
+    sales_30d integer,
+    stockout_days_30d integer
+)
+RETURNS numeric
+LANGUAGE plpgsql
+IMMUTABLE
+AS $function$
+BEGIN
+    RETURN sales_30d /
+        NULLIF(
+            GREATEST(
+                30.0 - stockout_days_30d,
+                CASE
+                    WHEN sales_30d > 0 THEN 14.0  -- If we have sales, ensure at least 14 days denominator
+                    ELSE 30.0                      -- If no sales, use full period
+                END
+            ),
+            0
+        );
+END;
+$function$;
+
+-- =============================================================================
+-- get_weighted_avg_cost: Weighted average cost from receivings up to a given date.
+-- Prefers receivings from the 365 days before p_date so decade-old costs don't
+-- weigh equally with recent ones; falls back to the lifetime average when the
+-- product had no receivings in that window.
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.get_weighted_avg_cost(
+    p_pid bigint,
+    p_date date
+)
+RETURNS numeric
+LANGUAGE plpgsql
+STABLE
+AS $function$
+DECLARE
+    weighted_cost NUMERIC;
+BEGIN
+    SELECT
+        CASE
+            WHEN SUM(qty_each) > 0 THEN SUM(cost_each * qty_each) / SUM(qty_each)
+            ELSE NULL
+        END INTO weighted_cost
+    FROM receivings
+    WHERE pid = p_pid
+        AND received_date <= p_date
+        AND received_date > p_date - INTERVAL '365 days'
+        AND status != 'canceled';
+
+    IF weighted_cost IS NULL THEN
+        SELECT
+            CASE
+                WHEN SUM(qty_each) > 0 THEN SUM(cost_each * qty_each) / SUM(qty_each)
+                ELSE NULL
+            END INTO weighted_cost
+        FROM receivings
+        WHERE pid = p_pid
+            AND received_date <= p_date
+            AND status != 'canceled';
+    END IF;
+
+    RETURN weighted_cost;
+END;
+$function$;
+
+-- =============================================================================
+-- classify_demand_pattern: Classifies demand based on average demand and
+-- coefficient of variation (CV). Standard inventory classification:
+--   zero:     no demand
+--   stable:   CV <= 0.2 (predictable, easy to forecast)
+--   variable: CV <= 0.5 (some variability, still forecastable)
+--   sporadic: low volume + high CV (intermittent demand)
+--   lumpy:    high volume + high CV (unpredictable bursts)
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.classify_demand_pattern(
+    avg_demand numeric,
+    cv numeric
+)
+RETURNS character varying
+LANGUAGE plpgsql
+IMMUTABLE
+AS $function$
+BEGIN
+    IF avg_demand IS NULL OR cv IS NULL THEN
+        RETURN NULL;
+    ELSIF avg_demand = 0 THEN
+        RETURN 'zero';
+    ELSIF cv <= 0.2 THEN
+        RETURN 'stable';
+    ELSIF cv <= 0.5 THEN
+        RETURN 'variable';
+    ELSIF avg_demand < 1.0 THEN
+        RETURN 'sporadic';
+    ELSE
+        RETURN 'lumpy';
+    END IF;
+END;
+$function$;
+
+-- =============================================================================
+-- detect_seasonal_pattern: Detects seasonality by comparing monthly average
+-- sales across the last 12 months. Uses coefficient of variation across months
+-- and peak-to-average ratio to classify patterns.
+--
+-- Returns:
+--   seasonal_pattern: 'none', 'moderate', or 'strong'
+--   seasonality_index: peak month avg / overall avg * 100 (100 = no seasonality)
+--   peak_season: name of peak month (e.g. 'January'), or NULL if none
+-- =============================================================================
+CREATE OR REPLACE FUNCTION public.detect_seasonal_pattern(p_pid bigint)
+RETURNS TABLE(seasonal_pattern character varying, seasonality_index numeric, peak_season character varying)
+LANGUAGE plpgsql
+STABLE
+AS $function$
+DECLARE
+    v_monthly_cv NUMERIC;
+    v_max_month_avg NUMERIC;
+    v_overall_avg NUMERIC;
+    v_monthly_stddev NUMERIC;
+    v_peak_month_num INT;
+    v_data_months INT;
+    v_seasonality_index NUMERIC;
+    v_seasonal_pattern VARCHAR;
+    v_peak_season VARCHAR;
+BEGIN
+    -- Gather monthly average sales and peak month in a single query
+    SELECT
+        COUNT(*),
+        AVG(month_avg),
+        STDDEV(month_avg),
+        MAX(month_avg),
+        (ARRAY_AGG(mo ORDER BY month_avg DESC))[1]::INT
+    INTO v_data_months, v_overall_avg, v_monthly_stddev, v_max_month_avg, v_peak_month_num
+    FROM (
+        SELECT EXTRACT(MONTH FROM snapshot_date) AS mo, AVG(units_sold) AS month_avg
+        FROM daily_product_snapshots
+        WHERE pid = p_pid AND snapshot_date >= CURRENT_DATE - INTERVAL '365 days'
+        GROUP BY EXTRACT(MONTH FROM snapshot_date)
+    ) monthly;
+
+    -- Need at least 3 months of data for meaningful seasonality detection
+    IF v_data_months < 3 OR v_overall_avg IS NULL OR v_overall_avg = 0 THEN
+        RETURN QUERY SELECT 'none'::VARCHAR, 100::NUMERIC, NULL::VARCHAR;
+        RETURN;
+    END IF;
+
+    -- CV of monthly averages
+    v_monthly_cv := v_monthly_stddev / v_overall_avg;
+
+    -- Seasonality index: peak month avg / overall avg * 100
+    v_seasonality_index := ROUND((v_max_month_avg / v_overall_avg * 100)::NUMERIC, 2);
+
+    IF v_monthly_cv > 0.5 AND v_seasonality_index > 150 THEN
+        v_seasonal_pattern := 'strong';
+        v_peak_season := TRIM(TO_CHAR(TO_DATE(v_peak_month_num::TEXT, 'MM'), 'Month'));
+    ELSIF v_monthly_cv > 0.3 AND v_seasonality_index > 120 THEN
+        v_seasonal_pattern := 'moderate';
+        v_peak_season := TRIM(TO_CHAR(TO_DATE(v_peak_month_num::TEXT, 'MM'), 'Month'));
+    ELSE
+        v_seasonal_pattern := 'none';
+        v_peak_season := NULL;
+        v_seasonality_index := 100;
+    END IF;
+
+    RETURN QUERY SELECT v_seasonal_pattern, v_seasonality_index, v_peak_season;
+END;
+$function$;
+
+-- =============================================================================
+-- category_hierarchy: Materialized view providing a recursive category tree
+-- with ancestor paths for efficient rollup queries.
+--
+-- Refresh after category changes: REFRESH MATERIALIZED VIEW category_hierarchy;
+-- =============================================================================
+-- DROP MATERIALIZED VIEW IF EXISTS category_hierarchy;
+-- CREATE MATERIALIZED VIEW category_hierarchy AS
+-- WITH RECURSIVE cat_tree AS (
+--     SELECT cat_id, name, type, parent_id,
+--            cat_id AS root_id, 0 AS level, ARRAY[cat_id] AS path
+--     FROM categories
+--     WHERE parent_id IS NULL
+--     UNION ALL
+--     SELECT c.cat_id, c.name, c.type, c.parent_id,
+--            ct.root_id, ct.level + 1, ct.path || c.cat_id
+--     FROM categories c
+--     JOIN cat_tree ct ON c.parent_id = ct.cat_id
+-- )
+-- SELECT cat_id, name, type, parent_id, root_id, level, path,
+--        (SELECT array_agg(unnest ORDER BY unnest DESC)
+--         FROM unnest(cat_tree.path) unnest
+--         WHERE unnest <> cat_tree.cat_id) AS ancestor_ids
+-- FROM cat_tree;
+--
+-- CREATE UNIQUE INDEX ON category_hierarchy (cat_id);

inventory-server/db/metrics-schema-new.sql

@@ -0,0 +1,343 @@
+-- Drop tables in reverse order of dependency
+DROP TABLE IF EXISTS public.product_metrics CASCADE;
+DROP TABLE IF EXISTS public.daily_product_snapshots CASCADE;
+
+-- Table Definition: daily_product_snapshots
+CREATE TABLE public.daily_product_snapshots (
+    snapshot_date DATE NOT NULL,
+    pid INT8 NOT NULL,
+    sku VARCHAR, -- Copied for convenience
+
+    -- Inventory Metrics (End of Day / Last Snapshot of Day)
+    eod_stock_quantity INT NOT NULL DEFAULT 0,
+    eod_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- Increased precision
+    eod_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    eod_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    stockout_flag BOOLEAN NOT NULL DEFAULT FALSE,
+
+    -- Sales Metrics (Aggregated for the snapshot_date)
+    units_sold INT NOT NULL DEFAULT 0,
+    units_returned INT NOT NULL DEFAULT 0,
+    gross_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    discounts NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    returns_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    net_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- gross_revenue - discounts
+    cogs NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    gross_regular_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    profit NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- net_revenue - cogs
+
+    -- Receiving Metrics (Aggregated for the snapshot_date)
+    units_received INT NOT NULL DEFAULT 0,
+    cost_received NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+
+    calculation_timestamp TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+    PRIMARY KEY (snapshot_date, pid) -- Composite primary key
+    -- CONSTRAINT fk_daily_snapshot_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE -- FK Optional on snapshot table
+);
+
+-- Add Indexes for daily_product_snapshots
+CREATE INDEX idx_daily_snapshot_pid_date ON public.daily_product_snapshots(pid, snapshot_date); -- Useful for product-specific time series
+
+
+-- Table Definition: product_metrics
+CREATE TABLE public.product_metrics (
+    pid INT8 PRIMARY KEY,
+    last_calculated TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+    -- Product Info (Copied for convenience/performance)
+    sku VARCHAR,
+    title VARCHAR,
+    brand VARCHAR,
+    vendor VARCHAR,
+    image_url VARCHAR, -- (e.g., products.image_175)
+    is_visible BOOLEAN,
+    is_replenishable BOOLEAN,
+    
+    -- Additional product fields
+    barcode VARCHAR,
+    harmonized_tariff_code VARCHAR,
+    vendor_reference VARCHAR,
+    notions_reference VARCHAR,
+    line VARCHAR,
+    subline VARCHAR,
+    artist VARCHAR,
+    moq INT,
+    rating NUMERIC(10, 2),
+    reviews INT,
+    weight NUMERIC(14, 4),
+    length NUMERIC(14, 4),
+    width NUMERIC(14, 4),
+    height NUMERIC(14, 4),
+    country_of_origin VARCHAR,
+    location VARCHAR,
+    baskets INT,
+    notifies INT,
+    preorder_count INT,
+    notions_inv_count INT,
+
+    -- Current Status (Refreshed Hourly)
+    current_price NUMERIC(10, 2),
+    current_regular_price NUMERIC(10, 2),
+    current_cost_price NUMERIC(10, 4), -- Increased precision for cost
+    current_stock INT NOT NULL DEFAULT 0,
+    current_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    current_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    current_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    on_order_qty INT NOT NULL DEFAULT 0,
+    on_order_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    on_order_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
+    earliest_expected_date DATE,
+    -- total_received_lifetime INT NOT NULL DEFAULT 0, -- Can calc if needed
+
+    -- Historical Dates (Calculated Once/Periodically)
+    date_created DATE,
+    date_first_received DATE,
+    date_last_received DATE,
+    date_first_sold DATE,
+    date_last_sold DATE,
+    age_days INT, -- Calculated based on LEAST(date_created, date_first_sold)
+
+    -- Rolling Period Metrics (Refreshed Hourly from daily_product_snapshots)
+    sales_7d INT, revenue_7d NUMERIC(14, 4),
+    sales_14d INT, revenue_14d NUMERIC(14, 4),
+    sales_30d INT, revenue_30d NUMERIC(14, 4),
+    cogs_30d NUMERIC(14, 4), profit_30d NUMERIC(14, 4),
+    returns_units_30d INT, returns_revenue_30d NUMERIC(14, 4),
+    discounts_30d NUMERIC(14, 4),
+    gross_revenue_30d NUMERIC(14, 4), gross_regular_revenue_30d NUMERIC(14, 4),
+    stockout_days_30d INT,
+    sales_365d INT, revenue_365d NUMERIC(14, 4),
+    avg_stock_units_30d NUMERIC(10, 2), avg_stock_cost_30d NUMERIC(14, 4),
+    avg_stock_retail_30d NUMERIC(14, 4), avg_stock_gross_30d NUMERIC(14, 4),
+    received_qty_30d INT, received_cost_30d NUMERIC(14, 4),
+
+    -- Lifetime Metrics (Recalculated Hourly/Daily from daily_product_snapshots)
+    lifetime_sales INT,
+    lifetime_revenue NUMERIC(16, 4),
+    lifetime_revenue_quality VARCHAR(10), -- 'exact', 'partial', 'estimated'
+
+    -- First Period Metrics (Calculated Once/Periodically from daily_product_snapshots)
+    first_7_days_sales INT, first_7_days_revenue NUMERIC(14, 4),
+    first_30_days_sales INT, first_30_days_revenue NUMERIC(14, 4),
+    first_60_days_sales INT, first_60_days_revenue NUMERIC(14, 4),
+    first_90_days_sales INT, first_90_days_revenue NUMERIC(14, 4),
+
+    -- Calculated KPIs (Refreshed Hourly based on rolling metrics)
+    asp_30d NUMERIC(10, 2), -- revenue_30d / sales_30d
+    acp_30d NUMERIC(10, 4), -- cogs_30d / sales_30d
+    avg_ros_30d NUMERIC(10, 4), -- profit_30d / sales_30d
+    avg_sales_per_day_30d NUMERIC(10, 2), -- sales_30d / 30.0
+    avg_sales_per_month_30d NUMERIC(10, 2), -- sales_30d (assuming 30d = 1 month for this metric)
+    margin_30d NUMERIC(8, 2), -- (profit_30d / revenue_30d) * 100
+    markup_30d NUMERIC(8, 2), -- (profit_30d / cogs_30d) * 100
+    gmroi_30d NUMERIC(10, 2), -- profit_30d / avg_stock_cost_30d
+    stockturn_30d NUMERIC(10, 2), -- sales_30d / avg_stock_units_30d
+    return_rate_30d NUMERIC(8, 2), -- returns_units_30d / (sales_30d + returns_units_30d) * 100
+    discount_rate_30d NUMERIC(8, 2), -- discounts_30d / gross_revenue_30d * 100
+    stockout_rate_30d NUMERIC(8, 2), -- stockout_days_30d / 30.0 * 100
+    markdown_30d NUMERIC(14, 4), -- gross_regular_revenue_30d - gross_revenue_30d
+    markdown_rate_30d NUMERIC(8, 2), -- markdown_30d / gross_regular_revenue_30d * 100
+    sell_through_30d NUMERIC(8, 2), -- sales_30d / (current_stock + sales_30d) * 100
+    avg_lead_time_days INT, -- Calculated Periodically from purchase_orders
+
+    -- Forecasting & Replenishment (Refreshed Hourly)
+    abc_class CHAR(1), -- Updated Periodically (e.g., Weekly)
+    sales_velocity_daily NUMERIC(10, 4), -- sales_30d / (30.0 - stockout_days_30d)
+    config_lead_time INT, -- From settings tables
+    config_days_of_stock INT, -- From settings tables
+    config_safety_stock INT, -- From settings_product
+    planning_period_days INT, -- config_lead_time + config_days_of_stock
+    lead_time_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_lead_time
+    days_of_stock_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_days_of_stock
+    planning_period_forecast_units NUMERIC(10, 2), -- lead_time_forecast_units + days_of_stock_forecast_units
+    lead_time_closing_stock NUMERIC(10, 2), -- current_stock + on_order_qty - lead_time_forecast_units
+    days_of_stock_closing_stock NUMERIC(10, 2), -- lead_time_closing_stock - days_of_stock_forecast_units
+    replenishment_needed_raw NUMERIC(10, 2), -- planning_period_forecast_units + config_safety_stock - current_stock - on_order_qty
+    replenishment_units INT, -- CEILING(GREATEST(0, replenishment_needed_raw))
+    replenishment_cost NUMERIC(14, 4), -- replenishment_units * current_cost_price
+    replenishment_retail NUMERIC(14, 4), -- replenishment_units * current_price
+    replenishment_profit NUMERIC(14, 4), -- replenishment_units * (current_price - current_cost_price)
+    to_order_units INT, -- Apply MOQ/UOM logic to replenishment_units
+    forecast_lost_sales_units NUMERIC(10, 2), -- GREATEST(0, -lead_time_closing_stock)
+    forecast_lost_revenue NUMERIC(14, 4), -- forecast_lost_sales_units * current_price
+    stock_cover_in_days NUMERIC(10, 1), -- current_stock / sales_velocity_daily
+    po_cover_in_days NUMERIC(10, 1), -- on_order_qty / sales_velocity_daily
+    sells_out_in_days NUMERIC(10, 1), -- (current_stock + on_order_qty) / sales_velocity_daily
+    replenish_date DATE, -- Calc based on when stock hits safety stock minus lead time
+    overstocked_units INT, -- GREATEST(0, current_stock - config_safety_stock - planning_period_forecast_units)
+    overstocked_cost NUMERIC(14, 4), -- overstocked_units * current_cost_price
+    overstocked_retail NUMERIC(14, 4), -- overstocked_units * current_price
+    is_old_stock BOOLEAN, -- Based on age, last sold, last received, on_order status
+
+    -- Yesterday's Metrics (Refreshed Hourly from daily_product_snapshots)
+    yesterday_sales INT,
+
+    -- Product Status (Calculated from metrics)
+    status VARCHAR, -- Stores status values like: Critical, Reorder Soon, Healthy, Overstock, At Risk, New
+
+    -- Growth Metrics (P3)
+    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
+    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
+    sales_growth_yoy NUMERIC(10, 2), -- Year-over-year sales growth %
+    revenue_growth_yoy NUMERIC(10, 2), -- Year-over-year revenue growth %
+
+    -- Demand Variability Metrics (P3)
+    sales_variance_30d NUMERIC(10, 2), -- Variance of daily sales
+    sales_std_dev_30d NUMERIC(10, 2), -- Standard deviation of daily sales
+    sales_cv_30d NUMERIC(10, 2), -- Coefficient of variation
+    demand_pattern VARCHAR(20), -- 'stable', 'variable', 'sporadic', 'lumpy'
+
+    -- Service Level & Fill Rate (P5)
+    fill_rate_30d NUMERIC(8, 2), -- % of demand fulfilled from stock
+    stockout_incidents_30d INT, -- Days with stockouts
+    service_level_30d NUMERIC(8, 2), -- % of days without stockouts
+    lost_sales_incidents_30d INT, -- Days with potential lost sales
+
+    -- Seasonality (P5)
+    seasonality_index NUMERIC(10, 2), -- Current vs average (100 = average)
+    seasonal_pattern VARCHAR(20), -- 'none', 'weekly', 'monthly', 'quarterly', 'yearly'
+    peak_season VARCHAR(20), -- e.g., 'Q4', 'summer', 'holiday'
+
+    CONSTRAINT fk_product_metrics_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
+);
+
+-- Add Indexes for product_metrics (adjust based on common filtering/sorting in frontend)
+CREATE INDEX idx_product_metrics_brand ON public.product_metrics(brand);
+CREATE INDEX idx_product_metrics_vendor ON public.product_metrics(vendor);
+CREATE INDEX idx_product_metrics_sku ON public.product_metrics(sku);
+CREATE INDEX idx_product_metrics_abc_class ON public.product_metrics(abc_class);
+CREATE INDEX idx_product_metrics_revenue_30d ON public.product_metrics(revenue_30d DESC NULLS LAST); -- Example sorting index
+CREATE INDEX idx_product_metrics_sales_30d ON public.product_metrics(sales_30d DESC NULLS LAST); -- Example sorting index
+CREATE INDEX idx_product_metrics_current_stock ON public.product_metrics(current_stock);
+CREATE INDEX idx_product_metrics_sells_out_in_days ON public.product_metrics(sells_out_in_days ASC NULLS LAST); -- Example sorting index
+CREATE INDEX idx_product_metrics_status ON public.product_metrics(status); -- Index for status filtering
+
+-- Add new vendor, category, and brand metrics tables
+-- Drop tables in reverse order if they exist
+DROP TABLE IF EXISTS public.brand_metrics CASCADE;
+DROP TABLE IF EXISTS public.vendor_metrics CASCADE;
+DROP TABLE IF EXISTS public.category_metrics CASCADE;
+
+-- ========= Category Metrics =========
+CREATE TABLE public.category_metrics (
+    category_id INT8 PRIMARY KEY, -- Foreign key to categories.cat_id
+    category_name VARCHAR,        -- Denormalized for convenience
+    category_type INT2,           -- Denormalized for convenience
+    parent_id INT8,             -- Denormalized for convenience
+    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+    -- ROLLED-UP METRICS (includes this category + all descendants)
+    -- Counts & Basic Info
+    product_count INT NOT NULL DEFAULT 0,           -- Total products linked
+    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products linked
+    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products linked
+
+    -- Current Stock Value (approximated using current product costs/prices)
+    current_stock_units INT NOT NULL DEFAULT 0,
+    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+
+    -- Rolling Period Aggregates (Summed from product_metrics)
+    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
+
+    -- DIRECT METRICS (only products directly in this category)
+    direct_product_count INT NOT NULL DEFAULT 0,           -- Products directly in this category
+    direct_active_product_count INT NOT NULL DEFAULT 0,    -- Visible products directly in this category
+    direct_replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products directly in this category
+
+    -- Direct Current Stock Value
+    direct_current_stock_units INT NOT NULL DEFAULT 0,
+    direct_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    direct_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+
+    -- Direct Rolling Period Aggregates
+    direct_sales_7d INT NOT NULL DEFAULT 0, direct_revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    direct_sales_30d INT NOT NULL DEFAULT 0, direct_revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    direct_profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, direct_cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    direct_sales_365d INT NOT NULL DEFAULT 0, direct_revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    direct_lifetime_sales INT NOT NULL DEFAULT 0, direct_lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
+
+    -- Calculated KPIs (Based on 30d aggregates) - Apply to rolled-up metrics
+    avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
+    stock_turn_30d NUMERIC(10, 3), -- sales_units / avg_stock_units (Needs avg stock calc)
+    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
+    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
+
+    CONSTRAINT fk_category_metrics_cat_id FOREIGN KEY (category_id) REFERENCES public.categories(cat_id) ON DELETE CASCADE ON UPDATE CASCADE
+);
+CREATE INDEX idx_category_metrics_name ON public.category_metrics(category_name);
+CREATE INDEX idx_category_metrics_type ON public.category_metrics(category_type);
+
+-- ========= Vendor Metrics =========
+CREATE TABLE public.vendor_metrics (
+    vendor_name VARCHAR PRIMARY KEY, -- Matches products.vendor
+    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+    -- Counts & Basic Info
+    product_count INT NOT NULL DEFAULT 0,           -- Total products from this vendor
+    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products
+    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
+
+    -- Current Stock Value (approximated)
+    current_stock_units INT NOT NULL DEFAULT 0,
+    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+
+    -- On Order Value
+    on_order_units INT NOT NULL DEFAULT 0,
+    on_order_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+
+    -- PO Performance (Simplified)
+    po_count_365d INT NOT NULL DEFAULT 0, -- Count of distinct POs created in last year
+    avg_lead_time_days INT,             -- Calculated from received POs historically
+
+    -- Rolling Period Aggregates (Summed from product_metrics)
+    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
+
+    -- Calculated KPIs (Based on 30d aggregates)
+    avg_margin_30d NUMERIC(14, 4), -- (profit / revenue) * 100
+    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
+    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
+    -- Add more KPIs if needed (e.g., avg product value, sell-through rate for vendor)
+);
+CREATE INDEX idx_vendor_metrics_active_count ON public.vendor_metrics(active_product_count);
+
+
+-- ========= Brand Metrics =========
+CREATE TABLE public.brand_metrics (
+    brand_name VARCHAR PRIMARY KEY, -- Matches products.brand (use 'Unbranded' for NULLs)
+    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+
+    -- Counts & Basic Info
+    product_count INT NOT NULL DEFAULT 0,           -- Total products of this brand
+    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products
+    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
+
+    -- Current Stock Value (approximated)
+    current_stock_units INT NOT NULL DEFAULT 0,
+    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+
+    -- Rolling Period Aggregates (Summed from product_metrics)
+    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
+    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
+
+    -- Calculated KPIs (Based on 30d aggregates)
+    avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
+    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
+    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
+    -- Add more KPIs if needed (e.g., avg product value, sell-through rate for brand)
+);
+CREATE INDEX idx_brand_metrics_active_count ON public.brand_metrics(active_product_count);

inventory-server/db/migrations/add-date-online-and-shop-score.sql

@@ -0,0 +1,20 @@
+-- Migration: Add date_online and shop_score columns to products table
+-- These fields are imported from production to improve newsletter recommendation accuracy:
+--   date_online  = products.date_ol in production (date product went live on the shop)
+--   shop_score   = products.score in production (sales-based popularity score)
+--
+-- After running this migration, do a full (non-incremental) import to backfill:
+--   INCREMENTAL_UPDATE=false node scripts/import-from-prod.js
+
+-- Add date_online column (production: products.date_ol)
+ALTER TABLE products ADD COLUMN IF NOT EXISTS date_online TIMESTAMP WITH TIME ZONE;
+
+-- Add shop_score column (production: products.score)
+-- Using NUMERIC(10,2) to preserve the decimal precision from production
+ALTER TABLE products ADD COLUMN IF NOT EXISTS shop_score NUMERIC(10, 2) DEFAULT 0;
+
+-- If shop_score was previously created as INTEGER, convert it
+ALTER TABLE products ALTER COLUMN shop_score TYPE NUMERIC(10, 2);
+
+-- Index on date_online for the newsletter "new products" filter
+CREATE INDEX IF NOT EXISTS idx_products_date_online ON products(date_online);

inventory-server/db/schema.sql

@@ -4,7 +4,12 @@ SET session_replication_role = 'replica';  -- Disable foreign key checks tempora
 -- Create function for updating timestamps
 CREATE OR REPLACE FUNCTION update_updated_column() RETURNS TRIGGER AS $func$
 BEGIN
-    NEW.updated = CURRENT_TIMESTAMP;
+    -- Check which table is being updated and use the appropriate column
+    IF TG_TABLE_NAME = 'categories' THEN
+        NEW.updated_at = CURRENT_TIMESTAMP;
+    ELSIF TG_TABLE_NAME IN ('products', 'orders', 'purchase_orders', 'receivings') THEN
+        NEW.updated = CURRENT_TIMESTAMP;
+    END IF;
     RETURN NEW;
 END;
 $func$ language plpgsql;
@@ -12,52 +17,53 @@ $func$ language plpgsql;
 -- Create tables
 CREATE TABLE products (
     pid BIGINT NOT NULL,
-    title VARCHAR(255) NOT NULL,
+    title TEXT NOT NULL,
     description TEXT,
-    SKU VARCHAR(50) NOT NULL,
+    sku TEXT NOT NULL,
     created_at TIMESTAMP WITH TIME ZONE,
+    date_online TIMESTAMP WITH TIME ZONE,
     first_received TIMESTAMP WITH TIME ZONE,
     stock_quantity INTEGER DEFAULT 0,
     preorder_count INTEGER DEFAULT 0,
     notions_inv_count INTEGER DEFAULT 0,
-    price DECIMAL(10, 3) NOT NULL,
+    price NUMERIC(14, 4) NOT NULL,
-    regular_price DECIMAL(10, 3) NOT NULL,
+    regular_price NUMERIC(14, 4) NOT NULL,
-    cost_price DECIMAL(10, 3),
+    cost_price NUMERIC(14, 4),
-    landing_cost_price DECIMAL(10, 3),
+    barcode TEXT,
-    barcode VARCHAR(50),
+    harmonized_tariff_code TEXT,
-    harmonized_tariff_code VARCHAR(20),
     updated_at TIMESTAMP WITH TIME ZONE,
     visible BOOLEAN DEFAULT true,
     managing_stock BOOLEAN DEFAULT true,
     replenishable BOOLEAN DEFAULT true,
-    vendor VARCHAR(100),
+    vendor TEXT,
-    vendor_reference VARCHAR(100),
+    vendor_reference TEXT,
-    notions_reference VARCHAR(100),
+    notions_reference TEXT,
-    permalink VARCHAR(255),
+    permalink TEXT,
     categories TEXT,
-    image VARCHAR(255),
+    image TEXT,
-    image_175 VARCHAR(255),
+    image_175 TEXT,
-    image_full VARCHAR(255),
+    image_full TEXT,
-    brand VARCHAR(100),
+    brand TEXT,
-    line VARCHAR(100),
+    line TEXT,
-    subline VARCHAR(100),
+    subline TEXT,
-    artist VARCHAR(100),
+    artist TEXT,
     options TEXT,
     tags TEXT,
     moq INTEGER DEFAULT 1,
     uom INTEGER DEFAULT 1,
-    rating DECIMAL(10,2) DEFAULT 0.00,
+    rating NUMERIC(14, 4) DEFAULT 0.00,
     reviews INTEGER DEFAULT 0,
-    weight DECIMAL(10,3),
+    weight NUMERIC(14, 4),
-    length DECIMAL(10,3),
+    length NUMERIC(14, 4),
-    width DECIMAL(10,3),
+    width NUMERIC(14, 4),
-    height DECIMAL(10,3),
+    height NUMERIC(14, 4),
-    country_of_origin VARCHAR(5),
+    country_of_origin TEXT,
-    location VARCHAR(50),
+    location TEXT,
     total_sold INTEGER DEFAULT 0,
     baskets INTEGER DEFAULT 0,
     notifies INTEGER DEFAULT 0,
     date_last_sold DATE,
+    shop_score NUMERIC(10, 2) DEFAULT 0,
     updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (pid)
 );
@@ -69,25 +75,25 @@ CREATE TRIGGER update_products_updated
     EXECUTE FUNCTION update_updated_column();
 
 -- Create indexes for products table
-CREATE INDEX idx_products_sku ON products(SKU);
+CREATE INDEX idx_products_sku ON products(sku);
 CREATE INDEX idx_products_vendor ON products(vendor);
 CREATE INDEX idx_products_brand ON products(brand);
-CREATE INDEX idx_products_location ON products(location);
+CREATE INDEX idx_products_visible ON products(visible);
-CREATE INDEX idx_products_total_sold ON products(total_sold);
+CREATE INDEX idx_products_replenishable ON products(replenishable);
-CREATE INDEX idx_products_date_last_sold ON products(date_last_sold);
 CREATE INDEX idx_products_updated ON products(updated);
 
 -- Create categories table with hierarchy support
 CREATE TABLE categories (
     cat_id BIGINT PRIMARY KEY,
-    name VARCHAR(100) NOT NULL,
+    name TEXT NOT NULL,
     type SMALLINT NOT NULL,
     parent_id BIGINT,
     description TEXT,
     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    status VARCHAR(20) DEFAULT 'active',
+    updated TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY (parent_id) REFERENCES categories(cat_id)
+    status TEXT DEFAULT 'active',
+    FOREIGN KEY (parent_id) REFERENCES categories(cat_id) ON DELETE SET NULL
 );
 
 -- Create trigger for categories
@@ -101,6 +107,7 @@ COMMENT ON COLUMN categories.type IS '10=section, 11=category, 12=subcategory, 1
 CREATE INDEX idx_categories_parent ON categories(parent_id);
 CREATE INDEX idx_categories_type ON categories(type);
 CREATE INDEX idx_categories_status ON categories(status);
+CREATE INDEX idx_categories_name ON categories(name);
 CREATE INDEX idx_categories_name_type ON categories(name, type);
 
 -- Create product_categories junction table
@@ -113,28 +120,28 @@ CREATE TABLE product_categories (
 );
 
 CREATE INDEX idx_product_categories_category ON product_categories(cat_id);
-CREATE INDEX idx_product_categories_product ON product_categories(pid);
 
 -- Create orders table with its indexes
 CREATE TABLE orders (
     id BIGSERIAL PRIMARY KEY,
-    order_number VARCHAR(50) NOT NULL,
+    order_number TEXT NOT NULL,
     pid BIGINT NOT NULL,
-    SKU VARCHAR(50) NOT NULL,
+    sku TEXT NOT NULL,
-    date DATE NOT NULL,
+    date TIMESTAMP WITH TIME ZONE NOT NULL,
-    price DECIMAL(10,3) NOT NULL,
+    price NUMERIC(14, 4) NOT NULL,
     quantity INTEGER NOT NULL,
-    discount DECIMAL(10,3) DEFAULT 0.000,
+    discount NUMERIC(14, 4) DEFAULT 0.0000,
-    tax DECIMAL(10,3) DEFAULT 0.000,
+    tax NUMERIC(14, 4) DEFAULT 0.0000,
     tax_included BOOLEAN DEFAULT false,
-    shipping DECIMAL(10,3) DEFAULT 0.000,
+    shipping NUMERIC(14, 4) DEFAULT 0.0000,
-    costeach DECIMAL(10,3) DEFAULT 0.000,
+    costeach NUMERIC(14, 4) DEFAULT 0.0000,
-    customer VARCHAR(50) NOT NULL,
+    customer TEXT NOT NULL,
-    customer_name VARCHAR(100),
+    customer_name TEXT,
-    status VARCHAR(20) DEFAULT 'pending',
+    status TEXT DEFAULT 'pending',
     canceled BOOLEAN DEFAULT false,
     updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    UNIQUE (order_number, pid)
+    UNIQUE (order_number, pid),
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE RESTRICT
 );
 
 -- Create trigger for orders
@@ -145,36 +152,34 @@ CREATE TRIGGER update_orders_updated
 
 CREATE INDEX idx_orders_number ON orders(order_number);
 CREATE INDEX idx_orders_pid ON orders(pid);
+CREATE INDEX idx_orders_sku ON orders(sku);
 CREATE INDEX idx_orders_customer ON orders(customer);
 CREATE INDEX idx_orders_date ON orders(date);
 CREATE INDEX idx_orders_status ON orders(status);
-CREATE INDEX idx_orders_metrics ON orders(pid, date, canceled);
+CREATE INDEX idx_orders_pid_date ON orders(pid, date);
 CREATE INDEX idx_orders_updated ON orders(updated);
 
 -- Create purchase_orders table with its indexes
+-- This table now focuses solely on purchase order intent, not receivings
 CREATE TABLE purchase_orders (
     id BIGSERIAL PRIMARY KEY,
-    po_id VARCHAR(50) NOT NULL,
+    po_id TEXT NOT NULL,
-    vendor VARCHAR(100) NOT NULL,
+    vendor TEXT NOT NULL,
-    date DATE NOT NULL,
+    date TIMESTAMP WITH TIME ZONE NOT NULL,
     expected_date DATE,
     pid BIGINT NOT NULL,
-    sku VARCHAR(50) NOT NULL,
+    sku TEXT NOT NULL,
-    name VARCHAR(100) NOT NULL,
+    name TEXT NOT NULL,
-    cost_price DECIMAL(10, 3) NOT NULL,
+    po_cost_price NUMERIC(14, 4) NOT NULL,
-    po_cost_price DECIMAL(10, 3) NOT NULL,
+    status TEXT DEFAULT 'created',
-    status SMALLINT DEFAULT 1,
-    receiving_status SMALLINT DEFAULT 1,
     notes TEXT,
     long_note TEXT,
     ordered INTEGER NOT NULL,
-    received INTEGER DEFAULT 0,
+    supplier_id INTEGER,
-    received_date DATE,
+    date_created TIMESTAMP WITH TIME ZONE,
-    last_received_date DATE,
+    date_ordered TIMESTAMP WITH TIME ZONE,
-    received_by VARCHAR(100),
-    receiving_history JSONB,
     updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY (pid) REFERENCES products(pid),
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
     UNIQUE (po_id, pid)
 );
 
@@ -185,22 +190,116 @@ CREATE TRIGGER update_purchase_orders_updated
     EXECUTE FUNCTION update_updated_column();
 
 COMMENT ON COLUMN purchase_orders.name IS 'Product name from products.description';
-COMMENT ON COLUMN purchase_orders.po_cost_price IS 'Original cost from PO, before receiving adjustments';
+COMMENT ON COLUMN purchase_orders.po_cost_price IS 'Original cost from PO';
-COMMENT ON COLUMN purchase_orders.status IS '0=canceled,1=created,10=electronically_ready_send,11=ordered,12=preordered,13=electronically_sent,15=receiving_started,50=done';
+COMMENT ON COLUMN purchase_orders.status IS 'canceled, created, electronically_ready_send, ordered, preordered, electronically_sent, receiving_started, done';
-COMMENT ON COLUMN purchase_orders.receiving_status IS '0=canceled,1=created,30=partial_received,40=full_received,50=paid';
-COMMENT ON COLUMN purchase_orders.receiving_history IS 'Array of receiving records with qty, date, cost, receiving_id, and alt_po flag';
 
 CREATE INDEX idx_po_id ON purchase_orders(po_id);
+CREATE INDEX idx_po_sku ON purchase_orders(sku);
 CREATE INDEX idx_po_vendor ON purchase_orders(vendor);
 CREATE INDEX idx_po_status ON purchase_orders(status);
-CREATE INDEX idx_po_receiving_status ON purchase_orders(receiving_status);
+CREATE INDEX idx_po_expected_date ON purchase_orders(expected_date);
-CREATE INDEX idx_po_metrics ON purchase_orders(pid, date, status, ordered, received);
+CREATE INDEX idx_po_pid_status ON purchase_orders(pid, status);
-CREATE INDEX idx_po_metrics_receiving ON purchase_orders(pid, date, receiving_status, received_date);
+CREATE INDEX idx_po_pid_date ON purchase_orders(pid, date);
-CREATE INDEX idx_po_product_date ON purchase_orders(pid, date);
-CREATE INDEX idx_po_product_status ON purchase_orders(pid, status);
 CREATE INDEX idx_po_updated ON purchase_orders(updated);
+CREATE INDEX idx_po_supplier_id ON purchase_orders(supplier_id);
+
+-- Create receivings table to track actual receipt of goods
+CREATE TABLE receivings (
+    id BIGSERIAL PRIMARY KEY,
+    receiving_id TEXT NOT NULL,
+    pid BIGINT NOT NULL,
+    sku TEXT NOT NULL,
+    name TEXT NOT NULL,
+    vendor TEXT,
+    qty_each INTEGER NOT NULL,
+    qty_each_orig INTEGER,
+    cost_each NUMERIC(14, 5) NOT NULL,
+    cost_each_orig NUMERIC(14, 5),
+    received_by INTEGER,
+    received_by_name TEXT,
+    received_date TIMESTAMP WITH TIME ZONE NOT NULL,
+    receiving_created_date TIMESTAMP WITH TIME ZONE,
+    supplier_id INTEGER,
+    status TEXT DEFAULT 'created',
+    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
+    UNIQUE (receiving_id, pid)
+);
+
+-- Create trigger for receivings
+CREATE TRIGGER update_receivings_updated
+    BEFORE UPDATE ON receivings
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_column();
+
+COMMENT ON COLUMN receivings.status IS 'canceled, created, partial_received, full_received, paid';
+COMMENT ON COLUMN receivings.qty_each_orig IS 'Original quantity from the source system';
+COMMENT ON COLUMN receivings.cost_each_orig IS 'Original cost from the source system';
+COMMENT ON COLUMN receivings.vendor IS 'Vendor name, same as in purchase_orders';
+
+CREATE INDEX idx_receivings_id ON receivings(receiving_id);
+CREATE INDEX idx_receivings_pid ON receivings(pid);
+CREATE INDEX idx_receivings_sku ON receivings(sku);
+CREATE INDEX idx_receivings_status ON receivings(status);
+CREATE INDEX idx_receivings_received_date ON receivings(received_date);
+CREATE INDEX idx_receivings_supplier_id ON receivings(supplier_id);
+CREATE INDEX idx_receivings_vendor ON receivings(vendor);
+CREATE INDEX idx_receivings_updated ON receivings(updated);
 
 SET session_replication_role = 'origin';  -- Re-enable foreign key checks
 
 -- Create views for common calculations
--- product_sales_trends view moved to metrics-schema.sql 
+-- product_sales_trends view moved to metrics-schema.sql 
+
+-- -- Historical data tables imported from production
+-- CREATE TABLE imported_product_current_prices (
+--   price_id BIGSERIAL PRIMARY KEY,
+--   pid BIGINT NOT NULL,
+--   qty_buy SMALLINT NOT NULL,
+--   is_min_qty_buy BOOLEAN NOT NULL,
+--   price_each NUMERIC(10,3) NOT NULL,
+--   qty_limit SMALLINT NOT NULL,
+--   no_promo BOOLEAN NOT NULL,
+--   checkout_offer BOOLEAN NOT NULL,
+--   active BOOLEAN NOT NULL,
+--   date_active TIMESTAMP WITH TIME ZONE,
+--   date_deactive TIMESTAMP WITH TIME ZONE,
+--   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
+-- );
+
+-- CREATE INDEX idx_imported_product_current_prices_pid ON imported_product_current_prices(pid, active, qty_buy);
+-- CREATE INDEX idx_imported_product_current_prices_checkout ON imported_product_current_prices(checkout_offer, active);
+-- CREATE INDEX idx_imported_product_current_prices_deactive ON imported_product_current_prices(date_deactive, active);
+-- CREATE INDEX idx_imported_product_current_prices_active ON imported_product_current_prices(date_active, active);
+
+-- CREATE TABLE imported_daily_inventory (
+--   date DATE NOT NULL,
+--   pid BIGINT NOT NULL,
+--   amountsold SMALLINT NOT NULL DEFAULT 0,
+--   times_sold SMALLINT NOT NULL DEFAULT 0,
+--   qtyreceived SMALLINT NOT NULL DEFAULT 0,
+--   price NUMERIC(7,2) NOT NULL DEFAULT 0,
+--   costeach NUMERIC(7,2) NOT NULL DEFAULT 0,
+--   stamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+--   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+--   PRIMARY KEY (date, pid)
+-- );
+
+-- CREATE INDEX idx_imported_daily_inventory_pid ON imported_daily_inventory(pid);
+
+-- CREATE TABLE imported_product_stat_history (
+--   pid BIGINT NOT NULL,
+--   date DATE NOT NULL,
+--   score NUMERIC(10,2) NOT NULL,
+--   score2 NUMERIC(10,2) NOT NULL,
+--   qty_in_baskets SMALLINT NOT NULL,
+--   qty_sold SMALLINT NOT NULL,
+--   notifies_set SMALLINT NOT NULL,
+--   visibility_score NUMERIC(10,2) NOT NULL,
+--   health_score VARCHAR(5) NOT NULL,
+--   sold_view_score NUMERIC(6,3) NOT NULL,
+--   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+--   PRIMARY KEY (pid, date)
+-- );
+
+-- CREATE INDEX idx_imported_product_stat_history_date ON imported_product_stat_history(date); 

inventory-server/db/setup-schema.sql

@@ -49,6 +49,30 @@ CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_system_prompt
 ON ai_prompts (prompt_type) 
 WHERE prompt_type = 'system';
 
+-- Reusable Images table for storing persistent images
+CREATE TABLE IF NOT EXISTS reusable_images (
+    id SERIAL PRIMARY KEY,
+    name TEXT NOT NULL,
+    filename TEXT NOT NULL,
+    file_path TEXT NOT NULL,
+    image_url TEXT NOT NULL,
+    is_global BOOLEAN NOT NULL DEFAULT false,
+    company TEXT,
+    mime_type TEXT,
+    file_size INTEGER,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    CONSTRAINT company_required_for_non_global CHECK (
+        (is_global = true AND company IS NULL) OR
+        (is_global = false AND company IS NOT NULL)
+    )
+);
+
+-- Create index on company for efficient querying
+CREATE INDEX IF NOT EXISTS idx_reusable_images_company ON reusable_images(company);
+-- Create index on is_global for efficient querying
+CREATE INDEX IF NOT EXISTS idx_reusable_images_is_global ON reusable_images(is_global);
+
 -- AI Validation Performance Tracking
 CREATE TABLE IF NOT EXISTS ai_validation_performance (
     id SERIAL PRIMARY KEY,
@@ -82,4 +106,10 @@ CREATE TRIGGER update_templates_updated_at
 CREATE TRIGGER update_ai_prompts_updated_at
     BEFORE UPDATE ON ai_prompts
     FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+    
+-- Trigger to automatically update the updated_at column for reusable_images
+CREATE TRIGGER update_reusable_images_updated_at
+    BEFORE UPDATE ON reusable_images
+    FOR EACH ROW
     EXECUTE FUNCTION update_updated_at_column(); 

inventory-server/migrations/001_create_import_sessions.sql

@@ -0,0 +1,29 @@
+-- Migration: Create import_sessions table
+-- Run this against your PostgreSQL database
+
+CREATE TABLE IF NOT EXISTS import_sessions (
+  id SERIAL PRIMARY KEY,
+  user_id INTEGER NOT NULL,
+  name VARCHAR(255),                    -- NULL for unnamed/autosave sessions
+  current_step VARCHAR(50) NOT NULL,    -- 'validation' | 'imageUpload'
+  data JSONB NOT NULL,                  -- Product rows
+  product_images JSONB,                 -- Image assignments
+  global_selections JSONB,              -- Supplier, company, line, subline
+  validation_state JSONB,               -- Errors, UPC status, generated item numbers
+  created_at TIMESTAMP DEFAULT NOW(),
+  updated_at TIMESTAMP DEFAULT NOW()
+);
+
+-- Ensure only one unnamed session per user (autosave slot)
+CREATE UNIQUE INDEX IF NOT EXISTS idx_unnamed_session_per_user
+  ON import_sessions (user_id)
+  WHERE name IS NULL;
+
+-- Index for fast user lookups
+CREATE INDEX IF NOT EXISTS idx_import_sessions_user_id
+  ON import_sessions (user_id);
+
+-- Add comment for documentation
+COMMENT ON TABLE import_sessions IS 'Stores in-progress product import sessions for users';
+COMMENT ON COLUMN import_sessions.name IS 'Session name - NULL indicates the single unnamed/autosave session per user';
+COMMENT ON COLUMN import_sessions.current_step IS 'Which step the user was on: validation or imageUpload';

inventory-server/migrations/002_ai_prompts_extensible_types.sql

@@ -0,0 +1,57 @@
+-- Migration: Make AI prompts extensible with is_singleton column
+-- Date: 2024-01-19
+-- Description: Removes hardcoded prompt_type CHECK constraint, adds is_singleton column
+--              for dynamic uniqueness enforcement, and creates appropriate indexes.
+
+-- 1. Drop the old CHECK constraints on prompt_type (allows any string value now)
+ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS ai_prompts_prompt_type_check;
+ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS company_required_for_specific;
+
+-- 2. Add is_singleton column (defaults to true for backwards compatibility)
+ALTER TABLE ai_prompts ADD COLUMN IF NOT EXISTS is_singleton BOOLEAN NOT NULL DEFAULT true;
+
+-- 3. Drop ALL old unique constraints and indexes (cleanup)
+-- Some were created as CONSTRAINTS (via ADD CONSTRAINT), others as standalone indexes
+-- Must drop constraints first, then remaining standalone indexes
+
+-- Drop constraints (these also remove their backing indexes)
+ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS unique_company_prompt;
+ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS idx_unique_general_prompt;
+ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS idx_unique_system_prompt;
+
+-- Drop standalone indexes (IF EXISTS handles cases where they don't exist)
+DROP INDEX IF EXISTS idx_unique_general_prompt;
+DROP INDEX IF EXISTS idx_unique_system_prompt;
+DROP INDEX IF EXISTS idx_unique_name_validation_system;
+DROP INDEX IF EXISTS idx_unique_name_validation_general;
+DROP INDEX IF EXISTS idx_unique_description_validation_system;
+DROP INDEX IF EXISTS idx_unique_description_validation_general;
+DROP INDEX IF EXISTS idx_unique_sanity_check_system;
+DROP INDEX IF EXISTS idx_unique_sanity_check_general;
+DROP INDEX IF EXISTS idx_unique_bulk_validation_system;
+DROP INDEX IF EXISTS idx_unique_bulk_validation_general;
+DROP INDEX IF EXISTS idx_unique_name_validation_company;
+DROP INDEX IF EXISTS idx_unique_description_validation_company;
+DROP INDEX IF EXISTS idx_unique_bulk_validation_company;
+
+-- 4. Create new partial unique indexes based on is_singleton
+-- For singleton types WITHOUT company (only one per prompt_type)
+CREATE UNIQUE INDEX IF NOT EXISTS idx_singleton_no_company
+  ON ai_prompts (prompt_type)
+  WHERE is_singleton = true AND company IS NULL;
+
+-- For singleton types WITH company (only one per prompt_type + company combination)
+CREATE UNIQUE INDEX IF NOT EXISTS idx_singleton_with_company
+  ON ai_prompts (prompt_type, company)
+  WHERE is_singleton = true AND company IS NOT NULL;
+
+-- 5. Add index for fast lookups by type
+CREATE INDEX IF NOT EXISTS idx_prompt_type ON ai_prompts (prompt_type);
+
+-- NOTE: After running this migration, you should:
+-- 1. Delete existing prompts with old types (general, system, company_specific)
+-- 2. Create new prompts with the new type naming convention:
+--    - name_validation_system, name_validation_general, name_validation_company_specific
+--    - description_validation_system, description_validation_general, description_validation_company_specific
+--    - sanity_check_system, sanity_check_general
+--    - bulk_validation_system, bulk_validation_general, bulk_validation_company_specific

inventory-server/migrations/003_create_import_audit_log.sql

@@ -0,0 +1,53 @@
+-- Migration: Create import_audit_log table
+-- Permanent audit trail of all product import submissions sent to the API
+-- Run this against your PostgreSQL database
+
+CREATE TABLE IF NOT EXISTS import_audit_log (
+  id SERIAL PRIMARY KEY,
+
+  -- Who initiated the import
+  user_id INTEGER NOT NULL,
+  username VARCHAR(255),
+
+  -- What was submitted
+  product_count INTEGER NOT NULL,
+  request_payload JSONB NOT NULL,           -- The exact JSON array of products sent to the API
+  environment VARCHAR(10) NOT NULL,         -- 'dev' or 'prod'
+  target_endpoint VARCHAR(255),             -- The API URL that was called
+  use_test_data_source BOOLEAN DEFAULT FALSE,
+
+  -- What came back
+  success BOOLEAN NOT NULL,
+  response_payload JSONB,                   -- Full API response
+  error_message TEXT,                       -- Extracted error message on failure
+  created_count INTEGER DEFAULT 0,          -- Number of products successfully created
+  errored_count INTEGER DEFAULT 0,          -- Number of products that errored
+
+  -- Metadata
+  session_id INTEGER,                       -- Optional link to the import_session used (if any)
+  duration_ms INTEGER,                      -- How long the API call took
+  created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
+);
+
+-- Index for looking up logs by user
+CREATE INDEX IF NOT EXISTS idx_import_audit_log_user_id
+  ON import_audit_log (user_id);
+
+-- Index for filtering by success/failure
+CREATE INDEX IF NOT EXISTS idx_import_audit_log_success
+  ON import_audit_log (success);
+
+-- Index for time-based queries
+CREATE INDEX IF NOT EXISTS idx_import_audit_log_created_at
+  ON import_audit_log (created_at DESC);
+
+-- Composite index for user + time queries
+CREATE INDEX IF NOT EXISTS idx_import_audit_log_user_created
+  ON import_audit_log (user_id, created_at DESC);
+
+COMMENT ON TABLE import_audit_log IS 'Permanent audit log of all product import API submissions';
+COMMENT ON COLUMN import_audit_log.request_payload IS 'Exact JSON products array sent to the external API';
+COMMENT ON COLUMN import_audit_log.response_payload IS 'Full response received from the external API';
+COMMENT ON COLUMN import_audit_log.environment IS 'dev or prod - which API endpoint was targeted';
+COMMENT ON COLUMN import_audit_log.session_id IS 'Optional reference to import_sessions.id if session was active';
+COMMENT ON COLUMN import_audit_log.duration_ms IS 'Round-trip time of the API call in milliseconds';

inventory-server/migrations/004_create_product_editor_audit_log.sql

@@ -0,0 +1,54 @@
+-- Migration: Create product_editor_audit_log table
+-- Permanent audit trail of all product editor API submissions
+-- Run this against your PostgreSQL database
+
+CREATE TABLE IF NOT EXISTS product_editor_audit_log (
+  id SERIAL PRIMARY KEY,
+
+  -- Who made the edit
+  user_id INTEGER NOT NULL,
+  username VARCHAR(255),
+
+  -- Which product
+  pid INTEGER NOT NULL,
+
+  -- What was submitted
+  action VARCHAR(50) NOT NULL,               -- 'product_edit', 'image_changes', 'taxonomy_set'
+  request_payload JSONB NOT NULL,            -- The exact payload sent to the external API
+  target_endpoint VARCHAR(255),              -- The API URL that was called
+
+  -- What came back
+  success BOOLEAN NOT NULL,
+  response_payload JSONB,                    -- Full API response
+  error_message TEXT,                        -- Extracted error message on failure
+
+  -- Metadata
+  duration_ms INTEGER,                       -- How long the API call took
+  created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
+);
+
+-- Index for looking up edits by product
+CREATE INDEX IF NOT EXISTS idx_pe_audit_log_pid
+  ON product_editor_audit_log (pid);
+
+-- Index for looking up edits by user
+CREATE INDEX IF NOT EXISTS idx_pe_audit_log_user_id
+  ON product_editor_audit_log (user_id);
+
+-- Index for time-based queries
+CREATE INDEX IF NOT EXISTS idx_pe_audit_log_created_at
+  ON product_editor_audit_log (created_at DESC);
+
+-- Composite index for product + time queries
+CREATE INDEX IF NOT EXISTS idx_pe_audit_log_pid_created
+  ON product_editor_audit_log (pid, created_at DESC);
+
+-- Composite index for user + time queries
+CREATE INDEX IF NOT EXISTS idx_pe_audit_log_user_created
+  ON product_editor_audit_log (user_id, created_at DESC);
+
+COMMENT ON TABLE product_editor_audit_log IS 'Permanent audit log of all product editor API submissions';
+COMMENT ON COLUMN product_editor_audit_log.action IS 'Type of edit: product_edit, image_changes, or taxonomy_set';
+COMMENT ON COLUMN product_editor_audit_log.request_payload IS 'Exact payload sent to the external API';
+COMMENT ON COLUMN product_editor_audit_log.response_payload IS 'Full response received from the external API';
+COMMENT ON COLUMN product_editor_audit_log.duration_ms IS 'Round-trip time of the API call in milliseconds';

inventory-server/migrations/005_phase6_permission_codes.sql

@@ -0,0 +1,52 @@
+-- Phase 6.2: per-route permission codes
+-- Seeds the permission codes referenced by Phase 6 hardening middleware.
+-- Safe to run multiple times (ON CONFLICT DO NOTHING).
+--
+-- Codes follow the plan's spec (CONSOLIDATION_PLAN.md §6.2):
+--   product_import    — POST/PUT/DELETE on /api/import
+--   data_management   — POST/PUT/DELETE on /api/csv (data-management.js)
+--   ai_admin          — POST/PUT/DELETE on /api/ai-prompts, /api/ai-validation
+--   templates_write   — POST/PUT/DELETE on /api/templates
+--   image_admin       — POST/DELETE on /api/reusable-images
+--   audit_read        — reserved for future read-gating on audit logs
+--   acot_admin        — reserved for acot-server (Phase 5 scope)
+--   klaviyo_* / meta_* / google_* / typeform_*  — reserved for dashboard-server (Phase 4 scope)
+--
+-- Admin users (is_admin = true) automatically pass any requirePermission() check,
+-- so this migration does not auto-grant codes to admins. New non-admin users get
+-- write access only when explicitly granted via the user-management UI.
+
+INSERT INTO permissions (code, name, category, description) VALUES
+  ('product_import', 'Product Import (write)', 'Imports',
+   'Allows POST/PUT/DELETE on /api/import — uploads, deletes, generate-upc, etc.'),
+  ('data_management', 'Data Management (write)', 'Data',
+   'Allows POST/PUT/DELETE on /api/csv — CSV operations, full updates, full resets.'),
+  ('ai_admin', 'AI Settings Admin', 'AI',
+   'Allows write access to AI prompts and AI validation endpoints.'),
+  ('templates_write', 'Template Editing', 'Templates',
+   'Allows POST/PUT/DELETE on /api/templates.'),
+  ('image_admin', 'Image Management', 'Images',
+   'Allows uploads and deletions on /api/reusable-images.'),
+  ('audit_read', 'Audit Log Access', 'Audit',
+   'Reserved for future read-gating of import + product-editor audit logs.'),
+  ('klaviyo_write', 'Klaviyo Write', 'Dashboard',
+   'Reserved for dashboard-server: mutates Klaviyo lists/segments.'),
+  ('klaviyo_admin', 'Klaviyo Admin', 'Dashboard',
+   'Reserved for dashboard-server: triggers campaign syncs.'),
+  ('meta_write', 'Meta Write', 'Dashboard',
+   'Reserved for dashboard-server: Meta API write operations.'),
+  ('google_write', 'Google Analytics Write', 'Dashboard',
+   'Reserved for dashboard-server: GA write operations.'),
+  ('typeform_write', 'Typeform Write', 'Dashboard',
+   'Reserved for dashboard-server: Typeform write operations.'),
+  ('acot_admin', 'ACOT Server Admin', 'ACOT',
+   'Reserved for acot-server admin endpoints.')
+ON CONFLICT (code) DO NOTHING;
+
+-- Phase 2 deviation #6 cleanup: drop defunct frontend permissions if present.
+-- These corresponded to the removed Aircall/Gorgias dashboards.
+DELETE FROM user_permissions
+ WHERE permission_id IN (
+   SELECT id FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls')
+ );
+DELETE FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls');

inventory-server/migrations/006_permissions_ui_cleanup.sql

@@ -0,0 +1,175 @@
+-- Permissions UI cleanup — 2026-05-28
+--
+-- WHAT THIS DOES
+--   Rewrites permissions.name and permissions.category for clarity.
+--   Consolidates 17 categories down to 10. Renames ambiguous entries so
+--   the User Management UI reads cleanly. Does NOT touch permissions.code,
+--   so every existing route gate (backend requirePermission and frontend
+--   Protected page=) and every row in user_permissions keeps working
+--   without any code change or remapping.
+--
+-- WHAT THIS DOES *NOT* DO
+--   No DROP/DELETE of any permission (except in the optional block at the
+--   very bottom — commented out by default). No changes to permissions.code.
+--   No INSERT of new permissions. No changes to other tables.
+--
+-- SAFETY
+--   Wrapped in a transaction. Run end-to-end; if any row count is wrong,
+--   ROLLBACK and inspect. The auth middleware caches the user's loaded
+--   permissions for 60s — after this runs, names refresh on next cache
+--   miss for the admin UI. user_permissions joins by id, so granted
+--   permissions remain granted across renames.
+
+BEGIN;
+
+------------------------------------------------------------------------
+-- 1. Category consolidation
+------------------------------------------------------------------------
+
+-- 1a. Orphan "Pages" (just the Settings page) → Pages/Settings
+UPDATE permissions SET category = 'Pages/Settings'
+ WHERE code = 'access:settings';
+
+-- 1b. Existing settings:* tab-access permissions stay under a renamed
+--     "Settings Tabs" category (was "Settings") so it reads less ambiguously
+--     next to "Pages/Settings" and "Write Actions".
+UPDATE permissions SET category = 'Settings Tabs'
+ WHERE category = 'Settings';
+
+-- 1c. Dashboard widget perms keep their codes but get a clearer category name.
+UPDATE permissions SET category = 'Dashboard Widgets'
+ WHERE category = 'Dashboard Components';
+
+-- 1d. Collapse the 7 single-member "new permission" categories
+--     (Imports, Data, AI, Templates, Images, Audit, ACOT, Dashboard
+--     [the dashboard-server one — distinct from Pages/Dashboard])
+--     into a single "Write Actions" category.
+UPDATE permissions SET category = 'Write Actions'
+ WHERE category IN ('Imports', 'Data', 'AI', 'Templates',
+                    'Images', 'Audit', 'ACOT', 'Dashboard');
+
+-- 1e. The "Admin" category (just Show Debug) stays as-is — single member
+--     but conceptually distinct enough that bucketing under Write Actions
+--     would muddy the meaning. Leaving it alone.
+
+------------------------------------------------------------------------
+-- 2. Name renames for clarity
+------------------------------------------------------------------------
+
+-- 2a. Distinguish "page that lets you do X" from "permission to do X"
+--     by suffixing the page-access permissions with " (page)" where the
+--     name otherwise collides with the corresponding write permission.
+UPDATE permissions SET name = 'Import Products (page)'
+ WHERE code = 'access:import';
+
+UPDATE permissions SET name = 'Product Editor (page)'
+ WHERE code = 'access:product_editor';
+
+UPDATE permissions SET name = 'Bulk Edit (page)'
+ WHERE code = 'access:bulk_edit';
+
+-- 2b. Settings tab-access perms get a uniform "Settings: X Tab" name so
+--     they sort together and read as "what you're seeing access to" not
+--     "the feature itself."
+UPDATE permissions SET name = 'Settings: Data Management Tab'
+ WHERE code = 'settings:data_management';
+
+UPDATE permissions SET name = 'Settings: Reusable Images Tab'
+ WHERE code = 'settings:library_management';
+
+UPDATE permissions SET name = 'Settings: AI Prompts Tab'
+ WHERE code = 'settings:prompt_management';
+
+UPDATE permissions SET name = 'Settings: Templates Tab'
+ WHERE code = 'settings:templates';
+
+UPDATE permissions SET name = 'Settings: User Management Tab'
+ WHERE code = 'settings:user_management';
+
+UPDATE permissions SET name = 'Settings: Audit Log Tab'
+ WHERE code = 'settings:audit_log';
+
+UPDATE permissions SET name = 'Settings: Global Tab'
+ WHERE code = 'settings:global';
+
+UPDATE permissions SET name = 'Settings: Products Tab'
+ WHERE code = 'settings:products';
+
+UPDATE permissions SET name = 'Settings: Vendors Tab'
+ WHERE code = 'settings:vendors';
+
+-- 2c. Write-action perms get verb-leading names so it's obvious what
+--     granting them actually allows.
+UPDATE permissions
+   SET name = 'Product Import: Upload & Submit',
+       description = 'Allows POST/PUT/DELETE on /api/import — image uploads, '
+                  || 'product submission, deletions, and generate-upc. Does NOT '
+                  || 'grant access to the Import Products page (access:import).'
+ WHERE code = 'product_import';
+
+UPDATE permissions
+   SET name = 'Data Management: Run Operations',
+       description = 'Allows POST/PUT/DELETE on /api/csv — CSV operations, '
+                  || 'full updates, full resets. Does NOT grant access to the '
+                  || 'Data Management settings tab (settings:data_management).'
+ WHERE code = 'data_management';
+
+UPDATE permissions
+   SET name = 'Reusable Images: Upload & Delete',
+       description = 'Allows uploads and deletions on /api/reusable-images. '
+                  || 'Distinct from product_import (which gates uploads inside '
+                  || 'the product import flow).'
+ WHERE code = 'image_admin';
+
+UPDATE permissions
+   SET name = 'Templates: Create & Edit',
+       description = 'Allows POST/PUT/DELETE on /api/templates.'
+ WHERE code = 'templates_write';
+
+UPDATE permissions
+   SET name = 'AI: Edit Prompts & Validation',
+       description = 'Allows write access to /api/ai-prompts and /api/ai-validation.'
+ WHERE code = 'ai_admin';
+
+UPDATE permissions
+   SET name = 'Klaviyo: Clear Cache',
+       description = 'Allows POST /api/klaviyo/events/clearCache.'
+ WHERE code = 'klaviyo_admin';
+
+UPDATE permissions
+   SET name = 'Meta: Mutate Campaigns',
+       description = 'Allows PATCH/POST on /api/meta/campaigns/*.'
+ WHERE code = 'meta_write';
+
+------------------------------------------------------------------------
+-- 3. Verification — should all return non-zero
+------------------------------------------------------------------------
+-- Uncomment to inspect before commit:
+-- SELECT category, COUNT(*) FROM permissions GROUP BY category ORDER BY category;
+-- SELECT code, name, category FROM permissions
+--   WHERE category IN ('Write Actions', 'Settings Tabs', 'Pages/Settings')
+--   ORDER BY category, name;
+
+COMMIT;
+
+------------------------------------------------------------------------
+-- 4. OPTIONAL — drop unused "Reserved for future" codes
+------------------------------------------------------------------------
+-- These five codes are referenced only in their own description ("Reserved
+-- for…") and appear in NO route gate, NO Protected page=, and NO frontend
+-- permissions.includes() check. Verified 2026-05-28.
+--
+-- Run this block separately if you want to drop them. user_permissions has
+-- ON DELETE CASCADE on permission_id is NOT configured (only on user_id),
+-- so we must clear user_permissions rows first.
+--
+-- BEGIN;
+--   DELETE FROM user_permissions
+--    WHERE permission_id IN (SELECT id FROM permissions
+--                             WHERE code IN ('klaviyo_write', 'google_write',
+--                                            'typeform_write', 'acot_admin',
+--                                            'audit_read'));
+--   DELETE FROM permissions
+--    WHERE code IN ('klaviyo_write', 'google_write', 'typeform_write',
+--                   'acot_admin', 'audit_read');
+-- COMMIT;

inventory-server/old/backfill-snapshots.js

@@ -0,0 +1,426 @@
+const path = require('path');
+const fs = require('fs');
+const progress = require('../scripts/metrics-new/utils/progress'); // Assuming progress utils are here
+const { getConnection, closePool } = require('../scripts/metrics-new/utils/db'); // Assuming db utils are here
+const os = require('os'); // For detecting number of CPU cores
+
+// --- Configuration ---
+const BATCH_SIZE_DAYS = 1; // Process 1 day per database function call
+const SQL_FUNCTION_FILE = path.resolve(__dirname, 'backfill_historical_snapshots.sql'); // Correct path
+const LOG_PROGRESS_INTERVAL_MS = 5000; // Update console progress roughly every 5 seconds
+const HISTORY_TYPE = 'backfill_snapshots'; // Identifier for history table
+const MAX_WORKERS = Math.max(1, Math.floor(os.cpus().length / 2)); // Use half of available CPU cores
+const USE_PARALLEL = false; // Set to true to enable parallel processing
+const PG_STATEMENT_TIMEOUT_MS = 1800000; // 30 minutes max per query
+
+// --- Cancellation Handling ---
+let isCancelled = false;
+let runningQueryPromise = null; // To potentially track the active query
+
+function requestCancellation() {
+    if (!isCancelled) {
+        isCancelled = true;
+        console.warn('\nCancellation requested. Finishing current batch then stopping...');
+        // Note: We are NOT forcefully cancelling the backend query anymore.
+    }
+}
+
+process.on('SIGINT', requestCancellation); // Handle Ctrl+C
+process.on('SIGTERM', requestCancellation); // Handle termination signals
+
+// --- Main Backfill Function ---
+async function backfillSnapshots(cmdStartDate, cmdEndDate, cmdStartBatch = 1) {
+    let connection;
+    const overallStartTime = Date.now();
+    let calculateHistoryId = null;
+    let processedDaysTotal = 0; // Track total days processed across all batches executed in this run
+    let currentBatchNum = cmdStartBatch > 0 ? cmdStartBatch : 1;
+    let totalBatches = 0; // Initialize totalBatches
+    let totalDays = 0; // Initialize totalDays
+
+    console.log(`Starting snapshot backfill process...`);
+    console.log(`SQL Function definition file: ${SQL_FUNCTION_FILE}`);
+    if (!fs.existsSync(SQL_FUNCTION_FILE)) {
+         console.error(`FATAL: SQL file not found at ${SQL_FUNCTION_FILE}`);
+         process.exit(1); // Exit early if file doesn't exist
+    }
+
+    try {
+        // Set up a connection with higher memory limits
+        connection = await getConnection({
+            // Add performance-related settings
+            application_name: 'backfill_snapshots',
+            statement_timeout: PG_STATEMENT_TIMEOUT_MS, // 30 min timeout per statement
+            // These parameters may need to be configured in your database:
+            // work_mem: '1GB',
+            // maintenance_work_mem: '2GB',
+            // temp_buffers: '1GB',
+        });
+        
+        console.log('Database connection acquired.');
+
+        // --- Ensure Function Exists ---
+        console.log('Ensuring database function is up-to-date...');
+        try {
+            const sqlFunctionDef = fs.readFileSync(SQL_FUNCTION_FILE, 'utf8');
+            if (!sqlFunctionDef.includes('CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final')) {
+                throw new Error(`SQL file ${SQL_FUNCTION_FILE} does not seem to contain the function definition.`);
+            }
+            await connection.query(sqlFunctionDef); // Execute the whole file
+            console.log('Database function `backfill_daily_snapshots_range_final` created/updated.');
+            
+            // Add performance query hints to the database
+            await connection.query(`
+                -- Analyze tables for better query planning
+                ANALYZE public.products;
+                ANALYZE public.imported_daily_inventory;
+                ANALYZE public.imported_product_stat_history;
+                ANALYZE public.daily_product_snapshots;
+                ANALYZE public.imported_product_current_prices;
+            `).catch(err => {
+                // Non-fatal if analyze fails
+                console.warn('Failed to analyze tables (non-fatal):', err.message);
+            });
+            
+        } catch (err) {
+            console.error(`Error processing SQL function file ${SQL_FUNCTION_FILE}:`, err);
+            throw new Error(`Failed to create or replace DB function: ${err.message}`);
+        }
+
+        // --- Prepare History Record ---
+        console.log('Preparing calculation history record...');
+         // Ensure history table exists (optional, could be done elsewhere)
+         await connection.query(`
+            CREATE TABLE IF NOT EXISTS public.calculate_history (
+                id SERIAL PRIMARY KEY,
+                start_time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+                end_time TIMESTAMPTZ,
+                duration_seconds INTEGER,
+                status VARCHAR(20) NOT NULL, -- e.g., 'running', 'completed', 'failed', 'cancelled'
+                error_message TEXT,
+                additional_info JSONB -- Store type, file, batch info etc.
+            );
+         `);
+        // Mark previous runs of this type as potentially failed if they were left 'running'
+        await connection.query(`
+            UPDATE public.calculate_history
+            SET status = 'failed', error_message = 'Interrupted by new run.'
+            WHERE status = 'running' AND additional_info->>'type' = $1;
+        `, [HISTORY_TYPE]);
+
+        // Create new history record
+        const historyResult = await connection.query(`
+            INSERT INTO public.calculate_history (start_time, status, additional_info)
+            VALUES (NOW(), 'running', jsonb_build_object('type', $1::text, 'sql_file', $2::text, 'start_batch', $3::integer))
+            RETURNING id;
+        `, [HISTORY_TYPE, path.basename(SQL_FUNCTION_FILE), cmdStartBatch]);
+        calculateHistoryId = historyResult.rows[0].id;
+        console.log(`Calculation history record created with ID: ${calculateHistoryId}`);
+
+
+        // --- Determine Date Range ---
+        console.log('Determining date range...');
+        let effectiveStartDate, effectiveEndDate;
+
+        // Use command-line dates if provided, otherwise query DB
+        if (cmdStartDate) {
+             effectiveStartDate = cmdStartDate;
+        } else {
+             const minDateResult = await connection.query(`
+                SELECT LEAST(
+                    COALESCE((SELECT MIN(date) FROM public.imported_daily_inventory WHERE date > '1970-01-01'), CURRENT_DATE),
+                    COALESCE((SELECT MIN(date) FROM public.imported_product_stat_history WHERE date > '1970-01-01'), CURRENT_DATE)
+                )::date as min_date;
+            `);
+             effectiveStartDate = minDateResult.rows[0]?.min_date || new Date().toISOString().split('T')[0]; // Fallback
+             console.log(`Auto-detected start date: ${effectiveStartDate}`);
+        }
+
+        if (cmdEndDate) {
+            effectiveEndDate = cmdEndDate;
+        } else {
+             const maxDateResult = await connection.query(`
+                 SELECT GREATEST(
+                    COALESCE((SELECT MAX(date) FROM public.imported_daily_inventory WHERE date < CURRENT_DATE), '1970-01-01'::date),
+                    COALESCE((SELECT MAX(date) FROM public.imported_product_stat_history WHERE date < CURRENT_DATE), '1970-01-01'::date)
+                )::date as max_date;
+             `);
+             // Ensure end date is not today or in the future
+             effectiveEndDate = maxDateResult.rows[0]?.max_date || new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Default yesterday
+             if (new Date(effectiveEndDate) >= new Date(new Date().toISOString().split('T')[0])) {
+                 effectiveEndDate = new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Set to yesterday if >= today
+             }
+             console.log(`Auto-detected end date: ${effectiveEndDate}`);
+        }
+
+         // Validate dates
+         const dStart = new Date(effectiveStartDate);
+         const dEnd = new Date(effectiveEndDate);
+         if (isNaN(dStart.getTime()) || isNaN(dEnd.getTime()) || dStart > dEnd) {
+             throw new Error(`Invalid date range: Start "${effectiveStartDate}", End "${effectiveEndDate}"`);
+         }
+
+        // --- Batch Processing ---
+        totalDays = Math.ceil((dEnd - dStart) / (1000 * 60 * 60 * 24)) + 1; // Inclusive
+        totalBatches = Math.ceil(totalDays / BATCH_SIZE_DAYS);
+
+        console.log(`Target Date Range: ${effectiveStartDate} to ${effectiveEndDate} (${totalDays} days)`);
+        console.log(`Total Batches: ${totalBatches} (Batch Size: ${BATCH_SIZE_DAYS} days)`);
+        console.log(`Starting from Batch: ${currentBatchNum}`);
+
+        // Initial progress update
+        progress.outputProgress({ 
+            status: 'running', 
+            operation: 'Starting Batch Processing',
+            currentBatch: currentBatchNum, 
+            totalBatches: totalBatches, 
+            totalDays: totalDays,
+            elapsed: '0s', 
+            remaining: 'Calculating...', 
+            rate: 0,
+            historyId: calculateHistoryId // Include history ID in the object
+        });
+
+        while (currentBatchNum <= totalBatches && !isCancelled) {
+            const batchOffset = (currentBatchNum - 1) * BATCH_SIZE_DAYS;
+            const batchStartDate = new Date(dStart);
+            batchStartDate.setDate(dStart.getDate() + batchOffset);
+
+            const batchEndDate = new Date(batchStartDate);
+            batchEndDate.setDate(batchStartDate.getDate() + BATCH_SIZE_DAYS - 1);
+
+            // Clamp batch end date to the overall effective end date
+            if (batchEndDate > dEnd) {
+                batchEndDate.setTime(dEnd.getTime());
+            }
+
+            const batchStartDateStr = batchStartDate.toISOString().split('T')[0];
+            const batchEndDateStr = batchEndDate.toISOString().split('T')[0];
+            const batchStartTime = Date.now();
+
+            console.log(`\n--- Processing Batch ${currentBatchNum} / ${totalBatches} ---`);
+            console.log(`    Dates: ${batchStartDateStr} to ${batchEndDateStr}`);
+
+            // Execute the function for the batch
+             try {
+                progress.outputProgress({
+                     status: 'running', 
+                     operation: `Executing DB function for batch ${currentBatchNum}...`,
+                     currentBatch: currentBatchNum, 
+                     totalBatches: totalBatches, 
+                     totalDays: totalDays,
+                     elapsed: progress.formatElapsedTime(overallStartTime), 
+                     remaining: 'Executing...', 
+                     rate: 0,
+                     historyId: calculateHistoryId
+                 });
+
+                // Performance improvement: Add batch processing hint
+                await connection.query('SET LOCAL enable_parallel_append = on; SET LOCAL enable_parallel_hash = on; SET LOCAL max_parallel_workers_per_gather = 4;');
+
+                // Store promise in case we need to try and cancel (though not implemented forcefully)
+                runningQueryPromise = connection.query(
+                    `SELECT backfill_daily_snapshots_range_final($1::date, $2::date);`,
+                    [batchStartDateStr, batchEndDateStr]
+                );
+                await runningQueryPromise; // Wait for the function call to complete
+                runningQueryPromise = null; // Clear the promise
+
+                const batchDurationMs = Date.now() - batchStartTime;
+                const daysInThisBatch = Math.ceil((batchEndDate - batchStartDate) / (1000 * 60 * 60 * 24)) + 1;
+                processedDaysTotal += daysInThisBatch;
+
+                console.log(`    Batch ${currentBatchNum} completed in ${progress.formatElapsedTime(batchStartTime)}.`);
+
+                // --- Update Progress & History ---
+                const overallElapsedSec = Math.round((Date.now() - overallStartTime) / 1000);
+                progress.outputProgress({
+                    status: 'running', 
+                    operation: `Completed batch ${currentBatchNum}`,
+                    currentBatch: currentBatchNum, 
+                    totalBatches: totalBatches, 
+                    totalDays: totalDays,
+                    processedDays: processedDaysTotal,
+                    elapsed: progress.formatElapsedTime(overallStartTime),
+                    remaining: progress.estimateRemaining(overallStartTime, processedDaysTotal, totalDays),
+                    rate: progress.calculateRate(overallStartTime, processedDaysTotal),
+                    batchDuration: progress.formatElapsedTime(batchStartTime),
+                    historyId: calculateHistoryId
+                });
+
+                // Save checkpoint in history
+                await connection.query(`
+                    UPDATE public.calculate_history
+                    SET additional_info = jsonb_set(additional_info, '{last_completed_batch}', $1::jsonb)
+                                        || jsonb_build_object('last_processed_date', $2::text)
+                    WHERE id = $3::integer;
+                `, [JSON.stringify(currentBatchNum), batchEndDateStr, calculateHistoryId]);
+
+
+             } catch (batchError) {
+                 console.error(`\n--- ERROR in Batch ${currentBatchNum} (${batchStartDateStr} to ${batchEndDateStr}) ---`);
+                 console.error('    Database Error:', batchError.message);
+                 console.error('    DB Error Code:', batchError.code);
+                 // Log detailed error to history and re-throw to stop the process
+                 await connection.query(`
+                    UPDATE public.calculate_history
+                    SET status = 'failed',
+                        end_time = NOW(),
+                        duration_seconds = $1::integer,
+                        error_message = $2::text,
+                        additional_info = additional_info || jsonb_build_object('failed_batch', $3::integer, 'failed_date_range', $4::text)
+                    WHERE id = $5::integer;
+                 `, [
+                     Math.round((Date.now() - overallStartTime) / 1000),
+                     `Batch ${currentBatchNum} failed: ${batchError.message} (Code: ${batchError.code || 'N/A'})`,
+                     currentBatchNum,
+                     `${batchStartDateStr} to ${batchEndDateStr}`,
+                     calculateHistoryId
+                 ]);
+                 throw batchError; // Stop execution
+             }
+
+            currentBatchNum++;
+            // Optional delay between batches
+            // await new Promise(resolve => setTimeout(resolve, 500));
+
+        } // End while loop
+
+        // --- Final Outcome ---
+        const finalStatus = isCancelled ? 'cancelled' : 'completed';
+        const finalMessage = isCancelled ? `Calculation stopped after completing batch ${currentBatchNum - 1}.` : 'Historical snapshots backfill completed successfully.';
+        const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
+
+        console.log(`\n--- Backfill ${finalStatus.toUpperCase()} ---`);
+        console.log(finalMessage);
+        console.log(`Total duration: ${progress.formatElapsedTime(overallStartTime)}`);
+
+        // Update history record
+        await connection.query(`
+            UPDATE public.calculate_history SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3
+            WHERE id = $4::integer;
+        `, [finalStatus, finalDurationSec, (isCancelled ? 'User cancelled' : null), calculateHistoryId]);
+
+        if (!isCancelled) {
+            progress.clearProgress(); // Clear progress state only on successful completion
+        } else {
+             progress.outputProgress({ // Final cancelled status update
+                status: 'cancelled', 
+                operation: finalMessage,
+                currentBatch: currentBatchNum - 1, 
+                totalBatches: totalBatches, 
+                totalDays: totalDays,
+                processedDays: processedDaysTotal,
+                elapsed: progress.formatElapsedTime(overallStartTime), 
+                remaining: 'Cancelled', 
+                rate: 0,
+                historyId: calculateHistoryId
+             });
+        }
+
+        return { success: true, status: finalStatus, message: finalMessage, duration: finalDurationSec };
+
+    } catch (error) {
+        console.error('\n--- Backfill encountered an unrecoverable error ---');
+        console.error(error.message);
+        const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
+
+        // Update history if possible
+        if (connection && calculateHistoryId) {
+             try {
+                 await connection.query(`
+                    UPDATE public.calculate_history
+                    SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3::text
+                    WHERE id = $4::integer;
+                 `, [
+                     isCancelled ? 'cancelled' : 'failed',
+                     finalDurationSec,
+                     error.message,
+                     calculateHistoryId
+                 ]);
+             } catch (histError) {
+                 console.error("Failed to update history record with error state:", histError);
+             }
+        } else {
+            console.error("Could not update history record (no ID or connection).");
+        }
+
+        // FIX: Use initialized value or a default if loop never started
+        const batchNumForError = currentBatchNum > cmdStartBatch ? currentBatchNum - 1 : cmdStartBatch - 1;
+        
+        // Update progress.outputProgress call to match actual function signature
+        try {
+            // Create progress data object
+            const progressData = {
+                status: 'failed',
+                operation: 'Backfill failed',
+                message: error.message,
+                currentBatch: batchNumForError,
+                totalBatches: totalBatches,
+                totalDays: totalDays,
+                processedDays: processedDaysTotal,
+                elapsed: progress.formatElapsedTime(overallStartTime),
+                remaining: 'Failed',
+                rate: 0,
+                // Include history ID in progress data if needed
+                historyId: calculateHistoryId
+            };
+            
+            // Call with single object parameter (not separate historyId)
+            progress.outputProgress(progressData);
+        } catch (progressError) {
+            console.error('Failed to report progress:', progressError);
+        }
+
+        return { success: false, status: 'failed', error: error.message, duration: finalDurationSec };
+
+    } finally {
+        if (connection) {
+            console.log('Releasing database connection.');
+            connection.release();
+        }
+        // Close pool only if this script is meant to be standalone
+        // If part of a larger app, the app should manage pool closure
+        // console.log('Closing database pool.');
+        // await closePool();
+    }
+}
+
+// --- Script Execution ---
+
+// Parse command-line arguments
+const args = process.argv.slice(2);
+let cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg = 1; // Default start batch is 1
+
+for (let i = 0; i < args.length; i++) {
+    if (args[i] === '--start-date' && args[i+1]) cmdStartDateArg = args[++i];
+    else if (args[i] === '--end-date' && args[i+1]) cmdEndDateArg = args[++i];
+    else if (args[i] === '--start-batch' && args[i+1]) cmdStartBatchArg = parseInt(args[++i], 10);
+}
+
+if (isNaN(cmdStartBatchArg) || cmdStartBatchArg < 1) {
+    console.warn(`Invalid --start-batch value. Defaulting to 1.`);
+    cmdStartBatchArg = 1;
+}
+
+// Run the backfill process
+backfillSnapshots(cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg)
+    .then(result => {
+        if (result.success) {
+            console.log(`\n✅ ${result.message} (Duration: ${result.duration}s)`);
+            process.exitCode = 0; // Success
+        } else {
+            console.error(`\n❌ Backfill failed: ${result.error || 'Unknown error'} (Duration: ${result.duration}s)`);
+            process.exitCode = 1; // Failure
+        }
+    })
+    .catch(err => {
+        console.error('\n❌ Unexpected error during backfill execution:', err);
+        process.exitCode = 1; // Failure
+    })
+    .finally(async () => {
+        // Ensure pool is closed if run standalone
+         console.log('Backfill script finished. Closing pool.');
+         await closePool(); // Make sure closePool exists and works in your db utils
+         process.exit(process.exitCode); // Exit with appropriate code
+    });

inventory-server/old/backfill_historical_snapshots.sql

@@ -0,0 +1,161 @@
+-- Description: Backfills the daily_product_snapshots table using imported historical unit data
+--              (daily inventory/stats) and historical price data (current prices table).
+--              - Uses imported daily sales/receipt UNIT counts for accuracy.
+--              - ESTIMATES historical stock levels using a forward calculation.
+--              - APPROXIMATES historical REVENUE using looked-up historical base prices.
+--              - APPROXIMATES historical COGS, PROFIT, and STOCK VALUE using CURRENT product costs/prices.
+--              Run ONCE after importing historical data and before initial product_metrics population.
+-- Dependencies: Core import tables (products), imported history tables (imported_daily_inventory,
+--               imported_product_stat_history, imported_product_current_prices),
+--               daily_product_snapshots table must exist.
+-- Frequency: Run ONCE.
+
+CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final(
+    _start_date DATE,
+    _end_date DATE
+)
+RETURNS VOID AS $$
+DECLARE
+    _current_processing_date DATE := _start_date;
+    _batch_start_time TIMESTAMPTZ;
+    _row_count INTEGER;
+BEGIN
+    RAISE NOTICE 'Starting FINAL historical snapshot backfill from % to %.', _start_date, _end_date;
+    RAISE NOTICE 'Using historical units and historical prices (for revenue approximation).';
+    RAISE NOTICE 'WARNING: Historical COGS, Profit, and Stock Value use CURRENT product costs/prices.';
+
+    -- Ensure end date is not in the future
+    IF _end_date >= CURRENT_DATE THEN
+        _end_date := CURRENT_DATE - INTERVAL '1 day';
+        RAISE NOTICE 'Adjusted end date to % to avoid conflict with hourly script.', _end_date;
+    END IF;
+
+    -- Performance: Create temporary table with product info to avoid repeated lookups
+    CREATE TEMP TABLE IF NOT EXISTS temp_product_info AS 
+    SELECT 
+        pid, 
+        sku,
+        COALESCE(landing_cost_price, cost_price, 0.00) as effective_cost_price,
+        COALESCE(price, 0.00) as current_price,
+        COALESCE(regular_price, 0.00) as current_regular_price
+    FROM public.products;
+    
+    -- Performance: Create index on temporary table
+    CREATE INDEX IF NOT EXISTS temp_product_info_pid_idx ON temp_product_info(pid);
+    
+    ANALYZE temp_product_info;
+    
+    RAISE NOTICE 'Created temporary product info table with % products', (SELECT COUNT(*) FROM temp_product_info);
+
+    WHILE _current_processing_date <= _end_date LOOP
+        _batch_start_time := clock_timestamp();
+        RAISE NOTICE 'Processing date: %', _current_processing_date;
+
+        -- Get Daily Transaction Unit Info from imported history
+        WITH DailyHistoryUnits AS (
+            SELECT
+                pids.pid,
+                -- Prioritize daily_inventory, fallback to product_stat_history for sold qty
+                COALESCE(di.amountsold, ps.qty_sold, 0)::integer as units_sold_today,
+                COALESCE(di.qtyreceived, 0)::integer as units_received_today
+            FROM
+               (SELECT DISTINCT pid FROM temp_product_info) pids -- Ensure all products are considered
+               LEFT JOIN public.imported_daily_inventory di
+                   ON pids.pid = di.pid AND di.date = _current_processing_date
+               LEFT JOIN public.imported_product_stat_history ps
+                   ON pids.pid = ps.pid AND ps.date = _current_processing_date
+            -- Removed WHERE clause to ensure snapshots are created even for days with 0 activity,
+            -- allowing stock carry-over. The main query will handle products properly.
+        ),
+        HistoricalPrice AS (
+            -- Find the base price (qty_buy=1) active on the processing date
+            SELECT DISTINCT ON (pid)
+                pid,
+                price_each
+            FROM public.imported_product_current_prices
+            WHERE
+                qty_buy = 1
+            -- Use TIMESTAMPTZ comparison logic:
+            AND date_active <= (_current_processing_date + interval '1 day' - interval '1 second') -- Active sometime on or before end of processing day
+            AND (date_deactive IS NULL OR date_deactive > _current_processing_date) -- Not deactivated before start of processing day
+            -- Assuming 'active' flag isn't needed if dates are correct; add 'AND active != 0' if necessary
+            ORDER BY
+                pid, date_active DESC -- Get the most recently activated price
+        ),
+        PreviousStock AS (
+            -- Get the estimated stock from the PREVIOUS day snapshot
+            SELECT pid, eod_stock_quantity
+            FROM public.daily_product_snapshots
+            WHERE snapshot_date = _current_processing_date - INTERVAL '1 day'
+        )
+        -- Insert into the daily snapshots table
+        INSERT INTO public.daily_product_snapshots (
+            snapshot_date, pid, sku,
+            eod_stock_quantity, eod_stock_cost, eod_stock_retail, eod_stock_gross, stockout_flag,
+            units_sold, units_returned,
+            gross_revenue, discounts, returns_revenue,
+            net_revenue, cogs, gross_regular_revenue, profit,
+            units_received, cost_received,
+            calculation_timestamp
+        )
+        SELECT
+            _current_processing_date AS snapshot_date,
+            p.pid,
+            p.sku,
+            -- Estimated EOD Stock (using historical daily units)
+            -- Handle potential NULL from joins with COALESCE 0
+            COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0) AS estimated_eod_stock,
+            -- Valued Stock (using estimated stock and CURRENT prices/costs - APPROXIMATION)
+            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.effective_cost_price AS eod_stock_cost,
+            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_price AS eod_stock_retail, -- Stock retail uses current price
+            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_regular_price AS eod_stock_gross, -- Stock gross uses current regular price
+            -- Stockout Flag (based on estimated stock)
+            (COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) <= 0 AS stockout_flag,
+
+            -- Today's Unit Aggregates from History
+            COALESCE(dh.units_sold_today, 0) as units_sold,
+            0 AS units_returned, -- Placeholder: Cannot determine returns from daily summary
+
+            -- Monetary Values using looked-up Historical Price and CURRENT Cost/RegPrice
+            COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS gross_revenue, -- Approx Revenue
+            0 AS discounts, -- Placeholder
+            0 AS returns_revenue, -- Placeholder
+            COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS net_revenue, -- Approx Net Revenue
+            COALESCE(dh.units_sold_today, 0) * p.effective_cost_price AS cogs, -- Approx COGS (uses CURRENT cost)
+            COALESCE(dh.units_sold_today, 0) * p.current_regular_price AS gross_regular_revenue, -- Approx Gross Regular Revenue
+            -- Approx Profit
+            (COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price)) - (COALESCE(dh.units_sold_today, 0) * p.effective_cost_price) AS profit,
+
+            COALESCE(dh.units_received_today, 0) as units_received,
+            -- Estimate received cost using CURRENT product cost
+            COALESCE(dh.units_received_today, 0) * p.effective_cost_price AS cost_received, -- Approx
+
+            clock_timestamp() -- Timestamp of this specific calculation
+        FROM temp_product_info p -- Use the temp table for better performance
+        LEFT JOIN PreviousStock ps ON p.pid = ps.pid
+        LEFT JOIN DailyHistoryUnits dh ON p.pid = dh.pid -- Join today's historical activity
+        LEFT JOIN HistoricalPrice hp ON p.pid = hp.pid -- Join the looked-up historical price
+        -- Optimization: Only process products with activity or previous stock
+        WHERE (dh.units_sold_today > 0 OR dh.units_received_today > 0 OR COALESCE(ps.eod_stock_quantity, 0) > 0)
+
+        ON CONFLICT (snapshot_date, pid) DO NOTHING; -- Avoid errors if rerunning parts, but prefer clean runs
+
+        GET DIAGNOSTICS _row_count = ROW_COUNT;
+        RAISE NOTICE 'Processed %: Inserted/Skipped % rows. Duration: %',
+                     _current_processing_date,
+                     _row_count,
+                     clock_timestamp() - _batch_start_time;
+
+        _current_processing_date := _current_processing_date + INTERVAL '1 day';
+
+    END LOOP;
+    
+    -- Clean up temporary tables
+    DROP TABLE IF EXISTS temp_product_info;
+
+    RAISE NOTICE 'Finished FINAL historical snapshot backfill.';
+END;
+$$ LANGUAGE plpgsql;
+
+-- Example usage:
+-- SELECT backfill_daily_snapshots_range_final('2023-01-01'::date, '2023-12-31'::date);

inventory-server/old/calculate-metrics.js

@@ -57,18 +57,20 @@ const TEMP_TABLES = [
     'temp_daily_sales',
     'temp_product_stats',
     'temp_category_sales',
-    'temp_category_stats'
+    'temp_category_stats',
+    'temp_beginning_inventory',
+    'temp_monthly_inventory'
 ];
 
 // Add cleanup function for temporary tables
 async function cleanupTemporaryTables(connection) {
     try {
+        // Drop each temporary table if it exists
         for (const table of TEMP_TABLES) {
-            await connection.query(`DROP TEMPORARY TABLE IF EXISTS ${table}`);
+            await connection.query(`DROP TABLE IF EXISTS ${table}`);
         }
-    } catch (error) {
+    } catch (err) {
-        logError(error, 'Error cleaning up temporary tables');
+        console.error('Error cleaning up temporary tables:', err);
-        throw error; // Re-throw to be handled by the caller
     }
 }
 
@@ -86,22 +88,42 @@ let isCancelled = false;
 
 function cancelCalculation() {
     isCancelled = true;
-    global.clearProgress();
+    console.log('Calculation has been cancelled by user');
-    // Format as SSE event
+    
-    const event = {
+    // Force-terminate any query that's been running for more than 5 seconds
-        progress: {
+    try {
-            status: 'cancelled',
+        const connection = getConnection();
-            operation: 'Calculation cancelled',
+        connection.then(async (conn) => {
-            current: 0,
+            try {
-            total: 0,
+                // Identify and terminate long-running queries from our application
-            elapsed: null,
+                await conn.query(`
-            remaining: null,
+                    SELECT pg_cancel_backend(pid) 
-            rate: 0,
+                    FROM pg_stat_activity 
-            timestamp: Date.now()
+                    WHERE query_start < now() - interval '5 seconds'
-        }
+                    AND application_name LIKE '%node%'
+                    AND query NOT LIKE '%pg_cancel_backend%'
+                `);
+                
+                // Clean up any temporary tables
+                await cleanupTemporaryTables(conn);
+                
+                // Release connection
+                conn.release();
+            } catch (err) {
+                console.error('Error during force cancellation:', err);
+                conn.release();
+            }
+        }).catch(err => {
+            console.error('Could not get connection for cancellation:', err);
+        });
+    } catch (err) {
+        console.error('Failed to terminate running queries:', err);
+    }
+    
+    return {
+        success: true,
+        message: 'Calculation has been cancelled'
     };
-    process.stdout.write(JSON.stringify(event) + '\n');
-    process.exit(0);
 }
 
 // Handle SIGTERM signal for cancellation
@@ -119,6 +141,15 @@ async function calculateMetrics() {
     let totalPurchaseOrders = 0;
     let calculateHistoryId;
     
+    // Set a maximum execution time (30 minutes)
+    const MAX_EXECUTION_TIME = 30 * 60 * 1000;
+    const timeout = setTimeout(() => {
+        console.error(`Calculation timed out after ${MAX_EXECUTION_TIME/1000} seconds, forcing termination`);
+        // Call cancel and force exit
+        cancelCalculation();
+        process.exit(1);
+    }, MAX_EXECUTION_TIME);
+    
     try {
         // Clean up any previously running calculations
         connection = await getConnection();
@@ -127,24 +158,24 @@ async function calculateMetrics() {
             SET 
                 status = 'cancelled',
                 end_time = NOW(),
-                duration_seconds = TIMESTAMPDIFF(SECOND, start_time, NOW()),
+                duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
                 error_message = 'Previous calculation was not completed properly'
             WHERE status = 'running'
         `);
 
         // Get counts from all relevant tables
-        const [[productCount], [orderCount], [poCount]] = await Promise.all([
+        const [productCountResult, orderCountResult, poCountResult] = await Promise.all([
             connection.query('SELECT COUNT(*) as total FROM products'),
             connection.query('SELECT COUNT(*) as total FROM orders'),
             connection.query('SELECT COUNT(*) as total FROM purchase_orders')
         ]);
 
-        totalProducts = productCount.total;
+        totalProducts = parseInt(productCountResult.rows[0].total);
-        totalOrders = orderCount.total;
+        totalOrders = parseInt(orderCountResult.rows[0].total);
-        totalPurchaseOrders = poCount.total;
+        totalPurchaseOrders = parseInt(poCountResult.rows[0].total);
 
         // Create history record for this calculation
-        const [historyResult] = await connection.query(`
+        const historyResult = await connection.query(`
             INSERT INTO calculate_history (
                 start_time,
                 status,
@@ -155,19 +186,19 @@ async function calculateMetrics() {
             ) VALUES (
                 NOW(),
                 'running',
-                ?,
+                $1,
-                ?,
+                $2,
-                ?,
+                $3,
-                JSON_OBJECT(
+                jsonb_build_object(
-                    'skip_product_metrics', ?,
+                    'skip_product_metrics', ($4::int > 0),
-                    'skip_time_aggregates', ?,
+                    'skip_time_aggregates', ($5::int > 0),
-                    'skip_financial_metrics', ?,
+                    'skip_financial_metrics', ($6::int > 0),
-                    'skip_vendor_metrics', ?,
+                    'skip_vendor_metrics', ($7::int > 0),
-                    'skip_category_metrics', ?,
+                    'skip_category_metrics', ($8::int > 0),
-                    'skip_brand_metrics', ?,
+                    'skip_brand_metrics', ($9::int > 0),
-                    'skip_sales_forecasts', ?
+                    'skip_sales_forecasts', ($10::int > 0)
                 )
-            )
+            ) RETURNING id
         `, [
             totalProducts,
             totalOrders,
@@ -180,8 +211,7 @@ async function calculateMetrics() {
             SKIP_BRAND_METRICS,
             SKIP_SALES_FORECASTS
         ]);
-        calculateHistoryId = historyResult.insertId;
+        calculateHistoryId = historyResult.rows[0].id;
-        connection.release();
 
         // Add debug logging for the progress functions
         console.log('Debug - Progress functions:', {
@@ -199,6 +229,8 @@ async function calculateMetrics() {
             throw err;
         }
 
+        // Release the connection before getting a new one
+        connection.release();
         isCancelled = false;
         connection = await getConnection();
 
@@ -234,10 +266,10 @@ async function calculateMetrics() {
                 await connection.query(`
                     UPDATE calculate_history 
                     SET 
-                        processed_products = ?,
+                        processed_products = $1,
-                        processed_orders = ?,
+                        processed_orders = $2,
-                        processed_purchase_orders = ?
+                        processed_purchase_orders = $3
-                    WHERE id = ?
+                    WHERE id = $4
                 `, [safeProducts, safeOrders, safePurchaseOrders, calculateHistoryId]);
             };
 
@@ -359,216 +391,6 @@ async function calculateMetrics() {
                 console.log('Skipping sales forecasts calculation');
             }
 
-            // Calculate ABC classification
-            outputProgress({
-                status: 'running',
-                operation: 'Starting ABC classification',
-                current: processedProducts || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedProducts || 0),
-                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            if (isCancelled) return {
-                processedProducts: processedProducts || 0,
-                processedOrders: processedOrders || 0,
-                processedPurchaseOrders: 0,
-                success: false
-            };
-
-            const [abcConfig] = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
-            const abcThresholds = abcConfig[0] || { a_threshold: 20, b_threshold: 50 };
-
-            // First, create and populate the rankings table with an index
-            await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
-            await connection.query(`
-                CREATE TEMPORARY TABLE temp_revenue_ranks (
-                    pid BIGINT NOT NULL,
-                    total_revenue DECIMAL(10,3),
-                    rank_num INT,
-                    total_count INT,
-                    PRIMARY KEY (pid),
-                    INDEX (rank_num)
-                ) ENGINE=MEMORY
-            `);
-
-            outputProgress({
-                status: 'running',
-                operation: 'Creating revenue rankings',
-                current: processedProducts || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedProducts || 0),
-                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            if (isCancelled) return {
-                processedProducts: processedProducts || 0,
-                processedOrders: processedOrders || 0,
-                processedPurchaseOrders: 0,
-                success: false
-            };
-
-            await connection.query(`
-                INSERT INTO temp_revenue_ranks
-                SELECT 
-                    pid,
-                    total_revenue,
-                    @rank := @rank + 1 as rank_num,
-                    @total_count := @rank as total_count
-                FROM (
-                    SELECT pid, total_revenue 
-                    FROM product_metrics 
-                    WHERE total_revenue > 0 
-                    ORDER BY total_revenue DESC
-                ) ranked, 
-                (SELECT @rank := 0) r
-            `);
-
-            // Get total count for percentage calculation
-            const [rankingCount] = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
-            const totalCount = rankingCount[0].total_count || 1;
-            const max_rank = totalCount; // Store max_rank for use in classification
-
-            outputProgress({
-                status: 'running',
-                operation: 'Updating ABC classifications',
-                current: processedProducts || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedProducts || 0),
-                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            if (isCancelled) return {
-                processedProducts: processedProducts || 0,
-                processedOrders: processedOrders || 0,
-                processedPurchaseOrders: 0,
-                success: false
-            };
-
-            // ABC classification progress tracking
-            let abcProcessedCount = 0;
-            const batchSize = 5000;
-            let lastProgressUpdate = Date.now();
-            const progressUpdateInterval = 1000; // Update every second
-
-            while (true) {
-                if (isCancelled) return {
-                    processedProducts: Number(processedProducts) || 0,
-                    processedOrders: Number(processedOrders) || 0,
-                    processedPurchaseOrders: 0,
-                    success: false
-                };
-
-                // First get a batch of PIDs that need updating
-                const [pids] = await connection.query(`
-                    SELECT pm.pid 
-                    FROM product_metrics pm
-                    LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
-                    WHERE pm.abc_class IS NULL 
-                    OR pm.abc_class != 
-                        CASE 
-                            WHEN tr.rank_num IS NULL THEN 'C'
-                            WHEN (tr.rank_num / ?) * 100 <= ? THEN 'A'
-                            WHEN (tr.rank_num / ?) * 100 <= ? THEN 'B'
-                            ELSE 'C'
-                        END
-                    LIMIT ?
-                `, [max_rank, abcThresholds.a_threshold,
-                    max_rank, abcThresholds.b_threshold,
-                    batchSize]);
-
-                if (pids.length === 0) {
-                    break;
-                }
-
-                // Then update just those PIDs
-                const [result] = await connection.query(`
-                    UPDATE product_metrics pm
-                    LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
-                    SET pm.abc_class = 
-                        CASE 
-                            WHEN tr.rank_num IS NULL THEN 'C'
-                            WHEN (tr.rank_num / ?) * 100 <= ? THEN 'A'
-                            WHEN (tr.rank_num / ?) * 100 <= ? THEN 'B'
-                            ELSE 'C'
-                        END,
-                        pm.last_calculated_at = NOW()
-                    WHERE pm.pid IN (?)
-                `, [max_rank, abcThresholds.a_threshold,
-                    max_rank, abcThresholds.b_threshold,
-                    pids.map(row => row.pid)]);
-
-                abcProcessedCount += result.affectedRows;
-                
-                // Calculate progress ensuring valid numbers
-                const currentProgress = Math.floor(totalProducts * (0.99 + (abcProcessedCount / (totalCount || 1)) * 0.01));
-                processedProducts = Number(currentProgress) || processedProducts || 0;
-
-                // Only update progress at most once per second
-                const now = Date.now();
-                if (now - lastProgressUpdate >= progressUpdateInterval) {
-                    const progress = ensureValidProgress(processedProducts, totalProducts);
-                    
-                    outputProgress({
-                        status: 'running',
-                        operation: 'ABC classification progress',
-                        current: progress.current,
-                        total: progress.total,
-                        elapsed: formatElapsedTime(startTime),
-                        remaining: estimateRemaining(startTime, progress.current, progress.total),
-                        rate: calculateRate(startTime, progress.current),
-                        percentage: progress.percentage,
-                        timing: {
-                            start_time: new Date(startTime).toISOString(),
-                            end_time: new Date().toISOString(),
-                            elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                        }
-                    });
-                    
-                    lastProgressUpdate = now;
-                }
-
-                // Update database progress
-                await updateProgress(processedProducts, processedOrders, processedPurchaseOrders);
-
-                // Small delay between batches to allow other transactions
-                await new Promise(resolve => setTimeout(resolve, 100));
-            }
-
-            // Clean up
-            await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
-
-            const endTime = Date.now();
-            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
-
-            // Update calculate_status for ABC classification
-            await connection.query(`
-                INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-                VALUES ('abc_classification', NOW())
-                ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
-            `);
-
             // Final progress update with guaranteed valid numbers
             const finalProgress = ensureValidProgress(totalProducts, totalProducts);
             
@@ -578,14 +400,14 @@ async function calculateMetrics() {
                 operation: 'Metrics calculation complete',
                 current: finalProgress.current,
                 total: finalProgress.total,
-                elapsed: formatElapsedTime(startTime),
+                elapsed: global.formatElapsedTime(startTime),
                 remaining: '0s',
-                rate: calculateRate(startTime, finalProgress.current),
+                rate: global.calculateRate(startTime, finalProgress.current),
                 percentage: '100',
                 timing: {
                     start_time: new Date(startTime).toISOString(),
                     end_time: new Date().toISOString(),
-                    elapsed_seconds: totalElapsedSeconds
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
             });
 
@@ -601,13 +423,13 @@ async function calculateMetrics() {
                 UPDATE calculate_history 
                 SET 
                     end_time = NOW(),
-                    duration_seconds = ?,
+                    duration_seconds = $1,
-                    processed_products = ?,
+                    processed_products = $2,
-                    processed_orders = ?,
+                    processed_orders = $3,
-                    processed_purchase_orders = ?,
+                    processed_purchase_orders = $4,
                     status = 'completed'
-                WHERE id = ?
+                WHERE id = $5
-            `, [totalElapsedSeconds, 
+            `, [Math.round((Date.now() - startTime) / 1000), 
                 finalStats.processedProducts,
                 finalStats.processedOrders, 
                 finalStats.processedPurchaseOrders,
@@ -616,6 +438,11 @@ async function calculateMetrics() {
             // Clear progress file on successful completion
             global.clearProgress();
 
+            return {
+                success: true,
+                message: 'Calculation completed successfully',
+                duration: Math.round((Date.now() - startTime) / 1000)
+            };
         } catch (error) {
             const endTime = Date.now();
             const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
@@ -625,13 +452,13 @@ async function calculateMetrics() {
                 UPDATE calculate_history 
                 SET 
                     end_time = NOW(),
-                    duration_seconds = ?,
+                    duration_seconds = $1,
-                    processed_products = ?,
+                    processed_products = $2,
-                    processed_orders = ?,
+                    processed_orders = $3,
-                    processed_purchase_orders = ?,
+                    processed_purchase_orders = $4,
-                    status = ?,
+                    status = $5,
-                    error_message = ?
+                    error_message = $6
-                WHERE id = ?
+                WHERE id = $7
             `, [
                 totalElapsedSeconds,
                 processedProducts || 0,  // Ensure we have a valid number
@@ -677,17 +504,38 @@ async function calculateMetrics() {
             }
             throw error;
         } finally {
+            // Clear the timeout to prevent forced termination
+            clearTimeout(timeout);
+            
+            // Always clean up and release connection
             if (connection) {
-                // Ensure temporary tables are cleaned up
+                try {
-                await cleanupTemporaryTables(connection);
+                    await cleanupTemporaryTables(connection);
-                connection.release();
+                    connection.release();
+                } catch (err) {
+                    console.error('Error in final cleanup:', err);
+                }
             }
-            // Close the connection pool when we're done
-            await closePool();
         }
     } catch (error) {
-        success = false;
+        console.error('Error in metrics calculation', error);
-        logError(error, 'Error in metrics calculation');
+        
+        try {
+            if (connection) {
+                await connection.query(`
+                    UPDATE calculate_history 
+                    SET 
+                        status = 'failed',
+                        end_time = NOW(),
+                        duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
+                        error_message = $1
+                    WHERE id = $2
+                `, [error.message.substring(0, 500), calculateHistoryId]);
+            }
+        } catch (updateError) {
+            console.error('Error updating calculation history:', updateError);
+        }
+        
         throw error;
     }
 }

inventory-server/old/config-schema.sql

@@ -0,0 +1,242 @@
+-- -- Configuration tables schema
+
+
+
+-- -- Stock threshold configurations
+-- CREATE TABLE stock_thresholds (
+--     id INTEGER NOT NULL,
+--     category_id BIGINT,  -- NULL means default/global threshold
+--     vendor VARCHAR(100), -- NULL means applies to all vendors
+--     critical_days INTEGER NOT NULL DEFAULT 7,
+--     reorder_days INTEGER NOT NULL DEFAULT 14,
+--     overstock_days INTEGER NOT NULL DEFAULT 90,
+--     low_stock_threshold INTEGER NOT NULL DEFAULT 5,
+--     min_reorder_quantity INTEGER NOT NULL DEFAULT 1,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (id),
+--     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
+--     UNIQUE (category_id, vendor)
+-- );
+
+-- CREATE TRIGGER update_stock_thresholds_updated
+--     BEFORE UPDATE ON stock_thresholds
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- CREATE INDEX idx_st_metrics ON stock_thresholds(category_id, vendor);
+
+-- -- Lead time threshold configurations
+-- CREATE TABLE lead_time_thresholds (
+--     id INTEGER NOT NULL,
+--     category_id BIGINT,  -- NULL means default/global threshold
+--     vendor VARCHAR(100), -- NULL means applies to all vendors
+--     target_days INTEGER NOT NULL DEFAULT 14,
+--     warning_days INTEGER NOT NULL DEFAULT 21,
+--     critical_days INTEGER NOT NULL DEFAULT 30,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (id),
+--     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
+--     UNIQUE (category_id, vendor)
+-- );
+
+-- CREATE TRIGGER update_lead_time_thresholds_updated
+--     BEFORE UPDATE ON lead_time_thresholds
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- -- Sales velocity window configurations
+-- CREATE TABLE sales_velocity_config (
+--     id INTEGER NOT NULL,
+--     category_id BIGINT,  -- NULL means default/global threshold
+--     vendor VARCHAR(100), -- NULL means applies to all vendors
+--     daily_window_days INTEGER NOT NULL DEFAULT 30,
+--     weekly_window_days INTEGER NOT NULL DEFAULT 7,
+--     monthly_window_days INTEGER NOT NULL DEFAULT 90,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (id),
+--     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
+--     UNIQUE (category_id, vendor)
+-- );
+
+-- CREATE TRIGGER update_sales_velocity_config_updated
+--     BEFORE UPDATE ON sales_velocity_config
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- CREATE INDEX idx_sv_metrics ON sales_velocity_config(category_id, vendor);
+
+-- -- ABC Classification configurations
+-- CREATE TABLE abc_classification_config (
+--     id INTEGER NOT NULL PRIMARY KEY,
+--     a_threshold DECIMAL(5,2) NOT NULL DEFAULT 20.0,
+--     b_threshold DECIMAL(5,2) NOT NULL DEFAULT 50.0,
+--     classification_period_days INTEGER NOT NULL DEFAULT 90,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
+-- );
+
+-- CREATE TRIGGER update_abc_classification_config_updated
+--     BEFORE UPDATE ON abc_classification_config
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- -- Safety stock configurations
+-- CREATE TABLE safety_stock_config (
+--     id INTEGER NOT NULL,
+--     category_id BIGINT,  -- NULL means default/global threshold
+--     vendor VARCHAR(100), -- NULL means applies to all vendors
+--     coverage_days INTEGER NOT NULL DEFAULT 14,
+--     service_level DECIMAL(5,2) NOT NULL DEFAULT 95.0,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (id),
+--     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
+--     UNIQUE (category_id, vendor)
+-- );
+
+-- CREATE TRIGGER update_safety_stock_config_updated
+--     BEFORE UPDATE ON safety_stock_config
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- CREATE INDEX idx_ss_metrics ON safety_stock_config(category_id, vendor);
+
+-- -- Turnover rate configurations
+-- CREATE TABLE turnover_config (
+--     id INTEGER NOT NULL,
+--     category_id BIGINT,  -- NULL means default/global threshold
+--     vendor VARCHAR(100), -- NULL means applies to all vendors
+--     calculation_period_days INTEGER NOT NULL DEFAULT 30,
+--     target_rate DECIMAL(10,2) NOT NULL DEFAULT 1.0,
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (id),
+--     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
+--     UNIQUE (category_id, vendor)
+-- );
+
+-- CREATE TRIGGER update_turnover_config_updated
+--     BEFORE UPDATE ON turnover_config
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- -- Create table for sales seasonality factors
+-- CREATE TABLE sales_seasonality (
+--     month INTEGER NOT NULL,
+--     seasonality_factor DECIMAL(5,3) DEFAULT 0,
+--     last_updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+--     PRIMARY KEY (month),
+--     CONSTRAINT month_range CHECK (month BETWEEN 1 AND 12),
+--     CONSTRAINT seasonality_range CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
+-- );
+
+-- CREATE TRIGGER update_sales_seasonality_updated
+--     BEFORE UPDATE ON sales_seasonality
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- -- Create table for financial calculation parameters
+-- CREATE TABLE financial_calc_config (
+--     id INTEGER NOT NULL PRIMARY KEY,
+--     order_cost DECIMAL(10,2) NOT NULL DEFAULT 25.00, -- The fixed cost per purchase order (used in EOQ)
+--     holding_rate DECIMAL(10,4) NOT NULL DEFAULT 0.25, -- The annual inventory holding cost as a percentage of unit cost (used in EOQ)
+--     service_level_z_score DECIMAL(10,4) NOT NULL DEFAULT 1.96, -- Z-score for ~95% service level (used in Safety Stock)
+--     min_reorder_qty INTEGER NOT NULL DEFAULT 1, -- Minimum reorder quantity 
+--     default_reorder_qty INTEGER NOT NULL DEFAULT 5, -- Default reorder quantity when sales data is insufficient
+--     default_safety_stock INTEGER NOT NULL DEFAULT 5, -- Default safety stock when sales data is insufficient
+--     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+--     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
+-- );
+
+-- CREATE TRIGGER update_financial_calc_config_updated
+--     BEFORE UPDATE ON financial_calc_config
+--     FOR EACH ROW
+--     EXECUTE FUNCTION update_updated_at_column();
+
+-- -- Insert default global thresholds
+-- INSERT INTO stock_thresholds (id, category_id, vendor, critical_days, reorder_days, overstock_days)
+-- VALUES (1, NULL, NULL, 7, 14, 90)
+-- ON CONFLICT (id) DO UPDATE SET
+--     critical_days = EXCLUDED.critical_days,
+--     reorder_days = EXCLUDED.reorder_days,
+--     overstock_days = EXCLUDED.overstock_days;
+
+-- INSERT INTO lead_time_thresholds (id, category_id, vendor, target_days, warning_days, critical_days)
+-- VALUES (1, NULL, NULL, 14, 21, 30)
+-- ON CONFLICT (id) DO UPDATE SET
+--     target_days = EXCLUDED.target_days,
+--     warning_days = EXCLUDED.warning_days,
+--     critical_days = EXCLUDED.critical_days;
+
+-- INSERT INTO sales_velocity_config (id, category_id, vendor, daily_window_days, weekly_window_days, monthly_window_days)
+-- VALUES (1, NULL, NULL, 30, 7, 90)
+-- ON CONFLICT (id) DO UPDATE SET
+--     daily_window_days = EXCLUDED.daily_window_days,
+--     weekly_window_days = EXCLUDED.weekly_window_days,
+--     monthly_window_days = EXCLUDED.monthly_window_days;
+
+-- INSERT INTO abc_classification_config (id, a_threshold, b_threshold, classification_period_days)
+-- VALUES (1, 20.0, 50.0, 90)
+-- ON CONFLICT (id) DO UPDATE SET
+--     a_threshold = EXCLUDED.a_threshold,
+--     b_threshold = EXCLUDED.b_threshold,
+--     classification_period_days = EXCLUDED.classification_period_days;
+
+-- INSERT INTO safety_stock_config (id, category_id, vendor, coverage_days, service_level)
+-- VALUES (1, NULL, NULL, 14, 95.0)
+-- ON CONFLICT (id) DO UPDATE SET
+--     coverage_days = EXCLUDED.coverage_days,
+--     service_level = EXCLUDED.service_level;
+
+-- INSERT INTO turnover_config (id, category_id, vendor, calculation_period_days, target_rate)
+-- VALUES (1, NULL, NULL, 30, 1.0)
+-- ON CONFLICT (id) DO UPDATE SET
+--     calculation_period_days = EXCLUDED.calculation_period_days,
+--     target_rate = EXCLUDED.target_rate;
+
+-- -- Insert default seasonality factors (neutral)
+-- INSERT INTO sales_seasonality (month, seasonality_factor)
+-- VALUES 
+--     (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
+--     (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
+-- ON CONFLICT (month) DO UPDATE SET
+--     last_updated = CURRENT_TIMESTAMP;
+
+-- -- Insert default values
+-- INSERT INTO financial_calc_config (id, order_cost, holding_rate, service_level_z_score, min_reorder_qty, default_reorder_qty, default_safety_stock)
+-- VALUES (1, 25.00, 0.25, 1.96, 1, 5, 5)
+-- ON CONFLICT (id) DO UPDATE SET
+--     order_cost = EXCLUDED.order_cost,
+--     holding_rate = EXCLUDED.holding_rate,
+--     service_level_z_score = EXCLUDED.service_level_z_score,
+--     min_reorder_qty = EXCLUDED.min_reorder_qty,
+--     default_reorder_qty = EXCLUDED.default_reorder_qty,
+--     default_safety_stock = EXCLUDED.default_safety_stock;
+
+-- -- View to show thresholds with category names
+-- CREATE OR REPLACE VIEW stock_thresholds_view AS
+-- SELECT 
+--     st.*,
+--     c.name as category_name,
+--     CASE 
+--         WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 'Global Default'
+--         WHEN st.category_id IS NULL THEN 'Vendor: ' || st.vendor
+--         WHEN st.vendor IS NULL THEN 'Category: ' || c.name
+--         ELSE 'Category: ' || c.name || ' / Vendor: ' || st.vendor
+--     END as threshold_scope
+-- FROM 
+--     stock_thresholds st
+-- LEFT JOIN 
+--     categories c ON st.category_id = c.cat_id
+-- ORDER BY 
+--     CASE 
+--         WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 1
+--         WHEN st.category_id IS NULL THEN 2
+--         WHEN st.vendor IS NULL THEN 3
+--         ELSE 4
+--     END,
+--     c.name,
+--     st.vendor;

inventory-server/old/historical-data.js

@@ -0,0 +1,961 @@
+const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../scripts/metrics-new/utils/progress');
+const fs = require('fs');
+const path = require('path');
+const { pipeline } = require('stream');
+const { promisify } = require('util');
+
+// Configuration constants to control which tables get imported
+const IMPORT_PRODUCT_CURRENT_PRICES = false;
+const IMPORT_DAILY_INVENTORY = false; 
+const IMPORT_PRODUCT_STAT_HISTORY = true;
+
+// For product stat history, limit to more recent data for faster initial import
+const USE_RECENT_MONTHS = 12; // Just use the most recent months for product_stat_history
+
+/**
+ * Validates a date from MySQL before inserting it into PostgreSQL
+ * @param {string|Date|null} mysqlDate - Date string or object from MySQL
+ * @returns {string|null} Valid date string or null if invalid
+ */
+function validateDate(mysqlDate) {
+  // Handle null, undefined, or empty values
+  if (!mysqlDate) {
+    return null;
+  }
+  
+  // Convert to string if it's not already
+  const dateStr = String(mysqlDate);
+  
+  // Handle MySQL zero dates and empty values
+  if (dateStr === '0000-00-00' || 
+      dateStr === '0000-00-00 00:00:00' ||
+      dateStr.indexOf('0000-00-00') !== -1 ||
+      dateStr === '') {
+    return null;
+  }
+  
+  // Check if the date is valid
+  const date = new Date(mysqlDate);
+  
+  // If the date is invalid or suspiciously old (pre-1970), return null
+  if (isNaN(date.getTime()) || date.getFullYear() < 1970) {
+    return null;
+  }
+  
+  return mysqlDate;
+}
+
+/**
+ * Imports historical data from MySQL to PostgreSQL
+ */
+async function importHistoricalData(
+  prodConnection, 
+  localConnection, 
+  options = {}
+) {
+  const {
+    incrementalUpdate = true,
+    oneYearAgo = new Date(new Date().setFullYear(new Date().getFullYear() - 1))
+  } = options;
+
+  const oneYearAgoStr = oneYearAgo.toISOString().split('T')[0];
+  const startTime = Date.now();
+  
+  // Use larger batch sizes to improve performance
+  const BATCH_SIZE = 5000;  // For fetching from small tables
+  const INSERT_BATCH_SIZE = 500; // For inserting to small tables
+  const LARGE_BATCH_SIZE = 10000; // For fetching from large tables
+  const LARGE_INSERT_BATCH_SIZE = 1000; // For inserting to large tables
+  
+  // Calculate date for recent data
+  const recentDateStr = new Date(
+    new Date().setMonth(new Date().getMonth() - USE_RECENT_MONTHS)
+  ).toISOString().split('T')[0];
+
+  console.log(`Starting import with:
+    - One year ago date: ${oneYearAgoStr}
+    - Recent months date: ${recentDateStr} (for product_stat_history)
+    - Incremental update: ${incrementalUpdate}
+    - Standard batch size: ${BATCH_SIZE}
+    - Standard insert batch size: ${INSERT_BATCH_SIZE}
+    - Large table batch size: ${LARGE_BATCH_SIZE}
+    - Large table insert batch size: ${LARGE_INSERT_BATCH_SIZE}
+    - Import product_current_prices: ${IMPORT_PRODUCT_CURRENT_PRICES}
+    - Import daily_inventory: ${IMPORT_DAILY_INVENTORY}
+    - Import product_stat_history: ${IMPORT_PRODUCT_STAT_HISTORY}`);
+
+  try {
+    // Get last sync time for incremental updates
+    const lastSyncTimes = {};
+    
+    if (incrementalUpdate) {
+      try {
+        const syncResult = await localConnection.query(`
+          SELECT table_name, last_sync_timestamp
+          FROM sync_status
+          WHERE table_name IN (
+            'imported_product_current_prices', 
+            'imported_daily_inventory', 
+            'imported_product_stat_history'
+          )
+        `);
+        
+        // Add check for rows existence and type
+        if (syncResult && Array.isArray(syncResult.rows)) {
+          for (const row of syncResult.rows) {
+            lastSyncTimes[row.table_name] = row.last_sync_timestamp;
+            console.log(`Last sync time for ${row.table_name}: ${row.last_sync_timestamp}`);
+          }
+        } else {
+          console.warn('Sync status query did not return expected rows. Proceeding without last sync times.');
+        }
+      } catch (error) {
+        console.error('Error fetching sync status:', error);
+      }
+    }
+
+    // Determine how many tables will be imported
+    const tablesCount = [
+      IMPORT_PRODUCT_CURRENT_PRICES,
+      IMPORT_DAILY_INVENTORY,
+      IMPORT_PRODUCT_STAT_HISTORY
+    ].filter(Boolean).length;
+
+    // Run all imports sequentially for better reliability
+    console.log(`Starting sequential imports for ${tablesCount} tables...`);
+    outputProgress({
+      status: "running",
+      operation: "Historical data import",
+      message: `Starting sequential imports for ${tablesCount} tables...`,
+      current: 0,
+      total: tablesCount,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    let progressCount = 0;
+    let productCurrentPricesResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
+    let dailyInventoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
+    let productStatHistoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
+
+    // Import product current prices
+    if (IMPORT_PRODUCT_CURRENT_PRICES) {
+      console.log('Importing product current prices...');
+      productCurrentPricesResult = await importProductCurrentPrices(
+        prodConnection, 
+        localConnection, 
+        oneYearAgoStr, 
+        lastSyncTimes['imported_product_current_prices'], 
+        BATCH_SIZE,
+        INSERT_BATCH_SIZE,
+        incrementalUpdate,
+        startTime
+      );
+      progressCount++;
+      
+      outputProgress({
+        status: "running",
+        operation: "Historical data import",
+        message: `Completed import ${progressCount} of ${tablesCount}`,
+        current: progressCount,
+        total: tablesCount,
+        elapsed: formatElapsedTime(startTime)
+      });
+    }
+    
+    // Import daily inventory
+    if (IMPORT_DAILY_INVENTORY) {
+      console.log('Importing daily inventory...');
+      dailyInventoryResult = await importDailyInventory(
+        prodConnection, 
+        localConnection, 
+        oneYearAgoStr, 
+        lastSyncTimes['imported_daily_inventory'], 
+        BATCH_SIZE,
+        INSERT_BATCH_SIZE,
+        incrementalUpdate,
+        startTime
+      );
+      progressCount++;
+      
+      outputProgress({
+        status: "running",
+        operation: "Historical data import",
+        message: `Completed import ${progressCount} of ${tablesCount}`,
+        current: progressCount,
+        total: tablesCount,
+        elapsed: formatElapsedTime(startTime)
+      });
+    }
+    
+    // Import product stat history - using optimized approach
+    if (IMPORT_PRODUCT_STAT_HISTORY) {
+      console.log('Importing product stat history...');
+      productStatHistoryResult = await importProductStatHistory(
+        prodConnection, 
+        localConnection, 
+        recentDateStr, // Use more recent date for this massive table 
+        lastSyncTimes['imported_product_stat_history'], 
+        LARGE_BATCH_SIZE,
+        LARGE_INSERT_BATCH_SIZE,
+        incrementalUpdate,
+        startTime,
+        USE_RECENT_MONTHS // Pass the recent months constant
+      );
+      progressCount++;
+      
+      outputProgress({
+        status: "running",
+        operation: "Historical data import",
+        message: `Completed import ${progressCount} of ${tablesCount}`,
+        current: progressCount,
+        total: tablesCount,
+        elapsed: formatElapsedTime(startTime)
+      });
+    }
+
+    // Aggregate results
+    const totalRecordsAdded = 
+      productCurrentPricesResult.recordsAdded + 
+      dailyInventoryResult.recordsAdded + 
+      productStatHistoryResult.recordsAdded;
+    
+    const totalRecordsUpdated = 
+      productCurrentPricesResult.recordsUpdated + 
+      dailyInventoryResult.recordsUpdated + 
+      productStatHistoryResult.recordsUpdated;
+    
+    const totalProcessed = 
+      productCurrentPricesResult.totalProcessed + 
+      dailyInventoryResult.totalProcessed + 
+      productStatHistoryResult.totalProcessed;
+    
+    const allErrors = [
+      ...productCurrentPricesResult.errors,
+      ...dailyInventoryResult.errors,
+      ...productStatHistoryResult.errors
+    ];
+
+    // Log import summary
+    console.log(`
+Historical data import complete:
+-------------------------------
+Records added:   ${totalRecordsAdded}
+Records updated: ${totalRecordsUpdated}
+Total processed: ${totalProcessed}
+Errors:          ${allErrors.length}
+Time taken:      ${formatElapsedTime(startTime)}
+    `);
+
+    // Final progress update
+    outputProgress({
+      status: "complete",
+      operation: "Historical data import",
+      message: `Import complete. Added: ${totalRecordsAdded}, Updated: ${totalRecordsUpdated}, Errors: ${allErrors.length}`,
+      current: tablesCount,
+      total: tablesCount,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    // Log any errors
+    if (allErrors.length > 0) {
+      console.log('Errors encountered during import:');
+      console.log(JSON.stringify(allErrors, null, 2));
+    }
+
+    // Calculate duration
+    const endTime = Date.now();
+    const durationSeconds = Math.round((endTime - startTime) / 1000);
+    const finalStatus = allErrors.length === 0 ? 'complete' : 'failed';
+    const errorMessage = allErrors.length > 0 ? JSON.stringify(allErrors) : null;
+
+    // Update import history
+    await localConnection.query(`
+      INSERT INTO import_history (
+        table_name, 
+        end_time, 
+        duration_seconds, 
+        records_added, 
+        records_updated, 
+        is_incremental, 
+        status, 
+        error_message,
+        additional_info
+      )
+      VALUES ($1, NOW(), $2, $3, $4, $5, $6, $7, $8)
+    `, [
+      'historical_data_combined',
+      durationSeconds,
+      totalRecordsAdded,
+      totalRecordsUpdated,
+      incrementalUpdate,
+      finalStatus,
+      errorMessage,
+      JSON.stringify({ 
+        totalProcessed,
+        tablesImported: {
+          imported_product_current_prices: IMPORT_PRODUCT_CURRENT_PRICES,
+          imported_daily_inventory: IMPORT_DAILY_INVENTORY,
+          imported_product_stat_history: IMPORT_PRODUCT_STAT_HISTORY
+        }
+      })
+    ]);
+
+    // Return summary
+    return {
+      recordsAdded: totalRecordsAdded,
+      recordsUpdated: totalRecordsUpdated,
+      totalProcessed,
+      errors: allErrors,
+      timeTaken: formatElapsedTime(startTime)
+    };
+  } catch (error) {
+    console.error('Error importing historical data:', error);
+    
+    // Final progress update on error
+    outputProgress({
+      status: "failed",
+      operation: "Historical data import",
+      message: `Import failed: ${error.message}`,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    throw error;
+  }
+}
+
+/**
+ * Imports product_current_prices data from MySQL to PostgreSQL
+ */
+async function importProductCurrentPrices(
+  prodConnection, 
+  localConnection, 
+  oneYearAgoStr, 
+  lastSyncTime, 
+  batchSize,
+  insertBatchSize,
+  incrementalUpdate,
+  startTime
+) {
+  let recordsAdded = 0;
+  let recordsUpdated = 0;
+  let totalProcessed = 0;
+  let errors = [];
+  let offset = 0;
+  let allProcessed = false;
+
+  try {
+    // Get total count for progress reporting
+    const [countResult] = await prodConnection.query(`
+      SELECT COUNT(*) as total
+      FROM product_current_prices
+      WHERE (date_active >= ? OR date_deactive >= ?)
+        ${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
+    `, [oneYearAgoStr, oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
+
+    const totalCount = countResult[0].total;
+
+    outputProgress({
+      status: "running",
+      operation: "Historical data import - Product Current Prices",
+      message: `Found ${totalCount} records to process`,
+      current: 0,
+      total: totalCount,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    // Process in batches for better performance
+    while (!allProcessed) {
+      try {
+        // Fetch batch from production
+        const [rows] = await prodConnection.query(`
+          SELECT
+            price_id,
+            pid,
+            qty_buy,
+            is_min_qty_buy,
+            price_each,
+            qty_limit,
+            no_promo,
+            checkout_offer,
+            active,
+            date_active,
+            date_deactive
+          FROM product_current_prices
+          WHERE (date_active >= ? OR date_deactive >= ?)
+            ${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
+          ORDER BY price_id
+          LIMIT ? OFFSET ?
+        `, [
+          oneYearAgoStr, 
+          oneYearAgoStr, 
+          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
+          batchSize,
+          offset
+        ]);
+
+        if (rows.length === 0) {
+          allProcessed = true;
+          break;
+        }
+
+        // Process rows in smaller batches for better performance
+        for (let i = 0; i < rows.length; i += insertBatchSize) {
+          const batch = rows.slice(i, i + insertBatchSize);
+          
+          if (batch.length === 0) continue;
+          
+          try {
+            // Build parameterized query to handle NULL values properly
+            const values = [];
+            const placeholders = [];
+            let placeholderIndex = 1;
+            
+            for (const row of batch) {
+              const rowPlaceholders = [
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`
+              ];
+              
+              placeholders.push(`(${rowPlaceholders.join(', ')})`);
+              
+              values.push(
+                row.price_id,
+                row.pid,
+                row.qty_buy,
+                row.is_min_qty_buy ? true : false,
+                row.price_each,
+                row.qty_limit, // PostgreSQL will handle null values properly
+                row.no_promo ? true : false,
+                row.checkout_offer ? true : false,
+                row.active ? true : false,
+                validateDate(row.date_active),
+                validateDate(row.date_deactive)
+              );
+            }
+            
+            // Execute batch insert
+            const result = await localConnection.query(`
+              WITH ins AS (
+                INSERT INTO imported_product_current_prices (
+                  price_id, pid, qty_buy, is_min_qty_buy, price_each, qty_limit, 
+                  no_promo, checkout_offer, active, date_active, date_deactive
+                )
+                VALUES ${placeholders.join(',\n')}
+                ON CONFLICT (price_id) DO UPDATE SET
+                  pid = EXCLUDED.pid,
+                  qty_buy = EXCLUDED.qty_buy,
+                  is_min_qty_buy = EXCLUDED.is_min_qty_buy,
+                  price_each = EXCLUDED.price_each,
+                  qty_limit = EXCLUDED.qty_limit,
+                  no_promo = EXCLUDED.no_promo,
+                  checkout_offer = EXCLUDED.checkout_offer,
+                  active = EXCLUDED.active,
+                  date_active = EXCLUDED.date_active,
+                  date_deactive = EXCLUDED.date_deactive,
+                  updated = CURRENT_TIMESTAMP
+                RETURNING (xmax = 0) AS inserted
+              )
+              SELECT 
+                COUNT(*) FILTER (WHERE inserted) AS inserted_count, 
+                COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
+              FROM ins
+            `, values);
+            
+            // Safely update counts based on the result
+            if (result && result.rows && result.rows.length > 0) {
+              const insertedCount = parseInt(result.rows[0].inserted_count || 0);
+              const updatedCount = parseInt(result.rows[0].updated_count || 0);
+              
+              recordsAdded += insertedCount;
+              recordsUpdated += updatedCount;
+            }
+          } catch (error) {
+            console.error(`Error in batch import of product_current_prices at offset ${i}:`, error);
+            errors.push({
+              table: 'imported_product_current_prices',
+              batchOffset: i,
+              batchSize: batch.length,
+              error: error.message
+            });
+          }
+        }
+
+        totalProcessed += rows.length;
+        offset += rows.length;
+
+        // Update progress
+        outputProgress({
+          status: "running",
+          operation: "Historical data import - Product Current Prices",
+          message: `Processed ${totalProcessed} of ${totalCount} records`,
+          current: totalProcessed,
+          total: totalCount,
+          elapsed: formatElapsedTime(startTime),
+          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
+          rate: calculateRate(startTime, totalProcessed)
+        });
+      } catch (error) {
+        console.error('Error in batch import of product_current_prices:', error);
+        errors.push({
+          table: 'imported_product_current_prices',
+          error: error.message,
+          offset: offset,
+          batchSize: batchSize
+        });
+        
+        // Try to continue with next batch
+        offset += batchSize;
+      }
+    }
+
+    // Update sync status
+    await localConnection.query(`
+      INSERT INTO sync_status (table_name, last_sync_timestamp)
+      VALUES ('imported_product_current_prices', NOW())
+      ON CONFLICT (table_name) DO UPDATE SET
+        last_sync_timestamp = NOW()
+    `);
+
+    return { recordsAdded, recordsUpdated, totalProcessed, errors };
+  } catch (error) {
+    console.error('Error in product current prices import:', error);
+    return { 
+      recordsAdded, 
+      recordsUpdated, 
+      totalProcessed, 
+      errors: [...errors, { 
+        table: 'imported_product_current_prices',
+        error: error.message
+      }] 
+    };
+  }
+}
+
+/**
+ * Imports daily_inventory data from MySQL to PostgreSQL
+ */
+async function importDailyInventory(
+  prodConnection, 
+  localConnection, 
+  oneYearAgoStr, 
+  lastSyncTime, 
+  batchSize,
+  insertBatchSize,
+  incrementalUpdate,
+  startTime
+) {
+  let recordsAdded = 0;
+  let recordsUpdated = 0;
+  let totalProcessed = 0;
+  let errors = [];
+  let offset = 0;
+  let allProcessed = false;
+
+  try {
+    // Get total count for progress reporting
+    const [countResult] = await prodConnection.query(`
+      SELECT COUNT(*) as total
+      FROM daily_inventory
+      WHERE date >= ?
+        ${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
+    `, [oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
+
+    const totalCount = countResult[0].total;
+
+    outputProgress({
+      status: "running",
+      operation: "Historical data import - Daily Inventory",
+      message: `Found ${totalCount} records to process`,
+      current: 0,
+      total: totalCount,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    // Process in batches for better performance
+    while (!allProcessed) {
+      try {
+        // Fetch batch from production
+        const [rows] = await prodConnection.query(`
+          SELECT
+            date,
+            pid,
+            amountsold,
+            times_sold,
+            qtyreceived,
+            price,
+            costeach,
+            stamp
+          FROM daily_inventory
+          WHERE date >= ?
+            ${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
+          ORDER BY date, pid
+          LIMIT ? OFFSET ?
+        `, [
+          oneYearAgoStr, 
+          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
+          batchSize,
+          offset
+        ]);
+
+        if (rows.length === 0) {
+          allProcessed = true;
+          break;
+        }
+
+        // Process rows in smaller batches for better performance
+        for (let i = 0; i < rows.length; i += insertBatchSize) {
+          const batch = rows.slice(i, i + insertBatchSize);
+          
+          if (batch.length === 0) continue;
+          
+          try {
+            // Build parameterized query to handle NULL values properly
+            const values = [];
+            const placeholders = [];
+            let placeholderIndex = 1;
+            
+            for (const row of batch) {
+              const rowPlaceholders = [
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`
+              ];
+              
+              placeholders.push(`(${rowPlaceholders.join(', ')})`);
+              
+              values.push(
+                validateDate(row.date),
+                row.pid,
+                row.amountsold || 0,
+                row.times_sold || 0,
+                row.qtyreceived || 0,
+                row.price || 0,
+                row.costeach || 0,
+                validateDate(row.stamp)
+              );
+            }
+            
+            // Execute batch insert
+            const result = await localConnection.query(`
+              WITH ins AS (
+                INSERT INTO imported_daily_inventory (
+                  date, pid, amountsold, times_sold, qtyreceived, price, costeach, stamp
+                )
+                VALUES ${placeholders.join(',\n')}
+                ON CONFLICT (date, pid) DO UPDATE SET
+                  amountsold = EXCLUDED.amountsold,
+                  times_sold = EXCLUDED.times_sold,
+                  qtyreceived = EXCLUDED.qtyreceived,
+                  price = EXCLUDED.price,
+                  costeach = EXCLUDED.costeach,
+                  stamp = EXCLUDED.stamp,
+                  updated = CURRENT_TIMESTAMP
+                RETURNING (xmax = 0) AS inserted
+              )
+              SELECT 
+                COUNT(*) FILTER (WHERE inserted) AS inserted_count, 
+                COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
+              FROM ins
+            `, values);
+            
+            // Safely update counts based on the result
+            if (result && result.rows && result.rows.length > 0) {
+              const insertedCount = parseInt(result.rows[0].inserted_count || 0);
+              const updatedCount = parseInt(result.rows[0].updated_count || 0);
+              
+              recordsAdded += insertedCount;
+              recordsUpdated += updatedCount;
+            }
+          } catch (error) {
+            console.error(`Error in batch import of daily_inventory at offset ${i}:`, error);
+            errors.push({
+              table: 'imported_daily_inventory',
+              batchOffset: i,
+              batchSize: batch.length,
+              error: error.message
+            });
+          }
+        }
+
+        totalProcessed += rows.length;
+        offset += rows.length;
+
+        // Update progress
+        outputProgress({
+          status: "running",
+          operation: "Historical data import - Daily Inventory",
+          message: `Processed ${totalProcessed} of ${totalCount} records`,
+          current: totalProcessed,
+          total: totalCount,
+          elapsed: formatElapsedTime(startTime),
+          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
+          rate: calculateRate(startTime, totalProcessed)
+        });
+      } catch (error) {
+        console.error('Error in batch import of daily_inventory:', error);
+        errors.push({
+          table: 'imported_daily_inventory',
+          error: error.message,
+          offset: offset,
+          batchSize: batchSize
+        });
+        
+        // Try to continue with next batch
+        offset += batchSize;
+      }
+    }
+
+    // Update sync status
+    await localConnection.query(`
+      INSERT INTO sync_status (table_name, last_sync_timestamp)
+      VALUES ('imported_daily_inventory', NOW())
+      ON CONFLICT (table_name) DO UPDATE SET
+        last_sync_timestamp = NOW()
+    `);
+
+    return { recordsAdded, recordsUpdated, totalProcessed, errors };
+  } catch (error) {
+    console.error('Error in daily inventory import:', error);
+    return { 
+      recordsAdded, 
+      recordsUpdated, 
+      totalProcessed, 
+      errors: [...errors, { 
+        table: 'imported_daily_inventory',
+        error: error.message
+      }] 
+    };
+  }
+}
+
+/**
+ * Imports product_stat_history data from MySQL to PostgreSQL
+ * Using fast direct inserts without conflict checking
+ */
+async function importProductStatHistory(
+  prodConnection, 
+  localConnection, 
+  recentDateStr, // Use more recent date instead of one year ago
+  lastSyncTime, 
+  batchSize,
+  insertBatchSize,
+  incrementalUpdate,
+  startTime,
+  recentMonths // Add parameter for recent months
+) {
+  let recordsAdded = 0;
+  let recordsUpdated = 0;
+  let totalProcessed = 0;
+  let errors = [];
+  let offset = 0;
+  let allProcessed = false;
+  let lastRateCheck = Date.now();
+  let lastProcessed = 0;
+
+  try {
+    // Get total count for progress reporting
+    const [countResult] = await prodConnection.query(`
+      SELECT COUNT(*) as total
+      FROM product_stat_history
+      WHERE date >= ?
+        ${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
+    `, [recentDateStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
+
+    const totalCount = countResult[0].total;
+    console.log(`Found ${totalCount} records to process in product_stat_history (using recent date: ${recentDateStr})`);
+
+    // Progress indicator
+    outputProgress({
+      status: "running",
+      operation: "Historical data import - Product Stat History",
+      message: `Found ${totalCount} records to process (last ${recentMonths} months only)`,
+      current: 0,
+      total: totalCount,
+      elapsed: formatElapsedTime(startTime)
+    });
+
+    // If not incremental, truncate the table first for better performance
+    if (!incrementalUpdate) {
+      console.log('Truncating imported_product_stat_history for full import...');
+      await localConnection.query('TRUNCATE TABLE imported_product_stat_history');
+    } else if (lastSyncTime) {
+      // For incremental updates, delete records that will be reimported
+      console.log(`Deleting records from imported_product_stat_history since ${lastSyncTime}...`);
+      await localConnection.query('DELETE FROM imported_product_stat_history WHERE date > $1', [lastSyncTime]);
+    }
+
+    // Process in batches for better performance
+    while (!allProcessed) {
+      try {
+        // Fetch batch from production with minimal filtering and no sorting
+        const [rows] = await prodConnection.query(`
+          SELECT
+            pid,
+            date,
+            COALESCE(score, 0) as score,
+            COALESCE(score2, 0) as score2,
+            COALESCE(qty_in_baskets, 0) as qty_in_baskets,
+            COALESCE(qty_sold, 0) as qty_sold,
+            COALESCE(notifies_set, 0) as notifies_set,
+            COALESCE(visibility_score, 0) as visibility_score,
+            COALESCE(health_score, 0) as health_score,
+            COALESCE(sold_view_score, 0) as sold_view_score
+          FROM product_stat_history
+          WHERE date >= ?
+            ${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
+          LIMIT ? OFFSET ?
+        `, [
+          recentDateStr, 
+          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
+          batchSize,
+          offset
+        ]);
+
+        if (rows.length === 0) {
+          allProcessed = true;
+          break;
+        }
+
+        // Process rows in smaller batches for better performance
+        for (let i = 0; i < rows.length; i += insertBatchSize) {
+          const batch = rows.slice(i, i + insertBatchSize);
+          
+          if (batch.length === 0) continue;
+          
+          try {
+            // Build parameterized query to handle NULL values properly
+            const values = [];
+            const placeholders = [];
+            let placeholderIndex = 1;
+            
+            for (const row of batch) {
+              const rowPlaceholders = [
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`,
+                `$${placeholderIndex++}`
+              ];
+              
+              placeholders.push(`(${rowPlaceholders.join(', ')})`);
+              
+              values.push(
+                row.pid,
+                validateDate(row.date),
+                row.score,
+                row.score2,
+                row.qty_in_baskets,
+                row.qty_sold,
+                row.notifies_set,
+                row.visibility_score,
+                row.health_score,
+                row.sold_view_score
+              );
+            }
+            
+            // Execute direct batch insert without conflict checking
+            await localConnection.query(`
+              INSERT INTO imported_product_stat_history (
+                pid, date, score, score2, qty_in_baskets, qty_sold, notifies_set,
+                visibility_score, health_score, sold_view_score
+              )
+              VALUES ${placeholders.join(',\n')}
+            `, values);
+            
+            // All inserts are new records when using this approach
+            recordsAdded += batch.length;
+          } catch (error) {
+            console.error(`Error in batch insert of product_stat_history at offset ${i}:`, error);
+            errors.push({
+              table: 'imported_product_stat_history',
+              batchOffset: i,
+              batchSize: batch.length,
+              error: error.message
+            });
+          }
+        }
+
+        totalProcessed += rows.length;
+        offset += rows.length;
+
+        // Calculate current rate every 10 seconds or 100,000 records
+        const now = Date.now();
+        if (now - lastRateCheck > 10000 || totalProcessed - lastProcessed > 100000) {
+          const timeElapsed = (now - lastRateCheck) / 1000; // seconds
+          const recordsProcessed = totalProcessed - lastProcessed;
+          const currentRate = Math.round(recordsProcessed / timeElapsed);
+          
+          console.log(`Current import rate: ${currentRate} records/second`);
+          
+          lastRateCheck = now;
+          lastProcessed = totalProcessed;
+        }
+
+        // Update progress
+        outputProgress({
+          status: "running",
+          operation: "Historical data import - Product Stat History",
+          message: `Processed ${totalProcessed} of ${totalCount} records`,
+          current: totalProcessed,
+          total: totalCount,
+          elapsed: formatElapsedTime(startTime),
+          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
+          rate: calculateRate(startTime, totalProcessed)
+        });
+      } catch (error) {
+        console.error('Error in batch import of product_stat_history:', error);
+        errors.push({
+          table: 'imported_product_stat_history',
+          error: error.message,
+          offset: offset,
+          batchSize: batchSize
+        });
+        
+        // Try to continue with next batch
+        offset += batchSize;
+      }
+    }
+
+    // Update sync status
+    await localConnection.query(`
+      INSERT INTO sync_status (table_name, last_sync_timestamp)
+      VALUES ('imported_product_stat_history', NOW())
+      ON CONFLICT (table_name) DO UPDATE SET
+        last_sync_timestamp = NOW()
+    `);
+
+    return { recordsAdded, recordsUpdated, totalProcessed, errors };
+  } catch (error) {
+    console.error('Error in product stat history import:', error);
+    return { 
+      recordsAdded, 
+      recordsUpdated, 
+      totalProcessed, 
+      errors: [...errors, { 
+        table: 'imported_product_stat_history',
+        error: error.message
+      }] 
+    };
+  }
+}
+
+module.exports = importHistoricalData; 

inventory-server/old/metrics-schema.sql

@@ -11,15 +11,17 @@ CREATE TABLE temp_sales_metrics (
     avg_margin_percent DECIMAL(10,3),
     first_sale_date DATE,
     last_sale_date DATE,
+    stddev_daily_sales DECIMAL(10,3),
     PRIMARY KEY (pid)
 );
 
 CREATE TABLE temp_purchase_metrics (
     pid BIGINT NOT NULL,
-    avg_lead_time_days INTEGER,
+    avg_lead_time_days DECIMAL(10,2),
     last_purchase_date DATE,
     first_received_date DATE,
     last_received_date DATE,
+    stddev_lead_time_days DECIMAL(10,2),
     PRIMARY KEY (pid)
 );
 
@@ -50,7 +52,7 @@ CREATE TABLE product_metrics (
     gross_profit DECIMAL(10,3),
     gmroi DECIMAL(10,3),
     -- Purchase metrics
-    avg_lead_time_days INTEGER,
+    avg_lead_time_days DECIMAL(10,2),
     last_purchase_date DATE,
     first_received_date DATE,
     last_received_date DATE,

inventory-server/old/metrics/brand-metrics.js

@@ -32,12 +32,12 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
         }
 
         // Get order count that will be processed
-        const [orderCount] = await connection.query(`
+        const orderCount = await connection.query(`
             SELECT COUNT(*) as count 
             FROM orders o 
             WHERE o.canceled = false
         `);
-        processedOrders = orderCount[0].count;
+        processedOrders = parseInt(orderCount.rows[0].count);
 
         outputProgress({
             status: 'running',
@@ -98,14 +98,14 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
                     SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as period_margin,
                     COUNT(DISTINCT DATE(o.date)) as period_days,
                     CASE 
-                        WHEN o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH) THEN 'current'
+                        WHEN o.date >= CURRENT_DATE - INTERVAL '3 months' THEN 'current'
-                        WHEN o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                        WHEN o.date BETWEEN CURRENT_DATE - INTERVAL '15 months' 
-                            AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH) THEN 'previous'
+                            AND CURRENT_DATE - INTERVAL '12 months' THEN 'previous'
                     END as period_type
                 FROM filtered_products p
                 JOIN orders o ON p.pid = o.pid 
                 WHERE o.canceled = false
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH)
+                    AND o.date >= CURRENT_DATE - INTERVAL '15 months'
                 GROUP BY p.brand, period_type
             ),
             brand_data AS (
@@ -165,15 +165,16 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
             LEFT JOIN sales_periods sp ON bd.brand = sp.brand
             GROUP BY bd.brand, bd.product_count, bd.active_products, bd.total_stock_units, 
                      bd.total_stock_cost, bd.total_stock_retail, bd.total_revenue, bd.avg_margin
-            ON DUPLICATE KEY UPDATE
+            ON CONFLICT (brand) DO UPDATE
-                product_count = VALUES(product_count),
+            SET
-                active_products = VALUES(active_products),
+                product_count = EXCLUDED.product_count,
-                total_stock_units = VALUES(total_stock_units),
+                active_products = EXCLUDED.active_products,
-                total_stock_cost = VALUES(total_stock_cost),
+                total_stock_units = EXCLUDED.total_stock_units,
-                total_stock_retail = VALUES(total_stock_retail),
+                total_stock_cost = EXCLUDED.total_stock_cost,
-                total_revenue = VALUES(total_revenue),
+                total_stock_retail = EXCLUDED.total_stock_retail,
-                avg_margin = VALUES(avg_margin),
+                total_revenue = EXCLUDED.total_revenue,
-                growth_rate = VALUES(growth_rate),
+                avg_margin = EXCLUDED.avg_margin,
+                growth_rate = EXCLUDED.growth_rate,
                 last_calculated_at = CURRENT_TIMESTAMP
         `);
 
@@ -230,8 +231,8 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
             monthly_metrics AS (
                 SELECT 
                     p.brand,
-                    YEAR(o.date) as year,
+                    EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
-                    MONTH(o.date) as month,
+                    EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
                     COUNT(DISTINCT p.valid_pid) as product_count,
                     COUNT(DISTINCT p.active_pid) as active_products,
                     SUM(p.valid_stock) as total_stock_units,
@@ -255,19 +256,20 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
                     END as avg_margin
                 FROM filtered_products p
                 LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
-                WHERE o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                WHERE o.date >= CURRENT_DATE - INTERVAL '12 months'
-                GROUP BY p.brand, YEAR(o.date), MONTH(o.date)
+                GROUP BY p.brand, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
             )
             SELECT *
             FROM monthly_metrics
-            ON DUPLICATE KEY UPDATE
+            ON CONFLICT (brand, year, month) DO UPDATE
-                product_count = VALUES(product_count),
+            SET
-                active_products = VALUES(active_products),
+                product_count = EXCLUDED.product_count,
-                total_stock_units = VALUES(total_stock_units),
+                active_products = EXCLUDED.active_products,
-                total_stock_cost = VALUES(total_stock_cost),
+                total_stock_units = EXCLUDED.total_stock_units,
-                total_stock_retail = VALUES(total_stock_retail),
+                total_stock_cost = EXCLUDED.total_stock_cost,
-                total_revenue = VALUES(total_revenue),
+                total_stock_retail = EXCLUDED.total_stock_retail,
-                avg_margin = VALUES(avg_margin)
+                total_revenue = EXCLUDED.total_revenue,
+                avg_margin = EXCLUDED.avg_margin
         `);
 
         processedCount = Math.floor(totalProducts * 0.99);
@@ -294,7 +296,8 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount =
         await connection.query(`
             INSERT INTO calculate_status (module_name, last_calculation_timestamp)
             VALUES ('brand_metrics', NOW())
-            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+            ON CONFLICT (module_name) DO UPDATE
+            SET last_calculation_timestamp = NOW()
         `);
 
         return {

inventory-server/old/metrics/category-metrics.js

@@ -32,12 +32,12 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
         }
 
         // Get order count that will be processed
-        const [orderCount] = await connection.query(`
+        const orderCount = await connection.query(`
             SELECT COUNT(*) as count 
             FROM orders o 
             WHERE o.canceled = false
         `);
-        processedOrders = orderCount[0].count;
+        processedOrders = parseInt(orderCount.rows[0].count);
 
         outputProgress({
             status: 'running',
@@ -76,12 +76,13 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             LEFT JOIN product_categories pc ON c.cat_id = pc.cat_id
             LEFT JOIN products p ON pc.pid = p.pid
             GROUP BY c.cat_id, c.status
-            ON DUPLICATE KEY UPDATE
+            ON CONFLICT (category_id) DO UPDATE
-                product_count = VALUES(product_count),
+            SET
-                active_products = VALUES(active_products),
+                product_count = EXCLUDED.product_count,
-                total_value = VALUES(total_value),
+                active_products = EXCLUDED.active_products,
-                status = VALUES(status),
+                total_value = EXCLUDED.total_value,
-                last_calculated_at = VALUES(last_calculated_at)
+                status = EXCLUDED.status,
+                last_calculated_at = EXCLUDED.last_calculated_at
         `);
 
         processedCount = Math.floor(totalProducts * 0.90);
@@ -127,17 +128,13 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                     (tc.category_id IS NULL AND tc.vendor = p.vendor) OR
                     (tc.category_id IS NULL AND tc.vendor IS NULL)
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL COALESCE(tc.calculation_period_days, 30) DAY)
+                    AND o.date >= CURRENT_DATE - (COALESCE(tc.calculation_period_days, 30) || ' days')::INTERVAL
                 GROUP BY pc.cat_id
             )
-            UPDATE category_metrics cm
+            UPDATE category_metrics
-            JOIN category_sales cs ON cm.category_id = cs.cat_id
-            LEFT JOIN turnover_config tc ON 
-                (tc.category_id = cm.category_id AND tc.vendor IS NULL) OR
-                (tc.category_id IS NULL AND tc.vendor IS NULL)
             SET 
-                cm.avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
+                avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
-                cm.turnover_rate = CASE 
+                turnover_rate = CASE 
                     WHEN cs.avg_stock > 0 AND cs.active_days > 0 
                     THEN LEAST(
                         (cs.units_sold / cs.avg_stock) * (365.0 / cs.active_days),
@@ -145,7 +142,9 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                     )
                     ELSE 0 
                 END,
-                cm.last_calculated_at = NOW()
+                last_calculated_at = NOW()
+            FROM category_sales cs
+            WHERE category_id = cs.cat_id
         `);
 
         processedCount = Math.floor(totalProducts * 0.95);
@@ -184,9 +183,9 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
+                    AND o.date >= CURRENT_DATE - INTERVAL '3 months'
                 GROUP BY pc.cat_id
             ),
             previous_period AS (
@@ -198,26 +197,26 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
                 WHERE o.canceled = false 
-                    AND o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                    AND o.date BETWEEN CURRENT_DATE - INTERVAL '15 months' 
-                    AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND CURRENT_DATE - INTERVAL '12 months'
                 GROUP BY pc.cat_id
             ),
             trend_data AS (
                 SELECT 
                     pc.cat_id,
-                    MONTH(o.date) as month,
+                    EXTRACT(MONTH FROM o.date) as month,
                     SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
                         (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
                     COUNT(DISTINCT DATE(o.date)) as days_in_month
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH)
+                    AND o.date >= CURRENT_DATE - INTERVAL '15 months'
-                GROUP BY pc.cat_id, MONTH(o.date)
+                GROUP BY pc.cat_id, EXTRACT(MONTH FROM o.date)
             ),
             trend_stats AS (
                 SELECT 
@@ -261,16 +260,42 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
+                    AND o.date >= CURRENT_DATE - INTERVAL '3 months'
                 GROUP BY pc.cat_id
+            ),
+            combined_metrics AS (
+                SELECT 
+                    COALESCE(cp.cat_id, pp.cat_id) as category_id,
+                    CASE
+                        WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
+                        WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
+                        WHEN ta.trend_slope IS NOT NULL THEN
+                            GREATEST(
+                                -100.0,
+                                LEAST(
+                                    (ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
+                                    999.99
+                                )
+                            )
+                        ELSE
+                            GREATEST(
+                                -100.0,
+                                LEAST(
+                                    ((COALESCE(cp.revenue, 0) - pp.revenue) / 
+                                        NULLIF(ABS(pp.revenue), 0)) * 100.0,
+                                    999.99
+                                )
+                            )
+                    END as growth_rate,
+                    mc.avg_margin
+                FROM current_period cp
+                FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
+                LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
+                LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
             )
             UPDATE category_metrics cm
-            LEFT JOIN current_period cp ON cm.category_id = cp.cat_id
-            LEFT JOIN previous_period pp ON cm.category_id = pp.cat_id
-            LEFT JOIN trend_analysis ta ON cm.category_id = ta.cat_id
-            LEFT JOIN margin_calc mc ON cm.category_id = mc.cat_id
             SET 
-                cm.growth_rate = CASE
+                growth_rate = CASE
                     WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
                     WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
                     WHEN ta.trend_slope IS NOT NULL THEN
@@ -291,9 +316,13 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                             )
                         )
                 END,
-                cm.avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
+                avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
-                cm.last_calculated_at = NOW()
+                last_calculated_at = NOW()
-            WHERE cp.cat_id IS NOT NULL OR pp.cat_id IS NOT NULL
+            FROM current_period cp
+            FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
+            LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
+            LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
+            WHERE cm.category_id = COALESCE(cp.cat_id, pp.cat_id)
         `);
 
         processedCount = Math.floor(totalProducts * 0.97);
@@ -335,8 +364,8 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             )
             SELECT 
                 pc.cat_id,
-                YEAR(o.date) as year,
+                EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
-                MONTH(o.date) as month,
+                EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
                 COUNT(DISTINCT p.pid) as product_count,
                 COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
                 SUM(p.stock_quantity * p.cost_price) as total_value,
@@ -364,15 +393,16 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             JOIN products p ON pc.pid = p.pid
             JOIN orders o ON p.pid = o.pid
             WHERE o.canceled = false
-                AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                AND o.date >= CURRENT_DATE - INTERVAL '12 months'
-            GROUP BY pc.cat_id, YEAR(o.date), MONTH(o.date)
+            GROUP BY pc.cat_id, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
-            ON DUPLICATE KEY UPDATE
+            ON CONFLICT (category_id, year, month) DO UPDATE
-                product_count = VALUES(product_count),
+            SET
-                active_products = VALUES(active_products),
+                product_count = EXCLUDED.product_count,
-                total_value = VALUES(total_value),
+                active_products = EXCLUDED.active_products,
-                total_revenue = VALUES(total_revenue),
+                total_value = EXCLUDED.total_value,
-                avg_margin = VALUES(avg_margin),
+                total_revenue = EXCLUDED.total_revenue,
-                turnover_rate = VALUES(turnover_rate)
+                avg_margin = EXCLUDED.avg_margin,
+                turnover_rate = EXCLUDED.turnover_rate
         `);
 
         processedCount = Math.floor(totalProducts * 0.99);
@@ -414,20 +444,20 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             )
             WITH date_ranges AS (
                 SELECT 
-                    DATE_SUB(CURRENT_DATE, INTERVAL 30 DAY) as period_start,
+                    CURRENT_DATE - INTERVAL '30 days' as period_start,
                     CURRENT_DATE as period_end
                 UNION ALL
                 SELECT 
-                    DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY),
+                    CURRENT_DATE - INTERVAL '90 days',
-                    DATE_SUB(CURRENT_DATE, INTERVAL 31 DAY)
+                    CURRENT_DATE - INTERVAL '31 days'
                 UNION ALL
                 SELECT 
-                    DATE_SUB(CURRENT_DATE, INTERVAL 180 DAY),
+                    CURRENT_DATE - INTERVAL '180 days',
-                    DATE_SUB(CURRENT_DATE, INTERVAL 91 DAY)
+                    CURRENT_DATE - INTERVAL '91 days'
                 UNION ALL
                 SELECT 
-                    DATE_SUB(CURRENT_DATE, INTERVAL 365 DAY),
+                    CURRENT_DATE - INTERVAL '365 days',
-                    DATE_SUB(CURRENT_DATE, INTERVAL 181 DAY)
+                    CURRENT_DATE - INTERVAL '181 days'
             ),
             sales_data AS (
                 SELECT 
@@ -466,12 +496,13 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 END as avg_price,
                 NOW() as last_calculated_at
             FROM sales_data
-            ON DUPLICATE KEY UPDATE
+            ON CONFLICT (category_id, brand, period_start, period_end) DO UPDATE
-                avg_daily_sales = VALUES(avg_daily_sales),
+            SET
-                total_sold = VALUES(total_sold),
+                avg_daily_sales = EXCLUDED.avg_daily_sales,
-                num_products = VALUES(num_products),
+                total_sold = EXCLUDED.total_sold,
-                avg_price = VALUES(avg_price),
+                num_products = EXCLUDED.num_products,
-                last_calculated_at = VALUES(last_calculated_at)
+                avg_price = EXCLUDED.avg_price,
+                last_calculated_at = EXCLUDED.last_calculated_at
         `);
 
         processedCount = Math.floor(totalProducts * 1.0);
@@ -498,7 +529,8 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
         await connection.query(`
             INSERT INTO calculate_status (module_name, last_calculation_timestamp)
             VALUES ('category_metrics', NOW())
-            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+            ON CONFLICT (module_name) DO UPDATE
+            SET last_calculation_timestamp = NOW()
         `);
 
         return {

inventory-server/old/metrics/financial-metrics.js

@@ -32,13 +32,13 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
         }
 
         // Get order count that will be processed
-        const [orderCount] = await connection.query(`
+        const orderCount = await connection.query(`
             SELECT COUNT(*) as count 
             FROM orders o 
             WHERE o.canceled = false
-                AND DATE(o.date) >= DATE_SUB(CURDATE(), INTERVAL 12 MONTH)
+                AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'
         `);
-        processedOrders = orderCount[0].count;
+        processedOrders = parseInt(orderCount.rows[0].count);
 
         outputProgress({
             status: 'running',
@@ -56,38 +56,97 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             }
         });
 
-        // Calculate financial metrics with optimized query
+        // First, calculate beginning inventory values (12 months ago)
+        await connection.query(`
+            CREATE TEMPORARY TABLE IF NOT EXISTS temp_beginning_inventory AS
+            WITH beginning_inventory_calc AS (
+                SELECT 
+                    p.pid,
+                    p.stock_quantity as current_quantity,
+                    COALESCE(SUM(o.quantity), 0) as sold_quantity,
+                    COALESCE(SUM(po.received), 0) as received_quantity,
+                    GREATEST(0, (p.stock_quantity + COALESCE(SUM(o.quantity), 0) - COALESCE(SUM(po.received), 0))) as beginning_quantity,
+                    p.cost_price
+                FROM 
+                    products p
+                LEFT JOIN 
+                    orders o ON p.pid = o.pid 
+                    AND o.canceled = false 
+                    AND o.date >= CURRENT_DATE - INTERVAL '12 months'::interval
+                LEFT JOIN 
+                    purchase_orders po ON p.pid = po.pid 
+                    AND po.received_date IS NOT NULL 
+                    AND po.received_date >= CURRENT_DATE - INTERVAL '12 months'::interval
+                GROUP BY 
+                    p.pid, p.stock_quantity, p.cost_price
+            )
+            SELECT 
+                pid,
+                beginning_quantity,
+                beginning_quantity * cost_price as beginning_value,
+                current_quantity * cost_price as current_value,
+                ((beginning_quantity * cost_price) + (current_quantity * cost_price)) / 2 as average_inventory_value
+            FROM 
+                beginning_inventory_calc
+        `);
+
+        processedCount = Math.floor(totalProducts * 0.60);
+        outputProgress({
+            status: 'running',
+            operation: 'Beginning inventory values calculated, computing financial metrics',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        // Calculate financial metrics with optimized query and standard formulas
         await connection.query(`
             WITH product_financials AS (
                 SELECT 
                     p.pid,
-                    p.cost_price * p.stock_quantity as inventory_value,
+                    COALESCE(bi.average_inventory_value, p.cost_price * p.stock_quantity) as avg_inventory_value,
-                    SUM(o.quantity * o.price) as total_revenue,
+                    p.cost_price * p.stock_quantity as current_inventory_value,
-                    SUM(o.quantity * p.cost_price) as cost_of_goods_sold,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as total_revenue,
-                    SUM(o.quantity * (o.price - p.cost_price)) as gross_profit,
+                    SUM(o.quantity * COALESCE(o.costeach, 0)) as cost_of_goods_sold,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - COALESCE(o.costeach, 0))) as gross_profit,
                     MIN(o.date) as first_sale_date,
                     MAX(o.date) as last_sale_date,
-                    DATEDIFF(MAX(o.date), MIN(o.date)) + 1 as calculation_period_days,
+                    EXTRACT(DAY FROM (MAX(o.date)::timestamp with time zone - MIN(o.date)::timestamp with time zone)) + 1 as calculation_period_days,
                     COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM products p
                 LEFT JOIN orders o ON p.pid = o.pid
+                LEFT JOIN temp_beginning_inventory bi ON p.pid = bi.pid
                 WHERE o.canceled = false
-                    AND DATE(o.date) >= DATE_SUB(CURDATE(), INTERVAL 12 MONTH)
+                    AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'::interval
-                GROUP BY p.pid
+                GROUP BY p.pid, p.cost_price, p.stock_quantity, bi.average_inventory_value
             )
             UPDATE product_metrics pm
-            JOIN product_financials pf ON pm.pid = pf.pid
             SET 
-                pm.inventory_value = COALESCE(pf.inventory_value, 0),
+                inventory_value = COALESCE(pf.current_inventory_value, 0)::decimal(10,3),
-                pm.total_revenue = COALESCE(pf.total_revenue, 0),
+                total_revenue = COALESCE(pf.total_revenue, 0)::decimal(10,3),
-                pm.cost_of_goods_sold = COALESCE(pf.cost_of_goods_sold, 0),
+                cost_of_goods_sold = COALESCE(pf.cost_of_goods_sold, 0)::decimal(10,3),
-                pm.gross_profit = COALESCE(pf.gross_profit, 0),
+                gross_profit = COALESCE(pf.gross_profit, 0)::decimal(10,3),
-                pm.gmroi = CASE 
+                turnover_rate = CASE 
-                    WHEN COALESCE(pf.inventory_value, 0) > 0 AND pf.active_days > 0 THEN 
+                    WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN 
-                        (COALESCE(pf.gross_profit, 0) * (365.0 / pf.active_days)) / COALESCE(pf.inventory_value, 0)
+                        COALESCE(pf.cost_of_goods_sold, 0) / NULLIF(pf.avg_inventory_value, 0)
                     ELSE 0 
-                END,
+                END::decimal(12,3),
-                pm.last_calculated_at = CURRENT_TIMESTAMP
+                gmroi = CASE 
+                    WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN 
+                        COALESCE(pf.gross_profit, 0) / NULLIF(pf.avg_inventory_value, 0)
+                    ELSE 0 
+                END::decimal(10,3),
+                last_calculated_at = CURRENT_TIMESTAMP
+            FROM product_financials pf
+            WHERE pm.pid = pf.pid
         `);
 
         processedCount = Math.floor(totalProducts * 0.65);
@@ -114,52 +173,8 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             success
         };
 
-        // Update time-based aggregates with optimized query
+        // Clean up temporary tables
-        await connection.query(`
+        await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
-            WITH monthly_financials AS (
-                SELECT 
-                    p.pid,
-                    YEAR(o.date) as year,
-                    MONTH(o.date) as month,
-                    p.cost_price * p.stock_quantity as inventory_value,
-                    SUM(o.quantity * (o.price - p.cost_price)) as gross_profit,
-                    COUNT(DISTINCT DATE(o.date)) as active_days,
-                    MIN(o.date) as period_start,
-                    MAX(o.date) as period_end
-                FROM products p
-                LEFT JOIN orders o ON p.pid = o.pid
-                WHERE o.canceled = false
-                GROUP BY p.pid, YEAR(o.date), MONTH(o.date)
-            )
-            UPDATE product_time_aggregates pta
-            JOIN monthly_financials mf ON pta.pid = mf.pid 
-                AND pta.year = mf.year 
-                AND pta.month = mf.month
-            SET 
-                pta.inventory_value = COALESCE(mf.inventory_value, 0),
-                pta.gmroi = CASE 
-                    WHEN COALESCE(mf.inventory_value, 0) > 0 AND mf.active_days > 0 THEN 
-                        (COALESCE(mf.gross_profit, 0) * (365.0 / mf.active_days)) / COALESCE(mf.inventory_value, 0)
-                    ELSE 0 
-                END
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.70);
-        outputProgress({
-            status: 'running',
-            operation: 'Time-based aggregates updated',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
 
         // If we get here, everything completed successfully
         success = true;
@@ -168,7 +183,8 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
         await connection.query(`
             INSERT INTO calculate_status (module_name, last_calculation_timestamp)
             VALUES ('financial_metrics', NOW())
-            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+            ON CONFLICT (module_name) DO UPDATE
+            SET last_calculation_timestamp = NOW()
         `);
 
         return {
@@ -184,6 +200,12 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
         throw error;
     } finally {
         if (connection) {
+            try {
+                // Make sure temporary tables are always cleaned up
+                await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
+            } catch (err) {
+                console.error('Error cleaning up temp tables:', err);
+            }
             connection.release();
         }
     }

inventory-server/old/metrics/product-metrics.js

@@ -0,0 +1,736 @@
+const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
+const { getConnection } = require('./utils/db');
+
+// Helper function to handle NaN and undefined values
+function sanitizeValue(value) {
+    if (value === undefined || value === null || Number.isNaN(value)) {
+        return null;
+    }
+    return value;
+}
+
+async function calculateProductMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
+    let connection;
+    let success = false;
+    let processedOrders = 0;
+    const BATCH_SIZE = 5000;
+    
+    try {
+        connection = await getConnection();
+        // Skip flags are inherited from the parent scope
+        const SKIP_PRODUCT_BASE_METRICS = 0;
+        const SKIP_PRODUCT_TIME_AGGREGATES = 0;
+
+        // Get total product count if not provided
+        if (!totalProducts) {
+            const productCount = await connection.query('SELECT COUNT(*) as count FROM products');
+            totalProducts = parseInt(productCount.rows[0].count);
+        }
+
+        if (isCancelled) {
+            outputProgress({
+                status: 'cancelled',
+                operation: 'Product metrics calculation cancelled',
+                current: processedCount,
+                total: totalProducts,
+                elapsed: formatElapsedTime(startTime),
+                remaining: null,
+                rate: calculateRate(startTime, processedCount),
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // First ensure all products have a metrics record
+        await connection.query(`
+            INSERT INTO product_metrics (pid, last_calculated_at)
+            SELECT pid, NOW()
+            FROM products
+            ON CONFLICT (pid) DO NOTHING
+        `);
+
+        // Get threshold settings once
+        const thresholds = await connection.query(`
+            SELECT critical_days, reorder_days, overstock_days, low_stock_threshold
+            FROM stock_thresholds 
+            WHERE category_id IS NULL AND vendor IS NULL
+            LIMIT 1
+        `);
+        
+        // Check if threshold data was returned
+        if (!thresholds.rows || thresholds.rows.length === 0) {
+            console.warn('No default thresholds found in the database. Using explicit type casting in the query.');
+        }
+        
+        const defaultThresholds = thresholds.rows[0];
+
+        // Get financial calculation configuration parameters
+        const financialConfig = await connection.query(`
+            SELECT 
+                order_cost, 
+                holding_rate, 
+                service_level_z_score,
+                min_reorder_qty,
+                default_reorder_qty,
+                default_safety_stock
+            FROM financial_calc_config 
+            WHERE id = 1
+            LIMIT 1
+        `);
+        const finConfig = financialConfig.rows[0] || { 
+            order_cost: 25.00, 
+            holding_rate: 0.25, 
+            service_level_z_score: 1.96,
+            min_reorder_qty: 1,
+            default_reorder_qty: 5,
+            default_safety_stock: 5
+        };
+
+        // Calculate base product metrics
+        if (!SKIP_PRODUCT_BASE_METRICS) {
+            outputProgress({
+                status: 'running',
+                operation: 'Starting base product metrics calculation',
+                current: processedCount,
+                total: totalProducts,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount, totalProducts),
+                rate: calculateRate(startTime, processedCount),
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+
+            // Get order count that will be processed
+            const orderCount = await connection.query(`
+                SELECT COUNT(*) as count 
+                FROM orders o 
+                WHERE o.canceled = false
+            `);
+            processedOrders = parseInt(orderCount.rows[0].count);
+
+            // Clear temporary tables
+            await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
+            await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
+
+            // Create temp_sales_metrics
+            await connection.query(`
+                CREATE TEMPORARY TABLE temp_sales_metrics (
+                    pid BIGINT NOT NULL,
+                    daily_sales_avg DECIMAL(10,3),
+                    weekly_sales_avg DECIMAL(10,3),
+                    monthly_sales_avg DECIMAL(10,3),
+                    total_revenue DECIMAL(10,3),
+                    avg_margin_percent DECIMAL(10,3),
+                    first_sale_date DATE,
+                    last_sale_date DATE,
+                    stddev_daily_sales DECIMAL(10,3),
+                    PRIMARY KEY (pid)
+                )
+            `);
+
+            // Create temp_purchase_metrics
+            await connection.query(`
+                CREATE TEMPORARY TABLE temp_purchase_metrics (
+                    pid BIGINT NOT NULL,
+                    avg_lead_time_days DECIMAL(10,2),
+                    last_purchase_date DATE,
+                    first_received_date DATE,
+                    last_received_date DATE,
+                    stddev_lead_time_days DECIMAL(10,2),
+                    PRIMARY KEY (pid)
+                )
+            `);
+
+            // Populate temp_sales_metrics with base stats and sales averages
+            await connection.query(`
+                INSERT INTO temp_sales_metrics
+                SELECT 
+                    p.pid,
+                    COALESCE(SUM(o.quantity) / NULLIF(COUNT(DISTINCT DATE(o.date)), 0), 0) as daily_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 7), 0), 0) as weekly_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 30), 0), 0) as monthly_sales_avg,
+                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
+                    CASE 
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN ((SUM(o.quantity * o.price) - SUM(o.quantity * p.cost_price)) / SUM(o.quantity * o.price)) * 100 
+                        ELSE 0 
+                    END as avg_margin_percent,
+                    MIN(o.date) as first_sale_date,
+                    MAX(o.date) as last_sale_date,
+                    COALESCE(STDDEV_SAMP(daily_qty.quantity), 0) as stddev_daily_sales
+                FROM products p
+                LEFT JOIN orders o ON p.pid = o.pid 
+                    AND o.canceled = false
+                    AND o.date >= CURRENT_DATE - INTERVAL '90 days'
+                LEFT JOIN (
+                    SELECT 
+                        pid, 
+                        DATE(date) as sale_date, 
+                        SUM(quantity) as quantity
+                    FROM orders
+                    WHERE canceled = false
+                    AND date >= CURRENT_DATE - INTERVAL '90 days'
+                    GROUP BY pid, DATE(date)
+                ) daily_qty ON p.pid = daily_qty.pid
+                GROUP BY p.pid
+            `);
+
+            // Populate temp_purchase_metrics with timeout protection
+            await Promise.race([
+                connection.query(`
+                    INSERT INTO temp_purchase_metrics
+                    SELECT 
+                        p.pid,
+                        AVG(
+                            CASE 
+                                WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL 
+                                THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
+                                ELSE NULL
+                            END
+                        ) as avg_lead_time_days,
+                        MAX(po.date) as last_purchase_date,
+                        MIN(po.received_date) as first_received_date,
+                        MAX(po.received_date) as last_received_date,
+                        STDDEV_SAMP(
+                            CASE 
+                                WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL 
+                                THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
+                                ELSE NULL
+                            END
+                        ) as stddev_lead_time_days
+                    FROM products p
+                    LEFT JOIN purchase_orders po ON p.pid = po.pid 
+                        AND po.received_date IS NOT NULL
+                        AND po.date IS NOT NULL
+                        AND po.date >= CURRENT_DATE - INTERVAL '365 days'
+                    GROUP BY p.pid
+                `),
+                new Promise((_, reject) => 
+                    setTimeout(() => reject(new Error('Timeout: temp_purchase_metrics query took too long')), 60000)
+                )
+            ]).catch(async (err) => {
+                logError(err, 'Error populating temp_purchase_metrics, continuing with empty table');
+                // Create an empty fallback to continue processing
+                await connection.query(`
+                    INSERT INTO temp_purchase_metrics
+                    SELECT 
+                        p.pid,
+                        30.0 as avg_lead_time_days,
+                        NULL as last_purchase_date,
+                        NULL as first_received_date,
+                        NULL as last_received_date,
+                        0.0 as stddev_lead_time_days
+                    FROM products p
+                    LEFT JOIN temp_purchase_metrics tpm ON p.pid = tpm.pid
+                    WHERE tpm.pid IS NULL
+                `);
+            });
+
+            // Process updates in batches
+            let lastPid = 0;
+            let batchCount = 0;
+            const MAX_BATCHES = 1000; // Safety limit for number of batches to prevent infinite loops
+            
+            while (batchCount < MAX_BATCHES) {
+                if (isCancelled) break;
+                
+                batchCount++;
+                const batch = await connection.query(
+                    'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
+                    [lastPid, BATCH_SIZE]
+                );
+
+                if (batch.rows.length === 0) break;
+
+                // Process the entire batch in a single efficient query
+                const lowStockThreshold = parseInt(defaultThresholds?.low_stock_threshold) || 5;
+                const criticalDays = parseInt(defaultThresholds?.critical_days) || 7;
+                const reorderDays = parseInt(defaultThresholds?.reorder_days) || 14;
+                const overstockDays = parseInt(defaultThresholds?.overstock_days) || 90;
+                const serviceLevel = parseFloat(finConfig?.service_level_z_score) || 1.96;
+                const defaultSafetyStock = parseInt(finConfig?.default_safety_stock) || 5;
+                const defaultReorderQty = parseInt(finConfig?.default_reorder_qty) || 5;
+                const orderCost = parseFloat(finConfig?.order_cost) || 25.00;
+                const holdingRate = parseFloat(finConfig?.holding_rate) || 0.25;
+                const minReorderQty = parseInt(finConfig?.min_reorder_qty) || 1;
+
+                await connection.query(`
+                    UPDATE product_metrics pm
+                    SET 
+                        inventory_value = p.stock_quantity * NULLIF(p.cost_price, 0),
+                        daily_sales_avg = COALESCE(sm.daily_sales_avg, 0),
+                        weekly_sales_avg = COALESCE(sm.weekly_sales_avg, 0),
+                        monthly_sales_avg = COALESCE(sm.monthly_sales_avg, 0),
+                        total_revenue = COALESCE(sm.total_revenue, 0),
+                        avg_margin_percent = COALESCE(sm.avg_margin_percent, 0),
+                        first_sale_date = sm.first_sale_date,
+                        last_sale_date = sm.last_sale_date,
+                        avg_lead_time_days = COALESCE(lm.avg_lead_time_days, 30.0),
+                        days_of_inventory = CASE 
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.daily_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        weeks_of_inventory = CASE 
+                            WHEN COALESCE(sm.weekly_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.weekly_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        stock_status = CASE
+                            WHEN p.stock_quantity <= 0 THEN 'Out of Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 AND p.stock_quantity <= ${lowStockThreshold} THEN 'Low Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 THEN 'In Stock'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${criticalDays} THEN 'Critical'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${reorderDays} THEN 'Reorder'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays} THEN 'Overstocked'
+                            ELSE 'Healthy'
+                        END,
+                        safety_stock = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
+                                CEIL(
+                                    ${serviceLevel} * SQRT(
+                                        GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
+                                        POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
+                                    )
+                                )
+                            ELSE ${defaultSafetyStock}
+                        END,
+                        reorder_point = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
+                                CEIL(sm.daily_sales_avg * GREATEST(0, COALESCE(lm.avg_lead_time_days, 30.0))) + 
+                                (CASE
+                                    WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
+                                        CEIL(
+                                            ${serviceLevel} * SQRT(
+                                                GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
+                                                POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
+                                            )
+                                        )
+                                    ELSE ${defaultSafetyStock}
+                                END)
+                            ELSE ${lowStockThreshold}
+                        END,
+                        reorder_qty = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND NULLIF(p.cost_price, 0) IS NOT NULL AND NULLIF(p.cost_price, 0) > 0 THEN
+                                GREATEST(
+                                    CEIL(SQRT(
+                                        (2 * (sm.daily_sales_avg * 365) * ${orderCost}) / 
+                                        NULLIF(p.cost_price * ${holdingRate}, 0) 
+                                    )),
+                                    ${minReorderQty}
+                                )
+                            ELSE ${defaultReorderQty}
+                        END,
+                        overstocked_amt = CASE
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays} 
+                            THEN GREATEST(0, p.stock_quantity - CEIL(sm.daily_sales_avg * ${overstockDays}))
+                            ELSE 0
+                        END,
+                        last_calculated_at = NOW()
+                    FROM products p
+                    LEFT JOIN temp_sales_metrics sm ON p.pid = sm.pid
+                    LEFT JOIN temp_purchase_metrics lm ON p.pid = lm.pid
+                    WHERE p.pid = ANY($1::BIGINT[])
+                    AND pm.pid = p.pid
+                `, [batch.rows.map(row => row.pid)]);
+
+                lastPid = batch.rows[batch.rows.length - 1].pid;
+                processedCount += batch.rows.length;
+
+                outputProgress({
+                    status: 'running',
+                    operation: 'Processing base metrics batch',
+                    current: processedCount,
+                    total: totalProducts,
+                    elapsed: formatElapsedTime(startTime),
+                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
+                    rate: calculateRate(startTime, processedCount),
+                    percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
+                });
+            }
+
+            // Add safety check if the loop processed MAX_BATCHES
+            if (batchCount >= MAX_BATCHES) {
+                logError(new Error(`Reached maximum batch count (${MAX_BATCHES}). Process may have entered an infinite loop.`), 'Batch processing safety limit reached');
+            }
+        }
+
+        // Calculate forecast accuracy and bias in batches
+        let forecastPid = 0;
+        while (true) {
+            if (isCancelled) break;
+
+            const forecastBatch = await connection.query(
+                'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
+                [forecastPid, BATCH_SIZE]
+            );
+
+            if (forecastBatch.rows.length === 0) break;
+
+            const forecastPidArray = forecastBatch.rows.map(row => row.pid);
+
+            // Use array_to_string to convert the array to a string of comma-separated values
+            await connection.query(`
+                WITH forecast_metrics AS (
+                    SELECT 
+                        sf.pid,
+                        AVG(CASE 
+                            WHEN o.quantity > 0 
+                            THEN ABS(sf.forecast_quantity - o.quantity) / o.quantity * 100
+                            ELSE 100 
+                        END) as avg_forecast_error,
+                        AVG(CASE 
+                            WHEN o.quantity > 0 
+                            THEN (sf.forecast_quantity - o.quantity) / o.quantity * 100
+                            ELSE 0 
+                        END) as avg_forecast_bias,
+                        MAX(sf.forecast_date) as last_forecast_date
+                    FROM sales_forecasts sf
+                    JOIN orders o ON sf.pid = o.pid 
+                        AND DATE(o.date) = sf.forecast_date
+                    WHERE o.canceled = false
+                        AND sf.forecast_date >= CURRENT_DATE - INTERVAL '90 days'
+                        AND sf.pid = ANY('{${forecastPidArray.join(',')}}'::BIGINT[])
+                    GROUP BY sf.pid
+                )
+                UPDATE product_metrics pm
+                SET 
+                    forecast_accuracy = GREATEST(0, 100 - LEAST(fm.avg_forecast_error, 100)),
+                    forecast_bias = GREATEST(-100, LEAST(fm.avg_forecast_bias, 100)),
+                    last_forecast_date = fm.last_forecast_date,
+                    last_calculated_at = NOW()
+                FROM forecast_metrics fm
+                WHERE pm.pid = fm.pid
+            `);
+
+            forecastPid = forecastBatch.rows[forecastBatch.rows.length - 1].pid;
+        }
+
+        // Calculate product time aggregates
+        if (!SKIP_PRODUCT_TIME_AGGREGATES) {
+            outputProgress({
+                status: 'running',
+                operation: 'Starting product time aggregates calculation',
+                current: processedCount || 0,
+                total: totalProducts || 0,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+
+            // Note: The time-aggregates calculation has been moved to time-aggregates.js
+            // This module will not duplicate that functionality
+            processedCount = Math.floor(totalProducts * 0.6);
+            outputProgress({
+                status: 'running',
+                operation: 'Product time aggregates calculation delegated to time-aggregates module',
+                current: processedCount || 0,
+                total: totalProducts || 0,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+        } else {
+            processedCount = Math.floor(totalProducts * 0.6);
+            outputProgress({
+                status: 'running',
+                operation: 'Skipping product time aggregates calculation',
+                current: processedCount || 0,
+                total: totalProducts || 0,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+        }
+
+        // Calculate ABC classification
+        outputProgress({
+            status: 'running',
+            operation: 'Starting ABC classification',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
+        const abcConfig = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
+        const abcThresholds = abcConfig.rows[0] || { a_threshold: 20, b_threshold: 50 };
+        
+        // Extract values and ensure they are valid numbers
+        const aThreshold = parseFloat(abcThresholds.a_threshold) || 20;
+        const bThreshold = parseFloat(abcThresholds.b_threshold) || 50;
+
+        // First, create and populate the rankings table with an index
+        await connection.query('DROP TABLE IF EXISTS temp_revenue_ranks');
+        await connection.query(`
+            CREATE TEMPORARY TABLE temp_revenue_ranks (
+                pid BIGINT NOT NULL,
+                total_revenue DECIMAL(10,3),
+                rank_num INT,
+                dense_rank_num INT,
+                percentile DECIMAL(5,2),
+                total_count INT,
+                PRIMARY KEY (pid)
+            )
+        `);
+        await connection.query('CREATE INDEX ON temp_revenue_ranks (rank_num)');
+        await connection.query('CREATE INDEX ON temp_revenue_ranks (dense_rank_num)');
+        await connection.query('CREATE INDEX ON temp_revenue_ranks (percentile)');
+
+        // Calculate rankings with proper tie handling
+        await connection.query(`
+            INSERT INTO temp_revenue_ranks
+            WITH revenue_data AS (
+                SELECT 
+                    pid,
+                    total_revenue,
+                    COUNT(*) OVER () as total_count,
+                    PERCENT_RANK() OVER (ORDER BY total_revenue DESC) * 100 as percentile,
+                    RANK() OVER (ORDER BY total_revenue DESC) as rank_num,
+                    DENSE_RANK() OVER (ORDER BY total_revenue DESC) as dense_rank_num
+                FROM product_metrics 
+                WHERE total_revenue > 0
+            )
+            SELECT 
+                pid,
+                total_revenue,
+                rank_num,
+                dense_rank_num,
+                percentile,
+                total_count
+            FROM revenue_data
+        `);
+
+        // Get total count for percentage calculation
+        const rankingCount = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
+        const totalCount = parseInt(rankingCount.rows[0].total_count) || 1;
+
+        // Process updates in batches
+        let abcProcessedCount = 0;
+        const batchSize = 5000;
+        const maxPid = await connection.query('SELECT MAX(pid) as max_pid FROM products');
+        const maxProductId = parseInt(maxPid.rows[0].max_pid);
+
+        while (abcProcessedCount < maxProductId) {
+            if (isCancelled) return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,
+                success
+            };
+
+            // Get a batch of PIDs that need updating
+            const pids = await connection.query(`
+                SELECT pm.pid 
+                FROM product_metrics pm
+                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
+                WHERE pm.pid > $1
+                    AND (pm.abc_class IS NULL 
+                    OR pm.abc_class != 
+                        CASE 
+                            WHEN tr.pid IS NULL THEN 'C'
+                            WHEN tr.percentile <= ${aThreshold} THEN 'A'
+                            WHEN tr.percentile <= ${bThreshold} THEN 'B'
+                            ELSE 'C'
+                        END)
+                ORDER BY pm.pid
+                LIMIT $2
+            `, [abcProcessedCount, batchSize]);
+
+            if (pids.rows.length === 0) break;
+
+            const pidValues = pids.rows.map(row => row.pid);
+
+            await connection.query(`
+                UPDATE product_metrics pm
+                SET abc_class = 
+                    CASE 
+                        WHEN tr.pid IS NULL THEN 'C'
+                        WHEN tr.percentile <= ${aThreshold} THEN 'A'
+                        WHEN tr.percentile <= ${bThreshold} THEN 'B'
+                        ELSE 'C'
+                    END,
+                    last_calculated_at = NOW()
+                FROM (SELECT pid, percentile FROM temp_revenue_ranks) tr
+                WHERE pm.pid = tr.pid AND pm.pid = ANY($1::BIGINT[])
+                OR (pm.pid = ANY($1::BIGINT[]) AND tr.pid IS NULL)
+            `, [pidValues]);
+
+            // Now update turnover rate with proper handling of zero inventory periods
+            await connection.query(`
+                UPDATE product_metrics pm
+                SET 
+                    turnover_rate = CASE 
+                        WHEN sales.avg_nonzero_stock > 0 AND sales.active_days > 0 
+                        THEN LEAST(
+                            (sales.total_sold / sales.avg_nonzero_stock) * (365.0 / sales.active_days),
+                            999.99
+                        )
+                        ELSE 0 
+                    END,
+                    last_calculated_at = NOW()
+                FROM (
+                    SELECT 
+                        o.pid,
+                        SUM(o.quantity) as total_sold,
+                        COUNT(DISTINCT DATE(o.date)) as active_days,
+                        AVG(CASE 
+                            WHEN p.stock_quantity > 0 THEN p.stock_quantity 
+                            ELSE NULL 
+                        END) as avg_nonzero_stock
+                    FROM orders o
+                    JOIN products p ON o.pid = p.pid
+                    WHERE o.canceled = false 
+                    AND o.date >= CURRENT_DATE - INTERVAL '90 days'
+                    AND o.pid = ANY($1::BIGINT[])
+                    GROUP BY o.pid
+                ) sales
+                WHERE pm.pid = sales.pid
+            `, [pidValues]);
+
+            abcProcessedCount = pids.rows[pids.rows.length - 1].pid;
+            
+            // Calculate progress proportionally to total products
+            processedCount = Math.floor(totalProducts * (0.60 + (abcProcessedCount / maxProductId) * 0.2));
+            
+            outputProgress({
+                status: 'running',
+                operation: 'ABC classification progress',
+                current: processedCount,
+                total: totalProducts,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount, totalProducts),
+                rate: calculateRate(startTime, processedCount),
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+        }
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('product_metrics', NOW())
+            ON CONFLICT (module_name) DO UPDATE
+            SET last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount || 0,
+            processedOrders: processedOrders || 0,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
+    } catch (error) {
+        success = false;
+        logError(error, 'Error calculating product metrics');
+        throw error;
+    } finally {
+        // Always clean up temporary tables, even if an error occurred
+        if (connection) {
+            try {
+                await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
+                await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
+            } catch (err) {
+                console.error('Error cleaning up temporary tables:', err);
+            }
+            
+            // Make sure to release the connection
+            connection.release();
+        }
+    }
+}
+
+function calculateStockStatus(stock, config, daily_sales_avg, weekly_sales_avg, monthly_sales_avg) {
+    if (stock <= 0) {
+        return 'Out of Stock';
+    }
+
+    // Use the most appropriate sales average based on data quality
+    let sales_avg = daily_sales_avg;
+    if (sales_avg === 0) {
+        sales_avg = weekly_sales_avg / 7;
+    }
+    if (sales_avg === 0) {
+        sales_avg = monthly_sales_avg / 30;
+    }
+
+    if (sales_avg === 0) {
+        return stock <= config.low_stock_threshold ? 'Low Stock' : 'In Stock';
+    }
+
+    const days_of_stock = stock / sales_avg;
+
+    if (days_of_stock <= config.critical_days) {
+        return 'Critical';
+    } else if (days_of_stock <= config.reorder_days) {
+        return 'Reorder';
+    } else if (days_of_stock > config.overstock_days) {
+        return 'Overstocked';
+    }
+
+    return 'Healthy';
+}
+
+// Note: calculateReorderQuantities function has been removed as its logic has been incorporated 
+// in the main SQL query with configurable parameters
+
+module.exports = calculateProductMetrics;