matt/inventory
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: matt/inventory:master
Branches Tags
matt/inventory:master matt/inventory:merge-dashboard matt/inventory:fix-numbers matt/inventory:check-numbers matt/inventory:move-to-postgresql matt/inventory:add-company-specific-ai-prompts matt/inventory:add-permissions matt/inventory:add-product-upload-page matt/inventory:Improve-calcs matt/inventory:Improve-data-import-and-calcs
..
compare: matt/inventory:546a675b5ff9d298a3747ed97e59eaf69e578e18
Branches Tags
matt/inventory:master matt/inventory:merge-dashboard matt/inventory:fix-numbers matt/inventory:check-numbers matt/inventory:move-to-postgresql matt/inventory:add-company-specific-ai-prompts matt/inventory:add-permissions matt/inventory:add-product-upload-page matt/inventory:Improve-calcs matt/inventory:Improve-data-import-and-calcs

1 Commits
master .. 546a675b5f

Author SHA1 Message Date
matt 546a675b5f Add repeat orders page 2026-04-09 10:01:30 -04:00
426 changed files with 8611 additions and 82053 deletions
Expand all files Collapse all files
.claude/CLAUDE.md
+2 -2
View File
@@ -84,7 +84,7 @@ npm run setup # Create required directories (logs, uploads)
- PostgreSQL with connection pooling (pg library)
- Pool initialized in `utils/db.js` via `initPool()`
- Pool attached to `app.locals.pool` for route access
- Environment variables loaded from `/var/www/inventory/.env` (production path)
- Environment variables loaded from `/var/www/html/inventory/.env` (production path)
**API Routes:** All prefixed with `/api/`
- `/api/products` - Product CRUD operations
@@ -164,7 +164,7 @@ Run tests for individual components or features:
## Important Notes
- Environment variables must be configured in `/var/www/inventory/.env` for production
- Environment variables must be configured in `/var/www/html/inventory/.env` for production
- The frontend expects the backend at `/api` (proxied in dev, served together in production)
- PM2 is used for production process management
- Database uses PostgreSQL with SSL support (configurable via `DB_SSL` env var)
.gitignore
-4
View File
@@ -50,10 +50,6 @@ dashboard-server/meta-server/._package-lock.json
dashboard-server/meta-server/._services
*.tsbuildinfo
# Python bytecode
__pycache__/
*.pyc
uploads/*
uploads/**/*
**/uploads/*
CONSOLIDATION_PLAN.md
-1141
View File
File diff suppressed because it is too large Load Diff
FORECAST_FIX_PLAN.md
-343
View File
@@ -1,343 +0,0 @@
# Forecast Accuracy Fix Plan
**Written:** 2026-06-10, from a code + live-data review of the forecasting pipeline.
**Goal:** eliminate the systematic ~1.7–2x over-forecast bias, recover demand the model currently ignores, and fix the accuracy measurement so improvements are visible and long-lead forecasts are validated.
Read this whole document before starting. Fixes are grouped into phases; each phase is independently deployable and has its own validation step. Line numbers are as of 2026-06-10 — re-locate by function name if the file has drifted.
---
## 1. Diagnosis summary (measured 2026-06-10)
The dashboard headline is **202% WMAPE**. Decomposition of that number, all measured against `forecast_accuracy` run 129 and ad-hoc queries:
| Finding | Evidence |
|---|---|
| Daily-grain WMAPE has a ~190% *floor* for this catalog | Avg demand ≈ 0.11 units/product/day. A perfect rate forecast of intermittent demand scores ≈ 2e^−λ ≈ 190%. A trivial trailing-30d-average naive forecast scores **204%** on the same products/days; the engine scores 221% (slightly *worse than naive*). |
| Same forecasts at 21-day-per-product grain: **109%**; bias-corrected: **75%** | Half the headline is metric grain, most of the rest is bias. |
| Aggregate over-forecast **+70%** (227,690 forecast vs 133,861 actual units) | Portfolio daily ratio is 1.5–2.5x on most days. |
| Decay phase 2.47x over (fc 51,675 / act 20,915) | Root cause F1: velocity inflated **4.07x** (measured: 1.353 vs true 0.332 units/day) by averaging over sparse snapshot rows. |
| Preorder phase 2.15x over (fc 67,212 / act 31,189) | Root cause F4: launch curve applied at age=0 starting *today*, ignoring that the product hasn't arrived. |
| Mature phase 1.69x over (fc 57,857 / act 34,313) | Root causes F2 (history edge truncation) + F3 (seasonal double-count). |
| Dormant products sold **16,180 units** (~11% of demand) against zero forecasts | Root cause F5; also excluded from the headline metric, so invisible. |
| All 879,800 accuracy samples are in the **1–7d lead bucket** | Root cause F7: archiving design only ever saves yesterday's slice. 30–90d forecasts (what purchasing uses) are never validated. |
| Launch phase is healthy: WMAPE 100%, bias −6%, beats naive | The lifecycle-curve concept works; its calibration inputs are broken. Don't redesign it. |
**Key data fact** underlying several fixes: `daily_product_snapshots` is **activity-based and sparse** — only ~500–1,800 of ~38K products have a row on a given day. Verified: every pid-day with an order DOES have a snapshot row and units match (5,234/5,234 pid-days, 8,980 vs 8,984 units over 7 days). So *missing row = zero sales*, and any query that aggregates over only the rows that exist is averaging over sold-days.
---
## 2. Environment & operational notes
- **Files:** engine is `inventory-server/scripts/forecast/forecast_engine.py`; orchestrator `run_forecast.js` in the same dir; consumer endpoints in `inventory-server/src/routes/dashboard.js` (`/forecast/metrics` ~line 308, `/forecast/accuracy` ~line 647); overview UI in `inventory/src/components/overview/ForecastMetrics.tsx` and `ForecastAccuracy.tsx`.
- **Local `inventory-server/` is NFS-mounted to `/var/www/inventory/` on the netcup server.** Edits made locally appear on the server immediately — no copy step. Do NOT run bulk `grep`/`find`/`node --check` over `inventory-server/` locally (the mount hangs); `ssh netcup` and run them there.
- **Avoid the glob tool** for search in this repo; use bash (`grep`/`rg` via ssh for server-side trees).
- **Scheduling:** the engine runs daily at **09:30:01 server time** (runs table is conclusive), but the cron entry is NOT in matt's crontab, `/etc/cron.d`, or pm2. Likely root's crontab (`sudo crontab -l` to confirm). You do not need to touch the schedule for these fixes; just know a run fires at 09:30 daily and occasionally skips days (e.g. 2026-06-07/08).
- **Manual test runs:** `ssh netcup`, then `cd /var/www/inventory/scripts/forecast && node run_forecast.js`. Takes ~3.5–4 min. Safe to run any time: the engine TRUNCATEs and rebuilds `product_forecasts`, archives prior past-dated rows, and records a new `forecast_runs` row. Python deps live in the server venv (`venv/`); `run_forecast.js` handles env + venv automatically.
- **DB access for validation:** `ssh netcup`, then `PGPASSWORD=6D3GUkxuFgi2UghwgnUd psql -h localhost -U inventory_readonly -d inventory_db`. The engine itself connects with the write user via env vars loaded from `/var/www/inventory/.env` — schema changes should be made idempotently *inside the engine code* (the file already uses `CREATE TABLE IF NOT EXISTS` / `CREATE INDEX IF NOT EXISTS`; use `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` the same way) so no manual migration is needed.
- **Python gotchas already handled in this file (don't regress):** numpy types must go through the registered psycopg2 adapters; `pd.Series.combine_first()` keeps zeros over real data — use `reindex(..., fill_value=0.0)`.
- Engine runtime budget: currently ~212–227s. Phases 1–2 shouldn't move it meaningfully; Phase 3's extra archiving adds one INSERT…SELECT. If runtime balloons past ~6 min, investigate before shipping.
- `--backfill` mode (`backfill_accuracy_data`) is an in-sample backtest using the *old* formulas. **Do not run it anymore**; there is enough real out-of-sample history. Updating it to match the new logic is optional/low priority (F11).
---
## Phase 1 — Bias bugs in the engine (no schema changes)
### F1. Decay velocity: stop averaging over sparse snapshot rows
**Where:** `forecast_engine.py`, `batch_load_product_data()`, the decay query (~lines 697–710).
**Problem:** `AVG(COALESCE(dps.units_sold, 0))` runs over only the snapshot rows that exist — mostly sold-days. Measured inflation on the current 975 decay products: **4.07x** (1.353 vs 0.332 true units/day). This feeds `compute_scale_factor()` for the decay phase and is the single largest bias source.
**Fix:** divide the sum by calendar days in the window, clipped to the product's age (decay products are 14–60 days old, so a 20-day-old product's window is 20 days, not 30):
```sql
SELECT dps.pid,
SUM(COALESCE(dps.units_sold, 0))::float
/ GREATEST(LEAST(30, (CURRENT_DATE - pm.date_first_received::date)), 1) AS avg_daily
FROM daily_product_snapshots dps
JOIN product_metrics pm ON pm.pid = dps.pid
WHERE dps.pid = ANY(%s)
AND dps.snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
AND dps.snapshot_date >= pm.date_first_received::date
GROUP BY dps.pid, pm.date_first_received
```
No Python-side changes needed; `data['decay_velocity']` keeps the same shape. Products with zero snapshot rows in the window still get no entry → existing `scale = 1.0` fallback applies (acceptable: decay classification requires `sales_velocity_daily > 0`, so truly dead products don't reach this path).
### F2. Mature history: reindex over the full calendar window
**Where:** `forecast_engine.py`, `forecast_mature()` (~lines 833–836).
**Problem:** `hist.set_index('snapshot_date').resample('D').sum()` only spans first-snapshot → last-snapshot. Interior gaps correctly become zeros, but **leading and trailing quiet periods are absent**, so the Holt level is fitted on the product's busy span. A marginal mature product whose activity clusters in 2 of the last 8 weeks gets a level ~4x too high.
**Fix:** replace the resample with an explicit reindex over the full `EXP_SMOOTHING_WINDOW` ending yesterday:
```python
hist = history_df.copy()
hist['snapshot_date'] = pd.to_datetime(hist['snapshot_date'])
hist = hist.set_index('snapshot_date')['units_sold']
full_index = pd.date_range(
end=pd.Timestamp(date.today() - timedelta(days=1)),
periods=EXP_SMOOTHING_WINDOW, freq='D')
series = hist.reindex(full_index, fill_value=0.0).values.astype(float)
```
Notes: (pid, snapshot_date) is unique in `daily_product_snapshots`, so no duplicate-index risk. `observed_mean` and the `cap` recompute over the full window automatically (intended — the cap gets correspondingly tighter). Mature products are by definition >60 days old, so the 60-day window never predates first receipt. Do NOT use `combine_first` (see gotchas above).
### F3. Stop double-applying the monthly seasonal index
**Where:** `forecast_engine.py`, `generate_all_forecasts()` — the `seasonal_multipliers` pre-compute (~lines 959–961) and application (~line 1050).
**Problem:** every per-product calibration (decay velocity, mature Holt level, launch first-week scale, preorder rate, slow-mover velocity) is fitted on *raw recent actuals*, which already embed the current month's seasonal level. The forecast then multiplies by the **absolute** monthly index of the target date. Example from the live indices (`forecast_runs.phase_counts` for run 129): May = 1.224 (sale month), June = 0.982. Early-June forecasts were calibrated on May-sale-inflated velocities and barely discounted — a structural ~25% over-forecast at that transition, and it'll be worse around November (1.316).
**Fix:** apply the seasonal index *relative to the calibration period*. Compute a calibration index as the average monthly index over the trailing 30 calendar days (robust at month boundaries), then divide:
```python
today = date.today()
trailing = [today - timedelta(days=i) for i in range(1, 31)]
calibration_index = float(np.mean([monthly_indices.get(d.month, 1.0) for d in trailing]))
seasonal_multipliers = [
monthly_indices.get(d.month, 1.0) / max(calibration_index, 0.1)
for d in forecast_dates
]
```
Leave the DOW multipliers absolute — every calibration is a multi-week average and therefore DOW-neutral, so reshaping by absolute DOW indices is correct.
**Optional sub-fix (same area, low priority):** the monthly indices are computed from a single trailing 365-day window, so each month appears once and YoY growth contaminates "seasonality". A cheap improvement is widening `SEASONAL_LOOKBACK_DAYS` to 730 and averaging the two observations of each month. Do this only after the main fixes are validated.
### Phase 1 validation
Deploy (edit locally; NFS propagates), run the engine manually once, wait for 3–5 daily cycles, then:
```sql
-- Portfolio ratio per day (target: drifts from ~2.0 toward 0.8–1.3)
WITH ranked AS (
SELECT pfh.pid, pfh.forecast_date, pfh.forecast_units, pfh.lifecycle_phase,
ROW_NUMBER() OVER (PARTITION BY pfh.pid, pfh.forecast_date ORDER BY fr.started_at DESC) rn
FROM product_forecasts_history pfh
JOIN forecast_runs fr ON fr.id = pfh.run_id
WHERE pfh.forecast_date >= CURRENT_DATE - 7)
SELECT r.forecast_date, round(SUM(r.forecast_units),0) AS fc,
SUM(COALESCE(dps.units_sold,0)) AS act,
round(SUM(r.forecast_units)/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS ratio
FROM ranked r
LEFT JOIN daily_product_snapshots dps ON dps.pid = r.pid AND dps.snapshot_date = r.forecast_date
WHERE r.rn = 1 AND r.lifecycle_phase != 'dormant'
GROUP BY 1 ORDER BY 1;
```
Also check `forecast_accuracy` `by_phase` rows for the newest run: decay bias should fall from +0.35 toward ~0, mature from +0.17 toward ~0. (Accuracy lags ~1 day behind each fix since it evaluates yesterday's forecasts.)
---
## Phase 2 — Demand the model currently ignores or mistimes
### F4. Preorder: forecast the preorder rate until arrival, launch curve after
**Where:** `forecast_engine.py` — `batch_load_product_data()` (add arrival dates), `generate_all_forecasts()` preorder branch (~lines 1005–1009), and `forecast_from_curve()` (or a small wrapper).
**Problem:** preorder products run the launch curve from `age=0` starting **today**, i.e. full first-week launch sales while the product is still weeks from arriving. Actual preorder-period sales are a much slower trickle.
**Fix:**
1. Batch-load each preorder product's expected arrival from `purchase_orders` (line-item grain: it has `pid` and `expected_date` directly). Open statuses verified against live data: `created`, `ordered`, `electronically_sent`, `receiving_started` (~705 open line items currently have a future `expected_date`):
```sql
SELECT pid, MIN(expected_date) AS expected_arrival
FROM purchase_orders
WHERE pid = ANY(%s)
AND status IN ('created', 'ordered', 'electronically_sent', 'receiving_started')
AND expected_date IS NOT NULL
AND expected_date >= CURRENT_DATE
GROUP BY pid
```
Fallbacks, in order: (a) an open PO with a *past* `expected_date` → assume arrival in 7 days; (b) no PO at all → arrival in 14 days (and log a counter of how many hit this default).
2. In the preorder branch, build the daily array piecewise. Let `days_until_arrival = (expected_arrival - today).days`:
- Days `0 .. days_until_arrival-1`: flat observed preorder daily rate = `preorder_sales[pid] / max(preorder_days[pid], 1)` (both already batch-loaded), clamped to ≤ the curve's scaled week-0 daily value.
- Days `days_until_arrival .. horizon`: `forecast_from_curve(curve_info, scale, age_days=0, ...)` shifted so the curve's day 0 lands on the arrival date (i.e. pass `horizon_days - days_until_arrival` and offset into the output array).
- Keep the existing `compute_scale_factor('preorder', ...)` for the post-arrival curve; the pre-arrival segment doesn't use it.
This is consistent with how the reference curves were built: historical preorder units were recorded on their **order dates** (pre-arrival), so week-0 of the fitted curves reflects post-receipt orders, not the backlog.
### F5. Dormant products: small positive rate instead of hard zero, and count them
**Where:** `forecast_engine.py` — `generate_all_forecasts()` dormant branch (~lines 1040–1042), `batch_load_product_data()`, and `compute_accuracy()`.
**Problem:** all ~28K dormant products are forecast at exactly 0, yet they sold 16,180 units in the eval window (~11% of all demand) — restocks, promos, long-tail. Worse, dormant is *excluded* from the headline accuracy filter, so this miss is invisible.
**Fix (cheap version, do this now):**
1. Batch-load a trailing-180-day order rate for dormant products (11,362 of them have ≥1 sale in 180d — verified):
```sql
SELECT o.pid, SUM(o.quantity) / 180.0 AS rate
FROM orders o
WHERE o.pid = ANY(%s)
AND o.canceled IS DISTINCT FROM TRUE
AND o.date >= CURRENT_DATE - INTERVAL '180 days'
GROUP BY o.pid
```
2. Dormant branch: if the product has a rate > 0, forecast it flat with `method = 'velocity'`; else keep zeros with `method = 'zero'`. Apply the same DOW/seasonal multipliers as everything else (automatic — they're applied after the branch).
3. In `compute_accuracy()`, add a second overall row: `metric_type='overall', dimension_value='all_incl_dormant'` with no dormant filter (keep the existing `'all'` row unchanged for trend continuity). One extra entry in the `dimensions`/`filter_clauses` dicts.
**Upgrade path (optional, Phase 4):** replace flat rates for `slow_mover` + dormant-with-sales with TSB (Teunter–Syntetos–Babai), the standard intermittent-demand method with obsolescence handling. Per product over a daily series `d_t` (build it from snapshots the F2 way — full calendar reindex):
```
if d_t > 0: p_t = p_{t-1} + β·(1 − p_{t-1}); z_t = z_{t-1} + α·(d_t − z_{t-1})
else: p_t = p_{t-1}·(1 − β); z_t = z_{t-1}
forecast = p_T · z_T (flat across horizon)
```
Start with α=0.1, β=0.05, initialize p = (nonzero days / total days), z = mean of nonzero demands. Scope: slow_mover (~6K) + dormant with 180d sales (~11K); series from up to 180 days of snapshots (sparse rows → ~manageable volume). Only do this after Phase 3 measurement exists to prove it beats the flat rates.
### Phase 2 validation
After 3–5 cycles: preorder `by_phase` bias should drop from +0.85 toward < +0.3; the new `all_incl_dormant` row should appear and its `total_actual_units` minus `'all'`'s should be largely *covered* rather than all-miss (dormant `bias` rising from −1.36 toward ~−0.3 or better).
---
## Phase 3 — Fix the measurement (schema + engine + API + UI)
> Without this phase you cannot see whether Phases 1–2 worked except by ad-hoc SQL, the lead-time chart stays a single bucket forever, and the dashboard keeps displaying a number with a 190% floor in red.
### F7. Archive long-lead forecasts so 15/30/60/90d accuracy exists
**Where:** `forecast_engine.py` — `archive_forecasts()` (~lines 1086–1154), `compute_accuracy()` CTE (~lines 1201–1228).
**Problem:** the current design archives only *past-dated* rows of the previous run before truncation. With daily runs, that's only ever the 1-day-ahead slice — all 879,800 accuracy samples sit in the '1-7d' bucket and the longer buckets in the UI chart can never populate. Purchasing decisions ride on 30–60d forecasts that are never validated.
**Fix:**
1. Keep the existing past-date archiving exactly as is (it provides dense short-lead coverage).
2. After `generate_all_forecasts()` completes, additionally archive a **sampled set of future leads** from the new run, non-dormant only, attributed to the *current* run id (correct attribution, unlike the past-date path which attributes to the previous run):
```sql
INSERT INTO product_forecasts_history
(run_id, pid, forecast_date, forecast_units, forecast_revenue,
lifecycle_phase, forecast_method, confidence_lower, confidence_upper, generated_at)
SELECT %(run_id)s, pid, forecast_date, forecast_units, forecast_revenue,
lifecycle_phase, forecast_method, confidence_lower, confidence_upper, generated_at
FROM product_forecasts
WHERE lifecycle_phase != 'dormant'
AND forecast_date - CURRENT_DATE IN (7, 14, 30, 60, 89)
ON CONFLICT (run_id, pid, forecast_date) DO NOTHING
```
Volume: ~10K non-dormant products × 5 leads ≈ 50K rows/day; the existing 90-day prune (`forecast_date < CURRENT_DATE - 90`) bounds steady state at a few million rows. Note future-dated rows survive until their date passes + 90 days — that's intended.
3. **CRITICAL companion change** in `compute_accuracy()`: the accuracy CTE must now exclude not-yet-realized rows, or future-dated archives get scored against actual=0:
```sql
FROM product_forecasts_history pfh
JOIN forecast_runs fr ON fr.id = pfh.run_id
WHERE pfh.forecast_date < CURRENT_DATE -- ADD THIS
```
4. **Dedup semantics change.** Today's `ROW_NUMBER() OVER (PARTITION BY pid, forecast_date ORDER BY started_at DESC)` keeps only the latest (= shortest-lead) row per pid/date, which would silently discard all the new long-lead rows. Restructure:
- Compute `lead_days = forecast_date - started_at::date` and the lead bucket *inside* `ranked_history`.
- For `by_lead_time`: dedup `PARTITION BY pid, forecast_date, lead_bucket` (one sample per pid/date/bucket, latest run wins within a bucket).
- For everything else (`overall`, `by_phase`, `by_method`, `daily`, and the new weekly metric below): restrict to `lead_days BETWEEN 0 AND 6` and keep the existing per-(pid, date) dedup. This preserves the current meaning of the headline metrics (short-lead) while the lead-time table becomes real.
### F8. Track a naive baseline (forecast value-added)
**Where:** `archive_forecasts()` (both INSERT paths), `compute_accuracy()`, `forecast_accuracy` schema, `/forecast/accuracy` endpoint.
**Problem:** the engine currently *loses* to a trailing-average naive forecast (221% vs 204% daily WMAPE) and nothing on the dashboard would ever reveal that. Every accuracy improvement should be judged as value-over-naive.
**Fix:**
1. Schema (idempotent, in the ensure blocks): `ALTER TABLE product_forecasts_history ADD COLUMN IF NOT EXISTS naive_units NUMERIC(10,2);` and `ALTER TABLE forecast_accuracy ADD COLUMN IF NOT EXISTS naive_wmape NUMERIC(10,4), ADD COLUMN IF NOT EXISTS fva NUMERIC(10,4);`
2. Populate `naive_units` during both archive INSERTs via a join — naive = flat trailing-28-day average daily units as of archive time (28 days = DOW-balanced; information available at generation; same value at every lead, which is exactly what a naive baseline means):
```sql
LEFT JOIN (
SELECT o.pid, SUM(o.quantity) / 28.0 AS naive_daily
FROM orders o
WHERE o.canceled IS DISTINCT FROM TRUE
AND o.date >= CURRENT_DATE - INTERVAL '28 days' AND o.date < CURRENT_DATE
GROUP BY o.pid
) nv ON nv.pid = pf.pid
-- select COALESCE(nv.naive_daily, 0) AS naive_units
```
3. In `compute_accuracy()`, add to each dimension's aggregate: `SUM(ABS(naive_units - actual_units)) / NULLIF(SUM(actual_units),0) AS naive_wmape` and store `fva = 1 - wmape / naive_wmape` (NULL-safe). Rows archived before this change have `naive_units` NULL — treat NULL as excluded (`FILTER (WHERE naive_units IS NOT NULL)` on the naive sums) rather than as zero.
4. Endpoint: include `naiveWmape` and `fva` in the `overall` (and per-phase) payload of `/dashboard/forecast/accuracy` in `dashboard.js`.
### F9. Weekly-grain headline metric + bias as a percentage
**Where:** `compute_accuracy()`, `/forecast/accuracy` endpoint, `ForecastAccuracy.tsx`.
**Problem:** daily-grain WMAPE on this catalog has a ~190% floor — as a headline it's noise. The informative numbers are (a) weekly-per-product WMAPE (currently ~109%, target ~70–85% post-fix) and (b) aggregate bias, which the UI currently renders as `+0.108 units` — indistinguishable from zero while the reality is +70%.
**Fix:**
1. New metric in `compute_accuracy()`: `metric_type='overall_weekly', dimension_value='all'`. Definition: using the short-lead deduped rows (lead ≤ 6, non-dormant), aggregate per `(pid, date_trunc('week', forecast_date))` keeping only complete weeks (`COUNT(*) = 7`), then `WMAPE = SUM(ABS(fc_week − act_week)) / SUM(act_week)`, excluding pid-weeks where both are 0. Store sample_size = number of pid-weeks. Compute `naive_wmape`/`fva` the same way from `naive_units`.
2. Endpoint: expose as `overallWeekly`; also add a weekly variant to the `accuracyTrend` query (`metric_type='overall_weekly'`). The trend will start empty (old runs lack the row) — that's fine; don't backfill.
3. `ForecastAccuracy.tsx`:
- Headline WMAPE → `overallWeekly.wmape`, labeled "WMAPE (weekly)". Keep daily WMAPE available in a tooltip if desired.
- Color thresholds for weekly grain: green ≤ 60, yellow ≤ 90, red above (tunable; document that they're calibrated for intermittent retail demand).
- Replace the bias row: show `(totalForecast / totalActual − 1)` as a signed percentage labeled "Forecast vs actual" (both totals already arrive in `overall`). Keep MAE.
- Add a "vs naive" line: naive weekly WMAPE and FVA. FVA > 0 = engine adds value.
- The lead-time chart needs no code change — buckets will populate as F7 rows mature (7d lead evaluable after 7 days, 30d after 30, etc.).
4. `confidenceLevel` in `/forecast/metrics` ([dashboard.js ~line 360]) is "share of products forecast via lifecycle curves", not confidence. It only feeds a per-day tooltip field — rename the JSON field to `curveCoverage` and update the one consumer in `ForecastMetrics.tsx`, or leave it and add a comment; low priority.
### Phase 3 validation
- Next run after deploy: `forecast_accuracy` contains `overall_weekly` and `fva` values; `/dashboard/forecast/accuracy` returns them; the overview popover renders weekly WMAPE, bias %, and the naive comparison.
- After 7/14/30 days: `by_lead_time` rows appear for '8-14d', '15-30d', '31-60d' buckets respectively (61-90d after ~60 days).
- Confirm engine runtime still < ~5 min and `product_forecasts_history` growth ≈ 50–70K rows/day.
---
## Phase 4 — Optional / after the above is proven
- **F6. TSB for slow movers + dormant** (spec in F5). Gate on Phase 3 measurement: ship only if weekly FVA improves on those phases.
- **F10. Confidence-margin source:** `load_accuracy_margins()` feeds daily-grain per-phase WMAPE (clamped to 1.0) into the intervals, so every interval is ±100% — uninformative. Once `overall_weekly` exists, add per-phase weekly rows (`by_phase_weekly`) and source margins from those instead.
- **F11.** Update or delete `backfill_accuracy_data()` (it encodes the old formulas). Until then, just don't run `--backfill`.
- **F12.** `compute_dow_indices()` weights by revenue but the multipliers are applied to units — switch `SUM(o.price * o.quantity)` to `SUM(o.quantity)`. Tiny effect.
- **F13.** Longer term: for reorder decisions the right target is P(lead-time demand > stock), not a point forecast. Evaluate quantile (pinball) loss at lead-time horizons using the existing confidence-interval columns. Design separately.
---
## 4. Success criteria
1. Rolling-14-day portfolio forecast/actual ratio within **0.8–1.25** (currently 1.5–2.5).
2. Weekly-grain WMAPE ≤ **90%** and **FVA > 0** (engine beats naive) sustained for 2+ weeks.
3. Decay/preorder/mature per-phase bias within ±0.1 units/day (currently +0.35 / +0.85 / +0.17).
4. `all_incl_dormant` actuals covered: dormant bias better than −0.4 (currently −1.36, i.e. 100% miss).
5. Lead-time buckets through 31–60d populated with ≥10K samples each within ~6 weeks.
6. Launch phase stays healthy (bias within ±0.15, WMAPE not degraded) — regression guard for F3/F4 changes.
## 5. Re-measurement appendix
The naive-vs-engine comparison used in the diagnosis (rerun any time; adjust dates):
```sql
WITH ranked AS (
SELECT pfh.pid, pfh.forecast_date, pfh.forecast_units, pfh.lifecycle_phase,
ROW_NUMBER() OVER (PARTITION BY pfh.pid, pfh.forecast_date ORDER BY fr.started_at DESC) rn
FROM product_forecasts_history pfh
JOIN forecast_runs fr ON fr.id = pfh.run_id
WHERE pfh.forecast_date BETWEEN CURRENT_DATE - 9 AND CURRENT_DATE - 1),
eng AS (SELECT * FROM ranked WHERE rn = 1 AND lifecycle_phase != 'dormant'),
naive AS (
SELECT o.pid, SUM(o.quantity)/30.0 AS naive_daily FROM orders o
WHERE o.canceled IS DISTINCT FROM TRUE
AND o.date >= CURRENT_DATE - 39 AND o.date < CURRENT_DATE - 9
GROUP BY o.pid)
SELECT e.lifecycle_phase, COUNT(*) AS n, SUM(COALESCE(dps.units_sold,0)) AS actual,
round(SUM(e.forecast_units),0) AS engine_fc, round(SUM(COALESCE(nv.naive_daily,0)),0) AS naive_fc,
round(SUM(ABS(e.forecast_units - COALESCE(dps.units_sold,0)))/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS engine_wmape,
round(SUM(ABS(COALESCE(nv.naive_daily,0) - COALESCE(dps.units_sold,0)))/NULLIF(SUM(COALESCE(dps.units_sold,0)),0),2) AS naive_wmape
FROM eng e
LEFT JOIN naive nv ON nv.pid = e.pid
LEFT JOIN daily_product_snapshots dps ON dps.pid = e.pid AND dps.snapshot_date = e.forecast_date
GROUP BY ROLLUP(e.lifecycle_phase) ORDER BY 1;
```
Baseline numbers to beat (June 1–9, 2026): engine 221% / naive 204% daily WMAPE; engine_fc/actual = 1.82; per-phase table in §1.
IMPORT_METRICS_FIX_PLAN.md
-449
View File
@@ -1,449 +0,0 @@
# Import & Metrics Pipeline Fix Plan
Fixes for issues found in a full review (2026-06-10) of the `full-update.js` pipeline:
`inventory-server/scripts/full-update.js` → `import-from-prod.js` (6 importers in `scripts/import/`)
→ `calculate-metrics-new.js` (7 SQL modules in `scripts/metrics-new/`).
Every issue below was verified against the code, and where marked **[verified-live]**, against the
live MySQL source (`sg` on 192.168.1.5 via the acot-db tooling / `ssh workpi`) and live PostgreSQL
(`inventory_db` — `ssh netcup`, then `psql -U inventory_readonly`, password in `/Users/matt/Dev/inventory/CLAUDE.md`).
Write credentials for migrations: see `/var/www/inventory/.env` on netcup (`inventory_user`).
## Operational context (read first)
- Local `inventory-server/` is **NFS-mounted** to `/var/www/inventory/` on the netcup server — edits
appear on the server with no copy step. Run heavy validation/grep/find **on the server via
`ssh netcup`**, not locally (NFS hangs + AppleDouble `._*` noise).
- The PG server timezone is **Europe/Berlin**. The business operates in **America/Chicago**. This
matters for Fix 2.
- MySQL server is America/Chicago; the mysql2 driver is configured `timezone: '-05:00'` and
corrected at runtime by `adjustDateForMySQL()` in `scripts/import/utils.js` (see
`memory/TIMEZONE_ISSUE.md`). Don't "fix" that part — it already works.
- Orders/PO/products imports are incremental by default (`INCREMENTAL_UPDATE !== 'false'`); a full
orders sync = run with `INCREMENTAL_UPDATE=false` (5-year window).
- Existing rebuild tooling: `scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (rebuilds
`daily_product_snapshots` from `orders`/`receivings`). The full-pipeline order after data fixes:
re-import → rebuild snapshots → `node scripts/calculate-metrics-new.js`.
- Precedent: `scripts/metrics-new/migrations/002_fix_discount_double_counting.sql` documents the
procedure used last time a discount formula changed. Follow the same pattern (migration doc +
code fix + full re-import + rebuild).
---
## P0 — Data correctness (do both, then ONE re-import + rebuild)
### Fix 1: Item-level promo discounts dropped (~$26K / 30 days ≈ 10% of product revenue) [verified-live]
**File:** `scripts/import/orders.js` — `order_totals` CTE (~lines 604-623) and the discount fetch in
`processDiscountsBatch` (~lines 379-383).
**Problem.** The discount applied to each PG `orders` row is:
prorated `summary_discount_subtotal` + item-level promo discounts. The item-level part is gated:
```sql
SUM(CASE WHEN COALESCE(md.discount_amount_subtotal, 0) > 0 THEN id.amount ELSE 0 END)
```
In the PHP source (`/Users/matt/Dev/acot/website/website/lib/neworder.class.php`):
- `order_items.prod_price` is the **pre-promo** price; `summary_subtotal = Σ prod_price·qty` (line ~3087).
- Item-level promo discounts live in `order_discount_items` with `which = 2`; they are applied to the
order total via `summary_discount += amount + products_disc_sum` (line ~6567) — i.e. they are **not**
part of `discount_amount_subtotal` and **not** baked into `prod_price`.
- Live data (90 days): of 10,010 type-10 promo discounts, **8,070 have item rows but only 8 have
`discount_amount_subtotal > 0`** — the gate zeroes essentially all item-level promo discounts.
- Live impact (30 days): **$25,989 dropped** across 2,021 orders, vs only $13,574 captured via the
prorated subtotal component. Order discount components, 30d: total $54,957 = $13,574 subtotal +
$15,395 shipping + ~$25,989 item-level. (Shipping discounts correctly excluded from product revenue.)
**Consequence.** `orders.discount` understated → `net_revenue`, `profit_30d`, `margin_30d` overstated
by ~10% of revenue; `discounts_30d` / `discount_rate_30d` ~3x understated. Flows into daily snapshots,
product/brand/vendor/category metrics, and dashboards.
**Fix.**
1. In `processDiscountsBatch`, fetch only real item discounts:
`SELECT order_id, pid, discount_id, amount FROM order_discount_items WHERE order_id IN (?) AND which = 2`.
(`which=1` rows store prices of free promo-added items; `which=3` are usage records — neither is a
discount amount.)
2. In the `order_totals` CTE, remove the gate — sum `id.amount` unconditionally:
`SUM(COALESCE(id.amount, 0)) AS promo_discount_sum` (drop the join/CASE on `temp_main_discounts`;
`temp_main_discounts` becomes unused and can be removed entirely along with its insert loop).
3. Sanity guard (optional, recommended): clamp final per-row discount to `price * quantity`.
**Verification.** After a FULL orders re-import, for a recent 30-day window PG should satisfy:
`SUM(discount)` ≈ MySQL `Σ summary_discount_subtotal` + `Σ order_discount_items.amount (which=2)`
over the same orders (± rounding from proration). Spot-check an order with a type-10 promo:
discount on the affected pid ≈ the `which=2` amount. Re-run migration 002's verification query too
(pids 624756, 614513) to confirm no regression of the prior fix.
### Fix 2: Daily snapshots bucket sales by Europe/Berlin days, not business days [verified-live]
**Files:** `scripts/metrics-new/update_daily_snapshots.sql` (SalesData join `o.date::date = _target_date`
~line 138; gap-fill and stale-detection aggregates at lines ~47-83);
`scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (same pattern — check & fix);
`scripts/metrics-new/update_product_metrics.sql` (`HistoricalDates` `MIN(o.date)::date` etc., lines ~131-147).
**Problem.** `orders.date` is `timestamptz`; `::date` casts in the server TZ (**Europe/Berlin**,
verified via `SHOW timezone`). Berlin is 7-8h ahead of Central, so every order placed after
~5 PM Central lands on the **next** snapshot day. This shifts a large evening slice of daily sales
forward one day; skews `yesterday_sales`, day-of-week patterns (the forecast engine's DOW
multipliers, daily-grain forecast accuracy — see `FORECAST_FIX_PLAN.md`), and is inconsistent with
`stock_snapshots`, whose dates come from a Central-time MySQL cron.
**Fix.** Bucket all order/receiving dates in business time. Replace every `o.date::date` /
`received_date::date` used for *day bucketing* in the two snapshot SQL files with:
```sql
(o.date AT TIME ZONE 'America/Chicago')::date
```
Apply consistently in: SalesData, ReceivingData, the gap-fill date lists, the stale-detection
aggregates (they must match SalesData or every day looks permanently stale), and the rebuild script.
`HistoricalDates` in update_product_metrics (first/last sold dates) should match too.
Add an index to keep the per-day loop fast, e.g.
`CREATE INDEX ON orders ( ((date AT TIME ZONE 'America/Chicago')::date) );` and equivalent on
`receivings(received_date)`; check `EXPLAIN` on the SalesData query afterward.
Note: `receivings.received_date` came from MySQL DATETIME (Central literal) inserted as timestamptz —
it was interpreted in the *session* TZ at insert. Before converting, spot-check a few receivings
against MySQL to confirm which TZ the stored instants actually represent; the conversion expression
must yield the Central calendar day MySQL shows. Same check for `orders.date` (it originates from
`_order.date_placed`, a TIMESTAMP column, so it should be a correct instant — `AT TIME ZONE
'America/Chicago'` is right for it).
**Verification.** Pick 2-3 recent days; compare per-day `units_sold` totals in
`daily_product_snapshots` against MySQL
`SELECT date_placed_onlydate, SUM(qty_ordered) ... WHERE order_status >= 20 GROUP BY 1`
(MySQL stores Central days). They should now match closely (small diffs from canceled-status timing).
### P0 execution order (single pass)
1. Land Fix 1 (orders.js) and Fix 2 (both snapshot SQL files + product-metrics date CTE).
2. Full orders re-import: `INCREMENTAL_UPDATE=false node scripts/import-from-prod.js` (or at minimum
the orders step) — run on the server, it's long.
3. Rebuild snapshots: `psql -f scripts/metrics-new/backfill/rebuild_daily_snapshots.sql` (after
confirming it contains the TZ fix). The hourly job's 90-day self-heal will NOT fix history beyond
90 days by itself; the explicit rebuild is required.
4. `node scripts/calculate-metrics-new.js`.
5. Expect dashboards to show: margins down ~8-10 points (real), daily sales curves shifted, DOW
profile changed. Tell the user before/after numbers.
---
## P1 — Wrong or drifting numbers, fix soon
### Fix 3: Vendor avg lead time computed over a near-cartesian join
**File:** `scripts/metrics-new/calculate_vendor_metrics.sql`, `VendorPOAggregates` (lines ~62-83).
**Problem.** Joins each done-PO line to **every** receiving of the same (pid, supplier) after the PO
date — a product received 10 times contributes 10 ever-growing lead times → overstated, busy-product-
weighted vendor lead time. The per-product version in `update_periodic_metrics.sql` (lines 27-48)
is correct (MIN receiving per PO within 180 days, then average).
**Fix.** Reuse the periodic shape, aggregated to vendor:
```sql
WITH po_first_receiving AS (
SELECT po.vendor, po.po_id, po.pid, po.date::date AS po_date,
MIN(r.received_date::date) AS first_receive_date
FROM purchase_orders po
JOIN receivings r ON r.pid = po.pid AND r.supplier_id = po.supplier_id
AND r.received_date >= po.date
AND r.received_date <= po.date + INTERVAL '180 days'
WHERE po.status = 'done' AND po.date >= CURRENT_DATE - INTERVAL '1 year'
AND po.vendor IS NOT NULL AND po.vendor <> ''
GROUP BY po.vendor, po.po_id, po.pid, po.date
)
SELECT vendor, COUNT(DISTINCT po_id) AS po_count_365d,
ROUND(AVG(GREATEST(1, first_receive_date - po_date)))::int AS avg_lead_time_days_hist
FROM po_first_receiving GROUP BY vendor
```
**Verification.** For a few vendors compare old vs new values; new should be materially lower and
roughly match `AVG(product_metrics.avg_lead_time_days)` for that vendor's products.
### Fix 4: Deleted order items & combined orders never reconciled in PG [verified-live]
**File:** `scripts/import/orders.js`.
**Problem.** The orders import upserts but never deletes:
- Items removed from an order in MySQL (`DELETE FROM order_items ...` happens, e.g.
neworder.class.php ~line 6500 for unpicked promo items, plus staff edits) leave stale rows in PG
forever. May 2026 check: PG has 49,841 item rows vs MySQL 49,377 (+0.9%) — and PG should be ≤
MySQL.
- Combining orders (`combine_orders`, neworder.class.php ~11946) sets the source orders to status 16
AND **zeroes `date_placed`**, then copies all items to a NEW order. Because the import query
filters `o.date_placed >= …`, a combined source order can never be re-fetched, so its stale
'placed' rows would double-count with the new merged order. Currently latent (last combine
2024-07, predating current PG data — verified no stale rows exist today), but it will silently
corrupt the day combining is used again.
**Fix.** Two parts, both inside the orders import after the upsert phase:
1. **Item-set reconciliation** for re-imported orders: the import already knows the set of changed
`orderIds` and inserted their current items into `temp_order_items`. Mirror the PO import's
pattern (`purchase-orders.js` lines ~683-694):
```sql
DELETE FROM orders o
WHERE o.order_number = ANY($1) -- orders fetched this run
AND NOT EXISTS (SELECT 1 FROM temp_order_items t
WHERE t.order_id = o.order_number AND t.pid = o.pid);
```
2. **Combined/cancelled sweep** that does NOT depend on `date_placed`: each run, fetch from MySQL
`SELECT order_id, order_status FROM _order WHERE order_status IN (15,16) AND stamp > ?`
(no date_placed filter) and update matching PG rows' `status`/`canceled`
('combined' rows are then excluded from metrics — see Fix 5). Cheap (small result set).
**Verification.** Re-run the May-2026 row-count comparison (MySQL vs PG for one month) after one full
run; counts should converge (PG ≤ MySQL, diff explained by TZ window edges only).
### Fix 5: 'combined' orders are counted as sales
**Files:** `scripts/metrics-new/update_daily_snapshots.sql` (status filters, lines ~77, 120-134),
`update_product_metrics.sql` (`HistoricalDates` line ~145, `LifetimeRevenue` line ~249),
`backfill/rebuild_daily_snapshots.sql`.
**Problem.** Sales filters exclude only `('canceled', 'returned')`. Status 16 'combined' = "merged
into another order" — the new order carries the same items, so counting both double-counts. 826
combined orders exist in MySQL; today none are in PG (see Fix 4), but once Fix 4's sweep starts
marking rows 'combined', the metrics filters must exclude them.
**Fix.** Change every `NOT IN ('canceled', 'returned')` in the metrics SQL to
`NOT IN ('canceled', 'returned', 'combined')`. Grep for the pattern in `scripts/metrics-new/` and
`src/routes/` (dashboard endpoints replicate these filters — see CLAUDE.md analytics-filters note).
### Fix 6: Incremental sync watermark race (silent permanent misses)
**Files:** `scripts/import/orders.js` (~772), `products.js` (~934), `purchase-orders.js` (~833).
**Problem.** `sync_status.last_sync_timestamp` is set to `NOW()` *after* the import finishes. Any
MySQL row modified between the source query and that write is below the new watermark but was never
fetched → permanently skipped (until a full sync or the row changes again). Long imports widen the
window; PG/MySQL clock skew adds to it.
**Fix.** Capture the watermark **before** the source query and write that value:
```js
const [[{ now: sourceNow }]] = await prodConnection.query('SELECT NOW() as now');
// ... do the import ...
await localConnection.query(
`INSERT INTO sync_status ... VALUES ('orders', $1) ON CONFLICT ... SET last_sync_timestamp = $1`,
[sourceNow]);
```
Using MySQL's own clock also eliminates cross-server skew. Note `sourceNow` comes back through the
mysql2 driver TZ conversion — verify round-tripping with `adjustDateForMySQL` produces a correct
comparison value, or store `UTC_TIMESTAMP()` and compare against `CONVERT_TZ`-normalized stamps.
Overlap (re-importing rows changed during the run) is harmless — everything is upserted.
### Fix 7: Stockout days / service level / fill rate / avg stock built on activity-only snapshots
**Files:** `scripts/metrics-new/update_product_metrics.sql` — `SnapshotAggregates`
(`stockout_days_30d`, `avg_stock_*_30d`, lines ~177-189), `ServiceLevels` (lines ~304-323),
plus `calculate_sales_velocity` usage.
**Problem.** `daily_product_snapshots` only has rows on days with sales/receivings. So:
- A product that is out of stock (and therefore sells nothing) gets **no row** → `stockout_days_30d`
≈ 0 exactly when stockouts matter → `calculate_sales_velocity(sales, stockout_days)`'s adjustment
is inert → velocity and replenishment understated for constrained products.
- `service_level_30d` divides stockout days by COUNT(activity days), not 30.
- `avg_stock_units_30d` / `avg_stock_cost_30d` average only activity days (biased toward in-stock
days) → GMROI / stockturn / sell-through denominators biased.
- `fill_rate_30d`'s `units_sold * 0.2` lost-sales heuristic is arbitrary — fine to keep, but document.
**Fix.** Derive stock-presence metrics from `stock_snapshots` (full daily coverage from MySQL
`snap_product_value`, imported by `stock-snapshots.js`) instead of `daily_product_snapshots`:
```sql
StockCoverage AS (
SELECT pid,
COUNT(*) FILTER (WHERE stock_quantity <= 0) AS stockout_days_30d,
AVG(stock_quantity) AS avg_stock_units_30d,
AVG(stock_value) AS avg_stock_cost_30d
FROM stock_snapshots
WHERE snapshot_date >= _current_date - INTERVAL '29 days'
GROUP BY pid
)
```
Treat products absent from `stock_snapshots` for a day as unknown (NULL), not in-stock. Keep
`daily_product_snapshots` for sales/revenue aggregates. `service_level_30d` denominator becomes the
count of covered days. Note `stock_snapshots` has no `eod_stock_retail`; keep retail/gross averages
on the old source or compute as `stock_quantity * current price` explicitly.
**Verification.** Pick products that had a known stockout period; `stockout_days_30d` should now be
> 0 and `sales_velocity_daily` should rise accordingly.
---
## P2 — Definition / robustness improvements
### Fix 8: Returns don't reduce COGS; LifetimeRevenue ignores returns
`update_daily_snapshots.sql` SalesData: COGS accrues only on `quantity > 0` rows; return rows
(negative qty — 15,875 rows live) subtract revenue but never COGS → margin understated in
return-heavy periods. Add a returns-COGS term mirroring the sales-COGS COALESCE chain
(`SUM(... WHEN quantity < 0 THEN cost * ABS(quantity))`) and subtract it in `cogs` (or store
`returns_cogs` separately and use `cogs - returns_cogs` in profit). Also `LifetimeRevenue` in
`update_product_metrics.sql` (line ~242) filters `quantity > 0` — include negative-qty rows so
lifetime revenue nets out returns (drop the quantity filter; `price*quantity` is already signed,
but check the `- discount` term sign for return rows).
### Fix 9: return_rate_30d definition
`update_product_metrics.sql` line ~468: `returns / (sales + returns)` → industry standard is
`returns / sales`. Change denominator to `NULLIF(sa.sales_30d, 0)`.
### Fix 10: GMROI not annualized
Line ~466: `profit_30d / avg_stock_cost_30d` is a monthly GMROI (~1/12 of the conventional annual
figure, benchmark ≥ 2-3). Either annualize (`* 12.17`) or rename the column/label "monthly".
Decision for Matt; annualizing is recommended for comparability. Frontend displays must be checked
either way.
### Fix 11: get_weighted_avg_cost is a lifetime WAC
`db/functions.sql` (~line 81, deployed identically): averages ALL receivings ≤ date — decade-old
costs weigh equally. Recommended: window to recent receivings, e.g. last 365 days falling back to
lifetime when none. Used as fallback COGS when `o.costeach` is NULL, so impact is modest but real
for long-lived SKUs. Apply with `CREATE OR REPLACE FUNCTION` in `db/functions.sql` AND on the live DB.
### Fix 12: exclude_from_forecast removes products from product_metrics entirely
`update_product_metrics.sql` line ~627 (`WHERE s.exclude_forecast IS FALSE OR ... IS NULL`): the
flag's name implies forecast-only, but excluded products get NO metrics row → vanish from brand/
vendor/category rollups and dashboards. Fix: always emit the row; instead NULL the
forecast/replenishment columns when excluded (wrap those expressions in
`CASE WHEN s.exclude_forecast THEN NULL ELSE ... END`).
### Fix 13: Incremental products import misses category-only changes
`products.js` incremental WHERE (~lines 433-440) keys on `p.stamp`, `ci.stamp`, price/b2b dates —
`product_category_index` changes don't bump any of those → PG `product_categories` goes stale. Also
the `needs_update` comparison (~lines 604-625) doesn't compare `categories`, so even refetched rows
skip the category rewrite. Fix both: add `t.categories IS NOT DISTINCT FROM p.categories` to the
needs_update comparison (note: `products.categories` is the GROUP_CONCAT string — confirm PG column
holds the same representation), and add a cheap full-sweep (e.g. weekly, or compare
`COUNT(*) GROUP BY pid` hashes) OR include `EXISTS (SELECT 1 FROM product_category_index pci WHERE
pci.pid = p.pid AND pci.stamp > ?)` in the incremental WHERE if that table has a stamp column —
verify schema first (`DESCRIBE product_category_index`).
### Fix 14: PO/receivings OFFSET pagination over a moving filter
`purchase-orders.js` (~lines 275-298, 447-470): `LIMIT/OFFSET` with a `date_updated > ?` predicate;
concurrent updates shift rows between pages → silent skips. Fix: keyset pagination —
`WHERE ... AND p.po_id > ? ORDER BY p.po_id LIMIT 500`, carrying the last seen po_id (drop OFFSET).
Same for receivings on `receiving_id`.
### Fix 15: Status map gaps and unsafe defaults
- `orders.js` orderStatusMap lacks 45 (`payment_pending`) and 67 (`remote_send`) → imported as
numeric strings. Add both (mirror in `migrations/001_map_order_statuses.sql` as a follow-up update
for existing rows).
- `purchase-orders.js` `poStatusMap[po.status] || 'created'` (line ~335): an unknown *cancel-like*
code would be treated as an open PO and inflate on-order FIFO. Default to a sentinel like
`'unknown_<code>'` instead, and make the FIFO/on-order CTEs in `update_product_metrics.sql` treat
only the known-open statuses as open (they already whitelist open statuses — so the sentinel is
safe there; just ensure nothing treats unknown as 'created'). Same for receivingStatusMap.
### Fix 16: Transactions issued through the pool wrapper land on arbitrary connections
`categories.js` (lines ~17-152) and `daily-deals.js` (~27-130) call `query('BEGIN')` /
`query('COMMIT')` on the wrapper, which checks out a client per call — BEGIN/work/COMMIT are not
guaranteed to share a connection (works only by pool-LIFO accident). The categories
`DISABLE TRIGGER` rides on this too. Fix: use the wrapper's `beginTransaction()/commit()/rollback()`
(see `utils.js` lines 121-148) exactly as orders.js does. In categories.js also move the
post-COMMIT `ENABLE TRIGGER` inside the transaction (DISABLE/ENABLE both inside), or drop the
trigger toggling entirely if the trigger isn't actually problematic anymore.
### Fix 17: stock-snapshots import swallows batch errors → permanent holes
`stock-snapshots.js` (~lines 153-155): a failed batch is logged and skipped, but the next
incremental starts at `MAX(snapshot_date)` — the hole is never revisited. Fix: rethrow (fail the
step) or collect failed date ranges and retry once, then fail if still failing. Also line ~168:
`calculateRate(processedRows, startTime)` — arguments reversed (signature is
`calculateRate(startTime, current)`, see `metrics-new/utils/progress.js:70`).
### Fix 18: Metrics cancellation targets an application_name that's never set
`calculate-metrics-new.js` line ~180 cancels backends `WHERE application_name =
'node-metrics-calculator'`, but the Pool config never sets it → cancellation no-ops (the 30-min
`statement_timeout` is the only real guard). Fix: add `application_name: 'node-metrics-calculator'`
to both dbConfig branches.
### Fix 19: Aggregate-table change-detection lists miss cost-only changes
`calculate_brand_metrics.sql` / `calculate_vendor_metrics.sql` / `calculate_category_metrics.sql`
ON CONFLICT WHERE lists don't include `profit_30d`/`cogs_30d` — a cost revision with unchanged
sales/revenue leaves stale rows (product_metrics has a 1-day staleness net; rollups don't). Add
`... OR x.profit_30d IS DISTINCT FROM EXCLUDED.profit_30d OR x.cogs_30d IS DISTINCT FROM
EXCLUDED.cogs_30d` to each, or add a `last_calculated < NOW() - INTERVAL '1 day'` net like
product_metrics line ~707.
### Fix 20: Snapshot stale-detection only compares unit counts
`update_daily_snapshots.sql` lines ~57-85: detects mismatches in `units_sold`/`units_received` only;
price/discount/costeach corrections older than the 2-day recheck are never repaired. Add a
revenue comparison to the stale check: compare `SUM(net_revenue)` per day against the equivalent
recomputed from `orders` (ROUND both to 2dp to avoid float-noise churn).
### Fix 21: Category metrics positive-only revenue asymmetry
`calculate_category_metrics.sql` (lines ~27-36, 64-73): revenue summed only when `> 0` while
cogs/profit use COALESCE-all → margin numerator/denominator from different populations, and
inconsistent with brand/vendor (plain COALESCE). Change the revenue/sales CASEs to
`COALESCE(pm.revenue_7d, 0)` etc., matching brand_metrics.
### Fix 22 (decision needed): Demand-pattern & seasonality definitions
- `classify_demand_pattern` (db/functions.sql): CV thresholds 0.2/0.5 + avg<1/day. Industry standard
is Syntetos-Boylan: ADI ≥ 1.32 and CV² ≥ 0.49 quadrants (smooth/erratic/intermittent/lumpy).
Today everything classifies sporadic/lumpy. If adopting SB: ADI = 30 / COUNT(days with sales),
CV² computed on nonzero-demand sizes. Changes the vocabulary consumed by the forecast engine
(`scripts/forecast/forecast_engine.py` reads `demand_pattern`) — coordinate before changing.
- SeasonalityAnalysis (`update_product_metrics.sql` ~360): `month_avg = AVG(units_sold)` over rows
with sales only → intensity, not volume. Use monthly totals (SUM, with zero months counted) /
overall monthly average for the index.
- Safety stock: currently static config units; `sales_std_dev_30d` exists but is unused. Optional
upgrade: `safety = z * σ_d * sqrt(lead_time)` with z from a service-level setting.
These change user-facing semantics — confirm with Matt before implementing.
---
## Verified non-issues (no action, or cleanup only)
- **`costeach` fallback `price * 0.5`** (orders.js line ~615): fires on **2.1%** of item rows
(729/34,833, last 30d, live-verified). Accepted by Matt — 50% margin is a fair estimate for these
products. Optional: nothing.
- **Missing-product order skips**: zero occurrences — MySQL has no orphan order_items (1-year check),
PG products is a superset of MySQL products (687,579 vs 687,576), last 7 import runs all logged
`totalSkipped: 0`. Cleanup only: remove the unused `importMissingProducts` import line at
`orders.js:2` (the function itself stays in products.js — harmless utility).
- **Status 30 'cancelled_old'** in `total_sold >= 20` filter: zero rows live in `_order` — safe.
- **Duplicate (order_id, pid) order items**: none exist in MySQL — the upsert PK is safe.
- **base_discount** in orders.js: computed/stored in temp table but unused since migration 002 —
remove the column from temp table + queries for clarity (no behavior change).
- **`full-update.js` `runScript`**: try/catch around `console.log` is dead code; per-step
`status:'complete'` messages could confuse a UI parser. Cosmetic only — tidy if touching the file.
## Suggested implementation order
| Step | Fixes | Re-import/rebuild needed |
|---|---|---|
| 1 | Fix 1 + Fix 2 (+ Fix 5 filters, Fix 8/9 while editing the same SQL) | FULL orders re-import → snapshot rebuild → metrics (once) |
| 2 | Fix 4 + Fix 6 (orders.js reconciliation + watermarks; POs/products watermarks too) | no |
| 3 | Fix 3, Fix 7 (metrics SQL only) | metrics run |
| 4 | Fix 13-21 (robustness batch) | no |
| 5 | Fix 10-12, Fix 22 after Matt's sign-off (definition changes) | metrics run |
After step 1, expect: margin_30d down ~8-10 points, discounts_30d ~3x up, daily curves shifted to
correct business days. Communicate before/after so the change isn't mistaken for a data incident.
## Reference: verification snippets used in the review
```sql
-- MySQL: item-level discounts dropped by the gate (30d)
SELECT COUNT(DISTINCT o.order_id), ROUND(SUM(odi.amount),2)
FROM order_discount_items odi
JOIN order_discounts od ON od.order_id=odi.order_id AND od.discount_id=odi.discount_id
JOIN _order o ON o.order_id=odi.order_id
WHERE odi.which=2 AND o.date_placed >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
AND o.order_status >= 20 AND COALESCE(od.discount_amount_subtotal,0)=0;
-- → 2,021 orders / $25,989 (2026-06-10)
-- MySQL: costeach fallback frequency (30d)
SELECT COUNT(*),
SUM(CASE WHEN NOT EXISTS (SELECT 1 FROM order_costs oc WHERE oc.orderid=oi.order_id
AND oc.pid=oi.prod_pid AND oc.pending=0)
AND NOT EXISTS (SELECT 1 FROM product_inventory pi WHERE pi.pid=oi.prod_pid)
THEN 1 ELSE 0 END)
FROM order_items oi JOIN _order o ON o.order_id=oi.order_id
WHERE o.order_status >= 20 AND o.date_placed >= DATE_SUB(CURDATE(), INTERVAL 30 DAY);
-- → 729 / 34,833 = 2.1% (2026-06-10)
-- PG: timezone check
SHOW timezone; -- Europe/Berlin (2026-06-10)
-- Row drift, May 2026: MySQL 49,377 items / PG 49,841 (+0.9%)
```
TIME_UNIFICATION_PLAN.md
-217
View File
@@ -1,217 +0,0 @@
# Time Unification Plan
Audit date: 2026-06-12. Companion to `IMPORT_METRICS_FIX_PLAN.md` (import/metrics day-bucketing fixes, implemented 2026-06-11).
> **IMPLEMENTATION STATUS (2026-06-12):** Phases 1–6 code implemented. Shared
> module at `inventory-server/shared/business-time/` (+ frontend mirror
> `inventory/src/utils/businessTime.ts`); both duplicate utils deleted;
> acot-server hour-late bug fixed; `dateStrings: true` on all mysql2 configs
> except the import pipeline (dynamic Chicago offset instead);
> `forecast_engine.py` anchors on `business_today()`; pg pools pin
> `TimeZone=America/Chicago`; `/var/www/ecosystem.config.cjs` has
> `TZ: 'America/Chicago'` (backup at `~/ecosystem.config.cjs.bak.2026-06-12`);
> supervised import+metrics cycle ran clean. Docs at
> `inventory-server/docs/TIME.md`.
>
> **Still pending (Matt):** ① `sudo -u postgres psql -c "ALTER DATABASE
> inventory_db SET timezone = 'America/Chicago';"` ② `pm2 reload
> ecosystem.config.cjs --update-env` (mind the JWT_SECRET shell-var gotcha)
> ③ flush dashboard Redis ④ frontend `npm run build` (deploys) ⑤ Phase 2
> before/after verification + Klaviyo UTC spot-check. Deferred (non-correctness):
> GA4/Meta footnote labels. (UPDATE 2026-06-13: the two-selector-kit UI
> consolidation — `BusinessRangeSelect` + `HorizonTabs` — is now implemented; see Phase 5.)
## The Convention (single source of truth)
**A business day runs 1:00:00am Eastern → 12:59:59am Eastern the next day.**
Because 1am Eastern == midnight Central year-round (both observe US DST), this is identical to:
> **Business date = the America/Chicago calendar date.**
Practical consequences — these are the ONLY approved idioms:
| Context | Correct idiom |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| PG `timestamptz` → business date | `(ts AT TIME ZONE 'America/Chicago')::date` — or plain `ts::date` / `CURRENT_DATE` once Phase 1 sets the session TZ |
| MySQL DATETIME literal (stores Central wall-clock) → business date | `DATE(col)` — **no hour shift** |
| MySQL WHERE bounds for a business day | Central wall-clock strings: `col >= 'YYYY-MM-DD 00:00:00' AND col < 'YYYY-MM-DD+1 00:00:00'` |
| Luxon | `dt.setZone('America/Chicago')` (real conversion). Never `keepLocalTime`, never a fixed `UTC-05:00` offset |
| Intervals | **Half-open** `[start, nextStart)`. No `-1 minute`, no `-1 ms` endpoints |
| "Today" in JS | a `businessToday()` helper (Chicago date), never `new Date().toISOString().split('T')[0]` (UTC) and never bare `CURRENT_DATE` reasoning in Node |
## Verified facts (2026-06-11/12)
- MySQL (192.168.1.5) session/global TZ = SYSTEM = America/Chicago. `NOW()` = UTC-5 (CDT) ✓ live-checked.
- `_order.date_placed`, `_order.date_shipped` are DATETIME storing **Central wall-clock** ✓ (recent order: `date_placed 18:28` at 23:45 UTC).
- `timeclock.TimeStamp` stores **Central wall-clock** ✓ (8am-ET punch-ins recorded as ~07:03–07:06; `NOW()` = 08:14 at ~9:14am ET).
- `picking_ticket.createddate` — PHP-written, Central (same as all PHP DATETIME writes; spot-check during Phase 2 anyway).
- PostgreSQL `inventory_db` session TZ = **Europe/Berlin** ✓ (`SHOW timezone`).
- Node processes on netcup run with system TZ Europe/Berlin (no `TZ` in `.env`/ecosystem).
- Browsers/office = America/New_York.
- GA4 property reporting TZ = America/New_York (per Matt); API `NdaysAgo` resolves in property TZ.
- Klaviyo account TZ = Eastern; API documented to return UTC — **verify in Phase 5** (one event spot-check).
- PG columns: `orders.date`, `receivings.received_date`, `purchase_orders.date`, `products.created_at/date_online`, `product_metrics.last_calculated` = timestamptz; `stock_snapshots.snapshot_date`, `daily_product_snapshots.snapshot_date`, `purchase_orders.expected_date`, `product_forecasts.forecast_date` = DATE.
## Known defects this plan fixes
1. **acot-server dashboard windows are 1 hour late** (today = 2am–2am ET): `toDatabaseSqlString()` in `dashboard/acot-server/utils/timeUtils.js` passes Eastern wall-clock (via `keepLocalTime: true` + fixed `UTC-05:00`) to a Central-literal DB.
2. **Fixed `UTC-05:00` offsets** in `timeUtils.js` (`DB_TIMEZONE`) and `events.js:639` — wrong every winter (CST = UTC-6).
3. `**DATE_SUB(col, INTERVAL 1 HOUR)` daily bucketing** in operations/employee metrics — assumed Eastern storage; columns are Central, so buckets are also 1h late.
4. **payroll-metrics punch parsing**: mysql2 `timezone: 'Z'` turns `07:58` Central into `07:58Z`, then luxon renders it as 03:58 ET — punches shifted −5h before pay-period math; overnight punches (midnight–5am CT) land on the wrong day.
5. **PG routes use Berlin "today"**: every `CURRENT_DATE` / `NOW()` / `::date` / `DATE(col)` in `src/routes/`* and the window anchors in `scripts/metrics-new/*.sql` flip to the next business day at 6pm ET.
6. **JS UTC "today"** (`new Date().toISOString().split('T')[0]`) in `src/routes/dashboard.js` and others flips at 8pm ET — a third clock.
7. **DATE-column serialization off-by-one**: node-postgres returns `DATE` as a JS Date at *Berlin local midnight* (= 22:00/23:00 UTC the previous day); `d.toISOString().split('T')[0]` then emits the **previous day**. Affects `src/routes/dashboard.js` (~lines 335, 379, 478, 542, 779, 799, 812) and `src/routes/products.js:1006` wherever the value is a true DATE column.
8. **Two duplicate luxon utils disagree**: `TimeManager.getDayEnd` = next 1am −1 *minute* (59.999s hole); `timeUtils.getDayEnd` = −1 ms. `TimeManager.groupEventsByInterval` groups by `startOf('day')` in the event string's own zone — neither ET nor the 1am rule.
9. **Routes that ignore the convention**: `discounts.js` (zoneless `DateTime.fromISO` + `startOf('day')` = Berlin midnight), `customers.js` (MySQL `NOW()` windows), `events.js` hourly chart (Central hour labels shown to an Eastern office).
10. **Three mysql2 timezone configs for the same DB**: import `'-05:00'` (+ dynamic outbound fix), acot-server `'Z'`, `src/utils/dbConnection.js` + `src/routes/import.js` `'Z'`.
11. **Frontend**: custom ranges are browser-local midnight→23:59:59 (not business days); Forecasting's "Last Month" preset = *trailing* month while everywhere else it's *calendar* month; BlackFridayDashboard does UTC date math; 5 different range-option vocabularies; `date-range-picker-narrow.tsx` is dead code.
---
## Phase 1 — PostgreSQL & Node clocks (highest leverage, smallest diff)
**1.1 Set PG database timezone to business time.**
```sql
ALTER DATABASE inventory_db SET timezone = 'America/Chicago';
```
Then reconnect all pools (PM2 reload, see gotcha note at bottom; the 15-min cron picks it up on next run automatically since it opens fresh connections).
Effect: every `CURRENT_DATE`, `NOW()::date`, `ts::date`, `DATE(ts)`, `date_trunc('day', ts)` in routes and metrics SQL becomes business-correct *without touching the queries*. Explicit `AT TIME ZONE 'America/Chicago'` expressions are unaffected (they're absolute) — leave them; they now read as documentation.
- [ ] Run `ALTER DATABASE`; verify with `SHOW timezone` from a fresh `psql` and from the app pool (add a temporary log of `SELECT current_setting('timezone')` on pool connect).
- [ ] Belt-and-suspenders: also set it in the pool config in `src/utils/db.js` (`options: '-c TimeZone=America/Chicago'` or a `SET TIME ZONE` on connect) so a DB restore/migration can't silently revert it. Same for the pg pools in `scripts/import/`* and `scripts/forecast/run_forecast.js`.
- [ ] Sweep for anything that *wanted* UTC or Berlin semantics: `grep -rn "NOW()\|CURRENT_DATE\|CURRENT_TIMESTAMP" src/ scripts/ db/`. Audit found none — confirm. (Pipeline bookkeeping like `last_calculated >= NOW() - INTERVAL '5 minutes'` is interval-based and unaffected.)
- [ ] Check `forecast_engine.py`: any `datetime.now()`/`date.today()` used to anchor horizons should become Chicago-aware (`zoneinfo.ZoneInfo('America/Chicago')`). The engine writes `forecast_date` DATE rows — those must mean business dates.
**1.2 Set Node process TZ to business time.**
Add `TZ=America/Chicago` to the PM2 ecosystem env for `inventory-server` and `dashboard-server` (and the cron wrapper for `full-update.js`).
Why: (a) fixes defect #7 — node-postgres will materialize DATE columns at *Chicago* midnight, which round-trips through `toISOString().split('T')[0]` to the **same** date (Chicago is west of UTC); (b) makes incidental `new Date()` / `setHours` / `getHours` math business-aligned instead of Berlin.
- [ ] Add `TZ` to `ecosystem.config.cjs` env blocks; `pm2 reload ecosystem.config.cjs --update-env` (NOT `pm2 restart` — see PM2 env gotcha).
- [ ] Even with TZ set, replace the `.toISOString().split('T')[0]` idiom on DATE columns with `::text` in the SQL (cheapest, unambiguous) or a `formatDateCol()` helper — do this opportunistically in Phase 4; the TZ change makes it correct immediately.
- [ ] Note: `new Date().toISOString()` still yields UTC instants — fine for instants, never for "today". `businessToday()` from the shared module (Phase 3) is the only sanctioned "today".
**Verification (Phase 1):**
- [ ] Before/after at ~7pm ET (the danger window): `SELECT CURRENT_DATE` must equal the Chicago date, not tomorrow.
- [ ] `GET /api/dashboard` forecast series: dates must match `SELECT forecast_date::text` directly from PG (defect #7 regression check).
- [ ] Run a full `full-update.js` cycle; diff `product_metrics` sales_30d counts for a handful of pids against hand-computed Chicago windows.
## Phase 2 — Fix the acot-server hour-late bug (biggest visible-number impact)
All in `dashboard/acot-server/` (= `/var/www/inventory/dashboard/acot-server/`).
**2.1 `utils/timeUtils.js`:**
- [ ] Replace `DB_TIMEZONE = 'UTC-05:00'` with `DB_TIMEZONE = 'America/Chicago'`.
- [ ] `toDatabaseSqlString()`: `normalized.setZone(DB_TIMEZONE).toFormat(...)` — **delete `keepLocalTime: true*`*. 1:00am ET now correctly serializes as `00:00:00` (Central).
- [ ] `parseBusinessDate()` / `formatMySQLDate()`: same zone constant; they become DST-correct automatically.
- [ ] `getDayEnd()`: switch to half-open — return next day-start; change `getTimeRangeConditions` to emit `date_placed >= ? AND date_placed < ?`. (Eliminates the −1ms endpoint and any boundary-order ambiguity.)
- [ ] Keep `BUSINESS_DAY_START_HOUR = 1` ET — it's the same convention; the *conversion* was the bug.
**2.2 Daily bucketing in routes — remove the hand-rolled hour shift:**
- [ ] `routes/operations-metrics.js` (~~184–214) and `routes/employee-metrics.js` (~~324–351): `DATE_FORMAT(DATE_SUB(col, INTERVAL 1 HOUR), '%Y-%m-%d')` → `DATE(col)` for `pt.createddate`, `o.date_shipped`, `tc.TimeStamp` (all Central literals — verified). Update the misleading "Eastern timezone" comments.
- [ ] Where the JS then re-parses those date strings (`DateTime.fromFormat(date, 'yyyy-MM-dd', { zone: TIMEZONE })`) keep zone ET only for *labeling*; the string itself is already the business date — pass it through.
**2.3 `routes/events.js`:**
- [ ] Line ~639: `DateTime.fromSQL(value, { zone: 'UTC-05:00' })` → `{ zone: 'America/Chicago' }` (or route through `parseBusinessDate`).
- [ ] Hourly chart (today/yesterday): currently `HOUR(date_placed)` returns **Central** hours; office reads the axis as Eastern. Convert for display: emit the business timestamp and format hours in ET on the frontend, or `HOUR(CONVERT_TZ(date_placed,'America/Chicago','America/New_York'))` if the MySQL tz tables are loaded (verify with `SELECT CONVERT_TZ(NOW(),'America/Chicago','America/New_York')`; fall back to `ADDTIME(date_placed, '01:00:00')` — ET = CT + 1h always). Same for the `currentHour` source.
- [ ] Previous-period custom math (~553–561) is instant-math and stays as-is once bounds are correct.
**2.4 Routes not on the util:**
- [ ] `routes/discounts.js` (~62–85): replace zoneless `DateTime.fromISO` + Berlin `startOf('day')`/`endOf('day')` with `getTimeRangeConditions('custom', …)` / business-day helpers.
- [ ] `routes/customers.js` (~238–244): `DATE_SUB(NOW(), INTERVAL 3 MONTH)` is a rough trailing window — acceptable; add a comment that `NOW()` here is Central and that's intended. (Or switch to bound params from the util for uniformity.)
- [ ] `routes/payroll-metrics.js`: punches must be interpreted as Central. Either (preferred, see 2.5) get strings from the driver and parse with `DateTime.fromSQL(s, { zone: 'America/Chicago' })`, or correct the `'Z'`-mangled Dates explicitly. All pay-period boundary math stays in ET — that's display convention; the parse is what's broken. Re-test: a 7:58am-CT punch must render 8:58am ET and land on the right pay-period day; test an overnight punch (e.g. 00:30 CT).
**2.5 Driver config — make values unambiguous:**
- [ ] `db/connection.js`: add `dateStrings: true` (and drop `timezone: 'Z'`). All DATETIME/TIMESTAMP values arrive as literal strings; every consumer parses through the shared helper with zone Chicago. This converts the implicit-zone footgun into a compile-time-visible parse step. Sweep acot-server routes for `instanceof Date` assumptions and `new Date(row.X)` on datetime fields and route them through the parser.
**Verification (Phase 2):**
- [ ] Live-DB boundary check: place/find an order with `date_placed` between `00:00` and `01:00` CT (i.e. 1–2am ET); `timeRange=today` must include orders from `00:00:00` CT onward. Before the fix it starts at `01:00:00` CT.
- [ ] Compare `/stats?timeRange=yesterday` totals before/after; expect ~the 1am–2am ET hour to migrate between days.
- [ ] Employee daily hours for a known day vs the punch list by hand.
- [ ] Flush the dashboard Redis cache after deploy (cached responses are keyed by `timeRange` and hold pre-fix numbers).
## Phase 3 — One shared business-time module (kill the duplicate utils)
Create `inventory-server/shared/business-time/` (consistent with the shared-lib location and the side-service convention — this is a leaf dependency, not a middle layer). Export, roughly:
```
BUSINESS_TZ = 'America/Chicago' // canonical
OFFICE_TZ = 'America/New_York' // display only
businessToday() // Chicago calendar date (DateTime)
businessDayStart(d) / businessDayRange(d) // [start, nextStart)
rangeForPreset(name, now?) // today|yesterday|last7days|...|lastMonth — one vocabulary
previousRange(name|range)
toMySqlBound(dt) // Central wall-clock 'YYYY-MM-DD HH:mm:ss'
parseMySql(s) // zone Chicago → DateTime
businessDateOf(instant) // → 'YYYY-MM-DD'
formatOffice(dt, fmt) // ET display formatting
```
- [ ] Port `timeUtils.js` consumers (events/operations/employee/discounts/payroll) to it; delete `dashboard/acot-server/utils/timeUtils.js`.
- [ ] Port `TimeManager` consumers (klaviyo routes + `events.service.js`, `campaigns.service.js`, `reporting.service.js`, `redis.service.js`) to it; delete `dashboard/utils/time.utils.js`. Klaviyo API bounds stay UTC ISO (that part was right).
- [ ] Fix `groupEventsByInterval` while porting: group by `businessDateOf(eventInstant)` (day) and `setZone(OFFICE_TZ)` for hour/week/month — never the event string's own offset.
- [ ] One range vocabulary: the union of today's two sets — `today, yesterday, twoDaysAgo, thisWeek, lastWeek, thisMonth, lastMonth, last7days, last30days, last90days, previous7/30/90days, custom`. Both servers accept exactly this list; unknown → 400, not a silent default.
- [ ] ESM/CJS: dashboard tree is ESM, `src/` is CJS — publish the module dual (or `.mjs` with a tiny CJS wrapper), same pattern as existing shared lib.
- [ ] Klaviyo verification (per Matt): pull one known recent order event and confirm `attributes.datetime` is UTC (`Z`/+00:00) and matches the PG `orders.date` instant. If it ever arrives as ET-naive, add an explicit zone at the single parse point in the module.
## Phase 4 — inventory-server `src/routes` cleanup (mostly mechanical after Phase 1)
- [ ] `dashboard.js`: replace `new Date()`/`toISOString().split('T')[0]` "today"/default-range derivations (~309–316, 1041–1043, 1115–1120) with `businessToday()` / date-only strings; emit DATE columns via `::text` (lines ~335, 379, 478, 542, 769–815).
- [ ] `analytics.js`: default ranges (~19–24, 180–184) — drop the UTC truncation dance; accept `YYYY-MM-DD` business dates, default via `businessToday()`. `o.date::date` joins become business-correct after Phase 1 — change to the explicit `AT TIME ZONE 'America/Chicago'` idiom anyway in the hot endpoints so they don't regress if session TZ ever changes.
- [ ] `orders.js`: `fromDate/toDate` (~~13–45) — treat date-only input as business-day bounds (start of `from`'s business day to start of day after `to`), not UTC midnights; `DATE(date)` windows (~~115–131) fine post-Phase 1.
- [ ] `products.js:1006`, `purchase-orders.js:719/772` (UTC-today defaults), `repeat-orders.js`, `newsletter.js`: same two substitutions (business today; `::text` for DATE output). All `CURRENT_DATE` usages are correct after Phase 1 — leave them.
- [ ] `metrics-new/*.sql`: window anchors (`CURRENT_DATE - INTERVAL '59 days'` etc., `update_product_metrics.sql:11` `_current_date`) become business-correct via Phase 1 — no edits required; add a header comment to each file stating the session-TZ assumption.
- [ ] Standardize `src/utils/dbConnection.js` and the inline config in `src/routes/import.js` to `dateStrings: true` + shared parsing (they currently use `timezone: 'Z'`; consumers are product-text-centric so risk is low, but one convention).
- [ ] Import pipeline (`scripts/import/`): keep `adjustDateForMySQL()` for outbound params (it's the correct dynamic approach) — or migrate to `dateStrings: true` + `toMySqlBound()` for symmetry. Either way, inbound DATETIME *reads* must stop being parsed at fixed `-05:00`: audit `purchase-orders.js`/`daily-deals.js` for fields that survive into PG as instants and route them through `parseMySql()`. (`orders` uses `stamp` TIMESTAMP — wire-format follows session tz = Central; same parse rule applies.)
## Phase 5 — Frontend consolidation
**Principle: the server owns all boundary math.** The client sends either a named range or date-only strings (`YYYY-MM-DD`, meaning business dates); it never sends `toISOString()` of a browser-local midnight.
- [x] **Two sanctioned selector kits** — *implemented 2026-06-13* at `inventory/src/components/dashboard/shared/{BusinessRangeSelect,HorizonTabs}.tsx`.
1. `BusinessRangeSelect` — the `TIME_RANGES` presets. Migrated StatCards (×2), SalesChart, ProductGrid, KlaviyoCampaigns (all were identical `<Select>`-over-`TIME_RANGES` blocks). Shipped preset-only: those cards never had a custom-range UI, and FinancialOverview/OperationsMetrics keep their own natural-language period pickers (boundary math already fixed earlier in Phase 5). A `Custom…` popover can be added to the shared component later if a card needs it.
2. `HorizonTabs` — trailing-day windows (`7 / 30 / 90 / 365`, label map incl. `14D`/`1Y`). Migrated SalesMetrics (`7|30|90`), InventoryTrends/ValueTrend (`30|90|365`), InventoryFlow (`30|90`). **Corrections vs. this plan's list:** GrowthMomentum has *no* period selector (nothing to migrate), and the `30|90|'year'` union is actually ForecastMetrics — left alone because its 3rd option is a forward EOY forecast horizon, not a trailing window. Each card keeps its own fetch; this was a pure presentational/DRY swap. The GA4/Meta day-granular cards (MetaCampaigns/UserBehavior/Analytics) are left for the deferred Phase 6 footnote pass.
- [ ] `FinancialOverview` / `OperationsMetrics` custom periods: send the parsed period descriptor (`{type, startYear, startMonth|startQuarter, count}` from `naturalLanguagePeriod`) or derived date-only strings — not browser-midnight ISO instants (currently `setHours(0,0,0,0)`/`23:59:59.999` + `toISOString()`, FinancialOverview ~lines 560–620, 762–763). Server resolves to 1am-ET bounds. Client-side bucket alignment (`setHours(0,…)` ~390/432) should key on server-provided business-date strings instead.
- [ ] `MetaCampaigns` / `UserBehaviorDashboard` / `AnalyticsDashboard`: keep numeric `7/14/30/90` where the upstream API is day-granular (GA4/Meta), but render through `HorizonTabs` for visual consistency; label the GA/Meta cards' footnote "calendar days, midnight ET" (platform limitation — accepted deviation, see Phase 6).
- [ ] Forecasting `DateRangePickerQuick`: rename presets to "Trailing month / 3 / 6 / 12 months" (they are `addMonths(now, -N)`, not calendar months) or convert to calendar months — decide one; today the label collides with the dashboards' calendar "Last Month".
- [ ] Delete `src/components/ui/date-range-picker-narrow.tsx` (zero importers).
- [ ] `BlackFridayDashboard`: replace `getUTCFullYear/getUTCMonth` anchors with business-date helpers (a small `businessToday()` util in `src/utils/` — Chicago date via `Intl.DateTimeFormat('en-CA', { timeZone: 'America/Chicago' })`).
- [ ] `lib/dashboard/constants.js`: keep `TIME_RANGES` as the one preset list; `formatDateForInput`/`parseDateFromInput` stay (datetime-local is inherently browser-local) but custom values must be converted to date-only/business params at the request boundary, not `toISOString()`.
- [ ] `DateTime.jsx` (dashboard clock): browser-local is fine for a wall clock; pin to `America/New_York` via `toLocaleString` options so a remote viewer sees office time.
## Phase 6 — External services: document accepted deviations
These platforms bucket at **midnight ET**, not 1am ET. The 1-hour difference is not worth fighting their APIs; record it instead.
- [ ] GA4: property TZ = America/New_York; `NdaysAgo` = calendar days midnight ET. Accepted deviation (affects 0:00–1:00am ET traffic attribution only).
- [ ] Meta insights: `time_zone: 'America/New_York'` — same deviation, same note.
- [ ] Klaviyo: data fetched with exact UTC bounds from our own range math → fully on-convention once Phase 3 lands (plus the UTC-return verification).
- [ ] Add a `docs/TIME.md` (or a section in `.claude/CLAUDE.md`) stating: the convention, the approved idioms table, the accepted deviations, and "never hand-roll an hour offset."
## Rollout order & risk notes
1. Phase 2 (acot-server fix) is independent of everything — ship first; it corrects the most-watched numbers. Flush Redis after deploy.
2. Phase 1 next, off-hours (ideally before 6pm ET so Berlin/Chicago dates agree during the change); pause the auto-update (`touch inventory-server/.pause-auto-update`), apply, run one supervised `full-update.js`, compare metrics row counts, unpause.
3. Phases 3–4 together (module + ports), then 5, then 6.
4. PM2: env changes need `pm2 reload ecosystem.config.cjs --update-env` (restart won't pick up new env, except dashboard-server which loads dotenv itself).
5. DST regression dates for tests: **Nov 1 2026** (fall back) and **Mar 14 2027** (spring forward) — boundary tests for `rangeForPreset('today')`, `toMySqlBound`, and the metrics windows on both.
6. Expect small visible shifts in historical dashboard numbers after Phase 2 (the 1am–2am ET hour moves to its correct day) and after Phase 1 (evening-run metrics windows stop drifting). Worth a heads-up to anyone comparing screenshots.
## Decisions needed (Matt)
- [ ] Forecasting quick-presets: trailing (rename) or calendar months? trailing
- [ ] Hourly "today" chart axis: display Eastern hours (recommended) — confirm. - yes
- [ ] Custom calendar periods (e.g. "March 2026") = business month (Mar 1 1am ET → Apr 1 1am ET, recommended) — confirm, so FinancialOverview presets and custom agree. yes
docs/setup-chat.md
+1 -1
View File
@@ -13,7 +13,7 @@ Not all of the information in this database is relevant as it's a direct export
Server-side files should use similar conventions and the same technologies as the inventory-server (inventor-server root) and auth-server (inventory-server/auth). I will provide my current pm2 ecosystem file upon request for you to add the configuration for the new "chat-server". I use Caddy on the server and can provide my caddyfile to assist with configuring the api routes. All configuration and routes for the chat-server should go in the inventory-server/chat folder or subfolders you create.
The folder you see as inventory-server is actually a direct mount of the /var/www/inventory folder on the server. You can read and write files from there like usual, but any terminal commands for the server I will have to run myself.
The folder you see as inventory-server is actually a direct mount of the /var/www/html/inventory folder on the server. You can read and write files from there like usual, but any terminal commands for the server I will have to run myself.
The "Chat" page should be added to the main application sidebar and a similar page to the others should be created in inventory/src/pages. All other frontend pages should go in inventory/src/components/chat.
inventory-server/auth/add-user.js
-75
View File
@@ -1,75 +0,0 @@
import bcrypt from 'bcrypt';
import pg from 'pg';
import inquirer from 'inquirer';
const { Pool } = pg;
import { config as loadEnv } from 'dotenv';
import { fileURLToPath } from 'node:url';
import { dirname, resolve as resolvePath } from 'node:path';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
loadEnv({ path: resolvePath(__dirname, '../.env') });
const pool = new Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: Number(process.env.DB_PORT) || 5432,
});
async function promptUser() {
return inquirer.prompt([
{
type: 'input',
name: 'username',
message: 'Enter username:',
validate: (input) => input.length >= 3 || 'Username must be at least 3 characters long',
},
{
type: 'password',
name: 'password',
message: 'Enter password:',
mask: '*',
validate: (input) => input.length >= 8 || 'Password must be at least 8 characters long',
},
{
type: 'password',
name: 'confirmPassword',
message: 'Confirm password:',
mask: '*',
validate: (input, answers) => input === answers.password || 'Passwords do not match',
},
]);
}
async function addUser() {
try {
const { username, password } = await promptUser();
const hashedPassword = await bcrypt.hash(password, 10);
const checkResult = await pool.query(
'SELECT id FROM users WHERE username = $1',
[username]
);
if (checkResult.rows.length > 0) {
console.error('Error: Username already exists');
process.exit(1);
}
const result = await pool.query(
'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id',
[username, hashedPassword]
);
console.log(`User ${username} created successfully with id ${result.rows[0].id}`);
} catch (error) {
console.error('Error creating user:', error);
if (error.code) console.error('Error code:', error.code);
} finally {
await pool.end();
}
}
addUser();
inventory-server/auth/package-lock.json
Generated
-2275
View File
File diff suppressed because it is too large Load Diff
inventory-server/auth/package.json
-23
View File
@@ -1,23 +0,0 @@
{
"name": "inventory-auth-server",
"version": "1.0.0",
"description": "Authentication server for inventory management system",
"type": "module",
"main": "server.js",
"scripts": {
"start": "node server.js",
"add-user": "node add-user.js"
},
"dependencies": {
"bcrypt": "^5.1.1",
"cors": "^2.8.5",
"dotenv": "^16.4.7",
"express": "^4.18.2",
"express-rate-limit": "^7.4.0",
"inquirer": "^8.2.6",
"jsonwebtoken": "^9.0.2",
"pg": "^8.11.3",
"pino": "^9.5.0",
"pino-http": "^10.3.0"
}
}
inventory-server/auth/permissions.js
-73
View File
@@ -1,73 +0,0 @@
export function createPermissionHelpers({ pool }) {
async function checkPermission(userId, permissionCode) {
try {
const adminResult = await pool.query(
'SELECT is_admin FROM users WHERE id = $1',
[userId]
);
if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) return true;
const result = await pool.query(
`SELECT COUNT(*) AS has_permission
FROM user_permissions up
JOIN permissions p ON up.permission_id = p.id
WHERE up.user_id = $1 AND p.code = $2`,
[userId, permissionCode]
);
return Number(result.rows[0].has_permission) > 0;
} catch (error) {
console.error('Error checking permission:', error);
return false;
}
}
function requirePermission(permissionCode) {
return async (req, res, next) => {
try {
if (!req.user?.id) {
return res.status(401).json({ error: 'Authentication required' });
}
const hasPermission = await checkPermission(req.user.id, permissionCode);
if (!hasPermission) {
return res.status(403).json({
error: 'Insufficient permissions',
requiredPermission: permissionCode,
});
}
next();
} catch (error) {
console.error('Permission middleware error:', error);
res.status(500).json({ error: 'Server error checking permissions' });
}
};
}
async function getUserPermissions(userId) {
try {
const adminResult = await pool.query(
'SELECT is_admin FROM users WHERE id = $1',
[userId]
);
if (adminResult.rows.length === 0) return [];
if (adminResult.rows[0].is_admin) {
const allPermissions = await pool.query('SELECT code FROM permissions');
return allPermissions.rows.map((p) => p.code);
}
const permissions = await pool.query(
`SELECT p.code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1`,
[userId]
);
return permissions.rows.map((p) => p.code);
} catch (error) {
console.error('Error getting user permissions:', error);
return [];
}
}
return { checkPermission, requirePermission, getUserPermissions };
}
inventory-server/auth/routes.js
-325
View File
@@ -1,325 +0,0 @@
import express from 'express';
import bcrypt from 'bcrypt';
import jwt from 'jsonwebtoken';
import { createPermissionHelpers } from './permissions.js';
export function createAuthRoutes({ pool }) {
const router = express.Router();
const { requirePermission, getUserPermissions } = createPermissionHelpers({ pool });
// Local authenticate(): used by user-management endpoints that need req.user populated
// with id/username/email/is_admin. NOT the per-service authenticate() — that lives in
// shared/auth/middleware.js and is used by downstream services. Auth-server's surface is
// small enough that a local copy is fine; the security boundary is the JWT verify step.
async function authenticate(req, res, next) {
const authHeader = req.headers.authorization;
if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({ error: 'Authentication required' });
}
const token = authHeader.split(' ')[1];
let decoded;
try {
decoded = jwt.verify(token, process.env.JWT_SECRET);
} catch (error) {
return res.status(401).json({ error: 'Invalid token' });
}
// DB failures are 500, not 401 — the frontend ends the session on 401,
// and a transient outage must not log users out.
try {
const result = await pool.query(
'SELECT id, username, email, is_admin, rocket_chat_user_id FROM users WHERE id = $1',
[decoded.userId]
);
if (result.rows.length === 0) {
return res.status(401).json({ error: 'User not found' });
}
req.user = result.rows[0];
next();
} catch (error) {
console.error('Auth DB error:', error);
res.status(500).json({ error: 'Server error' });
}
}
router.post('/login', async (req, res) => {
try {
const { username, password } = req.body;
const result = await pool.query(
'SELECT id, username, password, is_admin, is_active, rocket_chat_user_id FROM users WHERE username = $1',
[username]
);
if (result.rows.length === 0) {
return res.status(401).json({ error: 'Invalid username or password' });
}
const user = result.rows[0];
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
const validPassword = await bcrypt.compare(password, user.password);
if (!validPassword) {
return res.status(401).json({ error: 'Invalid username or password' });
}
await pool.query(
'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
[user.id]
);
const token = jwt.sign(
{ userId: user.id, username: user.username },
process.env.JWT_SECRET,
{ expiresIn: '7d' }
);
const permissions = await getUserPermissions(user.id);
res.json({
token,
user: {
id: user.id,
username: user.username,
is_admin: user.is_admin,
rocket_chat_user_id: user.rocket_chat_user_id,
permissions,
},
});
} catch (error) {
console.error('Login error:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.get('/me', authenticate, async (req, res) => {
try {
const permissions = await getUserPermissions(req.user.id);
res.json({
id: req.user.id,
username: req.user.username,
email: req.user.email,
is_admin: req.user.is_admin,
rocket_chat_user_id: req.user.rocket_chat_user_id,
permissions,
});
} catch (error) {
console.error('Error getting current user:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const result = await pool.query(`
SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
FROM users
ORDER BY username
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting users:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const userId = req.params.id;
const userResult = await pool.query(`
SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
FROM users
WHERE id = $1
`, [userId]);
if (userResult.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
const permissionsResult = await pool.query(`
SELECT p.id, p.name, p.code, p.category, p.description
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
ORDER BY p.category, p.name
`, [userId]);
res.json({
...userResult.rows[0],
permissions: permissionsResult.rows,
});
} catch (error) {
console.error('Error getting user:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
const client = await pool.connect();
try {
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
if (!username || !password) {
return res.status(400).json({ error: 'Username and password are required' });
}
const existingUser = await client.query(
'SELECT id FROM users WHERE username = $1',
[username]
);
if (existingUser.rows.length > 0) {
return res.status(400).json({ error: 'Username already exists' });
}
await client.query('BEGIN');
const hashedPassword = await bcrypt.hash(password, 10);
const rcUserId = rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null;
const userResult = await client.query(`
INSERT INTO users (username, email, password, is_admin, is_active, rocket_chat_user_id, created_at)
VALUES ($1, $2, $3, $4, $5, $6, CURRENT_TIMESTAMP)
RETURNING id
`, [username, email || null, hashedPassword, !!is_admin, is_active !== false, rcUserId]);
const userId = userResult.rows[0].id;
if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
const permissionIds = normalizePermissionIds(permissions);
if (permissionIds.length > 0) {
await client.query(
`INSERT INTO user_permissions (user_id, permission_id)
SELECT $1, unnest($2::int[])
ON CONFLICT DO NOTHING`,
[userId, permissionIds]
);
}
}
await client.query('COMMIT');
res.status(201).json({ id: userId, message: 'User created successfully' });
} catch (error) {
await client.query('ROLLBACK');
console.error('Error creating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
const client = await pool.connect();
try {
const userId = req.params.id;
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
const userExists = await client.query('SELECT id FROM users WHERE id = $1', [userId]);
if (userExists.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
await client.query('BEGIN');
const updateFields = [];
const updateValues = [userId];
let paramIndex = 2;
if (username !== undefined) { updateFields.push(`username = $${paramIndex++}`); updateValues.push(username); }
if (email !== undefined) { updateFields.push(`email = $${paramIndex++}`); updateValues.push(email || null); }
if (is_admin !== undefined) { updateFields.push(`is_admin = $${paramIndex++}`); updateValues.push(!!is_admin); }
if (is_active !== undefined) { updateFields.push(`is_active = $${paramIndex++}`); updateValues.push(!!is_active); }
if (rocket_chat_user_id !== undefined) {
updateFields.push(`rocket_chat_user_id = $${paramIndex++}`);
updateValues.push(rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null);
}
if (password) {
const hashedPassword = await bcrypt.hash(password, 10);
updateFields.push(`password = $${paramIndex++}`);
updateValues.push(hashedPassword);
}
if (updateFields.length > 0) {
updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
await client.query(`
UPDATE users SET ${updateFields.join(', ')} WHERE id = $1
`, updateValues);
}
if (Array.isArray(permissions)) {
await client.query('DELETE FROM user_permissions WHERE user_id = $1', [userId]);
const newIsAdmin = is_admin !== undefined
? is_admin
: (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
if (!newIsAdmin && permissions.length > 0) {
const permissionIds = normalizePermissionIds(permissions);
if (permissionIds.length > 0) {
await client.query(
`INSERT INTO user_permissions (user_id, permission_id)
SELECT $1, unnest($2::int[])
ON CONFLICT DO NOTHING`,
[userId, permissionIds]
);
}
}
}
await client.query('COMMIT');
res.json({ message: 'User updated successfully' });
} catch (error) {
await client.query('ROLLBACK');
console.error('Error updating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => {
try {
const userId = req.params.id;
if (req.user.id === parseInt(userId, 10)) {
return res.status(400).json({ error: 'Cannot delete your own account' });
}
const result = await pool.query(
'DELETE FROM users WHERE id = $1 RETURNING id',
[userId]
);
if (result.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
res.json({ message: 'User deleted successfully' });
} catch (error) {
console.error('Error deleting user:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const result = await pool.query(`
SELECT category, json_agg(
json_build_object(
'id', id, 'name', name, 'code', code, 'description', description
) ORDER BY name
) as permissions
FROM permissions
GROUP BY category
ORDER BY category
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
}
});
router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const result = await pool.query(`
SELECT * FROM permissions ORDER BY category, name
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
}
});
return router;
}
function normalizePermissionIds(permissions) {
return permissions
.map((p) => {
if (typeof p === 'object' && p?.id) return parseInt(p.id, 10);
if (typeof p === 'number') return p;
if (typeof p === 'string' && !Number.isNaN(parseInt(p, 10))) return parseInt(p, 10);
return null;
})
.filter((id) => id !== null && !Number.isNaN(id));
}
inventory-server/auth/schema.sql
-89
View File
@@ -1,89 +0,0 @@
CREATE TABLE users (
id SERIAL PRIMARY KEY,
username VARCHAR(255) NOT NULL UNIQUE,
password VARCHAR(255) NOT NULL,
email VARCHAR UNIQUE,
is_admin BOOLEAN DEFAULT FALSE,
is_active BOOLEAN DEFAULT TRUE,
last_login TIMESTAMP WITH TIME ZONE,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Function to update the updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = CURRENT_TIMESTAMP;
RETURN NEW;
END;
$$ language 'plpgsql';
-- Sequence and defined type for users table if not exists
CREATE SEQUENCE IF NOT EXISTS users_id_seq;
-- Create permissions table
CREATE TABLE IF NOT EXISTS "public"."permissions" (
"id" SERIAL PRIMARY KEY,
"name" varchar NOT NULL UNIQUE,
"code" varchar NOT NULL UNIQUE,
"description" text,
"category" varchar NOT NULL,
"created_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
"updated_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP
);
-- Create user_permissions junction table
CREATE TABLE IF NOT EXISTS "public"."user_permissions" (
"user_id" int4 NOT NULL REFERENCES "public"."users"("id") ON DELETE CASCADE,
"permission_id" int4 NOT NULL REFERENCES "public"."permissions"("id") ON DELETE CASCADE,
"created_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY ("user_id", "permission_id")
);
-- Add triggers for updated_at on users and permissions
DROP TRIGGER IF EXISTS update_users_updated_at ON users;
CREATE TRIGGER update_users_updated_at
BEFORE UPDATE ON users
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
DROP TRIGGER IF EXISTS update_permissions_updated_at ON permissions;
CREATE TRIGGER update_permissions_updated_at
BEFORE UPDATE ON permissions
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
-- Insert default permissions by page - only the ones used in application
INSERT INTO permissions (name, code, description, category) VALUES
('Dashboard Access', 'access:dashboard', 'Can access the Dashboard page', 'Pages'),
('Products Access', 'access:products', 'Can access the Products page', 'Pages'),
('Categories Access', 'access:categories', 'Can access the Categories page', 'Pages'),
('Vendors Access', 'access:vendors', 'Can access the Vendors page', 'Pages'),
('Analytics Access', 'access:analytics', 'Can access the Analytics page', 'Pages'),
('Forecasting Access', 'access:forecasting', 'Can access the Forecasting page', 'Pages'),
('Purchase Orders Access', 'access:purchase_orders', 'Can access the Purchase Orders page', 'Pages'),
('Import Access', 'access:import', 'Can access the Import page', 'Pages'),
('Settings Access', 'access:settings', 'Can access the Settings page', 'Pages'),
('AI Validation Debug Access', 'access:ai_validation_debug', 'Can access the AI Validation Debug page', 'Pages')
ON CONFLICT (code) DO NOTHING;
-- Settings section permissions
INSERT INTO permissions (name, code, description, category) VALUES
('Data Management', 'settings:data_management', 'Access to the Data Management settings section', 'Settings'),
('Stock Management', 'settings:stock_management', 'Access to the Stock Management settings section', 'Settings'),
('Performance Metrics', 'settings:performance_metrics', 'Access to the Performance Metrics settings section', 'Settings'),
('Calculation Settings', 'settings:calculation_settings', 'Access to the Calculation Settings section', 'Settings'),
('Template Management', 'settings:templates', 'Access to the Template Management settings section', 'Settings'),
('User Management', 'settings:user_management', 'Access to the User Management settings section', 'Settings')
ON CONFLICT (code) DO NOTHING;
-- Set any existing users as admin
UPDATE users SET is_admin = TRUE WHERE is_admin IS NULL;
-- Grant all permissions to admin users
INSERT INTO user_permissions (user_id, permission_id)
SELECT u.id, p.id
FROM users u, permissions p
WHERE u.is_admin = TRUE
ON CONFLICT DO NOTHING;
inventory-server/auth/server.js
+152 -65
View File
@@ -1,89 +1,176 @@
import 'dotenv/config';
import express from 'express';
import cors from 'cors';
import pg from 'pg';
import { fileURLToPath } from 'node:url';
require('dotenv').config({ path: '../.env' });
const express = require('express');
const cors = require('cors');
const bcrypt = require('bcrypt');
const jwt = require('jsonwebtoken');
const { Pool } = require('pg');
const morgan = require('morgan');
const authRoutes = require('./routes');
const { Pool } = pg;
import { dirname, resolve as resolvePath } from 'node:path';
import { config as loadEnv } from 'dotenv';
import { corsOptions } from '../shared/cors/policy.js';
import { requestLog } from '../shared/logging/request-log.js';
import { logger } from '../shared/logging/logger.js';
import { errorHandler } from '../shared/errors/handler.js';
import { loginLimiter, verifyLimiter } from '../shared/rate-limit/login.js';
import { extractBearerToken, verifyToken, TokenError } from '../shared/auth/verify.js';
import { createAuthRoutes } from './routes.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// auth/ lives at inventory-server/auth/, so .env one level up
loadEnv({ path: resolvePath(__dirname, '../.env') });
if (!process.env.JWT_SECRET) {
logger.error('JWT_SECRET is not set; refusing to start');
process.exit(1);
}
logger.info({
// Log startup configuration
console.log('Starting auth server with config:', {
host: process.env.DB_HOST,
user: process.env.DB_USER,
database: process.env.DB_NAME,
port: process.env.DB_PORT,
auth_port: process.env.AUTH_PORT,
}, 'starting auth server');
auth_port: process.env.AUTH_PORT
});
const app = express();
const port = Number(process.env.AUTH_PORT) || 3011;
// Behind Caddy on localhost: without this, req.ip is ::1 for every request and
// the rate limiters put ALL users in one shared bucket (10 logins/15min for
// the whole office). Same setting as inventory-server's server.js.
app.set('trust proxy', 'loopback');
const port = process.env.AUTH_PORT || 3011;
// Database configuration
const pool = new Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: Number(process.env.DB_PORT) || 5432,
port: process.env.DB_PORT,
});
app.use(requestLog());
app.use(express.json({ limit: '1mb' }));
app.use(cors(corsOptions));
// Make pool available globally
global.pool = pool;
// Middleware
app.use(express.json());
app.use(morgan('combined'));
app.use(cors({
origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw', 'https://tools.acherryontop.com', 'https://tools.acherryontop.com'],
credentials: true
}));
// Login endpoint
app.post('/login', async (req, res) => {
const { username, password } = req.body;
// Caddy forward_auth target: JWT signature check only, no DB hit.
// Returns 200 with X-User-Id / X-User-Username on success; 401 otherwise.
// Per-service middleware re-verifies independently; these headers are informational.
app.all('/verify', verifyLimiter, (req, res) => {
try {
const token = extractBearerToken(req.headers.authorization);
const decoded = verifyToken(token, process.env.JWT_SECRET);
res.set('X-User-Id', String(decoded.userId));
if (decoded.username) res.set('X-User-Username', decoded.username);
res.status(200).end();
} catch (err) {
if (err instanceof TokenError) {
return res.status(401).json({ error: err.message });
// Get user from database
const result = await pool.query(
'SELECT id, username, password, is_admin, is_active FROM users WHERE username = $1',
[username]
);
const user = result.rows[0];
// Check if user exists and password is correct
if (!user || !(await bcrypt.compare(password, user.password))) {
return res.status(401).json({ error: 'Invalid username or password' });
}
// Check if user is active
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
// Update last login timestamp
await pool.query(
'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
[user.id]
);
// Generate JWT token
const token = jwt.sign(
{ userId: user.id, username: user.username },
process.env.JWT_SECRET,
{ expiresIn: '24h' }
);
// Get user permissions for the response
const permissionsResult = await pool.query(`
SELECT code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
`, [user.id]);
const permissions = permissionsResult.rows.map(row => row.code);
res.json({
token,
user: {
id: user.id,
username: user.username,
is_admin: user.is_admin,
permissions: user.is_admin ? [] : permissions
}
});
} catch (error) {
console.error('Login error:', error);
res.status(500).json({ error: 'Internal server error' });
}
});
// User info endpoint
app.get('/me', async (req, res) => {
const authHeader = req.headers.authorization;
if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({ error: 'No token provided' });
}
try {
const token = authHeader.split(' ')[1];
const decoded = jwt.verify(token, process.env.JWT_SECRET);
// Get user details from database
const userResult = await pool.query(
'SELECT id, username, email, is_admin, rocket_chat_user_id, is_active FROM users WHERE id = $1',
[decoded.userId]
);
if (userResult.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
const user = userResult.rows[0];
// Check if user is active
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
// Get user permissions
let permissions = [];
if (!user.is_admin) {
const permissionsResult = await pool.query(`
SELECT code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
`, [user.id]);
permissions = permissionsResult.rows.map(row => row.code);
}
res.json({
id: user.id,
username: user.username,
email: user.email,
rocket_chat_user_id: user.rocket_chat_user_id,
is_admin: user.is_admin,
permissions: permissions
});
} catch (error) {
console.error('Token verification error:', error);
res.status(401).json({ error: 'Invalid token' });
}
});
// Login route gets its own rate limiter to slow credential stuffing.
app.use('/login', loginLimiter);
// Mount all routes from routes.js
app.use('/', authRoutes);
// Mount user-management + /login + /me from routes.js
app.use('/', createAuthRoutes({ pool }));
app.get('/health', (req, res) => res.json({ status: 'healthy' }));
app.use(errorHandler);
app.listen(port, () => {
logger.info({ port }, 'auth server listening');
// Health check endpoint
app.get('/health', (req, res) => {
res.json({ status: 'healthy' });
});
// Error handling middleware
app.use((err, req, res, next) => {
console.error(err.stack);
res.status(500).json({ error: 'Something broke!' });
});
// Start server
app.listen(port, () => {
console.log(`Auth server running on port ${port}`);
});
inventory-server/chat/create-new-database.sql
-45
View File
@@ -1,45 +0,0 @@
-- PostgreSQL Database Creation Script for New Server
-- Run as: sudo -u postgres psql -f create-new-database.sql
-- Terminate all connections to the database (if it exists)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'rocketchat_converted' AND pid <> pg_backend_pid();
-- Drop the database if it exists
DROP DATABASE IF EXISTS rocketchat_converted;
-- Create fresh database
CREATE DATABASE rocketchat_converted;
-- Create user (if not exists) - UPDATE PASSWORD BEFORE RUNNING!
DO $$
BEGIN
IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'rocketchat_user') THEN
CREATE USER rocketchat_user WITH PASSWORD 'HKjLgt23gWuPXzEAn3rW';
END IF;
END $$;
-- Grant database privileges
GRANT CONNECT ON DATABASE rocketchat_converted TO rocketchat_user;
GRANT CREATE ON DATABASE rocketchat_converted TO rocketchat_user;
-- Connect to the new database
\c rocketchat_converted;
-- Grant schema privileges
GRANT CREATE ON SCHEMA public TO rocketchat_user;
GRANT USAGE ON SCHEMA public TO rocketchat_user;
-- Grant privileges on all future tables and sequences
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO rocketchat_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO rocketchat_user;
-- Display success message
\echo 'Database created successfully!'
\echo 'IMPORTANT: Update the password for rocketchat_user before proceeding'
\echo 'Next steps:'
\echo '1. Update the password in this file'
\echo '2. Run export-chat-data.sh on your current server'
\echo '3. Transfer the exported files to this server'
\echo '4. Run import-chat-data.sh on this server'
inventory-server/chat/db-convert/mongo_to_postgres_converter.py
-881
View File
@@ -1,881 +0,0 @@
#!/usr/bin/env python3
"""
MongoDB to PostgreSQL Converter for Rocket.Chat
Converts MongoDB BSON export files to PostgreSQL database
Usage:
python3 mongo_to_postgres_converter.py \
--mongo-path db/database/62df06d44234d20001289144 \
--pg-database rocketchat_converted \
--pg-user rocketchat_user \
--pg-password your_password \
--debug
"""
import json
import os
import re
import subprocess
import sys
import struct
from datetime import datetime
from pathlib import Path
from typing import Dict, Any, List, Optional
import argparse
import traceback
# Auto-install dependencies if needed
try:
import bson
import psycopg2
except ImportError:
print("Installing required packages...")
subprocess.check_call([sys.executable, "-m", "pip", "install", "pymongo", "psycopg2-binary"])
import bson
import psycopg2
class MongoToPostgresConverter:
def __init__(self, mongo_db_path: str, postgres_config: Dict[str, str], debug_mode: bool = False, debug_collections: List[str] = None):
self.mongo_db_path = Path(mongo_db_path)
self.postgres_config = postgres_config
self.debug_mode = debug_mode
self.debug_collections = debug_collections or []
self.collections = {}
self.schema_info = {}
self.error_log = {}
def log_debug(self, message: str, collection: str = None):
"""Log debug messages if debug mode is enabled and collection is in debug list"""
if self.debug_mode and (not self.debug_collections or collection in self.debug_collections):
print(f"DEBUG: {message}")
def log_error(self, collection: str, error_type: str, details: str):
"""Log detailed error information"""
if collection not in self.error_log:
self.error_log[collection] = []
self.error_log[collection].append({
'type': error_type,
'details': details,
'timestamp': datetime.now().isoformat()
})
def sample_documents(self, collection_name: str, max_samples: int = 3) -> List[Dict]:
"""Sample documents from a collection for debugging"""
if not self.debug_mode or (self.debug_collections and collection_name not in self.debug_collections):
return []
print(f"\n🔍 Sampling documents from {collection_name}:")
bson_file = self.collections[collection_name]['bson_file']
if bson_file.stat().st_size == 0:
print(" Collection is empty")
return []
samples = []
try:
with open(bson_file, 'rb') as f:
sample_count = 0
while sample_count < max_samples:
try:
doc_size = int.from_bytes(f.read(4), byteorder='little')
if doc_size <= 0:
break
f.seek(-4, 1)
doc_bytes = f.read(doc_size)
if len(doc_bytes) != doc_size:
break
doc = bson.decode(doc_bytes)
samples.append(doc)
sample_count += 1
print(f" Sample {sample_count} - Keys: {list(doc.keys())}")
# Show a few key fields with their types and truncated values
for key, value in list(doc.items())[:3]:
value_preview = str(value)[:50] + "..." if len(str(value)) > 50 else str(value)
print(f" {key}: {type(value).__name__} = {value_preview}")
if len(doc) > 3:
print(f" ... and {len(doc) - 3} more fields")
print()
except (bson.InvalidBSON, struct.error, OSError) as e:
self.log_error(collection_name, 'document_parsing', str(e))
break
except Exception as e:
self.log_error(collection_name, 'file_reading', str(e))
print(f" Error reading collection: {e}")
return samples
def discover_collections(self):
"""Discover all BSON files and their metadata"""
print("Discovering MongoDB collections...")
for bson_file in self.mongo_db_path.glob("*.bson"):
collection_name = bson_file.stem
metadata_file = bson_file.with_suffix(".metadata.json")
# Read metadata if available
metadata = {}
if metadata_file.exists():
try:
with open(metadata_file, 'r', encoding='utf-8') as f:
metadata = json.load(f)
except (UnicodeDecodeError, json.JSONDecodeError) as e:
print(f"Warning: Could not read metadata for {collection_name}: {e}")
metadata = {}
# Get file size and document count estimate
file_size = bson_file.stat().st_size
doc_count = self._estimate_document_count(bson_file)
self.collections[collection_name] = {
'bson_file': bson_file,
'metadata': metadata,
'file_size': file_size,
'estimated_docs': doc_count
}
print(f"Found {len(self.collections)} collections")
for name, info in self.collections.items():
print(f" - {name}: {info['file_size']/1024/1024:.1f}MB (~{info['estimated_docs']} docs)")
def _estimate_document_count(self, bson_file: Path) -> int:
"""Estimate document count by reading first few documents"""
if bson_file.stat().st_size == 0:
return 0
try:
with open(bson_file, 'rb') as f:
docs_sampled = 0
bytes_sampled = 0
max_sample_size = min(1024 * 1024, bson_file.stat().st_size) # 1MB or file size
while bytes_sampled < max_sample_size:
try:
doc_size = int.from_bytes(f.read(4), byteorder='little')
if doc_size <= 0 or doc_size > 16 * 1024 * 1024: # MongoDB doc size limit
break
f.seek(-4, 1) # Go back
doc_bytes = f.read(doc_size)
if len(doc_bytes) != doc_size:
break
bson.decode(doc_bytes) # Validate it's a valid BSON document
docs_sampled += 1
bytes_sampled += doc_size
except (bson.InvalidBSON, struct.error, OSError):
break
if docs_sampled > 0 and bytes_sampled > 0:
avg_doc_size = bytes_sampled / docs_sampled
return int(bson_file.stat().st_size / avg_doc_size)
except Exception:
pass
return 0
def analyze_schema(self, collection_name: str, sample_size: int = 100) -> Dict[str, Any]:
"""Analyze collection schema by sampling documents"""
print(f"Analyzing schema for {collection_name}...")
bson_file = self.collections[collection_name]['bson_file']
if bson_file.stat().st_size == 0:
return {}
schema = {}
docs_analyzed = 0
try:
with open(bson_file, 'rb') as f:
while docs_analyzed < sample_size:
try:
doc_size = int.from_bytes(f.read(4), byteorder='little')
if doc_size <= 0:
break
f.seek(-4, 1)
doc_bytes = f.read(doc_size)
if len(doc_bytes) != doc_size:
break
doc = bson.decode(doc_bytes)
self._analyze_document_schema(doc, schema)
docs_analyzed += 1
except (bson.InvalidBSON, struct.error, OSError):
break
except Exception as e:
print(f"Error analyzing {collection_name}: {e}")
self.schema_info[collection_name] = schema
return schema
def _analyze_document_schema(self, doc: Dict[str, Any], schema: Dict[str, Any], prefix: str = ""):
"""Recursively analyze document structure"""
for key, value in doc.items():
full_key = f"{prefix}.{key}" if prefix else key
if full_key not in schema:
schema[full_key] = {
'types': set(),
'null_count': 0,
'total_count': 0,
'is_array': False,
'nested_schema': {}
}
schema[full_key]['total_count'] += 1
if value is None:
schema[full_key]['null_count'] += 1
schema[full_key]['types'].add('null')
elif isinstance(value, dict):
schema[full_key]['types'].add('object')
if 'nested_schema' not in schema[full_key]:
schema[full_key]['nested_schema'] = {}
self._analyze_document_schema(value, schema[full_key]['nested_schema'])
elif isinstance(value, list):
schema[full_key]['types'].add('array')
schema[full_key]['is_array'] = True
if value and isinstance(value[0], dict):
if 'array_item_schema' not in schema[full_key]:
schema[full_key]['array_item_schema'] = {}
for item in value[:5]: # Sample first 5 items
if isinstance(item, dict):
self._analyze_document_schema(item, schema[full_key]['array_item_schema'])
else:
schema[full_key]['types'].add(type(value).__name__)
def generate_postgres_schema(self) -> Dict[str, str]:
"""Generate PostgreSQL CREATE TABLE statements"""
print("Generating PostgreSQL schema...")
table_definitions = {}
for collection_name, schema in self.schema_info.items():
if not schema: # Empty collection
continue
table_name = self._sanitize_table_name(collection_name)
columns = []
# Always add an id column (PostgreSQL doesn't use _id like MongoDB)
columns.append("id SERIAL PRIMARY KEY")
for field_name, field_info in schema.items():
if field_name == '_id':
columns.append("mongo_id TEXT") # Always allow NULL for mongo_id
continue
col_name = self._sanitize_column_name(field_name)
# Handle conflicts with PostgreSQL auto-generated columns
if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
col_name = f"field_{col_name}"
col_type = self._determine_postgres_type(field_info)
# Make all fields nullable by default to avoid constraint violations
columns.append(f"{col_name} {col_type}")
# Add metadata columns
columns.extend([
"created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP",
"updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP"
])
column_definitions = ',\n '.join(columns)
table_sql = f"""
CREATE TABLE IF NOT EXISTS {table_name} (
{column_definitions}
);
-- Create indexes based on MongoDB indexes
"""
# Get list of actual columns that will exist in the table
existing_columns = set(['id', 'mongo_id', 'created_at', 'updated_at'])
for field_name in schema.keys():
if field_name != '_id':
col_name = self._sanitize_column_name(field_name)
# Handle conflicts with PostgreSQL auto-generated columns
if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
col_name = f"field_{col_name}"
existing_columns.add(col_name)
# Add indexes from MongoDB metadata
metadata = self.collections[collection_name].get('metadata', {})
indexes = metadata.get('indexes', [])
for index in indexes:
if index['name'] != '_id_': # Skip the default _id index
# Sanitize index name - remove special characters
sanitized_index_name = re.sub(r'[^a-zA-Z0-9_]', '_', index['name'])
index_name = f"idx_{table_name}_{sanitized_index_name}"
index_keys = list(index['key'].keys())
if index_keys:
sanitized_keys = []
for key in index_keys:
if key != '_id':
sanitized_key = self._sanitize_column_name(key)
# Handle conflicts with PostgreSQL auto-generated columns
if sanitized_key in ['id', 'mongo_id', 'created_at', 'updated_at']:
sanitized_key = f"field_{sanitized_key}"
# Only add if the column actually exists in our table
if sanitized_key in existing_columns:
sanitized_keys.append(sanitized_key)
if sanitized_keys:
table_sql += f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name} ({', '.join(sanitized_keys)});\n"
table_definitions[collection_name] = table_sql
return table_definitions
def _sanitize_table_name(self, name: str) -> str:
"""Convert MongoDB collection name to PostgreSQL table name"""
# Remove rocketchat_ prefix if present
if name.startswith('rocketchat_'):
name = name[11:]
# Replace special characters with underscores
name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
# Ensure it starts with a letter
if name and name[0].isdigit():
name = 'table_' + name
return name.lower()
def _sanitize_column_name(self, name: str) -> str:
"""Convert MongoDB field name to PostgreSQL column name"""
# Handle nested field names (convert dots to underscores)
name = name.replace('.', '_')
# Replace special characters with underscores
name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
# Ensure it starts with a letter or underscore
if name and name[0].isdigit():
name = 'col_' + name
# Handle PostgreSQL reserved words
reserved = {
'user', 'order', 'group', 'table', 'index', 'key', 'value', 'date', 'time', 'timestamp',
'default', 'select', 'from', 'where', 'insert', 'update', 'delete', 'create', 'drop',
'alter', 'grant', 'revoke', 'commit', 'rollback', 'begin', 'end', 'case', 'when',
'then', 'else', 'if', 'null', 'not', 'and', 'or', 'in', 'exists', 'between',
'like', 'limit', 'offset', 'union', 'join', 'inner', 'outer', 'left', 'right',
'full', 'cross', 'natural', 'on', 'using', 'distinct', 'all', 'any', 'some',
'desc', 'asc', 'primary', 'foreign', 'references', 'constraint', 'unique',
'check', 'cascade', 'restrict', 'action', 'match', 'partial', 'full'
}
if name.lower() in reserved:
name = name + '_col'
return name.lower()
def _determine_postgres_type(self, field_info: Dict[str, Any]) -> str:
"""Determine PostgreSQL column type from MongoDB field analysis with improved logic"""
types = field_info['types']
# Convert set to list for easier checking
type_list = list(types)
# If there's only one type (excluding null), use specific typing
non_null_types = [t for t in type_list if t != 'null']
if len(non_null_types) == 1:
single_type = non_null_types[0]
if single_type == 'bool':
return 'BOOLEAN'
elif single_type == 'int':
return 'INTEGER'
elif single_type == 'float':
return 'NUMERIC'
elif single_type == 'str':
return 'TEXT'
elif single_type == 'datetime':
return 'TIMESTAMP'
elif single_type == 'ObjectId':
return 'TEXT'
# Handle mixed types more conservatively
if 'array' in types or field_info.get('is_array', False):
return 'JSONB' # Arrays always go to JSONB
elif 'object' in types:
return 'JSONB' # Objects always go to JSONB
elif len(non_null_types) > 1:
# Multiple non-null types - check for common combinations
if set(non_null_types) <= {'int', 'float'}:
return 'NUMERIC' # Can handle both int and float
elif set(non_null_types) <= {'bool', 'str'}:
return 'TEXT' # Convert everything to text
elif set(non_null_types) <= {'str', 'ObjectId'}:
return 'TEXT' # Both are string-like
else:
return 'JSONB' # Complex mixed types go to JSONB
elif 'ObjectId' in types:
return 'TEXT'
elif 'datetime' in types:
return 'TIMESTAMP'
elif 'bool' in types:
return 'BOOLEAN'
elif 'int' in types:
return 'INTEGER'
elif 'float' in types:
return 'NUMERIC'
elif 'str' in types:
return 'TEXT'
else:
return 'TEXT' # Default fallback
def create_postgres_database(self, table_definitions: Dict[str, str]):
"""Create PostgreSQL database and tables"""
print("Creating PostgreSQL database schema...")
try:
# Connect to PostgreSQL
conn = psycopg2.connect(**self.postgres_config)
conn.autocommit = True
cursor = conn.cursor()
# Create tables
for collection_name, table_sql in table_definitions.items():
print(f"Creating table for {collection_name}...")
cursor.execute(table_sql)
cursor.close()
conn.close()
print("Database schema created successfully!")
except Exception as e:
print(f"Error creating database schema: {e}")
raise
def convert_and_insert_data(self, batch_size: int = 1000):
"""Convert BSON data and insert into PostgreSQL"""
print("Converting and inserting data...")
try:
conn = psycopg2.connect(**self.postgres_config)
conn.autocommit = False
for collection_name in self.collections:
print(f"Processing {collection_name}...")
self._convert_collection(conn, collection_name, batch_size)
conn.close()
print("Data conversion completed successfully!")
except Exception as e:
print(f"Error converting data: {e}")
raise
def _convert_collection(self, conn, collection_name: str, batch_size: int):
"""Convert a single collection"""
bson_file = self.collections[collection_name]['bson_file']
if bson_file.stat().st_size == 0:
print(f" Skipping empty collection {collection_name}")
return
table_name = self._sanitize_table_name(collection_name)
cursor = conn.cursor()
batch = []
total_inserted = 0
errors = 0
try:
with open(bson_file, 'rb') as f:
while True:
try:
doc_size = int.from_bytes(f.read(4), byteorder='little')
if doc_size <= 0:
break
f.seek(-4, 1)
doc_bytes = f.read(doc_size)
if len(doc_bytes) != doc_size:
break
doc = bson.decode(doc_bytes)
batch.append(doc)
if len(batch) >= batch_size:
inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
total_inserted += inserted
errors += batch_errors
batch = []
conn.commit()
if total_inserted % 5000 == 0: # Less frequent progress updates
print(f" Inserted {total_inserted} documents...")
except (bson.InvalidBSON, struct.error, OSError):
break
# Insert remaining documents
if batch:
inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
total_inserted += inserted
errors += batch_errors
conn.commit()
if errors > 0:
print(f" Completed {collection_name}: {total_inserted} documents inserted ({errors} errors)")
else:
print(f" Completed {collection_name}: {total_inserted} documents inserted")
except Exception as e:
print(f" Error processing {collection_name}: {e}")
conn.rollback()
finally:
cursor.close()
def _insert_batch(self, cursor, table_name: str, documents: List[Dict], collection_name: str):
"""Insert a batch of documents with proper transaction handling"""
if not documents:
return 0, 0
# Get schema info for this collection
schema = self.schema_info.get(collection_name, {})
# Build column list
columns = ['mongo_id']
for field_name in schema.keys():
if field_name != '_id':
col_name = self._sanitize_column_name(field_name)
# Handle conflicts with PostgreSQL auto-generated columns
if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
col_name = f"field_{col_name}"
columns.append(col_name)
# Build INSERT statement
placeholders = ', '.join(['%s'] * len(columns))
sql = f"INSERT INTO {table_name} ({', '.join(columns)}) VALUES ({placeholders})"
self.log_debug(f"SQL: {sql}", collection_name)
# Convert documents to tuples
rows = []
errors = 0
for doc_idx, doc in enumerate(documents):
try:
row = []
# Add mongo_id
row.append(str(doc.get('_id', '')))
# Add other fields
for field_name in schema.keys():
if field_name != '_id':
try:
value = self._get_nested_value(doc, field_name)
converted_value = self._convert_value_for_postgres(value, field_name, schema)
row.append(converted_value)
except Exception as e:
self.log_error(collection_name, 'field_conversion',
f"Field '{field_name}' in doc {doc_idx}: {str(e)}")
# Only show debug for collections we're focusing on
if collection_name in self.debug_collections:
print(f" ⚠️ Error converting field '{field_name}': {e}")
row.append(None) # Use NULL for problematic fields
rows.append(tuple(row))
except Exception as e:
self.log_error(collection_name, 'document_conversion', f"Document {doc_idx}: {str(e)}")
errors += 1
continue
# Execute batch insert
if rows:
try:
cursor.executemany(sql, rows)
return len(rows), errors
except Exception as batch_error:
self.log_error(collection_name, 'batch_insert', str(batch_error))
# Only show detailed debugging for targeted collections
if collection_name in self.debug_collections:
print(f" 🔴 Batch insert failed for {collection_name}: {batch_error}")
print(" Trying individual inserts with rollback handling...")
# Rollback the failed transaction
cursor.connection.rollback()
# Try inserting one by one in individual transactions
success_count = 0
for row_idx, row in enumerate(rows):
try:
cursor.execute(sql, row)
cursor.connection.commit() # Commit each successful insert
success_count += 1
except Exception as row_error:
cursor.connection.rollback() # Rollback failed insert
self.log_error(collection_name, 'row_insert', f"Row {row_idx}: {str(row_error)}")
# Show detailed error only for the first few failures and only for targeted collections
if collection_name in self.debug_collections and errors < 3:
print(f" Row {row_idx} failed: {row_error}")
print(f" Row data: {len(row)} values, expected {len(columns)} columns")
errors += 1
continue
return success_count, errors
return 0, errors
def _get_nested_value(self, doc: Dict, field_path: str):
"""Get value from nested document using dot notation"""
keys = field_path.split('.')
value = doc
for key in keys:
if isinstance(value, dict) and key in value:
value = value[key]
else:
return None
return value
def _convert_value_for_postgres(self, value, field_name: str = None, schema: Dict = None):
"""Convert MongoDB value to PostgreSQL compatible value with schema-aware conversion"""
if value is None:
return None
# Get the expected PostgreSQL type for this field if available
expected_type = None
if schema and field_name and field_name in schema:
field_info = schema[field_name]
expected_type = self._determine_postgres_type(field_info)
# Handle conversion based on expected type
if expected_type == 'BOOLEAN':
if isinstance(value, bool):
return value
elif isinstance(value, str):
return value.lower() in ('true', '1', 'yes', 'on')
elif isinstance(value, (int, float)):
return bool(value)
else:
return None
elif expected_type == 'INTEGER':
if isinstance(value, int):
return value
elif isinstance(value, float):
return int(value)
elif isinstance(value, str) and value.isdigit():
return int(value)
elif isinstance(value, bool):
return int(value)
else:
return None
elif expected_type == 'NUMERIC':
if isinstance(value, (int, float)):
return value
elif isinstance(value, str):
try:
return float(value)
except ValueError:
return None
elif isinstance(value, bool):
return float(value)
else:
return None
elif expected_type == 'TEXT':
if isinstance(value, str):
return value
elif value is not None:
str_value = str(value)
# Handle very long strings
if len(str_value) > 65535:
return str_value[:65535]
return str_value
else:
return None
elif expected_type == 'TIMESTAMP':
if hasattr(value, 'isoformat'):
return value.isoformat()
elif isinstance(value, str):
return value
else:
return str(value) if value is not None else None
elif expected_type == 'JSONB':
if isinstance(value, (dict, list)):
return json.dumps(value, default=self._json_serializer)
elif isinstance(value, str):
# Check if it's already valid JSON
try:
json.loads(value)
return value
except (json.JSONDecodeError, TypeError):
# Not valid JSON, wrap it
return json.dumps(value)
else:
return json.dumps(value, default=self._json_serializer)
# Fallback to original logic if no expected type or type not recognized
if isinstance(value, bool):
return value
elif isinstance(value, (int, float)):
return value
elif isinstance(value, str):
return value
elif isinstance(value, (dict, list)):
return json.dumps(value, default=self._json_serializer)
elif hasattr(value, 'isoformat'): # datetime
return value.isoformat()
elif hasattr(value, '__str__'):
str_value = str(value)
if len(str_value) > 65535:
return str_value[:65535]
return str_value
else:
return str(value)
def _json_serializer(self, obj):
"""Custom JSON serializer for complex objects with better error handling"""
try:
if hasattr(obj, 'isoformat'): # datetime
return obj.isoformat()
elif hasattr(obj, '__str__'):
return str(obj)
else:
return None
except Exception as e:
self.log_debug(f"JSON serialization error: {e}")
return str(obj)
def run_conversion(self, sample_size: int = 100, batch_size: int = 1000):
"""Run the full conversion process with focused debugging"""
print("Starting MongoDB to PostgreSQL conversion...")
print("This will convert your Rocket.Chat database from MongoDB to PostgreSQL")
if self.debug_mode:
if self.debug_collections:
print(f"🐛 DEBUG MODE: Focusing on collections: {', '.join(self.debug_collections)}")
else:
print("🐛 DEBUG MODE: All collections")
print("=" * 70)
# Step 1: Discover collections
self.discover_collections()
# Step 2: Analyze schemas
print("\nAnalyzing collection schemas...")
for collection_name in self.collections:
self.analyze_schema(collection_name, sample_size)
# Sample problematic collections if debugging
if self.debug_mode and self.debug_collections:
for coll in self.debug_collections:
if coll in self.collections:
self.sample_documents(coll, 2)
# Step 3: Generate PostgreSQL schema
table_definitions = self.generate_postgres_schema()
# Step 4: Create database schema
self.create_postgres_database(table_definitions)
# Step 5: Convert and insert data
self.convert_and_insert_data(batch_size)
# Step 6: Show error summary
self._print_error_summary()
print("=" * 70)
print("✅ Conversion completed!")
print(f" Database: {self.postgres_config['database']}")
print(f" Tables created: {len(table_definitions)}")
def _print_error_summary(self):
"""Print a focused summary of errors"""
if not self.error_log:
print("\n✅ No errors encountered during conversion!")
return
print("\n⚠️ ERROR SUMMARY:")
print("=" * 50)
# Sort by error count descending
sorted_collections = sorted(self.error_log.items(),
key=lambda x: len(x[1]), reverse=True)
for collection, errors in sorted_collections:
error_types = {}
for error in errors:
error_type = error['type']
if error_type not in error_types:
error_types[error_type] = []
error_types[error_type].append(error['details'])
print(f"\n🔴 {collection} ({len(errors)} total errors):")
for error_type, details_list in error_types.items():
print(f" {error_type}: {len(details_list)} errors")
# Show sample errors for critical collections
if collection in ['rocketchat_settings', 'rocketchat_room'] and len(details_list) > 0:
print(f" Sample: {details_list[0][:100]}...")
def main():
parser = argparse.ArgumentParser(
description='Convert MongoDB BSON export to PostgreSQL',
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="""
Examples:
# Basic usage
python3 mongo_to_postgres_converter.py \\
--mongo-path db/database/62df06d44234d20001289144 \\
--pg-database rocketchat_converted \\
--pg-user rocketchat_user \\
--pg-password mypassword
# Debug specific failing collections
python3 mongo_to_postgres_converter.py \\
--mongo-path db/database/62df06d44234d20001289144 \\
--pg-database rocketchat_converted \\
--pg-user rocketchat_user \\
--pg-password mypassword \\
--debug-collections rocketchat_settings rocketchat_room
Before running this script:
1. Run: sudo -u postgres psql -f reset_database.sql
2. Update the password in reset_database.sql
"""
)
parser.add_argument('--mongo-path', required=True, help='Path to MongoDB export directory')
parser.add_argument('--pg-host', default='localhost', help='PostgreSQL host (default: localhost)')
parser.add_argument('--pg-port', default='5432', help='PostgreSQL port (default: 5432)')
parser.add_argument('--pg-database', required=True, help='PostgreSQL database name')
parser.add_argument('--pg-user', required=True, help='PostgreSQL username')
parser.add_argument('--pg-password', required=True, help='PostgreSQL password')
parser.add_argument('--sample-size', type=int, default=100, help='Number of documents to sample for schema analysis (default: 100)')
parser.add_argument('--batch-size', type=int, default=1000, help='Batch size for data insertion (default: 1000)')
parser.add_argument('--debug', action='store_true', help='Enable debug mode with detailed error logging')
parser.add_argument('--debug-collections', nargs='*', help='Specific collections to debug (e.g., rocketchat_settings rocketchat_room)')
args = parser.parse_args()
postgres_config = {
'host': args.pg_host,
'port': args.pg_port,
'database': args.pg_database,
'user': args.pg_user,
'password': args.pg_password
}
# Enable debug mode if debug collections are specified
debug_mode = args.debug or (args.debug_collections is not None)
converter = MongoToPostgresConverter(args.mongo_path, postgres_config, debug_mode, args.debug_collections)
converter.run_conversion(args.sample_size, args.batch_size)
if __name__ == '__main__':
main()
inventory-server/chat/db-convert/reset_database.sql
-41
View File
@@ -1,41 +0,0 @@
-- PostgreSQL Database Reset Script for Rocket.Chat Import
-- Run as: sudo -u postgres psql -f reset_database.sql
-- Terminate all connections to the database (force disconnect users)
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'rocketchat_converted' AND pid <> pg_backend_pid();
-- Drop the database if it exists
DROP DATABASE IF EXISTS rocketchat_converted;
-- Create fresh database
CREATE DATABASE rocketchat_converted;
-- Create user (if not exists)
DO $$
BEGIN
IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'rocketchat_user') THEN
CREATE USER rocketchat_user WITH PASSWORD 'HKjLgt23gWuPXzEAn3rW';
END IF;
END $$;
-- Grant database privileges
GRANT CONNECT ON DATABASE rocketchat_converted TO rocketchat_user;
GRANT CREATE ON DATABASE rocketchat_converted TO rocketchat_user;
-- Connect to the new database
\c rocketchat_converted;
-- Grant schema privileges
GRANT CREATE ON SCHEMA public TO rocketchat_user;
GRANT USAGE ON SCHEMA public TO rocketchat_user;
-- Grant privileges on all future tables and sequences
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO rocketchat_user;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO rocketchat_user;
-- Display success message
\echo 'Database reset completed successfully!'
\echo 'You can now run the converter with:'
\echo 'python3 mongo_to_postgres_converter.py --mongo-path db/database/62df06d44234d20001289144 --pg-database rocketchat_converted --pg-user rocketchat_user --pg-password your_password'
inventory-server/chat/db-convert/test_converter.py
-54
View File
@@ -1,54 +0,0 @@
#!/usr/bin/env python3
"""
Quick test script to verify the converter fixes work for problematic collections
"""
from mongo_to_postgres_converter import MongoToPostgresConverter
def test_problematic_collections():
print("🧪 Testing converter fixes for problematic collections...")
postgres_config = {
'host': 'localhost',
'port': '5432',
'database': 'rocketchat_test',
'user': 'rocketchat_user',
'password': 'password123'
}
converter = MongoToPostgresConverter(
'db/database/62df06d44234d20001289144',
postgres_config,
debug_mode=True,
debug_collections=['rocketchat_settings', 'rocketchat_room']
)
# Test just discovery and schema analysis
print("\n1. Testing collection discovery...")
converter.discover_collections()
print("\n2. Testing schema analysis...")
if 'rocketchat_settings' in converter.collections:
settings_schema = converter.analyze_schema('rocketchat_settings', 10)
print(f"Settings schema fields: {len(settings_schema)}")
# Check specific problematic fields
if 'packageValue' in settings_schema:
packagevalue_info = settings_schema['packageValue']
pg_type = converter._determine_postgres_type(packagevalue_info)
print(f"packageValue types: {packagevalue_info['types']} -> PostgreSQL: {pg_type}")
if 'rocketchat_room' in converter.collections:
room_schema = converter.analyze_schema('rocketchat_room', 10)
print(f"Room schema fields: {len(room_schema)}")
# Check specific problematic fields
if 'sysMes' in room_schema:
sysmes_info = room_schema['sysMes']
pg_type = converter._determine_postgres_type(sysmes_info)
print(f"sysMes types: {sysmes_info['types']} -> PostgreSQL: {pg_type}")
print("\n✅ Test completed - check the type mappings above!")
if __name__ == '__main__':
test_problematic_collections()
inventory-server/chat/export-chat-data.sh
-147
View File
@@ -1,147 +0,0 @@
#!/bin/bash
# Chat Database Export Script
# This script exports the chat database schema and data for migration
set -e # Exit on any error
echo "🚀 Starting chat database export..."
# Configuration - Update these values for your setup
DB_HOST="${CHAT_DB_HOST:-localhost}"
DB_PORT="${CHAT_DB_PORT:-5432}"
DB_NAME="${CHAT_DB_NAME:-rocketchat_converted}"
DB_USER="${CHAT_DB_USER:-rocketchat_user}"
# Check if database connection info is available
if [ -z "$CHAT_DB_PASSWORD" ]; then
echo "⚠️ CHAT_DB_PASSWORD environment variable not set"
echo "Please set it with: export CHAT_DB_PASSWORD='your_password'"
exit 1
fi
echo "📊 Database: $DB_NAME on $DB_HOST:$DB_PORT"
# Create export directory
EXPORT_DIR="chat-migration-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$EXPORT_DIR"
echo "📁 Export directory: $EXPORT_DIR"
# Export database schema
echo "📋 Exporting database schema..."
PGPASSWORD="$CHAT_DB_PASSWORD" pg_dump \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
--schema-only \
--no-owner \
--no-privileges \
-f "$EXPORT_DIR/chat-schema.sql"
if [ $? -eq 0 ]; then
echo "✅ Schema exported successfully"
else
echo "❌ Schema export failed"
exit 1
fi
# Export database data
echo "💾 Exporting database data..."
PGPASSWORD="$CHAT_DB_PASSWORD" pg_dump \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
--data-only \
--no-owner \
--no-privileges \
--disable-triggers \
--column-inserts \
-f "$EXPORT_DIR/chat-data.sql"
if [ $? -eq 0 ]; then
echo "✅ Data exported successfully"
else
echo "❌ Data export failed"
exit 1
fi
# Export file uploads and avatars
echo "📎 Exporting chat files (uploads and avatars)..."
if [ -d "db-convert/db/files" ]; then
cd db-convert/db
tar -czf "../../$EXPORT_DIR/chat-files.tar.gz" files/
cd ../..
echo "✅ Files exported successfully"
else
echo "⚠️ No files directory found at db-convert/db/files"
echo " This is normal if you have no file uploads"
touch "$EXPORT_DIR/chat-files.tar.gz"
fi
# Get table statistics for verification
echo "📈 Generating export statistics..."
PGPASSWORD="$CHAT_DB_PASSWORD" psql \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
-c "
SELECT
schemaname,
tablename,
n_tup_ins as inserted_rows,
n_tup_upd as updated_rows,
n_tup_del as deleted_rows,
n_live_tup as live_rows,
n_dead_tup as dead_rows
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;
" > "$EXPORT_DIR/table-stats.txt"
# Create export summary
cat > "$EXPORT_DIR/export-summary.txt" << EOF
Chat Database Export Summary
===========================
Export Date: $(date)
Database: $DB_NAME
Host: $DB_HOST:$DB_PORT
User: $DB_USER
Files Generated:
- chat-schema.sql: Database schema (tables, indexes, constraints)
- chat-data.sql: All table data
- chat-files.tar.gz: Uploaded files and avatars
- table-stats.txt: Database statistics
- export-summary.txt: This summary
Next Steps:
1. Transfer these files to your new server
2. Run create-new-database.sql on the new server first
3. Run import-chat-data.sh on the new server
4. Update your application configuration
5. Run verify-migration.js to validate the migration
Important Notes:
- Keep these files secure as they contain your chat data
- Ensure the new server has enough disk space
- Plan for application downtime during the migration
EOF
echo ""
echo "🎉 Export completed successfully!"
echo "📁 Files are in: $EXPORT_DIR/"
echo ""
echo "📋 Export Summary:"
ls -lh "$EXPORT_DIR/"
echo ""
echo "🚚 Next steps:"
echo "1. Transfer the $EXPORT_DIR/ directory to your new server"
echo "2. Run create-new-database.sql on the new server (update password first!)"
echo "3. Run import-chat-data.sh on the new server"
echo ""
echo "💡 To transfer files to new server:"
echo " scp -r $EXPORT_DIR/ user@new-server:/tmp/"
inventory-server/chat/import-chat-data.sh
-167
View File
@@ -1,167 +0,0 @@
#!/bin/bash
# Chat Database Import Script
# This script imports the chat database schema and data on the new server
set -e # Exit on any error
echo "🚀 Starting chat database import..."
# Configuration - Update these values for your new server
DB_HOST="${CHAT_DB_HOST:-localhost}"
DB_PORT="${CHAT_DB_PORT:-5432}"
DB_NAME="${CHAT_DB_NAME:-rocketchat_converted}"
DB_USER="${CHAT_DB_USER:-rocketchat_user}"
# Check if database connection info is available
if [ -z "$CHAT_DB_PASSWORD" ]; then
echo "⚠️ CHAT_DB_PASSWORD environment variable not set"
echo "Please set it with: export CHAT_DB_PASSWORD='your_password'"
exit 1
fi
# Find the migration directory
MIGRATION_DIR=""
if [ -d "/tmp" ]; then
MIGRATION_DIR=$(find /tmp -maxdepth 1 -name "chat-migration-*" -type d | head -1)
fi
if [ -z "$MIGRATION_DIR" ]; then
echo "❌ No migration directory found in /tmp/"
echo "Please specify the migration directory:"
read -p "Enter full path to migration directory: " MIGRATION_DIR
fi
if [ ! -d "$MIGRATION_DIR" ]; then
echo "❌ Migration directory not found: $MIGRATION_DIR"
exit 1
fi
echo "📁 Using migration directory: $MIGRATION_DIR"
echo "📊 Target database: $DB_NAME on $DB_HOST:$DB_PORT"
# Verify required files exist
REQUIRED_FILES=("chat-schema.sql" "chat-data.sql" "chat-files.tar.gz")
for file in "${REQUIRED_FILES[@]}"; do
if [ ! -f "$MIGRATION_DIR/$file" ]; then
echo "❌ Required file not found: $MIGRATION_DIR/$file"
exit 1
fi
done
echo "✅ All required files found"
# Test database connection
echo "🔗 Testing database connection..."
PGPASSWORD="$CHAT_DB_PASSWORD" psql \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
-c "SELECT version();" > /dev/null
if [ $? -eq 0 ]; then
echo "✅ Database connection successful"
else
echo "❌ Database connection failed"
echo "Please ensure:"
echo " 1. PostgreSQL is running"
echo " 2. Database '$DB_NAME' exists"
echo " 3. User '$DB_USER' has access"
echo " 4. Password is correct"
exit 1
fi
# Import database schema
echo "📋 Importing database schema..."
PGPASSWORD="$CHAT_DB_PASSWORD" psql \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
-f "$MIGRATION_DIR/chat-schema.sql"
if [ $? -eq 0 ]; then
echo "✅ Schema imported successfully"
else
echo "❌ Schema import failed"
exit 1
fi
# Import database data
echo "💾 Importing database data..."
echo " This may take a while depending on data size..."
PGPASSWORD="$CHAT_DB_PASSWORD" psql \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
-f "$MIGRATION_DIR/chat-data.sql"
if [ $? -eq 0 ]; then
echo "✅ Data imported successfully"
else
echo "❌ Data import failed"
echo "Check the error messages above for details"
exit 1
fi
# Create files directory and import files
echo "📎 Setting up files directory..."
mkdir -p "db-convert/db"
if [ -s "$MIGRATION_DIR/chat-files.tar.gz" ]; then
echo "📂 Extracting chat files..."
cd db-convert/db
tar -xzf "$MIGRATION_DIR/chat-files.tar.gz"
cd ../..
# Set proper permissions
if [ -d "db-convert/db/files" ]; then
chmod -R 755 db-convert/db/files
echo "✅ Files imported and permissions set"
else
echo "⚠️ Files directory not created properly"
fi
else
echo "ℹ️ No files to import (empty archive)"
mkdir -p "db-convert/db/files/uploads"
mkdir -p "db-convert/db/files/avatars"
fi
# Get final table statistics
echo "📈 Generating import statistics..."
PGPASSWORD="$CHAT_DB_PASSWORD" psql \
-h "$DB_HOST" \
-p "$DB_PORT" \
-U "$DB_USER" \
-d "$DB_NAME" \
-c "
SELECT
tablename,
n_live_tup as row_count
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY n_live_tup DESC;
"
# Create import summary
echo ""
echo "🎉 Import completed successfully!"
echo ""
echo "📋 Import Summary:"
echo " Database: $DB_NAME"
echo " Host: $DB_HOST:$DB_PORT"
echo " Files location: $(pwd)/db-convert/db/files/"
echo ""
echo "🔍 Next steps:"
echo "1. Update your application configuration to use this database"
echo "2. Run verify-migration.js to validate the migration"
echo "3. Test your application thoroughly"
echo "4. Update DNS/load balancer to point to new server"
echo ""
echo "⚠️ Important:"
echo "- Keep the original data as backup until migration is fully validated"
echo "- Monitor the application closely after switching"
echo "- Have a rollback plan ready"
inventory-server/chat/migrate-to-new-server.md
-86
View File
@@ -1,86 +0,0 @@
# Chat Database Migration Guide
This guide will help you migrate your chat database from the current server to a new PostgreSQL server.
## Overview
Your chat system uses:
- Database: `rocketchat_converted` (PostgreSQL)
- Main tables: users, message, room, uploads, avatars, subscription
- File storage: db-convert/db/files/ directory with uploads and avatars
- Environment configuration for database connection
## Migration Steps
### 1. Pre-Migration Setup
On your **new server**, ensure PostgreSQL is installed and running:
```bash
# Install PostgreSQL (if not already done)
sudo apt update
sudo apt install postgresql postgresql-contrib
# Start PostgreSQL service
sudo systemctl start postgresql
sudo systemctl enable postgresql
```
### 2. Create Database Schema on New Server
Run the provided migration script:
```bash
# On new server
sudo -u postgres psql -f create-new-database.sql
```
### 3. Export Data from Current Server
Run the export script:
```bash
# On current server
./export-chat-data.sh
```
This will create:
- `chat-schema.sql` - Database schema
- `chat-data.sql` - All table data
- `chat-files.tar.gz` - All uploaded files and avatars
### 4. Transfer Data to New Server
```bash
# Copy files to new server
scp chat-schema.sql chat-data.sql chat-files.tar.gz user@new-server:/tmp/
```
### 5. Import Data on New Server
```bash
# On new server
./import-chat-data.sh
```
### 6. Update Configuration
Update your environment variables to point to the new database server.
### 7. Verify Migration
Run the verification script to ensure everything transferred correctly:
```bash
node verify-migration.js
```
## Files Provided
1. `create-new-database.sql` - Creates database and user on new server
2. `export-chat-data.sh` - Exports data from current server
3. `import-chat-data.sh` - Imports data to new server
4. `verify-migration.js` - Verifies data integrity
5. `update-config-template.env` - Template for new configuration
## Important Notes
- **Backup first**: Always backup your current database before migration
- **Downtime**: Plan for application downtime during migration
- **File permissions**: Ensure file permissions are preserved during transfer
- **Network access**: Ensure new server can accept connections from your application
inventory-server/chat/package-lock.json
Generated
-1446
View File
File diff suppressed because it is too large Load Diff
inventory-server/chat/package.json
-24
View File
@@ -1,24 +0,0 @@
{
"name": "chat-server",
"version": "1.0.0",
"description": "Chat archive server for Rocket.Chat data",
"type": "module",
"main": "server.js",
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js"
},
"dependencies": {
"express": "^4.18.2",
"cors": "^2.8.5",
"pg": "^8.11.0",
"dotenv": "^16.0.3",
"morgan": "^1.10.0",
"jsonwebtoken": "^9.0.2",
"pino": "^9.5.0",
"pino-http": "^10.3.0"
},
"devDependencies": {
"nodemon": "^2.0.22"
}
}
inventory-server/chat/routes.js
-656
View File
@@ -1,656 +0,0 @@
import express from 'express';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
// ESM polyfill — Phase 9 §9.1. Handlers below use __dirname to resolve the
// db-convert/db/files/{uploads,avatars} static asset paths.
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const router = express.Router();
// Serve uploaded files with proper mapping from database paths to actual file locations
router.get('/files/uploads/*', async (req, res) => {
try {
// Extract the path from the URL (everything after /files/uploads/)
const requestPath = req.params[0];
// The URL path will be like: ufs/AmazonS3:Uploads/274Mf9CyHNG72oF86/filename.jpg
// We need to extract the mongo_id (274Mf9CyHNG72oF86) from this path
const pathParts = requestPath.split('/');
let mongoId = null;
// Find the mongo_id in the path structure
for (let i = 0; i < pathParts.length; i++) {
if (pathParts[i].includes('AmazonS3:Uploads') && i + 1 < pathParts.length) {
mongoId = pathParts[i + 1];
break;
}
// Sometimes the mongo_id might be the last part of ufs/AmazonS3:Uploads/mongoId
if (pathParts[i] === 'AmazonS3:Uploads' && i + 1 < pathParts.length) {
mongoId = pathParts[i + 1];
break;
}
}
if (!mongoId) {
// Try to get mongo_id from database by matching the full path
const result = await global.pool.query(`
SELECT mongo_id, name, type
FROM uploads
WHERE path = $1 OR url = $1
LIMIT 1
`, [`/ufs/AmazonS3:Uploads/${requestPath}`, `/ufs/AmazonS3:Uploads/${requestPath}`]);
if (result.rows.length > 0) {
mongoId = result.rows[0].mongo_id;
}
}
if (!mongoId) {
return res.status(404).json({ error: 'File not found' });
}
// The actual file is stored with just the mongo_id as filename
const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
// Get file info from database for proper content-type
const fileInfo = await global.pool.query(`
SELECT name, type
FROM uploads
WHERE mongo_id = $1
LIMIT 1
`, [mongoId]);
if (fileInfo.rows.length === 0) {
return res.status(404).json({ error: 'File metadata not found' });
}
const { name, type } = fileInfo.rows[0];
// Set proper content type
if (type) {
res.set('Content-Type', type);
}
// Set content disposition with original filename
if (name) {
res.set('Content-Disposition', `inline; filename="${name}"`);
}
// Send the file
res.sendFile(filePath, (err) => {
if (err) {
console.error('Error serving file:', err);
if (!res.headersSent) {
res.status(404).json({ error: 'File not found on disk' });
}
}
});
} catch (error) {
console.error('Error serving upload:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Also serve files directly by mongo_id for simpler access
router.get('/files/by-id/:mongoId', async (req, res) => {
try {
const { mongoId } = req.params;
// Get file info from database
const fileInfo = await global.pool.query(`
SELECT name, type
FROM uploads
WHERE mongo_id = $1
LIMIT 1
`, [mongoId]);
if (fileInfo.rows.length === 0) {
return res.status(404).json({ error: 'File not found' });
}
const { name, type } = fileInfo.rows[0];
const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
// Set proper content type and filename
if (type) {
res.set('Content-Type', type);
}
if (name) {
res.set('Content-Disposition', `inline; filename="${name}"`);
}
// Send the file
res.sendFile(filePath, (err) => {
if (err) {
console.error('Error serving file:', err);
if (!res.headersSent) {
res.status(404).json({ error: 'File not found on disk' });
}
}
});
} catch (error) {
console.error('Error serving upload by ID:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Serve user avatars by mongo_id
router.get('/avatar/:mongoId', async (req, res) => {
try {
const { mongoId } = req.params;
console.log(`[Avatar Debug] Looking up avatar for user mongo_id: ${mongoId}`);
// First try to find avatar by user's avataretag
const userResult = await global.pool.query(`
SELECT avataretag, username FROM users WHERE mongo_id = $1
`, [mongoId]);
let avatarPath = null;
if (userResult.rows.length > 0) {
const username = userResult.rows[0].username;
const avataretag = userResult.rows[0].avataretag;
// Try method 1: Look up by avataretag -> etag (for users with avataretag set)
if (avataretag) {
console.log(`[Avatar Debug] Found user ${username} with avataretag: ${avataretag}`);
const avatarResult = await global.pool.query(`
SELECT url, path FROM avatars WHERE etag = $1
`, [avataretag]);
if (avatarResult.rows.length > 0) {
const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
console.log(`[Avatar Debug] Found avatar record with path: ${dbPath}`);
if (dbPath) {
const pathParts = dbPath.split('/');
for (let i = 0; i < pathParts.length; i++) {
if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
const avatarMongoId = pathParts[i + 1];
avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
break;
}
}
}
} else {
console.log(`[Avatar Debug] No avatar record found for etag: ${avataretag}`);
}
}
// Try method 2: Look up by userid directly (for users without avataretag)
if (!avatarPath) {
console.log(`[Avatar Debug] Trying direct userid lookup for user ${username} (${mongoId})`);
const avatarResult = await global.pool.query(`
SELECT url, path FROM avatars WHERE userid = $1
`, [mongoId]);
if (avatarResult.rows.length > 0) {
const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
console.log(`[Avatar Debug] Found avatar record by userid with path: ${dbPath}`);
if (dbPath) {
const pathParts = dbPath.split('/');
for (let i = 0; i < pathParts.length; i++) {
if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
const avatarMongoId = pathParts[i + 1];
avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
break;
}
}
}
} else {
console.log(`[Avatar Debug] No avatar record found for userid: ${mongoId}`);
}
}
} else {
console.log(`[Avatar Debug] No user found for mongo_id: ${mongoId}`);
}
// Fallback: try direct lookup by user mongo_id
if (!avatarPath) {
avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', mongoId);
console.log(`[Avatar Debug] Using fallback path: ${avatarPath}`);
}
// Set proper content type for images
res.set('Content-Type', 'image/jpeg'); // Most avatars are likely JPEG
// Send the file
res.sendFile(avatarPath, (err) => {
if (err) {
// If avatar doesn't exist, send a default 404 or generate initials
console.log(`[Avatar Debug] Avatar file not found at path: ${avatarPath}, error:`, err.message);
if (!res.headersSent) {
res.status(404).json({ error: 'Avatar not found' });
}
} else {
console.log(`[Avatar Debug] Successfully served avatar from: ${avatarPath}`);
}
});
} catch (error) {
console.error('Error serving avatar:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Serve avatars statically as fallback
router.use('/files/avatars', express.static(path.join(__dirname, 'db-convert/db/files/avatars')));
// Get all users for the "view as" dropdown (active and inactive)
router.get('/users', async (req, res) => {
try {
const result = await global.pool.query(`
SELECT id, username, name, type, active, status, lastlogin,
statustext, utcoffset, statusconnection, mongo_id, avataretag
FROM users
WHERE type = 'user'
ORDER BY
active DESC, -- Active users first
CASE
WHEN status = 'online' THEN 1
WHEN status = 'away' THEN 2
WHEN status = 'busy' THEN 3
ELSE 4
END,
name ASC
`);
res.json({
status: 'success',
users: result.rows
});
} catch (error) {
console.error('Error fetching users:', error);
res.status(500).json({
status: 'error',
error: 'Failed to fetch users',
details: error.message
});
}
});
// Get rooms for a specific user with enhanced room names for direct messages
router.get('/users/:userId/rooms', async (req, res) => {
const { userId } = req.params;
try {
// Get the current user's mongo_id for filtering
const userResult = await global.pool.query(`
SELECT mongo_id, username FROM users WHERE id = $1
`, [userId]);
if (userResult.rows.length === 0) {
return res.status(404).json({
status: 'error',
error: 'User not found'
});
}
const currentUserMongoId = userResult.rows[0].mongo_id;
const currentUsername = userResult.rows[0].username;
// Get rooms where the user is a member with proper naming from subscription table
// Include archived and closed rooms but sort them at the bottom
const result = await global.pool.query(`
SELECT DISTINCT
r.id,
r.mongo_id as room_mongo_id,
r.name,
r.fname,
r.t as type,
r.msgs,
r.lm as last_message_date,
r.usernames,
r.uids,
r.userscount,
r.description,
r.teamid,
r.archived,
s.open,
-- Use the subscription's name for direct messages (excludes current user)
-- For channels/groups, use room's fname or name
CASE
WHEN r.t = 'd' THEN COALESCE(s.fname, s.name, 'Unknown User')
ELSE COALESCE(r.fname, r.name, 'Unnamed Room')
END as display_name
FROM room r
JOIN subscription s ON s.rid = r.mongo_id
WHERE s.u->>'_id' = $1
ORDER BY
s.open DESC NULLS LAST, -- Open rooms first
r.archived NULLS FIRST, -- Non-archived first (nulls treated as false)
r.lm DESC NULLS LAST
LIMIT 50
`, [currentUserMongoId]);
// Enhance rooms with participant information for direct messages
const enhancedRooms = await Promise.all(result.rows.map(async (room) => {
if (room.type === 'd' && room.uids) {
// Get participant info (excluding current user) for direct messages
const participantResult = await global.pool.query(`
SELECT u.username, u.name, u.mongo_id, u.avataretag
FROM users u
WHERE u.mongo_id = ANY($1::text[])
AND u.mongo_id != $2
`, [room.uids, currentUserMongoId]);
room.participants = participantResult.rows;
}
return room;
}));
res.json({
status: 'success',
rooms: enhancedRooms
});
} catch (error) {
console.error('Error fetching user rooms:', error);
res.status(500).json({
status: 'error',
error: 'Failed to fetch user rooms',
details: error.message
});
}
});
// Get room details including participants
router.get('/rooms/:roomId', async (req, res) => {
const { roomId } = req.params;
const { userId } = req.query; // Accept current user ID as query parameter
try {
const result = await global.pool.query(`
SELECT r.id, r.name, r.fname, r.t as type, r.msgs, r.description,
r.lm as last_message_date, r.usernames, r.uids, r.userscount, r.teamid
FROM room r
WHERE r.id = $1
`, [roomId]);
if (result.rows.length === 0) {
return res.status(404).json({
status: 'error',
error: 'Room not found'
});
}
const room = result.rows[0];
// For direct messages, get the proper display name based on current user
if (room.type === 'd' && room.uids && userId) {
// Get current user's mongo_id
const userResult = await global.pool.query(`
SELECT mongo_id FROM users WHERE id = $1
`, [userId]);
if (userResult.rows.length > 0) {
const currentUserMongoId = userResult.rows[0].mongo_id;
// Get display name from subscription table for this user
// Use room mongo_id to match with subscription.rid
const roomMongoResult = await global.pool.query(`
SELECT mongo_id FROM room WHERE id = $1
`, [roomId]);
if (roomMongoResult.rows.length > 0) {
const roomMongoId = roomMongoResult.rows[0].mongo_id;
const subscriptionResult = await global.pool.query(`
SELECT fname, name FROM subscription
WHERE rid = $1 AND u->>'_id' = $2
`, [roomMongoId, currentUserMongoId]);
if (subscriptionResult.rows.length > 0) {
const sub = subscriptionResult.rows[0];
room.display_name = sub.fname || sub.name || 'Unknown User';
}
}
}
// Get all participants for additional info
const participantResult = await global.pool.query(`
SELECT username, name
FROM users
WHERE mongo_id = ANY($1::text[])
`, [room.uids]);
room.participants = participantResult.rows;
} else {
// For channels/groups, use room's fname or name
room.display_name = room.fname || room.name || 'Unnamed Room';
}
res.json({
status: 'success',
room: room
});
} catch (error) {
console.error('Error fetching room details:', error);
res.status(500).json({
status: 'error',
error: 'Failed to fetch room details',
details: error.message
});
}
});
// Get messages for a specific room (fast, without attachments)
router.get('/rooms/:roomId/messages', async (req, res) => {
const { roomId } = req.params;
const { limit = 50, offset = 0, before } = req.query;
try {
// Fast query - just get messages without expensive attachment joins
let query = `
SELECT m.id, m.msg, m.ts, m.u, m._updatedat, m.urls, m.mentions, m.md
FROM message m
JOIN room r ON m.rid = r.mongo_id
WHERE r.id = $1
`;
const params = [roomId];
if (before) {
query += ` AND m.ts < $${params.length + 1}`;
params.push(before);
}
query += ` ORDER BY m.ts DESC LIMIT $${params.length + 1} OFFSET $${params.length + 2}`;
params.push(limit, offset);
const result = await global.pool.query(query, params);
// Add empty attachments array for now - attachments will be loaded separately if needed
const messages = result.rows.map(msg => ({
...msg,
attachments: []
}));
res.json({
status: 'success',
messages: messages.reverse() // Reverse to show oldest first
});
} catch (error) {
console.error('Error fetching messages:', error);
res.status(500).json({
status: 'error',
error: 'Failed to fetch messages',
details: error.message
});
}
});
// Get attachments for specific messages (called separately for performance)
router.post('/messages/attachments', async (req, res) => {
const { messageIds } = req.body;
if (!messageIds || !Array.isArray(messageIds) || messageIds.length === 0) {
return res.json({ status: 'success', attachments: {} });
}
try {
// Get room mongo_id from first message to limit search scope
const roomQuery = await global.pool.query(`
SELECT r.mongo_id as room_mongo_id
FROM message m
JOIN room r ON m.rid = r.mongo_id
WHERE m.id = $1
LIMIT 1
`, [messageIds[0]]);
if (roomQuery.rows.length === 0) {
return res.json({ status: 'success', attachments: {} });
}
const roomMongoId = roomQuery.rows[0].room_mongo_id;
// Get messages and their upload timestamps
const messagesQuery = await global.pool.query(`
SELECT m.id, m.ts, m.u->>'_id' as user_id
FROM message m
WHERE m.id = ANY($1::int[])
`, [messageIds]);
if (messagesQuery.rows.length === 0) {
return res.json({ status: 'success', attachments: {} });
}
// Build a map of user_id -> array of message timestamps for efficient lookup
const userTimeMap = {};
const messageMap = {};
messagesQuery.rows.forEach(msg => {
if (!userTimeMap[msg.user_id]) {
userTimeMap[msg.user_id] = [];
}
userTimeMap[msg.user_id].push(msg.ts);
messageMap[msg.id] = { ts: msg.ts, user_id: msg.user_id };
});
// Get attachments for this room and these users
const uploadsQuery = await global.pool.query(`
SELECT mongo_id, name, size, type, url, path, typegroup, identify,
userid, uploadedat
FROM uploads
WHERE rid = $1
AND userid = ANY($2::text[])
ORDER BY uploadedat
`, [roomMongoId, Object.keys(userTimeMap)]);
// Match attachments to messages based on timestamp proximity (within 5 minutes)
const attachmentsByMessage = {};
uploadsQuery.rows.forEach(upload => {
const uploadTime = new Date(upload.uploadedat).getTime();
// Find the closest message from this user within 5 minutes
let closestMessageId = null;
let closestTimeDiff = Infinity;
Object.entries(messageMap).forEach(([msgId, msgData]) => {
if (msgData.user_id === upload.userid) {
const msgTime = new Date(msgData.ts).getTime();
const timeDiff = Math.abs(uploadTime - msgTime);
if (timeDiff < 300000 && timeDiff < closestTimeDiff) { // 5 minutes = 300000ms
closestMessageId = msgId;
closestTimeDiff = timeDiff;
}
}
});
if (closestMessageId) {
if (!attachmentsByMessage[closestMessageId]) {
attachmentsByMessage[closestMessageId] = [];
}
attachmentsByMessage[closestMessageId].push({
id: upload.id,
mongo_id: upload.mongo_id,
name: upload.name,
size: upload.size,
type: upload.type,
url: upload.url,
path: upload.path,
typegroup: upload.typegroup,
identify: upload.identify
});
}
});
res.json({
status: 'success',
attachments: attachmentsByMessage
});
} catch (error) {
console.error('Error fetching message attachments:', error);
res.status(500).json({
status: 'error',
error: 'Failed to fetch attachments',
details: error.message
});
}
});
// Search messages in accessible rooms for a user
router.get('/users/:userId/search', async (req, res) => {
const { userId } = req.params;
const { q, limit = 20 } = req.query;
if (!q || q.length < 2) {
return res.status(400).json({
status: 'error',
error: 'Search query must be at least 2 characters'
});
}
try {
const userResult = await global.pool.query(`
SELECT mongo_id FROM users WHERE id = $1
`, [userId]);
if (userResult.rows.length === 0) {
return res.status(404).json({
status: 'error',
error: 'User not found'
});
}
const currentUserMongoId = userResult.rows[0].mongo_id;
const result = await global.pool.query(`
SELECT m.id, m.msg, m.ts, m.u, r.id as room_id, r.name as room_name, r.fname as room_fname, r.t as room_type
FROM message m
JOIN room r ON m.rid = r.mongo_id
JOIN subscription s ON s.rid = r.mongo_id AND s.u->>'_id' = $1
WHERE m.msg ILIKE $2
AND r.archived IS NOT TRUE
ORDER BY m.ts DESC
LIMIT $3
`, [currentUserMongoId, `%${q}%`, limit]);
res.json({
status: 'success',
results: result.rows
});
} catch (error) {
console.error('Error searching messages:', error);
res.status(500).json({
status: 'error',
error: 'Failed to search messages',
details: error.message
});
}
});
export default router;
inventory-server/chat/server.js
+47 -96
View File
@@ -1,62 +1,23 @@
// chat-server — Phase 9 §9.1 of CONSOLIDATION_PLAN.md.
//
// ESM conversion + in-process authenticate() defense-in-depth. Previously this
// service relied on the Caddy `forward_auth` gate alone — `localhost:3014`
// was reachable unauthenticated. Now:
// 1. Bound to 127.0.0.1 (was 0.0.0.0) so direct-port access is impossible.
// 2. authenticate() runs against an in-process `inventory_db` pool before
// any route handler sees the request.
//
// Two pools intentionally:
// - `inventoryPool`: used by authenticate() for users/permissions lookups
// against the main inventory_db (matches DB_* env vars).
// - `pool` (set as global.pool for routes.js): the existing
// `rocketchat_converted` pool driven by CHAT_DB_* env vars. routes.js
// reads global.pool throughout — no handler-body changes needed.
import { config as loadEnv } from 'dotenv';
import express from 'express';
import cors from 'cors';
import morgan from 'morgan';
import pg from 'pg';
import path from 'node:path';
import fs from 'node:fs';
import { fileURLToPath } from 'node:url';
import { authenticate } from '../shared/auth/middleware.js';
import { corsOptions } from '../shared/cors/policy.js';
import { errorHandler } from '../shared/errors/handler.js';
import { requestLog } from '../shared/logging/request-log.js';
import chatRoutes from './routes.js';
const { Pool } = pg;
const __dirname = path.dirname(fileURLToPath(import.meta.url));
// Env layering matches dashboard-server (Deviation #18): shared .env wins on
// collisions for security-critical vars, local .env supplies CHAT_DB_*.
const sharedEnvPath = '/var/www/inventory/.env';
const localEnvPath = path.resolve(__dirname, '.env');
if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
if (fs.existsSync(localEnvPath)) loadEnv({ path: localEnvPath });
if (!process.env.JWT_SECRET) {
console.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
process.exit(1);
}
const app = express();
const port = Number(process.env.CHAT_PORT) || 3014;
require('dotenv').config({ path: '../.env' });
const express = require('express');
const cors = require('cors');
const { Pool } = require('pg');
const morgan = require('morgan');
const chatRoutes = require('./routes');
// Log startup configuration
console.log('Starting chat server with config:', {
host: process.env.CHAT_DB_HOST,
user: process.env.CHAT_DB_USER,
database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
port: process.env.CHAT_DB_PORT,
chat_port: port,
chat_port: process.env.CHAT_PORT || 3014
});
// Rocket.Chat archive pool — routes.js reads it via global.pool.
const app = express();
const port = process.env.CHAT_PORT || 3014;
// Database configuration for rocketchat_converted database
const pool = new Pool({
host: process.env.CHAT_DB_HOST,
user: process.env.CHAT_DB_USER,
@@ -64,69 +25,59 @@ const pool = new Pool({
database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
port: process.env.CHAT_DB_PORT,
});
// Make pool available globally
global.pool = pool;
// inventory_db pool — used by authenticate() for user/permission lookups.
const inventoryPool = new Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: Number(process.env.DB_PORT) || 5432,
ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
});
app.use(requestLog());
// Middleware
app.use(express.json());
app.use(morgan('combined'));
app.use(cors(corsOptions));
app.use(cors({
origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw', 'https://tools.acherryontop.com', 'https://tools.acherryontop.com'],
credentials: true
}));
// /health stays unauthenticated for out-of-band probes — mounted BEFORE
// authenticate() so monitoring tools on the host can poll without a JWT.
// Only reachable via localhost:3014 directly (Caddy routes /health to
// inventory-server:3010, not here).
app.get('/health', (req, res) => res.json({ status: 'healthy' }));
// Phase 9 §9.1 — per-server auth re-verification. Every chat route must pass
// authenticate() in addition to the Caddy forward_auth gate.
app.use(authenticate({ pool: inventoryPool, secret: process.env.JWT_SECRET }));
app.get('/test-db', async (req, res, next) => {
// Test database connection endpoint
app.get('/test-db', async (req, res) => {
try {
const result = await pool.query('SELECT COUNT(*) as user_count FROM users WHERE active = true');
const messageResult = await pool.query('SELECT COUNT(*) as message_count FROM message');
const roomResult = await pool.query('SELECT COUNT(*) as room_count FROM room');
res.json({
status: 'success',
database: 'rocketchat_converted',
stats: {
active_users: parseInt(result.rows[0].user_count, 10),
total_messages: parseInt(messageResult.rows[0].message_count, 10),
total_rooms: parseInt(roomResult.rows[0].room_count, 10),
},
active_users: parseInt(result.rows[0].user_count),
total_messages: parseInt(messageResult.rows[0].message_count),
total_rooms: parseInt(roomResult.rows[0].room_count)
}
});
} catch (error) {
console.error('Database test error:', error);
res.status(500).json({
status: 'error',
error: 'Database connection failed',
details: error.message
});
} catch (err) {
next(err);
}
});
// Mount all routes from routes.js
app.use('/', chatRoutes);
app.use(errorHandler);
// Phase 9 §9.1 — bind to 127.0.0.1. Caddy reverse_proxy targets localhost:3014
// already; this closes the gap where unauthenticated direct-port access from
// any host on the network was possible.
const server = app.listen(port, '127.0.0.1', () => {
console.log(`Chat server running on 127.0.0.1:${port}`);
// Health check endpoint
app.get('/health', (req, res) => {
res.json({ status: 'healthy' });
});
const shutdown = async (signal) => {
console.log(`chat-server shutting down (${signal})`);
server.close();
try { await pool.end(); } catch { /* ignore */ }
try { await inventoryPool.end(); } catch { /* ignore */ }
process.exit(0);
};
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));
// Error handling middleware
app.use((err, req, res, next) => {
console.error(err.stack);
res.status(500).json({ error: 'Something broke!' });
});
// Start server
app.listen(port, () => {
console.log(`Chat server running on port ${port}`);
});
inventory-server/chat/update-config-template.env
-26
View File
@@ -1,26 +0,0 @@
# Chat Server Database Configuration Template
# Copy this to your .env file and update the values for your new server
# Database Configuration for New Server
CHAT_DB_HOST=your-new-server-ip-or-hostname
CHAT_DB_PORT=5432
CHAT_DB_NAME=rocketchat_converted
CHAT_DB_USER=rocketchat_user
CHAT_DB_PASSWORD=your-secure-password
# Chat Server Port
CHAT_PORT=3014
# Example configuration:
# CHAT_DB_HOST=192.168.1.100
# CHAT_DB_PORT=5432
# CHAT_DB_NAME=rocketchat_converted
# CHAT_DB_USER=rocketchat_user
# CHAT_DB_PASSWORD=MySecureP@ssw0rd123
# Notes:
# - Replace 'your-new-server-ip-or-hostname' with actual server address
# - Use a strong password for CHAT_DB_PASSWORD
# - Ensure the new server allows connections from your application server
# - Update any firewall rules to allow PostgreSQL connections (port 5432)
# - Test connectivity before updating production configuration
inventory-server/chat/verify-migration.cjs
-231
View File
@@ -1,231 +0,0 @@
#!/usr/bin/env node
/**
* Chat Database Migration Verification Script
*
* This script verifies that the chat database migration was successful
* by comparing record counts and testing basic functionality.
*/
require('dotenv').config({ path: '../.env' });
const { Pool } = require('pg');
// Database configuration
const pool = new Pool({
host: process.env.CHAT_DB_HOST || 'localhost',
user: process.env.CHAT_DB_USER || 'rocketchat_user',
password: process.env.CHAT_DB_PASSWORD,
database: process.env.CHAT_DB_NAME || 'rocketchat_converted',
port: process.env.CHAT_DB_PORT || 5432,
});
const originalStats = process.argv[2] ? JSON.parse(process.argv[2]) : null;
async function verifyMigration() {
console.log('🔍 Starting migration verification...\n');
try {
// Test basic connection
console.log('🔗 Testing database connection...');
const versionResult = await pool.query('SELECT version()');
console.log('✅ Database connection successful');
console.log(` PostgreSQL version: ${versionResult.rows[0].version.split(' ')[1]}\n`);
// Get table statistics
console.log('📊 Checking table statistics...');
const statsResult = await pool.query(`
SELECT
tablename,
n_live_tup as row_count,
n_dead_tup as dead_rows,
schemaname
FROM pg_stat_user_tables
WHERE schemaname = 'public'
ORDER BY n_live_tup DESC
`);
if (statsResult.rows.length === 0) {
console.log('❌ No tables found! Migration may have failed.');
return false;
}
console.log('📋 Table Statistics:');
console.log(' Table Name | Row Count | Dead Rows');
console.log(' -------------------|-----------|----------');
let totalRows = 0;
const tableStats = {};
for (const row of statsResult.rows) {
const rowCount = parseInt(row.row_count) || 0;
const deadRows = parseInt(row.dead_rows) || 0;
totalRows += rowCount;
tableStats[row.tablename] = rowCount;
console.log(` ${row.tablename.padEnd(18)} | ${rowCount.toString().padStart(9)} | ${deadRows.toString().padStart(8)}`);
}
console.log(`\n Total rows across all tables: ${totalRows}\n`);
// Verify critical tables exist and have data
const criticalTables = ['users', 'message', 'room'];
console.log('🔑 Checking critical tables...');
for (const table of criticalTables) {
if (tableStats[table] > 0) {
console.log(`✅ ${table}: ${tableStats[table]} rows`);
} else if (tableStats[table] === 0) {
console.log(`⚠️ ${table}: table exists but is empty`);
} else {
console.log(`❌ ${table}: table not found`);
return false;
}
}
// Test specific functionality
console.log('\n🧪 Testing specific functionality...');
// Test users table
const userTest = await pool.query(`
SELECT COUNT(*) as total_users,
COUNT(*) FILTER (WHERE active = true) as active_users,
COUNT(*) FILTER (WHERE type = 'user') as regular_users
FROM users
`);
if (userTest.rows[0]) {
const { total_users, active_users, regular_users } = userTest.rows[0];
console.log(`✅ Users: ${total_users} total, ${active_users} active, ${regular_users} regular users`);
}
// Test messages table
const messageTest = await pool.query(`
SELECT COUNT(*) as total_messages,
COUNT(DISTINCT rid) as unique_rooms,
MIN(ts) as oldest_message,
MAX(ts) as newest_message
FROM message
`);
if (messageTest.rows[0]) {
const { total_messages, unique_rooms, oldest_message, newest_message } = messageTest.rows[0];
console.log(`✅ Messages: ${total_messages} total across ${unique_rooms} rooms`);
if (oldest_message && newest_message) {
console.log(` Date range: ${oldest_message.toISOString().split('T')[0]} to ${newest_message.toISOString().split('T')[0]}`);
}
}
// Test rooms table
const roomTest = await pool.query(`
SELECT COUNT(*) as total_rooms,
COUNT(*) FILTER (WHERE t = 'c') as channels,
COUNT(*) FILTER (WHERE t = 'p') as private_groups,
COUNT(*) FILTER (WHERE t = 'd') as direct_messages
FROM room
`);
if (roomTest.rows[0]) {
const { total_rooms, channels, private_groups, direct_messages } = roomTest.rows[0];
console.log(`✅ Rooms: ${total_rooms} total (${channels} channels, ${private_groups} private, ${direct_messages} DMs)`);
}
// Test file uploads if table exists
if (tableStats.uploads > 0) {
const uploadTest = await pool.query(`
SELECT COUNT(*) as total_uploads,
COUNT(DISTINCT typegroup) as file_types,
pg_size_pretty(SUM(size)) as total_size
FROM uploads
WHERE size IS NOT NULL
`);
if (uploadTest.rows[0]) {
const { total_uploads, file_types, total_size } = uploadTest.rows[0];
console.log(`✅ Uploads: ${total_uploads} files, ${file_types} types, ${total_size || 'unknown size'}`);
}
}
// Test server health endpoint simulation
console.log('\n🏥 Testing application endpoints simulation...');
try {
const healthTest = await pool.query(`
SELECT
(SELECT COUNT(*) FROM users WHERE active = true) as active_users,
(SELECT COUNT(*) FROM message) as total_messages,
(SELECT COUNT(*) FROM room) as total_rooms
`);
if (healthTest.rows[0]) {
const stats = healthTest.rows[0];
console.log('✅ Health check simulation passed');
console.log(` Active users: ${stats.active_users}`);
console.log(` Total messages: ${stats.total_messages}`);
console.log(` Total rooms: ${stats.total_rooms}`);
}
} catch (error) {
console.log(`⚠️ Health check simulation failed: ${error.message}`);
}
// Check indexes
console.log('\n📇 Checking database indexes...');
const indexResult = await pool.query(`
SELECT
schemaname,
tablename,
indexname,
indexdef
FROM pg_indexes
WHERE schemaname = 'public'
ORDER BY tablename, indexname
`);
const indexesByTable = {};
for (const idx of indexResult.rows) {
if (!indexesByTable[idx.tablename]) {
indexesByTable[idx.tablename] = [];
}
indexesByTable[idx.tablename].push(idx.indexname);
}
for (const [table, indexes] of Object.entries(indexesByTable)) {
console.log(` ${table}: ${indexes.length} indexes`);
}
console.log('\n🎉 Migration verification completed successfully!');
console.log('\n✅ Summary:');
console.log(` - Database connection: Working`);
console.log(` - Tables created: ${statsResult.rows.length}`);
console.log(` - Total data rows: ${totalRows}`);
console.log(` - Critical tables: All present`);
console.log(` - Indexes: ${indexResult.rows.length} total`);
console.log('\n🚀 Next steps:');
console.log(' 1. Update your application configuration');
console.log(' 2. Start your chat server');
console.log(' 3. Test chat functionality in the browser');
console.log(' 4. Monitor logs for any issues');
return true;
} catch (error) {
console.error('❌ Migration verification failed:', error.message);
console.error('\n🔧 Troubleshooting steps:');
console.error(' 1. Check database connection settings');
console.error(' 2. Verify database and user exist');
console.error(' 3. Check PostgreSQL logs');
console.error(' 4. Ensure import completed without errors');
return false;
} finally {
await pool.end();
}
}
// Run verification
if (require.main === module) {
verifyMigration().then(success => {
process.exit(success ? 0 : 1);
});
}
module.exports = { verifyMigration };
inventory-server/dashboard/.env.example
-32
View File
@@ -1,32 +0,0 @@
# dashboard-server .env template (Phase 4)
#
# The merged dashboard-server reads /var/www/inventory/.env FIRST (provides
# JWT_SECRET, DB_*, REDIS_*) and then layers this .env on top for vendor keys.
# Shared/security-critical vars stay in /var/www/inventory/.env so they aren't
# duplicated; vendor keys live here.
#
# Copy to .env and populate. Do NOT commit the populated file.
# Port the merged service listens on
DASHBOARD_PORT=3015
# Klaviyo (replaces klaviyo-server/.env)
KLAVIYO_API_KEY=
KLAVIYO_API_REVISION=2024-02-15
KLAVIYO_API_URL=https://a.klaviyo.com/api
# Meta / Facebook Ads (replaces meta-server/.env)
META_ACCESS_TOKEN=
META_AD_ACCOUNT_ID=
META_API_VERSION=v21.0
# Google Analytics (replaces google-server/.env)
GA_PROPERTY_ID=
GOOGLE_APPLICATION_CREDENTIALS_JSON=
# Typeform (replaces typeform-server/.env)
TYPEFORM_ACCESS_TOKEN=
# Vendors share the inventory REDIS_URL or REDIS_HOST/PORT/USERNAME/PASSWORD
# from the parent .env. Do NOT redeclare here unless you need a vendor-only
# override (rare; would need to fork shared/db/redis.js too).
inventory-server/dashboard/acot-server/README.md
-205
View File
@@ -1,205 +0,0 @@
# ACOT Server
This server replaces the Klaviyo integration with direct database queries to the production MySQL database via SSH tunnel. It provides seamless API compatibility for all frontend components without requiring any frontend changes.
## Setup
1. **Environment Variables**: Copy `.env.example` to `.env` and configure:
```
DB_HOST=localhost
DB_PORT=3306
DB_USER=your_db_user
DB_PASSWORD=your_db_password
DB_NAME=your_db_name
PORT=3007
NODE_ENV=development
```
2. **SSH Tunnel**: Ensure your SSH tunnel to the production database is running on localhost:3306.
3. **Install Dependencies**:
```bash
npm install
```
4. **Start Server**:
```bash
npm start
```
## API Endpoints
All endpoints provide exact API compatibility with the previous Klaviyo implementation:
### Main Statistics
- `GET /api/acot/events/stats` - Complete statistics dashboard data
- Query params: `timeRange` (today, yesterday, thisWeek, lastWeek, thisMonth, lastMonth, last7days, last30days, last90days) or `startDate`/`endDate` for custom ranges
- Returns: Revenue, orders, AOV, shipping data, order types, brands/categories, refunds, cancellations, best day, peak hour, order ranges, period progress, projections
### Daily Details
- `GET /api/acot/events/stats/details` - Daily breakdown with previous period comparisons
- Query params: `timeRange`, `metric` (revenue, orders, average_order, etc.), `daily=true`
- Returns: Array of daily data points with trend comparisons
### Products
- `GET /api/acot/events/products` - Top products with sales data
- Query params: `timeRange`
- Returns: Product list with images, sales quantities, revenue, and order counts
### Projections
- `GET /api/acot/events/projection` - Smart revenue projections for incomplete periods
- Query params: `timeRange`
- Returns: Projected revenue with confidence levels based on historical patterns
### Health Check
- `GET /api/acot/test` - Server health and database connectivity test
## Database Schema
The server queries the following main tables:
### Orders (`_order`)
- **Key fields**: `order_id`, `date_placed`, `summary_total`, `order_status`, `ship_method_selected`, `stats_waiting_preorder`
- **Valid orders**: `order_status > 15`
- **Cancelled orders**: `order_status = 15`
- **Shipped orders**: `order_status IN (100, 92)`
- **Pre-orders**: `stats_waiting_preorder > 0`
- **Local pickup**: `ship_method_selected = 'localpickup'`
- **On-hold orders**: `ship_method_selected = 'holdit'`
### Order Items (`order_items`)
- **Fields**: `order_id`, `prod_pid`, `qty_ordered`, `prod_price`
- **Purpose**: Links orders to products for detailed analysis
### Products (`products`)
- **Fields**: `pid`, `description` (product name), `company`
- **Purpose**: Product information and brand data
### Product Images (`product_images`)
- **Fields**: `pid`, `iid`, `order` (priority)
- **Primary image**: `order = 255` (highest priority)
- **Image URL generation**: `https://sbing.com/i/products/0000/{prefix}/{pid}-{type}-{iid}.jpg`
### Payments (`order_payment`)
- **Refunds**: `payment_amount < 0`
- **Purpose**: Track refund amounts and counts
## Business Logic
### Time Handling
- **Timezone**: All calculations in UTC-5 (Eastern Time)
- **Business Day**: 1 AM - 12:59 AM Eastern (25-hour business day)
- **Format**: MySQL DATETIME format (YYYY-MM-DD HH:MM:SS)
- **Period Boundaries**: Calculated using `timeUtils.js` for consistent time range handling
### Order Processing
- **Revenue Calculation**: Only includes orders with `order_status > 15`
- **Order Types**:
- Pre-orders: `stats_waiting_preorder > 0`
- Local pickup: `ship_method_selected = 'localpickup'`
- On-hold: `ship_method_selected = 'holdit'`
- **Shipping Methods**: Mapped to friendly names (e.g., `usps_ground_advantage` → "USPS Ground Advantage")
### Projections
- **Period Progress**: Calculated based on current time within the selected period
- **Simple Projection**: Linear extrapolation based on current progress
- **Smart Projection**: Uses historical data patterns for more accurate forecasting
- **Confidence Levels**: Based on data consistency and historical accuracy
### Image URL Generation
- **Pattern**: `https://sbing.com/i/products/0000/{prefix}/{pid}-{type}-{iid}.jpg`
- **Prefix**: First 2 digits of product ID
- **Type**: "main" for primary images
- **Fallback**: Uses primary image (order=255) when available
## Frontend Integration
### Service Layer (`services/acotService.js`)
- **Purpose**: Replaces direct Klaviyo API calls with acot-server calls
- **Methods**: `getStats()`, `getStatsDetails()`, `getProducts()`, `getProjection()`
- **Logging**: Axios interceptors for request/response logging
- **Environment**: Automatic URL handling (proxy in dev, direct in production)
### Component Updates
All 5 main components updated to use `acotService`:
- **StatCards.jsx**: Main dashboard statistics
- **MiniStatCards.jsx**: Compact statistics view
- **SalesChart.jsx**: Revenue and order trends
- **MiniSalesChart.jsx**: Compact chart view
- **ProductGrid.jsx**: Top products table
### Proxy Configuration (`vite.config.js`)
```javascript
'/api/acot': {
target: 'http://localhost:3007',
changeOrigin: true,
secure: false
}
```
## Key Features
### Complete Business Intelligence
- **Revenue Analytics**: Total revenue, trends, projections
- **Order Analysis**: Counts, types, status tracking
- **Product Performance**: Top sellers, revenue contribution
- **Shipping Intelligence**: Methods, locations, distribution
- **Customer Insights**: Order value ranges, patterns
- **Operational Metrics**: Refunds, cancellations, peak hours
### Performance Optimizations
- **Connection Pooling**: Efficient database connection management
- **Query Optimization**: Indexed queries with proper WHERE clauses
- **Caching Strategy**: Frontend caching for detail views
- **Batch Processing**: Efficient data aggregation
### Error Handling
- **Database Connectivity**: Graceful handling of connection issues
- **Query Failures**: Detailed error logging and user-friendly messages
- **Data Validation**: Input sanitization and validation
- **Fallback Mechanisms**: Default values for missing data
## Simplified Elements
Due to database complexity, some features are simplified:
- **Brands**: Shows "Various Brands" (companies table structure complex)
- **Categories**: Shows "General" (category relationships complex)
These can be enhanced in future iterations with proper category mapping.
## Testing
Test the server functionality:
```bash
# Health check
curl http://localhost:3007/api/acot/test
# Today's stats
curl http://localhost:3007/api/acot/events/stats?timeRange=today
# Last 30 days with details
curl http://localhost:3007/api/acot/events/stats/details?timeRange=last30days&daily=true
# Top products
curl http://localhost:3007/api/acot/events/products?timeRange=thisWeek
# Revenue projection
curl http://localhost:3007/api/acot/events/projection?timeRange=today
```
## Development Notes
- **No Frontend Changes**: Complete drop-in replacement for Klaviyo
- **API Compatibility**: Maintains exact response structure
- **Business Logic**: Implements all complex e-commerce calculations
- **Scalability**: Designed for production workloads
- **Maintainability**: Well-documented code with clear separation of concerns
## Future Enhancements
- Enhanced category and brand mapping
- Real-time notifications for significant events
- Advanced analytics and forecasting
- Customer segmentation analysis
- Inventory integration
inventory-server/dashboard/acot-server/db/connection.js
-307
View File
@@ -1,307 +0,0 @@
// Per Deviation #13 in CONSOLIDATION_PLAN.md: `ssh2` is CJS and its named export
// (`Client`) isn't reliably detected by Node's CJS→ESM interop static analysis.
// Default-import + destructure is the bulletproof pattern.
import ssh2 from 'ssh2';
import mysql from 'mysql2/promise';
import fs from 'node:fs';
const { Client } = ssh2;
// Connection pool configuration
const connectionPool = {
connections: [],
maxConnections: 20,
currentConnections: 0,
pendingRequests: [],
// Cache for query results (key: query string, value: {data, timestamp})
queryCache: new Map(),
// Cache duration for different query types in milliseconds
cacheDuration: {
'stats': 60 * 1000, // 1 minute for stats
'products': 5 * 60 * 1000, // 5 minutes for products
'orders': 60 * 1000, // 1 minute for orders
'default': 60 * 1000 // 1 minute default
},
// Circuit breaker state
circuitBreaker: {
failures: 0,
lastFailure: 0,
isOpen: false,
threshold: 5,
timeout: 30000 // 30 seconds
}
};
/**
* Get a database connection from the pool
* @returns {Promise<{connection: object, release: function}>} The database connection and release function
*/
async function getDbConnection() {
return new Promise(async (resolve, reject) => {
// Check circuit breaker
const now = Date.now();
if (connectionPool.circuitBreaker.isOpen) {
if (now - connectionPool.circuitBreaker.lastFailure > connectionPool.circuitBreaker.timeout) {
// Reset circuit breaker
connectionPool.circuitBreaker.isOpen = false;
connectionPool.circuitBreaker.failures = 0;
console.log('Circuit breaker reset');
} else {
reject(new Error('Circuit breaker is open - too many connection failures'));
return;
}
}
// Check if there's an available connection in the pool
if (connectionPool.connections.length > 0) {
const conn = connectionPool.connections.pop();
console.log(`Using pooled connection. Pool size: ${connectionPool.connections.length}`);
resolve({
connection: conn.connection,
release: () => releaseConnection(conn)
});
return;
}
// If we haven't reached max connections, create a new one
if (connectionPool.currentConnections < connectionPool.maxConnections) {
try {
console.log(`Creating new connection. Current: ${connectionPool.currentConnections}/${connectionPool.maxConnections}`);
connectionPool.currentConnections++;
const tunnel = await setupSshTunnel();
const { ssh, stream, dbConfig } = tunnel;
const connection = await mysql.createConnection({
...dbConfig,
stream
});
const conn = { ssh, connection, inUse: true, created: Date.now() };
console.log('Database connection established');
// Reset circuit breaker on successful connection
if (connectionPool.circuitBreaker.failures > 0) {
connectionPool.circuitBreaker.failures = 0;
connectionPool.circuitBreaker.isOpen = false;
}
resolve({
connection: conn.connection,
release: () => releaseConnection(conn)
});
} catch (error) {
connectionPool.currentConnections--;
// Track circuit breaker failures
connectionPool.circuitBreaker.failures++;
connectionPool.circuitBreaker.lastFailure = Date.now();
if (connectionPool.circuitBreaker.failures >= connectionPool.circuitBreaker.threshold) {
connectionPool.circuitBreaker.isOpen = true;
console.log(`Circuit breaker opened after ${connectionPool.circuitBreaker.failures} failures`);
}
reject(error);
}
return;
}
// Pool is full, queue the request with timeout
console.log('Connection pool full, queuing request...');
const timeoutId = setTimeout(() => {
// Remove from queue if still there
const index = connectionPool.pendingRequests.findIndex(req => req.resolve === resolve);
if (index !== -1) {
connectionPool.pendingRequests.splice(index, 1);
reject(new Error('Connection pool queue timeout after 15 seconds'));
}
}, 15000);
connectionPool.pendingRequests.push({
resolve,
reject,
timeoutId,
timestamp: Date.now()
});
});
}
/**
* Release a connection back to the pool
*/
function releaseConnection(conn) {
conn.inUse = false;
// Check if there are pending requests
if (connectionPool.pendingRequests.length > 0) {
const { resolve, timeoutId } = connectionPool.pendingRequests.shift();
// Clear the timeout since we're serving the request
if (timeoutId) {
clearTimeout(timeoutId);
}
conn.inUse = true;
console.log(`Serving queued request. Queue length: ${connectionPool.pendingRequests.length}`);
resolve({
connection: conn.connection,
release: () => releaseConnection(conn)
});
} else {
// Return to pool
connectionPool.connections.push(conn);
console.log(`Connection returned to pool. Pool size: ${connectionPool.connections.length}, Active: ${connectionPool.currentConnections}`);
}
}
/**
* Get cached query results or execute query if not cached
* @param {string} cacheKey - Unique key to identify the query
* @param {string} queryType - Type of query (stats, products, orders, etc.)
* @param {Function} queryFn - Function to execute if cache miss
* @returns {Promise<any>} The query result
*/
async function getCachedQuery(cacheKey, queryType, queryFn) {
// Get cache duration based on query type
const cacheDuration = connectionPool.cacheDuration[queryType] || connectionPool.cacheDuration.default;
// Check if we have a valid cached result
const cachedResult = connectionPool.queryCache.get(cacheKey);
const now = Date.now();
if (cachedResult && (now - cachedResult.timestamp < cacheDuration)) {
console.log(`Cache hit for ${queryType} query: ${cacheKey}`);
return cachedResult.data;
}
// No valid cache found, execute the query
console.log(`Cache miss for ${queryType} query: ${cacheKey}`);
const result = await queryFn();
// Cache the result
connectionPool.queryCache.set(cacheKey, {
data: result,
timestamp: now
});
return result;
}
/**
* Setup SSH tunnel to production database
* @private - Should only be used by getDbConnection
* @returns {Promise<{ssh: object, stream: object, dbConfig: object}>}
*/
async function setupSshTunnel() {
const sshConfig = {
host: process.env.PROD_SSH_HOST,
port: process.env.PROD_SSH_PORT || 22,
username: process.env.PROD_SSH_USER,
privateKey: process.env.PROD_SSH_KEY_PATH
? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
: undefined,
compress: true
};
const dbConfig = {
host: process.env.PROD_DB_HOST || 'localhost',
user: process.env.PROD_DB_USER,
password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME,
port: process.env.PROD_DB_PORT || 3306,
// DATETIME columns in this DB store Central wall-clock literals. Return
// them as literal strings so consumers parse them explicitly with
// parseMySql() (zone America/Chicago) from shared/business-time — never
// let the driver guess a zone (the old `timezone: 'Z'` mangled every
// value by labeling Central wall-clock as UTC).
dateStrings: true
};
return new Promise((resolve, reject) => {
const ssh = new Client();
ssh.on('error', (err) => {
console.error('SSH connection error:', err);
reject(err);
});
ssh.on('ready', () => {
ssh.forwardOut(
'127.0.0.1',
0,
dbConfig.host,
dbConfig.port,
(err, stream) => {
if (err) reject(err);
resolve({ ssh, stream, dbConfig });
}
);
}).connect(sshConfig);
});
}
/**
* Clear cached query results
* @param {string} [cacheKey] - Specific cache key to clear (clears all if not provided)
*/
function clearQueryCache(cacheKey) {
if (cacheKey) {
connectionPool.queryCache.delete(cacheKey);
console.log(`Cleared cache for key: ${cacheKey}`);
} else {
connectionPool.queryCache.clear();
console.log('Cleared all query cache');
}
}
/**
* Force close all active connections
* Useful for server shutdown or manual connection reset
*/
async function closeAllConnections() {
// Close all pooled connections
for (const conn of connectionPool.connections) {
try {
await conn.connection.end();
conn.ssh.end();
console.log('Closed pooled connection');
} catch (error) {
console.error('Error closing pooled connection:', error);
}
}
// Reset pool state
connectionPool.connections = [];
connectionPool.currentConnections = 0;
connectionPool.pendingRequests = [];
connectionPool.queryCache.clear();
console.log('All connections closed and pool reset');
}
/**
* Get connection pool status for debugging
*/
function getPoolStatus() {
return {
poolSize: connectionPool.connections.length,
activeConnections: connectionPool.currentConnections,
maxConnections: connectionPool.maxConnections,
pendingRequests: connectionPool.pendingRequests.length,
cacheSize: connectionPool.queryCache.size,
queuedRequests: connectionPool.pendingRequests.map(req => ({
waitTime: Date.now() - req.timestamp,
hasTimeout: !!req.timeoutId
}))
};
}
export {
getDbConnection,
getCachedQuery,
clearQueryCache,
closeAllConnections,
getPoolStatus,
};
inventory-server/dashboard/acot-server/package-lock.json
Generated
-1700
View File
File diff suppressed because it is too large Load Diff
inventory-server/dashboard/acot-server/package.json
-25
View File
@@ -1,25 +0,0 @@
{
"name": "acot-server",
"version": "1.0.0",
"description": "A Cherry On Top production database server",
"type": "module",
"main": "server.js",
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js"
},
"dependencies": {
"compression": "^1.7.4",
"cors": "^2.8.5",
"dotenv": "^16.3.1",
"express": "^4.18.2",
"luxon": "^3.5.0",
"morgan": "^1.10.0",
"mysql2": "^3.6.5",
"pg": "^8.21.0",
"ssh2": "^1.14.0"
},
"devDependencies": {
"nodemon": "^3.0.1"
}
}
inventory-server/dashboard/acot-server/routes/customers.js
-325
View File
@@ -1,325 +0,0 @@
// Customer lookup for the phone app (acot-phone-server).
//
// All queries hit the MySQL `sg` database via the shared SSH-tunneled pool in
// db/connection.js. The stats/orders logic mirrors the freescout
// ACOTCustomerData module so both apps display the same numbers for a given
// customer — the difference is that we key by phone, not email.
//
// NOTE: `users.phone` is not yet indexed in production. Admin will add
// `idx_phone (phone)` — queries here assume that exists for acceptable latency.
import express from 'express';
import { getDbConnection, getCachedQuery } from '../db/connection.js';
import { requirePhoneApiKey } from '../utils/phoneAuth.js';
const router = express.Router();
// Order status labels mirror ACOTCustomerDataServiceProvider.php.
const ORDER_STATUS_LABEL = {
0: 'Created', 10: 'Incomplete', 15: 'Cancelled', 16: 'Combined',
20: 'Placed', 22: 'Placed (Incomplete)', 40: 'Awaiting Payment',
45: 'Payment Pending', 50: 'Awaiting Products', 55: 'Shipping Later',
56: 'Shipping Together', 60: 'Ready', 61: 'Flagged', 62: 'Fix Before Pick',
65: 'Manual Picking', 67: 'Remote Send', 70: 'In PT', 80: 'Picked',
90: 'Awaiting Shipment', 91: 'Remote Wait', 92: 'Awaiting Pickup',
93: 'Fix Before Ship', 95: 'Shipped (Confirmed)', 100: 'Shipped',
};
const ORDER_STATUS_SHORT = {
0: 'Created', 10: 'Incomplete', 15: 'Cancelled', 16: 'Combined',
20: 'Placed', 22: 'Plcd Incomp', 40: 'Await Payment', 45: 'Pymt Pending',
50: 'Await Products', 55: 'Ship Later', 56: 'Ship Togethr', 60: 'Ready',
61: 'Flagged', 62: 'Fix Bfr Pick', 65: 'Manual Pick', 67: 'Remote Send',
70: 'In PT', 80: 'Picked', 90: 'Await Ship', 91: 'Remote Wait',
92: 'Await Pickup', 93: 'Fix Bfr Ship', 95: 'Shpd Confirm', 100: 'Shipped',
};
function statusLabel(s) { return ORDER_STATUS_LABEL[s] ?? `Unknown (${s})`; }
function statusShort(s) { return ORDER_STATUS_SHORT[s] ?? `Unknown (${s})`; }
// SIP trunks and historical CRM imports all disagree on phone format. Rather
// than normalize everything upstream, we search across the most common
// variations for US/Canada numbers. Falls through to the raw input for
// international numbers we can't safely reformat.
function phoneVariations(input) {
const raw = String(input || '').trim();
if (!raw) return [];
const digits = raw.replace(/\D/g, '');
const out = new Set([raw, digits]);
if (digits.length === 10) {
out.add(`+1${digits}`);
out.add(`1${digits}`);
} else if (digits.length === 11 && digits.startsWith('1')) {
out.add(`+${digits}`);
out.add(digits.slice(1)); // 10-digit form
out.add(`+1${digits.slice(1)}`);
}
return Array.from(out).filter(Boolean);
}
function trackingLink(method, tracking) {
if (!tracking) return '';
if (typeof method === 'string') {
if (method.startsWith('usps_') || method === 'fedex_smartpost') {
return `https://tools.usps.com/go/TrackConfirmAction?qtc_tLabels1=${tracking}`;
}
if (method.startsWith('fedex_')) {
return `https://www.fedex.com/fedextrack/?trknbr=${tracking}`;
}
}
return '';
}
// Matches ACOTCustomerDataServiceProvider::imageUrl — sbing.com/i/products/<dir1>/<dir2>/<pid>-t-<iid>.jpg
function imageUrl(pid, iid = 1) {
const padded = String(pid).padStart(10, '0');
const dir1 = padded.slice(0, 4);
const dir2 = padded.slice(4, 7);
return `https://sbing.com/i/products/${dir1}/${dir2}/${pid}-t-${iid}.jpg`;
}
router.use(requirePhoneApiKey);
// ── GET /by-phone ──────────────────────────────────────────────────────────
// Returns top-line customer info for the incoming-call overlay.
router.get('/by-phone', async (req, res) => {
const phone = String(req.query.phone || '').trim();
if (!phone) return res.status(400).json({ success: false, error: 'phone required' });
const variations = phoneVariations(phone);
if (variations.length === 0) return res.json({ success: true, customer: null });
try {
const data = await getCachedQuery(
`customer-by-phone:${variations.join('|')}`,
'default',
async () => {
const { connection, release } = await getDbConnection();
try {
const placeholders = variations.map(() => '?').join(',');
// Tie-break by highest LTV per user instructions: subquery computes LTV
// for every matching user, then we pick the biggest.
const [users] = await connection.execute(
`SELECT u.cid, u.uid, u.firstname, u.lastname, u.email, u.phone, u.points,
COALESCE((
SELECT SUM(summary_total)
FROM _order
WHERE order_cid = u.cid AND order_status >= 50
), 0) AS lifetime_value,
COALESCE((
SELECT COUNT(*)
FROM _order
WHERE order_cid = u.cid AND order_status >= 20
), 0) AS num_orders,
(
SELECT AVG(summary_total)
FROM _order
WHERE order_cid = u.cid AND order_status >= 20
) AS avg_order
FROM users u
WHERE u.phone IN (${placeholders})
ORDER BY lifetime_value DESC
LIMIT 1`,
variations
);
return users[0] ?? null;
} finally {
release();
}
}
);
if (!data) return res.json({ success: true, customer: null });
res.json({
success: true,
customer: {
cid: Number(data.cid),
uid: data.uid,
firstName: data.firstname || null,
lastName: data.lastname || null,
email: data.email || null,
phone: data.phone,
points: Number(data.points) || 0,
lifetimeValue: Number(data.lifetime_value) || 0,
orderCount: Number(data.num_orders) || 0,
avgOrderValue: data.avg_order != null ? Number(data.avg_order) : 0,
},
});
} catch (err) {
console.error('customers/by-phone failed:', err);
res.status(500).json({ success: false, error: 'query_failed' });
}
});
// ── GET /search ────────────────────────────────────────────────────────────
// Name search for the dialer. Accepts a free-text query; splits on whitespace.
// - 1 token: LIKE against firstname OR lastname (prefix).
// - 2+ tokens: firstname LIKE A% AND lastname LIKE B% (order-sensitive on purpose).
router.get('/search', async (req, res) => {
const q = String(req.query.q || '').trim();
const limit = Math.min(Math.max(parseInt(req.query.limit || '10', 10) || 10, 1), 25);
if (q.length < 2) return res.json({ success: true, results: [] });
try {
const data = await getCachedQuery(
`customer-search:${q}:${limit}`,
'default',
async () => {
const { connection, release } = await getDbConnection();
try {
const tokens = q.split(/\s+/).filter(Boolean);
let sql;
let params;
if (tokens.length === 1) {
const pattern = `${tokens[0]}%`;
sql = `SELECT cid, firstname, lastname, email, phone
FROM users
WHERE (firstname LIKE ? OR lastname LIKE ?)
AND phone <> ''
ORDER BY lastname, firstname
LIMIT ?`;
params = [pattern, pattern, limit];
} else {
const firstPat = `${tokens[0]}%`;
const lastPat = `${tokens.slice(1).join(' ')}%`;
sql = `SELECT cid, firstname, lastname, email, phone
FROM users
WHERE firstname LIKE ? AND lastname LIKE ?
AND phone <> ''
ORDER BY lastname, firstname
LIMIT ?`;
params = [firstPat, lastPat, limit];
}
const [rows] = await connection.execute(sql, params);
return rows;
} finally {
release();
}
}
);
res.json({
success: true,
results: data.map((r) => ({
cid: Number(r.cid),
firstName: r.firstname || null,
lastName: r.lastname || null,
email: r.email || null,
phone: r.phone,
})),
});
} catch (err) {
console.error('customers/search failed:', err);
res.status(500).json({ success: false, error: 'query_failed' });
}
});
// ── GET /:cid/orders ───────────────────────────────────────────────────────
// Recent orders for the active-call screen — mirrors the freescout sidebar.
router.get('/:cid/orders', async (req, res) => {
const cid = Number(req.params.cid);
if (!Number.isFinite(cid) || cid <= 0) {
return res.status(400).json({ success: false, error: 'bad_cid' });
}
try {
const data = await getCachedQuery(
`customer-orders:${cid}`,
'orders',
async () => {
const { connection, release } = await getDbConnection();
try {
// MySQL-safe equivalent of the Laravel query in the freescout module.
// Active = placed OR shipped within the last 3 months.
// NOW() here is Central (server TZ) — intended: this is a rough
// trailing window, not a business-day boundary.
const [ordersRaw] = await connection.execute(
`SELECT order_id, order_status, order_type, summary_total,
date_placed, ship_method_type, ship_method_tracking,
CASE
WHEN (order_status BETWEEN 20 AND 92
OR date_shipped > DATE_SUB(NOW(), INTERVAL 3 MONTH))
THEN 1 ELSE 0
END AS _is_active
FROM _order
WHERE order_cid = ?
AND (order_status >= 20
OR date_shipped > DATE_SUB(NOW(), INTERVAL 3 MONTH))
ORDER BY _is_active DESC, date_placed DESC`,
[cid]
);
const active = ordersRaw.filter((o) => o._is_active === 1);
const inactive = ordersRaw.filter((o) => o._is_active === 0);
const orders = active.concat(inactive.slice(0, Math.max(0, 10 - active.length)));
if (orders.length === 0) return [];
const orderIds = orders.map((o) => o.order_id);
const idPlaceholders = orderIds.map(() => '?').join(',');
const [items] = await connection.execute(
`SELECT order_id, prod_pid, prod_itemnumber, prod_description, prod_price, qty_ordered
FROM order_items
WHERE order_id IN (${idPlaceholders})`,
orderIds
);
// Main-image lookup: per-pid highest \`order\` at type=3 (matches the
// freescout module's raw SQL).
const pids = [...new Set(items.map((i) => Number(i.prod_pid)).filter(Boolean))];
const mainImagesByPid = new Map();
if (pids.length > 0) {
const pidList = pids.join(',');
const [imgRows] = await connection.execute(
`SELECT pi.pid, pi.iid
FROM product_images pi
INNER JOIN (
SELECT pid, MAX(\`order\`) AS max_order
FROM product_images
WHERE pid IN (${pidList}) AND type = 3
GROUP BY pid
) pm ON pi.pid = pm.pid AND pi.\`order\` = pm.max_order AND pi.type = 3`
);
for (const r of imgRows) mainImagesByPid.set(Number(r.pid), Number(r.iid));
}
const itemsByOrder = new Map();
for (const it of items) {
const oid = Number(it.order_id);
if (!itemsByOrder.has(oid)) itemsByOrder.set(oid, []);
const iid = mainImagesByPid.get(Number(it.prod_pid)) ?? 1;
itemsByOrder.get(oid).push({
pid: Number(it.prod_pid),
sku: it.prod_itemnumber || null,
name: it.prod_description || null,
price: Number(it.prod_price) || 0,
quantity: Number(it.qty_ordered) || 0,
imageUrl: imageUrl(it.prod_pid, iid),
});
}
return orders.map((o) => ({
orderId: Number(o.order_id),
datePlaced: o.date_placed,
total: Number(o.summary_total) || 0,
status: Number(o.order_status),
statusLabel: statusLabel(Number(o.order_status)),
statusShort: statusShort(Number(o.order_status)),
trackingNumber: o.ship_method_tracking || '',
trackingUrl: trackingLink(o.ship_method_type, o.ship_method_tracking),
items: itemsByOrder.get(Number(o.order_id)) || [],
}));
} finally {
release();
}
}
);
res.json({ success: true, orders: data });
} catch (err) {
console.error('customers/:cid/orders failed:', err);
res.status(500).json({ success: false, error: 'query_failed' });
}
});
export default router;
inventory-server/dashboard/acot-server/routes/discounts.js
-579
View File
@@ -1,579 +0,0 @@
import express from 'express';
import { DateTime } from 'luxon';
import { getDbConnection } from '../db/connection.js';
import { BUSINESS_TZ, businessNow, toMySqlBound } from '../../../shared/business-time/index.js';
const router = express.Router();
// Bucket boundaries by summary_subtotal (post-item-sale, pre-order-promo).
// The final entry is open-ended: all orders >= the last bound land there.
const RANGE_BOUNDS = [
10, 20, 30, 40, 50, 60, 70, 80, 90,
100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200,
300, 400, 500, 1000, 1500
];
const FINAL_BUCKET_KEY = '99999';
function buildRangeDefinitions() {
const ranges = [];
let previous = 0;
for (const bound of RANGE_BOUNDS) {
const key = bound.toString().padStart(5, '0');
ranges.push({
min: previous,
max: bound,
label: `$${previous.toLocaleString()} - $${bound.toLocaleString()}`,
key,
});
previous = bound;
}
const lastBound = RANGE_BOUNDS[RANGE_BOUNDS.length - 1];
ranges.push({
min: lastBound,
max: null,
label: `$${lastBound.toLocaleString()}+`,
key: FINAL_BUCKET_KEY,
});
return ranges;
}
const RANGE_DEFINITIONS = buildRangeDefinitions();
function bucketKeyFor(subtotal) {
for (const range of RANGE_DEFINITIONS) {
if (range.max == null) return range.key;
if (subtotal <= range.max) return range.key;
}
return FINAL_BUCKET_KEY;
}
const DEFAULT_POINT_DOLLAR_VALUE = 0.005;
const DEFAULTS = {
merchantFeePercent: 2.9,
fixedCostPerOrder: 1.25,
pointDollarValue: DEFAULT_POINT_DOLLAR_VALUE,
};
// Parse date inputs in BUSINESS time (date-only strings = business dates;
// zoned instants keep their instant). Never zoneless — that was Berlin.
function parseDate(value, fallback) {
if (!value) {
return fallback;
}
const parsed = DateTime.fromISO(value, { zone: BUSINESS_TZ });
if (!parsed.isValid) {
return fallback;
}
return parsed;
}
const formatDateForSql = (dt) => toMySqlBound(dt);
router.get('/promos', async (req, res) => {
let connection;
try {
const { connection: conn, release } = await getDbConnection();
connection = conn;
const releaseConnection = release;
const { startDate, endDate } = req.query || {};
// Half-open business-day range [start, dayAfterEnd) in Chicago time.
const now = businessNow();
const defaultStart = now.minus({ years: 3 }).startOf('day');
const parsedStart = startDate ? parseDate(startDate, defaultStart).startOf('day') : defaultStart;
const parsedEnd = (endDate ? parseDate(endDate, now) : now).startOf('day').plus({ days: 1 });
const rangeStart = parsedStart <= parsedEnd ? parsedStart : parsedEnd;
const rangeEnd = parsedEnd >= parsedStart ? parsedEnd : parsedStart;
const rangeStartSql = formatDateForSql(rangeStart);
const rangeEndSql = formatDateForSql(rangeEnd);
const sql = `
SELECT
p.promo_id AS id,
p.promo_code AS code,
p.promo_description_online AS description_online,
p.promo_description_private AS description_private,
p.date_start,
p.date_end,
COALESCE(u.usage_count, 0) AS usage_count
FROM promos p
LEFT JOIN (
SELECT
discount_code,
COUNT(DISTINCT order_id) AS usage_count
FROM order_discounts
WHERE discount_type = 10 AND discount_active = 1
GROUP BY discount_code
) u ON u.discount_code = p.promo_id
WHERE p.date_start IS NOT NULL
AND p.date_end IS NOT NULL
AND NOT (p.date_end < ? OR p.date_start >= ?)
AND p.store = 1
AND p.date_start >= '2010-01-01'
ORDER BY p.promo_id DESC
LIMIT 200
`;
const [rows] = await connection.execute(sql, [rangeStartSql, rangeEndSql]);
releaseConnection();
const promos = rows.map(row => ({
id: Number(row.id),
code: row.code,
description: row.description_online || row.description_private || '',
privateDescription: row.description_private || '',
promo_description_online: row.description_online || '',
promo_description_private: row.description_private || '',
dateStart: row.date_start,
dateEnd: row.date_end,
usageCount: Number(row.usage_count || 0)
}));
res.json({ promos });
} catch (error) {
if (connection) {
try {
connection.destroy();
} catch (destroyError) {
console.error('Failed to destroy connection after error:', destroyError);
}
}
console.error('Error fetching promos:', error);
res.status(500).json({ error: 'Failed to fetch promos' });
}
});
function emptyBucketAccumulator(range) {
return {
key: range.key,
label: range.label,
min: range.min,
max: range.max,
orderCount: 0,
sumOrderValue: 0,
sumProductDiscountAmount: 0,
sumPromoProductDiscount: 0,
sumCustomerItemCost: 0,
sumShippingChargeBase: 0,
sumShippingAfterAuto: 0,
sumShipPromoDiscount: 0,
sumShippingSurcharge: 0,
sumOrderSurcharge: 0,
sumCustomerShipCost: 0,
sumActualShippingCost: 0,
sumTotalRevenue: 0,
sumProductCogs: 0,
sumMerchantFees: 0,
sumPointsCost: 0,
sumFixedCosts: 0,
sumTotalCosts: 0,
sumProfit: 0,
};
}
function simulateOrder(order, config, derived) {
const orderValue = Number(order.summary_subtotal) || 0;
const retail = Number(order.summary_subtotal_retail) || orderValue;
const productDiscountAmount = Number(order.summary_discount_subtotal) || 0;
const pointsRedeemedDollars = Number(order.points_redeemed) || 0;
// summary_discount_subtotal is a kitchen-sink rollup that includes points
// redemptions (type 20). pointsCost already accrues for points awarded, so
// the points portion of historical discount must be excluded here to avoid
// double-counting it on orders that redeemed points.
const historicalProductDiscountExPoints = Math.max(0, productDiscountAmount - pointsRedeemedDollars);
const shippingChargeBase =
(Number(order.summary_shipping) || 0) + (Number(order.summary_shipping_rush) || 0);
const actualShippingCost = Number(order.ship_method_cost) || 0;
const cogs = Number(order.total_cogs) || 0;
let promoProductDiscount = 0;
if (config.productPromo.type === 'percentage_subtotal' && orderValue >= config.productPromo.minSubtotal) {
promoProductDiscount = orderValue * (config.productPromo.value / 100);
} else if (config.productPromo.type === 'percentage_regular' && orderValue >= config.productPromo.minSubtotal) {
const targetRate = config.productPromo.value / 100;
const targetCustomerPrice = retail * (1 - targetRate);
promoProductDiscount = Math.max(0, orderValue - targetCustomerPrice);
} else if (config.productPromo.type === 'fixed_amount' && orderValue >= config.productPromo.minSubtotal) {
promoProductDiscount = config.productPromo.value;
} else if (config.productPromo.type === 'none' && config.applyHistoricalProductPromo) {
promoProductDiscount = historicalProductDiscountExPoints;
}
promoProductDiscount = Math.max(0, Math.min(promoProductDiscount, orderValue));
let shippingAfterAuto = shippingChargeBase;
for (const tier of config.shippingTiers) {
if (orderValue >= tier.threshold) {
if (tier.mode === 'percentage') {
shippingAfterAuto = shippingChargeBase * Math.max(0, 1 - tier.value / 100);
} else if (tier.mode === 'flat') {
shippingAfterAuto = tier.value;
}
}
}
let shipPromoDiscount = 0;
if (config.shippingPromo.type !== 'none' && orderValue >= config.shippingPromo.minSubtotal) {
if (config.shippingPromo.type === 'percentage') {
shipPromoDiscount = shippingAfterAuto * (config.shippingPromo.value / 100);
} else if (config.shippingPromo.type === 'fixed') {
shipPromoDiscount = config.shippingPromo.value;
}
if (config.shippingPromo.maxDiscount > 0) {
shipPromoDiscount = Math.min(shipPromoDiscount, config.shippingPromo.maxDiscount);
}
shipPromoDiscount = Math.min(shipPromoDiscount, shippingAfterAuto);
}
let shippingSurcharge = 0;
let orderSurcharge = 0;
for (const surcharge of config.surcharges) {
const meetsMin = orderValue >= surcharge.threshold;
const meetsMax = surcharge.maxThreshold == null || orderValue < surcharge.maxThreshold;
if (meetsMin && meetsMax) {
if (surcharge.target === 'shipping') shippingSurcharge += surcharge.amount;
else if (surcharge.target === 'order') orderSurcharge += surcharge.amount;
}
}
const customerShipCost = Math.max(0, shippingAfterAuto - shipPromoDiscount + shippingSurcharge);
const customerItemCost = Math.max(0, orderValue - promoProductDiscount + orderSurcharge);
const totalRevenue = customerItemCost + customerShipCost;
const productCogs = config.cogsCalculationMode === 'average'
? orderValue * derived.overallCogsPercentage
: cogs;
const merchantFees = totalRevenue * (config.merchantFeePercent / 100);
const pointsCost = orderValue * derived.pointsPerDollar * derived.redemptionRate * derived.pointDollarValue;
const fixedCosts = config.fixedCostPerOrder;
const totalCosts = productCogs + actualShippingCost + merchantFees + pointsCost + fixedCosts;
const profit = totalRevenue - totalCosts;
return {
orderValue,
productDiscountAmount,
promoProductDiscount,
customerItemCost,
shippingChargeBase,
shippingAfterAuto,
shipPromoDiscount,
shippingSurcharge,
orderSurcharge,
customerShipCost,
actualShippingCost,
totalRevenue,
productCogs,
merchantFees,
pointsCost,
fixedCosts,
totalCosts,
profit,
};
}
function accumulate(bucket, sim) {
bucket.orderCount += 1;
bucket.sumOrderValue += sim.orderValue;
bucket.sumProductDiscountAmount += sim.productDiscountAmount;
bucket.sumPromoProductDiscount += sim.promoProductDiscount;
bucket.sumCustomerItemCost += sim.customerItemCost;
bucket.sumShippingChargeBase += sim.shippingChargeBase;
bucket.sumShippingAfterAuto += sim.shippingAfterAuto;
bucket.sumShipPromoDiscount += sim.shipPromoDiscount;
bucket.sumShippingSurcharge += sim.shippingSurcharge;
bucket.sumOrderSurcharge += sim.orderSurcharge;
bucket.sumCustomerShipCost += sim.customerShipCost;
bucket.sumActualShippingCost += sim.actualShippingCost;
bucket.sumTotalRevenue += sim.totalRevenue;
bucket.sumProductCogs += sim.productCogs;
bucket.sumMerchantFees += sim.merchantFees;
bucket.sumPointsCost += sim.pointsCost;
bucket.sumFixedCosts += sim.fixedCosts;
bucket.sumTotalCosts += sim.totalCosts;
bucket.sumProfit += sim.profit;
}
function finalizeBucket(b, totalOrders) {
const n = b.orderCount;
const avg = (sum) => (n > 0 ? sum / n : 0);
return {
key: b.key,
label: b.label,
min: b.min,
max: b.max,
orderCount: n,
weight: totalOrders > 0 ? n / totalOrders : 0,
orderValue: avg(b.sumOrderValue),
productDiscountAmount: avg(b.sumProductDiscountAmount),
promoProductDiscount: avg(b.sumPromoProductDiscount),
customerItemCost: avg(b.sumCustomerItemCost),
shippingChargeBase: avg(b.sumShippingChargeBase),
shippingAfterAuto: avg(b.sumShippingAfterAuto),
shipPromoDiscount: avg(b.sumShipPromoDiscount),
shippingSurcharge: avg(b.sumShippingSurcharge),
orderSurcharge: avg(b.sumOrderSurcharge),
customerShipCost: avg(b.sumCustomerShipCost),
actualShippingCost: avg(b.sumActualShippingCost),
totalRevenue: avg(b.sumTotalRevenue),
productCogs: avg(b.sumProductCogs),
merchantFees: avg(b.sumMerchantFees),
pointsCost: avg(b.sumPointsCost),
fixedCosts: avg(b.sumFixedCosts),
totalCosts: avg(b.sumTotalCosts),
profit: avg(b.sumProfit),
profitPercent: b.sumTotalRevenue > 0 ? b.sumProfit / b.sumTotalRevenue : 0,
};
}
router.post('/simulate', async (req, res) => {
const {
dateRange = {},
filters = {},
productPromo = {},
shippingPromo = {},
shippingTiers = [],
surcharges = [],
merchantFeePercent,
fixedCostPerOrder,
cogsCalculationMode = 'actual',
applyHistoricalProductPromo = false,
pointsConfig = {}
} = req.body || {};
// Half-open business-day bounds [startOfDay, dayAfterEnd) in Chicago time.
const endDefault = businessNow();
const startDefault = endDefault.minus({ months: 6 });
const startDt = parseDate(dateRange.start, startDefault).startOf('day');
const endDt = parseDate(dateRange.end, endDefault).startOf('day').plus({ days: 1 });
const shipCountry = filters.shipCountry || 'US';
const promoIds = Array.from(
new Set(
[
...(Array.isArray(filters.promoIds) ? filters.promoIds : []),
...(Array.isArray(filters.promoCodes) ? filters.promoCodes : []),
]
.map((value) => {
if (typeof value === 'string') return value.trim();
if (typeof value === 'number') return String(value);
return '';
})
.filter((value) => value.length > 0)
)
);
const config = {
merchantFeePercent: typeof merchantFeePercent === 'number' ? merchantFeePercent : DEFAULTS.merchantFeePercent,
fixedCostPerOrder: typeof fixedCostPerOrder === 'number' ? fixedCostPerOrder : DEFAULTS.fixedCostPerOrder,
cogsCalculationMode,
applyHistoricalProductPromo: applyHistoricalProductPromo === true,
productPromo: {
type: productPromo.type || 'none',
value: Number(productPromo.value || 0),
minSubtotal: Number(productPromo.minSubtotal || 0)
},
shippingPromo: {
type: shippingPromo.type || 'none',
value: Number(shippingPromo.value || 0),
minSubtotal: Number(shippingPromo.minSubtotal || 0),
maxDiscount: Number(shippingPromo.maxDiscount || 0)
},
shippingTiers: Array.isArray(shippingTiers)
? shippingTiers
.map(tier => ({
threshold: Number(tier.threshold || 0),
mode: tier.mode === 'percentage' || tier.mode === 'flat' ? tier.mode : 'percentage',
value: Number(tier.value || 0)
}))
.filter(tier => tier.threshold >= 0 && tier.value >= 0)
.sort((a, b) => a.threshold - b.threshold)
: [],
surcharges: Array.isArray(surcharges)
? surcharges
.map(s => ({
threshold: Number(s.threshold || 0),
maxThreshold: typeof s.maxThreshold === 'number' && s.maxThreshold > 0 ? s.maxThreshold : null,
target: s.target === 'shipping' || s.target === 'order' ? s.target : 'shipping',
amount: Number(s.amount || 0)
}))
.filter(s => s.threshold >= 0 && s.amount >= 0)
.sort((a, b) => a.threshold - b.threshold)
: [],
points: {
pointsPerDollar: typeof pointsConfig.pointsPerDollar === 'number' ? pointsConfig.pointsPerDollar : null,
redemptionRate: typeof pointsConfig.redemptionRate === 'number' ? pointsConfig.redemptionRate : null,
pointDollarValue: typeof pointsConfig.pointDollarValue === 'number'
? pointsConfig.pointDollarValue
: DEFAULT_POINT_DOLLAR_VALUE
}
};
let connection;
let release;
try {
const dbConn = await getDbConnection();
connection = dbConn.connection;
release = dbConn.release;
const params = [shipCountry, formatDateForSql(startDt), formatDateForSql(endDt)];
let promoExistsClause = '';
if (promoIds.length > 0) {
const placeholders = promoIds.map(() => '?').join(',');
promoExistsClause = `
AND EXISTS (
SELECT 1 FROM order_discounts od
WHERE od.order_id = o.order_id
AND od.discount_active = 1
AND od.discount_type = 10
AND od.discount_code IN (${placeholders})
)
`;
params.push(...promoIds);
}
const ordersQuery = `
SELECT
o.order_id,
o.summary_subtotal,
COALESCE(o.summary_subtotal_retail, o.summary_subtotal) AS summary_subtotal_retail,
COALESCE(o.summary_discount_subtotal, 0) AS summary_discount_subtotal,
COALESCE(o.summary_shipping, 0) AS summary_shipping,
COALESCE(o.summary_shipping_rush, 0) AS summary_shipping_rush,
COALESCE(o.ship_method_cost, 0) AS ship_method_cost,
COALESCE(o.summary_points, 0) AS summary_points,
COALESCE(c.total_cogs, 0) AS total_cogs,
COALESCE(p.points_redeemed, 0) AS points_redeemed
FROM _order o
LEFT JOIN (
SELECT order_id, SUM(cogs_amount) AS total_cogs
FROM report_sales_data
WHERE action IN (1,2,3)
GROUP BY order_id
) c ON c.order_id = o.order_id
LEFT JOIN (
SELECT order_id, SUM(discount_amount_subtotal) AS points_redeemed
FROM order_discounts
WHERE discount_type = 20 AND discount_active = 1
GROUP BY order_id
) p ON p.order_id = o.order_id
WHERE o.summary_total > 0
AND o.order_status >= 20
AND o.ship_method_selected <> 'holdit'
AND o.ship_country = ?
AND o.date_placed >= ? AND o.date_placed < ?
${promoExistsClause}
`;
const [orders] = await connection.execute(ordersQuery, params);
if (release) {
release();
release = null;
}
let totalSubtotal = 0;
let totalProductDiscount = 0;
let totalCogs = 0;
let totalPointsAwarded = 0;
let totalPointsRedeemedDollars = 0;
for (const o of orders) {
totalSubtotal += Number(o.summary_subtotal) || 0;
totalProductDiscount += Number(o.summary_discount_subtotal) || 0;
totalCogs += Number(o.total_cogs) || 0;
totalPointsAwarded += Number(o.summary_points) || 0;
totalPointsRedeemedDollars += Number(o.points_redeemed) || 0;
}
const productDiscountRate = totalSubtotal > 0 ? totalProductDiscount / totalSubtotal : 0;
const overallCogsPercentage = totalSubtotal > 0 ? totalCogs / totalSubtotal : 0;
const pointsPerDollar = config.points.pointsPerDollar != null
? config.points.pointsPerDollar
: (totalSubtotal > 0 ? totalPointsAwarded / totalSubtotal : 0);
const pointDollarValue = config.points.pointDollarValue || DEFAULT_POINT_DOLLAR_VALUE;
let redemptionRate;
if (config.points.redemptionRate != null) {
redemptionRate = config.points.redemptionRate;
} else if (totalPointsAwarded > 0 && pointDollarValue > 0) {
const totalRedeemedPoints = totalPointsRedeemedDollars / pointDollarValue;
redemptionRate = Math.min(1, totalRedeemedPoints / totalPointsAwarded);
} else {
redemptionRate = 0;
}
const derived = {
overallCogsPercentage,
pointsPerDollar,
redemptionRate,
pointDollarValue,
};
const buckets = new Map();
for (const range of RANGE_DEFINITIONS) {
buckets.set(range.key, emptyBucketAccumulator(range));
}
let grandTotalProfit = 0;
let grandTotalRevenue = 0;
for (const order of orders) {
const sim = simulateOrder(order, config, derived);
const bucketKey = bucketKeyFor(sim.orderValue);
const bucket = buckets.get(bucketKey);
accumulate(bucket, sim);
grandTotalProfit += sim.profit;
grandTotalRevenue += sim.totalRevenue;
}
const totalOrders = orders.length;
const bucketResults = RANGE_DEFINITIONS.map((range) =>
finalizeBucket(buckets.get(range.key), totalOrders)
);
const weightedProfitAmount = totalOrders > 0 ? grandTotalProfit / totalOrders : 0;
const weightedProfitPercent = grandTotalRevenue > 0 ? grandTotalProfit / grandTotalRevenue : 0;
res.json({
dateRange: {
start: startDt.toISO(),
end: endDt.toISO()
},
totals: {
orders: totalOrders,
subtotal: totalSubtotal,
productDiscountRate,
pointsPerDollar,
redemptionRate,
pointDollarValue,
weightedProfitAmount,
weightedProfitPercent,
overallCogsPercentage: cogsCalculationMode === 'average' ? overallCogsPercentage : undefined
},
buckets: bucketResults
});
} catch (error) {
if (release) {
try {
release();
} catch (releaseError) {
console.error('Failed to release connection after error:', releaseError);
}
} else if (connection) {
try {
connection.destroy();
} catch (destroyError) {
console.error('Failed to destroy connection after error:', destroyError);
}
}
console.error('Error running discount simulation:', error);
res.status(500).json({ error: 'Failed to run discount simulation' });
}
});
export default router;
inventory-server/dashboard/acot-server/routes/employee-metrics.js
-687
View File
@@ -1,687 +0,0 @@
import express from 'express';
import { DateTime } from 'luxon';
import { getDbConnection, getPoolStatus } from '../db/connection.js';
import {
getTimeRangeConditions,
customRange,
previousRange,
parseMySql,
BUSINESS_TZ,
} from '../../../shared/business-time/index.js';
const router = express.Router();
const TIMEZONE = 'America/New_York';
// timeclock.TimeStamp stores Central wall-clock literals; the driver returns
// them as strings (dateStrings: true). Parse explicitly in business time.
const punchTime = (punch) => {
const dt = parseMySql(punch.TimeStamp);
return dt ? dt.toJSDate() : new Date(NaN);
};
// Punch types from the database
const PUNCH_TYPES = {
OUT: 0,
IN: 1,
BREAK_START: 2,
BREAK_END: 3,
};
// Standard hours for FTE calculation (40 hours per week)
const STANDARD_WEEKLY_HOURS = 40;
/**
* Calculate working hours from timeclock entries
* Groups punches by employee and date, pairs in/out punches
* Returns both total hours (with breaks, for FTE) and productive hours (without breaks, for productivity)
*/
function calculateHoursFromPunches(punches) {
// Group by employee
const byEmployee = new Map();
punches.forEach(punch => {
if (!byEmployee.has(punch.EmployeeID)) {
byEmployee.set(punch.EmployeeID, []);
}
byEmployee.get(punch.EmployeeID).push(punch);
});
const employeeHours = [];
let totalHours = 0;
let totalBreakHours = 0;
byEmployee.forEach((employeePunches, employeeId) => {
// Sort by timestamp
employeePunches.sort((a, b) => punchTime(a) - punchTime(b));
let hours = 0;
let breakHours = 0;
let currentIn = null;
let breakStart = null;
employeePunches.forEach(punch => {
const punchAt = punchTime(punch);
switch (punch.PunchType) {
case PUNCH_TYPES.IN:
currentIn = punchAt;
break;
case PUNCH_TYPES.OUT:
if (currentIn) {
hours += (punchAt - currentIn) / (1000 * 60 * 60); // Convert ms to hours
currentIn = null;
}
break;
case PUNCH_TYPES.BREAK_START:
breakStart = punchAt;
break;
case PUNCH_TYPES.BREAK_END:
if (breakStart) {
breakHours += (punchAt - breakStart) / (1000 * 60 * 60);
breakStart = null;
}
break;
}
});
totalHours += hours;
totalBreakHours += breakHours;
employeeHours.push({
employeeId,
hours,
breakHours,
productiveHours: hours - breakHours,
});
});
return {
employeeHours,
totalHours,
totalBreakHours,
totalProductiveHours: totalHours - totalBreakHours
};
}
/**
* Calculate FTE (Full Time Equivalents) for a period
* @param {number} totalHours - Total hours worked
* @param {Date} startDate - Period start
* @param {Date} endDate - Period end
*/
function calculateFTE(totalHours, startDate, endDate) {
const start = new Date(startDate);
const end = new Date(endDate);
const days = Math.max(1, (end - start) / (1000 * 60 * 60 * 24));
const weeks = days / 7;
const expectedHours = weeks * STANDARD_WEEKLY_HOURS;
return expectedHours > 0 ? totalHours / expectedHours : 0;
}
// Main employee metrics endpoint
router.get('/', async (req, res) => {
const startTime = Date.now();
console.log(`[EMPLOYEE-METRICS] Starting request for timeRange: ${req.query.timeRange}`);
const timeoutPromise = new Promise((_, reject) => {
setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
});
try {
const mainOperation = async () => {
const { timeRange, startDate, endDate } = req.query;
console.log(`[EMPLOYEE-METRICS] Getting DB connection...`);
const { connection, release } = await getDbConnection();
console.log(`[EMPLOYEE-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
// Adapt where clause for timeclock table (uses TimeStamp instead of date_placed)
const timeclockWhere = whereClause.replace(/date_placed/g, 'tc.TimeStamp');
// Query for timeclock data with employee names
const timeclockQuery = `
SELECT
tc.EmployeeID,
tc.TimeStamp,
tc.PunchType,
e.firstname,
e.lastname
FROM timeclock tc
LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
WHERE ${timeclockWhere}
AND e.hidden = 0
AND e.disabled = 0
ORDER BY tc.EmployeeID, tc.TimeStamp
`;
const [timeclockRows] = await connection.execute(timeclockQuery, params);
// Calculate hours (includes both total hours for FTE and productive hours for productivity)
const { employeeHours, totalHours, totalBreakHours, totalProductiveHours } = calculateHoursFromPunches(timeclockRows);
// Get employee names for the results
const employeeNames = new Map();
timeclockRows.forEach(row => {
if (!employeeNames.has(row.EmployeeID)) {
employeeNames.set(row.EmployeeID, {
firstname: row.firstname || '',
lastname: row.lastname || '',
});
}
});
// Enrich employee hours with names
const enrichedEmployeeHours = employeeHours.map(eh => ({
...eh,
name: employeeNames.has(eh.employeeId)
? `${employeeNames.get(eh.employeeId).firstname} ${employeeNames.get(eh.employeeId).lastname}`.trim()
: `Employee ${eh.employeeId}`,
})).sort((a, b) => b.hours - a.hours);
// Query for picking tickets - using subquery to avoid duplication from bucket join
// Ship-together orders: only count main orders (is_sub = 0 or NULL), not sub-orders
const pickingWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
// First get picking ticket stats without the bucket join (to avoid duplication)
const pickingStatsQuery = `
SELECT
pt.createdby as employeeId,
e.firstname,
e.lastname,
COUNT(DISTINCT pt.pickingid) as ticketCount,
SUM(pt.totalpieces_picked) as piecesPicked,
SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds,
AVG(NULLIF(pt.picking_speed, 0)) as avgPickingSpeed
FROM picking_ticket pt
LEFT JOIN employees e ON pt.createdby = e.employeeid
WHERE ${pickingWhere}
AND pt.closeddate IS NOT NULL
GROUP BY pt.createdby, e.firstname, e.lastname
`;
// Separate query for order counts (needs bucket join for ship-together handling)
const orderCountQuery = `
SELECT
pt.createdby as employeeId,
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${pickingWhere}
AND pt.closeddate IS NOT NULL
GROUP BY pt.createdby
`;
const [[pickingStatsRows], [orderCountRows]] = await Promise.all([
connection.execute(pickingStatsQuery, params),
connection.execute(orderCountQuery, params)
]);
// Merge the results
const orderCountMap = new Map();
orderCountRows.forEach(row => {
orderCountMap.set(row.employeeId, parseInt(row.ordersPicked || 0));
});
// Aggregate picking totals
let totalOrdersPicked = 0;
let totalPiecesPicked = 0;
let totalTickets = 0;
let totalPickingTimeSeconds = 0;
let pickingSpeedSum = 0;
let pickingSpeedCount = 0;
const pickingByEmployee = pickingStatsRows.map(row => {
const ordersPicked = orderCountMap.get(row.employeeId) || 0;
totalOrdersPicked += ordersPicked;
totalPiecesPicked += parseInt(row.piecesPicked || 0);
totalTickets += parseInt(row.ticketCount || 0);
totalPickingTimeSeconds += parseInt(row.pickingTimeSeconds || 0);
if (row.avgPickingSpeed && row.avgPickingSpeed > 0) {
pickingSpeedSum += parseFloat(row.avgPickingSpeed);
pickingSpeedCount++;
}
const empPickingHours = parseInt(row.pickingTimeSeconds || 0) / 3600;
return {
employeeId: row.employeeId,
name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeId}`,
ticketCount: parseInt(row.ticketCount || 0),
ordersPicked,
piecesPicked: parseInt(row.piecesPicked || 0),
pickingHours: empPickingHours,
avgPickingSpeed: row.avgPickingSpeed ? parseFloat(row.avgPickingSpeed) : null,
};
});
const totalPickingHours = totalPickingTimeSeconds / 3600;
const avgPickingSpeed = pickingSpeedCount > 0 ? pickingSpeedSum / pickingSpeedCount : 0;
// Query for shipped orders - totals
// Ship-together orders: only count main orders (order_type != 8 for sub-orders, or use parent tracking)
const shippingWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
const shippingQuery = `
SELECT
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
WHERE ${shippingWhere}
AND o.order_status IN (100, 92)
`;
const [shippingRows] = await connection.execute(shippingQuery, params);
const shipping = shippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
// Query for shipped orders by employee
const shippingByEmployeeQuery = `
SELECT
e.employeeid,
e.firstname,
e.lastname,
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
JOIN employees e ON o.stats_cid_shipped = e.cid
WHERE ${shippingWhere}
AND o.order_status IN (100, 92)
AND e.hidden = 0
AND e.disabled = 0
GROUP BY e.employeeid, e.firstname, e.lastname
ORDER BY ordersShipped DESC
`;
const [shippingByEmployeeRows] = await connection.execute(shippingByEmployeeQuery, params);
const shippingByEmployee = shippingByEmployeeRows.map(row => ({
employeeId: row.employeeid,
name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeid}`,
ordersShipped: parseInt(row.ordersShipped || 0),
piecesShipped: parseInt(row.piecesShipped || 0),
}));
// Calculate period dates for FTE calculation
let periodStart, periodEnd;
if (dateRange?.start) {
periodStart = new Date(dateRange.start);
} else if (params[0]) {
periodStart = new Date(params[0]);
} else {
periodStart = new Date();
periodStart.setDate(periodStart.getDate() - 30);
}
if (dateRange?.end) {
periodEnd = new Date(dateRange.end);
} else if (params[1]) {
periodEnd = new Date(params[1]);
} else {
periodEnd = new Date();
}
const fte = calculateFTE(totalHours, periodStart, periodEnd);
const activeEmployees = enrichedEmployeeHours.filter(e => e.hours > 0).length;
// Calculate weeks in period for weekly averages
const periodDays = Math.max(1, (periodEnd - periodStart) / (1000 * 60 * 60 * 24));
const weeksInPeriod = periodDays / 7;
// Get daily trend data for hours.
// TimeStamp stores Central wall-clock literals; business date = the
// Central calendar date, so DATE_FORMAT(col) needs no hour shift.
const trendWhere = whereClause.replace(/date_placed/g, 'tc.TimeStamp');
const trendQuery = `
SELECT
DATE_FORMAT(tc.TimeStamp, '%Y-%m-%d') as date,
tc.EmployeeID,
tc.TimeStamp,
tc.PunchType
FROM timeclock tc
LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
WHERE ${trendWhere}
AND e.hidden = 0
AND e.disabled = 0
ORDER BY date, tc.EmployeeID, tc.TimeStamp
`;
const [trendRows] = await connection.execute(trendQuery, params);
// Get daily picking data for trend
// Ship-together orders: only count main orders (is_sub = 0 or NULL)
// createddate is a Central wall-clock literal — DATE_FORMAT(col) is the
// business date with no hour shift.
const pickingTrendWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
const pickingTrendQuery = `
SELECT
DATE_FORMAT(pt.createddate, '%Y-%m-%d') as date,
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked,
COALESCE(SUM(pt.totalpieces_picked), 0) as piecesPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${pickingTrendWhere}
AND pt.closeddate IS NOT NULL
GROUP BY DATE_FORMAT(pt.createddate, '%Y-%m-%d')
ORDER BY date
`;
const [pickingTrendRows] = await connection.execute(pickingTrendQuery, params);
// Create a map of picking data by date
const pickingByDate = new Map();
pickingTrendRows.forEach(row => {
// Date is already a string in YYYY-MM-DD format from DATE_FORMAT
const date = String(row.date);
pickingByDate.set(date, {
ordersPicked: parseInt(row.ordersPicked || 0),
piecesPicked: parseInt(row.piecesPicked || 0),
});
});
// Group timeclock by date for trend
const byDate = new Map();
trendRows.forEach(row => {
// Date is already a string in YYYY-MM-DD format from DATE_FORMAT
const date = String(row.date);
if (!byDate.has(date)) {
byDate.set(date, []);
}
byDate.get(date).push(row);
});
// Generate all business dates in the half-open period [start, end) for
// complete trend data. Business date = Chicago calendar date.
const allDatesInRange = [];
const startDt = DateTime.fromJSDate(periodStart).setZone(BUSINESS_TZ).startOf('day');
const endDt = DateTime.fromJSDate(periodEnd).setZone(BUSINESS_TZ);
let currentDt = startDt;
while (currentDt < endDt) {
allDatesInRange.push(currentDt.toFormat('yyyy-MM-dd'));
currentDt = currentDt.plus({ days: 1 });
}
// Build trend data for all dates in range, filling zeros for missing days
const trend = allDatesInRange.map(date => {
const punches = byDate.get(date) || [];
const { totalHours: dayHours, employeeHours: dayEmployeeHours } = calculateHoursFromPunches(punches);
const picking = pickingByDate.get(date) || { ordersPicked: 0, piecesPicked: 0 };
// The string is already the business date; the zone here is only for
// the display timestamp label (office time).
const dateDt = DateTime.fromFormat(date, 'yyyy-MM-dd', { zone: TIMEZONE });
return {
date,
timestamp: dateDt.toISO(),
hours: dayHours,
activeEmployees: dayEmployeeHours.filter(e => e.hours > 0).length,
ordersPicked: picking.ordersPicked,
piecesPicked: picking.piecesPicked,
};
});
// Get previous period data for comparison
const previousRange = getPreviousPeriodRange(timeRange, startDate, endDate);
let comparison = null;
let previousTotals = null;
if (previousRange) {
const prevTimeclockWhere = previousRange.whereClause.replace(/date_placed/g, 'tc.TimeStamp');
const [prevTimeclockRows] = await connection.execute(
`SELECT tc.EmployeeID, tc.TimeStamp, tc.PunchType
FROM timeclock tc
LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
WHERE ${prevTimeclockWhere}
AND e.hidden = 0
AND e.disabled = 0
ORDER BY tc.EmployeeID, tc.TimeStamp`,
previousRange.params
);
const {
totalHours: prevTotalHours,
totalProductiveHours: prevProductiveHours,
employeeHours: prevEmployeeHours
} = calculateHoursFromPunches(prevTimeclockRows);
const prevActiveEmployees = prevEmployeeHours.filter(e => e.hours > 0).length;
// Previous picking data (ship-together fix applied)
// Use separate queries to avoid duplication from bucket join
const prevPickingWhere = previousRange.whereClause.replace(/date_placed/g, 'pt.createddate');
const [[prevPickingStatsRows], [prevOrderCountRows]] = await Promise.all([
connection.execute(
`SELECT
SUM(pt.totalpieces_picked) as piecesPicked,
SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds
FROM picking_ticket pt
WHERE ${prevPickingWhere}
AND pt.closeddate IS NOT NULL`,
previousRange.params
),
connection.execute(
`SELECT
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${prevPickingWhere}
AND pt.closeddate IS NOT NULL`,
previousRange.params
)
]);
const prevPickingStats = prevPickingStatsRows[0] || { piecesPicked: 0, pickingTimeSeconds: 0 };
const prevOrderCount = prevOrderCountRows[0] || { ordersPicked: 0 };
const prevPicking = {
ordersPicked: parseInt(prevOrderCount.ordersPicked || 0),
piecesPicked: parseInt(prevPickingStats.piecesPicked || 0),
pickingTimeSeconds: parseInt(prevPickingStats.pickingTimeSeconds || 0)
};
const prevPickingHours = prevPicking.pickingTimeSeconds / 3600;
// Previous shipping data
const prevShippingWhere = previousRange.whereClause.replace(/date_placed/g, 'o.date_shipped');
const [prevShippingRows] = await connection.execute(
`SELECT
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
WHERE ${prevShippingWhere}
AND o.order_status IN (100, 92)`,
previousRange.params
);
const prevShipping = prevShippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
// Calculate previous period FTE and productivity
const prevFte = calculateFTE(prevTotalHours, previousRange.start || periodStart, previousRange.end || periodEnd);
const prevOrdersPerHour = prevProductiveHours > 0 ? parseInt(prevPicking.ordersPicked || 0) / prevProductiveHours : 0;
const prevPiecesPerHour = prevProductiveHours > 0 ? parseInt(prevPicking.piecesPicked || 0) / prevProductiveHours : 0;
previousTotals = {
hours: prevTotalHours,
productiveHours: prevProductiveHours,
activeEmployees: prevActiveEmployees,
fte: prevFte,
ordersPicked: parseInt(prevPicking.ordersPicked || 0),
piecesPicked: parseInt(prevPicking.piecesPicked || 0),
pickingHours: prevPickingHours,
ordersShipped: parseInt(prevShipping.ordersShipped || 0),
piecesShipped: parseInt(prevShipping.piecesShipped || 0),
ordersPerHour: prevOrdersPerHour,
piecesPerHour: prevPiecesPerHour,
};
// Calculate productivity metrics for comparison
const currentOrdersPerHour = totalProductiveHours > 0 ? totalOrdersPicked / totalProductiveHours : 0;
const currentPiecesPerHour = totalProductiveHours > 0 ? totalPiecesPicked / totalProductiveHours : 0;
comparison = {
hours: calculateComparison(totalHours, prevTotalHours),
productiveHours: calculateComparison(totalProductiveHours, prevProductiveHours),
activeEmployees: calculateComparison(activeEmployees, prevActiveEmployees),
fte: calculateComparison(fte, prevFte),
ordersPicked: calculateComparison(totalOrdersPicked, parseInt(prevPicking.ordersPicked || 0)),
piecesPicked: calculateComparison(totalPiecesPicked, parseInt(prevPicking.piecesPicked || 0)),
ordersShipped: calculateComparison(parseInt(shipping.ordersShipped || 0), parseInt(prevShipping.ordersShipped || 0)),
piecesShipped: calculateComparison(parseInt(shipping.piecesShipped || 0), parseInt(prevShipping.piecesShipped || 0)),
ordersPerHour: calculateComparison(currentOrdersPerHour, prevOrdersPerHour),
piecesPerHour: calculateComparison(currentPiecesPerHour, prevPiecesPerHour),
};
}
// Calculate efficiency (picking time vs productive hours)
const pickingEfficiency = totalProductiveHours > 0 ? (totalPickingHours / totalProductiveHours) * 100 : 0;
const response = {
dateRange,
totals: {
// Time metrics
hours: totalHours,
breakHours: totalBreakHours,
productiveHours: totalProductiveHours,
pickingHours: totalPickingHours,
// Employee metrics
activeEmployees,
fte,
weeksInPeriod,
// Picking metrics
ordersPicked: totalOrdersPicked,
piecesPicked: totalPiecesPicked,
ticketCount: totalTickets,
// Shipping metrics
ordersShipped: parseInt(shipping.ordersShipped || 0),
piecesShipped: parseInt(shipping.piecesShipped || 0),
// Calculated metrics - standardized to weekly
hoursPerWeek: weeksInPeriod > 0 ? totalHours / weeksInPeriod : 0,
hoursPerEmployeePerWeek: activeEmployees > 0 && weeksInPeriod > 0
? (totalHours / activeEmployees) / weeksInPeriod
: 0,
// Productivity metrics (uses productive hours - excludes breaks)
ordersPerHour: totalProductiveHours > 0 ? totalOrdersPicked / totalProductiveHours : 0,
piecesPerHour: totalProductiveHours > 0 ? totalPiecesPicked / totalProductiveHours : 0,
// Picking speed from database (more accurate, only counts picking time)
avgPickingSpeed,
// Efficiency metrics
pickingEfficiency,
},
previousTotals,
comparison,
byEmployee: {
hours: enrichedEmployeeHours,
picking: pickingByEmployee,
shipping: shippingByEmployee,
},
trend,
};
return { response, release };
};
let result;
try {
result = await Promise.race([mainOperation(), timeoutPromise]);
} catch (error) {
if (error.message.includes('timeout')) {
console.log(`[EMPLOYEE-METRICS] Request timed out in ${Date.now() - startTime}ms`);
throw error;
}
throw error;
}
const { response, release } = result;
if (release) release();
console.log(`[EMPLOYEE-METRICS] Request completed in ${Date.now() - startTime}ms`);
res.json(response);
} catch (error) {
console.error('Error in /employee-metrics:', error);
console.log(`[EMPLOYEE-METRICS] Request failed in ${Date.now() - startTime}ms`);
res.status(500).json({ error: error.message });
}
});
// Health check
router.get('/health', async (req, res) => {
try {
const { connection, release } = await getDbConnection();
await connection.execute('SELECT 1 as test');
release();
res.json({
status: 'healthy',
timestamp: new Date().toISOString(),
pool: getPoolStatus(),
});
} catch (error) {
res.status(500).json({
status: 'unhealthy',
timestamp: new Date().toISOString(),
error: error.message,
});
}
});
// Helper functions
function calculateComparison(currentValue, previousValue) {
if (typeof previousValue !== 'number') {
return { absolute: null, percentage: null };
}
const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
const percentage =
absolute !== null && previousValue !== 0
? (absolute / Math.abs(previousValue)) * 100
: null;
return { absolute, percentage };
}
function getPreviousPeriodRange(timeRange, startDate, endDate) {
if (timeRange && timeRange !== 'custom') {
const prevTimeRange = getPreviousTimeRange(timeRange);
if (!prevTimeRange || prevTimeRange === timeRange) {
return null;
}
return getTimeRangeConditions(prevTimeRange);
}
const hasCustomDates = (timeRange === 'custom' || !timeRange) && startDate && endDate;
if (!hasCustomDates) {
return null;
}
try {
const prev = previousRange(customRange(startDate, endDate));
if (!prev) return null;
return getTimeRangeConditions('custom', prev.start.toISO(), prev.end.toISO());
} catch {
return null;
}
}
function getPreviousTimeRange(timeRange) {
const map = {
today: 'yesterday',
thisWeek: 'lastWeek',
thisMonth: 'lastMonth',
last7days: 'previous7days',
last30days: 'previous30days',
last90days: 'previous90days',
yesterday: 'twoDaysAgo'
};
return map[timeRange] || timeRange;
}
export default router;
inventory-server/dashboard/acot-server/routes/events.js
+302 -1192
View File
File diff suppressed because it is too large Load Diff
inventory-server/dashboard/acot-server/routes/operations-metrics.js
-479
View File
@@ -1,479 +0,0 @@
import express from 'express';
import { DateTime } from 'luxon';
import { getDbConnection, getPoolStatus } from '../db/connection.js';
import {
getTimeRangeConditions,
customRange,
previousRange,
BUSINESS_TZ,
} from '../../../shared/business-time/index.js';
const router = express.Router();
const TIMEZONE = 'America/New_York';
// Main operations metrics endpoint - focused on picking and shipping
router.get('/', async (req, res) => {
const startTime = Date.now();
console.log(`[OPERATIONS-METRICS] Starting request for timeRange: ${req.query.timeRange}`);
const timeoutPromise = new Promise((_, reject) => {
setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
});
try {
const mainOperation = async () => {
const { timeRange, startDate, endDate } = req.query;
console.log(`[OPERATIONS-METRICS] Getting DB connection...`);
const { connection, release } = await getDbConnection();
console.log(`[OPERATIONS-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
try {
const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
// Query for picking tickets - using subquery to avoid duplication from bucket join
// Ship-together orders: only count main orders (is_sub = 0 or NULL), not sub-orders
const pickingWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
// First get picking ticket stats without the bucket join (to avoid duplication)
const pickingStatsQuery = `
SELECT
pt.createdby as employeeId,
e.firstname,
e.lastname,
COUNT(DISTINCT pt.pickingid) as ticketCount,
SUM(pt.totalpieces_picked) as piecesPicked,
SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds,
AVG(NULLIF(pt.picking_speed, 0)) as avgPickingSpeed
FROM picking_ticket pt
LEFT JOIN employees e ON pt.createdby = e.employeeid
WHERE ${pickingWhere}
AND pt.closeddate IS NOT NULL
GROUP BY pt.createdby, e.firstname, e.lastname
`;
// Separate query for order counts (needs bucket join for ship-together handling)
const orderCountQuery = `
SELECT
pt.createdby as employeeId,
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${pickingWhere}
AND pt.closeddate IS NOT NULL
GROUP BY pt.createdby
`;
const [[pickingStatsRows], [orderCountRows]] = await Promise.all([
connection.execute(pickingStatsQuery, params),
connection.execute(orderCountQuery, params)
]);
// Merge the results
const orderCountMap = new Map();
orderCountRows.forEach(row => {
orderCountMap.set(row.employeeId, parseInt(row.ordersPicked || 0));
});
// Aggregate picking totals
let totalOrdersPicked = 0;
let totalPiecesPicked = 0;
let totalTickets = 0;
let totalPickingTimeSeconds = 0;
let pickingSpeedSum = 0;
let pickingSpeedCount = 0;
const pickingByEmployee = pickingStatsRows.map(row => {
const ordersPicked = orderCountMap.get(row.employeeId) || 0;
totalOrdersPicked += ordersPicked;
totalPiecesPicked += parseInt(row.piecesPicked || 0);
totalTickets += parseInt(row.ticketCount || 0);
totalPickingTimeSeconds += parseInt(row.pickingTimeSeconds || 0);
if (row.avgPickingSpeed && row.avgPickingSpeed > 0) {
pickingSpeedSum += parseFloat(row.avgPickingSpeed);
pickingSpeedCount++;
}
const empPickingHours = parseInt(row.pickingTimeSeconds || 0) / 3600;
return {
employeeId: row.employeeId,
name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeId}`,
ticketCount: parseInt(row.ticketCount || 0),
ordersPicked,
piecesPicked: parseInt(row.piecesPicked || 0),
pickingHours: empPickingHours,
avgPickingSpeed: row.avgPickingSpeed ? parseFloat(row.avgPickingSpeed) : null,
};
});
const totalPickingHours = totalPickingTimeSeconds / 3600;
const avgPickingSpeed = pickingSpeedCount > 0 ? pickingSpeedSum / pickingSpeedCount : 0;
// Query for shipped orders - totals
// Ship-together orders: only count main orders (order_type != 8 for sub-orders)
const shippingWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
const shippingQuery = `
SELECT
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
WHERE ${shippingWhere}
AND o.order_status IN (100, 92)
`;
const [shippingRows] = await connection.execute(shippingQuery, params);
const shipping = shippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
// Query for shipped orders by employee
const shippingByEmployeeQuery = `
SELECT
e.employeeid,
e.firstname,
e.lastname,
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
JOIN employees e ON o.stats_cid_shipped = e.cid
WHERE ${shippingWhere}
AND o.order_status IN (100, 92)
AND e.hidden = 0
AND e.disabled = 0
GROUP BY e.employeeid, e.firstname, e.lastname
ORDER BY ordersShipped DESC
`;
const [shippingByEmployeeRows] = await connection.execute(shippingByEmployeeQuery, params);
const shippingByEmployee = shippingByEmployeeRows.map(row => ({
employeeId: row.employeeid,
name: `${row.firstname || ''} ${row.lastname || ''}`.trim() || `Employee ${row.employeeid}`,
ordersShipped: parseInt(row.ordersShipped || 0),
piecesShipped: parseInt(row.piecesShipped || 0),
}));
// Calculate period dates
let periodStart, periodEnd;
if (dateRange?.start) {
periodStart = new Date(dateRange.start);
} else if (params[0]) {
periodStart = new Date(params[0]);
} else {
periodStart = new Date();
periodStart.setDate(periodStart.getDate() - 30);
}
if (dateRange?.end) {
periodEnd = new Date(dateRange.end);
} else if (params[1]) {
periodEnd = new Date(params[1]);
} else {
periodEnd = new Date();
}
// Calculate productivity (orders/pieces per picking hour)
const ordersPerHour = totalPickingHours > 0 ? totalOrdersPicked / totalPickingHours : 0;
const piecesPerHour = totalPickingHours > 0 ? totalPiecesPicked / totalPickingHours : 0;
// Get daily trend data for picking.
// Columns store Central wall-clock literals, and the business date IS
// the Central calendar date — so DATE(col) is the business date with
// no hour shift.
const pickingTrendWhere = whereClause.replace(/date_placed/g, 'pt.createddate');
const pickingTrendQuery = `
SELECT
pt_agg.date,
COALESCE(order_counts.ordersPicked, 0) as ordersPicked,
pt_agg.piecesPicked
FROM (
SELECT
DATE_FORMAT(pt.createddate, '%Y-%m-%d') as date,
COALESCE(SUM(pt.totalpieces_picked), 0) as piecesPicked
FROM picking_ticket pt
WHERE ${pickingTrendWhere}
AND pt.closeddate IS NOT NULL
GROUP BY DATE_FORMAT(pt.createddate, '%Y-%m-%d')
) pt_agg
LEFT JOIN (
SELECT
DATE_FORMAT(pt.createddate, '%Y-%m-%d') as date,
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${pickingTrendWhere}
AND pt.closeddate IS NOT NULL
GROUP BY DATE_FORMAT(pt.createddate, '%Y-%m-%d')
) order_counts ON pt_agg.date = order_counts.date
ORDER BY pt_agg.date
`;
// Get shipping trend data
const shippingTrendWhere = whereClause.replace(/date_placed/g, 'o.date_shipped');
const shippingTrendQuery = `
SELECT
DATE_FORMAT(o.date_shipped, '%Y-%m-%d') as date,
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
WHERE ${shippingTrendWhere}
AND o.order_status IN (100, 92)
GROUP BY DATE_FORMAT(o.date_shipped, '%Y-%m-%d')
ORDER BY date
`;
const [[pickingTrendRows], [shippingTrendRows]] = await Promise.all([
connection.execute(pickingTrendQuery, [...params, ...params]),
connection.execute(shippingTrendQuery, params),
]);
// Create maps for trend data
const pickingByDate = new Map();
pickingTrendRows.forEach(row => {
const date = String(row.date);
pickingByDate.set(date, {
ordersPicked: parseInt(row.ordersPicked || 0),
piecesPicked: parseInt(row.piecesPicked || 0),
});
});
const shippingByDate = new Map();
shippingTrendRows.forEach(row => {
const date = String(row.date);
shippingByDate.set(date, {
ordersShipped: parseInt(row.ordersShipped || 0),
piecesShipped: parseInt(row.piecesShipped || 0),
});
});
// Generate all business dates in the half-open period [start, end) for
// complete trend data. Business date = Chicago calendar date.
const allDatesInRange = [];
const startDt = DateTime.fromJSDate(periodStart).setZone(BUSINESS_TZ).startOf('day');
const endDt = DateTime.fromJSDate(periodEnd).setZone(BUSINESS_TZ);
let currentDt = startDt;
while (currentDt < endDt) {
allDatesInRange.push(currentDt.toFormat('yyyy-MM-dd'));
currentDt = currentDt.plus({ days: 1 });
}
// Build trend data for all dates in range
const trend = allDatesInRange.map(date => {
const picking = pickingByDate.get(date) || { ordersPicked: 0, piecesPicked: 0 };
const shippingData = shippingByDate.get(date) || { ordersShipped: 0, piecesShipped: 0 };
// The string is already the business date; the zone here is only for
// the display timestamp label (office time).
const dateDt = DateTime.fromFormat(date, 'yyyy-MM-dd', { zone: TIMEZONE });
return {
date,
timestamp: dateDt.toISO(),
ordersPicked: picking.ordersPicked,
piecesPicked: picking.piecesPicked,
ordersShipped: shippingData.ordersShipped,
piecesShipped: shippingData.piecesShipped,
};
});
// Get previous period data for comparison
const previousRange = getPreviousPeriodRange(timeRange, startDate, endDate);
let comparison = null;
let previousTotals = null;
if (previousRange) {
// Previous picking data
const prevPickingWhere = previousRange.whereClause.replace(/date_placed/g, 'pt.createddate');
const [[prevPickingStatsRows], [prevOrderCountRows]] = await Promise.all([
connection.execute(
`SELECT
SUM(pt.totalpieces_picked) as piecesPicked,
SUM(TIMESTAMPDIFF(SECOND, pt.createddate, pt.closeddate)) as pickingTimeSeconds
FROM picking_ticket pt
WHERE ${prevPickingWhere}
AND pt.closeddate IS NOT NULL`,
previousRange.params
),
connection.execute(
`SELECT
COUNT(DISTINCT CASE WHEN ptb.is_sub = 0 OR ptb.is_sub IS NULL THEN ptb.orderid END) as ordersPicked
FROM picking_ticket pt
LEFT JOIN picking_ticket_buckets ptb ON pt.pickingid = ptb.pickingid
WHERE ${prevPickingWhere}
AND pt.closeddate IS NOT NULL`,
previousRange.params
)
]);
const prevPickingStats = prevPickingStatsRows[0] || { piecesPicked: 0, pickingTimeSeconds: 0 };
const prevOrderCount = prevOrderCountRows[0] || { ordersPicked: 0 };
const prevPicking = {
ordersPicked: parseInt(prevOrderCount.ordersPicked || 0),
piecesPicked: parseInt(prevPickingStats.piecesPicked || 0),
pickingTimeSeconds: parseInt(prevPickingStats.pickingTimeSeconds || 0)
};
const prevPickingHours = prevPicking.pickingTimeSeconds / 3600;
// Previous shipping data
const prevShippingWhere = previousRange.whereClause.replace(/date_placed/g, 'o.date_shipped');
const [prevShippingRows] = await connection.execute(
`SELECT
COUNT(DISTINCT CASE WHEN o.order_type != 8 OR o.order_type IS NULL THEN o.order_id END) as ordersShipped,
COALESCE(SUM(o.stats_prod_pieces), 0) as piecesShipped
FROM _order o
WHERE ${prevShippingWhere}
AND o.order_status IN (100, 92)`,
previousRange.params
);
const prevShipping = prevShippingRows[0] || { ordersShipped: 0, piecesShipped: 0 };
// Calculate previous productivity
const prevOrdersPerHour = prevPickingHours > 0 ? parseInt(prevPicking.ordersPicked || 0) / prevPickingHours : 0;
const prevPiecesPerHour = prevPickingHours > 0 ? parseInt(prevPicking.piecesPicked || 0) / prevPickingHours : 0;
previousTotals = {
ordersPicked: parseInt(prevPicking.ordersPicked || 0),
piecesPicked: parseInt(prevPicking.piecesPicked || 0),
pickingHours: prevPickingHours,
ordersShipped: parseInt(prevShipping.ordersShipped || 0),
piecesShipped: parseInt(prevShipping.piecesShipped || 0),
ordersPerHour: prevOrdersPerHour,
piecesPerHour: prevPiecesPerHour,
};
comparison = {
ordersPicked: calculateComparison(totalOrdersPicked, parseInt(prevPicking.ordersPicked || 0)),
piecesPicked: calculateComparison(totalPiecesPicked, parseInt(prevPicking.piecesPicked || 0)),
ordersShipped: calculateComparison(parseInt(shipping.ordersShipped || 0), parseInt(prevShipping.ordersShipped || 0)),
piecesShipped: calculateComparison(parseInt(shipping.piecesShipped || 0), parseInt(prevShipping.piecesShipped || 0)),
ordersPerHour: calculateComparison(ordersPerHour, prevOrdersPerHour),
piecesPerHour: calculateComparison(piecesPerHour, prevPiecesPerHour),
};
}
const response = {
dateRange,
totals: {
// Picking metrics
ordersPicked: totalOrdersPicked,
piecesPicked: totalPiecesPicked,
ticketCount: totalTickets,
pickingHours: totalPickingHours,
// Shipping metrics
ordersShipped: parseInt(shipping.ordersShipped || 0),
piecesShipped: parseInt(shipping.piecesShipped || 0),
// Productivity metrics
ordersPerHour,
piecesPerHour,
avgPickingSpeed,
},
previousTotals,
comparison,
byEmployee: {
picking: pickingByEmployee,
shipping: shippingByEmployee,
},
trend,
};
return response;
} finally {
// Always release the connection regardless of who wins Promise.race.
// If the timeout wins, this IIFE keeps running until MySQL responds; this
// finally ensures the connection still returns to the pool.
release();
}
};
const response = await Promise.race([mainOperation(), timeoutPromise]);
console.log(`[OPERATIONS-METRICS] Request completed in ${Date.now() - startTime}ms`);
res.json(response);
} catch (error) {
if (error.message.includes('timeout')) {
console.log(`[OPERATIONS-METRICS] Request timed out in ${Date.now() - startTime}ms`);
} else {
console.error('Error in /operations-metrics:', error);
}
console.log(`[OPERATIONS-METRICS] Request failed in ${Date.now() - startTime}ms`);
res.status(500).json({ error: error.message });
}
});
// Health check
router.get('/health', async (req, res) => {
try {
const { connection, release } = await getDbConnection();
await connection.execute('SELECT 1 as test');
release();
res.json({
status: 'healthy',
timestamp: new Date().toISOString(),
pool: getPoolStatus(),
});
} catch (error) {
res.status(500).json({
status: 'unhealthy',
timestamp: new Date().toISOString(),
error: error.message,
});
}
});
// Helper functions
function calculateComparison(currentValue, previousValue) {
if (typeof previousValue !== 'number') {
return { absolute: null, percentage: null };
}
const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
const percentage =
absolute !== null && previousValue !== 0
? (absolute / Math.abs(previousValue)) * 100
: null;
return { absolute, percentage };
}
function getPreviousPeriodRange(timeRange, startDate, endDate) {
if (timeRange && timeRange !== 'custom') {
const prevTimeRange = getPreviousTimeRange(timeRange);
if (!prevTimeRange || prevTimeRange === timeRange) {
return null;
}
return getTimeRangeConditions(prevTimeRange);
}
const hasCustomDates = (timeRange === 'custom' || !timeRange) && startDate && endDate;
if (!hasCustomDates) {
return null;
}
try {
const prev = previousRange(customRange(startDate, endDate));
if (!prev) return null;
return getTimeRangeConditions('custom', prev.start.toISO(), prev.end.toISO());
} catch {
return null;
}
}
function getPreviousTimeRange(timeRange) {
const map = {
today: 'yesterday',
thisWeek: 'lastWeek',
thisMonth: 'lastMonth',
last7days: 'previous7days',
last30days: 'previous30days',
last90days: 'previous90days',
yesterday: 'twoDaysAgo'
};
return map[timeRange] || timeRange;
}
export default router;
inventory-server/dashboard/acot-server/routes/payroll-metrics.js
-514
View File
@@ -1,514 +0,0 @@
import express from 'express';
import { DateTime } from 'luxon';
import { getDbConnection, getPoolStatus } from '../db/connection.js';
import { parseMySql, toMySqlBound } from '../../../shared/business-time/index.js';
const router = express.Router();
const TIMEZONE = 'America/New_York';
// timeclock.TimeStamp stores Central wall-clock literals; the driver returns
// them as strings (dateStrings: true). Parse in business time, compare/display
// in ET (pay-period boundaries are an ET display convention).
const punchDateTime = (punch) => {
const dt = parseMySql(punch.TimeStamp);
return dt ? dt.setZone(TIMEZONE) : null;
};
// Punch types from the database
const PUNCH_TYPES = {
OUT: 0,
IN: 1,
BREAK_START: 2,
BREAK_END: 3,
};
// Standard hours for overtime calculation (40 hours per week)
const STANDARD_WEEKLY_HOURS = 40;
// Reference pay period start date (January 25, 2026 is a Sunday, first day of a pay period)
const PAY_PERIOD_REFERENCE = DateTime.fromObject(
{ year: 2026, month: 1, day: 25 },
{ zone: TIMEZONE }
);
/**
* Calculate the pay period that contains a given date
* Pay periods are 14 days starting on Sunday
* @param {DateTime} date - The date to find the pay period for
* @returns {{ start: DateTime, end: DateTime, week1: { start: DateTime, end: DateTime }, week2: { start: DateTime, end: DateTime } }}
*/
function getPayPeriodForDate(date) {
const dt = DateTime.isDateTime(date) ? date : DateTime.fromJSDate(date, { zone: TIMEZONE });
// Calculate days since reference
const daysSinceReference = Math.floor(dt.diff(PAY_PERIOD_REFERENCE, 'days').days);
// Find which pay period this falls into (can be negative for dates before reference)
const payPeriodIndex = Math.floor(daysSinceReference / 14);
// Calculate the start of this pay period
const start = PAY_PERIOD_REFERENCE.plus({ days: payPeriodIndex * 14 }).startOf('day');
const end = start.plus({ days: 13 }).endOf('day');
// Week 1: Sunday through Saturday
const week1Start = start;
const week1End = start.plus({ days: 6 }).endOf('day');
// Week 2: Sunday through Saturday
const week2Start = start.plus({ days: 7 }).startOf('day');
const week2End = end;
return {
start,
end,
week1: { start: week1Start, end: week1End },
week2: { start: week2Start, end: week2End },
};
}
/**
* Get the current pay period
*/
function getCurrentPayPeriod() {
return getPayPeriodForDate(DateTime.now().setZone(TIMEZONE));
}
/**
* Navigate to previous or next pay period
* @param {DateTime} currentStart - Current pay period start
* @param {number} offset - Number of pay periods to move (negative for previous)
*/
function navigatePayPeriod(currentStart, offset) {
const newStart = currentStart.plus({ days: offset * 14 });
return getPayPeriodForDate(newStart);
}
/**
* Calculate working hours from timeclock entries, broken down by week
* @param {Array} punches - Timeclock punch entries
* @param {Object} payPeriod - Pay period with week boundaries
*/
function calculateHoursByWeek(punches, payPeriod) {
// Group by employee
const byEmployee = new Map();
punches.forEach(punch => {
if (!byEmployee.has(punch.EmployeeID)) {
byEmployee.set(punch.EmployeeID, {
employeeId: punch.EmployeeID,
firstname: punch.firstname || '',
lastname: punch.lastname || '',
punches: [],
});
}
byEmployee.get(punch.EmployeeID).punches.push(punch);
});
const employeeResults = [];
let totalHours = 0;
let totalBreakHours = 0;
let totalOvertimeHours = 0;
let totalRegularHours = 0;
let week1TotalHours = 0;
let week1TotalOvertime = 0;
let week2TotalHours = 0;
let week2TotalOvertime = 0;
byEmployee.forEach((employeeData) => {
// Sort punches by timestamp
employeeData.punches.sort((a, b) => (punchDateTime(a)?.toMillis() ?? 0) - (punchDateTime(b)?.toMillis() ?? 0));
// Calculate hours for each week
const week1Punches = employeeData.punches.filter(p => {
const dt = punchDateTime(p);
return dt && dt >= payPeriod.week1.start && dt <= payPeriod.week1.end;
});
const week2Punches = employeeData.punches.filter(p => {
const dt = punchDateTime(p);
return dt && dt >= payPeriod.week2.start && dt <= payPeriod.week2.end;
});
const week1Hours = calculateHoursFromPunches(week1Punches);
const week2Hours = calculateHoursFromPunches(week2Punches);
// Calculate overtime per week (anything over 40 hours)
const week1Overtime = Math.max(0, week1Hours.hours - STANDARD_WEEKLY_HOURS);
const week2Overtime = Math.max(0, week2Hours.hours - STANDARD_WEEKLY_HOURS);
const week1Regular = week1Hours.hours - week1Overtime;
const week2Regular = week2Hours.hours - week2Overtime;
const employeeTotal = week1Hours.hours + week2Hours.hours;
const employeeBreaks = week1Hours.breakHours + week2Hours.breakHours;
const employeeOvertime = week1Overtime + week2Overtime;
const employeeRegular = employeeTotal - employeeOvertime;
totalHours += employeeTotal;
totalBreakHours += employeeBreaks;
totalOvertimeHours += employeeOvertime;
totalRegularHours += employeeRegular;
week1TotalHours += week1Hours.hours;
week1TotalOvertime += week1Overtime;
week2TotalHours += week2Hours.hours;
week2TotalOvertime += week2Overtime;
employeeResults.push({
employeeId: employeeData.employeeId,
name: `${employeeData.firstname} ${employeeData.lastname}`.trim() || `Employee ${employeeData.employeeId}`,
week1Hours: week1Hours.hours,
week1BreakHours: week1Hours.breakHours,
week1Overtime,
week1Regular,
week2Hours: week2Hours.hours,
week2BreakHours: week2Hours.breakHours,
week2Overtime,
week2Regular,
totalHours: employeeTotal,
totalBreakHours: employeeBreaks,
overtimeHours: employeeOvertime,
regularHours: employeeRegular,
});
});
// Sort by total hours descending
employeeResults.sort((a, b) => b.totalHours - a.totalHours);
return {
byEmployee: employeeResults,
totals: {
hours: totalHours,
breakHours: totalBreakHours,
overtimeHours: totalOvertimeHours,
regularHours: totalRegularHours,
activeEmployees: employeeResults.filter(e => e.totalHours > 0).length,
},
byWeek: [
{
week: 1,
start: payPeriod.week1.start.toISODate(),
end: payPeriod.week1.end.toISODate(),
hours: week1TotalHours,
overtime: week1TotalOvertime,
regular: week1TotalHours - week1TotalOvertime,
},
{
week: 2,
start: payPeriod.week2.start.toISODate(),
end: payPeriod.week2.end.toISODate(),
hours: week2TotalHours,
overtime: week2TotalOvertime,
regular: week2TotalHours - week2TotalOvertime,
},
],
};
}
/**
* Calculate hours from a set of punches
*/
function calculateHoursFromPunches(punches) {
let hours = 0;
let breakHours = 0;
let currentIn = null;
let breakStart = null;
punches.forEach(punch => {
const punchTime = punchDateTime(punch)?.toJSDate() ?? new Date(NaN);
switch (punch.PunchType) {
case PUNCH_TYPES.IN:
currentIn = punchTime;
break;
case PUNCH_TYPES.OUT:
if (currentIn) {
hours += (punchTime - currentIn) / (1000 * 60 * 60);
currentIn = null;
}
break;
case PUNCH_TYPES.BREAK_START:
breakStart = punchTime;
break;
case PUNCH_TYPES.BREAK_END:
if (breakStart) {
breakHours += (punchTime - breakStart) / (1000 * 60 * 60);
breakStart = null;
}
break;
}
});
return { hours, breakHours };
}
/**
* Calculate FTE for a pay period (based on 80 hours = 1 FTE for 2-week period)
* @param {number} totalHours - Total hours worked
* @param {number} elapsedFraction - Fraction of the period elapsed (0-1). Defaults to 1 for complete periods.
*/
function calculateFTE(totalHours, elapsedFraction = 1) {
const fullTimePeriodHours = STANDARD_WEEKLY_HOURS * 2; // 80 hours for 2 weeks
const proratedHours = fullTimePeriodHours * elapsedFraction;
return proratedHours > 0 ? totalHours / proratedHours : 0;
}
// Main payroll metrics endpoint
router.get('/', async (req, res) => {
const startTime = Date.now();
console.log(`[PAYROLL-METRICS] Starting request`);
const timeoutPromise = new Promise((_, reject) => {
setTimeout(() => reject(new Error('Request timeout after 30 seconds')), 30000);
});
try {
const mainOperation = async () => {
const { payPeriodStart, navigate } = req.query;
let payPeriod;
if (payPeriodStart) {
// Parse the provided start date
const startDate = DateTime.fromISO(payPeriodStart, { zone: TIMEZONE });
if (!startDate.isValid) {
return res.status(400).json({ error: 'Invalid payPeriodStart date format' });
}
payPeriod = getPayPeriodForDate(startDate);
} else {
// Default to current pay period
payPeriod = getCurrentPayPeriod();
}
// Handle navigation if requested
if (navigate) {
const offset = parseInt(navigate, 10);
if (!isNaN(offset)) {
payPeriod = navigatePayPeriod(payPeriod.start, offset);
}
}
console.log(`[PAYROLL-METRICS] Getting DB connection...`);
const { connection, release } = await getDbConnection();
console.log(`[PAYROLL-METRICS] DB connection obtained in ${Date.now() - startTime}ms`);
try {
// Build query for the pay period. Bounds are serialized to Central
// wall-clock strings (the column's literal zone), half-open
// [start, start + 14 days).
const periodStart = toMySqlBound(payPeriod.start);
const periodEnd = toMySqlBound(payPeriod.start.plus({ days: 14 }));
const timeclockQuery = `
SELECT
tc.EmployeeID,
tc.TimeStamp,
tc.PunchType,
e.firstname,
e.lastname
FROM timeclock tc
LEFT JOIN employees e ON tc.EmployeeID = e.employeeid
WHERE tc.TimeStamp >= ? AND tc.TimeStamp < ?
AND e.hidden = 0
AND e.disabled = 0
ORDER BY tc.EmployeeID, tc.TimeStamp
`;
const [timeclockRows] = await connection.execute(timeclockQuery, [periodStart, periodEnd]);
// Calculate hours with week breakdown
const hoursData = calculateHoursByWeek(timeclockRows, payPeriod);
// Calculate FTE — prorate for in-progress periods so the value reflects
// the pace employees are on rather than raw hours / 80
let elapsedFraction = 1;
if (isCurrentPayPeriod(payPeriod)) {
const now = DateTime.now().setZone(TIMEZONE);
const elapsedDays = Math.max(1, Math.ceil(now.diff(payPeriod.start, 'days').days));
elapsedFraction = Math.min(1, elapsedDays / 14);
}
const fte = calculateFTE(hoursData.totals.hours, elapsedFraction);
const activeEmployees = hoursData.totals.activeEmployees;
const avgHoursPerEmployee = activeEmployees > 0 ? hoursData.totals.hours / activeEmployees : 0;
// Get previous pay period data for comparison
const prevPayPeriod = navigatePayPeriod(payPeriod.start, -1);
const [prevTimeclockRows] = await connection.execute(timeclockQuery, [
toMySqlBound(prevPayPeriod.start),
toMySqlBound(prevPayPeriod.start.plus({ days: 14 })),
]);
const prevHoursData = calculateHoursByWeek(prevTimeclockRows, prevPayPeriod);
const prevFte = calculateFTE(prevHoursData.totals.hours);
// Calculate comparisons
const comparison = {
hours: calculateComparison(hoursData.totals.hours, prevHoursData.totals.hours),
overtimeHours: calculateComparison(hoursData.totals.overtimeHours, prevHoursData.totals.overtimeHours),
fte: calculateComparison(fte, prevFte),
activeEmployees: calculateComparison(hoursData.totals.activeEmployees, prevHoursData.totals.activeEmployees),
};
const response = {
payPeriod: {
start: payPeriod.start.toISODate(),
end: payPeriod.end.toISODate(),
label: formatPayPeriodLabel(payPeriod),
week1: {
start: payPeriod.week1.start.toISODate(),
end: payPeriod.week1.end.toISODate(),
label: formatWeekLabel(payPeriod.week1),
},
week2: {
start: payPeriod.week2.start.toISODate(),
end: payPeriod.week2.end.toISODate(),
label: formatWeekLabel(payPeriod.week2),
},
isCurrent: isCurrentPayPeriod(payPeriod),
},
totals: {
hours: hoursData.totals.hours,
breakHours: hoursData.totals.breakHours,
overtimeHours: hoursData.totals.overtimeHours,
regularHours: hoursData.totals.regularHours,
activeEmployees,
fte,
avgHoursPerEmployee,
},
previousTotals: {
hours: prevHoursData.totals.hours,
overtimeHours: prevHoursData.totals.overtimeHours,
activeEmployees: prevHoursData.totals.activeEmployees,
fte: prevFte,
},
comparison,
byEmployee: hoursData.byEmployee,
byWeek: hoursData.byWeek,
};
return response;
} finally {
// Always release the connection regardless of who wins Promise.race.
// If the timeout wins, this IIFE keeps running until MySQL responds; this
// finally ensures the connection still returns to the pool.
release();
}
};
const response = await Promise.race([mainOperation(), timeoutPromise]);
console.log(`[PAYROLL-METRICS] Request completed in ${Date.now() - startTime}ms`);
res.json(response);
} catch (error) {
if (error.message.includes('timeout')) {
console.log(`[PAYROLL-METRICS] Request timed out in ${Date.now() - startTime}ms`);
} else {
console.error('Error in /payroll-metrics:', error);
}
console.log(`[PAYROLL-METRICS] Request failed in ${Date.now() - startTime}ms`);
res.status(500).json({ error: error.message });
}
});
// Get pay period info endpoint (for navigation without full data)
router.get('/period-info', async (req, res) => {
try {
const { payPeriodStart, navigate } = req.query;
let payPeriod;
if (payPeriodStart) {
const startDate = DateTime.fromISO(payPeriodStart, { zone: TIMEZONE });
if (!startDate.isValid) {
return res.status(400).json({ error: 'Invalid payPeriodStart date format' });
}
payPeriod = getPayPeriodForDate(startDate);
} else {
payPeriod = getCurrentPayPeriod();
}
if (navigate) {
const offset = parseInt(navigate, 10);
if (!isNaN(offset)) {
payPeriod = navigatePayPeriod(payPeriod.start, offset);
}
}
res.json({
payPeriod: {
start: payPeriod.start.toISODate(),
end: payPeriod.end.toISODate(),
label: formatPayPeriodLabel(payPeriod),
week1: {
start: payPeriod.week1.start.toISODate(),
end: payPeriod.week1.end.toISODate(),
label: formatWeekLabel(payPeriod.week1),
},
week2: {
start: payPeriod.week2.start.toISODate(),
end: payPeriod.week2.end.toISODate(),
label: formatWeekLabel(payPeriod.week2),
},
isCurrent: isCurrentPayPeriod(payPeriod),
},
});
} catch (error) {
console.error('Error in /payroll-metrics/period-info:', error);
res.status(500).json({ error: error.message });
}
});
// Health check
router.get('/health', async (req, res) => {
try {
const { connection, release } = await getDbConnection();
await connection.execute('SELECT 1 as test');
release();
res.json({
status: 'healthy',
timestamp: new Date().toISOString(),
pool: getPoolStatus(),
});
} catch (error) {
res.status(500).json({
status: 'unhealthy',
timestamp: new Date().toISOString(),
error: error.message,
});
}
});
// Helper functions
function calculateComparison(currentValue, previousValue) {
if (typeof previousValue !== 'number') {
return { absolute: null, percentage: null };
}
const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
const percentage =
absolute !== null && previousValue !== 0
? (absolute / Math.abs(previousValue)) * 100
: null;
return { absolute, percentage };
}
function formatPayPeriodLabel(payPeriod) {
const startStr = payPeriod.start.toFormat('MMM d');
const endStr = payPeriod.end.toFormat('MMM d, yyyy');
return `${startStr} – ${endStr}`;
}
function formatWeekLabel(week) {
const startStr = week.start.toFormat('MMM d');
const endStr = week.end.toFormat('MMM d');
return `${startStr} – ${endStr}`;
}
function isCurrentPayPeriod(payPeriod) {
const now = DateTime.now().setZone(TIMEZONE);
return now >= payPeriod.start && now <= payPeriod.end;
}
export default router;
inventory-server/dashboard/acot-server/routes/test.js
-58
View File
@@ -1,58 +0,0 @@
import express from 'express';
import { getDbConnection, getCachedQuery } from '../db/connection.js';
const router = express.Router();
// Test endpoint to count orders
router.get('/order-count', async (req, res) => {
try {
const { connection } = await getDbConnection();
// Simple query to count orders from _order table
const queryFn = async () => {
const [rows] = await connection.execute('SELECT COUNT(*) as count FROM _order');
return rows[0].count;
};
const cacheKey = 'order-count';
const count = await getCachedQuery(cacheKey, 'default', queryFn);
res.json({
success: true,
data: {
orderCount: count,
timestamp: new Date().toISOString()
}
});
} catch (error) {
console.error('Error fetching order count:', error);
res.status(500).json({
success: false,
error: error.message
});
}
});
// Test connection endpoint
router.get('/test-connection', async (req, res) => {
try {
const { connection } = await getDbConnection();
// Test the connection with a simple query
const [rows] = await connection.execute('SELECT 1 as test');
res.json({
success: true,
message: 'Database connection successful',
data: rows[0]
});
} catch (error) {
console.error('Error testing connection:', error);
res.status(500).json({
success: false,
error: error.message
});
}
});
export default router;
inventory-server/dashboard/acot-server/server.js
-163
View File
@@ -1,163 +0,0 @@
// acot-server — Phase 5 of CONSOLIDATION_PLAN.md.
// Standalone service on ACOT_PORT (default 3012) exposing /api/acot/* against
// the production MySQL `sg` database via an ssh2 tunnel (see db/connection.js).
//
// Auth model (two flavors, deliberate):
// - /api/acot/customers/* : x-acot-api-key shared secret (used by acot-phone-server).
// Mounted BEFORE authenticate() so its requirePhoneApiKey
// path is the only gate.
// - everything else : JWT Bearer via shared/auth/middleware.js authenticate().
// Defense-in-depth on top of Caddy forward_auth.
//
// Shared infrastructure (Phase 2 + Phase 6):
// - shared/auth/middleware.js authenticate() for SPA-served routes
// - shared/cors/policy.js explicit allowed-origins list (Phase 6.6)
// - shared/logging/request-log.js pino-http, Authorization/Cookie redacted (Phase 6.5/6.9)
// - shared/errors/handler.js consistent error envelope, no leak in prod
//
// Env layering: /var/www/inventory/.env loaded FIRST (JWT_SECRET, DB_* for the
// shared PG pool used by authenticate to look up user permissions). Local .env
// loaded SECOND for ACOT-specific keys (PROD_DB_*, PROD_SSH_*, ACOT_PHONE_API_KEY).
// dotenv defaults to override:false, so the first file wins on collisions.
import { config as loadEnv } from 'dotenv';
import express from 'express';
import cors from 'cors';
import compression from 'compression';
import morgan from 'morgan';
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import pg from 'pg';
import { authenticate } from '../../shared/auth/middleware.js';
import { corsOptions } from '../../shared/cors/policy.js';
import { errorHandler } from '../../shared/errors/handler.js';
import { logger } from '../../shared/logging/logger.js';
import { requestLog } from '../../shared/logging/request-log.js';
import { closeAllConnections } from './db/connection.js';
import testRouter from './routes/test.js';
import eventsRouter from './routes/events.js';
import discountsRouter from './routes/discounts.js';
import employeeMetricsRouter from './routes/employee-metrics.js';
import payrollMetricsRouter from './routes/payroll-metrics.js';
import operationsMetricsRouter from './routes/operations-metrics.js';
import customersRouter from './routes/customers.js';
const { Pool } = pg;
const __dirname = path.dirname(fileURLToPath(import.meta.url));
// Layer envs: shared inventory .env first (JWT_SECRET, DB_*) then acot .env.
const sharedEnvPath = '/var/www/inventory/.env';
const localEnvPath = path.resolve(__dirname, '.env');
if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
if (fs.existsSync(localEnvPath)) loadEnv({ path: localEnvPath });
// Phase 6.4 — refuse to start without JWT_SECRET. authenticate() would reject
// every request anyway; failing fast surfaces the misconfiguration immediately.
if (!process.env.JWT_SECRET) {
logger.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
process.exit(1);
}
const app = express();
const PORT = Number(process.env.ACOT_PORT) || 3012;
// Trust X-Forwarded-* only when the immediate hop is loopback (Caddy on the same
// host). Required for the KIOSK_IPS bypass in shared/auth/middleware.js to see
// real client IPs instead of 127.0.0.1.
app.set('trust proxy', 'loopback');
// Postgres pool for authenticate() (user/permission lookups against inventory_db).
// All MySQL access goes through db/connection.js (separate, ssh-tunneled).
const pool = new Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: Number(process.env.DB_PORT) || 5432,
});
// Per-app access log on disk (kept from pre-conversion behavior; pino request-log
// is mounted below for structured/redacted server-side logging).
const logDir = path.join(__dirname, 'logs/app');
if (!fs.existsSync(logDir)) {
fs.mkdirSync(logDir, { recursive: true });
}
const accessLogStream = fs.createWriteStream(path.join(logDir, 'access.log'), { flags: 'a' });
app.use(requestLog());
app.use(compression());
app.use(cors(corsOptions));
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
if (process.env.NODE_ENV === 'production') {
app.use(morgan('combined', { stream: accessLogStream }));
} else {
app.use(morgan('dev'));
}
app.get('/health', (req, res) => {
res.json({
status: 'healthy',
service: 'acot-server',
timestamp: new Date().toISOString(),
uptime: process.uptime(),
});
});
// Customers route uses x-acot-api-key (shared secret with acot-phone-server),
// NOT JWT — mount BEFORE authenticate() so requirePhoneApiKey is the only gate.
app.use('/api/acot/customers', customersRouter);
// All remaining /api/acot/* routes require a valid JWT.
app.use('/api/acot', authenticate({ pool, secret: process.env.JWT_SECRET }));
app.use('/api/acot/test', testRouter);
app.use('/api/acot/events', eventsRouter);
app.use('/api/acot/discounts', discountsRouter);
app.use('/api/acot/employee-metrics', employeeMetricsRouter);
app.use('/api/acot/payroll-metrics', payrollMetricsRouter);
app.use('/api/acot/operations-metrics', operationsMetricsRouter);
// 404 for unmatched /api routes (keeps prior behavior).
app.use((req, res) => {
res.status(404).json({ success: false, error: 'Route not found' });
});
app.use(errorHandler);
const server = app.listen(PORT, '0.0.0.0', () => {
logger.info({ port: PORT, mode: process.env.NODE_ENV || 'development' }, 'acot-server listening');
});
const gracefulShutdown = async (signal) => {
logger.info({ signal }, 'acot-server shutting down');
server.close(async () => {
try {
await closeAllConnections();
} catch (err) {
logger.error({ err: { message: err.message } }, 'error closing MySQL pool');
}
try {
await pool.end();
} catch { /* ignore */ }
process.exit(0);
});
};
process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));
process.on('uncaughtException', (err) => {
logger.error({ err: { message: err.message, stack: err.stack } }, 'uncaughtException');
process.exit(1);
});
process.on('unhandledRejection', (reason) => {
logger.error({ reason }, 'unhandledRejection');
});
export default app;
inventory-server/dashboard/acot-server/utils/phoneAuth.js
-26
View File
@@ -1,26 +0,0 @@
// Shared-secret auth for customer-lookup endpoints that expose PII.
// The acot-phone-server sends `x-acot-api-key` on every request; we compare
// against ACOT_PHONE_API_KEY from the environment using timing-safe comparison.
import crypto from 'node:crypto';
export function requirePhoneApiKey(req, res, next) {
const expected = process.env.ACOT_PHONE_API_KEY;
if (!expected) {
console.error('ACOT_PHONE_API_KEY not configured; rejecting all requests');
return res.status(503).json({ success: false, error: 'auth_not_configured' });
}
const provided = req.get('x-acot-api-key') || '';
const expectedBuf = Buffer.from(expected);
const providedBuf = Buffer.from(provided);
if (
providedBuf.length !== expectedBuf.length ||
!crypto.timingSafeEqual(providedBuf, expectedBuf)
) {
return res.status(401).json({ success: false, error: 'unauthorized' });
}
next();
}
inventory-server/dashboard/package-lock.json
Generated
-2939
View File
File diff suppressed because it is too large Load Diff
inventory-server/dashboard/package.json
-29
View File
@@ -1,29 +0,0 @@
{
"name": "dashboard-server",
"version": "1.0.0",
"description": "Merged ESM dashboard server (klaviyo + meta + google-analytics + typeform). Phase 4 of CONSOLIDATION_PLAN.md.",
"main": "server.js",
"type": "module",
"scripts": {
"start": "node server.js",
"dev": "nodemon server.js"
},
"dependencies": {
"@google-analytics/data": "^4.0.0",
"axios": "^1.7.9",
"cors": "^2.8.5",
"dotenv": "^16.4.7",
"express": "^4.21.2",
"ioredis": "^5.4.1",
"jsonwebtoken": "^9.0.2",
"lodash": "^4.17.21",
"luxon": "^3.5.0",
"node-fetch": "^3.3.2",
"pg": "^8.18.0",
"pino": "^9.5.0",
"pino-http": "^10.3.0"
},
"devDependencies": {
"nodemon": "^3.0.2"
}
}
inventory-server/dashboard/routes/freescout/index.js
-30
View File
@@ -1,30 +0,0 @@
// FreeScout customer-service report router. Read-only — authenticated-only is
// sufficient (same stance as Google/Typeform); no requirePermission() gate.
import express from 'express';
import { FreescoutService } from '../../services/freescout/freescout.service.js';
const ALLOWED_DAYS = new Set([7, 14, 30, 90, 180, 365]);
export function createFreescoutRouter({ redis, freescoutPool, phonePool }) {
const router = express.Router();
const freescout = new FreescoutService(redis, freescoutPool, phonePool);
router.get('/report', async (req, res) => {
const days = Number(req.query.days) || 30;
if (!ALLOWED_DAYS.has(days)) {
return res.status(400).json({
error: 'Invalid days parameter',
details: `days must be one of ${[...ALLOWED_DAYS].join(', ')}`,
});
}
try {
res.json(await freescout.getReport(days));
} catch (error) {
console.error('FreeScout report error:', { days, error: error.message });
res.status(500).json({ error: 'Failed to build FreeScout report', details: error.message });
}
});
return router;
}
inventory-server/dashboard/routes/google/index.js
-55
View File
@@ -1,55 +0,0 @@
// Google Analytics router — ESM conversion of google-server/routes/analytics.routes.js.
// All routes are read-only — authenticated-only is sufficient; no extra permission.
// google_write is reserved for future write endpoints (per migration 005).
import express from 'express';
import { AnalyticsService } from '../../services/google/analytics.service.js';
export function createGoogleRouter({ redis }) {
const router = express.Router();
const service = new AnalyticsService(redis);
router.get('/metrics', async (req, res) => {
try {
const { startDate = '7daysAgo' } = req.query;
const data = await service.getBasicMetrics(startDate);
res.json({ success: true, data });
} catch (error) {
console.error('Metrics error:', { startDate: req.query.startDate, error: error.message });
res.status(500).json({ success: false, error: 'Failed to fetch metrics', details: error.message });
}
});
router.get('/realtime/basic', async (req, res) => {
try {
const data = await service.getRealTimeBasicData();
res.json({ success: true, data });
} catch (error) {
console.error('Realtime basic error:', { error: error.message });
res.status(500).json({ success: false, error: 'Failed to fetch realtime basic data', details: error.message });
}
});
router.get('/realtime/detailed', async (req, res) => {
try {
const data = await service.getRealTimeDetailedData();
res.json({ success: true, data });
} catch (error) {
console.error('Realtime detailed error:', { error: error.message });
res.status(500).json({ success: false, error: 'Failed to fetch realtime detailed data', details: error.message });
}
});
router.get('/user-behavior', async (req, res) => {
try {
const { timeRange = '30' } = req.query;
const data = await service.getUserBehavior(timeRange);
res.json({ success: true, data });
} catch (error) {
console.error('User behavior error:', { timeRange: req.query.timeRange, error: error.message });
res.status(500).json({ success: false, error: 'Failed to fetch user behavior data', details: error.message });
}
});
return router;
}
inventory-server/dashboard/routes/klaviyo/campaigns.routes.js
-71
View File
@@ -1,71 +0,0 @@
import express from 'express';
import { CampaignsService } from '../../services/klaviyo/campaigns.service.js';
import { TimeManager } from '../../../shared/business-time/index.js';
export function createCampaignsRouter(apiKey, apiRevision, redis) {
const router = express.Router();
const timeManager = new TimeManager();
const campaignsService = new CampaignsService(apiKey, apiRevision, redis);
// Get campaigns with optional filtering
router.get('/', async (req, res) => {
try {
const params = {
pageSize: parseInt(req.query.pageSize) || 50,
sort: req.query.sort || '-send_time',
status: req.query.status,
startDate: req.query.startDate,
endDate: req.query.endDate,
pageCursor: req.query.pageCursor
};
console.log('[Campaigns Route] Fetching campaigns with params:', params);
const data = await campaignsService.getCampaigns(params);
console.log('[Campaigns Route] Success:', {
count: data.data?.length || 0
});
res.json(data);
} catch (error) {
console.error('[Campaigns Route] Error:', error);
res.status(500).json({
status: 'error',
message: error.message,
details: error.response?.data || null
});
}
});
// Get campaigns by time range
router.get('/:timeRange', async (req, res) => {
try {
const { timeRange } = req.params;
const { status } = req.query;
let result;
if (timeRange === 'custom') {
const { startDate, endDate } = req.query;
if (!startDate || !endDate) {
return res.status(400).json({ error: 'Custom range requires startDate and endDate' });
}
result = await campaignsService.getCampaigns({
startDate,
endDate,
status
});
} else {
result = await campaignsService.getCampaignsByTimeRange(
timeRange,
{ status }
);
}
res.json(result);
} catch (error) {
console.error("[Campaigns Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
return router;
}
inventory-server/dashboard/routes/klaviyo/events.routes.js
-485
View File
@@ -1,485 +0,0 @@
import express from 'express';
import { EventsService } from '../../services/klaviyo/events.service.js';
import { TimeManager } from '../../../shared/business-time/index.js';
import { RedisService } from '../../services/klaviyo/redis.service.js';
import { requirePermission } from '../../../shared/auth/middleware.js';
// Import METRIC_IDS from events service
const METRIC_IDS = {
PLACED_ORDER: 'Y8cqcF',
SHIPPED_ORDER: 'VExpdL',
ACCOUNT_CREATED: 'TeeypV',
CANCELED_ORDER: 'YjVMNg',
NEW_BLOG_POST: 'YcxeDr',
PAYMENT_REFUNDED: 'R7XUYh'
};
export function createEventsRouter(apiKey, apiRevision, redis) {
const router = express.Router();
const timeManager = new TimeManager();
const eventsService = new EventsService(apiKey, apiRevision, redis);
const redisService = new RedisService(redis);
// Phase 6.2: clearCache is operational maintenance — requires klaviyo_admin.
// Mounted as path-level middleware so the existing POST handler below stays untouched.
router.use('/clearCache', requirePermission('klaviyo_admin'));
// Get events with optional filtering
router.get('/', async (req, res) => {
try {
const params = {
pageSize: parseInt(req.query.pageSize) || 50,
sort: req.query.sort || '-datetime',
metricId: req.query.metricId,
startDate: req.query.startDate,
endDate: req.query.endDate,
pageCursor: req.query.pageCursor,
fields: {}
};
// Parse fields parameter if provided
if (req.query.fields) {
try {
params.fields = JSON.parse(req.query.fields);
} catch (e) {
console.warn('[Events Route] Invalid fields parameter:', e);
}
}
console.log('[Events Route] Fetching events with params:', params);
const data = await eventsService.getEvents(params);
console.log('[Events Route] Success:', {
count: data.data?.length || 0,
included: data.included?.length || 0
});
res.json(data);
} catch (error) {
console.error('[Events Route] Error:', error);
res.status(500).json({
status: 'error',
message: error.message,
details: error.response?.data || null
});
}
});
// Get events by time range
router.get('/by-time/:timeRange', async (req, res) => {
try {
const { timeRange } = req.params;
const { metricId, startDate, endDate } = req.query;
let result;
if (timeRange === 'custom') {
if (!startDate || !endDate) {
return res.status(400).json({ error: 'Custom range requires startDate and endDate' });
}
const range = timeManager.getCustomRange(startDate, endDate);
if (!range) {
return res.status(400).json({ error: 'Invalid date range' });
}
result = await eventsService.getEvents({
metricId,
startDate: range.start.toISO(),
endDate: range.end.toISO()
});
} else {
result = await eventsService.getEventsByTimeRange(
timeRange,
{ metricId }
);
}
res.json(result);
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Get comprehensive statistics for a time period
router.get('/stats', async (req, res) => {
try {
const { timeRange, startDate, endDate } = req.query;
console.log('[Events Route] Stats request:', {
timeRange,
startDate,
endDate
});
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO()
};
console.log('[Events Route] Calculating period stats with params:', params);
const stats = await eventsService.calculatePeriodStats(params);
console.log('[Events Route] Stats response:', {
timeRange: {
start: range.start.toISO(),
end: range.end.toISO()
},
shippedCount: stats?.shipping?.shippedCount,
totalOrders: stats?.orderCount
});
res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
stats
});
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Add new route for smart revenue projection
router.get('/projection', async (req, res) => {
try {
const { timeRange, startDate, endDate } = req.query;
console.log('[Events Route] Projection request:', {
timeRange,
startDate,
endDate
});
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO()
};
// Try to get from cache first with a short TTL
const cacheKey = redisService._getCacheKey('projection', params);
const cachedData = await redisService.get(cacheKey);
if (cachedData) {
console.log('[Events Route] Cache hit for projection');
return res.json(cachedData);
}
console.log('[Events Route] Calculating smart projection with params:', params);
const projection = await eventsService.calculateSmartProjection(params);
// Cache the results with a short TTL (5 minutes)
await redisService.set(cacheKey, projection, 300);
res.json(projection);
} catch (error) {
console.error("[Events Route] Error calculating projection:", error);
res.status(500).json({ error: error.message });
}
});
// Add new route for detailed stats
router.get('/stats/details', async (req, res) => {
try {
const { timeRange, startDate, endDate, metric, daily = false } = req.query;
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO(),
metric,
daily: daily === 'true' || daily === true
};
// Try to get from cache first
const cacheKey = redisService._getCacheKey('stats:details', params);
const cachedData = await redisService.get(cacheKey);
if (cachedData) {
console.log('[Events Route] Cache hit for detailed stats');
return res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
stats: cachedData
});
}
const stats = await eventsService.calculateDetailedStats(params);
// Cache the results
const ttl = redisService._getTTL(timeRange);
await redisService.set(cacheKey, stats, ttl);
res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
stats
});
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Get product statistics for a time period
router.get('/products', async (req, res) => {
try {
const { timeRange, startDate, endDate } = req.query;
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO()
};
// Try to get from cache first
const cacheKey = redisService._getCacheKey('events', params);
const cachedData = await redisService.getEventData('products', params);
if (cachedData) {
console.log('[Events Route] Cache hit for products');
return res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
stats: {
products: cachedData
}
});
}
const stats = await eventsService.calculatePeriodStats(params);
res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
stats
});
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Get event feed (multiple event types sorted by time)
router.get('/feed', async (req, res) => {
try {
const { timeRange, startDate, endDate, metricIds } = req.query;
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO(),
metricIds: metricIds ? JSON.parse(metricIds) : null
};
const result = await eventsService.getMultiMetricEvents(params);
res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
...result
});
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Get aggregated events data
router.get('/aggregate', async (req, res) => {
try {
const { timeRange, startDate, endDate, interval = 'day', metricId, property } = req.query;
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else if (timeRange) {
range = timeManager.getDateRange(timeRange);
} else {
return res.status(400).json({ error: 'Must provide either timeRange or startDate and endDate' });
}
if (!range) {
return res.status(400).json({ error: 'Invalid time range' });
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO(),
metricId,
interval,
property
};
const result = await eventsService.getEvents(params);
const groupedData = timeManager.groupEventsByInterval(result.data, interval, property);
res.json({
timeRange: {
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
},
data: groupedData
});
} catch (error) {
console.error("[Events Route] Error:", error);
res.status(500).json({ error: error.message });
}
});
// Get date range for a given time period
router.get("/dateRange", async (req, res) => {
try {
const { timeRange, startDate, endDate } = req.query;
let range;
if (startDate && endDate) {
range = timeManager.getCustomRange(startDate, endDate);
} else {
range = timeManager.getDateRange(timeRange || 'today');
}
if (!range) {
return res.status(400).json({
error: "Invalid time range parameters"
});
}
res.json({
start: range.start.toISO(),
end: range.end.toISO(),
displayStart: timeManager.formatForDisplay(range.start),
displayEnd: timeManager.formatForDisplay(range.end)
});
} catch (error) {
console.error('Error getting date range:', error);
res.status(500).json({
error: "Failed to get date range"
});
}
});
// Clear cache for a specific time range
router.post("/clearCache", async (req, res) => {
try {
const { timeRange, startDate, endDate } = req.body;
await redisService.clearCache({ timeRange, startDate, endDate });
res.json({ message: "Cache cleared successfully" });
} catch (error) {
console.error('Error clearing cache:', error);
res.status(500).json({ error: "Failed to clear cache" });
}
});
// Add new batch metrics endpoint
router.get('/batch', async (req, res) => {
try {
const { timeRange, startDate, endDate, metrics } = req.query;
// Parse metrics array from query
const metricsList = metrics ? JSON.parse(metrics) : [];
const params = timeRange === 'custom'
? { startDate, endDate, metrics: metricsList }
: { timeRange, metrics: metricsList };
const results = await eventsService.getBatchMetrics(params);
res.json(results);
} catch (error) {
console.error('[Events Route] Error in batch request:', error);
res.status(500).json({ error: error.message });
}
});
return router;
}
inventory-server/dashboard/routes/klaviyo/index.js
-48
View File
@@ -1,48 +0,0 @@
// Klaviyo router factory. Phase 4 merge: takes the injected redis client and
// the env-resolved API key/revision, returns the mounted /api/klaviyo router
// (matches Caddy proxy path; no rewrite needed).
import express from 'express';
import rateLimit from 'express-rate-limit';
import { requirePermission } from '../../../shared/auth/middleware.js';
import { createEventsRouter } from './events.routes.js';
import { createMetricsRouter } from './metrics.routes.js';
import { createCampaignsRouter } from './campaigns.routes.js';
import { createReportingRouter } from './reporting.routes.js';
export function createKlaviyoRouter({ redis }) {
const apiKey = process.env.KLAVIYO_API_KEY;
const apiRevision = process.env.KLAVIYO_API_REVISION || '2024-02-15';
if (!apiKey) {
// Loud at startup; the routes themselves will 500 on every call without it.
console.warn('[klaviyo] KLAVIYO_API_KEY not set — Klaviyo endpoints will fail');
}
const router = express.Router();
// Phase 4 carryover from klaviyo-server: throttle the heavy /reporting/campaign-values-reports
// endpoint. authenticate() already runs upstream so we don't add a per-user limiter here.
const reportingLimiter = rateLimit({
windowMs: 10 * 60 * 1000,
max: 10,
message: 'Too many requests to reporting endpoint, please try again later',
keyGenerator: (req) => `${req.ip}-klaviyo-reporting`,
skip: (req) => !req.path.includes('campaign-values-reports'),
standardHeaders: true,
legacyHeaders: false,
});
router.use('/reporting', reportingLimiter);
router.use('/events', createEventsRouter(apiKey, apiRevision, redis));
router.use('/metrics', createMetricsRouter(apiKey, apiRevision));
router.use('/campaigns', createCampaignsRouter(apiKey, apiRevision, redis));
router.use('/reporting', createReportingRouter(apiKey, apiRevision, redis));
return router;
}
// Re-exported so the dashboard server / future tests can attach the
// klaviyo_admin gate without reaching into the events router file.
export { requirePermission };
inventory-server/dashboard/routes/klaviyo/metrics.routes.js
-28
View File
@@ -1,28 +0,0 @@
import express from 'express';
import { MetricsService } from '../../services/klaviyo/metrics.service.js';
export function createMetricsRouter(apiKey, apiRevision) {
const router = express.Router();
const metricsService = new MetricsService(apiKey, apiRevision);
// Get all metrics
router.get('/', async (req, res) => {
try {
console.log('[Metrics Route] Fetching metrics');
const data = await metricsService.getMetrics();
console.log('[Metrics Route] Success:', {
count: data.data?.length || 0
});
res.json(data);
} catch (error) {
console.error('[Metrics Route] Error:', error);
res.status(500).json({
status: 'error',
message: error.message,
details: error.response?.data || null
});
}
});
return router;
}
inventory-server/dashboard/routes/klaviyo/reporting.routes.js
-29
View File
@@ -1,29 +0,0 @@
import express from 'express';
import { ReportingService } from '../../services/klaviyo/reporting.service.js';
import { TimeManager } from '../../../shared/business-time/index.js';
export function createReportingRouter(apiKey, apiRevision, redis) {
const router = express.Router();
const reportingService = new ReportingService(apiKey, apiRevision, redis);
const timeManager = new TimeManager();
// Get campaign reports by time range
router.get('/campaigns/:timeRange', async (req, res) => {
try {
const { timeRange } = req.params;
const { channel } = req.query;
const reports = await reportingService.getCampaignReports({
timeRange,
channel
});
res.json(reports);
} catch (error) {
console.error('[ReportingRoutes] Error fetching campaign reports:', error);
res.status(500).json({ error: error.message });
}
});
return router;
}
inventory-server/dashboard/routes/meta/index.js
-89
View File
@@ -1,89 +0,0 @@
// Meta router factory — ESM conversion of meta-server/routes/campaigns.routes.js.
// Phase 6.2: mutations (PATCH /campaigns/:id/budget, POST /campaigns/:id/:action)
// require the `meta_write` permission. Reads (GET) stay authenticated-only.
import express from 'express';
import { requirePermission } from '../../../shared/auth/middleware.js';
import {
fetchCampaigns,
fetchAccountInsights,
updateCampaignBudget,
updateCampaignStatus,
} from '../../services/meta/meta.service.js';
export function createMetaRouter() {
const router = express.Router();
// Reads — authenticated-only
router.get('/campaigns', async (req, res, next) => {
try {
const { since, until } = req.query;
if (!since || !until) {
return res.status(400).json({ error: 'Date range is required (since, until)' });
}
const campaigns = await fetchCampaigns(since, until);
res.json(campaigns);
} catch (error) {
console.error('Campaign fetch error:', error);
res.status(500).json({
error: 'Failed to fetch campaigns',
details: error.response?.data?.error?.message || error.message,
});
}
});
router.get('/account-insights', async (req, res) => {
try {
const { since, until } = req.query;
if (!since || !until) {
return res.status(400).json({ error: 'Date range is required (since, until)' });
}
const insights = await fetchAccountInsights(since, until);
res.json(insights);
} catch (error) {
console.error('Account insights fetch error:', error);
res.status(500).json({
error: 'Failed to fetch account insights',
details: error.response?.data?.error?.message || error.message,
});
}
});
// Writes — meta_write
router.patch('/campaigns/:campaignId/budget', requirePermission('meta_write'), async (req, res) => {
try {
const { campaignId } = req.params;
const { budget } = req.body;
if (!budget) {
return res.status(400).json({ error: 'Budget is required' });
}
const result = await updateCampaignBudget(campaignId, budget);
res.json(result);
} catch (error) {
console.error('Budget update error:', error);
res.status(500).json({
error: 'Failed to update campaign budget',
details: error.response?.data?.error?.message || error.message,
});
}
});
router.post('/campaigns/:campaignId/:action', requirePermission('meta_write'), async (req, res) => {
try {
const { campaignId, action } = req.params;
if (!['pause', 'unpause'].includes(action)) {
return res.status(400).json({ error: 'Invalid action. Use "pause" or "unpause"' });
}
const result = await updateCampaignStatus(campaignId, action);
res.json(result);
} catch (error) {
console.error('Status update error:', error);
res.status(500).json({
error: 'Failed to update campaign status',
details: error.response?.data?.error?.message || error.message,
});
}
});
return router;
}
inventory-server/dashboard/routes/typeform/index.js
-84
View File
@@ -1,84 +0,0 @@
// Typeform router — ESM conversion of typeform-server/routes/typeform.routes.js.
// All routes read-only — authenticated-only is sufficient; typeform_write reserved
// for future write endpoints (per migration 005).
import express from 'express';
import { TypeformService } from '../../services/typeform/typeform.service.js';
export function createTypeformRouter({ redis }) {
const router = express.Router();
const typeform = new TypeformService(redis);
router.get('/forms/:formId/responses', async (req, res) => {
try {
const { formId } = req.params;
const filters = req.query;
if (!formId) {
return res.status(400).json({ error: 'Missing form ID', details: 'The form ID parameter is required' });
}
const data = await typeform.getFormResponsesWithFilters(formId, filters);
if (!data) {
return res.status(404).json({ error: 'No data found', details: `No responses found for form ${formId}` });
}
res.json(data);
} catch (error) {
console.error('Form responses error:', {
formId: req.params.formId,
filters: req.query,
error: error.message,
response: error.response?.data,
});
if (error.response?.status === 401) {
return res.status(401).json({ error: 'Authentication failed', details: 'Invalid Typeform API credentials' });
}
if (error.response?.status === 404) {
return res.status(404).json({ error: 'Not found', details: `Form '${req.params.formId}' not found` });
}
if (error.response?.status === 400) {
return res.status(400).json({
error: 'Invalid request',
details: error.response?.data?.message || 'The request was invalid',
data: error.response?.data,
});
}
res.status(500).json({
error: 'Failed to fetch form responses',
details: error.response?.data?.message || error.message,
data: error.response?.data,
});
}
});
router.get('/forms/:formId/insights', async (req, res) => {
try {
const { formId } = req.params;
if (!formId) {
return res.status(400).json({ error: 'Missing form ID', details: 'The form ID parameter is required' });
}
const data = await typeform.getFormInsights(formId);
if (!data) {
return res.status(404).json({ error: 'No data found', details: `No insights found for form ${formId}` });
}
res.json(data);
} catch (error) {
console.error('Form insights error:', {
formId: req.params.formId,
error: error.message,
response: error.response?.data,
});
if (error.response?.status === 401) {
return res.status(401).json({ error: 'Authentication failed', details: 'Invalid Typeform API credentials' });
}
if (error.response?.status === 404) {
return res.status(404).json({ error: 'Not found', details: `Form '${req.params.formId}' not found` });
}
res.status(500).json({
error: 'Failed to fetch form insights',
details: error.response?.data?.message || error.message,
data: error.response?.data,
});
}
});
return router;
}
inventory-server/dashboard/scripts/create-campaign-products-table.sql
-30
View File
@@ -1,30 +0,0 @@
-- Stores individual product links found in Klaviyo campaign emails
CREATE TABLE IF NOT EXISTS klaviyo_campaign_products (
id SERIAL PRIMARY KEY,
campaign_id TEXT NOT NULL,
campaign_name TEXT,
sent_at TIMESTAMPTZ,
pid BIGINT NOT NULL,
product_url TEXT,
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(campaign_id, pid)
);
CREATE INDEX IF NOT EXISTS idx_kcp_campaign_id ON klaviyo_campaign_products(campaign_id);
CREATE INDEX IF NOT EXISTS idx_kcp_pid ON klaviyo_campaign_products(pid);
CREATE INDEX IF NOT EXISTS idx_kcp_sent_at ON klaviyo_campaign_products(sent_at);
-- Stores non-product shop links (categories, filters, etc.) found in campaigns
CREATE TABLE IF NOT EXISTS klaviyo_campaign_links (
id SERIAL PRIMARY KEY,
campaign_id TEXT NOT NULL,
campaign_name TEXT,
sent_at TIMESTAMPTZ,
link_url TEXT NOT NULL,
link_type TEXT, -- 'category', 'brand', 'filter', 'clearance', 'deals', 'other'
created_at TIMESTAMPTZ DEFAULT NOW(),
UNIQUE(campaign_id, link_url)
);
CREATE INDEX IF NOT EXISTS idx_kcl_campaign_id ON klaviyo_campaign_links(campaign_id);
CREATE INDEX IF NOT EXISTS idx_kcl_sent_at ON klaviyo_campaign_links(sent_at);
inventory-server/dashboard/scripts/import-campaign-products.js
-279
View File
@@ -1,279 +0,0 @@
/**
* Extract products featured in Klaviyo campaign emails and store in DB.
*
* - Fetches recent sent campaigns from Klaviyo API
* - Gets template HTML for each campaign message
* - Parses out product links (/shop/{id}) and other shop links
* - Inserts into klaviyo_campaign_products and klaviyo_campaign_links tables
*
* Usage: node scripts/poc-campaign-products.js [limit] [offset]
* limit: number of sent campaigns to process (default: 10)
* offset: number of sent campaigns to skip before processing (default: 0)
*
* Requires DB_* env vars (from inventory-server .env) and KLAVIYO_API_KEY.
*/
import fetch from 'node-fetch';
import pg from 'pg';
import dotenv from 'dotenv';
import path from 'path';
import fs from 'fs';
import { fileURLToPath } from 'url';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
// Load klaviyo .env for API key
dotenv.config({ path: path.resolve(__dirname, '../.env') });
// Also load the main inventory-server .env for DB credentials
const mainEnvPath = '/var/www/inventory/.env';
if (fs.existsSync(mainEnvPath)) {
dotenv.config({ path: mainEnvPath });
}
const API_KEY = process.env.KLAVIYO_API_KEY;
const REVISION = process.env.KLAVIYO_API_REVISION || '2026-01-15';
const BASE_URL = 'https://a.klaviyo.com/api';
if (!API_KEY) {
console.error('KLAVIYO_API_KEY not set in .env');
process.exit(1);
}
// ── Klaviyo API helpers ──────────────────────────────────────────────
const headers = {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': `Klaviyo-API-Key ${API_KEY}`,
'revision': REVISION,
};
async function klaviyoGet(endpoint, params = {}) {
const url = new URL(`${BASE_URL}${endpoint}`);
for (const [k, v] of Object.entries(params)) {
url.searchParams.append(k, v);
}
return klaviyoFetch(url.toString());
}
async function klaviyoFetch(url) {
const res = await fetch(url, { headers });
if (!res.ok) {
const body = await res.text();
throw new Error(`Klaviyo ${res.status} on ${url}: ${body}`);
}
return res.json();
}
async function getRecentCampaigns(limit, offset = 0) {
const campaigns = [];
const messageMap = {};
let skipped = 0;
let data = await klaviyoGet('/campaigns', {
'filter': 'equals(messages.channel,"email")',
'sort': '-scheduled_at',
'include': 'campaign-messages',
});
while (true) {
for (const c of (data.data || [])) {
if (c.attributes?.status === 'Sent') {
if (skipped < offset) {
skipped++;
continue;
}
campaigns.push(c);
if (campaigns.length >= limit) break;
}
}
for (const inc of (data.included || [])) {
if (inc.type === 'campaign-message') {
messageMap[inc.id] = inc;
}
}
const nextUrl = data.links?.next;
if (campaigns.length >= limit || !nextUrl) break;
const progress = skipped < offset
? `Skipped ${skipped}/${offset}...`
: `Fetched ${campaigns.length}/${limit} sent campaigns, loading next page...`;
console.log(` ${progress}`);
await new Promise(r => setTimeout(r, 200));
data = await klaviyoFetch(nextUrl);
}
return { campaigns: campaigns.slice(0, limit), messageMap };
}
async function getTemplateHtml(messageId) {
const data = await klaviyoGet(`/campaign-messages/${messageId}/template`, {
'fields[template]': 'html,name',
});
return {
templateId: data.data?.id,
templateName: data.data?.attributes?.name,
html: data.data?.attributes?.html || '',
};
}
// ── HTML parsing ─────────────────────────────────────────────────────
function parseProductsFromHtml(html) {
const seen = new Set();
const products = [];
const linkRegex = /href="([^"]*acherryontop\.com\/shop\/(\d+))[^"]*"/gi;
let match;
while ((match = linkRegex.exec(html)) !== null) {
const productId = match[2];
if (!seen.has(productId)) {
seen.add(productId);
products.push({
siteProductId: productId,
url: match[1],
});
}
}
const categoryLinks = [];
const catRegex = /href="([^"]*acherryontop\.com\/shop\/[^"]+)"/gi;
while ((match = catRegex.exec(html)) !== null) {
const url = match[1];
if (/\/shop\/\d+$/.test(url)) continue;
if (!categoryLinks.includes(url)) categoryLinks.push(url);
}
return { products, categoryLinks };
}
function classifyLink(url) {
if (/\/shop\/(new|pre-order|backinstock)/.test(url)) return 'filter';
if (/\/shop\/company\//.test(url)) return 'brand';
if (/\/shop\/clearance/.test(url)) return 'clearance';
if (/\/shop\/daily_deals/.test(url)) return 'deals';
if (/\/shop\/category\//.test(url)) return 'category';
return 'other';
}
// ── Database ─────────────────────────────────────────────────────────
function createPool() {
return new pg.Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: process.env.DB_PORT || 5432,
ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
});
}
async function insertProducts(pool, campaignId, campaignName, sentAt, products) {
if (products.length === 0) return 0;
let inserted = 0;
for (const p of products) {
try {
await pool.query(
`INSERT INTO klaviyo_campaign_products
(campaign_id, campaign_name, sent_at, pid, product_url)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (campaign_id, pid) DO NOTHING`,
[campaignId, campaignName, sentAt, parseInt(p.siteProductId), p.url]
);
inserted++;
} catch (err) {
console.error(` Error inserting product ${p.siteProductId}: ${err.message}`);
}
}
return inserted;
}
async function insertLinks(pool, campaignId, campaignName, sentAt, links) {
if (links.length === 0) return 0;
let inserted = 0;
for (const url of links) {
try {
await pool.query(
`INSERT INTO klaviyo_campaign_links
(campaign_id, campaign_name, sent_at, link_url, link_type)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (campaign_id, link_url) DO NOTHING`,
[campaignId, campaignName, sentAt, url, classifyLink(url)]
);
inserted++;
} catch (err) {
console.error(` Error inserting link: ${err.message}`);
}
}
return inserted;
}
// ── Main ─────────────────────────────────────────────────────────────
async function main() {
const limit = parseInt(process.argv[2]) || 10;
const offset = parseInt(process.argv[3]) || 0;
const pool = createPool();
try {
// Fetch campaigns
console.log(`Fetching up to ${limit} recent campaigns (offset: ${offset})...\n`);
const { campaigns, messageMap } = await getRecentCampaigns(limit, offset);
console.log(`Found ${campaigns.length} sent campaigns.\n`);
let totalProducts = 0;
let totalLinks = 0;
for (const campaign of campaigns) {
const name = campaign.attributes?.name || 'Unnamed';
const sentAt = campaign.attributes?.send_time;
console.log(`━━━ ${name} (${sentAt?.slice(0, 10) || 'no date'}) ━━━`);
const msgIds = (campaign.relationships?.['campaign-messages']?.data || [])
.map(r => r.id);
if (msgIds.length === 0) {
console.log(' No messages.\n');
continue;
}
for (const msgId of msgIds) {
const msg = messageMap[msgId];
const subject = msg?.attributes?.definition?.content?.subject;
if (subject) console.log(` Subject: ${subject}`);
try {
const template = await getTemplateHtml(msgId);
const { products, categoryLinks } = parseProductsFromHtml(template.html);
const pInserted = await insertProducts(pool, campaign.id, name, sentAt, products);
const lInserted = await insertLinks(pool, campaign.id, name, sentAt, categoryLinks);
console.log(` ${products.length} products (${pInserted} new), ${categoryLinks.length} links (${lInserted} new)`);
totalProducts += pInserted;
totalLinks += lInserted;
await new Promise(r => setTimeout(r, 200));
} catch (err) {
console.log(` Error: ${err.message}`);
}
}
console.log('');
}
console.log(`Done. Inserted ${totalProducts} product rows, ${totalLinks} link rows.`);
} finally {
await pool.end();
}
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});
inventory-server/dashboard/scripts/test-freescout-report.mjs
-37
View File
@@ -1,37 +0,0 @@
// Smoke test for the FreeScout CS report service, bypassing HTTP auth.
// Run on the server: cd /var/www/inventory/dashboard && node scripts/test-freescout-report.mjs [days]
import { config as loadEnv } from 'dotenv';
import { createPool } from '../../shared/db/pg.js';
import { FreescoutService } from '../services/freescout/freescout.service.js';
loadEnv({ path: new URL('../.env', import.meta.url).pathname });
const days = Number(process.argv[2]) || 30;
const freescoutPool = createPool('FREESCOUT_DB', {
user: process.env.FREESCOUT_DB_USERNAME,
database: process.env.FREESCOUT_DB_DATABASE,
max: 2,
});
const phonePool = process.env.ACOT_PHONE_DB_HOST
? createPool('ACOT_PHONE_DB', {
user: process.env.ACOT_PHONE_DB_USERNAME,
database: process.env.ACOT_PHONE_DB_DATABASE,
max: 2,
})
: null;
// Stub redis in "end" status so the service skips caching entirely.
const redisStub = { status: 'end' };
const service = new FreescoutService(redisStub, freescoutPool, phonePool);
const t0 = Date.now();
try {
const report = await service.getReport(days);
console.log(JSON.stringify(report, null, 2));
console.error(`\nOK — ${days}d report in ${Date.now() - t0}ms`);
} finally {
await freescoutPool.end();
await phonePool?.end();
}
inventory-server/dashboard/server.js
-160
View File
@@ -1,160 +0,0 @@
// dashboard-server — Phase 4 of CONSOLIDATION_PLAN.md.
// Merges the four per-vendor PM2 apps (klaviyo, meta, google-analytics, typeform)
// into a single ESM service on DASHBOARD_PORT (default 3015).
//
// Mount points (matches Caddy proxy paths):
// /api/klaviyo/* → routes/klaviyo (was klaviyo-server :3004)
// /api/meta/* → routes/meta (was meta-server :3005)
// /api/dashboard-analytics/* → routes/google (was google-server :3007 via Caddy /api/analytics rewrite)
// /api/typeform/* → routes/typeform (was typeform-server :3008)
// /api/freescout/* → routes/freescout (FreeScout + acot_phone CS report, new)
//
// Shared infrastructure (Phase 2 + Phase 6):
// - shared/auth/middleware.js authenticate() guards /api/* (Phase 6.1/6.2 — second line of defense)
// - shared/cors/policy.js explicit allowed-origins list (Phase 6.6)
// - shared/logging/request-log.js pino-http, Authorization/Cookie redacted (Phase 6.5/6.9)
// - shared/errors/handler.js consistent error envelope, no leak in prod
// - shared/db/pg.js / shared/db/redis.js one Pool + one ioredis client for all vendors
//
// Per-route permission gates (Phase 6.2):
// - meta_write PATCH/POST mutations to Meta campaigns
// - klaviyo_admin POST /api/klaviyo/events/clearCache (operational maintenance)
// Read-only Google + Typeform endpoints stay authenticated-only.
import { config as loadEnv } from 'dotenv';
import express from 'express';
import cors from 'cors';
import path from 'node:path';
import fs from 'node:fs';
import { fileURLToPath } from 'node:url';
import { authenticate } from '../shared/auth/middleware.js';
import { corsOptions } from '../shared/cors/policy.js';
import { createPool } from '../shared/db/pg.js';
import { createRedis } from '../shared/db/redis.js';
import { errorHandler } from '../shared/errors/handler.js';
import { logger } from '../shared/logging/logger.js';
import { requestLog } from '../shared/logging/request-log.js';
import { createKlaviyoRouter } from './routes/klaviyo/index.js';
import { createMetaRouter } from './routes/meta/index.js';
import { createGoogleRouter } from './routes/google/index.js';
import { createTypeformRouter } from './routes/typeform/index.js';
import { createFreescoutRouter } from './routes/freescout/index.js';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
// Layer envs: shared inventory .env wins on collisions (security-critical vars come
// from one place); vendor-specific keys come from the per-service .env.
//
// dotenv defaults to override:false so the first file loaded wins. Order matters.
const sharedEnvPath = '/var/www/inventory/.env';
const dashboardEnvPath = path.resolve(__dirname, '.env');
if (fs.existsSync(sharedEnvPath)) loadEnv({ path: sharedEnvPath });
if (fs.existsSync(dashboardEnvPath)) loadEnv({ path: dashboardEnvPath });
// Phase 6.4 — refuse to start without JWT_SECRET. Without it authenticate() falls
// back to res.status(401) on every request and the service is useless anyway.
if (!process.env.JWT_SECRET) {
logger.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
process.exit(1);
}
const app = express();
const PORT = Number(process.env.DASHBOARD_PORT) || 3015;
// Trust X-Forwarded-* only when the immediate hop is loopback (Caddy on the same
// host). Required for the KIOSK_IPS bypass in shared/auth/middleware.js to see
// real client IPs instead of 127.0.0.1.
app.set('trust proxy', 'loopback');
// Single Postgres pool — used by authenticate() to load user permissions.
// All four vendors share this pool (auth lookups are the only DB hits at runtime).
const pool = createPool('DB');
// Single ioredis client shared across all vendors. lazyConnect:true means the
// first .get/.set triggers the actual connect — keeps startup non-blocking even
// if Redis is temporarily unavailable, and aligns with shared/db/redis.js defaults.
const redis = createRedis();
// Read-only pools for the FreeScout CS section. The .env keys use USERNAME /
// DATABASE where createPool() reads USER / NAME, hence the overrides. Both are
// optional: without FREESCOUT_DB_* the route isn't mounted; without
// ACOT_PHONE_DB_* the report's phone block is null.
const freescoutPool = process.env.FREESCOUT_DB_HOST
? createPool('FREESCOUT_DB', {
user: process.env.FREESCOUT_DB_USERNAME,
database: process.env.FREESCOUT_DB_DATABASE,
max: 5,
})
: null;
const phonePool = process.env.ACOT_PHONE_DB_HOST
? createPool('ACOT_PHONE_DB', {
user: process.env.ACOT_PHONE_DB_USERNAME,
database: process.env.ACOT_PHONE_DB_DATABASE,
max: 3,
})
: null;
app.use(requestLog());
app.use(cors(corsOptions));
app.use(express.json({ limit: '10mb' }));
// Phase 6.1/6.2: every /api request requires a valid JWT. authenticate() also
// loads user permissions, which the per-route requirePermission() checks rely on.
app.use('/api', authenticate({ pool, secret: process.env.JWT_SECRET }));
app.use('/api/klaviyo', createKlaviyoRouter({ redis }));
app.use('/api/meta', createMetaRouter());
// Note: frontend calls /api/dashboard-analytics (Caddy used to rewrite it to
// /api/analytics for the standalone google-server). Mount at the public path so
// Caddy can drop the rewrite — see Caddyfile.proposed.
app.use('/api/dashboard-analytics', createGoogleRouter({ redis }));
app.use('/api/typeform', createTypeformRouter({ redis }));
if (freescoutPool) {
app.use('/api/freescout', createFreescoutRouter({ redis, freescoutPool, phonePool }));
} else {
logger.warn('FREESCOUT_DB_* not set — /api/freescout not mounted');
}
app.get('/health', (req, res) => {
res.json({
status: 'ok',
timestamp: new Date().toISOString(),
service: 'dashboard-server',
redis: redis.status,
});
});
app.use(errorHandler);
// Connect Redis up front so the first request doesn't pay the connect cost.
// Failures here are non-fatal — vendors degrade to cache-miss → upstream fetch.
redis.connect().catch((err) => {
logger.error({ err: { message: err.message, code: err.code } }, 'redis lazy-connect failed');
});
const server = app.listen(PORT, '0.0.0.0', () => {
logger.info({ port: PORT, mode: process.env.NODE_ENV || 'development' }, 'dashboard-server listening');
});
const shutdown = async (signal) => {
logger.info({ signal }, 'dashboard-server shutting down');
server.close();
try { await redis.quit(); } catch { /* ignore */ }
try { await pool.end(); } catch { /* ignore */ }
try { await freescoutPool?.end(); } catch { /* ignore */ }
try { await phonePool?.end(); } catch { /* ignore */ }
process.exit(0);
};
process.on('SIGTERM', () => shutdown('SIGTERM'));
process.on('SIGINT', () => shutdown('SIGINT'));
process.on('uncaughtException', (err) => {
logger.error({ err: { message: err.message, stack: err.stack } }, 'uncaughtException');
process.exit(1);
});
process.on('unhandledRejection', (reason) => {
logger.error({ reason }, 'unhandledRejection');
});
inventory-server/dashboard/services/freescout/freescout.service.js
-409
View File
@@ -1,409 +0,0 @@
// FreeScout customer-service report — direct read-only queries against the
// freescout and acot_phone Postgres databases (same instance as inventory_db).
// FreeScout's Reports module has no JSON API (its /reports/ajax needs an agent
// session cookie and returns rendered Blade HTML), so we reproduce its formulas
// from Modules/Reports/Http/Controllers/ReportsController.php instead.
//
// Timezone: freescout.* timestamp columns are NAIVE literals in America/New_York
// (Laravel APP_TZ). Every comparison re-anchors them with AT TIME ZONE
// 'America/New_York'; day bucketing then converts to America/Chicago business
// days per shared/business-time. acot_phone columns are proper timestamptz.
//
// First-response / resolution times are NOT recomputed from threads — FreeScout's
// `freescout:reports-collect-data` cron precomputes them into conversations.meta
// under the "rpt" key (frt / rst / rnt / rtr / rfr, seconds). We read that JSON.
import { DateTime } from 'luxon';
import { BUSINESS_TZ } from '../../../shared/business-time/index.js';
// Enum values from freescout app/Conversation.php and app/Thread.php, and
// Modules/SatRatings (rating: 1=great 2=okay 3=bad).
const CONV_STATE_PUBLISHED = 2;
const CONV_STATUS_CLOSED = 3;
const CONV_STATUS_SPAM = 4;
const CONV_TYPES = { 1: 'email', 2: 'phone', 3: 'chat' };
const THREAD_TYPE_CUSTOMER = 1;
const THREAD_TYPE_MESSAGE = 2;
// FreeScout's report-table time buckets (Reports module getTimeTablePattern),
// used for the first-response-time histogram. Thresholds in seconds; SQL
// width_bucket() returns the index into this list.
const FRT_BUCKET_LABELS = ['<15m', '15-30m', '30-60m', '1-2h', '2-3h', '3-6h', '6-12h', '12-24h', '1-2d', '>2d'];
const FRT_BUCKET_THRESHOLDS = [900, 1800, 3600, 7200, 10800, 21600, 43200, 86400, 172800];
// Re-anchor a naive America/New_York timestamp column as timestamptz.
const ny = (col) => `(${col} AT TIME ZONE 'America/New_York')`;
// Bucket it to an America/Chicago business date.
const nyBizDate = (col) => `(${ny(col)} AT TIME ZONE '${BUSINESS_TZ}')::date`;
const num = (v) => (v === null || v === undefined ? null : Number(v));
export class FreescoutService {
constructor(redis, freescoutPool, phonePool = null) {
if (!redis) throw new Error('FreescoutService requires an ioredis client');
if (!freescoutPool) throw new Error('FreescoutService requires the freescout pg Pool');
this.redis = redis;
this.fs = freescoutPool;
this.phone = phonePool; // optional — phone card hidden when absent
}
get _redisReady() {
return this.redis.status === 'ready' || this.redis.status === 'connect';
}
async _cacheGet(key) {
if (!this._redisReady) return null;
try {
const raw = await this.redis.get(key);
return raw ? JSON.parse(raw) : null;
} catch (err) {
console.warn('[Freescout] cache get failed:', err.message);
return null;
}
}
async _cacheSet(key, value, ttlSec) {
if (!this._redisReady) return;
try {
await this.redis.setex(key, ttlSec, JSON.stringify(value));
} catch (err) {
console.warn('[Freescout] cache set failed:', err.message);
}
}
async getReport(days) {
const cacheKey = `freescout:report:v2:${days}`;
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const todayStart = DateTime.now().setZone(BUSINESS_TZ).startOf('day');
const end = todayStart.plus({ days: 1 }); // exclusive — current period includes today-so-far
const start = todayStart.minus({ days: days - 1 });
const prevEnd = start;
const prevStart = start.minus({ days });
const cur = [start.toISO(), end.toISO()];
const prev = [prevStart.toISO(), prevEnd.toISO()];
const [
overview, overviewPrev,
times, timesPrev,
satisfaction, satisfactionPrev,
timeseries, channels, agents,
frtDistribution,
phone, phonePrev,
] = await Promise.all([
this._overview(cur), this._overview(prev),
this._responseTimes(cur), this._responseTimes(prev),
this._satisfaction(cur), this._satisfaction(prev),
this._timeseries(cur, start, todayStart),
this._channels(cur),
this._agents(cur),
this._frtDistribution(cur),
this._phoneStats(cur), this._phoneStats(prev),
]);
const report = {
range: {
days,
start: start.toISODate(),
end: todayStart.toISODate(),
prevStart: prevStart.toISODate(),
prevEnd: prevEnd.minus({ days: 1 }).toISODate(),
},
overview: { current: overview, previous: overviewPrev },
responseTimes: { current: times, previous: timesPrev },
satisfaction: { current: satisfaction, previous: satisfactionPrev },
timeseries,
channels,
agents,
frtDistribution,
phone: phone ? { current: phone, previous: phonePrev } : null,
};
await this._cacheSet(cacheKey, report, 300);
return report;
}
// Headline counts. Formulas mirror ReportsController: newConversations =
// countNewConv, messagesReceived = countMessages (customer threads),
// repliesSent = countRepliesSent, customersHelped = countCustomersHelped.
// "resolved" uses conversations.closed_at (simpler than the module's
// closing-thread reconstruction, same intent: closes that happened in range).
async _overview([from, to]) {
const { rows } = await this.fs.query(
`SELECT
(SELECT count(*) FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status <> ${CONV_STATUS_SPAM}
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2) AS new_conversations,
(SELECT count(*) FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status = ${CONV_STATUS_CLOSED}
AND closed_at IS NOT NULL
AND ${ny('closed_at')} >= $1 AND ${ny('closed_at')} < $2) AS resolved,
(SELECT count(*) FROM threads
WHERE type = ${THREAD_TYPE_CUSTOMER}
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2) AS messages_received,
(SELECT count(*) FROM threads
WHERE type = ${THREAD_TYPE_MESSAGE} AND state = ${CONV_STATE_PUBLISHED}
AND created_by_user_id IS NOT NULL
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2) AS replies_sent,
(SELECT count(DISTINCT c.customer_id)
FROM threads t JOIN conversations c ON c.id = t.conversation_id
WHERE t.type = ${THREAD_TYPE_MESSAGE} AND t.state = ${CONV_STATE_PUBLISHED}
AND t.created_by_user_id IS NOT NULL
AND ${ny('t.created_at')} >= $1 AND ${ny('t.created_at')} < $2) AS customers_helped`,
[from, to],
);
const r = rows[0];
return {
newConversations: num(r.new_conversations),
resolved: num(r.resolved),
messagesReceived: num(r.messages_received),
repliesSent: num(r.replies_sent),
customersHelped: num(r.customers_helped),
};
}
// Precomputed per-conversation metrics from conversations.meta->rpt (seconds):
// frt = first response time, rnt = resolution time, rfr = resolved on first reply.
async _responseTimes([from, to]) {
const { rows } = await this.fs.query(
`SELECT count(frt) AS frt_count,
round(avg(frt)) AS frt_avg,
round(percentile_cont(0.5) WITHIN GROUP (ORDER BY frt)) AS frt_median,
count(rnt) AS rnt_count,
round(avg(rnt)) AS rnt_avg,
round(percentile_cont(0.5) WITHIN GROUP (ORDER BY rnt)) AS rnt_median,
count(*) FILTER (WHERE rfr) AS rfr_count
FROM (
SELECT nullif(meta::json #>> '{rpt,frt}', '')::numeric AS frt,
nullif(meta::json #>> '{rpt,rnt}', '')::numeric AS rnt,
(meta::json #>> '{rpt,rfr}') IN ('1', 'true') AS rfr
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status <> ${CONV_STATUS_SPAM}
AND meta LIKE '{%' AND meta LIKE '%"rpt"%'
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
) rpt`,
[from, to],
);
const r = rows[0];
return {
firstResponse: { count: num(r.frt_count), avgSec: num(r.frt_avg), medianSec: num(r.frt_median) },
resolution: { count: num(r.rnt_count), avgSec: num(r.rnt_avg), medianSec: num(r.rnt_median) },
resolvedFirstReply: num(r.rfr_count),
};
}
// Histogram of first-response times (current period only), zero-filled over
// FreeScout's bucket boundaries.
async _frtDistribution([from, to]) {
const { rows } = await this.fs.query(
`SELECT width_bucket(frt, ARRAY[${FRT_BUCKET_THRESHOLDS.join(',')}]::numeric[]) AS bucket,
count(*) AS n
FROM (
SELECT nullif(meta::json #>> '{rpt,frt}', '')::numeric AS frt
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status <> ${CONV_STATUS_SPAM}
AND meta LIKE '{%' AND meta LIKE '%"rpt"%'
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
) rpt
WHERE frt IS NOT NULL
GROUP BY 1`,
[from, to],
);
const counts = new Array(FRT_BUCKET_LABELS.length).fill(0);
for (const r of rows) counts[Number(r.bucket)] = Number(r.n);
return FRT_BUCKET_LABELS.map((label, i) => ({ bucket: label, count: counts[i] }));
}
// Ratings live on the rated reply thread (threads.rating, SatRatings module).
// Score formula per calcSatisfactionScore: ceil(great% - bad%).
async _satisfaction([from, to]) {
const { rows } = await this.fs.query(
`SELECT count(*) FILTER (WHERE rating = 1) AS great,
count(*) FILTER (WHERE rating = 2) AS okay,
count(*) FILTER (WHERE rating = 3) AS bad
FROM threads
WHERE rating IS NOT NULL AND rating > 0
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2`,
[from, to],
);
const great = num(rows[0].great);
const okay = num(rows[0].okay);
const bad = num(rows[0].bad);
const total = great + okay + bad;
return {
great,
okay,
bad,
total,
score: total ? Math.ceil((great * 100) / total - (bad * 100) / total) : null,
};
}
// Per-business-day new / resolved / customer messages, zero-filled.
async _timeseries([from, to], start, todayStart) {
const [news, closes, msgs] = await Promise.all([
this.fs.query(
// ::text so pg returns 'YYYY-MM-DD' strings — the driver's DATE→JS Date
// parsing is zone-ambiguous and can shift the day
`SELECT ${nyBizDate('created_at')}::text AS day, count(*) AS n
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status <> ${CONV_STATUS_SPAM}
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
GROUP BY 1`,
[from, to],
),
this.fs.query(
`SELECT ${nyBizDate('closed_at')}::text AS day, count(*) AS n
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status = ${CONV_STATUS_CLOSED}
AND closed_at IS NOT NULL
AND ${ny('closed_at')} >= $1 AND ${ny('closed_at')} < $2
GROUP BY 1`,
[from, to],
),
this.fs.query(
`SELECT ${nyBizDate('created_at')}::text AS day, count(*) AS n
FROM threads
WHERE type = ${THREAD_TYPE_CUSTOMER}
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
GROUP BY 1`,
[from, to],
),
]);
const toMap = (res) => new Map(res.rows.map((row) => [row.day, Number(row.n)]));
const newsM = toMap(news);
const closesM = toMap(closes);
const msgsM = toMap(msgs);
const series = [];
for (let d = start; d <= todayStart; d = d.plus({ days: 1 })) {
const key = d.toISODate();
series.push({
date: key,
newConversations: newsM.get(key) ?? 0,
resolved: closesM.get(key) ?? 0,
messages: msgsM.get(key) ?? 0,
});
}
return series;
}
async _channels([from, to]) {
const { rows } = await this.fs.query(
`SELECT type, count(*) AS n
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status <> ${CONV_STATUS_SPAM}
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
GROUP BY type`,
[from, to],
);
const channels = { email: 0, phone: 0, chat: 0, other: 0 };
for (const r of rows) {
channels[CONV_TYPES[r.type] ?? 'other'] += Number(r.n);
}
return channels;
}
// Per-agent table: replies + customers helped + ratings keyed off the reply
// thread's author; closes keyed off conversations.closed_by_user_id.
async _agents([from, to]) {
const [replies, ratings, closes, users] = await Promise.all([
this.fs.query(
`SELECT t.created_by_user_id AS user_id,
count(*) AS replies,
count(DISTINCT c.customer_id) AS customers_helped
FROM threads t JOIN conversations c ON c.id = t.conversation_id
WHERE t.type = ${THREAD_TYPE_MESSAGE} AND t.state = ${CONV_STATE_PUBLISHED}
AND t.created_by_user_id IS NOT NULL
AND ${ny('t.created_at')} >= $1 AND ${ny('t.created_at')} < $2
GROUP BY 1`,
[from, to],
),
this.fs.query(
`SELECT created_by_user_id AS user_id,
count(*) FILTER (WHERE rating = 1) AS great,
count(*) FILTER (WHERE rating = 2) AS okay,
count(*) FILTER (WHERE rating = 3) AS bad
FROM threads
WHERE rating IS NOT NULL AND rating > 0 AND created_by_user_id IS NOT NULL
AND ${ny('created_at')} >= $1 AND ${ny('created_at')} < $2
GROUP BY 1`,
[from, to],
),
this.fs.query(
`SELECT closed_by_user_id AS user_id, count(*) AS closed
FROM conversations
WHERE state = ${CONV_STATE_PUBLISHED} AND status = ${CONV_STATUS_CLOSED}
AND closed_at IS NOT NULL AND closed_by_user_id IS NOT NULL
AND ${ny('closed_at')} >= $1 AND ${ny('closed_at')} < $2
GROUP BY 1`,
[from, to],
),
// type=1 excludes the Workflows robot user
this.fs.query(`SELECT id, first_name, last_name FROM users WHERE type = 1`),
]);
const agents = new Map();
const entry = (id) => {
if (!agents.has(id)) {
agents.set(id, {
id, name: null, replies: 0, customersHelped: 0, closed: 0,
great: 0, okay: 0, bad: 0, satScore: null,
});
}
return agents.get(id);
};
for (const r of replies.rows) {
const a = entry(r.user_id);
a.replies = Number(r.replies);
a.customersHelped = Number(r.customers_helped);
}
for (const r of closes.rows) entry(r.user_id).closed = Number(r.closed);
for (const r of ratings.rows) {
const a = entry(r.user_id);
a.great = Number(r.great);
a.okay = Number(r.okay);
a.bad = Number(r.bad);
const total = a.great + a.okay + a.bad;
if (total) a.satScore = Math.ceil((a.great * 100) / total - (a.bad * 100) / total);
}
const names = new Map(users.rows.map((u) => [u.id, `${u.first_name} ${u.last_name}`.trim()]));
return [...agents.values()]
.filter((a) => names.has(a.id)) // drop deleted/robot authors
.map((a) => ({ ...a, name: names.get(a.id) }))
.sort((a, b) => b.replies - a.replies);
}
// acot_phone bridge DB (timestamptz — no NY re-anchoring). Inbound calls always
// get answered_at set by the PBX, so "missed" is the voicemail count, not
// answered_at IS NULL.
async _phoneStats([from, to]) {
if (!this.phone) return null;
const [calls, vms] = await Promise.all([
this.phone.query(
`SELECT count(*) AS total,
count(*) FILTER (WHERE direction = 'inbound') AS inbound,
count(*) FILTER (WHERE direction = 'outbound') AS outbound,
round(avg(duration_seconds) FILTER (WHERE answered_at IS NOT NULL)) AS avg_duration_sec
FROM calls
WHERE started_at >= $1 AND started_at < $2`,
[from, to],
),
this.phone.query(
`SELECT count(*) AS voicemails FROM voicemails WHERE created_at >= $1 AND created_at < $2`,
[from, to],
),
]);
const c = calls.rows[0];
return {
total: num(c.total),
inbound: num(c.inbound),
outbound: num(c.outbound),
avgDurationSec: num(c.avg_duration_sec),
voicemails: num(vms.rows[0].voicemails),
};
}
}
inventory-server/dashboard/services/google/analytics.service.js
-195
View File
@@ -1,195 +0,0 @@
// Google Analytics (GA4) service — ESM conversion of google-server/services/analytics.service.js.
// Phase 4: accepts injected ioredis client (was self-constructing node-redis v4 before).
// node-redis v4 set syntax `{ EX: 300 }` is translated to ioredis `setex(key, 300, val)`.
import { BetaAnalyticsDataClient } from '@google-analytics/data';
const CACHE_DURATIONS = {
REALTIME_BASIC: 60,
REALTIME_DETAILED: 300,
BASIC_METRICS: 3600,
USER_BEHAVIOR: 3600,
};
export class AnalyticsService {
constructor(redis) {
if (!redis) {
throw new Error('AnalyticsService requires an ioredis client (Phase 4: injected)');
}
this.redis = redis;
const credentials = process.env.GOOGLE_APPLICATION_CREDENTIALS_JSON;
this.analyticsClient = new BetaAnalyticsDataClient({
credentials: typeof credentials === 'string' ? JSON.parse(credentials) : credentials,
});
this.propertyId = process.env.GA_PROPERTY_ID;
}
get _redisReady() {
return this.redis.status === 'ready' || this.redis.status === 'connect';
}
async _cacheGet(key) {
if (!this._redisReady) return null;
try {
const raw = await this.redis.get(key);
return raw ? JSON.parse(raw) : null;
} catch (err) {
console.warn('[AnalyticsService] cache get failed:', err.message);
return null;
}
}
async _cacheSet(key, value, ttlSec) {
if (!this._redisReady) return;
try {
await this.redis.setex(key, ttlSec, JSON.stringify(value));
} catch (err) {
console.warn('[AnalyticsService] cache set failed:', err.message);
}
}
async getBasicMetrics(startDate = '7daysAgo') {
const cacheKey = `analytics:basic_metrics:${startDate}`;
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const [response] = await this.analyticsClient.runReport({
property: `properties/${this.propertyId}`,
dateRanges: [{ startDate, endDate: 'today' }],
dimensions: [{ name: 'date' }],
metrics: [
{ name: 'activeUsers' },
{ name: 'newUsers' },
{ name: 'averageSessionDuration' },
{ name: 'screenPageViews' },
{ name: 'bounceRate' },
{ name: 'conversions' },
],
returnPropertyQuota: true,
});
await this._cacheSet(cacheKey, response, CACHE_DURATIONS.BASIC_METRICS);
return response;
}
async getRealTimeBasicData() {
const cacheKey = 'analytics:realtime:basic';
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const [userResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
metrics: [{ name: 'activeUsers' }],
returnPropertyQuota: true,
});
const [fiveMinResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
metrics: [{ name: 'activeUsers' }],
minuteRanges: [{ startMinutesAgo: 5, endMinutesAgo: 0 }],
});
const [timeSeriesResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
dimensions: [{ name: 'minutesAgo' }],
metrics: [{ name: 'activeUsers' }],
});
const response = {
userResponse,
fiveMinResponse,
timeSeriesResponse,
quotaInfo: {
projectHourly: userResponse.propertyQuota.tokensPerProjectPerHour,
daily: userResponse.propertyQuota.tokensPerDay,
serverErrors: userResponse.propertyQuota.serverErrorsPerProjectPerHour,
thresholdedRequests: userResponse.propertyQuota.potentiallyThresholdedRequestsPerHour,
},
};
await this._cacheSet(cacheKey, response, CACHE_DURATIONS.REALTIME_BASIC);
return response;
}
async getRealTimeDetailedData() {
const cacheKey = 'analytics:realtime:detailed';
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const [pageResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
dimensions: [{ name: 'unifiedScreenName' }],
metrics: [{ name: 'screenPageViews' }],
orderBy: [{ metric: { metricName: 'screenPageViews' }, desc: true }],
limit: 25,
});
const [eventResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
dimensions: [{ name: 'eventName' }],
metrics: [{ name: 'eventCount' }],
orderBy: [{ metric: { metricName: 'eventCount' }, desc: true }],
limit: 25,
});
const [deviceResponse] = await this.analyticsClient.runRealtimeReport({
property: `properties/${this.propertyId}`,
dimensions: [{ name: 'deviceCategory' }],
metrics: [{ name: 'activeUsers' }],
orderBy: [{ metric: { metricName: 'activeUsers' }, desc: true }],
limit: 10,
returnPropertyQuota: true,
});
const response = {
pageResponse,
eventResponse,
sourceResponse: deviceResponse,
};
await this._cacheSet(cacheKey, response, CACHE_DURATIONS.REALTIME_DETAILED);
return response;
}
async getUserBehavior(timeRange = '30') {
const cacheKey = `analytics:user_behavior:${timeRange}`;
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const [pageResponse] = await this.analyticsClient.runReport({
property: `properties/${this.propertyId}`,
dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
dimensions: [{ name: 'pagePath' }],
metrics: [
{ name: 'screenPageViews' },
{ name: 'averageSessionDuration' },
{ name: 'bounceRate' },
{ name: 'sessions' },
],
orderBy: [{ metric: { metricName: 'screenPageViews' }, desc: true }],
limit: 25,
});
const [deviceResponse] = await this.analyticsClient.runReport({
property: `properties/${this.propertyId}`,
dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
dimensions: [{ name: 'deviceCategory' }],
metrics: [{ name: 'screenPageViews' }, { name: 'sessions' }],
});
const [sourceResponse] = await this.analyticsClient.runReport({
property: `properties/${this.propertyId}`,
dateRanges: [{ startDate: `${timeRange}daysAgo`, endDate: 'today' }],
dimensions: [{ name: 'sessionSource' }],
metrics: [{ name: 'sessions' }, { name: 'conversions' }],
orderBy: [{ metric: { metricName: 'sessions' }, desc: true }],
limit: 25,
returnPropertyQuota: true,
});
const response = { pageResponse, deviceResponse, sourceResponse };
await this._cacheSet(cacheKey, response, CACHE_DURATIONS.USER_BEHAVIOR);
return response;
}
}
inventory-server/dashboard/services/klaviyo/campaigns.service.js
-206
View File
@@ -1,206 +0,0 @@
import fetch from 'node-fetch';
import { TimeManager } from '../../../shared/business-time/index.js';
import { RedisService } from './redis.service.js';
export class CampaignsService {
constructor(apiKey, apiRevision, redis) {
this.apiKey = apiKey;
this.apiRevision = apiRevision;
this.baseUrl = 'https://a.klaviyo.com/api';
this.timeManager = new TimeManager();
this.redisService = new RedisService(redis);
}
async getCampaigns(params = {}) {
try {
// Add request debouncing
const requestKey = JSON.stringify(params);
if (this._pendingRequests && this._pendingRequests[requestKey]) {
return this._pendingRequests[requestKey];
}
// Try to get from cache first
const cacheKey = this.redisService._getCacheKey('campaigns', params);
let cachedData = null;
try {
cachedData = await this.redisService.get(`${cacheKey}:raw`);
if (cachedData) {
return cachedData;
}
} catch (cacheError) {
console.warn('[CampaignsService] Cache error:', cacheError);
}
this._pendingRequests = this._pendingRequests || {};
this._pendingRequests[requestKey] = (async () => {
let allCampaigns = [];
let nextCursor = params.pageCursor;
let pageCount = 0;
const filter = params.filter || this._buildFilter(params);
do {
const queryParams = new URLSearchParams();
if (filter) {
queryParams.append('filter', filter);
}
queryParams.append('sort', params.sort || '-send_time');
if (nextCursor) {
queryParams.append('page[cursor]', nextCursor);
}
const url = `${this.baseUrl}/campaigns?${queryParams.toString()}`;
try {
const response = await fetch(url, {
method: 'GET',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
'revision': this.apiRevision
}
});
if (!response.ok) {
const errorData = await response.json();
console.error('[CampaignsService] API Error:', errorData);
throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
}
const responseData = await response.json();
allCampaigns = allCampaigns.concat(responseData.data || []);
pageCount++;
nextCursor = responseData.links?.next ?
new URL(responseData.links.next).searchParams.get('page[cursor]') : null;
if (nextCursor) {
await new Promise(resolve => setTimeout(resolve, 50));
}
} catch (fetchError) {
console.error('[CampaignsService] Fetch error:', fetchError);
throw fetchError;
}
} while (nextCursor);
const transformedCampaigns = this._transformCampaigns(allCampaigns);
const result = {
data: transformedCampaigns,
meta: {
total_count: transformedCampaigns.length,
page_count: pageCount
}
};
try {
const ttl = this.redisService._getTTL(params.timeRange);
await this.redisService.set(`${cacheKey}:raw`, result, ttl);
} catch (cacheError) {
console.warn('[CampaignsService] Cache set error:', cacheError);
}
delete this._pendingRequests[requestKey];
return result;
})();
return await this._pendingRequests[requestKey];
} catch (error) {
console.error('[CampaignsService] Error fetching campaigns:', error);
throw error;
}
}
_buildFilter(params) {
const filters = [];
if (params.startDate && params.endDate) {
const startUtc = this.timeManager.formatForAPI(params.startDate);
const endUtc = this.timeManager.formatForAPI(params.endDate);
filters.push(`greater-or-equal(send_time,${startUtc})`);
filters.push(`less-than(send_time,${endUtc})`);
}
if (params.status) {
filters.push(`equals(status,"${params.status}")`);
}
if (params.customFilters) {
filters.push(...params.customFilters);
}
return filters.length > 0 ? (filters.length > 1 ? `and(${filters.join(',')})` : filters[0]) : null;
}
async getCampaignsByTimeRange(timeRange, options = {}) {
const range = this.timeManager.getDateRange(timeRange);
if (!range) {
throw new Error('Invalid time range specified');
}
const params = {
timeRange,
startDate: range.start.toISO(),
endDate: range.end.toISO(),
...options
};
// Try to get from cache first
const cacheKey = this.redisService._getCacheKey('campaigns', params);
let cachedData = null;
try {
cachedData = await this.redisService.get(`${cacheKey}:raw`);
if (cachedData) {
return cachedData;
}
} catch (cacheError) {
console.warn('[CampaignsService] Cache error:', cacheError);
}
return this.getCampaigns(params);
}
_transformCampaigns(campaigns) {
if (!Array.isArray(campaigns)) {
console.warn('[CampaignsService] Campaigns is not an array:', campaigns);
return [];
}
return campaigns.map(campaign => {
try {
const stats = campaign.attributes?.campaign_message?.stats || {};
return {
id: campaign.id,
name: campaign.attributes?.name || "Unnamed Campaign",
subject: campaign.attributes?.campaign_message?.subject || "",
send_time: campaign.attributes?.send_time,
stats: {
delivery_rate: stats.delivery_rate || 0,
delivered: stats.delivered || 0,
recipients: stats.recipients || 0,
open_rate: stats.open_rate || 0,
opens_unique: stats.opens_unique || 0,
opens: stats.opens || 0,
clicks_unique: stats.clicks_unique || 0,
click_rate: stats.click_rate || 0,
click_to_open_rate: stats.click_to_open_rate || 0,
conversion_value: stats.conversion_value || 0,
conversion_uniques: stats.conversion_uniques || 0
}
};
} catch (error) {
console.error('[CampaignsService] Error transforming campaign:', error, campaign);
return {
id: campaign.id || 'unknown',
name: 'Error Processing Campaign',
stats: {}
};
}
});
}
}
inventory-server/dashboard/services/klaviyo/events.service.js
-2202
View File
File diff suppressed because it is too large Load Diff
inventory-server/dashboard/services/klaviyo/metrics.service.js
-38
View File
@@ -1,38 +0,0 @@
import fetch from 'node-fetch';
export class MetricsService {
constructor(apiKey, apiRevision) {
this.apiKey = apiKey;
this.apiRevision = apiRevision;
this.baseUrl = 'https://a.klaviyo.com/api';
}
async getMetrics() {
try {
const response = await fetch(`${this.baseUrl}/metrics/`, {
headers: {
'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
'revision': this.apiRevision,
'Content-Type': 'application/json',
'Accept': 'application/json'
}
});
if (!response.ok) {
const errorData = await response.json();
console.error('[MetricsService] API Error:', errorData);
throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
}
const data = await response.json();
// Sort the results by name before returning
if (data.data) {
data.data.sort((a, b) => a.attributes.name.localeCompare(b.attributes.name));
}
return data;
} catch (error) {
console.error('[MetricsService] Error fetching metrics:', error);
throw error;
}
}
}
inventory-server/dashboard/services/klaviyo/redis.service.js
-146
View File
@@ -1,146 +0,0 @@
// Klaviyo cache wrapper. Was a self-instantiating ioredis client per service in
// the standalone klaviyo-server; now accepts an injected client so the merged
// dashboard-server shares one connection across all vendors (Phase 4).
//
// Public surface kept identical to the original so the ~3K LOC of klaviyo
// service code (events/campaigns/reporting) needs no other changes:
// - get(key)
// - set(key, data, ttl)
// - _getCacheKey(type, params)
// - _getTTL(timeRange)
// - getEventData(type, params) / cacheEventData(type, params, data)
// - clearCache(params)
//
// Reads short-circuit to null when the client isn't ready; writes are no-ops.
// Same "Redis hiccup → fall through to upstream" behavior as before.
import { TimeManager } from '../../../shared/business-time/index.js';
export class RedisService {
constructor(redis) {
if (!redis) {
throw new Error('RedisService requires an ioredis client (Phase 4: injected, no longer self-constructed)');
}
this.client = redis;
this.timeManager = new TimeManager();
this.DEFAULT_TTL = 5 * 60;
}
get isConnected() {
// ioredis: 'wait' | 'reconnecting' | 'connecting' | 'connect' | 'ready' | 'close' | 'end'
return this.client.status === 'ready' || this.client.status === 'connect';
}
async get(key) {
if (!this.isConnected) return null;
try {
const data = await this.client.get(key);
return data ? JSON.parse(data) : null;
} catch (error) {
console.error('[RedisService] Error getting data:', error);
return null;
}
}
async set(key, data, ttl = this.DEFAULT_TTL) {
if (!this.isConnected) return;
try {
await this.client.setex(key, ttl, JSON.stringify(data));
} catch (error) {
console.error('[RedisService] Error setting data:', error);
}
}
_getCacheKey(type, params = {}) {
const {
timeRange,
startDate,
endDate,
metricId,
metric,
daily,
cacheKey,
isPreviousPeriod,
customFilters,
} = params;
let key = `klaviyo:${type}`;
if (type === 'stats:details') {
key += `:${metric || 'all'}`;
if (daily) key += ':daily';
if (customFilters?.length) {
const filterHash = customFilters.join('').replace(/[^a-zA-Z0-9]/g, '');
key += `:${filterHash}`;
}
}
if (cacheKey) {
key += `:${cacheKey}`;
} else if (timeRange) {
key += `:${timeRange}`;
if (metricId) key += `:${metricId}`;
if (isPreviousPeriod) key += ':prev';
} else if (startDate && endDate) {
key += `:custom:${startDate}:${endDate}`;
if (metricId) key += `:${metricId}`;
if (isPreviousPeriod) key += ':prev';
}
if (['pre_orders', 'local_pickup', 'on_hold'].includes(metric)) {
key += `:${metric}`;
}
return key;
}
_getTTL(timeRange) {
const TTL_MAP = {
today: 2 * 60,
yesterday: 30 * 60,
thisWeek: 5 * 60,
lastWeek: 60 * 60,
thisMonth: 10 * 60,
lastMonth: 2 * 60 * 60,
last7days: 5 * 60,
last30days: 15 * 60,
custom: 15 * 60,
};
return TTL_MAP[timeRange] || this.DEFAULT_TTL;
}
async getEventData(type, params) {
if (!this.isConnected) return null;
try {
const baseKey = this._getCacheKey('events', params);
return await this.get(`${baseKey}:${type}`);
} catch (error) {
console.error('[RedisService] Error getting event data:', error);
return null;
}
}
async cacheEventData(type, params, data) {
if (!this.isConnected) return;
try {
const ttl = this._getTTL(params.timeRange);
const baseKey = this._getCacheKey('events', params);
await this.set(`${baseKey}:${type}`, data, ttl);
} catch (error) {
console.error('[RedisService] Error caching event data:', error);
}
}
async clearCache(params = {}) {
if (!this.isConnected) return;
try {
const pattern = this._getCacheKey('events', params) + '*';
const keys = await this.client.keys(pattern);
if (keys.length > 0) {
await this.client.del(...keys);
}
} catch (error) {
console.error('[RedisService] Error clearing cache:', error);
}
}
}
inventory-server/dashboard/services/klaviyo/reporting.service.js
-254
View File
@@ -1,254 +0,0 @@
import fetch from 'node-fetch';
import { TimeManager } from '../../../shared/business-time/index.js';
import { RedisService } from './redis.service.js';
const METRIC_IDS = {
PLACED_ORDER: 'Y8cqcF'
};
export class ReportingService {
constructor(apiKey, apiRevision, redis) {
this.apiKey = apiKey;
this.apiRevision = apiRevision;
this.baseUrl = 'https://a.klaviyo.com/api';
this.timeManager = new TimeManager();
this.redisService = new RedisService(redis);
this._pendingReportRequest = null;
}
async getCampaignReports(params = {}) {
try {
// Check if there's a pending request
if (this._pendingReportRequest) {
console.log('[ReportingService] Using pending campaign report request');
return this._pendingReportRequest;
}
// Try to get from cache first
const cacheKey = this.redisService._getCacheKey('campaign_reports', params);
let cachedData = null;
try {
cachedData = await this.redisService.get(`${cacheKey}:raw`);
if (cachedData) {
console.log('[ReportingService] Using cached campaign report data');
return cachedData;
}
} catch (cacheError) {
console.warn('[ReportingService] Cache error:', cacheError);
}
// Create new request promise
this._pendingReportRequest = (async () => {
console.log('[ReportingService] Fetching fresh campaign report data');
const range = this.timeManager.getDateRange(params.timeRange || 'last30days');
// Determine which channels to fetch based on params
const channelsToFetch = params.channel === 'all' || !params.channel
? ['email', 'sms']
: [params.channel];
const allResults = [];
// Fetch each channel
for (const channel of channelsToFetch) {
const payload = {
data: {
type: "campaign-values-report",
attributes: {
timeframe: {
start: range.start.toISO(),
end: range.end.toISO()
},
statistics: [
"delivery_rate",
"delivered",
"recipients",
"open_rate",
"opens_unique",
"opens",
"click_rate",
"clicks_unique",
"click_to_open_rate",
"conversion_value",
"conversion_uniques"
],
conversion_metric_id: METRIC_IDS.PLACED_ORDER,
filter: `equals(send_channel,"${channel}")`
}
}
};
const response = await fetch(`${this.baseUrl}/campaign-values-reports`, {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
'revision': this.apiRevision
},
body: JSON.stringify(payload)
});
if (!response.ok) {
const errorData = await response.json();
console.error('[ReportingService] API Error:', errorData);
throw new Error(`Klaviyo API error: ${response.status} ${response.statusText}`);
}
const reportData = await response.json();
console.log(`[ReportingService] Raw ${channel} report data:`, JSON.stringify(reportData, null, 2));
// Get campaign IDs from the report
const campaignIds = reportData.data?.attributes?.results?.map(result =>
result.groupings?.campaign_id
).filter(Boolean) || [];
if (campaignIds.length > 0) {
// Get campaign details including send time and subject lines
const campaignDetails = await this.getCampaignDetails(campaignIds);
// Process results for this channel
const channelResults = reportData.data.attributes.results.map(result => {
const campaignId = result.groupings.campaign_id;
const details = campaignDetails.find(detail => detail.id === campaignId);
return {
id: campaignId,
name: details.attributes.name,
subject: details.attributes.subject,
send_time: details.attributes.send_time,
channel: channel, // Use the channel we're currently processing
stats: {
delivery_rate: result.statistics.delivery_rate,
delivered: result.statistics.delivered,
recipients: result.statistics.recipients,
open_rate: result.statistics.open_rate,
opens_unique: result.statistics.opens_unique,
opens: result.statistics.opens,
click_rate: result.statistics.click_rate,
clicks_unique: result.statistics.clicks_unique,
click_to_open_rate: result.statistics.click_to_open_rate,
conversion_value: result.statistics.conversion_value,
conversion_uniques: result.statistics.conversion_uniques
}
};
});
allResults.push(...channelResults);
}
}
// Sort all results by date
const enrichedData = {
data: allResults.sort((a, b) => {
const dateA = new Date(a.send_time);
const dateB = new Date(b.send_time);
return dateB - dateA; // Sort by date descending
})
};
console.log('[ReportingService] Enriched data:', JSON.stringify(enrichedData, null, 2));
// Cache the enriched response for 10 minutes
try {
await this.redisService.set(`${cacheKey}:raw`, enrichedData, 600);
} catch (cacheError) {
console.warn('[ReportingService] Cache set error:', cacheError);
}
return enrichedData;
})();
const result = await this._pendingReportRequest;
this._pendingReportRequest = null;
return result;
} catch (error) {
console.error('[ReportingService] Error fetching campaign reports:', error);
this._pendingReportRequest = null;
throw error;
}
}
async getCampaignDetails(campaignIds = []) {
if (!Array.isArray(campaignIds) || campaignIds.length === 0) {
return [];
}
const fetchWithTimeout = async (campaignId, retries = 3) => {
for (let i = 0; i < retries; i++) {
try {
const controller = new AbortController();
const timeoutId = setTimeout(() => controller.abort(), 10000); // 10 second timeout
const response = await fetch(
`${this.baseUrl}/campaigns/${campaignId}?include=campaign-messages`,
{
headers: {
'Accept': 'application/json',
'Authorization': `Klaviyo-API-Key ${this.apiKey}`,
'revision': this.apiRevision
},
signal: controller.signal
}
);
clearTimeout(timeoutId);
if (!response.ok) {
throw new Error(`Failed to fetch campaign ${campaignId}: ${response.status}`);
}
const data = await response.json();
if (!data.data) {
throw new Error(`Invalid response for campaign ${campaignId}`);
}
const message = data.included?.find(item => item.type === 'campaign-message');
console.log('[ReportingService] Campaign details for ID:', campaignId, {
send_channel: data.data.attributes.send_channel,
raw_attributes: data.data.attributes
});
return {
id: data.data.id,
type: data.data.type,
attributes: {
...data.data.attributes,
name: data.data.attributes.name,
send_time: data.data.attributes.send_time,
subject: message?.attributes?.content?.subject,
send_channel: data.data.attributes.send_channel || 'email'
}
};
} catch (error) {
if (i === retries - 1) throw error;
await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1))); // Exponential backoff
}
}
};
// Process in smaller chunks to avoid overwhelming the API
const chunkSize = 10;
const campaignDetails = [];
for (let i = 0; i < campaignIds.length; i += chunkSize) {
const chunk = campaignIds.slice(i, i + chunkSize);
const results = await Promise.all(
chunk.map(id => fetchWithTimeout(id).catch(error => {
console.error(`Failed to fetch campaign ${id}:`, error);
return null;
}))
);
campaignDetails.push(...results.filter(Boolean));
if (i + chunkSize < campaignIds.length) {
await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between chunks
}
}
return campaignDetails;
}
}
inventory-server/dashboard/services/meta/meta.service.js
-104
View File
@@ -1,104 +0,0 @@
// Meta (Facebook Ads) service — ESM conversion of meta-server/services/meta.service.js.
// No Redis caching (matches the original — Meta calls are cheap-enough; reach/spend
// rolls over once per request). Uses axios.
import axios from 'axios';
function getConfig() {
const version = process.env.META_API_VERSION || 'v21.0';
return {
baseUrl: `https://graph.facebook.com/${version}`,
accessToken: process.env.META_ACCESS_TOKEN,
adAccountId: process.env.META_AD_ACCOUNT_ID,
};
}
async function metaApiRequest(endpoint, params = {}) {
const { baseUrl, accessToken } = getConfig();
try {
const response = await axios.get(`${baseUrl}/${endpoint}`, {
params: {
access_token: accessToken,
time_zone: 'America/New_York',
...params,
},
});
return response.data;
} catch (error) {
console.error('Meta API Error:', {
message: error.message,
response: error.response?.data,
endpoint,
});
throw error;
}
}
export async function fetchCampaigns(since, until) {
const { adAccountId } = getConfig();
const campaigns = await metaApiRequest(`act_${adAccountId}/campaigns`, {
fields: [
'id',
'name',
'status',
'objective',
'daily_budget',
'lifetime_budget',
'adsets{daily_budget,lifetime_budget}',
`insights.time_range({'since':'${since}','until':'${until}'}).level(campaign){
spend,
impressions,
clicks,
ctr,
reach,
frequency,
cpm,
cpc,
actions,
action_values,
cost_per_action_type
}`,
].join(','),
limit: 100,
});
return campaigns.data.filter((c) => c.insights?.data?.[0]?.spend > 0);
}
export async function fetchAccountInsights(since, until) {
const { adAccountId } = getConfig();
const accountInsights = await metaApiRequest(`act_${adAccountId}/insights`, {
fields: 'reach,spend,impressions,clicks,ctr,cpm,actions,action_values',
time_range: JSON.stringify({ since, until }),
});
return accountInsights.data[0] || null;
}
export async function updateCampaignBudget(campaignId, budget) {
const { baseUrl, accessToken } = getConfig();
try {
const response = await axios.post(`${baseUrl}/${campaignId}`, {
access_token: accessToken,
daily_budget: budget * 100, // dollars → cents
});
return response.data;
} catch (error) {
console.error('Update campaign budget error:', error);
throw error;
}
}
export async function updateCampaignStatus(campaignId, action) {
const { baseUrl, accessToken } = getConfig();
try {
const status = action === 'pause' ? 'PAUSED' : 'ACTIVE';
const response = await axios.post(`${baseUrl}/${campaignId}`, {
access_token: accessToken,
status,
});
return response.data;
} catch (error) {
console.error('Update campaign status error:', error);
throw error;
}
}
inventory-server/dashboard/services/typeform/typeform.service.js
-80
View File
@@ -1,80 +0,0 @@
// Typeform service — ESM conversion of typeform-server/services/typeform.service.js.
// Phase 4: accepts injected ioredis client. node-redis v4 set syntax `{ EX: 300 }`
// translated to ioredis `setex(key, 300, val)`.
import axios from 'axios';
export class TypeformService {
constructor(redis) {
if (!redis) {
throw new Error('TypeformService requires an ioredis client (Phase 4: injected)');
}
this.redis = redis;
const token = process.env.TYPEFORM_ACCESS_TOKEN;
if (!token) {
console.warn('[Typeform] TYPEFORM_ACCESS_TOKEN not set — all calls will 401');
}
this.apiClient = axios.create({
baseURL: 'https://api.typeform.com',
headers: {
Authorization: `Bearer ${token}`,
'Content-Type': 'application/json',
},
});
}
get _redisReady() {
return this.redis.status === 'ready' || this.redis.status === 'connect';
}
async _cacheGet(key) {
if (!this._redisReady) return null;
try {
const raw = await this.redis.get(key);
return raw ? JSON.parse(raw) : null;
} catch (err) {
console.warn('[Typeform] cache get failed:', err.message);
return null;
}
}
async _cacheSet(key, value, ttlSec) {
if (!this._redisReady) return;
try {
await this.redis.setex(key, ttlSec, JSON.stringify(value));
} catch (err) {
console.warn('[Typeform] cache set failed:', err.message);
}
}
async getFormResponses(formId, params = {}) {
const cacheKey = `typeform:responses:${formId}:${JSON.stringify(params)}`;
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const response = await this.apiClient.get(`/forms/${formId}/responses`, { params });
const data = response.data;
await this._cacheSet(cacheKey, data, 300);
return data;
}
async getFormInsights(formId) {
const cacheKey = `typeform:insights:${formId}`;
const cached = await this._cacheGet(cacheKey);
if (cached) return cached;
const response = await this.apiClient.get(`/insights/${formId}/summary`);
const data = response.data;
await this._cacheSet(cacheKey, data, 300);
return data;
}
async getFormResponsesWithFilters(formId, { since, until, pageSize = 25, ...otherParams } = {}) {
const params = { page_size: pageSize, ...otherParams };
if (since) params.since = new Date(since).toISOString();
if (until) params.until = new Date(until).toISOString();
return this.getFormResponses(formId, params);
}
}
inventory-server/db/config-schema-new.sql
-196
View File
@@ -1,196 +0,0 @@
-- Create function for updating timestamps if it doesn't exist
CREATE OR REPLACE FUNCTION update_updated_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated = CURRENT_TIMESTAMP;
RETURN NEW;
END;
$$ language 'plpgsql';
-- Create function for updating updated_at timestamps
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = CURRENT_TIMESTAMP;
RETURN NEW;
END;
$$ language 'plpgsql';
-- Drop tables in reverse order of dependency
DROP TABLE IF EXISTS public.settings_product CASCADE;
DROP TABLE IF EXISTS public.settings_vendor CASCADE;
DROP TABLE IF EXISTS public.settings_global CASCADE;
-- Table Definition: settings_global
CREATE TABLE public.settings_global (
setting_key VARCHAR PRIMARY KEY,
setting_value VARCHAR NOT NULL,
description TEXT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
-- Table Definition: settings_vendor
CREATE TABLE public.settings_vendor (
vendor VARCHAR PRIMARY KEY, -- Matches products.vendor
default_lead_time_days INT,
default_days_of_stock INT,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
);
-- Index for faster lookups if needed (PK usually sufficient)
-- CREATE INDEX idx_settings_vendor_vendor ON public.settings_vendor(vendor);
-- Table Definition: settings_product
CREATE TABLE public.settings_product (
pid INT8 PRIMARY KEY,
lead_time_days INT, -- Overrides vendor/global
days_of_stock INT, -- Overrides vendor/global
safety_stock INT DEFAULT 0, -- Minimum desired stock level
forecast_method VARCHAR DEFAULT 'standard', -- e.g., 'standard', 'seasonal'
exclude_from_forecast BOOLEAN DEFAULT FALSE,
updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT fk_settings_product_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
);
-- Description: Inserts or updates standard default global settings.
-- Safe to rerun; will update existing keys with these default values.
-- Dependencies: `settings_global` table must exist.
-- Frequency: Run once initially, or rerun if you want to reset global defaults.
INSERT INTO public.settings_global (setting_key, setting_value, description) VALUES
('abc_revenue_threshold_a', '0.80', 'Revenue percentage for Class A (cumulative)'),
('abc_revenue_threshold_b', '0.95', 'Revenue percentage for Class B (cumulative)'),
('abc_calculation_basis', 'revenue_30d', 'Metric for ABC calc (revenue_30d, sales_30d, lifetime_revenue)'),
('abc_calculation_period', '30', 'Days period for ABC calculation if not lifetime'),
('default_forecast_method', 'standard', 'Default forecast method (standard, seasonal)'),
('default_lead_time_days', '14', 'Global default lead time in days'),
('default_days_of_stock', '30', 'Global default days of stock coverage target'),
-- Set default safety stock to 0 units. Can be overridden per product.
-- If you wanted safety stock in days, you'd store 'days' here and calculate units later.
('default_safety_stock_units', '0', 'Global default safety stock in units')
ON CONFLICT (setting_key) DO UPDATE SET
setting_value = EXCLUDED.setting_value,
description = EXCLUDED.description,
updated_at = CURRENT_TIMESTAMP; -- Update timestamp if default value changes
-- Description: Creates placeholder rows in `settings_vendor` for each unique vendor
-- found in the `products` table. Does NOT set specific overrides.
-- Safe to rerun; will NOT overwrite existing vendor settings.
-- Dependencies: `settings_vendor` table must exist, `products` table populated.
-- Frequency: Run once after initial product load, or periodically if new vendors are added.
INSERT INTO public.settings_vendor (
vendor,
default_lead_time_days,
default_days_of_stock
-- updated_at will use its default CURRENT_TIMESTAMP on insert
)
SELECT
DISTINCT p.vendor,
-- Explicitly cast NULL to INTEGER to resolve type mismatch
CAST(NULL AS INTEGER),
CAST(NULL AS INTEGER)
FROM
public.products p
WHERE
p.vendor IS NOT NULL
AND p.vendor <> '' -- Exclude blank vendors if necessary
ON CONFLICT (vendor) DO NOTHING; -- IMPORTANT: Do not overwrite existing vendor settings
SELECT COUNT(*) FROM public.settings_vendor; -- Verify rows were inserted
-- Description: Creates placeholder rows in `settings_product` for each unique product
-- found in the `products` table. Sets basic defaults but no specific overrides.
-- Safe to rerun; will NOT overwrite existing product settings.
-- Dependencies: `settings_product` table must exist, `products` table populated.
-- Frequency: Run once after initial product load, or periodically if new products are added.
INSERT INTO public.settings_product (
pid,
lead_time_days, -- NULL = Inherit from Vendor/Global
days_of_stock, -- NULL = Inherit from Vendor/Global
safety_stock, -- Default to 0 units initially
forecast_method, -- NULL = Inherit from Global ('standard')
exclude_from_forecast -- Default to FALSE
-- updated_at will use its default CURRENT_TIMESTAMP on insert
)
SELECT
p.pid,
CAST(NULL AS INTEGER), -- Explicitly cast NULL to INTEGER
CAST(NULL AS INTEGER), -- Explicitly cast NULL to INTEGER
COALESCE((SELECT setting_value::int FROM settings_global WHERE setting_key = 'default_safety_stock_units'), 0), -- Use global default safety stock units
CAST(NULL AS VARCHAR), -- Cast NULL to VARCHAR for forecast_method (already varchar, but explicit)
FALSE -- Default: Include in forecast
FROM
public.products p
ON CONFLICT (pid) DO NOTHING; -- IMPORTANT: Do not overwrite existing product-specific settings
-- History and status tables
CREATE TABLE IF NOT EXISTS calculate_history (
id BIGSERIAL PRIMARY KEY,
start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
end_time TIMESTAMP WITH TIME ZONE NULL,
duration_seconds INTEGER,
duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
total_products INTEGER DEFAULT 0,
total_orders INTEGER DEFAULT 0,
total_purchase_orders INTEGER DEFAULT 0,
processed_products INTEGER DEFAULT 0,
processed_orders INTEGER DEFAULT 0,
processed_purchase_orders INTEGER DEFAULT 0,
status calculation_status DEFAULT 'running',
error_message TEXT,
additional_info JSONB
);
CREATE TABLE IF NOT EXISTS calculate_status (
module_name text PRIMARY KEY,
last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS sync_status (
table_name TEXT PRIMARY KEY,
last_sync_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_sync_id BIGINT
);
CREATE TABLE IF NOT EXISTS import_history (
id BIGSERIAL PRIMARY KEY,
table_name VARCHAR(50) NOT NULL,
start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
end_time TIMESTAMP WITH TIME ZONE NULL,
duration_seconds INTEGER,
duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
records_added INTEGER DEFAULT 0,
records_updated INTEGER DEFAULT 0,
records_deleted INTEGER DEFAULT 0,
records_skipped INTEGER DEFAULT 0,
total_processed INTEGER DEFAULT 0,
is_incremental BOOLEAN DEFAULT FALSE,
status calculation_status DEFAULT 'running',
error_message TEXT,
additional_info JSONB
);
-- Create all indexes after tables are fully created
CREATE INDEX IF NOT EXISTS idx_last_calc ON calculate_status(last_calculation_timestamp);
CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status(last_sync_timestamp);
CREATE INDEX IF NOT EXISTS idx_table_time ON import_history(table_name, start_time);
CREATE INDEX IF NOT EXISTS idx_import_history_status ON import_history(status);
CREATE INDEX IF NOT EXISTS idx_calculate_history_status ON calculate_history(status);
-- Add comments for documentation
COMMENT ON TABLE import_history IS 'Tracks history of data import operations with detailed statistics';
COMMENT ON COLUMN import_history.records_deleted IS 'Number of records deleted during this import';
COMMENT ON COLUMN import_history.records_skipped IS 'Number of records skipped (e.g., unchanged, invalid)';
COMMENT ON COLUMN import_history.total_processed IS 'Total number of records examined/processed, including skipped';
COMMENT ON TABLE calculate_history IS 'Tracks history of metrics calculation runs with performance data';
COMMENT ON COLUMN calculate_history.duration_seconds IS 'Total duration of the calculation in seconds';
COMMENT ON COLUMN calculate_history.additional_info IS 'JSON object containing step timings, row counts, and other detailed metrics';
inventory-server/db/daily-deals-schema.sql
-17
View File
@@ -1,17 +0,0 @@
-- Daily Deals schema for local PostgreSQL
-- Synced from production MySQL product_daily_deals + product_current_prices
CREATE TABLE IF NOT EXISTS product_daily_deals (
deal_id serial PRIMARY KEY,
deal_date date NOT NULL,
pid bigint NOT NULL,
price_id bigint NOT NULL,
-- Denormalized from product_current_prices so we don't need to sync that whole table
deal_price numeric(10,3),
created_at timestamptz DEFAULT NOW(),
CONSTRAINT fk_daily_deals_pid FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_daily_deals_date ON product_daily_deals(deal_date);
CREATE INDEX IF NOT EXISTS idx_daily_deals_pid ON product_daily_deals(pid);
CREATE UNIQUE INDEX IF NOT EXISTS idx_daily_deals_unique ON product_daily_deals(deal_date, pid);
inventory-server/db/functions.sql
-249
View File
@@ -1,249 +0,0 @@
-- Custom PostgreSQL functions used by the metrics pipeline
-- These must exist in the database before running calculate-metrics-new.js
--
-- To install/update: psql -d inventory_db -f functions.sql
-- All functions use CREATE OR REPLACE so they are safe to re-run.
-- =============================================================================
-- safe_divide: Division helper that returns a default value instead of erroring
-- on NULL or zero denominators.
-- =============================================================================
CREATE OR REPLACE FUNCTION public.safe_divide(
numerator numeric,
denominator numeric,
default_value numeric DEFAULT NULL::numeric
)
RETURNS numeric
LANGUAGE plpgsql
IMMUTABLE
AS $function$
BEGIN
IF denominator IS NULL OR denominator = 0 THEN
RETURN default_value;
ELSE
RETURN numerator / denominator;
END IF;
END;
$function$;
-- =============================================================================
-- std_numeric: Standardized rounding helper for consistent numeric precision.
-- =============================================================================
CREATE OR REPLACE FUNCTION public.std_numeric(
value numeric,
precision_digits integer DEFAULT 2
)
RETURNS numeric
LANGUAGE plpgsql
IMMUTABLE
AS $function$
BEGIN
IF value IS NULL THEN
RETURN NULL;
ELSE
RETURN ROUND(value, precision_digits);
END IF;
END;
$function$;
-- =============================================================================
-- calculate_sales_velocity: Daily sales velocity adjusted for stockout days.
-- Ensures at least 14-day denominator for products with sales to avoid
-- inflated velocity from short windows.
-- =============================================================================
CREATE OR REPLACE FUNCTION public.calculate_sales_velocity(
sales_30d integer,
stockout_days_30d integer
)
RETURNS numeric
LANGUAGE plpgsql
IMMUTABLE
AS $function$
BEGIN
RETURN sales_30d /
NULLIF(
GREATEST(
30.0 - stockout_days_30d,
CASE
WHEN sales_30d > 0 THEN 14.0 -- If we have sales, ensure at least 14 days denominator
ELSE 30.0 -- If no sales, use full period
END
),
0
);
END;
$function$;
-- =============================================================================
-- get_weighted_avg_cost: Weighted average cost from receivings up to a given date.
-- Prefers receivings from the 365 days before p_date so decade-old costs don't
-- weigh equally with recent ones; falls back to the lifetime average when the
-- product had no receivings in that window.
-- =============================================================================
CREATE OR REPLACE FUNCTION public.get_weighted_avg_cost(
p_pid bigint,
p_date date
)
RETURNS numeric
LANGUAGE plpgsql
STABLE
AS $function$
DECLARE
weighted_cost NUMERIC;
BEGIN
SELECT
CASE
WHEN SUM(qty_each) > 0 THEN SUM(cost_each * qty_each) / SUM(qty_each)
ELSE NULL
END INTO weighted_cost
FROM receivings
WHERE pid = p_pid
AND received_date <= p_date
AND received_date > p_date - INTERVAL '365 days'
AND status != 'canceled';
IF weighted_cost IS NULL THEN
SELECT
CASE
WHEN SUM(qty_each) > 0 THEN SUM(cost_each * qty_each) / SUM(qty_each)
ELSE NULL
END INTO weighted_cost
FROM receivings
WHERE pid = p_pid
AND received_date <= p_date
AND status != 'canceled';
END IF;
RETURN weighted_cost;
END;
$function$;
-- =============================================================================
-- classify_demand_pattern: Classifies demand based on average demand and
-- coefficient of variation (CV). Standard inventory classification:
-- zero: no demand
-- stable: CV <= 0.2 (predictable, easy to forecast)
-- variable: CV <= 0.5 (some variability, still forecastable)
-- sporadic: low volume + high CV (intermittent demand)
-- lumpy: high volume + high CV (unpredictable bursts)
-- =============================================================================
CREATE OR REPLACE FUNCTION public.classify_demand_pattern(
avg_demand numeric,
cv numeric
)
RETURNS character varying
LANGUAGE plpgsql
IMMUTABLE
AS $function$
BEGIN
IF avg_demand IS NULL OR cv IS NULL THEN
RETURN NULL;
ELSIF avg_demand = 0 THEN
RETURN 'zero';
ELSIF cv <= 0.2 THEN
RETURN 'stable';
ELSIF cv <= 0.5 THEN
RETURN 'variable';
ELSIF avg_demand < 1.0 THEN
RETURN 'sporadic';
ELSE
RETURN 'lumpy';
END IF;
END;
$function$;
-- =============================================================================
-- detect_seasonal_pattern: Detects seasonality by comparing monthly average
-- sales across the last 12 months. Uses coefficient of variation across months
-- and peak-to-average ratio to classify patterns.
--
-- Returns:
-- seasonal_pattern: 'none', 'moderate', or 'strong'
-- seasonality_index: peak month avg / overall avg * 100 (100 = no seasonality)
-- peak_season: name of peak month (e.g. 'January'), or NULL if none
-- =============================================================================
CREATE OR REPLACE FUNCTION public.detect_seasonal_pattern(p_pid bigint)
RETURNS TABLE(seasonal_pattern character varying, seasonality_index numeric, peak_season character varying)
LANGUAGE plpgsql
STABLE
AS $function$
DECLARE
v_monthly_cv NUMERIC;
v_max_month_avg NUMERIC;
v_overall_avg NUMERIC;
v_monthly_stddev NUMERIC;
v_peak_month_num INT;
v_data_months INT;
v_seasonality_index NUMERIC;
v_seasonal_pattern VARCHAR;
v_peak_season VARCHAR;
BEGIN
-- Gather monthly average sales and peak month in a single query
SELECT
COUNT(*),
AVG(month_avg),
STDDEV(month_avg),
MAX(month_avg),
(ARRAY_AGG(mo ORDER BY month_avg DESC))[1]::INT
INTO v_data_months, v_overall_avg, v_monthly_stddev, v_max_month_avg, v_peak_month_num
FROM (
SELECT EXTRACT(MONTH FROM snapshot_date) AS mo, AVG(units_sold) AS month_avg
FROM daily_product_snapshots
WHERE pid = p_pid AND snapshot_date >= CURRENT_DATE - INTERVAL '365 days'
GROUP BY EXTRACT(MONTH FROM snapshot_date)
) monthly;
-- Need at least 3 months of data for meaningful seasonality detection
IF v_data_months < 3 OR v_overall_avg IS NULL OR v_overall_avg = 0 THEN
RETURN QUERY SELECT 'none'::VARCHAR, 100::NUMERIC, NULL::VARCHAR;
RETURN;
END IF;
-- CV of monthly averages
v_monthly_cv := v_monthly_stddev / v_overall_avg;
-- Seasonality index: peak month avg / overall avg * 100
v_seasonality_index := ROUND((v_max_month_avg / v_overall_avg * 100)::NUMERIC, 2);
IF v_monthly_cv > 0.5 AND v_seasonality_index > 150 THEN
v_seasonal_pattern := 'strong';
v_peak_season := TRIM(TO_CHAR(TO_DATE(v_peak_month_num::TEXT, 'MM'), 'Month'));
ELSIF v_monthly_cv > 0.3 AND v_seasonality_index > 120 THEN
v_seasonal_pattern := 'moderate';
v_peak_season := TRIM(TO_CHAR(TO_DATE(v_peak_month_num::TEXT, 'MM'), 'Month'));
ELSE
v_seasonal_pattern := 'none';
v_peak_season := NULL;
v_seasonality_index := 100;
END IF;
RETURN QUERY SELECT v_seasonal_pattern, v_seasonality_index, v_peak_season;
END;
$function$;
-- =============================================================================
-- category_hierarchy: Materialized view providing a recursive category tree
-- with ancestor paths for efficient rollup queries.
--
-- Refresh after category changes: REFRESH MATERIALIZED VIEW category_hierarchy;
-- =============================================================================
-- DROP MATERIALIZED VIEW IF EXISTS category_hierarchy;
-- CREATE MATERIALIZED VIEW category_hierarchy AS
-- WITH RECURSIVE cat_tree AS (
-- SELECT cat_id, name, type, parent_id,
-- cat_id AS root_id, 0 AS level, ARRAY[cat_id] AS path
-- FROM categories
-- WHERE parent_id IS NULL
-- UNION ALL
-- SELECT c.cat_id, c.name, c.type, c.parent_id,
-- ct.root_id, ct.level + 1, ct.path || c.cat_id
-- FROM categories c
-- JOIN cat_tree ct ON c.parent_id = ct.cat_id
-- )
-- SELECT cat_id, name, type, parent_id, root_id, level, path,
-- (SELECT array_agg(unnest ORDER BY unnest DESC)
-- FROM unnest(cat_tree.path) unnest
-- WHERE unnest <> cat_tree.cat_id) AS ancestor_ids
-- FROM cat_tree;
--
-- CREATE UNIQUE INDEX ON category_hierarchy (cat_id);
inventory-server/db/metrics-schema-new.sql
-343
View File
@@ -1,343 +0,0 @@
-- Drop tables in reverse order of dependency
DROP TABLE IF EXISTS public.product_metrics CASCADE;
DROP TABLE IF EXISTS public.daily_product_snapshots CASCADE;
-- Table Definition: daily_product_snapshots
CREATE TABLE public.daily_product_snapshots (
snapshot_date DATE NOT NULL,
pid INT8 NOT NULL,
sku VARCHAR, -- Copied for convenience
-- Inventory Metrics (End of Day / Last Snapshot of Day)
eod_stock_quantity INT NOT NULL DEFAULT 0,
eod_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- Increased precision
eod_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
eod_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
stockout_flag BOOLEAN NOT NULL DEFAULT FALSE,
-- Sales Metrics (Aggregated for the snapshot_date)
units_sold INT NOT NULL DEFAULT 0,
units_returned INT NOT NULL DEFAULT 0,
gross_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
discounts NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
returns_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
net_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- gross_revenue - discounts
cogs NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
gross_regular_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
profit NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- net_revenue - cogs
-- Receiving Metrics (Aggregated for the snapshot_date)
units_received INT NOT NULL DEFAULT 0,
cost_received NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
calculation_timestamp TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (snapshot_date, pid) -- Composite primary key
-- CONSTRAINT fk_daily_snapshot_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE -- FK Optional on snapshot table
);
-- Add Indexes for daily_product_snapshots
CREATE INDEX idx_daily_snapshot_pid_date ON public.daily_product_snapshots(pid, snapshot_date); -- Useful for product-specific time series
-- Table Definition: product_metrics
CREATE TABLE public.product_metrics (
pid INT8 PRIMARY KEY,
last_calculated TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Product Info (Copied for convenience/performance)
sku VARCHAR,
title VARCHAR,
brand VARCHAR,
vendor VARCHAR,
image_url VARCHAR, -- (e.g., products.image_175)
is_visible BOOLEAN,
is_replenishable BOOLEAN,
-- Additional product fields
barcode VARCHAR,
harmonized_tariff_code VARCHAR,
vendor_reference VARCHAR,
notions_reference VARCHAR,
line VARCHAR,
subline VARCHAR,
artist VARCHAR,
moq INT,
rating NUMERIC(10, 2),
reviews INT,
weight NUMERIC(14, 4),
length NUMERIC(14, 4),
width NUMERIC(14, 4),
height NUMERIC(14, 4),
country_of_origin VARCHAR,
location VARCHAR,
baskets INT,
notifies INT,
preorder_count INT,
notions_inv_count INT,
-- Current Status (Refreshed Hourly)
current_price NUMERIC(10, 2),
current_regular_price NUMERIC(10, 2),
current_cost_price NUMERIC(10, 4), -- Increased precision for cost
current_stock INT NOT NULL DEFAULT 0,
current_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
current_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
current_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
on_order_qty INT NOT NULL DEFAULT 0,
on_order_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
on_order_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
earliest_expected_date DATE,
-- total_received_lifetime INT NOT NULL DEFAULT 0, -- Can calc if needed
-- Historical Dates (Calculated Once/Periodically)
date_created DATE,
date_first_received DATE,
date_last_received DATE,
date_first_sold DATE,
date_last_sold DATE,
age_days INT, -- Calculated based on LEAST(date_created, date_first_sold)
-- Rolling Period Metrics (Refreshed Hourly from daily_product_snapshots)
sales_7d INT, revenue_7d NUMERIC(14, 4),
sales_14d INT, revenue_14d NUMERIC(14, 4),
sales_30d INT, revenue_30d NUMERIC(14, 4),
cogs_30d NUMERIC(14, 4), profit_30d NUMERIC(14, 4),
returns_units_30d INT, returns_revenue_30d NUMERIC(14, 4),
discounts_30d NUMERIC(14, 4),
gross_revenue_30d NUMERIC(14, 4), gross_regular_revenue_30d NUMERIC(14, 4),
stockout_days_30d INT,
sales_365d INT, revenue_365d NUMERIC(14, 4),
avg_stock_units_30d NUMERIC(10, 2), avg_stock_cost_30d NUMERIC(14, 4),
avg_stock_retail_30d NUMERIC(14, 4), avg_stock_gross_30d NUMERIC(14, 4),
received_qty_30d INT, received_cost_30d NUMERIC(14, 4),
-- Lifetime Metrics (Recalculated Hourly/Daily from daily_product_snapshots)
lifetime_sales INT,
lifetime_revenue NUMERIC(16, 4),
lifetime_revenue_quality VARCHAR(10), -- 'exact', 'partial', 'estimated'
-- First Period Metrics (Calculated Once/Periodically from daily_product_snapshots)
first_7_days_sales INT, first_7_days_revenue NUMERIC(14, 4),
first_30_days_sales INT, first_30_days_revenue NUMERIC(14, 4),
first_60_days_sales INT, first_60_days_revenue NUMERIC(14, 4),
first_90_days_sales INT, first_90_days_revenue NUMERIC(14, 4),
-- Calculated KPIs (Refreshed Hourly based on rolling metrics)
asp_30d NUMERIC(10, 2), -- revenue_30d / sales_30d
acp_30d NUMERIC(10, 4), -- cogs_30d / sales_30d
avg_ros_30d NUMERIC(10, 4), -- profit_30d / sales_30d
avg_sales_per_day_30d NUMERIC(10, 2), -- sales_30d / 30.0
avg_sales_per_month_30d NUMERIC(10, 2), -- sales_30d (assuming 30d = 1 month for this metric)
margin_30d NUMERIC(8, 2), -- (profit_30d / revenue_30d) * 100
markup_30d NUMERIC(8, 2), -- (profit_30d / cogs_30d) * 100
gmroi_30d NUMERIC(10, 2), -- profit_30d / avg_stock_cost_30d
stockturn_30d NUMERIC(10, 2), -- sales_30d / avg_stock_units_30d
return_rate_30d NUMERIC(8, 2), -- returns_units_30d / (sales_30d + returns_units_30d) * 100
discount_rate_30d NUMERIC(8, 2), -- discounts_30d / gross_revenue_30d * 100
stockout_rate_30d NUMERIC(8, 2), -- stockout_days_30d / 30.0 * 100
markdown_30d NUMERIC(14, 4), -- gross_regular_revenue_30d - gross_revenue_30d
markdown_rate_30d NUMERIC(8, 2), -- markdown_30d / gross_regular_revenue_30d * 100
sell_through_30d NUMERIC(8, 2), -- sales_30d / (current_stock + sales_30d) * 100
avg_lead_time_days INT, -- Calculated Periodically from purchase_orders
-- Forecasting & Replenishment (Refreshed Hourly)
abc_class CHAR(1), -- Updated Periodically (e.g., Weekly)
sales_velocity_daily NUMERIC(10, 4), -- sales_30d / (30.0 - stockout_days_30d)
config_lead_time INT, -- From settings tables
config_days_of_stock INT, -- From settings tables
config_safety_stock INT, -- From settings_product
planning_period_days INT, -- config_lead_time + config_days_of_stock
lead_time_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_lead_time
days_of_stock_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_days_of_stock
planning_period_forecast_units NUMERIC(10, 2), -- lead_time_forecast_units + days_of_stock_forecast_units
lead_time_closing_stock NUMERIC(10, 2), -- current_stock + on_order_qty - lead_time_forecast_units
days_of_stock_closing_stock NUMERIC(10, 2), -- lead_time_closing_stock - days_of_stock_forecast_units
replenishment_needed_raw NUMERIC(10, 2), -- planning_period_forecast_units + config_safety_stock - current_stock - on_order_qty
replenishment_units INT, -- CEILING(GREATEST(0, replenishment_needed_raw))
replenishment_cost NUMERIC(14, 4), -- replenishment_units * current_cost_price
replenishment_retail NUMERIC(14, 4), -- replenishment_units * current_price
replenishment_profit NUMERIC(14, 4), -- replenishment_units * (current_price - current_cost_price)
to_order_units INT, -- Apply MOQ/UOM logic to replenishment_units
forecast_lost_sales_units NUMERIC(10, 2), -- GREATEST(0, -lead_time_closing_stock)
forecast_lost_revenue NUMERIC(14, 4), -- forecast_lost_sales_units * current_price
stock_cover_in_days NUMERIC(10, 1), -- current_stock / sales_velocity_daily
po_cover_in_days NUMERIC(10, 1), -- on_order_qty / sales_velocity_daily
sells_out_in_days NUMERIC(10, 1), -- (current_stock + on_order_qty) / sales_velocity_daily
replenish_date DATE, -- Calc based on when stock hits safety stock minus lead time
overstocked_units INT, -- GREATEST(0, current_stock - config_safety_stock - planning_period_forecast_units)
overstocked_cost NUMERIC(14, 4), -- overstocked_units * current_cost_price
overstocked_retail NUMERIC(14, 4), -- overstocked_units * current_price
is_old_stock BOOLEAN, -- Based on age, last sold, last received, on_order status
-- Yesterday's Metrics (Refreshed Hourly from daily_product_snapshots)
yesterday_sales INT,
-- Product Status (Calculated from metrics)
status VARCHAR, -- Stores status values like: Critical, Reorder Soon, Healthy, Overstock, At Risk, New
-- Growth Metrics (P3)
sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
sales_growth_yoy NUMERIC(10, 2), -- Year-over-year sales growth %
revenue_growth_yoy NUMERIC(10, 2), -- Year-over-year revenue growth %
-- Demand Variability Metrics (P3)
sales_variance_30d NUMERIC(10, 2), -- Variance of daily sales
sales_std_dev_30d NUMERIC(10, 2), -- Standard deviation of daily sales
sales_cv_30d NUMERIC(10, 2), -- Coefficient of variation
demand_pattern VARCHAR(20), -- 'stable', 'variable', 'sporadic', 'lumpy'
-- Service Level & Fill Rate (P5)
fill_rate_30d NUMERIC(8, 2), -- % of demand fulfilled from stock
stockout_incidents_30d INT, -- Days with stockouts
service_level_30d NUMERIC(8, 2), -- % of days without stockouts
lost_sales_incidents_30d INT, -- Days with potential lost sales
-- Seasonality (P5)
seasonality_index NUMERIC(10, 2), -- Current vs average (100 = average)
seasonal_pattern VARCHAR(20), -- 'none', 'weekly', 'monthly', 'quarterly', 'yearly'
peak_season VARCHAR(20), -- e.g., 'Q4', 'summer', 'holiday'
CONSTRAINT fk_product_metrics_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
);
-- Add Indexes for product_metrics (adjust based on common filtering/sorting in frontend)
CREATE INDEX idx_product_metrics_brand ON public.product_metrics(brand);
CREATE INDEX idx_product_metrics_vendor ON public.product_metrics(vendor);
CREATE INDEX idx_product_metrics_sku ON public.product_metrics(sku);
CREATE INDEX idx_product_metrics_abc_class ON public.product_metrics(abc_class);
CREATE INDEX idx_product_metrics_revenue_30d ON public.product_metrics(revenue_30d DESC NULLS LAST); -- Example sorting index
CREATE INDEX idx_product_metrics_sales_30d ON public.product_metrics(sales_30d DESC NULLS LAST); -- Example sorting index
CREATE INDEX idx_product_metrics_current_stock ON public.product_metrics(current_stock);
CREATE INDEX idx_product_metrics_sells_out_in_days ON public.product_metrics(sells_out_in_days ASC NULLS LAST); -- Example sorting index
CREATE INDEX idx_product_metrics_status ON public.product_metrics(status); -- Index for status filtering
-- Add new vendor, category, and brand metrics tables
-- Drop tables in reverse order if they exist
DROP TABLE IF EXISTS public.brand_metrics CASCADE;
DROP TABLE IF EXISTS public.vendor_metrics CASCADE;
DROP TABLE IF EXISTS public.category_metrics CASCADE;
-- ========= Category Metrics =========
CREATE TABLE public.category_metrics (
category_id INT8 PRIMARY KEY, -- Foreign key to categories.cat_id
category_name VARCHAR, -- Denormalized for convenience
category_type INT2, -- Denormalized for convenience
parent_id INT8, -- Denormalized for convenience
last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- ROLLED-UP METRICS (includes this category + all descendants)
-- Counts & Basic Info
product_count INT NOT NULL DEFAULT 0, -- Total products linked
active_product_count INT NOT NULL DEFAULT 0, -- Visible products linked
replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products linked
-- Current Stock Value (approximated using current product costs/prices)
current_stock_units INT NOT NULL DEFAULT 0,
current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-- Rolling Period Aggregates (Summed from product_metrics)
sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-- DIRECT METRICS (only products directly in this category)
direct_product_count INT NOT NULL DEFAULT 0, -- Products directly in this category
direct_active_product_count INT NOT NULL DEFAULT 0, -- Visible products directly in this category
direct_replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products directly in this category
-- Direct Current Stock Value
direct_current_stock_units INT NOT NULL DEFAULT 0,
direct_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
direct_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-- Direct Rolling Period Aggregates
direct_sales_7d INT NOT NULL DEFAULT 0, direct_revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
direct_sales_30d INT NOT NULL DEFAULT 0, direct_revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
direct_profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, direct_cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
direct_sales_365d INT NOT NULL DEFAULT 0, direct_revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
direct_lifetime_sales INT NOT NULL DEFAULT 0, direct_lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-- Calculated KPIs (Based on 30d aggregates) - Apply to rolled-up metrics
avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
stock_turn_30d NUMERIC(10, 3), -- sales_units / avg_stock_units (Needs avg stock calc)
sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
CONSTRAINT fk_category_metrics_cat_id FOREIGN KEY (category_id) REFERENCES public.categories(cat_id) ON DELETE CASCADE ON UPDATE CASCADE
);
CREATE INDEX idx_category_metrics_name ON public.category_metrics(category_name);
CREATE INDEX idx_category_metrics_type ON public.category_metrics(category_type);
-- ========= Vendor Metrics =========
CREATE TABLE public.vendor_metrics (
vendor_name VARCHAR PRIMARY KEY, -- Matches products.vendor
last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- Counts & Basic Info
product_count INT NOT NULL DEFAULT 0, -- Total products from this vendor
active_product_count INT NOT NULL DEFAULT 0, -- Visible products
replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
-- Current Stock Value (approximated)
current_stock_units INT NOT NULL DEFAULT 0,
current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-- On Order Value
on_order_units INT NOT NULL DEFAULT 0,
on_order_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-- PO Performance (Simplified)
po_count_365d INT NOT NULL DEFAULT 0, -- Count of distinct POs created in last year
avg_lead_time_days INT, -- Calculated from received POs historically
-- Rolling Period Aggregates (Summed from product_metrics)
sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-- Calculated KPIs (Based on 30d aggregates)
avg_margin_30d NUMERIC(14, 4), -- (profit / revenue) * 100
sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
-- Add more KPIs if needed (e.g., avg product value, sell-through rate for vendor)
);
CREATE INDEX idx_vendor_metrics_active_count ON public.vendor_metrics(active_product_count);
-- ========= Brand Metrics =========
CREATE TABLE public.brand_metrics (
brand_name VARCHAR PRIMARY KEY, -- Matches products.brand (use 'Unbranded' for NULLs)
last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-- Counts & Basic Info
product_count INT NOT NULL DEFAULT 0, -- Total products of this brand
active_product_count INT NOT NULL DEFAULT 0, -- Visible products
replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
-- Current Stock Value (approximated)
current_stock_units INT NOT NULL DEFAULT 0,
current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-- Rolling Period Aggregates (Summed from product_metrics)
sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-- Calculated KPIs (Based on 30d aggregates)
avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
-- Add more KPIs if needed (e.g., avg product value, sell-through rate for brand)
);
CREATE INDEX idx_brand_metrics_active_count ON public.brand_metrics(active_product_count);
inventory-server/db/migrations/add-date-online-and-shop-score.sql
-20
View File
@@ -1,20 +0,0 @@
-- Migration: Add date_online and shop_score columns to products table
-- These fields are imported from production to improve newsletter recommendation accuracy:
-- date_online = products.date_ol in production (date product went live on the shop)
-- shop_score = products.score in production (sales-based popularity score)
--
-- After running this migration, do a full (non-incremental) import to backfill:
-- INCREMENTAL_UPDATE=false node scripts/import-from-prod.js
-- Add date_online column (production: products.date_ol)
ALTER TABLE products ADD COLUMN IF NOT EXISTS date_online TIMESTAMP WITH TIME ZONE;
-- Add shop_score column (production: products.score)
-- Using NUMERIC(10,2) to preserve the decimal precision from production
ALTER TABLE products ADD COLUMN IF NOT EXISTS shop_score NUMERIC(10, 2) DEFAULT 0;
-- If shop_score was previously created as INTEGER, convert it
ALTER TABLE products ALTER COLUMN shop_score TYPE NUMERIC(10, 2);
-- Index on date_online for the newsletter "new products" filter
CREATE INDEX IF NOT EXISTS idx_products_date_online ON products(date_online);
inventory-server/db/schema.sql
-305
View File
@@ -1,305 +0,0 @@
-- Enable strict error reporting
SET session_replication_role = 'replica'; -- Disable foreign key checks temporarily
-- Create function for updating timestamps
CREATE OR REPLACE FUNCTION update_updated_column() RETURNS TRIGGER AS $func$
BEGIN
-- Check which table is being updated and use the appropriate column
IF TG_TABLE_NAME = 'categories' THEN
NEW.updated_at = CURRENT_TIMESTAMP;
ELSIF TG_TABLE_NAME IN ('products', 'orders', 'purchase_orders', 'receivings') THEN
NEW.updated = CURRENT_TIMESTAMP;
END IF;
RETURN NEW;
END;
$func$ language plpgsql;
-- Create tables
CREATE TABLE products (
pid BIGINT NOT NULL,
title TEXT NOT NULL,
description TEXT,
sku TEXT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE,
date_online TIMESTAMP WITH TIME ZONE,
first_received TIMESTAMP WITH TIME ZONE,
stock_quantity INTEGER DEFAULT 0,
preorder_count INTEGER DEFAULT 0,
notions_inv_count INTEGER DEFAULT 0,
price NUMERIC(14, 4) NOT NULL,
regular_price NUMERIC(14, 4) NOT NULL,
cost_price NUMERIC(14, 4),
barcode TEXT,
harmonized_tariff_code TEXT,
updated_at TIMESTAMP WITH TIME ZONE,
visible BOOLEAN DEFAULT true,
managing_stock BOOLEAN DEFAULT true,
replenishable BOOLEAN DEFAULT true,
vendor TEXT,
vendor_reference TEXT,
notions_reference TEXT,
permalink TEXT,
categories TEXT,
image TEXT,
image_175 TEXT,
image_full TEXT,
brand TEXT,
line TEXT,
subline TEXT,
artist TEXT,
options TEXT,
tags TEXT,
moq INTEGER DEFAULT 1,
uom INTEGER DEFAULT 1,
rating NUMERIC(14, 4) DEFAULT 0.00,
reviews INTEGER DEFAULT 0,
weight NUMERIC(14, 4),
length NUMERIC(14, 4),
width NUMERIC(14, 4),
height NUMERIC(14, 4),
country_of_origin TEXT,
location TEXT,
total_sold INTEGER DEFAULT 0,
baskets INTEGER DEFAULT 0,
notifies INTEGER DEFAULT 0,
date_last_sold DATE,
shop_score NUMERIC(10, 2) DEFAULT 0,
updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (pid)
);
-- Create trigger for products
CREATE TRIGGER update_products_updated
BEFORE UPDATE ON products
FOR EACH ROW
EXECUTE FUNCTION update_updated_column();
-- Create indexes for products table
CREATE INDEX idx_products_sku ON products(sku);
CREATE INDEX idx_products_vendor ON products(vendor);
CREATE INDEX idx_products_brand ON products(brand);
CREATE INDEX idx_products_visible ON products(visible);
CREATE INDEX idx_products_replenishable ON products(replenishable);
CREATE INDEX idx_products_updated ON products(updated);
-- Create categories table with hierarchy support
CREATE TABLE categories (
cat_id BIGINT PRIMARY KEY,
name TEXT NOT NULL,
type SMALLINT NOT NULL,
parent_id BIGINT,
description TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
status TEXT DEFAULT 'active',
FOREIGN KEY (parent_id) REFERENCES categories(cat_id) ON DELETE SET NULL
);
-- Create trigger for categories
CREATE TRIGGER update_categories_updated_at
BEFORE UPDATE ON categories
FOR EACH ROW
EXECUTE FUNCTION update_updated_column();
COMMENT ON COLUMN categories.type IS '10=section, 11=category, 12=subcategory, 13=subsubcategory, 1=company, 2=line, 3=subline, 40=artist';
CREATE INDEX idx_categories_parent ON categories(parent_id);
CREATE INDEX idx_categories_type ON categories(type);
CREATE INDEX idx_categories_status ON categories(status);
CREATE INDEX idx_categories_name ON categories(name);
CREATE INDEX idx_categories_name_type ON categories(name, type);
-- Create product_categories junction table
CREATE TABLE product_categories (
cat_id BIGINT NOT NULL,
pid BIGINT NOT NULL,
PRIMARY KEY (pid, cat_id),
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
FOREIGN KEY (cat_id) REFERENCES categories(cat_id) ON DELETE CASCADE
);
CREATE INDEX idx_product_categories_category ON product_categories(cat_id);
-- Create orders table with its indexes
CREATE TABLE orders (
id BIGSERIAL PRIMARY KEY,
order_number TEXT NOT NULL,
pid BIGINT NOT NULL,
sku TEXT NOT NULL,
date TIMESTAMP WITH TIME ZONE NOT NULL,
price NUMERIC(14, 4) NOT NULL,
quantity INTEGER NOT NULL,
discount NUMERIC(14, 4) DEFAULT 0.0000,
tax NUMERIC(14, 4) DEFAULT 0.0000,
tax_included BOOLEAN DEFAULT false,
shipping NUMERIC(14, 4) DEFAULT 0.0000,
costeach NUMERIC(14, 4) DEFAULT 0.0000,
customer TEXT NOT NULL,
customer_name TEXT,
status TEXT DEFAULT 'pending',
canceled BOOLEAN DEFAULT false,
updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE (order_number, pid),
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE RESTRICT
);
-- Create trigger for orders
CREATE TRIGGER update_orders_updated
BEFORE UPDATE ON orders
FOR EACH ROW
EXECUTE FUNCTION update_updated_column();
CREATE INDEX idx_orders_number ON orders(order_number);
CREATE INDEX idx_orders_pid ON orders(pid);
CREATE INDEX idx_orders_sku ON orders(sku);
CREATE INDEX idx_orders_customer ON orders(customer);
CREATE INDEX idx_orders_date ON orders(date);
CREATE INDEX idx_orders_status ON orders(status);
CREATE INDEX idx_orders_pid_date ON orders(pid, date);
CREATE INDEX idx_orders_updated ON orders(updated);
-- Create purchase_orders table with its indexes
-- This table now focuses solely on purchase order intent, not receivings
CREATE TABLE purchase_orders (
id BIGSERIAL PRIMARY KEY,
po_id TEXT NOT NULL,
vendor TEXT NOT NULL,
date TIMESTAMP WITH TIME ZONE NOT NULL,
expected_date DATE,
pid BIGINT NOT NULL,
sku TEXT NOT NULL,
name TEXT NOT NULL,
po_cost_price NUMERIC(14, 4) NOT NULL,
status TEXT DEFAULT 'created',
notes TEXT,
long_note TEXT,
ordered INTEGER NOT NULL,
supplier_id INTEGER,
date_created TIMESTAMP WITH TIME ZONE,
date_ordered TIMESTAMP WITH TIME ZONE,
updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
UNIQUE (po_id, pid)
);
-- Create trigger for purchase_orders
CREATE TRIGGER update_purchase_orders_updated
BEFORE UPDATE ON purchase_orders
FOR EACH ROW
EXECUTE FUNCTION update_updated_column();
COMMENT ON COLUMN purchase_orders.name IS 'Product name from products.description';
COMMENT ON COLUMN purchase_orders.po_cost_price IS 'Original cost from PO';
COMMENT ON COLUMN purchase_orders.status IS 'canceled, created, electronically_ready_send, ordered, preordered, electronically_sent, receiving_started, done';
CREATE INDEX idx_po_id ON purchase_orders(po_id);
CREATE INDEX idx_po_sku ON purchase_orders(sku);
CREATE INDEX idx_po_vendor ON purchase_orders(vendor);
CREATE INDEX idx_po_status ON purchase_orders(status);
CREATE INDEX idx_po_expected_date ON purchase_orders(expected_date);
CREATE INDEX idx_po_pid_status ON purchase_orders(pid, status);
CREATE INDEX idx_po_pid_date ON purchase_orders(pid, date);
CREATE INDEX idx_po_updated ON purchase_orders(updated);
CREATE INDEX idx_po_supplier_id ON purchase_orders(supplier_id);
-- Create receivings table to track actual receipt of goods
CREATE TABLE receivings (
id BIGSERIAL PRIMARY KEY,
receiving_id TEXT NOT NULL,
pid BIGINT NOT NULL,
sku TEXT NOT NULL,
name TEXT NOT NULL,
vendor TEXT,
qty_each INTEGER NOT NULL,
qty_each_orig INTEGER,
cost_each NUMERIC(14, 5) NOT NULL,
cost_each_orig NUMERIC(14, 5),
received_by INTEGER,
received_by_name TEXT,
received_date TIMESTAMP WITH TIME ZONE NOT NULL,
receiving_created_date TIMESTAMP WITH TIME ZONE,
supplier_id INTEGER,
status TEXT DEFAULT 'created',
updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
UNIQUE (receiving_id, pid)
);
-- Create trigger for receivings
CREATE TRIGGER update_receivings_updated
BEFORE UPDATE ON receivings
FOR EACH ROW
EXECUTE FUNCTION update_updated_column();
COMMENT ON COLUMN receivings.status IS 'canceled, created, partial_received, full_received, paid';
COMMENT ON COLUMN receivings.qty_each_orig IS 'Original quantity from the source system';
COMMENT ON COLUMN receivings.cost_each_orig IS 'Original cost from the source system';
COMMENT ON COLUMN receivings.vendor IS 'Vendor name, same as in purchase_orders';
CREATE INDEX idx_receivings_id ON receivings(receiving_id);
CREATE INDEX idx_receivings_pid ON receivings(pid);
CREATE INDEX idx_receivings_sku ON receivings(sku);
CREATE INDEX idx_receivings_status ON receivings(status);
CREATE INDEX idx_receivings_received_date ON receivings(received_date);
CREATE INDEX idx_receivings_supplier_id ON receivings(supplier_id);
CREATE INDEX idx_receivings_vendor ON receivings(vendor);
CREATE INDEX idx_receivings_updated ON receivings(updated);
SET session_replication_role = 'origin'; -- Re-enable foreign key checks
-- Create views for common calculations
-- product_sales_trends view moved to metrics-schema.sql
-- -- Historical data tables imported from production
-- CREATE TABLE imported_product_current_prices (
-- price_id BIGSERIAL PRIMARY KEY,
-- pid BIGINT NOT NULL,
-- qty_buy SMALLINT NOT NULL,
-- is_min_qty_buy BOOLEAN NOT NULL,
-- price_each NUMERIC(10,3) NOT NULL,
-- qty_limit SMALLINT NOT NULL,
-- no_promo BOOLEAN NOT NULL,
-- checkout_offer BOOLEAN NOT NULL,
-- active BOOLEAN NOT NULL,
-- date_active TIMESTAMP WITH TIME ZONE,
-- date_deactive TIMESTAMP WITH TIME ZONE,
-- updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
-- );
-- CREATE INDEX idx_imported_product_current_prices_pid ON imported_product_current_prices(pid, active, qty_buy);
-- CREATE INDEX idx_imported_product_current_prices_checkout ON imported_product_current_prices(checkout_offer, active);
-- CREATE INDEX idx_imported_product_current_prices_deactive ON imported_product_current_prices(date_deactive, active);
-- CREATE INDEX idx_imported_product_current_prices_active ON imported_product_current_prices(date_active, active);
-- CREATE TABLE imported_daily_inventory (
-- date DATE NOT NULL,
-- pid BIGINT NOT NULL,
-- amountsold SMALLINT NOT NULL DEFAULT 0,
-- times_sold SMALLINT NOT NULL DEFAULT 0,
-- qtyreceived SMALLINT NOT NULL DEFAULT 0,
-- price NUMERIC(7,2) NOT NULL DEFAULT 0,
-- costeach NUMERIC(7,2) NOT NULL DEFAULT 0,
-- stamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (date, pid)
-- );
-- CREATE INDEX idx_imported_daily_inventory_pid ON imported_daily_inventory(pid);
-- CREATE TABLE imported_product_stat_history (
-- pid BIGINT NOT NULL,
-- date DATE NOT NULL,
-- score NUMERIC(10,2) NOT NULL,
-- score2 NUMERIC(10,2) NOT NULL,
-- qty_in_baskets SMALLINT NOT NULL,
-- qty_sold SMALLINT NOT NULL,
-- notifies_set SMALLINT NOT NULL,
-- visibility_score NUMERIC(10,2) NOT NULL,
-- health_score VARCHAR(5) NOT NULL,
-- sold_view_score NUMERIC(6,3) NOT NULL,
-- updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (pid, date)
-- );
-- CREATE INDEX idx_imported_product_stat_history_date ON imported_product_stat_history(date);
inventory-server/db/setup-schema.sql
-115
View File
@@ -1,115 +0,0 @@
-- Templates table for storing import templates
CREATE TABLE IF NOT EXISTS templates (
id SERIAL PRIMARY KEY,
company TEXT NOT NULL,
product_type TEXT NOT NULL,
supplier TEXT,
msrp DECIMAL(10,2),
cost_each DECIMAL(10,2),
qty_per_unit INTEGER,
case_qty INTEGER,
hts_code TEXT,
description TEXT,
weight DECIMAL(10,2),
length DECIMAL(10,2),
width DECIMAL(10,2),
height DECIMAL(10,2),
tax_cat TEXT,
size_cat TEXT,
categories TEXT[],
ship_restrictions TEXT[],
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
UNIQUE(company, product_type)
);
-- AI Prompts table for storing validation prompts
CREATE TABLE IF NOT EXISTS ai_prompts (
id SERIAL PRIMARY KEY,
prompt_text TEXT NOT NULL,
prompt_type TEXT NOT NULL CHECK (prompt_type IN ('general', 'company_specific', 'system')),
company TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT unique_company_prompt UNIQUE (company),
CONSTRAINT company_required_for_specific CHECK (
(prompt_type = 'general' AND company IS NULL) OR
(prompt_type = 'system' AND company IS NULL) OR
(prompt_type = 'company_specific' AND company IS NOT NULL)
)
);
-- Create a unique partial index to ensure only one general prompt
CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_general_prompt
ON ai_prompts (prompt_type)
WHERE prompt_type = 'general';
-- Create a unique partial index to ensure only one system prompt
CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_system_prompt
ON ai_prompts (prompt_type)
WHERE prompt_type = 'system';
-- Reusable Images table for storing persistent images
CREATE TABLE IF NOT EXISTS reusable_images (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
filename TEXT NOT NULL,
file_path TEXT NOT NULL,
image_url TEXT NOT NULL,
is_global BOOLEAN NOT NULL DEFAULT false,
company TEXT,
mime_type TEXT,
file_size INTEGER,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT company_required_for_non_global CHECK (
(is_global = true AND company IS NULL) OR
(is_global = false AND company IS NOT NULL)
)
);
-- Create index on company for efficient querying
CREATE INDEX IF NOT EXISTS idx_reusable_images_company ON reusable_images(company);
-- Create index on is_global for efficient querying
CREATE INDEX IF NOT EXISTS idx_reusable_images_is_global ON reusable_images(is_global);
-- AI Validation Performance Tracking
CREATE TABLE IF NOT EXISTS ai_validation_performance (
id SERIAL PRIMARY KEY,
prompt_length INTEGER NOT NULL,
product_count INTEGER NOT NULL,
start_time TIMESTAMP WITH TIME ZONE NOT NULL,
end_time TIMESTAMP WITH TIME ZONE NOT NULL,
duration_seconds DECIMAL(10,2) GENERATED ALWAYS AS (EXTRACT(EPOCH FROM (end_time - start_time))) STORED,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
-- Create index on prompt_length for efficient querying
CREATE INDEX IF NOT EXISTS idx_ai_validation_prompt_length ON ai_validation_performance(prompt_length);
-- Function to update the updated_at timestamp
CREATE OR REPLACE FUNCTION update_updated_at_column()
RETURNS TRIGGER AS $$
BEGIN
NEW.updated_at = CURRENT_TIMESTAMP;
RETURN NEW;
END;
$$ language 'plpgsql';
-- Trigger to automatically update the updated_at column
CREATE TRIGGER update_templates_updated_at
BEFORE UPDATE ON templates
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
-- Trigger to automatically update the updated_at column for ai_prompts
CREATE TRIGGER update_ai_prompts_updated_at
BEFORE UPDATE ON ai_prompts
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
-- Trigger to automatically update the updated_at column for reusable_images
CREATE TRIGGER update_reusable_images_updated_at
BEFORE UPDATE ON reusable_images
FOR EACH ROW
EXECUTE FUNCTION update_updated_at_column();
inventory-server/docs/TIME.md
-67
View File
@@ -1,67 +0,0 @@
# TIME.md — the business-time convention
Adopted 2026-06-12 (see `TIME_UNIFICATION_PLAN.md` at the repo root for the
audit and rollout). **Never hand-roll an hour offset.**
## The convention
**A business day runs 1:00am Eastern → 1:00am Eastern the next day.**
Because 1am Eastern == midnight Central year-round (both observe US DST),
this is identical to:
> **Business date = the America/Chicago calendar date.**
- `BUSINESS_TZ = 'America/Chicago'` — canonical, used for all boundary math.
- `OFFICE_TZ = 'America/New_York'` — display only (the office reads ET).
The single source of truth is `shared/business-time/index.js`
(`@inventory/shared/business-time`). The frontend mirror for the few
client-side needs is `inventory/src/utils/businessTime.ts`.
## Approved idioms
| Context | Correct idiom |
| --- | --- |
| PG `timestamptz` → business date | `(ts AT TIME ZONE 'America/Chicago')::date`, or plain `ts::date` / `CURRENT_DATE` (the session TZ is pinned to America/Chicago in every pool config AND as the `inventory_db` database default) |
| MySQL DATETIME literal (stores **Central wall-clock**) → business date | `DATE(col)` / `DATE_FORMAT(col, '%Y-%m-%d')` — **no hour shift** |
| MySQL WHERE bounds for business days | Central wall-clock strings via `toMySqlBound()`: `col >= ? AND col < ?` |
| MySQL hour-of-day for the office | `HOUR(ADDTIME(col, '01:00:00'))` — ET = CT + 1h year-round |
| Luxon | `dt.setZone('America/Chicago')` (real conversion). Never `keepLocalTime`, never a fixed `UTC-05:00` offset (wrong every winter) |
| Intervals | **Half-open** `[start, nextStart)`. No `-1 minute` / `-1 ms` endpoints (legacy exception: the `TimeManager` compat class keeps an inclusive −1ms end for its existing consumers) |
| "Today" in JS | `businessTodayStr()` / `businessToday()` from the shared module — never `new Date().toISOString().split('T')[0]` (UTC) |
| node-postgres DATE columns → string | `formatDateCol(value)` (local-time formatting round-trips correctly) or `::text` in SQL — never `.toISOString().split('T')[0]` (UTC shifts a day) |
| mysql2 driver config | `dateStrings: true` + parse with `parseMySql()` (zone Chicago). The import pipeline instead pins a **dynamic** Chicago offset (`currentChicagoOffset()` in `scripts/import-from-prod.js`) |
| Client → server | Named range presets, or **date-only strings** (`YYYY-MM-DD` = business dates). Never `toISOString()` of a browser-local midnight |
## Range vocabulary (both servers, one list)
`today, yesterday, twoDaysAgo, thisWeek, lastWeek, thisMonth, lastMonth,
last7days, last30days, last90days, previous7days, previous30days,
previous90days, custom`
`custom` takes `startDate`/`endDate` as date-only business dates (preferred)
or zoned ISO instants (used as-is, end exclusive). Calendar periods
("March 2026") mean the business month: Mar 1 1am ET → Apr 1 1am ET.
## Accepted deviations (documented, not fought)
- **GA4**: property TZ = America/New_York; `NdaysAgo` = calendar days,
midnight ET. Affects only 0:00–1:00am ET traffic attribution.
- **Meta insights**: `time_zone: 'America/New_York'` — same 1-hour deviation,
same note.
- **Klaviyo**: fetched with exact UTC bounds computed from our own range math
— fully on-convention. API returns UTC datetimes.
- **Pay periods** (payroll dashboard): 14-day periods aligned to **midnight
ET** Sundays — an HR display convention, intentionally not 1am-ET days.
## Infrastructure pins (belt-and-suspenders)
1. `ALTER DATABASE inventory_db SET timezone = 'America/Chicago'` (database default).
2. Every pg Pool passes `options: '-c TimeZone=America/Chicago'`
(`src/utils/db.js`, `shared/db/pg.js`, `scripts/metrics-new/utils/db.js`,
`scripts/calculate-metrics-new.js`, `scripts/import/utils.js`).
3. Node processes run with `TZ=America/Chicago`
(`/var/www/ecosystem.config.cjs` env blocks; `scripts/full-update.js` and
`scripts/forecast/run_forecast.js` set it themselves for cron contexts).
4. `forecast_engine.py` anchors on `business_today()` (zoneinfo Chicago).
inventory-server/migrations/001_create_import_sessions.sql
-29
View File
@@ -1,29 +0,0 @@
-- Migration: Create import_sessions table
-- Run this against your PostgreSQL database
CREATE TABLE IF NOT EXISTS import_sessions (
id SERIAL PRIMARY KEY,
user_id INTEGER NOT NULL,
name VARCHAR(255), -- NULL for unnamed/autosave sessions
current_step VARCHAR(50) NOT NULL, -- 'validation' | 'imageUpload'
data JSONB NOT NULL, -- Product rows
product_images JSONB, -- Image assignments
global_selections JSONB, -- Supplier, company, line, subline
validation_state JSONB, -- Errors, UPC status, generated item numbers
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Ensure only one unnamed session per user (autosave slot)
CREATE UNIQUE INDEX IF NOT EXISTS idx_unnamed_session_per_user
ON import_sessions (user_id)
WHERE name IS NULL;
-- Index for fast user lookups
CREATE INDEX IF NOT EXISTS idx_import_sessions_user_id
ON import_sessions (user_id);
-- Add comment for documentation
COMMENT ON TABLE import_sessions IS 'Stores in-progress product import sessions for users';
COMMENT ON COLUMN import_sessions.name IS 'Session name - NULL indicates the single unnamed/autosave session per user';
COMMENT ON COLUMN import_sessions.current_step IS 'Which step the user was on: validation or imageUpload';
inventory-server/migrations/002_ai_prompts_extensible_types.sql
-57
View File
@@ -1,57 +0,0 @@
-- Migration: Make AI prompts extensible with is_singleton column
-- Date: 2024-01-19
-- Description: Removes hardcoded prompt_type CHECK constraint, adds is_singleton column
-- for dynamic uniqueness enforcement, and creates appropriate indexes.
-- 1. Drop the old CHECK constraints on prompt_type (allows any string value now)
ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS ai_prompts_prompt_type_check;
ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS company_required_for_specific;
-- 2. Add is_singleton column (defaults to true for backwards compatibility)
ALTER TABLE ai_prompts ADD COLUMN IF NOT EXISTS is_singleton BOOLEAN NOT NULL DEFAULT true;
-- 3. Drop ALL old unique constraints and indexes (cleanup)
-- Some were created as CONSTRAINTS (via ADD CONSTRAINT), others as standalone indexes
-- Must drop constraints first, then remaining standalone indexes
-- Drop constraints (these also remove their backing indexes)
ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS unique_company_prompt;
ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS idx_unique_general_prompt;
ALTER TABLE ai_prompts DROP CONSTRAINT IF EXISTS idx_unique_system_prompt;
-- Drop standalone indexes (IF EXISTS handles cases where they don't exist)
DROP INDEX IF EXISTS idx_unique_general_prompt;
DROP INDEX IF EXISTS idx_unique_system_prompt;
DROP INDEX IF EXISTS idx_unique_name_validation_system;
DROP INDEX IF EXISTS idx_unique_name_validation_general;
DROP INDEX IF EXISTS idx_unique_description_validation_system;
DROP INDEX IF EXISTS idx_unique_description_validation_general;
DROP INDEX IF EXISTS idx_unique_sanity_check_system;
DROP INDEX IF EXISTS idx_unique_sanity_check_general;
DROP INDEX IF EXISTS idx_unique_bulk_validation_system;
DROP INDEX IF EXISTS idx_unique_bulk_validation_general;
DROP INDEX IF EXISTS idx_unique_name_validation_company;
DROP INDEX IF EXISTS idx_unique_description_validation_company;
DROP INDEX IF EXISTS idx_unique_bulk_validation_company;
-- 4. Create new partial unique indexes based on is_singleton
-- For singleton types WITHOUT company (only one per prompt_type)
CREATE UNIQUE INDEX IF NOT EXISTS idx_singleton_no_company
ON ai_prompts (prompt_type)
WHERE is_singleton = true AND company IS NULL;
-- For singleton types WITH company (only one per prompt_type + company combination)
CREATE UNIQUE INDEX IF NOT EXISTS idx_singleton_with_company
ON ai_prompts (prompt_type, company)
WHERE is_singleton = true AND company IS NOT NULL;
-- 5. Add index for fast lookups by type
CREATE INDEX IF NOT EXISTS idx_prompt_type ON ai_prompts (prompt_type);
-- NOTE: After running this migration, you should:
-- 1. Delete existing prompts with old types (general, system, company_specific)
-- 2. Create new prompts with the new type naming convention:
-- - name_validation_system, name_validation_general, name_validation_company_specific
-- - description_validation_system, description_validation_general, description_validation_company_specific
-- - sanity_check_system, sanity_check_general
-- - bulk_validation_system, bulk_validation_general, bulk_validation_company_specific
inventory-server/migrations/003_create_import_audit_log.sql
-53
View File
@@ -1,53 +0,0 @@
-- Migration: Create import_audit_log table
-- Permanent audit trail of all product import submissions sent to the API
-- Run this against your PostgreSQL database
CREATE TABLE IF NOT EXISTS import_audit_log (
id SERIAL PRIMARY KEY,
-- Who initiated the import
user_id INTEGER NOT NULL,
username VARCHAR(255),
-- What was submitted
product_count INTEGER NOT NULL,
request_payload JSONB NOT NULL, -- The exact JSON array of products sent to the API
environment VARCHAR(10) NOT NULL, -- 'dev' or 'prod'
target_endpoint VARCHAR(255), -- The API URL that was called
use_test_data_source BOOLEAN DEFAULT FALSE,
-- What came back
success BOOLEAN NOT NULL,
response_payload JSONB, -- Full API response
error_message TEXT, -- Extracted error message on failure
created_count INTEGER DEFAULT 0, -- Number of products successfully created
errored_count INTEGER DEFAULT 0, -- Number of products that errored
-- Metadata
session_id INTEGER, -- Optional link to the import_session used (if any)
duration_ms INTEGER, -- How long the API call took
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- Index for looking up logs by user
CREATE INDEX IF NOT EXISTS idx_import_audit_log_user_id
ON import_audit_log (user_id);
-- Index for filtering by success/failure
CREATE INDEX IF NOT EXISTS idx_import_audit_log_success
ON import_audit_log (success);
-- Index for time-based queries
CREATE INDEX IF NOT EXISTS idx_import_audit_log_created_at
ON import_audit_log (created_at DESC);
-- Composite index for user + time queries
CREATE INDEX IF NOT EXISTS idx_import_audit_log_user_created
ON import_audit_log (user_id, created_at DESC);
COMMENT ON TABLE import_audit_log IS 'Permanent audit log of all product import API submissions';
COMMENT ON COLUMN import_audit_log.request_payload IS 'Exact JSON products array sent to the external API';
COMMENT ON COLUMN import_audit_log.response_payload IS 'Full response received from the external API';
COMMENT ON COLUMN import_audit_log.environment IS 'dev or prod - which API endpoint was targeted';
COMMENT ON COLUMN import_audit_log.session_id IS 'Optional reference to import_sessions.id if session was active';
COMMENT ON COLUMN import_audit_log.duration_ms IS 'Round-trip time of the API call in milliseconds';
inventory-server/migrations/004_create_product_editor_audit_log.sql
-54
View File
@@ -1,54 +0,0 @@
-- Migration: Create product_editor_audit_log table
-- Permanent audit trail of all product editor API submissions
-- Run this against your PostgreSQL database
CREATE TABLE IF NOT EXISTS product_editor_audit_log (
id SERIAL PRIMARY KEY,
-- Who made the edit
user_id INTEGER NOT NULL,
username VARCHAR(255),
-- Which product
pid INTEGER NOT NULL,
-- What was submitted
action VARCHAR(50) NOT NULL, -- 'product_edit', 'image_changes', 'taxonomy_set'
request_payload JSONB NOT NULL, -- The exact payload sent to the external API
target_endpoint VARCHAR(255), -- The API URL that was called
-- What came back
success BOOLEAN NOT NULL,
response_payload JSONB, -- Full API response
error_message TEXT, -- Extracted error message on failure
-- Metadata
duration_ms INTEGER, -- How long the API call took
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- Index for looking up edits by product
CREATE INDEX IF NOT EXISTS idx_pe_audit_log_pid
ON product_editor_audit_log (pid);
-- Index for looking up edits by user
CREATE INDEX IF NOT EXISTS idx_pe_audit_log_user_id
ON product_editor_audit_log (user_id);
-- Index for time-based queries
CREATE INDEX IF NOT EXISTS idx_pe_audit_log_created_at
ON product_editor_audit_log (created_at DESC);
-- Composite index for product + time queries
CREATE INDEX IF NOT EXISTS idx_pe_audit_log_pid_created
ON product_editor_audit_log (pid, created_at DESC);
-- Composite index for user + time queries
CREATE INDEX IF NOT EXISTS idx_pe_audit_log_user_created
ON product_editor_audit_log (user_id, created_at DESC);
COMMENT ON TABLE product_editor_audit_log IS 'Permanent audit log of all product editor API submissions';
COMMENT ON COLUMN product_editor_audit_log.action IS 'Type of edit: product_edit, image_changes, or taxonomy_set';
COMMENT ON COLUMN product_editor_audit_log.request_payload IS 'Exact payload sent to the external API';
COMMENT ON COLUMN product_editor_audit_log.response_payload IS 'Full response received from the external API';
COMMENT ON COLUMN product_editor_audit_log.duration_ms IS 'Round-trip time of the API call in milliseconds';
inventory-server/migrations/005_phase6_permission_codes.sql
-52
View File
@@ -1,52 +0,0 @@
-- Phase 6.2: per-route permission codes
-- Seeds the permission codes referenced by Phase 6 hardening middleware.
-- Safe to run multiple times (ON CONFLICT DO NOTHING).
--
-- Codes follow the plan's spec (CONSOLIDATION_PLAN.md §6.2):
-- product_import — POST/PUT/DELETE on /api/import
-- data_management — POST/PUT/DELETE on /api/csv (data-management.js)
-- ai_admin — POST/PUT/DELETE on /api/ai-prompts, /api/ai-validation
-- templates_write — POST/PUT/DELETE on /api/templates
-- image_admin — POST/DELETE on /api/reusable-images
-- audit_read — reserved for future read-gating on audit logs
-- acot_admin — reserved for acot-server (Phase 5 scope)
-- klaviyo_* / meta_* / google_* / typeform_* — reserved for dashboard-server (Phase 4 scope)
--
-- Admin users (is_admin = true) automatically pass any requirePermission() check,
-- so this migration does not auto-grant codes to admins. New non-admin users get
-- write access only when explicitly granted via the user-management UI.
INSERT INTO permissions (code, name, category, description) VALUES
('product_import', 'Product Import (write)', 'Imports',
'Allows POST/PUT/DELETE on /api/import — uploads, deletes, generate-upc, etc.'),
('data_management', 'Data Management (write)', 'Data',
'Allows POST/PUT/DELETE on /api/csv — CSV operations, full updates, full resets.'),
('ai_admin', 'AI Settings Admin', 'AI',
'Allows write access to AI prompts and AI validation endpoints.'),
('templates_write', 'Template Editing', 'Templates',
'Allows POST/PUT/DELETE on /api/templates.'),
('image_admin', 'Image Management', 'Images',
'Allows uploads and deletions on /api/reusable-images.'),
('audit_read', 'Audit Log Access', 'Audit',
'Reserved for future read-gating of import + product-editor audit logs.'),
('klaviyo_write', 'Klaviyo Write', 'Dashboard',
'Reserved for dashboard-server: mutates Klaviyo lists/segments.'),
('klaviyo_admin', 'Klaviyo Admin', 'Dashboard',
'Reserved for dashboard-server: triggers campaign syncs.'),
('meta_write', 'Meta Write', 'Dashboard',
'Reserved for dashboard-server: Meta API write operations.'),
('google_write', 'Google Analytics Write', 'Dashboard',
'Reserved for dashboard-server: GA write operations.'),
('typeform_write', 'Typeform Write', 'Dashboard',
'Reserved for dashboard-server: Typeform write operations.'),
('acot_admin', 'ACOT Server Admin', 'ACOT',
'Reserved for acot-server admin endpoints.')
ON CONFLICT (code) DO NOTHING;
-- Phase 2 deviation #6 cleanup: drop defunct frontend permissions if present.
-- These corresponded to the removed Aircall/Gorgias dashboards.
DELETE FROM user_permissions
WHERE permission_id IN (
SELECT id FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls')
);
DELETE FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls');
inventory-server/migrations/006_permissions_ui_cleanup.sql
-175
View File
@@ -1,175 +0,0 @@
-- Permissions UI cleanup — 2026-05-28
--
-- WHAT THIS DOES
-- Rewrites permissions.name and permissions.category for clarity.
-- Consolidates 17 categories down to 10. Renames ambiguous entries so
-- the User Management UI reads cleanly. Does NOT touch permissions.code,
-- so every existing route gate (backend requirePermission and frontend
-- Protected page=) and every row in user_permissions keeps working
-- without any code change or remapping.
--
-- WHAT THIS DOES *NOT* DO
-- No DROP/DELETE of any permission (except in the optional block at the
-- very bottom — commented out by default). No changes to permissions.code.
-- No INSERT of new permissions. No changes to other tables.
--
-- SAFETY
-- Wrapped in a transaction. Run end-to-end; if any row count is wrong,
-- ROLLBACK and inspect. The auth middleware caches the user's loaded
-- permissions for 60s — after this runs, names refresh on next cache
-- miss for the admin UI. user_permissions joins by id, so granted
-- permissions remain granted across renames.
BEGIN;
------------------------------------------------------------------------
-- 1. Category consolidation
------------------------------------------------------------------------
-- 1a. Orphan "Pages" (just the Settings page) → Pages/Settings
UPDATE permissions SET category = 'Pages/Settings'
WHERE code = 'access:settings';
-- 1b. Existing settings:* tab-access permissions stay under a renamed
-- "Settings Tabs" category (was "Settings") so it reads less ambiguously
-- next to "Pages/Settings" and "Write Actions".
UPDATE permissions SET category = 'Settings Tabs'
WHERE category = 'Settings';
-- 1c. Dashboard widget perms keep their codes but get a clearer category name.
UPDATE permissions SET category = 'Dashboard Widgets'
WHERE category = 'Dashboard Components';
-- 1d. Collapse the 7 single-member "new permission" categories
-- (Imports, Data, AI, Templates, Images, Audit, ACOT, Dashboard
-- [the dashboard-server one — distinct from Pages/Dashboard])
-- into a single "Write Actions" category.
UPDATE permissions SET category = 'Write Actions'
WHERE category IN ('Imports', 'Data', 'AI', 'Templates',
'Images', 'Audit', 'ACOT', 'Dashboard');
-- 1e. The "Admin" category (just Show Debug) stays as-is — single member
-- but conceptually distinct enough that bucketing under Write Actions
-- would muddy the meaning. Leaving it alone.
------------------------------------------------------------------------
-- 2. Name renames for clarity
------------------------------------------------------------------------
-- 2a. Distinguish "page that lets you do X" from "permission to do X"
-- by suffixing the page-access permissions with " (page)" where the
-- name otherwise collides with the corresponding write permission.
UPDATE permissions SET name = 'Import Products (page)'
WHERE code = 'access:import';
UPDATE permissions SET name = 'Product Editor (page)'
WHERE code = 'access:product_editor';
UPDATE permissions SET name = 'Bulk Edit (page)'
WHERE code = 'access:bulk_edit';
-- 2b. Settings tab-access perms get a uniform "Settings: X Tab" name so
-- they sort together and read as "what you're seeing access to" not
-- "the feature itself."
UPDATE permissions SET name = 'Settings: Data Management Tab'
WHERE code = 'settings:data_management';
UPDATE permissions SET name = 'Settings: Reusable Images Tab'
WHERE code = 'settings:library_management';
UPDATE permissions SET name = 'Settings: AI Prompts Tab'
WHERE code = 'settings:prompt_management';
UPDATE permissions SET name = 'Settings: Templates Tab'
WHERE code = 'settings:templates';
UPDATE permissions SET name = 'Settings: User Management Tab'
WHERE code = 'settings:user_management';
UPDATE permissions SET name = 'Settings: Audit Log Tab'
WHERE code = 'settings:audit_log';
UPDATE permissions SET name = 'Settings: Global Tab'
WHERE code = 'settings:global';
UPDATE permissions SET name = 'Settings: Products Tab'
WHERE code = 'settings:products';
UPDATE permissions SET name = 'Settings: Vendors Tab'
WHERE code = 'settings:vendors';
-- 2c. Write-action perms get verb-leading names so it's obvious what
-- granting them actually allows.
UPDATE permissions
SET name = 'Product Import: Upload & Submit',
description = 'Allows POST/PUT/DELETE on /api/import — image uploads, '
|| 'product submission, deletions, and generate-upc. Does NOT '
|| 'grant access to the Import Products page (access:import).'
WHERE code = 'product_import';
UPDATE permissions
SET name = 'Data Management: Run Operations',
description = 'Allows POST/PUT/DELETE on /api/csv — CSV operations, '
|| 'full updates, full resets. Does NOT grant access to the '
|| 'Data Management settings tab (settings:data_management).'
WHERE code = 'data_management';
UPDATE permissions
SET name = 'Reusable Images: Upload & Delete',
description = 'Allows uploads and deletions on /api/reusable-images. '
|| 'Distinct from product_import (which gates uploads inside '
|| 'the product import flow).'
WHERE code = 'image_admin';
UPDATE permissions
SET name = 'Templates: Create & Edit',
description = 'Allows POST/PUT/DELETE on /api/templates.'
WHERE code = 'templates_write';
UPDATE permissions
SET name = 'AI: Edit Prompts & Validation',
description = 'Allows write access to /api/ai-prompts and /api/ai-validation.'
WHERE code = 'ai_admin';
UPDATE permissions
SET name = 'Klaviyo: Clear Cache',
description = 'Allows POST /api/klaviyo/events/clearCache.'
WHERE code = 'klaviyo_admin';
UPDATE permissions
SET name = 'Meta: Mutate Campaigns',
description = 'Allows PATCH/POST on /api/meta/campaigns/*.'
WHERE code = 'meta_write';
------------------------------------------------------------------------
-- 3. Verification — should all return non-zero
------------------------------------------------------------------------
-- Uncomment to inspect before commit:
-- SELECT category, COUNT(*) FROM permissions GROUP BY category ORDER BY category;
-- SELECT code, name, category FROM permissions
-- WHERE category IN ('Write Actions', 'Settings Tabs', 'Pages/Settings')
-- ORDER BY category, name;
COMMIT;
------------------------------------------------------------------------
-- 4. OPTIONAL — drop unused "Reserved for future" codes
------------------------------------------------------------------------
-- These five codes are referenced only in their own description ("Reserved
-- for…") and appear in NO route gate, NO Protected page=, and NO frontend
-- permissions.includes() check. Verified 2026-05-28.
--
-- Run this block separately if you want to drop them. user_permissions has
-- ON DELETE CASCADE on permission_id is NOT configured (only on user_id),
-- so we must clear user_permissions rows first.
--
-- BEGIN;
-- DELETE FROM user_permissions
-- WHERE permission_id IN (SELECT id FROM permissions
-- WHERE code IN ('klaviyo_write', 'google_write',
-- 'typeform_write', 'acot_admin',
-- 'audit_read'));
-- DELETE FROM permissions
-- WHERE code IN ('klaviyo_write', 'google_write', 'typeform_write',
-- 'acot_admin', 'audit_read');
-- COMMIT;
inventory-server/old/backfill-snapshots.js
-426
View File
@@ -1,426 +0,0 @@
const path = require('path');
const fs = require('fs');
const progress = require('../scripts/metrics-new/utils/progress'); // Assuming progress utils are here
const { getConnection, closePool } = require('../scripts/metrics-new/utils/db'); // Assuming db utils are here
const os = require('os'); // For detecting number of CPU cores
// --- Configuration ---
const BATCH_SIZE_DAYS = 1; // Process 1 day per database function call
const SQL_FUNCTION_FILE = path.resolve(__dirname, 'backfill_historical_snapshots.sql'); // Correct path
const LOG_PROGRESS_INTERVAL_MS = 5000; // Update console progress roughly every 5 seconds
const HISTORY_TYPE = 'backfill_snapshots'; // Identifier for history table
const MAX_WORKERS = Math.max(1, Math.floor(os.cpus().length / 2)); // Use half of available CPU cores
const USE_PARALLEL = false; // Set to true to enable parallel processing
const PG_STATEMENT_TIMEOUT_MS = 1800000; // 30 minutes max per query
// --- Cancellation Handling ---
let isCancelled = false;
let runningQueryPromise = null; // To potentially track the active query
function requestCancellation() {
if (!isCancelled) {
isCancelled = true;
console.warn('\nCancellation requested. Finishing current batch then stopping...');
// Note: We are NOT forcefully cancelling the backend query anymore.
}
}
process.on('SIGINT', requestCancellation); // Handle Ctrl+C
process.on('SIGTERM', requestCancellation); // Handle termination signals
// --- Main Backfill Function ---
async function backfillSnapshots(cmdStartDate, cmdEndDate, cmdStartBatch = 1) {
let connection;
const overallStartTime = Date.now();
let calculateHistoryId = null;
let processedDaysTotal = 0; // Track total days processed across all batches executed in this run
let currentBatchNum = cmdStartBatch > 0 ? cmdStartBatch : 1;
let totalBatches = 0; // Initialize totalBatches
let totalDays = 0; // Initialize totalDays
console.log(`Starting snapshot backfill process...`);
console.log(`SQL Function definition file: ${SQL_FUNCTION_FILE}`);
if (!fs.existsSync(SQL_FUNCTION_FILE)) {
console.error(`FATAL: SQL file not found at ${SQL_FUNCTION_FILE}`);
process.exit(1); // Exit early if file doesn't exist
}
try {
// Set up a connection with higher memory limits
connection = await getConnection({
// Add performance-related settings
application_name: 'backfill_snapshots',
statement_timeout: PG_STATEMENT_TIMEOUT_MS, // 30 min timeout per statement
// These parameters may need to be configured in your database:
// work_mem: '1GB',
// maintenance_work_mem: '2GB',
// temp_buffers: '1GB',
});
console.log('Database connection acquired.');
// --- Ensure Function Exists ---
console.log('Ensuring database function is up-to-date...');
try {
const sqlFunctionDef = fs.readFileSync(SQL_FUNCTION_FILE, 'utf8');
if (!sqlFunctionDef.includes('CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final')) {
throw new Error(`SQL file ${SQL_FUNCTION_FILE} does not seem to contain the function definition.`);
}
await connection.query(sqlFunctionDef); // Execute the whole file
console.log('Database function `backfill_daily_snapshots_range_final` created/updated.');
// Add performance query hints to the database
await connection.query(`
-- Analyze tables for better query planning
ANALYZE public.products;
ANALYZE public.imported_daily_inventory;
ANALYZE public.imported_product_stat_history;
ANALYZE public.daily_product_snapshots;
ANALYZE public.imported_product_current_prices;
`).catch(err => {
// Non-fatal if analyze fails
console.warn('Failed to analyze tables (non-fatal):', err.message);
});
} catch (err) {
console.error(`Error processing SQL function file ${SQL_FUNCTION_FILE}:`, err);
throw new Error(`Failed to create or replace DB function: ${err.message}`);
}
// --- Prepare History Record ---
console.log('Preparing calculation history record...');
// Ensure history table exists (optional, could be done elsewhere)
await connection.query(`
CREATE TABLE IF NOT EXISTS public.calculate_history (
id SERIAL PRIMARY KEY,
start_time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
end_time TIMESTAMPTZ,
duration_seconds INTEGER,
status VARCHAR(20) NOT NULL, -- e.g., 'running', 'completed', 'failed', 'cancelled'
error_message TEXT,
additional_info JSONB -- Store type, file, batch info etc.
);
`);
// Mark previous runs of this type as potentially failed if they were left 'running'
await connection.query(`
UPDATE public.calculate_history
SET status = 'failed', error_message = 'Interrupted by new run.'
WHERE status = 'running' AND additional_info->>'type' = $1;
`, [HISTORY_TYPE]);
// Create new history record
const historyResult = await connection.query(`
INSERT INTO public.calculate_history (start_time, status, additional_info)
VALUES (NOW(), 'running', jsonb_build_object('type', $1::text, 'sql_file', $2::text, 'start_batch', $3::integer))
RETURNING id;
`, [HISTORY_TYPE, path.basename(SQL_FUNCTION_FILE), cmdStartBatch]);
calculateHistoryId = historyResult.rows[0].id;
console.log(`Calculation history record created with ID: ${calculateHistoryId}`);
// --- Determine Date Range ---
console.log('Determining date range...');
let effectiveStartDate, effectiveEndDate;
// Use command-line dates if provided, otherwise query DB
if (cmdStartDate) {
effectiveStartDate = cmdStartDate;
} else {
const minDateResult = await connection.query(`
SELECT LEAST(
COALESCE((SELECT MIN(date) FROM public.imported_daily_inventory WHERE date > '1970-01-01'), CURRENT_DATE),
COALESCE((SELECT MIN(date) FROM public.imported_product_stat_history WHERE date > '1970-01-01'), CURRENT_DATE)
)::date as min_date;
`);
effectiveStartDate = minDateResult.rows[0]?.min_date || new Date().toISOString().split('T')[0]; // Fallback
console.log(`Auto-detected start date: ${effectiveStartDate}`);
}
if (cmdEndDate) {
effectiveEndDate = cmdEndDate;
} else {
const maxDateResult = await connection.query(`
SELECT GREATEST(
COALESCE((SELECT MAX(date) FROM public.imported_daily_inventory WHERE date < CURRENT_DATE), '1970-01-01'::date),
COALESCE((SELECT MAX(date) FROM public.imported_product_stat_history WHERE date < CURRENT_DATE), '1970-01-01'::date)
)::date as max_date;
`);
// Ensure end date is not today or in the future
effectiveEndDate = maxDateResult.rows[0]?.max_date || new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Default yesterday
if (new Date(effectiveEndDate) >= new Date(new Date().toISOString().split('T')[0])) {
effectiveEndDate = new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Set to yesterday if >= today
}
console.log(`Auto-detected end date: ${effectiveEndDate}`);
}
// Validate dates
const dStart = new Date(effectiveStartDate);
const dEnd = new Date(effectiveEndDate);
if (isNaN(dStart.getTime()) || isNaN(dEnd.getTime()) || dStart > dEnd) {
throw new Error(`Invalid date range: Start "${effectiveStartDate}", End "${effectiveEndDate}"`);
}
// --- Batch Processing ---
totalDays = Math.ceil((dEnd - dStart) / (1000 * 60 * 60 * 24)) + 1; // Inclusive
totalBatches = Math.ceil(totalDays / BATCH_SIZE_DAYS);
console.log(`Target Date Range: ${effectiveStartDate} to ${effectiveEndDate} (${totalDays} days)`);
console.log(`Total Batches: ${totalBatches} (Batch Size: ${BATCH_SIZE_DAYS} days)`);
console.log(`Starting from Batch: ${currentBatchNum}`);
// Initial progress update
progress.outputProgress({
status: 'running',
operation: 'Starting Batch Processing',
currentBatch: currentBatchNum,
totalBatches: totalBatches,
totalDays: totalDays,
elapsed: '0s',
remaining: 'Calculating...',
rate: 0,
historyId: calculateHistoryId // Include history ID in the object
});
while (currentBatchNum <= totalBatches && !isCancelled) {
const batchOffset = (currentBatchNum - 1) * BATCH_SIZE_DAYS;
const batchStartDate = new Date(dStart);
batchStartDate.setDate(dStart.getDate() + batchOffset);
const batchEndDate = new Date(batchStartDate);
batchEndDate.setDate(batchStartDate.getDate() + BATCH_SIZE_DAYS - 1);
// Clamp batch end date to the overall effective end date
if (batchEndDate > dEnd) {
batchEndDate.setTime(dEnd.getTime());
}
const batchStartDateStr = batchStartDate.toISOString().split('T')[0];
const batchEndDateStr = batchEndDate.toISOString().split('T')[0];
const batchStartTime = Date.now();
console.log(`\n--- Processing Batch ${currentBatchNum} / ${totalBatches} ---`);
console.log(` Dates: ${batchStartDateStr} to ${batchEndDateStr}`);
// Execute the function for the batch
try {
progress.outputProgress({
status: 'running',
operation: `Executing DB function for batch ${currentBatchNum}...`,
currentBatch: currentBatchNum,
totalBatches: totalBatches,
totalDays: totalDays,
elapsed: progress.formatElapsedTime(overallStartTime),
remaining: 'Executing...',
rate: 0,
historyId: calculateHistoryId
});
// Performance improvement: Add batch processing hint
await connection.query('SET LOCAL enable_parallel_append = on; SET LOCAL enable_parallel_hash = on; SET LOCAL max_parallel_workers_per_gather = 4;');
// Store promise in case we need to try and cancel (though not implemented forcefully)
runningQueryPromise = connection.query(
`SELECT backfill_daily_snapshots_range_final($1::date, $2::date);`,
[batchStartDateStr, batchEndDateStr]
);
await runningQueryPromise; // Wait for the function call to complete
runningQueryPromise = null; // Clear the promise
const batchDurationMs = Date.now() - batchStartTime;
const daysInThisBatch = Math.ceil((batchEndDate - batchStartDate) / (1000 * 60 * 60 * 24)) + 1;
processedDaysTotal += daysInThisBatch;
console.log(` Batch ${currentBatchNum} completed in ${progress.formatElapsedTime(batchStartTime)}.`);
// --- Update Progress & History ---
const overallElapsedSec = Math.round((Date.now() - overallStartTime) / 1000);
progress.outputProgress({
status: 'running',
operation: `Completed batch ${currentBatchNum}`,
currentBatch: currentBatchNum,
totalBatches: totalBatches,
totalDays: totalDays,
processedDays: processedDaysTotal,
elapsed: progress.formatElapsedTime(overallStartTime),
remaining: progress.estimateRemaining(overallStartTime, processedDaysTotal, totalDays),
rate: progress.calculateRate(overallStartTime, processedDaysTotal),
batchDuration: progress.formatElapsedTime(batchStartTime),
historyId: calculateHistoryId
});
// Save checkpoint in history
await connection.query(`
UPDATE public.calculate_history
SET additional_info = jsonb_set(additional_info, '{last_completed_batch}', $1::jsonb)
|| jsonb_build_object('last_processed_date', $2::text)
WHERE id = $3::integer;
`, [JSON.stringify(currentBatchNum), batchEndDateStr, calculateHistoryId]);
} catch (batchError) {
console.error(`\n--- ERROR in Batch ${currentBatchNum} (${batchStartDateStr} to ${batchEndDateStr}) ---`);
console.error(' Database Error:', batchError.message);
console.error(' DB Error Code:', batchError.code);
// Log detailed error to history and re-throw to stop the process
await connection.query(`
UPDATE public.calculate_history
SET status = 'failed',
end_time = NOW(),
duration_seconds = $1::integer,
error_message = $2::text,
additional_info = additional_info || jsonb_build_object('failed_batch', $3::integer, 'failed_date_range', $4::text)
WHERE id = $5::integer;
`, [
Math.round((Date.now() - overallStartTime) / 1000),
`Batch ${currentBatchNum} failed: ${batchError.message} (Code: ${batchError.code || 'N/A'})`,
currentBatchNum,
`${batchStartDateStr} to ${batchEndDateStr}`,
calculateHistoryId
]);
throw batchError; // Stop execution
}
currentBatchNum++;
// Optional delay between batches
// await new Promise(resolve => setTimeout(resolve, 500));
} // End while loop
// --- Final Outcome ---
const finalStatus = isCancelled ? 'cancelled' : 'completed';
const finalMessage = isCancelled ? `Calculation stopped after completing batch ${currentBatchNum - 1}.` : 'Historical snapshots backfill completed successfully.';
const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
console.log(`\n--- Backfill ${finalStatus.toUpperCase()} ---`);
console.log(finalMessage);
console.log(`Total duration: ${progress.formatElapsedTime(overallStartTime)}`);
// Update history record
await connection.query(`
UPDATE public.calculate_history SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3
WHERE id = $4::integer;
`, [finalStatus, finalDurationSec, (isCancelled ? 'User cancelled' : null), calculateHistoryId]);
if (!isCancelled) {
progress.clearProgress(); // Clear progress state only on successful completion
} else {
progress.outputProgress({ // Final cancelled status update
status: 'cancelled',
operation: finalMessage,
currentBatch: currentBatchNum - 1,
totalBatches: totalBatches,
totalDays: totalDays,
processedDays: processedDaysTotal,
elapsed: progress.formatElapsedTime(overallStartTime),
remaining: 'Cancelled',
rate: 0,
historyId: calculateHistoryId
});
}
return { success: true, status: finalStatus, message: finalMessage, duration: finalDurationSec };
} catch (error) {
console.error('\n--- Backfill encountered an unrecoverable error ---');
console.error(error.message);
const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
// Update history if possible
if (connection && calculateHistoryId) {
try {
await connection.query(`
UPDATE public.calculate_history
SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3::text
WHERE id = $4::integer;
`, [
isCancelled ? 'cancelled' : 'failed',
finalDurationSec,
error.message,
calculateHistoryId
]);
} catch (histError) {
console.error("Failed to update history record with error state:", histError);
}
} else {
console.error("Could not update history record (no ID or connection).");
}
// FIX: Use initialized value or a default if loop never started
const batchNumForError = currentBatchNum > cmdStartBatch ? currentBatchNum - 1 : cmdStartBatch - 1;
// Update progress.outputProgress call to match actual function signature
try {
// Create progress data object
const progressData = {
status: 'failed',
operation: 'Backfill failed',
message: error.message,
currentBatch: batchNumForError,
totalBatches: totalBatches,
totalDays: totalDays,
processedDays: processedDaysTotal,
elapsed: progress.formatElapsedTime(overallStartTime),
remaining: 'Failed',
rate: 0,
// Include history ID in progress data if needed
historyId: calculateHistoryId
};
// Call with single object parameter (not separate historyId)
progress.outputProgress(progressData);
} catch (progressError) {
console.error('Failed to report progress:', progressError);
}
return { success: false, status: 'failed', error: error.message, duration: finalDurationSec };
} finally {
if (connection) {
console.log('Releasing database connection.');
connection.release();
}
// Close pool only if this script is meant to be standalone
// If part of a larger app, the app should manage pool closure
// console.log('Closing database pool.');
// await closePool();
}
}
// --- Script Execution ---
// Parse command-line arguments
const args = process.argv.slice(2);
let cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg = 1; // Default start batch is 1
for (let i = 0; i < args.length; i++) {
if (args[i] === '--start-date' && args[i+1]) cmdStartDateArg = args[++i];
else if (args[i] === '--end-date' && args[i+1]) cmdEndDateArg = args[++i];
else if (args[i] === '--start-batch' && args[i+1]) cmdStartBatchArg = parseInt(args[++i], 10);
}
if (isNaN(cmdStartBatchArg) || cmdStartBatchArg < 1) {
console.warn(`Invalid --start-batch value. Defaulting to 1.`);
cmdStartBatchArg = 1;
}
// Run the backfill process
backfillSnapshots(cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg)
.then(result => {
if (result.success) {
console.log(`\n✅ ${result.message} (Duration: ${result.duration}s)`);
process.exitCode = 0; // Success
} else {
console.error(`\n❌ Backfill failed: ${result.error || 'Unknown error'} (Duration: ${result.duration}s)`);
process.exitCode = 1; // Failure
}
})
.catch(err => {
console.error('\n❌ Unexpected error during backfill execution:', err);
process.exitCode = 1; // Failure
})
.finally(async () => {
// Ensure pool is closed if run standalone
console.log('Backfill script finished. Closing pool.');
await closePool(); // Make sure closePool exists and works in your db utils
process.exit(process.exitCode); // Exit with appropriate code
});
inventory-server/old/backfill_historical_snapshots.sql
-161
View File
@@ -1,161 +0,0 @@
-- Description: Backfills the daily_product_snapshots table using imported historical unit data
-- (daily inventory/stats) and historical price data (current prices table).
-- - Uses imported daily sales/receipt UNIT counts for accuracy.
-- - ESTIMATES historical stock levels using a forward calculation.
-- - APPROXIMATES historical REVENUE using looked-up historical base prices.
-- - APPROXIMATES historical COGS, PROFIT, and STOCK VALUE using CURRENT product costs/prices.
-- Run ONCE after importing historical data and before initial product_metrics population.
-- Dependencies: Core import tables (products), imported history tables (imported_daily_inventory,
-- imported_product_stat_history, imported_product_current_prices),
-- daily_product_snapshots table must exist.
-- Frequency: Run ONCE.
CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final(
_start_date DATE,
_end_date DATE
)
RETURNS VOID AS $$
DECLARE
_current_processing_date DATE := _start_date;
_batch_start_time TIMESTAMPTZ;
_row_count INTEGER;
BEGIN
RAISE NOTICE 'Starting FINAL historical snapshot backfill from % to %.', _start_date, _end_date;
RAISE NOTICE 'Using historical units and historical prices (for revenue approximation).';
RAISE NOTICE 'WARNING: Historical COGS, Profit, and Stock Value use CURRENT product costs/prices.';
-- Ensure end date is not in the future
IF _end_date >= CURRENT_DATE THEN
_end_date := CURRENT_DATE - INTERVAL '1 day';
RAISE NOTICE 'Adjusted end date to % to avoid conflict with hourly script.', _end_date;
END IF;
-- Performance: Create temporary table with product info to avoid repeated lookups
CREATE TEMP TABLE IF NOT EXISTS temp_product_info AS
SELECT
pid,
sku,
COALESCE(landing_cost_price, cost_price, 0.00) as effective_cost_price,
COALESCE(price, 0.00) as current_price,
COALESCE(regular_price, 0.00) as current_regular_price
FROM public.products;
-- Performance: Create index on temporary table
CREATE INDEX IF NOT EXISTS temp_product_info_pid_idx ON temp_product_info(pid);
ANALYZE temp_product_info;
RAISE NOTICE 'Created temporary product info table with % products', (SELECT COUNT(*) FROM temp_product_info);
WHILE _current_processing_date <= _end_date LOOP
_batch_start_time := clock_timestamp();
RAISE NOTICE 'Processing date: %', _current_processing_date;
-- Get Daily Transaction Unit Info from imported history
WITH DailyHistoryUnits AS (
SELECT
pids.pid,
-- Prioritize daily_inventory, fallback to product_stat_history for sold qty
COALESCE(di.amountsold, ps.qty_sold, 0)::integer as units_sold_today,
COALESCE(di.qtyreceived, 0)::integer as units_received_today
FROM
(SELECT DISTINCT pid FROM temp_product_info) pids -- Ensure all products are considered
LEFT JOIN public.imported_daily_inventory di
ON pids.pid = di.pid AND di.date = _current_processing_date
LEFT JOIN public.imported_product_stat_history ps
ON pids.pid = ps.pid AND ps.date = _current_processing_date
-- Removed WHERE clause to ensure snapshots are created even for days with 0 activity,
-- allowing stock carry-over. The main query will handle products properly.
),
HistoricalPrice AS (
-- Find the base price (qty_buy=1) active on the processing date
SELECT DISTINCT ON (pid)
pid,
price_each
FROM public.imported_product_current_prices
WHERE
qty_buy = 1
-- Use TIMESTAMPTZ comparison logic:
AND date_active <= (_current_processing_date + interval '1 day' - interval '1 second') -- Active sometime on or before end of processing day
AND (date_deactive IS NULL OR date_deactive > _current_processing_date) -- Not deactivated before start of processing day
-- Assuming 'active' flag isn't needed if dates are correct; add 'AND active != 0' if necessary
ORDER BY
pid, date_active DESC -- Get the most recently activated price
),
PreviousStock AS (
-- Get the estimated stock from the PREVIOUS day snapshot
SELECT pid, eod_stock_quantity
FROM public.daily_product_snapshots
WHERE snapshot_date = _current_processing_date - INTERVAL '1 day'
)
-- Insert into the daily snapshots table
INSERT INTO public.daily_product_snapshots (
snapshot_date, pid, sku,
eod_stock_quantity, eod_stock_cost, eod_stock_retail, eod_stock_gross, stockout_flag,
units_sold, units_returned,
gross_revenue, discounts, returns_revenue,
net_revenue, cogs, gross_regular_revenue, profit,
units_received, cost_received,
calculation_timestamp
)
SELECT
_current_processing_date AS snapshot_date,
p.pid,
p.sku,
-- Estimated EOD Stock (using historical daily units)
-- Handle potential NULL from joins with COALESCE 0
COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0) AS estimated_eod_stock,
-- Valued Stock (using estimated stock and CURRENT prices/costs - APPROXIMATION)
GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.effective_cost_price AS eod_stock_cost,
GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_price AS eod_stock_retail, -- Stock retail uses current price
GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_regular_price AS eod_stock_gross, -- Stock gross uses current regular price
-- Stockout Flag (based on estimated stock)
(COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) <= 0 AS stockout_flag,
-- Today's Unit Aggregates from History
COALESCE(dh.units_sold_today, 0) as units_sold,
0 AS units_returned, -- Placeholder: Cannot determine returns from daily summary
-- Monetary Values using looked-up Historical Price and CURRENT Cost/RegPrice
COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS gross_revenue, -- Approx Revenue
0 AS discounts, -- Placeholder
0 AS returns_revenue, -- Placeholder
COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS net_revenue, -- Approx Net Revenue
COALESCE(dh.units_sold_today, 0) * p.effective_cost_price AS cogs, -- Approx COGS (uses CURRENT cost)
COALESCE(dh.units_sold_today, 0) * p.current_regular_price AS gross_regular_revenue, -- Approx Gross Regular Revenue
-- Approx Profit
(COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price)) - (COALESCE(dh.units_sold_today, 0) * p.effective_cost_price) AS profit,
COALESCE(dh.units_received_today, 0) as units_received,
-- Estimate received cost using CURRENT product cost
COALESCE(dh.units_received_today, 0) * p.effective_cost_price AS cost_received, -- Approx
clock_timestamp() -- Timestamp of this specific calculation
FROM temp_product_info p -- Use the temp table for better performance
LEFT JOIN PreviousStock ps ON p.pid = ps.pid
LEFT JOIN DailyHistoryUnits dh ON p.pid = dh.pid -- Join today's historical activity
LEFT JOIN HistoricalPrice hp ON p.pid = hp.pid -- Join the looked-up historical price
-- Optimization: Only process products with activity or previous stock
WHERE (dh.units_sold_today > 0 OR dh.units_received_today > 0 OR COALESCE(ps.eod_stock_quantity, 0) > 0)
ON CONFLICT (snapshot_date, pid) DO NOTHING; -- Avoid errors if rerunning parts, but prefer clean runs
GET DIAGNOSTICS _row_count = ROW_COUNT;
RAISE NOTICE 'Processed %: Inserted/Skipped % rows. Duration: %',
_current_processing_date,
_row_count,
clock_timestamp() - _batch_start_time;
_current_processing_date := _current_processing_date + INTERVAL '1 day';
END LOOP;
-- Clean up temporary tables
DROP TABLE IF EXISTS temp_product_info;
RAISE NOTICE 'Finished FINAL historical snapshot backfill.';
END;
$$ LANGUAGE plpgsql;
-- Example usage:
-- SELECT backfill_daily_snapshots_range_final('2023-01-01'::date, '2023-12-31'::date);
inventory-server/old/calculate-metrics.js
-558
View File
@@ -1,558 +0,0 @@
const path = require('path');
// Change working directory to script directory
process.chdir(path.dirname(__filename));
require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
// Configuration flags for controlling which metrics to calculate
// Set to 1 to skip the corresponding calculation, 0 to run it
const SKIP_PRODUCT_METRICS = 0;
const SKIP_TIME_AGGREGATES = 0;
const SKIP_FINANCIAL_METRICS = 0;
const SKIP_VENDOR_METRICS = 0;
const SKIP_CATEGORY_METRICS = 0;
const SKIP_BRAND_METRICS = 0;
const SKIP_SALES_FORECASTS = 0;
// Add error handler for uncaught exceptions
process.on('uncaughtException', (error) => {
console.error('Uncaught Exception:', error);
process.exit(1);
});
// Add error handler for unhandled promise rejections
process.on('unhandledRejection', (reason, promise) => {
console.error('Unhandled Rejection at:', promise, 'reason:', reason);
process.exit(1);
});
const progress = require('./metrics/utils/progress');
console.log('Progress module loaded:', {
modulePath: require.resolve('./metrics/utils/progress'),
exports: Object.keys(progress),
currentDir: process.cwd(),
scriptDir: __dirname
});
// Store progress functions in global scope to ensure availability
global.formatElapsedTime = progress.formatElapsedTime;
global.estimateRemaining = progress.estimateRemaining;
global.calculateRate = progress.calculateRate;
global.outputProgress = progress.outputProgress;
global.clearProgress = progress.clearProgress;
global.getProgress = progress.getProgress;
global.logError = progress.logError;
// List of temporary tables used in the calculation process
const TEMP_TABLES = [
'temp_revenue_ranks',
'temp_sales_metrics',
'temp_purchase_metrics',
'temp_product_metrics',
'temp_vendor_metrics',
'temp_category_metrics',
'temp_brand_metrics',
'temp_forecast_dates',
'temp_daily_sales',
'temp_product_stats',
'temp_category_sales',
'temp_category_stats',
'temp_beginning_inventory',
'temp_monthly_inventory'
];
// Add cleanup function for temporary tables
async function cleanupTemporaryTables(connection) {
try {
// Drop each temporary table if it exists
for (const table of TEMP_TABLES) {
await connection.query(`DROP TABLE IF EXISTS ${table}`);
}
} catch (err) {
console.error('Error cleaning up temporary tables:', err);
}
}
const { getConnection, closePool } = require('./metrics/utils/db');
const calculateProductMetrics = require('./metrics/product-metrics');
const calculateTimeAggregates = require('./metrics/time-aggregates');
const calculateFinancialMetrics = require('./metrics/financial-metrics');
const calculateVendorMetrics = require('./metrics/vendor-metrics');
const calculateCategoryMetrics = require('./metrics/category-metrics');
const calculateBrandMetrics = require('./metrics/brand-metrics');
const calculateSalesForecasts = require('./metrics/sales-forecasts');
// Add cancel handler
let isCancelled = false;
function cancelCalculation() {
isCancelled = true;
console.log('Calculation has been cancelled by user');
// Force-terminate any query that's been running for more than 5 seconds
try {
const connection = getConnection();
connection.then(async (conn) => {
try {
// Identify and terminate long-running queries from our application
await conn.query(`
SELECT pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE query_start < now() - interval '5 seconds'
AND application_name LIKE '%node%'
AND query NOT LIKE '%pg_cancel_backend%'
`);
// Clean up any temporary tables
await cleanupTemporaryTables(conn);
// Release connection
conn.release();
} catch (err) {
console.error('Error during force cancellation:', err);
conn.release();
}
}).catch(err => {
console.error('Could not get connection for cancellation:', err);
});
} catch (err) {
console.error('Failed to terminate running queries:', err);
}
return {
success: true,
message: 'Calculation has been cancelled'
};
}
// Handle SIGTERM signal for cancellation
process.on('SIGTERM', cancelCalculation);
// Update the main calculation function to use the new modular structure
async function calculateMetrics() {
let connection;
const startTime = Date.now();
let processedProducts = 0;
let processedOrders = 0;
let processedPurchaseOrders = 0;
let totalProducts = 0;
let totalOrders = 0;
let totalPurchaseOrders = 0;
let calculateHistoryId;
// Set a maximum execution time (30 minutes)
const MAX_EXECUTION_TIME = 30 * 60 * 1000;
const timeout = setTimeout(() => {
console.error(`Calculation timed out after ${MAX_EXECUTION_TIME/1000} seconds, forcing termination`);
// Call cancel and force exit
cancelCalculation();
process.exit(1);
}, MAX_EXECUTION_TIME);
try {
// Clean up any previously running calculations
connection = await getConnection();
await connection.query(`
UPDATE calculate_history
SET
status = 'cancelled',
end_time = NOW(),
duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
error_message = 'Previous calculation was not completed properly'
WHERE status = 'running'
`);
// Get counts from all relevant tables
const [productCountResult, orderCountResult, poCountResult] = await Promise.all([
connection.query('SELECT COUNT(*) as total FROM products'),
connection.query('SELECT COUNT(*) as total FROM orders'),
connection.query('SELECT COUNT(*) as total FROM purchase_orders')
]);
totalProducts = parseInt(productCountResult.rows[0].total);
totalOrders = parseInt(orderCountResult.rows[0].total);
totalPurchaseOrders = parseInt(poCountResult.rows[0].total);
// Create history record for this calculation
const historyResult = await connection.query(`
INSERT INTO calculate_history (
start_time,
status,
total_products,
total_orders,
total_purchase_orders,
additional_info
) VALUES (
NOW(),
'running',
$1,
$2,
$3,
jsonb_build_object(
'skip_product_metrics', ($4::int > 0),
'skip_time_aggregates', ($5::int > 0),
'skip_financial_metrics', ($6::int > 0),
'skip_vendor_metrics', ($7::int > 0),
'skip_category_metrics', ($8::int > 0),
'skip_brand_metrics', ($9::int > 0),
'skip_sales_forecasts', ($10::int > 0)
)
) RETURNING id
`, [
totalProducts,
totalOrders,
totalPurchaseOrders,
SKIP_PRODUCT_METRICS,
SKIP_TIME_AGGREGATES,
SKIP_FINANCIAL_METRICS,
SKIP_VENDOR_METRICS,
SKIP_CATEGORY_METRICS,
SKIP_BRAND_METRICS,
SKIP_SALES_FORECASTS
]);
calculateHistoryId = historyResult.rows[0].id;
// Add debug logging for the progress functions
console.log('Debug - Progress functions:', {
formatElapsedTime: typeof global.formatElapsedTime,
estimateRemaining: typeof global.estimateRemaining,
calculateRate: typeof global.calculateRate,
startTime: startTime
});
try {
const elapsed = global.formatElapsedTime(startTime);
console.log('Debug - formatElapsedTime test successful:', elapsed);
} catch (err) {
console.error('Debug - Error testing formatElapsedTime:', err);
throw err;
}
// Release the connection before getting a new one
connection.release();
isCancelled = false;
connection = await getConnection();
try {
global.outputProgress({
status: 'running',
operation: 'Starting metrics calculation',
current: 0,
total: 100,
elapsed: '0s',
remaining: 'Calculating...',
rate: 0,
percentage: '0',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Update progress periodically
const updateProgress = async (products = null, orders = null, purchaseOrders = null) => {
// Ensure all values are valid numbers or default to previous value
if (products !== null) processedProducts = Number(products) || processedProducts || 0;
if (orders !== null) processedOrders = Number(orders) || processedOrders || 0;
if (purchaseOrders !== null) processedPurchaseOrders = Number(purchaseOrders) || processedPurchaseOrders || 0;
// Ensure we never send NaN to the database
const safeProducts = Number(processedProducts) || 0;
const safeOrders = Number(processedOrders) || 0;
const safePurchaseOrders = Number(processedPurchaseOrders) || 0;
await connection.query(`
UPDATE calculate_history
SET
processed_products = $1,
processed_orders = $2,
processed_purchase_orders = $3
WHERE id = $4
`, [safeProducts, safeOrders, safePurchaseOrders, calculateHistoryId]);
};
// Helper function to ensure valid progress numbers
const ensureValidProgress = (current, total) => ({
current: Number(current) || 0,
total: Number(total) || 1, // Default to 1 to avoid division by zero
percentage: (((Number(current) || 0) / (Number(total) || 1)) * 100).toFixed(1)
});
// Initial progress
const initialProgress = ensureValidProgress(0, totalProducts);
global.outputProgress({
status: 'running',
operation: 'Starting metrics calculation',
current: initialProgress.current,
total: initialProgress.total,
elapsed: '0s',
remaining: 'Calculating...',
rate: 0,
percentage: initialProgress.percentage,
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (!SKIP_PRODUCT_METRICS) {
const result = await calculateProductMetrics(startTime, totalProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Product metrics calculation failed');
}
} else {
console.log('Skipping product metrics calculation...');
processedProducts = Math.floor(totalProducts * 0.6);
await updateProgress(processedProducts);
global.outputProgress({
status: 'running',
operation: 'Skipping product metrics calculation',
current: processedProducts,
total: totalProducts,
elapsed: global.formatElapsedTime(startTime),
remaining: global.estimateRemaining(startTime, processedProducts, totalProducts),
rate: global.calculateRate(startTime, processedProducts),
percentage: '60',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
}
// Calculate time-based aggregates
if (!SKIP_TIME_AGGREGATES) {
const result = await calculateTimeAggregates(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Time aggregates calculation failed');
}
} else {
console.log('Skipping time aggregates calculation');
}
// Calculate financial metrics
if (!SKIP_FINANCIAL_METRICS) {
const result = await calculateFinancialMetrics(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Financial metrics calculation failed');
}
} else {
console.log('Skipping financial metrics calculation');
}
// Calculate vendor metrics
if (!SKIP_VENDOR_METRICS) {
const result = await calculateVendorMetrics(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Vendor metrics calculation failed');
}
} else {
console.log('Skipping vendor metrics calculation');
}
// Calculate category metrics
if (!SKIP_CATEGORY_METRICS) {
const result = await calculateCategoryMetrics(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Category metrics calculation failed');
}
} else {
console.log('Skipping category metrics calculation');
}
// Calculate brand metrics
if (!SKIP_BRAND_METRICS) {
const result = await calculateBrandMetrics(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Brand metrics calculation failed');
}
} else {
console.log('Skipping brand metrics calculation');
}
// Calculate sales forecasts
if (!SKIP_SALES_FORECASTS) {
const result = await calculateSalesForecasts(startTime, totalProducts, processedProducts);
await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
if (!result.success) {
throw new Error('Sales forecasts calculation failed');
}
} else {
console.log('Skipping sales forecasts calculation');
}
// Final progress update with guaranteed valid numbers
const finalProgress = ensureValidProgress(totalProducts, totalProducts);
// Final success message
outputProgress({
status: 'complete',
operation: 'Metrics calculation complete',
current: finalProgress.current,
total: finalProgress.total,
elapsed: global.formatElapsedTime(startTime),
remaining: '0s',
rate: global.calculateRate(startTime, finalProgress.current),
percentage: '100',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Ensure all values are valid numbers before final update
const finalStats = {
processedProducts: Number(processedProducts) || 0,
processedOrders: Number(processedOrders) || 0,
processedPurchaseOrders: Number(processedPurchaseOrders) || 0
};
// Update history with completion
await connection.query(`
UPDATE calculate_history
SET
end_time = NOW(),
duration_seconds = $1,
processed_products = $2,
processed_orders = $3,
processed_purchase_orders = $4,
status = 'completed'
WHERE id = $5
`, [Math.round((Date.now() - startTime) / 1000),
finalStats.processedProducts,
finalStats.processedOrders,
finalStats.processedPurchaseOrders,
calculateHistoryId]);
// Clear progress file on successful completion
global.clearProgress();
return {
success: true,
message: 'Calculation completed successfully',
duration: Math.round((Date.now() - startTime) / 1000)
};
} catch (error) {
const endTime = Date.now();
const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
// Update history with error
await connection.query(`
UPDATE calculate_history
SET
end_time = NOW(),
duration_seconds = $1,
processed_products = $2,
processed_orders = $3,
processed_purchase_orders = $4,
status = $5,
error_message = $6
WHERE id = $7
`, [
totalElapsedSeconds,
processedProducts || 0, // Ensure we have a valid number
processedOrders || 0, // Ensure we have a valid number
processedPurchaseOrders || 0, // Ensure we have a valid number
isCancelled ? 'cancelled' : 'failed',
error.message,
calculateHistoryId
]);
if (isCancelled) {
global.outputProgress({
status: 'cancelled',
operation: 'Calculation cancelled',
current: processedProducts,
total: totalProducts || 0,
elapsed: global.formatElapsedTime(startTime),
remaining: null,
rate: global.calculateRate(startTime, processedProducts),
percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
} else {
global.outputProgress({
status: 'error',
operation: 'Error: ' + error.message,
current: processedProducts,
total: totalProducts || 0,
elapsed: global.formatElapsedTime(startTime),
remaining: null,
rate: global.calculateRate(startTime, processedProducts),
percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
}
throw error;
} finally {
// Clear the timeout to prevent forced termination
clearTimeout(timeout);
// Always clean up and release connection
if (connection) {
try {
await cleanupTemporaryTables(connection);
connection.release();
} catch (err) {
console.error('Error in final cleanup:', err);
}
}
}
} catch (error) {
console.error('Error in metrics calculation', error);
try {
if (connection) {
await connection.query(`
UPDATE calculate_history
SET
status = 'failed',
end_time = NOW(),
duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
error_message = $1
WHERE id = $2
`, [error.message.substring(0, 500), calculateHistoryId]);
}
} catch (updateError) {
console.error('Error updating calculation history:', updateError);
}
throw error;
}
}
// Export as a module with all necessary functions
module.exports = {
calculateMetrics,
cancelCalculation,
getProgress: global.getProgress
};
// Run directly if called from command line
if (require.main === module) {
calculateMetrics().catch(error => {
if (!error.message.includes('Operation cancelled')) {
console.error('Error:', error);
}
process.exit(1);
});
}
inventory-server/old/config-schema.sql
-242
View File
@@ -1,242 +0,0 @@
-- -- Configuration tables schema
-- -- Stock threshold configurations
-- CREATE TABLE stock_thresholds (
-- id INTEGER NOT NULL,
-- category_id BIGINT, -- NULL means default/global threshold
-- vendor VARCHAR(100), -- NULL means applies to all vendors
-- critical_days INTEGER NOT NULL DEFAULT 7,
-- reorder_days INTEGER NOT NULL DEFAULT 14,
-- overstock_days INTEGER NOT NULL DEFAULT 90,
-- low_stock_threshold INTEGER NOT NULL DEFAULT 5,
-- min_reorder_quantity INTEGER NOT NULL DEFAULT 1,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (id),
-- FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-- UNIQUE (category_id, vendor)
-- );
-- CREATE TRIGGER update_stock_thresholds_updated
-- BEFORE UPDATE ON stock_thresholds
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- CREATE INDEX idx_st_metrics ON stock_thresholds(category_id, vendor);
-- -- Lead time threshold configurations
-- CREATE TABLE lead_time_thresholds (
-- id INTEGER NOT NULL,
-- category_id BIGINT, -- NULL means default/global threshold
-- vendor VARCHAR(100), -- NULL means applies to all vendors
-- target_days INTEGER NOT NULL DEFAULT 14,
-- warning_days INTEGER NOT NULL DEFAULT 21,
-- critical_days INTEGER NOT NULL DEFAULT 30,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (id),
-- FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-- UNIQUE (category_id, vendor)
-- );
-- CREATE TRIGGER update_lead_time_thresholds_updated
-- BEFORE UPDATE ON lead_time_thresholds
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- -- Sales velocity window configurations
-- CREATE TABLE sales_velocity_config (
-- id INTEGER NOT NULL,
-- category_id BIGINT, -- NULL means default/global threshold
-- vendor VARCHAR(100), -- NULL means applies to all vendors
-- daily_window_days INTEGER NOT NULL DEFAULT 30,
-- weekly_window_days INTEGER NOT NULL DEFAULT 7,
-- monthly_window_days INTEGER NOT NULL DEFAULT 90,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (id),
-- FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-- UNIQUE (category_id, vendor)
-- );
-- CREATE TRIGGER update_sales_velocity_config_updated
-- BEFORE UPDATE ON sales_velocity_config
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- CREATE INDEX idx_sv_metrics ON sales_velocity_config(category_id, vendor);
-- -- ABC Classification configurations
-- CREATE TABLE abc_classification_config (
-- id INTEGER NOT NULL PRIMARY KEY,
-- a_threshold DECIMAL(5,2) NOT NULL DEFAULT 20.0,
-- b_threshold DECIMAL(5,2) NOT NULL DEFAULT 50.0,
-- classification_period_days INTEGER NOT NULL DEFAULT 90,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
-- );
-- CREATE TRIGGER update_abc_classification_config_updated
-- BEFORE UPDATE ON abc_classification_config
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- -- Safety stock configurations
-- CREATE TABLE safety_stock_config (
-- id INTEGER NOT NULL,
-- category_id BIGINT, -- NULL means default/global threshold
-- vendor VARCHAR(100), -- NULL means applies to all vendors
-- coverage_days INTEGER NOT NULL DEFAULT 14,
-- service_level DECIMAL(5,2) NOT NULL DEFAULT 95.0,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (id),
-- FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-- UNIQUE (category_id, vendor)
-- );
-- CREATE TRIGGER update_safety_stock_config_updated
-- BEFORE UPDATE ON safety_stock_config
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- CREATE INDEX idx_ss_metrics ON safety_stock_config(category_id, vendor);
-- -- Turnover rate configurations
-- CREATE TABLE turnover_config (
-- id INTEGER NOT NULL,
-- category_id BIGINT, -- NULL means default/global threshold
-- vendor VARCHAR(100), -- NULL means applies to all vendors
-- calculation_period_days INTEGER NOT NULL DEFAULT 30,
-- target_rate DECIMAL(10,2) NOT NULL DEFAULT 1.0,
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (id),
-- FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-- UNIQUE (category_id, vendor)
-- );
-- CREATE TRIGGER update_turnover_config_updated
-- BEFORE UPDATE ON turnover_config
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- -- Create table for sales seasonality factors
-- CREATE TABLE sales_seasonality (
-- month INTEGER NOT NULL,
-- seasonality_factor DECIMAL(5,3) DEFAULT 0,
-- last_updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- PRIMARY KEY (month),
-- CONSTRAINT month_range CHECK (month BETWEEN 1 AND 12),
-- CONSTRAINT seasonality_range CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
-- );
-- CREATE TRIGGER update_sales_seasonality_updated
-- BEFORE UPDATE ON sales_seasonality
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- -- Create table for financial calculation parameters
-- CREATE TABLE financial_calc_config (
-- id INTEGER NOT NULL PRIMARY KEY,
-- order_cost DECIMAL(10,2) NOT NULL DEFAULT 25.00, -- The fixed cost per purchase order (used in EOQ)
-- holding_rate DECIMAL(10,4) NOT NULL DEFAULT 0.25, -- The annual inventory holding cost as a percentage of unit cost (used in EOQ)
-- service_level_z_score DECIMAL(10,4) NOT NULL DEFAULT 1.96, -- Z-score for ~95% service level (used in Safety Stock)
-- min_reorder_qty INTEGER NOT NULL DEFAULT 1, -- Minimum reorder quantity
-- default_reorder_qty INTEGER NOT NULL DEFAULT 5, -- Default reorder quantity when sales data is insufficient
-- default_safety_stock INTEGER NOT NULL DEFAULT 5, -- Default safety stock when sales data is insufficient
-- created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-- updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
-- );
-- CREATE TRIGGER update_financial_calc_config_updated
-- BEFORE UPDATE ON financial_calc_config
-- FOR EACH ROW
-- EXECUTE FUNCTION update_updated_at_column();
-- -- Insert default global thresholds
-- INSERT INTO stock_thresholds (id, category_id, vendor, critical_days, reorder_days, overstock_days)
-- VALUES (1, NULL, NULL, 7, 14, 90)
-- ON CONFLICT (id) DO UPDATE SET
-- critical_days = EXCLUDED.critical_days,
-- reorder_days = EXCLUDED.reorder_days,
-- overstock_days = EXCLUDED.overstock_days;
-- INSERT INTO lead_time_thresholds (id, category_id, vendor, target_days, warning_days, critical_days)
-- VALUES (1, NULL, NULL, 14, 21, 30)
-- ON CONFLICT (id) DO UPDATE SET
-- target_days = EXCLUDED.target_days,
-- warning_days = EXCLUDED.warning_days,
-- critical_days = EXCLUDED.critical_days;
-- INSERT INTO sales_velocity_config (id, category_id, vendor, daily_window_days, weekly_window_days, monthly_window_days)
-- VALUES (1, NULL, NULL, 30, 7, 90)
-- ON CONFLICT (id) DO UPDATE SET
-- daily_window_days = EXCLUDED.daily_window_days,
-- weekly_window_days = EXCLUDED.weekly_window_days,
-- monthly_window_days = EXCLUDED.monthly_window_days;
-- INSERT INTO abc_classification_config (id, a_threshold, b_threshold, classification_period_days)
-- VALUES (1, 20.0, 50.0, 90)
-- ON CONFLICT (id) DO UPDATE SET
-- a_threshold = EXCLUDED.a_threshold,
-- b_threshold = EXCLUDED.b_threshold,
-- classification_period_days = EXCLUDED.classification_period_days;
-- INSERT INTO safety_stock_config (id, category_id, vendor, coverage_days, service_level)
-- VALUES (1, NULL, NULL, 14, 95.0)
-- ON CONFLICT (id) DO UPDATE SET
-- coverage_days = EXCLUDED.coverage_days,
-- service_level = EXCLUDED.service_level;
-- INSERT INTO turnover_config (id, category_id, vendor, calculation_period_days, target_rate)
-- VALUES (1, NULL, NULL, 30, 1.0)
-- ON CONFLICT (id) DO UPDATE SET
-- calculation_period_days = EXCLUDED.calculation_period_days,
-- target_rate = EXCLUDED.target_rate;
-- -- Insert default seasonality factors (neutral)
-- INSERT INTO sales_seasonality (month, seasonality_factor)
-- VALUES
-- (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
-- (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
-- ON CONFLICT (month) DO UPDATE SET
-- last_updated = CURRENT_TIMESTAMP;
-- -- Insert default values
-- INSERT INTO financial_calc_config (id, order_cost, holding_rate, service_level_z_score, min_reorder_qty, default_reorder_qty, default_safety_stock)
-- VALUES (1, 25.00, 0.25, 1.96, 1, 5, 5)
-- ON CONFLICT (id) DO UPDATE SET
-- order_cost = EXCLUDED.order_cost,
-- holding_rate = EXCLUDED.holding_rate,
-- service_level_z_score = EXCLUDED.service_level_z_score,
-- min_reorder_qty = EXCLUDED.min_reorder_qty,
-- default_reorder_qty = EXCLUDED.default_reorder_qty,
-- default_safety_stock = EXCLUDED.default_safety_stock;
-- -- View to show thresholds with category names
-- CREATE OR REPLACE VIEW stock_thresholds_view AS
-- SELECT
-- st.*,
-- c.name as category_name,
-- CASE
-- WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 'Global Default'
-- WHEN st.category_id IS NULL THEN 'Vendor: ' || st.vendor
-- WHEN st.vendor IS NULL THEN 'Category: ' || c.name
-- ELSE 'Category: ' || c.name || ' / Vendor: ' || st.vendor
-- END as threshold_scope
-- FROM
-- stock_thresholds st
-- LEFT JOIN
-- categories c ON st.category_id = c.cat_id
-- ORDER BY
-- CASE
-- WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 1
-- WHEN st.category_id IS NULL THEN 2
-- WHEN st.vendor IS NULL THEN 3
-- ELSE 4
-- END,
-- c.name,
-- st.vendor;
inventory-server/old/historical-data.js
-961
View File
@@ -1,961 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../scripts/metrics-new/utils/progress');
const fs = require('fs');
const path = require('path');
const { pipeline } = require('stream');
const { promisify } = require('util');
// Configuration constants to control which tables get imported
const IMPORT_PRODUCT_CURRENT_PRICES = false;
const IMPORT_DAILY_INVENTORY = false;
const IMPORT_PRODUCT_STAT_HISTORY = true;
// For product stat history, limit to more recent data for faster initial import
const USE_RECENT_MONTHS = 12; // Just use the most recent months for product_stat_history
/**
* Validates a date from MySQL before inserting it into PostgreSQL
* @param {string|Date|null} mysqlDate - Date string or object from MySQL
* @returns {string|null} Valid date string or null if invalid
*/
function validateDate(mysqlDate) {
// Handle null, undefined, or empty values
if (!mysqlDate) {
return null;
}
// Convert to string if it's not already
const dateStr = String(mysqlDate);
// Handle MySQL zero dates and empty values
if (dateStr === '0000-00-00' ||
dateStr === '0000-00-00 00:00:00' ||
dateStr.indexOf('0000-00-00') !== -1 ||
dateStr === '') {
return null;
}
// Check if the date is valid
const date = new Date(mysqlDate);
// If the date is invalid or suspiciously old (pre-1970), return null
if (isNaN(date.getTime()) || date.getFullYear() < 1970) {
return null;
}
return mysqlDate;
}
/**
* Imports historical data from MySQL to PostgreSQL
*/
async function importHistoricalData(
prodConnection,
localConnection,
options = {}
) {
const {
incrementalUpdate = true,
oneYearAgo = new Date(new Date().setFullYear(new Date().getFullYear() - 1))
} = options;
const oneYearAgoStr = oneYearAgo.toISOString().split('T')[0];
const startTime = Date.now();
// Use larger batch sizes to improve performance
const BATCH_SIZE = 5000; // For fetching from small tables
const INSERT_BATCH_SIZE = 500; // For inserting to small tables
const LARGE_BATCH_SIZE = 10000; // For fetching from large tables
const LARGE_INSERT_BATCH_SIZE = 1000; // For inserting to large tables
// Calculate date for recent data
const recentDateStr = new Date(
new Date().setMonth(new Date().getMonth() - USE_RECENT_MONTHS)
).toISOString().split('T')[0];
console.log(`Starting import with:
- One year ago date: ${oneYearAgoStr}
- Recent months date: ${recentDateStr} (for product_stat_history)
- Incremental update: ${incrementalUpdate}
- Standard batch size: ${BATCH_SIZE}
- Standard insert batch size: ${INSERT_BATCH_SIZE}
- Large table batch size: ${LARGE_BATCH_SIZE}
- Large table insert batch size: ${LARGE_INSERT_BATCH_SIZE}
- Import product_current_prices: ${IMPORT_PRODUCT_CURRENT_PRICES}
- Import daily_inventory: ${IMPORT_DAILY_INVENTORY}
- Import product_stat_history: ${IMPORT_PRODUCT_STAT_HISTORY}`);
try {
// Get last sync time for incremental updates
const lastSyncTimes = {};
if (incrementalUpdate) {
try {
const syncResult = await localConnection.query(`
SELECT table_name, last_sync_timestamp
FROM sync_status
WHERE table_name IN (
'imported_product_current_prices',
'imported_daily_inventory',
'imported_product_stat_history'
)
`);
// Add check for rows existence and type
if (syncResult && Array.isArray(syncResult.rows)) {
for (const row of syncResult.rows) {
lastSyncTimes[row.table_name] = row.last_sync_timestamp;
console.log(`Last sync time for ${row.table_name}: ${row.last_sync_timestamp}`);
}
} else {
console.warn('Sync status query did not return expected rows. Proceeding without last sync times.');
}
} catch (error) {
console.error('Error fetching sync status:', error);
}
}
// Determine how many tables will be imported
const tablesCount = [
IMPORT_PRODUCT_CURRENT_PRICES,
IMPORT_DAILY_INVENTORY,
IMPORT_PRODUCT_STAT_HISTORY
].filter(Boolean).length;
// Run all imports sequentially for better reliability
console.log(`Starting sequential imports for ${tablesCount} tables...`);
outputProgress({
status: "running",
operation: "Historical data import",
message: `Starting sequential imports for ${tablesCount} tables...`,
current: 0,
total: tablesCount,
elapsed: formatElapsedTime(startTime)
});
let progressCount = 0;
let productCurrentPricesResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
let dailyInventoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
let productStatHistoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
// Import product current prices
if (IMPORT_PRODUCT_CURRENT_PRICES) {
console.log('Importing product current prices...');
productCurrentPricesResult = await importProductCurrentPrices(
prodConnection,
localConnection,
oneYearAgoStr,
lastSyncTimes['imported_product_current_prices'],
BATCH_SIZE,
INSERT_BATCH_SIZE,
incrementalUpdate,
startTime
);
progressCount++;
outputProgress({
status: "running",
operation: "Historical data import",
message: `Completed import ${progressCount} of ${tablesCount}`,
current: progressCount,
total: tablesCount,
elapsed: formatElapsedTime(startTime)
});
}
// Import daily inventory
if (IMPORT_DAILY_INVENTORY) {
console.log('Importing daily inventory...');
dailyInventoryResult = await importDailyInventory(
prodConnection,
localConnection,
oneYearAgoStr,
lastSyncTimes['imported_daily_inventory'],
BATCH_SIZE,
INSERT_BATCH_SIZE,
incrementalUpdate,
startTime
);
progressCount++;
outputProgress({
status: "running",
operation: "Historical data import",
message: `Completed import ${progressCount} of ${tablesCount}`,
current: progressCount,
total: tablesCount,
elapsed: formatElapsedTime(startTime)
});
}
// Import product stat history - using optimized approach
if (IMPORT_PRODUCT_STAT_HISTORY) {
console.log('Importing product stat history...');
productStatHistoryResult = await importProductStatHistory(
prodConnection,
localConnection,
recentDateStr, // Use more recent date for this massive table
lastSyncTimes['imported_product_stat_history'],
LARGE_BATCH_SIZE,
LARGE_INSERT_BATCH_SIZE,
incrementalUpdate,
startTime,
USE_RECENT_MONTHS // Pass the recent months constant
);
progressCount++;
outputProgress({
status: "running",
operation: "Historical data import",
message: `Completed import ${progressCount} of ${tablesCount}`,
current: progressCount,
total: tablesCount,
elapsed: formatElapsedTime(startTime)
});
}
// Aggregate results
const totalRecordsAdded =
productCurrentPricesResult.recordsAdded +
dailyInventoryResult.recordsAdded +
productStatHistoryResult.recordsAdded;
const totalRecordsUpdated =
productCurrentPricesResult.recordsUpdated +
dailyInventoryResult.recordsUpdated +
productStatHistoryResult.recordsUpdated;
const totalProcessed =
productCurrentPricesResult.totalProcessed +
dailyInventoryResult.totalProcessed +
productStatHistoryResult.totalProcessed;
const allErrors = [
...productCurrentPricesResult.errors,
...dailyInventoryResult.errors,
...productStatHistoryResult.errors
];
// Log import summary
console.log(`
Historical data import complete:
-------------------------------
Records added: ${totalRecordsAdded}
Records updated: ${totalRecordsUpdated}
Total processed: ${totalProcessed}
Errors: ${allErrors.length}
Time taken: ${formatElapsedTime(startTime)}
`);
// Final progress update
outputProgress({
status: "complete",
operation: "Historical data import",
message: `Import complete. Added: ${totalRecordsAdded}, Updated: ${totalRecordsUpdated}, Errors: ${allErrors.length}`,
current: tablesCount,
total: tablesCount,
elapsed: formatElapsedTime(startTime)
});
// Log any errors
if (allErrors.length > 0) {
console.log('Errors encountered during import:');
console.log(JSON.stringify(allErrors, null, 2));
}
// Calculate duration
const endTime = Date.now();
const durationSeconds = Math.round((endTime - startTime) / 1000);
const finalStatus = allErrors.length === 0 ? 'complete' : 'failed';
const errorMessage = allErrors.length > 0 ? JSON.stringify(allErrors) : null;
// Update import history
await localConnection.query(`
INSERT INTO import_history (
table_name,
end_time,
duration_seconds,
records_added,
records_updated,
is_incremental,
status,
error_message,
additional_info
)
VALUES ($1, NOW(), $2, $3, $4, $5, $6, $7, $8)
`, [
'historical_data_combined',
durationSeconds,
totalRecordsAdded,
totalRecordsUpdated,
incrementalUpdate,
finalStatus,
errorMessage,
JSON.stringify({
totalProcessed,
tablesImported: {
imported_product_current_prices: IMPORT_PRODUCT_CURRENT_PRICES,
imported_daily_inventory: IMPORT_DAILY_INVENTORY,
imported_product_stat_history: IMPORT_PRODUCT_STAT_HISTORY
}
})
]);
// Return summary
return {
recordsAdded: totalRecordsAdded,
recordsUpdated: totalRecordsUpdated,
totalProcessed,
errors: allErrors,
timeTaken: formatElapsedTime(startTime)
};
} catch (error) {
console.error('Error importing historical data:', error);
// Final progress update on error
outputProgress({
status: "failed",
operation: "Historical data import",
message: `Import failed: ${error.message}`,
elapsed: formatElapsedTime(startTime)
});
throw error;
}
}
/**
* Imports product_current_prices data from MySQL to PostgreSQL
*/
async function importProductCurrentPrices(
prodConnection,
localConnection,
oneYearAgoStr,
lastSyncTime,
batchSize,
insertBatchSize,
incrementalUpdate,
startTime
) {
let recordsAdded = 0;
let recordsUpdated = 0;
let totalProcessed = 0;
let errors = [];
let offset = 0;
let allProcessed = false;
try {
// Get total count for progress reporting
const [countResult] = await prodConnection.query(`
SELECT COUNT(*) as total
FROM product_current_prices
WHERE (date_active >= ? OR date_deactive >= ?)
${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
`, [oneYearAgoStr, oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
const totalCount = countResult[0].total;
outputProgress({
status: "running",
operation: "Historical data import - Product Current Prices",
message: `Found ${totalCount} records to process`,
current: 0,
total: totalCount,
elapsed: formatElapsedTime(startTime)
});
// Process in batches for better performance
while (!allProcessed) {
try {
// Fetch batch from production
const [rows] = await prodConnection.query(`
SELECT
price_id,
pid,
qty_buy,
is_min_qty_buy,
price_each,
qty_limit,
no_promo,
checkout_offer,
active,
date_active,
date_deactive
FROM product_current_prices
WHERE (date_active >= ? OR date_deactive >= ?)
${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
ORDER BY price_id
LIMIT ? OFFSET ?
`, [
oneYearAgoStr,
oneYearAgoStr,
...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
batchSize,
offset
]);
if (rows.length === 0) {
allProcessed = true;
break;
}
// Process rows in smaller batches for better performance
for (let i = 0; i < rows.length; i += insertBatchSize) {
const batch = rows.slice(i, i + insertBatchSize);
if (batch.length === 0) continue;
try {
// Build parameterized query to handle NULL values properly
const values = [];
const placeholders = [];
let placeholderIndex = 1;
for (const row of batch) {
const rowPlaceholders = [
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`
];
placeholders.push(`(${rowPlaceholders.join(', ')})`);
values.push(
row.price_id,
row.pid,
row.qty_buy,
row.is_min_qty_buy ? true : false,
row.price_each,
row.qty_limit, // PostgreSQL will handle null values properly
row.no_promo ? true : false,
row.checkout_offer ? true : false,
row.active ? true : false,
validateDate(row.date_active),
validateDate(row.date_deactive)
);
}
// Execute batch insert
const result = await localConnection.query(`
WITH ins AS (
INSERT INTO imported_product_current_prices (
price_id, pid, qty_buy, is_min_qty_buy, price_each, qty_limit,
no_promo, checkout_offer, active, date_active, date_deactive
)
VALUES ${placeholders.join(',\n')}
ON CONFLICT (price_id) DO UPDATE SET
pid = EXCLUDED.pid,
qty_buy = EXCLUDED.qty_buy,
is_min_qty_buy = EXCLUDED.is_min_qty_buy,
price_each = EXCLUDED.price_each,
qty_limit = EXCLUDED.qty_limit,
no_promo = EXCLUDED.no_promo,
checkout_offer = EXCLUDED.checkout_offer,
active = EXCLUDED.active,
date_active = EXCLUDED.date_active,
date_deactive = EXCLUDED.date_deactive,
updated = CURRENT_TIMESTAMP
RETURNING (xmax = 0) AS inserted
)
SELECT
COUNT(*) FILTER (WHERE inserted) AS inserted_count,
COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
FROM ins
`, values);
// Safely update counts based on the result
if (result && result.rows && result.rows.length > 0) {
const insertedCount = parseInt(result.rows[0].inserted_count || 0);
const updatedCount = parseInt(result.rows[0].updated_count || 0);
recordsAdded += insertedCount;
recordsUpdated += updatedCount;
}
} catch (error) {
console.error(`Error in batch import of product_current_prices at offset ${i}:`, error);
errors.push({
table: 'imported_product_current_prices',
batchOffset: i,
batchSize: batch.length,
error: error.message
});
}
}
totalProcessed += rows.length;
offset += rows.length;
// Update progress
outputProgress({
status: "running",
operation: "Historical data import - Product Current Prices",
message: `Processed ${totalProcessed} of ${totalCount} records`,
current: totalProcessed,
total: totalCount,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, totalProcessed, totalCount),
rate: calculateRate(startTime, totalProcessed)
});
} catch (error) {
console.error('Error in batch import of product_current_prices:', error);
errors.push({
table: 'imported_product_current_prices',
error: error.message,
offset: offset,
batchSize: batchSize
});
// Try to continue with next batch
offset += batchSize;
}
}
// Update sync status
await localConnection.query(`
INSERT INTO sync_status (table_name, last_sync_timestamp)
VALUES ('imported_product_current_prices', NOW())
ON CONFLICT (table_name) DO UPDATE SET
last_sync_timestamp = NOW()
`);
return { recordsAdded, recordsUpdated, totalProcessed, errors };
} catch (error) {
console.error('Error in product current prices import:', error);
return {
recordsAdded,
recordsUpdated,
totalProcessed,
errors: [...errors, {
table: 'imported_product_current_prices',
error: error.message
}]
};
}
}
/**
* Imports daily_inventory data from MySQL to PostgreSQL
*/
async function importDailyInventory(
prodConnection,
localConnection,
oneYearAgoStr,
lastSyncTime,
batchSize,
insertBatchSize,
incrementalUpdate,
startTime
) {
let recordsAdded = 0;
let recordsUpdated = 0;
let totalProcessed = 0;
let errors = [];
let offset = 0;
let allProcessed = false;
try {
// Get total count for progress reporting
const [countResult] = await prodConnection.query(`
SELECT COUNT(*) as total
FROM daily_inventory
WHERE date >= ?
${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
`, [oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
const totalCount = countResult[0].total;
outputProgress({
status: "running",
operation: "Historical data import - Daily Inventory",
message: `Found ${totalCount} records to process`,
current: 0,
total: totalCount,
elapsed: formatElapsedTime(startTime)
});
// Process in batches for better performance
while (!allProcessed) {
try {
// Fetch batch from production
const [rows] = await prodConnection.query(`
SELECT
date,
pid,
amountsold,
times_sold,
qtyreceived,
price,
costeach,
stamp
FROM daily_inventory
WHERE date >= ?
${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
ORDER BY date, pid
LIMIT ? OFFSET ?
`, [
oneYearAgoStr,
...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
batchSize,
offset
]);
if (rows.length === 0) {
allProcessed = true;
break;
}
// Process rows in smaller batches for better performance
for (let i = 0; i < rows.length; i += insertBatchSize) {
const batch = rows.slice(i, i + insertBatchSize);
if (batch.length === 0) continue;
try {
// Build parameterized query to handle NULL values properly
const values = [];
const placeholders = [];
let placeholderIndex = 1;
for (const row of batch) {
const rowPlaceholders = [
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`
];
placeholders.push(`(${rowPlaceholders.join(', ')})`);
values.push(
validateDate(row.date),
row.pid,
row.amountsold || 0,
row.times_sold || 0,
row.qtyreceived || 0,
row.price || 0,
row.costeach || 0,
validateDate(row.stamp)
);
}
// Execute batch insert
const result = await localConnection.query(`
WITH ins AS (
INSERT INTO imported_daily_inventory (
date, pid, amountsold, times_sold, qtyreceived, price, costeach, stamp
)
VALUES ${placeholders.join(',\n')}
ON CONFLICT (date, pid) DO UPDATE SET
amountsold = EXCLUDED.amountsold,
times_sold = EXCLUDED.times_sold,
qtyreceived = EXCLUDED.qtyreceived,
price = EXCLUDED.price,
costeach = EXCLUDED.costeach,
stamp = EXCLUDED.stamp,
updated = CURRENT_TIMESTAMP
RETURNING (xmax = 0) AS inserted
)
SELECT
COUNT(*) FILTER (WHERE inserted) AS inserted_count,
COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
FROM ins
`, values);
// Safely update counts based on the result
if (result && result.rows && result.rows.length > 0) {
const insertedCount = parseInt(result.rows[0].inserted_count || 0);
const updatedCount = parseInt(result.rows[0].updated_count || 0);
recordsAdded += insertedCount;
recordsUpdated += updatedCount;
}
} catch (error) {
console.error(`Error in batch import of daily_inventory at offset ${i}:`, error);
errors.push({
table: 'imported_daily_inventory',
batchOffset: i,
batchSize: batch.length,
error: error.message
});
}
}
totalProcessed += rows.length;
offset += rows.length;
// Update progress
outputProgress({
status: "running",
operation: "Historical data import - Daily Inventory",
message: `Processed ${totalProcessed} of ${totalCount} records`,
current: totalProcessed,
total: totalCount,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, totalProcessed, totalCount),
rate: calculateRate(startTime, totalProcessed)
});
} catch (error) {
console.error('Error in batch import of daily_inventory:', error);
errors.push({
table: 'imported_daily_inventory',
error: error.message,
offset: offset,
batchSize: batchSize
});
// Try to continue with next batch
offset += batchSize;
}
}
// Update sync status
await localConnection.query(`
INSERT INTO sync_status (table_name, last_sync_timestamp)
VALUES ('imported_daily_inventory', NOW())
ON CONFLICT (table_name) DO UPDATE SET
last_sync_timestamp = NOW()
`);
return { recordsAdded, recordsUpdated, totalProcessed, errors };
} catch (error) {
console.error('Error in daily inventory import:', error);
return {
recordsAdded,
recordsUpdated,
totalProcessed,
errors: [...errors, {
table: 'imported_daily_inventory',
error: error.message
}]
};
}
}
/**
* Imports product_stat_history data from MySQL to PostgreSQL
* Using fast direct inserts without conflict checking
*/
async function importProductStatHistory(
prodConnection,
localConnection,
recentDateStr, // Use more recent date instead of one year ago
lastSyncTime,
batchSize,
insertBatchSize,
incrementalUpdate,
startTime,
recentMonths // Add parameter for recent months
) {
let recordsAdded = 0;
let recordsUpdated = 0;
let totalProcessed = 0;
let errors = [];
let offset = 0;
let allProcessed = false;
let lastRateCheck = Date.now();
let lastProcessed = 0;
try {
// Get total count for progress reporting
const [countResult] = await prodConnection.query(`
SELECT COUNT(*) as total
FROM product_stat_history
WHERE date >= ?
${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
`, [recentDateStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
const totalCount = countResult[0].total;
console.log(`Found ${totalCount} records to process in product_stat_history (using recent date: ${recentDateStr})`);
// Progress indicator
outputProgress({
status: "running",
operation: "Historical data import - Product Stat History",
message: `Found ${totalCount} records to process (last ${recentMonths} months only)`,
current: 0,
total: totalCount,
elapsed: formatElapsedTime(startTime)
});
// If not incremental, truncate the table first for better performance
if (!incrementalUpdate) {
console.log('Truncating imported_product_stat_history for full import...');
await localConnection.query('TRUNCATE TABLE imported_product_stat_history');
} else if (lastSyncTime) {
// For incremental updates, delete records that will be reimported
console.log(`Deleting records from imported_product_stat_history since ${lastSyncTime}...`);
await localConnection.query('DELETE FROM imported_product_stat_history WHERE date > $1', [lastSyncTime]);
}
// Process in batches for better performance
while (!allProcessed) {
try {
// Fetch batch from production with minimal filtering and no sorting
const [rows] = await prodConnection.query(`
SELECT
pid,
date,
COALESCE(score, 0) as score,
COALESCE(score2, 0) as score2,
COALESCE(qty_in_baskets, 0) as qty_in_baskets,
COALESCE(qty_sold, 0) as qty_sold,
COALESCE(notifies_set, 0) as notifies_set,
COALESCE(visibility_score, 0) as visibility_score,
COALESCE(health_score, 0) as health_score,
COALESCE(sold_view_score, 0) as sold_view_score
FROM product_stat_history
WHERE date >= ?
${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
LIMIT ? OFFSET ?
`, [
recentDateStr,
...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
batchSize,
offset
]);
if (rows.length === 0) {
allProcessed = true;
break;
}
// Process rows in smaller batches for better performance
for (let i = 0; i < rows.length; i += insertBatchSize) {
const batch = rows.slice(i, i + insertBatchSize);
if (batch.length === 0) continue;
try {
// Build parameterized query to handle NULL values properly
const values = [];
const placeholders = [];
let placeholderIndex = 1;
for (const row of batch) {
const rowPlaceholders = [
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`,
`$${placeholderIndex++}`
];
placeholders.push(`(${rowPlaceholders.join(', ')})`);
values.push(
row.pid,
validateDate(row.date),
row.score,
row.score2,
row.qty_in_baskets,
row.qty_sold,
row.notifies_set,
row.visibility_score,
row.health_score,
row.sold_view_score
);
}
// Execute direct batch insert without conflict checking
await localConnection.query(`
INSERT INTO imported_product_stat_history (
pid, date, score, score2, qty_in_baskets, qty_sold, notifies_set,
visibility_score, health_score, sold_view_score
)
VALUES ${placeholders.join(',\n')}
`, values);
// All inserts are new records when using this approach
recordsAdded += batch.length;
} catch (error) {
console.error(`Error in batch insert of product_stat_history at offset ${i}:`, error);
errors.push({
table: 'imported_product_stat_history',
batchOffset: i,
batchSize: batch.length,
error: error.message
});
}
}
totalProcessed += rows.length;
offset += rows.length;
// Calculate current rate every 10 seconds or 100,000 records
const now = Date.now();
if (now - lastRateCheck > 10000 || totalProcessed - lastProcessed > 100000) {
const timeElapsed = (now - lastRateCheck) / 1000; // seconds
const recordsProcessed = totalProcessed - lastProcessed;
const currentRate = Math.round(recordsProcessed / timeElapsed);
console.log(`Current import rate: ${currentRate} records/second`);
lastRateCheck = now;
lastProcessed = totalProcessed;
}
// Update progress
outputProgress({
status: "running",
operation: "Historical data import - Product Stat History",
message: `Processed ${totalProcessed} of ${totalCount} records`,
current: totalProcessed,
total: totalCount,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, totalProcessed, totalCount),
rate: calculateRate(startTime, totalProcessed)
});
} catch (error) {
console.error('Error in batch import of product_stat_history:', error);
errors.push({
table: 'imported_product_stat_history',
error: error.message,
offset: offset,
batchSize: batchSize
});
// Try to continue with next batch
offset += batchSize;
}
}
// Update sync status
await localConnection.query(`
INSERT INTO sync_status (table_name, last_sync_timestamp)
VALUES ('imported_product_stat_history', NOW())
ON CONFLICT (table_name) DO UPDATE SET
last_sync_timestamp = NOW()
`);
return { recordsAdded, recordsUpdated, totalProcessed, errors };
} catch (error) {
console.error('Error in product stat history import:', error);
return {
recordsAdded,
recordsUpdated,
totalProcessed,
errors: [...errors, {
table: 'imported_product_stat_history',
error: error.message
}]
};
}
}
module.exports = importHistoricalData;
inventory-server/old/metrics-schema.sql
-377
View File
@@ -1,377 +0,0 @@
-- Disable foreign key checks
SET session_replication_role = 'replica';
-- Temporary tables for batch metrics processing
CREATE TABLE temp_sales_metrics (
pid BIGINT NOT NULL,
daily_sales_avg DECIMAL(10,3),
weekly_sales_avg DECIMAL(10,3),
monthly_sales_avg DECIMAL(10,3),
total_revenue DECIMAL(10,3),
avg_margin_percent DECIMAL(10,3),
first_sale_date DATE,
last_sale_date DATE,
stddev_daily_sales DECIMAL(10,3),
PRIMARY KEY (pid)
);
CREATE TABLE temp_purchase_metrics (
pid BIGINT NOT NULL,
avg_lead_time_days DECIMAL(10,2),
last_purchase_date DATE,
first_received_date DATE,
last_received_date DATE,
stddev_lead_time_days DECIMAL(10,2),
PRIMARY KEY (pid)
);
-- New table for product metrics
CREATE TABLE product_metrics (
pid BIGINT NOT NULL,
last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Sales velocity metrics
daily_sales_avg DECIMAL(10,3),
weekly_sales_avg DECIMAL(10,3),
monthly_sales_avg DECIMAL(10,3),
avg_quantity_per_order DECIMAL(10,3),
number_of_orders INTEGER,
first_sale_date DATE,
last_sale_date DATE,
-- Stock metrics
days_of_inventory INTEGER,
weeks_of_inventory INTEGER,
reorder_point INTEGER,
safety_stock INTEGER,
reorder_qty INTEGER DEFAULT 0,
overstocked_amt INTEGER DEFAULT 0,
-- Financial metrics
avg_margin_percent DECIMAL(10,3),
total_revenue DECIMAL(10,3),
inventory_value DECIMAL(10,3),
cost_of_goods_sold DECIMAL(10,3),
gross_profit DECIMAL(10,3),
gmroi DECIMAL(10,3),
-- Purchase metrics
avg_lead_time_days DECIMAL(10,2),
last_purchase_date DATE,
first_received_date DATE,
last_received_date DATE,
-- Classification metrics
abc_class CHAR(1),
stock_status VARCHAR(20),
-- Turnover metrics
turnover_rate DECIMAL(12,3),
-- Lead time metrics
current_lead_time INTEGER,
target_lead_time INTEGER,
lead_time_status VARCHAR(20),
-- Forecast metrics
forecast_accuracy DECIMAL(5,2) DEFAULT NULL,
forecast_bias DECIMAL(5,2) DEFAULT NULL,
last_forecast_date DATE DEFAULT NULL,
PRIMARY KEY (pid),
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
);
CREATE INDEX idx_metrics_revenue ON product_metrics(total_revenue);
CREATE INDEX idx_metrics_stock_status ON product_metrics(stock_status);
CREATE INDEX idx_metrics_lead_time ON product_metrics(lead_time_status);
CREATE INDEX idx_metrics_turnover ON product_metrics(turnover_rate);
CREATE INDEX idx_metrics_last_calculated ON product_metrics(last_calculated_at);
CREATE INDEX idx_metrics_abc ON product_metrics(abc_class);
CREATE INDEX idx_metrics_sales ON product_metrics(daily_sales_avg, weekly_sales_avg, monthly_sales_avg);
CREATE INDEX idx_metrics_forecast ON product_metrics(forecast_accuracy, forecast_bias);
-- New table for time-based aggregates
CREATE TABLE product_time_aggregates (
pid BIGINT NOT NULL,
year INTEGER NOT NULL,
month INTEGER NOT NULL,
-- Sales metrics
total_quantity_sold INTEGER DEFAULT 0,
total_revenue DECIMAL(10,3) DEFAULT 0,
total_cost DECIMAL(10,3) DEFAULT 0,
order_count INTEGER DEFAULT 0,
-- Stock changes
stock_received INTEGER DEFAULT 0,
stock_ordered INTEGER DEFAULT 0,
-- Calculated fields
avg_price DECIMAL(10,3),
profit_margin DECIMAL(10,3),
inventory_value DECIMAL(10,3),
gmroi DECIMAL(10,3),
PRIMARY KEY (pid, year, month),
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
);
CREATE INDEX idx_date ON product_time_aggregates(year, month);
-- Create vendor_details table
CREATE TABLE vendor_details (
vendor VARCHAR(100) PRIMARY KEY,
contact_name VARCHAR(100),
email VARCHAR(255),
phone VARCHAR(50),
status VARCHAR(20) DEFAULT 'active',
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
CREATE INDEX idx_vendor_details_status ON vendor_details(status);
-- New table for vendor metrics
CREATE TABLE vendor_metrics (
vendor VARCHAR(100) NOT NULL,
last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Performance metrics
avg_lead_time_days DECIMAL(10,3),
on_time_delivery_rate DECIMAL(5,2),
order_fill_rate DECIMAL(5,2),
total_orders INTEGER DEFAULT 0,
total_late_orders INTEGER DEFAULT 0,
total_purchase_value DECIMAL(10,3) DEFAULT 0,
avg_order_value DECIMAL(10,3),
-- Product metrics
active_products INTEGER DEFAULT 0,
total_products INTEGER DEFAULT 0,
-- Financial metrics
total_revenue DECIMAL(10,3) DEFAULT 0,
avg_margin_percent DECIMAL(5,2),
-- Status
status VARCHAR(20) DEFAULT 'active',
PRIMARY KEY (vendor),
FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
);
CREATE INDEX idx_vendor_performance ON vendor_metrics(on_time_delivery_rate);
CREATE INDEX idx_vendor_status ON vendor_metrics(status);
CREATE INDEX idx_vendor_metrics_last_calculated ON vendor_metrics(last_calculated_at);
CREATE INDEX idx_vendor_metrics_orders ON vendor_metrics(total_orders, total_late_orders);
-- New table for category metrics
CREATE TABLE category_metrics (
category_id BIGINT NOT NULL,
last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Product metrics
product_count INTEGER DEFAULT 0,
active_products INTEGER DEFAULT 0,
-- Financial metrics
total_value DECIMAL(15,3) DEFAULT 0,
avg_margin DECIMAL(5,2),
turnover_rate DECIMAL(12,3),
growth_rate DECIMAL(5,2),
-- Status
status VARCHAR(20) DEFAULT 'active',
PRIMARY KEY (category_id),
FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
);
CREATE INDEX idx_category_status ON category_metrics(status);
CREATE INDEX idx_category_growth ON category_metrics(growth_rate);
CREATE INDEX idx_metrics_last_calculated_cat ON category_metrics(last_calculated_at);
CREATE INDEX idx_category_metrics_products ON category_metrics(product_count, active_products);
-- New table for vendor time-based metrics
CREATE TABLE vendor_time_metrics (
vendor VARCHAR(100) NOT NULL,
year INTEGER NOT NULL,
month INTEGER NOT NULL,
-- Order metrics
total_orders INTEGER DEFAULT 0,
late_orders INTEGER DEFAULT 0,
avg_lead_time_days DECIMAL(10,3),
-- Financial metrics
total_purchase_value DECIMAL(10,3) DEFAULT 0,
total_revenue DECIMAL(10,3) DEFAULT 0,
avg_margin_percent DECIMAL(5,2),
PRIMARY KEY (vendor, year, month),
FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
);
CREATE INDEX idx_vendor_date ON vendor_time_metrics(year, month);
-- New table for category time-based metrics
CREATE TABLE category_time_metrics (
category_id BIGINT NOT NULL,
year INTEGER NOT NULL,
month INTEGER NOT NULL,
-- Product metrics
product_count INTEGER DEFAULT 0,
active_products INTEGER DEFAULT 0,
-- Financial metrics
total_value DECIMAL(15,3) DEFAULT 0,
total_revenue DECIMAL(15,3) DEFAULT 0,
avg_margin DECIMAL(5,2),
turnover_rate DECIMAL(12,3),
PRIMARY KEY (category_id, year, month),
FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
);
CREATE INDEX idx_category_date ON category_time_metrics(year, month);
-- New table for category-based sales metrics
CREATE TABLE category_sales_metrics (
category_id BIGINT NOT NULL,
brand VARCHAR(100) NOT NULL,
period_start DATE NOT NULL,
period_end DATE NOT NULL,
avg_daily_sales DECIMAL(10,3) DEFAULT 0,
total_sold INTEGER DEFAULT 0,
num_products INTEGER DEFAULT 0,
avg_price DECIMAL(10,3) DEFAULT 0,
last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (category_id, brand, period_start, period_end),
FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
);
CREATE INDEX idx_category_brand ON category_sales_metrics(category_id, brand);
CREATE INDEX idx_period ON category_sales_metrics(period_start, period_end);
-- New table for brand metrics
CREATE TABLE brand_metrics (
brand VARCHAR(100) NOT NULL,
last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-- Product metrics
product_count INTEGER DEFAULT 0,
active_products INTEGER DEFAULT 0,
-- Stock metrics
total_stock_units INTEGER DEFAULT 0,
total_stock_cost DECIMAL(15,2) DEFAULT 0,
total_stock_retail DECIMAL(15,2) DEFAULT 0,
-- Sales metrics
total_revenue DECIMAL(15,2) DEFAULT 0,
avg_margin DECIMAL(5,2) DEFAULT 0,
growth_rate DECIMAL(5,2) DEFAULT 0,
PRIMARY KEY (brand)
);
CREATE INDEX idx_brand_metrics_last_calculated ON brand_metrics(last_calculated_at);
CREATE INDEX idx_brand_metrics_revenue ON brand_metrics(total_revenue);
CREATE INDEX idx_brand_metrics_growth ON brand_metrics(growth_rate);
-- New table for brand time-based metrics
CREATE TABLE brand_time_metrics (
brand VARCHAR(100) NOT NULL,
year INTEGER NOT NULL,
month INTEGER NOT NULL,
-- Product metrics
product_count INTEGER DEFAULT 0,
active_products INTEGER DEFAULT 0,
-- Stock metrics
total_stock_units INTEGER DEFAULT 0,
total_stock_cost DECIMAL(15,2) DEFAULT 0,
total_stock_retail DECIMAL(15,2) DEFAULT 0,
-- Sales metrics
total_revenue DECIMAL(15,2) DEFAULT 0,
avg_margin DECIMAL(5,2) DEFAULT 0,
growth_rate DECIMAL(5,2) DEFAULT 0,
PRIMARY KEY (brand, year, month)
);
CREATE INDEX idx_brand_time_date ON brand_time_metrics(year, month);
-- New table for sales forecasts
CREATE TABLE sales_forecasts (
pid BIGINT NOT NULL,
forecast_date DATE NOT NULL,
forecast_quantity INTEGER,
confidence_level DECIMAL(5,2),
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (pid, forecast_date),
FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
);
CREATE INDEX idx_forecast_date ON sales_forecasts(forecast_date);
-- New table for category forecasts
CREATE TABLE category_forecasts (
category_id BIGINT NOT NULL,
forecast_date DATE NOT NULL,
forecast_revenue DECIMAL(15,2),
forecast_units INTEGER,
confidence_level DECIMAL(5,2),
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (category_id, forecast_date),
FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
);
CREATE INDEX idx_cat_forecast_date ON category_forecasts(forecast_date);
-- Create views for common calculations
CREATE OR REPLACE VIEW inventory_health AS
WITH stock_levels AS (
SELECT
p.pid,
p.title,
p.SKU,
p.stock_quantity,
p.preorder_count,
pm.daily_sales_avg,
pm.weekly_sales_avg,
pm.monthly_sales_avg,
pm.reorder_point,
pm.safety_stock,
pm.days_of_inventory,
pm.weeks_of_inventory,
pm.stock_status,
pm.abc_class,
pm.turnover_rate,
pm.avg_lead_time_days,
pm.current_lead_time,
pm.target_lead_time,
pm.lead_time_status,
p.cost_price,
p.price,
pm.inventory_value,
pm.gmroi
FROM products p
LEFT JOIN product_metrics pm ON p.pid = pm.pid
WHERE p.managing_stock = true AND p.visible = true
)
SELECT
*,
CASE
WHEN stock_quantity <= safety_stock THEN 'Critical'
WHEN stock_quantity <= reorder_point THEN 'Low'
WHEN stock_quantity > (reorder_point * 3) THEN 'Excess'
ELSE 'Healthy'
END as inventory_status,
CASE
WHEN lead_time_status = 'delayed' AND stock_status = 'low' THEN 'High'
WHEN lead_time_status = 'delayed' OR stock_status = 'low' THEN 'Medium'
ELSE 'Low'
END as risk_level
FROM stock_levels;
-- Create view for category performance trends
CREATE OR REPLACE VIEW category_performance_trends AS
WITH monthly_trends AS (
SELECT
c.cat_id,
c.name as category_name,
ctm.year,
ctm.month,
ctm.product_count,
ctm.active_products,
ctm.total_value,
ctm.total_revenue,
ctm.avg_margin,
ctm.turnover_rate,
LAG(ctm.total_revenue) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_revenue,
LAG(ctm.turnover_rate) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_turnover
FROM categories c
JOIN category_time_metrics ctm ON c.cat_id = ctm.category_id
)
SELECT
*,
CASE
WHEN prev_month_revenue IS NULL THEN 0
ELSE ((total_revenue - prev_month_revenue) / prev_month_revenue) * 100
END as revenue_growth_percent,
CASE
WHEN prev_month_turnover IS NULL THEN 0
ELSE ((turnover_rate - prev_month_turnover) / prev_month_turnover) * 100
END as turnover_growth_percent
FROM monthly_trends;
SET session_replication_role = 'origin';
inventory-server/old/metrics/brand-metrics.js
-321
View File
@@ -1,321 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateBrandMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Brand metrics calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders: 0,
processedPurchaseOrders: 0,
success
};
}
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
`);
processedOrders = parseInt(orderCount.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting brand metrics calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Calculate brand metrics with optimized queries
await connection.query(`
INSERT INTO brand_metrics (
brand,
product_count,
active_products,
total_stock_units,
total_stock_cost,
total_stock_retail,
total_revenue,
avg_margin,
growth_rate
)
WITH filtered_products AS (
SELECT
p.*,
CASE
WHEN p.stock_quantity <= 5000 AND p.stock_quantity >= 0
THEN p.pid
END as valid_pid,
CASE
WHEN p.visible = true
AND p.stock_quantity <= 5000
AND p.stock_quantity >= 0
THEN p.pid
END as active_pid,
CASE
WHEN p.stock_quantity IS NULL
OR p.stock_quantity < 0
OR p.stock_quantity > 5000
THEN 0
ELSE p.stock_quantity
END as valid_stock
FROM products p
WHERE p.brand IS NOT NULL
),
sales_periods AS (
SELECT
p.brand,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as period_revenue,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as period_margin,
COUNT(DISTINCT DATE(o.date)) as period_days,
CASE
WHEN o.date >= CURRENT_DATE - INTERVAL '3 months' THEN 'current'
WHEN o.date BETWEEN CURRENT_DATE - INTERVAL '15 months'
AND CURRENT_DATE - INTERVAL '12 months' THEN 'previous'
END as period_type
FROM filtered_products p
JOIN orders o ON p.pid = o.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '15 months'
GROUP BY p.brand, period_type
),
brand_data AS (
SELECT
p.brand,
COUNT(DISTINCT p.valid_pid) as product_count,
COUNT(DISTINCT p.active_pid) as active_products,
SUM(p.valid_stock) as total_stock_units,
SUM(p.valid_stock * p.cost_price) as total_stock_cost,
SUM(p.valid_stock * p.price) as total_stock_retail,
COALESCE(SUM(o.quantity * (o.price - COALESCE(o.discount, 0))), 0) as total_revenue,
CASE
WHEN SUM(o.quantity * o.price) > 0
THEN GREATEST(
-100.0,
LEAST(
100.0,
(
SUM(o.quantity * o.price) - -- Use gross revenue (before discounts)
SUM(o.quantity * COALESCE(p.cost_price, 0)) -- Total costs
) * 100.0 /
NULLIF(SUM(o.quantity * o.price), 0) -- Divide by gross revenue
)
)
ELSE 0
END as avg_margin
FROM filtered_products p
LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
GROUP BY p.brand
)
SELECT
bd.brand,
bd.product_count,
bd.active_products,
bd.total_stock_units,
bd.total_stock_cost,
bd.total_stock_retail,
bd.total_revenue,
bd.avg_margin,
CASE
WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0
AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0
THEN 100.0
WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0
THEN 0.0
ELSE GREATEST(
-100.0,
LEAST(
((MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) -
MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)) /
NULLIF(ABS(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)), 0)) * 100.0,
999.99
)
)
END as growth_rate
FROM brand_data bd
LEFT JOIN sales_periods sp ON bd.brand = sp.brand
GROUP BY bd.brand, bd.product_count, bd.active_products, bd.total_stock_units,
bd.total_stock_cost, bd.total_stock_retail, bd.total_revenue, bd.avg_margin
ON CONFLICT (brand) DO UPDATE
SET
product_count = EXCLUDED.product_count,
active_products = EXCLUDED.active_products,
total_stock_units = EXCLUDED.total_stock_units,
total_stock_cost = EXCLUDED.total_stock_cost,
total_stock_retail = EXCLUDED.total_stock_retail,
total_revenue = EXCLUDED.total_revenue,
avg_margin = EXCLUDED.avg_margin,
growth_rate = EXCLUDED.growth_rate,
last_calculated_at = CURRENT_TIMESTAMP
`);
processedCount = Math.floor(totalProducts * 0.97);
outputProgress({
status: 'running',
operation: 'Brand metrics calculated, starting time-based metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Calculate brand time-based metrics with optimized query
await connection.query(`
INSERT INTO brand_time_metrics (
brand,
year,
month,
product_count,
active_products,
total_stock_units,
total_stock_cost,
total_stock_retail,
total_revenue,
avg_margin
)
WITH filtered_products AS (
SELECT
p.*,
CASE WHEN p.stock_quantity <= 5000 THEN p.pid END as valid_pid,
CASE WHEN p.visible = true AND p.stock_quantity <= 5000 THEN p.pid END as active_pid,
CASE
WHEN p.stock_quantity IS NULL OR p.stock_quantity < 0 OR p.stock_quantity > 5000 THEN 0
ELSE p.stock_quantity
END as valid_stock
FROM products p
WHERE p.brand IS NOT NULL
),
monthly_metrics AS (
SELECT
p.brand,
EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
COUNT(DISTINCT p.valid_pid) as product_count,
COUNT(DISTINCT p.active_pid) as active_products,
SUM(p.valid_stock) as total_stock_units,
SUM(p.valid_stock * p.cost_price) as total_stock_cost,
SUM(p.valid_stock * p.price) as total_stock_retail,
SUM(o.quantity * o.price) as total_revenue,
CASE
WHEN SUM(o.quantity * o.price) > 0
THEN GREATEST(
-100.0,
LEAST(
100.0,
(
SUM(o.quantity * o.price) - -- Use gross revenue (before discounts)
SUM(o.quantity * COALESCE(p.cost_price, 0)) -- Total costs
) * 100.0 /
NULLIF(SUM(o.quantity * o.price), 0) -- Divide by gross revenue
)
)
ELSE 0
END as avg_margin
FROM filtered_products p
LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
WHERE o.date >= CURRENT_DATE - INTERVAL '12 months'
GROUP BY p.brand, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
)
SELECT *
FROM monthly_metrics
ON CONFLICT (brand, year, month) DO UPDATE
SET
product_count = EXCLUDED.product_count,
active_products = EXCLUDED.active_products,
total_stock_units = EXCLUDED.total_stock_units,
total_stock_cost = EXCLUDED.total_stock_cost,
total_stock_retail = EXCLUDED.total_stock_retail,
total_revenue = EXCLUDED.total_revenue,
avg_margin = EXCLUDED.avg_margin
`);
processedCount = Math.floor(totalProducts * 0.99);
outputProgress({
status: 'running',
operation: 'Brand time-based metrics calculated',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('brand_metrics', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating brand metrics');
throw error;
} finally {
if (connection) {
connection.release();
}
}
}
module.exports = calculateBrandMetrics;
inventory-server/old/metrics/category-metrics.js
-554
View File
@@ -1,554 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateCategoryMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Category metrics calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders: 0,
processedPurchaseOrders: 0,
success
};
}
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
`);
processedOrders = parseInt(orderCount.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting category metrics calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// First, calculate base category metrics
await connection.query(`
INSERT INTO category_metrics (
category_id,
product_count,
active_products,
total_value,
status,
last_calculated_at
)
SELECT
c.cat_id,
COUNT(DISTINCT p.pid) as product_count,
COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
COALESCE(SUM(p.stock_quantity * p.cost_price), 0) as total_value,
c.status,
NOW() as last_calculated_at
FROM categories c
LEFT JOIN product_categories pc ON c.cat_id = pc.cat_id
LEFT JOIN products p ON pc.pid = p.pid
GROUP BY c.cat_id, c.status
ON CONFLICT (category_id) DO UPDATE
SET
product_count = EXCLUDED.product_count,
active_products = EXCLUDED.active_products,
total_value = EXCLUDED.total_value,
status = EXCLUDED.status,
last_calculated_at = EXCLUDED.last_calculated_at
`);
processedCount = Math.floor(totalProducts * 0.90);
outputProgress({
status: 'running',
operation: 'Base category metrics calculated, updating with margin data',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Then update with margin and turnover data
await connection.query(`
WITH category_sales AS (
SELECT
pc.cat_id,
SUM(o.quantity * o.price) as total_sales,
SUM(o.quantity * (o.price - p.cost_price)) as total_margin,
SUM(o.quantity) as units_sold,
AVG(GREATEST(p.stock_quantity, 0)) as avg_stock,
COUNT(DISTINCT DATE(o.date)) as active_days
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
LEFT JOIN turnover_config tc ON
(tc.category_id = pc.cat_id AND tc.vendor = p.vendor) OR
(tc.category_id = pc.cat_id AND tc.vendor IS NULL) OR
(tc.category_id IS NULL AND tc.vendor = p.vendor) OR
(tc.category_id IS NULL AND tc.vendor IS NULL)
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - (COALESCE(tc.calculation_period_days, 30) || ' days')::INTERVAL
GROUP BY pc.cat_id
)
UPDATE category_metrics
SET
avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
turnover_rate = CASE
WHEN cs.avg_stock > 0 AND cs.active_days > 0
THEN LEAST(
(cs.units_sold / cs.avg_stock) * (365.0 / cs.active_days),
999.99
)
ELSE 0
END,
last_calculated_at = NOW()
FROM category_sales cs
WHERE category_id = cs.cat_id
`);
processedCount = Math.floor(totalProducts * 0.95);
outputProgress({
status: 'running',
operation: 'Margin data updated, calculating growth rates',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Finally update growth rates
await connection.query(`
WITH current_period AS (
SELECT
pc.cat_id,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) /
(1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as gross_profit,
COUNT(DISTINCT DATE(o.date)) as days
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '3 months'
GROUP BY pc.cat_id
),
previous_period AS (
SELECT
pc.cat_id,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) /
(1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
COUNT(DISTINCT DATE(o.date)) as days
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
WHERE o.canceled = false
AND o.date BETWEEN CURRENT_DATE - INTERVAL '15 months'
AND CURRENT_DATE - INTERVAL '12 months'
GROUP BY pc.cat_id
),
trend_data AS (
SELECT
pc.cat_id,
EXTRACT(MONTH FROM o.date) as month,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) /
(1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
COUNT(DISTINCT DATE(o.date)) as days_in_month
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '15 months'
GROUP BY pc.cat_id, EXTRACT(MONTH FROM o.date)
),
trend_stats AS (
SELECT
cat_id,
COUNT(*) as n,
AVG(month) as avg_x,
AVG(revenue / NULLIF(days_in_month, 0)) as avg_y,
SUM(month * (revenue / NULLIF(days_in_month, 0))) as sum_xy,
SUM(month * month) as sum_xx
FROM trend_data
GROUP BY cat_id
HAVING COUNT(*) >= 6
),
trend_analysis AS (
SELECT
cat_id,
((n * sum_xy) - (avg_x * n * avg_y)) /
NULLIF((n * sum_xx) - (n * avg_x * avg_x), 0) as trend_slope,
avg_y as avg_daily_revenue
FROM trend_stats
),
margin_calc AS (
SELECT
pc.cat_id,
CASE
WHEN SUM(o.quantity * o.price) > 0 THEN
GREATEST(
-100.0,
LEAST(
100.0,
(
SUM(o.quantity * o.price) - -- Use gross revenue (before discounts)
SUM(o.quantity * COALESCE(p.cost_price, 0)) -- Total costs
) * 100.0 /
NULLIF(SUM(o.quantity * o.price), 0) -- Divide by gross revenue
)
)
ELSE NULL
END as avg_margin
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '3 months'
GROUP BY pc.cat_id
),
combined_metrics AS (
SELECT
COALESCE(cp.cat_id, pp.cat_id) as category_id,
CASE
WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
WHEN ta.trend_slope IS NOT NULL THEN
GREATEST(
-100.0,
LEAST(
(ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
999.99
)
)
ELSE
GREATEST(
-100.0,
LEAST(
((COALESCE(cp.revenue, 0) - pp.revenue) /
NULLIF(ABS(pp.revenue), 0)) * 100.0,
999.99
)
)
END as growth_rate,
mc.avg_margin
FROM current_period cp
FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
)
UPDATE category_metrics cm
SET
growth_rate = CASE
WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
WHEN ta.trend_slope IS NOT NULL THEN
GREATEST(
-100.0,
LEAST(
(ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
999.99
)
)
ELSE
GREATEST(
-100.0,
LEAST(
((COALESCE(cp.revenue, 0) - pp.revenue) /
NULLIF(ABS(pp.revenue), 0)) * 100.0,
999.99
)
)
END,
avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
last_calculated_at = NOW()
FROM current_period cp
FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
WHERE cm.category_id = COALESCE(cp.cat_id, pp.cat_id)
`);
processedCount = Math.floor(totalProducts * 0.97);
outputProgress({
status: 'running',
operation: 'Growth rates calculated, updating time-based metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Calculate time-based metrics
await connection.query(`
INSERT INTO category_time_metrics (
category_id,
year,
month,
product_count,
active_products,
total_value,
total_revenue,
avg_margin,
turnover_rate
)
SELECT
pc.cat_id,
EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
COUNT(DISTINCT p.pid) as product_count,
COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
SUM(p.stock_quantity * p.cost_price) as total_value,
SUM(o.quantity * o.price) as total_revenue,
CASE
WHEN SUM(o.quantity * o.price) > 0 THEN
LEAST(
GREATEST(
SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0))) * 100.0 /
SUM(o.quantity * o.price),
-100
),
100
)
ELSE 0
END as avg_margin,
COALESCE(
LEAST(
SUM(o.quantity) / NULLIF(AVG(GREATEST(p.stock_quantity, 0)), 0),
999.99
),
0
) as turnover_rate
FROM product_categories pc
JOIN products p ON pc.pid = p.pid
JOIN orders o ON p.pid = o.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '12 months'
GROUP BY pc.cat_id, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
ON CONFLICT (category_id, year, month) DO UPDATE
SET
product_count = EXCLUDED.product_count,
active_products = EXCLUDED.active_products,
total_value = EXCLUDED.total_value,
total_revenue = EXCLUDED.total_revenue,
avg_margin = EXCLUDED.avg_margin,
turnover_rate = EXCLUDED.turnover_rate
`);
processedCount = Math.floor(totalProducts * 0.99);
outputProgress({
status: 'running',
operation: 'Time-based metrics calculated, updating category-sales metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Calculate category-sales metrics
await connection.query(`
INSERT INTO category_sales_metrics (
category_id,
brand,
period_start,
period_end,
avg_daily_sales,
total_sold,
num_products,
avg_price,
last_calculated_at
)
WITH date_ranges AS (
SELECT
CURRENT_DATE - INTERVAL '30 days' as period_start,
CURRENT_DATE as period_end
UNION ALL
SELECT
CURRENT_DATE - INTERVAL '90 days',
CURRENT_DATE - INTERVAL '31 days'
UNION ALL
SELECT
CURRENT_DATE - INTERVAL '180 days',
CURRENT_DATE - INTERVAL '91 days'
UNION ALL
SELECT
CURRENT_DATE - INTERVAL '365 days',
CURRENT_DATE - INTERVAL '181 days'
),
sales_data AS (
SELECT
pc.cat_id,
COALESCE(p.brand, 'Unknown') as brand,
dr.period_start,
dr.period_end,
COUNT(DISTINCT p.pid) as num_products,
SUM(o.quantity) as total_sold,
SUM(o.quantity * o.price) as total_revenue,
COUNT(DISTINCT DATE(o.date)) as num_days
FROM products p
JOIN product_categories pc ON p.pid = pc.pid
JOIN orders o ON p.pid = o.pid
CROSS JOIN date_ranges dr
WHERE o.canceled = false
AND o.date BETWEEN dr.period_start AND dr.period_end
GROUP BY pc.cat_id, p.brand, dr.period_start, dr.period_end
)
SELECT
cat_id as category_id,
brand,
period_start,
period_end,
CASE
WHEN num_days > 0
THEN total_sold / num_days
ELSE 0
END as avg_daily_sales,
total_sold,
num_products,
CASE
WHEN total_sold > 0
THEN total_revenue / total_sold
ELSE 0
END as avg_price,
NOW() as last_calculated_at
FROM sales_data
ON CONFLICT (category_id, brand, period_start, period_end) DO UPDATE
SET
avg_daily_sales = EXCLUDED.avg_daily_sales,
total_sold = EXCLUDED.total_sold,
num_products = EXCLUDED.num_products,
avg_price = EXCLUDED.avg_price,
last_calculated_at = EXCLUDED.last_calculated_at
`);
processedCount = Math.floor(totalProducts * 1.0);
outputProgress({
status: 'running',
operation: 'Category-sales metrics calculated',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('category_metrics', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating category metrics');
throw error;
} finally {
if (connection) {
connection.release();
}
}
}
module.exports = calculateCategoryMetrics;
inventory-server/old/metrics/financial-metrics.js
-214
View File
@@ -1,214 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateFinancialMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Financial metrics calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders: 0,
processedPurchaseOrders: 0,
success
};
}
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'
`);
processedOrders = parseInt(orderCount.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting financial metrics calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// First, calculate beginning inventory values (12 months ago)
await connection.query(`
CREATE TEMPORARY TABLE IF NOT EXISTS temp_beginning_inventory AS
WITH beginning_inventory_calc AS (
SELECT
p.pid,
p.stock_quantity as current_quantity,
COALESCE(SUM(o.quantity), 0) as sold_quantity,
COALESCE(SUM(po.received), 0) as received_quantity,
GREATEST(0, (p.stock_quantity + COALESCE(SUM(o.quantity), 0) - COALESCE(SUM(po.received), 0))) as beginning_quantity,
p.cost_price
FROM
products p
LEFT JOIN
orders o ON p.pid = o.pid
AND o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '12 months'::interval
LEFT JOIN
purchase_orders po ON p.pid = po.pid
AND po.received_date IS NOT NULL
AND po.received_date >= CURRENT_DATE - INTERVAL '12 months'::interval
GROUP BY
p.pid, p.stock_quantity, p.cost_price
)
SELECT
pid,
beginning_quantity,
beginning_quantity * cost_price as beginning_value,
current_quantity * cost_price as current_value,
((beginning_quantity * cost_price) + (current_quantity * cost_price)) / 2 as average_inventory_value
FROM
beginning_inventory_calc
`);
processedCount = Math.floor(totalProducts * 0.60);
outputProgress({
status: 'running',
operation: 'Beginning inventory values calculated, computing financial metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Calculate financial metrics with optimized query and standard formulas
await connection.query(`
WITH product_financials AS (
SELECT
p.pid,
COALESCE(bi.average_inventory_value, p.cost_price * p.stock_quantity) as avg_inventory_value,
p.cost_price * p.stock_quantity as current_inventory_value,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as total_revenue,
SUM(o.quantity * COALESCE(o.costeach, 0)) as cost_of_goods_sold,
SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - COALESCE(o.costeach, 0))) as gross_profit,
MIN(o.date) as first_sale_date,
MAX(o.date) as last_sale_date,
EXTRACT(DAY FROM (MAX(o.date)::timestamp with time zone - MIN(o.date)::timestamp with time zone)) + 1 as calculation_period_days,
COUNT(DISTINCT DATE(o.date)) as active_days
FROM products p
LEFT JOIN orders o ON p.pid = o.pid
LEFT JOIN temp_beginning_inventory bi ON p.pid = bi.pid
WHERE o.canceled = false
AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'::interval
GROUP BY p.pid, p.cost_price, p.stock_quantity, bi.average_inventory_value
)
UPDATE product_metrics pm
SET
inventory_value = COALESCE(pf.current_inventory_value, 0)::decimal(10,3),
total_revenue = COALESCE(pf.total_revenue, 0)::decimal(10,3),
cost_of_goods_sold = COALESCE(pf.cost_of_goods_sold, 0)::decimal(10,3),
gross_profit = COALESCE(pf.gross_profit, 0)::decimal(10,3),
turnover_rate = CASE
WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN
COALESCE(pf.cost_of_goods_sold, 0) / NULLIF(pf.avg_inventory_value, 0)
ELSE 0
END::decimal(12,3),
gmroi = CASE
WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN
COALESCE(pf.gross_profit, 0) / NULLIF(pf.avg_inventory_value, 0)
ELSE 0
END::decimal(10,3),
last_calculated_at = CURRENT_TIMESTAMP
FROM product_financials pf
WHERE pm.pid = pf.pid
`);
processedCount = Math.floor(totalProducts * 0.65);
outputProgress({
status: 'running',
operation: 'Base financial metrics calculated, updating time aggregates',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Clean up temporary tables
await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('financial_metrics', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating financial metrics');
throw error;
} finally {
if (connection) {
try {
// Make sure temporary tables are always cleaned up
await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
} catch (err) {
console.error('Error cleaning up temp tables:', err);
}
connection.release();
}
}
}
module.exports = calculateFinancialMetrics;
inventory-server/old/metrics/product-metrics.js
-736
View File
@@ -1,736 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
// Helper function to handle NaN and undefined values
function sanitizeValue(value) {
if (value === undefined || value === null || Number.isNaN(value)) {
return null;
}
return value;
}
async function calculateProductMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
let connection;
let success = false;
let processedOrders = 0;
const BATCH_SIZE = 5000;
try {
connection = await getConnection();
// Skip flags are inherited from the parent scope
const SKIP_PRODUCT_BASE_METRICS = 0;
const SKIP_PRODUCT_TIME_AGGREGATES = 0;
// Get total product count if not provided
if (!totalProducts) {
const productCount = await connection.query('SELECT COUNT(*) as count FROM products');
totalProducts = parseInt(productCount.rows[0].count);
}
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Product metrics calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
}
// First ensure all products have a metrics record
await connection.query(`
INSERT INTO product_metrics (pid, last_calculated_at)
SELECT pid, NOW()
FROM products
ON CONFLICT (pid) DO NOTHING
`);
// Get threshold settings once
const thresholds = await connection.query(`
SELECT critical_days, reorder_days, overstock_days, low_stock_threshold
FROM stock_thresholds
WHERE category_id IS NULL AND vendor IS NULL
LIMIT 1
`);
// Check if threshold data was returned
if (!thresholds.rows || thresholds.rows.length === 0) {
console.warn('No default thresholds found in the database. Using explicit type casting in the query.');
}
const defaultThresholds = thresholds.rows[0];
// Get financial calculation configuration parameters
const financialConfig = await connection.query(`
SELECT
order_cost,
holding_rate,
service_level_z_score,
min_reorder_qty,
default_reorder_qty,
default_safety_stock
FROM financial_calc_config
WHERE id = 1
LIMIT 1
`);
const finConfig = financialConfig.rows[0] || {
order_cost: 25.00,
holding_rate: 0.25,
service_level_z_score: 1.96,
min_reorder_qty: 1,
default_reorder_qty: 5,
default_safety_stock: 5
};
// Calculate base product metrics
if (!SKIP_PRODUCT_BASE_METRICS) {
outputProgress({
status: 'running',
operation: 'Starting base product metrics calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
`);
processedOrders = parseInt(orderCount.rows[0].count);
// Clear temporary tables
await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
// Create temp_sales_metrics
await connection.query(`
CREATE TEMPORARY TABLE temp_sales_metrics (
pid BIGINT NOT NULL,
daily_sales_avg DECIMAL(10,3),
weekly_sales_avg DECIMAL(10,3),
monthly_sales_avg DECIMAL(10,3),
total_revenue DECIMAL(10,3),
avg_margin_percent DECIMAL(10,3),
first_sale_date DATE,
last_sale_date DATE,
stddev_daily_sales DECIMAL(10,3),
PRIMARY KEY (pid)
)
`);
// Create temp_purchase_metrics
await connection.query(`
CREATE TEMPORARY TABLE temp_purchase_metrics (
pid BIGINT NOT NULL,
avg_lead_time_days DECIMAL(10,2),
last_purchase_date DATE,
first_received_date DATE,
last_received_date DATE,
stddev_lead_time_days DECIMAL(10,2),
PRIMARY KEY (pid)
)
`);
// Populate temp_sales_metrics with base stats and sales averages
await connection.query(`
INSERT INTO temp_sales_metrics
SELECT
p.pid,
COALESCE(SUM(o.quantity) / NULLIF(COUNT(DISTINCT DATE(o.date)), 0), 0) as daily_sales_avg,
COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 7), 0), 0) as weekly_sales_avg,
COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 30), 0), 0) as monthly_sales_avg,
COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
CASE
WHEN SUM(o.quantity * o.price) > 0
THEN ((SUM(o.quantity * o.price) - SUM(o.quantity * p.cost_price)) / SUM(o.quantity * o.price)) * 100
ELSE 0
END as avg_margin_percent,
MIN(o.date) as first_sale_date,
MAX(o.date) as last_sale_date,
COALESCE(STDDEV_SAMP(daily_qty.quantity), 0) as stddev_daily_sales
FROM products p
LEFT JOIN orders o ON p.pid = o.pid
AND o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '90 days'
LEFT JOIN (
SELECT
pid,
DATE(date) as sale_date,
SUM(quantity) as quantity
FROM orders
WHERE canceled = false
AND date >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY pid, DATE(date)
) daily_qty ON p.pid = daily_qty.pid
GROUP BY p.pid
`);
// Populate temp_purchase_metrics with timeout protection
await Promise.race([
connection.query(`
INSERT INTO temp_purchase_metrics
SELECT
p.pid,
AVG(
CASE
WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL
THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
ELSE NULL
END
) as avg_lead_time_days,
MAX(po.date) as last_purchase_date,
MIN(po.received_date) as first_received_date,
MAX(po.received_date) as last_received_date,
STDDEV_SAMP(
CASE
WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL
THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
ELSE NULL
END
) as stddev_lead_time_days
FROM products p
LEFT JOIN purchase_orders po ON p.pid = po.pid
AND po.received_date IS NOT NULL
AND po.date IS NOT NULL
AND po.date >= CURRENT_DATE - INTERVAL '365 days'
GROUP BY p.pid
`),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('Timeout: temp_purchase_metrics query took too long')), 60000)
)
]).catch(async (err) => {
logError(err, 'Error populating temp_purchase_metrics, continuing with empty table');
// Create an empty fallback to continue processing
await connection.query(`
INSERT INTO temp_purchase_metrics
SELECT
p.pid,
30.0 as avg_lead_time_days,
NULL as last_purchase_date,
NULL as first_received_date,
NULL as last_received_date,
0.0 as stddev_lead_time_days
FROM products p
LEFT JOIN temp_purchase_metrics tpm ON p.pid = tpm.pid
WHERE tpm.pid IS NULL
`);
});
// Process updates in batches
let lastPid = 0;
let batchCount = 0;
const MAX_BATCHES = 1000; // Safety limit for number of batches to prevent infinite loops
while (batchCount < MAX_BATCHES) {
if (isCancelled) break;
batchCount++;
const batch = await connection.query(
'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
[lastPid, BATCH_SIZE]
);
if (batch.rows.length === 0) break;
// Process the entire batch in a single efficient query
const lowStockThreshold = parseInt(defaultThresholds?.low_stock_threshold) || 5;
const criticalDays = parseInt(defaultThresholds?.critical_days) || 7;
const reorderDays = parseInt(defaultThresholds?.reorder_days) || 14;
const overstockDays = parseInt(defaultThresholds?.overstock_days) || 90;
const serviceLevel = parseFloat(finConfig?.service_level_z_score) || 1.96;
const defaultSafetyStock = parseInt(finConfig?.default_safety_stock) || 5;
const defaultReorderQty = parseInt(finConfig?.default_reorder_qty) || 5;
const orderCost = parseFloat(finConfig?.order_cost) || 25.00;
const holdingRate = parseFloat(finConfig?.holding_rate) || 0.25;
const minReorderQty = parseInt(finConfig?.min_reorder_qty) || 1;
await connection.query(`
UPDATE product_metrics pm
SET
inventory_value = p.stock_quantity * NULLIF(p.cost_price, 0),
daily_sales_avg = COALESCE(sm.daily_sales_avg, 0),
weekly_sales_avg = COALESCE(sm.weekly_sales_avg, 0),
monthly_sales_avg = COALESCE(sm.monthly_sales_avg, 0),
total_revenue = COALESCE(sm.total_revenue, 0),
avg_margin_percent = COALESCE(sm.avg_margin_percent, 0),
first_sale_date = sm.first_sale_date,
last_sale_date = sm.last_sale_date,
avg_lead_time_days = COALESCE(lm.avg_lead_time_days, 30.0),
days_of_inventory = CASE
WHEN COALESCE(sm.daily_sales_avg, 0) > 0
THEN FLOOR(p.stock_quantity / NULLIF(sm.daily_sales_avg, 0))
ELSE NULL
END,
weeks_of_inventory = CASE
WHEN COALESCE(sm.weekly_sales_avg, 0) > 0
THEN FLOOR(p.stock_quantity / NULLIF(sm.weekly_sales_avg, 0))
ELSE NULL
END,
stock_status = CASE
WHEN p.stock_quantity <= 0 THEN 'Out of Stock'
WHEN COALESCE(sm.daily_sales_avg, 0) = 0 AND p.stock_quantity <= ${lowStockThreshold} THEN 'Low Stock'
WHEN COALESCE(sm.daily_sales_avg, 0) = 0 THEN 'In Stock'
WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${criticalDays} THEN 'Critical'
WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${reorderDays} THEN 'Reorder'
WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays} THEN 'Overstocked'
ELSE 'Healthy'
END,
safety_stock = CASE
WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
CEIL(
${serviceLevel} * SQRT(
GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
)
)
ELSE ${defaultSafetyStock}
END,
reorder_point = CASE
WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
CEIL(sm.daily_sales_avg * GREATEST(0, COALESCE(lm.avg_lead_time_days, 30.0))) +
(CASE
WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
CEIL(
${serviceLevel} * SQRT(
GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
)
)
ELSE ${defaultSafetyStock}
END)
ELSE ${lowStockThreshold}
END,
reorder_qty = CASE
WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND NULLIF(p.cost_price, 0) IS NOT NULL AND NULLIF(p.cost_price, 0) > 0 THEN
GREATEST(
CEIL(SQRT(
(2 * (sm.daily_sales_avg * 365) * ${orderCost}) /
NULLIF(p.cost_price * ${holdingRate}, 0)
)),
${minReorderQty}
)
ELSE ${defaultReorderQty}
END,
overstocked_amt = CASE
WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays}
THEN GREATEST(0, p.stock_quantity - CEIL(sm.daily_sales_avg * ${overstockDays}))
ELSE 0
END,
last_calculated_at = NOW()
FROM products p
LEFT JOIN temp_sales_metrics sm ON p.pid = sm.pid
LEFT JOIN temp_purchase_metrics lm ON p.pid = lm.pid
WHERE p.pid = ANY($1::BIGINT[])
AND pm.pid = p.pid
`, [batch.rows.map(row => row.pid)]);
lastPid = batch.rows[batch.rows.length - 1].pid;
processedCount += batch.rows.length;
outputProgress({
status: 'running',
operation: 'Processing base metrics batch',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
}
// Add safety check if the loop processed MAX_BATCHES
if (batchCount >= MAX_BATCHES) {
logError(new Error(`Reached maximum batch count (${MAX_BATCHES}). Process may have entered an infinite loop.`), 'Batch processing safety limit reached');
}
}
// Calculate forecast accuracy and bias in batches
let forecastPid = 0;
while (true) {
if (isCancelled) break;
const forecastBatch = await connection.query(
'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
[forecastPid, BATCH_SIZE]
);
if (forecastBatch.rows.length === 0) break;
const forecastPidArray = forecastBatch.rows.map(row => row.pid);
// Use array_to_string to convert the array to a string of comma-separated values
await connection.query(`
WITH forecast_metrics AS (
SELECT
sf.pid,
AVG(CASE
WHEN o.quantity > 0
THEN ABS(sf.forecast_quantity - o.quantity) / o.quantity * 100
ELSE 100
END) as avg_forecast_error,
AVG(CASE
WHEN o.quantity > 0
THEN (sf.forecast_quantity - o.quantity) / o.quantity * 100
ELSE 0
END) as avg_forecast_bias,
MAX(sf.forecast_date) as last_forecast_date
FROM sales_forecasts sf
JOIN orders o ON sf.pid = o.pid
AND DATE(o.date) = sf.forecast_date
WHERE o.canceled = false
AND sf.forecast_date >= CURRENT_DATE - INTERVAL '90 days'
AND sf.pid = ANY('{${forecastPidArray.join(',')}}'::BIGINT[])
GROUP BY sf.pid
)
UPDATE product_metrics pm
SET
forecast_accuracy = GREATEST(0, 100 - LEAST(fm.avg_forecast_error, 100)),
forecast_bias = GREATEST(-100, LEAST(fm.avg_forecast_bias, 100)),
last_forecast_date = fm.last_forecast_date,
last_calculated_at = NOW()
FROM forecast_metrics fm
WHERE pm.pid = fm.pid
`);
forecastPid = forecastBatch.rows[forecastBatch.rows.length - 1].pid;
}
// Calculate product time aggregates
if (!SKIP_PRODUCT_TIME_AGGREGATES) {
outputProgress({
status: 'running',
operation: 'Starting product time aggregates calculation',
current: processedCount || 0,
total: totalProducts || 0,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
rate: calculateRate(startTime, processedCount || 0),
percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Note: The time-aggregates calculation has been moved to time-aggregates.js
// This module will not duplicate that functionality
processedCount = Math.floor(totalProducts * 0.6);
outputProgress({
status: 'running',
operation: 'Product time aggregates calculation delegated to time-aggregates module',
current: processedCount || 0,
total: totalProducts || 0,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
rate: calculateRate(startTime, processedCount || 0),
percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
} else {
processedCount = Math.floor(totalProducts * 0.6);
outputProgress({
status: 'running',
operation: 'Skipping product time aggregates calculation',
current: processedCount || 0,
total: totalProducts || 0,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
rate: calculateRate(startTime, processedCount || 0),
percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
}
// Calculate ABC classification
outputProgress({
status: 'running',
operation: 'Starting ABC classification',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0, // This module doesn't process POs
success
};
const abcConfig = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
const abcThresholds = abcConfig.rows[0] || { a_threshold: 20, b_threshold: 50 };
// Extract values and ensure they are valid numbers
const aThreshold = parseFloat(abcThresholds.a_threshold) || 20;
const bThreshold = parseFloat(abcThresholds.b_threshold) || 50;
// First, create and populate the rankings table with an index
await connection.query('DROP TABLE IF EXISTS temp_revenue_ranks');
await connection.query(`
CREATE TEMPORARY TABLE temp_revenue_ranks (
pid BIGINT NOT NULL,
total_revenue DECIMAL(10,3),
rank_num INT,
dense_rank_num INT,
percentile DECIMAL(5,2),
total_count INT,
PRIMARY KEY (pid)
)
`);
await connection.query('CREATE INDEX ON temp_revenue_ranks (rank_num)');
await connection.query('CREATE INDEX ON temp_revenue_ranks (dense_rank_num)');
await connection.query('CREATE INDEX ON temp_revenue_ranks (percentile)');
// Calculate rankings with proper tie handling
await connection.query(`
INSERT INTO temp_revenue_ranks
WITH revenue_data AS (
SELECT
pid,
total_revenue,
COUNT(*) OVER () as total_count,
PERCENT_RANK() OVER (ORDER BY total_revenue DESC) * 100 as percentile,
RANK() OVER (ORDER BY total_revenue DESC) as rank_num,
DENSE_RANK() OVER (ORDER BY total_revenue DESC) as dense_rank_num
FROM product_metrics
WHERE total_revenue > 0
)
SELECT
pid,
total_revenue,
rank_num,
dense_rank_num,
percentile,
total_count
FROM revenue_data
`);
// Get total count for percentage calculation
const rankingCount = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
const totalCount = parseInt(rankingCount.rows[0].total_count) || 1;
// Process updates in batches
let abcProcessedCount = 0;
const batchSize = 5000;
const maxPid = await connection.query('SELECT MAX(pid) as max_pid FROM products');
const maxProductId = parseInt(maxPid.rows[0].max_pid);
while (abcProcessedCount < maxProductId) {
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Get a batch of PIDs that need updating
const pids = await connection.query(`
SELECT pm.pid
FROM product_metrics pm
LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
WHERE pm.pid > $1
AND (pm.abc_class IS NULL
OR pm.abc_class !=
CASE
WHEN tr.pid IS NULL THEN 'C'
WHEN tr.percentile <= ${aThreshold} THEN 'A'
WHEN tr.percentile <= ${bThreshold} THEN 'B'
ELSE 'C'
END)
ORDER BY pm.pid
LIMIT $2
`, [abcProcessedCount, batchSize]);
if (pids.rows.length === 0) break;
const pidValues = pids.rows.map(row => row.pid);
await connection.query(`
UPDATE product_metrics pm
SET abc_class =
CASE
WHEN tr.pid IS NULL THEN 'C'
WHEN tr.percentile <= ${aThreshold} THEN 'A'
WHEN tr.percentile <= ${bThreshold} THEN 'B'
ELSE 'C'
END,
last_calculated_at = NOW()
FROM (SELECT pid, percentile FROM temp_revenue_ranks) tr
WHERE pm.pid = tr.pid AND pm.pid = ANY($1::BIGINT[])
OR (pm.pid = ANY($1::BIGINT[]) AND tr.pid IS NULL)
`, [pidValues]);
// Now update turnover rate with proper handling of zero inventory periods
await connection.query(`
UPDATE product_metrics pm
SET
turnover_rate = CASE
WHEN sales.avg_nonzero_stock > 0 AND sales.active_days > 0
THEN LEAST(
(sales.total_sold / sales.avg_nonzero_stock) * (365.0 / sales.active_days),
999.99
)
ELSE 0
END,
last_calculated_at = NOW()
FROM (
SELECT
o.pid,
SUM(o.quantity) as total_sold,
COUNT(DISTINCT DATE(o.date)) as active_days,
AVG(CASE
WHEN p.stock_quantity > 0 THEN p.stock_quantity
ELSE NULL
END) as avg_nonzero_stock
FROM orders o
JOIN products p ON o.pid = p.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '90 days'
AND o.pid = ANY($1::BIGINT[])
GROUP BY o.pid
) sales
WHERE pm.pid = sales.pid
`, [pidValues]);
abcProcessedCount = pids.rows[pids.rows.length - 1].pid;
// Calculate progress proportionally to total products
processedCount = Math.floor(totalProducts * (0.60 + (abcProcessedCount / maxProductId) * 0.2));
outputProgress({
status: 'running',
operation: 'ABC classification progress',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
}
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('product_metrics', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount || 0,
processedOrders: processedOrders || 0,
processedPurchaseOrders: 0, // This module doesn't process POs
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating product metrics');
throw error;
} finally {
// Always clean up temporary tables, even if an error occurred
if (connection) {
try {
await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
} catch (err) {
console.error('Error cleaning up temporary tables:', err);
}
// Make sure to release the connection
connection.release();
}
}
}
function calculateStockStatus(stock, config, daily_sales_avg, weekly_sales_avg, monthly_sales_avg) {
if (stock <= 0) {
return 'Out of Stock';
}
// Use the most appropriate sales average based on data quality
let sales_avg = daily_sales_avg;
if (sales_avg === 0) {
sales_avg = weekly_sales_avg / 7;
}
if (sales_avg === 0) {
sales_avg = monthly_sales_avg / 30;
}
if (sales_avg === 0) {
return stock <= config.low_stock_threshold ? 'Low Stock' : 'In Stock';
}
const days_of_stock = stock / sales_avg;
if (days_of_stock <= config.critical_days) {
return 'Critical';
} else if (days_of_stock <= config.reorder_days) {
return 'Reorder';
} else if (days_of_stock > config.overstock_days) {
return 'Overstocked';
}
return 'Healthy';
}
// Note: calculateReorderQuantities function has been removed as its logic has been incorporated
// in the main SQL query with configurable parameters
module.exports = calculateProductMetrics;
inventory-server/old/metrics/sales-forecasts.js
-440
View File
@@ -1,440 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateSalesForecasts(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Sales forecasts calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders: 0,
processedPurchaseOrders: 0,
success
};
}
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '90 days'
`);
processedOrders = parseInt(orderCount.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting sales forecasts calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// First, create a temporary table for forecast dates
await connection.query(`
CREATE TEMPORARY TABLE IF NOT EXISTS temp_forecast_dates (
forecast_date DATE,
day_of_week INT,
month INT,
PRIMARY KEY (forecast_date)
)
`);
await connection.query(`
INSERT INTO temp_forecast_dates
SELECT
CURRENT_DATE + (n || ' days')::INTERVAL as forecast_date,
EXTRACT(DOW FROM CURRENT_DATE + (n || ' days')::INTERVAL) + 1 as day_of_week,
EXTRACT(MONTH FROM CURRENT_DATE + (n || ' days')::INTERVAL) as month
FROM (
SELECT a.n + b.n * 10 as n
FROM
(SELECT 0 as n UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION
SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
(SELECT 0 as n UNION SELECT 1 UNION SELECT 2) b
ORDER BY n
LIMIT 31
) numbers
`);
processedCount = Math.floor(totalProducts * 0.92);
outputProgress({
status: 'running',
operation: 'Forecast dates prepared, calculating daily sales stats',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Create temporary table for daily sales stats
await connection.query(`
CREATE TEMPORARY TABLE temp_daily_sales AS
SELECT
o.pid,
EXTRACT(DOW FROM o.date) + 1 as day_of_week,
SUM(o.quantity) as daily_quantity,
SUM(o.price * o.quantity) as daily_revenue,
COUNT(DISTINCT DATE(o.date)) as day_count
FROM orders o
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY o.pid, EXTRACT(DOW FROM o.date) + 1
`);
processedCount = Math.floor(totalProducts * 0.94);
outputProgress({
status: 'running',
operation: 'Daily sales stats calculated, preparing product stats',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Create temporary table for product stats
await connection.query(`
CREATE TEMPORARY TABLE temp_product_stats AS
SELECT
pid,
AVG(daily_revenue) as overall_avg_revenue,
SUM(day_count) as total_days
FROM temp_daily_sales
GROUP BY pid
`);
processedCount = Math.floor(totalProducts * 0.96);
outputProgress({
status: 'running',
operation: 'Product stats prepared, calculating product-level forecasts',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Calculate product-level forecasts
await connection.query(`
INSERT INTO sales_forecasts (
pid,
forecast_date,
forecast_quantity,
confidence_level,
created_at
)
WITH daily_stats AS (
SELECT
ds.pid,
AVG(ds.daily_quantity) as avg_daily_qty,
STDDEV(ds.daily_quantity) as std_daily_qty,
COUNT(DISTINCT ds.day_count) as data_points,
SUM(ds.day_count) as total_days,
AVG(ds.daily_revenue) as avg_daily_revenue,
STDDEV(ds.daily_revenue) as std_daily_revenue,
MIN(ds.daily_quantity) as min_daily_qty,
MAX(ds.daily_quantity) as max_daily_qty,
-- Calculate variance without using LAG
COALESCE(
STDDEV(ds.daily_quantity) / NULLIF(AVG(ds.daily_quantity), 0),
0
) as daily_variance_ratio
FROM temp_daily_sales ds
GROUP BY ds.pid
HAVING AVG(ds.daily_quantity) > 0
)
SELECT
ds.pid,
fd.forecast_date,
GREATEST(0,
ROUND(
ds.avg_daily_qty *
(1 + COALESCE(sf.seasonality_factor, 0))
)
) as forecast_quantity,
CASE
WHEN ds.total_days >= 60 AND ds.daily_variance_ratio < 0.5 THEN 90
WHEN ds.total_days >= 60 THEN 85
WHEN ds.total_days >= 30 AND ds.daily_variance_ratio < 0.5 THEN 80
WHEN ds.total_days >= 30 THEN 75
WHEN ds.total_days >= 14 AND ds.daily_variance_ratio < 0.5 THEN 70
WHEN ds.total_days >= 14 THEN 65
ELSE 60
END as confidence_level,
NOW() as created_at
FROM daily_stats ds
JOIN temp_product_stats ps ON ds.pid = ps.pid
CROSS JOIN temp_forecast_dates fd
LEFT JOIN sales_seasonality sf ON fd.month = sf.month
GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, sf.seasonality_factor,
ds.avg_daily_qty, ds.std_daily_qty, ds.avg_daily_qty, ds.total_days, ds.daily_variance_ratio
ON CONFLICT (pid, forecast_date) DO UPDATE
SET
forecast_quantity = EXCLUDED.forecast_quantity,
confidence_level = EXCLUDED.confidence_level,
created_at = NOW()
`);
processedCount = Math.floor(totalProducts * 0.98);
outputProgress({
status: 'running',
operation: 'Product forecasts calculated, preparing category stats',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Create temporary table for category stats
await connection.query(`
CREATE TEMPORARY TABLE temp_category_sales AS
SELECT
pc.cat_id,
EXTRACT(DOW FROM o.date) + 1 as day_of_week,
SUM(o.quantity) as daily_quantity,
SUM(o.price * o.quantity) as daily_revenue,
COUNT(DISTINCT DATE(o.date)) as day_count
FROM orders o
JOIN product_categories pc ON o.pid = pc.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '90 days'
GROUP BY pc.cat_id, EXTRACT(DOW FROM o.date) + 1
`);
await connection.query(`
CREATE TEMPORARY TABLE temp_category_stats AS
SELECT
cat_id,
AVG(daily_revenue) as overall_avg_revenue,
SUM(day_count) as total_days
FROM temp_category_sales
GROUP BY cat_id
`);
processedCount = Math.floor(totalProducts * 0.99);
outputProgress({
status: 'running',
operation: 'Category stats prepared, calculating category-level forecasts',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Calculate category-level forecasts
await connection.query(`
INSERT INTO category_forecasts (
category_id,
forecast_date,
forecast_units,
forecast_revenue,
confidence_level,
created_at
)
SELECT
cs.cat_id::bigint as category_id,
fd.forecast_date,
GREATEST(0,
ROUND(AVG(cs.daily_quantity) *
(1 + COALESCE(sf.seasonality_factor, 0)))
) as forecast_units,
GREATEST(0,
COALESCE(
CASE
WHEN SUM(cs.day_count) >= 4 THEN AVG(cs.daily_revenue)
ELSE ct.overall_avg_revenue
END *
(1 + COALESCE(sf.seasonality_factor, 0)),
0
)
) as forecast_revenue,
CASE
WHEN ct.total_days >= 60 THEN 90
WHEN ct.total_days >= 30 THEN 80
WHEN ct.total_days >= 14 THEN 70
ELSE 60
END as confidence_level,
NOW() as created_at
FROM temp_category_sales cs
JOIN temp_category_stats ct ON cs.cat_id = ct.cat_id
CROSS JOIN temp_forecast_dates fd
LEFT JOIN sales_seasonality sf ON fd.month = sf.month
GROUP BY
cs.cat_id,
fd.forecast_date,
ct.overall_avg_revenue,
ct.total_days,
sf.seasonality_factor,
sf.month
HAVING AVG(cs.daily_quantity) > 0
ON CONFLICT (category_id, forecast_date) DO UPDATE
SET
forecast_units = EXCLUDED.forecast_units,
forecast_revenue = EXCLUDED.forecast_revenue,
confidence_level = EXCLUDED.confidence_level,
created_at = NOW()
`);
// Clean up temporary tables
await connection.query(`
DROP TABLE IF EXISTS temp_forecast_dates;
DROP TABLE IF EXISTS temp_daily_sales;
DROP TABLE IF EXISTS temp_product_stats;
DROP TABLE IF EXISTS temp_category_sales;
DROP TABLE IF EXISTS temp_category_stats;
`);
processedCount = Math.floor(totalProducts * 1.0);
outputProgress({
status: 'running',
operation: 'Category forecasts calculated and temporary tables cleaned up',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('sales_forecasts', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating sales forecasts');
throw error;
} finally {
if (connection) {
try {
// Ensure temporary tables are cleaned up
await connection.query(`
DROP TABLE IF EXISTS temp_forecast_dates;
DROP TABLE IF EXISTS temp_daily_sales;
DROP TABLE IF EXISTS temp_product_stats;
DROP TABLE IF EXISTS temp_category_sales;
DROP TABLE IF EXISTS temp_category_stats;
`);
} catch (err) {
console.error('Error cleaning up temporary tables:', err);
}
connection.release();
}
}
}
module.exports = calculateSalesForecasts;
inventory-server/old/metrics/time-aggregates.js
-344
View File
@@ -1,344 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateTimeAggregates(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Time aggregates calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders: 0,
processedPurchaseOrders: 0,
success
};
}
// Get order count that will be processed
const orderCount = await connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
`);
processedOrders = parseInt(orderCount.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting time aggregates calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Create a temporary table for end-of-month inventory values
await connection.query(`
CREATE TEMPORARY TABLE IF NOT EXISTS temp_monthly_inventory AS
WITH months AS (
-- Generate all year/month combinations for the last 12 months
SELECT
EXTRACT(YEAR FROM month_date)::INTEGER as year,
EXTRACT(MONTH FROM month_date)::INTEGER as month,
month_date as start_date,
(month_date + INTERVAL '1 month'::interval - INTERVAL '1 day'::interval)::DATE as end_date
FROM (
SELECT generate_series(
DATE_TRUNC('month', CURRENT_DATE - INTERVAL '12 months'::interval)::DATE,
DATE_TRUNC('month', CURRENT_DATE)::DATE,
INTERVAL '1 month'::interval
) as month_date
) dates
),
monthly_inventory_calc AS (
SELECT
p.pid,
m.year,
m.month,
m.end_date,
p.stock_quantity as current_quantity,
-- Calculate sold during period (before end_date)
COALESCE(SUM(
CASE
WHEN o.date <= m.end_date THEN o.quantity
ELSE 0
END
), 0) as sold_after_end_date,
-- Calculate received during period (before end_date)
COALESCE(SUM(
CASE
WHEN po.received_date <= m.end_date THEN po.received
ELSE 0
END
), 0) as received_after_end_date,
p.cost_price
FROM
products p
CROSS JOIN
months m
LEFT JOIN
orders o ON p.pid = o.pid
AND o.canceled = false
AND o.date > m.end_date
AND o.date <= CURRENT_DATE
LEFT JOIN
purchase_orders po ON p.pid = po.pid
AND po.received_date IS NOT NULL
AND po.received_date > m.end_date
AND po.received_date <= CURRENT_DATE
GROUP BY
p.pid, m.year, m.month, m.end_date, p.stock_quantity, p.cost_price
)
SELECT
pid,
year,
month,
-- End of month quantity = current quantity - sold after + received after
GREATEST(0, current_quantity - sold_after_end_date + received_after_end_date) as end_of_month_quantity,
-- End of month inventory value
GREATEST(0, current_quantity - sold_after_end_date + received_after_end_date) * cost_price as end_of_month_value,
cost_price
FROM
monthly_inventory_calc
`);
processedCount = Math.floor(totalProducts * 0.40);
outputProgress({
status: 'running',
operation: 'Monthly inventory values calculated, processing time aggregates',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// Initial insert of time-based aggregates
await connection.query(`
INSERT INTO product_time_aggregates (
pid,
year,
month,
total_quantity_sold,
total_revenue,
total_cost,
order_count,
stock_received,
stock_ordered,
avg_price,
profit_margin,
inventory_value,
gmroi
)
WITH monthly_sales AS (
SELECT
o.pid,
EXTRACT(YEAR FROM o.date::timestamp with time zone)::INTEGER as year,
EXTRACT(MONTH FROM o.date::timestamp with time zone)::INTEGER as month,
SUM(o.quantity) as total_quantity_sold,
SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) as total_revenue,
SUM(COALESCE(o.costeach, 0) * o.quantity) as total_cost,
COUNT(DISTINCT o.order_number) as order_count,
AVG(o.price - COALESCE(o.discount, 0)) as avg_price,
CASE
WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) > 0
THEN ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - SUM(COALESCE(o.costeach, 0) * o.quantity))
/ SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
ELSE 0
END as profit_margin,
COUNT(DISTINCT DATE(o.date)) as active_days
FROM orders o
JOIN products p ON o.pid = p.pid
WHERE o.canceled = false
GROUP BY o.pid, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
),
monthly_stock AS (
SELECT
pid,
EXTRACT(YEAR FROM date::timestamp with time zone)::INTEGER as year,
EXTRACT(MONTH FROM date::timestamp with time zone)::INTEGER as month,
SUM(received) as stock_received,
SUM(ordered) as stock_ordered
FROM purchase_orders
GROUP BY pid, EXTRACT(YEAR FROM date::timestamp with time zone), EXTRACT(MONTH FROM date::timestamp with time zone)
)
SELECT
COALESCE(s.pid, ms.pid, mi.pid) as pid,
COALESCE(s.year, ms.year, mi.year) as year,
COALESCE(s.month, ms.month, mi.month) as month,
COALESCE(s.total_quantity_sold, 0)::INTEGER as total_quantity_sold,
COALESCE(s.total_revenue, 0)::DECIMAL(10,3) as total_revenue,
COALESCE(s.total_cost, 0)::DECIMAL(10,3) as total_cost,
COALESCE(s.order_count, 0)::INTEGER as order_count,
COALESCE(ms.stock_received, 0)::INTEGER as stock_received,
COALESCE(ms.stock_ordered, 0)::INTEGER as stock_ordered,
COALESCE(s.avg_price, 0)::DECIMAL(10,3) as avg_price,
COALESCE(s.profit_margin, 0)::DECIMAL(10,3) as profit_margin,
COALESCE(mi.end_of_month_value, 0)::DECIMAL(10,3) as inventory_value,
CASE
WHEN COALESCE(mi.end_of_month_value, 0) > 0
THEN (COALESCE(s.total_revenue, 0) - COALESCE(s.total_cost, 0))
/ NULLIF(COALESCE(mi.end_of_month_value, 0), 0)
ELSE 0
END::DECIMAL(10,3) as gmroi
FROM (
SELECT * FROM monthly_sales s
UNION ALL
SELECT
pid,
year,
month,
0 as total_quantity_sold,
0 as total_revenue,
0 as total_cost,
0 as order_count,
NULL as avg_price,
0 as profit_margin,
0 as active_days
FROM monthly_stock ms
WHERE NOT EXISTS (
SELECT 1 FROM monthly_sales s2
WHERE s2.pid = ms.pid
AND s2.year = ms.year
AND s2.month = ms.month
)
UNION ALL
SELECT
pid,
year,
month,
0 as total_quantity_sold,
0 as total_revenue,
0 as total_cost,
0 as order_count,
NULL as avg_price,
0 as profit_margin,
0 as active_days
FROM temp_monthly_inventory mi
WHERE NOT EXISTS (
SELECT 1 FROM monthly_sales s3
WHERE s3.pid = mi.pid
AND s3.year = mi.year
AND s3.month = mi.month
)
AND NOT EXISTS (
SELECT 1 FROM monthly_stock ms3
WHERE ms3.pid = mi.pid
AND ms3.year = mi.year
AND ms3.month = mi.month
)
) s
LEFT JOIN monthly_stock ms
ON s.pid = ms.pid
AND s.year = ms.year
AND s.month = ms.month
LEFT JOIN temp_monthly_inventory mi
ON s.pid = mi.pid
AND s.year = mi.year
AND s.month = mi.month
ON CONFLICT (pid, year, month) DO UPDATE
SET
total_quantity_sold = EXCLUDED.total_quantity_sold,
total_revenue = EXCLUDED.total_revenue,
total_cost = EXCLUDED.total_cost,
order_count = EXCLUDED.order_count,
stock_received = EXCLUDED.stock_received,
stock_ordered = EXCLUDED.stock_ordered,
avg_price = EXCLUDED.avg_price,
profit_margin = EXCLUDED.profit_margin,
inventory_value = EXCLUDED.inventory_value,
gmroi = EXCLUDED.gmroi
`);
processedCount = Math.floor(totalProducts * 0.60);
outputProgress({
status: 'running',
operation: 'Base time aggregates calculated',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
// Clean up temporary tables
await connection.query('DROP TABLE IF EXISTS temp_monthly_inventory');
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('time_aggregates', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders: 0,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating time aggregates');
throw error;
} finally {
if (connection) {
try {
// Ensure temporary tables are cleaned up
await connection.query('DROP TABLE IF EXISTS temp_monthly_inventory');
} catch (err) {
console.error('Error cleaning up temporary tables:', err);
}
connection.release();
}
}
}
module.exports = calculateTimeAggregates;
inventory-server/old/metrics/utils/db.js
-39
View File
@@ -1,39 +0,0 @@
const { Pool } = require('pg');
const path = require('path');
require('dotenv').config({ path: path.resolve(__dirname, '../../..', '.env') });
// Database configuration
const dbConfig = {
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: process.env.DB_PORT || 5432,
ssl: process.env.DB_SSL === 'true',
// Add performance optimizations
max: 10, // connection pool max size
idleTimeoutMillis: 30000,
connectionTimeoutMillis: 60000
};
// Create a single pool instance to be reused
const pool = new Pool(dbConfig);
// Add event handlers for pool
pool.on('error', (err, client) => {
console.error('Unexpected error on idle client', err);
});
async function getConnection() {
return await pool.connect();
}
async function closePool() {
await pool.end();
}
module.exports = {
dbConfig,
getConnection,
closePool
};
inventory-server/old/metrics/utils/progress.js
-158
View File
@@ -1,158 +0,0 @@
const fs = require('fs');
const path = require('path');
// Helper function to format elapsed time
function formatElapsedTime(elapsed) {
// If elapsed is a timestamp, convert to elapsed milliseconds
if (elapsed instanceof Date || elapsed > 1000000000000) {
elapsed = Date.now() - elapsed;
} else {
// If elapsed is in seconds, convert to milliseconds
elapsed = elapsed * 1000;
}
const seconds = Math.floor(elapsed / 1000);
const minutes = Math.floor(seconds / 60);
const hours = Math.floor(minutes / 60);
if (hours > 0) {
return `${hours}h ${minutes % 60}m`;
} else if (minutes > 0) {
return `${minutes}m ${seconds % 60}s`;
} else {
return `${seconds}s`;
}
}
// Helper function to estimate remaining time
function estimateRemaining(startTime, current, total) {
if (current === 0) return null;
const elapsed = Date.now() - startTime;
const rate = current / elapsed;
const remaining = (total - current) / rate;
const minutes = Math.floor(remaining / 60000);
const seconds = Math.floor((remaining % 60000) / 1000);
if (minutes > 0) {
return `${minutes}m ${seconds}s`;
} else {
return `${seconds}s`;
}
}
// Helper function to calculate rate
function calculateRate(startTime, current) {
const elapsed = (Date.now() - startTime) / 1000; // Convert to seconds
return elapsed > 0 ? Math.round(current / elapsed) : 0;
}
// Set up logging
const LOG_DIR = path.join(__dirname, '../../../logs');
const ERROR_LOG = path.join(LOG_DIR, 'import-errors.log');
const IMPORT_LOG = path.join(LOG_DIR, 'import.log');
const STATUS_FILE = path.join(LOG_DIR, 'metrics-status.json');
// Ensure log directory exists
if (!fs.existsSync(LOG_DIR)) {
fs.mkdirSync(LOG_DIR, { recursive: true });
}
// Helper function to log errors
function logError(error, context = '') {
const timestamp = new Date().toISOString();
const errorMessage = `[${timestamp}] ${context}\nError: ${error.message}\nStack: ${error.stack}\n\n`;
// Log to error file
fs.appendFileSync(ERROR_LOG, errorMessage);
// Also log to console
console.error(`\n${context}\nError: ${error.message}`);
}
// Helper function to log import progress
function logImport(message) {
const timestamp = new Date().toISOString();
const logMessage = `[${timestamp}] ${message}\n`;
fs.appendFileSync(IMPORT_LOG, logMessage);
}
// Helper function to output progress
function outputProgress(data) {
// Save progress to file for resumption
saveProgress(data);
// Format as SSE event
const event = {
progress: data
};
// Always send to stdout for frontend
process.stdout.write(JSON.stringify(event) + '\n');
// Log significant events to disk
const isSignificant =
// Operation starts
(data.operation && !data.current) ||
// Operation completions and errors
data.status === 'complete' ||
data.status === 'error' ||
// Major phase changes
data.operation?.includes('Starting ABC classification') ||
data.operation?.includes('Starting time-based aggregates') ||
data.operation?.includes('Starting vendor metrics');
if (isSignificant) {
logImport(`${data.operation || 'Operation'}${data.message ? ': ' + data.message : ''}${data.error ? ' Error: ' + data.error : ''}${data.status ? ' Status: ' + data.status : ''}`);
}
}
function saveProgress(progress) {
try {
fs.writeFileSync(STATUS_FILE, JSON.stringify({
...progress,
timestamp: Date.now()
}));
} catch (err) {
console.error('Failed to save progress:', err);
}
}
function clearProgress() {
try {
if (fs.existsSync(STATUS_FILE)) {
fs.unlinkSync(STATUS_FILE);
}
} catch (err) {
console.error('Failed to clear progress:', err);
}
}
function getProgress() {
try {
if (fs.existsSync(STATUS_FILE)) {
const progress = JSON.parse(fs.readFileSync(STATUS_FILE, 'utf8'));
// Check if the progress is still valid (less than 1 hour old)
if (progress.timestamp && Date.now() - progress.timestamp < 3600000) {
return progress;
} else {
// Clear old progress
clearProgress();
}
}
} catch (err) {
console.error('Failed to read progress:', err);
clearProgress();
}
return null;
}
module.exports = {
formatElapsedTime,
estimateRemaining,
calculateRate,
logError,
logImport,
outputProgress,
saveProgress,
clearProgress,
getProgress
};
inventory-server/old/metrics/vendor-metrics.js
-378
View File
@@ -1,378 +0,0 @@
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
const { getConnection } = require('./utils/db');
async function calculateVendorMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
const connection = await getConnection();
let success = false;
let processedOrders = 0;
let processedPurchaseOrders = 0;
try {
if (isCancelled) {
outputProgress({
status: 'cancelled',
operation: 'Vendor metrics calculation cancelled',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders,
success
};
}
// Get counts of records that will be processed
const [orderCountResult, poCountResult] = await Promise.all([
connection.query(`
SELECT COUNT(*) as count
FROM orders o
WHERE o.canceled = false
`),
connection.query(`
SELECT COUNT(*) as count
FROM purchase_orders po
WHERE po.status != 0
`)
]);
processedOrders = parseInt(orderCountResult.rows[0].count);
processedPurchaseOrders = parseInt(poCountResult.rows[0].count);
outputProgress({
status: 'running',
operation: 'Starting vendor metrics calculation',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// First ensure all vendors exist in vendor_details
await connection.query(`
INSERT INTO vendor_details (vendor, status, created_at, updated_at)
SELECT DISTINCT
vendor,
'active' as status,
NOW() as created_at,
NOW() as updated_at
FROM products
WHERE vendor IS NOT NULL
ON CONFLICT (vendor) DO NOTHING
`);
processedCount = Math.floor(totalProducts * 0.8);
outputProgress({
status: 'running',
operation: 'Vendor details updated, calculating metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders,
success
};
// Now calculate vendor metrics
await connection.query(`
INSERT INTO vendor_metrics (
vendor,
total_revenue,
total_orders,
total_late_orders,
avg_lead_time_days,
on_time_delivery_rate,
order_fill_rate,
avg_order_value,
active_products,
total_products,
total_purchase_value,
avg_margin_percent,
status,
last_calculated_at
)
WITH vendor_sales AS (
SELECT
p.vendor,
SUM(o.quantity * o.price) as total_revenue,
COUNT(DISTINCT o.id) as total_orders,
COUNT(DISTINCT p.pid) as active_products,
SUM(o.quantity * (o.price - p.cost_price)) as total_margin
FROM products p
JOIN orders o ON p.pid = o.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '12 months'
GROUP BY p.vendor
),
vendor_po AS (
SELECT
p.vendor,
COUNT(DISTINCT CASE WHEN po.receiving_status = 40 THEN po.id END) as received_orders,
COUNT(DISTINCT po.id) as total_orders,
AVG(CASE
WHEN po.receiving_status = 40
AND po.received_date IS NOT NULL
AND po.date IS NOT NULL
THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
ELSE NULL
END) as avg_lead_time_days,
SUM(po.ordered * po.po_cost_price) as total_purchase_value
FROM products p
JOIN purchase_orders po ON p.pid = po.pid
WHERE po.date >= CURRENT_DATE - INTERVAL '12 months'
GROUP BY p.vendor
),
vendor_products AS (
SELECT
vendor,
COUNT(DISTINCT pid) as total_products
FROM products
GROUP BY vendor
)
SELECT
vs.vendor,
COALESCE(vs.total_revenue, 0) as total_revenue,
COALESCE(vp.total_orders, 0) as total_orders,
COALESCE(vp.total_orders - vp.received_orders, 0) as total_late_orders,
COALESCE(vp.avg_lead_time_days, 0) as avg_lead_time_days,
CASE
WHEN vp.total_orders > 0
THEN (vp.received_orders / vp.total_orders) * 100
ELSE 0
END as on_time_delivery_rate,
CASE
WHEN vp.total_orders > 0
THEN (vp.received_orders / vp.total_orders) * 100
ELSE 0
END as order_fill_rate,
CASE
WHEN vs.total_orders > 0
THEN vs.total_revenue / vs.total_orders
ELSE 0
END as avg_order_value,
COALESCE(vs.active_products, 0) as active_products,
COALESCE(vpr.total_products, 0) as total_products,
COALESCE(vp.total_purchase_value, 0) as total_purchase_value,
CASE
WHEN vs.total_revenue > 0
THEN (vs.total_margin / vs.total_revenue) * 100
ELSE 0
END as avg_margin_percent,
'active' as status,
NOW() as last_calculated_at
FROM vendor_sales vs
LEFT JOIN vendor_po vp ON vs.vendor = vp.vendor
LEFT JOIN vendor_products vpr ON vs.vendor = vpr.vendor
WHERE vs.vendor IS NOT NULL
ON CONFLICT (vendor) DO UPDATE
SET
total_revenue = EXCLUDED.total_revenue,
total_orders = EXCLUDED.total_orders,
total_late_orders = EXCLUDED.total_late_orders,
avg_lead_time_days = EXCLUDED.avg_lead_time_days,
on_time_delivery_rate = EXCLUDED.on_time_delivery_rate,
order_fill_rate = EXCLUDED.order_fill_rate,
avg_order_value = EXCLUDED.avg_order_value,
active_products = EXCLUDED.active_products,
total_products = EXCLUDED.total_products,
total_purchase_value = EXCLUDED.total_purchase_value,
avg_margin_percent = EXCLUDED.avg_margin_percent,
status = EXCLUDED.status,
last_calculated_at = EXCLUDED.last_calculated_at
`);
processedCount = Math.floor(totalProducts * 0.9);
outputProgress({
status: 'running',
operation: 'Vendor metrics calculated, updating time-based metrics',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
if (isCancelled) return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders,
success
};
// Calculate time-based metrics
await connection.query(`
INSERT INTO vendor_time_metrics (
vendor,
year,
month,
total_orders,
late_orders,
avg_lead_time_days,
total_purchase_value,
total_revenue,
avg_margin_percent
)
WITH monthly_orders AS (
SELECT
p.vendor,
EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
COUNT(DISTINCT o.id) as total_orders,
SUM(o.quantity * o.price) as total_revenue,
SUM(o.quantity * (o.price - p.cost_price)) as total_margin
FROM products p
JOIN orders o ON p.pid = o.pid
WHERE o.canceled = false
AND o.date >= CURRENT_DATE - INTERVAL '12 months'
AND p.vendor IS NOT NULL
GROUP BY p.vendor, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
),
monthly_po AS (
SELECT
p.vendor,
EXTRACT(YEAR FROM po.date::timestamp with time zone) as year,
EXTRACT(MONTH FROM po.date::timestamp with time zone) as month,
COUNT(DISTINCT po.id) as total_po,
COUNT(DISTINCT CASE
WHEN po.receiving_status = 40 AND po.received_date > po.expected_date
THEN po.id
END) as late_orders,
AVG(CASE
WHEN po.receiving_status = 40
AND po.received_date IS NOT NULL
AND po.date IS NOT NULL
THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
ELSE NULL
END) as avg_lead_time_days,
SUM(po.ordered * po.po_cost_price) as total_purchase_value
FROM products p
JOIN purchase_orders po ON p.pid = po.pid
WHERE po.date >= CURRENT_DATE - INTERVAL '12 months'
AND p.vendor IS NOT NULL
GROUP BY p.vendor, EXTRACT(YEAR FROM po.date::timestamp with time zone), EXTRACT(MONTH FROM po.date::timestamp with time zone)
)
SELECT
mo.vendor,
mo.year,
mo.month,
COALESCE(mp.total_po, 0) as total_orders,
COALESCE(mp.late_orders, 0) as late_orders,
COALESCE(mp.avg_lead_time_days, 0) as avg_lead_time_days,
COALESCE(mp.total_purchase_value, 0) as total_purchase_value,
mo.total_revenue,
CASE
WHEN mo.total_revenue > 0
THEN (mo.total_margin / mo.total_revenue) * 100
ELSE 0
END as avg_margin_percent
FROM monthly_orders mo
LEFT JOIN monthly_po mp ON mo.vendor = mp.vendor
AND mo.year = mp.year
AND mo.month = mp.month
UNION
SELECT
mp.vendor,
mp.year,
mp.month,
mp.total_po as total_orders,
mp.late_orders,
mp.avg_lead_time_days,
mp.total_purchase_value,
0 as total_revenue,
0 as avg_margin_percent
FROM monthly_po mp
LEFT JOIN monthly_orders mo ON mp.vendor = mo.vendor
AND mp.year = mo.year
AND mp.month = mo.month
WHERE mo.vendor IS NULL
ON CONFLICT (vendor, year, month) DO UPDATE
SET
total_orders = EXCLUDED.total_orders,
late_orders = EXCLUDED.late_orders,
avg_lead_time_days = EXCLUDED.avg_lead_time_days,
total_purchase_value = EXCLUDED.total_purchase_value,
total_revenue = EXCLUDED.total_revenue,
avg_margin_percent = EXCLUDED.avg_margin_percent
`);
processedCount = Math.floor(totalProducts * 0.95);
outputProgress({
status: 'running',
operation: 'Time-based vendor metrics calculated',
current: processedCount,
total: totalProducts,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, processedCount, totalProducts),
rate: calculateRate(startTime, processedCount),
percentage: ((processedCount / totalProducts) * 100).toFixed(1),
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
}
});
// If we get here, everything completed successfully
success = true;
// Update calculate_status
await connection.query(`
INSERT INTO calculate_status (module_name, last_calculation_timestamp)
VALUES ('vendor_metrics', NOW())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = NOW()
`);
return {
processedProducts: processedCount,
processedOrders,
processedPurchaseOrders,
success
};
} catch (error) {
success = false;
logError(error, 'Error calculating vendor metrics');
throw error;
} finally {
if (connection) {
connection.release();
}
}
}
module.exports = calculateVendorMetrics;
inventory-server/old/old_csv/import-csv.js
-1049
View File
File diff suppressed because it is too large Load Diff
inventory-server/old/old_csv/update-csv.js
-180
View File
@@ -1,180 +0,0 @@
const path = require('path');
const fs = require('fs');
const axios = require('axios');
const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../metrics/utils/progress');
// Change working directory to script directory
process.chdir(path.dirname(__filename));
require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
const FILES = [
{
name: '39f2x83-products.csv',
url: process.env.PRODUCTS_CSV_URL
},
{
name: '39f2x83-orders.csv',
url: process.env.ORDERS_CSV_URL
},
{
name: '39f2x83-purchase_orders.csv',
url: process.env.PURCHASE_ORDERS_CSV_URL
}
];
let isCancelled = false;
function cancelUpdate() {
isCancelled = true;
outputProgress({
status: 'cancelled',
operation: 'CSV update cancelled',
current: 0,
total: FILES.length,
elapsed: null,
remaining: null,
rate: 0
});
}
async function downloadFile(file, index, startTime) {
if (isCancelled) return;
const csvDir = path.join(__dirname, '../csv');
if (!fs.existsSync(csvDir)) {
fs.mkdirSync(csvDir, { recursive: true });
}
const writer = fs.createWriteStream(path.join(csvDir, file.name));
try {
const response = await axios({
url: file.url,
method: 'GET',
responseType: 'stream'
});
const totalLength = response.headers['content-length'];
let downloadedLength = 0;
let lastProgressUpdate = Date.now();
const PROGRESS_INTERVAL = 1000; // Update progress every second
response.data.on('data', (chunk) => {
if (isCancelled) {
writer.end();
return;
}
downloadedLength += chunk.length;
// Update progress based on time interval
const now = Date.now();
if (now - lastProgressUpdate >= PROGRESS_INTERVAL) {
const progress = (downloadedLength / totalLength) * 100;
outputProgress({
status: 'running',
operation: `Downloading ${file.name}`,
current: index + (downloadedLength / totalLength),
total: FILES.length,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, index + (downloadedLength / totalLength), FILES.length),
rate: calculateRate(startTime, index + (downloadedLength / totalLength)),
percentage: progress.toFixed(1),
file_progress: {
name: file.name,
downloaded: downloadedLength,
total: totalLength,
percentage: progress.toFixed(1)
}
});
lastProgressUpdate = now;
}
});
response.data.pipe(writer);
return new Promise((resolve, reject) => {
writer.on('finish', resolve);
writer.on('error', reject);
});
} catch (error) {
fs.unlinkSync(path.join(csvDir, file.name));
throw error;
}
}
// Main function to update all files
async function updateFiles() {
const startTime = Date.now();
outputProgress({
status: 'running',
operation: 'Starting CSV update',
current: 0,
total: FILES.length,
elapsed: '0s',
remaining: null,
rate: 0,
percentage: '0'
});
try {
for (let i = 0; i < FILES.length; i++) {
if (isCancelled) {
return;
}
const file = FILES[i];
await downloadFile(file, i, startTime);
outputProgress({
status: 'running',
operation: 'CSV update in progress',
current: i + 1,
total: FILES.length,
elapsed: formatElapsedTime(startTime),
remaining: estimateRemaining(startTime, i + 1, FILES.length),
rate: calculateRate(startTime, i + 1),
percentage: (((i + 1) / FILES.length) * 100).toFixed(1)
});
}
outputProgress({
status: 'complete',
operation: 'CSV update complete',
current: FILES.length,
total: FILES.length,
elapsed: formatElapsedTime(startTime),
remaining: '0s',
rate: calculateRate(startTime, FILES.length),
percentage: '100'
});
} catch (error) {
outputProgress({
status: 'error',
operation: 'CSV update failed',
error: error.message,
current: 0,
total: FILES.length,
elapsed: formatElapsedTime(startTime),
remaining: null,
rate: 0
});
throw error;
}
}
// Run the update only if this is the main module
if (require.main === module) {
updateFiles().catch((error) => {
console.error('Error updating CSV files:', error);
process.exit(1);
});
}
// Export the functions needed by the route
module.exports = {
updateFiles,
cancelUpdate
};
inventory-server/old/populate-initial-metrics.js
-677
View File
@@ -1,677 +0,0 @@
const path = require('path');
const fs = require('fs');
const os = require('os'); // For detecting CPU cores
// Get the base directory (the directory containing the inventory-server folder)
const baseDir = path.resolve(__dirname, '../../..');
// Load environment variables from the inventory-server directory
require('dotenv').config({ path: path.resolve(__dirname, '../..', '.env') });
// Configure statement timeout (30 minutes)
const PG_STATEMENT_TIMEOUT_MS = 1800000;
// Add error handler for uncaught exceptions
process.on('uncaughtException', (error) => {
console.error('Uncaught Exception:', error);
process.exit(1);
});
// Add error handler for unhandled promise rejections
process.on('unhandledRejection', (reason, promise) => {
console.error('Unhandled Rejection at:', promise, 'reason:', reason);
process.exit(1);
});
// Load progress module
const progress = require('../scripts/metrics-new/utils/progress');
// Store progress functions in global scope to ensure availability
global.formatElapsedTime = progress.formatElapsedTime;
global.estimateRemaining = progress.estimateRemaining;
global.calculateRate = progress.calculateRate;
global.outputProgress = progress.outputProgress;
global.clearProgress = progress.clearProgress;
global.getProgress = progress.getProgress;
global.logError = progress.logError;
// Load database module
const { getConnection, closePool } = require('../scripts/metrics-new/utils/db');
// Add cancel handler
let isCancelled = false;
let runningQueryPromise = null;
function cancelCalculation() {
if (!isCancelled) {
isCancelled = true;
console.log('Calculation has been cancelled by user');
// Store the query promise to potentially cancel it
const queryToCancel = runningQueryPromise;
if (queryToCancel) {
console.log('Attempting to cancel the running query...');
}
// Force-terminate any query that's been running for more than 5 seconds
try {
const connection = getConnection();
connection.then(async (conn) => {
try {
// Identify and terminate long-running queries from our application
await conn.query(`
SELECT pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE query_start < now() - interval '5 seconds'
AND application_name = 'populate_metrics'
AND query NOT LIKE '%pg_cancel_backend%'
`);
// Release connection
conn.release();
} catch (err) {
console.error('Error during force cancellation:', err);
conn.release();
}
}).catch(err => {
console.error('Could not get connection for cancellation:', err);
});
} catch (err) {
console.error('Failed to terminate running queries:', err);
}
}
return {
success: true,
message: 'Calculation has been cancelled'
};
}
// Handle SIGTERM signal for cancellation
process.on('SIGTERM', cancelCalculation);
process.on('SIGINT', cancelCalculation);
const calculateInitialMetrics = (client, onProgress) => {
return client.query(`
-- Truncate the existing metrics tables to ensure clean data
TRUNCATE TABLE public.daily_product_snapshots;
TRUNCATE TABLE public.product_metrics;
-- First let's create daily snapshots for all products with order activity
WITH SalesData AS (
SELECT
p.pid,
p.sku,
o.date::date AS order_date,
-- Count orders to ensure we only include products with real activity
COUNT(o.id) as order_count,
-- Aggregate Sales (Quantity > 0, Status not Canceled/Returned)
COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.quantity ELSE 0 END), 0) AS units_sold,
COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.price * o.quantity ELSE 0 END), 0.00) AS gross_revenue_unadjusted,
COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.discount ELSE 0 END), 0.00) AS discounts,
COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN COALESCE(o.costeach, p.landing_cost_price, p.cost_price) * o.quantity ELSE 0 END), 0.00) AS cogs,
COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN p.regular_price * o.quantity ELSE 0 END), 0.00) AS gross_regular_revenue,
-- Aggregate Returns (Quantity < 0 or Status = Returned)
COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN ABS(o.quantity) ELSE 0 END), 0) AS units_returned,
COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN o.price * ABS(o.quantity) ELSE 0 END), 0.00) AS returns_revenue
FROM public.products p
LEFT JOIN public.orders o ON p.pid = o.pid
GROUP BY p.pid, p.sku, o.date::date
HAVING COUNT(o.id) > 0 -- Only include products with actual orders
),
ReceivingData AS (
SELECT
r.pid,
r.received_date::date AS receiving_date,
-- Count receiving documents to ensure we only include products with real activity
COUNT(DISTINCT r.receiving_id) as receiving_count,
-- Calculate received quantity for this day
SUM(r.received_quantity) AS units_received,
-- Calculate received cost for this day
SUM(r.received_quantity * r.unit_cost) AS cost_received
FROM public.receivings r
GROUP BY r.pid, r.received_date::date
HAVING COUNT(DISTINCT r.receiving_id) > 0 OR SUM(r.received_quantity) > 0
),
-- Get current stock quantities
StockData AS (
SELECT
p.pid,
p.stock_quantity,
COALESCE(p.landing_cost_price, p.cost_price, 0.00) as effective_cost_price,
COALESCE(p.price, 0.00) as current_price,
COALESCE(p.regular_price, 0.00) as current_regular_price
FROM public.products p
),
-- Combine sales and receiving dates to get all activity dates
DatePidCombos AS (
SELECT DISTINCT pid, order_date AS activity_date FROM SalesData
UNION
SELECT DISTINCT pid, receiving_date FROM ReceivingData
),
-- Insert daily snapshots for all product-date combinations
SnapshotInsert AS (
INSERT INTO public.daily_product_snapshots (
snapshot_date,
pid,
sku,
eod_stock_quantity,
eod_stock_cost,
eod_stock_retail,
eod_stock_gross,
stockout_flag,
units_sold,
units_returned,
gross_revenue,
discounts,
returns_revenue,
net_revenue,
cogs,
gross_regular_revenue,
profit,
units_received,
cost_received,
calculation_timestamp
)
SELECT
d.activity_date AS snapshot_date,
d.pid,
p.sku,
-- Use current stock as approximation, since historical stock data is not available
s.stock_quantity AS eod_stock_quantity,
s.stock_quantity * s.effective_cost_price AS eod_stock_cost,
s.stock_quantity * s.current_price AS eod_stock_retail,
s.stock_quantity * s.current_regular_price AS eod_stock_gross,
(s.stock_quantity <= 0) AS stockout_flag,
-- Sales metrics
COALESCE(sd.units_sold, 0),
COALESCE(sd.units_returned, 0),
COALESCE(sd.gross_revenue_unadjusted, 0.00),
COALESCE(sd.discounts, 0.00),
COALESCE(sd.returns_revenue, 0.00),
COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00) AS net_revenue,
COALESCE(sd.cogs, 0.00),
COALESCE(sd.gross_regular_revenue, 0.00),
(COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00)) - COALESCE(sd.cogs, 0.00) AS profit,
-- Receiving metrics
COALESCE(rd.units_received, 0),
COALESCE(rd.cost_received, 0.00),
now() -- calculation timestamp
FROM DatePidCombos d
JOIN public.products p ON d.pid = p.pid
LEFT JOIN SalesData sd ON d.pid = sd.pid AND d.activity_date = sd.order_date
LEFT JOIN ReceivingData rd ON d.pid = rd.pid AND d.activity_date = rd.receiving_date
LEFT JOIN StockData s ON d.pid = s.pid
RETURNING pid, snapshot_date
),
-- Now build the aggregated product metrics from the daily snapshots
MetricsInsert AS (
INSERT INTO public.product_metrics (
pid,
sku,
current_stock_quantity,
current_stock_cost,
current_stock_retail,
current_stock_msrp,
is_out_of_stock,
total_units_sold,
total_units_returned,
return_rate,
gross_revenue,
total_discounts,
total_returns,
net_revenue,
total_cogs,
total_gross_revenue,
total_profit,
profit_margin,
avg_daily_units,
reorder_point,
reorder_alert,
days_of_supply,
sales_velocity,
sales_velocity_score,
rank_by_revenue,
rank_by_quantity,
rank_by_profit,
total_received_quantity,
total_received_cost,
last_sold_date,
last_received_date,
days_since_last_sale,
days_since_last_received,
calculation_timestamp
)
SELECT
p.pid,
p.sku,
p.stock_quantity AS current_stock_quantity,
p.stock_quantity * COALESCE(p.landing_cost_price, p.cost_price, 0) AS current_stock_cost,
p.stock_quantity * COALESCE(p.price, 0) AS current_stock_retail,
p.stock_quantity * COALESCE(p.regular_price, 0) AS current_stock_msrp,
(p.stock_quantity <= 0) AS is_out_of_stock,
-- Aggregate metrics
COALESCE(SUM(ds.units_sold), 0) AS total_units_sold,
COALESCE(SUM(ds.units_returned), 0) AS total_units_returned,
CASE
WHEN COALESCE(SUM(ds.units_sold), 0) > 0
THEN COALESCE(SUM(ds.units_returned), 0)::float / NULLIF(COALESCE(SUM(ds.units_sold), 0), 0)
ELSE 0
END AS return_rate,
COALESCE(SUM(ds.gross_revenue), 0) AS gross_revenue,
COALESCE(SUM(ds.discounts), 0) AS total_discounts,
COALESCE(SUM(ds.returns_revenue), 0) AS total_returns,
COALESCE(SUM(ds.net_revenue), 0) AS net_revenue,
COALESCE(SUM(ds.cogs), 0) AS total_cogs,
COALESCE(SUM(ds.gross_regular_revenue), 0) AS total_gross_revenue,
COALESCE(SUM(ds.profit), 0) AS total_profit,
CASE
WHEN COALESCE(SUM(ds.net_revenue), 0) > 0
THEN COALESCE(SUM(ds.profit), 0) / NULLIF(COALESCE(SUM(ds.net_revenue), 0), 0)
ELSE 0
END AS profit_margin,
-- Calculate average daily units
COALESCE(AVG(ds.units_sold), 0) AS avg_daily_units,
-- Calculate reorder point (simplified, can be enhanced with lead time and safety stock)
CEILING(COALESCE(AVG(ds.units_sold) * 14, 0)) AS reorder_point,
(p.stock_quantity <= CEILING(COALESCE(AVG(ds.units_sold) * 14, 0))) AS reorder_alert,
-- Days of supply based on average daily sales
CASE
WHEN COALESCE(AVG(ds.units_sold), 0) > 0
THEN p.stock_quantity / NULLIF(COALESCE(AVG(ds.units_sold), 0), 0)
ELSE NULL
END AS days_of_supply,
-- Sales velocity (average units sold per day over last 30 days)
(SELECT COALESCE(AVG(recent.units_sold), 0)
FROM public.daily_product_snapshots recent
WHERE recent.pid = p.pid
AND recent.snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
) AS sales_velocity,
-- Placeholder for sales velocity score (can be calculated based on velocity)
0 AS sales_velocity_score,
-- Will be updated later by ranking procedure
0 AS rank_by_revenue,
0 AS rank_by_quantity,
0 AS rank_by_profit,
-- Receiving data
COALESCE(SUM(ds.units_received), 0) AS total_received_quantity,
COALESCE(SUM(ds.cost_received), 0) AS total_received_cost,
-- Date metrics
(SELECT MAX(sd.snapshot_date)
FROM public.daily_product_snapshots sd
WHERE sd.pid = p.pid AND sd.units_sold > 0
) AS last_sold_date,
(SELECT MAX(rd.snapshot_date)
FROM public.daily_product_snapshots rd
WHERE rd.pid = p.pid AND rd.units_received > 0
) AS last_received_date,
-- Calculate days since last sale/received
CASE
WHEN (SELECT MAX(sd.snapshot_date)
FROM public.daily_product_snapshots sd
WHERE sd.pid = p.pid AND sd.units_sold > 0) IS NOT NULL
THEN (CURRENT_DATE - (SELECT MAX(sd.snapshot_date)
FROM public.daily_product_snapshots sd
WHERE sd.pid = p.pid AND sd.units_sold > 0))::integer
ELSE NULL
END AS days_since_last_sale,
CASE
WHEN (SELECT MAX(rd.snapshot_date)
FROM public.daily_product_snapshots rd
WHERE rd.pid = p.pid AND rd.units_received > 0) IS NOT NULL
THEN (CURRENT_DATE - (SELECT MAX(rd.snapshot_date)
FROM public.daily_product_snapshots rd
WHERE rd.pid = p.pid AND rd.units_received > 0))::integer
ELSE NULL
END AS days_since_last_received,
now() -- calculation timestamp
FROM public.products p
LEFT JOIN public.daily_product_snapshots ds ON p.pid = ds.pid
GROUP BY p.pid, p.sku, p.stock_quantity, p.landing_cost_price, p.cost_price, p.price, p.regular_price
)
-- Update the calculate_status table
INSERT INTO public.calculate_status (module_name, last_calculation_timestamp)
VALUES
('daily_snapshots', now()),
('product_metrics', now())
ON CONFLICT (module_name) DO UPDATE
SET last_calculation_timestamp = now();
-- Finally, update the ranks for products
UPDATE public.product_metrics pm SET
rank_by_revenue = rev_ranks.rank
FROM (
SELECT pid, RANK() OVER (ORDER BY net_revenue DESC) AS rank
FROM public.product_metrics
WHERE net_revenue > 0
) rev_ranks
WHERE pm.pid = rev_ranks.pid;
UPDATE public.product_metrics pm SET
rank_by_quantity = qty_ranks.rank
FROM (
SELECT pid, RANK() OVER (ORDER BY total_units_sold DESC) AS rank
FROM public.product_metrics
WHERE total_units_sold > 0
) qty_ranks
WHERE pm.pid = qty_ranks.pid;
UPDATE public.product_metrics pm SET
rank_by_profit = profit_ranks.rank
FROM (
SELECT pid, RANK() OVER (ORDER BY total_profit DESC) AS rank
FROM public.product_metrics
WHERE total_profit > 0
) profit_ranks
WHERE pm.pid = profit_ranks.pid;
-- Return count of products with metrics
SELECT COUNT(*) AS product_count FROM public.product_metrics
`);
};
async function populateInitialMetrics() {
let connection;
const startTime = Date.now();
let calculateHistoryId;
try {
// Clean up any previously running calculations
connection = await getConnection({
// Add performance-related settings
application_name: 'populate_metrics',
statement_timeout: PG_STATEMENT_TIMEOUT_MS, // 30 min timeout per statement
});
// Ensure the calculate_status table exists and has the correct structure
await connection.query(`
CREATE TABLE IF NOT EXISTS calculate_status (
module_name TEXT PRIMARY KEY,
last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
)
`);
await connection.query(`
UPDATE calculate_history
SET
status = 'cancelled',
end_time = NOW(),
duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
error_message = 'Previous calculation was not completed properly'
WHERE status = 'running' AND additional_info->>'type' = 'populate_initial_metrics'
`);
// Create history record for this calculation
const historyResult = await connection.query(`
INSERT INTO calculate_history (
start_time,
status,
additional_info
) VALUES (
NOW(),
'running',
jsonb_build_object(
'type', 'populate_initial_metrics',
'sql_file', 'populate_initial_product_metrics.sql'
)
) RETURNING id
`);
calculateHistoryId = historyResult.rows[0].id;
// Initialize progress
global.outputProgress({
status: 'running',
operation: 'Starting initial product metrics population',
current: 0,
total: 100,
elapsed: '0s',
remaining: 'Calculating... (this may take a while)',
rate: 0,
percentage: '0',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
},
historyId: calculateHistoryId
});
// Prepare the database - analyze tables
global.outputProgress({
status: 'running',
operation: 'Analyzing database tables for better query performance',
current: 2,
total: 100,
elapsed: global.formatElapsedTime(startTime),
remaining: 'Analyzing...',
rate: 0,
percentage: '2',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
},
historyId: calculateHistoryId
});
// Enable better query planning and parallel operations
await connection.query(`
-- Analyze tables for better query planning
ANALYZE public.products;
ANALYZE public.purchase_orders;
ANALYZE public.daily_product_snapshots;
ANALYZE public.orders;
-- Enable parallel operations
SET LOCAL enable_parallel_append = on;
SET LOCAL enable_parallel_hash = on;
SET LOCAL max_parallel_workers_per_gather = 4;
-- Larger work memory for complex sorts/joins
SET LOCAL work_mem = '128MB';
`).catch(err => {
// Non-fatal if analyze fails
console.warn('Failed to analyze tables (non-fatal):', err.message);
});
// Execute the SQL query
global.outputProgress({
status: 'running',
operation: 'Executing initial metrics SQL query',
current: 5,
total: 100,
elapsed: global.formatElapsedTime(startTime),
remaining: 'Calculating... (this could take several hours with 150M+ records)',
rate: 0,
percentage: '5',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
},
historyId: calculateHistoryId
});
// Read the SQL file
const sqlFilePath = path.resolve(__dirname, 'populate_initial_product_metrics.sql');
console.log('Base directory:', baseDir);
console.log('Script directory:', __dirname);
console.log('SQL file path:', sqlFilePath);
console.log('Current working directory:', process.cwd());
if (!fs.existsSync(sqlFilePath)) {
throw new Error(`SQL file not found at ${sqlFilePath}`);
}
// Read and clean up the SQL (Slightly more robust cleaning)
const sqlQuery = fs.readFileSync(sqlFilePath, 'utf8')
.replace(/\r\n/g, '\n') // Handle Windows endings
.replace(/\r/g, '\n') // Handle old Mac endings
.trim(); // Remove leading/trailing whitespace VERY IMPORTANT
// Log details again AFTER cleaning
console.log('SQL Query length (cleaned):', sqlQuery.length);
console.log('SQL Query structure validation:');
console.log('- Contains DO block:', sqlQuery.includes('DO $$') || sqlQuery.includes('DO $')); // Check both types of tag start
console.log('- Contains BEGIN:', sqlQuery.includes('BEGIN'));
console.log('- Contains END:', sqlQuery.includes('END $$;') || sqlQuery.includes('END $')); // Check both types of tag end
console.log('- First 50 chars:', JSON.stringify(sqlQuery.slice(0, 50)));
console.log('- Last 100 chars (cleaned):', JSON.stringify(sqlQuery.slice(-100)));
// Final check to ensure clean SQL ending
if (!sqlQuery.endsWith('END $$;')) {
console.warn('WARNING: SQL does not end with "END $$;". This might cause issues.');
console.log('Exact ending:', JSON.stringify(sqlQuery.slice(-20)));
}
// Execute the script
console.log('Starting initial product metrics population...');
// Track the query promise for potential cancellation
runningQueryPromise = connection.query({
text: sqlQuery,
rowMode: 'array'
});
await runningQueryPromise;
runningQueryPromise = null;
// Update progress to 100%
global.outputProgress({
status: 'complete',
operation: 'Initial product metrics population complete',
current: 100,
total: 100,
elapsed: global.formatElapsedTime(startTime),
remaining: '0s',
rate: 0,
percentage: '100',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
},
historyId: calculateHistoryId
});
// Update history with completion
await connection.query(`
UPDATE calculate_history
SET
end_time = NOW(),
duration_seconds = $1,
status = 'completed'
WHERE id = $2
`, [Math.round((Date.now() - startTime) / 1000), calculateHistoryId]);
// Clear progress file on successful completion
global.clearProgress();
return {
success: true,
message: 'Initial product metrics population completed successfully',
duration: Math.round((Date.now() - startTime) / 1000)
};
} catch (error) {
const endTime = Date.now();
const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
// Enhanced error logging
console.error('Error details:', {
message: error.message,
code: error.code,
hint: error.hint,
position: error.position,
detail: error.detail,
where: error.where ? error.where.substring(0, 500) + '...' : undefined, // Truncate to avoid huge logs
severity: error.severity,
file: error.file,
line: error.line,
routine: error.routine
});
// Update history with error
if (connection && calculateHistoryId) {
await connection.query(`
UPDATE calculate_history
SET
end_time = NOW(),
duration_seconds = $1,
status = $2,
error_message = $3
WHERE id = $4
`, [
totalElapsedSeconds,
isCancelled ? 'cancelled' : 'failed',
error.message,
calculateHistoryId
]);
}
if (isCancelled) {
global.outputProgress({
status: 'cancelled',
operation: 'Calculation cancelled',
current: 50,
total: 100,
elapsed: global.formatElapsedTime(startTime),
remaining: null,
rate: 0,
percentage: '50',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: totalElapsedSeconds
},
historyId: calculateHistoryId
});
} else {
global.outputProgress({
status: 'error',
operation: 'Error during initial product metrics population',
message: error.message,
current: 0,
total: 100,
elapsed: global.formatElapsedTime(startTime),
remaining: null,
rate: 0,
percentage: '0',
timing: {
start_time: new Date(startTime).toISOString(),
end_time: new Date().toISOString(),
elapsed_seconds: totalElapsedSeconds
},
historyId: calculateHistoryId
});
}
console.error('Error during initial product metrics population:', error);
return {
success: false,
error: error.message,
duration: totalElapsedSeconds
};
} finally {
if (connection) {
connection.release();
}
await closePool();
}
}
// Start population process
populateInitialMetrics()
.then(result => {
if (result.success) {
console.log(`Initial product metrics population completed successfully in ${result.duration} seconds`);
process.exit(0);
} else {
console.error(`Initial product metrics population failed: ${result.error}`);
process.exit(1);
}
})
.catch(err => {
console.error('Unexpected error:', err);
process.exit(1);
});
inventory-server/old/psql-csv-import.sh
-428
View File
@@ -1,428 +0,0 @@
#!/bin/bash
# Simple script to import CSV to PostgreSQL using psql
# Usage: ./psql-csv-import.sh <csv-file> <table-name> [start-batch]
# Exit on error
set -e
# Get arguments
CSV_FILE=$1
TABLE_NAME=$2
BATCH_SIZE=500000 # Process 500,000 rows at a time
START_BATCH=${3:-1} # Optional third parameter to start from a specific batch
if [ -z "$CSV_FILE" ] || [ -z "$TABLE_NAME" ]; then
echo "Usage: ./psql-csv-import.sh <csv-file> <table-name> [start-batch]"
exit 1
fi
# Check if file exists (only needed for batch 1)
if [ "$START_BATCH" -eq 1 ] && [ ! -f "$CSV_FILE" ]; then
echo "Error: CSV file '$CSV_FILE' not found"
exit 1
fi
# Load environment variables
if [ -f "../.env" ]; then
source "../.env"
else
echo "Warning: .env file not found, using default connection parameters"
fi
# Set default connection parameters if not from .env
DB_HOST=${DB_HOST:-localhost}
DB_PORT=${DB_PORT:-5432}
DB_NAME=${DB_NAME:-inventory_db}
DB_USER=${DB_USER:-postgres}
export PGPASSWORD=${DB_PASSWORD:-} # Export password for psql
# Common psql parameters
PSQL_OPTS="-h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME"
# Function to clean up database state
cleanup_and_optimize() {
echo "Cleaning up and optimizing database state..."
# Analyze the target table to update statistics
psql $PSQL_OPTS -c "ANALYZE $TABLE_NAME;"
# Perform vacuum to reclaim space and update stats
psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
# Reset connection pool
psql $PSQL_OPTS -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = current_database() AND pid <> pg_backend_pid();"
# Clean up shared memory
psql $PSQL_OPTS -c "DISCARD ALL;"
echo "Optimization complete."
}
# Show connection info
echo "Importing $CSV_FILE into $TABLE_NAME"
echo "Database: $DB_NAME on $DB_HOST:$DB_PORT with batch size: $BATCH_SIZE starting at batch $START_BATCH"
# Start timer
START_TIME=$(date +%s)
# Create progress tracking file
PROGRESS_FILE="/tmp/import_progress_${TABLE_NAME}.txt"
touch "$PROGRESS_FILE"
echo "Starting import at $(date), batch $START_BATCH" >> "$PROGRESS_FILE"
# If we're resuming, run cleanup first
if [ "$START_BATCH" -gt 1 ]; then
cleanup_and_optimize
fi
# For imported_product_stat_history, use optimized approach with hardcoded column names
if [ "$TABLE_NAME" = "imported_product_stat_history" ]; then
echo "Using optimized import for $TABLE_NAME"
# Only drop constraints/indexes and create staging table for batch 1
if [ "$START_BATCH" -eq 1 ]; then
# Extract CSV header
CSV_HEADER=$(head -n 1 "$CSV_FILE")
echo "CSV header: $CSV_HEADER"
# Step 1: Drop constraints and indexes
echo "Dropping constraints and indexes..."
psql $PSQL_OPTS -c "
DO \$\$
DECLARE
constraint_name TEXT;
BEGIN
-- Drop primary key constraint if exists
SELECT conname INTO constraint_name
FROM pg_constraint
WHERE conrelid = '$TABLE_NAME'::regclass AND contype = 'p';
IF FOUND THEN
EXECUTE 'ALTER TABLE $TABLE_NAME DROP CONSTRAINT IF EXISTS ' || constraint_name;
RAISE NOTICE 'Dropped primary key constraint: %', constraint_name;
END IF;
END \$\$;
"
# Drop all indexes on the table
psql $PSQL_OPTS -c "
DO \$\$
DECLARE
index_name TEXT;
index_record RECORD;
BEGIN
FOR index_record IN
SELECT indexname
FROM pg_indexes
WHERE tablename = '$TABLE_NAME'
LOOP
EXECUTE 'DROP INDEX IF EXISTS ' || index_record.indexname;
RAISE NOTICE 'Dropped index: %', index_record.indexname;
END LOOP;
END \$\$;
"
# Step 2: Set maintenance_work_mem and disable triggers
echo "Setting maintenance_work_mem and disabling triggers..."
psql $PSQL_OPTS -c "
SET maintenance_work_mem = '1GB';
ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;
"
# Step 3: Create staging table
echo "Creating staging table..."
psql $PSQL_OPTS -c "
DROP TABLE IF EXISTS staging_import;
CREATE UNLOGGED TABLE staging_import (
pid TEXT,
date TEXT,
score TEXT,
score2 TEXT,
qty_in_baskets TEXT,
qty_sold TEXT,
notifies_set TEXT,
visibility_score TEXT,
health_score TEXT,
sold_view_score TEXT
);
-- Create an index on staging_import to improve OFFSET performance
CREATE INDEX ON staging_import (pid);
"
# Step 4: Import CSV into staging table
echo "Importing CSV into staging table..."
psql $PSQL_OPTS -c "\copy staging_import FROM '$CSV_FILE' WITH CSV HEADER DELIMITER ','"
else
echo "Resuming import from batch $START_BATCH - skipping table creation and CSV import"
# Check if staging table exists
STAGING_EXISTS=$(psql $PSQL_OPTS -t -c "SELECT EXISTS(SELECT 1 FROM pg_tables WHERE tablename='staging_import');" | tr -d '[:space:]')
if [ "$STAGING_EXISTS" != "t" ]; then
echo "Error: Staging table 'staging_import' does not exist. Run without batch parameter first."
exit 1
fi
# Ensure triggers are disabled
psql $PSQL_OPTS -c "ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;"
# Optimize PostgreSQL for better performance
psql $PSQL_OPTS -c "
-- Increase work mem for this session
SET work_mem = '256MB';
SET maintenance_work_mem = '1GB';
"
fi
# Step 5: Get total row count
TOTAL_ROWS=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM staging_import;" | tr -d '[:space:]')
echo "Total rows to import: $TOTAL_ROWS"
# Calculate starting point
PROCESSED=$(( ($START_BATCH - 1) * $BATCH_SIZE ))
if [ $PROCESSED -ge $TOTAL_ROWS ]; then
echo "Error: Start batch $START_BATCH is beyond the available rows ($TOTAL_ROWS)"
exit 1
fi
# Step 6: Process in batches with shell loop
BATCH_NUM=$(( $START_BATCH - 1 ))
# We'll process batches in chunks of 10 before cleaning up
CHUNKS_SINCE_CLEANUP=0
while [ $PROCESSED -lt $TOTAL_ROWS ]; do
BATCH_NUM=$(( $BATCH_NUM + 1 ))
BATCH_START=$(date +%s)
MAX_ROWS=$(( $PROCESSED + $BATCH_SIZE ))
if [ $MAX_ROWS -gt $TOTAL_ROWS ]; then
MAX_ROWS=$TOTAL_ROWS
fi
echo "Processing batch $BATCH_NUM (rows $PROCESSED to $MAX_ROWS)..."
# Optimize query buffer for this batch
psql $PSQL_OPTS -c "SET work_mem = '256MB';"
# Insert batch with type casts
psql $PSQL_OPTS -c "
INSERT INTO $TABLE_NAME (
pid, date, score, score2, qty_in_baskets, qty_sold,
notifies_set, visibility_score, health_score, sold_view_score
)
SELECT
pid::bigint,
date::date,
score::numeric,
score2::numeric,
qty_in_baskets::smallint,
qty_sold::smallint,
notifies_set::smallint,
visibility_score::numeric,
health_score::varchar,
sold_view_score::numeric
FROM staging_import
LIMIT $BATCH_SIZE
OFFSET $PROCESSED;
"
# Update progress
BATCH_END=$(date +%s)
BATCH_ELAPSED=$(( $BATCH_END - $BATCH_START ))
PROGRESS_PCT=$(echo "scale=2; $MAX_ROWS * 100 / $TOTAL_ROWS" | bc)
echo "Batch $BATCH_NUM committed in ${BATCH_ELAPSED}s, $MAX_ROWS of $TOTAL_ROWS rows processed ($PROGRESS_PCT%)" | tee -a "$PROGRESS_FILE"
# Increment counter
PROCESSED=$(( $PROCESSED + $BATCH_SIZE ))
CHUNKS_SINCE_CLEANUP=$(( $CHUNKS_SINCE_CLEANUP + 1 ))
# Check current row count every 10 batches
if [ $(( $BATCH_NUM % 10 )) -eq 0 ]; then
CURRENT_COUNT=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM $TABLE_NAME;" | tr -d '[:space:]')
echo "Current row count in $TABLE_NAME: $CURRENT_COUNT" | tee -a "$PROGRESS_FILE"
# Every 10 batches, run an intermediate cleanup
if [ $CHUNKS_SINCE_CLEANUP -ge 10 ]; then
echo "Running intermediate cleanup and optimization..."
psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
CHUNKS_SINCE_CLEANUP=0
fi
fi
# Optional - write a checkpoint file to know where to restart
echo "$BATCH_NUM" > "/tmp/import_last_batch_${TABLE_NAME}.txt"
done
# Only recreate indexes if we've completed the import
if [ $PROCESSED -ge $TOTAL_ROWS ]; then
# Step 7: Re-enable triggers and recreate primary key
echo "Re-enabling triggers and recreating primary key..."
psql $PSQL_OPTS -c "
ALTER TABLE $TABLE_NAME ENABLE TRIGGER ALL;
ALTER TABLE $TABLE_NAME ADD PRIMARY KEY (pid, date);
"
# Step 8: Clean up and get final count
echo "Cleaning up and getting final count..."
psql $PSQL_OPTS -c "
DROP TABLE staging_import;
VACUUM ANALYZE $TABLE_NAME;
SELECT COUNT(*) AS \"Total rows in $TABLE_NAME\" FROM $TABLE_NAME;
"
else
echo "Import interrupted at batch $BATCH_NUM. To resume, run:"
echo "./psql-csv-import.sh $CSV_FILE $TABLE_NAME $BATCH_NUM"
fi
else
# Generic approach for other tables
if [ "$START_BATCH" -eq 1 ]; then
# Extract CSV header
CSV_HEADER=$(head -n 1 "$CSV_FILE")
echo "CSV header: $CSV_HEADER"
# Extract CSV header and format it for SQL
CSV_COLUMNS=$(echo "$CSV_HEADER" | tr ',' '\n' | sed 's/^/"/;s/$/"/' | tr '\n' ',' | sed 's/,$//')
TEMP_COLUMNS=$(echo "$CSV_HEADER" | tr ',' '\n' | sed 's/$/ TEXT/' | tr '\n' ',' | sed 's/,$//')
echo "Importing columns: $CSV_COLUMNS"
# Step 1: Set maintenance_work_mem and disable triggers
echo "Setting maintenance_work_mem and disabling triggers..."
psql $PSQL_OPTS -c "
SET maintenance_work_mem = '1GB';
ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;
"
# Step 2: Create temp table
echo "Creating temporary table..."
psql $PSQL_OPTS -c "
DROP TABLE IF EXISTS temp_import;
CREATE UNLOGGED TABLE temp_import ($TEMP_COLUMNS);
-- Create an index on temp_import to improve OFFSET performance
CREATE INDEX ON temp_import ((1)); -- Index on first column
"
# Step 3: Import CSV into temp table
echo "Importing CSV into temporary table..."
psql $PSQL_OPTS -c "\copy temp_import FROM '$CSV_FILE' WITH CSV HEADER DELIMITER ','"
else
echo "Resuming import from batch $START_BATCH - skipping table creation and CSV import"
# Check if temp table exists
TEMP_EXISTS=$(psql $PSQL_OPTS -t -c "SELECT EXISTS(SELECT 1 FROM pg_tables WHERE tablename='temp_import');" | tr -d '[:space:]')
if [ "$TEMP_EXISTS" != "t" ]; then
echo "Error: Temporary table 'temp_import' does not exist. Run without batch parameter first."
exit 1
fi
# Ensure triggers are disabled
psql $PSQL_OPTS -c "ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;"
# Optimize PostgreSQL for better performance
psql $PSQL_OPTS -c "
-- Increase work mem for this session
SET work_mem = '256MB';
SET maintenance_work_mem = '1GB';
"
# Hard-code columns since we know them
CSV_COLUMNS='"pid","date","score","score2","qty_in_baskets","qty_sold","notifies_set","visibility_score","health_score","sold_view_score"'
echo "Using standard columns: $CSV_COLUMNS"
fi
# Step 4: Get total row count
TOTAL_ROWS=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM temp_import;" | tr -d '[:space:]')
echo "Total rows to import: $TOTAL_ROWS"
# Calculate starting point
PROCESSED=$(( ($START_BATCH - 1) * $BATCH_SIZE ))
if [ $PROCESSED -ge $TOTAL_ROWS ]; then
echo "Error: Start batch $START_BATCH is beyond the available rows ($TOTAL_ROWS)"
exit 1
fi
# Step 5: Process in batches with shell loop
BATCH_NUM=$(( $START_BATCH - 1 ))
# We'll process batches in chunks of 10 before cleaning up
CHUNKS_SINCE_CLEANUP=0
while [ $PROCESSED -lt $TOTAL_ROWS ]; do
BATCH_NUM=$(( $BATCH_NUM + 1 ))
BATCH_START=$(date +%s)
MAX_ROWS=$(( $PROCESSED + $BATCH_SIZE ))
if [ $MAX_ROWS -gt $TOTAL_ROWS ]; then
MAX_ROWS=$TOTAL_ROWS
fi
echo "Processing batch $BATCH_NUM (rows $PROCESSED to $MAX_ROWS)..."
# Optimize query buffer for this batch
psql $PSQL_OPTS -c "SET work_mem = '256MB';"
# Insert batch
psql $PSQL_OPTS -c "
INSERT INTO $TABLE_NAME ($CSV_COLUMNS)
SELECT $CSV_COLUMNS
FROM temp_import
LIMIT $BATCH_SIZE
OFFSET $PROCESSED;
"
# Update progress
BATCH_END=$(date +%s)
BATCH_ELAPSED=$(( $BATCH_END - $BATCH_START ))
PROGRESS_PCT=$(echo "scale=2; $MAX_ROWS * 100 / $TOTAL_ROWS" | bc)
echo "Batch $BATCH_NUM committed in ${BATCH_ELAPSED}s, $MAX_ROWS of $TOTAL_ROWS rows processed ($PROGRESS_PCT%)" | tee -a "$PROGRESS_FILE"
# Increment counter
PROCESSED=$(( $PROCESSED + $BATCH_SIZE ))
CHUNKS_SINCE_CLEANUP=$(( $CHUNKS_SINCE_CLEANUP + 1 ))
# Check current row count every 10 batches
if [ $(( $BATCH_NUM % 10 )) -eq 0 ]; then
CURRENT_COUNT=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM $TABLE_NAME;" | tr -d '[:space:]')
echo "Current row count in $TABLE_NAME: $CURRENT_COUNT" | tee -a "$PROGRESS_FILE"
# Every 10 batches, run an intermediate cleanup
if [ $CHUNKS_SINCE_CLEANUP -ge 10 ]; then
echo "Running intermediate cleanup and optimization..."
psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
CHUNKS_SINCE_CLEANUP=0
fi
fi
# Optional - write a checkpoint file to know where to restart
echo "$BATCH_NUM" > "/tmp/import_last_batch_${TABLE_NAME}.txt"
done
# Only clean up if we've completed the import
if [ $PROCESSED -ge $TOTAL_ROWS ]; then
# Step 6: Re-enable triggers and clean up
echo "Re-enabling triggers and cleaning up..."
psql $PSQL_OPTS -c "
ALTER TABLE $TABLE_NAME ENABLE TRIGGER ALL;
DROP TABLE temp_import;
VACUUM ANALYZE $TABLE_NAME;
SELECT COUNT(*) AS \"Total rows in $TABLE_NAME\" FROM $TABLE_NAME;
"
else
echo "Import interrupted at batch $BATCH_NUM. To resume, run:"
echo "./psql-csv-import.sh $CSV_FILE $TABLE_NAME $BATCH_NUM"
fi
fi
# Calculate elapsed time
END_TIME=$(date +%s)
ELAPSED=$((END_TIME - START_TIME))
echo "Import completed successfully in ${ELAPSED}s ($(($ELAPSED / 60)) minutes)"
echo "Progress log saved to $PROGRESS_FILE"
inventory-server/old/reset-metrics.js
-378
View File
@@ -1,378 +0,0 @@
const { Client } = require('pg');
const path = require('path');
const fs = require('fs');
require('dotenv').config({ path: path.resolve(__dirname, '../.env') });
const dbConfig = {
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: process.env.DB_PORT || 5432
};
function outputProgress(data) {
if (!data.status) {
data = {
status: 'running',
...data
};
}
console.log(JSON.stringify(data));
}
// Explicitly define all metrics-related tables in dependency order
const METRICS_TABLES = [
'brand_metrics',
'brand_time_metrics',
'category_forecasts',
'category_metrics',
'category_sales_metrics',
'category_time_metrics',
'product_metrics',
'product_time_aggregates',
'sales_forecasts',
'temp_purchase_metrics',
'temp_sales_metrics',
'vendor_metrics',
'vendor_time_metrics',
'vendor_details'
];
// Tables to always protect from being dropped
const PROTECTED_TABLES = [
'users',
'permissions',
'user_permissions',
'calculate_history',
'import_history',
'ai_prompts',
'ai_validation_performance',
'templates',
'reusable_images'
];
// Split SQL into individual statements
function splitSQLStatements(sql) {
sql = sql.replace(/\r\n/g, '\n');
let statements = [];
let currentStatement = '';
let inString = false;
let stringChar = '';
for (let i = 0; i < sql.length; i++) {
const char = sql[i];
const nextChar = sql[i + 1] || '';
if ((char === "'" || char === '"') && sql[i - 1] !== '\\') {
if (!inString) {
inString = true;
stringChar = char;
} else if (char === stringChar) {
inString = false;
}
}
if (!inString && char === '-' && nextChar === '-') {
while (i < sql.length && sql[i] !== '\n') i++;
continue;
}
if (!inString && char === '/' && nextChar === '*') {
i += 2;
while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
i++;
continue;
}
if (!inString && char === ';') {
if (currentStatement.trim()) {
statements.push(currentStatement.trim());
}
currentStatement = '';
} else {
currentStatement += char;
}
}
if (currentStatement.trim()) {
statements.push(currentStatement.trim());
}
return statements;
}
async function resetMetrics() {
let client;
try {
outputProgress({
operation: 'Starting metrics reset',
message: 'Connecting to database...'
});
client = new Client(dbConfig);
await client.connect();
// Explicitly begin a transaction
await client.query('BEGIN');
// First verify current state
const initialTables = await client.query(`
SELECT tablename as name
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = ANY($1)
AND tablename NOT IN (SELECT unnest($2::text[]))
`, [METRICS_TABLES, PROTECTED_TABLES]);
outputProgress({
operation: 'Initial state',
message: `Found ${initialTables.rows.length} existing metrics tables: ${initialTables.rows.map(t => t.name).join(', ')}`
});
// Disable foreign key checks at the start
await client.query('SET session_replication_role = \'replica\'');
// Drop all metrics tables in reverse order to handle dependencies
outputProgress({
operation: 'Dropping metrics tables',
message: 'Removing existing metrics tables...'
});
for (const table of [...METRICS_TABLES].reverse()) {
// Skip protected tables
if (PROTECTED_TABLES.includes(table)) {
outputProgress({
operation: 'Protected table',
message: `Skipping protected table: ${table}`
});
continue;
}
try {
// Use NOWAIT to avoid hanging if there's a lock
await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
// Verify the table was actually dropped
const checkDrop = await client.query(`
SELECT COUNT(*) as count
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = $1
`, [table]);
if (parseInt(checkDrop.rows[0].count) > 0) {
throw new Error(`Failed to drop table ${table} - table still exists`);
}
outputProgress({
operation: 'Table dropped',
message: `Successfully dropped table: ${table}`
});
// Commit after each table drop to ensure locks are released
await client.query('COMMIT');
// Start a new transaction for the next table
await client.query('BEGIN');
// Re-disable foreign key constraints for the new transaction
await client.query('SET session_replication_role = \'replica\'');
} catch (err) {
outputProgress({
status: 'error',
operation: 'Drop table error',
message: `Error dropping table ${table}: ${err.message}`
});
await client.query('ROLLBACK');
// Re-start transaction for next table
await client.query('BEGIN');
await client.query('SET session_replication_role = \'replica\'');
}
}
// Verify all tables were dropped
const afterDrop = await client.query(`
SELECT tablename as name
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = ANY($1)
`, [METRICS_TABLES]);
if (afterDrop.rows.length > 0) {
throw new Error(`Failed to drop all tables. Remaining tables: ${afterDrop.rows.map(t => t.name).join(', ')}`);
}
// Make sure we have a fresh transaction here
await client.query('COMMIT');
await client.query('BEGIN');
await client.query('SET session_replication_role = \'replica\'');
// Read metrics schema
outputProgress({
operation: 'Reading schema',
message: 'Loading metrics schema file...'
});
const schemaPath = path.resolve(__dirname, '../db/metrics-schema.sql');
if (!fs.existsSync(schemaPath)) {
throw new Error(`Schema file not found at: ${schemaPath}`);
}
const schemaSQL = fs.readFileSync(schemaPath, 'utf8');
const statements = splitSQLStatements(schemaSQL);
outputProgress({
operation: 'Schema loaded',
message: `Found ${statements.length} SQL statements to execute`
});
// Execute schema statements
for (let i = 0; i < statements.length; i++) {
const stmt = statements[i];
try {
const result = await client.query(stmt);
// If this is a CREATE TABLE statement, verify the table was created
if (stmt.trim().toLowerCase().startsWith('create table')) {
const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(\w+)["]?/i)?.[1];
if (tableName) {
const checkCreate = await client.query(`
SELECT tablename as name
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = $1
`, [tableName]);
if (checkCreate.rows.length === 0) {
throw new Error(`Failed to create table ${tableName} - table does not exist after CREATE statement`);
}
outputProgress({
operation: 'Table created',
message: `Successfully created table: ${tableName}`
});
}
}
outputProgress({
operation: 'SQL Progress',
message: {
statement: i + 1,
total: statements.length,
preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
rowCount: result.rowCount
}
});
// Commit every 10 statements to avoid long-running transactions
if (i > 0 && i % 10 === 0) {
await client.query('COMMIT');
await client.query('BEGIN');
await client.query('SET session_replication_role = \'replica\'');
}
} catch (sqlError) {
outputProgress({
status: 'error',
operation: 'SQL Error',
message: {
error: sqlError.message,
statement: stmt,
statementNumber: i + 1
}
});
await client.query('ROLLBACK');
throw sqlError;
}
}
// Final commit for any pending statements
await client.query('COMMIT');
// Start new transaction for final checks
await client.query('BEGIN');
// Re-enable foreign key checks after all tables are created
await client.query('SET session_replication_role = \'origin\'');
// Verify metrics tables were created
outputProgress({
operation: 'Verifying metrics tables',
message: 'Checking all metrics tables were created...'
});
const metricsTablesResult = await client.query(`
SELECT tablename as name
FROM pg_tables
WHERE schemaname = 'public'
AND tablename = ANY($1)
`, [METRICS_TABLES]);
outputProgress({
operation: 'Tables found',
message: `Found ${metricsTablesResult.rows.length} tables: ${metricsTablesResult.rows.map(t => t.name).join(', ')}`
});
const existingMetricsTables = metricsTablesResult.rows.map(t => t.name);
const missingMetricsTables = METRICS_TABLES.filter(t => !existingMetricsTables.includes(t));
if (missingMetricsTables.length > 0) {
// Do one final check of the actual tables
const finalCheck = await client.query(`
SELECT tablename as name
FROM pg_tables
WHERE schemaname = 'public'
`);
outputProgress({
operation: 'Final table check',
message: `All database tables: ${finalCheck.rows.map(t => t.name).join(', ')}`
});
await client.query('ROLLBACK');
throw new Error(`Failed to create metrics tables: ${missingMetricsTables.join(', ')}`);
}
// Commit final transaction
await client.query('COMMIT');
outputProgress({
status: 'complete',
operation: 'Reset complete',
message: 'All metrics tables have been reset successfully'
});
} catch (error) {
outputProgress({
status: 'error',
operation: 'Reset failed',
message: error.message,
stack: error.stack
});
if (client) {
try {
await client.query('ROLLBACK');
} catch (rollbackError) {
console.error('Error during rollback:', rollbackError);
}
// Make sure to re-enable foreign key checks even if there's an error
await client.query('SET session_replication_role = \'origin\'').catch(() => {});
}
throw error;
} finally {
if (client) {
// One final attempt to ensure foreign key checks are enabled
await client.query('SET session_replication_role = \'origin\'').catch(() => {});
await client.end();
}
}
}
// Export if required as a module
if (typeof module !== 'undefined' && module.exports) {
module.exports = resetMetrics;
}
// Run if called from command line
if (require.main === module) {
resetMetrics().catch(error => {
console.error('Error:', error);
process.exit(1);
});
}
inventory-server/old/scripts.js
-180
View File
@@ -1,180 +0,0 @@
const readline = require('readline');
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
});
const question = (query) => new Promise((resolve) => rl.question(query, resolve));
async function loadScript(name) {
try {
return await require(name);
} catch (error) {
console.error(`Failed to load script ${name}:`, error);
return null;
}
}
async function runWithTimeout(fn) {
return new Promise((resolve, reject) => {
// Create a child process for the script
const child = require('child_process').fork(fn.toString(), [], {
stdio: 'inherit'
});
child.on('exit', (code) => {
if (code === 0) {
resolve();
} else {
reject(new Error(`Script exited with code ${code}`));
}
});
child.on('error', (err) => {
reject(err);
});
});
}
function clearScreen() {
process.stdout.write('\x1Bc');
}
const scripts = {
'Import Scripts': {
'1': { name: 'Full Import From Production', path: './import-from-prod' },
'2': { name: 'Individual Import Scripts ▸', submenu: {
'1': { name: 'Import Orders', path: './import/orders', key: 'importOrders' },
'2': { name: 'Import Products', path: './import/products', key: 'importProducts' },
'3': { name: 'Import Purchase Orders', path: './import/purchase-orders' },
'4': { name: 'Import Categories', path: './import/categories' },
'b': { name: 'Back to Main Menu' }
}}
},
'Metrics': {
'3': { name: 'Calculate All Metrics', path: './calculate-metrics' },
'4': { name: 'Individual Metric Scripts ▸', submenu: {
'1': { name: 'Brand Metrics', path: './metrics/brand-metrics' },
'2': { name: 'Category Metrics', path: './metrics/category-metrics' },
'3': { name: 'Financial Metrics', path: './metrics/financial-metrics' },
'4': { name: 'Product Metrics', path: './metrics/product-metrics' },
'5': { name: 'Sales Forecasts', path: './metrics/sales-forecasts' },
'6': { name: 'Time Aggregates', path: './metrics/time-aggregates' },
'7': { name: 'Vendor Metrics', path: './metrics/vendor-metrics' },
'b': { name: 'Back to Main Menu' }
}}
},
'Database Management': {
'5': { name: 'Test Production Connection', path: './test-prod-connection' }
},
'Reset Scripts': {
'6': { name: 'Reset Database', path: './reset-db' },
'7': { name: 'Reset Metrics', path: './reset-metrics' }
}
};
let lastRun = null;
async function displayMenu(menuItems, title = 'Inventory Management Script Runner') {
clearScreen();
console.log(`\n${title}\n`);
for (const [category, items] of Object.entries(menuItems)) {
console.log(`\n${category}:`);
Object.entries(items).forEach(([key, script]) => {
console.log(`${key}. ${script.name}`);
});
}
if (lastRun) {
console.log('\nQuick Access:');
console.log(`r. Repeat Last Script (${lastRun.name})`);
}
console.log('\nq. Quit\n');
}
async function handleSubmenu(submenu, title) {
while (true) {
await displayMenu({"Individual Scripts": submenu}, title);
const choice = await question('Select an option (or b to go back): ');
if (choice.toLowerCase() === 'b') {
return null;
}
if (submenu[choice]) {
return submenu[choice];
}
console.log('Invalid selection. Please try again.');
await new Promise(resolve => setTimeout(resolve, 1000));
}
}
async function runScript(script) {
console.log(`\nRunning: ${script.name}`);
try {
const scriptPath = require.resolve(script.path);
await runWithTimeout(scriptPath);
console.log('\nScript completed successfully');
lastRun = script;
} catch (error) {
console.error('\nError running script:', error);
}
await question('\nPress Enter to continue...');
}
async function main() {
while (true) {
await displayMenu(scripts);
const choice = await question('Select an option: ');
if (choice.toLowerCase() === 'q') {
break;
}
if (choice.toLowerCase() === 'r' && lastRun) {
await runScript(lastRun);
continue;
}
let selectedScript = null;
for (const category of Object.values(scripts)) {
if (category[choice]) {
selectedScript = category[choice];
break;
}
}
if (!selectedScript) {
console.log('Invalid selection. Please try again.');
await new Promise(resolve => setTimeout(resolve, 1000));
continue;
}
if (selectedScript.submenu) {
const submenuChoice = await handleSubmenu(
selectedScript.submenu,
selectedScript.name
);
if (submenuChoice && submenuChoice.path) {
await runScript(submenuChoice);
}
} else if (selectedScript.path) {
await runScript(selectedScript);
}
}
rl.close();
process.exit(0);
}
if (require.main === module) {
main().catch(error => {
console.error('Fatal error:', error);
process.exit(1);
});
}

Some files were not shown because too many files have changed in this diff Show More