Phase 3 + 6

This commit is contained in:
2026-05-23 19:38:12 -04:00
parent 1ab14ba45f
commit 82e568d455
60 changed files with 1983 additions and 2720 deletions
+139 -9
View File
@@ -10,15 +10,18 @@ Audit-driven plan to (a) reduce 12 PM2 processes to 3 application servers + 1 au
|---|---|---| |---|---|---|
| 1 — Decommission dead services | **Complete** | aircall/gorgias/clarity/legacy-auth-server deleted from repo + PM2 + Caddyfile + ecosystem.cjs | | 1 — Decommission dead services | **Complete** | aircall/gorgias/clarity/legacy-auth-server deleted from repo + PM2 + Caddyfile + ecosystem.cjs |
| 2 — Build shared `lib/` | **Complete** | Lives at `inventory-server/shared/` (see Deviations). `/verify` endpoint live on auth-server | | 2 — Build shared `lib/` | **Complete** | Lives at `inventory-server/shared/` (see Deviations). `/verify` endpoint live on auth-server |
| 3 — Convert auth-server + inventory-server to ESM | Not started | Next up. Auth-server is still CJS but has the new `/verify` route added in CJS form | | 3 — Convert auth-server + inventory-server to ESM | **Complete (code)** | All 58 server-side files ESM; verified 0 import failures on netcup. Pending: `npm install` on server + pm2 reload to actually run the new code. See Deviations #1013 |
| 4 — Build `dashboard-server` (the merge) | Not started | klaviyo/meta/google/typeform still run as 4 separate PM2 apps | | 4 — Build `dashboard-server` (the merge) | Not started | klaviyo/meta/google/typeform still run as 4 separate PM2 apps |
| 5 — Convert `acot-server` to ESM | Not started | | | 5 — Convert `acot-server` to ESM | Not started | |
| 6 — Auth hardening | Not started | Shared modules exist (`shared/rate-limit`, `shared/cors`, `shared/logging`) but no service consumes them yet. JWT_SECRET footgun discovered — see 6.4 | | 6 — Auth hardening | **Complete (code) — gated on Phase F1** | All in-process items wired (rate-limit, JWT precondition, CORS lockdown, request-log, upload allowlist, `requirePermission` on sensitive routes, permissions seed migration). `authenticate()` is live on `/api/*`. Server-side artefacts (Caddyfile, ecosystem.cjs) written to `inventory-server/deploy/` for review. 6.11 (audit logging) deferred. **Frontend cannot use the app until Phase F1 ships** — see below |
| 7 — Caddyfile final form | Partial | Dead routes removed; `forward_auth` gate + `/uploads/*` gating + per-vendor cleanup deferred to after Phase 4 | | **F1 — Frontend fetch wrapper (NEW)** | **Not started — CRITICAL** | Frontend uses raw `fetch()` in ~220 sites; only 7 send `Authorization: Bearer`. With Phase 6's `authenticate()` middleware live, every refresh 401s until the frontend uniformly attaches the token. See "Phase F1" below |
| 8ecosystem.config.cjs final form | Partial | Dead apps removed; final shape depends on Phase 4 merge | | 7Caddyfile final form | Partial | Proposed file at `inventory-server/deploy/Caddyfile.proposed`. Apply blocked on F1 (forward_auth would 401 every page load until then) |
| 8 — ecosystem.config.cjs final form | Partial | Proposed at `inventory-server/deploy/ecosystem.config.cjs.proposed`. Includes Phase 6.4 JWT_SECRET footgun fix and 6.10 lt-wordlist token move |
**Live PM2 count: 10** (down from 13). Target after Phase 4: 5 application apps + acot-phone-server + lt-wordlist-api. **Live PM2 count: 10** (down from 13). Target after Phase 4: 5 application apps + acot-phone-server + lt-wordlist-api.
**Apply order from current state:** (a) `npm install` on netcup to install the new shared-module deps (`pino`, `pino-http`, `ioredis`, `express-rate-limit`, `jsonwebtoken`), (b) ship Phase F1 frontend fetch wrapper, (c) `pm2 reload inventory-server new-auth-server` (Phase 3+6 code goes live, requests carry tokens, app keeps working), (d) apply `deploy/ecosystem.config.cjs.proposed` (Phase 6.4 + 6.10), (e) apply `deploy/Caddyfile.proposed` (Phase 6.1 — edge gate).
--- ---
## Goals ## Goals
@@ -199,7 +202,11 @@ Caddy's `forward_auth` only needs "is this token valid? give me a user-id." Toda
## Phase 3 — Convert `auth-server` and `inventory-server` to ESM ## Phase 3 — Convert `auth-server` and `inventory-server` to ESM
Status: **Not started.** Lift the easy ones first. These two stay standalone (don't merge into anything), so they're isolated changes. The auth-server's new `/verify` route (added in Phase 2) is currently CJS — refactor it during this phase to import from `../shared/auth/verify.js`. Status: **Complete (code) — 2026-05-23.** Both servers + all sub-trees converted to ESM. 58 importable .js files load cleanly on netcup (verified via dynamic-import sweep). Two latent bugs surfaced and fixed: `??`/`||` precedence in `shared/db/{pg,redis}.js`, and CJS named-import of `Pool` from `pg` in both auth files (now uses `import pg from 'pg'; const { Pool } = pg`).
Scripts under `inventory-server/scripts/` (one-shot maintenance / orchestrators) kept CommonJS via a sibling `scripts/package.json` declaring `"type": "commonjs"` — Node's package-type resolution walks up directory by directory, so this overrides the parent's `"type": "module"` without renaming any file or touching any `spawn()` callsite. Convert individual scripts to ESM if/when touched.
Pending to actually go live: `npm install` on netcup (new deps: `pino`, `pino-http`, `ioredis`, `express-rate-limit`, `jsonwebtoken`) + `pm2 reload`. See "Phase F1" — the frontend fetch wrapper should ship in the same deploy or this immediately breaks the app.
### Mechanical conversion ### Mechanical conversion
@@ -357,7 +364,23 @@ Same as inventory-server: start with PM2, smoke-test the most-used `/api/acot/*`
## Phase 6 — Auth hardening ## Phase 6 — Auth hardening
Status: **Not started.** This is the security work that justifies the whole refactor. Runs in parallel with phases 35 where possible. Shared building blocks already exist (`shared/rate-limit/login.js`, `shared/cors/policy.js`, `shared/logging/request-log.js`, `shared/errors/handler.js`) — Phase 6 is about *applying* them per-service. Status: **Complete (code) — 2026-05-23. Application gated on Phase F1.** All in-process hardening shipped alongside the Phase 3 ESM conversion. The `authenticate()` middleware is wired live on `/api/*` in inventory-server — **the moment that code reaches production, the frontend stops working until Phase F1 lands**, because today's frontend doesn't include `Authorization: Bearer` on the vast majority of fetch calls (see Phase F1 below for the diagnosis).
Per-item status:
| # | Item | Status | Where |
|---|---|---|---|
| 6.1 | Caddy `forward_auth` gate | **Proposed** — apply *after* F1 | `inventory-server/deploy/Caddyfile.proposed` |
| 6.2 | `requirePermission` on sensitive routes + permissions migration | **Done** | inline in `config.js`, `data-management.js`, `import.js`, `ai-prompts.js`, `ai-validation.js`, `templates.js`, `reusable-images.js`; codes seeded by `migrations/005_phase6_permission_codes.sql` |
| 6.3 | Login rate-limit + `/verify` rate-limit | **Done** | `auth/server.js` uses `shared/rate-limit/login.js` (`loginLimiter`, `verifyLimiter`) |
| 6.4 | JWT_SECRET as startup precondition + ecosystem footgun fix | **Done in code; proposed for ecosystem.cjs** | Both auth-server and inventory-server `process.exit(1)` if `JWT_SECRET` is unset. `inventory-server/deploy/ecosystem.config.cjs.proposed` removes the `JWT_SECRET: process.env.JWT_SECRET` override that was shadowing `.env` |
| 6.5 | Structured request logging w/ redaction | **Done** | `shared/logging/request-log.js` (pino-http, redacts Authorization/Cookie); mounted in both `auth/server.js` and `src/server.js` |
| 6.6 | CORS lockdown | **Done** | `src/middleware/cors.js` now re-exports `shared/cors/policy.js`. LAN wildcards (`192.168.*`, `10.*`) and `*` defaults gone |
| 6.7 | Upload hardening | **Done** | Exact-match MIME+extension allowlist on `routes/import.js` and `routes/reusable-images.js`; dead `multer({ dest })` removed from `routes/products.js` (no upload route was using it — strongest hardening was deletion) |
| 6.8 | Frontend token storage stays localStorage + XSS audit | **Audited** | Confirmed `dangerouslySetInnerHTML` is sanitized in `ProductEditor.tsx`. **Flagged: `ChatRoom.tsx:277,392` renders user-controlled chat content as raw HTML — real XSS vector, separate fix needed** |
| 6.9 | Remove debug middleware | **Done** | The header-dumping `app.use((req,res,next)=>{ console.log(... req.headers ...) })` block removed from `src/server.js`. Replaced with `shared/logging/request-log.js` (which redacts). |
| 6.10 | `lt-wordlist-api` token move | **Proposed for ecosystem.cjs** | `inventory-server/deploy/ecosystem.config.cjs.proposed` shows the entry without inline token; apply alongside rotating the secret value into `/opt/lt-wordlist-api/.env` |
| 6.11 | Audit logging for sensitive ops | **Deferred** | Out of scope for this pass per user direction. Existing `import_audit_log` and `product_editor_audit_log` tables stay as-is; generic `system_audit_log` table + middleware is its own project |
### 6.1 Caddy `forward_auth` gate ### 6.1 Caddy `forward_auth` gate
@@ -494,8 +517,99 @@ Already have `import-audit-log` and `product-editor-audit-log` tables. Extend th
--- ---
## Phase F1 — Frontend fetch wrapper (NEW — 2026-05-23)
Status: **Not started. CRITICAL. Blocks the Phase 3+6 deploy from being usable.**
### The discovery
While wiring `authenticate()` on `/api/*` in Phase 6.1/6.2, we audited the frontend's fetch usage and found:
- **7** call sites send `Authorization: Bearer ${token}` explicitly (all in `AuthContext.tsx` for `/me` + `/login`, plus a couple of `settings/*` pages).
- **~220** other `fetch(...)` / `axios.*(...)` call sites across `inventory/src/services/`, `inventory/src/pages/`, `inventory/src/components/` send **no** Authorization header at all.
- There is no global fetch wrapper, axios interceptor, or service-worker shim that injects the token.
Today this works because nothing on the server checks. Caddy currently has no `forward_auth` gate (Phase 6.1 is a Caddyfile change that hasn't shipped yet) and the previous inventory-server had no `authenticate()` middleware. The frontend's auth model was "you log in once to get the token; the token is checked only by `/me`; everything else is implicitly trusted at the network layer."
With Phase 6 code in production, **every page refresh 401s** on the first API call after the next pm2 reload. The user explicitly accepted this when authorising the Phase 6 work — but the fix is its own deliverable, and shipping Phase 3+6 to PM2 without F1 in the same window means an outage window measured in *however long F1 takes* (not minutes).
### Recommended approach
Add a single fetch wrapper at `inventory/src/utils/api.ts` (or similar) and migrate the ~220 call sites to use it. The wrapper:
1. Reads `localStorage.getItem('token')` on every call (cheap; localStorage is sync).
2. Merges `Authorization: Bearer ${token}` into the request headers if a token exists.
3. Intercepts 401 responses → fires `window.dispatchEvent(new Event('auth:logout'))` (a listener already exists in `AuthContext.tsx:117`) so the user gets bounced to `/login` cleanly instead of seeing broken pages.
4. Preserves the existing call shape — `apiFetch(url, init)` should be a drop-in for `fetch(url, init)` so the migration is mechanical.
```ts
// inventory/src/utils/api.ts (sketch)
export async function apiFetch(input: RequestInfo | URL, init: RequestInit = {}): Promise<Response> {
const token = localStorage.getItem('token');
const headers = new Headers(init.headers);
if (token && !headers.has('Authorization')) {
headers.set('Authorization', `Bearer ${token}`);
}
const res = await fetch(input, { ...init, headers });
if (res.status === 401 && token) {
// Token expired or revoked — bounce to /login. AuthContext already listens.
window.dispatchEvent(new Event('auth:logout'));
}
return res;
}
```
Same shape for axios:
```ts
// inventory/src/utils/apiClient.ts (sketch)
import axios from 'axios';
export const apiClient = axios.create();
apiClient.interceptors.request.use((config) => {
const token = localStorage.getItem('token');
if (token) config.headers.Authorization = `Bearer ${token}`;
return config;
});
apiClient.interceptors.response.use(
(r) => r,
(err) => {
if (err?.response?.status === 401) window.dispatchEvent(new Event('auth:logout'));
return Promise.reject(err);
},
);
```
### Migration plan
1. Land the two wrapper modules above. ~50 LOC total.
2. Codemod or sed-loop: in `inventory/src/`, replace `fetch(``apiFetch(` (with the right import) and `axios.get/post/...``apiClient.get/post/...`. ~220 call sites — a half-day of careful find-and-replace plus per-page verification. Spot-check the ones with custom `Content-Type` (multipart uploads especially) so the wrapper doesn't clobber multipart boundaries.
3. Leave the `AuthContext.tsx` `/login` and `/me` calls alone — they already work and migrating them adds no value.
4. Run the SPA: log in, exercise Overview / Products / Analytics / Dashboard / etc. with browser devtools open watching for `Authorization` header on every `/api/*` request.
### Sequencing with Phase 3+6 deploy
**Two options:**
A) **Ship F1 first** (recommended). Frontend goes out with the wrapper; nothing changes server-side. Then `pm2 reload` Phase 3+6. Zero-downtime, zero broken-page window.
B) **Ship together.** F1 and Phase 3+6 land in the same deploy. Brief window (seconds) where the frontend has the wrapper but the server hasn't reloaded yet — wrapper just sends extra headers the old server ignores. Safe.
Do **not** ship Phase 3+6 first and F1 second. That gives a broken app for as long as F1 takes.
### Out of scope (kept on `localStorage`)
Per Phase 6.8, we're not migrating to httpOnly cookie auth. F1 is the minimum work to make the per-service `authenticate()` (Phase 6) actually usable. A future Phase F2 could move to cookies + CSRF double-submit, but that's a much larger change touching the AuthContext, the login flow, and every backend that reads tokens. Not justified for an internal tool with no public sign-up.
### Note on `/uploads/*` gating (Phase 6.7's Caddyfile change)
The proposed Caddyfile moves `/uploads/*` behind `forward_auth`. Most product images today are referenced from `<img src="/uploads/...">` in the SPA — those requests are made by the browser, which **does not include `Authorization` headers on image requests**. Fixing this is part of F1's scope too: either (a) keep `/uploads/*` public (revert that part of 6.7) and accept that uploaded images leak to anyone who guesses a URL, or (b) issue per-image signed URLs from the API and gate those at Caddy. Decide before applying the Caddyfile.
---
## Phase 7 — Caddyfile final form ## Phase 7 — Caddyfile final form
Status: **Proposed (2026-05-23). Apply blocked on Phase F1.** The full proposed file lives at `inventory-server/deploy/Caddyfile.proposed` and matches the spec below except that vendor handle blocks still point to per-vendor PM2 apps (Phase 4 hasn't merged them yet). See `inventory-server/deploy/README.md` for the apply commands (admin-API + sudo cp pattern from Phase 2 deviation #8).
After all phases, the `tools.acherryontop.com` block looks like: After all phases, the `tools.acherryontop.com` block looks like:
```caddyfile ```caddyfile
@@ -571,6 +685,8 @@ Removed: `/dashboard-auth/*`, `/api/aircall/*`, `/api/gorgias/*`, `/api/clarity/
## Phase 8 — ecosystem.config.cjs final form ## Phase 8 — ecosystem.config.cjs final form
Status: **Proposed (2026-05-23).** Full proposed file at `inventory-server/deploy/ecosystem.config.cjs.proposed`. Includes the Phase 6.4 `JWT_SECRET` shadow-override fix and the Phase 6.10 `lt-wordlist-api` token move. Still lists per-vendor PM2 apps until Phase 4 merge ships — that's the only thing keeping app count at 10 instead of the target 5.
```js ```js
module.exports = { module.exports = {
apps: [ apps: [
@@ -646,10 +762,11 @@ Phase 1 unblocks everything (fewer services to convert).
Phase 2 is the foundation; nothing else can start until shared `lib/` exists. Phase 2 is the foundation; nothing else can start until shared `lib/` exists.
Phases 35 can run in parallel; they touch independent services. Phases 35 can run in parallel; they touch independent services.
Phase 6's sub-items can be developed alongside 35 but **enabled** only after them (no point adding `requirePermission` to a route that doesn't yet have `authenticate`). Phase 6's sub-items can be developed alongside 35 but **enabled** only after them (no point adding `requirePermission` to a route that doesn't yet have `authenticate`).
Phase 7 is the cutover: Caddyfile flip happens when all backend changes are deployed. **Phase F1 must precede the Phase 3+6 pm2 reload** — without the fetch wrapper, the moment the new code goes live the SPA breaks. Discovered during Phase 3+6 implementation; see Phase F1.
Phase 7 is the cutover: Caddyfile flip happens after F1 ships AND after the `/uploads/*` gating decision in F1 is made.
Phase 8 is cleanup: remove dead PM2 entries. Phase 8 is cleanup: remove dead PM2 entries.
Estimated effort, end-to-end: **~3 weeks of focused work** by one engineer. Phase 1 ≈ 1 day, Phase 2 ≈ 2 days, Phase 3 ≈ 3 days (both services), Phase 4 ≈ 57 days (the merge), Phase 5 ≈ 23 days, Phase 6 ≈ 34 days, Phase 7+8 ≈ 1 day. Estimated effort, end-to-end: **~3 weeks of focused work** by one engineer. Phase 1 ≈ 1 day, Phase 2 ≈ 2 days, Phase 3 ≈ 3 days (both services), Phase 4 ≈ 57 days (the merge), Phase 5 ≈ 23 days, Phase 6 ≈ 34 days, Phase F1 ≈ 0.51 day, Phase 7+8 ≈ 1 day.
--- ---
@@ -699,12 +816,13 @@ Each phase produces an independently deployable state. Rollback per phase:
These came up in the audit but aren't part of this refactor: These came up in the audit but aren't part of this refactor:
- `httpOnly` cookie auth (deferred — current `localStorage` acceptable for internal tool). - `httpOnly` cookie auth ("Phase F2" — deferred). Phase F1 keeps `localStorage` + Bearer header because that's the minimum to unblock the Phase 6 `authenticate()` rollout. A future move to cookie auth would touch `AuthContext`, every backend that reads tokens, and introduce CSRF concerns — much larger project.
- Replacing PM2 with systemd or Docker. - Replacing PM2 with systemd or Docker.
- Test coverage beyond the auth-critical surface. - Test coverage beyond the auth-critical surface.
- `apiv2`/`apiv2-test` proxies to `backend.acherryontop.com` — separate system, not touched. - `apiv2`/`apiv2-test` proxies to `backend.acherryontop.com` — separate system, not touched.
- `acot-phone-server` and `lt-wordlist-api` — staying as-is. - `acot-phone-server` and `lt-wordlist-api` — staying as-is.
- Centralized observability stack (Prometheus, Grafana). The logger work in Phase 6.5 sets up the data, but shipping it somewhere is future work. - Centralized observability stack (Prometheus, Grafana). The logger work in Phase 6.5 sets up the data, but shipping it somewhere is future work.
- ChatRoom XSS remediation (flagged during Phase 6.8 audit — `inventory/src/components/chat/ChatRoom.tsx:277,392` renders user-controlled chat content via `dangerouslySetInnerHTML` without sanitization). Real vulnerability for an internal-but-multi-user tool; separate fix.
--- ---
@@ -749,3 +867,15 @@ These are decisions made during Phase 1/2 implementation that amend the spec abo
8. **Caddyfile changes via admin API on `:2020`.** The Caddyfile is owned by root and matt has no passwordless sudo. Cutover used `curl -X POST .../load` on the Caddy admin port (which matt can hit), then a separate `sudo cp /home/matt/Caddyfile.new /etc/caddy/Caddyfile` step to persist the on-disk file. Future Caddyfile changes can follow the same pattern. Backup convention: `/etc/caddy/Caddyfile.bak.YYYY-MM-DD`. 8. **Caddyfile changes via admin API on `:2020`.** The Caddyfile is owned by root and matt has no passwordless sudo. Cutover used `curl -X POST .../load` on the Caddy admin port (which matt can hit), then a separate `sudo cp /home/matt/Caddyfile.new /etc/caddy/Caddyfile` step to persist the on-disk file. Future Caddyfile changes can follow the same pattern. Backup convention: `/etc/caddy/Caddyfile.bak.YYYY-MM-DD`.
9. **Path-naming.** Plan uses `inventory/` as the top-level (server-side path convention). Locally the equivalent is `inventory-server/`. Whenever the plan says `inventory/dashboard/foo/`, read that as `/var/www/inventory/dashboard/foo/` on the server or `inventory-server/dashboard/foo/` locally. 9. **Path-naming.** Plan uses `inventory/` as the top-level (server-side path convention). Locally the equivalent is `inventory-server/`. Whenever the plan says `inventory/dashboard/foo/`, read that as `/var/www/inventory/dashboard/foo/` on the server or `inventory-server/dashboard/foo/` locally.
10. **Scripts directory kept CJS via package.json shim.** Original plan called for converting "any spawned script" to ESM alongside its caller. Implemented: added `inventory-server/scripts/package.json` with `"type": "commonjs"`. Node's package-type resolution walks up directory by directory, so this overrides the parent's `"type": "module"` for the entire `scripts/` tree (≈15 files including `import/*.js`, `metrics-new/utils/*`, the orchestrator scripts) without renaming any file or touching any `spawn()` callsite. Convert individual scripts to ESM when touched; don't bulk-migrate.
11. **`src/routes/products.js` had dead multer setup.** Phase 6.7 spec called for hardening the upload route in products.js. There was no upload route — the `multer({ dest })` instance and `importProductsFromCSV` import were dead code left over from a long-ago migration. Strongest 6.7 hardening was deletion: no upload handler = no attack surface. The two real upload paths (`/api/import/upload-image` and `/api/reusable-images/upload`) got tightened MIME+extension allowlists instead.
12. **Two pre-existing syntax errors in shared/db/ surfaced.** `shared/db/pg.js:13` and `shared/db/redis.js:22` both had `?? Number(...) || N` — mixing `??` and `||` without parentheses is a TC39 syntax error. They passed Phase 2 because nothing imported them yet; Phase 3 smoke-test exposed it. Fixed with parens.
13. **`import { Pool } from 'pg'` doesn't work in ESM.** The `pg` package is CJS using `module.exports = { Pool, ... }`. Node's ESM-from-CJS interop fails to detect `Pool` as a named export via static analysis. The bulletproof pattern, now used everywhere: `import pg from 'pg'; const { Pool } = pg;`. Same idea for any future CJS-only deps. `src/utils/db.js` already had it; the two auth files needed the fix during execution.
14. **Frontend Bearer-header gap discovered (drives new Phase F1).** Phase 6 was specified assuming the frontend already sends `Authorization: Bearer` on every API call. It does not — only 7 of ~220 call sites do. Phase 6's `authenticate()` middleware is shipped and ready to enable, but until F1 lands the SPA will 401 on every page. The plan now has Phase F1 to address this explicitly; until then, the Phase 3+6 pm2 reload should not ship unless F1 ships in the same window.
15. **macOS NFS workflow note.** The `inventory-server/` directory locally is an NFS mount of `/var/www/inventory/` on netcup. Bulk operations (`find`/`grep -r`/mass `node --check`/`npm install`) hang or take minutes locally and pollute file listings with macOS AppleDouble `._*` sidecar files. Default to `ssh netcup` for any sweep across the tree — individual file edits via the editor are fine.
+23 -51
View File
@@ -1,103 +1,75 @@
require('dotenv').config({ path: '../.env' }); import bcrypt from 'bcrypt';
const bcrypt = require('bcrypt'); import pg from 'pg';
const { Pool } = require('pg'); import inquirer from 'inquirer';
const inquirer = require('inquirer');
// Log connection details for debugging (remove in production) const { Pool } = pg;
console.log('Attempting to connect with:', { import { config as loadEnv } from 'dotenv';
host: process.env.DB_HOST, import { fileURLToPath } from 'node:url';
user: process.env.DB_USER, import { dirname, resolve as resolvePath } from 'node:path';
database: process.env.DB_NAME,
port: process.env.DB_PORT const __filename = fileURLToPath(import.meta.url);
}); const __dirname = dirname(__filename);
loadEnv({ path: resolvePath(__dirname, '../.env') });
const pool = new Pool({ const pool = new Pool({
host: process.env.DB_HOST, host: process.env.DB_HOST,
user: process.env.DB_USER, user: process.env.DB_USER,
password: process.env.DB_PASSWORD, password: process.env.DB_PASSWORD,
database: process.env.DB_NAME, database: process.env.DB_NAME,
port: process.env.DB_PORT, port: Number(process.env.DB_PORT) || 5432,
}); });
async function promptUser() { async function promptUser() {
const questions = [ return inquirer.prompt([
{ {
type: 'input', type: 'input',
name: 'username', name: 'username',
message: 'Enter username:', message: 'Enter username:',
validate: (input) => { validate: (input) => input.length >= 3 || 'Username must be at least 3 characters long',
if (input.length < 3) {
return 'Username must be at least 3 characters long';
}
return true;
}
}, },
{ {
type: 'password', type: 'password',
name: 'password', name: 'password',
message: 'Enter password:', message: 'Enter password:',
mask: '*', mask: '*',
validate: (input) => { validate: (input) => input.length >= 8 || 'Password must be at least 8 characters long',
if (input.length < 8) {
return 'Password must be at least 8 characters long';
}
return true;
}
}, },
{ {
type: 'password', type: 'password',
name: 'confirmPassword', name: 'confirmPassword',
message: 'Confirm password:', message: 'Confirm password:',
mask: '*', mask: '*',
validate: (input, answers) => { validate: (input, answers) => input === answers.password || 'Passwords do not match',
if (input !== answers.password) { },
return 'Passwords do not match'; ]);
}
return true;
}
}
];
return inquirer.prompt(questions);
} }
async function addUser() { async function addUser() {
try { try {
// Get user input const { username, password } = await promptUser();
const answers = await promptUser(); const hashedPassword = await bcrypt.hash(password, 10);
const { username, password } = answers;
// Hash password
const saltRounds = 10;
const hashedPassword = await bcrypt.hash(password, saltRounds);
// Check if user already exists
const checkResult = await pool.query( const checkResult = await pool.query(
'SELECT id FROM users WHERE username = $1', 'SELECT id FROM users WHERE username = $1',
[username] [username]
); );
if (checkResult.rows.length > 0) { if (checkResult.rows.length > 0) {
console.error('Error: Username already exists'); console.error('Error: Username already exists');
process.exit(1); process.exit(1);
} }
// Insert new user
const result = await pool.query( const result = await pool.query(
'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id', 'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id',
[username, hashedPassword] [username, hashedPassword]
); );
console.log(`User ${username} created successfully with id ${result.rows[0].id}`); console.log(`User ${username} created successfully with id ${result.rows[0].id}`);
} catch (error) { } catch (error) {
console.error('Error creating user:', error); console.error('Error creating user:', error);
console.error('Error details:', error.message); if (error.code) console.error('Error code:', error.code);
if (error.code) {
console.error('Error code:', error.code);
}
} finally { } finally {
await pool.end(); await pool.end();
} }
} }
addUser(); addUser();
+7 -3
View File
@@ -2,18 +2,22 @@
"name": "inventory-auth-server", "name": "inventory-auth-server",
"version": "1.0.0", "version": "1.0.0",
"description": "Authentication server for inventory management system", "description": "Authentication server for inventory management system",
"type": "module",
"main": "server.js", "main": "server.js",
"scripts": { "scripts": {
"start": "node server.js" "start": "node server.js",
"add-user": "node add-user.js"
}, },
"dependencies": { "dependencies": {
"bcrypt": "^5.1.1", "bcrypt": "^5.1.1",
"cors": "^2.8.5", "cors": "^2.8.5",
"dotenv": "^16.4.7", "dotenv": "^16.4.7",
"express": "^4.18.2", "express": "^4.18.2",
"express-rate-limit": "^7.4.0",
"inquirer": "^8.2.6", "inquirer": "^8.2.6",
"jsonwebtoken": "^9.0.2", "jsonwebtoken": "^9.0.2",
"morgan": "^1.10.0", "pg": "^8.11.3",
"pg": "^8.11.3" "pino": "^9.5.0",
"pino-http": "^10.3.0"
} }
} }
+69 -124
View File
@@ -1,128 +1,73 @@
// Get pool from global or create a new one if not available export function createPermissionHelpers({ pool }) {
let pool; async function checkPermission(userId, permissionCode) {
if (typeof global.pool !== 'undefined') {
pool = global.pool;
} else {
// If global pool is not available, create a new connection
const { Pool } = require('pg');
pool = new Pool({
host: process.env.DB_HOST,
user: process.env.DB_USER,
password: process.env.DB_PASSWORD,
database: process.env.DB_NAME,
port: process.env.DB_PORT,
});
console.log('Created new database pool in permissions.js');
}
/**
* Check if a user has a specific permission
* @param {number} userId - The user ID to check
* @param {string} permissionCode - The permission code to check
* @returns {Promise<boolean>} - Whether the user has the permission
*/
async function checkPermission(userId, permissionCode) {
try {
// First check if the user is an admin
const adminResult = await pool.query(
'SELECT is_admin FROM users WHERE id = $1',
[userId]
);
// If user is admin, automatically grant permission
if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) {
return true;
}
// Otherwise check for specific permission
const result = await pool.query(
`SELECT COUNT(*) AS has_permission
FROM user_permissions up
JOIN permissions p ON up.permission_id = p.id
WHERE up.user_id = $1 AND p.code = $2`,
[userId, permissionCode]
);
return result.rows[0].has_permission > 0;
} catch (error) {
console.error('Error checking permission:', error);
return false;
}
}
/**
* Middleware to require a specific permission
* @param {string} permissionCode - The permission code required
* @returns {Function} - Express middleware function
*/
function requirePermission(permissionCode) {
return async (req, res, next) => {
try { try {
// Check if user is authenticated const adminResult = await pool.query(
if (!req.user || !req.user.id) { 'SELECT is_admin FROM users WHERE id = $1',
return res.status(401).json({ error: 'Authentication required' });
}
const hasPermission = await checkPermission(req.user.id, permissionCode);
if (!hasPermission) {
return res.status(403).json({
error: 'Insufficient permissions',
requiredPermission: permissionCode
});
}
next();
} catch (error) {
console.error('Permission middleware error:', error);
res.status(500).json({ error: 'Server error checking permissions' });
}
};
}
/**
* Get all permissions for a user
* @param {number} userId - The user ID
* @returns {Promise<string[]>} - Array of permission codes
*/
async function getUserPermissions(userId) {
try {
// Check if user is admin
const adminResult = await pool.query(
'SELECT is_admin FROM users WHERE id = $1',
[userId]
);
if (adminResult.rows.length === 0) {
return [];
}
const isAdmin = adminResult.rows[0].is_admin;
if (isAdmin) {
// Admin gets all permissions
const allPermissions = await pool.query('SELECT code FROM permissions');
return allPermissions.rows.map(p => p.code);
} else {
// Get assigned permissions
const permissions = await pool.query(
`SELECT p.code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1`,
[userId] [userId]
); );
if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) return true;
return permissions.rows.map(p => p.code);
}
} catch (error) {
console.error('Error getting user permissions:', error);
return [];
}
}
module.exports = { const result = await pool.query(
checkPermission, `SELECT COUNT(*) AS has_permission
requirePermission, FROM user_permissions up
getUserPermissions JOIN permissions p ON up.permission_id = p.id
}; WHERE up.user_id = $1 AND p.code = $2`,
[userId, permissionCode]
);
return Number(result.rows[0].has_permission) > 0;
} catch (error) {
console.error('Error checking permission:', error);
return false;
}
}
function requirePermission(permissionCode) {
return async (req, res, next) => {
try {
if (!req.user?.id) {
return res.status(401).json({ error: 'Authentication required' });
}
const hasPermission = await checkPermission(req.user.id, permissionCode);
if (!hasPermission) {
return res.status(403).json({
error: 'Insufficient permissions',
requiredPermission: permissionCode,
});
}
next();
} catch (error) {
console.error('Permission middleware error:', error);
res.status(500).json({ error: 'Server error checking permissions' });
}
};
}
async function getUserPermissions(userId) {
try {
const adminResult = await pool.query(
'SELECT is_admin FROM users WHERE id = $1',
[userId]
);
if (adminResult.rows.length === 0) return [];
if (adminResult.rows[0].is_admin) {
const allPermissions = await pool.query('SELECT code FROM permissions');
return allPermissions.rows.map((p) => p.code);
}
const permissions = await pool.query(
`SELECT p.code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1`,
[userId]
);
return permissions.rows.map((p) => p.code);
} catch (error) {
console.error('Error getting user permissions:', error);
return [];
}
}
return { checkPermission, requirePermission, getUserPermissions };
}
+299 -515
View File
@@ -1,533 +1,317 @@
const express = require('express'); import express from 'express';
const router = express.Router(); import bcrypt from 'bcrypt';
const bcrypt = require('bcrypt'); import jwt from 'jsonwebtoken';
const jwt = require('jsonwebtoken'); import { createPermissionHelpers } from './permissions.js';
const { requirePermission, getUserPermissions } = require('./permissions');
// Get pool from global or create a new one if not available export function createAuthRoutes({ pool }) {
let pool; const router = express.Router();
if (typeof global.pool !== 'undefined') { const { requirePermission, getUserPermissions } = createPermissionHelpers({ pool });
pool = global.pool;
} else { // Local authenticate(): used by user-management endpoints that need req.user populated
// If global pool is not available, create a new connection // with id/username/email/is_admin. NOT the per-service authenticate() — that lives in
const { Pool } = require('pg'); // shared/auth/middleware.js and is used by downstream services. Auth-server's surface is
pool = new Pool({ // small enough that a local copy is fine; the security boundary is the JWT verify step.
host: process.env.DB_HOST, async function authenticate(req, res, next) {
user: process.env.DB_USER, try {
password: process.env.DB_PASSWORD, const authHeader = req.headers.authorization;
database: process.env.DB_NAME, if (!authHeader || !authHeader.startsWith('Bearer ')) {
port: process.env.DB_PORT, return res.status(401).json({ error: 'Authentication required' });
}
const token = authHeader.split(' ')[1];
const decoded = jwt.verify(token, process.env.JWT_SECRET);
const result = await pool.query(
'SELECT id, username, email, is_admin, rocket_chat_user_id FROM users WHERE id = $1',
[decoded.userId]
);
if (result.rows.length === 0) {
return res.status(401).json({ error: 'User not found' });
}
req.user = result.rows[0];
next();
} catch (error) {
res.status(401).json({ error: 'Invalid token' });
}
}
router.post('/login', async (req, res) => {
try {
const { username, password } = req.body;
const result = await pool.query(
'SELECT id, username, password, is_admin, is_active, rocket_chat_user_id FROM users WHERE username = $1',
[username]
);
if (result.rows.length === 0) {
return res.status(401).json({ error: 'Invalid username or password' });
}
const user = result.rows[0];
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
const validPassword = await bcrypt.compare(password, user.password);
if (!validPassword) {
return res.status(401).json({ error: 'Invalid username or password' });
}
await pool.query(
'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
[user.id]
);
const token = jwt.sign(
{ userId: user.id, username: user.username },
process.env.JWT_SECRET,
{ expiresIn: '8h' }
);
const permissions = await getUserPermissions(user.id);
res.json({
token,
user: {
id: user.id,
username: user.username,
is_admin: user.is_admin,
rocket_chat_user_id: user.rocket_chat_user_id,
permissions,
},
});
} catch (error) {
console.error('Login error:', error);
res.status(500).json({ error: 'Server error' });
}
}); });
console.log('Created new database pool in routes.js');
}
// Authentication middleware router.get('/me', authenticate, async (req, res) => {
const authenticate = async (req, res, next) => { try {
try { const permissions = await getUserPermissions(req.user.id);
const authHeader = req.headers.authorization; res.json({
if (!authHeader || !authHeader.startsWith('Bearer ')) { id: req.user.id,
return res.status(401).json({ error: 'Authentication required' }); username: req.user.username,
email: req.user.email,
is_admin: req.user.is_admin,
rocket_chat_user_id: req.user.rocket_chat_user_id,
permissions,
});
} catch (error) {
console.error('Error getting current user:', error);
res.status(500).json({ error: 'Server error' });
} }
});
const token = authHeader.split(' ')[1]; router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
const decoded = jwt.verify(token, process.env.JWT_SECRET); try {
const result = await pool.query(`
// Get user from database SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
const result = await pool.query( FROM users
'SELECT id, username, email, is_admin, rocket_chat_user_id FROM users WHERE id = $1', ORDER BY username
[decoded.userId] `);
); res.json(result.rows);
} catch (error) {
console.log('Database query result for user', decoded.userId, ':', result.rows[0]); console.error('Error getting users:', error);
res.status(500).json({ error: 'Server error' });
if (result.rows.length === 0) {
return res.status(401).json({ error: 'User not found' });
} }
});
// Attach user to request
req.user = result.rows[0];
next();
} catch (error) {
console.error('Authentication error:', error);
res.status(401).json({ error: 'Invalid token' });
}
};
// Login route router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
router.post('/login', async (req, res) => { try {
try { const userId = req.params.id;
const { username, password } = req.body; const userResult = await pool.query(`
SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
// Get user from database FROM users
const result = await pool.query(
'SELECT id, username, password, is_admin, is_active, rocket_chat_user_id FROM users WHERE username = $1',
[username]
);
if (result.rows.length === 0) {
return res.status(401).json({ error: 'Invalid username or password' });
}
const user = result.rows[0];
// Check if user is active
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
// Verify password
const validPassword = await bcrypt.compare(password, user.password);
if (!validPassword) {
return res.status(401).json({ error: 'Invalid username or password' });
}
// Update last login
await pool.query(
'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
[user.id]
);
// Generate JWT
const token = jwt.sign(
{ userId: user.id, username: user.username },
process.env.JWT_SECRET,
{ expiresIn: '8h' }
);
// Get user permissions
const permissions = await getUserPermissions(user.id);
res.json({
token,
user: {
id: user.id,
username: user.username,
is_admin: user.is_admin,
rocket_chat_user_id: user.rocket_chat_user_id,
permissions
}
});
} catch (error) {
console.error('Login error:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Get current user
router.get('/me', authenticate, async (req, res) => {
try {
// Get user permissions
const permissions = await getUserPermissions(req.user.id);
res.json({
id: req.user.id,
username: req.user.username,
email: req.user.email,
is_admin: req.user.is_admin,
rocket_chat_user_id: req.user.rocket_chat_user_id,
permissions,
// Debug info
_debug_raw_user: req.user,
_server_identifier: "INVENTORY_AUTH_SERVER_MODIFIED"
});
} catch (error) {
console.error('Error getting current user:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Get all users
router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const result = await pool.query(`
SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
FROM users
ORDER BY username
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting users:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Get user with permissions
router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
try {
const userId = req.params.id;
// Get user details
const userResult = await pool.query(`
SELECT id, username, email, is_admin, is_active, rocket_chat_user_id, created_at, last_login
FROM users
WHERE id = $1
`, [userId]);
if (userResult.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
// Get user permissions
const permissionsResult = await pool.query(`
SELECT p.id, p.name, p.code, p.category, p.description
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
ORDER BY p.category, p.name
`, [userId]);
// Combine user and permissions
const user = {
...userResult.rows[0],
permissions: permissionsResult.rows
};
res.json(user);
} catch (error) {
console.error('Error getting user:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Create new user
router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
const client = await pool.connect();
try {
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
console.log("Create user request:", {
username,
email,
is_admin,
is_active,
rocket_chat_user_id,
permissions: permissions || []
});
// Validate required fields
if (!username || !password) {
return res.status(400).json({ error: 'Username and password are required' });
}
// Check if username is taken
const existingUser = await client.query(
'SELECT id FROM users WHERE username = $1',
[username]
);
if (existingUser.rows.length > 0) {
return res.status(400).json({ error: 'Username already exists' });
}
// Start transaction
await client.query('BEGIN');
// Hash password
const saltRounds = 10;
const hashedPassword = await bcrypt.hash(password, saltRounds);
// Insert new user
// Convert rocket_chat_user_id to integer if provided
const rcUserId = rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null;
const userResult = await client.query(`
INSERT INTO users (username, email, password, is_admin, is_active, rocket_chat_user_id, created_at)
VALUES ($1, $2, $3, $4, $5, $6, CURRENT_TIMESTAMP)
RETURNING id
`, [username, email || null, hashedPassword, !!is_admin, is_active !== false, rcUserId]);
const userId = userResult.rows[0].id;
// Assign permissions if provided and not admin
if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
console.log("Adding permissions for new user:", userId);
console.log("Permissions received:", permissions);
// Check permission format
const permissionIds = permissions.map(p => {
if (typeof p === 'object' && p.id) {
console.log("Permission is an object with ID:", p.id);
return parseInt(p.id, 10);
} else if (typeof p === 'number') {
console.log("Permission is a number:", p);
return p;
} else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
console.log("Permission is a string that can be parsed as a number:", p);
return parseInt(p, 10);
} else {
console.log("Unknown permission format:", typeof p, p);
// If it's a permission code, we need to look up the ID
return null;
}
}).filter(id => id !== null);
console.log("Filtered permission IDs:", permissionIds);
if (permissionIds.length > 0) {
const permissionValues = permissionIds
.map(permId => `(${userId}, ${permId})`)
.join(',');
console.log("Inserting permission values:", permissionValues);
try {
await client.query(`
INSERT INTO user_permissions (user_id, permission_id)
VALUES ${permissionValues}
ON CONFLICT DO NOTHING
`);
console.log("Successfully inserted permissions for new user:", userId);
} catch (err) {
console.error("Error inserting permissions for new user:", err);
throw err;
}
} else {
console.log("No valid permission IDs found to insert for new user");
}
} else {
console.log("Not adding permissions: is_admin =", is_admin, "permissions array:", Array.isArray(permissions), "length:", permissions ? permissions.length : 0);
}
await client.query('COMMIT');
res.status(201).json({
id: userId,
message: 'User created successfully'
});
} catch (error) {
await client.query('ROLLBACK');
console.error('Error creating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
// Update user
router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
const client = await pool.connect();
try {
const userId = req.params.id;
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
console.log("Update user request:", {
userId,
username,
email,
is_admin,
is_active,
rocket_chat_user_id,
permissions: permissions || []
});
// Check if user exists
const userExists = await client.query(
'SELECT id FROM users WHERE id = $1',
[userId]
);
if (userExists.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
// Start transaction
await client.query('BEGIN');
// Build update fields
const updateFields = [];
const updateValues = [userId]; // First parameter is the user ID
let paramIndex = 2;
if (username !== undefined) {
updateFields.push(`username = $${paramIndex++}`);
updateValues.push(username);
}
if (email !== undefined) {
updateFields.push(`email = $${paramIndex++}`);
updateValues.push(email || null);
}
if (is_admin !== undefined) {
updateFields.push(`is_admin = $${paramIndex++}`);
updateValues.push(!!is_admin);
}
if (is_active !== undefined) {
updateFields.push(`is_active = $${paramIndex++}`);
updateValues.push(!!is_active);
}
if (rocket_chat_user_id !== undefined) {
updateFields.push(`rocket_chat_user_id = $${paramIndex++}`);
// Convert to integer if not null/undefined, otherwise null
const rcUserId = rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null;
updateValues.push(rcUserId);
}
// Update password if provided
if (password) {
const saltRounds = 10;
const hashedPassword = await bcrypt.hash(password, saltRounds);
updateFields.push(`password = $${paramIndex++}`);
updateValues.push(hashedPassword);
}
// Update user if there are fields to update
if (updateFields.length > 0) {
updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
await client.query(`
UPDATE users
SET ${updateFields.join(', ')}
WHERE id = $1 WHERE id = $1
`, updateValues); `, [userId]);
if (userResult.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
const permissionsResult = await pool.query(`
SELECT p.id, p.name, p.code, p.category, p.description
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
ORDER BY p.category, p.name
`, [userId]);
res.json({
...userResult.rows[0],
permissions: permissionsResult.rows,
});
} catch (error) {
console.error('Error getting user:', error);
res.status(500).json({ error: 'Server error' });
} }
});
// Update permissions if provided
if (Array.isArray(permissions)) { router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
console.log("Updating permissions for user:", userId); const client = await pool.connect();
console.log("Permissions received:", permissions); try {
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
// First remove existing permissions if (!username || !password) {
await client.query( return res.status(400).json({ error: 'Username and password are required' });
'DELETE FROM user_permissions WHERE user_id = $1', }
const existingUser = await client.query(
'SELECT id FROM users WHERE username = $1',
[username]
);
if (existingUser.rows.length > 0) {
return res.status(400).json({ error: 'Username already exists' });
}
await client.query('BEGIN');
const hashedPassword = await bcrypt.hash(password, 10);
const rcUserId = rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null;
const userResult = await client.query(`
INSERT INTO users (username, email, password, is_admin, is_active, rocket_chat_user_id, created_at)
VALUES ($1, $2, $3, $4, $5, $6, CURRENT_TIMESTAMP)
RETURNING id
`, [username, email || null, hashedPassword, !!is_admin, is_active !== false, rcUserId]);
const userId = userResult.rows[0].id;
if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
const permissionIds = normalizePermissionIds(permissions);
if (permissionIds.length > 0) {
await client.query(
`INSERT INTO user_permissions (user_id, permission_id)
SELECT $1, unnest($2::int[])
ON CONFLICT DO NOTHING`,
[userId, permissionIds]
);
}
}
await client.query('COMMIT');
res.status(201).json({ id: userId, message: 'User created successfully' });
} catch (error) {
await client.query('ROLLBACK');
console.error('Error creating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
const client = await pool.connect();
try {
const userId = req.params.id;
const { username, email, password, is_admin, is_active, rocket_chat_user_id, permissions } = req.body;
const userExists = await client.query('SELECT id FROM users WHERE id = $1', [userId]);
if (userExists.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
await client.query('BEGIN');
const updateFields = [];
const updateValues = [userId];
let paramIndex = 2;
if (username !== undefined) { updateFields.push(`username = $${paramIndex++}`); updateValues.push(username); }
if (email !== undefined) { updateFields.push(`email = $${paramIndex++}`); updateValues.push(email || null); }
if (is_admin !== undefined) { updateFields.push(`is_admin = $${paramIndex++}`); updateValues.push(!!is_admin); }
if (is_active !== undefined) { updateFields.push(`is_active = $${paramIndex++}`); updateValues.push(!!is_active); }
if (rocket_chat_user_id !== undefined) {
updateFields.push(`rocket_chat_user_id = $${paramIndex++}`);
updateValues.push(rocket_chat_user_id ? parseInt(rocket_chat_user_id, 10) : null);
}
if (password) {
const hashedPassword = await bcrypt.hash(password, 10);
updateFields.push(`password = $${paramIndex++}`);
updateValues.push(hashedPassword);
}
if (updateFields.length > 0) {
updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
await client.query(`
UPDATE users SET ${updateFields.join(', ')} WHERE id = $1
`, updateValues);
}
if (Array.isArray(permissions)) {
await client.query('DELETE FROM user_permissions WHERE user_id = $1', [userId]);
const newIsAdmin = is_admin !== undefined
? is_admin
: (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
if (!newIsAdmin && permissions.length > 0) {
const permissionIds = normalizePermissionIds(permissions);
if (permissionIds.length > 0) {
await client.query(
`INSERT INTO user_permissions (user_id, permission_id)
SELECT $1, unnest($2::int[])
ON CONFLICT DO NOTHING`,
[userId, permissionIds]
);
}
}
}
await client.query('COMMIT');
res.json({ message: 'User updated successfully' });
} catch (error) {
await client.query('ROLLBACK');
console.error('Error updating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => {
try {
const userId = req.params.id;
if (req.user.id === parseInt(userId, 10)) {
return res.status(400).json({ error: 'Cannot delete your own account' });
}
const result = await pool.query(
'DELETE FROM users WHERE id = $1 RETURNING id',
[userId] [userId]
); );
console.log("Deleted existing permissions for user:", userId); if (result.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
// Add new permissions if any and not admin
const newIsAdmin = is_admin !== undefined ? is_admin : (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
console.log("User is admin:", newIsAdmin);
if (!newIsAdmin && permissions.length > 0) {
console.log("Adding permissions:", permissions);
// Check permission format
const permissionIds = permissions.map(p => {
if (typeof p === 'object' && p.id) {
console.log("Permission is an object with ID:", p.id);
return parseInt(p.id, 10);
} else if (typeof p === 'number') {
console.log("Permission is a number:", p);
return p;
} else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
console.log("Permission is a string that can be parsed as a number:", p);
return parseInt(p, 10);
} else {
console.log("Unknown permission format:", typeof p, p);
// If it's a permission code, we need to look up the ID
return null;
}
}).filter(id => id !== null);
console.log("Filtered permission IDs:", permissionIds);
if (permissionIds.length > 0) {
const permissionValues = permissionIds
.map(permId => `(${userId}, ${permId})`)
.join(',');
console.log("Inserting permission values:", permissionValues);
try {
await client.query(`
INSERT INTO user_permissions (user_id, permission_id)
VALUES ${permissionValues}
ON CONFLICT DO NOTHING
`);
console.log("Successfully inserted permissions for user:", userId);
} catch (err) {
console.error("Error inserting permissions:", err);
throw err;
}
} else {
console.log("No valid permission IDs found to insert");
}
} }
res.json({ message: 'User deleted successfully' });
} catch (error) {
console.error('Error deleting user:', error);
res.status(500).json({ error: 'Server error' });
} }
});
await client.query('COMMIT');
res.json({ message: 'User updated successfully' });
} catch (error) {
await client.query('ROLLBACK');
console.error('Error updating user:', error);
res.status(500).json({ error: 'Server error' });
} finally {
client.release();
}
});
// Delete user router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => {
router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => { try {
try { const result = await pool.query(`
const userId = req.params.id; SELECT category, json_agg(
json_build_object(
// Check that user is not deleting themselves 'id', id, 'name', name, 'code', code, 'description', description
if (req.user.id === parseInt(userId, 10)) { ) ORDER BY name
return res.status(400).json({ error: 'Cannot delete your own account' }); ) as permissions
FROM permissions
GROUP BY category
ORDER BY category
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
} }
});
// Delete user (this will cascade to user_permissions due to FK constraints)
const result = await pool.query( router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => {
'DELETE FROM users WHERE id = $1 RETURNING id', try {
[userId] const result = await pool.query(`
); SELECT * FROM permissions ORDER BY category, name
`);
if (result.rows.length === 0) { res.json(result.rows);
return res.status(404).json({ error: 'User not found' }); } catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
} }
});
res.json({ message: 'User deleted successfully' });
} catch (error) {
console.error('Error deleting user:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Get all permissions grouped by category return router;
router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => { }
try {
const result = await pool.query(`
SELECT category, json_agg(
json_build_object(
'id', id,
'name', name,
'code', code,
'description', description
) ORDER BY name
) as permissions
FROM permissions
GROUP BY category
ORDER BY category
`);
res.json(result.rows);
} catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
}
});
// Get all permissions function normalizePermissionIds(permissions) {
router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => { return permissions
try { .map((p) => {
const result = await pool.query(` if (typeof p === 'object' && p?.id) return parseInt(p.id, 10);
SELECT * if (typeof p === 'number') return p;
FROM permissions if (typeof p === 'string' && !Number.isNaN(parseInt(p, 10))) return parseInt(p, 10);
ORDER BY category, name return null;
`); })
.filter((id) => id !== null && !Number.isNaN(id));
res.json(result.rows); }
} catch (error) {
console.error('Error getting permissions:', error);
res.status(500).json({ error: 'Server error' });
}
});
module.exports = router;
+54 -165
View File
@@ -1,195 +1,84 @@
require('dotenv').config({ path: '../.env' }); import 'dotenv/config';
const express = require('express'); import express from 'express';
const cors = require('cors'); import cors from 'cors';
const bcrypt = require('bcrypt'); import pg from 'pg';
const jwt = require('jsonwebtoken'); import { fileURLToPath } from 'node:url';
const { Pool } = require('pg');
const morgan = require('morgan');
const authRoutes = require('./routes');
// Log startup configuration const { Pool } = pg;
console.log('Starting auth server with config:', { import { dirname, resolve as resolvePath } from 'node:path';
import { config as loadEnv } from 'dotenv';
import { corsOptions } from '../shared/cors/policy.js';
import { requestLog } from '../shared/logging/request-log.js';
import { logger } from '../shared/logging/logger.js';
import { errorHandler } from '../shared/errors/handler.js';
import { loginLimiter, verifyLimiter } from '../shared/rate-limit/login.js';
import { extractBearerToken, verifyToken, TokenError } from '../shared/auth/verify.js';
import { createAuthRoutes } from './routes.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);
// auth/ lives at inventory-server/auth/, so .env one level up
loadEnv({ path: resolvePath(__dirname, '../.env') });
if (!process.env.JWT_SECRET) {
logger.error('JWT_SECRET is not set; refusing to start');
process.exit(1);
}
logger.info({
host: process.env.DB_HOST, host: process.env.DB_HOST,
user: process.env.DB_USER, user: process.env.DB_USER,
database: process.env.DB_NAME, database: process.env.DB_NAME,
port: process.env.DB_PORT, port: process.env.DB_PORT,
auth_port: process.env.AUTH_PORT auth_port: process.env.AUTH_PORT,
}); }, 'starting auth server');
const app = express(); const app = express();
const port = process.env.AUTH_PORT || 3011; const port = Number(process.env.AUTH_PORT) || 3011;
// Database configuration
const pool = new Pool({ const pool = new Pool({
host: process.env.DB_HOST, host: process.env.DB_HOST,
user: process.env.DB_USER, user: process.env.DB_USER,
password: process.env.DB_PASSWORD, password: process.env.DB_PASSWORD,
database: process.env.DB_NAME, database: process.env.DB_NAME,
port: process.env.DB_PORT, port: Number(process.env.DB_PORT) || 5432,
}); });
// Make pool available globally app.use(requestLog());
global.pool = pool; app.use(express.json({ limit: '1mb' }));
app.use(cors(corsOptions));
// Middleware
app.use(express.json());
app.use(morgan('combined'));
app.use(cors({
origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw', 'https://acot.site', 'https://tools.acherryontop.com'],
credentials: true
}));
// Login endpoint
app.post('/login', async (req, res) => {
const { username, password } = req.body;
try {
// Get user from database
const result = await pool.query(
'SELECT id, username, password, is_admin, is_active FROM users WHERE username = $1',
[username]
);
const user = result.rows[0];
// Check if user exists and password is correct
if (!user || !(await bcrypt.compare(password, user.password))) {
return res.status(401).json({ error: 'Invalid username or password' });
}
// Check if user is active
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
// Update last login timestamp
await pool.query(
'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
[user.id]
);
// Generate JWT token
const token = jwt.sign(
{ userId: user.id, username: user.username },
process.env.JWT_SECRET,
{ expiresIn: '24h' }
);
// Get user permissions for the response
const permissionsResult = await pool.query(`
SELECT code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
`, [user.id]);
const permissions = permissionsResult.rows.map(row => row.code);
res.json({
token,
user: {
id: user.id,
username: user.username,
is_admin: user.is_admin,
permissions: user.is_admin ? [] : permissions
}
});
} catch (error) {
console.error('Login error:', error);
res.status(500).json({ error: 'Internal server error' });
}
});
// User info endpoint
app.get('/me', async (req, res) => {
const authHeader = req.headers.authorization;
if (!authHeader || !authHeader.startsWith('Bearer ')) {
return res.status(401).json({ error: 'No token provided' });
}
try {
const token = authHeader.split(' ')[1];
const decoded = jwt.verify(token, process.env.JWT_SECRET);
// Get user details from database
const userResult = await pool.query(
'SELECT id, username, email, is_admin, rocket_chat_user_id, is_active FROM users WHERE id = $1',
[decoded.userId]
);
if (userResult.rows.length === 0) {
return res.status(404).json({ error: 'User not found' });
}
const user = userResult.rows[0];
// Check if user is active
if (!user.is_active) {
return res.status(403).json({ error: 'Account is inactive' });
}
// Get user permissions
let permissions = [];
if (!user.is_admin) {
const permissionsResult = await pool.query(`
SELECT code
FROM permissions p
JOIN user_permissions up ON p.id = up.permission_id
WHERE up.user_id = $1
`, [user.id]);
permissions = permissionsResult.rows.map(row => row.code);
}
res.json({
id: user.id,
username: user.username,
email: user.email,
rocket_chat_user_id: user.rocket_chat_user_id,
is_admin: user.is_admin,
permissions: permissions
});
} catch (error) {
console.error('Token verification error:', error);
res.status(401).json({ error: 'Invalid token' });
}
});
// Caddy forward_auth target: JWT signature check only, no DB hit. // Caddy forward_auth target: JWT signature check only, no DB hit.
// Returns 200 with X-User-Id / X-User-Username on success, 401 otherwise. // Returns 200 with X-User-Id / X-User-Username on success; 401 otherwise.
// Per-service middleware re-verifies the token independently; these headers // Per-service middleware re-verifies independently; these headers are informational.
// are informational and must not be trusted by upstreams. app.all('/verify', verifyLimiter, (req, res) => {
app.all('/verify', (req, res) => {
const header = req.headers.authorization;
if (!header || !header.startsWith('Bearer ')) {
return res.status(401).json({ error: 'No token provided' });
}
try { try {
const decoded = jwt.verify(header.slice(7), process.env.JWT_SECRET); const token = extractBearerToken(req.headers.authorization);
const decoded = verifyToken(token, process.env.JWT_SECRET);
res.set('X-User-Id', String(decoded.userId)); res.set('X-User-Id', String(decoded.userId));
if (decoded.username) res.set('X-User-Username', decoded.username); if (decoded.username) res.set('X-User-Username', decoded.username);
res.status(200).end(); res.status(200).end();
} catch (err) { } catch (err) {
res.status(401).json({ error: err.name === 'TokenExpiredError' ? 'Token expired' : 'Invalid token' }); if (err instanceof TokenError) {
return res.status(401).json({ error: err.message });
}
res.status(401).json({ error: 'Invalid token' });
} }
}); });
// Mount all routes from routes.js // Login route gets its own rate limiter to slow credential stuffing.
app.use('/', authRoutes); app.use('/login', loginLimiter);
// Health check endpoint // Mount user-management + /login + /me from routes.js
app.get('/health', (req, res) => { app.use('/', createAuthRoutes({ pool }));
res.json({ status: 'healthy' });
});
// Error handling middleware app.get('/health', (req, res) => res.json({ status: 'healthy' }));
app.use((err, req, res, next) => {
console.error(err.stack); app.use(errorHandler);
res.status(500).json({ error: 'Something broke!' });
});
// Start server
app.listen(port, () => { app.listen(port, () => {
console.log(`Auth server running on port ${port}`); logger.info({ port }, 'auth server listening');
}); });
@@ -0,0 +1,87 @@
# Phase 6.1 + 6.6 + 6.7: tools.acherryontop.com final form
#
# Apply on the server with:
# curl -X POST http://localhost:2020/load \
# -H 'Content-Type: text/caddyfile' \
# --data-binary @/home/matt/Caddyfile.new
# sudo cp /home/matt/Caddyfile.new /etc/caddy/Caddyfile
# sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak.$(date +%F)
#
# Differences from current /etc/caddy/Caddyfile:
# 1. forward_auth gate added in front of /api/* and /chat-api/* (Phase 6.1).
# 2. /uploads/* moved behind the forward_auth gate (Phase 6.7 — was public).
# 3. LAN wildcards / Access-Control-Allow-Origin "*" defaults dropped from /api/* (Phase 6.6).
# 4. Removed dead /api/{aircall,gorgias,clarity}/* routes (Phase 1 — already cleaned up here).
#
# Phase 4 (dashboard-server merge) is NOT yet reflected — klaviyo/meta/google/typeform
# still route to their per-vendor PM2 apps in the live Caddyfile. Update those handle
# blocks to localhost:3015 when dashboard-server ships.
tools.acherryontop.com {
import security_headers
# Public: login endpoint
handle /auth-inv/* {
uri strip_prefix /auth-inv
reverse_proxy localhost:3011
}
# Public: static frontend assets (long-cache)
@static path *.js *.css *.png *.jpg *.jpeg *.gif *.ico *.svg *.woff *.woff2
handle @static {
header Cache-Control "public, max-age=2592000"
root * /var/www/inventory/frontend/build
file_server
}
# ----- Authenticated zone -----
# Phase 6.1: forward_auth subrequest to auth-server:/verify. 2xx → proceeds.
# 401/403 → Caddy returns auth-server response to client; backend never sees it.
@gated path /api/* /chat-api/* /uploads/*
handle @gated {
forward_auth localhost:3011 {
uri /verify
copy_headers Authorization
}
# Phase 6.7: /uploads/* now behind the gate (was a public file_server before)
handle /uploads/* {
root * /var/www/inventory
file_server
}
# Vendor dashboard routes
# NOTE: pre-Phase-4 these are still on separate ports; updates here when merged.
handle /api/klaviyo/* { reverse_proxy localhost:3015 }
handle /api/meta/* { reverse_proxy localhost:3015 }
handle /api/google-analytics/* { reverse_proxy localhost:3015 }
handle /api/typeform/* { reverse_proxy localhost:3015 }
# ACOT
handle /api/acot/* { reverse_proxy localhost:3012 }
# Chat
handle /chat-api/* {
uri strip_prefix /chat-api
reverse_proxy localhost:3014
}
# Catch-all: inventory-server
handle /api/* { reverse_proxy localhost:3010 }
}
# Out-of-band probes (unauthenticated)
handle /health { reverse_proxy localhost:3010 }
# SPA fallback (public assets)
handle {
root * /var/www/inventory/frontend/build
try_files {path} /index.html
file_server
encode gzip
}
handle_errors {
respond "{err.status_code} {err.status_text}"
}
}
+61
View File
@@ -0,0 +1,61 @@
# Server-side deployment artefacts for Phase 3 + 6
This directory contains proposed versions of files that live outside the
inventory-server tree on production. Each is a recommendation — apply
deliberately and only after the Node-side ESM + auth changes are deployed and
smoke-tested.
| Source | Target | Phase |
| --------------------------------------- | ------------------------------------- | -------- |
| `Caddyfile.proposed` | `/etc/caddy/Caddyfile` | 6.1, 6.6, 6.7 |
| `ecosystem.config.cjs.proposed` | `/var/www/ecosystem.config.cjs` | 6.4, 6.10 |
## Recommended apply order
1. **Deploy the Node code first** (this repo). PM2 reload picks up the new
ESM-mode inventory-server and auth-server. At this point the frontend will
start hitting 401s on every API call because the new `authenticate()`
middleware is live and the frontend doesn't carry Bearer tokens on most
fetches. **This is expected per the discussion in CONSOLIDATION_PLAN.md
§6** — the frontend fetch-wrapper work is the next deliverable.
2. **Apply the ecosystem.cjs change** (Phase 6.4) to fix the `JWT_SECRET`
shadow-override before the next pm2 restart silently re-introduces it.
3. **Apply the Caddyfile change** (Phase 6.1) only after the frontend is
sending Bearer tokens. Until then, `forward_auth` will reject every page
refresh at the edge.
## Caddyfile apply pattern
Caddy admin API is on `:2020` (matt has access). On-disk file needs root.
```bash
# Upload + load atomically into the running Caddy
curl -X POST http://localhost:2020/load \
-H 'Content-Type: text/caddyfile' \
--data-binary @/home/matt/Caddyfile.new
# Persist to disk (separate sudo step)
sudo cp /etc/caddy/Caddyfile /etc/caddy/Caddyfile.bak.$(date +%F)
sudo cp /home/matt/Caddyfile.new /etc/caddy/Caddyfile
```
## ecosystem.cjs apply pattern
```bash
sudo cp /var/www/ecosystem.config.cjs /var/www/ecosystem.config.cjs.bak.$(date +%F)
sudo cp /home/matt/ecosystem.config.cjs.new /var/www/ecosystem.config.cjs
pm2 reload ecosystem.config.cjs --update-env
pm2 env new-auth-server | grep -i jwt # JWT_SECRET from .env only
```
## Rollback
Every applied file leaves a `.bak.YYYY-MM-DD` next to it. `sudo cp <bak>
<original>` then `caddy reload` / `pm2 reload`.
Phase 6 changes are *additive* — if `forward_auth` causes problems, comment
out the directive in the live Caddyfile and per-server middleware
(`authenticate()` in inventory-server, in particular) continues protecting
routes.
@@ -0,0 +1,114 @@
// Phase 6.4 + 6.10: proposed /var/www/ecosystem.config.cjs
//
// Diffs from the live file (CONSOLIDATION_PLAN.md §6.4 / §6.10):
// 1. Drop the `JWT_SECRET: process.env.JWT_SECRET` override in new-auth-server's
// env block. That override shadowed the .env value with whatever shell var was
// exported when pm2 was last started — causing the silent-divergence footgun
// called out in CLAUDE.md memory. With it removed, .env is the single source.
// 2. Move ADD_WORD_TOKEN out of the inline env into /opt/lt-wordlist-api/.env.
// The PM2 entry below no longer references it.
// 3. Rename the placeholder script paths if you have any half-finished ones; this
// version only lists the apps that actually run today (post-Phase-1 cleanup).
//
// To apply:
// sudo cp /var/www/ecosystem.config.cjs /var/www/ecosystem.config.cjs.bak.$(date +%F)
// sudo cp /home/matt/ecosystem.config.cjs.new /var/www/ecosystem.config.cjs
// pm2 reload ecosystem.config.cjs --update-env
//
// Verify after reload:
// pm2 env new-auth-server | grep -i jwt # should show JWT_SECRET from .env only
// pm2 env new-auth-server | grep ADD_WORD # should be empty
const inventoryEnv = require('dotenv').config({ path: '/var/www/inventory/.env' }).parsed;
module.exports = {
apps: [
{
name: 'new-auth-server', // Phase 8 may rename to 'auth-server' — cosmetic
script: './inventory/auth/server.js',
cwd: '/var/www',
env: {
...inventoryEnv,
NODE_ENV: 'production',
AUTH_PORT: 3011,
// PHASE 6.4 FIX: no JWT_SECRET override here. .env wins.
},
max_memory_restart: '500M',
error_file: '/var/log/pm2/new-auth-server-error.log',
out_file: '/var/log/pm2/new-auth-server-out.log',
},
{
name: 'inventory-server',
script: './inventory/src/server.js',
cwd: '/var/www',
env: {
...inventoryEnv,
NODE_ENV: 'production',
PORT: 3010,
UPLOADS_DIR: '/var/www/inventory/uploads',
},
max_memory_restart: '1G',
error_file: '/var/log/pm2/inventory-server-error.log',
out_file: '/var/log/pm2/inventory-server-out.log',
},
{
name: 'chat-server',
script: './inventory/chat/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production', PORT: 3014 },
max_memory_restart: '500M',
},
{
name: 'acot-server',
script: './inventory/dashboard/acot-server/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production', ACOT_PORT: 3012 },
max_memory_restart: '1G',
},
// Per-vendor dashboard apps stay until Phase 4 merge ships.
{
name: 'klaviyo-server',
script: './inventory/dashboard/klaviyo-server/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production' },
max_memory_restart: '500M',
},
{
name: 'meta-server',
script: './inventory/dashboard/meta-server/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production' },
max_memory_restart: '300M',
},
{
name: 'google-server',
script: './inventory/dashboard/google-server/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production' },
max_memory_restart: '300M',
},
{
name: 'typeform-server',
script: './inventory/dashboard/typeform-server/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production' },
max_memory_restart: '300M',
},
// PHASE 6.10: lt-wordlist-api now loads ADD_WORD_TOKEN from /opt/lt-wordlist-api/.env
// (no longer hardcoded here). Rotate that token's value when applying this change.
{
name: 'lt-wordlist-api',
script: '/opt/lt-wordlist-api/server.js',
cwd: '/opt/lt-wordlist-api',
env: { NODE_ENV: 'production' },
max_memory_restart: '200M',
},
{
name: 'acot-phone-server',
script: './inventory/acot-phone/server.js',
cwd: '/var/www',
env: { ...inventoryEnv, NODE_ENV: 'production' },
max_memory_restart: '300M',
},
],
};
@@ -0,0 +1,52 @@
-- Phase 6.2: per-route permission codes
-- Seeds the permission codes referenced by Phase 6 hardening middleware.
-- Safe to run multiple times (ON CONFLICT DO NOTHING).
--
-- Codes follow the plan's spec (CONSOLIDATION_PLAN.md §6.2):
-- product_import — POST/PUT/DELETE on /api/import
-- data_management — POST/PUT/DELETE on /api/csv (data-management.js)
-- ai_admin — POST/PUT/DELETE on /api/ai-prompts, /api/ai-validation
-- templates_write — POST/PUT/DELETE on /api/templates
-- image_admin — POST/DELETE on /api/reusable-images
-- audit_read — reserved for future read-gating on audit logs
-- acot_admin — reserved for acot-server (Phase 5 scope)
-- klaviyo_* / meta_* / google_* / typeform_* — reserved for dashboard-server (Phase 4 scope)
--
-- Admin users (is_admin = true) automatically pass any requirePermission() check,
-- so this migration does not auto-grant codes to admins. New non-admin users get
-- write access only when explicitly granted via the user-management UI.
INSERT INTO permissions (code, name, category, description) VALUES
('product_import', 'Product Import (write)', 'Imports',
'Allows POST/PUT/DELETE on /api/import — uploads, deletes, generate-upc, etc.'),
('data_management', 'Data Management (write)', 'Data',
'Allows POST/PUT/DELETE on /api/csv — CSV operations, full updates, full resets.'),
('ai_admin', 'AI Settings Admin', 'AI',
'Allows write access to AI prompts and AI validation endpoints.'),
('templates_write', 'Template Editing', 'Templates',
'Allows POST/PUT/DELETE on /api/templates.'),
('image_admin', 'Image Management', 'Images',
'Allows uploads and deletions on /api/reusable-images.'),
('audit_read', 'Audit Log Access', 'Audit',
'Reserved for future read-gating of import + product-editor audit logs.'),
('klaviyo_write', 'Klaviyo Write', 'Dashboard',
'Reserved for dashboard-server: mutates Klaviyo lists/segments.'),
('klaviyo_admin', 'Klaviyo Admin', 'Dashboard',
'Reserved for dashboard-server: triggers campaign syncs.'),
('meta_write', 'Meta Write', 'Dashboard',
'Reserved for dashboard-server: Meta API write operations.'),
('google_write', 'Google Analytics Write', 'Dashboard',
'Reserved for dashboard-server: GA write operations.'),
('typeform_write', 'Typeform Write', 'Dashboard',
'Reserved for dashboard-server: Typeform write operations.'),
('acot_admin', 'ACOT Server Admin', 'ACOT',
'Reserved for acot-server admin endpoints.')
ON CONFLICT (code) DO NOTHING;
-- Phase 2 deviation #6 cleanup: drop defunct frontend permissions if present.
-- These corresponded to the removed Aircall/Gorgias dashboards.
DELETE FROM user_permissions
WHERE permission_id IN (
SELECT id FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls')
);
DELETE FROM permissions WHERE code IN ('dashboard:gorgias', 'dashboard:calls');
+367
View File
@@ -18,10 +18,15 @@
"diff": "^7.0.0", "diff": "^7.0.0",
"dotenv": "^16.4.7", "dotenv": "^16.4.7",
"express": "^4.18.2", "express": "^4.18.2",
"express-rate-limit": "^7.4.0",
"ioredis": "^5.10.1",
"jsonwebtoken": "^9.0.2",
"multer": "^1.4.5-lts.1", "multer": "^1.4.5-lts.1",
"mysql2": "^3.12.0", "mysql2": "^3.12.0",
"openai": "^6.0.0", "openai": "^6.0.0",
"pg": "^8.14.1", "pg": "^8.14.1",
"pino": "^9.5.0",
"pino-http": "^10.3.0",
"pm2": "^5.3.0", "pm2": "^5.3.0",
"sharp": "^0.33.5", "sharp": "^0.33.5",
"ssh2": "^1.16.0", "ssh2": "^1.16.0",
@@ -409,6 +414,12 @@
"url": "https://opencollective.com/libvips" "url": "https://opencollective.com/libvips"
} }
}, },
"node_modules/@ioredis/commands": {
"version": "1.5.1",
"resolved": "https://registry.npmjs.org/@ioredis/commands/-/commands-1.5.1.tgz",
"integrity": "sha512-JH8ZL/ywcJyR9MmJ5BNqZllXNZQqQbnVZOqpPQqE1vHiFgAw4NHbvE0FOduNU8IX9babitBT46571OnPTT0Zcw==",
"license": "MIT"
},
"node_modules/@mapbox/node-pre-gyp": { "node_modules/@mapbox/node-pre-gyp": {
"version": "1.0.11", "version": "1.0.11",
"resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz", "resolved": "https://registry.npmjs.org/@mapbox/node-pre-gyp/-/node-pre-gyp-1.0.11.tgz",
@@ -477,6 +488,12 @@
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/@pinojs/redact": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/@pinojs/redact/-/redact-0.4.0.tgz",
"integrity": "sha512-k2ENnmBugE/rzQfEcdWHcCY+/FM3VLzH9cYEsbdsoqrvzAKRhUZeRNhAZvB8OitQJ1TBed3yqWtdjzS6wJKBwg==",
"license": "MIT"
},
"node_modules/@pm2/agent": { "node_modules/@pm2/agent": {
"version": "2.0.4", "version": "2.0.4",
"resolved": "https://registry.npmjs.org/@pm2/agent/-/agent-2.0.4.tgz", "resolved": "https://registry.npmjs.org/@pm2/agent/-/agent-2.0.4.tgz",
@@ -958,6 +975,15 @@
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/atomic-sleep": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/atomic-sleep/-/atomic-sleep-1.0.0.tgz",
"integrity": "sha512-kNOjDqAh7px0XWNI+4QbzoiR/nTkHAWNud2uvnJquD1/x5a7EQZMJT0AczqK0Qn67oY/TTQ1LbUKajZpp3I9tQ==",
"license": "MIT",
"engines": {
"node": ">=8.0.0"
}
},
"node_modules/aws-ssl-profiles": { "node_modules/aws-ssl-profiles": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/aws-ssl-profiles/-/aws-ssl-profiles-1.1.2.tgz", "resolved": "https://registry.npmjs.org/aws-ssl-profiles/-/aws-ssl-profiles-1.1.2.tgz",
@@ -1092,6 +1118,12 @@
"node": ">=8" "node": ">=8"
} }
}, },
"node_modules/buffer-equal-constant-time": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/buffer-equal-constant-time/-/buffer-equal-constant-time-1.0.1.tgz",
"integrity": "sha512-zRpUiDwd/xk6ADqPMATG8vc9VPrkck7T07OIx0gnjmJAnHnTVXNQG3vfvWNuiZIkwu9KrKdA1iJKfsfTVxE6NA==",
"license": "BSD-3-Clause"
},
"node_modules/buffer-from": { "node_modules/buffer-from": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz", "resolved": "https://registry.npmjs.org/buffer-from/-/buffer-from-1.1.2.tgz",
@@ -1219,6 +1251,15 @@
"node": ">=8.10.0" "node": ">=8.10.0"
} }
}, },
"node_modules/cluster-key-slot": {
"version": "1.1.2",
"resolved": "https://registry.npmjs.org/cluster-key-slot/-/cluster-key-slot-1.1.2.tgz",
"integrity": "sha512-RMr0FhtfXemyinomL4hrWcYJxmX6deFdCxpJzhDttxgO1+bcCnkk+9drydLVDmAMG7NE6aN/fl4F7ucU/90gAA==",
"license": "Apache-2.0",
"engines": {
"node": ">=0.10.0"
}
},
"node_modules/color": { "node_modules/color": {
"version": "4.2.3", "version": "4.2.3",
"resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz", "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz",
@@ -1529,6 +1570,15 @@
"node": ">= 0.4" "node": ">= 0.4"
} }
}, },
"node_modules/ecdsa-sig-formatter": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz",
"integrity": "sha512-nagl3RYrbNv6kQkeJIpt6NJZy8twLB/2vtz6yN9Z4vRKHN4/QZJIEbqohALSgwKdnksuY3k5Addp5lg8sVoVcQ==",
"license": "Apache-2.0",
"dependencies": {
"safe-buffer": "^5.0.1"
}
},
"node_modules/ee-first": { "node_modules/ee-first": {
"version": "1.1.1", "version": "1.1.1",
"resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz", "resolved": "https://registry.npmjs.org/ee-first/-/ee-first-1.1.1.tgz",
@@ -1738,6 +1788,21 @@
"url": "https://opencollective.com/express" "url": "https://opencollective.com/express"
} }
}, },
"node_modules/express-rate-limit": {
"version": "7.5.1",
"resolved": "https://registry.npmjs.org/express-rate-limit/-/express-rate-limit-7.5.1.tgz",
"integrity": "sha512-7iN8iPMDzOMHPUYllBEsQdWVB6fPDMPqwjBaFrgr4Jgr/+okjvzAy+UHlYYL/Vs0OsOrMkwS6PJDkFlJwoxUnw==",
"license": "MIT",
"engines": {
"node": ">= 16"
},
"funding": {
"url": "https://github.com/sponsors/express-rate-limit"
},
"peerDependencies": {
"express": ">= 4.11"
}
},
"node_modules/extrareqp2": { "node_modules/extrareqp2": {
"version": "1.0.0", "version": "1.0.0",
"resolved": "https://registry.npmjs.org/extrareqp2/-/extrareqp2-1.0.0.tgz", "resolved": "https://registry.npmjs.org/extrareqp2/-/extrareqp2-1.0.0.tgz",
@@ -1912,6 +1977,15 @@
"is-property": "^1.0.2" "is-property": "^1.0.2"
} }
}, },
"node_modules/get-caller-file": {
"version": "2.0.5",
"resolved": "https://registry.npmjs.org/get-caller-file/-/get-caller-file-2.0.5.tgz",
"integrity": "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg==",
"license": "ISC",
"engines": {
"node": "6.* || 8.* || >= 10.*"
}
},
"node_modules/get-intrinsic": { "node_modules/get-intrinsic": {
"version": "1.2.7", "version": "1.2.7",
"resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.2.7.tgz", "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.2.7.tgz",
@@ -2227,6 +2301,53 @@
"integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==",
"license": "ISC" "license": "ISC"
}, },
"node_modules/ioredis": {
"version": "5.10.1",
"resolved": "https://registry.npmjs.org/ioredis/-/ioredis-5.10.1.tgz",
"integrity": "sha512-HuEDBTI70aYdx1v6U97SbNx9F1+svQKBDo30o0b9fw055LMepzpOOd0Ccg9Q6tbqmBSJaMuY0fB7yw9/vjBYCA==",
"license": "MIT",
"dependencies": {
"@ioredis/commands": "1.5.1",
"cluster-key-slot": "^1.1.0",
"debug": "^4.3.4",
"denque": "^2.1.0",
"lodash.defaults": "^4.2.0",
"lodash.isarguments": "^3.1.0",
"redis-errors": "^1.2.0",
"redis-parser": "^3.0.0",
"standard-as-callback": "^2.1.0"
},
"engines": {
"node": ">=12.22.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/ioredis"
}
},
"node_modules/ioredis/node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",
"integrity": "sha512-RGwwWnwQvkVfavKVt22FGLw+xYSdzARwm0ru6DhTVA3umU5hZc28V3kO4stgYryrTlLpuvgI9GiijltAjNbcqA==",
"license": "MIT",
"dependencies": {
"ms": "^2.1.3"
},
"engines": {
"node": ">=6.0"
},
"peerDependenciesMeta": {
"supports-color": {
"optional": true
}
}
},
"node_modules/ioredis/node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"license": "MIT"
},
"node_modules/ip-address": { "node_modules/ip-address": {
"version": "9.0.5", "version": "9.0.5",
"resolved": "https://registry.npmjs.org/ip-address/-/ip-address-9.0.5.tgz", "resolved": "https://registry.npmjs.org/ip-address/-/ip-address-9.0.5.tgz",
@@ -2376,6 +2497,55 @@
"license": "ISC", "license": "ISC",
"optional": true "optional": true
}, },
"node_modules/jsonwebtoken": {
"version": "9.0.3",
"resolved": "https://registry.npmjs.org/jsonwebtoken/-/jsonwebtoken-9.0.3.tgz",
"integrity": "sha512-MT/xP0CrubFRNLNKvxJ2BYfy53Zkm++5bX9dtuPbqAeQpTVe0MQTFhao8+Cp//EmJp244xt6Drw/GVEGCUj40g==",
"license": "MIT",
"dependencies": {
"jws": "^4.0.1",
"lodash.includes": "^4.3.0",
"lodash.isboolean": "^3.0.3",
"lodash.isinteger": "^4.0.4",
"lodash.isnumber": "^3.0.3",
"lodash.isplainobject": "^4.0.6",
"lodash.isstring": "^4.0.1",
"lodash.once": "^4.0.0",
"ms": "^2.1.1",
"semver": "^7.5.4"
},
"engines": {
"node": ">=12",
"npm": ">=6"
}
},
"node_modules/jsonwebtoken/node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"license": "MIT"
},
"node_modules/jwa": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/jwa/-/jwa-2.0.1.tgz",
"integrity": "sha512-hRF04fqJIP8Abbkq5NKGN0Bbr3JxlQ+qhZufXVr0DvujKy93ZCbXZMHDL4EOtodSbCWxOqR8MS1tXA5hwqCXDg==",
"license": "MIT",
"dependencies": {
"buffer-equal-constant-time": "^1.0.1",
"ecdsa-sig-formatter": "1.0.11",
"safe-buffer": "^5.0.1"
}
},
"node_modules/jws": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/jws/-/jws-4.0.1.tgz",
"integrity": "sha512-EKI/M/yqPncGUUh44xz0PxSidXFr/+r0pA70+gIYhjv+et7yxM+s29Y+VGDkovRofQem0fs7Uvf4+YmAdyRduA==",
"license": "MIT",
"dependencies": {
"jwa": "^2.0.1",
"safe-buffer": "^5.0.1"
}
},
"node_modules/lazy": { "node_modules/lazy": {
"version": "1.0.11", "version": "1.0.11",
"resolved": "https://registry.npmjs.org/lazy/-/lazy-1.0.11.tgz", "resolved": "https://registry.npmjs.org/lazy/-/lazy-1.0.11.tgz",
@@ -2391,6 +2561,60 @@
"integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==", "integrity": "sha512-v2kDEe57lecTulaDIuNTPy3Ry4gLGJ6Z1O3vE1krgXZNrsQ+LFTGHVxVjcXPs17LhbZVGedAJv8XZ1tvj5FvSg==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/lodash.defaults": {
"version": "4.2.0",
"resolved": "https://registry.npmjs.org/lodash.defaults/-/lodash.defaults-4.2.0.tgz",
"integrity": "sha512-qjxPLHd3r5DnsdGacqOMU6pb/avJzdh9tFX2ymgoZE27BmjXrNy/y4LoaiTeAb+O3gL8AfpJGtqfX/ae2leYYQ==",
"license": "MIT"
},
"node_modules/lodash.includes": {
"version": "4.3.0",
"resolved": "https://registry.npmjs.org/lodash.includes/-/lodash.includes-4.3.0.tgz",
"integrity": "sha512-W3Bx6mdkRTGtlJISOvVD/lbqjTlPPUDTMnlXZFnVwi9NKJ6tiAk6LVdlhZMm17VZisqhKcgzpO5Wz91PCt5b0w==",
"license": "MIT"
},
"node_modules/lodash.isarguments": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz",
"integrity": "sha512-chi4NHZlZqZD18a0imDHnZPrDeBbTtVN7GXMwuGdRH9qotxAjYs3aVLKc7zNOG9eddR5Ksd8rvFEBc9SsggPpg==",
"license": "MIT"
},
"node_modules/lodash.isboolean": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/lodash.isboolean/-/lodash.isboolean-3.0.3.tgz",
"integrity": "sha512-Bz5mupy2SVbPHURB98VAcw+aHh4vRV5IPNhILUCsOzRmsTmSQ17jIuqopAentWoehktxGd9e/hbIXq980/1QJg==",
"license": "MIT"
},
"node_modules/lodash.isinteger": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/lodash.isinteger/-/lodash.isinteger-4.0.4.tgz",
"integrity": "sha512-DBwtEWN2caHQ9/imiNeEA5ys1JoRtRfY3d7V9wkqtbycnAmTvRRmbHKDV4a0EYc678/dia0jrte4tjYwVBaZUA==",
"license": "MIT"
},
"node_modules/lodash.isnumber": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/lodash.isnumber/-/lodash.isnumber-3.0.3.tgz",
"integrity": "sha512-QYqzpfwO3/CWf3XP+Z+tkQsfaLL/EnUlXWVkIk5FUPc4sBdTehEqZONuyRt2P67PXAk+NXmTBcc97zw9t1FQrw==",
"license": "MIT"
},
"node_modules/lodash.isplainobject": {
"version": "4.0.6",
"resolved": "https://registry.npmjs.org/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz",
"integrity": "sha512-oSXzaWypCMHkPC3NvBEaPHf0KsA5mvPrOPgQWDsbg8n7orZ290M0BmC/jgRZ4vcJ6DTAhjrsSYgdsW/F+MFOBA==",
"license": "MIT"
},
"node_modules/lodash.isstring": {
"version": "4.0.1",
"resolved": "https://registry.npmjs.org/lodash.isstring/-/lodash.isstring-4.0.1.tgz",
"integrity": "sha512-0wJxfxH1wgO3GrbuP+dTTk7op+6L41QCXbGINEmD+ny/G/eCqGzxyCsh7159S+mgDDcoarnBw6PC1PS5+wUGgw==",
"license": "MIT"
},
"node_modules/lodash.once": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/lodash.once/-/lodash.once-4.1.1.tgz",
"integrity": "sha512-Sb487aTOCr9drQVL8pIxOzVhafOjZN9UU54hiN8PU3uAiSV7lx1yYNpbNmex2PK6dSJoNTSJUUswT651yww3Mg==",
"license": "MIT"
},
"node_modules/long": { "node_modules/long": {
"version": "5.2.3", "version": "5.2.3",
"resolved": "https://registry.npmjs.org/long/-/long-5.2.3.tgz", "resolved": "https://registry.npmjs.org/long/-/long-5.2.3.tgz",
@@ -2910,6 +3134,15 @@
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
}, },
"node_modules/on-exit-leak-free": {
"version": "2.1.2",
"resolved": "https://registry.npmjs.org/on-exit-leak-free/-/on-exit-leak-free-2.1.2.tgz",
"integrity": "sha512-0eJJY6hXLGf1udHwfNftBqH+g73EU4B504nZeKpz1sYRKafAghwxEJunB2O7rDZkL4PGfsMVnTXZ2EjibbqcsA==",
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/on-finished": { "node_modules/on-finished": {
"version": "2.4.1", "version": "2.4.1",
"resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz", "resolved": "https://registry.npmjs.org/on-finished/-/on-finished-2.4.1.tgz",
@@ -3156,6 +3389,55 @@
"node": ">=10" "node": ">=10"
} }
}, },
"node_modules/pino": {
"version": "9.14.0",
"resolved": "https://registry.npmjs.org/pino/-/pino-9.14.0.tgz",
"integrity": "sha512-8OEwKp5juEvb/MjpIc4hjqfgCNysrS94RIOMXYvpYCdm/jglrKEiAYmiumbmGhCvs+IcInsphYDFwqrjr7398w==",
"license": "MIT",
"dependencies": {
"@pinojs/redact": "^0.4.0",
"atomic-sleep": "^1.0.0",
"on-exit-leak-free": "^2.1.0",
"pino-abstract-transport": "^2.0.0",
"pino-std-serializers": "^7.0.0",
"process-warning": "^5.0.0",
"quick-format-unescaped": "^4.0.3",
"real-require": "^0.2.0",
"safe-stable-stringify": "^2.3.1",
"sonic-boom": "^4.0.1",
"thread-stream": "^3.0.0"
},
"bin": {
"pino": "bin.js"
}
},
"node_modules/pino-abstract-transport": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/pino-abstract-transport/-/pino-abstract-transport-2.0.0.tgz",
"integrity": "sha512-F63x5tizV6WCh4R6RHyi2Ml+M70DNRXt/+HANowMflpgGFMAym/VKm6G7ZOQRjqN7XbGxK1Lg9t6ZrtzOaivMw==",
"license": "MIT",
"dependencies": {
"split2": "^4.0.0"
}
},
"node_modules/pino-http": {
"version": "10.5.0",
"resolved": "https://registry.npmjs.org/pino-http/-/pino-http-10.5.0.tgz",
"integrity": "sha512-hD91XjgaKkSsdn8P7LaebrNzhGTdB086W3pyPihX0EzGPjq5uBJBXo4N5guqNaK6mUjg9aubMF7wDViYek9dRA==",
"license": "MIT",
"dependencies": {
"get-caller-file": "^2.0.5",
"pino": "^9.0.0",
"pino-std-serializers": "^7.0.0",
"process-warning": "^5.0.0"
}
},
"node_modules/pino-std-serializers": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/pino-std-serializers/-/pino-std-serializers-7.1.0.tgz",
"integrity": "sha512-BndPH67/JxGExRgiX1dX0w1FvZck5Wa4aal9198SrRhZjH3GxKQUKIBnYJTdj2HDN3UQAS06HlfcSbQj2OHmaw==",
"license": "MIT"
},
"node_modules/pm2": { "node_modules/pm2": {
"version": "5.4.3", "version": "5.4.3",
"resolved": "https://registry.npmjs.org/pm2/-/pm2-5.4.3.tgz", "resolved": "https://registry.npmjs.org/pm2/-/pm2-5.4.3.tgz",
@@ -3426,6 +3708,22 @@
"integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==", "integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/process-warning": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/process-warning/-/process-warning-5.0.0.tgz",
"integrity": "sha512-a39t9ApHNx2L4+HBnQKqxxHNs1r7KF+Intd8Q/g1bUh6q0WIp9voPXJ/x0j+ZL45KF1pJd9+q2jLIRMfvEshkA==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/fastify"
},
{
"type": "opencollective",
"url": "https://opencollective.com/fastify"
}
],
"license": "MIT"
},
"node_modules/promptly": { "node_modules/promptly": {
"version": "2.2.0", "version": "2.2.0",
"resolved": "https://registry.npmjs.org/promptly/-/promptly-2.2.0.tgz", "resolved": "https://registry.npmjs.org/promptly/-/promptly-2.2.0.tgz",
@@ -3518,6 +3816,12 @@
"url": "https://github.com/sponsors/ljharb" "url": "https://github.com/sponsors/ljharb"
} }
}, },
"node_modules/quick-format-unescaped": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/quick-format-unescaped/-/quick-format-unescaped-4.0.4.tgz",
"integrity": "sha512-tYC1Q1hgyRuHgloV/YXs2w15unPVh8qfu/qCTfhTYamaw7fyhumKa2yGpdSo87vY32rIclj+4fWYQXUMs9EHvg==",
"license": "MIT"
},
"node_modules/range-parser": { "node_modules/range-parser": {
"version": "1.2.1", "version": "1.2.1",
"resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz", "resolved": "https://registry.npmjs.org/range-parser/-/range-parser-1.2.1.tgz",
@@ -3587,6 +3891,36 @@
"node": ">=8.10.0" "node": ">=8.10.0"
} }
}, },
"node_modules/real-require": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/real-require/-/real-require-0.2.0.tgz",
"integrity": "sha512-57frrGM/OCTLqLOAh0mhVA9VBMHd+9U7Zb2THMGdBUoZVOtGbJzjxsYGDJ3A9AYYCP4hn6y1TVbaOfzWtm5GFg==",
"license": "MIT",
"engines": {
"node": ">= 12.13.0"
}
},
"node_modules/redis-errors": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/redis-errors/-/redis-errors-1.2.0.tgz",
"integrity": "sha512-1qny3OExCf0UvUV/5wpYKf2YwPcOqXzkwKKSmKHiE6ZMQs5heeE/c8eXK+PNllPvmjgAbfnsbpkGZWy8cBpn9w==",
"license": "MIT",
"engines": {
"node": ">=4"
}
},
"node_modules/redis-parser": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-3.0.0.tgz",
"integrity": "sha512-DJnGAeenTdpMEH6uAJRK/uiyEIH9WVsUmoLwzudwGJUwZPp80PDBWPHXSAGNPwNvIXAbe7MSUB1zQFugFml66A==",
"license": "MIT",
"dependencies": {
"redis-errors": "^1.0.0"
},
"engines": {
"node": ">=4"
}
},
"node_modules/require-in-the-middle": { "node_modules/require-in-the-middle": {
"version": "5.2.0", "version": "5.2.0",
"resolved": "https://registry.npmjs.org/require-in-the-middle/-/require-in-the-middle-5.2.0.tgz", "resolved": "https://registry.npmjs.org/require-in-the-middle/-/require-in-the-middle-5.2.0.tgz",
@@ -3700,6 +4034,15 @@
], ],
"license": "MIT" "license": "MIT"
}, },
"node_modules/safe-stable-stringify": {
"version": "2.5.0",
"resolved": "https://registry.npmjs.org/safe-stable-stringify/-/safe-stable-stringify-2.5.0.tgz",
"integrity": "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA==",
"license": "MIT",
"engines": {
"node": ">=10"
}
},
"node_modules/safer-buffer": { "node_modules/safer-buffer": {
"version": "2.1.2", "version": "2.1.2",
"resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz", "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
@@ -4001,6 +4344,15 @@
"integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/sonic-boom": {
"version": "4.2.1",
"resolved": "https://registry.npmjs.org/sonic-boom/-/sonic-boom-4.2.1.tgz",
"integrity": "sha512-w6AxtubXa2wTXAUsZMMWERrsIRAdrK0Sc+FUytWvYAhBJLyuI4llrMIC1DtlNSdI99EI86KZum2MMq3EAZlF9Q==",
"license": "MIT",
"dependencies": {
"atomic-sleep": "^1.0.0"
}
},
"node_modules/source-map": { "node_modules/source-map": {
"version": "0.6.1", "version": "0.6.1",
"resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz", "resolved": "https://registry.npmjs.org/source-map/-/source-map-0.6.1.tgz",
@@ -4061,6 +4413,12 @@
"nan": "^2.20.0" "nan": "^2.20.0"
} }
}, },
"node_modules/standard-as-callback": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/standard-as-callback/-/standard-as-callback-2.1.0.tgz",
"integrity": "sha512-qoRRSyROncaz1z0mvYqIE4lCd9p2R90i6GxW3uZv5ucSu8tU7B5HXUP1gG8pVZsYNVaXjk8ClXHPttLyxAL48A==",
"license": "MIT"
},
"node_modules/statuses": { "node_modules/statuses": {
"version": "2.0.1", "version": "2.0.1",
"resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz", "resolved": "https://registry.npmjs.org/statuses/-/statuses-2.0.1.tgz",
@@ -4187,6 +4545,15 @@
"node": ">=10" "node": ">=10"
} }
}, },
"node_modules/thread-stream": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/thread-stream/-/thread-stream-3.1.0.tgz",
"integrity": "sha512-OqyPZ9u96VohAyMfJykzmivOrY2wfMSf3C5TtFJVgN+Hm6aj+voFhlK+kZEIv2FBh1X6Xp3DlnCOfEQ3B2J86A==",
"license": "MIT",
"dependencies": {
"real-require": "^0.2.0"
}
},
"node_modules/to-regex-range": { "node_modules/to-regex-range": {
"version": "5.0.1", "version": "5.0.1",
"resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz", "resolved": "https://registry.npmjs.org/to-regex-range/-/to-regex-range-5.0.1.tgz",
+6
View File
@@ -2,6 +2,7 @@
"name": "inventory-server", "name": "inventory-server",
"version": "1.0.0", "version": "1.0.0",
"description": "Backend server for inventory management system", "description": "Backend server for inventory management system",
"type": "module",
"main": "src/server.js", "main": "src/server.js",
"scripts": { "scripts": {
"start": "node src/server.js", "start": "node src/server.js",
@@ -27,10 +28,15 @@
"diff": "^7.0.0", "diff": "^7.0.0",
"dotenv": "^16.4.7", "dotenv": "^16.4.7",
"express": "^4.18.2", "express": "^4.18.2",
"express-rate-limit": "^7.4.0",
"ioredis": "^5.10.1",
"jsonwebtoken": "^9.0.2",
"multer": "^1.4.5-lts.1", "multer": "^1.4.5-lts.1",
"mysql2": "^3.12.0", "mysql2": "^3.12.0",
"openai": "^6.0.0", "openai": "^6.0.0",
"pg": "^8.14.1", "pg": "^8.14.1",
"pino": "^9.5.0",
"pino-http": "^10.3.0",
"pm2": "^5.3.0", "pm2": "^5.3.0",
"sharp": "^0.33.5", "sharp": "^0.33.5",
"ssh2": "^1.16.0", "ssh2": "^1.16.0",
+6
View File
@@ -0,0 +1,6 @@
{
"name": "inventory-server-scripts",
"private": true,
"type": "commonjs",
"description": "One-shot maintenance scripts (imports, metrics, forecasting orchestration). Kept CommonJS so the existing `require()` graph still works after the parent server moved to ESM. Convert individual scripts when touching them."
}
+1 -1
View File
@@ -10,7 +10,7 @@ export function createPool(envPrefix = 'DB', overrides = {}) {
user: overrides.user ?? get('USER'), user: overrides.user ?? get('USER'),
password: overrides.password ?? get('PASSWORD'), password: overrides.password ?? get('PASSWORD'),
database: overrides.database ?? get('NAME'), database: overrides.database ?? get('NAME'),
port: overrides.port ?? Number(get('PORT')) || 5432, port: overrides.port ?? (Number(get('PORT')) || 5432),
ssl: (overrides.ssl ?? get('SSL')) === 'true' ? { rejectUnauthorized: false } : false, ssl: (overrides.ssl ?? get('SSL')) === 'true' ? { rejectUnauthorized: false } : false,
max: overrides.max ?? 20, max: overrides.max ?? 20,
idleTimeoutMillis: overrides.idleTimeoutMillis ?? 30_000, idleTimeoutMillis: overrides.idleTimeoutMillis ?? 30_000,
+1 -1
View File
@@ -19,7 +19,7 @@ export function createRedis(overrides = {}) {
return new Redis({ return new Redis({
host: overrides.host ?? process.env.REDIS_HOST ?? 'localhost', host: overrides.host ?? process.env.REDIS_HOST ?? 'localhost',
port: overrides.port ?? Number(process.env.REDIS_PORT) || 6379, port: overrides.port ?? (Number(process.env.REDIS_PORT) || 6379),
username: overrides.username ?? process.env.REDIS_USERNAME, username: overrides.username ?? process.env.REDIS_USERNAME,
password: overrides.password ?? process.env.REDIS_PASSWORD, password: overrides.password ?? process.env.REDIS_PASSWORD,
...options, ...options,
+8 -35
View File
@@ -1,41 +1,14 @@
const cors = require('cors'); import cors from 'cors';
import { corsOptions } from '../../shared/cors/policy.js';
// Single CORS middleware for all endpoints export const corsMiddleware = cors(corsOptions);
const corsMiddleware = cors({
origin: [
'https://inventory.kent.pw',
'http://localhost:5175',
'https://acot.site',
'https://tools.acherryontop.com',
/^http:\/\/192\.168\.\d+\.\d+(:\d+)?$/,
/^http:\/\/10\.\d+\.\d+\.\d+(:\d+)?$/
],
methods: ['GET', 'POST', 'PUT', 'DELETE', 'OPTIONS'],
allowedHeaders: ['Content-Type', 'Authorization'],
exposedHeaders: ['Content-Type'],
credentials: true
});
// Error handler for CORS export function corsErrorHandler(err, req, res, next) {
const corsErrorHandler = (err, req, res, next) => { if (err && err.message === 'CORS not allowed') {
if (err.message === 'CORS not allowed') { return res.status(403).json({
console.error('CORS Error:', {
origin: req.get('Origin'),
method: req.method,
path: req.path,
headers: req.headers
});
res.status(403).json({
error: 'CORS not allowed', error: 'CORS not allowed',
origin: req.get('Origin'), origin: req.get('Origin'),
message: 'Origin not in allowed list: https://inventory.kent.pw, https://acot.site, https://tools.acherryontop.com, localhost:5175, 192.168.x.x, or 10.x.x.x'
}); });
} else {
next(err);
} }
}; next(err);
}
module.exports = {
corsMiddleware,
corsErrorHandler
};
+10 -2
View File
@@ -1,6 +1,14 @@
const express = require('express'); import express from 'express';
import { requirePermission } from '../../shared/auth/middleware.js';
const router = express.Router(); const router = express.Router();
// Phase 6.2: prompt edits require ai_admin. Reads remain authenticated-only.
router.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
return requirePermission('ai_admin')(req, res, next);
});
// Get all AI prompts // Get all AI prompts
router.get('/', async (req, res) => { router.get('/', async (req, res) => {
try { try {
@@ -307,4 +315,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
+22 -9
View File
@@ -1,12 +1,25 @@
const express = require("express"); import express from "express";
import OpenAI from "openai";
import { promises as fs } from "node:fs";
import path from "node:path";
import { fileURLToPath } from "node:url";
import dotenv from "dotenv";
import mysql from 'mysql2/promise';
import { Client } from 'ssh2';
import { getDbConnection, closeAllConnections } from '../utils/dbConnection.js'; // Import the optimized connection function
import { requirePermission } from '../../shared/auth/middleware.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const router = express.Router(); const router = express.Router();
const OpenAI = require("openai");
const fs = require("fs").promises; // Phase 6.2: AI validation runs (which trigger OpenAI calls + DB writes) require ai_admin.
const path = require("path"); // Status/health reads stay authenticated-only.
const dotenv = require("dotenv"); router.use((req, res, next) => {
const mysql = require('mysql2/promise'); if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
const { Client } = require('ssh2'); return requirePermission('ai_admin')(req, res, next);
const { getDbConnection, closeAllConnections } = require('../utils/dbConnection'); // Import the optimized connection function });
// Ensure environment variables are loaded // Ensure environment variables are loaded
dotenv.config({ path: path.join(__dirname, "../../.env") }); dotenv.config({ path: path.join(__dirname, "../../.env") });
@@ -1401,7 +1414,7 @@ router.get("/test-taxonomy", async (req, res) => {
} }
}); });
module.exports = router; export default router;
function extractResponseText(response) { function extractResponseText(response) {
if (!response) return ""; if (!response) return "";
+6 -6
View File
@@ -5,10 +5,11 @@
* Provides embedding generation and similarity-based suggestions. * Provides embedding generation and similarity-based suggestions.
*/ */
const express = require('express'); import express from 'express';
import aiService from '../services/ai/index.js';
import { getDbConnection, closeAllConnections } from '../utils/dbConnection.js';
const router = express.Router(); const router = express.Router();
const aiService = require('../services/ai');
const { getDbConnection, closeAllConnections } = require('../utils/dbConnection');
// Track initialization state // Track initialization state
let initializationPromise = null; let initializationPromise = null;
@@ -440,11 +441,10 @@ router.post('/validate/sanity-check', async (req, res) => {
* Call once from server startup so the taxonomy embeddings are ready before * Call once from server startup so the taxonomy embeddings are ready before
* the first user request hits a taxonomy dropdown. * the first user request hits a taxonomy dropdown.
*/ */
function initInBackground() { export function initInBackground() {
ensureInitialized().catch(err => ensureInitialized().catch(err =>
console.error('[AI Routes] Background initialization failed:', err) console.error('[AI Routes] Background initialization failed:', err)
); );
} }
module.exports = router; export default router;
module.exports.initInBackground = initInBackground;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Forecasting: summarize sales for products received in a period by brand // Forecasting: summarize sales for products received in a period by brand
@@ -980,4 +980,4 @@ router.get('/seasonal', async (req, res) => {
} }
}); });
module.exports = router; export default router;
@@ -1,6 +1,6 @@
const express = require('express'); import express from 'express';
import { parseValue } from '../utils/apiHelpers.js'; // Adjust path if needed
const router = express.Router(); const router = express.Router();
const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
// --- Configuration & Helpers --- // --- Configuration & Helpers ---
const DEFAULT_PAGE_LIMIT = 50; const DEFAULT_PAGE_LIMIT = 50;
@@ -281,4 +281,4 @@ router.get('/', async (req, res) => {
// GET /brands-aggregate/:name (Get single brand metric) // GET /brands-aggregate/:name (Get single brand metric)
// Implement if needed, remember to URL-decode the name parameter // Implement if needed, remember to URL-decode the name parameter
module.exports = router; export default router;
@@ -1,6 +1,6 @@
const express = require('express'); import express from 'express';
import { parseValue } from '../utils/apiHelpers.js'; // Adjust path if needed
const router = express.Router(); const router = express.Router();
const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
// --- Configuration & Helpers --- // --- Configuration & Helpers ---
const DEFAULT_PAGE_LIMIT = 50; const DEFAULT_PAGE_LIMIT = 50;
@@ -360,4 +360,4 @@ router.get('/', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+8 -5
View File
@@ -1,10 +1,13 @@
const express = require('express'); import express from 'express';
import { requireAdmin } from '../../shared/auth/middleware.js';
const router = express.Router(); const router = express.Router();
// Debug middleware // Phase 6.2: global settings are admin-only on write. Reads pass through to any
// authenticated user (the server-level authenticate() already gates that).
router.use((req, res, next) => { router.use((req, res, next) => {
console.log(`[Config Route] ${req.method} ${req.path}`); if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
next(); return requireAdmin(req, res, next);
}); });
// ===== GLOBAL SETTINGS ===== // ===== GLOBAL SETTINGS =====
@@ -322,4 +325,4 @@ router.post('/vendors/:vendor/reset', async (req, res) => {
}); });
// Export the router // Export the router
module.exports = router; export default router;
+3 -3
View File
@@ -1,6 +1,6 @@
const express = require('express'); import express from 'express';
import db from '../utils/db.js';
const router = express.Router(); const router = express.Router();
const db = require('../utils/db');
// Helper function to execute queries using the connection pool // Helper function to execute queries using the connection pool
async function executeQuery(sql, params = []) { async function executeQuery(sql, params = []) {
@@ -1288,4 +1288,4 @@ router.get('/replenish/products', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+16 -9
View File
@@ -1,13 +1,20 @@
const express = require('express'); import express from 'express';
const router = express.Router(); import { spawn } from 'node:child_process';
const { spawn } = require('child_process'); import path from 'node:path';
const path = require('path'); import { fileURLToPath } from 'node:url';
const db = require('../utils/db'); import db from '../utils/db.js';
import { requirePermission } from '../../shared/auth/middleware.js';
// Debug middleware MUST be first const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const router = express.Router();
// Phase 6.2: CSV / full-update / full-reset / metrics-recalc are destructive.
// Writes require data_management; reads (status polls, SSE streams) pass through.
router.use((req, res, next) => { router.use((req, res, next) => {
console.log(`[CSV Route Debug] ${req.method} ${req.path}`); if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
next(); return requirePermission('data_management')(req, res, next);
}); });
// Store active processes and their progress // Store active processes and their progress
@@ -437,4 +444,4 @@ router.get('/status/table-counts', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// GET /api/hts-lookup?search=term // GET /api/hts-lookup?search=term
@@ -167,4 +167,4 @@ router.get('/', async (req, res) => {
} }
}); });
module.exports = router; export default router;
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Create a new audit log entry // Create a new audit log entry
@@ -190,4 +190,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Get all import sessions for a user (named + unnamed) // Get all import sessions for a user (named + unnamed)
@@ -334,4 +334,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
+37 -19
View File
@@ -1,14 +1,23 @@
const express = require('express'); import express from 'express';
import { Client } from 'ssh2';
import mysql from 'mysql2/promise';
import multer from 'multer';
import path from 'node:path';
import fs from 'node:fs';
import sharp from 'sharp';
import axios from 'axios';
import net from 'node:net';
import { requirePermission } from '../../shared/auth/middleware.js';
const router = express.Router(); const router = express.Router();
const { Client } = require('ssh2');
const mysql = require('mysql2/promise');
const multer = require('multer');
const path = require('path');
const fs = require('fs');
const fsp = fs.promises; const fsp = fs.promises;
const sharp = require('sharp');
const axios = require('axios'); // Phase 6.2: imports, uploads, generate-upc and deletions all require product_import.
const net = require('net'); // Reads (list-uploads, status checks) remain authenticated-only.
router.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
return requirePermission('product_import')(req, res, next);
});
// Create uploads directory if it doesn't exist // Create uploads directory if it doesn't exist
const uploadsDir = path.join('/var/www/inventory/uploads/products'); const uploadsDir = path.join('/var/www/inventory/uploads/products');
@@ -515,21 +524,30 @@ const storage = multer.diskStorage({
const MAX_UPLOAD_BYTES = 25 * 1024 * 1024; const MAX_UPLOAD_BYTES = 25 * 1024 * 1024;
// Phase 6.7: exact-match MIME + extension allowlist. Substring-based regex
// matchers (the previous /jpeg|png|.../ approach) accepted MIMEs like
// `application/jpeg-payload` because of partial matches; this rejects them.
const ALLOWED_MIME_TYPES = new Set([
'image/jpeg',
'image/png',
'image/gif',
'image/webp',
'image/tiff',
]);
const ALLOWED_EXTENSIONS = new Set(['.jpg', '.jpeg', '.png', '.gif', '.webp', '.tif', '.tiff']);
const upload = multer({ const upload = multer({
storage: storage, storage: storage,
limits: { limits: {
fileSize: MAX_UPLOAD_BYTES, fileSize: MAX_UPLOAD_BYTES,
files: 1,
}, },
fileFilter: function (req, file, cb) { fileFilter: function (req, file, cb) {
// Accept only image files const ext = path.extname(file.originalname).toLowerCase();
const filetypes = /jpeg|jpg|png|gif|webp|tiff?/; if (!ALLOWED_MIME_TYPES.has(file.mimetype) || !ALLOWED_EXTENSIONS.has(ext)) {
const mimetype = filetypes.test(file.mimetype); return cb(new Error('Only image files are allowed (jpg, png, gif, webp, tiff)'));
const extname = filetypes.test(path.extname(file.originalname).toLowerCase());
if (mimetype && extname) {
return cb(null, true);
} }
cb(new Error('Only image files are allowed')); cb(null, true);
} }
}); });
@@ -647,7 +665,7 @@ async function setupSshTunnel() {
port: process.env.PROD_SSH_PORT || 22, port: process.env.PROD_SSH_PORT || 22,
username: process.env.PROD_SSH_USER, username: process.env.PROD_SSH_USER,
privateKey: process.env.PROD_SSH_KEY_PATH privateKey: process.env.PROD_SSH_KEY_PATH
? require('fs').readFileSync(process.env.PROD_SSH_KEY_PATH) ? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
: undefined, : undefined,
compress: true compress: true
}; };
@@ -2872,4 +2890,4 @@ router.get('/query-products', async (req, res) => {
} }
}); });
module.exports = router; export default router;
@@ -1,6 +1,6 @@
const express = require('express'); import express from 'express';
import { parseValue } from '../utils/apiHelpers.js';
const router = express.Router(); const router = express.Router();
const { parseValue } = require('../utils/apiHelpers');
// --- Configuration & Helpers --- // --- Configuration & Helpers ---
const DEFAULT_PAGE_LIMIT = 50; const DEFAULT_PAGE_LIMIT = 50;
@@ -378,4 +378,4 @@ router.get('/:brand/:line/products', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// --- Configuration & Helpers --- // --- Configuration & Helpers ---
@@ -645,4 +645,4 @@ function parseValue(value, type) {
} }
} }
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Shared CTE fragment for the reference date. // Shared CTE fragment for the reference date.
@@ -721,4 +721,4 @@ router.get('/campaigns/links', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Get all orders with pagination, filtering, and sorting // Get all orders with pagination, filtering, and sorting
@@ -258,4 +258,4 @@ router.get('/:orderNumber', async (req, res) => {
} }
}); });
module.exports = router; export default router;
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Create a new audit log entry // Create a new audit log entry
@@ -192,4 +192,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
+4 -24
View File
@@ -1,27 +1,7 @@
const express = require('express'); import express from 'express';
import { PurchaseOrderStatus, ReceivingStatus } from '../types/status-codes.js';
const router = express.Router(); const router = express.Router();
const multer = require('multer');
const path = require('path');
const fs = require('fs');
const { importProductsFromCSV } = require('../utils/csvImporter');
const { PurchaseOrderStatus, ReceivingStatus } = require('../types/status-codes');
// Configure multer for file uploads without silent fallbacks
const configuredUploadsDir = process.env.UPLOADS_DIR;
const uploadsDir = configuredUploadsDir
? (path.isAbsolute(configuredUploadsDir)
? configuredUploadsDir
: path.resolve(__dirname, '../../', configuredUploadsDir))
: path.resolve(__dirname, '../../uploads');
try {
fs.mkdirSync(uploadsDir, { recursive: true });
} catch (error) {
console.error(`Failed to initialize uploads directory at ${uploadsDir}:`, error);
throw error;
}
const upload = multer({ dest: uploadsDir });
// Get unique brands // Get unique brands
router.get('/brands', async (req, res) => { router.get('/brands', async (req, res) => {
@@ -983,4 +963,4 @@ router.get('/:id/forecast', async (req, res) => {
} }
}); });
module.exports = router; export default router;
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Status code constants // Status code constants
@@ -1277,4 +1277,4 @@ router.get('/pipeline', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
// Stale PO statuses to exclude from our counting — these are the "still active" // Stale PO statuses to exclude from our counting — these are the "still active"
@@ -388,4 +388,4 @@ router.get('/:pid/history', async (req, res) => {
} }
}); });
module.exports = router; export default router;
+26 -15
View File
@@ -1,8 +1,16 @@
const express = require('express'); import express from 'express';
import multer from 'multer';
import path from 'node:path';
import fs from 'node:fs';
import { requirePermission } from '../../shared/auth/middleware.js';
const router = express.Router(); const router = express.Router();
const multer = require('multer');
const path = require('path'); // Phase 6.2: uploads + deletions of reusable images require image_admin.
const fs = require('fs'); router.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
return requirePermission('image_admin')(req, res, next);
});
// Create reusable uploads directory if it doesn't exist // Create reusable uploads directory if it doesn't exist
const uploadsDir = path.join('/var/www/inventory/uploads/reusable'); const uploadsDir = path.join('/var/www/inventory/uploads/reusable');
@@ -38,21 +46,24 @@ const storage = multer.diskStorage({
} }
}); });
const upload = multer({ // Phase 6.7: exact-match MIME + extension allowlist.
const ALLOWED_MIME_TYPES = new Set([
'image/jpeg', 'image/png', 'image/gif', 'image/webp',
]);
const ALLOWED_EXTENSIONS = new Set(['.jpg', '.jpeg', '.png', '.gif', '.webp']);
const upload = multer({
storage: storage, storage: storage,
limits: { limits: {
fileSize: 5 * 1024 * 1024, // 5MB max file size fileSize: 5 * 1024 * 1024,
files: 1,
}, },
fileFilter: function (req, file, cb) { fileFilter: function (req, file, cb) {
// Accept only image files const ext = path.extname(file.originalname).toLowerCase();
const filetypes = /jpeg|jpg|png|gif|webp/; if (!ALLOWED_MIME_TYPES.has(file.mimetype) || !ALLOWED_EXTENSIONS.has(ext)) {
const mimetype = filetypes.test(file.mimetype); return cb(new Error('Only image files are allowed (jpg, png, gif, webp)'));
const extname = filetypes.test(path.extname(file.originalname).toLowerCase());
if (mimetype && extname) {
return cb(null, true);
} }
cb(new Error('Only image files are allowed')); cb(null, true);
} }
}); });
@@ -393,4 +404,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
+2 -2
View File
@@ -1,4 +1,4 @@
const express = require('express'); import express from 'express';
const router = express.Router(); const router = express.Router();
const MAX_MATCHES = 500; const MAX_MATCHES = 500;
@@ -267,4 +267,4 @@ function descriptionAggregate(products) {
return { duplicates, samples }; return { duplicates, samples };
} }
module.exports = router; export default router;
+16 -5
View File
@@ -1,12 +1,23 @@
const express = require('express'); import express from 'express';
const { getPool } = require('../utils/db'); import dotenv from 'dotenv';
const dotenv = require('dotenv'); import path from 'node:path';
const path = require('path'); import { fileURLToPath } from 'node:url';
import { getPool } from '../utils/db.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
dotenv.config({ path: path.join(__dirname, "../../.env") }); dotenv.config({ path: path.join(__dirname, "../../.env") });
const router = express.Router(); const router = express.Router();
// Phase 6.2: template edits require templates_write. Reads pass through.
import { requirePermission } from '../../shared/auth/middleware.js';
router.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD' || req.method === 'OPTIONS') return next();
return requirePermission('templates_write')(req, res, next);
});
// Get all templates // Get all templates
router.get('/', async (req, res) => { router.get('/', async (req, res) => {
try { try {
@@ -278,4 +289,4 @@ router.use((err, req, res, next) => {
}); });
}); });
module.exports = router; export default router;
@@ -1,6 +1,6 @@
const express = require('express'); import express from 'express';
import { parseValue } from '../utils/apiHelpers.js'; // Adjust path if needed
const router = express.Router(); const router = express.Router();
const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
// --- Configuration & Helpers --- // --- Configuration & Helpers ---
const DEFAULT_PAGE_LIMIT = 50; const DEFAULT_PAGE_LIMIT = 50;
@@ -320,4 +320,4 @@ router.get('/', async (req, res) => {
// GET /vendors-aggregate/:name (Get single vendor metric) // GET /vendors-aggregate/:name (Get single vendor metric)
// Implement if needed, remember to URL-decode the name parameter // Implement if needed, remember to URL-decode the name parameter
module.exports = router; export default router;
+79 -153
View File
@@ -1,59 +1,64 @@
const express = require('express'); import { config as loadEnv } from 'dotenv';
const cors = require('cors'); import express from 'express';
const { spawn } = require('child_process'); import path from 'node:path';
const path = require('path'); import fs from 'node:fs';
const fs = require('fs'); import { fileURLToPath } from 'node:url';
const { corsMiddleware, corsErrorHandler } = require('./middleware/cors');
const { initPool } = require('./utils/db'); import { corsMiddleware, corsErrorHandler } from './middleware/cors.js';
const productsRouter = require('./routes/products'); import { initPool } from './utils/db.js';
const dashboardRouter = require('./routes/dashboard');
const ordersRouter = require('./routes/orders'); import { authenticate } from '../shared/auth/middleware.js';
const csvRouter = require('./routes/data-management'); import { requestLog } from '../shared/logging/request-log.js';
const analyticsRouter = require('./routes/analytics'); import { logger } from '../shared/logging/logger.js';
const purchaseOrdersRouter = require('./routes/purchase-orders'); import { errorHandler } from '../shared/errors/handler.js';
const configRouter = require('./routes/config');
const metricsRouter = require('./routes/metrics'); import productsRouter from './routes/products.js';
const importRouter = require('./routes/import'); import dashboardRouter from './routes/dashboard.js';
const aiValidationRouter = require('./routes/ai-validation'); import ordersRouter from './routes/orders.js';
const aiRouter = require('./routes/ai'); import csvRouter from './routes/data-management.js';
const templatesRouter = require('./routes/templates'); import analyticsRouter from './routes/analytics.js';
const aiPromptsRouter = require('./routes/ai-prompts'); import purchaseOrdersRouter from './routes/purchase-orders.js';
const reusableImagesRouter = require('./routes/reusable-images'); import configRouter from './routes/config.js';
const categoriesAggregateRouter = require('./routes/categoriesAggregate'); import metricsRouter from './routes/metrics.js';
const vendorsAggregateRouter = require('./routes/vendorsAggregate'); import importRouter from './routes/import.js';
const brandsAggregateRouter = require('./routes/brandsAggregate'); import aiValidationRouter from './routes/ai-validation.js';
const htsLookupRouter = require('./routes/hts-lookup'); import aiRouter, { initInBackground as initAiInBackground } from './routes/ai.js';
const specLookupRouter = require('./routes/spec-lookup'); import templatesRouter from './routes/templates.js';
const importSessionsRouter = require('./routes/import-sessions'); import aiPromptsRouter from './routes/ai-prompts.js';
const importAuditLogRouter = require('./routes/import-audit-log'); import reusableImagesRouter from './routes/reusable-images.js';
const productEditorAuditLogRouter = require('./routes/product-editor-audit-log'); import categoriesAggregateRouter from './routes/categoriesAggregate.js';
const newsletterRouter = require('./routes/newsletter'); import vendorsAggregateRouter from './routes/vendorsAggregate.js';
const linesAggregateRouter = require('./routes/linesAggregate'); import brandsAggregateRouter from './routes/brandsAggregate.js';
const repeatOrdersRouter = require('./routes/repeat-orders'); import htsLookupRouter from './routes/hts-lookup.js';
import specLookupRouter from './routes/spec-lookup.js';
import importSessionsRouter from './routes/import-sessions.js';
import importAuditLogRouter from './routes/import-audit-log.js';
import productEditorAuditLogRouter from './routes/product-editor-audit-log.js';
import newsletterRouter from './routes/newsletter.js';
import linesAggregateRouter from './routes/linesAggregate.js';
import repeatOrdersRouter from './routes/repeat-orders.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
// Get the absolute path to the .env file
const envPath = '/var/www/inventory/.env'; const envPath = '/var/www/inventory/.env';
console.log('Looking for .env file at:', envPath); loadEnv({ path: envPath });
console.log('.env file exists:', fs.existsSync(envPath)); logger.info({
envPath,
envExists: fs.existsSync(envPath),
NODE_ENV: process.env.NODE_ENV || 'not set',
PORT: process.env.PORT || 'not set',
DB_HOST: process.env.DB_HOST || 'not set',
DB_NAME: process.env.DB_NAME || 'not set',
DB_PASSWORD: process.env.DB_PASSWORD ? '[set]' : 'not set',
DB_SSL: process.env.DB_SSL || 'not set',
}, 'inventory-server starting');
try { if (!process.env.JWT_SECRET) {
require('dotenv').config({ path: envPath }); logger.error('JWT_SECRET is not set; refusing to start (per Phase 6.4)');
console.log('.env file loaded successfully'); process.exit(1);
console.log('Environment check:', {
NODE_ENV: process.env.NODE_ENV || 'not set',
PORT: process.env.PORT || 'not set',
DB_HOST: process.env.DB_HOST || 'not set',
DB_USER: process.env.DB_USER || 'not set',
DB_NAME: process.env.DB_NAME || 'not set',
DB_PASSWORD: process.env.DB_PASSWORD ? '[password set]' : 'not set',
DB_PORT: process.env.DB_PORT || 'not set',
DB_SSL: process.env.DB_SSL || 'not set'
});
} catch (error) {
console.error('Error loading .env file:', error);
} }
// Resolve important directories relative to the project root
const serverRoot = path.resolve(__dirname, '..'); const serverRoot = path.resolve(__dirname, '..');
const configuredUploadsDir = process.env.UPLOADS_DIR; const configuredUploadsDir = process.env.UPLOADS_DIR;
const uploadsDir = configuredUploadsDir const uploadsDir = configuredUploadsDir
@@ -62,12 +67,10 @@ const uploadsDir = configuredUploadsDir
: path.resolve(serverRoot, configuredUploadsDir)) : path.resolve(serverRoot, configuredUploadsDir))
: path.resolve(serverRoot, 'uploads'); : path.resolve(serverRoot, 'uploads');
// Persist the resolved uploads directory so downstream modules share the same path
process.env.UPLOADS_DIR = uploadsDir; process.env.UPLOADS_DIR = uploadsDir;
const requiredDirs = [path.resolve(serverRoot, 'logs'), uploadsDir]; const requiredDirs = [path.resolve(serverRoot, 'logs'), uploadsDir];
requiredDirs.forEach((dir) => {
requiredDirs.forEach(dir => {
if (!fs.existsSync(dir)) { if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true }); fs.mkdirSync(dir, { recursive: true });
} }
@@ -75,28 +78,18 @@ requiredDirs.forEach(dir => {
const app = express(); const app = express();
// Debug middleware to log request details // Phase 6.5/6.9: structured access log (replaces the previous header-dumping debug
app.use((req, res, next) => { // middleware that wrote raw Authorization values to stdout). Pino redaction strips
console.log('Request details:', { // `authorization` and `cookie` automatically — see shared/logging/logger.js.
method: req.method, app.use(requestLog());
url: req.url,
origin: req.get('Origin'),
headers: req.headers
});
next();
});
// Apply CORS middleware first, before any other middleware
app.use(corsMiddleware); app.use(corsMiddleware);
// Body parser middleware
app.use(express.json({ limit: '10mb' })); app.use(express.json({ limit: '10mb' }));
app.use(express.urlencoded({ extended: true, limit: '10mb' })); app.use(express.urlencoded({ extended: true, limit: '10mb' }));
// Initialize database pool and start server
async function startServer() { async function startServer() {
try { try {
// Initialize database pool
const pool = await initPool({ const pool = await initPool({
host: process.env.DB_HOST, host: process.env.DB_HOST,
user: process.env.DB_USER, user: process.env.DB_USER,
@@ -104,17 +97,18 @@ async function startServer() {
database: process.env.DB_NAME, database: process.env.DB_NAME,
port: process.env.DB_PORT || 5432, port: process.env.DB_PORT || 5432,
max: process.env.NODE_ENV === 'production' ? 20 : 10, max: process.env.NODE_ENV === 'production' ? 20 : 10,
idleTimeoutMillis: 30000, idleTimeoutMillis: 30_000,
connectionTimeoutMillis: 2000, connectionTimeoutMillis: 2_000,
ssl: process.env.DB_SSL === 'true' ? { ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
rejectUnauthorized: false
} : false
}); });
// Make pool available to routes
app.locals.pool = pool; app.locals.pool = pool;
// Set up routes after pool is initialized // Phase 6.1/6.2: every /api request requires a valid JWT. Defense in depth — Caddy
// forward_auth (when enabled) is the first reject; this is the second. Frontend
// service files MUST include `Authorization: Bearer <token>` on every fetch.
app.use('/api', authenticate({ pool, secret: process.env.JWT_SECRET }));
app.use('/api/products', productsRouter); app.use('/api/products', productsRouter);
app.use('/api/dashboard', dashboardRouter); app.use('/api/dashboard', dashboardRouter);
app.use('/api/orders', ordersRouter); app.use('/api/orders', ordersRouter);
@@ -123,10 +117,8 @@ async function startServer() {
app.use('/api/purchase-orders', purchaseOrdersRouter); app.use('/api/purchase-orders', purchaseOrdersRouter);
app.use('/api/config', configRouter); app.use('/api/config', configRouter);
app.use('/api/metrics', metricsRouter); app.use('/api/metrics', metricsRouter);
// Use only the aggregate routes for vendors and categories app.use('/api/vendors', vendorsAggregateRouter);
app.use('/api/vendors', vendorsAggregateRouter);
app.use('/api/categories', categoriesAggregateRouter); app.use('/api/categories', categoriesAggregateRouter);
// Keep the aggregate-specific endpoints for backward compatibility
app.use('/api/categories-aggregate', categoriesAggregateRouter); app.use('/api/categories-aggregate', categoriesAggregateRouter);
app.use('/api/vendors-aggregate', vendorsAggregateRouter); app.use('/api/vendors-aggregate', vendorsAggregateRouter);
app.use('/api/brands-aggregate', brandsAggregateRouter); app.use('/api/brands-aggregate', brandsAggregateRouter);
@@ -145,101 +137,35 @@ async function startServer() {
app.use('/api/lines-aggregate', linesAggregateRouter); app.use('/api/lines-aggregate', linesAggregateRouter);
app.use('/api/repeat-orders', repeatOrdersRouter); app.use('/api/repeat-orders', repeatOrdersRouter);
// Basic health check route
app.get('/health', (req, res) => { app.get('/health', (req, res) => {
res.json({ res.json({
status: 'ok', status: 'ok',
timestamp: new Date().toISOString(), timestamp: new Date().toISOString(),
environment: process.env.NODE_ENV environment: process.env.NODE_ENV,
}); });
}); });
// CORS error handler - must be before other error handlers
app.use(corsErrorHandler); app.use(corsErrorHandler);
app.use(errorHandler);
// Error handling middleware - MUST be after routes and CORS error handler
app.use((err, req, res, next) => {
console.error(`[${new Date().toISOString()}] Error:`, err);
// Send detailed error in development, generic in production
const error = process.env.NODE_ENV === 'production'
? 'An internal server error occurred'
: err.message || err;
res.status(err.status || 500).json({ error });
});
const PORT = process.env.PORT || 3000; const PORT = process.env.PORT || 3000;
app.listen(PORT, () => { app.listen(PORT, () => {
console.log(`[Server] Running in ${process.env.NODE_ENV || 'development'} mode on port ${PORT}`); logger.info({ port: PORT, mode: process.env.NODE_ENV || 'development' }, 'inventory-server listening');
// Pre-warm AI service so taxonomy embeddings are ready before first user request initAiInBackground();
aiRouter.initInBackground();
}); });
} catch (error) { } catch (error) {
console.error('Failed to start server:', error); logger.error({ err: error }, 'Failed to start server');
process.exit(1); process.exit(1);
} }
} }
// Handle uncaught exceptions
process.on('uncaughtException', (err) => { process.on('uncaughtException', (err) => {
console.error(`[${new Date().toISOString()}] Uncaught Exception:`, err); logger.error({ err }, 'Uncaught Exception');
process.exit(1); process.exit(1);
}); });
process.on('unhandledRejection', (reason, promise) => { process.on('unhandledRejection', (reason, promise) => {
console.error(`[${new Date().toISOString()}] Unhandled Rejection at:`, promise, 'reason:', reason); logger.error({ reason, promise }, 'Unhandled Rejection');
}); });
// Initialize client sets for SSE startServer();
const importClients = new Set();
const updateClients = new Set();
const resetClients = new Set();
const resetMetricsClients = new Set();
// Helper function to send progress to SSE clients
const sendProgressToClients = (clients, data) => {
clients.forEach(client => {
try {
client.write(`data: ${JSON.stringify(data)}\n\n`);
} catch (error) {
console.error('Error sending SSE update:', error);
}
});
};
// Setup SSE connection
const setupSSE = (req, res) => {
const { type } = req.params;
// Set headers for SSE
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
'Access-Control-Allow-Origin': req.headers.origin || '*',
'Access-Control-Allow-Credentials': 'true'
});
// Send initial message
res.write('data: {"status":"connected"}\n\n');
// Add client to appropriate set
const clientSet = type === 'import' ? importClients :
type === 'update' ? updateClients :
type === 'reset' ? resetClients :
type === 'reset-metrics' ? resetMetricsClients :
null;
if (clientSet) {
clientSet.add(res);
// Remove client when connection closes
req.on('close', () => {
clientSet.delete(res);
});
}
};
// Start the server
startServer();
@@ -1,82 +1,36 @@
/** export function cosineSimilarity(a, b) {
* Vector similarity utilities if (!a || !b || a.length !== b.length) return 0;
*/
/**
* Compute cosine similarity between two vectors
* @param {number[]} a
* @param {number[]} b
* @returns {number} Similarity score between -1 and 1
*/
function cosineSimilarity(a, b) {
if (!a || !b || a.length !== b.length) {
return 0;
}
let dotProduct = 0; let dotProduct = 0;
let normA = 0; let normA = 0;
let normB = 0; let normB = 0;
for (let i = 0; i < a.length; i++) { for (let i = 0; i < a.length; i++) {
dotProduct += a[i] * b[i]; dotProduct += a[i] * b[i];
normA += a[i] * a[i]; normA += a[i] * a[i];
normB += b[i] * b[i]; normB += b[i] * b[i];
} }
const denominator = Math.sqrt(normA) * Math.sqrt(normB); const denominator = Math.sqrt(normA) * Math.sqrt(normB);
if (denominator === 0) return 0; return denominator === 0 ? 0 : dotProduct / denominator;
return dotProduct / denominator;
} }
/** export function findTopMatches(queryEmbedding, items, topK = 10) {
* Find top K most similar items from a collection if (!queryEmbedding || !items || items.length === 0) return [];
* @param {number[]} queryEmbedding - The embedding to search for const scored = items.map((item) => ({
* @param {Array<{id: any, embedding: number[]}>} items - Items with embeddings
* @param {number} topK - Number of results to return
* @returns {Array<{id: any, similarity: number}>}
*/
function findTopMatches(queryEmbedding, items, topK = 10) {
if (!queryEmbedding || !items || items.length === 0) {
return [];
}
const scored = items.map(item => ({
id: item.id, id: item.id,
similarity: cosineSimilarity(queryEmbedding, item.embedding) similarity: cosineSimilarity(queryEmbedding, item.embedding),
})); }));
scored.sort((a, b) => b.similarity - a.similarity); scored.sort((a, b) => b.similarity - a.similarity);
return scored.slice(0, topK); return scored.slice(0, topK);
} }
/** export function findMatchesAboveThreshold(queryEmbedding, items, threshold = 0.5) {
* Find matches above a similarity threshold if (!queryEmbedding || !items || items.length === 0) return [];
* @param {number[]} queryEmbedding
* @param {Array<{id: any, embedding: number[]}>} items
* @param {number} threshold - Minimum similarity (0-1)
* @returns {Array<{id: any, similarity: number}>}
*/
function findMatchesAboveThreshold(queryEmbedding, items, threshold = 0.5) {
if (!queryEmbedding || !items || items.length === 0) {
return [];
}
const scored = items const scored = items
.map(item => ({ .map((item) => ({
id: item.id, id: item.id,
similarity: cosineSimilarity(queryEmbedding, item.embedding) similarity: cosineSimilarity(queryEmbedding, item.embedding),
})) }))
.filter(item => item.similarity >= threshold); .filter((item) => item.similarity >= threshold);
scored.sort((a, b) => b.similarity - a.similarity); scored.sort((a, b) => b.similarity - a.similarity);
return scored; return scored;
} }
module.exports = {
cosineSimilarity,
findTopMatches,
findMatchesAboveThreshold
};
@@ -1,74 +1,52 @@
/** /**
* Taxonomy Embedding Service * Taxonomy Embedding Service
* *
* Generates and caches embeddings for categories, themes, and colors.
* Excludes "Black Friday", "Gifts", "Deals" categories and their children. * Excludes "Black Friday", "Gifts", "Deals" categories and their children.
* * Disk cache at data/taxonomy-embeddings.json; content-hash invalidated.
* Disk cache: embeddings are saved to data/taxonomy-embeddings.json and reused
* across server restarts. Cache is invalidated by content hash — if the taxonomy
* rows in MySQL change, the next check will detect it and regenerate automatically.
*
* Background check: after initialization, call startBackgroundCheck(getConnectionFn)
* to poll for taxonomy changes on a configurable interval (default 1h).
*/ */
const fs = require('fs'); import fs from 'node:fs';
const path = require('path'); import path from 'node:path';
const crypto = require('crypto'); import crypto from 'node:crypto';
const { findTopMatches } = require('./similarity'); import { fileURLToPath } from 'node:url';
import { findTopMatches } from './similarity.js';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
// Categories to exclude (and all their children)
const EXCLUDED_CATEGORY_NAMES = ['black friday', 'gifts', 'deals']; const EXCLUDED_CATEGORY_NAMES = ['black friday', 'gifts', 'deals'];
// Disk cache config
const CACHE_PATH = path.join(__dirname, '..', '..', '..', '..', 'data', 'taxonomy-embeddings.json'); const CACHE_PATH = path.join(__dirname, '..', '..', '..', '..', 'data', 'taxonomy-embeddings.json');
class TaxonomyEmbeddings { export class TaxonomyEmbeddings {
constructor({ provider, logger }) { constructor({ provider, logger }) {
this.provider = provider; this.provider = provider;
this.logger = logger || console; this.logger = logger || console;
// Cached taxonomy with embeddings
this.categories = []; this.categories = [];
this.themes = []; this.themes = [];
this.colors = []; this.colors = [];
// Raw data without embeddings (for lookup)
this.categoryMap = new Map(); this.categoryMap = new Map();
this.themeMap = new Map(); this.themeMap = new Map();
this.colorMap = new Map(); this.colorMap = new Map();
// Content hash of the last successfully built taxonomy (from DB rows)
this.contentHash = null; this.contentHash = null;
this.initialized = false; this.initialized = false;
this.initializing = false; this.initializing = false;
this._checkInterval = null; this._checkInterval = null;
this._regenerating = false; this._regenerating = false;
} }
/**
* Initialize embeddings — fetches raw taxonomy rows to compute a content hash,
* then either loads the matching disk cache or generates fresh embeddings.
*/
async initialize(connection) { async initialize(connection) {
if (this.initialized) { if (this.initialized) {
return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length }; return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length };
} }
if (this.initializing) { if (this.initializing) {
// Wait for existing initialization
while (this.initializing) { while (this.initializing) {
await new Promise(resolve => setTimeout(resolve, 100)); await new Promise((resolve) => setTimeout(resolve, 100));
} }
return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length }; return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length };
} }
this.initializing = true; this.initializing = true;
try { try {
// Always fetch raw rows first — cheap (~10ms), no OpenAI calls.
// Used to compute a content hash for cache validation.
const rawRows = await this._fetchRawRows(connection); const rawRows = await this._fetchRawRows(connection);
const freshHash = this._computeContentHash(rawRows); const freshHash = this._computeContentHash(rawRows);
@@ -77,9 +55,9 @@ class TaxonomyEmbeddings {
this.categories = cached.categories; this.categories = cached.categories;
this.themes = cached.themes; this.themes = cached.themes;
this.colors = cached.colors; this.colors = cached.colors;
this.categoryMap = new Map(this.categories.map(c => [c.id, c])); this.categoryMap = new Map(this.categories.map((c) => [c.id, c]));
this.themeMap = new Map(this.themes.map(t => [t.id, t])); this.themeMap = new Map(this.themes.map((t) => [t.id, t]));
this.colorMap = new Map(this.colors.map(c => [c.id, c])); this.colorMap = new Map(this.colors.map((c) => [c.id, c]));
this.contentHash = freshHash; this.contentHash = freshHash;
this.initialized = true; this.initialized = true;
this.logger.info(`[TaxonomyEmbeddings] Loaded from cache: ${this.categories.length} categories, ${this.themes.length} themes, ${this.colors.length} colors`); this.logger.info(`[TaxonomyEmbeddings] Loaded from cache: ${this.categories.length} categories, ${this.themes.length} themes, ${this.colors.length} colors`);
@@ -95,7 +73,6 @@ class TaxonomyEmbeddings {
await this._buildAndEmbed(rawRows, freshHash); await this._buildAndEmbed(rawRows, freshHash);
this.initialized = true; this.initialized = true;
this.logger.info('[TaxonomyEmbeddings] Initialization complete'); this.logger.info('[TaxonomyEmbeddings] Initialization complete');
return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length }; return { categories: this.categories.length, themes: this.themes.length, colors: this.colors.length };
} catch (error) { } catch (error) {
this.logger.error('[TaxonomyEmbeddings] Initialization failed:', error); this.logger.error('[TaxonomyEmbeddings] Initialization failed:', error);
@@ -105,26 +82,16 @@ class TaxonomyEmbeddings {
} }
} }
/**
* Start a background interval that checks for taxonomy changes and regenerates
* embeddings automatically if the content hash differs.
*
* @param {Function} getConnectionFn - async function returning { connection }
* @param {number} intervalMs - check interval, default 1 hour
*/
startBackgroundCheck(getConnectionFn, intervalMs = 60 * 60 * 1000) { startBackgroundCheck(getConnectionFn, intervalMs = 60 * 60 * 1000) {
if (this._checkInterval) return; if (this._checkInterval) return;
this.logger.info(`[TaxonomyEmbeddings] Background taxonomy check started (every ${intervalMs / 60000} min)`); this.logger.info(`[TaxonomyEmbeddings] Background taxonomy check started (every ${intervalMs / 60000} min)`);
this._checkInterval = setInterval(async () => { this._checkInterval = setInterval(async () => {
if (this._regenerating) return; if (this._regenerating) return;
try { try {
const { connection } = await getConnectionFn(); const { connection } = await getConnectionFn();
const rawRows = await this._fetchRawRows(connection); const rawRows = await this._fetchRawRows(connection);
const freshHash = this._computeContentHash(rawRows); const freshHash = this._computeContentHash(rawRows);
if (freshHash === this.contentHash) return; if (freshHash === this.contentHash) return;
this.logger.info('[TaxonomyEmbeddings] Taxonomy changed, regenerating embeddings in background...'); this.logger.info('[TaxonomyEmbeddings] Taxonomy changed, regenerating embeddings in background...');
@@ -146,123 +113,79 @@ class TaxonomyEmbeddings {
} }
} }
/**
* Find similar categories for a product embedding
*/
findSimilarCategories(productEmbedding, topK = 10) { findSimilarCategories(productEmbedding, topK = 10) {
if (!this.initialized || !productEmbedding) { if (!this.initialized || !productEmbedding) return [];
return [];
}
const matches = findTopMatches(productEmbedding, this.categories, topK); const matches = findTopMatches(productEmbedding, this.categories, topK);
return matches.map((match) => {
return matches.map(match => {
const cat = this.categoryMap.get(match.id); const cat = this.categoryMap.get(match.id);
return { return {
id: match.id, id: match.id,
name: cat?.name || '', name: cat?.name || '',
fullPath: cat?.fullPath || '', fullPath: cat?.fullPath || '',
similarity: match.similarity similarity: match.similarity,
}; };
}); });
} }
/**
* Find similar themes for a product embedding
*/
findSimilarThemes(productEmbedding, topK = 5) { findSimilarThemes(productEmbedding, topK = 5) {
if (!this.initialized || !productEmbedding) { if (!this.initialized || !productEmbedding) return [];
return [];
}
const matches = findTopMatches(productEmbedding, this.themes, topK); const matches = findTopMatches(productEmbedding, this.themes, topK);
return matches.map((match) => {
return matches.map(match => {
const theme = this.themeMap.get(match.id); const theme = this.themeMap.get(match.id);
return { return {
id: match.id, id: match.id,
name: theme?.name || '', name: theme?.name || '',
fullPath: theme?.fullPath || '', fullPath: theme?.fullPath || '',
similarity: match.similarity similarity: match.similarity,
}; };
}); });
} }
/**
* Find similar colors for a product embedding
*/
findSimilarColors(productEmbedding, topK = 5) { findSimilarColors(productEmbedding, topK = 5) {
if (!this.initialized || !productEmbedding) { if (!this.initialized || !productEmbedding) return [];
return [];
}
const matches = findTopMatches(productEmbedding, this.colors, topK); const matches = findTopMatches(productEmbedding, this.colors, topK);
return matches.map((match) => {
return matches.map(match => {
const color = this.colorMap.get(match.id); const color = this.colorMap.get(match.id);
return { return {
id: match.id, id: match.id,
name: color?.name || '', name: color?.name || '',
similarity: match.similarity similarity: match.similarity,
}; };
}); });
} }
/**
* Get all taxonomy data (without embeddings) for frontend
*/
getTaxonomyData() { getTaxonomyData() {
return { return {
categories: this.categories.map(({ id, name, fullPath, parentId }) => ({ id, name, fullPath, parentId })), categories: this.categories.map(({ id, name, fullPath, parentId }) => ({ id, name, fullPath, parentId })),
themes: this.themes.map(({ id, name, fullPath, parentId }) => ({ id, name, fullPath, parentId })), themes: this.themes.map(({ id, name, fullPath, parentId }) => ({ id, name, fullPath, parentId })),
colors: this.colors.map(({ id, name }) => ({ id, name })) colors: this.colors.map(({ id, name }) => ({ id, name })),
}; };
} }
/**
* Check if service is ready
*/
isReady() { isReady() {
return this.initialized; return this.initialized;
} }
// ============================================================================
// Private Methods
// ============================================================================
/**
* Fetch minimal raw rows from MySQL — used for content hash computation.
* This is the cheap path: no path-building, no embeddings, just the raw data.
*/
async _fetchRawRows(connection) { async _fetchRawRows(connection) {
const [[catRows], [themeRows], [colorRows]] = await Promise.all([ const [[catRows], [themeRows], [colorRows]] = await Promise.all([
connection.query('SELECT cat_id, name, master_cat_id, type FROM product_categories WHERE type IN (10, 11, 12, 13) ORDER BY cat_id'), connection.query('SELECT cat_id, name, master_cat_id, type FROM product_categories WHERE type IN (10, 11, 12, 13) ORDER BY cat_id'),
connection.query('SELECT cat_id, name, master_cat_id, type FROM product_categories WHERE type IN (20, 21) ORDER BY cat_id'), connection.query('SELECT cat_id, name, master_cat_id, type FROM product_categories WHERE type IN (20, 21) ORDER BY cat_id'),
connection.query('SELECT color, name, hex_color FROM product_color_list ORDER BY `order`') connection.query('SELECT color, name, hex_color FROM product_color_list ORDER BY `order`'),
]); ]);
return { catRows, themeRows, colorRows }; return { catRows, themeRows, colorRows };
} }
/**
* Compute a stable SHA-256 hash of the taxonomy row content.
* Any change to IDs, names, or parent relationships will produce a different hash.
*/
_computeContentHash({ catRows, themeRows, colorRows }) { _computeContentHash({ catRows, themeRows, colorRows }) {
const content = JSON.stringify({ const content = JSON.stringify({
cats: catRows.map(r => [r.cat_id, r.name, r.master_cat_id]).sort((a, b) => a[0] - b[0]), cats: catRows.map((r) => [r.cat_id, r.name, r.master_cat_id]).sort((a, b) => a[0] - b[0]),
themes: themeRows.map(r => [r.cat_id, r.name, r.master_cat_id]).sort((a, b) => a[0] - b[0]), themes: themeRows.map((r) => [r.cat_id, r.name, r.master_cat_id]).sort((a, b) => a[0] - b[0]),
colors: colorRows.map(r => [r.color, r.name]).sort() colors: colorRows.map((r) => [r.color, r.name]).sort(),
}); });
return crypto.createHash('sha256').update(content).digest('hex').slice(0, 16); return crypto.createHash('sha256').update(content).digest('hex').slice(0, 16);
} }
/**
* Build full taxonomy objects and generate embeddings, then atomically swap
* the in-memory state. Called on cache miss and on background change detection.
*/
async _buildAndEmbed(rawRows, contentHash) { async _buildAndEmbed(rawRows, contentHash) {
const { catRows, themeRows, colorRows } = rawRows; const { catRows, themeRows, colorRows } = rawRows;
const categories = this._buildCategories(catRows); const categories = this._buildCategories(catRows);
const themes = this._buildThemes(themeRows); const themes = this._buildThemes(themeRows);
const colors = this._buildColors(colorRows); const colors = this._buildColors(colorRows);
@@ -272,23 +195,22 @@ class TaxonomyEmbeddings {
const [catEmbeddings, themeEmbeddings, colorEmbeddings] = await Promise.all([ const [catEmbeddings, themeEmbeddings, colorEmbeddings] = await Promise.all([
this._generateEmbeddings(categories, 'categories'), this._generateEmbeddings(categories, 'categories'),
this._generateEmbeddings(themes, 'themes'), this._generateEmbeddings(themes, 'themes'),
this._generateEmbeddings(colors, 'colors') this._generateEmbeddings(colors, 'colors'),
]); ]);
// Atomic in-memory swap (single-threaded JS — readers always see a consistent state)
this.categories = catEmbeddings; this.categories = catEmbeddings;
this.themes = themeEmbeddings; this.themes = themeEmbeddings;
this.colors = colorEmbeddings; this.colors = colorEmbeddings;
this.categoryMap = new Map(this.categories.map(c => [c.id, c])); this.categoryMap = new Map(this.categories.map((c) => [c.id, c]));
this.themeMap = new Map(this.themes.map(t => [t.id, t])); this.themeMap = new Map(this.themes.map((t) => [t.id, t]));
this.colorMap = new Map(this.colors.map(c => [c.id, c])); this.colorMap = new Map(this.colors.map((c) => [c.id, c]));
this.contentHash = contentHash; this.contentHash = contentHash;
this._saveCache(); this._saveCache();
} }
_buildCategories(rows) { _buildCategories(rows) {
const byId = new Map(rows.map(r => [r.cat_id, r])); const byId = new Map(rows.map((r) => [r.cat_id, r]));
const excludedIds = new Set(); const excludedIds = new Set();
for (const row of rows) { for (const row of rows) {
@@ -297,7 +219,6 @@ class TaxonomyEmbeddings {
} }
} }
// Multiple passes to find all descendants of excluded categories
let foundNew = true; let foundNew = true;
while (foundNew) { while (foundNew) {
foundNew = false; foundNew = false;
@@ -314,98 +235,80 @@ class TaxonomyEmbeddings {
const categories = []; const categories = [];
for (const row of rows) { for (const row of rows) {
if (excludedIds.has(row.cat_id)) continue; if (excludedIds.has(row.cat_id)) continue;
const pathParts = []; const pathParts = [];
let current = row; let current = row;
while (current) { while (current) {
pathParts.unshift(current.name); pathParts.unshift(current.name);
current = current.master_cat_id ? byId.get(current.master_cat_id) : null; current = current.master_cat_id ? byId.get(current.master_cat_id) : null;
} }
categories.push({ categories.push({
id: row.cat_id, id: row.cat_id,
name: row.name, name: row.name,
parentId: row.master_cat_id, parentId: row.master_cat_id,
type: row.type, type: row.type,
fullPath: pathParts.join(' > '), fullPath: pathParts.join(' > '),
embeddingText: pathParts.join(' ') embeddingText: pathParts.join(' '),
}); });
} }
return categories; return categories;
} }
_buildThemes(rows) { _buildThemes(rows) {
const byId = new Map(rows.map(r => [r.cat_id, r])); const byId = new Map(rows.map((r) => [r.cat_id, r]));
return rows.map((row) => {
return rows.map(row => {
const pathParts = []; const pathParts = [];
let current = row; let current = row;
while (current) { while (current) {
pathParts.unshift(current.name); pathParts.unshift(current.name);
current = current.master_cat_id ? byId.get(current.master_cat_id) : null; current = current.master_cat_id ? byId.get(current.master_cat_id) : null;
} }
return { return {
id: row.cat_id, id: row.cat_id,
name: row.name, name: row.name,
parentId: row.master_cat_id, parentId: row.master_cat_id,
type: row.type, type: row.type,
fullPath: pathParts.join(' > '), fullPath: pathParts.join(' > '),
embeddingText: pathParts.join(' ') embeddingText: pathParts.join(' '),
}; };
}); });
} }
_buildColors(rows) { _buildColors(rows) {
return rows.map(row => ({ return rows.map((row) => ({
id: row.color, id: row.color,
name: row.name, name: row.name,
hexColor: row.hex_color, hexColor: row.hex_color,
embeddingText: row.name embeddingText: row.name,
})); }));
} }
async _generateEmbeddings(items, label) { async _generateEmbeddings(items, label) {
if (items.length === 0) { if (items.length === 0) return items;
return items;
}
const startTime = Date.now(); const startTime = Date.now();
const texts = items.map(item => item.embeddingText); const texts = items.map((item) => item.embeddingText);
const results = [...items]; const results = [...items];
// Process in batches
for await (const chunk of this.provider.embedBatchChunked(texts, { batchSize: 100 })) { for await (const chunk of this.provider.embedBatchChunked(texts, { batchSize: 100 })) {
for (let i = 0; i < chunk.embeddings.length; i++) { for (let i = 0; i < chunk.embeddings.length; i++) {
const globalIndex = chunk.startIndex + i; const globalIndex = chunk.startIndex + i;
results[globalIndex] = { results[globalIndex] = { ...results[globalIndex], embedding: chunk.embeddings[i] };
...results[globalIndex],
embedding: chunk.embeddings[i]
};
} }
} }
const elapsed = Date.now() - startTime; const elapsed = Date.now() - startTime;
this.logger.info(`[TaxonomyEmbeddings] Generated ${items.length} ${label} embeddings in ${elapsed}ms`); this.logger.info(`[TaxonomyEmbeddings] Generated ${items.length} ${label} embeddings in ${elapsed}ms`);
return results; return results;
} }
// ============================================================================
// Disk Cache Methods
// ============================================================================
_loadCache() { _loadCache() {
try { try {
if (!fs.existsSync(CACHE_PATH)) return null; if (!fs.existsSync(CACHE_PATH)) return null;
const data = JSON.parse(fs.readFileSync(CACHE_PATH, 'utf8')); const data = JSON.parse(fs.readFileSync(CACHE_PATH, 'utf8'));
if (!data.contentHash || !data.categories?.length || !data.themes?.length || !data.colors?.length) { if (!data.contentHash || !data.categories?.length || !data.themes?.length || !data.colors?.length) {
this.logger.warn('[TaxonomyEmbeddings] Disk cache malformed or missing content hash, will regenerate'); this.logger.warn('[TaxonomyEmbeddings] Disk cache malformed or missing content hash, will regenerate');
return null; return null;
} }
return data; return data;
} catch (err) { } catch (err) {
this.logger.warn('[TaxonomyEmbeddings] Failed to load disk cache:', err.message); this.logger.warn('[TaxonomyEmbeddings] Failed to load disk cache:', err.message);
@@ -429,5 +332,3 @@ class TaxonomyEmbeddings {
} }
} }
} }
module.exports = { TaxonomyEmbeddings };
+55 -220
View File
@@ -1,17 +1,16 @@
/** /**
* AI Service * AI Service
* *
* Main entry point for AI functionality including: * Main entry point for AI functionality (embeddings + chat completions + task registry).
* - Embeddings for taxonomy suggestions (OpenAI)
* - Chat completions for validation tasks (Groq)
* - Task registry for AI operations
*/ */
const { OpenAIProvider } = require('./providers/openaiProvider'); import { OpenAIProvider } from './providers/openaiProvider.js';
const { GroqProvider, MODELS: GROQ_MODELS } = require('./providers/groqProvider'); import { GroqProvider, MODELS as GROQ_MODELS } from './providers/groqProvider.js';
const { TaxonomyEmbeddings } = require('./embeddings/taxonomyEmbeddings'); import { TaxonomyEmbeddings } from './embeddings/taxonomyEmbeddings.js';
const { cosineSimilarity, findTopMatches } = require('./embeddings/similarity'); import { cosineSimilarity, findTopMatches } from './embeddings/similarity.js';
const { getRegistry, TASK_IDS, registerAllTasks } = require('./tasks'); import { getRegistry, TASK_IDS, registerAllTasks } from './tasks/index.js';
export { TASK_IDS, GROQ_MODELS, cosineSimilarity, findTopMatches };
let initialized = false; let initialized = false;
let initializing = false; let initializing = false;
@@ -19,54 +18,28 @@ let openaiProvider = null;
let groqProvider = null; let groqProvider = null;
let taxonomyEmbeddings = null; let taxonomyEmbeddings = null;
let logger = console; let logger = console;
// Store pool reference for task access
let appPool = null; let appPool = null;
/** export async function initialize({ openaiApiKey, groqApiKey, mysqlConnection, pool, logger: customLogger }) {
* Initialize the AI service if (initialized) return { success: true, message: 'Already initialized' };
* @param {Object} options
* @param {string} options.openaiApiKey - OpenAI API key (for embeddings)
* @param {string} [options.groqApiKey] - Groq API key (for chat completions)
* @param {Object} options.mysqlConnection - MySQL connection for taxonomy data
* @param {Object} [options.pool] - PostgreSQL pool for prompt loading
* @param {Object} [options.logger] - Logger instance
*/
async function initialize({ openaiApiKey, groqApiKey, mysqlConnection, pool, logger: customLogger }) {
if (initialized) {
return { success: true, message: 'Already initialized' };
}
if (initializing) { if (initializing) {
// Wait for existing initialization
while (initializing) { while (initializing) {
await new Promise(resolve => setTimeout(resolve, 100)); await new Promise((resolve) => setTimeout(resolve, 100));
} }
return { success: initialized, message: initialized ? 'Initialized' : 'Initialization failed' }; return { success: initialized, message: initialized ? 'Initialized' : 'Initialization failed' };
} }
initializing = true; initializing = true;
try { try {
if (customLogger) { if (customLogger) logger = customLogger;
logger = customLogger; if (!openaiApiKey) throw new Error('OpenAI API key is required');
}
if (!openaiApiKey) {
throw new Error('OpenAI API key is required');
}
logger.info('[AI] Initializing AI service...'); logger.info('[AI] Initializing AI service...');
if (pool) appPool = pool;
// Store pool reference for tasks
if (pool) {
appPool = pool;
}
// Create OpenAI provider (for embeddings)
openaiProvider = new OpenAIProvider({ apiKey: openaiApiKey }); openaiProvider = new OpenAIProvider({ apiKey: openaiApiKey });
// Create Groq provider (for chat completions) if API key provided
if (groqApiKey) { if (groqApiKey) {
groqProvider = new GroqProvider({ apiKey: groqApiKey }); groqProvider = new GroqProvider({ apiKey: groqApiKey });
logger.info('[AI] Groq provider initialized for chat completions'); logger.info('[AI] Groq provider initialized for chat completions');
@@ -74,32 +47,19 @@ async function initialize({ openaiApiKey, groqApiKey, mysqlConnection, pool, log
logger.warn('[AI] No Groq API key provided - chat completion tasks will not be available'); logger.warn('[AI] No Groq API key provided - chat completion tasks will not be available');
} }
// Create and initialize taxonomy embeddings taxonomyEmbeddings = new TaxonomyEmbeddings({ provider: openaiProvider, logger });
taxonomyEmbeddings = new TaxonomyEmbeddings({
provider: openaiProvider,
logger
});
const stats = await taxonomyEmbeddings.initialize(mysqlConnection); const stats = await taxonomyEmbeddings.initialize(mysqlConnection);
// Register validation tasks if Groq is available if (groqProvider) registerValidationTasks();
if (groqProvider) {
registerValidationTasks();
}
initialized = true; initialized = true;
logger.info('[AI] AI service initialized', { logger.info('[AI] AI service initialized', {
...stats, ...stats,
groqEnabled: !!groqProvider, groqEnabled: !!groqProvider,
tasksRegistered: getRegistry().list() tasksRegistered: getRegistry().list(),
}); });
return { return { success: true, message: 'Initialized', stats, groqEnabled: !!groqProvider };
success: true,
message: 'Initialized',
stats,
groqEnabled: !!groqProvider
};
} catch (error) { } catch (error) {
logger.error('[AI] Initialization failed:', error); logger.error('[AI] Initialization failed:', error);
return { success: false, message: error.message }; return { success: false, message: error.message };
@@ -108,41 +68,20 @@ async function initialize({ openaiApiKey, groqApiKey, mysqlConnection, pool, log
} }
} }
/**
* Register validation tasks with the task registry
* Called during initialization if Groq is available
*/
function registerValidationTasks() { function registerValidationTasks() {
registerAllTasks(logger); registerAllTasks(logger);
logger.info('[AI] Validation tasks registered'); logger.info('[AI] Validation tasks registered');
} }
/** export function isReady() {
* Check if service is ready
*/
function isReady() {
return initialized && taxonomyEmbeddings?.isReady(); return initialized && taxonomyEmbeddings?.isReady();
} }
/** export function startBackgroundCheck(getConnectionFn, intervalMs) {
* Start background taxonomy change detection.
* Call once after initialization, passing a function that returns { connection }.
* @param {Function} getConnectionFn
* @param {number} [intervalMs] - default 1 hour
*/
function startBackgroundCheck(getConnectionFn, intervalMs) {
if (!initialized || !taxonomyEmbeddings) return; if (!initialized || !taxonomyEmbeddings) return;
taxonomyEmbeddings.startBackgroundCheck(getConnectionFn, intervalMs); taxonomyEmbeddings.startBackgroundCheck(getConnectionFn, intervalMs);
} }
/**
* Build weighted product text for embedding.
* Weights the product name heavily by repeating it, and truncates long descriptions
* to prevent verbose marketing copy from drowning out the product signal.
*
* @param {Object} product - Product with name, description, company, line
* @returns {string} - Combined text for embedding
*/
function buildProductText(product) { function buildProductText(product) {
const parts = []; const parts = [];
const name = product.name?.trim(); const name = product.name?.trim();
@@ -150,20 +89,9 @@ function buildProductText(product) {
const company = (product.company_name || product.company)?.trim(); const company = (product.company_name || product.company)?.trim();
const line = (product.line_name || product.line)?.trim(); const line = (product.line_name || product.line)?.trim();
// Name is most important - repeat 3x to weight it heavily in the embedding if (name) parts.push(name, name, name);
if (name) { if (company) parts.push(company);
parts.push(name, name, name); if (line) parts.push(line);
}
// Company and line provide context
if (company) {
parts.push(company);
}
if (line) {
parts.push(line);
}
// Truncate description to prevent it from overwhelming the signal
if (description) { if (description) {
const truncated = description.length > 500 const truncated = description.length > 500
? description.substring(0, 500) + '...' ? description.substring(0, 500) + '...'
@@ -174,74 +102,35 @@ function buildProductText(product) {
return parts.join(' ').trim(); return parts.join(' ').trim();
} }
/** export async function getProductEmbedding(product) {
* Generate embedding for a product if (!initialized || !openaiProvider) throw new Error('AI service not initialized');
* @param {Object} product - Product with name, description, company, line
* @returns {Promise<{embedding: number[], latencyMs: number}>}
*/
async function getProductEmbedding(product) {
if (!initialized || !openaiProvider) {
throw new Error('AI service not initialized');
}
const text = buildProductText(product); const text = buildProductText(product);
if (!text) return { embedding: null, latencyMs: 0 };
if (!text) {
return { embedding: null, latencyMs: 0 };
}
const result = await openaiProvider.embed(text); const result = await openaiProvider.embed(text);
return { embedding: result.embeddings[0], latencyMs: result.latencyMs };
return {
embedding: result.embeddings[0],
latencyMs: result.latencyMs
};
} }
/** export async function getProductEmbeddings(products) {
* Generate embeddings for multiple products if (!initialized || !openaiProvider) throw new Error('AI service not initialized');
* @param {Object[]} products - Array of products
* @returns {Promise<{embeddings: Array<{index: number, embedding: number[]}>, latencyMs: number}>}
*/
async function getProductEmbeddings(products) {
if (!initialized || !openaiProvider) {
throw new Error('AI service not initialized');
}
const texts = products.map(buildProductText); const texts = products.map(buildProductText);
const validIndices = texts.map((t, i) => t ? i : -1).filter((i) => i >= 0);
const validTexts = texts.filter((t) => t);
// Track which products have empty text if (validTexts.length === 0) return { embeddings: [], latencyMs: 0 };
const validIndices = texts.map((t, i) => t ? i : -1).filter(i => i >= 0);
const validTexts = texts.filter(t => t);
if (validTexts.length === 0) {
return { embeddings: [], latencyMs: 0 };
}
const result = await openaiProvider.embed(validTexts); const result = await openaiProvider.embed(validTexts);
// Map embeddings back to original indices
const embeddings = validIndices.map((originalIndex, resultIndex) => ({ const embeddings = validIndices.map((originalIndex, resultIndex) => ({
index: originalIndex, index: originalIndex,
embedding: result.embeddings[resultIndex] embedding: result.embeddings[resultIndex],
})); }));
return { embeddings, latencyMs: result.latencyMs };
return {
embeddings,
latencyMs: result.latencyMs
};
} }
/** export function findSimilarTaxonomy(productEmbedding, options = {}) {
* Find similar taxonomy items for a product embedding if (!initialized || !taxonomyEmbeddings) throw new Error('AI service not initialized');
* @param {number[]} productEmbedding
* @param {Object} options
* @returns {{categories: Array, themes: Array, colors: Array}}
*/
function findSimilarTaxonomy(productEmbedding, options = {}) {
if (!initialized || !taxonomyEmbeddings) {
throw new Error('AI service not initialized');
}
const topCategories = options.topCategories ?? 10; const topCategories = options.topCategories ?? 10;
const topThemes = options.topThemes ?? 5; const topThemes = options.topThemes ?? 5;
@@ -250,25 +139,15 @@ function findSimilarTaxonomy(productEmbedding, options = {}) {
return { return {
categories: taxonomyEmbeddings.findSimilarCategories(productEmbedding, topCategories), categories: taxonomyEmbeddings.findSimilarCategories(productEmbedding, topCategories),
themes: taxonomyEmbeddings.findSimilarThemes(productEmbedding, topThemes), themes: taxonomyEmbeddings.findSimilarThemes(productEmbedding, topThemes),
colors: taxonomyEmbeddings.findSimilarColors(productEmbedding, topColors) colors: taxonomyEmbeddings.findSimilarColors(productEmbedding, topColors),
}; };
} }
/** export async function getSuggestionsForProduct(product, options = {}) {
* Get product embedding and find similar taxonomy in one call
* @param {Object} product
* @param {Object} options
*/
async function getSuggestionsForProduct(product, options = {}) {
const { embedding, latencyMs: embeddingLatency } = await getProductEmbedding(product); const { embedding, latencyMs: embeddingLatency } = await getProductEmbedding(product);
if (!embedding) { if (!embedding) {
return { return { categories: [], themes: [], colors: [], latencyMs: embeddingLatency };
categories: [],
themes: [],
colors: [],
latencyMs: embeddingLatency
};
} }
const startSearch = Date.now(); const startSearch = Date.now();
@@ -279,27 +158,17 @@ async function getSuggestionsForProduct(product, options = {}) {
...suggestions, ...suggestions,
latencyMs: embeddingLatency + searchLatency, latencyMs: embeddingLatency + searchLatency,
embeddingLatencyMs: embeddingLatency, embeddingLatencyMs: embeddingLatency,
searchLatencyMs: searchLatency searchLatencyMs: searchLatency,
}; };
} }
/** export function getTaxonomyData() {
* Get all taxonomy data (without embeddings) for frontend if (!initialized || !taxonomyEmbeddings) throw new Error('AI service not initialized');
*/
function getTaxonomyData() {
if (!initialized || !taxonomyEmbeddings) {
throw new Error('AI service not initialized');
}
return taxonomyEmbeddings.getTaxonomyData(); return taxonomyEmbeddings.getTaxonomyData();
} }
/** export function getStatus() {
* Get service status
*/
function getStatus() {
const registry = getRegistry(); const registry = getRegistry();
return { return {
initialized, initialized,
ready: isReady(), ready: isReady(),
@@ -309,90 +178,56 @@ function getStatus() {
taxonomyStats: taxonomyEmbeddings ? { taxonomyStats: taxonomyEmbeddings ? {
categories: taxonomyEmbeddings.categories?.length || 0, categories: taxonomyEmbeddings.categories?.length || 0,
themes: taxonomyEmbeddings.themes?.length || 0, themes: taxonomyEmbeddings.themes?.length || 0,
colors: taxonomyEmbeddings.colors?.length || 0 colors: taxonomyEmbeddings.colors?.length || 0,
} : null, } : null,
tasks: { tasks: {
registered: registry.list(), registered: registry.list(),
count: registry.size() count: registry.size(),
} },
}; };
} }
/** export async function runTask(taskId, payload = {}) {
* Run an AI task by ID if (!initialized) throw new Error('AI service not initialized');
* @param {string} taskId - Task identifier from TASK_IDS if (!groqProvider) throw new Error('Groq provider not available - chat completion tasks require GROQ_API_KEY');
* @param {Object} payload - Task-specific input
* @returns {Promise<Object>} Task result
*/
async function runTask(taskId, payload = {}) {
if (!initialized) {
throw new Error('AI service not initialized');
}
if (!groqProvider) {
throw new Error('Groq provider not available - chat completion tasks require GROQ_API_KEY');
}
const registry = getRegistry(); const registry = getRegistry();
return registry.runTask(taskId, { return registry.runTask(taskId, {
...payload, ...payload,
// Inject dependencies tasks may need
provider: groqProvider, provider: groqProvider,
// Use pool from payload if provided (from route), fall back to stored appPool
pool: payload.pool || appPool, pool: payload.pool || appPool,
logger logger,
}); });
} }
/** export function getGroqProvider() {
* Get the Groq provider instance (for direct use if needed)
* @returns {GroqProvider|null}
*/
function getGroqProvider() {
return groqProvider; return groqProvider;
} }
/** export function getPool() {
* Get the PostgreSQL pool (for tasks that need DB access)
* @returns {Object|null}
*/
function getPool() {
return appPool; return appPool;
} }
/** export function hasChatCompletion() {
* Check if chat completion tasks are available
* @returns {boolean}
*/
function hasChatCompletion() {
return !!groqProvider; return !!groqProvider;
} }
module.exports = { export default {
// Initialization
initialize, initialize,
isReady, isReady,
getStatus, getStatus,
startBackgroundCheck, startBackgroundCheck,
// Embeddings (OpenAI)
getProductEmbedding, getProductEmbedding,
getProductEmbeddings, getProductEmbeddings,
findSimilarTaxonomy, findSimilarTaxonomy,
getSuggestionsForProduct, getSuggestionsForProduct,
getTaxonomyData, getTaxonomyData,
// Chat completions (Groq)
runTask, runTask,
hasChatCompletion, hasChatCompletion,
getGroqProvider, getGroqProvider,
getPool, getPool,
// Constants
TASK_IDS, TASK_IDS,
GROQ_MODELS, GROQ_MODELS,
// Re-export utilities
cosineSimilarity, cosineSimilarity,
findTopMatches findTopMatches,
}; };
@@ -1,81 +1,41 @@
/** /**
* Description Validation Prompts * Description Validation Prompts
*
* Functions for building and parsing description validation prompts.
* System and general prompts are loaded from the database.
*/ */
/**
* Sanitize an issue string from AI response
* AI sometimes returns malformed strings with escape sequences
*
* @param {string} issue - Raw issue string
* @returns {string} Cleaned issue string
*/
function sanitizeIssue(issue) { function sanitizeIssue(issue) {
if (!issue || typeof issue !== 'string') return ''; if (!issue || typeof issue !== 'string') return '';
return issue
let cleaned = issue
// Remove trailing backslashes (incomplete escapes)
.replace(/\\+$/, '') .replace(/\\+$/, '')
// Fix malformed escaped quotes at end of string
.replace(/\\",?\)?$/, '') .replace(/\\",?\)?$/, '')
// Clean up double-escaped quotes
.replace(/\\\\"/g, '"') .replace(/\\\\"/g, '"')
// Clean up single escaped quotes that aren't needed
.replace(/\\"/g, '"') .replace(/\\"/g, '"')
// Remove any remaining trailing punctuation artifacts
.replace(/[,\s]+$/, '') .replace(/[,\s]+$/, '')
// Trim whitespace
.trim(); .trim();
return cleaned;
} }
/** export function buildDescriptionUserPrompt(product, prompts) {
* Build the user prompt for description validation
* Combines database prompts with product data
*
* @param {Object} product - Product data
* @param {string} product.name - Product name
* @param {string} product.description - Current description
* @param {string} [product.company_name] - Company name
* @param {string} [product.categories] - Product categories
* @param {Object} prompts - Prompts loaded from database
* @param {string} prompts.general - General description guidelines
* @param {string} [prompts.companySpecific] - Company-specific rules
* @returns {string} Complete user prompt
*/
function buildDescriptionUserPrompt(product, prompts) {
const parts = []; const parts = [];
// Add general prompt/guidelines if provided
if (prompts.general) { if (prompts.general) {
parts.push(prompts.general); parts.push(prompts.general);
parts.push(''); // Empty line for separation parts.push('');
} }
// Add company-specific rules if provided
if (prompts.companySpecific) { if (prompts.companySpecific) {
parts.push(`COMPANY-SPECIFIC RULES FOR ${product.company_name || 'THIS COMPANY'}:`); parts.push(`COMPANY-SPECIFIC RULES FOR ${product.company_name || 'THIS COMPANY'}:`);
parts.push(prompts.companySpecific); parts.push(prompts.companySpecific);
parts.push(''); // Empty line for separation parts.push('');
} }
// Add product information
parts.push('PRODUCT TO VALIDATE:'); parts.push('PRODUCT TO VALIDATE:');
parts.push(`NAME: "${product.name || ''}"`); parts.push(`NAME: "${product.name || ''}"`);
parts.push(`COMPANY: ${product.company_name || 'Unknown'}`); parts.push(`COMPANY: ${product.company_name || 'Unknown'}`);
if (product.categories) parts.push(`CATEGORIES: ${product.categories}`);
if (product.categories) {
parts.push(`CATEGORIES: ${product.categories}`);
}
parts.push(''); parts.push('');
parts.push('CURRENT DESCRIPTION:'); parts.push('CURRENT DESCRIPTION:');
parts.push(`"${product.description || '(empty)'}"`); parts.push(`"${product.description || '(empty)'}"`);
// Add response format instructions
parts.push(''); parts.push('');
parts.push('CRITICAL RULES:'); parts.push('CRITICAL RULES:');
parts.push('- If isValid is false, you MUST provide a suggestion with the improved description'); parts.push('- If isValid is false, you MUST provide a suggestion with the improved description');
@@ -86,68 +46,40 @@ function buildDescriptionUserPrompt(product, prompts) {
parts.push(JSON.stringify({ parts.push(JSON.stringify({
isValid: 'true if perfect, false if ANY changes needed', isValid: 'true if perfect, false if ANY changes needed',
suggestion: 'REQUIRED when isValid is false - the complete improved description', suggestion: 'REQUIRED when isValid is false - the complete improved description',
issues: ['list each problem found (empty array only if isValid is true)'] issues: ['list each problem found (empty array only if isValid is true)'],
}, null, 2)); }, null, 2));
return parts.join('\n'); return parts.join('\n');
} }
/** export function parseDescriptionResponse(parsed, content) {
* Parse the AI response for description validation
*
* @param {Object|null} parsed - Parsed JSON from AI
* @param {string} content - Raw response content
* @returns {Object}
*/
function parseDescriptionResponse(parsed, content) {
// If we got valid parsed JSON, use it
if (parsed && typeof parsed.isValid === 'boolean') { if (parsed && typeof parsed.isValid === 'boolean') {
// Sanitize issues - AI sometimes returns malformed escape sequences
const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : []; const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : [];
const issues = rawIssues const issues = rawIssues.map(sanitizeIssue).filter((issue) => issue.length > 0);
.map(sanitizeIssue)
.filter(issue => issue.length > 0);
const suggestion = parsed.suggestion || null; const suggestion = parsed.suggestion || null;
// IMPORTANT: LLMs sometimes return contradictory data (isValid: true with issues).
// If there are issues, treat as invalid regardless of what the AI said.
// Also if there's a suggestion, the AI thought something needed to change.
const isValid = parsed.isValid && issues.length === 0 && !suggestion; const isValid = parsed.isValid && issues.length === 0 && !suggestion;
return { isValid, suggestion, issues }; return { isValid, suggestion, issues };
} }
// Handle case where isValid is a string "true"/"false" instead of boolean
if (parsed && typeof parsed.isValid === 'string') { if (parsed && typeof parsed.isValid === 'string') {
const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : []; const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : [];
const issues = rawIssues const issues = rawIssues.map(sanitizeIssue).filter((issue) => issue.length > 0);
.map(sanitizeIssue)
.filter(issue => issue.length > 0);
const suggestion = parsed.suggestion || null; const suggestion = parsed.suggestion || null;
const rawIsValid = parsed.isValid.toLowerCase() !== 'false'; const rawIsValid = parsed.isValid.toLowerCase() !== 'false';
// Same defensive logic: if there are issues, it's not valid
const isValid = rawIsValid && issues.length === 0 && !suggestion; const isValid = rawIsValid && issues.length === 0 && !suggestion;
return { isValid, suggestion, issues }; return { isValid, suggestion, issues };
} }
// Try to extract from content if parsing failed
try { try {
// Look for isValid pattern
const isValidMatch = content.match(/"isValid"\s*:\s*(true|false)/i); const isValidMatch = content.match(/"isValid"\s*:\s*(true|false)/i);
const isValid = isValidMatch ? isValidMatch[1].toLowerCase() === 'true' : true; const isValid = isValidMatch ? isValidMatch[1].toLowerCase() === 'true' : true;
// Look for suggestion (might be multiline)
const suggestionMatch = content.match(/"suggestion"\s*:\s*"((?:[^"\\]|\\.)*)"/s); const suggestionMatch = content.match(/"suggestion"\s*:\s*"((?:[^"\\]|\\.)*)"/s);
let suggestion = suggestionMatch ? suggestionMatch[1] : null; let suggestion = suggestionMatch ? suggestionMatch[1] : null;
if (suggestion) { if (suggestion) {
// Unescape common escapes
suggestion = suggestion.replace(/\\n/g, '\n').replace(/\\"/g, '"'); suggestion = suggestion.replace(/\\n/g, '\n').replace(/\\"/g, '"');
} }
// Look for issues array
const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/); const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/);
let issues = []; let issues = [];
if (issuesMatch) { if (issuesMatch) {
@@ -155,22 +87,13 @@ function parseDescriptionResponse(parsed, content) {
const issueStrings = issuesContent.match(/"([^"]+)"/g); const issueStrings = issuesContent.match(/"([^"]+)"/g);
if (issueStrings) { if (issueStrings) {
issues = issueStrings issues = issueStrings
.map(s => sanitizeIssue(s.replace(/"/g, ''))) .map((s) => sanitizeIssue(s.replace(/"/g, '')))
.filter(issue => issue.length > 0); .filter((issue) => issue.length > 0);
} }
} }
// Same logic: if there are issues, it's not valid
const finalIsValid = isValid && issues.length === 0 && !suggestion; const finalIsValid = isValid && issues.length === 0 && !suggestion;
return { isValid: finalIsValid, suggestion, issues }; return { isValid: finalIsValid, suggestion, issues };
} catch { } catch {
// Default to valid if we can't parse anything
return { isValid: true, suggestion: null, issues: [] }; return { isValid: true, suggestion: null, issues: [] };
} }
} }
module.exports = {
buildDescriptionUserPrompt,
parseDescriptionResponse
};
@@ -1,164 +1,94 @@
/** /**
* Name Validation Prompts * Name Validation Prompts
*
* Functions for building and parsing name validation prompts.
* System and general prompts are loaded from the database.
*/ */
/**
* Sanitize an issue string from AI response
* AI sometimes returns malformed strings with escape sequences
*
* @param {string} issue - Raw issue string
* @returns {string} Cleaned issue string
*/
function sanitizeIssue(issue) { function sanitizeIssue(issue) {
if (!issue || typeof issue !== 'string') return ''; if (!issue || typeof issue !== 'string') return '';
return issue
let cleaned = issue
// Remove trailing backslashes (incomplete escapes)
.replace(/\\+$/, '') .replace(/\\+$/, '')
// Fix malformed escaped quotes at end of string
.replace(/\\",?\)?$/, '') .replace(/\\",?\)?$/, '')
// Clean up double-escaped quotes
.replace(/\\\\"/g, '"') .replace(/\\\\"/g, '"')
// Clean up single escaped quotes that aren't needed
.replace(/\\"/g, '"') .replace(/\\"/g, '"')
// Remove any remaining trailing punctuation artifacts
.replace(/[,\s]+$/, '') .replace(/[,\s]+$/, '')
// Trim whitespace
.trim(); .trim();
return cleaned;
} }
/** export function buildNameUserPrompt(product, prompts) {
* Build the user prompt for name validation
* Combines database prompts with product data
*
* @param {Object} product - Product data
* @param {string} product.name - Current product name
* @param {string} [product.company_name] - Company name
* @param {string} [product.line_name] - Product line name
* @param {string} [product.subline_name] - Product subline name
* @param {string[]} [product.siblingNames] - Names of other products in the same line
* @param {Object} prompts - Prompts loaded from database
* @param {string} prompts.general - General naming conventions
* @param {string} [prompts.companySpecific] - Company-specific rules
* @returns {string} Complete user prompt
*/
function buildNameUserPrompt(product, prompts) {
const parts = []; const parts = [];
// Add general prompt/conventions if provided
if (prompts.general) { if (prompts.general) {
parts.push(prompts.general); parts.push(prompts.general);
parts.push(''); // Empty line for separation parts.push('');
} }
// Add company-specific rules if provided
if (prompts.companySpecific) { if (prompts.companySpecific) {
parts.push(`COMPANY-SPECIFIC RULES FOR ${product.company_name || 'THIS COMPANY'}:`); parts.push(`COMPANY-SPECIFIC RULES FOR ${product.company_name || 'THIS COMPANY'}:`);
parts.push(prompts.companySpecific); parts.push(prompts.companySpecific);
parts.push(''); // Empty line for separation parts.push('');
} }
// Add product information
parts.push('PRODUCT TO VALIDATE:'); parts.push('PRODUCT TO VALIDATE:');
parts.push(`NAME: "${product.name || ''}"`); parts.push(`NAME: "${product.name || ''}"`);
parts.push(`COMPANY: ${product.company_name || 'Unknown'}`); parts.push(`COMPANY: ${product.company_name || 'Unknown'}`);
parts.push(`LINE: ${product.line_name || 'None'}`); parts.push(`LINE: ${product.line_name || 'None'}`);
if (product.subline_name) { if (product.subline_name) parts.push(`SUBLINE: ${product.subline_name}`);
parts.push(`SUBLINE: ${product.subline_name}`);
}
// Add sibling context for naming decisions
if (product.siblingNames && product.siblingNames.length > 0) { if (product.siblingNames && product.siblingNames.length > 0) {
parts.push(''); parts.push('');
parts.push(`OTHER PRODUCTS IN THIS LINE (${product.siblingNames.length + 1} total including this one):`); parts.push(`OTHER PRODUCTS IN THIS LINE (${product.siblingNames.length + 1} total including this one):`);
product.siblingNames.forEach(name => { product.siblingNames.forEach((name) => parts.push(`- ${name}`));
parts.push(`- ${name}`);
});
} }
// Add response format instructions
parts.push(''); parts.push('');
parts.push('RESPOND WITH JSON:'); parts.push('RESPOND WITH JSON:');
parts.push(JSON.stringify({ parts.push(JSON.stringify({
isValid: 'true/false', isValid: 'true/false',
suggestion: 'corrected name if changes needed, or null if valid', suggestion: 'corrected name if changes needed, or null if valid',
issues: ['issue 1', 'issue 2 (empty array if valid)'] issues: ['issue 1', 'issue 2 (empty array if valid)'],
}, null, 2)); }, null, 2));
return parts.join('\n'); return parts.join('\n');
} }
/** export function parseNameResponse(parsed, content) {
* Parse the AI response for name validation
*
* @param {Object|null} parsed - Parsed JSON from AI
* @param {string} content - Raw response content
* @returns {Object}
*/
function parseNameResponse(parsed, content) {
// Debug: Log what we're trying to parse
console.log('[parseNameResponse] Input:', { console.log('[parseNameResponse] Input:', {
hasParsed: !!parsed, hasParsed: !!parsed,
parsedIsValid: parsed?.isValid, parsedIsValid: parsed?.isValid,
parsedType: typeof parsed?.isValid, parsedType: typeof parsed?.isValid,
contentPreview: content?.substring(0, 3000) contentPreview: content?.substring(0, 3000),
}); });
// If we got valid parsed JSON, use it
if (parsed && typeof parsed.isValid === 'boolean') { if (parsed && typeof parsed.isValid === 'boolean') {
// Sanitize issues - AI sometimes returns malformed escape sequences
const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : []; const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : [];
const issues = rawIssues const issues = rawIssues.map(sanitizeIssue).filter((issue) => issue.length > 0);
.map(sanitizeIssue)
.filter(issue => issue.length > 0);
const suggestion = parsed.suggestion || null; const suggestion = parsed.suggestion || null;
// IMPORTANT: LLMs sometimes return contradictory data (isValid: true with issues).
// If there are issues, treat as invalid regardless of what the AI said.
const isValid = parsed.isValid && issues.length === 0 && !suggestion; const isValid = parsed.isValid && issues.length === 0 && !suggestion;
return { isValid, suggestion, issues }; return { isValid, suggestion, issues };
} }
// Handle case where isValid is a string "true"/"false" instead of boolean
if (parsed && typeof parsed.isValid === 'string') { if (parsed && typeof parsed.isValid === 'string') {
const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : []; const rawIssues = Array.isArray(parsed.issues) ? parsed.issues : [];
const issues = rawIssues const issues = rawIssues.map(sanitizeIssue).filter((issue) => issue.length > 0);
.map(sanitizeIssue)
.filter(issue => issue.length > 0);
const suggestion = parsed.suggestion || null; const suggestion = parsed.suggestion || null;
const rawIsValid = parsed.isValid.toLowerCase() !== 'false'; const rawIsValid = parsed.isValid.toLowerCase() !== 'false';
// Same defensive logic: if there are issues, it's not valid
const isValid = rawIsValid && issues.length === 0 && !suggestion; const isValid = rawIsValid && issues.length === 0 && !suggestion;
console.log('[parseNameResponse] Parsed isValid as string:', parsed.isValid, '→', isValid); console.log('[parseNameResponse] Parsed isValid as string:', parsed.isValid, '→', isValid);
return { isValid, suggestion, issues }; return { isValid, suggestion, issues };
} }
// Try to extract from content if parsing failed
try { try {
// Look for isValid pattern - handle both boolean and quoted string
// Matches: "isValid": true, "isValid": false, "isValid": "true", "isValid": "false"
const isValidMatch = content.match(/"isValid"\s*:\s*"?(true|false)"?/i); const isValidMatch = content.match(/"isValid"\s*:\s*"?(true|false)"?/i);
const isValid = isValidMatch ? isValidMatch[1].toLowerCase() === 'true' : true; const isValid = isValidMatch ? isValidMatch[1].toLowerCase() === 'true' : true;
console.log('[parseNameResponse] Regex extraction:', { console.log('[parseNameResponse] Regex extraction:', {
isValidMatch: isValidMatch?.[0], isValidMatch: isValidMatch?.[0],
isValidValue: isValidMatch?.[1], isValidValue: isValidMatch?.[1],
resultIsValid: isValid resultIsValid: isValid,
}); });
// Look for suggestion - handle escaped quotes and null
const suggestionMatch = content.match(/"suggestion"\s*:\s*(?:"([^"\\]*(?:\\.[^"\\]*)*)"|null)/); const suggestionMatch = content.match(/"suggestion"\s*:\s*(?:"([^"\\]*(?:\\.[^"\\]*)*)"|null)/);
const suggestion = suggestionMatch ? (suggestionMatch[1] || null) : null; const suggestion = suggestionMatch ? (suggestionMatch[1] || null) : null;
// Look for issues array
const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/); const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/);
let issues = []; let issues = [];
if (issuesMatch) { if (issuesMatch) {
@@ -166,22 +96,13 @@ function parseNameResponse(parsed, content) {
const issueStrings = issuesContent.match(/"([^"]+)"/g); const issueStrings = issuesContent.match(/"([^"]+)"/g);
if (issueStrings) { if (issueStrings) {
issues = issueStrings issues = issueStrings
.map(s => sanitizeIssue(s.replace(/"/g, ''))) .map((s) => sanitizeIssue(s.replace(/"/g, '')))
.filter(issue => issue.length > 0); .filter((issue) => issue.length > 0);
} }
} }
// Same defensive logic: if there are issues, it's not valid
const finalIsValid = isValid && issues.length === 0 && !suggestion; const finalIsValid = isValid && issues.length === 0 && !suggestion;
return { isValid: finalIsValid, suggestion, issues }; return { isValid: finalIsValid, suggestion, issues };
} catch { } catch {
// Default to valid if we can't parse anything
return { isValid: true, suggestion: null, issues: [] }; return { isValid: true, suggestion: null, issues: [] };
} }
} }
module.exports = {
buildNameUserPrompt,
parseNameResponse
};
@@ -1,35 +1,18 @@
/** /**
* Prompt Loader * Prompt Loader — loads AI prompts from the ai_prompts PostgreSQL table.
*
* Utilities to load AI prompts from the ai_prompts PostgreSQL table.
* Supports loading prompts by base type (e.g., 'name_validation' loads
* name_validation_system, name_validation_general, and optionally
* name_validation_company_specific).
*/ */
/** export async function loadPromptByType(pool, promptType, company = null) {
* Load a single prompt by exact type
* @param {Object} pool - PostgreSQL pool
* @param {string} promptType - Exact prompt type (e.g., 'name_validation_system')
* @param {string} [company] - Company identifier (for company_specific types)
* @returns {Promise<string|null>} Prompt text or null if not found
*/
async function loadPromptByType(pool, promptType, company = null) {
try { try {
let result; const result = company
? await pool.query(
if (company) { 'SELECT prompt_text FROM ai_prompts WHERE prompt_type = $1 AND company = $2',
result = await pool.query( [promptType, company]
'SELECT prompt_text FROM ai_prompts WHERE prompt_type = $1 AND company = $2', )
[promptType, company] : await pool.query(
); 'SELECT prompt_text FROM ai_prompts WHERE prompt_type = $1 AND company IS NULL',
} else { [promptType]
result = await pool.query( );
'SELECT prompt_text FROM ai_prompts WHERE prompt_type = $1 AND company IS NULL',
[promptType]
);
}
return result.rows[0]?.prompt_text || null; return result.rows[0]?.prompt_text || null;
} catch (error) { } catch (error) {
console.error(`[PromptLoader] Error loading ${promptType} prompt:`, error.message); console.error(`[PromptLoader] Error loading ${promptType} prompt:`, error.message);
@@ -37,93 +20,46 @@ async function loadPromptByType(pool, promptType, company = null) {
} }
} }
/** export async function loadPromptsByType(pool, baseType, company = null) {
* Load all prompts for a task type (system, general, and optionally company-specific)
*
* @param {Object} pool - PostgreSQL pool
* @param {string} baseType - Base type name (e.g., 'name_validation', 'description_validation')
* @param {string|null} [company] - Optional company ID for company-specific prompts
* @returns {Promise<{system: string|null, general: string|null, companySpecific: string|null}>}
*/
async function loadPromptsByType(pool, baseType, company = null) {
const systemType = `${baseType}_system`; const systemType = `${baseType}_system`;
const generalType = `${baseType}_general`; const generalType = `${baseType}_general`;
const companyType = `${baseType}_company_specific`; const companyType = `${baseType}_company_specific`;
// Load system and general prompts in parallel
const [system, general] = await Promise.all([ const [system, general] = await Promise.all([
loadPromptByType(pool, systemType), loadPromptByType(pool, systemType),
loadPromptByType(pool, generalType) loadPromptByType(pool, generalType),
]); ]);
// Load company-specific prompt if company is provided
let companySpecific = null; let companySpecific = null;
if (company) { if (company) {
companySpecific = await loadPromptByType(pool, companyType, company); companySpecific = await loadPromptByType(pool, companyType, company);
} }
return { system, general, companySpecific };
return {
system,
general,
companySpecific
};
} }
/** export function loadNameValidationPrompts(pool, company = null) {
* Load name validation prompts
* @param {Object} pool - PostgreSQL pool
* @param {string|null} [company] - Optional company ID
* @returns {Promise<{system: string|null, general: string|null, companySpecific: string|null}>}
*/
async function loadNameValidationPrompts(pool, company = null) {
return loadPromptsByType(pool, 'name_validation', company); return loadPromptsByType(pool, 'name_validation', company);
} }
/** export function loadDescriptionValidationPrompts(pool, company = null) {
* Load description validation prompts
* @param {Object} pool - PostgreSQL pool
* @param {string|null} [company] - Optional company ID
* @returns {Promise<{system: string|null, general: string|null, companySpecific: string|null}>}
*/
async function loadDescriptionValidationPrompts(pool, company = null) {
return loadPromptsByType(pool, 'description_validation', company); return loadPromptsByType(pool, 'description_validation', company);
} }
/** export function loadSanityCheckPrompts(pool) {
* Load sanity check prompts (no company-specific variant)
* @param {Object} pool - PostgreSQL pool
* @returns {Promise<{system: string|null, general: string|null, companySpecific: null}>}
*/
async function loadSanityCheckPrompts(pool) {
return loadPromptsByType(pool, 'sanity_check', null); return loadPromptsByType(pool, 'sanity_check', null);
} }
/** export function loadBulkValidationPrompts(pool, company = null) {
* Load bulk validation prompts (GPT-5 validation)
* @param {Object} pool - PostgreSQL pool
* @param {string|null} [company] - Optional company ID
* @returns {Promise<{system: string|null, general: string|null, companySpecific: string|null}>}
*/
async function loadBulkValidationPrompts(pool, company = null) {
return loadPromptsByType(pool, 'bulk_validation', company); return loadPromptsByType(pool, 'bulk_validation', company);
} }
/** export async function loadBulkValidationPromptsForCompanies(pool, companyIds = []) {
* Load bulk validation prompts for multiple companies at once
* @param {Object} pool - PostgreSQL pool
* @param {string[]} companyIds - Array of company IDs
* @returns {Promise<{system: string|null, general: string|null, companyPrompts: Map<string, string>}>}
*/
async function loadBulkValidationPromptsForCompanies(pool, companyIds = []) {
// Load system and general prompts
const [system, general] = await Promise.all([ const [system, general] = await Promise.all([
loadPromptByType(pool, 'bulk_validation_system'), loadPromptByType(pool, 'bulk_validation_system'),
loadPromptByType(pool, 'bulk_validation_general') loadPromptByType(pool, 'bulk_validation_general'),
]); ]);
// Load company-specific prompts for all provided companies
const companyPrompts = new Map(); const companyPrompts = new Map();
if (companyIds.length > 0) { if (companyIds.length > 0) {
try { try {
const result = await pool.query( const result = await pool.query(
@@ -132,7 +68,6 @@ async function loadBulkValidationPromptsForCompanies(pool, companyIds = []) {
AND company = ANY($1)`, AND company = ANY($1)`,
[companyIds] [companyIds]
); );
for (const row of result.rows) { for (const row of result.rows) {
companyPrompts.set(row.company, row.prompt_text); companyPrompts.set(row.company, row.prompt_text);
} }
@@ -140,35 +75,14 @@ async function loadBulkValidationPromptsForCompanies(pool, companyIds = []) {
console.error('[PromptLoader] Error loading company-specific prompts:', error.message); console.error('[PromptLoader] Error loading company-specific prompts:', error.message);
} }
} }
return { system, general, companyPrompts };
return {
system,
general,
companyPrompts
};
} }
/** export function validateRequiredPrompts(prompts, baseType, options = {}) {
* Validate that required prompts exist, throw error if missing
* @param {Object} prompts - Prompts object from loadPromptsByType
* @param {string} baseType - Base type for error messages
* @param {Object} options - Validation options
* @param {boolean} [options.requireSystem=true] - Require system prompt
* @param {boolean} [options.requireGeneral=true] - Require general prompt
* @throws {Error} If required prompts are missing
*/
function validateRequiredPrompts(prompts, baseType, options = {}) {
const { requireSystem = true, requireGeneral = true } = options; const { requireSystem = true, requireGeneral = true } = options;
const missing = []; const missing = [];
if (requireSystem && !prompts.system) missing.push(`${baseType}_system`);
if (requireSystem && !prompts.system) { if (requireGeneral && !prompts.general) missing.push(`${baseType}_general`);
missing.push(`${baseType}_system`);
}
if (requireGeneral && !prompts.general) {
missing.push(`${baseType}_general`);
}
if (missing.length > 0) { if (missing.length > 0) {
throw new Error( throw new Error(
`Missing required AI prompts: ${missing.join(', ')}. ` + `Missing required AI prompts: ${missing.join(', ')}. ` +
@@ -176,19 +90,3 @@ function validateRequiredPrompts(prompts, baseType, options = {}) {
); );
} }
} }
module.exports = {
// Core loader
loadPromptByType,
loadPromptsByType,
// Task-specific loaders
loadNameValidationPrompts,
loadDescriptionValidationPrompts,
loadSanityCheckPrompts,
loadBulkValidationPrompts,
loadBulkValidationPromptsForCompanies,
// Validation
validateRequiredPrompts
};
@@ -1,21 +1,8 @@
/** /**
* Sanity Check Prompts * Sanity Check Prompts
*
* Functions for building and parsing batch product consistency validation prompts.
* System and general prompts are loaded from the database.
*/ */
/** export function buildSanityCheckUserPrompt(products, prompts) {
* Build the user prompt for sanity check
* Combines database prompts with product data
*
* @param {Object[]} products - Array of product data (limited fields for context)
* @param {Object} prompts - Prompts loaded from database
* @param {string} prompts.general - General sanity check rules
* @returns {string} Complete user prompt
*/
function buildSanityCheckUserPrompt(products, prompts) {
// Build a simplified product list for the prompt
const productSummaries = products.map((p, index) => ({ const productSummaries = products.map((p, index) => ({
index, index,
name: p.name, name: p.name,
@@ -33,22 +20,17 @@ function buildSanityCheckUserPrompt(products, prompts) {
weight: p.weight, weight: p.weight,
length: p.length, length: p.length,
width: p.width, width: p.width,
height: p.height height: p.height,
})); }));
const parts = []; const parts = [];
// Add general prompt/rules if provided
if (prompts.general) { if (prompts.general) {
parts.push(prompts.general); parts.push(prompts.general);
parts.push(''); // Empty line for separation parts.push('');
} }
// Add products to review
parts.push(`PRODUCTS TO REVIEW (${products.length} items):`); parts.push(`PRODUCTS TO REVIEW (${products.length} items):`);
parts.push(JSON.stringify(productSummaries, null, 2)); parts.push(JSON.stringify(productSummaries, null, 2));
// Add response format
parts.push(''); parts.push('');
parts.push('RESPOND WITH JSON:'); parts.push('RESPOND WITH JSON:');
parts.push(JSON.stringify({ parts.push(JSON.stringify({
@@ -57,10 +39,10 @@ function buildSanityCheckUserPrompt(products, prompts) {
productIndex: 0, productIndex: 0,
field: 'msrp', field: 'msrp',
issue: 'Description of the issue found', issue: 'Description of the issue found',
suggestion: 'Suggested fix or verification (optional)' suggestion: 'Suggested fix or verification (optional)',
} },
], ],
summary: '2-3 sentences summarizing the overall product quality' summary: '2-3 sentences summarizing the overall product quality',
}, null, 2)); }, null, 2));
parts.push(''); parts.push('');
@@ -69,60 +51,40 @@ function buildSanityCheckUserPrompt(products, prompts) {
return parts.join('\n'); return parts.join('\n');
} }
/** export function parseSanityCheckResponse(parsed, content) {
* Parse the AI response for sanity check
*
* @param {Object|null} parsed - Parsed JSON from AI
* @param {string} content - Raw response content
* @returns {Object}
*/
function parseSanityCheckResponse(parsed, content) {
// If we got valid parsed JSON, use it
if (parsed && Array.isArray(parsed.issues)) { if (parsed && Array.isArray(parsed.issues)) {
return { return {
issues: parsed.issues.map(issue => ({ issues: parsed.issues.map((issue) => ({
productIndex: issue.productIndex ?? issue.index ?? 0, productIndex: issue.productIndex ?? issue.index ?? 0,
field: issue.field || 'unknown', field: issue.field || 'unknown',
issue: issue.issue || issue.message || '', issue: issue.issue || issue.message || '',
suggestion: issue.suggestion || null suggestion: issue.suggestion || null,
})), })),
summary: parsed.summary || 'Review complete' summary: parsed.summary || 'Review complete',
}; };
} }
// Try to extract from content if parsing failed
try { try {
// Try to find issues array
const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/); const issuesMatch = content.match(/"issues"\s*:\s*\[([\s\S]*?)\]/);
let issues = []; let issues = [];
if (issuesMatch) { if (issuesMatch) {
// Try to parse the array content
try { try {
const arrayContent = `[${issuesMatch[1]}]`; const arrayContent = `[${issuesMatch[1]}]`;
const parsedIssues = JSON.parse(arrayContent); const parsedIssues = JSON.parse(arrayContent);
issues = parsedIssues.map(issue => ({ issues = parsedIssues.map((issue) => ({
productIndex: issue.productIndex ?? issue.index ?? 0, productIndex: issue.productIndex ?? issue.index ?? 0,
field: issue.field || 'unknown', field: issue.field || 'unknown',
issue: issue.issue || issue.message || '', issue: issue.issue || issue.message || '',
suggestion: issue.suggestion || null suggestion: issue.suggestion || null,
})); }));
} catch { } catch {
// Couldn't parse the array /* fall through */
} }
} }
// Try to find summary
const summaryMatch = content.match(/"summary"\s*:\s*"([^"]+)"/); const summaryMatch = content.match(/"summary"\s*:\s*"([^"]+)"/);
const summary = summaryMatch ? summaryMatch[1] : 'Review complete'; const summary = summaryMatch ? summaryMatch[1] : 'Review complete';
return { issues, summary }; return { issues, summary };
} catch { } catch {
return { issues: [], summary: 'Could not parse review results' }; return { issues: [], summary: 'Could not parse review results' };
} }
} }
module.exports = {
buildSanityCheckUserPrompt,
parseSanityCheckResponse
};
@@ -1,25 +1,15 @@
/** /**
* Groq Provider - Handles chat completions via Groq's OpenAI-compatible API * Groq Provider - chat completions via Groq's OpenAI-compatible API
*
* Uses Groq's fast inference for real-time AI validation tasks.
* Supports models like openai/gpt-oss-120b (complex) and openai/gpt-oss-20b (simple).
*/ */
const GROQ_BASE_URL = 'https://api.groq.com/openai/v1'; export const GROQ_BASE_URL = 'https://api.groq.com/openai/v1';
// Default models export const MODELS = {
const MODELS = { LARGE: 'openai/gpt-oss-120b',
LARGE: 'openai/gpt-oss-120b', // For complex tasks (descriptions, sanity checks) SMALL: 'openai/gpt-oss-20b',
SMALL: 'openai/gpt-oss-20b' // For simple tasks (name validation)
}; };
class GroqProvider { export class GroqProvider {
/**
* @param {Object} options
* @param {string} options.apiKey - Groq API key
* @param {string} [options.baseUrl] - Override base URL
* @param {number} [options.timeoutMs=30000] - Default timeout
*/
constructor({ apiKey, baseUrl = GROQ_BASE_URL, timeoutMs = 30000 }) { constructor({ apiKey, baseUrl = GROQ_BASE_URL, timeoutMs = 30000 }) {
if (!apiKey) { if (!apiKey) {
throw new Error('Groq API key is required'); throw new Error('Groq API key is required');
@@ -29,41 +19,25 @@ class GroqProvider {
this.timeoutMs = timeoutMs; this.timeoutMs = timeoutMs;
} }
/**
* Send a chat completion request
*
* @param {Object} params
* @param {Array<{role: string, content: string}>} params.messages - Conversation messages
* @param {string} [params.model] - Model to use (defaults to LARGE)
* @param {number} [params.temperature=0.3] - Response randomness (0-2)
* @param {number} [params.maxTokens=500] - Max tokens in response
* @param {Object} [params.responseFormat] - For JSON mode: { type: 'json_object' }
* @param {number} [params.timeoutMs] - Request timeout override
* @returns {Promise<{content: string, parsed: Object|null, usage: Object, latencyMs: number, model: string}>}
*/
async chatCompletion({ async chatCompletion({
messages, messages,
model = MODELS.LARGE, model = MODELS.LARGE,
temperature = 0.3, temperature = 0.3,
maxTokens = 500, maxTokens = 500,
responseFormat = null, responseFormat = null,
timeoutMs = this.timeoutMs timeoutMs = this.timeoutMs,
}) { }) {
const started = Date.now(); const started = Date.now();
const body = { const body = {
model, model,
messages, messages,
temperature, temperature,
max_completion_tokens: maxTokens max_completion_tokens: maxTokens,
}; };
// Enable JSON mode if requested
if (responseFormat?.type === 'json_object') { if (responseFormat?.type === 'json_object') {
body.response_format = { type: 'json_object' }; body.response_format = { type: 'json_object' };
} }
// Debug: Log request being sent
console.log('[Groq] Request:', { console.log('[Groq] Request:', {
model: body.model, model: body.model,
temperature: body.temperature, temperature: body.temperature,
@@ -71,12 +45,11 @@ class GroqProvider {
hasResponseFormat: !!body.response_format, hasResponseFormat: !!body.response_format,
messageCount: body.messages?.length, messageCount: body.messages?.length,
systemPromptLength: body.messages?.[0]?.content?.length, systemPromptLength: body.messages?.[0]?.content?.length,
userPromptLength: body.messages?.[1]?.content?.length userPromptLength: body.messages?.[1]?.content?.length,
}); });
const response = await this._makeRequest('chat/completions', body, timeoutMs); const response = await this._makeRequest('chat/completions', body, timeoutMs);
// Debug: Log raw response structure
console.log('[Groq] Raw response:', { console.log('[Groq] Raw response:', {
hasChoices: !!response.choices, hasChoices: !!response.choices,
choicesLength: response.choices?.length, choicesLength: response.choices?.length,
@@ -84,22 +57,20 @@ class GroqProvider {
finishReason: response.choices[0].finish_reason, finishReason: response.choices[0].finish_reason,
hasMessage: !!response.choices[0].message, hasMessage: !!response.choices[0].message,
contentLength: response.choices[0].message?.content?.length, contentLength: response.choices[0].message?.content?.length,
contentPreview: response.choices[0].message?.content?.substring(0, 200) contentPreview: response.choices[0].message?.content?.substring(0, 200),
} : null, } : null,
usage: response.usage, usage: response.usage,
model: response.model model: response.model,
}); });
const content = response.choices?.[0]?.message?.content || ''; const content = response.choices?.[0]?.message?.content || '';
const usage = response.usage || {}; const usage = response.usage || {};
// Attempt to parse JSON if response format was requested
let parsed = null; let parsed = null;
if (responseFormat && content) { if (responseFormat && content) {
try { try {
parsed = JSON.parse(content); parsed = JSON.parse(content);
} catch { } catch {
// Content isn't valid JSON - try to extract JSON from markdown
parsed = this._extractJson(content); parsed = this._extractJson(content);
} }
} }
@@ -110,74 +81,50 @@ class GroqProvider {
usage: { usage: {
promptTokens: usage.prompt_tokens || 0, promptTokens: usage.prompt_tokens || 0,
completionTokens: usage.completion_tokens || 0, completionTokens: usage.completion_tokens || 0,
totalTokens: usage.total_tokens || 0 totalTokens: usage.total_tokens || 0,
}, },
latencyMs: Date.now() - started, latencyMs: Date.now() - started,
model: response.model || model model: response.model || model,
}; };
} }
/**
* Extract JSON from content that might be wrapped in markdown code blocks
* @private
*/
_extractJson(content) { _extractJson(content) {
// Try to find JSON in code blocks
const codeBlockMatch = content.match(/```(?:json)?\s*([\s\S]*?)```/); const codeBlockMatch = content.match(/```(?:json)?\s*([\s\S]*?)```/);
if (codeBlockMatch) { if (codeBlockMatch) {
try { try { return JSON.parse(codeBlockMatch[1].trim()); } catch { /* fall through */ }
return JSON.parse(codeBlockMatch[1].trim());
} catch {
// Fall through
}
} }
// Try to find JSON object/array directly
const jsonMatch = content.match(/(\{[\s\S]*\}|\[[\s\S]*\])/); const jsonMatch = content.match(/(\{[\s\S]*\}|\[[\s\S]*\])/);
if (jsonMatch) { if (jsonMatch) {
try { try { return JSON.parse(jsonMatch[1]); } catch { /* fall through */ }
return JSON.parse(jsonMatch[1]);
} catch {
// Fall through
}
} }
return null; return null;
} }
/**
* Make an HTTP request to Groq API
* @private
*/
async _makeRequest(endpoint, body, timeoutMs) { async _makeRequest(endpoint, body, timeoutMs) {
const controller = new AbortController(); const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs); const timeout = setTimeout(() => controller.abort(), timeoutMs);
try { try {
const response = await fetch(`${this.baseUrl}/${endpoint}`, { const response = await fetch(`${this.baseUrl}/${endpoint}`, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
'Authorization': `Bearer ${this.apiKey}` 'Authorization': `Bearer ${this.apiKey}`,
}, },
body: JSON.stringify(body), body: JSON.stringify(body),
signal: controller.signal signal: controller.signal,
}); });
if (!response.ok) { if (!response.ok) {
const error = await response.json().catch(() => ({})); const error = await response.json().catch(() => ({}));
const message = error.error?.message || `Groq API error: ${response.status}`; const message = error.error?.message || `Groq API error: ${response.status}`;
const err = new Error(message); const err = new Error(message);
err.status = response.status; err.status = response.status;
err.code = error.error?.code; err.code = error.error?.code;
// Include failed_generation if available (for JSON mode failures)
if (error.error?.failed_generation) { if (error.error?.failed_generation) {
err.failedGeneration = error.error.failed_generation; err.failedGeneration = error.error.failed_generation;
console.error('[Groq] JSON validation failed. Model output:', error.error.failed_generation); console.error('[Groq] JSON validation failed. Model output:', error.error.failed_generation);
} }
throw err; throw err;
} }
return response.json(); return response.json();
} catch (error) { } catch (error) {
if (error.name === 'AbortError') { if (error.name === 'AbortError') {
@@ -191,13 +138,7 @@ class GroqProvider {
} }
} }
/**
* Check if the provider is properly configured
* @returns {boolean}
*/
isConfigured() { isConfigured() {
return !!this.apiKey; return !!this.apiKey;
} }
} }
module.exports = { GroqProvider, MODELS, GROQ_BASE_URL };
@@ -2,11 +2,11 @@
* OpenAI Provider - Handles embedding generation * OpenAI Provider - Handles embedding generation
*/ */
const EMBEDDING_MODEL = 'text-embedding-3-small'; export const EMBEDDING_MODEL = 'text-embedding-3-small';
const EMBEDDING_DIMENSIONS = 1536; export const EMBEDDING_DIMENSIONS = 1536;
const MAX_BATCH_SIZE = 2048; const MAX_BATCH_SIZE = 2048;
class OpenAIProvider { export class OpenAIProvider {
constructor({ apiKey, baseUrl = 'https://api.openai.com/v1', timeoutMs = 60000 }) { constructor({ apiKey, baseUrl = 'https://api.openai.com/v1', timeoutMs = 60000 }) {
if (!apiKey) { if (!apiKey) {
throw new Error('OpenAI API key is required'); throw new Error('OpenAI API key is required');
@@ -16,12 +16,6 @@ class OpenAIProvider {
this.timeoutMs = timeoutMs; this.timeoutMs = timeoutMs;
} }
/**
* Generate embeddings for one or more texts
* @param {string|string[]} input - Text or array of texts
* @param {Object} options
* @returns {Promise<{embeddings: number[][], usage: Object, model: string, latencyMs: number}>}
*/
async embed(input, options = {}) { async embed(input, options = {}) {
const texts = Array.isArray(input) ? input : [input]; const texts = Array.isArray(input) ? input : [input];
const model = options.model || EMBEDDING_MODEL; const model = options.model || EMBEDDING_MODEL;
@@ -33,56 +27,39 @@ class OpenAIProvider {
} }
const started = Date.now(); const started = Date.now();
const cleanedTexts = texts.map((t) =>
// Clean and truncate input texts
const cleanedTexts = texts.map(t =>
(t || '').replace(/\n+/g, ' ').trim().substring(0, 8000) (t || '').replace(/\n+/g, ' ').trim().substring(0, 8000)
); );
const body = { const body = { input: cleanedTexts, model, encoding_format: 'float' };
input: cleanedTexts, if (model.includes('embedding-3')) body.dimensions = dimensions;
model,
encoding_format: 'float'
};
// Only embedding-3 models support dimensions parameter
if (model.includes('embedding-3')) {
body.dimensions = dimensions;
}
const response = await this._makeRequest('embeddings', body, timeoutMs); const response = await this._makeRequest('embeddings', body, timeoutMs);
// Sort by index to ensure order matches input
const sortedData = response.data.sort((a, b) => a.index - b.index); const sortedData = response.data.sort((a, b) => a.index - b.index);
return { return {
embeddings: sortedData.map(item => item.embedding), embeddings: sortedData.map((item) => item.embedding),
usage: { usage: {
promptTokens: response.usage?.prompt_tokens || 0, promptTokens: response.usage?.prompt_tokens || 0,
totalTokens: response.usage?.total_tokens || 0 totalTokens: response.usage?.total_tokens || 0,
}, },
model: response.model || model, model: response.model || model,
latencyMs: Date.now() - started latencyMs: Date.now() - started,
}; };
} }
/**
* Generator for processing large batches in chunks
*/
async *embedBatchChunked(texts, options = {}) { async *embedBatchChunked(texts, options = {}) {
const batchSize = Math.min(options.batchSize || 100, MAX_BATCH_SIZE); const batchSize = Math.min(options.batchSize || 100, MAX_BATCH_SIZE);
for (let i = 0; i < texts.length; i += batchSize) { for (let i = 0; i < texts.length; i += batchSize) {
const chunk = texts.slice(i, i + batchSize); const chunk = texts.slice(i, i + batchSize);
const result = await this.embed(chunk, options); const result = await this.embed(chunk, options);
yield { yield {
embeddings: result.embeddings, embeddings: result.embeddings,
startIndex: i, startIndex: i,
endIndex: i + chunk.length, endIndex: i + chunk.length,
usage: result.usage, usage: result.usage,
model: result.model, model: result.model,
latencyMs: result.latencyMs latencyMs: result.latencyMs,
}; };
} }
} }
@@ -90,28 +67,23 @@ class OpenAIProvider {
async _makeRequest(endpoint, body, timeoutMs) { async _makeRequest(endpoint, body, timeoutMs) {
const controller = new AbortController(); const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs); const timeout = setTimeout(() => controller.abort(), timeoutMs);
try { try {
const response = await fetch(`${this.baseUrl}/${endpoint}`, { const response = await fetch(`${this.baseUrl}/${endpoint}`, {
method: 'POST', method: 'POST',
headers: { headers: {
'Content-Type': 'application/json', 'Content-Type': 'application/json',
'Authorization': `Bearer ${this.apiKey}` 'Authorization': `Bearer ${this.apiKey}`,
}, },
body: JSON.stringify(body), body: JSON.stringify(body),
signal: controller.signal signal: controller.signal,
}); });
if (!response.ok) { if (!response.ok) {
const error = await response.json().catch(() => ({})); const error = await response.json().catch(() => ({}));
throw new Error(error.error?.message || `OpenAI API error: ${response.status}`); throw new Error(error.error?.message || `OpenAI API error: ${response.status}`);
} }
return response.json(); return response.json();
} finally { } finally {
clearTimeout(timeout); clearTimeout(timeout);
} }
} }
} }
module.exports = { OpenAIProvider, EMBEDDING_MODEL, EMBEDDING_DIMENSIONS };
@@ -1,132 +1,84 @@
/** /**
* Description Validation Task * Description Validation Task
*
* Validates a product description for quality, accuracy, and guideline compliance.
* Uses Groq with the larger model for better reasoning about content quality.
* Loads all prompts from the database (no hardcoded prompts).
*/ */
const { MODELS } = require('../providers/groqProvider'); import { MODELS } from '../providers/groqProvider.js';
const { import {
loadDescriptionValidationPrompts, loadDescriptionValidationPrompts,
validateRequiredPrompts validateRequiredPrompts,
} = require('../prompts/promptLoader'); } from '../prompts/promptLoader.js';
const { import {
buildDescriptionUserPrompt, buildDescriptionUserPrompt,
parseDescriptionResponse parseDescriptionResponse,
} = require('../prompts/descriptionPrompts'); } from '../prompts/descriptionPrompts.js';
const TASK_ID = 'validate.description'; export const TASK_ID = 'validate.description';
/** export function createDescriptionValidationTask() {
* Create the description validation task
*
* @returns {Object} Task definition
*/
function createDescriptionValidationTask() {
return { return {
id: TASK_ID, id: TASK_ID,
description: 'Validate product description for quality and guideline compliance', description: 'Validate product description for quality and guideline compliance',
/**
* Run the description validation
*
* @param {Object} payload
* @param {Object} payload.product - Product data
* @param {string} payload.product.name - Product name (for context)
* @param {string} payload.product.description - Description to validate
* @param {string} [payload.product.company_name] - Company name
* @param {string} [payload.product.company_id] - Company ID for loading specific rules
* @param {string} [payload.product.categories] - Product categories
* @param {Object} payload.provider - Groq provider instance
* @param {Object} payload.pool - PostgreSQL pool
* @param {Object} [payload.logger] - Logger instance
* @returns {Promise<Object>}
*/
async run(payload) { async run(payload) {
const { product, provider, pool, logger } = payload; const { product, provider, pool, logger } = payload;
const log = logger || console; const log = logger || console;
// Validate required input
if (!product?.name && !product?.description) { if (!product?.name && !product?.description) {
return { return { isValid: true, suggestion: null, issues: [], skipped: true, reason: 'No name or description provided' };
isValid: true,
suggestion: null,
issues: [],
skipped: true,
reason: 'No name or description provided'
};
}
if (!provider) {
throw new Error('Groq provider not available');
}
if (!pool) {
throw new Error('Database pool not available');
} }
if (!provider) throw new Error('Groq provider not available');
if (!pool) throw new Error('Database pool not available');
try { try {
// Load prompts from database
const companyKey = product.company_id || product.company_name || product.company; const companyKey = product.company_id || product.company_name || product.company;
const prompts = await loadDescriptionValidationPrompts(pool, companyKey); const prompts = await loadDescriptionValidationPrompts(pool, companyKey);
// Validate required prompts exist
validateRequiredPrompts(prompts, 'description_validation'); validateRequiredPrompts(prompts, 'description_validation');
// Build the user prompt with database-loaded prompts
const userPrompt = buildDescriptionUserPrompt(product, prompts); const userPrompt = buildDescriptionUserPrompt(product, prompts);
let response; let response;
let result; let result;
try { try {
// Try with JSON mode first
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.LARGE, // openai/gpt-oss-120b - better for content analysis model: MODELS.LARGE,
temperature: 0.3, // Slightly higher for creative suggestions temperature: 0.3,
maxTokens: 2000, // Reasoning models need extra tokens for thinking maxTokens: 2000,
responseFormat: { type: 'json_object' } responseFormat: { type: 'json_object' },
}); });
// Log full raw response for debugging
log.info('[DescriptionValidation] Raw AI response:', { log.info('[DescriptionValidation] Raw AI response:', {
parsed: response.parsed, parsed: response.parsed,
content: response.content, content: response.content,
contentLength: response.content?.length contentLength: response.content?.length,
}); });
// Parse the response
result = parseDescriptionResponse(response.parsed, response.content); result = parseDescriptionResponse(response.parsed, response.content);
} catch (jsonError) { } catch (jsonError) {
// If JSON mode failed, check if we have failedGeneration to parse
if (jsonError.failedGeneration) { if (jsonError.failedGeneration) {
log.warn('[DescriptionValidation] JSON mode failed, attempting to parse failed_generation:', { log.warn('[DescriptionValidation] JSON mode failed, attempting to parse failed_generation:', {
failedGeneration: jsonError.failedGeneration failedGeneration: jsonError.failedGeneration,
}); });
result = parseDescriptionResponse(null, jsonError.failedGeneration); result = parseDescriptionResponse(null, jsonError.failedGeneration);
response = { latencyMs: 0, usage: {}, model: MODELS.LARGE }; response = { latencyMs: 0, usage: {}, model: MODELS.LARGE };
} else { } else {
// Retry without JSON mode
log.warn('[DescriptionValidation] JSON mode failed, retrying without JSON mode'); log.warn('[DescriptionValidation] JSON mode failed, retrying without JSON mode');
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.LARGE, model: MODELS.LARGE,
temperature: 0.3, temperature: 0.3,
maxTokens: 2000 // Reasoning models need extra tokens for thinking maxTokens: 2000,
// No responseFormat - let the model respond freely
}); });
log.info('[DescriptionValidation] Raw AI response (no JSON mode):', { log.info('[DescriptionValidation] Raw AI response (no JSON mode):', {
parsed: response.parsed, parsed: response.parsed,
content: response.content, content: response.content,
contentLength: response.content?.length contentLength: response.content?.length,
}); });
result = parseDescriptionResponse(response.parsed, response.content); result = parseDescriptionResponse(response.parsed, response.content);
} }
@@ -135,24 +87,19 @@ function createDescriptionValidationTask() {
log.info(`[DescriptionValidation] Validated description for "${product.name}" in ${response.latencyMs}ms`, { log.info(`[DescriptionValidation] Validated description for "${product.name}" in ${response.latencyMs}ms`, {
isValid: result.isValid, isValid: result.isValid,
hasSuggestion: !!result.suggestion, hasSuggestion: !!result.suggestion,
issueCount: result.issues.length issueCount: result.issues.length,
}); });
return { return {
...result, ...result,
latencyMs: response.latencyMs, latencyMs: response.latencyMs,
usage: response.usage, usage: response.usage,
model: response.model model: response.model,
}; };
} catch (error) { } catch (error) {
log.error('[DescriptionValidation] Error:', error.message); log.error('[DescriptionValidation] Error:', error.message);
throw error; throw error;
} }
} },
}; };
} }
module.exports = {
TASK_ID,
createDescriptionValidationTask
};
+18 -113
View File
@@ -1,166 +1,87 @@
/** /**
* AI Task Registry * AI Task Registry
*
* Simple registry pattern for AI tasks. Each task has:
* - id: Unique identifier
* - run: Async function that executes the task
*
* This allows adding new AI capabilities without modifying core code.
*/ */
const { createNameValidationTask, TASK_ID: NAME_TASK_ID } = require('./nameValidationTask'); import { createNameValidationTask, TASK_ID as NAME_TASK_ID } from './nameValidationTask.js';
const { createDescriptionValidationTask, TASK_ID: DESC_TASK_ID } = require('./descriptionValidationTask'); import { createDescriptionValidationTask, TASK_ID as DESC_TASK_ID } from './descriptionValidationTask.js';
const { createSanityCheckTask, TASK_ID: SANITY_TASK_ID } = require('./sanityCheckTask'); import { createSanityCheckTask, TASK_ID as SANITY_TASK_ID } from './sanityCheckTask.js';
/** export { createNameValidationTask, createDescriptionValidationTask, createSanityCheckTask };
* Task IDs - frozen constants for type safety
*/ export const TASK_IDS = Object.freeze({
const TASK_IDS = Object.freeze({
// Inline validation (triggered on field blur)
VALIDATE_NAME: NAME_TASK_ID, VALIDATE_NAME: NAME_TASK_ID,
VALIDATE_DESCRIPTION: DESC_TASK_ID, VALIDATE_DESCRIPTION: DESC_TASK_ID,
SANITY_CHECK: SANITY_TASK_ID,
// Batch operations (triggered on user action)
SANITY_CHECK: SANITY_TASK_ID
}); });
/** export class TaskRegistry {
* Task Registry
*/
class TaskRegistry {
constructor() { constructor() {
this.tasks = new Map(); this.tasks = new Map();
} }
/**
* Register a task
* @param {Object} task
* @param {string} task.id - Unique task identifier
* @param {Function} task.run - Async function: (payload) => result
* @param {string} [task.description] - Human-readable description
*/
register(task) { register(task) {
if (!task?.id) { if (!task?.id) throw new Error('Task must have an id');
throw new Error('Task must have an id'); if (typeof task.run !== 'function') throw new Error(`Task ${task.id} must have a run function`);
} if (this.tasks.has(task.id)) throw new Error(`Task ${task.id} is already registered`);
if (typeof task.run !== 'function') {
throw new Error(`Task ${task.id} must have a run function`);
}
if (this.tasks.has(task.id)) {
throw new Error(`Task ${task.id} is already registered`);
}
this.tasks.set(task.id, task); this.tasks.set(task.id, task);
return this; return this;
} }
/**
* Get a task by ID
* @param {string} taskId
* @returns {Object|null}
*/
get(taskId) { get(taskId) {
return this.tasks.get(taskId) || null; return this.tasks.get(taskId) || null;
} }
/**
* Check if a task exists
* @param {string} taskId
* @returns {boolean}
*/
has(taskId) { has(taskId) {
return this.tasks.has(taskId); return this.tasks.has(taskId);
} }
/**
* Run a task by ID
* @param {string} taskId
* @param {Object} payload - Task-specific input
* @returns {Promise<Object>} Task result
*/
async runTask(taskId, payload = {}) { async runTask(taskId, payload = {}) {
const task = this.get(taskId); const task = this.get(taskId);
if (!task) { if (!task) throw new Error(`Unknown task: ${taskId}`);
throw new Error(`Unknown task: ${taskId}`);
}
try { try {
const result = await task.run(payload); const result = await task.run(payload);
return { return { success: true, taskId, ...result };
success: true,
taskId,
...result
};
} catch (error) { } catch (error) {
return { return {
success: false, success: false,
taskId, taskId,
error: error.message, error: error.message,
code: error.code code: error.code,
}; };
} }
} }
/**
* List all registered task IDs
* @returns {string[]}
*/
list() { list() {
return Array.from(this.tasks.keys()); return Array.from(this.tasks.keys());
} }
/**
* Get count of registered tasks
* @returns {number}
*/
size() { size() {
return this.tasks.size; return this.tasks.size;
} }
} }
// Singleton instance
let registry = null; let registry = null;
/** export function getRegistry() {
* Get or create the task registry if (!registry) registry = new TaskRegistry();
* @returns {TaskRegistry}
*/
function getRegistry() {
if (!registry) {
registry = new TaskRegistry();
}
return registry; return registry;
} }
/** export function resetRegistry() {
* Reset the registry (mainly for testing)
*/
function resetRegistry() {
registry = null; registry = null;
} }
/** export function registerAllTasks(logger = console) {
* Register all validation tasks with the registry
* Call this during initialization after the registry is created
*
* @param {Object} [logger] - Optional logger
*/
function registerAllTasks(logger = console) {
const reg = getRegistry(); const reg = getRegistry();
// Register name validation
if (!reg.has(TASK_IDS.VALIDATE_NAME)) { if (!reg.has(TASK_IDS.VALIDATE_NAME)) {
reg.register(createNameValidationTask()); reg.register(createNameValidationTask());
logger.info(`[Tasks] Registered: ${TASK_IDS.VALIDATE_NAME}`); logger.info(`[Tasks] Registered: ${TASK_IDS.VALIDATE_NAME}`);
} }
// Register description validation
if (!reg.has(TASK_IDS.VALIDATE_DESCRIPTION)) { if (!reg.has(TASK_IDS.VALIDATE_DESCRIPTION)) {
reg.register(createDescriptionValidationTask()); reg.register(createDescriptionValidationTask());
logger.info(`[Tasks] Registered: ${TASK_IDS.VALIDATE_DESCRIPTION}`); logger.info(`[Tasks] Registered: ${TASK_IDS.VALIDATE_DESCRIPTION}`);
} }
// Register sanity check
if (!reg.has(TASK_IDS.SANITY_CHECK)) { if (!reg.has(TASK_IDS.SANITY_CHECK)) {
reg.register(createSanityCheckTask()); reg.register(createSanityCheckTask());
logger.info(`[Tasks] Registered: ${TASK_IDS.SANITY_CHECK}`); logger.info(`[Tasks] Registered: ${TASK_IDS.SANITY_CHECK}`);
@@ -168,19 +89,3 @@ function registerAllTasks(logger = console) {
return reg; return reg;
} }
module.exports = {
// Constants
TASK_IDS,
// Registry
TaskRegistry,
getRegistry,
resetRegistry,
registerAllTasks,
// Task factories (for custom registration)
createNameValidationTask,
createDescriptionValidationTask,
createSanityCheckTask
};
@@ -1,77 +1,38 @@
/** /**
* Name Validation Task * Name Validation Task
*
* Validates a product name for spelling, grammar, and naming conventions.
* Uses Groq with the smaller model for fast response times.
* Loads all prompts from the database (no hardcoded prompts).
*/ */
const { MODELS } = require('../providers/groqProvider'); import { MODELS } from '../providers/groqProvider.js';
const { import {
loadNameValidationPrompts, loadNameValidationPrompts,
validateRequiredPrompts validateRequiredPrompts,
} = require('../prompts/promptLoader'); } from '../prompts/promptLoader.js';
const { import {
buildNameUserPrompt, buildNameUserPrompt,
parseNameResponse parseNameResponse,
} = require('../prompts/namePrompts'); } from '../prompts/namePrompts.js';
const TASK_ID = 'validate.name'; export const TASK_ID = 'validate.name';
/** export function createNameValidationTask() {
* Create the name validation task
*
* @returns {Object} Task definition
*/
function createNameValidationTask() {
return { return {
id: TASK_ID, id: TASK_ID,
description: 'Validate product name for spelling, grammar, and conventions', description: 'Validate product name for spelling, grammar, and conventions',
/**
* Run the name validation
*
* @param {Object} payload
* @param {Object} payload.product - Product data
* @param {string} payload.product.name - Product name to validate
* @param {string} [payload.product.company_name] - Company name
* @param {string} [payload.product.company_id] - Company ID for loading specific rules
* @param {string} [payload.product.line_name] - Product line
* @param {string} [payload.product.description] - Description for context
* @param {Object} payload.provider - Groq provider instance
* @param {Object} payload.pool - PostgreSQL pool
* @param {Object} [payload.logger] - Logger instance
* @returns {Promise<Object>}
*/
async run(payload) { async run(payload) {
const { product, provider, pool, logger } = payload; const { product, provider, pool, logger } = payload;
const log = logger || console; const log = logger || console;
// Validate required input
if (!product?.name) { if (!product?.name) {
return { return { isValid: true, suggestion: null, issues: [], skipped: true, reason: 'No name provided' };
isValid: true,
suggestion: null,
issues: [],
skipped: true,
reason: 'No name provided'
};
}
if (!provider) {
throw new Error('Groq provider not available');
}
if (!pool) {
throw new Error('Database pool not available');
} }
if (!provider) throw new Error('Groq provider not available');
if (!pool) throw new Error('Database pool not available');
try { try {
// Load prompts from database
const companyKey = product.company_id || product.company_name || product.company; const companyKey = product.company_id || product.company_name || product.company;
const prompts = await loadNameValidationPrompts(pool, companyKey); const prompts = await loadNameValidationPrompts(pool, companyKey);
// Debug: Log loaded prompts
log.info('[NameValidation] Loaded prompts:', { log.info('[NameValidation] Loaded prompts:', {
hasSystem: !!prompts.system, hasSystem: !!prompts.system,
systemLength: prompts.system?.length || 0, systemLength: prompts.system?.length || 0,
@@ -79,68 +40,57 @@ function createNameValidationTask() {
generalLength: prompts.general?.length || 0, generalLength: prompts.general?.length || 0,
generalPreview: prompts.general?.substring(0, 100) || '(empty)', generalPreview: prompts.general?.substring(0, 100) || '(empty)',
hasCompanySpecific: !!prompts.companySpecific, hasCompanySpecific: !!prompts.companySpecific,
companyKey companyKey,
}); });
// Validate required prompts exist
validateRequiredPrompts(prompts, 'name_validation'); validateRequiredPrompts(prompts, 'name_validation');
// Build the user prompt with database-loaded prompts
const userPrompt = buildNameUserPrompt(product, prompts); const userPrompt = buildNameUserPrompt(product, prompts);
// Debug: Log the full user prompt being sent
log.info('[NameValidation] User prompt:', userPrompt.substring(0, 500)); log.info('[NameValidation] User prompt:', userPrompt.substring(0, 500));
let response; let response;
let result; let result;
try { try {
// Try with JSON mode first
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.LARGE, // openai/gpt-oss-120b - reasoning model model: MODELS.LARGE,
temperature: 0.2, // Low temperature for consistent results temperature: 0.2,
maxTokens: 3000, // Reasoning models need extra tokens for thinking maxTokens: 3000,
responseFormat: { type: 'json_object' } responseFormat: { type: 'json_object' },
}); });
// Log full raw response for debugging
log.info('[NameValidation] Raw AI response:', { log.info('[NameValidation] Raw AI response:', {
parsed: response.parsed, parsed: response.parsed,
content: response.content, content: response.content,
contentLength: response.content?.length contentLength: response.content?.length,
}); });
// Parse the response
result = parseNameResponse(response.parsed, response.content); result = parseNameResponse(response.parsed, response.content);
} catch (jsonError) { } catch (jsonError) {
// If JSON mode failed, check if we have failedGeneration to parse
if (jsonError.failedGeneration) { if (jsonError.failedGeneration) {
log.warn('[NameValidation] JSON mode failed, attempting to parse failed_generation:', { log.warn('[NameValidation] JSON mode failed, attempting to parse failed_generation:', {
failedGeneration: jsonError.failedGeneration failedGeneration: jsonError.failedGeneration,
}); });
result = parseNameResponse(null, jsonError.failedGeneration); result = parseNameResponse(null, jsonError.failedGeneration);
response = { latencyMs: 0, usage: {}, model: MODELS.SMALL }; response = { latencyMs: 0, usage: {}, model: MODELS.SMALL };
} else { } else {
// Retry without JSON mode
log.warn('[NameValidation] JSON mode failed, retrying without JSON mode'); log.warn('[NameValidation] JSON mode failed, retrying without JSON mode');
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.SMALL, model: MODELS.SMALL,
temperature: 0.2, temperature: 0.2,
maxTokens: 1500 // Reasoning models need extra tokens for thinking maxTokens: 1500,
// No responseFormat - let the model respond freely
}); });
log.info('[NameValidation] Raw AI response (no JSON mode):', { log.info('[NameValidation] Raw AI response (no JSON mode):', {
parsed: response.parsed, parsed: response.parsed,
content: response.content, content: response.content,
contentLength: response.content?.length contentLength: response.content?.length,
}); });
result = parseNameResponse(response.parsed, response.content); result = parseNameResponse(response.parsed, response.content);
} }
@@ -149,24 +99,19 @@ function createNameValidationTask() {
log.info(`[NameValidation] Validated "${product.name}" in ${response.latencyMs}ms`, { log.info(`[NameValidation] Validated "${product.name}" in ${response.latencyMs}ms`, {
isValid: result.isValid, isValid: result.isValid,
hassuggestion: !!result.suggestion, hassuggestion: !!result.suggestion,
issueCount: result.issues.length issueCount: result.issues.length,
}); });
return { return {
...result, ...result,
latencyMs: response.latencyMs, latencyMs: response.latencyMs,
usage: response.usage, usage: response.usage,
model: response.model model: response.model,
}; };
} catch (error) { } catch (error) {
log.error('[NameValidation] Error:', error.message); log.error('[NameValidation] Error:', error.message);
throw error; throw error;
} }
} },
}; };
} }
module.exports = {
TASK_ID,
createNameValidationTask
};
@@ -1,96 +1,55 @@
/** /**
* Sanity Check Task * Sanity Check Task
*
* Reviews a batch of products for consistency and appropriateness.
* Uses Groq with the larger model for complex batch analysis.
* Loads all prompts from the database (no hardcoded prompts).
*/ */
const { MODELS } = require('../providers/groqProvider'); import { MODELS } from '../providers/groqProvider.js';
const { import {
loadSanityCheckPrompts, loadSanityCheckPrompts,
validateRequiredPrompts validateRequiredPrompts,
} = require('../prompts/promptLoader'); } from '../prompts/promptLoader.js';
const { import {
buildSanityCheckUserPrompt, buildSanityCheckUserPrompt,
parseSanityCheckResponse parseSanityCheckResponse,
} = require('../prompts/sanityCheckPrompts'); } from '../prompts/sanityCheckPrompts.js';
const TASK_ID = 'sanity.check'; export const TASK_ID = 'sanity.check';
export const MAX_PRODUCTS_PER_REQUEST = 50;
// Maximum products to send in a single request (to avoid token limits) export function createSanityCheckTask() {
const MAX_PRODUCTS_PER_REQUEST = 50;
/**
* Create the sanity check task
*
* @returns {Object} Task definition
*/
function createSanityCheckTask() {
return { return {
id: TASK_ID, id: TASK_ID,
description: 'Review batch of products for consistency and appropriateness', description: 'Review batch of products for consistency and appropriateness',
/**
* Run the sanity check
*
* @param {Object} payload
* @param {Object[]} payload.products - Array of products to check
* @param {Object} payload.provider - Groq provider instance
* @param {Object} payload.pool - PostgreSQL pool
* @param {Object} [payload.logger] - Logger instance
* @returns {Promise<Object>}
*/
async run(payload) { async run(payload) {
const { products, provider, pool, logger } = payload; const { products, provider, pool, logger } = payload;
const log = logger || console; const log = logger || console;
// Validate required input
if (!Array.isArray(products) || products.length === 0) { if (!Array.isArray(products) || products.length === 0) {
return { return { issues: [], summary: 'No products to check', skipped: true };
issues: [],
summary: 'No products to check',
skipped: true
};
}
if (!provider) {
throw new Error('Groq provider not available');
}
if (!pool) {
throw new Error('Database pool not available');
} }
if (!provider) throw new Error('Groq provider not available');
if (!pool) throw new Error('Database pool not available');
try { try {
// Load prompts from database
const prompts = await loadSanityCheckPrompts(pool); const prompts = await loadSanityCheckPrompts(pool);
// Validate required prompts exist
validateRequiredPrompts(prompts, 'sanity_check'); validateRequiredPrompts(prompts, 'sanity_check');
// If batch is small enough, process in one request
if (products.length <= MAX_PRODUCTS_PER_REQUEST) { if (products.length <= MAX_PRODUCTS_PER_REQUEST) {
return await checkBatch(products, prompts, provider, log); return await checkBatch(products, prompts, provider, log);
} }
// Otherwise, process in chunks and combine results
log.info(`[SanityCheck] Processing ${products.length} products in chunks`); log.info(`[SanityCheck] Processing ${products.length} products in chunks`);
const allIssues = []; const allIssues = [];
const summaries = []; const summaries = [];
for (let i = 0; i < products.length; i += MAX_PRODUCTS_PER_REQUEST) { for (let i = 0; i < products.length; i += MAX_PRODUCTS_PER_REQUEST) {
const chunk = products.slice(i, i + MAX_PRODUCTS_PER_REQUEST); const chunk = products.slice(i, i + MAX_PRODUCTS_PER_REQUEST);
const chunkOffset = i; // To adjust product indices in results const chunkOffset = i;
const result = await checkBatch(chunk, prompts, provider, log); const result = await checkBatch(chunk, prompts, provider, log);
const adjustedIssues = result.issues.map((issue) => ({
// Adjust product indices to match original array
const adjustedIssues = result.issues.map(issue => ({
...issue, ...issue,
productIndex: issue.productIndex + chunkOffset productIndex: issue.productIndex + chunkOffset,
})); }));
allIssues.push(...adjustedIssues); allIssues.push(...adjustedIssues);
summaries.push(result.summary); summaries.push(result.summary);
} }
@@ -101,82 +60,61 @@ function createSanityCheckTask() {
? `Reviewed ${products.length} products in ${summaries.length} batches. ${allIssues.length} issues found.` ? `Reviewed ${products.length} products in ${summaries.length} batches. ${allIssues.length} issues found.`
: summaries[0], : summaries[0],
totalProducts: products.length, totalProducts: products.length,
issueCount: allIssues.length issueCount: allIssues.length,
}; };
} catch (error) { } catch (error) {
log.error('[SanityCheck] Error:', error.message); log.error('[SanityCheck] Error:', error.message);
throw error; throw error;
} }
} },
}; };
} }
/**
* Check a single batch of products
*
* @param {Object[]} products - Products to check
* @param {Object} prompts - Loaded prompts from database
* @param {Object} provider - Groq provider
* @param {Object} log - Logger
* @returns {Promise<Object>}
*/
async function checkBatch(products, prompts, provider, log) { async function checkBatch(products, prompts, provider, log) {
const userPrompt = buildSanityCheckUserPrompt(products, prompts); const userPrompt = buildSanityCheckUserPrompt(products, prompts);
let response; let response;
let result; let result;
try { try {
// Try with JSON mode first
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.LARGE, // openai/gpt-oss-120b - needed for complex batch analysis model: MODELS.LARGE,
temperature: 0.2, // Low temperature for consistent analysis temperature: 0.2,
maxTokens: 2000, // More tokens for batch results maxTokens: 2000,
responseFormat: { type: 'json_object' } responseFormat: { type: 'json_object' },
}); });
result = parseSanityCheckResponse(response.parsed, response.content); result = parseSanityCheckResponse(response.parsed, response.content);
} catch (jsonError) { } catch (jsonError) {
// If JSON mode failed, check if we have failedGeneration to parse
if (jsonError.failedGeneration) { if (jsonError.failedGeneration) {
log.warn('[SanityCheck] JSON mode failed, attempting to parse failed_generation'); log.warn('[SanityCheck] JSON mode failed, attempting to parse failed_generation');
result = parseSanityCheckResponse(null, jsonError.failedGeneration); result = parseSanityCheckResponse(null, jsonError.failedGeneration);
response = { latencyMs: 0, usage: {}, model: MODELS.LARGE }; response = { latencyMs: 0, usage: {}, model: MODELS.LARGE };
} else { } else {
// Retry without JSON mode
log.warn('[SanityCheck] JSON mode failed, retrying without JSON mode'); log.warn('[SanityCheck] JSON mode failed, retrying without JSON mode');
response = await provider.chatCompletion({ response = await provider.chatCompletion({
messages: [ messages: [
{ role: 'system', content: prompts.system }, { role: 'system', content: prompts.system },
{ role: 'user', content: userPrompt } { role: 'user', content: userPrompt },
], ],
model: MODELS.LARGE, model: MODELS.LARGE,
temperature: 0.2, temperature: 0.2,
maxTokens: 2000 maxTokens: 2000,
// No responseFormat - let the model respond freely
}); });
result = parseSanityCheckResponse(response.parsed, response.content); result = parseSanityCheckResponse(response.parsed, response.content);
} }
} }
log.info(`[SanityCheck] Checked ${products.length} products in ${response.latencyMs}ms`, { log.info(`[SanityCheck] Checked ${products.length} products in ${response.latencyMs}ms`, {
issueCount: result.issues.length issueCount: result.issues.length,
}); });
return { return {
...result, ...result,
latencyMs: response.latencyMs, latencyMs: response.latencyMs,
usage: response.usage, usage: response.usage,
model: response.model model: response.model,
}; };
} }
module.exports = {
TASK_ID,
createSanityCheckTask,
MAX_PRODUCTS_PER_REQUEST
};
+42 -58
View File
@@ -1,79 +1,63 @@
// Purchase Order Status Codes // Purchase Order Status Codes
const PurchaseOrderStatus = { export const PurchaseOrderStatus = {
Canceled: 0, Canceled: 0,
Created: 1, Created: 1,
ElectronicallyReadySend: 10, ElectronicallyReadySend: 10,
Ordered: 11, Ordered: 11,
Preordered: 12, Preordered: 12,
ElectronicallySent: 13, ElectronicallySent: 13,
ReceivingStarted: 15, ReceivingStarted: 15,
Done: 50 Done: 50,
}; };
// Receiving Status Codes // Receiving Status Codes
const ReceivingStatus = { export const ReceivingStatus = {
Canceled: 0, Canceled: 0,
Created: 1, Created: 1,
PartialReceived: 30, PartialReceived: 30,
FullReceived: 40, FullReceived: 40,
Paid: 50 Paid: 50,
}; };
// Status Code Display Names export const PurchaseOrderStatusLabels = {
const PurchaseOrderStatusLabels = { [PurchaseOrderStatus.Canceled]: 'Canceled',
[PurchaseOrderStatus.Canceled]: 'Canceled', [PurchaseOrderStatus.Created]: 'Created',
[PurchaseOrderStatus.Created]: 'Created', [PurchaseOrderStatus.ElectronicallyReadySend]: 'Ready to Send',
[PurchaseOrderStatus.ElectronicallyReadySend]: 'Ready to Send', [PurchaseOrderStatus.Ordered]: 'Ordered',
[PurchaseOrderStatus.Ordered]: 'Ordered', [PurchaseOrderStatus.Preordered]: 'Preordered',
[PurchaseOrderStatus.Preordered]: 'Preordered', [PurchaseOrderStatus.ElectronicallySent]: 'Sent',
[PurchaseOrderStatus.ElectronicallySent]: 'Sent', [PurchaseOrderStatus.ReceivingStarted]: 'Receiving Started',
[PurchaseOrderStatus.ReceivingStarted]: 'Receiving Started', [PurchaseOrderStatus.Done]: 'Done',
[PurchaseOrderStatus.Done]: 'Done'
}; };
const ReceivingStatusLabels = { export const ReceivingStatusLabels = {
[ReceivingStatus.Canceled]: 'Canceled', [ReceivingStatus.Canceled]: 'Canceled',
[ReceivingStatus.Created]: 'Created', [ReceivingStatus.Created]: 'Created',
[ReceivingStatus.PartialReceived]: 'Partially Received', [ReceivingStatus.PartialReceived]: 'Partially Received',
[ReceivingStatus.FullReceived]: 'Fully Received', [ReceivingStatus.FullReceived]: 'Fully Received',
[ReceivingStatus.Paid]: 'Paid' [ReceivingStatus.Paid]: 'Paid',
}; };
// Helper functions export function getPurchaseOrderStatusLabel(status) {
function getPurchaseOrderStatusLabel(status) { return PurchaseOrderStatusLabels[status] || 'Unknown';
return PurchaseOrderStatusLabels[status] || 'Unknown';
} }
function getReceivingStatusLabel(status) { export function getReceivingStatusLabel(status) {
return ReceivingStatusLabels[status] || 'Unknown'; return ReceivingStatusLabels[status] || 'Unknown';
} }
// Status checks export function isReceivingComplete(status) {
function isReceivingComplete(status) { return status >= ReceivingStatus.PartialReceived;
return status >= ReceivingStatus.PartialReceived;
} }
function isPurchaseOrderComplete(status) { export function isPurchaseOrderComplete(status) {
return status === PurchaseOrderStatus.Done; return status === PurchaseOrderStatus.Done;
} }
function isPurchaseOrderCanceled(status) { export function isPurchaseOrderCanceled(status) {
return status === PurchaseOrderStatus.Canceled; return status === PurchaseOrderStatus.Canceled;
} }
function isReceivingCanceled(status) { export function isReceivingCanceled(status) {
return status === ReceivingStatus.Canceled; return status === ReceivingStatus.Canceled;
} }
module.exports = {
PurchaseOrderStatus,
ReceivingStatus,
PurchaseOrderStatusLabels,
ReceivingStatusLabels,
getPurchaseOrderStatusLabel,
getReceivingStatusLabel,
isReceivingComplete,
isPurchaseOrderComplete,
isPurchaseOrderCanceled,
isReceivingCanceled
};
+23 -39
View File
@@ -1,45 +1,29 @@
/** /**
* Parses a query parameter value based on its expected type. * Parses a query parameter value based on its expected type.
* Throws error for invalid formats. Adjust date handling as needed. * Throws on invalid formats.
*/ */
function parseValue(value, type) { export function parseValue(value, type) {
if (value === null || value === undefined || value === '') return null; if (value === null || value === undefined || value === '') return null;
console.log(`Parsing value: "${value}" as type: "${type}"`); switch (type) {
case 'number': {
switch (type) { const num = parseFloat(value);
case 'number': if (Number.isNaN(num)) throw new Error(`Invalid number format: "${value}"`);
const num = parseFloat(value); return num;
if (isNaN(num)) {
console.error(`Invalid number format: "${value}"`);
throw new Error(`Invalid number format: "${value}"`);
}
return num;
case 'integer': // Specific type for integer IDs etc.
const int = parseInt(value, 10);
if (isNaN(int)) {
console.error(`Invalid integer format: "${value}"`);
throw new Error(`Invalid integer format: "${value}"`);
}
console.log(`Successfully parsed integer: ${int}`);
return int;
case 'boolean':
if (String(value).toLowerCase() === 'true') return true;
if (String(value).toLowerCase() === 'false') return false;
console.error(`Invalid boolean format: "${value}"`);
throw new Error(`Invalid boolean format: "${value}"`);
case 'date':
// Basic ISO date format validation (YYYY-MM-DD)
if (!String(value).match(/^\d{4}-\d{2}-\d{2}$/)) {
console.warn(`Potentially invalid date format passed: "${value}"`);
// Optionally throw an error or return null depending on strictness
// throw new Error(`Invalid date format (YYYY-MM-DD expected): "${value}"`);
}
return String(value); // Send as string, let DB handle casting/comparison
case 'string':
default:
return String(value);
} }
case 'integer': {
const int = parseInt(value, 10);
if (Number.isNaN(int)) throw new Error(`Invalid integer format: "${value}"`);
return int;
}
case 'boolean':
if (String(value).toLowerCase() === 'true') return true;
if (String(value).toLowerCase() === 'false') return false;
throw new Error(`Invalid boolean format: "${value}"`);
case 'date':
return String(value);
case 'string':
default:
return String(value);
}
} }
module.exports = { parseValue };
+9 -23
View File
@@ -1,45 +1,37 @@
const fs = require('fs'); import fs from 'node:fs';
const { parse } = require('csv-parse'); import { parse } from 'csv-parse';
const { v4: uuidv4 } = require('uuid'); import { v4 as uuidv4 } from 'uuid';
async function importProductsFromCSV(filePath, pool) { export async function importProductsFromCSV(filePath, pool) {
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const products = []; const products = [];
fs.createReadStream(filePath) fs.createReadStream(filePath)
.pipe(parse({ .pipe(parse({ columns: true, skip_empty_lines: true }))
columns: true, .on('data', (row) => {
skip_empty_lines: true
}))
.on('data', async (row) => {
products.push({ products.push({
id: uuidv4(), id: uuidv4(),
sku: row.sku, sku: row.sku,
name: row.name, name: row.name,
description: row.description || null, description: row.description || null,
category: row.category || null category: row.category || null,
}); });
}) })
.on('end', async () => { .on('end', async () => {
try { try {
const connection = await pool.getConnection(); const connection = await pool.getConnection();
try { try {
await connection.beginTransaction(); await connection.beginTransaction();
for (const product of products) { for (const product of products) {
await connection.query( await connection.query(
'INSERT INTO products (id, sku, name, description, category) VALUES (?, ?, ?, ?, ?)', 'INSERT INTO products (id, sku, name, description, category) VALUES (?, ?, ?, ?, ?)',
[product.id, product.sku, product.name, product.description, product.category] [product.id, product.sku, product.name, product.description, product.category]
); );
// Initialize inventory level for the product
await connection.query( await connection.query(
'INSERT INTO inventory_levels (id, product_id, quantity) VALUES (?, ?, 0)', 'INSERT INTO inventory_levels (id, product_id, quantity) VALUES (?, ?, 0)',
[uuidv4(), product.id] [uuidv4(), product.id]
); );
} }
await connection.commit(); await connection.commit();
resolve({ imported: products.length }); resolve({ imported: products.length });
} catch (error) { } catch (error) {
@@ -52,12 +44,6 @@ async function importProductsFromCSV(filePath, pool) {
reject(error); reject(error);
} }
}) })
.on('error', (error) => { .on('error', reject);
reject(error);
});
}); });
} }
module.exports = {
importProductsFromCSV
};
+10 -8
View File
@@ -1,21 +1,23 @@
const { Pool } = require('pg'); import pg from 'pg';
const { Pool } = pg;
let pool; let pool;
function initPool(config) { export function initPool(config) {
pool = new Pool(config); pool = new Pool(config);
return pool; return pool;
} }
async function getConnection() { export async function getConnection() {
if (!pool) { if (!pool) {
throw new Error('Database pool not initialized'); throw new Error('Database pool not initialized');
} }
return pool.connect(); return pool.connect();
} }
module.exports = { export function getPool() {
initPool, return pool;
getConnection, }
getPool: () => pool
}; export default { initPool, getConnection, getPool };
+48 -150
View File
@@ -1,158 +1,94 @@
const { Client } = require('ssh2'); import { Client } from 'ssh2';
const mysql = require('mysql2/promise'); import mysql from 'mysql2/promise';
const fs = require('fs'); import fs from 'node:fs';
// Connection pooling and cache configuration
const connectionCache = { const connectionCache = {
ssh: null, ssh: null,
dbConnection: null, dbConnection: null,
lastUsed: 0, lastUsed: 0,
isConnecting: false, isConnecting: false,
connectionPromise: null, connectionPromise: null,
// Cache expiration time in milliseconds (5 minutes)
expirationTime: 5 * 60 * 1000, expirationTime: 5 * 60 * 1000,
// Cache for query results (key: query string, value: {data, timestamp})
queryCache: new Map(), queryCache: new Map(),
// Cache duration for different query types in milliseconds
cacheDuration: { cacheDuration: {
'field-options': 30 * 60 * 1000, // 30 minutes for field options 'field-options': 30 * 60 * 1000,
'product-lines': 10 * 60 * 1000, // 10 minutes for product lines 'product-lines': 10 * 60 * 1000,
'sublines': 10 * 60 * 1000, // 10 minutes for sublines 'sublines': 10 * 60 * 1000,
'taxonomy': 30 * 60 * 1000, // 30 minutes for taxonomy data 'taxonomy': 30 * 60 * 1000,
'default': 60 * 1000 // 1 minute default 'default': 60 * 1000,
} },
}; };
/** export async function getDbConnection() {
* Get a database connection with connection pooling
* @returns {Promise<{ssh: object, connection: object}>} The SSH and database connection
*/
async function getDbConnection() {
const now = Date.now(); const now = Date.now();
const needsRefresh = !connectionCache.ssh
// Check if we need to refresh the connection due to inactivity || !connectionCache.dbConnection
const needsRefresh = !connectionCache.ssh || || (now - connectionCache.lastUsed > connectionCache.expirationTime);
!connectionCache.dbConnection ||
(now - connectionCache.lastUsed > connectionCache.expirationTime);
// If connection is still valid, update last used time and return existing connection
if (!needsRefresh) { if (!needsRefresh) {
connectionCache.lastUsed = now; connectionCache.lastUsed = now;
return { return { ssh: connectionCache.ssh, connection: connectionCache.dbConnection };
ssh: connectionCache.ssh,
connection: connectionCache.dbConnection
};
} }
// If another request is already establishing a connection, wait for that promise
if (connectionCache.isConnecting && connectionCache.connectionPromise) { if (connectionCache.isConnecting && connectionCache.connectionPromise) {
try { try {
await connectionCache.connectionPromise; await connectionCache.connectionPromise;
return { return { ssh: connectionCache.ssh, connection: connectionCache.dbConnection };
ssh: connectionCache.ssh,
connection: connectionCache.dbConnection
};
} catch (error) { } catch (error) {
// If that connection attempt failed, we'll try again below
console.error('Error waiting for existing connection:', error); console.error('Error waiting for existing connection:', error);
} }
} }
// Close existing connections if they exist
if (connectionCache.dbConnection) { if (connectionCache.dbConnection) {
try { try { await connectionCache.dbConnection.end(); }
await connectionCache.dbConnection.end(); catch (error) { console.error('Error closing existing database connection:', error); }
} catch (error) {
console.error('Error closing existing database connection:', error);
}
} }
if (connectionCache.ssh) { if (connectionCache.ssh) {
try { try { connectionCache.ssh.end(); }
connectionCache.ssh.end(); catch (error) { console.error('Error closing existing SSH connection:', error); }
} catch (error) {
console.error('Error closing existing SSH connection:', error);
}
} }
// Mark that we're establishing a new connection
connectionCache.isConnecting = true; connectionCache.isConnecting = true;
connectionCache.connectionPromise = setupSshTunnel().then((tunnel) => {
// Create a new promise for this connection attempt
connectionCache.connectionPromise = setupSshTunnel().then(tunnel => {
const { ssh, stream, dbConfig } = tunnel; const { ssh, stream, dbConfig } = tunnel;
return mysql.createConnection({ ...dbConfig, stream }).then((connection) => {
return mysql.createConnection({
...dbConfig,
stream
}).then(connection => {
// Store the new connections
connectionCache.ssh = ssh; connectionCache.ssh = ssh;
connectionCache.dbConnection = connection; connectionCache.dbConnection = connection;
connectionCache.lastUsed = Date.now(); connectionCache.lastUsed = Date.now();
connectionCache.isConnecting = false; connectionCache.isConnecting = false;
return { ssh, connection };
return {
ssh,
connection
};
}); });
}).catch(error => { }).catch((error) => {
connectionCache.isConnecting = false; connectionCache.isConnecting = false;
throw error; throw error;
}); });
// Wait for the connection to be established
return connectionCache.connectionPromise; return connectionCache.connectionPromise;
} }
/** export async function getCachedQuery(cacheKey, queryType, queryFn) {
* Get cached query results or execute query if not cached
* @param {string} cacheKey - Unique key to identify the query
* @param {string} queryType - Type of query (field-options, product-lines, etc.)
* @param {Function} queryFn - Function to execute if cache miss
* @returns {Promise<any>} The query result
*/
async function getCachedQuery(cacheKey, queryType, queryFn) {
// Get cache duration based on query type
const cacheDuration = connectionCache.cacheDuration[queryType] || connectionCache.cacheDuration.default; const cacheDuration = connectionCache.cacheDuration[queryType] || connectionCache.cacheDuration.default;
// Check if we have a valid cached result
const cachedResult = connectionCache.queryCache.get(cacheKey); const cachedResult = connectionCache.queryCache.get(cacheKey);
const now = Date.now(); const now = Date.now();
if (cachedResult && (now - cachedResult.timestamp < cacheDuration)) { if (cachedResult && (now - cachedResult.timestamp < cacheDuration)) {
console.log(`Cache hit for ${queryType} query: ${cacheKey}`);
return cachedResult.data; return cachedResult.data;
} }
// No valid cache found, execute the query
console.log(`Cache miss for ${queryType} query: ${cacheKey}`);
const result = await queryFn(); const result = await queryFn();
connectionCache.queryCache.set(cacheKey, { data: result, timestamp: now });
// Cache the result
connectionCache.queryCache.set(cacheKey, {
data: result,
timestamp: now
});
return result; return result;
} }
/**
* Setup SSH tunnel to production database
* @private - Should only be used by getDbConnection
* @returns {Promise<{ssh: object, stream: object, dbConfig: object}>}
*/
async function setupSshTunnel() { async function setupSshTunnel() {
const sshConfig = { const sshConfig = {
host: process.env.PROD_SSH_HOST, host: process.env.PROD_SSH_HOST,
port: process.env.PROD_SSH_PORT || 22, port: Number(process.env.PROD_SSH_PORT) || 22,
username: process.env.PROD_SSH_USER, username: process.env.PROD_SSH_USER,
privateKey: process.env.PROD_SSH_KEY_PATH privateKey: process.env.PROD_SSH_KEY_PATH
? fs.readFileSync(process.env.PROD_SSH_KEY_PATH) ? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
: undefined, : undefined,
compress: true compress: true,
}; };
const dbConfig = { const dbConfig = {
@@ -160,80 +96,42 @@ async function setupSshTunnel() {
user: process.env.PROD_DB_USER, user: process.env.PROD_DB_USER,
password: process.env.PROD_DB_PASSWORD, password: process.env.PROD_DB_PASSWORD,
database: process.env.PROD_DB_NAME, database: process.env.PROD_DB_NAME,
port: process.env.PROD_DB_PORT || 3306, port: Number(process.env.PROD_DB_PORT) || 3306,
timezone: 'Z' timezone: 'Z',
}; };
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
const ssh = new Client(); const ssh = new Client();
ssh.on('error', (err) => { ssh.on('error', (err) => {
console.error('SSH connection error:', err); console.error('SSH connection error:', err);
reject(err); reject(err);
}); });
ssh.on('ready', () => { ssh.on('ready', () => {
ssh.forwardOut( ssh.forwardOut('127.0.0.1', 0, dbConfig.host, dbConfig.port, (err, stream) => {
'127.0.0.1', if (err) reject(err);
0, else resolve({ ssh, stream, dbConfig });
dbConfig.host, });
dbConfig.port,
(err, stream) => {
if (err) reject(err);
resolve({ ssh, stream, dbConfig });
}
);
}).connect(sshConfig); }).connect(sshConfig);
}); });
} }
/** export function clearQueryCache(cacheKey) {
* Clear cached query results if (cacheKey) connectionCache.queryCache.delete(cacheKey);
* @param {string} [cacheKey] - Specific cache key to clear (clears all if not provided) else connectionCache.queryCache.clear();
*/
function clearQueryCache(cacheKey) {
if (cacheKey) {
connectionCache.queryCache.delete(cacheKey);
console.log(`Cleared cache for key: ${cacheKey}`);
} else {
connectionCache.queryCache.clear();
console.log('Cleared all query cache');
}
} }
/** export async function closeAllConnections() {
* Force close all active connections
* Useful for server shutdown or manual connection reset
*/
async function closeAllConnections() {
if (connectionCache.dbConnection) { if (connectionCache.dbConnection) {
try { try { await connectionCache.dbConnection.end(); }
await connectionCache.dbConnection.end(); catch (error) { console.error('Error closing database connection:', error); }
console.log('Closed database connection');
} catch (error) {
console.error('Error closing database connection:', error);
}
connectionCache.dbConnection = null; connectionCache.dbConnection = null;
} }
if (connectionCache.ssh) { if (connectionCache.ssh) {
try { try { connectionCache.ssh.end(); }
connectionCache.ssh.end(); catch (error) { console.error('Error closing SSH connection:', error); }
console.log('Closed SSH connection');
} catch (error) {
console.error('Error closing SSH connection:', error);
}
connectionCache.ssh = null; connectionCache.ssh = null;
} }
connectionCache.lastUsed = 0; connectionCache.lastUsed = 0;
connectionCache.isConnecting = false; connectionCache.isConnecting = false;
connectionCache.connectionPromise = null; connectionCache.connectionPromise = null;
} }
module.exports = {
getDbConnection,
getCachedQuery,
clearQueryCache,
closeAllConnections
};