Product editor tweaks

.claude/CLAUDE.md

@@ -0,0 +1,172 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a full-stack inventory management system with a React + TypeScript frontend and Node.js/Express backend using PostgreSQL. The system includes product management, analytics, forecasting, purchase orders, and a comprehensive dashboard for business metrics.
+
+**Monorepo Structure:**
+- `inventory/` - Vite-based React frontend with TypeScript
+- `inventory-server/` - Express backend API server
+- Root `package.json` contains shared dependencies
+
+## Development Commands
+
+### Frontend (inventory/)
+```bash
+cd inventory
+npm run dev        # Start dev server on port 5175
+npm run build      # Build for production (outputs to build/ then copies to ../inventory-server/frontend/build)
+npm run lint       # Run ESLint
+npm run preview    # Preview production build
+```
+
+### Backend (inventory-server/)
+```bash
+cd inventory-server
+npm run dev        # Start with nodemon (auto-reload)
+npm start          # Start server (production)
+npm run prod       # Start with PM2 for production
+npm run prod:stop  # Stop PM2 instance
+npm run prod:restart  # Restart PM2 instance
+npm run prod:logs  # View PM2 logs
+npm run setup      # Create required directories (logs, uploads)
+```
+
+## Architecture
+
+### Frontend Architecture
+
+**Router Structure:** React Router with lazy loading for code splitting:
+- Main chunks: Core inventory, Dashboard, Product Import, Chat Archive
+- Authentication flow uses `RequireAuth` and `Protected` components with permission-based access
+- All routes except `/login` and `/small` require authentication
+
+**Key Directories:**
+- `src/pages/` - Top-level page components (Overview, Products, Analytics, Dashboard, etc.)
+- `src/components/` - Organized by feature (dashboard/, products/, analytics/, etc.)
+- `src/components/ui/` - shadcn/ui components
+- `src/types/` - TypeScript type definitions
+- `src/contexts/` - React contexts (AuthContext, DashboardScrollContext)
+- `src/hooks/` - Custom React hooks (use-toast, useDebounce, use-mobile)
+- `src/utils/` - Utility functions (emojiUtils, productUtils, naturalLanguagePeriod)
+- `src/services/` - API service layer
+- `src/config/` - Configuration files
+
+**State Management:**
+- React Context for auth and global state
+- @tanstack/react-query for server state management
+- zustand for client state management
+- Local storage for auth tokens, session storage for login state
+
+**Key Dependencies:**
+- UI: Radix UI primitives, shadcn/ui, Tailwind CSS, Framer Motion
+- Data: @tanstack/react-table, react-data-grid, @tanstack/react-virtual
+- Forms: react-hook-form, zod
+- Charts: recharts, chart.js, react-chartjs-2
+- File handling: xlsx for Excel export, react-dropzone for uploads
+- Other: axios for HTTP, date-fns/luxon for dates
+
+**Path Alias:** `@/` maps to `./src/`
+
+### Backend Architecture
+
+**Entry Point:** `inventory-server/src/server.js`
+
+**Key Directories:**
+- `src/routes/` - Express route handlers (products, dashboard, analytics, import, etc.)
+- `src/middleware/` - Express middleware (CORS, auth, etc.)
+- `src/utils/` - Utility functions (database connection, API helpers)
+- `src/types/` - Type definitions (e.g., status-codes)
+
+**Database:**
+- PostgreSQL with connection pooling (pg library)
+- Pool initialized in `utils/db.js` via `initPool()`
+- Pool attached to `app.locals.pool` for route access
+- Environment variables loaded from `/var/www/html/inventory/.env` (production path)
+
+**API Routes:** All prefixed with `/api/`
+- `/api/products` - Product CRUD operations
+- `/api/dashboard` - Dashboard metrics and data
+- `/api/analytics` - Analytics and reporting
+- `/api/orders` - Order management
+- `/api/purchase-orders` - Purchase order management
+- `/api/csv` - CSV import/export (data management)
+- `/api/import` - Product import workflows
+- `/api/config` - Configuration management
+- `/api/metrics` - System metrics
+- `/api/ai-validation` - AI-powered validation
+- `/api/ai-prompts` - AI prompt management
+- `/api/templates` - Template management
+- `/api/reusable-images` - Image management
+- `/api/categoriesAggregate`, `/api/vendorsAggregate`, `/api/brandsAggregate` - Aggregate data endpoints
+
+**Authentication:**
+- External auth service at `/auth-inv` endpoint
+- Token-based authentication (Bearer tokens)
+- Frontend stores tokens in localStorage
+- Protected routes verify tokens via auth service `/me` endpoint
+
+**File Uploads:**
+- Multer middleware for file handling
+- Uploads directory: `inventory-server/uploads/`
+
+### Development Proxy Setup
+
+The Vite dev server (port 5175) proxies API requests to `https://inventory.kent.pw`:
+- `/api/*` → production API
+- `/auth-inv/*` → authentication service
+- `/chat-api/*` → chat service
+- `/uploads/*` → uploaded files
+- Various third-party services (Aircall, Klaviyo, Meta, Gorgias, Typeform, ACOT, Clarity)
+
+### Build Process
+
+When building the frontend:
+1. TypeScript compilation (`tsc -b`)
+2. Vite build (outputs to `inventory/build/`)
+3. Custom Vite plugin copies build to `inventory-server/frontend/build/`
+4. Manual chunks for vendor splitting (react-vendor, ui-vendor, query-vendor)
+
+## Testing
+
+Run tests for individual components or features:
+```bash
+# No test suite currently configured
+# Tests would typically use Jest or Vitest with React Testing Library
+```
+
+## Common Development Workflows
+
+### Adding a New Page
+1. Create page component in `inventory/src/pages/YourPage.tsx`
+2. Add lazy import in `inventory/src/App.tsx`
+3. Add route with `<Protected>` wrapper and permission check
+4. Add corresponding backend route in `inventory-server/src/routes/`
+5. Update permission system if needed
+
+### Adding a New API Endpoint
+1. Create or update route file in `inventory-server/src/routes/`
+2. Use `executeQuery()` helper for database queries
+3. Register router in `inventory-server/src/server.js`
+4. Frontend can access at `/api/{route-name}`
+
+### Working with Database
+- Use parameterized queries: `executeQuery(sql, [param1, param2])`
+- Pool is accessed via `db.getPool()` or `app.locals.pool`
+- Connection helper: `db.getConnection()` returns a client for transactions
+
+### Permissions System
+- User permissions stored in `user.permissions` array (permission codes)
+- Check permissions in `<Protected page="permission_code">` component
+- Admin users (`is_admin: true`) have access to all pages
+
+## Important Notes
+
+- Environment variables must be configured in `/var/www/html/inventory/.env` for production
+- The frontend expects the backend at `/api` (proxied in dev, served together in production)
+- PM2 is used for production process management
+- Database uses PostgreSQL with SSL support (configurable via `DB_SSL` env var)
+- File uploads stored in `inventory-server/uploads/` directory
+- Build artifacts in `inventory/build/` are copied to `inventory-server/frontend/build/`

.gitignore

@@ -73,4 +73,10 @@ inventory-server/scripts/.fuse_hidden00000fa20000000a
 */chat/db-convert/mongo_converter_env/*
 
 # Ignore compiled Vite config to avoid duplication
-vite.config.js
+vite.config.js
+inventory-server/inventory_backup.sql
+chat-files.tar.gz
+chat-migration*/
+**/chat-migration*/
+chat-migration*/**
+**/chat-migration*/**

CLAUDE.md

@@ -0,0 +1,2 @@
+* Avoid using glob tool for search as it may not work properly on this codebase. Search using bash instead.
+* If you use the task tool to have an agent investigate something, make sure to let it know to avoid using glob

PRODUCT_IMPORT_ENHANCEMENTS.md

@@ -0,0 +1,375 @@
+# Product Import Module - Enhancement & Issues Outline
+
+This document outlines the investigation and implementation requirements for each requested enhancement to the product import module.
+
+---
+
+## 1. UPC Import - Strip Quotes and Spaces ✅ IMPLEMENTED
+
+**Issue:** When importing UPCs, strip `'`, `"` characters and any spaces, leaving only numbers.
+
+**Implementation (Completed):**
+- Modified `normalizeUpcValue()` in [Import.tsx:661-667](inventory/src/pages/Import.tsx#L661-L667)
+- Strips single quotes, double quotes, smart quotes (`'"`), and whitespace before processing
+- Then handles scientific notation and extracts only digits
+
+**Files Modified:**
+- `inventory/src/pages/Import.tsx` - `normalizeUpcValue()` function
+
+---
+
+## 2. AI Context Columns in Validation Payloads ✅ IMPLEMENTED
+
+**Issue:** The match columns step has a setting to use a field only for AI context (`isAiSupplemental`). Update AI description validation to include any columns selected with this option in the payload. Also include in sanity check payload. Not needed for names.
+
+**Current Implementation:**
+- AI Supplemental toggle: [MatchColumnsStep.tsx:102-118](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L102-L118)
+- AI supplemental data stored in `__aiSupplemental` field on each row
+- Description payload builder: [inlineAiPayload.ts:183-195](inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts#L183-L195)
+
+**Implementation:**
+1. **Update `buildDescriptionValidationPayload()` in `inlineAiPayload.ts`** to include AI supplemental data:
+   ```typescript
+   export const buildDescriptionValidationPayload = (
+     row: Data<string>,
+     fieldOptions: FieldOptionsMap,
+     productLinesCache: Map<string, SelectOption[]>,
+     sublinesCache: Map<string, SelectOption[]>
+   ) => {
+     const payload: Record<string, unknown> = {
+       name: row.name,
+       description: row.description,
+       company_name: getFieldOptionLabel(row.company, fieldOptions, 'company'),
+       company_id: row.company,
+       categories: getFieldOptionLabel(row.category, fieldOptions, 'category'),
+     };
+
+     // Add AI supplemental context if present
+     if (row.__aiSupplemental && typeof row.__aiSupplemental === 'object') {
+       payload.additional_context = row.__aiSupplemental;
+     }
+
+     return payload;
+   };
+   ```
+
+2. **Update sanity check payload** - Locate sanity check submission logic and include `__aiSupplemental` data
+
+3. **Verify `__aiSupplemental` is properly populated** from MatchColumnsStep when columns are marked as AI context only
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts`
+- Backend sanity check endpoint (if separate from description validation)
+- Verify data flow in `MatchColumnsStep.tsx` → `ValidationStep`
+
+---
+
+## 3. Fresh Taxonomy Data Per Session ✅ IMPLEMENTED
+
+**Issue:** Ensure taxonomy data is brought in fresh with each session - cache should be invalidated if we exit the import flow and start again.
+
+**Current Implementation:**
+- Field options cached 5 minutes: [ValidationStep/index.tsx:128-133](inventory/src/components/product-import/steps/ValidationStep/index.tsx#L128-L133)
+- Product lines cache: `productLinesCache` in Zustand store
+- Sublines cache: `sublinesCache` in Zustand store
+- Caches set to 10-minute stale time
+
+**Implementation:**
+1. **Add cache invalidation on import flow mount/unmount** in `UploadFlow.tsx`:
+   ```typescript
+   useEffect(() => {
+     // On mount - invalidate import-related query cache
+     queryClient.invalidateQueries({ queryKey: ['import-field-options'] });
+
+     return () => {
+       // On unmount - clear caches
+       queryClient.removeQueries({ queryKey: ['import-field-options'] });
+       queryClient.removeQueries({ queryKey: ['product-lines'] });
+       queryClient.removeQueries({ queryKey: ['sublines'] });
+     };
+   }, []);
+   ```
+
+2. **Clear Zustand store caches** when exiting import flow:
+   - Add action to `validationStore.ts` to clear `productLinesCache` and `sublinesCache`
+   - Call this action on unmount of `UploadFlow` or when navigating away
+
+3. **Consider adding a `sessionId`** that changes on each import flow start, used as part of cache keys
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/UploadFlow.tsx` - Add cleanup effect
+- `inventory/src/components/product-import/steps/ValidationStep/store/validationStore.ts` - Add cache clear action
+- Potentially `inventory/src/components/product-import/steps/ValidationStep/index.tsx` - Query key updates
+
+---
+
+## 4. Save Template from Confirmation Page ✅ IMPLEMENTED
+
+**Issue:** Add option to save rows of submitted data as a new template on the confirmation page after completing the import flow. Verify this works with new validation step changes.
+
+**Current Implementation:**
+- **Import Results section already exists** inline in [Import.tsx:968-1150](inventory/src/pages/Import.tsx#L968-L1150)
+  - Shows created products (lines 1021-1097) with image, name, UPC, item number
+  - Shows errored products (lines 1100-1138) with error details
+  - "Fix products with errors" button resumes validation flow for failed items
+- Template saving logic in ValidationStep: [useTemplateManagement.ts:204-266](inventory/src/components/product-import/steps/ValidationStep/hooks/useTemplateManagement.ts#L204-L266)
+- Saves via `POST /api/templates`
+- `importOutcome.submittedProducts` contains the full product data for each row
+
+**Implementation:**
+1. **Add "Save as Template" button** to each created product row in the results section (around line 1087-1092 in Import.tsx):
+   ```typescript
+   // Add button after the item number display
+   <Button
+     variant="ghost"
+     size="sm"
+     onClick={() => handleSaveAsTemplate(index)}
+   >
+     <BookmarkPlus className="h-4 w-4" />
+   </Button>
+   ```
+
+2. **Add state and dialog** for template saving in Import.tsx:
+   ```typescript
+   const [templateSaveDialogOpen, setTemplateSaveDialogOpen] = useState(false);
+   const [selectedProductForTemplate, setSelectedProductForTemplate] = useState<NormalizedProduct | null>(null);
+   ```
+
+3. **Extract/reuse template save logic** from `useTemplateManagement.ts`:
+   - The `saveNewTemplate()` function (lines 204-266) can be extracted into a shared utility
+   - Or create a `SaveTemplateDialog` component that can be used in both places
+   - Key fields needed: `company` (for template name), `product_type`, and all product field values
+
+4. **Data mapping consideration:**
+   - `importOutcome.submittedProducts` uses `NormalizedProduct` type
+   - Templates expect raw field values - may need to map back from normalized format
+   - Exclude metadata fields: `['id', '__index', '__meta', '__template', '__original', '__corrected', '__changes', '__aiSupplemental']`
+
+**Files to Modify:**
+- `inventory/src/pages/Import.tsx` - Add save template button, state, and dialog
+- Consider creating `inventory/src/components/product-import/SaveTemplateDialog.tsx` for reusability
+- Potentially extract core save logic from `useTemplateManagement.ts` into shared utility
+
+---
+
+## 5. Sheet Preview on Select Sheet Step ✅ IMPLEMENTED
+
+**Issue:** On the select sheet step, show a preview of the first 10 lines or so of each sheet underneath the options.
+
+**Implementation (Completed):**
+- Added `workbook` prop to `SelectSheetStep` component
+- Added `sheetPreviews` memoized computation using `XLSXLib.utils.sheet_to_json()`
+- Shows first 10 rows, 8 columns max per sheet
+- Added `truncateCell()` helper to limit cell content to 30 characters with ellipsis
+- Each sheet option is now a clickable card with:
+  - Radio button and sheet name
+  - Row count indicator
+  - Scrollable preview table with horizontal scroll
+  - Selected state highlighted with primary border
+- Updated `UploadFlow.tsx` to pass workbook prop
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectSheetStep/SelectSheetStep.tsx`
+- `inventory/src/components/product-import/steps/UploadFlow.tsx`
+
+---
+
+## 6. Empty Row Removal ✅ IMPLEMENTED
+
+**Issue:** When importing a sheet, automatically remove completely empty rows.
+
+**Current Implementation:**
+- Empty columns are filtered: [MatchColumnsStep.tsx:616-634](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L616-L634)
+- A "Remove empty/duplicates" button exists that removes empty rows, single-value rows, AND duplicates
+- The automatic removal should ONLY remove completely empty rows, not duplicates or single-value rows
+
+**Implementation (Completed):**
+- Added `isRowCompletelyEmpty()` helper function to [SelectHeaderStep.tsx](inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx)
+- Added `useMemo` to filter empty rows on initial data load
+- Uses `Object.values(row)` to check all cell values (matches existing button logic)
+- Only removes rows where ALL values are undefined, null, or whitespace-only strings
+- Manual "Remove Empty/Duplicates" button still available for additional cleanup (duplicates, single-value rows)
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx`
+
+---
+
+## 7. Unit Conversion for Weight/Dimensions ✅ IMPLEMENTED
+
+**Issue:** Add unit conversion feature for weight and dimensions columns - similar to calculator button on cost/msrp, add button that opens popover with options to convert grams → oz, lbs → oz for the whole column at once.
+
+**Current Implementation:**
+- Calculator button on price columns: [ValidationTable.tsx:1491-1627](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1491-L1627)
+- `PriceColumnHeader` component shows calculator icon on hover
+- Weight field defined in config with validation
+
+**Implementation:**
+1. **Create `UnitConversionColumnHeader` component** (similar to `PriceColumnHeader`):
+   ```typescript
+   const UnitConversionColumnHeader = ({ field, table }) => {
+     const [showPopover, setShowPopover] = useState(false);
+
+     const conversions = {
+       weight: [
+         { label: 'Grams → Ounces', factor: 0.035274 },
+         { label: 'Pounds → Ounces', factor: 16 },
+         { label: 'Kilograms → Ounces', factor: 35.274 },
+       ],
+       dimensions: [
+         { label: 'Centimeters → Inches', factor: 0.393701 },
+         { label: 'Millimeters → Inches', factor: 0.0393701 },
+       ]
+     };
+
+     const applyConversion = (factor: number) => {
+       // Batch update all cells in column
+       table.rows.forEach((row, index) => {
+         const currentValue = parseFloat(row[field.key]);
+         if (!isNaN(currentValue)) {
+           updateCell(index, field.key, (currentValue * factor).toFixed(2));
+         }
+       });
+     };
+
+     return (
+       <Popover open={showPopover} onOpenChange={setShowPopover}>
+         <PopoverTrigger>
+           <Scale className="h-4 w-4" /> {/* or similar icon */}
+         </PopoverTrigger>
+         <PopoverContent>
+           {conversions[fieldType].map(conv => (
+             <Button key={conv.label} onClick={() => applyConversion(conv.factor)}>
+               {conv.label}
+             </Button>
+           ))}
+         </PopoverContent>
+       </Popover>
+     );
+   };
+   ```
+
+2. **Identify weight/dimension fields** in config:
+   - `weight_oz`, `length_in`, `width_in`, `height_in` (check actual field keys)
+
+3. **Add to column header render logic** in ValidationTable
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx`
+- Potentially create new component file for `UnitConversionColumnHeader`
+- Update column header rendering to use new component for weight/dimension fields
+
+---
+
+## 8. Expanded MSRP Auto-Fill from Cost ✅ IMPLEMENTED
+
+**Issue:** Expand auto-fill functionality for MSRP from cost - open small popover with options for 2x, 2.1x, 2.2x, 2.3x, 2.4x, 2.5x multipliers, plus checkbox to round up to nearest 9.
+
+**Current Implementation:**
+- Calculator on MSRP column: [ValidationTable.tsx:1540-1584](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1540-L1584)
+- Currently only does `Cost × 2` then subtracts 0.01 if whole number
+
+**Implementation:**
+1. **Replace simple click with popover** in `PriceColumnHeader`:
+   ```typescript
+   const [selectedMultiplier, setSelectedMultiplier] = useState(2.0);
+   const [roundToNine, setRoundToNine] = useState(false);
+   const multipliers = [2.0, 2.1, 2.2, 2.3, 2.4, 2.5];
+
+   const roundUpToNine = (value: number): number => {
+     // 1.41 → 1.49, 2.78 → 2.79, 12.32 → 12.39
+     const wholePart = Math.floor(value);
+     const decimal = value - wholePart;
+     if (decimal <= 0.09) return wholePart + 0.09;
+     if (decimal <= 0.19) return wholePart + 0.19;
+     // ... continue pattern, or:
+     const lastDigit = Math.floor(decimal * 10);
+     return wholePart + (lastDigit / 10) + 0.09;
+   };
+
+   const calculateMsrp = (cost: number): number => {
+     let result = cost * selectedMultiplier;
+     if (roundToNine) {
+       result = roundUpToNine(result);
+     }
+     return result;
+   };
+   ```
+
+2. **Create popover UI**:
+   ```tsx
+   <Popover>
+     <PopoverTrigger><Calculator className="h-4 w-4" /></PopoverTrigger>
+     <PopoverContent className="w-48">
+       <div className="space-y-2">
+         <Label>Multiplier</Label>
+         <div className="grid grid-cols-3 gap-1">
+           {multipliers.map(m => (
+             <Button
+               key={m}
+               variant={selectedMultiplier === m ? 'default' : 'outline'}
+               size="sm"
+               onClick={() => setSelectedMultiplier(m)}
+             >
+               {m}x
+             </Button>
+           ))}
+         </div>
+         <div className="flex items-center gap-2">
+           <Checkbox checked={roundToNine} onCheckedChange={setRoundToNine} />
+           <Label>Round to .X9</Label>
+         </div>
+         <Button onClick={applyCalculation} className="w-full">
+           Apply
+         </Button>
+       </div>
+     </PopoverContent>
+   </Popover>
+   ```
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx` - `PriceColumnHeader` component
+
+---
+
+## 9. Debug Mode - Skip API Submission ✅ IMPLEMENTED
+
+**Issue:** Add a third switch in the footer of image upload step (visible only to users with `admin:debug` permission) that will not submit data to any API, only complete the process and show results page as if it had worked.
+
+**Implementation (Completed):**
+- Added `skipApiSubmission` state to `ImageUploadStep.tsx`
+- Added amber-colored "Skip API (Debug)" switch (visible only with `admin:debug` permission)
+- When skip is active, "Use Test API" and "Use Test Database" switches are hidden
+- Added `skipApiSubmission?: boolean` to `SubmitOptions` type in `types.ts`
+- In `Import.tsx`, when `skipApiSubmission` is true:
+  - Skips the actual API call entirely
+  - Generates mock success response with mock PIDs
+  - Shows `[DEBUG]` prefix in toast and result message
+  - Displays results page as if submission succeeded
+
+**Files Modified:**
+- `inventory/src/components/product-import/types.ts` - Added `skipApiSubmission` to `SubmitOptions`
+- `inventory/src/components/product-import/steps/ImageUploadStep/ImageUploadStep.tsx` - Added switch UI
+- `inventory/src/pages/Import.tsx` - Added skip logic in `handleData()`
+
+---
+
+## Summary
+
+| # | Enhancement | Complexity | Status |
+|---|-------------|------------|--------|
+| 1 | Strip UPC quotes/spaces | Low | ✅ Implemented |
+| 2 | AI context in validation | Medium | ✅ Implemented |
+| 3 | Fresh taxonomy per session | Medium | ✅ Implemented |
+| 4 | Save template from confirmation | Medium-High | ✅ Implemented |
+| 5 | Sheet preview | Low-Medium | ✅ Implemented |
+| 6 | Remove empty rows | Low | ✅ Implemented |
+| 7 | Unit conversion | Medium | ✅ Implemented |
+| 8 | MSRP multiplier options | Medium | ✅ Implemented |
+| 9 | Debug skip API | Low-Medium | ✅ Implemented |
+
+**Implemented:** 9 of 9 items - All enhancements complete!
+
+---
+
+*Document generated: 2026-01-25*

docs/PERMISSIONS.md

@@ -7,12 +7,13 @@ This document outlines the permission system implemented in the Inventory Manage
 Permissions follow this naming convention:
 
 - Page access: `access:{page_name}`
-- Actions: `{action}:{resource}`
+- Settings sections: `settings:{section_name}`
+- Admin features: `admin:{feature}`
 
 Examples:
 - `access:products` - Can access the Products page
-- `create:products` - Can create new products
-- `edit:users` - Can edit user accounts
+- `settings:user_management` - Can access User Management settings
+- `admin:debug` - Can see debug information
 
 ## Permission Components
 
@@ -22,10 +23,10 @@ The core component that conditionally renders content based on permissions.
 
 ```tsx
 <PermissionGuard 
-  permission="create:products"
+  permission="settings:user_management"
   fallback={<p>No permission</p>}
 >
-  <button>Create Product</button>
+  <button>Manage Users</button>
 </PermissionGuard>
 ```
 
@@ -81,7 +82,7 @@ Specific component for settings with built-in permission checks.
 <SettingsSection
   title="System Settings"
   description="Configure global settings"
-  permission="edit:system_settings"
+  permission="settings:global"
 >
   {/* Settings content */}
 </SettingsSection>
@@ -95,8 +96,8 @@ Core hook for checking any permission.
 
 ```tsx
 const { hasPermission, hasPageAccess, isAdmin } = usePermissions();
-if (hasPermission('delete:products')) {
-  // Can delete products
+if (hasPermission('settings:user_management')) {
+  // Can access user management
 }
 ```
 
@@ -106,8 +107,8 @@ Specialized hook for page-level permissions.
 
 ```tsx
 const { canView, canCreate, canEdit, canDelete } = usePagePermission('products');
-if (canEdit()) {
-  // Can edit products
+if (canView()) {
+  // Can view products
 }
 ```
 
@@ -119,18 +120,43 @@ Permissions are stored in the database:
 
 Admin users automatically have all permissions.
 
-## Common Permission Codes
+## Implemented Permission Codes
 
+### Page Access Permissions
 | Code | Description |
 |------|-------------|
 | `access:dashboard` | Access to Dashboard page |
+| `access:overview` | Access to Overview page |
 | `access:products` | Access to Products page |
-| `create:products` | Create new products |
-| `edit:products` | Edit existing products |
-| `delete:products` | Delete products |
-| `view:users` | View user accounts |
-| `edit:users` | Edit user accounts |
-| `manage:permissions` | Assign permissions to users |
+| `access:categories` | Access to Categories page |
+| `access:brands` | Access to Brands page |
+| `access:vendors` | Access to Vendors page |
+| `access:purchase_orders` | Access to Purchase Orders page |
+| `access:analytics` | Access to Analytics page |
+| `access:forecasting` | Access to Forecasting page |
+| `access:import` | Access to Import page |
+| `access:settings` | Access to Settings page |
+| `access:chat` | Access to Chat Archive page |
+
+### Settings Permissions
+| Code | Description |
+|------|-------------|
+| `settings:global` | Access to Global Settings section |
+| `settings:products` | Access to Product Settings section |
+| `settings:vendors` | Access to Vendor Settings section |
+| `settings:data_management` | Access to Data Management settings |
+| `settings:calculation_settings` | Access to Calculation Settings |
+| `settings:library_management` | Access to Image Library Management |
+| `settings:performance_metrics` | Access to Performance Metrics |
+| `settings:prompt_management` | Access to AI Prompt Management |
+| `settings:stock_management` | Access to Stock Management |
+| `settings:templates` | Access to Template Management |
+| `settings:user_management` | Access to User Management |
+
+### Admin Permissions
+| Code | Description |
+|------|-------------|
+| `admin:debug` | Can see debug information and features |
 
 ## Implementation Examples
 
@@ -148,25 +174,31 @@ In `App.tsx`:
 ### Component Level Protection
 
 ```tsx
-const { canEdit } = usePagePermission('products');
+const { hasPermission } = usePermissions();
 
-function handleEdit() {
-  if (!canEdit()) {
+function handleAction() {
+  if (!hasPermission('settings:user_management')) {
     toast.error("You don't have permission");
     return;
   }
-  // Edit logic
+  // Action logic
 }
 ```
 
 ### UI Element Protection
 
 ```tsx
-<PermissionButton 
-  page="products" 
-  action="delete"
-  onClick={handleDelete}
->
-  Delete
-</PermissionButton>
-``` 
+<PermissionGuard permission="settings:user_management">
+  <button onClick={handleManageUsers}>
+    Manage Users
+  </button>
+</PermissionGuard>
+```
+
+## Notes
+
+- **Page Access**: These permissions control which pages a user can navigate to
+- **Settings Access**: These permissions control access to different sections within the Settings page
+- **Admin Features**: Special permissions for administrative functions
+- **CRUD Operations**: The application currently focuses on viewing and managing data rather than creating/editing/deleting individual records
+- **User Management**: User CRUD operations are handled through the settings interface rather than dedicated user management pages 

ecosystem.config.js

@@ -1,222 +0,0 @@
-// ecosystem.config.js
-const path = require('path');
-const dotenv = require('dotenv');
-
-// Load environment variables safely with error handling
-const loadEnvFile = (envPath) => {
-  try {
-    console.log('Loading env from:', envPath);
-    const result = dotenv.config({ path: envPath });
-    if (result.error) {
-      console.warn(`Warning: .env file not found or invalid at ${envPath}:`, result.error.message);
-      return {};
-    }
-    console.log('Env variables loaded from', envPath, ':', Object.keys(result.parsed || {}));
-    return result.parsed || {};
-  } catch (error) {
-    console.warn(`Warning: Error loading .env file at ${envPath}:`, error.message);
-    return {};
-  }
-}
-
-// Load environment variables for each server
-const authEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/auth-server/.env'));
-const aircallEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/aircall-server/.env'));
-const klaviyoEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/klaviyo-server/.env'));
-const metaEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/meta-server/.env'));
-const googleAnalyticsEnv = require('dotenv').config({ 
-  path: path.resolve(__dirname, 'dashboard/google-server/.env') 
-}).parsed || {};
-const typeformEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/typeform-server/.env'));
-const inventoryEnv = loadEnvFile(path.resolve(__dirname, 'inventory/.env'));
-
-// Common log settings for all apps
-const logSettings = {
-  log_rotate: true,
-  max_size: '10M',
-  retain: '10',
-  log_date_format: 'YYYY-MM-DD HH:mm:ss'
-};
-
-// Common app settings
-const commonSettings = {
-  instances: 1,
-  exec_mode: 'fork',
-  autorestart: true,
-  watch: false,
-  max_memory_restart: '1G',
-  time: true,
-  ...logSettings,
-  ignore_watch: [
-    'node_modules',
-    'logs',
-    '.git',
-    '*.log'
-  ],
-  min_uptime: 5000,
-  max_restarts: 5,
-  restart_delay: 4000,
-  listen_timeout: 50000,
-  kill_timeout: 5000,
-  node_args: '--max-old-space-size=1536'
-};
-
-module.exports = {
-  apps: [
-    {
-      ...commonSettings,
-      name: 'auth-server',
-      script: './dashboard/auth-server/index.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3003,
-        ...authEnv
-      },
-      error_file: 'dashboard/auth-server/logs/pm2/err.log',
-      out_file: 'dashboard/auth-server/logs/pm2/out.log',
-      log_file: 'dashboard/auth-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3003
-      },
-      env_development: {
-        NODE_ENV: 'development',
-        PORT: 3003
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'aircall-server',
-      script: './dashboard/aircall-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        AIRCALL_PORT: 3002,
-        ...aircallEnv
-      },
-      error_file: 'dashboard/aircall-server/logs/pm2/err.log',
-      out_file: 'dashboard/aircall-server/logs/pm2/out.log',
-      log_file: 'dashboard/aircall-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        AIRCALL_PORT: 3002
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'klaviyo-server',
-      script: './dashboard/klaviyo-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        KLAVIYO_PORT: 3004,
-        ...klaviyoEnv
-      },
-      error_file: 'dashboard/klaviyo-server/logs/pm2/err.log',
-      out_file: 'dashboard/klaviyo-server/logs/pm2/out.log',
-      log_file: 'dashboard/klaviyo-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        KLAVIYO_PORT: 3004
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'meta-server',
-      script: './dashboard/meta-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3005,
-        ...metaEnv
-      },
-      error_file: 'dashboard/meta-server/logs/pm2/err.log',
-      out_file: 'dashboard/meta-server/logs/pm2/out.log',
-      log_file: 'dashboard/meta-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3005
-      }
-    },
-    {
-      name: "gorgias-server",
-      script: "./dashboard/gorgias-server/server.js",
-      env: {
-        NODE_ENV: "development",
-        PORT: 3006
-      },
-      env_production: {
-        NODE_ENV: "production",
-        PORT: 3006
-      },
-      error_file: "dashboard/logs/gorgias-server-error.log",
-      out_file: "dashboard/logs/gorgias-server-out.log",
-      log_file: "dashboard/logs/gorgias-server-combined.log",
-      time: true
-    },
-    {
-      ...commonSettings,
-      name: 'google-server',
-      script: path.resolve(__dirname, 'dashboard/google-server/server.js'),
-      watch: false,
-      env: {
-        NODE_ENV: 'production',
-        GOOGLE_ANALYTICS_PORT: 3007,
-        ...googleAnalyticsEnv
-      },
-      error_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/err.log'),
-      out_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/out.log'),
-      log_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/combined.log'),
-      env_production: {
-        NODE_ENV: 'production',
-        GOOGLE_ANALYTICS_PORT: 3007
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'typeform-server',
-      script: './dashboard/typeform-server/server.js',
-      env: {
-        NODE_ENV: 'production',
-        TYPEFORM_PORT: 3008,
-        ...typeformEnv
-      },
-      error_file: 'dashboard/typeform-server/logs/pm2/err.log',
-      out_file: 'dashboard/typeform-server/logs/pm2/out.log',
-      log_file: 'dashboard/typeform-server/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        TYPEFORM_PORT: 3008
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'inventory-server',
-      script: './inventory/src/server.js',
-      env: {
-        NODE_ENV: 'production',
-        PORT: 3010,
-        ...inventoryEnv
-      },
-      error_file: 'inventory/logs/pm2/err.log',
-      out_file: 'inventory/logs/pm2/out.log',
-      log_file: 'inventory/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        PORT: 3010,
-        ...inventoryEnv
-      }
-    },
-    {
-      ...commonSettings,
-      name: 'new-auth-server',
-      script: './inventory-server/auth/server.js',
-      env: {
-        NODE_ENV: 'production',
-        AUTH_PORT: 3011,
-        ...inventoryEnv,
-        JWT_SECRET: process.env.JWT_SECRET
-      },
-      error_file: 'inventory-server/auth/logs/pm2/err.log',
-      out_file: 'inventory-server/auth/logs/pm2/out.log',
-      log_file: 'inventory-server/auth/logs/pm2/combined.log'
-    }
-  ]
-};

inventory-server/auth/add-user.js

@@ -1,103 +0,0 @@
-require('dotenv').config({ path: '../.env' });
-const bcrypt = require('bcrypt');
-const { Pool } = require('pg');
-const inquirer = require('inquirer');
-
-// Log connection details for debugging (remove in production)
-console.log('Attempting to connect with:', {
-  host: process.env.DB_HOST,
-  user: process.env.DB_USER,
-  database: process.env.DB_NAME,
-  port: process.env.DB_PORT
-});
-
-const pool = new Pool({
-  host: process.env.DB_HOST,
-  user: process.env.DB_USER,
-  password: process.env.DB_PASSWORD,
-  database: process.env.DB_NAME,
-  port: process.env.DB_PORT,
-});
-
-async function promptUser() {
-  const questions = [
-    {
-      type: 'input',
-      name: 'username',
-      message: 'Enter username:',
-      validate: (input) => {
-        if (input.length < 3) {
-          return 'Username must be at least 3 characters long';
-        }
-        return true;
-      }
-    },
-    {
-      type: 'password',
-      name: 'password',
-      message: 'Enter password:',
-      mask: '*',
-      validate: (input) => {
-        if (input.length < 8) {
-          return 'Password must be at least 8 characters long';
-        }
-        return true;
-      }
-    },
-    {
-      type: 'password',
-      name: 'confirmPassword',
-      message: 'Confirm password:',
-      mask: '*',
-      validate: (input, answers) => {
-        if (input !== answers.password) {
-          return 'Passwords do not match';
-        }
-        return true;
-      }
-    }
-  ];
-
-  return inquirer.prompt(questions);
-}
-
-async function addUser() {
-  try {
-    // Get user input
-    const answers = await promptUser();
-    const { username, password } = answers;
-
-    // Hash password
-    const saltRounds = 10;
-    const hashedPassword = await bcrypt.hash(password, saltRounds);
-
-    // Check if user already exists
-    const checkResult = await pool.query(
-      'SELECT id FROM users WHERE username = $1',
-      [username]
-    );
-
-    if (checkResult.rows.length > 0) {
-      console.error('Error: Username already exists');
-      process.exit(1);
-    }
-
-    // Insert new user
-    const result = await pool.query(
-      'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id',
-      [username, hashedPassword]
-    );
-
-    console.log(`User ${username} created successfully with id ${result.rows[0].id}`);
-  } catch (error) {
-    console.error('Error creating user:', error);
-    console.error('Error details:', error.message);
-    if (error.code) {
-      console.error('Error code:', error.code);
-    }
-  } finally {
-    await pool.end();
-  }
-}
-
-addUser(); 

inventory-server/auth/package.json

@@ -1,19 +0,0 @@
-{
-  "name": "inventory-auth-server",
-  "version": "1.0.0",
-  "description": "Authentication server for inventory management system",
-  "main": "server.js",
-  "scripts": {
-    "start": "node server.js"
-  },
-  "dependencies": {
-    "bcrypt": "^5.1.1",
-    "cors": "^2.8.5",
-    "dotenv": "^16.4.7",
-    "express": "^4.18.2",
-    "inquirer": "^8.2.6",
-    "jsonwebtoken": "^9.0.2",
-    "morgan": "^1.10.0",
-    "pg": "^8.11.3"
-  }
-}

inventory-server/auth/permissions.js

@@ -1,128 +0,0 @@
-// Get pool from global or create a new one if not available
-let pool;
-if (typeof global.pool !== 'undefined') {
-  pool = global.pool;
-} else {
-  // If global pool is not available, create a new connection
-  const { Pool } = require('pg');
-  pool = new Pool({
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT,
-  });
-  console.log('Created new database pool in permissions.js');
-}
-
-/**
- * Check if a user has a specific permission
- * @param {number} userId - The user ID to check
- * @param {string} permissionCode - The permission code to check
- * @returns {Promise<boolean>} - Whether the user has the permission
- */
-async function checkPermission(userId, permissionCode) {
-  try {
-    // First check if the user is an admin
-    const adminResult = await pool.query(
-      'SELECT is_admin FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    // If user is admin, automatically grant permission
-    if (adminResult.rows.length > 0 && adminResult.rows[0].is_admin) {
-      return true;
-    }
-    
-    // Otherwise check for specific permission
-    const result = await pool.query(
-      `SELECT COUNT(*) AS has_permission
-       FROM user_permissions up
-       JOIN permissions p ON up.permission_id = p.id
-       WHERE up.user_id = $1 AND p.code = $2`,
-      [userId, permissionCode]
-    );
-    
-    return result.rows[0].has_permission > 0;
-  } catch (error) {
-    console.error('Error checking permission:', error);
-    return false;
-  }
-}
-
-/**
- * Middleware to require a specific permission
- * @param {string} permissionCode - The permission code required
- * @returns {Function} - Express middleware function
- */
-function requirePermission(permissionCode) {
-  return async (req, res, next) => {
-    try {
-      // Check if user is authenticated
-      if (!req.user || !req.user.id) {
-        return res.status(401).json({ error: 'Authentication required' });
-      }
-      
-      const hasPermission = await checkPermission(req.user.id, permissionCode);
-      
-      if (!hasPermission) {
-        return res.status(403).json({ 
-          error: 'Insufficient permissions',
-          requiredPermission: permissionCode
-        });
-      }
-      
-      next();
-    } catch (error) {
-      console.error('Permission middleware error:', error);
-      res.status(500).json({ error: 'Server error checking permissions' });
-    }
-  };
-}
-
-/**
- * Get all permissions for a user
- * @param {number} userId - The user ID
- * @returns {Promise<string[]>} - Array of permission codes
- */
-async function getUserPermissions(userId) {
-  try {
-    // Check if user is admin
-    const adminResult = await pool.query(
-      'SELECT is_admin FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    if (adminResult.rows.length === 0) {
-      return [];
-    }
-    
-    const isAdmin = adminResult.rows[0].is_admin;
-    
-    if (isAdmin) {
-      // Admin gets all permissions
-      const allPermissions = await pool.query('SELECT code FROM permissions');
-      return allPermissions.rows.map(p => p.code);
-    } else {
-      // Get assigned permissions
-      const permissions = await pool.query(
-        `SELECT p.code
-         FROM permissions p
-         JOIN user_permissions up ON p.id = up.permission_id
-         WHERE up.user_id = $1`,
-        [userId]
-      );
-      
-      return permissions.rows.map(p => p.code);
-    }
-  } catch (error) {
-    console.error('Error getting user permissions:', error);
-    return [];
-  }
-}
-
-module.exports = {
-  checkPermission,
-  requirePermission,
-  getUserPermissions
-}; 

inventory-server/auth/routes.js

@@ -1,513 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const bcrypt = require('bcrypt');
-const jwt = require('jsonwebtoken');
-const { requirePermission, getUserPermissions } = require('./permissions');
-
-// Get pool from global or create a new one if not available
-let pool;
-if (typeof global.pool !== 'undefined') {
-  pool = global.pool;
-} else {
-  // If global pool is not available, create a new connection
-  const { Pool } = require('pg');
-  pool = new Pool({
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT,
-  });
-  console.log('Created new database pool in routes.js');
-}
-
-// Authentication middleware
-const authenticate = async (req, res, next) => {
-  try {
-    const authHeader = req.headers.authorization;
-    if (!authHeader || !authHeader.startsWith('Bearer ')) {
-      return res.status(401).json({ error: 'Authentication required' });
-    }
-
-    const token = authHeader.split(' ')[1];
-    const decoded = jwt.verify(token, process.env.JWT_SECRET);
-    
-    // Get user from database
-    const result = await pool.query(
-      'SELECT id, username, is_admin FROM users WHERE id = $1',
-      [decoded.userId]
-    );
-    
-    if (result.rows.length === 0) {
-      return res.status(401).json({ error: 'User not found' });
-    }
-    
-    // Attach user to request
-    req.user = result.rows[0];
-    next();
-  } catch (error) {
-    console.error('Authentication error:', error);
-    res.status(401).json({ error: 'Invalid token' });
-  }
-};
-
-// Login route
-router.post('/login', async (req, res) => {
-  try {
-    const { username, password } = req.body;
-    
-    // Get user from database
-    const result = await pool.query(
-      'SELECT id, username, password, is_admin, is_active FROM users WHERE username = $1',
-      [username]
-    );
-    
-    if (result.rows.length === 0) {
-      return res.status(401).json({ error: 'Invalid username or password' });
-    }
-    
-    const user = result.rows[0];
-    
-    // Check if user is active
-    if (!user.is_active) {
-      return res.status(403).json({ error: 'Account is inactive' });
-    }
-    
-    // Verify password
-    const validPassword = await bcrypt.compare(password, user.password);
-    if (!validPassword) {
-      return res.status(401).json({ error: 'Invalid username or password' });
-    }
-    
-    // Update last login
-    await pool.query(
-      'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
-      [user.id]
-    );
-    
-    // Generate JWT
-    const token = jwt.sign(
-      { userId: user.id, username: user.username },
-      process.env.JWT_SECRET,
-      { expiresIn: '8h' }
-    );
-    
-    // Get user permissions
-    const permissions = await getUserPermissions(user.id);
-    
-    res.json({
-      token,
-      user: {
-        id: user.id,
-        username: user.username,
-        is_admin: user.is_admin,
-        permissions
-      }
-    });
-  } catch (error) {
-    console.error('Login error:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get current user
-router.get('/me', authenticate, async (req, res) => {
-  try {
-    // Get user permissions
-    const permissions = await getUserPermissions(req.user.id);
-    
-    res.json({
-      id: req.user.id,
-      username: req.user.username,
-      is_admin: req.user.is_admin,
-      permissions
-    });
-  } catch (error) {
-    console.error('Error getting current user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get all users
-router.get('/users', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const result = await pool.query(`
-      SELECT id, username, email, is_admin, is_active, created_at, last_login
-      FROM users
-      ORDER BY username
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error getting users:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get user with permissions
-router.get('/users/:id', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const userId = req.params.id;
-    
-    // Get user details
-    const userResult = await pool.query(`
-      SELECT id, username, email, is_admin, is_active, created_at, last_login
-      FROM users
-      WHERE id = $1
-    `, [userId]);
-    
-    if (userResult.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    // Get user permissions
-    const permissionsResult = await pool.query(`
-      SELECT p.id, p.name, p.code, p.category, p.description
-      FROM permissions p
-      JOIN user_permissions up ON p.id = up.permission_id
-      WHERE up.user_id = $1
-      ORDER BY p.category, p.name
-    `, [userId]);
-    
-    // Combine user and permissions
-    const user = {
-      ...userResult.rows[0],
-      permissions: permissionsResult.rows
-    };
-    
-    res.json(user);
-  } catch (error) {
-    console.error('Error getting user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Create new user
-router.post('/users', authenticate, requirePermission('create:users'), async (req, res) => {
-  const client = await pool.connect();
-  
-  try {
-    const { username, email, password, is_admin, is_active, permissions } = req.body;
-    
-    console.log("Create user request:", {
-      username,
-      email,
-      is_admin,
-      is_active,
-      permissions: permissions || []
-    });
-    
-    // Validate required fields
-    if (!username || !password) {
-      return res.status(400).json({ error: 'Username and password are required' });
-    }
-    
-    // Check if username is taken
-    const existingUser = await client.query(
-      'SELECT id FROM users WHERE username = $1',
-      [username]
-    );
-    
-    if (existingUser.rows.length > 0) {
-      return res.status(400).json({ error: 'Username already exists' });
-    }
-    
-    // Start transaction
-    await client.query('BEGIN');
-    
-    // Hash password
-    const saltRounds = 10;
-    const hashedPassword = await bcrypt.hash(password, saltRounds);
-    
-    // Insert new user
-    const userResult = await client.query(`
-      INSERT INTO users (username, email, password, is_admin, is_active, created_at)
-      VALUES ($1, $2, $3, $4, $5, CURRENT_TIMESTAMP)
-      RETURNING id
-    `, [username, email || null, hashedPassword, !!is_admin, is_active !== false]);
-    
-    const userId = userResult.rows[0].id;
-    
-    // Assign permissions if provided and not admin
-    if (!is_admin && Array.isArray(permissions) && permissions.length > 0) {
-      console.log("Adding permissions for new user:", userId);
-      console.log("Permissions received:", permissions);
-      
-      // Check permission format
-      const permissionIds = permissions.map(p => {
-        if (typeof p === 'object' && p.id) {
-          console.log("Permission is an object with ID:", p.id);
-          return parseInt(p.id, 10);
-        } else if (typeof p === 'number') {
-          console.log("Permission is a number:", p);
-          return p;
-        } else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
-          console.log("Permission is a string that can be parsed as a number:", p);
-          return parseInt(p, 10);
-        } else {
-          console.log("Unknown permission format:", typeof p, p);
-          // If it's a permission code, we need to look up the ID
-          return null;
-        }
-      }).filter(id => id !== null);
-      
-      console.log("Filtered permission IDs:", permissionIds);
-      
-      if (permissionIds.length > 0) {
-        const permissionValues = permissionIds
-          .map(permId => `(${userId}, ${permId})`)
-          .join(',');
-        
-        console.log("Inserting permission values:", permissionValues);
-        
-        try {
-          await client.query(`
-            INSERT INTO user_permissions (user_id, permission_id)
-            VALUES ${permissionValues}
-            ON CONFLICT DO NOTHING
-          `);
-          console.log("Successfully inserted permissions for new user:", userId);
-        } catch (err) {
-          console.error("Error inserting permissions for new user:", err);
-          throw err;
-        }
-      } else {
-        console.log("No valid permission IDs found to insert for new user");
-      }
-    } else {
-      console.log("Not adding permissions: is_admin =", is_admin, "permissions array:", Array.isArray(permissions), "length:", permissions ? permissions.length : 0);
-    }
-    
-    await client.query('COMMIT');
-    
-    res.status(201).json({ 
-      id: userId, 
-      message: 'User created successfully' 
-    });
-  } catch (error) {
-    await client.query('ROLLBACK');
-    console.error('Error creating user:', error);
-    res.status(500).json({ error: 'Server error' });
-  } finally {
-    client.release();
-  }
-});
-
-// Update user
-router.put('/users/:id', authenticate, requirePermission('edit:users'), async (req, res) => {
-  const client = await pool.connect();
-  
-  try {
-    const userId = req.params.id;
-    const { username, email, password, is_admin, is_active, permissions } = req.body;
-    
-    console.log("Update user request:", {
-      userId,
-      username,
-      email,
-      is_admin,
-      is_active,
-      permissions: permissions || []
-    });
-    
-    // Check if user exists
-    const userExists = await client.query(
-      'SELECT id FROM users WHERE id = $1',
-      [userId]
-    );
-    
-    if (userExists.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    // Start transaction
-    await client.query('BEGIN');
-    
-    // Build update fields
-    const updateFields = [];
-    const updateValues = [userId]; // First parameter is the user ID
-    let paramIndex = 2;
-    
-    if (username !== undefined) {
-      updateFields.push(`username = $${paramIndex++}`);
-      updateValues.push(username);
-    }
-    
-    if (email !== undefined) {
-      updateFields.push(`email = $${paramIndex++}`);
-      updateValues.push(email || null);
-    }
-    
-    if (is_admin !== undefined) {
-      updateFields.push(`is_admin = $${paramIndex++}`);
-      updateValues.push(!!is_admin);
-    }
-    
-    if (is_active !== undefined) {
-      updateFields.push(`is_active = $${paramIndex++}`);
-      updateValues.push(!!is_active);
-    }
-    
-    // Update password if provided
-    if (password) {
-      const saltRounds = 10;
-      const hashedPassword = await bcrypt.hash(password, saltRounds);
-      updateFields.push(`password = $${paramIndex++}`);
-      updateValues.push(hashedPassword);
-    }
-    
-    // Update user if there are fields to update
-    if (updateFields.length > 0) {
-      updateFields.push(`updated_at = CURRENT_TIMESTAMP`);
-      
-      await client.query(`
-        UPDATE users
-        SET ${updateFields.join(', ')}
-        WHERE id = $1
-      `, updateValues);
-    }
-    
-    // Update permissions if provided
-    if (Array.isArray(permissions)) {
-      console.log("Updating permissions for user:", userId);
-      console.log("Permissions received:", permissions);
-      
-      // First remove existing permissions
-      await client.query(
-        'DELETE FROM user_permissions WHERE user_id = $1',
-        [userId]
-      );
-      console.log("Deleted existing permissions for user:", userId);
-      
-      // Add new permissions if any and not admin
-      const newIsAdmin = is_admin !== undefined ? is_admin : (await client.query('SELECT is_admin FROM users WHERE id = $1', [userId])).rows[0].is_admin;
-      
-      console.log("User is admin:", newIsAdmin);
-      
-      if (!newIsAdmin && permissions.length > 0) {
-        console.log("Adding permissions:", permissions);
-        
-        // Check permission format
-        const permissionIds = permissions.map(p => {
-          if (typeof p === 'object' && p.id) {
-            console.log("Permission is an object with ID:", p.id);
-            return parseInt(p.id, 10);
-          } else if (typeof p === 'number') {
-            console.log("Permission is a number:", p);
-            return p;
-          } else if (typeof p === 'string' && !isNaN(parseInt(p, 10))) {
-            console.log("Permission is a string that can be parsed as a number:", p);
-            return parseInt(p, 10);
-          } else {
-            console.log("Unknown permission format:", typeof p, p);
-            // If it's a permission code, we need to look up the ID
-            return null;
-          }
-        }).filter(id => id !== null);
-        
-        console.log("Filtered permission IDs:", permissionIds);
-        
-        if (permissionIds.length > 0) {
-          const permissionValues = permissionIds
-            .map(permId => `(${userId}, ${permId})`)
-            .join(',');
-          
-          console.log("Inserting permission values:", permissionValues);
-          
-          try {
-            await client.query(`
-              INSERT INTO user_permissions (user_id, permission_id)
-              VALUES ${permissionValues}
-              ON CONFLICT DO NOTHING
-            `);
-            console.log("Successfully inserted permissions for user:", userId);
-          } catch (err) {
-            console.error("Error inserting permissions:", err);
-            throw err;
-          }
-        } else {
-          console.log("No valid permission IDs found to insert");
-        }
-      }
-    }
-    
-    await client.query('COMMIT');
-    
-    res.json({ message: 'User updated successfully' });
-  } catch (error) {
-    await client.query('ROLLBACK');
-    console.error('Error updating user:', error);
-    res.status(500).json({ error: 'Server error' });
-  } finally {
-    client.release();
-  }
-});
-
-// Delete user
-router.delete('/users/:id', authenticate, requirePermission('delete:users'), async (req, res) => {
-  try {
-    const userId = req.params.id;
-    
-    // Check that user is not deleting themselves
-    if (req.user.id === parseInt(userId, 10)) {
-      return res.status(400).json({ error: 'Cannot delete your own account' });
-    }
-    
-    // Delete user (this will cascade to user_permissions due to FK constraints)
-    const result = await pool.query(
-      'DELETE FROM users WHERE id = $1 RETURNING id',
-      [userId]
-    );
-    
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'User not found' });
-    }
-    
-    res.json({ message: 'User deleted successfully' });
-  } catch (error) {
-    console.error('Error deleting user:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get all permissions grouped by category
-router.get('/permissions/categories', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const result = await pool.query(`
-      SELECT category, json_agg(
-        json_build_object(
-          'id', id,
-          'name', name,
-          'code', code,
-          'description', description
-        ) ORDER BY name
-      ) as permissions
-      FROM permissions
-      GROUP BY category
-      ORDER BY category
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error getting permissions:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Get all permissions
-router.get('/permissions', authenticate, requirePermission('view:users'), async (req, res) => {
-  try {
-    const result = await pool.query(`
-      SELECT *
-      FROM permissions
-      ORDER BY category, name
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error getting permissions:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-module.exports = router; 

inventory-server/auth/schema.sql

@@ -1,89 +0,0 @@
-CREATE TABLE users (
-  id SERIAL PRIMARY KEY,
-  username VARCHAR(255) NOT NULL UNIQUE,
-  password VARCHAR(255) NOT NULL,
-  email VARCHAR UNIQUE,
-  is_admin BOOLEAN DEFAULT FALSE,
-  is_active BOOLEAN DEFAULT TRUE,
-  last_login TIMESTAMP WITH TIME ZONE,
-  updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-); 
-
--- Function to update the updated_at timestamp
-CREATE OR REPLACE FUNCTION update_updated_at_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated_at = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Sequence and defined type for users table if not exists
-CREATE SEQUENCE IF NOT EXISTS users_id_seq;
-
--- Create permissions table
-CREATE TABLE IF NOT EXISTS "public"."permissions" (
-    "id" SERIAL PRIMARY KEY,
-    "name" varchar NOT NULL UNIQUE,
-    "code" varchar NOT NULL UNIQUE,
-    "description" text,
-    "category" varchar NOT NULL,
-    "created_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
-    "updated_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP
-);
-
--- Create user_permissions junction table
-CREATE TABLE IF NOT EXISTS "public"."user_permissions" (
-    "user_id" int4 NOT NULL REFERENCES "public"."users"("id") ON DELETE CASCADE,
-    "permission_id" int4 NOT NULL REFERENCES "public"."permissions"("id") ON DELETE CASCADE,
-    "created_at" timestamp with time zone DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY ("user_id", "permission_id")
-);
-
--- Add triggers for updated_at on users and permissions
-DROP TRIGGER IF EXISTS update_users_updated_at ON users;
-CREATE TRIGGER update_users_updated_at
-    BEFORE UPDATE ON users
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
-DROP TRIGGER IF EXISTS update_permissions_updated_at ON permissions;
-CREATE TRIGGER update_permissions_updated_at
-    BEFORE UPDATE ON permissions
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Insert default permissions by page - only the ones used in application
-INSERT INTO permissions (name, code, description, category) VALUES 
-    ('Dashboard Access', 'access:dashboard', 'Can access the Dashboard page', 'Pages'),
-    ('Products Access', 'access:products', 'Can access the Products page', 'Pages'),
-    ('Categories Access', 'access:categories', 'Can access the Categories page', 'Pages'),
-    ('Vendors Access', 'access:vendors', 'Can access the Vendors page', 'Pages'),
-    ('Analytics Access', 'access:analytics', 'Can access the Analytics page', 'Pages'),
-    ('Forecasting Access', 'access:forecasting', 'Can access the Forecasting page', 'Pages'),
-    ('Purchase Orders Access', 'access:purchase_orders', 'Can access the Purchase Orders page', 'Pages'),
-    ('Import Access', 'access:import', 'Can access the Import page', 'Pages'),
-    ('Settings Access', 'access:settings', 'Can access the Settings page', 'Pages'),
-    ('AI Validation Debug Access', 'access:ai_validation_debug', 'Can access the AI Validation Debug page', 'Pages')
-ON CONFLICT (code) DO NOTHING;
-
--- Settings section permissions
-INSERT INTO permissions (name, code, description, category) VALUES
-    ('Data Management', 'settings:data_management', 'Access to the Data Management settings section', 'Settings'),
-    ('Stock Management', 'settings:stock_management', 'Access to the Stock Management settings section', 'Settings'),
-    ('Performance Metrics', 'settings:performance_metrics', 'Access to the Performance Metrics settings section', 'Settings'),
-    ('Calculation Settings', 'settings:calculation_settings', 'Access to the Calculation Settings section', 'Settings'),
-    ('Template Management', 'settings:templates', 'Access to the Template Management settings section', 'Settings'),
-    ('User Management', 'settings:user_management', 'Access to the User Management settings section', 'Settings')
-ON CONFLICT (code) DO NOTHING;
-
--- Set any existing users as admin
-UPDATE users SET is_admin = TRUE WHERE is_admin IS NULL;
-
--- Grant all permissions to admin users
-INSERT INTO user_permissions (user_id, permission_id)
-SELECT u.id, p.id
-FROM users u, permissions p
-WHERE u.is_admin = TRUE
-ON CONFLICT DO NOTHING; 

inventory-server/auth/server.js

@@ -35,7 +35,7 @@ global.pool = pool;
 app.use(express.json());
 app.use(morgan('combined'));
 app.use(cors({
-  origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw'],
+  origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw', 'https://tools.acherryontop.com', 'https://tools.acherryontop.com'],
   credentials: true
 }));
 
@@ -62,6 +62,12 @@ app.post('/login', async (req, res) => {
       return res.status(403).json({ error: 'Account is inactive' });
     }
 
+    // Update last login timestamp
+    await pool.query(
+      'UPDATE users SET last_login = CURRENT_TIMESTAMP WHERE id = $1',
+      [user.id]
+    );
+
     // Generate JWT token
     const token = jwt.sign(
       { userId: user.id, username: user.username },
@@ -76,7 +82,7 @@ app.post('/login', async (req, res) => {
       JOIN user_permissions up ON p.id = up.permission_id
       WHERE up.user_id = $1
     `, [user.id]);
-    
+
     const permissions = permissionsResult.rows.map(row => row.code);
 
     res.json({ 
@@ -108,7 +114,7 @@ app.get('/me', async (req, res) => {
     
     // Get user details from database
     const userResult = await pool.query(
-      'SELECT id, username, email, is_admin, is_active FROM users WHERE id = $1',
+      'SELECT id, username, email, is_admin, rocket_chat_user_id, is_active FROM users WHERE id = $1',
       [decoded.userId]
     );
     
@@ -135,6 +141,7 @@ app.get('/me', async (req, res) => {
       id: user.id,
       username: user.username,
       email: user.email,
+      rocket_chat_user_id: user.rocket_chat_user_id,
       is_admin: user.is_admin,
       permissions: permissions
     });

inventory-server/chat/db-convert/mongo_to_postgres_converter.py

@@ -1,881 +0,0 @@
-#!/usr/bin/env python3
-"""
-MongoDB to PostgreSQL Converter for Rocket.Chat
-Converts MongoDB BSON export files to PostgreSQL database
-
-Usage:
-    python3 mongo_to_postgres_converter.py \
-        --mongo-path db/database/62df06d44234d20001289144 \
-        --pg-database rocketchat_converted \
-        --pg-user rocketchat_user \
-        --pg-password your_password \
-        --debug
-"""
-
-import json
-import os
-import re
-import subprocess
-import sys
-import struct
-from datetime import datetime
-from pathlib import Path
-from typing import Dict, Any, List, Optional
-import argparse
-import traceback
-
-# Auto-install dependencies if needed
-try:
-    import bson
-    import psycopg2
-except ImportError:
-    print("Installing required packages...")
-    subprocess.check_call([sys.executable, "-m", "pip", "install", "pymongo", "psycopg2-binary"])
-    import bson
-    import psycopg2
-
-class MongoToPostgresConverter:
-    def __init__(self, mongo_db_path: str, postgres_config: Dict[str, str], debug_mode: bool = False, debug_collections: List[str] = None):
-        self.mongo_db_path = Path(mongo_db_path)
-        self.postgres_config = postgres_config
-        self.debug_mode = debug_mode
-        self.debug_collections = debug_collections or []
-        self.collections = {}
-        self.schema_info = {}
-        self.error_log = {}
-        
-    def log_debug(self, message: str, collection: str = None):
-        """Log debug messages if debug mode is enabled and collection is in debug list"""
-        if self.debug_mode and (not self.debug_collections or collection in self.debug_collections):
-            print(f"DEBUG: {message}")
-            
-    def log_error(self, collection: str, error_type: str, details: str):
-        """Log detailed error information"""
-        if collection not in self.error_log:
-            self.error_log[collection] = []
-        self.error_log[collection].append({
-            'type': error_type,
-            'details': details,
-            'timestamp': datetime.now().isoformat()
-        })
-        
-    def sample_documents(self, collection_name: str, max_samples: int = 3) -> List[Dict]:
-        """Sample documents from a collection for debugging"""
-        if not self.debug_mode or (self.debug_collections and collection_name not in self.debug_collections):
-            return []
-            
-        print(f"\n🔍 Sampling documents from {collection_name}:")
-        
-        bson_file = self.collections[collection_name]['bson_file']
-        if bson_file.stat().st_size == 0:
-            print("  Collection is empty")
-            return []
-            
-        samples = []
-        
-        try:
-            with open(bson_file, 'rb') as f:
-                sample_count = 0
-                while sample_count < max_samples:
-                    try:
-                        doc_size = int.from_bytes(f.read(4), byteorder='little')
-                        if doc_size <= 0:
-                            break
-                        f.seek(-4, 1)
-                        doc_bytes = f.read(doc_size)
-                        if len(doc_bytes) != doc_size:
-                            break
-                            
-                        doc = bson.decode(doc_bytes)
-                        samples.append(doc)
-                        sample_count += 1
-                        
-                        print(f"  Sample {sample_count} - Keys: {list(doc.keys())}")
-                        # Show a few key fields with their types and truncated values
-                        for key, value in list(doc.items())[:3]:
-                            value_preview = str(value)[:50] + "..." if len(str(value)) > 50 else str(value)
-                            print(f"    {key}: {type(value).__name__} = {value_preview}")
-                        if len(doc) > 3:
-                            print(f"    ... and {len(doc) - 3} more fields")
-                        print()
-                        
-                    except (bson.InvalidBSON, struct.error, OSError) as e:
-                        self.log_error(collection_name, 'document_parsing', str(e))
-                        break
-                        
-        except Exception as e:
-            self.log_error(collection_name, 'file_reading', str(e))
-            print(f"  Error reading collection: {e}")
-            
-        return samples
-
-    def discover_collections(self):
-        """Discover all BSON files and their metadata"""
-        print("Discovering MongoDB collections...")
-        
-        for bson_file in self.mongo_db_path.glob("*.bson"):
-            collection_name = bson_file.stem
-            metadata_file = bson_file.with_suffix(".metadata.json")
-            
-            # Read metadata if available
-            metadata = {}
-            if metadata_file.exists():
-                try:
-                    with open(metadata_file, 'r', encoding='utf-8') as f:
-                        metadata = json.load(f)
-                except (UnicodeDecodeError, json.JSONDecodeError) as e:
-                    print(f"Warning: Could not read metadata for {collection_name}: {e}")
-                    metadata = {}
-            
-            # Get file size and document count estimate
-            file_size = bson_file.stat().st_size
-            doc_count = self._estimate_document_count(bson_file)
-            
-            self.collections[collection_name] = {
-                'bson_file': bson_file,
-                'metadata': metadata,
-                'file_size': file_size,
-                'estimated_docs': doc_count
-            }
-            
-        print(f"Found {len(self.collections)} collections")
-        for name, info in self.collections.items():
-            print(f"  - {name}: {info['file_size']/1024/1024:.1f}MB (~{info['estimated_docs']} docs)")
-    
-    def _estimate_document_count(self, bson_file: Path) -> int:
-        """Estimate document count by reading first few documents"""
-        if bson_file.stat().st_size == 0:
-            return 0
-            
-        try:
-            with open(bson_file, 'rb') as f:
-                docs_sampled = 0
-                bytes_sampled = 0
-                max_sample_size = min(1024 * 1024, bson_file.stat().st_size)  # 1MB or file size
-                
-                while bytes_sampled < max_sample_size:
-                    try:
-                        doc_size = int.from_bytes(f.read(4), byteorder='little')
-                        if doc_size <= 0 or doc_size > 16 * 1024 * 1024:  # MongoDB doc size limit
-                            break
-                        f.seek(-4, 1)  # Go back
-                        doc_bytes = f.read(doc_size)
-                        if len(doc_bytes) != doc_size:
-                            break
-                        bson.decode(doc_bytes)  # Validate it's a valid BSON document
-                        docs_sampled += 1
-                        bytes_sampled += doc_size
-                    except (bson.InvalidBSON, struct.error, OSError):
-                        break
-                
-                if docs_sampled > 0 and bytes_sampled > 0:
-                    avg_doc_size = bytes_sampled / docs_sampled
-                    return int(bson_file.stat().st_size / avg_doc_size)
-                    
-        except Exception:
-            pass
-            
-        return 0
-    
-    def analyze_schema(self, collection_name: str, sample_size: int = 100) -> Dict[str, Any]:
-        """Analyze collection schema by sampling documents"""
-        print(f"Analyzing schema for {collection_name}...")
-        
-        bson_file = self.collections[collection_name]['bson_file']
-        if bson_file.stat().st_size == 0:
-            return {}
-            
-        schema = {}
-        docs_analyzed = 0
-        
-        try:
-            with open(bson_file, 'rb') as f:
-                while docs_analyzed < sample_size:
-                    try:
-                        doc_size = int.from_bytes(f.read(4), byteorder='little')
-                        if doc_size <= 0:
-                            break
-                        f.seek(-4, 1)
-                        doc_bytes = f.read(doc_size)
-                        if len(doc_bytes) != doc_size:
-                            break
-                            
-                        doc = bson.decode(doc_bytes)
-                        self._analyze_document_schema(doc, schema)
-                        docs_analyzed += 1
-                        
-                    except (bson.InvalidBSON, struct.error, OSError):
-                        break
-                        
-        except Exception as e:
-            print(f"Error analyzing {collection_name}: {e}")
-            
-        self.schema_info[collection_name] = schema
-        return schema
-    
-    def _analyze_document_schema(self, doc: Dict[str, Any], schema: Dict[str, Any], prefix: str = ""):
-        """Recursively analyze document structure"""
-        for key, value in doc.items():
-            full_key = f"{prefix}.{key}" if prefix else key
-            
-            if full_key not in schema:
-                schema[full_key] = {
-                    'types': set(),
-                    'null_count': 0,
-                    'total_count': 0,
-                    'is_array': False,
-                    'nested_schema': {}
-                }
-            
-            schema[full_key]['total_count'] += 1
-            
-            if value is None:
-                schema[full_key]['null_count'] += 1
-                schema[full_key]['types'].add('null')
-            elif isinstance(value, dict):
-                schema[full_key]['types'].add('object')
-                if 'nested_schema' not in schema[full_key]:
-                    schema[full_key]['nested_schema'] = {}
-                self._analyze_document_schema(value, schema[full_key]['nested_schema'])
-            elif isinstance(value, list):
-                schema[full_key]['types'].add('array')
-                schema[full_key]['is_array'] = True
-                if value and isinstance(value[0], dict):
-                    if 'array_item_schema' not in schema[full_key]:
-                        schema[full_key]['array_item_schema'] = {}
-                    for item in value[:5]:  # Sample first 5 items
-                        if isinstance(item, dict):
-                            self._analyze_document_schema(item, schema[full_key]['array_item_schema'])
-            else:
-                schema[full_key]['types'].add(type(value).__name__)
-    
-    def generate_postgres_schema(self) -> Dict[str, str]:
-        """Generate PostgreSQL CREATE TABLE statements"""
-        print("Generating PostgreSQL schema...")
-        
-        table_definitions = {}
-        
-        for collection_name, schema in self.schema_info.items():
-            if not schema:  # Empty collection
-                continue
-                
-            table_name = self._sanitize_table_name(collection_name)
-            columns = []
-            
-            # Always add an id column (PostgreSQL doesn't use _id like MongoDB)
-            columns.append("id SERIAL PRIMARY KEY")
-            
-            for field_name, field_info in schema.items():
-                if field_name == '_id':
-                    columns.append("mongo_id TEXT")  # Always allow NULL for mongo_id
-                    continue
-                    
-                col_name = self._sanitize_column_name(field_name)
-                
-                # Handle conflicts with PostgreSQL auto-generated columns
-                if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
-                    col_name = f"field_{col_name}"
-                
-                col_type = self._determine_postgres_type(field_info)
-                
-                # Make all fields nullable by default to avoid constraint violations
-                columns.append(f"{col_name} {col_type}")
-            
-            # Add metadata columns
-            columns.extend([
-                "created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP",
-                "updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP"
-            ])
-            
-            column_definitions = ',\n    '.join(columns)
-            table_sql = f"""
-CREATE TABLE IF NOT EXISTS {table_name} (
-    {column_definitions}
-);
-
--- Create indexes based on MongoDB indexes
-"""
-            
-            # Get list of actual columns that will exist in the table
-            existing_columns = set(['id', 'mongo_id', 'created_at', 'updated_at'])
-            for field_name in schema.keys():
-                if field_name != '_id':
-                    col_name = self._sanitize_column_name(field_name)
-                    # Handle conflicts with PostgreSQL auto-generated columns
-                    if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
-                        col_name = f"field_{col_name}"
-                    existing_columns.add(col_name)
-            
-            # Add indexes from MongoDB metadata
-            metadata = self.collections[collection_name].get('metadata', {})
-            indexes = metadata.get('indexes', [])
-            
-            for index in indexes:
-                if index['name'] != '_id_':  # Skip the default _id index
-                    # Sanitize index name - remove special characters
-                    sanitized_index_name = re.sub(r'[^a-zA-Z0-9_]', '_', index['name'])
-                    index_name = f"idx_{table_name}_{sanitized_index_name}"
-                    index_keys = list(index['key'].keys())
-                    if index_keys:
-                        sanitized_keys = []
-                        for key in index_keys:
-                            if key != '_id':
-                                sanitized_key = self._sanitize_column_name(key)
-                                # Handle conflicts with PostgreSQL auto-generated columns
-                                if sanitized_key in ['id', 'mongo_id', 'created_at', 'updated_at']:
-                                    sanitized_key = f"field_{sanitized_key}"
-                                # Only add if the column actually exists in our table
-                                if sanitized_key in existing_columns:
-                                    sanitized_keys.append(sanitized_key)
-                        
-                        if sanitized_keys:
-                            table_sql += f"CREATE INDEX IF NOT EXISTS {index_name} ON {table_name} ({', '.join(sanitized_keys)});\n"
-            
-            table_definitions[collection_name] = table_sql
-            
-        return table_definitions
-    
-    def _sanitize_table_name(self, name: str) -> str:
-        """Convert MongoDB collection name to PostgreSQL table name"""
-        # Remove rocketchat_ prefix if present
-        if name.startswith('rocketchat_'):
-            name = name[11:]
-        
-        # Replace special characters with underscores
-        name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
-        
-        # Ensure it starts with a letter
-        if name and name[0].isdigit():
-            name = 'table_' + name
-            
-        return name.lower()
-    
-    def _sanitize_column_name(self, name: str) -> str:
-        """Convert MongoDB field name to PostgreSQL column name"""
-        # Handle nested field names (convert dots to underscores)
-        name = name.replace('.', '_')
-        
-        # Replace special characters with underscores
-        name = re.sub(r'[^a-zA-Z0-9_]', '_', name)
-        
-        # Ensure it starts with a letter or underscore
-        if name and name[0].isdigit():
-            name = 'col_' + name
-            
-        # Handle PostgreSQL reserved words
-        reserved = {
-            'user', 'order', 'group', 'table', 'index', 'key', 'value', 'date', 'time', 'timestamp',
-            'default', 'select', 'from', 'where', 'insert', 'update', 'delete', 'create', 'drop',
-            'alter', 'grant', 'revoke', 'commit', 'rollback', 'begin', 'end', 'case', 'when',
-            'then', 'else', 'if', 'null', 'not', 'and', 'or', 'in', 'exists', 'between',
-            'like', 'limit', 'offset', 'union', 'join', 'inner', 'outer', 'left', 'right',
-            'full', 'cross', 'natural', 'on', 'using', 'distinct', 'all', 'any', 'some',
-            'desc', 'asc', 'primary', 'foreign', 'references', 'constraint', 'unique',
-            'check', 'cascade', 'restrict', 'action', 'match', 'partial', 'full'
-        }
-        if name.lower() in reserved:
-            name = name + '_col'
-            
-        return name.lower()
-    
-    def _determine_postgres_type(self, field_info: Dict[str, Any]) -> str:
-        """Determine PostgreSQL column type from MongoDB field analysis with improved logic"""
-        types = field_info['types']
-        
-        # Convert set to list for easier checking
-        type_list = list(types)
-        
-        # If there's only one type (excluding null), use specific typing
-        non_null_types = [t for t in type_list if t != 'null']
-        
-        if len(non_null_types) == 1:
-            single_type = non_null_types[0]
-            if single_type == 'bool':
-                return 'BOOLEAN'
-            elif single_type == 'int':
-                return 'INTEGER'
-            elif single_type == 'float':
-                return 'NUMERIC'
-            elif single_type == 'str':
-                return 'TEXT'
-            elif single_type == 'datetime':
-                return 'TIMESTAMP'
-            elif single_type == 'ObjectId':
-                return 'TEXT'
-        
-        # Handle mixed types more conservatively
-        if 'array' in types or field_info.get('is_array', False):
-            return 'JSONB'  # Arrays always go to JSONB
-        elif 'object' in types:
-            return 'JSONB'  # Objects always go to JSONB
-        elif len(non_null_types) > 1:
-            # Multiple non-null types - check for common combinations
-            if set(non_null_types) <= {'int', 'float'}:
-                return 'NUMERIC'  # Can handle both int and float
-            elif set(non_null_types) <= {'bool', 'str'}:
-                return 'TEXT'  # Convert everything to text
-            elif set(non_null_types) <= {'str', 'ObjectId'}:
-                return 'TEXT'  # Both are string-like
-            else:
-                return 'JSONB'  # Complex mixed types go to JSONB
-        elif 'ObjectId' in types:
-            return 'TEXT'
-        elif 'datetime' in types:
-            return 'TIMESTAMP'
-        elif 'bool' in types:
-            return 'BOOLEAN'
-        elif 'int' in types:
-            return 'INTEGER'
-        elif 'float' in types:
-            return 'NUMERIC'
-        elif 'str' in types:
-            return 'TEXT'
-        else:
-            return 'TEXT'  # Default fallback
-    
-    def create_postgres_database(self, table_definitions: Dict[str, str]):
-        """Create PostgreSQL database and tables"""
-        print("Creating PostgreSQL database schema...")
-        
-        try:
-            # Connect to PostgreSQL
-            conn = psycopg2.connect(**self.postgres_config)
-            conn.autocommit = True
-            cursor = conn.cursor()
-            
-            # Create tables
-            for collection_name, table_sql in table_definitions.items():
-                print(f"Creating table for {collection_name}...")
-                cursor.execute(table_sql)
-            
-            cursor.close()
-            conn.close()
-            print("Database schema created successfully!")
-            
-        except Exception as e:
-            print(f"Error creating database schema: {e}")
-            raise
-    
-    def convert_and_insert_data(self, batch_size: int = 1000):
-        """Convert BSON data and insert into PostgreSQL"""
-        print("Converting and inserting data...")
-        
-        try:
-            conn = psycopg2.connect(**self.postgres_config)
-            conn.autocommit = False
-            
-            for collection_name in self.collections:
-                print(f"Processing {collection_name}...")
-                self._convert_collection(conn, collection_name, batch_size)
-            
-            conn.close()
-            print("Data conversion completed successfully!")
-            
-        except Exception as e:
-            print(f"Error converting data: {e}")
-            raise
-    
-    def _convert_collection(self, conn, collection_name: str, batch_size: int):
-        """Convert a single collection"""
-        bson_file = self.collections[collection_name]['bson_file']
-        
-        if bson_file.stat().st_size == 0:
-            print(f"  Skipping empty collection {collection_name}")
-            return
-            
-        table_name = self._sanitize_table_name(collection_name)
-        cursor = conn.cursor()
-        
-        batch = []
-        total_inserted = 0
-        errors = 0
-        
-        try:
-            with open(bson_file, 'rb') as f:
-                while True:
-                    try:
-                        doc_size = int.from_bytes(f.read(4), byteorder='little')
-                        if doc_size <= 0:
-                            break
-                        f.seek(-4, 1)
-                        doc_bytes = f.read(doc_size)
-                        if len(doc_bytes) != doc_size:
-                            break
-                            
-                        doc = bson.decode(doc_bytes)
-                        batch.append(doc)
-                        
-                        if len(batch) >= batch_size:
-                            inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
-                            total_inserted += inserted
-                            errors += batch_errors
-                            batch = []
-                            conn.commit()
-                            if total_inserted % 5000 == 0:  # Less frequent progress updates
-                                print(f"  Inserted {total_inserted} documents...")
-                            
-                    except (bson.InvalidBSON, struct.error, OSError):
-                        break
-                
-                # Insert remaining documents
-                if batch:
-                    inserted, batch_errors = self._insert_batch(cursor, table_name, batch, collection_name)
-                    total_inserted += inserted
-                    errors += batch_errors
-                    conn.commit()
-                    
-            if errors > 0:
-                print(f"  Completed {collection_name}: {total_inserted} documents inserted ({errors} errors)")
-            else:
-                print(f"  Completed {collection_name}: {total_inserted} documents inserted")
-            
-        except Exception as e:
-            print(f"  Error processing {collection_name}: {e}")
-            conn.rollback()
-        finally:
-            cursor.close()
-    
-    def _insert_batch(self, cursor, table_name: str, documents: List[Dict], collection_name: str):
-        """Insert a batch of documents with proper transaction handling"""
-        if not documents:
-            return 0, 0
-            
-        # Get schema info for this collection
-        schema = self.schema_info.get(collection_name, {})
-        
-        # Build column list
-        columns = ['mongo_id']
-        for field_name in schema.keys():
-            if field_name != '_id':
-                col_name = self._sanitize_column_name(field_name)
-                # Handle conflicts with PostgreSQL auto-generated columns
-                if col_name in ['id', 'mongo_id', 'created_at', 'updated_at']:
-                    col_name = f"field_{col_name}"
-                columns.append(col_name)
-        
-        # Build INSERT statement
-        placeholders = ', '.join(['%s'] * len(columns))
-        sql = f"INSERT INTO {table_name} ({', '.join(columns)}) VALUES ({placeholders})"
-        
-        self.log_debug(f"SQL: {sql}", collection_name)
-        
-        # Convert documents to tuples
-        rows = []
-        errors = 0
-        
-        for doc_idx, doc in enumerate(documents):
-            try:
-                row = []
-                
-                # Add mongo_id
-                row.append(str(doc.get('_id', '')))
-                
-                # Add other fields
-                for field_name in schema.keys():
-                    if field_name != '_id':
-                        try:
-                            value = self._get_nested_value(doc, field_name)
-                            converted_value = self._convert_value_for_postgres(value, field_name, schema)
-                            row.append(converted_value)
-                        except Exception as e:
-                            self.log_error(collection_name, 'field_conversion', 
-                                         f"Field '{field_name}' in doc {doc_idx}: {str(e)}")
-                            # Only show debug for collections we're focusing on
-                            if collection_name in self.debug_collections:
-                                print(f"    ⚠️  Error converting field '{field_name}': {e}")
-                            row.append(None)  # Use NULL for problematic fields
-                
-                rows.append(tuple(row))
-                
-            except Exception as e:
-                self.log_error(collection_name, 'document_conversion', f"Document {doc_idx}: {str(e)}")
-                errors += 1
-                continue
-        
-        # Execute batch insert
-        if rows:
-            try:
-                cursor.executemany(sql, rows)
-                return len(rows), errors
-            except Exception as batch_error:
-                self.log_error(collection_name, 'batch_insert', str(batch_error))
-                
-                # Only show detailed debugging for targeted collections
-                if collection_name in self.debug_collections:
-                    print(f"    🔴 Batch insert failed for {collection_name}: {batch_error}")
-                    print("    Trying individual inserts with rollback handling...")
-                
-                # Rollback the failed transaction
-                cursor.connection.rollback()
-                
-                # Try inserting one by one in individual transactions
-                success_count = 0
-                for row_idx, row in enumerate(rows):
-                    try:
-                        cursor.execute(sql, row)
-                        cursor.connection.commit()  # Commit each successful insert
-                        success_count += 1
-                    except Exception as row_error:
-                        cursor.connection.rollback()  # Rollback failed insert
-                        self.log_error(collection_name, 'row_insert', f"Row {row_idx}: {str(row_error)}")
-                        
-                        # Show detailed error only for the first few failures and only for targeted collections
-                        if collection_name in self.debug_collections and errors < 3:
-                            print(f"      Row {row_idx} failed: {row_error}")
-                            print(f"      Row data: {len(row)} values, expected {len(columns)} columns")
-                            
-                        errors += 1
-                        continue
-                return success_count, errors
-        
-        return 0, errors
-    
-    def _get_nested_value(self, doc: Dict, field_path: str):
-        """Get value from nested document using dot notation"""
-        keys = field_path.split('.')
-        value = doc
-        
-        for key in keys:
-            if isinstance(value, dict) and key in value:
-                value = value[key]
-            else:
-                return None
-                
-        return value
-    
-    def _convert_value_for_postgres(self, value, field_name: str = None, schema: Dict = None):
-        """Convert MongoDB value to PostgreSQL compatible value with schema-aware conversion"""
-        if value is None:
-            return None
-        
-        # Get the expected PostgreSQL type for this field if available
-        expected_type = None
-        if schema and field_name and field_name in schema:
-            field_info = schema[field_name]
-            expected_type = self._determine_postgres_type(field_info)
-        
-        # Handle conversion based on expected type
-        if expected_type == 'BOOLEAN':
-            if isinstance(value, bool):
-                return value
-            elif isinstance(value, str):
-                return value.lower() in ('true', '1', 'yes', 'on')
-            elif isinstance(value, (int, float)):
-                return bool(value)
-            else:
-                return None
-        elif expected_type == 'INTEGER':
-            if isinstance(value, int):
-                return value
-            elif isinstance(value, float):
-                return int(value)
-            elif isinstance(value, str) and value.isdigit():
-                return int(value)
-            elif isinstance(value, bool):
-                return int(value)
-            else:
-                return None
-        elif expected_type == 'NUMERIC':
-            if isinstance(value, (int, float)):
-                return value
-            elif isinstance(value, str):
-                try:
-                    return float(value)
-                except ValueError:
-                    return None
-            elif isinstance(value, bool):
-                return float(value)
-            else:
-                return None
-        elif expected_type == 'TEXT':
-            if isinstance(value, str):
-                return value
-            elif value is not None:
-                str_value = str(value)
-                # Handle very long strings
-                if len(str_value) > 65535:
-                    return str_value[:65535]
-                return str_value
-            else:
-                return None
-        elif expected_type == 'TIMESTAMP':
-            if hasattr(value, 'isoformat'):
-                return value.isoformat()
-            elif isinstance(value, str):
-                return value
-            else:
-                return str(value) if value is not None else None
-        elif expected_type == 'JSONB':
-            if isinstance(value, (dict, list)):
-                return json.dumps(value, default=self._json_serializer)
-            elif isinstance(value, str):
-                # Check if it's already valid JSON
-                try:
-                    json.loads(value)
-                    return value
-                except (json.JSONDecodeError, TypeError):
-                    # Not valid JSON, wrap it
-                    return json.dumps(value)
-            else:
-                return json.dumps(value, default=self._json_serializer)
-        
-        # Fallback to original logic if no expected type or type not recognized
-        if isinstance(value, bool):
-            return value
-        elif isinstance(value, (int, float)):
-            return value
-        elif isinstance(value, str):
-            return value
-        elif isinstance(value, (dict, list)):
-            return json.dumps(value, default=self._json_serializer)
-        elif hasattr(value, 'isoformat'):  # datetime
-            return value.isoformat()
-        elif hasattr(value, '__str__'):
-            str_value = str(value)
-            if len(str_value) > 65535:
-                return str_value[:65535]
-            return str_value
-        else:
-            return str(value)
-    
-    def _json_serializer(self, obj):
-        """Custom JSON serializer for complex objects with better error handling"""
-        try:
-            if hasattr(obj, 'isoformat'):  # datetime
-                return obj.isoformat()
-            elif hasattr(obj, '__str__'):
-                return str(obj)
-            else:
-                return None
-        except Exception as e:
-            self.log_debug(f"JSON serialization error: {e}")
-            return str(obj)
-    
-    def run_conversion(self, sample_size: int = 100, batch_size: int = 1000):
-        """Run the full conversion process with focused debugging"""
-        print("Starting MongoDB to PostgreSQL conversion...")
-        print("This will convert your Rocket.Chat database from MongoDB to PostgreSQL")
-        if self.debug_mode:
-            if self.debug_collections:
-                print(f"🐛 DEBUG MODE: Focusing on collections: {', '.join(self.debug_collections)}")
-            else:
-                print("🐛 DEBUG MODE: All collections")
-        print("=" * 70)
-        
-        # Step 1: Discover collections
-        self.discover_collections()
-        
-        # Step 2: Analyze schemas
-        print("\nAnalyzing collection schemas...")
-        for collection_name in self.collections:
-            self.analyze_schema(collection_name, sample_size)
-        
-        # Sample problematic collections if debugging
-        if self.debug_mode and self.debug_collections:
-            for coll in self.debug_collections:
-                if coll in self.collections:
-                    self.sample_documents(coll, 2)
-        
-        # Step 3: Generate PostgreSQL schema
-        table_definitions = self.generate_postgres_schema()
-        
-        # Step 4: Create database schema
-        self.create_postgres_database(table_definitions)
-        
-        # Step 5: Convert and insert data
-        self.convert_and_insert_data(batch_size)
-        
-        # Step 6: Show error summary
-        self._print_error_summary()
-        
-        print("=" * 70)
-        print("✅ Conversion completed!")
-        print(f"   Database: {self.postgres_config['database']}")
-        print(f"   Tables created: {len(table_definitions)}")
-        
-    def _print_error_summary(self):
-        """Print a focused summary of errors"""
-        if not self.error_log:
-            print("\n✅ No errors encountered during conversion!")
-            return
-            
-        print("\n⚠️  ERROR SUMMARY:")
-        print("=" * 50)
-        
-        # Sort by error count descending
-        sorted_collections = sorted(self.error_log.items(), 
-                                  key=lambda x: len(x[1]), reverse=True)
-        
-        for collection, errors in sorted_collections:
-            error_types = {}
-            for error in errors:
-                error_type = error['type']
-                if error_type not in error_types:
-                    error_types[error_type] = []
-                error_types[error_type].append(error['details'])
-            
-            print(f"\n🔴 {collection} ({len(errors)} total errors):")
-            for error_type, details_list in error_types.items():
-                print(f"   {error_type}: {len(details_list)} errors")
-                
-                # Show sample errors for critical collections
-                if collection in ['rocketchat_settings', 'rocketchat_room'] and len(details_list) > 0:
-                    print(f"     Sample: {details_list[0][:100]}...")
-
-def main():
-    parser = argparse.ArgumentParser(
-        description='Convert MongoDB BSON export to PostgreSQL',
-        formatter_class=argparse.RawDescriptionHelpFormatter,
-        epilog="""
-Examples:
-  # Basic usage
-  python3 mongo_to_postgres_converter.py \\
-    --mongo-path db/database/62df06d44234d20001289144 \\
-    --pg-database rocketchat_converted \\
-    --pg-user rocketchat_user \\
-    --pg-password mypassword
-
-  # Debug specific failing collections
-  python3 mongo_to_postgres_converter.py \\
-    --mongo-path db/database/62df06d44234d20001289144 \\
-    --pg-database rocketchat_converted \\
-    --pg-user rocketchat_user \\
-    --pg-password mypassword \\
-    --debug-collections rocketchat_settings rocketchat_room
-
-Before running this script:
-1. Run: sudo -u postgres psql -f reset_database.sql
-2. Update the password in reset_database.sql
-        """
-    )
-    
-    parser.add_argument('--mongo-path', required=True, help='Path to MongoDB export directory')
-    parser.add_argument('--pg-host', default='localhost', help='PostgreSQL host (default: localhost)')
-    parser.add_argument('--pg-port', default='5432', help='PostgreSQL port (default: 5432)')
-    parser.add_argument('--pg-database', required=True, help='PostgreSQL database name')
-    parser.add_argument('--pg-user', required=True, help='PostgreSQL username')
-    parser.add_argument('--pg-password', required=True, help='PostgreSQL password')
-    parser.add_argument('--sample-size', type=int, default=100, help='Number of documents to sample for schema analysis (default: 100)')
-    parser.add_argument('--batch-size', type=int, default=1000, help='Batch size for data insertion (default: 1000)')
-    parser.add_argument('--debug', action='store_true', help='Enable debug mode with detailed error logging')
-    parser.add_argument('--debug-collections', nargs='*', help='Specific collections to debug (e.g., rocketchat_settings rocketchat_room)')
-    
-    args = parser.parse_args()
-    
-    postgres_config = {
-        'host': args.pg_host,
-        'port': args.pg_port,
-        'database': args.pg_database,
-        'user': args.pg_user,
-        'password': args.pg_password
-    }
-    
-    # Enable debug mode if debug collections are specified
-    debug_mode = args.debug or (args.debug_collections is not None)
-    
-    converter = MongoToPostgresConverter(args.mongo_path, postgres_config, debug_mode, args.debug_collections)
-    converter.run_conversion(args.sample_size, args.batch_size)
-
-
-if __name__ == '__main__':
-    main() 

inventory-server/chat/db-convert/reset_database.sql

@@ -1,41 +0,0 @@
--- PostgreSQL Database Reset Script for Rocket.Chat Import
--- Run as: sudo -u postgres psql -f reset_database.sql
-
--- Terminate all connections to the database (force disconnect users)
-SELECT pg_terminate_backend(pid)
-FROM pg_stat_activity
-WHERE datname = 'rocketchat_converted' AND pid <> pg_backend_pid();
-
--- Drop the database if it exists
-DROP DATABASE IF EXISTS rocketchat_converted;
-
--- Create fresh database
-CREATE DATABASE rocketchat_converted;
-
--- Create user (if not exists)
-DO $$ 
-BEGIN
-    IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'rocketchat_user') THEN
-        CREATE USER rocketchat_user WITH PASSWORD 'HKjLgt23gWuPXzEAn3rW';
-    END IF;
-END $$;
-
--- Grant database privileges
-GRANT CONNECT ON DATABASE rocketchat_converted TO rocketchat_user;
-GRANT CREATE ON DATABASE rocketchat_converted TO rocketchat_user;
-
--- Connect to the new database
-\c rocketchat_converted;
-
--- Grant schema privileges
-GRANT CREATE ON SCHEMA public TO rocketchat_user;
-GRANT USAGE ON SCHEMA public TO rocketchat_user;
-
--- Grant privileges on all future tables and sequences
-ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO rocketchat_user;
-ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE, SELECT ON SEQUENCES TO rocketchat_user;
-
--- Display success message
-\echo 'Database reset completed successfully!'
-\echo 'You can now run the converter with:'
-\echo 'python3 mongo_to_postgres_converter.py --mongo-path db/database/62df06d44234d20001289144 --pg-database rocketchat_converted --pg-user rocketchat_user --pg-password your_password' 

inventory-server/chat/db-convert/test_converter.py

@@ -1,54 +0,0 @@
-#!/usr/bin/env python3
-"""
-Quick test script to verify the converter fixes work for problematic collections
-"""
-
-from mongo_to_postgres_converter import MongoToPostgresConverter
-
-def test_problematic_collections():
-    print("🧪 Testing converter fixes for problematic collections...")
-    
-    postgres_config = {
-        'host': 'localhost',
-        'port': '5432',
-        'database': 'rocketchat_test',
-        'user': 'rocketchat_user',
-        'password': 'password123'
-    }
-    
-    converter = MongoToPostgresConverter(
-        'db/database/62df06d44234d20001289144',
-        postgres_config,
-        debug_mode=True,
-        debug_collections=['rocketchat_settings', 'rocketchat_room']
-    )
-    
-    # Test just discovery and schema analysis
-    print("\n1. Testing collection discovery...")
-    converter.discover_collections()
-    
-    print("\n2. Testing schema analysis...")
-    if 'rocketchat_settings' in converter.collections:
-        settings_schema = converter.analyze_schema('rocketchat_settings', 10)
-        print(f"Settings schema fields: {len(settings_schema)}")
-        
-        # Check specific problematic fields
-        if 'packageValue' in settings_schema:
-            packagevalue_info = settings_schema['packageValue']
-            pg_type = converter._determine_postgres_type(packagevalue_info)
-            print(f"packageValue types: {packagevalue_info['types']} -> PostgreSQL: {pg_type}")
-            
-    if 'rocketchat_room' in converter.collections:
-        room_schema = converter.analyze_schema('rocketchat_room', 10)
-        print(f"Room schema fields: {len(room_schema)}")
-        
-        # Check specific problematic fields
-        if 'sysMes' in room_schema:
-            sysmes_info = room_schema['sysMes']
-            pg_type = converter._determine_postgres_type(sysmes_info)
-            print(f"sysMes types: {sysmes_info['types']} -> PostgreSQL: {pg_type}")
-    
-    print("\n✅ Test completed - check the type mappings above!")
-
-if __name__ == '__main__':
-    test_problematic_collections() 

inventory-server/chat/package.json

@@ -1,20 +0,0 @@
-{
-  "name": "chat-server",
-  "version": "1.0.0",
-  "description": "Chat archive server for Rocket.Chat data",
-  "main": "server.js",
-  "scripts": {
-    "start": "node server.js",
-    "dev": "nodemon server.js"
-  },
-  "dependencies": {
-    "express": "^4.18.2",
-    "cors": "^2.8.5",
-    "pg": "^8.11.0",
-    "dotenv": "^16.0.3",
-    "morgan": "^1.10.0"
-  },
-  "devDependencies": {
-    "nodemon": "^2.0.22"
-  }
-} 

inventory-server/chat/routes.js

@@ -1,649 +0,0 @@
-const express = require('express');
-const path = require('path');
-const router = express.Router();
-
-// Serve uploaded files with proper mapping from database paths to actual file locations
-router.get('/files/uploads/*', async (req, res) => {
-  try {
-    // Extract the path from the URL (everything after /files/uploads/)
-    const requestPath = req.params[0];
-    
-    // The URL path will be like: ufs/AmazonS3:Uploads/274Mf9CyHNG72oF86/filename.jpg
-    // We need to extract the mongo_id (274Mf9CyHNG72oF86) from this path
-    const pathParts = requestPath.split('/');
-    let mongoId = null;
-    
-    // Find the mongo_id in the path structure
-    for (let i = 0; i < pathParts.length; i++) {
-      if (pathParts[i].includes('AmazonS3:Uploads') && i + 1 < pathParts.length) {
-        mongoId = pathParts[i + 1];
-        break;
-      }
-      // Sometimes the mongo_id might be the last part of ufs/AmazonS3:Uploads/mongoId
-      if (pathParts[i] === 'AmazonS3:Uploads' && i + 1 < pathParts.length) {
-        mongoId = pathParts[i + 1];
-        break;
-      }
-    }
-    
-    if (!mongoId) {
-      // Try to get mongo_id from database by matching the full path
-      const result = await global.pool.query(`
-        SELECT mongo_id, name, type 
-        FROM uploads 
-        WHERE path = $1 OR url = $1
-        LIMIT 1
-      `, [`/ufs/AmazonS3:Uploads/${requestPath}`, `/ufs/AmazonS3:Uploads/${requestPath}`]);
-      
-      if (result.rows.length > 0) {
-        mongoId = result.rows[0].mongo_id;
-      }
-    }
-    
-    if (!mongoId) {
-      return res.status(404).json({ error: 'File not found' });
-    }
-    
-    // The actual file is stored with just the mongo_id as filename
-    const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
-    
-    // Get file info from database for proper content-type
-    const fileInfo = await global.pool.query(`
-      SELECT name, type 
-      FROM uploads 
-      WHERE mongo_id = $1
-      LIMIT 1
-    `, [mongoId]);
-    
-    if (fileInfo.rows.length === 0) {
-      return res.status(404).json({ error: 'File metadata not found' });
-    }
-    
-    const { name, type } = fileInfo.rows[0];
-    
-    // Set proper content type
-    if (type) {
-      res.set('Content-Type', type);
-    }
-    
-    // Set content disposition with original filename
-    if (name) {
-      res.set('Content-Disposition', `inline; filename="${name}"`);
-    }
-    
-    // Send the file
-    res.sendFile(filePath, (err) => {
-      if (err) {
-        console.error('Error serving file:', err);
-        if (!res.headersSent) {
-          res.status(404).json({ error: 'File not found on disk' });
-        }
-      }
-    });
-    
-  } catch (error) {
-    console.error('Error serving upload:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Also serve files directly by mongo_id for simpler access
-router.get('/files/by-id/:mongoId', async (req, res) => {
-  try {
-    const { mongoId } = req.params;
-    
-    // Get file info from database
-    const fileInfo = await global.pool.query(`
-      SELECT name, type 
-      FROM uploads 
-      WHERE mongo_id = $1
-      LIMIT 1
-    `, [mongoId]);
-    
-    if (fileInfo.rows.length === 0) {
-      return res.status(404).json({ error: 'File not found' });
-    }
-    
-    const { name, type } = fileInfo.rows[0];
-    const filePath = path.join(__dirname, 'db-convert/db/files/uploads', mongoId);
-    
-    // Set proper content type and filename
-    if (type) {
-      res.set('Content-Type', type);
-    }
-    
-    if (name) {
-      res.set('Content-Disposition', `inline; filename="${name}"`);
-    }
-    
-    // Send the file
-    res.sendFile(filePath, (err) => {
-      if (err) {
-        console.error('Error serving file:', err);
-        if (!res.headersSent) {
-          res.status(404).json({ error: 'File not found on disk' });
-        }
-      }
-    });
-    
-  } catch (error) {
-    console.error('Error serving upload by ID:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Serve user avatars by mongo_id
-router.get('/avatar/:mongoId', async (req, res) => {
-  try {
-    const { mongoId } = req.params;
-    
-    console.log(`[Avatar Debug] Looking up avatar for user mongo_id: ${mongoId}`);
-    
-    // First try to find avatar by user's avataretag
-    const userResult = await global.pool.query(`
-      SELECT avataretag, username FROM users WHERE mongo_id = $1
-    `, [mongoId]);
-    
-    let avatarPath = null;
-    
-    if (userResult.rows.length > 0) {
-      const username = userResult.rows[0].username;
-      const avataretag = userResult.rows[0].avataretag;
-      
-      // Try method 1: Look up by avataretag -> etag (for users with avataretag set)
-      if (avataretag) {
-        console.log(`[Avatar Debug] Found user ${username} with avataretag: ${avataretag}`);
-        
-        const avatarResult = await global.pool.query(`
-          SELECT url, path FROM avatars WHERE etag = $1
-        `, [avataretag]);
-        
-        if (avatarResult.rows.length > 0) {
-          const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
-          console.log(`[Avatar Debug] Found avatar record with path: ${dbPath}`);
-          
-          if (dbPath) {
-            const pathParts = dbPath.split('/');
-            for (let i = 0; i < pathParts.length; i++) {
-              if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
-                const avatarMongoId = pathParts[i + 1];
-                avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
-                console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
-                break;
-              }
-            }
-          }
-        } else {
-          console.log(`[Avatar Debug] No avatar record found for etag: ${avataretag}`);
-        }
-      }
-      
-      // Try method 2: Look up by userid directly (for users without avataretag)
-      if (!avatarPath) {
-        console.log(`[Avatar Debug] Trying direct userid lookup for user ${username} (${mongoId})`);
-        
-        const avatarResult = await global.pool.query(`
-          SELECT url, path FROM avatars WHERE userid = $1
-        `, [mongoId]);
-        
-        if (avatarResult.rows.length > 0) {
-          const dbPath = avatarResult.rows[0].path || avatarResult.rows[0].url;
-          console.log(`[Avatar Debug] Found avatar record by userid with path: ${dbPath}`);
-          
-          if (dbPath) {
-            const pathParts = dbPath.split('/');
-            for (let i = 0; i < pathParts.length; i++) {
-              if (pathParts[i].includes('AmazonS3:Avatars') && i + 1 < pathParts.length) {
-                const avatarMongoId = pathParts[i + 1];
-                avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', avatarMongoId);
-                console.log(`[Avatar Debug] Extracted avatar mongo_id: ${avatarMongoId}, full path: ${avatarPath}`);
-                break;
-              }
-            }
-          }
-        } else {
-          console.log(`[Avatar Debug] No avatar record found for userid: ${mongoId}`);
-        }
-      }
-    } else {
-      console.log(`[Avatar Debug] No user found for mongo_id: ${mongoId}`);
-    }
-    
-    // Fallback: try direct lookup by user mongo_id
-    if (!avatarPath) {
-      avatarPath = path.join(__dirname, 'db-convert/db/files/avatars', mongoId);
-      console.log(`[Avatar Debug] Using fallback path: ${avatarPath}`);
-    }
-    
-    // Set proper content type for images
-    res.set('Content-Type', 'image/jpeg'); // Most avatars are likely JPEG
-    
-    // Send the file
-    res.sendFile(avatarPath, (err) => {
-      if (err) {
-        // If avatar doesn't exist, send a default 404 or generate initials
-        console.log(`[Avatar Debug] Avatar file not found at path: ${avatarPath}, error:`, err.message);
-        if (!res.headersSent) {
-          res.status(404).json({ error: 'Avatar not found' });
-        }
-      } else {
-        console.log(`[Avatar Debug] Successfully served avatar from: ${avatarPath}`);
-      }
-    });
-    
-  } catch (error) {
-    console.error('Error serving avatar:', error);
-    res.status(500).json({ error: 'Server error' });
-  }
-});
-
-// Serve avatars statically as fallback
-router.use('/files/avatars', express.static(path.join(__dirname, 'db-convert/db/files/avatars')));
-
-// Get all users for the "view as" dropdown (active and inactive)
-router.get('/users', async (req, res) => {
-  try {
-    const result = await global.pool.query(`
-      SELECT id, username, name, type, active, status, lastlogin, 
-             statustext, utcoffset, statusconnection, mongo_id, avataretag
-      FROM users 
-      WHERE type = 'user'
-      ORDER BY 
-        active DESC,  -- Active users first
-        CASE 
-          WHEN status = 'online' THEN 1
-          WHEN status = 'away' THEN 2  
-          WHEN status = 'busy' THEN 3
-          ELSE 4
-        END,
-        name ASC
-    `);
-    
-    res.json({
-      status: 'success',
-      users: result.rows
-    });
-  } catch (error) {
-    console.error('Error fetching users:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to fetch users',
-      details: error.message
-    });
-  }
-});
-
-// Get rooms for a specific user with enhanced room names for direct messages
-router.get('/users/:userId/rooms', async (req, res) => {
-  const { userId } = req.params;
-  
-  try {
-    // Get the current user's mongo_id for filtering
-    const userResult = await global.pool.query(`
-      SELECT mongo_id, username FROM users WHERE id = $1
-    `, [userId]);
-    
-    if (userResult.rows.length === 0) {
-      return res.status(404).json({
-        status: 'error',
-        error: 'User not found'
-      });
-    }
-    
-    const currentUserMongoId = userResult.rows[0].mongo_id;
-    const currentUsername = userResult.rows[0].username;
-    
-    // Get rooms where the user is a member with proper naming from subscription table
-    // Include archived and closed rooms but sort them at the bottom
-    const result = await global.pool.query(`
-      SELECT DISTINCT 
-        r.id, 
-        r.mongo_id as room_mongo_id, 
-        r.name, 
-        r.fname, 
-        r.t as type, 
-        r.msgs, 
-        r.lm as last_message_date, 
-        r.usernames, 
-        r.uids,
-        r.userscount,
-        r.description,
-        r.teamid,
-        r.archived,
-        s.open,
-        -- Use the subscription's name for direct messages (excludes current user)
-        -- For channels/groups, use room's fname or name
-        CASE 
-          WHEN r.t = 'd' THEN COALESCE(s.fname, s.name, 'Unknown User')
-          ELSE COALESCE(r.fname, r.name, 'Unnamed Room')
-        END as display_name
-      FROM room r
-      JOIN subscription s ON s.rid = r.mongo_id 
-      WHERE s.u->>'_id' = $1
-      ORDER BY 
-        s.open DESC NULLS LAST,  -- Open rooms first
-        r.archived NULLS FIRST,  -- Non-archived first (nulls treated as false)
-        r.lm DESC NULLS LAST
-      LIMIT 50
-    `, [currentUserMongoId]);
-
-    // Enhance rooms with participant information for direct messages
-    const enhancedRooms = await Promise.all(result.rows.map(async (room) => {
-      if (room.type === 'd' && room.uids) {
-        // Get participant info (excluding current user) for direct messages
-        const participantResult = await global.pool.query(`
-          SELECT u.username, u.name, u.mongo_id, u.avataretag
-          FROM users u 
-          WHERE u.mongo_id = ANY($1::text[])
-          AND u.mongo_id != $2
-        `, [room.uids, currentUserMongoId]);
-        
-        room.participants = participantResult.rows;
-      }
-      return room;
-    }));
-    
-    res.json({
-      status: 'success',
-      rooms: enhancedRooms
-    });
-  } catch (error) {
-    console.error('Error fetching user rooms:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to fetch user rooms',
-      details: error.message
-    });
-  }
-});
-
-// Get room details including participants
-router.get('/rooms/:roomId', async (req, res) => {
-  const { roomId } = req.params;
-  const { userId } = req.query; // Accept current user ID as query parameter
-  
-  try {
-    const result = await global.pool.query(`
-      SELECT r.id, r.name, r.fname, r.t as type, r.msgs, r.description,
-             r.lm as last_message_date, r.usernames, r.uids, r.userscount, r.teamid
-      FROM room r
-      WHERE r.id = $1
-    `, [roomId]);
-    
-    if (result.rows.length === 0) {
-      return res.status(404).json({
-        status: 'error',
-        error: 'Room not found'
-      });
-    }
-    
-    const room = result.rows[0];
-    
-    // For direct messages, get the proper display name based on current user
-    if (room.type === 'd' && room.uids && userId) {
-      // Get current user's mongo_id
-      const userResult = await global.pool.query(`
-        SELECT mongo_id FROM users WHERE id = $1
-      `, [userId]);
-      
-      if (userResult.rows.length > 0) {
-        const currentUserMongoId = userResult.rows[0].mongo_id;
-        
-        // Get display name from subscription table for this user
-        // Use room mongo_id to match with subscription.rid
-        const roomMongoResult = await global.pool.query(`
-          SELECT mongo_id FROM room WHERE id = $1
-        `, [roomId]);
-        
-        if (roomMongoResult.rows.length > 0) {
-          const roomMongoId = roomMongoResult.rows[0].mongo_id;
-          
-          const subscriptionResult = await global.pool.query(`
-            SELECT fname, name FROM subscription 
-            WHERE rid = $1 AND u->>'_id' = $2
-          `, [roomMongoId, currentUserMongoId]);
-          
-          if (subscriptionResult.rows.length > 0) {
-            const sub = subscriptionResult.rows[0];
-            room.display_name = sub.fname || sub.name || 'Unknown User';
-          }
-        }
-      }
-      
-      // Get all participants for additional info
-      const participantResult = await global.pool.query(`
-        SELECT username, name
-        FROM users 
-        WHERE mongo_id = ANY($1::text[])
-      `, [room.uids]);
-      
-      room.participants = participantResult.rows;
-    } else {
-      // For channels/groups, use room's fname or name
-      room.display_name = room.fname || room.name || 'Unnamed Room';
-    }
-    
-    res.json({
-      status: 'success',
-      room: room
-    });
-  } catch (error) {
-    console.error('Error fetching room details:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to fetch room details',
-      details: error.message
-    });
-  }
-});
-
-// Get messages for a specific room (fast, without attachments)
-router.get('/rooms/:roomId/messages', async (req, res) => {
-  const { roomId } = req.params;
-  const { limit = 50, offset = 0, before } = req.query;
-  
-  try {
-    // Fast query - just get messages without expensive attachment joins
-    let query = `
-      SELECT m.id, m.msg, m.ts, m.u, m._updatedat, m.urls, m.mentions, m.md
-      FROM message m
-      JOIN room r ON m.rid = r.mongo_id
-      WHERE r.id = $1
-    `;
-    
-    const params = [roomId];
-    
-    if (before) {
-      query += ` AND m.ts < $${params.length + 1}`;
-      params.push(before);
-    }
-    
-    query += ` ORDER BY m.ts DESC LIMIT $${params.length + 1} OFFSET $${params.length + 2}`;
-    params.push(limit, offset);
-    
-    const result = await global.pool.query(query, params);
-    
-    // Add empty attachments array for now - attachments will be loaded separately if needed
-    const messages = result.rows.map(msg => ({
-      ...msg,
-      attachments: []
-    }));
-    
-    res.json({
-      status: 'success',
-      messages: messages.reverse() // Reverse to show oldest first
-    });
-  } catch (error) {
-    console.error('Error fetching messages:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to fetch messages',
-      details: error.message
-    });
-  }
-});
-
-// Get attachments for specific messages (called separately for performance)
-router.post('/messages/attachments', async (req, res) => {
-  const { messageIds } = req.body;
-  
-  if (!messageIds || !Array.isArray(messageIds) || messageIds.length === 0) {
-    return res.json({ status: 'success', attachments: {} });
-  }
-  
-  try {
-    // Get room mongo_id from first message to limit search scope
-    const roomQuery = await global.pool.query(`
-      SELECT r.mongo_id as room_mongo_id
-      FROM message m
-      JOIN room r ON m.rid = r.mongo_id
-      WHERE m.id = $1
-      LIMIT 1
-    `, [messageIds[0]]);
-    
-    if (roomQuery.rows.length === 0) {
-      return res.json({ status: 'success', attachments: {} });
-    }
-    
-    const roomMongoId = roomQuery.rows[0].room_mongo_id;
-    
-    // Get messages and their upload timestamps
-    const messagesQuery = await global.pool.query(`
-      SELECT m.id, m.ts, m.u->>'_id' as user_id
-      FROM message m
-      WHERE m.id = ANY($1::int[])
-    `, [messageIds]);
-    
-    if (messagesQuery.rows.length === 0) {
-      return res.json({ status: 'success', attachments: {} });
-    }
-    
-    // Build a map of user_id -> array of message timestamps for efficient lookup
-    const userTimeMap = {};
-    const messageMap = {};
-    messagesQuery.rows.forEach(msg => {
-      if (!userTimeMap[msg.user_id]) {
-        userTimeMap[msg.user_id] = [];
-      }
-      userTimeMap[msg.user_id].push(msg.ts);
-      messageMap[msg.id] = { ts: msg.ts, user_id: msg.user_id };
-    });
-    
-    // Get attachments for this room and these users
-    const uploadsQuery = await global.pool.query(`
-      SELECT mongo_id, name, size, type, url, path, typegroup, identify, 
-             userid, uploadedat
-      FROM uploads 
-      WHERE rid = $1 
-      AND userid = ANY($2::text[])
-      ORDER BY uploadedat
-    `, [roomMongoId, Object.keys(userTimeMap)]);
-    
-    // Match attachments to messages based on timestamp proximity (within 5 minutes)
-    const attachmentsByMessage = {};
-    
-    uploadsQuery.rows.forEach(upload => {
-      const uploadTime = new Date(upload.uploadedat).getTime();
-      
-      // Find the closest message from this user within 5 minutes
-      let closestMessageId = null;
-      let closestTimeDiff = Infinity;
-      
-      Object.entries(messageMap).forEach(([msgId, msgData]) => {
-        if (msgData.user_id === upload.userid) {
-          const msgTime = new Date(msgData.ts).getTime();
-          const timeDiff = Math.abs(uploadTime - msgTime);
-          
-          if (timeDiff < 300000 && timeDiff < closestTimeDiff) { // 5 minutes = 300000ms
-            closestMessageId = msgId;
-            closestTimeDiff = timeDiff;
-          }
-        }
-      });
-      
-      if (closestMessageId) {
-        if (!attachmentsByMessage[closestMessageId]) {
-          attachmentsByMessage[closestMessageId] = [];
-        }
-        
-        attachmentsByMessage[closestMessageId].push({
-          id: upload.id,
-          mongo_id: upload.mongo_id,
-          name: upload.name,
-          size: upload.size,
-          type: upload.type,
-          url: upload.url,
-          path: upload.path,
-          typegroup: upload.typegroup,
-          identify: upload.identify
-        });
-      }
-    });
-    
-    res.json({
-      status: 'success',
-      attachments: attachmentsByMessage
-    });
-    
-  } catch (error) {
-    console.error('Error fetching message attachments:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to fetch attachments',
-      details: error.message
-    });
-  }
-});
-
-// Search messages in accessible rooms for a user
-router.get('/users/:userId/search', async (req, res) => {
-  const { userId } = req.params;
-  const { q, limit = 20 } = req.query;
-  
-  if (!q || q.length < 2) {
-    return res.status(400).json({
-      status: 'error',
-      error: 'Search query must be at least 2 characters'
-    });
-  }
-  
-  try {
-    const userResult = await global.pool.query(`
-      SELECT mongo_id FROM users WHERE id = $1
-    `, [userId]);
-    
-    if (userResult.rows.length === 0) {
-      return res.status(404).json({
-        status: 'error',
-        error: 'User not found'
-      });
-    }
-    
-    const currentUserMongoId = userResult.rows[0].mongo_id;
-    
-    const result = await global.pool.query(`
-      SELECT m.id, m.msg, m.ts, m.u, r.id as room_id, r.name as room_name, r.fname as room_fname, r.t as room_type
-      FROM message m
-      JOIN room r ON m.rid = r.mongo_id
-      JOIN subscription s ON s.rid = r.mongo_id AND s.u->>'_id' = $1
-      WHERE m.msg ILIKE $2
-      AND r.archived IS NOT TRUE
-      ORDER BY m.ts DESC
-      LIMIT $3
-    `, [currentUserMongoId, `%${q}%`, limit]);
-    
-    res.json({
-      status: 'success',
-      results: result.rows
-    });
-  } catch (error) {
-    console.error('Error searching messages:', error);
-    res.status(500).json({ 
-      status: 'error',
-      error: 'Failed to search messages',
-      details: error.message
-    });
-  }
-});
-
-module.exports = router; 

inventory-server/chat/server.js

@@ -33,7 +33,7 @@ global.pool = pool;
 app.use(express.json());
 app.use(morgan('combined'));
 app.use(cors({
-  origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw'],
+  origin: ['http://localhost:5175', 'http://localhost:5174', 'https://inventory.kent.pw', 'https://tools.acherryontop.com', 'https://tools.acherryontop.com'],
   credentials: true
 }));
 

inventory-server/dashboard/acot-server/routes/events.js

@@ -0,0 +1,973 @@
+const express = require('express');
+const router = express.Router();
+const { getDbConnection, getPoolStatus } = require('../db/connection');
+const { getTimeRangeConditions, formatBusinessDate, getBusinessDayBounds } = require('../utils/timeUtils');
+
+// Image URL generation utility
+const getImageUrls = (pid, iid = 1) => {
+  const imageUrlBase = 'https://sbing.com/i/products/0000/';
+  const paddedPid = pid.toString().padStart(6, '0');
+  const prefix = paddedPid.slice(0, 3);
+  const basePath = `${imageUrlBase}${prefix}/${pid}`;
+  return {
+    image: `${basePath}-t-${iid}.jpg`,
+    image_175: `${basePath}-175x175-${iid}.jpg`,
+    image_full: `${basePath}-o-${iid}.jpg`,
+    ImgThumb: `${basePath}-175x175-${iid}.jpg` // For ProductGrid component
+  };
+};
+
+// Main stats endpoint - replaces /api/klaviyo/events/stats
+router.get('/stats', async (req, res) => {
+  const startTime = Date.now();
+  console.log(`[STATS] Starting request for timeRange: ${req.query.timeRange}`);
+  
+  // Set a timeout for the entire operation
+  const timeoutPromise = new Promise((_, reject) => {
+    setTimeout(() => reject(new Error('Request timeout after 15 seconds')), 15000);
+  });
+  
+  try {
+    const mainOperation = async () => {
+      const { timeRange, startDate, endDate } = req.query;
+      console.log(`[STATS] Getting DB connection...`);
+      const { connection, release } = await getDbConnection();
+      console.log(`[STATS] DB connection obtained in ${Date.now() - startTime}ms`);
+    
+    const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
+    
+    // Main order stats query
+    const mainStatsQuery = `
+      SELECT 
+        COUNT(*) as orderCount,
+        SUM(summary_total) as revenue,
+        SUM(stats_prod_pieces) as itemCount,
+        AVG(summary_total) as averageOrderValue,
+        AVG(stats_prod_pieces) as averageItemsPerOrder,
+        SUM(CASE WHEN stats_waiting_preorder > 0 THEN 1 ELSE 0 END) as preOrderCount,
+        SUM(CASE WHEN ship_method_selected = 'localpickup' THEN 1 ELSE 0 END) as localPickupCount,
+        SUM(CASE WHEN ship_method_selected = 'holdit' THEN 1 ELSE 0 END) as onHoldCount,
+        SUM(CASE WHEN order_status IN (100, 92) THEN 1 ELSE 0 END) as shippedCount,
+        SUM(CASE WHEN order_status = 15 THEN 1 ELSE 0 END) as cancelledCount,
+        SUM(CASE WHEN order_status = 15 THEN summary_total ELSE 0 END) as cancelledTotal
+      FROM _order 
+      WHERE order_status > 15 AND ${whereClause}
+    `;
+    
+    const [mainStats] = await connection.execute(mainStatsQuery, params);
+    const stats = mainStats[0];
+    
+    // Refunds query
+    const refundsQuery = `
+      SELECT 
+        COUNT(*) as refundCount,
+        ABS(SUM(payment_amount)) as refundTotal
+      FROM order_payment op
+      JOIN _order o ON op.order_id = o.order_id
+      WHERE payment_amount < 0 AND o.order_status > 15 AND ${whereClause.replace('date_placed', 'o.date_placed')}
+    `;
+    
+    const [refundStats] = await connection.execute(refundsQuery, params);
+    
+    // Best revenue day query
+    const bestDayQuery = `
+      SELECT 
+        DATE(date_placed) as date,
+        SUM(summary_total) as revenue,
+        COUNT(*) as orders
+      FROM _order 
+      WHERE order_status > 15 AND ${whereClause}
+      GROUP BY DATE(date_placed)
+      ORDER BY revenue DESC
+      LIMIT 1
+    `;
+    
+    const [bestDayResult] = await connection.execute(bestDayQuery, params);
+    
+    // Peak hour query (for single day periods)
+    let peakHour = null;
+    if (['today', 'yesterday'].includes(timeRange)) {
+      const peakHourQuery = `
+        SELECT 
+          HOUR(date_placed) as hour,
+          COUNT(*) as count
+        FROM _order 
+        WHERE order_status > 15 AND ${whereClause}
+        GROUP BY HOUR(date_placed)
+        ORDER BY count DESC
+        LIMIT 1
+      `;
+      
+      const [peakHourResult] = await connection.execute(peakHourQuery, params);
+      if (peakHourResult.length > 0) {
+        const hour = peakHourResult[0].hour;
+        const date = new Date();
+        date.setHours(hour, 0, 0);
+        peakHour = {
+          hour,
+          count: peakHourResult[0].count,
+          displayHour: date.toLocaleString("en-US", { hour: "numeric", hour12: true })
+        };
+      }
+    }
+    
+    // Brands and categories query - simplified for now since we don't have the category tables
+    // We'll use a simple approach without company table for now
+    const brandsQuery = `
+      SELECT 
+        'Various Brands' as brandName,
+        COUNT(DISTINCT oi.order_id) as orderCount,
+        SUM(oi.qty_ordered) as itemCount,
+        SUM(oi.qty_ordered * oi.prod_price) as revenue
+      FROM order_items oi
+      JOIN _order o ON oi.order_id = o.order_id
+      JOIN products p ON oi.prod_pid = p.pid
+      WHERE o.order_status > 15 AND ${whereClause.replace('date_placed', 'o.date_placed')}
+      HAVING revenue > 0
+    `;
+    
+    const [brandsResult] = await connection.execute(brandsQuery, params);
+    
+    // For categories, we'll use a simplified approach
+    const categoriesQuery = `
+      SELECT 
+        'General' as categoryName,
+        COUNT(DISTINCT oi.order_id) as orderCount,
+        SUM(oi.qty_ordered) as itemCount,
+        SUM(oi.qty_ordered * oi.prod_price) as revenue
+      FROM order_items oi
+      JOIN _order o ON oi.order_id = o.order_id
+      JOIN products p ON oi.prod_pid = p.pid
+      WHERE o.order_status > 15 AND ${whereClause.replace('date_placed', 'o.date_placed')}
+      HAVING revenue > 0
+    `;
+    
+    const [categoriesResult] = await connection.execute(categoriesQuery, params);
+    
+    // Shipping locations query
+    const shippingQuery = `
+      SELECT 
+        ship_country,
+        ship_state,
+        ship_method_selected,
+        COUNT(*) as count
+      FROM _order 
+      WHERE order_status IN (100, 92) AND ${whereClause}
+      GROUP BY ship_country, ship_state, ship_method_selected
+    `;
+    
+    const [shippingResult] = await connection.execute(shippingQuery, params);
+    
+    // Process shipping data
+    const shippingStats = processShippingData(shippingResult, stats.shippedCount);
+    
+    // Order value range query
+    const orderRangeQuery = `
+      SELECT 
+        MIN(summary_total) as smallest,
+        MAX(summary_total) as largest
+      FROM _order 
+      WHERE order_status > 15 AND ${whereClause}
+    `;
+    
+    const [orderRangeResult] = await connection.execute(orderRangeQuery, params);
+    
+    // Calculate period progress for incomplete periods
+    let periodProgress = 100;
+    if (['today', 'thisWeek', 'thisMonth'].includes(timeRange)) {
+      periodProgress = calculatePeriodProgress(timeRange);
+    }
+    
+    // Previous period comparison data
+    const prevPeriodData = await getPreviousPeriodData(connection, timeRange, startDate, endDate);
+    
+    const response = {
+      timeRange: dateRange,
+      stats: {
+        revenue: parseFloat(stats.revenue || 0),
+        orderCount: parseInt(stats.orderCount || 0),
+        itemCount: parseInt(stats.itemCount || 0),
+        averageOrderValue: parseFloat(stats.averageOrderValue || 0),
+        averageItemsPerOrder: parseFloat(stats.averageItemsPerOrder || 0),
+        
+        // Order types
+        orderTypes: {
+          preOrders: {
+            count: parseInt(stats.preOrderCount || 0),
+            percentage: stats.orderCount > 0 ? (stats.preOrderCount / stats.orderCount) * 100 : 0
+          },
+          localPickup: {
+            count: parseInt(stats.localPickupCount || 0),
+            percentage: stats.orderCount > 0 ? (stats.localPickupCount / stats.orderCount) * 100 : 0
+          },
+          heldItems: {
+            count: parseInt(stats.onHoldCount || 0),
+            percentage: stats.orderCount > 0 ? (stats.onHoldCount / stats.orderCount) * 100 : 0
+          }
+        },
+        
+        // Shipping
+        shipping: {
+          shippedCount: parseInt(stats.shippedCount || 0),
+          locations: shippingStats.locations,
+          methodStats: shippingStats.methods
+        },
+        
+        // Brands and categories
+        brands: {
+          total: brandsResult.length,
+          list: brandsResult.slice(0, 50).map(brand => ({
+            name: brand.brandName,
+            count: parseInt(brand.itemCount),
+            revenue: parseFloat(brand.revenue)
+          }))
+        },
+        
+        categories: {
+          total: categoriesResult.length,
+          list: categoriesResult.slice(0, 50).map(category => ({
+            name: category.categoryName,
+            count: parseInt(category.itemCount),
+            revenue: parseFloat(category.revenue)
+          }))
+        },
+        
+        // Refunds and cancellations
+        refunds: {
+          total: parseFloat(refundStats[0]?.refundTotal || 0),
+          count: parseInt(refundStats[0]?.refundCount || 0)
+        },
+        
+        canceledOrders: {
+          total: parseFloat(stats.cancelledTotal || 0),
+          count: parseInt(stats.cancelledCount || 0)
+        },
+        
+        // Best day
+        bestRevenueDay: bestDayResult.length > 0 ? {
+          amount: parseFloat(bestDayResult[0].revenue),
+          displayDate: bestDayResult[0].date,
+          orders: parseInt(bestDayResult[0].orders)
+        } : null,
+        
+        // Peak hour (for single days)
+        peakOrderHour: peakHour,
+        
+        // Order value range
+        orderValueRange: orderRangeResult.length > 0 ? {
+          smallest: parseFloat(orderRangeResult[0].smallest || 0),
+          largest: parseFloat(orderRangeResult[0].largest || 0)
+        } : { smallest: 0, largest: 0 },
+        
+        // Period progress and projections
+        periodProgress,
+        projectedRevenue: periodProgress < 100 ? (stats.revenue / (periodProgress / 100)) : stats.revenue,
+        
+        // Previous period comparison
+        prevPeriodRevenue: prevPeriodData.revenue,
+        prevPeriodOrders: prevPeriodData.orderCount,
+        prevPeriodAOV: prevPeriodData.averageOrderValue
+      }
+    };
+    
+    return { response, release };
+    };
+    
+    // Race between the main operation and timeout
+    let result;
+    try {
+      result = await Promise.race([mainOperation(), timeoutPromise]);
+    } catch (error) {
+      // If it's a timeout, we don't have a release function to call
+      if (error.message.includes('timeout')) {
+        console.log(`[STATS] Request timed out in ${Date.now() - startTime}ms`);
+        throw error;
+      }
+      // For other errors, re-throw
+      throw error;
+    }
+    
+    const { response, release } = result;
+    
+    // Release connection back to pool
+    if (release) release();
+    
+    console.log(`[STATS] Request completed in ${Date.now() - startTime}ms`);
+    res.json(response);
+    
+  } catch (error) {
+    console.error('Error in /stats:', error);
+    console.log(`[STATS] Request failed in ${Date.now() - startTime}ms`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Daily details endpoint - replaces /api/klaviyo/events/stats/details
+router.get('/stats/details', async (req, res) => {
+  let release;
+  try {
+    const { timeRange, startDate, endDate, metric, daily } = req.query;
+    const { connection, release: releaseConn } = await getDbConnection();
+    release = releaseConn;
+
+    const { whereClause, params } = getTimeRangeConditions(timeRange, startDate, endDate);
+
+    // Daily breakdown query
+    const dailyQuery = `
+      SELECT 
+        DATE(date_placed) as date,
+        COUNT(*) as orders,
+        SUM(summary_total) as revenue,
+        AVG(summary_total) as averageOrderValue,
+        SUM(stats_prod_pieces) as itemCount
+      FROM _order 
+      WHERE order_status > 15 AND ${whereClause}
+      GROUP BY DATE(date_placed)
+      ORDER BY DATE(date_placed)
+    `;
+    
+    const [dailyResults] = await connection.execute(dailyQuery, params);
+    
+    // Get previous period data using the same logic as main stats endpoint
+    let prevWhereClause, prevParams;
+    
+    if (timeRange && timeRange !== 'custom') {
+      const prevTimeRange = getPreviousTimeRange(timeRange);
+      const result = getTimeRangeConditions(prevTimeRange);
+      prevWhereClause = result.whereClause;
+      prevParams = result.params;
+    } else {
+      // Custom date range - go back by the same duration
+      const start = new Date(startDate);
+      const end = new Date(endDate);
+      const duration = end.getTime() - start.getTime();
+      
+      const prevEnd = new Date(start.getTime() - 1);
+      const prevStart = new Date(prevEnd.getTime() - duration);
+      
+      prevWhereClause = 'date_placed >= ? AND date_placed <= ?';
+      prevParams = [prevStart.toISOString(), prevEnd.toISOString()];
+    }
+
+    // Get previous period daily data
+    const prevQuery = `
+      SELECT 
+        DATE(date_placed) as date,
+        COUNT(*) as prevOrders,
+        SUM(summary_total) as prevRevenue,
+        AVG(summary_total) as prevAvgOrderValue
+      FROM _order 
+      WHERE order_status > 15 AND ${prevWhereClause}
+      GROUP BY DATE(date_placed)
+    `;
+    
+    const [prevResults] = await connection.execute(prevQuery, prevParams);
+    
+    // Create a map for quick lookup of previous period data
+    const prevMap = new Map();
+    prevResults.forEach(prev => {
+      const key = new Date(prev.date).toISOString().split('T')[0];
+      prevMap.set(key, prev);
+    });
+    
+    // For period-to-period comparison, we need to map days by relative position
+    // since dates won't match exactly (e.g., current week vs previous week)
+    const dailyArray = dailyResults.map(day => ({
+      timestamp: day.date,
+      date: day.date,
+      orders: parseInt(day.orders),
+      revenue: parseFloat(day.revenue),
+      averageOrderValue: parseFloat(day.averageOrderValue || 0),
+      itemCount: parseInt(day.itemCount)
+    }));
+
+    const prevArray = prevResults.map(day => ({
+      orders: parseInt(day.prevOrders),
+      revenue: parseFloat(day.prevRevenue),
+      averageOrderValue: parseFloat(day.prevAvgOrderValue || 0)
+    }));
+
+    // Combine current and previous period data by matching relative positions
+    const statsWithComparison = dailyArray.map((day, index) => {
+      const prev = prevArray[index] || { orders: 0, revenue: 0, averageOrderValue: 0 };
+      
+      return {
+        ...day,
+        prevOrders: prev.orders,
+        prevRevenue: prev.revenue,
+        prevAvgOrderValue: prev.averageOrderValue
+      };
+    });
+    
+    res.json({ stats: statsWithComparison });
+    
+  } catch (error) {
+    console.error('Error in /stats/details:', error);
+    res.status(500).json({ error: error.message });
+  } finally {
+    // Release connection back to pool
+    if (release) release();
+  }
+});
+
+// Financial performance endpoint
+router.get('/financials', async (req, res) => {
+  let release;
+  try {
+    const { timeRange, startDate, endDate } = req.query;
+    const { connection, release: releaseConn } = await getDbConnection();
+    release = releaseConn;
+
+    const { whereClause, params, dateRange } = getTimeRangeConditions(timeRange, startDate, endDate);
+    const financialWhere = whereClause.replace(/date_placed/g, 'DATE_SUB(date_change, INTERVAL 1 HOUR)');
+
+    const [totalsRows] = await connection.execute(
+      buildFinancialTotalsQuery(financialWhere),
+      params
+    );
+
+    const totals = normalizeFinancialTotals(totalsRows[0]);
+
+    const [trendRows] = await connection.execute(
+      buildFinancialTrendQuery(financialWhere),
+      params
+    );
+
+    const trend = trendRows.map(normalizeFinancialTrendRow);
+
+    let previousTotals = null;
+    let comparison = null;
+
+    const previousRange = getPreviousPeriodRange(timeRange, startDate, endDate);
+    if (previousRange) {
+      const prevWhere = previousRange.whereClause.replace(/date_placed/g, 'DATE_SUB(date_change, INTERVAL 1 HOUR)');
+      const [previousRows] = await connection.execute(
+        buildFinancialTotalsQuery(prevWhere),
+        previousRange.params
+      );
+      previousTotals = normalizeFinancialTotals(previousRows[0]);
+      comparison = {
+        grossSales: calculateComparison(totals.grossSales, previousTotals.grossSales),
+        refunds: calculateComparison(totals.refunds, previousTotals.refunds),
+        taxCollected: calculateComparison(totals.taxCollected, previousTotals.taxCollected),
+        discounts: calculateComparison(totals.discounts, previousTotals.discounts),
+        cogs: calculateComparison(totals.cogs, previousTotals.cogs),
+        income: calculateComparison(totals.income, previousTotals.income),
+        profit: calculateComparison(totals.profit, previousTotals.profit),
+        margin: calculateComparison(totals.margin, previousTotals.margin),
+      };
+    }
+
+    res.json({
+      dateRange,
+      totals,
+      previousTotals,
+      comparison,
+      trend,
+    });
+  } catch (error) {
+    console.error('Error in /financials:', error);
+    res.status(500).json({ error: error.message });
+  } finally {
+    if (release) release();
+  }
+});
+
+// Products endpoint - replaces /api/klaviyo/events/products
+router.get('/products', async (req, res) => {
+  let release;
+  try {
+    const { timeRange, startDate, endDate } = req.query;
+    const { connection, release: releaseConn } = await getDbConnection();
+    release = releaseConn;
+    
+    const { whereClause, params } = getTimeRangeConditions(timeRange, startDate, endDate);
+    
+    const productsQuery = `
+      SELECT 
+        p.pid,
+        p.description as name,
+        SUM(oi.qty_ordered) as totalQuantity,
+        SUM(oi.qty_ordered * oi.prod_price) as totalRevenue,
+        COUNT(DISTINCT oi.order_id) as orderCount,
+        (SELECT pi.iid FROM product_images pi WHERE pi.pid = p.pid AND pi.order = 255 LIMIT 1) as primary_iid
+      FROM order_items oi
+      JOIN _order o ON oi.order_id = o.order_id
+      JOIN products p ON oi.prod_pid = p.pid
+      WHERE o.order_status > 15 AND ${whereClause.replace('date_placed', 'o.date_placed')}
+      GROUP BY p.pid, p.description
+      ORDER BY totalRevenue DESC
+      LIMIT 500
+    `;
+    
+    const [productsResult] = await connection.execute(productsQuery, params);
+    
+    // Add image URLs to each product
+    const productsWithImages = productsResult.map(product => {
+      const imageUrls = getImageUrls(product.pid, product.primary_iid || 1);
+      return {
+        id: product.pid,
+        name: product.name,
+        totalQuantity: parseInt(product.totalQuantity),
+        totalRevenue: parseFloat(product.totalRevenue),
+        orderCount: parseInt(product.orderCount),
+        ...imageUrls
+      };
+    });
+    
+    res.json({
+      stats: {
+        products: {
+          total: productsWithImages.length,
+          list: productsWithImages
+        }
+      }
+    });
+    
+  } catch (error) {
+    console.error('Error in /products:', error);
+    res.status(500).json({ error: error.message });
+  } finally {
+    // Release connection back to pool
+    if (release) release();
+  }
+});
+
+// Projection endpoint - replaces /api/klaviyo/events/projection
+router.get('/projection', async (req, res) => {
+  let release;
+  try {
+    const { timeRange, startDate, endDate } = req.query;
+    
+    // Only provide projections for incomplete periods
+    if (!['today', 'thisWeek', 'thisMonth'].includes(timeRange)) {
+      return res.json({ projectedRevenue: 0, confidence: 0 });
+    }
+    
+    const { connection, release: releaseConn } = await getDbConnection();
+    release = releaseConn;
+    
+    // Get current period data
+    const { whereClause, params } = getTimeRangeConditions(timeRange, startDate, endDate);
+    
+    const currentQuery = `
+      SELECT 
+        SUM(summary_total) as currentRevenue,
+        COUNT(*) as currentOrders
+      FROM _order 
+      WHERE order_status > 15 AND ${whereClause}
+    `;
+    
+    const [currentResult] = await connection.execute(currentQuery, params);
+    const current = currentResult[0];
+    
+    // Get historical data for the same period type
+    const historicalQuery = await getHistoricalProjectionData(connection, timeRange);
+    
+    // Calculate projection based on current progress and historical patterns
+    const periodProgress = calculatePeriodProgress(timeRange);
+    const projection = calculateSmartProjection(
+      parseFloat(current.currentRevenue || 0),
+      parseInt(current.currentOrders || 0),
+      periodProgress,
+      historicalQuery
+    );
+    
+    res.json(projection);
+    
+  } catch (error) {
+    console.error('Error in /projection:', error);
+    res.status(500).json({ error: error.message });
+  } finally {
+    // Release connection back to pool
+    if (release) release();
+  }
+});
+
+// Debug endpoint to check connection pool status
+router.get('/debug/pool', (req, res) => {
+  res.json(getPoolStatus());
+});
+
+// Health check endpoint
+router.get('/health', async (req, res) => {
+  try {
+    const { connection, release } = await getDbConnection();
+    
+    // Simple query to test connection
+    const [result] = await connection.execute('SELECT 1 as test');
+    release();
+    
+    res.json({ 
+      status: 'healthy', 
+      timestamp: new Date().toISOString(),
+      pool: getPoolStatus(),
+      dbTest: result[0]
+    });
+  } catch (error) {
+    res.status(500).json({ 
+      status: 'unhealthy', 
+      error: error.message,
+      timestamp: new Date().toISOString(),
+      pool: getPoolStatus()
+    });
+  }
+});
+
+// Helper functions
+function processShippingData(shippingResult, totalShipped) {
+  const countries = {};
+  const states = {};
+  const methods = {};
+  
+  shippingResult.forEach(row => {
+    // Countries
+    if (row.ship_country) {
+      countries[row.ship_country] = (countries[row.ship_country] || 0) + row.count;
+    }
+    
+    // States
+    if (row.ship_state) {
+      states[row.ship_state] = (states[row.ship_state] || 0) + row.count;
+    }
+    
+    // Methods
+    if (row.ship_method_selected) {
+      methods[row.ship_method_selected] = (methods[row.ship_method_selected] || 0) + row.count;
+    }
+  });
+  
+  return {
+    locations: {
+      total: totalShipped,
+      byCountry: Object.entries(countries)
+        .map(([country, count]) => ({
+          country,
+          count,
+          percentage: (count / totalShipped) * 100
+        }))
+        .sort((a, b) => b.count - a.count),
+      byState: Object.entries(states)
+        .map(([state, count]) => ({
+          state,
+          count,
+          percentage: (count / totalShipped) * 100
+        }))
+        .sort((a, b) => b.count - a.count)
+    },
+    methods: Object.entries(methods)
+      .map(([name, value]) => ({ name, value }))
+      .sort((a, b) => b.value - a.value)
+  };
+}
+
+function calculatePeriodProgress(timeRange) {
+  const now = new Date();
+  const easternTime = new Date(now.getTime() - (5 * 60 * 60 * 1000)); // UTC-5
+  
+  switch (timeRange) {
+    case 'today': {
+      const { start } = getBusinessDayBounds('today');
+      const businessStart = new Date(start);
+      const businessEnd = new Date(businessStart);
+      businessEnd.setDate(businessEnd.getDate() + 1);
+      businessEnd.setHours(0, 59, 59, 999); // 12:59 AM next day
+      
+      const elapsed = easternTime.getTime() - businessStart.getTime();
+      const total = businessEnd.getTime() - businessStart.getTime();
+      return Math.min(100, Math.max(0, (elapsed / total) * 100));
+    }
+    case 'thisWeek': {
+      const startOfWeek = new Date(easternTime);
+      startOfWeek.setDate(easternTime.getDate() - easternTime.getDay()); // Sunday
+      startOfWeek.setHours(1, 0, 0, 0); // 1 AM business day start
+      
+      const endOfWeek = new Date(startOfWeek);
+      endOfWeek.setDate(endOfWeek.getDate() + 7);
+      
+      const elapsed = easternTime.getTime() - startOfWeek.getTime();
+      const total = endOfWeek.getTime() - startOfWeek.getTime();
+      return Math.min(100, Math.max(0, (elapsed / total) * 100));
+    }
+    case 'thisMonth': {
+      const startOfMonth = new Date(easternTime.getFullYear(), easternTime.getMonth(), 1, 1, 0, 0, 0);
+      const endOfMonth = new Date(easternTime.getFullYear(), easternTime.getMonth() + 1, 1, 0, 59, 59, 999);
+      
+      const elapsed = easternTime.getTime() - startOfMonth.getTime();
+      const total = endOfMonth.getTime() - startOfMonth.getTime();
+      return Math.min(100, Math.max(0, (elapsed / total) * 100));
+    }
+    default:
+      return 100;
+  }
+}
+
+function buildFinancialTotalsQuery(whereClause) {
+  return `
+    SELECT 
+      COALESCE(SUM(sale_amount), 0) as grossSales,
+      COALESCE(SUM(refund_amount), 0) as refunds,
+      COALESCE(SUM(shipping_collected_amount + small_order_fee_amount + rush_fee_amount), 0) as shippingFees,
+      COALESCE(SUM(tax_collected_amount), 0) as taxCollected,
+      COALESCE(SUM(discount_total_amount), 0) as discounts,
+      COALESCE(SUM(cogs_amount), 0) as cogs
+    FROM report_sales_data
+    WHERE ${whereClause}
+      AND action IN (1, 2, 3)
+  `;
+}
+
+function buildFinancialTrendQuery(whereClause) {
+  return `
+    SELECT 
+      DATE(DATE_SUB(date_change, INTERVAL 1 HOUR)) as date,
+      SUM(sale_amount) as grossSales,
+      SUM(refund_amount) as refunds,
+      SUM(shipping_collected_amount + small_order_fee_amount + rush_fee_amount) as shippingFees,
+      SUM(tax_collected_amount) as taxCollected,
+      SUM(discount_total_amount) as discounts,
+      SUM(cogs_amount) as cogs
+    FROM report_sales_data
+    WHERE ${whereClause}
+      AND action IN (1, 2, 3)
+    GROUP BY DATE(DATE_SUB(date_change, INTERVAL 1 HOUR))
+    ORDER BY date ASC
+  `;
+}
+
+function normalizeFinancialTotals(row = {}) {
+  const grossSales = parseFloat(row.grossSales || 0);
+  const refunds = parseFloat(row.refunds || 0);
+  const shippingFees = parseFloat(row.shippingFees || 0);
+  const taxCollected = parseFloat(row.taxCollected || 0);
+  const discounts = parseFloat(row.discounts || 0);
+  const cogs = parseFloat(row.cogs || 0);
+  const productNet = grossSales - refunds - discounts;
+  const income = productNet + shippingFees;
+  const profit = income - cogs;
+  const margin = income !== 0 ? (profit / income) * 100 : 0;
+
+  return {
+    grossSales,
+    refunds,
+    shippingFees,
+    taxCollected,
+    discounts,
+    cogs,
+    income,
+    profit,
+    margin,
+  };
+}
+
+function normalizeFinancialTrendRow(row = {}) {
+  const grossSales = parseFloat(row.grossSales || 0);
+  const refunds = parseFloat(row.refunds || 0);
+  const shippingFees = parseFloat(row.shippingFees || 0);
+  const taxCollected = parseFloat(row.taxCollected || 0);
+  const discounts = parseFloat(row.discounts || 0);
+  const cogs = parseFloat(row.cogs || 0);
+  const productNet = grossSales - refunds - discounts;
+  const income = productNet + shippingFees;
+  const profit = income - cogs;
+  const margin = income !== 0 ? (profit / income) * 100 : 0;
+  let timestamp = null;
+  let dateValue = null;
+
+  if (row.date instanceof Date) {
+    dateValue = row.date.toISOString().slice(0, 10);
+  } else if (typeof row.date === 'string') {
+    dateValue = row.date;
+  }
+
+  if (typeof dateValue === 'string') {
+    timestamp = new Date(`${dateValue}T06:00:00.000Z`).toISOString();
+  }
+
+  return {
+    date: dateValue,
+    grossSales,
+    refunds,
+    shippingFees,
+    taxCollected,
+    discounts,
+    cogs,
+    income,
+    profit,
+    margin,
+    timestamp,
+  };
+}
+
+function calculateComparison(currentValue, previousValue) {
+  if (typeof previousValue !== 'number') {
+    return { absolute: null, percentage: null };
+  }
+
+  const absolute = typeof currentValue === 'number' ? currentValue - previousValue : null;
+  const percentage =
+    absolute !== null && previousValue !== 0
+      ? (absolute / Math.abs(previousValue)) * 100
+      : null;
+
+  return { absolute, percentage };
+}
+
+function getPreviousPeriodRange(timeRange, startDate, endDate) {
+  if (timeRange && timeRange !== 'custom') {
+    const prevTimeRange = getPreviousTimeRange(timeRange);
+    if (!prevTimeRange || prevTimeRange === timeRange) {
+      return null;
+    }
+    return getTimeRangeConditions(prevTimeRange);
+  }
+
+  const hasCustomDates = (timeRange === 'custom' || !timeRange) && startDate && endDate;
+  if (!hasCustomDates) {
+    return null;
+  }
+
+  const start = new Date(startDate);
+  const end = new Date(endDate);
+
+  if (Number.isNaN(start.getTime()) || Number.isNaN(end.getTime())) {
+    return null;
+  }
+
+  const duration = end.getTime() - start.getTime();
+  if (!Number.isFinite(duration) || duration <= 0) {
+    return null;
+  }
+
+  const prevEnd = new Date(start.getTime() - 1);
+  const prevStart = new Date(prevEnd.getTime() - duration);
+
+  return getTimeRangeConditions('custom', prevStart.toISOString(), prevEnd.toISOString());
+}
+
+async function getPreviousPeriodData(connection, timeRange, startDate, endDate) {
+  // Calculate previous period dates
+  let prevWhereClause, prevParams;
+  
+  if (timeRange && timeRange !== 'custom') {
+    const prevTimeRange = getPreviousTimeRange(timeRange);
+    const result = getTimeRangeConditions(prevTimeRange);
+    prevWhereClause = result.whereClause;
+    prevParams = result.params;
+  } else {
+    // Custom date range - go back by the same duration
+    const start = new Date(startDate);
+    const end = new Date(endDate);
+    const duration = end.getTime() - start.getTime();
+    
+    const prevEnd = new Date(start.getTime() - 1);
+    const prevStart = new Date(prevEnd.getTime() - duration);
+    
+    prevWhereClause = 'date_placed >= ? AND date_placed <= ?';
+    prevParams = [prevStart.toISOString(), prevEnd.toISOString()];
+  }
+  
+  const prevQuery = `
+    SELECT 
+      COUNT(*) as orderCount,
+      SUM(summary_total) as revenue,
+      AVG(summary_total) as averageOrderValue
+    FROM _order 
+    WHERE order_status > 15 AND ${prevWhereClause}
+  `;
+  
+  const [prevResult] = await connection.execute(prevQuery, prevParams);
+  const prev = prevResult[0] || { orderCount: 0, revenue: 0, averageOrderValue: 0 };
+  
+  return {
+    orderCount: parseInt(prev.orderCount || 0),
+    revenue: parseFloat(prev.revenue || 0),
+    averageOrderValue: parseFloat(prev.averageOrderValue || 0)
+  };
+}
+
+function getPreviousTimeRange(timeRange) {
+  const map = {
+    today: 'yesterday',
+    thisWeek: 'lastWeek',
+    thisMonth: 'lastMonth',
+    last7days: 'previous7days',
+    last30days: 'previous30days',
+    last90days: 'previous90days',
+    yesterday: 'twoDaysAgo'
+  };
+  return map[timeRange] || timeRange;
+}
+
+async function getHistoricalProjectionData(connection, timeRange) {
+  // Get historical data for projection calculations
+  // This is a simplified version - you could make this more sophisticated
+  const historicalQuery = `
+    SELECT 
+      SUM(summary_total) as revenue,
+      COUNT(*) as orders
+    FROM _order 
+    WHERE order_status > 15 
+      AND date_placed >= DATE_SUB(NOW(), INTERVAL 30 DAY)
+      AND date_placed < DATE_SUB(NOW(), INTERVAL 1 DAY)
+  `;
+  
+  const [result] = await connection.execute(historicalQuery);
+  return result;
+}
+
+function calculateSmartProjection(currentRevenue, currentOrders, periodProgress, historicalData) {
+  if (periodProgress >= 100) {
+    return { projectedRevenue: currentRevenue, projectedOrders: currentOrders, confidence: 1.0 };
+  }
+  
+  // Simple linear projection with confidence based on how much of the period has elapsed
+  const projectedRevenue = currentRevenue / (periodProgress / 100);
+  const projectedOrders = Math.round(currentOrders / (periodProgress / 100));
+  
+  // Confidence increases with more data (higher period progress)
+  const confidence = Math.min(0.95, Math.max(0.1, periodProgress / 100));
+  
+  return {
+    projectedRevenue,
+    projectedOrders,
+    confidence
+  };
+}
+
+// Health check endpoint
+router.get('/health', async (req, res) => {
+  try {
+    const poolStatus = getPoolStatus();
+    
+    // Test database connectivity
+    const { connection, release } = await getDbConnection();
+    await connection.execute('SELECT 1 as test');
+    release();
+    
+    res.json({
+      status: 'healthy',
+      timestamp: new Date().toISOString(),
+      pool: poolStatus,
+      database: 'connected'
+    });
+  } catch (error) {
+    console.error('Health check failed:', error);
+    res.status(500).json({
+      status: 'unhealthy',
+      timestamp: new Date().toISOString(),
+      error: error.message,
+      pool: getPoolStatus()
+    });
+  }
+});
+
+// Debug endpoint for pool status
+router.get('/debug/pool', (req, res) => {
+  res.json({
+    timestamp: new Date().toISOString(),
+    pool: getPoolStatus()
+  });
+});
+
+module.exports = router; 

inventory-server/db/config-schema-new.sql

@@ -1,196 +0,0 @@
--- Create function for updating timestamps if it doesn't exist
-CREATE OR REPLACE FUNCTION update_updated_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Create function for updating updated_at timestamps
-CREATE OR REPLACE FUNCTION update_updated_at_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated_at = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Drop tables in reverse order of dependency
-DROP TABLE IF EXISTS public.settings_product CASCADE;
-DROP TABLE IF EXISTS public.settings_vendor CASCADE;
-DROP TABLE IF EXISTS public.settings_global CASCADE;
-
--- Table Definition: settings_global
-CREATE TABLE public.settings_global (
-    setting_key VARCHAR PRIMARY KEY,
-    setting_value VARCHAR NOT NULL,
-    description TEXT,
-    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
-);
-
--- Table Definition: settings_vendor
-CREATE TABLE public.settings_vendor (
-    vendor VARCHAR PRIMARY KEY, -- Matches products.vendor
-    default_lead_time_days INT,
-    default_days_of_stock INT,
-    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP
-);
--- Index for faster lookups if needed (PK usually sufficient)
--- CREATE INDEX idx_settings_vendor_vendor ON public.settings_vendor(vendor);
-
--- Table Definition: settings_product
-CREATE TABLE public.settings_product (
-    pid INT8 PRIMARY KEY,
-    lead_time_days INT, -- Overrides vendor/global
-    days_of_stock INT, -- Overrides vendor/global
-    safety_stock INT DEFAULT 0, -- Minimum desired stock level
-    forecast_method VARCHAR DEFAULT 'standard', -- e.g., 'standard', 'seasonal'
-    exclude_from_forecast BOOLEAN DEFAULT FALSE,
-    updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    CONSTRAINT fk_settings_product_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
-);
-
-
--- Description: Inserts or updates standard default global settings.
---              Safe to rerun; will update existing keys with these default values.
--- Dependencies: `settings_global` table must exist.
--- Frequency: Run once initially, or rerun if you want to reset global defaults.
-
-INSERT INTO public.settings_global (setting_key, setting_value, description) VALUES
-    ('abc_revenue_threshold_a', '0.80', 'Revenue percentage for Class A (cumulative)'),
-    ('abc_revenue_threshold_b', '0.95', 'Revenue percentage for Class B (cumulative)'),
-    ('abc_calculation_basis', 'revenue_30d', 'Metric for ABC calc (revenue_30d, sales_30d, lifetime_revenue)'),
-    ('abc_calculation_period', '30', 'Days period for ABC calculation if not lifetime'),
-    ('default_forecast_method', 'standard', 'Default forecast method (standard, seasonal)'),
-    ('default_lead_time_days', '14', 'Global default lead time in days'),
-    ('default_days_of_stock', '30', 'Global default days of stock coverage target'),
-    -- Set default safety stock to 0 units. Can be overridden per product.
-    -- If you wanted safety stock in days, you'd store 'days' here and calculate units later.
-    ('default_safety_stock_units', '0', 'Global default safety stock in units')
-ON CONFLICT (setting_key) DO UPDATE SET
-    setting_value = EXCLUDED.setting_value,
-    description = EXCLUDED.description,
-    updated_at = CURRENT_TIMESTAMP; -- Update timestamp if default value changes
-
-
-      
--- Description: Creates placeholder rows in `settings_vendor` for each unique vendor
---              found in the `products` table. Does NOT set specific overrides.
---              Safe to rerun; will NOT overwrite existing vendor settings.
--- Dependencies: `settings_vendor` table must exist, `products` table populated.
--- Frequency: Run once after initial product load, or periodically if new vendors are added.
-
-INSERT INTO public.settings_vendor (
-    vendor,
-    default_lead_time_days,
-    default_days_of_stock
-    -- updated_at will use its default CURRENT_TIMESTAMP on insert
-)
-SELECT
-    DISTINCT p.vendor,
-    -- Explicitly cast NULL to INTEGER to resolve type mismatch
-    CAST(NULL AS INTEGER),
-    CAST(NULL AS INTEGER)
-FROM
-    public.products p
-WHERE
-    p.vendor IS NOT NULL
-    AND p.vendor <> '' -- Exclude blank vendors if necessary
-
-ON CONFLICT (vendor) DO NOTHING; -- IMPORTANT: Do not overwrite existing vendor settings
-
-SELECT COUNT(*) FROM public.settings_vendor; -- Verify rows were inserted
-
-      
--- Description: Creates placeholder rows in `settings_product` for each unique product
---              found in the `products` table. Sets basic defaults but no specific overrides.
---              Safe to rerun; will NOT overwrite existing product settings.
--- Dependencies: `settings_product` table must exist, `products` table populated.
--- Frequency: Run once after initial product load, or periodically if new products are added.
-
-INSERT INTO public.settings_product (
-    pid,
-    lead_time_days,         -- NULL = Inherit from Vendor/Global
-    days_of_stock,          -- NULL = Inherit from Vendor/Global
-    safety_stock,           -- Default to 0 units initially
-    forecast_method,        -- NULL = Inherit from Global ('standard')
-    exclude_from_forecast   -- Default to FALSE
-    -- updated_at will use its default CURRENT_TIMESTAMP on insert
-)
-SELECT
-    p.pid,
-    CAST(NULL AS INTEGER),  -- Explicitly cast NULL to INTEGER
-    CAST(NULL AS INTEGER),  -- Explicitly cast NULL to INTEGER
-    COALESCE((SELECT setting_value::int FROM settings_global WHERE setting_key = 'default_safety_stock_units'), 0), -- Use global default safety stock units
-    CAST(NULL AS VARCHAR), -- Cast NULL to VARCHAR for forecast_method (already varchar, but explicit)
-    FALSE  -- Default: Include in forecast
-FROM
-    public.products p
-
-ON CONFLICT (pid) DO NOTHING; -- IMPORTANT: Do not overwrite existing product-specific settings
-
-
--- History and status tables
-CREATE TABLE IF NOT EXISTS calculate_history (
-    id BIGSERIAL PRIMARY KEY,
-    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    end_time TIMESTAMP WITH TIME ZONE NULL,
-    duration_seconds INTEGER,
-    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
-    total_products INTEGER DEFAULT 0,
-    total_orders INTEGER DEFAULT 0,
-    total_purchase_orders INTEGER DEFAULT 0,
-    processed_products INTEGER DEFAULT 0,
-    processed_orders INTEGER DEFAULT 0,
-    processed_purchase_orders INTEGER DEFAULT 0,
-    status calculation_status DEFAULT 'running',
-    error_message TEXT,
-    additional_info JSONB
-);
-
-CREATE TABLE IF NOT EXISTS calculate_status (
-    module_name text PRIMARY KEY,
-    last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE TABLE IF NOT EXISTS sync_status (
-    table_name TEXT PRIMARY KEY,
-    last_sync_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    last_sync_id BIGINT
-);
-
-CREATE TABLE IF NOT EXISTS import_history (
-    id BIGSERIAL PRIMARY KEY,
-    table_name VARCHAR(50) NOT NULL,
-    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    end_time TIMESTAMP WITH TIME ZONE NULL,
-    duration_seconds INTEGER,
-    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
-    records_added INTEGER DEFAULT 0,
-    records_updated INTEGER DEFAULT 0,
-    records_deleted INTEGER DEFAULT 0,
-    records_skipped INTEGER DEFAULT 0,
-    total_processed INTEGER DEFAULT 0,
-    is_incremental BOOLEAN DEFAULT FALSE,
-    status calculation_status DEFAULT 'running',
-    error_message TEXT,
-    additional_info JSONB
-);
-
--- Create all indexes after tables are fully created
-CREATE INDEX IF NOT EXISTS idx_last_calc ON calculate_status(last_calculation_timestamp);
-CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status(last_sync_timestamp);
-CREATE INDEX IF NOT EXISTS idx_table_time ON import_history(table_name, start_time);
-CREATE INDEX IF NOT EXISTS idx_import_history_status ON import_history(status);
-CREATE INDEX IF NOT EXISTS idx_calculate_history_status ON calculate_history(status);
-
--- Add comments for documentation
-COMMENT ON TABLE import_history IS 'Tracks history of data import operations with detailed statistics';
-COMMENT ON COLUMN import_history.records_deleted IS 'Number of records deleted during this import';
-COMMENT ON COLUMN import_history.records_skipped IS 'Number of records skipped (e.g., unchanged, invalid)';
-COMMENT ON COLUMN import_history.total_processed IS 'Total number of records examined/processed, including skipped';
-
-COMMENT ON TABLE calculate_history IS 'Tracks history of metrics calculation runs with performance data';
-COMMENT ON COLUMN calculate_history.duration_seconds IS 'Total duration of the calculation in seconds';
-COMMENT ON COLUMN calculate_history.additional_info IS 'JSON object containing step timings, row counts, and other detailed metrics'; 

inventory-server/db/metrics-schema-new.sql

@@ -1,344 +0,0 @@
--- Drop tables in reverse order of dependency
-DROP TABLE IF EXISTS public.product_metrics CASCADE;
-DROP TABLE IF EXISTS public.daily_product_snapshots CASCADE;
-
--- Table Definition: daily_product_snapshots
-CREATE TABLE public.daily_product_snapshots (
-    snapshot_date DATE NOT NULL,
-    pid INT8 NOT NULL,
-    sku VARCHAR, -- Copied for convenience
-
-    -- Inventory Metrics (End of Day / Last Snapshot of Day)
-    eod_stock_quantity INT NOT NULL DEFAULT 0,
-    eod_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- Increased precision
-    eod_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    eod_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    stockout_flag BOOLEAN NOT NULL DEFAULT FALSE,
-
-    -- Sales Metrics (Aggregated for the snapshot_date)
-    units_sold INT NOT NULL DEFAULT 0,
-    units_returned INT NOT NULL DEFAULT 0,
-    gross_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    discounts NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    returns_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    net_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- gross_revenue - discounts
-    cogs NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    gross_regular_revenue NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    profit NUMERIC(14, 4) NOT NULL DEFAULT 0.00, -- net_revenue - cogs
-
-    -- Receiving Metrics (Aggregated for the snapshot_date)
-    units_received INT NOT NULL DEFAULT 0,
-    cost_received NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-
-    calculation_timestamp TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
-
-    PRIMARY KEY (snapshot_date, pid) -- Composite primary key
-    -- CONSTRAINT fk_daily_snapshot_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE -- FK Optional on snapshot table
-);
-
--- Add Indexes for daily_product_snapshots
-CREATE INDEX idx_daily_snapshot_pid_date ON public.daily_product_snapshots(pid, snapshot_date); -- Useful for product-specific time series
-
-
--- Table Definition: product_metrics
-CREATE TABLE public.product_metrics (
-    pid INT8 PRIMARY KEY,
-    last_calculated TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP,
-
-    -- Product Info (Copied for convenience/performance)
-    sku VARCHAR,
-    title VARCHAR,
-    brand VARCHAR,
-    vendor VARCHAR,
-    image_url VARCHAR, -- (e.g., products.image_175)
-    is_visible BOOLEAN,
-    is_replenishable BOOLEAN,
-    
-    -- Additional product fields
-    barcode VARCHAR,
-    harmonized_tariff_code VARCHAR,
-    vendor_reference VARCHAR,
-    notions_reference VARCHAR,
-    line VARCHAR,
-    subline VARCHAR,
-    artist VARCHAR,
-    moq INT,
-    rating NUMERIC(10, 2),
-    reviews INT,
-    weight NUMERIC(14, 4),
-    length NUMERIC(14, 4),
-    width NUMERIC(14, 4),
-    height NUMERIC(14, 4),
-    country_of_origin VARCHAR,
-    location VARCHAR,
-    baskets INT,
-    notifies INT,
-    preorder_count INT,
-    notions_inv_count INT,
-
-    -- Current Status (Refreshed Hourly)
-    current_price NUMERIC(10, 2),
-    current_regular_price NUMERIC(10, 2),
-    current_cost_price NUMERIC(10, 4), -- Increased precision for cost
-    current_landing_cost_price NUMERIC(10, 4), -- Increased precision for cost
-    current_stock INT NOT NULL DEFAULT 0,
-    current_stock_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    current_stock_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    current_stock_gross NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    on_order_qty INT NOT NULL DEFAULT 0,
-    on_order_cost NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    on_order_retail NUMERIC(14, 4) NOT NULL DEFAULT 0.00,
-    earliest_expected_date DATE,
-    -- total_received_lifetime INT NOT NULL DEFAULT 0, -- Can calc if needed
-
-    -- Historical Dates (Calculated Once/Periodically)
-    date_created DATE,
-    date_first_received DATE,
-    date_last_received DATE,
-    date_first_sold DATE,
-    date_last_sold DATE,
-    age_days INT, -- Calculated based on LEAST(date_created, date_first_sold)
-
-    -- Rolling Period Metrics (Refreshed Hourly from daily_product_snapshots)
-    sales_7d INT, revenue_7d NUMERIC(14, 4),
-    sales_14d INT, revenue_14d NUMERIC(14, 4),
-    sales_30d INT, revenue_30d NUMERIC(14, 4),
-    cogs_30d NUMERIC(14, 4), profit_30d NUMERIC(14, 4),
-    returns_units_30d INT, returns_revenue_30d NUMERIC(14, 4),
-    discounts_30d NUMERIC(14, 4),
-    gross_revenue_30d NUMERIC(14, 4), gross_regular_revenue_30d NUMERIC(14, 4),
-    stockout_days_30d INT,
-    sales_365d INT, revenue_365d NUMERIC(14, 4),
-    avg_stock_units_30d NUMERIC(10, 2), avg_stock_cost_30d NUMERIC(14, 4),
-    avg_stock_retail_30d NUMERIC(14, 4), avg_stock_gross_30d NUMERIC(14, 4),
-    received_qty_30d INT, received_cost_30d NUMERIC(14, 4),
-
-    -- Lifetime Metrics (Recalculated Hourly/Daily from daily_product_snapshots)
-    lifetime_sales INT,
-    lifetime_revenue NUMERIC(16, 4),
-    lifetime_revenue_quality VARCHAR(10), -- 'exact', 'partial', 'estimated'
-
-    -- First Period Metrics (Calculated Once/Periodically from daily_product_snapshots)
-    first_7_days_sales INT, first_7_days_revenue NUMERIC(14, 4),
-    first_30_days_sales INT, first_30_days_revenue NUMERIC(14, 4),
-    first_60_days_sales INT, first_60_days_revenue NUMERIC(14, 4),
-    first_90_days_sales INT, first_90_days_revenue NUMERIC(14, 4),
-
-    -- Calculated KPIs (Refreshed Hourly based on rolling metrics)
-    asp_30d NUMERIC(10, 2), -- revenue_30d / sales_30d
-    acp_30d NUMERIC(10, 4), -- cogs_30d / sales_30d
-    avg_ros_30d NUMERIC(10, 4), -- profit_30d / sales_30d
-    avg_sales_per_day_30d NUMERIC(10, 2), -- sales_30d / 30.0
-    avg_sales_per_month_30d NUMERIC(10, 2), -- sales_30d (assuming 30d = 1 month for this metric)
-    margin_30d NUMERIC(8, 2), -- (profit_30d / revenue_30d) * 100
-    markup_30d NUMERIC(8, 2), -- (profit_30d / cogs_30d) * 100
-    gmroi_30d NUMERIC(10, 2), -- profit_30d / avg_stock_cost_30d
-    stockturn_30d NUMERIC(10, 2), -- sales_30d / avg_stock_units_30d
-    return_rate_30d NUMERIC(8, 2), -- returns_units_30d / (sales_30d + returns_units_30d) * 100
-    discount_rate_30d NUMERIC(8, 2), -- discounts_30d / gross_revenue_30d * 100
-    stockout_rate_30d NUMERIC(8, 2), -- stockout_days_30d / 30.0 * 100
-    markdown_30d NUMERIC(14, 4), -- gross_regular_revenue_30d - gross_revenue_30d
-    markdown_rate_30d NUMERIC(8, 2), -- markdown_30d / gross_regular_revenue_30d * 100
-    sell_through_30d NUMERIC(8, 2), -- sales_30d / (current_stock + sales_30d) * 100
-    avg_lead_time_days INT, -- Calculated Periodically from purchase_orders
-
-    -- Forecasting & Replenishment (Refreshed Hourly)
-    abc_class CHAR(1), -- Updated Periodically (e.g., Weekly)
-    sales_velocity_daily NUMERIC(10, 4), -- sales_30d / (30.0 - stockout_days_30d)
-    config_lead_time INT, -- From settings tables
-    config_days_of_stock INT, -- From settings tables
-    config_safety_stock INT, -- From settings_product
-    planning_period_days INT, -- config_lead_time + config_days_of_stock
-    lead_time_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_lead_time
-    days_of_stock_forecast_units NUMERIC(10, 2), -- sales_velocity_daily * config_days_of_stock
-    planning_period_forecast_units NUMERIC(10, 2), -- lead_time_forecast_units + days_of_stock_forecast_units
-    lead_time_closing_stock NUMERIC(10, 2), -- current_stock + on_order_qty - lead_time_forecast_units
-    days_of_stock_closing_stock NUMERIC(10, 2), -- lead_time_closing_stock - days_of_stock_forecast_units
-    replenishment_needed_raw NUMERIC(10, 2), -- planning_period_forecast_units + config_safety_stock - current_stock - on_order_qty
-    replenishment_units INT, -- CEILING(GREATEST(0, replenishment_needed_raw))
-    replenishment_cost NUMERIC(14, 4), -- replenishment_units * COALESCE(current_landing_cost_price, current_cost_price)
-    replenishment_retail NUMERIC(14, 4), -- replenishment_units * current_price
-    replenishment_profit NUMERIC(14, 4), -- replenishment_units * (current_price - COALESCE(current_landing_cost_price, current_cost_price))
-    to_order_units INT, -- Apply MOQ/UOM logic to replenishment_units
-    forecast_lost_sales_units NUMERIC(10, 2), -- GREATEST(0, -lead_time_closing_stock)
-    forecast_lost_revenue NUMERIC(14, 4), -- forecast_lost_sales_units * current_price
-    stock_cover_in_days NUMERIC(10, 1), -- current_stock / sales_velocity_daily
-    po_cover_in_days NUMERIC(10, 1), -- on_order_qty / sales_velocity_daily
-    sells_out_in_days NUMERIC(10, 1), -- (current_stock + on_order_qty) / sales_velocity_daily
-    replenish_date DATE, -- Calc based on when stock hits safety stock minus lead time
-    overstocked_units INT, -- GREATEST(0, current_stock - config_safety_stock - planning_period_forecast_units)
-    overstocked_cost NUMERIC(14, 4), -- overstocked_units * COALESCE(current_landing_cost_price, current_cost_price)
-    overstocked_retail NUMERIC(14, 4), -- overstocked_units * current_price
-    is_old_stock BOOLEAN, -- Based on age, last sold, last received, on_order status
-
-    -- Yesterday's Metrics (Refreshed Hourly from daily_product_snapshots)
-    yesterday_sales INT,
-
-    -- Product Status (Calculated from metrics)
-    status VARCHAR, -- Stores status values like: Critical, Reorder Soon, Healthy, Overstock, At Risk, New
-
-    -- Growth Metrics (P3)
-    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
-    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth current 30d vs prev 30d
-    sales_growth_yoy NUMERIC(10, 2), -- Year-over-year sales growth %
-    revenue_growth_yoy NUMERIC(10, 2), -- Year-over-year revenue growth %
-
-    -- Demand Variability Metrics (P3)
-    sales_variance_30d NUMERIC(10, 2), -- Variance of daily sales
-    sales_std_dev_30d NUMERIC(10, 2), -- Standard deviation of daily sales
-    sales_cv_30d NUMERIC(10, 2), -- Coefficient of variation
-    demand_pattern VARCHAR(20), -- 'stable', 'variable', 'sporadic', 'lumpy'
-
-    -- Service Level & Fill Rate (P5)
-    fill_rate_30d NUMERIC(8, 2), -- % of demand fulfilled from stock
-    stockout_incidents_30d INT, -- Days with stockouts
-    service_level_30d NUMERIC(8, 2), -- % of days without stockouts
-    lost_sales_incidents_30d INT, -- Days with potential lost sales
-
-    -- Seasonality (P5)
-    seasonality_index NUMERIC(10, 2), -- Current vs average (100 = average)
-    seasonal_pattern VARCHAR(20), -- 'none', 'weekly', 'monthly', 'quarterly', 'yearly'
-    peak_season VARCHAR(20), -- e.g., 'Q4', 'summer', 'holiday'
-
-    CONSTRAINT fk_product_metrics_pid FOREIGN KEY (pid) REFERENCES public.products(pid) ON DELETE CASCADE ON UPDATE CASCADE
-);
-
--- Add Indexes for product_metrics (adjust based on common filtering/sorting in frontend)
-CREATE INDEX idx_product_metrics_brand ON public.product_metrics(brand);
-CREATE INDEX idx_product_metrics_vendor ON public.product_metrics(vendor);
-CREATE INDEX idx_product_metrics_sku ON public.product_metrics(sku);
-CREATE INDEX idx_product_metrics_abc_class ON public.product_metrics(abc_class);
-CREATE INDEX idx_product_metrics_revenue_30d ON public.product_metrics(revenue_30d DESC NULLS LAST); -- Example sorting index
-CREATE INDEX idx_product_metrics_sales_30d ON public.product_metrics(sales_30d DESC NULLS LAST); -- Example sorting index
-CREATE INDEX idx_product_metrics_current_stock ON public.product_metrics(current_stock);
-CREATE INDEX idx_product_metrics_sells_out_in_days ON public.product_metrics(sells_out_in_days ASC NULLS LAST); -- Example sorting index
-CREATE INDEX idx_product_metrics_status ON public.product_metrics(status); -- Index for status filtering
-
--- Add new vendor, category, and brand metrics tables
--- Drop tables in reverse order if they exist
-DROP TABLE IF EXISTS public.brand_metrics CASCADE;
-DROP TABLE IF EXISTS public.vendor_metrics CASCADE;
-DROP TABLE IF EXISTS public.category_metrics CASCADE;
-
--- ========= Category Metrics =========
-CREATE TABLE public.category_metrics (
-    category_id INT8 PRIMARY KEY, -- Foreign key to categories.cat_id
-    category_name VARCHAR,        -- Denormalized for convenience
-    category_type INT2,           -- Denormalized for convenience
-    parent_id INT8,             -- Denormalized for convenience
-    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-
-    -- ROLLED-UP METRICS (includes this category + all descendants)
-    -- Counts & Basic Info
-    product_count INT NOT NULL DEFAULT 0,           -- Total products linked
-    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products linked
-    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products linked
-
-    -- Current Stock Value (approximated using current product costs/prices)
-    current_stock_units INT NOT NULL DEFAULT 0,
-    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-
-    -- Rolling Period Aggregates (Summed from product_metrics)
-    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-
-    -- DIRECT METRICS (only products directly in this category)
-    direct_product_count INT NOT NULL DEFAULT 0,           -- Products directly in this category
-    direct_active_product_count INT NOT NULL DEFAULT 0,    -- Visible products directly in this category
-    direct_replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products directly in this category
-
-    -- Direct Current Stock Value
-    direct_current_stock_units INT NOT NULL DEFAULT 0,
-    direct_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    direct_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-
-    -- Direct Rolling Period Aggregates
-    direct_sales_7d INT NOT NULL DEFAULT 0, direct_revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    direct_sales_30d INT NOT NULL DEFAULT 0, direct_revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    direct_profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, direct_cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    direct_sales_365d INT NOT NULL DEFAULT 0, direct_revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    direct_lifetime_sales INT NOT NULL DEFAULT 0, direct_lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-
-    -- Calculated KPIs (Based on 30d aggregates) - Apply to rolled-up metrics
-    avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
-    stock_turn_30d NUMERIC(10, 3), -- sales_units / avg_stock_units (Needs avg stock calc)
-    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
-    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
-
-    CONSTRAINT fk_category_metrics_cat_id FOREIGN KEY (category_id) REFERENCES public.categories(cat_id) ON DELETE CASCADE ON UPDATE CASCADE
-);
-CREATE INDEX idx_category_metrics_name ON public.category_metrics(category_name);
-CREATE INDEX idx_category_metrics_type ON public.category_metrics(category_type);
-
--- ========= Vendor Metrics =========
-CREATE TABLE public.vendor_metrics (
-    vendor_name VARCHAR PRIMARY KEY, -- Matches products.vendor
-    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-
-    -- Counts & Basic Info
-    product_count INT NOT NULL DEFAULT 0,           -- Total products from this vendor
-    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products
-    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
-
-    -- Current Stock Value (approximated)
-    current_stock_units INT NOT NULL DEFAULT 0,
-    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-
-    -- On Order Value
-    on_order_units INT NOT NULL DEFAULT 0,
-    on_order_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-
-    -- PO Performance (Simplified)
-    po_count_365d INT NOT NULL DEFAULT 0, -- Count of distinct POs created in last year
-    avg_lead_time_days INT,             -- Calculated from received POs historically
-
-    -- Rolling Period Aggregates (Summed from product_metrics)
-    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-
-    -- Calculated KPIs (Based on 30d aggregates)
-    avg_margin_30d NUMERIC(14, 4), -- (profit / revenue) * 100
-    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
-    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
-    -- Add more KPIs if needed (e.g., avg product value, sell-through rate for vendor)
-);
-CREATE INDEX idx_vendor_metrics_active_count ON public.vendor_metrics(active_product_count);
-
-
--- ========= Brand Metrics =========
-CREATE TABLE public.brand_metrics (
-    brand_name VARCHAR PRIMARY KEY, -- Matches products.brand (use 'Unbranded' for NULLs)
-    last_calculated TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-
-    -- Counts & Basic Info
-    product_count INT NOT NULL DEFAULT 0,           -- Total products of this brand
-    active_product_count INT NOT NULL DEFAULT 0,    -- Visible products
-    replenishable_product_count INT NOT NULL DEFAULT 0,-- Replenishable products
-
-    -- Current Stock Value (approximated)
-    current_stock_units INT NOT NULL DEFAULT 0,
-    current_stock_cost NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    current_stock_retail NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-
-    -- Rolling Period Aggregates (Summed from product_metrics)
-    sales_7d INT NOT NULL DEFAULT 0, revenue_7d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_30d INT NOT NULL DEFAULT 0, revenue_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    profit_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00, cogs_30d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    sales_365d INT NOT NULL DEFAULT 0, revenue_365d NUMERIC(16, 4) NOT NULL DEFAULT 0.00,
-    lifetime_sales INT NOT NULL DEFAULT 0, lifetime_revenue NUMERIC(18, 4) NOT NULL DEFAULT 0.00,
-
-    -- Calculated KPIs (Based on 30d aggregates)
-    avg_margin_30d NUMERIC(7, 3), -- (profit / revenue) * 100
-    sales_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in sales units
-    revenue_growth_30d_vs_prev NUMERIC(10, 2), -- % growth in revenue
-    -- Add more KPIs if needed (e.g., avg product value, sell-through rate for brand)
-);
-CREATE INDEX idx_brand_metrics_active_count ON public.brand_metrics(active_product_count);

inventory-server/db/schema.sql

@@ -1,304 +0,0 @@
--- Enable strict error reporting
-SET session_replication_role = 'replica';  -- Disable foreign key checks temporarily
-
--- Create function for updating timestamps
-CREATE OR REPLACE FUNCTION update_updated_column() RETURNS TRIGGER AS $func$
-BEGIN
-    -- Check which table is being updated and use the appropriate column
-    IF TG_TABLE_NAME = 'categories' THEN
-        NEW.updated_at = CURRENT_TIMESTAMP;
-    ELSIF TG_TABLE_NAME IN ('products', 'orders', 'purchase_orders', 'receivings') THEN
-        NEW.updated = CURRENT_TIMESTAMP;
-    END IF;
-    RETURN NEW;
-END;
-$func$ language plpgsql;
-
--- Create tables
-CREATE TABLE products (
-    pid BIGINT NOT NULL,
-    title TEXT NOT NULL,
-    description TEXT,
-    sku TEXT NOT NULL,
-    created_at TIMESTAMP WITH TIME ZONE,
-    first_received TIMESTAMP WITH TIME ZONE,
-    stock_quantity INTEGER DEFAULT 0,
-    preorder_count INTEGER DEFAULT 0,
-    notions_inv_count INTEGER DEFAULT 0,
-    price NUMERIC(14, 4) NOT NULL,
-    regular_price NUMERIC(14, 4) NOT NULL,
-    cost_price NUMERIC(14, 4),
-    landing_cost_price NUMERIC(14, 4),
-    barcode TEXT,
-    harmonized_tariff_code TEXT,
-    updated_at TIMESTAMP WITH TIME ZONE,
-    visible BOOLEAN DEFAULT true,
-    managing_stock BOOLEAN DEFAULT true,
-    replenishable BOOLEAN DEFAULT true,
-    vendor TEXT,
-    vendor_reference TEXT,
-    notions_reference TEXT,
-    permalink TEXT,
-    categories TEXT,
-    image TEXT,
-    image_175 TEXT,
-    image_full TEXT,
-    brand TEXT,
-    line TEXT,
-    subline TEXT,
-    artist TEXT,
-    options TEXT,
-    tags TEXT,
-    moq INTEGER DEFAULT 1,
-    uom INTEGER DEFAULT 1,
-    rating NUMERIC(14, 4) DEFAULT 0.00,
-    reviews INTEGER DEFAULT 0,
-    weight NUMERIC(14, 4),
-    length NUMERIC(14, 4),
-    width NUMERIC(14, 4),
-    height NUMERIC(14, 4),
-    country_of_origin TEXT,
-    location TEXT,
-    total_sold INTEGER DEFAULT 0,
-    baskets INTEGER DEFAULT 0,
-    notifies INTEGER DEFAULT 0,
-    date_last_sold DATE,
-    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (pid)
-);
-
--- Create trigger for products
-CREATE TRIGGER update_products_updated
-    BEFORE UPDATE ON products
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_column();
-
--- Create indexes for products table
-CREATE INDEX idx_products_sku ON products(sku);
-CREATE INDEX idx_products_vendor ON products(vendor);
-CREATE INDEX idx_products_brand ON products(brand);
-CREATE INDEX idx_products_visible ON products(visible);
-CREATE INDEX idx_products_replenishable ON products(replenishable);
-CREATE INDEX idx_products_updated ON products(updated);
-
--- Create categories table with hierarchy support
-CREATE TABLE categories (
-    cat_id BIGINT PRIMARY KEY,
-    name TEXT NOT NULL,
-    type SMALLINT NOT NULL,
-    parent_id BIGINT,
-    description TEXT,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    status TEXT DEFAULT 'active',
-    FOREIGN KEY (parent_id) REFERENCES categories(cat_id) ON DELETE SET NULL
-);
-
--- Create trigger for categories
-CREATE TRIGGER update_categories_updated_at
-    BEFORE UPDATE ON categories
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_column();
-
-COMMENT ON COLUMN categories.type IS '10=section, 11=category, 12=subcategory, 13=subsubcategory, 1=company, 2=line, 3=subline, 40=artist';
-
-CREATE INDEX idx_categories_parent ON categories(parent_id);
-CREATE INDEX idx_categories_type ON categories(type);
-CREATE INDEX idx_categories_status ON categories(status);
-CREATE INDEX idx_categories_name ON categories(name);
-CREATE INDEX idx_categories_name_type ON categories(name, type);
-
--- Create product_categories junction table
-CREATE TABLE product_categories (
-    cat_id BIGINT NOT NULL,
-    pid BIGINT NOT NULL,
-    PRIMARY KEY (pid, cat_id),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    FOREIGN KEY (cat_id) REFERENCES categories(cat_id) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_product_categories_category ON product_categories(cat_id);
-
--- Create orders table with its indexes
-CREATE TABLE orders (
-    id BIGSERIAL PRIMARY KEY,
-    order_number TEXT NOT NULL,
-    pid BIGINT NOT NULL,
-    sku TEXT NOT NULL,
-    date TIMESTAMP WITH TIME ZONE NOT NULL,
-    price NUMERIC(14, 4) NOT NULL,
-    quantity INTEGER NOT NULL,
-    discount NUMERIC(14, 4) DEFAULT 0.0000,
-    tax NUMERIC(14, 4) DEFAULT 0.0000,
-    tax_included BOOLEAN DEFAULT false,
-    shipping NUMERIC(14, 4) DEFAULT 0.0000,
-    costeach NUMERIC(14, 4) DEFAULT 0.0000,
-    customer TEXT NOT NULL,
-    customer_name TEXT,
-    status TEXT DEFAULT 'pending',
-    canceled BOOLEAN DEFAULT false,
-    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    UNIQUE (order_number, pid),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE RESTRICT
-);
-
--- Create trigger for orders
-CREATE TRIGGER update_orders_updated
-    BEFORE UPDATE ON orders
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_column();
-
-CREATE INDEX idx_orders_number ON orders(order_number);
-CREATE INDEX idx_orders_pid ON orders(pid);
-CREATE INDEX idx_orders_sku ON orders(sku);
-CREATE INDEX idx_orders_customer ON orders(customer);
-CREATE INDEX idx_orders_date ON orders(date);
-CREATE INDEX idx_orders_status ON orders(status);
-CREATE INDEX idx_orders_pid_date ON orders(pid, date);
-CREATE INDEX idx_orders_updated ON orders(updated);
-
--- Create purchase_orders table with its indexes
--- This table now focuses solely on purchase order intent, not receivings
-CREATE TABLE purchase_orders (
-    id BIGSERIAL PRIMARY KEY,
-    po_id TEXT NOT NULL,
-    vendor TEXT NOT NULL,
-    date TIMESTAMP WITH TIME ZONE NOT NULL,
-    expected_date DATE,
-    pid BIGINT NOT NULL,
-    sku TEXT NOT NULL,
-    name TEXT NOT NULL,
-    po_cost_price NUMERIC(14, 4) NOT NULL,
-    status TEXT DEFAULT 'created',
-    notes TEXT,
-    long_note TEXT,
-    ordered INTEGER NOT NULL,
-    supplier_id INTEGER,
-    date_created TIMESTAMP WITH TIME ZONE,
-    date_ordered TIMESTAMP WITH TIME ZONE,
-    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    UNIQUE (po_id, pid)
-);
-
--- Create trigger for purchase_orders
-CREATE TRIGGER update_purchase_orders_updated
-    BEFORE UPDATE ON purchase_orders
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_column();
-
-COMMENT ON COLUMN purchase_orders.name IS 'Product name from products.description';
-COMMENT ON COLUMN purchase_orders.po_cost_price IS 'Original cost from PO';
-COMMENT ON COLUMN purchase_orders.status IS 'canceled, created, electronically_ready_send, ordered, preordered, electronically_sent, receiving_started, done';
-
-CREATE INDEX idx_po_id ON purchase_orders(po_id);
-CREATE INDEX idx_po_sku ON purchase_orders(sku);
-CREATE INDEX idx_po_vendor ON purchase_orders(vendor);
-CREATE INDEX idx_po_status ON purchase_orders(status);
-CREATE INDEX idx_po_expected_date ON purchase_orders(expected_date);
-CREATE INDEX idx_po_pid_status ON purchase_orders(pid, status);
-CREATE INDEX idx_po_pid_date ON purchase_orders(pid, date);
-CREATE INDEX idx_po_updated ON purchase_orders(updated);
-CREATE INDEX idx_po_supplier_id ON purchase_orders(supplier_id);
-
--- Create receivings table to track actual receipt of goods
-CREATE TABLE receivings (
-    id BIGSERIAL PRIMARY KEY,
-    receiving_id TEXT NOT NULL,
-    pid BIGINT NOT NULL,
-    sku TEXT NOT NULL,
-    name TEXT NOT NULL,
-    vendor TEXT,
-    qty_each INTEGER NOT NULL,
-    qty_each_orig INTEGER,
-    cost_each NUMERIC(14, 5) NOT NULL,
-    cost_each_orig NUMERIC(14, 5),
-    received_by INTEGER,
-    received_by_name TEXT,
-    received_date TIMESTAMP WITH TIME ZONE NOT NULL,
-    receiving_created_date TIMESTAMP WITH TIME ZONE,
-    supplier_id INTEGER,
-    status TEXT DEFAULT 'created',
-    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    UNIQUE (receiving_id, pid)
-);
-
--- Create trigger for receivings
-CREATE TRIGGER update_receivings_updated
-    BEFORE UPDATE ON receivings
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_column();
-
-COMMENT ON COLUMN receivings.status IS 'canceled, created, partial_received, full_received, paid';
-COMMENT ON COLUMN receivings.qty_each_orig IS 'Original quantity from the source system';
-COMMENT ON COLUMN receivings.cost_each_orig IS 'Original cost from the source system';
-COMMENT ON COLUMN receivings.vendor IS 'Vendor name, same as in purchase_orders';
-
-CREATE INDEX idx_receivings_id ON receivings(receiving_id);
-CREATE INDEX idx_receivings_pid ON receivings(pid);
-CREATE INDEX idx_receivings_sku ON receivings(sku);
-CREATE INDEX idx_receivings_status ON receivings(status);
-CREATE INDEX idx_receivings_received_date ON receivings(received_date);
-CREATE INDEX idx_receivings_supplier_id ON receivings(supplier_id);
-CREATE INDEX idx_receivings_vendor ON receivings(vendor);
-CREATE INDEX idx_receivings_updated ON receivings(updated);
-
-SET session_replication_role = 'origin';  -- Re-enable foreign key checks
-
--- Create views for common calculations
--- product_sales_trends view moved to metrics-schema.sql 
-
--- -- Historical data tables imported from production
--- CREATE TABLE imported_product_current_prices (
---   price_id BIGSERIAL PRIMARY KEY,
---   pid BIGINT NOT NULL,
---   qty_buy SMALLINT NOT NULL,
---   is_min_qty_buy BOOLEAN NOT NULL,
---   price_each NUMERIC(10,3) NOT NULL,
---   qty_limit SMALLINT NOT NULL,
---   no_promo BOOLEAN NOT NULL,
---   checkout_offer BOOLEAN NOT NULL,
---   active BOOLEAN NOT NULL,
---   date_active TIMESTAMP WITH TIME ZONE,
---   date_deactive TIMESTAMP WITH TIME ZONE,
---   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
--- );
-
--- CREATE INDEX idx_imported_product_current_prices_pid ON imported_product_current_prices(pid, active, qty_buy);
--- CREATE INDEX idx_imported_product_current_prices_checkout ON imported_product_current_prices(checkout_offer, active);
--- CREATE INDEX idx_imported_product_current_prices_deactive ON imported_product_current_prices(date_deactive, active);
--- CREATE INDEX idx_imported_product_current_prices_active ON imported_product_current_prices(date_active, active);
-
--- CREATE TABLE imported_daily_inventory (
---   date DATE NOT NULL,
---   pid BIGINT NOT NULL,
---   amountsold SMALLINT NOT NULL DEFAULT 0,
---   times_sold SMALLINT NOT NULL DEFAULT 0,
---   qtyreceived SMALLINT NOT NULL DEFAULT 0,
---   price NUMERIC(7,2) NOT NULL DEFAULT 0,
---   costeach NUMERIC(7,2) NOT NULL DEFAULT 0,
---   stamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
---   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
---   PRIMARY KEY (date, pid)
--- );
-
--- CREATE INDEX idx_imported_daily_inventory_pid ON imported_daily_inventory(pid);
-
--- CREATE TABLE imported_product_stat_history (
---   pid BIGINT NOT NULL,
---   date DATE NOT NULL,
---   score NUMERIC(10,2) NOT NULL,
---   score2 NUMERIC(10,2) NOT NULL,
---   qty_in_baskets SMALLINT NOT NULL,
---   qty_sold SMALLINT NOT NULL,
---   notifies_set SMALLINT NOT NULL,
---   visibility_score NUMERIC(10,2) NOT NULL,
---   health_score VARCHAR(5) NOT NULL,
---   sold_view_score NUMERIC(6,3) NOT NULL,
---   updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
---   PRIMARY KEY (pid, date)
--- );
-
--- CREATE INDEX idx_imported_product_stat_history_date ON imported_product_stat_history(date); 

inventory-server/db/setup-schema.sql

@@ -1,115 +0,0 @@
--- Templates table for storing import templates
-CREATE TABLE IF NOT EXISTS templates (
-    id SERIAL PRIMARY KEY,
-    company TEXT NOT NULL,
-    product_type TEXT NOT NULL,
-    supplier TEXT,
-    msrp DECIMAL(10,2),
-    cost_each DECIMAL(10,2),
-    qty_per_unit INTEGER,
-    case_qty INTEGER,
-    hts_code TEXT,
-    description TEXT,
-    weight DECIMAL(10,2),
-    length DECIMAL(10,2),
-    width DECIMAL(10,2),
-    height DECIMAL(10,2),
-    tax_cat TEXT,
-    size_cat TEXT,
-    categories TEXT[],
-    ship_restrictions TEXT[],
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    UNIQUE(company, product_type)
-);
-
--- AI Prompts table for storing validation prompts
-CREATE TABLE IF NOT EXISTS ai_prompts (
-    id SERIAL PRIMARY KEY,
-    prompt_text TEXT NOT NULL,
-    prompt_type TEXT NOT NULL CHECK (prompt_type IN ('general', 'company_specific', 'system')),
-    company TEXT,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    CONSTRAINT unique_company_prompt UNIQUE (company),
-    CONSTRAINT company_required_for_specific CHECK (
-        (prompt_type = 'general' AND company IS NULL) OR 
-        (prompt_type = 'system' AND company IS NULL) OR 
-        (prompt_type = 'company_specific' AND company IS NOT NULL)
-    )
-);
-
--- Create a unique partial index to ensure only one general prompt
-CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_general_prompt 
-ON ai_prompts (prompt_type) 
-WHERE prompt_type = 'general';
-
--- Create a unique partial index to ensure only one system prompt
-CREATE UNIQUE INDEX IF NOT EXISTS idx_unique_system_prompt 
-ON ai_prompts (prompt_type) 
-WHERE prompt_type = 'system';
-
--- Reusable Images table for storing persistent images
-CREATE TABLE IF NOT EXISTS reusable_images (
-    id SERIAL PRIMARY KEY,
-    name TEXT NOT NULL,
-    filename TEXT NOT NULL,
-    file_path TEXT NOT NULL,
-    image_url TEXT NOT NULL,
-    is_global BOOLEAN NOT NULL DEFAULT false,
-    company TEXT,
-    mime_type TEXT,
-    file_size INTEGER,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-    CONSTRAINT company_required_for_non_global CHECK (
-        (is_global = true AND company IS NULL) OR
-        (is_global = false AND company IS NOT NULL)
-    )
-);
-
--- Create index on company for efficient querying
-CREATE INDEX IF NOT EXISTS idx_reusable_images_company ON reusable_images(company);
--- Create index on is_global for efficient querying
-CREATE INDEX IF NOT EXISTS idx_reusable_images_is_global ON reusable_images(is_global);
-
--- AI Validation Performance Tracking
-CREATE TABLE IF NOT EXISTS ai_validation_performance (
-    id SERIAL PRIMARY KEY,
-    prompt_length INTEGER NOT NULL,
-    product_count INTEGER NOT NULL,
-    start_time TIMESTAMP WITH TIME ZONE NOT NULL,
-    end_time TIMESTAMP WITH TIME ZONE NOT NULL,
-    duration_seconds DECIMAL(10,2) GENERATED ALWAYS AS (EXTRACT(EPOCH FROM (end_time - start_time))) STORED,
-    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
-);
-
--- Create index on prompt_length for efficient querying
-CREATE INDEX IF NOT EXISTS idx_ai_validation_prompt_length ON ai_validation_performance(prompt_length);
-
--- Function to update the updated_at timestamp
-CREATE OR REPLACE FUNCTION update_updated_at_column()
-RETURNS TRIGGER AS $$
-BEGIN
-    NEW.updated_at = CURRENT_TIMESTAMP;
-    RETURN NEW;
-END;
-$$ language 'plpgsql';
-
--- Trigger to automatically update the updated_at column
-CREATE TRIGGER update_templates_updated_at
-    BEFORE UPDATE ON templates
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-
--- Trigger to automatically update the updated_at column for ai_prompts
-CREATE TRIGGER update_ai_prompts_updated_at
-    BEFORE UPDATE ON ai_prompts
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column();
-    
--- Trigger to automatically update the updated_at column for reusable_images
-CREATE TRIGGER update_reusable_images_updated_at
-    BEFORE UPDATE ON reusable_images
-    FOR EACH ROW
-    EXECUTE FUNCTION update_updated_at_column(); 

inventory-server/old/backfill-snapshots.js

@@ -1,426 +0,0 @@
-const path = require('path');
-const fs = require('fs');
-const progress = require('../scripts/metrics-new/utils/progress'); // Assuming progress utils are here
-const { getConnection, closePool } = require('../scripts/metrics-new/utils/db'); // Assuming db utils are here
-const os = require('os'); // For detecting number of CPU cores
-
-// --- Configuration ---
-const BATCH_SIZE_DAYS = 1; // Process 1 day per database function call
-const SQL_FUNCTION_FILE = path.resolve(__dirname, 'backfill_historical_snapshots.sql'); // Correct path
-const LOG_PROGRESS_INTERVAL_MS = 5000; // Update console progress roughly every 5 seconds
-const HISTORY_TYPE = 'backfill_snapshots'; // Identifier for history table
-const MAX_WORKERS = Math.max(1, Math.floor(os.cpus().length / 2)); // Use half of available CPU cores
-const USE_PARALLEL = false; // Set to true to enable parallel processing
-const PG_STATEMENT_TIMEOUT_MS = 1800000; // 30 minutes max per query
-
-// --- Cancellation Handling ---
-let isCancelled = false;
-let runningQueryPromise = null; // To potentially track the active query
-
-function requestCancellation() {
-    if (!isCancelled) {
-        isCancelled = true;
-        console.warn('\nCancellation requested. Finishing current batch then stopping...');
-        // Note: We are NOT forcefully cancelling the backend query anymore.
-    }
-}
-
-process.on('SIGINT', requestCancellation); // Handle Ctrl+C
-process.on('SIGTERM', requestCancellation); // Handle termination signals
-
-// --- Main Backfill Function ---
-async function backfillSnapshots(cmdStartDate, cmdEndDate, cmdStartBatch = 1) {
-    let connection;
-    const overallStartTime = Date.now();
-    let calculateHistoryId = null;
-    let processedDaysTotal = 0; // Track total days processed across all batches executed in this run
-    let currentBatchNum = cmdStartBatch > 0 ? cmdStartBatch : 1;
-    let totalBatches = 0; // Initialize totalBatches
-    let totalDays = 0; // Initialize totalDays
-
-    console.log(`Starting snapshot backfill process...`);
-    console.log(`SQL Function definition file: ${SQL_FUNCTION_FILE}`);
-    if (!fs.existsSync(SQL_FUNCTION_FILE)) {
-         console.error(`FATAL: SQL file not found at ${SQL_FUNCTION_FILE}`);
-         process.exit(1); // Exit early if file doesn't exist
-    }
-
-    try {
-        // Set up a connection with higher memory limits
-        connection = await getConnection({
-            // Add performance-related settings
-            application_name: 'backfill_snapshots',
-            statement_timeout: PG_STATEMENT_TIMEOUT_MS, // 30 min timeout per statement
-            // These parameters may need to be configured in your database:
-            // work_mem: '1GB',
-            // maintenance_work_mem: '2GB',
-            // temp_buffers: '1GB',
-        });
-        
-        console.log('Database connection acquired.');
-
-        // --- Ensure Function Exists ---
-        console.log('Ensuring database function is up-to-date...');
-        try {
-            const sqlFunctionDef = fs.readFileSync(SQL_FUNCTION_FILE, 'utf8');
-            if (!sqlFunctionDef.includes('CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final')) {
-                throw new Error(`SQL file ${SQL_FUNCTION_FILE} does not seem to contain the function definition.`);
-            }
-            await connection.query(sqlFunctionDef); // Execute the whole file
-            console.log('Database function `backfill_daily_snapshots_range_final` created/updated.');
-            
-            // Add performance query hints to the database
-            await connection.query(`
-                -- Analyze tables for better query planning
-                ANALYZE public.products;
-                ANALYZE public.imported_daily_inventory;
-                ANALYZE public.imported_product_stat_history;
-                ANALYZE public.daily_product_snapshots;
-                ANALYZE public.imported_product_current_prices;
-            `).catch(err => {
-                // Non-fatal if analyze fails
-                console.warn('Failed to analyze tables (non-fatal):', err.message);
-            });
-            
-        } catch (err) {
-            console.error(`Error processing SQL function file ${SQL_FUNCTION_FILE}:`, err);
-            throw new Error(`Failed to create or replace DB function: ${err.message}`);
-        }
-
-        // --- Prepare History Record ---
-        console.log('Preparing calculation history record...');
-         // Ensure history table exists (optional, could be done elsewhere)
-         await connection.query(`
-            CREATE TABLE IF NOT EXISTS public.calculate_history (
-                id SERIAL PRIMARY KEY,
-                start_time TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-                end_time TIMESTAMPTZ,
-                duration_seconds INTEGER,
-                status VARCHAR(20) NOT NULL, -- e.g., 'running', 'completed', 'failed', 'cancelled'
-                error_message TEXT,
-                additional_info JSONB -- Store type, file, batch info etc.
-            );
-         `);
-        // Mark previous runs of this type as potentially failed if they were left 'running'
-        await connection.query(`
-            UPDATE public.calculate_history
-            SET status = 'failed', error_message = 'Interrupted by new run.'
-            WHERE status = 'running' AND additional_info->>'type' = $1;
-        `, [HISTORY_TYPE]);
-
-        // Create new history record
-        const historyResult = await connection.query(`
-            INSERT INTO public.calculate_history (start_time, status, additional_info)
-            VALUES (NOW(), 'running', jsonb_build_object('type', $1::text, 'sql_file', $2::text, 'start_batch', $3::integer))
-            RETURNING id;
-        `, [HISTORY_TYPE, path.basename(SQL_FUNCTION_FILE), cmdStartBatch]);
-        calculateHistoryId = historyResult.rows[0].id;
-        console.log(`Calculation history record created with ID: ${calculateHistoryId}`);
-
-
-        // --- Determine Date Range ---
-        console.log('Determining date range...');
-        let effectiveStartDate, effectiveEndDate;
-
-        // Use command-line dates if provided, otherwise query DB
-        if (cmdStartDate) {
-             effectiveStartDate = cmdStartDate;
-        } else {
-             const minDateResult = await connection.query(`
-                SELECT LEAST(
-                    COALESCE((SELECT MIN(date) FROM public.imported_daily_inventory WHERE date > '1970-01-01'), CURRENT_DATE),
-                    COALESCE((SELECT MIN(date) FROM public.imported_product_stat_history WHERE date > '1970-01-01'), CURRENT_DATE)
-                )::date as min_date;
-            `);
-             effectiveStartDate = minDateResult.rows[0]?.min_date || new Date().toISOString().split('T')[0]; // Fallback
-             console.log(`Auto-detected start date: ${effectiveStartDate}`);
-        }
-
-        if (cmdEndDate) {
-            effectiveEndDate = cmdEndDate;
-        } else {
-             const maxDateResult = await connection.query(`
-                 SELECT GREATEST(
-                    COALESCE((SELECT MAX(date) FROM public.imported_daily_inventory WHERE date < CURRENT_DATE), '1970-01-01'::date),
-                    COALESCE((SELECT MAX(date) FROM public.imported_product_stat_history WHERE date < CURRENT_DATE), '1970-01-01'::date)
-                )::date as max_date;
-             `);
-             // Ensure end date is not today or in the future
-             effectiveEndDate = maxDateResult.rows[0]?.max_date || new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Default yesterday
-             if (new Date(effectiveEndDate) >= new Date(new Date().toISOString().split('T')[0])) {
-                 effectiveEndDate = new Date(Date.now() - 86400000).toISOString().split('T')[0]; // Set to yesterday if >= today
-             }
-             console.log(`Auto-detected end date: ${effectiveEndDate}`);
-        }
-
-         // Validate dates
-         const dStart = new Date(effectiveStartDate);
-         const dEnd = new Date(effectiveEndDate);
-         if (isNaN(dStart.getTime()) || isNaN(dEnd.getTime()) || dStart > dEnd) {
-             throw new Error(`Invalid date range: Start "${effectiveStartDate}", End "${effectiveEndDate}"`);
-         }
-
-        // --- Batch Processing ---
-        totalDays = Math.ceil((dEnd - dStart) / (1000 * 60 * 60 * 24)) + 1; // Inclusive
-        totalBatches = Math.ceil(totalDays / BATCH_SIZE_DAYS);
-
-        console.log(`Target Date Range: ${effectiveStartDate} to ${effectiveEndDate} (${totalDays} days)`);
-        console.log(`Total Batches: ${totalBatches} (Batch Size: ${BATCH_SIZE_DAYS} days)`);
-        console.log(`Starting from Batch: ${currentBatchNum}`);
-
-        // Initial progress update
-        progress.outputProgress({ 
-            status: 'running', 
-            operation: 'Starting Batch Processing',
-            currentBatch: currentBatchNum, 
-            totalBatches: totalBatches, 
-            totalDays: totalDays,
-            elapsed: '0s', 
-            remaining: 'Calculating...', 
-            rate: 0,
-            historyId: calculateHistoryId // Include history ID in the object
-        });
-
-        while (currentBatchNum <= totalBatches && !isCancelled) {
-            const batchOffset = (currentBatchNum - 1) * BATCH_SIZE_DAYS;
-            const batchStartDate = new Date(dStart);
-            batchStartDate.setDate(dStart.getDate() + batchOffset);
-
-            const batchEndDate = new Date(batchStartDate);
-            batchEndDate.setDate(batchStartDate.getDate() + BATCH_SIZE_DAYS - 1);
-
-            // Clamp batch end date to the overall effective end date
-            if (batchEndDate > dEnd) {
-                batchEndDate.setTime(dEnd.getTime());
-            }
-
-            const batchStartDateStr = batchStartDate.toISOString().split('T')[0];
-            const batchEndDateStr = batchEndDate.toISOString().split('T')[0];
-            const batchStartTime = Date.now();
-
-            console.log(`\n--- Processing Batch ${currentBatchNum} / ${totalBatches} ---`);
-            console.log(`    Dates: ${batchStartDateStr} to ${batchEndDateStr}`);
-
-            // Execute the function for the batch
-             try {
-                progress.outputProgress({
-                     status: 'running', 
-                     operation: `Executing DB function for batch ${currentBatchNum}...`,
-                     currentBatch: currentBatchNum, 
-                     totalBatches: totalBatches, 
-                     totalDays: totalDays,
-                     elapsed: progress.formatElapsedTime(overallStartTime), 
-                     remaining: 'Executing...', 
-                     rate: 0,
-                     historyId: calculateHistoryId
-                 });
-
-                // Performance improvement: Add batch processing hint
-                await connection.query('SET LOCAL enable_parallel_append = on; SET LOCAL enable_parallel_hash = on; SET LOCAL max_parallel_workers_per_gather = 4;');
-
-                // Store promise in case we need to try and cancel (though not implemented forcefully)
-                runningQueryPromise = connection.query(
-                    `SELECT backfill_daily_snapshots_range_final($1::date, $2::date);`,
-                    [batchStartDateStr, batchEndDateStr]
-                );
-                await runningQueryPromise; // Wait for the function call to complete
-                runningQueryPromise = null; // Clear the promise
-
-                const batchDurationMs = Date.now() - batchStartTime;
-                const daysInThisBatch = Math.ceil((batchEndDate - batchStartDate) / (1000 * 60 * 60 * 24)) + 1;
-                processedDaysTotal += daysInThisBatch;
-
-                console.log(`    Batch ${currentBatchNum} completed in ${progress.formatElapsedTime(batchStartTime)}.`);
-
-                // --- Update Progress & History ---
-                const overallElapsedSec = Math.round((Date.now() - overallStartTime) / 1000);
-                progress.outputProgress({
-                    status: 'running', 
-                    operation: `Completed batch ${currentBatchNum}`,
-                    currentBatch: currentBatchNum, 
-                    totalBatches: totalBatches, 
-                    totalDays: totalDays,
-                    processedDays: processedDaysTotal,
-                    elapsed: progress.formatElapsedTime(overallStartTime),
-                    remaining: progress.estimateRemaining(overallStartTime, processedDaysTotal, totalDays),
-                    rate: progress.calculateRate(overallStartTime, processedDaysTotal),
-                    batchDuration: progress.formatElapsedTime(batchStartTime),
-                    historyId: calculateHistoryId
-                });
-
-                // Save checkpoint in history
-                await connection.query(`
-                    UPDATE public.calculate_history
-                    SET additional_info = jsonb_set(additional_info, '{last_completed_batch}', $1::jsonb)
-                                        || jsonb_build_object('last_processed_date', $2::text)
-                    WHERE id = $3::integer;
-                `, [JSON.stringify(currentBatchNum), batchEndDateStr, calculateHistoryId]);
-
-
-             } catch (batchError) {
-                 console.error(`\n--- ERROR in Batch ${currentBatchNum} (${batchStartDateStr} to ${batchEndDateStr}) ---`);
-                 console.error('    Database Error:', batchError.message);
-                 console.error('    DB Error Code:', batchError.code);
-                 // Log detailed error to history and re-throw to stop the process
-                 await connection.query(`
-                    UPDATE public.calculate_history
-                    SET status = 'failed',
-                        end_time = NOW(),
-                        duration_seconds = $1::integer,
-                        error_message = $2::text,
-                        additional_info = additional_info || jsonb_build_object('failed_batch', $3::integer, 'failed_date_range', $4::text)
-                    WHERE id = $5::integer;
-                 `, [
-                     Math.round((Date.now() - overallStartTime) / 1000),
-                     `Batch ${currentBatchNum} failed: ${batchError.message} (Code: ${batchError.code || 'N/A'})`,
-                     currentBatchNum,
-                     `${batchStartDateStr} to ${batchEndDateStr}`,
-                     calculateHistoryId
-                 ]);
-                 throw batchError; // Stop execution
-             }
-
-            currentBatchNum++;
-            // Optional delay between batches
-            // await new Promise(resolve => setTimeout(resolve, 500));
-
-        } // End while loop
-
-        // --- Final Outcome ---
-        const finalStatus = isCancelled ? 'cancelled' : 'completed';
-        const finalMessage = isCancelled ? `Calculation stopped after completing batch ${currentBatchNum - 1}.` : 'Historical snapshots backfill completed successfully.';
-        const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
-
-        console.log(`\n--- Backfill ${finalStatus.toUpperCase()} ---`);
-        console.log(finalMessage);
-        console.log(`Total duration: ${progress.formatElapsedTime(overallStartTime)}`);
-
-        // Update history record
-        await connection.query(`
-            UPDATE public.calculate_history SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3
-            WHERE id = $4::integer;
-        `, [finalStatus, finalDurationSec, (isCancelled ? 'User cancelled' : null), calculateHistoryId]);
-
-        if (!isCancelled) {
-            progress.clearProgress(); // Clear progress state only on successful completion
-        } else {
-             progress.outputProgress({ // Final cancelled status update
-                status: 'cancelled', 
-                operation: finalMessage,
-                currentBatch: currentBatchNum - 1, 
-                totalBatches: totalBatches, 
-                totalDays: totalDays,
-                processedDays: processedDaysTotal,
-                elapsed: progress.formatElapsedTime(overallStartTime), 
-                remaining: 'Cancelled', 
-                rate: 0,
-                historyId: calculateHistoryId
-             });
-        }
-
-        return { success: true, status: finalStatus, message: finalMessage, duration: finalDurationSec };
-
-    } catch (error) {
-        console.error('\n--- Backfill encountered an unrecoverable error ---');
-        console.error(error.message);
-        const finalDurationSec = Math.round((Date.now() - overallStartTime) / 1000);
-
-        // Update history if possible
-        if (connection && calculateHistoryId) {
-             try {
-                 await connection.query(`
-                    UPDATE public.calculate_history
-                    SET status = $1::calculation_status, end_time = NOW(), duration_seconds = $2::integer, error_message = $3::text
-                    WHERE id = $4::integer;
-                 `, [
-                     isCancelled ? 'cancelled' : 'failed',
-                     finalDurationSec,
-                     error.message,
-                     calculateHistoryId
-                 ]);
-             } catch (histError) {
-                 console.error("Failed to update history record with error state:", histError);
-             }
-        } else {
-            console.error("Could not update history record (no ID or connection).");
-        }
-
-        // FIX: Use initialized value or a default if loop never started
-        const batchNumForError = currentBatchNum > cmdStartBatch ? currentBatchNum - 1 : cmdStartBatch - 1;
-        
-        // Update progress.outputProgress call to match actual function signature
-        try {
-            // Create progress data object
-            const progressData = {
-                status: 'failed',
-                operation: 'Backfill failed',
-                message: error.message,
-                currentBatch: batchNumForError,
-                totalBatches: totalBatches,
-                totalDays: totalDays,
-                processedDays: processedDaysTotal,
-                elapsed: progress.formatElapsedTime(overallStartTime),
-                remaining: 'Failed',
-                rate: 0,
-                // Include history ID in progress data if needed
-                historyId: calculateHistoryId
-            };
-            
-            // Call with single object parameter (not separate historyId)
-            progress.outputProgress(progressData);
-        } catch (progressError) {
-            console.error('Failed to report progress:', progressError);
-        }
-
-        return { success: false, status: 'failed', error: error.message, duration: finalDurationSec };
-
-    } finally {
-        if (connection) {
-            console.log('Releasing database connection.');
-            connection.release();
-        }
-        // Close pool only if this script is meant to be standalone
-        // If part of a larger app, the app should manage pool closure
-        // console.log('Closing database pool.');
-        // await closePool();
-    }
-}
-
-// --- Script Execution ---
-
-// Parse command-line arguments
-const args = process.argv.slice(2);
-let cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg = 1; // Default start batch is 1
-
-for (let i = 0; i < args.length; i++) {
-    if (args[i] === '--start-date' && args[i+1]) cmdStartDateArg = args[++i];
-    else if (args[i] === '--end-date' && args[i+1]) cmdEndDateArg = args[++i];
-    else if (args[i] === '--start-batch' && args[i+1]) cmdStartBatchArg = parseInt(args[++i], 10);
-}
-
-if (isNaN(cmdStartBatchArg) || cmdStartBatchArg < 1) {
-    console.warn(`Invalid --start-batch value. Defaulting to 1.`);
-    cmdStartBatchArg = 1;
-}
-
-// Run the backfill process
-backfillSnapshots(cmdStartDateArg, cmdEndDateArg, cmdStartBatchArg)
-    .then(result => {
-        if (result.success) {
-            console.log(`\n✅ ${result.message} (Duration: ${result.duration}s)`);
-            process.exitCode = 0; // Success
-        } else {
-            console.error(`\n❌ Backfill failed: ${result.error || 'Unknown error'} (Duration: ${result.duration}s)`);
-            process.exitCode = 1; // Failure
-        }
-    })
-    .catch(err => {
-        console.error('\n❌ Unexpected error during backfill execution:', err);
-        process.exitCode = 1; // Failure
-    })
-    .finally(async () => {
-        // Ensure pool is closed if run standalone
-         console.log('Backfill script finished. Closing pool.');
-         await closePool(); // Make sure closePool exists and works in your db utils
-         process.exit(process.exitCode); // Exit with appropriate code
-    });

inventory-server/old/backfill_historical_snapshots.sql

@@ -1,161 +0,0 @@
--- Description: Backfills the daily_product_snapshots table using imported historical unit data
---              (daily inventory/stats) and historical price data (current prices table).
---              - Uses imported daily sales/receipt UNIT counts for accuracy.
---              - ESTIMATES historical stock levels using a forward calculation.
---              - APPROXIMATES historical REVENUE using looked-up historical base prices.
---              - APPROXIMATES historical COGS, PROFIT, and STOCK VALUE using CURRENT product costs/prices.
---              Run ONCE after importing historical data and before initial product_metrics population.
--- Dependencies: Core import tables (products), imported history tables (imported_daily_inventory,
---               imported_product_stat_history, imported_product_current_prices),
---               daily_product_snapshots table must exist.
--- Frequency: Run ONCE.
-
-CREATE OR REPLACE FUNCTION backfill_daily_snapshots_range_final(
-    _start_date DATE,
-    _end_date DATE
-)
-RETURNS VOID AS $$
-DECLARE
-    _current_processing_date DATE := _start_date;
-    _batch_start_time TIMESTAMPTZ;
-    _row_count INTEGER;
-BEGIN
-    RAISE NOTICE 'Starting FINAL historical snapshot backfill from % to %.', _start_date, _end_date;
-    RAISE NOTICE 'Using historical units and historical prices (for revenue approximation).';
-    RAISE NOTICE 'WARNING: Historical COGS, Profit, and Stock Value use CURRENT product costs/prices.';
-
-    -- Ensure end date is not in the future
-    IF _end_date >= CURRENT_DATE THEN
-        _end_date := CURRENT_DATE - INTERVAL '1 day';
-        RAISE NOTICE 'Adjusted end date to % to avoid conflict with hourly script.', _end_date;
-    END IF;
-
-    -- Performance: Create temporary table with product info to avoid repeated lookups
-    CREATE TEMP TABLE IF NOT EXISTS temp_product_info AS 
-    SELECT 
-        pid, 
-        sku,
-        COALESCE(landing_cost_price, cost_price, 0.00) as effective_cost_price,
-        COALESCE(price, 0.00) as current_price,
-        COALESCE(regular_price, 0.00) as current_regular_price
-    FROM public.products;
-    
-    -- Performance: Create index on temporary table
-    CREATE INDEX IF NOT EXISTS temp_product_info_pid_idx ON temp_product_info(pid);
-    
-    ANALYZE temp_product_info;
-    
-    RAISE NOTICE 'Created temporary product info table with % products', (SELECT COUNT(*) FROM temp_product_info);
-
-    WHILE _current_processing_date <= _end_date LOOP
-        _batch_start_time := clock_timestamp();
-        RAISE NOTICE 'Processing date: %', _current_processing_date;
-
-        -- Get Daily Transaction Unit Info from imported history
-        WITH DailyHistoryUnits AS (
-            SELECT
-                pids.pid,
-                -- Prioritize daily_inventory, fallback to product_stat_history for sold qty
-                COALESCE(di.amountsold, ps.qty_sold, 0)::integer as units_sold_today,
-                COALESCE(di.qtyreceived, 0)::integer as units_received_today
-            FROM
-               (SELECT DISTINCT pid FROM temp_product_info) pids -- Ensure all products are considered
-               LEFT JOIN public.imported_daily_inventory di
-                   ON pids.pid = di.pid AND di.date = _current_processing_date
-               LEFT JOIN public.imported_product_stat_history ps
-                   ON pids.pid = ps.pid AND ps.date = _current_processing_date
-            -- Removed WHERE clause to ensure snapshots are created even for days with 0 activity,
-            -- allowing stock carry-over. The main query will handle products properly.
-        ),
-        HistoricalPrice AS (
-            -- Find the base price (qty_buy=1) active on the processing date
-            SELECT DISTINCT ON (pid)
-                pid,
-                price_each
-            FROM public.imported_product_current_prices
-            WHERE
-                qty_buy = 1
-            -- Use TIMESTAMPTZ comparison logic:
-            AND date_active <= (_current_processing_date + interval '1 day' - interval '1 second') -- Active sometime on or before end of processing day
-            AND (date_deactive IS NULL OR date_deactive > _current_processing_date) -- Not deactivated before start of processing day
-            -- Assuming 'active' flag isn't needed if dates are correct; add 'AND active != 0' if necessary
-            ORDER BY
-                pid, date_active DESC -- Get the most recently activated price
-        ),
-        PreviousStock AS (
-            -- Get the estimated stock from the PREVIOUS day snapshot
-            SELECT pid, eod_stock_quantity
-            FROM public.daily_product_snapshots
-            WHERE snapshot_date = _current_processing_date - INTERVAL '1 day'
-        )
-        -- Insert into the daily snapshots table
-        INSERT INTO public.daily_product_snapshots (
-            snapshot_date, pid, sku,
-            eod_stock_quantity, eod_stock_cost, eod_stock_retail, eod_stock_gross, stockout_flag,
-            units_sold, units_returned,
-            gross_revenue, discounts, returns_revenue,
-            net_revenue, cogs, gross_regular_revenue, profit,
-            units_received, cost_received,
-            calculation_timestamp
-        )
-        SELECT
-            _current_processing_date AS snapshot_date,
-            p.pid,
-            p.sku,
-            -- Estimated EOD Stock (using historical daily units)
-            -- Handle potential NULL from joins with COALESCE 0
-            COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0) AS estimated_eod_stock,
-            -- Valued Stock (using estimated stock and CURRENT prices/costs - APPROXIMATION)
-            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.effective_cost_price AS eod_stock_cost,
-            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_price AS eod_stock_retail, -- Stock retail uses current price
-            GREATEST(0, COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) * p.current_regular_price AS eod_stock_gross, -- Stock gross uses current regular price
-            -- Stockout Flag (based on estimated stock)
-            (COALESCE(ps.eod_stock_quantity, 0) + COALESCE(dh.units_received_today, 0) - COALESCE(dh.units_sold_today, 0)) <= 0 AS stockout_flag,
-
-            -- Today's Unit Aggregates from History
-            COALESCE(dh.units_sold_today, 0) as units_sold,
-            0 AS units_returned, -- Placeholder: Cannot determine returns from daily summary
-
-            -- Monetary Values using looked-up Historical Price and CURRENT Cost/RegPrice
-            COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS gross_revenue, -- Approx Revenue
-            0 AS discounts, -- Placeholder
-            0 AS returns_revenue, -- Placeholder
-            COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price) AS net_revenue, -- Approx Net Revenue
-            COALESCE(dh.units_sold_today, 0) * p.effective_cost_price AS cogs, -- Approx COGS (uses CURRENT cost)
-            COALESCE(dh.units_sold_today, 0) * p.current_regular_price AS gross_regular_revenue, -- Approx Gross Regular Revenue
-            -- Approx Profit
-            (COALESCE(dh.units_sold_today, 0) * COALESCE(hp.price_each, p.current_price)) - (COALESCE(dh.units_sold_today, 0) * p.effective_cost_price) AS profit,
-
-            COALESCE(dh.units_received_today, 0) as units_received,
-            -- Estimate received cost using CURRENT product cost
-            COALESCE(dh.units_received_today, 0) * p.effective_cost_price AS cost_received, -- Approx
-
-            clock_timestamp() -- Timestamp of this specific calculation
-        FROM temp_product_info p -- Use the temp table for better performance
-        LEFT JOIN PreviousStock ps ON p.pid = ps.pid
-        LEFT JOIN DailyHistoryUnits dh ON p.pid = dh.pid -- Join today's historical activity
-        LEFT JOIN HistoricalPrice hp ON p.pid = hp.pid -- Join the looked-up historical price
-        -- Optimization: Only process products with activity or previous stock
-        WHERE (dh.units_sold_today > 0 OR dh.units_received_today > 0 OR COALESCE(ps.eod_stock_quantity, 0) > 0)
-
-        ON CONFLICT (snapshot_date, pid) DO NOTHING; -- Avoid errors if rerunning parts, but prefer clean runs
-
-        GET DIAGNOSTICS _row_count = ROW_COUNT;
-        RAISE NOTICE 'Processed %: Inserted/Skipped % rows. Duration: %',
-                     _current_processing_date,
-                     _row_count,
-                     clock_timestamp() - _batch_start_time;
-
-        _current_processing_date := _current_processing_date + INTERVAL '1 day';
-
-    END LOOP;
-    
-    -- Clean up temporary tables
-    DROP TABLE IF EXISTS temp_product_info;
-
-    RAISE NOTICE 'Finished FINAL historical snapshot backfill.';
-END;
-$$ LANGUAGE plpgsql;
-
--- Example usage:
--- SELECT backfill_daily_snapshots_range_final('2023-01-01'::date, '2023-12-31'::date);

inventory-server/old/calculate-metrics.js

@@ -1,558 +0,0 @@
-const path = require('path');
-
-// Change working directory to script directory
-process.chdir(path.dirname(__filename));
-
-require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
-
-// Configuration flags for controlling which metrics to calculate
-// Set to 1 to skip the corresponding calculation, 0 to run it
-const SKIP_PRODUCT_METRICS = 0;
-const SKIP_TIME_AGGREGATES = 0;
-const SKIP_FINANCIAL_METRICS = 0;
-const SKIP_VENDOR_METRICS = 0;
-const SKIP_CATEGORY_METRICS = 0;
-const SKIP_BRAND_METRICS = 0;
-const SKIP_SALES_FORECASTS = 0;
-
-// Add error handler for uncaught exceptions
-process.on('uncaughtException', (error) => {
-    console.error('Uncaught Exception:', error);
-    process.exit(1);
-});
-
-// Add error handler for unhandled promise rejections
-process.on('unhandledRejection', (reason, promise) => {
-    console.error('Unhandled Rejection at:', promise, 'reason:', reason);
-    process.exit(1);
-});
-
-const progress = require('./metrics/utils/progress');
-console.log('Progress module loaded:', {
-    modulePath: require.resolve('./metrics/utils/progress'),
-    exports: Object.keys(progress),
-    currentDir: process.cwd(),
-    scriptDir: __dirname
-});
-
-// Store progress functions in global scope to ensure availability
-global.formatElapsedTime = progress.formatElapsedTime;
-global.estimateRemaining = progress.estimateRemaining;
-global.calculateRate = progress.calculateRate;
-global.outputProgress = progress.outputProgress;
-global.clearProgress = progress.clearProgress;
-global.getProgress = progress.getProgress;
-global.logError = progress.logError;
-
-// List of temporary tables used in the calculation process
-const TEMP_TABLES = [
-    'temp_revenue_ranks',
-    'temp_sales_metrics',
-    'temp_purchase_metrics',
-    'temp_product_metrics',
-    'temp_vendor_metrics',
-    'temp_category_metrics',
-    'temp_brand_metrics',
-    'temp_forecast_dates',
-    'temp_daily_sales',
-    'temp_product_stats',
-    'temp_category_sales',
-    'temp_category_stats',
-    'temp_beginning_inventory',
-    'temp_monthly_inventory'
-];
-
-// Add cleanup function for temporary tables
-async function cleanupTemporaryTables(connection) {
-    try {
-        // Drop each temporary table if it exists
-        for (const table of TEMP_TABLES) {
-            await connection.query(`DROP TABLE IF EXISTS ${table}`);
-        }
-    } catch (err) {
-        console.error('Error cleaning up temporary tables:', err);
-    }
-}
-
-const { getConnection, closePool } = require('./metrics/utils/db');
-const calculateProductMetrics = require('./metrics/product-metrics');
-const calculateTimeAggregates = require('./metrics/time-aggregates');
-const calculateFinancialMetrics = require('./metrics/financial-metrics');
-const calculateVendorMetrics = require('./metrics/vendor-metrics');
-const calculateCategoryMetrics = require('./metrics/category-metrics');
-const calculateBrandMetrics = require('./metrics/brand-metrics');
-const calculateSalesForecasts = require('./metrics/sales-forecasts');
-
-// Add cancel handler
-let isCancelled = false;
-
-function cancelCalculation() {
-    isCancelled = true;
-    console.log('Calculation has been cancelled by user');
-    
-    // Force-terminate any query that's been running for more than 5 seconds
-    try {
-        const connection = getConnection();
-        connection.then(async (conn) => {
-            try {
-                // Identify and terminate long-running queries from our application
-                await conn.query(`
-                    SELECT pg_cancel_backend(pid) 
-                    FROM pg_stat_activity 
-                    WHERE query_start < now() - interval '5 seconds'
-                    AND application_name LIKE '%node%'
-                    AND query NOT LIKE '%pg_cancel_backend%'
-                `);
-                
-                // Clean up any temporary tables
-                await cleanupTemporaryTables(conn);
-                
-                // Release connection
-                conn.release();
-            } catch (err) {
-                console.error('Error during force cancellation:', err);
-                conn.release();
-            }
-        }).catch(err => {
-            console.error('Could not get connection for cancellation:', err);
-        });
-    } catch (err) {
-        console.error('Failed to terminate running queries:', err);
-    }
-    
-    return {
-        success: true,
-        message: 'Calculation has been cancelled'
-    };
-}
-
-// Handle SIGTERM signal for cancellation
-process.on('SIGTERM', cancelCalculation);
-
-// Update the main calculation function to use the new modular structure
-async function calculateMetrics() {
-    let connection;
-    const startTime = Date.now();
-    let processedProducts = 0;
-    let processedOrders = 0;
-    let processedPurchaseOrders = 0;
-    let totalProducts = 0;
-    let totalOrders = 0;
-    let totalPurchaseOrders = 0;
-    let calculateHistoryId;
-    
-    // Set a maximum execution time (30 minutes)
-    const MAX_EXECUTION_TIME = 30 * 60 * 1000;
-    const timeout = setTimeout(() => {
-        console.error(`Calculation timed out after ${MAX_EXECUTION_TIME/1000} seconds, forcing termination`);
-        // Call cancel and force exit
-        cancelCalculation();
-        process.exit(1);
-    }, MAX_EXECUTION_TIME);
-    
-    try {
-        // Clean up any previously running calculations
-        connection = await getConnection();
-        await connection.query(`
-            UPDATE calculate_history 
-            SET 
-                status = 'cancelled',
-                end_time = NOW(),
-                duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
-                error_message = 'Previous calculation was not completed properly'
-            WHERE status = 'running'
-        `);
-
-        // Get counts from all relevant tables
-        const [productCountResult, orderCountResult, poCountResult] = await Promise.all([
-            connection.query('SELECT COUNT(*) as total FROM products'),
-            connection.query('SELECT COUNT(*) as total FROM orders'),
-            connection.query('SELECT COUNT(*) as total FROM purchase_orders')
-        ]);
-
-        totalProducts = parseInt(productCountResult.rows[0].total);
-        totalOrders = parseInt(orderCountResult.rows[0].total);
-        totalPurchaseOrders = parseInt(poCountResult.rows[0].total);
-
-        // Create history record for this calculation
-        const historyResult = await connection.query(`
-            INSERT INTO calculate_history (
-                start_time,
-                status,
-                total_products,
-                total_orders,
-                total_purchase_orders,
-                additional_info
-            ) VALUES (
-                NOW(),
-                'running',
-                $1,
-                $2,
-                $3,
-                jsonb_build_object(
-                    'skip_product_metrics', ($4::int > 0),
-                    'skip_time_aggregates', ($5::int > 0),
-                    'skip_financial_metrics', ($6::int > 0),
-                    'skip_vendor_metrics', ($7::int > 0),
-                    'skip_category_metrics', ($8::int > 0),
-                    'skip_brand_metrics', ($9::int > 0),
-                    'skip_sales_forecasts', ($10::int > 0)
-                )
-            ) RETURNING id
-        `, [
-            totalProducts,
-            totalOrders,
-            totalPurchaseOrders,
-            SKIP_PRODUCT_METRICS,
-            SKIP_TIME_AGGREGATES,
-            SKIP_FINANCIAL_METRICS, 
-            SKIP_VENDOR_METRICS,
-            SKIP_CATEGORY_METRICS,
-            SKIP_BRAND_METRICS,
-            SKIP_SALES_FORECASTS
-        ]);
-        calculateHistoryId = historyResult.rows[0].id;
-
-        // Add debug logging for the progress functions
-        console.log('Debug - Progress functions:', {
-            formatElapsedTime: typeof global.formatElapsedTime,
-            estimateRemaining: typeof global.estimateRemaining,
-            calculateRate: typeof global.calculateRate,
-            startTime: startTime
-        });
-
-        try {
-            const elapsed = global.formatElapsedTime(startTime);
-            console.log('Debug - formatElapsedTime test successful:', elapsed);
-        } catch (err) {
-            console.error('Debug - Error testing formatElapsedTime:', err);
-            throw err;
-        }
-
-        // Release the connection before getting a new one
-        connection.release();
-        isCancelled = false;
-        connection = await getConnection();
-
-        try {
-            global.outputProgress({
-                status: 'running',
-                operation: 'Starting metrics calculation',
-                current: 0,
-                total: 100,
-                elapsed: '0s',
-                remaining: 'Calculating...',
-                rate: 0,
-                percentage: '0',
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            // Update progress periodically
-            const updateProgress = async (products = null, orders = null, purchaseOrders = null) => {
-                // Ensure all values are valid numbers or default to previous value
-                if (products !== null) processedProducts = Number(products) || processedProducts || 0;
-                if (orders !== null) processedOrders = Number(orders) || processedOrders || 0;
-                if (purchaseOrders !== null) processedPurchaseOrders = Number(purchaseOrders) || processedPurchaseOrders || 0;
-
-                // Ensure we never send NaN to the database
-                const safeProducts = Number(processedProducts) || 0;
-                const safeOrders = Number(processedOrders) || 0;
-                const safePurchaseOrders = Number(processedPurchaseOrders) || 0;
-
-                await connection.query(`
-                    UPDATE calculate_history 
-                    SET 
-                        processed_products = $1,
-                        processed_orders = $2,
-                        processed_purchase_orders = $3
-                    WHERE id = $4
-                `, [safeProducts, safeOrders, safePurchaseOrders, calculateHistoryId]);
-            };
-
-            // Helper function to ensure valid progress numbers
-            const ensureValidProgress = (current, total) => ({
-                current: Number(current) || 0,
-                total: Number(total) || 1,  // Default to 1 to avoid division by zero
-                percentage: (((Number(current) || 0) / (Number(total) || 1)) * 100).toFixed(1)
-            });
-
-            // Initial progress
-            const initialProgress = ensureValidProgress(0, totalProducts);
-            global.outputProgress({
-                status: 'running',
-                operation: 'Starting metrics calculation',
-                current: initialProgress.current,
-                total: initialProgress.total,
-                elapsed: '0s',
-                remaining: 'Calculating...',
-                rate: 0,
-                percentage: initialProgress.percentage,
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            if (!SKIP_PRODUCT_METRICS) {
-                const result = await calculateProductMetrics(startTime, totalProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Product metrics calculation failed');
-                }
-            } else {
-                console.log('Skipping product metrics calculation...');
-                processedProducts = Math.floor(totalProducts * 0.6);
-                await updateProgress(processedProducts);
-                global.outputProgress({
-                    status: 'running',
-                    operation: 'Skipping product metrics calculation',
-                    current: processedProducts,
-                    total: totalProducts,
-                    elapsed: global.formatElapsedTime(startTime),
-                    remaining: global.estimateRemaining(startTime, processedProducts, totalProducts),
-                    rate: global.calculateRate(startTime, processedProducts),
-                    percentage: '60',
-                    timing: {
-                        start_time: new Date(startTime).toISOString(),
-                        end_time: new Date().toISOString(),
-                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                    }
-                });
-            }
-
-            // Calculate time-based aggregates
-            if (!SKIP_TIME_AGGREGATES) {
-                const result = await calculateTimeAggregates(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Time aggregates calculation failed');
-                }
-            } else {
-                console.log('Skipping time aggregates calculation');
-            }
-
-            // Calculate financial metrics
-            if (!SKIP_FINANCIAL_METRICS) {
-                const result = await calculateFinancialMetrics(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Financial metrics calculation failed');
-                }
-            } else {
-                console.log('Skipping financial metrics calculation');
-            }
-            
-            // Calculate vendor metrics
-            if (!SKIP_VENDOR_METRICS) {
-                const result = await calculateVendorMetrics(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Vendor metrics calculation failed');
-                }
-            } else {
-                console.log('Skipping vendor metrics calculation');
-            }
-
-            // Calculate category metrics
-            if (!SKIP_CATEGORY_METRICS) {
-                const result = await calculateCategoryMetrics(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Category metrics calculation failed');
-                }
-            } else {
-                console.log('Skipping category metrics calculation');
-            }
-
-            // Calculate brand metrics
-            if (!SKIP_BRAND_METRICS) {
-                const result = await calculateBrandMetrics(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Brand metrics calculation failed');
-                }
-            } else {
-                console.log('Skipping brand metrics calculation');
-            }
-
-            // Calculate sales forecasts
-            if (!SKIP_SALES_FORECASTS) {
-                const result = await calculateSalesForecasts(startTime, totalProducts, processedProducts);
-                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
-                if (!result.success) {
-                    throw new Error('Sales forecasts calculation failed');
-                }
-            } else {
-                console.log('Skipping sales forecasts calculation');
-            }
-
-            // Final progress update with guaranteed valid numbers
-            const finalProgress = ensureValidProgress(totalProducts, totalProducts);
-            
-            // Final success message
-            outputProgress({
-                status: 'complete',
-                operation: 'Metrics calculation complete',
-                current: finalProgress.current,
-                total: finalProgress.total,
-                elapsed: global.formatElapsedTime(startTime),
-                remaining: '0s',
-                rate: global.calculateRate(startTime, finalProgress.current),
-                percentage: '100',
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            // Ensure all values are valid numbers before final update
-            const finalStats = {
-                processedProducts: Number(processedProducts) || 0,
-                processedOrders: Number(processedOrders) || 0,
-                processedPurchaseOrders: Number(processedPurchaseOrders) || 0
-            };
-
-            // Update history with completion
-            await connection.query(`
-                UPDATE calculate_history 
-                SET 
-                    end_time = NOW(),
-                    duration_seconds = $1,
-                    processed_products = $2,
-                    processed_orders = $3,
-                    processed_purchase_orders = $4,
-                    status = 'completed'
-                WHERE id = $5
-            `, [Math.round((Date.now() - startTime) / 1000), 
-                finalStats.processedProducts,
-                finalStats.processedOrders, 
-                finalStats.processedPurchaseOrders,
-                calculateHistoryId]);
-
-            // Clear progress file on successful completion
-            global.clearProgress();
-
-            return {
-                success: true,
-                message: 'Calculation completed successfully',
-                duration: Math.round((Date.now() - startTime) / 1000)
-            };
-        } catch (error) {
-            const endTime = Date.now();
-            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
-
-            // Update history with error
-            await connection.query(`
-                UPDATE calculate_history 
-                SET 
-                    end_time = NOW(),
-                    duration_seconds = $1,
-                    processed_products = $2,
-                    processed_orders = $3,
-                    processed_purchase_orders = $4,
-                    status = $5,
-                    error_message = $6
-                WHERE id = $7
-            `, [
-                totalElapsedSeconds,
-                processedProducts || 0,  // Ensure we have a valid number
-                processedOrders || 0,    // Ensure we have a valid number
-                processedPurchaseOrders || 0,  // Ensure we have a valid number
-                isCancelled ? 'cancelled' : 'failed',
-                error.message,
-                calculateHistoryId
-            ]);
-
-            if (isCancelled) {
-                global.outputProgress({
-                    status: 'cancelled',
-                    operation: 'Calculation cancelled',
-                    current: processedProducts,
-                    total: totalProducts || 0,
-                    elapsed: global.formatElapsedTime(startTime),
-                    remaining: null,
-                    rate: global.calculateRate(startTime, processedProducts),
-                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
-                    timing: {
-                        start_time: new Date(startTime).toISOString(),
-                        end_time: new Date().toISOString(),
-                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                    }
-                });
-            } else {
-                global.outputProgress({
-                    status: 'error',
-                    operation: 'Error: ' + error.message,
-                    current: processedProducts,
-                    total: totalProducts || 0,
-                    elapsed: global.formatElapsedTime(startTime),
-                    remaining: null,
-                    rate: global.calculateRate(startTime, processedProducts),
-                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
-                    timing: {
-                        start_time: new Date(startTime).toISOString(),
-                        end_time: new Date().toISOString(),
-                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                    }
-                });
-            }
-            throw error;
-        } finally {
-            // Clear the timeout to prevent forced termination
-            clearTimeout(timeout);
-            
-            // Always clean up and release connection
-            if (connection) {
-                try {
-                    await cleanupTemporaryTables(connection);
-                    connection.release();
-                } catch (err) {
-                    console.error('Error in final cleanup:', err);
-                }
-            }
-        }
-    } catch (error) {
-        console.error('Error in metrics calculation', error);
-        
-        try {
-            if (connection) {
-                await connection.query(`
-                    UPDATE calculate_history 
-                    SET 
-                        status = 'failed',
-                        end_time = NOW(),
-                        duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
-                        error_message = $1
-                    WHERE id = $2
-                `, [error.message.substring(0, 500), calculateHistoryId]);
-            }
-        } catch (updateError) {
-            console.error('Error updating calculation history:', updateError);
-        }
-        
-        throw error;
-    }
-}
-
-// Export as a module with all necessary functions
-module.exports = {
-    calculateMetrics,
-    cancelCalculation,
-    getProgress: global.getProgress
-};
-
-// Run directly if called from command line
-if (require.main === module) {
-    calculateMetrics().catch(error => {
-        if (!error.message.includes('Operation cancelled')) {
-            console.error('Error:', error);
-        }
-        process.exit(1);
-    });
-} 

inventory-server/old/config-schema.sql

@@ -1,242 +0,0 @@
--- -- Configuration tables schema
-
-
-
--- -- Stock threshold configurations
--- CREATE TABLE stock_thresholds (
---     id INTEGER NOT NULL,
---     category_id BIGINT,  -- NULL means default/global threshold
---     vendor VARCHAR(100), -- NULL means applies to all vendors
---     critical_days INTEGER NOT NULL DEFAULT 7,
---     reorder_days INTEGER NOT NULL DEFAULT 14,
---     overstock_days INTEGER NOT NULL DEFAULT 90,
---     low_stock_threshold INTEGER NOT NULL DEFAULT 5,
---     min_reorder_quantity INTEGER NOT NULL DEFAULT 1,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (id),
---     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
---     UNIQUE (category_id, vendor)
--- );
-
--- CREATE TRIGGER update_stock_thresholds_updated
---     BEFORE UPDATE ON stock_thresholds
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- CREATE INDEX idx_st_metrics ON stock_thresholds(category_id, vendor);
-
--- -- Lead time threshold configurations
--- CREATE TABLE lead_time_thresholds (
---     id INTEGER NOT NULL,
---     category_id BIGINT,  -- NULL means default/global threshold
---     vendor VARCHAR(100), -- NULL means applies to all vendors
---     target_days INTEGER NOT NULL DEFAULT 14,
---     warning_days INTEGER NOT NULL DEFAULT 21,
---     critical_days INTEGER NOT NULL DEFAULT 30,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (id),
---     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
---     UNIQUE (category_id, vendor)
--- );
-
--- CREATE TRIGGER update_lead_time_thresholds_updated
---     BEFORE UPDATE ON lead_time_thresholds
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- -- Sales velocity window configurations
--- CREATE TABLE sales_velocity_config (
---     id INTEGER NOT NULL,
---     category_id BIGINT,  -- NULL means default/global threshold
---     vendor VARCHAR(100), -- NULL means applies to all vendors
---     daily_window_days INTEGER NOT NULL DEFAULT 30,
---     weekly_window_days INTEGER NOT NULL DEFAULT 7,
---     monthly_window_days INTEGER NOT NULL DEFAULT 90,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (id),
---     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
---     UNIQUE (category_id, vendor)
--- );
-
--- CREATE TRIGGER update_sales_velocity_config_updated
---     BEFORE UPDATE ON sales_velocity_config
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- CREATE INDEX idx_sv_metrics ON sales_velocity_config(category_id, vendor);
-
--- -- ABC Classification configurations
--- CREATE TABLE abc_classification_config (
---     id INTEGER NOT NULL PRIMARY KEY,
---     a_threshold DECIMAL(5,2) NOT NULL DEFAULT 20.0,
---     b_threshold DECIMAL(5,2) NOT NULL DEFAULT 50.0,
---     classification_period_days INTEGER NOT NULL DEFAULT 90,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
--- );
-
--- CREATE TRIGGER update_abc_classification_config_updated
---     BEFORE UPDATE ON abc_classification_config
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- -- Safety stock configurations
--- CREATE TABLE safety_stock_config (
---     id INTEGER NOT NULL,
---     category_id BIGINT,  -- NULL means default/global threshold
---     vendor VARCHAR(100), -- NULL means applies to all vendors
---     coverage_days INTEGER NOT NULL DEFAULT 14,
---     service_level DECIMAL(5,2) NOT NULL DEFAULT 95.0,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (id),
---     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
---     UNIQUE (category_id, vendor)
--- );
-
--- CREATE TRIGGER update_safety_stock_config_updated
---     BEFORE UPDATE ON safety_stock_config
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- CREATE INDEX idx_ss_metrics ON safety_stock_config(category_id, vendor);
-
--- -- Turnover rate configurations
--- CREATE TABLE turnover_config (
---     id INTEGER NOT NULL,
---     category_id BIGINT,  -- NULL means default/global threshold
---     vendor VARCHAR(100), -- NULL means applies to all vendors
---     calculation_period_days INTEGER NOT NULL DEFAULT 30,
---     target_rate DECIMAL(10,2) NOT NULL DEFAULT 1.0,
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (id),
---     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
---     UNIQUE (category_id, vendor)
--- );
-
--- CREATE TRIGGER update_turnover_config_updated
---     BEFORE UPDATE ON turnover_config
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- -- Create table for sales seasonality factors
--- CREATE TABLE sales_seasonality (
---     month INTEGER NOT NULL,
---     seasonality_factor DECIMAL(5,3) DEFAULT 0,
---     last_updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
---     PRIMARY KEY (month),
---     CONSTRAINT month_range CHECK (month BETWEEN 1 AND 12),
---     CONSTRAINT seasonality_range CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
--- );
-
--- CREATE TRIGGER update_sales_seasonality_updated
---     BEFORE UPDATE ON sales_seasonality
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- -- Create table for financial calculation parameters
--- CREATE TABLE financial_calc_config (
---     id INTEGER NOT NULL PRIMARY KEY,
---     order_cost DECIMAL(10,2) NOT NULL DEFAULT 25.00, -- The fixed cost per purchase order (used in EOQ)
---     holding_rate DECIMAL(10,4) NOT NULL DEFAULT 0.25, -- The annual inventory holding cost as a percentage of unit cost (used in EOQ)
---     service_level_z_score DECIMAL(10,4) NOT NULL DEFAULT 1.96, -- Z-score for ~95% service level (used in Safety Stock)
---     min_reorder_qty INTEGER NOT NULL DEFAULT 1, -- Minimum reorder quantity 
---     default_reorder_qty INTEGER NOT NULL DEFAULT 5, -- Default reorder quantity when sales data is insufficient
---     default_safety_stock INTEGER NOT NULL DEFAULT 5, -- Default safety stock when sales data is insufficient
---     created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
---     updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
--- );
-
--- CREATE TRIGGER update_financial_calc_config_updated
---     BEFORE UPDATE ON financial_calc_config
---     FOR EACH ROW
---     EXECUTE FUNCTION update_updated_at_column();
-
--- -- Insert default global thresholds
--- INSERT INTO stock_thresholds (id, category_id, vendor, critical_days, reorder_days, overstock_days)
--- VALUES (1, NULL, NULL, 7, 14, 90)
--- ON CONFLICT (id) DO UPDATE SET
---     critical_days = EXCLUDED.critical_days,
---     reorder_days = EXCLUDED.reorder_days,
---     overstock_days = EXCLUDED.overstock_days;
-
--- INSERT INTO lead_time_thresholds (id, category_id, vendor, target_days, warning_days, critical_days)
--- VALUES (1, NULL, NULL, 14, 21, 30)
--- ON CONFLICT (id) DO UPDATE SET
---     target_days = EXCLUDED.target_days,
---     warning_days = EXCLUDED.warning_days,
---     critical_days = EXCLUDED.critical_days;
-
--- INSERT INTO sales_velocity_config (id, category_id, vendor, daily_window_days, weekly_window_days, monthly_window_days)
--- VALUES (1, NULL, NULL, 30, 7, 90)
--- ON CONFLICT (id) DO UPDATE SET
---     daily_window_days = EXCLUDED.daily_window_days,
---     weekly_window_days = EXCLUDED.weekly_window_days,
---     monthly_window_days = EXCLUDED.monthly_window_days;
-
--- INSERT INTO abc_classification_config (id, a_threshold, b_threshold, classification_period_days)
--- VALUES (1, 20.0, 50.0, 90)
--- ON CONFLICT (id) DO UPDATE SET
---     a_threshold = EXCLUDED.a_threshold,
---     b_threshold = EXCLUDED.b_threshold,
---     classification_period_days = EXCLUDED.classification_period_days;
-
--- INSERT INTO safety_stock_config (id, category_id, vendor, coverage_days, service_level)
--- VALUES (1, NULL, NULL, 14, 95.0)
--- ON CONFLICT (id) DO UPDATE SET
---     coverage_days = EXCLUDED.coverage_days,
---     service_level = EXCLUDED.service_level;
-
--- INSERT INTO turnover_config (id, category_id, vendor, calculation_period_days, target_rate)
--- VALUES (1, NULL, NULL, 30, 1.0)
--- ON CONFLICT (id) DO UPDATE SET
---     calculation_period_days = EXCLUDED.calculation_period_days,
---     target_rate = EXCLUDED.target_rate;
-
--- -- Insert default seasonality factors (neutral)
--- INSERT INTO sales_seasonality (month, seasonality_factor)
--- VALUES 
---     (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
---     (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
--- ON CONFLICT (month) DO UPDATE SET
---     last_updated = CURRENT_TIMESTAMP;
-
--- -- Insert default values
--- INSERT INTO financial_calc_config (id, order_cost, holding_rate, service_level_z_score, min_reorder_qty, default_reorder_qty, default_safety_stock)
--- VALUES (1, 25.00, 0.25, 1.96, 1, 5, 5)
--- ON CONFLICT (id) DO UPDATE SET
---     order_cost = EXCLUDED.order_cost,
---     holding_rate = EXCLUDED.holding_rate,
---     service_level_z_score = EXCLUDED.service_level_z_score,
---     min_reorder_qty = EXCLUDED.min_reorder_qty,
---     default_reorder_qty = EXCLUDED.default_reorder_qty,
---     default_safety_stock = EXCLUDED.default_safety_stock;
-
--- -- View to show thresholds with category names
--- CREATE OR REPLACE VIEW stock_thresholds_view AS
--- SELECT 
---     st.*,
---     c.name as category_name,
---     CASE 
---         WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 'Global Default'
---         WHEN st.category_id IS NULL THEN 'Vendor: ' || st.vendor
---         WHEN st.vendor IS NULL THEN 'Category: ' || c.name
---         ELSE 'Category: ' || c.name || ' / Vendor: ' || st.vendor
---     END as threshold_scope
--- FROM 
---     stock_thresholds st
--- LEFT JOIN 
---     categories c ON st.category_id = c.cat_id
--- ORDER BY 
---     CASE 
---         WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 1
---         WHEN st.category_id IS NULL THEN 2
---         WHEN st.vendor IS NULL THEN 3
---         ELSE 4
---     END,
---     c.name,
---     st.vendor;

inventory-server/old/historical-data.js

@@ -1,961 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../scripts/metrics-new/utils/progress');
-const fs = require('fs');
-const path = require('path');
-const { pipeline } = require('stream');
-const { promisify } = require('util');
-
-// Configuration constants to control which tables get imported
-const IMPORT_PRODUCT_CURRENT_PRICES = false;
-const IMPORT_DAILY_INVENTORY = false; 
-const IMPORT_PRODUCT_STAT_HISTORY = true;
-
-// For product stat history, limit to more recent data for faster initial import
-const USE_RECENT_MONTHS = 12; // Just use the most recent months for product_stat_history
-
-/**
- * Validates a date from MySQL before inserting it into PostgreSQL
- * @param {string|Date|null} mysqlDate - Date string or object from MySQL
- * @returns {string|null} Valid date string or null if invalid
- */
-function validateDate(mysqlDate) {
-  // Handle null, undefined, or empty values
-  if (!mysqlDate) {
-    return null;
-  }
-  
-  // Convert to string if it's not already
-  const dateStr = String(mysqlDate);
-  
-  // Handle MySQL zero dates and empty values
-  if (dateStr === '0000-00-00' || 
-      dateStr === '0000-00-00 00:00:00' ||
-      dateStr.indexOf('0000-00-00') !== -1 ||
-      dateStr === '') {
-    return null;
-  }
-  
-  // Check if the date is valid
-  const date = new Date(mysqlDate);
-  
-  // If the date is invalid or suspiciously old (pre-1970), return null
-  if (isNaN(date.getTime()) || date.getFullYear() < 1970) {
-    return null;
-  }
-  
-  return mysqlDate;
-}
-
-/**
- * Imports historical data from MySQL to PostgreSQL
- */
-async function importHistoricalData(
-  prodConnection, 
-  localConnection, 
-  options = {}
-) {
-  const {
-    incrementalUpdate = true,
-    oneYearAgo = new Date(new Date().setFullYear(new Date().getFullYear() - 1))
-  } = options;
-
-  const oneYearAgoStr = oneYearAgo.toISOString().split('T')[0];
-  const startTime = Date.now();
-  
-  // Use larger batch sizes to improve performance
-  const BATCH_SIZE = 5000;  // For fetching from small tables
-  const INSERT_BATCH_SIZE = 500; // For inserting to small tables
-  const LARGE_BATCH_SIZE = 10000; // For fetching from large tables
-  const LARGE_INSERT_BATCH_SIZE = 1000; // For inserting to large tables
-  
-  // Calculate date for recent data
-  const recentDateStr = new Date(
-    new Date().setMonth(new Date().getMonth() - USE_RECENT_MONTHS)
-  ).toISOString().split('T')[0];
-
-  console.log(`Starting import with:
-    - One year ago date: ${oneYearAgoStr}
-    - Recent months date: ${recentDateStr} (for product_stat_history)
-    - Incremental update: ${incrementalUpdate}
-    - Standard batch size: ${BATCH_SIZE}
-    - Standard insert batch size: ${INSERT_BATCH_SIZE}
-    - Large table batch size: ${LARGE_BATCH_SIZE}
-    - Large table insert batch size: ${LARGE_INSERT_BATCH_SIZE}
-    - Import product_current_prices: ${IMPORT_PRODUCT_CURRENT_PRICES}
-    - Import daily_inventory: ${IMPORT_DAILY_INVENTORY}
-    - Import product_stat_history: ${IMPORT_PRODUCT_STAT_HISTORY}`);
-
-  try {
-    // Get last sync time for incremental updates
-    const lastSyncTimes = {};
-    
-    if (incrementalUpdate) {
-      try {
-        const syncResult = await localConnection.query(`
-          SELECT table_name, last_sync_timestamp
-          FROM sync_status
-          WHERE table_name IN (
-            'imported_product_current_prices', 
-            'imported_daily_inventory', 
-            'imported_product_stat_history'
-          )
-        `);
-        
-        // Add check for rows existence and type
-        if (syncResult && Array.isArray(syncResult.rows)) {
-          for (const row of syncResult.rows) {
-            lastSyncTimes[row.table_name] = row.last_sync_timestamp;
-            console.log(`Last sync time for ${row.table_name}: ${row.last_sync_timestamp}`);
-          }
-        } else {
-          console.warn('Sync status query did not return expected rows. Proceeding without last sync times.');
-        }
-      } catch (error) {
-        console.error('Error fetching sync status:', error);
-      }
-    }
-
-    // Determine how many tables will be imported
-    const tablesCount = [
-      IMPORT_PRODUCT_CURRENT_PRICES,
-      IMPORT_DAILY_INVENTORY,
-      IMPORT_PRODUCT_STAT_HISTORY
-    ].filter(Boolean).length;
-
-    // Run all imports sequentially for better reliability
-    console.log(`Starting sequential imports for ${tablesCount} tables...`);
-    outputProgress({
-      status: "running",
-      operation: "Historical data import",
-      message: `Starting sequential imports for ${tablesCount} tables...`,
-      current: 0,
-      total: tablesCount,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    let progressCount = 0;
-    let productCurrentPricesResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
-    let dailyInventoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
-    let productStatHistoryResult = { recordsAdded: 0, recordsUpdated: 0, totalProcessed: 0, errors: [] };
-
-    // Import product current prices
-    if (IMPORT_PRODUCT_CURRENT_PRICES) {
-      console.log('Importing product current prices...');
-      productCurrentPricesResult = await importProductCurrentPrices(
-        prodConnection, 
-        localConnection, 
-        oneYearAgoStr, 
-        lastSyncTimes['imported_product_current_prices'], 
-        BATCH_SIZE,
-        INSERT_BATCH_SIZE,
-        incrementalUpdate,
-        startTime
-      );
-      progressCount++;
-      
-      outputProgress({
-        status: "running",
-        operation: "Historical data import",
-        message: `Completed import ${progressCount} of ${tablesCount}`,
-        current: progressCount,
-        total: tablesCount,
-        elapsed: formatElapsedTime(startTime)
-      });
-    }
-    
-    // Import daily inventory
-    if (IMPORT_DAILY_INVENTORY) {
-      console.log('Importing daily inventory...');
-      dailyInventoryResult = await importDailyInventory(
-        prodConnection, 
-        localConnection, 
-        oneYearAgoStr, 
-        lastSyncTimes['imported_daily_inventory'], 
-        BATCH_SIZE,
-        INSERT_BATCH_SIZE,
-        incrementalUpdate,
-        startTime
-      );
-      progressCount++;
-      
-      outputProgress({
-        status: "running",
-        operation: "Historical data import",
-        message: `Completed import ${progressCount} of ${tablesCount}`,
-        current: progressCount,
-        total: tablesCount,
-        elapsed: formatElapsedTime(startTime)
-      });
-    }
-    
-    // Import product stat history - using optimized approach
-    if (IMPORT_PRODUCT_STAT_HISTORY) {
-      console.log('Importing product stat history...');
-      productStatHistoryResult = await importProductStatHistory(
-        prodConnection, 
-        localConnection, 
-        recentDateStr, // Use more recent date for this massive table 
-        lastSyncTimes['imported_product_stat_history'], 
-        LARGE_BATCH_SIZE,
-        LARGE_INSERT_BATCH_SIZE,
-        incrementalUpdate,
-        startTime,
-        USE_RECENT_MONTHS // Pass the recent months constant
-      );
-      progressCount++;
-      
-      outputProgress({
-        status: "running",
-        operation: "Historical data import",
-        message: `Completed import ${progressCount} of ${tablesCount}`,
-        current: progressCount,
-        total: tablesCount,
-        elapsed: formatElapsedTime(startTime)
-      });
-    }
-
-    // Aggregate results
-    const totalRecordsAdded = 
-      productCurrentPricesResult.recordsAdded + 
-      dailyInventoryResult.recordsAdded + 
-      productStatHistoryResult.recordsAdded;
-    
-    const totalRecordsUpdated = 
-      productCurrentPricesResult.recordsUpdated + 
-      dailyInventoryResult.recordsUpdated + 
-      productStatHistoryResult.recordsUpdated;
-    
-    const totalProcessed = 
-      productCurrentPricesResult.totalProcessed + 
-      dailyInventoryResult.totalProcessed + 
-      productStatHistoryResult.totalProcessed;
-    
-    const allErrors = [
-      ...productCurrentPricesResult.errors,
-      ...dailyInventoryResult.errors,
-      ...productStatHistoryResult.errors
-    ];
-
-    // Log import summary
-    console.log(`
-Historical data import complete:
--------------------------------
-Records added:   ${totalRecordsAdded}
-Records updated: ${totalRecordsUpdated}
-Total processed: ${totalProcessed}
-Errors:          ${allErrors.length}
-Time taken:      ${formatElapsedTime(startTime)}
-    `);
-
-    // Final progress update
-    outputProgress({
-      status: "complete",
-      operation: "Historical data import",
-      message: `Import complete. Added: ${totalRecordsAdded}, Updated: ${totalRecordsUpdated}, Errors: ${allErrors.length}`,
-      current: tablesCount,
-      total: tablesCount,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    // Log any errors
-    if (allErrors.length > 0) {
-      console.log('Errors encountered during import:');
-      console.log(JSON.stringify(allErrors, null, 2));
-    }
-
-    // Calculate duration
-    const endTime = Date.now();
-    const durationSeconds = Math.round((endTime - startTime) / 1000);
-    const finalStatus = allErrors.length === 0 ? 'complete' : 'failed';
-    const errorMessage = allErrors.length > 0 ? JSON.stringify(allErrors) : null;
-
-    // Update import history
-    await localConnection.query(`
-      INSERT INTO import_history (
-        table_name, 
-        end_time, 
-        duration_seconds, 
-        records_added, 
-        records_updated, 
-        is_incremental, 
-        status, 
-        error_message,
-        additional_info
-      )
-      VALUES ($1, NOW(), $2, $3, $4, $5, $6, $7, $8)
-    `, [
-      'historical_data_combined',
-      durationSeconds,
-      totalRecordsAdded,
-      totalRecordsUpdated,
-      incrementalUpdate,
-      finalStatus,
-      errorMessage,
-      JSON.stringify({ 
-        totalProcessed,
-        tablesImported: {
-          imported_product_current_prices: IMPORT_PRODUCT_CURRENT_PRICES,
-          imported_daily_inventory: IMPORT_DAILY_INVENTORY,
-          imported_product_stat_history: IMPORT_PRODUCT_STAT_HISTORY
-        }
-      })
-    ]);
-
-    // Return summary
-    return {
-      recordsAdded: totalRecordsAdded,
-      recordsUpdated: totalRecordsUpdated,
-      totalProcessed,
-      errors: allErrors,
-      timeTaken: formatElapsedTime(startTime)
-    };
-  } catch (error) {
-    console.error('Error importing historical data:', error);
-    
-    // Final progress update on error
-    outputProgress({
-      status: "failed",
-      operation: "Historical data import",
-      message: `Import failed: ${error.message}`,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    throw error;
-  }
-}
-
-/**
- * Imports product_current_prices data from MySQL to PostgreSQL
- */
-async function importProductCurrentPrices(
-  prodConnection, 
-  localConnection, 
-  oneYearAgoStr, 
-  lastSyncTime, 
-  batchSize,
-  insertBatchSize,
-  incrementalUpdate,
-  startTime
-) {
-  let recordsAdded = 0;
-  let recordsUpdated = 0;
-  let totalProcessed = 0;
-  let errors = [];
-  let offset = 0;
-  let allProcessed = false;
-
-  try {
-    // Get total count for progress reporting
-    const [countResult] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM product_current_prices
-      WHERE (date_active >= ? OR date_deactive >= ?)
-        ${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
-    `, [oneYearAgoStr, oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
-
-    const totalCount = countResult[0].total;
-
-    outputProgress({
-      status: "running",
-      operation: "Historical data import - Product Current Prices",
-      message: `Found ${totalCount} records to process`,
-      current: 0,
-      total: totalCount,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    // Process in batches for better performance
-    while (!allProcessed) {
-      try {
-        // Fetch batch from production
-        const [rows] = await prodConnection.query(`
-          SELECT
-            price_id,
-            pid,
-            qty_buy,
-            is_min_qty_buy,
-            price_each,
-            qty_limit,
-            no_promo,
-            checkout_offer,
-            active,
-            date_active,
-            date_deactive
-          FROM product_current_prices
-          WHERE (date_active >= ? OR date_deactive >= ?)
-            ${incrementalUpdate && lastSyncTime ? `AND date_deactive > ?` : ''}
-          ORDER BY price_id
-          LIMIT ? OFFSET ?
-        `, [
-          oneYearAgoStr, 
-          oneYearAgoStr, 
-          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
-          batchSize,
-          offset
-        ]);
-
-        if (rows.length === 0) {
-          allProcessed = true;
-          break;
-        }
-
-        // Process rows in smaller batches for better performance
-        for (let i = 0; i < rows.length; i += insertBatchSize) {
-          const batch = rows.slice(i, i + insertBatchSize);
-          
-          if (batch.length === 0) continue;
-          
-          try {
-            // Build parameterized query to handle NULL values properly
-            const values = [];
-            const placeholders = [];
-            let placeholderIndex = 1;
-            
-            for (const row of batch) {
-              const rowPlaceholders = [
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`
-              ];
-              
-              placeholders.push(`(${rowPlaceholders.join(', ')})`);
-              
-              values.push(
-                row.price_id,
-                row.pid,
-                row.qty_buy,
-                row.is_min_qty_buy ? true : false,
-                row.price_each,
-                row.qty_limit, // PostgreSQL will handle null values properly
-                row.no_promo ? true : false,
-                row.checkout_offer ? true : false,
-                row.active ? true : false,
-                validateDate(row.date_active),
-                validateDate(row.date_deactive)
-              );
-            }
-            
-            // Execute batch insert
-            const result = await localConnection.query(`
-              WITH ins AS (
-                INSERT INTO imported_product_current_prices (
-                  price_id, pid, qty_buy, is_min_qty_buy, price_each, qty_limit, 
-                  no_promo, checkout_offer, active, date_active, date_deactive
-                )
-                VALUES ${placeholders.join(',\n')}
-                ON CONFLICT (price_id) DO UPDATE SET
-                  pid = EXCLUDED.pid,
-                  qty_buy = EXCLUDED.qty_buy,
-                  is_min_qty_buy = EXCLUDED.is_min_qty_buy,
-                  price_each = EXCLUDED.price_each,
-                  qty_limit = EXCLUDED.qty_limit,
-                  no_promo = EXCLUDED.no_promo,
-                  checkout_offer = EXCLUDED.checkout_offer,
-                  active = EXCLUDED.active,
-                  date_active = EXCLUDED.date_active,
-                  date_deactive = EXCLUDED.date_deactive,
-                  updated = CURRENT_TIMESTAMP
-                RETURNING (xmax = 0) AS inserted
-              )
-              SELECT 
-                COUNT(*) FILTER (WHERE inserted) AS inserted_count, 
-                COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
-              FROM ins
-            `, values);
-            
-            // Safely update counts based on the result
-            if (result && result.rows && result.rows.length > 0) {
-              const insertedCount = parseInt(result.rows[0].inserted_count || 0);
-              const updatedCount = parseInt(result.rows[0].updated_count || 0);
-              
-              recordsAdded += insertedCount;
-              recordsUpdated += updatedCount;
-            }
-          } catch (error) {
-            console.error(`Error in batch import of product_current_prices at offset ${i}:`, error);
-            errors.push({
-              table: 'imported_product_current_prices',
-              batchOffset: i,
-              batchSize: batch.length,
-              error: error.message
-            });
-          }
-        }
-
-        totalProcessed += rows.length;
-        offset += rows.length;
-
-        // Update progress
-        outputProgress({
-          status: "running",
-          operation: "Historical data import - Product Current Prices",
-          message: `Processed ${totalProcessed} of ${totalCount} records`,
-          current: totalProcessed,
-          total: totalCount,
-          elapsed: formatElapsedTime(startTime),
-          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
-          rate: calculateRate(startTime, totalProcessed)
-        });
-      } catch (error) {
-        console.error('Error in batch import of product_current_prices:', error);
-        errors.push({
-          table: 'imported_product_current_prices',
-          error: error.message,
-          offset: offset,
-          batchSize: batchSize
-        });
-        
-        // Try to continue with next batch
-        offset += batchSize;
-      }
-    }
-
-    // Update sync status
-    await localConnection.query(`
-      INSERT INTO sync_status (table_name, last_sync_timestamp)
-      VALUES ('imported_product_current_prices', NOW())
-      ON CONFLICT (table_name) DO UPDATE SET
-        last_sync_timestamp = NOW()
-    `);
-
-    return { recordsAdded, recordsUpdated, totalProcessed, errors };
-  } catch (error) {
-    console.error('Error in product current prices import:', error);
-    return { 
-      recordsAdded, 
-      recordsUpdated, 
-      totalProcessed, 
-      errors: [...errors, { 
-        table: 'imported_product_current_prices',
-        error: error.message
-      }] 
-    };
-  }
-}
-
-/**
- * Imports daily_inventory data from MySQL to PostgreSQL
- */
-async function importDailyInventory(
-  prodConnection, 
-  localConnection, 
-  oneYearAgoStr, 
-  lastSyncTime, 
-  batchSize,
-  insertBatchSize,
-  incrementalUpdate,
-  startTime
-) {
-  let recordsAdded = 0;
-  let recordsUpdated = 0;
-  let totalProcessed = 0;
-  let errors = [];
-  let offset = 0;
-  let allProcessed = false;
-
-  try {
-    // Get total count for progress reporting
-    const [countResult] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM daily_inventory
-      WHERE date >= ?
-        ${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
-    `, [oneYearAgoStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
-
-    const totalCount = countResult[0].total;
-
-    outputProgress({
-      status: "running",
-      operation: "Historical data import - Daily Inventory",
-      message: `Found ${totalCount} records to process`,
-      current: 0,
-      total: totalCount,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    // Process in batches for better performance
-    while (!allProcessed) {
-      try {
-        // Fetch batch from production
-        const [rows] = await prodConnection.query(`
-          SELECT
-            date,
-            pid,
-            amountsold,
-            times_sold,
-            qtyreceived,
-            price,
-            costeach,
-            stamp
-          FROM daily_inventory
-          WHERE date >= ?
-            ${incrementalUpdate && lastSyncTime ? `AND stamp > ?` : ''}
-          ORDER BY date, pid
-          LIMIT ? OFFSET ?
-        `, [
-          oneYearAgoStr, 
-          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
-          batchSize,
-          offset
-        ]);
-
-        if (rows.length === 0) {
-          allProcessed = true;
-          break;
-        }
-
-        // Process rows in smaller batches for better performance
-        for (let i = 0; i < rows.length; i += insertBatchSize) {
-          const batch = rows.slice(i, i + insertBatchSize);
-          
-          if (batch.length === 0) continue;
-          
-          try {
-            // Build parameterized query to handle NULL values properly
-            const values = [];
-            const placeholders = [];
-            let placeholderIndex = 1;
-            
-            for (const row of batch) {
-              const rowPlaceholders = [
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`
-              ];
-              
-              placeholders.push(`(${rowPlaceholders.join(', ')})`);
-              
-              values.push(
-                validateDate(row.date),
-                row.pid,
-                row.amountsold || 0,
-                row.times_sold || 0,
-                row.qtyreceived || 0,
-                row.price || 0,
-                row.costeach || 0,
-                validateDate(row.stamp)
-              );
-            }
-            
-            // Execute batch insert
-            const result = await localConnection.query(`
-              WITH ins AS (
-                INSERT INTO imported_daily_inventory (
-                  date, pid, amountsold, times_sold, qtyreceived, price, costeach, stamp
-                )
-                VALUES ${placeholders.join(',\n')}
-                ON CONFLICT (date, pid) DO UPDATE SET
-                  amountsold = EXCLUDED.amountsold,
-                  times_sold = EXCLUDED.times_sold,
-                  qtyreceived = EXCLUDED.qtyreceived,
-                  price = EXCLUDED.price,
-                  costeach = EXCLUDED.costeach,
-                  stamp = EXCLUDED.stamp,
-                  updated = CURRENT_TIMESTAMP
-                RETURNING (xmax = 0) AS inserted
-              )
-              SELECT 
-                COUNT(*) FILTER (WHERE inserted) AS inserted_count, 
-                COUNT(*) FILTER (WHERE NOT inserted) AS updated_count
-              FROM ins
-            `, values);
-            
-            // Safely update counts based on the result
-            if (result && result.rows && result.rows.length > 0) {
-              const insertedCount = parseInt(result.rows[0].inserted_count || 0);
-              const updatedCount = parseInt(result.rows[0].updated_count || 0);
-              
-              recordsAdded += insertedCount;
-              recordsUpdated += updatedCount;
-            }
-          } catch (error) {
-            console.error(`Error in batch import of daily_inventory at offset ${i}:`, error);
-            errors.push({
-              table: 'imported_daily_inventory',
-              batchOffset: i,
-              batchSize: batch.length,
-              error: error.message
-            });
-          }
-        }
-
-        totalProcessed += rows.length;
-        offset += rows.length;
-
-        // Update progress
-        outputProgress({
-          status: "running",
-          operation: "Historical data import - Daily Inventory",
-          message: `Processed ${totalProcessed} of ${totalCount} records`,
-          current: totalProcessed,
-          total: totalCount,
-          elapsed: formatElapsedTime(startTime),
-          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
-          rate: calculateRate(startTime, totalProcessed)
-        });
-      } catch (error) {
-        console.error('Error in batch import of daily_inventory:', error);
-        errors.push({
-          table: 'imported_daily_inventory',
-          error: error.message,
-          offset: offset,
-          batchSize: batchSize
-        });
-        
-        // Try to continue with next batch
-        offset += batchSize;
-      }
-    }
-
-    // Update sync status
-    await localConnection.query(`
-      INSERT INTO sync_status (table_name, last_sync_timestamp)
-      VALUES ('imported_daily_inventory', NOW())
-      ON CONFLICT (table_name) DO UPDATE SET
-        last_sync_timestamp = NOW()
-    `);
-
-    return { recordsAdded, recordsUpdated, totalProcessed, errors };
-  } catch (error) {
-    console.error('Error in daily inventory import:', error);
-    return { 
-      recordsAdded, 
-      recordsUpdated, 
-      totalProcessed, 
-      errors: [...errors, { 
-        table: 'imported_daily_inventory',
-        error: error.message
-      }] 
-    };
-  }
-}
-
-/**
- * Imports product_stat_history data from MySQL to PostgreSQL
- * Using fast direct inserts without conflict checking
- */
-async function importProductStatHistory(
-  prodConnection, 
-  localConnection, 
-  recentDateStr, // Use more recent date instead of one year ago
-  lastSyncTime, 
-  batchSize,
-  insertBatchSize,
-  incrementalUpdate,
-  startTime,
-  recentMonths // Add parameter for recent months
-) {
-  let recordsAdded = 0;
-  let recordsUpdated = 0;
-  let totalProcessed = 0;
-  let errors = [];
-  let offset = 0;
-  let allProcessed = false;
-  let lastRateCheck = Date.now();
-  let lastProcessed = 0;
-
-  try {
-    // Get total count for progress reporting
-    const [countResult] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM product_stat_history
-      WHERE date >= ?
-        ${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
-    `, [recentDateStr, ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : [])]);
-
-    const totalCount = countResult[0].total;
-    console.log(`Found ${totalCount} records to process in product_stat_history (using recent date: ${recentDateStr})`);
-
-    // Progress indicator
-    outputProgress({
-      status: "running",
-      operation: "Historical data import - Product Stat History",
-      message: `Found ${totalCount} records to process (last ${recentMonths} months only)`,
-      current: 0,
-      total: totalCount,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    // If not incremental, truncate the table first for better performance
-    if (!incrementalUpdate) {
-      console.log('Truncating imported_product_stat_history for full import...');
-      await localConnection.query('TRUNCATE TABLE imported_product_stat_history');
-    } else if (lastSyncTime) {
-      // For incremental updates, delete records that will be reimported
-      console.log(`Deleting records from imported_product_stat_history since ${lastSyncTime}...`);
-      await localConnection.query('DELETE FROM imported_product_stat_history WHERE date > $1', [lastSyncTime]);
-    }
-
-    // Process in batches for better performance
-    while (!allProcessed) {
-      try {
-        // Fetch batch from production with minimal filtering and no sorting
-        const [rows] = await prodConnection.query(`
-          SELECT
-            pid,
-            date,
-            COALESCE(score, 0) as score,
-            COALESCE(score2, 0) as score2,
-            COALESCE(qty_in_baskets, 0) as qty_in_baskets,
-            COALESCE(qty_sold, 0) as qty_sold,
-            COALESCE(notifies_set, 0) as notifies_set,
-            COALESCE(visibility_score, 0) as visibility_score,
-            COALESCE(health_score, 0) as health_score,
-            COALESCE(sold_view_score, 0) as sold_view_score
-          FROM product_stat_history
-          WHERE date >= ?
-            ${incrementalUpdate && lastSyncTime ? `AND date > ?` : ''}
-          LIMIT ? OFFSET ?
-        `, [
-          recentDateStr, 
-          ...(incrementalUpdate && lastSyncTime ? [lastSyncTime] : []),
-          batchSize,
-          offset
-        ]);
-
-        if (rows.length === 0) {
-          allProcessed = true;
-          break;
-        }
-
-        // Process rows in smaller batches for better performance
-        for (let i = 0; i < rows.length; i += insertBatchSize) {
-          const batch = rows.slice(i, i + insertBatchSize);
-          
-          if (batch.length === 0) continue;
-          
-          try {
-            // Build parameterized query to handle NULL values properly
-            const values = [];
-            const placeholders = [];
-            let placeholderIndex = 1;
-            
-            for (const row of batch) {
-              const rowPlaceholders = [
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`,
-                `$${placeholderIndex++}`
-              ];
-              
-              placeholders.push(`(${rowPlaceholders.join(', ')})`);
-              
-              values.push(
-                row.pid,
-                validateDate(row.date),
-                row.score,
-                row.score2,
-                row.qty_in_baskets,
-                row.qty_sold,
-                row.notifies_set,
-                row.visibility_score,
-                row.health_score,
-                row.sold_view_score
-              );
-            }
-            
-            // Execute direct batch insert without conflict checking
-            await localConnection.query(`
-              INSERT INTO imported_product_stat_history (
-                pid, date, score, score2, qty_in_baskets, qty_sold, notifies_set,
-                visibility_score, health_score, sold_view_score
-              )
-              VALUES ${placeholders.join(',\n')}
-            `, values);
-            
-            // All inserts are new records when using this approach
-            recordsAdded += batch.length;
-          } catch (error) {
-            console.error(`Error in batch insert of product_stat_history at offset ${i}:`, error);
-            errors.push({
-              table: 'imported_product_stat_history',
-              batchOffset: i,
-              batchSize: batch.length,
-              error: error.message
-            });
-          }
-        }
-
-        totalProcessed += rows.length;
-        offset += rows.length;
-
-        // Calculate current rate every 10 seconds or 100,000 records
-        const now = Date.now();
-        if (now - lastRateCheck > 10000 || totalProcessed - lastProcessed > 100000) {
-          const timeElapsed = (now - lastRateCheck) / 1000; // seconds
-          const recordsProcessed = totalProcessed - lastProcessed;
-          const currentRate = Math.round(recordsProcessed / timeElapsed);
-          
-          console.log(`Current import rate: ${currentRate} records/second`);
-          
-          lastRateCheck = now;
-          lastProcessed = totalProcessed;
-        }
-
-        // Update progress
-        outputProgress({
-          status: "running",
-          operation: "Historical data import - Product Stat History",
-          message: `Processed ${totalProcessed} of ${totalCount} records`,
-          current: totalProcessed,
-          total: totalCount,
-          elapsed: formatElapsedTime(startTime),
-          remaining: estimateRemaining(startTime, totalProcessed, totalCount),
-          rate: calculateRate(startTime, totalProcessed)
-        });
-      } catch (error) {
-        console.error('Error in batch import of product_stat_history:', error);
-        errors.push({
-          table: 'imported_product_stat_history',
-          error: error.message,
-          offset: offset,
-          batchSize: batchSize
-        });
-        
-        // Try to continue with next batch
-        offset += batchSize;
-      }
-    }
-
-    // Update sync status
-    await localConnection.query(`
-      INSERT INTO sync_status (table_name, last_sync_timestamp)
-      VALUES ('imported_product_stat_history', NOW())
-      ON CONFLICT (table_name) DO UPDATE SET
-        last_sync_timestamp = NOW()
-    `);
-
-    return { recordsAdded, recordsUpdated, totalProcessed, errors };
-  } catch (error) {
-    console.error('Error in product stat history import:', error);
-    return { 
-      recordsAdded, 
-      recordsUpdated, 
-      totalProcessed, 
-      errors: [...errors, { 
-        table: 'imported_product_stat_history',
-        error: error.message
-      }] 
-    };
-  }
-}
-
-module.exports = importHistoricalData; 

inventory-server/old/metrics-schema.sql

@@ -1,377 +0,0 @@
--- Disable foreign key checks
-SET session_replication_role = 'replica';
-
--- Temporary tables for batch metrics processing
-CREATE TABLE temp_sales_metrics (
-    pid BIGINT NOT NULL,
-    daily_sales_avg DECIMAL(10,3),
-    weekly_sales_avg DECIMAL(10,3),
-    monthly_sales_avg DECIMAL(10,3),
-    total_revenue DECIMAL(10,3),
-    avg_margin_percent DECIMAL(10,3),
-    first_sale_date DATE,
-    last_sale_date DATE,
-    stddev_daily_sales DECIMAL(10,3),
-    PRIMARY KEY (pid)
-);
-
-CREATE TABLE temp_purchase_metrics (
-    pid BIGINT NOT NULL,
-    avg_lead_time_days DECIMAL(10,2),
-    last_purchase_date DATE,
-    first_received_date DATE,
-    last_received_date DATE,
-    stddev_lead_time_days DECIMAL(10,2),
-    PRIMARY KEY (pid)
-);
-
--- New table for product metrics
-CREATE TABLE product_metrics (
-    pid BIGINT NOT NULL,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    -- Sales velocity metrics
-    daily_sales_avg DECIMAL(10,3),
-    weekly_sales_avg DECIMAL(10,3),
-    monthly_sales_avg DECIMAL(10,3),
-    avg_quantity_per_order DECIMAL(10,3),
-    number_of_orders INTEGER,
-    first_sale_date DATE,
-    last_sale_date DATE,
-    -- Stock metrics
-    days_of_inventory INTEGER,
-    weeks_of_inventory INTEGER,
-    reorder_point INTEGER,
-    safety_stock INTEGER,
-    reorder_qty INTEGER DEFAULT 0,
-    overstocked_amt INTEGER DEFAULT 0,
-    -- Financial metrics
-    avg_margin_percent DECIMAL(10,3),
-    total_revenue DECIMAL(10,3),
-    inventory_value DECIMAL(10,3),
-    cost_of_goods_sold DECIMAL(10,3),
-    gross_profit DECIMAL(10,3),
-    gmroi DECIMAL(10,3),
-    -- Purchase metrics
-    avg_lead_time_days DECIMAL(10,2),
-    last_purchase_date DATE,
-    first_received_date DATE,
-    last_received_date DATE,
-    -- Classification metrics
-    abc_class CHAR(1),
-    stock_status VARCHAR(20),
-    -- Turnover metrics
-    turnover_rate DECIMAL(12,3),
-    -- Lead time metrics
-    current_lead_time INTEGER,
-    target_lead_time INTEGER,
-    lead_time_status VARCHAR(20),
-    -- Forecast metrics
-    forecast_accuracy DECIMAL(5,2) DEFAULT NULL,
-    forecast_bias DECIMAL(5,2) DEFAULT NULL,
-    last_forecast_date DATE DEFAULT NULL,
-    PRIMARY KEY (pid),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_metrics_revenue ON product_metrics(total_revenue);
-CREATE INDEX idx_metrics_stock_status ON product_metrics(stock_status);
-CREATE INDEX idx_metrics_lead_time ON product_metrics(lead_time_status);
-CREATE INDEX idx_metrics_turnover ON product_metrics(turnover_rate);
-CREATE INDEX idx_metrics_last_calculated ON product_metrics(last_calculated_at);
-CREATE INDEX idx_metrics_abc ON product_metrics(abc_class);
-CREATE INDEX idx_metrics_sales ON product_metrics(daily_sales_avg, weekly_sales_avg, monthly_sales_avg);
-CREATE INDEX idx_metrics_forecast ON product_metrics(forecast_accuracy, forecast_bias);
-
--- New table for time-based aggregates
-CREATE TABLE product_time_aggregates (
-    pid BIGINT NOT NULL,
-    year INTEGER NOT NULL,
-    month INTEGER NOT NULL,
-    -- Sales metrics
-    total_quantity_sold INTEGER DEFAULT 0,
-    total_revenue DECIMAL(10,3) DEFAULT 0,
-    total_cost DECIMAL(10,3) DEFAULT 0,
-    order_count INTEGER DEFAULT 0,
-    -- Stock changes
-    stock_received INTEGER DEFAULT 0,
-    stock_ordered INTEGER DEFAULT 0,
-    -- Calculated fields
-    avg_price DECIMAL(10,3),
-    profit_margin DECIMAL(10,3),
-    inventory_value DECIMAL(10,3),
-    gmroi DECIMAL(10,3),
-    PRIMARY KEY (pid, year, month),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_date ON product_time_aggregates(year, month);
-
--- Create vendor_details table
-CREATE TABLE vendor_details (
-    vendor VARCHAR(100) PRIMARY KEY,
-    contact_name VARCHAR(100),
-    email VARCHAR(255),
-    phone VARCHAR(50),
-    status VARCHAR(20) DEFAULT 'active',
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
-);
-
-CREATE INDEX idx_vendor_details_status ON vendor_details(status);
-
--- New table for vendor metrics
-CREATE TABLE vendor_metrics (
-    vendor VARCHAR(100) NOT NULL,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    -- Performance metrics
-    avg_lead_time_days DECIMAL(10,3),
-    on_time_delivery_rate DECIMAL(5,2),
-    order_fill_rate DECIMAL(5,2),
-    total_orders INTEGER DEFAULT 0,
-    total_late_orders INTEGER DEFAULT 0,
-    total_purchase_value DECIMAL(10,3) DEFAULT 0,
-    avg_order_value DECIMAL(10,3),
-    -- Product metrics
-    active_products INTEGER DEFAULT 0,
-    total_products INTEGER DEFAULT 0,
-    -- Financial metrics
-    total_revenue DECIMAL(10,3) DEFAULT 0,
-    avg_margin_percent DECIMAL(5,2),
-    -- Status
-    status VARCHAR(20) DEFAULT 'active',
-    PRIMARY KEY (vendor),
-    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_vendor_performance ON vendor_metrics(on_time_delivery_rate);
-CREATE INDEX idx_vendor_status ON vendor_metrics(status);
-CREATE INDEX idx_vendor_metrics_last_calculated ON vendor_metrics(last_calculated_at);
-CREATE INDEX idx_vendor_metrics_orders ON vendor_metrics(total_orders, total_late_orders);
-
--- New table for category metrics
-CREATE TABLE category_metrics (
-    category_id BIGINT NOT NULL,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    -- Product metrics
-    product_count INTEGER DEFAULT 0,
-    active_products INTEGER DEFAULT 0,
-    -- Financial metrics
-    total_value DECIMAL(15,3) DEFAULT 0,
-    avg_margin DECIMAL(5,2),
-    turnover_rate DECIMAL(12,3),
-    growth_rate DECIMAL(5,2),
-    -- Status
-    status VARCHAR(20) DEFAULT 'active',
-    PRIMARY KEY (category_id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_category_status ON category_metrics(status);
-CREATE INDEX idx_category_growth ON category_metrics(growth_rate);
-CREATE INDEX idx_metrics_last_calculated_cat ON category_metrics(last_calculated_at);
-CREATE INDEX idx_category_metrics_products ON category_metrics(product_count, active_products);
-
--- New table for vendor time-based metrics
-CREATE TABLE vendor_time_metrics (
-    vendor VARCHAR(100) NOT NULL,
-    year INTEGER NOT NULL,
-    month INTEGER NOT NULL,
-    -- Order metrics
-    total_orders INTEGER DEFAULT 0,
-    late_orders INTEGER DEFAULT 0,
-    avg_lead_time_days DECIMAL(10,3),
-    -- Financial metrics
-    total_purchase_value DECIMAL(10,3) DEFAULT 0,
-    total_revenue DECIMAL(10,3) DEFAULT 0,
-    avg_margin_percent DECIMAL(5,2),
-    PRIMARY KEY (vendor, year, month),
-    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_vendor_date ON vendor_time_metrics(year, month);
-
--- New table for category time-based metrics
-CREATE TABLE category_time_metrics (
-    category_id BIGINT NOT NULL,
-    year INTEGER NOT NULL,
-    month INTEGER NOT NULL,
-    -- Product metrics
-    product_count INTEGER DEFAULT 0,
-    active_products INTEGER DEFAULT 0,
-    -- Financial metrics
-    total_value DECIMAL(15,3) DEFAULT 0,
-    total_revenue DECIMAL(15,3) DEFAULT 0,
-    avg_margin DECIMAL(5,2),
-    turnover_rate DECIMAL(12,3),
-    PRIMARY KEY (category_id, year, month),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_category_date ON category_time_metrics(year, month);
-
--- New table for category-based sales metrics
-CREATE TABLE category_sales_metrics (
-    category_id BIGINT NOT NULL,
-    brand VARCHAR(100) NOT NULL,
-    period_start DATE NOT NULL,
-    period_end DATE NOT NULL,
-    avg_daily_sales DECIMAL(10,3) DEFAULT 0,
-    total_sold INTEGER DEFAULT 0,
-    num_products INTEGER DEFAULT 0,
-    avg_price DECIMAL(10,3) DEFAULT 0,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (category_id, brand, period_start, period_end),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_category_brand ON category_sales_metrics(category_id, brand);
-CREATE INDEX idx_period ON category_sales_metrics(period_start, period_end);
-
--- New table for brand metrics
-CREATE TABLE brand_metrics (
-    brand VARCHAR(100) NOT NULL,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    -- Product metrics
-    product_count INTEGER DEFAULT 0,
-    active_products INTEGER DEFAULT 0,
-    -- Stock metrics
-    total_stock_units INTEGER DEFAULT 0,
-    total_stock_cost DECIMAL(15,2) DEFAULT 0,
-    total_stock_retail DECIMAL(15,2) DEFAULT 0,
-    -- Sales metrics
-    total_revenue DECIMAL(15,2) DEFAULT 0,
-    avg_margin DECIMAL(5,2) DEFAULT 0,
-    growth_rate DECIMAL(5,2) DEFAULT 0,
-    PRIMARY KEY (brand)
-);
-
-CREATE INDEX idx_brand_metrics_last_calculated ON brand_metrics(last_calculated_at);
-CREATE INDEX idx_brand_metrics_revenue ON brand_metrics(total_revenue);
-CREATE INDEX idx_brand_metrics_growth ON brand_metrics(growth_rate);
-
--- New table for brand time-based metrics
-CREATE TABLE brand_time_metrics (
-    brand VARCHAR(100) NOT NULL,
-    year INTEGER NOT NULL,
-    month INTEGER NOT NULL,
-    -- Product metrics
-    product_count INTEGER DEFAULT 0,
-    active_products INTEGER DEFAULT 0,
-    -- Stock metrics
-    total_stock_units INTEGER DEFAULT 0,
-    total_stock_cost DECIMAL(15,2) DEFAULT 0,
-    total_stock_retail DECIMAL(15,2) DEFAULT 0,
-    -- Sales metrics
-    total_revenue DECIMAL(15,2) DEFAULT 0,
-    avg_margin DECIMAL(5,2) DEFAULT 0,
-    growth_rate DECIMAL(5,2) DEFAULT 0,
-    PRIMARY KEY (brand, year, month)
-);
-
-CREATE INDEX idx_brand_time_date ON brand_time_metrics(year, month);
-
--- New table for sales forecasts
-CREATE TABLE sales_forecasts (
-    pid BIGINT NOT NULL,
-    forecast_date DATE NOT NULL,
-    forecast_quantity INTEGER,
-    confidence_level DECIMAL(5,2),
-    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (pid, forecast_date),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_forecast_date ON sales_forecasts(forecast_date);
-
--- New table for category forecasts
-CREATE TABLE category_forecasts (
-    category_id BIGINT NOT NULL,
-    forecast_date DATE NOT NULL,
-    forecast_revenue DECIMAL(15,2),
-    forecast_units INTEGER,
-    confidence_level DECIMAL(5,2),
-    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (category_id, forecast_date),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
-);
-
-CREATE INDEX idx_cat_forecast_date ON category_forecasts(forecast_date);
-
--- Create views for common calculations
-CREATE OR REPLACE VIEW inventory_health AS
-WITH stock_levels AS (
-    SELECT 
-        p.pid,
-        p.title,
-        p.SKU,
-        p.stock_quantity,
-        p.preorder_count,
-        pm.daily_sales_avg,
-        pm.weekly_sales_avg,
-        pm.monthly_sales_avg,
-        pm.reorder_point,
-        pm.safety_stock,
-        pm.days_of_inventory,
-        pm.weeks_of_inventory,
-        pm.stock_status,
-        pm.abc_class,
-        pm.turnover_rate,
-        pm.avg_lead_time_days,
-        pm.current_lead_time,
-        pm.target_lead_time,
-        pm.lead_time_status,
-        p.cost_price,
-        p.price,
-        pm.inventory_value,
-        pm.gmroi
-    FROM products p
-    LEFT JOIN product_metrics pm ON p.pid = pm.pid
-    WHERE p.managing_stock = true AND p.visible = true
-)
-SELECT 
-    *,
-    CASE 
-        WHEN stock_quantity <= safety_stock THEN 'Critical'
-        WHEN stock_quantity <= reorder_point THEN 'Low'
-        WHEN stock_quantity > (reorder_point * 3) THEN 'Excess'
-        ELSE 'Healthy'
-    END as inventory_status,
-    CASE 
-        WHEN lead_time_status = 'delayed' AND stock_status = 'low' THEN 'High'
-        WHEN lead_time_status = 'delayed' OR stock_status = 'low' THEN 'Medium'
-        ELSE 'Low'
-    END as risk_level
-FROM stock_levels;
-
--- Create view for category performance trends
-CREATE OR REPLACE VIEW category_performance_trends AS
-WITH monthly_trends AS (
-    SELECT 
-        c.cat_id,
-        c.name as category_name,
-        ctm.year,
-        ctm.month,
-        ctm.product_count,
-        ctm.active_products,
-        ctm.total_value,
-        ctm.total_revenue,
-        ctm.avg_margin,
-        ctm.turnover_rate,
-        LAG(ctm.total_revenue) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_revenue,
-        LAG(ctm.turnover_rate) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_turnover
-    FROM categories c
-    JOIN category_time_metrics ctm ON c.cat_id = ctm.category_id
-)
-SELECT 
-    *,
-    CASE 
-        WHEN prev_month_revenue IS NULL THEN 0
-        ELSE ((total_revenue - prev_month_revenue) / prev_month_revenue) * 100 
-    END as revenue_growth_percent,
-    CASE 
-        WHEN prev_month_turnover IS NULL THEN 0
-        ELSE ((turnover_rate - prev_month_turnover) / prev_month_turnover) * 100 
-    END as turnover_growth_percent
-FROM monthly_trends;
-
-SET session_replication_role = 'origin';

inventory-server/old/metrics/brand-metrics.js

@@ -1,321 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateBrandMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Brand metrics calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders: 0,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // Get order count that will be processed
-        const orderCount = await connection.query(`
-            SELECT COUNT(*) as count 
-            FROM orders o 
-            WHERE o.canceled = false
-        `);
-        processedOrders = parseInt(orderCount.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting brand metrics calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // Calculate brand metrics with optimized queries
-        await connection.query(`
-            INSERT INTO brand_metrics (
-                brand,
-                product_count,
-                active_products,
-                total_stock_units,
-                total_stock_cost,
-                total_stock_retail,
-                total_revenue,
-                avg_margin,
-                growth_rate
-            )
-            WITH filtered_products AS (
-                SELECT 
-                    p.*,
-                    CASE 
-                        WHEN p.stock_quantity <= 5000 AND p.stock_quantity >= 0 
-                        THEN p.pid 
-                    END as valid_pid,
-                    CASE 
-                        WHEN p.visible = true 
-                        AND p.stock_quantity <= 5000 
-                        AND p.stock_quantity >= 0 
-                        THEN p.pid 
-                    END as active_pid,
-                    CASE 
-                        WHEN p.stock_quantity IS NULL 
-                        OR p.stock_quantity < 0 
-                        OR p.stock_quantity > 5000 
-                        THEN 0 
-                        ELSE p.stock_quantity 
-                    END as valid_stock
-                FROM products p
-                WHERE p.brand IS NOT NULL
-            ),
-            sales_periods AS (
-                SELECT 
-                    p.brand,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as period_revenue,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as period_margin,
-                    COUNT(DISTINCT DATE(o.date)) as period_days,
-                    CASE 
-                        WHEN o.date >= CURRENT_DATE - INTERVAL '3 months' THEN 'current'
-                        WHEN o.date BETWEEN CURRENT_DATE - INTERVAL '15 months' 
-                            AND CURRENT_DATE - INTERVAL '12 months' THEN 'previous'
-                    END as period_type
-                FROM filtered_products p
-                JOIN orders o ON p.pid = o.pid 
-                WHERE o.canceled = false
-                    AND o.date >= CURRENT_DATE - INTERVAL '15 months'
-                GROUP BY p.brand, period_type
-            ),
-            brand_data AS (
-                SELECT 
-                    p.brand,
-                    COUNT(DISTINCT p.valid_pid) as product_count,
-                    COUNT(DISTINCT p.active_pid) as active_products,
-                    SUM(p.valid_stock) as total_stock_units,
-                    SUM(p.valid_stock * p.cost_price) as total_stock_cost,
-                    SUM(p.valid_stock * p.price) as total_stock_retail,
-                    COALESCE(SUM(o.quantity * (o.price - COALESCE(o.discount, 0))), 0) as total_revenue,
-                    CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 
-                        THEN GREATEST(
-                            -100.0,
-                            LEAST(
-                                100.0,
-                                (
-                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
-                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
-                                ) * 100.0 / 
-                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
-                            )
-                        )
-                        ELSE 0 
-                    END as avg_margin
-                FROM filtered_products p
-                LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
-                GROUP BY p.brand
-            )
-            SELECT 
-                bd.brand,
-                bd.product_count,
-                bd.active_products,
-                bd.total_stock_units,
-                bd.total_stock_cost,
-                bd.total_stock_retail,
-                bd.total_revenue,
-                bd.avg_margin,
-                CASE
-                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
-                         AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0 
-                    THEN 100.0
-                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
-                    THEN 0.0
-                    ELSE GREATEST(
-                        -100.0,
-                        LEAST(
-                            ((MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) - 
-                              MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)) / 
-                             NULLIF(ABS(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)), 0)) * 100.0,
-                            999.99
-                        )
-                    )
-                END as growth_rate
-            FROM brand_data bd
-            LEFT JOIN sales_periods sp ON bd.brand = sp.brand
-            GROUP BY bd.brand, bd.product_count, bd.active_products, bd.total_stock_units, 
-                     bd.total_stock_cost, bd.total_stock_retail, bd.total_revenue, bd.avg_margin
-            ON CONFLICT (brand) DO UPDATE
-            SET
-                product_count = EXCLUDED.product_count,
-                active_products = EXCLUDED.active_products,
-                total_stock_units = EXCLUDED.total_stock_units,
-                total_stock_cost = EXCLUDED.total_stock_cost,
-                total_stock_retail = EXCLUDED.total_stock_retail,
-                total_revenue = EXCLUDED.total_revenue,
-                avg_margin = EXCLUDED.avg_margin,
-                growth_rate = EXCLUDED.growth_rate,
-                last_calculated_at = CURRENT_TIMESTAMP
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.97);
-        outputProgress({
-            status: 'running',
-            operation: 'Brand metrics calculated, starting time-based metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Calculate brand time-based metrics with optimized query
-        await connection.query(`
-            INSERT INTO brand_time_metrics (
-                brand,
-                year,
-                month,
-                product_count,
-                active_products,
-                total_stock_units,
-                total_stock_cost,
-                total_stock_retail,
-                total_revenue,
-                avg_margin
-            )
-            WITH filtered_products AS (
-                SELECT 
-                    p.*,
-                    CASE WHEN p.stock_quantity <= 5000 THEN p.pid END as valid_pid,
-                    CASE WHEN p.visible = true AND p.stock_quantity <= 5000 THEN p.pid END as active_pid,
-                    CASE 
-                        WHEN p.stock_quantity IS NULL OR p.stock_quantity < 0 OR p.stock_quantity > 5000 THEN 0 
-                        ELSE p.stock_quantity 
-                    END as valid_stock
-                FROM products p
-                WHERE p.brand IS NOT NULL
-            ),
-            monthly_metrics AS (
-                SELECT 
-                    p.brand,
-                    EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
-                    EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
-                    COUNT(DISTINCT p.valid_pid) as product_count,
-                    COUNT(DISTINCT p.active_pid) as active_products,
-                    SUM(p.valid_stock) as total_stock_units,
-                    SUM(p.valid_stock * p.cost_price) as total_stock_cost,
-                    SUM(p.valid_stock * p.price) as total_stock_retail,
-                    SUM(o.quantity * o.price) as total_revenue,
-                    CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 
-                        THEN GREATEST(
-                            -100.0,
-                            LEAST(
-                                100.0,
-                                (
-                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
-                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
-                                ) * 100.0 / 
-                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
-                            )
-                        )
-                        ELSE 0 
-                    END as avg_margin
-                FROM filtered_products p
-                LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
-                WHERE o.date >= CURRENT_DATE - INTERVAL '12 months'
-                GROUP BY p.brand, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
-            )
-            SELECT *
-            FROM monthly_metrics
-            ON CONFLICT (brand, year, month) DO UPDATE
-            SET
-                product_count = EXCLUDED.product_count,
-                active_products = EXCLUDED.active_products,
-                total_stock_units = EXCLUDED.total_stock_units,
-                total_stock_cost = EXCLUDED.total_stock_cost,
-                total_stock_retail = EXCLUDED.total_stock_retail,
-                total_revenue = EXCLUDED.total_revenue,
-                avg_margin = EXCLUDED.avg_margin
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.99);
-        outputProgress({
-            status: 'running',
-            operation: 'Brand time-based metrics calculated',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('brand_metrics', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating brand metrics');
-        throw error;
-    } finally {
-        if (connection) {
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateBrandMetrics; 

inventory-server/old/metrics/category-metrics.js

@@ -1,554 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateCategoryMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Category metrics calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders: 0,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // Get order count that will be processed
-        const orderCount = await connection.query(`
-            SELECT COUNT(*) as count 
-            FROM orders o 
-            WHERE o.canceled = false
-        `);
-        processedOrders = parseInt(orderCount.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting category metrics calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-        
-        // First, calculate base category metrics
-        await connection.query(`
-            INSERT INTO category_metrics (
-                category_id,
-                product_count,
-                active_products,
-                total_value,
-                status,
-                last_calculated_at
-            )
-            SELECT 
-                c.cat_id,
-                COUNT(DISTINCT p.pid) as product_count,
-                COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
-                COALESCE(SUM(p.stock_quantity * p.cost_price), 0) as total_value,
-                c.status,
-                NOW() as last_calculated_at
-            FROM categories c
-            LEFT JOIN product_categories pc ON c.cat_id = pc.cat_id
-            LEFT JOIN products p ON pc.pid = p.pid
-            GROUP BY c.cat_id, c.status
-            ON CONFLICT (category_id) DO UPDATE
-            SET
-                product_count = EXCLUDED.product_count,
-                active_products = EXCLUDED.active_products,
-                total_value = EXCLUDED.total_value,
-                status = EXCLUDED.status,
-                last_calculated_at = EXCLUDED.last_calculated_at
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.90);
-        outputProgress({
-            status: 'running',
-            operation: 'Base category metrics calculated, updating with margin data',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Then update with margin and turnover data
-        await connection.query(`
-            WITH category_sales AS (
-                SELECT 
-                    pc.cat_id,
-                    SUM(o.quantity * o.price) as total_sales,
-                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin,
-                    SUM(o.quantity) as units_sold,
-                    AVG(GREATEST(p.stock_quantity, 0)) as avg_stock,
-                    COUNT(DISTINCT DATE(o.date)) as active_days
-                FROM product_categories pc
-                JOIN products p ON pc.pid = p.pid
-                JOIN orders o ON p.pid = o.pid
-                LEFT JOIN turnover_config tc ON 
-                    (tc.category_id = pc.cat_id AND tc.vendor = p.vendor) OR
-                    (tc.category_id = pc.cat_id AND tc.vendor IS NULL) OR
-                    (tc.category_id IS NULL AND tc.vendor = p.vendor) OR
-                    (tc.category_id IS NULL AND tc.vendor IS NULL)
-                WHERE o.canceled = false 
-                    AND o.date >= CURRENT_DATE - (COALESCE(tc.calculation_period_days, 30) || ' days')::INTERVAL
-                GROUP BY pc.cat_id
-            )
-            UPDATE category_metrics
-            SET 
-                avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
-                turnover_rate = CASE 
-                    WHEN cs.avg_stock > 0 AND cs.active_days > 0 
-                    THEN LEAST(
-                        (cs.units_sold / cs.avg_stock) * (365.0 / cs.active_days),
-                        999.99
-                    )
-                    ELSE 0 
-                END,
-                last_calculated_at = NOW()
-            FROM category_sales cs
-            WHERE category_id = cs.cat_id
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.95);
-        outputProgress({
-            status: 'running',
-            operation: 'Margin data updated, calculating growth rates',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Finally update growth rates
-        await connection.query(`
-            WITH current_period AS (
-                SELECT 
-                    pc.cat_id,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
-                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as gross_profit,
-                    COUNT(DISTINCT DATE(o.date)) as days
-                FROM product_categories pc
-                JOIN products p ON pc.pid = p.pid
-                JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
-                WHERE o.canceled = false 
-                    AND o.date >= CURRENT_DATE - INTERVAL '3 months'
-                GROUP BY pc.cat_id
-            ),
-            previous_period AS (
-                SELECT 
-                    pc.cat_id,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
-                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
-                    COUNT(DISTINCT DATE(o.date)) as days
-                FROM product_categories pc
-                JOIN products p ON pc.pid = p.pid
-                JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
-                WHERE o.canceled = false 
-                    AND o.date BETWEEN CURRENT_DATE - INTERVAL '15 months' 
-                    AND CURRENT_DATE - INTERVAL '12 months'
-                GROUP BY pc.cat_id
-            ),
-            trend_data AS (
-                SELECT 
-                    pc.cat_id,
-                    EXTRACT(MONTH FROM o.date) as month,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
-                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
-                    COUNT(DISTINCT DATE(o.date)) as days_in_month
-                FROM product_categories pc
-                JOIN products p ON pc.pid = p.pid
-                JOIN orders o ON p.pid = o.pid
-                LEFT JOIN sales_seasonality ss ON EXTRACT(MONTH FROM o.date) = ss.month
-                WHERE o.canceled = false 
-                    AND o.date >= CURRENT_DATE - INTERVAL '15 months'
-                GROUP BY pc.cat_id, EXTRACT(MONTH FROM o.date)
-            ),
-            trend_stats AS (
-                SELECT 
-                    cat_id,
-                    COUNT(*) as n,
-                    AVG(month) as avg_x,
-                    AVG(revenue / NULLIF(days_in_month, 0)) as avg_y,
-                    SUM(month * (revenue / NULLIF(days_in_month, 0))) as sum_xy,
-                    SUM(month * month) as sum_xx
-                FROM trend_data
-                GROUP BY cat_id
-                HAVING COUNT(*) >= 6
-            ),
-            trend_analysis AS (
-                SELECT 
-                    cat_id,
-                    ((n * sum_xy) - (avg_x * n * avg_y)) / 
-                        NULLIF((n * sum_xx) - (n * avg_x * avg_x), 0) as trend_slope,
-                    avg_y as avg_daily_revenue
-                FROM trend_stats
-            ),
-            margin_calc AS (
-                SELECT 
-                    pc.cat_id,
-                    CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 THEN
-                            GREATEST(
-                                -100.0,
-                                LEAST(
-                                    100.0,
-                                    (
-                                        SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
-                                        SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
-                                    ) * 100.0 / 
-                                    NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
-                                )
-                            )
-                        ELSE NULL
-                    END as avg_margin
-                FROM product_categories pc
-                JOIN products p ON pc.pid = p.pid
-                JOIN orders o ON p.pid = o.pid
-                WHERE o.canceled = false 
-                    AND o.date >= CURRENT_DATE - INTERVAL '3 months'
-                GROUP BY pc.cat_id
-            ),
-            combined_metrics AS (
-                SELECT 
-                    COALESCE(cp.cat_id, pp.cat_id) as category_id,
-                    CASE
-                        WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
-                        WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
-                        WHEN ta.trend_slope IS NOT NULL THEN
-                            GREATEST(
-                                -100.0,
-                                LEAST(
-                                    (ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
-                                    999.99
-                                )
-                            )
-                        ELSE
-                            GREATEST(
-                                -100.0,
-                                LEAST(
-                                    ((COALESCE(cp.revenue, 0) - pp.revenue) / 
-                                        NULLIF(ABS(pp.revenue), 0)) * 100.0,
-                                    999.99
-                                )
-                            )
-                    END as growth_rate,
-                    mc.avg_margin
-                FROM current_period cp
-                FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
-                LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
-                LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
-            )
-            UPDATE category_metrics cm
-            SET 
-                growth_rate = CASE
-                    WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
-                    WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
-                    WHEN ta.trend_slope IS NOT NULL THEN
-                        GREATEST(
-                            -100.0,
-                            LEAST(
-                                (ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
-                                999.99
-                            )
-                        )
-                    ELSE
-                        GREATEST(
-                            -100.0,
-                            LEAST(
-                                ((COALESCE(cp.revenue, 0) - pp.revenue) / 
-                                    NULLIF(ABS(pp.revenue), 0)) * 100.0,
-                                999.99
-                            )
-                        )
-                END,
-                avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
-                last_calculated_at = NOW()
-            FROM current_period cp
-            FULL OUTER JOIN previous_period pp ON cp.cat_id = pp.cat_id
-            LEFT JOIN trend_analysis ta ON COALESCE(cp.cat_id, pp.cat_id) = ta.cat_id
-            LEFT JOIN margin_calc mc ON COALESCE(cp.cat_id, pp.cat_id) = mc.cat_id
-            WHERE cm.category_id = COALESCE(cp.cat_id, pp.cat_id)
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.97);
-        outputProgress({
-            status: 'running',
-            operation: 'Growth rates calculated, updating time-based metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Calculate time-based metrics
-        await connection.query(`
-            INSERT INTO category_time_metrics (
-                category_id,
-                year,
-                month,
-                product_count,
-                active_products,
-                total_value,
-                total_revenue,
-                avg_margin,
-                turnover_rate
-            )
-            SELECT 
-                pc.cat_id,
-                EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
-                EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
-                COUNT(DISTINCT p.pid) as product_count,
-                COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
-                SUM(p.stock_quantity * p.cost_price) as total_value,
-                SUM(o.quantity * o.price) as total_revenue,
-                CASE
-                    WHEN SUM(o.quantity * o.price) > 0 THEN
-                        LEAST(
-                            GREATEST(
-                                SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0))) * 100.0 / 
-                                SUM(o.quantity * o.price),
-                                -100
-                            ),
-                            100
-                        )
-                    ELSE 0
-                END as avg_margin,
-                COALESCE(
-                    LEAST(
-                        SUM(o.quantity) / NULLIF(AVG(GREATEST(p.stock_quantity, 0)), 0),
-                        999.99
-                    ),
-                    0
-                ) as turnover_rate
-            FROM product_categories pc
-            JOIN products p ON pc.pid = p.pid
-            JOIN orders o ON p.pid = o.pid
-            WHERE o.canceled = false
-                AND o.date >= CURRENT_DATE - INTERVAL '12 months'
-            GROUP BY pc.cat_id, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
-            ON CONFLICT (category_id, year, month) DO UPDATE
-            SET
-                product_count = EXCLUDED.product_count,
-                active_products = EXCLUDED.active_products,
-                total_value = EXCLUDED.total_value,
-                total_revenue = EXCLUDED.total_revenue,
-                avg_margin = EXCLUDED.avg_margin,
-                turnover_rate = EXCLUDED.turnover_rate
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.99);
-        outputProgress({
-            status: 'running',
-            operation: 'Time-based metrics calculated, updating category-sales metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Calculate category-sales metrics
-        await connection.query(`
-            INSERT INTO category_sales_metrics (
-                category_id,
-                brand,
-                period_start,
-                period_end,
-                avg_daily_sales,
-                total_sold,
-                num_products,
-                avg_price,
-                last_calculated_at
-            )
-            WITH date_ranges AS (
-                SELECT 
-                    CURRENT_DATE - INTERVAL '30 days' as period_start,
-                    CURRENT_DATE as period_end
-                UNION ALL
-                SELECT 
-                    CURRENT_DATE - INTERVAL '90 days',
-                    CURRENT_DATE - INTERVAL '31 days'
-                UNION ALL
-                SELECT 
-                    CURRENT_DATE - INTERVAL '180 days',
-                    CURRENT_DATE - INTERVAL '91 days'
-                UNION ALL
-                SELECT 
-                    CURRENT_DATE - INTERVAL '365 days',
-                    CURRENT_DATE - INTERVAL '181 days'
-            ),
-            sales_data AS (
-                SELECT 
-                    pc.cat_id,
-                    COALESCE(p.brand, 'Unknown') as brand,
-                    dr.period_start,
-                    dr.period_end,
-                    COUNT(DISTINCT p.pid) as num_products,
-                    SUM(o.quantity) as total_sold,
-                    SUM(o.quantity * o.price) as total_revenue,
-                    COUNT(DISTINCT DATE(o.date)) as num_days
-                FROM products p
-                JOIN product_categories pc ON p.pid = pc.pid
-                JOIN orders o ON p.pid = o.pid
-                CROSS JOIN date_ranges dr
-                WHERE o.canceled = false
-                    AND o.date BETWEEN dr.period_start AND dr.period_end
-                GROUP BY pc.cat_id, p.brand, dr.period_start, dr.period_end
-            )
-            SELECT 
-                cat_id as category_id,
-                brand,
-                period_start,
-                period_end,
-                CASE 
-                    WHEN num_days > 0 
-                    THEN total_sold / num_days 
-                    ELSE 0 
-                END as avg_daily_sales,
-                total_sold,
-                num_products,
-                CASE 
-                    WHEN total_sold > 0 
-                    THEN total_revenue / total_sold 
-                    ELSE 0 
-                END as avg_price,
-                NOW() as last_calculated_at
-            FROM sales_data
-            ON CONFLICT (category_id, brand, period_start, period_end) DO UPDATE
-            SET
-                avg_daily_sales = EXCLUDED.avg_daily_sales,
-                total_sold = EXCLUDED.total_sold,
-                num_products = EXCLUDED.num_products,
-                avg_price = EXCLUDED.avg_price,
-                last_calculated_at = EXCLUDED.last_calculated_at
-        `);
-
-        processedCount = Math.floor(totalProducts * 1.0);
-        outputProgress({
-            status: 'running',
-            operation: 'Category-sales metrics calculated',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('category_metrics', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating category metrics');
-        throw error;
-    } finally {
-        if (connection) {
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateCategoryMetrics; 

inventory-server/old/metrics/financial-metrics.js

@@ -1,214 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateFinancialMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Financial metrics calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders: 0,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // Get order count that will be processed
-        const orderCount = await connection.query(`
-            SELECT COUNT(*) as count 
-            FROM orders o 
-            WHERE o.canceled = false
-                AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'
-        `);
-        processedOrders = parseInt(orderCount.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting financial metrics calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // First, calculate beginning inventory values (12 months ago)
-        await connection.query(`
-            CREATE TEMPORARY TABLE IF NOT EXISTS temp_beginning_inventory AS
-            WITH beginning_inventory_calc AS (
-                SELECT 
-                    p.pid,
-                    p.stock_quantity as current_quantity,
-                    COALESCE(SUM(o.quantity), 0) as sold_quantity,
-                    COALESCE(SUM(po.received), 0) as received_quantity,
-                    GREATEST(0, (p.stock_quantity + COALESCE(SUM(o.quantity), 0) - COALESCE(SUM(po.received), 0))) as beginning_quantity,
-                    p.cost_price
-                FROM 
-                    products p
-                LEFT JOIN 
-                    orders o ON p.pid = o.pid 
-                    AND o.canceled = false 
-                    AND o.date >= CURRENT_DATE - INTERVAL '12 months'::interval
-                LEFT JOIN 
-                    purchase_orders po ON p.pid = po.pid 
-                    AND po.received_date IS NOT NULL 
-                    AND po.received_date >= CURRENT_DATE - INTERVAL '12 months'::interval
-                GROUP BY 
-                    p.pid, p.stock_quantity, p.cost_price
-            )
-            SELECT 
-                pid,
-                beginning_quantity,
-                beginning_quantity * cost_price as beginning_value,
-                current_quantity * cost_price as current_value,
-                ((beginning_quantity * cost_price) + (current_quantity * cost_price)) / 2 as average_inventory_value
-            FROM 
-                beginning_inventory_calc
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.60);
-        outputProgress({
-            status: 'running',
-            operation: 'Beginning inventory values calculated, computing financial metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // Calculate financial metrics with optimized query and standard formulas
-        await connection.query(`
-            WITH product_financials AS (
-                SELECT 
-                    p.pid,
-                    COALESCE(bi.average_inventory_value, p.cost_price * p.stock_quantity) as avg_inventory_value,
-                    p.cost_price * p.stock_quantity as current_inventory_value,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as total_revenue,
-                    SUM(o.quantity * COALESCE(o.costeach, 0)) as cost_of_goods_sold,
-                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - COALESCE(o.costeach, 0))) as gross_profit,
-                    MIN(o.date) as first_sale_date,
-                    MAX(o.date) as last_sale_date,
-                    EXTRACT(DAY FROM (MAX(o.date)::timestamp with time zone - MIN(o.date)::timestamp with time zone)) + 1 as calculation_period_days,
-                    COUNT(DISTINCT DATE(o.date)) as active_days
-                FROM products p
-                LEFT JOIN orders o ON p.pid = o.pid
-                LEFT JOIN temp_beginning_inventory bi ON p.pid = bi.pid
-                WHERE o.canceled = false
-                    AND DATE(o.date) >= CURRENT_DATE - INTERVAL '12 months'::interval
-                GROUP BY p.pid, p.cost_price, p.stock_quantity, bi.average_inventory_value
-            )
-            UPDATE product_metrics pm
-            SET 
-                inventory_value = COALESCE(pf.current_inventory_value, 0)::decimal(10,3),
-                total_revenue = COALESCE(pf.total_revenue, 0)::decimal(10,3),
-                cost_of_goods_sold = COALESCE(pf.cost_of_goods_sold, 0)::decimal(10,3),
-                gross_profit = COALESCE(pf.gross_profit, 0)::decimal(10,3),
-                turnover_rate = CASE 
-                    WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN 
-                        COALESCE(pf.cost_of_goods_sold, 0) / NULLIF(pf.avg_inventory_value, 0)
-                    ELSE 0 
-                END::decimal(12,3),
-                gmroi = CASE 
-                    WHEN COALESCE(pf.avg_inventory_value, 0) > 0 THEN 
-                        COALESCE(pf.gross_profit, 0) / NULLIF(pf.avg_inventory_value, 0)
-                    ELSE 0 
-                END::decimal(10,3),
-                last_calculated_at = CURRENT_TIMESTAMP
-            FROM product_financials pf
-            WHERE pm.pid = pf.pid
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.65);
-        outputProgress({
-            status: 'running',
-            operation: 'Base financial metrics calculated, updating time aggregates',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Clean up temporary tables
-        await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('financial_metrics', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating financial metrics');
-        throw error;
-    } finally {
-        if (connection) {
-            try {
-                // Make sure temporary tables are always cleaned up
-                await connection.query('DROP TABLE IF EXISTS temp_beginning_inventory');
-            } catch (err) {
-                console.error('Error cleaning up temp tables:', err);
-            }
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateFinancialMetrics; 

inventory-server/old/metrics/product-metrics.js

@@ -1,736 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-// Helper function to handle NaN and undefined values
-function sanitizeValue(value) {
-    if (value === undefined || value === null || Number.isNaN(value)) {
-        return null;
-    }
-    return value;
-}
-
-async function calculateProductMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    let connection;
-    let success = false;
-    let processedOrders = 0;
-    const BATCH_SIZE = 5000;
-    
-    try {
-        connection = await getConnection();
-        // Skip flags are inherited from the parent scope
-        const SKIP_PRODUCT_BASE_METRICS = 0;
-        const SKIP_PRODUCT_TIME_AGGREGATES = 0;
-
-        // Get total product count if not provided
-        if (!totalProducts) {
-            const productCount = await connection.query('SELECT COUNT(*) as count FROM products');
-            totalProducts = parseInt(productCount.rows[0].count);
-        }
-
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Product metrics calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // First ensure all products have a metrics record
-        await connection.query(`
-            INSERT INTO product_metrics (pid, last_calculated_at)
-            SELECT pid, NOW()
-            FROM products
-            ON CONFLICT (pid) DO NOTHING
-        `);
-
-        // Get threshold settings once
-        const thresholds = await connection.query(`
-            SELECT critical_days, reorder_days, overstock_days, low_stock_threshold
-            FROM stock_thresholds 
-            WHERE category_id IS NULL AND vendor IS NULL
-            LIMIT 1
-        `);
-        
-        // Check if threshold data was returned
-        if (!thresholds.rows || thresholds.rows.length === 0) {
-            console.warn('No default thresholds found in the database. Using explicit type casting in the query.');
-        }
-        
-        const defaultThresholds = thresholds.rows[0];
-
-        // Get financial calculation configuration parameters
-        const financialConfig = await connection.query(`
-            SELECT 
-                order_cost, 
-                holding_rate, 
-                service_level_z_score,
-                min_reorder_qty,
-                default_reorder_qty,
-                default_safety_stock
-            FROM financial_calc_config 
-            WHERE id = 1
-            LIMIT 1
-        `);
-        const finConfig = financialConfig.rows[0] || { 
-            order_cost: 25.00, 
-            holding_rate: 0.25, 
-            service_level_z_score: 1.96,
-            min_reorder_qty: 1,
-            default_reorder_qty: 5,
-            default_safety_stock: 5
-        };
-
-        // Calculate base product metrics
-        if (!SKIP_PRODUCT_BASE_METRICS) {
-            outputProgress({
-                status: 'running',
-                operation: 'Starting base product metrics calculation',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            // Get order count that will be processed
-            const orderCount = await connection.query(`
-                SELECT COUNT(*) as count 
-                FROM orders o 
-                WHERE o.canceled = false
-            `);
-            processedOrders = parseInt(orderCount.rows[0].count);
-
-            // Clear temporary tables
-            await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
-            await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
-
-            // Create temp_sales_metrics
-            await connection.query(`
-                CREATE TEMPORARY TABLE temp_sales_metrics (
-                    pid BIGINT NOT NULL,
-                    daily_sales_avg DECIMAL(10,3),
-                    weekly_sales_avg DECIMAL(10,3),
-                    monthly_sales_avg DECIMAL(10,3),
-                    total_revenue DECIMAL(10,3),
-                    avg_margin_percent DECIMAL(10,3),
-                    first_sale_date DATE,
-                    last_sale_date DATE,
-                    stddev_daily_sales DECIMAL(10,3),
-                    PRIMARY KEY (pid)
-                )
-            `);
-
-            // Create temp_purchase_metrics
-            await connection.query(`
-                CREATE TEMPORARY TABLE temp_purchase_metrics (
-                    pid BIGINT NOT NULL,
-                    avg_lead_time_days DECIMAL(10,2),
-                    last_purchase_date DATE,
-                    first_received_date DATE,
-                    last_received_date DATE,
-                    stddev_lead_time_days DECIMAL(10,2),
-                    PRIMARY KEY (pid)
-                )
-            `);
-
-            // Populate temp_sales_metrics with base stats and sales averages
-            await connection.query(`
-                INSERT INTO temp_sales_metrics
-                SELECT 
-                    p.pid,
-                    COALESCE(SUM(o.quantity) / NULLIF(COUNT(DISTINCT DATE(o.date)), 0), 0) as daily_sales_avg,
-                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 7), 0), 0) as weekly_sales_avg,
-                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 30), 0), 0) as monthly_sales_avg,
-                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
-                    CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 
-                        THEN ((SUM(o.quantity * o.price) - SUM(o.quantity * p.cost_price)) / SUM(o.quantity * o.price)) * 100 
-                        ELSE 0 
-                    END as avg_margin_percent,
-                    MIN(o.date) as first_sale_date,
-                    MAX(o.date) as last_sale_date,
-                    COALESCE(STDDEV_SAMP(daily_qty.quantity), 0) as stddev_daily_sales
-                FROM products p
-                LEFT JOIN orders o ON p.pid = o.pid 
-                    AND o.canceled = false
-                    AND o.date >= CURRENT_DATE - INTERVAL '90 days'
-                LEFT JOIN (
-                    SELECT 
-                        pid, 
-                        DATE(date) as sale_date, 
-                        SUM(quantity) as quantity
-                    FROM orders
-                    WHERE canceled = false
-                    AND date >= CURRENT_DATE - INTERVAL '90 days'
-                    GROUP BY pid, DATE(date)
-                ) daily_qty ON p.pid = daily_qty.pid
-                GROUP BY p.pid
-            `);
-
-            // Populate temp_purchase_metrics with timeout protection
-            await Promise.race([
-                connection.query(`
-                    INSERT INTO temp_purchase_metrics
-                    SELECT 
-                        p.pid,
-                        AVG(
-                            CASE 
-                                WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL 
-                                THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
-                                ELSE NULL
-                            END
-                        ) as avg_lead_time_days,
-                        MAX(po.date) as last_purchase_date,
-                        MIN(po.received_date) as first_received_date,
-                        MAX(po.received_date) as last_received_date,
-                        STDDEV_SAMP(
-                            CASE 
-                                WHEN po.received_date IS NOT NULL AND po.date IS NOT NULL 
-                                THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
-                                ELSE NULL
-                            END
-                        ) as stddev_lead_time_days
-                    FROM products p
-                    LEFT JOIN purchase_orders po ON p.pid = po.pid 
-                        AND po.received_date IS NOT NULL
-                        AND po.date IS NOT NULL
-                        AND po.date >= CURRENT_DATE - INTERVAL '365 days'
-                    GROUP BY p.pid
-                `),
-                new Promise((_, reject) => 
-                    setTimeout(() => reject(new Error('Timeout: temp_purchase_metrics query took too long')), 60000)
-                )
-            ]).catch(async (err) => {
-                logError(err, 'Error populating temp_purchase_metrics, continuing with empty table');
-                // Create an empty fallback to continue processing
-                await connection.query(`
-                    INSERT INTO temp_purchase_metrics
-                    SELECT 
-                        p.pid,
-                        30.0 as avg_lead_time_days,
-                        NULL as last_purchase_date,
-                        NULL as first_received_date,
-                        NULL as last_received_date,
-                        0.0 as stddev_lead_time_days
-                    FROM products p
-                    LEFT JOIN temp_purchase_metrics tpm ON p.pid = tpm.pid
-                    WHERE tpm.pid IS NULL
-                `);
-            });
-
-            // Process updates in batches
-            let lastPid = 0;
-            let batchCount = 0;
-            const MAX_BATCHES = 1000; // Safety limit for number of batches to prevent infinite loops
-            
-            while (batchCount < MAX_BATCHES) {
-                if (isCancelled) break;
-                
-                batchCount++;
-                const batch = await connection.query(
-                    'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
-                    [lastPid, BATCH_SIZE]
-                );
-
-                if (batch.rows.length === 0) break;
-
-                // Process the entire batch in a single efficient query
-                const lowStockThreshold = parseInt(defaultThresholds?.low_stock_threshold) || 5;
-                const criticalDays = parseInt(defaultThresholds?.critical_days) || 7;
-                const reorderDays = parseInt(defaultThresholds?.reorder_days) || 14;
-                const overstockDays = parseInt(defaultThresholds?.overstock_days) || 90;
-                const serviceLevel = parseFloat(finConfig?.service_level_z_score) || 1.96;
-                const defaultSafetyStock = parseInt(finConfig?.default_safety_stock) || 5;
-                const defaultReorderQty = parseInt(finConfig?.default_reorder_qty) || 5;
-                const orderCost = parseFloat(finConfig?.order_cost) || 25.00;
-                const holdingRate = parseFloat(finConfig?.holding_rate) || 0.25;
-                const minReorderQty = parseInt(finConfig?.min_reorder_qty) || 1;
-
-                await connection.query(`
-                    UPDATE product_metrics pm
-                    SET 
-                        inventory_value = p.stock_quantity * NULLIF(p.cost_price, 0),
-                        daily_sales_avg = COALESCE(sm.daily_sales_avg, 0),
-                        weekly_sales_avg = COALESCE(sm.weekly_sales_avg, 0),
-                        monthly_sales_avg = COALESCE(sm.monthly_sales_avg, 0),
-                        total_revenue = COALESCE(sm.total_revenue, 0),
-                        avg_margin_percent = COALESCE(sm.avg_margin_percent, 0),
-                        first_sale_date = sm.first_sale_date,
-                        last_sale_date = sm.last_sale_date,
-                        avg_lead_time_days = COALESCE(lm.avg_lead_time_days, 30.0),
-                        days_of_inventory = CASE 
-                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 
-                            THEN FLOOR(p.stock_quantity / NULLIF(sm.daily_sales_avg, 0))
-                            ELSE NULL 
-                        END,
-                        weeks_of_inventory = CASE 
-                            WHEN COALESCE(sm.weekly_sales_avg, 0) > 0 
-                            THEN FLOOR(p.stock_quantity / NULLIF(sm.weekly_sales_avg, 0))
-                            ELSE NULL 
-                        END,
-                        stock_status = CASE
-                            WHEN p.stock_quantity <= 0 THEN 'Out of Stock'
-                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 AND p.stock_quantity <= ${lowStockThreshold} THEN 'Low Stock'
-                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 THEN 'In Stock'
-                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${criticalDays} THEN 'Critical'
-                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ${reorderDays} THEN 'Reorder'
-                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays} THEN 'Overstocked'
-                            ELSE 'Healthy'
-                        END,
-                        safety_stock = CASE
-                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
-                                CEIL(
-                                    ${serviceLevel} * SQRT(
-                                        GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
-                                        POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
-                                    )
-                                )
-                            ELSE ${defaultSafetyStock}
-                        END,
-                        reorder_point = CASE
-                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
-                                CEIL(sm.daily_sales_avg * GREATEST(0, COALESCE(lm.avg_lead_time_days, 30.0))) + 
-                                (CASE
-                                    WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND COALESCE(lm.avg_lead_time_days, 0) > 0 THEN
-                                        CEIL(
-                                            ${serviceLevel} * SQRT(
-                                                GREATEST(0, COALESCE(lm.avg_lead_time_days, 0)) * POWER(COALESCE(sm.stddev_daily_sales, 0), 2) +
-                                                POWER(COALESCE(sm.daily_sales_avg, 0), 2) * POWER(COALESCE(lm.stddev_lead_time_days, 0), 2)
-                                            )
-                                        )
-                                    ELSE ${defaultSafetyStock}
-                                END)
-                            ELSE ${lowStockThreshold}
-                        END,
-                        reorder_qty = CASE
-                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND NULLIF(p.cost_price, 0) IS NOT NULL AND NULLIF(p.cost_price, 0) > 0 THEN
-                                GREATEST(
-                                    CEIL(SQRT(
-                                        (2 * (sm.daily_sales_avg * 365) * ${orderCost}) / 
-                                        NULLIF(p.cost_price * ${holdingRate}, 0) 
-                                    )),
-                                    ${minReorderQty}
-                                )
-                            ELSE ${defaultReorderQty}
-                        END,
-                        overstocked_amt = CASE
-                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ${overstockDays} 
-                            THEN GREATEST(0, p.stock_quantity - CEIL(sm.daily_sales_avg * ${overstockDays}))
-                            ELSE 0
-                        END,
-                        last_calculated_at = NOW()
-                    FROM products p
-                    LEFT JOIN temp_sales_metrics sm ON p.pid = sm.pid
-                    LEFT JOIN temp_purchase_metrics lm ON p.pid = lm.pid
-                    WHERE p.pid = ANY($1::BIGINT[])
-                    AND pm.pid = p.pid
-                `, [batch.rows.map(row => row.pid)]);
-
-                lastPid = batch.rows[batch.rows.length - 1].pid;
-                processedCount += batch.rows.length;
-
-                outputProgress({
-                    status: 'running',
-                    operation: 'Processing base metrics batch',
-                    current: processedCount,
-                    total: totalProducts,
-                    elapsed: formatElapsedTime(startTime),
-                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                    rate: calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                    timing: {
-                        start_time: new Date(startTime).toISOString(),
-                        end_time: new Date().toISOString(),
-                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                    }
-                });
-            }
-
-            // Add safety check if the loop processed MAX_BATCHES
-            if (batchCount >= MAX_BATCHES) {
-                logError(new Error(`Reached maximum batch count (${MAX_BATCHES}). Process may have entered an infinite loop.`), 'Batch processing safety limit reached');
-            }
-        }
-
-        // Calculate forecast accuracy and bias in batches
-        let forecastPid = 0;
-        while (true) {
-            if (isCancelled) break;
-
-            const forecastBatch = await connection.query(
-                'SELECT pid FROM products WHERE pid > $1 ORDER BY pid LIMIT $2',
-                [forecastPid, BATCH_SIZE]
-            );
-
-            if (forecastBatch.rows.length === 0) break;
-
-            const forecastPidArray = forecastBatch.rows.map(row => row.pid);
-
-            // Use array_to_string to convert the array to a string of comma-separated values
-            await connection.query(`
-                WITH forecast_metrics AS (
-                    SELECT 
-                        sf.pid,
-                        AVG(CASE 
-                            WHEN o.quantity > 0 
-                            THEN ABS(sf.forecast_quantity - o.quantity) / o.quantity * 100
-                            ELSE 100 
-                        END) as avg_forecast_error,
-                        AVG(CASE 
-                            WHEN o.quantity > 0 
-                            THEN (sf.forecast_quantity - o.quantity) / o.quantity * 100
-                            ELSE 0 
-                        END) as avg_forecast_bias,
-                        MAX(sf.forecast_date) as last_forecast_date
-                    FROM sales_forecasts sf
-                    JOIN orders o ON sf.pid = o.pid 
-                        AND DATE(o.date) = sf.forecast_date
-                    WHERE o.canceled = false
-                        AND sf.forecast_date >= CURRENT_DATE - INTERVAL '90 days'
-                        AND sf.pid = ANY('{${forecastPidArray.join(',')}}'::BIGINT[])
-                    GROUP BY sf.pid
-                )
-                UPDATE product_metrics pm
-                SET 
-                    forecast_accuracy = GREATEST(0, 100 - LEAST(fm.avg_forecast_error, 100)),
-                    forecast_bias = GREATEST(-100, LEAST(fm.avg_forecast_bias, 100)),
-                    last_forecast_date = fm.last_forecast_date,
-                    last_calculated_at = NOW()
-                FROM forecast_metrics fm
-                WHERE pm.pid = fm.pid
-            `);
-
-            forecastPid = forecastBatch.rows[forecastBatch.rows.length - 1].pid;
-        }
-
-        // Calculate product time aggregates
-        if (!SKIP_PRODUCT_TIME_AGGREGATES) {
-            outputProgress({
-                status: 'running',
-                operation: 'Starting product time aggregates calculation',
-                current: processedCount || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedCount || 0),
-                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-
-            // Note: The time-aggregates calculation has been moved to time-aggregates.js
-            // This module will not duplicate that functionality
-            processedCount = Math.floor(totalProducts * 0.6);
-            outputProgress({
-                status: 'running',
-                operation: 'Product time aggregates calculation delegated to time-aggregates module',
-                current: processedCount || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedCount || 0),
-                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-        } else {
-            processedCount = Math.floor(totalProducts * 0.6);
-            outputProgress({
-                status: 'running',
-                operation: 'Skipping product time aggregates calculation',
-                current: processedCount || 0,
-                total: totalProducts || 0,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
-                rate: calculateRate(startTime, processedCount || 0),
-                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-        }
-
-        // Calculate ABC classification
-        outputProgress({
-            status: 'running',
-            operation: 'Starting ABC classification',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,  // This module doesn't process POs
-            success
-        };
-
-        const abcConfig = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
-        const abcThresholds = abcConfig.rows[0] || { a_threshold: 20, b_threshold: 50 };
-        
-        // Extract values and ensure they are valid numbers
-        const aThreshold = parseFloat(abcThresholds.a_threshold) || 20;
-        const bThreshold = parseFloat(abcThresholds.b_threshold) || 50;
-
-        // First, create and populate the rankings table with an index
-        await connection.query('DROP TABLE IF EXISTS temp_revenue_ranks');
-        await connection.query(`
-            CREATE TEMPORARY TABLE temp_revenue_ranks (
-                pid BIGINT NOT NULL,
-                total_revenue DECIMAL(10,3),
-                rank_num INT,
-                dense_rank_num INT,
-                percentile DECIMAL(5,2),
-                total_count INT,
-                PRIMARY KEY (pid)
-            )
-        `);
-        await connection.query('CREATE INDEX ON temp_revenue_ranks (rank_num)');
-        await connection.query('CREATE INDEX ON temp_revenue_ranks (dense_rank_num)');
-        await connection.query('CREATE INDEX ON temp_revenue_ranks (percentile)');
-
-        // Calculate rankings with proper tie handling
-        await connection.query(`
-            INSERT INTO temp_revenue_ranks
-            WITH revenue_data AS (
-                SELECT 
-                    pid,
-                    total_revenue,
-                    COUNT(*) OVER () as total_count,
-                    PERCENT_RANK() OVER (ORDER BY total_revenue DESC) * 100 as percentile,
-                    RANK() OVER (ORDER BY total_revenue DESC) as rank_num,
-                    DENSE_RANK() OVER (ORDER BY total_revenue DESC) as dense_rank_num
-                FROM product_metrics 
-                WHERE total_revenue > 0
-            )
-            SELECT 
-                pid,
-                total_revenue,
-                rank_num,
-                dense_rank_num,
-                percentile,
-                total_count
-            FROM revenue_data
-        `);
-
-        // Get total count for percentage calculation
-        const rankingCount = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
-        const totalCount = parseInt(rankingCount.rows[0].total_count) || 1;
-
-        // Process updates in batches
-        let abcProcessedCount = 0;
-        const batchSize = 5000;
-        const maxPid = await connection.query('SELECT MAX(pid) as max_pid FROM products');
-        const maxProductId = parseInt(maxPid.rows[0].max_pid);
-
-        while (abcProcessedCount < maxProductId) {
-            if (isCancelled) return {
-                processedProducts: processedCount,
-                processedOrders,
-                processedPurchaseOrders: 0,
-                success
-            };
-
-            // Get a batch of PIDs that need updating
-            const pids = await connection.query(`
-                SELECT pm.pid 
-                FROM product_metrics pm
-                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
-                WHERE pm.pid > $1
-                    AND (pm.abc_class IS NULL 
-                    OR pm.abc_class != 
-                        CASE 
-                            WHEN tr.pid IS NULL THEN 'C'
-                            WHEN tr.percentile <= ${aThreshold} THEN 'A'
-                            WHEN tr.percentile <= ${bThreshold} THEN 'B'
-                            ELSE 'C'
-                        END)
-                ORDER BY pm.pid
-                LIMIT $2
-            `, [abcProcessedCount, batchSize]);
-
-            if (pids.rows.length === 0) break;
-
-            const pidValues = pids.rows.map(row => row.pid);
-
-            await connection.query(`
-                UPDATE product_metrics pm
-                SET abc_class = 
-                    CASE 
-                        WHEN tr.pid IS NULL THEN 'C'
-                        WHEN tr.percentile <= ${aThreshold} THEN 'A'
-                        WHEN tr.percentile <= ${bThreshold} THEN 'B'
-                        ELSE 'C'
-                    END,
-                    last_calculated_at = NOW()
-                FROM (SELECT pid, percentile FROM temp_revenue_ranks) tr
-                WHERE pm.pid = tr.pid AND pm.pid = ANY($1::BIGINT[])
-                OR (pm.pid = ANY($1::BIGINT[]) AND tr.pid IS NULL)
-            `, [pidValues]);
-
-            // Now update turnover rate with proper handling of zero inventory periods
-            await connection.query(`
-                UPDATE product_metrics pm
-                SET 
-                    turnover_rate = CASE 
-                        WHEN sales.avg_nonzero_stock > 0 AND sales.active_days > 0 
-                        THEN LEAST(
-                            (sales.total_sold / sales.avg_nonzero_stock) * (365.0 / sales.active_days),
-                            999.99
-                        )
-                        ELSE 0 
-                    END,
-                    last_calculated_at = NOW()
-                FROM (
-                    SELECT 
-                        o.pid,
-                        SUM(o.quantity) as total_sold,
-                        COUNT(DISTINCT DATE(o.date)) as active_days,
-                        AVG(CASE 
-                            WHEN p.stock_quantity > 0 THEN p.stock_quantity 
-                            ELSE NULL 
-                        END) as avg_nonzero_stock
-                    FROM orders o
-                    JOIN products p ON o.pid = p.pid
-                    WHERE o.canceled = false 
-                    AND o.date >= CURRENT_DATE - INTERVAL '90 days'
-                    AND o.pid = ANY($1::BIGINT[])
-                    GROUP BY o.pid
-                ) sales
-                WHERE pm.pid = sales.pid
-            `, [pidValues]);
-
-            abcProcessedCount = pids.rows[pids.rows.length - 1].pid;
-            
-            // Calculate progress proportionally to total products
-            processedCount = Math.floor(totalProducts * (0.60 + (abcProcessedCount / maxProductId) * 0.2));
-            
-            outputProgress({
-                status: 'running',
-                operation: 'ABC classification progress',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-        }
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('product_metrics', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount || 0,
-            processedOrders: processedOrders || 0,
-            processedPurchaseOrders: 0,  // This module doesn't process POs
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating product metrics');
-        throw error;
-    } finally {
-        // Always clean up temporary tables, even if an error occurred
-        if (connection) {
-            try {
-                await connection.query('DROP TABLE IF EXISTS temp_sales_metrics');
-                await connection.query('DROP TABLE IF EXISTS temp_purchase_metrics');
-            } catch (err) {
-                console.error('Error cleaning up temporary tables:', err);
-            }
-            
-            // Make sure to release the connection
-            connection.release();
-        }
-    }
-}
-
-function calculateStockStatus(stock, config, daily_sales_avg, weekly_sales_avg, monthly_sales_avg) {
-    if (stock <= 0) {
-        return 'Out of Stock';
-    }
-
-    // Use the most appropriate sales average based on data quality
-    let sales_avg = daily_sales_avg;
-    if (sales_avg === 0) {
-        sales_avg = weekly_sales_avg / 7;
-    }
-    if (sales_avg === 0) {
-        sales_avg = monthly_sales_avg / 30;
-    }
-
-    if (sales_avg === 0) {
-        return stock <= config.low_stock_threshold ? 'Low Stock' : 'In Stock';
-    }
-
-    const days_of_stock = stock / sales_avg;
-
-    if (days_of_stock <= config.critical_days) {
-        return 'Critical';
-    } else if (days_of_stock <= config.reorder_days) {
-        return 'Reorder';
-    } else if (days_of_stock > config.overstock_days) {
-        return 'Overstocked';
-    }
-
-    return 'Healthy';
-}
-
-// Note: calculateReorderQuantities function has been removed as its logic has been incorporated 
-// in the main SQL query with configurable parameters
-
-module.exports = calculateProductMetrics; 

inventory-server/old/metrics/sales-forecasts.js

@@ -1,440 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateSalesForecasts(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Sales forecasts calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders: 0,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // Get order count that will be processed
-        const orderCount = await connection.query(`
-            SELECT COUNT(*) as count 
-            FROM orders o 
-            WHERE o.canceled = false
-                AND o.date >= CURRENT_DATE - INTERVAL '90 days'
-        `);
-        processedOrders = parseInt(orderCount.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting sales forecasts calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // First, create a temporary table for forecast dates
-        await connection.query(`
-            CREATE TEMPORARY TABLE IF NOT EXISTS temp_forecast_dates (
-                forecast_date DATE,
-                day_of_week INT,
-                month INT,
-                PRIMARY KEY (forecast_date)
-            )
-        `);
-
-        await connection.query(`
-            INSERT INTO temp_forecast_dates
-            SELECT 
-                CURRENT_DATE + (n || ' days')::INTERVAL as forecast_date,
-                EXTRACT(DOW FROM CURRENT_DATE + (n || ' days')::INTERVAL) + 1 as day_of_week,
-                EXTRACT(MONTH FROM CURRENT_DATE + (n || ' days')::INTERVAL) as month
-            FROM (
-                SELECT a.n + b.n * 10 as n 
-                FROM 
-                    (SELECT 0 as n UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION
-                     SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) a,
-                    (SELECT 0 as n UNION SELECT 1 UNION SELECT 2) b
-                ORDER BY n
-                LIMIT 31
-            ) numbers
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.92);
-        outputProgress({
-            status: 'running',
-            operation: 'Forecast dates prepared, calculating daily sales stats',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Create temporary table for daily sales stats
-        await connection.query(`
-            CREATE TEMPORARY TABLE temp_daily_sales AS
-            SELECT 
-                o.pid,
-                EXTRACT(DOW FROM o.date) + 1 as day_of_week,
-                SUM(o.quantity) as daily_quantity,
-                SUM(o.price * o.quantity) as daily_revenue,
-                COUNT(DISTINCT DATE(o.date)) as day_count
-            FROM orders o
-            WHERE o.canceled = false
-                AND o.date >= CURRENT_DATE - INTERVAL '90 days'
-            GROUP BY o.pid, EXTRACT(DOW FROM o.date) + 1
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.94);
-        outputProgress({
-            status: 'running',
-            operation: 'Daily sales stats calculated, preparing product stats',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Create temporary table for product stats
-        await connection.query(`
-            CREATE TEMPORARY TABLE temp_product_stats AS
-            SELECT 
-                pid,
-                AVG(daily_revenue) as overall_avg_revenue,
-                SUM(day_count) as total_days
-            FROM temp_daily_sales
-            GROUP BY pid
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.96);
-        outputProgress({
-            status: 'running',
-            operation: 'Product stats prepared, calculating product-level forecasts',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Calculate product-level forecasts
-        await connection.query(`
-            INSERT INTO sales_forecasts (
-                pid,
-                forecast_date,
-                forecast_quantity,
-                confidence_level,
-                created_at
-            )
-            WITH daily_stats AS (
-                SELECT 
-                    ds.pid,
-                    AVG(ds.daily_quantity) as avg_daily_qty,
-                    STDDEV(ds.daily_quantity) as std_daily_qty,
-                    COUNT(DISTINCT ds.day_count) as data_points,
-                    SUM(ds.day_count) as total_days,
-                    AVG(ds.daily_revenue) as avg_daily_revenue,
-                    STDDEV(ds.daily_revenue) as std_daily_revenue,
-                    MIN(ds.daily_quantity) as min_daily_qty,
-                    MAX(ds.daily_quantity) as max_daily_qty,
-                    -- Calculate variance without using LAG
-                    COALESCE(
-                        STDDEV(ds.daily_quantity) / NULLIF(AVG(ds.daily_quantity), 0),
-                        0
-                    ) as daily_variance_ratio
-                FROM temp_daily_sales ds
-                GROUP BY ds.pid
-                HAVING AVG(ds.daily_quantity) > 0
-            )
-            SELECT 
-                ds.pid,
-                fd.forecast_date,
-                GREATEST(0, 
-                    ROUND(
-                        ds.avg_daily_qty * 
-                        (1 + COALESCE(sf.seasonality_factor, 0))
-                    )
-                ) as forecast_quantity,
-                CASE
-                    WHEN ds.total_days >= 60 AND ds.daily_variance_ratio < 0.5 THEN 90
-                    WHEN ds.total_days >= 60 THEN 85
-                    WHEN ds.total_days >= 30 AND ds.daily_variance_ratio < 0.5 THEN 80
-                    WHEN ds.total_days >= 30 THEN 75
-                    WHEN ds.total_days >= 14 AND ds.daily_variance_ratio < 0.5 THEN 70
-                    WHEN ds.total_days >= 14 THEN 65
-                    ELSE 60
-                END as confidence_level,
-                NOW() as created_at
-            FROM daily_stats ds
-            JOIN temp_product_stats ps ON ds.pid = ps.pid
-            CROSS JOIN temp_forecast_dates fd
-            LEFT JOIN sales_seasonality sf ON fd.month = sf.month
-            GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, sf.seasonality_factor, 
-                    ds.avg_daily_qty, ds.std_daily_qty, ds.avg_daily_qty, ds.total_days, ds.daily_variance_ratio
-            ON CONFLICT (pid, forecast_date) DO UPDATE
-            SET
-                forecast_quantity = EXCLUDED.forecast_quantity,
-                confidence_level = EXCLUDED.confidence_level,
-                created_at = NOW()
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.98);
-        outputProgress({
-            status: 'running',
-            operation: 'Product forecasts calculated, preparing category stats',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Create temporary table for category stats
-        await connection.query(`
-            CREATE TEMPORARY TABLE temp_category_sales AS
-            SELECT 
-                pc.cat_id,
-                EXTRACT(DOW FROM o.date) + 1 as day_of_week,
-                SUM(o.quantity) as daily_quantity,
-                SUM(o.price * o.quantity) as daily_revenue,
-                COUNT(DISTINCT DATE(o.date)) as day_count
-            FROM orders o
-            JOIN product_categories pc ON o.pid = pc.pid
-            WHERE o.canceled = false
-                AND o.date >= CURRENT_DATE - INTERVAL '90 days'
-            GROUP BY pc.cat_id, EXTRACT(DOW FROM o.date) + 1
-        `);
-
-        await connection.query(`
-            CREATE TEMPORARY TABLE temp_category_stats AS
-            SELECT 
-                cat_id,
-                AVG(daily_revenue) as overall_avg_revenue,
-                SUM(day_count) as total_days
-            FROM temp_category_sales
-            GROUP BY cat_id
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.99);
-        outputProgress({
-            status: 'running',
-            operation: 'Category stats prepared, calculating category-level forecasts',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-        // Calculate category-level forecasts
-        await connection.query(`
-            INSERT INTO category_forecasts (
-                category_id,
-                forecast_date,
-                forecast_units,
-                forecast_revenue,
-                confidence_level,
-                created_at
-            )
-            SELECT 
-                cs.cat_id::bigint as category_id,
-                fd.forecast_date,
-                GREATEST(0, 
-                    ROUND(AVG(cs.daily_quantity) * 
-                    (1 + COALESCE(sf.seasonality_factor, 0)))
-                ) as forecast_units,
-                GREATEST(0, 
-                    COALESCE(
-                        CASE 
-                            WHEN SUM(cs.day_count) >= 4 THEN AVG(cs.daily_revenue)
-                            ELSE ct.overall_avg_revenue
-                        END * 
-                        (1 + COALESCE(sf.seasonality_factor, 0)),
-                        0
-                    )
-                ) as forecast_revenue,
-                CASE
-                    WHEN ct.total_days >= 60 THEN 90
-                    WHEN ct.total_days >= 30 THEN 80
-                    WHEN ct.total_days >= 14 THEN 70
-                    ELSE 60
-                END as confidence_level,
-                NOW() as created_at
-            FROM temp_category_sales cs
-            JOIN temp_category_stats ct ON cs.cat_id = ct.cat_id
-            CROSS JOIN temp_forecast_dates fd
-            LEFT JOIN sales_seasonality sf ON fd.month = sf.month
-            GROUP BY 
-                cs.cat_id, 
-                fd.forecast_date, 
-                ct.overall_avg_revenue, 
-                ct.total_days, 
-                sf.seasonality_factor,
-                sf.month
-            HAVING AVG(cs.daily_quantity) > 0
-            ON CONFLICT (category_id, forecast_date) DO UPDATE
-            SET
-                forecast_units = EXCLUDED.forecast_units,
-                forecast_revenue = EXCLUDED.forecast_revenue,
-                confidence_level = EXCLUDED.confidence_level,
-                created_at = NOW()
-        `);
-
-        // Clean up temporary tables
-        await connection.query(`
-            DROP TABLE IF EXISTS temp_forecast_dates;
-            DROP TABLE IF EXISTS temp_daily_sales;
-            DROP TABLE IF EXISTS temp_product_stats;
-            DROP TABLE IF EXISTS temp_category_sales;
-            DROP TABLE IF EXISTS temp_category_stats;
-        `);
-
-        processedCount = Math.floor(totalProducts * 1.0);
-        outputProgress({
-            status: 'running',
-            operation: 'Category forecasts calculated and temporary tables cleaned up',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('sales_forecasts', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating sales forecasts');
-        throw error;
-    } finally {
-        if (connection) {
-            try {
-                // Ensure temporary tables are cleaned up
-                await connection.query(`
-                    DROP TABLE IF EXISTS temp_forecast_dates;
-                    DROP TABLE IF EXISTS temp_daily_sales;
-                    DROP TABLE IF EXISTS temp_product_stats;
-                    DROP TABLE IF EXISTS temp_category_sales;
-                    DROP TABLE IF EXISTS temp_category_stats;
-                `);
-            } catch (err) {
-                console.error('Error cleaning up temporary tables:', err);
-            }
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateSalesForecasts; 

inventory-server/old/metrics/time-aggregates.js

@@ -1,344 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateTimeAggregates(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Time aggregates calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders: 0,
-                processedPurchaseOrders: 0,
-                success
-            };
-        }
-
-        // Get order count that will be processed
-        const orderCount = await connection.query(`
-            SELECT COUNT(*) as count 
-            FROM orders o 
-            WHERE o.canceled = false
-        `);
-        processedOrders = parseInt(orderCount.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting time aggregates calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // Create a temporary table for end-of-month inventory values
-        await connection.query(`
-            CREATE TEMPORARY TABLE IF NOT EXISTS temp_monthly_inventory AS
-            WITH months AS (
-                -- Generate all year/month combinations for the last 12 months
-                SELECT 
-                    EXTRACT(YEAR FROM month_date)::INTEGER as year,
-                    EXTRACT(MONTH FROM month_date)::INTEGER as month,
-                    month_date as start_date,
-                    (month_date + INTERVAL '1 month'::interval - INTERVAL '1 day'::interval)::DATE as end_date
-                FROM (
-                    SELECT generate_series(
-                        DATE_TRUNC('month', CURRENT_DATE - INTERVAL '12 months'::interval)::DATE,
-                        DATE_TRUNC('month', CURRENT_DATE)::DATE,
-                        INTERVAL '1 month'::interval
-                    ) as month_date
-                ) dates
-            ),
-            monthly_inventory_calc AS (
-                SELECT 
-                    p.pid,
-                    m.year,
-                    m.month,
-                    m.end_date,
-                    p.stock_quantity as current_quantity,
-                    -- Calculate sold during period (before end_date)
-                    COALESCE(SUM(
-                        CASE 
-                            WHEN o.date <= m.end_date THEN o.quantity 
-                            ELSE 0 
-                        END
-                    ), 0) as sold_after_end_date,
-                    -- Calculate received during period (before end_date)
-                    COALESCE(SUM(
-                        CASE 
-                            WHEN po.received_date <= m.end_date THEN po.received 
-                            ELSE 0 
-                        END
-                    ), 0) as received_after_end_date,
-                    p.cost_price
-                FROM 
-                    products p
-                CROSS JOIN 
-                    months m
-                LEFT JOIN 
-                    orders o ON p.pid = o.pid 
-                    AND o.canceled = false 
-                    AND o.date > m.end_date
-                    AND o.date <= CURRENT_DATE
-                LEFT JOIN 
-                    purchase_orders po ON p.pid = po.pid 
-                    AND po.received_date IS NOT NULL 
-                    AND po.received_date > m.end_date
-                    AND po.received_date <= CURRENT_DATE
-                GROUP BY 
-                    p.pid, m.year, m.month, m.end_date, p.stock_quantity, p.cost_price
-            )
-            SELECT 
-                pid,
-                year,
-                month,
-                -- End of month quantity = current quantity - sold after + received after
-                GREATEST(0, current_quantity - sold_after_end_date + received_after_end_date) as end_of_month_quantity,
-                -- End of month inventory value
-                GREATEST(0, current_quantity - sold_after_end_date + received_after_end_date) * cost_price as end_of_month_value,
-                cost_price
-            FROM 
-                monthly_inventory_calc
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.40);
-        outputProgress({
-            status: 'running',
-            operation: 'Monthly inventory values calculated, processing time aggregates',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // Initial insert of time-based aggregates
-        await connection.query(`
-            INSERT INTO product_time_aggregates (
-                pid,
-                year,
-                month,
-                total_quantity_sold,
-                total_revenue,
-                total_cost,
-                order_count,
-                stock_received,
-                stock_ordered,
-                avg_price,
-                profit_margin,
-                inventory_value,
-                gmroi
-            )
-            WITH monthly_sales AS (
-                SELECT 
-                    o.pid,
-                    EXTRACT(YEAR FROM o.date::timestamp with time zone)::INTEGER as year,
-                    EXTRACT(MONTH FROM o.date::timestamp with time zone)::INTEGER as month,
-                    SUM(o.quantity) as total_quantity_sold,
-                    SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) as total_revenue,
-                    SUM(COALESCE(o.costeach, 0) * o.quantity) as total_cost,
-                    COUNT(DISTINCT o.order_number) as order_count,
-                    AVG(o.price - COALESCE(o.discount, 0)) as avg_price,
-                    CASE 
-                        WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) > 0 
-                        THEN ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - SUM(COALESCE(o.costeach, 0) * o.quantity)) 
-                            / SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
-                        ELSE 0 
-                    END as profit_margin,
-                    COUNT(DISTINCT DATE(o.date)) as active_days
-                FROM orders o
-                JOIN products p ON o.pid = p.pid
-                WHERE o.canceled = false
-                GROUP BY o.pid, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
-            ),
-            monthly_stock AS (
-                SELECT 
-                    pid,
-                    EXTRACT(YEAR FROM date::timestamp with time zone)::INTEGER as year,
-                    EXTRACT(MONTH FROM date::timestamp with time zone)::INTEGER as month,
-                    SUM(received) as stock_received,
-                    SUM(ordered) as stock_ordered
-                FROM purchase_orders
-                GROUP BY pid, EXTRACT(YEAR FROM date::timestamp with time zone), EXTRACT(MONTH FROM date::timestamp with time zone)
-            )
-            SELECT 
-                COALESCE(s.pid, ms.pid, mi.pid) as pid,
-                COALESCE(s.year, ms.year, mi.year) as year,
-                COALESCE(s.month, ms.month, mi.month) as month,
-                COALESCE(s.total_quantity_sold, 0)::INTEGER as total_quantity_sold,
-                COALESCE(s.total_revenue, 0)::DECIMAL(10,3) as total_revenue,
-                COALESCE(s.total_cost, 0)::DECIMAL(10,3) as total_cost,
-                COALESCE(s.order_count, 0)::INTEGER as order_count,
-                COALESCE(ms.stock_received, 0)::INTEGER as stock_received,
-                COALESCE(ms.stock_ordered, 0)::INTEGER as stock_ordered,
-                COALESCE(s.avg_price, 0)::DECIMAL(10,3) as avg_price,
-                COALESCE(s.profit_margin, 0)::DECIMAL(10,3) as profit_margin,
-                COALESCE(mi.end_of_month_value, 0)::DECIMAL(10,3) as inventory_value,
-                CASE 
-                    WHEN COALESCE(mi.end_of_month_value, 0) > 0 
-                    THEN (COALESCE(s.total_revenue, 0) - COALESCE(s.total_cost, 0)) 
-                        / NULLIF(COALESCE(mi.end_of_month_value, 0), 0)
-                    ELSE 0 
-                END::DECIMAL(10,3) as gmroi
-            FROM (
-                SELECT * FROM monthly_sales s
-                UNION ALL
-                SELECT 
-                    pid,
-                    year,
-                    month,
-                    0 as total_quantity_sold,
-                    0 as total_revenue,
-                    0 as total_cost,
-                    0 as order_count,
-                    NULL as avg_price,
-                    0 as profit_margin,
-                    0 as active_days
-                FROM monthly_stock ms
-                WHERE NOT EXISTS (
-                    SELECT 1 FROM monthly_sales s2
-                    WHERE s2.pid = ms.pid
-                    AND s2.year = ms.year
-                    AND s2.month = ms.month
-                )
-                UNION ALL
-                SELECT 
-                    pid,
-                    year,
-                    month,
-                    0 as total_quantity_sold,
-                    0 as total_revenue,
-                    0 as total_cost,
-                    0 as order_count,
-                    NULL as avg_price,
-                    0 as profit_margin,
-                    0 as active_days
-                FROM temp_monthly_inventory mi
-                WHERE NOT EXISTS (
-                    SELECT 1 FROM monthly_sales s3
-                    WHERE s3.pid = mi.pid
-                    AND s3.year = mi.year
-                    AND s3.month = mi.month
-                )
-                AND NOT EXISTS (
-                    SELECT 1 FROM monthly_stock ms3
-                    WHERE ms3.pid = mi.pid
-                    AND ms3.year = mi.year
-                    AND ms3.month = mi.month
-                )
-            ) s
-            LEFT JOIN monthly_stock ms 
-                ON s.pid = ms.pid 
-                AND s.year = ms.year 
-                AND s.month = ms.month
-            LEFT JOIN temp_monthly_inventory mi
-                ON s.pid = mi.pid
-                AND s.year = mi.year
-                AND s.month = mi.month
-            ON CONFLICT (pid, year, month) DO UPDATE
-            SET
-                total_quantity_sold = EXCLUDED.total_quantity_sold,
-                total_revenue = EXCLUDED.total_revenue,
-                total_cost = EXCLUDED.total_cost,
-                order_count = EXCLUDED.order_count,
-                stock_received = EXCLUDED.stock_received,
-                stock_ordered = EXCLUDED.stock_ordered,
-                avg_price = EXCLUDED.avg_price,
-                profit_margin = EXCLUDED.profit_margin,
-                inventory_value = EXCLUDED.inventory_value,
-                gmroi = EXCLUDED.gmroi
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.60);
-        outputProgress({
-            status: 'running',
-            operation: 'Base time aggregates calculated',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-        
-        // Clean up temporary tables
-        await connection.query('DROP TABLE IF EXISTS temp_monthly_inventory');
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('time_aggregates', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders: 0,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating time aggregates');
-        throw error;
-    } finally {
-        if (connection) {
-            try {
-                // Ensure temporary tables are cleaned up
-                await connection.query('DROP TABLE IF EXISTS temp_monthly_inventory');
-            } catch (err) {
-                console.error('Error cleaning up temporary tables:', err);
-            }
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateTimeAggregates; 

inventory-server/old/metrics/utils/db.js

@@ -1,39 +0,0 @@
-const { Pool } = require('pg');
-const path = require('path');
-require('dotenv').config({ path: path.resolve(__dirname, '../../..', '.env') });
-
-// Database configuration
-const dbConfig = {
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432,
-    ssl: process.env.DB_SSL === 'true',
-    // Add performance optimizations
-    max: 10, // connection pool max size
-    idleTimeoutMillis: 30000,
-    connectionTimeoutMillis: 60000
-};
-
-// Create a single pool instance to be reused
-const pool = new Pool(dbConfig);
-
-// Add event handlers for pool
-pool.on('error', (err, client) => {
-    console.error('Unexpected error on idle client', err);
-});
-
-async function getConnection() {
-    return await pool.connect();
-}
-
-async function closePool() {
-    await pool.end();
-}
-
-module.exports = {
-    dbConfig,
-    getConnection,
-    closePool
-}; 

inventory-server/old/metrics/utils/progress.js

@@ -1,158 +0,0 @@
-const fs = require('fs');
-const path = require('path');
-
-// Helper function to format elapsed time
-function formatElapsedTime(elapsed) {
-    // If elapsed is a timestamp, convert to elapsed milliseconds
-    if (elapsed instanceof Date || elapsed > 1000000000000) {
-        elapsed = Date.now() - elapsed;
-    } else {
-        // If elapsed is in seconds, convert to milliseconds
-        elapsed = elapsed * 1000;
-    }
-    
-    const seconds = Math.floor(elapsed / 1000);
-    const minutes = Math.floor(seconds / 60);
-    const hours = Math.floor(minutes / 60);
-    
-    if (hours > 0) {
-        return `${hours}h ${minutes % 60}m`;
-    } else if (minutes > 0) {
-        return `${minutes}m ${seconds % 60}s`;
-    } else {
-        return `${seconds}s`;
-    }
-}
-
-// Helper function to estimate remaining time
-function estimateRemaining(startTime, current, total) {
-    if (current === 0) return null;
-    const elapsed = Date.now() - startTime;
-    const rate = current / elapsed;
-    const remaining = (total - current) / rate;
-    
-    const minutes = Math.floor(remaining / 60000);
-    const seconds = Math.floor((remaining % 60000) / 1000);
-    
-    if (minutes > 0) {
-        return `${minutes}m ${seconds}s`;
-    } else {
-        return `${seconds}s`;
-    }
-}
-
-// Helper function to calculate rate
-function calculateRate(startTime, current) {
-    const elapsed = (Date.now() - startTime) / 1000; // Convert to seconds
-    return elapsed > 0 ? Math.round(current / elapsed) : 0;
-}
-
-// Set up logging
-const LOG_DIR = path.join(__dirname, '../../../logs');
-const ERROR_LOG = path.join(LOG_DIR, 'import-errors.log');
-const IMPORT_LOG = path.join(LOG_DIR, 'import.log');
-const STATUS_FILE = path.join(LOG_DIR, 'metrics-status.json');
-
-// Ensure log directory exists
-if (!fs.existsSync(LOG_DIR)) {
-    fs.mkdirSync(LOG_DIR, { recursive: true });
-}
-
-// Helper function to log errors
-function logError(error, context = '') {
-    const timestamp = new Date().toISOString();
-    const errorMessage = `[${timestamp}] ${context}\nError: ${error.message}\nStack: ${error.stack}\n\n`;
-    
-    // Log to error file
-    fs.appendFileSync(ERROR_LOG, errorMessage);
-    
-    // Also log to console
-    console.error(`\n${context}\nError: ${error.message}`);
-}
-
-// Helper function to log import progress
-function logImport(message) {
-    const timestamp = new Date().toISOString();
-    const logMessage = `[${timestamp}] ${message}\n`;
-    fs.appendFileSync(IMPORT_LOG, logMessage);
-}
-
-// Helper function to output progress
-function outputProgress(data) {
-    // Save progress to file for resumption
-    saveProgress(data);
-    // Format as SSE event
-    const event = {
-        progress: data
-    };
-    // Always send to stdout for frontend
-    process.stdout.write(JSON.stringify(event) + '\n');
-    
-    // Log significant events to disk
-    const isSignificant = 
-        // Operation starts
-        (data.operation && !data.current) ||
-        // Operation completions and errors
-        data.status === 'complete' || 
-        data.status === 'error' ||
-        // Major phase changes
-        data.operation?.includes('Starting ABC classification') ||
-        data.operation?.includes('Starting time-based aggregates') ||
-        data.operation?.includes('Starting vendor metrics');
-    
-    if (isSignificant) {
-        logImport(`${data.operation || 'Operation'}${data.message ? ': ' + data.message : ''}${data.error ? ' Error: ' + data.error : ''}${data.status ? ' Status: ' + data.status : ''}`);
-    }
-}
-
-function saveProgress(progress) {
-    try {
-        fs.writeFileSync(STATUS_FILE, JSON.stringify({
-            ...progress,
-            timestamp: Date.now()
-        }));
-    } catch (err) {
-        console.error('Failed to save progress:', err);
-    }
-}
-
-function clearProgress() {
-    try {
-        if (fs.existsSync(STATUS_FILE)) {
-            fs.unlinkSync(STATUS_FILE);
-        }
-    } catch (err) {
-        console.error('Failed to clear progress:', err);
-    }
-}
-
-function getProgress() {
-    try {
-        if (fs.existsSync(STATUS_FILE)) {
-            const progress = JSON.parse(fs.readFileSync(STATUS_FILE, 'utf8'));
-            // Check if the progress is still valid (less than 1 hour old)
-            if (progress.timestamp && Date.now() - progress.timestamp < 3600000) {
-                return progress;
-            } else {
-                // Clear old progress
-                clearProgress();
-            }
-        }
-    } catch (err) {
-        console.error('Failed to read progress:', err);
-        clearProgress();
-    }
-    return null;
-}
-
-module.exports = {
-    formatElapsedTime,
-    estimateRemaining,
-    calculateRate,
-    logError,
-    logImport,
-    outputProgress,
-    saveProgress,
-    clearProgress,
-    getProgress
-}; 

inventory-server/old/metrics/vendor-metrics.js

@@ -1,378 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
-const { getConnection } = require('./utils/db');
-
-async function calculateVendorMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
-    const connection = await getConnection();
-    let success = false;
-    let processedOrders = 0;
-    let processedPurchaseOrders = 0;
-    
-    try {
-        if (isCancelled) {
-            outputProgress({
-                status: 'cancelled',
-                operation: 'Vendor metrics calculation cancelled',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: null,
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-                }
-            });
-            return {
-                processedProducts: processedCount,
-                processedOrders,
-                processedPurchaseOrders,
-                success
-            };
-        }
-
-        // Get counts of records that will be processed
-        const [orderCountResult, poCountResult] = await Promise.all([
-            connection.query(`
-                SELECT COUNT(*) as count 
-                FROM orders o 
-                WHERE o.canceled = false
-            `),
-            connection.query(`
-                SELECT COUNT(*) as count 
-                FROM purchase_orders po 
-                WHERE po.status != 0
-            `)
-        ]);
-        processedOrders = parseInt(orderCountResult.rows[0].count);
-        processedPurchaseOrders = parseInt(poCountResult.rows[0].count);
-
-        outputProgress({
-            status: 'running',
-            operation: 'Starting vendor metrics calculation',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // First ensure all vendors exist in vendor_details
-        await connection.query(`
-            INSERT INTO vendor_details (vendor, status, created_at, updated_at)
-            SELECT DISTINCT 
-                vendor,
-                'active' as status,
-                NOW() as created_at,
-                NOW() as updated_at
-            FROM products
-            WHERE vendor IS NOT NULL
-            ON CONFLICT (vendor) DO NOTHING
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.8);
-        outputProgress({
-            status: 'running',
-            operation: 'Vendor details updated, calculating metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders,
-            success
-        };
-
-        // Now calculate vendor metrics
-        await connection.query(`
-            INSERT INTO vendor_metrics (
-                vendor,
-                total_revenue,
-                total_orders,
-                total_late_orders,
-                avg_lead_time_days,
-                on_time_delivery_rate,
-                order_fill_rate,
-                avg_order_value,
-                active_products,
-                total_products,
-                total_purchase_value,
-                avg_margin_percent,
-                status,
-                last_calculated_at
-            )
-            WITH vendor_sales AS (
-                SELECT 
-                    p.vendor,
-                    SUM(o.quantity * o.price) as total_revenue,
-                    COUNT(DISTINCT o.id) as total_orders,
-                    COUNT(DISTINCT p.pid) as active_products,
-                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
-                FROM products p
-                JOIN orders o ON p.pid = o.pid
-                WHERE o.canceled = false
-                    AND o.date >= CURRENT_DATE - INTERVAL '12 months'
-                GROUP BY p.vendor
-            ),
-            vendor_po AS (
-                SELECT 
-                    p.vendor,
-                    COUNT(DISTINCT CASE WHEN po.receiving_status = 40 THEN po.id END) as received_orders,
-                    COUNT(DISTINCT po.id) as total_orders,
-                    AVG(CASE 
-                        WHEN po.receiving_status = 40 
-                            AND po.received_date IS NOT NULL 
-                            AND po.date IS NOT NULL
-                        THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
-                        ELSE NULL
-                    END) as avg_lead_time_days,
-                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
-                FROM products p
-                JOIN purchase_orders po ON p.pid = po.pid
-                WHERE po.date >= CURRENT_DATE - INTERVAL '12 months'
-                GROUP BY p.vendor
-            ),
-            vendor_products AS (
-                SELECT 
-                    vendor,
-                    COUNT(DISTINCT pid) as total_products
-                FROM products
-                GROUP BY vendor
-            )
-            SELECT 
-                vs.vendor,
-                COALESCE(vs.total_revenue, 0) as total_revenue,
-                COALESCE(vp.total_orders, 0) as total_orders,
-                COALESCE(vp.total_orders - vp.received_orders, 0) as total_late_orders,
-                COALESCE(vp.avg_lead_time_days, 0) as avg_lead_time_days,
-                CASE 
-                    WHEN vp.total_orders > 0 
-                    THEN (vp.received_orders / vp.total_orders) * 100 
-                    ELSE 0 
-                END as on_time_delivery_rate,
-                CASE 
-                    WHEN vp.total_orders > 0 
-                    THEN (vp.received_orders / vp.total_orders) * 100 
-                    ELSE 0 
-                END as order_fill_rate,
-                CASE 
-                    WHEN vs.total_orders > 0 
-                    THEN vs.total_revenue / vs.total_orders 
-                    ELSE 0 
-                END as avg_order_value,
-                COALESCE(vs.active_products, 0) as active_products,
-                COALESCE(vpr.total_products, 0) as total_products,
-                COALESCE(vp.total_purchase_value, 0) as total_purchase_value,
-                CASE 
-                    WHEN vs.total_revenue > 0 
-                    THEN (vs.total_margin / vs.total_revenue) * 100 
-                    ELSE 0 
-                END as avg_margin_percent,
-                'active' as status,
-                NOW() as last_calculated_at
-            FROM vendor_sales vs
-            LEFT JOIN vendor_po vp ON vs.vendor = vp.vendor
-            LEFT JOIN vendor_products vpr ON vs.vendor = vpr.vendor
-            WHERE vs.vendor IS NOT NULL
-            ON CONFLICT (vendor) DO UPDATE
-            SET
-                total_revenue = EXCLUDED.total_revenue,
-                total_orders = EXCLUDED.total_orders,
-                total_late_orders = EXCLUDED.total_late_orders,
-                avg_lead_time_days = EXCLUDED.avg_lead_time_days,
-                on_time_delivery_rate = EXCLUDED.on_time_delivery_rate,
-                order_fill_rate = EXCLUDED.order_fill_rate,
-                avg_order_value = EXCLUDED.avg_order_value,
-                active_products = EXCLUDED.active_products,
-                total_products = EXCLUDED.total_products,
-                total_purchase_value = EXCLUDED.total_purchase_value,
-                avg_margin_percent = EXCLUDED.avg_margin_percent,
-                status = EXCLUDED.status,
-                last_calculated_at = EXCLUDED.last_calculated_at
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.9);
-        outputProgress({
-            status: 'running',
-            operation: 'Vendor metrics calculated, updating time-based metrics',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        if (isCancelled) return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders,
-            success
-        };
-
-        // Calculate time-based metrics
-        await connection.query(`
-            INSERT INTO vendor_time_metrics (
-                vendor,
-                year,
-                month,
-                total_orders,
-                late_orders,
-                avg_lead_time_days,
-                total_purchase_value,
-                total_revenue,
-                avg_margin_percent
-            )
-            WITH monthly_orders AS (
-                SELECT 
-                    p.vendor,
-                    EXTRACT(YEAR FROM o.date::timestamp with time zone) as year,
-                    EXTRACT(MONTH FROM o.date::timestamp with time zone) as month,
-                    COUNT(DISTINCT o.id) as total_orders,
-                    SUM(o.quantity * o.price) as total_revenue,
-                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
-                FROM products p
-                JOIN orders o ON p.pid = o.pid
-                WHERE o.canceled = false
-                    AND o.date >= CURRENT_DATE - INTERVAL '12 months'
-                    AND p.vendor IS NOT NULL
-                GROUP BY p.vendor, EXTRACT(YEAR FROM o.date::timestamp with time zone), EXTRACT(MONTH FROM o.date::timestamp with time zone)
-            ),
-            monthly_po AS (
-                SELECT 
-                    p.vendor,
-                    EXTRACT(YEAR FROM po.date::timestamp with time zone) as year,
-                    EXTRACT(MONTH FROM po.date::timestamp with time zone) as month,
-                    COUNT(DISTINCT po.id) as total_po,
-                    COUNT(DISTINCT CASE 
-                        WHEN po.receiving_status = 40 AND po.received_date > po.expected_date 
-                        THEN po.id 
-                    END) as late_orders,
-                    AVG(CASE 
-                        WHEN po.receiving_status = 40 
-                            AND po.received_date IS NOT NULL 
-                            AND po.date IS NOT NULL
-                        THEN EXTRACT(EPOCH FROM (po.received_date::timestamp with time zone - po.date::timestamp with time zone)) / 86400.0
-                        ELSE NULL
-                    END) as avg_lead_time_days,
-                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
-                FROM products p
-                JOIN purchase_orders po ON p.pid = po.pid
-                WHERE po.date >= CURRENT_DATE - INTERVAL '12 months'
-                    AND p.vendor IS NOT NULL
-                GROUP BY p.vendor, EXTRACT(YEAR FROM po.date::timestamp with time zone), EXTRACT(MONTH FROM po.date::timestamp with time zone)
-            )
-            SELECT 
-                mo.vendor,
-                mo.year,
-                mo.month,
-                COALESCE(mp.total_po, 0) as total_orders,
-                COALESCE(mp.late_orders, 0) as late_orders,
-                COALESCE(mp.avg_lead_time_days, 0) as avg_lead_time_days,
-                COALESCE(mp.total_purchase_value, 0) as total_purchase_value,
-                mo.total_revenue,
-                CASE 
-                    WHEN mo.total_revenue > 0 
-                    THEN (mo.total_margin / mo.total_revenue) * 100 
-                    ELSE 0 
-                END as avg_margin_percent
-            FROM monthly_orders mo
-            LEFT JOIN monthly_po mp ON mo.vendor = mp.vendor 
-                AND mo.year = mp.year 
-                AND mo.month = mp.month
-            UNION
-            SELECT 
-                mp.vendor,
-                mp.year,
-                mp.month,
-                mp.total_po as total_orders,
-                mp.late_orders,
-                mp.avg_lead_time_days,
-                mp.total_purchase_value,
-                0 as total_revenue,
-                0 as avg_margin_percent
-            FROM monthly_po mp
-            LEFT JOIN monthly_orders mo ON mp.vendor = mo.vendor 
-                AND mp.year = mo.year 
-                AND mp.month = mo.month
-            WHERE mo.vendor IS NULL
-            ON CONFLICT (vendor, year, month) DO UPDATE
-            SET
-                total_orders = EXCLUDED.total_orders,
-                late_orders = EXCLUDED.late_orders,
-                avg_lead_time_days = EXCLUDED.avg_lead_time_days,
-                total_purchase_value = EXCLUDED.total_purchase_value,
-                total_revenue = EXCLUDED.total_revenue,
-                avg_margin_percent = EXCLUDED.avg_margin_percent
-        `);
-
-        processedCount = Math.floor(totalProducts * 0.95);
-        outputProgress({
-            status: 'running',
-            operation: 'Time-based vendor metrics calculated',
-            current: processedCount,
-            total: totalProducts,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalProducts),
-            rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            }
-        });
-
-        // If we get here, everything completed successfully
-        success = true;
-        
-        // Update calculate_status
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ('vendor_metrics', NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = NOW()
-        `);
-
-        return {
-            processedProducts: processedCount,
-            processedOrders,
-            processedPurchaseOrders,
-            success
-        };
-
-    } catch (error) {
-        success = false;
-        logError(error, 'Error calculating vendor metrics');
-        throw error;
-    } finally {
-        if (connection) {
-            connection.release();
-        }
-    }
-}
-
-module.exports = calculateVendorMetrics;

inventory-server/old/old_csv/update-csv.js

@@ -1,180 +0,0 @@
-const path = require('path');
-const fs = require('fs');
-const axios = require('axios');
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../metrics/utils/progress');
-
-// Change working directory to script directory
-process.chdir(path.dirname(__filename));
-
-require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
-
-const FILES = [
-    {
-        name: '39f2x83-products.csv',
-        url: process.env.PRODUCTS_CSV_URL
-    },
-    {
-        name: '39f2x83-orders.csv',
-        url: process.env.ORDERS_CSV_URL
-    },
-    {
-        name: '39f2x83-purchase_orders.csv',
-        url: process.env.PURCHASE_ORDERS_CSV_URL
-    }
-];
-
-let isCancelled = false;
-
-function cancelUpdate() {
-    isCancelled = true;
-    outputProgress({
-        status: 'cancelled',
-        operation: 'CSV update cancelled',
-        current: 0,
-        total: FILES.length,
-        elapsed: null,
-        remaining: null,
-        rate: 0
-    });
-}
-
-async function downloadFile(file, index, startTime) {
-    if (isCancelled) return;
-
-    const csvDir = path.join(__dirname, '../csv');
-    if (!fs.existsSync(csvDir)) {
-        fs.mkdirSync(csvDir, { recursive: true });
-    }
-
-    const writer = fs.createWriteStream(path.join(csvDir, file.name));
-    
-    try {
-        const response = await axios({
-            url: file.url,
-            method: 'GET',
-            responseType: 'stream'
-        });
-
-        const totalLength = response.headers['content-length'];
-        let downloadedLength = 0;
-        let lastProgressUpdate = Date.now();
-        const PROGRESS_INTERVAL = 1000; // Update progress every second
-
-        response.data.on('data', (chunk) => {
-            if (isCancelled) {
-                writer.end();
-                return;
-            }
-
-            downloadedLength += chunk.length;
-            
-            // Update progress based on time interval
-            const now = Date.now();
-            if (now - lastProgressUpdate >= PROGRESS_INTERVAL) {
-                const progress = (downloadedLength / totalLength) * 100;
-                outputProgress({
-                    status: 'running',
-                    operation: `Downloading ${file.name}`,
-                    current: index + (downloadedLength / totalLength),
-                    total: FILES.length,
-                    elapsed: formatElapsedTime(startTime),
-                    remaining: estimateRemaining(startTime, index + (downloadedLength / totalLength), FILES.length),
-                    rate: calculateRate(startTime, index + (downloadedLength / totalLength)),
-                    percentage: progress.toFixed(1),
-                    file_progress: {
-                        name: file.name,
-                        downloaded: downloadedLength,
-                        total: totalLength,
-                        percentage: progress.toFixed(1)
-                    }
-                });
-                lastProgressUpdate = now;
-            }
-        });
-
-        response.data.pipe(writer);
-
-        return new Promise((resolve, reject) => {
-            writer.on('finish', resolve);
-            writer.on('error', reject);
-        });
-    } catch (error) {
-        fs.unlinkSync(path.join(csvDir, file.name));
-        throw error;
-    }
-}
-
-// Main function to update all files
-async function updateFiles() {
-    const startTime = Date.now();
-    
-    outputProgress({
-        status: 'running',
-        operation: 'Starting CSV update',
-        current: 0,
-        total: FILES.length,
-        elapsed: '0s',
-        remaining: null,
-        rate: 0,
-        percentage: '0'
-    });
-
-    try {
-        for (let i = 0; i < FILES.length; i++) {
-            if (isCancelled) {
-                return;
-            }
-
-            const file = FILES[i];
-            await downloadFile(file, i, startTime);
-
-            outputProgress({
-                status: 'running',
-                operation: 'CSV update in progress',
-                current: i + 1,
-                total: FILES.length,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, i + 1, FILES.length),
-                rate: calculateRate(startTime, i + 1),
-                percentage: (((i + 1) / FILES.length) * 100).toFixed(1)
-            });
-        }
-
-        outputProgress({
-            status: 'complete',
-            operation: 'CSV update complete',
-            current: FILES.length,
-            total: FILES.length,
-            elapsed: formatElapsedTime(startTime),
-            remaining: '0s',
-            rate: calculateRate(startTime, FILES.length),
-            percentage: '100'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'CSV update failed',
-            error: error.message,
-            current: 0,
-            total: FILES.length,
-            elapsed: formatElapsedTime(startTime),
-            remaining: null,
-            rate: 0
-        });
-        throw error;
-    }
-}
-
-// Run the update only if this is the main module
-if (require.main === module) {
-    updateFiles().catch((error) => {
-        console.error('Error updating CSV files:', error);
-        process.exit(1);
-    });
-}
-
-// Export the functions needed by the route
-module.exports = {
-    updateFiles,
-    cancelUpdate
-}; 

inventory-server/old/populate-initial-metrics.js

@@ -1,677 +0,0 @@
-const path = require('path');
-const fs = require('fs');
-const os = require('os'); // For detecting CPU cores
-
-// Get the base directory (the directory containing the inventory-server folder)
-const baseDir = path.resolve(__dirname, '../../..');
-
-// Load environment variables from the inventory-server directory
-require('dotenv').config({ path: path.resolve(__dirname, '../..', '.env') });
-
-// Configure statement timeout (30 minutes)
-const PG_STATEMENT_TIMEOUT_MS = 1800000;
-
-// Add error handler for uncaught exceptions
-process.on('uncaughtException', (error) => {
-    console.error('Uncaught Exception:', error);
-    process.exit(1);
-});
-
-// Add error handler for unhandled promise rejections
-process.on('unhandledRejection', (reason, promise) => {
-    console.error('Unhandled Rejection at:', promise, 'reason:', reason);
-    process.exit(1);
-});
-
-// Load progress module
-const progress = require('../scripts/metrics-new/utils/progress');
-
-// Store progress functions in global scope to ensure availability
-global.formatElapsedTime = progress.formatElapsedTime;
-global.estimateRemaining = progress.estimateRemaining;
-global.calculateRate = progress.calculateRate;
-global.outputProgress = progress.outputProgress;
-global.clearProgress = progress.clearProgress;
-global.getProgress = progress.getProgress;
-global.logError = progress.logError;
-
-// Load database module
-const { getConnection, closePool } = require('../scripts/metrics-new/utils/db');
-
-// Add cancel handler
-let isCancelled = false;
-let runningQueryPromise = null;
-
-function cancelCalculation() {
-    if (!isCancelled) {
-        isCancelled = true;
-        console.log('Calculation has been cancelled by user');
-        
-        // Store the query promise to potentially cancel it
-        const queryToCancel = runningQueryPromise;
-        if (queryToCancel) {
-            console.log('Attempting to cancel the running query...');
-        }
-        
-        // Force-terminate any query that's been running for more than 5 seconds
-        try {
-            const connection = getConnection();
-            connection.then(async (conn) => {
-                try {
-                    // Identify and terminate long-running queries from our application
-                    await conn.query(`
-                        SELECT pg_cancel_backend(pid) 
-                        FROM pg_stat_activity 
-                        WHERE query_start < now() - interval '5 seconds'
-                        AND application_name = 'populate_metrics'
-                        AND query NOT LIKE '%pg_cancel_backend%'
-                    `);
-                    
-                    // Release connection
-                    conn.release();
-                } catch (err) {
-                    console.error('Error during force cancellation:', err);
-                    conn.release();
-                }
-            }).catch(err => {
-                console.error('Could not get connection for cancellation:', err);
-            });
-        } catch (err) {
-            console.error('Failed to terminate running queries:', err);
-        }
-    }
-    
-    return {
-        success: true,
-        message: 'Calculation has been cancelled'
-    };
-}
-
-// Handle SIGTERM signal for cancellation
-process.on('SIGTERM', cancelCalculation);
-process.on('SIGINT', cancelCalculation);
-
-const calculateInitialMetrics = (client, onProgress) => {
-  return client.query(`
-    -- Truncate the existing metrics tables to ensure clean data
-    TRUNCATE TABLE public.daily_product_snapshots;
-    TRUNCATE TABLE public.product_metrics;
-
-    -- First let's create daily snapshots for all products with order activity
-    WITH SalesData AS (
-      SELECT
-        p.pid,
-        p.sku,
-        o.date::date AS order_date,
-        -- Count orders to ensure we only include products with real activity
-        COUNT(o.id) as order_count,
-        -- Aggregate Sales (Quantity > 0, Status not Canceled/Returned)
-        COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.quantity ELSE 0 END), 0) AS units_sold,
-        COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.price * o.quantity ELSE 0 END), 0.00) AS gross_revenue_unadjusted,
-        COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.discount ELSE 0 END), 0.00) AS discounts,
-        COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN COALESCE(o.costeach, p.landing_cost_price, p.cost_price) * o.quantity ELSE 0 END), 0.00) AS cogs,
-        COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN p.regular_price * o.quantity ELSE 0 END), 0.00) AS gross_regular_revenue,
-
-        -- Aggregate Returns (Quantity < 0 or Status = Returned)
-        COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN ABS(o.quantity) ELSE 0 END), 0) AS units_returned,
-        COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN o.price * ABS(o.quantity) ELSE 0 END), 0.00) AS returns_revenue
-      FROM public.products p
-      LEFT JOIN public.orders o ON p.pid = o.pid
-      GROUP BY p.pid, p.sku, o.date::date
-      HAVING COUNT(o.id) > 0 -- Only include products with actual orders
-    ),
-    ReceivingData AS (
-      SELECT
-        r.pid,
-        r.received_date::date AS receiving_date,
-        -- Count receiving documents to ensure we only include products with real activity
-        COUNT(DISTINCT r.receiving_id) as receiving_count,
-        -- Calculate received quantity for this day
-        SUM(r.received_quantity) AS units_received,
-        -- Calculate received cost for this day
-        SUM(r.received_quantity * r.unit_cost) AS cost_received
-      FROM public.receivings r
-      GROUP BY r.pid, r.received_date::date
-      HAVING COUNT(DISTINCT r.receiving_id) > 0 OR SUM(r.received_quantity) > 0
-    ),
-    -- Get current stock quantities
-    StockData AS (
-      SELECT
-        p.pid,
-        p.stock_quantity,
-        COALESCE(p.landing_cost_price, p.cost_price, 0.00) as effective_cost_price,
-        COALESCE(p.price, 0.00) as current_price,
-        COALESCE(p.regular_price, 0.00) as current_regular_price
-      FROM public.products p
-    ),
-    -- Combine sales and receiving dates to get all activity dates
-    DatePidCombos AS (
-      SELECT DISTINCT pid, order_date AS activity_date FROM SalesData
-      UNION
-      SELECT DISTINCT pid, receiving_date FROM ReceivingData
-    ),
-    -- Insert daily snapshots for all product-date combinations
-    SnapshotInsert AS (
-      INSERT INTO public.daily_product_snapshots (
-        snapshot_date,
-        pid,
-        sku,
-        eod_stock_quantity,
-        eod_stock_cost,
-        eod_stock_retail,
-        eod_stock_gross,
-        stockout_flag,
-        units_sold,
-        units_returned,
-        gross_revenue,
-        discounts,
-        returns_revenue,
-        net_revenue,
-        cogs,
-        gross_regular_revenue,
-        profit,
-        units_received,
-        cost_received,
-        calculation_timestamp
-      )
-      SELECT
-        d.activity_date AS snapshot_date,
-        d.pid,
-        p.sku,
-        -- Use current stock as approximation, since historical stock data is not available
-        s.stock_quantity AS eod_stock_quantity,
-        s.stock_quantity * s.effective_cost_price AS eod_stock_cost,
-        s.stock_quantity * s.current_price AS eod_stock_retail,
-        s.stock_quantity * s.current_regular_price AS eod_stock_gross,
-        (s.stock_quantity <= 0) AS stockout_flag,
-        -- Sales metrics
-        COALESCE(sd.units_sold, 0),
-        COALESCE(sd.units_returned, 0),
-        COALESCE(sd.gross_revenue_unadjusted, 0.00),
-        COALESCE(sd.discounts, 0.00),
-        COALESCE(sd.returns_revenue, 0.00),
-        COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00) AS net_revenue,
-        COALESCE(sd.cogs, 0.00),
-        COALESCE(sd.gross_regular_revenue, 0.00),
-        (COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00)) - COALESCE(sd.cogs, 0.00) AS profit,
-        -- Receiving metrics
-        COALESCE(rd.units_received, 0),
-        COALESCE(rd.cost_received, 0.00),
-        now() -- calculation timestamp
-      FROM DatePidCombos d
-      JOIN public.products p ON d.pid = p.pid
-      LEFT JOIN SalesData sd ON d.pid = sd.pid AND d.activity_date = sd.order_date
-      LEFT JOIN ReceivingData rd ON d.pid = rd.pid AND d.activity_date = rd.receiving_date
-      LEFT JOIN StockData s ON d.pid = s.pid
-      RETURNING pid, snapshot_date
-    ),
-    -- Now build the aggregated product metrics from the daily snapshots
-    MetricsInsert AS (
-      INSERT INTO public.product_metrics (
-        pid,
-        sku,
-        current_stock_quantity,
-        current_stock_cost,
-        current_stock_retail,
-        current_stock_msrp,
-        is_out_of_stock,
-        total_units_sold,
-        total_units_returned,
-        return_rate,
-        gross_revenue,
-        total_discounts,
-        total_returns,
-        net_revenue,
-        total_cogs,
-        total_gross_revenue,
-        total_profit,
-        profit_margin,
-        avg_daily_units,
-        reorder_point,
-        reorder_alert,
-        days_of_supply,
-        sales_velocity,
-        sales_velocity_score,
-        rank_by_revenue,
-        rank_by_quantity,
-        rank_by_profit,
-        total_received_quantity,
-        total_received_cost,
-        last_sold_date,
-        last_received_date,
-        days_since_last_sale,
-        days_since_last_received,
-        calculation_timestamp
-      )
-      SELECT
-        p.pid,
-        p.sku,
-        p.stock_quantity AS current_stock_quantity,
-        p.stock_quantity * COALESCE(p.landing_cost_price, p.cost_price, 0) AS current_stock_cost,
-        p.stock_quantity * COALESCE(p.price, 0) AS current_stock_retail,
-        p.stock_quantity * COALESCE(p.regular_price, 0) AS current_stock_msrp,
-        (p.stock_quantity <= 0) AS is_out_of_stock,
-        -- Aggregate metrics
-        COALESCE(SUM(ds.units_sold), 0) AS total_units_sold,
-        COALESCE(SUM(ds.units_returned), 0) AS total_units_returned,
-        CASE 
-          WHEN COALESCE(SUM(ds.units_sold), 0) > 0 
-          THEN COALESCE(SUM(ds.units_returned), 0)::float / NULLIF(COALESCE(SUM(ds.units_sold), 0), 0) 
-          ELSE 0 
-        END AS return_rate,
-        COALESCE(SUM(ds.gross_revenue), 0) AS gross_revenue,
-        COALESCE(SUM(ds.discounts), 0) AS total_discounts,
-        COALESCE(SUM(ds.returns_revenue), 0) AS total_returns,
-        COALESCE(SUM(ds.net_revenue), 0) AS net_revenue,
-        COALESCE(SUM(ds.cogs), 0) AS total_cogs,
-        COALESCE(SUM(ds.gross_regular_revenue), 0) AS total_gross_revenue,
-        COALESCE(SUM(ds.profit), 0) AS total_profit,
-        CASE 
-          WHEN COALESCE(SUM(ds.net_revenue), 0) > 0 
-          THEN COALESCE(SUM(ds.profit), 0) / NULLIF(COALESCE(SUM(ds.net_revenue), 0), 0) 
-          ELSE 0 
-        END AS profit_margin,
-        -- Calculate average daily units
-        COALESCE(AVG(ds.units_sold), 0) AS avg_daily_units,
-        -- Calculate reorder point (simplified, can be enhanced with lead time and safety stock)
-        CEILING(COALESCE(AVG(ds.units_sold) * 14, 0)) AS reorder_point,
-        (p.stock_quantity <= CEILING(COALESCE(AVG(ds.units_sold) * 14, 0))) AS reorder_alert,
-        -- Days of supply based on average daily sales
-        CASE 
-          WHEN COALESCE(AVG(ds.units_sold), 0) > 0 
-          THEN p.stock_quantity / NULLIF(COALESCE(AVG(ds.units_sold), 0), 0) 
-          ELSE NULL 
-        END AS days_of_supply,
-        -- Sales velocity (average units sold per day over last 30 days)
-        (SELECT COALESCE(AVG(recent.units_sold), 0)
-         FROM public.daily_product_snapshots recent
-         WHERE recent.pid = p.pid
-         AND recent.snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
-        ) AS sales_velocity,
-        -- Placeholder for sales velocity score (can be calculated based on velocity)
-        0 AS sales_velocity_score,
-        -- Will be updated later by ranking procedure
-        0 AS rank_by_revenue,
-        0 AS rank_by_quantity,
-        0 AS rank_by_profit,
-        -- Receiving data
-        COALESCE(SUM(ds.units_received), 0) AS total_received_quantity,
-        COALESCE(SUM(ds.cost_received), 0) AS total_received_cost,
-        -- Date metrics
-        (SELECT MAX(sd.snapshot_date)
-         FROM public.daily_product_snapshots sd
-         WHERE sd.pid = p.pid AND sd.units_sold > 0
-        ) AS last_sold_date,
-        (SELECT MAX(rd.snapshot_date)
-         FROM public.daily_product_snapshots rd
-         WHERE rd.pid = p.pid AND rd.units_received > 0
-        ) AS last_received_date,
-        -- Calculate days since last sale/received
-        CASE 
-          WHEN (SELECT MAX(sd.snapshot_date)
-                FROM public.daily_product_snapshots sd
-                WHERE sd.pid = p.pid AND sd.units_sold > 0) IS NOT NULL
-          THEN (CURRENT_DATE - (SELECT MAX(sd.snapshot_date)
-                                FROM public.daily_product_snapshots sd
-                                WHERE sd.pid = p.pid AND sd.units_sold > 0))::integer
-          ELSE NULL
-        END AS days_since_last_sale,
-        CASE 
-          WHEN (SELECT MAX(rd.snapshot_date)
-                FROM public.daily_product_snapshots rd
-                WHERE rd.pid = p.pid AND rd.units_received > 0) IS NOT NULL
-          THEN (CURRENT_DATE - (SELECT MAX(rd.snapshot_date)
-                               FROM public.daily_product_snapshots rd
-                               WHERE rd.pid = p.pid AND rd.units_received > 0))::integer
-          ELSE NULL
-        END AS days_since_last_received,
-        now() -- calculation timestamp
-      FROM public.products p
-      LEFT JOIN public.daily_product_snapshots ds ON p.pid = ds.pid
-      GROUP BY p.pid, p.sku, p.stock_quantity, p.landing_cost_price, p.cost_price, p.price, p.regular_price
-    )
-    
-    -- Update the calculate_status table
-    INSERT INTO public.calculate_status (module_name, last_calculation_timestamp)
-    VALUES 
-      ('daily_snapshots', now()),
-      ('product_metrics', now())
-    ON CONFLICT (module_name) DO UPDATE 
-    SET last_calculation_timestamp = now();
-    
-    -- Finally, update the ranks for products
-    UPDATE public.product_metrics pm SET
-      rank_by_revenue = rev_ranks.rank
-    FROM (
-      SELECT pid, RANK() OVER (ORDER BY net_revenue DESC) AS rank
-      FROM public.product_metrics
-      WHERE net_revenue > 0
-    ) rev_ranks
-    WHERE pm.pid = rev_ranks.pid;
-    
-    UPDATE public.product_metrics pm SET
-      rank_by_quantity = qty_ranks.rank
-    FROM (
-      SELECT pid, RANK() OVER (ORDER BY total_units_sold DESC) AS rank
-      FROM public.product_metrics
-      WHERE total_units_sold > 0
-    ) qty_ranks
-    WHERE pm.pid = qty_ranks.pid;
-    
-    UPDATE public.product_metrics pm SET
-      rank_by_profit = profit_ranks.rank
-    FROM (
-      SELECT pid, RANK() OVER (ORDER BY total_profit DESC) AS rank
-      FROM public.product_metrics
-      WHERE total_profit > 0
-    ) profit_ranks
-    WHERE pm.pid = profit_ranks.pid;
-    
-    -- Return count of products with metrics
-    SELECT COUNT(*) AS product_count FROM public.product_metrics
-  `);
-};
-
-async function populateInitialMetrics() {
-    let connection;
-    const startTime = Date.now();
-    let calculateHistoryId;
-    
-    try {
-        // Clean up any previously running calculations
-        connection = await getConnection({
-            // Add performance-related settings
-            application_name: 'populate_metrics',
-            statement_timeout: PG_STATEMENT_TIMEOUT_MS, // 30 min timeout per statement
-        });
-        
-        // Ensure the calculate_status table exists and has the correct structure
-        await connection.query(`
-            CREATE TABLE IF NOT EXISTS calculate_status (
-                module_name TEXT PRIMARY KEY,
-                last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
-            )
-        `);
-        
-        await connection.query(`
-            UPDATE calculate_history 
-            SET 
-                status = 'cancelled',
-                end_time = NOW(),
-                duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
-                error_message = 'Previous calculation was not completed properly'
-            WHERE status = 'running' AND additional_info->>'type' = 'populate_initial_metrics'
-        `);
-
-        // Create history record for this calculation
-        const historyResult = await connection.query(`
-            INSERT INTO calculate_history (
-                start_time,
-                status,
-                additional_info
-            ) VALUES (
-                NOW(),
-                'running',
-                jsonb_build_object(
-                    'type', 'populate_initial_metrics',
-                    'sql_file', 'populate_initial_product_metrics.sql'
-                )
-            ) RETURNING id
-        `);
-        calculateHistoryId = historyResult.rows[0].id;
-
-        // Initialize progress
-        global.outputProgress({
-            status: 'running',
-            operation: 'Starting initial product metrics population',
-            current: 0,
-            total: 100,
-            elapsed: '0s',
-            remaining: 'Calculating... (this may take a while)',
-            rate: 0,
-            percentage: '0',
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            },
-            historyId: calculateHistoryId
-        });
-
-        // Prepare the database - analyze tables
-        global.outputProgress({
-            status: 'running',
-            operation: 'Analyzing database tables for better query performance',
-            current: 2,
-            total: 100,
-            elapsed: global.formatElapsedTime(startTime),
-            remaining: 'Analyzing...',
-            rate: 0,
-            percentage: '2',
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            },
-            historyId: calculateHistoryId
-        });
-        
-        // Enable better query planning and parallel operations
-        await connection.query(`
-            -- Analyze tables for better query planning
-            ANALYZE public.products;
-            ANALYZE public.purchase_orders;
-            ANALYZE public.daily_product_snapshots;
-            ANALYZE public.orders;
-            
-            -- Enable parallel operations
-            SET LOCAL enable_parallel_append = on;
-            SET LOCAL enable_parallel_hash = on;
-            SET LOCAL max_parallel_workers_per_gather = 4;
-            
-            -- Larger work memory for complex sorts/joins
-            SET LOCAL work_mem = '128MB';
-        `).catch(err => {
-            // Non-fatal if analyze fails
-            console.warn('Failed to analyze tables (non-fatal):', err.message);
-        });
-
-        // Execute the SQL query
-        global.outputProgress({
-            status: 'running',
-            operation: 'Executing initial metrics SQL query',
-            current: 5,
-            total: 100,
-            elapsed: global.formatElapsedTime(startTime),
-            remaining: 'Calculating... (this could take several hours with 150M+ records)',
-            rate: 0,
-            percentage: '5',
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            },
-            historyId: calculateHistoryId
-        });
-
-        // Read the SQL file
-        const sqlFilePath = path.resolve(__dirname, 'populate_initial_product_metrics.sql');
-        console.log('Base directory:', baseDir);
-        console.log('Script directory:', __dirname);
-        console.log('SQL file path:', sqlFilePath);
-        console.log('Current working directory:', process.cwd());
-        
-        if (!fs.existsSync(sqlFilePath)) {
-            throw new Error(`SQL file not found at ${sqlFilePath}`);
-        }
-        
-        // Read and clean up the SQL (Slightly more robust cleaning)
-        const sqlQuery = fs.readFileSync(sqlFilePath, 'utf8')
-            .replace(/\r\n/g, '\n') // Handle Windows endings
-            .replace(/\r/g, '\n')   // Handle old Mac endings
-            .trim(); // Remove leading/trailing whitespace VERY IMPORTANT
-
-        // Log details again AFTER cleaning
-        console.log('SQL Query length (cleaned):', sqlQuery.length);
-        console.log('SQL Query structure validation:');
-        console.log('- Contains DO block:', sqlQuery.includes('DO $$') || sqlQuery.includes('DO $')); // Check both types of tag start
-        console.log('- Contains BEGIN:', sqlQuery.includes('BEGIN'));
-        console.log('- Contains END:', sqlQuery.includes('END $$;') || sqlQuery.includes('END $')); // Check both types of tag end
-        console.log('- First 50 chars:', JSON.stringify(sqlQuery.slice(0, 50)));
-        console.log('- Last 100 chars (cleaned):', JSON.stringify(sqlQuery.slice(-100)));
-        
-        // Final check to ensure clean SQL ending
-        if (!sqlQuery.endsWith('END $$;')) {
-            console.warn('WARNING: SQL does not end with "END $$;". This might cause issues.');
-            console.log('Exact ending:', JSON.stringify(sqlQuery.slice(-20)));
-        }
-        
-        // Execute the script
-        console.log('Starting initial product metrics population...');
-        
-        // Track the query promise for potential cancellation
-        runningQueryPromise = connection.query({
-            text: sqlQuery,
-            rowMode: 'array'
-        });
-        await runningQueryPromise;
-        runningQueryPromise = null;
-
-        // Update progress to 100%
-        global.outputProgress({
-            status: 'complete',
-            operation: 'Initial product metrics population complete',
-            current: 100,
-            total: 100,
-            elapsed: global.formatElapsedTime(startTime),
-            remaining: '0s',
-            rate: 0,
-            percentage: '100',
-            timing: {
-                start_time: new Date(startTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
-            },
-            historyId: calculateHistoryId
-        });
-
-        // Update history with completion
-        await connection.query(`
-            UPDATE calculate_history 
-            SET 
-                end_time = NOW(),
-                duration_seconds = $1,
-                status = 'completed'
-            WHERE id = $2
-        `, [Math.round((Date.now() - startTime) / 1000), calculateHistoryId]);
-
-        // Clear progress file on successful completion
-        global.clearProgress();
-
-        return {
-            success: true,
-            message: 'Initial product metrics population completed successfully',
-            duration: Math.round((Date.now() - startTime) / 1000)
-        };
-    } catch (error) {
-        const endTime = Date.now();
-        const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
-
-        // Enhanced error logging
-        console.error('Error details:', {
-            message: error.message,
-            code: error.code,
-            hint: error.hint,
-            position: error.position,
-            detail: error.detail,
-            where: error.where ? error.where.substring(0, 500) + '...' : undefined, // Truncate to avoid huge logs
-            severity: error.severity,
-            file: error.file,
-            line: error.line,
-            routine: error.routine
-        });
-
-        // Update history with error
-        if (connection && calculateHistoryId) {
-            await connection.query(`
-                UPDATE calculate_history 
-                SET 
-                    end_time = NOW(),
-                    duration_seconds = $1,
-                    status = $2,
-                    error_message = $3
-                WHERE id = $4
-            `, [
-                totalElapsedSeconds,
-                isCancelled ? 'cancelled' : 'failed',
-                error.message,
-                calculateHistoryId
-            ]);
-        }
-
-        if (isCancelled) {
-            global.outputProgress({
-                status: 'cancelled',
-                operation: 'Calculation cancelled',
-                current: 50,
-                total: 100,
-                elapsed: global.formatElapsedTime(startTime),
-                remaining: null,
-                rate: 0,
-                percentage: '50',
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: totalElapsedSeconds
-                },
-                historyId: calculateHistoryId
-            });
-        } else {
-            global.outputProgress({
-                status: 'error',
-                operation: 'Error during initial product metrics population',
-                message: error.message,
-                current: 0,
-                total: 100,
-                elapsed: global.formatElapsedTime(startTime),
-                remaining: null,
-                rate: 0,
-                percentage: '0',
-                timing: {
-                    start_time: new Date(startTime).toISOString(),
-                    end_time: new Date().toISOString(),
-                    elapsed_seconds: totalElapsedSeconds
-                },
-                historyId: calculateHistoryId
-            });
-        }
-
-        console.error('Error during initial product metrics population:', error);
-        return {
-            success: false,
-            error: error.message,
-            duration: totalElapsedSeconds
-        };
-    } finally {
-        if (connection) {
-            connection.release();
-        }
-        await closePool();
-    }
-}
-
-// Start population process
-populateInitialMetrics()
-    .then(result => {
-        if (result.success) {
-            console.log(`Initial product metrics population completed successfully in ${result.duration} seconds`);
-            process.exit(0);
-        } else {
-            console.error(`Initial product metrics population failed: ${result.error}`);
-            process.exit(1);
-        }
-    })
-    .catch(err => {
-        console.error('Unexpected error:', err);
-        process.exit(1);
-    }); 

inventory-server/old/psql-csv-import.sh

@@ -1,428 +0,0 @@
-#!/bin/bash
-
-# Simple script to import CSV to PostgreSQL using psql
-# Usage: ./psql-csv-import.sh <csv-file> <table-name> [start-batch]
-
-# Exit on error
-set -e
-
-# Get arguments
-CSV_FILE=$1
-TABLE_NAME=$2
-BATCH_SIZE=500000  # Process 500,000 rows at a time
-START_BATCH=${3:-1}  # Optional third parameter to start from a specific batch
-
-if [ -z "$CSV_FILE" ] || [ -z "$TABLE_NAME" ]; then
-  echo "Usage: ./psql-csv-import.sh <csv-file> <table-name> [start-batch]"
-  exit 1
-fi
-
-# Check if file exists (only needed for batch 1)
-if [ "$START_BATCH" -eq 1 ] && [ ! -f "$CSV_FILE" ]; then
-  echo "Error: CSV file '$CSV_FILE' not found"
-  exit 1
-fi
-
-# Load environment variables
-if [ -f "../.env" ]; then
-  source "../.env"
-else
-  echo "Warning: .env file not found, using default connection parameters"
-fi
-
-# Set default connection parameters if not from .env
-DB_HOST=${DB_HOST:-localhost}
-DB_PORT=${DB_PORT:-5432}
-DB_NAME=${DB_NAME:-inventory_db}
-DB_USER=${DB_USER:-postgres}
-export PGPASSWORD=${DB_PASSWORD:-}  # Export password for psql
-
-# Common psql parameters
-PSQL_OPTS="-h $DB_HOST -p $DB_PORT -U $DB_USER -d $DB_NAME"
-
-# Function to clean up database state
-cleanup_and_optimize() {
-  echo "Cleaning up and optimizing database state..."
-  
-  # Analyze the target table to update statistics
-  psql $PSQL_OPTS -c "ANALYZE $TABLE_NAME;"
-  
-  # Perform vacuum to reclaim space and update stats
-  psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
-  
-  # Reset connection pool
-  psql $PSQL_OPTS -c "SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = current_database() AND pid <> pg_backend_pid();"
-  
-  # Clean up shared memory
-  psql $PSQL_OPTS -c "DISCARD ALL;"
-  
-  echo "Optimization complete."
-}
-
-# Show connection info
-echo "Importing $CSV_FILE into $TABLE_NAME"
-echo "Database: $DB_NAME on $DB_HOST:$DB_PORT with batch size: $BATCH_SIZE starting at batch $START_BATCH"
-
-# Start timer
-START_TIME=$(date +%s)
-
-# Create progress tracking file
-PROGRESS_FILE="/tmp/import_progress_${TABLE_NAME}.txt"
-touch "$PROGRESS_FILE"
-echo "Starting import at $(date), batch $START_BATCH" >> "$PROGRESS_FILE"
-
-# If we're resuming, run cleanup first
-if [ "$START_BATCH" -gt 1 ]; then
-  cleanup_and_optimize
-fi
-
-# For imported_product_stat_history, use optimized approach with hardcoded column names
-if [ "$TABLE_NAME" = "imported_product_stat_history" ]; then
-  echo "Using optimized import for $TABLE_NAME"
-  
-  # Only drop constraints/indexes and create staging table for batch 1
-  if [ "$START_BATCH" -eq 1 ]; then
-    # Extract CSV header
-    CSV_HEADER=$(head -n 1 "$CSV_FILE")
-    echo "CSV header: $CSV_HEADER"
-  
-    # Step 1: Drop constraints and indexes
-    echo "Dropping constraints and indexes..."
-    psql $PSQL_OPTS -c "
-    DO \$\$
-    DECLARE
-      constraint_name TEXT;
-    BEGIN
-      -- Drop primary key constraint if exists
-      SELECT conname INTO constraint_name
-      FROM pg_constraint 
-      WHERE conrelid = '$TABLE_NAME'::regclass AND contype = 'p';
-      
-      IF FOUND THEN
-        EXECUTE 'ALTER TABLE $TABLE_NAME DROP CONSTRAINT IF EXISTS ' || constraint_name;
-        RAISE NOTICE 'Dropped primary key constraint: %', constraint_name;
-      END IF;
-    END \$\$;
-    "
-    
-    # Drop all indexes on the table
-    psql $PSQL_OPTS -c "
-    DO \$\$
-    DECLARE
-      index_name TEXT;
-      index_record RECORD;
-    BEGIN
-      FOR index_record IN 
-        SELECT indexname 
-        FROM pg_indexes 
-        WHERE tablename = '$TABLE_NAME' 
-      LOOP
-        EXECUTE 'DROP INDEX IF EXISTS ' || index_record.indexname;
-        RAISE NOTICE 'Dropped index: %', index_record.indexname;
-      END LOOP;
-    END \$\$;
-    "
-    
-    # Step 2: Set maintenance_work_mem and disable triggers
-    echo "Setting maintenance_work_mem and disabling triggers..."
-    psql $PSQL_OPTS -c "
-    SET maintenance_work_mem = '1GB';
-    ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;
-    "
-    
-    # Step 3: Create staging table
-    echo "Creating staging table..."
-    psql $PSQL_OPTS -c "
-    DROP TABLE IF EXISTS staging_import;
-    CREATE UNLOGGED TABLE staging_import (
-      pid TEXT,
-      date TEXT,
-      score TEXT,
-      score2 TEXT,
-      qty_in_baskets TEXT,
-      qty_sold TEXT,
-      notifies_set TEXT,
-      visibility_score TEXT,
-      health_score TEXT,
-      sold_view_score TEXT
-    );
-    
-    -- Create an index on staging_import to improve OFFSET performance
-    CREATE INDEX ON staging_import (pid);
-    "
-    
-    # Step 4: Import CSV into staging table
-    echo "Importing CSV into staging table..."
-    psql $PSQL_OPTS -c "\copy staging_import FROM '$CSV_FILE' WITH CSV HEADER DELIMITER ','"
-  else
-    echo "Resuming import from batch $START_BATCH - skipping table creation and CSV import"
-    
-    # Check if staging table exists
-    STAGING_EXISTS=$(psql $PSQL_OPTS -t -c "SELECT EXISTS(SELECT 1 FROM pg_tables WHERE tablename='staging_import');" | tr -d '[:space:]')
-    
-    if [ "$STAGING_EXISTS" != "t" ]; then
-      echo "Error: Staging table 'staging_import' does not exist. Run without batch parameter first."
-      exit 1
-    fi
-    
-    # Ensure triggers are disabled
-    psql $PSQL_OPTS -c "ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;"
-    
-    # Optimize PostgreSQL for better performance
-    psql $PSQL_OPTS -c "
-    -- Increase work mem for this session
-    SET work_mem = '256MB';
-    SET maintenance_work_mem = '1GB';
-    "
-  fi
-  
-  # Step 5: Get total row count
-  TOTAL_ROWS=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM staging_import;" | tr -d '[:space:]')
-  echo "Total rows to import: $TOTAL_ROWS"
-  
-  # Calculate starting point
-  PROCESSED=$(( ($START_BATCH - 1) * $BATCH_SIZE ))
-  if [ $PROCESSED -ge $TOTAL_ROWS ]; then
-    echo "Error: Start batch $START_BATCH is beyond the available rows ($TOTAL_ROWS)"
-    exit 1
-  fi
-  
-  # Step 6: Process in batches with shell loop
-  BATCH_NUM=$(( $START_BATCH - 1 ))
-  
-  # We'll process batches in chunks of 10 before cleaning up
-  CHUNKS_SINCE_CLEANUP=0
-  
-  while [ $PROCESSED -lt $TOTAL_ROWS ]; do
-    BATCH_NUM=$(( $BATCH_NUM + 1 ))
-    BATCH_START=$(date +%s)
-    MAX_ROWS=$(( $PROCESSED + $BATCH_SIZE ))
-    if [ $MAX_ROWS -gt $TOTAL_ROWS ]; then
-      MAX_ROWS=$TOTAL_ROWS
-    fi
-    
-    echo "Processing batch $BATCH_NUM (rows $PROCESSED to $MAX_ROWS)..."
-    
-    # Optimize query buffer for this batch
-    psql $PSQL_OPTS -c "SET work_mem = '256MB';"
-    
-    # Insert batch with type casts
-    psql $PSQL_OPTS -c "
-    INSERT INTO $TABLE_NAME (
-      pid, date, score, score2, qty_in_baskets, qty_sold, 
-      notifies_set, visibility_score, health_score, sold_view_score
-    )
-    SELECT 
-      pid::bigint, 
-      date::date, 
-      score::numeric, 
-      score2::numeric,
-      qty_in_baskets::smallint,
-      qty_sold::smallint,
-      notifies_set::smallint,
-      visibility_score::numeric,
-      health_score::varchar,
-      sold_view_score::numeric
-    FROM staging_import
-    LIMIT $BATCH_SIZE
-    OFFSET $PROCESSED;
-    "
-    
-    # Update progress
-    BATCH_END=$(date +%s)
-    BATCH_ELAPSED=$(( $BATCH_END - $BATCH_START ))
-    PROGRESS_PCT=$(echo "scale=2; $MAX_ROWS * 100 / $TOTAL_ROWS" | bc)
-    
-    echo "Batch $BATCH_NUM committed in ${BATCH_ELAPSED}s, $MAX_ROWS of $TOTAL_ROWS rows processed ($PROGRESS_PCT%)" | tee -a "$PROGRESS_FILE"
-    
-    # Increment counter
-    PROCESSED=$(( $PROCESSED + $BATCH_SIZE ))
-    CHUNKS_SINCE_CLEANUP=$(( $CHUNKS_SINCE_CLEANUP + 1 ))
-    
-    # Check current row count every 10 batches
-    if [ $(( $BATCH_NUM % 10 )) -eq 0 ]; then
-      CURRENT_COUNT=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM $TABLE_NAME;" | tr -d '[:space:]')
-      echo "Current row count in $TABLE_NAME: $CURRENT_COUNT" | tee -a "$PROGRESS_FILE"
-      
-      # Every 10 batches, run an intermediate cleanup
-      if [ $CHUNKS_SINCE_CLEANUP -ge 10 ]; then
-        echo "Running intermediate cleanup and optimization..."
-        psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
-        CHUNKS_SINCE_CLEANUP=0
-      fi
-    fi
-    
-    # Optional - write a checkpoint file to know where to restart
-    echo "$BATCH_NUM" > "/tmp/import_last_batch_${TABLE_NAME}.txt"
-  done
-  
-  # Only recreate indexes if we've completed the import
-  if [ $PROCESSED -ge $TOTAL_ROWS ]; then
-    # Step 7: Re-enable triggers and recreate primary key
-    echo "Re-enabling triggers and recreating primary key..."
-    psql $PSQL_OPTS -c "
-    ALTER TABLE $TABLE_NAME ENABLE TRIGGER ALL;
-    ALTER TABLE $TABLE_NAME ADD PRIMARY KEY (pid, date);
-    "
-    
-    # Step 8: Clean up and get final count
-    echo "Cleaning up and getting final count..."
-    psql $PSQL_OPTS -c "
-    DROP TABLE staging_import;
-    VACUUM ANALYZE $TABLE_NAME;
-    SELECT COUNT(*) AS \"Total rows in $TABLE_NAME\" FROM $TABLE_NAME;
-    "
-  else
-    echo "Import interrupted at batch $BATCH_NUM. To resume, run:"
-    echo "./psql-csv-import.sh $CSV_FILE $TABLE_NAME $BATCH_NUM"
-  fi
-
-else
-  # Generic approach for other tables
-  if [ "$START_BATCH" -eq 1 ]; then
-    # Extract CSV header
-    CSV_HEADER=$(head -n 1 "$CSV_FILE")
-    echo "CSV header: $CSV_HEADER"
-    
-    # Extract CSV header and format it for SQL
-    CSV_COLUMNS=$(echo "$CSV_HEADER" | tr ',' '\n' | sed 's/^/"/;s/$/"/' | tr '\n' ',' | sed 's/,$//')
-    TEMP_COLUMNS=$(echo "$CSV_HEADER" | tr ',' '\n' | sed 's/$/ TEXT/' | tr '\n' ',' | sed 's/,$//')
-    
-    echo "Importing columns: $CSV_COLUMNS"
-    
-    # Step 1: Set maintenance_work_mem and disable triggers
-    echo "Setting maintenance_work_mem and disabling triggers..."
-    psql $PSQL_OPTS -c "
-    SET maintenance_work_mem = '1GB';
-    ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;
-    "
-    
-    # Step 2: Create temp table
-    echo "Creating temporary table..."
-    psql $PSQL_OPTS -c "
-    DROP TABLE IF EXISTS temp_import;
-    CREATE UNLOGGED TABLE temp_import ($TEMP_COLUMNS);
-    
-    -- Create an index on temp_import to improve OFFSET performance
-    CREATE INDEX ON temp_import ((1));  -- Index on first column
-    "
-    
-    # Step 3: Import CSV into temp table
-    echo "Importing CSV into temporary table..."
-    psql $PSQL_OPTS -c "\copy temp_import FROM '$CSV_FILE' WITH CSV HEADER DELIMITER ','"
-  else
-    echo "Resuming import from batch $START_BATCH - skipping table creation and CSV import"
-    
-    # Check if temp table exists
-    TEMP_EXISTS=$(psql $PSQL_OPTS -t -c "SELECT EXISTS(SELECT 1 FROM pg_tables WHERE tablename='temp_import');" | tr -d '[:space:]')
-    
-    if [ "$TEMP_EXISTS" != "t" ]; then
-      echo "Error: Temporary table 'temp_import' does not exist. Run without batch parameter first."
-      exit 1
-    fi
-    
-    # Ensure triggers are disabled
-    psql $PSQL_OPTS -c "ALTER TABLE $TABLE_NAME DISABLE TRIGGER ALL;"
-    
-    # Optimize PostgreSQL for better performance
-    psql $PSQL_OPTS -c "
-    -- Increase work mem for this session
-    SET work_mem = '256MB';
-    SET maintenance_work_mem = '1GB';
-    "
-    
-    # Hard-code columns since we know them
-    CSV_COLUMNS='"pid","date","score","score2","qty_in_baskets","qty_sold","notifies_set","visibility_score","health_score","sold_view_score"'
-    
-    echo "Using standard columns: $CSV_COLUMNS"
-  fi
-  
-  # Step 4: Get total row count
-  TOTAL_ROWS=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM temp_import;" | tr -d '[:space:]')
-  echo "Total rows to import: $TOTAL_ROWS"
-  
-  # Calculate starting point
-  PROCESSED=$(( ($START_BATCH - 1) * $BATCH_SIZE ))
-  if [ $PROCESSED -ge $TOTAL_ROWS ]; then
-    echo "Error: Start batch $START_BATCH is beyond the available rows ($TOTAL_ROWS)"
-    exit 1
-  fi
-  
-  # Step 5: Process in batches with shell loop
-  BATCH_NUM=$(( $START_BATCH - 1 ))
-  
-  # We'll process batches in chunks of 10 before cleaning up
-  CHUNKS_SINCE_CLEANUP=0
-  
-  while [ $PROCESSED -lt $TOTAL_ROWS ]; do
-    BATCH_NUM=$(( $BATCH_NUM + 1 ))
-    BATCH_START=$(date +%s)
-    MAX_ROWS=$(( $PROCESSED + $BATCH_SIZE ))
-    if [ $MAX_ROWS -gt $TOTAL_ROWS ]; then
-      MAX_ROWS=$TOTAL_ROWS
-    fi
-    
-    echo "Processing batch $BATCH_NUM (rows $PROCESSED to $MAX_ROWS)..."
-    
-    # Optimize query buffer for this batch
-    psql $PSQL_OPTS -c "SET work_mem = '256MB';"
-    
-    # Insert batch
-    psql $PSQL_OPTS -c "
-    INSERT INTO $TABLE_NAME ($CSV_COLUMNS)
-    SELECT $CSV_COLUMNS 
-    FROM temp_import
-    LIMIT $BATCH_SIZE
-    OFFSET $PROCESSED;
-    "
-    
-    # Update progress
-    BATCH_END=$(date +%s)
-    BATCH_ELAPSED=$(( $BATCH_END - $BATCH_START ))
-    PROGRESS_PCT=$(echo "scale=2; $MAX_ROWS * 100 / $TOTAL_ROWS" | bc)
-    
-    echo "Batch $BATCH_NUM committed in ${BATCH_ELAPSED}s, $MAX_ROWS of $TOTAL_ROWS rows processed ($PROGRESS_PCT%)" | tee -a "$PROGRESS_FILE"
-    
-    # Increment counter
-    PROCESSED=$(( $PROCESSED + $BATCH_SIZE ))
-    CHUNKS_SINCE_CLEANUP=$(( $CHUNKS_SINCE_CLEANUP + 1 ))
-    
-    # Check current row count every 10 batches
-    if [ $(( $BATCH_NUM % 10 )) -eq 0 ]; then
-      CURRENT_COUNT=$(psql $PSQL_OPTS -t -c "SELECT COUNT(*) FROM $TABLE_NAME;" | tr -d '[:space:]')
-      echo "Current row count in $TABLE_NAME: $CURRENT_COUNT" | tee -a "$PROGRESS_FILE"
-      
-      # Every 10 batches, run an intermediate cleanup
-      if [ $CHUNKS_SINCE_CLEANUP -ge 10 ]; then
-        echo "Running intermediate cleanup and optimization..."
-        psql $PSQL_OPTS -c "VACUUM $TABLE_NAME;"
-        CHUNKS_SINCE_CLEANUP=0
-      fi
-    fi
-    
-    # Optional - write a checkpoint file to know where to restart
-    echo "$BATCH_NUM" > "/tmp/import_last_batch_${TABLE_NAME}.txt"
-  done
-  
-  # Only clean up if we've completed the import
-  if [ $PROCESSED -ge $TOTAL_ROWS ]; then
-    # Step 6: Re-enable triggers and clean up
-    echo "Re-enabling triggers and cleaning up..."
-    psql $PSQL_OPTS -c "
-    ALTER TABLE $TABLE_NAME ENABLE TRIGGER ALL;
-    DROP TABLE temp_import;
-    VACUUM ANALYZE $TABLE_NAME;
-    SELECT COUNT(*) AS \"Total rows in $TABLE_NAME\" FROM $TABLE_NAME;
-    "
-  else
-    echo "Import interrupted at batch $BATCH_NUM. To resume, run:"
-    echo "./psql-csv-import.sh $CSV_FILE $TABLE_NAME $BATCH_NUM"
-  fi
-fi
-
-# Calculate elapsed time
-END_TIME=$(date +%s)
-ELAPSED=$((END_TIME - START_TIME))
-
-echo "Import completed successfully in ${ELAPSED}s ($(($ELAPSED / 60)) minutes)"
-echo "Progress log saved to $PROGRESS_FILE" 

inventory-server/old/reset-metrics.js

@@ -1,378 +0,0 @@
-const { Client } = require('pg');
-const path = require('path');
-const fs = require('fs');
-require('dotenv').config({ path: path.resolve(__dirname, '../.env') });
-
-const dbConfig = {
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432
-};
-
-function outputProgress(data) {
-    if (!data.status) {
-        data = {
-            status: 'running',
-            ...data
-        };
-    }
-    console.log(JSON.stringify(data));
-}
-
-// Explicitly define all metrics-related tables in dependency order
-const METRICS_TABLES = [
-    'brand_metrics',
-    'brand_time_metrics',
-    'category_forecasts',
-    'category_metrics',
-    'category_sales_metrics',
-    'category_time_metrics',
-    'product_metrics',
-    'product_time_aggregates',
-    'sales_forecasts',
-    'temp_purchase_metrics',
-    'temp_sales_metrics',
-    'vendor_metrics',
-    'vendor_time_metrics',
-    'vendor_details'
-];
-
-// Tables to always protect from being dropped
-const PROTECTED_TABLES = [
-    'users', 
-    'permissions', 
-    'user_permissions', 
-    'calculate_history', 
-    'import_history', 
-    'ai_prompts', 
-    'ai_validation_performance', 
-    'templates', 
-    'reusable_images'
-];
-
-// Split SQL into individual statements
-function splitSQLStatements(sql) {
-    sql = sql.replace(/\r\n/g, '\n');
-    let statements = [];
-    let currentStatement = '';
-    let inString = false;
-    let stringChar = '';
-    
-    for (let i = 0; i < sql.length; i++) {
-        const char = sql[i];
-        const nextChar = sql[i + 1] || '';
-        
-        if ((char === "'" || char === '"') && sql[i - 1] !== '\\') {
-            if (!inString) {
-                inString = true;
-                stringChar = char;
-            } else if (char === stringChar) {
-                inString = false;
-            }
-        }
-        
-        if (!inString && char === '-' && nextChar === '-') {
-            while (i < sql.length && sql[i] !== '\n') i++;
-            continue;
-        }
-        
-        if (!inString && char === '/' && nextChar === '*') {
-            i += 2;
-            while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
-            i++;
-            continue;
-        }
-        
-        if (!inString && char === ';') {
-            if (currentStatement.trim()) {
-                statements.push(currentStatement.trim());
-            }
-            currentStatement = '';
-        } else {
-            currentStatement += char;
-        }
-    }
-    
-    if (currentStatement.trim()) {
-        statements.push(currentStatement.trim());
-    }
-    
-    return statements;
-}
-
-async function resetMetrics() {
-    let client;
-    try {
-        outputProgress({
-            operation: 'Starting metrics reset',
-            message: 'Connecting to database...'
-        });
-
-        client = new Client(dbConfig);
-        await client.connect();
-
-        // Explicitly begin a transaction
-        await client.query('BEGIN');
-
-        // First verify current state
-        const initialTables = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-            AND tablename NOT IN (SELECT unnest($2::text[]))
-        `, [METRICS_TABLES, PROTECTED_TABLES]);
-
-        outputProgress({
-            operation: 'Initial state',
-            message: `Found ${initialTables.rows.length} existing metrics tables: ${initialTables.rows.map(t => t.name).join(', ')}`
-        });
-
-        // Disable foreign key checks at the start
-        await client.query('SET session_replication_role = \'replica\'');
-
-        // Drop all metrics tables in reverse order to handle dependencies
-        outputProgress({
-            operation: 'Dropping metrics tables',
-            message: 'Removing existing metrics tables...'
-        });
-
-        for (const table of [...METRICS_TABLES].reverse()) {
-            // Skip protected tables
-            if (PROTECTED_TABLES.includes(table)) {
-                outputProgress({
-                    operation: 'Protected table',
-                    message: `Skipping protected table: ${table}`
-                });
-                continue;
-            }
-            
-            try {
-                // Use NOWAIT to avoid hanging if there's a lock
-                await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
-                
-                // Verify the table was actually dropped
-                const checkDrop = await client.query(`
-                    SELECT COUNT(*) as count 
-                    FROM pg_tables 
-                    WHERE schemaname = 'public' 
-                    AND tablename = $1
-                `, [table]);
-                
-                if (parseInt(checkDrop.rows[0].count) > 0) {
-                    throw new Error(`Failed to drop table ${table} - table still exists`);
-                }
-
-                outputProgress({
-                    operation: 'Table dropped',
-                    message: `Successfully dropped table: ${table}`
-                });
-                
-                // Commit after each table drop to ensure locks are released
-                await client.query('COMMIT');
-                // Start a new transaction for the next table
-                await client.query('BEGIN');
-                // Re-disable foreign key constraints for the new transaction
-                await client.query('SET session_replication_role = \'replica\'');
-            } catch (err) {
-                outputProgress({
-                    status: 'error',
-                    operation: 'Drop table error',
-                    message: `Error dropping table ${table}: ${err.message}`
-                });
-                await client.query('ROLLBACK');
-                // Re-start transaction for next table
-                await client.query('BEGIN');
-                await client.query('SET session_replication_role = \'replica\'');
-            }
-        }
-
-        // Verify all tables were dropped
-        const afterDrop = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-        `, [METRICS_TABLES]);
-
-        if (afterDrop.rows.length > 0) {
-            throw new Error(`Failed to drop all tables. Remaining tables: ${afterDrop.rows.map(t => t.name).join(', ')}`);
-        }
-
-        // Make sure we have a fresh transaction here
-        await client.query('COMMIT');
-        await client.query('BEGIN');
-        await client.query('SET session_replication_role = \'replica\'');
-
-        // Read metrics schema
-        outputProgress({
-            operation: 'Reading schema',
-            message: 'Loading metrics schema file...'
-        });
-
-        const schemaPath = path.resolve(__dirname, '../db/metrics-schema.sql');
-        if (!fs.existsSync(schemaPath)) {
-            throw new Error(`Schema file not found at: ${schemaPath}`);
-        }
-
-        const schemaSQL = fs.readFileSync(schemaPath, 'utf8');
-        const statements = splitSQLStatements(schemaSQL);
-
-        outputProgress({
-            operation: 'Schema loaded',
-            message: `Found ${statements.length} SQL statements to execute`
-        });
-
-        // Execute schema statements
-        for (let i = 0; i < statements.length; i++) {
-            const stmt = statements[i];
-            try {
-                const result = await client.query(stmt);
-
-                // If this is a CREATE TABLE statement, verify the table was created
-                if (stmt.trim().toLowerCase().startsWith('create table')) {
-                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(\w+)["]?/i)?.[1];
-                    if (tableName) {
-                        const checkCreate = await client.query(`
-                            SELECT tablename as name
-                            FROM pg_tables 
-                            WHERE schemaname = 'public' 
-                            AND tablename = $1
-                        `, [tableName]);
-                        
-                        if (checkCreate.rows.length === 0) {
-                            throw new Error(`Failed to create table ${tableName} - table does not exist after CREATE statement`);
-                        }
-
-                        outputProgress({
-                            operation: 'Table created',
-                            message: `Successfully created table: ${tableName}`
-                        });
-                    }
-                }
-                
-                outputProgress({
-                    operation: 'SQL Progress',
-                    message: {
-                        statement: i + 1,
-                        total: statements.length,
-                        preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                        rowCount: result.rowCount
-                    }
-                });
-                
-                // Commit every 10 statements to avoid long-running transactions
-                if (i > 0 && i % 10 === 0) {
-                    await client.query('COMMIT');
-                    await client.query('BEGIN');
-                    await client.query('SET session_replication_role = \'replica\'');
-                }
-            } catch (sqlError) {
-                outputProgress({
-                    status: 'error',
-                    operation: 'SQL Error',
-                    message: {
-                        error: sqlError.message,
-                        statement: stmt,
-                        statementNumber: i + 1
-                    }
-                });
-                await client.query('ROLLBACK');
-                throw sqlError;
-            }
-        }
-
-        // Final commit for any pending statements
-        await client.query('COMMIT');
-        
-        // Start new transaction for final checks
-        await client.query('BEGIN');
-
-        // Re-enable foreign key checks after all tables are created
-        await client.query('SET session_replication_role = \'origin\'');
-
-        // Verify metrics tables were created
-        outputProgress({
-            operation: 'Verifying metrics tables',
-            message: 'Checking all metrics tables were created...'
-        });
-
-        const metricsTablesResult = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-        `, [METRICS_TABLES]);
-
-        outputProgress({
-            operation: 'Tables found',
-            message: `Found ${metricsTablesResult.rows.length} tables: ${metricsTablesResult.rows.map(t => t.name).join(', ')}`
-        });
-
-        const existingMetricsTables = metricsTablesResult.rows.map(t => t.name);
-        const missingMetricsTables = METRICS_TABLES.filter(t => !existingMetricsTables.includes(t));
-
-        if (missingMetricsTables.length > 0) {
-            // Do one final check of the actual tables
-            const finalCheck = await client.query(`
-                SELECT tablename as name
-                FROM pg_tables 
-                WHERE schemaname = 'public'
-            `);
-            outputProgress({
-                operation: 'Final table check',
-                message: `All database tables: ${finalCheck.rows.map(t => t.name).join(', ')}`
-            });
-            await client.query('ROLLBACK');
-            throw new Error(`Failed to create metrics tables: ${missingMetricsTables.join(', ')}`);
-        }
-
-        // Commit final transaction
-        await client.query('COMMIT');
-        
-        outputProgress({
-            status: 'complete',
-            operation: 'Reset complete',
-            message: 'All metrics tables have been reset successfully'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'Reset failed',
-            message: error.message,
-            stack: error.stack
-        });
-        
-        if (client) {
-            try {
-                await client.query('ROLLBACK');
-            } catch (rollbackError) {
-                console.error('Error during rollback:', rollbackError);
-            }
-            // Make sure to re-enable foreign key checks even if there's an error
-            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
-        }
-        throw error;
-    } finally {
-        if (client) {
-            // One final attempt to ensure foreign key checks are enabled
-            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
-            await client.end();
-        }
-    }
-}
-
-// Export if required as a module
-if (typeof module !== 'undefined' && module.exports) {
-    module.exports = resetMetrics;
-}
-
-// Run if called from command line
-if (require.main === module) {
-    resetMetrics().catch(error => {
-        console.error('Error:', error);
-        process.exit(1);
-    });
-} 

inventory-server/old/scripts.js

@@ -1,180 +0,0 @@
-const readline = require('readline');
-
-const rl = readline.createInterface({
-  input: process.stdin,
-  output: process.stdout
-});
-
-const question = (query) => new Promise((resolve) => rl.question(query, resolve));
-
-async function loadScript(name) {
-  try {
-    return await require(name);
-  } catch (error) {
-    console.error(`Failed to load script ${name}:`, error);
-    return null;
-  }
-}
-
-async function runWithTimeout(fn) {
-  return new Promise((resolve, reject) => {
-    // Create a child process for the script
-    const child = require('child_process').fork(fn.toString(), [], {
-      stdio: 'inherit'
-    });
-
-    child.on('exit', (code) => {
-      if (code === 0) {
-        resolve();
-      } else {
-        reject(new Error(`Script exited with code ${code}`));
-      }
-    });
-
-    child.on('error', (err) => {
-      reject(err);
-    });
-  });
-}
-
-function clearScreen() {
-  process.stdout.write('\x1Bc');
-}
-
-const scripts = {
-  'Import Scripts': {
-    '1': { name: 'Full Import From Production', path: './import-from-prod' },
-    '2': { name: 'Individual Import Scripts ▸', submenu: {
-      '1': { name: 'Import Orders', path: './import/orders', key: 'importOrders' },
-      '2': { name: 'Import Products', path: './import/products', key: 'importProducts' },
-      '3': { name: 'Import Purchase Orders', path: './import/purchase-orders' },
-      '4': { name: 'Import Categories', path: './import/categories' },
-      'b': { name: 'Back to Main Menu' }
-    }}
-  },
-  'Metrics': {
-    '3': { name: 'Calculate All Metrics', path: './calculate-metrics' },
-    '4': { name: 'Individual Metric Scripts ▸', submenu: {
-      '1': { name: 'Brand Metrics', path: './metrics/brand-metrics' },
-      '2': { name: 'Category Metrics', path: './metrics/category-metrics' },
-      '3': { name: 'Financial Metrics', path: './metrics/financial-metrics' },
-      '4': { name: 'Product Metrics', path: './metrics/product-metrics' },
-      '5': { name: 'Sales Forecasts', path: './metrics/sales-forecasts' },
-      '6': { name: 'Time Aggregates', path: './metrics/time-aggregates' },
-      '7': { name: 'Vendor Metrics', path: './metrics/vendor-metrics' },
-      'b': { name: 'Back to Main Menu' }
-    }}
-  },
-  'Database Management': {
-    '5': { name: 'Test Production Connection', path: './test-prod-connection' }
-  },
-  'Reset Scripts': {
-    '6': { name: 'Reset Database', path: './reset-db' },
-    '7': { name: 'Reset Metrics', path: './reset-metrics' }
-  }
-};
-
-let lastRun = null;
-
-async function displayMenu(menuItems, title = 'Inventory Management Script Runner') {
-  clearScreen();
-  console.log(`\n${title}\n`);
-  
-  for (const [category, items] of Object.entries(menuItems)) {
-    console.log(`\n${category}:`);
-    Object.entries(items).forEach(([key, script]) => {
-      console.log(`${key}. ${script.name}`);
-    });
-  }
-
-  if (lastRun) {
-    console.log('\nQuick Access:');
-    console.log(`r. Repeat Last Script (${lastRun.name})`);
-  }
-  
-  console.log('\nq. Quit\n');
-}
-
-async function handleSubmenu(submenu, title) {
-  while (true) {
-    await displayMenu({"Individual Scripts": submenu}, title);
-    const choice = await question('Select an option (or b to go back): ');
-    
-    if (choice.toLowerCase() === 'b') {
-      return null;
-    }
-
-    if (submenu[choice]) {
-      return submenu[choice];
-    }
-
-    console.log('Invalid selection. Please try again.');
-    await new Promise(resolve => setTimeout(resolve, 1000));
-  }
-}
-
-async function runScript(script) {
-  console.log(`\nRunning: ${script.name}`);
-  try {
-    const scriptPath = require.resolve(script.path);
-    await runWithTimeout(scriptPath);
-    console.log('\nScript completed successfully');
-    lastRun = script;
-  } catch (error) {
-    console.error('\nError running script:', error);
-  }
-  await question('\nPress Enter to continue...');
-}
-
-async function main() {
-  while (true) {
-    await displayMenu(scripts);
-    
-    const choice = await question('Select an option: ');
-    
-    if (choice.toLowerCase() === 'q') {
-      break;
-    }
-
-    if (choice.toLowerCase() === 'r' && lastRun) {
-      await runScript(lastRun);
-      continue;
-    }
-
-    let selectedScript = null;
-    for (const category of Object.values(scripts)) {
-      if (category[choice]) {
-        selectedScript = category[choice];
-        break;
-      }
-    }
-
-    if (!selectedScript) {
-      console.log('Invalid selection. Please try again.');
-      await new Promise(resolve => setTimeout(resolve, 1000));
-      continue;
-    }
-
-    if (selectedScript.submenu) {
-      const submenuChoice = await handleSubmenu(
-        selectedScript.submenu,
-        selectedScript.name
-      );
-      if (submenuChoice && submenuChoice.path) {
-        await runScript(submenuChoice);
-      }
-    } else if (selectedScript.path) {
-      await runScript(selectedScript);
-    }
-  }
-
-  rl.close();
-  process.exit(0);
-}
-
-if (require.main === module) {
-  main().catch(error => {
-    console.error('Fatal error:', error);
-    process.exit(1);
-  });
-} 

inventory-server/old/test-connection.js

@@ -1,22 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const { testConnection } = require('../../scripts/test-prod-connection');
-
-router.get('/test-prod-connection', async (req, res) => {
-  try {
-    const productCount = await testConnection();
-    res.json({ 
-      success: true, 
-      message: 'Successfully connected to production database',
-      productCount 
-    });
-  } catch (error) {
-    console.error('Production connection test failed:', error);
-    res.status(500).json({ 
-      success: false, 
-      error: error.message || 'Failed to connect to production database' 
-    });
-  }
-});
-
-module.exports = router; 

inventory-server/old/test-prod-connection.js

@@ -1,89 +0,0 @@
-const mysql = require('mysql2/promise');
-const { Client } = require('ssh2');
-const dotenv = require('dotenv');
-const path = require('path');
-
-dotenv.config({ path: path.join(__dirname, '../.env') });
-
-// SSH configuration
-const sshConfig = {
-  host: process.env.PROD_SSH_HOST,
-  port: process.env.PROD_SSH_PORT || 22,
-  username: process.env.PROD_SSH_USER,
-  privateKey: process.env.PROD_SSH_KEY_PATH ? require('fs').readFileSync(process.env.PROD_SSH_KEY_PATH) : undefined
-};
-
-// Database configuration
-const dbConfig = {
-  host: process.env.PROD_DB_HOST || 'localhost', // Usually localhost when tunneling
-  user: process.env.PROD_DB_USER,
-  password: process.env.PROD_DB_PASSWORD,
-  database: process.env.PROD_DB_NAME,
-  port: process.env.PROD_DB_PORT || 3306
-};
-
-async function testConnection() {
-  const ssh = new Client();
-
-  try {
-    // Create new Promise for SSH connection
-    await new Promise((resolve, reject) => {
-      ssh.on('ready', resolve)
-         .on('error', reject)
-         .connect(sshConfig);
-    });
-
-    console.log('SSH Connection successful!');
-
-    // Forward local port to remote MySQL port
-    const tunnel = await new Promise((resolve, reject) => {
-      ssh.forwardOut(
-        '127.0.0.1',
-        0,
-        dbConfig.host,
-        dbConfig.port,
-        (err, stream) => {
-          if (err) reject(err);
-          resolve(stream);
-        }
-      );
-    });
-
-    console.log('Port forwarding established');
-
-    // Create MySQL connection over SSH tunnel
-    const connection = await mysql.createConnection({
-      ...dbConfig,
-      stream: tunnel
-    });
-
-    console.log('MySQL Connection successful!');
-
-    // Test query
-    const [rows] = await connection.query('SELECT COUNT(*) as count FROM products');
-    console.log('Query successful! Product count:', rows[0].count);
-
-    // Clean up
-    await connection.end();
-    ssh.end();
-    console.log('Connections closed successfully');
-    return rows[0].count;
-
-  } catch (error) {
-    console.error('Error:', error);
-    if (ssh) ssh.end();
-    throw error;
-  }
-}
-
-// If running directly (not imported)
-if (require.main === module) {
-  testConnection()
-    .then(() => process.exit(0))
-    .catch(error => {
-      console.error('Test failed:', error);
-      process.exit(1);
-    });
-}
-
-module.exports = { testConnection }; 

inventory-server/old/update-order-costs.js

@@ -1,337 +0,0 @@
-/**
- * This script updates the costeach values for existing orders from the original MySQL database
- * without needing to run the full import process.
- */
-const dotenv = require("dotenv");
-const path = require("path");
-const fs = require("fs");
-const { setupConnections, closeConnections } = require('../scripts/import/utils');
-const { outputProgress, formatElapsedTime } = require('./metrics/utils/progress');
-
-dotenv.config({ path: path.join(__dirname, "../.env") });
-
-// SSH configuration
-const sshConfig = {
-  ssh: {
-    host: process.env.PROD_SSH_HOST,
-    port: process.env.PROD_SSH_PORT || 22,
-    username: process.env.PROD_SSH_USER,
-    privateKey: process.env.PROD_SSH_KEY_PATH
-      ? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
-      : undefined,
-    compress: true, // Enable SSH compression
-  },
-  prodDbConfig: {
-    // MySQL config for production
-    host: process.env.PROD_DB_HOST || "localhost",
-    user: process.env.PROD_DB_USER,
-    password: process.env.PROD_DB_PASSWORD,
-    database: process.env.PROD_DB_NAME,
-    port: process.env.PROD_DB_PORT || 3306,
-    timezone: 'Z',
-  },
-  localDbConfig: {
-    // PostgreSQL config for local
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432,
-    ssl: process.env.DB_SSL === 'true',
-    connectionTimeoutMillis: 60000,
-    idleTimeoutMillis: 30000,
-    max: 10 // connection pool max size
-  }
-};
-
-async function updateOrderCosts() {
-  const startTime = Date.now();
-  let connections;
-  let updatedCount = 0;
-  let errorCount = 0;
-
-  try {
-    outputProgress({
-      status: "running",
-      operation: "Order costs update",
-      message: "Initializing SSH tunnel..."
-    });
-
-    connections = await setupConnections(sshConfig);
-    const { prodConnection, localConnection } = connections;
-
-    // 1. Get all orders from local database that need cost updates
-    outputProgress({
-      status: "running",
-      operation: "Order costs update",
-      message: "Getting orders from local database..."
-    });
-
-    const [orders] = await localConnection.query(`
-      SELECT DISTINCT order_number, pid 
-      FROM orders 
-      WHERE costeach = 0 OR costeach IS NULL
-      ORDER BY order_number
-    `);
-
-    if (!orders || !orders.rows || orders.rows.length === 0) {
-      console.log("No orders found that need cost updates");
-      return { updatedCount: 0, errorCount: 0 };
-    }
-
-    const totalOrders = orders.rows.length;
-    console.log(`Found ${totalOrders} orders that need cost updates`);
-
-    // Process in batches of 1000 orders
-    const BATCH_SIZE = 500;
-    for (let i = 0; i < orders.rows.length; i += BATCH_SIZE) {
-      try {
-        // Start transaction for this batch
-        await localConnection.beginTransaction();
-        
-        const batch = orders.rows.slice(i, i + BATCH_SIZE);
-        
-        const orderNumbers = [...new Set(batch.map(o => o.order_number))];
-        
-        // 2. Fetch costs from production database for these orders
-        outputProgress({
-          status: "running",
-          operation: "Order costs update",
-          message: `Fetching costs for orders ${i + 1} to ${Math.min(i + BATCH_SIZE, totalOrders)} of ${totalOrders}`,
-          current: i,
-          total: totalOrders,
-          elapsed: formatElapsedTime((Date.now() - startTime) / 1000)
-        });
-
-        const [costs] = await prodConnection.query(`
-          SELECT 
-            oc.orderid as order_number,
-            oc.pid,
-            oc.costeach
-          FROM order_costs oc
-          INNER JOIN (
-            SELECT 
-              orderid,
-              pid, 
-              MAX(id) as max_id
-            FROM order_costs
-            WHERE orderid IN (?) 
-              AND pending = 0
-            GROUP BY orderid, pid
-          ) latest ON oc.orderid = latest.orderid AND oc.pid = latest.pid AND oc.id = latest.max_id
-        `, [orderNumbers]);
-
-        // Create a map of costs for easy lookup
-        const costMap = {};
-        if (costs && costs.length) {
-          costs.forEach(c => {
-            costMap[`${c.order_number}-${c.pid}`] = c.costeach || 0;
-          });
-        }
-
-        // 3. Update costs in local database by batches
-        // Using a more efficient update approach with a temporary table
-        
-        // Create a temporary table for each batch
-        await localConnection.query(`
-          DROP TABLE IF EXISTS temp_order_costs;
-          CREATE TEMP TABLE temp_order_costs (
-            order_number VARCHAR(50) NOT NULL,
-            pid BIGINT NOT NULL,
-            costeach DECIMAL(10,3) NOT NULL,
-            PRIMARY KEY (order_number, pid)
-          );
-        `);
-        
-        // Insert cost data into the temporary table
-        const costEntries = [];
-        for (const order of batch) {
-          const key = `${order.order_number}-${order.pid}`;
-          if (key in costMap) {
-            costEntries.push({
-              order_number: order.order_number,
-              pid: order.pid,
-              costeach: costMap[key]
-            });
-          }
-        }
-        
-        // Insert in sub-batches of 100
-        const DB_BATCH_SIZE = 50;
-        for (let j = 0; j < costEntries.length; j += DB_BATCH_SIZE) {
-          const subBatch = costEntries.slice(j, j + DB_BATCH_SIZE);
-          if (subBatch.length === 0) continue;
-          
-          const placeholders = subBatch.map((_, idx) => 
-            `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
-          ).join(',');
-          
-          const values = subBatch.flatMap(item => [
-            item.order_number,
-            item.pid,
-            item.costeach
-          ]);
-          
-          await localConnection.query(`
-            INSERT INTO temp_order_costs (order_number, pid, costeach)
-            VALUES ${placeholders}
-          `, values);
-        }
-        
-        // Perform bulk update from the temporary table
-        const [updateResult] = await localConnection.query(`
-          UPDATE orders o
-          SET costeach = t.costeach
-          FROM temp_order_costs t
-          WHERE o.order_number = t.order_number AND o.pid = t.pid
-          RETURNING o.id
-        `);
-        
-        const batchUpdated = updateResult.rowCount || 0;
-        updatedCount += batchUpdated;
-
-        // Commit transaction for this batch
-        await localConnection.commit();
-
-        outputProgress({
-          status: "running",
-          operation: "Order costs update",
-          message: `Updated ${updatedCount} orders with costs from production (batch: ${batchUpdated})`,
-          current: i + batch.length,
-          total: totalOrders,
-          elapsed: formatElapsedTime((Date.now() - startTime) / 1000)
-        });
-      } catch (error) {
-        // If a batch fails, roll back that batch's transaction and continue
-        try {
-          await localConnection.rollback();
-        } catch (rollbackError) {
-          console.error("Error during batch rollback:", rollbackError);
-        }
-        
-        console.error(`Error processing batch ${i}-${i + BATCH_SIZE}:`, error);
-        errorCount++;
-      }
-    }
-
-    // 4. For orders with no matching costs, set a default based on price
-    outputProgress({
-      status: "running",
-      operation: "Order costs update",
-      message: "Setting default costs for remaining orders..."
-    });
-
-    // Process remaining updates in smaller batches
-    const DEFAULT_BATCH_SIZE = 10000;
-    let totalDefaultUpdated = 0;
-    
-    try {
-      // Start with a count query to determine how many records need the default update
-      const [countResult] = await localConnection.query(`
-        SELECT COUNT(*) as count FROM orders
-        WHERE (costeach = 0 OR costeach IS NULL)
-      `);
-      
-      const totalToUpdate = parseInt(countResult.rows[0]?.count || 0);
-      
-      if (totalToUpdate > 0) {
-        console.log(`Applying default cost to ${totalToUpdate} orders`);
-        
-        // Apply the default in batches with separate transactions
-        for (let i = 0; i < totalToUpdate; i += DEFAULT_BATCH_SIZE) {
-          try {
-            await localConnection.beginTransaction();
-            
-            const [defaultUpdates] = await localConnection.query(`
-              WITH orders_to_update AS (
-                SELECT id FROM orders
-                WHERE (costeach = 0 OR costeach IS NULL)
-                LIMIT ${DEFAULT_BATCH_SIZE}
-              )
-              UPDATE orders o
-              SET costeach = price * 0.5
-              FROM orders_to_update otu
-              WHERE o.id = otu.id
-              RETURNING o.id
-            `);
-            
-            const batchDefaultUpdated = defaultUpdates.rowCount || 0;
-            totalDefaultUpdated += batchDefaultUpdated;
-            
-            await localConnection.commit();
-            
-            outputProgress({
-              status: "running",
-              operation: "Order costs update",
-              message: `Applied default costs to ${totalDefaultUpdated} of ${totalToUpdate} orders`,
-              current: totalDefaultUpdated,
-              total: totalToUpdate,
-              elapsed: formatElapsedTime((Date.now() - startTime) / 1000)
-            });
-          } catch (error) {
-            try {
-              await localConnection.rollback();
-            } catch (rollbackError) {
-              console.error("Error during default update rollback:", rollbackError);
-            }
-            
-            console.error(`Error applying default costs batch ${i}-${i + DEFAULT_BATCH_SIZE}:`, error);
-            errorCount++;
-          }
-        }
-      }
-    } catch (error) {
-      console.error("Error counting or updating remaining orders:", error);
-      errorCount++;
-    }
-
-    updatedCount += totalDefaultUpdated;
-    
-    const endTime = Date.now();
-    const totalSeconds = (endTime - startTime) / 1000;
-
-    outputProgress({
-      status: "complete",
-      operation: "Order costs update",
-      message: `Updated ${updatedCount} orders (${totalDefaultUpdated} with default values) in ${formatElapsedTime(totalSeconds)}`,
-      elapsed: formatElapsedTime(totalSeconds)
-    });
-
-    return {
-      status: "complete",
-      updatedCount,
-      errorCount
-    };
-  } catch (error) {
-    console.error("Error during order costs update:", error);
-    
-    return {
-      status: "error",
-      error: error.message,
-      updatedCount,
-      errorCount
-    };
-  } finally {
-    if (connections) {
-      await closeConnections(connections).catch(err => {
-        console.error("Error closing connections:", err);
-      });
-    }
-  }
-}
-
-// Run the script only if this is the main module
-if (require.main === module) {
-  updateOrderCosts().then((results) => {
-    console.log('Cost update completed:', results);
-    // Force exit after a small delay to ensure all logs are written
-    setTimeout(() => process.exit(0), 500);
-  }).catch((error) => {
-    console.error("Unhandled error:", error);
-    // Force exit with error code after a small delay
-    setTimeout(() => process.exit(1), 500);
-  });
-}
-
-// Export the function for use in other scripts
-module.exports = updateOrderCosts; 

inventory-server/package.json

@@ -1,41 +0,0 @@
-{
-  "name": "inventory-server",
-  "version": "1.0.0",
-  "description": "Backend server for inventory management system",
-  "main": "src/server.js",
-  "scripts": {
-    "start": "node src/server.js",
-    "dev": "nodemon src/server.js",
-    "prod": "pm2 start ecosystem.config.js",
-    "prod:stop": "pm2 stop inventory-server",
-    "prod:restart": "pm2 restart inventory-server",
-    "prod:logs": "pm2 logs inventory-server",
-    "prod:status": "pm2 status inventory-server",
-    "setup": "mkdir -p logs uploads",
-    "test": "echo \"Error: no test specified\" && exit 1"
-  },
-  "keywords": [],
-  "author": "",
-  "license": "ISC",
-  "dependencies": {
-    "@types/diff": "^7.0.1",
-    "axios": "^1.8.1",
-    "bcrypt": "^5.1.1",
-    "commander": "^13.1.0",
-    "cors": "^2.8.5",
-    "csv-parse": "^5.6.0",
-    "diff": "^7.0.0",
-    "dotenv": "^16.4.7",
-    "express": "^4.18.2",
-    "multer": "^1.4.5-lts.1",
-    "mysql2": "^3.12.0",
-    "openai": "^4.85.3",
-    "pg": "^8.14.1",
-    "pm2": "^5.3.0",
-    "ssh2": "^1.16.0",
-    "uuid": "^9.0.1"
-  },
-  "devDependencies": {
-    "nodemon": "^3.0.2"
-  }
-}

inventory-server/scripts/calculate-metrics-new.js

@@ -1,908 +0,0 @@
-// run-all-updates.js
-const path = require('path');
-const fs = require('fs');
-const { Pool } = require('pg'); // Assuming you use 'pg'
-
-// --- Configuration ---
-// Toggle these constants to enable/disable specific steps for testing
-const RUN_DAILY_SNAPSHOTS = true;
-const RUN_PRODUCT_METRICS = true;
-const RUN_PERIODIC_METRICS = true;
-const RUN_BRAND_METRICS = true;
-const RUN_VENDOR_METRICS = true;
-const RUN_CATEGORY_METRICS = true;
-
-// Maximum execution time for the entire sequence (e.g., 90 minutes)
-const MAX_EXECUTION_TIME_TOTAL = 90 * 60 * 1000;
-// Maximum execution time per individual SQL step (e.g., 30 minutes)
-const MAX_EXECUTION_TIME_PER_STEP = 30 * 60 * 1000;
-// Query cancellation timeout
-const CANCEL_QUERY_AFTER_SECONDS = 5;
-// --- End Configuration ---
-
-// Change working directory to script directory
-process.chdir(path.dirname(__filename));
-
-// Log script path for debugging
-console.log('Script running from:', __dirname);
-
-// Try to load environment variables from multiple locations
-const envPaths = [
-  path.resolve(__dirname, '../..', '.env'),     // Two levels up (inventory/.env)
-  path.resolve(__dirname, '..', '.env'),       // One level up (inventory-server/.env)
-  path.resolve(__dirname, '.env'),             // Same directory
-  '/var/www/html/inventory/.env'               // Server absolute path
-];
-
-let envLoaded = false;
-for (const envPath of envPaths) {
-  if (fs.existsSync(envPath)) {
-    console.log(`Loading environment from: ${envPath}`);
-    require('dotenv').config({ path: envPath });
-    envLoaded = true;
-    break;
-  }
-}
-
-if (!envLoaded) {
-  console.warn('WARNING: Could not find .env file in any of the expected locations.');
-  console.warn('Checked paths:', envPaths);
-}
-
-// --- Database Setup ---
-// Make sure we have the required DB credentials
-if (!process.env.DB_HOST && !process.env.DATABASE_URL) {
-    console.error('WARNING: Neither DB_HOST nor DATABASE_URL environment variables found');
-}
-
-// Only validate individual parameters if not using connection string
-if (!process.env.DATABASE_URL) {
-    if (!process.env.DB_USER) console.error('WARNING: DB_USER environment variable is missing');
-    if (!process.env.DB_NAME) console.error('WARNING: DB_NAME environment variable is missing');
-
-    // Password must be a string for PostgreSQL SCRAM authentication
-    if (!process.env.DB_PASSWORD || typeof process.env.DB_PASSWORD !== 'string') {
-        console.error('WARNING: DB_PASSWORD environment variable is missing or not a string');
-    }
-}
-
-// Configure database connection to match individual scripts
-let dbConfig;
-
-// Check if a DATABASE_URL exists (common in production environments)
-if (process.env.DATABASE_URL && typeof process.env.DATABASE_URL === 'string') {
-    console.log('Using DATABASE_URL for connection');
-    dbConfig = {
-        connectionString: process.env.DATABASE_URL,
-        ssl: process.env.DB_SSL === 'true' ? { rejectUnauthorized: false } : false,
-        // Add performance optimizations
-        max: 10, // connection pool max size
-        idleTimeoutMillis: 30000,
-        connectionTimeoutMillis: 60000,
-        // Set timeouts for long-running queries
-        statement_timeout: 1800000, // 30 minutes
-        query_timeout: 1800000 // 30 minutes
-    };
-} else {
-    // Use individual connection parameters
-    dbConfig = {
-        host: process.env.DB_HOST,
-        user: process.env.DB_USER,
-        password: process.env.DB_PASSWORD,
-        database: process.env.DB_NAME,
-        port: process.env.DB_PORT || 5432,
-        ssl: process.env.DB_SSL === 'true',
-        // Add performance optimizations
-        max: 10, // connection pool max size
-        idleTimeoutMillis: 30000,
-        connectionTimeoutMillis: 60000,
-        // Set timeouts for long-running queries
-        statement_timeout: 1800000, // 30 minutes
-        query_timeout: 1800000 // 30 minutes
-    };
-}
-
-// Try to load from utils DB module as a last resort
-try {
-    if (!process.env.DB_HOST && !process.env.DATABASE_URL) {
-        console.log('Attempting to load DB config from individual script modules...');
-        const dbModule = require('./metrics-new/utils/db');
-        if (dbModule && dbModule.dbConfig) {
-            console.log('Found DB config in individual script module');
-            dbConfig = {
-                ...dbModule.dbConfig,
-                // Add performance optimizations if not present
-                max: dbModule.dbConfig.max || 10,
-                idleTimeoutMillis: dbModule.dbConfig.idleTimeoutMillis || 30000,
-                connectionTimeoutMillis: dbModule.dbConfig.connectionTimeoutMillis || 60000,
-                statement_timeout: 1800000,
-                query_timeout: 1800000
-            };
-        }
-    }
-} catch (err) {
-    console.warn('Could not load DB config from individual script modules:', err.message);
-}
-
-// Debug log connection info (without password)
-console.log('DB Connection Info:', {
-    connectionString: dbConfig.connectionString ? 'PROVIDED' : undefined,
-    host: dbConfig.host,
-    user: dbConfig.user,
-    database: dbConfig.database,
-    port: dbConfig.port,
-    ssl: dbConfig.ssl ? 'ENABLED' : 'DISABLED',
-    password: (dbConfig.password || dbConfig.connectionString) ? '****' : 'MISSING' // Only show if credentials exist
-});
-
-const pool = new Pool(dbConfig);
-
-const getConnection = () => {
-    return pool.connect();
-};
-
-const closePool = () => {
-    console.log("Closing database connection pool.");
-    return pool.end();
-};
-
-// --- Progress Utilities ---
-// Using functions directly instead of globals
-const progressUtils = require('./metrics-new/utils/progress'); // Assuming utils/progress.js exports these
-
-// --- State & Cancellation ---
-let isCancelled = false;
-let currentStep = ''; // Track which step is running for cancellation message
-let overallStartTime = null;
-let mainTimeoutHandle = null;
-let stepTimeoutHandle = null;
-let combinedHistoryId = null; // ID for the combined history record
-
-async function cancelCalculation(reason = 'cancelled by user') {
-    if (isCancelled) return; // Prevent multiple cancellations
-    isCancelled = true;
-    console.log(`Calculation ${reason}. Attempting to cancel active step: ${currentStep}`);
-
-    // Clear timeouts
-    if (mainTimeoutHandle) clearTimeout(mainTimeoutHandle);
-    if (stepTimeoutHandle) clearTimeout(stepTimeoutHandle);
-
-    // Attempt to cancel the long-running query in Postgres
-    let conn = null;
-    try {
-        console.log(`Attempting to cancel queries running longer than ${CANCEL_QUERY_AFTER_SECONDS} seconds...`);
-        conn = await getConnection();
-        const result = await conn.query(`
-            SELECT pg_cancel_backend(pid)
-            FROM pg_stat_activity
-            WHERE query_start < now() - interval '${CANCEL_QUERY_AFTER_SECONDS} seconds'
-            AND application_name = 'node-metrics-calculator' -- Match specific app name
-            AND state = 'active' -- Only cancel active queries
-            AND query NOT LIKE '%pg_cancel_backend%'
-            AND pid <> pg_backend_pid(); -- Don't cancel self
-        `);
-        console.log(`Sent ${result.rowCount} cancellation signal(s).`);
-
-        // Update the combined history record to show cancellation
-        if (combinedHistoryId) {
-            const totalDuration = Math.round((Date.now() - overallStartTime) / 1000);
-            await conn.query(`
-                UPDATE calculate_history
-                SET
-                    status = 'cancelled'::calculation_status,
-                    end_time = NOW(),
-                    duration_seconds = $1::integer,
-                    error_message = $2::text
-                WHERE id = $3::integer;
-            `, [totalDuration, `Calculation ${reason} during step: ${currentStep}`, combinedHistoryId]);
-            console.log(`Updated combined history record ${combinedHistoryId} with cancellation status`);
-        }
-
-        conn.release();
-    } catch (err) {
-        console.error('Error during database query cancellation:', err.message);
-        if (conn) {
-            try { conn.release(); } catch (e) { console.error("Error releasing cancellation connection", e); }
-        }
-        // Proceed with script termination attempt even if DB cancel fails
-    } finally {
-         // Update progress to show cancellation
-        progressUtils.outputProgress({
-            status: 'cancelled',
-            operation: `Calculation ${reason} during step: ${currentStep}`,
-            current: 0, // Reset progress indicators
-            total: 100,
-            elapsed: overallStartTime ? progressUtils.formatElapsedTime(overallStartTime) : 'N/A',
-            remaining: null,
-            rate: 0,
-            percentage: '0', // Or keep last known percentage?
-            timing: {
-                start_time: overallStartTime ? new Date(overallStartTime).toISOString() : 'N/A',
-                end_time: new Date().toISOString(),
-                elapsed_seconds: overallStartTime ? Math.round((Date.now() - overallStartTime) / 1000) : 0
-            }
-        });
-    }
-
-    // Note: We don't force exit here anymore. We let the main function's error
-    // handling catch the cancellation error thrown by executeSqlStep or the timeout.
-    return {
-        success: true, // Indicates cancellation was initiated
-        message: `Calculation ${reason}`
-    };
-}
-
-// Handle SIGINT (Ctrl+C) and SIGTERM (kill) signals
-process.on('SIGINT', () => {
-    console.log('\nReceived SIGINT (Ctrl+C).');
-    cancelCalculation('cancelled by user (SIGINT)');
-    // Give cancellation a moment to propagate before force-exiting if needed
-    setTimeout(() => process.exit(1), 2000);
-});
-process.on('SIGTERM', () => {
-    console.log('Received SIGTERM.');
-    cancelCalculation('cancelled by system (SIGTERM)');
-    // Give cancellation a moment to propagate before force-exiting if needed
-    setTimeout(() => process.exit(1), 2000);
-});
-
-// Add error handlers for uncaught exceptions/rejections
-process.on('uncaughtException', (error) => {
-    console.error('Uncaught Exception:', error);
-    // Attempt graceful shutdown/logging if possible, then exit
-    cancelCalculation('failed due to uncaught exception').finally(() => {
-        closePool().finally(() => process.exit(1));
-    });
-});
-
-process.on('unhandledRejection', (reason, promise) => {
-    console.error('Unhandled Rejection at:', promise, 'reason:', reason);
-    // Attempt graceful shutdown/logging if possible, then exit
-    cancelCalculation('failed due to unhandled rejection').finally(() => {
-        closePool().finally(() => process.exit(1));
-    });
-});
-
-
-// --- Core Logic ---
-
-/**
- * Ensures all products have entries in the settings_product table
- * This is important after importing new products
- */
-async function syncSettingsProductTable() {
-    let conn = null;
-    try {
-        currentStep = 'Syncing settings_product table';
-        progressUtils.outputProgress({
-            operation: 'Syncing product settings',
-            message: 'Ensuring all products have settings entries'
-        });
-
-        conn = await getConnection();
-        
-        // Get counts before sync
-        const beforeCounts = await conn.query(`
-            SELECT 
-                (SELECT COUNT(*) FROM products) AS products_count,
-                (SELECT COUNT(*) FROM settings_product) AS settings_count
-        `);
-        
-        const productsCount = parseInt(beforeCounts.rows[0].products_count);
-        const settingsCount = parseInt(beforeCounts.rows[0].settings_count);
-        
-        progressUtils.outputProgress({
-            operation: 'Settings product sync',
-            message: `Found ${productsCount} products and ${settingsCount} settings entries`
-        });
-        
-        // Insert missing product settings
-        const result = await conn.query(`
-            INSERT INTO settings_product (
-                pid,
-                lead_time_days,
-                days_of_stock,
-                safety_stock,
-                forecast_method,
-                exclude_from_forecast
-            )
-            SELECT
-                p.pid,
-                CAST(NULL AS INTEGER),
-                CAST(NULL AS INTEGER),
-                COALESCE((SELECT setting_value::int FROM settings_global WHERE setting_key = 'default_safety_stock_units'), 0),
-                CAST(NULL AS VARCHAR),
-                FALSE
-            FROM
-                public.products p
-            WHERE
-                NOT EXISTS (
-                    SELECT 1 FROM settings_product sp WHERE sp.pid = p.pid
-                )
-            ON CONFLICT (pid) DO NOTHING
-        `);
-        
-        // Get counts after sync
-        const afterCounts = await conn.query(`
-            SELECT COUNT(*) AS settings_count FROM settings_product
-        `);
-        
-        const newSettingsCount = parseInt(afterCounts.rows[0].settings_count);
-        const addedCount = newSettingsCount - settingsCount;
-        
-        progressUtils.outputProgress({
-            operation: 'Settings product sync',
-            message: `Added ${addedCount} new settings entries. Now have ${newSettingsCount} total entries.`,
-            status: 'complete'
-        });
-        
-        conn.release();
-        return addedCount;
-    } catch (err) {
-        progressUtils.outputProgress({
-            status: 'error',
-            operation: 'Settings product sync failed',
-            error: err.message
-        });
-        if (conn) conn.release();
-        throw err;
-    }
-}
-
-/**
- * Executes a single SQL calculation step.
- * @param {object} config - Configuration for the step.
- * @param {string} config.name - User-friendly name of the step.
- * @param {string} config.sqlFile - Path to the SQL file.
- * @param {string} config.historyType - Type identifier for calculate_history.
- * @param {string} config.statusModule - Module name for calculate_status.
- * @param {object} progress - Progress utility functions.
- * @returns {Promise<{success: boolean, message: string, duration: number, rowsAffected: number}>}
- */
-async function executeSqlStep(config, progress) {
-    if (isCancelled) throw new Error(`Calculation skipped step ${config.name} due to prior cancellation.`);
-
-    currentStep = config.name; // Update global state
-    console.log(`\n--- Starting Step: ${config.name} ---`);
-    const stepStartTime = Date.now();
-    let connection = null;
-    let rowsAffected = 0; // Track rows affected by this step
-
-    // Set timeout for this specific step
-    if (stepTimeoutHandle) clearTimeout(stepTimeoutHandle); // Clear previous step's timeout
-    stepTimeoutHandle = setTimeout(() => {
-        // Don't exit directly, throw an error to be caught by the main loop
-        const timeoutError = new Error(`Step "${config.name}" timed out after ${MAX_EXECUTION_TIME_PER_STEP / 1000} seconds.`);
-        cancelCalculation(`timed out during step: ${config.name}`); // Initiate cancellation process
-        // The error will likely be thrown before cancelCalculation fully completes,
-        // but cancelCalculation attempts to stop the query.
-        // The main catch block will handle cleanup.
-    }, MAX_EXECUTION_TIME_PER_STEP);
-
-
-    try {
-        // 1. Read SQL File
-        const sqlFilePath = path.resolve(__dirname, config.sqlFile);
-        if (!fs.existsSync(sqlFilePath)) {
-            throw new Error(`SQL file not found: ${sqlFilePath}`);
-        }
-        const sqlQuery = fs.readFileSync(sqlFilePath, 'utf8');
-        console.log(`Read SQL file: ${config.sqlFile}`);
-        
-        // Check for potential parameter references that might cause issues
-        const parameterMatches = sqlQuery.match(/\$\d+(?!\:\:)/g);
-        if (parameterMatches && parameterMatches.length > 0) {
-            console.warn(`WARNING: Found ${parameterMatches.length} untyped parameters in SQL: ${parameterMatches.slice(0, 5).join(', ')}${parameterMatches.length > 5 ? '...' : ''}`);
-            console.warn('These might cause "could not determine data type of parameter" errors.');
-        }
-
-        // 2. Get Database Connection
-        connection = await getConnection();
-        console.log("Database connection acquired.");
-
-        // 3. Ensure calculate_status table exists
-        await connection.query(`
-            CREATE TABLE IF NOT EXISTS calculate_status (
-                module_name TEXT PRIMARY KEY,
-                last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
-            );
-        `);
-
-        // 4. Initial Progress Update
-        progress.outputProgress({
-            status: 'running',
-            operation: `Starting: ${config.name}`,
-            current: 0, total: 100,
-            elapsed: progress.formatElapsedTime(stepStartTime),
-            remaining: 'Calculating...', rate: 0, percentage: '0',
-            timing: { 
-                start_time: new Date(stepStartTime).toISOString(),
-                step_start_ms: stepStartTime
-            }
-        });
-
-        // 5. Execute the Main SQL Query
-        progress.outputProgress({
-            status: 'running',
-            operation: `Executing SQL: ${config.name}`,
-            current: 25, total: 100,
-            elapsed: progress.formatElapsedTime(stepStartTime),
-            remaining: 'Executing query...', rate: 0, percentage: '25',
-            timing: { 
-                start_time: new Date(stepStartTime).toISOString(),
-                step_start_ms: stepStartTime
-            }
-        });
-        console.log(`Executing SQL for ${config.name}...`);
-
-        try {
-            // Try executing exactly as individual scripts do
-            const result = await connection.query(sqlQuery);
-            
-            // Try to extract row count from result
-            if (result && result.rowCount !== undefined) {
-                rowsAffected = result.rowCount;
-            } else if (Array.isArray(result) && result[0] && result[0].rowCount !== undefined) {
-                rowsAffected = result[0].rowCount;
-            }
-            
-            // Check if the query returned a result set with row count info
-            if (result && result.rows && result.rows.length > 0 && result.rows[0].rows_processed) {
-                rowsAffected = parseInt(result.rows[0].rows_processed) || rowsAffected;
-                console.log(`SQL returned metrics: ${JSON.stringify(result.rows[0])}`);
-            } else if (Array.isArray(result) && result[0] && result[0].rows && result[0].rows[0] && result[0].rows[0].rows_processed) {
-                rowsAffected = parseInt(result[0].rows[0].rows_processed) || rowsAffected;
-                console.log(`SQL returned metrics: ${JSON.stringify(result[0].rows[0])}`);
-            }
-            
-            console.log(`SQL affected ${rowsAffected} rows`);
-        } catch (sqlError) {
-            if (sqlError.message.includes('could not determine data type of parameter')) {
-                console.log('Simple query failed with parameter type error, trying alternative method...');
-                try {
-                    // Execute with explicit text mode to avoid parameter confusion
-                    await connection.query({
-                        text: sqlQuery,
-                        rowMode: 'text'
-                    });
-                } catch (altError) {
-                    console.error('Alternative execution method also failed:', altError.message);
-                    throw altError; // Re-throw the alternative error
-                }
-            } else {
-                console.error('SQL Execution Error:', sqlError.message);
-                if (sqlError.position) {
-                    // If the error has a position, try to show the relevant part of the SQL query
-                    const position = parseInt(sqlError.position, 10);
-                    const startPos = Math.max(0, position - 100);
-                    const endPos = Math.min(sqlQuery.length, position + 100);
-                    console.error('SQL Error Context:');
-                    console.error('...' + sqlQuery.substring(startPos, position) + ' [ERROR HERE] ' + sqlQuery.substring(position, endPos) + '...');
-                }
-                throw sqlError; // Re-throw to be caught by the main try/catch
-            }
-        }
-
-        // Check for cancellation immediately after query finishes
-        if (isCancelled) throw new Error(`Calculation cancelled during SQL execution for ${config.name}`);
-
-        console.log(`SQL execution finished for ${config.name}.`);
-
-        // 6. Update Status table only
-        await connection.query(`
-            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
-            VALUES ($1::text, NOW())
-            ON CONFLICT (module_name) DO UPDATE
-            SET last_calculation_timestamp = EXCLUDED.last_calculation_timestamp;
-        `, [config.statusModule]);
-
-        const stepDuration = Math.round((Date.now() - stepStartTime) / 1000);
-
-        // 7. Final Progress Update for Step
-        progress.outputProgress({
-            status: 'complete',
-            operation: `Completed: ${config.name}`,
-            current: 100, total: 100,
-            elapsed: progress.formatElapsedTime(stepStartTime),
-            remaining: '0s', rate: 0, percentage: '100',
-            timing: {
-                start_time: new Date(stepStartTime).toISOString(),
-                end_time: new Date().toISOString(),
-                elapsed_seconds: stepDuration
-            }
-        });
-        console.log(`--- Finished Step: ${config.name} (Duration: ${progress.formatElapsedTime(stepStartTime)}) ---`);
-
-        return {
-            success: true,
-            message: `${config.name} completed successfully`,
-            duration: stepDuration,
-            rowsAffected: rowsAffected
-        };
-
-    } catch (error) {
-        clearTimeout(stepTimeoutHandle); // Clear timeout on error
-        const errorEndTime = Date.now();
-        const errorDuration = Math.round((errorEndTime - stepStartTime) / 1000);
-        const finalStatus = isCancelled ? 'cancelled' : 'failed';
-        const errorMessage = error.message || 'Unknown error';
-
-        console.error(`--- ERROR in Step: ${config.name} ---`);
-        console.error(error); // Log the full error
-        console.error(`------------------------------------`);
-
-        // Update progress file with error/cancellation
-        progress.outputProgress({
-            status: finalStatus,
-            operation: `Error in ${config.name}: ${errorMessage.split('\n')[0]}`, // Show first line of error
-            current: 50, total: 100, // Indicate partial completion
-            elapsed: progress.formatElapsedTime(stepStartTime),
-            remaining: null, rate: 0, percentage: '50',
-            timing: {
-                start_time: new Date(stepStartTime).toISOString(),
-                end_time: new Date(errorEndTime).toISOString(),
-                elapsed_seconds: errorDuration
-            }
-        });
-
-        // Rethrow the error to be caught by the main runCalculations function
-        throw error; // Add context if needed: new Error(`Step ${config.name} failed: ${errorMessage}`)
-
-    } finally {
-        clearTimeout(stepTimeoutHandle); // Ensure timeout is cleared
-        currentStep = ''; // Reset current step
-        if (connection) {
-            try {
-                await connection.release();
-                console.log("Database connection released.");
-            } catch (releaseError) {
-                console.error("Error releasing database connection:", releaseError);
-            }
-        }
-    }
-}
-
-/**
- * Main function to run all calculation steps sequentially.
- */
-async function runAllCalculations() {
-    overallStartTime = Date.now();
-    isCancelled = false; // Reset cancellation flag at start
-
-    // Overall timeout for the entire script
-     mainTimeoutHandle = setTimeout(() => {
-        console.error(`--- OVERALL TIMEOUT REACHED (${MAX_EXECUTION_TIME_TOTAL / 1000}s) ---`);
-        cancelCalculation(`overall timeout reached`);
-        // The process should exit via the unhandled rejection/exception handlers
-        // or the SIGTERM/SIGINT handlers after cancellation attempt.
-    }, MAX_EXECUTION_TIME_TOTAL);
-
-    const steps = [
-        {
-            run: RUN_DAILY_SNAPSHOTS,
-            name: 'Daily Snapshots Update',
-            sqlFile: 'metrics-new/update_daily_snapshots.sql',
-            historyType: 'daily_snapshots',
-            statusModule: 'daily_snapshots'
-        },
-        {
-            run: RUN_PRODUCT_METRICS,
-            name: 'Product Metrics Update',
-            sqlFile: 'metrics-new/update_product_metrics.sql', // ASSUMING the initial population is now part of a regular update
-            historyType: 'product_metrics',
-            statusModule: 'product_metrics'
-        },
-        {
-            run: RUN_PERIODIC_METRICS,
-            name: 'Periodic Metrics Update',
-            sqlFile: 'metrics-new/update_periodic_metrics.sql',
-            historyType: 'periodic_metrics',
-            statusModule: 'periodic_metrics'
-        },
-        {
-            run: RUN_BRAND_METRICS,
-            name: 'Brand Metrics Update',
-            sqlFile: 'metrics-new/calculate_brand_metrics.sql',
-            historyType: 'brand_metrics',
-            statusModule: 'brand_metrics'
-        },
-        {
-            run: RUN_VENDOR_METRICS,
-            name: 'Vendor Metrics Update',
-            sqlFile: 'metrics-new/calculate_vendor_metrics.sql',
-            historyType: 'vendor_metrics',
-            statusModule: 'vendor_metrics'
-        },
-        {
-            run: RUN_CATEGORY_METRICS,
-            name: 'Category Metrics Update',
-            sqlFile: 'metrics-new/calculate_category_metrics.sql',
-            historyType: 'category_metrics',
-            statusModule: 'category_metrics'
-        }
-    ];
-
-    // Build a list of steps we will actually run
-    const stepsToRun = steps.filter(step => step.run);
-    const stepNames = stepsToRun.map(step => step.name);
-    const sqlFiles = stepsToRun.map(step => step.sqlFile);
-    
-    let overallSuccess = true;
-    let connection = null;
-
-    try {
-        // Create a single history record before starting all calculations
-        try {
-            connection = await getConnection();
-            
-            // Ensure calculate_history table exists (basic structure)
-            await connection.query(`
-                CREATE TABLE IF NOT EXISTS calculate_history (
-                    id SERIAL PRIMARY KEY,
-                    start_time TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
-                    end_time TIMESTAMP WITH TIME ZONE,
-                    duration_seconds INTEGER,
-                    status TEXT, -- Will be altered to enum if needed below
-                    error_message TEXT,
-                    additional_info JSONB
-                );
-            `);
-
-            // Ensure the calculation_status enum type exists if needed
-            await connection.query(`
-                DO $$
-                BEGIN
-                    IF NOT EXISTS (SELECT 1 FROM pg_type WHERE typname = 'calculation_status') THEN
-                        CREATE TYPE calculation_status AS ENUM ('running', 'completed', 'failed', 'cancelled');
-                        
-                        -- If needed, alter the existing table to use the enum
-                        ALTER TABLE calculate_history 
-                        ALTER COLUMN status TYPE calculation_status 
-                        USING status::calculation_status;
-                    END IF;
-                END
-                $$;
-            `);
-
-            // Mark any previous running combined calculations as cancelled
-            await connection.query(`
-                UPDATE calculate_history
-                SET
-                    status = 'cancelled'::calculation_status,
-                    end_time = NOW(),
-                    duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
-                    error_message = 'Previous calculation was not completed properly or was superseded.'
-                WHERE status = 'running'::calculation_status AND additional_info->>'type' = 'combined_metrics';
-            `);
-
-            // Create a single history record for this run
-            const historyResult = await connection.query(`
-                INSERT INTO calculate_history (status, additional_info)
-                VALUES ('running'::calculation_status, jsonb_build_object(
-                    'type', 'combined_metrics', 
-                    'steps', $1::jsonb,
-                    'sql_files', $2::jsonb
-                ))
-                RETURNING id;
-            `, [JSON.stringify(stepNames), JSON.stringify(sqlFiles)]);
-            
-            combinedHistoryId = historyResult.rows[0].id;
-            console.log(`Created combined history record ID: ${combinedHistoryId}`);
-            
-            // Get initial counts for tracking
-            const productCount = await connection.query('SELECT COUNT(*) as count FROM products');
-            const totalProducts = parseInt(productCount.rows[0].count);
-            
-            // Update history with initial counts
-            await connection.query(`
-                UPDATE calculate_history
-                SET additional_info = additional_info || jsonb_build_object('total_products', $1::integer)
-                WHERE id = $2
-            `, [totalProducts, combinedHistoryId]);
-            
-            connection.release();
-        } catch (historyError) {
-            console.error('Error creating combined history record:', historyError);
-            if (connection) connection.release();
-            // Continue without history tracking if it fails
-        }
-
-        // First, sync the settings_product table to ensure all products have entries
-        progressUtils.outputProgress({
-            operation: 'Starting metrics calculation',
-            message: 'Preparing product settings...'
-        });
-        
-        try {
-            const addedCount = await syncSettingsProductTable();
-            
-            progressUtils.outputProgress({
-                operation: 'Preparation complete',
-                message: `Added ${addedCount} missing product settings entries`,
-                status: 'complete'
-            });
-        } catch (syncError) {
-            console.error('Warning: Failed to sync product settings, continuing with metrics calculations:', syncError);
-            // Don't fail the entire process if settings sync fails
-        }
-
-        // Track completed steps
-        const completedSteps = [];
-        const stepTimings = {};
-        const stepRowCounts = {};
-        let currentStepIndex = 0;
-        
-        // Now run the calculation steps
-        for (const step of stepsToRun) {
-            if (isCancelled) {
-                console.log(`Skipping step "${step.name}" due to cancellation.`);
-                overallSuccess = false; // Mark as not fully successful if steps are skipped due to cancel
-                continue; // Skip to next step
-            }
-            
-            currentStepIndex++;
-            
-            // Update overall progress
-            progressUtils.outputProgress({
-                status: 'running',
-                operation: 'Running calculations',
-                message: `Step ${currentStepIndex} of ${stepsToRun.length}: ${step.name}`,
-                current: currentStepIndex - 1,
-                total: stepsToRun.length,
-                elapsed: progressUtils.formatElapsedTime(overallStartTime),
-                remaining: progressUtils.estimateRemaining(overallStartTime, currentStepIndex - 1, stepsToRun.length),
-                percentage: Math.round(((currentStepIndex - 1) / stepsToRun.length) * 100).toString(),
-                timing: {
-                    overall_start_time: new Date(overallStartTime).toISOString(),
-                    current_step: step.name,
-                    completed_steps: completedSteps.length
-                }
-            });
-            
-            // Pass the progress utilities to the step executor
-            const result = await executeSqlStep(step, progressUtils);
-            
-            if (result.success) {
-                completedSteps.push({
-                    name: step.name,
-                    duration: result.duration,
-                    status: 'completed',
-                    rowsAffected: result.rowsAffected
-                });
-                stepTimings[step.name] = result.duration;
-                stepRowCounts[step.name] = result.rowsAffected;
-            }
-        }
-
-        // If we finished naturally (no errors thrown out)
-        clearTimeout(mainTimeoutHandle); // Clear the main timeout
-
-        // Update the combined history record on successful completion
-        if (combinedHistoryId) {
-            try {
-                connection = await getConnection();
-                const totalDuration = Math.round((Date.now() - overallStartTime) / 1000);
-                
-                // Get final processed counts
-                const processedCounts = await connection.query(`
-                    SELECT 
-                        (SELECT COUNT(*) FROM product_metrics WHERE last_calculated >= $1) as processed_products
-                `, [new Date(overallStartTime)]);
-                
-                await connection.query(`
-                    UPDATE calculate_history
-                    SET
-                        end_time = NOW(),
-                        duration_seconds = $1::integer,
-                        status = $2::calculation_status,
-                        additional_info = additional_info || jsonb_build_object(
-                            'processed_products', $3::integer,
-                            'completed_steps', $4::jsonb,
-                            'step_timings', $5::jsonb,
-                            'step_row_counts', $6::jsonb
-                        )
-                    WHERE id = $7::integer;
-                `, [
-                    totalDuration, 
-                    isCancelled ? 'cancelled' : 'completed',
-                    processedCounts.rows[0].processed_products,
-                    JSON.stringify(completedSteps),
-                    JSON.stringify(stepTimings),
-                    JSON.stringify(stepRowCounts),
-                    combinedHistoryId
-                ]);
-                
-                connection.release();
-            } catch (historyError) {
-                console.error('Error updating combined history record on completion:', historyError);
-                if (connection) connection.release();
-            }
-        }
-
-        if (isCancelled) {
-             console.log("\n--- Calculation finished with cancellation ---");
-             overallSuccess = false;
-        } else {
-            console.log("\n--- All enabled calculations finished successfully ---");
-            
-            // Send final completion progress
-            progressUtils.outputProgress({
-                status: 'complete',
-                operation: 'All calculations completed',
-                message: `Successfully completed ${completedSteps.length} of ${stepsToRun.length} steps`,
-                current: stepsToRun.length,
-                total: stepsToRun.length,
-                elapsed: progressUtils.formatElapsedTime(overallStartTime),
-                remaining: '0s',
-                percentage: '100',
-                timing: {
-                    overall_start_time: new Date(overallStartTime).toISOString(),
-                    overall_end_time: new Date().toISOString(),
-                    total_duration_seconds: Math.round((Date.now() - overallStartTime) / 1000),
-                    step_timings: stepTimings,
-                    completed_steps: completedSteps.length
-                }
-            });
-            
-            progressUtils.clearProgress(); // Clear progress only on full success
-        }
-
-    } catch (error) {
-        clearTimeout(mainTimeoutHandle); // Clear the main timeout
-        console.error("\n--- SCRIPT EXECUTION FAILED ---");
-        // Error details were already logged by executeSqlStep or global handlers
-        overallSuccess = false;
-        
-        // Update the combined history record on error
-        if (combinedHistoryId) {
-            try {
-                connection = await getConnection();
-                const totalDuration = Math.round((Date.now() - overallStartTime) / 1000);
-                
-                await connection.query(`
-                    UPDATE calculate_history
-                    SET
-                        end_time = NOW(),
-                        duration_seconds = $1::integer,
-                        status = $2::calculation_status,
-                        error_message = $3::text
-                    WHERE id = $4::integer;
-                `, [
-                    totalDuration, 
-                    isCancelled ? 'cancelled' : 'failed', 
-                    error.message.substring(0, 1000), 
-                    combinedHistoryId
-                ]);
-                
-                connection.release();
-            } catch (historyError) {
-                console.error('Error updating combined history record on error:', historyError);
-                if (connection) connection.release();
-            }
-        }
-    } finally {
-        await closePool();
-        console.log(`Total execution time: ${progressUtils.formatElapsedTime(overallStartTime)}`);
-        process.exit(overallSuccess ? 0 : 1);
-    }
-}
-
-// --- Script Execution ---
-if (require.main === module) {
-    runAllCalculations();
-} else {
-    // Export functions if needed as a module (e.g., for testing or API)
-    module.exports = {
-        runAllCalculations,
-        cancelCalculation,
-        syncSettingsProductTable,
-        // Expose individual steps if useful, wrapping them slightly
-        runDailySnapshots: () => executeSqlStep({ name: 'Daily Snapshots Update', sqlFile: 'update_daily_snapshots.sql', historyType: 'daily_snapshots', statusModule: 'daily_snapshots' }, progressUtils),
-        runProductMetrics: () => executeSqlStep({ name: 'Product Metrics Update', sqlFile: 'update_product_metrics.sql', historyType: 'product_metrics', statusModule: 'product_metrics' }, progressUtils),
-        runPeriodicMetrics: () => executeSqlStep({ name: 'Periodic Metrics Update', sqlFile: 'update_periodic_metrics.sql', historyType: 'periodic_metrics', statusModule: 'periodic_metrics' }, progressUtils),
-        runBrandMetrics: () => executeSqlStep({ name: 'Brand Metrics Update', sqlFile: 'calculate_brand_metrics.sql', historyType: 'brand_metrics', statusModule: 'brand_metrics' }, progressUtils),
-        runVendorMetrics: () => executeSqlStep({ name: 'Vendor Metrics Update', sqlFile: 'calculate_vendor_metrics.sql', historyType: 'vendor_metrics', statusModule: 'vendor_metrics' }, progressUtils),
-        runCategoryMetrics: () => executeSqlStep({ name: 'Category Metrics Update', sqlFile: 'calculate_category_metrics.sql', historyType: 'category_metrics', statusModule: 'category_metrics' }, progressUtils),
-        getProgress: progressUtils.getProgress
-    };
-}

inventory-server/scripts/full-reset.js

@@ -1,115 +0,0 @@
-const path = require('path');
-const { spawn } = require('child_process');
-
-function outputProgress(data) {
-    if (!data.status) {
-        data = {
-            status: 'running',
-            ...data
-        };
-    }
-    console.log(JSON.stringify(data));
-}
-
-function runScript(scriptPath) {
-    return new Promise((resolve, reject) => {
-        const child = spawn('node', [scriptPath], {
-            stdio: ['inherit', 'pipe', 'pipe'],
-            env: {
-                ...process.env,
-                PGHOST: process.env.DB_HOST,
-                PGUSER: process.env.DB_USER,
-                PGPASSWORD: process.env.DB_PASSWORD,
-                PGDATABASE: process.env.DB_NAME,
-                PGPORT: process.env.DB_PORT || '5432'
-            }
-        });
-
-        let output = '';
-
-        child.stdout.on('data', (data) => {
-            const lines = data.toString().split('\n');
-            lines.filter(line => line.trim()).forEach(line => {
-                try {
-                    console.log(line); // Pass through the JSON output
-                    output += line + '\n';
-                } catch (e) {
-                    console.log(line); // If not JSON, just log it directly
-                }
-            });
-        });
-
-        child.stderr.on('data', (data) => {
-            console.error(data.toString());
-        });
-
-        child.on('close', (code) => {
-            if (code !== 0) {
-                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
-            } else {
-                resolve(output);
-            }
-        });
-
-        child.on('error', (err) => {
-            reject(err);
-        });
-    });
-}
-
-async function fullReset() {
-    try {
-        // Step 1: Reset Database
-        outputProgress({
-            operation: 'Starting full reset',
-            message: 'Step 1/3: Resetting database...'
-        });
-        await runScript(path.join(__dirname, 'reset-db.js'));
-        outputProgress({
-            status: 'complete',
-            operation: 'Database reset step complete',
-            message: 'Database reset finished, moving to import...'
-        });
-
-        // Step 2: Import from Production
-        outputProgress({
-            operation: 'Starting import',
-            message: 'Step 2/3: Importing from production...'
-        });
-        await runScript(path.join(__dirname, 'import-from-prod.js'));
-        outputProgress({
-            status: 'complete',
-            operation: 'Import step complete',
-            message: 'Import finished, moving to metrics calculation...'
-        });
-
-        // Step 3: Calculate Metrics
-        outputProgress({
-            operation: 'Starting metrics calculation',
-            message: 'Step 3/3: Calculating metrics...'
-        });
-        await runScript(path.join(__dirname, 'calculate-metrics-new.js'));
-        
-        // Final completion message
-        outputProgress({
-            status: 'complete',
-            operation: 'Full reset complete',
-            message: 'Successfully completed all steps: database reset, import, and metrics calculation'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'Full reset failed',
-            error: error.message,
-            stack: error.stack
-        });
-        process.exit(1);
-    }
-}
-
-// Run if called directly
-if (require.main === module) {
-    fullReset();
-}
-
-module.exports = fullReset; 

inventory-server/scripts/full-update.js

@@ -1,100 +0,0 @@
-const path = require('path');
-const { spawn } = require('child_process');
-
-function outputProgress(data) {
-    if (!data.status) {
-        data = {
-            status: 'running',
-            ...data
-        };
-    }
-    console.log(JSON.stringify(data));
-}
-
-function runScript(scriptPath) {
-    return new Promise((resolve, reject) => {
-        const child = spawn('node', [scriptPath], {
-            stdio: ['inherit', 'pipe', 'pipe']
-        });
-
-        let output = '';
-
-        child.stdout.on('data', (data) => {
-            const lines = data.toString().split('\n');
-            lines.filter(line => line.trim()).forEach(line => {
-                try {
-                    console.log(line); // Pass through the JSON output
-                    output += line + '\n';
-                } catch (e) {
-                    console.log(line); // If not JSON, just log it directly
-                }
-            });
-        });
-
-        child.stderr.on('data', (data) => {
-            console.error(data.toString());
-        });
-
-        child.on('close', (code) => {
-            if (code !== 0) {
-                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
-            } else {
-                resolve(output);
-            }
-        });
-
-        child.on('error', (err) => {
-            reject(err);
-        });
-    });
-}
-
-async function fullUpdate() {
-    try {
-        // Step 1: Import from Production
-        outputProgress({
-            operation: 'Starting full update',
-            message: 'Step 1/2: Importing from production...'
-        });
-        await runScript(path.join(__dirname, 'import-from-prod.js'));
-        outputProgress({
-            status: 'complete',
-            operation: 'Import step complete',
-            message: 'Import finished, moving to metrics calculation...'
-        });
-
-        // Step 2: Calculate Metrics
-        outputProgress({
-            operation: 'Starting metrics calculation',
-            message: 'Step 2/2: Calculating metrics...'
-        });
-        await runScript(path.join(__dirname, 'calculate-metrics-new.js'));
-        outputProgress({
-            status: 'complete',
-            operation: 'Metrics step complete',
-            message: 'Metrics calculation finished'
-        });
-
-        // Final completion message
-        outputProgress({
-            status: 'complete',
-            operation: 'Full update complete',
-            message: 'Successfully completed all steps: import and metrics calculation'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'Full update failed',
-            error: error.message,
-            stack: error.stack
-        });
-        process.exit(1);
-    }
-}
-
-// Run if called directly
-if (require.main === module) {
-    fullUpdate();
-}
-
-module.exports = fullUpdate; 

inventory-server/scripts/import-from-prod.js

@@ -1,352 +0,0 @@
-const dotenv = require("dotenv");
-const path = require("path");
-const { outputProgress, formatElapsedTime } = require('./metrics-new/utils/progress');
-const { setupConnections, closeConnections } = require('./import/utils');
-const importCategories = require('./import/categories');
-const { importProducts } = require('./import/products');
-const importOrders = require('./import/orders');
-const importPurchaseOrders = require('./import/purchase-orders');
-
-dotenv.config({ path: path.join(__dirname, "../.env") });
-
-// Constants to control which imports run
-const IMPORT_CATEGORIES = true;
-const IMPORT_PRODUCTS = true;
-const IMPORT_ORDERS = true;
-const IMPORT_PURCHASE_ORDERS = true;
-
-// Add flag for incremental updates
-const INCREMENTAL_UPDATE = process.env.INCREMENTAL_UPDATE !== 'false';  // Default to true unless explicitly set to false
-
-// SSH configuration
-const sshConfig = {
-  ssh: {
-    host: process.env.PROD_SSH_HOST,
-    port: process.env.PROD_SSH_PORT || 22,
-    username: process.env.PROD_SSH_USER,
-    privateKey: process.env.PROD_SSH_KEY_PATH
-      ? require("fs").readFileSync(process.env.PROD_SSH_KEY_PATH)
-      : undefined,
-    compress: true, // Enable SSH compression
-  },
-  prodDbConfig: {
-    // MySQL config for production
-    host: process.env.PROD_DB_HOST || "localhost",
-    user: process.env.PROD_DB_USER,
-    password: process.env.PROD_DB_PASSWORD,
-    database: process.env.PROD_DB_NAME,
-    port: process.env.PROD_DB_PORT || 3306,
-    timezone: '-05:00', // Production DB always stores times in EST (UTC-5) regardless of DST
-  },
-  localDbConfig: {
-    // PostgreSQL config for local
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432,
-    ssl: process.env.DB_SSL === 'true',
-    connectionTimeoutMillis: 60000,
-    idleTimeoutMillis: 30000,
-    max: 10 // connection pool max size
-  }
-};
-
-let isImportCancelled = false;
-
-// Add cancel function
-function cancelImport() {
-    isImportCancelled = true;
-    outputProgress({
-        status: 'cancelled',
-        operation: 'Import process',
-        message: 'Import cancelled by user',
-        current: 0,
-        total: 0,
-        elapsed: null,
-        remaining: null,
-        rate: 0
-    });
-}
-
-async function main() {
-  const startTime = Date.now();
-  let connections;
-  let completedSteps = 0;
-  let importHistoryId;
-  const totalSteps = [
-    IMPORT_CATEGORIES,
-    IMPORT_PRODUCTS,
-    IMPORT_ORDERS,
-    IMPORT_PURCHASE_ORDERS
-  ].filter(Boolean).length;
-
-  try {
-    // Initial progress update
-    outputProgress({
-      status: "running",
-      operation: "Import process",
-      message: `Initializing SSH tunnel for ${INCREMENTAL_UPDATE ? 'incremental' : 'full'} import...`,
-      current: completedSteps,
-      total: totalSteps,
-      elapsed: formatElapsedTime(startTime)
-    });
-
-    connections = await setupConnections(sshConfig);
-    const { prodConnection, localConnection } = connections;
-
-    if (isImportCancelled) throw new Error("Import cancelled");
-
-    // Clean up any previously running imports that weren't completed
-    await localConnection.query(`
-      UPDATE import_history 
-      SET 
-        status = 'cancelled',
-        end_time = NOW(),
-        duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
-        error_message = 'Previous import was not completed properly'
-      WHERE status = 'running'
-    `);
-
-    // Create import history record for the overall session
-    try {
-      const [historyResult] = await localConnection.query(`
-        INSERT INTO import_history (
-          table_name, 
-          start_time, 
-          is_incremental,
-          status,
-          additional_info
-        ) VALUES (
-          'all_tables',
-          NOW(),
-          $1::boolean,
-          'running',
-          jsonb_build_object(
-            'categories_enabled', $2::boolean,
-            'products_enabled', $3::boolean,
-            'orders_enabled', $4::boolean,
-            'purchase_orders_enabled', $5::boolean
-          )
-        ) RETURNING id
-      `, [INCREMENTAL_UPDATE, IMPORT_CATEGORIES, IMPORT_PRODUCTS, IMPORT_ORDERS, IMPORT_PURCHASE_ORDERS]);
-      importHistoryId = historyResult.rows[0].id;
-    } catch (error) {
-      console.error("Error creating import history record:", error);
-      outputProgress({
-        status: "error",
-        operation: "Import process",
-        message: "Failed to create import history record",
-        error: error.message
-      });
-      throw error;
-    }
-
-    const results = {
-      categories: null,
-      products: null,
-      orders: null,
-      purchaseOrders: null
-    };
-
-    let totalRecordsAdded = 0;
-    let totalRecordsUpdated = 0;
-    let totalRecordsDeleted = 0;  // Add tracking for deleted records
-    let totalRecordsSkipped = 0;  // Track skipped/filtered records
-    const stepTimings = {};
-
-    // Run each import based on constants
-    if (IMPORT_CATEGORIES) {
-      const stepStart = Date.now();
-      results.categories = await importCategories(prodConnection, localConnection);
-      stepTimings.categories = Math.round((Date.now() - stepStart) / 1000);
-      
-      if (isImportCancelled) throw new Error("Import cancelled");
-      completedSteps++;
-      console.log('Categories import result:', results.categories);
-      totalRecordsAdded += parseInt(results.categories?.recordsAdded || 0);
-      totalRecordsUpdated += parseInt(results.categories?.recordsUpdated || 0);
-    }
-
-    if (IMPORT_PRODUCTS) {
-      const stepStart = Date.now();
-      results.products = await importProducts(prodConnection, localConnection, INCREMENTAL_UPDATE);
-      stepTimings.products = Math.round((Date.now() - stepStart) / 1000);
-      
-      if (isImportCancelled) throw new Error("Import cancelled");
-      completedSteps++;
-      console.log('Products import result:', results.products);
-      totalRecordsAdded += parseInt(results.products?.recordsAdded || 0);
-      totalRecordsUpdated += parseInt(results.products?.recordsUpdated || 0);
-      totalRecordsSkipped += parseInt(results.products?.skippedUnchanged || 0);
-    }
-
-    if (IMPORT_ORDERS) {
-      const stepStart = Date.now();
-      results.orders = await importOrders(prodConnection, localConnection, INCREMENTAL_UPDATE);
-      stepTimings.orders = Math.round((Date.now() - stepStart) / 1000);
-      
-      if (isImportCancelled) throw new Error("Import cancelled");
-      completedSteps++;
-      console.log('Orders import result:', results.orders);
-      totalRecordsAdded += parseInt(results.orders?.recordsAdded || 0);
-      totalRecordsUpdated += parseInt(results.orders?.recordsUpdated || 0);
-      totalRecordsSkipped += parseInt(results.orders?.totalSkipped || 0);
-    }
-
-    if (IMPORT_PURCHASE_ORDERS) {
-      try {
-        const stepStart = Date.now();
-        results.purchaseOrders = await importPurchaseOrders(prodConnection, localConnection, INCREMENTAL_UPDATE);
-        stepTimings.purchaseOrders = Math.round((Date.now() - stepStart) / 1000);
-        
-        if (isImportCancelled) throw new Error("Import cancelled");
-        completedSteps++;
-        console.log('Purchase orders import result:', results.purchaseOrders);
-        
-        // Handle potential error status
-        if (results.purchaseOrders?.status === 'error') {
-          console.error('Purchase orders import had an error:', results.purchaseOrders.error);
-        } else {
-          totalRecordsAdded += parseInt(results.purchaseOrders?.recordsAdded || 0);
-          totalRecordsUpdated += parseInt(results.purchaseOrders?.recordsUpdated || 0);
-          totalRecordsDeleted += parseInt(results.purchaseOrders?.recordsDeleted || 0);
-        }
-      } catch (error) {
-        console.error('Error during purchase orders import:', error);
-        // Continue with other imports, don't fail the whole process
-        results.purchaseOrders = {
-          status: 'error',
-          error: error.message,
-          recordsAdded: 0,
-          recordsUpdated: 0
-        };
-      }
-    }
-
-    const endTime = Date.now();
-    const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
-
-    // Update import history with final stats
-    await localConnection.query(`
-      UPDATE import_history 
-      SET 
-        end_time = NOW(),
-        duration_seconds = $1,
-        records_added = $2,
-        records_updated = $3,
-        status = 'completed',
-        additional_info = jsonb_build_object(
-          'categories_enabled', $4::boolean,
-          'products_enabled', $5::boolean,
-          'orders_enabled', $6::boolean,
-          'purchase_orders_enabled', $7::boolean,
-          'categories_result', COALESCE($8::jsonb, 'null'::jsonb),
-          'products_result', COALESCE($9::jsonb, 'null'::jsonb),
-          'orders_result', COALESCE($10::jsonb, 'null'::jsonb),
-          'purchase_orders_result', COALESCE($11::jsonb, 'null'::jsonb),
-          'total_deleted', $12::integer,
-          'total_skipped', $13::integer,
-          'step_timings', $14::jsonb
-        )
-      WHERE id = $15
-    `, [
-      totalElapsedSeconds,
-      parseInt(totalRecordsAdded),
-      parseInt(totalRecordsUpdated),
-      IMPORT_CATEGORIES,
-      IMPORT_PRODUCTS,
-      IMPORT_ORDERS,
-      IMPORT_PURCHASE_ORDERS,
-      JSON.stringify(results.categories),
-      JSON.stringify(results.products),
-      JSON.stringify(results.orders),
-      JSON.stringify(results.purchaseOrders),
-      totalRecordsDeleted,
-      totalRecordsSkipped,
-      JSON.stringify(stepTimings),
-      importHistoryId
-    ]);
-
-    outputProgress({
-      status: "complete",
-      operation: "Import process",
-      message: `${INCREMENTAL_UPDATE ? 'Incremental' : 'Full'} import completed successfully in ${formatElapsedTime(totalElapsedSeconds)}`,
-      current: completedSteps,
-      total: totalSteps,
-      elapsed: formatElapsedTime(startTime),
-      timing: {
-        start_time: new Date(startTime).toISOString(),
-        end_time: new Date(endTime).toISOString(),
-        elapsed_time: formatElapsedTime(startTime),
-        elapsed_seconds: totalElapsedSeconds,
-        total_duration: formatElapsedTime(totalElapsedSeconds)
-      },
-      results
-    });
-
-    return results;
-  } catch (error) {
-    const endTime = Date.now();
-    const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
-
-    // Update import history with error
-    if (importHistoryId && connections?.localConnection) {
-      await connections.localConnection.query(`
-        UPDATE import_history 
-        SET 
-          end_time = NOW(),
-          duration_seconds = $1,
-          status = $2,
-          error_message = $3
-        WHERE id = $4
-      `, [totalElapsedSeconds, error.message === "Import cancelled" ? 'cancelled' : 'failed', error.message, importHistoryId]);
-    }
-
-    console.error("Error during import process:", error);
-    outputProgress({
-      status: error.message === "Import cancelled" ? "cancelled" : "error",
-      operation: "Import process",
-      message: error.message === "Import cancelled" 
-        ? `${INCREMENTAL_UPDATE ? 'Incremental' : 'Full'} import cancelled by user after ${formatElapsedTime(totalElapsedSeconds)}`
-        : `${INCREMENTAL_UPDATE ? 'Incremental' : 'Full'} import failed after ${formatElapsedTime(totalElapsedSeconds)}`,
-      error: error.message,
-      current: completedSteps,
-      total: totalSteps,
-      elapsed: formatElapsedTime(startTime),
-      timing: {
-        start_time: new Date(startTime).toISOString(),
-        end_time: new Date(endTime).toISOString(),
-        elapsed_time: formatElapsedTime(startTime),
-        elapsed_seconds: totalElapsedSeconds,
-        total_duration: formatElapsedTime(totalElapsedSeconds)
-      }
-    });
-    throw error;
-  } finally {
-    if (connections) {
-      await closeConnections(connections).catch(err => {
-        console.error("Error closing connections:", err);
-      });
-    }
-  }
-}
-
-// Run the import only if this is the main module
-if (require.main === module) {
-  main().then((results) => {
-    console.log('Import completed successfully:', results);
-    // Force exit after a small delay to ensure all logs are written
-    setTimeout(() => process.exit(0), 500);
-  }).catch((error) => {
-    console.error("Unhandled error in main process:", error);
-    // Force exit with error code after a small delay
-    setTimeout(() => process.exit(1), 500);
-  });
-}
-
-// Export the functions needed by the route
-module.exports = {
-  main,
-  cancelImport,
-};

inventory-server/scripts/import/categories.js

@@ -1,210 +0,0 @@
-const { outputProgress, formatElapsedTime } = require('../metrics-new/utils/progress');
-
-async function importCategories(prodConnection, localConnection) {
-  outputProgress({
-    operation: "Starting categories import",
-    status: "running",
-  });
-
-  const startTime = Date.now();
-  const typeOrder = [10, 20, 11, 21, 12, 13];
-  let totalInserted = 0;
-  let totalUpdated = 0;
-  let skippedCategories = [];
-
-  try {
-    // Start a single transaction for the entire import
-    await localConnection.query('BEGIN');
-    
-    // Temporarily disable the trigger that's causing problems
-    await localConnection.query('ALTER TABLE categories DISABLE TRIGGER update_categories_updated_at');
-
-    // Process each type in order with its own savepoint
-    for (const type of typeOrder) {
-      try {
-        // Create a savepoint for this type
-        await localConnection.query(`SAVEPOINT category_type_${type}`);
-
-        // Production query remains MySQL compatible
-        const [categories] = await prodConnection.query(
-          `
-          SELECT 
-            pc.cat_id,
-            pc.name,
-            pc.type,
-            CASE 
-              WHEN pc.type IN (10, 20) THEN NULL  -- Top level categories should have no parent
-              WHEN pc.master_cat_id IS NULL THEN NULL
-              ELSE pc.master_cat_id
-            END as parent_id,
-            pc.combined_name as description
-          FROM product_categories pc
-          WHERE pc.type = ?
-          ORDER BY pc.cat_id
-        `,
-          [type]
-        );
-
-        if (categories.length === 0) {
-          await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
-          continue;
-        }
-
-        console.log(`Processing ${categories.length} type ${type} categories`);
-
-        // For types that can have parents (11, 21, 12, 13), we'll proceed directly
-        // No need to check for parent existence since we process in hierarchical order
-        let categoriesToInsert = categories;
-
-        if (categoriesToInsert.length === 0) {
-          console.log(`No valid categories of type ${type} to insert`);
-          await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
-          continue;
-        }
-
-        // PostgreSQL upsert query with parameterized values
-        const values = categoriesToInsert.flatMap((cat) => [
-          cat.cat_id,
-          cat.name,
-          cat.type,
-          cat.parent_id,
-          cat.description,
-          'active',
-          new Date(),
-          new Date()
-        ]);
-
-        const placeholders = categoriesToInsert
-          .map((_, i) => `($${i * 8 + 1}, $${i * 8 + 2}, $${i * 8 + 3}, $${i * 8 + 4}, $${i * 8 + 5}, $${i * 8 + 6}, $${i * 8 + 7}, $${i * 8 + 8})`)
-          .join(',');
-
-        // Insert categories with ON CONFLICT clause for PostgreSQL
-        const query = `
-          WITH inserted_categories AS (
-            INSERT INTO categories (
-              cat_id, name, type, parent_id, description, status, created_at, updated_at
-            )
-            VALUES ${placeholders}
-            ON CONFLICT (cat_id) DO UPDATE SET
-              name = EXCLUDED.name,
-              type = EXCLUDED.type,
-              parent_id = EXCLUDED.parent_id,
-              description = EXCLUDED.description,
-              status = EXCLUDED.status,
-              updated_at = EXCLUDED.updated_at
-            WHERE -- Only update if at least one field has changed
-              categories.name IS DISTINCT FROM EXCLUDED.name OR
-              categories.type IS DISTINCT FROM EXCLUDED.type OR
-              categories.parent_id IS DISTINCT FROM EXCLUDED.parent_id OR
-              categories.description IS DISTINCT FROM EXCLUDED.description OR
-              categories.status IS DISTINCT FROM EXCLUDED.status
-            RETURNING 
-              cat_id,
-              CASE 
-                WHEN xmax = 0 THEN true
-                ELSE false
-              END as is_insert
-          )
-          SELECT 
-            COUNT(*) as total,
-            COUNT(*) FILTER (WHERE is_insert) as inserted,
-            COUNT(*) FILTER (WHERE NOT is_insert) as updated
-          FROM inserted_categories`;
-        
-        const result = await localConnection.query(query, values);
-
-        // Get the first result since query returns an array
-        const queryResult = Array.isArray(result) ? result[0] : result;
-
-        if (!queryResult || !queryResult.rows || !queryResult.rows[0]) {
-          console.error('Query failed to return results');
-          throw new Error('Query did not return expected results');
-        }
-
-        const total = parseInt(queryResult.rows[0].total) || 0;
-        const inserted = parseInt(queryResult.rows[0].inserted) || 0;
-        const updated = parseInt(queryResult.rows[0].updated) || 0;
-
-        console.log(`Total: ${total}, Inserted: ${inserted}, Updated: ${updated}`);
-        
-        totalInserted += inserted;
-        totalUpdated += updated;
-
-        // Release the savepoint for this type
-        await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
-
-        outputProgress({
-          status: "running",
-          operation: "Categories import",
-          message: `Imported ${inserted} (updated ${updated}) categories of type ${type}`,
-          current: totalInserted + totalUpdated,
-          total: categories.length,
-          elapsed: formatElapsedTime(startTime),
-        });
-      } catch (error) {
-        // Rollback to the savepoint for this type
-        await localConnection.query(`ROLLBACK TO SAVEPOINT category_type_${type}`);
-        throw error;
-      }
-    }
-
-    // Commit the entire transaction - we'll do this even if we have skipped categories
-    await localConnection.query('COMMIT');
-
-    // Update sync status
-    await localConnection.query(`
-      INSERT INTO sync_status (table_name, last_sync_timestamp)
-      VALUES ('categories', NOW())
-      ON CONFLICT (table_name) DO UPDATE SET
-        last_sync_timestamp = NOW()
-    `);
-    
-    // Re-enable the trigger
-    await localConnection.query('ALTER TABLE categories ENABLE TRIGGER update_categories_updated_at');
-
-    outputProgress({
-      status: "complete",
-      operation: "Categories import completed",
-      current: totalInserted + totalUpdated,
-      total: totalInserted + totalUpdated,
-      duration: formatElapsedTime(startTime),
-      warnings: skippedCategories.length > 0 ? {
-        message: "Some categories were skipped due to missing parents",
-        skippedCategories
-      } : undefined
-    });
-
-    return {
-      status: "complete",
-      recordsAdded: totalInserted,
-      recordsUpdated: totalUpdated,
-      totalRecords: totalInserted + totalUpdated,
-      warnings: skippedCategories.length > 0 ? {
-        message: "Some categories were skipped due to missing parents",
-        skippedCategories
-      } : undefined
-    };
-  } catch (error) {
-    console.error("Error importing categories:", error);
-    
-    // Only rollback if we haven't committed yet
-    try {
-      await localConnection.query('ROLLBACK');
-      
-      // Make sure we re-enable the trigger even if there was an error
-      await localConnection.query('ALTER TABLE categories ENABLE TRIGGER update_categories_updated_at');
-    } catch (rollbackError) {
-      console.error("Error during rollback:", rollbackError);
-    }
-    
-    outputProgress({
-      status: "error",
-      operation: "Categories import failed",
-      error: error.message
-    });
-
-    throw error;
-  }
-}
-
-module.exports = importCategories; 

inventory-server/scripts/import/orders.js

@@ -1,779 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../metrics-new/utils/progress');
-const { importMissingProducts, setupTemporaryTables, cleanupTemporaryTables, materializeCalculations } = require('./products');
-
-/**
- * Imports orders from a production MySQL database to a local PostgreSQL database.
- * It can run in two modes:
- *   1. Incremental update mode (default): Only fetch orders that have changed since the last sync time.
- *   2. Full update mode: Fetch all eligible orders within the last 5 years regardless of timestamp.
- *
- * @param {object} prodConnection - A MySQL connection to production DB (MySQL 5.7).
- * @param {object} localConnection - A MySQL connection to local DB (MySQL 8.0).
- * @param {boolean} incrementalUpdate - Set to false for a full sync; true for incremental.
- *
- * @returns {object} Information about the sync operation.
- */
-async function importOrders(prodConnection, localConnection, incrementalUpdate = true) {
-  const startTime = Date.now();
-  const skippedOrders = new Set();
-  const missingProducts = new Set();
-  let recordsAdded = 0;
-  let recordsUpdated = 0;
-  let processedCount = 0;
-  let importedCount = 0;
-  let totalOrderItems = 0;
-  let totalUniqueOrders = 0;
-  let cumulativeProcessedOrders = 0;
-
-  try {
-    // Get last sync info - NOT in a transaction anymore
-    const [syncInfo] = await localConnection.query(
-      "SELECT last_sync_timestamp FROM sync_status WHERE table_name = 'orders'"
-    );
-    const lastSyncTime = syncInfo?.rows?.[0]?.last_sync_timestamp || '1970-01-01';
-
-    console.log('Orders: Using last sync time:', lastSyncTime);
-
-    // First get count of order items - Keep MySQL compatible for production
-    const [[{ total }]] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM order_items oi
-      JOIN _order o ON oi.order_id = o.order_id
-      WHERE o.order_status >= 15
-        AND o.date_placed >= DATE_SUB(CURRENT_DATE, INTERVAL ${incrementalUpdate ? '1' : '5'} YEAR)
-        AND o.date_placed IS NOT NULL
-        ${incrementalUpdate ? `
-          AND (
-            o.stamp > ? 
-            OR oi.stamp > ?
-            OR EXISTS (
-              SELECT 1 FROM order_discount_items odi 
-              WHERE odi.order_id = o.order_id 
-              AND odi.pid = oi.prod_pid
-            )
-            OR EXISTS (
-              SELECT 1 FROM order_tax_info oti 
-              JOIN order_tax_info_products otip ON oti.taxinfo_id = otip.taxinfo_id
-              WHERE oti.order_id = o.order_id 
-              AND otip.pid = oi.prod_pid
-              AND oti.stamp > ?
-            )
-          )
-        ` : ''}
-    `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime] : []);
-
-    totalOrderItems = total;
-    console.log('Orders: Found changes:', totalOrderItems);
-
-    // Get order items - Keep MySQL compatible for production
-    console.log('Orders: Starting MySQL query...');
-    const [orderItems] = await prodConnection.query(`
-      SELECT 
-        oi.order_id,
-        oi.prod_pid,
-        COALESCE(NULLIF(TRIM(oi.prod_itemnumber), ''), 'NO-SKU') as SKU,
-        oi.prod_price as price,
-        oi.qty_ordered as quantity,
-        COALESCE(oi.prod_price_reg - oi.prod_price, 0) as base_discount,
-        oi.stamp as last_modified
-      FROM order_items oi
-      JOIN _order o ON oi.order_id = o.order_id
-      WHERE o.order_status >= 15
-        AND o.date_placed >= DATE_SUB(CURRENT_DATE, INTERVAL ${incrementalUpdate ? '1' : '5'} YEAR)
-        AND o.date_placed IS NOT NULL
-        ${incrementalUpdate ? `
-          AND (
-            o.stamp > ? 
-            OR oi.stamp > ?
-            OR EXISTS (
-              SELECT 1 FROM order_discount_items odi 
-              WHERE odi.order_id = o.order_id 
-              AND odi.pid = oi.prod_pid
-            )
-            OR EXISTS (
-              SELECT 1 FROM order_tax_info oti 
-              JOIN order_tax_info_products otip ON oti.taxinfo_id = otip.taxinfo_id
-              WHERE oti.order_id = o.order_id 
-              AND otip.pid = oi.prod_pid
-              AND oti.stamp > ?
-            )
-          )
-        ` : ''}
-    `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime] : []);
-
-    console.log('Orders: Found', orderItems.length, 'order items to process');
-
-    // Create tables in PostgreSQL for data processing
-    // Start a transaction just for creating the temp tables
-    await localConnection.beginTransaction();
-    try {
-      await localConnection.query(`
-        DROP TABLE IF EXISTS temp_order_items;
-        DROP TABLE IF EXISTS temp_order_meta;
-        DROP TABLE IF EXISTS temp_order_discounts;
-        DROP TABLE IF EXISTS temp_order_taxes;
-        DROP TABLE IF EXISTS temp_order_costs;
-        DROP TABLE IF EXISTS temp_main_discounts;
-        DROP TABLE IF EXISTS temp_item_discounts;
-
-        CREATE TEMP TABLE temp_order_items (
-          order_id INTEGER NOT NULL,
-          pid INTEGER NOT NULL,
-          sku TEXT NOT NULL,
-          price NUMERIC(14, 4) NOT NULL,
-          quantity INTEGER NOT NULL,
-          base_discount NUMERIC(14, 4) DEFAULT 0,
-          PRIMARY KEY (order_id, pid)
-        );
-
-        CREATE TEMP TABLE temp_order_meta (
-          order_id INTEGER NOT NULL,
-          date TIMESTAMP WITH TIME ZONE NOT NULL,
-          customer TEXT NOT NULL,
-          customer_name TEXT NOT NULL,
-          status TEXT,
-          canceled BOOLEAN,
-          summary_discount NUMERIC(14, 4) DEFAULT 0.0000,
-          summary_subtotal NUMERIC(14, 4) DEFAULT 0.0000,
-          summary_discount_subtotal NUMERIC(14, 4) DEFAULT 0.0000,
-          PRIMARY KEY (order_id)
-        );
-
-        CREATE TEMP TABLE temp_order_discounts (
-          order_id INTEGER NOT NULL,
-          pid INTEGER NOT NULL,
-          discount NUMERIC(14, 4) NOT NULL,
-          PRIMARY KEY (order_id, pid)
-        );
-
-        CREATE TEMP TABLE temp_main_discounts (
-          order_id INTEGER NOT NULL,
-          discount_id INTEGER NOT NULL,
-          discount_amount_subtotal NUMERIC(14, 4) DEFAULT 0.0000,
-          PRIMARY KEY (order_id, discount_id)
-        );
-
-        CREATE TEMP TABLE temp_item_discounts (
-          order_id INTEGER NOT NULL,
-          pid INTEGER NOT NULL,
-          discount_id INTEGER NOT NULL,
-          amount NUMERIC(14, 4) NOT NULL,
-          PRIMARY KEY (order_id, pid, discount_id)
-        );
-
-        CREATE TEMP TABLE temp_order_taxes (
-          order_id INTEGER NOT NULL,
-          pid INTEGER NOT NULL,
-          tax NUMERIC(14, 4) NOT NULL,
-          PRIMARY KEY (order_id, pid)
-        );
-
-        CREATE TEMP TABLE temp_order_costs (
-          order_id INTEGER NOT NULL,
-          pid INTEGER NOT NULL,
-          costeach NUMERIC(14, 4) DEFAULT 0.0000,
-          PRIMARY KEY (order_id, pid)
-        );
-
-        CREATE INDEX idx_temp_order_items_pid ON temp_order_items(pid);
-        CREATE INDEX idx_temp_order_meta_order_id ON temp_order_meta(order_id);
-        CREATE INDEX idx_temp_order_discounts_order_pid ON temp_order_discounts(order_id, pid);
-        CREATE INDEX idx_temp_order_taxes_order_pid ON temp_order_taxes(order_id, pid);
-        CREATE INDEX idx_temp_order_costs_order_pid ON temp_order_costs(order_id, pid);
-        CREATE INDEX idx_temp_main_discounts_discount_id ON temp_main_discounts(discount_id);
-        CREATE INDEX idx_temp_item_discounts_order_pid ON temp_item_discounts(order_id, pid);
-        CREATE INDEX idx_temp_item_discounts_discount_id ON temp_item_discounts(discount_id);
-      `);
-      await localConnection.commit();
-    } catch (error) {
-      await localConnection.rollback();
-      throw error;
-    }
-
-    // Insert order items in batches - each batch gets its own transaction
-    for (let i = 0; i < orderItems.length; i += 5000) {
-      await localConnection.beginTransaction();
-      try {
-        const batch = orderItems.slice(i, Math.min(i + 5000, orderItems.length));
-        const placeholders = batch.map((_, idx) => 
-          `($${idx * 6 + 1}, $${idx * 6 + 2}, $${idx * 6 + 3}, $${idx * 6 + 4}, $${idx * 6 + 5}, $${idx * 6 + 6})`
-        ).join(",");
-        const values = batch.flatMap(item => [
-          item.order_id, item.prod_pid, item.SKU, item.price, item.quantity, item.base_discount
-        ]);
-
-        await localConnection.query(`
-          INSERT INTO temp_order_items (order_id, pid, sku, price, quantity, base_discount)
-          VALUES ${placeholders}
-          ON CONFLICT (order_id, pid) DO UPDATE SET
-            sku = EXCLUDED.sku,
-            price = EXCLUDED.price,
-            quantity = EXCLUDED.quantity,
-            base_discount = EXCLUDED.base_discount
-        `, values);
-
-        await localConnection.commit();
-        
-        processedCount = i + batch.length;
-        outputProgress({
-          status: "running",
-          operation: "Orders import",
-          message: `Loading order items: ${processedCount} of ${totalOrderItems}`,
-          current: processedCount,
-          total: totalOrderItems,
-                      elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, processedCount, totalOrderItems),
-          rate: calculateRate(startTime, processedCount)
-        });
-      } catch (error) {
-        await localConnection.rollback();
-        throw error;
-      }
-    }
-
-    // Get unique order IDs
-    const orderIds = [...new Set(orderItems.map(item => item.order_id))];
-    totalUniqueOrders = orderIds.length;
-    console.log('Orders: Processing', totalUniqueOrders, 'unique orders');
-
-    // Reset processed count for order processing phase
-    processedCount = 0;
-
-    // Process metadata, discounts, taxes, and costs in parallel
-    const METADATA_BATCH_SIZE = 2000;
-    const PG_BATCH_SIZE = 200;
-    
-    // Add a helper function for title case conversion
-    function toTitleCase(str) {
-      if (!str) return '';
-      return str.toLowerCase().split(' ').map(word => {
-        return word.charAt(0).toUpperCase() + word.slice(1);
-      }).join(' ');
-    }
-
-    const processMetadataBatch = async (batchIds) => {
-      const [orders] = await prodConnection.query(`
-        SELECT 
-          o.order_id,
-          o.date_placed as date,
-          o.order_cid as customer,
-          CONCAT(COALESCE(u.firstname, ''), ' ', COALESCE(u.lastname, '')) as customer_name,
-          o.order_status as status,
-          CASE WHEN o.date_cancelled != '0000-00-00 00:00:00' THEN 1 ELSE 0 END as canceled,
-          o.summary_discount,
-          o.summary_subtotal,
-          o.summary_discount_subtotal
-        FROM _order o
-        LEFT JOIN users u ON o.order_cid = u.cid
-        WHERE o.order_id IN (?)
-      `, [batchIds]);
-
-      // Process in sub-batches for PostgreSQL
-      await localConnection.beginTransaction();
-      try {
-        for (let j = 0; j < orders.length; j += PG_BATCH_SIZE) {
-          const subBatch = orders.slice(j, j + PG_BATCH_SIZE);
-          if (subBatch.length === 0) continue;
-
-          const placeholders = subBatch.map((_, idx) => 
-            `($${idx * 9 + 1}, $${idx * 9 + 2}, $${idx * 9 + 3}, $${idx * 9 + 4}, $${idx * 9 + 5}, $${idx * 9 + 6}, $${idx * 9 + 7}, $${idx * 9 + 8}, $${idx * 9 + 9})`
-          ).join(",");
-          
-          const values = subBatch.flatMap(order => [
-            order.order_id,
-            new Date(order.date), // Convert to TIMESTAMP WITH TIME ZONE
-            order.customer,
-            toTitleCase(order.customer_name) || '',
-            order.status.toString(), // Convert status to TEXT
-            order.canceled,
-            order.summary_discount || 0,
-            order.summary_subtotal || 0,
-            order.summary_discount_subtotal || 0
-          ]);
-
-          await localConnection.query(`
-            INSERT INTO temp_order_meta (
-              order_id, date, customer, customer_name, status, canceled,
-              summary_discount, summary_subtotal, summary_discount_subtotal
-            )
-            VALUES ${placeholders}
-            ON CONFLICT (order_id) DO UPDATE SET
-              date = EXCLUDED.date,
-              customer = EXCLUDED.customer,
-              customer_name = EXCLUDED.customer_name,
-              status = EXCLUDED.status,
-              canceled = EXCLUDED.canceled,
-              summary_discount = EXCLUDED.summary_discount,
-              summary_subtotal = EXCLUDED.summary_subtotal,
-              summary_discount_subtotal = EXCLUDED.summary_discount_subtotal
-          `, values);
-        }
-        await localConnection.commit();
-      } catch (error) {
-        await localConnection.rollback();
-        throw error;
-      }
-    };
-
-    const processDiscountsBatch = async (batchIds) => {
-      // First, load main discount records
-      const [mainDiscounts] = await prodConnection.query(`
-        SELECT order_id, discount_id, discount_amount_subtotal
-        FROM order_discounts
-        WHERE order_id IN (?)
-      `, [batchIds]);
-
-      if (mainDiscounts.length > 0) {
-        await localConnection.beginTransaction();
-        try {
-          for (let j = 0; j < mainDiscounts.length; j += PG_BATCH_SIZE) {
-            const subBatch = mainDiscounts.slice(j, j + PG_BATCH_SIZE);
-            if (subBatch.length === 0) continue;
-
-            const placeholders = subBatch.map((_, idx) => 
-              `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
-            ).join(",");
-            
-            const values = subBatch.flatMap(d => [
-              d.order_id,
-              d.discount_id,
-              d.discount_amount_subtotal || 0
-            ]);
-
-            await localConnection.query(`
-              INSERT INTO temp_main_discounts (order_id, discount_id, discount_amount_subtotal)
-              VALUES ${placeholders}
-              ON CONFLICT (order_id, discount_id) DO UPDATE SET
-                discount_amount_subtotal = EXCLUDED.discount_amount_subtotal
-            `, values);
-          }
-          await localConnection.commit();
-        } catch (error) {
-          await localConnection.rollback();
-          throw error;
-        }
-      }
-
-      // Then, load item discount records
-      const [discounts] = await prodConnection.query(`
-        SELECT order_id, pid, discount_id, amount
-        FROM order_discount_items
-        WHERE order_id IN (?)
-      `, [batchIds]);
-
-      if (discounts.length === 0) return;
-
-      // Process in memory to handle potential duplicates
-      const discountMap = new Map();
-      for (const d of discounts) {
-        const key = `${d.order_id}-${d.pid}-${d.discount_id}`;
-        discountMap.set(key, d);
-      }
-
-      const uniqueDiscounts = Array.from(discountMap.values());
-
-      await localConnection.beginTransaction();
-      try {
-        for (let j = 0; j < uniqueDiscounts.length; j += PG_BATCH_SIZE) {
-          const subBatch = uniqueDiscounts.slice(j, j + PG_BATCH_SIZE);
-          if (subBatch.length === 0) continue;
-
-          const placeholders = subBatch.map((_, idx) => 
-            `($${idx * 4 + 1}, $${idx * 4 + 2}, $${idx * 4 + 3}, $${idx * 4 + 4})`
-          ).join(",");
-          
-          const values = subBatch.flatMap(d => [
-            d.order_id,
-            d.pid,
-            d.discount_id,
-            d.amount || 0
-          ]);
-
-          await localConnection.query(`
-            INSERT INTO temp_item_discounts (order_id, pid, discount_id, amount)
-            VALUES ${placeholders}
-            ON CONFLICT (order_id, pid, discount_id) DO UPDATE SET
-              amount = EXCLUDED.amount
-          `, values);
-        }
-
-        // Create aggregated view with a simpler, safer query that avoids duplicates
-        await localConnection.query(`
-          TRUNCATE temp_order_discounts;
-          
-          INSERT INTO temp_order_discounts (order_id, pid, discount)
-          SELECT order_id, pid, SUM(amount) as discount
-          FROM temp_item_discounts
-          GROUP BY order_id, pid
-        `);
-
-        await localConnection.commit();
-      } catch (error) {
-        await localConnection.rollback();
-        throw error;
-      }
-    };
-
-    const processTaxesBatch = async (batchIds) => {
-      // Optimized tax query to avoid subquery
-      const [taxes] = await prodConnection.query(`
-        SELECT oti.order_id, otip.pid, otip.item_taxes_to_collect as tax
-        FROM (
-          SELECT order_id, MAX(taxinfo_id) as latest_taxinfo_id
-          FROM order_tax_info
-          WHERE order_id IN (?)
-          GROUP BY order_id
-        ) latest_info
-        JOIN order_tax_info oti ON oti.order_id = latest_info.order_id 
-          AND oti.taxinfo_id = latest_info.latest_taxinfo_id
-        JOIN order_tax_info_products otip ON oti.taxinfo_id = otip.taxinfo_id
-      `, [batchIds]);
-
-      if (taxes.length === 0) return;
-
-      await localConnection.beginTransaction();
-      try {
-        for (let j = 0; j < taxes.length; j += PG_BATCH_SIZE) {
-          const subBatch = taxes.slice(j, j + PG_BATCH_SIZE);
-          if (subBatch.length === 0) continue;
-
-          const placeholders = subBatch.map((_, idx) => 
-            `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
-          ).join(",");
-          
-          const values = subBatch.flatMap(t => [
-            t.order_id,
-            t.pid,
-            t.tax || 0
-          ]);
-
-          await localConnection.query(`
-            INSERT INTO temp_order_taxes (order_id, pid, tax)
-            VALUES ${placeholders}
-            ON CONFLICT (order_id, pid) DO UPDATE SET
-              tax = EXCLUDED.tax
-          `, values);
-        }
-        await localConnection.commit();
-      } catch (error) {
-        await localConnection.rollback();
-        throw error;
-      }
-    };
-
-    const processCostsBatch = async (batchIds) => {
-      // Modified query to ensure one row per order_id/pid by using a subquery
-      const [costs] = await prodConnection.query(`
-        SELECT 
-          oc.orderid as order_id,
-          oc.pid,
-          oc.costeach
-        FROM order_costs oc
-        INNER JOIN (
-          SELECT 
-            orderid,
-            pid, 
-            MAX(id) as max_id
-          FROM order_costs
-          WHERE orderid IN (?) 
-            AND pending = 0
-          GROUP BY orderid, pid
-        ) latest ON oc.orderid = latest.orderid AND oc.pid = latest.pid AND oc.id = latest.max_id
-      `, [batchIds]);
-
-      if (costs.length === 0) return;
-
-      await localConnection.beginTransaction();
-      try {
-        for (let j = 0; j < costs.length; j += PG_BATCH_SIZE) {
-          const subBatch = costs.slice(j, j + PG_BATCH_SIZE);
-          if (subBatch.length === 0) continue;
-
-          const placeholders = subBatch.map((_, idx) => 
-            `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
-          ).join(",");
-          
-          const values = subBatch.flatMap(c => [
-            c.order_id,
-            c.pid,
-            c.costeach || 0
-          ]);
-
-          await localConnection.query(`
-            INSERT INTO temp_order_costs (order_id, pid, costeach)
-            VALUES ${placeholders}
-            ON CONFLICT (order_id, pid) DO UPDATE SET
-              costeach = EXCLUDED.costeach
-          `, values);
-        }
-        await localConnection.commit();
-      } catch (error) {
-        await localConnection.rollback();
-        throw error;
-      }
-    };
-
-    // Process all data types SEQUENTIALLY for each batch - not in parallel
-    for (let i = 0; i < orderIds.length; i += METADATA_BATCH_SIZE) {
-      const batchIds = orderIds.slice(i, i + METADATA_BATCH_SIZE);
-      
-      // Run these sequentially instead of in parallel to avoid transaction conflicts
-      await processMetadataBatch(batchIds);
-      await processDiscountsBatch(batchIds);
-      await processTaxesBatch(batchIds);
-      await processCostsBatch(batchIds);
-
-      processedCount = i + batchIds.length;
-      outputProgress({
-        status: "running",
-        operation: "Orders import",
-        message: `Loading order data: ${processedCount} of ${totalUniqueOrders}`,
-        current: processedCount,
-        total: totalUniqueOrders,
-                        elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalUniqueOrders),
-        rate: calculateRate(startTime, processedCount)
-      });
-    }
-
-    // Pre-check all products at once
-    const allOrderPids = [...new Set(orderItems.map(item => item.prod_pid))];
-    console.log('Orders: Checking', allOrderPids.length, 'unique products');
-    
-    const [existingProducts] = allOrderPids.length > 0 ? await localConnection.query(
-      "SELECT pid FROM products WHERE pid = ANY($1::bigint[])",
-      [allOrderPids]
-    ) : [[]];
-    
-    const existingPids = new Set(existingProducts.rows.map(p => p.pid));
-    
-    // Process in smaller batches
-    for (let i = 0; i < orderIds.length; i += 2000) {  // Increased from 1000 to 2000
-      const batchIds = orderIds.slice(i, i + 2000);
-
-      // Get combined data for this batch in sub-batches
-      const PG_BATCH_SIZE = 200; // Increased from 100 to 200
-      for (let j = 0; j < batchIds.length; j += PG_BATCH_SIZE) {
-        const subBatchIds = batchIds.slice(j, j + PG_BATCH_SIZE);
-        
-        // Start a transaction for this sub-batch
-        await localConnection.beginTransaction();
-        try {
-          const [orders] = await localConnection.query(`
-            WITH order_totals AS (
-              SELECT 
-                oi.order_id,
-                oi.pid,
-                -- Instead of using ARRAY_AGG which can cause duplicate issues, use SUM with a CASE
-                SUM(CASE 
-                  WHEN COALESCE(md.discount_amount_subtotal, 0) > 0 THEN id.amount 
-                  ELSE 0 
-                END) as promo_discount_sum,
-                COALESCE(ot.tax, 0) as total_tax,
-                COALESCE(oc.costeach, oi.price * 0.5) as costeach
-              FROM temp_order_items oi
-              LEFT JOIN temp_item_discounts id ON oi.order_id = id.order_id AND oi.pid = id.pid
-              LEFT JOIN temp_main_discounts md ON id.order_id = md.order_id AND id.discount_id = md.discount_id
-              LEFT JOIN temp_order_taxes ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
-              LEFT JOIN temp_order_costs oc ON oi.order_id = oc.order_id AND oi.pid = oc.pid
-              WHERE oi.order_id = ANY($1)
-              GROUP BY oi.order_id, oi.pid, ot.tax, oc.costeach
-            )
-            SELECT 
-              oi.order_id as order_number,
-              oi.pid::bigint as pid,
-              oi.sku,
-              om.date,
-              oi.price,
-              oi.quantity,
-              (
-                -- Part 1: Sale Savings for the Line
-                (oi.base_discount * oi.quantity)
-                +
-                -- Part 2: Prorated Points Discount (if applicable)
-                CASE
-                  WHEN om.summary_discount_subtotal > 0 AND om.summary_subtotal > 0 THEN
-                    COALESCE(ROUND((om.summary_discount_subtotal * (oi.price * oi.quantity)) / NULLIF(om.summary_subtotal, 0), 4), 0)
-                  ELSE 0
-                END
-                +
-                -- Part 3: Specific Item-Level Discount (only if parent discount affected subtotal)
-                COALESCE(ot.promo_discount_sum, 0)
-              )::NUMERIC(14, 4) as discount,
-              COALESCE(ot.total_tax, 0)::NUMERIC(14, 4) as tax,
-              false as tax_included,
-              0 as shipping,
-              om.customer,
-              om.customer_name,
-              om.status,
-              om.canceled,
-              COALESCE(ot.costeach, oi.price * 0.5)::NUMERIC(14, 4) as costeach
-            FROM temp_order_items oi
-            JOIN temp_order_meta om ON oi.order_id = om.order_id
-            LEFT JOIN order_totals ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
-            WHERE oi.order_id = ANY($1)
-            ORDER BY oi.order_id, oi.pid
-          `, [subBatchIds]);
-
-          // Filter orders and track missing products
-          const validOrders = [];
-          const processedOrderItems = new Set();
-          const processedOrders = new Set();
-          
-          for (const order of orders.rows) {
-            if (!existingPids.has(order.pid)) {
-              missingProducts.add(order.pid);
-              skippedOrders.add(order.order_number);
-              continue;
-            }
-            validOrders.push(order);
-            processedOrderItems.add(`${order.order_number}-${order.pid}`);
-            processedOrders.add(order.order_number);
-          }
-
-          // Process valid orders in smaller sub-batches
-          const FINAL_BATCH_SIZE = 100;  // Increased from 50 to 100
-          for (let k = 0; k < validOrders.length; k += FINAL_BATCH_SIZE) {
-            const subBatch = validOrders.slice(k, k + FINAL_BATCH_SIZE);
-            
-            const placeholders = subBatch.map((_, idx) => {
-              const base = idx * 15;  // 15 columns including costeach
-              return `($${base + 1}, $${base + 2}, $${base + 3}, $${base + 4}, $${base + 5}, $${base + 6}, $${base + 7}, $${base + 8}, $${base + 9}, $${base + 10}, $${base + 11}, $${base + 12}, $${base + 13}, $${base + 14}, $${base + 15})`;
-            }).join(',');
-
-            const batchValues = subBatch.flatMap(o => [
-              o.order_number,
-              o.pid,
-              o.sku || 'NO-SKU',
-              o.date,  // This is now a TIMESTAMP WITH TIME ZONE
-              o.price,
-              o.quantity,
-              o.discount,
-              o.tax,
-              o.tax_included,
-              o.shipping,
-              o.customer,
-              o.customer_name,
-              o.status.toString(), // Convert status to TEXT
-              o.canceled,
-              o.costeach
-            ]);
-
-            const [result] = await localConnection.query(`
-              WITH inserted_orders AS (
-                INSERT INTO orders (
-                  order_number, pid, sku, date, price, quantity, discount,
-                  tax, tax_included, shipping, customer, customer_name,
-                  status, canceled, costeach
-                )
-                VALUES ${placeholders}
-                ON CONFLICT (order_number, pid) DO UPDATE SET
-                  sku = EXCLUDED.sku,
-                  date = EXCLUDED.date,
-                  price = EXCLUDED.price,
-                  quantity = EXCLUDED.quantity,
-                  discount = EXCLUDED.discount,
-                  tax = EXCLUDED.tax,
-                  tax_included = EXCLUDED.tax_included,
-                  shipping = EXCLUDED.shipping,
-                  customer = EXCLUDED.customer,
-                  customer_name = EXCLUDED.customer_name,
-                  status = EXCLUDED.status,
-                  canceled = EXCLUDED.canceled,
-                  costeach = EXCLUDED.costeach
-                WHERE -- Only update if at least one key field has changed
-                  orders.price IS DISTINCT FROM EXCLUDED.price OR
-                  orders.quantity IS DISTINCT FROM EXCLUDED.quantity OR
-                  orders.discount IS DISTINCT FROM EXCLUDED.discount OR
-                  orders.tax IS DISTINCT FROM EXCLUDED.tax OR
-                  orders.status IS DISTINCT FROM EXCLUDED.status OR
-                  orders.canceled IS DISTINCT FROM EXCLUDED.canceled OR
-                  orders.costeach IS DISTINCT FROM EXCLUDED.costeach OR
-                  orders.date IS DISTINCT FROM EXCLUDED.date
-                RETURNING xmax = 0 as inserted
-              )
-              SELECT 
-                COUNT(*) FILTER (WHERE inserted) as inserted,
-                COUNT(*) FILTER (WHERE NOT inserted) as updated
-              FROM inserted_orders
-            `, batchValues);
-            
-            const { inserted, updated } = result.rows[0];
-            recordsAdded += parseInt(inserted) || 0;
-            recordsUpdated += parseInt(updated) || 0;
-            importedCount += subBatch.length;
-          }
-
-          await localConnection.commit();
-          
-          cumulativeProcessedOrders += processedOrders.size;
-          outputProgress({
-            status: "running",
-            operation: "Orders import",
-            message: `Importing orders: ${cumulativeProcessedOrders} of ${totalUniqueOrders}`,
-            current: cumulativeProcessedOrders,
-            total: totalUniqueOrders,
-            elapsed: formatElapsedTime(startTime),
-            remaining: estimateRemaining(startTime, cumulativeProcessedOrders, totalUniqueOrders),
-            rate: calculateRate(startTime, cumulativeProcessedOrders)
-          });
-        } catch (error) {
-          await localConnection.rollback();
-          throw error;
-        }
-      }
-    }
-
-    // Start a transaction for updating sync status and dropping temp tables
-    await localConnection.beginTransaction();
-    try {
-      // Update sync status
-      await localConnection.query(`
-        INSERT INTO sync_status (table_name, last_sync_timestamp)
-        VALUES ('orders', NOW())
-        ON CONFLICT (table_name) DO UPDATE SET
-          last_sync_timestamp = NOW()
-      `);
-      
-      // Cleanup temporary tables
-      await localConnection.query(`
-        DROP TABLE IF EXISTS temp_order_items;
-        DROP TABLE IF EXISTS temp_order_meta;
-        DROP TABLE IF EXISTS temp_order_discounts;
-        DROP TABLE IF EXISTS temp_order_taxes;
-        DROP TABLE IF EXISTS temp_order_costs;
-        DROP TABLE IF EXISTS temp_main_discounts;
-        DROP TABLE IF EXISTS temp_item_discounts;
-      `);
-      
-      // Commit final transaction
-      await localConnection.commit();
-    } catch (error) {
-      await localConnection.rollback();
-      throw error; 
-    }
-
-    return {
-      status: "complete",
-      totalImported: Math.floor(importedCount) || 0,
-      recordsAdded: parseInt(recordsAdded) || 0,
-      recordsUpdated: parseInt(recordsUpdated) || 0,
-      totalSkipped: skippedOrders.size || 0,
-      missingProducts: missingProducts.size || 0,
-      totalProcessed: orderItems.length,  // Total order items in source
-      incrementalUpdate,
-      lastSyncTime,
-      details: {
-        uniqueOrdersProcessed: cumulativeProcessedOrders,
-        totalOrderItems: orderItems.length,
-        skippedDueToMissingProducts: skippedOrders.size,
-        missingProductIds: Array.from(missingProducts).slice(0, 100) // First 100 for debugging
-      }
-    };
-  } catch (error) {
-    console.error("Error during orders import:", error);
-    throw error;
-  }
-}
-
-module.exports = importOrders;

inventory-server/scripts/import/products.js

@@ -1,950 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../metrics-new/utils/progress');
-const BATCH_SIZE = 1000; // Smaller batch size for better progress tracking
-const MAX_RETRIES = 3;
-const RETRY_DELAY = 5000; // 5 seconds
-const dotenv = require("dotenv");
-const path = require("path");
-dotenv.config({ path: path.join(__dirname, "../../.env") });
-
-// Utility functions
-const imageUrlBase = process.env.PRODUCT_IMAGE_URL_BASE || 'https://sbing.com/i/products/0000/';
-const getImageUrls = (pid, iid = 1) => {
-  const paddedPid = pid.toString().padStart(6, '0');
-  // Use padded PID only for the first 3 digits
-  const prefix = paddedPid.slice(0, 3);
-  // Use the actual pid for the rest of the URL
-  const basePath = `${imageUrlBase}${prefix}/${pid}`;
-  return {
-    image: `${basePath}-t-${iid}.jpg`,
-    image_175: `${basePath}-175x175-${iid}.jpg`,
-    image_full: `${basePath}-o-${iid}.jpg`
-  };
-};
-
-// Add helper function for retrying operations with exponential backoff
-async function withRetry(operation, errorMessage) {
-  let lastError;
-  for (let attempt = 1; attempt <= MAX_RETRIES; attempt++) {
-    try {
-      return await operation();
-    } catch (error) {
-      lastError = error;
-      console.error(`${errorMessage} (Attempt ${attempt}/${MAX_RETRIES}):`, error);
-      if (attempt < MAX_RETRIES) {
-        const backoffTime = RETRY_DELAY * Math.pow(2, attempt - 1);
-        await new Promise(resolve => setTimeout(resolve, backoffTime));
-      }
-    }
-  }
-  throw lastError;
-}
-
-// Add helper function at the top of the file
-function validateDate(mysqlDate) {
-  if (!mysqlDate || mysqlDate === '0000-00-00' || mysqlDate === '0000-00-00 00:00:00') {
-    return null;
-  }
-  // Check if the date is valid
-  const date = new Date(mysqlDate);
-  return isNaN(date.getTime()) ? null : mysqlDate;
-}
-
-async function setupTemporaryTables(connection) {
-  // Drop the table if it exists
-  await connection.query('DROP TABLE IF EXISTS temp_products');
-
-  // Create the temporary table
-  await connection.query(`
-    CREATE TEMP TABLE temp_products (
-      pid BIGINT NOT NULL,
-      title TEXT,
-      description TEXT,
-      sku TEXT,
-      stock_quantity INTEGER DEFAULT 0,
-      preorder_count INTEGER DEFAULT 0,
-      notions_inv_count INTEGER DEFAULT 0,
-      price NUMERIC(14, 4) NOT NULL DEFAULT 0,
-      regular_price NUMERIC(14, 4) NOT NULL DEFAULT 0,
-      cost_price NUMERIC(14, 4),
-      vendor TEXT,
-      vendor_reference TEXT,
-      notions_reference TEXT,
-      brand TEXT,
-      line TEXT,
-      subline TEXT,
-      artist TEXT,
-      categories TEXT,
-      created_at TIMESTAMP WITH TIME ZONE,
-      first_received TIMESTAMP WITH TIME ZONE,
-      landing_cost_price NUMERIC(14, 4),
-      barcode TEXT,
-      harmonized_tariff_code TEXT,
-      updated_at TIMESTAMP WITH TIME ZONE,
-      visible BOOLEAN,
-      managing_stock BOOLEAN DEFAULT true,
-      replenishable BOOLEAN,
-      permalink TEXT,
-      moq INTEGER DEFAULT 1,
-      uom INTEGER DEFAULT 1,
-      rating NUMERIC(14, 4),
-      reviews INTEGER,
-      weight NUMERIC(14, 4),
-      length NUMERIC(14, 4),
-      width NUMERIC(14, 4),
-      height NUMERIC(14, 4),
-      country_of_origin TEXT,
-      location TEXT,
-      total_sold INTEGER,
-      baskets INTEGER,
-      notifies INTEGER,
-      date_last_sold TIMESTAMP WITH TIME ZONE,
-      primary_iid INTEGER,
-      image TEXT,
-      image_175 TEXT,
-      image_full TEXT,
-      options TEXT,
-      tags TEXT,
-      needs_update BOOLEAN DEFAULT TRUE,
-      PRIMARY KEY (pid)
-    )`);
-
-  // Create the index
-  await connection.query('CREATE INDEX idx_temp_products_needs_update ON temp_products (needs_update)');
-}
-
-async function cleanupTemporaryTables(connection) {
-  await connection.query('DROP TABLE IF EXISTS temp_products');
-}
-
-async function importMissingProducts(prodConnection, localConnection, missingPids) {
-  if (!missingPids || missingPids.length === 0) {
-    return {
-      status: "complete",
-      recordsAdded: 0,
-      message: "No missing products to import"
-    };
-  }
-
-  try {
-    // Setup temporary tables
-    await setupTemporaryTables(localConnection);
-
-    // Get product data from production - Keep MySQL compatible
-    const [prodData] = await prodConnection.query(`
-      SELECT 
-        p.pid,
-        p.description AS title,
-        p.notes AS description,
-        p.itemnumber AS sku,
-        p.date_created,
-        p.datein AS first_received,
-        p.location,
-        p.upc AS barcode,
-        p.harmonized_tariff_code,
-        p.stamp AS updated_at,
-        CASE WHEN si.show + si.buyable > 0 THEN 1 ELSE 0 END AS visible,
-        CASE 
-          WHEN p.reorder < 0 THEN 0
-          WHEN p.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR) THEN 1
-          WHEN COALESCE(pnb.inventory, 0) > 0 THEN 1
-          WHEN (
-            (COALESCE(pls.date_sold, '0000-00-00') = '0000-00-00' OR pls.date_sold <= DATE_SUB(CURRENT_DATE, INTERVAL 5 YEAR))
-            AND (p.datein = '0000-00-00 00:00:00' OR p.datein <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
-            AND (p.date_refill = '0000-00-00 00:00:00' OR p.date_refill <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
-          ) THEN 0
-          ELSE 1
-        END AS replenishable,
-        COALESCE(si.available_local, 0) as stock_quantity,
-        0 as pending_qty,
-        COALESCE(ci.onpreorder, 0) as preorder_count,
-        COALESCE(pnb.inventory, 0) as notions_inv_count,
-        COALESCE(pcp.price_each, 0) as price,
-        COALESCE(p.sellingprice, 0) AS regular_price,
-        CASE 
-          WHEN EXISTS (SELECT 1 FROM product_inventory WHERE pid = p.pid AND count > 0) 
-          THEN (
-            SELECT ROUND(SUM(costeach * count) / SUM(count), 5) 
-            FROM product_inventory 
-            WHERE pid = p.pid AND count > 0
-          )
-          ELSE (SELECT costeach FROM product_inventory WHERE pid = p.pid ORDER BY daterec DESC LIMIT 1)
-        END AS cost_price,
-        NULL as landing_cost_price,
-        s.companyname AS vendor,
-        CASE 
-          WHEN s.companyname = 'Notions' THEN sid.notions_itemnumber 
-          ELSE sid.supplier_itemnumber 
-        END AS vendor_reference,
-        sid.notions_itemnumber AS notions_reference,
-        CONCAT('https://www.acherryontop.com/shop/product/', p.pid) AS permalink,
-        pc1.name AS brand,
-        pc2.name AS line,
-        pc3.name AS subline,
-        pc4.name AS artist,
-        COALESCE(CASE 
-          WHEN sid.supplier_id = 92 THEN sid.notions_qty_per_unit
-          ELSE sid.supplier_qty_per_unit
-        END, sid.notions_qty_per_unit) AS moq,
-        p.rating,
-        p.rating_votes AS reviews,
-        p.weight,
-        p.length,
-        p.width,
-        p.height,
-        p.country_of_origin,
-        (SELECT COUNT(*) FROM mybasket mb WHERE mb.item = p.pid AND mb.qty > 0) AS baskets,
-        (SELECT COUNT(*) FROM product_notify pn WHERE pn.pid = p.pid) AS notifies,
-        (SELECT COALESCE(SUM(oi.qty_ordered), 0) 
-         FROM order_items oi 
-         JOIN _order o ON oi.order_id = o.order_id 
-         WHERE oi.prod_pid = p.pid AND o.order_status >= 20) AS total_sold,
-        pls.date_sold as date_last_sold,
-        (SELECT iid FROM product_images WHERE pid = p.pid AND \`order\` = 255 LIMIT 1) AS primary_iid,
-        GROUP_CONCAT(DISTINCT CASE 
-          WHEN pc.cat_id IS NOT NULL 
-          AND pc.type IN (10, 20, 11, 21, 12, 13)
-          AND pci.cat_id NOT IN (16, 17)
-          THEN pci.cat_id 
-        END) as category_ids
-      FROM products p
-      LEFT JOIN shop_inventory si ON p.pid = si.pid AND si.store = 0
-      LEFT JOIN current_inventory ci ON p.pid = ci.pid
-      LEFT JOIN product_notions_b2b pnb ON p.pid = pnb.pid
-      LEFT JOIN product_current_prices pcp ON p.pid = pcp.pid AND pcp.active = 1
-      LEFT JOIN supplier_item_data sid ON p.pid = sid.pid
-      LEFT JOIN suppliers s ON sid.supplier_id = s.supplierid
-      LEFT JOIN product_category_index pci ON p.pid = pci.pid
-      LEFT JOIN product_categories pc ON pci.cat_id = pc.cat_id
-      LEFT JOIN product_categories pc1 ON p.company = pc1.cat_id
-      LEFT JOIN product_categories pc2 ON p.line = pc2.cat_id
-      LEFT JOIN product_categories pc3 ON p.subline = pc3.cat_id
-      LEFT JOIN product_categories pc4 ON p.artist = pc4.cat_id
-      LEFT JOIN product_last_sold pls ON p.pid = pls.pid
-      WHERE p.pid IN (?)
-      GROUP BY p.pid
-    `, [missingPids]);
-
-    if (!prodData || prodData.length === 0) {
-      return {
-        status: "complete",
-        recordsAdded: 0,
-        message: "No products found in production database"
-      };
-    }
-
-    // Process in batches
-    let recordsAdded = 0;
-    for (let i = 0; i < prodData.length; i += BATCH_SIZE) {
-      const batch = prodData.slice(i, i + BATCH_SIZE);
-      
-      const placeholders = batch.map((_, idx) => {
-        const base = idx * 48; // 48 columns
-        return `(${Array.from({ length: 48 }, (_, i) => `$${base + i + 1}`).join(', ')})`;
-      }).join(',');
-
-      const values = batch.flatMap(row => {
-        const imageUrls = getImageUrls(row.pid, row.primary_iid || 1);
-        return [
-          row.pid,
-          row.title,
-          row.description,
-          row.sku || '',
-          row.stock_quantity > 5000 ? 0 : Math.max(0, row.stock_quantity),
-          row.preorder_count,
-          row.notions_inv_count,
-          row.price,
-          row.regular_price,
-          row.cost_price,
-          row.vendor,
-          row.vendor_reference,
-          row.notions_reference,
-          row.brand,
-          row.line,
-          row.subline,
-          row.artist,
-          row.category_ids,
-          validateDate(row.date_created),
-          validateDate(row.first_received),
-          row.landing_cost_price,
-          row.barcode,
-          row.harmonized_tariff_code,
-          validateDate(row.updated_at),
-          row.visible,
-          true,
-          row.replenishable,
-          row.permalink,
-          Math.max(1, Math.round(row.moq || 1)),
-          1,
-          row.rating,
-          row.reviews,
-          row.weight,
-          row.length,
-          row.width,
-          row.height,
-          row.country_of_origin,
-          row.location,
-          row.total_sold,
-          row.baskets,
-          row.notifies,
-          validateDate(row.date_last_sold),
-          row.primary_iid,
-          imageUrls.image,
-          imageUrls.image_175,
-          imageUrls.image_full,
-          null,
-          null
-        ];
-      });
-
-      const [result] = await localConnection.query(`
-        WITH inserted_products AS (
-          INSERT INTO products (
-            pid, title, description, sku, stock_quantity, preorder_count, notions_inv_count,
-            price, regular_price, cost_price, vendor, vendor_reference, notions_reference,
-            brand, line, subline, artist, categories, created_at, first_received,
-            landing_cost_price, barcode, harmonized_tariff_code, updated_at, visible,
-            managing_stock, replenishable, permalink, moq, uom, rating, reviews,
-            weight, length, width, height, country_of_origin, location, total_sold,
-            baskets, notifies, date_last_sold, primary_iid, image, image_175, image_full, options, tags
-          )
-          VALUES ${placeholders}
-          ON CONFLICT (pid) DO NOTHING
-          RETURNING pid
-        )
-        SELECT COUNT(*) as inserted FROM inserted_products
-      `, values);
-
-      recordsAdded += parseInt(result.rows[0].inserted, 10) || 0;
-    }
-
-    return {
-      status: "complete",
-      recordsAdded,
-      message: `Successfully imported ${recordsAdded} missing products`
-    };
-  } catch (error) {
-    console.error('Error importing missing products:', error);
-    throw error;
-  }
-}
-
-async function materializeCalculations(prodConnection, localConnection, incrementalUpdate = true, lastSyncTime = '1970-01-01', startTime = Date.now()) {
-  outputProgress({
-    status: "running",
-    operation: "Products import",
-    message: "Fetching product data from production"
-  });
-
-  // Get all product data in a single optimized query - Keep MySQL compatible
-  const [prodData] = await prodConnection.query(`
-    SELECT 
-      p.pid,
-      p.description AS title,
-      p.notes AS description,
-      p.itemnumber AS sku,
-      p.date_created,
-      p.datein AS first_received,
-      p.location,
-      p.upc AS barcode,
-      p.harmonized_tariff_code,
-      p.stamp AS updated_at,
-      CASE WHEN si.show + si.buyable > 0 THEN 1 ELSE 0 END AS visible,
-      CASE 
-        WHEN p.reorder < 0 THEN 0
-        WHEN p.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR) THEN 1
-        WHEN COALESCE(pnb.inventory, 0) > 0 THEN 1
-        WHEN (
-          (COALESCE(pls.date_sold, '0000-00-00') = '0000-00-00' OR pls.date_sold <= DATE_SUB(CURRENT_DATE, INTERVAL 5 YEAR))
-          AND (p.datein = '0000-00-00 00:00:00' OR p.datein <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
-          AND (p.date_refill = '0000-00-00 00:00:00' OR p.date_refill <= DATE_SUB(CURRENT_TIMESTAMP, INTERVAL 5 YEAR))
-        ) THEN 0
-        ELSE 1
-      END AS replenishable,
-      COALESCE(si.available_local, 0) as stock_quantity,
-      0 as pending_qty,
-      COALESCE(ci.onpreorder, 0) as preorder_count,
-      COALESCE(pnb.inventory, 0) as notions_inv_count,
-      COALESCE(pcp.price_each, 0) as price,
-      COALESCE(p.sellingprice, 0) AS regular_price,
-      CASE 
-        WHEN EXISTS (SELECT 1 FROM product_inventory WHERE pid = p.pid AND count > 0) 
-        THEN (
-          SELECT ROUND(SUM(costeach * count) / SUM(count), 5) 
-          FROM product_inventory 
-          WHERE pid = p.pid AND count > 0
-        )
-        ELSE (SELECT costeach FROM product_inventory WHERE pid = p.pid ORDER BY daterec DESC LIMIT 1)
-      END AS cost_price,
-      NULL as landing_cost_price,
-      s.companyname AS vendor,
-      CASE 
-        WHEN s.companyname = 'Notions' THEN sid.notions_itemnumber 
-        ELSE sid.supplier_itemnumber 
-      END AS vendor_reference,
-      sid.notions_itemnumber AS notions_reference,
-      CONCAT('https://www.acherryontop.com/shop/product/', p.pid) AS permalink,
-      pc1.name AS brand,
-      pc2.name AS line,
-      pc3.name AS subline,
-      pc4.name AS artist,
-      COALESCE(CASE 
-        WHEN sid.supplier_id = 92 THEN sid.notions_qty_per_unit
-        ELSE sid.supplier_qty_per_unit
-      END, sid.notions_qty_per_unit) AS moq,
-      p.rating,
-      p.rating_votes AS reviews,
-      p.weight,
-      p.length,
-      p.width,
-      p.height,
-      p.country_of_origin,
-      (SELECT COUNT(*) FROM mybasket mb WHERE mb.item = p.pid AND mb.qty > 0) AS baskets,
-      (SELECT COUNT(*) FROM product_notify pn WHERE pn.pid = p.pid) AS notifies,
-      (SELECT COALESCE(SUM(oi.qty_ordered), 0) 
-       FROM order_items oi 
-       JOIN _order o ON oi.order_id = o.order_id 
-       WHERE oi.prod_pid = p.pid AND o.order_status >= 20) AS total_sold,
-      pls.date_sold as date_last_sold,
-      (SELECT iid FROM product_images WHERE pid = p.pid AND \`order\` = 255 LIMIT 1) AS primary_iid,
-      GROUP_CONCAT(DISTINCT CASE 
-        WHEN pc.cat_id IS NOT NULL 
-        AND pc.type IN (10, 20, 11, 21, 12, 13)
-        AND pci.cat_id NOT IN (16, 17)
-        THEN pci.cat_id 
-      END) as category_ids
-    FROM products p
-    LEFT JOIN shop_inventory si ON p.pid = si.pid AND si.store = 0
-    LEFT JOIN current_inventory ci ON p.pid = ci.pid
-    LEFT JOIN product_notions_b2b pnb ON p.pid = pnb.pid
-    LEFT JOIN product_current_prices pcp ON p.pid = pcp.pid AND pcp.active = 1
-    LEFT JOIN supplier_item_data sid ON p.pid = sid.pid
-    LEFT JOIN suppliers s ON sid.supplier_id = s.supplierid
-    LEFT JOIN product_category_index pci ON p.pid = pci.pid
-    LEFT JOIN product_categories pc ON pci.cat_id = pc.cat_id
-    LEFT JOIN product_categories pc1 ON p.company = pc1.cat_id
-    LEFT JOIN product_categories pc2 ON p.line = pc2.cat_id
-    LEFT JOIN product_categories pc3 ON p.subline = pc3.cat_id
-    LEFT JOIN product_categories pc4 ON p.artist = pc4.cat_id
-    LEFT JOIN product_last_sold pls ON p.pid = pls.pid
-    WHERE ${incrementalUpdate ? `
-      p.stamp > ? OR
-      ci.stamp > ? OR
-      pcp.date_deactive > ? OR
-      pcp.date_active > ? OR
-      pnb.date_updated > ?
-      -- Add condition for product_images changes if needed for incremental updates
-      -- OR EXISTS (SELECT 1 FROM product_images pi WHERE pi.pid = p.pid AND pi.stamp > ?)
-    ` : 'TRUE'}
-    GROUP BY p.pid
-  `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime /*, lastSyncTime */] : []);
-
-  outputProgress({
-    status: "running",
-    operation: "Products import",
-    message: `Processing ${prodData.length} product records`
-  });
-
-  // Insert all product data into temp table in batches
-  for (let i = 0; i < prodData.length; i += BATCH_SIZE) {
-    const batch = prodData.slice(i, Math.min(i + BATCH_SIZE, prodData.length));
-    
-    await withRetry(async () => {
-      const placeholders = batch.map((_, idx) => {
-        const base = idx * 48; // 48 columns
-        return `(${Array.from({ length: 48 }, (_, i) => `$${base + i + 1}`).join(', ')})`;
-      }).join(',');
-
-      const values = batch.flatMap(row => {
-        const imageUrls = getImageUrls(row.pid, row.primary_iid || 1);
-        return [
-          row.pid,
-          row.title,
-          row.description,
-          row.sku || '',
-          row.stock_quantity > 5000 ? 0 : Math.max(0, row.stock_quantity),
-          row.preorder_count,
-          row.notions_inv_count,
-          row.price,
-          row.regular_price,
-          row.cost_price,
-          row.vendor,
-          row.vendor_reference,
-          row.notions_reference,
-          row.brand,
-          row.line,
-          row.subline,
-          row.artist,
-          row.category_ids,
-          validateDate(row.date_created),
-          validateDate(row.first_received),
-          row.landing_cost_price,
-          row.barcode,
-          row.harmonized_tariff_code,
-          validateDate(row.updated_at),
-          row.visible,
-          true,
-          row.replenishable,
-          row.permalink,
-          Math.max(1, Math.round(row.moq || 1)),
-          1,
-          row.rating,
-          row.reviews,
-          row.weight,
-          row.length,
-          row.width,
-          row.height,
-          row.country_of_origin,
-          row.location,
-          row.total_sold,
-          row.baskets,
-          row.notifies,
-          validateDate(row.date_last_sold),
-          row.primary_iid,
-          imageUrls.image,
-          imageUrls.image_175,
-          imageUrls.image_full,
-          null,
-          null
-        ];
-      });
-
-      await localConnection.query(`
-        INSERT INTO temp_products (
-          pid, title, description, sku, stock_quantity, preorder_count, notions_inv_count,
-          price, regular_price, cost_price, vendor, vendor_reference, notions_reference,
-          brand, line, subline, artist, categories, created_at, first_received,
-          landing_cost_price, barcode, harmonized_tariff_code, updated_at, visible,
-          managing_stock, replenishable, permalink, moq, uom, rating, reviews,
-          weight, length, width, height, country_of_origin, location, total_sold,
-          baskets, notifies, date_last_sold, primary_iid, image, image_175, image_full, options, tags
-        ) VALUES ${placeholders}
-        ON CONFLICT (pid) DO UPDATE SET
-          title = EXCLUDED.title,
-          description = EXCLUDED.description,
-          sku = EXCLUDED.sku,
-          stock_quantity = EXCLUDED.stock_quantity,
-          preorder_count = EXCLUDED.preorder_count,
-          notions_inv_count = EXCLUDED.notions_inv_count,
-          price = EXCLUDED.price,
-          regular_price = EXCLUDED.regular_price,
-          cost_price = EXCLUDED.cost_price,
-          vendor = EXCLUDED.vendor,
-          vendor_reference = EXCLUDED.vendor_reference,
-          notions_reference = EXCLUDED.notions_reference,
-          brand = EXCLUDED.brand,
-          line = EXCLUDED.line,
-          subline = EXCLUDED.subline,
-          artist = EXCLUDED.artist,
-          created_at = EXCLUDED.created_at,
-          first_received = EXCLUDED.first_received,
-          landing_cost_price = EXCLUDED.landing_cost_price,
-          barcode = EXCLUDED.barcode,
-          harmonized_tariff_code = EXCLUDED.harmonized_tariff_code,
-          updated_at = EXCLUDED.updated_at,
-          visible = EXCLUDED.visible,
-          managing_stock = EXCLUDED.managing_stock,
-          replenishable = EXCLUDED.replenishable,
-          permalink = EXCLUDED.permalink,
-          moq = EXCLUDED.moq,
-          uom = EXCLUDED.uom,
-          rating = EXCLUDED.rating,
-          reviews = EXCLUDED.reviews,
-          weight = EXCLUDED.weight,
-          length = EXCLUDED.length,
-          width = EXCLUDED.width,
-          height = EXCLUDED.height,
-          country_of_origin = EXCLUDED.country_of_origin,
-          location = EXCLUDED.location,
-          total_sold = EXCLUDED.total_sold,
-          baskets = EXCLUDED.baskets,
-          notifies = EXCLUDED.notifies,
-          date_last_sold = EXCLUDED.date_last_sold,
-          primary_iid = EXCLUDED.primary_iid,
-          image = EXCLUDED.image,
-          image_175 = EXCLUDED.image_175,
-          image_full = EXCLUDED.image_full,
-          options = EXCLUDED.options,
-          tags = EXCLUDED.tags
-        RETURNING 
-          xmax = 0 as inserted
-      `, values);
-    }, `Error inserting batch ${i} to ${i + batch.length}`);
-
-    outputProgress({
-      status: "running",
-      operation: "Products import",
-      message: `Imported ${i + batch.length} of ${prodData.length} products`,
-      current: i + batch.length,
-      total: prodData.length,
-              elapsed: formatElapsedTime(startTime),
-        remaining: estimateRemaining(startTime, i + batch.length, prodData.length),
-      rate: calculateRate(startTime, i + batch.length)
-    });
-  }
-
-  outputProgress({
-    status: "running",
-    operation: "Products import",
-    message: "Finished materializing calculations"
-  });
-  
-  // Add step to identify which products actually need updating
-  outputProgress({
-    status: "running",
-    operation: "Products import",
-    message: "Identifying changed products"
-  });
-  
-  // Mark products that haven't changed as needs_update = false
-  await localConnection.query(`
-    UPDATE temp_products t
-    SET needs_update = FALSE
-    FROM products p
-    WHERE t.pid = p.pid
-      AND t.title IS NOT DISTINCT FROM p.title
-      AND t.description IS NOT DISTINCT FROM p.description
-      AND t.sku IS NOT DISTINCT FROM p.sku
-      AND t.stock_quantity = p.stock_quantity
-      AND t.price = p.price
-      AND t.regular_price = p.regular_price
-      AND t.cost_price IS NOT DISTINCT FROM p.cost_price
-      AND t.vendor IS NOT DISTINCT FROM p.vendor
-      AND t.brand IS NOT DISTINCT FROM p.brand
-      AND t.visible = p.visible
-      AND t.replenishable = p.replenishable
-      AND t.barcode IS NOT DISTINCT FROM p.barcode
-      AND t.updated_at IS NOT DISTINCT FROM p.updated_at
-      AND t.total_sold IS NOT DISTINCT FROM p.total_sold
-      -- Check key fields that are likely to change
-      -- We don't need to check every single field, just the important ones
-  `);
-  
-  // Get count of products that need updating
-  const [countResult] = await localConnection.query(`
-    SELECT 
-      COUNT(*) FILTER (WHERE needs_update = true) as update_count,
-      COUNT(*) FILTER (WHERE needs_update = false) as skip_count,
-      COUNT(*) as total_count
-    FROM temp_products
-  `);
-  
-  outputProgress({
-    status: "running",
-    operation: "Products import",
-    message: `Found ${countResult.rows[0].update_count} products that need updating, ${countResult.rows[0].skip_count} unchanged`
-  });
-  
-  // Return the total products processed
-  return {
-    totalProcessed: prodData.length,
-    needsUpdate: parseInt(countResult.rows[0].update_count),
-    skipped: parseInt(countResult.rows[0].skip_count)
-  };
-}
-
-async function importProducts(prodConnection, localConnection, incrementalUpdate = true) {
-  const startTime = Date.now();
-  let lastSyncTime = '1970-01-01';
-
-  try {
-    // Get last sync time if doing incremental update
-    if (incrementalUpdate) {
-      const [syncResult] = await localConnection.query(
-        "SELECT last_sync_timestamp FROM sync_status WHERE table_name = 'products'"
-      );
-      if (syncResult.rows.length > 0) {
-        lastSyncTime = syncResult.rows[0].last_sync_timestamp;
-      }
-    }
-
-    // Start a transaction to ensure temporary tables persist
-    await localConnection.beginTransaction();
-
-    try {
-      // Setup temporary tables
-      await setupTemporaryTables(localConnection);
-
-      // Materialize calculations into temp table
-      const materializeResult = await materializeCalculations(prodConnection, localConnection, incrementalUpdate, lastSyncTime, startTime);
-
-      // Get the list of products that need updating
-      const [products] = await localConnection.query(`
-        SELECT 
-          t.pid,
-          t.title,
-          t.description,
-          t.sku,
-          t.stock_quantity,
-          t.preorder_count,
-          t.notions_inv_count,
-          t.price,
-          t.regular_price,
-          t.cost_price,
-          t.vendor,
-          t.vendor_reference,
-          t.notions_reference,
-          t.brand,
-          t.line,
-          t.subline,
-          t.artist,
-          t.categories,
-          t.created_at,
-          t.first_received,
-          t.landing_cost_price,
-          t.barcode,
-          t.harmonized_tariff_code,
-          t.updated_at,
-          t.visible,
-          t.managing_stock,
-          t.replenishable,
-          t.permalink,
-          t.moq,
-          t.rating,
-          t.reviews,
-          t.weight,
-          t.length,
-          t.width,
-          t.height,
-          t.country_of_origin,
-          t.location,
-          t.total_sold,
-          t.baskets,
-          t.notifies,
-          t.date_last_sold,
-          t.primary_iid,
-          t.image,
-          t.image_175,
-          t.image_full,
-          t.options,
-          t.tags
-        FROM temp_products t
-        WHERE t.needs_update = true
-      `);
-
-      // Process products in batches
-      let recordsAdded = 0;
-      let recordsUpdated = 0;
-
-      for (let i = 0; i < products.rows.length; i += BATCH_SIZE) {
-        const batch = products.rows.slice(i, i + BATCH_SIZE);
-        
-        const placeholders = batch.map((_, idx) => {
-          const base = idx * 47; // 47 columns
-          return `(${Array.from({ length: 47 }, (_, i) => `$${base + i + 1}`).join(', ')})`;
-        }).join(',');
-
-        const values = batch.flatMap(row => {
-          const imageUrls = getImageUrls(row.pid, row.primary_iid || 1);
-          return [
-            row.pid,
-            row.title,
-            row.description,
-            row.sku || '',
-            row.stock_quantity > 5000 ? 0 : Math.max(0, row.stock_quantity),
-            row.preorder_count,
-            row.notions_inv_count,
-            row.price,
-            row.regular_price,
-            row.cost_price,
-            row.vendor,
-            row.vendor_reference,
-            row.notions_reference,
-            row.brand,
-            row.line,
-            row.subline,
-            row.artist,
-            row.categories,
-            validateDate(row.created_at),
-            validateDate(row.first_received),
-            row.landing_cost_price,
-            row.barcode,
-            row.harmonized_tariff_code,
-            validateDate(row.updated_at),
-            row.visible,
-            row.managing_stock,
-            row.replenishable,
-            row.permalink,
-            row.moq,
-            1,
-            row.rating,
-            row.reviews,
-            row.weight,
-            row.length,
-            row.width,
-            row.height,
-            row.country_of_origin,
-            row.location,
-            row.total_sold,
-            row.baskets,
-            row.notifies,
-            validateDate(row.date_last_sold),
-            imageUrls.image,
-            imageUrls.image_175,
-            imageUrls.image_full,
-            row.options,
-            row.tags
-          ];
-        });
-
-        const [result] = await localConnection.query(`
-          WITH upserted AS (
-            INSERT INTO products (
-              pid, title, description, sku, stock_quantity, preorder_count, notions_inv_count,
-              price, regular_price, cost_price, vendor, vendor_reference, notions_reference,
-              brand, line, subline, artist, categories, created_at, first_received,
-              landing_cost_price, barcode, harmonized_tariff_code, updated_at, visible,
-              managing_stock, replenishable, permalink, moq, uom, rating, reviews,
-              weight, length, width, height, country_of_origin, location, total_sold,
-              baskets, notifies, date_last_sold, image, image_175, image_full, options, tags
-            )
-            VALUES ${placeholders}
-            ON CONFLICT (pid) DO UPDATE SET
-              title = EXCLUDED.title,
-              description = EXCLUDED.description,
-              sku = EXCLUDED.sku,
-              stock_quantity = EXCLUDED.stock_quantity,
-              preorder_count = EXCLUDED.preorder_count,
-              notions_inv_count = EXCLUDED.notions_inv_count,
-              price = EXCLUDED.price,
-              regular_price = EXCLUDED.regular_price,
-              cost_price = EXCLUDED.cost_price,
-              vendor = EXCLUDED.vendor,
-              vendor_reference = EXCLUDED.vendor_reference,
-              notions_reference = EXCLUDED.notions_reference,
-              brand = EXCLUDED.brand,
-              line = EXCLUDED.line,
-              subline = EXCLUDED.subline,
-              artist = EXCLUDED.artist,
-              created_at = EXCLUDED.created_at,
-              first_received = EXCLUDED.first_received,
-              landing_cost_price = EXCLUDED.landing_cost_price,
-              barcode = EXCLUDED.barcode,
-              harmonized_tariff_code = EXCLUDED.harmonized_tariff_code,
-              updated_at = EXCLUDED.updated_at,
-              visible = EXCLUDED.visible,
-              managing_stock = EXCLUDED.managing_stock,
-              replenishable = EXCLUDED.replenishable,
-              permalink = EXCLUDED.permalink,
-              moq = EXCLUDED.moq,
-              uom = EXCLUDED.uom,
-              rating = EXCLUDED.rating,
-              reviews = EXCLUDED.reviews,
-              weight = EXCLUDED.weight,
-              length = EXCLUDED.length,
-              width = EXCLUDED.width,
-              height = EXCLUDED.height,
-              country_of_origin = EXCLUDED.country_of_origin,
-              location = EXCLUDED.location,
-              total_sold = EXCLUDED.total_sold,
-              baskets = EXCLUDED.baskets,
-              notifies = EXCLUDED.notifies,
-              date_last_sold = EXCLUDED.date_last_sold,
-              image = EXCLUDED.image,
-              image_175 = EXCLUDED.image_175,
-              image_full = EXCLUDED.image_full,
-              options = EXCLUDED.options,
-              tags = EXCLUDED.tags
-            RETURNING 
-              xmax = 0 as inserted
-          )
-          SELECT 
-            COUNT(*) FILTER (WHERE inserted) as inserted,
-            COUNT(*) FILTER (WHERE NOT inserted) as updated
-          FROM upserted
-        `, values);
-
-        recordsAdded += parseInt(result.rows[0].inserted, 10) || 0;
-        recordsUpdated += parseInt(result.rows[0].updated, 10) || 0;
-
-        // Process category relationships in batches
-        const allCategories = [];
-        for (const row of batch) {
-          if (row.categories) {
-            const categoryIds = row.categories.split(',').filter(id => id && id.trim());
-            if (categoryIds.length > 0) {
-              categoryIds.forEach(catId => {
-                allCategories.push([row.pid, parseInt(catId.trim(), 10)]);
-              });
-            }
-          }
-        }
-
-        // If we have categories to process
-        if (allCategories.length > 0) {
-          // First get all products in this batch
-          const productIds = batch.map(p => p.pid);
-          
-          // Delete all existing relationships for products in this batch
-          await localConnection.query(
-            'DELETE FROM product_categories WHERE pid = ANY($1)',
-            [productIds]
-          );
-          
-          // Insert all new relationships in one batch
-          const catPlaceholders = allCategories.map((_, idx) => 
-            `($${idx * 2 + 1}, $${idx * 2 + 2})`
-          ).join(',');
-          
-          const catValues = allCategories.flat();
-          
-          await localConnection.query(`
-            INSERT INTO product_categories (pid, cat_id)
-            VALUES ${catPlaceholders}
-            ON CONFLICT (pid, cat_id) DO NOTHING
-          `, catValues);
-        }
-
-        outputProgress({
-          status: "running",
-          operation: "Products import",
-          message: `Processing products: ${i + batch.length} of ${products.rows.length}`,
-          current: i + batch.length,
-          total: products.rows.length,
-                  elapsed: formatElapsedTime(startTime),
-        remaining: estimateRemaining(startTime, i + batch.length, products.rows.length),
-          rate: calculateRate(startTime, i + batch.length)
-        });
-      }
-
-      // Cleanup temporary tables
-      await cleanupTemporaryTables(localConnection);
-
-      // Commit the transaction
-      await localConnection.commit();
-
-      // Update sync status
-      await localConnection.query(`
-        INSERT INTO sync_status (table_name, last_sync_timestamp)
-        VALUES ('products', NOW())
-        ON CONFLICT (table_name) DO UPDATE SET
-          last_sync_timestamp = NOW()
-      `);
-
-      return {
-        status: 'complete',
-        recordsAdded,
-        recordsUpdated,
-        totalRecords: products.rows.length,
-        totalProcessed: materializeResult.totalProcessed,
-        duration: formatElapsedTime(startTime),
-        needsUpdate: materializeResult.needsUpdate,
-        skippedUnchanged: materializeResult.skipped
-      };
-    } catch (error) {
-      // Rollback on error
-      await localConnection.rollback();
-      throw error;
-    }
-  } catch (error) {
-    console.error('Error in importProducts:', error);
-    throw error;
-  }
-}
-
-module.exports = {
-  importProducts,
-  importMissingProducts,
-  setupTemporaryTables,
-  cleanupTemporaryTables,
-  materializeCalculations
-};

inventory-server/scripts/import/purchase-orders.js

@@ -1,884 +0,0 @@
-const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } = require('../metrics-new/utils/progress');
-
-/**
- * Validates a date from MySQL before inserting it into PostgreSQL
- * @param {string|Date|null} mysqlDate - Date string or object from MySQL
- * @returns {string|null} Valid date string or null if invalid
- */
-function validateDate(mysqlDate) {
-  // Handle null, undefined, or empty values
-  if (!mysqlDate) {
-    return null;
-  }
-  
-  // Convert to string if it's not already
-  const dateStr = String(mysqlDate);
-  
-  // Handle MySQL zero dates and empty values
-  if (dateStr === '0000-00-00' || 
-      dateStr === '0000-00-00 00:00:00' ||
-      dateStr.indexOf('0000-00-00') !== -1 ||
-      dateStr === '') {
-    return null;
-  }
-  
-  // Check if the date is valid
-  const date = new Date(mysqlDate);
-  
-  // If the date is invalid or suspiciously old (pre-1970), return null
-  if (isNaN(date.getTime()) || date.getFullYear() < 1970) {
-    return null;
-  }
-  
-  return mysqlDate;
-}
-
-/**
- * Imports purchase orders and receivings from a production MySQL database to a local PostgreSQL database.
- * Handles these as separate data streams without complex FIFO allocation.
- * 
- * @param {object} prodConnection - A MySQL connection to production DB
- * @param {object} localConnection - A PostgreSQL connection to local DB
- * @param {boolean} incrementalUpdate - Set to false for a full sync; true for incremental
- * @returns {object} Information about the sync operation
- */
-async function importPurchaseOrders(prodConnection, localConnection, incrementalUpdate = true) {
-  const startTime = Date.now();
-  let poRecordsAdded = 0;
-  let poRecordsUpdated = 0;
-  let poRecordsDeleted = 0;
-  let receivingRecordsAdded = 0;
-  let receivingRecordsUpdated = 0;
-  let receivingRecordsDeleted = 0;
-  let totalProcessed = 0;
-  
-  // Batch size constants
-  const PO_BATCH_SIZE = 500;
-  const INSERT_BATCH_SIZE = 100;
-
-  try {
-    // Begin transaction for the entire import process
-    await localConnection.beginTransaction();
-
-    // Get last sync info
-    const [syncInfo] = await localConnection.query(
-      "SELECT last_sync_timestamp FROM sync_status WHERE table_name = 'purchase_orders'"
-    );
-    const lastSyncTime = syncInfo?.rows?.[0]?.last_sync_timestamp || '1970-01-01';
-
-    console.log('Purchase Orders: Using last sync time:', lastSyncTime);
-
-    // Create temp tables for processing
-    await localConnection.query(`
-      DROP TABLE IF EXISTS temp_purchase_orders;
-      DROP TABLE IF EXISTS temp_receivings;
-      DROP TABLE IF EXISTS employee_names;
-      DROP TABLE IF EXISTS temp_supplier_names;
-      
-      -- Temporary table for purchase orders
-      CREATE TEMP TABLE temp_purchase_orders (
-        po_id TEXT NOT NULL,
-        pid BIGINT NOT NULL,
-        sku TEXT,
-        name TEXT,
-        vendor TEXT,
-        date TIMESTAMP WITH TIME ZONE,
-        expected_date DATE,
-        status TEXT,
-        notes TEXT,
-        long_note TEXT,
-        ordered INTEGER,
-        po_cost_price NUMERIC(14, 4),
-        supplier_id INTEGER,
-        date_created TIMESTAMP WITH TIME ZONE,
-        date_ordered TIMESTAMP WITH TIME ZONE,
-        PRIMARY KEY (po_id, pid)
-      );
-      
-      -- Temporary table for receivings
-      CREATE TEMP TABLE temp_receivings (
-        receiving_id TEXT NOT NULL,
-        pid BIGINT NOT NULL,
-        sku TEXT,
-        name TEXT,
-        vendor TEXT,
-        qty_each INTEGER,
-        qty_each_orig INTEGER,
-        cost_each NUMERIC(14, 5),
-        cost_each_orig NUMERIC(14, 5),
-        received_by INTEGER,
-        received_by_name TEXT,
-        received_date TIMESTAMP WITH TIME ZONE,
-        receiving_created_date TIMESTAMP WITH TIME ZONE,
-        supplier_id INTEGER,
-        status TEXT,
-        PRIMARY KEY (receiving_id, pid)
-      );
-      
-      -- Temporary table for employee names
-      CREATE TEMP TABLE employee_names (
-        employeeid INTEGER PRIMARY KEY,
-        firstname TEXT,
-        lastname TEXT
-      );
-      
-      -- Create indexes for efficient joins
-      CREATE INDEX idx_temp_po_pid ON temp_purchase_orders(pid);
-      CREATE INDEX idx_temp_receiving_pid ON temp_receivings(pid);
-    `);
-
-    // Map status codes to text values
-    const poStatusMap = {
-      0: 'canceled',
-      1: 'created',
-      10: 'electronically_ready_send',
-      11: 'ordered',
-      12: 'preordered',
-      13: 'electronically_sent',
-      15: 'receiving_started',
-      50: 'done'
-    };
-    
-    const receivingStatusMap = {
-      0: 'canceled',
-      1: 'created',
-      30: 'partial_received',
-      40: 'full_received',
-      50: 'paid'
-    };
-
-    // Get time window for data retrieval
-    const yearInterval = incrementalUpdate ? 1 : 5;
-
-    // Fetch employee data from production
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Fetching employee data"
-    });
-
-    const [employees] = await prodConnection.query(`
-      SELECT 
-        employeeid,
-        firstname,
-        lastname
-      FROM employees
-    `);
-
-    // Insert employee data into temp table
-    if (employees.length > 0) {
-      const employeeValues = employees.map(emp => [
-        emp.employeeid,
-        emp.firstname || '',
-        emp.lastname || ''
-      ]).flat();
-
-      const placeholders = employees.map((_, idx) => {
-        const base = idx * 3;
-        return `($${base + 1}, $${base + 2}, $${base + 3})`;
-      }).join(',');
-
-      await localConnection.query(`
-        INSERT INTO employee_names (employeeid, firstname, lastname)
-        VALUES ${placeholders}
-        ON CONFLICT (employeeid) DO UPDATE SET
-          firstname = EXCLUDED.firstname,
-          lastname = EXCLUDED.lastname
-      `, employeeValues);
-    }
-
-    // Add this section before the PO import to create a supplier names mapping
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Fetching supplier data for vendor mapping"
-    });
-
-    // Fetch supplier data from production and store in a temp table
-    const [suppliers] = await prodConnection.query(`
-      SELECT 
-        supplierid,
-        companyname
-      FROM suppliers
-      WHERE companyname IS NOT NULL AND companyname != ''
-    `);
-
-    if (suppliers.length > 0) {
-      // Create temp table for supplier names
-      await localConnection.query(`
-        DROP TABLE IF EXISTS temp_supplier_names;
-        CREATE TEMP TABLE temp_supplier_names (
-          supplier_id INTEGER PRIMARY KEY,
-          company_name TEXT NOT NULL
-        );
-      `);
-
-      // Insert supplier data in batches
-      for (let i = 0; i < suppliers.length; i += INSERT_BATCH_SIZE) {
-        const batch = suppliers.slice(i, i + INSERT_BATCH_SIZE);
-        
-        const placeholders = batch.map((_, idx) => {
-          const base = idx * 2;
-          return `($${base + 1}, $${base + 2})`;
-        }).join(',');
-        
-        const values = batch.flatMap(s => [
-          s.supplierid,
-          s.companyname || 'Unnamed Supplier'
-        ]);
-        
-        await localConnection.query(`
-          INSERT INTO temp_supplier_names (supplier_id, company_name)
-          VALUES ${placeholders}
-          ON CONFLICT (supplier_id) DO UPDATE SET
-            company_name = EXCLUDED.company_name
-        `, values);
-      }
-    }
-
-    // 1. Fetch and process purchase orders
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Fetching purchase orders"
-    });
-
-    const [poCount] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM po p
-      WHERE p.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL ${yearInterval} YEAR)
-        ${incrementalUpdate ? `
-          AND (
-            p.date_updated > ? 
-            OR p.date_ordered > ? 
-            OR p.date_estin > ?
-          )
-        ` : ''}
-    `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime] : []);
-
-    const totalPOs = poCount[0].total;
-    console.log(`Found ${totalPOs} relevant purchase orders`);
-
-    // Skip processing if no POs to process
-    if (totalPOs === 0) {
-      console.log('No purchase orders to process, skipping PO import step');
-    } else {
-      // Fetch and process POs in batches
-      let offset = 0;
-      let allPOsProcessed = false;
-      
-      while (!allPOsProcessed) {
-        const [poList] = await prodConnection.query(`
-          SELECT 
-            p.po_id,
-            p.supplier_id,
-            s.companyname AS vendor,
-            p.status,
-            p.notes AS long_note,
-            p.short_note AS notes,
-            p.date_created,
-            p.date_ordered,
-            p.date_estin
-          FROM po p
-          LEFT JOIN suppliers s ON p.supplier_id = s.supplierid
-          WHERE p.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL ${yearInterval} YEAR)
-            ${incrementalUpdate ? `
-              AND (
-                p.date_updated > ? 
-                OR p.date_ordered > ? 
-                OR p.date_estin > ?
-              )
-            ` : ''}
-          ORDER BY p.po_id
-          LIMIT ${PO_BATCH_SIZE} OFFSET ${offset}
-        `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime] : []);
-        
-        if (poList.length === 0) {
-          allPOsProcessed = true;
-          break;
-        }
-        
-        // Get products for these POs
-        const poIds = poList.map(po => po.po_id);
-        
-        const [poProducts] = await prodConnection.query(`
-          SELECT 
-            pp.po_id,
-            pp.pid,
-            pp.qty_each,
-            pp.cost_each,
-            COALESCE(p.itemnumber, 'NO-SKU') AS sku,
-            COALESCE(p.description, 'Unknown Product') AS name
-          FROM po_products pp
-          LEFT JOIN products p ON pp.pid = p.pid
-          WHERE pp.po_id IN (?)
-        `, [poIds]);
-        
-        // Build complete PO records
-        const completePOs = [];
-        for (const product of poProducts) {
-          const po = poList.find(p => p.po_id == product.po_id);
-          if (!po) continue;
-          
-          completePOs.push({
-            po_id: po.po_id.toString(),
-            pid: product.pid,
-            sku: product.sku,
-            name: product.name,
-            vendor: po.vendor || 'Unknown Vendor',
-            date: validateDate(po.date_ordered) || validateDate(po.date_created),
-            expected_date: validateDate(po.date_estin),
-            status: poStatusMap[po.status] || 'created',
-            notes: po.notes || '',
-            long_note: po.long_note || '',
-            ordered: product.qty_each,
-            po_cost_price: product.cost_each,
-            supplier_id: po.supplier_id,
-            date_created: validateDate(po.date_created),
-            date_ordered: validateDate(po.date_ordered)
-          });
-        }
-        
-        // Insert PO data in batches
-        for (let i = 0; i < completePOs.length; i += INSERT_BATCH_SIZE) {
-          const batch = completePOs.slice(i, i + INSERT_BATCH_SIZE);
-          
-          const placeholders = batch.map((_, idx) => {
-            const base = idx * 15;
-            return `($${base + 1}, $${base + 2}, $${base + 3}, $${base + 4}, $${base + 5}, $${base + 6}, $${base + 7}, $${base + 8}, $${base + 9}, $${base + 10}, $${base + 11}, $${base + 12}, $${base + 13}, $${base + 14}, $${base + 15})`;
-          }).join(',');
-          
-          const values = batch.flatMap(po => [
-            po.po_id,
-            po.pid,
-            po.sku,
-            po.name,
-            po.vendor,
-            po.date,
-            po.expected_date,
-            po.status,
-            po.notes,
-            po.long_note,
-            po.ordered,
-            po.po_cost_price,
-            po.supplier_id,
-            po.date_created,
-            po.date_ordered
-          ]);
-          
-          await localConnection.query(`
-            INSERT INTO temp_purchase_orders (
-              po_id, pid, sku, name, vendor, date, expected_date, status, notes, long_note,
-              ordered, po_cost_price, supplier_id, date_created, date_ordered
-            )
-            VALUES ${placeholders}
-            ON CONFLICT (po_id, pid) DO UPDATE SET
-              sku = EXCLUDED.sku,
-              name = EXCLUDED.name,
-              vendor = EXCLUDED.vendor,
-              date = EXCLUDED.date,
-              expected_date = EXCLUDED.expected_date,
-              status = EXCLUDED.status,
-              notes = EXCLUDED.notes,
-              long_note = EXCLUDED.long_note,
-              ordered = EXCLUDED.ordered,
-              po_cost_price = EXCLUDED.po_cost_price,
-              supplier_id = EXCLUDED.supplier_id,
-              date_created = EXCLUDED.date_created,
-              date_ordered = EXCLUDED.date_ordered
-          `, values);
-        }
-        
-        offset += poList.length;
-        totalProcessed += completePOs.length;
-        
-        outputProgress({
-          status: "running",
-          operation: "Purchase orders import",
-          message: `Processed ${offset} of ${totalPOs} purchase orders (${totalProcessed} line items)`,
-          current: offset,
-          total: totalPOs,
-          elapsed: formatElapsedTime(startTime),
-          remaining: estimateRemaining(startTime, offset, totalPOs),
-          rate: calculateRate(startTime, offset)
-        });
-        
-        if (poList.length < PO_BATCH_SIZE) {
-          allPOsProcessed = true;
-        }
-      }
-    }
-    
-    // 2. Next, fetch all relevant receivings
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Fetching receivings data"
-    });
-
-    const [receivingCount] = await prodConnection.query(`
-      SELECT COUNT(*) as total
-      FROM receivings r
-      WHERE r.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL ${yearInterval} YEAR)
-        ${incrementalUpdate ? `
-          AND (
-            r.date_updated > ?
-            OR r.date_created > ?
-          )
-        ` : ''}
-    `, incrementalUpdate ? [lastSyncTime, lastSyncTime] : []);
-
-    const totalReceivings = receivingCount[0].total;
-    console.log(`Found ${totalReceivings} relevant receivings`);
-
-    // Skip processing if no receivings to process
-    if (totalReceivings === 0) {
-      console.log('No receivings to process, skipping receivings import step');
-    } else {
-      // Fetch and process receivings in batches
-      offset = 0; // Reset offset for receivings
-      let allReceivingsProcessed = false;
-      
-      while (!allReceivingsProcessed) {
-        const [receivingList] = await prodConnection.query(`
-          SELECT 
-            r.receiving_id,
-            r.supplier_id,
-            r.status,
-            r.notes,
-            r.shipping,
-            r.total_amount,
-            r.hold,
-            r.for_storefront,
-            r.date_created,
-            r.date_paid,
-            r.date_checked
-          FROM receivings r
-          WHERE r.date_created >= DATE_SUB(CURRENT_DATE, INTERVAL ${yearInterval} YEAR)
-            ${incrementalUpdate ? `
-              AND (
-                r.date_updated > ?
-                OR r.date_created > ?
-              )
-            ` : ''}
-          ORDER BY r.receiving_id
-          LIMIT ${PO_BATCH_SIZE} OFFSET ${offset}
-        `, incrementalUpdate ? [lastSyncTime, lastSyncTime] : []);
-        
-        if (receivingList.length === 0) {
-          allReceivingsProcessed = true;
-          break;
-        }
-        
-        // Get products for these receivings
-        const receivingIds = receivingList.map(r => r.receiving_id);
-        
-        const [receivingProducts] = await prodConnection.query(`
-          SELECT 
-            rp.receiving_id,
-            rp.pid,
-            rp.qty_each,
-            rp.qty_each_orig,
-            rp.cost_each,
-            rp.cost_each_orig,
-            rp.received_by,
-            rp.received_date,
-            r.date_created as receiving_created_date,
-            COALESCE(p.itemnumber, 'NO-SKU') AS sku,
-            COALESCE(p.description, 'Unknown Product') AS name
-          FROM receivings_products rp
-          JOIN receivings r ON rp.receiving_id = r.receiving_id
-          LEFT JOIN products p ON rp.pid = p.pid
-          WHERE rp.receiving_id IN (?)
-        `, [receivingIds]);
-        
-        // Build complete receiving records
-        const completeReceivings = [];
-        for (const product of receivingProducts) {
-          const receiving = receivingList.find(r => r.receiving_id == product.receiving_id);
-          if (!receiving) continue;
-          
-          // Get employee name if available
-          let receivedByName = null;
-          if (product.received_by) {
-            const [employeeResult] = await localConnection.query(`
-              SELECT CONCAT(firstname, ' ', lastname) as full_name 
-              FROM employee_names 
-              WHERE employeeid = $1
-            `, [product.received_by]);
-            
-            if (employeeResult.rows.length > 0) {
-              receivedByName = employeeResult.rows[0].full_name;
-            }
-          }
-          
-          // Get vendor name if available
-          let vendorName = 'Unknown Vendor';
-          if (receiving.supplier_id) {
-            const [vendorResult] = await localConnection.query(`
-              SELECT company_name 
-              FROM temp_supplier_names 
-              WHERE supplier_id = $1
-            `, [receiving.supplier_id]);
-            
-            if (vendorResult.rows.length > 0) {
-              vendorName = vendorResult.rows[0].company_name;
-            }
-          }
-          
-          completeReceivings.push({
-            receiving_id: receiving.receiving_id.toString(),
-            pid: product.pid,
-            sku: product.sku,
-            name: product.name,
-            vendor: vendorName,
-            qty_each: product.qty_each,
-            qty_each_orig: product.qty_each_orig,
-            cost_each: product.cost_each,
-            cost_each_orig: product.cost_each_orig,
-            received_by: product.received_by,
-            received_by_name: receivedByName,
-            received_date: validateDate(product.received_date) || validateDate(product.receiving_created_date),
-            receiving_created_date: validateDate(product.receiving_created_date),
-            supplier_id: receiving.supplier_id,
-            status: receivingStatusMap[receiving.status] || 'created'
-          });
-        }
-        
-        // Insert receiving data in batches
-        for (let i = 0; i < completeReceivings.length; i += INSERT_BATCH_SIZE) {
-          const batch = completeReceivings.slice(i, i + INSERT_BATCH_SIZE);
-          
-          const placeholders = batch.map((_, idx) => {
-            const base = idx * 15;
-            return `($${base + 1}, $${base + 2}, $${base + 3}, $${base + 4}, $${base + 5}, $${base + 6}, $${base + 7}, $${base + 8}, $${base + 9}, $${base + 10}, $${base + 11}, $${base + 12}, $${base + 13}, $${base + 14}, $${base + 15})`;
-          }).join(',');
-          
-          const values = batch.flatMap(r => [
-            r.receiving_id,
-            r.pid,
-            r.sku,
-            r.name,
-            r.vendor,
-            r.qty_each,
-            r.qty_each_orig,
-            r.cost_each,
-            r.cost_each_orig,
-            r.received_by,
-            r.received_by_name,
-            r.received_date,
-            r.receiving_created_date,
-            r.supplier_id,
-            r.status
-          ]);
-          
-          await localConnection.query(`
-            INSERT INTO temp_receivings (
-              receiving_id, pid, sku, name, vendor, qty_each, qty_each_orig, 
-              cost_each, cost_each_orig, received_by, received_by_name, 
-              received_date, receiving_created_date, supplier_id, status
-            )
-            VALUES ${placeholders}
-            ON CONFLICT (receiving_id, pid) DO UPDATE SET
-              sku = EXCLUDED.sku,
-              name = EXCLUDED.name,
-              vendor = EXCLUDED.vendor,
-              qty_each = EXCLUDED.qty_each,
-              qty_each_orig = EXCLUDED.qty_each_orig,
-              cost_each = EXCLUDED.cost_each,
-              cost_each_orig = EXCLUDED.cost_each_orig,
-              received_by = EXCLUDED.received_by,
-              received_by_name = EXCLUDED.received_by_name,
-              received_date = EXCLUDED.received_date,
-              receiving_created_date = EXCLUDED.receiving_created_date,
-              supplier_id = EXCLUDED.supplier_id,
-              status = EXCLUDED.status
-          `, values);
-        }
-        
-        offset += receivingList.length;
-        totalProcessed += completeReceivings.length;
-        
-        outputProgress({
-          status: "running",
-          operation: "Purchase orders import",
-          message: `Processed ${offset} of ${totalReceivings} receivings (${totalProcessed} line items total)`,
-          current: offset,
-          total: totalReceivings,
-          elapsed: formatElapsedTime(startTime),
-          remaining: estimateRemaining(startTime, offset, totalReceivings),
-          rate: calculateRate(startTime, offset)
-        });
-        
-        if (receivingList.length < PO_BATCH_SIZE) {
-          allReceivingsProcessed = true;
-        }
-      }
-    }
-
-    // Add this section to filter out invalid PIDs before final import
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Validating product IDs before final import"
-    });
-
-    await localConnection.query(`
-      -- Create temp table to store invalid PIDs
-      DROP TABLE IF EXISTS temp_invalid_pids;
-      CREATE TEMP TABLE temp_invalid_pids AS (
-        -- Get all unique PIDs from our temp tables
-        WITH all_pids AS (
-          SELECT DISTINCT pid FROM temp_purchase_orders
-          UNION
-          SELECT DISTINCT pid FROM temp_receivings
-        )
-        -- Filter to only those that don't exist in products table
-        SELECT p.pid
-        FROM all_pids p
-        WHERE NOT EXISTS (
-          SELECT 1 FROM products WHERE pid = p.pid
-        )
-      );
-      
-      -- Remove purchase orders with invalid PIDs
-      DELETE FROM temp_purchase_orders
-      WHERE pid IN (SELECT pid FROM temp_invalid_pids);
-      
-      -- Remove receivings with invalid PIDs
-      DELETE FROM temp_receivings
-      WHERE pid IN (SELECT pid FROM temp_invalid_pids);
-    `);
-
-    // Get count of filtered items for reporting
-    const [filteredResult] = await localConnection.query(`
-      SELECT COUNT(*) as count FROM temp_invalid_pids
-    `);
-    const filteredCount = filteredResult.rows[0].count;
-
-    if (filteredCount > 0) {
-      console.log(`Filtered out ${filteredCount} items with invalid product IDs`);
-    }
-
-    // 3. Insert final purchase order records to the actual table
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Inserting final purchase order records"
-    });
-
-    // Create a temp table to track PO IDs being processed
-    await localConnection.query(`
-      DROP TABLE IF EXISTS processed_po_ids;
-      CREATE TEMP TABLE processed_po_ids AS (
-        SELECT DISTINCT po_id FROM temp_purchase_orders
-      );
-    `);
-      
-    // Delete products that were removed from POs and count them
-    const [poDeletedResult] = await localConnection.query(`
-      WITH deleted AS (
-        DELETE FROM purchase_orders
-        WHERE po_id IN (SELECT po_id FROM processed_po_ids)
-          AND NOT EXISTS (
-            SELECT 1 FROM temp_purchase_orders tp
-            WHERE purchase_orders.po_id = tp.po_id AND purchase_orders.pid = tp.pid
-          )
-        RETURNING po_id, pid
-      )
-      SELECT COUNT(*) as count FROM deleted
-    `);
-    
-    poRecordsDeleted = poDeletedResult.rows[0]?.count || 0;
-    console.log(`Deleted ${poRecordsDeleted} products that were removed from purchase orders`);
-
-    const [poResult] = await localConnection.query(`
-      INSERT INTO purchase_orders (
-        po_id, vendor, date, expected_date, pid, sku, name,
-        po_cost_price, status, notes, long_note,
-        ordered, supplier_id, date_created, date_ordered
-      )
-      SELECT 
-        po_id,
-        vendor,
-        COALESCE(date, date_created, now()) as date,
-        expected_date,
-        pid,
-        sku,
-        name,
-        po_cost_price,
-        status,
-        notes,
-        long_note,
-        ordered,
-        supplier_id,
-        date_created,
-        date_ordered
-      FROM temp_purchase_orders
-      ON CONFLICT (po_id, pid) DO UPDATE SET
-        vendor = EXCLUDED.vendor,
-        date = EXCLUDED.date,
-        expected_date = EXCLUDED.expected_date,
-        sku = EXCLUDED.sku,
-        name = EXCLUDED.name,
-        po_cost_price = EXCLUDED.po_cost_price,
-        status = EXCLUDED.status,
-        notes = EXCLUDED.notes,
-        long_note = EXCLUDED.long_note,
-        ordered = EXCLUDED.ordered,
-        supplier_id = EXCLUDED.supplier_id,
-        date_created = EXCLUDED.date_created,
-        date_ordered = EXCLUDED.date_ordered,
-        updated = CURRENT_TIMESTAMP
-      WHERE -- Only update if at least one key field has changed
-        purchase_orders.ordered IS DISTINCT FROM EXCLUDED.ordered OR
-        purchase_orders.po_cost_price IS DISTINCT FROM EXCLUDED.po_cost_price OR
-        purchase_orders.status IS DISTINCT FROM EXCLUDED.status OR
-        purchase_orders.expected_date IS DISTINCT FROM EXCLUDED.expected_date OR
-        purchase_orders.date IS DISTINCT FROM EXCLUDED.date OR
-        purchase_orders.vendor IS DISTINCT FROM EXCLUDED.vendor
-      RETURNING (xmax = 0) as inserted
-    `);
-
-    poRecordsAdded = poResult.rows.filter(r => r.inserted).length;
-    poRecordsUpdated = poResult.rows.filter(r => !r.inserted).length;
-
-    // 4. Insert final receiving records to the actual table
-    outputProgress({
-      status: "running",
-      operation: "Purchase orders import",
-      message: "Inserting final receiving records"
-    });
-
-    // Create a temp table to track receiving IDs being processed
-    await localConnection.query(`
-      DROP TABLE IF EXISTS processed_receiving_ids;
-      CREATE TEMP TABLE processed_receiving_ids AS (
-        SELECT DISTINCT receiving_id FROM temp_receivings
-      );
-    `);
-      
-    // Delete products that were removed from receivings and count them
-    const [receivingDeletedResult] = await localConnection.query(`
-      WITH deleted AS (
-        DELETE FROM receivings
-        WHERE receiving_id IN (SELECT receiving_id FROM processed_receiving_ids)
-          AND NOT EXISTS (
-            SELECT 1 FROM temp_receivings tr
-            WHERE receivings.receiving_id = tr.receiving_id AND receivings.pid = tr.pid
-          )
-        RETURNING receiving_id, pid
-      )
-      SELECT COUNT(*) as count FROM deleted
-    `);
-    
-    receivingRecordsDeleted = receivingDeletedResult.rows[0]?.count || 0;
-    console.log(`Deleted ${receivingRecordsDeleted} products that were removed from receivings`);
-
-    const [receivingsResult] = await localConnection.query(`
-      INSERT INTO receivings (
-        receiving_id, pid, sku, name, vendor, qty_each, qty_each_orig,
-        cost_each, cost_each_orig, received_by, received_by_name,
-        received_date, receiving_created_date, supplier_id, status
-      )
-      SELECT 
-        receiving_id,
-        pid,
-        sku,
-        name,
-        vendor,
-        qty_each,
-        qty_each_orig,
-        cost_each,
-        cost_each_orig,
-        received_by,
-        received_by_name,
-        COALESCE(received_date, receiving_created_date, now()) as received_date,
-        receiving_created_date,
-        supplier_id,
-        status
-      FROM temp_receivings
-      ON CONFLICT (receiving_id, pid) DO UPDATE SET
-        sku = EXCLUDED.sku,
-        name = EXCLUDED.name,
-        vendor = EXCLUDED.vendor,
-        qty_each = EXCLUDED.qty_each,
-        qty_each_orig = EXCLUDED.qty_each_orig,
-        cost_each = EXCLUDED.cost_each,
-        cost_each_orig = EXCLUDED.cost_each_orig,
-        received_by = EXCLUDED.received_by,
-        received_by_name = EXCLUDED.received_by_name,
-        received_date = EXCLUDED.received_date,
-        receiving_created_date = EXCLUDED.receiving_created_date,
-        supplier_id = EXCLUDED.supplier_id,
-        status = EXCLUDED.status,
-        updated = CURRENT_TIMESTAMP
-      WHERE -- Only update if at least one key field has changed
-        receivings.qty_each IS DISTINCT FROM EXCLUDED.qty_each OR
-        receivings.cost_each IS DISTINCT FROM EXCLUDED.cost_each OR
-        receivings.status IS DISTINCT FROM EXCLUDED.status OR
-        receivings.received_date IS DISTINCT FROM EXCLUDED.received_date OR
-        receivings.received_by IS DISTINCT FROM EXCLUDED.received_by
-      RETURNING (xmax = 0) as inserted
-    `);
-
-    receivingRecordsAdded = receivingsResult.rows.filter(r => r.inserted).length;
-    receivingRecordsUpdated = receivingsResult.rows.filter(r => !r.inserted).length;
-
-    // Update sync status
-    await localConnection.query(`
-      INSERT INTO sync_status (table_name, last_sync_timestamp)
-      VALUES ('purchase_orders', NOW())
-      ON CONFLICT (table_name) DO UPDATE SET
-        last_sync_timestamp = NOW()
-    `);
-
-    // Clean up temporary tables
-    await localConnection.query(`
-      DROP TABLE IF EXISTS temp_purchase_orders;
-      DROP TABLE IF EXISTS temp_receivings;
-      DROP TABLE IF EXISTS employee_names;
-      DROP TABLE IF EXISTS temp_supplier_names;
-      DROP TABLE IF EXISTS temp_invalid_pids;
-      DROP TABLE IF EXISTS processed_po_ids;
-      DROP TABLE IF EXISTS processed_receiving_ids;
-    `);
-    
-    // Commit transaction
-    await localConnection.commit();
-
-    return {
-      status: "complete",
-      recordsAdded: poRecordsAdded + receivingRecordsAdded,
-      recordsUpdated: poRecordsUpdated + receivingRecordsUpdated,
-      recordsDeleted: poRecordsDeleted + receivingRecordsDeleted,
-      poRecordsAdded,
-      poRecordsUpdated,
-      poRecordsDeleted,
-      receivingRecordsAdded,
-      receivingRecordsUpdated,
-      receivingRecordsDeleted,
-      totalRecords: totalProcessed
-    };
-  } catch (error) {
-    console.error("Error during purchase orders import:", error);
-    
-    // Rollback transaction
-    try {
-      await localConnection.rollback();
-    } catch (rollbackError) {
-      console.error('Error during rollback:', rollbackError.message);
-    }
-    
-    return {
-      status: "error",
-      error: error.message,
-      recordsAdded: 0,
-      recordsUpdated: 0,
-      recordsDeleted: 0,
-      totalRecords: 0
-    };
-  }
-}
-
-module.exports = importPurchaseOrders;

inventory-server/scripts/import/utils.js

@@ -1,156 +0,0 @@
-const mysql = require("mysql2/promise");
-const { Client } = require("ssh2");
-const { Pool } = require('pg');
-const dotenv = require("dotenv");
-const path = require("path");
-
-// Helper function to setup SSH tunnel
-async function setupSshTunnel(sshConfig) {
-  return new Promise((resolve, reject) => {
-    const ssh = new Client();
-
-    ssh.on('error', (err) => {
-      console.error('SSH connection error:', err);
-    });
-
-    ssh.on('end', () => {
-      console.log('SSH connection ended normally');
-    });
-
-    ssh.on('close', () => {
-      console.log('SSH connection closed');
-    });
-
-    ssh
-      .on("ready", () => {
-        ssh.forwardOut(
-          "127.0.0.1",
-          0,
-          sshConfig.prodDbConfig.host,
-          sshConfig.prodDbConfig.port,
-          async (err, stream) => {
-            if (err) reject(err);
-            resolve({ ssh, stream });
-          }
-        );
-      })
-      .connect(sshConfig.ssh);
-  });
-}
-
-// Helper function to setup database connections
-async function setupConnections(sshConfig) {
-  const tunnel = await setupSshTunnel(sshConfig);
-  
-  // Setup MySQL connection for production
-  const prodConnection = await mysql.createConnection({
-    ...sshConfig.prodDbConfig,
-    stream: tunnel.stream,
-  });
-
-  // Setup PostgreSQL connection pool for local
-  const localPool = new Pool(sshConfig.localDbConfig);
-
-  // Test the PostgreSQL connection
-  try {
-    const client = await localPool.connect();
-    await client.query('SELECT NOW()');
-    client.release();
-    console.log('PostgreSQL connection successful');
-  } catch (err) {
-    console.error('PostgreSQL connection error:', err);
-    throw err;
-  }
-
-  // Create a wrapper for the PostgreSQL pool to match MySQL interface
-  const localConnection = {
-    _client: null,
-    _transactionActive: false,
-
-    query: async (text, params) => {
-      // If we're not in a transaction, use the pool directly
-      if (!localConnection._transactionActive) {
-        const client = await localPool.connect();
-        try {
-          const result = await client.query(text, params);
-          return [result];
-        } finally {
-          client.release();
-        }
-      }
-      
-      // If we're in a transaction, use the dedicated client
-      if (!localConnection._client) {
-        throw new Error('No active transaction client');
-      }
-      const result = await localConnection._client.query(text, params);
-      return [result];
-    },
-
-    beginTransaction: async () => {
-      if (localConnection._transactionActive) {
-        throw new Error('Transaction already active');
-      }
-      localConnection._client = await localPool.connect();
-      await localConnection._client.query('BEGIN');
-      localConnection._transactionActive = true;
-    },
-
-    commit: async () => {
-      if (!localConnection._transactionActive) {
-        throw new Error('No active transaction to commit');
-      }
-      await localConnection._client.query('COMMIT');
-      localConnection._client.release();
-      localConnection._client = null;
-      localConnection._transactionActive = false;
-    },
-
-    rollback: async () => {
-      if (!localConnection._transactionActive) {
-        throw new Error('No active transaction to rollback');
-      }
-      await localConnection._client.query('ROLLBACK');
-      localConnection._client.release();
-      localConnection._client = null;
-      localConnection._transactionActive = false;
-    },
-
-    end: async () => {
-      if (localConnection._client) {
-        localConnection._client.release();
-        localConnection._client = null;
-      }
-      await localPool.end();
-    }
-  };
-
-  return { prodConnection, localConnection, tunnel };
-}
-
-// Helper function to close connections
-async function closeConnections(connections) {
-  const { ssh, prodConnection, localConnection } = connections;
-  
-  try {
-    if (prodConnection) await prodConnection.end();
-    if (localConnection) await localConnection.end();
-    
-    // Wait a bit for any pending data to be written before closing SSH
-    await new Promise(resolve => setTimeout(resolve, 100));
-    
-    if (ssh) {
-      ssh.on('close', () => {
-        console.log('SSH connection closed cleanly');
-      });
-      ssh.end();
-    }
-  } catch (err) {
-    console.error('Error during cleanup:', err);
-  }
-}
-
-module.exports = {
-  setupConnections,
-  closeConnections
-}; 

inventory-server/scripts/metrics-new/backfill/populate_initial_product_metrics.sql

@@ -1,444 +0,0 @@
--- Description: Performs the first population OR full recalculation of the product_metrics table based on
---              historically backfilled daily_product_snapshots and current product/PO data.
---              Calculates all metrics considering the full available history up to 'yesterday'.
---              Run ONCE after backfill_historical_snapshots_final.sql completes successfully.
--- Dependencies: Core import tables (products, purchase_orders, receivings), daily_product_snapshots (historically populated),
---               configuration tables (settings_*), product_metrics table must exist.
--- Frequency: Run ONCE.
-DO $$
-DECLARE
-    _module_name VARCHAR := 'product_metrics_population'; -- Generic name
-    _start_time TIMESTAMPTZ := clock_timestamp();
-    -- Calculate metrics AS OF the end of the last fully completed day
-    _calculation_date DATE := CURRENT_DATE - INTERVAL '1 day';
-BEGIN
-    RAISE NOTICE 'Running % module. Calculating AS OF: %. Start Time: %', _module_name, _calculation_date, _start_time;
-
-    -- Optional: Consider TRUNCATE if you want a completely fresh start,
-    -- otherwise ON CONFLICT will update existing rows if this is rerun.
-    -- TRUNCATE TABLE public.product_metrics;
-    RAISE NOTICE 'Populating product_metrics table. This may take some time...';
-
-    -- CTEs to gather necessary information AS OF _calculation_date
-    WITH CurrentInfo AS (
-        -- Fetches current product details, including costs/prices used for forecasting & fallbacks
-        SELECT
-            p.pid, p.sku, p.title, p.brand, p.vendor, COALESCE(p.image_175, p.image) as image_url,
-            p.visible as is_visible, p.replenishable,
-            COALESCE(p.price, 0.00) as current_price, COALESCE(p.regular_price, 0.00) as current_regular_price,
-            COALESCE(p.cost_price, 0.00) as current_cost_price,
-            COALESCE(p.landing_cost_price, p.cost_price, 0.00) as current_effective_cost, -- Use landing if available, else cost
-            p.stock_quantity as current_stock, -- Use actual current stock for forecast base
-            p.created_at, p.first_received, p.date_last_sold,
-            p.moq,
-            p.uom,
-            p.total_sold as historical_total_sold -- Add historical total_sold from products table
-        FROM public.products p
-    ),
-    OnOrderInfo AS (
-        -- Calculates current on-order quantities and costs
-        SELECT
-            pid,
-            SUM(ordered) AS on_order_qty,
-            SUM(ordered * po_cost_price) AS on_order_cost,
-            MIN(expected_date) AS earliest_expected_date
-        FROM public.purchase_orders
-        -- Use the most common statuses representing active, unfulfilled POs
-        WHERE status IN ('created', 'ordered', 'preordered', 'electronically_sent', 'electronically_ready_send', 'receiving_started')
-          AND status NOT IN ('canceled', 'done')
-        GROUP BY pid
-    ),
-    HistoricalDates AS (
-        -- Determines key historical dates from orders and receivings
-         SELECT
-            p.pid,
-            MIN(o.date)::date AS date_first_sold,
-            MAX(o.date)::date AS max_order_date, -- Used as fallback for date_last_sold
-            MIN(r.received_date)::date AS date_first_received_calc,
-            MAX(r.received_date)::date AS date_last_received_calc
-        FROM public.products p
-        LEFT JOIN public.orders o ON p.pid = o.pid AND o.quantity > 0 AND o.status NOT IN ('canceled', 'returned')
-        LEFT JOIN public.receivings r ON p.pid = r.pid
-        GROUP BY p.pid
-    ),
-    SnapshotAggregates AS (
-        -- Aggregates metrics from historical snapshots up to the _calculation_date
-        SELECT
-            pid,
-            -- Rolling periods relative to _calculation_date
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '6 days' AND _calculation_date THEN units_sold ELSE 0 END) AS sales_7d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '6 days' AND _calculation_date THEN net_revenue ELSE 0 END) AS revenue_7d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '13 days' AND _calculation_date THEN units_sold ELSE 0 END) AS sales_14d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '13 days' AND _calculation_date THEN net_revenue ELSE 0 END) AS revenue_14d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN units_sold ELSE 0 END) AS sales_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN net_revenue ELSE 0 END) AS revenue_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN cogs ELSE 0 END) AS cogs_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN profit ELSE 0 END) AS profit_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN units_returned ELSE 0 END) AS returns_units_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN returns_revenue ELSE 0 END) AS returns_revenue_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN discounts ELSE 0 END) AS discounts_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN gross_revenue ELSE 0 END) AS gross_revenue_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN gross_regular_revenue ELSE 0 END) AS gross_regular_revenue_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date AND stockout_flag THEN 1 ELSE 0 END) AS stockout_days_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '364 days' AND _calculation_date THEN units_sold ELSE 0 END) AS sales_365d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '364 days' AND _calculation_date THEN net_revenue ELSE 0 END) AS revenue_365d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN units_received ELSE 0 END) AS received_qty_30d,
-            SUM(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN cost_received ELSE 0 END) AS received_cost_30d,
-
-            -- Averages over the last 30 days ending _calculation_date
-            AVG(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN eod_stock_quantity END) AS avg_stock_units_30d,
-            AVG(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN eod_stock_cost END) AS avg_stock_cost_30d,
-            AVG(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN eod_stock_retail END) AS avg_stock_retail_30d,
-            AVG(CASE WHEN snapshot_date BETWEEN _calculation_date - INTERVAL '29 days' AND _calculation_date THEN eod_stock_gross END) AS avg_stock_gross_30d,
-
-            -- Lifetime (Using historical total from products table)
-            (SELECT total_sold FROM public.products WHERE public.products.pid = daily_product_snapshots.pid) AS lifetime_sales,
-            COALESCE(
-                -- Option 1: Use 30-day average price if available
-                CASE WHEN SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '29 days' AND snapshot_date <= _calculation_date THEN units_sold ELSE 0 END) > 0 THEN
-                    (SELECT total_sold FROM public.products WHERE public.products.pid = daily_product_snapshots.pid) * (
-                        SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '29 days' AND snapshot_date <= _calculation_date THEN net_revenue ELSE 0 END) / 
-                        NULLIF(SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '29 days' AND snapshot_date <= _calculation_date THEN units_sold ELSE 0 END), 0)
-                    )
-                ELSE NULL END,
-                -- Option 2: Try 365-day average price if available
-                CASE WHEN SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '364 days' AND snapshot_date <= _calculation_date THEN units_sold ELSE 0 END) > 0 THEN
-                    (SELECT total_sold FROM public.products WHERE public.products.pid = daily_product_snapshots.pid) * (
-                        SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '364 days' AND snapshot_date <= _calculation_date THEN net_revenue ELSE 0 END) / 
-                        NULLIF(SUM(CASE WHEN snapshot_date >= _calculation_date - INTERVAL '364 days' AND snapshot_date <= _calculation_date THEN units_sold ELSE 0 END), 0)
-                    )
-                ELSE NULL END,
-                -- Option 3: Use current price from products table
-                (SELECT total_sold * price FROM public.products WHERE public.products.pid = daily_product_snapshots.pid),
-                -- Option 4: Use regular price if current price might be zero
-                (SELECT total_sold * regular_price FROM public.products WHERE public.products.pid = daily_product_snapshots.pid),
-                -- Final fallback: Use accumulated revenue (less accurate for old products)
-                SUM(net_revenue)
-            ) AS lifetime_revenue,
-
-            -- Yesterday (Sales for the specific _calculation_date)
-            SUM(CASE WHEN snapshot_date = _calculation_date THEN units_sold ELSE 0 END) as yesterday_sales
-
-        FROM public.daily_product_snapshots
-        WHERE snapshot_date <= _calculation_date -- Ensure we only use data up to the calculation point
-        GROUP BY pid
-    ),
-    FirstPeriodMetrics AS (
-       -- Calculates sales/revenue for first X days after first sale date
-       -- Uses HistoricalDates CTE to get the first sale date
-        SELECT
-            pid, date_first_sold,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '6 days' THEN units_sold ELSE 0 END) AS first_7_days_sales,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '6 days' THEN net_revenue ELSE 0 END) AS first_7_days_revenue,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '29 days' THEN units_sold ELSE 0 END) AS first_30_days_sales,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '29 days' THEN net_revenue ELSE 0 END) AS first_30_days_revenue,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '59 days' THEN units_sold ELSE 0 END) AS first_60_days_sales,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '59 days' THEN net_revenue ELSE 0 END) AS first_60_days_revenue,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '89 days' THEN units_sold ELSE 0 END) AS first_90_days_sales,
-            SUM(CASE WHEN snapshot_date BETWEEN date_first_sold AND date_first_sold + INTERVAL '89 days' THEN net_revenue ELSE 0 END) AS first_90_days_revenue
-        FROM public.daily_product_snapshots ds
-        JOIN HistoricalDates hd USING(pid)
-        WHERE date_first_sold IS NOT NULL
-          AND snapshot_date >= date_first_sold -- Only consider snapshots after first sale
-          AND snapshot_date <= _calculation_date -- Only up to the overall calculation date
-        GROUP BY pid, date_first_sold
-    ),
-    Settings AS (
-       -- Fetches effective configuration settings (Product > Vendor > Global)
-       SELECT
-        p.pid,
-        COALESCE(sp.lead_time_days, sv.default_lead_time_days, (SELECT setting_value FROM settings_global WHERE setting_key = 'default_lead_time_days')::int, 14) AS effective_lead_time,
-        COALESCE(sp.days_of_stock, sv.default_days_of_stock, (SELECT setting_value FROM settings_global WHERE setting_key = 'default_days_of_stock')::int, 30) AS effective_days_of_stock,
-        COALESCE(sp.safety_stock, (SELECT setting_value::int FROM settings_global WHERE setting_key = 'default_safety_stock_units'), 0) AS effective_safety_stock,
-        COALESCE(sp.exclude_from_forecast, FALSE) AS exclude_forecast
-      FROM public.products p
-      LEFT JOIN public.settings_product sp ON p.pid = sp.pid
-      LEFT JOIN public.settings_vendor sv ON p.vendor = sv.vendor
-    ),
-    AvgLeadTime AS (
-        -- Calculate Average Lead Time by joining purchase_orders with receivings
-        SELECT
-            po.pid,
-            AVG(GREATEST(1, 
-                CASE 
-                    WHEN r.received_date IS NOT NULL AND po.date IS NOT NULL 
-                    THEN (r.received_date::date - po.date::date)
-                    ELSE 1
-                END
-            ))::int AS avg_lead_time_days_calc
-        FROM public.purchase_orders po
-        JOIN public.receivings r ON r.pid = po.pid
-        WHERE po.status = 'done' -- Completed POs
-          AND r.received_date IS NOT NULL
-          AND po.date IS NOT NULL
-          AND r.received_date >= po.date
-        GROUP BY po.pid
-    ),
-    RankedForABC AS (
-        -- Ranks products based on the configured ABC metric (using historical data)
-         SELECT
-            p.pid,
-            CASE COALESCE((SELECT setting_value FROM settings_global WHERE setting_key = 'abc_calculation_basis'), 'revenue_30d')
-                WHEN 'sales_30d' THEN COALESCE(sa.sales_30d, 0)
-                WHEN 'lifetime_revenue' THEN COALESCE(sa.lifetime_revenue, 0)::numeric
-                ELSE COALESCE(sa.revenue_30d, 0) -- Default to revenue_30d
-            END AS metric_value
-        FROM public.products p -- Use products as the base
-        JOIN SnapshotAggregates sa ON p.pid = sa.pid
-        WHERE p.replenishable = TRUE -- Only rank replenishable products
-          AND (CASE COALESCE((SELECT setting_value FROM settings_global WHERE setting_key = 'abc_calculation_basis'), 'revenue_30d')
-                WHEN 'sales_30d' THEN COALESCE(sa.sales_30d, 0)
-                WHEN 'lifetime_revenue' THEN COALESCE(sa.lifetime_revenue, 0)::numeric
-                ELSE COALESCE(sa.revenue_30d, 0)
-              END) > 0 -- Only include products with non-zero contribution
-    ),
-    CumulativeABC AS (
-        -- Calculates cumulative metric values for ABC ranking
-         SELECT
-            pid, metric_value,
-            SUM(metric_value) OVER (ORDER BY metric_value DESC NULLS LAST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as cumulative_metric,
-            SUM(metric_value) OVER () as total_metric
-        FROM RankedForABC
-    ),
-    FinalABC AS (
-        -- Assigns A, B, or C class based on thresholds
-        SELECT
-            pid,
-            CASE
-                WHEN cumulative_metric / NULLIF(total_metric, 0) <= COALESCE((SELECT setting_value::numeric FROM settings_global WHERE setting_key = 'abc_revenue_threshold_a'), 0.8) THEN 'A'::char(1)
-                WHEN cumulative_metric / NULLIF(total_metric, 0) <= COALESCE((SELECT setting_value::numeric FROM settings_global WHERE setting_key = 'abc_revenue_threshold_b'), 0.95) THEN 'B'::char(1)
-                ELSE 'C'::char(1)
-            END AS abc_class_calc
-         FROM CumulativeABC
-    )
-    -- Final INSERT/UPDATE statement using all the prepared CTEs
-    INSERT INTO public.product_metrics (
-        pid, last_calculated, sku, title, brand, vendor, image_url, is_visible, is_replenishable,
-        current_price, current_regular_price, current_cost_price, current_landing_cost_price,
-        current_stock, current_stock_cost, current_stock_retail, current_stock_gross,
-        on_order_qty, on_order_cost, on_order_retail, earliest_expected_date,
-        date_created, date_first_received, date_last_received, date_first_sold, date_last_sold, age_days,
-        sales_7d, revenue_7d, sales_14d, revenue_14d, sales_30d, revenue_30d, cogs_30d, profit_30d,
-        returns_units_30d, returns_revenue_30d, discounts_30d, gross_revenue_30d, gross_regular_revenue_30d,
-        stockout_days_30d, sales_365d, revenue_365d,
-        avg_stock_units_30d, avg_stock_cost_30d, avg_stock_retail_30d, avg_stock_gross_30d,
-        received_qty_30d, received_cost_30d,
-        lifetime_sales, lifetime_revenue,
-        first_7_days_sales, first_7_days_revenue, first_30_days_sales, first_30_days_revenue,
-        first_60_days_sales, first_60_days_revenue, first_90_days_sales, first_90_days_revenue,
-        asp_30d, acp_30d, avg_ros_30d, avg_sales_per_day_30d,
-        margin_30d, markup_30d, gmroi_30d, stockturn_30d, return_rate_30d, discount_rate_30d,
-        stockout_rate_30d, markdown_30d, markdown_rate_30d, sell_through_30d,
-        avg_lead_time_days, abc_class,
-        sales_velocity_daily, config_lead_time, config_days_of_stock, config_safety_stock,
-        planning_period_days, lead_time_forecast_units, days_of_stock_forecast_units,
-        planning_period_forecast_units, lead_time_closing_stock, days_of_stock_closing_stock,
-        replenishment_needed_raw, replenishment_units, replenishment_cost, replenishment_retail, replenishment_profit,
-        to_order_units, forecast_lost_sales_units, forecast_lost_revenue,
-        stock_cover_in_days, po_cover_in_days, sells_out_in_days, replenish_date,
-        overstocked_units, overstocked_cost, overstocked_retail, is_old_stock,
-        yesterday_sales
-    )
-    SELECT
-        -- Select columns in order, joining all CTEs by pid
-        ci.pid, _start_time, ci.sku, ci.title, ci.brand, ci.vendor, ci.image_url, ci.is_visible, ci.replenishable,
-        ci.current_price, ci.current_regular_price, ci.current_cost_price, ci.current_effective_cost,
-        ci.current_stock, (ci.current_stock * COALESCE(ci.current_effective_cost, 0.00))::numeric(12,2), (ci.current_stock * COALESCE(ci.current_price, 0.00))::numeric(12,2), (ci.current_stock * COALESCE(ci.current_regular_price, 0.00))::numeric(12,2),
-        COALESCE(ooi.on_order_qty, 0), COALESCE(ooi.on_order_cost, 0.00)::numeric(12,2), (COALESCE(ooi.on_order_qty, 0) * COALESCE(ci.current_price, 0.00))::numeric(12,2), ooi.earliest_expected_date,
-        
-        -- Fix type issue with date calculation - properly cast timestamps to dates before arithmetic
-        ci.created_at::date, 
-        COALESCE(ci.first_received::date, hd.date_first_received_calc), 
-        hd.date_last_received_calc, 
-        hd.date_first_sold, 
-        COALESCE(ci.date_last_sold, hd.max_order_date), 
-        -- Fix timestamp + integer error by ensuring we work only with dates
-        CASE 
-            WHEN LEAST(ci.created_at::date, COALESCE(hd.date_first_sold, ci.created_at::date)) IS NOT NULL 
-            THEN (_calculation_date::date - LEAST(ci.created_at::date, COALESCE(hd.date_first_sold, ci.created_at::date)))::int
-            ELSE NULL
-        END,
-        
-        COALESCE(sa.sales_7d, 0), COALESCE(sa.revenue_7d, 0), COALESCE(sa.sales_14d, 0), COALESCE(sa.revenue_14d, 0), COALESCE(sa.sales_30d, 0), COALESCE(sa.revenue_30d, 0), COALESCE(sa.cogs_30d, 0), COALESCE(sa.profit_30d, 0),
-        COALESCE(sa.returns_units_30d, 0), COALESCE(sa.returns_revenue_30d, 0), COALESCE(sa.discounts_30d, 0), COALESCE(sa.gross_revenue_30d, 0), COALESCE(sa.gross_regular_revenue_30d, 0),
-        COALESCE(sa.stockout_days_30d, 0), COALESCE(sa.sales_365d, 0), COALESCE(sa.revenue_365d, 0),
-        sa.avg_stock_units_30d, sa.avg_stock_cost_30d, sa.avg_stock_retail_30d, sa.avg_stock_gross_30d, -- Averages can be NULL if no data
-        COALESCE(sa.received_qty_30d, 0), COALESCE(sa.received_cost_30d, 0),
-        COALESCE(sa.lifetime_sales, 0), COALESCE(sa.lifetime_revenue, 0),
-        fpm.first_7_days_sales, fpm.first_7_days_revenue, fpm.first_30_days_sales, fpm.first_30_days_revenue,
-        fpm.first_60_days_sales, fpm.first_60_days_revenue, fpm.first_90_days_sales, fpm.first_90_days_revenue,
-
-        -- Calculated KPIs (using COALESCE on inputs where appropriate)
-        sa.revenue_30d / NULLIF(sa.sales_30d, 0) AS asp_30d,
-        sa.cogs_30d / NULLIF(sa.sales_30d, 0) AS acp_30d,
-        sa.profit_30d / NULLIF(sa.sales_30d, 0) AS avg_ros_30d,
-        COALESCE(sa.sales_30d, 0) / 30.0 AS avg_sales_per_day_30d,
-        
-        -- Fix for percentages - cast to numeric with appropriate precision
-        ((sa.profit_30d / NULLIF(sa.revenue_30d, 0)) * 100)::numeric(8,2) AS margin_30d,
-        ((sa.profit_30d / NULLIF(sa.cogs_30d, 0)) * 100)::numeric(8,2) AS markup_30d,
-        sa.profit_30d / NULLIF(sa.avg_stock_cost_30d, 0) AS gmroi_30d,
-        sa.sales_30d / NULLIF(sa.avg_stock_units_30d, 0) AS stockturn_30d,
-        ((sa.returns_units_30d / NULLIF(COALESCE(sa.sales_30d, 0) + COALESCE(sa.returns_units_30d, 0), 0)) * 100)::numeric(8,2) AS return_rate_30d,
-        ((sa.discounts_30d / NULLIF(sa.gross_revenue_30d, 0)) * 100)::numeric(8,2) AS discount_rate_30d,
-        ((COALESCE(sa.stockout_days_30d, 0) / 30.0) * 100)::numeric(8,2) AS stockout_rate_30d,
-        GREATEST(0, sa.gross_regular_revenue_30d - sa.gross_revenue_30d) AS markdown_30d, -- Ensure markdown isn't negative
-        ((GREATEST(0, sa.gross_regular_revenue_30d - sa.gross_revenue_30d) / NULLIF(sa.gross_regular_revenue_30d, 0)) * 100)::numeric(8,2) AS markdown_rate_30d,
-        -- Sell Through Rate: Sales / (Stock at end of period + Sales). This is one definition proxying for Sales / Beginning Stock.
-        ((sa.sales_30d / NULLIF(
-             (SELECT eod_stock_quantity FROM daily_product_snapshots WHERE snapshot_date = _calculation_date AND pid = ci.pid LIMIT 1) + COALESCE(sa.sales_30d, 0)
-             , 0)) * 100)::numeric(8,2) AS sell_through_30d,
-
-        -- Use calculated periodic metrics
-        alt.avg_lead_time_days_calc,
-        CASE
-            WHEN ci.replenishable = FALSE THEN NULL -- Non-replenishable don't get a class
-            ELSE COALESCE(fa.abc_class_calc, 'C') -- Default ranked replenishable but non-contributing to C
-        END,
-
-        -- Forecasting intermediate values (based on historical aggregates ending _calculation_date)
-        (COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) AS sales_velocity_daily, -- Ensure divisor > 0
-        s.effective_lead_time AS config_lead_time, s.effective_days_of_stock AS config_days_of_stock, s.effective_safety_stock AS config_safety_stock,
-        (s.effective_lead_time + s.effective_days_of_stock) AS planning_period_days,
-        -- Calculate raw forecast need components (using safe velocity)
-        (COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time AS lead_time_forecast_units,
-        (COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock AS days_of_stock_forecast_units,
-        -- Planning period forecast units (sum of lead time and DOS units)
-        CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-          + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock) AS planning_period_forecast_units,
-        -- Closing stock calculations (using raw forecast components for accuracy before rounding)
-        (ci.current_stock + COALESCE(ooi.on_order_qty, 0) - ((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)) AS lead_time_closing_stock,
-        ((ci.current_stock + COALESCE(ooi.on_order_qty, 0) - ((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)))
-          - ((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock) AS days_of_stock_closing_stock,
-        -- Raw replenishment needed
-        (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time) -- Use rounded forecast units
-           + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-           + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0) AS replenishment_needed_raw,
-
-        -- Final Forecasting Metrics
-        -- Replenishment Units (calculated need, before MOQ)
-        CEILING(GREATEST(0,
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-             + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-        ))::int AS replenishment_units,
-        -- Replenishment Cost/Retail/Profit (based on replenishment_units)
-        (CEILING(GREATEST(0, 
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-             + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-        ))::int) * COALESCE(ci.current_effective_cost, 0.00)::numeric(12,2) AS replenishment_cost,
-        (CEILING(GREATEST(0, 
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-             + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-        ))::int) * COALESCE(ci.current_price, 0.00)::numeric(12,2) AS replenishment_retail,
-        (CEILING(GREATEST(0, 
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-             + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-        ))::int) * (COALESCE(ci.current_price, 0.00) - COALESCE(ci.current_effective_cost, 0.00))::numeric(12,2) AS replenishment_profit,
-
-        -- *** FIX: To Order Units (Apply MOQ rounding) ***
-        CASE
-            WHEN COALESCE(ci.moq, 0) <= 1 THEN -- Treat no/invalid MOQ or MOQ=1 as no rounding needed
-                CEILING(GREATEST(0, 
-                   (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-                     + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-                     + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-                ))::int
-            ELSE -- Apply MOQ rounding: Round UP to nearest multiple of MOQ
-                 (CEILING(GREATEST(0, 
-                    (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-                      + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-                      + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0)
-                 ) / NULLIF(ci.moq::numeric, 0)) * COALESCE(ci.moq, 1))::int
-        END AS to_order_units,
-
-        -- Forecast Lost Sales (Units occurring during lead time if current+on_order is insufficient)
-        CEILING(GREATEST(0,
-            ((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time) -- Demand during lead time
-            - (ci.current_stock + COALESCE(ooi.on_order_qty, 0)) -- Supply available before order arrives
-        ))::int AS forecast_lost_sales_units,
-        -- Forecast Lost Revenue
-        (CEILING(GREATEST(0, 
-            ((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-            - (ci.current_stock + COALESCE(ooi.on_order_qty, 0))
-        ))::int) * COALESCE(ci.current_price, 0.00)::numeric(12,2) AS forecast_lost_revenue,
-
-        -- Stock Cover etc (using safe velocity)
-        ci.current_stock / NULLIF((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)), 0) AS stock_cover_in_days,
-        COALESCE(ooi.on_order_qty, 0) / NULLIF((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)), 0) AS po_cover_in_days,
-        (ci.current_stock + COALESCE(ooi.on_order_qty, 0)) / NULLIF((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)), 0) AS sells_out_in_days,
-         -- Replenish Date (Project forward from 'today', which is _calculation_date + 1 day)
-        CASE
-            WHEN (COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) > 0 -- Check for positive velocity
-            THEN
-                _calculation_date + INTERVAL '1 day' -- Today
-                + FLOOR(GREATEST(0, ci.current_stock - s.effective_safety_stock) -- Stock above safety
-                        / (COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) -- divided by velocity
-                  )::integer * INTERVAL '1 day' -- Gives date safety stock is hit
-                - s.effective_lead_time * INTERVAL '1 day' -- Subtract lead time
-            ELSE NULL -- Cannot calculate if no sales velocity
-        END AS replenish_date,
-        -- Overstocked Units (Stock above safety + planning period demand)
-        GREATEST(0, ci.current_stock - s.effective_safety_stock -
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time) -- Demand during lead time
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock)) -- Demand during DOS
-        )::int AS overstocked_units,
-        (GREATEST(0, ci.current_stock - s.effective_safety_stock -
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-        )::int) * COALESCE(ci.current_effective_cost, 0.00)::numeric(12,2) AS overstocked_cost,
-        (GREATEST(0, ci.current_stock - s.effective_safety_stock -
-           (CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_lead_time)
-             + CEILING((COALESCE(sa.sales_30d, 0) / NULLIF(GREATEST(1.0, 30.0 - COALESCE(sa.stockout_days_30d, 0)), 0)) * s.effective_days_of_stock))
-        )::int) * COALESCE(ci.current_price, 0.00)::numeric(12,2) AS overstocked_retail,
-        -- Old Stock Flag
-        (ci.created_at::date < (_calculation_date - INTERVAL '60 day')::date) AND
-        (COALESCE(ci.date_last_sold, hd.max_order_date) IS NULL OR COALESCE(ci.date_last_sold, hd.max_order_date) < (_calculation_date - INTERVAL '60 day')::date) AND
-        (hd.date_last_received_calc IS NULL OR hd.date_last_received_calc < (_calculation_date - INTERVAL '60 day')::date) AND
-        COALESCE(ooi.on_order_qty, 0) = 0 AS is_old_stock,
-        COALESCE(sa.yesterday_sales, 0) -- Sales for _calculation_date
-
-    FROM CurrentInfo ci
-    LEFT JOIN OnOrderInfo ooi ON ci.pid = ooi.pid
-    LEFT JOIN HistoricalDates hd ON ci.pid = hd.pid
-    LEFT JOIN SnapshotAggregates sa ON ci.pid = sa.pid
-    LEFT JOIN FirstPeriodMetrics fpm ON ci.pid = fpm.pid
-    LEFT JOIN Settings s ON ci.pid = s.pid
-    LEFT JOIN AvgLeadTime alt ON ci.pid = alt.pid -- Join calculated avg lead time
-    LEFT JOIN FinalABC fa ON ci.pid = fa.pid -- Join calculated ABC class
-    WHERE s.exclude_forecast IS FALSE OR s.exclude_forecast IS NULL
-
-    ON CONFLICT (pid) DO UPDATE SET
-        -- *** IMPORTANT: List ALL columns here, ensuring order matches INSERT list ***
-        -- Update ALL columns to ensure entire row is refreshed
-        last_calculated = EXCLUDED.last_calculated, sku = EXCLUDED.sku, title = EXCLUDED.title, brand = EXCLUDED.brand, vendor = EXCLUDED.vendor, image_url = EXCLUDED.image_url, is_visible = EXCLUDED.is_visible, is_replenishable = EXCLUDED.is_replenishable,
-        current_price = EXCLUDED.current_price, current_regular_price = EXCLUDED.current_regular_price, current_cost_price = EXCLUDED.current_cost_price, current_landing_cost_price = EXCLUDED.current_landing_cost_price,
-        current_stock = EXCLUDED.current_stock, current_stock_cost = EXCLUDED.current_stock_cost, current_stock_retail = EXCLUDED.current_stock_retail, current_stock_gross = EXCLUDED.current_stock_gross,
-        on_order_qty = EXCLUDED.on_order_qty, on_order_cost = EXCLUDED.on_order_cost, on_order_retail = EXCLUDED.on_order_retail, earliest_expected_date = EXCLUDED.earliest_expected_date,
-        date_created = EXCLUDED.date_created, date_first_received = EXCLUDED.date_first_received, date_last_received = EXCLUDED.date_last_received, date_first_sold = EXCLUDED.date_first_sold, date_last_sold = EXCLUDED.date_last_sold, age_days = EXCLUDED.age_days,
-        sales_7d = EXCLUDED.sales_7d, revenue_7d = EXCLUDED.revenue_7d, sales_14d = EXCLUDED.sales_14d, revenue_14d = EXCLUDED.revenue_14d, sales_30d = EXCLUDED.sales_30d, revenue_30d = EXCLUDED.revenue_30d, cogs_30d = EXCLUDED.cogs_30d, profit_30d = EXCLUDED.profit_30d,
-        returns_units_30d = EXCLUDED.returns_units_30d, returns_revenue_30d = EXCLUDED.returns_revenue_30d, discounts_30d = EXCLUDED.discounts_30d, gross_revenue_30d = EXCLUDED.gross_revenue_30d, gross_regular_revenue_30d = EXCLUDED.gross_regular_revenue_30d,
-        stockout_days_30d = EXCLUDED.stockout_days_30d, sales_365d = EXCLUDED.sales_365d, revenue_365d = EXCLUDED.revenue_365d,
-        avg_stock_units_30d = EXCLUDED.avg_stock_units_30d, avg_stock_cost_30d = EXCLUDED.avg_stock_cost_30d, avg_stock_retail_30d = EXCLUDED.avg_stock_retail_30d, avg_stock_gross_30d = EXCLUDED.avg_stock_gross_30d,
-        received_qty_30d = EXCLUDED.received_qty_30d, received_cost_30d = EXCLUDED.received_cost_30d,
-        lifetime_sales = EXCLUDED.lifetime_sales, lifetime_revenue = EXCLUDED.lifetime_revenue,
-        first_7_days_sales = EXCLUDED.first_7_days_sales, first_7_days_revenue = EXCLUDED.first_7_days_revenue, first_30_days_sales = EXCLUDED.first_30_days_sales, first_30_days_revenue = EXCLUDED.first_30_days_revenue,
-        first_60_days_sales = EXCLUDED.first_60_days_sales, first_60_days_revenue = EXCLUDED.first_60_days_revenue, first_90_days_sales = EXCLUDED.first_90_days_sales, first_90_days_revenue = EXCLUDED.first_90_days_revenue,
-        asp_30d = EXCLUDED.asp_30d, acp_30d = EXCLUDED.acp_30d, avg_ros_30d = EXCLUDED.avg_ros_30d, avg_sales_per_day_30d = EXCLUDED.avg_sales_per_day_30d, 
-        margin_30d = EXCLUDED.margin_30d, markup_30d = EXCLUDED.markup_30d, gmroi_30d = EXCLUDED.gmroi_30d, stockturn_30d = EXCLUDED.stockturn_30d, return_rate_30d = EXCLUDED.return_rate_30d, discount_rate_30d = EXCLUDED.discount_rate_30d,
-        stockout_rate_30d = EXCLUDED.stockout_rate_30d, markdown_30d = EXCLUDED.markdown_30d, markdown_rate_30d = EXCLUDED.markdown_rate_30d, sell_through_30d = EXCLUDED.sell_through_30d,
-        avg_lead_time_days = EXCLUDED.avg_lead_time_days, abc_class = EXCLUDED.abc_class,
-        sales_velocity_daily = EXCLUDED.sales_velocity_daily, config_lead_time = EXCLUDED.config_lead_time, config_days_of_stock = EXCLUDED.config_days_of_stock, config_safety_stock = EXCLUDED.config_safety_stock,
-        planning_period_days = EXCLUDED.planning_period_days, lead_time_forecast_units = EXCLUDED.lead_time_forecast_units, days_of_stock_forecast_units = EXCLUDED.days_of_stock_forecast_units,
-        planning_period_forecast_units = EXCLUDED.planning_period_forecast_units, lead_time_closing_stock = EXCLUDED.lead_time_closing_stock, days_of_stock_closing_stock = EXCLUDED.days_of_stock_closing_stock,
-        replenishment_needed_raw = EXCLUDED.replenishment_needed_raw, replenishment_units = EXCLUDED.replenishment_units, replenishment_cost = EXCLUDED.replenishment_cost, replenishment_retail = EXCLUDED.replenishment_retail, replenishment_profit = EXCLUDED.replenishment_profit,
-        to_order_units = EXCLUDED.to_order_units, -- *** Update to use EXCLUDED ***
-        forecast_lost_sales_units = EXCLUDED.forecast_lost_sales_units, forecast_lost_revenue = EXCLUDED.forecast_lost_revenue,
-        stock_cover_in_days = EXCLUDED.stock_cover_in_days, po_cover_in_days = EXCLUDED.po_cover_in_days, sells_out_in_days = EXCLUDED.sells_out_in_days, replenish_date = EXCLUDED.replenish_date,
-        overstocked_units = EXCLUDED.overstocked_units, overstocked_cost = EXCLUDED.overstocked_cost, overstocked_retail = EXCLUDED.overstocked_retail, is_old_stock = EXCLUDED.is_old_stock,
-        yesterday_sales = EXCLUDED.yesterday_sales;
-    RAISE NOTICE 'Finished % module. Duration: %', _module_name, clock_timestamp() - _start_time;
-END $$;

inventory-server/scripts/metrics-new/backfill/rebuild_daily_snapshots.sql

@@ -1,152 +0,0 @@
--- Description: Rebuilds daily product snapshots from scratch using real orders data.
---              Fixes issues with duplicated/inflated metrics.
--- Dependencies: Core import tables (products, orders, receivings).
--- Frequency: One-time run to clear out problematic data.
-
-DO $$
-DECLARE
-    _module_name TEXT := 'rebuild_daily_snapshots';
-    _start_time TIMESTAMPTZ := clock_timestamp();
-    _date DATE;
-    _count INT;
-    _total_records INT := 0;
-    _begin_date DATE := (SELECT MIN(date)::date FROM orders WHERE date >= '2024-01-01'); -- Starting point for data rebuild
-    _end_date DATE := CURRENT_DATE;
-BEGIN
-    RAISE NOTICE 'Beginning daily snapshots rebuild from % to %. Starting at %', _begin_date, _end_date, _start_time;
-
-    -- First truncate the existing snapshots to ensure a clean slate
-    TRUNCATE TABLE public.daily_product_snapshots;
-    RAISE NOTICE 'Cleared existing snapshot data';
-    
-    -- Now rebuild the snapshots day by day
-    _date := _begin_date;
-    
-    WHILE _date <= _end_date LOOP
-        RAISE NOTICE 'Processing date %...', _date;
-        
-        -- Create snapshots for this date
-        WITH SalesData AS (
-            SELECT
-                p.pid,
-                p.sku,
-                -- Count orders to ensure we only include products with real activity
-                COUNT(o.id) as order_count,
-                -- Aggregate Sales (Quantity > 0, Status not Canceled/Returned)
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.quantity ELSE 0 END), 0) AS units_sold,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.price * o.quantity ELSE 0 END), 0.00) AS gross_revenue_unadjusted,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.discount ELSE 0 END), 0.00) AS discounts,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN COALESCE(o.costeach, p.landing_cost_price, p.cost_price) * o.quantity ELSE 0 END), 0.00) AS cogs,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN p.regular_price * o.quantity ELSE 0 END), 0.00) AS gross_regular_revenue,
-
-                -- Aggregate Returns (Quantity < 0 or Status = Returned)
-                COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN ABS(o.quantity) ELSE 0 END), 0) AS units_returned,
-                COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN o.price * ABS(o.quantity) ELSE 0 END), 0.00) AS returns_revenue
-            FROM public.products p
-            LEFT JOIN public.orders o
-                ON p.pid = o.pid
-               AND o.date::date = _date
-            GROUP BY p.pid, p.sku
-            HAVING COUNT(o.id) > 0 -- Only include products with actual orders for this date
-        ),
-        ReceivingData AS (
-            SELECT
-                r.pid,
-                -- Count receiving documents to ensure we only include products with real activity
-                COUNT(DISTINCT r.receiving_id) as receiving_count,
-                -- Calculate received quantity for this day
-                SUM(r.qty_each) AS units_received,
-                -- Calculate received cost for this day
-                SUM(r.qty_each * r.cost_each) AS cost_received
-            FROM public.receivings r
-            WHERE r.received_date::date = _date
-            GROUP BY r.pid
-            HAVING COUNT(DISTINCT r.receiving_id) > 0 OR SUM(r.qty_each) > 0
-        ),
-        -- Get stock quantities for the day - note this is approximate since we're using current products data
-        StockData AS (
-            SELECT
-                p.pid,
-                p.stock_quantity,
-                COALESCE(p.landing_cost_price, p.cost_price, 0.00) as effective_cost_price,
-                COALESCE(p.price, 0.00) as current_price,
-                COALESCE(p.regular_price, 0.00) as current_regular_price
-            FROM public.products p
-        )
-        INSERT INTO public.daily_product_snapshots (
-            snapshot_date,
-            pid,
-            sku,
-            eod_stock_quantity,
-            eod_stock_cost,
-            eod_stock_retail,
-            eod_stock_gross,
-            stockout_flag,
-            units_sold,
-            units_returned,
-            gross_revenue,
-            discounts,
-            returns_revenue,
-            net_revenue,
-            cogs,
-            gross_regular_revenue,
-            profit,
-            units_received,
-            cost_received,
-            calculation_timestamp
-        )
-        SELECT
-            _date AS snapshot_date,
-            COALESCE(sd.pid, rd.pid) AS pid,
-            sd.sku,
-            -- Use current stock as approximation, since historical stock data may not be available
-            s.stock_quantity AS eod_stock_quantity,
-            s.stock_quantity * s.effective_cost_price AS eod_stock_cost,
-            s.stock_quantity * s.current_price AS eod_stock_retail,
-            s.stock_quantity * s.current_regular_price AS eod_stock_gross,
-            (s.stock_quantity <= 0) AS stockout_flag,
-            -- Sales metrics
-            COALESCE(sd.units_sold, 0),
-            COALESCE(sd.units_returned, 0),
-            COALESCE(sd.gross_revenue_unadjusted, 0.00),
-            COALESCE(sd.discounts, 0.00),
-            COALESCE(sd.returns_revenue, 0.00),
-            COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00) AS net_revenue,
-            COALESCE(sd.cogs, 0.00),
-            COALESCE(sd.gross_regular_revenue, 0.00),
-            (COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00)) - COALESCE(sd.cogs, 0.00) AS profit,
-            -- Receiving metrics
-            COALESCE(rd.units_received, 0),
-            COALESCE(rd.cost_received, 0.00),
-            _start_time
-        FROM SalesData sd
-        FULL OUTER JOIN ReceivingData rd ON sd.pid = rd.pid
-        LEFT JOIN StockData s ON COALESCE(sd.pid, rd.pid) = s.pid
-        WHERE (COALESCE(sd.order_count, 0) > 0 OR COALESCE(rd.receiving_count, 0) > 0);
-        
-        -- Get record count for this day
-        GET DIAGNOSTICS _count = ROW_COUNT;
-        _total_records := _total_records + _count;
-        
-        RAISE NOTICE 'Added % snapshot records for date %', _count, _date;
-        
-        -- Move to next day
-        _date := _date + INTERVAL '1 day';
-    END LOOP;
-    
-    RAISE NOTICE 'Rebuilding daily snapshots complete. Added % total records across % days.', _total_records, (_end_date - _begin_date)::integer + 1;
-    
-    -- Update the status table for daily_snapshots
-    INSERT INTO public.calculate_status (module_name, last_calculation_timestamp)
-    VALUES ('daily_snapshots', _start_time)
-    ON CONFLICT (module_name) DO UPDATE SET last_calculation_timestamp = _start_time;
-    
-    -- Now update product_metrics based on the rebuilt snapshots
-    RAISE NOTICE 'Triggering update of product_metrics table...';
-    
-    -- Call the update_product_metrics procedure directly
-    -- Your system might use a different method to trigger this update
-    PERFORM pg_notify('recalculate_metrics', 'product_metrics');
-    
-    RAISE NOTICE 'Rebuild complete. Duration: %', clock_timestamp() - _start_time;
-END $$; 

inventory-server/scripts/metrics-new/calculate_brand_metrics.sql

@@ -42,20 +42,6 @@ BEGIN
         JOIN public.products p ON pm.pid = p.pid
         GROUP BY brand_group
     ),
-    PreviousPeriodBrandMetrics AS (
-        -- Get previous period metrics for growth calculation
-        SELECT
-            COALESCE(p.brand, 'Unbranded') AS brand_group,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.units_sold ELSE 0 END) AS sales_prev_30d,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.net_revenue ELSE 0 END) AS revenue_prev_30d
-        FROM public.daily_product_snapshots dps
-        JOIN public.products p ON dps.pid = p.pid
-        GROUP BY brand_group
-    ),
     AllBrands AS (
          -- Ensure all brands from products table are included, mapping NULL/empty to 'Unbranded'
         SELECT DISTINCT COALESCE(brand, 'Unbranded') as brand_group
@@ -67,8 +53,7 @@ BEGIN
         current_stock_units, current_stock_cost, current_stock_retail,
         sales_7d, revenue_7d, sales_30d, revenue_30d, profit_30d, cogs_30d,
         sales_365d, revenue_365d, lifetime_sales, lifetime_revenue,
-        avg_margin_30d,
-        sales_growth_30d_vs_prev, revenue_growth_30d_vs_prev
+        avg_margin_30d
     )
     SELECT
         b.brand_group,
@@ -93,13 +78,9 @@ BEGIN
                 -- This is mathematically equivalent to profit/revenue but more explicit
                 ((COALESCE(ba.revenue_30d, 0) - COALESCE(ba.cogs_30d, 0)) / COALESCE(ba.revenue_30d, 1)) * 100.0
             ELSE NULL -- No margin for low/no revenue brands
-        END,
-        -- Growth metrics
-        std_numeric(safe_divide((ba.sales_30d - ppbm.sales_prev_30d) * 100.0, ppbm.sales_prev_30d), 2),
-        std_numeric(safe_divide((ba.revenue_30d - ppbm.revenue_prev_30d) * 100.0, ppbm.revenue_prev_30d), 2)
+        END
     FROM AllBrands b
     LEFT JOIN BrandAggregates ba ON b.brand_group = ba.brand_group
-    LEFT JOIN PreviousPeriodBrandMetrics ppbm ON b.brand_group = ppbm.brand_group
 
     ON CONFLICT (brand_name) DO UPDATE SET
         last_calculated = EXCLUDED.last_calculated,
@@ -114,9 +95,7 @@ BEGIN
         profit_30d = EXCLUDED.profit_30d, cogs_30d = EXCLUDED.cogs_30d,
         sales_365d = EXCLUDED.sales_365d, revenue_365d = EXCLUDED.revenue_365d,
         lifetime_sales = EXCLUDED.lifetime_sales, lifetime_revenue = EXCLUDED.lifetime_revenue,
-        avg_margin_30d = EXCLUDED.avg_margin_30d,
-        sales_growth_30d_vs_prev = EXCLUDED.sales_growth_30d_vs_prev,
-        revenue_growth_30d_vs_prev = EXCLUDED.revenue_growth_30d_vs_prev
+        avg_margin_30d = EXCLUDED.avg_margin_30d
     WHERE -- Only update if at least one value has changed
         brand_metrics.product_count IS DISTINCT FROM EXCLUDED.product_count OR
         brand_metrics.active_product_count IS DISTINCT FROM EXCLUDED.active_product_count OR

inventory-server/scripts/metrics-new/calculate_category_metrics.sql

@@ -1,5 +1,5 @@
--- Description: Calculates and updates aggregated metrics per category with hierarchy rollups.
--- Dependencies: product_metrics, products, categories, product_categories, category_hierarchy, calculate_status table.
+-- Description: Calculates and updates aggregated metrics per category.
+-- Dependencies: product_metrics, products, categories, product_categories, calculate_status table.
 -- Frequency: Daily (after product_metrics update).
 
 DO $$
@@ -9,21 +9,55 @@ DECLARE
     _min_revenue NUMERIC := 50.00; -- Minimum revenue threshold for margin calculation
 BEGIN
     RAISE NOTICE 'Running % calculation...', _module_name;
-    
-    -- Refresh the category hierarchy materialized view first
-    REFRESH MATERIALIZED VIEW CONCURRENTLY category_hierarchy;
 
-    -- First calculate direct metrics (products directly in each category)
-    WITH DirectCategoryMetrics AS (
+    WITH 
+    -- Identify the hierarchy depth for each category
+    CategoryDepth AS (
+        WITH RECURSIVE CategoryTree AS (
+            -- Base case: Start with categories without parents (root categories)
+            SELECT cat_id, name, parent_id, 0 AS depth
+            FROM public.categories
+            WHERE parent_id IS NULL
+            
+            UNION ALL
+            
+            -- Recursive step: Add child categories with incremented depth
+            SELECT c.cat_id, c.name, c.parent_id, ct.depth + 1
+            FROM public.categories c
+            JOIN CategoryTree ct ON c.parent_id = ct.cat_id
+        )
+        SELECT cat_id, depth
+        FROM CategoryTree
+    ),
+    -- For each product, find the most specific (deepest) category it belongs to
+    ProductDeepestCategory AS (
+        SELECT 
+            pc.pid,
+            pc.cat_id
+        FROM public.product_categories pc
+        JOIN CategoryDepth cd ON pc.cat_id = cd.cat_id
+        -- This is the key part: for each product, select only the category with maximum depth
+        WHERE (pc.pid, cd.depth) IN (
+            SELECT pc2.pid, MAX(cd2.depth)
+            FROM public.product_categories pc2
+            JOIN CategoryDepth cd2 ON pc2.cat_id = cd2.cat_id
+            GROUP BY pc2.pid
+        )
+    ),
+    -- Calculate metrics only at the most specific category level for each product
+    -- These are the direct metrics (only products directly in this category)
+    DirectCategoryMetrics AS (
         SELECT
-            pc.cat_id,
+            pdc.cat_id,
+            -- Counts
             COUNT(DISTINCT pm.pid) AS product_count,
             COUNT(DISTINCT CASE WHEN pm.is_visible THEN pm.pid END) AS active_product_count,
             COUNT(DISTINCT CASE WHEN pm.is_replenishable THEN pm.pid END) AS replenishable_product_count,
+            -- Current Stock
             SUM(pm.current_stock) AS current_stock_units,
             SUM(pm.current_stock_cost) AS current_stock_cost,
             SUM(pm.current_stock_retail) AS current_stock_retail,
-            -- Sales metrics with proper filtering
+            -- Rolling Periods - Only include products with actual sales in each period
             SUM(CASE WHEN pm.sales_7d > 0 THEN pm.sales_7d ELSE 0 END) AS sales_7d,
             SUM(CASE WHEN pm.revenue_7d > 0 THEN pm.revenue_7d ELSE 0 END) AS revenue_7d,
             SUM(CASE WHEN pm.sales_30d > 0 THEN pm.sales_30d ELSE 0 END) AS sales_30d,
@@ -33,141 +67,179 @@ BEGIN
             SUM(CASE WHEN pm.sales_365d > 0 THEN pm.sales_365d ELSE 0 END) AS sales_365d,
             SUM(CASE WHEN pm.revenue_365d > 0 THEN pm.revenue_365d ELSE 0 END) AS revenue_365d,
             SUM(CASE WHEN pm.lifetime_sales > 0 THEN pm.lifetime_sales ELSE 0 END) AS lifetime_sales,
-            SUM(CASE WHEN pm.lifetime_revenue > 0 THEN pm.lifetime_revenue ELSE 0 END) AS lifetime_revenue
-        FROM public.product_categories pc
-        JOIN public.product_metrics pm ON pc.pid = pm.pid
-        GROUP BY pc.cat_id
+            SUM(CASE WHEN pm.lifetime_revenue > 0 THEN pm.lifetime_revenue ELSE 0 END) AS lifetime_revenue,
+            -- Data for KPIs - Only average stock for products with stock
+            SUM(CASE WHEN pm.avg_stock_units_30d > 0 THEN pm.avg_stock_units_30d ELSE 0 END) AS total_avg_stock_units_30d
+        FROM public.product_metrics pm
+        JOIN ProductDeepestCategory pdc ON pm.pid = pdc.pid
+        GROUP BY pdc.cat_id
     ),
-    -- Calculate rolled-up metrics (including all descendant categories)
-    RolledUpMetrics AS (
+    -- Build a category lookup table for parent relationships
+    CategoryHierarchyPaths AS (
+        WITH RECURSIVE ParentPaths AS (
+            -- Base case: All categories with their immediate parents
+            SELECT 
+                cat_id,
+                cat_id as leaf_id, -- Every category is its own leaf initially
+                ARRAY[cat_id] as path
+            FROM public.categories
+            
+            UNION ALL
+            
+            -- Recursive step: Walk up the parent chain
+            SELECT
+                c.parent_id as cat_id,
+                pp.leaf_id, -- Keep the original leaf_id
+                c.parent_id || pp.path as path
+            FROM ParentPaths pp
+            JOIN public.categories c ON pp.cat_id = c.cat_id
+            WHERE c.parent_id IS NOT NULL -- Stop at root categories
+        )
+        -- Select distinct paths to avoid duplication
+        SELECT DISTINCT cat_id, leaf_id
+        FROM ParentPaths
+    ),
+    -- Aggregate metrics from leaf categories to their ancestors without duplication
+    -- These are the rolled-up metrics (including all child categories)
+    RollupMetrics AS (
         SELECT
-            ch.cat_id,
-            -- Sum metrics from this category and all its descendants
-            SUM(dcm.product_count) AS product_count,
-            SUM(dcm.active_product_count) AS active_product_count,
-            SUM(dcm.replenishable_product_count) AS replenishable_product_count,
-            SUM(dcm.current_stock_units) AS current_stock_units,
-            SUM(dcm.current_stock_cost) AS current_stock_cost,
-            SUM(dcm.current_stock_retail) AS current_stock_retail,
-            SUM(dcm.sales_7d) AS sales_7d,
-            SUM(dcm.revenue_7d) AS revenue_7d,
-            SUM(dcm.sales_30d) AS sales_30d,
-            SUM(dcm.revenue_30d) AS revenue_30d,
-            SUM(dcm.cogs_30d) AS cogs_30d,
-            SUM(dcm.profit_30d) AS profit_30d,
-            SUM(dcm.sales_365d) AS sales_365d,
-            SUM(dcm.revenue_365d) AS revenue_365d,
-            SUM(dcm.lifetime_sales) AS lifetime_sales,
-            SUM(dcm.lifetime_revenue) AS lifetime_revenue
-        FROM category_hierarchy ch
-        LEFT JOIN DirectCategoryMetrics dcm ON 
-            dcm.cat_id = ch.cat_id OR 
-            dcm.cat_id = ANY(SELECT cat_id FROM category_hierarchy WHERE ch.cat_id = ANY(ancestor_ids))
-        GROUP BY ch.cat_id
+            chp.cat_id,
+            -- For each parent category, count distinct products to avoid duplication
+            COUNT(DISTINCT dcm.cat_id) AS child_categories_count,
+            SUM(dcm.product_count) AS rollup_product_count,
+            SUM(dcm.active_product_count) AS rollup_active_product_count,
+            SUM(dcm.replenishable_product_count) AS rollup_replenishable_product_count,
+            SUM(dcm.current_stock_units) AS rollup_current_stock_units,
+            SUM(dcm.current_stock_cost) AS rollup_current_stock_cost,
+            SUM(dcm.current_stock_retail) AS rollup_current_stock_retail,
+            SUM(dcm.sales_7d) AS rollup_sales_7d,
+            SUM(dcm.revenue_7d) AS rollup_revenue_7d,
+            SUM(dcm.sales_30d) AS rollup_sales_30d,
+            SUM(dcm.revenue_30d) AS rollup_revenue_30d,
+            SUM(dcm.cogs_30d) AS rollup_cogs_30d,
+            SUM(dcm.profit_30d) AS rollup_profit_30d,
+            SUM(dcm.sales_365d) AS rollup_sales_365d,
+            SUM(dcm.revenue_365d) AS rollup_revenue_365d,
+            SUM(dcm.lifetime_sales) AS rollup_lifetime_sales,
+            SUM(dcm.lifetime_revenue) AS rollup_lifetime_revenue,
+            SUM(dcm.total_avg_stock_units_30d) AS rollup_total_avg_stock_units_30d
+        FROM CategoryHierarchyPaths chp
+        JOIN DirectCategoryMetrics dcm ON chp.leaf_id = dcm.cat_id
+        GROUP BY chp.cat_id
     ),
-    PreviousPeriodCategoryMetrics AS (
-        -- Get previous period metrics for growth calculation
+    -- Combine direct and rollup metrics
+    CombinedMetrics AS (
         SELECT
-            pc.cat_id,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.units_sold ELSE 0 END) AS sales_prev_30d,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.net_revenue ELSE 0 END) AS revenue_prev_30d
-        FROM public.daily_product_snapshots dps
-        JOIN public.product_categories pc ON dps.pid = pc.pid
-        GROUP BY pc.cat_id
-    ),
-    RolledUpPreviousPeriod AS (
-        -- Calculate rolled-up previous period metrics
-        SELECT
-            ch.cat_id,
-            SUM(ppcm.sales_prev_30d) AS sales_prev_30d,
-            SUM(ppcm.revenue_prev_30d) AS revenue_prev_30d
-        FROM category_hierarchy ch
-        LEFT JOIN PreviousPeriodCategoryMetrics ppcm ON 
-            ppcm.cat_id = ch.cat_id OR 
-            ppcm.cat_id = ANY(SELECT cat_id FROM category_hierarchy WHERE ch.cat_id = ANY(ancestor_ids))
-        GROUP BY ch.cat_id
-    ),
-    AllCategories AS (
-        -- Ensure all categories are included
-        SELECT 
             c.cat_id,
             c.name,
             c.type,
-            c.parent_id
+            c.parent_id,
+            -- Direct metrics (just this category)
+            COALESCE(dcm.product_count, 0) AS direct_product_count,
+            COALESCE(dcm.active_product_count, 0) AS direct_active_product_count,
+            COALESCE(dcm.replenishable_product_count, 0) AS direct_replenishable_product_count,
+            COALESCE(dcm.current_stock_units, 0) AS direct_current_stock_units,
+            COALESCE(dcm.current_stock_cost, 0) AS direct_current_stock_cost,
+            COALESCE(dcm.current_stock_retail, 0) AS direct_current_stock_retail,
+            COALESCE(dcm.sales_7d, 0) AS direct_sales_7d,
+            COALESCE(dcm.revenue_7d, 0) AS direct_revenue_7d,
+            COALESCE(dcm.sales_30d, 0) AS direct_sales_30d,
+            COALESCE(dcm.revenue_30d, 0) AS direct_revenue_30d,
+            COALESCE(dcm.cogs_30d, 0) AS direct_cogs_30d,
+            COALESCE(dcm.profit_30d, 0) AS direct_profit_30d,
+            COALESCE(dcm.sales_365d, 0) AS direct_sales_365d,
+            COALESCE(dcm.revenue_365d, 0) AS direct_revenue_365d,
+            COALESCE(dcm.lifetime_sales, 0) AS direct_lifetime_sales,
+            COALESCE(dcm.lifetime_revenue, 0) AS direct_lifetime_revenue,
+            COALESCE(dcm.total_avg_stock_units_30d, 0) AS direct_avg_stock_units_30d,
+            
+            -- Rolled up metrics (this category + all children)
+            COALESCE(rm.rollup_product_count, 0) AS product_count,
+            COALESCE(rm.rollup_active_product_count, 0) AS active_product_count,
+            COALESCE(rm.rollup_replenishable_product_count, 0) AS replenishable_product_count,
+            COALESCE(rm.rollup_current_stock_units, 0) AS current_stock_units,
+            COALESCE(rm.rollup_current_stock_cost, 0) AS current_stock_cost,
+            COALESCE(rm.rollup_current_stock_retail, 0) AS current_stock_retail,
+            COALESCE(rm.rollup_sales_7d, 0) AS sales_7d,
+            COALESCE(rm.rollup_revenue_7d, 0) AS revenue_7d,
+            COALESCE(rm.rollup_sales_30d, 0) AS sales_30d,
+            COALESCE(rm.rollup_revenue_30d, 0) AS revenue_30d,
+            COALESCE(rm.rollup_cogs_30d, 0) AS cogs_30d,
+            COALESCE(rm.rollup_profit_30d, 0) AS profit_30d,
+            COALESCE(rm.rollup_sales_365d, 0) AS sales_365d,
+            COALESCE(rm.rollup_revenue_365d, 0) AS revenue_365d,
+            COALESCE(rm.rollup_lifetime_sales, 0) AS lifetime_sales,
+            COALESCE(rm.rollup_lifetime_revenue, 0) AS lifetime_revenue,
+            COALESCE(rm.rollup_total_avg_stock_units_30d, 0) AS total_avg_stock_units_30d
         FROM public.categories c
-        WHERE c.status = 'active'
+        LEFT JOIN DirectCategoryMetrics dcm ON c.cat_id = dcm.cat_id
+        LEFT JOIN RollupMetrics rm ON c.cat_id = rm.cat_id
     )
     INSERT INTO public.category_metrics (
         category_id, category_name, category_type, parent_id, last_calculated,
-        -- Rolled-up metrics
+        -- Store all direct and rolled up metrics
         product_count, active_product_count, replenishable_product_count,
         current_stock_units, current_stock_cost, current_stock_retail,
         sales_7d, revenue_7d, sales_30d, revenue_30d, profit_30d, cogs_30d,
         sales_365d, revenue_365d, lifetime_sales, lifetime_revenue,
-        -- Direct metrics
+        -- Also store direct metrics with direct_ prefix
         direct_product_count, direct_active_product_count, direct_replenishable_product_count,
         direct_current_stock_units, direct_stock_cost, direct_stock_retail,
-        direct_sales_7d, direct_revenue_7d, direct_sales_30d, direct_revenue_30d,
+        direct_sales_7d, direct_revenue_7d, direct_sales_30d, direct_revenue_30d, 
         direct_profit_30d, direct_cogs_30d, direct_sales_365d, direct_revenue_365d,
         direct_lifetime_sales, direct_lifetime_revenue,
         -- KPIs
-        avg_margin_30d,
-        sales_growth_30d_vs_prev, revenue_growth_30d_vs_prev
+        avg_margin_30d, stock_turn_30d
     )
     SELECT
-        ac.cat_id,
-        ac.name,
-        ac.type,
-        ac.parent_id,
+        cm.cat_id,
+        cm.name,
+        cm.type,
+        cm.parent_id,
         _start_time,
-        -- Rolled-up metrics (includes descendants)
-        COALESCE(rum.product_count, 0),
-        COALESCE(rum.active_product_count, 0),
-        COALESCE(rum.replenishable_product_count, 0),
-        COALESCE(rum.current_stock_units, 0),
-        COALESCE(rum.current_stock_cost, 0.00),
-        COALESCE(rum.current_stock_retail, 0.00),
-        COALESCE(rum.sales_7d, 0), COALESCE(rum.revenue_7d, 0.00),
-        COALESCE(rum.sales_30d, 0), COALESCE(rum.revenue_30d, 0.00),
-        COALESCE(rum.profit_30d, 0.00), COALESCE(rum.cogs_30d, 0.00),
-        COALESCE(rum.sales_365d, 0), COALESCE(rum.revenue_365d, 0.00),
-        COALESCE(rum.lifetime_sales, 0), COALESCE(rum.lifetime_revenue, 0.00),
-        -- Direct metrics (only this category)
-        COALESCE(dcm.product_count, 0),
-        COALESCE(dcm.active_product_count, 0),
-        COALESCE(dcm.replenishable_product_count, 0),
-        COALESCE(dcm.current_stock_units, 0),
-        COALESCE(dcm.current_stock_cost, 0.00),
-        COALESCE(dcm.current_stock_retail, 0.00),
-        COALESCE(dcm.sales_7d, 0), COALESCE(dcm.revenue_7d, 0.00),
-        COALESCE(dcm.sales_30d, 0), COALESCE(dcm.revenue_30d, 0.00),
-        COALESCE(dcm.profit_30d, 0.00), COALESCE(dcm.cogs_30d, 0.00),
-        COALESCE(dcm.sales_365d, 0), COALESCE(dcm.revenue_365d, 0.00),
-        COALESCE(dcm.lifetime_sales, 0), COALESCE(dcm.lifetime_revenue, 0.00),
+        -- Rolled-up metrics (total including children)
+        cm.product_count,
+        cm.active_product_count,
+        cm.replenishable_product_count,
+        cm.current_stock_units,
+        cm.current_stock_cost,
+        cm.current_stock_retail,
+        cm.sales_7d, cm.revenue_7d,
+        cm.sales_30d, cm.revenue_30d, cm.profit_30d, cm.cogs_30d,
+        cm.sales_365d, cm.revenue_365d,
+        cm.lifetime_sales, cm.lifetime_revenue,
+        -- Direct metrics (just this category)
+        cm.direct_product_count,
+        cm.direct_active_product_count,
+        cm.direct_replenishable_product_count,
+        cm.direct_current_stock_units,
+        cm.direct_current_stock_cost,
+        cm.direct_current_stock_retail,
+        cm.direct_sales_7d, cm.direct_revenue_7d,
+        cm.direct_sales_30d, cm.direct_revenue_30d, cm.direct_profit_30d, cm.direct_cogs_30d,
+        cm.direct_sales_365d, cm.direct_revenue_365d,
+        cm.direct_lifetime_sales, cm.direct_lifetime_revenue,
         -- KPIs - Calculate margin only for categories with significant revenue
         CASE
-            WHEN COALESCE(rum.revenue_30d, 0) >= _min_revenue THEN
-                ((COALESCE(rum.revenue_30d, 0) - COALESCE(rum.cogs_30d, 0)) / COALESCE(rum.revenue_30d, 1)) * 100.0
-            ELSE NULL
+            WHEN cm.revenue_30d >= _min_revenue THEN
+                ((cm.revenue_30d - cm.cogs_30d) / cm.revenue_30d) * 100.0
+            ELSE NULL -- No margin for low/no revenue categories
         END,
-        -- Growth metrics for rolled-up values
-        std_numeric(safe_divide((rum.sales_30d - rupp.sales_prev_30d) * 100.0, rupp.sales_prev_30d), 2),
-        std_numeric(safe_divide((rum.revenue_30d - rupp.revenue_prev_30d) * 100.0, rupp.revenue_prev_30d), 2)
-    FROM AllCategories ac
-    LEFT JOIN DirectCategoryMetrics dcm ON ac.cat_id = dcm.cat_id
-    LEFT JOIN RolledUpMetrics rum ON ac.cat_id = rum.cat_id
-    LEFT JOIN RolledUpPreviousPeriod rupp ON ac.cat_id = rupp.cat_id
+        -- Stock Turn calculation
+        CASE
+            WHEN cm.total_avg_stock_units_30d > 0 THEN
+                cm.sales_30d / cm.total_avg_stock_units_30d
+            ELSE NULL -- No stock turn if no average stock
+        END
+    FROM CombinedMetrics cm
 
     ON CONFLICT (category_id) DO UPDATE SET
-        last_calculated = EXCLUDED.last_calculated,
         category_name = EXCLUDED.category_name,
         category_type = EXCLUDED.category_type,
         parent_id = EXCLUDED.parent_id,
-        -- Rolled-up metrics
+        last_calculated = EXCLUDED.last_calculated,
+        
+        -- ROLLED-UP METRICS (includes this category + all descendants)
         product_count = EXCLUDED.product_count,
         active_product_count = EXCLUDED.active_product_count,
         replenishable_product_count = EXCLUDED.replenishable_product_count,
@@ -179,7 +251,8 @@ BEGIN
         profit_30d = EXCLUDED.profit_30d, cogs_30d = EXCLUDED.cogs_30d,
         sales_365d = EXCLUDED.sales_365d, revenue_365d = EXCLUDED.revenue_365d,
         lifetime_sales = EXCLUDED.lifetime_sales, lifetime_revenue = EXCLUDED.lifetime_revenue,
-        -- Direct metrics
+        
+        -- DIRECT METRICS (only products directly in this category)
         direct_product_count = EXCLUDED.direct_product_count,
         direct_active_product_count = EXCLUDED.direct_active_product_count,
         direct_replenishable_product_count = EXCLUDED.direct_replenishable_product_count,
@@ -191,9 +264,10 @@ BEGIN
         direct_profit_30d = EXCLUDED.direct_profit_30d, direct_cogs_30d = EXCLUDED.direct_cogs_30d,
         direct_sales_365d = EXCLUDED.direct_sales_365d, direct_revenue_365d = EXCLUDED.direct_revenue_365d,
         direct_lifetime_sales = EXCLUDED.direct_lifetime_sales, direct_lifetime_revenue = EXCLUDED.direct_lifetime_revenue,
+        
+        -- Calculated KPIs
         avg_margin_30d = EXCLUDED.avg_margin_30d,
-        sales_growth_30d_vs_prev = EXCLUDED.sales_growth_30d_vs_prev,
-        revenue_growth_30d_vs_prev = EXCLUDED.revenue_growth_30d_vs_prev
+        stock_turn_30d = EXCLUDED.stock_turn_30d
     WHERE -- Only update if at least one value has changed
         category_metrics.product_count IS DISTINCT FROM EXCLUDED.product_count OR
         category_metrics.active_product_count IS DISTINCT FROM EXCLUDED.active_product_count OR
@@ -217,23 +291,19 @@ WITH update_stats AS (
     SELECT 
         COUNT(*) as total_categories,
         COUNT(*) FILTER (WHERE last_calculated >= NOW() - INTERVAL '5 minutes') as rows_processed,
-        COUNT(*) FILTER (WHERE category_type = 10) as sections,
-        COUNT(*) FILTER (WHERE category_type = 11) as categories,
-        COUNT(*) FILTER (WHERE category_type = 12) as subcategories,
-        SUM(product_count) as total_products_rolled,
-        SUM(direct_product_count) as total_products_direct,
-        SUM(sales_30d) as total_sales_30d,
-        SUM(revenue_30d) as total_revenue_30d
+        COUNT(*) FILTER (WHERE category_type = 11) as main_categories, -- 11 = category
+        COUNT(*) FILTER (WHERE category_type = 12) as subcategories,  -- 12 = subcategory
+        SUM(product_count) as total_products,
+        SUM(active_product_count) as total_active_products,
+        SUM(current_stock_units) as total_stock_units
     FROM public.category_metrics
 )
 SELECT 
     rows_processed,
     total_categories,
-    sections,
-    categories,
+    main_categories,
     subcategories,
-    total_products_rolled::int,
-    total_products_direct::int,
-    total_sales_30d::int,
-    ROUND(total_revenue_30d, 2) as total_revenue_30d
+    total_products::int,
+    total_active_products::int,
+    total_stock_units::int
 FROM update_stats;

inventory-server/scripts/metrics-new/calculate_vendor_metrics.sql

@@ -1,185 +0,0 @@
--- Description: Calculates and updates aggregated metrics per vendor.
--- Dependencies: product_metrics, products, purchase_orders, calculate_status table.
--- Frequency: Daily (after product_metrics update).
-
-DO $$
-DECLARE
-    _module_name VARCHAR := 'vendor_metrics';
-    _start_time TIMESTAMPTZ := clock_timestamp();
-BEGIN
-    RAISE NOTICE 'Running % calculation...', _module_name;
-
-    WITH VendorProductAggregates AS (
-        -- Aggregate metrics from product_metrics table per vendor
-        SELECT
-            p.vendor,
-            COUNT(DISTINCT pm.pid) AS product_count,
-            COUNT(DISTINCT CASE WHEN pm.is_visible THEN pm.pid END) AS active_product_count,
-            COUNT(DISTINCT CASE WHEN pm.is_replenishable THEN pm.pid END) AS replenishable_product_count,
-            SUM(pm.current_stock) AS current_stock_units,
-            SUM(pm.current_stock_cost) AS current_stock_cost,
-            SUM(pm.current_stock_retail) AS current_stock_retail,
-            SUM(pm.on_order_qty) AS on_order_units,
-            SUM(pm.on_order_cost) AS on_order_cost,
-            -- Only include products with valid sales data in each time period
-            COUNT(DISTINCT CASE WHEN pm.sales_7d > 0 THEN pm.pid END) AS products_with_sales_7d,
-            SUM(CASE WHEN pm.sales_7d > 0 THEN pm.sales_7d ELSE 0 END) AS sales_7d,
-            SUM(CASE WHEN pm.revenue_7d > 0 THEN pm.revenue_7d ELSE 0 END) AS revenue_7d,
-            
-            COUNT(DISTINCT CASE WHEN pm.sales_30d > 0 THEN pm.pid END) AS products_with_sales_30d,
-            SUM(CASE WHEN pm.sales_30d > 0 THEN pm.sales_30d ELSE 0 END) AS sales_30d,
-            SUM(CASE WHEN pm.revenue_30d > 0 THEN pm.revenue_30d ELSE 0 END) AS revenue_30d,
-            SUM(CASE WHEN pm.cogs_30d > 0 THEN pm.cogs_30d ELSE 0 END) AS cogs_30d,
-            SUM(CASE WHEN pm.profit_30d != 0 THEN pm.profit_30d ELSE 0 END) AS profit_30d,
-            
-            COUNT(DISTINCT CASE WHEN pm.sales_365d > 0 THEN pm.pid END) AS products_with_sales_365d,
-            SUM(CASE WHEN pm.sales_365d > 0 THEN pm.sales_365d ELSE 0 END) AS sales_365d,
-            SUM(CASE WHEN pm.revenue_365d > 0 THEN pm.revenue_365d ELSE 0 END) AS revenue_365d,
-            
-            COUNT(DISTINCT CASE WHEN pm.lifetime_sales > 0 THEN pm.pid END) AS products_with_lifetime_sales,
-            SUM(CASE WHEN pm.lifetime_sales > 0 THEN pm.lifetime_sales ELSE 0 END) AS lifetime_sales,
-            SUM(CASE WHEN pm.lifetime_revenue > 0 THEN pm.lifetime_revenue ELSE 0 END) AS lifetime_revenue
-        FROM public.product_metrics pm
-        JOIN public.products p ON pm.pid = p.pid
-        WHERE p.vendor IS NOT NULL AND p.vendor <> ''
-        GROUP BY p.vendor
-    ),
-    PreviousPeriodVendorMetrics AS (
-        -- Get previous period metrics for growth calculation
-        SELECT
-            p.vendor,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.units_sold ELSE 0 END) AS sales_prev_30d,
-            SUM(CASE WHEN dps.snapshot_date >= CURRENT_DATE - INTERVAL '59 days' 
-                     AND dps.snapshot_date < CURRENT_DATE - INTERVAL '29 days' 
-                THEN dps.net_revenue ELSE 0 END) AS revenue_prev_30d
-        FROM public.daily_product_snapshots dps
-        JOIN public.products p ON dps.pid = p.pid
-        WHERE p.vendor IS NOT NULL AND p.vendor <> ''
-        GROUP BY p.vendor
-    ),
-    VendorPOAggregates AS (
-        -- Aggregate PO related stats including lead time calculated from POs to receivings
-        SELECT
-            po.vendor,
-            COUNT(DISTINCT po.po_id) AS po_count_365d,
-            -- Calculate lead time by averaging the days between PO date and receiving date
-            AVG(GREATEST(1, CASE 
-                WHEN r.received_date IS NOT NULL AND po.date IS NOT NULL 
-                THEN (r.received_date::date - po.date::date) 
-                ELSE NULL 
-            END))::int AS avg_lead_time_days_hist -- Avg lead time from HISTORICAL received POs
-        FROM public.purchase_orders po
-        -- Join to receivings table to find when items were received
-        LEFT JOIN public.receivings r ON r.pid = po.pid
-        WHERE po.vendor IS NOT NULL AND po.vendor <> ''
-          AND po.date >= CURRENT_DATE - INTERVAL '1 year' -- Look at POs created in the last year
-          AND po.status = 'done' -- Only calculate lead time on completed POs
-          AND r.received_date IS NOT NULL
-          AND po.date IS NOT NULL
-          AND r.received_date >= po.date
-        GROUP BY po.vendor
-    ),
-    AllVendors AS (
-        -- Ensure all vendors from products table are included
-        SELECT DISTINCT vendor FROM public.products WHERE vendor IS NOT NULL AND vendor <> ''
-    )
-    INSERT INTO public.vendor_metrics (
-        vendor_name, last_calculated,
-        product_count, active_product_count, replenishable_product_count,
-        current_stock_units, current_stock_cost, current_stock_retail,
-        on_order_units, on_order_cost,
-        po_count_365d, avg_lead_time_days,
-        sales_7d, revenue_7d, sales_30d, revenue_30d, profit_30d, cogs_30d,
-        sales_365d, revenue_365d, lifetime_sales, lifetime_revenue,
-        avg_margin_30d,
-        sales_growth_30d_vs_prev, revenue_growth_30d_vs_prev
-    )
-    SELECT
-        v.vendor,
-        _start_time,
-        -- Base Aggregates
-        COALESCE(vpa.product_count, 0),
-        COALESCE(vpa.active_product_count, 0),
-        COALESCE(vpa.replenishable_product_count, 0),
-        COALESCE(vpa.current_stock_units, 0),
-        COALESCE(vpa.current_stock_cost, 0.00),
-        COALESCE(vpa.current_stock_retail, 0.00),
-        COALESCE(vpa.on_order_units, 0),
-        COALESCE(vpa.on_order_cost, 0.00),
-        -- PO Aggregates
-        COALESCE(vpoa.po_count_365d, 0),
-        vpoa.avg_lead_time_days_hist, -- Can be NULL if no received POs
-        -- Sales Aggregates
-        COALESCE(vpa.sales_7d, 0), COALESCE(vpa.revenue_7d, 0.00),
-        COALESCE(vpa.sales_30d, 0), COALESCE(vpa.revenue_30d, 0.00),
-        COALESCE(vpa.profit_30d, 0.00), COALESCE(vpa.cogs_30d, 0.00),
-        COALESCE(vpa.sales_365d, 0), COALESCE(vpa.revenue_365d, 0.00),
-        COALESCE(vpa.lifetime_sales, 0), COALESCE(vpa.lifetime_revenue, 0.00),
-        -- KPIs
-        (vpa.profit_30d / NULLIF(vpa.revenue_30d, 0)) * 100.0,
-        -- Growth metrics
-        std_numeric(safe_divide((vpa.sales_30d - ppvm.sales_prev_30d) * 100.0, ppvm.sales_prev_30d), 2),
-        std_numeric(safe_divide((vpa.revenue_30d - ppvm.revenue_prev_30d) * 100.0, ppvm.revenue_prev_30d), 2)
-    FROM AllVendors v
-    LEFT JOIN VendorProductAggregates vpa ON v.vendor = vpa.vendor
-    LEFT JOIN VendorPOAggregates vpoa ON v.vendor = vpoa.vendor
-    LEFT JOIN PreviousPeriodVendorMetrics ppvm ON v.vendor = ppvm.vendor
-
-    ON CONFLICT (vendor_name) DO UPDATE SET
-        last_calculated = EXCLUDED.last_calculated,
-        product_count = EXCLUDED.product_count,
-        active_product_count = EXCLUDED.active_product_count,
-        replenishable_product_count = EXCLUDED.replenishable_product_count,
-        current_stock_units = EXCLUDED.current_stock_units,
-        current_stock_cost = EXCLUDED.current_stock_cost,
-        current_stock_retail = EXCLUDED.current_stock_retail,
-        on_order_units = EXCLUDED.on_order_units,
-        on_order_cost = EXCLUDED.on_order_cost,
-        po_count_365d = EXCLUDED.po_count_365d,
-        avg_lead_time_days = EXCLUDED.avg_lead_time_days,
-        sales_7d = EXCLUDED.sales_7d, revenue_7d = EXCLUDED.revenue_7d,
-        sales_30d = EXCLUDED.sales_30d, revenue_30d = EXCLUDED.revenue_30d,
-        profit_30d = EXCLUDED.profit_30d, cogs_30d = EXCLUDED.cogs_30d,
-        sales_365d = EXCLUDED.sales_365d, revenue_365d = EXCLUDED.revenue_365d,
-        lifetime_sales = EXCLUDED.lifetime_sales, lifetime_revenue = EXCLUDED.lifetime_revenue,
-        avg_margin_30d = EXCLUDED.avg_margin_30d,
-        sales_growth_30d_vs_prev = EXCLUDED.sales_growth_30d_vs_prev,
-        revenue_growth_30d_vs_prev = EXCLUDED.revenue_growth_30d_vs_prev
-    WHERE -- Only update if at least one value has changed
-        vendor_metrics.product_count IS DISTINCT FROM EXCLUDED.product_count OR
-        vendor_metrics.active_product_count IS DISTINCT FROM EXCLUDED.active_product_count OR
-        vendor_metrics.current_stock_units IS DISTINCT FROM EXCLUDED.current_stock_units OR
-        vendor_metrics.on_order_units IS DISTINCT FROM EXCLUDED.on_order_units OR
-        vendor_metrics.sales_30d IS DISTINCT FROM EXCLUDED.sales_30d OR
-        vendor_metrics.revenue_30d IS DISTINCT FROM EXCLUDED.revenue_30d OR
-        vendor_metrics.lifetime_sales IS DISTINCT FROM EXCLUDED.lifetime_sales;
-
-    -- Update calculate_status
-    INSERT INTO public.calculate_status (module_name, last_calculation_timestamp)
-    VALUES (_module_name, _start_time)
-    ON CONFLICT (module_name) DO UPDATE SET last_calculation_timestamp = _start_time;
-
-    RAISE NOTICE 'Finished % calculation. Duration: %', _module_name, clock_timestamp() - _start_time;
-END $$;
-
--- Return metrics about the update operation for tracking
-WITH update_stats AS (
-    SELECT 
-        COUNT(*) as total_vendors,
-        COUNT(*) FILTER (WHERE last_calculated >= NOW() - INTERVAL '5 minutes') as rows_processed,
-        SUM(product_count) as total_products,
-        SUM(active_product_count) as total_active_products,
-        SUM(po_count_365d) as total_pos_365d,
-        AVG(avg_lead_time_days) as overall_avg_lead_time
-    FROM public.vendor_metrics
-)
-SELECT 
-    rows_processed,
-    total_vendors,
-    total_products::int,
-    total_active_products::int,
-    total_pos_365d::int,
-    ROUND(overall_avg_lead_time, 1) as overall_avg_lead_time
-FROM update_stats;

inventory-server/scripts/metrics-new/update_daily_snapshots.sql

@@ -1,222 +0,0 @@
--- Description: Calculates and updates daily aggregated product data for recent days.
---              Uses UPSERT (INSERT ON CONFLICT UPDATE) for idempotency.
--- Dependencies: Core import tables (products, orders, purchase_orders), calculate_status table.
--- Frequency: Hourly (Run ~5-10 minutes after hourly data import completes).
-
-DO $$
-DECLARE
-    _module_name TEXT := 'daily_snapshots';
-    _start_time TIMESTAMPTZ := clock_timestamp(); -- Time execution started
-    _last_calc_time TIMESTAMPTZ;
-    _target_date DATE; -- Will be set in the loop
-    _total_records INT := 0;
-    _has_orders BOOLEAN := FALSE;
-    _process_days INT := 5; -- Number of days to check/process (today plus previous 4 days)
-    _day_counter INT;
-    _missing_days INT[] := ARRAY[]::INT[]; -- Array to store days with missing or incomplete data
-BEGIN
-    -- Get the timestamp before the last successful run of this module
-    SELECT last_calculation_timestamp INTO _last_calc_time
-    FROM public.calculate_status
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Running % script. Start Time: %', _module_name, _start_time;
-    
-    -- First, check which days need processing by comparing orders data with snapshot data
-    FOR _day_counter IN 0..(_process_days-1) LOOP
-        _target_date := CURRENT_DATE - (_day_counter * INTERVAL '1 day');
-        
-        -- Check if this date needs updating by comparing orders to snapshot data
-        -- If the date has orders but not enough snapshots, or if snapshots show zero sales but orders exist, it's incomplete
-        SELECT 
-            CASE WHEN (
-                -- We have orders for this date but not enough snapshots, or snapshots with wrong total
-                (EXISTS (SELECT 1 FROM public.orders WHERE date::date = _target_date) AND
-                 (
-                    -- No snapshots exist for this date
-                    NOT EXISTS (SELECT 1 FROM public.daily_product_snapshots WHERE snapshot_date = _target_date) OR
-                    -- Or snapshots show zero sales but orders exist
-                    (SELECT COALESCE(SUM(units_sold), 0) FROM public.daily_product_snapshots WHERE snapshot_date = _target_date) = 0 OR
-                    -- Or the count of snapshot records is significantly less than distinct products in orders
-                    (SELECT COUNT(*) FROM public.daily_product_snapshots WHERE snapshot_date = _target_date) < 
-                    (SELECT COUNT(DISTINCT pid) FROM public.orders WHERE date::date = _target_date) * 0.8
-                 )
-                )
-            ) THEN TRUE ELSE FALSE END
-        INTO _has_orders;
-        
-        IF _has_orders THEN
-            -- This day needs processing - add to our array
-            _missing_days := _missing_days || _day_counter;
-            RAISE NOTICE 'Day % needs updating (incomplete or missing data)', _target_date;
-        END IF;
-    END LOOP;
-    
-    -- If no days need updating, exit early
-    IF array_length(_missing_days, 1) IS NULL THEN
-        RAISE NOTICE 'No days need updating - all snapshot data appears complete';
-        
-        -- Still update the calculate_status to record this run
-        UPDATE public.calculate_status
-        SET last_calculation_timestamp = _start_time
-        WHERE module_name = _module_name;
-        
-        RETURN;
-    END IF;
-    
-    RAISE NOTICE 'Need to update % days with missing or incomplete data', array_length(_missing_days, 1);
-
-    -- Process only the days that need updating
-    FOREACH _day_counter IN ARRAY _missing_days LOOP
-        _target_date := CURRENT_DATE - (_day_counter * INTERVAL '1 day');
-        RAISE NOTICE 'Processing date: %', _target_date;
-        
-        -- IMPORTANT: First delete any existing data for this date to prevent duplication
-        DELETE FROM public.daily_product_snapshots 
-        WHERE snapshot_date = _target_date;
-
-        -- Proceed with calculating daily metrics only for products with actual activity
-        WITH SalesData AS (
-            SELECT
-                p.pid,
-                p.sku,
-                -- Track number of orders to ensure we have real data
-                COUNT(o.id) as order_count,
-                -- Aggregate Sales (Quantity > 0, Status not Canceled/Returned)
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.quantity ELSE 0 END), 0) AS units_sold,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.price * o.quantity ELSE 0 END), 0.00) AS gross_revenue_unadjusted, -- Before discount
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN o.discount ELSE 0 END), 0.00) AS discounts,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN 
-                    COALESCE(
-                        o.costeach,  -- First use order-specific cost if available
-                        get_weighted_avg_cost(p.pid, o.date::date),  -- Then use weighted average cost
-                        p.landing_cost_price,  -- Fallback to landing cost
-                        p.cost_price  -- Final fallback to current cost
-                    ) * o.quantity 
-                ELSE 0 END), 0.00) AS cogs,
-                COALESCE(SUM(CASE WHEN o.quantity > 0 AND COALESCE(o.status, 'pending') NOT IN ('canceled', 'returned') THEN p.regular_price * o.quantity ELSE 0 END), 0.00) AS gross_regular_revenue, -- Use current regular price for simplicity here
-
-                -- Aggregate Returns (Quantity < 0 or Status = Returned)
-                COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN ABS(o.quantity) ELSE 0 END), 0) AS units_returned,
-                COALESCE(SUM(CASE WHEN o.quantity < 0 OR COALESCE(o.status, 'pending') = 'returned' THEN o.price * ABS(o.quantity) ELSE 0 END), 0.00) AS returns_revenue
-            FROM public.products p -- Start from products to include those with no orders today
-            JOIN public.orders o -- Changed to INNER JOIN to only process products with orders
-                ON p.pid = o.pid
-               AND o.date::date = _target_date -- Cast to date to ensure compatibility regardless of original type
-            GROUP BY p.pid, p.sku
-            -- No HAVING clause here - we always want to include all orders
-        ),
-        ReceivingData AS (
-            SELECT
-                r.pid,
-                -- Track number of receiving docs to ensure we have real data
-                COUNT(DISTINCT r.receiving_id) as receiving_doc_count,
-                -- Sum the quantities received on this date
-                SUM(r.qty_each) AS units_received,
-                -- Calculate the cost received (qty * cost)
-                SUM(r.qty_each * r.cost_each) AS cost_received
-            FROM public.receivings r
-            WHERE r.received_date::date = _target_date
-              -- Optional: Filter out canceled receivings if needed
-              -- AND r.status <> 'canceled'
-            GROUP BY r.pid
-            -- Only include products with actual receiving activity
-            HAVING COUNT(DISTINCT r.receiving_id) > 0 OR SUM(r.qty_each) > 0
-        ),
-        CurrentStock AS (
-            -- Select current stock values directly from products table
-            SELECT
-                pid,
-                stock_quantity,
-                COALESCE(landing_cost_price, cost_price, 0.00) as effective_cost_price,
-                COALESCE(price, 0.00) as current_price,
-                COALESCE(regular_price, 0.00) as current_regular_price
-            FROM public.products
-        ),
-        ProductsWithActivity AS (
-            -- Quick pre-filter to only process products with activity
-            SELECT DISTINCT pid 
-            FROM (
-                SELECT pid FROM SalesData
-                UNION
-                SELECT pid FROM ReceivingData
-            ) a
-        )
-        -- Now insert records, but ONLY for products with actual activity
-        INSERT INTO public.daily_product_snapshots (
-            snapshot_date,
-            pid,
-            sku,
-            eod_stock_quantity,
-            eod_stock_cost,
-            eod_stock_retail,
-            eod_stock_gross,
-            stockout_flag,
-            units_sold,
-            units_returned,
-            gross_revenue,
-            discounts,
-            returns_revenue,
-            net_revenue,
-            cogs,
-            gross_regular_revenue,
-            profit,
-            units_received,
-            cost_received,
-            calculation_timestamp
-        )
-        SELECT
-            _target_date AS snapshot_date,
-            COALESCE(sd.pid, rd.pid) AS pid,  -- Use sales or receiving PID
-            COALESCE(sd.sku, p.sku) AS sku,   -- Get SKU from sales data or products table
-            -- Inventory Metrics (Using CurrentStock)
-            cs.stock_quantity AS eod_stock_quantity,
-            cs.stock_quantity * cs.effective_cost_price AS eod_stock_cost,
-            cs.stock_quantity * cs.current_price AS eod_stock_retail,
-            cs.stock_quantity * cs.current_regular_price AS eod_stock_gross,
-            (cs.stock_quantity <= 0) AS stockout_flag,
-            -- Sales Metrics (From SalesData)
-            COALESCE(sd.units_sold, 0),
-            COALESCE(sd.units_returned, 0),
-            COALESCE(sd.gross_revenue_unadjusted, 0.00),
-            COALESCE(sd.discounts, 0.00),
-            COALESCE(sd.returns_revenue, 0.00),
-            COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00) AS net_revenue,
-            COALESCE(sd.cogs, 0.00),
-            COALESCE(sd.gross_regular_revenue, 0.00),
-            (COALESCE(sd.gross_revenue_unadjusted, 0.00) - COALESCE(sd.discounts, 0.00)) - COALESCE(sd.cogs, 0.00) AS profit, -- Basic profit: Net Revenue - COGS
-            -- Receiving Metrics (From ReceivingData)
-            COALESCE(rd.units_received, 0),
-            COALESCE(rd.cost_received, 0.00),
-            _start_time -- Timestamp of this calculation run
-        FROM SalesData sd
-        FULL OUTER JOIN ReceivingData rd ON sd.pid = rd.pid
-        JOIN ProductsWithActivity pwa ON COALESCE(sd.pid, rd.pid) = pwa.pid
-        LEFT JOIN public.products p ON COALESCE(sd.pid, rd.pid) = p.pid
-        LEFT JOIN CurrentStock cs ON COALESCE(sd.pid, rd.pid) = cs.pid
-        WHERE p.pid IS NOT NULL; -- Ensure we only insert for existing products
-
-        -- Get the total number of records inserted for this date
-        GET DIAGNOSTICS _total_records = ROW_COUNT;
-        RAISE NOTICE 'Created % daily snapshot records for % with sales/receiving activity', _total_records, _target_date;
-    END LOOP;
-
-    -- Update the status table with the timestamp from the START of this run
-    UPDATE public.calculate_status
-    SET last_calculation_timestamp = _start_time
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Finished % processing for multiple dates. Duration: %', _module_name, clock_timestamp() - _start_time;
-
-END $$;
-
--- Return the total records processed for tracking
-SELECT 
-    COUNT(*) as rows_processed,
-    COUNT(DISTINCT snapshot_date) as days_processed,
-    MIN(snapshot_date) as earliest_date,
-    MAX(snapshot_date) as latest_date,
-    SUM(units_sold) as total_units_sold,
-    SUM(units_received) as total_units_received
-FROM public.daily_product_snapshots
-WHERE calculation_timestamp >= (NOW() - INTERVAL '5 minutes'); -- Recent updates only

inventory-server/scripts/metrics-new/update_periodic_metrics.sql

@@ -1,139 +0,0 @@
--- Description: Calculates metrics that don't need hourly updates, like ABC class
---              and average lead time.
--- Dependencies: product_metrics, purchase_orders, settings_global, calculate_status.
--- Frequency: Daily or Weekly (e.g., run via cron job overnight).
-
-DO $$
-DECLARE
-    _module_name TEXT := 'periodic_metrics';
-    _start_time TIMESTAMPTZ := clock_timestamp();
-    _last_calc_time TIMESTAMPTZ;
-    _abc_basis VARCHAR;
-    _abc_period INT;
-    _threshold_a NUMERIC;
-    _threshold_b NUMERIC;
-BEGIN
-    -- Get the timestamp before the last successful run of this module
-    SELECT last_calculation_timestamp INTO _last_calc_time
-    FROM public.calculate_status
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Running % module. Start Time: %', _module_name, _start_time;
-
-    -- 1. Calculate Average Lead Time
-    RAISE NOTICE 'Calculating Average Lead Time...';
-    WITH LeadTimes AS (
-        SELECT
-            po.pid,
-            -- Calculate lead time by looking at when items ordered on POs were received
-            AVG(GREATEST(1, (r.received_date::date - po.date::date))) AS avg_days -- Use GREATEST(1,...) to avoid 0 or negative days
-        FROM public.purchase_orders po
-        -- Join to receivings table to find actual receipts
-        JOIN public.receivings r ON r.pid = po.pid
-        WHERE po.status = 'done' -- Only include completed POs
-          AND r.received_date >= po.date -- Ensure received date is not before order date
-          -- Optional: add check to make sure receiving is related to PO if you have source_po_id
-          -- AND (r.source_po_id = po.po_id OR r.source_po_id IS NULL)
-        GROUP BY po.pid
-    )
-    UPDATE public.product_metrics pm
-    SET avg_lead_time_days = lt.avg_days::int
-    FROM LeadTimes lt
-    WHERE pm.pid = lt.pid
-      AND pm.avg_lead_time_days IS DISTINCT FROM lt.avg_days::int; -- Only update if changed
-    RAISE NOTICE 'Finished Average Lead Time calculation.';
-
-
-    -- 2. Calculate ABC Classification
-    RAISE NOTICE 'Calculating ABC Classification...';
-    -- Get ABC settings
-    SELECT setting_value INTO _abc_basis FROM public.settings_global WHERE setting_key = 'abc_calculation_basis' LIMIT 1;
-    SELECT setting_value::numeric INTO _threshold_a FROM public.settings_global WHERE setting_key = 'abc_revenue_threshold_a' LIMIT 1;
-    SELECT setting_value::numeric INTO _threshold_b FROM public.settings_global WHERE setting_key = 'abc_revenue_threshold_b' LIMIT 1;
-    _abc_basis := COALESCE(_abc_basis, 'revenue_30d'); -- Default basis
-    _threshold_a := COALESCE(_threshold_a, 0.80);
-    _threshold_b := COALESCE(_threshold_b, 0.95);
-
-    RAISE NOTICE 'Using ABC Basis: %, Threshold A: %, Threshold B: %', _abc_basis, _threshold_a, _threshold_b;
-
-    WITH RankedProducts AS (
-        SELECT
-            pid,
-            -- Dynamically select the metric based on setting
-            CASE _abc_basis
-                WHEN 'sales_30d' THEN COALESCE(sales_30d, 0)
-                WHEN 'lifetime_revenue' THEN COALESCE(lifetime_revenue, 0)::numeric -- Cast needed if different type
-                ELSE COALESCE(revenue_30d, 0) -- Default to revenue_30d
-            END AS metric_value
-        FROM public.product_metrics
-        WHERE is_replenishable = TRUE -- Typically only classify replenishable items
-    ),
-    Cumulative AS (
-        SELECT
-            pid,
-            metric_value,
-            SUM(metric_value) OVER (ORDER BY metric_value DESC NULLS LAST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) as cumulative_metric,
-            SUM(metric_value) OVER () as total_metric
-        FROM RankedProducts
-        WHERE metric_value > 0 -- Exclude items with no contribution
-    )
-    UPDATE public.product_metrics pm
-    SET abc_class =
-        CASE
-            WHEN c.cumulative_metric / NULLIF(c.total_metric, 0) <= _threshold_a THEN 'A'
-            WHEN c.cumulative_metric / NULLIF(c.total_metric, 0) <= _threshold_b THEN 'B'
-            ELSE 'C'
-        END
-    FROM Cumulative c
-    WHERE pm.pid = c.pid
-      AND pm.abc_class IS DISTINCT FROM ( -- Only update if changed
-         CASE
-            WHEN c.cumulative_metric / NULLIF(c.total_metric, 0) <= _threshold_a THEN 'A'
-            WHEN c.cumulative_metric / NULLIF(c.total_metric, 0) <= _threshold_b THEN 'B'
-            ELSE 'C'
-         END);
-
-    -- Set non-contributing or non-replenishable to 'C' or NULL if preferred
-     UPDATE public.product_metrics
-     SET abc_class = 'C' -- Or NULL
-     WHERE abc_class IS NULL AND is_replenishable = TRUE; -- Catch those with 0 metric value
-
-     UPDATE public.product_metrics
-     SET abc_class = NULL -- Or 'N/A'?
-     WHERE is_replenishable = FALSE AND abc_class IS NOT NULL; -- Unclassify non-replenishable items
-
-
-    RAISE NOTICE 'Finished ABC Classification calculation.';
-
-    -- Add other periodic calculations here if needed (e.g., recalculating first/last dates)
-
-    -- Update the status table with the timestamp from the START of this run
-    UPDATE public.calculate_status
-    SET last_calculation_timestamp = _start_time
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Finished % module. Duration: %', _module_name, clock_timestamp() - _start_time;
-
-END $$;
-
--- Return metrics about the update operation for tracking
-WITH update_stats AS (
-    SELECT 
-        COUNT(*) as total_products,
-        COUNT(*) FILTER (WHERE last_calculated >= NOW() - INTERVAL '5 minutes') as rows_processed,
-        COUNT(*) FILTER (WHERE abc_class = 'A') as abc_a_count,
-        COUNT(*) FILTER (WHERE abc_class = 'B') as abc_b_count,
-        COUNT(*) FILTER (WHERE abc_class = 'C') as abc_c_count,
-        COUNT(*) FILTER (WHERE avg_lead_time_days IS NOT NULL) as products_with_lead_time,
-        AVG(avg_lead_time_days) as overall_avg_lead_time
-    FROM public.product_metrics
-)
-SELECT 
-    rows_processed,
-    total_products,
-    abc_a_count,
-    abc_b_count,
-    abc_c_count,
-    products_with_lead_time,
-    ROUND(overall_avg_lead_time, 1) as overall_avg_lead_time
-FROM update_stats;

inventory-server/scripts/metrics-new/update_product_metrics.sql

@@ -1,609 +0,0 @@
--- Description: Calculates and updates the main product_metrics table based on current data
---              and aggregated daily snapshots. Uses UPSERT for idempotency.
--- Dependencies: Core import tables, daily_product_snapshots, configuration tables, calculate_status.
--- Frequency: Hourly (Run AFTER update_daily_snapshots.sql completes).
-
-DO $$
-DECLARE
-    _module_name TEXT := 'product_metrics';
-    _start_time TIMESTAMPTZ := clock_timestamp();
-    _last_calc_time TIMESTAMPTZ;
-    _current_date DATE := CURRENT_DATE;
-BEGIN
-    -- Get the timestamp before the last successful run of this module
-    SELECT last_calculation_timestamp INTO _last_calc_time
-    FROM public.calculate_status
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Running % module. Start Time: %', _module_name, _start_time;
-
-    -- Use CTEs to gather all necessary information
-    WITH CurrentInfo AS (
-        SELECT
-            p.pid,
-            p.sku,
-            p.title,
-            p.brand,
-            p.vendor,
-            COALESCE(p.image_175, p.image) as image_url,
-            p.visible as is_visible,
-            p.replenishable as is_replenishable,
-            -- Add new product fields
-            p.barcode,
-            p.harmonized_tariff_code,
-            p.vendor_reference,
-            p.notions_reference,
-            p.line,
-            p.subline,
-            p.artist,
-            p.moq,
-            p.rating,
-            p.reviews,
-            p.weight,
-            p.length,
-            p.width,
-            p.height,
-            p.country_of_origin,
-            p.location,
-            p.baskets,
-            p.notifies,
-            p.preorder_count,
-            p.notions_inv_count,
-            COALESCE(p.price, 0.00) as current_price,
-            COALESCE(p.regular_price, 0.00) as current_regular_price,
-            COALESCE(p.cost_price, 0.00) as current_cost_price,
-            COALESCE(p.landing_cost_price, p.cost_price, 0.00) as current_effective_cost, -- Use landing if available, else cost
-            p.stock_quantity as current_stock,
-            p.created_at,
-            p.first_received,
-            p.date_last_sold,
-            p.total_sold as historical_total_sold, -- Add historical total_sold from products table
-            p.uom -- Assuming UOM logic is handled elsewhere or simple (e.g., 1=each)
-        FROM public.products p
-    ),
-    OnOrderInfo AS (
-        SELECT
-            pid,
-            SUM(ordered) AS on_order_qty,
-            SUM(ordered * po_cost_price) AS on_order_cost,
-            MIN(expected_date) AS earliest_expected_date
-        FROM public.purchase_orders
-        WHERE status IN ('created', 'ordered', 'preordered', 'electronically_sent', 'electronically_ready_send', 'receiving_started')
-          AND status NOT IN ('canceled', 'done')
-        GROUP BY pid
-    ),
-    HistoricalDates AS (
-        -- Note: Calculating these MIN/MAX values hourly can be slow on large tables.
-        -- Consider calculating periodically or storing on products if import can populate them.
-        SELECT
-            p.pid,
-            MIN(o.date)::date AS date_first_sold,
-            MAX(o.date)::date AS max_order_date, -- Use MAX for potential recalc of date_last_sold
-            
-            -- For first received, use the new receivings table
-            MIN(r.received_date)::date AS date_first_received_calc,
-            
-            -- For last received, use the new receivings table
-            MAX(r.received_date)::date AS date_last_received_calc
-        FROM public.products p
-        LEFT JOIN public.orders o ON p.pid = o.pid AND o.quantity > 0 AND o.status NOT IN ('canceled', 'returned')
-        LEFT JOIN public.receivings r ON p.pid = r.pid
-        GROUP BY p.pid
-    ),
-    SnapshotAggregates AS (
-        SELECT
-            pid,
-            -- Get the counts of all available data 
-            COUNT(DISTINCT snapshot_date) AS available_days,
-            
-            -- Rolling periods with no time constraint - just sum everything we have
-            SUM(units_sold) AS total_units_sold,
-            SUM(net_revenue) AS total_net_revenue,
-            
-            -- Specific time windows using date range boundaries precisely
-            -- Use _current_date - INTERVAL '6 days' to include 7 days (today + 6 previous days)
-            -- This ensures we count exactly the right number of days in each period
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '6 days' AND snapshot_date <= _current_date THEN units_sold ELSE 0 END) AS sales_7d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '6 days' AND snapshot_date <= _current_date THEN net_revenue ELSE 0 END) AS revenue_7d,
-            
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '13 days' AND snapshot_date <= _current_date THEN units_sold ELSE 0 END) AS sales_14d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '13 days' AND snapshot_date <= _current_date THEN net_revenue ELSE 0 END) AS revenue_14d,
-            
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN units_sold ELSE 0 END) AS sales_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN net_revenue ELSE 0 END) AS revenue_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN cogs ELSE 0 END) AS cogs_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN profit ELSE 0 END) AS profit_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN units_returned ELSE 0 END) AS returns_units_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN returns_revenue ELSE 0 END) AS returns_revenue_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN discounts ELSE 0 END) AS discounts_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN gross_revenue ELSE 0 END) AS gross_revenue_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN gross_regular_revenue ELSE 0 END) AS gross_regular_revenue_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date AND stockout_flag THEN 1 ELSE 0 END) AS stockout_days_30d,
-            
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '364 days' AND snapshot_date <= _current_date THEN units_sold ELSE 0 END) AS sales_365d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '364 days' AND snapshot_date <= _current_date THEN net_revenue ELSE 0 END) AS revenue_365d,
-            
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN units_received ELSE 0 END) AS received_qty_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN cost_received ELSE 0 END) AS received_cost_30d,
-
-            -- Averages for stock levels - only include dates within the specified period
-            AVG(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN eod_stock_quantity END) AS avg_stock_units_30d,
-            AVG(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN eod_stock_cost END) AS avg_stock_cost_30d,
-            AVG(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN eod_stock_retail END) AS avg_stock_retail_30d,
-            AVG(CASE WHEN snapshot_date >= _current_date - INTERVAL '29 days' AND snapshot_date <= _current_date THEN eod_stock_gross END) AS avg_stock_gross_30d,
-
-            -- Lifetime - should match total values above
-            SUM(units_sold) AS lifetime_sales,
-            SUM(net_revenue) AS lifetime_revenue,
-
-            -- Yesterday
-            SUM(CASE WHEN snapshot_date = _current_date - INTERVAL '1 day' THEN units_sold ELSE 0 END) as yesterday_sales
-
-        FROM public.daily_product_snapshots
-        GROUP BY pid
-    ),
-    FirstPeriodMetrics AS (
-      SELECT
-          pid,
-          date_first_sold,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '6 days' THEN units_sold ELSE 0 END) AS first_7_days_sales,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '6 days' THEN net_revenue ELSE 0 END) AS first_7_days_revenue,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '29 days' THEN units_sold ELSE 0 END) AS first_30_days_sales,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '29 days' THEN net_revenue ELSE 0 END) AS first_30_days_revenue,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '59 days' THEN units_sold ELSE 0 END) AS first_60_days_sales,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '59 days' THEN net_revenue ELSE 0 END) AS first_60_days_revenue,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '89 days' THEN units_sold ELSE 0 END) AS first_90_days_sales,
-          SUM(CASE WHEN snapshot_date >= date_first_sold AND snapshot_date <= date_first_sold + INTERVAL '89 days' THEN net_revenue ELSE 0 END) AS first_90_days_revenue
-      FROM public.daily_product_snapshots ds
-      JOIN HistoricalDates hd USING(pid)
-      WHERE date_first_sold IS NOT NULL
-        AND snapshot_date >= date_first_sold
-        AND snapshot_date <= date_first_sold + INTERVAL '90 days' -- Limit scan range
-      GROUP BY pid, date_first_sold
-    ),
-    Settings AS (
-      SELECT
-        p.pid,
-        COALESCE(sp.lead_time_days, sv.default_lead_time_days, (SELECT setting_value FROM settings_global WHERE setting_key = 'default_lead_time_days')::int, 14) AS effective_lead_time,
-        COALESCE(sp.days_of_stock, sv.default_days_of_stock, (SELECT setting_value FROM settings_global WHERE setting_key = 'default_days_of_stock')::int, 30) AS effective_days_of_stock,
-        COALESCE(sp.safety_stock, 0) AS effective_safety_stock, -- Assuming safety stock is units, not days from global for now
-        COALESCE(sp.exclude_from_forecast, FALSE) AS exclude_forecast
-      FROM public.products p
-      LEFT JOIN public.settings_product sp ON p.pid = sp.pid
-      LEFT JOIN public.settings_vendor sv ON p.vendor = sv.vendor
-    ),
-    LifetimeRevenue AS (
-        -- Calculate actual revenue from orders table
-        SELECT
-            o.pid,
-            SUM(o.price * o.quantity - COALESCE(o.discount, 0)) AS lifetime_revenue_from_orders,
-            SUM(o.quantity) AS lifetime_units_from_orders
-        FROM public.orders o
-        WHERE o.status NOT IN ('canceled', 'returned')
-          AND o.quantity > 0
-        GROUP BY o.pid
-    ),
-    PreviousPeriodMetrics AS (
-        -- Calculate metrics for previous 30-day period for growth comparison
-        SELECT
-            pid,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '59 days' 
-                     AND snapshot_date < _current_date - INTERVAL '29 days' 
-                THEN units_sold ELSE 0 END) AS sales_prev_30d,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '59 days' 
-                     AND snapshot_date < _current_date - INTERVAL '29 days' 
-                THEN net_revenue ELSE 0 END) AS revenue_prev_30d,
-            -- Year-over-year comparison
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '395 days' 
-                     AND snapshot_date < _current_date - INTERVAL '365 days' 
-                THEN units_sold ELSE 0 END) AS sales_30d_last_year,
-            SUM(CASE WHEN snapshot_date >= _current_date - INTERVAL '395 days' 
-                     AND snapshot_date < _current_date - INTERVAL '365 days' 
-                THEN net_revenue ELSE 0 END) AS revenue_30d_last_year
-        FROM public.daily_product_snapshots
-        GROUP BY pid
-    ),
-    DemandVariability AS (
-        -- Calculate variance and standard deviation of daily sales
-        SELECT
-            pid,
-            COUNT(*) AS days_with_data,
-            AVG(units_sold) AS avg_daily_sales,
-            VARIANCE(units_sold) AS sales_variance,
-            STDDEV(units_sold) AS sales_std_dev,
-            -- Coefficient of variation
-            CASE 
-                WHEN AVG(units_sold) > 0 THEN STDDEV(units_sold) / AVG(units_sold)
-                ELSE NULL 
-            END AS sales_cv
-        FROM public.daily_product_snapshots
-        WHERE snapshot_date >= _current_date - INTERVAL '29 days' 
-          AND snapshot_date <= _current_date
-        GROUP BY pid
-    ),
-    ServiceLevels AS (
-        -- Calculate service level and fill rate metrics
-        SELECT
-            pid,
-            COUNT(*) FILTER (WHERE stockout_flag = true) AS stockout_incidents_30d,
-            COUNT(*) FILTER (WHERE stockout_flag = true AND units_sold > 0) AS lost_sales_incidents_30d,
-            -- Service level: percentage of days without stockouts
-            (1.0 - (COUNT(*) FILTER (WHERE stockout_flag = true)::NUMERIC / NULLIF(COUNT(*), 0))) * 100 AS service_level_30d,
-            -- Fill rate: units sold / (units sold + potential lost sales)
-            CASE 
-                WHEN SUM(units_sold) > 0 THEN
-                    (SUM(units_sold)::NUMERIC / 
-                     (SUM(units_sold) + SUM(CASE WHEN stockout_flag THEN units_sold * 0.2 ELSE 0 END))) * 100
-                ELSE NULL
-            END AS fill_rate_30d
-        FROM public.daily_product_snapshots
-        WHERE snapshot_date >= _current_date - INTERVAL '29 days' 
-          AND snapshot_date <= _current_date
-        GROUP BY pid
-    ),
-    SeasonalityAnalysis AS (
-        -- Simple seasonality detection
-        SELECT 
-            p.pid,
-            sp.seasonal_pattern,
-            sp.seasonality_index,
-            sp.peak_season
-        FROM products p
-        CROSS JOIN LATERAL detect_seasonal_pattern(p.pid) sp
-    )
-    -- Final UPSERT into product_metrics
-    INSERT INTO public.product_metrics (
-        pid, last_calculated, sku, title, brand, vendor, image_url, is_visible, is_replenishable,
-        barcode, harmonized_tariff_code, vendor_reference, notions_reference, line, subline, artist, 
-        moq, rating, reviews, weight, length, width, height, country_of_origin, location, 
-        baskets, notifies, preorder_count, notions_inv_count,
-        current_price, current_regular_price, current_cost_price, current_landing_cost_price,
-        current_stock, current_stock_cost, current_stock_retail, current_stock_gross,
-        on_order_qty, on_order_cost, on_order_retail, earliest_expected_date,
-        date_created, date_first_received, date_last_received, date_first_sold, date_last_sold, age_days,
-        sales_7d, revenue_7d, sales_14d, revenue_14d, sales_30d, revenue_30d, cogs_30d, profit_30d,
-        returns_units_30d, returns_revenue_30d, discounts_30d, gross_revenue_30d, gross_regular_revenue_30d,
-        stockout_days_30d, sales_365d, revenue_365d,
-        avg_stock_units_30d, avg_stock_cost_30d, avg_stock_retail_30d, avg_stock_gross_30d,
-        received_qty_30d, received_cost_30d,
-        lifetime_sales, lifetime_revenue, lifetime_revenue_quality,
-        first_7_days_sales, first_7_days_revenue, first_30_days_sales, first_30_days_revenue,
-        first_60_days_sales, first_60_days_revenue, first_90_days_sales, first_90_days_revenue,
-        asp_30d, acp_30d, avg_ros_30d, avg_sales_per_day_30d, avg_sales_per_month_30d,
-        margin_30d, markup_30d, gmroi_30d, stockturn_30d, return_rate_30d, discount_rate_30d,
-        stockout_rate_30d, markdown_30d, markdown_rate_30d, sell_through_30d,
-        -- avg_lead_time_days, -- Calculated periodically
-        -- abc_class, -- Calculated periodically
-        sales_velocity_daily, config_lead_time, config_days_of_stock, config_safety_stock,
-        planning_period_days, lead_time_forecast_units, days_of_stock_forecast_units,
-        planning_period_forecast_units, lead_time_closing_stock, days_of_stock_closing_stock,
-        replenishment_needed_raw, replenishment_units, replenishment_cost, replenishment_retail, replenishment_profit,
-        to_order_units, forecast_lost_sales_units, forecast_lost_revenue,
-        stock_cover_in_days, po_cover_in_days, sells_out_in_days, replenish_date,
-        overstocked_units, overstocked_cost, overstocked_retail, is_old_stock,
-        yesterday_sales,
-        status,  -- Add status field for calculated status
-        -- New fields
-        sales_growth_30d_vs_prev, revenue_growth_30d_vs_prev,
-        sales_growth_yoy, revenue_growth_yoy,
-        sales_variance_30d, sales_std_dev_30d, sales_cv_30d, demand_pattern,
-        fill_rate_30d, stockout_incidents_30d, service_level_30d, lost_sales_incidents_30d,
-        seasonality_index, seasonal_pattern, peak_season
-    )
-    SELECT
-        ci.pid, _start_time, ci.sku, ci.title, ci.brand, ci.vendor, ci.image_url, ci.is_visible, ci.is_replenishable,
-        ci.barcode, ci.harmonized_tariff_code, ci.vendor_reference, ci.notions_reference, ci.line, ci.subline, ci.artist,
-        ci.moq, ci.rating, ci.reviews, ci.weight, ci.length, ci.width, ci.height, ci.country_of_origin, ci.location,
-        ci.baskets, ci.notifies, ci.preorder_count, ci.notions_inv_count,
-        ci.current_price, ci.current_regular_price, ci.current_cost_price, ci.current_effective_cost,
-        ci.current_stock, ci.current_stock * ci.current_effective_cost, ci.current_stock * ci.current_price, ci.current_stock * ci.current_regular_price,
-        COALESCE(ooi.on_order_qty, 0), COALESCE(ooi.on_order_cost, 0.00), COALESCE(ooi.on_order_qty, 0) * ci.current_price, ooi.earliest_expected_date,
-        ci.created_at::date, COALESCE(ci.first_received::date, hd.date_first_received_calc), hd.date_last_received_calc, hd.date_first_sold, COALESCE(ci.date_last_sold, hd.max_order_date), 
-        CASE 
-            WHEN ci.created_at IS NULL AND hd.date_first_sold IS NULL THEN 0
-            WHEN ci.created_at IS NULL THEN (_current_date - hd.date_first_sold)::integer
-            WHEN hd.date_first_sold IS NULL THEN (_current_date - ci.created_at::date)::integer
-            ELSE (_current_date - LEAST(ci.created_at::date, hd.date_first_sold))::integer
-        END AS age_days,
-        sa.sales_7d, sa.revenue_7d, sa.sales_14d, sa.revenue_14d, sa.sales_30d, sa.revenue_30d, sa.cogs_30d, sa.profit_30d,
-        sa.returns_units_30d, sa.returns_revenue_30d, sa.discounts_30d, sa.gross_revenue_30d, sa.gross_regular_revenue_30d,
-        sa.stockout_days_30d, sa.sales_365d, sa.revenue_365d,
-        sa.avg_stock_units_30d, sa.avg_stock_cost_30d, sa.avg_stock_retail_30d, sa.avg_stock_gross_30d,
-        sa.received_qty_30d, sa.received_cost_30d,
-        -- Use total_sold from products table as the source of truth for lifetime sales
-        -- This includes all historical data from the production database
-        ci.historical_total_sold AS lifetime_sales,
-        -- Calculate lifetime revenue using actual historical prices where available
-        CASE 
-            WHEN lr.lifetime_revenue_from_orders IS NOT NULL THEN
-                -- We have some order history - use it plus estimate for remaining
-                lr.lifetime_revenue_from_orders + 
-                (GREATEST(0, ci.historical_total_sold - COALESCE(lr.lifetime_units_from_orders, 0)) * 
-                 COALESCE(
-                    -- Use oldest known price from snapshots as proxy
-                    (SELECT revenue_7d / NULLIF(sales_7d, 0) 
-                     FROM daily_product_snapshots 
-                     WHERE pid = ci.pid AND sales_7d > 0 
-                     ORDER BY snapshot_date ASC 
-                     LIMIT 1),
-                    ci.current_price
-                 ))
-            ELSE
-                -- No order history - estimate using current price
-                ci.historical_total_sold * ci.current_price
-        END AS lifetime_revenue,
-        CASE 
-            WHEN lr.lifetime_units_from_orders >= ci.historical_total_sold * 0.9 THEN 'exact'
-            WHEN lr.lifetime_units_from_orders >= ci.historical_total_sold * 0.5 THEN 'partial'
-            ELSE 'estimated'
-        END AS lifetime_revenue_quality,
-        fpm.first_7_days_sales, fpm.first_7_days_revenue, fpm.first_30_days_sales, fpm.first_30_days_revenue,
-        fpm.first_60_days_sales, fpm.first_60_days_revenue, fpm.first_90_days_sales, fpm.first_90_days_revenue,
-        sa.revenue_30d / NULLIF(sa.sales_30d, 0) AS asp_30d,
-        sa.cogs_30d / NULLIF(sa.sales_30d, 0) AS acp_30d,
-        sa.profit_30d / NULLIF(sa.sales_30d, 0) AS avg_ros_30d,
-        sa.sales_30d / 30.0 AS avg_sales_per_day_30d,
-        sa.sales_30d AS avg_sales_per_month_30d, -- Using 30d sales as proxy for month
-        (sa.profit_30d / NULLIF(sa.revenue_30d, 0)) * 100 AS margin_30d,
-        (sa.profit_30d / NULLIF(sa.cogs_30d, 0)) * 100 AS markup_30d,
-        sa.profit_30d / NULLIF(sa.avg_stock_cost_30d, 0) AS gmroi_30d,
-        sa.sales_30d / NULLIF(sa.avg_stock_units_30d, 0) AS stockturn_30d,
-        (sa.returns_units_30d / NULLIF(sa.sales_30d + sa.returns_units_30d, 0)) * 100 AS return_rate_30d,
-        (sa.discounts_30d / NULLIF(sa.gross_revenue_30d, 0)) * 100 AS discount_rate_30d,
-        (sa.stockout_days_30d / 30.0) * 100 AS stockout_rate_30d,
-        sa.gross_regular_revenue_30d - sa.gross_revenue_30d AS markdown_30d,
-        ((sa.gross_regular_revenue_30d - sa.gross_revenue_30d) / NULLIF(sa.gross_regular_revenue_30d, 0)) * 100 AS markdown_rate_30d,
-        -- Fix sell-through rate: Industry standard is Units Sold / (Beginning Inventory + Units Received)
-        -- Approximating beginning inventory as current stock + units sold - units received
-        (sa.sales_30d / NULLIF(
-            ci.current_stock + sa.sales_30d + sa.returns_units_30d - sa.received_qty_30d, 
-            0
-        )) * 100 AS sell_through_30d,
-
-        -- Forecasting intermediate values
-        -- Use the calculate_sales_velocity function instead of repetitive calculation
-        calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) AS sales_velocity_daily,
-        s.effective_lead_time AS config_lead_time,
-        s.effective_days_of_stock AS config_days_of_stock,
-        s.effective_safety_stock AS config_safety_stock,
-        (s.effective_lead_time + s.effective_days_of_stock) AS planning_period_days,
-        
-        calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time AS lead_time_forecast_units,
-        
-        calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock AS days_of_stock_forecast_units,
-        
-        calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * (s.effective_lead_time + s.effective_days_of_stock) AS planning_period_forecast_units,
-        
-        (ci.current_stock + COALESCE(ooi.on_order_qty, 0) - (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time)) AS lead_time_closing_stock,
-        
-        ((ci.current_stock + COALESCE(ooi.on_order_qty, 0) - (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time))) - (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock) AS days_of_stock_closing_stock,
-        
-        ((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0) AS replenishment_needed_raw,
-
-        -- Final Forecasting / Replenishment Metrics
-        CEILING(GREATEST(0, (((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0))))::int AS replenishment_units,
-        (CEILING(GREATEST(0, (((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0))))::int) * ci.current_effective_cost AS replenishment_cost,
-        (CEILING(GREATEST(0, (((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0))))::int) * ci.current_price AS replenishment_retail,
-        (CEILING(GREATEST(0, (((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0))))::int) * (ci.current_price - ci.current_effective_cost) AS replenishment_profit,
-
-        -- To Order (Apply MOQ/UOM logic here if needed, otherwise equals replenishment)
-        CEILING(GREATEST(0, (((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)) + s.effective_safety_stock - ci.current_stock - COALESCE(ooi.on_order_qty, 0))))::int AS to_order_units,
-
-        GREATEST(0, - (ci.current_stock + COALESCE(ooi.on_order_qty, 0) - (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time))) AS forecast_lost_sales_units,
-        GREATEST(0, - (ci.current_stock + COALESCE(ooi.on_order_qty, 0) - (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time))) * ci.current_price AS forecast_lost_revenue,
-
-        ci.current_stock / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0) AS stock_cover_in_days,
-        COALESCE(ooi.on_order_qty, 0) / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0) AS po_cover_in_days,
-        (ci.current_stock + COALESCE(ooi.on_order_qty, 0)) / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0) AS sells_out_in_days,
-
-        -- Replenish Date: Date when stock is projected to hit safety stock, minus lead time
-        CASE
-            WHEN calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) > 0
-            THEN _current_date + FLOOR(GREATEST(0, ci.current_stock - s.effective_safety_stock) / calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int))::int - s.effective_lead_time
-            ELSE NULL
-        END AS replenish_date,
-
-        GREATEST(0, ci.current_stock - s.effective_safety_stock - ((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)))::int AS overstocked_units,
-        (GREATEST(0, ci.current_stock - s.effective_safety_stock - ((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)))) * ci.current_effective_cost AS overstocked_cost,
-        (GREATEST(0, ci.current_stock - s.effective_safety_stock - ((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock)))) * ci.current_price AS overstocked_retail,
-
-        -- Old Stock Flag
-        (ci.created_at::date < _current_date - INTERVAL '60 day') AND
-        (COALESCE(ci.date_last_sold, hd.max_order_date) IS NULL OR COALESCE(ci.date_last_sold, hd.max_order_date) < _current_date - INTERVAL '60 day') AND
-        (hd.date_last_received_calc IS NULL OR hd.date_last_received_calc < _current_date - INTERVAL '60 day') AND
-        COALESCE(ooi.on_order_qty, 0) = 0
-        AS is_old_stock,
-
-        sa.yesterday_sales,
-
-        -- Calculate status using direct CASE statements (inline logic)
-        CASE
-            -- Non-replenishable items default to Healthy
-            WHEN NOT ci.is_replenishable THEN 'Healthy'
-            
-            -- Calculate lead time and thresholds
-            ELSE
-                CASE
-                    -- Check for overstock first 
-                    WHEN GREATEST(0, ci.current_stock - s.effective_safety_stock - ((calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_lead_time) + (calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int) * s.effective_days_of_stock))) > 0 THEN 'Overstock'
-                    
-                    -- Check for Critical stock
-                    WHEN ci.current_stock <= 0 OR 
-                         (ci.current_stock / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0)) <= 0 THEN 'Critical'
-                    
-                    WHEN (ci.current_stock / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0)) < (COALESCE(s.effective_lead_time, 30) * 0.5) THEN 'Critical'
-                    
-                    -- Check for reorder soon
-                    WHEN ((ci.current_stock + COALESCE(ooi.on_order_qty, 0)) / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0)) < (COALESCE(s.effective_lead_time, 30) + 7) THEN 
-                         CASE
-                           WHEN (ci.current_stock / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0)) < (COALESCE(s.effective_lead_time, 30) * 0.5) THEN 'Critical'
-                           ELSE 'Reorder Soon'
-                         END
-                    
-                    -- Check for 'At Risk' - old stock
-                    WHEN (ci.created_at::date < _current_date - INTERVAL '60 day') AND
-                         (COALESCE(ci.date_last_sold, hd.max_order_date) IS NULL OR COALESCE(ci.date_last_sold, hd.max_order_date) < _current_date - INTERVAL '60 day') AND
-                         (hd.date_last_received_calc IS NULL OR hd.date_last_received_calc < _current_date - INTERVAL '60 day') AND
-                         COALESCE(ooi.on_order_qty, 0) = 0 THEN 'At Risk'
-                         
-                    -- Check for 'At Risk' - hasn't sold in a long time
-                    WHEN COALESCE(ci.date_last_sold, hd.max_order_date) IS NOT NULL 
-                      AND COALESCE(ci.date_last_sold, hd.max_order_date) < (_current_date - INTERVAL '90 days')
-                      AND (CASE 
-                            WHEN ci.created_at IS NULL AND hd.date_first_sold IS NULL THEN 0
-                            WHEN ci.created_at IS NULL THEN (_current_date - hd.date_first_sold)::integer
-                            WHEN hd.date_first_sold IS NULL THEN (_current_date - ci.created_at::date)::integer
-                            ELSE (_current_date - LEAST(ci.created_at::date, hd.date_first_sold))::integer
-                          END) > 180 THEN 'At Risk'
-                          
-                    -- Very high stock cover is at risk too
-                    WHEN (ci.current_stock / NULLIF(calculate_sales_velocity(sa.sales_30d::int, sa.stockout_days_30d::int), 0)) > 365 THEN 'At Risk'
-                         
-                    -- New products (less than 30 days old)
-                    WHEN (CASE 
-                           WHEN ci.created_at IS NULL AND hd.date_first_sold IS NULL THEN 0
-                           WHEN ci.created_at IS NULL THEN (_current_date - hd.date_first_sold)::integer
-                           WHEN hd.date_first_sold IS NULL THEN (_current_date - ci.created_at::date)::integer
-                           ELSE (_current_date - LEAST(ci.created_at::date, hd.date_first_sold))::integer
-                         END) <= 30 THEN 'New'
-                         
-                    -- If none of the above, assume Healthy
-                    ELSE 'Healthy'
-                END
-        END AS status,
-
-        -- Growth Metrics (P3) - using safe_divide and std_numeric for consistency
-        std_numeric(safe_divide((sa.sales_30d - ppm.sales_prev_30d) * 100.0, ppm.sales_prev_30d), 2) AS sales_growth_30d_vs_prev,
-        std_numeric(safe_divide((sa.revenue_30d - ppm.revenue_prev_30d) * 100.0, ppm.revenue_prev_30d), 2) AS revenue_growth_30d_vs_prev,
-        std_numeric(safe_divide((sa.sales_30d - ppm.sales_30d_last_year) * 100.0, ppm.sales_30d_last_year), 2) AS sales_growth_yoy,
-        std_numeric(safe_divide((sa.revenue_30d - ppm.revenue_30d_last_year) * 100.0, ppm.revenue_30d_last_year), 2) AS revenue_growth_yoy,
-
-        -- Demand Variability (P3)
-        std_numeric(dv.sales_variance, 2) AS sales_variance_30d,
-        std_numeric(dv.sales_std_dev, 2) AS sales_std_dev_30d,
-        std_numeric(dv.sales_cv, 2) AS sales_cv_30d,
-        classify_demand_pattern(dv.avg_daily_sales, dv.sales_cv) AS demand_pattern,
-
-        -- Service Levels (P5)
-        std_numeric(COALESCE(sl.fill_rate_30d, 100), 2) AS fill_rate_30d,
-        COALESCE(sl.stockout_incidents_30d, 0)::int AS stockout_incidents_30d,
-        std_numeric(COALESCE(sl.service_level_30d, 100), 2) AS service_level_30d,
-        COALESCE(sl.lost_sales_incidents_30d, 0)::int AS lost_sales_incidents_30d,
-
-        -- Seasonality (P5)
-        std_numeric(season.seasonality_index, 2) AS seasonality_index,
-        COALESCE(season.seasonal_pattern, 'none') AS seasonal_pattern,
-        season.peak_season
-
-    FROM CurrentInfo ci
-    LEFT JOIN OnOrderInfo ooi ON ci.pid = ooi.pid
-    LEFT JOIN HistoricalDates hd ON ci.pid = hd.pid
-    LEFT JOIN SnapshotAggregates sa ON ci.pid = sa.pid
-    LEFT JOIN FirstPeriodMetrics fpm ON ci.pid = fpm.pid
-    LEFT JOIN Settings s ON ci.pid = s.pid
-    LEFT JOIN LifetimeRevenue lr ON ci.pid = lr.pid
-    LEFT JOIN PreviousPeriodMetrics ppm ON ci.pid = ppm.pid
-    LEFT JOIN DemandVariability dv ON ci.pid = dv.pid
-    LEFT JOIN ServiceLevels sl ON ci.pid = sl.pid
-    LEFT JOIN SeasonalityAnalysis season ON ci.pid = season.pid
-    WHERE s.exclude_forecast IS FALSE OR s.exclude_forecast IS NULL -- Exclude products explicitly marked
-
-    ON CONFLICT (pid) DO UPDATE SET
-        last_calculated = EXCLUDED.last_calculated,
-        sku = EXCLUDED.sku, title = EXCLUDED.title, brand = EXCLUDED.brand, vendor = EXCLUDED.vendor, image_url = EXCLUDED.image_url, is_visible = EXCLUDED.is_visible, is_replenishable = EXCLUDED.is_replenishable,
-        barcode = EXCLUDED.barcode, harmonized_tariff_code = EXCLUDED.harmonized_tariff_code, vendor_reference = EXCLUDED.vendor_reference, notions_reference = EXCLUDED.notions_reference, line = EXCLUDED.line, subline = EXCLUDED.subline, artist = EXCLUDED.artist,
-        moq = EXCLUDED.moq, rating = EXCLUDED.rating, reviews = EXCLUDED.reviews, weight = EXCLUDED.weight, length = EXCLUDED.length, width = EXCLUDED.width, height = EXCLUDED.height, country_of_origin = EXCLUDED.country_of_origin, location = EXCLUDED.location,
-        baskets = EXCLUDED.baskets, notifies = EXCLUDED.notifies, preorder_count = EXCLUDED.preorder_count, notions_inv_count = EXCLUDED.notions_inv_count,
-        current_price = EXCLUDED.current_price, current_regular_price = EXCLUDED.current_regular_price, current_cost_price = EXCLUDED.current_cost_price, current_landing_cost_price = EXCLUDED.current_landing_cost_price,
-        current_stock = EXCLUDED.current_stock, current_stock_cost = EXCLUDED.current_stock_cost, current_stock_retail = EXCLUDED.current_stock_retail, current_stock_gross = EXCLUDED.current_stock_gross,
-        on_order_qty = EXCLUDED.on_order_qty, on_order_cost = EXCLUDED.on_order_cost, on_order_retail = EXCLUDED.on_order_retail, earliest_expected_date = EXCLUDED.earliest_expected_date,
-        date_created = EXCLUDED.date_created, date_first_received = EXCLUDED.date_first_received, date_last_received = EXCLUDED.date_last_received, date_first_sold = EXCLUDED.date_first_sold, date_last_sold = EXCLUDED.date_last_sold, age_days = EXCLUDED.age_days,
-        sales_7d = EXCLUDED.sales_7d, revenue_7d = EXCLUDED.revenue_7d, sales_14d = EXCLUDED.sales_14d, revenue_14d = EXCLUDED.revenue_14d, sales_30d = EXCLUDED.sales_30d, revenue_30d = EXCLUDED.revenue_30d, cogs_30d = EXCLUDED.cogs_30d, profit_30d = EXCLUDED.profit_30d,
-        returns_units_30d = EXCLUDED.returns_units_30d, returns_revenue_30d = EXCLUDED.returns_revenue_30d, discounts_30d = EXCLUDED.discounts_30d, gross_revenue_30d = EXCLUDED.gross_revenue_30d, gross_regular_revenue_30d = EXCLUDED.gross_regular_revenue_30d,
-        stockout_days_30d = EXCLUDED.stockout_days_30d, sales_365d = EXCLUDED.sales_365d, revenue_365d = EXCLUDED.revenue_365d,
-        avg_stock_units_30d = EXCLUDED.avg_stock_units_30d, avg_stock_cost_30d = EXCLUDED.avg_stock_cost_30d, avg_stock_retail_30d = EXCLUDED.avg_stock_retail_30d, avg_stock_gross_30d = EXCLUDED.avg_stock_gross_30d,
-        received_qty_30d = EXCLUDED.received_qty_30d, received_cost_30d = EXCLUDED.received_cost_30d,
-        lifetime_sales = EXCLUDED.lifetime_sales, lifetime_revenue = EXCLUDED.lifetime_revenue, lifetime_revenue_quality = EXCLUDED.lifetime_revenue_quality,
-        first_7_days_sales = EXCLUDED.first_7_days_sales, first_7_days_revenue = EXCLUDED.first_7_days_revenue, first_30_days_sales = EXCLUDED.first_30_days_sales, first_30_days_revenue = EXCLUDED.first_30_days_revenue,
-        first_60_days_sales = EXCLUDED.first_60_days_sales, first_60_days_revenue = EXCLUDED.first_60_days_revenue, first_90_days_sales = EXCLUDED.first_90_days_sales, first_90_days_revenue = EXCLUDED.first_90_days_revenue,
-        asp_30d = EXCLUDED.asp_30d, acp_30d = EXCLUDED.acp_30d, avg_ros_30d = EXCLUDED.avg_ros_30d, avg_sales_per_day_30d = EXCLUDED.avg_sales_per_day_30d, avg_sales_per_month_30d = EXCLUDED.avg_sales_per_month_30d,
-        margin_30d = EXCLUDED.margin_30d, markup_30d = EXCLUDED.markup_30d, gmroi_30d = EXCLUDED.gmroi_30d, stockturn_30d = EXCLUDED.stockturn_30d, return_rate_30d = EXCLUDED.return_rate_30d, discount_rate_30d = EXCLUDED.discount_rate_30d,
-        stockout_rate_30d = EXCLUDED.stockout_rate_30d, markdown_30d = EXCLUDED.markdown_30d, markdown_rate_30d = EXCLUDED.markdown_rate_30d, sell_through_30d = EXCLUDED.sell_through_30d,
-        -- avg_lead_time_days = EXCLUDED.avg_lead_time_days, -- Updated Periodically
-        -- abc_class = EXCLUDED.abc_class, -- Updated Periodically
-        sales_velocity_daily = EXCLUDED.sales_velocity_daily, config_lead_time = EXCLUDED.config_lead_time, config_days_of_stock = EXCLUDED.config_days_of_stock, config_safety_stock = EXCLUDED.config_safety_stock,
-        planning_period_days = EXCLUDED.planning_period_days, lead_time_forecast_units = EXCLUDED.lead_time_forecast_units, days_of_stock_forecast_units = EXCLUDED.days_of_stock_forecast_units,
-        planning_period_forecast_units = EXCLUDED.planning_period_forecast_units, lead_time_closing_stock = EXCLUDED.lead_time_closing_stock, days_of_stock_closing_stock = EXCLUDED.days_of_stock_closing_stock,
-        replenishment_needed_raw = EXCLUDED.replenishment_needed_raw, replenishment_units = EXCLUDED.replenishment_units, replenishment_cost = EXCLUDED.replenishment_cost, replenishment_retail = EXCLUDED.replenishment_retail, replenishment_profit = EXCLUDED.replenishment_profit,
-        to_order_units = EXCLUDED.to_order_units, forecast_lost_sales_units = EXCLUDED.forecast_lost_sales_units, forecast_lost_revenue = EXCLUDED.forecast_lost_revenue,
-        stock_cover_in_days = EXCLUDED.stock_cover_in_days, po_cover_in_days = EXCLUDED.po_cover_in_days, sells_out_in_days = EXCLUDED.sells_out_in_days, replenish_date = EXCLUDED.replenish_date,
-        overstocked_units = EXCLUDED.overstocked_units, overstocked_cost = EXCLUDED.overstocked_cost, overstocked_retail = EXCLUDED.overstocked_retail, is_old_stock = EXCLUDED.is_old_stock,
-        yesterday_sales = EXCLUDED.yesterday_sales,
-        status = EXCLUDED.status,
-        sales_growth_30d_vs_prev = EXCLUDED.sales_growth_30d_vs_prev,
-        revenue_growth_30d_vs_prev = EXCLUDED.revenue_growth_30d_vs_prev,
-        sales_growth_yoy = EXCLUDED.sales_growth_yoy,
-        revenue_growth_yoy = EXCLUDED.revenue_growth_yoy,
-        sales_variance_30d = EXCLUDED.sales_variance_30d,
-        sales_std_dev_30d = EXCLUDED.sales_std_dev_30d,
-        sales_cv_30d = EXCLUDED.sales_cv_30d,
-        demand_pattern = EXCLUDED.demand_pattern,
-        fill_rate_30d = EXCLUDED.fill_rate_30d,
-        stockout_incidents_30d = EXCLUDED.stockout_incidents_30d,
-        service_level_30d = EXCLUDED.service_level_30d,
-        lost_sales_incidents_30d = EXCLUDED.lost_sales_incidents_30d,
-        seasonality_index = EXCLUDED.seasonality_index,
-        seasonal_pattern = EXCLUDED.seasonal_pattern,
-        peak_season = EXCLUDED.peak_season
-        WHERE -- Only update if at least one key metric has changed
-            product_metrics.current_stock IS DISTINCT FROM EXCLUDED.current_stock OR
-            product_metrics.current_price IS DISTINCT FROM EXCLUDED.current_price OR
-            product_metrics.current_cost_price IS DISTINCT FROM EXCLUDED.current_cost_price OR
-            product_metrics.on_order_qty IS DISTINCT FROM EXCLUDED.on_order_qty OR
-            product_metrics.sales_7d IS DISTINCT FROM EXCLUDED.sales_7d OR
-            product_metrics.sales_30d IS DISTINCT FROM EXCLUDED.sales_30d OR
-            product_metrics.revenue_30d IS DISTINCT FROM EXCLUDED.revenue_30d OR
-            product_metrics.status IS DISTINCT FROM EXCLUDED.status OR
-            product_metrics.replenishment_units IS DISTINCT FROM EXCLUDED.replenishment_units OR
-            product_metrics.stock_cover_in_days IS DISTINCT FROM EXCLUDED.stock_cover_in_days OR
-            product_metrics.yesterday_sales IS DISTINCT FROM EXCLUDED.yesterday_sales OR
-            -- Check a few other important fields that might change
-            product_metrics.date_last_sold IS DISTINCT FROM EXCLUDED.date_last_sold OR
-            product_metrics.earliest_expected_date IS DISTINCT FROM EXCLUDED.earliest_expected_date OR
-            product_metrics.lifetime_sales IS DISTINCT FROM EXCLUDED.lifetime_sales OR
-            product_metrics.lifetime_revenue_quality IS DISTINCT FROM EXCLUDED.lifetime_revenue_quality
-        ;
-
-    -- Update the status table with the timestamp from the START of this run
-    UPDATE public.calculate_status
-    SET last_calculation_timestamp = _start_time
-    WHERE module_name = _module_name;
-
-    RAISE NOTICE 'Finished % module. Duration: %', _module_name, clock_timestamp() - _start_time;
-
-END $$;
-
--- Return metrics about the update operation
-WITH update_stats AS (
-    SELECT 
-        COUNT(*) as total_products,
-        COUNT(*) FILTER (WHERE last_calculated >= NOW() - INTERVAL '5 minutes') as rows_processed,
-        COUNT(*) FILTER (WHERE status = 'Critical') as critical_count,
-        COUNT(*) FILTER (WHERE status = 'Reorder Soon') as reorder_soon_count,
-        COUNT(*) FILTER (WHERE status = 'Healthy') as healthy_count,
-        COUNT(*) FILTER (WHERE status = 'Overstock') as overstock_count,
-        COUNT(*) FILTER (WHERE status = 'At Risk') as at_risk_count,
-        COUNT(*) FILTER (WHERE status = 'New') as new_count
-    FROM public.product_metrics
-)
-SELECT 
-    rows_processed,
-    total_products,
-    critical_count,
-    reorder_soon_count,
-    healthy_count,
-    overstock_count,
-    at_risk_count,
-    new_count,
-    ROUND((rows_processed::numeric / NULLIF(total_products, 0)) * 100, 2) as update_percentage
-FROM update_stats;

inventory-server/scripts/metrics-new/utils/db.js

@@ -1,39 +0,0 @@
-const { Pool } = require('pg');
-const path = require('path');
-require('dotenv').config({ path: path.resolve(__dirname, '../../..', '.env') });
-
-// Database configuration
-const dbConfig = {
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432,
-    ssl: process.env.DB_SSL === 'true',
-    // Add performance optimizations
-    max: 10, // connection pool max size
-    idleTimeoutMillis: 30000,
-    connectionTimeoutMillis: 60000
-};
-
-// Create a single pool instance to be reused
-const pool = new Pool(dbConfig);
-
-// Add event handlers for pool
-pool.on('error', (err, client) => {
-    console.error('Unexpected error on idle client', err);
-});
-
-async function getConnection() {
-    return await pool.connect();
-}
-
-async function closePool() {
-    await pool.end();
-}
-
-module.exports = {
-    dbConfig,
-    getConnection,
-    closePool
-}; 

inventory-server/scripts/metrics-new/utils/progress.js

@@ -1,183 +0,0 @@
-const fs = require('fs');
-const path = require('path');
-
-// Helper function to format elapsed time
-function formatElapsedTime(startTime) {
-    let elapsed;
-    
-    // If startTime is a timestamp (number representing milliseconds since epoch)
-    if (typeof startTime === 'number') {
-        // Check if it's a timestamp (will be a large number like 1700000000000)
-        if (startTime > 1000000000) { // timestamps are in milliseconds since 1970
-            elapsed = Date.now() - startTime;
-        } else {
-            // Assume it's already elapsed milliseconds
-            elapsed = startTime;
-        }
-    } else if (startTime instanceof Date) {
-        elapsed = Date.now() - startTime.getTime();
-    } else {
-        // Default to 0 if invalid input
-        elapsed = 0;
-    }
-    
-    const seconds = Math.floor(elapsed / 1000);
-    const minutes = Math.floor(seconds / 60);
-    const hours = Math.floor(minutes / 60);
-    
-    if (hours > 0) {
-        return `${hours}h ${minutes % 60}m ${seconds % 60}s`;
-    } else if (minutes > 0) {
-        return `${minutes}m ${seconds % 60}s`;
-    } else {
-        return `${seconds}s`;
-    }
-}
-
-// Helper function to estimate remaining time
-function estimateRemaining(startTime, current, total) {
-    // Handle edge cases
-    if (!current || current === 0 || !total || total === 0 || current >= total) {
-        return null;
-    }
-    
-    // Calculate elapsed time in milliseconds
-    const elapsed = Date.now() - startTime;
-    if (elapsed <= 0) return null;
-    
-    // Calculate rate (items per millisecond)
-    const rate = current / elapsed;
-    if (rate <= 0) return null;
-    
-    // Calculate remaining time in milliseconds
-    const remaining = (total - current) / rate;
-    
-    // Convert to readable format
-    const seconds = Math.floor(remaining / 1000);
-    const minutes = Math.floor(seconds / 60);
-    const hours = Math.floor(minutes / 60);
-    
-    if (hours > 0) {
-        return `${hours}h ${minutes % 60}m`;
-    } else if (minutes > 0) {
-        return `${minutes}m ${seconds % 60}s`;
-    } else {
-        return `${seconds}s`;
-    }
-}
-
-// Helper function to calculate rate
-function calculateRate(startTime, current) {
-    const elapsed = (Date.now() - startTime) / 1000; // Convert to seconds
-    return elapsed > 0 ? Math.round(current / elapsed) : 0;
-}
-
-// Set up logging
-const LOG_DIR = path.join(__dirname, '../../../logs');
-const ERROR_LOG = path.join(LOG_DIR, 'import-errors.log');
-const IMPORT_LOG = path.join(LOG_DIR, 'import.log');
-const STATUS_FILE = path.join(LOG_DIR, 'metrics-status.json');
-
-// Ensure log directory exists
-if (!fs.existsSync(LOG_DIR)) {
-    fs.mkdirSync(LOG_DIR, { recursive: true });
-}
-
-// Helper function to log errors
-function logError(error, context = '') {
-    const timestamp = new Date().toISOString();
-    const errorMessage = `[${timestamp}] ${context}\nError: ${error.message}\nStack: ${error.stack}\n\n`;
-    
-    // Log to error file
-    fs.appendFileSync(ERROR_LOG, errorMessage);
-    
-    // Also log to console
-    console.error(`\n${context}\nError: ${error.message}`);
-}
-
-// Helper function to log import progress
-function logImport(message) {
-    const timestamp = new Date().toISOString();
-    const logMessage = `[${timestamp}] ${message}\n`;
-    fs.appendFileSync(IMPORT_LOG, logMessage);
-}
-
-// Helper function to output progress
-function outputProgress(data) {
-    // Save progress to file for resumption
-    saveProgress(data);
-    // Format as SSE event
-    const event = {
-        progress: data
-    };
-    // Always send to stdout for frontend
-    process.stdout.write(JSON.stringify(event) + '\n');
-    
-    // Log significant events to disk
-    const isSignificant = 
-        // Operation starts
-        (data.operation && !data.current) ||
-        // Operation completions and errors
-        data.status === 'complete' || 
-        data.status === 'error' ||
-        // Major phase changes
-        data.operation?.includes('Starting ABC classification') ||
-        data.operation?.includes('Starting time-based aggregates') ||
-        data.operation?.includes('Starting vendor metrics');
-    
-    if (isSignificant) {
-        logImport(`${data.operation || 'Operation'}${data.message ? ': ' + data.message : ''}${data.error ? ' Error: ' + data.error : ''}${data.status ? ' Status: ' + data.status : ''}`);
-    }
-}
-
-function saveProgress(progress) {
-    try {
-        fs.writeFileSync(STATUS_FILE, JSON.stringify({
-            ...progress,
-            timestamp: Date.now()
-        }));
-    } catch (err) {
-        console.error('Failed to save progress:', err);
-    }
-}
-
-function clearProgress() {
-    try {
-        if (fs.existsSync(STATUS_FILE)) {
-            fs.unlinkSync(STATUS_FILE);
-        }
-    } catch (err) {
-        console.error('Failed to clear progress:', err);
-    }
-}
-
-function getProgress() {
-    try {
-        if (fs.existsSync(STATUS_FILE)) {
-            const progress = JSON.parse(fs.readFileSync(STATUS_FILE, 'utf8'));
-            // Check if the progress is still valid (less than 1 hour old)
-            if (progress.timestamp && Date.now() - progress.timestamp < 3600000) {
-                return progress;
-            } else {
-                // Clear old progress
-                clearProgress();
-            }
-        }
-    } catch (err) {
-        console.error('Failed to read progress:', err);
-        clearProgress();
-    }
-    return null;
-}
-
-module.exports = {
-    formatElapsedTime,
-    estimateRemaining,
-    calculateRate,
-    logError,
-    logImport,
-    outputProgress,
-    saveProgress,
-    clearProgress,
-    getProgress
-}; 

inventory-server/scripts/reset-db.js

@@ -1,599 +0,0 @@
-const { Client } = require('pg');
-const path = require('path');
-const dotenv = require('dotenv');
-const fs = require('fs');
-
-dotenv.config({ path: path.join(__dirname, '../.env') });
-
-const dbConfig = {
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432
-};
-
-// Tables to always protect from being dropped
-const PROTECTED_TABLES = [
-    'users', 
-    'permissions', 
-    'user_permissions', 
-    'calculate_history', 
-    'import_history', 
-    'ai_prompts', 
-    'ai_validation_performance', 
-    'templates', 
-    'reusable_images',
-    'imported_daily_inventory',
-    'imported_product_stat_history',
-    'imported_product_current_prices'
-];
-
-// Helper function to output progress in JSON format
-function outputProgress(data) {
-    if (!data.status) {
-        data = {
-            status: 'running',
-            ...data
-        };
-    }
-    console.log(JSON.stringify(data));
-}
-
-// Core tables that must be created
-const CORE_TABLES = [
-    'products',
-    'orders',
-    'purchase_orders',
-    'categories',
-    'product_categories'
-];
-
-// Split SQL into individual statements
-function splitSQLStatements(sql) {
-    // First, normalize line endings
-    sql = sql.replace(/\r\n/g, '\n');
-    
-    // Track statement boundaries
-    let statements = [];
-    let currentStatement = '';
-    let inString = false;
-    let stringChar = '';
-    let inDollarQuote = false;
-    let dollarQuoteTag = '';
-    
-    // Process character by character
-    for (let i = 0; i < sql.length; i++) {
-        const char = sql[i];
-        const nextChar = sql[i + 1] || '';
-        
-        // Handle dollar quotes
-        if (char === '$' && !inString) {
-            // Look ahead to find the dollar quote tag
-            let tag = '$';
-            let j = i + 1;
-            while (j < sql.length && sql[j] !== '$') {
-                tag += sql[j];
-                j++;
-            }
-            tag += '$';
-            
-            if (j < sql.length) { // Found closing $
-                if (!inDollarQuote) {
-                    inDollarQuote = true;
-                    dollarQuoteTag = tag;
-                    currentStatement += tag;
-                    i = j;
-                    continue;
-                } else if (sql.substring(i, j + 1) === dollarQuoteTag) {
-                    inDollarQuote = false;
-                    dollarQuoteTag = '';
-                    currentStatement += tag;
-                    i = j;
-                    continue;
-                }
-            }
-        }
-        
-        // Handle string literals (only if not in dollar quote)
-        if (!inDollarQuote && (char === "'" || char === '"') && sql[i - 1] !== '\\') {
-            if (!inString) {
-                inString = true;
-                stringChar = char;
-            } else if (char === stringChar) {
-                inString = false;
-            }
-        }
-        
-        // Handle comments (only if not in string or dollar quote)
-        if (!inString && !inDollarQuote) {
-            if (char === '-' && nextChar === '-') {
-                // Skip to end of line
-                while (i < sql.length && sql[i] !== '\n') i++;
-                continue;
-            }
-            
-            if (char === '/' && nextChar === '*') {
-                // Skip until closing */
-                i += 2;
-                while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
-                i++; // Skip the closing /
-                continue;
-            }
-        }
-        
-        // Handle statement boundaries (only if not in string or dollar quote)
-        if (!inString && !inDollarQuote && char === ';') {
-            if (currentStatement.trim()) {
-                statements.push(currentStatement.trim());
-            }
-            currentStatement = '';
-        } else {
-            currentStatement += char;
-        }
-    }
-    
-    // Add the last statement if it exists
-    if (currentStatement.trim()) {
-        statements.push(currentStatement.trim());
-    }
-    
-    return statements;
-}
-
-async function resetDatabase() {
-    outputProgress({
-        operation: 'Starting database reset',
-        message: 'Connecting to database...'
-    });
-
-    // Debug: Log current directory and file paths
-    outputProgress({
-        operation: 'Debug paths',
-        message: {
-            currentDir: process.cwd(),
-            __dirname: __dirname,
-            schemaPath: path.join(__dirname, '../db/schema.sql')
-        }
-    });
-
-    const client = new Client(dbConfig);
-    await client.connect();
-    
-    try {
-        // Check PostgreSQL version and user
-        outputProgress({
-            operation: 'Checking database',
-            message: 'Verifying PostgreSQL version and user privileges...'
-        });
-
-        const versionResult = await client.query('SELECT version()');
-        const userResult = await client.query('SELECT current_user, current_database()');
-        
-        outputProgress({
-            operation: 'Database info',
-            message: {
-                version: versionResult.rows[0].version,
-                user: userResult.rows[0].current_user,
-                database: userResult.rows[0].current_database
-            }
-        });
-
-        // Get list of all tables in the current database
-        outputProgress({
-            operation: 'Getting table list',
-            message: 'Retrieving all table names...'
-        });
-
-        const tablesResult = await client.query(`
-            SELECT string_agg(tablename, ', ') as tables
-            FROM pg_tables
-            WHERE schemaname = 'public'
-            AND tablename NOT IN (SELECT unnest($1::text[]));
-        `, [PROTECTED_TABLES]);
-
-        if (!tablesResult.rows[0].tables) {
-            outputProgress({
-                operation: 'No tables found',
-                message: 'Database is already empty'
-            });
-        } else {
-            outputProgress({
-                operation: 'Dropping tables',
-                message: 'Dropping all existing tables...'
-            });
-
-            // Disable triggers/foreign key checks
-            await client.query('SET session_replication_role = \'replica\';');
-            
-            // Drop all tables except users
-            const tables = tablesResult.rows[0].tables.split(', ');
-            for (const table of tables) {
-                if (!PROTECTED_TABLES.includes(table)) {
-                    await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
-                }
-            }
-
-            // Only drop types if we're not preserving history tables
-            const historyTablesExist = await client.query(`
-                SELECT EXISTS (
-                    SELECT FROM pg_tables 
-                    WHERE schemaname = 'public' 
-                    AND tablename IN ('calculate_history', 'import_history')
-                );
-            `);
-
-            if (!historyTablesExist.rows[0].exists) {
-                await client.query('DROP TYPE IF EXISTS calculation_status CASCADE;');
-                await client.query('DROP TYPE IF EXISTS module_name CASCADE;');
-            }
-            
-            // Re-enable triggers/foreign key checks
-            await client.query('SET session_replication_role = \'origin\';');
-        }
-
-        // Create enum types if they don't exist
-        outputProgress({
-            operation: 'Creating enum types',
-            message: 'Setting up required enum types...'
-        });
-
-        // Check if types exist before creating
-        const typesExist = await client.query(`
-            SELECT EXISTS (
-                SELECT 1 FROM pg_type 
-                WHERE typname = 'calculation_status'
-            ) as calc_status_exists,
-            EXISTS (
-                SELECT 1 FROM pg_type 
-                WHERE typname = 'module_name'
-            ) as module_name_exists;
-        `);
-
-        if (!typesExist.rows[0].calc_status_exists) {
-            await client.query(`CREATE TYPE calculation_status AS ENUM ('running', 'completed', 'failed', 'cancelled')`);
-        }
-
-        if (!typesExist.rows[0].module_name_exists) {
-            await client.query(`
-                CREATE TYPE module_name AS ENUM (
-                    'product_metrics',
-                    'time_aggregates',
-                    'financial_metrics',
-                    'vendor_metrics',
-                    'category_metrics',
-                    'brand_metrics',
-                    'sales_forecasts',
-                    'abc_classification',
-                    'daily_snapshots',
-                    'periodic_metrics'
-                )
-            `);
-        }
-
-        // Read and execute main schema first (core tables)
-        outputProgress({
-            operation: 'Running database setup',
-            message: 'Creating core tables...'
-        });
-        const schemaPath = path.join(__dirname, '../db/schema.sql');
-        
-        // Verify file exists
-        if (!fs.existsSync(schemaPath)) {
-            throw new Error(`Schema file not found at: ${schemaPath}`);
-        }
-
-        const schemaSQL = fs.readFileSync(schemaPath, 'utf8');
-        
-        outputProgress({
-            operation: 'Schema file',
-            message: {
-                path: schemaPath,
-                exists: fs.existsSync(schemaPath),
-                size: fs.statSync(schemaPath).size,
-                firstFewLines: schemaSQL.split('\n').slice(0, 5).join('\n')
-            }
-        });
-
-        // Execute schema statements one at a time
-        const statements = splitSQLStatements(schemaSQL);
-        outputProgress({
-            operation: 'SQL Execution',
-            message: {
-                totalStatements: statements.length,
-                statements: statements.map((stmt, i) => ({
-                    number: i + 1,
-                    preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : '')
-                }))
-            }
-        });
-
-        // Start a transaction for better error handling
-        await client.query('BEGIN');
-        try {
-            for (let i = 0; i < statements.length; i++) {
-                const stmt = statements[i];
-                try {
-                    const result = await client.query(stmt);
-                    
-                    // Verify if table was created (if this was a CREATE TABLE statement)
-                    if (stmt.trim().toLowerCase().startsWith('create table')) {
-                        const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(\w+)["]?/i)?.[1];
-                        if (tableName) {
-                            const tableExists = await client.query(`
-                                SELECT COUNT(*) as count 
-                                FROM information_schema.tables 
-                                WHERE table_schema = 'public' 
-                                AND table_name = $1
-                            `, [tableName]);
-                            
-                            outputProgress({
-                                operation: 'Table Creation Verification',
-                                message: {
-                                    table: tableName,
-                                    exists: tableExists.rows[0].count > 0
-                                }
-                            });
-                        }
-                    }
-                    
-                    outputProgress({
-                        operation: 'SQL Progress',
-                        message: {
-                            statement: i + 1,
-                            total: statements.length,
-                            preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                            rowCount: result.rowCount
-                        }
-                    });
-                    
-                    // Commit in chunks of 10 statements to avoid long-running transactions
-                    if (i > 0 && i % 10 === 0) {
-                        await client.query('COMMIT');
-                        await client.query('BEGIN');
-                    }
-                } catch (sqlError) {
-                    await client.query('ROLLBACK');
-                    outputProgress({
-                        status: 'error',
-                        operation: 'SQL Error',
-                        error: sqlError.message,
-                        statement: stmt,
-                        statementNumber: i + 1
-                    });
-                    throw sqlError;
-                }
-            }
-            // Commit the final transaction
-            await client.query('COMMIT');
-        } catch (error) {
-            await client.query('ROLLBACK');
-            throw error;
-        }
-
-        // Verify core tables were created
-        const existingTables = (await client.query(`
-            SELECT table_name
-            FROM information_schema.tables 
-            WHERE table_schema = 'public'
-        `)).rows.map(t => t.table_name);
-
-        outputProgress({
-            operation: 'Core tables verification',
-            message: {
-                found: existingTables,
-                expected: CORE_TABLES
-            }
-        });
-
-        const missingCoreTables = CORE_TABLES.filter(
-            t => !existingTables.includes(t)
-        );
-
-        if (missingCoreTables.length > 0) {
-            throw new Error(
-                `Failed to create core tables: ${missingCoreTables.join(', ')}`
-            );
-        }
-
-        outputProgress({
-            operation: 'Core tables created',
-            message: `Successfully created tables: ${CORE_TABLES.join(', ')}`
-        });
-
-        // Now read and execute config schema (since core tables exist)
-        outputProgress({
-            operation: 'Running config setup',
-            message: 'Creating configuration tables...'
-        });
-        const configSchemaPath = path.join(__dirname, '../db/config-schema-new.sql');
-        
-        // Verify file exists
-        if (!fs.existsSync(configSchemaPath)) {
-            throw new Error(`Config schema file not found at: ${configSchemaPath}`);
-        }
-        
-        const configSchemaSQL = fs.readFileSync(configSchemaPath, 'utf8');
-        
-        outputProgress({
-            operation: 'Config Schema file',
-            message: {
-                path: configSchemaPath,
-                exists: fs.existsSync(configSchemaPath),
-                size: fs.statSync(configSchemaPath).size,
-                firstFewLines: configSchemaSQL.split('\n').slice(0, 5).join('\n')
-            }
-        });
-
-        // Execute config schema statements one at a time
-        const configStatements = splitSQLStatements(configSchemaSQL);
-        outputProgress({
-            operation: 'Config SQL Execution',
-            message: {
-                totalStatements: configStatements.length,
-                statements: configStatements.map((stmt, i) => ({
-                    number: i + 1,
-                    preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : '')
-                }))
-            }
-        });
-
-        // Start a transaction for better error handling
-        await client.query('BEGIN');
-        try {
-            for (let i = 0; i < configStatements.length; i++) {
-                const stmt = configStatements[i];
-                try {
-                    const result = await client.query(stmt);
-                    
-                    outputProgress({
-                        operation: 'Config SQL Progress',
-                        message: {
-                            statement: i + 1,
-                            total: configStatements.length,
-                            preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                            rowCount: result.rowCount
-                        }
-                    });
-                    
-                    // Commit in chunks of 10 statements to avoid long-running transactions
-                    if (i > 0 && i % 10 === 0) {
-                        await client.query('COMMIT');
-                        await client.query('BEGIN');
-                    }
-                } catch (sqlError) {
-                    await client.query('ROLLBACK');
-                    outputProgress({
-                        status: 'error',
-                        operation: 'Config SQL Error',
-                        error: sqlError.message,
-                        statement: stmt,
-                        statementNumber: i + 1
-                    });
-                    throw sqlError;
-                }
-            }
-            // Commit the final transaction
-            await client.query('COMMIT');
-        } catch (error) {
-            await client.query('ROLLBACK');
-            throw error;
-        }
-
-        // Read and execute metrics schema (metrics tables)
-        outputProgress({
-            operation: 'Running metrics setup',
-            message: 'Creating metrics tables...'
-        });
-        const metricsSchemaPath = path.join(__dirname, '../db/metrics-schema-new.sql');
-        
-        // Verify file exists
-        if (!fs.existsSync(metricsSchemaPath)) {
-            throw new Error(`Metrics schema file not found at: ${metricsSchemaPath}`);
-        }
-        
-        const metricsSchemaSQL = fs.readFileSync(metricsSchemaPath, 'utf8');
-        
-        outputProgress({
-            operation: 'Metrics Schema file',
-            message: {
-                path: metricsSchemaPath,
-                exists: fs.existsSync(metricsSchemaPath),
-                size: fs.statSync(metricsSchemaPath).size,
-                firstFewLines: metricsSchemaSQL.split('\n').slice(0, 5).join('\n')
-            }
-        });
-
-        // Execute metrics schema statements one at a time
-        const metricsStatements = splitSQLStatements(metricsSchemaSQL);
-        outputProgress({
-            operation: 'Metrics SQL Execution',
-            message: {
-                totalStatements: metricsStatements.length,
-                statements: metricsStatements.map((stmt, i) => ({
-                    number: i + 1,
-                    preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : '')
-                }))
-            }
-        });
-
-        // Start a transaction for better error handling
-        await client.query('BEGIN');
-        try {
-            for (let i = 0; i < metricsStatements.length; i++) {
-                const stmt = metricsStatements[i];
-                try {
-                    const result = await client.query(stmt);
-                    
-                    outputProgress({
-                        operation: 'Metrics SQL Progress',
-                        message: {
-                            statement: i + 1,
-                            total: metricsStatements.length,
-                            preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                            rowCount: result.rowCount
-                        }
-                    });
-                    
-                    // Commit in chunks of 10 statements to avoid long-running transactions
-                    if (i > 0 && i % 10 === 0) {
-                        await client.query('COMMIT');
-                        await client.query('BEGIN');
-                    }
-                } catch (sqlError) {
-                    await client.query('ROLLBACK');
-                    outputProgress({
-                        status: 'error',
-                        operation: 'Metrics SQL Error',
-                        error: sqlError.message,
-                        statement: stmt,
-                        statementNumber: i + 1
-                    });
-                    throw sqlError;
-                }
-            }
-            // Commit the final transaction
-            await client.query('COMMIT');
-        } catch (error) {
-            await client.query('ROLLBACK');
-            throw error;
-        }
-
-        outputProgress({
-            status: 'complete',
-            operation: 'Database reset complete',
-            message: 'Database has been reset and all tables recreated'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'Failed to reset database',
-            error: error.message,
-            stack: error.stack
-        });
-        process.exit(1);
-    } finally {
-        // Make sure to re-enable foreign key checks if they were disabled
-        try {
-            await client.query('SET session_replication_role = \'origin\'');
-        } catch (e) {
-            console.error('Error re-enabling foreign key checks:', e.message);
-        }
-        
-        // Close the database connection
-        await client.end();
-    }
-}
-
-// Export if required as a module
-if (typeof module !== 'undefined' && module.exports) {
-    module.exports = resetDatabase;
-}
-
-// Run if called directly
-if (require.main === module) {
-    resetDatabase().catch(error => {
-        console.error('Error:', error);
-        process.exit(1);
-    });
-} 

inventory-server/scripts/reset-metrics-new.js

@@ -1,384 +0,0 @@
-const { Client } = require('pg');
-const path = require('path');
-const fs = require('fs');
-require('dotenv').config({ path: path.resolve(__dirname, '../.env') });
-
-const dbConfig = {
-    host: process.env.DB_HOST,
-    user: process.env.DB_USER,
-    password: process.env.DB_PASSWORD,
-    database: process.env.DB_NAME,
-    port: process.env.DB_PORT || 5432
-};
-
-function outputProgress(data) {
-    if (!data.status) {
-        data = {
-            status: 'running',
-            ...data
-        };
-    }
-    console.log(JSON.stringify(data));
-}
-
-// Tables to always protect from being dropped
-const PROTECTED_TABLES = [
-    'users', 
-    'permissions', 
-    'user_permissions', 
-    'calculate_history', 
-    'import_history', 
-    'ai_prompts', 
-    'ai_validation_performance', 
-    'templates', 
-    'reusable_images',
-    'imported_daily_inventory',
-    'imported_product_stat_history',
-    'imported_product_current_prices'
-];
-
-// Split SQL into individual statements
-function splitSQLStatements(sql) {
-    sql = sql.replace(/\r\n/g, '\n');
-    let statements = [];
-    let currentStatement = '';
-    let inString = false;
-    let stringChar = '';
-    
-    for (let i = 0; i < sql.length; i++) {
-        const char = sql[i];
-        const nextChar = sql[i + 1] || '';
-        
-        if ((char === "'" || char === '"') && sql[i - 1] !== '\\') {
-            if (!inString) {
-                inString = true;
-                stringChar = char;
-            } else if (char === stringChar) {
-                inString = false;
-            }
-        }
-        
-        if (!inString && char === '-' && nextChar === '-') {
-            while (i < sql.length && sql[i] !== '\n') i++;
-            continue;
-        }
-        
-        if (!inString && char === '/' && nextChar === '*') {
-            i += 2;
-            while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
-            i++;
-            continue;
-        }
-        
-        if (!inString && char === ';') {
-            if (currentStatement.trim()) {
-                statements.push(currentStatement.trim());
-            }
-            currentStatement = '';
-        } else {
-            currentStatement += char;
-        }
-    }
-    
-    if (currentStatement.trim()) {
-        statements.push(currentStatement.trim());
-    }
-    
-    return statements;
-}
-
-async function resetMetrics() {
-    let client;
-    try {
-        outputProgress({
-            operation: 'Starting metrics reset',
-            message: 'Connecting to database...'
-        });
-
-        client = new Client(dbConfig);
-        await client.connect();
-        
-        // Get metrics tables from the schema file by looking for CREATE TABLE statements
-        const schemaPath = path.resolve(__dirname, '../db/metrics-schema-new.sql');
-        if (!fs.existsSync(schemaPath)) {
-            throw new Error(`Schema file not found at: ${schemaPath}`);
-        }
-        
-        const schemaSQL = fs.readFileSync(schemaPath, 'utf8');
-        const createTableRegex = /create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(?:public\.)?(\w+)["]?/gi;
-        let metricsTables = [];
-        let match;
-        
-        while ((match = createTableRegex.exec(schemaSQL)) !== null) {
-            if (match[1] && !PROTECTED_TABLES.includes(match[1])) {
-                metricsTables.push(match[1]);
-            }
-        }
-        
-        if (metricsTables.length === 0) {
-            throw new Error('No tables found in the schema file');
-        }
-        
-        outputProgress({
-            operation: 'Schema analysis',
-            message: `Found ${metricsTables.length} metrics tables in schema: ${metricsTables.join(', ')}`
-        });
-
-        // Explicitly begin a transaction
-        await client.query('BEGIN');
-
-        // First verify current state
-        const initialTables = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-            AND tablename NOT IN (SELECT unnest($2::text[]))
-        `, [metricsTables, PROTECTED_TABLES]);
-
-        outputProgress({
-            operation: 'Initial state',
-            message: `Found ${initialTables.rows.length} existing metrics tables: ${initialTables.rows.map(t => t.name).join(', ')}`
-        });
-
-        // Disable foreign key checks at the start
-        await client.query('SET session_replication_role = \'replica\'');
-
-        // Drop all metrics tables in reverse order to handle dependencies
-        outputProgress({
-            operation: 'Dropping metrics tables',
-            message: 'Removing existing metrics tables...'
-        });
-
-        // Reverse the array to handle dependencies properly
-        for (const table of [...metricsTables].reverse()) {
-            // Skip protected tables (redundant check)
-            if (PROTECTED_TABLES.includes(table)) {
-                outputProgress({
-                    operation: 'Protected table',
-                    message: `Skipping protected table: ${table}`
-                });
-                continue;
-            }
-            
-            try {
-                // Use NOWAIT to avoid hanging if there's a lock
-                await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
-                
-                // Verify the table was actually dropped
-                const checkDrop = await client.query(`
-                    SELECT COUNT(*) as count 
-                    FROM pg_tables 
-                    WHERE schemaname = 'public' 
-                    AND tablename = $1
-                `, [table]);
-                
-                if (parseInt(checkDrop.rows[0].count) > 0) {
-                    throw new Error(`Failed to drop table ${table} - table still exists`);
-                }
-
-                outputProgress({
-                    operation: 'Table dropped',
-                    message: `Successfully dropped table: ${table}`
-                });
-                
-                // Commit after each table drop to ensure locks are released
-                await client.query('COMMIT');
-                // Start a new transaction for the next table
-                await client.query('BEGIN');
-                // Re-disable foreign key constraints for the new transaction
-                await client.query('SET session_replication_role = \'replica\'');
-            } catch (err) {
-                outputProgress({
-                    status: 'error',
-                    operation: 'Drop table error',
-                    message: `Error dropping table ${table}: ${err.message}`
-                });
-                await client.query('ROLLBACK');
-                // Re-start transaction for next table
-                await client.query('BEGIN');
-                await client.query('SET session_replication_role = \'replica\'');
-            }
-        }
-
-        // Verify all tables were dropped
-        const afterDrop = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-        `, [metricsTables]);
-
-        if (afterDrop.rows.length > 0) {
-            throw new Error(`Failed to drop all tables. Remaining tables: ${afterDrop.rows.map(t => t.name).join(', ')}`);
-        }
-
-        // Make sure we have a fresh transaction here
-        await client.query('COMMIT');
-        await client.query('BEGIN');
-        await client.query('SET session_replication_role = \'replica\'');
-
-        // Read metrics schema
-        outputProgress({
-            operation: 'Reading schema',
-            message: 'Loading metrics schema file...'
-        });
-
-        const statements = splitSQLStatements(schemaSQL);
-
-        outputProgress({
-            operation: 'Schema loaded',
-            message: `Found ${statements.length} SQL statements to execute`
-        });
-
-        // Execute schema statements
-        for (let i = 0; i < statements.length; i++) {
-            const stmt = statements[i];
-            try {
-                const result = await client.query(stmt);
-
-                // If this is a CREATE TABLE statement, verify the table was created
-                if (stmt.trim().toLowerCase().startsWith('create table')) {
-                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(?:public\.)?(\w+)["]?/i)?.[1];
-                    if (tableName) {
-                        const checkCreate = await client.query(`
-                            SELECT tablename as name
-                            FROM pg_tables 
-                            WHERE schemaname = 'public' 
-                            AND tablename = $1
-                        `, [tableName]);
-                        
-                        if (checkCreate.rows.length === 0) {
-                            throw new Error(`Failed to create table ${tableName} - table does not exist after CREATE statement`);
-                        }
-
-                        outputProgress({
-                            operation: 'Table created',
-                            message: `Successfully created table: ${tableName}`
-                        });
-                    }
-                }
-                
-                outputProgress({
-                    operation: 'SQL Progress',
-                    message: {
-                        statement: i + 1,
-                        total: statements.length,
-                        preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                        rowCount: result.rowCount
-                    }
-                });
-                
-                // Commit every 10 statements to avoid long-running transactions
-                if (i > 0 && i % 10 === 0) {
-                    await client.query('COMMIT');
-                    await client.query('BEGIN');
-                    await client.query('SET session_replication_role = \'replica\'');
-                }
-            } catch (sqlError) {
-                outputProgress({
-                    status: 'error',
-                    operation: 'SQL Error',
-                    message: {
-                        error: sqlError.message,
-                        statement: stmt,
-                        statementNumber: i + 1
-                    }
-                });
-                await client.query('ROLLBACK');
-                throw sqlError;
-            }
-        }
-
-        // Final commit for any pending statements
-        await client.query('COMMIT');
-        
-        // Start new transaction for final checks
-        await client.query('BEGIN');
-
-        // Re-enable foreign key checks after all tables are created
-        await client.query('SET session_replication_role = \'origin\'');
-
-        // Verify metrics tables were created
-        outputProgress({
-            operation: 'Verifying metrics tables',
-            message: 'Checking all metrics tables were created...'
-        });
-
-        const metricsTablesResult = await client.query(`
-            SELECT tablename as name
-            FROM pg_tables 
-            WHERE schemaname = 'public'
-            AND tablename = ANY($1)
-        `, [metricsTables]);
-
-        outputProgress({
-            operation: 'Tables found',
-            message: `Found ${metricsTablesResult.rows.length} tables: ${metricsTablesResult.rows.map(t => t.name).join(', ')}`
-        });
-
-        const existingMetricsTables = metricsTablesResult.rows.map(t => t.name);
-        const missingMetricsTables = metricsTables.filter(t => !existingMetricsTables.includes(t));
-
-        if (missingMetricsTables.length > 0) {
-            // Do one final check of the actual tables
-            const finalCheck = await client.query(`
-                SELECT tablename as name
-                FROM pg_tables 
-                WHERE schemaname = 'public'
-            `);
-            outputProgress({
-                operation: 'Final table check',
-                message: `All database tables: ${finalCheck.rows.map(t => t.name).join(', ')}`
-            });
-            await client.query('ROLLBACK');
-            throw new Error(`Failed to create metrics tables: ${missingMetricsTables.join(', ')}`);
-        }
-
-        // Commit final transaction
-        await client.query('COMMIT');
-        
-        outputProgress({
-            status: 'complete',
-            operation: 'Reset complete',
-            message: 'All metrics tables have been reset successfully'
-        });
-    } catch (error) {
-        outputProgress({
-            status: 'error',
-            operation: 'Reset failed',
-            message: error.message,
-            stack: error.stack
-        });
-        
-        if (client) {
-            try {
-                await client.query('ROLLBACK');
-            } catch (rollbackError) {
-                console.error('Error during rollback:', rollbackError);
-            }
-            // Make sure to re-enable foreign key checks even if there's an error
-            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
-        }
-        throw error;
-    } finally {
-        if (client) {
-            // One final attempt to ensure foreign key checks are enabled
-            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
-            await client.end();
-        }
-    }
-}
-
-// Export if required as a module
-if (typeof module !== 'undefined' && module.exports) {
-    module.exports = resetMetrics;
-}
-
-// Run if called from command line
-if (require.main === module) {
-    resetMetrics().catch(error => {
-        console.error('Error:', error);
-        process.exit(1);
-    });
-} 

inventory-server/src/middleware/cors.js

@@ -5,6 +5,8 @@ const corsMiddleware = cors({
   origin: [
     'https://inventory.kent.pw',
     'http://localhost:5175',
+    'https://tools.acherryontop.com',
+    'https://tools.acherryontop.com',
     /^http:\/\/192\.168\.\d+\.\d+(:\d+)?$/,
     /^http:\/\/10\.\d+\.\d+\.\d+(:\d+)?$/
   ],
@@ -26,7 +28,7 @@ const corsErrorHandler = (err, req, res, next) => {
     res.status(403).json({
       error: 'CORS not allowed',
       origin: req.get('Origin'),
-      message: 'Origin not in allowed list: https://inventory.kent.pw, localhost:5175, 192.168.x.x, or 10.x.x.x'
+      message: 'Origin not in allowed list: https://inventory.kent.pw, https://tools.acherryontop.com, https://tools.acherryontop.com, localhost:5175, 192.168.x.x, or 10.x.x.x'
     });
   } else {
     next(err);

inventory-server/src/routes/ai-prompts.js

@@ -1,319 +0,0 @@
-const express = require('express');
-const router = express.Router();
-
-// Get all AI prompts
-router.get('/', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM ai_prompts 
-      ORDER BY prompt_type ASC, company ASC
-    `);
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error fetching AI prompts:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch AI prompts',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get prompt by ID
-router.get('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM ai_prompts 
-      WHERE id = $1
-    `, [id]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'AI prompt not found' });
-    }
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error fetching AI prompt:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch AI prompt',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get prompt by type (general, system, company_specific)
-router.get('/by-type', async (req, res) => {
-  try {
-    const { type, company } = req.query;
-    const pool = req.app.locals.pool;
-    
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Validate prompt type
-    if (!type || !['general', 'system', 'company_specific'].includes(type)) {
-      return res.status(400).json({ 
-        error: 'Valid type query parameter is required (general, system, or company_specific)'
-      });
-    }
-    
-    // For company_specific type, company ID is required
-    if (type === 'company_specific' && !company) {
-      return res.status(400).json({
-        error: 'Company ID is required for company_specific prompt type'
-      });
-    }
-    
-    // For general and system types, company should not be provided
-    if ((type === 'general' || type === 'system') && company) {
-      return res.status(400).json({
-        error: 'Company ID should not be provided for general or system prompt types'
-      });
-    }
-    
-    // Build the query based on the type
-    let query, params;
-    if (type === 'company_specific') {
-      query = 'SELECT * FROM ai_prompts WHERE prompt_type = $1 AND company = $2';
-      params = [type, company];
-    } else {
-      query = 'SELECT * FROM ai_prompts WHERE prompt_type = $1';
-      params = [type];
-    }
-    
-    // Execute the query
-    const result = await pool.query(query, params);
-    
-    // Check if any prompt was found
-    if (result.rows.length === 0) {
-      let errorMessage;
-      if (type === 'company_specific') {
-        errorMessage = `AI prompt not found for company ${company}`;
-      } else {
-        errorMessage = `${type.charAt(0).toUpperCase() + type.slice(1)} AI prompt not found`;
-      }
-      return res.status(404).json({ error: errorMessage });
-    }
-    
-    // Return the first matching prompt
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error fetching AI prompt by type:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch AI prompt',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Create new AI prompt
-router.post('/', async (req, res) => {
-  try {
-    const {
-      prompt_text,
-      prompt_type,
-      company
-    } = req.body;
-
-    // Validate required fields
-    if (!prompt_text || !prompt_type) {
-      return res.status(400).json({ error: 'Prompt text and type are required' });
-    }
-    
-    // Validate prompt type
-    if (!['general', 'company_specific', 'system'].includes(prompt_type)) {
-      return res.status(400).json({ error: 'Prompt type must be either "general", "company_specific", or "system"' });
-    }
-    
-    // Validate company is provided for company-specific prompts
-    if (prompt_type === 'company_specific' && !company) {
-      return res.status(400).json({ error: 'Company is required for company-specific prompts' });
-    }
-
-    // Validate company is not provided for general or system prompts
-    if ((prompt_type === 'general' || prompt_type === 'system') && company) {
-      return res.status(400).json({ error: 'Company should not be provided for general or system prompts' });
-    }
-
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      INSERT INTO ai_prompts (
-        prompt_text,
-        prompt_type,
-        company
-      ) VALUES ($1, $2, $3)
-      RETURNING *
-    `, [
-      prompt_text,
-      prompt_type,
-      company
-    ]);
-
-    res.status(201).json(result.rows[0]);
-  } catch (error) {
-    console.error('Error creating AI prompt:', error);
-    
-    // Check for unique constraint violations
-    if (error instanceof Error && error.message.includes('unique constraint')) {
-      if (error.message.includes('unique_company_prompt')) {
-        return res.status(409).json({ 
-          error: 'A prompt already exists for this company',
-          details: error.message
-        });
-      } else if (error.message.includes('idx_unique_general_prompt')) {
-        return res.status(409).json({ 
-          error: 'A general prompt already exists',
-          details: error.message
-        });
-      } else if (error.message.includes('idx_unique_system_prompt')) {
-        return res.status(409).json({ 
-          error: 'A system prompt already exists',
-          details: error.message
-        });
-      }
-    }
-    
-    res.status(500).json({ 
-      error: 'Failed to create AI prompt',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Update AI prompt
-router.put('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const {
-      prompt_text,
-      prompt_type,
-      company
-    } = req.body;
-
-    // Validate required fields
-    if (!prompt_text || !prompt_type) {
-      return res.status(400).json({ error: 'Prompt text and type are required' });
-    }
-    
-    // Validate prompt type
-    if (!['general', 'company_specific', 'system'].includes(prompt_type)) {
-      return res.status(400).json({ error: 'Prompt type must be either "general", "company_specific", or "system"' });
-    }
-    
-    // Validate company is provided for company-specific prompts
-    if (prompt_type === 'company_specific' && !company) {
-      return res.status(400).json({ error: 'Company is required for company-specific prompts' });
-    }
-    
-    // Validate company is not provided for general or system prompts
-    if ((prompt_type === 'general' || prompt_type === 'system') && company) {
-      return res.status(400).json({ error: 'Company should not be provided for general or system prompts' });
-    }
-
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Check if the prompt exists
-    const checkResult = await pool.query('SELECT * FROM ai_prompts WHERE id = $1', [id]);
-    if (checkResult.rows.length === 0) {
-      return res.status(404).json({ error: 'AI prompt not found' });
-    }
-    
-    const result = await pool.query(`
-      UPDATE ai_prompts 
-      SET 
-        prompt_text = $1,
-        prompt_type = $2,
-        company = $3
-      WHERE id = $4
-      RETURNING *
-    `, [
-      prompt_text,
-      prompt_type,
-      company,
-      id
-    ]);
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error updating AI prompt:', error);
-    
-    // Check for unique constraint violations
-    if (error instanceof Error && error.message.includes('unique constraint')) {
-      if (error.message.includes('unique_company_prompt')) {
-        return res.status(409).json({ 
-          error: 'A prompt already exists for this company',
-          details: error.message
-        });
-      } else if (error.message.includes('idx_unique_general_prompt')) {
-        return res.status(409).json({ 
-          error: 'A general prompt already exists',
-          details: error.message
-        });
-      } else if (error.message.includes('idx_unique_system_prompt')) {
-        return res.status(409).json({ 
-          error: 'A system prompt already exists',
-          details: error.message
-        });
-      }
-    }
-    
-    res.status(500).json({ 
-      error: 'Failed to update AI prompt',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Delete AI prompt
-router.delete('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query('DELETE FROM ai_prompts WHERE id = $1 RETURNING *', [id]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'AI prompt not found' });
-    }
-
-    res.json({ message: 'AI prompt deleted successfully' });
-  } catch (error) {
-    console.error('Error deleting AI prompt:', error);
-    res.status(500).json({ 
-      error: 'Failed to delete AI prompt',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Error handling middleware
-router.use((err, req, res, next) => {
-  console.error('AI prompts route error:', err);
-  res.status(500).json({ 
-    error: 'Internal server error',
-    details: err.message
-  });
-});
-
-module.exports = router; 

inventory-server/src/routes/analytics.js

@@ -1,611 +1 @@
-const express = require('express');
-const router = express.Router();
-
-// Get overall analytics stats
-router.get('/stats', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    
-    const { rows: [results] } = await pool.query(`
-      WITH vendor_count AS (
-        SELECT COUNT(DISTINCT vendor_name) AS count
-        FROM vendor_metrics
-      ),
-      category_count AS (
-        SELECT COUNT(DISTINCT category_id) AS count
-        FROM category_metrics
-      ),
-      metrics_summary AS (
-        SELECT 
-          AVG(margin_30d) AS avg_profit_margin,
-          AVG(markup_30d) AS avg_markup,
-          AVG(stockturn_30d) AS avg_stock_turnover,
-          AVG(asp_30d) AS avg_order_value
-        FROM product_metrics
-        WHERE sales_30d > 0
-      )
-      SELECT
-        COALESCE(ms.avg_profit_margin, 0) AS profitMargin,
-        COALESCE(ms.avg_markup, 0) AS averageMarkup,
-        COALESCE(ms.avg_stock_turnover, 0) AS stockTurnoverRate,
-        COALESCE(vc.count, 0) AS vendorCount,
-        COALESCE(cc.count, 0) AS categoryCount,
-        COALESCE(ms.avg_order_value, 0) AS averageOrderValue
-      FROM metrics_summary ms
-      CROSS JOIN vendor_count vc
-      CROSS JOIN category_count cc
-    `);
-
-    // Ensure all values are numbers
-    const stats = {
-      profitMargin: Number(results.profitmargin) || 0,
-      averageMarkup: Number(results.averagemarkup) || 0,
-      stockTurnoverRate: Number(results.stockturnoverrate) || 0,
-      vendorCount: Number(results.vendorcount) || 0,
-      categoryCount: Number(results.categorycount) || 0,
-      averageOrderValue: Number(results.averageordervalue) || 0
-    };
-
-    res.json(stats);
-  } catch (error) {
-    console.error('Error fetching analytics stats:', error);
-    res.status(500).json({ error: 'Failed to fetch analytics stats' });
-  }
-});
-
-// Get profit analysis data
-router.get('/profit', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    
-    // Get profit margins by category with full path
-    const { rows: byCategory } = await pool.query(`
-      WITH RECURSIVE category_path AS (
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          c.name::text as path
-        FROM categories c
-        WHERE c.parent_id IS NULL
-        
-        UNION ALL
-        
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          (cp.path || ' > ' || c.name)::text
-        FROM categories c
-        JOIN category_path cp ON c.parent_id = cp.cat_id
-      )
-      SELECT 
-        cm.category_name as category,
-        COALESCE(cp.path, cm.category_name) as categorypath,
-        cm.avg_margin_30d as profitmargin,
-        cm.revenue_30d as revenue,
-        cm.cogs_30d as cost
-      FROM category_metrics cm
-      LEFT JOIN category_path cp ON cm.category_id = cp.cat_id
-      WHERE cm.revenue_30d > 0
-      ORDER BY cm.revenue_30d DESC
-      LIMIT 10
-    `);
-
-    // Get profit margin over time
-    const { rows: overTime } = await pool.query(`
-      WITH time_series AS (
-        SELECT
-          date_trunc('day', generate_series(
-            CURRENT_DATE - INTERVAL '30 days',
-            CURRENT_DATE,
-            '1 day'::interval
-          ))::date AS date
-      ),
-      daily_profits AS (
-        SELECT
-          snapshot_date as date,
-          SUM(net_revenue) as revenue,
-          SUM(cogs) as cost,
-          CASE
-            WHEN SUM(net_revenue) > 0 
-            THEN (SUM(net_revenue - cogs) / SUM(net_revenue)) * 100
-            ELSE 0
-          END as profit_margin
-        FROM daily_product_snapshots
-        WHERE snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
-        GROUP BY snapshot_date
-      )
-      SELECT
-        to_char(ts.date, 'YYYY-MM-DD') as date,
-        COALESCE(dp.profit_margin, 0) as profitmargin,
-        COALESCE(dp.revenue, 0) as revenue,
-        COALESCE(dp.cost, 0) as cost
-      FROM time_series ts
-      LEFT JOIN daily_profits dp ON ts.date = dp.date
-      ORDER BY ts.date
-    `);
-
-    // Get top performing products by profit margin
-    const { rows: topProducts } = await pool.query(`
-      WITH RECURSIVE category_path AS (
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          c.name::text as path
-        FROM categories c
-        WHERE c.parent_id IS NULL
-        
-        UNION ALL
-        
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          (cp.path || ' > ' || c.name)::text
-        FROM categories c
-        JOIN category_path cp ON c.parent_id = cp.cat_id
-      ),
-      product_categories AS (
-        SELECT 
-          pc.pid,
-          c.name as category,
-          COALESCE(cp.path, c.name) as categorypath
-        FROM product_categories pc
-        JOIN categories c ON pc.cat_id = c.cat_id
-        LEFT JOIN category_path cp ON c.cat_id = cp.cat_id
-      )
-      SELECT 
-        pm.title as product,
-        COALESCE(pc.category, 'Uncategorized') as category,
-        COALESCE(pc.categorypath, 'Uncategorized') as categorypath,
-        pm.margin_30d as profitmargin,
-        pm.revenue_30d as revenue,
-        pm.cogs_30d as cost
-      FROM product_metrics pm
-      LEFT JOIN product_categories pc ON pm.pid = pc.pid
-      WHERE pm.revenue_30d > 100
-      AND pm.margin_30d > 0
-      ORDER BY pm.margin_30d DESC
-      LIMIT 10
-    `);
-
-    res.json({ byCategory, overTime, topProducts });
-  } catch (error) {
-    console.error('Error fetching profit analysis:', error);
-    res.status(500).json({ error: 'Failed to fetch profit analysis' });
-  }
-});
-
-// Get vendor performance data
-router.get('/vendors', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    
-    // Set cache control headers to prevent 304
-    res.set({
-      'Cache-Control': 'no-cache, no-store, must-revalidate',
-      'Pragma': 'no-cache',
-      'Expires': '0'
-    });
-    
-    console.log('Fetching vendor performance data...');
-
-    // Get vendor performance metrics from the vendor_metrics table
-    const { rows: rawPerformance } = await pool.query(`
-      SELECT 
-        vendor_name as vendor,
-        revenue_30d as sales_volume,
-        avg_margin_30d as profit_margin,
-        COALESCE(
-          sales_30d / NULLIF(current_stock_units, 0),
-          0
-        ) as stock_turnover,
-        product_count,
-        -- Use actual growth metrics from the vendor_metrics table
-        sales_growth_30d_vs_prev as growth
-      FROM vendor_metrics
-      WHERE revenue_30d > 0
-      ORDER BY revenue_30d DESC
-      LIMIT 20
-    `);
-
-    // Format the performance data
-    const performance = rawPerformance.map(vendor => ({
-      vendor: vendor.vendor,
-      salesVolume: Number(vendor.sales_volume) || 0,
-      profitMargin: Number(vendor.profit_margin) || 0,
-      stockTurnover: Number(vendor.stock_turnover) || 0,
-      productCount: Number(vendor.product_count) || 0,
-      growth: Number(vendor.growth) || 0
-    }));
-
-    // Get vendor comparison metrics (sales per product vs margin)
-    const { rows: rawComparison } = await pool.query(`
-      SELECT 
-        vendor_name as vendor,
-        CASE 
-          WHEN active_product_count > 0 
-          THEN revenue_30d / active_product_count 
-          ELSE 0 
-        END as sales_per_product,
-        avg_margin_30d as average_margin,
-        product_count as size
-      FROM vendor_metrics
-      WHERE active_product_count > 0
-      ORDER BY sales_per_product DESC
-      LIMIT 10
-    `);
-
-    // Transform comparison data
-    const comparison = rawComparison.map(item => ({
-      vendor: item.vendor,
-      salesPerProduct: Number(item.sales_per_product) || 0,
-      averageMargin: Number(item.average_margin) || 0,
-      size: Number(item.size) || 0
-    }));
-
-    console.log('Performance data ready. Sending response...');
-
-    // Return complete structure that the front-end expects
-    res.json({ 
-      performance,
-      comparison,
-      // Add empty trends array to complete the structure
-      trends: []
-    });
-  } catch (error) {
-    console.error('Error fetching vendor performance:', error);
-    res.status(500).json({ error: 'Failed to fetch vendor performance data' });
-  }
-});
-
-// Get stock analysis data
-router.get('/stock', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    console.log('Fetching stock analysis data...');
-    
-    // Use the new metrics tables to get data
-    
-    // Get turnover by category
-    const { rows: turnoverByCategory } = await pool.query(`
-      WITH category_metrics_with_path AS (
-        WITH RECURSIVE category_path AS (
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            c.name::text as path
-          FROM categories c
-          WHERE c.parent_id IS NULL
-          
-          UNION ALL
-          
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            (cp.path || ' > ' || c.name)::text
-          FROM categories c
-          JOIN category_path cp ON c.parent_id = cp.cat_id
-        )
-        SELECT 
-          cm.category_id,
-          cm.category_name,
-          cp.path as category_path,
-          cm.current_stock_units,
-          cm.sales_30d,
-          cm.stock_turn_30d
-        FROM category_metrics cm
-        LEFT JOIN category_path cp ON cm.category_id = cp.cat_id
-        WHERE cm.sales_30d > 0
-      )
-      SELECT 
-        category_name as category,
-        COALESCE(stock_turn_30d, 0) as turnoverRate,
-        current_stock_units as averageStock,
-        sales_30d as totalSales
-      FROM category_metrics_with_path
-      ORDER BY stock_turn_30d DESC NULLS LAST
-      LIMIT 10
-    `);
-    
-    // Get stock levels over time (last 30 days)
-    const { rows: stockLevels } = await pool.query(`
-      WITH date_range AS (
-        SELECT generate_series(
-          CURRENT_DATE - INTERVAL '30 days',
-          CURRENT_DATE,
-          '1 day'::interval
-        )::date AS date
-      ),
-      daily_stock_counts AS (
-        SELECT
-          snapshot_date,
-          COUNT(DISTINCT pid) as total_products,
-          COUNT(DISTINCT CASE WHEN eod_stock_quantity > 5 THEN pid END) as in_stock,
-          COUNT(DISTINCT CASE WHEN eod_stock_quantity <= 5 AND eod_stock_quantity > 0 THEN pid END) as low_stock,
-          COUNT(DISTINCT CASE WHEN eod_stock_quantity = 0 THEN pid END) as out_of_stock
-        FROM daily_product_snapshots
-        WHERE snapshot_date >= CURRENT_DATE - INTERVAL '30 days'
-        GROUP BY snapshot_date
-      )
-      SELECT
-        to_char(dr.date, 'YYYY-MM-DD') as date,
-        COALESCE(dsc.in_stock, 0) as inStock,
-        COALESCE(dsc.low_stock, 0) as lowStock,
-        COALESCE(dsc.out_of_stock, 0) as outOfStock
-      FROM date_range dr
-      LEFT JOIN daily_stock_counts dsc ON dr.date = dsc.snapshot_date
-      ORDER BY dr.date
-    `);
-    
-    // Get critical items (products that need reordering)
-    const { rows: criticalItems } = await pool.query(`
-      SELECT
-        pm.title as product,
-        pm.sku as sku,
-        pm.current_stock as stockQuantity,
-        COALESCE(pm.config_safety_stock, 0) as reorderPoint,
-        COALESCE(pm.stockturn_30d, 0) as turnoverRate,
-        CASE
-          WHEN pm.sales_velocity_daily > 0 
-          THEN ROUND(pm.current_stock / pm.sales_velocity_daily)
-          ELSE 999
-        END as daysUntilStockout
-      FROM product_metrics pm
-      WHERE pm.is_visible = true
-        AND pm.is_replenishable = true
-        AND pm.sales_30d > 0
-        AND pm.current_stock <= pm.config_safety_stock * 2
-      ORDER BY 
-        CASE
-          WHEN pm.sales_velocity_daily > 0 
-          THEN pm.current_stock / pm.sales_velocity_daily
-          ELSE 999
-        END ASC,
-        pm.revenue_30d DESC
-      LIMIT 10
-    `);
-    
-    res.json({
-      turnoverByCategory,
-      stockLevels,
-      criticalItems
-    });
-  } catch (error) {
-    console.error('Error fetching stock analysis:', error);
-    res.status(500).json({ error: 'Failed to fetch stock analysis', details: error.message });
-  }
-});
-
-// Get price analysis data
-router.get('/pricing', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    
-    // Get price points analysis
-    const { rows: pricePoints } = await pool.query(`
-      SELECT 
-        CAST(p.price AS DECIMAL(15,3)) as price,
-        CAST(SUM(o.quantity) AS DECIMAL(15,3)) as salesVolume,
-        CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) as revenue,
-        c.name as category
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      JOIN product_categories pc ON p.pid = pc.pid
-      JOIN categories c ON pc.cat_id = c.cat_id
-      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
-      GROUP BY p.price, c.name
-      HAVING SUM(o.quantity) > 0
-      ORDER BY revenue DESC
-      LIMIT 50
-    `);
-
-    // Get price elasticity data (price changes vs demand)
-    const { rows: elasticity } = await pool.query(`
-      SELECT 
-        to_char(o.date, 'YYYY-MM-DD') as date,
-        CAST(AVG(o.price) AS DECIMAL(15,3)) as price,
-        CAST(SUM(o.quantity) AS DECIMAL(15,3)) as demand
-      FROM orders o
-      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
-      GROUP BY to_char(o.date, 'YYYY-MM-DD')
-      ORDER BY date
-    `);
-
-    // Get price optimization recommendations
-    const { rows: recommendations } = await pool.query(`
-      SELECT 
-        p.title as product,
-        CAST(p.price AS DECIMAL(15,3)) as currentPrice,
-        CAST(
-          ROUND(
-            CASE 
-              WHEN AVG(o.quantity) > 10 THEN p.price * 1.1
-              WHEN AVG(o.quantity) < 2 THEN p.price * 0.9
-              ELSE p.price
-            END, 2
-          ) AS DECIMAL(15,3)
-        ) as recommendedPrice,
-        CAST(
-          ROUND(
-            SUM(o.price * o.quantity) * 
-            CASE 
-              WHEN AVG(o.quantity) > 10 THEN 1.15
-              WHEN AVG(o.quantity) < 2 THEN 0.95
-              ELSE 1
-            END, 2
-          ) AS DECIMAL(15,3)
-        ) as potentialRevenue,
-        CASE 
-          WHEN AVG(o.quantity) > 10 THEN 85
-          WHEN AVG(o.quantity) < 2 THEN 75
-          ELSE 65
-        END as confidence
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
-      GROUP BY p.pid, p.price, p.title
-      HAVING ABS(
-        CAST(
-          ROUND(
-            CASE 
-              WHEN AVG(o.quantity) > 10 THEN p.price * 1.1
-              WHEN AVG(o.quantity) < 2 THEN p.price * 0.9
-              ELSE p.price
-            END, 2
-          ) AS DECIMAL(15,3)
-        ) - CAST(p.price AS DECIMAL(15,3))
-      ) > 0
-      ORDER BY 
-        CAST(
-          ROUND(
-            SUM(o.price * o.quantity) * 
-            CASE 
-              WHEN AVG(o.quantity) > 10 THEN 1.15
-              WHEN AVG(o.quantity) < 2 THEN 0.95
-              ELSE 1
-            END, 2
-          ) AS DECIMAL(15,3)
-        ) - CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) DESC
-      LIMIT 10
-    `);
-
-    res.json({ pricePoints, elasticity, recommendations });
-  } catch (error) {
-    console.error('Error fetching price analysis:', error);
-    res.status(500).json({ error: 'Failed to fetch price analysis' });
-  }
-});
-
-// Get category performance data
-router.get('/categories', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    
-    // Common CTE for category paths
-    const categoryPathCTE = `
-      WITH RECURSIVE category_path AS (
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          c.name::text as path
-        FROM categories c
-        WHERE c.parent_id IS NULL
-        
-        UNION ALL
-        
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          (cp.path || ' > ' || c.name)::text
-        FROM categories c
-        JOIN category_path cp ON c.parent_id = cp.cat_id
-      )
-    `;
-    
-    // Get category performance metrics with full path
-    const { rows: performance } = await pool.query(`
-      ${categoryPathCTE},
-      monthly_sales AS (
-        SELECT 
-          c.name,
-          cp.path,
-          SUM(CASE 
-            WHEN o.date >= CURRENT_DATE - INTERVAL '30 days' 
-            THEN o.price * o.quantity 
-            ELSE 0 
-          END) as current_month,
-          SUM(CASE 
-            WHEN o.date >= CURRENT_DATE - INTERVAL '60 days' 
-            AND o.date < CURRENT_DATE - INTERVAL '30 days'
-            THEN o.price * o.quantity 
-            ELSE 0 
-          END) as previous_month
-        FROM products p
-        LEFT JOIN orders o ON p.pid = o.pid
-        JOIN product_categories pc ON p.pid = pc.pid
-        JOIN categories c ON pc.cat_id = c.cat_id
-        JOIN category_path cp ON c.cat_id = cp.cat_id
-        WHERE o.date >= CURRENT_DATE - INTERVAL '60 days'
-        GROUP BY c.name, cp.path
-      )
-      SELECT 
-        c.name as category,
-        cp.path as categoryPath,
-        SUM(o.price * o.quantity) as revenue,
-        SUM(o.price * o.quantity - p.cost_price * o.quantity) as profit,
-        ROUND(
-          ((ms.current_month / NULLIF(ms.previous_month, 0)) - 1) * 100,
-          1
-        ) as growth,
-        COUNT(DISTINCT p.pid) as productCount
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      JOIN product_categories pc ON p.pid = pc.pid
-      JOIN categories c ON pc.cat_id = c.cat_id
-      JOIN category_path cp ON c.cat_id = cp.cat_id
-      LEFT JOIN monthly_sales ms ON c.name = ms.name AND cp.path = ms.path
-      WHERE o.date >= CURRENT_DATE - INTERVAL '60 days'
-      GROUP BY c.name, cp.path, ms.current_month, ms.previous_month
-      HAVING SUM(o.price * o.quantity) > 0
-      ORDER BY revenue DESC
-      LIMIT 10
-    `);
-
-    // Get category revenue distribution with full path
-    const { rows: distribution } = await pool.query(`
-      ${categoryPathCTE}
-      SELECT 
-        c.name as category,
-        cp.path as categoryPath,
-        SUM(o.price * o.quantity) as value
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      JOIN product_categories pc ON p.pid = pc.pid
-      JOIN categories c ON pc.cat_id = c.cat_id
-      JOIN category_path cp ON c.cat_id = cp.cat_id
-      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
-      GROUP BY c.name, cp.path
-      HAVING SUM(o.price * o.quantity) > 0
-      ORDER BY value DESC
-      LIMIT 6
-    `);
-
-    // Get category sales trends with full path
-    const { rows: trends } = await pool.query(`
-      ${categoryPathCTE}
-      SELECT 
-        c.name as category,
-        cp.path as categoryPath,
-        to_char(o.date, 'Mon YYYY') as month,
-        SUM(o.price * o.quantity) as sales
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      JOIN product_categories pc ON p.pid = pc.pid
-      JOIN categories c ON pc.cat_id = c.cat_id
-      JOIN category_path cp ON c.cat_id = cp.cat_id
-      WHERE o.date >= CURRENT_DATE - INTERVAL '6 months'
-      GROUP BY 
-        c.name,
-        cp.path,
-        to_char(o.date, 'Mon YYYY'),
-        to_char(o.date, 'YYYY-MM')
-      ORDER BY 
-        c.name,
-        to_char(o.date, 'YYYY-MM')
-    `);
-
-    res.json({ performance, distribution, trends });
-  } catch (error) {
-    console.error('Error fetching category performance:', error);
-    res.status(500).json({ error: 'Failed to fetch category performance' });
-  }
-});
-
-module.exports = router; 
+ 

inventory-server/src/routes/brandsAggregate.js

@@ -1,284 +1 @@
-const express = require('express');
-const router = express.Router();
-const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
-
-// --- Configuration & Helpers ---
-const DEFAULT_PAGE_LIMIT = 50;
-const MAX_PAGE_LIMIT = 200;
-
-// Maps query keys to DB columns in brand_metrics
-const COLUMN_MAP = {
-    brandName: { dbCol: 'bm.brand_name', type: 'string' },
-    productCount: { dbCol: 'bm.product_count', type: 'number' },
-    activeProductCount: { dbCol: 'bm.active_product_count', type: 'number' },
-    replenishableProductCount: { dbCol: 'bm.replenishable_product_count', type: 'number' },
-    currentStockUnits: { dbCol: 'bm.current_stock_units', type: 'number' },
-    currentStockCost: { dbCol: 'bm.current_stock_cost', type: 'number' },
-    currentStockRetail: { dbCol: 'bm.current_stock_retail', type: 'number' },
-    sales7d: { dbCol: 'bm.sales_7d', type: 'number' },
-    revenue7d: { dbCol: 'bm.revenue_7d', type: 'number' },
-    sales30d: { dbCol: 'bm.sales_30d', type: 'number' },
-    revenue30d: { dbCol: 'bm.revenue_30d', type: 'number' },
-    profit30d: { dbCol: 'bm.profit_30d', type: 'number' },
-    cogs30d: { dbCol: 'bm.cogs_30d', type: 'number' },
-    sales365d: { dbCol: 'bm.sales_365d', type: 'number' },
-    revenue365d: { dbCol: 'bm.revenue_365d', type: 'number' },
-    lifetimeSales: { dbCol: 'bm.lifetime_sales', type: 'number' },
-    lifetimeRevenue: { dbCol: 'bm.lifetime_revenue', type: 'number' },
-    avgMargin30d: { dbCol: 'bm.avg_margin_30d', type: 'number' },
-    // Growth metrics
-    salesGrowth30dVsPrev: { dbCol: 'bm.sales_growth_30d_vs_prev', type: 'number' },
-    revenueGrowth30dVsPrev: { dbCol: 'bm.revenue_growth_30d_vs_prev', type: 'number' },
-    // Add aliases if needed
-    name: { dbCol: 'bm.brand_name', type: 'string' },
-    // Add status for filtering
-    status: { dbCol: 'brand_status', type: 'string' },
-};
-
-function getSafeColumnInfo(queryParamKey) {
-    return COLUMN_MAP[queryParamKey] || null;
-}
-
-// --- Route Handlers ---
-
-// GET /brands-aggregate/filter-options (Just brands list for now)
-router.get('/filter-options', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /brands-aggregate/filter-options');
-    try {
-        // Get brand names
-        const { rows: brandRows } = await pool.query(`
-            SELECT DISTINCT brand_name FROM public.brand_metrics ORDER BY brand_name
-        `);
-        
-        // Get status values - calculate them since they're derived
-        const { rows: statusRows } = await pool.query(`
-            SELECT DISTINCT
-                CASE 
-                    WHEN active_product_count > 0 AND sales_30d > 0 THEN 'active'
-                    WHEN active_product_count > 0 THEN 'inactive'
-                    ELSE 'pending'
-                END as status
-            FROM public.brand_metrics
-            ORDER BY status
-        `);
-        
-        res.json({
-            brands: brandRows.map(r => r.brand_name),
-            statuses: statusRows.map(r => r.status)
-        });
-     } catch(error) {
-        console.error('Error fetching brand filter options:', error);
-        res.status(500).json({ error: 'Failed to fetch filter options' });
-     }
-});
-
-// GET /brands-aggregate/stats (Overall brand stats)
-router.get('/stats', async (req, res) => {
-     const pool = req.app.locals.pool;
-     console.log('GET /brands-aggregate/stats');
-     try {
-         const { rows: [stats] } = await pool.query(`
-            SELECT
-                COUNT(*) AS total_brands,
-                COUNT(CASE WHEN active_product_count > 0 THEN 1 END) AS active_brands,
-                SUM(active_product_count) AS total_active_products,
-                SUM(current_stock_cost) AS total_stock_value,
-                -- Weighted Average Margin
-                SUM(profit_30d) * 100.0 / NULLIF(SUM(revenue_30d), 0) AS overall_avg_margin_weighted
-            FROM public.brand_metrics bm
-         `);
-
-         res.json({
-             totalBrands: parseInt(stats?.total_brands || 0),
-             activeBrands: parseInt(stats?.active_brands || 0),
-             totalActiveProducts: parseInt(stats?.total_active_products || 0),
-             totalValue: parseFloat(stats?.total_stock_value || 0),
-             avgMargin: parseFloat(stats?.overall_avg_margin_weighted || 0),
-         });
-     } catch (error) {
-        console.error('Error fetching brand stats:', error);
-        res.status(500).json({ error: 'Failed to fetch brand stats.' });
-     }
-});
-
-// GET /brands-aggregate/ (List brands)
-router.get('/', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /brands-aggregate received query:', req.query);
-    try {
-        // --- Pagination ---
-        let page = parseInt(req.query.page, 10) || 1;
-        let limit = parseInt(req.query.limit, 10) || DEFAULT_PAGE_LIMIT;
-        limit = Math.min(limit, MAX_PAGE_LIMIT);
-        const offset = (page - 1) * limit;
-
-        // --- Sorting ---
-        const sortQueryKey = req.query.sort || 'brandName'; // Default sort
-        const sortColumnInfo = getSafeColumnInfo(sortQueryKey);
-        const sortColumn = sortColumnInfo ? sortColumnInfo.dbCol : 'bm.brand_name';
-        const sortDirection = req.query.order?.toLowerCase() === 'desc' ? 'DESC' : 'ASC';
-        const nullsOrder = (sortDirection === 'ASC' ? 'NULLS FIRST' : 'NULLS LAST');
-        const sortClause = `ORDER BY ${sortColumn} ${sortDirection} ${nullsOrder}`;
-
-        // --- Filtering ---
-        const conditions = [];
-        const params = [];
-        let paramCounter = 1;
-        // Build conditions based on req.query, using COLUMN_MAP and parseValue
-        for (const key in req.query) {
-             if (['page', 'limit', 'sort', 'order'].includes(key)) continue;
-
-             let filterKey = key;
-             let operator = '='; // Default operator
-             const value = req.query[key];
-
-             const operatorMatch = key.match(/^(.*)_(eq|ne|gt|gte|lt|lte|like|ilike|between|in)$/);
-             if (operatorMatch) {
-                 filterKey = operatorMatch[1];
-                 operator = operatorMatch[2];
-             }
-
-             const columnInfo = getSafeColumnInfo(filterKey);
-             if (columnInfo) {
-                 const dbColumn = columnInfo.dbCol;
-                 const valueType = columnInfo.type;
-                  try {
-                     let conditionFragment = '';
-                     let needsParam = true;
-                     switch (operator.toLowerCase()) { // Normalize operator
-                          case 'eq': operator = '='; break;
-                          case 'ne': operator = '<>'; break;
-                          case 'gt': operator = '>'; break;
-                          case 'gte': operator = '>='; break;
-                          case 'lt': operator = '<'; break;
-                          case 'lte': operator = '<='; break;
-                          case 'like': operator = 'LIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                          case 'ilike': operator = 'ILIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                          case 'between':
-                              const [val1, val2] = String(value).split(',');
-                              if (val1 !== undefined && val2 !== undefined) {
-                                  conditionFragment = `${dbColumn} BETWEEN $${paramCounter++} AND $${paramCounter++}`;
-                                  params.push(parseValue(val1, valueType), parseValue(val2, valueType));
-                                  needsParam = false;
-                              } else continue;
-                              break;
-                          case 'in':
-                               const inValues = String(value).split(',');
-                               if (inValues.length > 0) {
-                                  const placeholders = inValues.map(() => `$${paramCounter++}`).join(', ');
-                                  conditionFragment = `${dbColumn} IN (${placeholders})`;
-                                  params.push(...inValues.map(v => parseValue(v, valueType)));
-                                  needsParam = false;
-                               } else continue;
-                               break;
-                          default: operator = '='; break;
-                     }
-
-                     if (needsParam) {
-                          conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                          params.push(parseValue(value, valueType));
-                     } else if (!conditionFragment) { // For LIKE/ILIKE
-                          conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                     }
-
-
-                     if (conditionFragment) {
-                          conditions.push(`(${conditionFragment})`);
-                     }
-                  } catch (parseError) {
-                      console.warn(`Skipping filter for key "${key}" due to parsing error: ${parseError.message}`);
-                      if (needsParam) paramCounter--;
-                  }
-             } else {
-                  console.warn(`Invalid filter key ignored: ${key}`);
-             }
-         }
-
-
-        // --- Execute Queries ---
-        const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : '';
-        
-        // Status calculation similar to vendors
-        const statusCase = `
-            CASE 
-                WHEN active_product_count > 0 AND sales_30d > 0 THEN 'active'
-                WHEN active_product_count > 0 THEN 'inactive'
-                ELSE 'pending'
-            END as brand_status
-        `;
-        
-        const baseSql = `
-            FROM (
-                SELECT 
-                    bm.*,
-                    ${statusCase}
-                FROM public.brand_metrics bm
-            ) bm
-            ${whereClause}
-        `;
-
-        const countSql = `SELECT COUNT(*) AS total ${baseSql}`;
-        const dataSql = `
-            WITH brand_data AS (
-                SELECT 
-                    bm.*,
-                    ${statusCase}
-                FROM public.brand_metrics bm
-            )
-            SELECT bm.*
-            FROM brand_data bm
-            ${whereClause}
-            ${sortClause}
-            LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-        `;
-        const dataParams = [...params, limit, offset];
-
-        console.log("Count SQL:", countSql, params);
-        console.log("Data SQL:", dataSql, dataParams);
-
-        const [countResult, dataResult] = await Promise.all([
-            pool.query(countSql, params),
-            pool.query(dataSql, dataParams)
-        ]);
-
-        const total = parseInt(countResult.rows[0].total, 10);
-        const brands = dataResult.rows.map(row => {
-            // Create a new object with both snake_case and camelCase keys
-            const transformedRow = { ...row }; // Start with original data
-            
-            for (const key in row) {
-                // Skip null/undefined values
-                if (row[key] === null || row[key] === undefined) {
-                    continue; // Original already has the null value
-                }
-
-                // Transform keys to match frontend expectations (add camelCase versions)
-                // First handle cases like sales_7d -> sales7d
-                let camelKey = key.replace(/_(\d+[a-z])/g, '$1');
-                
-                // Then handle regular snake_case -> camelCase
-                camelKey = camelKey.replace(/_([a-z])/g, (_, letter) => letter.toUpperCase());
-                if (camelKey !== key) { // Only add if different from original
-                    transformedRow[camelKey] = row[key];
-                }
-            }
-            return transformedRow;
-        });
-
-        // --- Respond ---
-        res.json({
-            brands,
-            pagination: { total, pages: Math.ceil(total / limit), currentPage: page, limit },
-        });
-
-    } catch (error) {
-        console.error('Error fetching brand metrics list:', error);
-        res.status(500).json({ error: 'Failed to fetch brand metrics.' });
-    }
-});
-
-// GET /brands-aggregate/:name (Get single brand metric)
-// Implement if needed, remember to URL-decode the name parameter
-
-module.exports = router;
+ 

inventory-server/src/routes/categoriesAggregate.js

@@ -1,363 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
-
-// --- Configuration & Helpers ---
-const DEFAULT_PAGE_LIMIT = 50;
-const MAX_PAGE_LIMIT = 5000; // Increase this to allow retrieving all categories in one request
-
-// Maps query keys to DB columns in category_metrics and categories tables
-const COLUMN_MAP = {
-    categoryId: { dbCol: 'cm.category_id', type: 'integer' },
-    categoryName: { dbCol: 'cm.category_name', type: 'string' }, // From aggregate table
-    categoryType: { dbCol: 'cm.category_type', type: 'integer' }, // From aggregate table
-    parentId: { dbCol: 'cm.parent_id', type: 'integer' }, // From aggregate table
-    parentName: { dbCol: 'p.name', type: 'string' }, // Requires JOIN to categories
-    productCount: { dbCol: 'cm.product_count', type: 'number' },
-    activeProductCount: { dbCol: 'cm.active_product_count', type: 'number' },
-    replenishableProductCount: { dbCol: 'cm.replenishable_product_count', type: 'number' },
-    currentStockUnits: { dbCol: 'cm.current_stock_units', type: 'number' },
-    currentStockCost: { dbCol: 'cm.current_stock_cost', type: 'number' },
-    currentStockRetail: { dbCol: 'cm.current_stock_retail', type: 'number' },
-    sales7d: { dbCol: 'cm.sales_7d', type: 'number' },
-    revenue7d: { dbCol: 'cm.revenue_7d', type: 'number' },
-    sales30d: { dbCol: 'cm.sales_30d', type: 'number' },
-    revenue30d: { dbCol: 'cm.revenue_30d', type: 'number' },
-    profit30d: { dbCol: 'cm.profit_30d', type: 'number' },
-    cogs30d: { dbCol: 'cm.cogs_30d', type: 'number' },
-    sales365d: { dbCol: 'cm.sales_365d', type: 'number' },
-    revenue365d: { dbCol: 'cm.revenue_365d', type: 'number' },
-    lifetimeSales: { dbCol: 'cm.lifetime_sales', type: 'number' },
-    lifetimeRevenue: { dbCol: 'cm.lifetime_revenue', type: 'number' },
-    avgMargin30d: { dbCol: 'cm.avg_margin_30d', type: 'number' },
-    stockTurn30d: { dbCol: 'cm.stock_turn_30d', type: 'number' },
-    // Growth metrics
-    salesGrowth30dVsPrev: { dbCol: 'cm.sales_growth_30d_vs_prev', type: 'number' },
-    revenueGrowth30dVsPrev: { dbCol: 'cm.revenue_growth_30d_vs_prev', type: 'number' },
-    // Add status from the categories table for filtering
-    status: { dbCol: 'c.status', type: 'string' },
-};
-
-function getSafeColumnInfo(queryParamKey) {
-    return COLUMN_MAP[queryParamKey] || null;
-}
-
-// Type Labels (Consider moving to a shared config or fetching from DB)
-const TYPE_LABELS = {
-    10: 'Section', 11: 'Category', 12: 'Subcategory', 13: 'Sub-subcategory',
-    1: 'Company', 2: 'Line', 3: 'Subline', 40: 'Artist', // From old schema comments
-    20: 'Theme', 21: 'Subtheme' // Additional types from categories.js
-};
-
-// --- Route Handlers ---
-
-// GET /categories-aggregate/filter-options
-router.get('/filter-options', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /categories-aggregate/filter-options');
-    try {
-        // Fetch distinct types directly from the aggregate table if reliable
-        // Or join with categories table if source of truth is needed
-        const { rows: typeRows } = await pool.query(`
-            SELECT DISTINCT category_type
-            FROM public.category_metrics
-            ORDER BY category_type
-        `);
-
-        const typeOptions = typeRows.map(r => ({
-            value: r.category_type,
-            label: TYPE_LABELS[r.category_type] || `Type ${r.category_type}` // Add labels
-        }));
-
-        // Add status options for filtering (from categories.js)
-        const { rows: statusRows } = await pool.query(`
-            SELECT DISTINCT status FROM public.categories ORDER BY status
-        `);
-        
-        // Get type counts (from categories.js)
-        const { rows: typeCounts } = await pool.query(`
-            SELECT 
-                type,
-                COUNT(*)::integer as count
-            FROM categories
-            GROUP BY type
-            ORDER BY type
-        `);
-
-        res.json({
-            types: typeOptions,
-            statuses: statusRows.map(r => r.status),
-            typeCounts: typeCounts.map(tc => ({
-                type: tc.type,
-                count: tc.count
-            }))
-        });
-    } catch (error) {
-        console.error('Error fetching category filter options:', error);
-        res.status(500).json({ error: 'Failed to fetch filter options' });
-    }
-});
-
-// GET /categories-aggregate/stats
-router.get('/stats', async (req, res) => {
-     const pool = req.app.locals.pool;
-     console.log('GET /categories-aggregate/stats');
-     try {
-         // Calculate stats directly from the aggregate table
-         const { rows: [stats] } = await pool.query(`
-            SELECT
-                COUNT(*) AS total_categories,
-                -- Count active based on the source categories table status
-                COUNT(CASE WHEN c.status = 'active' THEN cm.category_id END) AS active_categories,
-                SUM(cm.active_product_count) AS total_active_products, -- Sum from aggregates
-                SUM(cm.current_stock_cost) AS total_stock_value, -- Sum from aggregates
-                -- Weighted Average Margin (Revenue as weight)
-                SUM(cm.profit_30d) * 100.0 / NULLIF(SUM(cm.revenue_30d), 0) AS overall_avg_margin_weighted,
-                -- Simple Average Margin (less accurate if categories vary greatly in size)
-                AVG(NULLIF(cm.avg_margin_30d, 0)) AS overall_avg_margin_simple
-                -- Growth rate can be calculated from 30d vs previous 30d revenue if needed
-            FROM public.category_metrics cm
-            JOIN public.categories c ON cm.category_id = c.cat_id -- Join to check category status
-         `);
-
-         res.json({
-             totalCategories: parseInt(stats?.total_categories || 0),
-             activeCategories: parseInt(stats?.active_categories || 0), // Based on categories.status
-             totalActiveProducts: parseInt(stats?.total_active_products || 0),
-             totalValue: parseFloat(stats?.total_stock_value || 0),
-             // Choose which avg margin calculation to expose
-             avgMargin: parseFloat(stats?.overall_avg_margin_weighted || stats?.overall_avg_margin_simple || 0)
-             // Growth rate could be added if we implement the calculation
-         });
-     } catch (error) {
-        console.error('Error fetching category stats:', error);
-        res.status(500).json({ error: 'Failed to fetch category stats.' });
-     }
-});
-
-// GET /categories-aggregate/ (List categories)
-router.get('/', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /categories-aggregate received query:', req.query);
-    try {
-        // --- Pagination ---
-        let page = parseInt(req.query.page, 10) || 1;
-        let limit = parseInt(req.query.limit, 10) || DEFAULT_PAGE_LIMIT;
-        limit = Math.min(limit, MAX_PAGE_LIMIT);
-        const offset = (page - 1) * limit;
-
-        // --- Sorting ---
-        const sortQueryKey = req.query.sort || 'categoryName';
-        const sortColumnInfo = getSafeColumnInfo(sortQueryKey);
-        
-        // Hierarchical sorting logic from categories.js
-        const hierarchicalSortOrder = `
-            ORDER BY 
-                CASE 
-                    WHEN cm.category_type = 10 THEN 1  -- sections first
-                    WHEN cm.category_type = 11 THEN 2  -- categories second
-                    WHEN cm.category_type = 12 THEN 3  -- subcategories third
-                    WHEN cm.category_type = 13 THEN 4  -- subsubcategories fourth
-                    WHEN cm.category_type = 20 THEN 5  -- themes fifth
-                    WHEN cm.category_type = 21 THEN 6  -- subthemes last
-                    ELSE 7
-                END,
-                cm.category_name ASC
-        `;
-        
-        // Use hierarchical sort as default
-        let sortClause = hierarchicalSortOrder;
-        
-        // Override with custom sort if specified
-        if (sortColumnInfo && sortQueryKey !== 'categoryName') {
-            const sortColumn = sortColumnInfo.dbCol;
-            const sortDirection = req.query.order?.toLowerCase() === 'desc' ? 'DESC' : 'ASC';
-            const nullsOrder = (sortDirection === 'ASC' ? 'NULLS FIRST' : 'NULLS LAST');
-            sortClause = `ORDER BY ${sortColumn} ${sortDirection} ${nullsOrder}`;
-        }
-
-        // --- Filtering ---
-        const conditions = [];
-        const params = [];
-        let paramCounter = 1;
-
-        console.log("Starting to process filters from query:", req.query);
-        
-        // Add filters based on req.query using COLUMN_MAP and parseValue
-        for (const key in req.query) {
-            if (['page', 'limit', 'sort', 'order'].includes(key)) continue;
-
-            let filterKey = key;
-            let operator = '='; // Default operator
-            const value = req.query[key];
-            
-            console.log(`Processing filter key: "${key}" with value: "${value}"`);
-
-            const operatorMatch = key.match(/^(.*)_(eq|ne|gt|gte|lt|lte|like|ilike|between|in)$/);
-            if (operatorMatch) {
-                filterKey = operatorMatch[1];
-                operator = operatorMatch[2];
-                console.log(`Parsed filter key: "${filterKey}" with operator: "${operator}"`);
-            }
-
-            // Special case for parentName requires join
-            const requiresJoin = filterKey === 'parentName';
-            const columnInfo = getSafeColumnInfo(filterKey);
-
-            if (columnInfo) {
-                console.log(`Column info for "${filterKey}":`, columnInfo);
-                const dbColumn = columnInfo.dbCol;
-                const valueType = columnInfo.type;
-                 try {
-                    let conditionFragment = '';
-                    let needsParam = true;
-                    switch (operator.toLowerCase()) {
-                        case 'eq': operator = '='; break;
-                        case 'ne': operator = '<>'; break;
-                        case 'gt': operator = '>'; break;
-                        case 'gte': operator = '>='; break;
-                        case 'lt': operator = '<'; break;
-                        case 'lte': operator = '<='; break;
-                        case 'like': operator = 'LIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                        case 'ilike': operator = 'ILIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                        case 'between':
-                            const [val1, val2] = String(value).split(',');
-                            if (val1 !== undefined && val2 !== undefined) {
-                                conditionFragment = `${dbColumn} BETWEEN $${paramCounter++} AND $${paramCounter++}`;
-                                params.push(parseValue(val1, valueType), parseValue(val2, valueType));
-                                needsParam = false;
-                            } else continue;
-                            break;
-                        case 'in':
-                             const inValues = String(value).split(',');
-                             if (inValues.length > 0) {
-                                const placeholders = inValues.map(() => `$${paramCounter++}`).join(', ');
-                                conditionFragment = `${dbColumn} IN (${placeholders})`;
-                                params.push(...inValues.map(v => parseValue(v, valueType)));
-                                needsParam = false;
-                             } else continue;
-                             break;
-                        default: operator = '='; break;
-                    }
-
-                    if (needsParam) {
-                         try {
-                             // Special handling for categoryType to ensure it works
-                             if (filterKey === 'categoryType') {
-                                 console.log(`Special handling for categoryType: ${value}`);
-                                 // Force conversion to integer
-                                 const numericValue = parseInt(value, 10);
-                                 if (!isNaN(numericValue)) {
-                                     console.log(`Successfully converted categoryType to integer: ${numericValue}`);
-                                     conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                                     params.push(numericValue);
-                                 } else {
-                                     console.error(`Failed to convert categoryType to integer: "${value}"`);
-                                     throw new Error(`Invalid categoryType value: "${value}"`);
-                                 }
-                             } else {
-                                 // Normal handling for other fields
-                                 const parsedValue = parseValue(value, valueType);
-                                 console.log(`Parsed "${value}" as ${valueType}: ${parsedValue}`);
-                                 conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                                 params.push(parsedValue);
-                             }
-                         } catch (innerError) {
-                             console.error(`Failed to parse "${value}" as ${valueType}:`, innerError);
-                             throw innerError;
-                         }
-                    } else if (!conditionFragment) { // For LIKE/ILIKE where needsParam is false
-                        conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`; // paramCounter was already incremented in push
-                    }
-
-
-                    if (conditionFragment) {
-                         console.log(`Adding condition: ${conditionFragment}`);
-                         conditions.push(`(${conditionFragment})`);
-                    }
-                 } catch (parseError) {
-                     console.error(`Skipping filter for key "${key}" due to parsing error:`, parseError);
-                     if (needsParam) paramCounter--; // Roll back counter if param push failed
-                 }
-            } else {
-                 console.warn(`Invalid filter key ignored: "${key}", not found in COLUMN_MAP`);
-            }
-        }
-
-        // --- Execute Queries ---
-        const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : '';
-
-        // Need JOIN for parent_name if sorting/filtering by it, or always include for display
-        const sortColumn = sortColumnInfo?.dbCol;
-        
-        // Always include the category and parent joins for status and parent_name
-        const joinSql = `
-            JOIN public.categories c ON cm.category_id = c.cat_id
-            LEFT JOIN public.categories p ON cm.parent_id = p.cat_id
-        `;
-
-        const baseSql = `
-            FROM public.category_metrics cm
-            ${joinSql}
-            ${whereClause}
-        `;
-
-        const countSql = `SELECT COUNT(*) AS total ${baseSql}`;
-        const dataSql = `
-            SELECT 
-                cm.*,
-                c.status,
-                c.description,
-                p.name as parent_name,
-                p.type as parent_type
-            ${baseSql}
-            ${sortClause}
-            LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-        `;
-        const dataParams = [...params, limit, offset];
-
-        console.log("Count SQL:", countSql, params);
-        console.log("Data SQL:", dataSql, dataParams);
-
-        const [countResult, dataResult] = await Promise.all([
-            pool.query(countSql, params),
-            pool.query(dataSql, dataParams)
-        ]);
-
-        const total = parseInt(countResult.rows[0].total, 10);
-        const categories = dataResult.rows.map(row => {
-            // Create a new object with both snake_case and camelCase keys
-            const transformedRow = { ...row }; // Start with original data
-            
-            for (const key in row) {
-                // Skip null/undefined values
-                if (row[key] === null || row[key] === undefined) {
-                    continue; // Original already has the null value
-                }
-
-                // Transform keys to match frontend expectations (add camelCase versions)
-                // First handle cases like sales_7d -> sales7d
-                let camelKey = key.replace(/_(\d+[a-z])/g, '$1');
-                
-                // Then handle regular snake_case -> camelCase
-                camelKey = camelKey.replace(/_([a-z])/g, (_, letter) => letter.toUpperCase());
-                if (camelKey !== key) { // Only add if different from original
-                    transformedRow[camelKey] = row[key];
-                }
-            }
-            return transformedRow;
-        });
-
-        // --- Respond ---
-        res.json({
-            categories,
-            pagination: { total, pages: Math.ceil(total / limit), currentPage: page, limit },
-        });
-
-    } catch (error) {
-        console.error('Error fetching category metrics list:', error);
-        res.status(500).json({ error: 'Failed to fetch category metrics.' });
-    }
-});
-
-module.exports = router;

inventory-server/src/routes/config.js

@@ -1,325 +0,0 @@
-const express = require('express');
-const router = express.Router();
-
-// Debug middleware
-router.use((req, res, next) => {
-  console.log(`[Config Route] ${req.method} ${req.path}`);
-  next();
-});
-
-// ===== GLOBAL SETTINGS =====
-
-// Get all global settings
-router.get('/global', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    console.log('[Config Route] Fetching global settings...');
-    
-    const { rows } = await pool.query('SELECT * FROM settings_global ORDER BY setting_key');
-    
-    console.log('[Config Route] Sending global settings:', rows);
-    res.json(rows);
-  } catch (error) {
-    console.error('[Config Route] Error fetching global settings:', error);
-    res.status(500).json({ error: 'Failed to fetch global settings', details: error.message });
-  }
-});
-
-// Update global settings
-router.put('/global', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    console.log('[Config Route] Updating global settings:', req.body);
-    
-    // Validate request
-    if (!Array.isArray(req.body)) {
-      return res.status(400).json({ error: 'Request body must be an array of settings' });
-    }
-    
-    // Begin transaction
-    const client = await pool.connect();
-    try {
-      await client.query('BEGIN');
-      
-      for (const setting of req.body) {
-        if (!setting.setting_key || !setting.setting_value) {
-          throw new Error('Each setting must have a key and value');
-        }
-        
-        await client.query(
-          `UPDATE settings_global 
-           SET setting_value = $1, 
-               updated_at = CURRENT_TIMESTAMP
-           WHERE setting_key = $2`,
-          [setting.setting_value, setting.setting_key]
-        );
-      }
-      
-      await client.query('COMMIT');
-      res.json({ success: true });
-    } catch (error) {
-      await client.query('ROLLBACK');
-      throw error;
-    } finally {
-      client.release();
-    }
-  } catch (error) {
-    console.error('[Config Route] Error updating global settings:', error);
-    res.status(500).json({ error: 'Failed to update global settings', details: error.message });
-  }
-});
-
-// ===== PRODUCT SETTINGS =====
-
-// Get product settings with pagination and search
-router.get('/products', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    console.log('[Config Route] Fetching product settings...');
-    
-    const page = parseInt(req.query.page) || 1;
-    const pageSize = parseInt(req.query.pageSize) || 10;
-    const offset = (page - 1) * pageSize;
-    const search = req.query.search || '';
-    
-    // Get total count for pagination
-    const countQuery = search
-      ? `SELECT COUNT(*) FROM settings_product sp
-         JOIN products p ON sp.pid::text = p.pid::text
-         WHERE sp.pid::text ILIKE $1 OR p.title ILIKE $1`
-      : 'SELECT COUNT(*) FROM settings_product';
-    
-    const countParams = search ? [`%${search}%`] : [];
-    const { rows: countResult } = await pool.query(countQuery, countParams);
-    const total = parseInt(countResult[0].count);
-    
-    // Get paginated settings
-    const query = search
-      ? `SELECT sp.*, p.title as product_name 
-         FROM settings_product sp
-         JOIN products p ON sp.pid::text = p.pid::text
-         WHERE sp.pid::text ILIKE $1 OR p.title ILIKE $1
-         ORDER BY sp.pid
-         LIMIT $2 OFFSET $3`
-      : `SELECT sp.*, p.title as product_name 
-         FROM settings_product sp
-         JOIN products p ON sp.pid::text = p.pid::text
-         ORDER BY sp.pid
-         LIMIT $1 OFFSET $2`;
-    
-    const queryParams = search 
-      ? [`%${search}%`, pageSize, offset]
-      : [pageSize, offset];
-    
-    const { rows } = await pool.query(query, queryParams);
-    
-    const response = {
-      items: rows,
-      total,
-      page,
-      pageSize
-    };
-    
-    console.log(`[Config Route] Sending ${rows.length} product settings`);
-    res.json(response);
-  } catch (error) {
-    console.error('[Config Route] Error fetching product settings:', error);
-    res.status(500).json({ error: 'Failed to fetch product settings', details: error.message });
-  }
-});
-
-// Update product settings
-router.put('/products/:pid', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const { pid } = req.params;
-    const { lead_time_days, days_of_stock, safety_stock, forecast_method, exclude_from_forecast } = req.body;
-    
-    console.log(`[Config Route] Updating product settings for ${pid}:`, req.body);
-    
-    // Check if product exists
-    const { rows: checkProduct } = await pool.query(
-      'SELECT 1 FROM settings_product WHERE pid::text = $1',
-      [pid]
-    );
-    
-    if (checkProduct.length === 0) {
-      // Insert if it doesn't exist
-      await pool.query(
-        `INSERT INTO settings_product 
-         (pid, lead_time_days, days_of_stock, safety_stock, forecast_method, exclude_from_forecast)
-         VALUES ($1, $2, $3, $4, $5, $6)`,
-        [pid, lead_time_days, days_of_stock, safety_stock, forecast_method, exclude_from_forecast]
-      );
-    } else {
-      // Update if it exists
-      await pool.query(
-        `UPDATE settings_product
-         SET lead_time_days = $2,
-             days_of_stock = $3,
-             safety_stock = $4,
-             forecast_method = $5,
-             exclude_from_forecast = $6,
-             updated_at = CURRENT_TIMESTAMP
-         WHERE pid::text = $1`,
-        [pid, lead_time_days, days_of_stock, safety_stock, forecast_method, exclude_from_forecast]
-      );
-    }
-    
-    res.json({ success: true });
-  } catch (error) {
-    console.error(`[Config Route] Error updating product settings for ${req.params.pid}:`, error);
-    res.status(500).json({ error: 'Failed to update product settings', details: error.message });
-  }
-});
-
-// Reset product settings to defaults
-router.post('/products/:pid/reset', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const { pid } = req.params;
-    
-    console.log(`[Config Route] Resetting product settings for ${pid}`);
-    
-    // Reset by setting everything to null/default
-    await pool.query(
-      `UPDATE settings_product
-       SET lead_time_days = NULL,
-           days_of_stock = NULL,
-           safety_stock = 0,
-           forecast_method = NULL,
-           exclude_from_forecast = false,
-           updated_at = CURRENT_TIMESTAMP
-       WHERE pid::text = $1`,
-      [pid]
-    );
-    
-    res.json({ success: true });
-  } catch (error) {
-    console.error(`[Config Route] Error resetting product settings for ${req.params.pid}:`, error);
-    res.status(500).json({ error: 'Failed to reset product settings', details: error.message });
-  }
-});
-
-// ===== VENDOR SETTINGS =====
-
-// Get vendor settings with pagination and search
-router.get('/vendors', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    console.log('[Config Route] Fetching vendor settings...');
-    
-    const page = parseInt(req.query.page) || 1;
-    const pageSize = parseInt(req.query.pageSize) || 10;
-    const offset = (page - 1) * pageSize;
-    const search = req.query.search || '';
-    
-    // Get total count for pagination
-    const countQuery = search
-      ? 'SELECT COUNT(*) FROM settings_vendor WHERE vendor ILIKE $1'
-      : 'SELECT COUNT(*) FROM settings_vendor';
-    
-    const countParams = search ? [`%${search}%`] : [];
-    const { rows: countResult } = await pool.query(countQuery, countParams);
-    const total = parseInt(countResult[0].count);
-    
-    // Get paginated settings
-    const query = search
-      ? `SELECT * FROM settings_vendor
-         WHERE vendor ILIKE $1
-         ORDER BY vendor
-         LIMIT $2 OFFSET $3`
-      : `SELECT * FROM settings_vendor
-         ORDER BY vendor
-         LIMIT $1 OFFSET $2`;
-    
-    const queryParams = search 
-      ? [`%${search}%`, pageSize, offset]
-      : [pageSize, offset];
-    
-    const { rows } = await pool.query(query, queryParams);
-    
-    const response = {
-      items: rows,
-      total,
-      page,
-      pageSize
-    };
-    
-    console.log(`[Config Route] Sending ${rows.length} vendor settings`);
-    res.json(response);
-  } catch (error) {
-    console.error('[Config Route] Error fetching vendor settings:', error);
-    res.status(500).json({ error: 'Failed to fetch vendor settings', details: error.message });
-  }
-});
-
-// Update vendor settings
-router.put('/vendors/:vendor', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const vendor = req.params.vendor;
-    const { default_lead_time_days, default_days_of_stock } = req.body;
-    
-    console.log(`[Config Route] Updating vendor settings for ${vendor}:`, req.body);
-    
-    // Check if vendor exists
-    const { rows: checkVendor } = await pool.query(
-      'SELECT 1 FROM settings_vendor WHERE vendor = $1',
-      [vendor]
-    );
-    
-    if (checkVendor.length === 0) {
-      // Insert if it doesn't exist
-      await pool.query(
-        `INSERT INTO settings_vendor 
-         (vendor, default_lead_time_days, default_days_of_stock)
-         VALUES ($1, $2, $3)`,
-        [vendor, default_lead_time_days, default_days_of_stock]
-      );
-    } else {
-      // Update if it exists
-      await pool.query(
-        `UPDATE settings_vendor
-         SET default_lead_time_days = $2,
-             default_days_of_stock = $3,
-             updated_at = CURRENT_TIMESTAMP
-         WHERE vendor = $1`,
-        [vendor, default_lead_time_days, default_days_of_stock]
-      );
-    }
-    
-    res.json({ success: true });
-  } catch (error) {
-    console.error(`[Config Route] Error updating vendor settings for ${req.params.vendor}:`, error);
-    res.status(500).json({ error: 'Failed to update vendor settings', details: error.message });
-  }
-});
-
-// Reset vendor settings to defaults
-router.post('/vendors/:vendor/reset', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const vendor = req.params.vendor;
-    
-    console.log(`[Config Route] Resetting vendor settings for ${vendor}`);
-    
-    // Reset by setting everything to null
-    await pool.query(
-      `UPDATE settings_vendor
-       SET default_lead_time_days = NULL,
-           default_days_of_stock = NULL,
-           updated_at = CURRENT_TIMESTAMP
-       WHERE vendor = $1`,
-      [vendor]
-    );
-    
-    res.json({ success: true });
-  } catch (error) {
-    console.error(`[Config Route] Error resetting vendor settings for ${req.params.vendor}:`, error);
-    res.status(500).json({ error: 'Failed to reset vendor settings', details: error.message });
-  }
-});
-
-// Export the router
-module.exports = router; 

inventory-server/src/routes/data-management.js

@@ -1,440 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const { spawn } = require('child_process');
-const path = require('path');
-const db = require('../utils/db');
-
-// Debug middleware MUST be first
-router.use((req, res, next) => {
-    console.log(`[CSV Route Debug] ${req.method} ${req.path}`);
-    next();
-});
-
-// Store active processes and their progress
-let activeImport = null;
-let importProgress = null;
-let activeFullUpdate = null;
-let activeFullReset = null;
-
-// SSE clients for progress updates
-const updateClients = new Set();
-const importClients = new Set();
-const resetClients = new Set();
-const resetMetricsClients = new Set();
-const calculateMetricsClients = new Set();
-const fullUpdateClients = new Set();
-const fullResetClients = new Set();
-
-// Helper to send progress to specific clients
-function sendProgressToClients(clients, data) {
-    // If data is a string, send it directly
-    // If it's an object, convert it to JSON
-    const message = typeof data === 'string' 
-        ? `data: ${data}\n\n`
-        : `data: ${JSON.stringify(data)}\n\n`;
-    
-    clients.forEach(client => {
-        try {
-            client.write(message);
-            // Immediately flush the response
-            if (typeof client.flush === 'function') {
-                client.flush();
-            }
-        } catch (error) {
-            // Silently remove failed client
-            clients.delete(client);
-        }
-    });
-}
-
-// Helper to run a script and stream progress
-function runScript(scriptPath, type, clients) {
-    return new Promise((resolve, reject) => {
-        // Kill any existing process of this type
-        let activeProcess;
-        switch (type) {
-            case 'update':
-                if (activeFullUpdate) {
-                    try { activeFullUpdate.kill(); } catch (e) { }
-                }
-                activeProcess = activeFullUpdate;
-                break;
-            case 'reset':
-                if (activeFullReset) {
-                    try { activeFullReset.kill(); } catch (e) { }
-                }
-                activeProcess = activeFullReset;
-                break;
-        }
-
-        const child = spawn('node', [scriptPath], {
-            stdio: ['inherit', 'pipe', 'pipe']
-        });
-
-        switch (type) {
-            case 'update':
-                activeFullUpdate = child;
-                break;
-            case 'reset':
-                activeFullReset = child;
-                break;
-        }
-
-        let output = '';
-
-        child.stdout.on('data', (data) => {
-            const text = data.toString();
-            output += text;
-            
-            // Split by lines to handle multiple JSON outputs
-            const lines = text.split('\n');
-            lines.filter(line => line.trim()).forEach(line => {
-                try {
-                    // Try to parse as JSON but don't let it affect the display
-                    const jsonData = JSON.parse(line);
-                    // Only end the process if we get a final status
-                    if (jsonData.status === 'complete' || jsonData.status === 'error' || jsonData.status === 'cancelled') {
-                        if (jsonData.status === 'complete' && !jsonData.operation?.includes('complete')) {
-                            // Don't close for intermediate completion messages
-                            sendProgressToClients(clients, line);
-                            return;
-                        }
-                        // Close only on final completion/error/cancellation
-                        switch (type) {
-                            case 'update':
-                                activeFullUpdate = null;
-                                break;
-                            case 'reset':
-                                activeFullReset = null;
-                                break;
-                        }
-                        if (jsonData.status === 'error') {
-                            reject(new Error(jsonData.error || 'Unknown error'));
-                        } else {
-                            resolve({ output });
-                        }
-                    }
-                } catch (e) {
-                    // Not JSON, just display as is
-                }
-                // Always send the raw line
-                sendProgressToClients(clients, line);
-            });
-        });
-
-        child.stderr.on('data', (data) => {
-            const text = data.toString();
-            console.error(text);
-            // Send stderr output directly too
-            sendProgressToClients(clients, text);
-        });
-
-        child.on('close', (code) => {
-            switch (type) {
-                case 'update':
-                    activeFullUpdate = null;
-                    break;
-                case 'reset':
-                    activeFullReset = null;
-                    break;
-            }
-
-            if (code !== 0) {
-                const error = `Script ${scriptPath} exited with code ${code}`;
-                sendProgressToClients(clients, error);
-                reject(new Error(error));
-            }
-            // Don't resolve here - let the completion message from the script trigger the resolve
-        });
-
-        child.on('error', (err) => {
-            switch (type) {
-                case 'update':
-                    activeFullUpdate = null;
-                    break;
-                case 'reset':
-                    activeFullReset = null;
-                    break;
-            }
-            sendProgressToClients(clients, err.message);
-            reject(err);
-        });
-    });
-}
-
-// Progress endpoints
-router.get('/:type/progress', (req, res) => {
-    const { type } = req.params;
-    if (!['update', 'reset'].includes(type)) {
-        return res.status(400).json({ error: 'Invalid operation type' });
-    }
-
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Add this client to the correct set
-    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
-    clients.add(res);
-
-    // Send initial connection message
-    sendProgressToClients(new Set([res]), JSON.stringify({
-        status: 'running',
-        operation: 'Initializing connection...'
-    }));
-
-    // Handle client disconnect
-    req.on('close', () => {
-        clients.delete(res);
-    });
-});
-
-// GET /status - Check for active processes
-router.get('/status', (req, res) => {
-    try {
-        const hasActiveUpdate = activeFullUpdate !== null;
-        const hasActiveReset = activeFullReset !== null;
-        
-        if (hasActiveUpdate || hasActiveReset) {
-            res.json({
-                active: true,
-                progress: {
-                    status: 'running',
-                    operation: hasActiveUpdate ? 'Full update in progress' : 'Full reset in progress',
-                    type: hasActiveUpdate ? 'update' : 'reset'
-                }
-            });
-        } else {
-            res.json({
-                active: false,
-                progress: null
-            });
-        }
-    } catch (error) {
-        console.error('Error checking status:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// Route to cancel active process
-router.post('/cancel', (req, res) => {
-    let killed = false;
-
-    // Get the operation type from the request
-    const { type } = req.query;
-    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
-    const activeProcess = type === 'update' ? activeFullUpdate : activeFullReset;
-
-    if (activeProcess) {
-        try {
-            activeProcess.kill('SIGTERM');
-            if (type === 'update') {
-                activeFullUpdate = null;
-            } else {
-                activeFullReset = null;
-            }
-            killed = true;
-            sendProgressToClients(clients, JSON.stringify({
-                status: 'cancelled',
-                operation: 'Operation cancelled'
-            }));
-        } catch (err) {
-            console.error(`Error killing ${type} process:`, err);
-        }
-    }
-
-    if (killed) {
-        res.json({ success: true });
-    } else {
-        res.status(404).json({ error: 'No active process to cancel' });
-    }
-});
-
-// POST /csv/full-update - Run full update script
-router.post('/full-update', async (req, res) => {
-    try {
-        const scriptPath = path.join(__dirname, '../../scripts/full-update.js');
-        runScript(scriptPath, 'update', fullUpdateClients)
-            .catch(error => {
-                console.error('Update failed:', error);
-            });
-        res.status(202).json({ message: 'Update started' });
-    } catch (error) {
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// POST /csv/full-reset - Run full reset script
-router.post('/full-reset', async (req, res) => {
-    try {
-        const scriptPath = path.join(__dirname, '../../scripts/full-reset.js');
-        runScript(scriptPath, 'reset', fullResetClients)
-            .catch(error => {
-                console.error('Reset failed:', error);
-            });
-        res.status(202).json({ message: 'Reset started' });
-    } catch (error) {
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// GET /history/import - Get recent import history
-router.get('/history/import', async (req, res) => {
-    try {
-        const pool = req.app.locals.pool;
-        
-        // First check which columns exist
-        const { rows: columns } = await pool.query(`
-            SELECT column_name 
-            FROM information_schema.columns 
-            WHERE table_name = 'import_history' 
-            AND column_name IN ('records_deleted', 'records_skipped', 'total_processed')
-        `);
-        
-        const hasDeletedColumn = columns.some(col => col.column_name === 'records_deleted');
-        const hasSkippedColumn = columns.some(col => col.column_name === 'records_skipped');
-        const hasTotalProcessedColumn = columns.some(col => col.column_name === 'total_processed');
-        
-        // Build query dynamically based on available columns
-        const query = `
-            SELECT 
-                id,
-                start_time,
-                end_time,
-                status,
-                error_message,
-                records_added::integer,
-                records_updated::integer,
-                ${hasDeletedColumn ? 'records_deleted::integer,' : '0 as records_deleted,'}
-                ${hasSkippedColumn ? 'records_skipped::integer,' : '0 as records_skipped,'}
-                ${hasTotalProcessedColumn ? 'total_processed::integer,' : '0 as total_processed,'}
-                is_incremental,
-                additional_info,
-                EXTRACT(EPOCH FROM (COALESCE(end_time, NOW()) - start_time)) / 60 as duration_minutes
-            FROM import_history 
-            ORDER BY start_time DESC 
-            LIMIT 20
-        `;
-        
-        const { rows } = await pool.query(query);
-        res.json(rows || []);
-    } catch (error) {
-        console.error('Error fetching import history:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// GET /history/calculate - Get recent calculation history
-router.get('/history/calculate', async (req, res) => {
-    try {
-        const pool = req.app.locals.pool;
-        const { rows } = await pool.query(`
-            SELECT 
-                id,
-                start_time,
-                end_time,
-                EXTRACT(EPOCH FROM (COALESCE(end_time, NOW()) - start_time)) / 60 as duration_minutes,
-                duration_seconds,
-                status,
-                error_message,
-                total_products,
-                total_orders,
-                total_purchase_orders,
-                processed_products,
-                processed_orders,
-                processed_purchase_orders,
-                additional_info
-            FROM calculate_history 
-            ORDER BY start_time DESC 
-            LIMIT 20
-        `);
-        res.json(rows || []);
-    } catch (error) {
-        console.error('Error fetching calculate history:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// GET /status/modules - Get module calculation status
-router.get('/status/modules', async (req, res) => {
-    try {
-        const pool = req.app.locals.pool;
-        const { rows } = await pool.query(`
-            SELECT 
-                module_name, 
-                last_calculation_timestamp::timestamp 
-            FROM calculate_status 
-            ORDER BY module_name
-        `);
-        res.json(rows || []);
-    } catch (error) {
-        console.error('Error fetching module status:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// GET /status/tables - Get table sync status
-router.get('/status/tables', async (req, res) => {
-    try {
-        const pool = req.app.locals.pool;
-        const { rows } = await pool.query(`
-            SELECT 
-                table_name, 
-                last_sync_timestamp::timestamp 
-            FROM sync_status 
-            ORDER BY table_name
-        `);
-        res.json(rows || []);
-    } catch (error) {
-        console.error('Error fetching table status:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-// GET /status/table-counts - Get record counts for all tables
-router.get('/status/table-counts', async (req, res) => {
-    try {
-        const pool = req.app.locals.pool;
-        const tables = [
-            // Core tables
-            'products', 'categories', 'product_categories', 'orders', 'purchase_orders', 'receivings',
-            // New metrics tables
-            'product_metrics', 'daily_product_snapshots','brand_metrics','category_metrics','vendor_metrics',
-            // Config tables
-            'settings_global', 'settings_vendor', 'settings_product'
-        ];
-
-        const counts = await Promise.all(
-            tables.map(table => 
-                pool.query(`SELECT COUNT(*) as count FROM ${table}`)
-                    .then(result => ({
-                        table_name: table,
-                        count: parseInt(result.rows[0].count)
-                    }))
-                    .catch(err => ({
-                        table_name: table,
-                        count: null,
-                        error: err.message
-                    }))
-            )
-        );
-
-        // Group tables by type
-        const groupedCounts = {
-            core: counts.filter(c => ['products', 'categories', 'product_categories', 'orders', 'purchase_orders', 'receivings'].includes(c.table_name)),
-            metrics: counts.filter(c => ['product_metrics', 'daily_product_snapshots','brand_metrics','category_metrics','vendor_metrics'].includes(c.table_name)),
-            config: counts.filter(c => ['settings_global', 'settings_vendor', 'settings_product'].includes(c.table_name))
-        };
-
-        res.json(groupedCounts);
-    } catch (error) {
-        console.error('Error fetching table counts:', error);
-        res.status(500).json({ error: error.message });
-    }
-});
-
-module.exports = router; 

inventory-server/src/routes/import.js

@@ -5,6 +5,8 @@ const mysql = require('mysql2/promise');
 const multer = require('multer');
 const path = require('path');
 const fs = require('fs');
+const fsp = fs.promises;
+const sharp = require('sharp');
 
 // Create uploads directory if it doesn't exist
 const uploadsDir = path.join('/var/www/html/inventory/uploads/products');
@@ -35,6 +37,9 @@ const connectionCache = {
   }
 };
 
+const MIN_IMAGE_DIMENSION = 1000;
+const MAX_IMAGE_SIZE_BYTES = 5 * 1024 * 1024;
+
 // Function to schedule image deletion after 24 hours
 const scheduleImageDeletion = (filename, filePath) => {
   // Only schedule deletion for images in the products folder
@@ -145,6 +150,255 @@ const cleanupImagesOnStartup = () => {
 // Run cleanup on server start
 cleanupImagesOnStartup();
 
+const bytesToMegabytes = (bytes) => Number((bytes / (1024 * 1024)).toFixed(2));
+
+const processUploadedImage = async (filePath, mimetype) => {
+  const notices = [];
+  const legacyWarnings = [];
+  const metadata = {};
+
+  const originalBuffer = await fsp.readFile(filePath);
+  let baseMetadata = await sharp(originalBuffer, { failOn: 'none' }).metadata();
+
+  metadata.width = baseMetadata.width || 0;
+  metadata.height = baseMetadata.height || 0;
+  metadata.size = originalBuffer.length;
+  metadata.colorSpace = baseMetadata.space || baseMetadata.colourspace || null;
+
+  if (
+    baseMetadata.width &&
+    baseMetadata.height &&
+    (baseMetadata.width < MIN_IMAGE_DIMENSION || baseMetadata.height < MIN_IMAGE_DIMENSION)
+  ) {
+    const message = `Image is ${baseMetadata.width}x${baseMetadata.height}. Recommended minimum is ${MIN_IMAGE_DIMENSION}x${MIN_IMAGE_DIMENSION}.`;
+    notices.push({
+      message,
+      level: 'warning',
+      code: 'dimensions_too_small',
+      source: 'server'
+    });
+    legacyWarnings.push(message);
+  }
+
+  const colorSpace = (baseMetadata.space || baseMetadata.colourspace || '').toLowerCase();
+  let shouldConvertToRgb = colorSpace === 'cmyk';
+
+  if (shouldConvertToRgb) {
+    const message = 'Converted image from CMYK to RGB.';
+    notices.push({
+      message,
+      level: 'info',
+      code: 'converted_to_rgb',
+      source: 'server'
+    });
+    legacyWarnings.push(message);
+  }
+
+  const format = (baseMetadata.format || '').toLowerCase();
+  if (format === 'gif') {
+    if (metadata.size > MAX_IMAGE_SIZE_BYTES) {
+      const message = `GIF optimization is limited; resulting size is ${bytesToMegabytes(metadata.size)}MB (target 5MB).`;
+      notices.push({
+        message,
+        level: 'warning',
+        code: 'gif_size_limit',
+        source: 'server'
+      });
+      legacyWarnings.push(message);
+    }
+    metadata.convertedToRgb = false;
+    metadata.resized = false;
+    return { notices, warnings: legacyWarnings, metadata, finalSize: metadata.size };
+  }
+
+  const supportsQuality = ['jpeg', 'jpg', 'webp'].includes(format);
+  let targetQuality = supportsQuality ? 90 : undefined;
+  let finalQuality = undefined;
+
+  let currentWidth = baseMetadata.width || null;
+  let currentHeight = baseMetadata.height || null;
+
+  let resized = false;
+  let mutated = false;
+  let finalBuffer = originalBuffer;
+  let finalInfo = baseMetadata;
+
+  const encode = async ({ width, height, quality }) => {
+    let pipeline = sharp(originalBuffer, { failOn: 'none' });
+
+    if (shouldConvertToRgb) {
+      pipeline = pipeline.toColorspace('srgb');
+    }
+
+    if (width || height) {
+      pipeline = pipeline.resize({
+        width: width ?? undefined,
+        height: height ?? undefined,
+        fit: 'inside',
+        withoutEnlargement: true,
+      });
+    }
+
+    switch (format) {
+      case 'png':
+        pipeline = pipeline.png({
+          compressionLevel: 9,
+          adaptiveFiltering: true,
+          palette: true,
+        });
+        break;
+      case 'webp':
+        pipeline = pipeline.webp({ quality: quality ?? 90 });
+        break;
+      case 'jpeg':
+      case 'jpg':
+      default:
+        pipeline = pipeline.jpeg({ quality: quality ?? 90, mozjpeg: true });
+        break;
+    }
+
+    return pipeline.toBuffer({ resolveWithObject: true });
+  };
+
+  const canResize =
+    (currentWidth && currentWidth > MIN_IMAGE_DIMENSION) ||
+    (currentHeight && currentHeight > MIN_IMAGE_DIMENSION);
+
+  if (metadata.size > MAX_IMAGE_SIZE_BYTES && (supportsQuality || canResize)) {
+    const maxAttempts = 8;
+
+    for (let attempt = 0; attempt < maxAttempts; attempt++) {
+      let targetWidth = currentWidth;
+      let targetHeight = currentHeight;
+      let resizedThisAttempt = false;
+
+      if (currentWidth && currentWidth > MIN_IMAGE_DIMENSION) {
+        targetWidth = Math.max(MIN_IMAGE_DIMENSION, Math.round(currentWidth * 0.85));
+      }
+
+      if (currentHeight && currentHeight > MIN_IMAGE_DIMENSION) {
+        targetHeight = Math.max(MIN_IMAGE_DIMENSION, Math.round(currentHeight * 0.85));
+      }
+
+      if (
+        (targetWidth && currentWidth && targetWidth < currentWidth) ||
+        (targetHeight && currentHeight && targetHeight < currentHeight)
+      ) {
+        resized = true;
+        resizedThisAttempt = true;
+        currentWidth = targetWidth;
+        currentHeight = targetHeight;
+      } else if (!supportsQuality || (targetQuality && targetQuality <= 70)) {
+        // Cannot resize further and quality cannot be adjusted
+        break;
+      }
+
+      const qualityForAttempt = supportsQuality ? targetQuality : undefined;
+      const { data, info } = await encode({
+        width: currentWidth,
+        height: currentHeight,
+        quality: qualityForAttempt,
+      });
+
+      mutated = true;
+      finalBuffer = data;
+      finalInfo = info;
+      metadata.optimizedSize = data.length;
+      if (info.width) metadata.width = info.width;
+      if (info.height) metadata.height = info.height;
+      if (info.width) currentWidth = info.width;
+      if (info.height) currentHeight = info.height;
+
+      if (supportsQuality && qualityForAttempt) {
+        finalQuality = qualityForAttempt;
+      }
+
+      if (data.length <= MAX_IMAGE_SIZE_BYTES) {
+        break;
+      }
+
+      if (resizedThisAttempt) {
+        continue;
+      }
+
+      if (supportsQuality && targetQuality && targetQuality > 70) {
+        const nextQuality = Math.max(70, targetQuality - 10);
+        if (nextQuality === targetQuality) {
+          break;
+        }
+        targetQuality = nextQuality;
+        continue;
+      }
+
+      break;
+    }
+
+    if (finalBuffer.length > MAX_IMAGE_SIZE_BYTES) {
+      const message = `Optimized image remains ${bytesToMegabytes(finalBuffer.length)}MB (target 5MB).`;
+      notices.push({
+        message,
+        level: 'warning',
+        code: 'size_over_limit',
+        source: 'server'
+      });
+      legacyWarnings.push(message);
+    }
+  } else if (shouldConvertToRgb) {
+    const { data, info } = await encode({ width: currentWidth, height: currentHeight });
+    mutated = true;
+    finalBuffer = data;
+    finalInfo = info;
+    metadata.optimizedSize = data.length;
+    if (info.width) metadata.width = info.width;
+    if (info.height) metadata.height = info.height;
+    if (info.width) currentWidth = info.width;
+    if (info.height) currentHeight = info.height;
+  }
+
+  if (mutated) {
+    await fsp.writeFile(filePath, finalBuffer);
+    metadata.optimizedSize = finalBuffer.length;
+  } else {
+    // No transformation occurred; still need to ensure we report original stats
+    metadata.optimizedSize = metadata.size;
+  }
+
+  metadata.convertedToRgb = shouldConvertToRgb && mutated;
+  metadata.resized = resized;
+  if (finalQuality) {
+    metadata.quality = finalQuality;
+  }
+
+  if (resized && metadata.width && metadata.height) {
+    const message = `Image resized to ${metadata.width}x${metadata.height} during optimization.`;
+    notices.push({
+      message,
+      level: 'info',
+      code: 'resized',
+      source: 'server'
+    });
+    legacyWarnings.push(message);
+  }
+
+  if (finalQuality && finalQuality < 90) {
+    const message = `Image quality adjusted to ${finalQuality} to reduce file size.`;
+    notices.push({
+      message,
+      level: 'info',
+      code: 'quality_adjusted',
+      source: 'server'
+    });
+    legacyWarnings.push(message);
+  }
+
+  return {
+    notices,
+    warnings: legacyWarnings,
+    metadata,
+    finalSize: finalBuffer.length,
+  };
+};
+
 // Configure multer for file uploads
 const storage = multer.diskStorage({
   destination: function (req, file, cb) {
@@ -178,7 +432,7 @@ const storage = multer.diskStorage({
 const upload = multer({ 
   storage: storage,
   limits: {
-    fileSize: 5 * 1024 * 1024, // 5MB max file size
+    fileSize: 15 * 1024 * 1024, // Allow bigger uploads; processing will reduce to 5MB
   },
   fileFilter: function (req, file, cb) {
     // Accept only image files
@@ -345,7 +599,7 @@ async function setupSshTunnel() {
 }
 
 // Image upload endpoint
-router.post('/upload-image', upload.single('image'), (req, res) => {
+router.post('/upload-image', upload.single('image'), async (req, res) => {
   try {
     if (!req.file) {
       return res.status(400).json({ error: 'No image file provided' });
@@ -375,9 +629,13 @@ router.post('/upload-image', upload.single('image'), (req, res) => {
       }
     });
 
+    // Process the image (resize/compress/color-space) before responding
+    const processingResult = await processUploadedImage(filePath, req.file.mimetype);
+    req.file.size = processingResult.finalSize;
+
     // Create URL for the uploaded file - using an absolute URL with domain
-    // This will generate a URL like: https://inventory.acot.site/uploads/products/filename.jpg
-    const baseUrl = 'https://inventory.acot.site';
+    // This will generate a URL like: https://tools.acherryontop.com/uploads/products/filename.jpg
+    const baseUrl = 'https://tools.acherryontop.com';
     const imageUrl = `${baseUrl}/uploads/products/${req.file.filename}`;
     
     // Schedule this image for deletion in 24 hours
@@ -390,11 +648,24 @@ router.post('/upload-image', upload.single('image'), (req, res) => {
       fileName: req.file.filename,
       mimetype: req.file.mimetype,
       fullPath: filePath,
+      notices: processingResult.notices,
+      warnings: processingResult.warnings,
+      metadata: processingResult.metadata,
       message: 'Image uploaded successfully (will auto-delete after 24 hours)'
     });
     
   } catch (error) {
     console.error('Error uploading image:', error);
+    if (req?.file?.filename) {
+      const cleanupPath = path.join(uploadsDir, req.file.filename);
+      if (fs.existsSync(cleanupPath)) {
+        try {
+          fs.unlinkSync(cleanupPath);
+        } catch (cleanupError) {
+          console.error('Failed to remove file after processing error:', cleanupError);
+        }
+      }
+    }
     res.status(500).json({ error: error.message || 'Failed to upload image' });
   }
 });
@@ -444,6 +715,26 @@ router.delete('/delete-image', (req, res) => {
   }
 });
 
+// Clear all taxonomy caches
+router.post('/clear-taxonomy-cache', (req, res) => {
+  try {
+    // Clear all entries from the query cache
+    const cacheSize = connectionCache.queryCache.size;
+    connectionCache.queryCache.clear();
+
+    console.log(`Cleared ${cacheSize} entries from taxonomy cache`);
+
+    res.json({
+      success: true,
+      message: `Cache cleared (${cacheSize} entries removed)`,
+      clearedEntries: cacheSize
+    });
+  } catch (error) {
+    console.error('Error clearing taxonomy cache:', error);
+    res.status(500).json({ error: 'Failed to clear cache' });
+  }
+});
+
 // Get all options for import fields
 router.get('/field-options', async (req, res) => {
     try {
@@ -955,6 +1246,155 @@ router.get('/search-products', async (req, res) => {
   }
 });
 
+const UPC_SUPPLIER_PREFIX_LEADING_DIGIT = '4';
+const UPC_MAX_SEQUENCE = 99999;
+const UPC_RESERVATION_TTL = 5 * 60 * 1000; // 5 minutes
+
+function buildSupplierPrefix(supplierId) {
+  const numericId = Number.parseInt(String(supplierId), 10);
+  if (Number.isNaN(numericId) || numericId < 0) {
+    return null;
+  }
+
+  const padded = String(numericId).padStart(5, '0');
+  const prefix = `${UPC_SUPPLIER_PREFIX_LEADING_DIGIT}${padded}`;
+  return prefix.length === 6 ? prefix : null;
+}
+
+function calculateUpcCheckDigit(upcWithoutCheckDigit) {
+  if (!/^\d{11}$/.test(upcWithoutCheckDigit)) {
+    throw new Error('UPC body must be 11 numeric characters');
+  }
+
+  let sum = 0;
+  for (let i = 0; i < upcWithoutCheckDigit.length; i += 1) {
+    const digit = Number.parseInt(upcWithoutCheckDigit[i], 10);
+    sum += (i % 2 === 0) ? digit * 3 : digit;
+  }
+
+  const mod = sum % 10;
+  return mod === 0 ? 0 : 10 - mod;
+}
+
+const upcReservationCache = new Map();
+const upcGenerationLocks = new Map();
+
+function getReservedSequence(prefix) {
+  const entry = upcReservationCache.get(prefix);
+  if (!entry) {
+    return 0;
+  }
+
+  if (Date.now() > entry.expiresAt) {
+    upcReservationCache.delete(prefix);
+    return 0;
+  }
+
+  return entry.lastSequence;
+}
+
+function setReservedSequence(prefix, sequence) {
+  upcReservationCache.set(prefix, {
+    lastSequence: sequence,
+    expiresAt: Date.now() + UPC_RESERVATION_TTL
+  });
+}
+
+async function runWithSupplierLock(prefix, task) {
+  const previous = upcGenerationLocks.get(prefix) || Promise.resolve();
+  const chained = previous.catch(() => {}).then(() => task());
+  upcGenerationLocks.set(prefix, chained);
+
+  try {
+    return await chained;
+  } finally {
+    if (upcGenerationLocks.get(prefix) === chained) {
+      upcGenerationLocks.delete(prefix);
+    }
+  }
+}
+
+router.post('/generate-upc', async (req, res) => {
+  const { supplierId, increment } = req.body || {};
+
+  if (supplierId === undefined || supplierId === null || String(supplierId).trim() === '') {
+    return res.status(400).json({ error: 'Supplier ID is required to generate a UPC' });
+  }
+
+  const supplierPrefix = buildSupplierPrefix(supplierId);
+  if (!supplierPrefix) {
+    return res.status(400).json({ error: 'Supplier ID must be a non-negative number with at most 5 digits' });
+  }
+
+  const step = Number.parseInt(increment, 10);
+  const sequenceIncrement = Number.isNaN(step) || step < 1 ? 1 : step;
+
+  try {
+    const result = await runWithSupplierLock(supplierPrefix, async () => {
+      const { connection } = await getDbConnection();
+
+      const [rows] = await connection.query(
+        `SELECT CAST(SUBSTRING(upc,7,5) AS UNSIGNED) AS num
+           FROM products
+          WHERE LEFT(upc, 6) = ? AND LENGTH(upc) = 12
+       ORDER BY num DESC
+          LIMIT 1`,
+        [supplierPrefix]
+      );
+
+      const lastSequenceFromDb = rows && rows.length > 0 && rows[0].num !== null
+        ? Number.parseInt(rows[0].num, 10) || 0
+        : 0;
+
+      const cachedSequence = getReservedSequence(supplierPrefix);
+      const baselineSequence = Math.max(lastSequenceFromDb, cachedSequence);
+
+      let nextSequence = baselineSequence + sequenceIncrement;
+      let candidateUpc = null;
+      let attempts = 0;
+
+      while (attempts < 10 && nextSequence <= UPC_MAX_SEQUENCE) {
+        const sequencePart = String(nextSequence).padStart(5, '0');
+        const upcBody = `${supplierPrefix}${sequencePart}`;
+        const checkDigit = calculateUpcCheckDigit(upcBody);
+        const fullUpc = `${upcBody}${checkDigit}`;
+
+        const [existing] = await connection.query(
+          'SELECT 1 FROM products WHERE upc = ? LIMIT 1',
+          [fullUpc]
+        );
+
+        if (!existing || existing.length === 0) {
+          candidateUpc = { upc: fullUpc, sequence: nextSequence };
+          break;
+        }
+
+        nextSequence += 1;
+        attempts += 1;
+      }
+
+      if (!candidateUpc) {
+        const reason = nextSequence > UPC_MAX_SEQUENCE
+          ? 'UPC range exhausted for this supplier'
+          : 'Unable to find an available UPC';
+        const error = new Error(reason);
+        error.status = 409;
+        throw error;
+      }
+
+      setReservedSequence(supplierPrefix, candidateUpc.sequence);
+      return candidateUpc.upc;
+    });
+
+    return res.json({ success: true, upc: result });
+  } catch (error) {
+    console.error('Error generating UPC:', error);
+    const status = error.status && Number.isInteger(error.status) ? error.status : 500;
+    const message = status === 500 ? 'Failed to generate UPC' : error.message;
+    return res.status(status).json({ error: message, details: status === 500 ? error.message : undefined });
+  }
+});
+
 // Endpoint to check UPC and generate item number
 router.get('/check-upc-and-generate-sku', async (req, res) => {
   const { upc, supplierId } = req.query;
@@ -974,7 +1414,7 @@ router.get('/check-upc-and-generate-sku', async (req, res) => {
     
     if (upcCheck.length > 0) {
       return res.status(409).json({ 
-        error: 'UPC already exists', 
+        error: 'A product with this UPC already exists', 
         existingProductId: upcCheck[0].pid,
         existingItemNumber: upcCheck[0].itemnumber
       });
@@ -1149,4 +1589,4 @@ router.get('/product-categories/:pid', async (req, res) => {
   }
 });
 
-module.exports = router; 
+module.exports = router; 

inventory-server/src/routes/metrics.js

@@ -1,590 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const { Pool } = require('pg'); // Assuming pg driver
-
-// --- Configuration & Helpers ---
-
-const DEFAULT_PAGE_LIMIT = 50;
-const MAX_PAGE_LIMIT = 200; // Prevent excessive data requests
-
-// Define direct mapping from frontend column names to database columns
-// This simplifies the code by eliminating conversion logic
-const COLUMN_MAP = {
-    // Product Info
-    pid: 'pm.pid',
-    sku: 'pm.sku',
-    title: 'pm.title',
-    brand: 'pm.brand',
-    vendor: 'pm.vendor',
-    imageUrl: 'pm.image_url',
-    isVisible: 'pm.is_visible',
-    isReplenishable: 'pm.is_replenishable',
-    // Additional Product Fields
-    barcode: 'pm.barcode',
-    harmonizedTariffCode: 'pm.harmonized_tariff_code',
-    vendorReference: 'pm.vendor_reference',
-    notionsReference: 'pm.notions_reference',
-    line: 'pm.line',
-    subline: 'pm.subline',
-    artist: 'pm.artist',
-    moq: 'pm.moq',
-    rating: 'pm.rating',
-    reviews: 'pm.reviews',
-    weight: 'pm.weight',
-    length: 'pm.length',
-    width: 'pm.width',
-    height: 'pm.height',
-    countryOfOrigin: 'pm.country_of_origin',
-    location: 'pm.location',
-    baskets: 'pm.baskets',
-    notifies: 'pm.notifies',
-    preorderCount: 'pm.preorder_count',
-    notionsInvCount: 'pm.notions_inv_count',
-    // Current Status
-    currentPrice: 'pm.current_price',
-    currentRegularPrice: 'pm.current_regular_price',
-    currentCostPrice: 'pm.current_cost_price',
-    currentLandingCostPrice: 'pm.current_landing_cost_price',
-    currentStock: 'pm.current_stock',
-    currentStockCost: 'pm.current_stock_cost',
-    currentStockRetail: 'pm.current_stock_retail',
-    currentStockGross: 'pm.current_stock_gross',
-    onOrderQty: 'pm.on_order_qty',
-    onOrderCost: 'pm.on_order_cost',
-    onOrderRetail: 'pm.on_order_retail',
-    earliestExpectedDate: 'pm.earliest_expected_date',
-    // Historical Dates
-    dateCreated: 'pm.date_created',
-    dateFirstReceived: 'pm.date_first_received',
-    dateLastReceived: 'pm.date_last_received',
-    dateFirstSold: 'pm.date_first_sold',
-    dateLastSold: 'pm.date_last_sold',
-    ageDays: 'pm.age_days',
-    // Rolling Period Metrics
-    sales7d: 'pm.sales_7d', 
-    revenue7d: 'pm.revenue_7d',
-    sales14d: 'pm.sales_14d', 
-    revenue14d: 'pm.revenue_14d',
-    sales30d: 'pm.sales_30d', 
-    revenue30d: 'pm.revenue_30d',
-    cogs30d: 'pm.cogs_30d', 
-    profit30d: 'pm.profit_30d',
-    returnsUnits30d: 'pm.returns_units_30d', 
-    returnsRevenue30d: 'pm.returns_revenue_30d',
-    discounts30d: 'pm.discounts_30d', 
-    grossRevenue30d: 'pm.gross_revenue_30d',
-    grossRegularRevenue30d: 'pm.gross_regular_revenue_30d',
-    stockoutDays30d: 'pm.stockout_days_30d',
-    sales365d: 'pm.sales_365d', 
-    revenue365d: 'pm.revenue_365d',
-    avgStockUnits30d: 'pm.avg_stock_units_30d', 
-    avgStockCost30d: 'pm.avg_stock_cost_30d',
-    avgStockRetail30d: 'pm.avg_stock_retail_30d', 
-    avgStockGross30d: 'pm.avg_stock_gross_30d',
-    receivedQty30d: 'pm.received_qty_30d', 
-    receivedCost30d: 'pm.received_cost_30d',
-    // Lifetime Metrics
-    lifetimeSales: 'pm.lifetime_sales', 
-    lifetimeRevenue: 'pm.lifetime_revenue',
-    // First Period Metrics
-    first7DaysSales: 'pm.first_7_days_sales', 
-    first7DaysRevenue: 'pm.first_7_days_revenue',
-    first30DaysSales: 'pm.first_30_days_sales', 
-    first30DaysRevenue: 'pm.first_30_days_revenue',
-    first60DaysSales: 'pm.first_60_days_sales', 
-    first60DaysRevenue: 'pm.first_60_days_revenue',
-    first90DaysSales: 'pm.first_90_days_sales', 
-    first90DaysRevenue: 'pm.first_90_days_revenue',
-    // Calculated KPIs
-    asp30d: 'pm.asp_30d', 
-    acp30d: 'pm.acp_30d', 
-    avgRos30d: 'pm.avg_ros_30d',
-    avgSalesPerDay30d: 'pm.avg_sales_per_day_30d', 
-    avgSalesPerMonth30d: 'pm.avg_sales_per_month_30d',
-    margin30d: 'pm.margin_30d', 
-    markup30d: 'pm.markup_30d', 
-    gmroi30d: 'pm.gmroi_30d',
-    stockturn30d: 'pm.stockturn_30d', 
-    returnRate30d: 'pm.return_rate_30d',
-    discountRate30d: 'pm.discount_rate_30d', 
-    stockoutRate30d: 'pm.stockout_rate_30d',
-    markdown30d: 'pm.markdown_30d', 
-    markdownRate30d: 'pm.markdown_rate_30d',
-    sellThrough30d: 'pm.sell_through_30d', 
-    avgLeadTimeDays: 'pm.avg_lead_time_days',
-    // Forecasting & Replenishment
-    abcClass: 'pm.abc_class', 
-    salesVelocityDaily: 'pm.sales_velocity_daily',
-    configLeadTime: 'pm.config_lead_time', 
-    configDaysOfStock: 'pm.config_days_of_stock',
-    configSafetyStock: 'pm.config_safety_stock', 
-    planningPeriodDays: 'pm.planning_period_days',
-    leadTimeForecastUnits: 'pm.lead_time_forecast_units', 
-    daysOfStockForecastUnits: 'pm.days_of_stock_forecast_units',
-    planningPeriodForecastUnits: 'pm.planning_period_forecast_units', 
-    leadTimeClosingStock: 'pm.lead_time_closing_stock',
-    daysOfStockClosingStock: 'pm.days_of_stock_closing_stock', 
-    replenishmentNeededRaw: 'pm.replenishment_needed_raw',
-    replenishmentUnits: 'pm.replenishment_units', 
-    replenishmentCost: 'pm.replenishment_cost',
-    replenishmentRetail: 'pm.replenishment_retail', 
-    replenishmentProfit: 'pm.replenishment_profit',
-    toOrderUnits: 'pm.to_order_units', 
-    forecastLostSalesUnits: 'pm.forecast_lost_sales_units',
-    forecastLostRevenue: 'pm.forecast_lost_revenue', 
-    stockCoverInDays: 'pm.stock_cover_in_days',
-    poCoverInDays: 'pm.po_cover_in_days', 
-    sellsOutInDays: 'pm.sells_out_in_days',
-    replenishDate: 'pm.replenish_date', 
-    overstockedUnits: 'pm.overstocked_units',
-    overstockedCost: 'pm.overstocked_cost', 
-    overstockedRetail: 'pm.overstocked_retail',
-    isOldStock: 'pm.is_old_stock',
-    // Yesterday
-    yesterdaySales: 'pm.yesterday_sales',
-    // Map status column - directly mapped now instead of calculated on frontend
-    status: 'pm.status',
-    
-    // Growth Metrics (P3)
-    salesGrowth30dVsPrev: 'pm.sales_growth_30d_vs_prev',
-    revenueGrowth30dVsPrev: 'pm.revenue_growth_30d_vs_prev',
-    salesGrowthYoy: 'pm.sales_growth_yoy',
-    revenueGrowthYoy: 'pm.revenue_growth_yoy',
-    
-    // Demand Variability Metrics (P3)
-    salesVariance30d: 'pm.sales_variance_30d',
-    salesStdDev30d: 'pm.sales_std_dev_30d',
-    salesCv30d: 'pm.sales_cv_30d',
-    demandPattern: 'pm.demand_pattern',
-    
-    // Service Level Metrics (P5)
-    fillRate30d: 'pm.fill_rate_30d',
-    stockoutIncidents30d: 'pm.stockout_incidents_30d',
-    serviceLevel30d: 'pm.service_level_30d',
-    lostSalesIncidents30d: 'pm.lost_sales_incidents_30d',
-    
-    // Seasonality Metrics (P5) 
-    seasonalityIndex: 'pm.seasonality_index',
-    seasonalPattern: 'pm.seasonal_pattern',
-    peakSeason: 'pm.peak_season',
-    
-    // Lifetime Revenue Quality
-    lifetimeRevenueQuality: 'pm.lifetime_revenue_quality'
-};
-
-// Define column types for use in sorting/filtering
-// This helps apply correct comparison operators and sorting logic
-const COLUMN_TYPES = {
-    // Numeric columns (use numeric operators and sorting)
-    numeric: [
-        'pid', 'currentPrice', 'currentRegularPrice', 'currentCostPrice', 'currentLandingCostPrice',
-        'currentStock', 'currentStockCost', 'currentStockRetail', 'currentStockGross',
-        'onOrderQty', 'onOrderCost', 'onOrderRetail', 'ageDays',
-        'sales7d', 'revenue7d', 'sales14d', 'revenue14d', 'sales30d', 'revenue30d',
-        'cogs30d', 'profit30d', 'returnsUnits30d', 'returnsRevenue30d', 'discounts30d',
-        'grossRevenue30d', 'grossRegularRevenue30d', 'stockoutDays30d', 'sales365d', 'revenue365d',
-        'avgStockUnits30d', 'avgStockCost30d', 'avgStockRetail30d', 'avgStockGross30d',
-        'receivedQty30d', 'receivedCost30d', 'lifetimeSales', 'lifetimeRevenue',
-        'first7DaysSales', 'first7DaysRevenue', 'first30DaysSales', 'first30DaysRevenue',
-        'first60DaysSales', 'first60DaysRevenue', 'first90DaysSales', 'first90DaysRevenue',
-        'asp30d', 'acp30d', 'avgRos30d', 'avgSalesPerDay30d', 'avgSalesPerMonth30d',
-        'margin30d', 'markup30d', 'gmroi30d', 'stockturn30d', 'returnRate30d', 'discountRate30d',
-        'stockoutRate30d', 'markdown30d', 'markdownRate30d', 'sellThrough30d', 'avgLeadTimeDays',
-        'salesVelocityDaily', 'configLeadTime', 'configDaysOfStock', 'configSafetyStock',
-        'planningPeriodDays', 'leadTimeForecastUnits', 'daysOfStockForecastUnits',
-        'planningPeriodForecastUnits', 'leadTimeClosingStock', 'daysOfStockClosingStock',
-        'replenishmentNeededRaw', 'replenishmentUnits', 'replenishmentCost', 'replenishmentRetail',
-        'replenishmentProfit', 'toOrderUnits', 'forecastLostSalesUnits', 'forecastLostRevenue',
-        'stockCoverInDays', 'poCoverInDays', 'sellsOutInDays', 'overstockedUnits',
-        'overstockedCost', 'overstockedRetail', 'yesterdaySales',
-        // New numeric columns
-        'moq', 'rating', 'reviews', 'weight', 'length', 'width', 'height', 
-        'baskets', 'notifies', 'preorderCount', 'notionsInvCount',
-        // Growth metrics
-        'salesGrowth30dVsPrev', 'revenueGrowth30dVsPrev', 'salesGrowthYoy', 'revenueGrowthYoy',
-        // Demand variability metrics
-        'salesVariance30d', 'salesStdDev30d', 'salesCv30d',
-        // Service level metrics
-        'fillRate30d', 'stockoutIncidents30d', 'serviceLevel30d', 'lostSalesIncidents30d',
-        // Seasonality metrics
-        'seasonalityIndex'
-    ],
-    // Date columns (use date operators and sorting)
-    date: [
-        'dateCreated', 'dateFirstReceived', 'dateLastReceived', 'dateFirstSold', 'dateLastSold',
-        'earliestExpectedDate', 'replenishDate', 'forecastedOutOfStockDate'
-    ],
-    // String columns (use string operators and sorting)
-    string: [
-        'sku', 'title', 'brand', 'vendor', 'imageUrl', 'abcClass', 'status',
-        // New string columns
-        'barcode', 'harmonizedTariffCode', 'vendorReference', 'notionsReference',
-        'line', 'subline', 'artist', 'countryOfOrigin', 'location',
-        // New string columns for patterns
-        'demandPattern', 'seasonalPattern', 'peakSeason', 'lifetimeRevenueQuality'
-    ],
-    // Boolean columns (use boolean operators and sorting)
-    boolean: ['isVisible', 'isReplenishable', 'isOldStock']
-};
-
-// Special sort handling for certain columns
-const SPECIAL_SORT_COLUMNS = {
-    // Percentage columns where we want to sort by the numeric value
-    margin30d: true,
-    markup30d: true,
-    sellThrough30d: true,
-    discountRate30d: true,
-    stockoutRate30d: true,
-    returnRate30d: true,
-    markdownRate30d: true,
-    
-    // Columns where we may want to sort by absolute value
-    profit30d: 'abs',
-    
-    // Velocity columns
-    salesVelocityDaily: true,
-    
-    // Growth rate columns
-    salesGrowth30dVsPrev: 'abs',
-    revenueGrowth30dVsPrev: 'abs',
-    salesGrowthYoy: 'abs',
-    revenueGrowthYoy: 'abs',
-    
-    // Status column needs special ordering
-    status: 'priority'
-};
-
-// Status priority for sorting (lower number = higher priority)
-const STATUS_PRIORITY = {
-    'Critical': 1,
-    'At Risk': 2,
-    'Reorder': 3,
-    'Overstocked': 4,
-    'Healthy': 5,
-    'New': 6
-    // Any other status will be sorted alphabetically after these
-};
-
-// Get database column name from frontend column name
-function getDbColumn(frontendColumn) {
-    return COLUMN_MAP[frontendColumn] || 'pm.title'; // Default to title if not found
-}
-
-// Get column type for proper sorting
-function getColumnType(frontendColumn) {
-    return COLUMN_TYPES[frontendColumn] || 'string';
-}
-
-// --- Route Handlers ---
-
-// GET /metrics/filter-options - Provide distinct values for filter dropdowns
-router.get('/filter-options', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /metrics/filter-options');
-    try {
-        const [vendorRes, brandRes, abcClassRes] = await Promise.all([
-            pool.query(`SELECT DISTINCT vendor FROM public.product_metrics WHERE vendor IS NOT NULL AND vendor <> '' ORDER BY vendor`),
-            pool.query(`SELECT DISTINCT COALESCE(brand, 'Unbranded') as brand FROM public.product_metrics WHERE brand IS NOT NULL AND brand <> '' ORDER BY brand`),
-            pool.query(`SELECT DISTINCT abc_class FROM public.product_metrics WHERE abc_class IS NOT NULL ORDER BY abc_class`)
-            // Add queries for other distinct options if needed (e.g., categories if stored on pm)
-        ]);
-
-        res.json({
-            vendors: vendorRes.rows.map(r => r.vendor),
-            brands: brandRes.rows.map(r => r.brand),
-            abcClasses: abcClassRes.rows.map(r => r.abc_class),
-        });
-    } catch (error) {
-        console.error('Error fetching filter options:', error);
-        res.status(500).json({ error: 'Failed to fetch filter options' });
-    }
-});
-
-
-// GET /metrics/ - List all product metrics with filtering, sorting, pagination
-router.get('/', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /metrics received query:', req.query);
-
-    try {
-        // --- Pagination ---
-        let page = parseInt(req.query.page, 10);
-        let limit = parseInt(req.query.limit, 10);
-        if (isNaN(page) || page < 1) page = 1;
-        if (isNaN(limit) || limit < 1) limit = DEFAULT_PAGE_LIMIT;
-        limit = Math.min(limit, MAX_PAGE_LIMIT); // Cap the limit
-        const offset = (page - 1) * limit;
-
-        // --- Sorting ---
-        const sortQueryKey = req.query.sort || 'title'; // Default sort field key
-        const dbColumn = getDbColumn(sortQueryKey);
-        const columnType = getColumnType(sortQueryKey);
-        
-        console.log(`Sorting request: ${sortQueryKey} -> ${dbColumn} (${columnType})`);
-        
-        const sortDirection = req.query.order?.toLowerCase() === 'desc' ? 'DESC' : 'ASC';
-
-        // Always put nulls last regardless of sort direction or column type
-        const nullsOrder = 'NULLS LAST';
-
-        // Build the ORDER BY clause based on column type and special handling
-        let orderByClause;
-
-        if (SPECIAL_SORT_COLUMNS[sortQueryKey] === 'abs') {
-            // Sort by absolute value for columns where negative values matter
-            orderByClause = `ABS(${dbColumn}::numeric) ${sortDirection} ${nullsOrder}`;
-        } else if (columnType === 'number' || SPECIAL_SORT_COLUMNS[sortQueryKey] === true) {
-            // For numeric columns, cast to numeric to ensure proper sorting
-            orderByClause = `${dbColumn}::numeric ${sortDirection} ${nullsOrder}`;
-        } else if (columnType === 'date') {
-            // For date columns, cast to timestamp to ensure proper sorting
-            orderByClause = `CASE WHEN ${dbColumn} IS NULL THEN 1 ELSE 0 END, ${dbColumn}::timestamp ${sortDirection}`;
-        } else if (columnType === 'status' || SPECIAL_SORT_COLUMNS[sortQueryKey] === 'priority') {
-            // Special handling for status column, using priority for known statuses
-            orderByClause = `
-                CASE WHEN ${dbColumn} IS NULL THEN 999
-                     WHEN ${dbColumn} = 'Critical' THEN 1
-                     WHEN ${dbColumn} = 'At Risk' THEN 2
-                     WHEN ${dbColumn} = 'Reorder' THEN 3
-                     WHEN ${dbColumn} = 'Overstocked' THEN 4
-                     WHEN ${dbColumn} = 'Healthy' THEN 5
-                     WHEN ${dbColumn} = 'New' THEN 6
-                     ELSE 100
-                END ${sortDirection} ${nullsOrder},
-                ${dbColumn} ${sortDirection}`;
-        } else {
-            // For string and boolean columns, no special casting needed
-            orderByClause = `CASE WHEN ${dbColumn} IS NULL THEN 1 ELSE 0 END, ${dbColumn} ${sortDirection}`;
-        }
-
-        // --- Filtering ---
-        const conditions = [];
-        const params = [];
-        let paramCounter = 1;
-
-        // Add default visibility/replenishable filters unless overridden
-        if (req.query.showInvisible !== 'true') conditions.push(`pm.is_visible = true`);
-        if (req.query.showNonReplenishable !== 'true') conditions.push(`pm.is_replenishable = true`);
-
-        // Special handling for stock_status
-        if (req.query.stock_status) {
-            const status = req.query.stock_status;
-            // Handle special case for "at-risk" which is stored as "At Risk" in the database
-            if (status.toLowerCase() === 'at-risk') {
-                conditions.push(`pm.status = $${paramCounter++}`);
-                params.push('At Risk');
-            } else {
-                // Capitalize first letter to match database values
-                conditions.push(`pm.status = $${paramCounter++}`);
-                params.push(status.charAt(0).toUpperCase() + status.slice(1));
-            }
-        }
-
-        // Process other filters from query parameters
-        for (const key in req.query) {
-            // Skip control params
-            if (['page', 'limit', 'sort', 'order', 'showInvisible', 'showNonReplenishable', 'stock_status'].includes(key)) continue;
-
-            let filterKey = key;
-            let operator = '='; // Default operator
-            let value = req.query[key];
-
-            // Check for operator suffixes (e.g., sales30d_gt, title_like)
-            const operatorMatch = key.match(/^(.*)_(eq|ne|gt|gte|lt|lte|like|ilike|between|in)$/);
-            if (operatorMatch) {
-                filterKey = operatorMatch[1]; // e.g., "sales30d"
-                operator = operatorMatch[2]; // e.g., "gt"
-            }
-
-            // Get the database column for this filter key
-            const dbColumn = getDbColumn(filterKey);
-            const valueType = getColumnType(filterKey);
-            
-            if (!dbColumn) {
-                console.warn(`Invalid filter key ignored: ${key}`);
-                continue; // Skip if the key doesn't map to a known column
-            }
-
-            // --- Build WHERE clause fragment ---
-            try {
-                let conditionFragment = '';
-                let needsParam = true; // Most operators need a parameter
-
-                switch (operator.toLowerCase()) {
-                    case 'eq': operator = '='; break;
-                    case 'ne': operator = '<>'; break;
-                    case 'gt': operator = '>'; break;
-                    case 'gte': operator = '>='; break;
-                    case 'lt': operator = '<'; break;
-                    case 'lte': operator = '<='; break;
-                    case 'like': operator = 'LIKE'; value = `%${value}%`; break; // Add wildcards for LIKE
-                    case 'ilike': operator = 'ILIKE'; value = `%${value}%`; break; // Add wildcards for ILIKE
-                    case 'between':
-                        const [val1, val2] = String(value).split(',');
-                        if (val1 !== undefined && val2 !== undefined) {
-                             conditionFragment = `${dbColumn} BETWEEN $${paramCounter++} AND $${paramCounter++}`;
-                             params.push(parseValue(val1, valueType), parseValue(val2, valueType));
-                             needsParam = false; // Params added manually
-                        } else {
-                             console.warn(`Invalid 'between' value for ${key}: ${value}`);
-                             continue; // Skip this filter
-                        }
-                        break;
-                    case 'in':
-                        const inValues = String(value).split(',');
-                        if (inValues.length > 0) {
-                            const placeholders = inValues.map(() => `$${paramCounter++}`).join(', ');
-                            conditionFragment = `${dbColumn} IN (${placeholders})`;
-                            params.push(...inValues.map(v => parseValue(v, valueType))); // Add all parsed values
-                             needsParam = false; // Params added manually
-                        } else {
-                             console.warn(`Invalid 'in' value for ${key}: ${value}`);
-                             continue; // Skip this filter
-                        }
-                        break;
-                    // Add other operators as needed (IS NULL, IS NOT NULL, etc.)
-                    case '=': // Keep default '='
-                    default: operator = '='; break; // Ensure default is handled
-                }
-
-                if (needsParam) {
-                    conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                    params.push(parseValue(value, valueType));
-                }
-
-                if (conditionFragment) {
-                    conditions.push(`(${conditionFragment})`); // Wrap condition in parentheses
-                }
-
-            } catch (parseError) {
-                 console.warn(`Skipping filter for key "${key}" due to parsing error: ${parseError.message}`);
-                 // Decrement counter if param wasn't actually used due to error
-                 if (needsParam) paramCounter--;
-            }
-        }
-
-        // --- Construct and Execute Queries ---
-        const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : '';
-
-        // Debug log of conditions and parameters
-        console.log('Constructed WHERE conditions:', conditions);
-        console.log('Parameters:', params);
-
-        // Count Query
-        const countSql = `SELECT COUNT(*) AS total FROM public.product_metrics pm ${whereClause}`;
-        console.log('Executing Count Query:', countSql, params);
-        const countPromise = pool.query(countSql, params);
-
-        // Data Query (Select all columns from metrics table for now)
-        const dataSql = `
-            SELECT pm.*
-            FROM public.product_metrics pm
-            ${whereClause}
-            ORDER BY ${orderByClause}
-            LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-        `;
-        const dataParams = [...params, limit, offset];
-
-        // Log detailed query information for debugging
-        console.log('Executing Data Query:');
-        console.log('  - Sort Column:', dbColumn);
-        console.log('  - Column Type:', columnType);
-        console.log('  - Sort Direction:', sortDirection);
-        console.log('  - Order By Clause:', orderByClause);
-        console.log('  - Full SQL:', dataSql);
-        console.log('  - Parameters:', dataParams);
-
-        const dataPromise = pool.query(dataSql, dataParams);
-
-        // Execute queries in parallel
-        const [countResult, dataResult] = await Promise.all([countPromise, dataPromise]);
-
-        const total = parseInt(countResult.rows[0].total, 10);
-        const metrics = dataResult.rows;
-        console.log(`Total: ${total}, Fetched: ${metrics.length} for page ${page}`);
-
-        // --- Respond ---
-        res.json({
-            metrics,
-            pagination: {
-                total,
-                pages: Math.ceil(total / limit),
-                currentPage: page,
-                limit,
-            },
-            // Optionally include applied filters/sort for frontend confirmation
-             appliedQuery: {
-                 filters: req.query, // Send back raw query filters
-                 sort: sortQueryKey,
-                 order: sortDirection.toLowerCase()
-             }
-        });
-
-    } catch (error) {
-        console.error('Error fetching metrics list:', error);
-        res.status(500).json({ error: 'Failed to fetch product metrics list.' });
-    }
-});
-
-// GET /metrics/:pid - Get metrics for a single product
-router.get('/:pid', async (req, res) => {
-    const pool = req.app.locals.pool;
-    const pid = parseInt(req.params.pid, 10);
-
-    if (isNaN(pid)) {
-        return res.status(400).json({ error: 'Invalid Product ID.' });
-    }
-
-    console.log(`GET /metrics/${pid}`);
-    try {
-        const { rows } = await pool.query(
-            `SELECT * FROM public.product_metrics WHERE pid = $1`,
-            [pid]
-        );
-
-        if (rows.length === 0) {
-            console.log(`Metrics not found for PID: ${pid}`);
-            return res.status(404).json({ error: 'Metrics not found for this product.' });
-        }
-
-        console.log(`Metrics found for PID: ${pid}`);
-        // Data is pre-calculated, return the first (only) row
-        res.json(rows[0]);
-
-    } catch (error) {
-        console.error(`Error fetching metrics for PID ${pid}:`, error);
-        res.status(500).json({ error: 'Failed to fetch product metrics.' });
-    }
-});
-
-
-/**
- * Parses a value based on its expected type.
- * Throws error for invalid formats.
- */
-function parseValue(value, type) {
-    if (value === null || value === undefined || value === '') return null; // Allow empty strings? Or handle differently?
-
-    switch (type) {
-        case 'number':
-            const num = parseFloat(value);
-            if (isNaN(num)) throw new Error(`Invalid number format: "${value}"`);
-            return num;
-        case 'boolean':
-            if (String(value).toLowerCase() === 'true') return true;
-            if (String(value).toLowerCase() === 'false') return false;
-            throw new Error(`Invalid boolean format: "${value}"`);
-        case 'date':
-            // Basic validation, rely on DB to handle actual date conversion
-            if (!String(value).match(/^\d{4}-\d{2}-\d{2}$/)) {
-               // Allow full timestamps too? Adjust regex if needed
-               // console.warn(`Potentially invalid date format: "${value}"`); // Warn instead of throwing?
-            }
-            return String(value); // Send as string, let DB handle it
-        case 'string':
-        default:
-            return String(value);
-    }
-}
-
-module.exports = router;

inventory-server/src/routes/orders.js

@@ -1,261 +0,0 @@
-const express = require('express');
-const router = express.Router();
-
-// Get all orders with pagination, filtering, and sorting
-router.get('/', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const page = parseInt(req.query.page) || 1;
-    const limit = parseInt(req.query.limit) || 50;
-    const offset = (page - 1) * limit;
-    const search = req.query.search || '';
-    const status = req.query.status || 'all';
-    const fromDate = req.query.fromDate ? new Date(req.query.fromDate) : null;
-    const toDate = req.query.toDate ? new Date(req.query.toDate) : null;
-    const minAmount = parseFloat(req.query.minAmount) || 0;
-    const maxAmount = req.query.maxAmount ? parseFloat(req.query.maxAmount) : null;
-    const sortColumn = req.query.sortColumn || 'date';
-    const sortDirection = req.query.sortDirection === 'desc' ? 'DESC' : 'ASC';
-
-    // Build the WHERE clause
-    const conditions = ['o1.canceled = false'];
-    const params = [];
-    let paramCounter = 1;
-
-    if (search) {
-      conditions.push(`(o1.order_number ILIKE $${paramCounter} OR o1.customer ILIKE $${paramCounter})`);
-      params.push(`%${search}%`);
-      paramCounter++;
-    }
-
-    if (status !== 'all') {
-      conditions.push(`o1.status = $${paramCounter}`);
-      params.push(status);
-      paramCounter++;
-    }
-
-    if (fromDate) {
-      conditions.push(`DATE(o1.date) >= DATE($${paramCounter})`);
-      params.push(fromDate.toISOString());
-      paramCounter++;
-    }
-
-    if (toDate) {
-      conditions.push(`DATE(o1.date) <= DATE($${paramCounter})`);
-      params.push(toDate.toISOString());
-      paramCounter++;
-    }
-
-    if (minAmount > 0) {
-      conditions.push(`total_amount >= $${paramCounter}`);
-      params.push(minAmount);
-      paramCounter++;
-    }
-
-    if (maxAmount) {
-      conditions.push(`total_amount <= $${paramCounter}`);
-      params.push(maxAmount);
-      paramCounter++;
-    }
-
-    // Get total count for pagination
-    const { rows: [countResult] } = await pool.query(`
-      SELECT COUNT(DISTINCT o1.order_number) as total 
-      FROM orders o1
-      LEFT JOIN (
-        SELECT order_number, SUM(price * quantity) as total_amount
-        FROM orders
-        GROUP BY order_number
-      ) totals ON o1.order_number = totals.order_number
-      WHERE ${conditions.join(' AND ')}
-    `, params);
-    
-    const total = countResult.total;
-
-    // Get paginated results
-    const query = `
-      SELECT 
-        o1.order_number,
-        o1.customer,
-        o1.date,
-        o1.status,
-        o1.payment_method,
-        o1.shipping_method,
-        COUNT(o2.pid) as items_count,
-        ROUND(SUM(o2.price * o2.quantity)::numeric, 3) as total_amount
-      FROM orders o1
-      JOIN orders o2 ON o1.order_number = o2.order_number
-      WHERE ${conditions.join(' AND ')}
-      GROUP BY 
-        o1.order_number,
-        o1.customer,
-        o1.date,
-        o1.status,
-        o1.payment_method,
-        o1.shipping_method
-      ORDER BY ${
-        sortColumn === 'items_count' || sortColumn === 'total_amount' 
-          ? `${sortColumn} ${sortDirection}`
-          : `o1.${sortColumn} ${sortDirection}`
-      }
-      LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-    `;
-
-    params.push(limit, offset);
-    const { rows } = await pool.query(query, params);
-
-    // Get order statistics
-    const { rows: [orderStats] } = await pool.query(`
-      WITH CurrentStats AS (
-        SELECT 
-          COUNT(DISTINCT order_number) as total_orders,
-          ROUND(SUM(price * quantity)::numeric, 3) as total_revenue
-        FROM orders
-        WHERE canceled = false
-          AND DATE(date) >= CURRENT_DATE - INTERVAL '30 days'
-      ),
-      PreviousStats AS (
-        SELECT 
-          COUNT(DISTINCT order_number) as prev_orders,
-          ROUND(SUM(price * quantity)::numeric, 3) as prev_revenue
-        FROM orders
-        WHERE canceled = false
-          AND DATE(date) BETWEEN CURRENT_DATE - INTERVAL '60 days' AND CURRENT_DATE - INTERVAL '30 days'
-      ),
-      OrderValues AS (
-        SELECT 
-          order_number,
-          ROUND(SUM(price * quantity)::numeric, 3) as order_value
-        FROM orders
-        WHERE canceled = false
-          AND DATE(date) >= CURRENT_DATE - INTERVAL '30 days'
-        GROUP BY order_number
-      )
-      SELECT 
-        cs.total_orders,
-        cs.total_revenue,
-        CASE 
-          WHEN ps.prev_orders > 0 
-          THEN ROUND(((cs.total_orders - ps.prev_orders)::numeric / ps.prev_orders * 100), 1)
-          ELSE 0 
-        END as order_growth,
-        CASE 
-          WHEN ps.prev_revenue > 0 
-          THEN ROUND(((cs.total_revenue - ps.prev_revenue)::numeric / ps.prev_revenue * 100), 1)
-          ELSE 0 
-        END as revenue_growth,
-        CASE 
-          WHEN cs.total_orders > 0 
-          THEN ROUND((cs.total_revenue::numeric / cs.total_orders), 3)
-          ELSE 0 
-        END as average_order_value,
-        CASE 
-          WHEN ps.prev_orders > 0 
-          THEN ROUND((ps.prev_revenue::numeric / ps.prev_orders), 3)
-          ELSE 0 
-        END as prev_average_order_value
-      FROM CurrentStats cs
-      CROSS JOIN PreviousStats ps
-    `);
-    
-    res.json({
-      orders: rows.map(row => ({
-        ...row,
-        total_amount: parseFloat(row.total_amount) || 0,
-        items_count: parseInt(row.items_count) || 0,
-        date: row.date
-      })),
-      pagination: {
-        total,
-        pages: Math.ceil(total / limit),
-        currentPage: page,
-        limit
-      },
-      stats: {
-        totalOrders: parseInt(orderStats.total_orders) || 0,
-        totalRevenue: parseFloat(orderStats.total_revenue) || 0,
-        orderGrowth: parseFloat(orderStats.order_growth) || 0,
-        revenueGrowth: parseFloat(orderStats.revenue_growth) || 0,
-        averageOrderValue: parseFloat(orderStats.average_order_value) || 0,
-        aovGrowth: orderStats.prev_average_order_value > 0 
-          ? ((orderStats.average_order_value - orderStats.prev_average_order_value) / orderStats.prev_average_order_value * 100)
-          : 0,
-        conversionRate: 2.5, // Placeholder - would need actual visitor data
-        conversionGrowth: 0.5 // Placeholder - would need actual visitor data
-      }
-    });
-  } catch (error) {
-    console.error('Error fetching orders:', error);
-    res.status(500).json({ error: 'Failed to fetch orders' });
-  }
-});
-
-// Get a single order with its items
-router.get('/:orderNumber', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    // Get order details
-    const { rows: orderRows } = await pool.query(`
-      SELECT DISTINCT
-        o1.order_number,
-        o1.customer,
-        o1.date,
-        o1.status,
-        o1.payment_method,
-        o1.shipping_method,
-        o1.shipping_address,
-        o1.billing_address,
-        COUNT(o2.pid) as items_count,
-        ROUND(SUM(o2.price * o2.quantity)::numeric, 3) as total_amount
-      FROM orders o1
-      JOIN orders o2 ON o1.order_number = o2.order_number
-      WHERE o1.order_number = $1 AND o1.canceled = false
-      GROUP BY 
-        o1.order_number,
-        o1.customer,
-        o1.date,
-        o1.status,
-        o1.payment_method,
-        o1.shipping_method,
-        o1.shipping_address,
-        o1.billing_address
-    `, [req.params.orderNumber]);
-
-    if (orderRows.length === 0) {
-      return res.status(404).json({ error: 'Order not found' });
-    }
-
-    // Get order items
-    const { rows: itemRows } = await pool.query(`
-      SELECT 
-        o.pid,
-        p.title,
-        p.SKU,
-        o.quantity,
-        o.price,
-        ROUND((o.price * o.quantity)::numeric, 3) as total
-      FROM orders o
-      JOIN products p ON o.pid = p.pid
-      WHERE o.order_number = $1 AND o.canceled = false
-    `, [req.params.orderNumber]);
-
-    const order = {
-      ...orderRows[0],
-      total_amount: parseFloat(orderRows[0].total_amount) || 0,
-      items_count: parseInt(orderRows[0].items_count) || 0,
-      items: itemRows.map(item => ({
-        ...item,
-        price: parseFloat(item.price) || 0,
-        total: parseFloat(item.total) || 0,
-        quantity: parseInt(item.quantity) || 0
-      }))
-    };
-
-    res.json(order);
-  } catch (error) {
-    console.error('Error fetching order:', error);
-    res.status(500).json({ error: 'Failed to fetch order' });
-  }
-});
-
-module.exports = router; 

inventory-server/src/routes/products.js

@@ -1,731 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const multer = require('multer');
-const { importProductsFromCSV } = require('../utils/csvImporter');
-const { PurchaseOrderStatus, ReceivingStatus } = require('../types/status-codes');
-
-// Configure multer for file uploads
-const upload = multer({ dest: 'uploads/' });
-
-// Get unique brands
-router.get('/brands', async (req, res) => {
-    console.log('Brands endpoint hit:', {
-        url: req.url,
-        method: req.method,
-        headers: req.headers,
-        path: req.path
-    });
-    
-    try {
-        const pool = req.app.locals.pool;
-        console.log('Fetching brands from database...');
-        
-        const { rows } = await pool.query(`
-            SELECT DISTINCT COALESCE(p.brand, 'Unbranded') as brand
-            FROM products p
-            WHERE p.visible = true
-            ORDER BY COALESCE(p.brand, 'Unbranded')
-        `);
-        
-        console.log(`Found ${rows.length} brands:`, rows.slice(0, 3));
-        res.json(rows.map(r => r.brand));
-    } catch (error) {
-        console.error('Error fetching brands:', error);
-        res.status(500).json({ error: 'Failed to fetch brands' });
-    }
-});
-
-// Get all products with pagination, filtering, and sorting
-router.get('/', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    const page = parseInt(req.query.page) || 1;
-    const limit = parseInt(req.query.limit) || 50;
-    const offset = (page - 1) * limit;
-    const sortColumn = req.query.sort || 'title';
-    const sortDirection = req.query.order === 'desc' ? 'DESC' : 'ASC';
-
-    const conditions = ['p.visible = true'];
-    const params = [];
-    let paramCounter = 1;
-
-    // Add default replenishable filter unless explicitly showing non-replenishable
-    if (req.query.showNonReplenishable !== 'true') {
-      conditions.push('p.replenishable = true');
-    }
-
-    // Handle search filter
-    if (req.query.search) {
-      conditions.push(`(p.title ILIKE $${paramCounter} OR p.SKU ILIKE $${paramCounter} OR p.barcode ILIKE $${paramCounter})`);
-      const searchTerm = `%${req.query.search}%`;
-      params.push(searchTerm);
-      paramCounter++;
-    }
-
-    // Handle text filters for specific fields
-    if (req.query.barcode) {
-      conditions.push(`p.barcode ILIKE $${paramCounter}`);
-      params.push(`%${req.query.barcode}%`);
-      paramCounter++;
-    }
-
-    if (req.query.vendor_reference) {
-      conditions.push(`p.vendor_reference ILIKE $${paramCounter}`);
-      params.push(`%${req.query.vendor_reference}%`);
-      paramCounter++;
-    }
-
-    // Add new text filters for the additional fields
-    if (req.query.description) {
-      conditions.push(`p.description ILIKE $${paramCounter}`);
-      params.push(`%${req.query.description}%`);
-      paramCounter++;
-    }
-
-    if (req.query.harmonized_tariff_code) {
-      conditions.push(`p.harmonized_tariff_code ILIKE $${paramCounter}`);
-      params.push(`%${req.query.harmonized_tariff_code}%`);
-      paramCounter++;
-    }
-
-    if (req.query.notions_reference) {
-      conditions.push(`p.notions_reference ILIKE $${paramCounter}`);
-      params.push(`%${req.query.notions_reference}%`);
-      paramCounter++;
-    }
-
-    if (req.query.line) {
-      conditions.push(`p.line ILIKE $${paramCounter}`);
-      params.push(`%${req.query.line}%`);
-      paramCounter++;
-    }
-
-    if (req.query.subline) {
-      conditions.push(`p.subline ILIKE $${paramCounter}`);
-      params.push(`%${req.query.subline}%`);
-      paramCounter++;
-    }
-
-    if (req.query.artist) {
-      conditions.push(`p.artist ILIKE $${paramCounter}`);
-      params.push(`%${req.query.artist}%`);
-      paramCounter++;
-    }
-
-    if (req.query.country_of_origin) {
-      conditions.push(`p.country_of_origin ILIKE $${paramCounter}`);
-      params.push(`%${req.query.country_of_origin}%`);
-      paramCounter++;
-    }
-
-    if (req.query.location) {
-      conditions.push(`p.location ILIKE $${paramCounter}`);
-      params.push(`%${req.query.location}%`);
-      paramCounter++;
-    }
-
-    // Handle numeric filters with operators
-    const numericFields = {
-      stock: 'p.stock_quantity',
-      price: 'p.price',
-      costPrice: 'p.cost_price',
-      landingCost: 'p.landing_cost_price',
-      dailySalesAvg: 'pm.daily_sales_avg',
-      weeklySalesAvg: 'pm.weekly_sales_avg',
-      monthlySalesAvg: 'pm.monthly_sales_avg',
-      avgQuantityPerOrder: 'pm.avg_quantity_per_order',
-      numberOfOrders: 'pm.number_of_orders',
-      margin: 'pm.avg_margin_percent',
-      gmroi: 'pm.gmroi',
-      inventoryValue: 'pm.inventory_value',
-      costOfGoodsSold: 'pm.cost_of_goods_sold',
-      grossProfit: 'pm.gross_profit',
-      turnoverRate: 'pm.turnover_rate',
-      leadTime: 'pm.current_lead_time',
-      currentLeadTime: 'pm.current_lead_time',
-      targetLeadTime: 'pm.target_lead_time',
-      stockCoverage: 'pm.days_of_inventory',
-      daysOfStock: 'pm.days_of_inventory',
-      weeksOfStock: 'pm.weeks_of_inventory',
-      reorderPoint: 'pm.reorder_point',
-      safetyStock: 'pm.safety_stock',
-      // Add new numeric fields
-      preorderCount: 'p.preorder_count',
-      notionsInvCount: 'p.notions_inv_count',
-      rating: 'p.rating',
-      reviews: 'p.reviews',
-      weight: 'p.weight',
-      totalSold: 'p.total_sold',
-      baskets: 'p.baskets',
-      notifies: 'p.notifies'
-    };
-
-    Object.entries(req.query).forEach(([key, value]) => {
-      const field = numericFields[key];
-      if (field) {
-        const operator = req.query[`${key}_operator`] || '=';
-        if (operator === 'between') {
-          try {
-            const [min, max] = JSON.parse(value);
-            conditions.push(`${field} BETWEEN $${paramCounter} AND $${paramCounter + 1}`);
-            params.push(min, max);
-            paramCounter += 2;
-          } catch (e) {
-            console.error(`Invalid between value for ${key}:`, value);
-          }
-        } else {
-          conditions.push(`${field} ${operator} $${paramCounter}`);
-          params.push(parseFloat(value));
-          paramCounter++;
-        }
-      }
-    });
-
-    // Handle date filters
-    const dateFields = {
-      firstSaleDate: 'pm.first_sale_date',
-      lastSaleDate: 'pm.last_sale_date',
-      lastPurchaseDate: 'pm.last_purchase_date',
-      firstReceivedDate: 'pm.first_received_date',
-      lastReceivedDate: 'pm.last_received_date'
-    };
-
-    Object.entries(req.query).forEach(([key, value]) => {
-      const field = dateFields[key];
-      if (field) {
-        conditions.push(`${field}::TEXT LIKE $${paramCounter}`);
-        params.push(`${value}%`); // Format like '2023-01%' to match by month or '2023-01-01' for exact date
-        paramCounter++;
-      }
-    });
-
-    // Handle select filters
-    if (req.query.vendor) {
-      conditions.push(`p.vendor = $${paramCounter}`);
-      params.push(req.query.vendor);
-      paramCounter++;
-    }
-
-    if (req.query.brand) {
-      conditions.push(`p.brand = $${paramCounter}`);
-      params.push(req.query.brand);
-      paramCounter++;
-    }
-
-    if (req.query.category) {
-      conditions.push(`p.categories ILIKE $${paramCounter}`);
-      params.push(`%${req.query.category}%`);
-      paramCounter++;
-    }
-
-    if (req.query.stockStatus && req.query.stockStatus !== 'all') {
-      conditions.push(`pm.stock_status = $${paramCounter}`);
-      params.push(req.query.stockStatus);
-      paramCounter++;
-    }
-
-    if (req.query.abcClass) {
-      conditions.push(`pm.abc_class = $${paramCounter}`);
-      params.push(req.query.abcClass);
-      paramCounter++;
-    }
-
-    if (req.query.leadTimeStatus) {
-      conditions.push(`pm.lead_time_status = $${paramCounter}`);
-      params.push(req.query.leadTimeStatus);
-      paramCounter++;
-    }
-
-    if (req.query.replenishable !== undefined) {
-      conditions.push(`p.replenishable = $${paramCounter}`);
-      params.push(req.query.replenishable === 'true');
-      paramCounter++;
-    }
-
-    if (req.query.managingStock !== undefined) {
-      conditions.push(`p.managing_stock = $${paramCounter}`);
-      params.push(req.query.managingStock === 'true');
-      paramCounter++;
-    }
-
-    // Combine all conditions with AND
-    const whereClause = conditions.length > 0 ? 'WHERE ' + conditions.join(' AND ') : '';
-
-    // Get total count for pagination
-    const countQuery = `
-      SELECT COUNT(DISTINCT p.pid) as total 
-      FROM products p
-      LEFT JOIN product_metrics pm ON p.pid = pm.pid
-      ${whereClause}
-    `;
-    const { rows: [countResult] } = await pool.query(countQuery, params);
-    const total = countResult.total;
-
-    // Get available filters
-    const { rows: categories } = await pool.query(
-      'SELECT name FROM categories ORDER BY name'
-    );
-    const { rows: vendors } = await pool.query(
-      'SELECT DISTINCT vendor FROM products WHERE visible = true AND vendor IS NOT NULL AND vendor != \'\' ORDER BY vendor'
-    );
-    const { rows: brands } = await pool.query(
-      'SELECT DISTINCT COALESCE(brand, \'Unbranded\') as brand FROM products WHERE visible = true ORDER BY brand'
-    );
-
-    // Main query with all fields
-    const query = `
-      WITH RECURSIVE 
-        category_path AS (
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            c.name::text as path
-          FROM categories c
-          WHERE c.parent_id IS NULL
-          
-          UNION ALL
-          
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            (cp.path || ' > ' || c.name)::text
-          FROM categories c
-          JOIN category_path cp ON c.parent_id = cp.cat_id
-        ),
-        product_thresholds AS (
-          SELECT 
-            p.pid,
-            COALESCE(
-              (SELECT overstock_days FROM stock_thresholds st 
-               WHERE st.category_id IN (
-                 SELECT pc.cat_id 
-                 FROM product_categories pc 
-                 WHERE pc.pid = p.pid
-               )
-               AND (st.vendor = p.vendor OR st.vendor IS NULL)
-               ORDER BY st.vendor IS NULL
-               LIMIT 1),
-              (SELECT overstock_days FROM stock_thresholds st 
-               WHERE st.category_id IS NULL 
-               AND (st.vendor = p.vendor OR st.vendor IS NULL)
-               ORDER BY st.vendor IS NULL
-               LIMIT 1),
-              90
-            ) as target_days
-          FROM products p
-        ),
-        product_leaf_categories AS (
-          SELECT DISTINCT pc.cat_id
-          FROM product_categories pc
-          WHERE NOT EXISTS (
-            SELECT 1 
-            FROM categories child
-            JOIN product_categories child_pc ON child.cat_id = child_pc.cat_id
-            WHERE child.parent_id = pc.cat_id
-            AND child_pc.pid = pc.pid
-          )
-        )
-      SELECT 
-        p.*,
-        COALESCE(p.brand, 'Unbranded') as brand,
-        string_agg(DISTINCT (c.cat_id || ':' || c.name), ',') as categories,
-        pm.daily_sales_avg,
-        pm.weekly_sales_avg,
-        pm.monthly_sales_avg,
-        pm.avg_quantity_per_order,
-        pm.number_of_orders,
-        pm.first_sale_date,
-        pm.last_sale_date,
-        pm.days_of_inventory,
-        pm.weeks_of_inventory,
-        pm.reorder_point,
-        pm.safety_stock,
-        pm.avg_margin_percent,
-        CAST(pm.total_revenue AS DECIMAL(15,3)) as total_revenue,
-        CAST(pm.inventory_value AS DECIMAL(15,3)) as inventory_value,
-        CAST(pm.cost_of_goods_sold AS DECIMAL(15,3)) as cost_of_goods_sold,
-        CAST(pm.gross_profit AS DECIMAL(15,3)) as gross_profit,
-        pm.gmroi,
-        pm.avg_lead_time_days,
-        pm.last_purchase_date,
-        pm.last_received_date,
-        pm.abc_class,
-        pm.stock_status,
-        pm.turnover_rate,
-        p.date_last_sold
-      FROM products p
-      LEFT JOIN product_metrics pm ON p.pid = pm.pid
-      LEFT JOIN product_categories pc ON p.pid = pc.pid
-      LEFT JOIN categories c ON pc.cat_id = c.cat_id
-      ${whereClause}
-      GROUP BY p.pid, pm.pid
-      ORDER BY ${sortColumn} ${sortDirection}
-      LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-    `;
-    
-    params.push(limit, offset);
-    const { rows: products } = await pool.query(query, params);
-
-    res.json({
-      products,
-      pagination: {
-        total,
-        pages: Math.ceil(total / limit),
-        currentPage: page,
-        limit
-      },
-      filters: {
-        categories: categories.map(c => c.name),
-        vendors: vendors.map(v => v.vendor),
-        brands: brands.map(b => b.brand)
-      }
-    });
-  } catch (error) {
-    console.error('Error fetching products:', error);
-    res.status(500).json({ error: 'Failed to fetch products' });
-  }
-});
-
-// Get trending products
-router.get('/trending', async (req, res) => {
-  const pool = req.app.locals.pool;
-  try {
-    // First check if we have any data
-    const { rows } = await pool.query(`
-      SELECT COUNT(*) as count, 
-             MAX(total_revenue) as max_revenue,
-             MAX(daily_sales_avg) as max_daily_sales,
-             COUNT(DISTINCT pid) as products_with_metrics
-      FROM product_metrics 
-      WHERE total_revenue > 0 OR daily_sales_avg > 0
-    `);
-    console.log('Product metrics stats:', rows[0]);
-
-    if (parseInt(rows[0].count) === 0) {
-      console.log('No products with metrics found');
-      return res.json([]);
-    }
-
-    // Get trending products
-    const { rows: trendingProducts } = await pool.query(`
-      SELECT 
-        p.pid,
-        p.sku,
-        p.title,
-        COALESCE(pm.daily_sales_avg, 0) as daily_sales_avg,
-        COALESCE(pm.weekly_sales_avg, 0) as weekly_sales_avg,
-        CASE 
-          WHEN pm.weekly_sales_avg > 0 AND pm.daily_sales_avg > 0
-          THEN ((pm.daily_sales_avg - pm.weekly_sales_avg) / pm.weekly_sales_avg) * 100
-          ELSE 0 
-        END as growth_rate,
-        COALESCE(pm.total_revenue, 0) as total_revenue
-      FROM products p
-      INNER JOIN product_metrics pm ON p.pid = pm.pid
-      WHERE (pm.total_revenue > 0 OR pm.daily_sales_avg > 0)
-        AND p.visible = true
-      ORDER BY growth_rate DESC
-      LIMIT 50
-    `);
-
-    console.log('Trending products:', trendingProducts);
-    res.json(trendingProducts);
-  } catch (error) {
-    console.error('Error fetching trending products:', error);
-    res.status(500).json({ error: 'Failed to fetch trending products' });
-  }
-});
-
-// Get a single product
-router.get('/:id', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    const id = parseInt(req.params.id);
-
-    // Common CTE for category paths
-    const categoryPathCTE = `
-      WITH RECURSIVE category_path AS (
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          c.name::text as path
-        FROM categories c
-        WHERE c.parent_id IS NULL
-        
-        UNION ALL
-        
-        SELECT 
-          c.cat_id,
-          c.name,
-          c.parent_id,
-          (cp.path || ' > ' || c.name)::text
-        FROM categories c
-        JOIN category_path cp ON c.parent_id = cp.cat_id
-      )
-    `;
-
-    // Get product details with category paths
-    const { rows: productRows } = await pool.query(`
-      SELECT 
-        p.*,
-        pm.daily_sales_avg,
-        pm.weekly_sales_avg,
-        pm.monthly_sales_avg,
-        pm.days_of_inventory,
-        pm.reorder_point,
-        pm.safety_stock,
-        pm.stock_status,
-        pm.abc_class,
-        pm.avg_margin_percent,
-        pm.total_revenue,
-        pm.inventory_value,
-        pm.turnover_rate,
-        pm.gmroi,
-        pm.cost_of_goods_sold,
-        pm.gross_profit,
-        pm.avg_lead_time_days,
-        pm.current_lead_time,
-        pm.target_lead_time,
-        pm.lead_time_status,
-        pm.reorder_qty,
-        pm.overstocked_amt
-      FROM products p
-      LEFT JOIN product_metrics pm ON p.pid = pm.pid
-      WHERE p.pid = $1
-    `, [id]);
-
-    if (!productRows.length) {
-      return res.status(404).json({ error: 'Product not found' });
-    }
-
-    // Get categories and their paths separately to avoid GROUP BY issues
-    const { rows: categoryRows } = await pool.query(`
-      WITH RECURSIVE 
-        category_path AS (
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            c.name::text as path
-          FROM categories c
-          WHERE c.parent_id IS NULL
-          
-          UNION ALL
-          
-          SELECT 
-            c.cat_id,
-            c.name,
-            c.parent_id,
-            (cp.path || ' > ' || c.name)::text
-          FROM categories c
-          JOIN category_path cp ON c.parent_id = cp.cat_id
-        ),
-        product_leaf_categories AS (
-          -- Find categories assigned to this product that aren't parents
-          -- of other categories assigned to this product
-          SELECT pc.cat_id
-          FROM product_categories pc
-          WHERE pc.pid = $1
-          AND NOT EXISTS (
-            -- Check if there are any child categories also assigned to this product
-            SELECT 1 
-            FROM categories child
-            JOIN product_categories child_pc ON child.cat_id = child_pc.cat_id
-            WHERE child.parent_id = pc.cat_id
-            AND child_pc.pid = pc.pid
-          )
-        )
-      SELECT 
-        c.cat_id,
-        c.name as category_name,
-        cp.path as full_path
-      FROM product_categories pc
-      JOIN categories c ON pc.cat_id = c.cat_id
-      JOIN category_path cp ON c.cat_id = cp.cat_id
-      JOIN product_leaf_categories plc ON c.cat_id = plc.cat_id
-      WHERE pc.pid = $2
-      ORDER BY cp.path
-    `, [id, id]);
-
-    // Transform the results
-    const categoryPathMap = categoryRows.reduce((acc, row) => {
-      // Use cat_id in the key to differentiate categories with the same name
-      acc[`${row.cat_id}:${row.category_name}`] = row.full_path;
-      return acc;
-    }, {});
-
-    const product = {
-      ...productRows[0],
-      // Include cat_id in categories array to match the keys in categoryPathMap
-      categories: categoryRows.map(row => `${row.cat_id}:${row.category_name}`),
-      category_paths: categoryPathMap,
-      price: parseFloat(productRows[0].price),
-      regular_price: parseFloat(productRows[0].regular_price),
-      cost_price: parseFloat(productRows[0].cost_price),
-      landing_cost_price: parseFloat(productRows[0].landing_cost_price),
-      stock_quantity: parseInt(productRows[0].stock_quantity),
-      moq: parseInt(productRows[0].moq),
-      uom: parseInt(productRows[0].uom),
-      managing_stock: Boolean(productRows[0].managing_stock),
-      replenishable: Boolean(productRows[0].replenishable),
-      // Format new fields
-      preorder_count: parseInt(productRows[0].preorder_count || 0),
-      notions_inv_count: parseInt(productRows[0].notions_inv_count || 0),
-      harmonized_tariff_code: productRows[0].harmonized_tariff_code || '',
-      notions_reference: productRows[0].notions_reference || '',
-      line: productRows[0].line || '',
-      subline: productRows[0].subline || '',
-      artist: productRows[0].artist || '',
-      rating: parseFloat(productRows[0].rating || 0),
-      reviews: parseInt(productRows[0].reviews || 0),
-      weight: parseFloat(productRows[0].weight || 0),
-      dimensions: {
-        length: parseFloat(productRows[0].length || 0),
-        width: parseFloat(productRows[0].width || 0),
-        height: parseFloat(productRows[0].height || 0),
-      },
-      country_of_origin: productRows[0].country_of_origin || '',
-      location: productRows[0].location || '',
-      total_sold: parseInt(productRows[0].total_sold || 0),
-      baskets: parseInt(productRows[0].baskets || 0),
-      notifies: parseInt(productRows[0].notifies || 0),
-      date_last_sold: productRows[0].date_last_sold || null,
-      // Format existing analytics fields
-      daily_sales_avg: parseFloat(productRows[0].daily_sales_avg) || 0,
-      weekly_sales_avg: parseFloat(productRows[0].weekly_sales_avg) || 0,
-      monthly_sales_avg: parseFloat(productRows[0].monthly_sales_avg) || 0,
-      avg_quantity_per_order: parseFloat(productRows[0].avg_quantity_per_order) || 0,
-      number_of_orders: parseInt(productRows[0].number_of_orders) || 0,
-      first_sale_date: productRows[0].first_sale_date || null,
-      last_sale_date: productRows[0].last_sale_date || null,
-      days_of_inventory: parseFloat(productRows[0].days_of_inventory) || 0,
-      weeks_of_inventory: parseFloat(productRows[0].weeks_of_inventory) || 0,
-      reorder_point: parseFloat(productRows[0].reorder_point) || 0,
-      safety_stock: parseFloat(productRows[0].safety_stock) || 0,
-      avg_margin_percent: parseFloat(productRows[0].avg_margin_percent) || 0,
-      total_revenue: parseFloat(productRows[0].total_revenue) || 0,
-      inventory_value: parseFloat(productRows[0].inventory_value) || 0,
-      cost_of_goods_sold: parseFloat(productRows[0].cost_of_goods_sold) || 0,
-      gross_profit: parseFloat(productRows[0].gross_profit) || 0,
-      gmroi: parseFloat(productRows[0].gmroi) || 0,
-      avg_lead_time_days: parseFloat(productRows[0].avg_lead_time_days) || 0,
-      current_lead_time: parseFloat(productRows[0].current_lead_time) || 0,
-      target_lead_time: parseFloat(productRows[0].target_lead_time) || 0,
-      lead_time_status: productRows[0].lead_time_status || null,
-      reorder_qty: parseInt(productRows[0].reorder_qty) || 0,
-      overstocked_amt: parseInt(productRows[0].overstocked_amt) || 0
-    };
-
-    res.json(product);
-  } catch (error) {
-    console.error('Error fetching product:', error);
-    res.status(500).json({ error: 'Failed to fetch product' });
-  }
-});
-
-// Get product time series data
-router.get('/:id/time-series', async (req, res) => {
-  const { id } = req.params;
-  try {
-    const pool = req.app.locals.pool;
-
-    // Get monthly sales data
-    const { rows: monthlySales } = await pool.query(`
-      SELECT 
-        TO_CHAR(date, 'YYYY-MM') as month,
-        COUNT(DISTINCT order_number) as order_count,
-        SUM(quantity) as units_sold,
-        ROUND(SUM(price * quantity)::numeric, 3) as revenue
-      FROM orders
-      WHERE pid = $1
-        AND canceled = false
-      GROUP BY TO_CHAR(date, 'YYYY-MM')
-      ORDER BY month DESC
-      LIMIT 12
-    `, [id]);
-
-    // Format monthly sales data
-    const formattedMonthlySales = monthlySales.map(month => ({
-      month: month.month,
-      order_count: parseInt(month.order_count),
-      units_sold: parseInt(month.units_sold),
-      revenue: parseFloat(month.revenue),
-      profit: 0 // Set to 0 since we don't have cost data in orders table
-    }));
-
-    // Get recent orders
-    const { rows: recentOrders } = await pool.query(`
-      SELECT 
-        TO_CHAR(date, 'YYYY-MM-DD') as date,
-        order_number,
-        quantity,
-        price,
-        discount,
-        tax,
-        shipping,
-        customer_name as customer,
-        status
-      FROM orders 
-      WHERE pid = $1
-        AND canceled = false
-      ORDER BY date DESC
-      LIMIT 10
-    `, [id]);
-
-    // Get recent purchase orders with detailed status
-    const { rows: recentPurchases } = await pool.query(`
-      SELECT 
-        TO_CHAR(date, 'YYYY-MM-DD') as date,
-        TO_CHAR(expected_date, 'YYYY-MM-DD') as expected_date,
-        TO_CHAR(received_date, 'YYYY-MM-DD') as received_date,
-        po_id,
-        ordered,
-        received,
-        status,
-        receiving_status,
-        cost_price,
-        notes,
-        CASE 
-          WHEN received_date IS NOT NULL THEN 
-            (received_date - date)
-          WHEN expected_date < CURRENT_DATE AND status < $2 THEN 
-            (CURRENT_DATE - expected_date)
-          ELSE NULL 
-        END as lead_time_days
-      FROM purchase_orders 
-      WHERE pid = $1
-        AND status != $3
-      ORDER BY date DESC
-      LIMIT 10
-    `, [id, PurchaseOrderStatus.ReceivingStarted, PurchaseOrderStatus.Canceled]);
-
-    res.json({
-      monthly_sales: formattedMonthlySales,
-      recent_orders: recentOrders.map(order => ({
-        ...order,
-        price: parseFloat(order.price),
-        discount: parseFloat(order.discount),
-        tax: parseFloat(order.tax),
-        shipping: parseFloat(order.shipping),
-        quantity: parseInt(order.quantity)
-      })),
-      recent_purchases: recentPurchases.map(po => ({
-        ...po,
-        ordered: parseInt(po.ordered),
-        received: parseInt(po.received),
-        status: parseInt(po.status),
-        receiving_status: parseInt(po.receiving_status),
-        cost_price: parseFloat(po.cost_price),
-        lead_time_days: po.lead_time_days ? parseInt(po.lead_time_days) : null
-      }))
-    });
-  } catch (error) {
-    console.error('Error fetching product time series:', error);
-    res.status(500).json({ error: 'Failed to fetch product time series' });
-  }
-});
-
-module.exports = router; 

inventory-server/src/routes/reusable-images.js

@@ -1,396 +0,0 @@
-const express = require('express');
-const router = express.Router();
-const multer = require('multer');
-const path = require('path');
-const fs = require('fs');
-
-// Create reusable uploads directory if it doesn't exist
-const uploadsDir = path.join('/var/www/html/inventory/uploads/reusable');
-fs.mkdirSync(uploadsDir, { recursive: true });
-
-// Configure multer for file uploads
-const storage = multer.diskStorage({
-  destination: function (req, file, cb) {
-    console.log(`Saving reusable image to: ${uploadsDir}`);
-    cb(null, uploadsDir);
-  },
-  filename: function (req, file, cb) {
-    // Create unique filename with original extension
-    const uniqueSuffix = Date.now() + '-' + Math.round(Math.random() * 1E9);
-    
-    // Make sure we preserve the original file extension
-    let fileExt = path.extname(file.originalname).toLowerCase();
-    
-    // Ensure there is a proper extension based on mimetype if none exists
-    if (!fileExt) {
-      switch (file.mimetype) {
-        case 'image/jpeg': fileExt = '.jpg'; break;
-        case 'image/png': fileExt = '.png'; break;
-        case 'image/gif': fileExt = '.gif'; break;
-        case 'image/webp': fileExt = '.webp'; break;
-        default: fileExt = '.jpg'; // Default to jpg
-      }
-    }
-    
-    const fileName = `reusable-${uniqueSuffix}${fileExt}`;
-    console.log(`Generated filename: ${fileName} with mimetype: ${file.mimetype}`);
-    cb(null, fileName);
-  }
-});
-
-const upload = multer({ 
-  storage: storage,
-  limits: {
-    fileSize: 5 * 1024 * 1024, // 5MB max file size
-  },
-  fileFilter: function (req, file, cb) {
-    // Accept only image files
-    const filetypes = /jpeg|jpg|png|gif|webp/;
-    const mimetype = filetypes.test(file.mimetype);
-    const extname = filetypes.test(path.extname(file.originalname).toLowerCase());
-    
-    if (mimetype && extname) {
-      return cb(null, true);
-    }
-    cb(new Error('Only image files are allowed'));
-  }
-});
-
-// Get all reusable images
-router.get('/', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM reusable_images 
-      ORDER BY created_at DESC
-    `);
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error fetching reusable images:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch reusable images',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get images by company or global images
-router.get('/by-company/:companyId', async (req, res) => {
-  try {
-    const { companyId } = req.params;
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Get images that are either global or belong to this company
-    const result = await pool.query(`
-      SELECT * FROM reusable_images 
-      WHERE is_global = true OR company = $1
-      ORDER BY created_at DESC
-    `, [companyId]);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error fetching reusable images by company:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch reusable images by company',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get global images only
-router.get('/global', async (req, res) => {
-  try {
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM reusable_images 
-      WHERE is_global = true
-      ORDER BY created_at DESC
-    `);
-    
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error fetching global reusable images:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch global reusable images',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get a single image by ID
-router.get('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM reusable_images 
-      WHERE id = $1
-    `, [id]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'Reusable image not found' });
-    }
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error fetching reusable image:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch reusable image',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Upload a new reusable image
-router.post('/upload', upload.single('image'), async (req, res) => {
-  try {
-    if (!req.file) {
-      return res.status(400).json({ error: 'No image file provided' });
-    }
-
-    const { name, is_global, company } = req.body;
-    
-    // Validate required fields
-    if (!name) {
-      return res.status(400).json({ error: 'Image name is required' });
-    }
-    
-    // Convert is_global from string to boolean
-    const isGlobal = is_global === 'true' || is_global === true;
-    
-    // Validate company is provided for non-global images
-    if (!isGlobal && !company) {
-      return res.status(400).json({ error: 'Company is required for non-global images' });
-    }
-
-    // Log file information
-    console.log('Reusable image uploaded:', {
-      filename: req.file.filename,
-      originalname: req.file.originalname,
-      mimetype: req.file.mimetype,
-      size: req.file.size,
-      path: req.file.path
-    });
-
-    // Ensure the file exists
-    const filePath = path.join(uploadsDir, req.file.filename);
-    if (!fs.existsSync(filePath)) {
-      return res.status(500).json({ error: 'File was not saved correctly' });
-    }
-
-    // Create URL for the uploaded file
-    const baseUrl = 'https://inventory.acot.site';
-    const imageUrl = `${baseUrl}/uploads/reusable/${req.file.filename}`;
-    
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Insert record into database
-    const result = await pool.query(`
-      INSERT INTO reusable_images (
-        name,
-        filename,
-        file_path,
-        image_url,
-        is_global,
-        company,
-        mime_type,
-        file_size
-      ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
-      RETURNING *
-    `, [
-      name,
-      req.file.filename,
-      filePath,
-      imageUrl,
-      isGlobal,
-      isGlobal ? null : company,
-      req.file.mimetype,
-      req.file.size
-    ]);
-
-    // Return success response with image data
-    res.status(201).json({
-      success: true,
-      image: result.rows[0],
-      message: 'Image uploaded successfully'
-    });
-    
-  } catch (error) {
-    console.error('Error uploading reusable image:', error);
-    res.status(500).json({ error: error.message || 'Failed to upload image' });
-  }
-});
-
-// Update image details (name, is_global, company)
-router.put('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const { name, is_global, company } = req.body;
-    
-    // Validate required fields
-    if (!name) {
-      return res.status(400).json({ error: 'Image name is required' });
-    }
-    
-    // Convert is_global from string to boolean if necessary
-    const isGlobal = typeof is_global === 'string' ? is_global === 'true' : !!is_global;
-    
-    // Validate company is provided for non-global images
-    if (!isGlobal && !company) {
-      return res.status(400).json({ error: 'Company is required for non-global images' });
-    }
-    
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Check if the image exists
-    const checkResult = await pool.query('SELECT * FROM reusable_images WHERE id = $1', [id]);
-    if (checkResult.rows.length === 0) {
-      return res.status(404).json({ error: 'Reusable image not found' });
-    }
-    
-    const result = await pool.query(`
-      UPDATE reusable_images 
-      SET 
-        name = $1,
-        is_global = $2,
-        company = $3
-      WHERE id = $4
-      RETURNING *
-    `, [
-      name,
-      isGlobal,
-      isGlobal ? null : company,
-      id
-    ]);
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error updating reusable image:', error);
-    res.status(500).json({ 
-      error: 'Failed to update reusable image',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Delete a reusable image
-router.delete('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const pool = req.app.locals.pool;
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    // Get the image data first to get the filename
-    const imageResult = await pool.query('SELECT * FROM reusable_images WHERE id = $1', [id]);
-    
-    if (imageResult.rows.length === 0) {
-      return res.status(404).json({ error: 'Reusable image not found' });
-    }
-    
-    const image = imageResult.rows[0];
-    
-    // Delete from database
-    await pool.query('DELETE FROM reusable_images WHERE id = $1', [id]);
-    
-    // Delete the file from filesystem
-    const filePath = path.join(uploadsDir, image.filename);
-    if (fs.existsSync(filePath)) {
-      fs.unlinkSync(filePath);
-    }
-
-    res.json({ 
-      message: 'Reusable image deleted successfully',
-      image
-    });
-  } catch (error) {
-    console.error('Error deleting reusable image:', error);
-    res.status(500).json({ 
-      error: 'Failed to delete reusable image',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Check if file exists and permissions
-router.get('/check-file/:filename', (req, res) => {
-  const { filename } = req.params;
-  
-  // Prevent directory traversal
-  if (filename.includes('..') || filename.includes('/')) {
-    return res.status(400).json({ error: 'Invalid filename' });
-  }
-  
-  const filePath = path.join(uploadsDir, filename);
-  
-  try {
-    // Check if file exists
-    if (!fs.existsSync(filePath)) {
-      return res.status(404).json({ 
-        error: 'File not found', 
-        path: filePath,
-        exists: false,
-        readable: false 
-      });
-    }
-    
-    // Check if file is readable
-    fs.accessSync(filePath, fs.constants.R_OK);
-    
-    // Get file stats
-    const stats = fs.statSync(filePath);
-    
-    return res.json({
-      filename,
-      path: filePath,
-      exists: true,
-      readable: true,
-      isFile: stats.isFile(),
-      isDirectory: stats.isDirectory(),
-      size: stats.size,
-      created: stats.birthtime,
-      modified: stats.mtime,
-      permissions: stats.mode.toString(8)
-    });
-  } catch (error) {
-    return res.status(500).json({ 
-      error: error.message,
-      path: filePath,
-      exists: fs.existsSync(filePath),
-      readable: false
-    });
-  }
-});
-
-// Error handling middleware
-router.use((err, req, res, next) => {
-  console.error('Reusable images route error:', err);
-  res.status(500).json({ 
-    error: 'Internal server error',
-    details: err.message
-  });
-});
-
-module.exports = router; 

inventory-server/src/routes/templates.js

@@ -1,283 +0,0 @@
-const express = require('express');
-const { getPool } = require('../utils/db');
-const dotenv = require('dotenv');
-const path = require('path');
-
-dotenv.config({ path: path.join(__dirname, "../../.env") });
-
-const router = express.Router();
-
-// Get all templates
-router.get('/', async (req, res) => {
-  try {
-    const pool = getPool();
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM templates 
-      ORDER BY company ASC, product_type ASC
-    `);
-    res.json(result.rows);
-  } catch (error) {
-    console.error('Error fetching templates:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch templates',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Get template by company and product type
-router.get('/:company/:productType', async (req, res) => {
-  try {
-    const { company, productType } = req.params;
-    const pool = getPool();
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      SELECT * FROM templates 
-      WHERE company = $1 AND product_type = $2
-    `, [company, productType]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'Template not found' });
-    }
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error fetching template:', error);
-    res.status(500).json({ 
-      error: 'Failed to fetch template',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Create new template
-router.post('/', async (req, res) => {
-  try {
-    const {
-      company,
-      product_type,
-      supplier,
-      msrp,
-      cost_each,
-      qty_per_unit,
-      case_qty,
-      hts_code,
-      description,
-      weight,
-      length,
-      width,
-      height,
-      tax_cat,
-      size_cat,
-      categories,
-      ship_restrictions
-    } = req.body;
-
-    // Validate required fields
-    if (!company || !product_type) {
-      return res.status(400).json({ error: 'Company and Product Type are required' });
-    }
-
-    const pool = getPool();
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      INSERT INTO templates (
-        company,
-        product_type,
-        supplier,
-        msrp,
-        cost_each,
-        qty_per_unit,
-        case_qty,
-        hts_code,
-        description,
-        weight,
-        length,
-        width,
-        height,
-        tax_cat,
-        size_cat,
-        categories,
-        ship_restrictions
-      ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
-      RETURNING *
-    `, [
-      company,
-      product_type,
-      supplier,
-      msrp,
-      cost_each,
-      qty_per_unit,
-      case_qty,
-      hts_code,
-      description,
-      weight,
-      length,
-      width,
-      height,
-      tax_cat,
-      size_cat,
-      categories,
-      ship_restrictions
-    ]);
-
-    res.status(201).json(result.rows[0]);
-  } catch (error) {
-    console.error('Error creating template:', error);
-    // Check for unique constraint violation
-    if (error instanceof Error && error.message.includes('unique constraint')) {
-      return res.status(409).json({ 
-        error: 'Template already exists for this company and product type',
-        details: error.message
-      });
-    }
-    res.status(500).json({ 
-      error: 'Failed to create template',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Update template
-router.put('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const {
-      company,
-      product_type,
-      supplier,
-      msrp,
-      cost_each,
-      qty_per_unit,
-      case_qty,
-      hts_code,
-      description,
-      weight,
-      length,
-      width,
-      height,
-      tax_cat,
-      size_cat,
-      categories,
-      ship_restrictions
-    } = req.body;
-
-    // Validate required fields
-    if (!company || !product_type) {
-      return res.status(400).json({ error: 'Company and Product Type are required' });
-    }
-
-    const pool = getPool();
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query(`
-      UPDATE templates 
-      SET 
-        company = $1,
-        product_type = $2,
-        supplier = $3,
-        msrp = $4,
-        cost_each = $5,
-        qty_per_unit = $6,
-        case_qty = $7,
-        hts_code = $8,
-        description = $9,
-        weight = $10,
-        length = $11,
-        width = $12,
-        height = $13,
-        tax_cat = $14,
-        size_cat = $15,
-        categories = $16,
-        ship_restrictions = $17
-      WHERE id = $18
-      RETURNING *
-    `, [
-      company,
-      product_type,
-      supplier,
-      msrp,
-      cost_each,
-      qty_per_unit,
-      case_qty,
-      hts_code,
-      description,
-      weight,
-      length,
-      width,
-      height,
-      tax_cat,
-      size_cat,
-      categories,
-      ship_restrictions,
-      id
-    ]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'Template not found' });
-    }
-
-    res.json(result.rows[0]);
-  } catch (error) {
-    console.error('Error updating template:', error);
-    // Check for unique constraint violation
-    if (error instanceof Error && error.message.includes('unique constraint')) {
-      return res.status(409).json({ 
-        error: 'Template already exists for this company and product type',
-        details: error.message
-      });
-    }
-    res.status(500).json({ 
-      error: 'Failed to update template',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Delete template
-router.delete('/:id', async (req, res) => {
-  try {
-    const { id } = req.params;
-    const pool = getPool();
-    if (!pool) {
-      throw new Error('Database pool not initialized');
-    }
-    
-    const result = await pool.query('DELETE FROM templates WHERE id = $1 RETURNING *', [id]);
-
-    if (result.rows.length === 0) {
-      return res.status(404).json({ error: 'Template not found' });
-    }
-
-    res.json({ message: 'Template deleted successfully' });
-  } catch (error) {
-    console.error('Error deleting template:', error);
-    res.status(500).json({ 
-      error: 'Failed to delete template',
-      details: error instanceof Error ? error.message : 'Unknown error'
-    });
-  }
-});
-
-// Error handling middleware
-router.use((err, req, res, next) => {
-  console.error('Template route error:', err);
-  res.status(500).json({ 
-    error: 'Internal server error',
-    details: err.message
-  });
-});
-
-module.exports = router; 

inventory-server/src/routes/vendorsAggregate.js

@@ -1,323 +1 @@
-const express = require('express');
-const router = express.Router();
-const { parseValue } = require('../utils/apiHelpers'); // Adjust path if needed
-
-// --- Configuration & Helpers ---
-const DEFAULT_PAGE_LIMIT = 50;
-const MAX_PAGE_LIMIT = 200;
-
-// Maps query keys to DB columns in vendor_metrics
-const COLUMN_MAP = {
-    vendorName: { dbCol: 'vm.vendor_name', type: 'string' },
-    productCount: { dbCol: 'vm.product_count', type: 'number' },
-    activeProductCount: { dbCol: 'vm.active_product_count', type: 'number' },
-    replenishableProductCount: { dbCol: 'vm.replenishable_product_count', type: 'number' },
-    currentStockUnits: { dbCol: 'vm.current_stock_units', type: 'number' },
-    currentStockCost: { dbCol: 'vm.current_stock_cost', type: 'number' },
-    currentStockRetail: { dbCol: 'vm.current_stock_retail', type: 'number' },
-    onOrderUnits: { dbCol: 'vm.on_order_units', type: 'number' },
-    onOrderCost: { dbCol: 'vm.on_order_cost', type: 'number' },
-    poCount365d: { dbCol: 'vm.po_count_365d', type: 'number' },
-    avgLeadTimeDays: { dbCol: 'vm.avg_lead_time_days', type: 'number' },
-    sales7d: { dbCol: 'vm.sales_7d', type: 'number' },
-    revenue7d: { dbCol: 'vm.revenue_7d', type: 'number' },
-    sales30d: { dbCol: 'vm.sales_30d', type: 'number' },
-    revenue30d: { dbCol: 'vm.revenue_30d', type: 'number' },
-    profit30d: { dbCol: 'vm.profit_30d', type: 'number' },
-    cogs30d: { dbCol: 'vm.cogs_30d', type: 'number' },
-    sales365d: { dbCol: 'vm.sales_365d', type: 'number' },
-    revenue365d: { dbCol: 'vm.revenue_365d', type: 'number' },
-    lifetimeSales: { dbCol: 'vm.lifetime_sales', type: 'number' },
-    lifetimeRevenue: { dbCol: 'vm.lifetime_revenue', type: 'number' },
-    avgMargin30d: { dbCol: 'vm.avg_margin_30d', type: 'number' },
-    // Growth metrics
-    salesGrowth30dVsPrev: { dbCol: 'vm.sales_growth_30d_vs_prev', type: 'number' },
-    revenueGrowth30dVsPrev: { dbCol: 'vm.revenue_growth_30d_vs_prev', type: 'number' },
-    // Add aliases if needed for frontend compatibility
-    name: { dbCol: 'vm.vendor_name', type: 'string' },
-    leadTime: { dbCol: 'vm.avg_lead_time_days', type: 'number' },
-    // Add status for filtering
-    status: { dbCol: 'vendor_status', type: 'string' },
-};
-
-function getSafeColumnInfo(queryParamKey) {
-    return COLUMN_MAP[queryParamKey] || null;
-}
-
-// --- Route Handlers ---
-
-// GET /vendors-aggregate/filter-options (Just vendors list for now)
-router.get('/filter-options', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /vendors-aggregate/filter-options');
-    try {
-        // Get vendor names
-        const { rows: vendorRows } = await pool.query(`
-            SELECT DISTINCT vendor_name FROM public.vendor_metrics ORDER BY vendor_name
-        `);
-        
-        // Get status values - calculate them since they're derived
-        const { rows: statusRows } = await pool.query(`
-            SELECT DISTINCT
-                CASE 
-                    WHEN po_count_365d > 0 AND sales_30d > 0 THEN 'active'
-                    WHEN po_count_365d > 0 THEN 'inactive'
-                    ELSE 'pending'
-                END as status
-            FROM public.vendor_metrics
-            ORDER BY status
-        `);
-        
-        res.json({
-            vendors: vendorRows.map(r => r.vendor_name),
-            statuses: statusRows.map(r => r.status)
-        });
-     } catch(error) {
-        console.error('Error fetching vendor filter options:', error);
-        res.status(500).json({ error: 'Failed to fetch filter options' });
-     }
-});
-
-// GET /vendors-aggregate/stats (Overall vendor stats)
-router.get('/stats', async (req, res) => {
-     const pool = req.app.locals.pool;
-     console.log('GET /vendors-aggregate/stats');
-     try {
-         // Get basic vendor stats from aggregate table
-         const { rows: [stats] } = await pool.query(`
-            SELECT
-                COUNT(*) AS total_vendors,
-                SUM(active_product_count) AS total_active_products,
-                SUM(current_stock_cost) AS total_stock_value,
-                SUM(on_order_cost) AS total_on_order_value,
-                AVG(NULLIF(avg_lead_time_days, 0)) AS overall_avg_lead_time
-            FROM public.vendor_metrics vm
-         `);
-         
-         // Count active vendors based on criteria (from old vendors.js)
-         const { rows: [activeStats] } = await pool.query(`
-            SELECT 
-                COUNT(DISTINCT CASE 
-                    WHEN po_count_365d > 0 
-                    THEN vendor_name 
-                END) as active_vendors
-            FROM public.vendor_metrics
-         `);
-         
-         // Get overall cost metrics from purchase orders
-         const { rows: [overallCostMetrics] } = await pool.query(`
-            SELECT 
-                ROUND((SUM(ordered * cost_price)::numeric / NULLIF(SUM(ordered), 0)), 2) as avg_unit_cost,
-                ROUND(SUM(ordered * cost_price)::numeric, 3) as total_spend
-            FROM purchase_orders
-            WHERE cost_price IS NOT NULL
-                AND ordered > 0
-                AND vendor IS NOT NULL AND vendor != ''
-         `);
-
-         res.json({
-             totalVendors: parseInt(stats?.total_vendors || 0),
-             activeVendors: parseInt(activeStats?.active_vendors || 0),
-             totalActiveProducts: parseInt(stats?.total_active_products || 0),
-             totalValue: parseFloat(stats?.total_stock_value || 0),
-             totalOnOrderValue: parseFloat(stats?.total_on_order_value || 0),
-             avgLeadTime: parseFloat(stats?.overall_avg_lead_time || 0),
-             avgUnitCost: parseFloat(overallCostMetrics?.avg_unit_cost || 0),
-             totalSpend: parseFloat(overallCostMetrics?.total_spend || 0)
-         });
-     } catch (error) {
-        console.error('Error fetching vendor stats:', error);
-        res.status(500).json({ error: 'Failed to fetch vendor stats.' });
-     }
-});
-
-// GET /vendors-aggregate/ (List vendors)
-router.get('/', async (req, res) => {
-    const pool = req.app.locals.pool;
-    console.log('GET /vendors-aggregate received query:', req.query);
-    try {
-        // --- Pagination ---
-        let page = parseInt(req.query.page, 10) || 1;
-        let limit = parseInt(req.query.limit, 10) || DEFAULT_PAGE_LIMIT;
-        limit = Math.min(limit, MAX_PAGE_LIMIT);
-        const offset = (page - 1) * limit;
-
-        // --- Sorting ---
-        const sortQueryKey = req.query.sort || 'vendorName'; // Default sort
-        const sortColumnInfo = getSafeColumnInfo(sortQueryKey);
-        const sortColumn = sortColumnInfo ? sortColumnInfo.dbCol : 'vm.vendor_name';
-        const sortDirection = req.query.order?.toLowerCase() === 'desc' ? 'DESC' : 'ASC';
-        const nullsOrder = (sortDirection === 'ASC' ? 'NULLS FIRST' : 'NULLS LAST');
-        const sortClause = `ORDER BY ${sortColumn} ${sortDirection} ${nullsOrder}`;
-
-        // --- Filtering ---
-        const conditions = [];
-        const params = [];
-        let paramCounter = 1;
-        // Build conditions based on req.query, using COLUMN_MAP and parseValue
-        for (const key in req.query) {
-            if (['page', 'limit', 'sort', 'order'].includes(key)) continue;
-
-            let filterKey = key;
-            let operator = '='; // Default operator
-            const value = req.query[key];
-
-            const operatorMatch = key.match(/^(.*)_(eq|ne|gt|gte|lt|lte|like|ilike|between|in)$/);
-            if (operatorMatch) {
-                filterKey = operatorMatch[1];
-                operator = operatorMatch[2];
-            }
-
-            const columnInfo = getSafeColumnInfo(filterKey);
-            if (columnInfo) {
-                const dbColumn = columnInfo.dbCol;
-                const valueType = columnInfo.type;
-                 try {
-                    let conditionFragment = '';
-                    let needsParam = true;
-                    switch (operator.toLowerCase()) { // Normalize operator
-                         case 'eq': operator = '='; break;
-                         case 'ne': operator = '<>'; break;
-                         case 'gt': operator = '>'; break;
-                         case 'gte': operator = '>='; break;
-                         case 'lt': operator = '<'; break;
-                         case 'lte': operator = '<='; break;
-                         case 'like': operator = 'LIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                         case 'ilike': operator = 'ILIKE'; needsParam=false; params.push(`%${parseValue(value, valueType)}%`); break;
-                         case 'between':
-                             const [val1, val2] = String(value).split(',');
-                             if (val1 !== undefined && val2 !== undefined) {
-                                 conditionFragment = `${dbColumn} BETWEEN $${paramCounter++} AND $${paramCounter++}`;
-                                 params.push(parseValue(val1, valueType), parseValue(val2, valueType));
-                                 needsParam = false;
-                             } else continue;
-                             break;
-                         case 'in':
-                              const inValues = String(value).split(',');
-                              if (inValues.length > 0) {
-                                 const placeholders = inValues.map(() => `$${paramCounter++}`).join(', ');
-                                 conditionFragment = `${dbColumn} IN (${placeholders})`;
-                                 params.push(...inValues.map(v => parseValue(v, valueType)));
-                                 needsParam = false;
-                              } else continue;
-                              break;
-                         default: operator = '='; break;
-                    }
-
-                    if (needsParam) {
-                         conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                         params.push(parseValue(value, valueType));
-                    } else if (!conditionFragment) { // For LIKE/ILIKE
-                         conditionFragment = `${dbColumn} ${operator} $${paramCounter++}`;
-                    }
-
-                    if (conditionFragment) {
-                         conditions.push(`(${conditionFragment})`);
-                    }
-                 } catch (parseError) {
-                     console.warn(`Skipping filter for key "${key}" due to parsing error: ${parseError.message}`);
-                     if (needsParam) paramCounter--;
-                 }
-            } else {
-                 console.warn(`Invalid filter key ignored: ${key}`);
-            }
-        }
-
-        // --- Execute Queries ---
-        const whereClause = conditions.length > 0 ? `WHERE ${conditions.join(' AND ')}` : '';
-        
-        // Status calculation from vendors.js
-        const statusCase = `
-            CASE 
-                WHEN po_count_365d > 0 AND sales_30d > 0 THEN 'active'
-                WHEN po_count_365d > 0 THEN 'inactive'
-                ELSE 'pending'
-            END as vendor_status
-        `;
-        
-        const baseSql = `
-            FROM (
-                SELECT 
-                    vm.*,
-                    ${statusCase}
-                FROM public.vendor_metrics vm
-            ) vm
-            ${whereClause}
-        `;
-
-        const countSql = `SELECT COUNT(*) AS total ${baseSql}`;
-        const dataSql = `
-            WITH vendor_data AS (
-                SELECT 
-                    vm.*,
-                    ${statusCase}
-                FROM public.vendor_metrics vm
-            )
-            SELECT 
-                vm.*,
-                COALESCE(po.avg_unit_cost, 0) as avg_unit_cost,
-                COALESCE(po.total_spend, 0) as total_spend
-            FROM vendor_data vm
-            LEFT JOIN (
-                SELECT 
-                    vendor,
-                    ROUND((SUM(ordered * cost_price)::numeric / NULLIF(SUM(ordered), 0)), 2) as avg_unit_cost,
-                    ROUND(SUM(ordered * cost_price)::numeric, 3) as total_spend
-                FROM purchase_orders
-                WHERE cost_price IS NOT NULL AND ordered > 0
-                GROUP BY vendor
-            ) po ON vm.vendor_name = po.vendor
-            ${whereClause}
-            ${sortClause}
-            LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
-        `;
-        const dataParams = [...params, limit, offset];
-
-        console.log("Count SQL:", countSql, params);
-        console.log("Data SQL:", dataSql, dataParams);
-
-        const [countResult, dataResult] = await Promise.all([
-            pool.query(countSql, params),
-            pool.query(dataSql, dataParams)
-        ]);
-
-        const total = parseInt(countResult.rows[0].total, 10);
-        const vendors = dataResult.rows.map(row => {
-            // Create a new object with both snake_case and camelCase keys
-            const transformedRow = { ...row }; // Start with original data
-            
-            for (const key in row) {
-                // Skip null/undefined values
-                if (row[key] === null || row[key] === undefined) {
-                    continue; // Original already has the null value
-                }
-
-                // Transform keys to match frontend expectations (add camelCase versions)
-                // First handle cases like sales_7d -> sales7d
-                let camelKey = key.replace(/_(\d+[a-z])/g, '$1');
-                
-                // Then handle regular snake_case -> camelCase
-                camelKey = camelKey.replace(/_([a-z])/g, (_, letter) => letter.toUpperCase());
-                if (camelKey !== key) { // Only add if different from original
-                    transformedRow[camelKey] = row[key];
-                }
-            }
-            return transformedRow;
-        });
-
-        // --- Respond ---
-        res.json({
-            vendors,
-            pagination: { total, pages: Math.ceil(total / limit), currentPage: page, limit },
-        });
-
-    } catch (error) {
-        console.error('Error fetching vendor metrics list:', error);
-        res.status(500).json({ error: 'Failed to fetch vendor metrics.' });
-    }
-});
-
-// GET /vendors-aggregate/:name (Get single vendor metric)
-// Implement if needed, remember to URL-decode the name parameter
-
-module.exports = router;
+ 

inventory-server/src/server.js

@@ -1,212 +0,0 @@
-const express = require('express');
-const cors = require('cors');
-const { spawn } = require('child_process');
-const path = require('path');
-const fs = require('fs');
-const { corsMiddleware, corsErrorHandler } = require('./middleware/cors');
-const { initPool } = require('./utils/db');
-const productsRouter = require('./routes/products');
-const dashboardRouter = require('./routes/dashboard');
-const ordersRouter = require('./routes/orders');
-const csvRouter = require('./routes/data-management');
-const analyticsRouter = require('./routes/analytics');
-const purchaseOrdersRouter = require('./routes/purchase-orders');
-const configRouter = require('./routes/config');
-const metricsRouter = require('./routes/metrics');
-const importRouter = require('./routes/import');
-const aiValidationRouter = require('./routes/ai-validation');
-const templatesRouter = require('./routes/templates');
-const aiPromptsRouter = require('./routes/ai-prompts');
-const reusableImagesRouter = require('./routes/reusable-images');
-const categoriesAggregateRouter = require('./routes/categoriesAggregate');
-const vendorsAggregateRouter = require('./routes/vendorsAggregate');
-const brandsAggregateRouter = require('./routes/brandsAggregate');
-
-// Get the absolute path to the .env file
-const envPath = '/var/www/html/inventory/.env';
-console.log('Looking for .env file at:', envPath);
-console.log('.env file exists:', fs.existsSync(envPath));
-
-try {
-  require('dotenv').config({ path: envPath });
-  console.log('.env file loaded successfully');
-  console.log('Environment check:', {
-    NODE_ENV: process.env.NODE_ENV || 'not set',
-    PORT: process.env.PORT || 'not set',
-    DB_HOST: process.env.DB_HOST || 'not set',
-    DB_USER: process.env.DB_USER || 'not set',
-    DB_NAME: process.env.DB_NAME || 'not set',
-    DB_PASSWORD: process.env.DB_PASSWORD ? '[password set]' : 'not set',
-    DB_PORT: process.env.DB_PORT || 'not set',
-    DB_SSL: process.env.DB_SSL || 'not set'
-  });
-} catch (error) {
-  console.error('Error loading .env file:', error);
-}
-
-// Ensure required directories exist
-['logs', 'uploads'].forEach(dir => {
-  if (!fs.existsSync(dir)) {
-    fs.mkdirSync(dir, { recursive: true });
-  }
-});
-
-const app = express();
-
-// Debug middleware to log request details
-app.use((req, res, next) => {
-  console.log('Request details:', {
-    method: req.method,
-    url: req.url,
-    origin: req.get('Origin'),
-    headers: req.headers
-  });
-  next();
-});
-
-// Apply CORS middleware first, before any other middleware
-app.use(corsMiddleware);
-
-// Body parser middleware
-app.use(express.json({ limit: '10mb' }));
-app.use(express.urlencoded({ extended: true, limit: '10mb' }));
-
-// Initialize database pool and start server
-async function startServer() {
-  try {
-    // Initialize database pool
-    const pool = await initPool({
-      host: process.env.DB_HOST,
-      user: process.env.DB_USER,
-      password: process.env.DB_PASSWORD,
-      database: process.env.DB_NAME,
-      port: process.env.DB_PORT || 5432,
-      max: process.env.NODE_ENV === 'production' ? 20 : 10,
-      idleTimeoutMillis: 30000,
-      connectionTimeoutMillis: 2000,
-      ssl: process.env.DB_SSL === 'true' ? {
-        rejectUnauthorized: false
-      } : false
-    });
-
-    // Make pool available to routes
-    app.locals.pool = pool;
-
-    // Set up routes after pool is initialized
-    app.use('/api/products', productsRouter);
-    app.use('/api/dashboard', dashboardRouter);
-    app.use('/api/orders', ordersRouter);
-    app.use('/api/csv', csvRouter);
-    app.use('/api/analytics', analyticsRouter);
-    app.use('/api/purchase-orders', purchaseOrdersRouter);
-    app.use('/api/config', configRouter);
-    app.use('/api/metrics', metricsRouter);
-    // Use only the aggregate routes for vendors and categories
-    app.use('/api/vendors', vendorsAggregateRouter); 
-    app.use('/api/categories', categoriesAggregateRouter);
-    // Keep the aggregate-specific endpoints for backward compatibility
-    app.use('/api/categories-aggregate', categoriesAggregateRouter);
-    app.use('/api/vendors-aggregate', vendorsAggregateRouter);
-    app.use('/api/brands-aggregate', brandsAggregateRouter);
-    app.use('/api/import', importRouter);
-    app.use('/api/ai-validation', aiValidationRouter);
-    app.use('/api/templates', templatesRouter);
-    app.use('/api/ai-prompts', aiPromptsRouter);
-    app.use('/api/reusable-images', reusableImagesRouter);
-
-    // Basic health check route
-    app.get('/health', (req, res) => {
-      res.json({ 
-        status: 'ok',
-        timestamp: new Date().toISOString(),
-        environment: process.env.NODE_ENV
-      });
-    });
-
-    // CORS error handler - must be before other error handlers
-    app.use(corsErrorHandler);
-
-    // Error handling middleware - MUST be after routes and CORS error handler
-    app.use((err, req, res, next) => {
-      console.error(`[${new Date().toISOString()}] Error:`, err);
-      
-      // Send detailed error in development, generic in production
-      const error = process.env.NODE_ENV === 'production'
-        ? 'An internal server error occurred'
-        : err.message || err;
-        
-      res.status(err.status || 500).json({ error });
-    });
-
-    const PORT = process.env.PORT || 3000;
-    app.listen(PORT, () => {
-      console.log(`[Server] Running in ${process.env.NODE_ENV || 'development'} mode on port ${PORT}`);
-    });
-  } catch (error) {
-    console.error('Failed to start server:', error);
-    process.exit(1);
-  }
-}
-
-// Handle uncaught exceptions
-process.on('uncaughtException', (err) => {
-  console.error(`[${new Date().toISOString()}] Uncaught Exception:`, err);
-  process.exit(1);
-});
-
-process.on('unhandledRejection', (reason, promise) => {
-  console.error(`[${new Date().toISOString()}] Unhandled Rejection at:`, promise, 'reason:', reason);
-});
-
-// Initialize client sets for SSE
-const importClients = new Set();
-const updateClients = new Set();
-const resetClients = new Set();
-const resetMetricsClients = new Set();
-
-// Helper function to send progress to SSE clients
-const sendProgressToClients = (clients, data) => {
-  clients.forEach(client => {
-    try {
-      client.write(`data: ${JSON.stringify(data)}\n\n`);
-    } catch (error) {
-      console.error('Error sending SSE update:', error);
-    }
-  });
-};
-
-// Setup SSE connection
-const setupSSE = (req, res) => {
-  const { type } = req.params;
-  
-  // Set headers for SSE
-  res.writeHead(200, {
-    'Content-Type': 'text/event-stream',
-    'Cache-Control': 'no-cache',
-    'Connection': 'keep-alive',
-    'Access-Control-Allow-Origin': req.headers.origin || '*',
-    'Access-Control-Allow-Credentials': 'true'
-  });
-
-  // Send initial message
-  res.write('data: {"status":"connected"}\n\n');
-
-  // Add client to appropriate set
-  const clientSet = type === 'import' ? importClients :
-                   type === 'update' ? updateClients :
-                   type === 'reset' ? resetClients :
-                   type === 'reset-metrics' ? resetMetricsClients :
-                   null;
-
-  if (clientSet) {
-    clientSet.add(res);
-    
-    // Remove client when connection closes
-    req.on('close', () => {
-      clientSet.delete(res);
-    });
-  }
-};
-
-// Start the server
-startServer(); 

inventory-server/src/types/status-codes.js

@@ -1,79 +0,0 @@
-// Purchase Order Status Codes
-const PurchaseOrderStatus = {
-    Canceled: 0,
-    Created: 1,
-    ElectronicallyReadySend: 10,
-    Ordered: 11,
-    Preordered: 12,
-    ElectronicallySent: 13,
-    ReceivingStarted: 15,
-    Done: 50
-};
-
-// Receiving Status Codes
-const ReceivingStatus = {
-    Canceled: 0,
-    Created: 1,
-    PartialReceived: 30,
-    FullReceived: 40,
-    Paid: 50
-};
-
-// Status Code Display Names
-const PurchaseOrderStatusLabels = {
-    [PurchaseOrderStatus.Canceled]: 'Canceled',
-    [PurchaseOrderStatus.Created]: 'Created',
-    [PurchaseOrderStatus.ElectronicallyReadySend]: 'Ready to Send',
-    [PurchaseOrderStatus.Ordered]: 'Ordered',
-    [PurchaseOrderStatus.Preordered]: 'Preordered',
-    [PurchaseOrderStatus.ElectronicallySent]: 'Sent',
-    [PurchaseOrderStatus.ReceivingStarted]: 'Receiving Started',
-    [PurchaseOrderStatus.Done]: 'Done'
-};
-
-const ReceivingStatusLabels = {
-    [ReceivingStatus.Canceled]: 'Canceled',
-    [ReceivingStatus.Created]: 'Created',
-    [ReceivingStatus.PartialReceived]: 'Partially Received',
-    [ReceivingStatus.FullReceived]: 'Fully Received',
-    [ReceivingStatus.Paid]: 'Paid'
-};
-
-// Helper functions
-function getPurchaseOrderStatusLabel(status) {
-    return PurchaseOrderStatusLabels[status] || 'Unknown';
-}
-
-function getReceivingStatusLabel(status) {
-    return ReceivingStatusLabels[status] || 'Unknown';
-}
-
-// Status checks
-function isReceivingComplete(status) {
-    return status >= ReceivingStatus.PartialReceived;
-}
-
-function isPurchaseOrderComplete(status) {
-    return status === PurchaseOrderStatus.Done;
-}
-
-function isPurchaseOrderCanceled(status) {
-    return status === PurchaseOrderStatus.Canceled;
-}
-
-function isReceivingCanceled(status) {
-    return status === ReceivingStatus.Canceled;
-}
-
-module.exports = {
-    PurchaseOrderStatus,
-    ReceivingStatus,
-    PurchaseOrderStatusLabels,
-    ReceivingStatusLabels,
-    getPurchaseOrderStatusLabel,
-    getReceivingStatusLabel,
-    isReceivingComplete,
-    isPurchaseOrderComplete,
-    isPurchaseOrderCanceled,
-    isReceivingCanceled
-}; 

inventory-server/src/utils/apiHelpers.js

@@ -1,45 +0,0 @@
-/**
- * Parses a query parameter value based on its expected type.
- * Throws error for invalid formats. Adjust date handling as needed.
- */
-function parseValue(value, type) {
-    if (value === null || value === undefined || value === '') return null;
-
-    console.log(`Parsing value: "${value}" as type: "${type}"`);
-    
-    switch (type) {
-        case 'number':
-            const num = parseFloat(value);
-            if (isNaN(num)) {
-              console.error(`Invalid number format: "${value}"`);
-              throw new Error(`Invalid number format: "${value}"`);
-            }
-            return num;
-        case 'integer': // Specific type for integer IDs etc.
-             const int = parseInt(value, 10);
-             if (isNaN(int)) {
-               console.error(`Invalid integer format: "${value}"`);
-               throw new Error(`Invalid integer format: "${value}"`);
-             }
-             console.log(`Successfully parsed integer: ${int}`);
-             return int;
-        case 'boolean':
-            if (String(value).toLowerCase() === 'true') return true;
-            if (String(value).toLowerCase() === 'false') return false;
-            console.error(`Invalid boolean format: "${value}"`);
-            throw new Error(`Invalid boolean format: "${value}"`);
-        case 'date':
-            // Basic ISO date format validation (YYYY-MM-DD)
-            if (!String(value).match(/^\d{4}-\d{2}-\d{2}$/)) {
-               console.warn(`Potentially invalid date format passed: "${value}"`);
-               // Optionally throw an error or return null depending on strictness
-               // throw new Error(`Invalid date format (YYYY-MM-DD expected): "${value}"`);
-            }
-            return String(value); // Send as string, let DB handle casting/comparison
-        case 'string':
-        default:
-            return String(value);
-    }
-}
-
-module.exports = { parseValue };

inventory-server/src/utils/csvImporter.js

@@ -1,63 +0,0 @@
-const fs = require('fs');
-const { parse } = require('csv-parse');
-const { v4: uuidv4 } = require('uuid');
-
-async function importProductsFromCSV(filePath, pool) {
-  return new Promise((resolve, reject) => {
-    const products = [];
-    
-    fs.createReadStream(filePath)
-      .pipe(parse({
-        columns: true,
-        skip_empty_lines: true
-      }))
-      .on('data', async (row) => {
-        products.push({
-          id: uuidv4(),
-          sku: row.sku,
-          name: row.name,
-          description: row.description || null,
-          category: row.category || null
-        });
-      })
-      .on('end', async () => {
-        try {
-          const connection = await pool.getConnection();
-          
-          try {
-            await connection.beginTransaction();
-            
-            for (const product of products) {
-              await connection.query(
-                'INSERT INTO products (id, sku, name, description, category) VALUES (?, ?, ?, ?, ?)',
-                [product.id, product.sku, product.name, product.description, product.category]
-              );
-              
-              // Initialize inventory level for the product
-              await connection.query(
-                'INSERT INTO inventory_levels (id, product_id, quantity) VALUES (?, ?, 0)',
-                [uuidv4(), product.id]
-              );
-            }
-            
-            await connection.commit();
-            resolve({ imported: products.length });
-          } catch (error) {
-            await connection.rollback();
-            reject(error);
-          } finally {
-            connection.release();
-          }
-        } catch (error) {
-          reject(error);
-        }
-      })
-      .on('error', (error) => {
-        reject(error);
-      });
-  });
-}
-
-module.exports = {
-  importProductsFromCSV
-}; 

inventory-server/src/utils/db.js

@@ -1,21 +0,0 @@
-const { Pool } = require('pg');
-
-let pool;
-
-function initPool(config) {
-  pool = new Pool(config);
-  return pool;
-}
-
-async function getConnection() {
-  if (!pool) {
-    throw new Error('Database pool not initialized');
-  }
-  return pool.connect();
-}
-
-module.exports = {
-  initPool,
-  getConnection,
-  getPool: () => pool
-}; 

inventory-server/src/utils/dbConnection.js

@@ -1,239 +0,0 @@
-const { Client } = require('ssh2');
-const mysql = require('mysql2/promise');
-const fs = require('fs');
-
-// Connection pooling and cache configuration
-const connectionCache = {
-  ssh: null,
-  dbConnection: null,
-  lastUsed: 0,
-  isConnecting: false,
-  connectionPromise: null,
-  // Cache expiration time in milliseconds (5 minutes)
-  expirationTime: 5 * 60 * 1000,
-  // Cache for query results (key: query string, value: {data, timestamp})
-  queryCache: new Map(),
-  // Cache duration for different query types in milliseconds
-  cacheDuration: {
-    'field-options': 30 * 60 * 1000, // 30 minutes for field options
-    'product-lines': 10 * 60 * 1000, // 10 minutes for product lines
-    'sublines': 10 * 60 * 1000,      // 10 minutes for sublines
-    'taxonomy': 30 * 60 * 1000,      // 30 minutes for taxonomy data
-    'default': 60 * 1000             // 1 minute default
-  }
-};
-
-/**
- * Get a database connection with connection pooling
- * @returns {Promise<{ssh: object, connection: object}>} The SSH and database connection
- */
-async function getDbConnection() {
-  const now = Date.now();
-  
-  // Check if we need to refresh the connection due to inactivity
-  const needsRefresh = !connectionCache.ssh || 
-                       !connectionCache.dbConnection || 
-                       (now - connectionCache.lastUsed > connectionCache.expirationTime);
-  
-  // If connection is still valid, update last used time and return existing connection
-  if (!needsRefresh) {
-    connectionCache.lastUsed = now;
-    return {
-      ssh: connectionCache.ssh,
-      connection: connectionCache.dbConnection
-    };
-  }
-  
-  // If another request is already establishing a connection, wait for that promise
-  if (connectionCache.isConnecting && connectionCache.connectionPromise) {
-    try {
-      await connectionCache.connectionPromise;
-      return {
-        ssh: connectionCache.ssh,
-        connection: connectionCache.dbConnection
-      };
-    } catch (error) {
-      // If that connection attempt failed, we'll try again below
-      console.error('Error waiting for existing connection:', error);
-    }
-  }
-  
-  // Close existing connections if they exist
-  if (connectionCache.dbConnection) {
-    try {
-      await connectionCache.dbConnection.end();
-    } catch (error) {
-      console.error('Error closing existing database connection:', error);
-    }
-  }
-  
-  if (connectionCache.ssh) {
-    try {
-      connectionCache.ssh.end();
-    } catch (error) {
-      console.error('Error closing existing SSH connection:', error);
-    }
-  }
-  
-  // Mark that we're establishing a new connection
-  connectionCache.isConnecting = true;
-  
-  // Create a new promise for this connection attempt
-  connectionCache.connectionPromise = setupSshTunnel().then(tunnel => {
-    const { ssh, stream, dbConfig } = tunnel;
-    
-    return mysql.createConnection({
-      ...dbConfig,
-      stream
-    }).then(connection => {
-      // Store the new connections
-      connectionCache.ssh = ssh;
-      connectionCache.dbConnection = connection;
-      connectionCache.lastUsed = Date.now();
-      connectionCache.isConnecting = false;
-      
-      return {
-        ssh,
-        connection
-      };
-    });
-  }).catch(error => {
-    connectionCache.isConnecting = false;
-    throw error;
-  });
-  
-  // Wait for the connection to be established
-  return connectionCache.connectionPromise;
-}
-
-/**
- * Get cached query results or execute query if not cached
- * @param {string} cacheKey - Unique key to identify the query
- * @param {string} queryType - Type of query (field-options, product-lines, etc.)
- * @param {Function} queryFn - Function to execute if cache miss
- * @returns {Promise<any>} The query result
- */
-async function getCachedQuery(cacheKey, queryType, queryFn) {
-  // Get cache duration based on query type
-  const cacheDuration = connectionCache.cacheDuration[queryType] || connectionCache.cacheDuration.default;
-  
-  // Check if we have a valid cached result
-  const cachedResult = connectionCache.queryCache.get(cacheKey);
-  const now = Date.now();
-  
-  if (cachedResult && (now - cachedResult.timestamp < cacheDuration)) {
-    console.log(`Cache hit for ${queryType} query: ${cacheKey}`);
-    return cachedResult.data;
-  }
-  
-  // No valid cache found, execute the query
-  console.log(`Cache miss for ${queryType} query: ${cacheKey}`);
-  const result = await queryFn();
-  
-  // Cache the result
-  connectionCache.queryCache.set(cacheKey, {
-    data: result,
-    timestamp: now
-  });
-  
-  return result;
-}
-
-/**
- * Setup SSH tunnel to production database
- * @private - Should only be used by getDbConnection
- * @returns {Promise<{ssh: object, stream: object, dbConfig: object}>}
- */
-async function setupSshTunnel() {
-  const sshConfig = {
-    host: process.env.PROD_SSH_HOST,
-    port: process.env.PROD_SSH_PORT || 22,
-    username: process.env.PROD_SSH_USER,
-    privateKey: process.env.PROD_SSH_KEY_PATH
-      ? fs.readFileSync(process.env.PROD_SSH_KEY_PATH)
-      : undefined,
-    compress: true
-  };
-
-  const dbConfig = {
-    host: process.env.PROD_DB_HOST || 'localhost',
-    user: process.env.PROD_DB_USER,
-    password: process.env.PROD_DB_PASSWORD,
-    database: process.env.PROD_DB_NAME,
-    port: process.env.PROD_DB_PORT || 3306,
-    timezone: 'Z'
-  };
-
-  return new Promise((resolve, reject) => {
-    const ssh = new Client();
-
-    ssh.on('error', (err) => {
-      console.error('SSH connection error:', err);
-      reject(err);
-    });
-
-    ssh.on('ready', () => {
-      ssh.forwardOut(
-        '127.0.0.1',
-        0,
-        dbConfig.host,
-        dbConfig.port,
-        (err, stream) => {
-          if (err) reject(err);
-          resolve({ ssh, stream, dbConfig });
-        }
-      );
-    }).connect(sshConfig);
-  });
-}
-
-/**
- * Clear cached query results
- * @param {string} [cacheKey] - Specific cache key to clear (clears all if not provided)
- */
-function clearQueryCache(cacheKey) {
-  if (cacheKey) {
-    connectionCache.queryCache.delete(cacheKey);
-    console.log(`Cleared cache for key: ${cacheKey}`);
-  } else {
-    connectionCache.queryCache.clear();
-    console.log('Cleared all query cache');
-  }
-}
-
-/**
- * Force close all active connections
- * Useful for server shutdown or manual connection reset
- */
-async function closeAllConnections() {
-  if (connectionCache.dbConnection) {
-    try {
-      await connectionCache.dbConnection.end();
-      console.log('Closed database connection');
-    } catch (error) {
-      console.error('Error closing database connection:', error);
-    }
-    connectionCache.dbConnection = null;
-  }
-  
-  if (connectionCache.ssh) {
-    try {
-      connectionCache.ssh.end();
-      console.log('Closed SSH connection');
-    } catch (error) {
-      console.error('Error closing SSH connection:', error);
-    }
-    connectionCache.ssh = null;
-  }
-  
-  connectionCache.lastUsed = 0;
-  connectionCache.isConnecting = false;
-  connectionCache.connectionPromise = null;
-}
-
-module.exports = {
-  getDbConnection,
-  getCachedQuery,
-  clearQueryCache,
-  closeAllConnections
-}; 

inventory/package.json

@@ -6,8 +6,10 @@
   "scripts": {
     "dev": "vite",
     "build": "tsc -b && vite build",
+    "build:deploy": "tsc -b && COPY_BUILD=true DEPLOY_TARGET=netcup DEPLOY_PATH=/var/www/html/inventory/frontend vite build",
     "lint": "eslint .",
-    "preview": "vite preview"
+    "preview": "vite preview",
+    "mount": "../mountremote.command"
   },
   "dependencies": {
     "@dnd-kit/core": "^6.3.1",
@@ -30,12 +32,12 @@
     "@radix-ui/react-scroll-area": "^1.2.2",
     "@radix-ui/react-select": "^2.1.4",
     "@radix-ui/react-separator": "^1.1.1",
-    "@radix-ui/react-slot": "^1.1.2",
+    "@radix-ui/react-slot": "^1.2.4",
     "@radix-ui/react-switch": "^1.1.2",
     "@radix-ui/react-tabs": "^1.1.2",
     "@radix-ui/react-toast": "^1.2.6",
-    "@radix-ui/react-toggle": "^1.1.1",
-    "@radix-ui/react-toggle-group": "^1.1.1",
+    "@radix-ui/react-toggle": "^1.1.10",
+    "@radix-ui/react-toggle-group": "^1.1.11",
     "@radix-ui/react-tooltip": "^1.1.6",
     "@shadcn/ui": "^0.0.4",
     "@tabler/icons-react": "^3.28.1",
@@ -53,7 +55,9 @@
     "cmdk": "^1.0.0",
     "date-fns": "^3.6.0",
     "diff": "^7.0.0",
+    "embla-carousel-react": "^8.6.0",
     "framer-motion": "^12.4.4",
+    "immer": "^11.1.3",
     "input-otp": "^1.4.1",
     "js-levenshtein": "^1.1.6",
     "lodash": "^4.17.21",
@@ -86,6 +90,7 @@
     "@eslint/js": "^9.17.0",
     "@types/fs-extra": "^11.0.4",
     "@types/lodash": "^4.17.15",
+    "@types/luxon": "^3.7.1",
     "@types/node": "^22.10.5",
     "@types/react": "^18.3.18",
     "@types/react-dom": "^18.3.5",

inventory/src/App.tsx

@@ -21,18 +21,24 @@ const Overview = lazy(() => import('./pages/Overview'));
 const Products = lazy(() => import('./pages/Products').then(module => ({ default: module.Products })));
 const Analytics = lazy(() => import('./pages/Analytics').then(module => ({ default: module.Analytics })));
 const Forecasting = lazy(() => import('./pages/Forecasting'));
+const DiscountSimulator = lazy(() => import('./pages/DiscountSimulator'));
+const HtsLookup = lazy(() => import('./pages/HtsLookup'));
 const Vendors = lazy(() => import('./pages/Vendors'));
 const Categories = lazy(() => import('./pages/Categories'));  
 const Brands = lazy(() => import('./pages/Brands'));
 const PurchaseOrders = lazy(() => import('./pages/PurchaseOrders'));
+const BlackFridayDashboard = lazy(() => import('./pages/BlackFridayDashboard'));
 
 // 2. Dashboard app - separate chunk
 const Dashboard = lazy(() => import('./pages/Dashboard'));
 const SmallDashboard = lazy(() => import('./pages/SmallDashboard'));
 
-// 3. Product import - separate chunk  
+// 3. Product import - separate chunk
 const Import = lazy(() => import('./pages/Import').then(module => ({ default: module.Import })));
 
+// Product editor
+const ProductEditor = lazy(() => import('./pages/ProductEditor'));
+
 // 4. Chat archive - separate chunk
 const Chat = lazy(() => import('./pages/Chat').then(module => ({ default: module.Chat })));
 
@@ -102,15 +108,10 @@ function App() {
             </RequireAuth>
           }>
             {/* Core inventory app routes - will be lazy loaded */}
-            <Route index element={
-              <Protected page="dashboard" fallback={<FirstAccessiblePage />}>
-                <Suspense fallback={<PageLoading />}>
-                  <Overview />
-                </Suspense>
-              </Protected>
-            } />
-            <Route path="/" element={
-              <Protected page="dashboard">
+            {/* Default route now prioritizes dashboard, then other pages */}
+            <Route index element={<FirstAccessiblePage />} />
+            <Route path="/overview" element={
+              <Protected page="overview">
                 <Suspense fallback={<PageLoading />}>
                   <Overview />
                 </Suspense>
@@ -158,6 +159,20 @@ function App() {
                 </Suspense>
               </Protected>
             } />
+            <Route path="/discount-simulator" element={
+              <Protected page="discount_simulator">
+                <Suspense fallback={<PageLoading />}>
+                  <DiscountSimulator />
+                </Suspense>
+              </Protected>
+            } />
+            <Route path="/hts-lookup" element={
+              <Protected page="hts_lookup">
+                <Suspense fallback={<PageLoading />}>
+                  <HtsLookup />
+                </Suspense>
+              </Protected>
+            } />
             <Route path="/forecasting" element={
               <Protected page="forecasting">
                 <Suspense fallback={<PageLoading />}>
@@ -173,6 +188,15 @@ function App() {
               </Protected>
             } />
             
+            {/* Product editor */}
+            <Route path="/product-editor" element={
+              <Protected page="product_editor">
+                <Suspense fallback={<PageLoading />}>
+                  <ProductEditor />
+                </Suspense>
+              </Protected>
+            } />
+
             {/* Product import - separate chunk */}
             <Route path="/import" element={
               <Protected page="import">
@@ -199,6 +223,13 @@ function App() {
                 </Suspense>
               </Protected>
             } />
+            <Route path="/dashboard/black-friday" element={
+              <Protected page="black_friday_dashboard">
+                <Suspense fallback={<PageLoading />}>
+                  <BlackFridayDashboard />
+                </Suspense>
+              </Protected>
+            } />
             
             <Route path="*" element={<Navigate to="/" replace />} />
           </Route>
@@ -209,4 +240,3 @@ function App() {
 }
 
 export default App;
-

inventory/src/components/auth/FirstAccessiblePage.tsx

@@ -3,12 +3,18 @@ import { Navigate } from "react-router-dom";
 import { AuthContext } from "@/contexts/AuthContext";
 
 // Define available pages in order of priority
+// Dashboard is first so users with dashboard access default to it
 const PAGES = [
+  { path: "/dashboard", permission: "access:dashboard" },
+  { path: "/dashboard/black-friday", permission: "access:black_friday_dashboard" },
+  { path: "/overview", permission: "access:overview" },
   { path: "/products", permission: "access:products" },
   { path: "/categories", permission: "access:categories" },
   { path: "/vendors", permission: "access:vendors" },
   { path: "/purchase-orders", permission: "access:purchase_orders" },
   { path: "/analytics", permission: "access:analytics" },
+  { path: "/discount-simulator", permission: "access:discount_simulator" },
+  { path: "/hts-lookup", permission: "access:hts_lookup" },
   { path: "/forecasting", permission: "access:forecasting" },
   { path: "/import", permission: "access:import" },
   { path: "/settings", permission: "access:settings" },
@@ -23,14 +29,12 @@ export function FirstAccessiblePage() {
     return null;
   }
   
-  // Admin users have access to all pages, so this component 
-  // shouldn't be rendering for them (handled by App.tsx)
-  if (user.is_admin) {
-    return null;
-  }
-  
-  // Find the first page the user has access to
+  // Admins can access every page, otherwise check explicit permissions
   const firstAccessiblePage = PAGES.find(page => {
+    if (user.is_admin) {
+      return true;
+    }
+
     return user.permissions?.includes(page.permission);
   });