Add remaining templates elements

.gitignore

@@ -26,6 +26,7 @@ dist-ssr
 dashboard/build/**
 dashboard-server/frontend/build/**
 **/build/**
+.fuse_hidden**
 ._*
 
 # Build directories

docs/calculate-issues.md

@@ -0,0 +1,185 @@
+1. **Missing Updates for Reorder Point and Safety Stock** [RESOLVED - product-metrics.js]
+- **Problem:** In the **product_metrics** table (used by the inventory health view), the fields **reorder_point** and **safety_stock** are never updated in the product metrics calculations. Although a helper function (`calculateReorderQuantities`) exists and computes these values, the update query in the `calculateProductMetrics` function does not assign any values to these columns.
+- **Effect:** The inventory health view relies on these fields (using COALESCE to default them to 0), which means that stock might never be classified as "Reorder" or "Healthy" based on the proper reorder point or safety stock calculations.
+- **Example:** Even if a product's base metrics would require a reorder (for example, if its days of inventory are low), the view always shows a value of 0 for reorder_point and safety_stock.
+- **Fix:** Update the product metrics query (or add a subsequent update) so that **pm.reorder_point** and **pm.safety_stock** are calculated (for instance, by integrating the logic from `calculateReorderQuantities`) and stored in the table.
+
+2. **Overwritten Module Exports When Combining Scripts** [RESOLVED - calculate-metrics.js]
+- **Problem:** The code provided shows two distinct exports. The main metrics calculation module exports `calculateMetrics` (along with cancel and getProgress helpers), but later in the same concatenated file the module exports are overwritten.
+- **Effect:** If these two code sections end up in a single module file, the export for the main calculation will be lost. This would break any code that calls the overall metrics calculation.
+- **Example:** An external caller expecting to run `calculateMetrics` would instead receive the `calculateProductMetrics` function.
+- **Fix:** Make sure each script resides in its own module file. Verify that the module boundaries and exports are not accidentally merged or overwritten when deployed.
+
+3. **Potential Formula Issue in EOQ Calculation (Reorder Qty)** [RESOLVED - product-metrics.js]
+- **Problem:** The helper function `calculateReorderQuantities` uses an EOQ formula with a holding cost expressed as a percentage (0.25) rather than a per‐unit cost.
+- **Effect:** If the intent was to use the traditional EOQ formula (which expects a holding cost per unit rather than a percentage), this could lead to an incorrect reorder quantity.
+- **Example:** For a given annual demand and fixed order cost, the computed reorder quantity might be higher or lower than expected.
+- **Fix:** Double-check the EOQ formula. If the intention is to compute based on a percentage, then document that clearly; otherwise, adjust the formula to use the proper holding cost value.
+
+4. **Potential Overlap or Redundancy in GMROI Calculation** [RESOLVED - time-aggregates.js]
+- **Problem:** In the time aggregates function, GMROI is calculated in two steps. The initial INSERT query computes GMROI as
+
+	`CASE WHEN s.inventory_value > 0 THEN (s.total_revenue - s.total_cost) / s.inventory_value ELSE 0 END`
+
+	and then a subsequent UPDATE query recalculates it as an annualized value using gross profit and active days.
+
+
+- **Effect:** Overwriting a computed value may be intentional to refine the metric, but if not coordinated it can cause confusion or unexpected output in the `product_time_aggregates` table.
+- **Example:** A product's GMROI might first appear as a simple ratio but then be updated to a scaled value based on the number of active days, which could lead to inconsistent reporting if not documented.
+- **Fix:** Consolidated the GMROI calculation into a single step in the initial INSERT query, properly handling annualization and NULL values.
+
+5. **Handling of Products Without Orders or Purchase Data** [RESOLVED - time-aggregates.js]
+- **Problem:** In the INSERT query of the time aggregates function, the UNION covers two cases: one for products with order data (from `monthly_sales`) and one for products that have entries in `monthly_stock` but no matching order data.
+- **Effect:** If a product has neither orders nor purchase orders, it won't get an entry in `product_time_aggregates`. Depending on business rules, this might be acceptable or might mean missing data.
+- **Example:** A product that's new or rarely ordered might not appear in the time aggregates view, potentially affecting downstream calculations.
+- **Fix:** Added an `all_products` CTE and modified the JOIN structure to ensure every product gets an entry with appropriate default values, even if it has no orders or purchase orders.
+
+6. **Redundant Recalculation of Vendor Metrics**
+- **Problem:** Similar concepts from prior scripts where cumulative metrics (like **total_revenue** and **total_cost**) are calculated in multiple query steps without necessary validation or optimization. In the vendor metrics script, calculations for total revenue and margin are performed within a `WITH` clause, which is then used in other parts of the process, making it more complex than needed.
+- **Effect:** There's unnecessary duplication in querying the same data multiple times across subqueries. It could result in decreased performance and may even lead to excess computation if the subqueries are not optimized or correctly indexed.
+- **Example:** Vendor sales and vendor purchase orders (PO) metrics are calculated in separate `WITH` clauses, leading to repeated calculations.
+- **Fix:** Synthesize the required metrics into fewer queries or reuse the results within the `WITH` clause itself. Avoid redundant calculations of **revenue** and **cost** unless truly necessary.
+
+7. **Handling Products Without Orders or Purchase Orders**
+- **Problem:** In your `calculateVendorMetrics` script, the initial insert for vendor sales doesn't fully address the products that might not have matching orders or purchase orders. If a vendor has products without any sales within the last 12 months, the results may not be fully accurate unless handled explicitly.
+- **Effect:** If no orders exist for a product associated with a particular vendor, that product will not contribute to the vendor's metrics, potentially omitting important data when calculating **total_orders** or **total_revenue**.
+- **Example:** The scripted statistics fill gaps, but products with no recent purchase or sales orders might not be counted accurately.
+- **Fix:** Include logic to handle scenarios where these products still need to be part of the vendor calculation. Use a `LEFT JOIN` wherever possible to account for cases without sales or purchase orders.
+
+8. **Redundant `ON DUPLICATE KEY UPDATE`**
+- **Problem:** Multiple queries in the `calculateVendorMetrics` script use `ON DUPLICATE KEY UPDATE` clauses to handle repeated metrics updates. This is useful for ensuring the most up-to-date calculations but can cause inconsistencies if multiple calculations happen for the same product or vendor simultaneously.
+- **Effect:** This approach can lead to an inaccurate update of brand-specific data when insertion and update overlap. Each time you add a new batch, an existing entry could be overwritten if not handled correctly.
+- **Example:** Vendor country, category, or sales-related metrics could unintentionally update during processing.
+- **Fix:** Match on current status more robustly in case of existing rows to avoid unnecessary updates. Ensure that the key used for `ON DUPLICATE KEY` aligns with any foreign key relationships that might indicate an already processed entry.
+
+9. **SQL Query Performance with Multiple Nested `WITH` Clauses**
+- **Problem:** Heavily nested queries (especially **WITH** clauses) may lead to slow performance depending on the size of the dataset.
+- **Effect:** Computational burden could be high when the database is large, e.g., querying **purchase orders**, **vendor sales**, and **product info** simultaneously. Even with proper indexes, the deployment might struggle in production environments.
+- **Example:** Multiple `WITH` clauses in the vendor and brand metrics calculation scripts might work fine in small datasets but degrade performance in production.
+- **Fix:** Combine some subqueries and reduce the layer of computations needed for calculating final metrics. Test performance on a production-sized dataset to see how nested queries are handled.
+
+10. **Missing Updates for Reorder Metrics (Vendor/Brand)**
+- **Previously Identified Issue:** Inconsistent updates for **reorder_point** and **safety_stock** across earlier scripts.
+- **Current Impact on This Script:** The vendor and brand metrics do not have explicit updates for reorder point or safety stock, which are essential for inventory evaluation.
+- **Effect:** The correct thresholds and reorder logic for vendor product inventory aren't fully accounted for in these scripts.
+- **Fix:** Integrate relevant logic to update **reorder_point** or **safety_stock** within the vendor and brand metrics calculations. Ensure that it's consistently computed and stored.
+
+11. **Data Integrity and Consistency** 
+
+**w**hen tracking sales growth or performance
+
+
+- **Problem:** Brand metrics include a sales growth clause where negative results can sometimes be skewed severely if period data varies considerably.
+- **Effect:** If period boundaries are incorrect or records are missing, this can create drastic growth rate calculations.
+- **Example:** If the "previous" period has no sales but "current" has a substantial increase, the growth rate will show as **100%**.
+- **Fix:** Implement checks that ensure both periods are valid and that the system calculates growth accurately, avoiding growth rates based solely on potential outliers. Replace consistent gaps with a no-growth rate or a meaningful zero.
+
+12. **Exclusion of Vendors With No Sales**
+
+	The vendor metrics query is driven by the `vendor_sales` CTE, which aggregates data only for vendors that have orders in the past 12 months.
+
+
+	- **Impact:** Vendors that have purchase activity (or simply exist in vendor_details) but no recent sales won't show up in vendor_metrics. This could cause the frontend to miss metrics for vendors that might still be important.
+	- **Fix:** Consider adding a UNION or changing the driving set so that all vendors (for example, from vendor_details) are included—even if they have zero sales.
+13. **Identical Formulas for On-Time Delivery and Order Fill Rates**
+
+	Both metrics are calculated as `(received_orders / total_orders) * 100`.
+
+
+	- **Impact:** If the business expects these to be distinct (for example, one might factor in on-time receipt versus mere receipt), then showing identical values on the frontend could be misleading.
+	- **Fix:** Verify and adjust the formulas if on-time delivery and order fill rates should be computed differently.
+14. **Handling Nulls and Defaults in Aggregations**
+
+	The query uses COALESCE in most places, but be sure that every aggregated value (like average lead time) correctly defaults when no data is present.
+
+
+	- **Impact:** Incorrect defaults might cause odd or missing numbers on the production interface.
+	- **Fix:** Double-check that all numeric aggregates reliably default to 0 where needed.
+
+15. **Inconsistent Stock Filtering Conditions**
+
+	In the main brand metrics query the CTE filters products with the condition
+
+	`p.stock_quantity <= 5000 AND p.stock_quantity >= 0`
+
+	whereas in the brand time-based metrics query the condition is only `p.stock_quantity <= 5000`.
+
+
+	- **Impact:** This discrepancy may lead to inconsistent numbers (for example, if any products have negative stock, which might be due to data issues) between overall brand metrics and time-based metrics on the frontend.
+	- **Fix:** Standardize the filtering criteria so that both queries treat out-of-range stock values in the same way.
+16. **Growth Rate Calculation Periods**
+
+	The growth rate is computed by comparing revenue from the last 3 months ("current") against a period from 15–12 months ago ("previous").
+
+
+	- **Impact:** This narrow window may not reflect typical year-over-year performance and could lead to volatile or unexpected growth percentages on the frontend.
+	- **Fix:** Revisit the business logic for growth—if a longer or different comparison period is preferred, adjust the date intervals accordingly.
+17. **Potential NULLs in Aggregated Time-Based Metrics**
+
+	In the brand time-based metrics query, aggregate expressions such as `SUM(o.quantity * o.price)` aren't wrapped with COALESCE.
+
+
+	- **Impact:** If there are no orders for a given brand/month, these sums might return NULL rather than 0, which could propagate into the frontend display.
+	- **Fix:** Wrap such aggregates in COALESCE (e.g. `COALESCE(SUM(o.quantity * o.price), 0)`) to ensure a default numeric value.
+
+18. **Grouping by Category Status in Base Metrics Insert**
+	- **Problem:** The INSERT for base category metrics groups by both `c.cat_id` and `c.status` even though the table's primary key is just `category_id`.
+	- **Effect:** If a category's status changes over time, the grouping may produce unexpected updates (or even multiple groups before the duplicate key update kicks in), possibly causing the wrong status or aggregated figures to be stored.
+	- **Example:** A category that toggles between "active" and "inactive" might have its metrics calculated differently on different runs.
+	- **Fix:** Ensure that the grouping keys match the primary key (or that the status update logic is exactly as intended) so that a single row per category is maintained.
+19. **Potential Null Handling in Margin Calculations**
+	- **Problem:** In the query for category time metrics, the calculation of average margin uses expressions such as `SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0)))` without using `COALESCE` on `p.cost_price`.
+	- **Effect:** If any product's `cost_price` is `NULL`, then `GREATEST(p.cost_price, 0)` returns `NULL` and the resulting sum (and thus the margin) could become `NULL` rather than defaulting to 0. This might lead to missing or misleading margin figures on the frontend.
+	- **Example:** A product with a missing cost price would make the entire margin expression evaluate to `NULL` even when sales exist.
+	- **Fix:** Replace `GREATEST(p.cost_price, 0)` with `GREATEST(COALESCE(p.cost_price, 0), 0)` (or simply use `COALESCE(p.cost_price, 0)`) to ensure that missing values are handled.
+20. **Data Coverage in Growth Rate Calculation**
+	- **Problem:** The growth rate update depends on multiple CTEs (current period, previous period, and trend analysis) that require a minimum amount of data (for instance, `HAVING COUNT(*) >= 6` in the trend_stats CTE).
+	- **Effect:** Categories with insufficient historical data will fall into the "ELSE" branch (or may even be skipped if no revenue is present), which might result in a growth rate of 0.0 or an unexpected value.
+	- **Example:** A newly created category that has only two months of data won't have trend analysis, so its growth rate will be calculated solely by the simple difference, which might not reflect true performance.
+	- **Fix:** Confirm that this fallback behavior is acceptable for production; if not, adjust the logic so that every category receives a consistent growth rate even with sparse data.
+21. **Omission of Forecasts for Zero–Sales Categories**
+	- **Observation:** The category–sales metrics query uses a `HAVING AVG(cs.daily_quantity) > 0` clause.
+	- **Effect:** Categories without any average daily sales will not receive a forecast record in `category_sales_metrics`. If the frontend expects a row (even with zeros) for every category, this will lead to missing data.
+	- **Fix:** Verify that it's acceptable for categories with no sales to have no forecast entry. If not, adjust the query so that a default forecast (with zeros) is inserted.
+
+22. **Randomness in Category-Level Forecast Revenue Calculation**
+	- **Problem:** In the category-level forecasts query, the forecast revenue is multiplied by a factor of `(0.95 + (RAND() * 0.1))`.
+	- **Effect:** This introduces randomness into the forecast figures so that repeated runs could yield slightly different values. If deterministic forecasts are expected on the production frontend, this could lead to inconsistent displays.
+	- **Example:** The same category might show a 5% higher forecast on one run and 3% on another because of the random multiplier.
+	- **Fix:** Confirm that this randomness is intentional for your forecasting model; if forecasts are meant to be reproducible, remove or replace the `RAND()` factor with a fixed multiplier.
+23. **Multi-Statement Cleanup of Temporary Tables**
+	- **Problem:** The cleanup query drops multiple temporary tables in one call (separated by semicolons).
+	- **Effect:** If your Node.js MySQL driver isn't configured to allow multi-statement execution, this query may fail, leaving temporary tables behind. Leftover temporary tables might eventually cause conflicts or resource issues.
+	- **Example:** Running the cleanup query could produce an error like "multi-statement queries not enabled," preventing proper cleanup.
+	- **Fix:** Either configure your database connection to allow multi-statements or issue separate queries for each temporary table drop to ensure that the cleanup runs successfully.
+24. **Handling Products with No Sales Data**
+	- **Problem:** In the product-level forecast calculation, the CTE `daily_stats` includes a `HAVING AVG(ds.daily_quantity) > 0` clause.
+	- **Effect:** Products that have no sales (or a zero average daily quantity) will be excluded from the forecasts. This means the frontend won't show forecasts for non–selling products, which might be acceptable but could also be a completeness issue.
+	- **Example:** A product that has never sold will not appear in the `sales_forecasts` table.
+	- **Fix:** Confirm that it is intended for forecasts to be generated only for products with some sales activity. If forecasts are required for all products, adjust the query to insert default forecast records for products with zero sales.
+25. **Complexity of the Forecast Formula Involving the Seasonality Factor**
+- **Issue:**
+
+	The sales forecast calculations incorporate an adjustment factor using `COALESCE(sf.seasonality_factor, 0)` to modify forecast units and revenue. This means that if the seasonality data is missing (or not populated), the factor defaults to 0.
+
+
+- **Potential Problem:**
+
+	A default value of 0 will drastically alter the forecast calculations—often leading to a forecast of 0 or an overly dampened forecast—when in reality the intended behavior might be to use a neutral multiplier (typically 1.0). This could result in forecasts that are not reflective of the actual seasonal impact, thereby skewing the figures that reach the frontend.
+
+
+- **Fix:**
+
+	Review your data source for seasonality (the `sales_seasonality` table) and ensure it's consistently populated. Alternatively, if missing seasonality data is possible, consider using a more neutral default (such as 1.0) in your COALESCE. This change would prevent the forecast formulas from over-simplifying (or even nullifying) the forecast output due to missing seasonality factors.
+
+
+26. **Group By with Seasonality Factor Variability**
+	- **Observation:** In the forecast insertion query, the GROUP BY clause includes `sf.seasonality_factor` along with other fields.
+	- **Effect:** If the seasonality factor differs (or is `NULL` versus a value) for different forecast dates, this might result in multiple rows for the same product and forecast date. However, the `ON DUPLICATE KEY UPDATE` clause will merge them—but only if the primary key (pid, forecast_date) is truly unique.
+	- **Fix:** Verify that the grouping produces exactly one row per product per forecast date. If there's potential for multiple rows due to seasonality variability, consider applying a COALESCE or an aggregation on the seasonality factor so that it does not affect grouping.
+
+27. **Memory Management for Temporary Tables** [RESOLVED - calculate-metrics.js]
+- **Problem:** In metrics calculations, temporary tables aren't always properly cleaned up if the process fails between creation and the DROP statement.
+- **Effect:** If a process fails after creating temporary tables but before dropping them, these tables remain in memory until the connection is closed. In a production environment with multiple calculation runs, this could lead to memory leaks or table name conflicts.
+- **Example:** The `temp_revenue_ranks` table creation in ABC classification could remain if the process fails before reaching the DROP statement.
+- **Fix:** Implement proper cleanup in a finally block or use transaction management that ensures temporary tables are always cleaned up, even in failure scenarios.

ecosystem.config.js

@@ -1,5 +1,209 @@
+// ecosystem.config.js
+const path = require('path');
+const dotenv = require('dotenv');
+
+// Load environment variables safely with error handling
+const loadEnvFile = (envPath) => {
+  try {
+    console.log('Loading env from:', envPath);
+    const result = dotenv.config({ path: envPath });
+    if (result.error) {
+      console.warn(`Warning: .env file not found or invalid at ${envPath}:`, result.error.message);
+      return {};
+    }
+    console.log('Env variables loaded from', envPath, ':', Object.keys(result.parsed || {}));
+    return result.parsed || {};
+  } catch (error) {
+    console.warn(`Warning: Error loading .env file at ${envPath}:`, error.message);
+    return {};
+  }
+}
+
+// Load environment variables for each server
+const authEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/auth-server/.env'));
+const aircallEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/aircall-server/.env'));
+const klaviyoEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/klaviyo-server/.env'));
+const metaEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/meta-server/.env'));
+const googleAnalyticsEnv = require('dotenv').config({ 
+  path: path.resolve(__dirname, 'dashboard/google-server/.env') 
+}).parsed || {};
+const typeformEnv = loadEnvFile(path.resolve(__dirname, 'dashboard/typeform-server/.env'));
+const inventoryEnv = loadEnvFile(path.resolve(__dirname, 'inventory/.env'));
+
+// Common log settings for all apps
+const logSettings = {
+  log_rotate: true,
+  max_size: '10M',
+  retain: '10',
+  log_date_format: 'YYYY-MM-DD HH:mm:ss'
+};
+
+// Common app settings
+const commonSettings = {
+  instances: 1,
+  exec_mode: 'fork',
+  autorestart: true,
+  watch: false,
+  max_memory_restart: '1G',
+  time: true,
+  ...logSettings,
+  ignore_watch: [
+    'node_modules',
+    'logs',
+    '.git',
+    '*.log'
+  ],
+  min_uptime: 5000,
+  max_restarts: 5,
+  restart_delay: 4000,
+  listen_timeout: 50000,
+  kill_timeout: 5000,
+  node_args: '--max-old-space-size=1536'
+};
+
 module.exports = {
   apps: [
+    {
+      ...commonSettings,
+      name: 'auth-server',
+      script: './dashboard/auth-server/index.js',
+      env: {
+        NODE_ENV: 'production',
+        PORT: 3003,
+        ...authEnv
+      },
+      error_file: 'dashboard/auth-server/logs/pm2/err.log',
+      out_file: 'dashboard/auth-server/logs/pm2/out.log',
+      log_file: 'dashboard/auth-server/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        PORT: 3003
+      },
+      env_development: {
+        NODE_ENV: 'development',
+        PORT: 3003
+      }
+    },
+    {
+      ...commonSettings,
+      name: 'aircall-server',
+      script: './dashboard/aircall-server/server.js',
+      env: {
+        NODE_ENV: 'production',
+        AIRCALL_PORT: 3002,
+        ...aircallEnv
+      },
+      error_file: 'dashboard/aircall-server/logs/pm2/err.log',
+      out_file: 'dashboard/aircall-server/logs/pm2/out.log',
+      log_file: 'dashboard/aircall-server/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        AIRCALL_PORT: 3002
+      }
+    },
+    {
+      ...commonSettings,
+      name: 'klaviyo-server',
+      script: './dashboard/klaviyo-server/server.js',
+      env: {
+        NODE_ENV: 'production',
+        KLAVIYO_PORT: 3004,
+        ...klaviyoEnv
+      },
+      error_file: 'dashboard/klaviyo-server/logs/pm2/err.log',
+      out_file: 'dashboard/klaviyo-server/logs/pm2/out.log',
+      log_file: 'dashboard/klaviyo-server/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        KLAVIYO_PORT: 3004
+      }
+    },
+    {
+      ...commonSettings,
+      name: 'meta-server',
+      script: './dashboard/meta-server/server.js',
+      env: {
+        NODE_ENV: 'production',
+        PORT: 3005,
+        ...metaEnv
+      },
+      error_file: 'dashboard/meta-server/logs/pm2/err.log',
+      out_file: 'dashboard/meta-server/logs/pm2/out.log',
+      log_file: 'dashboard/meta-server/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        PORT: 3005
+      }
+    },
+    {
+      name: "gorgias-server",
+      script: "./dashboard/gorgias-server/server.js",
+      env: {
+        NODE_ENV: "development",
+        PORT: 3006
+      },
+      env_production: {
+        NODE_ENV: "production",
+        PORT: 3006
+      },
+      error_file: "dashboard/logs/gorgias-server-error.log",
+      out_file: "dashboard/logs/gorgias-server-out.log",
+      log_file: "dashboard/logs/gorgias-server-combined.log",
+      time: true
+    },
+    {
+      ...commonSettings,
+      name: 'google-server',
+      script: path.resolve(__dirname, 'dashboard/google-server/server.js'),
+      watch: false,
+      env: {
+        NODE_ENV: 'production',
+        GOOGLE_ANALYTICS_PORT: 3007,
+        ...googleAnalyticsEnv
+      },
+      error_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/err.log'),
+      out_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/out.log'),
+      log_file: path.resolve(__dirname, 'dashboard/google-server/logs/pm2/combined.log'),
+      env_production: {
+        NODE_ENV: 'production',
+        GOOGLE_ANALYTICS_PORT: 3007
+      }
+    },
+    {
+      ...commonSettings,
+      name: 'typeform-server',
+      script: './dashboard/typeform-server/server.js',
+      env: {
+        NODE_ENV: 'production',
+        TYPEFORM_PORT: 3008,
+        ...typeformEnv
+      },
+      error_file: 'dashboard/typeform-server/logs/pm2/err.log',
+      out_file: 'dashboard/typeform-server/logs/pm2/out.log',
+      log_file: 'dashboard/typeform-server/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        TYPEFORM_PORT: 3008
+      }
+    },
+    {
+      ...commonSettings,
+      name: 'inventory-server',
+      script: './inventory/src/server.js',
+      env: {
+        NODE_ENV: 'production',
+        PORT: 3010,
+        ...inventoryEnv
+      },
+      error_file: 'inventory/logs/pm2/err.log',
+      out_file: 'inventory/logs/pm2/out.log',
+      log_file: 'inventory/logs/pm2/combined.log',
+      env_production: {
+        NODE_ENV: 'production',
+        PORT: 3010,
+        ...inventoryEnv
+      }
+    },
     {
       ...commonSettings,
       name: 'new-auth-server',
@@ -7,16 +211,12 @@ module.exports = {
       env: {
         NODE_ENV: 'production',
         AUTH_PORT: 3011,
+        ...inventoryEnv,
         JWT_SECRET: process.env.JWT_SECRET
       },
       error_file: 'inventory-server/auth/logs/pm2/err.log',
       out_file: 'inventory-server/auth/logs/pm2/out.log',
-      log_file: 'inventory-server/auth/logs/pm2/combined.log',
-      env_production: {
-        NODE_ENV: 'production',
-        AUTH_PORT: 3011,
-        JWT_SECRET: process.env.JWT_SECRET
-      }
+      log_file: 'inventory-server/auth/logs/pm2/combined.log'
     }
   ]
-}; 
+};

inventory-server/auth/add-user.js

@@ -0,0 +1,103 @@
+require('dotenv').config({ path: '../.env' });
+const bcrypt = require('bcrypt');
+const { Pool } = require('pg');
+const inquirer = require('inquirer');
+
+// Log connection details for debugging (remove in production)
+console.log('Attempting to connect with:', {
+  host: process.env.DB_HOST,
+  user: process.env.DB_USER,
+  database: process.env.DB_NAME,
+  port: process.env.DB_PORT
+});
+
+const pool = new Pool({
+  host: process.env.DB_HOST,
+  user: process.env.DB_USER,
+  password: process.env.DB_PASSWORD,
+  database: process.env.DB_NAME,
+  port: process.env.DB_PORT,
+});
+
+async function promptUser() {
+  const questions = [
+    {
+      type: 'input',
+      name: 'username',
+      message: 'Enter username:',
+      validate: (input) => {
+        if (input.length < 3) {
+          return 'Username must be at least 3 characters long';
+        }
+        return true;
+      }
+    },
+    {
+      type: 'password',
+      name: 'password',
+      message: 'Enter password:',
+      mask: '*',
+      validate: (input) => {
+        if (input.length < 8) {
+          return 'Password must be at least 8 characters long';
+        }
+        return true;
+      }
+    },
+    {
+      type: 'password',
+      name: 'confirmPassword',
+      message: 'Confirm password:',
+      mask: '*',
+      validate: (input, answers) => {
+        if (input !== answers.password) {
+          return 'Passwords do not match';
+        }
+        return true;
+      }
+    }
+  ];
+
+  return inquirer.prompt(questions);
+}
+
+async function addUser() {
+  try {
+    // Get user input
+    const answers = await promptUser();
+    const { username, password } = answers;
+
+    // Hash password
+    const saltRounds = 10;
+    const hashedPassword = await bcrypt.hash(password, saltRounds);
+
+    // Check if user already exists
+    const checkResult = await pool.query(
+      'SELECT id FROM users WHERE username = $1',
+      [username]
+    );
+
+    if (checkResult.rows.length > 0) {
+      console.error('Error: Username already exists');
+      process.exit(1);
+    }
+
+    // Insert new user
+    const result = await pool.query(
+      'INSERT INTO users (username, password) VALUES ($1, $2) RETURNING id',
+      [username, hashedPassword]
+    );
+
+    console.log(`User ${username} created successfully with id ${result.rows[0].id}`);
+  } catch (error) {
+    console.error('Error creating user:', error);
+    console.error('Error details:', error.message);
+    if (error.code) {
+      console.error('Error code:', error.code);
+    }
+  } finally {
+    await pool.end();
+  }
+}
+
+addUser(); 

inventory-server/auth/add_user.js

@@ -1,41 +0,0 @@
-const bcrypt = require('bcrypt');
-const mysql = require('mysql2/promise');
-const readline = require('readline').createInterface({
-  input: process.stdin,
-  output: process.stdout,
-});
-require('dotenv').config({ path: '../.env' });
-
-const dbConfig = {
-  host: process.env.DB_HOST,
-  user: process.env.DB_USER,
-  password: process.env.DB_PASSWORD,
-  database: process.env.DB_NAME,
-};
-
-async function addUser() {
-  const username = await askQuestion('Enter username: ');
-  const password = await askQuestion('Enter password: ');
-
-  const hashedPassword = await bcrypt.hash(password, 10);
-
-  const connection = await mysql.createConnection(dbConfig);
-
-  try {
-    await connection.query('INSERT INTO users (username, password) VALUES (?, ?)', [username, hashedPassword]);
-    console.log(`User ${username} added successfully.`);
-  } catch (error) {
-    console.error('Error adding user:', error);
-  } finally {
-    connection.end();
-    readline.close();
-  }
-}
-
-function askQuestion(query) {
-  return new Promise(resolve => readline.question(query, ans => {
-    resolve(ans);
-  }));
-}
-
-addUser(); 

inventory-server/auth/package.json

@@ -1,21 +1,19 @@
 {
-  "name": "auth-server",
+  "name": "inventory-auth-server",
   "version": "1.0.0",
-  "description": "Authentication server for inventory management",
+  "description": "Authentication server for inventory management system",
   "main": "server.js",
   "scripts": {
-    "start": "node server.js",
-    "dev": "nodemon server.js",
-    "add_user": "node add_user.js"
+    "start": "node server.js"
   },
   "dependencies": {
     "bcrypt": "^5.1.1",
     "cors": "^2.8.5",
-    "dotenv": "^16.4.5",
+    "dotenv": "^16.4.7",
     "express": "^4.18.2",
-    "jsonwebtoken": "^9.0.2"
-  },
-  "devDependencies": {
-    "nodemon": "^3.1.0"
+    "inquirer": "^8.2.6",
+    "jsonwebtoken": "^9.0.2",
+    "morgan": "^1.10.0",
+    "pg": "^8.11.3"
   }
-} 
+}

inventory-server/auth/schema.sql

@@ -1,6 +1,6 @@
-CREATE TABLE `users` (
-  `id` INT AUTO_INCREMENT PRIMARY KEY,
-  `username` VARCHAR(255) NOT NULL UNIQUE,
-  `password` VARCHAR(255) NOT NULL,
-  `created_at` TIMESTAMP DEFAULT CURRENT_TIMESTAMP
+CREATE TABLE users (
+  id SERIAL PRIMARY KEY,
+  username VARCHAR(255) NOT NULL UNIQUE,
+  password VARCHAR(255) NOT NULL,
+  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
 ); 

inventory-server/auth/server.js

@@ -1,135 +1,102 @@
+require('dotenv').config({ path: '../.env' });
 const express = require('express');
+const cors = require('cors');
 const bcrypt = require('bcrypt');
 const jwt = require('jsonwebtoken');
-const cors = require('cors');
-const mysql = require('mysql2/promise');
-require('dotenv').config({ path: '../.env' });
+const { Pool } = require('pg');
+const morgan = require('morgan');
+
+// Log startup configuration
+console.log('Starting auth server with config:', {
+  host: process.env.DB_HOST,
+  user: process.env.DB_USER,
+  database: process.env.DB_NAME,
+  port: process.env.DB_PORT,
+  auth_port: process.env.AUTH_PORT
+});
 
 const app = express();
-const PORT = process.env.AUTH_PORT || 3011;
+const port = process.env.AUTH_PORT || 3011;
 
 // Database configuration
-const dbConfig = {
+const pool = new Pool({
   host: process.env.DB_HOST,
   user: process.env.DB_USER,
   password: process.env.DB_PASSWORD,
   database: process.env.DB_NAME,
-};
+  port: process.env.DB_PORT,
+});
 
-// Create a connection pool
-const pool = mysql.createPool(dbConfig);
-
-app.use(cors({
-  origin: [
-    'https://inventory.kent.pw',
-    'http://localhost:5173',
-    'http://127.0.0.1:5173',
-    /^http:\/\/192\.168\.\d+\.\d+(:\d+)?$/,
-    /^http:\/\/10\.\d+\.\d+\.\d+(:\d+)?$/
-  ],
-  methods: ['GET', 'POST', 'OPTIONS'],
-  allowedHeaders: ['Content-Type', 'Authorization', 'X-Requested-With'],
-  credentials: true,
-  exposedHeaders: ['set-cookie']
-}));
+// Middleware
 app.use(express.json());
-
-// Debug middleware to log request details
-app.use((req, res, next) => {
-  console.log('Request details:', {
-    method: req.method,
-    url: req.url,
-    origin: req.get('Origin'),
-    headers: req.headers,
-    body: req.body,
-  });
-  next();
-});
-
-// Registration endpoint
-app.post('/register', async (req, res) => {
-  try {
-    const { username, password } = req.body;
-    const hashedPassword = await bcrypt.hash(password, 10);
-
-    const connection = await pool.getConnection();
-    await connection.query('INSERT INTO users (username, password) VALUES (?, ?)', [username, hashedPassword]);
-    connection.release();
-
-    res.status(201).json({ message: 'User registered successfully' });
-  } catch (error) {
-    console.error('Registration error:', error);
-    res.status(500).json({ error: 'Registration failed' });
-  }
-});
+app.use(morgan('combined'));
+app.use(cors({
+  origin: ['http://localhost:5173', 'https://inventory.kent.pw'],
+  credentials: true
+}));
 
 // Login endpoint
 app.post('/login', async (req, res) => {
+  const { username, password } = req.body;
+
   try {
-    const { username, password } = req.body;
-    console.log(`Login attempt for user: ${username}`);
-
-    const connection = await pool.getConnection();
-    const [rows] = await connection.query(
-      'SELECT * FROM users WHERE username = ?',
-      [username],
+    // Get user from database
+    const result = await pool.query(
+      'SELECT id, username, password FROM users WHERE username = $1',
+      [username]
     );
-    connection.release();
 
-    if (rows.length === 1) {
-      const user = rows[0];
-      const passwordMatch = await bcrypt.compare(password, user.password);
+    const user = result.rows[0];
 
-      if (passwordMatch) {
-        console.log(`User ${username} authenticated successfully`);
-        const token = jwt.sign(
-          { username: user.username },
-          process.env.JWT_SECRET,
-          { expiresIn: '1h' },
-        );
-        res.json({ token });
-      } else {
-        console.error(`Invalid password for user: ${username}`);
-        res.status(401).json({ error: 'Invalid credentials' });
-      }
-    } else {
-      console.error(`User not found: ${username}`);
-      res.status(401).json({ error: 'Invalid credentials' });
+    // Check if user exists and password is correct
+    if (!user || !(await bcrypt.compare(password, user.password))) {
+      return res.status(401).json({ error: 'Invalid username or password' });
     }
+
+    // Generate JWT token
+    const token = jwt.sign(
+      { userId: user.id, username: user.username },
+      process.env.JWT_SECRET,
+      { expiresIn: '24h' }
+    );
+
+    res.json({ token });
   } catch (error) {
     console.error('Login error:', error);
-    res.status(500).json({ error: 'Login failed' });
+    res.status(500).json({ error: 'Internal server error' });
   }
 });
 
-// Protected endpoint example
+// Protected route to verify token
 app.get('/protected', async (req, res) => {
   const authHeader = req.headers.authorization;
+
   if (!authHeader) {
-    return res.status(401).json({ error: 'Unauthorized' });
+    return res.status(401).json({ error: 'No token provided' });
   }
 
-  const token = authHeader.split(' ')[1];
   try {
+    const token = authHeader.split(' ')[1];
     const decoded = jwt.verify(token, process.env.JWT_SECRET);
-
-    // Optionally, you can fetch the user from the database here
-    // to verify that the user still exists or to get more user information
-    const connection = await pool.getConnection();
-    const [rows] = await connection.query('SELECT * FROM users WHERE username = ?', [decoded.username]);
-    connection.release();
-
-    if (rows.length === 0) {
-      return res.status(401).json({ error: 'User not found' });
-    }
-
-    res.json({ message: 'Protected resource accessed', user: decoded });
+    res.json({ userId: decoded.userId, username: decoded.username });
   } catch (error) {
-    console.error('Protected endpoint error:', error);
-    res.status(403).json({ error: 'Invalid token' });
+    console.error('Token verification error:', error);
+    res.status(401).json({ error: 'Invalid token' });
   }
 });
 
-app.listen(PORT, "0.0.0.0", () => {
-  console.log(`Auth server running on port ${PORT}`);
-}); 
+// Health check endpoint
+app.get('/health', (req, res) => {
+  res.json({ status: 'healthy' });
+});
+
+// Error handling middleware
+app.use((err, req, res, next) => {
+  console.error(err.stack);
+  res.status(500).json({ error: 'Something broke!' });
+});
+
+// Start server
+app.listen(port, () => {
+  console.log(`Auth server running on port ${port}`);
+});

inventory-server/db/config-schema.sql

@@ -1,150 +1,207 @@
 -- Configuration tables schema
 
+-- Create function for updating timestamps if it doesn't exist
+CREATE OR REPLACE FUNCTION update_updated_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$$ language 'plpgsql';
+
+-- Create function for updating updated_at timestamps
+CREATE OR REPLACE FUNCTION update_updated_at_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated_at = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$$ language 'plpgsql';
+
 -- Stock threshold configurations
-CREATE TABLE IF NOT EXISTS stock_thresholds (
-    id INT NOT NULL,
+CREATE TABLE stock_thresholds (
+    id INTEGER NOT NULL,
     category_id BIGINT,  -- NULL means default/global threshold
     vendor VARCHAR(100), -- NULL means applies to all vendors
-    critical_days INT NOT NULL DEFAULT 7,
-    reorder_days INT NOT NULL DEFAULT 14,
-    overstock_days INT NOT NULL DEFAULT 90,
-    low_stock_threshold INT NOT NULL DEFAULT 5,
-    min_reorder_quantity INT NOT NULL DEFAULT 1,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    critical_days INTEGER NOT NULL DEFAULT 7,
+    reorder_days INTEGER NOT NULL DEFAULT 14,
+    overstock_days INTEGER NOT NULL DEFAULT 90,
+    low_stock_threshold INTEGER NOT NULL DEFAULT 5,
+    min_reorder_quantity INTEGER NOT NULL DEFAULT 1,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE KEY unique_category_vendor (category_id, vendor),
-    INDEX idx_st_metrics (category_id, vendor)
+    UNIQUE (category_id, vendor)
 );
 
+CREATE TRIGGER update_stock_thresholds_updated
+    BEFORE UPDATE ON stock_thresholds
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
+CREATE INDEX idx_st_metrics ON stock_thresholds(category_id, vendor);
+
 -- Lead time threshold configurations
-CREATE TABLE IF NOT EXISTS lead_time_thresholds (
-    id INT NOT NULL,
+CREATE TABLE lead_time_thresholds (
+    id INTEGER NOT NULL,
     category_id BIGINT,  -- NULL means default/global threshold
     vendor VARCHAR(100), -- NULL means applies to all vendors
-    target_days INT NOT NULL DEFAULT 14,
-    warning_days INT NOT NULL DEFAULT 21,
-    critical_days INT NOT NULL DEFAULT 30,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    target_days INTEGER NOT NULL DEFAULT 14,
+    warning_days INTEGER NOT NULL DEFAULT 21,
+    critical_days INTEGER NOT NULL DEFAULT 30,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE KEY unique_category_vendor (category_id, vendor)
+    UNIQUE (category_id, vendor)
 );
 
+CREATE TRIGGER update_lead_time_thresholds_updated
+    BEFORE UPDATE ON lead_time_thresholds
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
 -- Sales velocity window configurations
-CREATE TABLE IF NOT EXISTS sales_velocity_config (
-    id INT NOT NULL,
+CREATE TABLE sales_velocity_config (
+    id INTEGER NOT NULL,
     category_id BIGINT,  -- NULL means default/global threshold
     vendor VARCHAR(100), -- NULL means applies to all vendors
-    daily_window_days INT NOT NULL DEFAULT 30,
-    weekly_window_days INT NOT NULL DEFAULT 7,
-    monthly_window_days INT NOT NULL DEFAULT 90,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    daily_window_days INTEGER NOT NULL DEFAULT 30,
+    weekly_window_days INTEGER NOT NULL DEFAULT 7,
+    monthly_window_days INTEGER NOT NULL DEFAULT 90,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE KEY unique_category_vendor (category_id, vendor),
-    INDEX idx_sv_metrics (category_id, vendor)
+    UNIQUE (category_id, vendor)
 );
 
+CREATE TRIGGER update_sales_velocity_config_updated
+    BEFORE UPDATE ON sales_velocity_config
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
+CREATE INDEX idx_sv_metrics ON sales_velocity_config(category_id, vendor);
+
 -- ABC Classification configurations
-CREATE TABLE IF NOT EXISTS abc_classification_config (
-    id INT NOT NULL PRIMARY KEY,
+CREATE TABLE abc_classification_config (
+    id INTEGER NOT NULL PRIMARY KEY,
     a_threshold DECIMAL(5,2) NOT NULL DEFAULT 20.0,
     b_threshold DECIMAL(5,2) NOT NULL DEFAULT 50.0,
-    classification_period_days INT NOT NULL DEFAULT 90,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
+    classification_period_days INTEGER NOT NULL DEFAULT 90,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
 );
 
+CREATE TRIGGER update_abc_classification_config_updated
+    BEFORE UPDATE ON abc_classification_config
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
 -- Safety stock configurations
-CREATE TABLE IF NOT EXISTS safety_stock_config (
-    id INT NOT NULL,
+CREATE TABLE safety_stock_config (
+    id INTEGER NOT NULL,
     category_id BIGINT,  -- NULL means default/global threshold
     vendor VARCHAR(100), -- NULL means applies to all vendors
-    coverage_days INT NOT NULL DEFAULT 14,
+    coverage_days INTEGER NOT NULL DEFAULT 14,
     service_level DECIMAL(5,2) NOT NULL DEFAULT 95.0,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE KEY unique_category_vendor (category_id, vendor),
-    INDEX idx_ss_metrics (category_id, vendor)
+    UNIQUE (category_id, vendor)
 );
 
+CREATE TRIGGER update_safety_stock_config_updated
+    BEFORE UPDATE ON safety_stock_config
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
+CREATE INDEX idx_ss_metrics ON safety_stock_config(category_id, vendor);
+
 -- Turnover rate configurations
-CREATE TABLE IF NOT EXISTS turnover_config (
-    id INT NOT NULL,
+CREATE TABLE turnover_config (
+    id INTEGER NOT NULL,
     category_id BIGINT,  -- NULL means default/global threshold
     vendor VARCHAR(100), -- NULL means applies to all vendors
-    calculation_period_days INT NOT NULL DEFAULT 30,
+    calculation_period_days INTEGER NOT NULL DEFAULT 30,
     target_rate DECIMAL(10,2) NOT NULL DEFAULT 1.0,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    UNIQUE KEY unique_category_vendor (category_id, vendor)
+    UNIQUE (category_id, vendor)
 );
 
+CREATE TRIGGER update_turnover_config_updated
+    BEFORE UPDATE ON turnover_config
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
 -- Create table for sales seasonality factors
-CREATE TABLE IF NOT EXISTS sales_seasonality (
-    month INT NOT NULL,
+CREATE TABLE sales_seasonality (
+    month INTEGER NOT NULL,
     seasonality_factor DECIMAL(5,3) DEFAULT 0,
-    last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    last_updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (month),
-    CHECK (month BETWEEN 1 AND 12),
-    CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
+    CONSTRAINT month_range CHECK (month BETWEEN 1 AND 12),
+    CONSTRAINT seasonality_range CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
 );
 
--- Insert default global thresholds if not exists
+CREATE TRIGGER update_sales_seasonality_updated
+    BEFORE UPDATE ON sales_seasonality
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column();
+
+-- Insert default global thresholds
 INSERT INTO stock_thresholds (id, category_id, vendor, critical_days, reorder_days, overstock_days)
 VALUES (1, NULL, NULL, 7, 14, 90)
-ON DUPLICATE KEY UPDATE
-    critical_days = VALUES(critical_days),
-    reorder_days = VALUES(reorder_days),
-    overstock_days = VALUES(overstock_days);
+ON CONFLICT (id) DO UPDATE SET
+    critical_days = EXCLUDED.critical_days,
+    reorder_days = EXCLUDED.reorder_days,
+    overstock_days = EXCLUDED.overstock_days;
 
 INSERT INTO lead_time_thresholds (id, category_id, vendor, target_days, warning_days, critical_days)
 VALUES (1, NULL, NULL, 14, 21, 30)
-ON DUPLICATE KEY UPDATE
-    target_days = VALUES(target_days),
-    warning_days = VALUES(warning_days),
-    critical_days = VALUES(critical_days);
+ON CONFLICT (id) DO UPDATE SET
+    target_days = EXCLUDED.target_days,
+    warning_days = EXCLUDED.warning_days,
+    critical_days = EXCLUDED.critical_days;
 
 INSERT INTO sales_velocity_config (id, category_id, vendor, daily_window_days, weekly_window_days, monthly_window_days)
 VALUES (1, NULL, NULL, 30, 7, 90)
-ON DUPLICATE KEY UPDATE
-    daily_window_days = VALUES(daily_window_days),
-    weekly_window_days = VALUES(weekly_window_days),
-    monthly_window_days = VALUES(monthly_window_days);
+ON CONFLICT (id) DO UPDATE SET
+    daily_window_days = EXCLUDED.daily_window_days,
+    weekly_window_days = EXCLUDED.weekly_window_days,
+    monthly_window_days = EXCLUDED.monthly_window_days;
 
 INSERT INTO abc_classification_config (id, a_threshold, b_threshold, classification_period_days)
 VALUES (1, 20.0, 50.0, 90)
-ON DUPLICATE KEY UPDATE
-    a_threshold = VALUES(a_threshold),
-    b_threshold = VALUES(b_threshold),
-    classification_period_days = VALUES(classification_period_days);
+ON CONFLICT (id) DO UPDATE SET
+    a_threshold = EXCLUDED.a_threshold,
+    b_threshold = EXCLUDED.b_threshold,
+    classification_period_days = EXCLUDED.classification_period_days;
 
 INSERT INTO safety_stock_config (id, category_id, vendor, coverage_days, service_level)
 VALUES (1, NULL, NULL, 14, 95.0)
-ON DUPLICATE KEY UPDATE
-    coverage_days = VALUES(coverage_days),
-    service_level = VALUES(service_level);
+ON CONFLICT (id) DO UPDATE SET
+    coverage_days = EXCLUDED.coverage_days,
+    service_level = EXCLUDED.service_level;
 
 INSERT INTO turnover_config (id, category_id, vendor, calculation_period_days, target_rate)
 VALUES (1, NULL, NULL, 30, 1.0)
-ON DUPLICATE KEY UPDATE
-    calculation_period_days = VALUES(calculation_period_days),
-    target_rate = VALUES(target_rate);
+ON CONFLICT (id) DO UPDATE SET
+    calculation_period_days = EXCLUDED.calculation_period_days,
+    target_rate = EXCLUDED.target_rate;
 
 -- Insert default seasonality factors (neutral)
 INSERT INTO sales_seasonality (month, seasonality_factor)
 VALUES 
     (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
     (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
-ON DUPLICATE KEY UPDATE last_updated = CURRENT_TIMESTAMP;
+ON CONFLICT (month) DO UPDATE SET
+    last_updated = CURRENT_TIMESTAMP;
 
 -- View to show thresholds with category names
 CREATE OR REPLACE VIEW stock_thresholds_view AS
@@ -153,9 +210,9 @@ SELECT
     c.name as category_name,
     CASE 
         WHEN st.category_id IS NULL AND st.vendor IS NULL THEN 'Global Default'
-        WHEN st.category_id IS NULL THEN CONCAT('Vendor: ', st.vendor)
-        WHEN st.vendor IS NULL THEN CONCAT('Category: ', c.name)
-        ELSE CONCAT('Category: ', c.name, ' / Vendor: ', st.vendor)
+        WHEN st.category_id IS NULL THEN 'Vendor: ' || st.vendor
+        WHEN st.vendor IS NULL THEN 'Category: ' || c.name
+        ELSE 'Category: ' || c.name || ' / Vendor: ' || st.vendor
     END as threshold_scope
 FROM 
     stock_thresholds st
@@ -171,26 +228,51 @@ ORDER BY
     c.name,
     st.vendor;
 
+-- History and status tables
+CREATE TABLE IF NOT EXISTS calculate_history (
+    id BIGSERIAL PRIMARY KEY,
+    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    end_time TIMESTAMP WITH TIME ZONE NULL,
+    duration_seconds INTEGER,
+    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
+    total_products INTEGER DEFAULT 0,
+    total_orders INTEGER DEFAULT 0,
+    total_purchase_orders INTEGER DEFAULT 0,
+    processed_products INTEGER DEFAULT 0,
+    processed_orders INTEGER DEFAULT 0,
+    processed_purchase_orders INTEGER DEFAULT 0,
+    status calculation_status DEFAULT 'running',
+    error_message TEXT,
+    additional_info JSONB
+);
+
+CREATE TABLE IF NOT EXISTS calculate_status (
+    module_name module_name PRIMARY KEY,
+    last_calculation_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP
+);
+
 CREATE TABLE IF NOT EXISTS sync_status (
     table_name VARCHAR(50) PRIMARY KEY,
-    last_sync_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    last_sync_id BIGINT,
-    INDEX idx_last_sync (last_sync_timestamp)
+    last_sync_timestamp TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    last_sync_id BIGINT
 );
 
 CREATE TABLE IF NOT EXISTS import_history (
-    id BIGINT AUTO_INCREMENT PRIMARY KEY,
+    id BIGSERIAL PRIMARY KEY,
     table_name VARCHAR(50) NOT NULL,
-    start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    end_time TIMESTAMP NULL,
-    duration_seconds INT,
-    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds / 60.0) STORED,
-    records_added INT DEFAULT 0,
-    records_updated INT DEFAULT 0,
+    start_time TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    end_time TIMESTAMP WITH TIME ZONE NULL,
+    duration_seconds INTEGER,
+    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds::decimal / 60.0) STORED,
+    records_added INTEGER DEFAULT 0,
+    records_updated INTEGER DEFAULT 0,
     is_incremental BOOLEAN DEFAULT FALSE,
-    status ENUM('running', 'completed', 'failed', 'cancelled') DEFAULT 'running',
+    status calculation_status DEFAULT 'running',
     error_message TEXT,
-    additional_info JSON,
-    INDEX idx_table_time (table_name, start_time),
-    INDEX idx_status (status)
-); 
+    additional_info JSONB
+);
+
+-- Create all indexes after tables are fully created
+CREATE INDEX IF NOT EXISTS idx_last_calc ON calculate_status(last_calculation_timestamp);
+CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status(last_sync_timestamp);
+CREATE INDEX IF NOT EXISTS idx_table_time ON import_history(table_name, start_time); 

inventory-server/db/metrics-schema.sql

@@ -1,8 +1,8 @@
 -- Disable foreign key checks
-SET FOREIGN_KEY_CHECKS = 0;
+SET session_replication_role = 'replica';
 
 -- Temporary tables for batch metrics processing
-CREATE TABLE IF NOT EXISTS temp_sales_metrics (
+CREATE TABLE temp_sales_metrics (
     pid BIGINT NOT NULL,
     daily_sales_avg DECIMAL(10,3),
     weekly_sales_avg DECIMAL(10,3),
@@ -14,9 +14,9 @@ CREATE TABLE IF NOT EXISTS temp_sales_metrics (
     PRIMARY KEY (pid)
 );
 
-CREATE TABLE IF NOT EXISTS temp_purchase_metrics (
+CREATE TABLE temp_purchase_metrics (
     pid BIGINT NOT NULL,
-    avg_lead_time_days INT,
+    avg_lead_time_days INTEGER,
     last_purchase_date DATE,
     first_received_date DATE,
     last_received_date DATE,
@@ -24,7 +24,7 @@ CREATE TABLE IF NOT EXISTS temp_purchase_metrics (
 );
 
 -- New table for product metrics
-CREATE TABLE IF NOT EXISTS product_metrics (
+CREATE TABLE product_metrics (
     pid BIGINT NOT NULL,
     last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     -- Sales velocity metrics
@@ -32,16 +32,16 @@ CREATE TABLE IF NOT EXISTS product_metrics (
     weekly_sales_avg DECIMAL(10,3),
     monthly_sales_avg DECIMAL(10,3),
     avg_quantity_per_order DECIMAL(10,3),
-    number_of_orders INT,
+    number_of_orders INTEGER,
     first_sale_date DATE,
     last_sale_date DATE,
     -- Stock metrics
-    days_of_inventory INT,
-    weeks_of_inventory INT,
-    reorder_point INT,
-    safety_stock INT,
-    reorder_qty INT DEFAULT 0,
-    overstocked_amt INT DEFAULT 0,
+    days_of_inventory INTEGER,
+    weeks_of_inventory INTEGER,
+    reorder_point INTEGER,
+    safety_stock INTEGER,
+    reorder_qty INTEGER DEFAULT 0,
+    overstocked_amt INTEGER DEFAULT 0,
     -- Financial metrics
     avg_margin_percent DECIMAL(10,3),
     total_revenue DECIMAL(10,3),
@@ -50,7 +50,7 @@ CREATE TABLE IF NOT EXISTS product_metrics (
     gross_profit DECIMAL(10,3),
     gmroi DECIMAL(10,3),
     -- Purchase metrics
-    avg_lead_time_days INT,
+    avg_lead_time_days INTEGER,
     last_purchase_date DATE,
     first_received_date DATE,
     last_received_date DATE,
@@ -60,97 +60,99 @@ CREATE TABLE IF NOT EXISTS product_metrics (
     -- Turnover metrics
     turnover_rate DECIMAL(12,3),
     -- Lead time metrics
-    current_lead_time INT,
-    target_lead_time INT,
+    current_lead_time INTEGER,
+    target_lead_time INTEGER,
     lead_time_status VARCHAR(20),
     -- Forecast metrics
     forecast_accuracy DECIMAL(5,2) DEFAULT NULL,
     forecast_bias DECIMAL(5,2) DEFAULT NULL,
     last_forecast_date DATE DEFAULT NULL,
     PRIMARY KEY (pid),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    INDEX idx_metrics_revenue (total_revenue),
-    INDEX idx_metrics_stock_status (stock_status),
-    INDEX idx_metrics_lead_time (lead_time_status),
-    INDEX idx_metrics_turnover (turnover_rate),
-    INDEX idx_metrics_last_calculated (last_calculated_at),
-    INDEX idx_metrics_abc (abc_class),
-    INDEX idx_metrics_sales (daily_sales_avg, weekly_sales_avg, monthly_sales_avg),
-    INDEX idx_metrics_forecast (forecast_accuracy, forecast_bias)
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_metrics_revenue ON product_metrics(total_revenue);
+CREATE INDEX idx_metrics_stock_status ON product_metrics(stock_status);
+CREATE INDEX idx_metrics_lead_time ON product_metrics(lead_time_status);
+CREATE INDEX idx_metrics_turnover ON product_metrics(turnover_rate);
+CREATE INDEX idx_metrics_last_calculated ON product_metrics(last_calculated_at);
+CREATE INDEX idx_metrics_abc ON product_metrics(abc_class);
+CREATE INDEX idx_metrics_sales ON product_metrics(daily_sales_avg, weekly_sales_avg, monthly_sales_avg);
+CREATE INDEX idx_metrics_forecast ON product_metrics(forecast_accuracy, forecast_bias);
+
 -- New table for time-based aggregates
-CREATE TABLE IF NOT EXISTS product_time_aggregates (
+CREATE TABLE product_time_aggregates (
     pid BIGINT NOT NULL,
-    year INT NOT NULL,
-    month INT NOT NULL,
+    year INTEGER NOT NULL,
+    month INTEGER NOT NULL,
     -- Sales metrics
-    total_quantity_sold INT DEFAULT 0,
+    total_quantity_sold INTEGER DEFAULT 0,
     total_revenue DECIMAL(10,3) DEFAULT 0,
     total_cost DECIMAL(10,3) DEFAULT 0,
-    order_count INT DEFAULT 0,
+    order_count INTEGER DEFAULT 0,
     -- Stock changes
-    stock_received INT DEFAULT 0,
-    stock_ordered INT DEFAULT 0,
+    stock_received INTEGER DEFAULT 0,
+    stock_ordered INTEGER DEFAULT 0,
     -- Calculated fields
     avg_price DECIMAL(10,3),
     profit_margin DECIMAL(10,3),
     inventory_value DECIMAL(10,3),
     gmroi DECIMAL(10,3),
     PRIMARY KEY (pid, year, month),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    INDEX idx_date (year, month)
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
 );
 
--- Create vendor details table
-CREATE TABLE IF NOT EXISTS vendor_details (
-    vendor VARCHAR(100) NOT NULL,
+CREATE INDEX idx_date ON product_time_aggregates(year, month);
+
+-- Create vendor_details table
+CREATE TABLE vendor_details (
+    vendor VARCHAR(100) PRIMARY KEY,
     contact_name VARCHAR(100),
-    email VARCHAR(100),
-    phone VARCHAR(20),
+    email VARCHAR(255),
+    phone VARCHAR(50),
     status VARCHAR(20) DEFAULT 'active',
-    notes TEXT,
-    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-    PRIMARY KEY (vendor),
-    INDEX idx_vendor_status (status)
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
 );
 
+CREATE INDEX idx_vendor_details_status ON vendor_details(status);
+
 -- New table for vendor metrics
-CREATE TABLE IF NOT EXISTS vendor_metrics (
+CREATE TABLE vendor_metrics (
     vendor VARCHAR(100) NOT NULL,
     last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     -- Performance metrics
     avg_lead_time_days DECIMAL(10,3),
     on_time_delivery_rate DECIMAL(5,2),
     order_fill_rate DECIMAL(5,2),
-    total_orders INT DEFAULT 0,
-    total_late_orders INT DEFAULT 0,
+    total_orders INTEGER DEFAULT 0,
+    total_late_orders INTEGER DEFAULT 0,
     total_purchase_value DECIMAL(10,3) DEFAULT 0,
     avg_order_value DECIMAL(10,3),
     -- Product metrics
-    active_products INT DEFAULT 0,
-    total_products INT DEFAULT 0,
+    active_products INTEGER DEFAULT 0,
+    total_products INTEGER DEFAULT 0,
     -- Financial metrics
     total_revenue DECIMAL(10,3) DEFAULT 0,
     avg_margin_percent DECIMAL(5,2),
     -- Status
     status VARCHAR(20) DEFAULT 'active',
     PRIMARY KEY (vendor),
-    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE,
-    INDEX idx_vendor_performance (on_time_delivery_rate),
-    INDEX idx_vendor_status (status),
-    INDEX idx_metrics_last_calculated (last_calculated_at),
-    INDEX idx_vendor_metrics_orders (total_orders, total_late_orders)
+    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_vendor_performance ON vendor_metrics(on_time_delivery_rate);
+CREATE INDEX idx_vendor_status ON vendor_metrics(status);
+CREATE INDEX idx_vendor_metrics_last_calculated ON vendor_metrics(last_calculated_at);
+CREATE INDEX idx_vendor_metrics_orders ON vendor_metrics(total_orders, total_late_orders);
+
 -- New table for category metrics
-CREATE TABLE IF NOT EXISTS category_metrics (
+CREATE TABLE category_metrics (
     category_id BIGINT NOT NULL,
     last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     -- Product metrics
-    product_count INT DEFAULT 0,
-    active_products INT DEFAULT 0,
+    product_count INTEGER DEFAULT 0,
+    active_products INTEGER DEFAULT 0,
     -- Financial metrics
     total_value DECIMAL(15,3) DEFAULT 0,
     avg_margin DECIMAL(5,2),
@@ -159,272 +161,215 @@ CREATE TABLE IF NOT EXISTS category_metrics (
     -- Status
     status VARCHAR(20) DEFAULT 'active',
     PRIMARY KEY (category_id),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    INDEX idx_category_status (status),
-    INDEX idx_category_growth (growth_rate),
-    INDEX idx_metrics_last_calculated (last_calculated_at),
-    INDEX idx_category_metrics_products (product_count, active_products)
+    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_category_status ON category_metrics(status);
+CREATE INDEX idx_category_growth ON category_metrics(growth_rate);
+CREATE INDEX idx_metrics_last_calculated_cat ON category_metrics(last_calculated_at);
+CREATE INDEX idx_category_metrics_products ON category_metrics(product_count, active_products);
+
 -- New table for vendor time-based metrics
-CREATE TABLE IF NOT EXISTS vendor_time_metrics (
+CREATE TABLE vendor_time_metrics (
     vendor VARCHAR(100) NOT NULL,
-    year INT NOT NULL,
-    month INT NOT NULL,
+    year INTEGER NOT NULL,
+    month INTEGER NOT NULL,
     -- Order metrics
-    total_orders INT DEFAULT 0,
-    late_orders INT DEFAULT 0,
+    total_orders INTEGER DEFAULT 0,
+    late_orders INTEGER DEFAULT 0,
     avg_lead_time_days DECIMAL(10,3),
     -- Financial metrics
     total_purchase_value DECIMAL(10,3) DEFAULT 0,
     total_revenue DECIMAL(10,3) DEFAULT 0,
     avg_margin_percent DECIMAL(5,2),
     PRIMARY KEY (vendor, year, month),
-    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE,
-    INDEX idx_vendor_date (year, month)
+    FOREIGN KEY (vendor) REFERENCES vendor_details(vendor) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_vendor_date ON vendor_time_metrics(year, month);
+
 -- New table for category time-based metrics
-CREATE TABLE IF NOT EXISTS category_time_metrics (
+CREATE TABLE category_time_metrics (
     category_id BIGINT NOT NULL,
-    year INT NOT NULL,
-    month INT NOT NULL,
+    year INTEGER NOT NULL,
+    month INTEGER NOT NULL,
     -- Product metrics
-    product_count INT DEFAULT 0,
-    active_products INT DEFAULT 0,
+    product_count INTEGER DEFAULT 0,
+    active_products INTEGER DEFAULT 0,
     -- Financial metrics
     total_value DECIMAL(15,3) DEFAULT 0,
     total_revenue DECIMAL(15,3) DEFAULT 0,
     avg_margin DECIMAL(5,2),
     turnover_rate DECIMAL(12,3),
     PRIMARY KEY (category_id, year, month),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    INDEX idx_category_date (year, month)
+    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_category_date ON category_time_metrics(year, month);
+
 -- New table for category-based sales metrics
-CREATE TABLE IF NOT EXISTS category_sales_metrics (
+CREATE TABLE category_sales_metrics (
     category_id BIGINT NOT NULL,
     brand VARCHAR(100) NOT NULL,
     period_start DATE NOT NULL,
     period_end DATE NOT NULL,
     avg_daily_sales DECIMAL(10,3) DEFAULT 0,
-    total_sold INT DEFAULT 0,
-    num_products INT DEFAULT 0,
+    total_sold INTEGER DEFAULT 0,
+    num_products INTEGER DEFAULT 0,
     avg_price DECIMAL(10,3) DEFAULT 0,
     last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (category_id, brand, period_start, period_end),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    INDEX idx_category_brand (category_id, brand),
-    INDEX idx_period (period_start, period_end)
+    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_category_brand ON category_sales_metrics(category_id, brand);
+CREATE INDEX idx_period ON category_sales_metrics(period_start, period_end);
+
 -- New table for brand metrics
-CREATE TABLE IF NOT EXISTS brand_metrics (
+CREATE TABLE brand_metrics (
     brand VARCHAR(100) NOT NULL,
     last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     -- Product metrics
-    product_count INT DEFAULT 0,
-    active_products INT DEFAULT 0,
+    product_count INTEGER DEFAULT 0,
+    active_products INTEGER DEFAULT 0,
     -- Stock metrics
-    total_stock_units INT DEFAULT 0,
+    total_stock_units INTEGER DEFAULT 0,
     total_stock_cost DECIMAL(15,2) DEFAULT 0,
     total_stock_retail DECIMAL(15,2) DEFAULT 0,
     -- Sales metrics
     total_revenue DECIMAL(15,2) DEFAULT 0,
     avg_margin DECIMAL(5,2) DEFAULT 0,
     growth_rate DECIMAL(5,2) DEFAULT 0,
-    PRIMARY KEY (brand),
-    INDEX idx_brand_metrics_last_calculated (last_calculated_at),
-    INDEX idx_brand_metrics_revenue (total_revenue),
-    INDEX idx_brand_metrics_growth (growth_rate)
+    PRIMARY KEY (brand)
 );
 
+CREATE INDEX idx_brand_metrics_last_calculated ON brand_metrics(last_calculated_at);
+CREATE INDEX idx_brand_metrics_revenue ON brand_metrics(total_revenue);
+CREATE INDEX idx_brand_metrics_growth ON brand_metrics(growth_rate);
+
 -- New table for brand time-based metrics
-CREATE TABLE IF NOT EXISTS brand_time_metrics (
+CREATE TABLE brand_time_metrics (
     brand VARCHAR(100) NOT NULL,
-    year INT NOT NULL,
-    month INT NOT NULL,
+    year INTEGER NOT NULL,
+    month INTEGER NOT NULL,
     -- Product metrics
-    product_count INT DEFAULT 0,
-    active_products INT DEFAULT 0,
+    product_count INTEGER DEFAULT 0,
+    active_products INTEGER DEFAULT 0,
     -- Stock metrics
-    total_stock_units INT DEFAULT 0,
+    total_stock_units INTEGER DEFAULT 0,
     total_stock_cost DECIMAL(15,2) DEFAULT 0,
     total_stock_retail DECIMAL(15,2) DEFAULT 0,
     -- Sales metrics
     total_revenue DECIMAL(15,2) DEFAULT 0,
     avg_margin DECIMAL(5,2) DEFAULT 0,
-    PRIMARY KEY (brand, year, month),
-    INDEX idx_brand_date (year, month)
+    growth_rate DECIMAL(5,2) DEFAULT 0,
+    PRIMARY KEY (brand, year, month)
 );
 
+CREATE INDEX idx_brand_time_date ON brand_time_metrics(year, month);
+
 -- New table for sales forecasts
-CREATE TABLE IF NOT EXISTS sales_forecasts (
+CREATE TABLE sales_forecasts (
     pid BIGINT NOT NULL,
     forecast_date DATE NOT NULL,
-    forecast_units DECIMAL(10,2) DEFAULT 0,
-    forecast_revenue DECIMAL(10,2) DEFAULT 0,
-    confidence_level DECIMAL(5,2) DEFAULT 0,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    forecast_quantity INTEGER,
+    confidence_level DECIMAL(5,2),
+    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (pid, forecast_date),
-    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    INDEX idx_forecast_date (forecast_date),
-    INDEX idx_forecast_last_calculated (last_calculated_at)
+    FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE
 );
 
+CREATE INDEX idx_forecast_date ON sales_forecasts(forecast_date);
+
 -- New table for category forecasts
-CREATE TABLE IF NOT EXISTS category_forecasts (
+CREATE TABLE category_forecasts (
     category_id BIGINT NOT NULL,
     forecast_date DATE NOT NULL,
-    forecast_units DECIMAL(10,2) DEFAULT 0,
-    forecast_revenue DECIMAL(10,2) DEFAULT 0,
-    confidence_level DECIMAL(5,2) DEFAULT 0,
-    last_calculated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    forecast_revenue DECIMAL(15,2),
+    forecast_units INTEGER,
+    confidence_level DECIMAL(5,2),
+    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
     PRIMARY KEY (category_id, forecast_date),
-    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    INDEX idx_category_forecast_date (forecast_date),
-    INDEX idx_category_forecast_last_calculated (last_calculated_at)
+    FOREIGN KEY (category_id) REFERENCES categories(cat_id) ON DELETE CASCADE
 );
 
--- Create view for inventory health
+CREATE INDEX idx_cat_forecast_date ON category_forecasts(forecast_date);
+
+-- Create views for common calculations
 CREATE OR REPLACE VIEW inventory_health AS
-WITH product_thresholds AS (
+WITH stock_levels AS (
     SELECT 
         p.pid,
-        COALESCE(
-            -- Try category+vendor specific
-            (SELECT critical_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Try category specific
-            (SELECT critical_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor IS NULL LIMIT 1),
-            -- Try vendor specific
-            (SELECT critical_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Fall back to default
-            (SELECT critical_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor IS NULL LIMIT 1),
-            7
-        ) as critical_days,
-        COALESCE(
-            -- Try category+vendor specific
-            (SELECT reorder_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Try category specific
-            (SELECT reorder_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor IS NULL LIMIT 1),
-            -- Try vendor specific
-            (SELECT reorder_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Fall back to default
-            (SELECT reorder_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor IS NULL LIMIT 1),
-            14
-        ) as reorder_days,
-        COALESCE(
-            -- Try category+vendor specific
-            (SELECT overstock_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Try category specific
-            (SELECT overstock_days FROM stock_thresholds st 
-            JOIN product_categories pc ON st.category_id = pc.cat_id
-            WHERE pc.pid = p.pid 
-            AND st.vendor IS NULL LIMIT 1),
-            -- Try vendor specific
-            (SELECT overstock_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor = p.vendor LIMIT 1),
-            -- Fall back to default
-            (SELECT overstock_days FROM stock_thresholds st 
-            WHERE st.category_id IS NULL 
-            AND st.vendor IS NULL LIMIT 1),
-            90
-        ) as overstock_days
+        p.title,
+        p.SKU,
+        p.stock_quantity,
+        p.preorder_count,
+        pm.daily_sales_avg,
+        pm.weekly_sales_avg,
+        pm.monthly_sales_avg,
+        pm.reorder_point,
+        pm.safety_stock,
+        pm.days_of_inventory,
+        pm.weeks_of_inventory,
+        pm.stock_status,
+        pm.abc_class,
+        pm.turnover_rate,
+        pm.avg_lead_time_days,
+        pm.current_lead_time,
+        pm.target_lead_time,
+        pm.lead_time_status,
+        p.cost_price,
+        p.price,
+        pm.inventory_value,
+        pm.gmroi
     FROM products p
+    LEFT JOIN product_metrics pm ON p.pid = pm.pid
+    WHERE p.managing_stock = true AND p.visible = true
 )
 SELECT 
-    p.pid,
-    p.SKU,
-    p.title,
-    p.stock_quantity,
-    COALESCE(pm.daily_sales_avg, 0) as daily_sales_avg,
-    COALESCE(pm.days_of_inventory, 0) as days_of_inventory,
-    COALESCE(pm.reorder_point, 0) as reorder_point,
-    COALESCE(pm.safety_stock, 0) as safety_stock,
+    *,
     CASE 
-        WHEN pm.daily_sales_avg = 0 THEN 'New'
-        WHEN p.stock_quantity <= CEIL(pm.daily_sales_avg * pt.critical_days) THEN 'Critical'
-        WHEN p.stock_quantity <= CEIL(pm.daily_sales_avg * pt.reorder_days) THEN 'Reorder'
-        WHEN p.stock_quantity > (pm.daily_sales_avg * pt.overstock_days) THEN 'Overstocked'
+        WHEN stock_quantity <= safety_stock THEN 'Critical'
+        WHEN stock_quantity <= reorder_point THEN 'Low'
+        WHEN stock_quantity > (reorder_point * 3) THEN 'Excess'
         ELSE 'Healthy'
-    END as stock_status
-FROM 
-    products p
-LEFT JOIN 
-    product_metrics pm ON p.pid = pm.pid
-LEFT JOIN
-    product_thresholds pt ON p.pid = pt.pid
-WHERE 
-    p.managing_stock = true;
+    END as inventory_status,
+    CASE 
+        WHEN lead_time_status = 'delayed' AND stock_status = 'low' THEN 'High'
+        WHEN lead_time_status = 'delayed' OR stock_status = 'low' THEN 'Medium'
+        ELSE 'Low'
+    END as risk_level
+FROM stock_levels;
 
 -- Create view for category performance trends
 CREATE OR REPLACE VIEW category_performance_trends AS
+WITH monthly_trends AS (
+    SELECT 
+        c.cat_id,
+        c.name as category_name,
+        ctm.year,
+        ctm.month,
+        ctm.product_count,
+        ctm.active_products,
+        ctm.total_value,
+        ctm.total_revenue,
+        ctm.avg_margin,
+        ctm.turnover_rate,
+        LAG(ctm.total_revenue) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_revenue,
+        LAG(ctm.turnover_rate) OVER (PARTITION BY c.cat_id ORDER BY ctm.year, ctm.month) as prev_month_turnover
+    FROM categories c
+    JOIN category_time_metrics ctm ON c.cat_id = ctm.category_id
+)
 SELECT 
-    c.cat_id as category_id,
-    c.name,
-    c.description,
-    p.name as parent_name,
-    c.status,
-    cm.product_count,
-    cm.active_products,
-    cm.total_value,
-    cm.avg_margin,
-    cm.turnover_rate,
-    cm.growth_rate,
+    *,
     CASE 
-        WHEN cm.growth_rate >= 20 THEN 'High Growth'
-        WHEN cm.growth_rate >= 5 THEN 'Growing'
-        WHEN cm.growth_rate >= -5 THEN 'Stable'
-        ELSE 'Declining'
-    END as performance_rating
-FROM 
-    categories c
-LEFT JOIN 
-    categories p ON c.parent_id = p.cat_id
-LEFT JOIN 
-    category_metrics cm ON c.cat_id = cm.category_id;
+        WHEN prev_month_revenue IS NULL THEN 0
+        ELSE ((total_revenue - prev_month_revenue) / prev_month_revenue) * 100 
+    END as revenue_growth_percent,
+    CASE 
+        WHEN prev_month_turnover IS NULL THEN 0
+        ELSE ((turnover_rate - prev_month_turnover) / prev_month_turnover) * 100 
+    END as turnover_growth_percent
+FROM monthly_trends;
 
--- Re-enable foreign key checks
-SET FOREIGN_KEY_CHECKS = 1;
-
--- Create table for sales seasonality factors
-CREATE TABLE IF NOT EXISTS sales_seasonality (
-    month INT NOT NULL,
-    seasonality_factor DECIMAL(5,3) DEFAULT 0,
-    last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (month),
-    CHECK (month BETWEEN 1 AND 12),
-    CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
-);
-
--- Insert default seasonality factors (neutral)
-INSERT INTO sales_seasonality (month, seasonality_factor)
-VALUES 
-    (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
-    (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
-ON DUPLICATE KEY UPDATE last_updated = CURRENT_TIMESTAMP; 
+SET session_replication_role = 'origin';

inventory-server/db/schema.sql

@@ -1,6 +1,13 @@
 -- Enable strict error reporting
-SET sql_mode = 'STRICT_ALL_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_ZERO_DATE,NO_ZERO_IN_DATE,NO_ENGINE_SUBSTITUTION';
-SET FOREIGN_KEY_CHECKS = 0;
+SET session_replication_role = 'replica';  -- Disable foreign key checks temporarily
+
+-- Create function for updating timestamps
+CREATE OR REPLACE FUNCTION update_updated_column() RETURNS TRIGGER AS $func$
+BEGIN
+    NEW.updated = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$func$ language plpgsql;
 
 -- Create tables
 CREATE TABLE products (
@@ -8,18 +15,18 @@ CREATE TABLE products (
     title VARCHAR(255) NOT NULL,
     description TEXT,
     SKU VARCHAR(50) NOT NULL,
-    created_at TIMESTAMP NULL,
-    first_received TIMESTAMP NULL,
-    stock_quantity INT DEFAULT 0,
-    preorder_count INT DEFAULT 0,
-    notions_inv_count INT DEFAULT 0,
+    created_at TIMESTAMP WITH TIME ZONE,
+    first_received TIMESTAMP WITH TIME ZONE,
+    stock_quantity INTEGER DEFAULT 0,
+    preorder_count INTEGER DEFAULT 0,
+    notions_inv_count INTEGER DEFAULT 0,
     price DECIMAL(10, 3) NOT NULL,
     regular_price DECIMAL(10, 3) NOT NULL,
     cost_price DECIMAL(10, 3),
     landing_cost_price DECIMAL(10, 3),
     barcode VARCHAR(50),
     harmonized_tariff_code VARCHAR(20),
-    updated_at TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE,
     visible BOOLEAN DEFAULT true,
     managing_stock BOOLEAN DEFAULT true,
     replenishable BOOLEAN DEFAULT true,
@@ -37,57 +44,64 @@ CREATE TABLE products (
     artist VARCHAR(100),
     options TEXT,
     tags TEXT,
-    moq INT DEFAULT 1,
-    uom INT DEFAULT 1,
+    moq INTEGER DEFAULT 1,
+    uom INTEGER DEFAULT 1,
     rating DECIMAL(10,2) DEFAULT 0.00,
-    reviews INT UNSIGNED DEFAULT 0,
+    reviews INTEGER DEFAULT 0,
     weight DECIMAL(10,3),
     length DECIMAL(10,3),
     width DECIMAL(10,3),
     height DECIMAL(10,3),
     country_of_origin VARCHAR(5),
     location VARCHAR(50),
-    total_sold INT UNSIGNED DEFAULT 0,
-    baskets INT UNSIGNED DEFAULT 0,
-    notifies INT UNSIGNED DEFAULT 0,
+    total_sold INTEGER DEFAULT 0,
+    baskets INTEGER DEFAULT 0,
+    notifies INTEGER DEFAULT 0,
     date_last_sold DATE,
-    PRIMARY KEY (pid),
-    INDEX idx_sku (SKU),
-    INDEX idx_vendor (vendor),
-    INDEX idx_brand (brand),
-    INDEX idx_location (location),
-    INDEX idx_total_sold (total_sold),
-    INDEX idx_date_last_sold (date_last_sold)
-) ENGINE=InnoDB;
+    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (pid)
+);
+
+-- Create trigger for products
+CREATE TRIGGER update_products_updated
+    BEFORE UPDATE ON products
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_column();
+
+-- Create indexes for products table
+CREATE INDEX idx_products_sku ON products(SKU);
+CREATE INDEX idx_products_vendor ON products(vendor);
+CREATE INDEX idx_products_brand ON products(brand);
+CREATE INDEX idx_products_location ON products(location);
+CREATE INDEX idx_products_total_sold ON products(total_sold);
+CREATE INDEX idx_products_date_last_sold ON products(date_last_sold);
+CREATE INDEX idx_products_updated ON products(updated);
 
 -- Create categories table with hierarchy support
 CREATE TABLE categories (
     cat_id BIGINT PRIMARY KEY,
     name VARCHAR(100) NOT NULL,
-    type SMALLINT NOT NULL COMMENT '10=section, 11=category, 12=subcategory, 13=subsubcategory, 1=company, 2=line, 3=subline, 40=artist',
+    type SMALLINT NOT NULL,
     parent_id BIGINT,
     description TEXT,
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
     status VARCHAR(20) DEFAULT 'active',
-    FOREIGN KEY (parent_id) REFERENCES categories(cat_id),
-    INDEX idx_parent (parent_id),
-    INDEX idx_type (type),
-    INDEX idx_status (status),
-    INDEX idx_name_type (name, type)
-) ENGINE=InnoDB;
+    FOREIGN KEY (parent_id) REFERENCES categories(cat_id)
+);
 
--- Create vendor_details table
-CREATE TABLE vendor_details (
-    vendor VARCHAR(100) PRIMARY KEY,
-    contact_name VARCHAR(100),
-    email VARCHAR(255),
-    phone VARCHAR(50),
-    status VARCHAR(20) DEFAULT 'active',
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-    INDEX idx_status (status)
-) ENGINE=InnoDB;
+-- Create trigger for categories
+CREATE TRIGGER update_categories_updated_at
+    BEFORE UPDATE ON categories
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_column();
+
+COMMENT ON COLUMN categories.type IS '10=section, 11=category, 12=subcategory, 13=subsubcategory, 1=company, 2=line, 3=subline, 40=artist';
+
+CREATE INDEX idx_categories_parent ON categories(parent_id);
+CREATE INDEX idx_categories_type ON categories(type);
+CREATE INDEX idx_categories_status ON categories(status);
+CREATE INDEX idx_categories_name_type ON categories(name, type);
 
 -- Create product_categories junction table
 CREATE TABLE product_categories (
@@ -95,74 +109,98 @@ CREATE TABLE product_categories (
     pid BIGINT NOT NULL,
     PRIMARY KEY (pid, cat_id),
     FOREIGN KEY (pid) REFERENCES products(pid) ON DELETE CASCADE,
-    FOREIGN KEY (cat_id) REFERENCES categories(cat_id) ON DELETE CASCADE,
-    INDEX idx_category (cat_id),
-    INDEX idx_product (pid)
-) ENGINE=InnoDB;
+    FOREIGN KEY (cat_id) REFERENCES categories(cat_id) ON DELETE CASCADE
+);
+
+CREATE INDEX idx_product_categories_category ON product_categories(cat_id);
+CREATE INDEX idx_product_categories_product ON product_categories(pid);
 
 -- Create orders table with its indexes
-CREATE TABLE IF NOT EXISTS orders (
-    id BIGINT NOT NULL AUTO_INCREMENT,
+CREATE TABLE orders (
+    id BIGSERIAL PRIMARY KEY,
     order_number VARCHAR(50) NOT NULL,
     pid BIGINT NOT NULL,
     SKU VARCHAR(50) NOT NULL,
     date DATE NOT NULL,
     price DECIMAL(10,3) NOT NULL,
-    quantity INT NOT NULL,
+    quantity INTEGER NOT NULL,
     discount DECIMAL(10,3) DEFAULT 0.000,
     tax DECIMAL(10,3) DEFAULT 0.000,
-    tax_included TINYINT(1) DEFAULT 0,
+    tax_included BOOLEAN DEFAULT false,
     shipping DECIMAL(10,3) DEFAULT 0.000,
     costeach DECIMAL(10,3) DEFAULT 0.000,
     customer VARCHAR(50) NOT NULL,
     customer_name VARCHAR(100),
     status VARCHAR(20) DEFAULT 'pending',
-    canceled TINYINT(1) DEFAULT 0,
-    PRIMARY KEY (id),
-    UNIQUE KEY unique_order_line (order_number, pid),
-    KEY order_number (order_number),
-    KEY pid (pid),
-    KEY customer (customer),
-    KEY date (date),
-    KEY status (status),
-    INDEX idx_orders_metrics (pid, date, canceled)
-) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
+    canceled BOOLEAN DEFAULT false,
+    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    UNIQUE (order_number, pid)
+);
+
+-- Create trigger for orders
+CREATE TRIGGER update_orders_updated
+    BEFORE UPDATE ON orders
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_column();
+
+CREATE INDEX idx_orders_number ON orders(order_number);
+CREATE INDEX idx_orders_pid ON orders(pid);
+CREATE INDEX idx_orders_customer ON orders(customer);
+CREATE INDEX idx_orders_date ON orders(date);
+CREATE INDEX idx_orders_status ON orders(status);
+CREATE INDEX idx_orders_metrics ON orders(pid, date, canceled);
+CREATE INDEX idx_orders_updated ON orders(updated);
 
 -- Create purchase_orders table with its indexes
 CREATE TABLE purchase_orders (
-    id BIGINT AUTO_INCREMENT PRIMARY KEY,
+    id BIGSERIAL PRIMARY KEY,
     po_id VARCHAR(50) NOT NULL,
     vendor VARCHAR(100) NOT NULL,
     date DATE NOT NULL,
     expected_date DATE,
     pid BIGINT NOT NULL,
     sku VARCHAR(50) NOT NULL,
-    name VARCHAR(100) NOT NULL COMMENT 'Product name from products.description',
+    name VARCHAR(100) NOT NULL,
     cost_price DECIMAL(10, 3) NOT NULL,
-    po_cost_price DECIMAL(10, 3) NOT NULL COMMENT 'Original cost from PO, before receiving adjustments',
-    status TINYINT UNSIGNED DEFAULT 1 COMMENT '0=canceled,1=created,10=electronically_ready_send,11=ordered,12=preordered,13=electronically_sent,15=receiving_started,50=done',
-    receiving_status TINYINT UNSIGNED DEFAULT 1 COMMENT '0=canceled,1=created,30=partial_received,40=full_received,50=paid',
+    po_cost_price DECIMAL(10, 3) NOT NULL,
+    status SMALLINT DEFAULT 1,
+    receiving_status SMALLINT DEFAULT 1,
     notes TEXT,
     long_note TEXT,
-    ordered INT NOT NULL,
-    received INT DEFAULT 0,
-    received_date DATE COMMENT 'Date of first receiving',
-    last_received_date DATE COMMENT 'Date of most recent receiving',
-    received_by INT,
-    receiving_history JSON COMMENT 'Array of receiving records with qty, date, cost, receiving_id, and alt_po flag',
+    ordered INTEGER NOT NULL,
+    received INTEGER DEFAULT 0,
+    received_date DATE,
+    last_received_date DATE,
+    received_by VARCHAR(100),
+    receiving_history JSONB,
+    updated TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT CURRENT_TIMESTAMP,
     FOREIGN KEY (pid) REFERENCES products(pid),
-    INDEX idx_po_id (po_id),
-    INDEX idx_vendor (vendor),
-    INDEX idx_status (status),
-    INDEX idx_receiving_status (receiving_status),
-    INDEX idx_purchase_orders_metrics (pid, date, status, ordered, received),
-    INDEX idx_po_metrics (pid, date, receiving_status, received_date),
-    INDEX idx_po_product_date (pid, date),
-    INDEX idx_po_product_status (pid, status),
-    UNIQUE KEY unique_po_product (po_id, pid)
-) ENGINE=InnoDB;
+    UNIQUE (po_id, pid)
+);
 
-SET FOREIGN_KEY_CHECKS = 1;
+-- Create trigger for purchase_orders
+CREATE TRIGGER update_purchase_orders_updated
+    BEFORE UPDATE ON purchase_orders
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_column();
+
+COMMENT ON COLUMN purchase_orders.name IS 'Product name from products.description';
+COMMENT ON COLUMN purchase_orders.po_cost_price IS 'Original cost from PO, before receiving adjustments';
+COMMENT ON COLUMN purchase_orders.status IS '0=canceled,1=created,10=electronically_ready_send,11=ordered,12=preordered,13=electronically_sent,15=receiving_started,50=done';
+COMMENT ON COLUMN purchase_orders.receiving_status IS '0=canceled,1=created,30=partial_received,40=full_received,50=paid';
+COMMENT ON COLUMN purchase_orders.receiving_history IS 'Array of receiving records with qty, date, cost, receiving_id, and alt_po flag';
+
+CREATE INDEX idx_po_id ON purchase_orders(po_id);
+CREATE INDEX idx_po_vendor ON purchase_orders(vendor);
+CREATE INDEX idx_po_status ON purchase_orders(status);
+CREATE INDEX idx_po_receiving_status ON purchase_orders(receiving_status);
+CREATE INDEX idx_po_metrics ON purchase_orders(pid, date, status, ordered, received);
+CREATE INDEX idx_po_metrics_receiving ON purchase_orders(pid, date, receiving_status, received_date);
+CREATE INDEX idx_po_product_date ON purchase_orders(pid, date);
+CREATE INDEX idx_po_product_status ON purchase_orders(pid, status);
+CREATE INDEX idx_po_updated ON purchase_orders(updated);
+
+SET session_replication_role = 'origin';  -- Re-enable foreign key checks
 
 -- Create views for common calculations
 -- product_sales_trends view moved to metrics-schema.sql 

inventory-server/db/templates.sql

@@ -0,0 +1,39 @@
+-- Templates table for storing import templates
+CREATE TABLE IF NOT EXISTS templates (
+    id SERIAL PRIMARY KEY,
+    company TEXT NOT NULL,
+    product_type TEXT NOT NULL,
+    supplier TEXT,
+    msrp DECIMAL(10,2),
+    cost_each DECIMAL(10,2),
+    qty_per_unit INTEGER,
+    case_qty INTEGER,
+    hts_code TEXT,
+    description TEXT,
+    weight DECIMAL(10,2),
+    length DECIMAL(10,2),
+    width DECIMAL(10,2),
+    height DECIMAL(10,2),
+    tax_cat TEXT,
+    size_cat TEXT,
+    categories TEXT[],
+    ship_restrictions TEXT[],
+    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
+    UNIQUE(company, product_type)
+);
+
+-- Function to update the updated_at timestamp
+CREATE OR REPLACE FUNCTION update_updated_at_column()
+RETURNS TRIGGER AS $$
+BEGIN
+    NEW.updated_at = CURRENT_TIMESTAMP;
+    RETURN NEW;
+END;
+$$ language 'plpgsql';
+
+-- Trigger to automatically update the updated_at column
+CREATE TRIGGER update_templates_updated_at
+    BEFORE UPDATE ON templates
+    FOR EACH ROW
+    EXECUTE FUNCTION update_updated_at_column(); 

inventory-server/package.json

@@ -18,12 +18,15 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
+    "bcrypt": "^5.1.1",
     "cors": "^2.8.5",
     "csv-parse": "^5.6.0",
     "dotenv": "^16.4.7",
     "express": "^4.18.2",
     "multer": "^1.4.5-lts.1",
     "mysql2": "^3.12.0",
+    "openai": "^4.85.3",
+    "pg": "^8.13.3",
     "pm2": "^5.3.0",
     "ssh2": "^1.16.0",
     "uuid": "^9.0.1"

inventory-server/scripts/calculate-metrics.js

@@ -7,13 +7,13 @@ require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
 
 // Configuration flags for controlling which metrics to calculate
 // Set to 1 to skip the corresponding calculation, 0 to run it
-const SKIP_PRODUCT_METRICS = 1;          // Skip all product metrics
-const SKIP_TIME_AGGREGATES = 1;          // Skip time aggregates
-const SKIP_FINANCIAL_METRICS = 1;        // Skip financial metrics
-const SKIP_VENDOR_METRICS = 1;           // Skip vendor metrics
-const SKIP_CATEGORY_METRICS = 1;         // Skip category metrics
-const SKIP_BRAND_METRICS = 1;            // Skip brand metrics
-const SKIP_SALES_FORECASTS = 1;          // Skip sales forecasts
+const SKIP_PRODUCT_METRICS = 0;
+const SKIP_TIME_AGGREGATES = 0;
+const SKIP_FINANCIAL_METRICS = 0;
+const SKIP_VENDOR_METRICS = 0;
+const SKIP_CATEGORY_METRICS = 0;
+const SKIP_BRAND_METRICS = 0;
+const SKIP_SALES_FORECASTS = 0;
 
 // Add error handler for uncaught exceptions
 process.on('uncaughtException', (error) => {
@@ -44,6 +44,34 @@ global.clearProgress = progress.clearProgress;
 global.getProgress = progress.getProgress;
 global.logError = progress.logError;
 
+// List of temporary tables used in the calculation process
+const TEMP_TABLES = [
+    'temp_revenue_ranks',
+    'temp_sales_metrics',
+    'temp_purchase_metrics',
+    'temp_product_metrics',
+    'temp_vendor_metrics',
+    'temp_category_metrics',
+    'temp_brand_metrics',
+    'temp_forecast_dates',
+    'temp_daily_sales',
+    'temp_product_stats',
+    'temp_category_sales',
+    'temp_category_stats'
+];
+
+// Add cleanup function for temporary tables
+async function cleanupTemporaryTables(connection) {
+    try {
+        for (const table of TEMP_TABLES) {
+            await connection.query(`DROP TEMPORARY TABLE IF EXISTS ${table}`);
+        }
+    } catch (error) {
+        logError(error, 'Error cleaning up temporary tables');
+        throw error; // Re-throw to be handled by the caller
+    }
+}
+
 const { getConnection, closePool } = require('./metrics/utils/db');
 const calculateProductMetrics = require('./metrics/product-metrics');
 const calculateTimeAggregates = require('./metrics/time-aggregates');
@@ -83,10 +111,78 @@ process.on('SIGTERM', cancelCalculation);
 async function calculateMetrics() {
     let connection;
     const startTime = Date.now();
-    let processedCount = 0;
+    let processedProducts = 0;
+    let processedOrders = 0;
+    let processedPurchaseOrders = 0;
     let totalProducts = 0;
+    let totalOrders = 0;
+    let totalPurchaseOrders = 0;
+    let calculateHistoryId;
     
     try {
+        // Clean up any previously running calculations
+        connection = await getConnection();
+        await connection.query(`
+            UPDATE calculate_history 
+            SET 
+                status = 'cancelled',
+                end_time = NOW(),
+                duration_seconds = TIMESTAMPDIFF(SECOND, start_time, NOW()),
+                error_message = 'Previous calculation was not completed properly'
+            WHERE status = 'running'
+        `);
+
+        // Get counts from all relevant tables
+        const [[productCount], [orderCount], [poCount]] = await Promise.all([
+            connection.query('SELECT COUNT(*) as total FROM products'),
+            connection.query('SELECT COUNT(*) as total FROM orders'),
+            connection.query('SELECT COUNT(*) as total FROM purchase_orders')
+        ]);
+
+        totalProducts = productCount.total;
+        totalOrders = orderCount.total;
+        totalPurchaseOrders = poCount.total;
+
+        // Create history record for this calculation
+        const [historyResult] = await connection.query(`
+            INSERT INTO calculate_history (
+                start_time,
+                status,
+                total_products,
+                total_orders,
+                total_purchase_orders,
+                additional_info
+            ) VALUES (
+                NOW(),
+                'running',
+                ?,
+                ?,
+                ?,
+                JSON_OBJECT(
+                    'skip_product_metrics', ?,
+                    'skip_time_aggregates', ?,
+                    'skip_financial_metrics', ?,
+                    'skip_vendor_metrics', ?,
+                    'skip_category_metrics', ?,
+                    'skip_brand_metrics', ?,
+                    'skip_sales_forecasts', ?
+                )
+            )
+        `, [
+            totalProducts,
+            totalOrders,
+            totalPurchaseOrders,
+            SKIP_PRODUCT_METRICS,
+            SKIP_TIME_AGGREGATES,
+            SKIP_FINANCIAL_METRICS, 
+            SKIP_VENDOR_METRICS,
+            SKIP_CATEGORY_METRICS,
+            SKIP_BRAND_METRICS,
+            SKIP_SALES_FORECASTS
+        ]);
+        calculateHistoryId = historyResult.insertId;
+        connection.release();
+
         // Add debug logging for the progress functions
         console.log('Debug - Progress functions:', {
             formatElapsedTime: typeof global.formatElapsedTime,
@@ -115,72 +211,150 @@ async function calculateMetrics() {
                 elapsed: '0s',
                 remaining: 'Calculating...',
                 rate: 0,
-                percentage: '0'
+                percentage: '0',
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+
+            // Update progress periodically
+            const updateProgress = async (products = null, orders = null, purchaseOrders = null) => {
+                // Ensure all values are valid numbers or default to previous value
+                if (products !== null) processedProducts = Number(products) || processedProducts || 0;
+                if (orders !== null) processedOrders = Number(orders) || processedOrders || 0;
+                if (purchaseOrders !== null) processedPurchaseOrders = Number(purchaseOrders) || processedPurchaseOrders || 0;
+
+                // Ensure we never send NaN to the database
+                const safeProducts = Number(processedProducts) || 0;
+                const safeOrders = Number(processedOrders) || 0;
+                const safePurchaseOrders = Number(processedPurchaseOrders) || 0;
+
+                await connection.query(`
+                    UPDATE calculate_history 
+                    SET 
+                        processed_products = ?,
+                        processed_orders = ?,
+                        processed_purchase_orders = ?
+                    WHERE id = ?
+                `, [safeProducts, safeOrders, safePurchaseOrders, calculateHistoryId]);
+            };
+
+            // Helper function to ensure valid progress numbers
+            const ensureValidProgress = (current, total) => ({
+                current: Number(current) || 0,
+                total: Number(total) || 1,  // Default to 1 to avoid division by zero
+                percentage: (((Number(current) || 0) / (Number(total) || 1)) * 100).toFixed(1)
+            });
+
+            // Initial progress
+            const initialProgress = ensureValidProgress(0, totalProducts);
+            global.outputProgress({
+                status: 'running',
+                operation: 'Starting metrics calculation',
+                current: initialProgress.current,
+                total: initialProgress.total,
+                elapsed: '0s',
+                remaining: 'Calculating...',
+                rate: 0,
+                percentage: initialProgress.percentage,
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            // Get total number of products
-            const [countResult] = await connection.query('SELECT COUNT(*) as total FROM products')
-                .catch(err => {
-                    global.logError(err, 'Failed to count products');
-                    throw err;
-                });
-            totalProducts = countResult[0].total;
-            
             if (!SKIP_PRODUCT_METRICS) {
-                processedCount = await calculateProductMetrics(startTime, totalProducts);
+                const result = await calculateProductMetrics(startTime, totalProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Product metrics calculation failed');
+                }
             } else {
                 console.log('Skipping product metrics calculation...');
-                processedCount = Math.floor(totalProducts * 0.6);
+                processedProducts = Math.floor(totalProducts * 0.6);
+                await updateProgress(processedProducts);
                 global.outputProgress({
                     status: 'running',
                     operation: 'Skipping product metrics calculation',
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts,
                     elapsed: global.formatElapsedTime(startTime),
-                    remaining: global.estimateRemaining(startTime, processedCount, totalProducts),
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: '60'
+                    remaining: global.estimateRemaining(startTime, processedProducts, totalProducts),
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: '60',
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             }
 
             // Calculate time-based aggregates
             if (!SKIP_TIME_AGGREGATES) {
-                processedCount = await calculateTimeAggregates(startTime, totalProducts, processedCount);
+                const result = await calculateTimeAggregates(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Time aggregates calculation failed');
+                }
             } else {
                 console.log('Skipping time aggregates calculation');
             }
 
             // Calculate financial metrics
             if (!SKIP_FINANCIAL_METRICS) {
-                processedCount = await calculateFinancialMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateFinancialMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Financial metrics calculation failed');
+                }
             } else {
                 console.log('Skipping financial metrics calculation');
             }
             
             // Calculate vendor metrics
             if (!SKIP_VENDOR_METRICS) {
-                processedCount = await calculateVendorMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateVendorMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Vendor metrics calculation failed');
+                }
             } else {
                 console.log('Skipping vendor metrics calculation');
             }
 
             // Calculate category metrics
             if (!SKIP_CATEGORY_METRICS) {
-                processedCount = await calculateCategoryMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateCategoryMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Category metrics calculation failed');
+                }
             } else {
                 console.log('Skipping category metrics calculation');
             }
 
             // Calculate brand metrics
             if (!SKIP_BRAND_METRICS) {
-                processedCount = await calculateBrandMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateBrandMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Brand metrics calculation failed');
+                }
             } else {
                 console.log('Skipping brand metrics calculation');
             }
 
             // Calculate sales forecasts
             if (!SKIP_SALES_FORECASTS) {
-                processedCount = await calculateSalesForecasts(startTime, totalProducts, processedCount);
+                const result = await calculateSalesForecasts(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Sales forecasts calculation failed');
+                }
             } else {
                 console.log('Skipping sales forecasts calculation');
             }
@@ -189,15 +363,25 @@ async function calculateMetrics() {
             outputProgress({
                 status: 'running',
                 operation: 'Starting ABC classification',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
             const [abcConfig] = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
             const abcThresholds = abcConfig[0] || { a_threshold: 20, b_threshold: 50 };
@@ -218,15 +402,25 @@ async function calculateMetrics() {
             outputProgress({
                 status: 'running',
                 operation: 'Creating revenue rankings',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
             await connection.query(`
                 INSERT INTO temp_revenue_ranks
@@ -247,26 +441,44 @@ async function calculateMetrics() {
             // Get total count for percentage calculation
             const [rankingCount] = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
             const totalCount = rankingCount[0].total_count || 1;
+            const max_rank = totalCount; // Store max_rank for use in classification
 
             outputProgress({
                 status: 'running',
                 operation: 'Updating ABC classifications',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
-            // Process updates in batches
+            // ABC classification progress tracking
             let abcProcessedCount = 0;
             const batchSize = 5000;
+            let lastProgressUpdate = Date.now();
+            const progressUpdateInterval = 1000; // Update every second
 
             while (true) {
-                if (isCancelled) return processedCount;
+                if (isCancelled) return {
+                    processedProducts: Number(processedProducts) || 0,
+                    processedOrders: Number(processedOrders) || 0,
+                    processedPurchaseOrders: 0,
+                    success: false
+                };
 
                 // First get a batch of PIDs that need updating
                 const [pids] = await connection.query(`
@@ -282,8 +494,8 @@ async function calculateMetrics() {
                             ELSE 'C'
                         END
                     LIMIT ?
-                `, [totalCount, abcThresholds.a_threshold,
-                    totalCount, abcThresholds.b_threshold,
+                `, [max_rank, abcThresholds.a_threshold,
+                    max_rank, abcThresholds.b_threshold,
                     batchSize]);
 
                 if (pids.length === 0) {
@@ -303,23 +515,42 @@ async function calculateMetrics() {
                         END,
                         pm.last_calculated_at = NOW()
                     WHERE pm.pid IN (?)
-                `, [totalCount, abcThresholds.a_threshold,
-                    totalCount, abcThresholds.b_threshold,
+                `, [max_rank, abcThresholds.a_threshold,
+                    max_rank, abcThresholds.b_threshold,
                     pids.map(row => row.pid)]);
 
                 abcProcessedCount += result.affectedRows;
-                processedCount = Math.floor(totalProducts * (0.99 + (abcProcessedCount / totalCount) * 0.01));
+                
+                // Calculate progress ensuring valid numbers
+                const currentProgress = Math.floor(totalProducts * (0.99 + (abcProcessedCount / (totalCount || 1)) * 0.01));
+                processedProducts = Number(currentProgress) || processedProducts || 0;
 
-                outputProgress({
-                    status: 'running',
-                    operation: 'ABC classification progress',
-                    current: processedCount,
-                    total: totalProducts,
-                    elapsed: formatElapsedTime(startTime),
-                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                    rate: calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-                });
+                // Only update progress at most once per second
+                const now = Date.now();
+                if (now - lastProgressUpdate >= progressUpdateInterval) {
+                    const progress = ensureValidProgress(processedProducts, totalProducts);
+                    
+                    outputProgress({
+                        status: 'running',
+                        operation: 'ABC classification progress',
+                        current: progress.current,
+                        total: progress.total,
+                        elapsed: formatElapsedTime(startTime),
+                        remaining: estimateRemaining(startTime, progress.current, progress.total),
+                        rate: calculateRate(startTime, progress.current),
+                        percentage: progress.percentage,
+                        timing: {
+                            start_time: new Date(startTime).toISOString(),
+                            end_time: new Date().toISOString(),
+                            elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                        }
+                    });
+                    
+                    lastProgressUpdate = now;
+                }
+
+                // Update database progress
+                await updateProgress(processedProducts, processedOrders, processedPurchaseOrders);
 
                 // Small delay between batches to allow other transactions
                 await new Promise(resolve => setTimeout(resolve, 100));
@@ -328,61 +559,145 @@ async function calculateMetrics() {
             // Clean up
             await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
 
+            const endTime = Date.now();
+            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
+
+            // Update calculate_status for ABC classification
+            await connection.query(`
+                INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+                VALUES ('abc_classification', NOW())
+                ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+            `);
+
+            // Final progress update with guaranteed valid numbers
+            const finalProgress = ensureValidProgress(totalProducts, totalProducts);
+            
             // Final success message
             outputProgress({
                 status: 'complete',
                 operation: 'Metrics calculation complete',
-                current: totalProducts,
-                total: totalProducts,
+                current: finalProgress.current,
+                total: finalProgress.total,
                 elapsed: formatElapsedTime(startTime),
                 remaining: '0s',
-                rate: calculateRate(startTime, totalProducts),
-                percentage: '100'
+                rate: calculateRate(startTime, finalProgress.current),
+                percentage: '100',
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: totalElapsedSeconds
+                }
             });
 
+            // Ensure all values are valid numbers before final update
+            const finalStats = {
+                processedProducts: Number(processedProducts) || 0,
+                processedOrders: Number(processedOrders) || 0,
+                processedPurchaseOrders: Number(processedPurchaseOrders) || 0
+            };
+
+            // Update history with completion
+            await connection.query(`
+                UPDATE calculate_history 
+                SET 
+                    end_time = NOW(),
+                    duration_seconds = ?,
+                    processed_products = ?,
+                    processed_orders = ?,
+                    processed_purchase_orders = ?,
+                    status = 'completed'
+                WHERE id = ?
+            `, [totalElapsedSeconds, 
+                finalStats.processedProducts,
+                finalStats.processedOrders, 
+                finalStats.processedPurchaseOrders,
+                calculateHistoryId]);
+
             // Clear progress file on successful completion
             global.clearProgress();
 
         } catch (error) {
+            const endTime = Date.now();
+            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
+
+            // Update history with error
+            await connection.query(`
+                UPDATE calculate_history 
+                SET 
+                    end_time = NOW(),
+                    duration_seconds = ?,
+                    processed_products = ?,
+                    processed_orders = ?,
+                    processed_purchase_orders = ?,
+                    status = ?,
+                    error_message = ?
+                WHERE id = ?
+            `, [
+                totalElapsedSeconds,
+                processedProducts || 0,  // Ensure we have a valid number
+                processedOrders || 0,    // Ensure we have a valid number
+                processedPurchaseOrders || 0,  // Ensure we have a valid number
+                isCancelled ? 'cancelled' : 'failed',
+                error.message,
+                calculateHistoryId
+            ]);
+
             if (isCancelled) {
                 global.outputProgress({
                     status: 'cancelled',
                     operation: 'Calculation cancelled',
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts || 0,
                     elapsed: global.formatElapsedTime(startTime),
                     remaining: null,
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / (totalProducts || 1)) * 100).toFixed(1)
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             } else {
                 global.outputProgress({
                     status: 'error',
                     operation: 'Error: ' + error.message,
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts || 0,
                     elapsed: global.formatElapsedTime(startTime),
                     remaining: null,
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / (totalProducts || 1)) * 100).toFixed(1)
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             }
             throw error;
         } finally {
             if (connection) {
-            connection.release();
+                // Ensure temporary tables are cleaned up
+                await cleanupTemporaryTables(connection);
+                connection.release();
             }
+            // Close the connection pool when we're done
+            await closePool();
         }
-    } finally {
-        // Close the connection pool when we're done
-        await closePool();
+    } catch (error) {
+        success = false;
+        logError(error, 'Error in metrics calculation');
+        throw error;
     }
 }
 
-// Export both functions and progress checker
-module.exports = calculateMetrics;
-module.exports.cancelCalculation = cancelCalculation;
-module.exports.getProgress = global.getProgress;
+// Export as a module with all necessary functions
+module.exports = {
+    calculateMetrics,
+    cancelCalculation,
+    getProgress: global.getProgress
+};
 
 // Run directly if called from command line
 if (require.main === module) {

inventory-server/scripts/full-reset.js

@@ -0,0 +1,115 @@
+const path = require('path');
+const { spawn } = require('child_process');
+
+function outputProgress(data) {
+    if (!data.status) {
+        data = {
+            status: 'running',
+            ...data
+        };
+    }
+    console.log(JSON.stringify(data));
+}
+
+function runScript(scriptPath) {
+    return new Promise((resolve, reject) => {
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe'],
+            env: {
+                ...process.env,
+                PGHOST: process.env.DB_HOST,
+                PGUSER: process.env.DB_USER,
+                PGPASSWORD: process.env.DB_PASSWORD,
+                PGDATABASE: process.env.DB_NAME,
+                PGPORT: process.env.DB_PORT || '5432'
+            }
+        });
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const lines = data.toString().split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    console.log(line); // Pass through the JSON output
+                    output += line + '\n';
+                } catch (e) {
+                    console.log(line); // If not JSON, just log it directly
+                }
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            console.error(data.toString());
+        });
+
+        child.on('close', (code) => {
+            if (code !== 0) {
+                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
+            } else {
+                resolve(output);
+            }
+        });
+
+        child.on('error', (err) => {
+            reject(err);
+        });
+    });
+}
+
+async function fullReset() {
+    try {
+        // Step 1: Reset Database
+        outputProgress({
+            operation: 'Starting full reset',
+            message: 'Step 1/3: Resetting database...'
+        });
+        await runScript(path.join(__dirname, 'reset-db.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Database reset step complete',
+            message: 'Database reset finished, moving to import...'
+        });
+
+        // Step 2: Import from Production
+        outputProgress({
+            operation: 'Starting import',
+            message: 'Step 2/3: Importing from production...'
+        });
+        await runScript(path.join(__dirname, 'import-from-prod.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Import step complete',
+            message: 'Import finished, moving to metrics calculation...'
+        });
+
+        // Step 3: Calculate Metrics
+        outputProgress({
+            operation: 'Starting metrics calculation',
+            message: 'Step 3/3: Calculating metrics...'
+        });
+        await runScript(path.join(__dirname, 'calculate-metrics.js'));
+        
+        // Final completion message
+        outputProgress({
+            status: 'complete',
+            operation: 'Full reset complete',
+            message: 'Successfully completed all steps: database reset, import, and metrics calculation'
+        });
+    } catch (error) {
+        outputProgress({
+            status: 'error',
+            operation: 'Full reset failed',
+            error: error.message,
+            stack: error.stack
+        });
+        process.exit(1);
+    }
+}
+
+// Run if called directly
+if (require.main === module) {
+    fullReset();
+}
+
+module.exports = fullReset; 

inventory-server/scripts/full-update.js

@@ -0,0 +1,100 @@
+const path = require('path');
+const { spawn } = require('child_process');
+
+function outputProgress(data) {
+    if (!data.status) {
+        data = {
+            status: 'running',
+            ...data
+        };
+    }
+    console.log(JSON.stringify(data));
+}
+
+function runScript(scriptPath) {
+    return new Promise((resolve, reject) => {
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe']
+        });
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const lines = data.toString().split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    console.log(line); // Pass through the JSON output
+                    output += line + '\n';
+                } catch (e) {
+                    console.log(line); // If not JSON, just log it directly
+                }
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            console.error(data.toString());
+        });
+
+        child.on('close', (code) => {
+            if (code !== 0) {
+                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
+            } else {
+                resolve(output);
+            }
+        });
+
+        child.on('error', (err) => {
+            reject(err);
+        });
+    });
+}
+
+async function fullUpdate() {
+    try {
+        // Step 1: Import from Production
+        outputProgress({
+            operation: 'Starting full update',
+            message: 'Step 1/2: Importing from production...'
+        });
+        await runScript(path.join(__dirname, 'import-from-prod.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Import step complete',
+            message: 'Import finished, moving to metrics calculation...'
+        });
+
+        // Step 2: Calculate Metrics
+        outputProgress({
+            operation: 'Starting metrics calculation',
+            message: 'Step 2/2: Calculating metrics...'
+        });
+        await runScript(path.join(__dirname, 'calculate-metrics.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Metrics step complete',
+            message: 'Metrics calculation finished'
+        });
+
+        // Final completion message
+        outputProgress({
+            status: 'complete',
+            operation: 'Full update complete',
+            message: 'Successfully completed all steps: import and metrics calculation'
+        });
+    } catch (error) {
+        outputProgress({
+            status: 'error',
+            operation: 'Full update failed',
+            error: error.message,
+            stack: error.stack
+        });
+        process.exit(1);
+    }
+}
+
+// Run if called directly
+if (require.main === module) {
+    fullUpdate();
+}
+
+module.exports = fullUpdate; 

inventory-server/scripts/import-from-prod.js

@@ -19,7 +19,6 @@ const IMPORT_PURCHASE_ORDERS = true;
 const INCREMENTAL_UPDATE = process.env.INCREMENTAL_UPDATE !== 'false';  // Default to true unless explicitly set to false
 
 // SSH configuration
-// In import-from-prod.js
 const sshConfig = {
   ssh: {
     host: process.env.PROD_SSH_HOST,
@@ -31,6 +30,7 @@ const sshConfig = {
     compress: true, // Enable SSH compression
   },
   prodDbConfig: {
+    // MySQL config for production
     host: process.env.PROD_DB_HOST || "localhost",
     user: process.env.PROD_DB_USER,
     password: process.env.PROD_DB_PASSWORD,
@@ -39,21 +39,16 @@ const sshConfig = {
     timezone: 'Z',
   },
   localDbConfig: {
+    // PostgreSQL config for local
     host: process.env.DB_HOST,
     user: process.env.DB_USER,
     password: process.env.DB_PASSWORD,
     database: process.env.DB_NAME,
-    multipleStatements: true,
-    waitForConnections: true,
-    connectionLimit: 10,
-    queueLimit: 0,
-    namedPlaceholders: true,
-    connectTimeout: 60000,
-    enableKeepAlive: true,
-    keepAliveInitialDelay: 10000,
-    compress: true,
-    timezone: 'Z',
-    stringifyObjects: false,
+    port: process.env.DB_PORT || 5432,
+    ssl: process.env.DB_SSL === 'true',
+    connectionTimeoutMillis: 60000,
+    idleTimeoutMillis: 30000,
+    max: 10 // connection pool max size
   }
 };
 
@@ -108,7 +103,7 @@ async function main() {
       SET 
         status = 'cancelled',
         end_time = NOW(),
-        duration_seconds = TIMESTAMPDIFF(SECOND, start_time, NOW()),
+        duration_seconds = EXTRACT(EPOCH FROM (NOW() - start_time))::INTEGER,
         error_message = 'Previous import was not completed properly'
       WHERE status = 'running'
     `);
@@ -118,9 +113,10 @@ async function main() {
       CREATE TABLE IF NOT EXISTS sync_status (
         table_name VARCHAR(50) PRIMARY KEY,
         last_sync_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-        last_sync_id BIGINT,
-        INDEX idx_last_sync (last_sync_timestamp)
+        last_sync_id BIGINT
       );
+
+      CREATE INDEX IF NOT EXISTS idx_last_sync ON sync_status (last_sync_timestamp);
     `);
 
     // Create import history record for the overall session
@@ -134,17 +130,17 @@ async function main() {
       ) VALUES (
         'all_tables',
         NOW(),
-        ?,
+        $1::boolean,
         'running',
-        JSON_OBJECT(
-          'categories_enabled', ?,
-          'products_enabled', ?,
-          'orders_enabled', ?,
-          'purchase_orders_enabled', ?
+        jsonb_build_object(
+          'categories_enabled', $2::boolean,
+          'products_enabled', $3::boolean,
+          'orders_enabled', $4::boolean,
+          'purchase_orders_enabled', $5::boolean
         )
-      )
+      ) RETURNING id
     `, [INCREMENTAL_UPDATE, IMPORT_CATEGORIES, IMPORT_PRODUCTS, IMPORT_ORDERS, IMPORT_PURCHASE_ORDERS]);
-    importHistoryId = historyResult.insertId;
+    importHistoryId = historyResult.rows[0].id;
 
     const results = {
       categories: null,
@@ -162,8 +158,8 @@ async function main() {
       if (isImportCancelled) throw new Error("Import cancelled");
       completedSteps++;
       console.log('Categories import result:', results.categories);
-      totalRecordsAdded += results.categories?.recordsAdded || 0;
-      totalRecordsUpdated += results.categories?.recordsUpdated || 0;
+      totalRecordsAdded += parseInt(results.categories?.recordsAdded || 0);
+      totalRecordsUpdated += parseInt(results.categories?.recordsUpdated || 0);
     }
 
     if (IMPORT_PRODUCTS) {
@@ -171,8 +167,8 @@ async function main() {
       if (isImportCancelled) throw new Error("Import cancelled");
       completedSteps++;
       console.log('Products import result:', results.products);
-      totalRecordsAdded += results.products?.recordsAdded || 0;
-      totalRecordsUpdated += results.products?.recordsUpdated || 0;
+      totalRecordsAdded += parseInt(results.products?.recordsAdded || 0);
+      totalRecordsUpdated += parseInt(results.products?.recordsUpdated || 0);
     }
 
     if (IMPORT_ORDERS) {
@@ -180,8 +176,8 @@ async function main() {
       if (isImportCancelled) throw new Error("Import cancelled");
       completedSteps++;
       console.log('Orders import result:', results.orders);
-      totalRecordsAdded += results.orders?.recordsAdded || 0;
-      totalRecordsUpdated += results.orders?.recordsUpdated || 0;
+      totalRecordsAdded += parseInt(results.orders?.recordsAdded || 0);
+      totalRecordsUpdated += parseInt(results.orders?.recordsUpdated || 0);
     }
 
     if (IMPORT_PURCHASE_ORDERS) {
@@ -189,8 +185,8 @@ async function main() {
       if (isImportCancelled) throw new Error("Import cancelled");
       completedSteps++;
       console.log('Purchase orders import result:', results.purchaseOrders);
-      totalRecordsAdded += results.purchaseOrders?.recordsAdded || 0;
-      totalRecordsUpdated += results.purchaseOrders?.recordsUpdated || 0;
+      totalRecordsAdded += parseInt(results.purchaseOrders?.recordsAdded || 0);
+      totalRecordsUpdated += parseInt(results.purchaseOrders?.recordsUpdated || 0);
     }
 
     const endTime = Date.now();
@@ -201,21 +197,21 @@ async function main() {
       UPDATE import_history 
       SET 
         end_time = NOW(),
-        duration_seconds = ?,
-        records_added = ?,
-        records_updated = ?,
+        duration_seconds = $1,
+        records_added = $2,
+        records_updated = $3,
         status = 'completed',
-        additional_info = JSON_OBJECT(
-          'categories_enabled', ?,
-          'products_enabled', ?,
-          'orders_enabled', ?,
-          'purchase_orders_enabled', ?,
-          'categories_result', CAST(? AS JSON),
-          'products_result', CAST(? AS JSON),
-          'orders_result', CAST(? AS JSON),
-          'purchase_orders_result', CAST(? AS JSON)
+        additional_info = jsonb_build_object(
+          'categories_enabled', $4::boolean,
+          'products_enabled', $5::boolean,
+          'orders_enabled', $6::boolean,
+          'purchase_orders_enabled', $7::boolean,
+          'categories_result', COALESCE($8::jsonb, 'null'::jsonb),
+          'products_result', COALESCE($9::jsonb, 'null'::jsonb),
+          'orders_result', COALESCE($10::jsonb, 'null'::jsonb),
+          'purchase_orders_result', COALESCE($11::jsonb, 'null'::jsonb)
         )
-      WHERE id = ?
+      WHERE id = $12
     `, [
       totalElapsedSeconds,
       totalRecordsAdded,
@@ -259,10 +255,10 @@ async function main() {
         UPDATE import_history 
         SET 
           end_time = NOW(),
-          duration_seconds = ?,
-          status = ?,
-          error_message = ?
-        WHERE id = ?
+          duration_seconds = $1,
+          status = $2,
+          error_message = $3
+        WHERE id = $4
       `, [totalElapsedSeconds, error.message === "Import cancelled" ? 'cancelled' : 'failed', error.message, importHistoryId]);
     }
 
@@ -288,16 +284,23 @@ async function main() {
     throw error;
   } finally {
     if (connections) {
-      await closeConnections(connections);
+      await closeConnections(connections).catch(err => {
+        console.error("Error closing connections:", err);
+      });
     }
   }
 }
 
 // Run the import only if this is the main module
 if (require.main === module) {
-  main().catch((error) => {
+  main().then((results) => {
+    console.log('Import completed successfully:', results);
+    // Force exit after a small delay to ensure all logs are written
+    setTimeout(() => process.exit(0), 500);
+  }).catch((error) => {
     console.error("Unhandled error in main process:", error);
-    process.exit(1);
+    // Force exit with error code after a small delay
+    setTimeout(() => process.exit(1), 500);
   });
 }
 

inventory-server/scripts/import/categories.js

@@ -9,170 +9,206 @@ async function importCategories(prodConnection, localConnection) {
   const startTime = Date.now();
   const typeOrder = [10, 20, 11, 21, 12, 13];
   let totalInserted = 0;
+  let totalUpdated = 0;
   let skippedCategories = [];
 
   try {
-    // Process each type in order with its own query
+    // Start a single transaction for the entire import
+    await localConnection.query('BEGIN');
+
+    // Process each type in order with its own savepoint
     for (const type of typeOrder) {
-      const [categories] = await prodConnection.query(
-        `
-        SELECT 
-          pc.cat_id,
-          pc.name,
-          pc.type,
-          CASE 
-            WHEN pc.type IN (10, 20) THEN NULL  -- Top level categories should have no parent
-            WHEN pc.master_cat_id IS NULL THEN NULL
-            ELSE pc.master_cat_id
-          END as parent_id,
-          pc.combined_name as description
-        FROM product_categories pc
-        WHERE pc.type = ?
-        ORDER BY pc.cat_id
-      `,
-        [type]
-      );
+      try {
+        // Create a savepoint for this type
+        await localConnection.query(`SAVEPOINT category_type_${type}`);
 
-      if (categories.length === 0) continue;
-
-      console.log(`\nProcessing ${categories.length} type ${type} categories`);
-      if (type === 10) {
-        console.log("Type 10 categories:", JSON.stringify(categories, null, 2));
-      }
-
-      // For types that can have parents (11, 21, 12, 13), verify parent existence
-      let categoriesToInsert = categories;
-      if (![10, 20].includes(type)) {
-        // Get all parent IDs
-        const parentIds = [
-          ...new Set(
-            categories.map((c) => c.parent_id).filter((id) => id !== null)
-          ),
-        ];
-
-        // Check which parents exist
-        const [existingParents] = await localConnection.query(
-          "SELECT cat_id FROM categories WHERE cat_id IN (?)",
-          [parentIds]
-        );
-        const existingParentIds = new Set(existingParents.map((p) => p.cat_id));
-
-        // Filter categories and track skipped ones
-        categoriesToInsert = categories.filter(
-          (cat) =>
-            cat.parent_id === null || existingParentIds.has(cat.parent_id)
-        );
-        const invalidCategories = categories.filter(
-          (cat) =>
-            cat.parent_id !== null && !existingParentIds.has(cat.parent_id)
+        // Production query remains MySQL compatible
+        const [categories] = await prodConnection.query(
+          `
+          SELECT 
+            pc.cat_id,
+            pc.name,
+            pc.type,
+            CASE 
+              WHEN pc.type IN (10, 20) THEN NULL  -- Top level categories should have no parent
+              WHEN pc.master_cat_id IS NULL THEN NULL
+              ELSE pc.master_cat_id
+            END as parent_id,
+            pc.combined_name as description
+          FROM product_categories pc
+          WHERE pc.type = ?
+          ORDER BY pc.cat_id
+        `,
+          [type]
         );
 
-        if (invalidCategories.length > 0) {
-          const skippedInfo = invalidCategories.map((c) => ({
-            id: c.cat_id,
-            name: c.name,
-            type: c.type,
-            missing_parent: c.parent_id,
-          }));
-          skippedCategories.push(...skippedInfo);
+        if (categories.length === 0) {
+          await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
+          continue;
+        }
 
-          console.log(
-            "\nSkipping categories with missing parents:",
-            invalidCategories
-              .map(
-                (c) =>
-                  `${c.cat_id} - ${c.name} (missing parent: ${c.parent_id})`
-              )
-              .join("\n")
-          );
+        console.log(`\nProcessing ${categories.length} type ${type} categories`);
+        if (type === 10) {
+          console.log("Type 10 categories:", JSON.stringify(categories, null, 2));
+        }
+
+        // For types that can have parents (11, 21, 12, 13), verify parent existence
+        let categoriesToInsert = categories;
+        if (![10, 20].includes(type)) {
+          // Get all parent IDs
+          const parentIds = [
+            ...new Set(
+              categories
+                .filter(c => c && c.parent_id !== null)
+                .map(c => c.parent_id)
+            ),
+          ];
+
+          console.log(`Processing ${categories.length} type ${type} categories with ${parentIds.length} unique parent IDs`);
+          console.log('Parent IDs:', parentIds);
+
+          // No need to check for parent existence - we trust they exist since they were just inserted
+          categoriesToInsert = categories;
         }
 
         if (categoriesToInsert.length === 0) {
           console.log(
-            `No valid categories of type ${type} to insert - all had missing parents`
+            `No valid categories of type ${type} to insert`
           );
+          await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
           continue;
         }
+
+        console.log(
+          `Inserting ${categoriesToInsert.length} type ${type} categories`
+        );
+
+        // PostgreSQL upsert query with parameterized values
+        const values = categoriesToInsert.flatMap((cat) => [
+          cat.cat_id,
+          cat.name,
+          cat.type,
+          cat.parent_id,
+          cat.description,
+          'active',
+          new Date(),
+          new Date()
+        ]);
+
+        console.log('Attempting to insert/update with values:', JSON.stringify(values, null, 2));
+
+        const placeholders = categoriesToInsert
+          .map((_, i) => `($${i * 8 + 1}, $${i * 8 + 2}, $${i * 8 + 3}, $${i * 8 + 4}, $${i * 8 + 5}, $${i * 8 + 6}, $${i * 8 + 7}, $${i * 8 + 8})`)
+          .join(',');
+
+        console.log('Using placeholders:', placeholders);
+
+        // Insert categories with ON CONFLICT clause for PostgreSQL
+        const query = `
+          WITH inserted_categories AS (
+            INSERT INTO categories (
+              cat_id, name, type, parent_id, description, status, created_at, updated_at
+            )
+            VALUES ${placeholders}
+            ON CONFLICT (cat_id) DO UPDATE SET
+              name = EXCLUDED.name,
+              type = EXCLUDED.type,
+              parent_id = EXCLUDED.parent_id,
+              description = EXCLUDED.description,
+              status = EXCLUDED.status,
+              updated_at = EXCLUDED.updated_at
+            RETURNING 
+              cat_id,
+              CASE 
+                WHEN xmax = 0 THEN true
+                ELSE false
+              END as is_insert
+          )
+          SELECT 
+            COUNT(*) as total,
+            COUNT(*) FILTER (WHERE is_insert) as inserted,
+            COUNT(*) FILTER (WHERE NOT is_insert) as updated
+          FROM inserted_categories`;
+
+        console.log('Executing query:', query);
+        
+        const result = await localConnection.query(query, values);
+        console.log('Query result:', result);
+
+        // Get the first result since query returns an array
+        const queryResult = Array.isArray(result) ? result[0] : result;
+
+        if (!queryResult || !queryResult.rows || !queryResult.rows[0]) {
+          console.error('Query failed to return results. Result:', queryResult);
+          throw new Error('Query did not return expected results');
+        }
+
+        const total = parseInt(queryResult.rows[0].total) || 0;
+        const inserted = parseInt(queryResult.rows[0].inserted) || 0;
+        const updated = parseInt(queryResult.rows[0].updated) || 0;
+
+        console.log(`Total: ${total}, Inserted: ${inserted}, Updated: ${updated}`);
+        
+        totalInserted += inserted;
+        totalUpdated += updated;
+
+        // Release the savepoint for this type
+        await localConnection.query(`RELEASE SAVEPOINT category_type_${type}`);
+
+        outputProgress({
+          status: "running",
+          operation: "Categories import",
+          message: `Imported ${inserted} (updated ${updated}) categories of type ${type}`,
+          current: totalInserted + totalUpdated,
+          total: categories.length,
+          elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
+        });
+      } catch (error) {
+        // Rollback to the savepoint for this type
+        await localConnection.query(`ROLLBACK TO SAVEPOINT category_type_${type}`);
+        throw error;
       }
-
-      console.log(
-        `Inserting ${categoriesToInsert.length} type ${type} categories`
-      );
-
-      const placeholders = categoriesToInsert
-        .map(() => "(?, ?, ?, ?, ?, ?, CURRENT_TIMESTAMP, CURRENT_TIMESTAMP)")
-        .join(",");
-
-      const values = categoriesToInsert.flatMap((cat) => [
-        cat.cat_id,
-        cat.name,
-        cat.type,
-        cat.parent_id,
-        cat.description,
-        "active",
-      ]);
-
-      // Insert categories and create relationships in one query to avoid race conditions
-      await localConnection.query(
-        `
-        INSERT INTO categories (cat_id, name, type, parent_id, description, status, created_at, updated_at)
-        VALUES ${placeholders}
-        ON DUPLICATE KEY UPDATE
-          name = VALUES(name),
-          type = VALUES(type),
-          parent_id = VALUES(parent_id),
-          description = VALUES(description),
-          status = VALUES(status),
-          updated_at = CURRENT_TIMESTAMP
-      `,
-        values
-      );
-
-      totalInserted += categoriesToInsert.length;
-      outputProgress({
-        status: "running",
-        operation: "Categories import",
-        current: totalInserted,
-        total: totalInserted,
-        elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
-      });
     }
 
-    // After all imports, if we skipped any categories, throw an error
-    if (skippedCategories.length > 0) {
-      const error = new Error(
-        "Categories import completed with errors - some categories were skipped due to missing parents"
-      );
-      error.skippedCategories = skippedCategories;
-      throw error;
-    }
+    // Commit the entire transaction - we'll do this even if we have skipped categories
+    await localConnection.query('COMMIT');
 
     outputProgress({
       status: "complete",
       operation: "Categories import completed",
-      current: totalInserted,
-      total: totalInserted,
+      current: totalInserted + totalUpdated,
+      total: totalInserted + totalUpdated,
       duration: formatElapsedTime((Date.now() - startTime) / 1000),
+      warnings: skippedCategories.length > 0 ? {
+        message: "Some categories were skipped due to missing parents",
+        skippedCategories
+      } : undefined
     });
 
     return {
       status: "complete",
-      totalImported: totalInserted
+      recordsAdded: totalInserted,
+      recordsUpdated: totalUpdated,
+      totalRecords: totalInserted + totalUpdated,
+      warnings: skippedCategories.length > 0 ? {
+        message: "Some categories were skipped due to missing parents",
+        skippedCategories
+      } : undefined
     };
   } catch (error) {
     console.error("Error importing categories:", error);
-    if (error.skippedCategories) {
-      console.error(
-        "Skipped categories:",
-        JSON.stringify(error.skippedCategories, null, 2)
-      );
+    
+    // Only rollback if we haven't committed yet
+    try {
+      await localConnection.query('ROLLBACK');
+    } catch (rollbackError) {
+      console.error("Error during rollback:", rollbackError);
     }
     
     outputProgress({
       status: "error",
       operation: "Categories import failed",
-      error: error.message,
-      skippedCategories: error.skippedCategories
+      error: error.message
     });
 
     throw error;

inventory-server/scripts/import/orders.js

@@ -2,7 +2,7 @@ const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate } =
 const { importMissingProducts, setupTemporaryTables, cleanupTemporaryTables, materializeCalculations } = require('./products');
 
 /**
- * Imports orders from a production MySQL database to a local MySQL database.
+ * Imports orders from a production MySQL database to a local PostgreSQL database.
  * It can run in two modes:
  *   1. Incremental update mode (default): Only fetch orders that have changed since the last sync time.
  *   2. Full update mode: Fetch all eligible orders within the last 5 years regardless of timestamp.
@@ -23,77 +23,18 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
   let importedCount = 0;
   let totalOrderItems = 0;
   let totalUniqueOrders = 0;
-
-  // Add a cumulative counter for processed orders before the loop
   let cumulativeProcessedOrders = 0;
 
   try {
-    // Insert temporary table creation queries
-    await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_items (
-        order_id INT UNSIGNED NOT NULL,
-        pid INT UNSIGNED NOT NULL,
-        SKU VARCHAR(50) NOT NULL,
-        price DECIMAL(10,2) NOT NULL,
-        quantity INT NOT NULL,
-        base_discount DECIMAL(10,2) DEFAULT 0,
-        PRIMARY KEY (order_id, pid)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-    `);
-    await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_meta (
-        order_id INT UNSIGNED NOT NULL,
-        date DATE NOT NULL,
-        customer VARCHAR(100) NOT NULL,
-        customer_name VARCHAR(150) NOT NULL,
-        status INT,
-        canceled TINYINT(1),
-        PRIMARY KEY (order_id)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-    `);
-    await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_discounts (
-        order_id INT UNSIGNED NOT NULL,
-        pid INT UNSIGNED NOT NULL,
-        discount DECIMAL(10,2) NOT NULL,
-        PRIMARY KEY (order_id, pid)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-    `);
-    await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_taxes (
-        order_id INT UNSIGNED NOT NULL,
-        pid INT UNSIGNED NOT NULL,
-        tax DECIMAL(10,2) NOT NULL,
-        PRIMARY KEY (order_id, pid)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-    `);
-    await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_costs (
-        order_id INT UNSIGNED NOT NULL,
-        pid INT UNSIGNED NOT NULL,
-        costeach DECIMAL(10,3) DEFAULT 0.000,
-        PRIMARY KEY (order_id, pid)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
-    `);
-
-    // Get column names from the local table
-    const [columns] = await localConnection.query(`
-      SELECT COLUMN_NAME 
-      FROM INFORMATION_SCHEMA.COLUMNS 
-      WHERE TABLE_NAME = 'orders'
-      ORDER BY ORDINAL_POSITION
-    `);
-    const columnNames = columns.map(col => col.COLUMN_NAME);
-
     // Get last sync info
     const [syncInfo] = await localConnection.query(
       "SELECT last_sync_timestamp FROM sync_status WHERE table_name = 'orders'"
     );
-    const lastSyncTime = syncInfo?.[0]?.last_sync_timestamp || '1970-01-01';
+    const lastSyncTime = syncInfo?.rows?.[0]?.last_sync_timestamp || '1970-01-01';
 
     console.log('Orders: Using last sync time:', lastSyncTime);
 
-    // First get count of order items
+    // First get count of order items - Keep MySQL compatible for production
     const [[{ total }]] = await prodConnection.query(`
       SELECT COUNT(*) as total
       FROM order_items oi
@@ -125,12 +66,13 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
     totalOrderItems = total;
     console.log('Orders: Found changes:', totalOrderItems);
 
-    // Get order items in batches
+    // Get order items - Keep MySQL compatible for production
+    console.log('Orders: Starting MySQL query...');
     const [orderItems] = await prodConnection.query(`
       SELECT 
         oi.order_id,
-        oi.prod_pid as pid,
-        oi.prod_itemnumber as SKU,
+        oi.prod_pid,
+        COALESCE(NULLIF(TRIM(oi.prod_itemnumber), ''), 'NO-SKU') as SKU,
         oi.prod_price as price,
         oi.qty_ordered as quantity,
         COALESCE(oi.prod_price_reg - oi.prod_price, 0) as base_discount,
@@ -161,24 +103,78 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
         ` : ''}
     `, incrementalUpdate ? [lastSyncTime, lastSyncTime, lastSyncTime] : []);
 
-    console.log('Orders: Processing', orderItems.length, 'order items');
+    console.log('Orders: Found', orderItems.length, 'order items to process');
+
+    // Create tables in PostgreSQL for debugging
+    await localConnection.query(`
+      DROP TABLE IF EXISTS debug_order_items;
+      DROP TABLE IF EXISTS debug_order_meta;
+      DROP TABLE IF EXISTS debug_order_discounts;
+      DROP TABLE IF EXISTS debug_order_taxes;
+      DROP TABLE IF EXISTS debug_order_costs;
+
+      CREATE TABLE debug_order_items (
+        order_id INTEGER NOT NULL,
+        pid INTEGER NOT NULL,
+        SKU VARCHAR(50) NOT NULL,
+        price DECIMAL(10,2) NOT NULL,
+        quantity INTEGER NOT NULL,
+        base_discount DECIMAL(10,2) DEFAULT 0,
+        PRIMARY KEY (order_id, pid)
+      );
+
+      CREATE TABLE debug_order_meta (
+        order_id INTEGER NOT NULL,
+        date DATE NOT NULL,
+        customer VARCHAR(100) NOT NULL,
+        customer_name VARCHAR(150) NOT NULL,
+        status INTEGER,
+        canceled BOOLEAN,
+        summary_discount DECIMAL(10,2) DEFAULT 0.00,
+        summary_subtotal DECIMAL(10,2) DEFAULT 0.00,
+        PRIMARY KEY (order_id)
+      );
+
+      CREATE TABLE debug_order_discounts (
+        order_id INTEGER NOT NULL,
+        pid INTEGER NOT NULL,
+        discount DECIMAL(10,2) NOT NULL,
+        PRIMARY KEY (order_id, pid)
+      );
+
+      CREATE TABLE debug_order_taxes (
+        order_id INTEGER NOT NULL,
+        pid INTEGER NOT NULL,
+        tax DECIMAL(10,2) NOT NULL,
+        PRIMARY KEY (order_id, pid)
+      );
+
+      CREATE TABLE debug_order_costs (
+        order_id INTEGER NOT NULL,
+        pid INTEGER NOT NULL,
+        costeach DECIMAL(10,3) DEFAULT 0.000,
+        PRIMARY KEY (order_id, pid)
+      );
+    `);
 
     // Insert order items in batches
     for (let i = 0; i < orderItems.length; i += 5000) {
       const batch = orderItems.slice(i, Math.min(i + 5000, orderItems.length));
-      const placeholders = batch.map(() => "(?, ?, ?, ?, ?, ?)").join(",");
+      const placeholders = batch.map((_, idx) => 
+        `($${idx * 6 + 1}, $${idx * 6 + 2}, $${idx * 6 + 3}, $${idx * 6 + 4}, $${idx * 6 + 5}, $${idx * 6 + 6})`
+      ).join(",");
       const values = batch.flatMap(item => [
-        item.order_id, item.pid, item.SKU, item.price, item.quantity, item.base_discount
+        item.order_id, item.prod_pid, item.SKU, item.price, item.quantity, item.base_discount
       ]);
 
       await localConnection.query(`
-        INSERT INTO temp_order_items (order_id, pid, SKU, price, quantity, base_discount)
+        INSERT INTO debug_order_items (order_id, pid, SKU, price, quantity, base_discount)
         VALUES ${placeholders}
-        ON DUPLICATE KEY UPDATE
-          SKU = VALUES(SKU),
-          price = VALUES(price),
-          quantity = VALUES(quantity),
-          base_discount = VALUES(base_discount)
+        ON CONFLICT (order_id, pid) DO UPDATE SET
+          SKU = EXCLUDED.SKU,
+          price = EXCLUDED.price,
+          quantity = EXCLUDED.quantity,
+          base_discount = EXCLUDED.base_discount
       `, values);
 
       processedCount = i + batch.length;
@@ -187,24 +183,26 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
         operation: "Orders import",
         message: `Loading order items: ${processedCount} of ${totalOrderItems}`,
         current: processedCount,
-        total: totalOrderItems
+        total: totalOrderItems,
+        elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
+        remaining: estimateRemaining(startTime, processedCount, totalOrderItems),
+        rate: calculateRate(startTime, processedCount)
       });
     }
 
     // Get unique order IDs
     const orderIds = [...new Set(orderItems.map(item => item.order_id))];
     totalUniqueOrders = orderIds.length;
-    console.log('Total unique order IDs:', totalUniqueOrders);
+    console.log('Orders: Processing', totalUniqueOrders, 'unique orders');
 
     // Reset processed count for order processing phase
     processedCount = 0;
 
-    // Get order metadata in batches
-    for (let i = 0; i < orderIds.length; i += 5000) {
-      const batchIds = orderIds.slice(i, i + 5000);
-      console.log(`Processing batch ${i/5000 + 1}, size: ${batchIds.length}`);
-      console.log('Sample of batch IDs:', batchIds.slice(0, 5));
-      
+    // Process metadata, discounts, taxes, and costs in parallel
+    const METADATA_BATCH_SIZE = 2000;
+    const PG_BATCH_SIZE = 200;
+    
+    const processMetadataBatch = async (batchIds) => {
       const [orders] = await prodConnection.query(`
         SELECT 
           o.order_id,
@@ -212,51 +210,53 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
           o.order_cid as customer,
           CONCAT(COALESCE(u.firstname, ''), ' ', COALESCE(u.lastname, '')) as customer_name,
           o.order_status as status,
-          CASE WHEN o.date_cancelled != '0000-00-00 00:00:00' THEN 1 ELSE 0 END as canceled
+          CASE WHEN o.date_cancelled != '0000-00-00 00:00:00' THEN 1 ELSE 0 END as canceled,
+          o.summary_discount,
+          o.summary_subtotal
         FROM _order o
         LEFT JOIN users u ON o.order_cid = u.cid
         WHERE o.order_id IN (?)
       `, [batchIds]);
-      
-      console.log(`Retrieved ${orders.length} orders for ${batchIds.length} IDs`);
-      const duplicates = orders.filter((order, index, self) => 
-        self.findIndex(o => o.order_id === order.order_id) !== index
-      );
-      if (duplicates.length > 0) {
-        console.log('Found duplicates:', duplicates);
+
+      // Process in sub-batches for PostgreSQL
+      for (let j = 0; j < orders.length; j += PG_BATCH_SIZE) {
+        const subBatch = orders.slice(j, j + PG_BATCH_SIZE);
+        if (subBatch.length === 0) continue;
+
+        const placeholders = subBatch.map((_, idx) => 
+          `($${idx * 8 + 1}, $${idx * 8 + 2}, $${idx * 8 + 3}, $${idx * 8 + 4}, $${idx * 8 + 5}, $${idx * 8 + 6}, $${idx * 8 + 7}, $${idx * 8 + 8})`
+        ).join(",");
+        
+        const values = subBatch.flatMap(order => [
+          order.order_id,
+          order.date,
+          order.customer,
+          order.customer_name || '',
+          order.status,
+          order.canceled,
+          order.summary_discount || 0,
+          order.summary_subtotal || 0
+        ]);
+
+        await localConnection.query(`
+          INSERT INTO debug_order_meta (
+            order_id, date, customer, customer_name, status, canceled,
+            summary_discount, summary_subtotal
+          )
+          VALUES ${placeholders}
+          ON CONFLICT (order_id) DO UPDATE SET
+            date = EXCLUDED.date,
+            customer = EXCLUDED.customer,
+            customer_name = EXCLUDED.customer_name,
+            status = EXCLUDED.status,
+            canceled = EXCLUDED.canceled,
+            summary_discount = EXCLUDED.summary_discount,
+            summary_subtotal = EXCLUDED.summary_subtotal
+        `, values);
       }
+    };
 
-      const placeholders = orders.map(() => "(?, ?, ?, ?, ?, ?)").join(",");
-      const values = orders.flatMap(order => [
-        order.order_id, order.date, order.customer, order.customer_name, order.status, order.canceled
-      ]);
-
-      await localConnection.query(`
-        INSERT INTO temp_order_meta VALUES ${placeholders}
-        ON DUPLICATE KEY UPDATE
-          date = VALUES(date),
-          customer = VALUES(customer),
-          customer_name = VALUES(customer_name),
-          status = VALUES(status),
-          canceled = VALUES(canceled)
-      `, values);
-
-      processedCount = i + orders.length;
-      outputProgress({
-        status: "running",
-        operation: "Orders import",
-        message: `Loading order metadata: ${processedCount} of ${totalUniqueOrders}`,
-        current: processedCount,
-        total: totalUniqueOrders
-      });
-    }
-
-    // Reset processed count for final phase
-    processedCount = 0;
-
-    // Get promotional discounts in batches
-    for (let i = 0; i < orderIds.length; i += 5000) {
-      const batchIds = orderIds.slice(i, i + 5000);
+    const processDiscountsBatch = async (batchIds) => {
       const [discounts] = await prodConnection.query(`
         SELECT order_id, pid, SUM(amount) as discount
         FROM order_discount_items
@@ -264,289 +264,296 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
         GROUP BY order_id, pid
       `, [batchIds]);
 
-      if (discounts.length > 0) {
-        const placeholders = discounts.map(() => "(?, ?, ?)").join(",");
-        const values = discounts.flatMap(d => [d.order_id, d.pid, d.discount]);
+      if (discounts.length === 0) return;
+
+      for (let j = 0; j < discounts.length; j += PG_BATCH_SIZE) {
+        const subBatch = discounts.slice(j, j + PG_BATCH_SIZE);
+        if (subBatch.length === 0) continue;
+
+        const placeholders = subBatch.map((_, idx) => 
+          `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
+        ).join(",");
+        
+        const values = subBatch.flatMap(d => [
+          d.order_id,
+          d.pid,
+          d.discount || 0
+        ]);
 
         await localConnection.query(`
-          INSERT INTO temp_order_discounts VALUES ${placeholders}
-          ON DUPLICATE KEY UPDATE
-            discount = VALUES(discount)
+          INSERT INTO debug_order_discounts (order_id, pid, discount)
+          VALUES ${placeholders}
+          ON CONFLICT (order_id, pid) DO UPDATE SET
+            discount = EXCLUDED.discount
         `, values);
       }
-    }
+    };
 
-    // Get tax information in batches
-    for (let i = 0; i < orderIds.length; i += 5000) {
-      const batchIds = orderIds.slice(i, i + 5000);
+    const processTaxesBatch = async (batchIds) => {
+      // Optimized tax query to avoid subquery
       const [taxes] = await prodConnection.query(`
-        SELECT DISTINCT
-          oti.order_id,
-          otip.pid,
-          otip.item_taxes_to_collect as tax
-        FROM order_tax_info oti
-        JOIN (
-          SELECT order_id, MAX(stamp) as max_stamp
+        SELECT oti.order_id, otip.pid, otip.item_taxes_to_collect as tax
+        FROM (
+          SELECT order_id, MAX(taxinfo_id) as latest_taxinfo_id
           FROM order_tax_info
           WHERE order_id IN (?)
           GROUP BY order_id
-        ) latest ON oti.order_id = latest.order_id AND oti.stamp = latest.max_stamp
+        ) latest_info
+        JOIN order_tax_info oti ON oti.order_id = latest_info.order_id 
+          AND oti.taxinfo_id = latest_info.latest_taxinfo_id
         JOIN order_tax_info_products otip ON oti.taxinfo_id = otip.taxinfo_id
       `, [batchIds]);
 
-      if (taxes.length > 0) {
-        // Remove any duplicates before inserting
-        const uniqueTaxes = new Map();
-        taxes.forEach(t => {
-          const key = `${t.order_id}-${t.pid}`;
-          uniqueTaxes.set(key, t);
-        });
+      if (taxes.length === 0) return;
 
-        const values = Array.from(uniqueTaxes.values()).flatMap(t => [t.order_id, t.pid, t.tax]);
-        if (values.length > 0) {
-          const placeholders = Array(uniqueTaxes.size).fill("(?, ?, ?)").join(",");
-          await localConnection.query(`
-            INSERT INTO temp_order_taxes VALUES ${placeholders}
-            ON DUPLICATE KEY UPDATE tax = VALUES(tax)
-          `, values);
-        }
-      }
-    }
+      for (let j = 0; j < taxes.length; j += PG_BATCH_SIZE) {
+        const subBatch = taxes.slice(j, j + PG_BATCH_SIZE);
+        if (subBatch.length === 0) continue;
 
-    // Get costeach values in batches
-    for (let i = 0; i < orderIds.length; i += 5000) {
-      const batchIds = orderIds.slice(i, i + 5000);
-      const [costs] = await prodConnection.query(`
-        SELECT orderid as order_id, pid, costeach
-        FROM order_costs
-        WHERE orderid IN (?)
-      `, [batchIds]);
+        const placeholders = subBatch.map((_, idx) => 
+          `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
+        ).join(",");
+        
+        const values = subBatch.flatMap(t => [
+          t.order_id,
+          t.pid,
+          t.tax || 0
+        ]);
 
-      if (costs.length > 0) {
-        const placeholders = costs.map(() => '(?, ?, ?)').join(",");
-        const values = costs.flatMap(c => [c.order_id, c.pid, c.costeach]);
         await localConnection.query(`
-          INSERT INTO temp_order_costs (order_id, pid, costeach)
+          INSERT INTO debug_order_taxes (order_id, pid, tax)
           VALUES ${placeholders}
-          ON DUPLICATE KEY UPDATE costeach = VALUES(costeach)
+          ON CONFLICT (order_id, pid) DO UPDATE SET
+            tax = EXCLUDED.tax
         `, values);
       }
-    }
+    };
 
-    // Now combine all the data and insert into orders table
-    // Pre-check all products at once instead of per batch
-    const allOrderPids = [...new Set(orderItems.map(item => item.pid))];
-    const [existingProducts] = allOrderPids.length > 0 ? await localConnection.query(
-      "SELECT pid FROM products WHERE pid IN (?)",
-      [allOrderPids]
-    ) : [[]];
-    const existingPids = new Set(existingProducts.map(p => p.pid));
-
-    // Process in larger batches
-    for (let i = 0; i < orderIds.length; i += 5000) {
-      const batchIds = orderIds.slice(i, i + 5000);
-
-      // Get combined data for this batch
-      const [orders] = await localConnection.query(`
+    const processCostsBatch = async (batchIds) => {
+      const [costs] = await prodConnection.query(`
         SELECT 
-          oi.order_id as order_number,
-          oi.pid,
-          oi.SKU,
-          om.date,
-          oi.price,
-          oi.quantity,
-          oi.base_discount + COALESCE(od.discount, 0) as discount,
-          COALESCE(ot.tax, 0) as tax,
-          0 as tax_included,
-          0 as shipping,
-          om.customer,
-          om.customer_name,
-          om.status,
-          om.canceled,
-          COALESCE(tc.costeach, 0) as costeach
-        FROM temp_order_items oi
-        JOIN temp_order_meta om ON oi.order_id = om.order_id
-        LEFT JOIN temp_order_discounts od ON oi.order_id = od.order_id AND oi.pid = od.pid
-        LEFT JOIN temp_order_taxes ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
-        LEFT JOIN temp_order_costs tc ON oi.order_id = tc.order_id AND oi.pid = tc.pid
-        WHERE oi.order_id IN (?)
+          oc.orderid as order_id,
+          oc.pid,
+          oc.costeach
+        FROM order_costs oc
+        WHERE oc.orderid IN (?)
+          AND oc.pending = 0
       `, [batchIds]);
 
-      // Filter orders and track missing products - do this in a single pass
-      const validOrders = [];
-      const values = [];
-      const processedOrderItems = new Set(); // Track unique order items
-      const processedOrders = new Set(); // Track unique orders
-      
-      for (const order of orders) {
-        if (!existingPids.has(order.pid)) {
-          missingProducts.add(order.pid);
-          skippedOrders.add(order.order_number);
-          continue;
-        }
-        validOrders.push(order);
-        values.push(...columnNames.map(col => order[col] ?? null));
-        processedOrderItems.add(`${order.order_number}-${order.pid}`);
-        processedOrders.add(order.order_number);
-      }
+      if (costs.length === 0) return;
 
-      if (validOrders.length > 0) {
-        // Pre-compute the placeholders string once
-        const singlePlaceholder = `(${columnNames.map(() => "?").join(",")})`;
-        const placeholders = Array(validOrders.length).fill(singlePlaceholder).join(",");
+      for (let j = 0; j < costs.length; j += PG_BATCH_SIZE) {
+        const subBatch = costs.slice(j, j + PG_BATCH_SIZE);
+        if (subBatch.length === 0) continue;
+
+        const placeholders = subBatch.map((_, idx) => 
+          `($${idx * 3 + 1}, $${idx * 3 + 2}, $${idx * 3 + 3})`
+        ).join(",");
         
-        const result = await localConnection.query(`
-          INSERT INTO orders (${columnNames.join(",")})
+        const values = subBatch.flatMap(c => [
+          c.order_id,
+          c.pid,
+          c.costeach || 0
+        ]);
+
+        await localConnection.query(`
+          INSERT INTO debug_order_costs (order_id, pid, costeach)
           VALUES ${placeholders}
-          ON DUPLICATE KEY UPDATE
-            SKU = VALUES(SKU),
-            date = VALUES(date),
-            price = VALUES(price),
-            quantity = VALUES(quantity),
-            discount = VALUES(discount),
-            tax = VALUES(tax),
-            tax_included = VALUES(tax_included),
-            shipping = VALUES(shipping),
-            customer = VALUES(customer),
-            customer_name = VALUES(customer_name),
-            status = VALUES(status),
-            canceled = VALUES(canceled),
-            costeach = VALUES(costeach)
-        `, validOrders.map(o => columnNames.map(col => o[col] ?? null)).flat());
-        
-        const affectedRows = result[0].affectedRows;
-        const updates = Math.floor(affectedRows / 2);
-        const inserts = affectedRows - (updates * 2);
-        
-        recordsAdded += inserts;
-        recordsUpdated += updates;
-        importedCount += processedOrderItems.size; // Count unique order items processed
+          ON CONFLICT (order_id, pid) DO UPDATE SET
+            costeach = EXCLUDED.costeach
+        `, values);
       }
+    };
 
-      // Update progress based on unique orders processed
-      cumulativeProcessedOrders += processedOrders.size;
+    // Process all data types in parallel for each batch
+    for (let i = 0; i < orderIds.length; i += METADATA_BATCH_SIZE) {
+      const batchIds = orderIds.slice(i, i + METADATA_BATCH_SIZE);
+      
+      await Promise.all([
+        processMetadataBatch(batchIds),
+        processDiscountsBatch(batchIds),
+        processTaxesBatch(batchIds),
+        processCostsBatch(batchIds)
+      ]);
+
+      processedCount = i + batchIds.length;
       outputProgress({
         status: "running",
         operation: "Orders import",
-        message: `Imported ${importedCount} order items (${cumulativeProcessedOrders} of ${totalUniqueOrders} orders processed)`,
-        current: cumulativeProcessedOrders,
+        message: `Loading order data: ${processedCount} of ${totalUniqueOrders}`,
+        current: processedCount,
         total: totalUniqueOrders,
         elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
-        remaining: estimateRemaining(startTime, cumulativeProcessedOrders, totalUniqueOrders),
-        rate: calculateRate(startTime, cumulativeProcessedOrders)
+        remaining: estimateRemaining(startTime, processedCount, totalUniqueOrders),
+        rate: calculateRate(startTime, processedCount)
       });
     }
 
-    // Now try to import any orders that were skipped due to missing products
-    if (skippedOrders.size > 0) {
-      try {
-        outputProgress({
-          status: "running",
-          operation: "Orders import",
-          message: `Retrying import of ${skippedOrders.size} orders with previously missing products`,
-        });
+    // Pre-check all products at once
+    const allOrderPids = [...new Set(orderItems.map(item => item.prod_pid))];
+    console.log('Orders: Checking', allOrderPids.length, 'unique products');
+    
+    const [existingProducts] = allOrderPids.length > 0 ? await localConnection.query(
+      "SELECT pid FROM products WHERE pid = ANY($1::bigint[])",
+      [allOrderPids]
+    ) : [[]];
+    
+    const existingPids = new Set(existingProducts.rows.map(p => p.pid));
+    
+    // Process in smaller batches
+    for (let i = 0; i < orderIds.length; i += 1000) {
+      const batchIds = orderIds.slice(i, i + 1000);
 
-        // Get the orders that were skipped
-        const [skippedProdOrders] = await localConnection.query(`
-          SELECT DISTINCT
+      // Get combined data for this batch in sub-batches
+      const PG_BATCH_SIZE = 100; // Process 100 records at a time
+      for (let j = 0; j < batchIds.length; j += PG_BATCH_SIZE) {
+        const subBatchIds = batchIds.slice(j, j + PG_BATCH_SIZE);
+        
+        const [orders] = await localConnection.query(`
+          WITH order_totals AS (
+            SELECT 
+              oi.order_id,
+              oi.pid,
+              SUM(COALESCE(od.discount, 0)) as promo_discount,
+              COALESCE(ot.tax, 0) as total_tax,
+              COALESCE(oi.price * 0.5, 0) as costeach
+            FROM debug_order_items oi
+            LEFT JOIN debug_order_discounts od ON oi.order_id = od.order_id AND oi.pid = od.pid
+            LEFT JOIN debug_order_taxes ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
+            GROUP BY oi.order_id, oi.pid, ot.tax
+          )
+          SELECT 
             oi.order_id as order_number,
-            oi.pid,
-            oi.SKU,
+            oi.pid::bigint as pid,
+            oi.SKU as sku,
             om.date,
             oi.price,
             oi.quantity,
-            oi.base_discount + COALESCE(od.discount, 0) as discount,
-            COALESCE(ot.tax, 0) as tax,
-            0 as tax_included,
+            (oi.base_discount + 
+             COALESCE(ot.promo_discount, 0) + 
+             CASE 
+              WHEN om.summary_discount > 0 AND om.summary_subtotal > 0 THEN 
+                ROUND((om.summary_discount * (oi.price * oi.quantity)) / NULLIF(om.summary_subtotal, 0), 2)
+              ELSE 0 
+             END)::DECIMAL(10,2) as discount,
+            COALESCE(ot.total_tax, 0)::DECIMAL(10,2) as tax,
+            false as tax_included,
             0 as shipping,
             om.customer,
             om.customer_name,
             om.status,
             om.canceled,
-            COALESCE(tc.costeach, 0) as costeach
-          FROM temp_order_items oi
-          JOIN temp_order_meta om ON oi.order_id = om.order_id
-          LEFT JOIN temp_order_discounts od ON oi.order_id = od.order_id AND oi.pid = od.pid
-          LEFT JOIN temp_order_taxes ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
-          LEFT JOIN temp_order_costs tc ON oi.order_id = tc.order_id AND oi.pid = tc.pid
-          WHERE oi.order_id IN (?)
-        `, [Array.from(skippedOrders)]);
+            COALESCE(ot.costeach, oi.price * 0.5)::DECIMAL(10,3) as costeach
+          FROM (
+            SELECT DISTINCT ON (order_id, pid)
+              order_id, pid, SKU, price, quantity, base_discount
+            FROM debug_order_items
+            WHERE order_id = ANY($1)
+            ORDER BY order_id, pid
+          ) oi
+          JOIN debug_order_meta om ON oi.order_id = om.order_id
+          LEFT JOIN order_totals ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
+          ORDER BY oi.order_id, oi.pid
+        `, [subBatchIds]);
 
-        // Check which products exist now
-        const skippedPids = [...new Set(skippedProdOrders.map(o => o.pid))];
-        const [existingProducts] = skippedPids.length > 0 ? await localConnection.query(
-          "SELECT pid FROM products WHERE pid IN (?)",
-          [skippedPids]
-        ) : [[]];
-        const existingPids = new Set(existingProducts.map(p => p.pid));
-
-        // Filter orders that can now be imported
-        const validOrders = skippedProdOrders.filter(order => existingPids.has(order.pid));
-        const retryOrderItems = new Set(); // Track unique order items in retry
-
-        if (validOrders.length > 0) {
-          const placeholders = validOrders.map(() => `(${columnNames.map(() => "?").join(", ")})`).join(",");
-          const values = validOrders.map(o => columnNames.map(col => o[col] ?? null)).flat();
-
-          const result = await localConnection.query(`
-            INSERT INTO orders (${columnNames.join(", ")})
-            VALUES ${placeholders}
-            ON DUPLICATE KEY UPDATE
-              SKU = VALUES(SKU),
-              date = VALUES(date),
-              price = VALUES(price),
-              quantity = VALUES(quantity),
-              discount = VALUES(discount),
-              tax = VALUES(tax),
-              tax_included = VALUES(tax_included),
-              shipping = VALUES(shipping),
-              customer = VALUES(customer),
-              customer_name = VALUES(customer_name),
-              status = VALUES(status),
-              canceled = VALUES(canceled),
-              costeach = VALUES(costeach)
-          `, values);
-
-          const affectedRows = result[0].affectedRows;
-          const updates = Math.floor(affectedRows / 2);
-          const inserts = affectedRows - (updates * 2);
-          
-          // Track unique order items
-          validOrders.forEach(order => {
-            retryOrderItems.add(`${order.order_number}-${order.pid}`);
-          });
-          
-          outputProgress({
-            status: "running",
-            operation: "Orders import",
-            message: `Successfully imported ${retryOrderItems.size} previously skipped order items`,
-          });
-
-          // Update the main counters
-          recordsAdded += inserts;
-          recordsUpdated += updates;
-          importedCount += retryOrderItems.size;
+        // Filter orders and track missing products
+        const validOrders = [];
+        const processedOrderItems = new Set();
+        const processedOrders = new Set();
+        
+        for (const order of orders.rows) {
+          if (!existingPids.has(order.pid)) {
+            missingProducts.add(order.pid);
+            skippedOrders.add(order.order_number);
+            continue;
+          }
+          validOrders.push(order);
+          processedOrderItems.add(`${order.order_number}-${order.pid}`);
+          processedOrders.add(order.order_number);
         }
-      } catch (error) {
-        console.warn('Warning: Failed to retry skipped orders:', error.message);
-        console.warn(`Skipped ${skippedOrders.size} orders due to ${missingProducts.size} missing products`);
+
+        // Process valid orders in smaller sub-batches
+        const FINAL_BATCH_SIZE = 50;
+        for (let k = 0; k < validOrders.length; k += FINAL_BATCH_SIZE) {
+          const subBatch = validOrders.slice(k, k + FINAL_BATCH_SIZE);
+          
+          const placeholders = subBatch.map((_, idx) => {
+            const base = idx * 14;  // 14 columns (removed updated)
+            return `($${base + 1}, $${base + 2}, $${base + 3}, $${base + 4}, $${base + 5}, $${base + 6}, $${base + 7}, $${base + 8}, $${base + 9}, $${base + 10}, $${base + 11}, $${base + 12}, $${base + 13}, $${base + 14})`;
+          }).join(',');
+
+          const batchValues = subBatch.flatMap(o => [
+            o.order_number,
+            o.pid,
+            o.sku || 'NO-SKU',
+            o.date,
+            o.price,
+            o.quantity,
+            o.discount,
+            o.tax,
+            o.tax_included,
+            o.shipping,
+            o.customer,
+            o.customer_name,
+            o.status,
+            o.canceled
+          ]);
+
+          const [result] = await localConnection.query(`
+            WITH inserted_orders AS (
+              INSERT INTO orders (
+                order_number, pid, sku, date, price, quantity, discount,
+                tax, tax_included, shipping, customer, customer_name,
+                status, canceled
+              )
+              VALUES ${placeholders}
+              ON CONFLICT (order_number, pid) DO UPDATE SET
+                sku = EXCLUDED.sku,
+                date = EXCLUDED.date,
+                price = EXCLUDED.price,
+                quantity = EXCLUDED.quantity,
+                discount = EXCLUDED.discount,
+                tax = EXCLUDED.tax,
+                tax_included = EXCLUDED.tax_included,
+                shipping = EXCLUDED.shipping,
+                customer = EXCLUDED.customer,
+                customer_name = EXCLUDED.customer_name,
+                status = EXCLUDED.status,
+                canceled = EXCLUDED.canceled
+              RETURNING xmax = 0 as inserted
+            )
+            SELECT 
+              COUNT(*) FILTER (WHERE inserted) as inserted,
+              COUNT(*) FILTER (WHERE NOT inserted) as updated
+            FROM inserted_orders
+          `, batchValues);
+          
+          const { inserted, updated } = result.rows[0];
+          recordsAdded += inserted;
+          recordsUpdated += updated;
+          importedCount += subBatch.length;
+        }
+
+        cumulativeProcessedOrders += processedOrders.size;
+        outputProgress({
+          status: "running",
+          operation: "Orders import",
+          message: `Importing orders: ${cumulativeProcessedOrders} of ${totalUniqueOrders}`,
+          current: cumulativeProcessedOrders,
+          total: totalUniqueOrders,
+          elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
+          remaining: estimateRemaining(startTime, cumulativeProcessedOrders, totalUniqueOrders),
+          rate: calculateRate(startTime, cumulativeProcessedOrders)
+        });
       }
     }
 
-    // Clean up temporary tables after ALL processing is complete
-    await localConnection.query(`
-      DROP TEMPORARY TABLE IF EXISTS temp_order_items;
-      DROP TEMPORARY TABLE IF EXISTS temp_order_meta;
-      DROP TEMPORARY TABLE IF EXISTS temp_order_discounts;
-      DROP TEMPORARY TABLE IF EXISTS temp_order_taxes;
-      DROP TEMPORARY TABLE IF EXISTS temp_order_costs;
-    `);
-
-    // Only update sync status if we get here (no errors thrown)
+    // Update sync status
     await localConnection.query(`
       INSERT INTO sync_status (table_name, last_sync_timestamp)
       VALUES ('orders', NOW())
-      ON DUPLICATE KEY UPDATE last_sync_timestamp = NOW()
+      ON CONFLICT (table_name) DO UPDATE SET
+        last_sync_timestamp = NOW()
     `);
 
     return {

inventory-server/scripts/import/purchase-orders.js

@@ -10,22 +10,42 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
     const [syncInfo] = await localConnection.query(
       "SELECT last_sync_timestamp FROM sync_status WHERE table_name = 'purchase_orders'"
     );
-    const lastSyncTime = syncInfo?.[0]?.last_sync_timestamp || '1970-01-01';
+    const lastSyncTime = syncInfo?.rows?.[0]?.last_sync_timestamp || '1970-01-01';
 
     console.log('Purchase Orders: Using last sync time:', lastSyncTime);
 
-    // Insert temporary table creation query for purchase orders
+    // Create temporary tables with PostgreSQL syntax
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_purchase_orders (
-        po_id INT UNSIGNED NOT NULL,
-        pid INT UNSIGNED NOT NULL,
+      DROP TABLE IF EXISTS temp_purchase_orders;
+      DROP TABLE IF EXISTS temp_po_receivings;
+
+      CREATE TEMP TABLE temp_purchase_orders (
+        po_id INTEGER NOT NULL,
+        pid INTEGER NOT NULL,
+        sku VARCHAR(50),
+        name VARCHAR(255),
         vendor VARCHAR(255),
-        date DATE,
-        expected_date DATE,
-        status INT,
+        date TIMESTAMP WITH TIME ZONE,
+        expected_date TIMESTAMP WITH TIME ZONE,
+        status INTEGER,
         notes TEXT,
+        ordered INTEGER,
+        cost_price DECIMAL(10,3),
         PRIMARY KEY (po_id, pid)
-      ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
+      );
+
+      CREATE TEMP TABLE temp_po_receivings (
+        po_id INTEGER,
+        pid INTEGER NOT NULL,
+        receiving_id INTEGER NOT NULL,
+        qty_each INTEGER,
+        cost_each DECIMAL(10,3),
+        received_date TIMESTAMP WITH TIME ZONE,
+        received_by INTEGER,
+        received_by_name VARCHAR(255),
+        is_alt_po INTEGER,
+        PRIMARY KEY (receiving_id, pid)
+      );
     `);
 
     outputProgress({
@@ -33,16 +53,15 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
       status: "running",
     });
 
-    // Get column names for the insert
-    const [columns] = await localConnection.query(`
+    // Get column names - Keep MySQL compatible for production
+    const [columns] = await prodConnection.query(`
       SELECT COLUMN_NAME 
       FROM INFORMATION_SCHEMA.COLUMNS 
       WHERE TABLE_NAME = 'purchase_orders'
+        AND COLUMN_NAME != 'updated'  -- Exclude the updated column
       ORDER BY ORDINAL_POSITION
     `);
-    const columnNames = columns
-      .map((col) => col.COLUMN_NAME)
-      .filter((name) => name !== "id");
+    const columnNames = columns.map(col => col.COLUMN_NAME);
 
     // Build incremental conditions
     const incrementalWhereClause = incrementalUpdate
@@ -61,7 +80,7 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
       ? [lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime, lastSyncTime]
       : [];
 
-    // First get all relevant PO IDs with basic info
+    // First get all relevant PO IDs with basic info - Keep MySQL compatible for production
     const [[{ total }]] = await prodConnection.query(`
       SELECT COUNT(*) as total
       FROM (
@@ -100,6 +119,7 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
 
     console.log('Purchase Orders: Found changes:', total);
 
+    // Get PO list - Keep MySQL compatible for production
     const [poList] = await prodConnection.query(`
       SELECT DISTINCT 
         COALESCE(p.po_id, r.receiving_id) as po_id,
@@ -110,12 +130,12 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
         ) as vendor,
         CASE 
           WHEN p.po_id IS NOT NULL THEN 
-            DATE(COALESCE(
+            COALESCE(
               NULLIF(p.date_ordered, '0000-00-00 00:00:00'),
               p.date_created
-            ))
+            )
           WHEN r.receiving_id IS NOT NULL THEN
-            DATE(r.date_created)
+            r.date_created
         END as date,
         CASE 
           WHEN p.date_estin = '0000-00-00' THEN NULL
@@ -186,14 +206,14 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
       const batch = poList.slice(i, Math.min(i + BATCH_SIZE, poList.length));
       const poIds = batch.map(po => po.po_id);
 
-      // Get all products for these POs in one query
+      // Get all products for these POs in one query - Keep MySQL compatible for production
       const [poProducts] = await prodConnection.query(`
         SELECT 
           pop.po_id,
           pop.pid,
           pr.itemnumber as sku,
           pr.description as name,
-          pop.cost_each,
+          pop.cost_each as cost_price,
           pop.qty_each as ordered
         FROM po_products pop
         USE INDEX (PRIMARY)
@@ -233,311 +253,397 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
           ORDER BY r.po_id, rp.pid, rp.received_date
         `, [batchPoIds, productPids]);
 
-        // Create maps for this sub-batch
-        const poProductMap = new Map();
-        productBatch.forEach(product => {
-          const key = `${product.po_id}-${product.pid}`;
-          poProductMap.set(key, product);
-        });
-
-        const receivingMap = new Map();
-        const altReceivingMap = new Map();
-        const noPOReceivingMap = new Map();
-        
-        receivings.forEach(receiving => {
-          const key = `${receiving.po_id}-${receiving.pid}`;
-          if (receiving.is_alt_po === 2) {
-            // No PO
-            if (!noPOReceivingMap.has(receiving.pid)) {
-              noPOReceivingMap.set(receiving.pid, []);
-            }
-            noPOReceivingMap.get(receiving.pid).push(receiving);
-          } else if (receiving.is_alt_po === 1) {
-            // Different PO
-            if (!altReceivingMap.has(receiving.pid)) {
-              altReceivingMap.set(receiving.pid, []);
-            }
-            altReceivingMap.get(receiving.pid).push(receiving);
-          } else {
-            // Original PO
-            if (!receivingMap.has(key)) {
-              receivingMap.set(key, []);
-            }
-            receivingMap.get(key).push(receiving);
-          }
-        });
-
-        // Verify PIDs exist
-        const [existingPids] = await localConnection.query(
-          'SELECT pid FROM products WHERE pid IN (?)',
-          [productPids]
-        );
-        const validPids = new Set(existingPids.map(p => p.pid));
-
-        // First check which PO lines already exist and get their current values
-        const poLines = Array.from(poProductMap.values())
-          .filter(p => validPids.has(p.pid))
-          .map(p => [p.po_id, p.pid]);
-
-        const [existingPOs] = await localConnection.query(
-          `SELECT ${columnNames.join(',')} FROM purchase_orders WHERE (po_id, pid) IN (${poLines.map(() => "(?,?)").join(",")})`,
-          poLines.flat()
-        );
-        const existingPOMap = new Map(
-          existingPOs.map(po => [`${po.po_id}-${po.pid}`, po])
-        );
-
-        // Split into inserts and updates
-        const insertsAndUpdates = { inserts: [], updates: [] };
-        let batchProcessed = 0;
-
-        for (const po of batch) {
-          const poProducts = Array.from(poProductMap.values())
-            .filter(p => p.po_id === po.po_id && validPids.has(p.pid));
-
-          for (const product of poProducts) {
-            const key = `${po.po_id}-${product.pid}`;
-            const receivingHistory = receivingMap.get(key) || [];
-            const altReceivingHistory = altReceivingMap.get(product.pid) || [];
-            const noPOReceivingHistory = noPOReceivingMap.get(product.pid) || [];
+        // Insert receivings into temp table
+        if (receivings.length > 0) {
+          // Process in smaller chunks to avoid parameter limits
+          const CHUNK_SIZE = 100; // Reduce chunk size to avoid parameter limits
+          for (let i = 0; i < receivings.length; i += CHUNK_SIZE) {
+            const chunk = receivings.slice(i, Math.min(i + CHUNK_SIZE, receivings.length));
             
-            // Combine all receivings and sort by date
-            const allReceivings = [
-              ...receivingHistory.map(r => ({ ...r, type: 'original' })),
-              ...altReceivingHistory.map(r => ({ ...r, type: 'alternate' })),
-              ...noPOReceivingHistory.map(r => ({ ...r, type: 'no_po' }))
-            ].sort((a, b) => new Date(a.received_date || '9999-12-31') - new Date(b.received_date || '9999-12-31'));
-
-            // Split receivings into original PO and others
-            const originalPOReceivings = allReceivings.filter(r => r.type === 'original');
-            const otherReceivings = allReceivings.filter(r => r.type !== 'original');
-
-            // Track FIFO fulfillment
-            let remainingToFulfill = product.ordered;
-            const fulfillmentTracking = [];
-            let totalReceived = 0;
-            let actualCost = null; // Will store the cost of the first receiving that fulfills this PO
-            let firstFulfillmentReceiving = null;
-            let lastFulfillmentReceiving = null;
-
-            for (const receiving of allReceivings) {
-              const qtyToApply = Math.min(remainingToFulfill, receiving.qty_each);
-              if (qtyToApply > 0) {
-                // If this is the first receiving being applied, use its cost
-                if (actualCost === null) {
-                  actualCost = receiving.cost_each;
-                  firstFulfillmentReceiving = receiving;
-                }
-                lastFulfillmentReceiving = receiving;
-                fulfillmentTracking.push({
-                  receiving_id: receiving.receiving_id,
-                  qty_applied: qtyToApply,
-                  qty_total: receiving.qty_each,
-                  cost: receiving.cost_each,
-                  date: receiving.received_date,
-                  received_by: receiving.received_by,
-                  received_by_name: receiving.received_by_name || 'Unknown',
-                  type: receiving.type,
-                  remaining_qty: receiving.qty_each - qtyToApply
-                });
-                remainingToFulfill -= qtyToApply;
-              } else {
-                // Track excess receivings
-                fulfillmentTracking.push({
-                  receiving_id: receiving.receiving_id,
-                  qty_applied: 0,
-                  qty_total: receiving.qty_each,
-                  cost: receiving.cost_each,
-                  date: receiving.received_date,
-                  received_by: receiving.received_by,
-                  received_by_name: receiving.received_by_name || 'Unknown',
-                  type: receiving.type,
-                  is_excess: true
-                });
-              }
-              totalReceived += receiving.qty_each;
-            }
-
-            const receiving_status = !totalReceived ? 1 : // created
-                                   remainingToFulfill > 0 ? 30 : // partial
-                                   40; // full
-
-            function formatDate(dateStr) {
-              if (!dateStr) return null;
-              if (dateStr === '0000-00-00' || dateStr === '0000-00-00 00:00:00') return null;
-              if (typeof dateStr === 'string' && !dateStr.match(/^\d{4}-\d{2}-\d{2}/)) return null;
-              try {
-                const date = new Date(dateStr);
-                if (isNaN(date.getTime())) return null;
-                if (date.getFullYear() < 1900 || date.getFullYear() > 2100) return null;
-                return date.toISOString().split('T')[0];
-              } catch (e) {
-                return null;
-              }
-            }
-
-            const rowValues = columnNames.map(col => {
-              switch (col) {
-                case 'po_id': return po.po_id;
-                case 'vendor': return po.vendor;
-                case 'date': return formatDate(po.date);
-                case 'expected_date': return formatDate(po.expected_date);
-                case 'pid': return product.pid;
-                case 'sku': return product.sku;
-                case 'name': return product.name;
-                case 'cost_price': return actualCost || product.cost_each;
-                case 'po_cost_price': return product.cost_each;
-                case 'status': return po.status;
-                case 'notes': return po.notes;
-                case 'long_note': return po.long_note;
-                case 'ordered': return product.ordered;
-                case 'received': return totalReceived;
-                case 'unfulfilled': return remainingToFulfill;
-                case 'excess_received': return Math.max(0, totalReceived - product.ordered);
-                case 'received_date': return formatDate(firstFulfillmentReceiving?.received_date);
-                case 'last_received_date': return formatDate(lastFulfillmentReceiving?.received_date);
-                case 'received_by': return firstFulfillmentReceiving?.received_by_name || null;
-                case 'receiving_status': return receiving_status;
-                case 'receiving_history': return JSON.stringify({
-                  fulfillment: fulfillmentTracking,
-                  ordered_qty: product.ordered,
-                  total_received: totalReceived,
-                  remaining_unfulfilled: remainingToFulfill,
-                  excess_received: Math.max(0, totalReceived - product.ordered),
-                  po_cost: product.cost_each,
-                  actual_cost: actualCost || product.cost_each
-                });
-                default: return null;
-              }
+            const values = [];
+            const placeholders = [];
+            
+            chunk.forEach((r, idx) => {
+              values.push(
+                r.po_id,
+                r.pid,
+                r.receiving_id,
+                r.qty_each,
+                r.cost_each,
+                r.received_date,
+                r.received_by,
+                r.received_by_name || null,
+                r.is_alt_po
+              );
+              
+              const offset = idx * 9;
+              placeholders.push(`($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9})`);
             });
 
-            if (existingPOMap.has(key)) {
-              const existing = existingPOMap.get(key);
-              // Check if any values are different
-              const hasChanges = columnNames.some(col => {
-                const newVal = rowValues[columnNames.indexOf(col)];
-                const oldVal = existing[col] ?? null;
-                // Special handling for numbers to avoid type coercion issues
-                if (typeof newVal === 'number' && typeof oldVal === 'number') {
-                  return Math.abs(newVal - oldVal) > 0.00001; // Allow for tiny floating point differences
-                }
-                // Special handling for receiving_history - parse and compare
-                if (col === 'receiving_history') {
-                  const newHistory = JSON.parse(newVal || '{}');
-                  const oldHistory = JSON.parse(oldVal || '{}');
-                  return JSON.stringify(newHistory) !== JSON.stringify(oldHistory);
-                }
-                return newVal !== oldVal;
-              });
-
-              if (hasChanges) {
-                insertsAndUpdates.updates.push({
-                  po_id: po.po_id,
-                  pid: product.pid,
-                  values: rowValues
-                });
-              }
-            } else {
-              insertsAndUpdates.inserts.push({
-                po_id: po.po_id,
-                pid: product.pid,
-                values: rowValues
-              });
-            }
-            batchProcessed++;
+            await localConnection.query(`
+              INSERT INTO temp_po_receivings (
+                po_id, pid, receiving_id, qty_each, cost_each, received_date,
+                received_by, received_by_name, is_alt_po
+              )
+              VALUES ${placeholders.join(',')}
+              ON CONFLICT (receiving_id, pid) DO UPDATE SET
+                po_id = EXCLUDED.po_id,
+                qty_each = EXCLUDED.qty_each,
+                cost_each = EXCLUDED.cost_each,
+                received_date = EXCLUDED.received_date,
+                received_by = EXCLUDED.received_by,
+                received_by_name = EXCLUDED.received_by_name,
+                is_alt_po = EXCLUDED.is_alt_po
+            `, values);
           }
         }
 
-        // Handle inserts
-        if (insertsAndUpdates.inserts.length > 0) {
-          const insertPlaceholders = insertsAndUpdates.inserts
-            .map(() => `(${Array(columnNames.length).fill("?").join(",")})`)
-            .join(",");
-
-          const insertResult = await localConnection.query(`
-            INSERT INTO purchase_orders (${columnNames.join(",")})
-            VALUES ${insertPlaceholders}
-          `, insertsAndUpdates.inserts.map(i => i.values).flat());
-
-          const affectedRows = insertResult[0].affectedRows;
-          // For an upsert, MySQL counts rows twice for updates
-          // So if affectedRows is odd, we have (updates * 2 + inserts)
-          const updates = Math.floor(affectedRows / 2);
-          const inserts = affectedRows - (updates * 2);
+        // Process each PO product in chunks
+        const PRODUCT_CHUNK_SIZE = 100;
+        for (let i = 0; i < productBatch.length; i += PRODUCT_CHUNK_SIZE) {
+          const chunk = productBatch.slice(i, Math.min(i + PRODUCT_CHUNK_SIZE, productBatch.length));
+          const values = [];
+          const placeholders = [];
           
-          recordsAdded += inserts;
-          recordsUpdated += Math.floor(updates); // Ensure we never have fractional updates
-          processed += batchProcessed;
-        }
-
-        // Handle updates - now we know these actually have changes
-        if (insertsAndUpdates.updates.length > 0) {
-          const updatePlaceholders = insertsAndUpdates.updates
-            .map(() => `(${Array(columnNames.length).fill("?").join(",")})`)
-            .join(",");
-
-          const updateResult = await localConnection.query(`
-            INSERT INTO purchase_orders (${columnNames.join(",")})
-            VALUES ${updatePlaceholders}
-            ON DUPLICATE KEY UPDATE ${columnNames
-              .filter((col) => col !== "po_id" && col !== "pid")
-              .map((col) => `${col} = VALUES(${col})`)
-              .join(",")};
-          `, insertsAndUpdates.updates.map(u => u.values).flat());
-
-          const affectedRows = updateResult[0].affectedRows;
-          // For an upsert, MySQL counts rows twice for updates
-          // So if affectedRows is odd, we have (updates * 2 + inserts)
-          const updates = Math.floor(affectedRows / 2);
-          const inserts = affectedRows - (updates * 2);
-          
-          recordsUpdated += Math.floor(updates); // Ensure we never have fractional updates
-          processed += batchProcessed;
-        }
-        
-        // Update progress based on time interval
-        const now = Date.now();
-        if (now - lastProgressUpdate >= PROGRESS_INTERVAL || processed === totalItems) {
-          outputProgress({
-            status: "running",
-            operation: "Purchase orders import",
-            current: processed,
-            total: totalItems,
-            elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
-            remaining: estimateRemaining(startTime, processed, totalItems),
-            rate: calculateRate(startTime, processed)
+          chunk.forEach((product, idx) => {
+            const po = batch.find(p => p.po_id === product.po_id);
+            if (!po) return;
+            
+            values.push(
+              product.po_id,
+              product.pid,
+              product.sku,
+              product.name,
+              po.vendor,
+              po.date,
+              po.expected_date,
+              po.status,
+              po.notes || po.long_note,
+              product.ordered,
+              product.cost_price
+            );
+            
+            const offset = idx * 11;  // Updated to match 11 fields
+            placeholders.push(`($${offset + 1}, $${offset + 2}, $${offset + 3}, $${offset + 4}, $${offset + 5}, $${offset + 6}, $${offset + 7}, $${offset + 8}, $${offset + 9}, $${offset + 10}, $${offset + 11})`);
           });
-          lastProgressUpdate = now;
+
+          if (placeholders.length > 0) {
+            await localConnection.query(`
+              INSERT INTO temp_purchase_orders (
+                po_id, pid, sku, name, vendor, date, expected_date, 
+                status, notes, ordered, cost_price
+              )
+              VALUES ${placeholders.join(',')}
+              ON CONFLICT (po_id, pid) DO UPDATE SET
+                sku = EXCLUDED.sku,
+                name = EXCLUDED.name,
+                vendor = EXCLUDED.vendor,
+                date = EXCLUDED.date,
+                expected_date = EXCLUDED.expected_date,
+                status = EXCLUDED.status,
+                notes = EXCLUDED.notes,
+                ordered = EXCLUDED.ordered,
+                cost_price = EXCLUDED.cost_price
+            `, values);
+          }
+
+          processed += chunk.length;
+          
+          // Update progress based on time interval
+          const now = Date.now();
+          if (now - lastProgressUpdate >= PROGRESS_INTERVAL || processed === totalItems) {
+            outputProgress({
+              status: "running",
+              operation: "Purchase orders import",
+              current: processed,
+              total: totalItems,
+              elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
+              remaining: estimateRemaining(startTime, processed, totalItems),
+              rate: calculateRate(startTime, processed)
+            });
+            lastProgressUpdate = now;
+          }
         }
       }
     }
 
-    // Only update sync status if we get here (no errors thrown)
+    // Insert final data into purchase_orders table in chunks
+    const FINAL_CHUNK_SIZE = 1000;
+    let totalProcessed = 0;
+    const totalPosResult = await localConnection.query('SELECT COUNT(*) as total_pos FROM temp_purchase_orders');
+    const total_pos = parseInt(totalPosResult.rows?.[0]?.total_pos || '0', 10);
+
+    outputProgress({
+      status: "running",
+      operation: "Purchase orders final import",
+      message: `Processing ${total_pos} purchase orders for final import`,
+      current: 0,
+      total: total_pos
+    });
+
+    // Process in chunks using cursor-based pagination
+    let lastPoId = 0;
+    let lastPid = 0;
+    let recordsAdded = 0;
+    let recordsUpdated = 0;
+
+    while (true) {
+      console.log('Fetching next chunk with lastPoId:', lastPoId, 'lastPid:', lastPid);
+      const chunkResult = await localConnection.query(`
+        SELECT po_id, pid FROM temp_purchase_orders 
+        WHERE (po_id, pid) > ($1, $2)
+        ORDER BY po_id, pid
+        LIMIT $3
+      `, [lastPoId, lastPid, FINAL_CHUNK_SIZE]);
+
+      if (!chunkResult?.rows) {
+        console.error('No rows returned from chunk query:', chunkResult);
+        break;
+      }
+
+      const chunk = chunkResult.rows;
+      console.log('Got chunk of size:', chunk.length);
+      if (chunk.length === 0) break;
+
+      const result = await localConnection.query(`
+        WITH inserted_pos AS (
+          INSERT INTO purchase_orders (
+            po_id, pid, sku, name, cost_price, po_cost_price,
+            vendor, date, expected_date, status, notes,
+            ordered, received, receiving_status,
+            received_date, last_received_date, received_by,
+            receiving_history
+          )
+          SELECT 
+            po.po_id,
+            po.pid,
+            po.sku,
+            po.name,
+            COALESCE(
+              (
+                SELECT cost_each 
+                FROM temp_po_receivings r2 
+                WHERE r2.pid = po.pid 
+                  AND r2.po_id = po.po_id
+                  AND r2.is_alt_po = 0 
+                  AND r2.cost_each > 0
+                ORDER BY r2.received_date 
+                LIMIT 1
+              ),
+              po.cost_price
+            ) as cost_price,
+            po.cost_price as po_cost_price,
+            po.vendor,
+            po.date,
+            po.expected_date,
+            po.status,
+            po.notes,
+            po.ordered,
+            COALESCE(SUM(CASE WHEN r.is_alt_po = 0 THEN r.qty_each END), 0) as received,
+            CASE 
+              WHEN COUNT(r.receiving_id) = 0 THEN 1  -- created
+              WHEN SUM(CASE WHEN r.is_alt_po = 0 THEN r.qty_each END) < po.ordered THEN 30  -- partial
+              ELSE 40  -- full
+            END as receiving_status,
+            MIN(CASE WHEN r.is_alt_po = 0 THEN r.received_date END) as received_date,
+            MAX(CASE WHEN r.is_alt_po = 0 THEN r.received_date END) as last_received_date,
+            (
+              SELECT r2.received_by_name 
+              FROM temp_po_receivings r2 
+              WHERE r2.pid = po.pid 
+                AND r2.is_alt_po = 0 
+              ORDER BY r2.received_date 
+              LIMIT 1
+            ) as received_by,
+            jsonb_build_object(
+              'ordered_qty', po.ordered,
+              'total_received', COALESCE(SUM(CASE WHEN r.is_alt_po = 0 THEN r.qty_each END), 0),
+              'remaining_unfulfilled', GREATEST(0, po.ordered - COALESCE(SUM(CASE WHEN r.is_alt_po = 0 THEN r.qty_each END), 0)),
+              'excess_received', GREATEST(0, COALESCE(SUM(CASE WHEN r.is_alt_po = 0 THEN r.qty_each END), 0) - po.ordered),
+              'po_cost', po.cost_price,
+              'actual_cost', COALESCE(
+                (
+                  SELECT cost_each 
+                  FROM temp_po_receivings r2 
+                  WHERE r2.pid = po.pid 
+                    AND r2.is_alt_po = 0 
+                    AND r2.cost_each > 0
+                  ORDER BY r2.received_date 
+                  LIMIT 1
+                ),
+                po.cost_price
+              ),
+              'fulfillment', (
+                SELECT jsonb_agg(
+                  jsonb_build_object(
+                    'receiving_id', r2.receiving_id,
+                    'qty_applied', CASE 
+                      WHEN r2.running_total <= po.ordered THEN r2.qty_each
+                      WHEN r2.running_total - r2.qty_each < po.ordered THEN po.ordered - (r2.running_total - r2.qty_each)
+                      ELSE 0
+                    END,
+                    'qty_total', r2.qty_each,
+                    'cost', r2.cost_each,
+                    'date', r2.received_date,
+                    'received_by', r2.received_by,
+                    'received_by_name', r2.received_by_name,
+                    'type', CASE r2.is_alt_po 
+                      WHEN 0 THEN 'original'
+                      WHEN 1 THEN 'alternate'
+                      ELSE 'no_po'
+                    END,
+                    'remaining_qty', CASE 
+                      WHEN r2.running_total <= po.ordered THEN 0
+                      WHEN r2.running_total - r2.qty_each < po.ordered THEN r2.running_total - po.ordered
+                      ELSE r2.qty_each
+                    END,
+                    'is_excess', r2.running_total > po.ordered
+                  )
+                  ORDER BY r2.received_date
+                ) 
+                FROM (
+                  SELECT 
+                    r2.*,
+                    SUM(r2.qty_each) OVER (
+                      PARTITION BY r2.pid 
+                      ORDER BY r2.received_date
+                      ROWS UNBOUNDED PRECEDING
+                    ) as running_total
+                  FROM temp_po_receivings r2
+                  WHERE r2.pid = po.pid
+                ) r2
+              ),
+              'alternate_po_receivings', (
+                SELECT jsonb_agg(
+                  jsonb_build_object(
+                    'receiving_id', r2.receiving_id,
+                    'qty', r2.qty_each,
+                    'cost', r2.cost_each,
+                    'date', r2.received_date,
+                    'received_by', r2.received_by,
+                    'received_by_name', r2.received_by_name
+                  )
+                  ORDER BY r2.received_date
+                )
+                FROM temp_po_receivings r2
+                WHERE r2.pid = po.pid AND r2.is_alt_po = 1
+              ),
+              'no_po_receivings', (
+                SELECT jsonb_agg(
+                  jsonb_build_object(
+                    'receiving_id', r2.receiving_id,
+                    'qty', r2.qty_each,
+                    'cost', r2.cost_each,
+                    'date', r2.received_date,
+                    'received_by', r2.received_by,
+                    'received_by_name', r2.received_by_name
+                  )
+                  ORDER BY r2.received_date
+                )
+                FROM temp_po_receivings r2
+                WHERE r2.pid = po.pid AND r2.is_alt_po = 2
+              )
+            ) as receiving_history
+          FROM temp_purchase_orders po
+          LEFT JOIN temp_po_receivings r ON po.pid = r.pid
+          WHERE (po.po_id, po.pid) IN (
+            SELECT po_id, pid FROM UNNEST($1::int[], $2::int[])
+          )
+          GROUP BY po.po_id, po.pid, po.sku, po.name, po.vendor, po.date, 
+                  po.expected_date, po.status, po.notes, po.ordered, po.cost_price
+          ON CONFLICT (po_id, pid) DO UPDATE SET
+            vendor = EXCLUDED.vendor,
+            date = EXCLUDED.date,
+            expected_date = EXCLUDED.expected_date,
+            status = EXCLUDED.status,
+            notes = EXCLUDED.notes,
+            ordered = EXCLUDED.ordered,
+            received = EXCLUDED.received,
+            receiving_status = EXCLUDED.receiving_status,
+            received_date = EXCLUDED.received_date,
+            last_received_date = EXCLUDED.last_received_date,
+            received_by = EXCLUDED.received_by,
+            receiving_history = EXCLUDED.receiving_history,
+            cost_price = EXCLUDED.cost_price,
+            po_cost_price = EXCLUDED.po_cost_price
+          RETURNING xmax
+        )
+        SELECT 
+          COUNT(*) FILTER (WHERE xmax = 0) as inserted,
+          COUNT(*) FILTER (WHERE xmax <> 0) as updated
+        FROM inserted_pos
+      `, [
+        chunk.map(r => r.po_id),
+        chunk.map(r => r.pid)
+      ]);
+
+      // Add debug logging
+      console.log('Insert result:', result?.rows?.[0]);
+
+      // Handle the result properly for PostgreSQL with more defensive coding
+      const resultRow = result?.rows?.[0] || {};
+      const insertCount = parseInt(resultRow.inserted || '0', 10);
+      const updateCount = parseInt(resultRow.updated || '0', 10);
+
+      recordsAdded += insertCount;
+      recordsUpdated += updateCount;
+      totalProcessed += chunk.length;
+
+      // Update progress
+      outputProgress({
+        status: "running",
+        operation: "Purchase orders final import",
+        message: `Processed ${totalProcessed} of ${total_pos} purchase orders`,
+        current: totalProcessed,
+        total: total_pos,
+        elapsed: formatElapsedTime((Date.now() - startTime) / 1000),
+        remaining: estimateRemaining(startTime, totalProcessed, total_pos),
+        rate: calculateRate(startTime, totalProcessed)
+      });
+
+      // Update last processed IDs for next chunk with safety check
+      if (chunk.length > 0) {
+        const lastItem = chunk[chunk.length - 1];
+        if (lastItem) {
+          lastPoId = lastItem.po_id;
+          lastPid = lastItem.pid;
+        }
+      }
+    }
+
+    // Update sync status
     await localConnection.query(`
       INSERT INTO sync_status (table_name, last_sync_timestamp)
       VALUES ('purchase_orders', NOW())
-      ON DUPLICATE KEY UPDATE 
-        last_sync_timestamp = NOW(),
-        last_sync_id = LAST_INSERT_ID(last_sync_id)
+      ON CONFLICT (table_name) DO UPDATE SET
+        last_sync_timestamp = NOW()
+    `);
+
+    // Clean up temporary tables
+    await localConnection.query(`
+      DROP TABLE IF EXISTS temp_purchase_orders;
+      DROP TABLE IF EXISTS temp_po_receivings;
     `);
 
     return {
       status: "complete",
-      totalImported: totalItems,
-      recordsAdded: recordsAdded || 0,
-      recordsUpdated: recordsUpdated || 0,
-      incrementalUpdate,
-      lastSyncTime
+      recordsAdded,
+      recordsUpdated,
+      totalRecords: processed
     };
-
   } catch (error) {
-    outputProgress({
-      operation: `${incrementalUpdate ? 'Incremental' : 'Full'} purchase orders import failed`,
-      status: "error",
-      error: error.message,
-    });
+    console.error("Error during purchase orders import:", error);
+    // Attempt cleanup on error
+    try {
+      await localConnection.query(`
+        DROP TABLE IF EXISTS temp_purchase_orders;
+        DROP TABLE IF EXISTS temp_po_receivings;
+      `);
+    } catch (cleanupError) {
+      console.error('Error during cleanup:', cleanupError);
+    }
     throw error;
   }
 }
 
-module.exports = importPurchaseOrders; 
+module.exports = importPurchaseOrders;

inventory-server/scripts/import/purchase_orders.js

@@ -1,82 +0,0 @@
-// Split into inserts and updates
-const insertsAndUpdates = batch.reduce((acc, po) => {
-  const key = `${po.po_id}-${po.pid}`;
-  if (existingPOMap.has(key)) {
-    const existing = existingPOMap.get(key);
-    // Check if any values are different
-    const hasChanges = columnNames.some(col => {
-      const newVal = po[col] ?? null;
-      const oldVal = existing[col] ?? null;
-      // Special handling for numbers to avoid type coercion issues
-      if (typeof newVal === 'number' && typeof oldVal === 'number') {
-        return Math.abs(newVal - oldVal) > 0.00001; // Allow for tiny floating point differences
-      }
-      // Special handling for receiving_history JSON
-      if (col === 'receiving_history') {
-        return JSON.stringify(newVal) !== JSON.stringify(oldVal);
-      }
-      return newVal !== oldVal;
-    });
-
-    if (hasChanges) {
-      console.log(`PO line changed: ${key}`, {
-        po_id: po.po_id,
-        pid: po.pid,
-        changes: columnNames.filter(col => {
-          const newVal = po[col] ?? null;
-          const oldVal = existing[col] ?? null;
-          if (typeof newVal === 'number' && typeof oldVal === 'number') {
-            return Math.abs(newVal - oldVal) > 0.00001;
-          }
-          if (col === 'receiving_history') {
-            return JSON.stringify(newVal) !== JSON.stringify(oldVal);
-          }
-          return newVal !== oldVal;
-        })
-      });
-      acc.updates.push({
-        po_id: po.po_id,
-        pid: po.pid,
-        values: columnNames.map(col => po[col] ?? null)
-      });
-    }
-  } else {
-    console.log(`New PO line: ${key}`);
-    acc.inserts.push({
-      po_id: po.po_id,
-      pid: po.pid,
-      values: columnNames.map(col => po[col] ?? null)
-    });
-  }
-  return acc;
-}, { inserts: [], updates: [] });
-
-// Handle inserts
-if (insertsAndUpdates.inserts.length > 0) {
-  const insertPlaceholders = Array(insertsAndUpdates.inserts.length).fill(placeholderGroup).join(",");
-  
-  const insertResult = await localConnection.query(`
-    INSERT INTO purchase_orders (${columnNames.join(",")})
-    VALUES ${insertPlaceholders}
-  `, insertsAndUpdates.inserts.map(i => i.values).flat());
-  
-  recordsAdded += insertResult[0].affectedRows;
-}
-
-// Handle updates
-if (insertsAndUpdates.updates.length > 0) {
-  const updatePlaceholders = Array(insertsAndUpdates.updates.length).fill(placeholderGroup).join(",");
-  
-  const updateResult = await localConnection.query(`
-    INSERT INTO purchase_orders (${columnNames.join(",")})
-    VALUES ${updatePlaceholders}
-    ON DUPLICATE KEY UPDATE
-      ${columnNames
-        .filter(col => col !== "po_id" && col !== "pid")
-        .map(col => `${col} = VALUES(${col})`)
-        .join(",")};
-  `, insertsAndUpdates.updates.map(u => u.values).flat());
-  
-  // Each update affects 2 rows in affectedRows, so we divide by 2 to get actual count
-  recordsUpdated += insertsAndUpdates.updates.length;
-} 

inventory-server/scripts/import/utils.js

@@ -1,5 +1,6 @@
 const mysql = require("mysql2/promise");
 const { Client } = require("ssh2");
+const { Pool } = require('pg');
 const dotenv = require("dotenv");
 const path = require("path");
 
@@ -41,23 +42,90 @@ async function setupSshTunnel(sshConfig) {
 async function setupConnections(sshConfig) {
   const tunnel = await setupSshTunnel(sshConfig);
   
+  // Setup MySQL connection for production
   const prodConnection = await mysql.createConnection({
     ...sshConfig.prodDbConfig,
     stream: tunnel.stream,
   });
 
-  const localConnection = await mysql.createPool({
-    ...sshConfig.localDbConfig,
-    waitForConnections: true,
-    connectionLimit: 10,
-    queueLimit: 0
-  });
+  // Setup PostgreSQL connection pool for local
+  const localPool = new Pool(sshConfig.localDbConfig);
 
-  return {
-    ssh: tunnel.ssh,
-    prodConnection,
-    localConnection
+  // Test the PostgreSQL connection
+  try {
+    const client = await localPool.connect();
+    await client.query('SELECT NOW()');
+    client.release();
+    console.log('PostgreSQL connection successful');
+  } catch (err) {
+    console.error('PostgreSQL connection error:', err);
+    throw err;
+  }
+
+  // Create a wrapper for the PostgreSQL pool to match MySQL interface
+  const localConnection = {
+    _client: null,
+    _transactionActive: false,
+
+    query: async (text, params) => {
+      // If we're not in a transaction, use the pool directly
+      if (!localConnection._transactionActive) {
+        const client = await localPool.connect();
+        try {
+          const result = await client.query(text, params);
+          return [result];
+        } finally {
+          client.release();
+        }
+      }
+      
+      // If we're in a transaction, use the dedicated client
+      if (!localConnection._client) {
+        throw new Error('No active transaction client');
+      }
+      const result = await localConnection._client.query(text, params);
+      return [result];
+    },
+
+    beginTransaction: async () => {
+      if (localConnection._transactionActive) {
+        throw new Error('Transaction already active');
+      }
+      localConnection._client = await localPool.connect();
+      await localConnection._client.query('BEGIN');
+      localConnection._transactionActive = true;
+    },
+
+    commit: async () => {
+      if (!localConnection._transactionActive) {
+        throw new Error('No active transaction to commit');
+      }
+      await localConnection._client.query('COMMIT');
+      localConnection._client.release();
+      localConnection._client = null;
+      localConnection._transactionActive = false;
+    },
+
+    rollback: async () => {
+      if (!localConnection._transactionActive) {
+        throw new Error('No active transaction to rollback');
+      }
+      await localConnection._client.query('ROLLBACK');
+      localConnection._client.release();
+      localConnection._client = null;
+      localConnection._transactionActive = false;
+    },
+
+    end: async () => {
+      if (localConnection._client) {
+        localConnection._client.release();
+        localConnection._client = null;
+      }
+      await localPool.end();
+    }
   };
+
+  return { prodConnection, localConnection, tunnel };
 }
 
 // Helper function to close connections

inventory-server/scripts/metrics/brand-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateBrandMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateBrandMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +16,29 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting brand metrics calculation',
@@ -26,7 +47,12 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Calculate brand metrics with optimized queries
@@ -45,10 +71,21 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             WITH filtered_products AS (
                 SELECT 
                     p.*,
-                    CASE WHEN p.stock_quantity <= 5000 THEN p.pid END as valid_pid,
-                    CASE WHEN p.visible = true AND p.stock_quantity <= 5000 THEN p.pid END as active_pid,
                     CASE 
-                        WHEN p.stock_quantity IS NULL OR p.stock_quantity < 0 OR p.stock_quantity > 5000 THEN 0 
+                        WHEN p.stock_quantity <= 5000 AND p.stock_quantity >= 0 
+                        THEN p.pid 
+                    END as valid_pid,
+                    CASE 
+                        WHEN p.visible = true 
+                        AND p.stock_quantity <= 5000 
+                        AND p.stock_quantity >= 0 
+                        THEN p.pid 
+                    END as active_pid,
+                    CASE 
+                        WHEN p.stock_quantity IS NULL 
+                        OR p.stock_quantity < 0 
+                        OR p.stock_quantity > 5000 
+                        THEN 0 
                         ELSE p.stock_quantity 
                     END as valid_stock
                 FROM products p
@@ -57,10 +94,13 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             sales_periods AS (
                 SELECT 
                     p.brand,
-                    SUM(o.quantity * o.price) as period_revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as period_revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as period_margin,
+                    COUNT(DISTINCT DATE(o.date)) as period_days,
                     CASE 
                         WHEN o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH) THEN 'current'
-                        WHEN o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH) THEN 'previous'
+                        WHEN o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                            AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH) THEN 'previous'
                     END as period_type
                 FROM filtered_products p
                 JOIN orders o ON p.pid = o.pid 
@@ -76,10 +116,20 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                     SUM(p.valid_stock) as total_stock_units,
                     SUM(p.valid_stock * p.cost_price) as total_stock_cost,
                     SUM(p.valid_stock * p.price) as total_stock_retail,
-                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
+                    COALESCE(SUM(o.quantity * (o.price - COALESCE(o.discount, 0))), 0) as total_revenue,
                     CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 THEN 
-                            (SUM((o.price - p.cost_price) * o.quantity) * 100.0) / SUM(o.price * o.quantity)
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN GREATEST(
+                            -100.0,
+                            LEAST(
+                                100.0,
+                                (
+                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                ) * 100.0 / 
+                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                            )
+                        )
                         ELSE 0 
                     END as avg_margin
                 FROM filtered_products p
@@ -97,16 +147,18 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                 bd.avg_margin,
                 CASE
                     WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
-                         AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0 THEN 100.0
-                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 THEN 0.0
-                    ELSE LEAST(
-                        GREATEST(
+                         AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0 
+                    THEN 100.0
+                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
+                    THEN 0.0
+                    ELSE GREATEST(
+                        -100.0,
+                        LEAST(
                             ((MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) - 
                               MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)) / 
-                             NULLIF(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END), 0)) * 100.0,
-                            -100.0
-                        ),
-                        999.99
+                             NULLIF(ABS(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)), 0)) * 100.0,
+                            999.99
+                        )
                     )
                 END as growth_rate
             FROM brand_data bd
@@ -134,10 +186,20 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate brand time-based metrics with optimized query
         await connection.query(`
@@ -177,8 +239,18 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                     SUM(p.valid_stock * p.price) as total_stock_retail,
                     SUM(o.quantity * o.price) as total_revenue,
                     CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 THEN 
-                            (SUM((o.price - p.cost_price) * o.quantity) * 100.0) / SUM(o.price * o.quantity)
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN GREATEST(
+                            -100.0,
+                            LEAST(
+                                100.0,
+                                (
+                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                ) * 100.0 / 
+                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                            )
+                        )
                         ELSE 0 
                     END as avg_margin
                 FROM filtered_products p
@@ -207,11 +279,33 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('brand_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating brand metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/category-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateCategoryMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateCategoryMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +16,29 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting category metrics calculation',
@@ -26,7 +47,12 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
         
         // First, calculate base category metrics
@@ -67,10 +93,20 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Then update with margin and turnover data
         await connection.query(`
@@ -80,19 +116,35 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                     SUM(o.quantity * o.price) as total_sales,
                     SUM(o.quantity * (o.price - p.cost_price)) as total_margin,
                     SUM(o.quantity) as units_sold,
-                    AVG(GREATEST(p.stock_quantity, 0)) as avg_stock
+                    AVG(GREATEST(p.stock_quantity, 0)) as avg_stock,
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
+                LEFT JOIN turnover_config tc ON 
+                    (tc.category_id = pc.cat_id AND tc.vendor = p.vendor) OR
+                    (tc.category_id = pc.cat_id AND tc.vendor IS NULL) OR
+                    (tc.category_id IS NULL AND tc.vendor = p.vendor) OR
+                    (tc.category_id IS NULL AND tc.vendor IS NULL)
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL COALESCE(tc.calculation_period_days, 30) DAY)
                 GROUP BY pc.cat_id
             )
             UPDATE category_metrics cm
             JOIN category_sales cs ON cm.category_id = cs.cat_id
+            LEFT JOIN turnover_config tc ON 
+                (tc.category_id = cm.category_id AND tc.vendor IS NULL) OR
+                (tc.category_id IS NULL AND tc.vendor IS NULL)
             SET 
                 cm.avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
-                cm.turnover_rate = LEAST(COALESCE(cs.units_sold / NULLIF(cs.avg_stock, 0), 0), 999.99),
+                cm.turnover_rate = CASE 
+                    WHEN cs.avg_stock > 0 AND cs.active_days > 0 
+                    THEN LEAST(
+                        (cs.units_sold / cs.avg_stock) * (365.0 / cs.active_days),
+                        999.99
+                    )
+                    ELSE 0 
+                END,
                 cm.last_calculated_at = NOW()
         `);
 
@@ -105,20 +157,34 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Finally update growth rates
         await connection.query(`
             WITH current_period AS (
                 SELECT 
                     pc.cat_id,
-                    SUM(o.quantity * o.price) as revenue
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as gross_profit,
+                    COUNT(DISTINCT DATE(o.date)) as days
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
                 WHERE o.canceled = false 
                     AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
                 GROUP BY pc.cat_id
@@ -126,30 +192,106 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             previous_period AS (
                 SELECT 
                     pc.cat_id,
-                    SUM(o.quantity * o.price) as revenue
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    COUNT(DISTINCT DATE(o.date)) as days
+                FROM product_categories pc
+                JOIN products p ON pc.pid = p.pid
+                JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                WHERE o.canceled = false 
+                    AND o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                    AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                GROUP BY pc.cat_id
+            ),
+            trend_data AS (
+                SELECT 
+                    pc.cat_id,
+                    MONTH(o.date) as month,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    COUNT(DISTINCT DATE(o.date)) as days_in_month
+                FROM product_categories pc
+                JOIN products p ON pc.pid = p.pid
+                JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                WHERE o.canceled = false 
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH)
+                GROUP BY pc.cat_id, MONTH(o.date)
+            ),
+            trend_stats AS (
+                SELECT 
+                    cat_id,
+                    COUNT(*) as n,
+                    AVG(month) as avg_x,
+                    AVG(revenue / NULLIF(days_in_month, 0)) as avg_y,
+                    SUM(month * (revenue / NULLIF(days_in_month, 0))) as sum_xy,
+                    SUM(month * month) as sum_xx
+                FROM trend_data
+                GROUP BY cat_id
+                HAVING COUNT(*) >= 6
+            ),
+            trend_analysis AS (
+                SELECT 
+                    cat_id,
+                    ((n * sum_xy) - (avg_x * n * avg_y)) / 
+                        NULLIF((n * sum_xx) - (n * avg_x * avg_x), 0) as trend_slope,
+                    avg_y as avg_daily_revenue
+                FROM trend_stats
+            ),
+            margin_calc AS (
+                SELECT 
+                    pc.cat_id,
+                    CASE 
+                        WHEN SUM(o.quantity * o.price) > 0 THEN
+                            GREATEST(
+                                -100.0,
+                                LEAST(
+                                    100.0,
+                                    (
+                                        SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                        SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                    ) * 100.0 / 
+                                    NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                                )
+                            )
+                        ELSE NULL
+                    END as avg_margin
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false 
-                    AND o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
-                    AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
                 GROUP BY pc.cat_id
             )
             UPDATE category_metrics cm
             LEFT JOIN current_period cp ON cm.category_id = cp.cat_id
             LEFT JOIN previous_period pp ON cm.category_id = pp.cat_id
+            LEFT JOIN trend_analysis ta ON cm.category_id = ta.cat_id
+            LEFT JOIN margin_calc mc ON cm.category_id = mc.cat_id
             SET 
                 cm.growth_rate = CASE
                     WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
-                    WHEN pp.revenue = 0 THEN 0.0
-                    ELSE LEAST(
+                    WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
+                    WHEN ta.trend_slope IS NOT NULL THEN
                         GREATEST(
-                            ((COALESCE(cp.revenue, 0) - pp.revenue) / pp.revenue) * 100.0,
-                            -100.0
-                        ),
-                        999.99
-                    )
+                            -100.0,
+                            LEAST(
+                                (ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
+                                999.99
+                            )
+                        )
+                    ELSE
+                        GREATEST(
+                            -100.0,
+                            LEAST(
+                                ((COALESCE(cp.revenue, 0) - pp.revenue) / 
+                                    NULLIF(ABS(pp.revenue), 0)) * 100.0,
+                                999.99
+                            )
+                        )
                 END,
+                cm.avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
                 cm.last_calculated_at = NOW()
             WHERE cp.cat_id IS NOT NULL OR pp.cat_id IS NOT NULL
         `);
@@ -163,10 +305,20 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate time-based metrics
         await connection.query(`
@@ -189,13 +341,23 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
                 SUM(p.stock_quantity * p.cost_price) as total_value,
                 SUM(o.quantity * o.price) as total_revenue,
+                CASE
+                    WHEN SUM(o.quantity * o.price) > 0 THEN
+                        LEAST(
+                            GREATEST(
+                                SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0))) * 100.0 / 
+                                SUM(o.quantity * o.price),
+                                -100
+                            ),
+                            100
+                        )
+                    ELSE 0
+                END as avg_margin,
                 COALESCE(
-                    SUM(o.quantity * (o.price - p.cost_price)) * 100.0 / 
-                    NULLIF(SUM(o.quantity * o.price), 0),
-                    0
-                ) as avg_margin,
-                COALESCE(
-                    SUM(o.quantity) / NULLIF(AVG(GREATEST(p.stock_quantity, 0)), 0),
+                    LEAST(
+                        SUM(o.quantity) / NULLIF(AVG(GREATEST(p.stock_quantity, 0)), 0),
+                        999.99
+                    ),
                     0
                 ) as turnover_rate
             FROM product_categories pc
@@ -216,17 +378,138 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
         processedCount = Math.floor(totalProducts * 0.99);
         outputProgress({
             status: 'running',
-            operation: 'Time-based metrics calculated',
+            operation: 'Time-based metrics calculated, updating category-sales metrics',
             current: processedCount,
             total: totalProducts,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
+        // Calculate category-sales metrics
+        await connection.query(`
+            INSERT INTO category_sales_metrics (
+                category_id,
+                brand,
+                period_start,
+                period_end,
+                avg_daily_sales,
+                total_sold,
+                num_products,
+                avg_price,
+                last_calculated_at
+            )
+            WITH date_ranges AS (
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 30 DAY) as period_start,
+                    CURRENT_DATE as period_end
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 31 DAY)
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 180 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 91 DAY)
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 365 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 181 DAY)
+            ),
+            sales_data AS (
+                SELECT 
+                    pc.cat_id,
+                    COALESCE(p.brand, 'Unknown') as brand,
+                    dr.period_start,
+                    dr.period_end,
+                    COUNT(DISTINCT p.pid) as num_products,
+                    SUM(o.quantity) as total_sold,
+                    SUM(o.quantity * o.price) as total_revenue,
+                    COUNT(DISTINCT DATE(o.date)) as num_days
+                FROM products p
+                JOIN product_categories pc ON p.pid = pc.pid
+                JOIN orders o ON p.pid = o.pid
+                CROSS JOIN date_ranges dr
+                WHERE o.canceled = false
+                    AND o.date BETWEEN dr.period_start AND dr.period_end
+                GROUP BY pc.cat_id, p.brand, dr.period_start, dr.period_end
+            )
+            SELECT 
+                cat_id as category_id,
+                brand,
+                period_start,
+                period_end,
+                CASE 
+                    WHEN num_days > 0 
+                    THEN total_sold / num_days 
+                    ELSE 0 
+                END as avg_daily_sales,
+                total_sold,
+                num_products,
+                CASE 
+                    WHEN total_sold > 0 
+                    THEN total_revenue / total_sold 
+                    ELSE 0 
+                END as avg_price,
+                NOW() as last_calculated_at
+            FROM sales_data
+            ON DUPLICATE KEY UPDATE
+                avg_daily_sales = VALUES(avg_daily_sales),
+                total_sold = VALUES(total_sold),
+                num_products = VALUES(num_products),
+                avg_price = VALUES(avg_price),
+                last_calculated_at = VALUES(last_calculated_at)
+        `);
+
+        processedCount = Math.floor(totalProducts * 1.0);
+        outputProgress({
+            status: 'running',
+            operation: 'Category-sales metrics calculated',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('category_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating category metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/financial-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateFinancialMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateFinancialMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +16,30 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+                AND DATE(o.date) >= DATE_SUB(CURDATE(), INTERVAL 12 MONTH)
+        `);
+        processedOrders = orderCount[0].count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting financial metrics calculation',
@@ -26,7 +48,12 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Calculate financial metrics with optimized query
@@ -59,7 +86,8 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
                     WHEN COALESCE(pf.inventory_value, 0) > 0 AND pf.active_days > 0 THEN 
                         (COALESCE(pf.gross_profit, 0) * (365.0 / pf.active_days)) / COALESCE(pf.inventory_value, 0)
                     ELSE 0 
-                END
+                END,
+                pm.last_calculated_at = CURRENT_TIMESTAMP
         `);
 
         processedCount = Math.floor(totalProducts * 0.65);
@@ -71,10 +99,20 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Update time-based aggregates with optimized query
         await connection.query(`
@@ -115,11 +153,33 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('financial_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating financial metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/product-metrics.js

@@ -11,11 +11,21 @@ function sanitizeValue(value) {
 
 async function calculateProductMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    const BATCH_SIZE = 5000;
+    
     try {
         // Skip flags are inherited from the parent scope
         const SKIP_PRODUCT_BASE_METRICS = 0;
         const SKIP_PRODUCT_TIME_AGGREGATES = 0;
 
+        // Get total product count if not provided
+        if (!totalProducts) {
+            const [productCount] = await connection.query('SELECT COUNT(*) as count FROM products');
+            totalProducts = productCount[0].count;
+        }
+
         if (isCancelled) {
             outputProgress({
                 status: 'cancelled',
@@ -25,11 +35,37 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // First ensure all products have a metrics record
+        await connection.query(`
+            INSERT IGNORE INTO product_metrics (pid, last_calculated_at)
+            SELECT pid, NOW()
+            FROM products
+        `);
+
+        // Get threshold settings once
+        const [thresholds] = await connection.query(`
+            SELECT critical_days, reorder_days, overstock_days, low_stock_threshold
+            FROM stock_thresholds 
+            WHERE category_id IS NULL AND vendor IS NULL
+            LIMIT 1
+        `);
+        const defaultThresholds = thresholds[0];
+
         // Calculate base product metrics
         if (!SKIP_PRODUCT_BASE_METRICS) {
             outputProgress({
@@ -40,89 +76,237 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: estimateRemaining(startTime, processedCount, totalProducts),
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            // Calculate base metrics
+            // Get order count that will be processed
+            const [orderCount] = await connection.query(`
+                SELECT COUNT(*) as count 
+                FROM orders o 
+                WHERE o.canceled = false
+            `);
+            processedOrders = orderCount[0].count;
+
+            // Clear temporary tables
+            await connection.query('TRUNCATE TABLE temp_sales_metrics');
+            await connection.query('TRUNCATE TABLE temp_purchase_metrics');
+
+            // Populate temp_sales_metrics with base stats and sales averages
             await connection.query(`
-                UPDATE product_metrics pm
-                JOIN (
-                    SELECT 
-                        p.pid,
-                        p.cost_price * p.stock_quantity as inventory_value,
-                        SUM(o.quantity) as total_quantity,
-                        COUNT(DISTINCT o.order_number) as number_of_orders,
-                        SUM(o.quantity * o.price) as total_revenue,
-                        SUM(o.quantity * p.cost_price) as cost_of_goods_sold,
-                        AVG(o.price) as avg_price,
-                        STDDEV(o.price) as price_std,
-                        MIN(o.date) as first_sale_date,
-                        MAX(o.date) as last_sale_date,
-                        COUNT(DISTINCT DATE(o.date)) as active_days
-                    FROM products p
-                    LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
-                    GROUP BY p.pid
-                ) stats ON pm.pid = stats.pid
-                SET 
-                    pm.inventory_value = COALESCE(stats.inventory_value, 0),
-                    pm.avg_quantity_per_order = COALESCE(stats.total_quantity / NULLIF(stats.number_of_orders, 0), 0),
-                    pm.number_of_orders = COALESCE(stats.number_of_orders, 0),
-                    pm.total_revenue = COALESCE(stats.total_revenue, 0),
-                    pm.cost_of_goods_sold = COALESCE(stats.cost_of_goods_sold, 0),
-                    pm.gross_profit = COALESCE(stats.total_revenue - stats.cost_of_goods_sold, 0),
-                    pm.avg_margin_percent = CASE 
-                        WHEN COALESCE(stats.total_revenue, 0) > 0 
-                        THEN ((stats.total_revenue - stats.cost_of_goods_sold) / stats.total_revenue) * 100 
+                INSERT INTO temp_sales_metrics
+                SELECT 
+                    p.pid,
+                    COALESCE(SUM(o.quantity) / NULLIF(COUNT(DISTINCT DATE(o.date)), 0), 0) as daily_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 7), 0), 0) as weekly_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 30), 0), 0) as monthly_sales_avg,
+                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
+                    CASE 
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN ((SUM(o.quantity * o.price) - SUM(o.quantity * p.cost_price)) / SUM(o.quantity * o.price)) * 100 
                         ELSE 0 
-                    END,
-                    pm.first_sale_date = stats.first_sale_date,
-                    pm.last_sale_date = stats.last_sale_date,
-                    pm.gmroi = CASE 
-                        WHEN COALESCE(stats.inventory_value, 0) > 0 
-                        THEN (stats.total_revenue - stats.cost_of_goods_sold) / stats.inventory_value
-                        ELSE 0 
-                    END,
-                    pm.last_calculated_at = NOW()
+                    END as avg_margin_percent,
+                    MIN(o.date) as first_sale_date,
+                    MAX(o.date) as last_sale_date
+                FROM products p
+                LEFT JOIN orders o ON p.pid = o.pid 
+                    AND o.canceled = false
+                    AND o.date >= DATE_SUB(CURDATE(), INTERVAL 90 DAY)
+                GROUP BY p.pid
             `);
 
-            processedCount = Math.floor(totalProducts * 0.4);
-            outputProgress({
-                status: 'running',
-                operation: 'Base product metrics calculated',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-        } else {
-            processedCount = Math.floor(totalProducts * 0.4);
-            outputProgress({
-                status: 'running',
-                operation: 'Skipping base product metrics calculation',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-        }
+            // Populate temp_purchase_metrics
+            await connection.query(`
+                INSERT INTO temp_purchase_metrics
+                SELECT 
+                    p.pid,
+                    AVG(DATEDIFF(po.received_date, po.date)) as avg_lead_time_days,
+                    MAX(po.date) as last_purchase_date,
+                    MIN(po.received_date) as first_received_date,
+                    MAX(po.received_date) as last_received_date
+                FROM products p
+                LEFT JOIN purchase_orders po ON p.pid = po.pid 
+                    AND po.received_date IS NOT NULL
+                    AND po.date >= DATE_SUB(CURDATE(), INTERVAL 365 DAY)
+                GROUP BY p.pid
+            `);
 
-        if (isCancelled) return processedCount;
+            // Process updates in batches
+            let lastPid = 0;
+            while (true) {
+                if (isCancelled) break;
+
+                const [batch] = await connection.query(
+                    'SELECT pid FROM products WHERE pid > ? ORDER BY pid LIMIT ?',
+                    [lastPid, BATCH_SIZE]
+                );
+
+                if (batch.length === 0) break;
+
+                await connection.query(`
+                    UPDATE product_metrics pm
+                    JOIN products p ON pm.pid = p.pid
+                    LEFT JOIN temp_sales_metrics sm ON pm.pid = sm.pid
+                    LEFT JOIN temp_purchase_metrics lm ON pm.pid = lm.pid
+                    SET 
+                        pm.inventory_value = p.stock_quantity * NULLIF(p.cost_price, 0),
+                        pm.daily_sales_avg = COALESCE(sm.daily_sales_avg, 0),
+                        pm.weekly_sales_avg = COALESCE(sm.weekly_sales_avg, 0),
+                        pm.monthly_sales_avg = COALESCE(sm.monthly_sales_avg, 0),
+                        pm.total_revenue = COALESCE(sm.total_revenue, 0),
+                        pm.avg_margin_percent = COALESCE(sm.avg_margin_percent, 0),
+                        pm.first_sale_date = sm.first_sale_date,
+                        pm.last_sale_date = sm.last_sale_date,
+                        pm.avg_lead_time_days = COALESCE(lm.avg_lead_time_days, 30),
+                        pm.days_of_inventory = CASE 
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.daily_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        pm.weeks_of_inventory = CASE 
+                            WHEN COALESCE(sm.weekly_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.weekly_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        pm.stock_status = CASE
+                            WHEN p.stock_quantity <= 0 THEN 'Out of Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 AND p.stock_quantity <= ? THEN 'Low Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 THEN 'In Stock'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ? THEN 'Critical'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ? THEN 'Reorder'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ? THEN 'Overstocked'
+                            ELSE 'Healthy'
+                        END,
+                        pm.safety_stock = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
+                                CEIL(sm.daily_sales_avg * SQRT(COALESCE(lm.avg_lead_time_days, 30)) * 1.96)
+                            ELSE ?
+                        END,
+                        pm.reorder_point = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
+                                CEIL(sm.daily_sales_avg * COALESCE(lm.avg_lead_time_days, 30)) + 
+                                CEIL(sm.daily_sales_avg * SQRT(COALESCE(lm.avg_lead_time_days, 30)) * 1.96)
+                            ELSE ?
+                        END,
+                        pm.reorder_qty = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND NULLIF(p.cost_price, 0) IS NOT NULL THEN
+                                GREATEST(
+                                    CEIL(SQRT((2 * (sm.daily_sales_avg * 365) * 25) / (NULLIF(p.cost_price, 0) * 0.25))),
+                                    ?
+                                )
+                            ELSE ?
+                        END,
+                        pm.overstocked_amt = CASE
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ? 
+                            THEN GREATEST(0, p.stock_quantity - CEIL(sm.daily_sales_avg * ?))
+                            ELSE 0
+                        END,
+                        pm.last_calculated_at = NOW()
+                    WHERE p.pid IN (${batch.map(() => '?').join(',')})
+                `,
+                [
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.critical_days,
+                    defaultThresholds.reorder_days,
+                    defaultThresholds.overstock_days,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.overstock_days,
+                    defaultThresholds.overstock_days,
+                    ...batch.map(row => row.pid)
+                ]
+                );
+
+                lastPid = batch[batch.length - 1].pid;
+                processedCount += batch.length;
+
+                outputProgress({
+                    status: 'running',
+                    operation: 'Processing base metrics batch',
+                    current: processedCount,
+                    total: totalProducts,
+                    elapsed: formatElapsedTime(startTime),
+                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
+                    rate: calculateRate(startTime, processedCount),
+                    percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
+                });
+            }
+
+            // Calculate forecast accuracy and bias in batches
+            lastPid = 0;
+            while (true) {
+                if (isCancelled) break;
+
+                const [batch] = await connection.query(
+                    'SELECT pid FROM products WHERE pid > ? ORDER BY pid LIMIT ?',
+                    [lastPid, BATCH_SIZE]
+                );
+
+                if (batch.length === 0) break;
+
+                await connection.query(`
+                    UPDATE product_metrics pm
+                    JOIN (
+                        SELECT 
+                            sf.pid,
+                            AVG(CASE 
+                                WHEN o.quantity > 0 
+                                THEN ABS(sf.forecast_units - o.quantity) / o.quantity * 100
+                                ELSE 100 
+                            END) as avg_forecast_error,
+                            AVG(CASE 
+                                WHEN o.quantity > 0 
+                                THEN (sf.forecast_units - o.quantity) / o.quantity * 100
+                                ELSE 0 
+                            END) as avg_forecast_bias,
+                            MAX(sf.forecast_date) as last_forecast_date
+                        FROM sales_forecasts sf
+                        JOIN orders o ON sf.pid = o.pid 
+                            AND DATE(o.date) = sf.forecast_date
+                        WHERE o.canceled = false
+                            AND sf.forecast_date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+                            AND sf.pid IN (?)
+                        GROUP BY sf.pid
+                    ) fa ON pm.pid = fa.pid
+                    SET 
+                        pm.forecast_accuracy = GREATEST(0, 100 - LEAST(fa.avg_forecast_error, 100)),
+                        pm.forecast_bias = GREATEST(-100, LEAST(fa.avg_forecast_bias, 100)),
+                        pm.last_forecast_date = fa.last_forecast_date,
+                        pm.last_calculated_at = NOW()
+                    WHERE pm.pid IN (?)
+                `, [batch.map(row => row.pid), batch.map(row => row.pid)]);
+
+                lastPid = batch[batch.length - 1].pid;
+            }
+        }
 
         // Calculate product time aggregates
         if (!SKIP_PRODUCT_TIME_AGGREGATES) {
             outputProgress({
                 status: 'running',
                 operation: 'Starting product time aggregates calculation',
-                current: processedCount,
-                total: totalProducts,
+                current: processedCount || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
             // Calculate time-based aggregates
@@ -179,29 +363,206 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
             outputProgress({
                 status: 'running',
                 operation: 'Product time aggregates calculated',
-                current: processedCount,
-                total: totalProducts,
+                current: processedCount || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
         } else {
             processedCount = Math.floor(totalProducts * 0.6);
             outputProgress({
                 status: 'running',
                 operation: 'Skipping product time aggregates calculation',
-                current: processedCount,
-                total: totalProducts,
+                current: processedCount || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
         }
 
-        return processedCount;
+        // Calculate ABC classification
+        outputProgress({
+            status: 'running',
+            operation: 'Starting ABC classification',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
+        const [abcConfig] = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
+        const abcThresholds = abcConfig[0] || { a_threshold: 20, b_threshold: 50 };
+
+        // First, create and populate the rankings table with an index
+        await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
+        await connection.query(`
+            CREATE TEMPORARY TABLE temp_revenue_ranks (
+                pid BIGINT NOT NULL,
+                total_revenue DECIMAL(10,3),
+                rank_num INT,
+                dense_rank_num INT,
+                percentile DECIMAL(5,2),
+                total_count INT,
+                PRIMARY KEY (pid),
+                INDEX (rank_num),
+                INDEX (dense_rank_num),
+                INDEX (percentile)
+            ) ENGINE=MEMORY
+        `);
+
+        // Calculate rankings with proper tie handling
+        await connection.query(`
+            INSERT INTO temp_revenue_ranks
+            WITH revenue_data AS (
+                SELECT 
+                    pid,
+                    total_revenue,
+                    COUNT(*) OVER () as total_count,
+                    PERCENT_RANK() OVER (ORDER BY total_revenue DESC) * 100 as percentile,
+                    RANK() OVER (ORDER BY total_revenue DESC) as rank_num,
+                    DENSE_RANK() OVER (ORDER BY total_revenue DESC) as dense_rank_num
+                FROM product_metrics 
+                WHERE total_revenue > 0
+            )
+            SELECT 
+                pid,
+                total_revenue,
+                rank_num,
+                dense_rank_num,
+                percentile,
+                total_count
+            FROM revenue_data
+        `);
+
+        // Get total count for percentage calculation
+        const [rankingCount] = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
+        const totalCount = rankingCount[0].total_count || 1;
+        const max_rank = totalCount;
+
+        // Process updates in batches
+        let abcProcessedCount = 0;
+        const batchSize = 5000;
+
+        while (true) {
+            if (isCancelled) return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,  // This module doesn't process POs
+                success
+            };
+
+            // Get a batch of PIDs that need updating
+            const [pids] = await connection.query(`
+                SELECT pm.pid 
+                FROM product_metrics pm
+                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
+                WHERE pm.abc_class IS NULL 
+                OR pm.abc_class != 
+                    CASE 
+                        WHEN tr.pid IS NULL THEN 'C'
+                        WHEN tr.percentile <= ? THEN 'A'
+                        WHEN tr.percentile <= ? THEN 'B'
+                        ELSE 'C'
+                    END
+                LIMIT ?
+            `, [abcThresholds.a_threshold, abcThresholds.b_threshold, batchSize]);
+
+            if (pids.length === 0) break;
+
+            await connection.query(`
+                UPDATE product_metrics pm
+                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
+                SET pm.abc_class = 
+                    CASE 
+                        WHEN tr.pid IS NULL THEN 'C'
+                        WHEN tr.percentile <= ? THEN 'A'
+                        WHEN tr.percentile <= ? THEN 'B'
+                        ELSE 'C'
+                    END,
+                    pm.last_calculated_at = NOW()
+                WHERE pm.pid IN (?)
+            `, [abcThresholds.a_threshold, abcThresholds.b_threshold, pids.map(row => row.pid)]);
+
+            // Now update turnover rate with proper handling of zero inventory periods
+            await connection.query(`
+                UPDATE product_metrics pm
+                JOIN (
+                    SELECT 
+                        o.pid,
+                        SUM(o.quantity) as total_sold,
+                        COUNT(DISTINCT DATE(o.date)) as active_days,
+                        AVG(CASE 
+                            WHEN p.stock_quantity > 0 THEN p.stock_quantity 
+                            ELSE NULL 
+                        END) as avg_nonzero_stock
+                    FROM orders o
+                    JOIN products p ON o.pid = p.pid
+                    WHERE o.canceled = false 
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+                    AND o.pid IN (?)
+                    GROUP BY o.pid
+                ) sales ON pm.pid = sales.pid
+                SET 
+                    pm.turnover_rate = CASE 
+                        WHEN sales.avg_nonzero_stock > 0 AND sales.active_days > 0 
+                        THEN LEAST(
+                            (sales.total_sold / sales.avg_nonzero_stock) * (365.0 / sales.active_days),
+                            999.99
+                        )
+                        ELSE 0 
+                    END,
+                    pm.last_calculated_at = NOW()
+                WHERE pm.pid IN (?)
+            `, [pids.map(row => row.pid), pids.map(row => row.pid)]);
+        }
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('product_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount || 0,
+            processedOrders: processedOrders || 0,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating product metrics');
         throw error;
     } finally {
@@ -257,9 +618,9 @@ function calculateReorderQuantities(stock, stock_status, daily_sales_avg, avg_le
     if (daily_sales_avg > 0) {
         const annual_demand = daily_sales_avg * 365;
         const order_cost = 25; // Fixed cost per order
-        const holding_cost_percent = 0.25; // 25% annual holding cost
+        const holding_cost = config.cost_price * 0.25; // 25% of unit cost as annual holding cost
         
-        reorder_qty = Math.ceil(Math.sqrt((2 * annual_demand * order_cost) / holding_cost_percent));
+        reorder_qty = Math.ceil(Math.sqrt((2 * annual_demand * order_cost) / holding_cost));
     } else {
         // If no sales data, use a basic calculation
         reorder_qty = Math.max(safety_stock, config.low_stock_threshold);

inventory-server/scripts/metrics/sales-forecasts.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateSalesForecasts(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateSalesForecasts(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +16,30 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+                AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+        `);
+        processedOrders = orderCount[0].count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting sales forecasts calculation',
@@ -26,7 +48,12 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // First, create a temporary table for forecast dates
@@ -65,10 +92,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for daily sales stats
         await connection.query(`
@@ -94,10 +131,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for product stats
         await connection.query(`
@@ -119,10 +166,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate product-level forecasts
         await connection.query(`
@@ -134,37 +191,76 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
                 confidence_level,
                 last_calculated_at
             )
+            WITH daily_stats AS (
+                SELECT 
+                    ds.pid,
+                    AVG(ds.daily_quantity) as avg_daily_qty,
+                    STDDEV(ds.daily_quantity) as std_daily_qty,
+                    COUNT(DISTINCT ds.day_count) as data_points,
+                    SUM(ds.day_count) as total_days,
+                    AVG(ds.daily_revenue) as avg_daily_revenue,
+                    STDDEV(ds.daily_revenue) as std_daily_revenue,
+                    MIN(ds.daily_quantity) as min_daily_qty,
+                    MAX(ds.daily_quantity) as max_daily_qty,
+                    -- Calculate variance without using LAG
+                    COALESCE(
+                        STDDEV(ds.daily_quantity) / NULLIF(AVG(ds.daily_quantity), 0),
+                        0
+                    ) as daily_variance_ratio
+                FROM temp_daily_sales ds
+                GROUP BY ds.pid
+                HAVING AVG(ds.daily_quantity) > 0
+            )
             SELECT 
                 ds.pid,
                 fd.forecast_date,
                 GREATEST(0, 
-                    AVG(ds.daily_quantity) * 
-                    (1 + COALESCE(sf.seasonality_factor, 0))
+                    ROUND(
+                        ds.avg_daily_qty * 
+                        (1 + COALESCE(sf.seasonality_factor, 0)) *
+                        CASE
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 1.5 THEN 0.85
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 1.0 THEN 0.9
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 0.5 THEN 0.95
+                            ELSE 1.0
+                        END,
+                        2
+                    )
                 ) as forecast_units,
                 GREATEST(0, 
-                    COALESCE(
-                        CASE 
-                            WHEN SUM(ds.day_count) >= 4 THEN AVG(ds.daily_revenue)
-                            ELSE ps.overall_avg_revenue
-                        END * 
-                        (1 + COALESCE(sf.seasonality_factor, 0)) *
-                        (0.95 + (RAND() * 0.1)),
-                        0
+                    ROUND(
+                        COALESCE(
+                            CASE 
+                                WHEN ds.data_points >= 4 THEN ds.avg_daily_revenue
+                                ELSE ps.overall_avg_revenue
+                            END * 
+                            (1 + COALESCE(sf.seasonality_factor, 0)) *
+                            CASE
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 1.5 THEN 0.85
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 1.0 THEN 0.9
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 0.5 THEN 0.95
+                                ELSE 1.0
+                            END,
+                            0
+                        ),
+                        2
                     )
                 ) as forecast_revenue,
                 CASE
-                    WHEN ps.total_days >= 60 THEN 90
-                    WHEN ps.total_days >= 30 THEN 80
-                    WHEN ps.total_days >= 14 THEN 70
+                    WHEN ds.total_days >= 60 AND ds.daily_variance_ratio < 0.5 THEN 90
+                    WHEN ds.total_days >= 60 THEN 85
+                    WHEN ds.total_days >= 30 AND ds.daily_variance_ratio < 0.5 THEN 80
+                    WHEN ds.total_days >= 30 THEN 75
+                    WHEN ds.total_days >= 14 AND ds.daily_variance_ratio < 0.5 THEN 70
+                    WHEN ds.total_days >= 14 THEN 65
                     ELSE 60
                 END as confidence_level,
                 NOW() as last_calculated_at
-            FROM temp_daily_sales ds
+            FROM daily_stats ds
             JOIN temp_product_stats ps ON ds.pid = ps.pid
             CROSS JOIN temp_forecast_dates fd
             LEFT JOIN sales_seasonality sf ON fd.month = sf.month
-            GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, ps.total_days, sf.seasonality_factor
-            HAVING AVG(ds.daily_quantity) > 0
+            GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, sf.seasonality_factor
             ON DUPLICATE KEY UPDATE
                 forecast_units = VALUES(forecast_units),
                 forecast_revenue = VALUES(forecast_revenue),
@@ -181,10 +277,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for category stats
         await connection.query(`
@@ -221,10 +327,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate category-level forecasts
         await connection.query(`
@@ -292,11 +408,33 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('sales_forecasts', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating sales forecasts');
         throw error;
     } finally {

inventory-server/scripts/metrics/time-aggregates.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateTimeAggregates(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateTimeAggregates(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +16,29 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
         }
 
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting time aggregates calculation',
@@ -26,7 +47,12 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Initial insert of time-based aggregates
@@ -42,9 +68,11 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 stock_received,
                 stock_ordered,
                 avg_price,
-                profit_margin
+                profit_margin,
+                inventory_value,
+                gmroi
             )
-            WITH sales_data AS (
+            WITH monthly_sales AS (
                 SELECT 
                     o.pid,
                     YEAR(o.date) as year,
@@ -55,17 +83,19 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     COUNT(DISTINCT o.order_number) as order_count,
                     AVG(o.price - COALESCE(o.discount, 0)) as avg_price,
                     CASE 
-                        WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) = 0 THEN 0
-                        ELSE ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - 
-                              SUM(COALESCE(p.cost_price, 0) * o.quantity)) / 
-                              SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
-                    END as profit_margin
+                        WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) > 0 
+                        THEN ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - SUM(COALESCE(p.cost_price, 0) * o.quantity)) 
+                            / SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
+                        ELSE 0 
+                    END as profit_margin,
+                    p.cost_price * p.stock_quantity as inventory_value,
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM orders o
                 JOIN products p ON o.pid = p.pid
-                WHERE o.canceled = 0
+                WHERE o.canceled = false
                 GROUP BY o.pid, YEAR(o.date), MONTH(o.date)
             ),
-            purchase_data AS (
+            monthly_stock AS (
                 SELECT 
                     pid,
                     YEAR(date) as year,
@@ -73,45 +103,100 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     SUM(received) as stock_received,
                     SUM(ordered) as stock_ordered
                 FROM purchase_orders
-                WHERE status = 50
                 GROUP BY pid, YEAR(date), MONTH(date)
+            ),
+            base_products AS (
+                SELECT 
+                    p.pid,
+                    p.cost_price * p.stock_quantity as inventory_value
+                FROM products p
             )
             SELECT 
-                s.pid,
-                s.year,
-                s.month,
-                s.total_quantity_sold,
-                s.total_revenue,
-                s.total_cost,
-                s.order_count,
-                COALESCE(p.stock_received, 0) as stock_received,
-                COALESCE(p.stock_ordered, 0) as stock_ordered,
-                s.avg_price,
-                s.profit_margin
-            FROM sales_data s
-            LEFT JOIN purchase_data p 
-                ON s.pid = p.pid 
-                AND s.year = p.year 
-                AND s.month = p.month
+                COALESCE(s.pid, ms.pid) as pid,
+                COALESCE(s.year, ms.year) as year,
+                COALESCE(s.month, ms.month) as month,
+                COALESCE(s.total_quantity_sold, 0) as total_quantity_sold,
+                COALESCE(s.total_revenue, 0) as total_revenue,
+                COALESCE(s.total_cost, 0) as total_cost,
+                COALESCE(s.order_count, 0) as order_count,
+                COALESCE(ms.stock_received, 0) as stock_received,
+                COALESCE(ms.stock_ordered, 0) as stock_ordered,
+                COALESCE(s.avg_price, 0) as avg_price,
+                COALESCE(s.profit_margin, 0) as profit_margin,
+                COALESCE(s.inventory_value, bp.inventory_value, 0) as inventory_value,
+                CASE 
+                    WHEN COALESCE(s.inventory_value, bp.inventory_value, 0) > 0 
+                        AND COALESCE(s.active_days, 0) > 0 
+                    THEN (COALESCE(s.total_revenue - s.total_cost, 0) * (365.0 / s.active_days)) 
+                        / COALESCE(s.inventory_value, bp.inventory_value)
+                    ELSE 0 
+                END as gmroi
+            FROM (
+                SELECT * FROM monthly_sales s
+                UNION ALL
+                SELECT 
+                    ms.pid,
+                    ms.year,
+                    ms.month,
+                    0 as total_quantity_sold,
+                    0 as total_revenue,
+                    0 as total_cost,
+                    0 as order_count,
+                    NULL as avg_price,
+                    0 as profit_margin,
+                    NULL as inventory_value,
+                    0 as active_days
+                FROM monthly_stock ms
+                WHERE NOT EXISTS (
+                    SELECT 1 FROM monthly_sales s2
+                    WHERE s2.pid = ms.pid
+                    AND s2.year = ms.year
+                    AND s2.month = ms.month
+                )
+            ) s
+            LEFT JOIN monthly_stock ms 
+                ON s.pid = ms.pid 
+                AND s.year = ms.year 
+                AND s.month = ms.month
+            JOIN base_products bp ON COALESCE(s.pid, ms.pid) = bp.pid
             UNION
             SELECT 
-                p.pid,
-                p.year,
-                p.month,
+                ms.pid,
+                ms.year,
+                ms.month,
                 0 as total_quantity_sold,
                 0 as total_revenue,
                 0 as total_cost,
                 0 as order_count,
-                p.stock_received,
-                p.stock_ordered,
+                ms.stock_received,
+                ms.stock_ordered,
                 0 as avg_price,
-                0 as profit_margin
-            FROM purchase_data p
-            LEFT JOIN sales_data s
-                ON p.pid = s.pid 
-                AND p.year = s.year 
-                AND p.month = s.month
-            WHERE s.pid IS NULL
+                0 as profit_margin,
+                bp.inventory_value,
+                0 as gmroi
+            FROM monthly_stock ms
+            JOIN base_products bp ON ms.pid = bp.pid
+            WHERE NOT EXISTS (
+                SELECT 1 FROM (
+                    SELECT * FROM monthly_sales
+                    UNION ALL
+                    SELECT 
+                        ms2.pid,
+                        ms2.year,
+                        ms2.month,
+                        0, 0, 0, 0, NULL, 0, NULL, 0
+                    FROM monthly_stock ms2
+                    WHERE NOT EXISTS (
+                        SELECT 1 FROM monthly_sales s2
+                        WHERE s2.pid = ms2.pid
+                        AND s2.year = ms2.year
+                        AND s2.month = ms2.month
+                    )
+                ) s
+                WHERE s.pid = ms.pid
+                AND s.year = ms.year
+                AND s.month = ms.month
+            )
             ON DUPLICATE KEY UPDATE
                 total_quantity_sold = VALUES(total_quantity_sold),
                 total_revenue = VALUES(total_revenue),
@@ -120,7 +205,9 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 stock_received = VALUES(stock_received),
                 stock_ordered = VALUES(stock_ordered),
                 avg_price = VALUES(avg_price),
-                profit_margin = VALUES(profit_margin)
+                profit_margin = VALUES(profit_margin),
+                inventory_value = VALUES(inventory_value),
+                gmroi = VALUES(gmroi)
         `);
 
         processedCount = Math.floor(totalProducts * 0.60);
@@ -132,10 +219,20 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Update with financial metrics
         await connection.query(`
@@ -147,7 +244,7 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     MONTH(o.date) as month,
                     p.cost_price * p.stock_quantity as inventory_value,
                     SUM(o.quantity * (o.price - p.cost_price)) as gross_profit,
-                    COUNT(DISTINCT DATE(o.date)) as days_in_period
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM products p
                 LEFT JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false
@@ -156,12 +253,7 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 AND pta.year = fin.year 
                 AND pta.month = fin.month
             SET 
-                pta.inventory_value = COALESCE(fin.inventory_value, 0),
-                pta.gmroi = CASE 
-                    WHEN COALESCE(fin.inventory_value, 0) > 0 AND fin.days_in_period > 0 THEN 
-                        (COALESCE(fin.gross_profit, 0) * (365.0 / fin.days_in_period)) / COALESCE(fin.inventory_value, 0)
-                    ELSE 0 
-                END
+                pta.inventory_value = COALESCE(fin.inventory_value, 0)
         `);
 
         processedCount = Math.floor(totalProducts * 0.65);
@@ -173,11 +265,33 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('time_aggregates', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating time aggregates');
         throw error;
     } finally {

inventory-server/scripts/metrics/vendor-metrics.js

@@ -1,8 +1,12 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateVendorMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateVendorMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    let processedPurchaseOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,11 +17,37 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            return processedCount;
+            return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders,
+                success
+            };
         }
 
+        // Get counts of records that will be processed
+        const [[orderCount], [poCount]] = await Promise.all([
+            connection.query(`
+                SELECT COUNT(*) as count 
+                FROM orders o 
+                WHERE o.canceled = false
+            `),
+            connection.query(`
+                SELECT COUNT(*) as count 
+                FROM purchase_orders po 
+                WHERE po.status != 0
+            `)
+        ]);
+        processedOrders = orderCount.count;
+        processedPurchaseOrders = poCount.count;
+
         outputProgress({
             status: 'running',
             operation: 'Starting vendor metrics calculation',
@@ -26,7 +56,12 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // First ensure all vendors exist in vendor_details
@@ -50,10 +85,20 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
 
         // Now calculate vendor metrics
         await connection.query(`
@@ -68,6 +113,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 avg_order_value,
                 active_products,
                 total_products,
+                total_purchase_value,
+                avg_margin_percent,
                 status,
                 last_calculated_at
             )
@@ -76,7 +123,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                     p.vendor,
                     SUM(o.quantity * o.price) as total_revenue,
                     COUNT(DISTINCT o.id) as total_orders,
-                    COUNT(DISTINCT p.pid) as active_products
+                    COUNT(DISTINCT p.pid) as active_products,
+                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
                 FROM products p
                 JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false
@@ -91,7 +139,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                     AVG(CASE 
                         WHEN po.receiving_status = 40 
                         THEN DATEDIFF(po.received_date, po.date) 
-                    END) as avg_lead_time_days
+                    END) as avg_lead_time_days,
+                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
                 FROM products p
                 JOIN purchase_orders po ON p.pid = po.pid
                 WHERE po.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
@@ -127,6 +176,12 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 END as avg_order_value,
                 COALESCE(vs.active_products, 0) as active_products,
                 COALESCE(vpr.total_products, 0) as total_products,
+                COALESCE(vp.total_purchase_value, 0) as total_purchase_value,
+                CASE 
+                    WHEN vs.total_revenue > 0 
+                    THEN (vs.total_margin / vs.total_revenue) * 100 
+                    ELSE 0 
+                END as avg_margin_percent,
                 'active' as status,
                 NOW() as last_calculated_at
             FROM vendor_sales vs
@@ -143,6 +198,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 avg_order_value = VALUES(avg_order_value),
                 active_products = VALUES(active_products),
                 total_products = VALUES(total_products),
+                total_purchase_value = VALUES(total_purchase_value),
+                avg_margin_percent = VALUES(avg_margin_percent),
                 status = VALUES(status),
                 last_calculated_at = VALUES(last_calculated_at)
         `);
@@ -150,17 +207,155 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
         processedCount = Math.floor(totalProducts * 0.9);
         outputProgress({
             status: 'running',
-            operation: 'Vendor metrics calculated',
+            operation: 'Vendor metrics calculated, updating time-based metrics',
             current: processedCount,
             total: totalProducts,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
+
+        // Calculate time-based metrics
+        await connection.query(`
+            INSERT INTO vendor_time_metrics (
+                vendor,
+                year,
+                month,
+                total_orders,
+                late_orders,
+                avg_lead_time_days,
+                total_purchase_value,
+                total_revenue,
+                avg_margin_percent
+            )
+            WITH monthly_orders AS (
+                SELECT 
+                    p.vendor,
+                    YEAR(o.date) as year,
+                    MONTH(o.date) as month,
+                    COUNT(DISTINCT o.id) as total_orders,
+                    SUM(o.quantity * o.price) as total_revenue,
+                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
+                FROM products p
+                JOIN orders o ON p.pid = o.pid
+                WHERE o.canceled = false
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND p.vendor IS NOT NULL
+                GROUP BY p.vendor, YEAR(o.date), MONTH(o.date)
+            ),
+            monthly_po AS (
+                SELECT 
+                    p.vendor,
+                    YEAR(po.date) as year,
+                    MONTH(po.date) as month,
+                    COUNT(DISTINCT po.id) as total_po,
+                    COUNT(DISTINCT CASE 
+                        WHEN po.receiving_status = 40 AND po.received_date > po.expected_date 
+                        THEN po.id 
+                    END) as late_orders,
+                    AVG(CASE 
+                        WHEN po.receiving_status = 40 
+                        THEN DATEDIFF(po.received_date, po.date)
+                    END) as avg_lead_time_days,
+                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
+                FROM products p
+                JOIN purchase_orders po ON p.pid = po.pid
+                WHERE po.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND p.vendor IS NOT NULL
+                GROUP BY p.vendor, YEAR(po.date), MONTH(po.date)
+            )
+            SELECT 
+                mo.vendor,
+                mo.year,
+                mo.month,
+                COALESCE(mp.total_po, 0) as total_orders,
+                COALESCE(mp.late_orders, 0) as late_orders,
+                COALESCE(mp.avg_lead_time_days, 0) as avg_lead_time_days,
+                COALESCE(mp.total_purchase_value, 0) as total_purchase_value,
+                mo.total_revenue,
+                CASE 
+                    WHEN mo.total_revenue > 0 
+                    THEN (mo.total_margin / mo.total_revenue) * 100 
+                    ELSE 0 
+                END as avg_margin_percent
+            FROM monthly_orders mo
+            LEFT JOIN monthly_po mp ON mo.vendor = mp.vendor 
+                AND mo.year = mp.year 
+                AND mo.month = mp.month
+            UNION
+            SELECT 
+                mp.vendor,
+                mp.year,
+                mp.month,
+                mp.total_po as total_orders,
+                mp.late_orders,
+                mp.avg_lead_time_days,
+                mp.total_purchase_value,
+                0 as total_revenue,
+                0 as avg_margin_percent
+            FROM monthly_po mp
+            LEFT JOIN monthly_orders mo ON mp.vendor = mo.vendor 
+                AND mp.year = mo.year 
+                AND mp.month = mo.month
+            WHERE mo.vendor IS NULL
+            ON DUPLICATE KEY UPDATE
+                total_orders = VALUES(total_orders),
+                late_orders = VALUES(late_orders),
+                avg_lead_time_days = VALUES(avg_lead_time_days),
+                total_purchase_value = VALUES(total_purchase_value),
+                total_revenue = VALUES(total_revenue),
+                avg_margin_percent = VALUES(avg_margin_percent)
+        `);
+
+        processedCount = Math.floor(totalProducts * 0.95);
+        outputProgress({
+            status: 'running',
+            operation: 'Time-based vendor metrics calculated',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('vendor_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating vendor metrics');
         throw error;
     } finally {

inventory-server/scripts/reset-db.js

@@ -1,4 +1,4 @@
-const mysql = require('mysql2/promise');
+const { Client } = require('pg');
 const path = require('path');
 const dotenv = require('dotenv');
 const fs = require('fs');
@@ -10,7 +10,7 @@ const dbConfig = {
     user: process.env.DB_USER,
     password: process.env.DB_PASSWORD,
     database: process.env.DB_NAME,
-    multipleStatements: true
+    port: process.env.DB_PORT || 5432
 };
 
 // Helper function to output progress in JSON format
@@ -54,14 +54,44 @@ function splitSQLStatements(sql) {
     let currentStatement = '';
     let inString = false;
     let stringChar = '';
+    let inDollarQuote = false;
+    let dollarQuoteTag = '';
     
     // Process character by character
     for (let i = 0; i < sql.length; i++) {
         const char = sql[i];
         const nextChar = sql[i + 1] || '';
         
-        // Handle string literals
-        if ((char === "'" || char === '"') && sql[i - 1] !== '\\') {
+        // Handle dollar quotes
+        if (char === '$' && !inString) {
+            // Look ahead to find the dollar quote tag
+            let tag = '$';
+            let j = i + 1;
+            while (j < sql.length && sql[j] !== '$') {
+                tag += sql[j];
+                j++;
+            }
+            tag += '$';
+            
+            if (j < sql.length) { // Found closing $
+                if (!inDollarQuote) {
+                    inDollarQuote = true;
+                    dollarQuoteTag = tag;
+                    currentStatement += tag;
+                    i = j;
+                    continue;
+                } else if (sql.substring(i, j + 1) === dollarQuoteTag) {
+                    inDollarQuote = false;
+                    dollarQuoteTag = '';
+                    currentStatement += tag;
+                    i = j;
+                    continue;
+                }
+            }
+        }
+        
+        // Handle string literals (only if not in dollar quote)
+        if (!inDollarQuote && (char === "'" || char === '"') && sql[i - 1] !== '\\') {
             if (!inString) {
                 inString = true;
                 stringChar = char;
@@ -70,23 +100,25 @@ function splitSQLStatements(sql) {
             }
         }
         
-        // Handle comments
-        if (!inString && char === '-' && nextChar === '-') {
-            // Skip to end of line
-            while (i < sql.length && sql[i] !== '\n') i++;
-            continue;
+        // Handle comments (only if not in string or dollar quote)
+        if (!inString && !inDollarQuote) {
+            if (char === '-' && nextChar === '-') {
+                // Skip to end of line
+                while (i < sql.length && sql[i] !== '\n') i++;
+                continue;
+            }
+            
+            if (char === '/' && nextChar === '*') {
+                // Skip until closing */
+                i += 2;
+                while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
+                i++; // Skip the closing /
+                continue;
+            }
         }
         
-        if (!inString && char === '/' && nextChar === '*') {
-            // Skip until closing */
-            i += 2;
-            while (i < sql.length && (sql[i] !== '*' || sql[i + 1] !== '/')) i++;
-            i++; // Skip the closing /
-            continue;
-        }
-        
-        // Handle statement boundaries
-        if (!inString && char === ';') {
+        // Handle statement boundaries (only if not in string or dollar quote)
+        if (!inString && !inDollarQuote && char === ';') {
             if (currentStatement.trim()) {
                 statements.push(currentStatement.trim());
             }
@@ -120,30 +152,26 @@ async function resetDatabase() {
         }
     });
 
-    const connection = await mysql.createConnection(dbConfig);
+    const client = new Client(dbConfig);
+    await client.connect();
     
     try {
-        // Check MySQL privileges
+        // Check PostgreSQL version and user
         outputProgress({
-            operation: 'Checking privileges',
-            message: 'Verifying MySQL user privileges...'
+            operation: 'Checking database',
+            message: 'Verifying PostgreSQL version and user privileges...'
         });
 
-        const [grants] = await connection.query('SHOW GRANTS');
-        outputProgress({
-            operation: 'User privileges',
-            message: {
-                grants: grants.map(g => Object.values(g)[0])
-            }
-        });
-
-        // Enable warnings as errors
-        await connection.query('SET SESSION sql_notes = 1');
+        const versionResult = await client.query('SELECT version()');
+        const userResult = await client.query('SELECT current_user, current_database()');
         
-        // Log database config (without sensitive info)
         outputProgress({
-            operation: 'Database config',
-            message: `Using database: ${dbConfig.database} on host: ${dbConfig.host}`
+            operation: 'Database info',
+            message: {
+                version: versionResult.rows[0].version,
+                user: userResult.rows[0].current_user,
+                database: userResult.rows[0].current_database
+            }
         });
 
         // Get list of all tables in the current database
@@ -152,14 +180,14 @@ async function resetDatabase() {
             message: 'Retrieving all table names...'
         });
 
-        const [tables] = await connection.query(`
-            SELECT GROUP_CONCAT(table_name) as tables 
-            FROM information_schema.tables 
-            WHERE table_schema = DATABASE()
-            AND table_name NOT IN ('users', 'import_history')
+        const tablesResult = await client.query(`
+            SELECT string_agg(tablename, ', ') as tables
+            FROM pg_tables
+            WHERE schemaname = 'public'
+            AND tablename NOT IN ('users', 'calculate_history', 'import_history');
         `);
 
-        if (!tables[0].tables) {
+        if (!tablesResult.rows[0].tables) {
             outputProgress({
                 operation: 'No tables found',
                 message: 'Database is already empty'
@@ -170,20 +198,73 @@ async function resetDatabase() {
                 message: 'Dropping all existing tables...'
             });
 
-            await connection.query('SET FOREIGN_KEY_CHECKS = 0');
-            const dropQuery = `
-                DROP TABLE IF EXISTS 
-                ${tables[0].tables
-                    .split(',')
-                    .filter(table => table !== 'users')
-                    .map(table => '`' + table + '`')
-                    .join(', ')}
-            `;
-            await connection.query(dropQuery);
-            await connection.query('SET FOREIGN_KEY_CHECKS = 1');
+            // Disable triggers/foreign key checks
+            await client.query('SET session_replication_role = \'replica\';');
+            
+            // Drop all tables except users
+            const tables = tablesResult.rows[0].tables.split(', ');
+            for (const table of tables) {
+                if (!['users'].includes(table)) {
+                    await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
+                }
+            }
+
+            // Only drop types if we're not preserving history tables
+            const historyTablesExist = await client.query(`
+                SELECT EXISTS (
+                    SELECT FROM pg_tables 
+                    WHERE schemaname = 'public' 
+                    AND tablename IN ('calculate_history', 'import_history')
+                );
+            `);
+
+            if (!historyTablesExist.rows[0].exists) {
+                await client.query('DROP TYPE IF EXISTS calculation_status CASCADE;');
+                await client.query('DROP TYPE IF EXISTS module_name CASCADE;');
+            }
+            
+            // Re-enable triggers/foreign key checks
+            await client.query('SET session_replication_role = \'origin\';');
         }
 
-        // Read and execute main schema (core tables)
+        // Create enum types if they don't exist
+        outputProgress({
+            operation: 'Creating enum types',
+            message: 'Setting up required enum types...'
+        });
+
+        // Check if types exist before creating
+        const typesExist = await client.query(`
+            SELECT EXISTS (
+                SELECT 1 FROM pg_type 
+                WHERE typname = 'calculation_status'
+            ) as calc_status_exists,
+            EXISTS (
+                SELECT 1 FROM pg_type 
+                WHERE typname = 'module_name'
+            ) as module_name_exists;
+        `);
+
+        if (!typesExist.rows[0].calc_status_exists) {
+            await client.query(`CREATE TYPE calculation_status AS ENUM ('running', 'completed', 'failed', 'cancelled')`);
+        }
+
+        if (!typesExist.rows[0].module_name_exists) {
+            await client.query(`
+                CREATE TYPE module_name AS ENUM (
+                    'product_metrics',
+                    'time_aggregates',
+                    'financial_metrics',
+                    'vendor_metrics',
+                    'category_metrics',
+                    'brand_metrics',
+                    'sales_forecasts',
+                    'abc_classification'
+                )
+            `);
+        }
+
+        // Read and execute main schema first (core tables)
         outputProgress({
             operation: 'Running database setup',
             message: 'Creating core tables...'
@@ -223,35 +304,24 @@ async function resetDatabase() {
         for (let i = 0; i < statements.length; i++) {
             const stmt = statements[i];
             try {
-                const [result, fields] = await connection.query(stmt);
-                
-                // Check for warnings
-                const [warnings] = await connection.query('SHOW WARNINGS');
-                if (warnings && warnings.length > 0) {
-                    outputProgress({
-                        status: 'warning',
-                        operation: 'SQL Warning',
-                        statement: i + 1,
-                        warnings: warnings
-                    });
-                }
+                const result = await client.query(stmt);
                 
                 // Verify if table was created (if this was a CREATE TABLE statement)
                 if (stmt.trim().toLowerCase().startsWith('create table')) {
-                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?`?(\w+)`?/i)?.[1];
+                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(\w+)["]?/i)?.[1];
                     if (tableName) {
-                        const [tableExists] = await connection.query(`
+                        const tableExists = await client.query(`
                             SELECT COUNT(*) as count 
                             FROM information_schema.tables 
-                            WHERE table_schema = DATABASE() 
-                            AND table_name = ?
+                            WHERE table_schema = 'public' 
+                            AND table_name = $1
                         `, [tableName]);
                         
                         outputProgress({
                             operation: 'Table Creation Verification',
                             message: {
                                 table: tableName,
-                                exists: tableExists[0].count > 0
+                                exists: tableExists.rows[0].count > 0
                             }
                         });
                     }
@@ -263,7 +333,7 @@ async function resetDatabase() {
                         statement: i + 1,
                         total: statements.length,
                         preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                        affectedRows: result.affectedRows
+                        rowCount: result.rowCount
                     }
                 });
             } catch (sqlError) {
@@ -271,8 +341,6 @@ async function resetDatabase() {
                     status: 'error',
                     operation: 'SQL Error',
                     error: sqlError.message,
-                    sqlState: sqlError.sqlState,
-                    errno: sqlError.errno,
                     statement: stmt,
                     statementNumber: i + 1
                 });
@@ -280,66 +348,12 @@ async function resetDatabase() {
             }
         }
 
-        // List all tables in the database after schema execution
-        outputProgress({
-            operation: 'Debug database',
-            message: {
-                currentDatabase: (await connection.query('SELECT DATABASE() as db'))[0][0].db
-            }
-        });
-
-        const [allTables] = await connection.query(`
-            SELECT 
-                table_schema,
-                table_name,
-                engine,
-                create_time,
-                table_rows
+        // Verify core tables were created
+        const existingTables = (await client.query(`
+            SELECT table_name
             FROM information_schema.tables 
-            WHERE table_schema = DATABASE()
-        `);
-
-        if (allTables.length === 0) {
-            outputProgress({
-                operation: 'Warning',
-                message: 'No tables found in database after schema execution'
-            });
-        } else {
-            outputProgress({
-                operation: 'Tables after schema execution',
-                message: {
-                    count: allTables.length,
-                    tables: allTables.map(t => ({
-                        schema: t.table_schema,
-                        name: t.table_name,
-                        engine: t.engine,
-                        created: t.create_time,
-                        rows: t.table_rows
-                    }))
-                }
-            });
-        }
-
-        // Also check table status
-        const [tableStatus] = await connection.query('SHOW TABLE STATUS');
-        outputProgress({
-            operation: 'Table Status',
-            message: {
-                tables: tableStatus.map(t => ({
-                    name: t.Name,
-                    engine: t.Engine,
-                    version: t.Version,
-                    rowFormat: t.Row_format,
-                    rows: t.Rows,
-                    createTime: t.Create_time,
-                    updateTime: t.Update_time
-                }))
-            }
-        });
-
-        // Verify core tables were created using SHOW TABLES
-        const [showTables] = await connection.query('SHOW TABLES');
-        const existingTables = showTables.map(t => Object.values(t)[0]);
+            WHERE table_schema = 'public'
+        `)).rows.map(t => t.table_name);
 
         outputProgress({
             operation: 'Core tables verification',
@@ -359,22 +373,12 @@ async function resetDatabase() {
             );
         }
 
-        // Verify all core tables use InnoDB
-        const [engineStatus] = await connection.query('SHOW TABLE STATUS WHERE Name IN (?)', [CORE_TABLES]);
-        const nonInnoDBTables = engineStatus.filter(t => t.Engine !== 'InnoDB');
-        
-        if (nonInnoDBTables.length > 0) {
-            throw new Error(
-                `Tables using non-InnoDB engine: ${nonInnoDBTables.map(t => t.Name).join(', ')}`
-            );
-        }
-
         outputProgress({
             operation: 'Core tables created',
             message: `Successfully created tables: ${CORE_TABLES.join(', ')}`
         });
 
-        // Read and execute config schema
+        // Now read and execute config schema (since core tables exist)
         outputProgress({
             operation: 'Running config setup',
             message: 'Creating configuration tables...'
@@ -400,18 +404,7 @@ async function resetDatabase() {
         for (let i = 0; i < configStatements.length; i++) {
             const stmt = configStatements[i];
             try {
-                const [result, fields] = await connection.query(stmt);
-                
-                // Check for warnings
-                const [warnings] = await connection.query('SHOW WARNINGS');
-                if (warnings && warnings.length > 0) {
-                    outputProgress({
-                        status: 'warning',
-                        operation: 'Config SQL Warning',
-                        statement: i + 1,
-                        warnings: warnings
-                    });
-                }
+                const result = await client.query(stmt);
                 
                 outputProgress({
                     operation: 'Config SQL Progress',
@@ -419,7 +412,7 @@ async function resetDatabase() {
                         statement: i + 1,
                         total: configStatements.length,
                         preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                        affectedRows: result.affectedRows
+                        rowCount: result.rowCount
                     }
                 });
             } catch (sqlError) {
@@ -427,8 +420,6 @@ async function resetDatabase() {
                     status: 'error',
                     operation: 'Config SQL Error',
                     error: sqlError.message,
-                    sqlState: sqlError.sqlState,
-                    errno: sqlError.errno,
                     statement: stmt,
                     statementNumber: i + 1
                 });
@@ -436,33 +427,6 @@ async function resetDatabase() {
             }
         }
 
-        // Verify config tables were created
-        const [showConfigTables] = await connection.query('SHOW TABLES');
-        const existingConfigTables = showConfigTables.map(t => Object.values(t)[0]);
-
-        outputProgress({
-            operation: 'Config tables verification',
-            message: {
-                found: existingConfigTables,
-                expected: CONFIG_TABLES
-            }
-        });
-
-        const missingConfigTables = CONFIG_TABLES.filter(
-            t => !existingConfigTables.includes(t)
-        );
-
-        if (missingConfigTables.length > 0) {
-            throw new Error(
-                `Failed to create config tables: ${missingConfigTables.join(', ')}`
-            );
-        }
-
-        outputProgress({
-            operation: 'Config tables created',
-            message: `Successfully created tables: ${CONFIG_TABLES.join(', ')}`
-        });
-
         // Read and execute metrics schema (metrics tables)
         outputProgress({
             operation: 'Running metrics setup',
@@ -489,18 +453,7 @@ async function resetDatabase() {
         for (let i = 0; i < metricsStatements.length; i++) {
             const stmt = metricsStatements[i];
             try {
-                const [result, fields] = await connection.query(stmt);
-                
-                // Check for warnings
-                const [warnings] = await connection.query('SHOW WARNINGS');
-                if (warnings && warnings.length > 0) {
-                    outputProgress({
-                        status: 'warning',
-                        operation: 'Metrics SQL Warning',
-                        statement: i + 1,
-                        warnings: warnings
-                    });
-                }
+                const result = await client.query(stmt);
                 
                 outputProgress({
                     operation: 'Metrics SQL Progress',
@@ -508,7 +461,7 @@ async function resetDatabase() {
                         statement: i + 1,
                         total: metricsStatements.length,
                         preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
-                        affectedRows: result.affectedRows
+                        rowCount: result.rowCount
                     }
                 });
             } catch (sqlError) {
@@ -516,8 +469,6 @@ async function resetDatabase() {
                     status: 'error',
                     operation: 'Metrics SQL Error',
                     error: sqlError.message,
-                    sqlState: sqlError.sqlState,
-                    errno: sqlError.errno,
                     statement: stmt,
                     statementNumber: i + 1
                 });
@@ -539,9 +490,19 @@ async function resetDatabase() {
         });
         process.exit(1);
     } finally {
-        await connection.end();
+        await client.end();
     }
 }
 
-// Run the reset
-resetDatabase(); 
+// Export if required as a module
+if (typeof module !== 'undefined' && module.exports) {
+    module.exports = resetDatabase;
+}
+
+// Run if called directly
+if (require.main === module) {
+    resetDatabase().catch(error => {
+        console.error('Error:', error);
+        process.exit(1);
+    });
+} 

inventory-server/scripts/reset-metrics.js

@@ -1,4 +1,4 @@
-const mysql = require('mysql2/promise');
+const { Client } = require('pg');
 const path = require('path');
 const fs = require('fs');
 require('dotenv').config({ path: path.resolve(__dirname, '../.env') });
@@ -8,7 +8,7 @@ const dbConfig = {
     user: process.env.DB_USER,
     password: process.env.DB_PASSWORD,
     database: process.env.DB_NAME,
-    multipleStatements: true
+    port: process.env.DB_PORT || 5432
 };
 
 function outputProgress(data) {
@@ -34,8 +34,8 @@ const METRICS_TABLES = [
     'sales_forecasts',
     'temp_purchase_metrics',
     'temp_sales_metrics',
-    'vendor_metrics', //before vendor_details for foreign key
-    'vendor_time_metrics', //before vendor_details for foreign key
+    'vendor_metrics',
+    'vendor_time_metrics',
     'vendor_details'
 ];
 
@@ -90,31 +90,31 @@ function splitSQLStatements(sql) {
 }
 
 async function resetMetrics() {
-    let connection;
+    let client;
     try {
         outputProgress({
             operation: 'Starting metrics reset',
             message: 'Connecting to database...'
         });
 
-        connection = await mysql.createConnection(dbConfig);
-        await connection.beginTransaction();
+        client = new Client(dbConfig);
+        await client.connect();
 
         // First verify current state
-        const [initialTables] = await connection.query(`
-            SELECT TABLE_NAME as name
-            FROM information_schema.tables 
-            WHERE TABLE_SCHEMA = DATABASE()
-            AND TABLE_NAME IN (?)
+        const initialTables = await client.query(`
+            SELECT tablename as name
+            FROM pg_tables 
+            WHERE schemaname = 'public'
+            AND tablename = ANY($1)
         `, [METRICS_TABLES]);
 
         outputProgress({
             operation: 'Initial state',
-            message: `Found ${initialTables.length} existing metrics tables: ${initialTables.map(t => t.name).join(', ')}`
+            message: `Found ${initialTables.rows.length} existing metrics tables: ${initialTables.rows.map(t => t.name).join(', ')}`
         });
 
         // Disable foreign key checks at the start
-        await connection.query('SET FOREIGN_KEY_CHECKS = 0');
+        await client.query('SET session_replication_role = \'replica\'');
 
         // Drop all metrics tables in reverse order to handle dependencies
         outputProgress({
@@ -124,17 +124,17 @@ async function resetMetrics() {
 
         for (const table of [...METRICS_TABLES].reverse()) {
             try {
-                await connection.query(`DROP TABLE IF EXISTS ${table}`);
+                await client.query(`DROP TABLE IF EXISTS "${table}" CASCADE`);
                 
                 // Verify the table was actually dropped
-                const [checkDrop] = await connection.query(`
+                const checkDrop = await client.query(`
                     SELECT COUNT(*) as count 
-                    FROM information_schema.tables 
-                    WHERE TABLE_SCHEMA = DATABASE() 
-                    AND TABLE_NAME = ?
+                    FROM pg_tables 
+                    WHERE schemaname = 'public' 
+                    AND tablename = $1
                 `, [table]);
                 
-                if (checkDrop[0].count > 0) {
+                if (parseInt(checkDrop.rows[0].count) > 0) {
                     throw new Error(`Failed to drop table ${table} - table still exists`);
                 }
 
@@ -153,15 +153,15 @@ async function resetMetrics() {
         }
 
         // Verify all tables were dropped
-        const [afterDrop] = await connection.query(`
-            SELECT TABLE_NAME as name
-            FROM information_schema.tables 
-            WHERE TABLE_SCHEMA = DATABASE()
-            AND TABLE_NAME IN (?)
+        const afterDrop = await client.query(`
+            SELECT tablename as name
+            FROM pg_tables 
+            WHERE schemaname = 'public'
+            AND tablename = ANY($1)
         `, [METRICS_TABLES]);
 
-        if (afterDrop.length > 0) {
-            throw new Error(`Failed to drop all tables. Remaining tables: ${afterDrop.map(t => t.name).join(', ')}`);
+        if (afterDrop.rows.length > 0) {
+            throw new Error(`Failed to drop all tables. Remaining tables: ${afterDrop.rows.map(t => t.name).join(', ')}`);
         }
 
         // Read metrics schema
@@ -187,39 +187,26 @@ async function resetMetrics() {
         for (let i = 0; i < statements.length; i++) {
             const stmt = statements[i];
             try {
-                await connection.query(stmt);
-                
-                // Check for warnings
-                const [warnings] = await connection.query('SHOW WARNINGS');
-                if (warnings && warnings.length > 0) {
-                    outputProgress({
-                        status: 'warning',
-                        operation: 'SQL Warning',
-                        message: {
-                            statement: i + 1,
-                            warnings: warnings
-                        }
-                    });
-                }
+                const result = await client.query(stmt);
 
                 // If this is a CREATE TABLE statement, verify the table was created
                 if (stmt.trim().toLowerCase().startsWith('create table')) {
-                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?`?(\w+)`?/i)?.[1];
+                    const tableName = stmt.match(/create\s+table\s+(?:if\s+not\s+exists\s+)?["]?(\w+)["]?/i)?.[1];
                     if (tableName) {
-                        const [checkCreate] = await connection.query(`
-                            SELECT TABLE_NAME as name, CREATE_TIME as created
-                            FROM information_schema.tables 
-                            WHERE TABLE_SCHEMA = DATABASE() 
-                            AND TABLE_NAME = ?
+                        const checkCreate = await client.query(`
+                            SELECT tablename as name
+                            FROM pg_tables 
+                            WHERE schemaname = 'public' 
+                            AND tablename = $1
                         `, [tableName]);
                         
-                        if (checkCreate.length === 0) {
+                        if (checkCreate.rows.length === 0) {
                             throw new Error(`Failed to create table ${tableName} - table does not exist after CREATE statement`);
                         }
 
                         outputProgress({
                             operation: 'Table created',
-                            message: `Successfully created table: ${tableName} at ${checkCreate[0].created}`
+                            message: `Successfully created table: ${tableName}`
                         });
                     }
                 }
@@ -229,7 +216,8 @@ async function resetMetrics() {
                     message: {
                         statement: i + 1,
                         total: statements.length,
-                        preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : '')
+                        preview: stmt.substring(0, 100) + (stmt.length > 100 ? '...' : ''),
+                        rowCount: result.rowCount
                     }
                 });
             } catch (sqlError) {
@@ -238,8 +226,6 @@ async function resetMetrics() {
                     operation: 'SQL Error',
                     message: {
                         error: sqlError.message,
-                        sqlState: sqlError.sqlState,
-                        errno: sqlError.errno,
                         statement: stmt,
                         statementNumber: i + 1
                     }
@@ -249,7 +235,7 @@ async function resetMetrics() {
         }
 
         // Re-enable foreign key checks after all tables are created
-        await connection.query('SET FOREIGN_KEY_CHECKS = 1');
+        await client.query('SET session_replication_role = \'origin\'');
 
         // Verify metrics tables were created
         outputProgress({
@@ -257,37 +243,36 @@ async function resetMetrics() {
             message: 'Checking all metrics tables were created...'
         });
 
-        const [metricsTablesResult] = await connection.query(`
-            SELECT 
-                TABLE_NAME as name, 
-                TABLE_ROWS as \`rows\`,
-                CREATE_TIME as created
-            FROM information_schema.tables 
-            WHERE TABLE_SCHEMA = DATABASE()
-            AND TABLE_NAME IN (?)
+        const metricsTablesResult = await client.query(`
+            SELECT tablename as name
+            FROM pg_tables 
+            WHERE schemaname = 'public'
+            AND tablename = ANY($1)
         `, [METRICS_TABLES]);
 
         outputProgress({
             operation: 'Tables found',
-            message: `Found ${metricsTablesResult.length} tables: ${metricsTablesResult.map(t => 
-                `${t.name} (created: ${t.created})`
-            ).join(', ')}`
+            message: `Found ${metricsTablesResult.rows.length} tables: ${metricsTablesResult.rows.map(t => t.name).join(', ')}`
         });
 
-        const existingMetricsTables = metricsTablesResult.map(t => t.name);
+        const existingMetricsTables = metricsTablesResult.rows.map(t => t.name);
         const missingMetricsTables = METRICS_TABLES.filter(t => !existingMetricsTables.includes(t));
 
         if (missingMetricsTables.length > 0) {
             // Do one final check of the actual tables
-            const [finalCheck] = await connection.query('SHOW TABLES');
+            const finalCheck = await client.query(`
+                SELECT tablename as name
+                FROM pg_tables 
+                WHERE schemaname = 'public'
+            `);
             outputProgress({
                 operation: 'Final table check',
-                message: `All database tables: ${finalCheck.map(t => Object.values(t)[0]).join(', ')}`
+                message: `All database tables: ${finalCheck.rows.map(t => t.name).join(', ')}`
             });
             throw new Error(`Failed to create metrics tables: ${missingMetricsTables.join(', ')}`);
         }
 
-        await connection.commit();
+        await client.query('COMMIT');
         
         outputProgress({
             status: 'complete',
@@ -302,17 +287,17 @@ async function resetMetrics() {
             stack: error.stack
         });
         
-        if (connection) {
-            await connection.rollback();
+        if (client) {
+            await client.query('ROLLBACK');
             // Make sure to re-enable foreign key checks even if there's an error
-            await connection.query('SET FOREIGN_KEY_CHECKS = 1').catch(() => {});
+            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
         }
         throw error;
     } finally {
-        if (connection) {
+        if (client) {
             // One final attempt to ensure foreign key checks are enabled
-            await connection.query('SET FOREIGN_KEY_CHECKS = 1').catch(() => {});
-            await connection.end();
+            await client.query('SET session_replication_role = \'origin\'').catch(() => {});
+            await client.end();
         }
     }
 }

inventory-server/src/prompts/product-validation.txt

@@ -0,0 +1,224 @@
+I will provide a JSON array with product data. Process the array by combining all products from validData and invalidData arrays into a single array, excluding any fields starting with “__”, such as “__index” or “__errors”. Process each product according to the reference guidelines below. If a field is not included in the data, do not include it in your response (e.g. do not include its key or any value) unless the specific field guidelines below say otherwise. If a product appears to be from an empty or entirely invalid line, do not include it in your response.
+
+Your response should be a JSON object with the following structure:
+{
+  "correctedData": [], // Array of corrected products
+  "changes": [], // Array of strings describing each change made
+  "warnings": [] // Array of strings with warnings or suggestions for manual review (see below for details)
+}
+
+Using the provided guidelines, focus on:
+1. Correcting typos and any incorrect spelling or grammar
+2. Standardizing product names
+3. Correcting and enhancing descriptions by adding details, keywords, and SEO-friendly language
+4. Fixing any obvious errors or inconsistencies between similar products in measurements, prices, or quantities
+5. Adding correct categories, themes, and colors
+
+Use only the provided data and your own knowledge to make changes. Do not make assumptions or make up information that you're not sure about. If you're unable to make a change you're confident about, leave the field as is. All data passed in should be validated, corrected, and returned. All values returned should be strings, not numbers. Do not leave out any fields that were present in the original data.
+
+Possible reasons for including a warning in the warnings array:
+- If you're unable to make a change you're confident about but you believe one needs to be made
+- If there are inconsistencies in the data that could be valid but need to be reviewed
+- If not enough information is provided to make a change that you believe is needed
+- If you infer a value for a required field based on context
+
+
+----------PRODUCT FIELD GUIDELINES----------
+
+Fields: supplier, private_notes, company, line, subline, artist
+Changes: Not allowed
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, return these fields exactly as provided with no changes
+
+Fields: upc, supplier_no, notions_no, item_number
+Changes: Formatting only
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, trim outside white space and return these fields exactly as provided with no other changes
+
+Fields: hts_code
+Changes: Minimal, you can correct formatting, obvious errors or inconsistencies
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, trim white space and any non-numeric characters, then return as a string. Do not validate in any other way.
+
+Fields: image_url
+Changes: Formatting only
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, convert all comma-separated values to valid https:// URLs and return
+
+Fields: msrp, cost_each
+Changes: Minimal, you can correct formatting, obvious errors or inconsistencies
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, strip any currency symbols and return as a string with exactly two decimal places, even if the last place is a 0.
+
+Fields: qty_per_unit, case_qty
+Changes: Minimal, you can correct formatting, obvious errors or inconsistencies
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, strip non-numeric characters and return
+
+Fields: ship_restrictions
+Changes: Only add a value if it's not already present
+Required: You must always return a value for this field, even if it's not provided in the original data. If no value is provided, return 0.
+Instructions: Always return a value exactly as provided, or return 0 if no value is provided.
+
+Fields: eta
+Changes: Minimal, you can correct formatting, obvious errors or inconsistencies
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, return a full month name, day is optional, no year ever (e.g. “January” or “March 3”). This value is not required if not provided.
+
+Fields: name
+Changes: Allowed to conform to guidelines, to fix typos or formatting
+Required: You must always return a value for this field, even if it's not provided in the original data. If no value is provided, return the most reasonable value possible based on the naming guidelines and the other information you have.
+Instructions: Always return a value that is corrected and enhanced per additional guidelines below
+
+Fields: description
+Changes: Full creative control allowed within guidelines
+Required: You must always return a value for this field, even if it's not provided in the original data. If no value is provided, return the most accurate description possible based on the description guidelines and the other information you have.
+Instructions: Always return a value that is corrected and enhanced per additional guidelines below
+
+Fields: weight, length, width, height
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: You must always return a value for this field, even if it's not provided in the original data. If no value is provided, return your best guess based on the other information you have or the dimensions for similar products.
+Instructions: Always return a reasonable value (weights in ounces and dimensions in inches) that is validated against similar provided products and your knowledge of general object measurements (e.g. a sheet of paper is not going to be 3 inches thick, a pack of stickers is not going to be 250 ounces, this sheet of paper is very likely going to be the same size as that other sheet of paper from the same line). If a value is unusual or unreasonable, even wildly so, change it to match similar products or to be more reasonable. When correcting unreasonable weights or dimensions, prioritize comparisons to products from the same company and product line first, then broader category matches or common knowledge if necessary.Do not return 0 or null for any of these fields.
+
+Fields: coo
+Changes: Formatting only
+Required: Return if present in the original data. Do not return if not present.
+Instructions: If present, convert all country names and abbreviations to the official ISO 3166-1 alpha-2 two-character country code. Convert any value with more than two characters to two characters only (e.g. "United States" or "USA" should both return "US").
+
+Fields: tax_cat
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: You must always return a value for this field, even if it's not provided in the original data. If no value is provided, return 0.
+Instructions: Always return a valid numerical tax code ID from the Available Tax Codes array below. Give preference to the value provided, but correct it if another value is more accurate. You must return a value for this field. 0 should be the default value in most cases.
+
+Fields: size_cat
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: Return if present in the original data or if not present and applicable. Do not return if not applicable (e.g. if no size categories apply based on what you know about the product).
+Instructions: If present or if applicable, return one valid numerical size category ID from the Available Size Categories array below. Give preference to the value provided, but correct it if another value is more accurate. A value is not required if none of the size categories apply, but it's important to include if one clearly applies, such as if the name contains 12x12, 6x8, 2oz, etc.
+
+Fields: themes
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: Return if present in the original data or if not present and applicable. Do not return any value if not applicable (e.g. if no themes apply based on what you know about the product).
+Instructions: If present, confirm that each provided theme matches what you understand to be a theme of the product. Remove any themes that do not match and add any themes that are missing. Most products will have zero or one theme. Return a comma-separated list of numerical theme IDs from the Available Themes array below. If you choose a sub-theme, you do not need to include its parent theme in the list.
+
+Fields: colors
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: Return if present in the original data or if not present and applicable. Do not return any value if not applicable (e.g. if no colors apply based on what you know about the product).
+Instructions: If present or if applicable, return a comma-separated list of numerical color IDs from the Available Colors array below, using the product name as the primary guide (e.g. if the name contains Blue or a blue variant, you should return the blue color ID). A value is not required if none of the colors apply. Most products will have zero colors.
+
+Fields: categories
+Changes: Allowed to correct obvious errors or inconsistencies or to add missing values
+Required: You must always return at least one value for this field, even if it's not provided in the original data. If no value is provided, return the most appropriate category or categories based on the other information you have.
+Instructions: Always return a comma-separated list of one or more valid numerical category IDs from the Available Categories array below. Give preference to the values provided, particularly if the other information isn't enough to determine a category, but correct them or add new categories if another value is more accurate. Do not return categories in the Deals or Black Friday categories, and strip these from the list if present. If you choose a subcategory at any level, you do not need to include its parent categories in the list. You must return at least one category and you can return multiple categories if applicable. All categories have equal value so their order is not important. Always try to return the most specific categories possible (e.g. one in the third level of the category hierarchy is better than one in the second level).
+
+----------PRODUCT NAMING GUIDELINES----------
+If there's only one of this type of product in a line: [Line Name] [Product Name] - [Company]
+Example: "Cosmos Infinity Chipboard - Stamperia"
+Example: "Serene Petals 6x6 Paper Pad - Prima"
+
+Multiple similar products in a line: [Differentiator] [Product Type] - [Line Name] - [Company]
+Example: "Ice & Shells Stencil - Arctic Antarctic - Stamperia"
+Example: "Astronomy Paper - Cosmos Infinity - Stamperia"
+
+Standalone products: [Product Name] - [Company]
+Example: "Hedwig Puffy Stickers - Paper House Productions"
+Example: "Heart Tree Dies - Lawn Fawn"
+
+Color-based products: [Color] [Product Name] - [Company]
+Example: "Green Valley Enamel Dots - Altenew"
+Example: "Magenta Aqua Pigment - Brutus Monroe"
+
+Complex products: [Differentiator] [Line] [Product Type] - [Company]
+Example: "Size 6 Round Black Velvet Watercolor Brush - Silver Brush Limited" (Size 6 Round is the differentiator, Black Velvet is the line, Watercolor Brush is the product type)
+
+These should not be included in the name, unless there are multiple products that are otherwise identical:
+- Product size
+- Product weight
+- Number of pages
+- How many are in the package
+
+Naming Conventions:
+- Paper sizes: Use "12x12", "8x8", "6x6" (no spaces or units of measure)
+- Company names must match backend exactly
+- Always capitalize every word in the name, including short articles like "The" and "An"
+- Use "Idea-ology" (not "idea-ology" or "Ideaology")
+- All stamps are "Stamp Set" (not "Clear Stamps" or "Rubber Stamps")
+- All dies are "Dies" or "Die" (not "Die Set")
+- Brands with their own naming conventions should be respected, such as "Doodle Cuts" for dies from Doodlebug
+
+Special Brand Rules - Ranger: 
+Format: [Product Name] - [Designer Line] - Ranger
+Possible Designers: Dylusions, Dina Wakley MEdia, Simon Hurley create., Wendy Vecchi
+Example: "Stacked Stencil - Dina Wakley MEdia - Ranger"
+
+Special Brand Rules - Tim Holtz products from Ranger:
+Format: [Color] [Product Name/Type] - Tim Holtz Distress - Ranger
+Example: "Mermaid Lagoon Tim Holtz Distress Oxide Ink Pad - Ranger"
+
+Special Brand Rules - Tim Holtz products from Sizzix or Stampers Anonymous:
+Format: [Product Name] [Product Type] by Tim Holtz - [Company]
+Example: "Leaf Fragments Thinlits Dies by Tim Holtz - Sizzix"
+
+Special Brand Rules - Tim Holtz products from Advantus/Idea-ology:
+Format: [Product Name] - Tim Holtz Idea-ology
+Example: "Tiny Vials - Tim Holtz Idea-ology"
+
+Special Brand Rules - Dies from Sizzix:
+Include die type plus "Dies" or "Die"
+Examples:
+"Art Nouveau 3-D Textured Impressions Embossing Folder - Sizzix"
+"Pocket Pals Thinlits Dies - Sizzix"
+"Butterfly Wishes Framelits Dies & Stamps - Sizzix"
+
+Important Notes
+- Ensure that product names are consistent across all products of the same type
+- Use the minimum amount of information needed to uniquely identify the product
+- Put detailed specifications in the product description, not its name
+
+Edge Cases
+- If the product is missing a company name, infer one from the other products included in the data
+- If the product is missing a clear differentiator and needs one to be unique, infer and add one from the other data provided (e.g. the description, existing size categories, etc.)
+
+Incorrect example: MVP Rugby - Collection Pack - Photoplay
+Notes: there should be no dash between the line and the product
+
+Incorrect Example: A2 Easel Cards - Black - Photoplay
+Notes: the differentiating factor should come first: “Black A2 Easel Cards - Photoplay”. Size is ok to include here because this is the name printed on the package.
+
+Incorrect Example: 6” - Scriber Needle Modeling Tool
+Notes: this product only comes in one size, so 6” isn’t needed. The company name should also be included.
+
+Incorrect Example: Slick - White - Tulip Dimensional Fabric Paint 4oz
+Notes: color should be first, then type, then product, then company, so “White Slick Dimensional Fabric Paint - Tulip”. It appears there’s only one size available so no need to differentiate in the name.
+
+Incorrect Example: Silhouette Adhesive Cork Sheets 5”X7” 8/Pkg
+Notes: should be “Adhesive Cork Sheets - Silhouette”
+
+Incorrect Example: Galaxy - Opaque - American Crafts Color Pour Resin Dyes
+Notes: “Galaxy Opaque Dye Set - Color Pour Resin - American Crafts”
+
+Incorrect Example: Slate - Lion Brand Truboo Yarn
+Notes: [Differentiator] [Line] [Product Type] - [Company] : “Slate Truboo Yarn - Lion Brand”
+
+Incorrect Example: Rose Quartz Dylusions Shimmer Paint
+Notes: “Rose Quartz Shimmer Paint - Dylusions - Ranger”
+
+
+----------PRODUCT DESCRIPTION GUIDELINES----------
+Product descriptions are an extremely important part of the listing and are the most important part of your response. Care should be taken to ensure they are correct, helpful, and SEO-friendly.
+
+If a description is provided in the data, use it as a starting point. Correct any spelling errors, typos, poor grammar, or awkward phrasing. If necessary and you have the information, add more details, describe how the customer could use it, etc. Use complete sentences and keep SEO in mind.
+
+If no description is provided, make one up using the product name, the information you have, and the other provided guidelines. At minimum, a description should be one complete sentence that starts with a capital letter and ends with a period. Unless the product is extremely complex, 2-4 sentences is usually sufficient if you have enough information.
+
+Important Notes:
+- Every description should state exactly what's included in the product (e.g. "Includes one 12x12 sheet of patterned cardstock." or "Includes one 6x12 sheet with 27 unique stickers." or "Includes 55 pieces." or "Package includes machine, power cord, 12 sheets of cardstock, 3 dies, and project instructions.")
+- Do not use the word "our" in the description (this usually shows up when we copy a description from the manufacturer). Instead use "these" or "[Company name] [product]" or similar. (e.g. don't use "Our journals are hand-made in the USA", instead use "These journals are hand made..." or "Archer & Olive journals are handmade...")
+- Don't include statements that add no value like “this is perfect for all your paper crafts”. If the product helps to solve a unique problem or has a unique feature, by all means describe it, but if it’s just a normal sheet of paper or pack of stickers, you don’t have to pretend like it’s the best thing ever. At the same time, ensure that you add enough copy to ensure good SEO.
+- State as many facts as you can about the product, considering the viewpoint of the customer and what they would want to know when looking at it. They probably want to know dimensions, what products it’s compatible with, how thick the paper is, how many sheets are included, whether the sheets are double-sided or not, which items are in the kit, etc. Say as much as you possibly can with the information that you have.
+- !!DO NOT make up information if you aren't sure about it. A minimal correct description is better than a long incorrect one!!
+
+Avoid/remove:
+- The word "Imported"
+- Any warnings about Prop 65, choking hazards, etc
+- The manufacturer's name if it's included as the very first thing in the description
+- Any statement similar to "comes in a variety of colors, each sold separately"

inventory-server/src/routes/ai-validation.js

@@ -0,0 +1,526 @@
+const express = require('express');
+const router = express.Router();
+const OpenAI = require('openai');
+const fs = require('fs').promises;
+const path = require('path');
+const dotenv = require('dotenv');
+
+// Ensure environment variables are loaded
+dotenv.config({ path: path.join(__dirname, '../../.env') });
+
+const openai = new OpenAI({
+  apiKey: process.env.OPENAI_API_KEY
+});
+
+if (!process.env.OPENAI_API_KEY) {
+  console.error('Warning: OPENAI_API_KEY is not set in environment variables');
+}
+
+// Cache configuration
+const CACHE_TTL = 60 * 60 * 1000; // 1 hour in milliseconds
+
+// Cache structure with TTL
+let cache = {
+  taxonomyData: null,
+  validationPrompt: null,
+  lastUpdated: null
+};
+
+// Function to check if cache is valid
+function isCacheValid() {
+  return cache.lastUpdated && (Date.now() - cache.lastUpdated) < CACHE_TTL;
+}
+
+// Function to clear cache
+function clearCache() {
+  cache = {
+    taxonomyData: null,
+    validationPrompt: null,
+    lastUpdated: null
+  };
+}
+
+// Debug endpoint for viewing prompt and cache status
+router.post('/debug', async (req, res) => {
+  try {
+    console.log('Debug POST endpoint called');
+    const pool = req.app.locals.pool;
+    const { products } = req.body;
+    
+    console.log('Received products:', {
+      isArray: Array.isArray(products),
+      length: products?.length,
+      firstProduct: products?.[0],
+      lastProduct: products?.[products?.length - 1]
+    });
+    
+    if (!Array.isArray(products)) {
+      console.error('Invalid input: products is not an array');
+      return res.status(400).json({ error: 'Products must be an array' });
+    }
+
+    if (products.length === 0) {
+      console.error('Invalid input: products array is empty');
+      return res.status(400).json({ error: 'Products array cannot be empty' });
+    }
+    
+    // Clean the products array to remove any internal fields
+    const cleanedProducts = products.map(product => {
+      const { __errors, __index, ...cleanProduct } = product;
+      return cleanProduct;
+    });
+    
+    return await generateDebugResponse(pool, cleanedProducts, res);
+  } catch (error) {
+    console.error('Debug POST endpoint error:', error);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Helper function to generate debug response
+async function generateDebugResponse(pool, productsToUse, res) {
+  // Load taxonomy data first
+  console.log('Loading taxonomy data...');
+  const taxonomy = await getTaxonomyData(pool);
+  console.log('Taxonomy data loaded:', {
+    categoriesCount: taxonomy.categories.length,
+    themesCount: taxonomy.themes.length,
+    colorsCount: taxonomy.colors.length,
+    taxCodesCount: taxonomy.taxCodes.length,
+    sizeCategoriesCount: taxonomy.sizeCategories.length,
+    suppliersCount: taxonomy.suppliers.length,
+    companiesCount: taxonomy.companies.length,
+    artistsCount: taxonomy.artists.length
+  });
+
+  // Load the prompt using the same function used by validation
+  console.log('Loading prompt...');
+  const prompt = await loadPrompt(pool, productsToUse);
+  const fullPrompt = prompt + '\n' + JSON.stringify(productsToUse);
+
+  const response = {
+    cacheStatus: {
+      isCacheValid: isCacheValid(),
+      lastUpdated: cache.lastUpdated ? new Date(cache.lastUpdated).toISOString() : null,
+      timeUntilExpiry: cache.lastUpdated ? 
+        Math.max(0, CACHE_TTL - (Date.now() - cache.lastUpdated)) / 1000 + ' seconds' : 
+        'expired',
+    },
+    taxonomyStats: taxonomy ? {
+      categories: countItems(taxonomy.categories),
+      themes: taxonomy.themes.length,
+      colors: taxonomy.colors.length,
+      taxCodes: taxonomy.taxCodes.length,
+      sizeCategories: taxonomy.sizeCategories.length,
+      suppliers: taxonomy.suppliers.length,
+      companies: taxonomy.companies.length,
+      artists: taxonomy.artists.length,
+      // Add filtered counts when products are provided
+      filtered: productsToUse ? {
+        suppliers: taxonomy.suppliers.filter(([id]) => 
+          productsToUse.some(p => Number(p.supplierid) === Number(id))).length,
+        companies: taxonomy.companies.filter(([id]) => 
+          productsToUse.some(p => Number(p.company) === Number(id))).length,
+        artists: taxonomy.artists.filter(([id]) => 
+          productsToUse.some(p => Number(p.artist) === Number(id))).length
+      } : null
+    } : null,
+    basePrompt: prompt,
+    sampleFullPrompt: fullPrompt,
+    promptLength: fullPrompt.length,
+  };
+
+  console.log('Sending response with stats:', response.taxonomyStats);
+  return res.json(response);
+}
+
+// Helper function to count total items in hierarchical structure
+function countItems(items) {
+  return items.reduce((count, item) => {
+    return count + 1 + (item.subcategories ? countItems(item.subcategories) : 0);
+  }, 0);
+}
+
+// Force cache refresh endpoint
+router.post('/refresh-cache', async (req, res) => {
+  try {
+    clearCache();
+    const pool = req.app.locals.pool;
+    await loadPrompt(pool); // This will rebuild the cache
+    res.json({ 
+      success: true, 
+      message: 'Cache refreshed successfully',
+      newCacheTime: new Date(cache.lastUpdated).toISOString()
+    });
+  } catch (error) {
+    console.error('Cache refresh error:', error);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+// Function to fetch and format taxonomy data
+async function getTaxonomyData(pool) {
+  if (cache.taxonomyData && isCacheValid()) {
+    return cache.taxonomyData;
+  }
+
+  // Fetch categories with hierarchy
+  const [categories] = await pool.query(`
+    SELECT cat_id, name, master_cat_id, level_order 
+    FROM (
+      SELECT cat_id,name,NULL AS master_cat_id,1 AS level_order 
+      FROM product_categories s 
+      WHERE type=10 
+      UNION ALL 
+      SELECT c.cat_id,c.name,c.master_cat_id,2 AS level_order 
+      FROM product_categories c 
+      JOIN product_categories s ON c.master_cat_id=s.cat_id 
+      WHERE c.type=11 AND s.type=10 
+      UNION ALL 
+      SELECT sc.cat_id,sc.name,sc.master_cat_id,3 AS level_order 
+      FROM product_categories sc 
+      JOIN product_categories c ON sc.master_cat_id=c.cat_id 
+      JOIN product_categories s ON c.master_cat_id=s.cat_id 
+      WHERE sc.type=12 AND c.type=11 AND s.type=10 
+      UNION ALL 
+      SELECT ssc.cat_id,ssc.name,ssc.master_cat_id,4 AS level_order 
+      FROM product_categories ssc 
+      JOIN product_categories sc ON ssc.master_cat_id=sc.cat_id 
+      JOIN product_categories c ON sc.master_cat_id=c.cat_id 
+      JOIN product_categories s ON c.master_cat_id=s.cat_id 
+      WHERE ssc.type=13 AND sc.type=12 AND c.type=11 AND s.type=10
+    ) AS hierarchy 
+    ORDER BY level_order,cat_id
+  `);
+
+  // Fetch themes with hierarchy
+  const [themes] = await pool.query(`
+    SELECT cat_id, name, master_cat_id, level_order 
+    FROM (
+      SELECT t.cat_id,t.name,null as master_cat_id,1 AS level_order 
+      FROM product_categories t 
+      WHERE t.type=20 
+      UNION ALL 
+      SELECT ts.cat_id,ts.name,ts.master_cat_id,2 AS level_order 
+      FROM product_categories ts 
+      JOIN product_categories t ON ts.master_cat_id=t.cat_id 
+      WHERE ts.type=21 AND t.type=20
+    ) AS hierarchy 
+    ORDER BY level_order,name
+  `);
+
+  // Fetch colors
+  const [colors] = await pool.query('SELECT color, name FROM product_color_list ORDER BY name');
+
+  // Fetch tax codes
+  const [taxCodes] = await pool.query('SELECT tax_code_id, name FROM product_tax_codes ORDER BY name');
+
+  // Fetch size categories
+  const [sizeCategories] = await pool.query('SELECT cat_id, name FROM product_categories WHERE type=50 ORDER BY name');
+
+  // Fetch suppliers
+  const [suppliers] = await pool.query(`
+    SELECT supplierid, companyname as name 
+    FROM suppliers 
+    WHERE companyname <> '' 
+    ORDER BY companyname
+  `);
+
+  // Fetch companies (type 1)
+  const [companies] = await pool.query(`
+    SELECT cat_id, name 
+    FROM product_categories 
+    WHERE type = 1 
+    ORDER BY name
+  `);
+
+  // Fetch artists (type 40)
+  const [artists] = await pool.query(`
+    SELECT cat_id, name 
+    FROM product_categories 
+    WHERE type = 40 
+    ORDER BY name
+  `);
+
+  // Fetch lines (type 2)
+  const [lines] = await pool.query(`
+    SELECT cat_id, name 
+    FROM product_categories 
+    WHERE type = 2 
+    ORDER BY name
+  `);
+
+  // Fetch sub-lines (type 3)
+  const [subLines] = await pool.query(`
+    SELECT cat_id, name 
+    FROM product_categories 
+    WHERE type = 3 
+    ORDER BY name
+  `);
+
+  // Format categories into a hierarchical structure
+  const formatHierarchy = (items, level = 1, parentId = null) => {
+    return items
+      .filter(item => item.level_order === level && item.master_cat_id === parentId)
+      .map(item => {
+        const children = formatHierarchy(items, level + 1, item.cat_id);
+        return children.length > 0 ? 
+          [item.cat_id, item.name, children] :
+          [item.cat_id, item.name];
+      });
+  };
+
+  // Format themes similarly but with only two levels
+  const formatThemes = (items) => {
+    return items
+      .filter(item => item.level_order === 1)
+      .map(item => {
+        const subthemes = items
+          .filter(subitem => subitem.master_cat_id === item.cat_id)
+          .map(subitem => [subitem.cat_id, subitem.name]);
+        return subthemes.length > 0 ? 
+          [item.cat_id, item.name, subthemes] :
+          [item.cat_id, item.name];
+      });
+  };
+
+  cache.taxonomyData = {
+    categories: formatHierarchy(categories),
+    themes: formatThemes(themes),
+    colors: colors.map(c => [c.color, c.name]),
+    taxCodes: (taxCodes || []).map(tc => [tc.tax_code_id, tc.name]),
+    sizeCategories: (sizeCategories || []).map(sc => [sc.cat_id, sc.name]),
+    suppliers: suppliers.map(s => [s.supplierid, s.name]),
+    companies: companies.map(c => [c.cat_id, c.name]),
+    artists: artists.map(a => [a.cat_id, a.name]),
+    lines: lines.map(l => [l.cat_id, l.name]),
+    subLines: subLines.map(sl => [sl.cat_id, sl.name])
+  };
+  cache.lastUpdated = Date.now();
+
+  return cache.taxonomyData;
+}
+
+// Load the prompt from file and inject taxonomy data
+async function loadPrompt(pool, productsToValidate = null) {
+  const promptPath = path.join(__dirname, '..', 'prompts', 'product-validation.txt');
+  const basePrompt = await fs.readFile(promptPath, 'utf8');
+  
+  // Get taxonomy data
+  const taxonomy = await getTaxonomyData(pool);
+
+  // Add system instructions to the prompt
+  const systemInstructions = `You are a specialized e-commerce product data processor for a crafting supplies website tasked with providing complete, correct, appealing, and SEO-friendly product listings. You should write professionally, but in a friendly and engaging tone.
+
+`;
+
+  // If we have products to validate, create a filtered prompt
+  if (productsToValidate) {
+    console.log('Creating filtered prompt for products:', productsToValidate);
+    
+    // Extract unique values from products for non-core attributes
+    const uniqueValues = {
+      supplierIds: new Set(),
+      companyIds: new Set(),
+      artistIds: new Set(),
+      lineIds: new Set(),
+      subLineIds: new Set()
+    };
+
+    // Collect any values that exist in the products
+    productsToValidate.forEach(product => {
+      Object.entries(product).forEach(([key, value]) => {
+        if (value === undefined || value === null) return;
+        
+        // Map field names to their respective sets
+        const fieldMap = {
+          supplierid: 'supplierIds',
+          supplier: 'supplierIds',
+          company: 'companyIds',
+          artist: 'artistIds',
+          line: 'lineIds',
+          subline: 'subLineIds'
+        };
+        
+        if (fieldMap[key]) {
+          uniqueValues[fieldMap[key]].add(Number(value));
+        }
+      });
+    });
+
+    console.log('Unique values collected:', {
+      suppliers: Array.from(uniqueValues.supplierIds),
+      companies: Array.from(uniqueValues.companyIds),
+      artists: Array.from(uniqueValues.artistIds),
+      lines: Array.from(uniqueValues.lineIds),
+      subLines: Array.from(uniqueValues.subLineIds)
+    });
+
+    // Create mixed taxonomy with filtered non-core data and full core data
+    const mixedTaxonomy = {
+      // Keep full data for core attributes
+      categories: taxonomy.categories,
+      themes: taxonomy.themes,
+      colors: taxonomy.colors,
+      taxCodes: taxonomy.taxCodes,
+      sizeCategories: taxonomy.sizeCategories,
+      // For non-core data, only include items that are actually used
+      suppliers: taxonomy.suppliers.filter(([id]) => uniqueValues.supplierIds.has(Number(id))),
+      companies: taxonomy.companies.filter(([id]) => uniqueValues.companyIds.has(Number(id))),
+      artists: taxonomy.artists.filter(([id]) => uniqueValues.artistIds.has(Number(id))),
+      lines: taxonomy.lines.filter(([id]) => uniqueValues.lineIds.has(Number(id))),
+      subLines: taxonomy.subLines.filter(([id]) => uniqueValues.subLineIds.has(Number(id)))
+    };
+
+    console.log('Filtered taxonomy counts:', {
+      suppliers: mixedTaxonomy.suppliers.length,
+      companies: mixedTaxonomy.companies.length,
+      artists: mixedTaxonomy.artists.length,
+      lines: mixedTaxonomy.lines.length,
+      subLines: mixedTaxonomy.subLines.length
+    });
+
+    // Format taxonomy data for the prompt, only including sections with values
+    const taxonomySection = `
+All Available Categories:
+${JSON.stringify(mixedTaxonomy.categories)}
+
+All Available Themes:
+${JSON.stringify(mixedTaxonomy.themes)}
+
+All Available Colors:
+${JSON.stringify(mixedTaxonomy.colors)}
+
+All Available Tax Codes:
+${JSON.stringify(mixedTaxonomy.taxCodes)}
+
+All Available Size Categories:
+${JSON.stringify(mixedTaxonomy.sizeCategories)}${mixedTaxonomy.suppliers.length ? `\n\nSuppliers Used In This Data:\n${JSON.stringify(mixedTaxonomy.suppliers)}` : ''}${mixedTaxonomy.companies.length ? `\n\nCompanies Used In This Data:\n${JSON.stringify(mixedTaxonomy.companies)}` : ''}${mixedTaxonomy.artists.length ? `\n\nArtists Used In This Data:\n${JSON.stringify(mixedTaxonomy.artists)}` : ''}${mixedTaxonomy.lines.length ? `\n\nLines Used In This Data:\n${JSON.stringify(mixedTaxonomy.lines)}` : ''}${mixedTaxonomy.subLines.length ? `\n\nSub-Lines Used In This Data:\n${JSON.stringify(mixedTaxonomy.subLines)}` : ''}
+
+----------Here is the product data to validate----------`;
+
+    // Return the filtered prompt without caching
+    return systemInstructions + basePrompt + '\n' + taxonomySection;
+  }
+
+  // For debug/display purposes, if no products provided and cache is valid, return cached prompt
+  if (!productsToValidate && cache.validationPrompt && isCacheValid()) {
+    return cache.validationPrompt;
+  }
+
+  // Generate and cache the full unfiltered prompt
+  const taxonomySection = `
+Available Categories:
+${JSON.stringify(taxonomy.categories)}
+
+Available Themes:
+${JSON.stringify(taxonomy.themes)}
+
+Available Colors:
+${JSON.stringify(taxonomy.colors)}
+
+Available Tax Codes:
+${JSON.stringify(taxonomy.taxCodes)}
+
+Available Size Categories:
+${JSON.stringify(taxonomy.sizeCategories)}
+
+Available Suppliers:
+${JSON.stringify(taxonomy.suppliers)}
+
+Available Companies:
+${JSON.stringify(taxonomy.companies)}
+
+Available Artists:
+${JSON.stringify(taxonomy.artists)}
+
+Available Shipping Restrictions:
+${JSON.stringify(taxonomy.shippingRestrictions)}
+
+Here is the product data to validate:`;
+
+  // Cache the full prompt only when no specific products are provided
+  cache.validationPrompt = systemInstructions + basePrompt + '\n' + taxonomySection;
+  cache.lastUpdated = Date.now();
+
+  return cache.validationPrompt;
+}
+
+// Set up cache clearing interval
+setInterval(clearCache, CACHE_TTL);
+
+router.post('/validate', async (req, res) => {
+  try {
+    const { products } = req.body;
+    console.log('🔍 Received products for validation:', JSON.stringify(products, null, 2));
+
+    if (!Array.isArray(products)) {
+      console.error('❌ Invalid input: products is not an array');
+      return res.status(400).json({ error: 'Products must be an array' });
+    }
+
+    // Load the prompt with the products data to filter taxonomy
+    const prompt = await loadPrompt(req.app.locals.pool, products);
+    const fullPrompt = prompt + '\n' + JSON.stringify(products);
+    console.log('📝 Generated prompt:', fullPrompt);
+
+    console.log('🤖 Sending request to OpenAI...');
+    const completion = await openai.chat.completions.create({
+      model: "gpt-4o",
+      messages: [
+        {
+          role: "user",
+          content: fullPrompt
+        }
+      ],
+      temperature: 0.2,
+      response_format: { type: "json_object" }
+    });
+
+    console.log('✅ Received response from OpenAI');
+    const rawResponse = completion.choices[0].message.content;
+    console.log('📄 Raw AI response:', rawResponse);
+
+    const aiResponse = JSON.parse(rawResponse);
+    console.log('🔄 Parsed AI response:', JSON.stringify(aiResponse, null, 2));
+
+    // Compare original and corrected data
+    if (aiResponse.correctedData) {
+      console.log('📊 Changes summary:');
+      products.forEach((original, index) => {
+        const corrected = aiResponse.correctedData[index];
+        if (corrected) {
+          const changes = Object.keys(corrected).filter(key => 
+            JSON.stringify(original[key]) !== JSON.stringify(corrected[key])
+          );
+          if (changes.length > 0) {
+            console.log(`\nProduct ${index + 1} changes:`);
+            changes.forEach(key => {
+              console.log(`  ${key}:`);
+              console.log(`    - Original: ${JSON.stringify(original[key])}`);
+              console.log(`    - Corrected: ${JSON.stringify(corrected[key])}`);
+            });
+          }
+        }
+      });
+    }
+
+    res.json({
+      success: true,
+      ...aiResponse
+    });
+  } catch (error) {
+    console.error('❌ AI Validation Error:', error);
+    console.error('Error details:', {
+      name: error.name,
+      message: error.message,
+      stack: error.stack
+    });
+    res.status(500).json({
+      success: false,
+      error: error.message || 'Error during AI validation'
+    });
+  }
+});
+
+module.exports = router; 

inventory-server/src/routes/analytics.js

@@ -6,24 +6,24 @@ router.get('/stats', async (req, res) => {
   try {
     const pool = req.app.locals.pool;
     
-    const [results] = await pool.query(`
+    const { rows: [results] } = await pool.query(`
       SELECT 
         COALESCE(
           ROUND(
             (SUM(o.price * o.quantity - p.cost_price * o.quantity) / 
-             NULLIF(SUM(o.price * o.quantity), 0)) * 100, 1
+             NULLIF(SUM(o.price * o.quantity), 0) * 100)::numeric, 1
           ),
           0
         ) as profitMargin,
         COALESCE(
           ROUND(
-            (AVG(p.price / NULLIF(p.cost_price, 0) - 1) * 100), 1
+            (AVG(p.price / NULLIF(p.cost_price, 0) - 1) * 100)::numeric, 1
           ),
           0
         ) as averageMarkup,
         COALESCE(
           ROUND(
-            SUM(o.quantity) / NULLIF(AVG(p.stock_quantity), 0), 2
+            (SUM(o.quantity) / NULLIF(AVG(p.stock_quantity), 0))::numeric, 2
           ),
           0
         ) as stockTurnoverRate,
@@ -31,23 +31,23 @@ router.get('/stats', async (req, res) => {
         COALESCE(COUNT(DISTINCT p.categories), 0) as categoryCount,
         COALESCE(
           ROUND(
-            AVG(o.price * o.quantity), 2
+            AVG(o.price * o.quantity)::numeric, 2
           ),
           0
         ) as averageOrderValue
       FROM products p
       LEFT JOIN orders o ON p.pid = o.pid
-      WHERE o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
     `);
 
     // Ensure all values are numbers
     const stats = {
-      profitMargin: Number(results[0].profitMargin) || 0,
-      averageMarkup: Number(results[0].averageMarkup) || 0,
-      stockTurnoverRate: Number(results[0].stockTurnoverRate) || 0,
-      vendorCount: Number(results[0].vendorCount) || 0,
-      categoryCount: Number(results[0].categoryCount) || 0,
-      averageOrderValue: Number(results[0].averageOrderValue) || 0
+      profitMargin: Number(results.profitmargin) || 0,
+      averageMarkup: Number(results.averagemarkup) || 0,
+      stockTurnoverRate: Number(results.stockturnoverrate) || 0,
+      vendorCount: Number(results.vendorcount) || 0,
+      categoryCount: Number(results.categorycount) || 0,
+      averageOrderValue: Number(results.averageordervalue) || 0
     };
 
     res.json(stats);
@@ -63,13 +63,13 @@ router.get('/profit', async (req, res) => {
     const pool = req.app.locals.pool;
     
     // Get profit margins by category with full path
-    const [byCategory] = await pool.query(`
+    const { rows: byCategory } = await pool.query(`
       WITH RECURSIVE category_path AS (
         SELECT 
           c.cat_id,
           c.name,
           c.parent_id,
-          CAST(c.name AS CHAR(1000)) as path
+          c.name::text as path
         FROM categories c
         WHERE c.parent_id IS NULL
         
@@ -79,7 +79,7 @@ router.get('/profit', async (req, res) => {
           c.cat_id,
           c.name,
           c.parent_id,
-          CONCAT(cp.path, ' > ', c.name)
+          cp.path || ' > ' || c.name
         FROM categories c
         JOIN category_path cp ON c.parent_id = cp.cat_id
       )
@@ -88,53 +88,46 @@ router.get('/profit', async (req, res) => {
         cp.path as categoryPath,
         ROUND(
           (SUM(o.price * o.quantity - p.cost_price * o.quantity) / 
-           NULLIF(SUM(o.price * o.quantity), 0)) * 100, 1
+           NULLIF(SUM(o.price * o.quantity), 0) * 100)::numeric, 1
         ) as profitMargin,
-        CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) as revenue,
-        CAST(SUM(p.cost_price * o.quantity) AS DECIMAL(15,3)) as cost
+        ROUND(SUM(o.price * o.quantity)::numeric, 3) as revenue,
+        ROUND(SUM(p.cost_price * o.quantity)::numeric, 3) as cost
       FROM products p
       LEFT JOIN orders o ON p.pid = o.pid
       JOIN product_categories pc ON p.pid = pc.pid
       JOIN categories c ON pc.cat_id = c.cat_id
       JOIN category_path cp ON c.cat_id = cp.cat_id
-      WHERE o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
       GROUP BY c.name, cp.path
       ORDER BY profitMargin DESC
       LIMIT 10
     `);
 
     // Get profit margin trend over time
-    const [overTime] = await pool.query(`
+    const { rows: overTime } = await pool.query(`
       SELECT 
-        formatted_date as date,
+        to_char(o.date, 'YYYY-MM-DD') as date,
         ROUND(
           (SUM(o.price * o.quantity - p.cost_price * o.quantity) / 
-           NULLIF(SUM(o.price * o.quantity), 0)) * 100, 1
+           NULLIF(SUM(o.price * o.quantity), 0) * 100)::numeric, 1
         ) as profitMargin,
-        CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) as revenue,
-        CAST(SUM(p.cost_price * o.quantity) AS DECIMAL(15,3)) as cost
+        ROUND(SUM(o.price * o.quantity)::numeric, 3) as revenue,
+        ROUND(SUM(p.cost_price * o.quantity)::numeric, 3) as cost
       FROM products p
       LEFT JOIN orders o ON p.pid = o.pid
-      CROSS JOIN (
-        SELECT DATE_FORMAT(o.date, '%Y-%m-%d') as formatted_date
-        FROM orders o
-        WHERE o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
-        GROUP BY DATE_FORMAT(o.date, '%Y-%m-%d')
-      ) dates
-      WHERE o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
-        AND DATE_FORMAT(o.date, '%Y-%m-%d') = dates.formatted_date
-      GROUP BY formatted_date
-      ORDER BY formatted_date
+      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
+      GROUP BY to_char(o.date, 'YYYY-MM-DD')
+      ORDER BY date
     `);
 
     // Get top performing products with category paths
-    const [topProducts] = await pool.query(`
+    const { rows: topProducts } = await pool.query(`
       WITH RECURSIVE category_path AS (
         SELECT 
           c.cat_id,
           c.name,
           c.parent_id,
-          CAST(c.name AS CHAR(1000)) as path
+          c.name::text as path
         FROM categories c
         WHERE c.parent_id IS NULL
         
@@ -144,7 +137,7 @@ router.get('/profit', async (req, res) => {
           c.cat_id,
           c.name,
           c.parent_id,
-          CONCAT(cp.path, ' > ', c.name)
+          cp.path || ' > ' || c.name
         FROM categories c
         JOIN category_path cp ON c.parent_id = cp.cat_id
       )
@@ -154,18 +147,18 @@ router.get('/profit', async (req, res) => {
         cp.path as categoryPath,
         ROUND(
           (SUM(o.price * o.quantity - p.cost_price * o.quantity) / 
-           NULLIF(SUM(o.price * o.quantity), 0)) * 100, 1
+           NULLIF(SUM(o.price * o.quantity), 0) * 100)::numeric, 1
         ) as profitMargin,
-        CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) as revenue,
-        CAST(SUM(p.cost_price * o.quantity) AS DECIMAL(15,3)) as cost
+        ROUND(SUM(o.price * o.quantity)::numeric, 3) as revenue,
+        ROUND(SUM(p.cost_price * o.quantity)::numeric, 3) as cost
       FROM products p
       LEFT JOIN orders o ON p.pid = o.pid
       JOIN product_categories pc ON p.pid = pc.pid
       JOIN categories c ON pc.cat_id = c.cat_id
       JOIN category_path cp ON c.cat_id = cp.cat_id
-      WHERE o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+      WHERE o.date >= CURRENT_DATE - INTERVAL '30 days'
       GROUP BY p.pid, p.title, c.name, cp.path
-      HAVING revenue > 0
+      HAVING SUM(o.price * o.quantity) > 0
       ORDER BY profitMargin DESC
       LIMIT 10
     `);
@@ -185,7 +178,7 @@ router.get('/vendors', async (req, res) => {
     console.log('Fetching vendor performance data...');
 
     // First check if we have any vendors with sales
-    const [checkData] = await pool.query(`
+    const { rows: [checkData] } = await pool.query(`
       SELECT COUNT(DISTINCT p.vendor) as vendor_count,
              COUNT(DISTINCT o.order_number) as order_count
       FROM products p
@@ -193,39 +186,39 @@ router.get('/vendors', async (req, res) => {
       WHERE p.vendor IS NOT NULL
     `);
     
-    console.log('Vendor data check:', checkData[0]);
+    console.log('Vendor data check:', checkData);
     
     // Get vendor performance metrics
-    const [performance] = await pool.query(`
+    const { rows: performance } = await pool.query(`
       WITH monthly_sales AS (
         SELECT 
           p.vendor,
-          CAST(SUM(CASE 
-            WHEN o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY) 
+          ROUND(SUM(CASE 
+            WHEN o.date >= CURRENT_DATE - INTERVAL '30 days'
             THEN o.price * o.quantity 
             ELSE 0 
-          END) AS DECIMAL(15,3)) as current_month,
-          CAST(SUM(CASE 
-            WHEN o.date >= DATE_SUB(CURDATE(), INTERVAL 60 DAY) 
-            AND o.date < DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+          END)::numeric, 3) as current_month,
+          ROUND(SUM(CASE 
+            WHEN o.date >= CURRENT_DATE - INTERVAL '60 days'
+            AND o.date < CURRENT_DATE - INTERVAL '30 days'
             THEN o.price * o.quantity 
             ELSE 0 
-          END) AS DECIMAL(15,3)) as previous_month
+          END)::numeric, 3) as previous_month
         FROM products p
         LEFT JOIN orders o ON p.pid = o.pid
         WHERE p.vendor IS NOT NULL
-          AND o.date >= DATE_SUB(CURDATE(), INTERVAL 60 DAY)
+          AND o.date >= CURRENT_DATE - INTERVAL '60 days'
         GROUP BY p.vendor
       )
       SELECT 
         p.vendor,
-        CAST(SUM(o.price * o.quantity) AS DECIMAL(15,3)) as salesVolume,
+        ROUND(SUM(o.price * o.quantity)::numeric, 3) as salesVolume,
         COALESCE(ROUND(
           (SUM(o.price * o.quantity - p.cost_price * o.quantity) / 
-           NULLIF(SUM(o.price * o.quantity), 0)) * 100, 1
+           NULLIF(SUM(o.price * o.quantity), 0) * 100)::numeric, 1
         ), 0) as profitMargin,
         COALESCE(ROUND(
-          SUM(o.quantity) / NULLIF(AVG(p.stock_quantity), 0), 1
+          (SUM(o.quantity) / NULLIF(AVG(p.stock_quantity), 0))::numeric, 1
         ), 0) as stockTurnover,
         COUNT(DISTINCT p.pid) as productCount,
         ROUND(
@@ -236,7 +229,7 @@ router.get('/vendors', async (req, res) => {
       LEFT JOIN orders o ON p.pid = o.pid
       LEFT JOIN monthly_sales ms ON p.vendor = ms.vendor
       WHERE p.vendor IS NOT NULL
-        AND o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+        AND o.date >= CURRENT_DATE - INTERVAL '30 days'
       GROUP BY p.vendor, ms.current_month, ms.previous_month
       ORDER BY salesVolume DESC
       LIMIT 10
@@ -244,45 +237,7 @@ router.get('/vendors', async (req, res) => {
 
     console.log('Performance data:', performance);
 
-    // Get vendor comparison data
-    const [comparison] = await pool.query(`
-      SELECT 
-        p.vendor,
-        CAST(COALESCE(ROUND(SUM(o.price * o.quantity) / NULLIF(COUNT(DISTINCT p.pid), 0), 2), 0) AS DECIMAL(15,3)) as salesPerProduct,
-        COALESCE(ROUND(AVG((o.price - p.cost_price) / NULLIF(o.price, 0) * 100), 1), 0) as averageMargin,
-        COUNT(DISTINCT p.pid) as size
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid AND o.date >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
-      WHERE p.vendor IS NOT NULL
-      GROUP BY p.vendor
-      ORDER BY salesPerProduct DESC
-      LIMIT 20
-    `);
-
-    console.log('Comparison data:', comparison);
-
-    // Get vendor sales trends
-    const [trends] = await pool.query(`
-      SELECT 
-        p.vendor,
-        DATE_FORMAT(o.date, '%b %Y') as month,
-        CAST(COALESCE(SUM(o.price * o.quantity), 0) AS DECIMAL(15,3)) as sales
-      FROM products p
-      LEFT JOIN orders o ON p.pid = o.pid
-      WHERE p.vendor IS NOT NULL
-        AND o.date >= DATE_SUB(CURDATE(), INTERVAL 6 MONTH)
-      GROUP BY 
-        p.vendor,
-        DATE_FORMAT(o.date, '%b %Y'),
-        DATE_FORMAT(o.date, '%Y-%m')
-      ORDER BY 
-        p.vendor,
-        DATE_FORMAT(o.date, '%Y-%m')
-    `);
-
-    console.log('Trends data:', trends);
-
-    res.json({ performance, comparison, trends });
+    res.json({ performance });
   } catch (error) {
     console.error('Error fetching vendor performance:', error);
     res.status(500).json({ error: 'Failed to fetch vendor performance' });

inventory-server/src/routes/categories.js

@@ -6,7 +6,7 @@ router.get('/', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     // Get all categories with metrics and hierarchy info
-    const [categories] = await pool.query(`
+    const { rows: categories } = await pool.query(`
       SELECT 
         c.cat_id,
         c.name,
@@ -18,7 +18,7 @@ router.get('/', async (req, res) => {
         p.type as parent_type,
         COALESCE(cm.product_count, 0) as product_count,
         COALESCE(cm.active_products, 0) as active_products,
-        CAST(COALESCE(cm.total_value, 0) AS DECIMAL(15,3)) as total_value,
+        ROUND(COALESCE(cm.total_value, 0)::numeric, 3) as total_value,
         COALESCE(cm.avg_margin, 0) as avg_margin,
         COALESCE(cm.turnover_rate, 0) as turnover_rate,
         COALESCE(cm.growth_rate, 0) as growth_rate
@@ -39,22 +39,22 @@ router.get('/', async (req, res) => {
     `);
 
     // Get overall stats
-    const [stats] = await pool.query(`
+    const { rows: [stats] } = await pool.query(`
       SELECT 
         COUNT(DISTINCT c.cat_id) as totalCategories,
         COUNT(DISTINCT CASE WHEN c.status = 'active' THEN c.cat_id END) as activeCategories,
-        CAST(COALESCE(SUM(cm.total_value), 0) AS DECIMAL(15,3)) as totalValue,
-        COALESCE(ROUND(AVG(NULLIF(cm.avg_margin, 0)), 1), 0) as avgMargin,
-        COALESCE(ROUND(AVG(NULLIF(cm.growth_rate, 0)), 1), 0) as avgGrowth
+        ROUND(COALESCE(SUM(cm.total_value), 0)::numeric, 3) as totalValue,
+        COALESCE(ROUND(AVG(NULLIF(cm.avg_margin, 0))::numeric, 1), 0) as avgMargin,
+        COALESCE(ROUND(AVG(NULLIF(cm.growth_rate, 0))::numeric, 1), 0) as avgGrowth
       FROM categories c
       LEFT JOIN category_metrics cm ON c.cat_id = cm.category_id
     `);
 
     // Get type counts for filtering
-    const [typeCounts] = await pool.query(`
+    const { rows: typeCounts } = await pool.query(`
       SELECT 
         type,
-        COUNT(*) as count
+        COUNT(*)::integer as count
       FROM categories
       GROUP BY type
       ORDER BY type
@@ -81,14 +81,14 @@ router.get('/', async (req, res) => {
       })),
       typeCounts: typeCounts.map(tc => ({
         type: tc.type,
-        count: parseInt(tc.count)
+        count: tc.count // Already cast to integer in the query
       })),
       stats: {
-        totalCategories: parseInt(stats[0].totalCategories),
-        activeCategories: parseInt(stats[0].activeCategories),
-        totalValue: parseFloat(stats[0].totalValue),
-        avgMargin: parseFloat(stats[0].avgMargin),
-        avgGrowth: parseFloat(stats[0].avgGrowth)
+        totalCategories: parseInt(stats.totalcategories),
+        activeCategories: parseInt(stats.activecategories),
+        totalValue: parseFloat(stats.totalvalue),
+        avgMargin: parseFloat(stats.avgmargin),
+        avgGrowth: parseFloat(stats.avggrowth)
       }
     });
   } catch (error) {

inventory-server/src/routes/config.js

@@ -13,22 +13,22 @@ router.get('/', async (req, res) => {
   try {
     console.log('[Config Route] Fetching configuration values...');
     
-    const [stockThresholds] = await pool.query('SELECT * FROM stock_thresholds WHERE id = 1');
+    const { rows: stockThresholds } = await pool.query('SELECT * FROM stock_thresholds WHERE id = 1');
     console.log('[Config Route] Stock thresholds:', stockThresholds);
     
-    const [leadTimeThresholds] = await pool.query('SELECT * FROM lead_time_thresholds WHERE id = 1');
+    const { rows: leadTimeThresholds } = await pool.query('SELECT * FROM lead_time_thresholds WHERE id = 1');
     console.log('[Config Route] Lead time thresholds:', leadTimeThresholds);
     
-    const [salesVelocityConfig] = await pool.query('SELECT * FROM sales_velocity_config WHERE id = 1');
+    const { rows: salesVelocityConfig } = await pool.query('SELECT * FROM sales_velocity_config WHERE id = 1');
     console.log('[Config Route] Sales velocity config:', salesVelocityConfig);
     
-    const [abcConfig] = await pool.query('SELECT * FROM abc_classification_config WHERE id = 1');
+    const { rows: abcConfig } = await pool.query('SELECT * FROM abc_classification_config WHERE id = 1');
     console.log('[Config Route] ABC config:', abcConfig);
     
-    const [safetyStockConfig] = await pool.query('SELECT * FROM safety_stock_config WHERE id = 1');
+    const { rows: safetyStockConfig } = await pool.query('SELECT * FROM safety_stock_config WHERE id = 1');
     console.log('[Config Route] Safety stock config:', safetyStockConfig);
     
-    const [turnoverConfig] = await pool.query('SELECT * FROM turnover_config WHERE id = 1');
+    const { rows: turnoverConfig } = await pool.query('SELECT * FROM turnover_config WHERE id = 1');
     console.log('[Config Route] Turnover config:', turnoverConfig);
 
     const response = {
@@ -53,14 +53,14 @@ router.put('/stock-thresholds/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { critical_days, reorder_days, overstock_days, low_stock_threshold, min_reorder_quantity } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE stock_thresholds 
-       SET critical_days = ?,
-           reorder_days = ?,
-           overstock_days = ?,
-           low_stock_threshold = ?,
-           min_reorder_quantity = ?
-       WHERE id = ?`,
+       SET critical_days = $1,
+           reorder_days = $2,
+           overstock_days = $3,
+           low_stock_threshold = $4,
+           min_reorder_quantity = $5
+       WHERE id = $6`,
       [critical_days, reorder_days, overstock_days, low_stock_threshold, min_reorder_quantity, req.params.id]
     );
     res.json({ success: true });
@@ -75,12 +75,12 @@ router.put('/lead-time-thresholds/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { target_days, warning_days, critical_days } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE lead_time_thresholds 
-       SET target_days = ?,
-           warning_days = ?,
-           critical_days = ?
-       WHERE id = ?`,
+       SET target_days = $1,
+           warning_days = $2,
+           critical_days = $3
+       WHERE id = $4`,
       [target_days, warning_days, critical_days, req.params.id]
     );
     res.json({ success: true });
@@ -95,12 +95,12 @@ router.put('/sales-velocity/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { daily_window_days, weekly_window_days, monthly_window_days } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE sales_velocity_config 
-       SET daily_window_days = ?,
-           weekly_window_days = ?,
-           monthly_window_days = ?
-       WHERE id = ?`,
+       SET daily_window_days = $1,
+           weekly_window_days = $2,
+           monthly_window_days = $3
+       WHERE id = $4`,
       [daily_window_days, weekly_window_days, monthly_window_days, req.params.id]
     );
     res.json({ success: true });
@@ -115,12 +115,12 @@ router.put('/abc-classification/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { a_threshold, b_threshold, classification_period_days } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE abc_classification_config 
-       SET a_threshold = ?,
-           b_threshold = ?,
-           classification_period_days = ?
-       WHERE id = ?`,
+       SET a_threshold = $1,
+           b_threshold = $2,
+           classification_period_days = $3
+       WHERE id = $4`,
       [a_threshold, b_threshold, classification_period_days, req.params.id]
     );
     res.json({ success: true });
@@ -135,11 +135,11 @@ router.put('/safety-stock/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { coverage_days, service_level } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE safety_stock_config 
-       SET coverage_days = ?,
-           service_level = ?
-       WHERE id = ?`,
+       SET coverage_days = $1,
+           service_level = $2
+       WHERE id = $3`,
       [coverage_days, service_level, req.params.id]
     );
     res.json({ success: true });
@@ -154,11 +154,11 @@ router.put('/turnover/:id', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     const { calculation_period_days, target_rate } = req.body;
-    const [result] = await pool.query(
+    const { rows } = await pool.query(
       `UPDATE turnover_config 
-       SET calculation_period_days = ?,
-           target_rate = ?
-       WHERE id = ?`,
+       SET calculation_period_days = $1,
+           target_rate = $2
+       WHERE id = $3`,
       [calculation_period_days, target_rate, req.params.id]
     );
     res.json({ success: true });

inventory-server/src/routes/csv.js

@@ -2,6 +2,7 @@ const express = require('express');
 const router = express.Router();
 const { spawn } = require('child_process');
 const path = require('path');
+const db = require('../utils/db');
 
 // Debug middleware MUST be first
 router.use((req, res, next) => {
@@ -9,9 +10,11 @@ router.use((req, res, next) => {
     next();
 });
 
-// Store active import process and its progress
+// Store active processes and their progress
 let activeImport = null;
 let importProgress = null;
+let activeFullUpdate = null;
+let activeFullReset = null;
 
 // SSE clients for progress updates
 const updateClients = new Set();
@@ -19,17 +22,16 @@ const importClients = new Set();
 const resetClients = new Set();
 const resetMetricsClients = new Set();
 const calculateMetricsClients = new Set();
+const fullUpdateClients = new Set();
+const fullResetClients = new Set();
 
 // Helper to send progress to specific clients
-function sendProgressToClients(clients, progress) {
-    const data = typeof progress === 'string' ? { progress } : progress;
-    
-    // Ensure we have a status field
-    if (!data.status) {
-        data.status = 'running';
-    }
-    
-    const message = `data: ${JSON.stringify(data)}\n\n`;
+function sendProgressToClients(clients, data) {
+    // If data is a string, send it directly
+    // If it's an object, convert it to JSON
+    const message = typeof data === 'string' 
+        ? `data: ${data}\n\n`
+        : `data: ${JSON.stringify(data)}\n\n`;
     
     clients.forEach(client => {
         try {
@@ -45,115 +47,149 @@ function sendProgressToClients(clients, progress) {
     });
 }
 
+// Helper to run a script and stream progress
+function runScript(scriptPath, type, clients) {
+    return new Promise((resolve, reject) => {
+        // Kill any existing process of this type
+        let activeProcess;
+        switch (type) {
+            case 'update':
+                if (activeFullUpdate) {
+                    try { activeFullUpdate.kill(); } catch (e) { }
+                }
+                activeProcess = activeFullUpdate;
+                break;
+            case 'reset':
+                if (activeFullReset) {
+                    try { activeFullReset.kill(); } catch (e) { }
+                }
+                activeProcess = activeFullReset;
+                break;
+        }
+
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe']
+        });
+
+        switch (type) {
+            case 'update':
+                activeFullUpdate = child;
+                break;
+            case 'reset':
+                activeFullReset = child;
+                break;
+        }
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const text = data.toString();
+            output += text;
+            
+            // Split by lines to handle multiple JSON outputs
+            const lines = text.split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    // Try to parse as JSON but don't let it affect the display
+                    const jsonData = JSON.parse(line);
+                    // Only end the process if we get a final status
+                    if (jsonData.status === 'complete' || jsonData.status === 'error' || jsonData.status === 'cancelled') {
+                        if (jsonData.status === 'complete' && !jsonData.operation?.includes('complete')) {
+                            // Don't close for intermediate completion messages
+                            sendProgressToClients(clients, line);
+                            return;
+                        }
+                        // Close only on final completion/error/cancellation
+                        switch (type) {
+                            case 'update':
+                                activeFullUpdate = null;
+                                break;
+                            case 'reset':
+                                activeFullReset = null;
+                                break;
+                        }
+                        if (jsonData.status === 'error') {
+                            reject(new Error(jsonData.error || 'Unknown error'));
+                        } else {
+                            resolve({ output });
+                        }
+                    }
+                } catch (e) {
+                    // Not JSON, just display as is
+                }
+                // Always send the raw line
+                sendProgressToClients(clients, line);
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            const text = data.toString();
+            console.error(text);
+            // Send stderr output directly too
+            sendProgressToClients(clients, text);
+        });
+
+        child.on('close', (code) => {
+            switch (type) {
+                case 'update':
+                    activeFullUpdate = null;
+                    break;
+                case 'reset':
+                    activeFullReset = null;
+                    break;
+            }
+
+            if (code !== 0) {
+                const error = `Script ${scriptPath} exited with code ${code}`;
+                sendProgressToClients(clients, error);
+                reject(new Error(error));
+            }
+            // Don't resolve here - let the completion message from the script trigger the resolve
+        });
+
+        child.on('error', (err) => {
+            switch (type) {
+                case 'update':
+                    activeFullUpdate = null;
+                    break;
+                case 'reset':
+                    activeFullReset = null;
+                    break;
+            }
+            sendProgressToClients(clients, err.message);
+            reject(err);
+        });
+    });
+}
+
 // Progress endpoints
-router.get('/update/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the update set
-    updateClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        updateClients.delete(res);
-    });
-});
-
-router.get('/import/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the import set
-    importClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        importClients.delete(res);
-    });
-});
-
-router.get('/reset/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the reset set
-    resetClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        resetClients.delete(res);
-    });
-});
-
-// Add reset-metrics progress endpoint
-router.get('/reset-metrics/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the reset-metrics set
-    resetMetricsClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        resetMetricsClients.delete(res);
-    });
-});
-
-// Add calculate-metrics progress endpoint
-router.get('/calculate-metrics/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send current progress if it exists
-    if (importProgress) {
-        res.write(`data: ${JSON.stringify(importProgress)}\n\n`);
-    } else {
-        res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
+router.get('/:type/progress', (req, res) => {
+    const { type } = req.params;
+    if (!['update', 'reset'].includes(type)) {
+        return res.status(400).json({ error: 'Invalid operation type' });
     }
 
-    // Add this client to the calculate-metrics set
-    calculateMetricsClients.add(res);
+    res.writeHead(200, {
+        'Content-Type': 'text/event-stream',
+        'Cache-Control': 'no-cache',
+        'Connection': 'keep-alive',
+        'Access-Control-Allow-Origin': req.headers.origin || '*',
+        'Access-Control-Allow-Credentials': 'true'
+    });
 
-    // Remove client when connection closes
+    // Add this client to the correct set
+    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
+    clients.add(res);
+
+    // Send initial connection message
+    sendProgressToClients(new Set([res]), JSON.stringify({
+        status: 'running',
+        operation: 'Initializing connection...'
+    }));
+
+    // Handle client disconnect
     req.on('close', () => {
-        calculateMetricsClients.delete(res);
+        clients.delete(res);
     });
 });
 
@@ -174,7 +210,6 @@ router.get('/status', (req, res) => {
 
 // Add calculate-metrics status endpoint
 router.get('/calculate-metrics/status', (req, res) => {
-    console.log('Calculate metrics status endpoint hit');
     const calculateMetrics = require('../../scripts/calculate-metrics');
     const progress = calculateMetrics.getProgress();
     
@@ -371,49 +406,35 @@ router.post('/import', async (req, res) => {
 
 // Route to cancel active process
 router.post('/cancel', (req, res) => {
-    if (!activeImport) {
-        return res.status(404).json({ error: 'No active process to cancel' });
+    let killed = false;
+
+    // Get the operation type from the request
+    const { type } = req.query;
+    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
+    const activeProcess = type === 'update' ? activeFullUpdate : activeFullReset;
+
+    if (activeProcess) {
+        try {
+            activeProcess.kill('SIGTERM');
+            if (type === 'update') {
+                activeFullUpdate = null;
+            } else {
+                activeFullReset = null;
+            }
+            killed = true;
+            sendProgressToClients(clients, JSON.stringify({
+                status: 'cancelled',
+                operation: 'Operation cancelled'
+            }));
+        } catch (err) {
+            console.error(`Error killing ${type} process:`, err);
+        }
     }
 
-    try {
-        // If it's the prod import module, call its cancel function
-        if (typeof activeImport.cancelImport === 'function') {
-            activeImport.cancelImport();
-        } else {
-            // Otherwise it's a child process
-            activeImport.kill('SIGTERM');
-        }
-        
-        // Get the operation type from the request
-        const { operation } = req.query;
-        
-        // Send cancel message only to the appropriate client set
-        const cancelMessage = { 
-            status: 'cancelled',
-            operation: 'Operation cancelled'
-        };
-
-        switch (operation) {
-            case 'update':
-                sendProgressToClients(updateClients, cancelMessage);
-                break;
-            case 'import':
-                sendProgressToClients(importClients, cancelMessage);
-                break;
-            case 'reset':
-                sendProgressToClients(resetClients, cancelMessage);
-                break;
-            case 'calculate-metrics':
-                sendProgressToClients(calculateMetricsClients, cancelMessage);
-                break;
-        }
-        
+    if (killed) {
         res.json({ success: true });
-    } catch (error) {
-        // Even if there's an error, try to clean up
-        activeImport = null;
-        importProgress = null;
-        res.status(500).json({ error: 'Failed to cancel process' });
+    } else {
+        res.status(404).json({ error: 'No active process to cancel' });
     }
 });
 
@@ -552,20 +573,6 @@ router.post('/reset-metrics', async (req, res) => {
     }
 });
 
-// Add calculate-metrics status endpoint
-router.get('/calculate-metrics/status', (req, res) => {
-    const calculateMetrics = require('../../scripts/calculate-metrics');
-    const progress = calculateMetrics.getProgress();
-    
-    // Only consider it active if both the process is running and we have progress
-    const isActive = !!activeImport && !!progress;
-    
-    res.json({
-        active: isActive,
-        progress: isActive ? progress : null
-    });
-});
-
 // Add calculate-metrics endpoint
 router.post('/calculate-metrics', async (req, res) => {
     if (activeImport) {
@@ -711,4 +718,116 @@ router.post('/import-from-prod', async (req, res) => {
     }
 });
 
+// POST /csv/full-update - Run full update script
+router.post('/full-update', async (req, res) => {
+    try {
+        const scriptPath = path.join(__dirname, '../../scripts/full-update.js');
+        runScript(scriptPath, 'update', fullUpdateClients)
+            .catch(error => {
+                console.error('Update failed:', error);
+            });
+        res.status(202).json({ message: 'Update started' });
+    } catch (error) {
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// POST /csv/full-reset - Run full reset script
+router.post('/full-reset', async (req, res) => {
+    try {
+        const scriptPath = path.join(__dirname, '../../scripts/full-reset.js');
+        runScript(scriptPath, 'reset', fullResetClients)
+            .catch(error => {
+                console.error('Reset failed:', error);
+            });
+        res.status(202).json({ message: 'Reset started' });
+    } catch (error) {
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /history/import - Get recent import history
+router.get('/history/import', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const { rows } = await pool.query(`
+            SELECT 
+                id,
+                start_time,
+                end_time,
+                status,
+                error_message,
+                rows_processed::integer,
+                files_processed::integer
+            FROM import_history 
+            ORDER BY start_time DESC 
+            LIMIT 20
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching import history:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /history/calculate - Get recent calculation history
+router.get('/history/calculate', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const { rows } = await pool.query(`
+            SELECT 
+                id,
+                start_time,
+                end_time,
+                status,
+                error_message,
+                modules_processed::integer,
+                total_modules::integer
+            FROM calculate_history 
+            ORDER BY start_time DESC 
+            LIMIT 20
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching calculate history:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /status/modules - Get module calculation status
+router.get('/status/modules', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const { rows } = await pool.query(`
+            SELECT 
+                module_name, 
+                last_calculation_timestamp::timestamp 
+            FROM calculate_status 
+            ORDER BY module_name
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching module status:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /status/tables - Get table sync status
+router.get('/status/tables', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const { rows } = await pool.query(`
+            SELECT 
+                table_name, 
+                last_sync_timestamp::timestamp 
+            FROM sync_status 
+            ORDER BY table_name
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching table status:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
 module.exports = router; 

inventory-server/src/routes/dashboard.js

@@ -19,16 +19,15 @@ async function executeQuery(sql, params = []) {
 router.get('/stock/metrics', async (req, res) => {
     try {
         // Get stock metrics
-        const [rows] = await executeQuery(`
+        const { rows: [stockMetrics] } = await executeQuery(`
             SELECT 
-                COALESCE(COUNT(*), 0) as total_products,
-                COALESCE(COUNT(CASE WHEN stock_quantity > 0 THEN 1 END), 0) as products_in_stock,
-                COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity END), 0) as total_units,
-                COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity * cost_price END), 0) as total_cost,
-                COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity * price END), 0) as total_retail
+                COALESCE(COUNT(*), 0)::integer as total_products,
+                COALESCE(COUNT(CASE WHEN stock_quantity > 0 THEN 1 END), 0)::integer as products_in_stock,
+                COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity END), 0)::integer as total_units,
+                ROUND(COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity * cost_price END), 0)::numeric, 3) as total_cost,
+                ROUND(COALESCE(SUM(CASE WHEN stock_quantity > 0 THEN stock_quantity * price END), 0)::numeric, 3) as total_retail
             FROM products
         `);
-        const stockMetrics = rows[0];
 
         console.log('Raw stockMetrics from database:', stockMetrics);
         console.log('stockMetrics.total_products:', stockMetrics.total_products);
@@ -38,26 +37,26 @@ router.get('/stock/metrics', async (req, res) => {
         console.log('stockMetrics.total_retail:', stockMetrics.total_retail);
 
         // Get brand stock values with Other category
-        const [brandValues] = await executeQuery(`
+        const { rows: brandValues } = await executeQuery(`
             WITH brand_totals AS (
                 SELECT 
                     COALESCE(brand, 'Unbranded') as brand,
-                    COUNT(DISTINCT pid) as variant_count,
-                    COALESCE(SUM(stock_quantity), 0) as stock_units,
-                    CAST(COALESCE(SUM(stock_quantity * cost_price), 0) AS DECIMAL(15,3)) as stock_cost,
-                    CAST(COALESCE(SUM(stock_quantity * price), 0) AS DECIMAL(15,3)) as stock_retail
+                    COUNT(DISTINCT pid)::integer as variant_count,
+                    COALESCE(SUM(stock_quantity), 0)::integer as stock_units,
+                    ROUND(COALESCE(SUM(stock_quantity * cost_price), 0)::numeric, 3) as stock_cost,
+                    ROUND(COALESCE(SUM(stock_quantity * price), 0)::numeric, 3) as stock_retail
                 FROM products
                 WHERE stock_quantity > 0
                 GROUP BY COALESCE(brand, 'Unbranded')
-                HAVING stock_cost > 0
+                HAVING ROUND(COALESCE(SUM(stock_quantity * cost_price), 0)::numeric, 3) > 0
             ),
             other_brands AS (
                 SELECT 
                     'Other' as brand,
-                    SUM(variant_count) as variant_count,
-                    SUM(stock_units) as stock_units,
-                    CAST(SUM(stock_cost) AS DECIMAL(15,3)) as stock_cost,
-                    CAST(SUM(stock_retail) AS DECIMAL(15,3)) as stock_retail
+                    SUM(variant_count)::integer as variant_count,
+                    SUM(stock_units)::integer as stock_units,
+                    ROUND(SUM(stock_cost)::numeric, 3) as stock_cost,
+                    ROUND(SUM(stock_retail)::numeric, 3) as stock_retail
                 FROM brand_totals
                 WHERE stock_cost <= 5000
             ),
@@ -101,51 +100,50 @@ router.get('/stock/metrics', async (req, res) => {
 // Returns purchase order metrics by vendor
 router.get('/purchase/metrics', async (req, res) => {
     try {
-        const [rows] = await executeQuery(`
+        const { rows: [poMetrics] } = await executeQuery(`
             SELECT 
                 COALESCE(COUNT(DISTINCT CASE 
-                    WHEN po.receiving_status < ${ReceivingStatus.PartialReceived} 
+                    WHEN po.receiving_status < $1 
                     THEN po.po_id 
-                END), 0) as active_pos,
+                END), 0)::integer as active_pos,
                 COALESCE(COUNT(DISTINCT CASE 
-                    WHEN po.receiving_status < ${ReceivingStatus.PartialReceived} 
-                    AND po.expected_date < CURDATE() 
+                    WHEN po.receiving_status < $1 
+                    AND po.expected_date < CURRENT_DATE 
                     THEN po.po_id 
-                END), 0) as overdue_pos,
+                END), 0)::integer as overdue_pos,
                 COALESCE(SUM(CASE 
-                    WHEN po.receiving_status < ${ReceivingStatus.PartialReceived} 
+                    WHEN po.receiving_status < $1 
                     THEN po.ordered 
                     ELSE 0 
-                END), 0) as total_units,
-                CAST(COALESCE(SUM(CASE 
-                    WHEN po.receiving_status < ${ReceivingStatus.PartialReceived} 
+                END), 0)::integer as total_units,
+                ROUND(COALESCE(SUM(CASE 
+                    WHEN po.receiving_status < $1 
                     THEN po.ordered * po.cost_price 
                     ELSE 0 
-                END), 0) AS DECIMAL(15,3)) as total_cost,
-                CAST(COALESCE(SUM(CASE 
-                    WHEN po.receiving_status < ${ReceivingStatus.PartialReceived} 
+                END), 0)::numeric, 3) as total_cost,
+                ROUND(COALESCE(SUM(CASE 
+                    WHEN po.receiving_status < $1 
                     THEN po.ordered * p.price 
                     ELSE 0 
-                END), 0) AS DECIMAL(15,3)) as total_retail
+                END), 0)::numeric, 3) as total_retail
             FROM purchase_orders po
             JOIN products p ON po.pid = p.pid
-        `);
-        const poMetrics = rows[0];
+        `, [ReceivingStatus.PartialReceived]);
 
-        const [vendorOrders] = await executeQuery(`
+        const { rows: vendorOrders } = await executeQuery(`
             SELECT 
                 po.vendor,
-                COUNT(DISTINCT po.po_id) as orders,
-                COALESCE(SUM(po.ordered), 0) as units,
-                CAST(COALESCE(SUM(po.ordered * po.cost_price), 0) AS DECIMAL(15,3)) as cost,
-                CAST(COALESCE(SUM(po.ordered * p.price), 0) AS DECIMAL(15,3)) as retail
+                COUNT(DISTINCT po.po_id)::integer as orders,
+                COALESCE(SUM(po.ordered), 0)::integer as units,
+                ROUND(COALESCE(SUM(po.ordered * po.cost_price), 0)::numeric, 3) as cost,
+                ROUND(COALESCE(SUM(po.ordered * p.price), 0)::numeric, 3) as retail
             FROM purchase_orders po
             JOIN products p ON po.pid = p.pid
-            WHERE po.receiving_status < ${ReceivingStatus.PartialReceived}
+            WHERE po.receiving_status < $1
             GROUP BY po.vendor
-            HAVING cost > 0
+            HAVING ROUND(COALESCE(SUM(po.ordered * po.cost_price), 0)::numeric, 3) > 0
             ORDER BY cost DESC
-        `);
+        `, [ReceivingStatus.PartialReceived]);
 
         // Format response to match PurchaseMetricsData interface
         const response = {
@@ -175,21 +173,21 @@ router.get('/purchase/metrics', async (req, res) => {
 router.get('/replenishment/metrics', async (req, res) => {
     try {
         // Get summary metrics
-        const [metrics] = await executeQuery(`
+        const { rows: [metrics] } = await executeQuery(`
             SELECT 
-                COUNT(DISTINCT p.pid) as products_to_replenish,
+                COUNT(DISTINCT p.pid)::integer as products_to_replenish,
                 COALESCE(SUM(CASE 
                     WHEN p.stock_quantity < 0 THEN ABS(p.stock_quantity) + pm.reorder_qty
                     ELSE pm.reorder_qty 
-                END), 0) as total_units_needed,
-                CAST(COALESCE(SUM(CASE 
+                END), 0)::integer as total_units_needed,
+                ROUND(COALESCE(SUM(CASE 
                     WHEN p.stock_quantity < 0 THEN (ABS(p.stock_quantity) + pm.reorder_qty) * p.cost_price
                     ELSE pm.reorder_qty * p.cost_price 
-                END), 0) AS DECIMAL(15,3)) as total_cost,
-                CAST(COALESCE(SUM(CASE 
+                END), 0)::numeric, 3) as total_cost,
+                ROUND(COALESCE(SUM(CASE 
                     WHEN p.stock_quantity < 0 THEN (ABS(p.stock_quantity) + pm.reorder_qty) * p.price
                     ELSE pm.reorder_qty * p.price 
-                END), 0) AS DECIMAL(15,3)) as total_retail
+                END), 0)::numeric, 3) as total_retail
             FROM products p
             JOIN product_metrics pm ON p.pid = pm.pid
             WHERE p.replenishable = true
@@ -199,23 +197,23 @@ router.get('/replenishment/metrics', async (req, res) => {
         `);
 
         // Get top variants to replenish
-        const [variants] = await executeQuery(`
+        const { rows: variants } = await executeQuery(`
             SELECT 
                 p.pid,
                 p.title,
-                p.stock_quantity as current_stock,
+                p.stock_quantity::integer as current_stock,
                 CASE 
                     WHEN p.stock_quantity < 0 THEN ABS(p.stock_quantity) + pm.reorder_qty
                     ELSE pm.reorder_qty 
-                END as replenish_qty,
-                CAST(CASE 
+                END::integer as replenish_qty,
+                ROUND(CASE 
                     WHEN p.stock_quantity < 0 THEN (ABS(p.stock_quantity) + pm.reorder_qty) * p.cost_price
                     ELSE pm.reorder_qty * p.cost_price 
-                END AS DECIMAL(15,3)) as replenish_cost,
-                CAST(CASE 
+                END::numeric, 3) as replenish_cost,
+                ROUND(CASE 
                     WHEN p.stock_quantity < 0 THEN (ABS(p.stock_quantity) + pm.reorder_qty) * p.price
                     ELSE pm.reorder_qty * p.price 
-                END AS DECIMAL(15,3)) as replenish_retail,
+                END::numeric, 3) as replenish_retail,
                 pm.stock_status
             FROM products p
             JOIN product_metrics pm ON p.pid = pm.pid
@@ -234,10 +232,10 @@ router.get('/replenishment/metrics', async (req, res) => {
 
         // Format response
         const response = {
-            productsToReplenish: parseInt(metrics[0].products_to_replenish) || 0,
-            unitsToReplenish: parseInt(metrics[0].total_units_needed) || 0,
-            replenishmentCost: parseFloat(metrics[0].total_cost) || 0,
-            replenishmentRetail: parseFloat(metrics[0].total_retail) || 0,
+            productsToReplenish: parseInt(metrics.products_to_replenish) || 0,
+            unitsToReplenish: parseInt(metrics.total_units_needed) || 0,
+            replenishmentCost: parseFloat(metrics.total_cost) || 0,
+            replenishmentRetail: parseFloat(metrics.total_retail) || 0,
             topVariants: variants.map(v => ({
                 id: v.pid,
                 title: v.title,

inventory-server/src/routes/import.js

@@ -0,0 +1,270 @@
+const express = require('express');
+const router = express.Router();
+const { Client } = require('ssh2');
+const mysql = require('mysql2/promise');
+
+// Helper function to setup SSH tunnel
+async function setupSshTunnel() {
+  const sshConfig = {
+    host: process.env.PROD_SSH_HOST,
+    port: process.env.PROD_SSH_PORT || 22,
+    username: process.env.PROD_SSH_USER,
+    privateKey: process.env.PROD_SSH_KEY_PATH
+      ? require('fs').readFileSync(process.env.PROD_SSH_KEY_PATH)
+      : undefined,
+    compress: true
+  };
+
+  const dbConfig = {
+    host: process.env.PROD_DB_HOST || 'localhost',
+    user: process.env.PROD_DB_USER,
+    password: process.env.PROD_DB_PASSWORD,
+    database: process.env.PROD_DB_NAME,
+    port: process.env.PROD_DB_PORT || 3306,
+    timezone: 'Z'
+  };
+
+  return new Promise((resolve, reject) => {
+    const ssh = new Client();
+
+    ssh.on('error', (err) => {
+      console.error('SSH connection error:', err);
+      reject(err);
+    });
+
+    ssh.on('ready', () => {
+      ssh.forwardOut(
+        '127.0.0.1',
+        0,
+        dbConfig.host,
+        dbConfig.port,
+        (err, stream) => {
+          if (err) reject(err);
+          resolve({ ssh, stream, dbConfig });
+        }
+      );
+    }).connect(sshConfig);
+  });
+}
+
+// Get all options for import fields
+router.get('/field-options', async (req, res) => {
+    let ssh;
+    let connection;
+    
+    try {
+        // Setup SSH tunnel and get database connection
+        const tunnel = await setupSshTunnel();
+        ssh = tunnel.ssh;
+        
+        // Create MySQL connection over SSH tunnel
+        connection = await mysql.createConnection({
+            ...tunnel.dbConfig,
+            stream: tunnel.stream
+        });
+
+        // Fetch companies (type 1)
+        const [companies] = await connection.query(`
+            SELECT cat_id, name 
+            FROM product_categories 
+            WHERE type = 1 
+            ORDER BY name
+        `);
+
+        // Fetch artists (type 40)
+        const [artists] = await connection.query(`
+            SELECT cat_id, name 
+            FROM product_categories 
+            WHERE type = 40 
+            ORDER BY name
+        `);
+
+        // Fetch sizes (type 50)
+        const [sizes] = await connection.query(`
+            SELECT cat_id, name 
+            FROM product_categories 
+            WHERE type = 50 
+            ORDER BY name
+        `);
+
+        // Fetch themes with subthemes
+        const [themes] = await connection.query(`
+            SELECT t.cat_id, t.name AS display_name, t.type, t.name AS sort_theme, 
+                   '' AS sort_subtheme, 1 AS level_order 
+            FROM product_categories t 
+            WHERE t.type = 20 
+            UNION ALL 
+            SELECT ts.cat_id, CONCAT(t.name,' - ',ts.name) AS display_name, ts.type,
+                   t.name AS sort_theme, ts.name AS sort_subtheme, 2 AS level_order 
+            FROM product_categories ts 
+            JOIN product_categories t ON ts.master_cat_id = t.cat_id 
+            WHERE ts.type = 21 AND t.type = 20 
+            ORDER BY sort_theme, sort_subtheme
+        `);
+
+        // Fetch categories with all levels
+        const [categories] = await connection.query(`
+            SELECT s.cat_id, s.name AS display_name, s.type, s.name AS sort_section,
+                   '' AS sort_category, '' AS sort_subcategory, '' AS sort_subsubcategory,
+                   1 AS level_order 
+            FROM product_categories s 
+            WHERE s.type = 10 
+            UNION ALL 
+            SELECT c.cat_id, CONCAT(s.name,' - ',c.name) AS display_name, c.type,
+                   s.name AS sort_section, c.name AS sort_category, '' AS sort_subcategory,
+                   '' AS sort_subsubcategory, 2 AS level_order 
+            FROM product_categories c 
+            JOIN product_categories s ON c.master_cat_id = s.cat_id 
+            WHERE c.type = 11 AND s.type = 10 
+            UNION ALL 
+            SELECT sc.cat_id, CONCAT(s.name,' - ',c.name,' - ',sc.name) AS display_name,
+                   sc.type, s.name AS sort_section, c.name AS sort_category,
+                   sc.name AS sort_subcategory, '' AS sort_subsubcategory, 3 AS level_order 
+            FROM product_categories sc 
+            JOIN product_categories c ON sc.master_cat_id = c.cat_id 
+            JOIN product_categories s ON c.master_cat_id = s.cat_id 
+            WHERE sc.type = 12 AND c.type = 11 AND s.type = 10 
+            UNION ALL 
+            SELECT ssc.cat_id, CONCAT(s.name,' - ',c.name,' - ',sc.name,' - ',ssc.name) AS display_name,
+                   ssc.type, s.name AS sort_section, c.name AS sort_category,
+                   sc.name AS sort_subcategory, ssc.name AS sort_subsubcategory, 4 AS level_order 
+            FROM product_categories ssc 
+            JOIN product_categories sc ON ssc.master_cat_id = sc.cat_id 
+            JOIN product_categories c ON sc.master_cat_id = c.cat_id 
+            JOIN product_categories s ON c.master_cat_id = s.cat_id 
+            WHERE ssc.type = 13 AND sc.type = 12 AND c.type = 11 AND s.type = 10 
+            ORDER BY sort_section, sort_category, sort_subcategory, sort_subsubcategory
+        `);
+
+        // Fetch colors
+        const [colors] = await connection.query(`
+            SELECT color, name, hex_color 
+            FROM product_color_list 
+            ORDER BY \`order\`
+        `);
+
+        // Fetch suppliers
+        const [suppliers] = await connection.query(`
+            SELECT supplierid as value, companyname as label 
+            FROM suppliers 
+            WHERE companyname <> '' 
+            ORDER BY companyname
+        `);
+
+        // Fetch tax categories
+        const [taxCategories] = await connection.query(`
+            SELECT tax_code_id as value, name as label 
+            FROM product_tax_codes 
+            ORDER BY tax_code_id = 0 DESC, name
+        `);
+
+        res.json({
+            companies: companies.map(c => ({ label: c.name, value: c.cat_id.toString() })),
+            artists: artists.map(a => ({ label: a.name, value: a.cat_id.toString() })),
+            sizes: sizes.map(s => ({ label: s.name, value: s.cat_id.toString() })),
+            themes: themes.map(t => ({ 
+                label: t.display_name, 
+                value: t.cat_id.toString(),
+                type: t.type,
+                level: t.level_order
+            })),
+            categories: categories.map(c => ({ 
+                label: c.display_name, 
+                value: c.cat_id.toString(),
+                type: c.type,
+                level: c.level_order
+            })),
+            colors: colors.map(c => ({ 
+                label: c.name, 
+                value: c.color,
+                hexColor: c.hex_color 
+            })),
+            suppliers: suppliers,
+            taxCategories: taxCategories,
+            shippingRestrictions: [
+                { label: "None", value: "0" },
+                { label: "US Only", value: "1" },
+                { label: "Limited Quantity", value: "2" },
+                { label: "US/CA Only", value: "3" },
+                { label: "No FedEx 2 Day", value: "4" },
+                { label: "North America Only", value: "5" }
+            ]
+        });
+    } catch (error) {
+        console.error('Error fetching import field options:', error);
+        res.status(500).json({ error: 'Failed to fetch import field options' });
+    } finally {
+        if (connection) await connection.end();
+        if (ssh) ssh.end();
+    }
+});
+
+// Get product lines for a specific company
+router.get('/product-lines/:companyId', async (req, res) => {
+    let ssh;
+    let connection;
+    
+    try {
+        // Setup SSH tunnel and get database connection
+        const tunnel = await setupSshTunnel();
+        ssh = tunnel.ssh;
+        
+        // Create MySQL connection over SSH tunnel
+        connection = await mysql.createConnection({
+            ...tunnel.dbConfig,
+            stream: tunnel.stream
+        });
+
+        const [lines] = await connection.query(`
+            SELECT cat_id as value, name as label 
+            FROM product_categories 
+            WHERE type = 2 
+            AND master_cat_id = ? 
+            ORDER BY name
+        `, [req.params.companyId]);
+
+        res.json(lines.map(l => ({ label: l.label, value: l.value.toString() })));
+    } catch (error) {
+        console.error('Error fetching product lines:', error);
+        res.status(500).json({ error: 'Failed to fetch product lines' });
+    } finally {
+        if (connection) await connection.end();
+        if (ssh) ssh.end();
+    }
+});
+
+// Get sublines for a specific product line
+router.get('/sublines/:lineId', async (req, res) => {
+    let ssh;
+    let connection;
+    
+    try {
+        // Setup SSH tunnel and get database connection
+        const tunnel = await setupSshTunnel();
+        ssh = tunnel.ssh;
+        
+        // Create MySQL connection over SSH tunnel
+        connection = await mysql.createConnection({
+            ...tunnel.dbConfig,
+            stream: tunnel.stream
+        });
+
+        const [sublines] = await connection.query(`
+            SELECT cat_id as value, name as label 
+            FROM product_categories 
+            WHERE type = 3 
+            AND master_cat_id = ? 
+            ORDER BY name
+        `, [req.params.lineId]);
+
+        res.json(sublines.map(s => ({ label: s.label, value: s.value.toString() })));
+    } catch (error) {
+        console.error('Error fetching sublines:', error);
+        res.status(500).json({ error: 'Failed to fetch sublines' });
+    } finally {
+        if (connection) await connection.end();
+        if (ssh) ssh.end();
+    }
+});
+
+module.exports = router; 

inventory-server/src/routes/metrics.js

@@ -5,26 +5,28 @@ const router = express.Router();
 router.get('/trends', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
-    const [rows] = await pool.query(`
+    const { rows } = await pool.query(`
       WITH MonthlyMetrics AS (
         SELECT 
-          DATE(CONCAT(pta.year, '-', LPAD(pta.month, 2, '0'), '-01')) as date,
-          CAST(COALESCE(SUM(pta.total_revenue), 0) AS DECIMAL(15,3)) as revenue,
-          CAST(COALESCE(SUM(pta.total_cost), 0) AS DECIMAL(15,3)) as cost,
-          CAST(COALESCE(SUM(pm.inventory_value), 0) AS DECIMAL(15,3)) as inventory_value,
+          make_date(pta.year, pta.month, 1) as date,
+          ROUND(COALESCE(SUM(pta.total_revenue), 0)::numeric, 3) as revenue,
+          ROUND(COALESCE(SUM(pta.total_cost), 0)::numeric, 3) as cost,
+          ROUND(COALESCE(SUM(pm.inventory_value), 0)::numeric, 3) as inventory_value,
           CASE 
             WHEN SUM(pm.inventory_value) > 0 
-            THEN CAST((SUM(pta.total_revenue - pta.total_cost) / SUM(pm.inventory_value)) * 100 AS DECIMAL(15,3))
+            THEN ROUND((SUM(pta.total_revenue - pta.total_cost) / SUM(pm.inventory_value) * 100)::numeric, 3)
             ELSE 0 
           END as gmroi
         FROM product_time_aggregates pta
         JOIN product_metrics pm ON pta.pid = pm.pid
-        WHERE (pta.year * 100 + pta.month) >= DATE_FORMAT(DATE_SUB(CURDATE(), INTERVAL 12 MONTH), '%Y%m')
+        WHERE (pta.year * 100 + pta.month) >= 
+          EXTRACT(YEAR FROM CURRENT_DATE - INTERVAL '12 months')::integer * 100 + 
+          EXTRACT(MONTH FROM CURRENT_DATE - INTERVAL '12 months')::integer
         GROUP BY pta.year, pta.month
         ORDER BY date ASC
       )
       SELECT 
-        DATE_FORMAT(date, '%b %y') as date,
+        to_char(date, 'Mon YY') as date,
         revenue,
         inventory_value,
         gmroi

inventory-server/src/routes/orders.js

@@ -20,39 +20,46 @@ router.get('/', async (req, res) => {
     // Build the WHERE clause
     const conditions = ['o1.canceled = false'];
     const params = [];
+    let paramCounter = 1;
 
     if (search) {
-      conditions.push('(o1.order_number LIKE ? OR o1.customer LIKE ?)');
-      params.push(`%${search}%`, `%${search}%`);
+      conditions.push(`(o1.order_number ILIKE $${paramCounter} OR o1.customer ILIKE $${paramCounter})`);
+      params.push(`%${search}%`);
+      paramCounter++;
     }
 
     if (status !== 'all') {
-      conditions.push('o1.status = ?');
+      conditions.push(`o1.status = $${paramCounter}`);
       params.push(status);
+      paramCounter++;
     }
 
     if (fromDate) {
-      conditions.push('DATE(o1.date) >= DATE(?)');
+      conditions.push(`DATE(o1.date) >= DATE($${paramCounter})`);
       params.push(fromDate.toISOString());
+      paramCounter++;
     }
 
     if (toDate) {
-      conditions.push('DATE(o1.date) <= DATE(?)');
+      conditions.push(`DATE(o1.date) <= DATE($${paramCounter})`);
       params.push(toDate.toISOString());
+      paramCounter++;
     }
 
     if (minAmount > 0) {
-      conditions.push('total_amount >= ?');
+      conditions.push(`total_amount >= $${paramCounter}`);
       params.push(minAmount);
+      paramCounter++;
     }
 
     if (maxAmount) {
-      conditions.push('total_amount <= ?');
+      conditions.push(`total_amount <= $${paramCounter}`);
       params.push(maxAmount);
+      paramCounter++;
     }
 
     // Get total count for pagination
-    const [countResult] = await pool.query(`
+    const { rows: [countResult] } = await pool.query(`
       SELECT COUNT(DISTINCT o1.order_number) as total 
       FROM orders o1
       LEFT JOIN (
@@ -63,7 +70,7 @@ router.get('/', async (req, res) => {
       WHERE ${conditions.join(' AND ')}
     `, params);
     
-    const total = countResult[0].total;
+    const total = countResult.total;
 
     // Get paginated results
     const query = `
@@ -75,7 +82,7 @@ router.get('/', async (req, res) => {
         o1.payment_method,
         o1.shipping_method,
         COUNT(o2.pid) as items_count,
-        CAST(SUM(o2.price * o2.quantity) AS DECIMAL(15,3)) as total_amount
+        ROUND(SUM(o2.price * o2.quantity)::numeric, 3) as total_amount
       FROM orders o1
       JOIN orders o2 ON o1.order_number = o2.order_number
       WHERE ${conditions.join(' AND ')}
@@ -91,36 +98,37 @@ router.get('/', async (req, res) => {
           ? `${sortColumn} ${sortDirection}`
           : `o1.${sortColumn} ${sortDirection}`
       }
-      LIMIT ? OFFSET ?
+      LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
     `;
 
-    const [rows] = await pool.query(query, [...params, limit, offset]);
+    params.push(limit, offset);
+    const { rows } = await pool.query(query, params);
 
     // Get order statistics
-    const [stats] = await pool.query(`
+    const { rows: [orderStats] } = await pool.query(`
       WITH CurrentStats AS (
         SELECT 
           COUNT(DISTINCT order_number) as total_orders,
-          CAST(SUM(price * quantity) AS DECIMAL(15,3)) as total_revenue
+          ROUND(SUM(price * quantity)::numeric, 3) as total_revenue
         FROM orders
         WHERE canceled = false
-          AND DATE(date) >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+          AND DATE(date) >= CURRENT_DATE - INTERVAL '30 days'
       ),
       PreviousStats AS (
         SELECT 
           COUNT(DISTINCT order_number) as prev_orders,
-          CAST(SUM(price * quantity) AS DECIMAL(15,3)) as prev_revenue
+          ROUND(SUM(price * quantity)::numeric, 3) as prev_revenue
         FROM orders
         WHERE canceled = false
-          AND DATE(date) BETWEEN DATE_SUB(CURDATE(), INTERVAL 60 DAY) AND DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+          AND DATE(date) BETWEEN CURRENT_DATE - INTERVAL '60 days' AND CURRENT_DATE - INTERVAL '30 days'
       ),
       OrderValues AS (
         SELECT 
           order_number,
-          CAST(SUM(price * quantity) AS DECIMAL(15,3)) as order_value
+          ROUND(SUM(price * quantity)::numeric, 3) as order_value
         FROM orders
         WHERE canceled = false
-          AND DATE(date) >= DATE_SUB(CURDATE(), INTERVAL 30 DAY)
+          AND DATE(date) >= CURRENT_DATE - INTERVAL '30 days'
         GROUP BY order_number
       )
       SELECT 
@@ -128,29 +136,27 @@ router.get('/', async (req, res) => {
         cs.total_revenue,
         CASE 
           WHEN ps.prev_orders > 0 
-          THEN ((cs.total_orders - ps.prev_orders) / ps.prev_orders * 100)
+          THEN ROUND(((cs.total_orders - ps.prev_orders)::numeric / ps.prev_orders * 100), 1)
           ELSE 0 
         END as order_growth,
         CASE 
           WHEN ps.prev_revenue > 0 
-          THEN ((cs.total_revenue - ps.prev_revenue) / ps.prev_revenue * 100)
+          THEN ROUND(((cs.total_revenue - ps.prev_revenue)::numeric / ps.prev_revenue * 100), 1)
           ELSE 0 
         END as revenue_growth,
         CASE 
           WHEN cs.total_orders > 0 
-          THEN CAST((cs.total_revenue / cs.total_orders) AS DECIMAL(15,3))
+          THEN ROUND((cs.total_revenue::numeric / cs.total_orders), 3)
           ELSE 0 
         END as average_order_value,
         CASE 
           WHEN ps.prev_orders > 0 
-          THEN CAST((ps.prev_revenue / ps.prev_orders) AS DECIMAL(15,3))
+          THEN ROUND((ps.prev_revenue::numeric / ps.prev_orders), 3)
           ELSE 0 
         END as prev_average_order_value
       FROM CurrentStats cs
       CROSS JOIN PreviousStats ps
     `);
-
-    const orderStats = stats[0];
     
     res.json({
       orders: rows.map(row => ({
@@ -189,7 +195,7 @@ router.get('/:orderNumber', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     // Get order details
-    const [orderRows] = await pool.query(`
+    const { rows: orderRows } = await pool.query(`
       SELECT DISTINCT
         o1.order_number,
         o1.customer,
@@ -200,10 +206,10 @@ router.get('/:orderNumber', async (req, res) => {
         o1.shipping_address,
         o1.billing_address,
         COUNT(o2.pid) as items_count,
-        CAST(SUM(o2.price * o2.quantity) AS DECIMAL(15,3)) as total_amount
+        ROUND(SUM(o2.price * o2.quantity)::numeric, 3) as total_amount
       FROM orders o1
       JOIN orders o2 ON o1.order_number = o2.order_number
-      WHERE o1.order_number = ? AND o1.canceled = false
+      WHERE o1.order_number = $1 AND o1.canceled = false
       GROUP BY 
         o1.order_number,
         o1.customer,
@@ -220,17 +226,17 @@ router.get('/:orderNumber', async (req, res) => {
     }
 
     // Get order items
-    const [itemRows] = await pool.query(`
+    const { rows: itemRows } = await pool.query(`
       SELECT 
         o.pid,
         p.title,
         p.SKU,
         o.quantity,
         o.price,
-        CAST((o.price * o.quantity) AS DECIMAL(15,3)) as total
+        ROUND((o.price * o.quantity)::numeric, 3) as total
       FROM orders o
       JOIN products p ON o.pid = p.pid
-      WHERE o.order_number = ? AND o.canceled = false
+      WHERE o.order_number = $1 AND o.canceled = false
     `, [req.params.orderNumber]);
 
     const order = {

inventory-server/src/routes/products.js

@@ -20,7 +20,7 @@ router.get('/brands', async (req, res) => {
         const pool = req.app.locals.pool;
         console.log('Fetching brands from database...');
         
-        const [results] = await pool.query(`
+        const { rows } = await pool.query(`
             SELECT DISTINCT COALESCE(p.brand, 'Unbranded') as brand
             FROM products p
             JOIN purchase_orders po ON p.pid = po.pid
@@ -30,8 +30,8 @@ router.get('/brands', async (req, res) => {
             ORDER BY COALESCE(p.brand, 'Unbranded')
         `);
         
-        console.log(`Found ${results.length} brands:`, results.slice(0, 3));
-        res.json(results.map(r => r.brand));
+        console.log(`Found ${rows.length} brands:`, rows.slice(0, 3));
+        res.json(rows.map(r => r.brand));
     } catch (error) {
         console.error('Error fetching brands:', error);
         res.status(500).json({ error: 'Failed to fetch brands' });
@@ -50,6 +50,7 @@ router.get('/', async (req, res) => {
 
     const conditions = ['p.visible = true'];
     const params = [];
+    let paramCounter = 1;
 
     // Add default replenishable filter unless explicitly showing non-replenishable
     if (req.query.showNonReplenishable !== 'true') {
@@ -58,9 +59,10 @@ router.get('/', async (req, res) => {
 
     // Handle search filter
     if (req.query.search) {
-      conditions.push('(p.title LIKE ? OR p.SKU LIKE ? OR p.barcode LIKE ?)');
+      conditions.push(`(p.title ILIKE $${paramCounter} OR p.SKU ILIKE $${paramCounter} OR p.barcode ILIKE $${paramCounter})`);
       const searchTerm = `%${req.query.search}%`;
-      params.push(searchTerm, searchTerm, searchTerm);
+      params.push(searchTerm);
+      paramCounter++;
     }
 
     // Handle numeric filters with operators
@@ -84,61 +86,69 @@ router.get('/', async (req, res) => {
       if (field) {
         const operator = req.query[`${key}_operator`] || '=';
         if (operator === 'between') {
-          // Handle between operator
           try {
             const [min, max] = JSON.parse(value);
-            conditions.push(`${field} BETWEEN ? AND ?`);
+            conditions.push(`${field} BETWEEN $${paramCounter} AND $${paramCounter + 1}`);
             params.push(min, max);
+            paramCounter += 2;
           } catch (e) {
             console.error(`Invalid between value for ${key}:`, value);
           }
         } else {
-          // Handle other operators
-          conditions.push(`${field} ${operator} ?`);
+          conditions.push(`${field} ${operator} $${paramCounter}`);
           params.push(parseFloat(value));
+          paramCounter++;
         }
       }
     });
 
     // Handle select filters
     if (req.query.vendor) {
-      conditions.push('p.vendor = ?');
+      conditions.push(`p.vendor = $${paramCounter}`);
       params.push(req.query.vendor);
+      paramCounter++;
     }
 
     if (req.query.brand) {
-      conditions.push('p.brand = ?');
+      conditions.push(`p.brand = $${paramCounter}`);
       params.push(req.query.brand);
+      paramCounter++;
     }
 
     if (req.query.category) {
-      conditions.push('p.categories LIKE ?');
+      conditions.push(`p.categories ILIKE $${paramCounter}`);
       params.push(`%${req.query.category}%`);
+      paramCounter++;
     }
 
     if (req.query.stockStatus && req.query.stockStatus !== 'all') {
-      conditions.push('pm.stock_status = ?');
+      conditions.push(`pm.stock_status = $${paramCounter}`);
       params.push(req.query.stockStatus);
+      paramCounter++;
     }
 
     if (req.query.abcClass) {
-      conditions.push('pm.abc_class = ?');
+      conditions.push(`pm.abc_class = $${paramCounter}`);
       params.push(req.query.abcClass);
+      paramCounter++;
     }
 
     if (req.query.leadTimeStatus) {
-      conditions.push('pm.lead_time_status = ?');
+      conditions.push(`pm.lead_time_status = $${paramCounter}`);
       params.push(req.query.leadTimeStatus);
+      paramCounter++;
     }
 
     if (req.query.replenishable !== undefined) {
-      conditions.push('p.replenishable = ?');
-      params.push(req.query.replenishable === 'true' ? 1 : 0);
+      conditions.push(`p.replenishable = $${paramCounter}`);
+      params.push(req.query.replenishable === 'true');
+      paramCounter++;
     }
 
     if (req.query.managingStock !== undefined) {
-      conditions.push('p.managing_stock = ?');
-      params.push(req.query.managingStock === 'true' ? 1 : 0);
+      conditions.push(`p.managing_stock = $${paramCounter}`);
+      params.push(req.query.managingStock === 'true');
+      paramCounter++;
     }
 
     // Combine all conditions with AND
@@ -151,17 +161,17 @@ router.get('/', async (req, res) => {
       LEFT JOIN product_metrics pm ON p.pid = pm.pid
       ${whereClause}
     `;
-    const [countResult] = await pool.query(countQuery, params);
-    const total = countResult[0].total;
+    const { rows: [countResult] } = await pool.query(countQuery, params);
+    const total = countResult.total;
 
     // Get available filters
-    const [categories] = await pool.query(
+    const { rows: categories } = await pool.query(
       'SELECT name FROM categories ORDER BY name'
     );
-    const [vendors] = await pool.query(
-      'SELECT DISTINCT vendor FROM products WHERE visible = true AND vendor IS NOT NULL AND vendor != "" ORDER BY vendor'
+    const { rows: vendors } = await pool.query(
+      'SELECT DISTINCT vendor FROM products WHERE visible = true AND vendor IS NOT NULL AND vendor != \'\' ORDER BY vendor'
     );
-    const [brands] = await pool.query(
+    const { rows: brands } = await pool.query(
       'SELECT DISTINCT COALESCE(brand, \'Unbranded\') as brand FROM products WHERE visible = true ORDER BY brand'
     );
 
@@ -173,7 +183,7 @@ router.get('/', async (req, res) => {
             c.cat_id,
             c.name,
             c.parent_id,
-            CAST(c.name AS CHAR(1000)) as path
+            CAST(c.name AS text) as path
           FROM categories c
           WHERE c.parent_id IS NULL
           
@@ -183,7 +193,7 @@ router.get('/', async (req, res) => {
             c.cat_id,
             c.name,
             c.parent_id,
-            CONCAT(cp.path, ' > ', c.name)
+            cp.path || ' > ' || c.name
           FROM categories c
           JOIN category_path cp ON c.parent_id = cp.cat_id
         ),
@@ -210,7 +220,6 @@ router.get('/', async (req, res) => {
           FROM products p
         ),
         product_leaf_categories AS (
-          -- Find categories that aren't parents to other categories for this product
           SELECT DISTINCT pc.cat_id
           FROM product_categories pc
           WHERE NOT EXISTS (
@@ -224,7 +233,7 @@ router.get('/', async (req, res) => {
       SELECT 
         p.*,
         COALESCE(p.brand, 'Unbranded') as brand,
-        GROUP_CONCAT(DISTINCT CONCAT(c.cat_id, ':', c.name)) as categories,
+        string_agg(DISTINCT (c.cat_id || ':' || c.name), ',') as categories,
         pm.daily_sales_avg,
         pm.weekly_sales_avg,
         pm.monthly_sales_avg,
@@ -247,83 +256,32 @@ router.get('/', async (req, res) => {
         pm.last_received_date,
         pm.abc_class,
         pm.stock_status,
-        pm.turnover_rate,
-        pm.current_lead_time,
-        pm.target_lead_time,
-        pm.lead_time_status,
-        pm.reorder_qty,
-        pm.overstocked_amt,
-        COALESCE(pm.days_of_inventory / NULLIF(pt.target_days, 0), 0) as stock_coverage_ratio
+        pm.turnover_rate
       FROM products p
       LEFT JOIN product_metrics pm ON p.pid = pm.pid
       LEFT JOIN product_categories pc ON p.pid = pc.pid
       LEFT JOIN categories c ON pc.cat_id = c.cat_id
-      LEFT JOIN product_thresholds pt ON p.pid = pt.pid
-      JOIN product_leaf_categories plc ON c.cat_id = plc.cat_id
-      ${whereClause ? 'WHERE ' + whereClause.substring(6) : ''}
-      GROUP BY p.pid
+      ${whereClause}
+      GROUP BY p.pid, pm.pid
       ORDER BY ${sortColumn} ${sortDirection}
-      LIMIT ? OFFSET ?
+      LIMIT $${paramCounter} OFFSET $${paramCounter + 1}
     `;
-
-    // Add pagination params to the main query params
-    const queryParams = [...params, limit, offset];
-    console.log('Query:', query.replace(/\s+/g, ' '));
-    console.log('Params:', queryParams);
-
-    const [rows] = await pool.query(query, queryParams);
-
-    // Transform the results
-    const products = rows.map(row => ({
-      ...row,
-      categories: row.categories ? row.categories.split(',') : [],
-      price: parseFloat(row.price),
-      cost_price: parseFloat(row.cost_price),
-      landing_cost_price: row.landing_cost_price ? parseFloat(row.landing_cost_price) : null,
-      stock_quantity: parseInt(row.stock_quantity),
-      daily_sales_avg: parseFloat(row.daily_sales_avg) || 0,
-      weekly_sales_avg: parseFloat(row.weekly_sales_avg) || 0,
-      monthly_sales_avg: parseFloat(row.monthly_sales_avg) || 0,
-      avg_quantity_per_order: parseFloat(row.avg_quantity_per_order) || 0,
-      number_of_orders: parseInt(row.number_of_orders) || 0,
-      first_sale_date: row.first_sale_date || null,
-      last_sale_date: row.last_sale_date || null,
-      days_of_inventory: parseFloat(row.days_of_inventory) || 0,
-      weeks_of_inventory: parseFloat(row.weeks_of_inventory) || 0,
-      reorder_point: parseFloat(row.reorder_point) || 0,
-      safety_stock: parseFloat(row.safety_stock) || 0,
-      avg_margin_percent: parseFloat(row.avg_margin_percent) || 0,
-      total_revenue: parseFloat(row.total_revenue) || 0,
-      inventory_value: parseFloat(row.inventory_value) || 0,
-      cost_of_goods_sold: parseFloat(row.cost_of_goods_sold) || 0,
-      gross_profit: parseFloat(row.gross_profit) || 0,
-      gmroi: parseFloat(row.gmroi) || 0,
-      avg_lead_time_days: parseFloat(row.avg_lead_time_days) || 0,
-      last_purchase_date: row.last_purchase_date || null,
-      last_received_date: row.last_received_date || null,
-      abc_class: row.abc_class || null,
-      stock_status: row.stock_status || null,
-      turnover_rate: parseFloat(row.turnover_rate) || 0,
-      current_lead_time: parseFloat(row.current_lead_time) || 0,
-      target_lead_time: parseFloat(row.target_lead_time) || 0,
-      lead_time_status: row.lead_time_status || null,
-      stock_coverage_ratio: parseFloat(row.stock_coverage_ratio) || 0,
-      reorder_qty: parseInt(row.reorder_qty) || 0,
-      overstocked_amt: parseInt(row.overstocked_amt) || 0
-    }));
+    
+    params.push(limit, offset);
+    const { rows: products } = await pool.query(query, params);
 
     res.json({
       products,
       pagination: {
         total,
-        currentPage: page,
         pages: Math.ceil(total / limit),
+        currentPage: page,
         limit
       },
       filters: {
-        categories: categories.map(category => category.name),
-        vendors: vendors.map(vendor => vendor.vendor),
-        brands: brands.map(brand => brand.brand)
+        categories: categories.map(c => c.name),
+        vendors: vendors.map(v => v.vendor),
+        brands: brands.map(b => b.brand)
       }
     });
   } catch (error) {

inventory-server/src/routes/purchase-orders.js

@@ -29,40 +29,46 @@ router.get('/', async (req, res) => {
     
     let whereClause = '1=1';
     const params = [];
+    let paramCounter = 1;
     
     if (search) {
-      whereClause += ' AND (po.po_id LIKE ? OR po.vendor LIKE ?)';
-      params.push(`%${search}%`, `%${search}%`);
+      whereClause += ` AND (po.po_id ILIKE $${paramCounter} OR po.vendor ILIKE $${paramCounter})`;
+      params.push(`%${search}%`);
+      paramCounter++;
     }
     
     if (status && status !== 'all') {
-      whereClause += ' AND po.status = ?';
+      whereClause += ` AND po.status = $${paramCounter}`;
       params.push(Number(status));
+      paramCounter++;
     }
     
     if (vendor && vendor !== 'all') {
-      whereClause += ' AND po.vendor = ?';
+      whereClause += ` AND po.vendor = $${paramCounter}`;
       params.push(vendor);
+      paramCounter++;
     }
     
     if (startDate) {
-      whereClause += ' AND po.date >= ?';
+      whereClause += ` AND po.date >= $${paramCounter}`;
       params.push(startDate);
+      paramCounter++;
     }
     
     if (endDate) {
-      whereClause += ' AND po.date <= ?';
+      whereClause += ` AND po.date <= $${paramCounter}`;
       params.push(endDate);
+      paramCounter++;
     }
 
     // Get filtered summary metrics
-    const [summary] = await pool.query(`
+    const { rows: [summary] } = await pool.query(`
       WITH po_totals AS (
         SELECT 
           po_id,
           SUM(ordered) as total_ordered,
           SUM(received) as total_received,
-          CAST(SUM(ordered * cost_price) AS DECIMAL(15,3)) as total_cost
+          ROUND(SUM(ordered * cost_price)::numeric, 3) as total_cost
         FROM purchase_orders po
         WHERE ${whereClause}
         GROUP BY po_id
@@ -72,26 +78,26 @@ router.get('/', async (req, res) => {
         SUM(total_ordered) as total_ordered,
         SUM(total_received) as total_received,
         ROUND(
-          SUM(total_received) / NULLIF(SUM(total_ordered), 0), 3
+          (SUM(total_received)::numeric / NULLIF(SUM(total_ordered), 0)), 3
         ) as fulfillment_rate,
-        CAST(SUM(total_cost) AS DECIMAL(15,3)) as total_value,
-        CAST(AVG(total_cost) AS DECIMAL(15,3)) as avg_cost
+        ROUND(SUM(total_cost)::numeric, 3) as total_value,
+        ROUND(AVG(total_cost)::numeric, 3) as avg_cost
       FROM po_totals
     `, params);
 
     // Get total count for pagination
-    const [countResult] = await pool.query(`
+    const { rows: [countResult] } = await pool.query(`
       SELECT COUNT(DISTINCT po_id) as total
       FROM purchase_orders po
       WHERE ${whereClause}
     `, params);
 
-    const total = countResult[0].total;
+    const total = countResult.total;
     const offset = (page - 1) * limit;
     const pages = Math.ceil(total / limit);
 
     // Get recent purchase orders
-    const [orders] = await pool.query(`
+    const { rows: orders } = await pool.query(`
       WITH po_totals AS (
         SELECT 
           po_id,
@@ -101,10 +107,10 @@ router.get('/', async (req, res) => {
           receiving_status,
           COUNT(DISTINCT pid) as total_items,
           SUM(ordered) as total_quantity,
-          CAST(SUM(ordered * cost_price) AS DECIMAL(15,3)) as total_cost,
+          ROUND(SUM(ordered * cost_price)::numeric, 3) as total_cost,
           SUM(received) as total_received,
           ROUND(
-            SUM(received) / NULLIF(SUM(ordered), 0), 3
+            (SUM(received)::numeric / NULLIF(SUM(ordered), 0)), 3
           ) as fulfillment_rate
         FROM purchase_orders po
         WHERE ${whereClause}
@@ -113,7 +119,7 @@ router.get('/', async (req, res) => {
       SELECT 
         po_id as id,
         vendor as vendor_name,
-        DATE_FORMAT(date, '%Y-%m-%d') as order_date,
+        to_char(date, 'YYYY-MM-DD') as order_date,
         status,
         receiving_status,
         total_items,
@@ -124,21 +130,21 @@ router.get('/', async (req, res) => {
       FROM po_totals
       ORDER BY 
         CASE 
-          WHEN ? = 'order_date' THEN date
-          WHEN ? = 'vendor_name' THEN vendor
-          WHEN ? = 'total_cost' THEN CAST(total_cost AS DECIMAL(15,3))
-          WHEN ? = 'total_received' THEN CAST(total_received AS DECIMAL(15,3))
-          WHEN ? = 'total_items' THEN CAST(total_items AS SIGNED)
-          WHEN ? = 'total_quantity' THEN CAST(total_quantity AS SIGNED)
-          WHEN ? = 'fulfillment_rate' THEN CAST(fulfillment_rate AS DECIMAL(5,3))
-          WHEN ? = 'status' THEN status
+          WHEN $${paramCounter} = 'order_date' THEN date
+          WHEN $${paramCounter} = 'vendor_name' THEN vendor
+          WHEN $${paramCounter} = 'total_cost' THEN total_cost
+          WHEN $${paramCounter} = 'total_received' THEN total_received
+          WHEN $${paramCounter} = 'total_items' THEN total_items
+          WHEN $${paramCounter} = 'total_quantity' THEN total_quantity
+          WHEN $${paramCounter} = 'fulfillment_rate' THEN fulfillment_rate
+          WHEN $${paramCounter} = 'status' THEN status
           ELSE date
         END ${sortDirection === 'desc' ? 'DESC' : 'ASC'}
-      LIMIT ? OFFSET ?
-    `, [...params, sortColumn, sortColumn, sortColumn, sortColumn, sortColumn, sortColumn, sortColumn, sortColumn, Number(limit), offset]);
+      LIMIT $${paramCounter + 1} OFFSET $${paramCounter + 2}
+    `, [...params, sortColumn, Number(limit), offset]);
 
     // Get unique vendors for filter options
-    const [vendors] = await pool.query(`
+    const { rows: vendors } = await pool.query(`
       SELECT DISTINCT vendor 
       FROM purchase_orders 
       WHERE vendor IS NOT NULL AND vendor != ''
@@ -146,7 +152,7 @@ router.get('/', async (req, res) => {
     `);
 
     // Get unique statuses for filter options
-    const [statuses] = await pool.query(`
+    const { rows: statuses } = await pool.query(`
       SELECT DISTINCT status 
       FROM purchase_orders 
       WHERE status IS NOT NULL
@@ -169,12 +175,12 @@ router.get('/', async (req, res) => {
 
     // Parse summary metrics
     const parsedSummary = {
-      order_count: Number(summary[0].order_count) || 0,
-      total_ordered: Number(summary[0].total_ordered) || 0,
-      total_received: Number(summary[0].total_received) || 0,
-      fulfillment_rate: Number(summary[0].fulfillment_rate) || 0,
-      total_value: Number(summary[0].total_value) || 0,
-      avg_cost: Number(summary[0].avg_cost) || 0
+      order_count: Number(summary.order_count) || 0,
+      total_ordered: Number(summary.total_ordered) || 0,
+      total_received: Number(summary.total_received) || 0,
+      fulfillment_rate: Number(summary.fulfillment_rate) || 0,
+      total_value: Number(summary.total_value) || 0,
+      avg_cost: Number(summary.avg_cost) || 0
     };
 
     res.json({
@@ -202,7 +208,7 @@ router.get('/vendor-metrics', async (req, res) => {
   try {
     const pool = req.app.locals.pool;
     
-    const [metrics] = await pool.query(`
+    const { rows: metrics } = await pool.query(`
       WITH delivery_metrics AS (
         SELECT 
           vendor,
@@ -213,7 +219,7 @@ router.get('/vendor-metrics', async (req, res) => {
           CASE 
             WHEN status >= ${STATUS.RECEIVING_STARTED} AND receiving_status >= ${RECEIVING_STATUS.PARTIAL_RECEIVED}
             AND received_date IS NOT NULL AND date IS NOT NULL
-            THEN DATEDIFF(received_date, date)
+            THEN (received_date - date)::integer
             ELSE NULL
           END as delivery_days
         FROM purchase_orders
@@ -226,18 +232,18 @@ router.get('/vendor-metrics', async (req, res) => {
         SUM(ordered) as total_ordered,
         SUM(received) as total_received,
         ROUND(
-          SUM(received) / NULLIF(SUM(ordered), 0), 3
+          (SUM(received)::numeric / NULLIF(SUM(ordered), 0)), 3
         ) as fulfillment_rate,
-        CAST(ROUND(
-          SUM(ordered * cost_price) / NULLIF(SUM(ordered), 0), 2
-        ) AS DECIMAL(15,3)) as avg_unit_cost,
-        CAST(SUM(ordered * cost_price) AS DECIMAL(15,3)) as total_spend,
         ROUND(
-          AVG(NULLIF(delivery_days, 0)), 1
+          (SUM(ordered * cost_price)::numeric / NULLIF(SUM(ordered), 0)), 2
+        ) as avg_unit_cost,
+        ROUND(SUM(ordered * cost_price)::numeric, 3) as total_spend,
+        ROUND(
+          AVG(NULLIF(delivery_days, 0))::numeric, 1
         ) as avg_delivery_days
       FROM delivery_metrics
       GROUP BY vendor
-      HAVING total_orders > 0
+      HAVING COUNT(DISTINCT po_id) > 0
       ORDER BY total_spend DESC
     `);
 
@@ -251,7 +257,7 @@ router.get('/vendor-metrics', async (req, res) => {
       fulfillment_rate: Number(vendor.fulfillment_rate) || 0,
       avg_unit_cost: Number(vendor.avg_unit_cost) || 0,
       total_spend: Number(vendor.total_spend) || 0,
-      avg_delivery_days: vendor.avg_delivery_days === null ? null : Number(vendor.avg_delivery_days)
+      avg_delivery_days: Number(vendor.avg_delivery_days) || 0
     }));
 
     res.json(parsedMetrics);

inventory-server/src/routes/templates.js

@@ -0,0 +1,283 @@
+const express = require('express');
+const { getPool } = require('../utils/db');
+const dotenv = require('dotenv');
+const path = require('path');
+
+dotenv.config({ path: path.join(__dirname, "../../.env") });
+
+const router = express.Router();
+
+// Get all templates
+router.get('/', async (req, res) => {
+  try {
+    const pool = getPool();
+    if (!pool) {
+      throw new Error('Database pool not initialized');
+    }
+    
+    const result = await pool.query(`
+      SELECT * FROM templates 
+      ORDER BY company ASC, product_type ASC
+    `);
+    res.json(result.rows);
+  } catch (error) {
+    console.error('Error fetching templates:', error);
+    res.status(500).json({ 
+      error: 'Failed to fetch templates',
+      details: error instanceof Error ? error.message : 'Unknown error'
+    });
+  }
+});
+
+// Get template by company and product type
+router.get('/:company/:productType', async (req, res) => {
+  try {
+    const { company, productType } = req.params;
+    const pool = getPool();
+    if (!pool) {
+      throw new Error('Database pool not initialized');
+    }
+    
+    const result = await pool.query(`
+      SELECT * FROM templates 
+      WHERE company = $1 AND product_type = $2
+    `, [company, productType]);
+
+    if (result.rows.length === 0) {
+      return res.status(404).json({ error: 'Template not found' });
+    }
+
+    res.json(result.rows[0]);
+  } catch (error) {
+    console.error('Error fetching template:', error);
+    res.status(500).json({ 
+      error: 'Failed to fetch template',
+      details: error instanceof Error ? error.message : 'Unknown error'
+    });
+  }
+});
+
+// Create new template
+router.post('/', async (req, res) => {
+  try {
+    const {
+      company,
+      product_type,
+      supplier,
+      msrp,
+      cost_each,
+      qty_per_unit,
+      case_qty,
+      hts_code,
+      description,
+      weight,
+      length,
+      width,
+      height,
+      tax_cat,
+      size_cat,
+      categories,
+      ship_restrictions
+    } = req.body;
+
+    // Validate required fields
+    if (!company || !product_type) {
+      return res.status(400).json({ error: 'Company and Product Type are required' });
+    }
+
+    const pool = getPool();
+    if (!pool) {
+      throw new Error('Database pool not initialized');
+    }
+    
+    const result = await pool.query(`
+      INSERT INTO templates (
+        company,
+        product_type,
+        supplier,
+        msrp,
+        cost_each,
+        qty_per_unit,
+        case_qty,
+        hts_code,
+        description,
+        weight,
+        length,
+        width,
+        height,
+        tax_cat,
+        size_cat,
+        categories,
+        ship_restrictions
+      ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17)
+      RETURNING *
+    `, [
+      company,
+      product_type,
+      supplier,
+      msrp,
+      cost_each,
+      qty_per_unit,
+      case_qty,
+      hts_code,
+      description,
+      weight,
+      length,
+      width,
+      height,
+      tax_cat,
+      size_cat,
+      categories,
+      ship_restrictions
+    ]);
+
+    res.status(201).json(result.rows[0]);
+  } catch (error) {
+    console.error('Error creating template:', error);
+    // Check for unique constraint violation
+    if (error instanceof Error && error.message.includes('unique constraint')) {
+      return res.status(409).json({ 
+        error: 'Template already exists for this company and product type',
+        details: error.message
+      });
+    }
+    res.status(500).json({ 
+      error: 'Failed to create template',
+      details: error instanceof Error ? error.message : 'Unknown error'
+    });
+  }
+});
+
+// Update template
+router.put('/:id', async (req, res) => {
+  try {
+    const { id } = req.params;
+    const {
+      company,
+      product_type,
+      supplier,
+      msrp,
+      cost_each,
+      qty_per_unit,
+      case_qty,
+      hts_code,
+      description,
+      weight,
+      length,
+      width,
+      height,
+      tax_cat,
+      size_cat,
+      categories,
+      ship_restrictions
+    } = req.body;
+
+    // Validate required fields
+    if (!company || !product_type) {
+      return res.status(400).json({ error: 'Company and Product Type are required' });
+    }
+
+    const pool = getPool();
+    if (!pool) {
+      throw new Error('Database pool not initialized');
+    }
+    
+    const result = await pool.query(`
+      UPDATE templates 
+      SET 
+        company = $1,
+        product_type = $2,
+        supplier = $3,
+        msrp = $4,
+        cost_each = $5,
+        qty_per_unit = $6,
+        case_qty = $7,
+        hts_code = $8,
+        description = $9,
+        weight = $10,
+        length = $11,
+        width = $12,
+        height = $13,
+        tax_cat = $14,
+        size_cat = $15,
+        categories = $16,
+        ship_restrictions = $17
+      WHERE id = $18
+      RETURNING *
+    `, [
+      company,
+      product_type,
+      supplier,
+      msrp,
+      cost_each,
+      qty_per_unit,
+      case_qty,
+      hts_code,
+      description,
+      weight,
+      length,
+      width,
+      height,
+      tax_cat,
+      size_cat,
+      categories,
+      ship_restrictions,
+      id
+    ]);
+
+    if (result.rows.length === 0) {
+      return res.status(404).json({ error: 'Template not found' });
+    }
+
+    res.json(result.rows[0]);
+  } catch (error) {
+    console.error('Error updating template:', error);
+    // Check for unique constraint violation
+    if (error instanceof Error && error.message.includes('unique constraint')) {
+      return res.status(409).json({ 
+        error: 'Template already exists for this company and product type',
+        details: error.message
+      });
+    }
+    res.status(500).json({ 
+      error: 'Failed to update template',
+      details: error instanceof Error ? error.message : 'Unknown error'
+    });
+  }
+});
+
+// Delete template
+router.delete('/:id', async (req, res) => {
+  try {
+    const { id } = req.params;
+    const pool = getPool();
+    if (!pool) {
+      throw new Error('Database pool not initialized');
+    }
+    
+    const result = await pool.query('DELETE FROM templates WHERE id = $1 RETURNING *', [id]);
+
+    if (result.rows.length === 0) {
+      return res.status(404).json({ error: 'Template not found' });
+    }
+
+    res.json({ message: 'Template deleted successfully' });
+  } catch (error) {
+    console.error('Error deleting template:', error);
+    res.status(500).json({ 
+      error: 'Failed to delete template',
+      details: error instanceof Error ? error.message : 'Unknown error'
+    });
+  }
+});
+
+// Error handling middleware
+router.use((err, req, res, next) => {
+  console.error('Template route error:', err);
+  res.status(500).json({ 
+    error: 'Internal server error',
+    details: err.message
+  });
+});
+
+module.exports = router; 

inventory-server/src/routes/vendors.js

@@ -6,7 +6,7 @@ router.get('/', async (req, res) => {
   const pool = req.app.locals.pool;
   try {
     // Get all vendors with metrics
-    const [vendors] = await pool.query(`
+    const { rows: vendors } = await pool.query(`
       SELECT DISTINCT
         p.vendor as name,
         COALESCE(vm.active_products, 0) as active_products,
@@ -26,16 +26,16 @@ router.get('/', async (req, res) => {
 
     // Get cost metrics for all vendors
     const vendorNames = vendors.map(v => v.name);
-    const [costMetrics] = await pool.query(`
+    const { rows: costMetrics } = await pool.query(`
       SELECT 
         vendor,
-        CAST(ROUND(SUM(ordered * cost_price) / NULLIF(SUM(ordered), 0), 2) AS DECIMAL(15,3)) as avg_unit_cost,
-        CAST(SUM(ordered * cost_price) AS DECIMAL(15,3)) as total_spend
+        ROUND((SUM(ordered * cost_price)::numeric / NULLIF(SUM(ordered), 0)), 2) as avg_unit_cost,
+        ROUND(SUM(ordered * cost_price)::numeric, 3) as total_spend
       FROM purchase_orders
       WHERE status = 'closed'
         AND cost_price IS NOT NULL
         AND ordered > 0
-        AND vendor IN (?)
+        AND vendor = ANY($1)
       GROUP BY vendor
     `, [vendorNames]);
 
@@ -49,26 +49,26 @@ router.get('/', async (req, res) => {
     }, {});
 
     // Get overall stats
-    const [stats] = await pool.query(`
+    const { rows: [stats] } = await pool.query(`
       SELECT 
         COUNT(DISTINCT p.vendor) as totalVendors,
         COUNT(DISTINCT CASE 
           WHEN COALESCE(vm.total_orders, 0) > 0 AND COALESCE(vm.order_fill_rate, 0) >= 75 
           THEN p.vendor 
         END) as activeVendors,
-        COALESCE(ROUND(AVG(NULLIF(vm.avg_lead_time_days, 0)), 1), 0) as avgLeadTime,
-        COALESCE(ROUND(AVG(NULLIF(vm.order_fill_rate, 0)), 1), 0) as avgFillRate,
-        COALESCE(ROUND(AVG(NULLIF(vm.on_time_delivery_rate, 0)), 1), 0) as avgOnTimeDelivery
+        COALESCE(ROUND(AVG(NULLIF(vm.avg_lead_time_days, 0))::numeric, 1), 0) as avgLeadTime,
+        COALESCE(ROUND(AVG(NULLIF(vm.order_fill_rate, 0))::numeric, 1), 0) as avgFillRate,
+        COALESCE(ROUND(AVG(NULLIF(vm.on_time_delivery_rate, 0))::numeric, 1), 0) as avgOnTimeDelivery
       FROM products p
       LEFT JOIN vendor_metrics vm ON p.vendor = vm.vendor
       WHERE p.vendor IS NOT NULL AND p.vendor != ''
     `);
 
     // Get overall cost metrics
-    const [overallCostMetrics] = await pool.query(`
+    const { rows: [overallCostMetrics] } = await pool.query(`
       SELECT 
-        CAST(ROUND(SUM(ordered * cost_price) / NULLIF(SUM(ordered), 0), 2) AS DECIMAL(15,3)) as avg_unit_cost,
-        CAST(SUM(ordered * cost_price) AS DECIMAL(15,3)) as total_spend
+        ROUND((SUM(ordered * cost_price)::numeric / NULLIF(SUM(ordered), 0)), 2) as avg_unit_cost,
+        ROUND(SUM(ordered * cost_price)::numeric, 3) as total_spend
       FROM purchase_orders
       WHERE status = 'closed'
         AND cost_price IS NOT NULL
@@ -90,13 +90,13 @@ router.get('/', async (req, res) => {
         total_spend: parseFloat(costMetricsMap[vendor.name]?.total_spend || 0)
       })),
       stats: {
-        totalVendors: parseInt(stats[0].totalVendors),
-        activeVendors: parseInt(stats[0].activeVendors),
-        avgLeadTime: parseFloat(stats[0].avgLeadTime),
-        avgFillRate: parseFloat(stats[0].avgFillRate),
-        avgOnTimeDelivery: parseFloat(stats[0].avgOnTimeDelivery),
-        avgUnitCost: parseFloat(overallCostMetrics[0].avg_unit_cost),
-        totalSpend: parseFloat(overallCostMetrics[0].total_spend)
+        totalVendors: parseInt(stats.totalvendors),
+        activeVendors: parseInt(stats.activevendors),
+        avgLeadTime: parseFloat(stats.avgleadtime),
+        avgFillRate: parseFloat(stats.avgfillrate),
+        avgOnTimeDelivery: parseFloat(stats.avgontimedelivery),
+        avgUnitCost: parseFloat(overallCostMetrics.avg_unit_cost),
+        totalSpend: parseFloat(overallCostMetrics.total_spend)
       }
     });
   } catch (error) {

inventory-server/src/server.js

@@ -3,7 +3,6 @@ const cors = require('cors');
 const { spawn } = require('child_process');
 const path = require('path');
 const fs = require('fs');
-const mysql = require('mysql2/promise');
 const { corsMiddleware, corsErrorHandler } = require('./middleware/cors');
 const { initPool } = require('./utils/db');
 const productsRouter = require('./routes/products');
@@ -16,11 +15,12 @@ const configRouter = require('./routes/config');
 const metricsRouter = require('./routes/metrics');
 const vendorsRouter = require('./routes/vendors');
 const categoriesRouter = require('./routes/categories');
-const testConnectionRouter = require('./routes/test-connection');
+const importRouter = require('./routes/import');
+const aiValidationRouter = require('./routes/ai-validation');
+const templatesRouter = require('./routes/templates');
 
 // Get the absolute path to the .env file
-const envPath = path.resolve(process.cwd(), '.env');
-console.log('Current working directory:', process.cwd());
+const envPath = '/var/www/html/inventory/.env';
 console.log('Looking for .env file at:', envPath);
 console.log('.env file exists:', fs.existsSync(envPath));
 
@@ -33,6 +33,9 @@ try {
     DB_HOST: process.env.DB_HOST || 'not set',
     DB_USER: process.env.DB_USER || 'not set',
     DB_NAME: process.env.DB_NAME || 'not set',
+    DB_PASSWORD: process.env.DB_PASSWORD ? '[password set]' : 'not set',
+    DB_PORT: process.env.DB_PORT || 'not set',
+    DB_SSL: process.env.DB_SSL || 'not set'
   });
 } catch (error) {
   console.error('Error loading .env file:', error);
@@ -62,61 +65,78 @@ app.use((req, res, next) => {
 app.use(corsMiddleware);
 
 // Body parser middleware
-app.use(express.json());
-app.use(express.urlencoded({ extended: true }));
+app.use(express.json({ limit: '10mb' }));
+app.use(express.urlencoded({ extended: true, limit: '10mb' }));
 
-// Initialize database pool
-const pool = initPool({
-  host: process.env.DB_HOST,
-  user: process.env.DB_USER,
-  password: process.env.DB_PASSWORD,
-  database: process.env.DB_NAME,
-  waitForConnections: true,
-  connectionLimit: process.env.NODE_ENV === 'production' ? 20 : 10,
-  queueLimit: 0,
-  enableKeepAlive: true,
-  keepAliveInitialDelay: 0
-});
+// Initialize database pool and start server
+async function startServer() {
+  try {
+    // Initialize database pool
+    const pool = await initPool({
+      host: process.env.DB_HOST,
+      user: process.env.DB_USER,
+      password: process.env.DB_PASSWORD,
+      database: process.env.DB_NAME,
+      port: process.env.DB_PORT || 5432,
+      max: process.env.NODE_ENV === 'production' ? 20 : 10,
+      idleTimeoutMillis: 30000,
+      connectionTimeoutMillis: 2000,
+      ssl: process.env.DB_SSL === 'true' ? {
+        rejectUnauthorized: false
+      } : false
+    });
 
-// Make pool available to routes
-app.locals.pool = pool;
+    // Make pool available to routes
+    app.locals.pool = pool;
 
-// Routes
-app.use('/api/products', productsRouter);
-app.use('/api/dashboard', dashboardRouter);
-app.use('/api/orders', ordersRouter);
-app.use('/api/csv', csvRouter);
-app.use('/api/analytics', analyticsRouter);
-app.use('/api/purchase-orders', purchaseOrdersRouter);
-app.use('/api/config', configRouter);
-app.use('/api/metrics', metricsRouter);
-app.use('/api/vendors', vendorsRouter);
-app.use('/api/categories', categoriesRouter);
-app.use('/api', testConnectionRouter);
+    // Set up routes after pool is initialized
+    app.use('/api/products', productsRouter);
+    app.use('/api/dashboard', dashboardRouter);
+    app.use('/api/orders', ordersRouter);
+    app.use('/api/csv', csvRouter);
+    app.use('/api/analytics', analyticsRouter);
+    app.use('/api/purchase-orders', purchaseOrdersRouter);
+    app.use('/api/config', configRouter);
+    app.use('/api/metrics', metricsRouter);
+    app.use('/api/vendors', vendorsRouter);
+    app.use('/api/categories', categoriesRouter);
+    app.use('/api/import', importRouter);
+    app.use('/api/ai-validation', aiValidationRouter);
+    app.use('/api/templates', templatesRouter);
 
-// Basic health check route
-app.get('/health', (req, res) => {
-  res.json({ 
-    status: 'ok',
-    timestamp: new Date().toISOString(),
-    environment: process.env.NODE_ENV
-  });
-});
+    // Basic health check route
+    app.get('/health', (req, res) => {
+      res.json({ 
+        status: 'ok',
+        timestamp: new Date().toISOString(),
+        environment: process.env.NODE_ENV
+      });
+    });
 
-// CORS error handler - must be before other error handlers
-app.use(corsErrorHandler);
+    // CORS error handler - must be before other error handlers
+    app.use(corsErrorHandler);
 
-// Error handling middleware - MUST be after routes and CORS error handler
-app.use((err, req, res, next) => {
-  console.error(`[${new Date().toISOString()}] Error:`, err);
-  
-  // Send detailed error in development, generic in production
-  const error = process.env.NODE_ENV === 'production'
-    ? 'An internal server error occurred'
-    : err.message || err;
-    
-  res.status(err.status || 500).json({ error });
-});
+    // Error handling middleware - MUST be after routes and CORS error handler
+    app.use((err, req, res, next) => {
+      console.error(`[${new Date().toISOString()}] Error:`, err);
+      
+      // Send detailed error in development, generic in production
+      const error = process.env.NODE_ENV === 'production'
+        ? 'An internal server error occurred'
+        : err.message || err;
+        
+      res.status(err.status || 500).json({ error });
+    });
+
+    const PORT = process.env.PORT || 3000;
+    app.listen(PORT, () => {
+      console.log(`[Server] Running in ${process.env.NODE_ENV || 'development'} mode on port ${PORT}`);
+    });
+  } catch (error) {
+    console.error('Failed to start server:', error);
+    process.exit(1);
+  }
+}
 
 // Handle uncaught exceptions
 process.on('uncaughtException', (err) => {
@@ -128,17 +148,6 @@ process.on('unhandledRejection', (reason, promise) => {
   console.error(`[${new Date().toISOString()}] Unhandled Rejection at:`, promise, 'reason:', reason);
 });
 
-// Test database connection
-pool.getConnection()
-  .then(connection => {
-    console.log('[Database] Connected successfully');
-    connection.release();
-  })
-  .catch(err => {
-    console.error('[Database] Error connecting:', err);
-    process.exit(1);
-  });
-
 // Initialize client sets for SSE
 const importClients = new Set();
 const updateClients = new Set();
@@ -189,62 +198,5 @@ const setupSSE = (req, res) => {
   }
 };
 
-// Update the status endpoint to include reset-metrics
-app.get('/csv/status', (req, res) => {
-    res.json({
-        active: !!currentOperation,
-        type: currentOperation?.type || null,
-        progress: currentOperation ? {
-            status: currentOperation.status,
-            operation: currentOperation.operation,
-            current: currentOperation.current,
-            total: currentOperation.total,
-            percentage: currentOperation.percentage
-        } : null
-    });
-});
-
-// Update progress endpoint mapping
-app.get('/csv/:type/progress', (req, res) => {
-    const { type } = req.params;
-    if (!['import', 'update', 'reset', 'reset-metrics'].includes(type)) {
-        res.status(400).json({ error: 'Invalid operation type' });
-        return;
-    }
-
-    setupSSE(req, res);
-});
-
-// Update the cancel endpoint to handle reset-metrics
-app.post('/csv/cancel', (req, res) => {
-    const { operation } = req.query;
-    
-    if (!currentOperation) {
-        res.status(400).json({ error: 'No operation in progress' });
-        return;
-    }
-
-    if (operation && operation.toLowerCase() !== currentOperation.type) {
-        res.status(400).json({ error: 'Operation type mismatch' });
-        return;
-    }
-
-    try {
-        // Handle cancellation based on operation type
-        if (currentOperation.type === 'reset-metrics') {
-            // Reset metrics doesn't need special cleanup
-            currentOperation = null;
-            res.json({ message: 'Reset metrics cancelled' });
-        } else {
-            // ... existing cancellation logic for other operations ...
-        }
-    } catch (error) {
-        console.error('Error during cancellation:', error);
-        res.status(500).json({ error: 'Failed to cancel operation' });
-    }
-});
-
-const PORT = process.env.PORT || 3000;
-app.listen(PORT, () => {
-  console.log(`[Server] Running in ${process.env.NODE_ENV || 'development'} mode on port ${PORT}`);
-}); 
+// Start the server
+startServer(); 

inventory-server/src/utils/db.js

@@ -1,17 +1,54 @@
-const mysql = require('mysql2/promise');
+const { Pool, Client } = require('pg');
+const { Client: SSHClient } = require('ssh2');
 
 let pool;
 
 function initPool(config) {
-  pool = mysql.createPool(config);
-  return pool;
+  // Log config without sensitive data
+  const safeConfig = {
+    host: config.host || process.env.DB_HOST,
+    user: config.user || process.env.DB_USER,
+    database: config.database || process.env.DB_NAME,
+    port: config.port || process.env.DB_PORT || 5432,
+    max: config.max || 10,
+    idleTimeoutMillis: config.idleTimeoutMillis || 30000,
+    connectionTimeoutMillis: config.connectionTimeoutMillis || 2000,
+    ssl: config.ssl || false,
+    password: (config.password || process.env.DB_PASSWORD) ? '[password set]' : '[no password]'
+  };
+  console.log('[Database] Initializing pool with config:', safeConfig);
+  
+  // Create the pool with the configuration
+  pool = new Pool({
+    host: config.host || process.env.DB_HOST,
+    user: config.user || process.env.DB_USER,
+    password: config.password || process.env.DB_PASSWORD,
+    database: config.database || process.env.DB_NAME,
+    port: config.port || process.env.DB_PORT || 5432,
+    max: config.max || 10,
+    idleTimeoutMillis: config.idleTimeoutMillis || 30000,
+    connectionTimeoutMillis: config.connectionTimeoutMillis || 2000,
+    ssl: config.ssl || false
+  });
+
+  // Test the pool connection
+  return pool.connect()
+    .then(client => {
+      console.log('[Database] Pool connection successful');
+      client.release();
+      return pool;
+    })
+    .catch(err => {
+      console.error('[Database] Connection failed:', err);
+      throw err;
+    });
 }
 
 async function getConnection() {
   if (!pool) {
     throw new Error('Database pool not initialized');
   }
-  return pool.getConnection();
+  return pool.connect();
 }
 
 module.exports = {

inventory/package.json

@@ -10,58 +10,90 @@
     "preview": "vite preview"
   },
   "dependencies": {
+    "@chakra-ui/button": "^2.1.0",
+    "@chakra-ui/checkbox": "^2.3.2",
+    "@chakra-ui/form-control": "^2.2.0",
+    "@chakra-ui/hooks": "^2.4.3",
+    "@chakra-ui/icons": "^2.2.4",
+    "@chakra-ui/input": "^2.1.2",
+    "@chakra-ui/layout": "^2.3.1",
+    "@chakra-ui/modal": "^2.3.1",
+    "@chakra-ui/popper": "^3.1.0",
+    "@chakra-ui/react": "^2.8.1",
+    "@chakra-ui/select": "^2.1.2",
+    "@chakra-ui/system": "^2.6.2",
+    "@chakra-ui/theme": "^3.4.7",
+    "@chakra-ui/theme-tools": "^2.2.7",
+    "@chakra-ui/utils": "^2.2.3",
     "@dnd-kit/core": "^6.3.1",
     "@dnd-kit/sortable": "^10.0.0",
     "@dnd-kit/utilities": "^3.2.2",
+    "@emotion/react": "^11.14.0",
+    "@emotion/styled": "^11.14.0",
     "@radix-ui/react-accordion": "^1.2.2",
     "@radix-ui/react-alert-dialog": "^1.1.4",
     "@radix-ui/react-avatar": "^1.1.2",
+    "@radix-ui/react-checkbox": "^1.1.4",
     "@radix-ui/react-collapsible": "^1.1.2",
-    "@radix-ui/react-dialog": "^1.1.4",
+    "@radix-ui/react-dialog": "^1.1.6",
     "@radix-ui/react-dropdown-menu": "^2.1.4",
     "@radix-ui/react-label": "^2.1.1",
-    "@radix-ui/react-popover": "^1.1.4",
+    "@radix-ui/react-popover": "^1.1.6",
     "@radix-ui/react-progress": "^1.1.1",
+    "@radix-ui/react-radio-group": "^1.2.3",
     "@radix-ui/react-scroll-area": "^1.2.2",
     "@radix-ui/react-select": "^2.1.4",
     "@radix-ui/react-separator": "^1.1.1",
     "@radix-ui/react-slot": "^1.1.1",
     "@radix-ui/react-switch": "^1.1.2",
     "@radix-ui/react-tabs": "^1.1.2",
+    "@radix-ui/react-toast": "^1.2.6",
     "@radix-ui/react-toggle": "^1.1.1",
     "@radix-ui/react-toggle-group": "^1.1.1",
     "@radix-ui/react-tooltip": "^1.1.6",
     "@shadcn/ui": "^0.0.4",
     "@tabler/icons-react": "^3.28.1",
-    "@tanstack/react-query": "^5.63.0",
+    "@tanstack/react-query": "^5.66.7",
     "@tanstack/react-table": "^8.20.6",
     "@tanstack/react-virtual": "^3.11.2",
     "@tanstack/virtual-core": "^3.11.2",
+    "@types/js-levenshtein": "^1.1.3",
+    "@types/uuid": "^10.0.0",
+    "chakra-react-select": "^4.7.5",
+    "chakra-ui-steps": "^2.0.4",
     "chart.js": "^4.4.7",
     "chartjs-adapter-date-fns": "^3.0.0",
     "class-variance-authority": "^0.7.1",
     "clsx": "^2.1.1",
     "cmdk": "^1.0.0",
     "date-fns": "^3.6.0",
+    "framer-motion": "^12.4.4",
+    "js-levenshtein": "^1.1.6",
+    "lodash": "^4.17.21",
     "lucide-react": "^0.469.0",
     "motion": "^11.18.0",
     "next-themes": "^0.4.4",
     "react": "^18.3.1",
     "react-chartjs-2": "^5.3.0",
+    "react-data-grid": "^7.0.0-beta.13",
     "react-day-picker": "^8.10.1",
     "react-dom": "^18.3.1",
+    "react-dropzone": "^14.3.5",
+    "react-icons": "^5.4.0",
     "react-router-dom": "^7.1.1",
     "recharts": "^2.15.0",
     "sonner": "^1.7.1",
     "tailwind-merge": "^2.6.0",
     "tailwindcss-animate": "^1.0.7",
     "tanstack": "^1.0.0",
-    "vaul": "^1.1.2"
+    "uuid": "^11.0.5",
+    "vaul": "^1.1.2",
+    "xlsx": "^0.18.5"
   },
   "devDependencies": {
     "@eslint/js": "^9.17.0",
     "@types/fs-extra": "^11.0.4",
-    "@types/lodash": "^4.17.14",
+    "@types/lodash": "^4.17.15",
     "@types/node": "^22.10.5",
     "@types/react": "^18.3.18",
     "@types/react-dom": "^18.3.5",

inventory/src/App.tsx

@@ -15,6 +15,9 @@ import { RequireAuth } from './components/auth/RequireAuth';
 import Forecasting from "@/pages/Forecasting";
 import { Vendors } from '@/pages/Vendors';
 import { Categories } from '@/pages/Categories';
+import { Import } from '@/pages/Import';
+import { ChakraProvider } from '@chakra-ui/react';
+import { AiValidationDebug } from "@/pages/AiValidationDebug"
 
 const queryClient = new QueryClient();
 
@@ -50,26 +53,30 @@ function App() {
 
   return (
     <QueryClientProvider client={queryClient}>
-      <Toaster richColors position="top-center" />
-      <Routes>
-        <Route path="/login" element={<Login />} />
-        <Route element={
-          <RequireAuth>
-            <MainLayout />
-          </RequireAuth>
-        }>
-          <Route path="/" element={<Dashboard />} />
-          <Route path="/products" element={<Products />} />
-          <Route path="/categories" element={<Categories />} />
-          <Route path="/vendors" element={<Vendors />} />
-          <Route path="/orders" element={<Orders />} />
-          <Route path="/purchase-orders" element={<PurchaseOrders />} />
-          <Route path="/analytics" element={<Analytics />} />
-          <Route path="/settings" element={<Settings />} />
-          <Route path="/forecasting" element={<Forecasting />} />
-          <Route path="*" element={<Navigate to="/" replace />} />
-        </Route>
-      </Routes>
+      <ChakraProvider>
+        <Toaster richColors position="top-center" />
+        <Routes>
+          <Route path="/login" element={<Login />} />
+          <Route element={
+            <RequireAuth>
+              <MainLayout />
+            </RequireAuth>
+          }>
+            <Route path="/" element={<Dashboard />} />
+            <Route path="/products" element={<Products />} />
+            <Route path="/import" element={<Import />} />
+            <Route path="/categories" element={<Categories />} />
+            <Route path="/vendors" element={<Vendors />} />
+            <Route path="/orders" element={<Orders />} />
+            <Route path="/purchase-orders" element={<PurchaseOrders />} />
+            <Route path="/analytics" element={<Analytics />} />
+            <Route path="/settings" element={<Settings />} />
+            <Route path="/forecasting" element={<Forecasting />} />
+            <Route path="/ai-validation/debug" element={<AiValidationDebug />} />
+            <Route path="*" element={<Navigate to="/" replace />} />
+          </Route>
+        </Routes>
+      </ChakraProvider>
     </QueryClientProvider>
   );
 }

inventory/src/components/layout/AppSidebar.tsx

@@ -8,6 +8,7 @@ import {
   LogOut,
   Users,
   Tags,
+  FileSpreadsheet,
 } from "lucide-react";
 import { IconCrystalBall } from "@tabler/icons-react";
 import {
@@ -35,6 +36,11 @@ const items = [
     icon: Package,
     url: "/products",
   },
+  {
+    title: "Import",
+    icon: FileSpreadsheet,
+    url: "/import",
+  },
   {
     title: "Forecasting",
     icon: IconCrystalBall,

inventory/src/components/settings/PerformanceMetrics.tsx

@@ -133,6 +133,10 @@ export function PerformanceMetrics() {
     }
   };
 
+  function getCategoryName(_cat_id: number): import("react").ReactNode {
+    throw new Error('Function not implemented.');
+  }
+
   return (
     <div className="max-w-[700px] space-y-4">
       {/* Lead Time Thresholds Card */}
@@ -205,11 +209,11 @@ export function PerformanceMetrics() {
             <Table>
               <TableHeader>
                 <TableRow>
-                  <TableHead>Category</TableHead>
-                  <TableHead>Vendor</TableHead>
-                  <TableHead className="text-right">A Threshold</TableHead>
-                  <TableHead className="text-right">B Threshold</TableHead>
-                  <TableHead className="text-right">Period Days</TableHead>
+                  <TableCell>Category</TableCell>
+                  <TableCell>Vendor</TableCell>
+                  <TableCell className="text-right">A Threshold</TableCell>
+                  <TableCell className="text-right">B Threshold</TableCell>
+                  <TableCell className="text-right">Period Days</TableCell>
                 </TableRow>
               </TableHeader>
               <TableBody>
@@ -242,10 +246,10 @@ export function PerformanceMetrics() {
             <Table>
               <TableHeader>
                 <TableRow>
-                  <TableHead>Category</TableHead>
-                  <TableHead>Vendor</TableHead>
-                  <TableHead className="text-right">Period Days</TableHead>
-                  <TableHead className="text-right">Target Rate</TableHead>
+                  <TableCell>Category</TableCell>
+                  <TableCell>Vendor</TableCell>
+                  <TableCell className="text-right">Period Days</TableCell>
+                  <TableCell className="text-right">Target Rate</TableCell>
                 </TableRow>
               </TableHeader>
               <TableBody>

inventory/src/components/settings/StockManagement.tsx

@@ -5,7 +5,6 @@ import { Input } from "@/components/ui/input";
 import { Label } from "@/components/ui/label";
 import { toast } from "sonner";
 import config from '../../config';
-import { Table, TableBody, TableCell, TableHeader, TableRow } from "@/components/ui/table";
 
 interface StockThreshold {
   id: number;
@@ -244,54 +243,6 @@ export function StockManagement() {
           </div>
         </CardContent>
       </Card>
-
-      <Table>
-        <TableHeader>
-          <TableRow>
-            <TableHead>Category</TableHead>
-            <TableHead>Vendor</TableHead>
-            <TableHead className="text-right">Critical Days</TableHead>
-            <TableHead className="text-right">Reorder Days</TableHead>
-            <TableHead className="text-right">Overstock Days</TableHead>
-            <TableHead className="text-right">Low Stock</TableHead>
-            <TableHead className="text-right">Min Reorder</TableHead>
-          </TableRow>
-        </TableHeader>
-        <TableBody>
-          {stockThresholds.map((threshold) => (
-            <TableRow key={`${threshold.cat_id}-${threshold.vendor}`}>
-              <TableCell>{threshold.cat_id ? getCategoryName(threshold.cat_id) : 'Global'}</TableCell>
-              <TableCell>{threshold.vendor || 'All Vendors'}</TableCell>
-              <TableCell className="text-right">{threshold.critical_days}</TableCell>
-              <TableCell className="text-right">{threshold.reorder_days}</TableCell>
-              <TableCell className="text-right">{threshold.overstock_days}</TableCell>
-              <TableCell className="text-right">{threshold.low_stock_threshold}</TableCell>
-              <TableCell className="text-right">{threshold.min_reorder_quantity}</TableCell>
-            </TableRow>
-          ))}
-        </TableBody>
-      </Table>
-
-      <Table>
-        <TableHeader>
-          <TableRow>
-            <TableHead>Category</TableHead>
-            <TableHead>Vendor</TableHead>
-            <TableHead className="text-right">Coverage Days</TableHead>
-            <TableHead className="text-right">Service Level</TableHead>
-          </TableRow>
-        </TableHeader>
-        <TableBody>
-          {safetyStockConfigs.map((config) => (
-            <TableRow key={`${config.cat_id}-${config.vendor}`}>
-              <TableCell>{config.cat_id ? getCategoryName(config.cat_id) : 'Global'}</TableCell>
-              <TableCell>{config.vendor || 'All Vendors'}</TableCell>
-              <TableCell className="text-right">{config.coverage_days}</TableCell>
-              <TableCell className="text-right">{config.service_level}%</TableCell>
-            </TableRow>
-          ))}
-        </TableBody>
-      </Table>
     </div>
   );
 } 

inventory/src/components/ui/card.tsx

@@ -9,7 +9,7 @@ const Card = React.forwardRef<
   <div
     ref={ref}
     className={cn(
-      "rounded-xl border bg-card text-card-foreground shadow",
+      "rounded-lg border bg-card text-card-foreground shadow-sm",
       className
     )}
     {...props}
@@ -30,22 +30,25 @@ const CardHeader = React.forwardRef<
 CardHeader.displayName = "CardHeader"
 
 const CardTitle = React.forwardRef<
-  HTMLDivElement,
-  React.HTMLAttributes<HTMLDivElement>
+  HTMLParagraphElement,
+  React.HTMLAttributes<HTMLHeadingElement>
 >(({ className, ...props }, ref) => (
-  <div
+  <h3
     ref={ref}
-    className={cn("font-semibold leading-none tracking-tight", className)}
+    className={cn(
+      "text-2xl font-semibold leading-none tracking-tight",
+      className
+    )}
     {...props}
   />
 ))
 CardTitle.displayName = "CardTitle"
 
 const CardDescription = React.forwardRef<
-  HTMLDivElement,
-  React.HTMLAttributes<HTMLDivElement>
+  HTMLParagraphElement,
+  React.HTMLAttributes<HTMLParagraphElement>
 >(({ className, ...props }, ref) => (
-  <div
+  <p
     ref={ref}
     className={cn("text-sm text-muted-foreground", className)}
     {...props}
@@ -73,4 +76,16 @@ const CardFooter = React.forwardRef<
 ))
 CardFooter.displayName = "CardFooter"
 
-export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent }
+const ScrollArea = React.forwardRef<
+  HTMLDivElement,
+  React.HTMLAttributes<HTMLDivElement>
+>(({ className, ...props }, ref) => (
+  <div
+    ref={ref}
+    className={cn("overflow-auto", className)}
+    {...props}
+  />
+))
+ScrollArea.displayName = "ScrollArea"
+
+export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent, ScrollArea }

inventory/src/components/ui/checkbox.tsx

@@ -0,0 +1,28 @@
+import * as React from "react"
+import * as CheckboxPrimitive from "@radix-ui/react-checkbox"
+import { Check } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const Checkbox = React.forwardRef<
+  React.ElementRef<typeof CheckboxPrimitive.Root>,
+  React.ComponentPropsWithoutRef<typeof CheckboxPrimitive.Root>
+>(({ className, ...props }, ref) => (
+  <CheckboxPrimitive.Root
+    ref={ref}
+    className={cn(
+      "peer h-4 w-4 shrink-0 rounded-sm border border-primary shadow focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 data-[state=checked]:bg-primary data-[state=checked]:text-primary-foreground",
+      className
+    )}
+    {...props}
+  >
+    <CheckboxPrimitive.Indicator
+      className={cn("flex items-center justify-center text-current")}
+    >
+      <Check className="h-4 w-4" />
+    </CheckboxPrimitive.Indicator>
+  </CheckboxPrimitive.Root>
+))
+Checkbox.displayName = CheckboxPrimitive.Root.displayName
+
+export { Checkbox }

inventory/src/components/ui/code.tsx

@@ -0,0 +1,23 @@
+import * as React from "react"
+
+import { cn } from "@/lib/utils"
+
+export interface CodeProps extends React.HTMLAttributes<HTMLPreElement> {}
+
+const Code = React.forwardRef<HTMLPreElement, CodeProps>(
+  ({ className, ...props }, ref) => {
+    return (
+      <pre
+        ref={ref}
+        className={cn(
+          "rounded-lg bg-muted px-4 py-4 font-mono text-sm",
+          className
+        )}
+        {...props}
+      />
+    )
+  }
+)
+Code.displayName = "Code"
+
+export { Code } 

inventory/src/components/ui/radio-group.tsx

@@ -0,0 +1,44 @@
+"use client"
+
+import * as React from "react"
+import * as RadioGroupPrimitive from "@radix-ui/react-radio-group"
+import { Circle } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const RadioGroup = React.forwardRef<
+  React.ElementRef<typeof RadioGroupPrimitive.Root>,
+  React.ComponentPropsWithoutRef<typeof RadioGroupPrimitive.Root>
+>(({ className, ...props }, ref) => {
+  return (
+    <RadioGroupPrimitive.Root
+      className={cn("grid gap-2", className)}
+      {...props}
+      ref={ref}
+    />
+  )
+})
+RadioGroup.displayName = RadioGroupPrimitive.Root.displayName
+
+const RadioGroupItem = React.forwardRef<
+  React.ElementRef<typeof RadioGroupPrimitive.Item>,
+  React.ComponentPropsWithoutRef<typeof RadioGroupPrimitive.Item>
+>(({ className, ...props }, ref) => {
+  return (
+    <RadioGroupPrimitive.Item
+      ref={ref}
+      className={cn(
+        "aspect-square h-4 w-4 rounded-full border border-primary text-primary shadow focus:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
+        className
+      )}
+      {...props}
+    >
+      <RadioGroupPrimitive.Indicator className="flex items-center justify-center">
+        <Circle className="h-3.5 w-3.5 fill-primary" />
+      </RadioGroupPrimitive.Indicator>
+    </RadioGroupPrimitive.Item>
+  )
+})
+RadioGroupItem.displayName = RadioGroupPrimitive.Item.displayName
+
+export { RadioGroup, RadioGroupItem }

inventory/src/components/ui/textarea.tsx

@@ -0,0 +1,22 @@
+import * as React from "react"
+
+import { cn } from "@/lib/utils"
+
+const Textarea = React.forwardRef<
+  HTMLTextAreaElement,
+  React.ComponentProps<"textarea">
+>(({ className, ...props }, ref) => {
+  return (
+    <textarea
+      className={cn(
+        "flex min-h-[60px] w-full rounded-md border border-input bg-transparent px-3 py-2 text-base shadow-sm placeholder:text-muted-foreground focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 md:text-sm",
+        className
+      )}
+      ref={ref}
+      {...props}
+    />
+  )
+})
+Textarea.displayName = "Textarea"
+
+export { Textarea }

inventory/src/components/ui/toast.tsx

@@ -0,0 +1,127 @@
+import * as React from "react"
+import * as ToastPrimitives from "@radix-ui/react-toast"
+import { cva, type VariantProps } from "class-variance-authority"
+import { X } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const ToastProvider = ToastPrimitives.Provider
+
+const ToastViewport = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Viewport>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Viewport>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Viewport
+    ref={ref}
+    className={cn(
+      "fixed top-0 z-[100] flex max-h-screen w-full flex-col-reverse p-4 sm:bottom-0 sm:right-0 sm:top-auto sm:flex-col md:max-w-[420px]",
+      className
+    )}
+    {...props}
+  />
+))
+ToastViewport.displayName = ToastPrimitives.Viewport.displayName
+
+const toastVariants = cva(
+  "group pointer-events-auto relative flex w-full items-center justify-between space-x-2 overflow-hidden rounded-md border p-4 pr-6 shadow-lg transition-all data-[swipe=cancel]:translate-x-0 data-[swipe=end]:translate-x-[var(--radix-toast-swipe-end-x)] data-[swipe=move]:translate-x-[var(--radix-toast-swipe-move-x)] data-[swipe=move]:transition-none data-[state=open]:animate-in data-[state=closed]:animate-out data-[swipe=end]:animate-out data-[state=closed]:fade-out-80 data-[state=closed]:slide-out-to-right-full data-[state=open]:slide-in-from-top-full data-[state=open]:sm:slide-in-from-bottom-full",
+  {
+    variants: {
+      variant: {
+        default: "border bg-background text-foreground",
+        destructive:
+          "destructive group border-destructive bg-destructive text-destructive-foreground",
+      },
+    },
+    defaultVariants: {
+      variant: "default",
+    },
+  }
+)
+
+const Toast = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Root>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Root> &
+    VariantProps<typeof toastVariants>
+>(({ className, variant, ...props }, ref) => {
+  return (
+    <ToastPrimitives.Root
+      ref={ref}
+      className={cn(toastVariants({ variant }), className)}
+      {...props}
+    />
+  )
+})
+Toast.displayName = ToastPrimitives.Root.displayName
+
+const ToastAction = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Action>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Action>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Action
+    ref={ref}
+    className={cn(
+      "inline-flex h-8 shrink-0 items-center justify-center rounded-md border bg-transparent px-3 text-sm font-medium transition-colors hover:bg-secondary focus:outline-none focus:ring-1 focus:ring-ring disabled:pointer-events-none disabled:opacity-50 group-[.destructive]:border-muted/40 group-[.destructive]:hover:border-destructive/30 group-[.destructive]:hover:bg-destructive group-[.destructive]:hover:text-destructive-foreground group-[.destructive]:focus:ring-destructive",
+      className
+    )}
+    {...props}
+  />
+))
+ToastAction.displayName = ToastPrimitives.Action.displayName
+
+const ToastClose = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Close>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Close>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Close
+    ref={ref}
+    className={cn(
+      "absolute right-1 top-1 rounded-md p-1 text-foreground/50 opacity-0 transition-opacity hover:text-foreground focus:opacity-100 focus:outline-none focus:ring-1 group-hover:opacity-100 group-[.destructive]:text-red-300 group-[.destructive]:hover:text-red-50 group-[.destructive]:focus:ring-red-400 group-[.destructive]:focus:ring-offset-red-600",
+      className
+    )}
+    toast-close=""
+    {...props}
+  >
+    <X className="h-4 w-4" />
+  </ToastPrimitives.Close>
+))
+ToastClose.displayName = ToastPrimitives.Close.displayName
+
+const ToastTitle = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Title>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Title>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Title
+    ref={ref}
+    className={cn("text-sm font-semibold [&+div]:text-xs", className)}
+    {...props}
+  />
+))
+ToastTitle.displayName = ToastPrimitives.Title.displayName
+
+const ToastDescription = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Description>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Description>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Description
+    ref={ref}
+    className={cn("text-sm opacity-90", className)}
+    {...props}
+  />
+))
+ToastDescription.displayName = ToastPrimitives.Description.displayName
+
+type ToastProps = React.ComponentPropsWithoutRef<typeof Toast>
+
+type ToastActionElement = React.ReactElement<typeof ToastAction>
+
+export {
+  type ToastProps,
+  type ToastActionElement,
+  ToastProvider,
+  ToastViewport,
+  Toast,
+  ToastTitle,
+  ToastDescription,
+  ToastClose,
+  ToastAction,
+}

inventory/src/components/ui/toaster.tsx

@@ -0,0 +1,33 @@
+import { useToast } from "@/hooks/use-toast"
+import {
+  Toast,
+  ToastClose,
+  ToastDescription,
+  ToastProvider,
+  ToastTitle,
+  ToastViewport,
+} from "@/components/ui/toast"
+
+export function Toaster() {
+  const { toasts } = useToast()
+
+  return (
+    <ToastProvider>
+      {toasts.map(function ({ id, title, description, action, ...props }) {
+        return (
+          <Toast key={id} {...props}>
+            <div className="grid gap-1">
+              {title && <ToastTitle>{title}</ToastTitle>}
+              {description && (
+                <ToastDescription>{description}</ToastDescription>
+              )}
+            </div>
+            {action}
+            <ToastClose />
+          </Toast>
+        )
+      })}
+      <ToastViewport />
+    </ToastProvider>
+  )
+}

inventory/src/hooks/use-toast.ts

@@ -0,0 +1,194 @@
+"use client"
+
+// Inspired by react-hot-toast library
+import * as React from "react"
+
+import type {
+  ToastActionElement,
+  ToastProps,
+} from "@/components/ui/toast"
+
+const TOAST_LIMIT = 1
+const TOAST_REMOVE_DELAY = 1000000
+
+type ToasterToast = ToastProps & {
+  id: string
+  title?: React.ReactNode
+  description?: React.ReactNode
+  action?: ToastActionElement
+}
+
+const actionTypes = {
+  ADD_TOAST: "ADD_TOAST",
+  UPDATE_TOAST: "UPDATE_TOAST",
+  DISMISS_TOAST: "DISMISS_TOAST",
+  REMOVE_TOAST: "REMOVE_TOAST",
+} as const
+
+let count = 0
+
+function genId() {
+  count = (count + 1) % Number.MAX_SAFE_INTEGER
+  return count.toString()
+}
+
+type ActionType = typeof actionTypes
+
+type Action =
+  | {
+      type: ActionType["ADD_TOAST"]
+      toast: ToasterToast
+    }
+  | {
+      type: ActionType["UPDATE_TOAST"]
+      toast: Partial<ToasterToast>
+    }
+  | {
+      type: ActionType["DISMISS_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+  | {
+      type: ActionType["REMOVE_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+
+interface State {
+  toasts: ToasterToast[]
+}
+
+const toastTimeouts = new Map<string, ReturnType<typeof setTimeout>>()
+
+const addToRemoveQueue = (toastId: string) => {
+  if (toastTimeouts.has(toastId)) {
+    return
+  }
+
+  const timeout = setTimeout(() => {
+    toastTimeouts.delete(toastId)
+    dispatch({
+      type: "REMOVE_TOAST",
+      toastId: toastId,
+    })
+  }, TOAST_REMOVE_DELAY)
+
+  toastTimeouts.set(toastId, timeout)
+}
+
+export const reducer = (state: State, action: Action): State => {
+  switch (action.type) {
+    case "ADD_TOAST":
+      return {
+        ...state,
+        toasts: [action.toast, ...state.toasts].slice(0, TOAST_LIMIT),
+      }
+
+    case "UPDATE_TOAST":
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === action.toast.id ? { ...t, ...action.toast } : t
+        ),
+      }
+
+    case "DISMISS_TOAST": {
+      const { toastId } = action
+
+      // ! Side effects ! - This could be extracted into a dismissToast() action,
+      // but I'll keep it here for simplicity
+      if (toastId) {
+        addToRemoveQueue(toastId)
+      } else {
+        state.toasts.forEach((toast) => {
+          addToRemoveQueue(toast.id)
+        })
+      }
+
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === toastId || toastId === undefined
+            ? {
+                ...t,
+                open: false,
+              }
+            : t
+        ),
+      }
+    }
+    case "REMOVE_TOAST":
+      if (action.toastId === undefined) {
+        return {
+          ...state,
+          toasts: [],
+        }
+      }
+      return {
+        ...state,
+        toasts: state.toasts.filter((t) => t.id !== action.toastId),
+      }
+  }
+}
+
+const listeners: Array<(state: State) => void> = []
+
+let memoryState: State = { toasts: [] }
+
+function dispatch(action: Action) {
+  memoryState = reducer(memoryState, action)
+  listeners.forEach((listener) => {
+    listener(memoryState)
+  })
+}
+
+type Toast = Omit<ToasterToast, "id">
+
+function toast({ ...props }: Toast) {
+  const id = genId()
+
+  const update = (props: ToasterToast) =>
+    dispatch({
+      type: "UPDATE_TOAST",
+      toast: { ...props, id },
+    })
+  const dismiss = () => dispatch({ type: "DISMISS_TOAST", toastId: id })
+
+  dispatch({
+    type: "ADD_TOAST",
+    toast: {
+      ...props,
+      id,
+      open: true,
+      onOpenChange: (open) => {
+        if (!open) dismiss()
+      },
+    },
+  })
+
+  return {
+    id: id,
+    dismiss,
+    update,
+  }
+}
+
+function useToast() {
+  const [state, setState] = React.useState<State>(memoryState)
+
+  React.useEffect(() => {
+    listeners.push(setState)
+    return () => {
+      const index = listeners.indexOf(setState)
+      if (index > -1) {
+        listeners.splice(index, 1)
+      }
+    }
+  }, [state])
+
+  return {
+    ...state,
+    toast,
+    dismiss: (toastId?: string) => dispatch({ type: "DISMISS_TOAST", toastId }),
+  }
+}
+
+export { useToast, toast }

inventory/src/index.css

@@ -99,3 +99,14 @@
     @apply bg-background text-foreground;
   }
 }
+
+
+
+@layer base {
+  * {
+    @apply border-border outline-ring/50;
+  }
+  body {
+    @apply bg-background text-foreground;
+  }
+}

inventory/src/lib/react-spreadsheet-import/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 UGNIS,
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inventory/src/lib/react-spreadsheet-import/README.md

@@ -0,0 +1,341 @@
+<h1 align="center">RSI react-spreadsheet-import ⚡️</h1>
+
+<div align="center">
+  
+![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/UgnisSoftware/react-spreadsheet-import/test.yaml)
+![GitHub](https://img.shields.io/github/license/UgnisSoftware/react-spreadsheet-import) [![npm](https://img.shields.io/npm/v/react-spreadsheet-import)](https://www.npmjs.com/package/react-spreadsheet-import)
+  
+</div>
+<br />
+
+A component used for importing XLS / XLSX / CSV documents built with [**Chakra UI**](https://chakra-ui.com). Import flow combines:
+
+- 📥 Uploader
+- ⚙️ Parser
+- 📊 File preview
+- 🧪 UI for column mapping
+- ✏ UI for validating and editing data
+
+✨ [**Demo**](https://ugnissoftware.github.io/react-spreadsheet-import/iframe.html?id=react-spreadsheet-import--basic&args=&viewMode=story) ✨
+<br />
+
+## Features
+
+- Custom styles - edit Chakra UI theme to match your project's styles 🎨
+- Custom validation rules - make sure valid data is being imported, easily spot and correct errors
+- Hooks - alter raw data after upload or make adjustments on data changes
+- Auto-mapping columns - automatically map most likely value to your template values, e.g. `name` -> `firstName`
+  <br />
+
+![rsi-preview](https://user-images.githubusercontent.com/45755753/159503528-90aacb69-128f-4ece-b45b-ab97d403a9d3.gif)
+
+## Figma
+
+We provide full figma designs. You can copy the designs
+[here](https://www.figma.com/community/file/1080776795891439629)
+
+## Getting started
+
+```sh
+npm i react-spreadsheet-import
+```
+
+Using the component: (it's up to you when the flow is open and what you do on submit with the imported data)
+
+```tsx
+import { ReactSpreadsheetImport } from "react-spreadsheet-import";
+
+<ReactSpreadsheetImport isOpen={isOpen} onClose={onClose} onSubmit={onSubmit} fields={fields} />
+```
+
+## Required Props
+
+```tsx
+  // Determines if modal is visible.
+  isOpen: Boolean
+  // Called when flow is closed without reaching submit.
+  onClose: () => void
+  // Called after user completes the flow. Provides data array, where data keys matches your field keys. 
+  onSubmit: (data, file) => void | Promise<any>
+```
+
+### Fields
+
+Fields describe what data you are trying to collect.
+
+```tsx
+const fields = [
+  {
+    // Visible in table header and when matching columns.
+    label: "Name",
+    // This is the key used for this field when we call onSubmit.
+    key: "name",
+    // Allows for better automatic column matching. Optional.
+    alternateMatches: ["first name", "first"],
+    // Used when editing and validating information.
+    fieldType: {
+      // There are 3 types - "input" / "checkbox" / "select".
+      type: "input",
+    },
+    // Used in the first step to provide an example of what data is expected in this field. Optional.
+    example: "Stephanie",
+    // Can have multiple validations that are visible in Validation Step table.
+    validations: [
+      {
+        // Can be "required" / "unique" / "regex"
+        rule: "required",
+        errorMessage: "Name is required",
+        // There can be "info" / "warning" / "error" levels. Optional. Default "error".
+        level: "error",
+      },
+    ],
+  },
+] as const
+```
+
+## Optional Props
+
+### Hooks
+
+You can transform and validate data with custom hooks. There are hooks after each step:
+
+- **uploadStepHook** - runs only once after uploading the file.
+- **selectHeaderStepHook** - runs only once after selecting the header row in spreadsheet.
+- **matchColumnsStepHook** - runs only once after column matching. Operations on data that are expensive should be done here.
+
+The last step - validation step has 2 unique hooks that run only in that step with different performance tradeoffs:
+
+- **tableHook** - runs at the start and on any change. Runs on all rows. Very expensive, but can change rows that depend on other rows.
+- **rowHook** - runs at the start and on any row change. Runs only on the rows changed. Fastest, most validations and transformations should be done here.
+
+Example:
+
+```tsx
+<ReactSpreadsheetImport
+  rowHook={(data, addError) => {
+    // Validation
+    if (data.name === "John") {
+      addError("name", { message: "No Johns allowed", level: "info" })
+    }
+    // Transformation
+    return { ...data, name: "Not John" }
+    // Sorry John
+  }}
+/>
+```
+
+### Initial state
+
+In rare case when you need to skip the beginning of the flow, you can start the flow from any of the steps.
+
+- **initialStepState** - initial state of component that will be rendered on load. 
+
+```tsx
+  initialStepState?: StepState
+  
+  type StepState =
+    | {
+        type: StepType.upload
+      }
+    | {
+        type: StepType.selectSheet
+        workbook: XLSX.WorkBook
+      }
+    | {
+        type: StepType.selectHeader
+        data: RawData[]
+      }
+    | {
+        type: StepType.matchColumns
+        data: RawData[]
+        headerValues: RawData
+      }
+    | {
+        type: StepType.validateData
+        data: any[]
+      }
+
+  type RawData = Array<string | undefined>
+
+  // XLSX.workbook type is native to SheetJS and can be viewed here: https://github.com/SheetJS/sheetjs/blob/83ddb4c1203f6bac052d8c1608b32fead02ea32f/types/index.d.ts#L269
+```
+
+Example:
+
+```tsx
+import { ReactSpreadsheetImport, StepType } from "react-spreadsheet-import";
+
+<ReactSpreadsheetImport
+  initialStepState={{
+    type: StepType.matchColumns,
+    data: [
+      ["Josh", "2"],
+      ["Charlie", "3"],
+      ["Lena", "50"],
+    ],
+    headerValues: ["name", "age"],
+  }}
+/>
+```
+
+### Dates and time
+
+Excel stores dates and times as numbers - offsets from an epoch. When reading xlsx files SheetJS provides date formatting helpers.
+**Default date import format** is `yyyy-mm-dd`. Date parsing with SheetJS sometimes yields unexpected results, therefore thorough date validations are recommended.
+
+- **dateFormat** - sets SheetJS `dateNF` option. Can be used to format dates when importing sheet data.
+- **parseRaw** - sets SheetJS `raw` option. If `true`, date formatting will be applied to XLSX date fields only. Default is `true`
+
+Common date-time formats can be viewed [here](https://docs.sheetjs.com/docs/csf/features/dates/#date-and-time-number-formats).
+
+### Other optional props
+
+```tsx
+  // Allows submitting with errors. Default: true
+  allowInvalidSubmit?: boolean
+  // Translations for each text. See customisation bellow
+  translations?: object
+  // Theme configuration passed to underlying Chakra-UI. See customisation bellow
+  customTheme?: object
+  // Specifies maximum number of rows for a single import
+  maxRecords?: number
+  // Maximum upload filesize (in bytes)
+  maxFileSize?: number
+  // Automatically map imported headers to specified fields if possible. Default: true
+  autoMapHeaders?: boolean
+  // When field type is "select", automatically match values if possible. Default: false
+  autoMapSelectValues?: boolean
+  // Headers matching accuracy: 1 for strict and up for more flexible matching. Default: 2
+  autoMapDistance?: number
+  // Enable navigation in stepper component and show back button. Default: false
+  isNavigationEnabled?: boolean
+```
+
+## Customisation
+
+### Customising styles (colors, fonts)
+
+You can see default theme we use [here](https://github.com/UgnisSoftware/react-spreadsheet-import/blob/master/src/theme.ts). Your override should match this object's structure.
+
+There are 3 ways you can style the component:
+
+1.) Change theme colors globally
+
+```jsx
+    <ReactSpreadsheetImport
+        {...mockRsiValues}
+        isOpen={isOpen}
+        onClose={onClose}
+        onSubmit={setData}
+        customTheme={{
+          colors: {
+            background: 'white',
+            ...
+            rsi: {
+              // your brand colors should go here
+              50: '...'
+              ...
+              500: 'teal',
+              ...
+              900: "...",
+            },
+          },
+        }}
+      />
+```
+
+<img width="1189" alt="Screenshot 2022-04-13 at 10 24 34" src="https://user-images.githubusercontent.com/5903616/163123718-15c05ad8-243b-4a81-8141-c47216047468.png">
+
+2.) Change all components of the same type, like all Buttons, at the same time
+
+```jsx
+<ReactSpreadsheetImport
+  {...mockRsiValues}
+  isOpen={isOpen}
+  onClose={onClose}
+  onSubmit={setData}
+  customTheme={{
+    components: {
+      Button: {
+        baseStyle: {
+          borderRadius: "none",
+        },
+        defaultProps: {
+          colorScheme: "yellow",
+        },
+      },
+    },
+  }}
+/>
+```
+
+<img width="1191" alt="Screenshot 2022-04-13 at 11 04 30" src="https://user-images.githubusercontent.com/5903616/163130213-82f955b4-5081-49e0-8f43-8857d480dacd.png">
+ 
+3.) Change components specifically in each Step.
+```jsx
+    <ReactSpreadsheetImport
+        {...mockRsiValues}
+        isOpen={isOpen}
+        onClose={onClose}
+        onSubmit={setData}
+        customTheme={{
+          components: {
+            UploadStep: {
+              baseStyle: {
+                dropzoneButton: {
+                  bg: "red",
+                },
+              },
+            },
+          },
+        }}
+      />
+```
+<img width="1182" alt="Screenshot 2022-04-13 at 10 21 58" src="https://user-images.githubusercontent.com/5903616/163123694-5b79179e-037e-4f9d-b1a9-6078f758bb7e.png">
+
+Underneath we use Chakra-UI, you can send in a custom theme for us to apply. Read more about themes [here](https://chakra-ui.com/docs/styled-system/theming/theme)
+
+### Changing text (translations)
+
+You can change any text in the flow:
+
+```tsx
+<ReactSpreadsheetImport
+  translations={{
+    uploadStep: {
+      title: "Upload Employees",
+    },
+  }}
+/>
+```
+
+You can see all the translation keys [here](https://github.com/UgnisSoftware/react-spreadsheet-import/blob/master/src/translationsRSIProps.ts)
+
+## VS other libraries
+
+Flatfile vs react-spreadsheet-import and Dromo vs react-spreadsheet-import:
+
+|                                | RSI            | Flatfile    | Dromo       |
+| ------------------------------ | -------------- | ----------- | ----------- |
+| Licence                        | MIT            | Proprietary | Proprietary |
+| Price                          | Free           | Paid        | Paid        |
+| Support                        | Github Issues  | Enterprise  | Enterprise  |
+| Self-host                      | Yes            | Paid        | Paid        |
+| Hosted solution                | In development | Yes         | Yes         |
+| On-prem deployment             | N/A            | Yes         | Yes         |
+| Hooks                          | Yes            | Yes         | Yes         |
+| Automatic header matching      | Yes            | Yes         | Yes         |
+| Data validation                | Yes            | Yes         | Yes         |
+| Custom styling                 | Yes            | Yes         | Yes         |
+| Translations                   | Yes            | Yes         | Yes         |
+| Trademarked words `Data Hooks` | No             | Yes         | No          |
+
+React-spreadsheet-import can be used as a free and open-source alternative to Flatfile and Dromo.
+
+## Contributing
+
+Feel free to open issues if you have any questions or notice bugs. If you want different component behaviour, consider forking the project.
+
+## Credits
+
+Created by Ugnis. [Julita Kriauciunaite](https://github.com/JulitorK) and [Karolis Masiulis](https://github.com/masiulis). You can contact us at `info@ugnis.com`

inventory/src/lib/react-spreadsheet-import/package.json

@@ -0,0 +1,174 @@
+{
+  "name": "react-spreadsheet-import",
+  "version": "4.7.1",
+  "description": "React spreadsheet import for xlsx and csv files with column matching and validation",
+  "main": "./dist-commonjs/index.js",
+  "module": "./dist/index.js",
+  "types": "./types/index.d.ts",
+  "files": [
+    "dist-commonjs",
+    "dist",
+    "types"
+  ],
+  "scripts": {
+    "start": "storybook dev -p 6006",
+    "test:unit": "jest",
+    "test:e2e": "npx playwright test",
+    "test:chromatic": "npx chromatic ",
+    "ts": "tsc",
+    "lint": "eslint \"src/**/*.{ts,tsx}\"",
+    "lint:fix": "eslint \"src/**/*.{ts,tsx}\" --fix",
+    "prebuild": "npm run clean",
+    "build": "rollup -c rollup.config.ts",
+    "build-storybook": "storybook build -o docs-build",
+    "release:patch": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version patch && git add -A && git push && git push --tags && npm publish",
+    "release:minor": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version minor && git add -A && git push && git push --tags && npm publish",
+    "release:major": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version major && git add -A && git push && git push --tags && npm publish",
+    "clean": "rimraf dist dist-commonjs types"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/UgnisSoftware/react-spreadsheet-import.git"
+  },
+  "keywords": [
+    "React",
+    "spreadsheet",
+    "import",
+    "upload",
+    "csv",
+    "xlsx",
+    "validate",
+    "automatic",
+    "match"
+  ],
+  "author": {
+    "name": "Ugnis"
+  },
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/UgnisSoftware/react-spreadsheet-import/issues"
+  },
+  "homepage": "https://github.com/UgnisSoftware/react-spreadsheet-import#readme",
+  "peerDependencies": {
+    "react": ">=18",
+    "react-dom": ">=18"
+  },
+  "dependencies": {
+    "@chakra-ui/react": "^2.8.1",
+    "@emotion/react": "^11.11.1",
+    "@emotion/styled": "^11.11.0",
+    "chakra-react-select": "^4.7.5",
+    "chakra-ui-steps": "2.0.4",
+    "framer-motion": "^10.16.4",
+    "js-levenshtein": "1.1.6",
+    "lodash": "4.17.21",
+    "react-data-grid": "7.0.0-beta.13",
+    "react-dropzone": "14.2.3",
+    "react-icons": "4.11.0",
+    "uuid": "^9.0.1",
+    "xlsx-ugnis": "0.20.3"
+  },
+  "devDependencies": {
+    "@babel/core": "7.23.2",
+    "@babel/preset-env": "7.23.2",
+    "@babel/preset-react": "7.22.15",
+    "@babel/preset-typescript": "7.23.2",
+    "@emotion/jest": "11.11.0",
+    "@jest/types": "27.5.1",
+    "@playwright/test": "^1.39.0",
+    "@storybook/addon-essentials": "7.5.1",
+    "@storybook/addon-interactions": "7.5.1",
+    "@storybook/addon-links": "7.5.1",
+    "@storybook/blocks": "7.5.1",
+    "@storybook/cli": "7.5.1",
+    "@storybook/react": "7.5.1",
+    "@storybook/react-webpack5": "7.5.1",
+    "@storybook/testing-library": "^0.0.14-next.2",
+    "@testing-library/dom": "9.3.0",
+    "@testing-library/jest-dom": "5.16.5",
+    "@testing-library/react": "14.0.0",
+    "@testing-library/user-event": "14.4.3",
+    "@types/jest": "27.4.1",
+    "@types/js-levenshtein": "1.1.1",
+    "@types/node": "^20.8.7",
+    "@types/react": "18.2.6",
+    "@types/react-dom": "18.2.4",
+    "@types/styled-system": "5.1.16",
+    "@types/uuid": "9.0.1",
+    "@typescript-eslint/eslint-plugin": "5.59.7",
+    "@typescript-eslint/parser": "5.59.7",
+    "babel-loader": "9.1.3",
+    "chromatic": "^7.4.0",
+    "eslint": "8.41.0",
+    "eslint-config-prettier": "8.8.0",
+    "eslint-plugin-prettier": "4.2.1",
+    "eslint-plugin-react": "7.32.2",
+    "eslint-plugin-react-hooks": "4.6.0",
+    "jest": "27.5.1",
+    "jest-watch-typeahead": "1.0.0",
+    "lint-staged": "13.2.2",
+    "prettier": "2.8.8",
+    "prop-types": "15.8.1",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-select-event": "5.5.1",
+    "rollup": "2.70.1",
+    "rollup-plugin-typescript2": "0.31.2",
+    "storybook": "7.5.1",
+    "ts-essentials": "9.3.2",
+    "ts-jest": "27.1.4",
+    "ttypescript": "1.5.15",
+    "typescript": "4.9.5",
+    "typescript-transform-paths": "3.4.6"
+  },
+  "overrides": {
+    "semver": "^7.5.3"
+  },
+  "lint-staged": {
+    "*.{ts,tsx}": "eslint",
+    "*.{js,ts,tsx,md,html,css,json}": "prettier --write"
+  },
+  "prettier": {
+    "tabWidth": 2,
+    "trailingComma": "all",
+    "semi": false,
+    "printWidth": 120
+  },
+  "jest": {
+    "preset": "ts-jest",
+    "testEnvironment": "jsdom",
+    "moduleFileExtensions": [
+      "ts",
+      "tsx",
+      "js",
+      "jsx",
+      "mjs"
+    ],
+    "transform": {
+      "^.+\\.(ts|tsx)?$": "ts-jest/dist",
+      "^.+\\.mjs$": "ts-jest/dist"
+    },
+    "moduleNameMapper": {
+      "~/(.*)": "<rootDir>/src/$1"
+    },
+    "modulePathIgnorePatterns": [
+      "<rootDir>/e2e/"
+    ],
+    "transformIgnorePatterns": [
+      "[/\\\\]node_modules[/\\\\].+\\.(js|jsx|mjs)$"
+    ],
+    "setupFiles": [
+      "./src/tests/setup.ts"
+    ],
+    "globals": {
+      "ts-jest": {
+        "tsconfig": "tsconfig.json"
+      }
+    },
+    "watchPlugins": [
+      "jest-watch-typeahead/filename",
+      "jest-watch-typeahead/testname"
+    ]
+  },
+  "readme": "ERROR: No README data found!"
+}

inventory/src/lib/react-spreadsheet-import/src/ReactSpreadsheetImport.tsx

@@ -0,0 +1,41 @@
+import merge from "lodash/merge"
+
+import { Steps } from "./steps/Steps"
+import { rtlThemeSupport, themeOverrides } from "./theme"
+import { Providers } from "./components/Providers"
+import type { RsiProps } from "./types"
+import { ModalWrapper } from "./components/ModalWrapper"
+import { translations } from "./translationsRSIProps"
+
+export const defaultTheme = themeOverrides
+
+export const defaultRSIProps: Partial<RsiProps<any>> = {
+  autoMapHeaders: true,
+  autoMapSelectValues: false,
+  allowInvalidSubmit: true,
+  autoMapDistance: 2,
+  isNavigationEnabled: false,
+  translations: translations,
+  uploadStepHook: async (value) => value,
+  selectHeaderStepHook: async (headerValues, data) => ({ headerValues, data }),
+  matchColumnsStepHook: async (table) => table,
+  dateFormat: "yyyy-mm-dd", // ISO 8601,
+  parseRaw: true,
+} as const
+
+export const ReactSpreadsheetImport = <T extends string>(propsWithoutDefaults: RsiProps<T>) => {
+  const props = merge({}, defaultRSIProps, propsWithoutDefaults)
+  const mergedTranslations =
+    props.translations !== translations ? merge(translations, props.translations) : translations
+  const mergedThemes = props.rtl
+    ? merge(defaultTheme, rtlThemeSupport, props.customTheme)
+    : merge(defaultTheme, props.customTheme)
+
+  return (
+    <Providers theme={mergedThemes} rsiValues={{ ...props, translations: mergedTranslations }}>
+      <ModalWrapper isOpen={props.isOpen} onClose={props.onClose}>
+        <Steps />
+      </ModalWrapper>
+    </Providers>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/ModalWrapper.tsx

@@ -0,0 +1,88 @@
+import type React from "react"
+import {
+  Dialog,
+  DialogContent,
+  DialogOverlay,
+  DialogPortal,
+  DialogClose,
+} from "@/components/ui/dialog"
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+  AlertDialogTrigger,
+  AlertDialogPortal,
+  AlertDialogOverlay,
+} from "@/components/ui/alert-dialog"
+import { useRsi } from "../hooks/useRsi"
+import { useState } from "react"
+
+type Props = {
+  children: React.ReactNode
+  isOpen: boolean
+  onClose: () => void
+}
+
+export const ModalWrapper = ({ children, isOpen, onClose }: Props) => {
+  const { rtl, translations } = useRsi()
+  const [showCloseAlert, setShowCloseAlert] = useState(false)
+  
+  return (
+    <>
+      <Dialog open={isOpen} onOpenChange={() => setShowCloseAlert(true)} modal>
+        <DialogPortal>
+          <DialogOverlay className="bg-background/80 backdrop-blur-sm" />
+          <DialogContent
+            onEscapeKeyDown={(e) => {
+              e.preventDefault()
+              setShowCloseAlert(true)
+            }}
+            onPointerDownOutside={(e) => e.preventDefault()}
+            className="fixed left-[50%] top-[50%] translate-x-[-50%] translate-y-[-50%] w-[calc(100%-2rem)] h-[calc(100%-2rem)] max-w-[100vw] max-h-[100vh] flex flex-col overflow-hidden rounded-lg border bg-background p-0 shadow-lg sm:w-[calc(100%-3rem)] sm:h-[calc(100%-3rem)] md:w-[calc(100%-4rem)] md:h-[calc(100%-4rem)]"
+          >
+            <AlertDialog>
+              <AlertDialogTrigger asChild>
+                <DialogClose className="absolute right-4 top-4" onClick={(e) => {
+                  e.preventDefault()
+                  setShowCloseAlert(true)
+                }} />
+              </AlertDialogTrigger>
+            </AlertDialog>
+            <div dir={rtl ? "rtl" : "ltr"} className="flex-1 overflow-auto">
+              {children}
+            </div>
+          </DialogContent>
+        </DialogPortal>
+      </Dialog>
+
+      <AlertDialog open={showCloseAlert} onOpenChange={setShowCloseAlert}>
+        <AlertDialogPortal>
+          <AlertDialogOverlay className="z-[1400]" />
+          <AlertDialogContent className="z-[1500]">
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {translations.alerts.confirmClose.headerTitle}
+              </AlertDialogTitle>
+              <AlertDialogDescription>
+                {translations.alerts.confirmClose.bodyText}
+              </AlertDialogDescription>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel onClick={() => setShowCloseAlert(false)}>
+                {translations.alerts.confirmClose.cancelButtonTitle}
+              </AlertDialogCancel>
+              <AlertDialogAction onClick={onClose}>
+                {translations.alerts.confirmClose.exitButtonTitle}
+              </AlertDialogAction>
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialogPortal>
+      </AlertDialog>
+    </>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/Providers.tsx

@@ -0,0 +1,33 @@
+import { ChakraProvider, extendTheme } from "@chakra-ui/react"
+import { createContext } from "react"
+import type { RsiProps } from "../types"
+import type { CustomTheme } from "../theme"
+
+export const RsiContext = createContext({} as any)
+
+type ProvidersProps<T extends string> = {
+  children: React.ReactNode
+  theme: CustomTheme
+  rsiValues: RsiProps<T>
+}
+
+export const rootId = "chakra-modal-rsi"
+
+export const Providers = <T extends string>({ children, theme, rsiValues }: ProvidersProps<T>) => {
+  const mergedTheme = extendTheme(theme)
+
+  if (!rsiValues.fields) {
+    throw new Error("Fields must be provided to react-spreadsheet-import")
+  }
+
+  return (
+    <RsiContext.Provider value={rsiValues}>
+      <ChakraProvider>
+        {/* cssVarsRoot used to override RSI defaultTheme but not the rest of chakra defaultTheme */}
+        <ChakraProvider cssVarsRoot={`#${rootId}`} theme={mergedTheme}>
+          {children}
+        </ChakraProvider>
+      </ChakraProvider>
+    </RsiContext.Provider>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/Table.tsx

@@ -0,0 +1,23 @@
+import type { DataGridProps, Column } from "react-data-grid"
+import DataGrid from "react-data-grid"
+import { useRsi } from "../hooks/useRsi"
+
+export type { Column }
+
+export type Props<TRow> = DataGridProps<TRow> & {
+  rowHeight?: number
+  hiddenHeader?: boolean
+  className?: string
+  style?: React.CSSProperties
+}
+
+export const Table = <TRow,>({ className, ...props }: Props<TRow>) => {
+  const { rtl } = useRsi()
+  return (
+    <DataGrid 
+      className={"rdg-light " + (className || "")} 
+      direction={rtl ? "rtl" : "ltr"} 
+      {...props} 
+    />
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/hooks/useRsi.ts

@@ -0,0 +1,9 @@
+import { useContext } from "react"
+import { RsiContext } from "../components/Providers"
+import type { RsiProps } from "../types"
+import type { MarkRequired } from "ts-essentials"
+import type { defaultRSIProps } from "../ReactSpreadsheetImport"
+import type { Translations } from "../translationsRSIProps"
+
+export const useRsi = <T extends string>() =>
+  useContext<MarkRequired<RsiProps<T>, keyof typeof defaultRSIProps> & { translations: Translations }>(RsiContext)

inventory/src/lib/react-spreadsheet-import/src/index.ts

@@ -0,0 +1,2 @@
+export { StepType } from "./steps/UploadFlow"
+export { ReactSpreadsheetImport } from "./ReactSpreadsheetImport"

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/MatchColumnsStep.tsx

@@ -0,0 +1,262 @@
+import { useCallback, useEffect, useMemo, useState } from "react"
+import { UserTableColumn } from "./components/UserTableColumn"
+import { useRsi } from "../../hooks/useRsi"
+import { TemplateColumn } from "./components/TemplateColumn"
+import { ColumnGrid } from "./components/ColumnGrid"
+import { setColumn } from "./utils/setColumn"
+import { setIgnoreColumn } from "./utils/setIgnoreColumn"
+import { setSubColumn } from "./utils/setSubColumn"
+import { normalizeTableData } from "./utils/normalizeTableData"
+import type { Field, RawData, Fields } from "../../types"
+import { getMatchedColumns } from "./utils/getMatchedColumns"
+import { findUnmatchedRequiredFields } from "./utils/findUnmatchedRequiredFields"
+import { toast } from "sonner"
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+  AlertDialogPortal,
+  AlertDialogOverlay,
+} from "@/components/ui/alert-dialog"
+import { DeepReadonly as TsDeepReadonly } from "ts-essentials"
+
+export type MatchColumnsProps<T extends string> = {
+  data: RawData[]
+  headerValues: RawData
+  onContinue: (data: any[], rawData: RawData[], columns: Columns<T>) => void
+  onBack?: () => void
+}
+
+export enum ColumnType {
+  empty,
+  ignored,
+  matched,
+  matchedCheckbox,
+  matchedSelect,
+  matchedSelectOptions,
+  matchedMultiInput,
+  matchedMultiSelect,
+}
+
+export type MatchedOptions<T> = {
+  entry: string
+  value: T
+}
+
+type EmptyColumn = { type: ColumnType.empty; index: number; header: string }
+type IgnoredColumn = { type: ColumnType.ignored; index: number; header: string }
+type MatchedColumn<T> = { type: ColumnType.matched; index: number; header: string; value: T }
+type MatchedSwitchColumn<T> = { type: ColumnType.matchedCheckbox; index: number; header: string; value: T }
+export type MatchedSelectColumn<T> = {
+  type: ColumnType.matchedSelect
+  index: number
+  header: string
+  value: T
+  matchedOptions: Partial<MatchedOptions<T>>[]
+}
+export type MatchedSelectOptionsColumn<T> = {
+  type: ColumnType.matchedSelectOptions
+  index: number
+  header: string
+  value: T
+  matchedOptions: MatchedOptions<T>[]
+}
+export type MatchedMultiInputColumn<T> = {
+  type: ColumnType.matchedMultiInput
+  index: number
+  header: string
+  value: T
+}
+export type MatchedMultiSelectColumn<T> = {
+  type: ColumnType.matchedMultiSelect
+  index: number
+  header: string
+  value: T
+  matchedOptions: MatchedOptions<T>[]
+}
+
+export type Column<T extends string> =
+  | EmptyColumn
+  | IgnoredColumn
+  | MatchedColumn<T>
+  | MatchedSwitchColumn<T>
+  | MatchedSelectColumn<T>
+  | MatchedSelectOptionsColumn<T>
+  | MatchedMultiInputColumn<T>
+  | MatchedMultiSelectColumn<T>
+
+export type Columns<T extends string> = Column<T>[]
+
+
+export const MatchColumnsStep = <T extends string>({
+  data,
+  headerValues,
+  onContinue,
+  onBack,
+}: MatchColumnsProps<T>) => {
+  const dataExample = data.slice(0, 2)
+  const { fields, autoMapHeaders, autoMapSelectValues, autoMapDistance, translations, allowInvalidSubmit } = useRsi<T>()
+  const [isLoading, setIsLoading] = useState(false)
+  const [columns, setColumns] = useState<Columns<T>>(
+    // Do not remove spread, it indexes empty array elements, otherwise map() skips over them
+    ([...headerValues] as string[]).map((value, index) => ({ type: ColumnType.empty, index, header: value ?? "" })),
+  )
+  const [showUnmatchedFieldsAlert, setShowUnmatchedFieldsAlert] = useState(false)
+
+  const onChange = useCallback(
+    (value: T, columnIndex: number) => {
+      const field = (fields as Fields<T>).find((f) => f.key === value)
+      if (!field) return
+
+      const existingFieldIndex = columns.findIndex((column) => "value" in column && column.value === field.key)
+      
+      setColumns(
+        columns.map<Column<T>>((column, index) => {
+          if (columnIndex === index) {
+            // Set the new column value
+            return setColumn(column, field as Field<T>, data, autoMapSelectValues)
+          } else if (index === existingFieldIndex) {
+            // Clear the old column that had this field
+            toast.warning(translations.matchColumnsStep.duplicateColumnWarningTitle, {
+              description: translations.matchColumnsStep.duplicateColumnWarningDescription,
+            })
+            return setColumn(column)
+          } else {
+            // Leave other columns unchanged
+            return column
+          }
+        }),
+      )
+    },
+    [
+      autoMapSelectValues,
+      columns,
+      data,
+      fields,
+      translations.matchColumnsStep.duplicateColumnWarningDescription,
+      translations.matchColumnsStep.duplicateColumnWarningTitle,
+    ],
+  )
+
+  const onIgnore = useCallback(
+    (columnIndex: number) => {
+      setColumns(columns.map((column, index) => (columnIndex === index ? setIgnoreColumn<T>(column) : column)))
+    },
+    [columns, setColumns],
+  )
+
+  const onRevertIgnore = useCallback(
+    (columnIndex: number) => {
+      setColumns(columns.map((column, index) => (columnIndex === index ? setColumn(column) : column)))
+    },
+    [columns, setColumns],
+  )
+
+  const onSubChange = useCallback(
+    (value: string, columnIndex: number, entry: string) => {
+      setColumns(
+        columns.map((column, index) =>
+          columnIndex === index && "matchedOptions" in column
+            ? setSubColumn(column as MatchedSelectColumn<T> | MatchedSelectOptionsColumn<T> | MatchedMultiSelectColumn<T>, entry, value)
+            : column,
+        ),
+      )
+    },
+    [columns, setColumns],
+  )
+  const unmatchedRequiredFields = useMemo(() => {
+    // Convert the fields to the expected type
+    const fieldsArray = Array.isArray(fields) ? fields : [fields]
+    const typedFields = fieldsArray.map(field => ({
+      ...field,
+      key: field.key as TsDeepReadonly<T>
+    })) as unknown as Fields<T>
+    
+    return findUnmatchedRequiredFields(typedFields, columns)
+  }, [fields, columns])
+
+  const handleOnContinue = useCallback(async () => {
+    if (unmatchedRequiredFields.length > 0) {
+      setShowUnmatchedFieldsAlert(true)
+    } else {
+      setIsLoading(true)
+      await onContinue(normalizeTableData(columns, data, fields), data, columns)
+      setIsLoading(false)
+    }
+  }, [unmatchedRequiredFields.length, onContinue, columns, data, fields])
+
+  const handleAlertOnContinue = useCallback(async () => {
+    setShowUnmatchedFieldsAlert(false)
+    setIsLoading(true)
+    await onContinue(normalizeTableData(columns, data, fields), data, columns)
+    setIsLoading(false)
+  }, [onContinue, columns, data, fields])
+
+  useEffect(
+    () => {
+      if (autoMapHeaders) {
+        setColumns(getMatchedColumns(columns, fields, data, autoMapDistance, autoMapSelectValues))
+      }
+    },
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+    [],
+  )
+
+  return (
+    <>
+      <AlertDialog open={showUnmatchedFieldsAlert} onOpenChange={setShowUnmatchedFieldsAlert}>
+        <AlertDialogPortal>
+          <AlertDialogOverlay className="z-[1400]" />
+          <AlertDialogContent className="z-[1500]">
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {translations.alerts.unmatchedRequiredFields.headerTitle}
+              </AlertDialogTitle>
+              <div className="space-y-3">
+                <AlertDialogDescription>
+                  {translations.alerts.unmatchedRequiredFields.bodyText}
+                </AlertDialogDescription>
+                <p className="text-sm text-muted-foreground">
+                  {translations.alerts.unmatchedRequiredFields.listTitle}{" "}
+                  <span className="font-bold">
+                    {unmatchedRequiredFields.join(", ")}
+                  </span>
+                </p>
+              </div>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel>
+                {translations.alerts.unmatchedRequiredFields.cancelButtonTitle}
+              </AlertDialogCancel>
+              {allowInvalidSubmit && (
+                <AlertDialogAction onClick={handleAlertOnContinue}>
+                  {translations.alerts.unmatchedRequiredFields.continueButtonTitle}
+                </AlertDialogAction>
+              )}
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialogPortal>
+      </AlertDialog>
+      <ColumnGrid
+        columns={columns}
+        onContinue={handleOnContinue}
+        onBack={onBack}
+        isLoading={isLoading}
+        userColumn={(column) => (
+          <UserTableColumn
+            column={column}
+            onIgnore={onIgnore}
+            onRevertIgnore={onRevertIgnore}
+            entries={dataExample.map((row) => row[column.index])}
+          />
+        )}
+        templateColumn={(column) => <TemplateColumn column={column} onChange={onChange} onSubChange={onSubChange} />}
+      />
+    </>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/ColumnGrid.tsx

@@ -0,0 +1,112 @@
+import type React from "react"
+import type { Column, Columns } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import { useRsi } from "../../../hooks/useRsi"
+import { Button } from "@/components/ui/button"
+import { ScrollArea, ScrollBar } from "@/components/ui/scroll-area"
+
+type ColumnGridProps<T extends string> = {
+  columns: Columns<T>
+  userColumn: (column: Column<T>) => React.ReactNode
+  templateColumn: (column: Column<T>) => React.ReactNode
+  onContinue: (val: Record<string, string>[]) => void
+  onBack?: () => void
+  isLoading: boolean
+}
+
+export const ColumnGrid = <T extends string>({
+  columns,
+  userColumn,
+  templateColumn,
+  onContinue,
+  onBack,
+  isLoading,
+}: ColumnGridProps<T>) => {
+  const { translations } = useRsi()
+  const normalColumnWidth = 250
+  const ignoredColumnWidth = 48 // 12 units = 3rem = 48px
+  const gap = 16
+  const totalWidth = columns.reduce((acc, col) => 
+    acc + (col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth) + gap, 
+    -gap // Subtract one gap since we need gaps between columns only
+  )
+
+  return (
+    <div className="flex h-[calc(100vh-10rem)] flex-col">
+      <div className="flex-1 overflow-hidden">
+        <div className="px-8 py-6">
+          <div className="mb-8">
+            <h2 className="text-3xl font-semibold text-foreground">
+              {translations.matchColumnsStep.title}
+            </h2>
+          </div>
+          <ScrollArea className="relative">
+            <div className="space-y-8" style={{ width: totalWidth }}>
+              {/* Your table section */}
+              <div>
+                <h3 className="mb-4 text-lg font-medium text-foreground">
+                  {translations.matchColumnsStep.userTableTitle}
+                </h3>
+                <div className="relative">
+                  <div 
+                    className="grid auto-cols-fr gap-4"
+                    style={{ 
+                      gridTemplateColumns: columns.map(col => 
+                        `${col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth}px`
+                      ).join(" "),
+                    }}
+                  >
+                    {columns.map((column, index) => (
+                      <div key={column.header + index}>
+                        {userColumn(column)}
+                      </div>
+                    ))}
+                  </div>
+                  <div className="pointer-events-none absolute bottom-0 left-0 right-0 h-16 bg-gradient-to-b from-transparent via-background/50 to-background" />
+                </div>
+              </div>
+
+              {/* Will become section */}
+              <div>
+                <h3 className="mb-4 text-lg font-medium text-foreground">
+                  {translations.matchColumnsStep.templateTitle}
+                </h3>
+                <div 
+                  className="grid auto-cols-fr gap-4"
+                  style={{ 
+                    gridTemplateColumns: columns.map(col => 
+                      `${col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth}px`
+                    ).join(" "),
+                  }}
+                >
+                  {columns.map((column, index) => (
+                    <div key={column.header + index}>
+                      {templateColumn(column)}
+                    </div>
+                  ))}
+                </div>
+              </div>
+            </div>
+            <ScrollBar orientation="horizontal" />
+          </ScrollArea>
+        </div>
+      </div>
+      <div className="border-t bg-muted px-8 py-4 -mb-1">
+        <div className="flex items-center justify-between">
+          {onBack && (
+            <Button variant="outline" onClick={onBack}>
+              {translations.matchColumnsStep.backButtonTitle}
+            </Button>
+          )}
+          <Button 
+            className="ml-auto"
+            disabled={isLoading}
+            onClick={() => onContinue([])}
+          >
+            {translations.matchColumnsStep.nextButtonTitle}
+          </Button>
+        </div>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/MatchIcon.tsx

@@ -0,0 +1,42 @@
+import { chakra, useStyleConfig, Flex } from "@chakra-ui/react"
+import { dataAttr } from "@chakra-ui/utils"
+import { motion } from "framer-motion"
+import { CgCheck } from "react-icons/cg"
+
+const MotionFlex = motion(Flex)
+
+const animationConfig = {
+  transition: {
+    duration: 0.1,
+  },
+  exit: { scale: 0.5, opacity: 0 },
+  initial: { scale: 0.5, opacity: 0 },
+  animate: { scale: 1, opacity: 1 },
+}
+type MatchIconProps = {
+  isChecked: boolean
+}
+
+export const MatchIcon = (props: MatchIconProps) => {
+  const style = useStyleConfig("MatchIcon", props)
+
+  return (
+    <chakra.div
+      __css={style}
+      minW={6}
+      minH={6}
+      w={6}
+      h={6}
+      ml="0.875rem"
+      mr={3}
+      data-highlighted={dataAttr(props.isChecked)}
+      data-testid="column-checkmark"
+    >
+      {props.isChecked && (
+        <MotionFlex {...animationConfig}>
+          <CgCheck size="24px" />
+        </MotionFlex>
+      )}
+    </chakra.div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/TemplateColumn.tsx

@@ -0,0 +1,133 @@
+import { useRsi } from "../../../hooks/useRsi"
+import type { Column } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { Fields } from "../../../types"
+import {
+  Card,
+  CardContent,
+  CardHeader,
+} from "@/components/ui/card"
+import {
+  Select,
+  SelectContent,
+  SelectItem,
+  SelectTrigger,
+  SelectValue,
+} from "@/components/ui/select"
+import {
+  Accordion,
+  AccordionContent,
+  AccordionItem,
+  AccordionTrigger,
+} from "@/components/ui/accordion"
+import { Check } from "lucide-react"
+
+type TemplateColumnProps<T extends string> = {
+  column: Column<T>
+  onChange: (value: T, columnIndex: number) => void
+  onSubChange: (value: string, columnIndex: number, entry: string) => void
+}
+
+const getAccordionTitle = <T extends string>(fields: Fields<T>, column: Column<T>, translations: any) => {
+  const field = fields.find((f) => "value" in column && f.key === column.value)
+  if (!field) return ""
+  return `${translations.matchColumnsStep.matchDropdownTitle} ${field.label} (${
+    "matchedOptions" in column ? column.matchedOptions.filter((option) => !option.value).length : 0
+  } ${translations.matchColumnsStep.unmatched})`
+}
+
+export const TemplateColumn = <T extends string>({ column, onChange, onSubChange }: TemplateColumnProps<T>) => {
+  const { translations, fields } = useRsi<T>()
+  const isIgnored = column.type === ColumnType.ignored
+  const isChecked =
+    column.type === ColumnType.matched ||
+    column.type === ColumnType.matchedCheckbox ||
+    column.type === ColumnType.matchedSelectOptions
+  const isSelect = "matchedOptions" in column
+  const selectOptions = fields.map(({ label, key }) => ({ value: key, label }))
+  const selectValue = column.type === ColumnType.empty ? undefined : 
+    selectOptions.find(({ value }) => "value" in column && column.value === value)?.value
+
+  if (isIgnored) {
+    return null
+  }
+
+  return (
+    <Card className="h-full">
+      <CardHeader className="flex flex-row items-center justify-between space-x-2 p-4">
+        <div className="flex-1">
+          <Select
+            key={`select-${column.index}-${("value" in column ? column.value : "empty")}`}
+            value={selectValue}
+            onValueChange={(value) => onChange(value as T, column.index)}
+          >
+            <SelectTrigger className="w-full">
+              <SelectValue placeholder={translations.matchColumnsStep.selectPlaceholder} />
+            </SelectTrigger>
+            <SelectContent 
+              side="bottom"
+              align="start"
+              className="z-[1500]"
+            >
+              {selectOptions.map((option) => (
+                <SelectItem key={option.value} value={option.value}>
+                  {option.label}
+                </SelectItem>
+              ))}
+            </SelectContent>
+          </Select>
+        </div>
+        {isChecked && (
+          <div className="flex h-8 w-8 items-center justify-center rounded-full border border-green-700 bg-green-300 dark:bg-green-900/20">
+            <Check className="h-4 w-4 text-green-700 dark:text-green-500" />
+          </div>
+        )}
+      </CardHeader>
+      {isSelect && (
+        <CardContent className="p-4">
+          <Accordion type="multiple" className="w-full">
+            <AccordionItem value="options" className="border-none">
+              <AccordionTrigger className="py-2 text-sm hover:no-underline">
+                {getAccordionTitle(fields, column, translations)}
+              </AccordionTrigger>
+              <AccordionContent>
+                <div className="space-y-2">
+                  {column.matchedOptions.map((option) => (
+                    <div key={option.entry} className="space-y-1">
+                      <p className="text-sm text-muted-foreground">
+                        {option.entry}
+                      </p>
+                      <Select
+                        value={option.value}
+                        onValueChange={(value) => onSubChange(value, column.index, option.entry!)}
+                      >
+                        <SelectTrigger className="w-full">
+                          <SelectValue placeholder={translations.matchColumnsStep.subSelectPlaceholder} />
+                        </SelectTrigger>
+                        <SelectContent 
+                          side="bottom"
+                          align="start"
+                          className="z-[1000]"
+                        >
+                          {(() => {
+                            const field = fields.find((f) => "value" in column && f.key === column.value)
+                            if (!field || !("fieldType" in field) || !("options" in field.fieldType)) return null
+                            return field.fieldType.options.map((option) => (
+                              <SelectItem key={option.value} value={option.value}>
+                                {option.label}
+                              </SelectItem>
+                            ))
+                          })()}
+                        </SelectContent>
+                      </Select>
+                    </div>
+                  ))}
+                </div>
+              </AccordionContent>
+            </AccordionItem>
+          </Accordion>
+        </CardContent>
+      )}
+    </Card>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/UserTableColumn.tsx

@@ -0,0 +1,74 @@
+import { Button } from "@/components/ui/button"
+import { Card, CardContent, CardHeader } from "@/components/ui/card"
+import { X, RotateCcw } from "lucide-react"
+import type { Column } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { RawData } from "../../../types"
+
+type UserTableColumnProps<T extends string> = {
+  column: Column<T>
+  entries: RawData
+  onIgnore: (index: number) => void
+  onRevertIgnore: (index: number) => void
+}
+
+export const UserTableColumn = <T extends string>(props: UserTableColumnProps<T>) => {
+  const {
+    column: { header, index, type },
+    entries,
+    onIgnore,
+    onRevertIgnore,
+  } = props
+  const isIgnored = type === ColumnType.ignored
+
+  if (isIgnored) {
+    return (
+      <Card className="h-full w-12 bg-muted/50">
+        <CardHeader className="flex flex-col items-center space-y-4 p-2">
+          <Button
+            variant="ghost"
+            size="icon"
+            onClick={() => onRevertIgnore(index)}
+            className="h-8 w-8"
+          >
+            <RotateCcw className="h-4 w-4" />
+          </Button>
+          <div 
+            className="vertical-text font-medium text-muted-foreground"
+            style={{ writingMode: 'vertical-rl', textOrientation: 'mixed', transform: 'rotate(0deg)' }}
+          >
+            {header}
+          </div>
+        </CardHeader>
+      </Card>
+    )
+  }
+
+  return (
+    <Card className="h-full">
+      <CardHeader className="flex flex-row items-center justify-between space-x-2 p-4">
+        <p className="font-medium">
+          {header}
+        </p>
+        <Button
+          variant="ghost"
+          size="icon"
+          onClick={() => onIgnore(index)}
+          className="h-8 w-8"
+        >
+          <X className="h-4 w-4" />
+        </Button>
+      </CardHeader>
+      <CardContent className="space-y-2 p-4">
+        {entries.map((entry, i) => (
+          <p 
+            key={`${entry || ""}-${i}`}
+            className="truncate text-sm text-muted-foreground"
+          >
+            {entry}
+          </p>
+        ))}
+      </CardContent>
+    </Card>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/findMatch.ts

@@ -0,0 +1,27 @@
+import lavenstein from "js-levenshtein"
+import type { Fields } from "../../../types"
+
+type AutoMatchAccumulator<T> = {
+  distance: number
+  value: T
+}
+
+export const findMatch = <T extends string>(
+  header: string,
+  fields: Fields<T>,
+  autoMapDistance: number,
+): T | undefined => {
+  const headerLower = header.toLowerCase()
+  const smallestValue = fields.reduce<AutoMatchAccumulator<T>>((acc, field) => {
+    const distance = Math.min(
+      ...[
+        lavenstein(field.key.toLowerCase(), headerLower),
+        ...(field.alternateMatches?.map((alternate) => lavenstein(alternate.toLowerCase(), headerLower)) || []),
+      ],
+    )
+    return distance < acc.distance || acc.distance === undefined
+      ? ({ value: field.key, distance } as AutoMatchAccumulator<T>)
+      : acc
+  }, {} as AutoMatchAccumulator<T>)
+  return smallestValue.distance <= autoMapDistance ? smallestValue.value : undefined
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/findUnmatchedRequiredFields.ts

@@ -0,0 +1,8 @@
+import type { Fields } from "../../../types"
+import type { Columns } from "../MatchColumnsStep"
+
+export const findUnmatchedRequiredFields = <T extends string>(fields: Fields<T>, columns: Columns<T>) =>
+  fields
+    .filter((field) => field.validations?.some((validation) => validation.rule === "required"))
+    .filter((field) => columns.findIndex((column) => "value" in column && column.value === field.key) === -1)
+    .map((field) => field.label) || []

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/getFieldOptions.ts

@@ -0,0 +1,6 @@
+import type { Fields } from "../../../types"
+
+export const getFieldOptions = <T extends string>(fields: Fields<T>, fieldKey: string) => {
+  const field = fields.find(({ key }) => fieldKey === key)!
+  return field.fieldType.type === "select" ? field.fieldType.options : []
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/getMatchedColumns.ts

@@ -0,0 +1,41 @@
+import lavenstein from "js-levenshtein"
+import { findMatch } from "./findMatch"
+import type { Field, Fields } from "../../../types"
+import { setColumn } from "./setColumn"
+import type { Column, Columns } from "../MatchColumnsStep"
+import type { MatchColumnsProps } from "../MatchColumnsStep"
+
+export const getMatchedColumns = <T extends string>(
+  columns: Columns<T>,
+  fields: Fields<T>,
+  data: MatchColumnsProps<T>["data"],
+  autoMapDistance: number,
+  autoMapSelectValues?: boolean,
+) =>
+  columns.reduce<Column<T>[]>((arr, column) => {
+    const autoMatch = findMatch(column.header, fields, autoMapDistance)
+    if (autoMatch) {
+      const field = fields.find((field) => field.key === autoMatch) as Field<T>
+      const duplicateIndex = arr.findIndex((column) => "value" in column && column.value === field.key)
+      const duplicate = arr[duplicateIndex]
+      if (duplicate && "value" in duplicate) {
+        return lavenstein(duplicate.value, duplicate.header) < lavenstein(autoMatch, column.header)
+          ? [
+              ...arr.slice(0, duplicateIndex),
+              setColumn(arr[duplicateIndex], field, data, autoMapSelectValues),
+              ...arr.slice(duplicateIndex + 1),
+              setColumn(column),
+            ]
+          : [
+              ...arr.slice(0, duplicateIndex),
+              setColumn(arr[duplicateIndex]),
+              ...arr.slice(duplicateIndex + 1),
+              setColumn(column, field, data, autoMapSelectValues),
+            ]
+      } else {
+        return [...arr, setColumn(column, field, data, autoMapSelectValues)]
+      }
+    } else {
+      return [...arr, column]
+    }
+  }, [])

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/normalizeCheckboxValue.ts

@@ -0,0 +1,13 @@
+const booleanWhitelist: Record<string, boolean> = {
+  yes: true,
+  no: false,
+  true: true,
+  false: false,
+}
+
+export const normalizeCheckboxValue = (value: string | undefined): boolean => {
+  if (value && value.toLowerCase() in booleanWhitelist) {
+    return booleanWhitelist[value.toLowerCase()]
+  }
+  return false
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/normalizeTableData.ts

@@ -0,0 +1,67 @@
+import type { Columns } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { Data, Fields, RawData } from "../../../types"
+import { normalizeCheckboxValue } from "./normalizeCheckboxValue"
+
+export const normalizeTableData = <T extends string>(columns: Columns<T>, data: RawData[], fields: Fields<T>) =>
+  data.map((row) =>
+    columns.reduce((acc, column, index) => {
+      const curr = row[index]
+      switch (column.type) {
+        case ColumnType.matchedCheckbox: {
+          const field = fields.find((field) => field.key === column.value)!
+          if ("booleanMatches" in field.fieldType && Object.keys(field.fieldType).length) {
+            const booleanMatchKey = Object.keys(field.fieldType.booleanMatches || []).find(
+              (key) => key.toLowerCase() === curr?.toLowerCase(),
+            )!
+            const booleanMatch = field.fieldType.booleanMatches?.[booleanMatchKey]
+            acc[column.value] = booleanMatchKey ? booleanMatch : normalizeCheckboxValue(curr)
+          } else {
+            acc[column.value] = normalizeCheckboxValue(curr)
+          }
+          return acc
+        }
+        case ColumnType.matched: {
+          acc[column.value] = curr === "" ? undefined : curr
+          return acc
+        }
+        case ColumnType.matchedMultiInput: {
+          const field = fields.find((field) => field.key === column.value)!
+          if (curr) {
+            const separator = field.fieldType.type === "multi-input" ? field.fieldType.separator || "," : ","
+            acc[column.value] = curr.split(separator).map(v => v.trim()).filter(Boolean)
+          } else {
+            acc[column.value] = undefined
+          }
+          return acc
+        }
+        case ColumnType.matchedSelect:
+        case ColumnType.matchedSelectOptions: {
+          const matchedOption = column.matchedOptions.find(({ entry, value }) => entry === curr)
+          acc[column.value] = matchedOption?.value || undefined
+          return acc
+        }
+        case ColumnType.matchedMultiSelect: {
+          const field = fields.find((field) => field.key === column.value)!
+          if (curr) {
+            const separator = field.fieldType.type === "multi-select" ? field.fieldType.separator || "," : ","
+            const entries = curr.split(separator).map(v => v.trim()).filter(Boolean)
+            const values = entries.map(entry => {
+              const matchedOption = column.matchedOptions.find(({ entry: optEntry }) => optEntry === entry)
+              return matchedOption?.value
+            }).filter(Boolean) as string[]
+            acc[column.value] = values.length ? values : undefined
+          } else {
+            acc[column.value] = undefined
+          }
+          return acc
+        }
+        case ColumnType.empty:
+        case ColumnType.ignored: {
+          return acc
+        }
+        default:
+          return acc
+      }
+    }, {} as Data<T>),
+  )

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setColumn.ts

@@ -0,0 +1,65 @@
+import type { Field, MultiSelect } from "../../../types"
+import { Column, ColumnType, MatchColumnsProps, MatchedOptions } from "../MatchColumnsStep"
+import { uniqueEntries } from "./uniqueEntries"
+
+export const setColumn = <T extends string>(
+  oldColumn: Column<T>,
+  field?: Field<T>,
+  data?: MatchColumnsProps<T>["data"],
+  autoMapSelectValues?: boolean,
+): Column<T> => {
+  switch (field?.fieldType.type) {
+    case "select":
+      const fieldOptions = field.fieldType.options
+      const uniqueData = uniqueEntries(data || [], oldColumn.index) as MatchedOptions<T>[]
+      const matchedOptions = autoMapSelectValues
+        ? uniqueData.map((record) => {
+            const value = fieldOptions.find(
+              (fieldOption) => fieldOption.value === record.entry || fieldOption.label === record.entry,
+            )?.value
+            return value ? ({ ...record, value } as MatchedOptions<T>) : (record as MatchedOptions<T>)
+          })
+        : uniqueData
+      const allMatched = matchedOptions.filter((o) => o.value).length == uniqueData?.length
+
+      return {
+        ...oldColumn,
+        type: allMatched ? ColumnType.matchedSelectOptions : ColumnType.matchedSelect,
+        value: field.key,
+        matchedOptions,
+      }
+    case "multi-select":
+      const multiSelectFieldType = field.fieldType as MultiSelect
+      const multiSelectFieldOptions = multiSelectFieldType.options
+      const multiSelectUniqueData = uniqueEntries(data || [], oldColumn.index) as MatchedOptions<T>[]
+      const multiSelectMatchedOptions = autoMapSelectValues
+        ? multiSelectUniqueData.map((record) => {
+            // Split the entry by the separator (default to comma)
+            const entries = record.entry.split(multiSelectFieldType.separator || ",").map(e => e.trim())
+            // Try to match each entry to an option
+            const values = entries.map(entry => {
+              const value = multiSelectFieldOptions.find(
+                (fieldOption) => fieldOption.value === entry || fieldOption.label === entry,
+              )?.value
+              return value
+            }).filter(Boolean) as T[]
+            return { ...record, value: values.length ? values[0] : undefined } as MatchedOptions<T>
+          })
+        : multiSelectUniqueData
+
+      return {
+        ...oldColumn,
+        type: ColumnType.matchedMultiSelect,
+        value: field.key,
+        matchedOptions: multiSelectMatchedOptions,
+      }
+    case "checkbox":
+      return { index: oldColumn.index, type: ColumnType.matchedCheckbox, value: field.key, header: oldColumn.header }
+    case "input":
+      return { index: oldColumn.index, type: ColumnType.matched, value: field.key, header: oldColumn.header }
+    case "multi-input":
+      return { index: oldColumn.index, type: ColumnType.matchedMultiInput, value: field.key, header: oldColumn.header }
+    default:
+      return { index: oldColumn.index, header: oldColumn.header, type: ColumnType.empty }
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setIgnoreColumn.ts

@@ -0,0 +1,7 @@
+import { Column, ColumnType } from "../MatchColumnsStep"
+
+export const setIgnoreColumn = <T extends string>({ header, index }: Column<T>): Column<T> => ({
+  header,
+  index,
+  type: ColumnType.ignored,
+})

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setSubColumn.ts

@@ -0,0 +1,20 @@
+import { ColumnType, MatchedOptions, MatchedSelectColumn, MatchedSelectOptionsColumn, MatchedMultiSelectColumn } from "../MatchColumnsStep"
+
+export const setSubColumn = <T>(
+  oldColumn: MatchedSelectColumn<T> | MatchedSelectOptionsColumn<T> | MatchedMultiSelectColumn<T>,
+  entry: string,
+  value: string,
+): MatchedSelectColumn<T> | MatchedSelectOptionsColumn<T> | MatchedMultiSelectColumn<T> => {
+  const options = oldColumn.matchedOptions.map((option) => (option.entry === entry ? { ...option, value } : option))
+  const allMatched = options.every(({ value }) => !!value)
+
+  if (oldColumn.type === ColumnType.matchedMultiSelect) {
+    return { ...oldColumn, matchedOptions: options as MatchedOptions<T>[] }
+  }
+  
+  if (allMatched) {
+    return { ...oldColumn, matchedOptions: options as MatchedOptions<T>[], type: ColumnType.matchedSelectOptions }
+  } else {
+    return { ...oldColumn, matchedOptions: options as MatchedOptions<T>[], type: ColumnType.matchedSelect }
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/uniqueEntries.ts

@@ -0,0 +1,11 @@
+import uniqBy from "lodash/uniqBy"
+import type { MatchColumnsProps, MatchedOptions } from "../MatchColumnsStep"
+
+export const uniqueEntries = <T extends string>(
+  data: MatchColumnsProps<T>["data"],
+  index: number,
+): Partial<MatchedOptions<T>>[] =>
+  uniqBy(
+    data.map((row) => ({ entry: row[index] })),
+    "entry",
+  ).filter(({ entry }) => !!entry)

inventory/src/lib/react-spreadsheet-import/src/steps/SelectHeaderStep/SelectHeaderStep.tsx

@@ -0,0 +1,57 @@
+import { useCallback, useState } from "react"
+import { SelectHeaderTable } from "./components/SelectHeaderTable"
+import { useRsi } from "../../hooks/useRsi"
+import type { RawData } from "../../types"
+import { Button } from "@/components/ui/button"
+
+type SelectHeaderProps = {
+  data: RawData[]
+  onContinue: (headerValues: RawData, data: RawData[]) => Promise<void>
+  onBack?: () => void
+}
+
+export const SelectHeaderStep = ({ data, onContinue, onBack }: SelectHeaderProps) => {
+  const { translations } = useRsi()
+  const [selectedRows, setSelectedRows] = useState<ReadonlySet<number>>(new Set([0]))
+  const [isLoading, setIsLoading] = useState(false)
+
+  const handleContinue = useCallback(async () => {
+    const [selectedRowIndex] = selectedRows
+    // We consider data above header to be redundant
+    const trimmedData = data.slice(selectedRowIndex + 1)
+    setIsLoading(true)
+    await onContinue(data[selectedRowIndex], trimmedData)
+    setIsLoading(false)
+  }, [onContinue, data, selectedRows])
+
+  return (
+    <div className="flex flex-col">
+      <div className="px-8 py-6">
+        <h2 className="text-2xl font-semibold text-foreground">
+          {translations.selectHeaderStep.title}
+        </h2>
+      </div>
+      <div className="flex-1 px-8 mb-12 overflow-auto">
+        <SelectHeaderTable 
+          data={data} 
+          selectedRows={selectedRows} 
+          setSelectedRows={setSelectedRows} 
+        />
+      </div>
+      <div className="flex items-center justify-between border-t bg-muted px-8 py-4 mt-2">
+        {onBack && (
+          <Button variant="outline" onClick={onBack}>
+            {translations.selectHeaderStep.backButtonTitle}
+          </Button>
+        )}
+        <Button 
+          className="ml-auto"
+          disabled={isLoading}
+          onClick={handleContinue}
+        >
+          {translations.selectHeaderStep.nextButtonTitle}
+        </Button>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/SelectHeaderStep/components/SelectHeaderTable.tsx

@@ -0,0 +1,106 @@
+import { useMemo } from "react"
+import type { RawData } from "../../../types"
+import {
+  Table,
+  TableBody,
+  TableCell,
+  TableHead,
+  TableHeader,
+  TableRow,
+} from "@/components/ui/table"
+import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
+import { Label } from "@/components/ui/label"
+import { cn } from "@/lib/utils"
+
+interface Props {
+  data: RawData[]
+  selectedRows: ReadonlySet<number>
+  setSelectedRows: (rows: ReadonlySet<number>) => void
+}
+
+export const SelectHeaderTable = ({ data, selectedRows, setSelectedRows }: Props) => {
+  const columns = useMemo(() => {
+    const longestRowLength = data.reduce((acc, curr) => (acc > curr.length ? acc : curr.length), 0)
+    return Array.from(Array(longestRowLength), (_, index) => ({
+      key: index.toString(),
+      name: `Column ${index + 1}`,
+    }))
+  }, [data])
+
+  if (!data || data.length === 0) {
+    return (
+      <div className="p-4">
+        <p className="text-sm text-muted-foreground">No data available to select headers from.</p>
+      </div>
+    )
+  }
+
+  const selectedRowIndex = Array.from(selectedRows)[0]
+  const gridTemplateColumns = `60px repeat(${columns.length}, minmax(150px, 300px))`
+
+  return (
+    <div className="rounded-md border p-3">
+      <p className="mb-2 p-2 text-sm text-muted-foreground">
+        Select the row that contains your column headers
+      </p>
+      <div className="h-[calc(100vh-27rem)] overflow-auto">
+        <Table className="relative w-full" style={{ tableLayout: 'fixed' }}>
+          <TableHeader>
+            <TableRow className="grid" style={{ gridTemplateColumns }}>
+              <TableHead className="sticky top-0 z-20 bg-background overflow-hidden">
+                <div className="truncate">Select</div>
+              </TableHead>
+              {columns.map((column) => (
+                <TableHead 
+                  key={column.key}
+                  className="sticky top-0 z-20 bg-background overflow-hidden"
+                >
+                  <div className="truncate">
+                    {column.name}
+                  </div>
+                </TableHead>
+              ))}
+            </TableRow>
+          </TableHeader>
+          <TableBody>
+            <RadioGroup
+              value={selectedRowIndex?.toString()}
+              onValueChange={(value) => setSelectedRows(new Set([parseInt(value)]))}
+            >
+              {data.map((row, rowIndex) => (
+                <TableRow 
+                  key={rowIndex}
+                  className={cn(
+                    "grid",
+                    selectedRowIndex === rowIndex && "bg-muted",
+                    "group hover:bg-muted/50"
+                  )}
+                  style={{ gridTemplateColumns }}
+                >
+                  <TableCell className="overflow-hidden">
+                    <div className="flex items-center">
+                      <RadioGroupItem value={rowIndex.toString()} id={`row-${rowIndex}`} />
+                      <Label htmlFor={`row-${rowIndex}`} className="sr-only">
+                        Select as header row
+                      </Label>
+                    </div>
+                  </TableCell>
+                  {columns.map((column, colIndex) => (
+                    <TableCell 
+                      key={`${rowIndex}-${column.key}`}
+                      className="overflow-hidden"
+                    >
+                      <div className="truncate">
+                        {row[colIndex] || ""}
+                      </div>
+                    </TableCell>
+                  ))}
+                </TableRow>
+              ))}
+            </RadioGroup>
+          </TableBody>
+        </Table>
+      </div>
+    </div>
+  )
+}