Fix up validation step to show proper components

.gitignore

@@ -26,6 +26,7 @@ dist-ssr
 dashboard/build/**
 dashboard-server/frontend/build/**
 **/build/**
+.fuse_hidden**
 ._*
 
 # Build directories

docs/calculate-issues.md

@@ -0,0 +1,185 @@
+1. **Missing Updates for Reorder Point and Safety Stock** [RESOLVED - product-metrics.js]
+- **Problem:** In the **product_metrics** table (used by the inventory health view), the fields **reorder_point** and **safety_stock** are never updated in the product metrics calculations. Although a helper function (`calculateReorderQuantities`) exists and computes these values, the update query in the `calculateProductMetrics` function does not assign any values to these columns.
+- **Effect:** The inventory health view relies on these fields (using COALESCE to default them to 0), which means that stock might never be classified as "Reorder" or "Healthy" based on the proper reorder point or safety stock calculations.
+- **Example:** Even if a product's base metrics would require a reorder (for example, if its days of inventory are low), the view always shows a value of 0 for reorder_point and safety_stock.
+- **Fix:** Update the product metrics query (or add a subsequent update) so that **pm.reorder_point** and **pm.safety_stock** are calculated (for instance, by integrating the logic from `calculateReorderQuantities`) and stored in the table.
+
+2. **Overwritten Module Exports When Combining Scripts** [RESOLVED - calculate-metrics.js]
+- **Problem:** The code provided shows two distinct exports. The main metrics calculation module exports `calculateMetrics` (along with cancel and getProgress helpers), but later in the same concatenated file the module exports are overwritten.
+- **Effect:** If these two code sections end up in a single module file, the export for the main calculation will be lost. This would break any code that calls the overall metrics calculation.
+- **Example:** An external caller expecting to run `calculateMetrics` would instead receive the `calculateProductMetrics` function.
+- **Fix:** Make sure each script resides in its own module file. Verify that the module boundaries and exports are not accidentally merged or overwritten when deployed.
+
+3. **Potential Formula Issue in EOQ Calculation (Reorder Qty)** [RESOLVED - product-metrics.js]
+- **Problem:** The helper function `calculateReorderQuantities` uses an EOQ formula with a holding cost expressed as a percentage (0.25) rather than a per‐unit cost.
+- **Effect:** If the intent was to use the traditional EOQ formula (which expects a holding cost per unit rather than a percentage), this could lead to an incorrect reorder quantity.
+- **Example:** For a given annual demand and fixed order cost, the computed reorder quantity might be higher or lower than expected.
+- **Fix:** Double-check the EOQ formula. If the intention is to compute based on a percentage, then document that clearly; otherwise, adjust the formula to use the proper holding cost value.
+
+4. **Potential Overlap or Redundancy in GMROI Calculation** [RESOLVED - time-aggregates.js]
+- **Problem:** In the time aggregates function, GMROI is calculated in two steps. The initial INSERT query computes GMROI as
+
+	`CASE WHEN s.inventory_value > 0 THEN (s.total_revenue - s.total_cost) / s.inventory_value ELSE 0 END`
+
+	and then a subsequent UPDATE query recalculates it as an annualized value using gross profit and active days.
+
+
+- **Effect:** Overwriting a computed value may be intentional to refine the metric, but if not coordinated it can cause confusion or unexpected output in the `product_time_aggregates` table.
+- **Example:** A product's GMROI might first appear as a simple ratio but then be updated to a scaled value based on the number of active days, which could lead to inconsistent reporting if not documented.
+- **Fix:** Consolidated the GMROI calculation into a single step in the initial INSERT query, properly handling annualization and NULL values.
+
+5. **Handling of Products Without Orders or Purchase Data** [RESOLVED - time-aggregates.js]
+- **Problem:** In the INSERT query of the time aggregates function, the UNION covers two cases: one for products with order data (from `monthly_sales`) and one for products that have entries in `monthly_stock` but no matching order data.
+- **Effect:** If a product has neither orders nor purchase orders, it won't get an entry in `product_time_aggregates`. Depending on business rules, this might be acceptable or might mean missing data.
+- **Example:** A product that's new or rarely ordered might not appear in the time aggregates view, potentially affecting downstream calculations.
+- **Fix:** Added an `all_products` CTE and modified the JOIN structure to ensure every product gets an entry with appropriate default values, even if it has no orders or purchase orders.
+
+6. **Redundant Recalculation of Vendor Metrics**
+- **Problem:** Similar concepts from prior scripts where cumulative metrics (like **total_revenue** and **total_cost**) are calculated in multiple query steps without necessary validation or optimization. In the vendor metrics script, calculations for total revenue and margin are performed within a `WITH` clause, which is then used in other parts of the process, making it more complex than needed.
+- **Effect:** There's unnecessary duplication in querying the same data multiple times across subqueries. It could result in decreased performance and may even lead to excess computation if the subqueries are not optimized or correctly indexed.
+- **Example:** Vendor sales and vendor purchase orders (PO) metrics are calculated in separate `WITH` clauses, leading to repeated calculations.
+- **Fix:** Synthesize the required metrics into fewer queries or reuse the results within the `WITH` clause itself. Avoid redundant calculations of **revenue** and **cost** unless truly necessary.
+
+7. **Handling Products Without Orders or Purchase Orders**
+- **Problem:** In your `calculateVendorMetrics` script, the initial insert for vendor sales doesn't fully address the products that might not have matching orders or purchase orders. If a vendor has products without any sales within the last 12 months, the results may not be fully accurate unless handled explicitly.
+- **Effect:** If no orders exist for a product associated with a particular vendor, that product will not contribute to the vendor's metrics, potentially omitting important data when calculating **total_orders** or **total_revenue**.
+- **Example:** The scripted statistics fill gaps, but products with no recent purchase or sales orders might not be counted accurately.
+- **Fix:** Include logic to handle scenarios where these products still need to be part of the vendor calculation. Use a `LEFT JOIN` wherever possible to account for cases without sales or purchase orders.
+
+8. **Redundant `ON DUPLICATE KEY UPDATE`**
+- **Problem:** Multiple queries in the `calculateVendorMetrics` script use `ON DUPLICATE KEY UPDATE` clauses to handle repeated metrics updates. This is useful for ensuring the most up-to-date calculations but can cause inconsistencies if multiple calculations happen for the same product or vendor simultaneously.
+- **Effect:** This approach can lead to an inaccurate update of brand-specific data when insertion and update overlap. Each time you add a new batch, an existing entry could be overwritten if not handled correctly.
+- **Example:** Vendor country, category, or sales-related metrics could unintentionally update during processing.
+- **Fix:** Match on current status more robustly in case of existing rows to avoid unnecessary updates. Ensure that the key used for `ON DUPLICATE KEY` aligns with any foreign key relationships that might indicate an already processed entry.
+
+9. **SQL Query Performance with Multiple Nested `WITH` Clauses**
+- **Problem:** Heavily nested queries (especially **WITH** clauses) may lead to slow performance depending on the size of the dataset.
+- **Effect:** Computational burden could be high when the database is large, e.g., querying **purchase orders**, **vendor sales**, and **product info** simultaneously. Even with proper indexes, the deployment might struggle in production environments.
+- **Example:** Multiple `WITH` clauses in the vendor and brand metrics calculation scripts might work fine in small datasets but degrade performance in production.
+- **Fix:** Combine some subqueries and reduce the layer of computations needed for calculating final metrics. Test performance on a production-sized dataset to see how nested queries are handled.
+
+10. **Missing Updates for Reorder Metrics (Vendor/Brand)**
+- **Previously Identified Issue:** Inconsistent updates for **reorder_point** and **safety_stock** across earlier scripts.
+- **Current Impact on This Script:** The vendor and brand metrics do not have explicit updates for reorder point or safety stock, which are essential for inventory evaluation.
+- **Effect:** The correct thresholds and reorder logic for vendor product inventory aren't fully accounted for in these scripts.
+- **Fix:** Integrate relevant logic to update **reorder_point** or **safety_stock** within the vendor and brand metrics calculations. Ensure that it's consistently computed and stored.
+
+11. **Data Integrity and Consistency** 
+
+**w**hen tracking sales growth or performance
+
+
+- **Problem:** Brand metrics include a sales growth clause where negative results can sometimes be skewed severely if period data varies considerably.
+- **Effect:** If period boundaries are incorrect or records are missing, this can create drastic growth rate calculations.
+- **Example:** If the "previous" period has no sales but "current" has a substantial increase, the growth rate will show as **100%**.
+- **Fix:** Implement checks that ensure both periods are valid and that the system calculates growth accurately, avoiding growth rates based solely on potential outliers. Replace consistent gaps with a no-growth rate or a meaningful zero.
+
+12. **Exclusion of Vendors With No Sales**
+
+	The vendor metrics query is driven by the `vendor_sales` CTE, which aggregates data only for vendors that have orders in the past 12 months.
+
+
+	- **Impact:** Vendors that have purchase activity (or simply exist in vendor_details) but no recent sales won't show up in vendor_metrics. This could cause the frontend to miss metrics for vendors that might still be important.
+	- **Fix:** Consider adding a UNION or changing the driving set so that all vendors (for example, from vendor_details) are included—even if they have zero sales.
+13. **Identical Formulas for On-Time Delivery and Order Fill Rates**
+
+	Both metrics are calculated as `(received_orders / total_orders) * 100`.
+
+
+	- **Impact:** If the business expects these to be distinct (for example, one might factor in on-time receipt versus mere receipt), then showing identical values on the frontend could be misleading.
+	- **Fix:** Verify and adjust the formulas if on-time delivery and order fill rates should be computed differently.
+14. **Handling Nulls and Defaults in Aggregations**
+
+	The query uses COALESCE in most places, but be sure that every aggregated value (like average lead time) correctly defaults when no data is present.
+
+
+	- **Impact:** Incorrect defaults might cause odd or missing numbers on the production interface.
+	- **Fix:** Double-check that all numeric aggregates reliably default to 0 where needed.
+
+15. **Inconsistent Stock Filtering Conditions**
+
+	In the main brand metrics query the CTE filters products with the condition
+
+	`p.stock_quantity <= 5000 AND p.stock_quantity >= 0`
+
+	whereas in the brand time-based metrics query the condition is only `p.stock_quantity <= 5000`.
+
+
+	- **Impact:** This discrepancy may lead to inconsistent numbers (for example, if any products have negative stock, which might be due to data issues) between overall brand metrics and time-based metrics on the frontend.
+	- **Fix:** Standardize the filtering criteria so that both queries treat out-of-range stock values in the same way.
+16. **Growth Rate Calculation Periods**
+
+	The growth rate is computed by comparing revenue from the last 3 months ("current") against a period from 15–12 months ago ("previous").
+
+
+	- **Impact:** This narrow window may not reflect typical year-over-year performance and could lead to volatile or unexpected growth percentages on the frontend.
+	- **Fix:** Revisit the business logic for growth—if a longer or different comparison period is preferred, adjust the date intervals accordingly.
+17. **Potential NULLs in Aggregated Time-Based Metrics**
+
+	In the brand time-based metrics query, aggregate expressions such as `SUM(o.quantity * o.price)` aren't wrapped with COALESCE.
+
+
+	- **Impact:** If there are no orders for a given brand/month, these sums might return NULL rather than 0, which could propagate into the frontend display.
+	- **Fix:** Wrap such aggregates in COALESCE (e.g. `COALESCE(SUM(o.quantity * o.price), 0)`) to ensure a default numeric value.
+
+18. **Grouping by Category Status in Base Metrics Insert**
+	- **Problem:** The INSERT for base category metrics groups by both `c.cat_id` and `c.status` even though the table's primary key is just `category_id`.
+	- **Effect:** If a category's status changes over time, the grouping may produce unexpected updates (or even multiple groups before the duplicate key update kicks in), possibly causing the wrong status or aggregated figures to be stored.
+	- **Example:** A category that toggles between "active" and "inactive" might have its metrics calculated differently on different runs.
+	- **Fix:** Ensure that the grouping keys match the primary key (or that the status update logic is exactly as intended) so that a single row per category is maintained.
+19. **Potential Null Handling in Margin Calculations**
+	- **Problem:** In the query for category time metrics, the calculation of average margin uses expressions such as `SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0)))` without using `COALESCE` on `p.cost_price`.
+	- **Effect:** If any product's `cost_price` is `NULL`, then `GREATEST(p.cost_price, 0)` returns `NULL` and the resulting sum (and thus the margin) could become `NULL` rather than defaulting to 0. This might lead to missing or misleading margin figures on the frontend.
+	- **Example:** A product with a missing cost price would make the entire margin expression evaluate to `NULL` even when sales exist.
+	- **Fix:** Replace `GREATEST(p.cost_price, 0)` with `GREATEST(COALESCE(p.cost_price, 0), 0)` (or simply use `COALESCE(p.cost_price, 0)`) to ensure that missing values are handled.
+20. **Data Coverage in Growth Rate Calculation**
+	- **Problem:** The growth rate update depends on multiple CTEs (current period, previous period, and trend analysis) that require a minimum amount of data (for instance, `HAVING COUNT(*) >= 6` in the trend_stats CTE).
+	- **Effect:** Categories with insufficient historical data will fall into the "ELSE" branch (or may even be skipped if no revenue is present), which might result in a growth rate of 0.0 or an unexpected value.
+	- **Example:** A newly created category that has only two months of data won't have trend analysis, so its growth rate will be calculated solely by the simple difference, which might not reflect true performance.
+	- **Fix:** Confirm that this fallback behavior is acceptable for production; if not, adjust the logic so that every category receives a consistent growth rate even with sparse data.
+21. **Omission of Forecasts for Zero–Sales Categories**
+	- **Observation:** The category–sales metrics query uses a `HAVING AVG(cs.daily_quantity) > 0` clause.
+	- **Effect:** Categories without any average daily sales will not receive a forecast record in `category_sales_metrics`. If the frontend expects a row (even with zeros) for every category, this will lead to missing data.
+	- **Fix:** Verify that it's acceptable for categories with no sales to have no forecast entry. If not, adjust the query so that a default forecast (with zeros) is inserted.
+
+22. **Randomness in Category-Level Forecast Revenue Calculation**
+	- **Problem:** In the category-level forecasts query, the forecast revenue is multiplied by a factor of `(0.95 + (RAND() * 0.1))`.
+	- **Effect:** This introduces randomness into the forecast figures so that repeated runs could yield slightly different values. If deterministic forecasts are expected on the production frontend, this could lead to inconsistent displays.
+	- **Example:** The same category might show a 5% higher forecast on one run and 3% on another because of the random multiplier.
+	- **Fix:** Confirm that this randomness is intentional for your forecasting model; if forecasts are meant to be reproducible, remove or replace the `RAND()` factor with a fixed multiplier.
+23. **Multi-Statement Cleanup of Temporary Tables**
+	- **Problem:** The cleanup query drops multiple temporary tables in one call (separated by semicolons).
+	- **Effect:** If your Node.js MySQL driver isn't configured to allow multi-statement execution, this query may fail, leaving temporary tables behind. Leftover temporary tables might eventually cause conflicts or resource issues.
+	- **Example:** Running the cleanup query could produce an error like "multi-statement queries not enabled," preventing proper cleanup.
+	- **Fix:** Either configure your database connection to allow multi-statements or issue separate queries for each temporary table drop to ensure that the cleanup runs successfully.
+24. **Handling Products with No Sales Data**
+	- **Problem:** In the product-level forecast calculation, the CTE `daily_stats` includes a `HAVING AVG(ds.daily_quantity) > 0` clause.
+	- **Effect:** Products that have no sales (or a zero average daily quantity) will be excluded from the forecasts. This means the frontend won't show forecasts for non–selling products, which might be acceptable but could also be a completeness issue.
+	- **Example:** A product that has never sold will not appear in the `sales_forecasts` table.
+	- **Fix:** Confirm that it is intended for forecasts to be generated only for products with some sales activity. If forecasts are required for all products, adjust the query to insert default forecast records for products with zero sales.
+25. **Complexity of the Forecast Formula Involving the Seasonality Factor**
+- **Issue:**
+
+	The sales forecast calculations incorporate an adjustment factor using `COALESCE(sf.seasonality_factor, 0)` to modify forecast units and revenue. This means that if the seasonality data is missing (or not populated), the factor defaults to 0.
+
+
+- **Potential Problem:**
+
+	A default value of 0 will drastically alter the forecast calculations—often leading to a forecast of 0 or an overly dampened forecast—when in reality the intended behavior might be to use a neutral multiplier (typically 1.0). This could result in forecasts that are not reflective of the actual seasonal impact, thereby skewing the figures that reach the frontend.
+
+
+- **Fix:**
+
+	Review your data source for seasonality (the `sales_seasonality` table) and ensure it's consistently populated. Alternatively, if missing seasonality data is possible, consider using a more neutral default (such as 1.0) in your COALESCE. This change would prevent the forecast formulas from over-simplifying (or even nullifying) the forecast output due to missing seasonality factors.
+
+
+26. **Group By with Seasonality Factor Variability**
+	- **Observation:** In the forecast insertion query, the GROUP BY clause includes `sf.seasonality_factor` along with other fields.
+	- **Effect:** If the seasonality factor differs (or is `NULL` versus a value) for different forecast dates, this might result in multiple rows for the same product and forecast date. However, the `ON DUPLICATE KEY UPDATE` clause will merge them—but only if the primary key (pid, forecast_date) is truly unique.
+	- **Fix:** Verify that the grouping produces exactly one row per product per forecast date. If there's potential for multiple rows due to seasonality variability, consider applying a COALESCE or an aggregation on the seasonality factor so that it does not affect grouping.
+
+27. **Memory Management for Temporary Tables** [RESOLVED - calculate-metrics.js]
+- **Problem:** In metrics calculations, temporary tables aren't always properly cleaned up if the process fails between creation and the DROP statement.
+- **Effect:** If a process fails after creating temporary tables but before dropping them, these tables remain in memory until the connection is closed. In a production environment with multiple calculation runs, this could lead to memory leaks or table name conflicts.
+- **Example:** The `temp_revenue_ranks` table creation in ABC classification could remain if the process fails before reaching the DROP statement.
+- **Fix:** Implement proper cleanup in a finally block or use transaction management that ensures temporary tables are always cleaned up, even in failure scenarios.

inventory-server/db/config-schema.sql

@@ -171,6 +171,39 @@ ORDER BY
     c.name,
     st.vendor;
 
+CREATE TABLE IF NOT EXISTS calculate_history (
+    id BIGINT AUTO_INCREMENT PRIMARY KEY,
+    start_time TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    end_time TIMESTAMP NULL,
+    duration_seconds INT,
+    duration_minutes DECIMAL(10,2) GENERATED ALWAYS AS (duration_seconds / 60.0) STORED,
+    total_products INT DEFAULT 0,
+    total_orders INT DEFAULT 0,
+    total_purchase_orders INT DEFAULT 0,
+    processed_products INT DEFAULT 0,
+    processed_orders INT DEFAULT 0,
+    processed_purchase_orders INT DEFAULT 0,
+    status ENUM('running', 'completed', 'failed', 'cancelled') DEFAULT 'running',
+    error_message TEXT,
+    additional_info JSON,
+    INDEX idx_status_time (status, start_time)
+);
+
+CREATE TABLE IF NOT EXISTS calculate_status (
+    module_name ENUM(
+        'product_metrics',
+        'time_aggregates',
+        'financial_metrics',
+        'vendor_metrics',
+        'category_metrics',
+        'brand_metrics',
+        'sales_forecasts',
+        'abc_classification'
+    ) PRIMARY KEY,
+    last_calculation_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    INDEX idx_last_calc (last_calculation_timestamp)
+);
+
 CREATE TABLE IF NOT EXISTS sync_status (
     table_name VARCHAR(50) PRIMARY KEY,
     last_sync_timestamp TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,

inventory-server/db/metrics-schema.sql

@@ -102,19 +102,17 @@ CREATE TABLE IF NOT EXISTS product_time_aggregates (
     INDEX idx_date (year, month)
 );
 
--- Create vendor details table
-CREATE TABLE IF NOT EXISTS vendor_details (
-    vendor VARCHAR(100) NOT NULL,
+-- Create vendor_details table
+CREATE TABLE vendor_details (
+    vendor VARCHAR(100) PRIMARY KEY,
     contact_name VARCHAR(100),
-    email VARCHAR(100),
-    phone VARCHAR(20),
+    email VARCHAR(255),
+    phone VARCHAR(50),
     status VARCHAR(20) DEFAULT 'active',
-    notes TEXT,
-    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-    PRIMARY KEY (vendor),
-    INDEX idx_vendor_status (status)
-);
+    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
+    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
+    INDEX idx_status (status)
+) ENGINE=InnoDB;
 
 -- New table for vendor metrics
 CREATE TABLE IF NOT EXISTS vendor_metrics (
@@ -411,20 +409,3 @@ LEFT JOIN
 
 -- Re-enable foreign key checks
 SET FOREIGN_KEY_CHECKS = 1;
-
--- Create table for sales seasonality factors
-CREATE TABLE IF NOT EXISTS sales_seasonality (
-    month INT NOT NULL,
-    seasonality_factor DECIMAL(5,3) DEFAULT 0,
-    last_updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
-    PRIMARY KEY (month),
-    CHECK (month BETWEEN 1 AND 12),
-    CHECK (seasonality_factor BETWEEN -1.0 AND 1.0)
-);
-
--- Insert default seasonality factors (neutral)
-INSERT INTO sales_seasonality (month, seasonality_factor)
-VALUES 
-    (1, 0), (2, 0), (3, 0), (4, 0), (5, 0), (6, 0),
-    (7, 0), (8, 0), (9, 0), (10, 0), (11, 0), (12, 0)
-ON DUPLICATE KEY UPDATE last_updated = CURRENT_TIMESTAMP; 

inventory-server/db/schema.sql

@@ -51,13 +51,15 @@ CREATE TABLE products (
     baskets INT UNSIGNED DEFAULT 0,
     notifies INT UNSIGNED DEFAULT 0,
     date_last_sold DATE,
+    updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     PRIMARY KEY (pid),
     INDEX idx_sku (SKU),
     INDEX idx_vendor (vendor),
     INDEX idx_brand (brand),
     INDEX idx_location (location),
     INDEX idx_total_sold (total_sold),
-    INDEX idx_date_last_sold (date_last_sold)
+    INDEX idx_date_last_sold (date_last_sold),
+    INDEX idx_updated (updated)
 ) ENGINE=InnoDB;
 
 -- Create categories table with hierarchy support
@@ -77,18 +79,6 @@ CREATE TABLE categories (
     INDEX idx_name_type (name, type)
 ) ENGINE=InnoDB;
 
--- Create vendor_details table
-CREATE TABLE vendor_details (
-    vendor VARCHAR(100) PRIMARY KEY,
-    contact_name VARCHAR(100),
-    email VARCHAR(255),
-    phone VARCHAR(50),
-    status VARCHAR(20) DEFAULT 'active',
-    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
-    INDEX idx_status (status)
-) ENGINE=InnoDB;
-
 -- Create product_categories junction table
 CREATE TABLE product_categories (
     cat_id BIGINT NOT NULL,
@@ -118,6 +108,7 @@ CREATE TABLE IF NOT EXISTS orders (
     customer_name VARCHAR(100),
     status VARCHAR(20) DEFAULT 'pending',
     canceled TINYINT(1) DEFAULT 0,
+    updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     PRIMARY KEY (id),
     UNIQUE KEY unique_order_line (order_number, pid),
     KEY order_number (order_number),
@@ -125,7 +116,8 @@ CREATE TABLE IF NOT EXISTS orders (
     KEY customer (customer),
     KEY date (date),
     KEY status (status),
-    INDEX idx_orders_metrics (pid, date, canceled)
+    INDEX idx_orders_metrics (pid, date, canceled),
+    INDEX idx_updated (updated)
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
 
 -- Create purchase_orders table with its indexes
@@ -148,8 +140,9 @@ CREATE TABLE purchase_orders (
     received INT DEFAULT 0,
     received_date DATE COMMENT 'Date of first receiving',
     last_received_date DATE COMMENT 'Date of most recent receiving',
-    received_by INT,
+    received_by VARCHAR(100) COMMENT 'Name of person who first received this PO line',
     receiving_history JSON COMMENT 'Array of receiving records with qty, date, cost, receiving_id, and alt_po flag',
+    updated TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
     FOREIGN KEY (pid) REFERENCES products(pid),
     INDEX idx_po_id (po_id),
     INDEX idx_vendor (vendor),
@@ -159,6 +152,7 @@ CREATE TABLE purchase_orders (
     INDEX idx_po_metrics (pid, date, receiving_status, received_date),
     INDEX idx_po_product_date (pid, date),
     INDEX idx_po_product_status (pid, status),
+    INDEX idx_updated (updated),
     UNIQUE KEY unique_po_product (po_id, pid)
 ) ENGINE=InnoDB;
 

inventory-server/scripts/calculate-metrics.js

@@ -7,13 +7,13 @@ require('dotenv').config({ path: path.resolve(__dirname, '..', '.env') });
 
 // Configuration flags for controlling which metrics to calculate
 // Set to 1 to skip the corresponding calculation, 0 to run it
-const SKIP_PRODUCT_METRICS = 1;          // Skip all product metrics
-const SKIP_TIME_AGGREGATES = 1;          // Skip time aggregates
-const SKIP_FINANCIAL_METRICS = 1;        // Skip financial metrics
-const SKIP_VENDOR_METRICS = 1;           // Skip vendor metrics
-const SKIP_CATEGORY_METRICS = 1;         // Skip category metrics
-const SKIP_BRAND_METRICS = 1;            // Skip brand metrics
-const SKIP_SALES_FORECASTS = 1;          // Skip sales forecasts
+const SKIP_PRODUCT_METRICS = 0;
+const SKIP_TIME_AGGREGATES = 0;
+const SKIP_FINANCIAL_METRICS = 0;
+const SKIP_VENDOR_METRICS = 0;
+const SKIP_CATEGORY_METRICS = 0;
+const SKIP_BRAND_METRICS = 0;
+const SKIP_SALES_FORECASTS = 0;
 
 // Add error handler for uncaught exceptions
 process.on('uncaughtException', (error) => {
@@ -44,6 +44,34 @@ global.clearProgress = progress.clearProgress;
 global.getProgress = progress.getProgress;
 global.logError = progress.logError;
 
+// List of temporary tables used in the calculation process
+const TEMP_TABLES = [
+    'temp_revenue_ranks',
+    'temp_sales_metrics',
+    'temp_purchase_metrics',
+    'temp_product_metrics',
+    'temp_vendor_metrics',
+    'temp_category_metrics',
+    'temp_brand_metrics',
+    'temp_forecast_dates',
+    'temp_daily_sales',
+    'temp_product_stats',
+    'temp_category_sales',
+    'temp_category_stats'
+];
+
+// Add cleanup function for temporary tables
+async function cleanupTemporaryTables(connection) {
+    try {
+        for (const table of TEMP_TABLES) {
+            await connection.query(`DROP TEMPORARY TABLE IF EXISTS ${table}`);
+        }
+    } catch (error) {
+        logError(error, 'Error cleaning up temporary tables');
+        throw error; // Re-throw to be handled by the caller
+    }
+}
+
 const { getConnection, closePool } = require('./metrics/utils/db');
 const calculateProductMetrics = require('./metrics/product-metrics');
 const calculateTimeAggregates = require('./metrics/time-aggregates');
@@ -83,10 +111,78 @@ process.on('SIGTERM', cancelCalculation);
 async function calculateMetrics() {
     let connection;
     const startTime = Date.now();
-    let processedCount = 0;
+    let processedProducts = 0;
+    let processedOrders = 0;
+    let processedPurchaseOrders = 0;
     let totalProducts = 0;
+    let totalOrders = 0;
+    let totalPurchaseOrders = 0;
+    let calculateHistoryId;
     
     try {
+        // Clean up any previously running calculations
+        connection = await getConnection();
+        await connection.query(`
+            UPDATE calculate_history 
+            SET 
+                status = 'cancelled',
+                end_time = NOW(),
+                duration_seconds = TIMESTAMPDIFF(SECOND, start_time, NOW()),
+                error_message = 'Previous calculation was not completed properly'
+            WHERE status = 'running'
+        `);
+
+        // Get counts from all relevant tables
+        const [[productCount], [orderCount], [poCount]] = await Promise.all([
+            connection.query('SELECT COUNT(*) as total FROM products'),
+            connection.query('SELECT COUNT(*) as total FROM orders'),
+            connection.query('SELECT COUNT(*) as total FROM purchase_orders')
+        ]);
+
+        totalProducts = productCount.total;
+        totalOrders = orderCount.total;
+        totalPurchaseOrders = poCount.total;
+
+        // Create history record for this calculation
+        const [historyResult] = await connection.query(`
+            INSERT INTO calculate_history (
+                start_time,
+                status,
+                total_products,
+                total_orders,
+                total_purchase_orders,
+                additional_info
+            ) VALUES (
+                NOW(),
+                'running',
+                ?,
+                ?,
+                ?,
+                JSON_OBJECT(
+                    'skip_product_metrics', ?,
+                    'skip_time_aggregates', ?,
+                    'skip_financial_metrics', ?,
+                    'skip_vendor_metrics', ?,
+                    'skip_category_metrics', ?,
+                    'skip_brand_metrics', ?,
+                    'skip_sales_forecasts', ?
+                )
+            )
+        `, [
+            totalProducts,
+            totalOrders,
+            totalPurchaseOrders,
+            SKIP_PRODUCT_METRICS,
+            SKIP_TIME_AGGREGATES,
+            SKIP_FINANCIAL_METRICS, 
+            SKIP_VENDOR_METRICS,
+            SKIP_CATEGORY_METRICS,
+            SKIP_BRAND_METRICS,
+            SKIP_SALES_FORECASTS
+        ]);
+        calculateHistoryId = historyResult.insertId;
+        connection.release();
+
         // Add debug logging for the progress functions
         console.log('Debug - Progress functions:', {
             formatElapsedTime: typeof global.formatElapsedTime,
@@ -115,72 +211,150 @@ async function calculateMetrics() {
                 elapsed: '0s',
                 remaining: 'Calculating...',
                 rate: 0,
-                percentage: '0'
+                percentage: '0',
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            // Get total number of products
-            const [countResult] = await connection.query('SELECT COUNT(*) as total FROM products')
-                .catch(err => {
-                    global.logError(err, 'Failed to count products');
-                    throw err;
+            // Update progress periodically
+            const updateProgress = async (products = null, orders = null, purchaseOrders = null) => {
+                // Ensure all values are valid numbers or default to previous value
+                if (products !== null) processedProducts = Number(products) || processedProducts || 0;
+                if (orders !== null) processedOrders = Number(orders) || processedOrders || 0;
+                if (purchaseOrders !== null) processedPurchaseOrders = Number(purchaseOrders) || processedPurchaseOrders || 0;
+
+                // Ensure we never send NaN to the database
+                const safeProducts = Number(processedProducts) || 0;
+                const safeOrders = Number(processedOrders) || 0;
+                const safePurchaseOrders = Number(processedPurchaseOrders) || 0;
+
+                await connection.query(`
+                    UPDATE calculate_history 
+                    SET 
+                        processed_products = ?,
+                        processed_orders = ?,
+                        processed_purchase_orders = ?
+                    WHERE id = ?
+                `, [safeProducts, safeOrders, safePurchaseOrders, calculateHistoryId]);
+            };
+
+            // Helper function to ensure valid progress numbers
+            const ensureValidProgress = (current, total) => ({
+                current: Number(current) || 0,
+                total: Number(total) || 1,  // Default to 1 to avoid division by zero
+                percentage: (((Number(current) || 0) / (Number(total) || 1)) * 100).toFixed(1)
+            });
+
+            // Initial progress
+            const initialProgress = ensureValidProgress(0, totalProducts);
+            global.outputProgress({
+                status: 'running',
+                operation: 'Starting metrics calculation',
+                current: initialProgress.current,
+                total: initialProgress.total,
+                elapsed: '0s',
+                remaining: 'Calculating...',
+                rate: 0,
+                percentage: initialProgress.percentage,
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
-            totalProducts = countResult[0].total;
 
             if (!SKIP_PRODUCT_METRICS) {
-                processedCount = await calculateProductMetrics(startTime, totalProducts);
+                const result = await calculateProductMetrics(startTime, totalProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Product metrics calculation failed');
+                }
             } else {
                 console.log('Skipping product metrics calculation...');
-                processedCount = Math.floor(totalProducts * 0.6);
+                processedProducts = Math.floor(totalProducts * 0.6);
+                await updateProgress(processedProducts);
                 global.outputProgress({
                     status: 'running',
                     operation: 'Skipping product metrics calculation',
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts,
                     elapsed: global.formatElapsedTime(startTime),
-                    remaining: global.estimateRemaining(startTime, processedCount, totalProducts),
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: '60'
+                    remaining: global.estimateRemaining(startTime, processedProducts, totalProducts),
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: '60',
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             }
 
             // Calculate time-based aggregates
             if (!SKIP_TIME_AGGREGATES) {
-                processedCount = await calculateTimeAggregates(startTime, totalProducts, processedCount);
+                const result = await calculateTimeAggregates(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Time aggregates calculation failed');
+                }
             } else {
                 console.log('Skipping time aggregates calculation');
             }
 
             // Calculate financial metrics
             if (!SKIP_FINANCIAL_METRICS) {
-                processedCount = await calculateFinancialMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateFinancialMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Financial metrics calculation failed');
+                }
             } else {
                 console.log('Skipping financial metrics calculation');
             }
             
             // Calculate vendor metrics
             if (!SKIP_VENDOR_METRICS) {
-                processedCount = await calculateVendorMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateVendorMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Vendor metrics calculation failed');
+                }
             } else {
                 console.log('Skipping vendor metrics calculation');
             }
 
             // Calculate category metrics
             if (!SKIP_CATEGORY_METRICS) {
-                processedCount = await calculateCategoryMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateCategoryMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Category metrics calculation failed');
+                }
             } else {
                 console.log('Skipping category metrics calculation');
             }
 
             // Calculate brand metrics
             if (!SKIP_BRAND_METRICS) {
-                processedCount = await calculateBrandMetrics(startTime, totalProducts, processedCount);
+                const result = await calculateBrandMetrics(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Brand metrics calculation failed');
+                }
             } else {
                 console.log('Skipping brand metrics calculation');
             }
 
             // Calculate sales forecasts
             if (!SKIP_SALES_FORECASTS) {
-                processedCount = await calculateSalesForecasts(startTime, totalProducts, processedCount);
+                const result = await calculateSalesForecasts(startTime, totalProducts, processedProducts);
+                await updateProgress(result.processedProducts, result.processedOrders, result.processedPurchaseOrders);
+                if (!result.success) {
+                    throw new Error('Sales forecasts calculation failed');
+                }
             } else {
                 console.log('Skipping sales forecasts calculation');
             }
@@ -189,15 +363,25 @@ async function calculateMetrics() {
             outputProgress({
                 status: 'running',
                 operation: 'Starting ABC classification',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
             const [abcConfig] = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
             const abcThresholds = abcConfig[0] || { a_threshold: 20, b_threshold: 50 };
@@ -218,15 +402,25 @@ async function calculateMetrics() {
             outputProgress({
                 status: 'running',
                 operation: 'Creating revenue rankings',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
             await connection.query(`
                 INSERT INTO temp_revenue_ranks
@@ -247,26 +441,44 @@ async function calculateMetrics() {
             // Get total count for percentage calculation
             const [rankingCount] = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
             const totalCount = rankingCount[0].total_count || 1;
+            const max_rank = totalCount; // Store max_rank for use in classification
 
             outputProgress({
                 status: 'running',
                 operation: 'Updating ABC classifications',
-                current: processedCount,
-                total: totalProducts,
+                current: processedProducts || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedProducts || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedProducts || 0),
+                percentage: (((processedProducts || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            if (isCancelled) return processedCount;
+            if (isCancelled) return {
+                processedProducts: processedProducts || 0,
+                processedOrders: processedOrders || 0,
+                processedPurchaseOrders: 0,
+                success: false
+            };
 
-            // Process updates in batches
+            // ABC classification progress tracking
             let abcProcessedCount = 0;
             const batchSize = 5000;
+            let lastProgressUpdate = Date.now();
+            const progressUpdateInterval = 1000; // Update every second
 
             while (true) {
-                if (isCancelled) return processedCount;
+                if (isCancelled) return {
+                    processedProducts: Number(processedProducts) || 0,
+                    processedOrders: Number(processedOrders) || 0,
+                    processedPurchaseOrders: 0,
+                    success: false
+                };
 
                 // First get a batch of PIDs that need updating
                 const [pids] = await connection.query(`
@@ -282,8 +494,8 @@ async function calculateMetrics() {
                             ELSE 'C'
                         END
                     LIMIT ?
-                `, [totalCount, abcThresholds.a_threshold,
-                    totalCount, abcThresholds.b_threshold,
+                `, [max_rank, abcThresholds.a_threshold,
+                    max_rank, abcThresholds.b_threshold,
                     batchSize]);
 
                 if (pids.length === 0) {
@@ -303,24 +515,43 @@ async function calculateMetrics() {
                         END,
                         pm.last_calculated_at = NOW()
                     WHERE pm.pid IN (?)
-                `, [totalCount, abcThresholds.a_threshold,
-                    totalCount, abcThresholds.b_threshold,
+                `, [max_rank, abcThresholds.a_threshold,
+                    max_rank, abcThresholds.b_threshold,
                     pids.map(row => row.pid)]);
 
                 abcProcessedCount += result.affectedRows;
-                processedCount = Math.floor(totalProducts * (0.99 + (abcProcessedCount / totalCount) * 0.01));
+                
+                // Calculate progress ensuring valid numbers
+                const currentProgress = Math.floor(totalProducts * (0.99 + (abcProcessedCount / (totalCount || 1)) * 0.01));
+                processedProducts = Number(currentProgress) || processedProducts || 0;
+
+                // Only update progress at most once per second
+                const now = Date.now();
+                if (now - lastProgressUpdate >= progressUpdateInterval) {
+                    const progress = ensureValidProgress(processedProducts, totalProducts);
                     
                     outputProgress({
                         status: 'running',
                         operation: 'ABC classification progress',
-                    current: processedCount,
-                    total: totalProducts,
+                        current: progress.current,
+                        total: progress.total,
                         elapsed: formatElapsedTime(startTime),
-                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                    rate: calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                        remaining: estimateRemaining(startTime, progress.current, progress.total),
+                        rate: calculateRate(startTime, progress.current),
+                        percentage: progress.percentage,
+                        timing: {
+                            start_time: new Date(startTime).toISOString(),
+                            end_time: new Date().toISOString(),
+                            elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                        }
                     });
                     
+                    lastProgressUpdate = now;
+                }
+
+                // Update database progress
+                await updateProgress(processedProducts, processedOrders, processedPurchaseOrders);
+
                 // Small delay between batches to allow other transactions
                 await new Promise(resolve => setTimeout(resolve, 100));
             }
@@ -328,61 +559,145 @@ async function calculateMetrics() {
             // Clean up
             await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
 
+            const endTime = Date.now();
+            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
+
+            // Update calculate_status for ABC classification
+            await connection.query(`
+                INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+                VALUES ('abc_classification', NOW())
+                ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+            `);
+
+            // Final progress update with guaranteed valid numbers
+            const finalProgress = ensureValidProgress(totalProducts, totalProducts);
+            
             // Final success message
             outputProgress({
                 status: 'complete',
                 operation: 'Metrics calculation complete',
-                current: totalProducts,
-                total: totalProducts,
+                current: finalProgress.current,
+                total: finalProgress.total,
                 elapsed: formatElapsedTime(startTime),
                 remaining: '0s',
-                rate: calculateRate(startTime, totalProducts),
-                percentage: '100'
+                rate: calculateRate(startTime, finalProgress.current),
+                percentage: '100',
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: totalElapsedSeconds
+                }
             });
 
+            // Ensure all values are valid numbers before final update
+            const finalStats = {
+                processedProducts: Number(processedProducts) || 0,
+                processedOrders: Number(processedOrders) || 0,
+                processedPurchaseOrders: Number(processedPurchaseOrders) || 0
+            };
+
+            // Update history with completion
+            await connection.query(`
+                UPDATE calculate_history 
+                SET 
+                    end_time = NOW(),
+                    duration_seconds = ?,
+                    processed_products = ?,
+                    processed_orders = ?,
+                    processed_purchase_orders = ?,
+                    status = 'completed'
+                WHERE id = ?
+            `, [totalElapsedSeconds, 
+                finalStats.processedProducts,
+                finalStats.processedOrders, 
+                finalStats.processedPurchaseOrders,
+                calculateHistoryId]);
+
             // Clear progress file on successful completion
             global.clearProgress();
 
         } catch (error) {
+            const endTime = Date.now();
+            const totalElapsedSeconds = Math.round((endTime - startTime) / 1000);
+
+            // Update history with error
+            await connection.query(`
+                UPDATE calculate_history 
+                SET 
+                    end_time = NOW(),
+                    duration_seconds = ?,
+                    processed_products = ?,
+                    processed_orders = ?,
+                    processed_purchase_orders = ?,
+                    status = ?,
+                    error_message = ?
+                WHERE id = ?
+            `, [
+                totalElapsedSeconds,
+                processedProducts || 0,  // Ensure we have a valid number
+                processedOrders || 0,    // Ensure we have a valid number
+                processedPurchaseOrders || 0,  // Ensure we have a valid number
+                isCancelled ? 'cancelled' : 'failed',
+                error.message,
+                calculateHistoryId
+            ]);
+
             if (isCancelled) {
                 global.outputProgress({
                     status: 'cancelled',
                     operation: 'Calculation cancelled',
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts || 0,
                     elapsed: global.formatElapsedTime(startTime),
                     remaining: null,
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / (totalProducts || 1)) * 100).toFixed(1)
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             } else {
                 global.outputProgress({
                     status: 'error',
                     operation: 'Error: ' + error.message,
-                    current: processedCount,
+                    current: processedProducts,
                     total: totalProducts || 0,
                     elapsed: global.formatElapsedTime(startTime),
                     remaining: null,
-                    rate: global.calculateRate(startTime, processedCount),
-                    percentage: ((processedCount / (totalProducts || 1)) * 100).toFixed(1)
+                    rate: global.calculateRate(startTime, processedProducts),
+                    percentage: ((processedProducts / (totalProducts || 1)) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
                 });
             }
             throw error;
         } finally {
             if (connection) {
+                // Ensure temporary tables are cleaned up
+                await cleanupTemporaryTables(connection);
                 connection.release();
             }
-        }
-    } finally {
             // Close the connection pool when we're done
             await closePool();
         }
+    } catch (error) {
+        success = false;
+        logError(error, 'Error in metrics calculation');
+        throw error;
+    }
 }
 
-// Export both functions and progress checker
-module.exports = calculateMetrics;
-module.exports.cancelCalculation = cancelCalculation;
-module.exports.getProgress = global.getProgress;
+// Export as a module with all necessary functions
+module.exports = {
+    calculateMetrics,
+    cancelCalculation,
+    getProgress: global.getProgress
+};
 
 // Run directly if called from command line
 if (require.main === module) {

inventory-server/scripts/full-reset.js

@@ -0,0 +1,107 @@
+const path = require('path');
+const { spawn } = require('child_process');
+
+function outputProgress(data) {
+    if (!data.status) {
+        data = {
+            status: 'running',
+            ...data
+        };
+    }
+    console.log(JSON.stringify(data));
+}
+
+function runScript(scriptPath) {
+    return new Promise((resolve, reject) => {
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe']
+        });
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const lines = data.toString().split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    console.log(line); // Pass through the JSON output
+                    output += line + '\n';
+                } catch (e) {
+                    console.log(line); // If not JSON, just log it directly
+                }
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            console.error(data.toString());
+        });
+
+        child.on('close', (code) => {
+            if (code !== 0) {
+                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
+            } else {
+                resolve(output);
+            }
+        });
+
+        child.on('error', (err) => {
+            reject(err);
+        });
+    });
+}
+
+async function fullReset() {
+    try {
+        // Step 1: Reset Database
+        outputProgress({
+            operation: 'Starting full reset',
+            message: 'Step 1/3: Resetting database...'
+        });
+        await runScript(path.join(__dirname, 'reset-db.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Database reset step complete',
+            message: 'Database reset finished, moving to import...'
+        });
+
+        // Step 2: Import from Production
+        outputProgress({
+            operation: 'Starting import',
+            message: 'Step 2/3: Importing from production...'
+        });
+        await runScript(path.join(__dirname, 'import-from-prod.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Import step complete',
+            message: 'Import finished, moving to metrics calculation...'
+        });
+
+        // Step 3: Calculate Metrics
+        outputProgress({
+            operation: 'Starting metrics calculation',
+            message: 'Step 3/3: Calculating metrics...'
+        });
+        await runScript(path.join(__dirname, 'calculate-metrics.js'));
+        
+        // Final completion message
+        outputProgress({
+            status: 'complete',
+            operation: 'Full reset complete',
+            message: 'Successfully completed all steps: database reset, import, and metrics calculation'
+        });
+    } catch (error) {
+        outputProgress({
+            status: 'error',
+            operation: 'Full reset failed',
+            error: error.message,
+            stack: error.stack
+        });
+        process.exit(1);
+    }
+}
+
+// Run if called directly
+if (require.main === module) {
+    fullReset();
+}
+
+module.exports = fullReset; 

inventory-server/scripts/full-update.js

@@ -0,0 +1,100 @@
+const path = require('path');
+const { spawn } = require('child_process');
+
+function outputProgress(data) {
+    if (!data.status) {
+        data = {
+            status: 'running',
+            ...data
+        };
+    }
+    console.log(JSON.stringify(data));
+}
+
+function runScript(scriptPath) {
+    return new Promise((resolve, reject) => {
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe']
+        });
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const lines = data.toString().split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    console.log(line); // Pass through the JSON output
+                    output += line + '\n';
+                } catch (e) {
+                    console.log(line); // If not JSON, just log it directly
+                }
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            console.error(data.toString());
+        });
+
+        child.on('close', (code) => {
+            if (code !== 0) {
+                reject(new Error(`Script ${scriptPath} exited with code ${code}`));
+            } else {
+                resolve(output);
+            }
+        });
+
+        child.on('error', (err) => {
+            reject(err);
+        });
+    });
+}
+
+async function fullUpdate() {
+    try {
+        // Step 1: Import from Production
+        outputProgress({
+            operation: 'Starting full update',
+            message: 'Step 1/2: Importing from production...'
+        });
+        await runScript(path.join(__dirname, 'import-from-prod.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Import step complete',
+            message: 'Import finished, moving to metrics calculation...'
+        });
+
+        // Step 2: Calculate Metrics
+        outputProgress({
+            operation: 'Starting metrics calculation',
+            message: 'Step 2/2: Calculating metrics...'
+        });
+        await runScript(path.join(__dirname, 'calculate-metrics.js'));
+        outputProgress({
+            status: 'complete',
+            operation: 'Metrics step complete',
+            message: 'Metrics calculation finished'
+        });
+
+        // Final completion message
+        outputProgress({
+            status: 'complete',
+            operation: 'Full update complete',
+            message: 'Successfully completed all steps: import and metrics calculation'
+        });
+    } catch (error) {
+        outputProgress({
+            status: 'error',
+            operation: 'Full update failed',
+            error: error.message,
+            stack: error.stack
+        });
+        process.exit(1);
+    }
+}
+
+// Run if called directly
+if (require.main === module) {
+    fullUpdate();
+}
+
+module.exports = fullUpdate; 

inventory-server/scripts/import/orders.js

@@ -28,9 +28,18 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
   let cumulativeProcessedOrders = 0;
 
   try {
-    // Insert temporary table creation queries
+    // Clean up any existing temp tables first
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_items (
+      DROP TEMPORARY TABLE IF EXISTS temp_order_items;
+      DROP TEMPORARY TABLE IF EXISTS temp_order_meta;
+      DROP TEMPORARY TABLE IF EXISTS temp_order_discounts;
+      DROP TEMPORARY TABLE IF EXISTS temp_order_taxes;
+      DROP TEMPORARY TABLE IF EXISTS temp_order_costs;
+    `);
+
+    // Create all temp tables with correct schema
+    await localConnection.query(`
+      CREATE TEMPORARY TABLE temp_order_items (
         order_id INT UNSIGNED NOT NULL,
         pid INT UNSIGNED NOT NULL,
         SKU VARCHAR(50) NOT NULL,
@@ -40,35 +49,41 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
         PRIMARY KEY (order_id, pid)
       ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
     `);
+
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_meta (
+      CREATE TEMPORARY TABLE temp_order_meta (
         order_id INT UNSIGNED NOT NULL,
         date DATE NOT NULL,
         customer VARCHAR(100) NOT NULL,
         customer_name VARCHAR(150) NOT NULL,
         status INT,
         canceled TINYINT(1),
+        summary_discount DECIMAL(10,2) DEFAULT 0.00,
+        summary_subtotal DECIMAL(10,2) DEFAULT 0.00,
         PRIMARY KEY (order_id)
       ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
     `);
+
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_discounts (
+      CREATE TEMPORARY TABLE temp_order_discounts (
         order_id INT UNSIGNED NOT NULL,
         pid INT UNSIGNED NOT NULL,
         discount DECIMAL(10,2) NOT NULL,
         PRIMARY KEY (order_id, pid)
       ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
     `);
+
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_taxes (
+      CREATE TEMPORARY TABLE temp_order_taxes (
         order_id INT UNSIGNED NOT NULL,
         pid INT UNSIGNED NOT NULL,
         tax DECIMAL(10,2) NOT NULL,
         PRIMARY KEY (order_id, pid)
       ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
     `);
+
     await localConnection.query(`
-      CREATE TABLE IF NOT EXISTS temp_order_costs (
+      CREATE TEMPORARY TABLE temp_order_costs (
         order_id INT UNSIGNED NOT NULL,
         pid INT UNSIGNED NOT NULL,
         costeach DECIMAL(10,3) DEFAULT 0.000,
@@ -81,6 +96,7 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
       SELECT COLUMN_NAME 
       FROM INFORMATION_SCHEMA.COLUMNS 
       WHERE TABLE_NAME = 'orders'
+        AND COLUMN_NAME != 'updated'  -- Exclude the updated column
       ORDER BY ORDINAL_POSITION
     `);
     const columnNames = columns.map(col => col.COLUMN_NAME);
@@ -212,7 +228,9 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
           o.order_cid as customer,
           CONCAT(COALESCE(u.firstname, ''), ' ', COALESCE(u.lastname, '')) as customer_name,
           o.order_status as status,
-          CASE WHEN o.date_cancelled != '0000-00-00 00:00:00' THEN 1 ELSE 0 END as canceled
+          CASE WHEN o.date_cancelled != '0000-00-00 00:00:00' THEN 1 ELSE 0 END as canceled,
+          o.summary_discount,
+          o.summary_subtotal
         FROM _order o
         LEFT JOIN users u ON o.order_cid = u.cid
         WHERE o.order_id IN (?)
@@ -226,19 +244,37 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
         console.log('Found duplicates:', duplicates);
       }
 
-      const placeholders = orders.map(() => "(?, ?, ?, ?, ?, ?)").join(",");
+      const placeholders = orders.map(() => "(?, ?, ?, ?, ?, ?, ?, ?)").join(",");
       const values = orders.flatMap(order => [
-        order.order_id, order.date, order.customer, order.customer_name, order.status, order.canceled
+        order.order_id, 
+        order.date, 
+        order.customer, 
+        order.customer_name, 
+        order.status, 
+        order.canceled,
+        order.summary_discount,
+        order.summary_subtotal
       ]);
 
       await localConnection.query(`
-        INSERT INTO temp_order_meta VALUES ${placeholders}
+        INSERT INTO temp_order_meta (
+          order_id, 
+          date, 
+          customer, 
+          customer_name, 
+          status, 
+          canceled,
+          summary_discount,
+          summary_subtotal
+        ) VALUES ${placeholders}
         ON DUPLICATE KEY UPDATE
           date = VALUES(date),
           customer = VALUES(customer),
           customer_name = VALUES(customer_name),
           status = VALUES(status),
-          canceled = VALUES(canceled)
+          canceled = VALUES(canceled),
+          summary_discount = VALUES(summary_discount),
+          summary_subtotal = VALUES(summary_subtotal)
       `, values);
 
       processedCount = i + orders.length;
@@ -317,14 +353,25 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
     for (let i = 0; i < orderIds.length; i += 5000) {
       const batchIds = orderIds.slice(i, i + 5000);
       const [costs] = await prodConnection.query(`
-        SELECT orderid as order_id, pid, costeach
-        FROM order_costs
-        WHERE orderid IN (?)
+        SELECT 
+          oc.orderid as order_id, 
+          oc.pid,
+          COALESCE(
+            oc.costeach,
+            (SELECT pi.costeach 
+             FROM product_inventory pi 
+             WHERE pi.pid = oc.pid 
+             AND pi.daterec <= o.date_placed
+             ORDER BY pi.daterec DESC LIMIT 1)
+          ) as costeach
+        FROM order_costs oc
+        JOIN _order o ON oc.orderid = o.order_id
+        WHERE oc.orderid IN (?)
       `, [batchIds]);
 
       if (costs.length > 0) {
         const placeholders = costs.map(() => '(?, ?, ?)').join(",");
-        const values = costs.flatMap(c => [c.order_id, c.pid, c.costeach]);
+        const values = costs.flatMap(c => [c.order_id, c.pid, c.costeach || 0]);
         await localConnection.query(`
           INSERT INTO temp_order_costs (order_id, pid, costeach)
           VALUES ${placeholders}
@@ -355,7 +402,13 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
           om.date,
           oi.price,
           oi.quantity,
-          oi.base_discount + COALESCE(od.discount, 0) as discount,
+          oi.base_discount + COALESCE(od.discount, 0) + 
+          CASE 
+            WHEN om.summary_discount > 0 THEN 
+              ROUND((om.summary_discount * (oi.price * oi.quantity)) / 
+                NULLIF(om.summary_subtotal, 0), 2)
+            ELSE 0 
+          END as discount,
           COALESCE(ot.tax, 0) as tax,
           0 as tax_included,
           0 as shipping,
@@ -455,7 +508,13 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
             om.date,
             oi.price,
             oi.quantity,
-            oi.base_discount + COALESCE(od.discount, 0) as discount,
+            oi.base_discount + COALESCE(od.discount, 0) + 
+            CASE 
+              WHEN o.summary_discount > 0 THEN 
+                ROUND((o.summary_discount * (oi.price * oi.quantity)) / 
+                  NULLIF(o.summary_subtotal, 0), 2)
+              ELSE 0 
+            END as discount,
             COALESCE(ot.tax, 0) as tax,
             0 as tax_included,
             0 as shipping,
@@ -466,6 +525,7 @@ async function importOrders(prodConnection, localConnection, incrementalUpdate =
             COALESCE(tc.costeach, 0) as costeach
           FROM temp_order_items oi
           JOIN temp_order_meta om ON oi.order_id = om.order_id
+          LEFT JOIN _order o ON oi.order_id = o.order_id
           LEFT JOIN temp_order_discounts od ON oi.order_id = od.order_id AND oi.pid = od.pid
           LEFT JOIN temp_order_taxes ot ON oi.order_id = ot.order_id AND oi.pid = ot.pid
           LEFT JOIN temp_order_costs tc ON oi.order_id = tc.order_id AND oi.pid = tc.pid

inventory-server/scripts/import/products.js

@@ -339,6 +339,7 @@ async function importProducts(prodConnection, localConnection, incrementalUpdate
       SELECT COLUMN_NAME 
       FROM INFORMATION_SCHEMA.COLUMNS 
       WHERE TABLE_NAME = 'products'
+        AND COLUMN_NAME != 'updated'  -- Exclude the updated column
       ORDER BY ORDINAL_POSITION
     `);
     const columnNames = columns.map(col => col.COLUMN_NAME);
@@ -470,7 +471,9 @@ async function importProducts(prodConnection, localConnection, incrementalUpdate
 
         // Process category relationships
         if (batch.some(p => p.category_ids)) {
-          const categoryRelationships = batch
+            // First get all valid categories
+            const allCategoryIds = [...new Set(
+                batch
                     .filter(p => p.category_ids)
                     .flatMap(product => 
                         product.category_ids
@@ -479,33 +482,92 @@ async function importProducts(prodConnection, localConnection, incrementalUpdate
                             .filter(id => id)
                             .map(Number)
                             .filter(id => !isNaN(id))
-                .map(catId => [catId, product.pid])
-            );
+                    )
+            )];
+
+            // Verify categories exist and get their hierarchy
+            const [categories] = await localConnection.query(`
+                WITH RECURSIVE category_hierarchy AS (
+                    SELECT 
+                        cat_id,
+                        parent_id,
+                        type,
+                        1 as level,
+                        CAST(cat_id AS CHAR(200)) as path
+                    FROM categories
+                    WHERE cat_id IN (?)
+                    UNION ALL
+                    SELECT 
+                        c.cat_id,
+                        c.parent_id,
+                        c.type,
+                        ch.level + 1,
+                        CONCAT(ch.path, ',', c.cat_id)
+                    FROM categories c
+                    JOIN category_hierarchy ch ON c.parent_id = ch.cat_id
+                    WHERE ch.level < 10  -- Prevent infinite recursion
+                )
+                SELECT 
+                    h.cat_id,
+                    h.parent_id,
+                    h.type,
+                    h.path,
+                    h.level
+                FROM (
+                    SELECT DISTINCT cat_id, parent_id, type, path, level
+                    FROM category_hierarchy
+                    WHERE cat_id IN (?)
+                ) h
+                ORDER BY h.level DESC
+            `, [allCategoryIds, allCategoryIds]);
+
+            const validCategories = new Map(categories.map(c => [c.cat_id, c]));
+            const validCategoryIds = new Set(categories.map(c => c.cat_id));
+
+            // Build category relationships ensuring proper hierarchy
+            const categoryRelationships = [];
+            batch
+                .filter(p => p.category_ids)
+                .forEach(product => {
+                    const productCategories = product.category_ids
+                        .split(',')
+                        .map(id => id.trim())
+                        .filter(id => id)
+                        .map(Number)
+                        .filter(id => !isNaN(id))
+                        .filter(id => validCategoryIds.has(id))
+                        .map(id => validCategories.get(id))
+                        .sort((a, b) => a.type - b.type);  // Sort by type to ensure proper hierarchy
+
+                    // Only add relationships that maintain proper hierarchy
+                    productCategories.forEach(category => {
+                        if (category.path.split(',').every(parentId => 
+                            validCategoryIds.has(Number(parentId))
+                        )) {
+                            categoryRelationships.push([category.cat_id, product.pid]);
+                        }
+                    });
+                });
 
             if (categoryRelationships.length > 0) {
-            // Verify categories exist before inserting relationships
-            const uniqueCatIds = [...new Set(categoryRelationships.map(([catId]) => catId))];
-            const [existingCats] = await localConnection.query(
-              "SELECT cat_id FROM categories WHERE cat_id IN (?)",
-              [uniqueCatIds]
-            );
-            const existingCatIds = new Set(existingCats.map(c => c.cat_id));
+                // First remove any existing relationships that will be replaced
+                await localConnection.query(`
+                    DELETE FROM product_categories 
+                    WHERE pid IN (?) AND cat_id IN (?)
+                `, [
+                    [...new Set(categoryRelationships.map(([_, pid]) => pid))],
+                    [...new Set(categoryRelationships.map(([catId, _]) => catId))]
+                ]);
 
-            // Filter relationships to only include existing categories
-            const validRelationships = categoryRelationships.filter(([catId]) => 
-              existingCatIds.has(catId)
-            );
-
-            if (validRelationships.length > 0) {
-              const catPlaceholders = validRelationships
+                // Then insert the new relationships
+                const placeholders = categoryRelationships
                     .map(() => "(?, ?)")
                     .join(",");
-              await localConnection.query(
-                `INSERT IGNORE INTO product_categories (cat_id, pid)
-                 VALUES ${catPlaceholders}`,
-                validRelationships.flat()
-              );
-            }
+
+                await localConnection.query(`
+                    INSERT INTO product_categories (cat_id, pid)
+                    VALUES ${placeholders}
+                `, categoryRelationships.flat());
             }
         }
       }
@@ -554,6 +616,7 @@ async function importMissingProducts(prodConnection, localConnection, missingPid
       SELECT COLUMN_NAME 
       FROM INFORMATION_SCHEMA.COLUMNS 
       WHERE TABLE_NAME = 'products'
+        AND COLUMN_NAME != 'updated'  -- Exclude the updated column
       ORDER BY ORDINAL_POSITION
     `);
     const columnNames = columns.map((col) => col.COLUMN_NAME);

inventory-server/scripts/import/purchase-orders.js

@@ -33,16 +33,15 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
       status: "running",
     });
 
-    // Get column names for the insert
+    // Get column names first
     const [columns] = await localConnection.query(`
       SELECT COLUMN_NAME 
       FROM INFORMATION_SCHEMA.COLUMNS 
       WHERE TABLE_NAME = 'purchase_orders'
+        AND COLUMN_NAME != 'updated'  -- Exclude the updated column
       ORDER BY ORDINAL_POSITION
     `);
-    const columnNames = columns
-      .map((col) => col.COLUMN_NAME)
-      .filter((name) => name !== "id");
+    const columnNames = columns.map(col => col.COLUMN_NAME);
 
     // Build incremental conditions
     const incrementalWhereClause = incrementalUpdate
@@ -321,10 +320,16 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
             let lastFulfillmentReceiving = null;
 
             for (const receiving of allReceivings) {
-              const qtyToApply = Math.min(remainingToFulfill, receiving.qty_each);
+                // Convert quantities to base units using supplier data
+                const baseQtyReceived = receiving.qty_each * (
+                    receiving.type === 'original' ? 1 : 
+                    Math.max(1, product.supplier_qty_per_unit || 1)
+                );
+                const qtyToApply = Math.min(remainingToFulfill, baseQtyReceived);
+                
                 if (qtyToApply > 0) {
                     // If this is the first receiving being applied, use its cost
-                if (actualCost === null) {
+                    if (actualCost === null && receiving.cost_each > 0) {
                         actualCost = receiving.cost_each;
                         firstFulfillmentReceiving = receiving;
                     }
@@ -332,13 +337,13 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
                     fulfillmentTracking.push({
                         receiving_id: receiving.receiving_id,
                         qty_applied: qtyToApply,
-                  qty_total: receiving.qty_each,
-                  cost: receiving.cost_each,
+                        qty_total: baseQtyReceived,
+                        cost: receiving.cost_each || actualCost || product.cost_each,
                         date: receiving.received_date,
                         received_by: receiving.received_by,
                         received_by_name: receiving.received_by_name || 'Unknown',
                         type: receiving.type,
-                  remaining_qty: receiving.qty_each - qtyToApply
+                        remaining_qty: baseQtyReceived - qtyToApply
                     });
                     remainingToFulfill -= qtyToApply;
                 } else {
@@ -346,8 +351,8 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
                     fulfillmentTracking.push({
                         receiving_id: receiving.receiving_id,
                         qty_applied: 0,
-                  qty_total: receiving.qty_each,
-                  cost: receiving.cost_each,
+                        qty_total: baseQtyReceived,
+                        cost: receiving.cost_each || actualCost || product.cost_each,
                         date: receiving.received_date,
                         received_by: receiving.received_by,
                         received_by_name: receiving.received_by_name || 'Unknown',
@@ -355,7 +360,7 @@ async function importPurchaseOrders(prodConnection, localConnection, incremental
                         is_excess: true
                     });
                 }
-              totalReceived += receiving.qty_each;
+                totalReceived += baseQtyReceived;
             }
 
             const receiving_status = !totalReceived ? 1 : // created

inventory-server/scripts/import/purchase_orders.js

@@ -1,82 +0,0 @@
-// Split into inserts and updates
-const insertsAndUpdates = batch.reduce((acc, po) => {
-  const key = `${po.po_id}-${po.pid}`;
-  if (existingPOMap.has(key)) {
-    const existing = existingPOMap.get(key);
-    // Check if any values are different
-    const hasChanges = columnNames.some(col => {
-      const newVal = po[col] ?? null;
-      const oldVal = existing[col] ?? null;
-      // Special handling for numbers to avoid type coercion issues
-      if (typeof newVal === 'number' && typeof oldVal === 'number') {
-        return Math.abs(newVal - oldVal) > 0.00001; // Allow for tiny floating point differences
-      }
-      // Special handling for receiving_history JSON
-      if (col === 'receiving_history') {
-        return JSON.stringify(newVal) !== JSON.stringify(oldVal);
-      }
-      return newVal !== oldVal;
-    });
-
-    if (hasChanges) {
-      console.log(`PO line changed: ${key}`, {
-        po_id: po.po_id,
-        pid: po.pid,
-        changes: columnNames.filter(col => {
-          const newVal = po[col] ?? null;
-          const oldVal = existing[col] ?? null;
-          if (typeof newVal === 'number' && typeof oldVal === 'number') {
-            return Math.abs(newVal - oldVal) > 0.00001;
-          }
-          if (col === 'receiving_history') {
-            return JSON.stringify(newVal) !== JSON.stringify(oldVal);
-          }
-          return newVal !== oldVal;
-        })
-      });
-      acc.updates.push({
-        po_id: po.po_id,
-        pid: po.pid,
-        values: columnNames.map(col => po[col] ?? null)
-      });
-    }
-  } else {
-    console.log(`New PO line: ${key}`);
-    acc.inserts.push({
-      po_id: po.po_id,
-      pid: po.pid,
-      values: columnNames.map(col => po[col] ?? null)
-    });
-  }
-  return acc;
-}, { inserts: [], updates: [] });
-
-// Handle inserts
-if (insertsAndUpdates.inserts.length > 0) {
-  const insertPlaceholders = Array(insertsAndUpdates.inserts.length).fill(placeholderGroup).join(",");
-  
-  const insertResult = await localConnection.query(`
-    INSERT INTO purchase_orders (${columnNames.join(",")})
-    VALUES ${insertPlaceholders}
-  `, insertsAndUpdates.inserts.map(i => i.values).flat());
-  
-  recordsAdded += insertResult[0].affectedRows;
-}
-
-// Handle updates
-if (insertsAndUpdates.updates.length > 0) {
-  const updatePlaceholders = Array(insertsAndUpdates.updates.length).fill(placeholderGroup).join(",");
-  
-  const updateResult = await localConnection.query(`
-    INSERT INTO purchase_orders (${columnNames.join(",")})
-    VALUES ${updatePlaceholders}
-    ON DUPLICATE KEY UPDATE
-      ${columnNames
-        .filter(col => col !== "po_id" && col !== "pid")
-        .map(col => `${col} = VALUES(${col})`)
-        .join(",")};
-  `, insertsAndUpdates.updates.map(u => u.values).flat());
-  
-  // Each update affects 2 rows in affectedRows, so we divide by 2 to get actual count
-  recordsUpdated += insertsAndUpdates.updates.length;
-} 

inventory-server/scripts/metrics/brand-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateBrandMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateBrandMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +16,28 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +47,12 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Calculate brand metrics with optimized queries
@@ -45,10 +71,21 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             WITH filtered_products AS (
                 SELECT 
                     p.*,
-                    CASE WHEN p.stock_quantity <= 5000 THEN p.pid END as valid_pid,
-                    CASE WHEN p.visible = true AND p.stock_quantity <= 5000 THEN p.pid END as active_pid,
                     CASE 
-                        WHEN p.stock_quantity IS NULL OR p.stock_quantity < 0 OR p.stock_quantity > 5000 THEN 0 
+                        WHEN p.stock_quantity <= 5000 AND p.stock_quantity >= 0 
+                        THEN p.pid 
+                    END as valid_pid,
+                    CASE 
+                        WHEN p.visible = true 
+                        AND p.stock_quantity <= 5000 
+                        AND p.stock_quantity >= 0 
+                        THEN p.pid 
+                    END as active_pid,
+                    CASE 
+                        WHEN p.stock_quantity IS NULL 
+                        OR p.stock_quantity < 0 
+                        OR p.stock_quantity > 5000 
+                        THEN 0 
                         ELSE p.stock_quantity 
                     END as valid_stock
                 FROM products p
@@ -57,10 +94,13 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             sales_periods AS (
                 SELECT 
                     p.brand,
-                    SUM(o.quantity * o.price) as period_revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0))) as period_revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as period_margin,
+                    COUNT(DISTINCT DATE(o.date)) as period_days,
                     CASE 
                         WHEN o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH) THEN 'current'
-                        WHEN o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH) THEN 'previous'
+                        WHEN o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                            AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH) THEN 'previous'
                     END as period_type
                 FROM filtered_products p
                 JOIN orders o ON p.pid = o.pid 
@@ -76,10 +116,20 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                     SUM(p.valid_stock) as total_stock_units,
                     SUM(p.valid_stock * p.cost_price) as total_stock_cost,
                     SUM(p.valid_stock * p.price) as total_stock_retail,
-                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
+                    COALESCE(SUM(o.quantity * (o.price - COALESCE(o.discount, 0))), 0) as total_revenue,
                     CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 THEN 
-                            (SUM((o.price - p.cost_price) * o.quantity) * 100.0) / SUM(o.price * o.quantity)
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN GREATEST(
+                            -100.0,
+                            LEAST(
+                                100.0,
+                                (
+                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                ) * 100.0 / 
+                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                            )
+                        )
                         ELSE 0 
                     END as avg_margin
                 FROM filtered_products p
@@ -97,17 +147,19 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                 bd.avg_margin,
                 CASE
                     WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
-                         AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0 THEN 100.0
-                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 THEN 0.0
-                    ELSE LEAST(
-                        GREATEST(
+                         AND MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) > 0 
+                    THEN 100.0
+                    WHEN MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END) = 0 
+                    THEN 0.0
+                    ELSE GREATEST(
+                        -100.0,
+                        LEAST(
                             ((MAX(CASE WHEN sp.period_type = 'current' THEN sp.period_revenue END) - 
                               MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)) / 
-                             NULLIF(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END), 0)) * 100.0,
-                            -100.0
-                        ),
+                             NULLIF(ABS(MAX(CASE WHEN sp.period_type = 'previous' THEN sp.period_revenue END)), 0)) * 100.0,
                             999.99
                         )
+                    )
                 END as growth_rate
             FROM brand_data bd
             LEFT JOIN sales_periods sp ON bd.brand = sp.brand
@@ -134,10 +186,20 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate brand time-based metrics with optimized query
         await connection.query(`
@@ -177,8 +239,18 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
                     SUM(p.valid_stock * p.price) as total_stock_retail,
                     SUM(o.quantity * o.price) as total_revenue,
                     CASE 
-                        WHEN SUM(o.quantity * o.price) > 0 THEN 
-                            (SUM((o.price - p.cost_price) * o.quantity) * 100.0) / SUM(o.price * o.quantity)
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN GREATEST(
+                            -100.0,
+                            LEAST(
+                                100.0,
+                                (
+                                    SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                    SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                ) * 100.0 / 
+                                NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                            )
+                        )
                         ELSE 0 
                     END as avg_margin
                 FROM filtered_products p
@@ -207,11 +279,33 @@ async function calculateBrandMetrics(startTime, totalProducts, processedCount, i
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('brand_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating brand metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/category-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateCategoryMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateCategoryMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +16,28 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +47,12 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
         
         // First, calculate base category metrics
@@ -67,10 +93,20 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Then update with margin and turnover data
         await connection.query(`
@@ -80,19 +116,35 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                     SUM(o.quantity * o.price) as total_sales,
                     SUM(o.quantity * (o.price - p.cost_price)) as total_margin,
                     SUM(o.quantity) as units_sold,
-                    AVG(GREATEST(p.stock_quantity, 0)) as avg_stock
+                    AVG(GREATEST(p.stock_quantity, 0)) as avg_stock,
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
+                LEFT JOIN turnover_config tc ON 
+                    (tc.category_id = pc.cat_id AND tc.vendor = p.vendor) OR
+                    (tc.category_id = pc.cat_id AND tc.vendor IS NULL) OR
+                    (tc.category_id IS NULL AND tc.vendor = p.vendor) OR
+                    (tc.category_id IS NULL AND tc.vendor IS NULL)
                 WHERE o.canceled = false 
-                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 1 YEAR)
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL COALESCE(tc.calculation_period_days, 30) DAY)
                 GROUP BY pc.cat_id
             )
             UPDATE category_metrics cm
             JOIN category_sales cs ON cm.category_id = cs.cat_id
+            LEFT JOIN turnover_config tc ON 
+                (tc.category_id = cm.category_id AND tc.vendor IS NULL) OR
+                (tc.category_id IS NULL AND tc.vendor IS NULL)
             SET 
                 cm.avg_margin = COALESCE(cs.total_margin * 100.0 / NULLIF(cs.total_sales, 0), 0),
-                cm.turnover_rate = LEAST(COALESCE(cs.units_sold / NULLIF(cs.avg_stock, 0), 0), 999.99),
+                cm.turnover_rate = CASE 
+                    WHEN cs.avg_stock > 0 AND cs.active_days > 0 
+                    THEN LEAST(
+                        (cs.units_sold / cs.avg_stock) * (365.0 / cs.active_days),
+                        999.99
+                    )
+                    ELSE 0 
+                END,
                 cm.last_calculated_at = NOW()
         `);
 
@@ -105,20 +157,34 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Finally update growth rates
         await connection.query(`
             WITH current_period AS (
                 SELECT 
                     pc.cat_id,
-                    SUM(o.quantity * o.price) as revenue
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0) - p.cost_price)) as gross_profit,
+                    COUNT(DISTINCT DATE(o.date)) as days
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
                 WHERE o.canceled = false 
                     AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
                 GROUP BY pc.cat_id
@@ -126,30 +192,106 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             previous_period AS (
                 SELECT 
                     pc.cat_id,
-                    SUM(o.quantity * o.price) as revenue
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    COUNT(DISTINCT DATE(o.date)) as days
+                FROM product_categories pc
+                JOIN products p ON pc.pid = p.pid
+                JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                WHERE o.canceled = false 
+                    AND o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
+                    AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                GROUP BY pc.cat_id
+            ),
+            trend_data AS (
+                SELECT 
+                    pc.cat_id,
+                    MONTH(o.date) as month,
+                    SUM(o.quantity * (o.price - COALESCE(o.discount, 0)) / 
+                        (1 + COALESCE(ss.seasonality_factor, 0))) as revenue,
+                    COUNT(DISTINCT DATE(o.date)) as days_in_month
+                FROM product_categories pc
+                JOIN products p ON pc.pid = p.pid
+                JOIN orders o ON p.pid = o.pid
+                LEFT JOIN sales_seasonality ss ON MONTH(o.date) = ss.month
+                WHERE o.canceled = false 
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH)
+                GROUP BY pc.cat_id, MONTH(o.date)
+            ),
+            trend_stats AS (
+                SELECT 
+                    cat_id,
+                    COUNT(*) as n,
+                    AVG(month) as avg_x,
+                    AVG(revenue / NULLIF(days_in_month, 0)) as avg_y,
+                    SUM(month * (revenue / NULLIF(days_in_month, 0))) as sum_xy,
+                    SUM(month * month) as sum_xx
+                FROM trend_data
+                GROUP BY cat_id
+                HAVING COUNT(*) >= 6
+            ),
+            trend_analysis AS (
+                SELECT 
+                    cat_id,
+                    ((n * sum_xy) - (avg_x * n * avg_y)) / 
+                        NULLIF((n * sum_xx) - (n * avg_x * avg_x), 0) as trend_slope,
+                    avg_y as avg_daily_revenue
+                FROM trend_stats
+            ),
+            margin_calc AS (
+                SELECT 
+                    pc.cat_id,
+                    CASE 
+                        WHEN SUM(o.quantity * o.price) > 0 THEN
+                            GREATEST(
+                                -100.0,
+                                LEAST(
+                                    100.0,
+                                    (
+                                        SUM(o.quantity * o.price) -  -- Use gross revenue (before discounts)
+                                        SUM(o.quantity * COALESCE(p.cost_price, 0))  -- Total costs
+                                    ) * 100.0 / 
+                                    NULLIF(SUM(o.quantity * o.price), 0)  -- Divide by gross revenue
+                                )
+                            )
+                        ELSE NULL
+                    END as avg_margin
                 FROM product_categories pc
                 JOIN products p ON pc.pid = p.pid
                 JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false 
-                    AND o.date BETWEEN DATE_SUB(CURRENT_DATE, INTERVAL 15 MONTH) 
-                    AND DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 3 MONTH)
                 GROUP BY pc.cat_id
             )
             UPDATE category_metrics cm
             LEFT JOIN current_period cp ON cm.category_id = cp.cat_id
             LEFT JOIN previous_period pp ON cm.category_id = pp.cat_id
+            LEFT JOIN trend_analysis ta ON cm.category_id = ta.cat_id
+            LEFT JOIN margin_calc mc ON cm.category_id = mc.cat_id
             SET 
                 cm.growth_rate = CASE
                     WHEN pp.revenue = 0 AND COALESCE(cp.revenue, 0) > 0 THEN 100.0
-                    WHEN pp.revenue = 0 THEN 0.0
-                    ELSE LEAST(
+                    WHEN pp.revenue = 0 OR cp.revenue IS NULL THEN 0.0
+                    WHEN ta.trend_slope IS NOT NULL THEN
                         GREATEST(
-                            ((COALESCE(cp.revenue, 0) - pp.revenue) / pp.revenue) * 100.0,
-                            -100.0
-                        ),
+                            -100.0,
+                            LEAST(
+                                (ta.trend_slope / NULLIF(ta.avg_daily_revenue, 0)) * 365 * 100,
                                 999.99
                             )
+                        )
+                    ELSE
+                        GREATEST(
+                            -100.0,
+                            LEAST(
+                                ((COALESCE(cp.revenue, 0) - pp.revenue) / 
+                                    NULLIF(ABS(pp.revenue), 0)) * 100.0,
+                                999.99
+                            )
+                        )
                 END,
+                cm.avg_margin = COALESCE(mc.avg_margin, cm.avg_margin),
                 cm.last_calculated_at = NOW()
             WHERE cp.cat_id IS NOT NULL OR pp.cat_id IS NOT NULL
         `);
@@ -163,10 +305,20 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate time-based metrics
         await connection.query(`
@@ -189,13 +341,23 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
                 COUNT(DISTINCT CASE WHEN p.visible = true THEN p.pid END) as active_products,
                 SUM(p.stock_quantity * p.cost_price) as total_value,
                 SUM(o.quantity * o.price) as total_revenue,
+                CASE
+                    WHEN SUM(o.quantity * o.price) > 0 THEN
+                        LEAST(
+                            GREATEST(
+                                SUM(o.quantity * (o.price - GREATEST(p.cost_price, 0))) * 100.0 / 
+                                SUM(o.quantity * o.price),
+                                -100
+                            ),
+                            100
+                        )
+                    ELSE 0
+                END as avg_margin,
                 COALESCE(
-                    SUM(o.quantity * (o.price - p.cost_price)) * 100.0 / 
-                    NULLIF(SUM(o.quantity * o.price), 0),
-                    0
-                ) as avg_margin,
-                COALESCE(
+                    LEAST(
                         SUM(o.quantity) / NULLIF(AVG(GREATEST(p.stock_quantity, 0)), 0),
+                        999.99
+                    ),
                     0
                 ) as turnover_rate
             FROM product_categories pc
@@ -216,17 +378,138 @@ async function calculateCategoryMetrics(startTime, totalProducts, processedCount
         processedCount = Math.floor(totalProducts * 0.99);
         outputProgress({
             status: 'running',
-            operation: 'Time-based metrics calculated',
+            operation: 'Time-based metrics calculated, updating category-sales metrics',
             current: processedCount,
             total: totalProducts,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
+        // Calculate category-sales metrics
+        await connection.query(`
+            INSERT INTO category_sales_metrics (
+                category_id,
+                brand,
+                period_start,
+                period_end,
+                avg_daily_sales,
+                total_sold,
+                num_products,
+                avg_price,
+                last_calculated_at
+            )
+            WITH date_ranges AS (
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 30 DAY) as period_start,
+                    CURRENT_DATE as period_end
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 31 DAY)
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 180 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 91 DAY)
+                UNION ALL
+                SELECT 
+                    DATE_SUB(CURRENT_DATE, INTERVAL 365 DAY),
+                    DATE_SUB(CURRENT_DATE, INTERVAL 181 DAY)
+            ),
+            sales_data AS (
+                SELECT 
+                    pc.cat_id,
+                    COALESCE(p.brand, 'Unknown') as brand,
+                    dr.period_start,
+                    dr.period_end,
+                    COUNT(DISTINCT p.pid) as num_products,
+                    SUM(o.quantity) as total_sold,
+                    SUM(o.quantity * o.price) as total_revenue,
+                    COUNT(DISTINCT DATE(o.date)) as num_days
+                FROM products p
+                JOIN product_categories pc ON p.pid = pc.pid
+                JOIN orders o ON p.pid = o.pid
+                CROSS JOIN date_ranges dr
+                WHERE o.canceled = false
+                    AND o.date BETWEEN dr.period_start AND dr.period_end
+                GROUP BY pc.cat_id, p.brand, dr.period_start, dr.period_end
+            )
+            SELECT 
+                cat_id as category_id,
+                brand,
+                period_start,
+                period_end,
+                CASE 
+                    WHEN num_days > 0 
+                    THEN total_sold / num_days 
+                    ELSE 0 
+                END as avg_daily_sales,
+                total_sold,
+                num_products,
+                CASE 
+                    WHEN total_sold > 0 
+                    THEN total_revenue / total_sold 
+                    ELSE 0 
+                END as avg_price,
+                NOW() as last_calculated_at
+            FROM sales_data
+            ON DUPLICATE KEY UPDATE
+                avg_daily_sales = VALUES(avg_daily_sales),
+                total_sold = VALUES(total_sold),
+                num_products = VALUES(num_products),
+                avg_price = VALUES(avg_price),
+                last_calculated_at = VALUES(last_calculated_at)
+        `);
+
+        processedCount = Math.floor(totalProducts * 1.0);
+        outputProgress({
+            status: 'running',
+            operation: 'Category-sales metrics calculated',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('category_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating category metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/financial-metrics.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateFinancialMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateFinancialMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +16,29 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+                AND DATE(o.date) >= DATE_SUB(CURDATE(), INTERVAL 12 MONTH)
+        `);
+        processedOrders = orderCount[0].count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +48,12 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Calculate financial metrics with optimized query
@@ -59,7 +86,8 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
                     WHEN COALESCE(pf.inventory_value, 0) > 0 AND pf.active_days > 0 THEN 
                         (COALESCE(pf.gross_profit, 0) * (365.0 / pf.active_days)) / COALESCE(pf.inventory_value, 0)
                     ELSE 0 
-                END
+                END,
+                pm.last_calculated_at = CURRENT_TIMESTAMP
         `);
 
         processedCount = Math.floor(totalProducts * 0.65);
@@ -71,10 +99,20 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Update time-based aggregates with optimized query
         await connection.query(`
@@ -115,11 +153,33 @@ async function calculateFinancialMetrics(startTime, totalProducts, processedCoun
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('financial_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating financial metrics');
         throw error;
     } finally {

inventory-server/scripts/metrics/product-metrics.js

@@ -11,11 +11,21 @@ function sanitizeValue(value) {
 
 async function calculateProductMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    const BATCH_SIZE = 5000;
+    
     try {
         // Skip flags are inherited from the parent scope
         const SKIP_PRODUCT_BASE_METRICS = 0;
         const SKIP_PRODUCT_TIME_AGGREGATES = 0;
 
+        // Get total product count if not provided
+        if (!totalProducts) {
+            const [productCount] = await connection.query('SELECT COUNT(*) as count FROM products');
+            totalProducts = productCount[0].count;
+        }
+
         if (isCancelled) {
             outputProgress({
                 status: 'cancelled',
@@ -25,10 +35,36 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // First ensure all products have a metrics record
+        await connection.query(`
+            INSERT IGNORE INTO product_metrics (pid, last_calculated_at)
+            SELECT pid, NOW()
+            FROM products
+        `);
+
+        // Get threshold settings once
+        const [thresholds] = await connection.query(`
+            SELECT critical_days, reorder_days, overstock_days, low_stock_threshold
+            FROM stock_thresholds 
+            WHERE category_id IS NULL AND vendor IS NULL
+            LIMIT 1
+        `);
+        const defaultThresholds = thresholds[0];
 
         // Calculate base product metrics
         if (!SKIP_PRODUCT_BASE_METRICS) {
@@ -40,89 +76,237 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
                 elapsed: formatElapsedTime(startTime),
                 remaining: estimateRemaining(startTime, processedCount, totalProducts),
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
-            // Calculate base metrics
+            // Get order count that will be processed
+            const [orderCount] = await connection.query(`
+                SELECT COUNT(*) as count 
+                FROM orders o 
+                WHERE o.canceled = false
+            `);
+            processedOrders = orderCount[0].count;
+
+            // Clear temporary tables
+            await connection.query('TRUNCATE TABLE temp_sales_metrics');
+            await connection.query('TRUNCATE TABLE temp_purchase_metrics');
+
+            // Populate temp_sales_metrics with base stats and sales averages
+            await connection.query(`
+                INSERT INTO temp_sales_metrics
+                SELECT 
+                    p.pid,
+                    COALESCE(SUM(o.quantity) / NULLIF(COUNT(DISTINCT DATE(o.date)), 0), 0) as daily_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 7), 0), 0) as weekly_sales_avg,
+                    COALESCE(SUM(o.quantity) / NULLIF(CEIL(COUNT(DISTINCT DATE(o.date)) / 30), 0), 0) as monthly_sales_avg,
+                    COALESCE(SUM(o.quantity * o.price), 0) as total_revenue,
+                    CASE 
+                        WHEN SUM(o.quantity * o.price) > 0 
+                        THEN ((SUM(o.quantity * o.price) - SUM(o.quantity * p.cost_price)) / SUM(o.quantity * o.price)) * 100 
+                        ELSE 0 
+                    END as avg_margin_percent,
+                    MIN(o.date) as first_sale_date,
+                    MAX(o.date) as last_sale_date
+                FROM products p
+                LEFT JOIN orders o ON p.pid = o.pid 
+                    AND o.canceled = false
+                    AND o.date >= DATE_SUB(CURDATE(), INTERVAL 90 DAY)
+                GROUP BY p.pid
+            `);
+
+            // Populate temp_purchase_metrics
+            await connection.query(`
+                INSERT INTO temp_purchase_metrics
+                SELECT 
+                    p.pid,
+                    AVG(DATEDIFF(po.received_date, po.date)) as avg_lead_time_days,
+                    MAX(po.date) as last_purchase_date,
+                    MIN(po.received_date) as first_received_date,
+                    MAX(po.received_date) as last_received_date
+                FROM products p
+                LEFT JOIN purchase_orders po ON p.pid = po.pid 
+                    AND po.received_date IS NOT NULL
+                    AND po.date >= DATE_SUB(CURDATE(), INTERVAL 365 DAY)
+                GROUP BY p.pid
+            `);
+
+            // Process updates in batches
+            let lastPid = 0;
+            while (true) {
+                if (isCancelled) break;
+
+                const [batch] = await connection.query(
+                    'SELECT pid FROM products WHERE pid > ? ORDER BY pid LIMIT ?',
+                    [lastPid, BATCH_SIZE]
+                );
+
+                if (batch.length === 0) break;
+
+                await connection.query(`
+                    UPDATE product_metrics pm
+                    JOIN products p ON pm.pid = p.pid
+                    LEFT JOIN temp_sales_metrics sm ON pm.pid = sm.pid
+                    LEFT JOIN temp_purchase_metrics lm ON pm.pid = lm.pid
+                    SET 
+                        pm.inventory_value = p.stock_quantity * NULLIF(p.cost_price, 0),
+                        pm.daily_sales_avg = COALESCE(sm.daily_sales_avg, 0),
+                        pm.weekly_sales_avg = COALESCE(sm.weekly_sales_avg, 0),
+                        pm.monthly_sales_avg = COALESCE(sm.monthly_sales_avg, 0),
+                        pm.total_revenue = COALESCE(sm.total_revenue, 0),
+                        pm.avg_margin_percent = COALESCE(sm.avg_margin_percent, 0),
+                        pm.first_sale_date = sm.first_sale_date,
+                        pm.last_sale_date = sm.last_sale_date,
+                        pm.avg_lead_time_days = COALESCE(lm.avg_lead_time_days, 30),
+                        pm.days_of_inventory = CASE 
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.daily_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        pm.weeks_of_inventory = CASE 
+                            WHEN COALESCE(sm.weekly_sales_avg, 0) > 0 
+                            THEN FLOOR(p.stock_quantity / NULLIF(sm.weekly_sales_avg, 0))
+                            ELSE NULL 
+                        END,
+                        pm.stock_status = CASE
+                            WHEN p.stock_quantity <= 0 THEN 'Out of Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 AND p.stock_quantity <= ? THEN 'Low Stock'
+                            WHEN COALESCE(sm.daily_sales_avg, 0) = 0 THEN 'In Stock'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ? THEN 'Critical'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) <= ? THEN 'Reorder'
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ? THEN 'Overstocked'
+                            ELSE 'Healthy'
+                        END,
+                        pm.safety_stock = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
+                                CEIL(sm.daily_sales_avg * SQRT(COALESCE(lm.avg_lead_time_days, 30)) * 1.96)
+                            ELSE ?
+                        END,
+                        pm.reorder_point = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 THEN
+                                CEIL(sm.daily_sales_avg * COALESCE(lm.avg_lead_time_days, 30)) + 
+                                CEIL(sm.daily_sales_avg * SQRT(COALESCE(lm.avg_lead_time_days, 30)) * 1.96)
+                            ELSE ?
+                        END,
+                        pm.reorder_qty = CASE
+                            WHEN COALESCE(sm.daily_sales_avg, 0) > 0 AND NULLIF(p.cost_price, 0) IS NOT NULL THEN
+                                GREATEST(
+                                    CEIL(SQRT((2 * (sm.daily_sales_avg * 365) * 25) / (NULLIF(p.cost_price, 0) * 0.25))),
+                                    ?
+                                )
+                            ELSE ?
+                        END,
+                        pm.overstocked_amt = CASE
+                            WHEN p.stock_quantity / NULLIF(sm.daily_sales_avg, 0) > ? 
+                            THEN GREATEST(0, p.stock_quantity - CEIL(sm.daily_sales_avg * ?))
+                            ELSE 0
+                        END,
+                        pm.last_calculated_at = NOW()
+                    WHERE p.pid IN (${batch.map(() => '?').join(',')})
+                `,
+                [
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.critical_days,
+                    defaultThresholds.reorder_days,
+                    defaultThresholds.overstock_days,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.low_stock_threshold,
+                    defaultThresholds.overstock_days,
+                    defaultThresholds.overstock_days,
+                    ...batch.map(row => row.pid)
+                ]
+                );
+
+                lastPid = batch[batch.length - 1].pid;
+                processedCount += batch.length;
+
+                outputProgress({
+                    status: 'running',
+                    operation: 'Processing base metrics batch',
+                    current: processedCount,
+                    total: totalProducts,
+                    elapsed: formatElapsedTime(startTime),
+                    remaining: estimateRemaining(startTime, processedCount, totalProducts),
+                    rate: calculateRate(startTime, processedCount),
+                    percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                    timing: {
+                        start_time: new Date(startTime).toISOString(),
+                        end_time: new Date().toISOString(),
+                        elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                    }
+                });
+            }
+
+            // Calculate forecast accuracy and bias in batches
+            lastPid = 0;
+            while (true) {
+                if (isCancelled) break;
+
+                const [batch] = await connection.query(
+                    'SELECT pid FROM products WHERE pid > ? ORDER BY pid LIMIT ?',
+                    [lastPid, BATCH_SIZE]
+                );
+
+                if (batch.length === 0) break;
+
                 await connection.query(`
                     UPDATE product_metrics pm
                     JOIN (
                         SELECT 
-                        p.pid,
-                        p.cost_price * p.stock_quantity as inventory_value,
-                        SUM(o.quantity) as total_quantity,
-                        COUNT(DISTINCT o.order_number) as number_of_orders,
-                        SUM(o.quantity * o.price) as total_revenue,
-                        SUM(o.quantity * p.cost_price) as cost_of_goods_sold,
-                        AVG(o.price) as avg_price,
-                        STDDEV(o.price) as price_std,
-                        MIN(o.date) as first_sale_date,
-                        MAX(o.date) as last_sale_date,
-                        COUNT(DISTINCT DATE(o.date)) as active_days
-                    FROM products p
-                    LEFT JOIN orders o ON p.pid = o.pid AND o.canceled = false
-                    GROUP BY p.pid
-                ) stats ON pm.pid = stats.pid
+                            sf.pid,
+                            AVG(CASE 
+                                WHEN o.quantity > 0 
+                                THEN ABS(sf.forecast_units - o.quantity) / o.quantity * 100
+                                ELSE 100 
+                            END) as avg_forecast_error,
+                            AVG(CASE 
+                                WHEN o.quantity > 0 
+                                THEN (sf.forecast_units - o.quantity) / o.quantity * 100
+                                ELSE 0 
+                            END) as avg_forecast_bias,
+                            MAX(sf.forecast_date) as last_forecast_date
+                        FROM sales_forecasts sf
+                        JOIN orders o ON sf.pid = o.pid 
+                            AND DATE(o.date) = sf.forecast_date
+                        WHERE o.canceled = false
+                            AND sf.forecast_date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+                            AND sf.pid IN (?)
+                        GROUP BY sf.pid
+                    ) fa ON pm.pid = fa.pid
                     SET 
-                    pm.inventory_value = COALESCE(stats.inventory_value, 0),
-                    pm.avg_quantity_per_order = COALESCE(stats.total_quantity / NULLIF(stats.number_of_orders, 0), 0),
-                    pm.number_of_orders = COALESCE(stats.number_of_orders, 0),
-                    pm.total_revenue = COALESCE(stats.total_revenue, 0),
-                    pm.cost_of_goods_sold = COALESCE(stats.cost_of_goods_sold, 0),
-                    pm.gross_profit = COALESCE(stats.total_revenue - stats.cost_of_goods_sold, 0),
-                    pm.avg_margin_percent = CASE 
-                        WHEN COALESCE(stats.total_revenue, 0) > 0 
-                        THEN ((stats.total_revenue - stats.cost_of_goods_sold) / stats.total_revenue) * 100 
-                        ELSE 0 
-                    END,
-                    pm.first_sale_date = stats.first_sale_date,
-                    pm.last_sale_date = stats.last_sale_date,
-                    pm.gmroi = CASE 
-                        WHEN COALESCE(stats.inventory_value, 0) > 0 
-                        THEN (stats.total_revenue - stats.cost_of_goods_sold) / stats.inventory_value
-                        ELSE 0 
-                    END,
+                        pm.forecast_accuracy = GREATEST(0, 100 - LEAST(fa.avg_forecast_error, 100)),
+                        pm.forecast_bias = GREATEST(-100, LEAST(fa.avg_forecast_bias, 100)),
+                        pm.last_forecast_date = fa.last_forecast_date,
                         pm.last_calculated_at = NOW()
-            `);
+                    WHERE pm.pid IN (?)
+                `, [batch.map(row => row.pid), batch.map(row => row.pid)]);
 
-            processedCount = Math.floor(totalProducts * 0.4);
-            outputProgress({
-                status: 'running',
-                operation: 'Base product metrics calculated',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-        } else {
-            processedCount = Math.floor(totalProducts * 0.4);
-            outputProgress({
-                status: 'running',
-                operation: 'Skipping base product metrics calculation',
-                current: processedCount,
-                total: totalProducts,
-                elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
+                lastPid = batch[batch.length - 1].pid;
+            }
         }
-
-        if (isCancelled) return processedCount;
 
         // Calculate product time aggregates
         if (!SKIP_PRODUCT_TIME_AGGREGATES) {
             outputProgress({
                 status: 'running',
                 operation: 'Starting product time aggregates calculation',
-                current: processedCount,
-                total: totalProducts,
+                current: processedCount || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
 
             // Calculate time-based aggregates
@@ -179,29 +363,206 @@ async function calculateProductMetrics(startTime, totalProducts, processedCount
             outputProgress({
                 status: 'running',
                 operation: 'Product time aggregates calculated',
-                current: processedCount,
-                total: totalProducts,
+                current: processedCount || 0,
+                total: totalProducts || 0,
                 elapsed: formatElapsedTime(startTime),
-                remaining: estimateRemaining(startTime, processedCount, totalProducts),
-                rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
             });
         } else {
             processedCount = Math.floor(totalProducts * 0.6);
             outputProgress({
                 status: 'running',
                 operation: 'Skipping product time aggregates calculation',
+                current: processedCount || 0,
+                total: totalProducts || 0,
+                elapsed: formatElapsedTime(startTime),
+                remaining: estimateRemaining(startTime, processedCount || 0, totalProducts || 0),
+                rate: calculateRate(startTime, processedCount || 0),
+                percentage: (((processedCount || 0) / (totalProducts || 1)) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+                }
+            });
+        }
+
+        // Calculate ABC classification
+        outputProgress({
+            status: 'running',
+            operation: 'Starting ABC classification',
             current: processedCount,
             total: totalProducts,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
+
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
+        const [abcConfig] = await connection.query('SELECT a_threshold, b_threshold FROM abc_classification_config WHERE id = 1');
+        const abcThresholds = abcConfig[0] || { a_threshold: 20, b_threshold: 50 };
+
+        // First, create and populate the rankings table with an index
+        await connection.query('DROP TEMPORARY TABLE IF EXISTS temp_revenue_ranks');
+        await connection.query(`
+            CREATE TEMPORARY TABLE temp_revenue_ranks (
+                pid BIGINT NOT NULL,
+                total_revenue DECIMAL(10,3),
+                rank_num INT,
+                dense_rank_num INT,
+                percentile DECIMAL(5,2),
+                total_count INT,
+                PRIMARY KEY (pid),
+                INDEX (rank_num),
+                INDEX (dense_rank_num),
+                INDEX (percentile)
+            ) ENGINE=MEMORY
+        `);
+
+        // Calculate rankings with proper tie handling
+        await connection.query(`
+            INSERT INTO temp_revenue_ranks
+            WITH revenue_data AS (
+                SELECT 
+                    pid,
+                    total_revenue,
+                    COUNT(*) OVER () as total_count,
+                    PERCENT_RANK() OVER (ORDER BY total_revenue DESC) * 100 as percentile,
+                    RANK() OVER (ORDER BY total_revenue DESC) as rank_num,
+                    DENSE_RANK() OVER (ORDER BY total_revenue DESC) as dense_rank_num
+                FROM product_metrics 
+                WHERE total_revenue > 0
+            )
+            SELECT 
+                pid,
+                total_revenue,
+                rank_num,
+                dense_rank_num,
+                percentile,
+                total_count
+            FROM revenue_data
+        `);
+
+        // Get total count for percentage calculation
+        const [rankingCount] = await connection.query('SELECT MAX(rank_num) as total_count FROM temp_revenue_ranks');
+        const totalCount = rankingCount[0].total_count || 1;
+        const max_rank = totalCount;
+
+        // Process updates in batches
+        let abcProcessedCount = 0;
+        const batchSize = 5000;
+
+        while (true) {
+            if (isCancelled) return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders: 0,  // This module doesn't process POs
+                success
+            };
+
+            // Get a batch of PIDs that need updating
+            const [pids] = await connection.query(`
+                SELECT pm.pid 
+                FROM product_metrics pm
+                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
+                WHERE pm.abc_class IS NULL 
+                OR pm.abc_class != 
+                    CASE 
+                        WHEN tr.pid IS NULL THEN 'C'
+                        WHEN tr.percentile <= ? THEN 'A'
+                        WHEN tr.percentile <= ? THEN 'B'
+                        ELSE 'C'
+                    END
+                LIMIT ?
+            `, [abcThresholds.a_threshold, abcThresholds.b_threshold, batchSize]);
+
+            if (pids.length === 0) break;
+
+            await connection.query(`
+                UPDATE product_metrics pm
+                LEFT JOIN temp_revenue_ranks tr ON pm.pid = tr.pid
+                SET pm.abc_class = 
+                    CASE 
+                        WHEN tr.pid IS NULL THEN 'C'
+                        WHEN tr.percentile <= ? THEN 'A'
+                        WHEN tr.percentile <= ? THEN 'B'
+                        ELSE 'C'
+                    END,
+                    pm.last_calculated_at = NOW()
+                WHERE pm.pid IN (?)
+            `, [abcThresholds.a_threshold, abcThresholds.b_threshold, pids.map(row => row.pid)]);
+
+            // Now update turnover rate with proper handling of zero inventory periods
+            await connection.query(`
+                UPDATE product_metrics pm
+                JOIN (
+                    SELECT 
+                        o.pid,
+                        SUM(o.quantity) as total_sold,
+                        COUNT(DISTINCT DATE(o.date)) as active_days,
+                        AVG(CASE 
+                            WHEN p.stock_quantity > 0 THEN p.stock_quantity 
+                            ELSE NULL 
+                        END) as avg_nonzero_stock
+                    FROM orders o
+                    JOIN products p ON o.pid = p.pid
+                    WHERE o.canceled = false 
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+                    AND o.pid IN (?)
+                    GROUP BY o.pid
+                ) sales ON pm.pid = sales.pid
+                SET 
+                    pm.turnover_rate = CASE 
+                        WHEN sales.avg_nonzero_stock > 0 AND sales.active_days > 0 
+                        THEN LEAST(
+                            (sales.total_sold / sales.avg_nonzero_stock) * (365.0 / sales.active_days),
+                            999.99
+                        )
+                        ELSE 0 
+                    END,
+                    pm.last_calculated_at = NOW()
+                WHERE pm.pid IN (?)
+            `, [pids.map(row => row.pid), pids.map(row => row.pid)]);
         }
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('product_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount || 0,
+            processedOrders: processedOrders || 0,
+            processedPurchaseOrders: 0,  // This module doesn't process POs
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating product metrics');
         throw error;
     } finally {
@@ -257,9 +618,9 @@ function calculateReorderQuantities(stock, stock_status, daily_sales_avg, avg_le
     if (daily_sales_avg > 0) {
         const annual_demand = daily_sales_avg * 365;
         const order_cost = 25; // Fixed cost per order
-        const holding_cost_percent = 0.25; // 25% annual holding cost
+        const holding_cost = config.cost_price * 0.25; // 25% of unit cost as annual holding cost
         
-        reorder_qty = Math.ceil(Math.sqrt((2 * annual_demand * order_cost) / holding_cost_percent));
+        reorder_qty = Math.ceil(Math.sqrt((2 * annual_demand * order_cost) / holding_cost));
     } else {
         // If no sales data, use a basic calculation
         reorder_qty = Math.max(safety_stock, config.low_stock_threshold);

inventory-server/scripts/metrics/sales-forecasts.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateSalesForecasts(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateSalesForecasts(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +16,29 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+                AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 90 DAY)
+        `);
+        processedOrders = orderCount[0].count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +48,12 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // First, create a temporary table for forecast dates
@@ -65,10 +92,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for daily sales stats
         await connection.query(`
@@ -94,10 +131,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for product stats
         await connection.query(`
@@ -119,10 +166,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate product-level forecasts
         await connection.query(`
@@ -134,37 +191,76 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
                 confidence_level,
                 last_calculated_at
             )
+            WITH daily_stats AS (
+                SELECT 
+                    ds.pid,
+                    AVG(ds.daily_quantity) as avg_daily_qty,
+                    STDDEV(ds.daily_quantity) as std_daily_qty,
+                    COUNT(DISTINCT ds.day_count) as data_points,
+                    SUM(ds.day_count) as total_days,
+                    AVG(ds.daily_revenue) as avg_daily_revenue,
+                    STDDEV(ds.daily_revenue) as std_daily_revenue,
+                    MIN(ds.daily_quantity) as min_daily_qty,
+                    MAX(ds.daily_quantity) as max_daily_qty,
+                    -- Calculate variance without using LAG
+                    COALESCE(
+                        STDDEV(ds.daily_quantity) / NULLIF(AVG(ds.daily_quantity), 0),
+                        0
+                    ) as daily_variance_ratio
+                FROM temp_daily_sales ds
+                GROUP BY ds.pid
+                HAVING AVG(ds.daily_quantity) > 0
+            )
             SELECT 
                 ds.pid,
                 fd.forecast_date,
                 GREATEST(0, 
-                    AVG(ds.daily_quantity) * 
-                    (1 + COALESCE(sf.seasonality_factor, 0))
+                    ROUND(
+                        ds.avg_daily_qty * 
+                        (1 + COALESCE(sf.seasonality_factor, 0)) *
+                        CASE
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 1.5 THEN 0.85
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 1.0 THEN 0.9
+                            WHEN ds.std_daily_qty / NULLIF(ds.avg_daily_qty, 0) > 0.5 THEN 0.95
+                            ELSE 1.0
+                        END,
+                        2
+                    )
                 ) as forecast_units,
                 GREATEST(0, 
+                    ROUND(
                         COALESCE(
                             CASE 
-                            WHEN SUM(ds.day_count) >= 4 THEN AVG(ds.daily_revenue)
+                                WHEN ds.data_points >= 4 THEN ds.avg_daily_revenue
                                 ELSE ps.overall_avg_revenue
                             END * 
                             (1 + COALESCE(sf.seasonality_factor, 0)) *
-                        (0.95 + (RAND() * 0.1)),
+                            CASE
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 1.5 THEN 0.85
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 1.0 THEN 0.9
+                                WHEN ds.std_daily_revenue / NULLIF(ds.avg_daily_revenue, 0) > 0.5 THEN 0.95
+                                ELSE 1.0
+                            END,
                             0
+                        ),
+                        2
                     )
                 ) as forecast_revenue,
                 CASE
-                    WHEN ps.total_days >= 60 THEN 90
-                    WHEN ps.total_days >= 30 THEN 80
-                    WHEN ps.total_days >= 14 THEN 70
+                    WHEN ds.total_days >= 60 AND ds.daily_variance_ratio < 0.5 THEN 90
+                    WHEN ds.total_days >= 60 THEN 85
+                    WHEN ds.total_days >= 30 AND ds.daily_variance_ratio < 0.5 THEN 80
+                    WHEN ds.total_days >= 30 THEN 75
+                    WHEN ds.total_days >= 14 AND ds.daily_variance_ratio < 0.5 THEN 70
+                    WHEN ds.total_days >= 14 THEN 65
                     ELSE 60
                 END as confidence_level,
                 NOW() as last_calculated_at
-            FROM temp_daily_sales ds
+            FROM daily_stats ds
             JOIN temp_product_stats ps ON ds.pid = ps.pid
             CROSS JOIN temp_forecast_dates fd
             LEFT JOIN sales_seasonality sf ON fd.month = sf.month
-            GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, ps.total_days, sf.seasonality_factor
-            HAVING AVG(ds.daily_quantity) > 0
+            GROUP BY ds.pid, fd.forecast_date, ps.overall_avg_revenue, sf.seasonality_factor
             ON DUPLICATE KEY UPDATE
                 forecast_units = VALUES(forecast_units),
                 forecast_revenue = VALUES(forecast_revenue),
@@ -181,10 +277,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Create temporary table for category stats
         await connection.query(`
@@ -221,10 +327,20 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Calculate category-level forecasts
         await connection.query(`
@@ -292,11 +408,33 @@ async function calculateSalesForecasts(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('sales_forecasts', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating sales forecasts');
         throw error;
     } finally {

inventory-server/scripts/metrics/time-aggregates.js

@@ -1,8 +1,11 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateTimeAggregates(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateTimeAggregates(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +16,28 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders: 0,
+                processedPurchaseOrders: 0,
+                success
+            };
+        }
+
+        // Get order count that will be processed
+        const [orderCount] = await connection.query(`
+            SELECT COUNT(*) as count 
+            FROM orders o 
+            WHERE o.canceled = false
+        `);
+        processedOrders = orderCount[0].count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +47,12 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // Initial insert of time-based aggregates
@@ -42,9 +68,11 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 stock_received,
                 stock_ordered,
                 avg_price,
-                profit_margin
+                profit_margin,
+                inventory_value,
+                gmroi
             )
-            WITH sales_data AS (
+            WITH monthly_sales AS (
                 SELECT 
                     o.pid,
                     YEAR(o.date) as year,
@@ -55,17 +83,19 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     COUNT(DISTINCT o.order_number) as order_count,
                     AVG(o.price - COALESCE(o.discount, 0)) as avg_price,
                     CASE 
-                        WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) = 0 THEN 0
-                        ELSE ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - 
-                              SUM(COALESCE(p.cost_price, 0) * o.quantity)) / 
-                              SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
-                    END as profit_margin
+                        WHEN SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) > 0 
+                        THEN ((SUM((o.price - COALESCE(o.discount, 0)) * o.quantity) - SUM(COALESCE(p.cost_price, 0) * o.quantity)) 
+                            / SUM((o.price - COALESCE(o.discount, 0)) * o.quantity)) * 100
+                        ELSE 0 
+                    END as profit_margin,
+                    p.cost_price * p.stock_quantity as inventory_value,
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM orders o
                 JOIN products p ON o.pid = p.pid
-                WHERE o.canceled = 0
+                WHERE o.canceled = false
                 GROUP BY o.pid, YEAR(o.date), MONTH(o.date)
             ),
-            purchase_data AS (
+            monthly_stock AS (
                 SELECT 
                     pid,
                     YEAR(date) as year,
@@ -73,45 +103,100 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     SUM(received) as stock_received,
                     SUM(ordered) as stock_ordered
                 FROM purchase_orders
-                WHERE status = 50
                 GROUP BY pid, YEAR(date), MONTH(date)
-            )
-            SELECT 
-                s.pid,
-                s.year,
-                s.month,
-                s.total_quantity_sold,
-                s.total_revenue,
-                s.total_cost,
-                s.order_count,
-                COALESCE(p.stock_received, 0) as stock_received,
-                COALESCE(p.stock_ordered, 0) as stock_ordered,
-                s.avg_price,
-                s.profit_margin
-            FROM sales_data s
-            LEFT JOIN purchase_data p 
-                ON s.pid = p.pid 
-                AND s.year = p.year 
-                AND s.month = p.month
-            UNION
+            ),
+            base_products AS (
                 SELECT 
                     p.pid,
-                p.year,
-                p.month,
+                    p.cost_price * p.stock_quantity as inventory_value
+                FROM products p
+            )
+            SELECT 
+                COALESCE(s.pid, ms.pid) as pid,
+                COALESCE(s.year, ms.year) as year,
+                COALESCE(s.month, ms.month) as month,
+                COALESCE(s.total_quantity_sold, 0) as total_quantity_sold,
+                COALESCE(s.total_revenue, 0) as total_revenue,
+                COALESCE(s.total_cost, 0) as total_cost,
+                COALESCE(s.order_count, 0) as order_count,
+                COALESCE(ms.stock_received, 0) as stock_received,
+                COALESCE(ms.stock_ordered, 0) as stock_ordered,
+                COALESCE(s.avg_price, 0) as avg_price,
+                COALESCE(s.profit_margin, 0) as profit_margin,
+                COALESCE(s.inventory_value, bp.inventory_value, 0) as inventory_value,
+                CASE 
+                    WHEN COALESCE(s.inventory_value, bp.inventory_value, 0) > 0 
+                        AND COALESCE(s.active_days, 0) > 0 
+                    THEN (COALESCE(s.total_revenue - s.total_cost, 0) * (365.0 / s.active_days)) 
+                        / COALESCE(s.inventory_value, bp.inventory_value)
+                    ELSE 0 
+                END as gmroi
+            FROM (
+                SELECT * FROM monthly_sales s
+                UNION ALL
+                SELECT 
+                    ms.pid,
+                    ms.year,
+                    ms.month,
                     0 as total_quantity_sold,
                     0 as total_revenue,
                     0 as total_cost,
                     0 as order_count,
-                p.stock_received,
-                p.stock_ordered,
+                    NULL as avg_price,
+                    0 as profit_margin,
+                    NULL as inventory_value,
+                    0 as active_days
+                FROM monthly_stock ms
+                WHERE NOT EXISTS (
+                    SELECT 1 FROM monthly_sales s2
+                    WHERE s2.pid = ms.pid
+                    AND s2.year = ms.year
+                    AND s2.month = ms.month
+                )
+            ) s
+            LEFT JOIN monthly_stock ms 
+                ON s.pid = ms.pid 
+                AND s.year = ms.year 
+                AND s.month = ms.month
+            JOIN base_products bp ON COALESCE(s.pid, ms.pid) = bp.pid
+            UNION
+            SELECT 
+                ms.pid,
+                ms.year,
+                ms.month,
+                0 as total_quantity_sold,
+                0 as total_revenue,
+                0 as total_cost,
+                0 as order_count,
+                ms.stock_received,
+                ms.stock_ordered,
                 0 as avg_price,
-                0 as profit_margin
-            FROM purchase_data p
-            LEFT JOIN sales_data s
-                ON p.pid = s.pid 
-                AND p.year = s.year 
-                AND p.month = s.month
-            WHERE s.pid IS NULL
+                0 as profit_margin,
+                bp.inventory_value,
+                0 as gmroi
+            FROM monthly_stock ms
+            JOIN base_products bp ON ms.pid = bp.pid
+            WHERE NOT EXISTS (
+                SELECT 1 FROM (
+                    SELECT * FROM monthly_sales
+                    UNION ALL
+                    SELECT 
+                        ms2.pid,
+                        ms2.year,
+                        ms2.month,
+                        0, 0, 0, 0, NULL, 0, NULL, 0
+                    FROM monthly_stock ms2
+                    WHERE NOT EXISTS (
+                        SELECT 1 FROM monthly_sales s2
+                        WHERE s2.pid = ms2.pid
+                        AND s2.year = ms2.year
+                        AND s2.month = ms2.month
+                    )
+                ) s
+                WHERE s.pid = ms.pid
+                AND s.year = ms.year
+                AND s.month = ms.month
+            )
             ON DUPLICATE KEY UPDATE
                 total_quantity_sold = VALUES(total_quantity_sold),
                 total_revenue = VALUES(total_revenue),
@@ -120,7 +205,9 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 stock_received = VALUES(stock_received),
                 stock_ordered = VALUES(stock_ordered),
                 avg_price = VALUES(avg_price),
-                profit_margin = VALUES(profit_margin)
+                profit_margin = VALUES(profit_margin),
+                inventory_value = VALUES(inventory_value),
+                gmroi = VALUES(gmroi)
         `);
 
         processedCount = Math.floor(totalProducts * 0.60);
@@ -132,10 +219,20 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
 
         // Update with financial metrics
         await connection.query(`
@@ -147,7 +244,7 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                     MONTH(o.date) as month,
                     p.cost_price * p.stock_quantity as inventory_value,
                     SUM(o.quantity * (o.price - p.cost_price)) as gross_profit,
-                    COUNT(DISTINCT DATE(o.date)) as days_in_period
+                    COUNT(DISTINCT DATE(o.date)) as active_days
                 FROM products p
                 LEFT JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false
@@ -156,12 +253,7 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
                 AND pta.year = fin.year 
                 AND pta.month = fin.month
             SET 
-                pta.inventory_value = COALESCE(fin.inventory_value, 0),
-                pta.gmroi = CASE 
-                    WHEN COALESCE(fin.inventory_value, 0) > 0 AND fin.days_in_period > 0 THEN 
-                        (COALESCE(fin.gross_profit, 0) * (365.0 / fin.days_in_period)) / COALESCE(fin.inventory_value, 0)
-                    ELSE 0 
-                END
+                pta.inventory_value = COALESCE(fin.inventory_value, 0)
         `);
 
         processedCount = Math.floor(totalProducts * 0.65);
@@ -173,11 +265,33 @@ async function calculateTimeAggregates(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('time_aggregates', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders: 0,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating time aggregates');
         throw error;
     } finally {

inventory-server/scripts/metrics/vendor-metrics.js

@@ -1,8 +1,12 @@
 const { outputProgress, formatElapsedTime, estimateRemaining, calculateRate, logError } = require('./utils/progress');
 const { getConnection } = require('./utils/db');
 
-async function calculateVendorMetrics(startTime, totalProducts, processedCount, isCancelled = false) {
+async function calculateVendorMetrics(startTime, totalProducts, processedCount = 0, isCancelled = false) {
     const connection = await getConnection();
+    let success = false;
+    let processedOrders = 0;
+    let processedPurchaseOrders = 0;
+    
     try {
         if (isCancelled) {
             outputProgress({
@@ -13,10 +17,36 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 elapsed: formatElapsedTime(startTime),
                 remaining: null,
                 rate: calculateRate(startTime, processedCount),
-                percentage: ((processedCount / totalProducts) * 100).toFixed(1)
-            });
-            return processedCount;
+                percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+                timing: {
+                    start_time: new Date(startTime).toISOString(),
+                    end_time: new Date().toISOString(),
+                    elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
                 }
+            });
+            return {
+                processedProducts: processedCount,
+                processedOrders,
+                processedPurchaseOrders,
+                success
+            };
+        }
+
+        // Get counts of records that will be processed
+        const [[orderCount], [poCount]] = await Promise.all([
+            connection.query(`
+                SELECT COUNT(*) as count 
+                FROM orders o 
+                WHERE o.canceled = false
+            `),
+            connection.query(`
+                SELECT COUNT(*) as count 
+                FROM purchase_orders po 
+                WHERE po.status != 0
+            `)
+        ]);
+        processedOrders = orderCount.count;
+        processedPurchaseOrders = poCount.count;
 
         outputProgress({
             status: 'running',
@@ -26,7 +56,12 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
         // First ensure all vendors exist in vendor_details
@@ -50,10 +85,20 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        if (isCancelled) return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
 
         // Now calculate vendor metrics
         await connection.query(`
@@ -68,6 +113,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 avg_order_value,
                 active_products,
                 total_products,
+                total_purchase_value,
+                avg_margin_percent,
                 status,
                 last_calculated_at
             )
@@ -76,7 +123,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                     p.vendor,
                     SUM(o.quantity * o.price) as total_revenue,
                     COUNT(DISTINCT o.id) as total_orders,
-                    COUNT(DISTINCT p.pid) as active_products
+                    COUNT(DISTINCT p.pid) as active_products,
+                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
                 FROM products p
                 JOIN orders o ON p.pid = o.pid
                 WHERE o.canceled = false
@@ -91,7 +139,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                     AVG(CASE 
                         WHEN po.receiving_status = 40 
                         THEN DATEDIFF(po.received_date, po.date) 
-                    END) as avg_lead_time_days
+                    END) as avg_lead_time_days,
+                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
                 FROM products p
                 JOIN purchase_orders po ON p.pid = po.pid
                 WHERE po.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
@@ -127,6 +176,12 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 END as avg_order_value,
                 COALESCE(vs.active_products, 0) as active_products,
                 COALESCE(vpr.total_products, 0) as total_products,
+                COALESCE(vp.total_purchase_value, 0) as total_purchase_value,
+                CASE 
+                    WHEN vs.total_revenue > 0 
+                    THEN (vs.total_margin / vs.total_revenue) * 100 
+                    ELSE 0 
+                END as avg_margin_percent,
                 'active' as status,
                 NOW() as last_calculated_at
             FROM vendor_sales vs
@@ -143,6 +198,8 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
                 avg_order_value = VALUES(avg_order_value),
                 active_products = VALUES(active_products),
                 total_products = VALUES(total_products),
+                total_purchase_value = VALUES(total_purchase_value),
+                avg_margin_percent = VALUES(avg_margin_percent),
                 status = VALUES(status),
                 last_calculated_at = VALUES(last_calculated_at)
         `);
@@ -150,17 +207,155 @@ async function calculateVendorMetrics(startTime, totalProducts, processedCount,
         processedCount = Math.floor(totalProducts * 0.9);
         outputProgress({
             status: 'running',
-            operation: 'Vendor metrics calculated',
+            operation: 'Vendor metrics calculated, updating time-based metrics',
             current: processedCount,
             total: totalProducts,
             elapsed: formatElapsedTime(startTime),
             remaining: estimateRemaining(startTime, processedCount, totalProducts),
             rate: calculateRate(startTime, processedCount),
-            percentage: ((processedCount / totalProducts) * 100).toFixed(1)
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
         });
 
-        return processedCount;
+        if (isCancelled) return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
+
+        // Calculate time-based metrics
+        await connection.query(`
+            INSERT INTO vendor_time_metrics (
+                vendor,
+                year,
+                month,
+                total_orders,
+                late_orders,
+                avg_lead_time_days,
+                total_purchase_value,
+                total_revenue,
+                avg_margin_percent
+            )
+            WITH monthly_orders AS (
+                SELECT 
+                    p.vendor,
+                    YEAR(o.date) as year,
+                    MONTH(o.date) as month,
+                    COUNT(DISTINCT o.id) as total_orders,
+                    SUM(o.quantity * o.price) as total_revenue,
+                    SUM(o.quantity * (o.price - p.cost_price)) as total_margin
+                FROM products p
+                JOIN orders o ON p.pid = o.pid
+                WHERE o.canceled = false
+                    AND o.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND p.vendor IS NOT NULL
+                GROUP BY p.vendor, YEAR(o.date), MONTH(o.date)
+            ),
+            monthly_po AS (
+                SELECT 
+                    p.vendor,
+                    YEAR(po.date) as year,
+                    MONTH(po.date) as month,
+                    COUNT(DISTINCT po.id) as total_po,
+                    COUNT(DISTINCT CASE 
+                        WHEN po.receiving_status = 40 AND po.received_date > po.expected_date 
+                        THEN po.id 
+                    END) as late_orders,
+                    AVG(CASE 
+                        WHEN po.receiving_status = 40 
+                        THEN DATEDIFF(po.received_date, po.date)
+                    END) as avg_lead_time_days,
+                    SUM(po.ordered * po.po_cost_price) as total_purchase_value
+                FROM products p
+                JOIN purchase_orders po ON p.pid = po.pid
+                WHERE po.date >= DATE_SUB(CURRENT_DATE, INTERVAL 12 MONTH)
+                    AND p.vendor IS NOT NULL
+                GROUP BY p.vendor, YEAR(po.date), MONTH(po.date)
+            )
+            SELECT 
+                mo.vendor,
+                mo.year,
+                mo.month,
+                COALESCE(mp.total_po, 0) as total_orders,
+                COALESCE(mp.late_orders, 0) as late_orders,
+                COALESCE(mp.avg_lead_time_days, 0) as avg_lead_time_days,
+                COALESCE(mp.total_purchase_value, 0) as total_purchase_value,
+                mo.total_revenue,
+                CASE 
+                    WHEN mo.total_revenue > 0 
+                    THEN (mo.total_margin / mo.total_revenue) * 100 
+                    ELSE 0 
+                END as avg_margin_percent
+            FROM monthly_orders mo
+            LEFT JOIN monthly_po mp ON mo.vendor = mp.vendor 
+                AND mo.year = mp.year 
+                AND mo.month = mp.month
+            UNION
+            SELECT 
+                mp.vendor,
+                mp.year,
+                mp.month,
+                mp.total_po as total_orders,
+                mp.late_orders,
+                mp.avg_lead_time_days,
+                mp.total_purchase_value,
+                0 as total_revenue,
+                0 as avg_margin_percent
+            FROM monthly_po mp
+            LEFT JOIN monthly_orders mo ON mp.vendor = mo.vendor 
+                AND mp.year = mo.year 
+                AND mp.month = mo.month
+            WHERE mo.vendor IS NULL
+            ON DUPLICATE KEY UPDATE
+                total_orders = VALUES(total_orders),
+                late_orders = VALUES(late_orders),
+                avg_lead_time_days = VALUES(avg_lead_time_days),
+                total_purchase_value = VALUES(total_purchase_value),
+                total_revenue = VALUES(total_revenue),
+                avg_margin_percent = VALUES(avg_margin_percent)
+        `);
+
+        processedCount = Math.floor(totalProducts * 0.95);
+        outputProgress({
+            status: 'running',
+            operation: 'Time-based vendor metrics calculated',
+            current: processedCount,
+            total: totalProducts,
+            elapsed: formatElapsedTime(startTime),
+            remaining: estimateRemaining(startTime, processedCount, totalProducts),
+            rate: calculateRate(startTime, processedCount),
+            percentage: ((processedCount / totalProducts) * 100).toFixed(1),
+            timing: {
+                start_time: new Date(startTime).toISOString(),
+                end_time: new Date().toISOString(),
+                elapsed_seconds: Math.round((Date.now() - startTime) / 1000)
+            }
+        });
+
+        // If we get here, everything completed successfully
+        success = true;
+        
+        // Update calculate_status
+        await connection.query(`
+            INSERT INTO calculate_status (module_name, last_calculation_timestamp)
+            VALUES ('vendor_metrics', NOW())
+            ON DUPLICATE KEY UPDATE last_calculation_timestamp = NOW()
+        `);
+
+        return {
+            processedProducts: processedCount,
+            processedOrders,
+            processedPurchaseOrders,
+            success
+        };
+
     } catch (error) {
+        success = false;
         logError(error, 'Error calculating vendor metrics');
         throw error;
     } finally {

inventory-server/scripts/reset-db.js

@@ -156,7 +156,7 @@ async function resetDatabase() {
             SELECT GROUP_CONCAT(table_name) as tables 
             FROM information_schema.tables 
             WHERE table_schema = DATABASE()
-            AND table_name NOT IN ('users', 'import_history')
+            AND table_name NOT IN ('users', 'import_history', 'calculate_history')
         `);
 
         if (!tables[0].tables) {
@@ -175,7 +175,7 @@ async function resetDatabase() {
                 DROP TABLE IF EXISTS 
                 ${tables[0].tables
                     .split(',')
-                    .filter(table => table !== 'users')
+                    .filter(table => !['users', 'calculate_history'].includes(table))
                     .map(table => '`' + table + '`')
                     .join(', ')}
             `;
@@ -543,5 +543,15 @@ async function resetDatabase() {
     }
 }
 
-// Run the reset
-resetDatabase(); 
+// Export if required as a module
+if (typeof module !== 'undefined' && module.exports) {
+    module.exports = resetDatabase;
+}
+
+// Run if called directly
+if (require.main === module) {
+    resetDatabase().catch(error => {
+        console.error('Error:', error);
+        process.exit(1);
+    });
+} 

inventory-server/src/routes/csv.js

@@ -2,6 +2,7 @@ const express = require('express');
 const router = express.Router();
 const { spawn } = require('child_process');
 const path = require('path');
+const db = require('../utils/db');
 
 // Debug middleware MUST be first
 router.use((req, res, next) => {
@@ -9,9 +10,11 @@ router.use((req, res, next) => {
     next();
 });
 
-// Store active import process and its progress
+// Store active processes and their progress
 let activeImport = null;
 let importProgress = null;
+let activeFullUpdate = null;
+let activeFullReset = null;
 
 // SSE clients for progress updates
 const updateClients = new Set();
@@ -19,17 +22,16 @@ const importClients = new Set();
 const resetClients = new Set();
 const resetMetricsClients = new Set();
 const calculateMetricsClients = new Set();
+const fullUpdateClients = new Set();
+const fullResetClients = new Set();
 
 // Helper to send progress to specific clients
-function sendProgressToClients(clients, progress) {
-    const data = typeof progress === 'string' ? { progress } : progress;
-    
-    // Ensure we have a status field
-    if (!data.status) {
-        data.status = 'running';
-    }
-    
-    const message = `data: ${JSON.stringify(data)}\n\n`;
+function sendProgressToClients(clients, data) {
+    // If data is a string, send it directly
+    // If it's an object, convert it to JSON
+    const message = typeof data === 'string' 
+        ? `data: ${data}\n\n`
+        : `data: ${JSON.stringify(data)}\n\n`;
     
     clients.forEach(client => {
         try {
@@ -45,115 +47,149 @@ function sendProgressToClients(clients, progress) {
     });
 }
 
-// Progress endpoints
-router.get('/update/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the update set
-    updateClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        updateClients.delete(res);
-    });
-});
-
-router.get('/import/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the import set
-    importClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        importClients.delete(res);
-    });
-});
-
-router.get('/reset/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the reset set
-    resetClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        resetClients.delete(res);
-    });
-});
-
-// Add reset-metrics progress endpoint
-router.get('/reset-metrics/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send an initial message to test the connection
-    res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
-
-    // Add this client to the reset-metrics set
-    resetMetricsClients.add(res);
-
-    // Remove client when connection closes
-    req.on('close', () => {
-        resetMetricsClients.delete(res);
-    });
-});
-
-// Add calculate-metrics progress endpoint
-router.get('/calculate-metrics/progress', (req, res) => {
-    res.writeHead(200, {
-        'Content-Type': 'text/event-stream',
-        'Cache-Control': 'no-cache',
-        'Connection': 'keep-alive',
-        'Access-Control-Allow-Origin': req.headers.origin || '*',
-        'Access-Control-Allow-Credentials': 'true'
-    });
-
-    // Send current progress if it exists
-    if (importProgress) {
-        res.write(`data: ${JSON.stringify(importProgress)}\n\n`);
-    } else {
-        res.write('data: {"status":"running","operation":"Initializing connection..."}\n\n');
+// Helper to run a script and stream progress
+function runScript(scriptPath, type, clients) {
+    return new Promise((resolve, reject) => {
+        // Kill any existing process of this type
+        let activeProcess;
+        switch (type) {
+            case 'update':
+                if (activeFullUpdate) {
+                    try { activeFullUpdate.kill(); } catch (e) { }
+                }
+                activeProcess = activeFullUpdate;
+                break;
+            case 'reset':
+                if (activeFullReset) {
+                    try { activeFullReset.kill(); } catch (e) { }
+                }
+                activeProcess = activeFullReset;
+                break;
         }
 
-    // Add this client to the calculate-metrics set
-    calculateMetricsClients.add(res);
+        const child = spawn('node', [scriptPath], {
+            stdio: ['inherit', 'pipe', 'pipe']
+        });
 
-    // Remove client when connection closes
+        switch (type) {
+            case 'update':
+                activeFullUpdate = child;
+                break;
+            case 'reset':
+                activeFullReset = child;
+                break;
+        }
+
+        let output = '';
+
+        child.stdout.on('data', (data) => {
+            const text = data.toString();
+            output += text;
+            
+            // Split by lines to handle multiple JSON outputs
+            const lines = text.split('\n');
+            lines.filter(line => line.trim()).forEach(line => {
+                try {
+                    // Try to parse as JSON but don't let it affect the display
+                    const jsonData = JSON.parse(line);
+                    // Only end the process if we get a final status
+                    if (jsonData.status === 'complete' || jsonData.status === 'error' || jsonData.status === 'cancelled') {
+                        if (jsonData.status === 'complete' && !jsonData.operation?.includes('complete')) {
+                            // Don't close for intermediate completion messages
+                            sendProgressToClients(clients, line);
+                            return;
+                        }
+                        // Close only on final completion/error/cancellation
+                        switch (type) {
+                            case 'update':
+                                activeFullUpdate = null;
+                                break;
+                            case 'reset':
+                                activeFullReset = null;
+                                break;
+                        }
+                        if (jsonData.status === 'error') {
+                            reject(new Error(jsonData.error || 'Unknown error'));
+                        } else {
+                            resolve({ output });
+                        }
+                    }
+                } catch (e) {
+                    // Not JSON, just display as is
+                }
+                // Always send the raw line
+                sendProgressToClients(clients, line);
+            });
+        });
+
+        child.stderr.on('data', (data) => {
+            const text = data.toString();
+            console.error(text);
+            // Send stderr output directly too
+            sendProgressToClients(clients, text);
+        });
+
+        child.on('close', (code) => {
+            switch (type) {
+                case 'update':
+                    activeFullUpdate = null;
+                    break;
+                case 'reset':
+                    activeFullReset = null;
+                    break;
+            }
+
+            if (code !== 0) {
+                const error = `Script ${scriptPath} exited with code ${code}`;
+                sendProgressToClients(clients, error);
+                reject(new Error(error));
+            }
+            // Don't resolve here - let the completion message from the script trigger the resolve
+        });
+
+        child.on('error', (err) => {
+            switch (type) {
+                case 'update':
+                    activeFullUpdate = null;
+                    break;
+                case 'reset':
+                    activeFullReset = null;
+                    break;
+            }
+            sendProgressToClients(clients, err.message);
+            reject(err);
+        });
+    });
+}
+
+// Progress endpoints
+router.get('/:type/progress', (req, res) => {
+    const { type } = req.params;
+    if (!['update', 'reset'].includes(type)) {
+        return res.status(400).json({ error: 'Invalid operation type' });
+    }
+
+    res.writeHead(200, {
+        'Content-Type': 'text/event-stream',
+        'Cache-Control': 'no-cache',
+        'Connection': 'keep-alive',
+        'Access-Control-Allow-Origin': req.headers.origin || '*',
+        'Access-Control-Allow-Credentials': 'true'
+    });
+
+    // Add this client to the correct set
+    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
+    clients.add(res);
+
+    // Send initial connection message
+    sendProgressToClients(new Set([res]), JSON.stringify({
+        status: 'running',
+        operation: 'Initializing connection...'
+    }));
+
+    // Handle client disconnect
     req.on('close', () => {
-        calculateMetricsClients.delete(res);
+        clients.delete(res);
     });
 });
 
@@ -174,7 +210,6 @@ router.get('/status', (req, res) => {
 
 // Add calculate-metrics status endpoint
 router.get('/calculate-metrics/status', (req, res) => {
-    console.log('Calculate metrics status endpoint hit');
     const calculateMetrics = require('../../scripts/calculate-metrics');
     const progress = calculateMetrics.getProgress();
     
@@ -371,49 +406,35 @@ router.post('/import', async (req, res) => {
 
 // Route to cancel active process
 router.post('/cancel', (req, res) => {
-    if (!activeImport) {
-        return res.status(404).json({ error: 'No active process to cancel' });
-    }
-
-    try {
-        // If it's the prod import module, call its cancel function
-        if (typeof activeImport.cancelImport === 'function') {
-            activeImport.cancelImport();
-        } else {
-            // Otherwise it's a child process
-            activeImport.kill('SIGTERM');
-        }
+    let killed = false;
 
     // Get the operation type from the request
-        const { operation } = req.query;
+    const { type } = req.query;
+    const clients = type === 'update' ? fullUpdateClients : fullResetClients;
+    const activeProcess = type === 'update' ? activeFullUpdate : activeFullReset;
 
-        // Send cancel message only to the appropriate client set
-        const cancelMessage = { 
+    if (activeProcess) {
+        try {
+            activeProcess.kill('SIGTERM');
+            if (type === 'update') {
+                activeFullUpdate = null;
+            } else {
+                activeFullReset = null;
+            }
+            killed = true;
+            sendProgressToClients(clients, JSON.stringify({
                 status: 'cancelled',
                 operation: 'Operation cancelled'
-        };
-
-        switch (operation) {
-            case 'update':
-                sendProgressToClients(updateClients, cancelMessage);
-                break;
-            case 'import':
-                sendProgressToClients(importClients, cancelMessage);
-                break;
-            case 'reset':
-                sendProgressToClients(resetClients, cancelMessage);
-                break;
-            case 'calculate-metrics':
-                sendProgressToClients(calculateMetricsClients, cancelMessage);
-                break;
+            }));
+        } catch (err) {
+            console.error(`Error killing ${type} process:`, err);
+        }
     }
 
+    if (killed) {
         res.json({ success: true });
-    } catch (error) {
-        // Even if there's an error, try to clean up
-        activeImport = null;
-        importProgress = null;
-        res.status(500).json({ error: 'Failed to cancel process' });
+    } else {
+        res.status(404).json({ error: 'No active process to cancel' });
     }
 });
 
@@ -552,20 +573,6 @@ router.post('/reset-metrics', async (req, res) => {
     }
 });
 
-// Add calculate-metrics status endpoint
-router.get('/calculate-metrics/status', (req, res) => {
-    const calculateMetrics = require('../../scripts/calculate-metrics');
-    const progress = calculateMetrics.getProgress();
-    
-    // Only consider it active if both the process is running and we have progress
-    const isActive = !!activeImport && !!progress;
-    
-    res.json({
-        active: isActive,
-        progress: isActive ? progress : null
-    });
-});
-
 // Add calculate-metrics endpoint
 router.post('/calculate-metrics', async (req, res) => {
     if (activeImport) {
@@ -711,4 +718,96 @@ router.post('/import-from-prod', async (req, res) => {
     }
 });
 
+// POST /csv/full-update - Run full update script
+router.post('/full-update', async (req, res) => {
+    try {
+        const scriptPath = path.join(__dirname, '../../scripts/full-update.js');
+        runScript(scriptPath, 'update', fullUpdateClients)
+            .catch(error => {
+                console.error('Update failed:', error);
+            });
+        res.status(202).json({ message: 'Update started' });
+    } catch (error) {
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// POST /csv/full-reset - Run full reset script
+router.post('/full-reset', async (req, res) => {
+    try {
+        const scriptPath = path.join(__dirname, '../../scripts/full-reset.js');
+        runScript(scriptPath, 'reset', fullResetClients)
+            .catch(error => {
+                console.error('Reset failed:', error);
+            });
+        res.status(202).json({ message: 'Reset started' });
+    } catch (error) {
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /history/import - Get recent import history
+router.get('/history/import', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const [rows] = await pool.query(`
+            SELECT * FROM import_history 
+            ORDER BY start_time DESC 
+            LIMIT 20
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching import history:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /history/calculate - Get recent calculation history
+router.get('/history/calculate', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const [rows] = await pool.query(`
+            SELECT * FROM calculate_history 
+            ORDER BY start_time DESC 
+            LIMIT 20
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching calculate history:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /status/modules - Get module calculation status
+router.get('/status/modules', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const [rows] = await pool.query(`
+            SELECT module_name, last_calculation_timestamp 
+            FROM calculate_status 
+            ORDER BY module_name
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching module status:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
+// GET /status/tables - Get table sync status
+router.get('/status/tables', async (req, res) => {
+    try {
+        const pool = req.app.locals.pool;
+        const [rows] = await pool.query(`
+            SELECT table_name, last_sync_timestamp 
+            FROM sync_status 
+            ORDER BY table_name
+        `);
+        res.json(rows || []);
+    } catch (error) {
+        console.error('Error fetching table status:', error);
+        res.status(500).json({ error: error.message });
+    }
+});
+
 module.exports = router; 

inventory/package.json

@@ -10,58 +10,90 @@
     "preview": "vite preview"
   },
   "dependencies": {
+    "@chakra-ui/button": "^2.1.0",
+    "@chakra-ui/checkbox": "^2.3.2",
+    "@chakra-ui/form-control": "^2.2.0",
+    "@chakra-ui/hooks": "^2.4.3",
+    "@chakra-ui/icons": "^2.2.4",
+    "@chakra-ui/input": "^2.1.2",
+    "@chakra-ui/layout": "^2.3.1",
+    "@chakra-ui/modal": "^2.3.1",
+    "@chakra-ui/popper": "^3.1.0",
+    "@chakra-ui/react": "^2.8.1",
+    "@chakra-ui/select": "^2.1.2",
+    "@chakra-ui/system": "^2.6.2",
+    "@chakra-ui/theme": "^3.4.7",
+    "@chakra-ui/theme-tools": "^2.2.7",
+    "@chakra-ui/utils": "^2.2.3",
     "@dnd-kit/core": "^6.3.1",
     "@dnd-kit/sortable": "^10.0.0",
     "@dnd-kit/utilities": "^3.2.2",
+    "@emotion/react": "^11.14.0",
+    "@emotion/styled": "^11.14.0",
     "@radix-ui/react-accordion": "^1.2.2",
     "@radix-ui/react-alert-dialog": "^1.1.4",
     "@radix-ui/react-avatar": "^1.1.2",
+    "@radix-ui/react-checkbox": "^1.1.4",
     "@radix-ui/react-collapsible": "^1.1.2",
     "@radix-ui/react-dialog": "^1.1.4",
     "@radix-ui/react-dropdown-menu": "^2.1.4",
     "@radix-ui/react-label": "^2.1.1",
     "@radix-ui/react-popover": "^1.1.4",
     "@radix-ui/react-progress": "^1.1.1",
+    "@radix-ui/react-radio-group": "^1.2.3",
     "@radix-ui/react-scroll-area": "^1.2.2",
     "@radix-ui/react-select": "^2.1.4",
     "@radix-ui/react-separator": "^1.1.1",
     "@radix-ui/react-slot": "^1.1.1",
     "@radix-ui/react-switch": "^1.1.2",
     "@radix-ui/react-tabs": "^1.1.2",
+    "@radix-ui/react-toast": "^1.2.6",
     "@radix-ui/react-toggle": "^1.1.1",
     "@radix-ui/react-toggle-group": "^1.1.1",
     "@radix-ui/react-tooltip": "^1.1.6",
     "@shadcn/ui": "^0.0.4",
     "@tabler/icons-react": "^3.28.1",
-    "@tanstack/react-query": "^5.63.0",
+    "@tanstack/react-query": "^5.66.7",
     "@tanstack/react-table": "^8.20.6",
     "@tanstack/react-virtual": "^3.11.2",
     "@tanstack/virtual-core": "^3.11.2",
+    "@types/js-levenshtein": "^1.1.3",
+    "@types/uuid": "^10.0.0",
+    "chakra-react-select": "^4.7.5",
+    "chakra-ui-steps": "^2.0.4",
     "chart.js": "^4.4.7",
     "chartjs-adapter-date-fns": "^3.0.0",
     "class-variance-authority": "^0.7.1",
     "clsx": "^2.1.1",
     "cmdk": "^1.0.0",
     "date-fns": "^3.6.0",
+    "framer-motion": "^12.4.4",
+    "js-levenshtein": "^1.1.6",
+    "lodash": "^4.17.21",
     "lucide-react": "^0.469.0",
     "motion": "^11.18.0",
     "next-themes": "^0.4.4",
     "react": "^18.3.1",
     "react-chartjs-2": "^5.3.0",
+    "react-data-grid": "^7.0.0-beta.13",
     "react-day-picker": "^8.10.1",
     "react-dom": "^18.3.1",
+    "react-dropzone": "^14.3.5",
+    "react-icons": "^5.4.0",
     "react-router-dom": "^7.1.1",
     "recharts": "^2.15.0",
     "sonner": "^1.7.1",
     "tailwind-merge": "^2.6.0",
     "tailwindcss-animate": "^1.0.7",
     "tanstack": "^1.0.0",
-    "vaul": "^1.1.2"
+    "uuid": "^11.0.5",
+    "vaul": "^1.1.2",
+    "xlsx": "^0.18.5"
   },
   "devDependencies": {
     "@eslint/js": "^9.17.0",
     "@types/fs-extra": "^11.0.4",
-    "@types/lodash": "^4.17.14",
+    "@types/lodash": "^4.17.15",
     "@types/node": "^22.10.5",
     "@types/react": "^18.3.18",
     "@types/react-dom": "^18.3.5",

inventory/src/App.tsx

@@ -15,6 +15,8 @@ import { RequireAuth } from './components/auth/RequireAuth';
 import Forecasting from "@/pages/Forecasting";
 import { Vendors } from '@/pages/Vendors';
 import { Categories } from '@/pages/Categories';
+import { Import } from '@/pages/import/Import';
+import { ChakraProvider } from '@chakra-ui/react';
 
 const queryClient = new QueryClient();
 
@@ -50,6 +52,7 @@ function App() {
 
   return (
     <QueryClientProvider client={queryClient}>
+      <ChakraProvider>
         <Toaster richColors position="top-center" />
         <Routes>
           <Route path="/login" element={<Login />} />
@@ -60,6 +63,7 @@ function App() {
           }>
             <Route path="/" element={<Dashboard />} />
             <Route path="/products" element={<Products />} />
+            <Route path="/import" element={<Import />} />
             <Route path="/categories" element={<Categories />} />
             <Route path="/vendors" element={<Vendors />} />
             <Route path="/orders" element={<Orders />} />
@@ -70,6 +74,7 @@ function App() {
             <Route path="*" element={<Navigate to="/" replace />} />
           </Route>
         </Routes>
+      </ChakraProvider>
     </QueryClientProvider>
   );
 }

inventory/src/components/layout/AppSidebar.tsx

@@ -8,6 +8,7 @@ import {
   LogOut,
   Users,
   Tags,
+  FileSpreadsheet,
 } from "lucide-react";
 import { IconCrystalBall } from "@tabler/icons-react";
 import {
@@ -35,6 +36,11 @@ const items = [
     icon: Package,
     url: "/products",
   },
+  {
+    title: "Import",
+    icon: FileSpreadsheet,
+    url: "/import",
+  },
   {
     title: "Forecasting",
     icon: IconCrystalBall,

inventory/src/components/settings/PerformanceMetrics.tsx

@@ -133,6 +133,10 @@ export function PerformanceMetrics() {
     }
   };
 
+  function getCategoryName(_cat_id: number): import("react").ReactNode {
+    throw new Error('Function not implemented.');
+  }
+
   return (
     <div className="max-w-[700px] space-y-4">
       {/* Lead Time Thresholds Card */}
@@ -205,11 +209,11 @@ export function PerformanceMetrics() {
             <Table>
               <TableHeader>
                 <TableRow>
-                  <TableHead>Category</TableHead>
-                  <TableHead>Vendor</TableHead>
-                  <TableHead className="text-right">A Threshold</TableHead>
-                  <TableHead className="text-right">B Threshold</TableHead>
-                  <TableHead className="text-right">Period Days</TableHead>
+                  <TableCell>Category</TableCell>
+                  <TableCell>Vendor</TableCell>
+                  <TableCell className="text-right">A Threshold</TableCell>
+                  <TableCell className="text-right">B Threshold</TableCell>
+                  <TableCell className="text-right">Period Days</TableCell>
                 </TableRow>
               </TableHeader>
               <TableBody>
@@ -242,10 +246,10 @@ export function PerformanceMetrics() {
             <Table>
               <TableHeader>
                 <TableRow>
-                  <TableHead>Category</TableHead>
-                  <TableHead>Vendor</TableHead>
-                  <TableHead className="text-right">Period Days</TableHead>
-                  <TableHead className="text-right">Target Rate</TableHead>
+                  <TableCell>Category</TableCell>
+                  <TableCell>Vendor</TableCell>
+                  <TableCell className="text-right">Period Days</TableCell>
+                  <TableCell className="text-right">Target Rate</TableCell>
                 </TableRow>
               </TableHeader>
               <TableBody>

inventory/src/components/settings/StockManagement.tsx

@@ -5,7 +5,6 @@ import { Input } from "@/components/ui/input";
 import { Label } from "@/components/ui/label";
 import { toast } from "sonner";
 import config from '../../config';
-import { Table, TableBody, TableCell, TableHeader, TableRow } from "@/components/ui/table";
 
 interface StockThreshold {
   id: number;
@@ -244,54 +243,6 @@ export function StockManagement() {
           </div>
         </CardContent>
       </Card>
-
-      <Table>
-        <TableHeader>
-          <TableRow>
-            <TableHead>Category</TableHead>
-            <TableHead>Vendor</TableHead>
-            <TableHead className="text-right">Critical Days</TableHead>
-            <TableHead className="text-right">Reorder Days</TableHead>
-            <TableHead className="text-right">Overstock Days</TableHead>
-            <TableHead className="text-right">Low Stock</TableHead>
-            <TableHead className="text-right">Min Reorder</TableHead>
-          </TableRow>
-        </TableHeader>
-        <TableBody>
-          {stockThresholds.map((threshold) => (
-            <TableRow key={`${threshold.cat_id}-${threshold.vendor}`}>
-              <TableCell>{threshold.cat_id ? getCategoryName(threshold.cat_id) : 'Global'}</TableCell>
-              <TableCell>{threshold.vendor || 'All Vendors'}</TableCell>
-              <TableCell className="text-right">{threshold.critical_days}</TableCell>
-              <TableCell className="text-right">{threshold.reorder_days}</TableCell>
-              <TableCell className="text-right">{threshold.overstock_days}</TableCell>
-              <TableCell className="text-right">{threshold.low_stock_threshold}</TableCell>
-              <TableCell className="text-right">{threshold.min_reorder_quantity}</TableCell>
-            </TableRow>
-          ))}
-        </TableBody>
-      </Table>
-
-      <Table>
-        <TableHeader>
-          <TableRow>
-            <TableHead>Category</TableHead>
-            <TableHead>Vendor</TableHead>
-            <TableHead className="text-right">Coverage Days</TableHead>
-            <TableHead className="text-right">Service Level</TableHead>
-          </TableRow>
-        </TableHeader>
-        <TableBody>
-          {safetyStockConfigs.map((config) => (
-            <TableRow key={`${config.cat_id}-${config.vendor}`}>
-              <TableCell>{config.cat_id ? getCategoryName(config.cat_id) : 'Global'}</TableCell>
-              <TableCell>{config.vendor || 'All Vendors'}</TableCell>
-              <TableCell className="text-right">{config.coverage_days}</TableCell>
-              <TableCell className="text-right">{config.service_level}%</TableCell>
-            </TableRow>
-          ))}
-        </TableBody>
-      </Table>
     </div>
   );
 } 

inventory/src/components/ui/card.tsx

@@ -9,7 +9,7 @@ const Card = React.forwardRef<
   <div
     ref={ref}
     className={cn(
-      "rounded-xl border bg-card text-card-foreground shadow",
+      "rounded-lg border bg-card text-card-foreground shadow-sm",
       className
     )}
     {...props}
@@ -30,22 +30,25 @@ const CardHeader = React.forwardRef<
 CardHeader.displayName = "CardHeader"
 
 const CardTitle = React.forwardRef<
-  HTMLDivElement,
-  React.HTMLAttributes<HTMLDivElement>
+  HTMLParagraphElement,
+  React.HTMLAttributes<HTMLHeadingElement>
 >(({ className, ...props }, ref) => (
-  <div
+  <h3
     ref={ref}
-    className={cn("font-semibold leading-none tracking-tight", className)}
+    className={cn(
+      "text-2xl font-semibold leading-none tracking-tight",
+      className
+    )}
     {...props}
   />
 ))
 CardTitle.displayName = "CardTitle"
 
 const CardDescription = React.forwardRef<
-  HTMLDivElement,
-  React.HTMLAttributes<HTMLDivElement>
+  HTMLParagraphElement,
+  React.HTMLAttributes<HTMLParagraphElement>
 >(({ className, ...props }, ref) => (
-  <div
+  <p
     ref={ref}
     className={cn("text-sm text-muted-foreground", className)}
     {...props}
@@ -73,4 +76,16 @@ const CardFooter = React.forwardRef<
 ))
 CardFooter.displayName = "CardFooter"
 
-export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent }
+const ScrollArea = React.forwardRef<
+  HTMLDivElement,
+  React.HTMLAttributes<HTMLDivElement>
+>(({ className, ...props }, ref) => (
+  <div
+    ref={ref}
+    className={cn("overflow-auto", className)}
+    {...props}
+  />
+))
+ScrollArea.displayName = "ScrollArea"
+
+export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent, ScrollArea }

inventory/src/components/ui/checkbox.tsx

@@ -0,0 +1,28 @@
+import * as React from "react"
+import * as CheckboxPrimitive from "@radix-ui/react-checkbox"
+import { Check } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const Checkbox = React.forwardRef<
+  React.ElementRef<typeof CheckboxPrimitive.Root>,
+  React.ComponentPropsWithoutRef<typeof CheckboxPrimitive.Root>
+>(({ className, ...props }, ref) => (
+  <CheckboxPrimitive.Root
+    ref={ref}
+    className={cn(
+      "peer h-4 w-4 shrink-0 rounded-sm border border-primary shadow focus-visible:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50 data-[state=checked]:bg-primary data-[state=checked]:text-primary-foreground",
+      className
+    )}
+    {...props}
+  >
+    <CheckboxPrimitive.Indicator
+      className={cn("flex items-center justify-center text-current")}
+    >
+      <Check className="h-4 w-4" />
+    </CheckboxPrimitive.Indicator>
+  </CheckboxPrimitive.Root>
+))
+Checkbox.displayName = CheckboxPrimitive.Root.displayName
+
+export { Checkbox }

inventory/src/components/ui/code.tsx

@@ -0,0 +1,23 @@
+import * as React from "react"
+
+import { cn } from "@/lib/utils"
+
+export interface CodeProps extends React.HTMLAttributes<HTMLPreElement> {}
+
+const Code = React.forwardRef<HTMLPreElement, CodeProps>(
+  ({ className, ...props }, ref) => {
+    return (
+      <pre
+        ref={ref}
+        className={cn(
+          "rounded-lg bg-muted px-4 py-4 font-mono text-sm",
+          className
+        )}
+        {...props}
+      />
+    )
+  }
+)
+Code.displayName = "Code"
+
+export { Code } 

inventory/src/components/ui/radio-group.tsx

@@ -0,0 +1,44 @@
+"use client"
+
+import * as React from "react"
+import * as RadioGroupPrimitive from "@radix-ui/react-radio-group"
+import { Circle } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const RadioGroup = React.forwardRef<
+  React.ElementRef<typeof RadioGroupPrimitive.Root>,
+  React.ComponentPropsWithoutRef<typeof RadioGroupPrimitive.Root>
+>(({ className, ...props }, ref) => {
+  return (
+    <RadioGroupPrimitive.Root
+      className={cn("grid gap-2", className)}
+      {...props}
+      ref={ref}
+    />
+  )
+})
+RadioGroup.displayName = RadioGroupPrimitive.Root.displayName
+
+const RadioGroupItem = React.forwardRef<
+  React.ElementRef<typeof RadioGroupPrimitive.Item>,
+  React.ComponentPropsWithoutRef<typeof RadioGroupPrimitive.Item>
+>(({ className, ...props }, ref) => {
+  return (
+    <RadioGroupPrimitive.Item
+      ref={ref}
+      className={cn(
+        "aspect-square h-4 w-4 rounded-full border border-primary text-primary shadow focus:outline-none focus-visible:ring-1 focus-visible:ring-ring disabled:cursor-not-allowed disabled:opacity-50",
+        className
+      )}
+      {...props}
+    >
+      <RadioGroupPrimitive.Indicator className="flex items-center justify-center">
+        <Circle className="h-3.5 w-3.5 fill-primary" />
+      </RadioGroupPrimitive.Indicator>
+    </RadioGroupPrimitive.Item>
+  )
+})
+RadioGroupItem.displayName = RadioGroupPrimitive.Item.displayName
+
+export { RadioGroup, RadioGroupItem }

inventory/src/components/ui/toast.tsx

@@ -0,0 +1,127 @@
+import * as React from "react"
+import * as ToastPrimitives from "@radix-ui/react-toast"
+import { cva, type VariantProps } from "class-variance-authority"
+import { X } from "lucide-react"
+
+import { cn } from "@/lib/utils"
+
+const ToastProvider = ToastPrimitives.Provider
+
+const ToastViewport = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Viewport>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Viewport>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Viewport
+    ref={ref}
+    className={cn(
+      "fixed top-0 z-[100] flex max-h-screen w-full flex-col-reverse p-4 sm:bottom-0 sm:right-0 sm:top-auto sm:flex-col md:max-w-[420px]",
+      className
+    )}
+    {...props}
+  />
+))
+ToastViewport.displayName = ToastPrimitives.Viewport.displayName
+
+const toastVariants = cva(
+  "group pointer-events-auto relative flex w-full items-center justify-between space-x-2 overflow-hidden rounded-md border p-4 pr-6 shadow-lg transition-all data-[swipe=cancel]:translate-x-0 data-[swipe=end]:translate-x-[var(--radix-toast-swipe-end-x)] data-[swipe=move]:translate-x-[var(--radix-toast-swipe-move-x)] data-[swipe=move]:transition-none data-[state=open]:animate-in data-[state=closed]:animate-out data-[swipe=end]:animate-out data-[state=closed]:fade-out-80 data-[state=closed]:slide-out-to-right-full data-[state=open]:slide-in-from-top-full data-[state=open]:sm:slide-in-from-bottom-full",
+  {
+    variants: {
+      variant: {
+        default: "border bg-background text-foreground",
+        destructive:
+          "destructive group border-destructive bg-destructive text-destructive-foreground",
+      },
+    },
+    defaultVariants: {
+      variant: "default",
+    },
+  }
+)
+
+const Toast = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Root>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Root> &
+    VariantProps<typeof toastVariants>
+>(({ className, variant, ...props }, ref) => {
+  return (
+    <ToastPrimitives.Root
+      ref={ref}
+      className={cn(toastVariants({ variant }), className)}
+      {...props}
+    />
+  )
+})
+Toast.displayName = ToastPrimitives.Root.displayName
+
+const ToastAction = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Action>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Action>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Action
+    ref={ref}
+    className={cn(
+      "inline-flex h-8 shrink-0 items-center justify-center rounded-md border bg-transparent px-3 text-sm font-medium transition-colors hover:bg-secondary focus:outline-none focus:ring-1 focus:ring-ring disabled:pointer-events-none disabled:opacity-50 group-[.destructive]:border-muted/40 group-[.destructive]:hover:border-destructive/30 group-[.destructive]:hover:bg-destructive group-[.destructive]:hover:text-destructive-foreground group-[.destructive]:focus:ring-destructive",
+      className
+    )}
+    {...props}
+  />
+))
+ToastAction.displayName = ToastPrimitives.Action.displayName
+
+const ToastClose = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Close>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Close>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Close
+    ref={ref}
+    className={cn(
+      "absolute right-1 top-1 rounded-md p-1 text-foreground/50 opacity-0 transition-opacity hover:text-foreground focus:opacity-100 focus:outline-none focus:ring-1 group-hover:opacity-100 group-[.destructive]:text-red-300 group-[.destructive]:hover:text-red-50 group-[.destructive]:focus:ring-red-400 group-[.destructive]:focus:ring-offset-red-600",
+      className
+    )}
+    toast-close=""
+    {...props}
+  >
+    <X className="h-4 w-4" />
+  </ToastPrimitives.Close>
+))
+ToastClose.displayName = ToastPrimitives.Close.displayName
+
+const ToastTitle = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Title>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Title>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Title
+    ref={ref}
+    className={cn("text-sm font-semibold [&+div]:text-xs", className)}
+    {...props}
+  />
+))
+ToastTitle.displayName = ToastPrimitives.Title.displayName
+
+const ToastDescription = React.forwardRef<
+  React.ElementRef<typeof ToastPrimitives.Description>,
+  React.ComponentPropsWithoutRef<typeof ToastPrimitives.Description>
+>(({ className, ...props }, ref) => (
+  <ToastPrimitives.Description
+    ref={ref}
+    className={cn("text-sm opacity-90", className)}
+    {...props}
+  />
+))
+ToastDescription.displayName = ToastPrimitives.Description.displayName
+
+type ToastProps = React.ComponentPropsWithoutRef<typeof Toast>
+
+type ToastActionElement = React.ReactElement<typeof ToastAction>
+
+export {
+  type ToastProps,
+  type ToastActionElement,
+  ToastProvider,
+  ToastViewport,
+  Toast,
+  ToastTitle,
+  ToastDescription,
+  ToastClose,
+  ToastAction,
+}

inventory/src/components/ui/toaster.tsx

@@ -0,0 +1,33 @@
+import { useToast } from "@/hooks/use-toast"
+import {
+  Toast,
+  ToastClose,
+  ToastDescription,
+  ToastProvider,
+  ToastTitle,
+  ToastViewport,
+} from "@/components/ui/toast"
+
+export function Toaster() {
+  const { toasts } = useToast()
+
+  return (
+    <ToastProvider>
+      {toasts.map(function ({ id, title, description, action, ...props }) {
+        return (
+          <Toast key={id} {...props}>
+            <div className="grid gap-1">
+              {title && <ToastTitle>{title}</ToastTitle>}
+              {description && (
+                <ToastDescription>{description}</ToastDescription>
+              )}
+            </div>
+            {action}
+            <ToastClose />
+          </Toast>
+        )
+      })}
+      <ToastViewport />
+    </ToastProvider>
+  )
+}

inventory/src/hooks/use-toast.ts

@@ -0,0 +1,194 @@
+"use client"
+
+// Inspired by react-hot-toast library
+import * as React from "react"
+
+import type {
+  ToastActionElement,
+  ToastProps,
+} from "@/components/ui/toast"
+
+const TOAST_LIMIT = 1
+const TOAST_REMOVE_DELAY = 1000000
+
+type ToasterToast = ToastProps & {
+  id: string
+  title?: React.ReactNode
+  description?: React.ReactNode
+  action?: ToastActionElement
+}
+
+const actionTypes = {
+  ADD_TOAST: "ADD_TOAST",
+  UPDATE_TOAST: "UPDATE_TOAST",
+  DISMISS_TOAST: "DISMISS_TOAST",
+  REMOVE_TOAST: "REMOVE_TOAST",
+} as const
+
+let count = 0
+
+function genId() {
+  count = (count + 1) % Number.MAX_SAFE_INTEGER
+  return count.toString()
+}
+
+type ActionType = typeof actionTypes
+
+type Action =
+  | {
+      type: ActionType["ADD_TOAST"]
+      toast: ToasterToast
+    }
+  | {
+      type: ActionType["UPDATE_TOAST"]
+      toast: Partial<ToasterToast>
+    }
+  | {
+      type: ActionType["DISMISS_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+  | {
+      type: ActionType["REMOVE_TOAST"]
+      toastId?: ToasterToast["id"]
+    }
+
+interface State {
+  toasts: ToasterToast[]
+}
+
+const toastTimeouts = new Map<string, ReturnType<typeof setTimeout>>()
+
+const addToRemoveQueue = (toastId: string) => {
+  if (toastTimeouts.has(toastId)) {
+    return
+  }
+
+  const timeout = setTimeout(() => {
+    toastTimeouts.delete(toastId)
+    dispatch({
+      type: "REMOVE_TOAST",
+      toastId: toastId,
+    })
+  }, TOAST_REMOVE_DELAY)
+
+  toastTimeouts.set(toastId, timeout)
+}
+
+export const reducer = (state: State, action: Action): State => {
+  switch (action.type) {
+    case "ADD_TOAST":
+      return {
+        ...state,
+        toasts: [action.toast, ...state.toasts].slice(0, TOAST_LIMIT),
+      }
+
+    case "UPDATE_TOAST":
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === action.toast.id ? { ...t, ...action.toast } : t
+        ),
+      }
+
+    case "DISMISS_TOAST": {
+      const { toastId } = action
+
+      // ! Side effects ! - This could be extracted into a dismissToast() action,
+      // but I'll keep it here for simplicity
+      if (toastId) {
+        addToRemoveQueue(toastId)
+      } else {
+        state.toasts.forEach((toast) => {
+          addToRemoveQueue(toast.id)
+        })
+      }
+
+      return {
+        ...state,
+        toasts: state.toasts.map((t) =>
+          t.id === toastId || toastId === undefined
+            ? {
+                ...t,
+                open: false,
+              }
+            : t
+        ),
+      }
+    }
+    case "REMOVE_TOAST":
+      if (action.toastId === undefined) {
+        return {
+          ...state,
+          toasts: [],
+        }
+      }
+      return {
+        ...state,
+        toasts: state.toasts.filter((t) => t.id !== action.toastId),
+      }
+  }
+}
+
+const listeners: Array<(state: State) => void> = []
+
+let memoryState: State = { toasts: [] }
+
+function dispatch(action: Action) {
+  memoryState = reducer(memoryState, action)
+  listeners.forEach((listener) => {
+    listener(memoryState)
+  })
+}
+
+type Toast = Omit<ToasterToast, "id">
+
+function toast({ ...props }: Toast) {
+  const id = genId()
+
+  const update = (props: ToasterToast) =>
+    dispatch({
+      type: "UPDATE_TOAST",
+      toast: { ...props, id },
+    })
+  const dismiss = () => dispatch({ type: "DISMISS_TOAST", toastId: id })
+
+  dispatch({
+    type: "ADD_TOAST",
+    toast: {
+      ...props,
+      id,
+      open: true,
+      onOpenChange: (open) => {
+        if (!open) dismiss()
+      },
+    },
+  })
+
+  return {
+    id: id,
+    dismiss,
+    update,
+  }
+}
+
+function useToast() {
+  const [state, setState] = React.useState<State>(memoryState)
+
+  React.useEffect(() => {
+    listeners.push(setState)
+    return () => {
+      const index = listeners.indexOf(setState)
+      if (index > -1) {
+        listeners.splice(index, 1)
+      }
+    }
+  }, [state])
+
+  return {
+    ...state,
+    toast,
+    dismiss: (toastId?: string) => dispatch({ type: "DISMISS_TOAST", toastId }),
+  }
+}
+
+export { useToast, toast }

inventory/src/lib/react-spreadsheet-import/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022 UGNIS,
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

inventory/src/lib/react-spreadsheet-import/README.md

@@ -0,0 +1,341 @@
+<h1 align="center">RSI react-spreadsheet-import ⚡️</h1>
+
+<div align="center">
+  
+![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/UgnisSoftware/react-spreadsheet-import/test.yaml)
+![GitHub](https://img.shields.io/github/license/UgnisSoftware/react-spreadsheet-import) [![npm](https://img.shields.io/npm/v/react-spreadsheet-import)](https://www.npmjs.com/package/react-spreadsheet-import)
+  
+</div>
+<br />
+
+A component used for importing XLS / XLSX / CSV documents built with [**Chakra UI**](https://chakra-ui.com). Import flow combines:
+
+- 📥 Uploader
+- ⚙️ Parser
+- 📊 File preview
+- 🧪 UI for column mapping
+- ✏ UI for validating and editing data
+
+✨ [**Demo**](https://ugnissoftware.github.io/react-spreadsheet-import/iframe.html?id=react-spreadsheet-import--basic&args=&viewMode=story) ✨
+<br />
+
+## Features
+
+- Custom styles - edit Chakra UI theme to match your project's styles 🎨
+- Custom validation rules - make sure valid data is being imported, easily spot and correct errors
+- Hooks - alter raw data after upload or make adjustments on data changes
+- Auto-mapping columns - automatically map most likely value to your template values, e.g. `name` -> `firstName`
+  <br />
+
+![rsi-preview](https://user-images.githubusercontent.com/45755753/159503528-90aacb69-128f-4ece-b45b-ab97d403a9d3.gif)
+
+## Figma
+
+We provide full figma designs. You can copy the designs
+[here](https://www.figma.com/community/file/1080776795891439629)
+
+## Getting started
+
+```sh
+npm i react-spreadsheet-import
+```
+
+Using the component: (it's up to you when the flow is open and what you do on submit with the imported data)
+
+```tsx
+import { ReactSpreadsheetImport } from "react-spreadsheet-import";
+
+<ReactSpreadsheetImport isOpen={isOpen} onClose={onClose} onSubmit={onSubmit} fields={fields} />
+```
+
+## Required Props
+
+```tsx
+  // Determines if modal is visible.
+  isOpen: Boolean
+  // Called when flow is closed without reaching submit.
+  onClose: () => void
+  // Called after user completes the flow. Provides data array, where data keys matches your field keys. 
+  onSubmit: (data, file) => void | Promise<any>
+```
+
+### Fields
+
+Fields describe what data you are trying to collect.
+
+```tsx
+const fields = [
+  {
+    // Visible in table header and when matching columns.
+    label: "Name",
+    // This is the key used for this field when we call onSubmit.
+    key: "name",
+    // Allows for better automatic column matching. Optional.
+    alternateMatches: ["first name", "first"],
+    // Used when editing and validating information.
+    fieldType: {
+      // There are 3 types - "input" / "checkbox" / "select".
+      type: "input",
+    },
+    // Used in the first step to provide an example of what data is expected in this field. Optional.
+    example: "Stephanie",
+    // Can have multiple validations that are visible in Validation Step table.
+    validations: [
+      {
+        // Can be "required" / "unique" / "regex"
+        rule: "required",
+        errorMessage: "Name is required",
+        // There can be "info" / "warning" / "error" levels. Optional. Default "error".
+        level: "error",
+      },
+    ],
+  },
+] as const
+```
+
+## Optional Props
+
+### Hooks
+
+You can transform and validate data with custom hooks. There are hooks after each step:
+
+- **uploadStepHook** - runs only once after uploading the file.
+- **selectHeaderStepHook** - runs only once after selecting the header row in spreadsheet.
+- **matchColumnsStepHook** - runs only once after column matching. Operations on data that are expensive should be done here.
+
+The last step - validation step has 2 unique hooks that run only in that step with different performance tradeoffs:
+
+- **tableHook** - runs at the start and on any change. Runs on all rows. Very expensive, but can change rows that depend on other rows.
+- **rowHook** - runs at the start and on any row change. Runs only on the rows changed. Fastest, most validations and transformations should be done here.
+
+Example:
+
+```tsx
+<ReactSpreadsheetImport
+  rowHook={(data, addError) => {
+    // Validation
+    if (data.name === "John") {
+      addError("name", { message: "No Johns allowed", level: "info" })
+    }
+    // Transformation
+    return { ...data, name: "Not John" }
+    // Sorry John
+  }}
+/>
+```
+
+### Initial state
+
+In rare case when you need to skip the beginning of the flow, you can start the flow from any of the steps.
+
+- **initialStepState** - initial state of component that will be rendered on load. 
+
+```tsx
+  initialStepState?: StepState
+  
+  type StepState =
+    | {
+        type: StepType.upload
+      }
+    | {
+        type: StepType.selectSheet
+        workbook: XLSX.WorkBook
+      }
+    | {
+        type: StepType.selectHeader
+        data: RawData[]
+      }
+    | {
+        type: StepType.matchColumns
+        data: RawData[]
+        headerValues: RawData
+      }
+    | {
+        type: StepType.validateData
+        data: any[]
+      }
+
+  type RawData = Array<string | undefined>
+
+  // XLSX.workbook type is native to SheetJS and can be viewed here: https://github.com/SheetJS/sheetjs/blob/83ddb4c1203f6bac052d8c1608b32fead02ea32f/types/index.d.ts#L269
+```
+
+Example:
+
+```tsx
+import { ReactSpreadsheetImport, StepType } from "react-spreadsheet-import";
+
+<ReactSpreadsheetImport
+  initialStepState={{
+    type: StepType.matchColumns,
+    data: [
+      ["Josh", "2"],
+      ["Charlie", "3"],
+      ["Lena", "50"],
+    ],
+    headerValues: ["name", "age"],
+  }}
+/>
+```
+
+### Dates and time
+
+Excel stores dates and times as numbers - offsets from an epoch. When reading xlsx files SheetJS provides date formatting helpers.
+**Default date import format** is `yyyy-mm-dd`. Date parsing with SheetJS sometimes yields unexpected results, therefore thorough date validations are recommended.
+
+- **dateFormat** - sets SheetJS `dateNF` option. Can be used to format dates when importing sheet data.
+- **parseRaw** - sets SheetJS `raw` option. If `true`, date formatting will be applied to XLSX date fields only. Default is `true`
+
+Common date-time formats can be viewed [here](https://docs.sheetjs.com/docs/csf/features/dates/#date-and-time-number-formats).
+
+### Other optional props
+
+```tsx
+  // Allows submitting with errors. Default: true
+  allowInvalidSubmit?: boolean
+  // Translations for each text. See customisation bellow
+  translations?: object
+  // Theme configuration passed to underlying Chakra-UI. See customisation bellow
+  customTheme?: object
+  // Specifies maximum number of rows for a single import
+  maxRecords?: number
+  // Maximum upload filesize (in bytes)
+  maxFileSize?: number
+  // Automatically map imported headers to specified fields if possible. Default: true
+  autoMapHeaders?: boolean
+  // When field type is "select", automatically match values if possible. Default: false
+  autoMapSelectValues?: boolean
+  // Headers matching accuracy: 1 for strict and up for more flexible matching. Default: 2
+  autoMapDistance?: number
+  // Enable navigation in stepper component and show back button. Default: false
+  isNavigationEnabled?: boolean
+```
+
+## Customisation
+
+### Customising styles (colors, fonts)
+
+You can see default theme we use [here](https://github.com/UgnisSoftware/react-spreadsheet-import/blob/master/src/theme.ts). Your override should match this object's structure.
+
+There are 3 ways you can style the component:
+
+1.) Change theme colors globally
+
+```jsx
+    <ReactSpreadsheetImport
+        {...mockRsiValues}
+        isOpen={isOpen}
+        onClose={onClose}
+        onSubmit={setData}
+        customTheme={{
+          colors: {
+            background: 'white',
+            ...
+            rsi: {
+              // your brand colors should go here
+              50: '...'
+              ...
+              500: 'teal',
+              ...
+              900: "...",
+            },
+          },
+        }}
+      />
+```
+
+<img width="1189" alt="Screenshot 2022-04-13 at 10 24 34" src="https://user-images.githubusercontent.com/5903616/163123718-15c05ad8-243b-4a81-8141-c47216047468.png">
+
+2.) Change all components of the same type, like all Buttons, at the same time
+
+```jsx
+<ReactSpreadsheetImport
+  {...mockRsiValues}
+  isOpen={isOpen}
+  onClose={onClose}
+  onSubmit={setData}
+  customTheme={{
+    components: {
+      Button: {
+        baseStyle: {
+          borderRadius: "none",
+        },
+        defaultProps: {
+          colorScheme: "yellow",
+        },
+      },
+    },
+  }}
+/>
+```
+
+<img width="1191" alt="Screenshot 2022-04-13 at 11 04 30" src="https://user-images.githubusercontent.com/5903616/163130213-82f955b4-5081-49e0-8f43-8857d480dacd.png">
+ 
+3.) Change components specifically in each Step.
+```jsx
+    <ReactSpreadsheetImport
+        {...mockRsiValues}
+        isOpen={isOpen}
+        onClose={onClose}
+        onSubmit={setData}
+        customTheme={{
+          components: {
+            UploadStep: {
+              baseStyle: {
+                dropzoneButton: {
+                  bg: "red",
+                },
+              },
+            },
+          },
+        }}
+      />
+```
+<img width="1182" alt="Screenshot 2022-04-13 at 10 21 58" src="https://user-images.githubusercontent.com/5903616/163123694-5b79179e-037e-4f9d-b1a9-6078f758bb7e.png">
+
+Underneath we use Chakra-UI, you can send in a custom theme for us to apply. Read more about themes [here](https://chakra-ui.com/docs/styled-system/theming/theme)
+
+### Changing text (translations)
+
+You can change any text in the flow:
+
+```tsx
+<ReactSpreadsheetImport
+  translations={{
+    uploadStep: {
+      title: "Upload Employees",
+    },
+  }}
+/>
+```
+
+You can see all the translation keys [here](https://github.com/UgnisSoftware/react-spreadsheet-import/blob/master/src/translationsRSIProps.ts)
+
+## VS other libraries
+
+Flatfile vs react-spreadsheet-import and Dromo vs react-spreadsheet-import:
+
+|                                | RSI            | Flatfile    | Dromo       |
+| ------------------------------ | -------------- | ----------- | ----------- |
+| Licence                        | MIT            | Proprietary | Proprietary |
+| Price                          | Free           | Paid        | Paid        |
+| Support                        | Github Issues  | Enterprise  | Enterprise  |
+| Self-host                      | Yes            | Paid        | Paid        |
+| Hosted solution                | In development | Yes         | Yes         |
+| On-prem deployment             | N/A            | Yes         | Yes         |
+| Hooks                          | Yes            | Yes         | Yes         |
+| Automatic header matching      | Yes            | Yes         | Yes         |
+| Data validation                | Yes            | Yes         | Yes         |
+| Custom styling                 | Yes            | Yes         | Yes         |
+| Translations                   | Yes            | Yes         | Yes         |
+| Trademarked words `Data Hooks` | No             | Yes         | No          |
+
+React-spreadsheet-import can be used as a free and open-source alternative to Flatfile and Dromo.
+
+## Contributing
+
+Feel free to open issues if you have any questions or notice bugs. If you want different component behaviour, consider forking the project.
+
+## Credits
+
+Created by Ugnis. [Julita Kriauciunaite](https://github.com/JulitorK) and [Karolis Masiulis](https://github.com/masiulis). You can contact us at `info@ugnis.com`

inventory/src/lib/react-spreadsheet-import/package.json

@@ -0,0 +1,174 @@
+{
+  "name": "react-spreadsheet-import",
+  "version": "4.7.1",
+  "description": "React spreadsheet import for xlsx and csv files with column matching and validation",
+  "main": "./dist-commonjs/index.js",
+  "module": "./dist/index.js",
+  "types": "./types/index.d.ts",
+  "files": [
+    "dist-commonjs",
+    "dist",
+    "types"
+  ],
+  "scripts": {
+    "start": "storybook dev -p 6006",
+    "test:unit": "jest",
+    "test:e2e": "npx playwright test",
+    "test:chromatic": "npx chromatic ",
+    "ts": "tsc",
+    "lint": "eslint \"src/**/*.{ts,tsx}\"",
+    "lint:fix": "eslint \"src/**/*.{ts,tsx}\" --fix",
+    "prebuild": "npm run clean",
+    "build": "rollup -c rollup.config.ts",
+    "build-storybook": "storybook build -o docs-build",
+    "release:patch": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version patch && git add -A && git push && git push --tags && npm publish",
+    "release:minor": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version minor && git add -A && git push && git push --tags && npm publish",
+    "release:major": "git checkout master && git pull && npm run test:unit && npm run ts && npm run build && npm version major && git add -A && git push && git push --tags && npm publish",
+    "clean": "rimraf dist dist-commonjs types"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/UgnisSoftware/react-spreadsheet-import.git"
+  },
+  "keywords": [
+    "React",
+    "spreadsheet",
+    "import",
+    "upload",
+    "csv",
+    "xlsx",
+    "validate",
+    "automatic",
+    "match"
+  ],
+  "author": {
+    "name": "Ugnis"
+  },
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/UgnisSoftware/react-spreadsheet-import/issues"
+  },
+  "homepage": "https://github.com/UgnisSoftware/react-spreadsheet-import#readme",
+  "peerDependencies": {
+    "react": ">=18",
+    "react-dom": ">=18"
+  },
+  "dependencies": {
+    "@chakra-ui/react": "^2.8.1",
+    "@emotion/react": "^11.11.1",
+    "@emotion/styled": "^11.11.0",
+    "chakra-react-select": "^4.7.5",
+    "chakra-ui-steps": "2.0.4",
+    "framer-motion": "^10.16.4",
+    "js-levenshtein": "1.1.6",
+    "lodash": "4.17.21",
+    "react-data-grid": "7.0.0-beta.13",
+    "react-dropzone": "14.2.3",
+    "react-icons": "4.11.0",
+    "uuid": "^9.0.1",
+    "xlsx-ugnis": "0.20.3"
+  },
+  "devDependencies": {
+    "@babel/core": "7.23.2",
+    "@babel/preset-env": "7.23.2",
+    "@babel/preset-react": "7.22.15",
+    "@babel/preset-typescript": "7.23.2",
+    "@emotion/jest": "11.11.0",
+    "@jest/types": "27.5.1",
+    "@playwright/test": "^1.39.0",
+    "@storybook/addon-essentials": "7.5.1",
+    "@storybook/addon-interactions": "7.5.1",
+    "@storybook/addon-links": "7.5.1",
+    "@storybook/blocks": "7.5.1",
+    "@storybook/cli": "7.5.1",
+    "@storybook/react": "7.5.1",
+    "@storybook/react-webpack5": "7.5.1",
+    "@storybook/testing-library": "^0.0.14-next.2",
+    "@testing-library/dom": "9.3.0",
+    "@testing-library/jest-dom": "5.16.5",
+    "@testing-library/react": "14.0.0",
+    "@testing-library/user-event": "14.4.3",
+    "@types/jest": "27.4.1",
+    "@types/js-levenshtein": "1.1.1",
+    "@types/node": "^20.8.7",
+    "@types/react": "18.2.6",
+    "@types/react-dom": "18.2.4",
+    "@types/styled-system": "5.1.16",
+    "@types/uuid": "9.0.1",
+    "@typescript-eslint/eslint-plugin": "5.59.7",
+    "@typescript-eslint/parser": "5.59.7",
+    "babel-loader": "9.1.3",
+    "chromatic": "^7.4.0",
+    "eslint": "8.41.0",
+    "eslint-config-prettier": "8.8.0",
+    "eslint-plugin-prettier": "4.2.1",
+    "eslint-plugin-react": "7.32.2",
+    "eslint-plugin-react-hooks": "4.6.0",
+    "jest": "27.5.1",
+    "jest-watch-typeahead": "1.0.0",
+    "lint-staged": "13.2.2",
+    "prettier": "2.8.8",
+    "prop-types": "15.8.1",
+    "react": "18.2.0",
+    "react-dom": "18.2.0",
+    "react-select-event": "5.5.1",
+    "rollup": "2.70.1",
+    "rollup-plugin-typescript2": "0.31.2",
+    "storybook": "7.5.1",
+    "ts-essentials": "9.3.2",
+    "ts-jest": "27.1.4",
+    "ttypescript": "1.5.15",
+    "typescript": "4.9.5",
+    "typescript-transform-paths": "3.4.6"
+  },
+  "overrides": {
+    "semver": "^7.5.3"
+  },
+  "lint-staged": {
+    "*.{ts,tsx}": "eslint",
+    "*.{js,ts,tsx,md,html,css,json}": "prettier --write"
+  },
+  "prettier": {
+    "tabWidth": 2,
+    "trailingComma": "all",
+    "semi": false,
+    "printWidth": 120
+  },
+  "jest": {
+    "preset": "ts-jest",
+    "testEnvironment": "jsdom",
+    "moduleFileExtensions": [
+      "ts",
+      "tsx",
+      "js",
+      "jsx",
+      "mjs"
+    ],
+    "transform": {
+      "^.+\\.(ts|tsx)?$": "ts-jest/dist",
+      "^.+\\.mjs$": "ts-jest/dist"
+    },
+    "moduleNameMapper": {
+      "~/(.*)": "<rootDir>/src/$1"
+    },
+    "modulePathIgnorePatterns": [
+      "<rootDir>/e2e/"
+    ],
+    "transformIgnorePatterns": [
+      "[/\\\\]node_modules[/\\\\].+\\.(js|jsx|mjs)$"
+    ],
+    "setupFiles": [
+      "./src/tests/setup.ts"
+    ],
+    "globals": {
+      "ts-jest": {
+        "tsconfig": "tsconfig.json"
+      }
+    },
+    "watchPlugins": [
+      "jest-watch-typeahead/filename",
+      "jest-watch-typeahead/testname"
+    ]
+  },
+  "readme": "ERROR: No README data found!"
+}

inventory/src/lib/react-spreadsheet-import/src/ReactSpreadsheetImport.tsx

@@ -0,0 +1,41 @@
+import merge from "lodash/merge"
+
+import { Steps } from "./steps/Steps"
+import { rtlThemeSupport, themeOverrides } from "./theme"
+import { Providers } from "./components/Providers"
+import type { RsiProps } from "./types"
+import { ModalWrapper } from "./components/ModalWrapper"
+import { translations } from "./translationsRSIProps"
+
+export const defaultTheme = themeOverrides
+
+export const defaultRSIProps: Partial<RsiProps<any>> = {
+  autoMapHeaders: true,
+  autoMapSelectValues: false,
+  allowInvalidSubmit: true,
+  autoMapDistance: 2,
+  isNavigationEnabled: false,
+  translations: translations,
+  uploadStepHook: async (value) => value,
+  selectHeaderStepHook: async (headerValues, data) => ({ headerValues, data }),
+  matchColumnsStepHook: async (table) => table,
+  dateFormat: "yyyy-mm-dd", // ISO 8601,
+  parseRaw: true,
+} as const
+
+export const ReactSpreadsheetImport = <T extends string>(propsWithoutDefaults: RsiProps<T>) => {
+  const props = merge({}, defaultRSIProps, propsWithoutDefaults)
+  const mergedTranslations =
+    props.translations !== translations ? merge(translations, props.translations) : translations
+  const mergedThemes = props.rtl
+    ? merge(defaultTheme, rtlThemeSupport, props.customTheme)
+    : merge(defaultTheme, props.customTheme)
+
+  return (
+    <Providers theme={mergedThemes} rsiValues={{ ...props, translations: mergedTranslations }}>
+      <ModalWrapper isOpen={props.isOpen} onClose={props.onClose}>
+        <Steps />
+      </ModalWrapper>
+    </Providers>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/ModalWrapper.tsx

@@ -0,0 +1,88 @@
+import type React from "react"
+import {
+  Dialog,
+  DialogContent,
+  DialogOverlay,
+  DialogPortal,
+  DialogClose,
+} from "@/components/ui/dialog"
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+  AlertDialogTrigger,
+  AlertDialogPortal,
+  AlertDialogOverlay,
+} from "@/components/ui/alert-dialog"
+import { useRsi } from "../hooks/useRsi"
+import { useState } from "react"
+
+type Props = {
+  children: React.ReactNode
+  isOpen: boolean
+  onClose: () => void
+}
+
+export const ModalWrapper = ({ children, isOpen, onClose }: Props) => {
+  const { rtl, translations } = useRsi()
+  const [showCloseAlert, setShowCloseAlert] = useState(false)
+  
+  return (
+    <>
+      <Dialog open={isOpen} onOpenChange={() => setShowCloseAlert(true)} modal>
+        <DialogPortal>
+          <DialogOverlay className="bg-background/80 backdrop-blur-sm" />
+          <DialogContent
+            onEscapeKeyDown={(e) => {
+              e.preventDefault()
+              setShowCloseAlert(true)
+            }}
+            onPointerDownOutside={(e) => e.preventDefault()}
+            className="fixed left-[50%] top-[50%] translate-x-[-50%] translate-y-[-50%] w-[calc(100%-2rem)] h-[calc(100%-2rem)] max-w-[100vw] max-h-[100vh] flex flex-col overflow-hidden rounded-lg border bg-background p-0 shadow-lg sm:w-[calc(100%-3rem)] sm:h-[calc(100%-3rem)] md:w-[calc(100%-4rem)] md:h-[calc(100%-4rem)]"
+          >
+            <AlertDialog>
+              <AlertDialogTrigger asChild>
+                <DialogClose className="absolute right-4 top-4" onClick={(e) => {
+                  e.preventDefault()
+                  setShowCloseAlert(true)
+                }} />
+              </AlertDialogTrigger>
+            </AlertDialog>
+            <div dir={rtl ? "rtl" : "ltr"} className="flex-1 overflow-auto">
+              {children}
+            </div>
+          </DialogContent>
+        </DialogPortal>
+      </Dialog>
+
+      <AlertDialog open={showCloseAlert} onOpenChange={setShowCloseAlert}>
+        <AlertDialogPortal>
+          <AlertDialogOverlay className="z-[1400]" />
+          <AlertDialogContent className="z-[1500]">
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {translations.alerts.confirmClose.headerTitle}
+              </AlertDialogTitle>
+              <AlertDialogDescription>
+                {translations.alerts.confirmClose.bodyText}
+              </AlertDialogDescription>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel onClick={() => setShowCloseAlert(false)}>
+                {translations.alerts.confirmClose.cancelButtonTitle}
+              </AlertDialogCancel>
+              <AlertDialogAction onClick={onClose}>
+                {translations.alerts.confirmClose.exitButtonTitle}
+              </AlertDialogAction>
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialogPortal>
+      </AlertDialog>
+    </>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/Providers.tsx

@@ -0,0 +1,33 @@
+import { ChakraProvider, extendTheme } from "@chakra-ui/react"
+import { createContext } from "react"
+import type { RsiProps } from "../types"
+import type { CustomTheme } from "../theme"
+
+export const RsiContext = createContext({} as any)
+
+type ProvidersProps<T extends string> = {
+  children: React.ReactNode
+  theme: CustomTheme
+  rsiValues: RsiProps<T>
+}
+
+export const rootId = "chakra-modal-rsi"
+
+export const Providers = <T extends string>({ children, theme, rsiValues }: ProvidersProps<T>) => {
+  const mergedTheme = extendTheme(theme)
+
+  if (!rsiValues.fields) {
+    throw new Error("Fields must be provided to react-spreadsheet-import")
+  }
+
+  return (
+    <RsiContext.Provider value={rsiValues}>
+      <ChakraProvider>
+        {/* cssVarsRoot used to override RSI defaultTheme but not the rest of chakra defaultTheme */}
+        <ChakraProvider cssVarsRoot={`#${rootId}`} theme={mergedTheme}>
+          {children}
+        </ChakraProvider>
+      </ChakraProvider>
+    </RsiContext.Provider>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/components/Table.tsx

@@ -0,0 +1,23 @@
+import type { DataGridProps, Column } from "react-data-grid"
+import DataGrid from "react-data-grid"
+import { useRsi } from "../hooks/useRsi"
+
+export type { Column }
+
+export type Props<TRow> = DataGridProps<TRow> & {
+  rowHeight?: number
+  hiddenHeader?: boolean
+  className?: string
+  style?: React.CSSProperties
+}
+
+export const Table = <TRow,>({ className, ...props }: Props<TRow>) => {
+  const { rtl } = useRsi()
+  return (
+    <DataGrid 
+      className={"rdg-light " + (className || "")} 
+      direction={rtl ? "rtl" : "ltr"} 
+      {...props} 
+    />
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/hooks/useRsi.ts

@@ -0,0 +1,9 @@
+import { useContext } from "react"
+import { RsiContext } from "../components/Providers"
+import type { RsiProps } from "../types"
+import type { MarkRequired } from "ts-essentials"
+import type { defaultRSIProps } from "../ReactSpreadsheetImport"
+import type { Translations } from "../translationsRSIProps"
+
+export const useRsi = <T extends string>() =>
+  useContext<MarkRequired<RsiProps<T>, keyof typeof defaultRSIProps> & { translations: Translations }>(RsiContext)

inventory/src/lib/react-spreadsheet-import/src/index.ts

@@ -0,0 +1,2 @@
+export { StepType } from "./steps/UploadFlow"
+export { ReactSpreadsheetImport } from "./ReactSpreadsheetImport"

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/MatchColumnsStep.tsx

@@ -0,0 +1,232 @@
+import { useCallback, useEffect, useMemo, useState } from "react"
+import { UserTableColumn } from "./components/UserTableColumn"
+import { useRsi } from "../../hooks/useRsi"
+import { TemplateColumn } from "./components/TemplateColumn"
+import { ColumnGrid } from "./components/ColumnGrid"
+import { setColumn } from "./utils/setColumn"
+import { setIgnoreColumn } from "./utils/setIgnoreColumn"
+import { setSubColumn } from "./utils/setSubColumn"
+import { normalizeTableData } from "./utils/normalizeTableData"
+import type { Field, RawData } from "../../types"
+import { getMatchedColumns } from "./utils/getMatchedColumns"
+import { findUnmatchedRequiredFields } from "./utils/findUnmatchedRequiredFields"
+import { toast } from "sonner"
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+  AlertDialogPortal,
+  AlertDialogOverlay,
+} from "@/components/ui/alert-dialog"
+
+export type MatchColumnsProps<T extends string> = {
+  data: RawData[]
+  headerValues: RawData
+  onContinue: (data: any[], rawData: RawData[], columns: Columns<T>) => void
+  onBack?: () => void
+}
+
+export enum ColumnType {
+  empty,
+  ignored,
+  matched,
+  matchedCheckbox,
+  matchedSelect,
+  matchedSelectOptions,
+}
+
+export type MatchedOptions<T> = {
+  entry: string
+  value: T
+}
+
+type EmptyColumn = { type: ColumnType.empty; index: number; header: string }
+type IgnoredColumn = { type: ColumnType.ignored; index: number; header: string }
+type MatchedColumn<T> = { type: ColumnType.matched; index: number; header: string; value: T }
+type MatchedSwitchColumn<T> = { type: ColumnType.matchedCheckbox; index: number; header: string; value: T }
+export type MatchedSelectColumn<T> = {
+  type: ColumnType.matchedSelect
+  index: number
+  header: string
+  value: T
+  matchedOptions: Partial<MatchedOptions<T>>[]
+}
+export type MatchedSelectOptionsColumn<T> = {
+  type: ColumnType.matchedSelectOptions
+  index: number
+  header: string
+  value: T
+  matchedOptions: MatchedOptions<T>[]
+}
+
+export type Column<T extends string> =
+  | EmptyColumn
+  | IgnoredColumn
+  | MatchedColumn<T>
+  | MatchedSwitchColumn<T>
+  | MatchedSelectColumn<T>
+  | MatchedSelectOptionsColumn<T>
+
+export type Columns<T extends string> = Column<T>[]
+
+export const MatchColumnsStep = <T extends string>({
+  data,
+  headerValues,
+  onContinue,
+  onBack,
+}: MatchColumnsProps<T>) => {
+  const dataExample = data.slice(0, 2)
+  const { fields, autoMapHeaders, autoMapSelectValues, autoMapDistance, translations, allowInvalidSubmit } = useRsi<T>()
+  const [isLoading, setIsLoading] = useState(false)
+  const [columns, setColumns] = useState<Columns<T>>(
+    // Do not remove spread, it indexes empty array elements, otherwise map() skips over them
+    ([...headerValues] as string[]).map((value, index) => ({ type: ColumnType.empty, index, header: value ?? "" })),
+  )
+  const [showUnmatchedFieldsAlert, setShowUnmatchedFieldsAlert] = useState(false)
+
+  const onChange = useCallback(
+    (value: T, columnIndex: number) => {
+      const field = fields.find((field: Field<T>) => field.key === value)
+      if (!field) return
+
+      const existingFieldIndex = columns.findIndex((column) => "value" in column && column.value === field.key)
+      
+      setColumns(
+        columns.map<Column<T>>((column, index) => {
+          if (columnIndex === index) {
+            // Set the new column value
+            return setColumn(column, field, data, autoMapSelectValues)
+          } else if (index === existingFieldIndex) {
+            // Clear the old column that had this field
+            toast.warning(translations.matchColumnsStep.duplicateColumnWarningTitle, {
+              description: translations.matchColumnsStep.duplicateColumnWarningDescription,
+            })
+            return setColumn(column)
+          } else {
+            // Leave other columns unchanged
+            return column
+          }
+        }),
+      )
+    },
+    [
+      autoMapSelectValues,
+      columns,
+      data,
+      fields,
+      translations.matchColumnsStep.duplicateColumnWarningDescription,
+      translations.matchColumnsStep.duplicateColumnWarningTitle,
+    ],
+  )
+
+  const onIgnore = useCallback(
+    (columnIndex: number) => {
+      setColumns(columns.map((column, index) => (columnIndex === index ? setIgnoreColumn<T>(column) : column)))
+    },
+    [columns, setColumns],
+  )
+
+  const onRevertIgnore = useCallback(
+    (columnIndex: number) => {
+      setColumns(columns.map((column, index) => (columnIndex === index ? setColumn(column) : column)))
+    },
+    [columns, setColumns],
+  )
+
+  const onSubChange = useCallback(
+    (value: string, columnIndex: number, entry: string) => {
+      setColumns(
+        columns.map((column, index) =>
+          columnIndex === index && "matchedOptions" in column ? setSubColumn(column, entry, value) : column,
+        ),
+      )
+    },
+    [columns, setColumns],
+  )
+  const unmatchedRequiredFields = useMemo(() => findUnmatchedRequiredFields(fields, columns), [fields, columns])
+
+  const handleOnContinue = useCallback(async () => {
+    if (unmatchedRequiredFields.length > 0) {
+      setShowUnmatchedFieldsAlert(true)
+    } else {
+      setIsLoading(true)
+      await onContinue(normalizeTableData(columns, data, fields), data, columns)
+      setIsLoading(false)
+    }
+  }, [unmatchedRequiredFields.length, onContinue, columns, data, fields])
+
+  const handleAlertOnContinue = useCallback(async () => {
+    setShowUnmatchedFieldsAlert(false)
+    setIsLoading(true)
+    await onContinue(normalizeTableData(columns, data, fields), data, columns)
+    setIsLoading(false)
+  }, [onContinue, columns, data, fields])
+
+  useEffect(
+    () => {
+      if (autoMapHeaders) {
+        setColumns(getMatchedColumns(columns, fields, data, autoMapDistance, autoMapSelectValues))
+      }
+    },
+    // eslint-disable-next-line react-hooks/exhaustive-deps
+    [],
+  )
+
+  return (
+    <>
+      <AlertDialog open={showUnmatchedFieldsAlert} onOpenChange={setShowUnmatchedFieldsAlert}>
+        <AlertDialogPortal>
+          <AlertDialogOverlay className="z-[1400]" />
+          <AlertDialogContent className="z-[1500]">
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {translations.alerts.unmatchedRequiredFields.headerTitle}
+              </AlertDialogTitle>
+              <div className="space-y-3">
+                <AlertDialogDescription>
+                  {translations.alerts.unmatchedRequiredFields.bodyText}
+                </AlertDialogDescription>
+                <p className="text-sm text-muted-foreground">
+                  {translations.alerts.unmatchedRequiredFields.listTitle}{" "}
+                  <span className="font-bold">
+                    {unmatchedRequiredFields.join(", ")}
+                  </span>
+                </p>
+              </div>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel>
+                {translations.alerts.unmatchedRequiredFields.cancelButtonTitle}
+              </AlertDialogCancel>
+              {allowInvalidSubmit && (
+                <AlertDialogAction onClick={handleAlertOnContinue}>
+                  {translations.alerts.unmatchedRequiredFields.continueButtonTitle}
+                </AlertDialogAction>
+              )}
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialogPortal>
+      </AlertDialog>
+      <ColumnGrid
+        columns={columns}
+        onContinue={handleOnContinue}
+        onBack={onBack}
+        isLoading={isLoading}
+        userColumn={(column) => (
+          <UserTableColumn
+            column={column}
+            onIgnore={onIgnore}
+            onRevertIgnore={onRevertIgnore}
+            entries={dataExample.map((row) => row[column.index])}
+          />
+        )}
+        templateColumn={(column) => <TemplateColumn column={column} onChange={onChange} onSubChange={onSubChange} />}
+      />
+    </>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/ColumnGrid.tsx

@@ -0,0 +1,112 @@
+import type React from "react"
+import type { Column, Columns } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import { useRsi } from "../../../hooks/useRsi"
+import { Button } from "@/components/ui/button"
+import { ScrollArea, ScrollBar } from "@/components/ui/scroll-area"
+
+type ColumnGridProps<T extends string> = {
+  columns: Columns<T>
+  userColumn: (column: Column<T>) => React.ReactNode
+  templateColumn: (column: Column<T>) => React.ReactNode
+  onContinue: (val: Record<string, string>[]) => void
+  onBack?: () => void
+  isLoading: boolean
+}
+
+export const ColumnGrid = <T extends string>({
+  columns,
+  userColumn,
+  templateColumn,
+  onContinue,
+  onBack,
+  isLoading,
+}: ColumnGridProps<T>) => {
+  const { translations } = useRsi()
+  const normalColumnWidth = 250
+  const ignoredColumnWidth = 48 // 12 units = 3rem = 48px
+  const gap = 16
+  const totalWidth = columns.reduce((acc, col) => 
+    acc + (col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth) + gap, 
+    -gap // Subtract one gap since we need gaps between columns only
+  )
+
+  return (
+    <div className="flex h-[calc(100vh-10rem)] flex-col">
+      <div className="flex-1 overflow-hidden">
+        <div className="px-8 py-6">
+          <div className="mb-8">
+            <h2 className="text-3xl font-semibold text-foreground">
+              {translations.matchColumnsStep.title}
+            </h2>
+          </div>
+          <ScrollArea className="relative">
+            <div className="space-y-8" style={{ width: totalWidth }}>
+              {/* Your table section */}
+              <div>
+                <h3 className="mb-4 text-lg font-medium text-foreground">
+                  {translations.matchColumnsStep.userTableTitle}
+                </h3>
+                <div className="relative">
+                  <div 
+                    className="grid auto-cols-fr gap-4"
+                    style={{ 
+                      gridTemplateColumns: columns.map(col => 
+                        `${col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth}px`
+                      ).join(" "),
+                    }}
+                  >
+                    {columns.map((column, index) => (
+                      <div key={column.header + index}>
+                        {userColumn(column)}
+                      </div>
+                    ))}
+                  </div>
+                  <div className="pointer-events-none absolute bottom-0 left-0 right-0 h-16 bg-gradient-to-b from-transparent via-background/50 to-background" />
+                </div>
+              </div>
+
+              {/* Will become section */}
+              <div>
+                <h3 className="mb-4 text-lg font-medium text-foreground">
+                  {translations.matchColumnsStep.templateTitle}
+                </h3>
+                <div 
+                  className="grid auto-cols-fr gap-4"
+                  style={{ 
+                    gridTemplateColumns: columns.map(col => 
+                      `${col.type === ColumnType.ignored ? ignoredColumnWidth : normalColumnWidth}px`
+                    ).join(" "),
+                  }}
+                >
+                  {columns.map((column, index) => (
+                    <div key={column.header + index}>
+                      {templateColumn(column)}
+                    </div>
+                  ))}
+                </div>
+              </div>
+            </div>
+            <ScrollBar orientation="horizontal" />
+          </ScrollArea>
+        </div>
+      </div>
+      <div className="border-t bg-muted px-8 py-4 -mb-1">
+        <div className="flex items-center justify-between">
+          {onBack && (
+            <Button variant="outline" onClick={onBack}>
+              {translations.matchColumnsStep.backButtonTitle}
+            </Button>
+          )}
+          <Button 
+            className="ml-auto"
+            disabled={isLoading}
+            onClick={() => onContinue([])}
+          >
+            {translations.matchColumnsStep.nextButtonTitle}
+          </Button>
+        </div>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/MatchIcon.tsx

@@ -0,0 +1,42 @@
+import { chakra, useStyleConfig, Flex } from "@chakra-ui/react"
+import { dataAttr } from "@chakra-ui/utils"
+import { motion } from "framer-motion"
+import { CgCheck } from "react-icons/cg"
+
+const MotionFlex = motion(Flex)
+
+const animationConfig = {
+  transition: {
+    duration: 0.1,
+  },
+  exit: { scale: 0.5, opacity: 0 },
+  initial: { scale: 0.5, opacity: 0 },
+  animate: { scale: 1, opacity: 1 },
+}
+type MatchIconProps = {
+  isChecked: boolean
+}
+
+export const MatchIcon = (props: MatchIconProps) => {
+  const style = useStyleConfig("MatchIcon", props)
+
+  return (
+    <chakra.div
+      __css={style}
+      minW={6}
+      minH={6}
+      w={6}
+      h={6}
+      ml="0.875rem"
+      mr={3}
+      data-highlighted={dataAttr(props.isChecked)}
+      data-testid="column-checkmark"
+    >
+      {props.isChecked && (
+        <MotionFlex {...animationConfig}>
+          <CgCheck size="24px" />
+        </MotionFlex>
+      )}
+    </chakra.div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/TemplateColumn.tsx

@@ -0,0 +1,129 @@
+import { useRsi } from "../../../hooks/useRsi"
+import type { Column } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { Fields, Field } from "../../../types"
+import {
+  Card,
+  CardContent,
+  CardHeader,
+} from "@/components/ui/card"
+import {
+  Select,
+  SelectContent,
+  SelectItem,
+  SelectTrigger,
+  SelectValue,
+} from "@/components/ui/select"
+import {
+  Accordion,
+  AccordionContent,
+  AccordionItem,
+  AccordionTrigger,
+} from "@/components/ui/accordion"
+import { Check } from "lucide-react"
+
+type TemplateColumnProps<T extends string> = {
+  column: Column<T>
+  onChange: (value: T, columnIndex: number) => void
+  onSubChange: (value: string, columnIndex: number, entry: string) => void
+}
+const getAccordionTitle = <T extends string>(fields: Fields<T>, column: Column<T>, translations: any) => {
+  const fieldLabel = fields.find((field: Field<T>) => "value" in column && field.key === column.value)!.label
+  return `${translations.matchColumnsStep.matchDropdownTitle} ${fieldLabel} (${
+    "matchedOptions" in column && column.matchedOptions.filter((option) => !option.value).length
+  } ${translations.matchColumnsStep.unmatched})`
+}
+
+export const TemplateColumn = <T extends string>({ column, onChange, onSubChange }: TemplateColumnProps<T>) => {
+  const { translations, fields } = useRsi<T>()
+  const isIgnored = column.type === ColumnType.ignored
+  const isChecked =
+    column.type === ColumnType.matched ||
+    column.type === ColumnType.matchedCheckbox ||
+    column.type === ColumnType.matchedSelectOptions
+  const isSelect = "matchedOptions" in column
+  const selectOptions = fields.map(({ label, key }: { label: string; key: string }) => ({ value: key, label }))
+  const selectValue = column.type === ColumnType.empty ? undefined : 
+    selectOptions.find(({ value }: { value: string }) => "value" in column && column.value === value)?.value
+
+  if (isIgnored) {
+    return null
+  }
+
+  return (
+    <Card className="h-full">
+      <CardHeader className="flex flex-row items-center justify-between space-x-2 p-4">
+        <div className="flex-1">
+          <Select
+            key={`select-${column.index}-${("value" in column ? column.value : "empty")}`}
+            value={selectValue}
+            onValueChange={(value) => onChange(value as T, column.index)}
+          >
+            <SelectTrigger className="w-full">
+              <SelectValue placeholder={translations.matchColumnsStep.selectPlaceholder} />
+            </SelectTrigger>
+            <SelectContent 
+              side="bottom"
+              align="start"
+              className="z-[1500]"
+            >
+              {selectOptions.map((option: { value: string; label: string }) => (
+                <SelectItem key={option.value} value={option.value}>
+                  {option.label}
+                </SelectItem>
+              ))}
+            </SelectContent>
+          </Select>
+        </div>
+        {isChecked && (
+          <div className="flex h-8 w-8 items-center justify-center rounded-full border border-green-700 bg-green-300 dark:bg-green-900/20">
+            <Check className="h-4 w-4 text-green-700 dark:text-green-500" />
+          </div>
+        )}
+      </CardHeader>
+      {isSelect && (
+        <CardContent className="p-4">
+          <Accordion type="multiple" className="w-full">
+            <AccordionItem value="options" className="border-none">
+              <AccordionTrigger className="py-2 text-sm hover:no-underline">
+                {getAccordionTitle<T>(fields, column, translations)}
+              </AccordionTrigger>
+              <AccordionContent>
+                <div className="space-y-2">
+                  {column.matchedOptions.map((option) => (
+                    <div key={option.entry} className="space-y-1">
+                      <p className="text-sm text-muted-foreground">
+                        {option.entry}
+                      </p>
+                      <Select
+                        value={option.value}
+                        onValueChange={(value) => onSubChange(value, column.index, option.entry!)}
+                      >
+                        <SelectTrigger className="w-full">
+                          <SelectValue placeholder={translations.matchColumnsStep.subSelectPlaceholder} />
+                        </SelectTrigger>
+                        <SelectContent 
+                          side="bottom"
+                          align="start"
+                          className="z-[1000]"
+                        >
+                          {fields
+                            .find((field: Field<T>) => "value" in column && field.key === column.value)
+                            ?.fieldType.options.map((option: { value: string; label: string }) => (
+                              <SelectItem key={option.value} value={option.value}>
+                                {option.label}
+                              </SelectItem>
+                            ))}
+                        </SelectContent>
+                      </Select>
+                    </div>
+                  ))}
+                </div>
+              </AccordionContent>
+            </AccordionItem>
+          </Accordion>
+        </CardContent>
+      )}
+    </Card>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/components/UserTableColumn.tsx

@@ -0,0 +1,74 @@
+import { Button } from "@/components/ui/button"
+import { Card, CardContent, CardHeader } from "@/components/ui/card"
+import { X, RotateCcw } from "lucide-react"
+import type { Column } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { RawData } from "../../../types"
+
+type UserTableColumnProps<T extends string> = {
+  column: Column<T>
+  entries: RawData
+  onIgnore: (index: number) => void
+  onRevertIgnore: (index: number) => void
+}
+
+export const UserTableColumn = <T extends string>(props: UserTableColumnProps<T>) => {
+  const {
+    column: { header, index, type },
+    entries,
+    onIgnore,
+    onRevertIgnore,
+  } = props
+  const isIgnored = type === ColumnType.ignored
+
+  if (isIgnored) {
+    return (
+      <Card className="h-full w-12 bg-muted/50">
+        <CardHeader className="flex flex-col items-center space-y-4 p-2">
+          <Button
+            variant="ghost"
+            size="icon"
+            onClick={() => onRevertIgnore(index)}
+            className="h-8 w-8"
+          >
+            <RotateCcw className="h-4 w-4" />
+          </Button>
+          <div 
+            className="vertical-text font-medium text-muted-foreground"
+            style={{ writingMode: 'vertical-rl', textOrientation: 'mixed', transform: 'rotate(180deg)' }}
+          >
+            {header}
+          </div>
+        </CardHeader>
+      </Card>
+    )
+  }
+
+  return (
+    <Card className="h-full">
+      <CardHeader className="flex flex-row items-center justify-between space-x-2 p-4">
+        <p className="font-medium">
+          {header}
+        </p>
+        <Button
+          variant="ghost"
+          size="icon"
+          onClick={() => onIgnore(index)}
+          className="h-8 w-8"
+        >
+          <X className="h-4 w-4" />
+        </Button>
+      </CardHeader>
+      <CardContent className="space-y-2 p-4">
+        {entries.map((entry, i) => (
+          <p 
+            key={`${entry || ""}-${i}`}
+            className="truncate text-sm text-muted-foreground"
+          >
+            {entry}
+          </p>
+        ))}
+      </CardContent>
+    </Card>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/findMatch.ts

@@ -0,0 +1,26 @@
+import lavenstein from "js-levenshtein"
+import type { Fields } from "../../../types"
+
+type AutoMatchAccumulator<T> = {
+  distance: number
+  value: T
+}
+
+export const findMatch = <T extends string>(
+  header: string,
+  fields: Fields<T>,
+  autoMapDistance: number,
+): T | undefined => {
+  const smallestValue = fields.reduce<AutoMatchAccumulator<T>>((acc, field) => {
+    const distance = Math.min(
+      ...[
+        lavenstein(field.key, header),
+        ...(field.alternateMatches?.map((alternate) => lavenstein(alternate, header)) || []),
+      ],
+    )
+    return distance < acc.distance || acc.distance === undefined
+      ? ({ value: field.key, distance } as AutoMatchAccumulator<T>)
+      : acc
+  }, {} as AutoMatchAccumulator<T>)
+  return smallestValue.distance <= autoMapDistance ? smallestValue.value : undefined
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/findUnmatchedRequiredFields.ts

@@ -0,0 +1,8 @@
+import type { Fields } from "../../../types"
+import type { Columns } from "../MatchColumnsStep"
+
+export const findUnmatchedRequiredFields = <T extends string>(fields: Fields<T>, columns: Columns<T>) =>
+  fields
+    .filter((field) => field.validations?.some((validation) => validation.rule === "required"))
+    .filter((field) => columns.findIndex((column) => "value" in column && column.value === field.key) === -1)
+    .map((field) => field.label) || []

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/getFieldOptions.ts

@@ -0,0 +1,6 @@
+import type { Fields } from "../../../types"
+
+export const getFieldOptions = <T extends string>(fields: Fields<T>, fieldKey: string) => {
+  const field = fields.find(({ key }) => fieldKey === key)!
+  return field.fieldType.type === "select" ? field.fieldType.options : []
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/getMatchedColumns.ts

@@ -0,0 +1,41 @@
+import lavenstein from "js-levenshtein"
+import { findMatch } from "./findMatch"
+import type { Field, Fields } from "../../../types"
+import { setColumn } from "./setColumn"
+import type { Column, Columns } from "../MatchColumnsStep"
+import type { MatchColumnsProps } from "../MatchColumnsStep"
+
+export const getMatchedColumns = <T extends string>(
+  columns: Columns<T>,
+  fields: Fields<T>,
+  data: MatchColumnsProps<T>["data"],
+  autoMapDistance: number,
+  autoMapSelectValues?: boolean,
+) =>
+  columns.reduce<Column<T>[]>((arr, column) => {
+    const autoMatch = findMatch(column.header, fields, autoMapDistance)
+    if (autoMatch) {
+      const field = fields.find((field) => field.key === autoMatch) as Field<T>
+      const duplicateIndex = arr.findIndex((column) => "value" in column && column.value === field.key)
+      const duplicate = arr[duplicateIndex]
+      if (duplicate && "value" in duplicate) {
+        return lavenstein(duplicate.value, duplicate.header) < lavenstein(autoMatch, column.header)
+          ? [
+              ...arr.slice(0, duplicateIndex),
+              setColumn(arr[duplicateIndex], field, data, autoMapSelectValues),
+              ...arr.slice(duplicateIndex + 1),
+              setColumn(column),
+            ]
+          : [
+              ...arr.slice(0, duplicateIndex),
+              setColumn(arr[duplicateIndex]),
+              ...arr.slice(duplicateIndex + 1),
+              setColumn(column, field, data, autoMapSelectValues),
+            ]
+      } else {
+        return [...arr, setColumn(column, field, data, autoMapSelectValues)]
+      }
+    } else {
+      return [...arr, column]
+    }
+  }, [])

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/normalizeCheckboxValue.ts

@@ -0,0 +1,13 @@
+const booleanWhitelist: Record<string, boolean> = {
+  yes: true,
+  no: false,
+  true: true,
+  false: false,
+}
+
+export const normalizeCheckboxValue = (value: string | undefined): boolean => {
+  if (value && value.toLowerCase() in booleanWhitelist) {
+    return booleanWhitelist[value.toLowerCase()]
+  }
+  return false
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/normalizeTableData.ts

@@ -0,0 +1,42 @@
+import type { Columns } from "../MatchColumnsStep"
+import { ColumnType } from "../MatchColumnsStep"
+import type { Data, Fields, RawData } from "../../../types"
+import { normalizeCheckboxValue } from "./normalizeCheckboxValue"
+
+export const normalizeTableData = <T extends string>(columns: Columns<T>, data: RawData[], fields: Fields<T>) =>
+  data.map((row) =>
+    columns.reduce((acc, column, index) => {
+      const curr = row[index]
+      switch (column.type) {
+        case ColumnType.matchedCheckbox: {
+          const field = fields.find((field) => field.key === column.value)!
+          if ("booleanMatches" in field.fieldType && Object.keys(field.fieldType).length) {
+            const booleanMatchKey = Object.keys(field.fieldType.booleanMatches || []).find(
+              (key) => key.toLowerCase() === curr?.toLowerCase(),
+            )!
+            const booleanMatch = field.fieldType.booleanMatches?.[booleanMatchKey]
+            acc[column.value] = booleanMatchKey ? booleanMatch : normalizeCheckboxValue(curr)
+          } else {
+            acc[column.value] = normalizeCheckboxValue(curr)
+          }
+          return acc
+        }
+        case ColumnType.matched: {
+          acc[column.value] = curr === "" ? undefined : curr
+          return acc
+        }
+        case ColumnType.matchedSelect:
+        case ColumnType.matchedSelectOptions: {
+          const matchedOption = column.matchedOptions.find(({ entry, value }) => entry === curr)
+          acc[column.value] = matchedOption?.value || undefined
+          return acc
+        }
+        case ColumnType.empty:
+        case ColumnType.ignored: {
+          return acc
+        }
+        default:
+          return acc
+      }
+    }, {} as Data<T>),
+  )

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setColumn.ts

@@ -0,0 +1,38 @@
+import type { Field } from "../../../types"
+import { Column, ColumnType, MatchColumnsProps, MatchedOptions } from "../MatchColumnsStep"
+import { uniqueEntries } from "./uniqueEntries"
+
+export const setColumn = <T extends string>(
+  oldColumn: Column<T>,
+  field?: Field<T>,
+  data?: MatchColumnsProps<T>["data"],
+  autoMapSelectValues?: boolean,
+): Column<T> => {
+  switch (field?.fieldType.type) {
+    case "select":
+      const fieldOptions = field.fieldType.options
+      const uniqueData = uniqueEntries(data || [], oldColumn.index) as MatchedOptions<T>[]
+      const matchedOptions = autoMapSelectValues
+        ? uniqueData.map((record) => {
+            const value = fieldOptions.find(
+              (fieldOption) => fieldOption.value === record.entry || fieldOption.label === record.entry,
+            )?.value
+            return value ? ({ ...record, value } as MatchedOptions<T>) : (record as MatchedOptions<T>)
+          })
+        : uniqueData
+      const allMatched = matchedOptions.filter((o) => o.value).length == uniqueData?.length
+
+      return {
+        ...oldColumn,
+        type: allMatched ? ColumnType.matchedSelectOptions : ColumnType.matchedSelect,
+        value: field.key,
+        matchedOptions,
+      }
+    case "checkbox":
+      return { index: oldColumn.index, type: ColumnType.matchedCheckbox, value: field.key, header: oldColumn.header }
+    case "input":
+      return { index: oldColumn.index, type: ColumnType.matched, value: field.key, header: oldColumn.header }
+    default:
+      return { index: oldColumn.index, header: oldColumn.header, type: ColumnType.empty }
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setIgnoreColumn.ts

@@ -0,0 +1,7 @@
+import { Column, ColumnType } from "../MatchColumnsStep"
+
+export const setIgnoreColumn = <T extends string>({ header, index }: Column<T>): Column<T> => ({
+  header,
+  index,
+  type: ColumnType.ignored,
+})

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/setSubColumn.ts

@@ -0,0 +1,14 @@
+import { ColumnType, MatchedOptions, MatchedSelectColumn, MatchedSelectOptionsColumn } from "../MatchColumnsStep"
+export const setSubColumn = <T>(
+  oldColumn: MatchedSelectColumn<T> | MatchedSelectOptionsColumn<T>,
+  entry: string,
+  value: string,
+): MatchedSelectColumn<T> | MatchedSelectOptionsColumn<T> => {
+  const options = oldColumn.matchedOptions.map((option) => (option.entry === entry ? { ...option, value } : option))
+  const allMathced = options.every(({ value }) => !!value)
+  if (allMathced) {
+    return { ...oldColumn, matchedOptions: options as MatchedOptions<T>[], type: ColumnType.matchedSelectOptions }
+  } else {
+    return { ...oldColumn, matchedOptions: options as MatchedOptions<T>[], type: ColumnType.matchedSelect }
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/MatchColumnsStep/utils/uniqueEntries.ts

@@ -0,0 +1,11 @@
+import uniqBy from "lodash/uniqBy"
+import type { MatchColumnsProps, MatchedOptions } from "../MatchColumnsStep"
+
+export const uniqueEntries = <T extends string>(
+  data: MatchColumnsProps<T>["data"],
+  index: number,
+): Partial<MatchedOptions<T>>[] =>
+  uniqBy(
+    data.map((row) => ({ entry: row[index] })),
+    "entry",
+  ).filter(({ entry }) => !!entry)

inventory/src/lib/react-spreadsheet-import/src/steps/SelectHeaderStep/SelectHeaderStep.tsx

@@ -0,0 +1,57 @@
+import { useCallback, useState } from "react"
+import { SelectHeaderTable } from "./components/SelectHeaderTable"
+import { useRsi } from "../../hooks/useRsi"
+import type { RawData } from "../../types"
+import { Button } from "@/components/ui/button"
+
+type SelectHeaderProps = {
+  data: RawData[]
+  onContinue: (headerValues: RawData, data: RawData[]) => Promise<void>
+  onBack?: () => void
+}
+
+export const SelectHeaderStep = ({ data, onContinue, onBack }: SelectHeaderProps) => {
+  const { translations } = useRsi()
+  const [selectedRows, setSelectedRows] = useState<ReadonlySet<number>>(new Set([0]))
+  const [isLoading, setIsLoading] = useState(false)
+
+  const handleContinue = useCallback(async () => {
+    const [selectedRowIndex] = selectedRows
+    // We consider data above header to be redundant
+    const trimmedData = data.slice(selectedRowIndex + 1)
+    setIsLoading(true)
+    await onContinue(data[selectedRowIndex], trimmedData)
+    setIsLoading(false)
+  }, [onContinue, data, selectedRows])
+
+  return (
+    <div className="flex flex-col">
+      <div className="px-8 py-6">
+        <h2 className="text-2xl font-semibold text-foreground">
+          {translations.selectHeaderStep.title}
+        </h2>
+      </div>
+      <div className="flex-1 px-8 mb-12 overflow-auto">
+        <SelectHeaderTable 
+          data={data} 
+          selectedRows={selectedRows} 
+          setSelectedRows={setSelectedRows} 
+        />
+      </div>
+      <div className="flex items-center justify-between border-t bg-muted px-8 py-4 mt-2">
+        {onBack && (
+          <Button variant="outline" onClick={onBack}>
+            {translations.selectHeaderStep.backButtonTitle}
+          </Button>
+        )}
+        <Button 
+          className="ml-auto"
+          disabled={isLoading}
+          onClick={handleContinue}
+        >
+          {translations.selectHeaderStep.nextButtonTitle}
+        </Button>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/SelectHeaderStep/components/SelectHeaderTable.tsx

@@ -0,0 +1,106 @@
+import { useMemo } from "react"
+import type { RawData } from "../../../types"
+import {
+  Table,
+  TableBody,
+  TableCell,
+  TableHead,
+  TableHeader,
+  TableRow,
+} from "@/components/ui/table"
+import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
+import { Label } from "@/components/ui/label"
+import { cn } from "@/lib/utils"
+
+interface Props {
+  data: RawData[]
+  selectedRows: ReadonlySet<number>
+  setSelectedRows: (rows: ReadonlySet<number>) => void
+}
+
+export const SelectHeaderTable = ({ data, selectedRows, setSelectedRows }: Props) => {
+  const columns = useMemo(() => {
+    const longestRowLength = data.reduce((acc, curr) => (acc > curr.length ? acc : curr.length), 0)
+    return Array.from(Array(longestRowLength), (_, index) => ({
+      key: index.toString(),
+      name: `Column ${index + 1}`,
+    }))
+  }, [data])
+
+  if (!data || data.length === 0) {
+    return (
+      <div className="p-4">
+        <p className="text-sm text-muted-foreground">No data available to select headers from.</p>
+      </div>
+    )
+  }
+
+  const selectedRowIndex = Array.from(selectedRows)[0]
+  const gridTemplateColumns = `60px repeat(${columns.length}, minmax(150px, 300px))`
+
+  return (
+    <div className="rounded-md border p-3">
+      <p className="mb-2 p-2 text-sm text-muted-foreground">
+        Select the row that contains your column headers
+      </p>
+      <div className="h-[calc(100vh-27rem)] overflow-auto">
+        <Table className="relative w-full" style={{ tableLayout: 'fixed' }}>
+          <TableHeader>
+            <TableRow className="grid" style={{ gridTemplateColumns }}>
+              <TableHead className="sticky top-0 z-20 bg-background overflow-hidden">
+                <div className="truncate">Select</div>
+              </TableHead>
+              {columns.map((column) => (
+                <TableHead 
+                  key={column.key}
+                  className="sticky top-0 z-20 bg-background overflow-hidden"
+                >
+                  <div className="truncate">
+                    {column.name}
+                  </div>
+                </TableHead>
+              ))}
+            </TableRow>
+          </TableHeader>
+          <TableBody>
+            <RadioGroup
+              value={selectedRowIndex?.toString()}
+              onValueChange={(value) => setSelectedRows(new Set([parseInt(value)]))}
+            >
+              {data.map((row, rowIndex) => (
+                <TableRow 
+                  key={rowIndex}
+                  className={cn(
+                    "grid",
+                    selectedRowIndex === rowIndex && "bg-muted",
+                    "group hover:bg-muted/50"
+                  )}
+                  style={{ gridTemplateColumns }}
+                >
+                  <TableCell className="overflow-hidden">
+                    <div className="flex items-center">
+                      <RadioGroupItem value={rowIndex.toString()} id={`row-${rowIndex}`} />
+                      <Label htmlFor={`row-${rowIndex}`} className="sr-only">
+                        Select as header row
+                      </Label>
+                    </div>
+                  </TableCell>
+                  {columns.map((column, colIndex) => (
+                    <TableCell 
+                      key={`${rowIndex}-${column.key}`}
+                      className="overflow-hidden"
+                    >
+                      <div className="truncate">
+                        {row[colIndex] || ""}
+                      </div>
+                    </TableCell>
+                  ))}
+                </TableRow>
+              ))}
+            </RadioGroup>
+          </TableBody>
+        </Table>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/SelectHeaderStep/components/columns.tsx

@@ -0,0 +1,68 @@
+import DataGrid, { Column, FormatterProps, useRowSelection } from "react-data-grid"
+import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
+import { Label } from "@/components/ui/label"
+import type { RawData } from "../../../types"
+import { cn } from "@/lib/utils"
+
+const SELECT_COLUMN_KEY = "select-row"
+
+function SelectFormatter(props: FormatterProps<unknown>) {
+  const [isRowSelected, onRowSelectionChange] = useRowSelection()
+
+  return (
+    <div className="flex h-full items-center pl-2">
+      <RadioGroup defaultValue={isRowSelected ? "selected" : undefined}>
+        <div className="flex items-center space-x-2">
+          <RadioGroupItem
+            value="selected"
+            id={`row-${props.rowIdx}`}
+            checked={isRowSelected}
+            onClick={(event) => {
+              onRowSelectionChange({
+                row: props.row,
+                checked: !isRowSelected,
+                isShiftClick: (event.nativeEvent as MouseEvent).shiftKey,
+              })
+            }}
+          />
+          <Label
+            htmlFor={`row-${props.rowIdx}`}
+            className="sr-only"
+          >
+            Select as header row
+          </Label>
+        </div>
+      </RadioGroup>
+    </div>
+  )
+}
+
+export const SelectColumn: Column<any, any> = {
+  key: SELECT_COLUMN_KEY,
+  name: "Select Header",
+  width: 100,
+  minWidth: 100,
+  maxWidth: 100,
+  resizable: false,
+  sortable: false,
+  frozen: true,
+  cellClass: "rdg-radio",
+  formatter: SelectFormatter,
+}
+
+export const generateSelectionColumns = (data: RawData[]) => {
+  const longestRowLength = data.reduce((acc, curr) => (acc > curr.length ? acc : curr.length), 0)
+  return [
+    SelectColumn,
+    ...Array.from(Array(longestRowLength), (_, index) => ({
+      key: index.toString(),
+      name: `Column ${index + 1}`,
+      width: 150,
+      formatter: ({ row }) => (
+        <div className="p-2 overflow-hidden text-ellipsis whitespace-nowrap">
+          {row[index]}
+        </div>
+      ),
+    })),
+  ]
+}

inventory/src/lib/react-spreadsheet-import/src/steps/SelectSheetStep/SelectSheetStep.tsx

@@ -0,0 +1,77 @@
+import { useCallback, useState } from "react"
+import { useRsi } from "../../hooks/useRsi"
+import { RadioGroup, RadioGroupItem } from "@/components/ui/radio-group"
+import { Label } from "@/components/ui/label"
+import { Button } from "@/components/ui/button"
+import { ChevronLeft } from "lucide-react"
+
+type SelectSheetProps = {
+  sheetNames: string[]
+  onContinue: (sheetName: string) => Promise<void>
+  onBack?: () => void
+}
+
+export const SelectSheetStep = ({ sheetNames, onContinue, onBack }: SelectSheetProps) => {
+  const [isLoading, setIsLoading] = useState(false)
+  const { translations } = useRsi()
+  const [value, setValue] = useState(sheetNames[0])
+
+  const handleOnContinue = useCallback(
+    async (data: typeof value) => {
+      setIsLoading(true)
+      await onContinue(data)
+      setIsLoading(false)
+    },
+    [onContinue],
+  )
+
+  return (
+    <div className="flex h-[calc(100vh-10rem)] flex-col">
+      <div className="flex-1 overflow-hidden">
+        <div className="px-8 py-6">
+          <div className="mb-8">
+            <h2 className="text-3xl font-semibold text-foreground">
+              {translations.uploadStep.selectSheet.title}
+            </h2>
+          </div>
+          <RadioGroup
+            value={value}
+            onValueChange={setValue}
+            className="space-y-4"
+          >
+            {sheetNames.map((sheetName) => (
+              <div key={sheetName} className="flex items-center space-x-2">
+                <RadioGroupItem value={sheetName} id={sheetName} />
+                <Label
+                  htmlFor={sheetName}
+                  className="text-base"
+                >
+                  {sheetName}
+                </Label>
+              </div>
+            ))}
+          </RadioGroup>
+        </div>
+      </div>
+      <div className="flex items-center justify-between border-t px-8 py-4 bg-muted -mb-1">
+        {onBack && (
+          <Button
+            variant="ghost"
+            onClick={onBack}
+            className="gap-2"
+          >
+            <ChevronLeft className="h-4 w-4" />
+            {translations.uploadStep.selectSheet.backButtonTitle}
+          </Button>
+        )}
+        <div className="flex-1" />
+        <Button
+          onClick={() => handleOnContinue(value)}
+          disabled={isLoading}
+        >
+          {translations.uploadStep.selectSheet.nextButtonTitle}
+        </Button>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/Steps.tsx

@@ -0,0 +1,81 @@
+import { StepState, StepType, UploadFlow } from "./UploadFlow"
+import { useRsi } from "../hooks/useRsi"
+import { useRef, useState } from "react"
+import { steps, stepTypeToStepIndex, stepIndexToStepType } from "../utils/steps"
+import { CgCheck } from "react-icons/cg"
+
+const CheckIcon = ({ color }: { color: string }) => <CgCheck size="24" className={color} />
+
+export const Steps = () => {
+  const { initialStepState, translations, isNavigationEnabled } = useRsi()
+  const initialStep = stepTypeToStepIndex(initialStepState?.type)
+  const [activeStep, setActiveStep] = useState(initialStep)
+  const [state, setState] = useState<StepState>(initialStepState || { type: StepType.upload })
+  const history = useRef<StepState[]>([])
+
+  const onClickStep = (stepIndex: number) => {
+    const type = stepIndexToStepType(stepIndex)
+    const historyIdx = history.current.findIndex((v) => v.type === type)
+    if (historyIdx === -1) return
+    const nextHistory = history.current.slice(0, historyIdx + 1)
+    history.current = nextHistory
+    setState(nextHistory[nextHistory.length - 1])
+    setActiveStep(stepIndex)
+  }
+
+  const onBack = () => {
+    onClickStep(Math.max(activeStep - 1, 0))
+  }
+
+  const onNext = (v: StepState) => {
+    history.current.push(state)
+    setState(v)
+    v.type !== StepType.selectSheet && setActiveStep(activeStep + 1)
+  }
+
+  return (
+    <>
+      <div className="hidden border-b bg-muted px-4 py-6 md:block">
+        <nav className="mx-auto flex items-center justify-center gap-4 lg:gap-24" aria-label="Steps">
+          {steps.map((key, index) => {
+            const isActive = index === activeStep
+            const isCompleted = index < activeStep
+            return (
+              <div key={key} className="flex items-center">
+                <button
+                  className={`group flex items-center ${isNavigationEnabled ? 'cursor-pointer' : 'cursor-default'}`}
+                  onClick={isNavigationEnabled ? () => onClickStep(index) : undefined}
+                  disabled={!isNavigationEnabled}
+                >
+                  <div className={`flex shrink-0 h-10 w-10 items-center justify-center rounded-full border-2 ${
+                    isActive 
+                      ? 'border-primary bg-primary text-primary-foreground'
+                      : isCompleted
+                      ? 'border-primary bg-primary text-primary-foreground'
+                      : 'border-muted-foreground/20 bg-background'
+                  }`}>
+                    {isCompleted ? (
+                      <CheckIcon color="text-primary-foreground" />
+                    ) : (
+                      <span className={`text-sm font-medium ${
+                        isActive ? 'text-primary-foreground' : 'text-muted-foreground'
+                      }`}>
+                        {index + 1}
+                      </span>
+                    )}
+                  </div>
+                  <span className={`ml-2 text-sm font-medium ${
+                    isActive ? 'text-foreground' : 'text-muted-foreground'
+                  }`}>
+                    {translations[key].title}
+                  </span>
+                </button>
+              </div>
+            )
+          })}
+        </nav>
+      </div>
+      <UploadFlow state={state} onNext={onNext} onBack={isNavigationEnabled ? onBack : undefined} />
+    </>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadFlow.tsx

@@ -0,0 +1,168 @@
+import { useCallback, useState } from "react"
+import type XLSX from "xlsx"
+import { UploadStep } from "./UploadStep/UploadStep"
+import { SelectHeaderStep } from "./SelectHeaderStep/SelectHeaderStep"
+import { SelectSheetStep } from "./SelectSheetStep/SelectSheetStep"
+import { mapWorkbook } from "../utils/mapWorkbook"
+import { ValidationStep } from "./ValidationStep/ValidationStep"
+import { addErrorsAndRunHooks } from "./ValidationStep/utils/dataMutations"
+import { MatchColumnsStep } from "./MatchColumnsStep/MatchColumnsStep"
+import { exceedsMaxRecords } from "../utils/exceedsMaxRecords"
+import { useRsi } from "../hooks/useRsi"
+import type { RawData } from "../types"
+import { Progress } from "@/components/ui/progress"
+import { useToast } from "@/hooks/use-toast"
+
+export enum StepType {
+  upload = "upload",
+  selectSheet = "selectSheet",
+  selectHeader = "selectHeader",
+  matchColumns = "matchColumns",
+  validateData = "validateData",
+}
+export type StepState =
+  | {
+      type: StepType.upload
+    }
+  | {
+      type: StepType.selectSheet
+      workbook: XLSX.WorkBook
+    }
+  | {
+      type: StepType.selectHeader
+      data: RawData[]
+    }
+  | {
+      type: StepType.matchColumns
+      data: RawData[]
+      headerValues: RawData
+    }
+  | {
+      type: StepType.validateData
+      data: any[]
+    }
+
+interface Props {
+  state: StepState
+  onNext: (v: StepState) => void
+  onBack?: () => void
+}
+
+export const UploadFlow = ({ state, onNext, onBack }: Props) => {
+  const {
+    maxRecords,
+    translations,
+    uploadStepHook,
+    selectHeaderStepHook,
+    matchColumnsStepHook,
+    fields,
+    rowHook,
+    tableHook,
+  } = useRsi()
+  const [uploadedFile, setUploadedFile] = useState<File | null>(null)
+  const { toast } = useToast()
+  const errorToast = useCallback(
+    (description: string) => {
+      toast({
+        variant: "destructive",
+        title: translations.alerts.toast.error,
+        description,
+      })
+    },
+    [toast, translations],
+  )
+
+  switch (state.type) {
+    case StepType.upload:
+      return (
+        <UploadStep
+          onContinue={async (workbook, file) => {
+            setUploadedFile(file)
+            const isSingleSheet = workbook.SheetNames.length === 1
+            if (isSingleSheet) {
+              if (maxRecords && exceedsMaxRecords(workbook.Sheets[workbook.SheetNames[0]], maxRecords)) {
+                errorToast(translations.uploadStep.maxRecordsExceeded(maxRecords.toString()))
+                return
+              }
+              try {
+                const mappedWorkbook = await uploadStepHook(mapWorkbook(workbook))
+                onNext({
+                  type: StepType.selectHeader,
+                  data: mappedWorkbook,
+                })
+              } catch (e) {
+                errorToast((e as Error).message)
+              }
+            } else {
+              onNext({ type: StepType.selectSheet, workbook })
+            }
+          }}
+        />
+      )
+    case StepType.selectSheet:
+      return (
+        <SelectSheetStep
+          sheetNames={state.workbook.SheetNames}
+          onContinue={async (sheetName) => {
+            if (maxRecords && exceedsMaxRecords(state.workbook.Sheets[sheetName], maxRecords)) {
+              errorToast(translations.uploadStep.maxRecordsExceeded(maxRecords.toString()))
+              return
+            }
+            try {
+              const mappedWorkbook = await uploadStepHook(mapWorkbook(state.workbook, sheetName))
+              onNext({
+                type: StepType.selectHeader,
+                data: mappedWorkbook,
+              })
+            } catch (e) {
+              errorToast((e as Error).message)
+            }
+          }}
+          onBack={onBack}
+        />
+      )
+    case StepType.selectHeader:
+      return (
+        <SelectHeaderStep
+          data={state.data}
+          onContinue={async (...args) => {
+            try {
+              const { data, headerValues } = await selectHeaderStepHook(...args)
+              onNext({
+                type: StepType.matchColumns,
+                data,
+                headerValues,
+              })
+            } catch (e) {
+              errorToast((e as Error).message)
+            }
+          }}
+          onBack={onBack}
+        />
+      )
+    case StepType.matchColumns:
+      return (
+        <MatchColumnsStep
+          data={state.data}
+          headerValues={state.headerValues}
+          onContinue={async (values, rawData, columns) => {
+            try {
+              const data = await matchColumnsStepHook(values, rawData, columns)
+              const dataWithMeta = await addErrorsAndRunHooks(data, fields, rowHook, tableHook)
+              onNext({
+                type: StepType.validateData,
+                data: dataWithMeta,
+              })
+            } catch (e) {
+              errorToast((e as Error).message)
+            }
+          }}
+          onBack={onBack}
+        />
+      )
+    case StepType.validateData:
+      return <ValidationStep initialData={state.data} file={uploadedFile!} onBack={onBack} />
+    default:
+      return <Progress value={33} className="w-full" />
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/UploadStep.tsx

@@ -0,0 +1,39 @@
+import type XLSX from "xlsx"
+import { useCallback, useState } from "react"
+import { useRsi } from "../../hooks/useRsi"
+import { DropZone } from "./components/DropZone"
+import { ExampleTable } from "./components/ExampleTable"
+import { FadingOverlay } from "./components/FadingOverlay"
+
+type UploadProps = {
+  onContinue: (data: XLSX.WorkBook, file: File) => Promise<void>
+}
+
+export const UploadStep = ({ onContinue }: UploadProps) => {
+  const [isLoading, setIsLoading] = useState(false)
+  const { translations, fields } = useRsi()
+  
+  const handleOnContinue = useCallback(
+    async (data: XLSX.WorkBook, file: File) => {
+      setIsLoading(true)
+      await onContinue(data, file)
+      setIsLoading(false)
+    },
+    [onContinue],
+  )
+  
+  return (
+    <div className="p-6">
+      <h2 className="text-2xl font-semibold mb-4">{translations.uploadStep.title}</h2>
+      <p className="text-lg mb-2">{translations.uploadStep.manifestTitle}</p>
+      <p className="text-muted-foreground mb-6">{translations.uploadStep.manifestDescription}</p>
+      <div className="relative mb-0 border-t  rounded-lg h-[80px]">
+        <div className="absolute inset-0">
+          <ExampleTable fields={fields} />
+        </div>
+        <FadingOverlay />
+      </div>
+      <DropZone onContinue={handleOnContinue} isLoading={isLoading} />
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/components/DropZone.tsx

@@ -0,0 +1,85 @@
+import { useDropzone } from "react-dropzone"
+import * as XLSX from "xlsx"
+import { useState } from "react"
+import { useRsi } from "../../../hooks/useRsi"
+import { readFileAsync } from "../utils/readFilesAsync"
+import { Button } from "@/components/ui/button"
+import { useToast } from "@/hooks/use-toast"
+import { cn } from "@/lib/utils"
+
+type DropZoneProps = {
+  onContinue: (data: XLSX.WorkBook, file: File) => void
+  isLoading: boolean
+}
+
+export const DropZone = ({ onContinue, isLoading }: DropZoneProps) => {
+  const { translations, maxFileSize, dateFormat, parseRaw } = useRsi()
+  const { toast } = useToast()
+  const [loading, setLoading] = useState(false)
+  const { getRootProps, getInputProps, isDragActive, open } = useDropzone({
+    noClick: true,
+    noKeyboard: true,
+    maxFiles: 1,
+    maxSize: maxFileSize,
+    accept: {
+      "application/vnd.ms-excel": [".xls"],
+      "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": [".xlsx"],
+      "text/csv": [".csv"],
+    },
+    onDropRejected: (fileRejections) => {
+      setLoading(false)
+      fileRejections.forEach((fileRejection) => {
+        toast({
+          variant: "destructive",
+          title: `${fileRejection.file.name} ${translations.uploadStep.dropzone.errorToastDescription}`,
+          description: fileRejection.errors[0].message,
+        })
+      })
+    },
+    onDropAccepted: async ([file]) => {
+      setLoading(true)
+      const arrayBuffer = await readFileAsync(file)
+      const workbook = XLSX.read(arrayBuffer, {
+        cellDates: true,
+        dateNF: dateFormat,
+        raw: parseRaw,
+        dense: true,
+        type: 'array',
+        codepage: 65001, // UTF-8
+        WTF: false // Don't throw on errors
+      })
+      setLoading(false)
+      onContinue(workbook, file)
+    },
+  })
+
+  return (
+    <div
+      {...getRootProps()}
+      className={cn(
+        "flex h-full w-full flex-1 flex-col items-center justify-center rounded-lg border-2 border-dashed border-secondary-foreground/30 bg-muted/90 p-12",
+        isDragActive && "border-primary bg-muted"
+      )}
+    >
+      <input {...getInputProps()} data-testid="rsi-dropzone" />
+      {isDragActive ? (
+        <p className="text-lg text-muted-foreground mb-1 py-6">
+          {translations.uploadStep.dropzone.activeDropzoneTitle}
+        </p>
+      ) : loading || isLoading ? (
+        <p className="text-lg text-muted-foreground">
+          {translations.uploadStep.dropzone.loadingTitle}
+        </p>
+      ) : (
+        <>
+          <p className="mb-4 text-lg text-muted-foreground">
+            {translations.uploadStep.dropzone.title}
+          </p>
+          <Button onClick={open} variant="default">
+            {translations.uploadStep.dropzone.buttonTitle}
+          </Button>
+        </>
+      )}
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/components/ExampleTable.tsx

@@ -0,0 +1,25 @@
+import type { Fields } from "../../../types"
+import { useMemo } from "react"
+import { Table } from "../../../components/Table"
+import { generateColumns } from "./columns"
+import { generateExampleRow } from "../utils/generateExampleRow"
+
+interface Props<T extends string> {
+  fields: Fields<T>
+}
+
+export const ExampleTable = <T extends string>({ fields }: Props<T>) => {
+  const data = useMemo(() => generateExampleRow(fields), [fields])
+  const columns = useMemo(() => generateColumns(fields), [fields])
+
+  return (
+    <div className="h-full w-full">
+      <Table 
+        rows={data} 
+        columns={columns} 
+        className="rdg-example h-full"
+        style={{ height: '100%' }}
+      />
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/components/FadingOverlay.tsx

@@ -0,0 +1,5 @@
+export const FadingOverlay = () => (
+  <div
+    className="absolute inset-x-0 bottom-0 h-12 pointer-events-none bg-gradient-to-t from-background to-transparent"
+  />
+)

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/components/columns.tsx

@@ -0,0 +1,44 @@
+import type { Column } from "react-data-grid"
+import type { Fields } from "../../../types"
+import { CgInfo } from "react-icons/cg"
+import {
+  Tooltip,
+  TooltipContent,
+  TooltipProvider,
+  TooltipTrigger,
+} from "@/components/ui/tooltip"
+
+export const generateColumns = <T extends string>(fields: Fields<T>) =>
+  fields.map(
+    (column): Column<any> => ({
+      key: column.key,
+      name: column.label,
+      minWidth: 150,
+      headerRenderer: () => (
+        <div className="flex items-center gap-1 relative">
+          <div className="flex-1 overflow-hidden text-ellipsis">
+            {column.label}
+          </div>
+          {column.description && (
+            <TooltipProvider>
+              <Tooltip>
+                <TooltipTrigger asChild>
+                  <div className="flex-none">
+                    <CgInfo className="h-4 w-4" />
+                  </div>
+                </TooltipTrigger>
+                <TooltipContent>
+                  {column.description}
+                </TooltipContent>
+              </Tooltip>
+            </TooltipProvider>
+          )}
+        </div>
+      ),
+      formatter: ({ row }) => (
+        <div className="min-w-full min-h-full overflow-hidden text-ellipsis">
+          {row[column.key]}
+        </div>
+      ),
+    }),
+  )

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/utils/generateExampleRow.ts

@@ -0,0 +1,14 @@
+import type { Field, Fields } from "../../../types"
+
+const titleMap: Record<Field<string>["fieldType"]["type"], string> = {
+  checkbox: "Boolean",
+  select: "Options",
+  input: "Text",
+}
+
+export const generateExampleRow = <T extends string>(fields: Fields<T>) => [
+  fields.reduce((acc, field) => {
+    acc[field.key as T] = field.example || titleMap[field.fieldType.type]
+    return acc
+  }, {} as Record<T, string>),
+]

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/utils/getDropZoneBorder.ts

@@ -0,0 +1,9 @@
+export const getDropZoneBorder = (color: string) => {
+  return {
+    bgGradient: `repeating-linear(0deg, ${color}, ${color} 10px, transparent 10px, transparent 20px, ${color} 20px), repeating-linear-gradient(90deg, ${color}, ${color} 10px, transparent 10px, transparent 20px, ${color} 20px), repeating-linear-gradient(180deg, ${color}, ${color} 10px, transparent 10px, transparent 20px, ${color} 20px), repeating-linear-gradient(270deg, ${color}, ${color} 10px, transparent 10px, transparent 20px, ${color} 20px)`,
+    backgroundSize: "2px 100%, 100% 2px, 2px 100% , 100% 2px",
+    backgroundPosition: "0 0, 0 0, 100% 0, 0 100%",
+    backgroundRepeat: "no-repeat",
+    borderRadius: "4px",
+  }
+}

inventory/src/lib/react-spreadsheet-import/src/steps/UploadStep/utils/readFilesAsync.ts

@@ -0,0 +1,13 @@
+export const readFileAsync = (file: File) => {
+  return new Promise((resolve, reject) => {
+    const reader = new FileReader()
+
+    reader.onload = () => {
+      resolve(reader.result)
+    }
+
+    reader.onerror = reject
+
+    reader.readAsArrayBuffer(file)
+  })
+}

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStep/ValidationStep.tsx

@@ -0,0 +1,518 @@
+import { useCallback, useMemo, useState } from "react"
+import { useRsi } from "../../hooks/useRsi"
+import type { Meta } from "./types"
+import { addErrorsAndRunHooks } from "./utils/dataMutations"
+import type { Data, Field, SelectOption } from "../../types"
+import {
+  Table,
+  TableBody,
+  TableCell,
+  TableHead,
+  TableHeader,
+  TableRow,
+} from "@/components/ui/table"
+import {
+  useReactTable,
+  getCoreRowModel,
+  type ColumnDef,
+  flexRender,
+  type RowSelectionState,
+} from "@tanstack/react-table"
+import { Checkbox } from "@/components/ui/checkbox"
+import { Input } from "@/components/ui/input"
+import {
+  AlertDialog,
+  AlertDialogAction,
+  AlertDialogCancel,
+  AlertDialogContent,
+  AlertDialogDescription,
+  AlertDialogFooter,
+  AlertDialogHeader,
+  AlertDialogTitle,
+  AlertDialogPortal,
+  AlertDialogOverlay,
+} from "@/components/ui/alert-dialog"
+import { Button } from "@/components/ui/button"
+import { Switch } from "@/components/ui/switch"
+import { useToast } from "@/hooks/use-toast"
+import {
+  Select,
+  SelectContent,
+  SelectItem,
+  SelectTrigger,
+  SelectValue,
+} from "@/components/ui/select"
+
+type Props<T extends string> = {
+  initialData: (Data<T> & Meta)[]
+  file: File
+  onBack?: () => void
+}
+
+type CellProps = {
+  value: any,
+  onChange: (value: any) => void,
+  error?: { level: string, message: string },
+  field: Field<string>
+}
+
+const EditableCell = ({ value, onChange, error, field }: CellProps) => {
+  const [isEditing, setIsEditing] = useState(false)
+  const [inputValue, setInputValue] = useState(value ?? "")
+  
+  const getDisplayValue = (value: any, fieldType: Field<string>["fieldType"]) => {
+    if (fieldType.type === "select") {
+      return fieldType.options.find((opt: SelectOption) => opt.value === value)?.label || value
+    }
+    if (fieldType.type === "checkbox") {
+      if (typeof value === "boolean") return value ? "Yes" : "No"
+      return value
+    }
+    return value
+  }
+
+  const isRequiredAndEmpty = field.validations?.some(v => v.rule === "required") && !value
+
+  // Show editing UI for:
+  // 1. Error cells
+  // 2. When actively editing
+  // 3. Required select fields that are empty
+  // 4. Checkbox fields (always show the checkbox)
+  const shouldShowEditUI = error?.level === "error" || 
+    isEditing || 
+    (field.fieldType.type === "select" && isRequiredAndEmpty) ||
+    field.fieldType.type === "checkbox"
+
+  if (shouldShowEditUI) {
+    switch (field.fieldType.type) {
+      case "select":
+        return (
+          <Select
+            defaultOpen={isEditing}
+            value={value as string || ""}
+            onValueChange={(newValue) => {
+              onChange(newValue)
+              setIsEditing(false)
+            }}
+          >
+            <SelectTrigger 
+              className={`w-full ${
+                (error?.level === "error" || isRequiredAndEmpty)
+                  ? "border-destructive text-destructive" 
+                  : ""
+              }`}
+            >
+              <SelectValue placeholder="Select..." />
+            </SelectTrigger>
+            <SelectContent>
+              {field.fieldType.options.map((option: SelectOption) => (
+                <SelectItem key={option.value} value={option.value}>
+                  {option.label}
+                </SelectItem>
+              ))}
+            </SelectContent>
+          </Select>
+        )
+      case "checkbox":
+        return (
+          <div className="flex items-center gap-2">
+            <Checkbox
+              checked={Boolean(value)}
+              onCheckedChange={(checked) => {
+                onChange(checked)
+              }}
+            />
+          </div>
+        )
+      default:
+        return (
+          <Input
+            value={inputValue}
+            onChange={(e) => setInputValue(e.target.value)}
+            onKeyDown={(e) => {
+              if (e.key === "Enter") {
+                onChange(inputValue)
+                if (!error?.level) {
+                  setIsEditing(false)
+                }
+              }
+            }}
+            onBlur={() => {
+              onChange(inputValue)
+              if (!error?.level) {
+                setIsEditing(false)
+              }
+            }}
+            className={`w-full bg-transparent ${
+              error?.level === "error" 
+                ? "border-destructive text-destructive" 
+                : ""
+            }`}
+            autoFocus={!error?.level}
+          />
+        )
+    }
+  }
+
+  // Display mode
+  return (
+    <div 
+      onClick={() => {
+        if (field.fieldType.type !== "checkbox") {
+          setIsEditing(true)
+          setInputValue(value ?? "")
+        }
+      }}
+      className={`cursor-text py-2 ${
+        error?.level === "error" ? "text-destructive" : ""
+      }`}
+    >
+      {getDisplayValue(value, field.fieldType)}
+    </div>
+  )
+}
+
+export const ValidationStep = <T extends string>({ initialData, file, onBack }: Props<T>) => {
+  const { translations, fields, onClose, onSubmit, rowHook, tableHook, allowInvalidSubmit } = useRsi<T>()
+  const { toast } = useToast()
+
+  const [data, setData] = useState<(Data<T> & Meta)[]>(initialData)
+  const [rowSelection, setRowSelection] = useState<RowSelectionState>({})
+  const [filterByErrors, setFilterByErrors] = useState(false)
+  const [showSubmitAlert, setShowSubmitAlert] = useState(false)
+  const [isSubmitting, setSubmitting] = useState(false)
+
+  // Memoize filtered data to prevent recalculation on every render
+  const filteredData = useMemo(() => {
+    if (!filterByErrors) return data
+    return data.filter(row => 
+      row.__errors && Object.values(row.__errors).some(err => err.level === "error")
+    )
+  }, [data, filterByErrors])
+
+  const updateData = useCallback(
+    async (rows: typeof data, indexes?: number[]) => {
+      // Check if hooks are async - if they are we want to apply changes optimistically for better UX
+      if (rowHook?.constructor.name === "AsyncFunction" || tableHook?.constructor.name === "AsyncFunction") {
+        setData(rows)
+      }
+      addErrorsAndRunHooks<T>(rows, fields, rowHook, tableHook, indexes).then((data) => setData(data))
+    },
+    [rowHook, tableHook, fields],
+  )
+
+  const updateRows = useCallback(
+    (rowIndex: number, columnId: string, value: string) => {
+      const newData = [...data]
+      // Get the actual row from the filtered or unfiltered data
+      const row = filteredData[rowIndex]
+      if (row) {
+        // Find the original index in the full dataset
+        const originalIndex = data.findIndex(r => r.__index === row.__index)
+        const updatedRow = {
+          ...row,
+          [columnId]: value,
+        }
+        newData[originalIndex] = updatedRow
+        updateData(newData, [originalIndex])
+      }
+    },
+    [data, filteredData, updateData],
+  )
+
+  const columns = useMemo<ColumnDef<Data<T> & Meta>[]>(() => {
+    const baseColumns: ColumnDef<Data<T> & Meta>[] = [
+      {
+        id: "select",
+        header: ({ table }) => (
+          <Checkbox
+            checked={table.getIsAllPageRowsSelected()}
+            onCheckedChange={(value) => table.toggleAllPageRowsSelected(!!value)}
+            aria-label="Select all"
+          />
+        ),
+        cell: ({ row }) => (
+          <Checkbox
+            checked={row.getIsSelected()}
+            onCheckedChange={(value) => row.toggleSelected(!!value)}
+            aria-label="Select row"
+          />
+        ),
+        enableSorting: false,
+        enableHiding: false,
+        size: 50,
+      },
+      ...fields.map((field: Field<T>): ColumnDef<Data<T> & Meta> => ({
+        accessorKey: field.key,
+        header: field.label,
+        cell: ({ row, column }) => {
+          const value = row.getValue(column.id)
+          const error = row.original.__errors?.[column.id]
+          const rowIndex = row.index
+          
+          return (
+            <EditableCell
+              value={value}
+              onChange={(newValue) => updateRows(rowIndex, column.id, newValue)}
+              error={error}
+              field={field}
+            />
+          )
+        },
+        // Use configured width or fallback to sensible defaults
+        size: (field as any).width || (
+          field.fieldType.type === "checkbox" ? 80 :
+          field.fieldType.type === "select" ? 150 : 
+          200
+        ),
+      })),
+    ]
+    return baseColumns
+  }, [fields, updateRows])
+
+  const table = useReactTable({
+    data: filteredData,
+    columns,
+    state: {
+      rowSelection,
+    },
+    enableRowSelection: true,
+    onRowSelectionChange: setRowSelection,
+    getCoreRowModel: getCoreRowModel(),
+  })
+
+  const deleteSelectedRows = () => {
+    if (Object.keys(rowSelection).length) {
+      const selectedRows = Object.keys(rowSelection).map(Number)
+      const newData = data.filter((_, index) => !selectedRows.includes(index))
+      updateData(newData)
+      setRowSelection({})
+    }
+  }
+
+  const normalizeValue = useCallback((value: any, field: Field<T>) => {
+    if (field.fieldType.type === "checkbox") {
+      if (typeof value === "boolean") return value
+      if (typeof value === "string") {
+        const normalizedValue = value.toLowerCase().trim()
+        if (field.fieldType.booleanMatches) {
+          return !!field.fieldType.booleanMatches[normalizedValue]
+        }
+        return ["yes", "true", "1"].includes(normalizedValue)
+      }
+      return false
+    }
+    if (field.fieldType.type === "select") {
+      // Ensure the value matches one of the options
+      if (field.fieldType.options.some(opt => opt.value === value)) {
+        return value
+      }
+      // Try to match by label
+      const matchByLabel = field.fieldType.options.find(
+        opt => opt.label.toLowerCase() === String(value).toLowerCase()
+      )
+      return matchByLabel ? matchByLabel.value : value
+    }
+    return value
+  }, [])
+
+  const submitData = async () => {
+    const calculatedData = data.reduce(
+      (acc, value) => {
+        const { __index, __errors, ...values } = value
+        
+        // Normalize values based on field types
+        const normalizedValues = Object.entries(values).reduce((obj, [key, val]) => {
+          const field = fields.find((f: Field<T>) => f.key === key)
+          if (field) {
+            obj[key as keyof Data<T>] = normalizeValue(val, field)
+          } else {
+            obj[key as keyof Data<T>] = val as string | boolean | undefined
+          }
+          return obj
+        }, {} as Data<T>)
+
+        if (__errors) {
+          for (const key in __errors) {
+            if (__errors[key].level === "error") {
+              acc.invalidData.push(normalizedValues)
+              return acc
+            }
+          }
+        }
+        acc.validData.push(normalizedValues)
+        return acc
+      },
+      { validData: [] as Data<T>[], invalidData: [] as Data<T>[], all: data },
+    )
+    setShowSubmitAlert(false)
+    setSubmitting(true)
+    const response = onSubmit(calculatedData, file)
+    if (response?.then) {
+      response
+        .then(() => {
+          onClose()
+        })
+        .catch((err: Error) => {
+          toast({
+            variant: "destructive",
+            title: translations.alerts.submitError.title,
+            description: err?.message || translations.alerts.submitError.defaultMessage,
+          })
+        })
+        .finally(() => {
+          setSubmitting(false)
+        })
+    } else {
+      onClose()
+    }
+  }
+  const onContinue = () => {
+    const invalidData = data.find((value) => {
+      if (value?.__errors) {
+        return !!Object.values(value.__errors)?.filter((err) => err.level === "error").length
+      }
+      return false
+    })
+    if (!invalidData) {
+      submitData()
+    } else {
+      setShowSubmitAlert(true)
+    }
+  }
+
+  return (
+    <div className="flex h-[calc(100vh-9.5rem)] flex-col">
+      <AlertDialog open={showSubmitAlert} onOpenChange={setShowSubmitAlert}>
+        <AlertDialogPortal>
+          <AlertDialogOverlay className="z-[1400]" />
+          <AlertDialogContent className="z-[1500]">
+            <AlertDialogHeader>
+              <AlertDialogTitle>
+                {translations.alerts.submitIncomplete.headerTitle}
+              </AlertDialogTitle>
+              <AlertDialogDescription>
+                {allowInvalidSubmit
+                  ? translations.alerts.submitIncomplete.bodyText
+                  : translations.alerts.submitIncomplete.bodyTextSubmitForbidden}
+              </AlertDialogDescription>
+            </AlertDialogHeader>
+            <AlertDialogFooter>
+              <AlertDialogCancel>
+                {translations.alerts.submitIncomplete.cancelButtonTitle}
+              </AlertDialogCancel>
+              {allowInvalidSubmit && (
+                <AlertDialogAction onClick={submitData}>
+                  {translations.alerts.submitIncomplete.finishButtonTitle}
+                </AlertDialogAction>
+              )}
+            </AlertDialogFooter>
+          </AlertDialogContent>
+        </AlertDialogPortal>
+      </AlertDialog>
+      <div className="flex-1 overflow-hidden">
+        <div className="px-8 py-6">
+          <div className="mb-8 flex flex-wrap items-center justify-between gap-2">
+            <h2 className="text-3xl font-semibold text-foreground">
+              {translations.validationStep.title}
+            </h2>
+            <div className="flex flex-wrap items-center gap-4">
+              <Button 
+                variant="outline" 
+                size="sm" 
+                onClick={deleteSelectedRows}
+              >
+                {translations.validationStep.discardButtonTitle}
+              </Button>
+              <div className="flex items-center gap-2">
+                <Switch
+                  checked={filterByErrors}
+                  onCheckedChange={setFilterByErrors}
+                  id="filter-errors"
+                />
+                <label htmlFor="filter-errors" className="text-sm text-muted-foreground">
+                  {translations.validationStep.filterSwitchTitle}
+                </label>
+              </div>
+            </div>
+          </div>
+          <div className="rounded-md border overflow-hidden">
+            <div className="overflow-x-auto">
+              <Table>
+                <TableHeader>
+                  {table.getHeaderGroups().map((headerGroup) => (
+                    <TableRow key={headerGroup.id}>
+                      {headerGroup.headers.map((header) => (
+                        <TableHead 
+                          key={header.id}
+                          style={{
+                            width: header.getSize(),
+                            minWidth: header.getSize(),
+                          }}
+                        >
+                          {header.isPlaceholder
+                            ? null
+                            : flexRender(
+                                header.column.columnDef.header,
+                                header.getContext()
+                              )}
+                        </TableHead>
+                      ))}
+                    </TableRow>
+                  ))}
+                </TableHeader>
+                <TableBody>
+                  {table.getRowModel().rows?.length ? (
+                    table.getRowModel().rows.map((row) => (
+                      <TableRow
+                        key={row.id}
+                        data-state={row.getIsSelected() && "selected"}
+                      >
+                        {row.getVisibleCells().map((cell) => (
+                          <TableCell 
+                            key={cell.id} 
+                            className="p-2"
+                            style={{
+                              width: cell.column.getSize(),
+                              minWidth: cell.column.getSize(),
+                            }}
+                          >
+                            {flexRender(cell.column.columnDef.cell, cell.getContext())}
+                          </TableCell>
+                        ))}
+                      </TableRow>
+                    ))
+                  ) : (
+                    <TableRow>
+                      <TableCell colSpan={columns.length} className="h-24 text-center">
+                        {filterByErrors
+                          ? translations.validationStep.noRowsMessageWhenFiltered
+                          : translations.validationStep.noRowsMessage}
+                      </TableCell>
+                    </TableRow>
+                  )}
+                </TableBody>
+              </Table>
+            </div>
+          </div>
+        </div>
+      </div>
+      <div className="border-t bg-muted px-8 py-4 mt-5">
+        <div className="flex items-center justify-between">
+          {onBack && (
+            <Button variant="outline" onClick={onBack}>
+              {translations.validationStep.backButtonTitle}
+            </Button>
+          )}
+          <Button 
+            className="ml-auto"
+            disabled={isSubmitting}
+            onClick={onContinue}
+          >
+            {translations.validationStep.nextButtonTitle}
+          </Button>
+        </div>
+      </div>
+    </div>
+  )
+}

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStep/components/columns.tsx

@@ -0,0 +1,191 @@
+import type { Column as RDGColumn, RenderEditCellProps, FormatterProps } from "react-data-grid"
+import { useRowSelection } from "react-data-grid"
+import { Checkbox, Input, Switch } from "@chakra-ui/react"
+import type { Data, Fields, Field, SelectOption } from "../../../types"
+import type { ChangeEvent } from "react"
+import type { Meta } from "../types"
+import { CgInfo } from "react-icons/cg"
+import {
+  Select,
+  SelectContent,
+  SelectItem,
+  SelectTrigger,
+  SelectValue,
+} from "@/components/ui/select"
+
+const SELECT_COLUMN_KEY = "select-row"
+
+function autoFocusAndSelect(input: HTMLInputElement | null) {
+  input?.focus()
+  input?.select()
+}
+
+type RowType<T extends string> = Data<T> & Meta
+
+export const generateColumns = <T extends string>(fields: Fields<T>): RDGColumn<RowType<T>>[] => [
+  {
+    key: SELECT_COLUMN_KEY,
+    name: "",
+    width: 35,
+    minWidth: 35,
+    maxWidth: 35,
+    resizable: false,
+    sortable: false,
+    frozen: true,
+    cellClass: "rdg-checkbox",
+    formatter: (props: FormatterProps<RowType<T>>) => {
+      // eslint-disable-next-line  react-hooks/rules-of-hooks
+      const [isRowSelected, onRowSelectionChange] = useRowSelection()
+      return (
+        <Checkbox
+          bg="white"
+          aria-label="Select"
+          isChecked={isRowSelected}
+          onChange={(event) => {
+            onRowSelectionChange({
+              row: props.row,
+              checked: Boolean(event.target.checked),
+              isShiftClick: (event.nativeEvent as MouseEvent).shiftKey,
+            })
+          }}
+        />
+      )
+    },
+  },
+  ...fields.map(
+    (column: Field<T>): RDGColumn<RowType<T>> => ({
+      key: column.key,
+      name: column.label,
+      minWidth: 150,
+      resizable: true,
+      headerRenderer: () => (
+        <div className="flex gap-1 items-center relative">
+          <div className="flex-1 overflow-hidden text-ellipsis">
+            {column.label}
+          </div>
+          {column.description && (
+            <div className="flex-none">
+              <CgInfo className="h-4 w-4" />
+            </div>
+          )}
+        </div>
+      ),
+      editable: column.fieldType.type !== "checkbox",
+      editor: ({ row, onRowChange, onClose }: RenderEditCellProps<RowType<T>>) => {
+        let component
+
+        switch (column.fieldType.type) {
+          case "select":
+            component = (
+              <Select
+                defaultOpen
+                value={row[column.key] as string}
+                onValueChange={(value) => {
+                  onRowChange({ ...row, [column.key]: value }, true)
+                }}
+              >
+                <SelectTrigger className="w-full border-0 focus:ring-0">
+                  <SelectValue />
+                </SelectTrigger>
+                <SelectContent 
+                  position="popper"
+                  className="z-[1000]"
+                  align="start"
+                  side="bottom"
+                >
+                  {column.fieldType.options.map((option: SelectOption) => (
+                    <SelectItem key={option.value} value={option.value}>
+                      {option.label}
+                    </SelectItem>
+                  ))}
+                </SelectContent>
+              </Select>
+            )
+            break
+          default:
+            component = (
+              <div className="pl-2">
+                <Input
+                  ref={autoFocusAndSelect}
+                  variant="unstyled"
+                  autoFocus
+                  size="small"
+                  value={row[column.key] as string}
+                  onChange={(event: ChangeEvent<HTMLInputElement>) => {
+                    onRowChange({ ...row, [column.key]: event.target.value })
+                  }}
+                  onBlur={() => onClose(true)}
+                />
+              </div>
+            )
+        }
+
+        return component
+      },
+      editorOptions: {
+        editOnClick: true,
+      },
+      formatter: ({ row, onRowChange }: FormatterProps<RowType<T>>) => {
+        let component
+
+        switch (column.fieldType.type) {
+          case "checkbox":
+            component = (
+              <div
+                className="flex items-center h-full"
+                onClick={(event) => {
+                  event.stopPropagation()
+                }}
+              >
+                <Switch
+                  isChecked={row[column.key] as boolean}
+                  onChange={() => {
+                    onRowChange({ ...row, [column.key]: !row[column.key as T] })
+                  }}
+                />
+              </div>
+            )
+            break
+          case "select":
+            component = (
+              <div className="min-w-full min-h-full overflow-hidden text-ellipsis">
+                {column.fieldType.options.find((option: SelectOption) => option.value === row[column.key as T])?.label || null}
+              </div>
+            )
+            break
+          default:
+            component = (
+              <div className="min-w-full min-h-full overflow-hidden text-ellipsis">
+                {row[column.key as T]}
+              </div>
+            )
+        }
+
+        if (row.__errors?.[column.key]) {
+          return (
+            <div className="relative group">
+              {component}
+              <div className="absolute left-0 -top-8 z-50 hidden group-hover:block bg-popover text-popover-foreground text-sm p-2 rounded shadow">
+                {row.__errors?.[column.key]?.message}
+              </div>
+            </div>
+          )
+        }
+
+        return component
+      },
+      cellClass: (row: Meta) => {
+        switch (row.__errors?.[column.key]?.level) {
+          case "error":
+            return "rdg-cell-error"
+          case "warning":
+            return "rdg-cell-warning"
+          case "info":
+            return "rdg-cell-info"
+          default:
+            return ""
+        }
+      },
+    }),
+  ),
+]

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStep/types.ts

@@ -0,0 +1,5 @@
+import { InfoWithSource } from "../../types"
+
+export type Meta = { __index: string; __errors?: Error | null }
+export type Error = { [key: string]: InfoWithSource }
+export type Errors = { [id: string]: Error }

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStep/utils/dataMutations.ts

@@ -0,0 +1,151 @@
+import type { Data, Fields, Info, RowHook, TableHook } from "../../../types"
+import type { Meta, Error, Errors } from "../types"
+import { v4 } from "uuid"
+import { ErrorSources } from "../../../types"
+
+export const addErrorsAndRunHooks = async <T extends string>(
+  data: (Data<T> & Partial<Meta>)[],
+  fields: Fields<T>,
+  rowHook?: RowHook<T>,
+  tableHook?: TableHook<T>,
+  changedRowIndexes?: number[],
+): Promise<(Data<T> & Meta)[]> => {
+  const errors: Errors = {}
+
+  const addError = (source: ErrorSources, rowIndex: number, fieldKey: T, error: Info) => {
+    errors[rowIndex] = {
+      ...errors[rowIndex],
+      [fieldKey]: { ...error, source },
+    }
+  }
+
+  if (tableHook) {
+    data = await tableHook(data, (...props) => addError(ErrorSources.Table, ...props))
+  }
+
+  if (rowHook) {
+    if (changedRowIndexes) {
+      for (const index of changedRowIndexes) {
+        data[index] = await rowHook(data[index], (...props) => addError(ErrorSources.Row, index, ...props), data)
+      }
+    } else {
+      data = await Promise.all(
+        data.map(async (value, index) =>
+          rowHook(value, (...props) => addError(ErrorSources.Row, index, ...props), data),
+        ),
+      )
+    }
+  }
+
+  fields.forEach((field) => {
+    field.validations?.forEach((validation) => {
+      switch (validation.rule) {
+        case "unique": {
+          const values = data.map((entry) => entry[field.key as T])
+
+          const taken = new Set() // Set of items used at least once
+          const duplicates = new Set() // Set of items used multiple times
+
+          values.forEach((value) => {
+            if (validation.allowEmpty && !value) {
+              // If allowEmpty is set, we will not validate falsy fields such as undefined or empty string.
+              return
+            }
+
+            if (taken.has(value)) {
+              duplicates.add(value)
+            } else {
+              taken.add(value)
+            }
+          })
+
+          values.forEach((value, index) => {
+            if (duplicates.has(value)) {
+              addError(ErrorSources.Table, index, field.key as T, {
+                level: validation.level || "error",
+                message: validation.errorMessage || "Field must be unique",
+              })
+            }
+          })
+          break
+        }
+        case "required": {
+          const dataToValidate = changedRowIndexes ? changedRowIndexes.map((index) => data[index]) : data
+          dataToValidate.forEach((entry, index) => {
+            const realIndex = changedRowIndexes ? changedRowIndexes[index] : index
+            if (entry[field.key as T] === null || entry[field.key as T] === undefined || entry[field.key as T] === "") {
+              addError(ErrorSources.Row, realIndex, field.key as T, {
+                level: validation.level || "error",
+                message: validation.errorMessage || "Field is required",
+              })
+            }
+          })
+
+          break
+        }
+        case "regex": {
+          const dataToValidate = changedRowIndexes ? changedRowIndexes.map((index) => data[index]) : data
+          const regex = new RegExp(validation.value, validation.flags)
+          dataToValidate.forEach((entry, index) => {
+            const realIndex = changedRowIndexes ? changedRowIndexes[index] : index
+            const value = entry[field.key]?.toString() ?? ""
+            if (!value.match(regex)) {
+              addError(ErrorSources.Row, realIndex, field.key as T, {
+                level: validation.level || "error",
+                message:
+                  validation.errorMessage || `Field did not match the regex /${validation.value}/${validation.flags} `,
+              })
+            }
+          })
+
+          break
+        }
+      }
+    })
+  })
+
+  return data.map((value, index) => {
+    // This is required only for table. Mutates to prevent needless rerenders
+    if (!("__index" in value)) {
+      value.__index = v4()
+    }
+    const newValue = value as Data<T> & Meta
+
+    // If we are validating all indexes, or we did full validation on this row - apply all errors
+    if (!changedRowIndexes || changedRowIndexes.includes(index)) {
+      if (errors[index]) {
+        return { ...newValue, __errors: errors[index] }
+      }
+
+      if (!errors[index] && value?.__errors) {
+        return { ...newValue, __errors: null }
+      }
+    }
+    // if we have not validated this row, keep it's row errors but apply global error changes
+    else {
+      // at this point errors[index] contains only table source errors, previous row and table errors are in value.__errors
+      const hasRowErrors =
+        value.__errors && Object.values(value.__errors).some((error) => error.source === ErrorSources.Row)
+
+      if (!hasRowErrors) {
+        if (errors[index]) {
+          return { ...newValue, __errors: errors[index] }
+        }
+        return newValue
+      }
+
+      const errorsWithoutTableError = Object.entries(value.__errors!).reduce((acc, [key, value]) => {
+        if (value.source === ErrorSources.Row) {
+          acc[key] = value
+        }
+        return acc
+      }, {} as Error)
+
+      const newErrors = { ...errorsWithoutTableError, ...errors[index] }
+
+      return { ...newValue, __errors: newErrors }
+    }
+
+    return newValue
+  })
+}

inventory/src/lib/react-spreadsheet-import/src/theme.ts

@@ -0,0 +1,509 @@
+import { StepsTheme } from "chakra-ui-steps"
+import type { CSSObject } from "@chakra-ui/react"
+import type { DeepPartial } from "ts-essentials"
+import type { ChakraStylesConfig } from "chakra-react-select"
+import type { SelectOption } from "./types"
+
+const StepsComponent: typeof StepsTheme = {
+  ...StepsTheme,
+  baseStyle: (props: any) => {
+    const navigationEnabled = !!props.onClickStep
+    return {
+      ...StepsTheme.baseStyle(props),
+      stepContainer: {
+        ...StepsTheme.baseStyle(props).stepContainer,
+        cursor: navigationEnabled ? "pointer" : "initial",
+      },
+      label: {
+        ...StepsTheme.baseStyle(props).label,
+        color: "textColor",
+      },
+    }
+  },
+  variants: {
+    circles: (props: any) => ({
+      ...StepsTheme.variants.circles(props),
+      step: {
+        ...StepsTheme.variants.circles(props).step,
+        "&:not(:last-child):after": {
+          ...StepsTheme.variants.circles(props).step["&:not(:last-child):after"],
+          backgroundColor: "background",
+        },
+      },
+      stepIconContainer: {
+        ...StepsTheme.variants.circles(props).stepIconContainer,
+        flex: "0 0 auto",
+        bg: "background",
+        borderColor: "background",
+      },
+    }),
+  },
+}
+
+const MatchIconTheme: any = {
+  baseStyle: (props: any) => {
+    return {
+      display: "flex",
+      alignItems: "center",
+      justifyContent: "center",
+      borderRadius: "50%",
+      borderWidth: "2px",
+      bg: "background",
+      borderColor: "yellow.500",
+      color: "background",
+      transitionDuration: "ultra-fast",
+      _highlighted: {
+        bg: "green.500",
+        borderColor: "green.500",
+      },
+    }
+  },
+  defaultProps: {
+    size: "md",
+    colorScheme: "green",
+  },
+}
+
+export const themeOverrides = {
+  colors: {
+    textColor: "#2D3748",
+    subtitleColor: "#718096",
+    inactiveColor: "#A0AEC0",
+    border: "#E2E8F0",
+    background: "white",
+    backgroundAlpha: "rgba(255,255,255,0)",
+    secondaryBackground: "#EDF2F7",
+    highlight: "#E2E8F0",
+    rsi: {
+      50: "#E6E6FF",
+      100: "#C4C6FF",
+      200: "#A2A5FC",
+      300: "#8888FC",
+      400: "#7069FA",
+      500: "#5D55FA",
+      600: "#4D3DF7",
+      700: "#3525E6",
+      800: "#1D0EBE",
+      900: "#0C008C",
+    },
+  },
+  shadows: {
+    outline: 0,
+  },
+  components: {
+    UploadStep: {
+      baseStyle: {
+        heading: {
+          fontSize: "3xl",
+          color: "textColor",
+          mb: "2rem",
+        },
+        title: {
+          fontSize: "2xl",
+          lineHeight: 8,
+          fontWeight: "semibold",
+          color: "textColor",
+        },
+        subtitle: {
+          fontSize: "md",
+          lineHeight: 6,
+          color: "subtitleColor",
+          mb: "1rem",
+        },
+        tableWrapper: {
+          mb: "0.5rem",
+          position: "relative",
+          h: "72px",
+        },
+        dropzoneText: {
+          size: "lg",
+          lineHeight: 7,
+          fontWeight: "semibold",
+          color: "textColor",
+        },
+        dropZoneBorder: "rsi.500",
+        dropzoneButton: {
+          mt: "1rem",
+        },
+      },
+    },
+    SelectSheetStep: {
+      baseStyle: {
+        heading: {
+          color: "textColor",
+          mb: 8,
+          fontSize: "3xl",
+        },
+        radio: {},
+        radioLabel: {
+          color: "textColor",
+        },
+      },
+    },
+    SelectHeaderStep: {
+      baseStyle: {
+        heading: {
+          color: "textColor",
+          mb: 8,
+          fontSize: "3xl",
+        },
+      },
+    },
+    MatchColumnsStep: {
+      baseStyle: {
+        heading: {
+          color: "textColor",
+          mb: 8,
+          fontSize: "3xl",
+        },
+        title: {
+          color: "textColor",
+          fontSize: "2xl",
+          lineHeight: 8,
+          fontWeight: "semibold",
+          mb: 4,
+        },
+        userTable: {
+          header: {
+            fontSize: "xs",
+            lineHeight: 4,
+            fontWeight: "bold",
+            letterSpacing: "wider",
+            color: "textColor",
+            overflow: "hidden",
+            whiteSpace: "nowrap",
+            textOverflow: "ellipsis",
+            ["&[data-ignored]"]: {
+              color: "inactiveColor",
+            },
+          },
+          cell: {
+            fontSize: "sm",
+            lineHeight: 5,
+            fontWeight: "medium",
+            color: "textColor",
+            px: 6,
+            py: 4,
+            overflow: "hidden",
+            whiteSpace: "nowrap",
+            textOverflow: "ellipsis",
+            ["&[data-ignored]"]: {
+              color: "inactiveColor",
+            },
+          },
+          ignoreButton: {
+            size: "xs",
+            colorScheme: "gray",
+            color: "textColor",
+          },
+        },
+        selectColumn: {
+          text: {
+            fontSize: "sm",
+            lineHeight: 5,
+            fontWeight: "normal",
+            color: "inactiveColor",
+            px: 4,
+          },
+          accordionLabel: {
+            color: "blue.600",
+            fontSize: "sm",
+            lineHeight: 5,
+            pl: 1,
+          },
+          selectLabel: {
+            pt: "0.375rem",
+            pb: 2,
+            fontSize: "md",
+            lineHeight: 6,
+            fontWeight: "medium",
+            color: "textColor",
+          },
+        },
+        select: {
+          control: (provided) => ({
+            ...provided,
+            borderColor: "border",
+            _hover: {
+              borderColor: "border",
+            },
+            ["&[data-focus-visible]"]: {
+              borderColor: "border",
+              boxShadow: "none",
+            },
+          }),
+          menu: (provided) => ({
+            ...provided,
+            p: 0,
+            mt: 0,
+          }),
+          menuList: (provided) => ({
+            ...provided,
+            bg: "background",
+            borderColor: "border",
+          }),
+          option: (provided, state) => ({
+            ...provided,
+            color: "textColor",
+            bg: state.isSelected || state.isFocused ? "highlight" : "background",
+            overflow: "hidden",
+            textOverflow: "ellipsis",
+            display: "block",
+            whiteSpace: "nowrap",
+            _hover: {
+              bg: "highlight",
+            },
+          }),
+          placeholder: (provided) => ({
+            ...provided,
+            color: "inactiveColor",
+          }),
+          noOptionsMessage: (provided) => ({
+            ...provided,
+            color: "inactiveColor",
+          }),
+        } as ChakraStylesConfig<SelectOption>,
+      },
+    },
+    ValidationStep: {
+      baseStyle: {
+        heading: {
+          color: "textColor",
+          fontSize: "3xl",
+        },
+        select: {
+          valueContainer: (provided) => ({
+            ...provided,
+            py: 0,
+            px: 1.5,
+          }),
+          inputContainer: (provided) => ({ ...provided, py: 0 }),
+          control: (provided) => ({ ...provided, border: "none" }),
+          input: (provided) => ({ ...provided, color: "textColor" }),
+          menu: (provided) => ({
+            ...provided,
+            p: 0,
+            mt: 0,
+          }),
+          menuList: (provided) => ({
+            ...provided,
+            bg: "background",
+            borderColor: "border",
+          }),
+          option: (provided, state) => ({
+            ...provided,
+            color: "textColor",
+            bg: state.isSelected || state.isFocused ? "highlight" : "background",
+            overflow: "hidden",
+            textOverflow: "ellipsis",
+            display: "block",
+            whiteSpace: "nowrap",
+          }),
+          noOptionsMessage: (provided) => ({
+            ...provided,
+            color: "inactiveColor",
+          }),
+        } as ChakraStylesConfig<SelectOption>,
+      },
+    },
+    MatchIcon: MatchIconTheme,
+    Steps: StepsComponent,
+    Modal: {
+      baseStyle: {
+        dialog: {
+          borderRadius: "md",
+          bg: "background",
+          fontSize: "lg",
+          color: "textColor",
+        },
+        closeModalButton: {},
+        backButton: {
+          gridColumn: "1",
+          gridRow: "1",
+          justifySelf: "start",
+        },
+        continueButton: {
+          gridColumn: "1 / 3",
+          gridRow: "1",
+          justifySelf: "center",
+        },
+      },
+      variants: {
+        rsi: {
+          header: {
+            bg: "secondaryBackground",
+            px: "2rem",
+            py: "1.5rem",
+          },
+          body: {
+            bg: "background",
+            display: "flex",
+            paddingX: "2rem",
+            paddingY: "2rem",
+            flexDirection: "column",
+            flex: 1,
+            overflow: "auto",
+            height: "100%",
+          },
+          footer: {
+            bg: "secondaryBackground",
+            py: "1.5rem",
+            px: "2rem",
+            justifyContent: "center",
+            display: "grid",
+            gridTemplateColumns: "1fr 1fr",
+            gridTemplateRows: "1fr",
+            gap: "1rem",
+          },
+          dialog: {
+            outline: "unset",
+            minH: "calc(var(--chakra-vh) - 4rem)",
+            maxW: "calc(var(--chakra-vw) - 4rem)",
+            my: "2rem",
+            borderRadius: "lg",
+            overflow: "hidden",
+          },
+        },
+      },
+    },
+    Button: {
+      defaultProps: {
+        colorScheme: "rsi",
+      },
+    },
+  },
+  styles: {
+    global: {
+      // supporting older browsers but avoiding fill-available CSS as it doesn't work https://github.com/chakra-ui/chakra-ui/blob/073bbcd21a9caa830d71b61d6302f47aaa5c154d/packages/components/css-reset/src/css-reset.tsx#L5
+      ":root": {
+        "--chakra-vh": "100vh",
+        "--chakra-vw": "100vw",
+      },
+      "@supports (height: 100dvh) and (width: 100dvw) ": {
+        ":root": {
+          "--chakra-vh": "100dvh",
+          "--chakra-vw": "100dvw",
+        },
+      },
+      ".rdg": {
+        contain: "size layout style paint",
+        borderRadius: "lg",
+        border: "none",
+        borderTop: "1px solid var(--rdg-border-color)",
+        blockSize: "100%",
+        flex: "1",
+
+        // we have to use vars here because chakra does not autotransform unknown props
+        "--rdg-row-height": "35px",
+        "--rdg-color": "var(--chakra-colors-textColor)",
+        "--rdg-background-color": "var(--chakra-colors-background)",
+        "--rdg-header-background-color": "var(--chakra-colors-background)",
+        "--rdg-row-hover-background-color": "var(--chakra-colors-background)",
+        "--rdg-selection-color": "var(--chakra-colors-blue-400)",
+        "--rdg-row-selected-background-color": "var(--chakra-colors-rsi-50)",
+        "--row-selected-hover-background-color": "var(--chakra-colors-rsi-100)",
+        "--rdg-error-cell-background-color": "var(--chakra-colors-red-50)",
+        "--rdg-warning-cell-background-color": "var(--chakra-colors-orange-50)",
+        "--rdg-info-cell-background-color": "var(--chakra-colors-blue-50)",
+        "--rdg-border-color": "var(--chakra-colors-border)",
+        "--rdg-frozen-cell-box-shadow": "none",
+        "--rdg-font-size": "var(--chakra-fontSizes-sm)",
+      },
+      ".rdg-header-row .rdg-cell": {
+        color: "textColor",
+        fontSize: "xs",
+        lineHeight: 10,
+        fontWeight: "bold",
+        letterSpacing: "wider",
+        textTransform: "uppercase",
+        "&:first-of-type": {
+          borderTopLeftRadius: "lg",
+        },
+        "&:last-child": {
+          borderTopRightRadius: "lg",
+        },
+      },
+      ".rdg-row:last-child .rdg-cell:first-of-type": {
+        borderBottomLeftRadius: "lg",
+      },
+      ".rdg-row:last-child .rdg-cell:last-child": {
+        borderBottomRightRadius: "lg",
+      },
+      ".rdg[dir='rtl']": {
+        ".rdg-row:last-child .rdg-cell:first-of-type": {
+          borderBottomRightRadius: "lg",
+          borderBottomLeftRadius: "none",
+        },
+        ".rdg-row:last-child .rdg-cell:last-child": {
+          borderBottomLeftRadius: "lg",
+          borderBottomRightRadius: "none",
+        },
+      },
+      ".rdg-cell": {
+        contain: "size layout style paint",
+        borderRight: "none",
+        borderInlineEnd: "none",
+        borderBottom: "1px solid var(--rdg-border-color)",
+        whiteSpace: "nowrap",
+        overflow: "hidden",
+        textOverflow: "ellipsis",
+        "&[aria-selected='true']": {
+          boxShadow: "inset 0 0 0 1px var(--rdg-selection-color)",
+        },
+        "&:first-of-type": {
+          boxShadow: "none",
+          borderInlineStart: "1px solid var(--rdg-border-color)",
+        },
+        "&:last-child": {
+          borderInlineEnd: "1px solid var(--rdg-border-color)",
+        },
+      },
+      ".rdg-cell-error": {
+        backgroundColor: "var(--rdg-error-cell-background-color)",
+      },
+      ".rdg-cell-warning": {
+        backgroundColor: "var(--rdg-warning-cell-background-color)",
+      },
+      ".rdg-cell-info": {
+        backgroundColor: "var(--rdg-info-cell-background-color)",
+      },
+      ".rdg-static": {
+        cursor: "pointer",
+      },
+      ".rdg-static .rdg-header-row": {
+        display: "none",
+      },
+      ".rdg-static .rdg-cell": {
+        "--rdg-selection-color": "none",
+      },
+      ".rdg-example .rdg-cell": {
+        "--rdg-selection-color": "none",
+        borderBottom: "none",
+      },
+
+      ".rdg-radio": {
+        display: "flex",
+        alignItems: "center",
+      },
+      ".rdg-checkbox": {
+        "--rdg-selection-color": "none",
+        display: "flex",
+        alignItems: "center",
+      },
+    },
+  },
+} as const
+
+export const rtlThemeSupport = {
+  components: {
+    Modal: {
+      baseStyle: {
+        dialog: {
+          direction: "rtl",
+        },
+      },
+    },
+  },
+} as const
+
+export type CustomTheme = DeepPartial<typeof themeOverrides>

inventory/src/lib/react-spreadsheet-import/src/translationsRSIProps.ts

@@ -0,0 +1,82 @@
+import type { DeepPartial } from "ts-essentials"
+
+export const translations = {
+  uploadStep: {
+    title: "Upload file",
+    manifestTitle: "Data that we expect:",
+    manifestDescription: "(You will have a chance to rename or remove columns in next steps)",
+    maxRecordsExceeded: (maxRecords: string) => `Too many records. Up to ${maxRecords} allowed`,
+    dropzone: {
+      title: "Upload .xlsx, .xls or .csv file",
+      errorToastDescription: "upload rejected",
+      activeDropzoneTitle: "Drop file here...",
+      buttonTitle: "Select file",
+      loadingTitle: "Processing...",
+    },
+    selectSheet: {
+      title: "Select the sheet to use",
+      nextButtonTitle: "Next",
+      backButtonTitle: "Back",
+    },
+  },
+  selectHeaderStep: {
+    title: "Select header row",
+    nextButtonTitle: "Next",
+    backButtonTitle: "Back",
+  },
+  matchColumnsStep: {
+    title: "Match Columns",
+    nextButtonTitle: "Next",
+    backButtonTitle: "Back",
+    userTableTitle: "Your table",
+    templateTitle: "Will become",
+    selectPlaceholder: "Select column...",
+    ignoredColumnText: "Column ignored",
+    subSelectPlaceholder: "Select...",
+    matchDropdownTitle: "Match",
+    unmatched: "Unmatched",
+    duplicateColumnWarningTitle: "Another column unselected",
+    duplicateColumnWarningDescription: "Columns cannot duplicate",
+  },
+  validationStep: {
+    title: "Validate data",
+    nextButtonTitle: "Confirm",
+    backButtonTitle: "Back",
+    noRowsMessage: "No data found",
+    noRowsMessageWhenFiltered: "No data containing errors",
+    discardButtonTitle: "Discard selected rows",
+    filterSwitchTitle: "Show only rows with errors",
+  },
+  alerts: {
+    confirmClose: {
+      headerTitle: "Exit import flow",
+      bodyText: "Are you sure? Your current information will not be saved.",
+      cancelButtonTitle: "Cancel",
+      exitButtonTitle: "Exit flow",
+    },
+    submitIncomplete: {
+      headerTitle: "Errors detected",
+      bodyText: "There are still some rows that contain errors. Rows with errors will be ignored when submitting.",
+      bodyTextSubmitForbidden: "There are still some rows containing errors.",
+      cancelButtonTitle: "Cancel",
+      finishButtonTitle: "Submit",
+    },
+    submitError: {
+      title: "Error",
+      defaultMessage: "An error occurred while submitting data",
+    },
+    unmatchedRequiredFields: {
+      headerTitle: "Not all columns matched",
+      bodyText: "There are required columns that are not matched or ignored. Do you want to continue?",
+      listTitle: "Columns not matched:",
+      cancelButtonTitle: "Cancel",
+      continueButtonTitle: "Continue",
+    },
+    toast: {
+      error: "Error",
+    },
+  },
+}
+
+export type TranslationsRSIProps = DeepPartial<typeof translations>
+export type Translations = typeof translations

inventory/src/lib/react-spreadsheet-import/src/types.ts

@@ -0,0 +1,160 @@
+import type { Meta } from "./steps/ValidationStep/types"
+import type { DeepReadonly } from "ts-essentials"
+import type { TranslationsRSIProps } from "./translationsRSIProps"
+import type { Columns } from "./steps/MatchColumnsStep/MatchColumnsStep"
+import type { StepState } from "./steps/UploadFlow"
+
+export type RsiProps<T extends string> = {
+  // Is modal visible.
+  isOpen: boolean
+  // callback when RSI is closed before final submit
+  onClose: () => void
+  // Field description for requested data
+  fields: Fields<T>
+  // Runs after file upload step, receives and returns raw sheet data
+  uploadStepHook?: (data: RawData[]) => Promise<RawData[]>
+  // Runs after header selection step, receives and returns raw sheet data
+  selectHeaderStepHook?: (headerValues: RawData, data: RawData[]) => Promise<{ headerValues: RawData; data: RawData[] }>
+  // Runs once before validation step, used for data mutations and if you want to change how columns were matched
+  matchColumnsStepHook?: (table: Data<T>[], rawData: RawData[], columns: Columns<T>) => Promise<Data<T>[]>
+  // Runs after column matching and on entry change
+  rowHook?: RowHook<T>
+  // Runs after column matching and on entry change
+  tableHook?: TableHook<T>
+  // Function called after user finishes the flow. You can return a promise that will be awaited.
+  onSubmit: (data: Result<T>, file: File) => void | Promise<any>
+  // Allows submitting with errors. Default: true
+  allowInvalidSubmit?: boolean
+  // Enable navigation in stepper component and show back button. Default: false
+  isNavigationEnabled?: boolean
+  // Translations for each text
+  translations?: TranslationsRSIProps
+  // Theme configuration passed to underlying Chakra-UI
+  customTheme?: object
+  // Specifies maximum number of rows for a single import
+  maxRecords?: number
+  // Maximum upload filesize (in bytes)
+  maxFileSize?: number
+  // Automatically map imported headers to specified fields if possible. Default: true
+  autoMapHeaders?: boolean
+  // When field type is "select", automatically match values if possible. Default: false
+  autoMapSelectValues?: boolean
+  // Headers matching accuracy: 1 for strict and up for more flexible matching
+  autoMapDistance?: number
+  // Initial Step state to be rendered on load
+  initialStepState?: StepState
+  // Sets SheetJS dateNF option. If date parsing is applied, date will be formatted e.g. "yyyy-mm-dd hh:mm:ss", "m/d/yy h:mm", 'mmm-yy', etc.
+  dateFormat?: string
+  // Sets SheetJS "raw" option. If true, parsing will only be applied to xlsx date fields.
+  parseRaw?: boolean
+  // Use for right-to-left (RTL) support
+  rtl?: boolean
+}
+
+export type RawData = Array<string | undefined>
+
+export type Data<T extends string> = { [key in T]: string | boolean | undefined }
+
+// Data model RSI uses for spreadsheet imports
+export type Fields<T extends string> = DeepReadonly<Field<T>[]>
+
+export type Field<T extends string> = {
+  // UI-facing field label
+  label: string
+  // Field's unique identifier
+  key: T
+  // UI-facing additional information displayed via tooltip and ? icon
+  description?: string
+  // Alternate labels used for fields' auto-matching, e.g. "fname" -> "firstName"
+  alternateMatches?: string[]
+  // Validations used for field entries
+  validations?: Validation[]
+  // Field entry component, default: Input
+  fieldType: Checkbox | Select | Input
+  // UI-facing values shown to user as field examples pre-upload phase
+  example?: string
+}
+
+export type Checkbox = {
+  type: "checkbox"
+  // Alternate values to be treated as booleans, e.g. {yes: true, no: false}
+  booleanMatches?: { [key: string]: boolean }
+}
+
+export type Select = {
+  type: "select"
+  // Options displayed in Select component
+  options: SelectOption[]
+}
+
+export type SelectOption = {
+  // UI-facing option label
+  label: string
+  // Field entry matching criteria as well as select output
+  value: string
+}
+
+export type Input = {
+  type: "input"
+}
+
+export type Validation = RequiredValidation | UniqueValidation | RegexValidation
+
+export type RequiredValidation = {
+  rule: "required"
+  errorMessage?: string
+  level?: ErrorLevel
+}
+
+export type UniqueValidation = {
+  rule: "unique"
+  allowEmpty?: boolean
+  errorMessage?: string
+  level?: ErrorLevel
+}
+
+export type RegexValidation = {
+  rule: "regex"
+  value: string
+  flags?: string
+  errorMessage: string
+  level?: ErrorLevel
+}
+
+export type RowHook<T extends string> = (
+  row: Data<T>,
+  addError: (fieldKey: T, error: Info) => void,
+  table: Data<T>[],
+) => Data<T> | Promise<Data<T>>
+export type TableHook<T extends string> = (
+  table: Data<T>[],
+  addError: (rowIndex: number, fieldKey: T, error: Info) => void,
+) => Data<T>[] | Promise<Data<T>[]>
+
+export type ErrorLevel = "info" | "warning" | "error"
+
+export type Info = {
+  message: string
+  level: ErrorLevel
+}
+
+export enum ErrorSources {
+  Table = "table",
+  Row = "row",
+}
+
+/*
+   Source determines whether the error is from the full table or row validation
+   Table validation is tableHook and "unique" validation
+   Row validation is rowHook and all other validations
+   it is used to determine if row.__errors should be updated or not depending on different validations
+*/
+export type InfoWithSource = Info & {
+  source: ErrorSources
+}
+
+export type Result<T extends string> = {
+  validData: Data<T>[]
+  invalidData: Data<T>[]
+  all: (Data<T> & Meta)[]
+}

inventory/src/lib/react-spreadsheet-import/src/utils/exceedsMaxRecords.ts

@@ -0,0 +1,6 @@
+import type XLSX from "xlsx-ugnis"
+
+export const exceedsMaxRecords = (workSheet: XLSX.WorkSheet, maxRecords: number) => {
+  const [top, bottom] = workSheet["!ref"]?.split(":").map((position) => parseInt(position.replace(/\D/g, ""), 10)) || []
+  return bottom - top > maxRecords
+}

inventory/src/lib/react-spreadsheet-import/src/utils/mapData.ts

@@ -0,0 +1,10 @@
+export const mapData = (data: string[][], valueMap: string[]) =>
+  data.map((row) =>
+    row.reduce<{ [k: string]: string }>((obj, value, index) => {
+      if (valueMap[index]) {
+        obj[valueMap[index]] = `${value}`
+        return obj
+      }
+      return obj
+    }, {}),
+  )

inventory/src/lib/react-spreadsheet-import/src/utils/mapWorkbook.ts

@@ -0,0 +1,15 @@
+import * as XLSX from "xlsx"
+import type { RawData } from "../types"
+
+export const mapWorkbook = (workbook: XLSX.WorkBook): RawData[] => {
+  const firstSheetName = workbook.SheetNames[0]
+  const worksheet = workbook.Sheets[firstSheetName]
+
+  const data = XLSX.utils.sheet_to_json<string[]>(worksheet, {
+    header: 1,
+    raw: false,
+    defval: "",
+  })
+
+  return data
+}

inventory/src/lib/react-spreadsheet-import/src/utils/steps.ts

@@ -0,0 +1,26 @@
+import { StepType } from "../steps/UploadFlow"
+
+export const steps = ["uploadStep", "selectHeaderStep", "matchColumnsStep", "validationStep"] as const
+const StepTypeToStepRecord: Record<StepType, (typeof steps)[number]> = {
+  [StepType.upload]: "uploadStep",
+  [StepType.selectSheet]: "uploadStep",
+  [StepType.selectHeader]: "selectHeaderStep",
+  [StepType.matchColumns]: "matchColumnsStep",
+  [StepType.validateData]: "validationStep",
+}
+const StepToStepTypeRecord: Record<(typeof steps)[number], StepType> = {
+  uploadStep: StepType.upload,
+  selectHeaderStep: StepType.selectHeader,
+  matchColumnsStep: StepType.matchColumns,
+  validationStep: StepType.validateData,
+}
+
+export const stepIndexToStepType = (stepIndex: number) => {
+  const step = steps[stepIndex]
+  return StepToStepTypeRecord[step] || StepType.upload
+}
+
+export const stepTypeToStepIndex = (type?: StepType) => {
+  const step = StepTypeToStepRecord[type || StepType.upload]
+  return Math.max(0, steps.indexOf(step))
+}

inventory/src/pages/import/Import.tsx

@@ -0,0 +1,249 @@
+import { useState } from "react";
+import { ReactSpreadsheetImport } from "@/lib/react-spreadsheet-import/src";
+import { Button } from "@/components/ui/button";
+import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
+import { Code } from "@/components/ui/code";
+import { toast } from "sonner";
+import { motion } from "framer-motion";
+
+const IMPORT_FIELDS = [
+  {
+    label: "Name",
+    key: "name",
+    alternateMatches: ["product", "product name", "item name", "title"],
+    fieldType: {
+      type: "input",
+    },
+    example: "Widget X",
+    description: "The name or title of the product",
+    width: 300,
+    validations: [
+      {
+        rule: "required",
+        errorMessage: "Name is required",
+        level: "error",
+      },
+    ],
+  },
+  {
+    label: "SKU",
+    key: "sku",
+    alternateMatches: ["item number", "product code", "product id", "item id"],
+    fieldType: {
+      type: "input",
+    },
+    example: "WX-123",
+    description: "Unique product identifier",
+    width: 120,
+    validations: [
+      {
+        rule: "required",
+        errorMessage: "SKU is required",
+        level: "error",
+      },
+      {
+        rule: "unique",
+        errorMessage: "SKU must be unique",
+        level: "error",
+      },
+    ],
+  },
+  {
+    label: "Category",
+    key: "category",
+    alternateMatches: ["product category", "type", "product type"],
+    fieldType: {
+      type: "select",
+      options: [
+        { label: "Electronics", value: "electronics" },
+        { label: "Clothing", value: "clothing" },
+        { label: "Food & Beverage", value: "food_beverage" },
+        { label: "Office Supplies", value: "office_supplies" },
+        { label: "Other", value: "other" },
+      ],
+    },
+    width: 150,
+    validations: [
+      {
+        rule: "required",
+        errorMessage: "Category is required",
+        level: "error",
+      },
+    ],
+    example: "Electronics",
+    description: "Product category",
+  },
+  {
+    label: "Quantity",
+    key: "quantity",
+    alternateMatches: ["qty", "stock", "amount", "inventory", "stock level"],
+    fieldType: {
+      type: "input",
+    },
+    example: "100",
+    description: "Current stock quantity",
+    width: 100,
+    validations: [
+      {
+        rule: "required",
+        errorMessage: "Quantity is required",
+        level: "error",
+      },
+      {
+        rule: "regex",
+        value: "^[0-9]+$",
+        errorMessage: "Quantity must be a positive number",
+        level: "error",
+      },
+    ],
+  },
+  {
+    label: "Price",
+    key: "price",
+    alternateMatches: ["unit price", "cost", "selling price", "retail price"],
+    fieldType: {
+      type: "input",
+    },
+    example: "29.99",
+    description: "Selling price per unit",
+    width: 100,
+    validations: [
+      {
+        rule: "required",
+        errorMessage: "Price is required",
+        level: "error",
+      },
+      {
+        rule: "regex",
+        value: "^\\d*\\.?\\d+$",
+        errorMessage: "Price must be a valid number",
+        level: "error",
+      },
+    ],
+  },
+  {
+    label: "In Stock",
+    key: "inStock",
+    alternateMatches: ["available", "active", "status"],
+    fieldType: {
+      type: "checkbox",
+      booleanMatches: {
+        yes: true,
+        no: false,
+        "in stock": true,
+        "out of stock": false,
+        available: true,
+        unavailable: false,
+      },
+    },
+    width: 80,
+    example: "Yes",
+    description: "Whether the item is currently in stock",
+  },
+  {
+    label: "Minimum Stock",
+    key: "minStock",
+    alternateMatches: ["min qty", "reorder point", "low stock level"],
+    fieldType: {
+      type: "input",
+    },
+    example: "10",
+    description: "Minimum stock level before reorder",
+    width: 100,
+    validations: [
+      {
+        rule: "regex",
+        value: "^[0-9]+$",
+        errorMessage: "Minimum stock must be a positive number",
+        level: "error",
+      },
+    ],
+  },
+  {
+    label: "Location",
+    key: "location",
+    alternateMatches: ["storage location", "warehouse", "shelf", "bin"],
+    fieldType: {
+      type: "select",
+      options: [
+        { label: "Warehouse A", value: "warehouse_a" },
+        { label: "Warehouse B", value: "warehouse_b" },
+        { label: "Store Front", value: "store_front" },
+        { label: "External Storage", value: "external" },
+      ],
+    },
+    width: 150,
+    example: "Warehouse A",
+    description: "Storage location of the product",
+  },
+];
+
+export function Import() {
+  const [isOpen, setIsOpen] = useState(false);
+  const [importedData, setImportedData] = useState<any[] | null>(null);
+
+  const handleData = async (data: any, file: File) => {
+    try {
+      console.log("Imported Data:", data);
+      console.log("File:", file);
+      setImportedData(data);
+      setIsOpen(false);
+      toast.success("Data imported successfully");
+    } catch (error) {
+      toast.error("Failed to import data");
+      console.error("Import error:", error);
+    }
+  };
+
+  return (
+    <motion.div 
+      layout
+      transition={{ 
+        layout: { 
+          duration: 0.15,
+          ease: [0.4, 0, 0.2, 1]
+        }
+      }}
+      className="container mx-auto py-6 space-y-4"
+    >
+      <motion.div 
+        layout="position"
+        transition={{ duration: 0.15 }}
+        className="flex items-center justify-between"
+      >
+        <h1 className="text-3xl font-bold tracking-tight">Add New Products</h1>
+      </motion.div>
+
+      <Card className="max-w-[400px]">
+        <CardHeader>
+          <CardTitle>Import Data</CardTitle>
+        </CardHeader>
+        <CardContent>
+          <Button onClick={() => setIsOpen(true)} className="w-full">
+            Upload Spreadsheet
+          </Button>
+        </CardContent>
+      </Card>
+
+      {importedData && (
+        <Card>
+          <CardHeader>
+            <CardTitle>Preview Imported Data</CardTitle>
+          </CardHeader>
+          <CardContent>
+            <Code className="p-4 w-full rounded-md border">
+              {JSON.stringify(importedData, null, 2)}
+            </Code>
+          </CardContent>
+        </Card>
+      )}
+
+      <ReactSpreadsheetImport
+        isOpen={isOpen}
+        onClose={() => setIsOpen(false)}
+        onSubmit={handleData}
+        fields={IMPORT_FIELDS}
+      />
+    </motion.div>
+  );
+} 

mountremote.command

@@ -1,7 +1,7 @@
 #!/bin/zsh
 
 #Clear previous mount in case it’s still there
-umount ~/Dev/inventory/inventory-server
+umount '/Users/matt/Library/Mobile Documents/com~apple~CloudDocs/Dev/inventory/inventory-server'
 
 #Mount
-sshfs matt@dashboard.kent.pw:/var/www/html/inventory -p 22122 ~/Dev/inventory/inventory-server/
+sshfs matt@dashboard.kent.pw:/var/www/html/inventory -p 22122 '/Users/matt/Library/Mobile Documents/com~apple~CloudDocs/Dev/inventory/inventory-server/'