Update calculate script to account for import changes

docs/metrics-changes.md

@@ -0,0 +1,181 @@
+# Metrics System Changes
+
+## Schema Changes
+
+### Product Identifiers
+- Changed `product_id` to `pid` throughout all metrics tables and queries
+- Changed `category_id` to `cat_id` in category-related queries
+
+### Purchase Orders
+- Changed status check from `status = 'closed'` to `receiving_status >= 30`
+- Added comment `-- Partial or fully received` for clarity
+- Now using `received_date` instead of relying on status changes
+
+### New Product Fields
+- Added support for `notions_inv_count`
+- Added support for `date_last_sold`
+- Added support for `total_sold`
+- Using `visible` flag for active product counts
+
+### Field Size Updates
+- Increased size of financial fields to handle larger numbers:
+  - Changed category metrics `total_value` from `DECIMAL(10,3)` to `DECIMAL(15,3)`
+  - Changed brand metrics financial fields from `DECIMAL(10,2)` to `DECIMAL(15,2)`
+  - Affects `total_stock_cost`, `total_stock_retail`, `total_revenue`
+
+## Metrics File Changes
+
+### Product Metrics (`product-metrics.js`)
+- Updated SQL queries to use new field names
+- Enhanced stock status calculations
+- Added financial metrics:
+  - `gross_profit`
+  - `gmroi`
+  - `avg_margin_percent`
+  - `inventory_value`
+- Improved reorder quantity calculations with:
+  - Enhanced safety stock calculation
+  - Lead time consideration
+  - Service level factors
+- Added NaN/NULL value handling:
+  - Added `sanitizeValue` helper function
+  - Properly converts JavaScript NaN to SQL NULL
+  - Ensures all numeric fields have valid values
+
+### Vendor Metrics (`vendor-metrics.js`)
+- Updated field references to use `pid`
+- Modified purchase order status checks
+- Enhanced vendor performance metrics:
+  - Order fill rate calculation
+  - On-time delivery rate
+  - Lead time tracking
+
+### Category Metrics (`category-metrics.js`)
+- Updated to use `cat_id` instead of `category_id`
+- Enhanced category performance tracking:
+  - Active vs total products
+  - Category growth rate
+  - Turnover rate
+- Added time-based metrics for:
+  - Product counts
+  - Revenue tracking
+  - Margin analysis
+- Added NULL brand handling:
+  - Uses 'Unbranded' for NULL brand values
+  - Maintains data integrity in category sales metrics
+
+### Brand Metrics (`brand-metrics.js`)
+- Updated product references to use `pid`
+- Enhanced brand performance metrics:
+  - Stock value calculations
+  - Revenue tracking
+  - Growth rate analysis
+- Added time-based aggregates for:
+  - Stock levels
+  - Sales performance
+  - Margin analysis
+- Increased field sizes to handle large retailers
+
+### Sales Forecasts (`sales-forecasts.js`)
+- Updated to use new product identifiers
+- Enhanced forecast calculations:
+  - Day-of-week patterns
+  - Seasonality factors
+  - Confidence levels
+- Added category-level forecasts with:
+  - Units and revenue predictions
+  - Confidence scoring
+  - Seasonal adjustments
+
+### Time Aggregates (`time-aggregates.js`)
+- Updated field references to use `pid`
+- Enhanced financial metrics:
+  - GMROI calculations
+  - Profit margin tracking
+- Added inventory value tracking
+- Improved purchase order integration
+
+## Database Impact
+
+### Tables Modified
+- `product_metrics`
+- `vendor_metrics`
+- `vendor_time_metrics`
+- `category_metrics`
+- `category_time_metrics`
+- `brand_metrics`
+- `brand_time_metrics`
+- `sales_forecasts`
+- `category_forecasts`
+- `product_time_aggregates`
+
+### New Fields Added
+Several tables have new fields for:
+- Enhanced financial tracking
+- Improved inventory metrics
+- Better performance monitoring
+- More accurate forecasting
+
+## Frontend Considerations
+
+### Data Access Changes
+- All product lookups need to use `pid` instead of `product_id`
+- Category references should use `cat_id`
+- Purchase order status checks need updating
+- Handle 'Unbranded' as a valid brand value
+
+### New Features Available
+- Enhanced stock status indicators
+- More detailed financial metrics
+- Improved forecasting data
+- Better category and brand performance tracking
+
+### UI Updates Needed
+- Update all API calls to use new field names
+- Modify data displays for new metrics
+- Add new financial performance indicators
+- Update stock status logic
+- Enhance forecast displays
+
+## API Route Updates Needed
+
+### Product Routes
+- Update ID field references
+- Modify stock status calculations
+- Add new financial metrics endpoints
+
+### Category Routes
+- Update to use `cat_id`
+- Add new performance metrics
+- Include time-based aggregates
+
+### Vendor Routes
+- Update product reference handling
+- Add enhanced performance metrics
+- Include new time-based data
+
+### Reporting Routes
+- Update all ID references
+- Add new metrics support
+- Include enhanced forecasting data
+
+## Migration Considerations
+
+### Data Migration
+- Update existing records to use new IDs
+- Backfill new metrics where possible
+- Verify data integrity after changes
+- Handle NULL to 'Unbranded' brand conversion
+
+### Code Updates
+- Update all API endpoints
+- Modify database queries
+- Update frontend components
+- Revise reporting logic
+
+### Testing Requirements
+- Verify ID changes throughout system
+- Test new metrics calculations
+- Validate forecasting accuracy
+- Check performance impact
+- Verify NULL value handling