inventory/docs/routes-cleanup.md

Analysis of Potential Issues

1. Obsolete Functionality:

   • config.js Legacy Endpoints: The endpoints GET /config/, PUT /config/stock-thresholds/:id, PUT /config/lead-time-thresholds/:id, PUT /config/sales-velocity/:id, PUT /config/abc-classification/:id, PUT /config/safety-stock/:id, and PUT /config/turnover/:id appear highly likely to be obsolete. They reference older, single-row config tables (stock_thresholds, etc.) while newer endpoints (/config/global, /config/products, /config/vendors) manage settings in more structured tables (settings_global, settings_product, settings_vendor). Unless specifically required for backward compatibility, these legacy endpoints should be removed to avoid confusion and potential data conflicts.
   • analytics.js Forecast Endpoint (GET /analytics/forecast): This endpoint uses MySQL syntax (DATEDIFF, DATE_FORMAT, JSON_OBJECT, ? placeholders) but seems intended to run within the analytics module which otherwise uses PostgreSQL (req.app.locals.pool, date_trunc, ::text, $1 placeholders). This endpoint is likely obsolete or misplaced and will not function correctly against the PostgreSQL database.
   • csv.js Redundant Actions:
     • POST /csv/update seems redundant with POST /csv/full-update. The latter uses the runScript helper and dedicated state (activeFullUpdate), appearing more robust. /csv/update might be older or incomplete.
     • POST /csv/reset seems redundant with POST /csv/full-reset. Similar reasoning applies; /csv/full-reset appears preferred.
   • products.js Import Endpoint (POST /products/import): This is dangerous duplication. The /csv module handles imports (/csv/import, /csv/import-from-prod) with locking (activeImport) to prevent concurrent operations. This endpoint lacks such locking and could corrupt data if run simultaneously with other CSV/reset operations. It should likely be removed.
   • products.js Metrics Endpoint (GET /products/:id/metrics): This is redundant. The /metrics/:pid endpoint provides the same, possibly more comprehensive, data directly from the product_metrics table. Clients should use /metrics/:pid instead.

2. Overlap or Inappropriate Duplication of Effort:

   • AI Prompt Getters: GET /ai-prompts/type/general and GET /ai-prompts/type/system could potentially be handled by adding a query parameter filter to GET /ai-prompts/ (e.g., GET /ai-prompts?prompt_type=general). However, dedicated endpoints for single, specific items can sometimes be simpler. This is more of a design choice than a major issue.
   • Vendor Performance/Metrics: There are multiple ways to get vendor performance data:
     • GET /analytics/vendors (uses vendor_metrics)
     • GET /dashboard/vendor/performance (uses purchase_orders)
     • GET /purchase-orders/vendor-metrics (uses purchase_orders)
     • GET /vendors-aggregate/ (uses vendor_metrics, augmented with purchase_orders) This suggests significant overlap. The /vendors-aggregate endpoint seems the most comprehensive, combining pre-aggregated data with some real-time info. The others, especially /dashboard/vendor/performance and /purchase-orders/vendor-metrics which calculate directly from purchase_orders, might be redundant or less performant.
   • Product Listing:
     • GET /products/ lists products joining products, product_metrics, and categories.
     • GET /metrics/ lists products primarily from product_metrics. They offer similar filtering/sorting. If product_metrics contains all necessary display fields, GET /products/ might be partly redundant for simple listing views, although it does provide aggregated category names. Evaluate if both full list endpoints are necessary.
   • Image Uploads/Management: Image handling is split:
     • products-import.js: Uploads temporary images for product import to /uploads/products/, schedules deletion.
     • reusable-images.js: Uploads persistent images to /uploads/reusable/, stores metadata in DB.
     • products-import.js has /check-file and /list-uploads that can see both directories, while reusable-images.js has a /check-file that only sees its own. This separation could be confusing. Clarify the purpose and lifecycle of images in each directory.
   • Background Task Management (csv.js): The use of activeImport for multiple unrelated tasks (import, reset, metrics calc) prevents concurrency, which might be too restrictive. The cancellation logic (/cancel) only targets full-update/full-reset, not tasks locked by activeImport. This needs unification.
   • Analytics/Dashboard Base Table Queries: Several endpoints in analytics.js (/pricing, /categories) and dashboard.js (/best-sellers, /sales/metrics, /trending/products, /key-metrics, /inventory-health, /sales-overview) query base tables (orders, products, purchase_orders) directly, while many others leverage pre-aggregated _metrics tables. This inconsistency can lead to performance differences and suggests potential for optimization by using aggregates where possible.

3. Obvious Mistakes / Data Issues:

   • AI Prompt Fetching: GET /ai-prompts/company/:companyId, /type/general, /type/system return result.rows[0]. This assumes uniqueness. If the underlying DB constraints (unique_company_prompt, etc.) fail or aren't present, this could silently hide data if multiple rows match. The use of unique constraint handling in POST/PUT suggests this is likely intended and safe if DB constraints are solid.
   • Mixed Databases & SSH Tunnels: The heavy reliance in ai_validation.js and products-import.js on connecting to a production MySQL DB via SSH tunnel while also using a local PostgreSQL DB adds significant architectural complexity.
     • Inefficiency: In ai_validation.js (generateDebugResponse), an SSH tunnel and MySQL connection (promptTunnel, promptConnection) are established but seem unused when fetching prompts (which correctly come from the PG pool res.app.locals.pool). This is wasted effort.
     • Improvement: The getDbConnection function in products-import.js implements caching/pooling for the SSH/MySQL connection – this is much better and should ideally be used consistently wherever the production DB is accessed (e.g., in ai_validation.js).
   • products.js Brand Filtering: GET /products/brands filters brands based on having associated purchase orders with a cost >= 500. This seems arbitrary for a general list of brands and might return incomplete results depending on the use case.
   • Type Handling: Ensure parseValue handles all required types and edge cases correctly, especially for filtering complex queries in *-aggregate and metrics routes. Explicit type casting in SQL (::numeric, ::text, etc.) is generally good practice in PostgreSQL.
   • Dummy Data: Several dashboard.js endpoints return hardcoded dummy data on errors or when no data is found. While this prevents UI crashes, it can mask real issues. Ensure logging is robust when fallbacks are used.

Summary of Endpoints

Here's a summary of the available endpoints, grouped by their likely file/module:

1. AI Prompts (ai_prompts.js) * GET /: Get all AI prompts. * GET /:id: Get a specific AI prompt by its ID. * GET /company/:companyId: Get the AI prompt for a specific company (expects one). (Deprecated) * GET /type/general: Get the general AI prompt (expects one). (Deprecated) * GET /type/system: Get the system AI prompt (expects one). (Deprecated) * GET /by-type: Get AI prompt by type (general, system, company_specific) with optional company parameter. (New Consolidated Endpoint) * POST /: Create a new AI prompt. * PUT /:id: Update an existing AI prompt. * DELETE /:id: Delete an AI prompt.

2. AI Validation (ai_validation.js) * POST /debug: Generate and view the structure of prompts and taxonomy data (for debugging, doesn't call OpenAI). Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance). * POST /validate: Validate product data using OpenAI. Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance). * GET /test-taxonomy: Test endpoint to query sample taxonomy data from Prod MySQL.

3. Analytics (analytics.js) * GET /stats: Get overall business statistics from metrics tables. * GET /profit: Get profit analysis data (by category, over time, top products) from metrics tables. * GET /vendors: Get vendor performance analysis from vendor_metrics. * GET /stock: Get stock analysis data (turnover, levels, critical items) from metrics tables. * GET /pricing: Get pricing analysis (price points, elasticity, recommendations) - uses orders table. * GET /categories: Get category performance analysis (revenue, profit, growth, distribution, trends) - uses orders and products tables. * GET /forecast: (Likely Obsolete/Broken) Attempts to get forecast data using MySQL syntax.

4. Brands Aggregate (brands-aggregate.js) * GET /filter-options: Get distinct brand names and statuses for UI filters (from brand_metrics). * GET /stats: Get overall statistics related to brands (from brand_metrics). * GET /: List brands with aggregated metrics, supporting filtering, sorting, pagination (from brand_metrics).

5. Categories Aggregate (categories-aggregate.js) * GET /filter-options: Get distinct category types, statuses, and counts for UI filters (from category_metrics & categories). * GET /stats: Get overall statistics related to categories (from category_metrics & categories). * GET /: List categories with aggregated metrics, supporting filtering, sorting (incl. hierarchy), pagination (from category_metrics & categories).

6. Configuration (config.js) * (New) GET /global: Get all global settings. * (New) PUT /global: Update global settings. * (New) GET /products: List product-specific settings with pagination/search. * (New) PUT /products/:pid: Update/Create product-specific settings. * (New) POST /products/:pid/reset: Reset product settings to defaults. * (New) GET /vendors: List vendor-specific settings with pagination/search. * (New) PUT /vendors/:vendor: Update/Create vendor-specific settings. * (New) POST /vendors/:vendor/reset: Reset vendor settings to defaults. * (Legacy/Obsolete) GET /: Get all config from old single-row tables. * (Legacy/Obsolete) PUT /stock-thresholds/:id: Update old stock thresholds. * (Legacy/Obsolete) PUT /lead-time-thresholds/:id: Update old lead time thresholds. * (Legacy/Obsolete) PUT /sales-velocity/:id: Update old sales velocity config. * (Legacy/Obsolete) PUT /abc-classification/:id: Update old ABC config. * (Legacy/Obsolete) PUT /safety-stock/:id: Update old safety stock config. * (Legacy/Obsolete) PUT /turnover/:id: Update old turnover config.

7. CSV Operations & Background Tasks (csv.js) * GET /:type/progress: SSE endpoint for full update/reset progress. * GET /test: Simple test endpoint. * GET /status: Check status of the generic background task lock (activeImport). * GET /calculate-metrics/status: Check status of metrics calculation. * GET /history/import: Get recent import history. * GET /history/calculate: Get recent metrics calculation history. * GET /status/modules: Get last calculation time per module. * GET /status/tables: Get last sync time per table. * GET /status/table-counts: Get record counts for key tables. * POST /update: (Potentially Obsolete) Trigger update-csv.js script. * POST /import: Trigger import-csv.js script. * POST /cancel: Cancel /full-update or /full-reset task. * POST /reset: (Potentially Obsolete) Trigger reset-db.js script. * POST /reset-metrics: Trigger reset-metrics.js script. * POST /calculate-metrics: Trigger calculate-metrics.js script. * POST /import-from-prod: Trigger import-from-prod.js script. * POST /full-update: Trigger full-update.js script (preferred update). * POST /full-reset: Trigger full-reset.js script (preferred reset).

8. Dashboard (dashboard.js) * GET /stock/metrics: Get dashboard stock summary metrics & brand breakdown. * GET /purchase/metrics: Get dashboard purchase order summary metrics & vendor breakdown. * GET /replenishment/metrics: Get dashboard replenishment summary & top variants. * GET /forecast/metrics: Get dashboard forecast summary, daily, and category breakdown. * GET /overstock/metrics: Get dashboard overstock summary & category breakdown. * GET /overstock/products: Get list of top overstocked products. * GET /best-sellers: Get dashboard best-selling products, brands, categories - uses orders, products. * GET /sales/metrics: Get dashboard sales summary for a period - uses orders. * GET /low-stock/products: Get list of top low stock/critical products. * GET /trending/products: Get list of trending products - uses orders, products. * GET /vendor/performance: Get dashboard vendor performance details - uses purchase_orders. * GET /key-metrics: Get dashboard summary KPIs - uses multiple base tables. * GET /inventory-health: Get dashboard inventory health overview - uses products, product_metrics. * GET /replenish/products: Get list of products needing replenishment (overlaps /low-stock/products). * GET /sales-overview: Get daily sales totals for chart - uses orders.

9. Product Import Utilities (products-import.js) * POST /upload-image: Upload temporary product image, schedule deletion. * DELETE /delete-image: Delete temporary product image. * GET /field-options: Get dropdown options for product fields from Prod MySQL (cached). * GET /product-lines/:companyId: Get product lines for a company from Prod MySQL (cached). * GET /sublines/:lineId: Get sublines for a line from Prod MySQL (cached). * GET /check-file/:filename: Check existence/permissions of uploaded file (temp or reusable). * GET /list-uploads: List files in upload directories. * GET /search-products: Search products in Prod MySQL DB. * GET /check-upc-and-generate-sku: Check UPC existence and generate SKU suggestion based on Prod MySQL data. * GET /product-categories/:pid: Get assigned categories for a product from Prod MySQL.

10. Product Metrics (product-metrics.js) * GET /filter-options: Get distinct filter values (vendor, brand, abcClass) from product_metrics. * GET /: List detailed product metrics with filtering, sorting, pagination (primary data access). * GET /:pid: Get full metrics record for a single product.

11. Orders (orders.js) * GET /: List orders with summary info, filtering, sorting, pagination, and stats. * GET /:orderNumber: Get details for a single order, including items.

12. Products (products.js) * GET /brands: Get distinct brands (filtered by PO value). * GET /: List products with core data + metrics, filtering, sorting, pagination. * GET /trending: Get trending products based on product_metrics. * GET /:id: Get details for a single product (core data + metrics). * POST /import: (Likely Obsolete/Dangerous) Import products from CSV. * PUT /:id: Update core product data. * GET /:id/metrics: (Redundant) Get metrics for a single product. * GET /:id/time-series: Get sales/PO history for a single product.

13. Purchase Orders (purchase-orders.js) * GET /: List purchase orders with summary info, filtering, sorting, pagination, and summary stats. * GET /vendor-metrics: Calculate vendor performance metrics from purchase_orders. * GET /cost-analysis: Calculate cost analysis by category from purchase_orders. * GET /receiving-status: Get summary counts based on PO receiving status. * GET /order-vs-received: List product ordered vs. received quantities.

14. Reusable Images (reusable-images.js) * GET /: List all reusable images. * GET /by-company/:companyId: List global and company-specific images. * GET /global: List only global images. * GET /:id: Get a single reusable image record. * POST /upload: Upload a new reusable image and create DB record. * PUT /:id: Update reusable image metadata (name, global, company). * DELETE /:id: Delete reusable image record and file. * GET /check-file/:filename: Check existence/permissions of a reusable image file.

15. Templates (templates.js) * GET /: List all product data templates. * GET /:company/:productType: Get a specific template. * POST /: Create a new template. * PUT /:id: Update an existing template. * DELETE /:id: Delete a template.

16. Vendors Aggregate (vendors-aggregate.js) * GET /filter-options: Get distinct vendor names and statuses for UI filters (from vendor_metrics). * GET /stats: Get overall statistics related to vendors (from vendor_metrics & purchase_orders). * GET /: List vendors with aggregated metrics, supporting filtering, sorting, pagination (from vendor_metrics & purchase_orders).

Recommendations:

1. Address Obsolete Endpoints: Prioritize removing or confirming the necessity of the endpoints marked as obsolete/redundant (legacy config, /analytics/forecast, /csv/update, /csv/reset, /products/import, /products/:id/metrics).
2. Consolidate Overlapping Functionality: Review the multiple vendor performance and product listing endpoints. Decide on the primary method (e.g., using aggregate tables via /vendors-aggregate and /metrics) and refactor or remove the others. Clarify the image upload strategies.
3. Standardize Data Access: Decide whether dashboard and analytics endpoints should primarily use aggregate tables (like /metrics, /brands-aggregate, etc.) or if direct access to base tables is sometimes necessary. Aim for consistency and document the reasoning. Optimize queries hitting base tables if they must remain.
4. Improve Background Task Management: Refactor csv.js to use a unified locking mechanism (maybe separate locks per task type?) and a consistent cancellation strategy for all spawned/managed processes. Clarify the purpose of update vs full-update and reset vs full-reset.
5. Optimize DB Connections: Ensure the getDbConnection pooling/caching helper from products-import.js is used consistently across all modules interacting with the production MySQL database (especially ai_validation.js). Remove unnecessary tunnel creations.
6. Review Data Integrity: Double-check the assumptions made (e.g., uniqueness of AI prompts) and ensure database constraints enforce them. Review the GET /products/brands filtering logic.

Changes Made

1. Removed Obsolete Legacy Endpoints in config.js:

   • Removed GET /config/ endpoint
   • Removed PUT /config/stock-thresholds/:id endpoint
   • Removed PUT /config/lead-time-thresholds/:id endpoint
   • Removed PUT /config/sales-velocity/:id endpoint
   • Removed PUT /config/abc-classification/:id endpoint
   • Removed PUT /config/safety-stock/:id endpoint
   • Removed PUT /config/turnover/:id endpoint

   These endpoints were obsolete as they referenced older, single-row config tables that have been replaced by newer endpoints using the structured tables settings_global, settings_product, and settings_vendor.

2. Removed MySQL Syntax /forecast Endpoint in analytics.js:

   • Removed GET /analytics/forecast endpoint that was using MySQL-specific syntax incompatible with the PostgreSQL database used elsewhere in the application.

3. Renamed and Removed Redundant Endpoints:

   • Renamed csv.js to data-management.js while maintaining the same /csv/* endpoint paths for consistency
   • Removed deprecated /csv/update endpoint (now fully replaced by /csv/full-update)
   • Removed deprecated /csv/reset endpoint (now fully replaced by /csv/full-reset)
   • Removed deprecated /products/import endpoint (now handled by /csv/import)
   • Removed deprecated /products/:id/metrics endpoint (now handled by /metrics/:pid)

4. Fixed Data Integrity Issues:

   • Improved GET /products/brands endpoint by removing the arbitrary filtering logic that was only showing brands with purchase orders that had a total cost of at least $500
   • The updated endpoint now returns all distinct brands from visible products, providing more complete data

5. Optimized Database Connections:

   • Created a new dbConnection.js utility file that encapsulates the optimized database connection management logic
   • Improved the ai-validation.js file to use this shared connection management, eliminating unnecessary repeated tunnel creation
   • Added proper connection pooling with timeout-based connection reuse, reducing the overhead of repeatedly creating SSH tunnels
   • Added query result caching for frequently accessed data to improve performance

These changes improve maintainability by removing duplicate code, enhance consistency by standardizing on the newer endpoint patterns, and optimize performance by reducing redundant database connections.

Additional Improvements

1. Further Database Connection Optimizations:

   • Extended the use of the optimized database connection utility to additional endpoints in ai-validation.js
   • Updated the /validate endpoint and /test-taxonomy endpoint to use getDbConnection
   • Ensured consistent connection management across all routes that access the production database

2. AI Prompts Data Integrity Verification:

   • Confirmed proper uniqueness constraints are in place in the database schema for AI prompts
   • The schema includes:
     • unique_company_prompt constraint ensuring only one prompt per company
     • idx_unique_general_prompt index ensuring only one general prompt in the system
     • idx_unique_system_prompt index ensuring only one system prompt in the system
   • Endpoint handlers properly handle uniqueness constraint violations with appropriate 409 Conflict responses
   • Validation ensures company-specific prompts have company IDs, while general/system prompts do not

3. AI Prompts Endpoint Consolidation:

   • Added a new consolidated /by-type endpoint that handles all types of prompts (general, system, company_specific)
   • Marked the existing separate endpoints as deprecated with console warnings
   • Maintained backward compatibility while providing a cleaner API moving forward

Completed Items

✅ Removed obsolete legacy endpoints in config.js
✅ Removed MySQL syntax /forecast endpoint in analytics.js
✅ Fixed GET /products/brands endpoint filtering logic
✅ Created reusable database connection utility (dbConnection.js)
✅ Optimized database connections in ai-validation.js
✅ Verified data integrity in AI prompts handling
✅ Consolidated AI prompts endpoints with a unified /by-type endpoint

Remaining Items

• Consider adding additional error handling and logging for database connections
• Perform load testing on the optimized database connections to ensure they handle high traffic properly