**Analysis of Potential Issues**

1.  **Obsolete Functionality:**
    *   **`config.js` Legacy Endpoints:** The endpoints `GET /config/`, `PUT /config/stock-thresholds/:id`, `PUT /config/lead-time-thresholds/:id`, `PUT /config/sales-velocity/:id`, `PUT /config/abc-classification/:id`, `PUT /config/safety-stock/:id`, and `PUT /config/turnover/:id` appear **highly likely to be obsolete**. They reference older, single-row config tables (`stock_thresholds`, etc.) while newer endpoints (`/config/global`, `/config/products`, `/config/vendors`) manage settings in more structured tables (`settings_global`, `settings_product`, `settings_vendor`). Unless specifically required for backward compatibility, these legacy endpoints should be removed to avoid confusion and potential data conflicts.
    *   **`analytics.js` Forecast Endpoint (`GET /analytics/forecast`):** This endpoint uses **MySQL syntax** (`DATEDIFF`, `DATE_FORMAT`, `JSON_OBJECT`, `?` placeholders) but seems intended to run within the analytics module which otherwise uses PostgreSQL (`req.app.locals.pool`, `date_trunc`, `::text`, `$1` placeholders). This endpoint is likely **obsolete or misplaced** and will not function correctly against the PostgreSQL database.
    *   **`csv.js` Redundant Actions:**
        *   `POST /csv/update` seems redundant with `POST /csv/full-update`. The latter uses the `runScript` helper and dedicated state (`activeFullUpdate`), appearing more robust. `/csv/update` might be older or incomplete.
        *   `POST /csv/reset` seems redundant with `POST /csv/full-reset`. Similar reasoning applies; `/csv/full-reset` appears preferred.
    *   **`products.js` Import Endpoint (`POST /products/import`):** This is **dangerous duplication**. The `/csv` module handles imports (`/csv/import`, `/csv/import-from-prod`) with locking (`activeImport`) to prevent concurrent operations. This endpoint lacks such locking and could corrupt data if run simultaneously with other CSV/reset operations. It should likely be removed.
    *   **`products.js` Metrics Endpoint (`GET /products/:id/metrics`):** This is redundant. The `/metrics/:pid` endpoint provides the same, possibly more comprehensive, data directly from the `product_metrics` table. Clients should use `/metrics/:pid` instead.

2.  **Overlap or Inappropriate Duplication of Effort:**
    *   **AI Prompt Getters:** `GET /ai-prompts/type/general` and `GET /ai-prompts/type/system` could potentially be handled by adding a query parameter filter to `GET /ai-prompts/` (e.g., `GET /ai-prompts?prompt_type=general`). However, dedicated endpoints for single, specific items can sometimes be simpler. This is more of a design choice than a major issue.
    *   **Vendor Performance/Metrics:** There are multiple ways to get vendor performance data:
        *   `GET /analytics/vendors` (uses `vendor_metrics`)
        *   `GET /dashboard/vendor/performance` (uses `purchase_orders`)
        *   `GET /purchase-orders/vendor-metrics` (uses `purchase_orders`)
        *   `GET /vendors-aggregate/` (uses `vendor_metrics`, augmented with `purchase_orders`)
        This suggests significant overlap. The `/vendors-aggregate` endpoint seems the most comprehensive, combining pre-aggregated data with some real-time info. The others, especially `/dashboard/vendor/performance` and `/purchase-orders/vendor-metrics` which calculate directly from `purchase_orders`, might be redundant or less performant.
    *   **Product Listing:**
        *   `GET /products/` lists products joining `products`, `product_metrics`, and `categories`.
        *   `GET /metrics/` lists products primarily from `product_metrics`.
        They offer similar filtering/sorting. If `product_metrics` contains all necessary display fields, `GET /products/` might be partly redundant for simple listing views, although it does provide aggregated category names. Evaluate if both full list endpoints are necessary.
    *   **Image Uploads/Management:** Image handling is split:
        *   `products-import.js`: Uploads temporary images for product import to `/uploads/products/`, schedules deletion.
        *   `reusable-images.js`: Uploads persistent images to `/uploads/reusable/`, stores metadata in DB.
        *   `products-import.js` has `/check-file` and `/list-uploads` that can see *both* directories, while `reusable-images.js` has a `/check-file` that only sees its own. This separation could be confusing. Clarify the purpose and lifecycle of images in each directory.
    *   **Background Task Management (`csv.js`):** The use of `activeImport` for multiple unrelated tasks (import, reset, metrics calc) prevents concurrency, which might be too restrictive. The cancellation logic (`/cancel`) only targets `full-update`/`full-reset`, not tasks locked by `activeImport`. This needs unification.
    *   **Analytics/Dashboard Base Table Queries:** Several endpoints in `analytics.js` (`/pricing`, `/categories`) and `dashboard.js` (`/best-sellers`, `/sales/metrics`, `/trending/products`, `/key-metrics`, `/inventory-health`, `/sales-overview`) query base tables (`orders`, `products`, `purchase_orders`) directly, while many others leverage pre-aggregated `_metrics` tables. This inconsistency can lead to performance differences and suggests potential for optimization by using aggregates where possible.

3.  **Obvious Mistakes / Data Issues:**
    *   **AI Prompt Fetching:** `GET /ai-prompts/company/:companyId`, `/type/general`, `/type/system` return `result.rows[0]`. This assumes uniqueness. If the underlying DB constraints (`unique_company_prompt`, etc.) fail or aren't present, this could silently hide data if multiple rows match. The use of unique constraint handling in POST/PUT suggests this is likely intended and safe *if* DB constraints are solid.
    *   **Mixed Databases & SSH Tunnels:** The heavy reliance in `ai_validation.js` and `products-import.js` on connecting to a production MySQL DB via SSH tunnel while also using a local PostgreSQL DB adds significant architectural complexity.
        *   **Inefficiency:** In `ai_validation.js` (`generateDebugResponse`), an SSH tunnel and MySQL connection (`promptTunnel`, `promptConnection`) are established but seem unused when fetching prompts (which correctly come from the PG pool `res.app.locals.pool`). This is wasted effort.
        *   **Improvement:** The `getDbConnection` function in `products-import.js` implements caching/pooling for the SSH/MySQL connection – this is much better and should ideally be used consistently wherever the production DB is accessed (e.g., in `ai_validation.js`).
    *   **`products.js` Brand Filtering:** `GET /products/brands` filters brands based on having associated purchase orders with a cost >= 500. This seems arbitrary for a general list of brands and might return incomplete results depending on the use case.
    *   **Type Handling:** Ensure `parseValue` handles all required types and edge cases correctly, especially for filtering complex queries in `*-aggregate` and `metrics` routes. Explicit type casting in SQL (`::numeric`, `::text`, etc.) is generally good practice in PostgreSQL.
    *   **Dummy Data:** Several `dashboard.js` endpoints return hardcoded dummy data on errors or when no data is found. While this prevents UI crashes, it can mask real issues. Ensure logging is robust when fallbacks are used.

**Summary of Endpoints**

Here's a summary of the available endpoints, grouped by their likely file/module:

**1. AI Prompts (`ai_prompts.js`)**
    *   `GET /`: Get all AI prompts.
    *   `GET /:id`: Get a specific AI prompt by its ID.
    *   `GET /company/:companyId`: Get the AI prompt for a specific company (expects one). **(Deprecated)**
    *   `GET /type/general`: Get the general AI prompt (expects one). **(Deprecated)**
    *   `GET /type/system`: Get the system AI prompt (expects one). **(Deprecated)**
    *   `GET /by-type`: Get AI prompt by type (general, system, company_specific) with optional company parameter. **(New Consolidated Endpoint)**
    *   `POST /`: Create a new AI prompt.
    *   `PUT /:id`: Update an existing AI prompt.
    *   `DELETE /:id`: Delete an AI prompt.

**2. AI Validation (`ai_validation.js`)**
    *   `POST /debug`: Generate and view the structure of prompts and taxonomy data (for debugging, doesn't call OpenAI). Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance).
    *   `POST /validate`: Validate product data using OpenAI. Connects to Prod MySQL (taxonomy) and Local PG (prompts, performance).
    *   `GET /test-taxonomy`: Test endpoint to query sample taxonomy data from Prod MySQL.

**3. Analytics (`analytics.js`)**
    *   `GET /stats`: Get overall business statistics from metrics tables.
    *   `GET /profit`: Get profit analysis data (by category, over time, top products) from metrics tables.
    *   `GET /vendors`: Get vendor performance analysis from `vendor_metrics`.
    *   `GET /stock`: Get stock analysis data (turnover, levels, critical items) from metrics tables.
    *   `GET /pricing`: Get pricing analysis (price points, elasticity, recommendations) - **uses `orders` table**.
    *   `GET /categories`: Get category performance analysis (revenue, profit, growth, distribution, trends) - **uses `orders` and `products` tables**.
    *   `GET /forecast`: (**Likely Obsolete/Broken**) Attempts to get forecast data using MySQL syntax.

**4. Brands Aggregate (`brands-aggregate.js`)**
    *   `GET /filter-options`: Get distinct brand names and statuses for UI filters (from `brand_metrics`).
    *   `GET /stats`: Get overall statistics related to brands (from `brand_metrics`).
    *   `GET /`: List brands with aggregated metrics, supporting filtering, sorting, pagination (from `brand_metrics`).

**5. Categories Aggregate (`categories-aggregate.js`)**
    *   `GET /filter-options`: Get distinct category types, statuses, and counts for UI filters (from `category_metrics` & `categories`).
    *   `GET /stats`: Get overall statistics related to categories (from `category_metrics` & `categories`).
    *   `GET /`: List categories with aggregated metrics, supporting filtering, sorting (incl. hierarchy), pagination (from `category_metrics` & `categories`).

**6. Configuration (`config.js`)**
    *   **(New)** `GET /global`: Get all global settings.
    *   **(New)** `PUT /global`: Update global settings.
    *   **(New)** `GET /products`: List product-specific settings with pagination/search.
    *   **(New)** `PUT /products/:pid`: Update/Create product-specific settings.
    *   **(New)** `POST /products/:pid/reset`: Reset product settings to defaults.
    *   **(New)** `GET /vendors`: List vendor-specific settings with pagination/search.
    *   **(New)** `PUT /vendors/:vendor`: Update/Create vendor-specific settings.
    *   **(New)** `POST /vendors/:vendor/reset`: Reset vendor settings to defaults.
    *   **(Legacy/Obsolete)** `GET /`: Get all config from old single-row tables.
    *   **(Legacy/Obsolete)** `PUT /stock-thresholds/:id`: Update old stock thresholds.
    *   **(Legacy/Obsolete)** `PUT /lead-time-thresholds/:id`: Update old lead time thresholds.
    *   **(Legacy/Obsolete)** `PUT /sales-velocity/:id`: Update old sales velocity config.
    *   **(Legacy/Obsolete)** `PUT /abc-classification/:id`: Update old ABC config.
    *   **(Legacy/Obsolete)** `PUT /safety-stock/:id`: Update old safety stock config.
    *   **(Legacy/Obsolete)** `PUT /turnover/:id`: Update old turnover config.

**7. CSV Operations & Background Tasks (`csv.js`)**
    *   `GET /:type/progress`: SSE endpoint for full update/reset progress.
    *   `GET /test`: Simple test endpoint.
    *   `GET /status`: Check status of the generic background task lock (`activeImport`).
    *   `GET /calculate-metrics/status`: Check status of metrics calculation.
    *   `GET /history/import`: Get recent import history.
    *   `GET /history/calculate`: Get recent metrics calculation history.
    *   `GET /status/modules`: Get last calculation time per module.
    *   `GET /status/tables`: Get last sync time per table.
    *   `GET /status/table-counts`: Get record counts for key tables.
    *   `POST /update`: (**Potentially Obsolete**) Trigger `update-csv.js` script.
    *   `POST /import`: Trigger `import-csv.js` script.
    *   `POST /cancel`: Cancel `/full-update` or `/full-reset` task.
    *   `POST /reset`: (**Potentially Obsolete**) Trigger `reset-db.js` script.
    *   `POST /reset-metrics`: Trigger `reset-metrics.js` script.
    *   `POST /calculate-metrics`: Trigger `calculate-metrics.js` script.
    *   `POST /import-from-prod`: Trigger `import-from-prod.js` script.
    *   `POST /full-update`: Trigger `full-update.js` script (preferred update).
    *   `POST /full-reset`: Trigger `full-reset.js` script (preferred reset).

**8. Dashboard (`dashboard.js`)**
    *   `GET /stock/metrics`: Get dashboard stock summary metrics & brand breakdown.
    *   `GET /purchase/metrics`: Get dashboard purchase order summary metrics & vendor breakdown.
    *   `GET /replenishment/metrics`: Get dashboard replenishment summary & top variants.
    *   `GET /forecast/metrics`: Get dashboard forecast summary, daily, and category breakdown.
    *   `GET /overstock/metrics`: Get dashboard overstock summary & category breakdown.
    *   `GET /overstock/products`: Get list of top overstocked products.
    *   `GET /best-sellers`: Get dashboard best-selling products, brands, categories - **uses `orders`, `products`**.
    *   `GET /sales/metrics`: Get dashboard sales summary for a period - **uses `orders`**.
    *   `GET /low-stock/products`: Get list of top low stock/critical products.
    *   `GET /trending/products`: Get list of trending products - **uses `orders`, `products`**.
    *   `GET /vendor/performance`: Get dashboard vendor performance details - **uses `purchase_orders`**.
    *   `GET /key-metrics`: Get dashboard summary KPIs - **uses multiple base tables**.
    *   `GET /inventory-health`: Get dashboard inventory health overview - **uses `products`, `product_metrics`**.
    *   `GET /replenish/products`: Get list of products needing replenishment (overlaps `/low-stock/products`).
    *   `GET /sales-overview`: Get daily sales totals for chart - **uses `orders`**.

**9. Product Import Utilities (`products-import.js`)**
    *   `POST /upload-image`: Upload temporary product image, schedule deletion.
    *   `DELETE /delete-image`: Delete temporary product image.
    *   `GET /field-options`: Get dropdown options for product fields from Prod MySQL (cached).
    *   `GET /product-lines/:companyId`: Get product lines for a company from Prod MySQL (cached).
    *   `GET /sublines/:lineId`: Get sublines for a line from Prod MySQL (cached).
    *   `GET /check-file/:filename`: Check existence/permissions of uploaded file (temp or reusable).
    *   `GET /list-uploads`: List files in upload directories.
    *   `GET /search-products`: Search products in Prod MySQL DB.
    *   `GET /check-upc-and-generate-sku`: Check UPC existence and generate SKU suggestion based on Prod MySQL data.
    *   `GET /product-categories/:pid`: Get assigned categories for a product from Prod MySQL.

**10. Product Metrics (`product-metrics.js`)**
    *   `GET /filter-options`: Get distinct filter values (vendor, brand, abcClass) from `product_metrics`.
    *   `GET /`: List detailed product metrics with filtering, sorting, pagination (primary data access).
    *   `GET /:pid`: Get full metrics record for a single product.

**11. Orders (`orders.js`)**
    *   `GET /`: List orders with summary info, filtering, sorting, pagination, and stats.
    *   `GET /:orderNumber`: Get details for a single order, including items.

**12. Products (`products.js`)**
    *   `GET /brands`: Get distinct brands (filtered by PO value).
    *   `GET /`: List products with core data + metrics, filtering, sorting, pagination.
    *   `GET /trending`: Get trending products based on `product_metrics`.
    *   `GET /:id`: Get details for a single product (core data + metrics).
    *   `POST /import`: (**Likely Obsolete/Dangerous**) Import products from CSV.
    *   `PUT /:id`: Update core product data.
    *   `GET /:id/metrics`: (**Redundant**) Get metrics for a single product.
    *   `GET /:id/time-series`: Get sales/PO history for a single product.

**13. Purchase Orders (`purchase-orders.js`)**
    *   `GET /`: List purchase orders with summary info, filtering, sorting, pagination, and summary stats.
    *   `GET /vendor-metrics`: Calculate vendor performance metrics from `purchase_orders`.
    *   `GET /cost-analysis`: Calculate cost analysis by category from `purchase_orders`.
    *   `GET /receiving-status`: Get summary counts based on PO receiving status.
    *   `GET /order-vs-received`: List product ordered vs. received quantities.

**14. Reusable Images (`reusable-images.js`)**
    *   `GET /`: List all reusable images.
    *   `GET /by-company/:companyId`: List global and company-specific images.
    *   `GET /global`: List only global images.
    *   `GET /:id`: Get a single reusable image record.
    *   `POST /upload`: Upload a new reusable image and create DB record.
    *   `PUT /:id`: Update reusable image metadata (name, global, company).
    *   `DELETE /:id`: Delete reusable image record and file.
    *   `GET /check-file/:filename`: Check existence/permissions of a reusable image file.

**15. Templates (`templates.js`)**
    *   `GET /`: List all product data templates.
    *   `GET /:company/:productType`: Get a specific template.
    *   `POST /`: Create a new template.
    *   `PUT /:id`: Update an existing template.
    *   `DELETE /:id`: Delete a template.

**16. Vendors Aggregate (`vendors-aggregate.js`)**
    *   `GET /filter-options`: Get distinct vendor names and statuses for UI filters (from `vendor_metrics`).
    *   `GET /stats`: Get overall statistics related to vendors (from `vendor_metrics` & `purchase_orders`).
    *   `GET /`: List vendors with aggregated metrics, supporting filtering, sorting, pagination (from `vendor_metrics` & `purchase_orders`).

**Recommendations:**

1.  **Address Obsolete Endpoints:** Prioritize removing or confirming the necessity of the endpoints marked as obsolete/redundant (legacy config, `/analytics/forecast`, `/csv/update`, `/csv/reset`, `/products/import`, `/products/:id/metrics`).
2.  **Consolidate Overlapping Functionality:** Review the multiple vendor performance and product listing endpoints. Decide on the primary method (e.g., using aggregate tables via `/vendors-aggregate` and `/metrics`) and refactor or remove the others. Clarify the image upload strategies.
3.  **Standardize Data Access:** Decide whether `dashboard` and `analytics` endpoints should primarily use aggregate tables (like `/metrics`, `/brands-aggregate`, etc.) or if direct access to base tables is sometimes necessary. Aim for consistency and document the reasoning. Optimize queries hitting base tables if they must remain.
4.  **Improve Background Task Management:** Refactor `csv.js` to use a unified locking mechanism (maybe separate locks per task type?) and a consistent cancellation strategy for all spawned/managed processes. Clarify the purpose of `update` vs `full-update` and `reset` vs `full-reset`.
5.  **Optimize DB Connections:** Ensure the `getDbConnection` pooling/caching helper from `products-import.js` is used *consistently* across all modules interacting with the production MySQL database (especially `ai_validation.js`). Remove unnecessary tunnel creations.
6.  **Review Data Integrity:** Double-check the assumptions made (e.g., uniqueness of AI prompts) and ensure database constraints enforce them. Review the `GET /products/brands` filtering logic.

## Changes Made

1. **Removed Obsolete Legacy Endpoints in `config.js`**:
   - Removed `GET /config/` endpoint
   - Removed `PUT /config/stock-thresholds/:id` endpoint
   - Removed `PUT /config/lead-time-thresholds/:id` endpoint
   - Removed `PUT /config/sales-velocity/:id` endpoint
   - Removed `PUT /config/abc-classification/:id` endpoint
   - Removed `PUT /config/safety-stock/:id` endpoint
   - Removed `PUT /config/turnover/:id` endpoint
   
   These endpoints were obsolete as they referenced older, single-row config tables that have been replaced by newer endpoints using the structured tables `settings_global`, `settings_product`, and `settings_vendor`.

2. **Removed MySQL Syntax `/forecast` Endpoint in `analytics.js`**:
   - Removed `GET /analytics/forecast` endpoint that was using MySQL-specific syntax incompatible with the PostgreSQL database used elsewhere in the application.

3. **Renamed and Removed Redundant Endpoints**:
   - Renamed `csv.js` to `data-management.js` while maintaining the same `/csv/*` endpoint paths for consistency
   - Removed deprecated `/csv/update` endpoint (now fully replaced by `/csv/full-update`)
   - Removed deprecated `/csv/reset` endpoint (now fully replaced by `/csv/full-reset`)
   - Removed deprecated `/products/import` endpoint (now handled by `/csv/import`)
   - Removed deprecated `/products/:id/metrics` endpoint (now handled by `/metrics/:pid`)

4. **Fixed Data Integrity Issues**:
   - Improved `GET /products/brands` endpoint by removing the arbitrary filtering logic that was only showing brands with purchase orders that had a total cost of at least $500
   - The updated endpoint now returns all distinct brands from visible products, providing more complete data

5. **Optimized Database Connections**:
   - Created a new `dbConnection.js` utility file that encapsulates the optimized database connection management logic
   - Improved the `ai-validation.js` file to use this shared connection management, eliminating unnecessary repeated tunnel creation
   - Added proper connection pooling with timeout-based connection reuse, reducing the overhead of repeatedly creating SSH tunnels
   - Added query result caching for frequently accessed data to improve performance

These changes improve maintainability by removing duplicate code, enhance consistency by standardizing on the newer endpoint patterns, and optimize performance by reducing redundant database connections.

## Additional Improvements

1. **Further Database Connection Optimizations**:
   - Extended the use of the optimized database connection utility to additional endpoints in `ai-validation.js`
   - Updated the `/validate` endpoint and `/test-taxonomy` endpoint to use `getDbConnection`
   - Ensured consistent connection management across all routes that access the production database

2. **AI Prompts Data Integrity Verification**:
   - Confirmed proper uniqueness constraints are in place in the database schema for AI prompts
   - The schema includes:
     - `unique_company_prompt` constraint ensuring only one prompt per company
     - `idx_unique_general_prompt` index ensuring only one general prompt in the system
     - `idx_unique_system_prompt` index ensuring only one system prompt in the system
   - Endpoint handlers properly handle uniqueness constraint violations with appropriate 409 Conflict responses
   - Validation ensures company-specific prompts have company IDs, while general/system prompts do not

3. **AI Prompts Endpoint Consolidation**:
   - Added a new consolidated `/by-type` endpoint that handles all types of prompts (general, system, company_specific)
   - Marked the existing separate endpoints as deprecated with console warnings
   - Maintained backward compatibility while providing a cleaner API moving forward

## Completed Items

✅ Removed obsolete legacy endpoints in `config.js`  
✅ Removed MySQL syntax `/forecast` endpoint in `analytics.js`  
✅ Fixed `GET /products/brands` endpoint filtering logic  
✅ Created reusable database connection utility (`dbConnection.js`)  
✅ Optimized database connections in `ai-validation.js`  
✅ Verified data integrity in AI prompts handling  
✅ Consolidated AI prompts endpoints with a unified `/by-type` endpoint  

## Remaining Items

- Consider adding additional error handling and logging for database connections
- Perform load testing on the optimized database connections to ensure they handle high traffic properly