Clean up/optimize validation step

.claude/CLAUDE.md

@@ -0,0 +1,172 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+This is a full-stack inventory management system with a React + TypeScript frontend and Node.js/Express backend using PostgreSQL. The system includes product management, analytics, forecasting, purchase orders, and a comprehensive dashboard for business metrics.
+
+**Monorepo Structure:**
+- `inventory/` - Vite-based React frontend with TypeScript
+- `inventory-server/` - Express backend API server
+- Root `package.json` contains shared dependencies
+
+## Development Commands
+
+### Frontend (inventory/)
+```bash
+cd inventory
+npm run dev        # Start dev server on port 5175
+npm run build      # Build for production (outputs to build/ then copies to ../inventory-server/frontend/build)
+npm run lint       # Run ESLint
+npm run preview    # Preview production build
+```
+
+### Backend (inventory-server/)
+```bash
+cd inventory-server
+npm run dev        # Start with nodemon (auto-reload)
+npm start          # Start server (production)
+npm run prod       # Start with PM2 for production
+npm run prod:stop  # Stop PM2 instance
+npm run prod:restart  # Restart PM2 instance
+npm run prod:logs  # View PM2 logs
+npm run setup      # Create required directories (logs, uploads)
+```
+
+## Architecture
+
+### Frontend Architecture
+
+**Router Structure:** React Router with lazy loading for code splitting:
+- Main chunks: Core inventory, Dashboard, Product Import, Chat Archive
+- Authentication flow uses `RequireAuth` and `Protected` components with permission-based access
+- All routes except `/login` and `/small` require authentication
+
+**Key Directories:**
+- `src/pages/` - Top-level page components (Overview, Products, Analytics, Dashboard, etc.)
+- `src/components/` - Organized by feature (dashboard/, products/, analytics/, etc.)
+- `src/components/ui/` - shadcn/ui components
+- `src/types/` - TypeScript type definitions
+- `src/contexts/` - React contexts (AuthContext, DashboardScrollContext)
+- `src/hooks/` - Custom React hooks (use-toast, useDebounce, use-mobile)
+- `src/utils/` - Utility functions (emojiUtils, productUtils, naturalLanguagePeriod)
+- `src/services/` - API service layer
+- `src/config/` - Configuration files
+
+**State Management:**
+- React Context for auth and global state
+- @tanstack/react-query for server state management
+- zustand for client state management
+- Local storage for auth tokens, session storage for login state
+
+**Key Dependencies:**
+- UI: Radix UI primitives, shadcn/ui, Tailwind CSS, Framer Motion
+- Data: @tanstack/react-table, react-data-grid, @tanstack/react-virtual
+- Forms: react-hook-form, zod
+- Charts: recharts, chart.js, react-chartjs-2
+- File handling: xlsx for Excel export, react-dropzone for uploads
+- Other: axios for HTTP, date-fns/luxon for dates
+
+**Path Alias:** `@/` maps to `./src/`
+
+### Backend Architecture
+
+**Entry Point:** `inventory-server/src/server.js`
+
+**Key Directories:**
+- `src/routes/` - Express route handlers (products, dashboard, analytics, import, etc.)
+- `src/middleware/` - Express middleware (CORS, auth, etc.)
+- `src/utils/` - Utility functions (database connection, API helpers)
+- `src/types/` - Type definitions (e.g., status-codes)
+
+**Database:**
+- PostgreSQL with connection pooling (pg library)
+- Pool initialized in `utils/db.js` via `initPool()`
+- Pool attached to `app.locals.pool` for route access
+- Environment variables loaded from `/var/www/html/inventory/.env` (production path)
+
+**API Routes:** All prefixed with `/api/`
+- `/api/products` - Product CRUD operations
+- `/api/dashboard` - Dashboard metrics and data
+- `/api/analytics` - Analytics and reporting
+- `/api/orders` - Order management
+- `/api/purchase-orders` - Purchase order management
+- `/api/csv` - CSV import/export (data management)
+- `/api/import` - Product import workflows
+- `/api/config` - Configuration management
+- `/api/metrics` - System metrics
+- `/api/ai-validation` - AI-powered validation
+- `/api/ai-prompts` - AI prompt management
+- `/api/templates` - Template management
+- `/api/reusable-images` - Image management
+- `/api/categoriesAggregate`, `/api/vendorsAggregate`, `/api/brandsAggregate` - Aggregate data endpoints
+
+**Authentication:**
+- External auth service at `/auth-inv` endpoint
+- Token-based authentication (Bearer tokens)
+- Frontend stores tokens in localStorage
+- Protected routes verify tokens via auth service `/me` endpoint
+
+**File Uploads:**
+- Multer middleware for file handling
+- Uploads directory: `inventory-server/uploads/`
+
+### Development Proxy Setup
+
+The Vite dev server (port 5175) proxies API requests to `https://inventory.kent.pw`:
+- `/api/*` → production API
+- `/auth-inv/*` → authentication service
+- `/chat-api/*` → chat service
+- `/uploads/*` → uploaded files
+- Various third-party services (Aircall, Klaviyo, Meta, Gorgias, Typeform, ACOT, Clarity)
+
+### Build Process
+
+When building the frontend:
+1. TypeScript compilation (`tsc -b`)
+2. Vite build (outputs to `inventory/build/`)
+3. Custom Vite plugin copies build to `inventory-server/frontend/build/`
+4. Manual chunks for vendor splitting (react-vendor, ui-vendor, query-vendor)
+
+## Testing
+
+Run tests for individual components or features:
+```bash
+# No test suite currently configured
+# Tests would typically use Jest or Vitest with React Testing Library
+```
+
+## Common Development Workflows
+
+### Adding a New Page
+1. Create page component in `inventory/src/pages/YourPage.tsx`
+2. Add lazy import in `inventory/src/App.tsx`
+3. Add route with `<Protected>` wrapper and permission check
+4. Add corresponding backend route in `inventory-server/src/routes/`
+5. Update permission system if needed
+
+### Adding a New API Endpoint
+1. Create or update route file in `inventory-server/src/routes/`
+2. Use `executeQuery()` helper for database queries
+3. Register router in `inventory-server/src/server.js`
+4. Frontend can access at `/api/{route-name}`
+
+### Working with Database
+- Use parameterized queries: `executeQuery(sql, [param1, param2])`
+- Pool is accessed via `db.getPool()` or `app.locals.pool`
+- Connection helper: `db.getConnection()` returns a client for transactions
+
+### Permissions System
+- User permissions stored in `user.permissions` array (permission codes)
+- Check permissions in `<Protected page="permission_code">` component
+- Admin users (`is_admin: true`) have access to all pages
+
+## Important Notes
+
+- Environment variables must be configured in `/var/www/html/inventory/.env` for production
+- The frontend expects the backend at `/api` (proxied in dev, served together in production)
+- PM2 is used for production process management
+- Database uses PostgreSQL with SSL support (configurable via `DB_SSL` env var)
+- File uploads stored in `inventory-server/uploads/` directory
+- Build artifacts in `inventory/build/` are copied to `inventory-server/frontend/build/`

inventory/src/components/product-import/config.ts

@@ -0,0 +1,340 @@
+import type { ErrorLevel } from "@/components/product-import";
+
+/**
+ * Base field configuration for product import
+ *
+ * These fields define the structure and validation rules for the import process.
+ * Options for select/multi-select fields are populated dynamically from the API.
+ */
+export const BASE_IMPORT_FIELDS = [
+  {
+    label: "Supplier",
+    key: "supplier",
+    description: "Primary supplier/manufacturer of the product",
+    fieldType: {
+      type: "select" as const,
+      options: [], // Will be populated from API
+    },
+    width: 220,
+    validations: [{ rule: "required" as const, errorMessage: "Required", level: "error" as ErrorLevel }],
+  },
+  {
+    label: "Company",
+    key: "company",
+    description: "Company/Brand name",
+    fieldType: {
+      type: "select",
+      options: [], // Will be populated from API
+    },
+    width: 220,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "Line",
+    key: "line",
+    description: "Product line",
+    alternateMatches: ["collection"],
+    fieldType: {
+      type: "select" as const,
+      options: [], // Will be populated dynamically based on company selection
+    },
+    width: 220,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "Sub Line",
+    key: "subline",
+    description: "Product sub-line",
+    fieldType: {
+      type: "select" as const,
+      options: [], // Will be populated dynamically based on line selection
+    },
+    width: 220,
+  },
+  {
+    label: "UPC",
+    key: "upc",
+    description: "Universal Product Code/Barcode",
+    alternateMatches: ["barcode", "bar code", "jan", "ean", "upc code"],
+    fieldType: { type: "input" },
+    width: 145,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "unique", errorMessage: "Must be unique", level: "error" },
+      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Item Number",
+    key: "item_number",
+    description: "Internal item reference number",
+    fieldType: { type: "input" },
+    width: 130,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "unique", errorMessage: "Must be unique", level: "error" },
+    ],
+  },
+  {
+    label: "Supplier #",
+    key: "supplier_no",
+    description: "Supplier's product identifier",
+    alternateMatches: ["sku", "item#", "mfg item #", "item", "supplier #"],
+    fieldType: { type: "input" },
+    width: 130,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "unique", errorMessage: "Must be unique", level: "error" },
+    ],
+  },
+  {
+    label: "Notions #",
+    key: "notions_no",
+    description: "Internal notions number",
+    alternateMatches: ["notions #","nmc"],
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "unique", errorMessage: "Must be unique", level: "error" },
+      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Name",
+    key: "name",
+    description: "Product name/title",
+    alternateMatches: ["sku description","product name"],
+    fieldType: { type: "input" },
+    width: 500,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "unique", errorMessage: "Must be unique", level: "error" },
+    ],
+  },
+  {
+    label: "MSRP",
+    key: "msrp",
+    description: "Manufacturer's Suggested Retail Price",
+    alternateMatches: ["retail", "retail price", "sugg retail", "price", "sugg. retail","default price"],
+    fieldType: {
+      type: "input",
+      price: true
+    },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Min Qty",
+    key: "qty_per_unit",
+    description: "Quantity of items per individual unit",
+    alternateMatches: ["inner pack", "inner", "min qty", "unit qty", "min. order qty", "supplier qty/unit"],
+    fieldType: { type: "input" },
+    width: 80,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Cost Each",
+    key: "cost_each",
+    description: "Wholesale cost per unit",
+    alternateMatches: ["wholesale", "wholesale price", "supplier cost each", "cost each"],
+    fieldType: {
+      type: "input",
+      price: true
+    },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Case Pack",
+    key: "case_qty",
+    description: "Number of units per case",
+    alternateMatches: ["mc qty","case qty","case pack","box ct"],
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Tax Category",
+    key: "tax_cat",
+    description: "Product tax category",
+    fieldType: {
+      type: "select",
+      options: [], // Will be populated from API
+    },
+    width: 200,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "Artist",
+    key: "artist",
+    description: "Artist/Designer name",
+    fieldType: {
+      type: "select",
+      options: [], // Will be populated from API
+    },
+    width: 200,
+  },
+  {
+    label: "ETA Date",
+    key: "eta",
+    description: "Estimated arrival date",
+    alternateMatches: ["shipping month"],
+    fieldType: { type: "input" },
+    width: 120,
+  },
+  {
+    label: "Weight",
+    key: "weight",
+    description: "Product weight (in lbs)",
+    alternateMatches: ["weight (lbs.)"],
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Length",
+    key: "length",
+    description: "Product length (in inches)",
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Width",
+    key: "width",
+    description: "Product width (in inches)",
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Height",
+    key: "height",
+    description: "Product height (in inches)",
+    fieldType: { type: "input" },
+    width: 100,
+    validations: [
+      { rule: "required", errorMessage: "Required", level: "error" },
+      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Shipping Restrictions",
+    key: "ship_restrictions",
+    description: "Product shipping restrictions",
+    fieldType: {
+      type: "select",
+      options: [], // Will be populated from API
+    },
+    width: 190,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "COO",
+    key: "coo",
+    description: "2-letter country code (ISO)",
+    alternateMatches: ["coo", "country of origin"],
+    fieldType: { type: "input" },
+    width: 70,
+    validations: [
+      { rule: "regex", value: "^[A-Z]{2}$", errorMessage: "Must be 2 letters", level: "error" },
+    ],
+  },
+  {
+    label: "HTS Code",
+    key: "hts_code",
+    description: "Harmonized Tariff Schedule code",
+    alternateMatches: ["taric","hts"],
+    fieldType: { type: "input" },
+    width: 130,
+    validations: [
+      { rule: "regex", value: "^[0-9.]+$", errorMessage: "Must be a number", level: "error" },
+    ],
+  },
+  {
+    label: "Size Category",
+    key: "size_cat",
+    description: "Product size category",
+    fieldType: {
+      type: "select",
+      options: [], // Will be populated from API
+    },
+    width: 180,
+  },
+  {
+    label: "Description",
+    key: "description",
+    description: "Detailed product description",
+    alternateMatches: ["details/description"],
+    fieldType: {
+      type: "input",
+      multiline: true
+    },
+    width: 500,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "Private Notes",
+    key: "priv_notes",
+    description: "Internal notes about the product",
+    fieldType: {
+      type: "input",
+      multiline: true
+    },
+    width: 300,
+  },
+  {
+    label: "Categories",
+    key: "categories",
+    description: "Product categories",
+    fieldType: {
+      type: "multi-select",
+      options: [], // Will be populated from API
+    },
+    width: 350,
+    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
+  },
+  {
+    label: "Themes",
+    key: "themes",
+    description: "Product themes/styles",
+    fieldType: {
+      type: "multi-select",
+      options: [], // Will be populated from API
+    },
+    width: 300,
+  },
+  {
+    label: "Colors",
+    key: "colors",
+    description: "Product colors",
+    fieldType: {
+      type: "multi-select",
+      options: [], // Will be populated from API
+    },
+    width: 200,
+  },
+] as const;
+
+export type ImportFieldKey = (typeof BASE_IMPORT_FIELDS)[number]["key"];

inventory/src/components/product-import/steps/ValidationStepNew/ValidationStepNew.tsx

@@ -1,80 +0,0 @@
-import React, { useState } from 'react';
-import { AiValidationDialogs } from './components/AiValidationDialogs';
-import { Product } from '../../../../types/products';
-import { 
-  AiValidationProgress, 
-  AiValidationDetails, 
-  CurrentPrompt as AiValidationCurrentPrompt 
-} from './hooks/useAiValidation';
-
-const ValidationStepNew: React.FC = () => {
-  const [aiValidationProgress, setAiValidationProgress] = useState<AiValidationProgress>({
-    isOpen: false,
-    status: 'idle',
-    step: 0
-  });
-  
-  const [aiValidationDetails, setAiValidationDetails] = useState<AiValidationDetails>({
-    changes: [],
-    warnings: [],
-    changeDetails: [],
-    isOpen: false
-  });
-  
-  const [currentPrompt, setCurrentPrompt] = useState<AiValidationCurrentPrompt>({
-    isOpen: false,
-    prompt: '',
-    isLoading: true,
-  });
-  
-  // Track reversion state (for internal use)
-  const [reversionState, setReversionState] = useState<Record<string, boolean>>({});
-  
-  const [fieldData] = useState<Product[]>([]);
-
-  // eslint-disable-next-line @typescript-eslint/no-unused-vars
-
-  const revertAiChange = (productIndex: number, fieldKey: string) => {
-    const key = `${productIndex}-${fieldKey}`;
-    setReversionState(prev => ({
-      ...prev,
-      [key]: true
-    }));
-  };
-
-  const isChangeReverted = (productIndex: number, fieldKey: string): boolean => {
-    const key = `${productIndex}-${fieldKey}`;
-    return !!reversionState[key];
-  };
-
-  const getFieldDisplayValueWithHighlight = (
-    _fieldKey: string,
-    originalValue: any,
-    correctedValue: any
-  ) => {
-    return {
-      originalHtml: String(originalValue),
-      correctedHtml: String(correctedValue)
-    };
-  };
-
-  return (
-    <div>
-      <AiValidationDialogs
-        aiValidationProgress={aiValidationProgress}
-        aiValidationDetails={aiValidationDetails}
-        currentPrompt={currentPrompt}
-        setAiValidationProgress={setAiValidationProgress}
-        setAiValidationDetails={setAiValidationDetails}
-        setCurrentPrompt={setCurrentPrompt}
-        revertAiChange={revertAiChange}
-        isChangeReverted={isChangeReverted}
-        getFieldDisplayValueWithHighlight={getFieldDisplayValueWithHighlight}
-        fields={fieldData}
-        debugData={currentPrompt.debugData}
-      />
-    </div>
-  );
-};
-
-export default ValidationStepNew; 

inventory/src/components/product-import/steps/ValidationStepNew/components/ValidationContainer.tsx

@@ -1,6 +1,6 @@
-import React, { useState, useEffect, useRef, useCallback, useMemo, useLayoutEffect } from 'react'
+import React, { useState, useEffect, useCallback, useMemo } from 'react'
 import { useValidationState } from '../hooks/useValidationState'
-import { Props } from '../hooks/validationTypes'
+import { Props, RowData } from '../hooks/validationTypes'
 import { Button } from '@/components/ui/button'
 import { Loader2, X, Plus, Edit3, Sparkles, FileText } from 'lucide-react'
 import { toast } from 'sonner'
@@ -14,11 +14,12 @@ import { SearchProductTemplateDialog } from '@/components/templates/SearchProduc
 import { TemplateForm } from '@/components/templates/TemplateForm'
 import axios from 'axios'
 import { RowSelectionState } from '@tanstack/react-table'
-import { useUpcValidation } from '../hooks/useUpcValidation'
 import { useProductLinesFetching } from '../hooks/useProductLinesFetching'
 import UpcValidationTableAdapter from './UpcValidationTableAdapter'
 import { Skeleton } from '@/components/ui/skeleton'
 import { Protected } from '@/components/auth/Protected'
+import { normalizeCountryCode } from '../utils/countryUtils'
+import { cleanPriceField } from '../utils/priceUtils'
 /**
  * ValidationContainer component - the main wrapper for the validation step
  * 
@@ -59,13 +60,29 @@ const ValidationContainer = <T extends string>({
     loadTemplates,
     setData,
     fields,
+    upcValidation,
     isLoadingTemplates,
+    isValidating,
+    isInitializing,
     validatingCells,
     setValidatingCells,
     editingCells,
-    setEditingCells
+    setEditingCells,
+    updateRow,
+    revalidateRows
   } = validationState
 
+  const dataIndexByRowId = useMemo(() => {
+    const map = new Map<any, number>()
+    data.forEach((row, index) => {
+      const rowId = (row as Record<string, any>).__index
+      if (rowId !== undefined && rowId !== null) {
+        map.set(rowId, index)
+      }
+    })
+    return map
+  }, [data])
+  
   // Use product lines fetching hook
   const {
     rowProductLines,
@@ -76,9 +93,6 @@ const ValidationContainer = <T extends string>({
     fetchSublines
   } = useProductLinesFetching(data);
   
-  // Use UPC validation hook
-  const upcValidation = useUpcValidation(data, setData);
-  
   // Function to check if a specific row is being validated - memoized
   const isRowValidatingUpc = upcValidation.isRowValidatingUpc;
   
@@ -115,11 +129,16 @@ const ValidationContainer = <T extends string>({
   // Add new state for template form dialog
   const [isTemplateFormOpen, setIsTemplateFormOpen] = useState(false)
   const [templateFormInitialData, setTemplateFormInitialData] = useState<any>(null)
+
+  const pendingInitializationTasks: string[] = []
+  if (isValidating) pendingInitializationTasks.push('validating rows')
+  if (upcValidation.validatingRows.size > 0) pendingInitializationTasks.push('checking UPCs')
+  if (isLoadingTemplates) pendingInitializationTasks.push('loading templates')
   const [fieldOptions, setFieldOptions] = useState<any>(null)
 
   // Track fields that need revalidation due to value changes
-  const [fieldsToRevalidate, setFieldsToRevalidate] = useState<Set<number>>(new Set());
+  // Combined state: Map<rowIndex, fieldKeys[]> - if empty array, revalidate all fields
-  const [fieldsToRevalidateMap, setFieldsToRevalidateMap] = useState<{[rowIndex: number]: string[]}>({});
+  const [fieldsToRevalidate, setFieldsToRevalidate] = useState<Map<number, string[]>>(new Map());
 
   // Function to mark a row for revalidation
   const markRowForRevalidation = useCallback((rowIndex: number, fieldKey?: string) => {
@@ -138,24 +157,17 @@ const ValidationContainer = <T extends string>({
     })();
 
     setFieldsToRevalidate(prev => {
-      const newSet = new Set(prev);
+      const newMap = new Map(prev);
-      newSet.add(originalIndex);
+      const existingFields = newMap.get(originalIndex) || [];
-      return newSet;
-    });
 
-    // Also track which specific field needs to be revalidated
+      if (fieldKey && !existingFields.includes(fieldKey)) {
-    if (fieldKey) {
+        newMap.set(originalIndex, [...existingFields, fieldKey]);
-      setFieldsToRevalidateMap(prev => {
+      } else if (!fieldKey) {
-        const newMap = { ...prev };
+        newMap.set(originalIndex, existingFields);
-        if (!newMap[originalIndex]) {
-          newMap[originalIndex] = [];
-        }
-        if (!newMap[originalIndex].includes(fieldKey)) {
-          newMap[originalIndex] = [...newMap[originalIndex], fieldKey];
       }
+
       return newMap;
     });
-    }
   }, [data, filteredData]);
 
   // Add a ref to track the last validation time
@@ -164,21 +176,19 @@ const ValidationContainer = <T extends string>({
   useEffect(() => {
     if (fieldsToRevalidate.size === 0) return;
 
-    // Revalidate the marked rows
+    // Extract rows and fields map
-    const rowsToRevalidate = Array.from(fieldsToRevalidate);
+    const rowsToRevalidate = Array.from(fieldsToRevalidate.keys());
+    const fieldsMap: {[rowIndex: number]: string[]} = {};
+    fieldsToRevalidate.forEach((fields, rowIndex) => {
+      fieldsMap[rowIndex] = fields;
+    });
 
-    // Clear the revalidation set
+    // Clear the revalidation map
-    setFieldsToRevalidate(new Set());
+    setFieldsToRevalidate(new Map());
-    
-    // Get the fields map for revalidation
-    const fieldsMap = { ...fieldsToRevalidateMap };
-    
-    // Clear the fields map
-    setFieldsToRevalidateMap({});
 
     // Revalidate each row with specific fields information
-    validationState.revalidateRows(rowsToRevalidate, fieldsMap);
+    revalidateRows(rowsToRevalidate, fieldsMap);
-  }, [fieldsToRevalidate, validationState, fieldsToRevalidateMap]);
+  }, [fieldsToRevalidate, revalidateRows]);
 
   // Function to fetch field options for template form
   const fetchFieldOptions = useCallback(async () => {
@@ -395,107 +405,40 @@ const ValidationContainer = <T extends string>({
     [setRowSelection]
   );
 
-  // Add scroll container ref at the container level
-  const scrollContainerRef = useRef<HTMLDivElement>(null);
-  const lastScrollPosition = useRef({ left: 0, top: 0 });
-  
   // Track if we're currently validating a UPC
 
   // Track last UPC update to prevent conflicting changes
 
   // Add these ref declarations here, at component level
 
-  // Memoize scroll handlers - simplified to avoid performance issues
+  // Helper: Process field value transformations
-  const handleScroll = useCallback((event: React.UIEvent<HTMLDivElement> | Event) => {
+  const processFieldValue = useCallback((key: T, value: any): any => {
-    // Store scroll position directly without conditions
-    const target = event.currentTarget as HTMLDivElement;
-    lastScrollPosition.current = {
-      left: target.scrollLeft,
-      top: target.scrollTop
-    };
-  }, []);
-
-  // Add scroll event listener
-  useEffect(() => {
-    const container = scrollContainerRef.current;
-    if (container) {
-      // Convert React event handler to native event handler
-      const nativeHandler = ((evt: Event) => {
-        handleScroll(evt);
-      }) as EventListener;
-      
-      container.addEventListener('scroll', nativeHandler, { passive: true });
-      return () => container.removeEventListener('scroll', nativeHandler);
-    }
-  }, [handleScroll]);
-
-  // Use a ref to track if we need to restore scroll position
-  const needScrollRestore = useRef(false);
-
-  // Set flag when data changes
-  useEffect(() => {
-    needScrollRestore.current = true;
-    // Only restore scroll on layout effects to avoid triggering rerenders
-  }, []);
-
-  // Use layout effect for DOM manipulations
-  useLayoutEffect(() => {
-    if (!needScrollRestore.current) return;
-    
-    const container = scrollContainerRef.current;
-    if (container && (lastScrollPosition.current.left > 0 || lastScrollPosition.current.top > 0)) {
-      container.scrollLeft = lastScrollPosition.current.left;
-      container.scrollTop = lastScrollPosition.current.top;
-      needScrollRestore.current = false;
-    }
-  }, []);
-
-  // Ensure manual edits to item numbers persist with minimal changes to validation logic
-  const handleUpdateRow = useCallback(async (rowIndex: number, key: T, value: any) => {
-    // Process value before updating data
     let processedValue = value;
 
-    // Strip dollar signs from price fields
+    // Clean price fields
-    if ((key === 'msrp' || key === 'cost_each') && typeof value === 'string') {
+    if ((key === 'msrp' || key === 'cost_each') && value !== undefined && value !== null) {
-      processedValue = value.replace(/[$,]/g, '');
+      processedValue = cleanPriceField(value);
+    }
 
-      // Also ensure it's a valid number
+    // Normalize country code
-      const numValue = parseFloat(processedValue);
+    if (key === 'coo' && typeof value === 'string' && value.trim()) {
-      if (!isNaN(numValue)) {
+      const normalized = normalizeCountryCode(value);
-        processedValue = numValue.toFixed(2);
+      if (normalized) {
+        processedValue = normalized;
+      } else {
+        const trimmed = value.trim();
+        if (trimmed.length === 2) {
+          processedValue = trimmed.toUpperCase();
+        }
       }
     }
 
-    // Find the row in the data
+    return processedValue;
-    const rowData = filteredData[rowIndex];
+  }, []);
-    if (!rowData) {
-      console.error(`No row data found for index ${rowIndex}`);
-      return;
-    }
 
-    // Use __index to find the actual row in the full data array
+  // Helper: Handle company change side effects
-    const rowId = rowData.__index;
+  const handleCompanyChange = useCallback((rowIndex: number, rowId: any, companyId: string) => {
-    const originalIndex = data.findIndex(item => item.__index === rowId);
+    // Clear line/subline values
-    
-    // Detect if this is a direct item_number edit
-    const isItemNumberEdit = key === 'item_number' as T;
-    
-    // For item_number edits, use core updateRow to atomically update + validate
-    if (isItemNumberEdit) {
-      const idx = originalIndex >= 0 ? originalIndex : rowIndex;
-      validationState.updateRow(idx, key as unknown as any, processedValue);
-      return;
-    }
-    
-    // For all other fields, use core updateRow for atomic update + validation
-    const idx = originalIndex >= 0 ? originalIndex : rowIndex;
-    validationState.updateRow(idx, key as unknown as any, processedValue);
-    
-    // Secondary effects - using requestAnimationFrame for better performance
-    requestAnimationFrame(() => {
-      // Handle company change - clear line/subline and fetch product lines
-      if (key === 'company' && value) {
-        // Clear any existing line/subline values immediately
     setData(prevData => {
       const newData = [...prevData];
       const idx = newData.findIndex(item => item.__index === rowId);
@@ -509,18 +452,9 @@ const ValidationContainer = <T extends string>({
       return newData;
     });
 
-        // Fetch product lines for the new company with debouncing
+    // Fetch product lines
-        if (rowId && value !== undefined) {
+    setValidatingCells(prev => new Set(prev).add(`${rowIndex}-line`));
-          const companyId = value.toString();
 
-          // Set loading state first
-          setValidatingCells(prev => {
-            const newSet = new Set(prev);
-            newSet.add(`${rowIndex}-line`);
-            return newSet;
-          });
-          
-          // Debounce the API call to prevent excessive requests
     setTimeout(() => {
       fetchProductLines(rowId, companyId)
         .catch(err => {
@@ -528,62 +462,18 @@ const ValidationContainer = <T extends string>({
           toast.error("Failed to load product lines");
         })
         .finally(() => {
-                // Clear loading indicator
           setValidatingCells(prev => {
             const newSet = new Set(prev);
             newSet.delete(`${rowIndex}-line`);
             return newSet;
           });
         });
-          }, 100); // 100ms debounce
+    }, 100);
-        }
+  }, [setData, fetchProductLines]);
-      }
 
-      // Handle supplier + UPC validation - using the most recent values
+  // Helper: Handle line change side effects
-      if (key === 'supplier' && value) {
+  const handleLineChange = useCallback((rowIndex: number, rowId: any, lineId: string) => {
-        // Get the latest UPC value from the updated row
+    // Clear subline value
-        const upcValue = (data[rowIndex] as any)?.upc || (data[rowIndex] as any)?.barcode;
-        
-        if (upcValue) {
-          
-          // Mark the item_number cell as being validated
-          const cellKey = `${rowIndex}-item_number`;
-          setValidatingCells(prev => {
-            const newSet = new Set(prev);
-            newSet.add(cellKey);
-            return newSet;
-          });
-          
-          // Use a regular promise-based approach instead of await
-          upcValidation.validateUpc(rowIndex, value.toString(), upcValue.toString())
-            .then(result => {
-              if (result.success) {
-                upcValidation.applyItemNumbersToData();
-                
-                // Mark for revalidation after item numbers are updated
-                setTimeout(() => {
-                  markRowForRevalidation(rowIndex, 'item_number');
-                }, 50);
-              }
-            })
-            .catch(err => {
-              console.error("Error validating UPC:", err);
-            })
-            .finally(() => {
-              // Clear validation state for the item_number cell
-              setValidatingCells(prev => {
-                const newSet = new Set(prev);
-                newSet.delete(cellKey);
-                return newSet;
-              });
-            });
-        }
-      }
-      
-      // Handle line change - clear subline and fetch sublines
-      if (key === 'line' && value) {
-        
-        // Clear any existing subline value
     setData(prevData => {
       const newData = [...prevData];
       const idx = newData.findIndex(item => item.__index === rowId);
@@ -598,270 +488,151 @@ const ValidationContainer = <T extends string>({
       return newData;
     });
 
-        // Fetch sublines for the new line
+    // Fetch sublines
-        if (rowId && value !== undefined) {
+    setValidatingCells(prev => new Set(prev).add(`${rowIndex}-subline`));
-          const lineId = value.toString();
-          
-          // Set loading state first
-          setValidatingCells(prev => {
-            const newSet = new Set(prev);
-            newSet.add(`${rowIndex}-subline`);
-            return newSet;
-          });
 
     fetchSublines(rowId, lineId)
-            .then(() => {
-            })
       .catch(err => {
         console.error(`Error fetching sublines for line ${lineId}:`, err);
         toast.error("Failed to load sublines");
       })
       .finally(() => {
-              // Clear loading indicator
         setValidatingCells(prev => {
           const newSet = new Set(prev);
           newSet.delete(`${rowIndex}-subline`);
           return newSet;
         });
       });
-        }
+  }, [setData, fetchSublines]);
-      }
 
-      // Add the UPC/barcode validation handler back:
+  // Helper: Handle UPC validation
-      // Handle UPC/barcode + supplier validation
+  const handleUpcValidation = useCallback((rowIndex: number, supplier: string, upc: string) => {
-      if ((key === 'upc' || key === 'barcode') && value) {
-        // Get latest supplier from the updated row
-        const supplier = (data[rowIndex] as any)?.supplier;
-        
-        if (supplier) {
-          
-          // Mark the item_number cell as being validated
     const cellKey = `${rowIndex}-item_number`;
-          setValidatingCells(prev => {
+    setValidatingCells(prev => new Set(prev).add(cellKey));
-            const newSet = new Set(prev);
-            newSet.add(cellKey);
-            return newSet;
-          });
 
-          // Use a regular promise-based approach
-          upcValidation.validateUpc(rowIndex, supplier.toString(), value.toString())
-            .then(result => {
-              if (result.success) {
-                upcValidation.applyItemNumbersToData();
-                
-                // Mark for revalidation after item numbers are updated
-                setTimeout(() => {
-                  markRowForRevalidation(rowIndex, 'item_number');
-                }, 50);
-              }
-            })
-            .catch(err => {
-              console.error("Error validating UPC:", err);
-            })
-            .finally(() => {
-              // Clear validation state for the item_number cell
-              setValidatingCells(prev => {
-                const newSet = new Set(prev);
-                newSet.delete(cellKey);
-                return newSet;
-              });
-            });
-        }
-      }
-    }); // Using requestAnimationFrame to defer execution until after the UI update
-  }, [data, filteredData, setData, fetchProductLines, fetchSublines, upcValidation, markRowForRevalidation]);
-  
-  // Fix the missing loading indicator clear code
-  const handleCopyDown = useCallback((rowIndex: number, fieldKey: string, endRowIndex?: number) => {
-    // Get the value to copy from the source row
-    const sourceRow = data[rowIndex];
-    if (!sourceRow) {
-      console.error(`Source row ${rowIndex} not found for copyDown`);
-      return;
-    }
-    
-    const valueToCopy = sourceRow[fieldKey];
-    
-    // Create a proper copy of the value to avoid reference issues, especially for arrays (MultiSelectCell)
-    const valueCopy = Array.isArray(valueToCopy) ? [...valueToCopy] : valueToCopy;
-
-    // Get all rows below the source row, up to endRowIndex if specified
-    const lastRowIndex = endRowIndex !== undefined ? Math.min(endRowIndex, data.length - 1) : data.length - 1;
-    const rowsToUpdate = Array.from({ length: lastRowIndex - rowIndex }, (_, i) => rowIndex + i + 1);
-    
-    // Mark all cells as updating at once
-    const updatingCells = new Set<string>();
-    rowsToUpdate.forEach(targetRowIndex => {
-      updatingCells.add(`${targetRowIndex}-${fieldKey}`);
-    });
-    
-    setValidatingCells(prev => {
-      const newSet = new Set(prev);
-      updatingCells.forEach(cell => newSet.add(cell));
-      return newSet;
-    });
-
-    // Update all rows at once efficiently with a single state update
-    setData(prevData => {
-      // Create a new copy of the data
-      const newData = [...prevData];
-      
-      // Update all rows at once
-      rowsToUpdate.forEach(targetRowIndex => {
-        // Find the original row using __index
-        const rowData = filteredData[targetRowIndex];
-        if (!rowData) return;
-        
-        const rowId = rowData.__index;
-        const originalIndex = newData.findIndex(item => item.__index === rowId);
-        
-        if (originalIndex !== -1) {
-          // Update the specific field on this row
-          newData[originalIndex] = {
-            ...newData[originalIndex],
-            [fieldKey]: valueCopy
-          };
-        } else {
-          // Fall back to direct index if __index not found
-          if (targetRowIndex < newData.length) {
-            newData[targetRowIndex] = {
-              ...newData[targetRowIndex],
-              [fieldKey]: valueCopy
-            };
-          }
-        }
-      });
-      
-      return newData;
-    });
-    
-    // Mark rows for revalidation
-    rowsToUpdate.forEach(targetRowIndex => {
-      markRowForRevalidation(targetRowIndex, fieldKey);
-    });
-    
-    // Clear the loading state for all cells efficiently
-    requestAnimationFrame(() => {
-      setValidatingCells(prev => {
-        if (prev.size === 0 || updatingCells.size === 0) return prev;
-        const newSet = new Set(prev);
-        updatingCells.forEach(cell => newSet.delete(cell));
-        return newSet;
-      });
-    });
-    
-    // If copying UPC or supplier fields, validate UPC for all rows
-    if (fieldKey === 'upc' || fieldKey === 'barcode' || fieldKey === 'supplier') {
-      // Process each row in parallel
-      const validationsToRun: {rowIndex: number, supplier: string, upc: string}[] = [];
-      
-      // Process each row separately to collect validation tasks
-      rowsToUpdate.forEach(targetRowIndex => {
-        const rowData = filteredData[targetRowIndex];
-        if (!rowData) return;
-        
-        // Only validate if both UPC and supplier are present after the update
-        const updatedRow = {
-          ...rowData,
-          [fieldKey]: valueCopy
-        };
-        
-        const hasUpc = updatedRow.upc || updatedRow.barcode;
-        const hasSupplier = updatedRow.supplier;
-        
-        if (hasUpc && hasSupplier) {
-          const upcValue = updatedRow.upc || updatedRow.barcode;
-          const supplierId = updatedRow.supplier;
-          
-          // Queue this validation if both values are defined
-          if (supplierId !== undefined && upcValue !== undefined) {
-            validationsToRun.push({
-              rowIndex: targetRowIndex,
-              supplier: supplierId.toString(),
-              upc: upcValue.toString()
-            });
-          }
-        }
-      });
-      
-      // Run validations in parallel but limit the batch size
-      if (validationsToRun.length > 0) {
-        
-        // Mark all cells as validating
-        validationsToRun.forEach(({ rowIndex }) => {
-          const cellKey = `${rowIndex}-item_number`;
-          setValidatingCells(prev => {
-            const newSet = new Set(prev);
-            newSet.add(cellKey);
-            return newSet;
-          });
-        });
-        
-        // Process in smaller batches to avoid overwhelming the system
-        const BATCH_SIZE = 5; // Process 5 validations at a time
-        const processBatch = (startIdx: number) => {
-          const endIdx = Math.min(startIdx + BATCH_SIZE, validationsToRun.length);
-          const batch = validationsToRun.slice(startIdx, endIdx);
-          
-          Promise.all(
-            batch.map(({ rowIndex, supplier, upc }) => 
     upcValidation.validateUpc(rowIndex, supplier, upc)
       .then(result => {
         if (result.success) {
-                    // Apply immediately for better UX
+          upcValidation.applyItemNumbersToData();
-                    if (startIdx + BATCH_SIZE >= validationsToRun.length) {
+          setTimeout(() => markRowForRevalidation(rowIndex, 'item_number'), 50);
-                      // Apply all updates at the end with callback to mark for revalidation
-                      upcValidation.applyItemNumbersToData(updatedRowIds => {
-                        // Mark these rows for revalidation after a delay
-                        setTimeout(() => {
-                          updatedRowIds.forEach(rowIdx => {
-                            markRowForRevalidation(rowIdx, 'item_number');
-                          });
-                        }, 100);
-                      });
         }
-                  }
-                  return { rowIndex, success: result.success };
-                })
-                .catch(err => {
-                  console.error(`Error validating UPC for row ${rowIndex}:`, err);
-                  return { rowIndex, success: false };
       })
+      .catch(err => console.error("Error validating UPC:", err))
       .finally(() => {
-                  // Clear validation state for this cell
-                  const cellKey = `${rowIndex}-item_number`;
         setValidatingCells(prev => {
-                    if (!prev.has(cellKey)) return prev;
           const newSet = new Set(prev);
           newSet.delete(cellKey);
           return newSet;
         });
-                })
-            )
-          ).then(() => {
-            // If there are more validations to run, process the next batch
-            if (endIdx < validationsToRun.length) {
-              // Add a small delay between batches to prevent UI freezing
-              setTimeout(() => processBatch(endIdx), 100);
-            } else {
-              // Final application of all item numbers if not done by individual batches
-              upcValidation.applyItemNumbersToData(updatedRowIds => {
-                // Mark these rows for revalidation after a delay
-                setTimeout(() => {
-                  updatedRowIds.forEach(rowIdx => {
-                    markRowForRevalidation(rowIdx, 'item_number');
       });
-                }, 100);
+  }, [upcValidation, markRowForRevalidation]);
-              });
-            }
-          });
-        };
 
-        // Start processing the first batch
+  // Main update handler - simplified to focus on core logic
-        processBatch(0);
+  const handleUpdateRow = useCallback(async (rowIndex: number, key: T, value: any) => {
+    // Process value transformations
+    const processedValue = processFieldValue(key, value);
+
+    // Find the row in the data
+    const rowData = filteredData[rowIndex];
+    if (!rowData) {
+      console.error(`No row data found for index ${rowIndex}`);
+      return;
+    }
+
+    // Use __index to find the actual row in the full data array
+    const rowId = rowData.__index;
+    const originalIndex = data.findIndex(item => item.__index === rowId);
+    const idx = originalIndex >= 0 ? originalIndex : rowIndex;
+
+    // Update the row with validation
+    updateRow(idx, key as unknown as any, processedValue);
+
+    // Handle secondary effects asynchronously
+    requestAnimationFrame(() => {
+      if (key === 'company' && value) {
+        handleCompanyChange(rowIndex, rowId, value.toString());
+      }
+
+      if (key === 'line' && value) {
+        handleLineChange(rowIndex, rowId, value.toString());
+      }
+
+      if (key === 'supplier' && value) {
+        const upcValue = (data[rowIndex] as any)?.upc || (data[rowIndex] as any)?.barcode;
+        if (upcValue) {
+          handleUpcValidation(rowIndex, value.toString(), upcValue.toString());
         }
       }
-  }, [data, filteredData, setData, setValidatingCells, upcValidation, markRowForRevalidation]);
+
+      if ((key === 'upc' || key === 'barcode') && value) {
+        const supplier = (data[rowIndex] as any)?.supplier;
+        if (supplier) {
+          handleUpcValidation(rowIndex, supplier.toString(), value.toString());
+        }
+      }
+    });
+  }, [data, filteredData, updateRow, processFieldValue, handleCompanyChange, handleLineChange, handleUpcValidation]);
+  
+  // Copy-down that keeps validations in sync for all affected rows
+  const handleCopyDown = useCallback((rowIndex: number, fieldKey: string, endRowIndex?: number) => {
+    const getActualIndex = (filteredIndex: number) => {
+      const filteredRow = filteredData[filteredIndex] as Record<string, any> | undefined
+      if (!filteredRow) return -1
+      const rowId = filteredRow.__index
+      if (rowId === undefined || rowId === null) return -1
+      const actualIndex = dataIndexByRowId.get(rowId)
+      return actualIndex === undefined ? -1 : actualIndex
+    }
+
+    const sourceActualIndex = getActualIndex(rowIndex)
+    if (sourceActualIndex < 0) {
+      console.error(`Unable to resolve source index ${rowIndex} for copyDown`)
+      return
+    }
+
+    const sourceRow = data[sourceActualIndex] as Record<string, any> | undefined
+    if (!sourceRow) {
+      console.error(`Source row ${sourceActualIndex} not found for copyDown`)
+      return
+    }
+
+    const baseValue = sourceRow[fieldKey]
+    const cloneValue = (value: any) => {
+      if (Array.isArray(value)) return [...value]
+      if (value && typeof value === 'object') return { ...value }
+      return value
+    }
+
+    const maxFilteredIndex = filteredData.length - 1
+    const lastFilteredIndex = endRowIndex !== undefined ? Math.min(endRowIndex, maxFilteredIndex) : maxFilteredIndex
+    if (lastFilteredIndex <= rowIndex) return
+
+    const updatedRows = new Set<number>()
+    updatedRows.add(sourceActualIndex)
+
+    const snapshot = data.map((row) => ({ ...row })) as RowData<T>[]
+
+    for (let idx = rowIndex + 1; idx <= lastFilteredIndex; idx++) {
+      const targetActualIndex = getActualIndex(idx)
+      if (targetActualIndex < 0) continue
+
+      updatedRows.add(targetActualIndex)
+      snapshot[targetActualIndex] = {
+        ...snapshot[targetActualIndex],
+        [fieldKey]: cloneValue(baseValue)
+      }
+
+      updateRow(targetActualIndex, fieldKey as T, cloneValue(baseValue))
+    }
+
+    const affectedIndexes = Array.from(updatedRows)
+    const fieldsMap: { [rowIndex: number]: string[] } = {}
+    affectedIndexes.forEach((idx) => {
+      fieldsMap[idx] = [fieldKey]
+    })
+
+    revalidateRows(affectedIndexes, fieldsMap, snapshot)
+  }, [data, dataIndexByRowId, filteredData, updateRow, revalidateRows])
   
   // Memoize the rendered validation table
   const renderValidationTable = useMemo(() => {
@@ -926,6 +697,20 @@ const ValidationContainer = <T extends string>({
     isLoadingSublines
   ]);
 
+  // Show loading state during initialization
+  if (isInitializing) {
+    return (
+      <div className="flex flex-col items-center justify-center h-[calc(100vh-10rem)]">
+        <Loader2 className="h-12 w-12 animate-spin text-primary mb-4" />
+        <h3 className="text-lg font-medium text-foreground mb-2">Initializing Validation</h3>
+        <p className="text-sm text-muted-foreground">Processing {data.length} rows...</p>
+        {pendingInitializationTasks.length > 0 && (
+          <p className="text-xs text-muted-foreground">Still {pendingInitializationTasks.join(' | ')}</p>
+        )}
+      </div>
+    );
+  }
+
   return (
     <div
       className="flex flex-col h-[calc(100vh-10rem)] overflow-hidden"
@@ -998,16 +783,15 @@ const ValidationContainer = <T extends string>({
                 <div className="rounded-md border h-full flex flex-col overflow-hidden">
                   <div className="flex-1 overflow-hidden w-full">
                     <div
-                      ref={scrollContainerRef}
                       className="overflow-auto max-h-[calc(100vh-320px)] w-full"
                       style={{
+                        scrollBehavior: 'smooth',
                         willChange: 'transform',
                         position: 'relative',
                         WebkitOverflowScrolling: 'touch', // Improve scroll performance on Safari
                         overscrollBehavior: 'contain', // Prevent scroll chaining
                         contain: 'paint', // Improve performance for sticky elements
                          }}
-                      onScroll={handleScroll}
                     >
                       {renderValidationTable}
                     </div>

inventory/src/components/product-import/steps/ValidationStepNew/components/ValidationTable.tsx

@@ -55,8 +55,8 @@ interface ValidationTableProps<T extends string> {
   [key: string]: any
 }
 
-// Create a memoized wrapper for template selects to prevent unnecessary re-renders
+// Simple template select component - let React handle optimization
-const MemoizedTemplateSelect = React.memo(({
+const TemplateSelectWrapper = ({
   templates,
   value,
   onValueChange,
@@ -88,84 +88,7 @@ const MemoizedTemplateSelect = React.memo(({
       defaultBrand={defaultBrand}
     />
   );
-}, (prev, next) => {
+};
-  return (
-    prev.value === next.value &&
-    prev.templates === next.templates &&
-    prev.defaultBrand === next.defaultBrand &&
-    prev.isLoading === next.isLoading
-  );
-});
-
-MemoizedTemplateSelect.displayName = 'MemoizedTemplateSelect';
-
-// Create a memoized cell component
-const MemoizedCell = React.memo(({
-  field,
-  value,
-  onChange,
-  errors,
-  isValidating,
-  fieldKey,
-  options,
-  itemNumber,
-  width,
-  rowIndex,
-  copyDown,
-  totalRows,
-  editingCells,
-  setEditingCells
-}: {
-  field: Field<string>,
-  value: any,
-  onChange: (value: any) => void,
-  errors: ErrorType[],
-  isValidating?: boolean,
-  fieldKey: string,
-  options?: readonly any[],
-  itemNumber?: string,
-  width: number,
-  rowIndex: number,
-  copyDown?: (endRowIndex?: number) => void,
-  totalRows: number,
-  editingCells: Set<string>,
-  setEditingCells: React.Dispatch<React.SetStateAction<Set<string>>>
-}) => {
-  return (
-    <ValidationCell
-      field={field}
-      value={value}
-      onChange={onChange}
-      errors={errors}
-      isValidating={isValidating}
-      fieldKey={fieldKey}
-      options={options}
-      itemNumber={itemNumber}
-      width={width}
-      rowIndex={rowIndex}
-      copyDown={copyDown}
-      totalRows={totalRows}
-      editingCells={editingCells}
-      setEditingCells={setEditingCells}
-    />
-  );
-}, (prev, next) => {
-  // For item_number cells, only re-render when itemNumber actually changes
-  if (prev.fieldKey === 'item_number') {
-    return prev.itemNumber === next.itemNumber && 
-           prev.value === next.value && 
-           prev.isValidating === next.isValidating;
-  }
-
-  // Simplified memo comparison - most expensive checks removed
-  // Note: editingCells changes are not checked here as they need immediate re-renders
-  return prev.value === next.value && 
-         prev.isValidating === next.isValidating &&
-         prev.errors === next.errors &&
-         prev.options === next.options;
-});
-
-MemoizedCell.displayName = 'MemoizedCell';
 
 const ValidationTable = <T extends string>({
   data,
@@ -194,49 +117,83 @@ const ValidationTable = <T extends string>({
 }: ValidationTableProps<T>) => {
   const { translations } = useRsi<T>();
 
-  // Add state for copy down selection mode
+  // Copy-down state combined into single object
-  const [isInCopyDownMode, setIsInCopyDownMode] = useState(false);
+  type CopyDownState = {
-  const [sourceRowIndex, setSourceRowIndex] = useState<number | null>(null);
+    sourceRowIndex: number;
-  const [sourceFieldKey, setSourceFieldKey] = useState<string | null>(null);
+    sourceFieldKey: string;
-  const [targetRowIndex, setTargetRowIndex] = useState<number | null>(null);
+    targetRowIndex: number | null;
+  };
+  const [copyDownState, setCopyDownState] = useState<CopyDownState | null>(null);
 
   // Handle copy down completion
   const handleCopyDownComplete = useCallback((sourceRowIndex: number, fieldKey: string, targetRowIndex: number) => {
-    // Call the copyDown function with the source row index, field key, and target row index
     copyDown(sourceRowIndex, fieldKey, targetRowIndex);
-    
+    setCopyDownState(null);
-    // Reset the copy down selection mode
-    setIsInCopyDownMode(false);
-    setSourceRowIndex(null);
-    setSourceFieldKey(null);
-    setTargetRowIndex(null);
   }, [copyDown]);
 
   // Create copy down context value
+  // Use a ref to track partial state during initialization
+  const partialCopyDownRef = React.useRef<{ rowIndex?: number; fieldKey?: string }>({});
+
   const copyDownContextValue = useMemo(() => ({
-    isInCopyDownMode,
+    isInCopyDownMode: copyDownState !== null,
-    sourceRowIndex,
+    sourceRowIndex: copyDownState?.sourceRowIndex ?? null,
-    sourceFieldKey,
+    sourceFieldKey: copyDownState?.sourceFieldKey ?? null,
-    targetRowIndex,
+    targetRowIndex: copyDownState?.targetRowIndex ?? null,
-    setIsInCopyDownMode,
+    setIsInCopyDownMode: (value: boolean) => {
-    setSourceRowIndex,
+      if (!value) {
-    setSourceFieldKey,
+        setCopyDownState(null);
-    setTargetRowIndex,
+        partialCopyDownRef.current = {};
+      }
+    },
+    setSourceRowIndex: (rowIndex: number | null) => {
+      if (rowIndex !== null) {
+        partialCopyDownRef.current.rowIndex = rowIndex;
+        // If we have both values, set the full state
+        if (partialCopyDownRef.current.fieldKey !== undefined) {
+          setCopyDownState({
+            sourceRowIndex: rowIndex,
+            sourceFieldKey: partialCopyDownRef.current.fieldKey,
+            targetRowIndex: null
+          });
+          partialCopyDownRef.current = {};
+        }
+      }
+    },
+    setSourceFieldKey: (fieldKey: string | null) => {
+      if (fieldKey !== null) {
+        partialCopyDownRef.current.fieldKey = fieldKey;
+        // If we have both values, set the full state
+        if (partialCopyDownRef.current.rowIndex !== undefined) {
+          setCopyDownState({
+            sourceRowIndex: partialCopyDownRef.current.rowIndex,
+            sourceFieldKey: fieldKey,
+            targetRowIndex: null
+          });
+          partialCopyDownRef.current = {};
+        }
+      }
+    },
+    setTargetRowIndex: (rowIndex: number | null) => {
+      if (copyDownState) {
+        setCopyDownState({
+          ...copyDownState,
+          targetRowIndex: rowIndex
+        });
+      }
+    },
     handleCopyDownComplete
-  }), [
+  }), [copyDownState, handleCopyDownComplete]);
-    isInCopyDownMode,
-    sourceRowIndex,
-    sourceFieldKey,
-    targetRowIndex,
-    handleCopyDownComplete
-  ]);
 
   // Update targetRowIndex when hovering over rows in copy down mode
   const handleRowMouseEnter = useCallback((rowIndex: number) => {
-    if (isInCopyDownMode && sourceRowIndex !== null && rowIndex > sourceRowIndex) {
+    if (copyDownState && copyDownState.sourceRowIndex < rowIndex) {
-      setTargetRowIndex(rowIndex);
+      setCopyDownState({
+        ...copyDownState,
+        targetRowIndex: rowIndex
+      });
     }
-  }, [isInCopyDownMode, sourceRowIndex]);
+  }, [copyDownState]);
 
   // Memoize the selection column with stable callback
   const handleSelectAll = useCallback((value: boolean, table: any) => {
@@ -290,7 +247,7 @@ const ValidationTable = <T extends string>({
       return (
         <TableCell className="p-1" style={{ width: '200px', minWidth: '200px', maxWidth: '200px', overflow: 'hidden' }}>
           <div className="w-full overflow-hidden">
-            <MemoizedTemplateSelect
+            <TemplateSelectWrapper
               templates={templates}
               value={templateValue || ''}
               onValueChange={(value) => handleTemplateChange(value, rowIndex)}
@@ -465,8 +422,8 @@ const ValidationTable = <T extends string>({
           : `cell-${row.index}-${fieldKey}`;
         
         return (
-          <MemoizedCell
+          <ValidationCell
-            key={cellKey} // CRITICAL: Add key to force re-render when itemNumber changes
+            key={cellKey}
             field={fieldWithType as Field<string>}
             value={currentValue}
             onChange={(value) => handleFieldUpdate(row.index, field.key as T, value)}
@@ -527,7 +484,7 @@ const ValidationTable = <T extends string>({
     <CopyDownContext.Provider value={copyDownContextValue}>
       <div className="min-w-max relative">
         {/* Add global styles for copy down mode */}
-        {isInCopyDownMode && (
+        {copyDownState && (
           <style>
             {`
               .copy-down-target-row,
@@ -543,7 +500,7 @@ const ValidationTable = <T extends string>({
             `}
           </style>
         )}
-        {isInCopyDownMode && sourceRowIndex !== null && sourceFieldKey !== null && (
+        {copyDownState && (
           <div className="sticky top-0 z-30 h-0 overflow-visible">
             <div 
               className="absolute w-[240px] top-16 bg-blue-50 border rounded-2xl shadow-lg border-blue-200 p-3 text-sm text-blue-700 flex items-center justify-between"
@@ -551,7 +508,7 @@ const ValidationTable = <T extends string>({
                 left: (() => {
                   // Find the column index
                   const colIndex = columns.findIndex(col =>
-                    'accessorKey' in col && col.accessorKey === sourceFieldKey
+                    'accessorKey' in col && col.accessorKey === copyDownState.sourceFieldKey
                   );
 
                   // If column not found, position at a default location
@@ -579,7 +536,7 @@ const ValidationTable = <T extends string>({
               <Button
                 variant="outline"
                 size="sm"
-                onClick={() => setIsInCopyDownMode(false)}
+                onClick={() => setCopyDownState(null)}
                 className="text-xs h-7 border-blue-200 text-blue-700 hover:bg-blue-100"
               >
                 Cancel
@@ -636,15 +593,14 @@ const ValidationTable = <T extends string>({
                                 Object.keys(validationErrors.get(parseInt(row.id)) || {}).length > 0;
 
                 // Precompute copy down target status
-                const isCopyDownTarget = isInCopyDownMode && 
+                const isCopyDownTarget = copyDownState !== null &&
-                                       sourceRowIndex !== null && 
+                                       parseInt(row.id) > copyDownState.sourceRowIndex;
-                                       parseInt(row.id) > sourceRowIndex;
 
                 // Using CSS variables for better performance on hover/state changes
                 const rowStyle = {
                   cursor: isCopyDownTarget ? 'pointer' : undefined,
                   position: 'relative' as const,
-                  willChange: isInCopyDownMode ? 'background-color' : 'auto',
+                  willChange: copyDownState ? 'background-color' : 'auto',
                   contain: 'layout',
                   transition: 'background-color 100ms ease-in-out'
                 };
@@ -677,34 +633,15 @@ const ValidationTable = <T extends string>({
   );
 };
 
-// Optimize memo comparison with more efficient checks
+// Simplified memo - React 18+ handles most optimizations well
+// Only check critical props that frequently change
 const areEqual = (prev: ValidationTableProps<any>, next: ValidationTableProps<any>) => {
-  // Check reference equality for simple props first
+  // Only prevent re-render if absolutely nothing changed (rare case)
-  if (prev.fields !== next.fields) return false;
+  return (
-  if (prev.templates !== next.templates) return false;
+    prev.data === next.data &&
-  if (prev.isLoadingTemplates !== next.isLoadingTemplates) return false;
+    prev.validationErrors === next.validationErrors &&
-  if (prev.filters?.showErrorsOnly !== next.filters?.showErrorsOnly) return false;
+    prev.rowSelection === next.rowSelection
-  
+  );
-  // Fast path: data length change always means re-render
-  if (prev.data.length !== next.data.length) return false;
-  
-  // CRITICAL: Check if data content has actually changed
-  // Simple reference equality check - if data array reference changed, re-render
-  if (prev.data !== next.data) return false;
-  
-  // Efficiently check row selection changes
-  const prevSelectionKeys = Object.keys(prev.rowSelection);
-  const nextSelectionKeys = Object.keys(next.rowSelection);
-  if (prevSelectionKeys.length !== nextSelectionKeys.length) return false;
-  if (!prevSelectionKeys.every(key => prev.rowSelection[key] === next.rowSelection[key])) return false;
-
-  // Use size for Map comparisons instead of deeper checks
-  if (prev.validationErrors.size !== next.validationErrors.size) return false;
-  if (prev.validatingCells.size !== next.validatingCells.size) return false;
-  if (prev.itemNumbers.size !== next.itemNumbers.size) return false;
-  
-  // If values haven't changed, component doesn't need to re-render
-  return true;
 };
 
 export default React.memo(ValidationTable, areEqual); 

inventory/src/components/product-import/steps/ValidationStepNew/hooks/useAiValidation.tsx

@@ -4,6 +4,7 @@ import { getApiUrl, RowData } from './validationTypes';
 import { Fields } from '../../../types';
 import { Meta } from '../types';
 import { addErrorsAndRunHooks } from '../utils/dataMutations';
+import { prepareDataForAiValidation } from '../utils/aiValidationUtils';
 import * as Diff from 'diff';
 
 // Define interfaces for AI validation
@@ -296,25 +297,8 @@ export const useAiValidation = <T extends string>(
         lastProduct: data[data.length - 1]
       });
 
-      // Build a complete row object including empty cells so API receives all fields
+      // Use utility function to prepare data for AI validation
-      const cleanedData = data.map(item => {
+      const cleanedData = prepareDataForAiValidation(data, fields);
-        const { __index, ...rest } = item as any;
-        // Ensure all known field keys are present, even if empty
-        const withAllKeys: Record<string, any> = {};
-        (fields as any[]).forEach((f) => {
-          const k = String(f.key);
-          // Preserve arrays (e.g., multi-select) as empty array if undefined
-          if (Array.isArray(rest[k])) {
-            withAllKeys[k] = rest[k];
-          } else if (rest[k] === undefined) {
-            // Use empty string to represent an empty cell
-            withAllKeys[k] = "";
-          } else {
-            withAllKeys[k] = rest[k];
-          }
-        });
-        return withAllKeys;
-      });
 
       console.log('Cleaned data sample:', {
         length: cleanedData.length,
@@ -323,7 +307,7 @@ export const useAiValidation = <T extends string>(
       });
       
       // Use POST to send products in request body
-      const response = await fetch(`${await getApiUrl()}/ai-validation/debug`, {
+      const response = await fetch(`${getApiUrl()}/ai-validation/debug`, {
         method: 'POST',
         headers: {
           'Content-Type': 'application/json',
@@ -435,22 +419,8 @@ export const useAiValidation = <T extends string>(
         });
       }, 1000) as unknown as NodeJS.Timeout;
 
-      // Build a complete row object including empty cells so API receives all fields
+      // Use utility function to prepare data for AI validation
-      const cleanedData = data.map(item => {
+      const cleanedData = prepareDataForAiValidation(data, fields);
-        const { __index, ...rest } = item as any;
-        const withAllKeys: Record<string, any> = {};
-        (fields as any[]).forEach((f) => {
-          const k = String(f.key);
-          if (Array.isArray(rest[k])) {
-            withAllKeys[k] = rest[k];
-          } else if (rest[k] === undefined) {
-            withAllKeys[k] = "";
-          } else {
-            withAllKeys[k] = rest[k];
-          }
-        });
-        return withAllKeys;
-      });
 
       console.log('Cleaned data for validation:', cleanedData);
 

inventory/src/components/product-import/steps/ValidationStepNew/hooks/useInitialValidation.tsx

@@ -0,0 +1,268 @@
+import { useState, useEffect, useRef } from 'react';
+import type { Dispatch, SetStateAction } from 'react';
+import type { Fields } from '@/components/product-import/types';
+import type { RowData } from './validationTypes';
+import { cleanPriceField } from '../utils/priceUtils';
+
+/**
+ * Custom hook for handling initial data validation
+ *
+ * Performs comprehensive validation on import data including:
+ * - Price field cleaning and formatting
+ * - Required field validation
+ * - Regex pattern validation
+ * - Batch processing for performance
+ *
+ * @param data - Array of row data to validate
+ * @param fields - Field configuration with validation rules
+ * @param setData - Function to update data after cleaning
+ * @param setValidationErrors - Function to set validation errors
+ * @param validateUniqueItemNumbers - Async function to validate uniqueness
+ * @param upcValidationComplete - Flag indicating UPC validation is done
+ * @param onComplete - Callback when validation is complete
+ */
+export function useInitialValidation<T extends string>({
+  data,
+  fields,
+  setData,
+  setValidationErrors,
+  validateUniqueItemNumbers,
+  upcValidationComplete,
+  onComplete,
+}: {
+  data: RowData<T>[];
+  fields: Fields<T>;
+  setData: (data: RowData<T>[]) => void;
+  setValidationErrors: Dispatch<SetStateAction<Map<number, Record<string, any[]>>>>;
+  validateUniqueItemNumbers: () => Promise<void>;
+  upcValidationComplete: boolean;
+  onComplete?: () => void;
+}) {
+  const hasRunRef = useRef(false);
+  const [isValidating, setIsValidating] = useState(false);
+
+  useEffect(() => {
+    // Only run once
+    if (hasRunRef.current) return;
+
+    // Wait for UPC validation to complete first
+    if (!upcValidationComplete) return;
+
+    // Handle empty dataset immediately
+    if (!data || data.length === 0) {
+      hasRunRef.current = true;
+      if (onComplete) {
+        onComplete();
+      }
+      return;
+    }
+
+    hasRunRef.current = true;
+    setIsValidating(true);
+
+    const runValidation = async () => {
+      console.log('Running initial validation...');
+
+      // Extract field groups for validation
+      const requiredFields = fields.filter((field) =>
+        field.validations?.some((v) => v.rule === 'required')
+      );
+      const regexFields = fields.filter((field) =>
+        field.validations?.some((v) => v.rule === 'regex')
+      );
+
+      console.log(`Validating ${requiredFields.length} required fields, ${regexFields.length} regex fields`);
+
+      // Dynamic batch size based on dataset size
+      const BATCH_SIZE = data.length <= 50 ? data.length : 25;
+      const totalBatches = Math.ceil(data.length / BATCH_SIZE);
+
+      // Initialize containers
+      const newData = [...data];
+      const validationErrorsTemp = new Map<number, Record<string, any[]>>();
+
+      // Process batches
+      for (let batchNum = 0; batchNum < totalBatches; batchNum++) {
+        const startIdx = batchNum * BATCH_SIZE;
+        const endIdx = Math.min(startIdx + BATCH_SIZE, data.length);
+
+        console.log(`Processing batch ${batchNum + 1}/${totalBatches} (rows ${startIdx}-${endIdx - 1})`);
+
+        // Process all rows in this batch
+        const batchPromises = [];
+
+        for (let rowIndex = startIdx; rowIndex < endIdx; rowIndex++) {
+          batchPromises.push(
+            processRow(
+              rowIndex,
+              data[rowIndex],
+              newData,
+              requiredFields,
+              regexFields,
+              validationErrorsTemp
+            )
+          );
+        }
+
+        await Promise.all(batchPromises);
+
+        // Yield to UI thread for large datasets
+        if (batchNum % 2 === 1 || data.length > 500) {
+          await new Promise((resolve) => setTimeout(resolve, data.length > 1000 ? 10 : 5));
+        }
+      }
+
+      console.log('Batch validation complete, applying results...');
+
+      // Apply validation errors
+      setValidationErrors(validationErrorsTemp);
+
+      // Apply data changes (price formatting)
+      if (JSON.stringify(data) !== JSON.stringify(newData)) {
+        setData(newData);
+      }
+
+      // Run uniqueness validation
+      console.log('Running uniqueness validation...');
+      await validateUniqueItemNumbers();
+
+      console.log('Initial validation complete');
+      setIsValidating(false);
+
+      // Notify completion
+      if (onComplete) {
+        onComplete();
+      }
+    };
+
+    runValidation().catch((error) => {
+      console.error('Error during initial validation:', error);
+      setIsValidating(false);
+      if (onComplete) {
+        onComplete();
+      }
+    });
+  }, [data, fields, setData, setValidationErrors, validateUniqueItemNumbers, upcValidationComplete, onComplete]);
+
+  return { isValidating };
+}
+
+/**
+ * Process a single row: clean data and validate
+ */
+function processRow<T extends string>(
+  rowIndex: number,
+  row: RowData<T>,
+  newData: RowData<T>[],
+  requiredFields: Fields<T>,
+  regexFields: Fields<T>,
+  validationErrorsTemp: Map<number, Record<string, any[]>>
+): Promise<void> {
+  return new Promise((resolve) => {
+    // Skip empty rows
+    if (!row) {
+      resolve();
+      return;
+    }
+
+    const fieldErrors: Record<string, any[]> = {};
+    let hasErrors = false;
+    const rowAsRecord = row as Record<string, any>;
+
+    // Clean price fields if needed
+    let needsUpdate = false;
+    let cleanedRow = row;
+
+    if (
+      rowAsRecord.msrp &&
+      typeof rowAsRecord.msrp === 'string' &&
+      (rowAsRecord.msrp.includes('$') || rowAsRecord.msrp.includes(','))
+    ) {
+      if (!needsUpdate) {
+        cleanedRow = { ...row } as RowData<T>;
+        needsUpdate = true;
+      }
+      (cleanedRow as Record<string, any>).msrp = cleanPriceField(rowAsRecord.msrp);
+    }
+
+    if (
+      rowAsRecord.cost_each &&
+      typeof rowAsRecord.cost_each === 'string' &&
+      (rowAsRecord.cost_each.includes('$') || rowAsRecord.cost_each.includes(','))
+    ) {
+      if (!needsUpdate) {
+        cleanedRow = { ...row } as RowData<T>;
+        needsUpdate = true;
+      }
+      (cleanedRow as Record<string, any>).cost_each = cleanPriceField(rowAsRecord.cost_each);
+    }
+
+    if (needsUpdate) {
+      newData[rowIndex] = cleanedRow;
+    }
+
+    // Validate required fields
+    for (const field of requiredFields) {
+      const key = String(field.key);
+      const value = row[key as keyof typeof row];
+
+      if (
+        value === undefined ||
+        value === null ||
+        value === '' ||
+        (Array.isArray(value) && value.length === 0) ||
+        (typeof value === 'object' && value !== null && Object.keys(value).length === 0)
+      ) {
+        fieldErrors[key] = [
+          {
+            message:
+              field.validations?.find((v) => v.rule === 'required')?.errorMessage ||
+              'This field is required',
+            level: 'error',
+            source: 'row',
+            type: 'required',
+          },
+        ];
+        hasErrors = true;
+      }
+    }
+
+    // Validate regex fields
+    for (const field of regexFields) {
+      const key = String(field.key);
+      const value = row[key as keyof typeof row];
+
+      // Skip empty values (handled by required validation)
+      if (value === undefined || value === null || value === '') {
+        continue;
+      }
+
+      const regexValidation = field.validations?.find((v) => v.rule === 'regex');
+      if (regexValidation) {
+        try {
+          const regex = new RegExp(regexValidation.value, regexValidation.flags);
+          if (!regex.test(String(value))) {
+            fieldErrors[key] = [
+              {
+                message: regexValidation.errorMessage,
+                level: regexValidation.level || 'error',
+                source: 'row',
+                type: 'regex',
+              },
+            ];
+            hasErrors = true;
+          }
+        } catch (error) {
+          console.error('Invalid regex in validation:', error);
+        }
+      }
+    }
+
+    // Store errors if any
+    if (hasErrors) {
+      validationErrorsTemp.set(rowIndex, fieldErrors);
+    }
+
+    resolve();
+  });
+}

inventory/src/components/product-import/steps/ValidationStepNew/hooks/useRowOperations.tsx

@@ -378,20 +378,22 @@ export const useRowOperations = <T extends string>(
   const revalidateRows = useCallback(
     async (
       rowIndexes: number[],
-      updatedFields?: { [rowIndex: number]: string[] }
+      updatedFields?: { [rowIndex: number]: string[] },
+      dataOverride?: RowData<T>[]
     ) => {
       // Process all specified rows using a single state update to avoid race conditions
       setValidationErrors((prev) => {
         let newErrors = new Map(prev);
+        const workingData = dataOverride ?? data;
 
         // Track which uniqueness fields need to be revalidated across the dataset
         const uniqueFieldsToCheck = new Set<string>();
 
         // Process each row
         for (const rowIndex of rowIndexes) {
-          if (rowIndex < 0 || rowIndex >= data.length) continue;
+          if (rowIndex < 0 || rowIndex >= workingData.length) continue;
 
-          const row = data[rowIndex];
+          const row = workingData[rowIndex];
           if (!row) continue;
 
           // If we have specific fields to update for this row
@@ -464,7 +466,7 @@ export const useRowOperations = <T extends string>(
 
         // Run per-field uniqueness checks and merge results
         if (uniqueFieldsToCheck.size > 0) {
-          newErrors = mergeUniqueErrorsForFields(newErrors, data, Array.from(uniqueFieldsToCheck));
+          newErrors = mergeUniqueErrorsForFields(newErrors, workingData, Array.from(uniqueFieldsToCheck));
         }
 
         return newErrors;

inventory/src/components/product-import/steps/ValidationStepNew/hooks/useUpcValidation.tsx

@@ -28,6 +28,7 @@ export const useUpcValidation = (
   
   // Cache for UPC validation results
   const processedUpcMapRef = useRef(new Map<string, string>());
+  const initialUpcValidationStartedRef = useRef(false);
   const initialUpcValidationDoneRef = useRef(false);
   
   // Helper to create cell key
@@ -317,13 +318,13 @@ export const useUpcValidation = (
   // Batch validate all UPCs in the data
   const validateAllUPCs = useCallback(async () => {
     // Skip if we've already done the initial validation
-    if (initialUpcValidationDoneRef.current) {
+    if (initialUpcValidationStartedRef.current) {
-      console.log('Initial UPC validation already done, skipping');
+      console.log('Initial UPC validation already in progress or complete, skipping');
       return;
     }
 
     // Mark that we've started the initial validation
-    initialUpcValidationDoneRef.current = true;
+    initialUpcValidationStartedRef.current = true;
 
     console.log('Starting initial UPC validation...');
     
@@ -345,6 +346,7 @@ export const useUpcValidation = (
     
     if (totalRows === 0) {
       setIsValidatingUpc(false);
+      initialUpcValidationDoneRef.current = true;
       return;
     }
     
@@ -442,6 +444,7 @@ export const useUpcValidation = (
     } catch (error) {
       console.error('Error in batch validation:', error);
     } finally {
+      initialUpcValidationDoneRef.current = true;
       // Make sure all validation states are cleared
       validationStateRef.current.validatingRows.clear();
       setValidatingRows(new Set());
@@ -453,10 +456,8 @@ export const useUpcValidation = (
   
   // Run initial UPC validation when data changes
   useEffect(() => {
-    // Skip if there's no data or we've already done the validation
+    if (initialUpcValidationStartedRef.current) return;
-    if (data.length === 0 || initialUpcValidationDoneRef.current) return;
 
-    // Run validation
     validateAllUPCs();
   }, [data, validateAllUPCs]);
 

inventory/src/components/product-import/steps/ValidationStepNew/hooks/useValidationState.tsx

@@ -11,41 +11,10 @@ import { useTemplateManagement } from "./useTemplateManagement";
 import { useFilterManagement } from "./useFilterManagement";
 import { useUniqueItemNumbersValidation } from "./useUniqueItemNumbersValidation";
 import { useUpcValidation } from "./useUpcValidation";
+import { useInitialValidation } from "./useInitialValidation";
 import { Props, RowData } from "./validationTypes";
-
+import { normalizeCountryCode } from "../utils/countryUtils";
-// Country normalization helper (common mappings) - function declaration for hoisting
+import { cleanPriceField } from "../utils/priceUtils";
-function normalizeCountryCode(input: string): string | null {
-  if (!input) return null;
-  const s = input.trim();
-  const upper = s.toUpperCase();
-  if (/^[A-Z]{2}$/.test(upper)) return upper; // already 2-letter
-  const iso3to2: Record<string, string> = {
-    USA: "US", GBR: "GB", UK: "GB", CHN: "CN", DEU: "DE", FRA: "FR", ITA: "IT", ESP: "ES",
-    CAN: "CA", MEX: "MX", AUS: "AU", NZL: "NZ", JPN: "JP", KOR: "KR", PRK: "KP", TWN: "TW",
-    VNM: "VN", THA: "TH", IDN: "ID", IND: "IN", BRA: "BR", ARG: "AR", CHL: "CL", PER: "PE",
-    ZAF: "ZA", RUS: "RU", UKR: "UA", NLD: "NL", BEL: "BE", CHE: "CH", SWE: "SE", NOR: "NO",
-    DNK: "DK", POL: "PL", AUT: "AT", PRT: "PT", GRC: "GR", CZE: "CZ", HUN: "HU", IRL: "IE",
-    ISR: "IL", PAK: "PK", BGD: "BD", PHL: "PH", MYS: "MY", SGP: "SG", HKG: "HK", MAC: "MO"
-  };
-  if (iso3to2[upper]) return iso3to2[upper];
-  const nameMap: Record<string, string> = {
-    "UNITED STATES": "US", "UNITED STATES OF AMERICA": "US", "AMERICA": "US", "U.S.": "US", "U.S.A": "US", "USA": "US",
-    "UNITED KINGDOM": "GB", "UK": "GB", "GREAT BRITAIN": "GB", "ENGLAND": "GB",
-    "CHINA": "CN", "PEOPLE'S REPUBLIC OF CHINA": "CN", "PRC": "CN",
-    "CANADA": "CA", "MEXICO": "MX", "JAPAN": "JP", "SOUTH KOREA": "KR", "KOREA, REPUBLIC OF": "KR",
-    "TAIWAN": "TW", "VIETNAM": "VN", "THAILAND": "TH", "INDONESIA": "ID", "INDIA": "IN",
-    "GERMANY": "DE", "FRANCE": "FR", "ITALY": "IT", "SPAIN": "ES", "NETHERLANDS": "NL", "BELGIUM": "BE",
-    "SWITZERLAND": "CH", "SWEDEN": "SE", "NORWAY": "NO", "DENMARK": "DK", "POLAND": "PL", "AUSTRIA": "AT",
-    "PORTUGAL": "PT", "GREECE": "GR", "CZECH REPUBLIC": "CZ", "CZECHIA": "CZ", "HUNGARY": "HU", "IRELAND": "IE",
-    "RUSSIA": "RU", "UKRAINE": "UA", "AUSTRALIA": "AU", "NEW ZEALAND": "NZ",
-    "BRAZIL": "BR", "ARGENTINA": "AR", "CHILE": "CL", "PERU": "PE", "SOUTH AFRICA": "ZA",
-    "ISRAEL": "IL", "PAKISTAN": "PK", "BANGLADESH": "BD", "PHILIPPINES": "PH", "MALAYSIA": "MY", "SINGAPORE": "SG",
-    "HONG KONG": "HK", "MACAU": "MO"
-  };
-  const normalizedName = s.replace(/\./g, "").trim().toUpperCase();
-  if (nameMap[normalizedName]) return nameMap[normalizedName];
-  return null;
-}
 
 export const useValidationState = <T extends string>({
   initialData,
@@ -69,22 +38,12 @@ export const useValidationState = <T extends string>({
     return initialData.map((row) => {
       const updatedRow = { ...row } as Record<string, any>;
 
-      // Clean MSRP
+      // Clean price fields using utility
-      if (typeof updatedRow.msrp === "string") {
+      if (updatedRow.msrp !== undefined) {
-        updatedRow.msrp = updatedRow.msrp.replace(/[$,]/g, "");
+        updatedRow.msrp = cleanPriceField(updatedRow.msrp);
-        const numValue = parseFloat(updatedRow.msrp);
-        if (!isNaN(numValue)) {
-          updatedRow.msrp = numValue.toFixed(2);
-        }
-      }
-
-      // Clean cost_each
-      if (typeof updatedRow.cost_each === "string") {
-        updatedRow.cost_each = updatedRow.cost_each.replace(/[$,]/g, "");
-        const numValue = parseFloat(updatedRow.cost_each);
-        if (!isNaN(numValue)) {
-          updatedRow.cost_each = numValue.toFixed(2);
       }
+      if (updatedRow.cost_each !== undefined) {
+        updatedRow.cost_each = cleanPriceField(updatedRow.cost_each);
       }
 
       // Set default tax category if not already set
@@ -126,7 +85,6 @@ export const useValidationState = <T extends string>({
   const [rowSelection, setRowSelection] = useState<RowSelectionState>({});
 
   // Validation state
-  const [isValidating] = useState(false);
   const [validationErrors, setValidationErrors] = useState<
     Map<number, Record<string, any[]>>
   >(new Map());
@@ -141,8 +99,8 @@ export const useValidationState = <T extends string>({
   const [editingCells, setEditingCells] = useState<Set<string>>(new Set());
   const hasEditingCells = editingCells.size > 0;
 
-  const initialValidationDoneRef = useRef(false);
+  // Track initial validation lifecycle
-  // isValidatingRef unused; remove to satisfy TS
+  const [initialValidationComplete, setInitialValidationComplete] = useState(false);
   // Track last seen item_number signature to drive targeted uniqueness checks
   const lastItemNumberSigRef = useRef<string | null>(null);
 
@@ -302,252 +260,33 @@ export const useValidationState = <T extends string>({
     },
     [data, onBack, onNext, validationErrors]
   );
+  const { isValidating: isInitialValidationRunning } = useInitialValidation<T>({
+    data,
+    fields,
+    setData,
+    setValidationErrors,
+    validateUniqueItemNumbers,
+    upcValidationComplete: upcValidation.initialValidationDone,
+    onComplete: () => {
+      setInitialValidationComplete(true);
+    },
+  });
 
-  // Initialize validation once, after initial UPC-based item number generation completes
   useEffect(() => {
-    if (initialValidationDoneRef.current) return;
+    if (initialValidationComplete) return;
-    // Wait for initial UPC validation to finish to avoid double work and ensure
+    if ((!data || data.length === 0) && upcValidation.initialValidationDone) {
-    // item_number values are in place before uniqueness checks
+      setInitialValidationComplete(true);
-    if (!upcValidation.initialValidationDone) return;
-
-    const runCompleteValidation = async () => {
-      if (!data || data.length === 0) return;
-
-      console.log("Running complete validation...");
-
-      // Get required fields
-      const requiredFields = fields.filter((field) =>
-        field.validations?.some((v) => v.rule === "required")
-      );
-      console.log(`Found ${requiredFields.length} required fields`);
-
-      // Get fields that have regex validation
-      const regexFields = fields.filter((field) =>
-        field.validations?.some((v) => v.rule === "regex")
-      );
-      console.log(`Found ${regexFields.length} fields with regex validation`);
-
-      // Get fields that need uniqueness validation
-      const uniqueFields = fields.filter((field) =>
-        field.validations?.some((v) => v.rule === "unique")
-      );
-      console.log(
-        `Found ${uniqueFields.length} fields requiring uniqueness validation`
-      );
-
-      // Dynamic batch size based on dataset size
-      const BATCH_SIZE = data.length <= 50 ? data.length : 25; // Process all at once for small datasets
-      const totalRows = data.length;
-
-      // Initialize new data for any modifications
-      const newData = [...data];
-
-      // Create a temporary Map to collect all validation errors
-      const validationErrorsTemp = new Map<
-        number,
-        Record<string, any[]>
-      >();
-
-      // Variables for batching
-      let currentBatch = 0;
-      const totalBatches = Math.ceil(totalRows / BATCH_SIZE);
-
-      const processBatch = async () => {
-        // Calculate batch range
-        const startIdx = currentBatch * BATCH_SIZE;
-        const endIdx = Math.min(startIdx + BATCH_SIZE, totalRows);
-        console.log(
-          `Processing batch ${
-            currentBatch + 1
-          }/${totalBatches} (rows ${startIdx} to ${endIdx - 1})`
-        );
-
-        // Process rows in this batch
-        const batchPromises: Promise<void>[] = [];
-
-        for (let rowIndex = startIdx; rowIndex < endIdx; rowIndex++) {
-          batchPromises.push(
-            new Promise<void>((resolve) => {
-              const row = data[rowIndex];
-
-              // Skip if row is empty or undefined
-              if (!row) {
-                resolve();
-                return;
     }
+  }, [data, initialValidationComplete, upcValidation.initialValidationDone]);
 
-              // Store field errors for this row
+  const hasPendingUpcValidation = upcValidation.validatingRows.size > 0;
-              const fieldErrors: Record<string, any[]> = {};
+  const isInitializing =
-              let hasErrors = false;
+    !initialValidationComplete ||
+    isInitialValidationRunning ||
+    templateManagement.isLoadingTemplates ||
+    hasPendingUpcValidation;
 
-              // Check if price fields need formatting
+  const isValidating = isInitialValidationRunning;
-              const rowAsRecord = row as Record<string, any>;
-              let mSrpNeedsProcessing = false;
-              let costEachNeedsProcessing = false;
-
-              if (
-                rowAsRecord.msrp &&
-                typeof rowAsRecord.msrp === "string" &&
-                (rowAsRecord.msrp.includes("$") ||
-                  rowAsRecord.msrp.includes(","))
-              ) {
-                mSrpNeedsProcessing = true;
-              }
-
-              if (
-                rowAsRecord.cost_each &&
-                typeof rowAsRecord.cost_each === "string" &&
-                (rowAsRecord.cost_each.includes("$") ||
-                  rowAsRecord.cost_each.includes(","))
-              ) {
-                costEachNeedsProcessing = true;
-              }
-
-              // Process price fields if needed
-              if (mSrpNeedsProcessing || costEachNeedsProcessing) {
-                // Create a clean copy only if needed
-                const cleanedRow = { ...row } as Record<string, any>;
-
-                if (mSrpNeedsProcessing) {
-                  const msrpValue = rowAsRecord.msrp.replace(/[$,]/g, "");
-                  const numValue = parseFloat(msrpValue);
-                  cleanedRow.msrp = !isNaN(numValue)
-                    ? numValue.toFixed(2)
-                    : msrpValue;
-                }
-
-                if (costEachNeedsProcessing) {
-                  const costValue = rowAsRecord.cost_each.replace(/[$,]/g, "");
-                  const numValue = parseFloat(costValue);
-                  cleanedRow.cost_each = !isNaN(numValue)
-                    ? numValue.toFixed(2)
-                    : costValue;
-                }
-
-                newData[rowIndex] = cleanedRow as RowData<T>;
-              }
-
-              // Validate required fields
-              for (const field of requiredFields) {
-                const key = String(field.key);
-                const value = row[key as keyof typeof row];
-
-                // Skip non-required empty fields
-                if (
-                  value === undefined ||
-                  value === null ||
-                  value === "" ||
-                  (Array.isArray(value) && value.length === 0) ||
-                  (typeof value === "object" &&
-                    value !== null &&
-                    Object.keys(value).length === 0)
-                ) {
-                  // Add error for empty required fields
-                  fieldErrors[key] = [
-                    {
-                      message:
-                        field.validations?.find((v) => v.rule === "required")
-                          ?.errorMessage || "This field is required",
-                      level: "error",
-                      source: "row",
-                      type: "required",
-                    },
-                  ];
-                  hasErrors = true;
-                }
-              }
-
-              // Validate regex fields - even if they have data
-              for (const field of regexFields) {
-                const key = String(field.key);
-                const value = row[key as keyof typeof row];
-
-                // Skip empty values as they're handled by required validation
-                if (value === undefined || value === null || value === "") {
-                  continue;
-                }
-
-                // Find regex validation
-                const regexValidation = field.validations?.find(
-                  (v) => v.rule === "regex"
-                );
-                if (regexValidation) {
-                  try {
-                    // Check if value matches regex
-                    const regex = new RegExp(
-                      regexValidation.value,
-                      regexValidation.flags
-                    );
-                    if (!regex.test(String(value))) {
-                      // Add regex validation error
-                      fieldErrors[key] = [
-                        {
-                          message: regexValidation.errorMessage,
-                          level: regexValidation.level || "error",
-                          source: "row",
-                          type: "regex",
-                        },
-                      ];
-                      hasErrors = true;
-                    }
-                  } catch (error) {
-                    console.error("Invalid regex in validation:", error);
-                  }
-                }
-              }
-
-              // Update validation errors for this row
-              if (hasErrors) {
-                validationErrorsTemp.set(rowIndex, fieldErrors);
-              }
-
-              resolve();
-            })
-          );
-        }
-
-        // Wait for all row validations to complete
-        await Promise.all(batchPromises);
-      };
-
-      const processAllBatches = async () => {
-        for (let batch = 0; batch < totalBatches; batch++) {
-          currentBatch = batch;
-          await processBatch();
-
-          // Yield to UI thread more frequently for large datasets
-          if (batch % 2 === 1 || totalRows > 500) {
-            await new Promise((resolve) => setTimeout(resolve, totalRows > 1000 ? 10 : 5));
-          }
-        }
-
-        // All batches complete
-        console.log("All initial validation batches complete");
-
-        // Apply collected validation errors all at once
-        setValidationErrors(validationErrorsTemp);
-
-        // Apply any data changes (like price formatting)
-        if (JSON.stringify(data) !== JSON.stringify(newData)) {
-          setData(newData);
-        }
-
-        // Run uniqueness validation after the basic validation
-        validateUniqueItemNumbers();
-
-        // Mark that initial validation is done
-        initialValidationDoneRef.current = true;
-
-        console.log("Initial validation complete");
-      };
-
-      // Start the validation process
-      processAllBatches();
-    };
-
-    // Run the complete validation
-    runCompleteValidation();
-  }, [data, fields, setData, setValidationErrors, validateUniqueItemNumbers, upcValidation.initialValidationDone]);
 
   // Targeted uniqueness revalidation: run only when item_number values change
   useEffect(() => {
@@ -693,6 +432,7 @@ export const useValidationState = <T extends string>({
 
     // Validation
     isValidating,
+    isInitializing,
     validationErrors,
     rowValidationStatus,
     validateRow,
@@ -730,6 +470,9 @@ export const useValidationState = <T extends string>({
     getTemplateDisplayText,
     refreshTemplates: templateManagement.refreshTemplates,
 
+    // UPC validation
+    upcValidation,
+
     // Filters
     filters: filterManagement.filters,
     filterFields: filterManagement.filterFields,

inventory/src/components/product-import/steps/ValidationStepNew/utils/aiValidationUtils.ts

@@ -0,0 +1,107 @@
+/**
+ * AI Validation utility functions
+ *
+ * Helper functions for processing AI validation data and managing progress
+ */
+
+import type { Fields } from '@/components/product-import/types';
+
+/**
+ * Clean data for AI validation by including all fields
+ *
+ * Ensures every field is present in the data sent to the API,
+ * converting undefined values to empty strings
+ */
+export function prepareDataForAiValidation<T extends string>(
+  data: any[],
+  fields: Fields<T>
+): Record<string, any>[] {
+  return data.map(item => {
+    const { __index, ...rest } = item;
+    const withAllKeys: Record<string, any> = {};
+
+    fields.forEach((f) => {
+      const k = String(f.key);
+      if (Array.isArray(rest[k])) {
+        withAllKeys[k] = rest[k];
+      } else if (rest[k] === undefined) {
+        withAllKeys[k] = "";
+      } else {
+        withAllKeys[k] = rest[k];
+      }
+    });
+
+    return withAllKeys;
+  });
+}
+
+/**
+ * Process AI-corrected data to handle multi-select and select fields
+ *
+ * Converts comma-separated strings to arrays for multi-select fields
+ * and handles label-to-value conversions for select fields
+ */
+export function processAiCorrectedData<T extends string>(
+  correctedData: any[],
+  originalData: any[],
+  fields: Fields<T>
+): any[] {
+  return correctedData.map((corrected: any, index: number) => {
+    // Start with original data to preserve metadata like __index
+    const original = originalData[index] || {};
+    const processed = { ...original, ...corrected };
+
+    // Process each field according to its type
+    Object.keys(processed).forEach(key => {
+      if (key.startsWith('__')) return; // Skip metadata fields
+
+      const fieldConfig = fields.find(f => String(f.key) === key);
+      if (!fieldConfig) return;
+
+      // Handle multi-select fields (comma-separated values → array)
+      if (fieldConfig?.fieldType.type === 'multi-select' && typeof processed[key] === 'string') {
+        processed[key] = processed[key]
+          .split(',')
+          .map((v: string) => v.trim())
+          .filter(Boolean);
+      }
+    });
+
+    return processed;
+  });
+}
+
+/**
+ * Calculate progress percentage based on elapsed time and estimates
+ *
+ * @param step - Current step number (1-5)
+ * @param elapsedSeconds - Time elapsed since start
+ * @param estimatedSeconds - Estimated total time (optional)
+ * @returns Progress percentage (0-95, never reaches 100 until complete)
+ */
+export function calculateProgressPercent(
+  step: number,
+  elapsedSeconds: number,
+  estimatedSeconds?: number
+): number {
+  if (estimatedSeconds && estimatedSeconds > 0) {
+    // Time-based progress
+    return Math.min(95, (elapsedSeconds / estimatedSeconds) * 100);
+  }
+
+  // Step-based progress with time adjustment
+  const baseProgress = (step / 5) * 100;
+  const timeAdjustment = step === 1 ? Math.min(20, elapsedSeconds * 0.5) : 0;
+  return Math.min(95, baseProgress + timeAdjustment);
+}
+
+/**
+ * Extract base status message by removing time information
+ *
+ * Removes patterns like "(5s remaining)" or "(1m 30s elapsed)"
+ */
+export function extractBaseStatus(status: string): string {
+  return status
+    .replace(/\s\(\d+[ms].+\)$/, '')
+    .replace(/\s\(\d+m \d+s.+\)$/, '');
+}

inventory/src/components/product-import/steps/ValidationStepNew/utils/countryUtils.ts

@@ -0,0 +1,66 @@
+/**
+ * Country code normalization utilities
+ *
+ * Converts various country code formats and country names to ISO 3166-1 alpha-2 codes
+ */
+
+/**
+ * Normalizes country codes and names to ISO 3166-1 alpha-2 format (2-letter codes)
+ *
+ * Supports:
+ * - ISO 3166-1 alpha-2 codes (e.g., "US", "GB")
+ * - ISO 3166-1 alpha-3 codes (e.g., "USA", "GBR")
+ * - Common country names (e.g., "United States", "China")
+ *
+ * @param input - Country code or name to normalize
+ * @returns ISO 3166-1 alpha-2 code or null if not recognized
+ *
+ * @example
+ * normalizeCountryCode("USA") // "US"
+ * normalizeCountryCode("United States") // "US"
+ * normalizeCountryCode("US") // "US"
+ * normalizeCountryCode("invalid") // null
+ */
+export function normalizeCountryCode(input: string): string | null {
+  if (!input) return null;
+
+  const s = input.trim();
+  const upper = s.toUpperCase();
+
+  // Already in ISO 3166-1 alpha-2 format
+  if (/^[A-Z]{2}$/.test(upper)) return upper;
+
+  // ISO 3166-1 alpha-3 to alpha-2 mapping
+  const iso3to2: Record<string, string> = {
+    USA: "US", GBR: "GB", UK: "GB", CHN: "CN", DEU: "DE", FRA: "FR", ITA: "IT", ESP: "ES",
+    CAN: "CA", MEX: "MX", AUS: "AU", NZL: "NZ", JPN: "JP", KOR: "KR", PRK: "KP", TWN: "TW",
+    VNM: "VN", THA: "TH", IDN: "ID", IND: "IN", BRA: "BR", ARG: "AR", CHL: "CL", PER: "PE",
+    ZAF: "ZA", RUS: "RU", UKR: "UA", NLD: "NL", BEL: "BE", CHE: "CH", SWE: "SE", NOR: "NO",
+    DNK: "DK", POL: "PL", AUT: "AT", PRT: "PT", GRC: "GR", CZE: "CZ", HUN: "HU", IRL: "IE",
+    ISR: "IL", PAK: "PK", BGD: "BD", PHL: "PH", MYS: "MY", SGP: "SG", HKG: "HK", MAC: "MO"
+  };
+
+  if (iso3to2[upper]) return iso3to2[upper];
+
+  // Country name to ISO 3166-1 alpha-2 mapping
+  const nameMap: Record<string, string> = {
+    "UNITED STATES": "US", "UNITED STATES OF AMERICA": "US", "AMERICA": "US", "U.S.": "US", "U.S.A": "US", "USA": "US",
+    "UNITED KINGDOM": "GB", "UK": "GB", "GREAT BRITAIN": "GB", "ENGLAND": "GB",
+    "CHINA": "CN", "PEOPLE'S REPUBLIC OF CHINA": "CN", "PRC": "CN",
+    "CANADA": "CA", "MEXICO": "MX", "JAPAN": "JP", "SOUTH KOREA": "KR", "KOREA, REPUBLIC OF": "KR",
+    "TAIWAN": "TW", "VIETNAM": "VN", "THAILAND": "TH", "INDONESIA": "ID", "INDIA": "IN",
+    "GERMANY": "DE", "FRANCE": "FR", "ITALY": "IT", "SPAIN": "ES", "NETHERLANDS": "NL", "BELGIUM": "BE",
+    "SWITZERLAND": "CH", "SWEDEN": "SE", "NORWAY": "NO", "DENMARK": "DK", "POLAND": "PL", "AUSTRIA": "AT",
+    "PORTUGAL": "PT", "GREECE": "GR", "CZECH REPUBLIC": "CZ", "CZECHIA": "CZ", "HUNGARY": "HU", "IRELAND": "IE",
+    "RUSSIA": "RU", "UKRAINE": "UA", "AUSTRALIA": "AU", "NEW ZEALAND": "NZ",
+    "BRAZIL": "BR", "ARGENTINA": "AR", "CHILE": "CL", "PERU": "PE", "SOUTH AFRICA": "ZA",
+    "ISRAEL": "IL", "PAKISTAN": "PK", "BANGLADESH": "BD", "PHILIPPINES": "PH", "MALAYSIA": "MY", "SINGAPORE": "SG",
+    "HONG KONG": "HK", "MACAU": "MO"
+  };
+
+  // Normalize input: remove dots, trim, uppercase
+  const normalizedName = s.replace(/\./g, "").trim().toUpperCase();
+  if (nameMap[normalizedName]) return nameMap[normalizedName];
+
+  return null;
+}

inventory/src/components/product-import/steps/ValidationStepNew/utils/priceUtils.ts

@@ -0,0 +1,62 @@
+/**
+ * Price field cleaning and formatting utilities
+ */
+
+/**
+ * Cleans a price field by removing currency symbols and formatting to 2 decimal places
+ *
+ * - Removes dollar signs ($) and commas (,)
+ * - Converts to number and formats with 2 decimal places
+ * - Returns original value if conversion fails
+ *
+ * @param value - Price value to clean (string or number)
+ * @returns Cleaned price string formatted to 2 decimals, or original value if invalid
+ *
+ * @example
+ * cleanPriceField("$1,234.56") // "1234.56"
+ * cleanPriceField("$99.9") // "99.90"
+ * cleanPriceField(123.456) // "123.46"
+ * cleanPriceField("invalid") // "invalid"
+ */
+export function cleanPriceField(value: string | number): string {
+  if (typeof value === "string") {
+    const cleaned = value.replace(/[$,]/g, "");
+    const numValue = parseFloat(cleaned);
+    if (!isNaN(numValue)) {
+      return numValue.toFixed(2);
+    }
+    return value;
+  }
+
+  if (typeof value === "number") {
+    return value.toFixed(2);
+  }
+
+  return String(value);
+}
+
+/**
+ * Cleans multiple price fields in a data object
+ *
+ * @param data - Object containing price fields
+ * @param priceFields - Array of field keys to clean
+ * @returns New object with cleaned price fields
+ *
+ * @example
+ * cleanPriceFields({ msrp: "$99.99", cost_each: "$50.00" }, ["msrp", "cost_each"])
+ * // { msrp: "99.99", cost_each: "50.00" }
+ */
+export function cleanPriceFields<T extends Record<string, any>>(
+  data: T,
+  priceFields: (keyof T)[]
+): T {
+  const cleaned = { ...data };
+
+  for (const field of priceFields) {
+    if (cleaned[field] !== undefined && cleaned[field] !== null) {
+      cleaned[field] = cleanPriceField(cleaned[field]) as any;
+    }
+  }
+
+  return cleaned;
+}

inventory/src/pages/Import.tsx

@@ -1,5 +1,5 @@
 import { useState } from "react";
-import { ReactSpreadsheetImport, ErrorLevel, StepType } from "@/components/product-import";
+import { ReactSpreadsheetImport, StepType } from "@/components/product-import";
 import { Button } from "@/components/ui/button";
 import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
 import { Code } from "@/components/ui/code";
@@ -9,342 +9,10 @@ import { useQuery } from "@tanstack/react-query";
 import config from "@/config";
 import { Loader2 } from "lucide-react";
 import type { DataValue, FieldType, Result } from "@/components/product-import/types";
-// Define base fields without dynamic options
+import { BASE_IMPORT_FIELDS, type ImportFieldKey } from "@/components/product-import/config";
-const BASE_IMPORT_FIELDS = [
-  {
-    label: "Supplier",
-    key: "supplier",
-    description: "Primary supplier/manufacturer of the product",
-    fieldType: {
-      type: "select" as const,
-      options: [], // Will be populated from API
-    },
-    width: 220,
-    validations: [{ rule: "required" as const, errorMessage: "Required", level: "error" as ErrorLevel }],
-  },
-  {
-    label: "Company",
-    key: "company",
-    description: "Company/Brand name",
-    fieldType: {
-      type: "select",
-      options: [], // Will be populated from API
-    },
-    width: 220,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "Line",
-    key: "line",
-    description: "Product line",
-    alternateMatches: ["collection"],
-    fieldType: {
-      type: "select" as const,
-      options: [], // Will be populated dynamically based on company selection
-    },
-    width: 220,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "Sub Line",
-    key: "subline",
-    description: "Product sub-line",
-    fieldType: {
-      type: "select" as const,
-      options: [], // Will be populated dynamically based on line selection
-    },
-    width: 220,
-  },
-  {
-    label: "UPC",
-    key: "upc",
-    description: "Universal Product Code/Barcode",
-    alternateMatches: ["barcode", "bar code", "jan", "ean", "upc code"],
-    fieldType: { type: "input" },
-    width: 145,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "unique", errorMessage: "Must be unique", level: "error" },
-      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Item Number",
-    key: "item_number",
-    description: "Internal item reference number",
-    fieldType: { type: "input" },
-    width: 130,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "unique", errorMessage: "Must be unique", level: "error" },
-    ],
-  },
-  {
-    label: "Supplier #",
-    key: "supplier_no",
-    description: "Supplier's product identifier",
-    alternateMatches: ["sku", "item#", "mfg item #", "item", "supplier #"],
-    fieldType: { type: "input" },
-    width: 130,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "unique", errorMessage: "Must be unique", level: "error" },
-    ],
-  },
-  {
-    label: "Notions #",
-    key: "notions_no",
-    description: "Internal notions number",
-    alternateMatches: ["notions #","nmc"],
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "unique", errorMessage: "Must be unique", level: "error" },
-      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Name",
-    key: "name",
-    description: "Product name/title",
-    alternateMatches: ["sku description","product name"],
-    fieldType: { type: "input" },
-    width: 500,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "unique", errorMessage: "Must be unique", level: "error" },
-    ],
-  },
-  {
-    label: "MSRP",
-    key: "msrp",
-    description: "Manufacturer's Suggested Retail Price",
-    alternateMatches: ["retail", "retail price", "sugg retail", "price", "sugg. retail","default price"],
-    fieldType: { 
-      type: "input",
-      price: true 
-    },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Min Qty",
-    key: "qty_per_unit",
-    description: "Quantity of items per individual unit",
-    alternateMatches: ["inner pack", "inner", "min qty", "unit qty", "min. order qty", "supplier qty/unit"],
-    fieldType: { type: "input" },
-    width: 80,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Cost Each",
-    key: "cost_each",
-    description: "Wholesale cost per unit",
-    alternateMatches: ["wholesale", "wholesale price", "supplier cost each", "cost each"],
-    fieldType: { 
-      type: "input",
-      price: true 
-    },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Case Pack",
-    key: "case_qty",
-    description: "Number of units per case",
-    alternateMatches: ["mc qty","case qty","case pack","box ct"],
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "regex", value: "^[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Tax Category",
-    key: "tax_cat",
-    description: "Product tax category",
-    fieldType: {
-      type: "select",
-      options: [], // Will be populated from API
-    },
-    width: 200,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "Artist",
-    key: "artist",
-    description: "Artist/Designer name",
-    fieldType: {
-      type: "select",
-      options: [], // Will be populated from API
-    },
-    width: 200,
-  },
-  {
-    label: "ETA Date",
-    key: "eta",
-    description: "Estimated arrival date",
-    alternateMatches: ["shipping month"],
-    fieldType: { type: "input" },
-    width: 120,
-  },
-  {
-    label: "Weight",
-    key: "weight",
-    description: "Product weight (in lbs)",
-    alternateMatches: ["weight (lbs.)"],
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Length",
-    key: "length",
-    description: "Product length (in inches)",
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Width",
-    key: "width",
-    description: "Product width (in inches)",
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Height",
-    key: "height",
-    description: "Product height (in inches)",
-    fieldType: { type: "input" },
-    width: 100,
-    validations: [
-      { rule: "required", errorMessage: "Required", level: "error" },
-      { rule: "regex", value: "^[0-9]*.?[0-9]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Shipping Restrictions",
-    key: "ship_restrictions",
-    description: "Product shipping restrictions",
-    fieldType: {
-      type: "select",
-      options: [], // Will be populated from API
-    },
-    width: 190,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "COO",
-    key: "coo",
-    description: "2-letter country code (ISO)",
-    alternateMatches: ["coo", "country of origin"],
-    fieldType: { type: "input" },
-    width: 70,
-    validations: [
-      { rule: "regex", value: "^[A-Z]{2}$", errorMessage: "Must be 2 letters", level: "error" },
-    ],
-  },
-  {
-    label: "HTS Code",
-    key: "hts_code",
-    description: "Harmonized Tariff Schedule code",
-    alternateMatches: ["taric","hts"],
-    fieldType: { type: "input" },
-    width: 130,
-    validations: [
-      { rule: "regex", value: "^[0-9.]+$", errorMessage: "Must be a number", level: "error" },
-    ],
-  },
-  {
-    label: "Size Category",
-    key: "size_cat",
-    description: "Product size category",
-    fieldType: {
-      type: "select",
-      options: [], // Will be populated from API
-    },
-    width: 180,
-  },
-  {
-    label: "Description",
-    key: "description",
-    description: "Detailed product description",
-    alternateMatches: ["details/description"],
-    fieldType: { 
-      type: "input",
-      multiline: true 
-    },
-    width: 500,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "Private Notes",
-    key: "priv_notes",
-    description: "Internal notes about the product",
-    fieldType: { 
-      type: "input",
-      multiline: true 
-    },
-    width: 300,
-  },
-  {
-    label: "Categories",
-    key: "categories",
-    description: "Product categories",
-    fieldType: {
-      type: "multi-select",
-      options: [], // Will be populated from API
-    },
-    width: 350,
-    validations: [{ rule: "required", errorMessage: "Required", level: "error" }],
-  },
-  {
-    label: "Themes",
-    key: "themes",
-    description: "Product themes/styles",
-    fieldType: {
-      type: "multi-select",
-      options: [], // Will be populated from API
-    },
-    width: 300,
-  },
-  {
-    label: "Colors",
-    key: "colors",
-    description: "Product colors",
-    fieldType: {
-      type: "multi-select",
-      options: [], // Will be populated from API
-    },
-    width: 200,
-  },
-] as const;
-
 
 export function Import() {
   const [isOpen, setIsOpen] = useState(false);
-  type ImportFieldKey = (typeof BASE_IMPORT_FIELDS)[number]["key"];
   type NormalizedProduct = Record<ImportFieldKey | "product_images", string | string[] | boolean | null>;
   type ImportResult = Result<string> & { all?: Result<string>["validData"] };