More validation table optimizations + create doc to track remaining fixes

docs/validate-table-changes.md

@@ -0,0 +1,303 @@
+# Current Issues to Address
+1. The red row background should go away when all cells in the row are valid and all required cells are populated
+2. Columns alignment with header is slightly off, gets worse the further right you go
+3. The copy down button is in the way of the validation error icon and the select open trigger - all three need to be in unique locations
+4. Validation isn't happening beyond checking if a cell is required or not - needs to respect rules in import.tsx
+    * Red cell outline if cell is required and it's empty
+    * Red outline + alert circle icon with tooltip if cell is NOT empty and isn't valid
+5. Description column needs to have an expanded view of some sort, maybe a popover to allow for easier editing
+    * Don't distort table to make it happen
+6. Need to ensure all cell's contents don't overflow the input (truncate). COO does this currently, probably more
+7. The template select cell is expanding, needs to be fixed size and truncate
+8. When you enter a value in 2+ cells before validation finishes, contents from all edited cells get erased when validation finishes
+9. Import dialog state not fully reset when closing? (validate data step appears scrolled to the middle of the table where I left it)
+10. UPC column doesn't need to show loading state when Item Number is being processed, only show on item number column
+11. Copy down needs to show a loading state on the cells that it will copy to
+12. Shipping restrictions/tax category should default to ID 0 if we didn't get it elsewhere
+13. Header row should be sticky (both up/down and left/right)
+14. Need a way to scroll around table if user doesn't have mouse wheel for left/right
+15. Need to remove all artificial virtualization, batching, artificial delays, and caching. Adds too much complexity and data set is not ever large enough for this to be helpful. Keep actual performance optimizations.
+
+## Do NOT change or edit
+* Anything related to AI validation
+* Anything about how templates or UPC validation work (only focus on specific issues described above)
+* Anything outside of the ValidationStepNew folder
+
+---------
+
+# Validation Step Components Overview
+
+## Core Components
+
+### ValidationContainer
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/ValidationContainer.tsx`
+- Main wrapper component for the validation step
+- Manages global state and coordinates between subcomponents
+- Handles navigation events (next, back)
+- Manages template application and validation state
+- Coordinates UPC validation and product line loading
+- Manages row selection and filtering
+- Contains cache management for UPC validation results
+- Maintains item number references separate from main data
+
+### ValidationTable
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/ValidationTable.tsx`
+- Handles data display and column configuration
+- Uses TanStack Table for core functionality
+- Features:
+  - Sticky header (both vertical and horizontal) - currently doesn't work properly
+  - Row selection with checkboxes
+  - Template selection column
+  - Dynamic column widths based on field types - specified in import.tsx component
+  - Copy down functionality for cell values
+  - Error highlighting for rows and cells
+  - Loading states for cells being validated
+
+### ValidationCell
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/ValidationCell.tsx`
+- Base cell component that renders different cell types based on field configuration
+- Handles error display with tooltips
+- Manages copy down button visibility
+- Supports loading states during validation
+- Cell Types:
+  1. InputCell: For single-value text input
+  2. SelectCell: For dropdown selection
+  3. MultiInputCell: For multiple value inputs
+  4. Template selection cells with SearchableTemplateSelect component
+
+### SearchableTemplateSelect
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/SearchableTemplateSelect.tsx`
+- Advanced template selection component with search functionality
+- Features:
+  - Real-time search filtering of templates
+  - Customizable display text for templates
+  - Support for default brand selection
+  - Accessible popover interface
+  - Keyboard navigation support
+  - Custom styling through className props
+  - Scroll event handling for nested scrollable areas
+
+### TemplateManager
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/TemplateManager.tsx`
+- Comprehensive template management interface
+- Features:
+  - Template selection with search functionality
+  - Save template dialog with name and type inputs
+  - Batch template application to selected rows
+  - Template count tracking
+  - Toast notifications for user feedback
+  - Dialog-based interface for template operations
+
+### AiValidationDialogs
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/AiValidationDialogs.tsx`
+- Manages AI-assisted validation dialogs and interactions
+
+### SaveTemplateDialog
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/SaveTemplateDialog.tsx`
+- Dialog component for saving new templates
+
+## Cell Components
+
+### InputCell
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/InputCell.tsx`
+- Handles single value text input
+- Features:
+  - Inline/edit mode switching
+  - Multiline support
+  - Price formatting
+  - Error state display
+  - Loading state during validation
+  - Width constraints
+  - Automated cleanPriceFields processing for "$" formatting
+
+### SelectCell
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/SelectCell.tsx`
+- Handles dropdown selection
+- Features:
+  - Searchable dropdown
+  - Custom option rendering
+  - Error state display
+  - Loading state during validation
+  - Width constraints
+  - Disabled state support
+  - Deferred search query handling for performance
+
+### MultiInputCell
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/MultiInputCell.tsx`
+- Handles multiple value inputs
+- Features:
+  - Comma-separated input support
+  - Multi-select dropdown for predefined options
+  - Custom separators
+  - Badge display for selected count
+  - Truncation for long values
+  - Width constraints
+  - Price formatting support
+  - Internal state management to avoid excessive re-renders
+
+## Validation System
+
+### useValidation Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useValidation.tsx`
+- Provides core validation logic
+- Validates at multiple levels:
+  1. Field-level validation (required, regex, unique)
+  2. Row-level validation (supplier, company fields)
+  3. Table-level validation
+  4. Custom validation hooks support
+- Error object structure includes message, level, and source properties
+- Handles debounced validation updates to avoid UI freezing
+
+### useAiValidation Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useAiValidation.tsx`
+- Manages AI-assisted validation logic and state
+- Features:
+  - Tracks detailed changes per product
+  - Manages validation progress with estimated completion time
+  - Handles warnings and change suggestions
+  - Supports diff generation for changes
+  - Progress tracking with step indicators
+  - Prompt management for AI interactions
+  - Timer management for long-running operations
+
+### useTemplates Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useTemplates.tsx`
+- Comprehensive template management system
+- Features:
+  - Template CRUD operations
+  - Template application logic
+  - Default value handling
+  - Template search and filtering
+  - Batch template operations
+  - Template validation
+
+### useUpcValidation Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useUpcValidation.tsx`
+- Dedicated UPC validation management
+- Features:
+  - UPC format validation
+  - Supplier data validation
+  - Cache management for validation results
+  - Batch processing of UPC validations
+  - Item number generation logic
+  - Loading state management
+
+### useFilters Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useFilters.tsx`
+- Advanced filtering system for table data
+- Features:
+  - Multiple filter criteria support
+  - Dynamic filter updates
+  - Filter persistence
+  - Filter combination logic
+  - Performance optimized filtering
+
+### useValidationState Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useValidationState.tsx`
+- Manages global validation state
+- Handles:
+  - Data updates
+  - Template management
+  - Error tracking using Map objects
+  - Row selection
+  - Filtering
+  - UPC validation with caching to prevent duplicate API calls
+  - Product line loading
+  - Batch processing of updates
+  - Default value application for tax_cat and ship_restrictions (defaulting to "0")
+  - Price field auto-formatting to remove "$" symbols
+
+### Utility Files
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/utils/validationUtils.ts`
+- Core validation utility functions
+- Includes:
+  - Field validation logic
+  - Error message formatting
+  - Validation rule processing
+  - Type checking utilities
+
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/utils/errorUtils.ts`
+- Error handling and formatting utilities
+- Includes:
+  - Error object creation
+  - Error message formatting
+  - Error source tracking
+  - Error level management
+
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/utils/dataMutations.ts`
+- Data transformation and mutation utilities
+- Includes:
+  - Row data updates
+  - Batch data processing
+  - Data structure conversions
+  - Change tracking
+
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/utils/validation-helper.js`
+- Helper functions for validation
+- Includes:
+  - Common validation patterns
+  - Validation state management
+  - Validation result processing
+
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/utils/upcValidation.ts`
+- UPC-specific validation utilities
+- Includes:
+  - UPC format checking
+  - Checksum validation
+  - Supplier data matching
+  - Cache management
+
+### Types
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/types.ts`
+- Core type definitions for the validation step
+
+### Validation Types
+1. Required field validation
+2. Regex pattern validation
+3. Unique value validation
+4. Custom field validation
+5. Row-level validation
+6. Table-level validation
+
+## State Management
+
+### useValidationState Hook
+`inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useValidationState.tsx`
+- Manages global validation state
+- Handles:
+  - Data updates
+  - Template management
+  - Error tracking using Map objects
+  - Row selection
+  - Filtering
+  - UPC validation with caching to prevent duplicate API calls
+  - Product line loading
+  - Batch processing of updates
+  - Default value application for tax_cat and ship_restrictions (defaulting to "0")
+  - Price field auto-formatting to remove "$" symbols
+
+## UPC Validation System
+
+### UPC Processing
+- Validates UPCs against supplier data
+- Cache system for UPC validation results
+- Batch processing of UPC validation requests
+- Auto-generation of item numbers based on UPC
+- Special loading states for UPC/item number fields
+- Separate state tracking to avoid unnecessary data structure updates
+
+## Template System
+
+### Template Management
+- Supports saving and loading templates
+- Template application to single/multiple rows
+- Default template values
+- Template search and filtering
+
+## Performance Optimizations
+1. Memoized components to prevent unnecessary renders
+2. Virtualized table for large datasets
+3. Deferred value updates for search inputs
+4. Efficient error state management
+5. Optimized cell update handling
+

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/ValidationCell.tsx

@@ -19,6 +19,14 @@ type ErrorObject = {
   source?: string;
 }
 
+// Helper function to check if a value is empty - utility function shared by all components
+const isEmpty = (val: any): boolean => 
+  val === undefined || 
+  val === null || 
+  val === '' || 
+  (Array.isArray(val) && val.length === 0) ||
+  (typeof val === 'object' && !Array.isArray(val) && Object.keys(val).length === 0);
+
 // Memoized validation icon component
 const ValidationIcon = React.memo(({ error }: { error: ErrorObject }) => (
   <TooltipProvider>
@@ -101,11 +109,17 @@ const BaseCellContent = React.memo(({
     />
   );
 }, (prev, next) => {
+  // Shallow array comparison for options if arrays
+  const optionsEqual = prev.options === next.options || 
+    (Array.isArray(prev.options) && Array.isArray(next.options) && 
+     prev.options.length === next.options.length &&
+     prev.options.every((opt, idx) => opt === (next.options as any[])[idx]));
+    
   return (
     prev.value === next.value &&
     prev.hasErrors === next.hasErrors &&
     prev.field === next.field &&
-    JSON.stringify(prev.options) === JSON.stringify(next.options)
+    optionsEqual
   );
 });
 
@@ -125,6 +139,82 @@ export interface ValidationCellProps {
   copyDown?: () => void
 }
 
+// Add efficient error message extraction function
+const getErrorMessage = (error: ErrorObject): string => error.message;
+
+// Add a utility function to process errors with appropriate caching
+function processErrors(value: any, errors: ErrorObject[]): {
+  filteredErrors: ErrorObject[];
+  hasError: boolean;
+  isRequiredButEmpty: boolean;
+  shouldShowErrorIcon: boolean;
+  errorMessages: string;
+} {
+  // Fast path - if no errors, return immediately
+  if (!errors || errors.length === 0) {
+    return {
+      filteredErrors: [],
+      hasError: false,
+      isRequiredButEmpty: false,
+      shouldShowErrorIcon: false,
+      errorMessages: ''
+    };
+  }
+  
+  // Check if value is empty - using local function for speed
+  const valueIsEmpty = value === undefined || 
+    value === null || 
+    value === '' || 
+    (Array.isArray(value) && value.length === 0) ||
+    (typeof value === 'object' && !Array.isArray(value) && Object.keys(value).length === 0);
+  
+  // If not empty, filter out required errors
+  // Create a new array only if we need to filter (avoid unnecessary allocations)
+  let filteredErrors: ErrorObject[];
+  let hasRequiredError = false;
+  
+  if (valueIsEmpty) {
+    // For empty values, check if there are required errors
+    hasRequiredError = errors.some(error => 
+      error.message?.toLowerCase().includes('required')
+    );
+    filteredErrors = errors;
+  } else {
+    // For non-empty values, filter out required errors
+    filteredErrors = errors.filter(error => 
+      !error.message?.toLowerCase().includes('required')
+    );
+  }
+  
+  // Determine if any actual errors exist after filtering
+  const hasError = filteredErrors.some(error => 
+    error.level === 'error' || error.level === 'warning'
+  );
+  
+  // Check if field is required but empty
+  const isRequiredButEmpty = valueIsEmpty && hasRequiredError;
+  
+  // Only show error icons for non-empty fields with actual errors
+  const shouldShowErrorIcon = hasError && !valueIsEmpty;
+  
+  // Get error messages for the tooltip - only if we need to show icon
+  let errorMessages = '';
+  if (shouldShowErrorIcon) {
+    errorMessages = filteredErrors
+      .filter(e => e.level === 'error' || e.level === 'warning')
+      .map(getErrorMessage)
+      .join('\n');
+  }
+  
+  return {
+    filteredErrors,
+    hasError,
+    isRequiredButEmpty,
+    shouldShowErrorIcon,
+    errorMessages
+  };
+}
+
 const ItemNumberCell = React.memo(({ 
   value, 
   itemNumber, 
@@ -144,34 +234,19 @@ const ItemNumberCell = React.memo(({
   onChange: (value: any) => void,
   copyDown?: () => void
 }) => {
-  // Helper function to check if a value is empty
-  const isEmpty = (val: any): boolean => 
-    val === undefined || 
-    val === null || 
-    val === '' || 
-    (Array.isArray(val) && val.length === 0) ||
-    (typeof val === 'object' && !Array.isArray(val) && Object.keys(val).length === 0);
-
   // If we have a value or itemNumber, ignore "required" errors
   const displayValue = itemNumber || value;
-  const filteredErrors = !isEmpty(displayValue) 
-    ? errors.filter(error => !error.message?.toLowerCase().includes('required'))
-    : errors;
   
-  // Determine if the field has an error after filtering
+  // Use the utility function to process errors once
-  const hasError = filteredErrors.some(error => error.level === 'error' || error.level === 'warning');
+  const {
-  
+    hasError,
-  // Determine if the field is required but empty
+    isRequiredButEmpty,
-  const isRequiredButEmpty = isEmpty(displayValue) && 
+    shouldShowErrorIcon,
-    errors.some(error => error.message?.toLowerCase().includes('required'));
+    errorMessages
-  
+  } = React.useMemo(() => 
-  // Only show error icons for non-empty fields with actual errors (not just required errors)
+    processErrors(displayValue, errors),
-  const shouldShowErrorIcon = hasError && !isEmpty(displayValue);
+    [displayValue, errors]
-  
+  );
-  // Get error messages for the tooltip
-  const errorMessages = shouldShowErrorIcon 
-    ? filteredErrors.filter(e => e.level === 'error' || e.level === 'warning').map(e => e.message).join('\n')
-    : '';
   
   return (
     <TableCell className="p-1 group relative" style={{ width: `${width}px`, minWidth: `${width}px` }}>
@@ -188,7 +263,7 @@ const ItemNumberCell = React.memo(({
               value={displayValue}
               onChange={onChange}
               hasErrors={hasError || isRequiredButEmpty}
-              options={[]}
+              options={(field.fieldType && typeof field.fieldType === 'object' && (field.fieldType as any).options) || []}
             />
           </div>
         )}
@@ -226,7 +301,7 @@ const ItemNumberCell = React.memo(({
   prev.value === next.value &&
   prev.itemNumber === next.itemNumber &&
   prev.isValidating === next.isValidating &&
-  JSON.stringify(prev.errors) === JSON.stringify(next.errors)
+  compareErrorArrays(prev.errors || [], next.errors || [])
 ));
 
 ItemNumberCell.displayName = 'ItemNumberCell';
@@ -241,7 +316,6 @@ const ValidationCell = ({
   options = [],
   itemNumber,
   width,
-  rowIndex,
   copyDown}: ValidationCellProps) => {
   // For item_number fields, use the specialized component
   if (fieldKey === 'item_number') {
@@ -259,19 +333,15 @@ const ValidationCell = ({
     );
   }
 
-  // Helper function to check if a value is empty
+  // Memoize filtered errors to avoid recalculation on every render
-  const isEmpty = (val: any): boolean => 
+  const filteredErrors = React.useMemo(() => {
-    val === undefined || 
+    return !isEmpty(value) 
-    val === null || 
-    val === '' || 
-    (Array.isArray(val) && val.length === 0) ||
-    (typeof val === 'object' && !Array.isArray(val) && Object.keys(val).length === 0);
-
-  // If we have a value, ignore "required" errors
-  const filteredErrors = !isEmpty(value) 
       ? errors.filter(error => !error.message?.toLowerCase().includes('required'))
       : errors;
+  }, [value, errors]);
   
+  // Memoize error state derivations
+  const { hasError, isRequiredButEmpty, shouldShowErrorIcon, errorMessages } = React.useMemo(() => {
     // Determine if the field has an error after filtering
     const hasError = filteredErrors.some(error => error.level === 'error' || error.level === 'warning');
     
@@ -287,15 +357,12 @@ const ValidationCell = ({
       ? filteredErrors.filter(e => e.level === 'error' || e.level === 'warning').map(e => e.message).join('\n')
       : '';
       
+    return { hasError, isRequiredButEmpty, shouldShowErrorIcon, errorMessages };
+  }, [filteredErrors, value, errors]);
+  
   // Check if this is a multiline field
-  const isMultiline = typeof field.fieldType === 'object' && 
-                      (field.fieldType.type === 'input' || field.fieldType.type === 'multi-input') && 
-                      field.fieldType.multiline === true;
   
   // Check for price field
-  const isPrice = typeof field.fieldType === 'object' && 
-                 (field.fieldType.type === 'input' || field.fieldType.type === 'multi-input') && 
-                 field.fieldType.price === true;
   
   return (
     <TableCell className="p-1 group relative" style={{ width: `${width}px`, minWidth: `${width}px` }}>
@@ -349,42 +416,61 @@ const ValidationCell = ({
 };
 
 export default React.memo(ValidationCell, (prev, next) => {
-  // Deep comparison of errors
-  const prevErrorsStr = JSON.stringify(prev.errors);
-  const nextErrorsStr = JSON.stringify(next.errors);
-  
-  // Deep comparison of options
-  const prevOptionsStr = JSON.stringify(prev.options);
-  const nextOptionsStr = JSON.stringify(next.options);
-  
   // For validating cells, always re-render
   if (prev.isValidating !== next.isValidating) {
     return false;
   }
   
-  // For item numbers, check if the item number changed
+  // Quick reference equality checks first for better performance
-  if (prev.fieldKey === 'item_number') {
+  if (prev.value !== next.value || prev.width !== next.width) {
-    return (
+    return false;
-      prev.value === next.value &&
-      prev.itemNumber === next.itemNumber &&
-      prevErrorsStr === nextErrorsStr
-    );
   }
   
-  // For select and multi-select fields, check if options changed
+  // Check for error arrays equality - avoid JSON.stringify
+  const errorsEqual = compareErrorArrays(prev.errors || [], next.errors || []);
+  if (!errorsEqual) return false;
+  
+  // Check options only when needed
   if (prev.field.fieldType.type === 'select' || prev.field.fieldType.type === 'multi-select') {
-    return (
+    if (prev.options !== next.options) {
-      prev.value === next.value &&
+      // Use safe defaults for options to handle undefined
-      prevErrorsStr === nextErrorsStr &&
+      const prevOpts = prev.options || [];
-      // Only do the deep comparison if the references are different
+      const nextOpts = next.options || [];
-      (prev.options === next.options || prevOptionsStr === nextOptionsStr)
+      
-    );
+      // Only do shallow comparison if references are different
+      if (prevOpts.length !== nextOpts.length) return false;
+      
+      // Quick length check before detailed comparison
+      for (let i = 0; i < prevOpts.length; i++) {
+        if (prevOpts[i] !== nextOpts[i]) return false;
+      }
+    }
   }
   
-  // For all other fields, check if value or errors changed
+  // For item numbers, check itemNumber equality
-  return (
+  if (prev.fieldKey === 'item_number' && prev.itemNumber !== next.itemNumber) {
-    prev.value === next.value &&
+    return false;
-    prevErrorsStr === nextErrorsStr &&
+  }
-    prev.width === next.width
+  
-  );
+  // If we got this far, the props are equal
+  return true;
 });
+
+// Helper function to compare error arrays efficiently
+function compareErrorArrays(prevErrors: ErrorObject[], nextErrors: ErrorObject[]): boolean {
+  if (prevErrors === nextErrors) return true;
+  if (prevErrors.length !== nextErrors.length) return false;
+  
+  for (let i = 0; i < prevErrors.length; i++) {
+    const prevError = prevErrors[i];
+    const nextError = nextErrors[i];
+    
+    if (prevError.message !== nextError.message || 
+        prevError.level !== nextError.level ||
+        prevError.source !== nextError.source) {
+      return false;
+    }
+  }
+  
+  return true;
+} 

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/ValidationTable.tsx

@@ -49,6 +49,106 @@ interface ValidationTableProps<T extends string> {
   [key: string]: any
 }
 
+// Create a memoized wrapper for template selects to prevent unnecessary re-renders
+const MemoizedTemplateSelect = React.memo(({
+  templates,
+  value,
+  onValueChange,
+  getTemplateDisplayText,
+  defaultBrand,
+  isLoading
+}: {
+  templates: Template[],
+  value: string,
+  onValueChange: (value: string) => void,
+  getTemplateDisplayText: (value: string | null) => string,
+  defaultBrand?: string,
+  isLoading?: boolean
+}) => {
+  if (isLoading) {
+    return (
+      <Button variant="outline" className="w-full justify-between" disabled>
+        <Loader2 className="h-4 w-4 mr-2 animate-spin" />
+        Loading...
+      </Button>
+    );
+  }
+
+  return (
+    <SearchableTemplateSelect
+      templates={templates}
+      value={value}
+      onValueChange={onValueChange}
+      getTemplateDisplayText={getTemplateDisplayText}
+      defaultBrand={defaultBrand}
+    />
+  );
+}, (prev, next) => {
+  return (
+    prev.value === next.value &&
+    prev.templates === next.templates &&
+    prev.defaultBrand === next.defaultBrand &&
+    prev.isLoading === next.isLoading
+  );
+});
+
+MemoizedTemplateSelect.displayName = 'MemoizedTemplateSelect';
+
+// Create a memoized cell component
+const MemoizedCell = React.memo(({
+  field,
+  value,
+  onChange,
+  errors,
+  isValidating,
+  fieldKey,
+  options,
+  itemNumber,
+  width,
+  rowIndex,
+  copyDown
+}: {
+  field: Field<string>,
+  value: any,
+  onChange: (value: any) => void,
+  errors: ErrorType[],
+  isValidating?: boolean,
+  fieldKey: string,
+  options?: readonly any[],
+  itemNumber?: string,
+  width: number,
+  rowIndex: number,
+  copyDown?: () => void
+}) => {
+  return (
+    <ValidationCell
+      field={field}
+      value={value}
+      onChange={onChange}
+      errors={errors}
+      isValidating={isValidating}
+      fieldKey={fieldKey}
+      options={options}
+      itemNumber={itemNumber}
+      width={width}
+      rowIndex={rowIndex}
+      copyDown={copyDown}
+    />
+  );
+}, (prev, next) => {
+  // Only re-render if these essential props change
+  return (
+    prev.value === next.value &&
+    prev.isValidating === next.isValidating &&
+    prev.itemNumber === next.itemNumber &&
+    // Deep compare errors
+    prev.errors === next.errors &&
+    prev.options === next.options
+  );
+});
+
+MemoizedCell.displayName = 'MemoizedCell';
+
 const ValidationTable = <T extends string>({
   data,
   fields,
@@ -118,25 +218,35 @@ const ValidationTable = <T extends string>({
       
       return (
         <TableCell className="p-1" style={{ width: '200px', minWidth: '200px' }}>
-          {isLoadingTemplates ? (
+          <MemoizedTemplateSelect
-            <Button variant="outline" className="w-full justify-between" disabled>
-              <Loader2 className="h-4 w-4 mr-2 animate-spin" />
-              Loading...
-            </Button>
-          ) : (
-            <SearchableTemplateSelect
             templates={templates}
             value={templateValue || ''}
             onValueChange={(value) => handleTemplateChange(value, rowIndex)}
             getTemplateDisplayText={getTemplateDisplayText}
             defaultBrand={defaultBrand}
+            isLoading={isLoadingTemplates}
           />
-          )}
         </TableCell>
       );
     }
   }), [templates, handleTemplateChange, getTemplateDisplayText, isLoadingTemplates, data]);
 
+  // Cache options by field key to avoid recreating arrays
+  const optionsCache = useMemo(() => {
+    const cache = new Map<string, readonly any[]>();
+    
+    fields.forEach((field) => {
+      if (field.disabled) return;
+      
+      if (field.fieldType.type === 'select' || field.fieldType.type === 'multi-select') {
+        const fieldKey = String(field.key);
+        cache.set(fieldKey, (field.fieldType as any).options || []);
+      }
+    });
+    
+    return cache;
+  }, [fields]);
+
   // Memoize the field update handler
   const handleFieldUpdate = useCallback((rowIndex: number, fieldKey: T, value: any) => {
     updateRow(rowIndex, fieldKey, value);
@@ -160,19 +270,23 @@ const ValidationTable = <T extends string>({
       150
     );
 
+    const fieldKey = String(field.key);
+    // Get cached options for this field
+    const fieldOptions = optionsCache.get(fieldKey) || [];
+
     return {
-      accessorKey: String(field.key),
+      accessorKey: fieldKey,
-      header: field.label || String(field.key),
+      header: field.label || fieldKey,
       size: fieldWidth,
       cell: ({ row }) => (
-        <ValidationCell
+        <MemoizedCell
           field={field}
           value={row.original[field.key]}
           onChange={(value) => handleFieldUpdate(row.index, field.key, value)}
-          errors={validationErrors.get(row.index)?.[String(field.key)] || []}
+          errors={validationErrors.get(row.index)?.[fieldKey] || []}
           isValidating={validatingCells.has(`${row.index}-${field.key}`)}
-          fieldKey={String(field.key)}
+          fieldKey={fieldKey}
-          options={(field.fieldType as any).options || []}
+          options={fieldOptions}
           itemNumber={itemNumbers.get(row.index)}
           width={fieldWidth}
           rowIndex={row.index}
@@ -181,7 +295,7 @@ const ValidationTable = <T extends string>({
       )
     };
   }).filter((col): col is ColumnDef<RowData<T>, any> => col !== null), 
-  [fields, validationErrors, validatingCells, itemNumbers, handleFieldUpdate, handleCopyDown]);
+  [fields, validationErrors, validatingCells, itemNumbers, handleFieldUpdate, handleCopyDown, optionsCache]);
 
   // Combine columns
   const columns = useMemo(() => [selectionColumn, templateColumn, ...fieldColumns], [selectionColumn, templateColumn, fieldColumns]);
@@ -269,7 +383,7 @@ const ValidationTable = <T extends string>({
   );
 };
 
-// Optimize memo comparison
+// Optimize memo comparison with more efficient checks
 const areEqual = (prev: ValidationTableProps<any>, next: ValidationTableProps<any>) => {
   // Check reference equality for simple props first
   if (prev.fields !== next.fields) return false;
@@ -277,34 +391,21 @@ const areEqual = (prev: ValidationTableProps<any>, next: ValidationTableProps<an
   if (prev.isLoadingTemplates !== next.isLoadingTemplates) return false;
   if (prev.filters?.showErrorsOnly !== next.filters?.showErrorsOnly) return false;
   
-  // Check data length and content
+  // Fast path: data length change always means re-render
   if (prev.data.length !== next.data.length) return false;
   
-  // Check row selection changes
+  // Efficiently check row selection changes
   const prevSelectionKeys = Object.keys(prev.rowSelection);
   const nextSelectionKeys = Object.keys(next.rowSelection);
   if (prevSelectionKeys.length !== nextSelectionKeys.length) return false;
   if (!prevSelectionKeys.every(key => prev.rowSelection[key] === next.rowSelection[key])) return false;
 
-  // Check validation errors
+  // Use size for Map comparisons instead of deeper checks
   if (prev.validationErrors.size !== next.validationErrors.size) return false;
-  for (const [key, value] of prev.validationErrors) {
-    const nextValue = next.validationErrors.get(key);
-    if (!nextValue || Object.keys(value).length !== Object.keys(nextValue).length) return false;
-  }
-
-  // Check validating cells
   if (prev.validatingCells.size !== next.validatingCells.size) return false;
-  for (const cell of prev.validatingCells) {
-    if (!next.validatingCells.has(cell)) return false;
-  }
-
-  // Check item numbers
   if (prev.itemNumbers.size !== next.itemNumbers.size) return false;
-  for (const [key, value] of prev.itemNumbers) {
-    if (next.itemNumbers.get(key) !== value) return false;
-  }
   
+  // If values haven't changed, component doesn't need to re-render
   return true;
 };
 

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/InputCell.tsx

@@ -1,4 +1,4 @@
-import React, { useState, useCallback, useDeferredValue, useTransition } from 'react'
+import React, { useState, useCallback, useDeferredValue, useTransition, useRef, useEffect } from 'react'
 import { Field } from '../../../../types'
 import { Input } from '@/components/ui/input'
 import { Textarea } from '@/components/ui/textarea'
@@ -16,6 +16,20 @@ interface InputCellProps<T extends string> {
   disabled?: boolean
 }
 
+// Add efficient price formatting utility
+const formatPrice = (value: string): string => {
+  // Remove any non-numeric characters except decimal point
+  const numericValue = value.replace(/[^\d.]/g, '');
+  
+  // Parse as float and format to 2 decimal places
+  const numValue = parseFloat(numericValue);
+  if (!isNaN(numValue)) {
+    return numValue.toFixed(2);
+  }
+  
+  return numericValue;
+};
+
 const InputCell = <T extends string>({
   value,
   onChange,
@@ -26,66 +40,81 @@ const InputCell = <T extends string>({
   isPrice = false,
   disabled = false
 }: InputCellProps<T>) => {
-  const [isEditing, setIsEditing] = useState(false)
+  const [isEditing, setIsEditing] = useState(false);
-  const [editValue, setEditValue] = useState('')
+  const [editValue, setEditValue] = useState('');
-  const [isPending, startTransition] = useTransition()
+  const [isPending, startTransition] = useTransition();
-  const deferredEditValue = useDeferredValue(editValue)
+  const deferredEditValue = useDeferredValue(editValue);
+  
+  // Use a ref to track if we need to process the value
+  const needsProcessingRef = useRef(false);
+  
+  // Efficiently handle price formatting without multiple rerenders
+  useEffect(() => {
+    if (isPrice && needsProcessingRef.current && !isEditing) {
+      needsProcessingRef.current = false;
+      
+      // Do price processing only when needed
+      const formattedValue = formatPrice(value);
+      if (formattedValue !== value) {
+        onChange(formattedValue);
+      }
+    }
+  }, [value, isPrice, isEditing, onChange]);
   
   // Handle focus event - optimized to be synchronous
   const handleFocus = useCallback(() => {
-    setIsEditing(true)
+    setIsEditing(true);
     
     // For price fields, strip formatting when focusing
-    if (isPrice && value !== undefined && value !== null) {
+    if (value !== undefined && value !== null) {
+      if (isPrice) {
         // Remove any non-numeric characters except decimal point
-      const numericValue = String(value).replace(/[^\d.]/g, '')
+        const numericValue = String(value).replace(/[^\d.]/g, '');
-      setEditValue(numericValue)
+        setEditValue(numericValue);
       } else {
-      setEditValue(value !== undefined && value !== null ? String(value) : '')
+        setEditValue(String(value));
+      }
+    } else {
+      setEditValue('');
     }
     
-    onStartEdit?.()
+    onStartEdit?.();
-  }, [value, onStartEdit, isPrice])
+  }, [value, onStartEdit, isPrice]);
   
   // Handle blur event - use transition for non-critical updates
   const handleBlur = useCallback(() => {
     startTransition(() => {
-      setIsEditing(false)
+      setIsEditing(false);
       
       // Format the value for storage (remove formatting like $ for price)
-      let processedValue = deferredEditValue
+      let processedValue = deferredEditValue.trim();
       
-      if (isPrice) {
+      if (isPrice && processedValue) {
-        // Remove any non-numeric characters except decimal point
+        needsProcessingRef.current = true;
-        processedValue = deferredEditValue.replace(/[^\d.]/g, '')
-        
-        // Parse as float and format to 2 decimal places to ensure valid number
-        const numValue = parseFloat(processedValue)
-        if (!isNaN(numValue)) {
-          processedValue = numValue.toFixed(2)
-        }
       }
       
-      onChange(processedValue)
+      onChange(processedValue);
-      onEndEdit?.()
+      onEndEdit?.();
-    })
+    });
-  }, [deferredEditValue, onChange, onEndEdit, isPrice])
+  }, [deferredEditValue, onChange, onEndEdit, isPrice]);
   
   // Handle direct input change - optimized to be synchronous for typing
   const handleChange = useCallback((e: React.ChangeEvent<HTMLInputElement | HTMLTextAreaElement>) => {
-    const newValue = isPrice ? e.target.value.replace(/[$,]/g, '') : e.target.value
+    const newValue = isPrice ? e.target.value.replace(/[$,]/g, '') : e.target.value;
-    setEditValue(newValue)
+    setEditValue(newValue);
-  }, [isPrice])
+  }, [isPrice]);
   
-  // Format price value for display - memoized and deferred
+  // Display value with efficient memoization
   const displayValue = useDeferredValue(
     isPrice && value ? 
-      parseFloat(String(value).replace(/[^\d.]/g, '')).toFixed(2) : 
+      typeof value === 'number' ? value.toFixed(2) :
+      typeof value === 'string' && /^-?\d+(\.\d+)?$/.test(value) ? parseFloat(value).toFixed(2) : 
+      value : 
       value ?? ''
-  )
+  );
   
   // Add outline even when not in focus
-  const outlineClass = "border focus-visible:ring-0 focus-visible:ring-offset-0"
+  const outlineClass = "border focus-visible:ring-0 focus-visible:ring-offset-0";
   
   // If disabled, just render the value without any interactivity
   if (disabled) {
@@ -148,11 +177,30 @@ const InputCell = <T extends string>({
 
 // Optimize memo comparison to focus on essential props
 export default React.memo(InputCell, (prev, next) => {
-  if (prev.isEditing !== next.isEditing) return false;
   if (prev.hasErrors !== next.hasErrors) return false;
   if (prev.isMultiline !== next.isMultiline) return false;
   if (prev.isPrice !== next.isPrice) return false;
-  // Only check value if not editing
+  if (prev.disabled !== next.disabled) return false;
-  if (!prev.isEditing && prev.value !== next.value) return false;
+  
+  // Only check value if not editing (to avoid expensive rerender during editing)
+  if (prev.value !== next.value) {
+    // For price values, do a more intelligent comparison
+    if (prev.isPrice) {
+      // Convert both to numeric values for comparison
+      const prevNum = typeof prev.value === 'number' ? prev.value : 
+                    typeof prev.value === 'string' ? parseFloat(prev.value) : 0;
+      const nextNum = typeof next.value === 'number' ? next.value : 
+                    typeof next.value === 'string' ? parseFloat(next.value) : 0;
+                    
+      // Only update if the actual numeric values differ
+      if (!isNaN(prevNum) && !isNaN(nextNum) && 
+          Math.abs(prevNum - nextNum) > 0.001) {
+        return false;
+      }
+    } else {
+      return false;
+    }
+  }
+  
   return true;
 }); 

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/MultiInputCell.tsx

@@ -32,6 +32,129 @@ interface MultiInputCellProps<T extends string> {
 // Add global CSS to ensure fixed width constraints - use !important to override other styles
 const fixedWidthClass = "!w-full !min-w-0 !max-w-full !flex-shrink-1 !flex-grow-0";
 
+// Memoized option item to prevent unnecessary renders for large option lists
+const OptionItem = React.memo(({
+  option,
+  isSelected,
+  onSelect
+}: {
+  option: FieldOption,
+  isSelected: boolean,
+  onSelect: (value: string) => void
+}) => (
+  <CommandItem
+    key={option.value}
+    value={option.value}
+    onSelect={() => onSelect(option.value)}
+    className="flex w-full"
+  >
+    <div className="flex items-center w-full overflow-hidden">
+      <Check
+        className={cn(
+          "mr-2 h-4 w-4 flex-shrink-0",
+          isSelected ? "opacity-100" : "opacity-0"
+        )}
+      />
+      <span className="truncate w-full">{option.label}</span>
+    </div>
+  </CommandItem>
+), (prev, next) => {
+  return prev.option.value === next.option.value && 
+         prev.isSelected === next.isSelected;
+});
+
+OptionItem.displayName = 'OptionItem';
+
+// Create a virtualized list component for large option lists
+const VirtualizedOptions = React.memo(({
+  options,
+  selectedValues,
+  onSelect,
+  maxHeight = 200
+}: {
+  options: FieldOption[],
+  selectedValues: Set<string>,
+  onSelect: (value: string) => void,
+  maxHeight?: number
+}) => {
+  const listRef = useRef<HTMLDivElement>(null);
+  
+  // Only render visible options for better performance with large lists
+  const [visibleOptions, setVisibleOptions] = useState<FieldOption[]>([]);
+  const [scrollPosition, setScrollPosition] = useState(0);
+  
+  // Constants for virtualization
+  const itemHeight = 32; // Height of each option item in pixels
+  const visibleCount = Math.ceil(maxHeight / itemHeight) + 2; // Number of visible items + buffer
+  
+  // Handle scroll events
+  const handleScroll = useCallback(() => {
+    if (listRef.current) {
+      setScrollPosition(listRef.current.scrollTop);
+    }
+  }, []);
+  
+  // Update visible options based on scroll position
+  useEffect(() => {
+    if (options.length <= visibleCount) {
+      // If fewer options than visible count, just show all
+      setVisibleOptions(options);
+      return;
+    }
+    
+    // Calculate start and end indices
+    const startIndex = Math.floor(scrollPosition / itemHeight);
+    const endIndex = Math.min(startIndex + visibleCount, options.length);
+    
+    // Update visible options
+    setVisibleOptions(options.slice(Math.max(0, startIndex), endIndex));
+  }, [options, scrollPosition, visibleCount, itemHeight]);
+  
+  // If fewer than the threshold, render all directly
+  if (options.length <= 100) {
+    return (
+      <div ref={listRef} className="max-h-[200px] overflow-y-auto" onScroll={handleScroll}>
+        {options.map(option => (
+          <OptionItem
+            key={option.value}
+            option={option}
+            isSelected={selectedValues.has(option.value)}
+            onSelect={onSelect}
+          />
+        ))}
+      </div>
+    );
+  }
+  
+  return (
+    <div 
+      ref={listRef} 
+      className="max-h-[200px] overflow-y-auto" 
+      onScroll={handleScroll} 
+      style={{ height: `${Math.min(maxHeight, options.length * itemHeight)}px` }}
+    >
+      <div style={{ height: `${options.length * itemHeight}px`, position: 'relative' }}>
+        <div style={{ 
+          position: 'absolute', 
+          top: `${Math.floor(scrollPosition / itemHeight) * itemHeight}px`,
+          width: '100%'
+        }}>
+          {visibleOptions.map(option => (
+            <OptionItem
+              key={option.value}
+              option={option}
+              isSelected={selectedValues.has(option.value)}
+              onSelect={onSelect}
+            />
+          ))}
+        </div>
+      </div>
+    </div>
+  );
+});
+
+VirtualizedOptions.displayName = 'VirtualizedOptions';
+
 const MultiInputCell = <T extends string>({
   field,
   value = [],
@@ -52,6 +175,9 @@ const MultiInputCell = <T extends string>({
   // Ref for the command list to enable scrolling
   const commandListRef = useRef<HTMLDivElement>(null)
 
+  // Create a memoized Set for fast lookups of selected values
+  const selectedValueSet = useMemo(() => new Set(internalValue), [internalValue]);
+
   // Sync internalValue with external value when component mounts or value changes externally
   useEffect(() => {
     if (!open) {
@@ -74,6 +200,7 @@ const MultiInputCell = <T extends string>({
     } else if (newOpen) {
       // Sync internal state with external state when opening
       setInternalValue(value);
+      setSearchQuery(""); // Reset search query on open
       if (onStartEdit) onStartEdit();
     }
   }, [open, internalValue, value, onChange, onStartEdit, onEndEdit]);
@@ -88,43 +215,84 @@ const MultiInputCell = <T extends string>({
       [];
     
     // Use provided options or field options, ensuring they have the correct shape
-    const availableOptions = (providedOptions || fieldOptions || []).map(option => ({
+    // Skip this work if we have a large number of options and they didn't change
+    if (providedOptions && providedOptions.length > 0) {
+      // Check if options are already in the right format
+      if (typeof providedOptions[0] === 'object' && 'label' in providedOptions[0] && 'value' in providedOptions[0]) {
+        return providedOptions as FieldOption[];
+      }
+      
+      return providedOptions.map(option => ({
         label: option.label || String(option.value),
         value: String(option.value)
       }));
-    
-    // Add default option if no options available
-    if (availableOptions.length === 0) {
-      availableOptions.push({ label: 'No options available', value: '' });
     }
     
-    return availableOptions;
+    // Check field options format
+    if (fieldOptions.length > 0) {
+      if (typeof fieldOptions[0] === 'object' && 'label' in fieldOptions[0] && 'value' in fieldOptions[0]) {
+        return fieldOptions as FieldOption[];
+      }
+      
+      return fieldOptions.map(option => ({
+        label: option.label || String(option.value),
+        value: String(option.value)
+      }));
+    }
+    
+    // Add default option if no options available
+    return [{ label: 'No options available', value: '' }];
   }, [field.fieldType, providedOptions]);
   
-  // Memoize filtered options based on search query
+  // Use deferredValue for search to prevent UI blocking with large lists
-  const filteredOptions = useMemo(() => {
+  const deferredSearchQuery = React.useDeferredValue(searchQuery);
-    if (!searchQuery) return selectOptions;
-    return selectOptions.filter(option =>
-      option.label.toLowerCase().includes(searchQuery.toLowerCase())
-    );
-  }, [selectOptions, searchQuery]);
   
-  // Sort options with selected items at the top for the dropdown
+  // Memoize filtered options based on search query - efficient filtering algorithm
+  const filteredOptions = useMemo(() => {
+    // If no search query, return all options
+    if (!deferredSearchQuery.trim()) return selectOptions;
+    
+    const query = deferredSearchQuery.toLowerCase();
+    
+    // Use faster algorithm for large option lists
+    if (selectOptions.length > 100) {
+      return selectOptions.filter(option => {
+        // First check starting with the query (most relevant)
+        if (option.label.toLowerCase().startsWith(query)) return true;
+        
+        // Then check includes for more general matches
+        return option.label.toLowerCase().includes(query);
+      });
+    }
+
+    // For smaller lists, do full text search
+    return selectOptions.filter(option =>
+      option.label.toLowerCase().includes(query)
+    );
+  }, [selectOptions, deferredSearchQuery]);
+
+  // Sort options with selected items at the top for the dropdown - only for smaller lists
   const sortedOptions = useMemo(() => {
+    // Skip expensive sorting for large lists
+    if (selectOptions.length > 100) return filteredOptions;
+    
     return [...filteredOptions].sort((a, b) => {
-      const aSelected = internalValue.includes(a.value);
+      const aSelected = selectedValueSet.has(a.value);
-      const bSelected = internalValue.includes(b.value);
+      const bSelected = selectedValueSet.has(b.value);
       
       if (aSelected && !bSelected) return -1;
       if (!aSelected && bSelected) return 1;
       return a.label.localeCompare(b.label);
     });
-  }, [filteredOptions, internalValue]);
+  }, [filteredOptions, selectedValueSet, selectOptions.length]);
 
   // Memoize selected values display
   const selectedValues = useMemo(() => {
+    // Use a map for looking up options by value for better performance
+    const optionsMap = new Map(selectOptions.map(opt => [opt.value, opt]));
+    
     return internalValue.map(v => {
-      const option = selectOptions.find(opt => String(opt.value) === String(v));
+      const option = optionsMap.get(v);
       return {
         value: v,
         label: option ? option.label : String(v)
@@ -141,7 +309,6 @@ const MultiInputCell = <T extends string>({
         return [...prev, selectedValue];
       }
     });
-    setSearchQuery("");
   }, []);
 
   // Handle focus
@@ -211,28 +378,6 @@ const MultiInputCell = <T extends string>({
     // Create a reference to the container element
     const containerRef = useRef<HTMLDivElement>(null);
     
-    // Use a layout effect to force the width after rendering
-    useLayoutEffect(() => {
-      if (containerRef.current) {
-        const container = containerRef.current;
-        
-        // Force direct style properties using the DOM API - simplified approach
-        container.style.width = `${cellWidth}px`;
-        container.style.minWidth = `${cellWidth}px`;
-        container.style.maxWidth = `${cellWidth}px`;
-        
-        // Apply to the button element as well
-        const button = container.querySelector('button');
-        if (button) {
-          // Cast to HTMLElement to access style property
-          const htmlButton = button as HTMLElement;
-          htmlButton.style.width = `${cellWidth}px`;
-          htmlButton.style.minWidth = `${cellWidth}px`;
-          htmlButton.style.maxWidth = `${cellWidth}px`;
-        }
-      }
-    }, [cellWidth]);
-    
     // Create a key-value map for inline styles with fixed width - simplified
     const fixedWidth = useMemo(() => ({
       width: `${cellWidth}px`,
@@ -241,6 +386,32 @@ const MultiInputCell = <T extends string>({
       boxSizing: 'border-box' as const,
     }), [cellWidth]);
     
+    // Use layout effect more efficiently - only for the button element
+    // since the container already uses inline styles
+    useLayoutEffect(() => {
+      // Skip if no width specified
+      if (!cellWidth) return;
+      
+      // Cache previous width to avoid unnecessary DOM updates
+      const prevWidth = containerRef.current?.getAttribute('data-prev-width');
+      
+      // Only update if width changed
+      if (prevWidth !== String(cellWidth) && containerRef.current) {
+        // Store new width for next comparison
+        containerRef.current.setAttribute('data-prev-width', String(cellWidth));
+        
+        // Only manipulate the button element directly since we can't 
+        // reliably style it with CSS in all cases
+        const button = containerRef.current.querySelector('button');
+        if (button) {
+          const htmlButton = button as HTMLElement;
+          htmlButton.style.width = `${cellWidth}px`;
+          htmlButton.style.minWidth = `${cellWidth}px`;
+          htmlButton.style.maxWidth = `${cellWidth}px`;
+        }
+      }
+    }, [cellWidth]);
+    
     return (
       <div 
         ref={containerRef}
@@ -305,30 +476,22 @@ const MultiInputCell = <T extends string>({
                 onValueChange={setSearchQuery}
               />
               <CommandList 
-                className="max-h-[200px] overflow-y-auto" 
+                className="overflow-hidden" 
                 ref={commandListRef}
                 onWheel={handleWheel}
               >
                 <CommandEmpty>No options found.</CommandEmpty>
                 <CommandGroup>
-                  {sortedOptions.map((option) => (
+                  {sortedOptions.length > 0 ? (
-                    <CommandItem
+                    <VirtualizedOptions 
-                      key={option.value}
+                      options={sortedOptions}
-                      value={option.value}
+                      selectedValues={selectedValueSet}
                       onSelect={handleSelect}
-                      className="flex w-full"
+                      maxHeight={200}
-                    >
-                      <div className="flex items-center w-full overflow-hidden">
-                        <Check
-                          className={cn(
-                            "mr-2 h-4 w-4 flex-shrink-0",
-                            internalValue.includes(option.value) ? "opacity-100" : "opacity-0"
-                          )}
                     />
-                        <span className="truncate w-full">{option.label}</span>
+                  ) : (
-                      </div>
+                    <div className="py-6 text-center text-sm">No options match your search</div>
-                    </CommandItem>
+                  )}
-                  ))}
                 </CommandGroup>
               </CommandList>
             </Command>
@@ -346,28 +509,6 @@ const MultiInputCell = <T extends string>({
     // Create a reference to the container element
     const containerRef = useRef<HTMLDivElement>(null);
     
-    // Use a layout effect to force the width after rendering
-    useLayoutEffect(() => {
-      if (containerRef.current) {
-        const container = containerRef.current;
-        
-        // Force direct style properties using the DOM API - simplified approach
-        container.style.width = `${cellWidth}px`;
-        container.style.minWidth = `${cellWidth}px`;
-        container.style.maxWidth = `${cellWidth}px`;
-        
-        // Apply to the button element as well
-        const button = container.querySelector('button');
-        if (button) {
-          // Cast to HTMLElement to access style property
-          const htmlButton = button as HTMLElement;
-          htmlButton.style.width = `${cellWidth}px`;
-          htmlButton.style.minWidth = `${cellWidth}px`;
-          htmlButton.style.maxWidth = `${cellWidth}px`;
-        }
-      }
-    }, [cellWidth]);
-    
     // Create a key-value map for inline styles with fixed width - simplified
     const fixedWidth = useMemo(() => ({
       width: `${cellWidth}px`,
@@ -376,6 +517,32 @@ const MultiInputCell = <T extends string>({
       boxSizing: 'border-box' as const,
     }), [cellWidth]);
     
+    // Use layout effect more efficiently - only for the button element
+    // since the container already uses inline styles
+    useLayoutEffect(() => {
+      // Skip if no width specified
+      if (!cellWidth) return;
+      
+      // Cache previous width to avoid unnecessary DOM updates
+      const prevWidth = containerRef.current?.getAttribute('data-prev-width');
+      
+      // Only update if width changed
+      if (prevWidth !== String(cellWidth) && containerRef.current) {
+        // Store new width for next comparison
+        containerRef.current.setAttribute('data-prev-width', String(cellWidth));
+        
+        // Only manipulate the button element directly since we can't 
+        // reliably style it with CSS in all cases
+        const button = containerRef.current.querySelector('button');
+        if (button) {
+          const htmlButton = button as HTMLElement;
+          htmlButton.style.width = `${cellWidth}px`;
+          htmlButton.style.minWidth = `${cellWidth}px`;
+          htmlButton.style.maxWidth = `${cellWidth}px`;
+        }
+      }
+    }, [cellWidth]);
+    
     return (
       <div 
         ref={containerRef}
@@ -460,4 +627,51 @@ const MultiInputCell = <T extends string>({
 
 MultiInputCell.displayName = 'MultiInputCell';
 
-export default React.memo(MultiInputCell); 
+export default React.memo(MultiInputCell, (prev, next) => {
+  // Quick check for reference equality of simple props
+  if (prev.hasErrors !== next.hasErrors || 
+      prev.disabled !== next.disabled ||
+      prev.isMultiline !== next.isMultiline ||
+      prev.isPrice !== next.isPrice ||
+      prev.separator !== next.separator) {
+    return false;
+  }
+  
+  // Array comparison for value
+  if (Array.isArray(prev.value) && Array.isArray(next.value)) {
+    if (prev.value.length !== next.value.length) return false;
+    
+    // Check each item in the array - optimize for large arrays
+    if (prev.value.length > 50) {
+      // For large arrays, JSON stringify is actually faster than iterating
+      return JSON.stringify(prev.value) === JSON.stringify(next.value);
+    }
+    
+    // For smaller arrays, iterative comparison is more efficient
+    for (let i = 0; i < prev.value.length; i++) {
+      if (prev.value[i] !== next.value[i]) return false;
+    }
+  } else if (prev.value !== next.value) {
+    return false;
+  }
+  
+  // Only do a full options comparison if they are different references and small arrays
+  if (prev.options !== next.options) {
+    if (!prev.options || !next.options) return false;
+    if (prev.options.length !== next.options.length) return false;
+    
+    // For large option lists, just check reference equality
+    if (prev.options.length > 100) return false;
+    
+    // For smaller lists, check if any values differ
+    for (let i = 0; i < prev.options.length; i++) {
+      const prevOpt = prev.options[i];
+      const nextOpt = next.options[i];
+      if (prevOpt.value !== nextOpt.value || prevOpt.label !== nextOpt.label) {
+        return false;
+      }
+    }
+  }
+  
+  return true;
+}); 

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells/SelectCell.tsx

@@ -1,4 +1,4 @@
-import { useState, useRef, useCallback, useMemo } from 'react'
+import { useState, useRef, useCallback, useMemo, useEffect } from 'react'
 import { Field } from '../../../../types'
 import { Check, ChevronsUpDown } from 'lucide-react'
 import { Button } from '@/components/ui/button'
@@ -47,7 +47,7 @@ const SelectCell = <T extends string>({
   const [isProcessing, setIsProcessing] = useState(false);
   
   // Update internal value when prop value changes
-  React.useEffect(() => {
+  useEffect(() => {
     setInternalValue(value);
     // When the value prop changes, it means validation is complete
     setIsProcessing(false);
@@ -55,32 +55,51 @@ const SelectCell = <T extends string>({
   
   // Memoize options processing to avoid recalculation on every render
   const selectOptions = useMemo(() => {
-    // Ensure we always have an array of options with the correct shape
+    // Fast path check - if we have raw options, just use those
-    const fieldType = field.fieldType;
+    if (options && options.length > 0) {
-    const fieldOptions = fieldType && 
+      // Check if options already have the correct structure to avoid mapping
-      (fieldType.type === 'select' || fieldType.type === 'multi-select') && 
+      if (typeof options[0] === 'object' && 'label' in options[0] && 'value' in options[0]) {
-      (fieldType as any).options ? 
+        return options as SelectOption[];
-      (fieldType as any).options : 
-      [];
-    
-    // Always ensure selectOptions is a valid array with at least a default option
-    const processedOptions = (options || fieldOptions || []).map((option: any) => ({
-      label: option.label || String(option.value),
-      value: String(option.value)
-    }));
-    
-    if (processedOptions.length === 0) {
-      processedOptions.push({ label: 'No options available', value: '' });
       }
       
-    return processedOptions;
+      // Optimize mapping to only convert what's needed
+      return options.map((option: any) => ({
+        label: option.label || String(option.value || option),
+        value: String(option.value || option)
+      }));
+    }
+    
+    // Fall back to field options if no direct options provided
+    const fieldType = field.fieldType;
+    if (fieldType && 
+        (fieldType.type === 'select' || fieldType.type === 'multi-select') && 
+        (fieldType as any).options) {
+      const fieldOptions = (fieldType as any).options;
+      
+      // Check if fieldOptions already have the correct structure
+      if (fieldOptions.length > 0 && typeof fieldOptions[0] === 'object' && 
+          'label' in fieldOptions[0] && 'value' in fieldOptions[0]) {
+        return fieldOptions as SelectOption[];
+      }
+      
+      return fieldOptions.map((option: any) => ({
+        label: option.label || String(option.value || option),
+        value: String(option.value || option)
+      }));
+    }
+    
+    // Return default empty option if no options available
+    return [{ label: 'No options available', value: '' }];
   }, [field.fieldType, options]);
   
   // Memoize display value to avoid recalculation on every render
   const displayValue = useMemo(() => {
-    return internalValue ? 
+    if (!internalValue) return 'Select...';
-      selectOptions.find((option: SelectOption) => String(option.value) === String(internalValue))?.label || String(internalValue) : 
+    
-      'Select...';
+    // Fast path: direct lookup by value using find
+    const stringValue = String(internalValue);
+    const found = selectOptions.find((option: SelectOption) => String(option.value) === stringValue);
+    return found ? found.label : stringValue;
   }, [internalValue, selectOptions]);
 
   // Handle wheel scroll in dropdown - optimized with passive event
@@ -112,27 +131,9 @@ const SelectCell = <T extends string>({
     }, 0);
   }, [onChange, onEndEdit]);
 
-  // Memoize the command items to avoid recreating them on every render
-  const commandItems = useMemo(() => {
-    return selectOptions.map((option: SelectOption) => (
-      <CommandItem
-        key={option.value}
-        value={option.label}
-        onSelect={() => handleSelect(option.value)}
-        className="cursor-pointer"
-      >
-        {option.label}
-        {String(option.value) === String(internalValue) && (
-          <Check className="ml-auto h-4 w-4" />
-        )}
-      </CommandItem>
-    ));
-  }, [selectOptions, internalValue, handleSelect]);
-
   // If disabled, render a static view
   if (disabled) {
-    const selectedOption = options.find(o => o.value === internalValue);
+    const displayText = displayValue;
-    const displayText = selectedOption ? selectedOption.label : internalValue;
     
     return (
       <div className={cn(
@@ -184,7 +185,7 @@ const SelectCell = <T extends string>({
         align="start"
         sideOffset={4}
       >
-        <Command shouldFilter={false}>
+        <Command shouldFilter={true}>
           <CommandInput 
             placeholder="Search..."
             className="h-9"
@@ -196,7 +197,19 @@ const SelectCell = <T extends string>({
           >
             <CommandEmpty>No results found.</CommandEmpty>
             <CommandGroup>
-              {commandItems}
+              {selectOptions.map((option: SelectOption) => (
+                <CommandItem
+                  key={option.value}
+                  value={option.label}
+                  onSelect={() => handleSelect(option.value)}
+                  className="cursor-pointer"
+                >
+                  {option.label}
+                  {String(option.value) === String(internalValue) && (
+                    <Check className="ml-auto h-4 w-4" />
+                  )}
+                </CommandItem>
+              ))}
             </CommandGroup>
           </CommandList>
         </Command>
@@ -208,10 +221,15 @@ const SelectCell = <T extends string>({
 // Optimize memo comparison to avoid unnecessary re-renders
 export default React.memo(SelectCell, (prev, next) => {
   // Only rerender when these critical props change
-  return (
+  if (prev.value !== next.value) return false;
-    prev.value === next.value && 
+  if (prev.hasErrors !== next.hasErrors) return false;
-    prev.hasErrors === next.hasErrors && 
+  if (prev.disabled !== next.disabled) return false;
-    prev.disabled === next.disabled &&
+  
-    prev.options === next.options
+  // Only check options array for reference equality - we're handling deep comparison internally
-  );
+  if (prev.options !== next.options && 
+      (prev.options.length !== next.options.length)) {
+    return false;
+  }
+  
+  return true;
 }); 

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useValidation.tsx

@@ -15,6 +15,14 @@ interface InfoWithSource {
   source: ErrorSources
 }
 
+// Shared utility function for checking empty values - defined once to avoid duplication
+const isEmpty = (value: any): boolean => 
+  value === undefined || 
+  value === null || 
+  value === '' || 
+  (Array.isArray(value) && value.length === 0) ||
+  (typeof value === 'object' && !Array.isArray(value) && Object.keys(value).length === 0);
+
 export const useValidation = <T extends string>(
   fields: Fields<T>,
   rowHook?: RowHook<T>,
@@ -32,15 +40,8 @@ export const useValidation = <T extends string>(
     field.validations.forEach(validation => {
       switch (validation.rule) {
         case 'required':
-          // More granular check for empty values
+          // Use the shared isEmpty function
-          const isEmpty = 
+          if (isEmpty(value)) {
-            value === undefined || 
-            value === null || 
-            value === '' || 
-            (Array.isArray(value) && value.length === 0) ||
-            (typeof value === 'object' && Object.keys(value).length === 0);
-            
-          if (isEmpty) {
             errors.push({
               message: validation.errorMessage || 'This field is required',
               level: validation.level || 'error'
@@ -82,13 +83,7 @@ export const useValidation = <T extends string>(
     // Run field-level validations
     const fieldErrors: Record<string, ValidationError[]> = {}
     
-    // Helper function to check if a value is empty
+    // Use the shared isEmpty function
-    const isEmpty = (value: any): boolean => 
-      value === undefined || 
-      value === null || 
-      value === '' || 
-      (Array.isArray(value) && value.length === 0) ||
-      (typeof value === 'object' && !Array.isArray(value) && Object.keys(value).length === 0);
     
     fields.forEach(field => {
       const value = row[String(field.key) as keyof typeof row]
@@ -230,7 +225,7 @@ export const useValidation = <T extends string>(
         const value = String(row[String(key) as keyof typeof row] || '')
         
         // Skip empty values if allowed
-        if (allowEmpty && (value === '' || value === undefined || value === null)) {
+        if (allowEmpty && isEmpty(value)) {
           return
         }
         
@@ -265,13 +260,7 @@ export const useValidation = <T extends string>(
 
   // Run complete validation
   const validateData = useCallback(async (data: RowData<T>[]) => {
-    // Helper function to check if a value is empty
+    // Use the shared isEmpty function
-    const isEmpty = (value: any): boolean => 
-      value === undefined || 
-      value === null || 
-      value === '' || 
-      (Array.isArray(value) && value.length === 0) ||
-      (typeof value === 'object' && !Array.isArray(value) && Object.keys(value).length === 0);
     
     // Step 1: Run field and row validation
     const rowValidations = await Promise.all(

inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/hooks/useValidationState.tsx

@@ -225,8 +225,49 @@ export const useValidationState = <T extends string>({
   const flushPendingUpdates = useCallback(() => {
     const updates = pendingUpdatesRef.current;
     
+    // Use a single setState call for validation errors if possible
     if (updates.errors.size > 0) {
       setValidationErrors(prev => {
+        // Create a new Map only if we're modifying it
+        const needsUpdate = Array.from(updates.errors.entries()).some(([rowIndex, errors]) => {
+          const prevErrors = prev.get(rowIndex);
+          const hasErrors = Object.keys(errors).length > 0;
+          
+          // Check if we need to update this row's errors
+          if (!prevErrors && hasErrors) return true;
+          if (prevErrors && !hasErrors) return true;
+          if (!prevErrors && !hasErrors) return false;
+          
+          // Check if the error objects are different
+          return Object.keys(errors).some(key => {
+            const prevError = prevErrors?.[key];
+            const nextError = errors[key];
+            
+            if (!prevError && nextError) return true;
+            if (prevError && !nextError) return true;
+            if (!prevError && !nextError) return false;
+            
+            // Compare the arrays if both exist
+            if (Array.isArray(prevError) && Array.isArray(nextError)) {
+              if (prevError.length !== nextError.length) return true;
+              
+              // Deep comparison of error objects
+              return prevError.some((err, i) => {
+                const nextErr = nextError[i];
+                return err.message !== nextErr.message || 
+                       err.level !== nextErr.level || 
+                       err.source !== nextErr.source;
+              });
+            }
+            
+            return true;
+          });
+        });
+        
+        // If no real changes, return the same state object
+        if (!needsUpdate) return prev;
+        
+        // Otherwise create a new Map with the updates
         const newErrors = new Map(prev);
         updates.errors.forEach((errors, rowIndex) => {
           if (Object.keys(errors).length === 0) {
@@ -235,30 +276,75 @@ export const useValidationState = <T extends string>({
             newErrors.set(rowIndex, errors);
           }
         });
+        
         return newErrors;
       });
+      
+      // Clear the updates
       updates.errors = new Map();
     }
 
+    // Use a single setState call for row validation statuses
     if (updates.statuses.size > 0) {
       setRowValidationStatus(prev => {
+        // Check if we need to update
+        const needsUpdate = Array.from(updates.statuses.entries()).some(([rowIndex, status]) => {
+          return prev.get(rowIndex) !== status;
+        });
+        
+        // If no real changes, return the same state object
+        if (!needsUpdate) return prev;
+        
+        // Create a new Map with updates
         const newStatuses = new Map(prev);
         updates.statuses.forEach((status, rowIndex) => {
           newStatuses.set(rowIndex, status);
         });
+        
         return newStatuses;
       });
+      
+      // Clear the updates
       updates.statuses = new Map();
     }
 
+    // Use a single setState call for data updates
     if (updates.data.length > 0) {
+      // Find non-empty items
+      const dataUpdates = updates.data.filter(item => item !== undefined);
+      
+      if (dataUpdates.length > 0) {
         setData(prev => {
-        const newData = [...prev];
+          // Check if we actually need to update
-        updates.data.forEach((row, index) => {
+          const needsUpdate = dataUpdates.some((row, index) => {
-          newData[index] = row;
+            const oldRow = prev[index];
+            if (!oldRow) return true;
+            
+            // Compare the rows
+            return Object.keys(row).some(key => {
+              // Skip meta fields that don't affect rendering
+              if (key.startsWith('__') && key !== '__template') return false;
+              
+              return oldRow[key] !== row[key];
             });
+          });
+          
+          // If no actual changes, return the same array
+          if (!needsUpdate) return prev;
+          
+          // Create a new array with the updates
+          const newData = [...prev];
+          dataUpdates.forEach((row, index) => {
+            if (index < newData.length) {
+              newData[index] = row;
+            }
+          });
+          
           return newData;
         });
+      }
+      
+      // Clear the updates
       updates.data = [];
     }
   }, []);
@@ -289,39 +375,62 @@ export const useValidationState = <T extends string>({
 
   // Update validateUniqueItemNumbers to use batch updates
   const validateUniqueItemNumbers = useCallback(async () => {
-    const duplicates = new Map<string, number[]>();
+    console.log('Validating unique item numbers');
-    const itemNumberMap = new Map<string, number>();
     
+    // Skip if no data
+    if (!data.length) return;
+    
+    // Use a more efficient Map to track duplicates
+    const itemNumberMap = new Map<string, number[]>();
+    
+    // Initialize batch updates
+    const errors = new Map<number, Record<string, ErrorType[]>>();
+    
+    // Single pass through data to identify all item numbers
     data.forEach((row, index) => {
       const itemNumber = row.item_number?.toString();
       if (itemNumber) {
-        if (itemNumberMap.has(itemNumber)) {
+        // Get or initialize the array of indices for this item number
-          const existingIndex = itemNumberMap.get(itemNumber)!;
+        const indices = itemNumberMap.get(itemNumber) || [];
-          if (!duplicates.has(itemNumber)) {
+        indices.push(index);
-            duplicates.set(itemNumber, [existingIndex]);
+        itemNumberMap.set(itemNumber, indices);
-          }
-          duplicates.get(itemNumber)!.push(index);
-        } else {
-          itemNumberMap.set(itemNumber, index);
-        }
       }
     });
     
-    duplicates.forEach((rowIndices, itemNumber) => {
+    // Process duplicates more efficiently
-      rowIndices.forEach(rowIndex => {
+    itemNumberMap.forEach((indices, itemNumber) => {
-        const errors = {
+      // Only process if there are duplicates
-          item_number: [{
+      if (indices.length > 1) {
+        const errorObj = {
           message: `Duplicate item number: ${itemNumber}`,
           level: 'error',
           source: 'validation'
-          }]
         };
-        queueUpdate(rowIndex, { errors });
+        
+        // Add error to each row with this item number
+        indices.forEach(rowIndex => {
+          const rowErrors = errors.get(rowIndex) || {};
+          rowErrors['item_number'] = [errorObj];
+          errors.set(rowIndex, rowErrors);
         });
+      }
     });
     
-    debouncedFlushUpdates();
+    // Apply batch updates
-  }, [data, queueUpdate, debouncedFlushUpdates]);
+    if (errors.size > 0) {
+      setValidationErrors(prev => {
+        const newMap = new Map(prev);
+        errors.forEach((rowErrors, rowIndex) => {
+          // Preserve existing errors for other fields
+          const existingErrors = newMap.get(rowIndex) || {};
+          newMap.set(rowIndex, { ...existingErrors, ...rowErrors });
+        });
+        return newMap;
+      });
+    }
+    
+    console.log('Unique item number validation complete');
+  }, [data]);
 
   // Fetch product by UPC from API - optimized with proper error handling and types
   const fetchProductByUpc = useCallback(async (supplier: string, upc: string): Promise<ValidationResult> => {
@@ -664,8 +773,14 @@ export const useValidationState = <T extends string>({
     const fieldErrors: Record<string, ErrorType[]> = {};
     let hasErrors = false;
     
+    // Get current errors for comparison
+    const currentErrors = validationErrors.get(rowIndex) || {};
+    
+    // Track if row has changes to original values
+    const originalRow = row.__original || {};
+    const changedFields = row.__changes || {};
+    
     // Use a more efficient approach - only validate fields that need validation
-    // This includes required fields and fields with values
     fields.forEach(field => {
       if (field.disabled) return;
       
@@ -678,15 +793,32 @@ export const useValidationState = <T extends string>({
         return;
       }
       
+      // Only validate if:
+      // 1. Field has changed (if we have change tracking)
+      // 2. No prior validation exists
+      // 3. This is a special field (supplier/company)
+      const hasChanged = changedFields[key] || 
+                         !currentErrors[key] || 
+                         key === 'supplier' || 
+                         key === 'company';
+                         
+      if (hasChanged) {
         // Validate the field
         const errors = validateField(value, field as Field<T>);
         if (errors.length > 0) {
           fieldErrors[key] = errors;
           hasErrors = true;
         }
+      } else {
+        // Keep existing errors if field hasn't changed
+        if (currentErrors[key] && currentErrors[key].length > 0) {
+          fieldErrors[key] = currentErrors[key];
+          hasErrors = true;
+        }
+      }
     });
     
-    // Special validation for supplier and company
+    // Special validation for supplier and company - always validate these
     if (!row.supplier) {
       fieldErrors['supplier'] = [{
         message: 'Supplier is required',
@@ -707,7 +839,11 @@ export const useValidationState = <T extends string>({
     // Update validation errors for this row
     setValidationErrors(prev => {
       const updated = new Map(prev);
+      if (Object.keys(fieldErrors).length > 0) {
         updated.set(rowIndex, fieldErrors);
+      } else {
+        updated.delete(rowIndex);
+      }
       return updated;
     });
     
@@ -717,7 +853,7 @@ export const useValidationState = <T extends string>({
       updated.set(rowIndex, hasErrors ? 'error' : 'validated');
       return updated;
     });
-  }, [data, fields, validateField]);
+  }, [data, fields, validateField, validationErrors]);
   
   // Update a row's field value
   const updateRow = useCallback((rowIndex: number, key: T, value: any) => {
@@ -926,7 +1062,7 @@ export const useValidationState = <T extends string>({
     }
   }, [data, rowSelection, setData]);
 
-  // Apply template to rows
+  // Apply template to rows - optimized version
   const applyTemplate = useCallback((templateId: string, rowIndexes: number[]) => {
     const template = templates.find(t => t.id.toString() === templateId);
     
@@ -936,7 +1072,6 @@ export const useValidationState = <T extends string>({
     }
     
     console.log(`Applying template ${templateId} to rows:`, rowIndexes);
-    console.log(`Template data:`, template);
     
     // Validate row indexes
     const validRowIndexes = rowIndexes.filter(index => 
@@ -949,11 +1084,6 @@ export const useValidationState = <T extends string>({
       return;
     }
     
-    if (validRowIndexes.length !== rowIndexes.length) {
-      console.warn('Some row indexes were invalid and will be skipped:', 
-        rowIndexes.filter(idx => !validRowIndexes.includes(idx)));
-    }
-    
     // Set the template application flag
     isApplyingTemplateRef.current = true;
     
@@ -963,59 +1093,58 @@ export const useValidationState = <T extends string>({
       top: window.scrollY
     };
     
-    // Track updated rows for UPC validation
+    // Create a copy of data and process all rows at once to minimize state updates
-    const updatedRows: number[] = [];
-    
-    // Create a copy of the data to track updates
     const newData = [...data];
+    const batchErrors = new Map<number, Record<string, ErrorType[]>>();
+    const batchStatuses = new Map<number, 'pending' | 'validating' | 'validated' | 'error'>();
+    
+    // Extract template fields once outside the loop
+    const templateFields = Object.entries(template).filter(([key]) => 
+      !['id', '__errors', '__meta', '__template', '__original', '__corrected', '__changes'].includes(key)
+    );
     
     // Apply template to each valid row
     validRowIndexes.forEach(index => {
       // Create a new row with template values
       const originalRow = newData[index];
-      console.log(`Applying to row at index ${index}:`, originalRow);
+      const updatedRow = { ...originalRow } as Record<string, any>;
-      
-      const updatedRow = { ...originalRow };
       
       // Clear existing errors
       delete updatedRow.__errors;
       
       // Apply template fields (excluding metadata fields)
-      Object.entries(template).forEach(([key, value]) => {
+      for (const [key, value] of templateFields) {
-        if (!['id', '__errors', '__meta', '__template', '__original', '__corrected', '__changes'].includes(key)) {
+        updatedRow[key] = value;
-          (updatedRow as any)[key] = value;
       }
-      });
       
       // Mark the row as using this template
       updatedRow.__template = templateId;
       
       // Update the row in the data array
-      newData[index] = updatedRow;
+      newData[index] = updatedRow as RowData<T>;
       
-      // Track which rows were updated
+      // Clear validation errors and mark as validated
-      updatedRows.push(index);
+      batchErrors.set(index, {});
-      
+      batchStatuses.set(index, 'validated');
-      console.log(`Row ${index} updated:`, updatedRow);
     });
     
-    // Update all data at once
+    // Perform a single update for all rows
     setData(newData);
     
-    // Clear validation errors and status for affected rows
+    // Update all validation errors and statuses at once
     setValidationErrors(prev => {
       const newErrors = new Map(prev);
-      validRowIndexes.forEach(index => {
+      for (const [rowIndex, errors] of batchErrors.entries()) {
-        newErrors.delete(index);
+        newErrors.set(rowIndex, errors);
-      });
+      }
       return newErrors;
     });
     
     setRowValidationStatus(prev => {
       const newStatus = new Map(prev);
-      validRowIndexes.forEach(index => {
+      for (const [rowIndex, status] of batchStatuses.entries()) {
-        newStatus.set(index, 'validated');  // Mark as validated immediately
+        newStatus.set(rowIndex, status);
-      });
+      }
       return newStatus;
     });
     
@@ -1031,32 +1160,56 @@ export const useValidationState = <T extends string>({
       toast.success(`Template applied to ${validRowIndexes.length} rows`);
     }
     
-    // Schedule UPC validation with a delay
+    // Check which rows need UPC validation
+    const upcValidationRows = validRowIndexes.filter(rowIndex => {
+      const row = newData[rowIndex];
+      return row && row.upc && row.supplier;
+    });
+    
+    // If there are rows needing UPC validation, process them
+    if (upcValidationRows.length > 0) {
+      // Batch UPC validation for better performance
       setTimeout(() => {
-      // Process rows in sequence to ensure validation state is consistent
+        // Process in batches to avoid overwhelming API
-      const processRows = async () => {
+        const processUpcValidations = async () => {
-        for (const rowIndex of updatedRows) {
+          const BATCH_SIZE = 5;
-          // Get the current row data after template application
-          const currentRow = newData[rowIndex];
           
-          // Check if UPC validation is needed
+          // Sort by upc for better caching
-          if (currentRow && currentRow.upc && currentRow.supplier) {
+          upcValidationRows.sort((a, b) => {
-            await validateUpc(rowIndex, String(currentRow.supplier), String(currentRow.upc));
+            const aUpc = String(newData[a].upc || '');
+            const bUpc = String(newData[b].upc || '');
+            return aUpc.localeCompare(bUpc);
+          });
+          
+          // Process in batches to avoid hammering the API
+          for (let i = 0; i < upcValidationRows.length; i += BATCH_SIZE) {
+            const batch = upcValidationRows.slice(i, i + BATCH_SIZE);
+            
+            // Process this batch in parallel
+            await Promise.all(batch.map(async (rowIndex) => {
+              const row = newData[rowIndex];
+              if (row && row.upc && row.supplier) {
+                await validateUpc(rowIndex, String(row.supplier), String(row.upc));
               }
+            }));
             
-          // Small delay between rows to prevent overwhelming the UI
+            // Add delay between batches to reduce server load
-          if (updatedRows.length > 1) {
+            if (i + BATCH_SIZE < upcValidationRows.length) {
-            await new Promise(resolve => setTimeout(resolve, 50));
+              await new Promise(r => setTimeout(r, 300));
             }
           }
           
-        // Reset the template application flag after all processing is done
+          // Reset template application flag
           isApplyingTemplateRef.current = false;
         };
         
-      // Start processing rows
+        // Start processing
-      processRows();
+        processUpcValidations();
-    }, 500);
+      }, 100);
+    } else {
+      // No UPC validation needed, reset flag immediately
+      isApplyingTemplateRef.current = false;
+    }
   }, [data, templates, validateUpc, setData, setValidationErrors, setRowValidationStatus]);
 
   // Apply template to selected rows
@@ -1234,24 +1387,38 @@ export const useValidationState = <T extends string>({
         return;
       }
 
+      // Create a copy for data modifications
       const newData = [...data];
-      const initialStatus = new Map();
+      // Use Maps for better performance with large datasets
-      const initialErrors = new Map();
+      const batchErrors = new Map<number, Record<string, ErrorType[]>>();
+      const batchStatuses = new Map<number, 'pending' | 'validating' | 'validated' | 'error'>();
 
       console.log(`Validating ${data.length} rows`);
 
       // Process in batches to avoid blocking the UI
-      const BATCH_SIZE = 100; // Increase batch size for better performance
+      const BATCH_SIZE = Math.min(100, Math.max(20, Math.floor(data.length / 10))); // Adaptive batch size
+      const totalBatches = Math.ceil(data.length / BATCH_SIZE);
       let currentBatch = 0;
-      let totalBatches = Math.ceil(data.length / BATCH_SIZE);
+      
+      // Pre-cache field validations
+      const requiredFields = fields.filter(f => f.validations?.some(v => v.rule === 'required'));
+      const requiredFieldKeys = new Set(requiredFields.map(f => String(f.key)));
+      
+      // Pre-process the supplier and company fields checks
+      const hasSupplierField = fields.some(field => String(field.key) === 'supplier');
+      const hasCompanyField = fields.some(field => String(field.key) === 'company');
       
       const processBatch = () => {
         const startIdx = currentBatch * BATCH_SIZE;
         const endIdx = Math.min(startIdx + BATCH_SIZE, data.length);
         
-        // Create a batch of validation promises
+        // Start validation time measurement for this batch
+        const batchStartTime = performance.now();
+        
+        // Create validation promises for all rows in the batch
         const batchPromises = [];
         
+        // Prepare a single batch processor for all rows
         for (let rowIndex = startIdx; rowIndex < endIdx; rowIndex++) {
           batchPromises.push(
             new Promise<void>(resolve => {
@@ -1274,36 +1441,52 @@ export const useValidationState = <T extends string>({
                 } as RowData<T>;
               }
               
-              // Process price fields to strip dollar signs - use the cleanPriceFields function
+              // Process price fields efficiently - use a single check for both fields
               const rowAsRecord = row as Record<string, any>;
-              if ((typeof rowAsRecord.msrp === 'string' && rowAsRecord.msrp.includes('$')) ||
+              const mSrpNeedsProcessing = typeof rowAsRecord.msrp === 'string' && rowAsRecord.msrp.includes('$'); 
-                  (typeof rowAsRecord.cost_each === 'string' && rowAsRecord.cost_each.includes('$'))) {
+              const costEachNeedsProcessing = typeof rowAsRecord.cost_each === 'string' && rowAsRecord.cost_each.includes('$');
-                // Clean just this row
+              
-                const cleanedRow = cleanPriceFields([row])[0];
+              if (mSrpNeedsProcessing || costEachNeedsProcessing) {
-                newData[rowIndex] = cleanedRow;
+                // Create a clean copy only if needed
+                const cleanedRow = {...row} as Record<string, any>;
+                
+                if (mSrpNeedsProcessing) {
+                  const msrpValue = rowAsRecord.msrp.replace(/[$,]/g, '');
+                  const numValue = parseFloat(msrpValue);
+                  cleanedRow.msrp = !isNaN(numValue) ? numValue.toFixed(2) : msrpValue;
                 }
                 
-              // Only validate required fields and fields with values
+                if (costEachNeedsProcessing) {
-              fields.forEach(field => {
+                  const costValue = rowAsRecord.cost_each.replace(/[$,]/g, '');
-                if (field.disabled) return;
+                  const numValue = parseFloat(costValue);
+                  cleanedRow.cost_each = !isNaN(numValue) ? numValue.toFixed(2) : costValue;
+                }
+                
+                newData[rowIndex] = cleanedRow as RowData<T>;
+              }
+
+              // Only validate required fields for efficiency
+              for (const field of requiredFields) {
                 const key = String(field.key);
                 const value = row[key as keyof typeof row];
                 
-                // Skip validation for empty non-required fields
+                // Skip non-required empty fields
-                const isRequired = field.validations?.some(v => v.rule === 'required');
+                if (value === undefined || value === null || value === '' || 
-                if (!isRequired && (value === undefined || value === null || value === '')) {
+                    (Array.isArray(value) && value.length === 0) ||
-                  return;
+                    (typeof value === 'object' && value !== null && Object.keys(value).length === 0)) {
-                }
                   
-                const errors = validateField(value, field as Field<T>);
+                  // Add error for empty required fields
-                if (errors.length > 0) {
+                  fieldErrors[key] = [{
-                  fieldErrors[key] = errors;
+                    message: field.validations?.find(v => v.rule === 'required')?.errorMessage || 'This field is required',
+                    level: 'error',
+                    source: 'required'
+                  }];
                   hasErrors = true;
                 }
-              });
+              }
               
               // Special validation for supplier and company
-              if (!row.supplier) {
+              if (hasSupplierField && !row.supplier) {
                 fieldErrors['supplier'] = [{
                   message: 'Supplier is required',
                   level: 'error',
@@ -1311,7 +1494,8 @@ export const useValidationState = <T extends string>({
                 }];
                 hasErrors = true;
               }
-              if (!row.company) {
+              
+              if (hasCompanyField && !row.company) {
                 fieldErrors['company'] = [{
                   message: 'Company is required',
                   level: 'error',
@@ -1320,11 +1504,13 @@ export const useValidationState = <T extends string>({
                 hasErrors = true;
               }
 
-              // Update validation errors for this row
+              // Only add errors if there are any
-              initialErrors.set(rowIndex, fieldErrors);
+              if (Object.keys(fieldErrors).length > 0) {
+                batchErrors.set(rowIndex, fieldErrors);
+              }
               
               // Update row validation status
-              initialStatus.set(rowIndex, hasErrors ? 'error' : 'validated');
+              batchStatuses.set(rowIndex, hasErrors ? 'error' : 'validated');
               
               resolve();
             })
@@ -1333,30 +1519,43 @@ export const useValidationState = <T extends string>({
         
         // Process all promises in the batch
         Promise.all(batchPromises).then(() => {
-          // Update state for this batch
+          // Measure batch completion time
+          const batchEndTime = performance.now();
+          const processingTime = batchEndTime - batchStartTime;
+          
+          // Update UI state for this batch more efficiently
+          if (batchErrors.size > 0) {
             setValidationErrors(prev => {
               const newMap = new Map(prev);
-            initialErrors.forEach((errors, rowIndex) => {
+              for (const [rowIndex, errors] of batchErrors.entries()) {
                 newMap.set(rowIndex, errors);
-            });
+              }
               return newMap;
             });
+          }
           
+          if (batchStatuses.size > 0) {
             setRowValidationStatus(prev => {
               const newMap = new Map(prev);
-            initialStatus.forEach((status, rowIndex) => {
+              for (const [rowIndex, status] of batchStatuses.entries()) {
                 newMap.set(rowIndex, status);
-            });
+              }
               return newMap;
             });
+          }
           
           // Move to the next batch or finish
           currentBatch++;
+          
+          // Log progress
+          console.log(`Batch ${currentBatch}/${totalBatches} completed in ${processingTime.toFixed(2)}ms`);
+          
           if (currentBatch < totalBatches) {
-            // Schedule the next batch with a small delay to allow UI updates
+            // Adaptive timeout based on processing time
-            setTimeout(processBatch, 10);
+            const nextDelay = Math.min(50, Math.max(5, Math.ceil(processingTime / 10)));
+            setTimeout(processBatch, nextDelay);
           } else {
-            // All batches processed, update the data
+            // All batches processed, update the data once
             setData(newData);
             console.log('Basic validation complete');
             initialValidationDoneRef.current = true;