inventory/docs/ValidationStep-Refactoring-Plan.md

ValidationStep Component Refactoring Plan

Overview

This document outlines a comprehensive plan to refactor the current ValidationStep component (4000+ lines) into a more maintainable, modular structure. The new implementation will be developed alongside the existing component without modifying the original code. Once completed, the previous step in the workflow will offer the option to continue to either the original ValidationStep or the new implementation.

Table of Contents

1. Current Component Analysis
2. New Architecture Design
3. Component Structure
4. State Management
5. Key Features Implementation
6. Integration Plan
7. Testing Strategy
8. Project Timeline
9. Design Principles
10. Appendix: Function Reference

Current Component Analysis

The current ValidationStep component has several issues:

• Size: At over 4000 lines, it's difficult to maintain and understand
• Multiple responsibilities: Handles validation, UI rendering, template management, and more
• Special cases: Contains numerous special case handlers and exceptions
• Complex state management: State is distributed across multiple useState calls
• Tightly coupled concerns: UI, validation logic, and business rules are intertwined

Key Features to Preserve

1. Data Validation

   • Field-level validation (required, regex, unique)
   • Row-level validation (supplier, company fields)
   • UPC validation with API integration
   • AI-assisted validation

2. Template Management

   • Saving, loading, and applying templates
   • Template-based validation

3. UI Components

   • Editable table with specialized cell renderers
   • Error display and management
   • Filtering and sorting capabilities
   • Status indicators and progress tracking

4. Special Field Handling

   • Input fields with price formatting
   • Multi-input fields with separator configuration
   • Select fields with dropdown options
   • Checkbox fields with boolean value mapping
   • UPC fields with specialized validation

5. User Interaction Flows

   • Tab and keyboard navigation
   • Bulk operations (select all, apply template)
   • Row validation on value change
   • Error reporting and display

New Architecture Design

The new architecture will follow these principles:

1. Separation of Concerns

   • UI rendering separate from business logic
   • Validation logic isolated from state management
   • Clear interfaces between components

2. Composable Components

   • Small, focused components with single responsibilities
   • Reusable pattern for different field types

3. Centralized State Management

   • Custom hooks for state management
   • Clear data flow patterns
   • Reduced prop drilling

4. Consistent Error Handling

   • Standardized error structure
   • Predictable error propagation
   • User-friendly error display

5. Performance Optimization

   • Virtualized table rendering
   • Memoization of expensive computations
   • Deferred validation for better user experience

Component Structure

The new ValidationStepNew folder has the following structure:

ValidationStepNew/
├── index.tsx                    # Main entry point that composes all pieces
├── components/                  # UI Components
│   ├── ValidationContainer.tsx  # Main wrapper component
│   ├── ValidationTable.tsx      # Table implementation
│   ├── ValidationCell.tsx       # Cell component
│   ├── ValidationSidebar.tsx    # Sidebar with controls
│   ├── ValidationToolbar.tsx    # Top toolbar (removed as unnecessary)
│   ├── TemplateManager.tsx      # Template management
│   ├── FilterPanel.tsx          # Filtering interface (integrated into Container)
│   └── cells/                   # Specialized cell renderers
│       ├── InputCell.tsx
│       ├── SelectCell.tsx
│       ├── MultiInputCell.tsx
│       └── CheckboxCell.tsx
├── hooks/                       # Custom hooks
│   ├── useValidationState.tsx   # Main state management
│   ├── useTemplates.tsx         # Template-related logic (integrated into ValidationState)
│   ├── useFilters.tsx           # Filtering logic (integrated into ValidationState)
│   └── useUpcValidation.tsx     # UPC-specific validation
└── utils/                       # Utility functions
    ├── validationUtils.ts       # Validation helper functions
    ├── formatters.ts            # Value formatting utilities 
    └── constants.ts             # Constant values and configuration

Component Responsibilities

ValidationContainer

• Main container component
• Coordinates between subcomponents
• Manages global state
• Handles navigation events (next, back)
• Contains filter controls

ValidationTable

• Displays the data in tabular form
• Manages selection state
• Handles keyboard navigation
• Integrates with TanStack Table
• Displays properly styled rows and cells

ValidationCell

• Factory component that renders appropriate cell type
• Manages cell-level state
• Handles validation errors display
• Manages edit mode
• Shows consistent error indicators

TemplateManager

• Handles template selection UI
• Provides template save/load functionality
• Manages template application to rows

Cell Components

• InputCell: Handles text input with multiline and price support
• MultiInputCell: Handles multiple values with separator configuration
• SelectCell: Command/popover component for single selection
• CheckboxCell: Boolean value selection with mapping support

State Management

Core State Interface

interface ValidationState<T extends string> {
  // Core data
  data: RowData<T>[];
  filteredData: RowData<T>[];
  
  // Validation state
  isValidating: boolean;
  validationErrors: Map<number, Record<string, Error[]>>;
  rowValidationStatus: Map<number, 'pending' | 'validating' | 'validated' | 'error'>;
  
  // Selection state
  rowSelection: RowSelectionState;
  
  // Template state
  templates: Template[];
  selectedTemplateId: string | null;
  
  // Filter state
  filters: FilterState;
  
  // Methods
  updateRow: (rowIndex: number, key: T, value: any) => void;
  validateRow: (rowIndex: number) => Promise<void>;
  validateUpc: (rowIndex: number, upcValue: string) => Promise<void>;
  applyTemplate: (templateId: string, rowIndexes: number[]) => void;
  saveTemplate: (name: string, type: string) => void;
  setFilters: (newFilters: Partial<FilterState>) => void;
  // Additional methods...
}

useValidationState Hook

The main state management hook handles:

• Data manipulation (update, sort, filter)
• Selection management
• Validation coordination
• Integration with validation utilities
• Template management
• Filtering and sorting

Key Features Implementation

1. Field Type Handling

Implemented a strategy pattern for different field types:

// In ValidationCell
const renderCellContent = () => {
  const fieldType = field.fieldType.type
  
  switch (fieldType) {
    case 'input':
      return <InputCell<T> field={field} value={value} onChange={onChange} ... />
    case 'multi-input':
      return <MultiInputCell<T> field={field} value={value} onChange={onChange} ... />
    case 'select':
      return <SelectCell<T> field={field} value={value} onChange={onChange} ... />
    // etc.
  }
}

2. Validation Logic

Validation is broken down into clear steps:

1. Field Validation: Apply field-level validations (required, regex, etc.)
2. Row Validation: Apply row-level validations and rowHook
3. Table Validation: Apply table-level validations (unique) and tableHook

Validation now happens automatically without explicit buttons, with immediate feedback on field blur.

3. UI Components

UI components follow these principles:

1. Consistent Styling: All components use shadcn UI for consistent look and feel
2. Visual Feedback: Errors are clearly indicated with icons and border styling
3. Intuitive Editing: Fields show outlines even when not in focus, and edit on click
4. Proper Command Pattern: Select and multi-select fields use command/popover pattern for better UX
5. Focus Management: Fields close when clicking away and perform validation on blur

Design Principles

Based on user preferences and best practices, the following design principles guide this refactoring:

1. Automatic Validation

   • Validation should happen automatically without explicit buttons
   • All validation should run on initial data load
   • Fields should validate on blur (when user clicks away)

2. Modern UI Patterns

   • Command/popover components for all selects and multi-selects
   • Consistent field outlines and borders even when not in focus
   • Badge patterns for multi-select items
   • Clear visual indicators for errors

3. Reduced Complexity

   • Remove unnecessary UI elements like "validate all" buttons
   • Eliminate redundant state and toast notifications
   • Simplify component hierarchy where possible
   • Find root causes rather than adding special cases

4. Consistent Component Behavior

   • Fields should close when clicking away
   • All inputs should follow the same editing pattern
   • Error handling should be consistent across all field types
   • Multi-select fields should allow selecting multiple items with clear visual feedback

Integration Plan

1. Creating the New Component Structure

Folder structure has been created without modifying the existing code:

mkdir -p inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/{components,hooks,utils}
mkdir -p inventory/src/lib/react-spreadsheet-import/src/steps/ValidationStepNew/components/cells

2. Implementing Basic Components

Core components have been implemented:

1. Created index.tsx as the main entry point
2. Implemented ValidationContainer with basic state management
3. Created ValidationTable for data display
4. Implemented basic cell rendering with specialized cell types

3. Implementing State Management

State management has been implemented:

1. Created useValidationState hook
2. Implemented data transformation utilities
3. Added validation logic

4. Integrating with Previous Step

The previous step component allows choosing between validation implementations, enabling gradual testing and adoption.

Testing Strategy

1. Unit Tests

   • Test individual utility functions
   • Test hooks in isolation
   • Test individual UI components

2. Integration Tests

   • Test component interactions
   • Test state management flow
   • Test validation logic integration

3. Comparison Tests

   • Compare output of new component with original
   • Verify that all functionality works the same

4. Performance Tests

   • Measure render times
   • Evaluate memory usage
   • Compare against original component

Project Timeline

1. Phase 1: Initial Structure (Completed)

   • Set up folder structure
   • Implement basic components
   • Create core state management

2. Phase 2: Core Functionality (In Progress)

   • Implement validation logic (completed)
   • Create cell renderers (completed)
   • Add template management (in progress)

3. Phase 3: Special Features (Upcoming)

   • Implement UPC validation
   • Add AI validation
   • Handle special cases

4. Phase 4: UI Refinement (Ongoing)

   • Improve error display (completed)
   • Enhance user interactions (completed)
   • Optimize performance (in progress)

5. Phase 5: Testing and Integration (Upcoming)

   • Write tests
   • Fix bugs
   • Integrate with previous step

Appendix: Function Reference

This section documents the core functions from the original ValidationStep that need to be preserved in the new implementation.

Validation Functions

1. validateRegex - Validates values against regex patterns
2. getValidationError - Determines field-level validation errors
3. validateAndCommit - Validates and commits new values
4. validateData - Validates all data rows
5. validateUpcAndGenerateItemNumbers - Validates UPC codes and generates item numbers

Formatting Functions

1. formatPrice - Formats price values
2. getDisplayValue - Gets formatted display value based on field type
3. isMultiInputType - Checks if field is multi-input type
4. getMultiInputSeparator - Gets separator for multi-input fields
5. isPriceField - Checks if field should be formatted as price

Template Functions

1. loadTemplates - Loads templates from storage
2. saveTemplate - Saves a new template
3. applyTemplate - Applies a template to selected rows
4. getTemplateDisplayText - Gets display text for a template

AI Validation Functions

1. handleAiValidation - Triggers AI validation
2. showCurrentPrompt - Shows current AI prompt
3. getFieldDisplayValue - Gets display value for a field
4. highlightDifferences - Highlights differences between original and corrected values
5. getFieldDisplayValueWithHighlight - Gets display value with highlighted differences
6. revertAiChange - Reverts an AI-suggested change
7. isChangeReverted - Checks if an AI change has been reverted

Event Handlers

1. handleUpcValueUpdate - Handles UPC value updates
2. handleBlur - Handles input blur events
3. handleWheel - Handles wheel events for navigation
4. copyValueDown - Copies a value to cells below
5. handleSkuGeneration - Generates SKUs

By following this refactoring plan, we continue to transform the monolithic ValidationStep component into a modular, maintainable set of components while preserving all existing functionality and aligning with user preferences for design and behavior.