Refactor validation error handling to use a single source of truth (validationErrors Map), speed up item number generation and loading lines/sublines

docs/validation-process-issues.md

@@ -62,14 +62,78 @@ const validationTimeoutsRef = useRef<Record<number, NodeJS.Timeout>>({});
 
 While there is cleanup on unmount, if rows are added/removed dynamically, timeouts for deleted rows might not be properly cleared.
 
-## 5. Inefficient Error Storage
+## 5. ✅ Inefficient Error Storage (RESOLVED)
 
-Errors are stored in multiple places:
-- In the `validationErrors` Map
+**Status: RESOLVED**
+
+### Problem
+
+Previously, validation errors were stored in multiple locations:
+- In the `validationErrors` Map in `useValidationState`
 - In the row data itself as `__errors`
-- In the UPC validation results
 
-This duplication makes it harder to maintain a single source of truth and could lead to inconsistencies.
+This redundancy caused several issues:
+- Inconsistent error states between the two storage locations
+- Increased memory usage by storing the same information twice
+- Complex state management to keep both sources in sync
+- Difficulty reasoning about where errors should be accessed from
+
+### Solution
+
+We've implemented a unified error storage approach by:
+- Making the `validationErrors` Map in `useValidationState` the single source of truth for all validation errors
+- Removed the `__errors` property from row data
+- Updated all validation functions to interact with the central error store instead of modifying row data
+- Modified UPC validation to use the central error store
+- Updated all components to read errors from the `validationErrors` Map instead of row data
+
+### Key Changes
+
+1. Modified `dataMutations.ts` to stop storing errors in row data
+2. Updated the `Meta` type to remove the `__errors` property
+3. Modified the `RowData` type to remove the `__errors` property
+4. Updated the `useValidation` hook to return errors separately from row data
+5. Modified the `useAiValidation` hook to work with the central error store
+6. Updated the `useFilters` hook to check for errors in the `validationErrors` Map
+7. Modified the `ValidationTable` and `ValidationCell` components to read errors from the `validationErrors` Map
+
+### Benefits
+
+- **Single Source of Truth**: All validation errors are now stored in one place
+- **Reduced Memory Usage**: No duplicate storage of error information
+- **Simplified State Management**: Only one state to update when errors change
+- **Cleaner Data Structure**: Row data no longer contains validation metadata
+- **More Maintainable Code**: Clearer separation of concerns between data and validation
+
+### Future Improvements
+
+While this refactoring addresses the core issue of inefficient error storage, there are still opportunities for further optimization:
+
+1. **Redundant Error Processing**: The validation process still performs some redundant calculations that could be optimized.
+2. **Race Conditions**: Async validation can lead to race conditions when multiple validations are triggered in quick succession.
+3. **Memory Leaks**: The timeout management for validation could be improved to prevent potential memory leaks.
+4. **Tight Coupling**: Components are still tightly coupled to the validation state structure.
+5. **Error Prioritization**: The system doesn't prioritize errors well, showing all errors at once rather than focusing on the most critical ones first.
+
+### Validation Flow
+
+The validation process now works as follows:
+
+1. **Error Generation**:
+   - Field-level validations generate errors based on validation rules
+   - Row-level hooks add custom validation errors
+   - Table-level validations (like uniqueness checks) add errors across rows
+
+2. **Error Storage**:
+   - All errors are stored in the `validationErrors` Map in `useValidationState`
+   - The Map uses row indices as keys and objects of field errors as values
+
+3. **Error Display**:
+   - The `ValidationTable` component checks the `validationErrors` Map to highlight rows with errors
+   - The `ValidationCell` component receives errors for specific fields from the `validationErrors` Map
+   - Errors are filtered in the UI to avoid showing "required" errors for fields with values
+
+This focused refactoring approach has successfully addressed a critical issue while keeping changes manageable and targeted.
 
 ## 6. Excessive Re-rendering