Product import fixes/enhancements

2026-01-25 21:59:57 -05:00
parent 100e398aae
commit ec8ab17d3f
15 changed files with 1161 additions and 127 deletions
--- a/PRODUCT_IMPORT_ENHANCEMENTS.md
+++ b/PRODUCT_IMPORT_ENHANCEMENTS.md
@@ -0,0 +1,375 @@
+# Product Import Module - Enhancement & Issues Outline
+
+This document outlines the investigation and implementation requirements for each requested enhancement to the product import module.
+
+---
+
+## 1. UPC Import - Strip Quotes and Spaces ✅ IMPLEMENTED
+
+**Issue:** When importing UPCs, strip `'`, `"` characters and any spaces, leaving only numbers.
+
+**Implementation (Completed):**
+- Modified `normalizeUpcValue()` in [Import.tsx:661-667](inventory/src/pages/Import.tsx#L661-L667)
+- Strips single quotes, double quotes, smart quotes (`'"`), and whitespace before processing
+- Then handles scientific notation and extracts only digits
+
+**Files Modified:**
+- `inventory/src/pages/Import.tsx` - `normalizeUpcValue()` function
+
+---
+
+## 2. AI Context Columns in Validation Payloads ✅ IMPLEMENTED
+
+**Issue:** The match columns step has a setting to use a field only for AI context (`isAiSupplemental`). Update AI description validation to include any columns selected with this option in the payload. Also include in sanity check payload. Not needed for names.
+
+**Current Implementation:**
+- AI Supplemental toggle: [MatchColumnsStep.tsx:102-118](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L102-L118)
+- AI supplemental data stored in `__aiSupplemental` field on each row
+- Description payload builder: [inlineAiPayload.ts:183-195](inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts#L183-L195)
+
+**Implementation:**
+1. **Update `buildDescriptionValidationPayload()` in `inlineAiPayload.ts`** to include AI supplemental data:
+   ```typescript
+   export const buildDescriptionValidationPayload = (
+     row: Data<string>,
+     fieldOptions: FieldOptionsMap,
+     productLinesCache: Map<string, SelectOption[]>,
+     sublinesCache: Map<string, SelectOption[]>
+   ) => {
+     const payload: Record<string, unknown> = {
+       name: row.name,
+       description: row.description,
+       company_name: getFieldOptionLabel(row.company, fieldOptions, 'company'),
+       company_id: row.company,
+       categories: getFieldOptionLabel(row.category, fieldOptions, 'category'),
+     };
+
+     // Add AI supplemental context if present
+     if (row.__aiSupplemental && typeof row.__aiSupplemental === 'object') {
+       payload.additional_context = row.__aiSupplemental;
+     }
+
+     return payload;
+   };
+   ```
+
+2. **Update sanity check payload** - Locate sanity check submission logic and include `__aiSupplemental` data
+
+3. **Verify `__aiSupplemental` is properly populated** from MatchColumnsStep when columns are marked as AI context only
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/utils/inlineAiPayload.ts`
+- Backend sanity check endpoint (if separate from description validation)
+- Verify data flow in `MatchColumnsStep.tsx` → `ValidationStep`
+
+---
+
+## 3. Fresh Taxonomy Data Per Session ✅ IMPLEMENTED
+
+**Issue:** Ensure taxonomy data is brought in fresh with each session - cache should be invalidated if we exit the import flow and start again.
+
+**Current Implementation:**
+- Field options cached 5 minutes: [ValidationStep/index.tsx:128-133](inventory/src/components/product-import/steps/ValidationStep/index.tsx#L128-L133)
+- Product lines cache: `productLinesCache` in Zustand store
+- Sublines cache: `sublinesCache` in Zustand store
+- Caches set to 10-minute stale time
+
+**Implementation:**
+1. **Add cache invalidation on import flow mount/unmount** in `UploadFlow.tsx`:
+   ```typescript
+   useEffect(() => {
+     // On mount - invalidate import-related query cache
+     queryClient.invalidateQueries({ queryKey: ['import-field-options'] });
+
+     return () => {
+       // On unmount - clear caches
+       queryClient.removeQueries({ queryKey: ['import-field-options'] });
+       queryClient.removeQueries({ queryKey: ['product-lines'] });
+       queryClient.removeQueries({ queryKey: ['sublines'] });
+     };
+   }, []);
+   ```
+
+2. **Clear Zustand store caches** when exiting import flow:
+   - Add action to `validationStore.ts` to clear `productLinesCache` and `sublinesCache`
+   - Call this action on unmount of `UploadFlow` or when navigating away
+
+3. **Consider adding a `sessionId`** that changes on each import flow start, used as part of cache keys
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/UploadFlow.tsx` - Add cleanup effect
+- `inventory/src/components/product-import/steps/ValidationStep/store/validationStore.ts` - Add cache clear action
+- Potentially `inventory/src/components/product-import/steps/ValidationStep/index.tsx` - Query key updates
+
+---
+
+## 4. Save Template from Confirmation Page ✅ IMPLEMENTED
+
+**Issue:** Add option to save rows of submitted data as a new template on the confirmation page after completing the import flow. Verify this works with new validation step changes.
+
+**Current Implementation:**
+- **Import Results section already exists** inline in [Import.tsx:968-1150](inventory/src/pages/Import.tsx#L968-L1150)
+  - Shows created products (lines 1021-1097) with image, name, UPC, item number
+  - Shows errored products (lines 1100-1138) with error details
+  - "Fix products with errors" button resumes validation flow for failed items
+- Template saving logic in ValidationStep: [useTemplateManagement.ts:204-266](inventory/src/components/product-import/steps/ValidationStep/hooks/useTemplateManagement.ts#L204-L266)
+- Saves via `POST /api/templates`
+- `importOutcome.submittedProducts` contains the full product data for each row
+
+**Implementation:**
+1. **Add "Save as Template" button** to each created product row in the results section (around line 1087-1092 in Import.tsx):
+   ```typescript
+   // Add button after the item number display
+   <Button
+     variant="ghost"
+     size="sm"
+     onClick={() => handleSaveAsTemplate(index)}
+   >
+     <BookmarkPlus className="h-4 w-4" />
+   </Button>
+   ```
+
+2. **Add state and dialog** for template saving in Import.tsx:
+   ```typescript
+   const [templateSaveDialogOpen, setTemplateSaveDialogOpen] = useState(false);
+   const [selectedProductForTemplate, setSelectedProductForTemplate] = useState<NormalizedProduct | null>(null);
+   ```
+
+3. **Extract/reuse template save logic** from `useTemplateManagement.ts`:
+   - The `saveNewTemplate()` function (lines 204-266) can be extracted into a shared utility
+   - Or create a `SaveTemplateDialog` component that can be used in both places
+   - Key fields needed: `company` (for template name), `product_type`, and all product field values
+
+4. **Data mapping consideration:**
+   - `importOutcome.submittedProducts` uses `NormalizedProduct` type
+   - Templates expect raw field values - may need to map back from normalized format
+   - Exclude metadata fields: `['id', '__index', '__meta', '__template', '__original', '__corrected', '__changes', '__aiSupplemental']`
+
+**Files to Modify:**
+- `inventory/src/pages/Import.tsx` - Add save template button, state, and dialog
+- Consider creating `inventory/src/components/product-import/SaveTemplateDialog.tsx` for reusability
+- Potentially extract core save logic from `useTemplateManagement.ts` into shared utility
+
+---
+
+## 5. Sheet Preview on Select Sheet Step ✅ IMPLEMENTED
+
+**Issue:** On the select sheet step, show a preview of the first 10 lines or so of each sheet underneath the options.
+
+**Implementation (Completed):**
+- Added `workbook` prop to `SelectSheetStep` component
+- Added `sheetPreviews` memoized computation using `XLSXLib.utils.sheet_to_json()`
+- Shows first 10 rows, 8 columns max per sheet
+- Added `truncateCell()` helper to limit cell content to 30 characters with ellipsis
+- Each sheet option is now a clickable card with:
+  - Radio button and sheet name
+  - Row count indicator
+  - Scrollable preview table with horizontal scroll
+  - Selected state highlighted with primary border
+- Updated `UploadFlow.tsx` to pass workbook prop
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectSheetStep/SelectSheetStep.tsx`
+- `inventory/src/components/product-import/steps/UploadFlow.tsx`
+
+---
+
+## 6. Empty Row Removal ✅ IMPLEMENTED
+
+**Issue:** When importing a sheet, automatically remove completely empty rows.
+
+**Current Implementation:**
+- Empty columns are filtered: [MatchColumnsStep.tsx:616-634](inventory/src/components/product-import/steps/MatchColumnsStep/MatchColumnsStep.tsx#L616-L634)
+- A "Remove empty/duplicates" button exists that removes empty rows, single-value rows, AND duplicates
+- The automatic removal should ONLY remove completely empty rows, not duplicates or single-value rows
+
+**Implementation (Completed):**
+- Added `isRowCompletelyEmpty()` helper function to [SelectHeaderStep.tsx](inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx)
+- Added `useMemo` to filter empty rows on initial data load
+- Uses `Object.values(row)` to check all cell values (matches existing button logic)
+- Only removes rows where ALL values are undefined, null, or whitespace-only strings
+- Manual "Remove Empty/Duplicates" button still available for additional cleanup (duplicates, single-value rows)
+
+**Files Modified:**
+- `inventory/src/components/product-import/steps/SelectHeaderStep/SelectHeaderStep.tsx`
+
+---
+
+## 7. Unit Conversion for Weight/Dimensions ✅ IMPLEMENTED
+
+**Issue:** Add unit conversion feature for weight and dimensions columns - similar to calculator button on cost/msrp, add button that opens popover with options to convert grams → oz, lbs → oz for the whole column at once.
+
+**Current Implementation:**
+- Calculator button on price columns: [ValidationTable.tsx:1491-1627](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1491-L1627)
+- `PriceColumnHeader` component shows calculator icon on hover
+- Weight field defined in config with validation
+
+**Implementation:**
+1. **Create `UnitConversionColumnHeader` component** (similar to `PriceColumnHeader`):
+   ```typescript
+   const UnitConversionColumnHeader = ({ field, table }) => {
+     const [showPopover, setShowPopover] = useState(false);
+
+     const conversions = {
+       weight: [
+         { label: 'Grams → Ounces', factor: 0.035274 },
+         { label: 'Pounds → Ounces', factor: 16 },
+         { label: 'Kilograms → Ounces', factor: 35.274 },
+       ],
+       dimensions: [
+         { label: 'Centimeters → Inches', factor: 0.393701 },
+         { label: 'Millimeters → Inches', factor: 0.0393701 },
+       ]
+     };
+
+     const applyConversion = (factor: number) => {
+       // Batch update all cells in column
+       table.rows.forEach((row, index) => {
+         const currentValue = parseFloat(row[field.key]);
+         if (!isNaN(currentValue)) {
+           updateCell(index, field.key, (currentValue * factor).toFixed(2));
+         }
+       });
+     };
+
+     return (
+       <Popover open={showPopover} onOpenChange={setShowPopover}>
+         <PopoverTrigger>
+           <Scale className="h-4 w-4" /> {/* or similar icon */}
+         </PopoverTrigger>
+         <PopoverContent>
+           {conversions[fieldType].map(conv => (
+             <Button key={conv.label} onClick={() => applyConversion(conv.factor)}>
+               {conv.label}
+             </Button>
+           ))}
+         </PopoverContent>
+       </Popover>
+     );
+   };
+   ```
+
+2. **Identify weight/dimension fields** in config:
+   - `weight_oz`, `length_in`, `width_in`, `height_in` (check actual field keys)
+
+3. **Add to column header render logic** in ValidationTable
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx`
+- Potentially create new component file for `UnitConversionColumnHeader`
+- Update column header rendering to use new component for weight/dimension fields
+
+---
+
+## 8. Expanded MSRP Auto-Fill from Cost ✅ IMPLEMENTED
+
+**Issue:** Expand auto-fill functionality for MSRP from cost - open small popover with options for 2x, 2.1x, 2.2x, 2.3x, 2.4x, 2.5x multipliers, plus checkbox to round up to nearest 9.
+
+**Current Implementation:**
+- Calculator on MSRP column: [ValidationTable.tsx:1540-1584](inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx#L1540-L1584)
+- Currently only does `Cost × 2` then subtracts 0.01 if whole number
+
+**Implementation:**
+1. **Replace simple click with popover** in `PriceColumnHeader`:
+   ```typescript
+   const [selectedMultiplier, setSelectedMultiplier] = useState(2.0);
+   const [roundToNine, setRoundToNine] = useState(false);
+   const multipliers = [2.0, 2.1, 2.2, 2.3, 2.4, 2.5];
+
+   const roundUpToNine = (value: number): number => {
+     // 1.41 → 1.49, 2.78 → 2.79, 12.32 → 12.39
+     const wholePart = Math.floor(value);
+     const decimal = value - wholePart;
+     if (decimal <= 0.09) return wholePart + 0.09;
+     if (decimal <= 0.19) return wholePart + 0.19;
+     // ... continue pattern, or:
+     const lastDigit = Math.floor(decimal * 10);
+     return wholePart + (lastDigit / 10) + 0.09;
+   };
+
+   const calculateMsrp = (cost: number): number => {
+     let result = cost * selectedMultiplier;
+     if (roundToNine) {
+       result = roundUpToNine(result);
+     }
+     return result;
+   };
+   ```
+
+2. **Create popover UI**:
+   ```tsx
+   <Popover>
+     <PopoverTrigger><Calculator className="h-4 w-4" /></PopoverTrigger>
+     <PopoverContent className="w-48">
+       <div className="space-y-2">
+         <Label>Multiplier</Label>
+         <div className="grid grid-cols-3 gap-1">
+           {multipliers.map(m => (
+             <Button
+               key={m}
+               variant={selectedMultiplier === m ? 'default' : 'outline'}
+               size="sm"
+               onClick={() => setSelectedMultiplier(m)}
+             >
+               {m}x
+             </Button>
+           ))}
+         </div>
+         <div className="flex items-center gap-2">
+           <Checkbox checked={roundToNine} onCheckedChange={setRoundToNine} />
+           <Label>Round to .X9</Label>
+         </div>
+         <Button onClick={applyCalculation} className="w-full">
+           Apply
+         </Button>
+       </div>
+     </PopoverContent>
+   </Popover>
+   ```
+
+**Files to Modify:**
+- `inventory/src/components/product-import/steps/ValidationStep/components/ValidationTable.tsx` - `PriceColumnHeader` component
+
+---
+
+## 9. Debug Mode - Skip API Submission ✅ IMPLEMENTED
+
+**Issue:** Add a third switch in the footer of image upload step (visible only to users with `admin:debug` permission) that will not submit data to any API, only complete the process and show results page as if it had worked.
+
+**Implementation (Completed):**
+- Added `skipApiSubmission` state to `ImageUploadStep.tsx`
+- Added amber-colored "Skip API (Debug)" switch (visible only with `admin:debug` permission)
+- When skip is active, "Use Test API" and "Use Test Database" switches are hidden
+- Added `skipApiSubmission?: boolean` to `SubmitOptions` type in `types.ts`
+- In `Import.tsx`, when `skipApiSubmission` is true:
+  - Skips the actual API call entirely
+  - Generates mock success response with mock PIDs
+  - Shows `[DEBUG]` prefix in toast and result message
+  - Displays results page as if submission succeeded
+
+**Files Modified:**
+- `inventory/src/components/product-import/types.ts` - Added `skipApiSubmission` to `SubmitOptions`
+- `inventory/src/components/product-import/steps/ImageUploadStep/ImageUploadStep.tsx` - Added switch UI
+- `inventory/src/pages/Import.tsx` - Added skip logic in `handleData()`
+
+---
+
+## Summary
+
+| # | Enhancement | Complexity | Status |
+|---|-------------|------------|--------|
+| 1 | Strip UPC quotes/spaces | Low | ✅ Implemented |
+| 2 | AI context in validation | Medium | ✅ Implemented |
+| 3 | Fresh taxonomy per session | Medium | ✅ Implemented |
+| 4 | Save template from confirmation | Medium-High | ✅ Implemented |
+| 5 | Sheet preview | Low-Medium | ✅ Implemented |
+| 6 | Remove empty rows | Low | ✅ Implemented |
+| 7 | Unit conversion | Medium | ✅ Implemented |
+| 8 | MSRP multiplier options | Medium | ✅ Implemented |
+| 9 | Debug skip API | Low-Medium | ✅ Implemented |
+
+**Implemented:** 9 of 9 items - All enhancements complete!
+
+---
+
+*Document generated: 2026-01-25*