80 KiB
80 KiB
AI Validation System Redesign
Status: Design Document (not yet implemented) Date: January 2025 Goal: Replace expensive batch AI validation with a multi-tier, real-time validation system
Table of Contents
- Executive Summary
- Current State Analysis
- Proposed Architecture
- Backend Implementation
- Frontend Implementation
- Database Schema
- Migration Strategy
- Cost Analysis
Executive Summary
Problem
The current AI validation system:
- Costs $0.30-0.50 per run (GPT-5.2 with reasoning)
- Takes 3-4 minutes to process
- Sends a giant prompt (18KB+ instructions + full taxonomy + all products)
- Processes everything in one batch at the end of the workflow
Solution
A multi-tier validation system that:
- Performs deterministic validation in code (no AI needed for formatting)
- Uses Groq (Llama 3.3) for real-time field suggestions (~200ms, ~$0.001/product)
- Pre-computes category/theme embeddings for fast similarity search
- Reserves expensive models only for complex description generation
- Validates incrementally as user works, not all at once
Expected Outcomes
| Metric | Current | Proposed |
|---|---|---|
| Cost per 50 products | $0.30-0.50 | $0.02-0.05 |
| Processing time | 3-4 minutes | Real-time + 10-30s batch |
| User experience | Wait at end | Inline suggestions |
Current State Analysis
What the Current System Does
Based on the general prompt in ai_prompts table, validation performs these tasks:
| Task | AI Required? | Proposed Approach |
|---|---|---|
Price formatting ($5.00 → 5.00) |
❌ No | JavaScript normalizer |
| UPC/SKU trimming | ❌ No | JavaScript normalizer |
Country code conversion (USA → US) |
❌ No | Lookup table |
| Date formatting (ETA field) | ❌ No | Date parser |
| Category assignment | ⚠️ Partial | Embeddings + small model |
| Theme detection | ⚠️ Partial | Embeddings + small model |
| Color extraction | ⚠️ Partial | Small model |
| Name standardization | ✅ Yes | Medium model |
| Description enhancement | ✅ Yes | Medium model (streaming) |
| Weight/dimension consistency | ✅ Yes | Batch comparison |
| Tax code assignment | ⚠️ Partial | Rules + small model |
Current Prompt Structure
[System Instructions] ~200 chars
[General Prompt] ~18,000 chars (field guidelines, naming rules, etc.)
[Company-Specific] ~variable
[Taxonomy Data] ~50,000-100,000 chars (categories, themes, colors, etc.)
[Product Data] ~variable (all products as JSON)
Total prompt size: Often 100,000+ characters = ~25,000+ tokens input
Current Model Usage
- Model: GPT-5.2 (reasoning model)
- Reasoning effort: Medium
- Max output tokens: 50,000
- Response format: Strict JSON schema
Proposed Architecture
System Overview
┌─────────────────────────────────────────────────────────────────────────────┐
│ VALIDATION STEP UI │
├─────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Field Edit │ │ Row Navigation │ │ Batch Action │ │
│ │ (on blur) │ │ (on row change)│ │ (user clicks) │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
└───────────┼─────────────────────┼─────────────────────┼─────────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ TIER 1: CLIENT-SIDE │
│ (Deterministic - No AI) │
├───────────────────────────────────────────────────────────────────────────┤
│ • Price formatting • UPC trimming • Country codes │
│ • Numeric validation • Required fields • Date parsing │
└───────────────────────────────────────────────────────────────────────────┘
│ │ │
▼ ▼ ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ TIER 2: REAL-TIME AI SUGGESTIONS │
│ (Groq - Llama 3.3 70B) │
├───────────────────────────────────────────────────────────────────────────┤
│ │
│ POST /api/ai/suggest/:field │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Name │ │ Category │ │ Theme │ │ Color │ │ Tax Code │ │
│ │ Task │ │ Task │ │ Task │ │ Task │ │ Task │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │ │ │ │ │
│ │ ▼ │ │ │ │
│ │ ┌────────────────┐ │ │ │ │
│ │ │ Embedding │ │ │ │ │
│ │ │ Pre-Filter │ │ │ │ │
│ │ └────────────────┘ │ │ │ │
│ │ │ │ │ │ │
└───────┼──────────────┼─────────────┼─────────────┼─────────────┼──────────┘
│ │ │ │ │
▼ ▼ ▼ ▼ ▼
┌───────────────────────────────────────────────────────────────────────────┐
│ TIER 3: BATCH AI ENHANCEMENT │
│ (Claude Haiku / GPT-4o-mini) │
├───────────────────────────────────────────────────────────────────────────┤
│ │
│ POST /api/ai/enhance/descriptions │
│ POST /api/ai/validate/consistency │
│ │
│ ┌────────────────────┐ ┌────────────────────┐ │
│ │ Description │ │ Cross-Product │ │
│ │ Generation │ │ Consistency │ │
│ │ (10-20 at once) │ │ (weights/dims) │ │
│ └────────────────────┘ └────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────────────┘
Validation Flow
User enters/maps data
│
▼
┌───────────────────┐
│ Tier 1: Code │ ◄── Instant (0ms)
│ - Format prices │
│ - Validate UPCs │
│ - Convert country │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Tier 2: Groq │ ◄── Fast (200-500ms per field)
│ - Suggest category│ Triggered on focus/blur
│ - Detect themes │
│ - Format name │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Tier 3: Batch │ ◄── On-demand (5-30s)
│ - Descriptions │ User clicks "Enhance"
│ - Consistency │
└───────────────────┘
Backend Implementation
File Structure
inventory-server/src/
├── services/
│ └── ai/
│ ├── index.js # Main entry, initialization, exports
│ ├── config.js # AI configuration management
│ ├── taskRegistry.js # Task registration system
│ ├── workQueue.js # Concurrency control
│ │
│ ├── providers/
│ │ ├── index.js # Provider factory
│ │ ├── groqProvider.js # Groq API client (chat)
│ │ ├── openaiProvider.js # OpenAI API client (embeddings + chat)
│ │ └── anthropicProvider.js # Claude API client (batch tasks)
│ │
│ ├── embeddings/
│ │ ├── index.js # Embedding service entry
│ │ ├── categoryEmbeddings.js # Category embedding management
│ │ ├── themeEmbeddings.js # Theme embedding management
│ │ ├── vectorStore.js # In-memory vector storage
│ │ └── similarity.js # Cosine similarity utilities
│ │
│ ├── tasks/
│ │ ├── index.js # Task exports
│ │ ├── nameSuggestionTask.js
│ │ ├── categorySuggestionTask.js
│ │ ├── themeSuggestionTask.js
│ │ ├── colorSuggestionTask.js
│ │ ├── taxCodeSuggestionTask.js
│ │ ├── descriptionEnhanceTask.js
│ │ ├── consistencyCheckTask.js
│ │ └── utils/
│ │ ├── productUtils.js # Product data helpers
│ │ └── responseParser.js # AI response parsing
│ │
│ ├── prompts/
│ │ ├── index.js # Prompt exports
│ │ ├── namePrompts.js
│ │ ├── categoryPrompts.js
│ │ ├── themePrompts.js
│ │ ├── colorPrompts.js
│ │ ├── descriptionPrompts.js
│ │ └── consistencyPrompts.js
│ │
│ └── normalizers/
│ ├── index.js # Normalizer exports
│ ├── priceNormalizer.js
│ ├── upcNormalizer.js
│ ├── countryCodeNormalizer.js
│ ├── dateNormalizer.js
│ └── numericNormalizer.js
│
└── routes/
└── ai.js # New AI routes (replaces ai-validation.js)
Provider System
Provider Interface
// services/ai/providers/index.js
/**
* All providers must implement this interface
*/
class AIProvider {
/**
* Chat completion
* @param {Object} params
* @param {Array<{role: string, content: string}>} params.messages
* @param {string} params.model
* @param {number} [params.temperature=0.3]
* @param {number} [params.maxTokens=500]
* @param {Object} [params.responseFormat] - JSON schema for structured output
* @param {number} [params.timeoutMs=30000]
* @returns {Promise<{content: string, parsed: Object|null, usage: Object, latencyMs: number, model: string}>}
*/
async chatCompletion(params) {
throw new Error('Not implemented');
}
/**
* Generate embeddings
* @param {string|string[]} input - Text or array of texts
* @param {Object} [options]
* @param {string} [options.model]
* @param {number} [options.dimensions]
* @returns {Promise<{embeddings: number[][], usage: Object, model: string, latencyMs: number}>}
*/
async embed(input, options) {
throw new Error('Not implemented');
}
}
/**
* Provider factory
*/
function createProvider(providerName, config) {
switch (providerName.toLowerCase()) {
case 'groq':
return new GroqProvider(config.providers.groq);
case 'openai':
return new OpenAIProvider(config.providers.openai);
case 'anthropic':
return new AnthropicProvider(config.providers.anthropic);
default:
throw new Error(`Unknown provider: ${providerName}`);
}
}
module.exports = { AIProvider, createProvider };
Groq Provider (Primary for Real-Time)
// services/ai/providers/groqProvider.js
const Groq = require('groq-sdk');
class GroqProvider {
constructor({ apiKey, baseUrl, timeoutMs = 30000 }) {
if (!apiKey) {
throw new Error('Groq API key is required');
}
this.client = new Groq({ apiKey, baseURL: baseUrl });
this.timeoutMs = timeoutMs;
}
async chatCompletion({
messages,
model = 'llama-3.3-70b-versatile',
temperature = 0.3,
maxTokens = 500,
responseFormat = null,
timeoutMs = this.timeoutMs
}) {
const started = Date.now();
const params = {
messages,
model,
temperature,
max_tokens: maxTokens
};
// Add JSON mode if requested
if (responseFormat) {
params.response_format = { type: 'json_object' };
}
const response = await this.client.chat.completions.create(params, {
timeout: timeoutMs
});
const content = response.choices[0]?.message?.content || '';
const usage = response.usage || {};
// Try to parse JSON if response format was requested
let parsed = null;
if (responseFormat && content) {
try {
parsed = JSON.parse(content);
} catch {
// Will return raw content
}
}
return {
content,
parsed,
usage: {
promptTokens: usage.prompt_tokens || 0,
completionTokens: usage.completion_tokens || 0,
totalTokens: usage.total_tokens || 0
},
latencyMs: Date.now() - started,
model: response.model || model
};
}
// Groq doesn't support embeddings, so this throws
async embed() {
throw new Error('Groq does not support embeddings. Use OpenAI provider.');
}
}
module.exports = { GroqProvider };
OpenAI Provider (Embeddings + Fallback Chat)
// services/ai/providers/openaiProvider.js
const MAX_EMBEDDING_BATCH_SIZE = 2048;
class OpenAIProvider {
constructor({
apiKey,
baseUrl = 'https://api.openai.com/v1',
embeddingModel = 'text-embedding-3-small',
embeddingDimensions = 1536,
chatModel = 'gpt-4o-mini',
timeoutMs = 60000
}) {
if (!apiKey) {
throw new Error('OpenAI API key is required');
}
this.apiKey = apiKey;
this.baseUrl = baseUrl;
this.embeddingModel = embeddingModel;
this.embeddingDimensions = embeddingDimensions;
this.chatModel = chatModel;
this.timeoutMs = timeoutMs;
}
async chatCompletion({
messages,
model = this.chatModel,
temperature = 0.3,
maxTokens = 500,
responseFormat = null,
timeoutMs = this.timeoutMs
}) {
const started = Date.now();
const body = {
model,
messages,
temperature,
max_tokens: maxTokens
};
if (responseFormat) {
body.response_format = { type: 'json_object' };
}
const response = await this._makeRequest('chat/completions', body, timeoutMs);
const content = response.choices[0]?.message?.content || '';
const usage = response.usage || {};
let parsed = null;
if (responseFormat && content) {
try {
parsed = JSON.parse(content);
} catch {
// Will return raw content
}
}
return {
content,
parsed,
usage: {
promptTokens: usage.prompt_tokens || 0,
completionTokens: usage.completion_tokens || 0,
totalTokens: usage.total_tokens || 0
},
latencyMs: Date.now() - started,
model: response.model || model
};
}
/**
* Generate embeddings for a single text or batch
*/
async embed(input, options = {}) {
const texts = Array.isArray(input) ? input : [input];
const model = options.model || this.embeddingModel;
const dimensions = options.dimensions || this.embeddingDimensions;
const timeoutMs = options.timeoutMs || this.timeoutMs;
if (texts.length > MAX_EMBEDDING_BATCH_SIZE) {
throw new Error(`Batch size ${texts.length} exceeds max of ${MAX_EMBEDDING_BATCH_SIZE}`);
}
const started = Date.now();
// Clean input texts
const cleanedTexts = texts.map(t =>
(t || '').replace(/\n+/g, ' ').trim().substring(0, 8000)
);
const body = {
input: cleanedTexts,
model,
encoding_format: 'float'
};
// Only embedding-3 models support dimensions parameter
if (model.includes('embedding-3')) {
body.dimensions = dimensions;
}
const response = await this._makeRequest('embeddings', body, timeoutMs);
// Sort by index to ensure order matches input
const sortedData = response.data.sort((a, b) => a.index - b.index);
const embeddings = sortedData.map(item => item.embedding);
return {
embeddings,
usage: {
promptTokens: response.usage?.prompt_tokens || 0,
totalTokens: response.usage?.total_tokens || 0
},
model: response.model || model,
latencyMs: Date.now() - started
};
}
/**
* Generator for processing large batches in chunks
*/
async *embedBatchChunked(texts, options = {}) {
const batchSize = Math.min(options.batchSize || 100, MAX_EMBEDDING_BATCH_SIZE);
for (let i = 0; i < texts.length; i += batchSize) {
const chunk = texts.slice(i, i + batchSize);
const result = await this.embed(chunk, options);
yield {
embeddings: result.embeddings,
startIndex: i,
endIndex: i + chunk.length,
usage: result.usage,
model: result.model,
latencyMs: result.latencyMs
};
}
}
async _makeRequest(endpoint, body, timeoutMs) {
const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), timeoutMs);
try {
const response = await fetch(`${this.baseUrl}/${endpoint}`, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Authorization': `Bearer ${this.apiKey}`
},
body: JSON.stringify(body),
signal: controller.signal
});
if (!response.ok) {
const error = await response.json().catch(() => ({}));
throw new Error(error.error?.message || `OpenAI API error: ${response.status}`);
}
return response.json();
} finally {
clearTimeout(timeout);
}
}
}
module.exports = { OpenAIProvider, MAX_EMBEDDING_BATCH_SIZE };
Anthropic Provider (Batch Enhancement)
// services/ai/providers/anthropicProvider.js
const Anthropic = require('@anthropic-ai/sdk');
class AnthropicProvider {
constructor({
apiKey,
defaultModel = 'claude-3-5-haiku-20241022',
timeoutMs = 120000
}) {
if (!apiKey) {
throw new Error('Anthropic API key is required');
}
this.client = new Anthropic({ apiKey });
this.defaultModel = defaultModel;
this.timeoutMs = timeoutMs;
}
async chatCompletion({
messages,
model = this.defaultModel,
temperature = 0.3,
maxTokens = 1000,
system = null,
timeoutMs = this.timeoutMs
}) {
const started = Date.now();
// Anthropic uses separate system parameter
const params = {
model,
max_tokens: maxTokens,
temperature,
messages: messages.filter(m => m.role !== 'system')
};
// Extract system message if present
const systemMessage = system || messages.find(m => m.role === 'system')?.content;
if (systemMessage) {
params.system = systemMessage;
}
const response = await this.client.messages.create(params);
const content = response.content
.filter(block => block.type === 'text')
.map(block => block.text)
.join('');
let parsed = null;
try {
parsed = JSON.parse(content);
} catch {
// Not JSON
}
return {
content,
parsed,
usage: {
promptTokens: response.usage?.input_tokens || 0,
completionTokens: response.usage?.output_tokens || 0,
totalTokens: (response.usage?.input_tokens || 0) + (response.usage?.output_tokens || 0)
},
latencyMs: Date.now() - started,
model: response.model || model
};
}
async embed() {
throw new Error('Anthropic does not support embeddings. Use OpenAI provider.');
}
}
module.exports = { AnthropicProvider };
Task Registry
// services/ai/taskRegistry.js
/**
* Registry for AI tasks
* Manages task registration, lookup, and execution
*/
class AiTaskRegistry {
constructor() {
this.tasks = new Map();
}
/**
* Register a task
* @param {Object} taskDefinition
* @param {string} taskDefinition.id - Unique task identifier
* @param {string} taskDefinition.description - Human-readable description
* @param {Function} taskDefinition.run - Async function to execute
* @param {Object} [taskDefinition.config] - Task-specific configuration
*/
register(taskDefinition) {
if (!taskDefinition?.id) {
throw new Error('Task must have an id');
}
if (typeof taskDefinition.run !== 'function') {
throw new Error(`Task ${taskDefinition.id} must have a run function`);
}
if (this.tasks.has(taskDefinition.id)) {
throw new Error(`Task ${taskDefinition.id} is already registered`);
}
this.tasks.set(taskDefinition.id, taskDefinition);
return this;
}
/**
* Get a task by ID
*/
get(taskId) {
return this.tasks.get(taskId) || null;
}
/**
* Check if a task exists
*/
has(taskId) {
return this.tasks.has(taskId);
}
/**
* List all registered task IDs
*/
list() {
return Array.from(this.tasks.keys());
}
}
/**
* Task IDs as frozen constants
*/
const TASK_IDS = Object.freeze({
// Tier 2: Real-time suggestions
SUGGEST_NAME: 'suggest.name',
SUGGEST_CATEGORIES: 'suggest.categories',
SUGGEST_THEMES: 'suggest.themes',
SUGGEST_COLORS: 'suggest.colors',
SUGGEST_TAX_CODE: 'suggest.taxCode',
SUGGEST_SIZE_CATEGORY: 'suggest.sizeCategory',
// Tier 3: Batch enhancement
ENHANCE_DESCRIPTIONS: 'enhance.descriptions',
CHECK_CONSISTENCY: 'check.consistency',
// Utility tasks
COMPUTE_EMBEDDINGS: 'util.computeEmbeddings'
});
module.exports = { AiTaskRegistry, TASK_IDS };
Work Queue (Concurrency Control)
// services/ai/workQueue.js
/**
* Simple concurrent work queue
* Prevents overwhelming AI providers with too many parallel requests
*/
class AiWorkQueue {
constructor(concurrency = 3) {
this.concurrency = Math.max(1, concurrency);
this.active = 0;
this.queue = [];
}
/**
* Enqueue a task for execution
* @param {Function} taskFactory - Async function to execute
* @returns {Promise} - Resolves with task result
*/
enqueue(taskFactory) {
return new Promise((resolve, reject) => {
const execute = async () => {
this.active += 1;
try {
const result = await taskFactory();
resolve(result);
} catch (error) {
reject(error);
} finally {
this.active -= 1;
this._processNext();
}
};
if (this.active < this.concurrency) {
execute();
} else {
this.queue.push(execute);
}
});
}
_processNext() {
if (this.queue.length === 0 || this.active >= this.concurrency) {
return;
}
const next = this.queue.shift();
if (next) {
next();
}
}
/**
* Get current queue statistics
*/
getStats() {
return {
active: this.active,
queued: this.queue.length,
concurrency: this.concurrency
};
}
}
module.exports = { AiWorkQueue };
Embedding System
Embedding Service Entry
// services/ai/embeddings/index.js
const { CategoryEmbeddings } = require('./categoryEmbeddings');
const { ThemeEmbeddings } = require('./themeEmbeddings');
const { VectorStore } = require('./vectorStore');
let categoryEmbeddings = null;
let themeEmbeddings = null;
let initialized = false;
/**
* Initialize the embedding system
* Should be called once at server startup
*/
async function initializeEmbeddings({ openaiProvider, mysqlConnection, logger }) {
if (initialized) {
return { categoryEmbeddings, themeEmbeddings };
}
logger?.info('[Embeddings] Initializing embedding system...');
// Initialize category embeddings
categoryEmbeddings = new CategoryEmbeddings({
provider: openaiProvider,
connection: mysqlConnection,
logger
});
// Initialize theme embeddings
themeEmbeddings = new ThemeEmbeddings({
provider: openaiProvider,
connection: mysqlConnection,
logger
});
// Load or compute embeddings
await Promise.all([
categoryEmbeddings.initialize(),
themeEmbeddings.initialize()
]);
initialized = true;
logger?.info('[Embeddings] Embedding system initialized');
return { categoryEmbeddings, themeEmbeddings };
}
/**
* Get category suggestions for a product
*/
async function suggestCategories(productText, topK = 10) {
if (!categoryEmbeddings) {
throw new Error('Embeddings not initialized');
}
return categoryEmbeddings.findSimilar(productText, topK);
}
/**
* Get theme suggestions for a product
*/
async function suggestThemes(productText, topK = 5) {
if (!themeEmbeddings) {
throw new Error('Embeddings not initialized');
}
return themeEmbeddings.findSimilar(productText, topK);
}
module.exports = {
initializeEmbeddings,
suggestCategories,
suggestThemes
};
Category Embeddings
// services/ai/embeddings/categoryEmbeddings.js
const { VectorStore } = require('./vectorStore');
const { cosineSimilarity } = require('./similarity');
class CategoryEmbeddings {
constructor({ provider, connection, logger }) {
this.provider = provider;
this.connection = connection;
this.logger = logger;
this.vectorStore = new VectorStore();
this.categories = []; // Raw category data
}
/**
* Initialize embeddings - load from cache or compute
*/
async initialize() {
this.logger?.info('[CategoryEmbeddings] Loading categories from database...');
// Fetch hierarchical categories
const [rows] = await this.connection.query(`
SELECT
cat_id,
name,
master_cat_id,
type
FROM product_categories
WHERE type IN (10, 11, 12, 13)
ORDER BY type, name
`);
// Build category paths (e.g., "Paper > Patterned Paper > 12x12 Single Sheets")
this.categories = this._buildCategoryPaths(rows);
this.logger?.info(`[CategoryEmbeddings] Built ${this.categories.length} category paths`);
// Check if we have cached embeddings
const cached = await this._loadCachedEmbeddings();
if (cached && cached.length === this.categories.length) {
this.logger?.info('[CategoryEmbeddings] Using cached embeddings');
this.vectorStore.load(cached);
return;
}
// Compute new embeddings
await this._computeAndCacheEmbeddings();
}
/**
* Find similar categories for a product
*/
async findSimilar(productText, topK = 10) {
// Get embedding for product
const { embeddings } = await this.provider.embed(productText);
const productEmbedding = embeddings[0];
// Find most similar categories
const results = this.vectorStore.search(productEmbedding, topK);
// Enrich with category data
return results.map(result => {
const category = this.categories.find(c => c.id === result.id);
return {
id: category.id,
name: category.name,
fullPath: category.fullPath,
parentId: category.parentId,
similarity: result.similarity
};
});
}
/**
* Build full paths for categories
*/
_buildCategoryPaths(rows) {
const byId = new Map(rows.map(r => [r.cat_id, r]));
const categories = [];
for (const row of rows) {
const path = [];
let current = row;
// Walk up the tree to build full path
while (current) {
path.unshift(current.name);
current = current.master_cat_id ? byId.get(current.master_cat_id) : null;
}
categories.push({
id: row.cat_id,
name: row.name,
parentId: row.master_cat_id,
type: row.type,
fullPath: path.join(' > '),
// Text for embedding includes path for context
embeddingText: path.join(' ')
});
}
return categories;
}
/**
* Compute embeddings for all categories
*/
async _computeAndCacheEmbeddings() {
this.logger?.info('[CategoryEmbeddings] Computing embeddings...');
const texts = this.categories.map(c => c.embeddingText);
const allEmbeddings = [];
// Process in chunks (OpenAI has batch limits)
for await (const chunk of this.provider.embedBatchChunked(texts, { batchSize: 100 })) {
for (let i = 0; i < chunk.embeddings.length; i++) {
const globalIndex = chunk.startIndex + i;
allEmbeddings.push({
id: this.categories[globalIndex].id,
embedding: chunk.embeddings[i]
});
}
this.logger?.info(`[CategoryEmbeddings] Processed ${chunk.endIndex}/${texts.length}`);
}
// Store in vector store
this.vectorStore.load(allEmbeddings);
// Cache to database for faster startup next time
await this._cacheEmbeddings(allEmbeddings);
this.logger?.info('[CategoryEmbeddings] Embeddings computed and cached');
}
async _loadCachedEmbeddings() {
// TODO: Load from ai_embeddings_cache table
return null;
}
async _cacheEmbeddings(embeddings) {
// TODO: Save to ai_embeddings_cache table
}
}
module.exports = { CategoryEmbeddings };
Vector Store
// services/ai/embeddings/vectorStore.js
const { cosineSimilarity } = require('./similarity');
/**
* In-memory vector store for fast similarity search
*/
class VectorStore {
constructor() {
this.vectors = []; // Array of { id, embedding }
}
/**
* Load vectors into the store
*/
load(vectors) {
this.vectors = vectors;
}
/**
* Add a single vector
*/
add(id, embedding) {
this.vectors.push({ id, embedding });
}
/**
* Search for most similar vectors
*/
search(queryEmbedding, topK = 10) {
const scored = this.vectors.map(item => ({
id: item.id,
similarity: cosineSimilarity(queryEmbedding, item.embedding)
}));
// Sort by similarity descending
scored.sort((a, b) => b.similarity - a.similarity);
return scored.slice(0, topK);
}
/**
* Get store size
*/
size() {
return this.vectors.length;
}
/**
* Clear the store
*/
clear() {
this.vectors = [];
}
}
module.exports = { VectorStore };
Similarity Utilities
// services/ai/embeddings/similarity.js
/**
* Compute cosine similarity between two vectors
*/
function cosineSimilarity(a, b) {
if (a.length !== b.length) {
throw new Error('Vectors must have same length');
}
let dotProduct = 0;
let normA = 0;
let normB = 0;
for (let i = 0; i < a.length; i++) {
dotProduct += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
const denominator = Math.sqrt(normA) * Math.sqrt(normB);
if (denominator === 0) return 0;
return dotProduct / denominator;
}
/**
* Compute Euclidean distance between two vectors
*/
function euclideanDistance(a, b) {
if (a.length !== b.length) {
throw new Error('Vectors must have same length');
}
let sum = 0;
for (let i = 0; i < a.length; i++) {
const diff = a[i] - b[i];
sum += diff * diff;
}
return Math.sqrt(sum);
}
module.exports = { cosineSimilarity, euclideanDistance };
Micro-Prompt Tasks
Name Suggestion Task
// services/ai/tasks/nameSuggestionTask.js
const { buildNamePrompt } = require('../prompts/namePrompts');
function createNameSuggestionTask({ provider, logger, config }) {
const taskConfig = config.tasks?.nameSuggestion || {};
async function run({ product }) {
if (!product?.name && !product?.description) {
return { suggestion: null, reason: 'No name or description provided' };
}
const prompt = buildNamePrompt(product);
const response = await provider.chatCompletion({
messages: [{ role: 'user', content: prompt }],
model: taskConfig.model || 'llama-3.3-70b-versatile',
temperature: taskConfig.temperature || 0.2,
maxTokens: taskConfig.maxTokens || 150
});
const suggestion = response.content.trim();
// Only return if different from original
if (suggestion === product.name) {
return { suggestion: null, unchanged: true };
}
return {
suggestion,
original: product.name,
usage: response.usage,
latencyMs: response.latencyMs
};
}
return {
id: 'suggest.name',
description: 'Suggest formatted product name',
run
};
}
module.exports = { createNameSuggestionTask };
Category Suggestion Task
// services/ai/tasks/categorySuggestionTask.js
const { suggestCategories } = require('../embeddings');
const { buildCategoryPrompt } = require('../prompts/categoryPrompts');
function createCategorySuggestionTask({ provider, logger, config }) {
const taskConfig = config.tasks?.categorySuggestion || {};
async function run({ product }) {
const productText = `${product.name || ''} ${product.description || ''}`.trim();
if (!productText) {
return { suggestions: [], reason: 'No product text provided' };
}
// Step 1: Get top candidates via embedding similarity
const embeddingMatches = await suggestCategories(productText, 10);
// Step 2: Use AI to pick best matches from candidates
const prompt = buildCategoryPrompt(product, embeddingMatches);
const response = await provider.chatCompletion({
messages: [{ role: 'user', content: prompt }],
model: taskConfig.model || 'llama-3.1-8b-instant',
temperature: taskConfig.temperature || 0.1,
maxTokens: taskConfig.maxTokens || 100,
responseFormat: { type: 'json_object' }
});
let selectedIds = [];
try {
const parsed = JSON.parse(response.content);
selectedIds = Array.isArray(parsed.categories) ? parsed.categories : parsed;
} catch {
logger?.warn('[CategorySuggestion] Failed to parse response', { content: response.content });
}
// Filter matches to only selected IDs
const suggestions = embeddingMatches
.filter(m => selectedIds.includes(m.id))
.map(m => ({
id: m.id,
name: m.name,
fullPath: m.fullPath,
similarity: m.similarity
}));
return {
suggestions,
allMatches: embeddingMatches, // Include all for fallback
usage: response.usage,
latencyMs: response.latencyMs
};
}
return {
id: 'suggest.categories',
description: 'Suggest product categories',
run
};
}
module.exports = { createCategorySuggestionTask };
Description Enhancement Task
// services/ai/tasks/descriptionEnhanceTask.js
const { buildDescriptionPrompt } = require('../prompts/descriptionPrompts');
function createDescriptionEnhanceTask({ provider, logger, config }) {
const taskConfig = config.tasks?.descriptionEnhance || {};
/**
* Enhance descriptions for a batch of products
*/
async function run({ products, mode = 'enhance' }) {
if (!Array.isArray(products) || products.length === 0) {
return { results: [], reason: 'No products provided' };
}
// Process in smaller batches for reliability
const batchSize = taskConfig.batchSize || 10;
const results = [];
for (let i = 0; i < products.length; i += batchSize) {
const batch = products.slice(i, i + batchSize);
const batchResults = await processBatch(batch, mode);
results.push(...batchResults);
}
return { results };
}
async function processBatch(products, mode) {
const prompt = buildDescriptionPrompt(products, mode);
const response = await provider.chatCompletion({
messages: [
{ role: 'system', content: getSystemPrompt() },
{ role: 'user', content: prompt }
],
model: taskConfig.model || 'claude-3-5-haiku-20241022',
temperature: taskConfig.temperature || 0.7,
maxTokens: taskConfig.maxTokens || 2000,
responseFormat: { type: 'json_object' }
});
let parsed = [];
try {
const result = JSON.parse(response.content);
parsed = result.descriptions || result;
} catch (error) {
logger?.warn('[DescriptionEnhance] Failed to parse response', { error: error.message });
}
// Match results back to products
return products.map((product, index) => {
const enhanced = parsed[index] || {};
return {
productId: product._index || product.upc || index,
original: product.description,
enhanced: enhanced.description || null,
changed: enhanced.description && enhanced.description !== product.description
};
});
}
function getSystemPrompt() {
return `You are a product copywriter for a craft supplies ecommerce store.
Write SEO-friendly, accurate descriptions that help customers understand what they're buying.
Always state what's included. Never use "our" - use "this" or the company name.
Keep descriptions to 2-4 sentences unless the product is complex.`;
}
return {
id: 'enhance.descriptions',
description: 'Enhance product descriptions in batch',
run
};
}
module.exports = { createDescriptionEnhanceTask };
Prompts
Name Prompts
// services/ai/prompts/namePrompts.js
function buildNamePrompt(product) {
return `Format this product name for a craft supplies store.
CURRENT NAME: "${product.name || ''}"
COMPANY: ${product.company_name || product.company || 'Unknown'}
LINE: ${product.line_name || product.line || 'None'}
PRODUCT TYPE: ${inferProductType(product)}
NAMING RULES:
- Single product in line: [Line Name] [Product Name] - [Company]
- Multiple similar products: [Differentiator] [Product Type] - [Line Name] - [Company]
- Standalone products: [Product Name] - [Company]
- Always capitalize every word (including "The", "And", etc.)
- Paper sizes: Use "12x12", "6x6" (no spaces or units)
- All stamps → "Stamp Set" (not "Clear Stamps")
- All dies → "Dies" (not "Die Set")
SPECIAL RULES:
- Tim Holtz from Ranger: "[Color] [Product] - Tim Holtz Distress - Ranger"
- Tim Holtz from Sizzix: "[Product] by Tim Holtz - Sizzix"
- Dylusions from Ranger: "[Product] - Dylusions - Ranger"
Return ONLY the corrected name, nothing else.`;
}
function inferProductType(product) {
const name = (product.name || '').toLowerCase();
const desc = (product.description || '').toLowerCase();
const text = `${name} ${desc}`;
if (text.includes('stamp')) return 'Stamps';
if (text.includes('die') || text.includes('thinlit')) return 'Dies';
if (text.includes('paper') || text.includes('cardstock')) return 'Paper';
if (text.includes('ink')) return 'Ink';
if (text.includes('sticker')) return 'Stickers';
if (text.includes('washi')) return 'Washi Tape';
return 'Unknown';
}
module.exports = { buildNamePrompt };
Category Prompts
// services/ai/prompts/categoryPrompts.js
function buildCategoryPrompt(product, categoryMatches) {
const matchList = categoryMatches
.map(c => `${c.id}: ${c.fullPath}`)
.join('\n');
return `Select the best categories for this craft product.
PRODUCT:
Name: "${product.name || ''}"
Description: "${(product.description || '').substring(0, 200)}"
Company: ${product.company_name || 'Unknown'}
CATEGORY OPTIONS:
${matchList}
RULES:
- Select 1-3 most specific categories
- Prefer deeper subcategories over parents
- If selecting a subcategory, don't also select its parent
- Never select "Deals" or "Black Friday" categories
Return JSON: {"categories": [id1, id2]}`;
}
module.exports = { buildCategoryPrompt };
Description Prompts
// services/ai/prompts/descriptionPrompts.js
function buildDescriptionPrompt(products, mode = 'enhance') {
const productList = products.map((p, i) => {
return `[${i}]
Name: ${p.name || 'Unknown'}
Company: ${p.company_name || 'Unknown'}
Current Description: ${p.description || '(none)'}
Categories: ${p.category_names?.join(', ') || 'Unknown'}
Dimensions: ${p.length || '?'}x${p.width || '?'} inches
Weight: ${p.weight || '?'} oz`;
}).join('\n\n');
const instruction = mode === 'generate'
? 'Write new descriptions for these products.'
: 'Improve these product descriptions. Fix grammar, add missing details, make SEO-friendly.';
return `${instruction}
PRODUCTS:
${productList}
RULES:
- 2-4 sentences each, professional but friendly
- Always state what's included (quantity, size)
- Don't use "our" - use "this" or company name
- Don't add generic filler ("perfect for all your crafts")
- State facts: dimensions, compatibility, materials
- Don't make up information you're not sure about
Return JSON: {"descriptions": [{"description": "..."}, ...]}`;
}
module.exports = { buildDescriptionPrompt };
Normalizers (Tier 1 - No AI)
// services/ai/normalizers/index.js
const priceNormalizer = require('./priceNormalizer');
const upcNormalizer = require('./upcNormalizer');
const countryCodeNormalizer = require('./countryCodeNormalizer');
const dateNormalizer = require('./dateNormalizer');
const numericNormalizer = require('./numericNormalizer');
/**
* Apply all relevant normalizers to a product
*/
function normalizeProduct(product, fieldMappings) {
const normalized = { ...product };
const changes = [];
// Price fields
for (const field of ['msrp', 'cost_each', 'price']) {
if (normalized[field] !== undefined) {
const result = priceNormalizer.normalize(normalized[field]);
if (result.value !== normalized[field]) {
changes.push({ field, from: normalized[field], to: result.value });
normalized[field] = result.value;
}
}
}
// UPC/SKU fields
for (const field of ['upc', 'supplier_no', 'notions_no', 'item_number']) {
if (normalized[field] !== undefined) {
const result = upcNormalizer.normalize(normalized[field]);
if (result.value !== normalized[field]) {
changes.push({ field, from: normalized[field], to: result.value });
normalized[field] = result.value;
}
}
}
// Country of origin
if (normalized.coo !== undefined) {
const result = countryCodeNormalizer.normalize(normalized.coo);
if (result.value !== normalized.coo) {
changes.push({ field: 'coo', from: normalized.coo, to: result.value });
normalized.coo = result.value;
}
}
// ETA date
if (normalized.eta !== undefined) {
const result = dateNormalizer.normalizeEta(normalized.eta);
if (result.value !== normalized.eta) {
changes.push({ field: 'eta', from: normalized.eta, to: result.value });
normalized.eta = result.value;
}
}
// Numeric fields
for (const field of ['qty_per_unit', 'case_qty']) {
if (normalized[field] !== undefined) {
const result = numericNormalizer.normalize(normalized[field]);
if (result.value !== normalized[field]) {
changes.push({ field, from: normalized[field], to: result.value });
normalized[field] = result.value;
}
}
}
return { normalized, changes };
}
module.exports = {
normalizeProduct,
priceNormalizer,
upcNormalizer,
countryCodeNormalizer,
dateNormalizer,
numericNormalizer
};
// services/ai/normalizers/priceNormalizer.js
/**
* Normalize price values
* Input: "$5.00", "5", "5.5", "$1,234.56"
* Output: "5.00", "5.00", "5.50", "1234.56"
*/
function normalize(value) {
if (value === null || value === undefined || value === '') {
return { value, changed: false };
}
const original = String(value);
// Remove currency symbols and commas
let cleaned = original.replace(/[$,]/g, '').trim();
// Try to parse as number
const num = parseFloat(cleaned);
if (isNaN(num)) {
return { value: original, changed: false, error: 'Not a valid number' };
}
// Format to 2 decimal places
const formatted = num.toFixed(2);
return {
value: formatted,
changed: formatted !== original
};
}
module.exports = { normalize };
// services/ai/normalizers/countryCodeNormalizer.js
const COUNTRY_MAP = {
// Full names
'united states': 'US',
'united states of america': 'US',
'china': 'CN',
'peoples republic of china': 'CN',
"people's republic of china": 'CN',
'taiwan': 'TW',
'japan': 'JP',
'south korea': 'KR',
'korea': 'KR',
'india': 'IN',
'germany': 'DE',
'united kingdom': 'GB',
'great britain': 'GB',
'france': 'FR',
'italy': 'IT',
'spain': 'ES',
'canada': 'CA',
'mexico': 'MX',
'brazil': 'BR',
'australia': 'AU',
'vietnam': 'VN',
'thailand': 'TH',
'indonesia': 'ID',
'philippines': 'PH',
'malaysia': 'MY',
// Common abbreviations
'usa': 'US',
'u.s.a.': 'US',
'u.s.': 'US',
'prc': 'CN',
'uk': 'GB',
'u.k.': 'GB',
'rok': 'KR'
};
function normalize(value) {
if (value === null || value === undefined || value === '') {
return { value, changed: false };
}
const original = String(value).trim();
const lookup = original.toLowerCase();
// Check if it's already a valid 2-letter code
if (/^[A-Z]{2}$/.test(original)) {
return { value: original, changed: false };
}
// Look up in map
if (COUNTRY_MAP[lookup]) {
return {
value: COUNTRY_MAP[lookup],
changed: true
};
}
// If 2 chars, uppercase and return
if (original.length === 2) {
const upper = original.toUpperCase();
return {
value: upper,
changed: upper !== original
};
}
// Take first 2 chars as fallback
const fallback = original.substring(0, 2).toUpperCase();
return {
value: fallback,
changed: true,
warning: `Unknown country "${original}", using "${fallback}"`
};
}
module.exports = { normalize };
Main AI Service Entry
// services/ai/index.js
const { AiTaskRegistry, TASK_IDS } = require('./taskRegistry');
const { AiWorkQueue } = require('./workQueue');
const { createProvider } = require('./providers');
const { initializeEmbeddings } = require('./embeddings');
const { normalizeProduct } = require('./normalizers');
// Task factories
const { createNameSuggestionTask } = require('./tasks/nameSuggestionTask');
const { createCategorySuggestionTask } = require('./tasks/categorySuggestionTask');
const { createThemeSuggestionTask } = require('./tasks/themeSuggestionTask');
const { createColorSuggestionTask } = require('./tasks/colorSuggestionTask');
const { createDescriptionEnhanceTask } = require('./tasks/descriptionEnhanceTask');
const { createConsistencyCheckTask } = require('./tasks/consistencyCheckTask');
let initialized = false;
let aiEnabled = false;
let registry = null;
let workQueue = null;
let providers = {};
/**
* Initialize the AI system
*/
async function initialize({ config, mysqlConnection, logger }) {
if (initialized) {
return { enabled: aiEnabled };
}
if (!config?.ai?.enabled) {
logger?.info('[AI] AI features disabled by configuration');
initialized = true;
aiEnabled = false;
return { enabled: false };
}
try {
// Initialize providers
providers.groq = createProvider('groq', config.ai);
providers.openai = createProvider('openai', config.ai);
if (config.ai.providers?.anthropic?.apiKey) {
providers.anthropic = createProvider('anthropic', config.ai);
}
// Initialize embeddings (requires OpenAI for embedding generation)
await initializeEmbeddings({
openaiProvider: providers.openai,
mysqlConnection,
logger
});
// Initialize work queue
workQueue = new AiWorkQueue(config.ai.maxConcurrentTasks || 5);
// Initialize task registry
registry = new AiTaskRegistry();
// Register Tier 2 tasks (real-time, Groq)
registry.register(createNameSuggestionTask({
provider: providers.groq,
logger,
config: config.ai
}));
registry.register(createCategorySuggestionTask({
provider: providers.groq,
logger,
config: config.ai
}));
registry.register(createThemeSuggestionTask({
provider: providers.groq,
logger,
config: config.ai
}));
registry.register(createColorSuggestionTask({
provider: providers.groq,
logger,
config: config.ai
}));
// Register Tier 3 tasks (batch, Anthropic/OpenAI)
const batchProvider = providers.anthropic || providers.openai;
registry.register(createDescriptionEnhanceTask({
provider: batchProvider,
logger,
config: config.ai
}));
registry.register(createConsistencyCheckTask({
provider: batchProvider,
logger,
config: config.ai
}));
initialized = true;
aiEnabled = true;
logger?.info('[AI] AI system initialized successfully');
return { enabled: true };
} catch (error) {
logger?.error('[AI] Failed to initialize AI system', { error: error.message });
initialized = true;
aiEnabled = false;
return { enabled: false, error: error.message };
}
}
/**
* Run a task by ID
*/
async function runTask(taskId, payload = {}) {
if (!aiEnabled) {
throw new Error('AI features are disabled');
}
const task = registry?.get(taskId);
if (!task) {
throw new Error(`Unknown task: ${taskId}`);
}
// Execute through work queue for concurrency control
return workQueue.enqueue(() => task.run(payload));
}
/**
* Normalize a product using Tier 1 (code-based) rules
*/
function normalize(product, fieldMappings) {
return normalizeProduct(product, fieldMappings);
}
/**
* Get system status
*/
function getStatus() {
return {
enabled: aiEnabled,
initialized,
tasks: registry?.list() || [],
queue: workQueue?.getStats() || null
};
}
module.exports = {
initialize,
runTask,
normalize,
getStatus,
TASK_IDS
};
API Routes
// routes/ai.js
const express = require('express');
const router = express.Router();
const ai = require('../services/ai');
const { TASK_IDS } = require('../services/ai/taskRegistry');
/**
* Get AI system status
*/
router.get('/status', (req, res) => {
res.json(ai.getStatus());
});
/**
* Normalize a product (Tier 1 - no AI)
*/
router.post('/normalize', (req, res) => {
try {
const { product, fieldMappings } = req.body;
const result = ai.normalize(product, fieldMappings);
res.json(result);
} catch (error) {
res.status(500).json({ error: error.message });
}
});
/**
* Normalize multiple products (Tier 1 - no AI)
*/
router.post('/normalize/batch', (req, res) => {
try {
const { products, fieldMappings } = req.body;
const results = products.map(product => ai.normalize(product, fieldMappings));
res.json({ results });
} catch (error) {
res.status(500).json({ error: error.message });
}
});
/**
* Get name suggestion (Tier 2 - Groq)
*/
router.post('/suggest/name', async (req, res) => {
try {
const { product } = req.body;
const result = await ai.runTask(TASK_IDS.SUGGEST_NAME, { product });
res.json(result);
} catch (error) {
console.error('[AI] Name suggestion error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Get category suggestions (Tier 2 - Embeddings + Groq)
*/
router.post('/suggest/categories', async (req, res) => {
try {
const { product } = req.body;
const result = await ai.runTask(TASK_IDS.SUGGEST_CATEGORIES, { product });
res.json(result);
} catch (error) {
console.error('[AI] Category suggestion error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Get theme suggestions (Tier 2 - Embeddings + Groq)
*/
router.post('/suggest/themes', async (req, res) => {
try {
const { product } = req.body;
const result = await ai.runTask(TASK_IDS.SUGGEST_THEMES, { product });
res.json(result);
} catch (error) {
console.error('[AI] Theme suggestion error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Get color suggestions (Tier 2 - Groq)
*/
router.post('/suggest/colors', async (req, res) => {
try {
const { product } = req.body;
const result = await ai.runTask(TASK_IDS.SUGGEST_COLORS, { product });
res.json(result);
} catch (error) {
console.error('[AI] Color suggestion error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Enhance descriptions (Tier 3 - Batch)
*/
router.post('/enhance/descriptions', async (req, res) => {
try {
const { products, mode = 'enhance' } = req.body;
const result = await ai.runTask(TASK_IDS.ENHANCE_DESCRIPTIONS, { products, mode });
res.json(result);
} catch (error) {
console.error('[AI] Description enhancement error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Check consistency across products (Tier 3 - Batch)
*/
router.post('/check/consistency', async (req, res) => {
try {
const { products } = req.body;
const result = await ai.runTask(TASK_IDS.CHECK_CONSISTENCY, { products });
res.json(result);
} catch (error) {
console.error('[AI] Consistency check error:', error);
res.status(500).json({ error: error.message });
}
});
/**
* Legacy endpoint - full validation (redirects to new system)
* Kept for backwards compatibility during migration
*/
router.post('/validate', async (req, res) => {
// TODO: Implement migration path
// For now, combine Tier 1 normalization with Tier 3 batch processing
res.status(501).json({
error: 'Legacy validation endpoint deprecated. Use /normalize, /suggest/*, and /enhance/* endpoints.'
});
});
module.exports = router;
Frontend Implementation
Suggestion Hooks
Base Hook
// hooks/ai/useAiSuggestion.ts
import { useState, useCallback } from 'react';
import { useDebouncedCallback } from 'use-debounce';
interface SuggestionResult<T> {
suggestion: T | null;
isLoading: boolean;
error: string | null;
latencyMs: number | null;
}
interface UseAiSuggestionOptions {
debounceMs?: number;
enabled?: boolean;
}
export function useAiSuggestion<T>(
endpoint: string,
options: UseAiSuggestionOptions = {}
) {
const { debounceMs = 300, enabled = true } = options;
const [result, setResult] = useState<SuggestionResult<T>>({
suggestion: null,
isLoading: false,
error: null,
latencyMs: null
});
const fetchSuggestion = useCallback(async (payload: unknown) => {
if (!enabled) return;
setResult(prev => ({ ...prev, isLoading: true, error: null }));
try {
const response = await fetch(`/api/ai${endpoint}`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(payload)
});
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const data = await response.json();
setResult({
suggestion: data.suggestion ?? data.suggestions ?? data,
isLoading: false,
error: null,
latencyMs: data.latencyMs ?? null
});
} catch (error) {
setResult(prev => ({
...prev,
isLoading: false,
error: error instanceof Error ? error.message : 'Unknown error'
}));
}
}, [endpoint, enabled]);
const debouncedFetch = useDebouncedCallback(fetchSuggestion, debounceMs);
const clear = useCallback(() => {
setResult({
suggestion: null,
isLoading: false,
error: null,
latencyMs: null
});
}, []);
return {
...result,
fetch: fetchSuggestion,
fetchDebounced: debouncedFetch,
clear
};
}
Name Suggestion Hook
// hooks/ai/useNameSuggestion.ts
import { useAiSuggestion } from './useAiSuggestion';
import { ProductRow } from '@/types/import';
interface NameSuggestionResult {
suggestion: string | null;
original: string;
unchanged?: boolean;
}
export function useNameSuggestion() {
const {
suggestion,
isLoading,
error,
fetch,
fetchDebounced,
clear
} = useAiSuggestion<NameSuggestionResult>('/suggest/name');
const suggest = (product: Partial<ProductRow>) => {
if (!product.name && !product.description) return;
fetchDebounced({ product });
};
const suggestImmediate = (product: Partial<ProductRow>) => {
if (!product.name && !product.description) return;
fetch({ product });
};
return {
suggestion: suggestion?.suggestion ?? null,
original: suggestion?.original ?? null,
unchanged: suggestion?.unchanged ?? false,
isLoading,
error,
suggest,
suggestImmediate,
clear
};
}
Category Suggestion Hook
// hooks/ai/useCategorySuggestion.ts
import { useAiSuggestion } from './useAiSuggestion';
import { ProductRow } from '@/types/import';
interface CategoryMatch {
id: number;
name: string;
fullPath: string;
similarity: number;
}
interface CategorySuggestionResult {
suggestions: CategoryMatch[];
allMatches: CategoryMatch[];
}
export function useCategorySuggestion() {
const {
suggestion,
isLoading,
error,
fetch,
clear
} = useAiSuggestion<CategorySuggestionResult>('/suggest/categories', {
debounceMs: 500 // Longer debounce for embedding lookup
});
const suggest = (product: Partial<ProductRow>) => {
const hasText = product.name || product.description;
if (!hasText) return;
fetch({ product });
};
return {
suggestions: suggestion?.suggestions ?? [],
allMatches: suggestion?.allMatches ?? [],
isLoading,
error,
suggest,
clear
};
}
Batch Enhancement Hook
// hooks/ai/useDescriptionEnhancement.ts
import { useState, useCallback } from 'react';
import { ProductRow } from '@/types/import';
interface EnhancementResult {
productId: string | number;
original: string | null;
enhanced: string | null;
changed: boolean;
}
interface UseDescriptionEnhancementResult {
results: EnhancementResult[];
isProcessing: boolean;
progress: { current: number; total: number };
error: string | null;
enhance: (products: ProductRow[], mode?: 'enhance' | 'generate') => Promise<void>;
cancel: () => void;
}
export function useDescriptionEnhancement(): UseDescriptionEnhancementResult {
const [results, setResults] = useState<EnhancementResult[]>([]);
const [isProcessing, setIsProcessing] = useState(false);
const [progress, setProgress] = useState({ current: 0, total: 0 });
const [error, setError] = useState<string | null>(null);
const [abortController, setAbortController] = useState<AbortController | null>(null);
const enhance = useCallback(async (
products: ProductRow[],
mode: 'enhance' | 'generate' = 'enhance'
) => {
const controller = new AbortController();
setAbortController(controller);
setIsProcessing(true);
setProgress({ current: 0, total: products.length });
setResults([]);
setError(null);
try {
const response = await fetch('/api/ai/enhance/descriptions', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ products, mode }),
signal: controller.signal
});
if (!response.ok) {
throw new Error(`API error: ${response.status}`);
}
const data = await response.json();
setResults(data.results || []);
setProgress({ current: products.length, total: products.length });
} catch (err) {
if (err instanceof Error && err.name === 'AbortError') {
// Cancelled by user
return;
}
setError(err instanceof Error ? err.message : 'Unknown error');
} finally {
setIsProcessing(false);
setAbortController(null);
}
}, []);
const cancel = useCallback(() => {
abortController?.abort();
}, [abortController]);
return {
results,
isProcessing,
progress,
error,
enhance,
cancel
};
}
UI Components
Suggestion Badge
// components/ai/SuggestionBadge.tsx
import { Check, X, Sparkles } from 'lucide-react';
import { Button } from '@/components/ui/button';
import { cn } from '@/lib/utils';
interface SuggestionBadgeProps {
suggestion: string;
onAccept: () => void;
onDismiss: () => void;
className?: string;
}
export function SuggestionBadge({
suggestion,
onAccept,
onDismiss,
className
}: SuggestionBadgeProps) {
return (
<div className={cn(
'flex items-center gap-2 p-2 mt-1 rounded-md',
'bg-purple-50 border border-purple-200',
'dark:bg-purple-950/30 dark:border-purple-800',
className
)}>
<Sparkles className="h-3.5 w-3.5 text-purple-500 flex-shrink-0" />
<span className="text-sm text-purple-700 dark:text-purple-300 flex-1 truncate">
{suggestion}
</span>
<div className="flex items-center gap-1 flex-shrink-0">
<Button
size="sm"
variant="ghost"
className="h-6 w-6 p-0 text-green-600 hover:text-green-700 hover:bg-green-100"
onClick={onAccept}
>
<Check className="h-3.5 w-3.5" />
</Button>
<Button
size="sm"
variant="ghost"
className="h-6 w-6 p-0 text-gray-400 hover:text-gray-600 hover:bg-gray-100"
onClick={onDismiss}
>
<X className="h-3.5 w-3.5" />
</Button>
</div>
</div>
);
}
Category Suggestion Dropdown
// components/ai/CategorySuggestionDropdown.tsx
import { useState, useEffect } from 'react';
import { Sparkles, ChevronDown } from 'lucide-react';
import {
DropdownMenu,
DropdownMenuContent,
DropdownMenuItem,
DropdownMenuSeparator,
DropdownMenuTrigger
} from '@/components/ui/dropdown-menu';
import { Button } from '@/components/ui/button';
import { Badge } from '@/components/ui/badge';
import { useCategorySuggestion } from '@/hooks/ai/useCategorySuggestion';
import { ProductRow } from '@/types/import';
interface CategorySuggestionDropdownProps {
product: ProductRow;
currentCategories: number[];
onSelect: (categoryId: number) => void;
allCategories: Array<{ id: number; name: string; fullPath: string }>;
}
export function CategorySuggestionDropdown({
product,
currentCategories,
onSelect,
allCategories
}: CategorySuggestionDropdownProps) {
const { suggestions, allMatches, isLoading, suggest, clear } = useCategorySuggestion();
const [isOpen, setIsOpen] = useState(false);
// Fetch suggestions when dropdown opens
useEffect(() => {
if (isOpen && suggestions.length === 0) {
suggest(product);
}
}, [isOpen, product, suggest, suggestions.length]);
// Clear when dropdown closes
useEffect(() => {
if (!isOpen) {
clear();
}
}, [isOpen, clear]);
return (
<DropdownMenu open={isOpen} onOpenChange={setIsOpen}>
<DropdownMenuTrigger asChild>
<Button variant="outline" size="sm" className="gap-2">
<span>Categories</span>
{currentCategories.length > 0 && (
<Badge variant="secondary" className="ml-1">
{currentCategories.length}
</Badge>
)}
<ChevronDown className="h-4 w-4" />
</Button>
</DropdownMenuTrigger>
<DropdownMenuContent className="w-80">
{/* AI Suggestions Section */}
{(isLoading || suggestions.length > 0) && (
<>
<div className="flex items-center gap-2 px-2 py-1.5 text-xs font-medium text-purple-600">
<Sparkles className="h-3 w-3" />
AI Suggested
{isLoading && <span className="text-gray-400">(loading...)</span>}
</div>
{suggestions.map(cat => (
<DropdownMenuItem
key={cat.id}
onClick={() => onSelect(cat.id)}
className="flex items-center justify-between"
>
<span className="truncate">{cat.fullPath}</span>
<Badge variant="outline" className="ml-2 text-xs">
{Math.round(cat.similarity * 100)}%
</Badge>
</DropdownMenuItem>
))}
<DropdownMenuSeparator />
</>
)}
{/* All Categories Section */}
<div className="px-2 py-1.5 text-xs font-medium text-gray-500">
All Categories
</div>
<div className="max-h-60 overflow-y-auto">
{allCategories.slice(0, 50).map(cat => (
<DropdownMenuItem
key={cat.id}
onClick={() => onSelect(cat.id)}
disabled={currentCategories.includes(cat.id)}
>
<span className="truncate">{cat.fullPath}</span>
</DropdownMenuItem>
))}
</div>
</DropdownMenuContent>
</DropdownMenu>
);
}
AI Validation Cell
// components/ai/AiValidationCell.tsx
import { useState, useEffect } from 'react';
import { Loader2 } from 'lucide-react';
import { Input } from '@/components/ui/input';
import { SuggestionBadge } from './SuggestionBadge';
import { useNameSuggestion } from '@/hooks/ai/useNameSuggestion';
import { ProductRow } from '@/types/import';
interface AiValidationCellProps {
field: 'name' | 'description';
value: string;
product: ProductRow;
onChange: (value: string) => void;
onBlur?: () => void;
}
export function AiValidationCell({
field,
value,
product,
onChange,
onBlur
}: AiValidationCellProps) {
const {
suggestion,
isLoading,
suggest,
clear
} = useNameSuggestion();
const [localValue, setLocalValue] = useState(value);
const [showSuggestion, setShowSuggestion] = useState(false);
// Sync external value changes
useEffect(() => {
setLocalValue(value);
}, [value]);
// Show suggestion when it arrives and differs from current value
useEffect(() => {
if (suggestion && suggestion !== localValue) {
setShowSuggestion(true);
}
}, [suggestion, localValue]);
const handleBlur = () => {
// Trigger AI suggestion on blur
if (field === 'name') {
suggest({ ...product, name: localValue });
}
onBlur?.();
};
const handleAccept = () => {
if (suggestion) {
setLocalValue(suggestion);
onChange(suggestion);
setShowSuggestion(false);
clear();
}
};
const handleDismiss = () => {
setShowSuggestion(false);
clear();
};
return (
<div className="relative">
<div className="relative">
<Input
value={localValue}
onChange={(e) => {
setLocalValue(e.target.value);
onChange(e.target.value);
}}
onBlur={handleBlur}
className="pr-8"
/>
{isLoading && (
<div className="absolute right-2 top-1/2 -translate-y-1/2">
<Loader2 className="h-4 w-4 animate-spin text-purple-500" />
</div>
)}
</div>
{showSuggestion && suggestion && (
<SuggestionBadge
suggestion={suggestion}
onAccept={handleAccept}
onDismiss={handleDismiss}
/>
)}
</div>
);
}
Batch Enhancement Button
// components/ai/EnhanceDescriptionsButton.tsx
import { useState } from 'react';
import { Sparkles, Loader2, CheckCircle } from 'lucide-react';
import { Button } from '@/components/ui/button';
import {
Dialog,
DialogContent,
DialogDescription,
DialogFooter,
DialogHeader,
DialogTitle
} from '@/components/ui/dialog';
import { Progress } from '@/components/ui/progress';
import { useDescriptionEnhancement } from '@/hooks/ai/useDescriptionEnhancement';
import { useValidationStore } from '../store/validationStore';
export function EnhanceDescriptionsButton() {
const [showDialog, setShowDialog] = useState(false);
const rows = useValidationStore(state => state.rows);
const updateRow = useValidationStore(state => state.updateRow);
const {
results,
isProcessing,
progress,
error,
enhance,
cancel
} = useDescriptionEnhancement();
const handleEnhance = async () => {
setShowDialog(true);
await enhance(rows.map(r => r.data));
};
const handleApply = () => {
// Apply enhanced descriptions to store
for (const result of results) {
if (result.changed && result.enhanced) {
const rowIndex = rows.findIndex(
r => r.data._index === result.productId || r.data.upc === result.productId
);
if (rowIndex !== -1) {
updateRow(rowIndex, { description: result.enhanced });
}
}
}
setShowDialog(false);
};
const changedCount = results.filter(r => r.changed).length;
const progressPercent = progress.total > 0
? Math.round((progress.current / progress.total) * 100)
: 0;
return (
<>
<Button
variant="outline"
onClick={handleEnhance}
disabled={isProcessing || rows.length === 0}
>
<Sparkles className="h-4 w-4 mr-2" />
Enhance Descriptions
</Button>
<Dialog open={showDialog} onOpenChange={setShowDialog}>
<DialogContent>
<DialogHeader>
<DialogTitle>Enhance Descriptions</DialogTitle>
<DialogDescription>
AI will improve product descriptions for SEO and clarity.
</DialogDescription>
</DialogHeader>
<div className="py-4">
{isProcessing ? (
<div className="space-y-4">
<div className="flex items-center gap-3">
<Loader2 className="h-5 w-5 animate-spin text-purple-500" />
<span>Processing {progress.current} of {progress.total} products...</span>
</div>
<Progress value={progressPercent} className="h-2" />
</div>
) : error ? (
<div className="text-red-500">{error}</div>
) : results.length > 0 ? (
<div className="space-y-2">
<div className="flex items-center gap-2 text-green-600">
<CheckCircle className="h-5 w-5" />
<span>Enhanced {changedCount} descriptions</span>
</div>
<p className="text-sm text-gray-500">
{results.length - changedCount} descriptions were already good or unchanged.
</p>
</div>
) : null}
</div>
<DialogFooter>
{isProcessing ? (
<Button variant="outline" onClick={cancel}>
Cancel
</Button>
) : results.length > 0 ? (
<>
<Button variant="outline" onClick={() => setShowDialog(false)}>
Discard
</Button>
<Button onClick={handleApply} disabled={changedCount === 0}>
Apply {changedCount} Changes
</Button>
</>
) : null}
</DialogFooter>
</DialogContent>
</Dialog>
</>
);
}
Integration with ValidationStep
// Example integration in ValidationContainer.tsx
import { AiValidationCell } from '@/components/ai/AiValidationCell';
import { CategorySuggestionDropdown } from '@/components/ai/CategorySuggestionDropdown';
import { EnhanceDescriptionsButton } from '@/components/ai/EnhanceDescriptionsButton';
// In the toolbar
<div className="flex items-center gap-2">
<EnhanceDescriptionsButton />
{/* Other toolbar items */}
</div>
// In the data grid column definitions
const columns = [
{
key: 'name',
name: 'Name',
renderCell: ({ row, onRowChange }) => (
<AiValidationCell
field="name"
value={row.name}
product={row}
onChange={(value) => onRowChange({ ...row, name: value })}
/>
)
},
{
key: 'categories',
name: 'Categories',
renderCell: ({ row, onRowChange }) => (
<CategorySuggestionDropdown
product={row}
currentCategories={parseCategories(row.categories)}
onSelect={(catId) => {
const current = parseCategories(row.categories);
onRowChange({
...row,
categories: [...current, catId].join(',')
});
}}
allCategories={allCategories}
/>
)
}
];
Database Schema
New Tables
-- Embedding cache for faster startup
CREATE TABLE IF NOT EXISTS ai_embedding_cache (
id SERIAL PRIMARY KEY,
entity_type VARCHAR(50) NOT NULL, -- 'category', 'theme', 'color'
entity_id INTEGER NOT NULL,
embedding_model VARCHAR(100) NOT NULL,
embedding VECTOR(1536), -- Using pgvector extension
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
UNIQUE(entity_type, entity_id, embedding_model)
);
-- Index for fast lookups
CREATE INDEX idx_embedding_cache_lookup
ON ai_embedding_cache(entity_type, embedding_model);
-- AI suggestion history (for analytics and improvement)
CREATE TABLE IF NOT EXISTS ai_suggestion_log (
id SERIAL PRIMARY KEY,
task_id VARCHAR(100) NOT NULL,
product_identifier VARCHAR(255),
suggestion JSONB,
accepted BOOLEAN DEFAULT NULL,
latency_ms INTEGER,
token_usage JSONB,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Index for analytics queries
CREATE INDEX idx_suggestion_log_task
ON ai_suggestion_log(task_id, created_at);
Configuration Table Updates
-- Add AI configuration to existing config or create new
CREATE TABLE IF NOT EXISTS ai_config (
id SERIAL PRIMARY KEY,
key VARCHAR(100) UNIQUE NOT NULL,
value JSONB NOT NULL,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);
-- Insert default config
INSERT INTO ai_config (key, value) VALUES
('providers', '{
"groq": {
"enabled": true,
"model": "llama-3.3-70b-versatile"
},
"openai": {
"enabled": true,
"embeddingModel": "text-embedding-3-small"
},
"anthropic": {
"enabled": true,
"model": "claude-3-5-haiku-20241022"
}
}'),
('tasks', '{
"nameSuggestion": {
"model": "llama-3.3-70b-versatile",
"temperature": 0.2,
"maxTokens": 150
},
"categorySuggestion": {
"model": "llama-3.1-8b-instant",
"temperature": 0.1,
"maxTokens": 100
},
"descriptionEnhance": {
"model": "claude-3-5-haiku-20241022",
"temperature": 0.7,
"maxTokens": 2000,
"batchSize": 10
}
}');
Migration Strategy
Phase 1: Add New System Alongside Old (Week 1-2)
- Implement
services/ai/structure - Add new
/api/ai/*routes - Keep old
/api/ai-validation/*routes working - Add feature flag to enable new system per-user
Phase 2: Frontend Integration (Week 2-3)
- Add suggestion hooks
- Integrate
AiValidationCellfor name field first - Add
CategorySuggestionDropdown - Add
EnhanceDescriptionsButtonfor batch descriptions
Phase 3: Replace Batch Validation (Week 3-4)
- Update "AI Validate" button to use new tiered system
- Remove old giant-prompt approach
- Keep old endpoint as deprecated fallback
- Monitor costs and latency
Phase 4: Cleanup (Week 4+)
- Remove old
ai-validation.jsroute - Remove old frontend components
- Archive old prompts
- Document new system
Cost Analysis
Current Costs (GPT-5.2 Reasoning)
| Metric | Value |
|---|---|
| Input tokens (est.) | ~25,000 |
| Output tokens (est.) | ~5,000 |
| Cost per run | $0.30-0.50 |
| Runs per day (est.) | 20 |
| Monthly cost | ~$200-300 |
Projected Costs (New System)
| Tier | Model | Cost per call | Calls per product | Cost per product |
|---|---|---|---|---|
| Tier 1 | Code | $0 | N/A | $0 |
| Tier 2 | Groq Llama 3.3 70B | ~$0.0003 | 3-5 | ~$0.001 |
| Tier 2 | OpenAI Embeddings | ~$0.00002 | 1 | ~$0.00002 |
| Tier 3 | Claude Haiku | ~$0.001 | 0.1 (batch) | ~$0.0001 |
Per 50 products:
- Tier 1: $0
- Tier 2: ~$0.05 (if all fields suggested)
- Tier 3: ~$0.01 (description batch)
- Total: ~$0.06
Monthly projection (20 runs/day × 50 products):
- Current: $200-300
- New: ~$36
- Savings: 80-90%
Next Steps
- Review this document - Confirm approach aligns with expectations
- Set up Groq account - Get API key for real-time inference
- Implement providers - Start with Groq + OpenAI
- Build embedding system - Pre-compute category embeddings
- Create first task - Start with name suggestion
- Integrate frontend - Add suggestion badge to name field
- Iterate - Add more tasks based on feedback
Appendix: Key Files Reference
Email App AI System (Reference)
/Users/matt/Dev/email/email-server/services/ai/index.js- Main entry/Users/matt/Dev/email/email-server/services/ai/providers/groqProvider.js- Groq implementation/Users/matt/Dev/email/email-server/services/ai/taskRegistry.js- Task system/Users/matt/Dev/email/email-server/services/ai/workQueue.js- Concurrency
Current Inventory AI System
/inventory-server/src/routes/ai-validation.js- Current monolithic implementation/inventory/src/components/product-import/steps/ValidationStep/hooks/useAiValidation/- Current frontend hooks