1.2 Technical Architecture & Design Approach¶

┌──────────────────────────────────────────────────────────────────────┐
│                          CLIENT LAYER                                │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐             │
│  │   Web UI      │  │   Mobile App  │  │   3rd Party   │             │
│  │  (React/TS)   │  │               │  │   Integration │             │
│  └───────┬───────┘  └───────┬───────┘  └───────┬───────┘             │
│          │                  │                  │                     │
│          └──────────────────┴──────────────────┘                     │
│                             │                                        │
│                    HTTP/JSON REST API                                │
└─────────────────────────────┼────────────────────────────────────────┘
                              │
┌─────────────────────────────┼────────────────────────────────────────┐
│                       API GATEWAY LAYER                              │
│  ┌────────────────────────────────────────────────────────────┐      │
│  │              FastAPI Application (apps/api/main.py)        │      │
│  │  • Request validation & preprocessing                      │      │
│  │  • Response caching (Redis)                                │      │
│  │  • Metrics collection (Prometheus)                         │      │
│  │  • Database persistence (PostgreSQL)                       │      │
│  │  • CORS, rate limiting, error handling                     │      │
│  └────────────────────┬───────────────────────────────────────┘      │
└───────────────────────┼──────────────────────────────────────────────┘
                        │
┌───────────────────────┼──────────────────────────────────────────────┐
│                 ORCHESTRATION LAYER                                  │
│  ┌────────────────────────────────────────────────────────────┐      │
│  │            Ensemble Router (core/model/ensemble_router.py) │      │
│  │  • Workflow orchestration                                  │      │
│  │  • Parallel execution (ThreadPoolExecutor)                 │      │
│  │  • Early-exit optimizations                                │      │
│  │  • Weighted voting & confidence calibration                │      │
│  │  • Method selection & fallback logic                       │      │
│  └────────────────────┬───────────────────────────────────────┘      │
└───────────────────────┼──────────────────────────────────────────────┘
                        │
┌───────────────────────┴──────────────────────────────────────────────┐
│                    PROCESSING PIPELINE                               │
│  ┌───────────────────────────────────────────────────────────┐       │
│  │  Stage 1: Normalization (core/normalize/normalizer.py)    │       │
│  │  ┌─────────────────────────────────────────────────────┐  │       │
│  │  │ • Unicode normalization & text cleaning             │  │       │
│  │  │ • Pattern extraction (channel, merchant, reference) │  │       │
│  │  │ • Amount & date parsing                             │  │       │
│  │  │ • Feature extraction (70+ handcrafted features)     │  │       │
│  │  └─────────────────────────────────────────────────────┘  │       │
│  └───────────────────────────────────────────────────────────┘       │
│                              │                                       │
│  ┌───────────────────────────┴───────────────────────────────┐       │
│  │  Stage 2: Merchant Resolution (core/resolve/resolver.py)  │       │
│  │  ┌─────────────────────────────────────────────────────┐  │       │
│  │  │ • Fuzzy string matching (RapidFuzz)                 │  │       │
│  │  │ • Merchant gazetteer lookup (3,000+ entries)        │  │       │
│  │  │ • High-confidence early exit (>70% similarity)      │  │       │
│  │  └─────────────────────────────────────────────────────┘  │       │
│  └───────────────────────────────────────────────────────────┘       │
└──────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────┼────────────────────────────────────────┐
│             CATEGORIZATION LAYER (Parallel Execution)                │
│                                                                      │
│  ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐       │
│  │ Method 1: MCC   │  │ Method 2: Rules │  │ Method 3: ML    │       │
│  │ Classifier      │  │ Engine          │  │ Embeddings      │       │
│  ├─────────────────┤  ├─────────────────┤  ├─────────────────┤       │
│  │• ISO 18245      │  │• Deterministic  │  │• Sentence       │       │
│  │  standard       │  │  pattern match  │  │  Transformers   │       │
│  │• 4-digit codes  │  │• Keyword search │  │• LightGBM       │       │
│  │• 95% confidence │  │• Regex patterns │  │• Calibrated     │       │
│  │• Instant lookup │  │• 60+ categories │  │  probabilities  │       │
│  │• Weight: 15%    │  │• Weight: 15%    │  │• Weight: 65%    │       │
│  └─────────────────┘  └─────────────────┘  └─────────────────┘       │
│         │                     │                     │                │
│         └─────────────────────┴─────────────────────┘                │
│                               │                                      │
│                  ┌────────────┴────────────┐                         │
│                  │ LLM Tiebreaker          │                         │
│                  │ (Triggered on Conflict) │                         │
│                  ├─────────────────────────┤                         │
│                  │• Ollama/Azure OpenAI    │                         │
│                  │• Few-shot prompting     │                         │
│                  │• Reasoning explanation  │                         │
│                  │• Weight: 5%             │                         │
│                  │• 120s timeout           │                         │
│                  └─────────────────────────┘                         │
└──────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────┼────────────────────────────────────────┐
│                   ENSEMBLE VOTING & DECISION                         │
│  ┌────────────────────────────────────────────────────────────┐      │
│  │  • Weighted voting across all methods                      │      │
│  │  • Confidence calibration (agreement-based)                │      │
│  │  • Category normalization & mapping                        │      │
│  │  • Ambiguity scoring & alternative rankings                │      │
│  │  • Human review flagging (category-specific thresholds)    │      │
│  └────────────────────────────────────────────────────────────┘      │
└──────────────────────────────────────────────────────────────────────┘
                              │
┌─────────────────────────────┼────────────────────────────────────────┐
│                      PERSISTENCE LAYER                               │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐                │
│  │  PostgreSQL  │  │    Redis     │  │   Feedback   │                │
│  │  Database    │  │    Cache     │  │   Storage    │                │
│  ├──────────────┤  ├──────────────┤  ├──────────────┤                │
│  │• Transactions│  │• Response    │  │• User        │                │
│  │• Feedback    │  │  caching     │  │  corrections │                │
│  │• Training    │  │• 10min TTL   │  │• Auto-retrain│                │
│  │  jobs        │  │• Deduplication│  │  trigger     │               │
│  └──────────────┘  └──────────────┘  └──────────────┘                │
└──────────────────────────────────────────────────────────────────────┘

MCC_WEIGHT=0.15          # 15% weight (high accuracy when available)
RULE_WEIGHT=0.15         # 15% weight (deterministic patterns)
ML_WEIGHT=0.65           # 65% weight (primary classifier)
LLM_WEIGHT=0.05          # 5% weight (tiebreaker only)

# Merchant-first strategy (>70% similarity)
if merchant_confidence >= 0.70:
    return merchant_category  # ~40% of transactions exit here

# Deterministic rule early exit (>95% confidence)
if rule_confidence >= 0.95:
    return rule_category  # Fraud, ATM, EMI patterns

# MCC early exit (>90% confidence)
if mcc_confidence >= 0.90:
    return mcc_category  # ISO standard codes

# Full ensemble voting for remaining cases (~50% of transactions)

with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {
        'mcc': executor.submit(run_mcc_classifier, text, mcc),
        'rule': executor.submit(run_rule_categorizer, text),
        'ml': executor.submit(run_ml_classifier, text),
        # LLM triggered conditionally (only on disagreement)
    }

{
  "normalized": {
    "merchant": "Starbucks",
    "amount": 250.00,
    "currency": "INR",
    "date": "2025-11-20",
    "channel": "UPI",
    "reference": "UPI/308912345"
  },
  "search_text": "starbucks coffee grande",
  "features": {...}  // 70+ numerical features
}

merchant_id,canonical_name,aliases,category,subcategory
1,Starbucks,"starbucks|sbux|starbux",food_dining,Cafes & Coffee
2,Amazon,"amazon.in|amazon india|amzn",shopping,Online Shopping

keyword_index = {
    "zomato": ["food_dining"],
    "uber": ["transport"],
    "netflix": ["subscriptions_memberships"],
    ...
}

pattern_index = {
    "(?i)upi/.*": {"category": "transfers_upi", "confidence": 0.95},
    "(?i).*atm.*withdrawal.*": {"category": "atm_cash", "confidence": 0.98},
    ...
}

MCC_MAPPINGS = {
    # Food & Dining
    "5812": {"category": "food_dining", "description": "Restaurants"},
    "5814": {"category": "food_dining", "description": "Fast Food"},

    # Travel
    "4511": {"category": "travel", "description": "Airlines"},
    "7011": {"category": "travel", "description": "Hotels"},

    # Fuel
    "5541": {"category": "fuel", "description": "Service Stations"},

    # Health
    "5912": {"category": "health", "description": "Pharmacies"},
    "8062": {"category": "health", "description": "Hospitals"},
    ...
}

result = mcc_classifier.categorize(text="Restaurant payment", mcc="5812")
# Returns: {"category": "food_dining", "confidence": 0.95, "mcc_code": "5812"}

predictions = ml_classifier.predict_single(
    text="netflix monthly subscription",
    handcrafted_features=features,
    top_k=3
)
# Returns: [
#   ("subscriptions_memberships", 0.92),
#   ("entertainment", 0.05),
#   ("shopping", 0.02)
# ]

# LLM invoked ONLY when:
# 1. Rule and ML disagree on category
# 2. Rule confidence < 80% OR ML confidence < 80%
# 3. LLM weight > 0 (configurable)

if rule_cat != ml_cat or rule_conf < 0.80 or ml_conf < 0.80:
    llm_result = llm_classifier.predict_single(text, amount)

"""You are a financial transaction categorization assistant.

Given a transaction description, classify it into ONE of these categories:
{taxonomy}

Few-shot examples:
{examples}

Transaction: "{text}"
Amount: {amount} INR

Provide your answer in this exact format:
CATEGORY: <category_id>
CONFIDENCE: <0.0-1.0>
REASONING: <brief explanation>
"""

def _ensemble_vote(mcc_result, rule_result, ml_result, llm_result):
    # Step 1: Normalize category names
    categories = normalize_all_categories([mcc, rule, ml, llm])

    # Step 2: Weighted voting
    votes = {}
    total_active_weight = 0.0

    if mcc_result:
        votes[mcc_category] += mcc_confidence * MCC_WEIGHT
        total_active_weight += MCC_WEIGHT

    if rule_result:
        votes[rule_category] += rule_confidence * RULE_WEIGHT
        total_active_weight += RULE_WEIGHT

    if ml_result:
        votes[ml_category] += ml_confidence * ML_WEIGHT
        total_active_weight += ML_WEIGHT

    if llm_result:
        votes[llm_category] += llm_confidence * LLM_WEIGHT
        total_active_weight += LLM_WEIGHT

    # Step 3: LLM Tiebreaker Logic
    if rule_cat != ml_cat and llm_result:
        # LLM makes FINAL DECISION on disagreement
        winner_category = llm_category
    else:
        # Standard voting: highest weighted vote wins
        winner_category = max(votes, key=votes.get)

    # Step 4: Confidence Calibration
    normalized_score = votes[winner_category] / total_active_weight

    # Agreement-based adjustment
    if full_agreement:
        adjustment = +0.20  # Boost confidence
    elif partial_agreement:
        adjustment = +0.10
    else:
        adjustment = -0.15  # Penalty for disagreement

    final_confidence = clip(normalized_score + adjustment, 0.05, 1.0)

    return CategorizationResult(
        category=winner_category,
        confidence=final_confidence,
        method="ensemble_unanimous" if full_agreement else "ensemble_mixed",
        ...
    )

CATEGORY_THRESHOLDS = {
    # Critical categories (higher thresholds)
    "Fraud & Security": {"auto_accept": 0.95, "review": 0.80},
    "Investments": {"auto_accept": 0.90, "review": 0.70},
    "Income/Salary": {"auto_accept": 0.90, "review": 0.70},

    # Standard categories
    "Travel": {"auto_accept": 0.85, "review": 0.60},
    "Health": {"auto_accept": 0.85, "review": 0.60},

    # Low-risk categories (lower thresholds)
    "Food & Dining": {"auto_accept": 0.80, "review": 0.50},
    "Shopping": {"auto_accept": 0.80, "review": 0.50},

    # Default
    "Other": {"auto_accept": 0.95, "review": 0.80}
}

version: "1.0.0"
categories:
  - name: "Food & Dining"
    id: "food_dining"
    description: "Restaurants, food delivery, cafes"
    mcc_codes: ["5812", "5814", "5813"]
    subcategories:
      - "Food Delivery"
      - "Restaurants"
      - "Cafes & Coffee"
    keywords:
      - "zomato"
      - "swiggy"
      - "restaurant"
    patterns:
      - "(?i)zomato.*"
      - "(?i)swiggy.*"

-- Transactions table
CREATE TABLE transactions (
    id SERIAL PRIMARY KEY,
    original_text TEXT NOT NULL,
    amount NUMERIC(15, 2),
    currency VARCHAR(10) DEFAULT 'INR',
    date DATE,
    category VARCHAR(100) NOT NULL,
    subcategory VARCHAR(100),
    confidence NUMERIC(5, 4),
    method VARCHAR(50),  -- 'ensemble_unanimous', 'rule', 'ml', etc.
    merchant VARCHAR(255),
    channel VARCHAR(50),
    reference VARCHAR(255),
    requires_review BOOLEAN DEFAULT FALSE,
    reviewed BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

-- Feedback table (for active learning)
CREATE TABLE feedback (
    id SERIAL PRIMARY KEY,
    transaction_text TEXT NOT NULL,
    predicted_category VARCHAR(100) NOT NULL,
    correct_category VARCHAR(100) NOT NULL,
    predicted_subcategory VARCHAR(100),
    correct_subcategory VARCHAR(100),
    amount NUMERIC(15, 2),
    date DATE,
    notes TEXT,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Training jobs table
CREATE TABLE training_jobs (
    id SERIAL PRIMARY KEY,
    job_id VARCHAR(255) UNIQUE NOT NULL,
    dataset_path TEXT,
    model_name VARCHAR(255),
    status VARCHAR(50) DEFAULT 'queued',
    accuracy NUMERIC(5, 4),
    metrics JSON,
    created_at TIMESTAMP DEFAULT NOW(),
    started_at TIMESTAMP,
    completed_at TIMESTAMP
);

CREATE INDEX idx_transactions_category ON transactions(category);
CREATE INDEX idx_transactions_date ON transactions(date);
CREATE INDEX idx_transactions_requires_review ON transactions(requires_review);
CREATE INDEX idx_feedback_predicted_category ON feedback(predicted_category);
CREATE INDEX idx_feedback_correct_category ON feedback(correct_category);

# Cache key generation
def build_cache_key(transaction):
    payload = f"{text}|{amount}|{date}|{currency}"
    return f"txn_cache:{sha256(payload)}"

# Cache hit (10-minute TTL)
cached_output = redis.get(cache_key)
if cached_output:
    return TransactionOutput(**json.loads(cached_output))

# Cache miss - categorize and store
output = router.categorize(text, amount, date, currency)
redis.setex(cache_key, 600, json.dumps(output))

services:
  # PostgreSQL database
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: txn_user
      POSTGRES_USER: txn_user
      POSTGRES_PASSWORD: txn_password
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      interval: 10s

  # Redis cache
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    command: redis-server --appendonly yes

  # Ollama LLM service
  llm-service:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Optional GPU support
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  # FastAPI application
  api:
    build:
      context: .
      dockerfile: infra/Dockerfile
    environment:
      - DATABASE_URL=postgresql://txn_user:txn_password@postgres:5432/txn_user
      - REDIS_URL=redis://redis:6379/0
      - LLM_URL=http://llm-service:11434
      - MCC_WEIGHT=0.15
      - RULE_WEIGHT=0.15
      - ML_WEIGHT=0.65
      - LLM_WEIGHT=0.05
    ports:
      - "8000:8000"
    depends_on:
      - postgres
      - redis
      - llm-service
    command: uvicorn apps.api.main:app --host 0.0.0.0 --port 8000

# Request counter
REQUEST_COUNTER = Counter(
    "categorization_requests_total",
    "Total API requests",
    ["endpoint"]
)

# Latency histogram
LATENCY_HIST = Histogram(
    "categorization_latency_seconds",
    "Request latency",
    ["endpoint"],
    buckets=(0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0)
)

# Method usage
METHOD_COUNTER = Counter(
    "method_usage_total",
    "Method usage count",
    ["method"]
)

# Review rate
REVIEW_COUNTER = Counter(
    "categorization_requires_review_total",
    "Transactions routed to manual review",
    ["endpoint"]
)

{
  "text": "Paid to YO DIMSUM Sec 57 Gurgaon",
  "amount": 850.00,
  "date": "2025-11-20",
  "currency": "INR",
  "mcc": "5812"
}

{
  "original_text": "Paid to YO DIMSUM Sec 57 Gurgaon",
  "category": "food_dining",
  "subcategory": "Restaurants",
  "confidence": 0.92,
  "method": "ensemble_rule+ml",
  "explanations": [
    "merchant_match=YO DIMSUM",
    "ml_embedding_classifier"
  ],
  "requires_review": false,
  "normalized": {
    "merchant": "YO DIMSUM",
    "amount": 850.00,
    "currency": "INR",
    "date": "2025-11-20",
    "channel": "UPI"
  },
  "alternatives": [
    {"category": "shopping", "confidence": 0.05},
    {"category": "other", "confidence": 0.03}
  ],
  "ensemble_votes": {
    "mcc": null,
    "rule": {"category": "food_dining", "confidence": 0.90},
    "ml": {"category": "food_dining", "confidence": 0.94},
    "llm": null,
    "agreement_count": 2,
    "total_methods": 2
  }
}

{
  "transactions": [
    "Starbucks coffee",
    "Netflix subscription",
    "Uber ride to airport"
  ]
}

{
  "results": [
    {
      "transaction": "Starbucks coffee",
      "category": "food_dining",
      "subcategory": "Cafes & Coffee",
      "confidence": 0.95,
      "method": "ensemble_unanimous",
      "status": "success"
    },
    ...
  ],
  "total": 3,
  "successful": 3,
  "failed": 0,
  "duration_seconds": 0.18
}

{
  "transaction_text": "URBAN COMPANY LIMITED",
  "predicted_category": "shopping",
  "correct_category": "personal_care",
  "notes": "Should be salon/spa service"
}

{
  "status": "success",
  "message": "Feedback stored in database",
  "feedback_id": "12345",
  "correction_count": 47,
  "retraining_triggered": false
}

# Trigger retraining at 50 correction intervals
if correction_count >= 50 and correction_count % 50 == 0:
    trigger_auto_retraining()

{
  "filename": "bank_statement_nov_2025.pdf",
  "results": [...],  // Same as batch-categorize
  "total": 156,
  "successful": 154,
  "failed": 2,
  "duration_seconds": 12.45
}

{
  "query": "starbux",
  "limit": 5
}

{
  "query": "starbux",
  "matches": [
    {
      "merchant_id": 1,
      "canonical_name": "Starbucks",
      "aliases": ["starbucks", "sbux", "starbux"],
      "category": "food_dining",
      "subcategory": "Cafes & Coffee",
      "similarity_score": 0.92
    }
  ]
}

{
  "status": "healthy",
  "version": "1.0.0",
  "timestamp": "2025-11-20T12:00:00Z",
  "components": {
    "router": "healthy",
    "normalizer": "healthy",
    "rule_categorizer": "healthy",
    "ml_classifier": "healthy",
    "llm_classifier": "healthy",
    "merchant_resolver": "healthy",
    "database": "healthy",
    "cache": "healthy"
  }
}

┌──────────────────────────────────────────────────────────────┐
│  User Interaction                                            │
│  ┌──────────────────────────────────────────────────────┐    │
│  │  1. Transaction categorized with low confidence      │    │
│  │  2. User reviews and corrects category               │    │
│  │  3. Feedback submitted via /feedback endpoint        │    │
│  └────────────────────────┬─────────────────────────────┘    │
└───────────────────────────┼──────────────────────────────────┘
                            │
┌───────────────────────────┼──────────────────────────────────┐
│  Feedback Storage                                            │
│  ┌────────────────────────────────────────────────────────┐  │
│  │  • Store in feedback table (PostgreSQL)                │  │
│  │  • Append to corrections.jsonl (file-based)            │  │
│  │  • Track correction count                              │  │
│  └────────────────────────┬───────────────────────────────┘  │
└───────────────────────────┼──────────────────────────────────┘
                            │
┌───────────────────────────┼──────────────────────────────────┐
│  Auto-Retraining Trigger                                     │
│  ┌────────────────────────────────────────────────────────┐  │
│  │  if correction_count >= 50 and correction_count % 50:  │  │
│  │      trigger_retraining()                              │  │
│  └────────────────────────┬───────────────────────────────┘  │
└───────────────────────────┼──────────────────────────────────┘
                            │
┌───────────────────────────┼──────────────────────────────────┐
│  Retraining Pipeline (scripts/train.py)                      │
│  ┌────────────────────────────────────────────────────────┐  │
│  │  1. Load original training data                        │  │
│  │  2. Merge with corrections.jsonl                       │  │
│  │  3. Balance dataset (oversampling weak categories)     │  │
│  │  4. Train new ML classifier                            │  │
│  │  5. Evaluate on validation set                         │  │
│  │  6. Save new model if accuracy improves                │  │
│  └────────────────────────┬───────────────────────────────┘  │
└───────────────────────────┼──────────────────────────────────┘
                            │
┌───────────────────────────┼──────────────────────────────────┐
│  Model Hot Swap (POST /reload-model)                         │
│  ┌────────────────────────────────────────────────────────┐  │
│  │  • Load new model from disk                            │  │
│  │  • Atomic swap in router (no downtime)                 │  │
│  │  • Clear Redis cache                                   │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

# scripts/learn_merchants_from_corrections.py
def learn_merchants_from_corrections():
    """Extract new merchants from user corrections"""
    corrections = load_corrections('data/corrections/corrections.jsonl')

    # Group by merchant pattern
    merchants = {}
    for correction in corrections:
        merchant = extract_merchant(correction['text'])
        if merchant:
            if merchant not in merchants:
                merchants[merchant] = {
                    'canonical_name': merchant,
                    'category': correction['correct_category'],
                    'frequency': 0
                }
            merchants[merchant]['frequency'] += 1

    # Add high-frequency merchants to gazetteer
    for merchant, info in merchants.items():
        if info['frequency'] >= 5:  # Min 5 occurrences
            add_to_gazetteer(merchant, info)

# Database
DATABASE_URL=postgresql://txn_user:txn_password@localhost:5432/txn_user
REDIS_URL=redis://localhost:6379/0

# LLM Provider
LLM_PROVIDER=ollama  # or 'azure'
LLM_URL=http://localhost:11434
LLM_MODEL=llama3.1:8b
LLM_TIMEOUT=120

# Azure OpenAI (alternative)
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com
AZURE_OPENAI_API_KEY=your-api-key
AZURE_OPENAI_CHAT_DEPLOYMENT=gpt-4.5

# Ensemble Weights
MCC_WEIGHT=0.15
RULE_WEIGHT=0.15
ML_WEIGHT=0.65
LLM_WEIGHT=0.05

# Thresholds
AUTO_ACCEPT_THRESHOLD=0.85
REVIEW_THRESHOLD=0.60
ML_CONFIDENCE_THRESHOLD=0.80
RULE_CONFIDENCE_THRESHOLD=0.80

# Performance
USE_ENSEMBLE=true
FAST_MODE=true
ENABLE_PARALLEL=true
MAX_WORKERS=4

# Caching
CACHE_TTL=600  # 10 minutes

# Monitoring
LOG_LEVEL=info
PROMETHEUS_ENABLED=false

# Model configuration
model:
  encoder: "sentence-transformers/all-MiniLM-L6-v2"
  classifier: "lightgbm"
  calibration: true

# Training parameters
training:
  test_size: 0.2
  random_state: 42
  balance_strategy: "oversample"  # or 'undersample', 'smote'

# LightGBM hyperparameters
lightgbm:
  num_leaves: 31
  learning_rate: 0.05
  n_estimators: 100
  max_depth: -1
  min_child_samples: 20
  subsample: 0.8
  colsample_bytree: 0.8

# Active learning
corrections:
  min_for_retraining: 50  # Trigger retraining after 50 corrections
  auto_retrain: true

# apps/api/main.py
allowed_origins = os.getenv(
    "ALLOWED_ORIGINS",
    "http://localhost:3000,http://localhost:3001"
).split(",")

app.add_middleware(
    CORSMiddleware,
    allow_origins=allowed_origins,
    allow_credentials=False,
    allow_methods=["GET", "POST", "PUT", "DELETE", "OPTIONS"],
    allow_headers=["Content-Type", "Authorization"],
)

# Planned implementation using Redis
from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)
app.state.limiter = limiter

@app.post("/categorize")
@limiter.limit("100/minute")
async def categorize_transaction(request: Request, ...):
    ...

Category               Precision  Recall  F1-Score  Support
─────────────────────────────────────────────────────────
food_dining            0.99       0.99    0.99      2,450
groceries              0.98       0.99    0.98      2,120
transport              0.99       0.98    0.99      1,890
bills                  0.98       0.99    0.98      1,780
shopping               0.97       0.98    0.97      2,340
health                 0.99       0.99    0.99      1,230
education              0.98       0.98    0.98      980
fuel                   1.00       0.99    0.99      1,450
travel                 0.98       0.98    0.98      1,120
subscriptions_memberships 0.97    0.97    0.97      890

Test Date: 2025-11-20
Transactions: 10 diverse merchant payments
Total Duration: 63.09 seconds
Success Rate: 100% (10/10)
Average Latency: 6.3 seconds per transaction

# Kubernetes deployment (example)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: txn-api
spec:
  replicas: 4  # Scale to 4 instances
  selector:
    matchLabels:
      app: txn-api
  template:
    spec:
      containers:
      - name: api
        image: txn-ai:latest
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        env:
          - name: DATABASE_URL
            valueFrom:
              secretKeyRef:
                name: db-credentials
                key: url

# Nginx load balancer
upstream txn_api {
    least_conn;  # Least connections algorithm
    server txn-api-1:8000;
    server txn-api-2:8000;
    server txn-api-3:8000;
    server txn-api-4:8000;
}

server {
    listen 80;
    server_name api.txn-categorizer.com;

    location / {
        proxy_pass http://txn_api;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_connect_timeout 120s;
        proxy_send_timeout 120s;
        proxy_read_timeout 120s;
    }
}

from sqlalchemy import create_engine
from sqlalchemy.pool import QueuePool

engine = create_engine(
    DATABASE_URL,
    poolclass=QueuePool,
    pool_size=20,           # Max 20 connections per instance
    max_overflow=10,        # Allow 10 overflow connections
    pool_pre_ping=True,     # Verify connections before use
    pool_recycle=3600       # Recycle connections every hour
)

# Primary (write)
primary_engine = create_engine(PRIMARY_DATABASE_URL)

# Read replica (read)
replica_engine = create_engine(REPLICA_DATABASE_URL)

# Route reads to replica
@app.get("/stats")
async def get_stats():
    with replica_engine.connect() as conn:
        result = conn.execute(query)