Transaction Schema Unification Refactor

Overview

This document explains a critical refactoring that unified transaction schemas across domains, fixed a movement_type extraction bug, and improved the overall architecture of our financial document parsing system.

The Problem We Faced

Initial Issue

Users were getting validation errors when uploading bank statements:

1Field required [type=missing, input_value={'date': '2025-04-07', 'd...202,06', 'category': ''}, input_type=dict]
2raw_statement.transactions.0.movement_type

Root Cause Analysis

Through systematic debugging, we discovered the issue wasn't with AI extraction (OpenAI was correctly extracting movement_type), but with our data flow architecture.

Architecture Problems

1. Duplicate Transaction Schemas

We had two different transaction schemas serving the same purpose:

2. Data Loss During Conversion

The statements service was manually converting between transaction formats:

Result: Even though OpenAI extracted movement_type correctly, it was dropped during conversion.

3. Wrong Domain Ownership

1❌ BEFORE: AI Layer defines TransactionData
2- AI concerns mixed with business logic
3- Transaction schema owned by infrastructure layer
4- Domains import from infrastructure
5
6✅ AFTER: Transactions Domain defines TransactionData
7- Business logic owns business schemas
8- AI layer imports from domain
9- Proper dependency direction

The Solution: Domain-Driven Schema Unification

Step 1: Move Schema to Correct Domain

Moved transaction schema to its rightful owner:

Step 2: Update Dependencies

AI Layer now imports from transactions domain:

Statements Domain uses same schema:

Step 3: Eliminate Redundant Conversion

BEFORE (Manual Conversion):

AFTER (Direct Usage):

Technical Implementation

Data Flow (Fixed)

11. PDF Upload → Extract Text
22. Text → OpenAI API with TransactionData schema
33. OpenAI extracts movement_type correctly ✅
44. Returns List[TransactionData] with all fields ✅
55. Direct assignment to RawBankStatement ✅
66. Validation succeeds ✅

Key Changes Made

Educational Insights

Why This Refactor Was Necessary

Domain-Driven Design Principles:

• Business concepts belong in business domains
• Infrastructure should depend on domain, not vice versa
• Avoid schema duplication across layers

Schema Evolution Problems:

• Manual conversion is error-prone - easy to forget fields
• Schema drift happens when definitions are duplicated
• Maintenance overhead increases with multiple definitions

How We Debugged This

Systematic Approach:

1. Traced data flow from AI response to validation
2. Checked each transformation step for data loss
3. Identified the exact line where movement_type was dropped
4. Root cause analysis revealed architectural issue

The error was NOT in AI extraction (which worked perfectly), but in our data transformation logic.

Lessons Learned

Benefits Achieved

Immediate Fixes

• ✅ movement_type extraction works - no more validation errors
• ✅ Cleaner codebase - eliminated redundant conversion logic
• ✅ Reduced complexity - fewer lines of code, fewer bugs

Long-term Improvements

• ✅ Easier maintenance - single place to update transaction schema
• ✅ Automatic compatibility - changes flow through automatically
• ✅ Better architecture - proper domain boundaries
• ✅ Future-proof - easier to add new transaction fields

Performance Gains

• ✅ No unnecessary object creation during conversion
• ✅ Direct object usage reduces memory allocations
• ✅ Simpler code paths improve readability and performance

Best Practices Established

Schema Design

Dependency Management

Data Transformation

Future Considerations

Schema Evolution

• Add new fields only to transactions domain
• Changes automatically propagate to all consumers
• Version migration can be handled in one place

Testing Strategy

• Test schema consistency across domains
• Validate data flow from AI to database
• Monitor for schema drift in CI/CD

Monitoring

• Log schema field counts to detect missing fields
• Track AI extraction success rates for new fields
• Alert on validation failures during parsing

Conclusion

This refactor demonstrates the importance of proper domain-driven design and schema management. By moving transaction schemas to their rightful domain and eliminating redundant conversions, we:

1. Fixed immediate bugs (movement_type extraction)
2. Improved system architecture (proper domain boundaries)
3. Reduced future maintenance (single source of truth)
4. Enhanced debuggability (cleaner data flow)

The lesson: Architecture problems often manifest as data transformation bugs. When debugging, look beyond the immediate error to understand the underlying structural issues.