Overview

A multi-strategy investment firm was receiving raw market and position data from six different systems (prime broker feeds, internal OMS, third-party risk engine, custody bank, fund administrator, and Bloomberg). Each source had different formats, delivery schedules, and data quality issues. Analytics and reporting downstream were consuming bad data.

What We Built

• Ingestion layer: adapters per source (SFTP, REST API, email attachment, direct DB query)
• Normalization engine: field mapping, type coercion, duplicate detection, late-arrival handling
• Data quality rules: threshold checks, cross-source reconciliation, anomaly flagging
• Load into Snowflake with SCD2 history preservation
• Operational dashboard: pipeline run status, row counts, quality score per feed

Stack

Python, Airflow, Snowflake, PostgreSQL (operational store), AWS S3, dbt for transformations.

Outcome

Eliminated manual data reconciliation work (~15 hours/week across ops team). Report generation time cut from overnight to under 30 minutes.