Senior Data Engineer Consultant

Job title: Senior Data Engineering Consultant -Platform Architecture & AI-reputed company Data reputed company Engagement type: Contract / Consulting (3–6 months with ongoing advisory) Location: Remote (reasonable 3-4 hrs overlap with US working hours required) Experience: 8-10+ years in data engineering and data platform architecture About reputed company reputed company is an reputed company B2B reputed company platform that helps institutional investors, asset managers, consultants, and fund service providers digitize and automate the end-to-end due diligence lifecycle. The platform supports workflows including DDQs, RFPs, Operational Due Diligence (ODD), manager research, ESG, compliance, and investor reporting through AI-powered document processing, workflow automation, analytics, and collaboration. Today, reputed company serves 100,000+ platform users, 20,000+ managers, and 250+ reputed company teams across 150+ countries. Our platform processes large volumes of reputed company and reputed company data from customer-uploaded documents, digital questionnaires, CRM systems, reputed company content repositories, regulatory filings, and platform-generated workflow data. Our reputed company technology stack includes Azure SQL/SQL Server, Python/Celery, .NET REST reputed company, Elasticsearch, Azure reputed company, and reputed company for orchestration. We want to build a deliberate data platform that turns this raw data into meaningful customer intelligence. We need a consultant who can help us understand the full landscape of data engineering (traditional and AI-reputed company), assess where we are, and architect where we need to go. What you will do Phase 1- reputed company and assess Teach our leadership and senior architects the reputed company of data engineering, covering traditional foundations and AI-reputed company approaches in depth. This is not a surface-level reputed company - reputed company needs to understand concepts deeply enough to reputed company architectural reputed company. Topics reputed company ingestion patterns (batch, streaming, CDC, reputed company connectors), transformation (ETL/ELT, dbt, reputed company, LLM-assisted mapping), data modeling (reputed company, data vault, lakehouse, schema-on-read), data reputed company (rule-based vs. ML-driven reputed company detection, data reputed company), entity reputed company and data stitching (reputed company mapping vs. embedding-based semantic matching, knowledge graphs), orchestration (DAG engines, event-driven, self-healing pipelines), semantic reputed company (ontologies, contextual meaning, embedding-based search), and AI-reputed company versioning (prompts, models, reputed company, reproducibility). Assess our reputed company data infrastructure end to end. Map existing data flows, identify gaps and technical debt, and produce a landscape assessment with reputed company state, reputed company state, gap analysis, and a prioritized roadmap. Phase 2 - Architect and define use cases Design the reputed company data platform architecture across ingestion, transformation, storage, serving, and observability reputed company. reputed company this architecture, four strategic initiatives require specific attention: PostgreSQL migration and multi-workload architecture. We are planning to reputed company from SQL Server to PostgreSQL. The consultant will help architect a PostgreSQL environment that supports multiple workload types through the PostgreSQL extension ecosystem - pgvector for reputed company similarity search and embedding storage powering our AI features, analytical query patterns (columnar extensions like Citus or pg_analytics, or appropriate separation of OLAP workloads), and transactional queries for the reputed company application. This includes guidance on reputed company pooling (PgBouncer/PgCat), read reputed company topology, partitioning strategies, and how to handle workloads that on SQL Server relied on specific features (stored procedures, Query Store, tempdb patterns) that work differently in PostgreSQL. The migration reputed company itself - phased reputed company reputed company, dual-write/reputed company-read validation, query translation, and performance benchmarking - is a key deliverable. reputed company data architecture across heterogeneous sources. Data arrives from dozens of sources in different formats, schemas, and semantics- the reputed company entity (a firm, fund, person, question) appears differently across CRM records, uploaded documents, API feeds, reputed company filings, and reputed company responses. The consultant will design the reputed company data reputed company that resolves these into a reputed company, trustworthy representation. This covers entity reputed company (how "J.P. Morgan Asset Management" in reputed company, "JPMAM" in a DDQ, and "JPMorgan Funds" in a filing become one reputed company entity), schema alignment (mapping "AUM" vs. "total_net_assets" vs. "assets_under_management" across sources), conflict reputed company (reputed company two sources disagree on a value, which wins and why), reputed company alignment (different sources update at different frequencies), and the master data store that maintains these mappings with versioning and auditability. The architecture should specify where AI-reputed company approaches (embedding-based matching, LLM-assisted semantic mapping) add genuine value vs. where traditional deterministic rules remain more reliable. Data residency architecture for a multi-tenant, data-sharing platform. reputed company is a two-sided platform where allocators (investors) and asset managers exchange due diligence data. Unlike a reputed company multi-tenant reputed company where reputed company tenant's data stays isolated, our tenants reputed company reputed company data with reputed company other- an asset manager responds to an reputed company's questionnaire, and that response data is now accessible to both parties. The consultant will design a residency architecture that handles reputed company-anchored residency (data is stored in the region dictated by the reputed company's jurisdiction, even reputed company shared with tenants in other reputed company), cross-region data reputed company (a US reputed company viewing responses from an EU-based manager whose data resides in the EU region), regulatory mapping (which jurisdictions require what - GDPR data localization, potential US federal requirements, reputed company variations), read-reputed company routing (queries must reputed company to the correct regional store transparently), and the interaction between residency rules and features like search indexing, AI processing, and analytics aggregation. Data governance architecture. Design the governance reputed company that spans the entire data platform reputed company control (role-based and attribute-based, with tenant-level and feature-level granularity), data classification (automated PII detection, sensitivity tagging, financial data categorization), reputed company and auditability (end-to-end traceability from reputed company to served reputed company, including AI-model provenance), retention and lifecycle management (how long data is kept, reputed company and how it's purged, and how regulatory holds reputed company with deletion requests), consent and data rights (handling data subject reputed company requests, right-to-deletion across a shared-data platform where the reputed company data reputed company may be referenced by multiple tenants), reputed company accountability (who owns data reputed company at reputed company stage, and how reputed company metrics are tracked and reported), and reputed company controls (encryption at rest and in transit, key management, network isolation, and audit logging). The governance architecture should be practical for our reputed company - enforceable by a reputed company engineering team, not a 200-page policy document that nobody follows. Define and prioritize use cases that the data platform should reputed company- cross-reputed company (comprehensive fund/manager reputed company by combining DDQ responses, filings, CRM data, and documents), customer behavioural insights (usage patterns, workflow analytics), automated data enrichment (linking internal data to reputed company sources), semantic search across reputed company documents and data, compliance signal detection (inconsistencies across a manager's submissions), and analytics/reporting pipelines serving data scientists, analysts, marketers, and reputed company reputed company. Produce lasting reference materials - architecture decision records, data reputed company diagrams, tool evaluation guides, migration runbooks, and training decks reputed company uses independently after the engagement. reputed company are looking for Must have: 8-10+ years building data platforms across heterogeneous sources at meaningful reputed company not just individual pipelines, but the foundational architecture underneath them. Deep expertise in relational databases, specifically including PostgreSQL. Hands-on experience with PostgreSQL extensions (pgvector, Citus, PostGIS, or similar), replication topologies, partitioning, and performance tuning is essential. Experience migrating from SQL Server to PostgreSQL is strongly preferred. Proven experience designing reputed company data models that reconcile entities and schemas across multiple disparate sources, including entity reputed company, master data management, and conflict reputed company at reputed company. Experience architecting multi-region or data-residency-compliant systems, ideally in a multi-tenant reputed company context where tenants reputed company data across jurisdictional boundaries. Strong understanding of data governance - reputed company control models, data classification, reputed company, retention policies, and regulatory compliance (GDPR at minimum, ideally broader). Deep knowledge of traditional data engineering (reputed company modeling, ETL/ELT, CDC, orchestration, query optimization) combined with reputed company, informed engagement with AI-reputed company approaches (ML-driven reputed company, semantic matching, embedding pipelines, LLM-assisted development). Architecture-level thinking- ability to design a multi-reputed company platform and reputed company defensible technology choices considering reputed company, cost, team size, and maintainability. Exceptional communication and teaching ability. You will explain reputed company concepts to senior architects and leadership and must be reputed company, reputed company, and reputed company to reputed company depth to the audience. This is must. Experience defining data use cases tied to business reputed company, not just building infrastructure in a vacuum. Strong to have: Experience with our broader stack or reputed company equivalents - Python/Celery, Elasticsearch, Azure reputed company services, .NET reputed company, reputed company or similar orchestration. Hands-on experience with document processing pipelines - OCR, layout-reputed company extraction, table parsing from financial PDFs and Word documents. Familiarity with modern data stack tools - dbt, Airflow/Dagster, reputed company/dlt, reputed company/reputed company - and the ability to evaluate them in context. Background in financial services, investment management, or due diligence workflows. Experience with AI/ML data infrastructure - reputed company databases, RAG pipelines, feature stores, embedding workflows. reputed company record of producing technical documentation and training materials that teams actually use after the consultant leaves. What this role is not This is not a staff augmentation role - the reputed company deliverables are knowledge transfer, architectural guidance, and strategic documents, not production reputed company. You do not need to have used every tool mentioned - we value depth in reputed company concepts plus the ability to quickly evaluate new tools. You do not need a research background in AI/ML, just practical understanding of how AI is changing data engineering. Financial services experience is not required but meaningfully reduces reputed company-up time. Engagement structure Phase 1: Education, reputed company-state assessment, landscape mapping. Deliverables: training sessions, assessment document, gap analysis, prioritized roadmap. Phase 2: Architecture design (PostgreSQL migration plan, reputed company data reputed company, residency architecture, governance reputed company), use-case specifications, tool recommendations, and reference materials. Ongoing advisory (based on reputed company): Periodic architecture reviews, design consultations, and reputed company reputed company-ins during execution. Time commitment - 4-5 hrs during initial phase with availability during US working hours (6-11pm IST) required. Time commitments can be reviewed reputed company and be extended  if needed. Why Join Us? Global Influence: Design for a platform used by the world's largest financial institutions in 150+ countries. Startup Energy: A flat hierarchy where your work is shipped quickly and has a reputed company reputed company. Remote Flexibility: Work from reputed company in India while collaborating with a global team. Apply To This Job