[Remote] DIRECTOR, DATA & AI ENGINEERING — DATABRICKS LEAD

Note: The job is a remote job and is open to candidates in USA. AbarVa is hiring a Director, Data & AI Engineering to lead and build its US Databricks engineering practice while remaining hands-on in architecture and implementation. The role covers data ingestion, migration and recovery of legacy data estates, governance, AI-ready data products, and engineering team development.

Responsibilities

• Lead and build: build and lead the US Databricks engineering team — hire, set standards, review code, mentor and grow engineers — while staying hands-on in the hardest problems yourself
• Ingestion framework: design and build a metadata-driven ingestion framework so a new source is onboarded by configuration, not another hand-written pipeline — and hand it to the client’s team as a working asset
• Estate assessment & migration: assess existing SQL, ETL, stored-procedure and warehouse estates with Databricks Lakebridge — inventory code, score complexity and dependencies, accelerate conversion, reconcile legacy-vs-Databricks parity — then turn the assessment into a sized, wave-based build plan
• Lineage & recovery: trace lineage end to end and reconstruct undocumented business logic from analytics estates being decommissioned; validate against source code, job metadata and owners; record the unknowns
• Medallion & data products: design scalable Bronze/Silver/Gold architectures and data products — named owner, contract, quality thresholds, lineage, documented consumers — run as a backlog and release cadence, not project phases
• Governance: design and implement Unity Catalog governance — lineage, security, access control, tenant isolation, data-product patterns
• Engineering velocity: stand up CI/CD and the toolchain for speed — Databricks Asset Bundles, Git workflows, automated testing, environment promotion, IaC, and observability that catches a broken pipeline before the business does
• Data for AI: model and certify Gold data for natural-language consumption through Databricks Genie and Claude on Bedrock — clear semantics, defined metrics, described columns, certified tables
• Partner & scale: partner directly with client CTO, CDAO, architecture, security and engineering leaders, and help build AbarVa’s Data & AI engineering capability as we scale

Skills

• An active Databricks certification — Data Engineer Professional, Solutions Architect, or an equivalent current credential
• Proven experience building and leading data engineering teams — hiring, standards, code review, mentoring — while remaining personally hands-on
• 12–18+ years of hands-on data engineering across modern data architecture, modeling and delivery, with exceptional depth in Databricks, Spark, SQL and Python
• Demonstrated experience building metadata-driven ingestion frameworks — Auto Loader / Lakeflow, CDC, schema evolution, data-quality expectations and reconciliation
• Hands-on Databricks Lakebridge experience (or directly equivalent assessment/transpile/reconcile tooling) to analyze existing data loads, size migrations and prove parity
• Deep Unity Catalog, medallion and data-product architecture; strong command of reconciliation, data quality, identity, master and reference data
• Real CI/CD practice for data
• Strong cloud architecture experience
• Familiarity with legacy platforms — Netezza, Teradata, Exadata, SQL Server, SAS
• Preferably AWS
• You use AI coding assistants daily and can set the team's standard for using them well — including where they're unreliable
• Executive presence that holds a room with senior client leaders while keeping technical credibility with engineers
• MVP or Champion status

Company Overview