Senior Databricks Data Engineer / Technical Lead

### Senior Databricks Data Engineer / Technical Lead

Job Summary

We are seeking a hands-on

Senior Databricks Data Engineer / Technical Lead

to support a U.S. Department of Transportation (DOT) data modernization initiative. The role will focus on building and migrating scalable data pipelines to a

Databricks Lakehouse

, modernizing legacy data sources, and supporting production data operations.

Key Responsibilities

• Design and develop scalable

Databricks data pipelines

using Medallion Architecture (Bronze/Silver/Gold).

• Build metadata-driven ingestion and transformation frameworks for legacy and external data sources.
• Develop and optimize

Python, PySpark, and SQL

workloads.

• Work with Databricks, Delta Lake, Unity Catalog, Auto Loader/Lakeflow, and Databricks Jobs/Workflows.
• Integrate data from

AWS/S3, CSV/JSON files, REST APIs

, and other sources.

• Implement data quality, validation, reconciliation, schema evolution, lineage, and governance.
• Manage orchestration, dependencies, monitoring, alerting, retries, and production support.
• Collaborate with architects, PMs, DevSecOps, security, governance, and business teams.
• Support migration to the

AWS/OneDOT

environment while ensuring security and performance.

Required Qualifications

• 7+ years of professional

Data Engineering

experience.

• Strong hands-on experience with

Databricks/Spark

and production data pipelines.

• Advanced

Python, PySpark, and SQL

skills.

• Experience with

Delta Lake, AWS/S3, Git, CI/CD, and data orchestration

.

• Experience handling sensitive data, access controls, and secrets management.
• Bachelor's degree in Computer Science, IT, Engineering, Data Engineering, or related field, or equivalent experience.

Preferred

• Experience migrating

Sybase/SAP IQ or similar legacy platforms to Databricks

.

• Experience with

Unity Catalog, metadata-driven ingestion, Auto Loader/Lakeflow, and Databricks Workflows

.

• Federal/DOT project experience.
• Databricks and/or AWS certifications.
• Experience with streaming, AI/ML data pipelines, data governance, and cloud cost optimization.