Data Engineer (Exp-8+Y) with Databricks exp-Full time-Remote
We are seeking a highly experienced Senior Data Engineer / Databricks Lead to design, develop, and optimize enterprise-scale data engineering solutions. The ideal candidate will have strong expertise in Databricks, PySpark, SQL, Delta Lake, and Unity Catalog, along with proven technical leadership and team mentoring experience.
Key Responsibilities
Lead the architecture and development of data solutions using Databricks, Delta Lake, Unity Catalog, and Medallion Architecture. Own end-to-end delivery of data engineering projects and provide technical guidance to engineering teams. Design, develop, and optimize scalable ETL/ELT pipelines using PySpark and SQL. Implement real-time and streaming data processing solutions using Structured Streaming and Delta Live Tables. Optimize Databricks workloads, improve performance, and manage cloud compute costs. Establish CI/CD pipelines, automated testing, monitoring, and data governance standards. Manage infrastructure and deployments using Terraform and Databricks Asset Bundles. Collaborate with business stakeholders, product owners, and cross-functional teams to translate requirements into technical solutions. Mentor engineers, conduct code reviews, and contribute to Agile sprint planning and delivery. Troubleshoot complex platform issues and ensure data reliability, security, and scalability. Required Skills 8+ years of experience in Data Engineering, including 5+ years of hands-on Databricks experience in production environments. Strong expertise in PySpark, Python, and SQL. Experience with Delta Lake, Unity Catalog, and Medallion Architecture. Hands-on experience with Structured Streaming and/or Delta Live Tables. Experience with Terraform and/or Databricks Asset Bundles. Experience with at least one major cloud platform: AWS, Azure, or Google Cloud Platform. Proven experience in technical leadership, mentoring, and team guidance. Strong understanding of CI/CD, data governance, monitoring, and performance optimization. Good knowledge of Agile/Scrum methodologies and excellent communication skills.
Preferred Qualifications
Databricks Professional Certification. Experience defining engineering standards across multiple teams or projects. Strong understanding of cloud cost optimization and production support. Ability to balance technical quality, business requirements, and delivery timelines.