[Remote] Senior Databricks Data Engineer

Note: The job is a remote job and is open to candidates in USA. Artmac is seeking a Senior Databricks Data Engineer to design, develop, and optimize scalable data engineering solutions. The role focuses on data pipelines, Lakehouse architecture, ETL/ELT workflows, cloud data platforms, and reliable, secure data processing.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks, PySpark, Apache Spark, and SQL
• Build and optimize ETL/ELT workflows to process large volumes of structured and semi-structured data
• Implement Lakehouse solutions using Delta Lake and Medallion Architecture (Bronze, Silver, and Gold layers)
• Develop batch and streaming data processing solutions to support business and analytics requirements
• Optimize Spark jobs, cluster configurations, partitioning, and data storage to improve performance and control processing costs
• Integrate data from multiple sources, including databases, APIs, cloud storage, and enterprise applications
• Implement data validation, reconciliation, error handling, and monitoring to ensure data accuracy and reliability
• Work with cloud platforms such as Microsoft Azure, AWS, or Google Cloud, depending on project requirements
• Implement data governance, access controls, and secure data management using Unity Catalog and related platform capabilities
• Develop automated workflows, job scheduling, and dependency management for production data pipelines
• Troubleshoot production issues, identify root causes, and implement long-term solutions

Skills

• 8 15 years of experience in data engineering, big data development, or related roles
• Strong hands-on experience with Databricks and Apache Spark
• Advanced programming skills in Python and PySpark
• Strong SQL skills, including complex queries, joins, aggregations, and query optimization
• Experience building production-grade ETL/ELT pipelines and data integration workflows
• Hands-on experience with Delta Lake, Delta tables, and Lakehouse architecture
• Understanding of Medallion Architecture and scalable data modeling practices
• Experience with at least one major cloud platform: Azure, AWS, or GCP
• Experience with data pipeline orchestration, scheduling, monitoring, and troubleshooting
• Knowledge of Git, CI/CD pipelines, automated testing, and deployment practices
• Strong understanding of data quality, data governance, security, and performance optimization
• Familiarity with Terraform, infrastructure as code, or cloud resource management
• Experience working with enterprise-scale data platforms and high-volume datasets
• Databricks certifications or relevant cloud data engineering certifications
• Bachelor's degree or equivalent combination of education and experience

Benefits

• Remote work

Company Overview

Company H1B Sponsorship