Databricks Architect
Job Description – Databricks Architect Company: Celebal Technologies Job Title: Databricks Architect Employment Type: Full-Time, Permanent Location: Remote (Must be open to travel as per business requirements) Work Authorization: Only US Citizens and Green Card holders About Celebal Technologies Celebal Technologies is a global technology company specializing in Data Engineering, Data Science, Artificial Intelligence, Generative AI, and Cloud Transformation. As a Databricks Elite Partner, we help enterprises accelerate their data and AI transformation journeys through innovative cloud-based solutions. Role Overview We are looking for an experienced Databricks Architect to lead the design, architecture, and implementation of scalable, secure, and high-performance data solutions using Databricks on AWS and/or Azure. The ideal candidate should have strong expertise in Databricks Lakehouse architecture, Delta Live Tables (DLT), Medallion Architecture, and Auto Loader. The candidate will be responsible for defining technical architecture, driving data engineering best practices, and delivering enterprise-scale data platforms.
Key Responsibilities
Design and implement scalable, secure, and high-performance Lakehouse architectures using Databricks. Define and implement end-to-end data engineering solutions on AWS and/or Azure. Architect and implement Medallion Architecture (Bronze, Silver, and Gold layers). Develop and optimize Delta Live Tables (DLT) pipelines for batch and streaming workloads. Implement Databricks Auto Loader for efficient and incremental ingestion of structured and unstructured data. Establish data governance, security, access controls, and best practices using Unity Catalog. Optimize Spark workloads, cluster configurations, job performance, and compute costs. Design and implement data integration and transformation frameworks using PySpark and SQL. Collaborate with business stakeholders, data engineers, and cross-functional teams to translate business requirements into technical solutions. Provide technical leadership, architecture guidance, code reviews, and mentorship to engineering teams. Ensure best practices around CI/CD, deployment automation, monitoring, and data quality. Required Skills & Qualifications Strong hands-on experience in Databricks architecture and Lakehouse implementation. Extensive experience with AWS and/or Microsoft Azure cloud platforms. Strong expertise in Delta Live Tables (DLT) and pipeline development. Hands-on experience implementing Medallion Architecture (Bronze, Silver, Gold). Strong experience with Databricks Auto Loader for incremental and scalable data ingestion. Proficiency in PySpark, Python, and Spark SQL. Experience with Delta Lake, Unity Catalog, and data governance. Strong understanding of batch and real-time/streaming data processing. Experience with performance tuning, optimization, and large-scale data processing. Knowledge of CI/CD, Git, and DevOps practices for Databricks deployments. Excellent communication, stakeholder management, and technical leadership skills.
Preferred Qualifications
Databricks certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Data Engineer Associate. Experience with Terraform and Infrastructure as Code. Knowledge of data modeling, enterprise data architecture, and data quality frameworks. Experience with cloud-native services such as AWS S3, Glue, Lambda, Azure Data Lake Storage, Azure Data Factory, and Azure Key Vault. Experience leading enterprise-level Data & AI transformation initiatives. Eligibility Criteria Must be a US Citizen or Green Card holder. Candidates must be comfortable working remotely and traveling as required by business needs. Must be available for full-time permanent employment.
What We Offer
Opportunity to work on enterprise-scale Data & AI transformation projects. Exposure to cutting-edge Databricks and cloud technologies. Opportunity to collaborate with global teams and industry-leading clients. A growth-oriented environment with opportunities for technical leadership and career development.