Data Engineer (Databricks, Azure, Kafka)
Job Title: Data Engineer (Databricks, Azure, Kafka) Location: Remote Job Type: W2 / C2C / 1099 Job Summary: We are looking for an experienced Data Engineer with 9+ years of experience to design, build, and maintain data pipelines using Databricks, Spark, and Azure cloud services. The ideal candidate has hands-on experience with modern data engineering tools, cloud automation, and event-driven architectures. Job Responsibilities: Design, develop, and optimize scalable data pipelines using Databricks and Spark (PySpark) Build and maintain ETL/ELT workflows integrating with Snowflake and other data platforms Develop and manage Azure-based data solutions using ADF, ADLS, Azure Functions, and Key Vault Implement CI/CD pipelines and infrastructure automation using GitHub Actions, Terraform, and DevOps best practices Build and maintain event-driven architectures using Kafka, including CDC and streaming integrations Work with AI/GenAI initiatives involving LLM fine-tuning, RAG, and prompt engineering Monitor, troubleshoot, and optimize data pipeline performance and reliability Collaborate with data architects, analysts, and business stakeholders to translate requirements into technical solutions Ensure data quality, security, and governance standards are met across pipelines Orchestrate and schedule workflows using Airflow Required Skills: 8+ years of experience in Data Engineering Data Engineering: Databricks, Spark (PySpark), Python/Scala, SQL, Snowflake Azure: Azure Cloud Services, ADF, ADLS, Azure Functions, Key Vault, DevOps AI/GenAI: LLM Fine-Tuning, RAG, Prompt Engineering Automation & DevOps: Airflow, GitHub, GitHub Actions, CI/CD, Terraform, Infrastructure Automation Kafka, Event-Driven Architecture, CDC/Streaming Integrations Preferred Qualifications: Bachelor's or Master's degree in Computer Science, Engineering, or related field Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect) is a plus Experience working in large-scale enterprise or regulated industry environments (finance, healthcare, insurance) Strong communication skills with ability to work directly with client stakeholders