[Remote] Senior Data Engineering Support Engineer - Databricks & PySpark

Note: The job is a remote job and is open to candidates in USA. NTT DATA is recruiting for a confidential client seeking a Senior Data Engineering Support Engineer. The role supports production Databricks and PySpark data pipelines through incident response, performance and cost optimization, reliability automation, and technical mentorship.

Responsibilities

• Critical Incident & Escalation Management: Act as the highest tier of internal escalation for complex, high-priority production failures in massive-scale batch and streaming PySpark jobs
• Deep-Dive Performance Tuning: Drive the resolution of complex distributed computing bottlenecks, including severe data skew, Out-Of-Memory (OOM) errors, improper partition pruning, and costly shuffle operations using Spark UI, logs, and Ganglia metrics
• Cost & Cluster Optimization: Analyze Databricks cluster utilization and run strategies to reduce cloud compute spend (e.g., configuring auto-scaling, spot instances, and choosing optimal driver/executor node types)
• Governance & Reliability Frameworks: Oversee and troubleshoot data access control, schema evolution, and data lineage tracking within Databricks Unity Catalog
• Automation & SRE Mindset: Build robust, automated frameworks and scripts in Python/PySpark to automate data reconciliation, system health checks, and self-healing pipelines
• Strategic Playbooks & Documentation: Author comprehensive, technical runbooks and post-mortem RCA documents. Establish SOPs (Standard Operating Procedures) to reduce future incident resolution time (MTTR)
• Mentorship & Leadership: Provide technical guidance to junior support engineers, conduct technical code reviews for hotfixes, and closely collaborate with core development teams to bridge the gap between engineering and operations

Skills

• 7+ years of total experience in Data Engineering
• 3+ years dedicated to supporting or optimizing large-scale Databricks and PySpark systems in production
• Advanced PySpark and Python engineering skills, with an explicit understanding of Spark architecture (lazy evaluation, DAG execution, execution memory vs. storage memory)
• High proficiency in advanced SQL for complex analytical troubleshooting
• Expert-level understanding of Delta Lake internals (ACID transactions, Time Travel, Liquid Clustering, Optimize/Z-Order commands)
• Proven track record of diagnosing and resolving distributed data pipeline issues at scale, including thread dumps, memory leaks, and serialization errors
• Strong architectural experience deploying and managing resources within major cloud networks (AWS, Azure, or GCP)
• Robust understanding of Git-based workflows, automated testing, and secure hotfix deployment via CI/CD pipelines
• Active Databricks Certifications (e.g., Databricks Certified Data Engineer Professional)
• Deep experience managing production environments utilizing Delta Live Tables (DLT) and advanced Unity Catalog rule engines

Benefits

• Medical, dental, and vision insurance with an employer contribution
• Flexible spending or health savings account
• Life and AD&D insurance
• Short and long term disability coverage
• Paid time off
• Employee assistance
• Participation in a 401k program with company match
• Additional voluntary or legally required benefits
• Remote role

Company Overview

Company H1B Sponsorship