Remote Position
[Remote] Lead Scala Data Engineer
🌍 Remote · United States
💼 Full-time
Note: The job is a remote job and is open to candidates in USA. ESG Consulting is seeking an experienced Lead Scala Data Engineer to serve as the primary technical owner of an enterprise data-processing framework supporting Medicaid Encounter Processing and enterprise data ingestion. The role focuses on maintaining and enhancing mission-critical Scala, Spark, Hive, Drools, and Cloudera Data Platform applications, including production support, performance optimization, releases, and cross-functional coordination.
Responsibilities
- Serve as the primary technical owner of the Scala/Spark application framework supporting Medicaid Encounter Processing
- Maintain and enhance production applications developed using Scala, Spark, Hive, and Drools
- Own Drools business-rule implementation and ongoing rule maintenance
- Support enterprise ingestion and processing of provider, member, reference, eligibility, and encounter data
- Own Spark and Hive batch-processing workflows
- Troubleshoot production issues, identify root causes, and resolve application defects
- Implement business, regulatory, and application changes
- Manage production releases, version control, and deployment coordination
- Perform Spark performance tuning and optimization
- Monitor and provide basic operational support for the Cloudera Data Platform (CDP)
- Maintain technical documentation, operational procedures, and knowledge-transfer materials
- Coordinate with infrastructure, cloud operations, QA, business, and other technical teams
Skills
- 7+ years of experience developing enterprise-scale distributed data-processing applications
- Strong hands-on Scala development experience, preferably 4–6+ years
- 4–6+ years of Apache Spark and Hive development experience
- Significant experience developing and maintaining applications on Cloudera Data Platform (CDP) 7.x or equivalent enterprise Hadoop environments
- Hands-on experience implementing business rules using the Drools Rules Engine, preferably 4–6 years
- Strong SQL development and query-optimization skills
- Experience supporting Linux-based production environments
- Strong experience troubleshooting distributed Spark applications in production
- Experience with Git and modern version-control practices
- Demonstrated experience taking technical ownership of production applications, including incidents, defects, enhancements, releases, deployments, and performance issues
- Capable of independently owning a production application across the complete Scala + Spark + Hive + Drools + Cloudera technology stack
- Medicaid or healthcare industry experience
- Experience with Medicaid Encounter Processing
- Experience with Cloudera Manager, HDFS, and YARN
- Experience integrating with IBM DataStage
- Familiarity with AWS infrastructure supporting Cloudera
- Experience working in Agile environments using Jira and Confluence
- Scala/Spark Data Engineer with strong Cloudera/CDP experience
Company Overview