[Remote] Senior Data Engineer – Databricks & AI Data Platform

Note: The job is a remote job and is open to candidates in USA. Brillio is hiring a Senior Data Engineer to build and scale its next-generation data and AI platform. The role focuses on developing scalable data pipelines and trusted, AI-ready data foundations while enabling analytics, knowledge management, and AI use cases with governance, performance, and compliance in mind.

Responsibilities

• Design, develop, and maintain highly scalable data pipelines on Databricks leveraging Delta Lake, Unity Catalog, and Databricks Workflows
• Ingest, integrate, and process large-scale datasets from research, commercial, content, operational, and third-party sources
• Implement batch and real-time data processing solutions to support enterprise analytics and AI initiatives
• Ensure reliable, secure, and high-performance data movement across cloud platforms and business systems
• Build the core data infrastructure supporting enterprise ontology, semantic layer, and knowledge graph initiatives
• Design and implement robust data models that enable efficient data discovery, contextual understanding, and relationship mapping
• Develop entity resolution frameworks to improve data consistency and create trusted views of business-critical information
• Create curated, reusable, and AI-ready data products that can be leveraged across multiple business domains
• Prepare and deliver high-quality datasets for Azure AI Foundry and advanced AI/ML applications
• Build and optimize chunking, embedding, vectorization, and retrieval pipelines for Generative AI and RAG-based solutions
• Develop feature engineering pipelines that support machine learning and predictive analytics initiatives
• Enable seamless data access for AI applications through real-time and batch processing frameworks
• Implement enterprise data governance standards using Unity Catalog
• Establish robust data quality controls, lineage tracking, metadata management, and auditing capabilities
• Define and maintain role-based access controls and security policies to ensure data privacy and regulatory compliance
• Collaborate with governance and compliance teams to support enterprise data management standards
• Monitor and optimize platform performance, scalability, and operational efficiency
• Implement cost optimization strategies across data storage, processing, and compute layers
• Develop reusable frameworks, templates, and engineering standards that accelerate onboarding of new data and AI use cases
• Champion engineering best practices including automation, CI/CD, testing, monitoring, and observability

Skills

• Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems, or a related field
• 7+ years of experience in Data Engineering, Data Platform Development, or Big Data environments
• Hands-on expertise with Databricks, including Delta Lake, Unity Catalog, Workflows, Spark, and Lakehouse architecture
• Strong experience building large-scale ETL/ELT pipelines and modern cloud data platforms
• Experience working with Azure Data Services and enterprise cloud environments
• Strong proficiency in Python, PySpark, and SQL
• Deep understanding of data modeling, metadata management, master data management, and data governance principles
• Experience designing and implementing enterprise-scale data architectures

Company Overview

Company H1B Sponsorship