Remote Position
Senior Data Engineer - DBT
🌍 Remote · United States
💼 Full-time
Role: Senior Data Engineer - DBT Location: NYC, NY (Remote) Duration: Long-Term Job Description: Must have below: 10+ Years Experience Great Communicator/Client Facing Individual Contributor 100% Hands on in the mentioned skills DBT Proficiency: model development: Experience in creating complex DBT models including incremental models, snapshots and documentation. Ability to write and maintain DBT macros for reusable code Testing and documentation: Proficiency in implementing DBT tests for data validation and quality checks Familiarity with generating and maintaining documentation using DBT's built in features Version control: Experience in managing DBT projects using git, including implementing CI/CD process from the scratch
- AWS Expertise:
- Data STORAGE solutions:
- In depth understanding of AWS S3 for data storage, including best practices for organization and security
- Experience with AWS redshift for data warehousing and performance optimization
- Data Integration:
- Familiarity with Aws glue for ETL processes and orchestration -Nice to have
- Experience with AWS lambda for serverless data processing tasks
- Workflow Orchestration:
- Proficiency in using Apache Airflow on AWS to design ,schedule and monitor complex data flows
- Ability to integrate Airflow with AWS services and DBT models such as triggering a DBT model or EMR or reading from s3 writing to redshift
- Data Lakes and Data warehousing:
- Understanding the architecture of data lakes vs data warehouses and when to use each
- Experience with amazon Athena for querying data directly in s3 using SQL
- Monitoring and Logging:
- Familiarity with AWS cloud watch for monitoring the pipelines and setting up alerts for workflow failures
- Cloud Security:
- Knowledge of AWS security best practices ,including IAM roles, encryption, DBT profiles access configurations Programming Skills:
- Python:
- Proficiency in Pandas and NumPy for data analysis and manipulation
- Ability to write scripts for automating ETL processes and scheduling jobs using airflow
- Experience in creating custom DBT macros using jinja and Python allowing for reusable components within dbt models
- Knowledge on how to implement conditional logic in DBT through python
- SQL:
- Advanced SQL skills, including complex joins ,window functions, CTE’s and subqueries
- Experience in optimizing SQL queries for performance and optimization Remote About the Company: Lorven Technologies Apply tot his job