Senior AWS Data Engineer / Lead
Designation: Senior AWS Data Engineer / Lead
Location: Bangalore
Experience: 8 - 10 years
Opening: 08
Join us as a Senior AWS Data Engineer / Lead and spearhead our data initiatives and leverage your expertise in the AWS cloud to design, build, and maintain data-driven solutions.
Job Description:
- Orchestration of Amazon MWAA: Expertise in managing Amazon Managed Workflows for Apache Airflow (MWAA) for data orchestration, including ingestion, transformation, and data integrity tasks.
- Transition to ELT: Implementing ELT (Extract, Load, Transform) approach aligned with Gartner’s data transformation strategy, focusing on metadata-driven integration design and rapid onboarding of new systems.
- DataOps Delivery: Adopting a DataOps approach with Informatica, pySpark, Python shell, and SQL programming skills to build data pipelines, along with CI/CD pipelines using GitHub Actions and Infrastructure as Code (IaC) with Terraform.
- Data Lake House Architecture: Utilizing AWS services like S3 and Amazon Redshift to create a true Data Lake house architecture with native Role-Based Access Control (RBAC) and security controls, integrating Snowflake for enhanced RBAC and observability using AWS CloudWatch and Apache Airflow.
- Event-Driven Architecture: Implementing event-based architecture with AWS EventBridge to monitor and audit AWS environments, enabling real-time response to operational changes and preventing infrastructure vulnerabilities.
Skills and Experience Required:
- Bachelor’s or Master’s degree in Computer Science or related fields with 8-10 years of professional experience.
- Proficiency in managing Amazon MWAA for Apache Airflow orchestration.
- Experience transitioning from ETL to ELT approach with a focus on metadata-driven integration.
- Strong programming skills in pySpark, Python shell, and SQL for building data pipelines.
- Hands-on experience with Informatica, GitHub Actions for CI/CD pipelines, and Terraform for Infrastructure as Code.
- Expertise in implementing Data Lake house architecture using AWS services like S3, Amazon Redshift, and Snowflake.
- Familiarity with Role-Based Access Control (RBAC), observability using AWS CloudWatch and Apache Airflow, and event-based architecture with AWS EventBridge.
