Data Engineering Manager

Summary

We are seeking a seasoned Data Engineering Manager to lead our data infrastructure initiatives within the IT Services sector. This pivotal role involves overseeing the design, implementation, and optimization of scalable data pipelines and architectures. The successful candidate will drive strategic decisions regarding data lake frameworks, real-time streaming solutions, and cloud-native deployment strategies, ensuring robust data availability and integrity to support enterprise-wide analytics and business intelligence goals.

Responsibilities

• Architect and govern large-scale data lake environments, ensuring high performance and reliability for both batch and real-time data processing workloads.
  
• Lead the development and maintenance of streaming data solutions using Apache Kafka and MSK, integrating change data capture (CDC) mechanisms via Debezium.
  
• Manage containerized data services and orchestration platforms, specifically utilizing ECS, EKS, and Docker to ensure seamless deployment and scaling.
  
• Oversee infrastructure-as-code practices using Terraform and CDK to automate provisioning and manage cloud resources efficiently.
  
• Direct the implementation of Avro serialization standards and optimize data ingestion patterns for complex analytical requirements.
  
• Collaborate with cross-functional teams to define data engineering roadmaps, resolve technical bottlenecks, and enforce best practices in data governance.
  
• Mentor senior engineers and foster a culture of technical excellence, code quality, and continuous improvement within the data engineering team.
  

Requirements

Requirements:

• Possess 10 to 15 years of professional experience in data engineering, with a proven track record of managing complex data ecosystems.
  
• Demonstrate deep expertise in streaming and batch processing architectures, with hands-on experience in Apache Kafka and MSK.
  
• Proficiency in containerization and orchestration tools, including Docker, ECS, and EKS.
  
• Strong command of infrastructure automation using Terraform or AWS CDK.
  
• Extensive background in designing and maintaining Data Lake architectures, with specific knowledge of Avro and Debezium CDC.
  
• Familiarity with real-time analytics engines such as Apache Pinot is highly desirable.
  
• Ability to operate within the IT Services industry standards, delivering high-quality solutions that meet client and business needs.