Azure Data Engineer

As an Azure Data Engineer, you will:

• Design, develop, and maintain complex and scalable data pipelines using Azure Data Factory (ADF) and Azure Databricks.
  
• Build and optimize large-scale data processing solutions using PySpark and Databricks.
  
• Develop robust ETL/ELT solutions in Python for data extraction, transformation, and loading.
  
• Collaborate with cross-functional teams to understand data requirements and deliver efficient data engineering solutions.
  
• Implement and maintain CI/CD pipelines using Azure DevOps for automated testing, integration, and deployment.
  
• Develop, test, and maintain reliable data solutions aligned with technical and business requirements.
  
• Monitor, troubleshoot, and optimize data pipelines to ensure performance, scalability, and reliability.
  
• Work with large datasets and use SQL for data querying and data management.
  
• Ensure data solutions follow applicable security, compliance, and engineering best practices.
  
• Stay updated with the latest trends, technologies, and best practices in Azure cloud data engineering.
  

What You Bring to the Table:

• 8–10 years of overall professional experience in data engineering, cloud data engineering, or related technology roles.
  
• Strong hands-on experience with Azure Data Factory for data orchestration, movement, and transformation.
  
• Strong experience with Azure Databricks for developing and managing scalable data processing solutions.
  
• Strong programming skills in Python.
  
• Hands-on experience with PySpark for large-scale data processing.
  
• Experience implementing CI/CD pipelines using Azure DevOps.
  
• Good knowledge of SQL and experience working with large datasets.
  
• Experience developing and maintaining ETL/ELT data pipelines.
  
• Understanding of data security, compliance, and cloud data engineering best practices.
  
• Strong problem-solving and analytical skills.
  
• Experience working independently as well as collaboratively within cross-functional teams.
  

You Should Possess the Ability To:

• Design and develop scalable data pipelines using Azure Data Factory and Azure Databricks.
  
• Develop efficient PySpark applications for processing and transforming large volumes of data.
  
• Write clean, maintainable, and reusable Python-based data engineering solutions.
  
• Build and manage automated CI/CD workflows in Azure DevOps.
  
• Perform data extraction, transformation, loading, validation, and integration across different data sources.
  
• Optimize data pipelines and Databricks workloads for performance and scalability.
  
• Monitor pipelines, identify failures or bottlenecks, and implement appropriate solutions.
  
• Troubleshoot data processing and pipeline-related issues effectively.
  
• Write SQL queries for data analysis, transformation, and validation.
  
• Work with stakeholders to understand requirements and translate them into technical data solutions.
  
• Follow appropriate data security, governance, compliance, and development standards.
  
• Document technical solutions, data processes, pipelines, and deployment procedures.
  
• Adapt to new Azure services, cloud data engineering technologies, and industry best practices.
  

What We Bring to the Table:

• An opportunity to work on enterprise-scale Azure cloud data engineering projects.
  
• Exposure to Azure Data Factory, Azure Databricks, PySpark, Python, Azure DevOps, and SQL.
  
• Opportunities to design and optimize large-scale data pipelines and processing frameworks.
  
• A collaborative environment involving cross-functional technical and business teams.
  
• Opportunities to contribute to cloud data modernization, automation, and CI/CD initiatives.
  

Let’s Connect

Want to discuss this opportunity in more detail? Feel free to reach out.

Recruiter: Hema Murali
Phone: +31 20 369 0609 ; Extn :148
Email: hema.m@stafide.nl
LinkedIn: https://www.linkedin.com/in/hema-murali-315999329/