Senior Data Engineer (Databricks + PySpark + ADF + Unity Catalog)

Overview

We are seeking a highly skilled Senior Data Engineer with hands-on experience in Databricks, Unity Catalog, and Azure Data Factory (ADF) to support a large-scale enterprise data modernization initiative. The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines, enabling secure data governance, and supporting cloud-based analytics solutions in Azure.

This role will collaborate closely with architects, business stakeholders, and cross-functional engineering teams to build reliable, high-performance data platforms and accelerate data-driven decision-making.

Responsibilities

Your Impact

• Design, develop, and optimize ETL/ELT pipelines using Azure Data Factory and Databricks.
• Build and maintain scalable data ingestion frameworks for structured and unstructured data sources.
• Develop data transformation processes using PySpark, Spark SQL, and Databricks notebooks.
• Implement and manage Unity Catalog for enterprise data governance, security, and access controls.
• Create and optimize Delta Lake architectures for analytics and reporting workloads.
• Monitor pipeline performance and troubleshoot production issues.
• Collaborate with data architects to implement enterprise data models and best practices.
• Ensure data quality, integrity, security, and compliance standards are met.
• Support CI/CD deployment processes and DevOps practices for data engineering solutions.
• Work closely with business and analytics teams to understand requirements and deliver scalable solutions.

Requirements

Your Skills & Experience

• 4-7+ years of experience in Data Engineering or Big Data development.
• Strong hands-on experience with Azure Databricks.
• Proficiency in PySpark, Spark SQL, and distributed data processing.
• Experience implementing and managing Unity Catalog.
• Strong experience with Azure Data Factory (ADF).
• Experience working with Delta Lake and modern Lakehouse architectures.
• Solid understanding of data modeling, ETL/ELT, and data warehousing concepts.
• Experience with Azure cloud services and ecosystem.
• Proficiency with Git, version control, and CI/CD processes.
• Strong analytical and problem-solving skills.
• Excellent communication and stakeholder management abilities.

Preferred Qualifications

• Experience with Azure Synapse Analytics.
• Familiarity with Collibra or other data governance platforms.
• Experience with Infrastructure as Code (Terraform, ARM Templates).
• Databricks Certification (Data Engineer Associate/Professional).
• Microsoft Azure Data Engineer Associate Certification.

Nice to Have

• Experience supporting enterprise-scale data migration projects.
• Knowledge of healthcare, retail, or consumer products data environments.
• Exposure to real-time data processing and streaming technologies.