Data Engineer

Designation – Data Engineer

Location – Pune, Maharashtra, India

Experience: 5–8 Years (Minimum 5 Years Relevant Experience)

Qualification: B.E./B.Tech, M.E./M.Tech, MCA, or equivalent in Computer Science/IT or a related field.

Work Mode: Hybrid

Primary Skill Set: Databricks, Python, Spark, SQL

Job Description – Data Engineer

Data Pipeline Development & Operations

• Design, build, and operate scalable and reliable data pipelines on the Databricks platform.
  
• Develop end-to-end data workflows from ingestion through transformation to consumption.
  
• Implement error handling, monitoring, and alerting mechanisms.
  
• Ensure pipeline reliability, performance, and maintainability.
  
• Optimize Spark jobs and cluster configurations for improved performance.
  
• Manage complex workflows using Databricks Jobs and Workflows.
  

Legacy Code Modernization

• Refactor legacy code and ETL pipelines to PySpark.
  
• Migrate traditional ETL processes to modern ELT patterns on Databricks.
  
• Identify opportunities for code and pipeline optimization.
  
• Ensure data integrity and backward compatibility during migrations.
  
• Document modernization approaches and migration playbooks.
  

Data Engineering Excellence

• Implement data quality checks and validation frameworks.
  
• Design and maintain Delta Lake tables with appropriate optimization strategies.
  
• Develop reusable data engineering frameworks and libraries.
  
• Follow best practices for version control, testing and CI/CD.
  
• Participate in code reviews and troubleshoot production issues.
  

Collaboration & Knowledge Sharing

• Work closely with data architects, analysts and business stakeholders.
  
• Collaborate with Infrastructure, Applications and Cyber teams.
  
• Share technical knowledge and best practices.
  
• Mentor junior Data Engineers on PySpark and Databricks.
  
• Maintain comprehensive technical documentation.
  

Essential Technical Skills

• Strong Data Engineering, ETL/ELT and data pipeline design knowledge.
  
• Hands-on PySpark, including DataFrames, Spark SQL and performance optimization.
  
• Practical experience with Databricks Workspace, Cluster Management, Notebooks and Job Orchestration.
  
• Knowledge of Databricks Workspace AI Agent capabilities and integration.
  
• Data Modelling experience – Dimensional Modeling, Data Vault or Lakehouse Architecture.
  
• Strong understanding of Delta Lake, including ACID Transactions, Schema Evolution and optimization.
  
• Strong Python skills for data processing and automation.
  

Additional Technical Skills

• Strong SQL proficiency.
  
• Cloud experience – Azure, AWS or GCP.
  
• Data Governance and Security understanding.
  
• Knowledge of Structured Streaming.
  
• DevOps and CI/CD practices.
  
• Version control using Git.
  
• Data Quality and Testing methodologies.
  

Professional Experience

• Minimum 5 years relevant Data Engineering experience.
  
• At least 2–3 years hands-on Databricks experience.
  
• Proven experience in legacy ETL/code modernization to modern frameworks.
  
• Experience building and maintaining production-scale data pipelines.
  
• Experience working with multiple data sources and formats.
  
• Experience in Agile development environments.
  

Required Certification – Mandatory

At least one of the following:

• Databricks Certified Data Engineer Associate
  
• Databricks Certified Data Engineer Professional
  

Preferred Certifications

• Databricks Certified Associate Developer for Apache Spark.
  
• Azure Data Engineer Associate.
  
• AWS Certified Data Analytics.
  
• Google Cloud Professional Data Engineer.
  
• Other relevant Data Engineering / Big Data certifications.
  

Soft Skills

• Strong problem-solving and analytical skills.
  
• Excellent communication and technical explanation skills.
  
• Strong collaboration and stakeholder management.
  
• Self-motivated with attention to detail.
  
• Adaptable to changing priorities and technologies.
  
• Client-focused approach with commitment to quality delivery.