DataOps/Cloud Data Engineer – Senior

DataOps/Cloud Data Engineer – Senior
Client: Community Services Cluster, Ontario Public Sector
Ministry: Ministry of Public and Business Service Delivery and Procurement
Location: 777 Bay Street, Toronto, ON
Work Arrangement: 100% Onsite – 5 days per week
Contract: September 1, 2026 – September 1, 2027
Security Clearance: No clearance required

Position Overview
We are seeking a Senior DataOps/Cloud Data Engineer to design, develop, optimize, and support modern cloud data solutions within a large-scale public-sector environment. The successful candidate will have strong hands-on expertise with Azure Data Factory, Databricks, Informatica, Python, SQL, and Medallion Architecture, along with proven experience building enterprise-scale data pipelines, lakehouses, data models, and cloud data integrations.
The consultant will play a key role in modernizing existing ETL processes, including the migration of Informatica ETL to Azure Data Factory and Databricks ELT, while ensuring data quality, security, governance, performance, and operational reliability.
Key Responsibilities

• Design, develop, optimize, and support Azure Data Factory and Databricks pipelines integrating Oracle databases with cloud Lakehouse environments.
  
• Develop and optimize Databricks solutions using Medallion Architecture, including Bronze, Silver, and Gold layers.
  
• Design and optimize data connections between Databricks/Lakehouse environments and on-premises data sources for downstream consumers.
  
• Design and implement scalable relational data models, dimensional models, fact/dimension models, star schemas, data warehouses, data lakes, and data lakehouses.
  
• Translate and modernize existing Informatica ETL processes into Azure Data Factory and Databricks ELT.
  
• Develop data pipelines and workflows for orchestration, deployment, automation, parallel processing, data movement, and Delta Lake management/indexing.
  
• Work extensively with Python, SQL, T-SQL, PL/SQL, Informatica, Azure Data Factory, SSIS, Databricks, and Microsoft Fabric.
  
• Implement CI/CD and automated data provisioning processes using Azure DevOps and related technologies.
  
• Develop and manage cloud data services, including Data Lakehouse, Delta Lake, Azure Storage Accounts, Key Vault, virtual machines, repositories, Parquet files, and related Azure services.
  
• Work with structured, semi-structured, and unstructured data across enterprise data warehouse, data lake, and lakehouse environments.
  
• Design and implement high-volume data migration and transformation solutions across OLTP and OLAP environments and cloud SaaS, PaaS, and IaaS platforms.
  
• Implement data quality processes covering data validation, profiling, cleansing, monitoring, and reconciliation.
  
• Apply data governance and information architecture standards and develop conceptual, logical, and physical data models.
  
• Implement secure data access using Microsoft Entra ID, including authentication, authorization, and role-based access control.
  
• Implement data anonymization and masking techniques for sensitive and regulated information.
  
• Develop and maintain data lineage to provide end-to-end visibility into data movement and transformations.
  
• Perform DataOps performance monitoring, troubleshooting, tuning, and production support.
  
• Investigate defects and production issues and develop and deploy appropriate fixes.
  
• Participate in system, unit, SIT, SAT, and UAT testing.
  
• Prepare and maintain technical, system, business, design, support, release, and training documentation.
  
• Collaborate with business stakeholders, project managers, architects, developers, data teams, and other technical teams.
  
• Provide technical leadership, guidance, and solution design expertise across multiple teams and workstreams.
  
• Support Agile and Waterfall delivery methodologies and established SDLC processes.
  

Requirements

Mandatory Technical Skills
Candidates must demonstrate strong hands-on experience in the following:

• Informatica ETL
  
• Azure Data Factory (ADF)
  
• Azure Databricks
  
• Databricks Medallion Architecture
  
• Python
  
• SQL
  
• T-SQL
  
• PL/SQL
  
• SSIS
  
• Microsoft Fabric
  
• Lakeflow
  
• SQL Optimization
  
• Delta Lake
  
• Data pipeline development, workflow orchestration, deployment, and automation
  
• Dataflow, parallelism, data movement, and Delta Lake indexing
  
• Cloud data engineering and cloud data platforms
  
• Data Lake, Data Warehouse, and Lakehouse solutions
  
• Relational and dimensional data modelling
  
• Fact/dimension modelling and star schema
  
• Enterprise data warehouse and data lake/lakehouse development
  
• Cloud and on-premises data integration
  
• CI/CD and automated data provisioning
  
• Azure DevOps
  
• Data quality, profiling, cleansing, validation, and monitoring
  
• Data governance and data lineage
  
• Microsoft Entra ID and RBAC
  
• Data anonymization and masking
  
• High-volume/VLD data migration and transformation
  
• OLTP and OLAP environments
  

Core Qualifications

• Strong analytical, problem-solving, decision-making, communication, presentation, and interpersonal skills.
  
• Proven experience designing and developing complex data and business intelligence solutions.
  
• Strong DataOps best practices and Agile development/deployment experience.
  
• Experience working with large and diverse structured and unstructured datasets.
  
• Experience with data analysis, data profiling, KPI development, and data visualization.
  
• Strong production troubleshooting and issue-resolution experience.
  
• Experience coordinating modernization of large, complex, multi-platform and multi-tier technology environments.
  
• Experience creating detailed business, system, technical, and solution design documentation.
  
• Experience with AODA/WCAG compliance.
  
• Demonstrated ability to work collaboratively across multiple teams and meet strict deadlines.
  

Project & Delivery Experience

• Strong experience with SDLC, Agile, and Waterfall methodologies.
  
• Experience conducting unit testing, system testing, SIT, SAT, and UAT.
  
• Experience tracking, reporting, and resolving project issues and risks.
  
• Experience gathering and consolidating business and technical requirements.
  
• Experience leading requirements-gathering sessions and supporting change requests and project artifacts.
  
• Experience coordinating multiple projects and competing priorities.
  
• Ability to provide technical leadership and guidance to development and business teams.
  
• Ability to ensure business requirements are accurately reflected in solution designs and technical specifications.
  

Leadership & Collaboration
The successful candidate will:

• Lead and coordinate technical teams and workstreams.
  
• Provide technical advice and mentorship to team members.
  
• Work effectively with business users, project managers, architects, developers, and IT teams.
  
• Facilitate solution integration across multiple teams and platforms.
  
• Communicate technical concepts clearly to both technical and non-technical stakeholders.
  
• Demonstrate strong ownership, accountability, and delivery focus.
  

Public Sector Experience

• Previous experience working in a public-sector organization of comparable size is strongly preferred.
  
• Experience delivering enterprise-scale data modernization, cloud migration, or data integration initiatives within government or large regulated environments is an asset.
  

Certifications
Databricks and/or Microsoft Fabric certifications are considered an asset.
Knowledge Transfer Requirement
The consultant will be responsible for transferring knowledge to the designated Community Services Cluster resource.
Knowledge transfer must be completed one week before the project end date or one week before the consultant leaves the Ministry, whichever applies.
Knowledge transfer will include:

• All deliverables and supporting documentation.
  
• Design, support, release, and training documentation.
  
• Project requirements and solution design documentation.
  
• Documentation stored in designated repositories such as Git, SharePoint, HPQC, TFS, or other approved project repositories.
  
• Written communication to the Project Manager and designated Ministry staff identifying where documentation has been stored.
  
• At least one walkthrough of the project documentation.
  
• One-on-one knowledge-transfer sessions, email communication, document updates, and documentation reviews.
  

Ideal Candidate Profile
The ideal candidate is a senior hands-on Cloud Data Engineer/DataOps professional who combines deep expertise in Informatica, Azure Data Factory, Databricks, Python, SQL, and Medallion Architecture with strong enterprise data modelling, cloud migration, DataOps, CI/CD, data governance, and production support experience.
Important: This is a 100% onsite position requiring the consultant to work from 777 Bay Street, Toronto, five days per week.