We are seeking a skilled Data Engineer to help build and enhance our modern Lakehouse architecture. This role is responsible for pipeline development, data modelling, governance, and automation, working within a cross-functional team that delivers end-to-end data products and analytics solutions. You will also play a key role in CI/CD automation, monitoring, and data quality enforcement, ensuring data consistency, reliability, and compliance across all workflows.
Key Responsibilities:
- Develop and maintain data pipelines for efficient data ingestion and transformation using Azure Databricks and Spark.
- Collaborate with Product Owners, Business Analysts, and Data Analysts to translate business needs into scalable data solutions.
- Ensure data quality and governance, implementing validation rules to maintain consistency and compliance.
- Work closely with System Analysts and Business Teams to understand data lifecycles and optimize solution designs for scalability and performance.
- Automate CI/CD pipelines for seamless deployment, testing, and monitoring of data models and pipelines.
- Implement Infrastructure as Code (IaC) to manage cloud resources efficiently using Terraform.
- Support Agile methodologies, contributing to continuous improvement and rapid iterations in a fast-paced environment.
Required Skills & Qualifications:
- 5+ years in Data Engineering, designing and managing modern data architectures.
- Proficiency in Spark & Azure Databricks for designing and optimizing data pipelines.
- Advanced SQL and Python skills for complex data queries and modelling, CI/CD pipeline development using GitHub, GitHub Actions, and Databricks.
- Experience with Infrastructure as Code (IaC) using Terraform for cloud automation.
- Experience in Agile environments, demonstrating adaptability and a proactive approach to change.
To apply please click the Apply button or send us your updated profile to [email protected]
EA Licence No.:18S9405 /EA Rgn No:R1436009
Percept Solutions is undergoing a growth phase and are on the lookout for talent. Applicants are encouraged to follow Percept Solutions on LinkedIn @ https://www.linkedin.com/company/percept-solutions/ to stay up to date on our upcoming roles and events
We are seeking a skilled Data Engineer to help build and enhance our modern Lakehouse architecture. This role is responsible for pipeline development, data modelling, governance, and automation, working within a cross-functional team that delivers end-to-end data products and analytics solutions. You will also play a key role in CI/CD automation, monitoring, and data quality enforcement, ensuring data consistency, reliability, and compliance across all workflows.
Key Responsibilities:
- Develop and maintain data pipelines for efficient data ingestion and transformation using Azure Databricks and Spark.
- Collaborate with Product Owners, Business Analysts, and Data Analysts to translate business needs into scalable data solutions.
- Ensure data quality and governance, implementing validation rules to maintain consistency and compliance.
- Work closely with System Analysts and Business Teams to understand data lifecycles and optimize solution designs for scalability and performance.
- Automate CI/CD pipelines for seamless deployment, testing, and monitoring of data models and pipelines.
- Implement Infrastructure as Code (IaC) to manage cloud resources efficiently using Terraform.
- Support Agile methodologies, contributing to continuous improvement and rapid iterations in a fast-paced environment.
Required Skills & Qualifications:
- 5+ years in Data Engineering, designing and managing modern data architectures.
- Proficiency in Spark & Azure Databricks for designing and optimizing data pipelines.
- Advanced SQL and Python skills for complex data queries and modelling, CI/CD pipeline development using GitHub, GitHub Actions, and Databricks.
- Experience with Infrastructure as Code (IaC) using Terraform for cloud automation.
- Experience in Agile environments, demonstrating adaptability and a proactive approach to change.
To apply please click the Apply button or send us your updated profile to [email protected]
EA Licence No.:18S9405 /EA Rgn No:R1436009
Percept Solutions is undergoing a growth phase and are on the lookout for talent. Applicants are encouraged to follow Percept Solutions on LinkedIn @ https://www.linkedin.com/company/percept-solutions/ to stay up to date on our upcoming roles and events