Lead Data Engineer
About the job
A Software solution company is looking for a highly skilled and experienced Lead Data Engineer to join their dynamic team. The team is passionate about leveraging advanced analytics and data-driven strategies to deliver exceptional solutions to the clients.
Your new company
As a Lead Data Engineer, you will play a pivotal role in designing, building, and optimizing our data infrastructure to support the advanced analytics needs. You will lead a team of talented data engineers and collaborate closely with data analysts, the DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data solutions.
Your new role
As the Lead Data Engineer, you will be responsible for:
- Design, develop, and implement Spark Scala applications and data processing pipelines to handle large volumes of structured and unstructured data.
- Integrate Elasticsearch with Spark to enable efficient data indexing, querying, and retrieval.
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.
- Design and deploy data engineering solutions on the OpenShift Container Platform (OCP) using containerization and orchestration techniques.
- Optimize data engineering workflows for containerized deployment and efficient resource utilization.
- Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability.
- Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.
- Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure.
- Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference.
- Stay updated with emerging technologies, industry trends, and best practices in data engineering and DevOps.
- Provide technical leadership, mentorship, and guidance to junior team members, fostering a culture of continuous learning and innovation to enhance the bank's analytics capabilities.
What you need
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field
- At least 10 years of experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
- Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc.)
- Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
- Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
- Experience with Grafana, Prometheus, and Splunk is a plus
- Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services is a plus
- Develop and validate hypotheses for threat hunting
- Generate insights and provide supporting evidence through Big Data Analytics for informed decision-making
- Assemble and disseminate intelligence to peers for swift intervention actions
- Data Management to ensure that data is stored securely and efficiently, following best practices and compliance regulations
- Continuously improving data processes and methodologies to enhance the quality and efficiency of data analysis
What you'll get in return
In return, you'll be part of an organization that values its employees. You’ll be rewarded with:
- structured career growth and plenty of developmental opportunities
- Hybrid and stable working environment
What you need to do now
If you're interested in this role, click 'apply now' or for more information and a confidential discussion on this role or to find out about more opportunities in Technology contact Yuki Cheung or email [email protected]. Referrals are welcome.
At Hays, we value diversity and are passionate about placing people in a role where they can flourish and succeed. We actively encourage people from diverse background to apply.
EA Reg Number: R22110258 | EA License Number: 07C3924 | Company Registration No: 200609504D
About the job
A Software solution company is looking for a highly skilled and experienced Lead Data Engineer to join their dynamic team. The team is passionate about leveraging advanced analytics and data-driven strategies to deliver exceptional solutions to the clients.
Your new company
As a Lead Data Engineer, you will play a pivotal role in designing, building, and optimizing our data infrastructure to support the advanced analytics needs. You will lead a team of talented data engineers and collaborate closely with data analysts, the DevOps team, infrastructure engineers, and other stakeholders to deliver high-quality data solutions.
Your new role
As the Lead Data Engineer, you will be responsible for:
- Design, develop, and implement Spark Scala applications and data processing pipelines to handle large volumes of structured and unstructured data.
- Integrate Elasticsearch with Spark to enable efficient data indexing, querying, and retrieval.
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch.
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions.
- Design and deploy data engineering solutions on the OpenShift Container Platform (OCP) using containerization and orchestration techniques.
- Optimize data engineering workflows for containerized deployment and efficient resource utilization.
- Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability.
- Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements.
- Implement monitoring and logging mechanisms to ensure the health, availability, and performance of the data infrastructure.
- Document data engineering processes, workflows, and infrastructure configurations for knowledge sharing and reference.
- Stay updated with emerging technologies, industry trends, and best practices in data engineering and DevOps.
- Provide technical leadership, mentorship, and guidance to junior team members, fostering a culture of continuous learning and innovation to enhance the bank's analytics capabilities.
What you need
- Bachelor's degree in Computer Science, Data Engineering, Information Technology, or a related field
- At least 10 years of experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
- Strong expertise in designing and developing data infrastructure using Hadoop, Spark, and related tools (HDFS, Hive, Ranger, etc.)
- Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
- Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
- Experience with Grafana, Prometheus, and Splunk is a plus
- Experience with cloud platforms (e.g., AWS, Azure, GCP) and their data services is a plus
- Develop and validate hypotheses for threat hunting
- Generate insights and provide supporting evidence through Big Data Analytics for informed decision-making
- Assemble and disseminate intelligence to peers for swift intervention actions
- Data Management to ensure that data is stored securely and efficiently, following best practices and compliance regulations
- Continuously improving data processes and methodologies to enhance the quality and efficiency of data analysis
What you'll get in return
In return, you'll be part of an organization that values its employees. You’ll be rewarded with:
- structured career growth and plenty of developmental opportunities
- Hybrid and stable working environment
What you need to do now
If you're interested in this role, click 'apply now' or for more information and a confidential discussion on this role or to find out about more opportunities in Technology contact Yuki Cheung or email [email protected]. Referrals are welcome.
At Hays, we value diversity and are passionate about placing people in a role where they can flourish and succeed. We actively encourage people from diverse background to apply.
EA Reg Number: R22110258 | EA License Number: 07C3924 | Company Registration No: 200609504D