Data engineer (Azure)
$ 7,000 - $ 9,000 / month
Checking job availability...
Original
Simplified
- Bachelor’s Degree in Computer Science or Engineering with 3-5 years of experience in Azure Data engineering, Python, Pyspark or Big Data development
- Sound Knowledge of Azure Synapse analytics for pipelines, orchestration, set up
- 1-2 experience in Visualization design and development with Power BI. Knowledge on row-level security, access control
- Sound experience in SQL, Datawarehouse, data marts, data ingestion with Pyspark and Python
- Expertise in developing and maintaining ETL processing pipelines in cloud-based platforms such as AWS, Azure, etc. (Azure Synapse or data factory preferred)
- Team player with good interpersonal, communication, and problem-solving skills.
- Design, review and development of Pyspark scripts. Testing, troubleshooting of data pipelines, orchestration
- Designing and developing reports and dashboards in Power BI, setting up access control with row-level security, DAX query experience
- Establishing connections to source data systems, including Sembcorp internal systems e.g. SAP, Historians, Data Lake, etc. as well as external systems such as Web APIs, etc
- Managing the collected data in appropriate storage/data-base solutions e.g. file systems, SQL servers, Big Data platforms such as Hadoop, HANA, etc. as required by the specific project requirements
- Design, development of data marts and relevant data pipelines using pyspark, data copy activities for batch ingestion
- Deployment of pipeline artifacts from one environment to the other using Azure Devops
- Performing data integration e.g. using database table joins, or other mechanisms at an appropriate level as required by the analysis requirements of the project.
- Data catalog with Purview enabling effective metadata management, lineage tracking, and data discovery
- Candidates should demonstrate the ability to leverage Purview to ensure data governance, compliance, and efficient data exploration within Azure environments.
- Able to work independently on assignment according to agreed schedule without much supervision
- Own assignment and take initiative to resolve issues hinder completion of assignment Proactively reach out for help/guidance whenever required.
- Bachelor’s Degree in Computer Science or Engineering with 3-5 years of experience in Azure Data engineering, Python, Pyspark or Big Data development
- Sound Knowledge of Azure Synapse analytics for pipelines, orchestration, set up
- 1-2 experience in Visualization design and development with Power BI. Knowledge on row-level security, access control
- Sound experience in SQL, Datawarehouse, data marts, data ingestion with Pyspark and Python
- Expertise in developing and maintaining ETL processing pipelines in cloud-based platforms such as AWS, Azure, etc. (Azure Synapse or data factory preferred)
- Team player with good interpersonal, communication, and problem-solving skills.
- Design, review and development of Pyspark scripts. Testing, troubleshooting of data pipelines, orchestration
- Designing and developing reports and dashboards in Power BI, setting up access control with row-level security, DAX query experience
- Establishing connections to source data systems, including Sembcorp internal systems e.g. SAP, Historians, Data Lake, etc. as well as external systems such as Web APIs, etc
- Managing the collected data in appropriate storage/data-base solutions e.g. file systems, SQL servers, Big Data platforms such as Hadoop, HANA, etc. as required by the specific project requirements
- Design, development of data marts and relevant data pipelines using pyspark, data copy activities for batch ingestion
- Deployment of pipeline artifacts from one environment to the other using Azure Devops
- Performing data integration e.g. using database table joins, or other mechanisms at an appropriate level as required by the analysis requirements of the project.
- Data catalog with Purview enabling effective metadata management, lineage tracking, and data discovery
- Candidates should demonstrate the ability to leverage Purview to ensure data governance, compliance, and efficient data exploration within Azure environments.
- Able to work independently on assignment according to agreed schedule without much supervision
- Own assignment and take initiative to resolve issues hinder completion of assignment Proactively reach out for help/guidance whenever required.