Checking job availability...
Original
Simplified
ABOUT US
BIT is a full-featured cryptocurrency exchange that is designed for use by professionals, with advanced risk management and fund efficiency supported by Portfolio Margin and Unified Margin.
BIT was launched in August 2020 as a spinoff from Matrixport, a prominent crypto financial service company. BIT is one of the top crypto options trading platforms which offers innovative USD-margined trading pairs.
BIT is always striving to widen its selection of tokens, enhance its trading tools, and inspire the creation of innovative financial products.
ABOUT THE ROLE
Data developer Engineer
Job Responsibilities:
- ETL & Data Pipeline Development:
- Design, develop, and optimize ETL processes to synchronize data from MySQL and other data sources to ByteHouse.
- Responsible for data collection, cleansing, transformation, and loading (ETL) to improve data quality and consistency.
- Maintain and optimize ETL jobs to ensure stability and efficiency in data processing.
- Design and implement data pipeline monitoring, task scheduling, and error handling optimizations.
- Data Lake/ Warehouse Management & Performance Optimization:
- Manage storage, table design, and query optimization for ByteHouse (similar to ClickHouse).
- Design data layer architectures and build efficient data warehouse models.
- Implement data governance, including data lineage, data quality management, and schema evolution.
- Research and optimize cold and hot data storage layers to improve query performance.
- Data Integration & Data Governance:
- Responsible for real-time/offline data integration (Kafka, Canal, Flink) to improve data timeliness.
- Ensure data consistency, handling issues like data deduplication, anomaly detection, and primary key conflicts.
- Design data access control mechanisms to ensure compliance and security of data access.
Job Requirements:
Essential Skills:
- Proficient in SQL, with experience in MySQL, ByteHouse / ClickHouse.
- Familiar with ETL development, with experience in Airflow / AWS Glue / Flink / Spark.
- Familiar with data lake/data warehouse architecture design.
- Knowledge of Kafka/Canal and data flow mechanisms.
- Experience with Kubernetes (K8s) and AWS.
Preferred Skills:
- Knowledge of data governance (data quality, lineage analysis, metadata management).
- Experience with distributed storage optimization, such as S3 / HDFS + ByteHouse.
- Proficient in Python / Java / Go development, with the ability to develop ETL automation tools.
- Experience in optimizing high-concurrency queries for ClickHouse/ByteHouse.