Position Overview:

We are looking for a technically proficient and experienced Big Data Development Engineer to join our core data platform team. You will be deeply involved in the design and construction of the company’s next-generation data platform, responsible for orchestrating and optimizing high-performance, scalable data processing architectures that cover the entire pipeline from data collection, cleansing, and storage to real-time processing and service delivery. This platform will directly support the company's data-driven strategy, enabling rapid business response to market changes and providing robust data support for algorithm, analytics, and product teams.

Here, you will have the opportunity to work with massive datasets, high-concurrency stream processing, and diverse technology components (such as Flink, ClickHouse, Kafka, etc.), and participate in the critical system implementation from 0 to 1. If you are passionate about data engineering and wish to challenge technical boundaries in an international, multi-team collaborative environment, we cordially invite you to join our team!

Job Responsibilities:

•       Participate in the design, development, and optimization of the company’s big data platform to support continuous business growth and data-driven decision-making.

•       Be responsible for the setup, performance tuning, monitoring, and daily operations of the ClickHouse cluster, ensuring high availability and performance of the system.

•       Participate in the development, deployment, and maintenance of Flink streaming processing tasks to ensure the stability of real-time data processing pipelines.

•       Support business data service needs by developing data API services based on Java Spring Boot.

•       Assist the data science team in building data pipelines to support algorithm model training and online services.

•       Continuously optimize data processing workflows and system architecture to enhance system stability and scalability.

•       Proactively take responsibility in projects, drive problem-solving, demonstrate ownership, and maintain excellent execution and communication under pressure.

Position Requirements:

•       Bachelor's degree or above, preferably in Computer Science, Software Engineering, Information Technology, or related fields.

•       Over 5 years of relevant experience in big data.

•       Solid foundation in databases: familiar with relational databases (such as MySQL, PostgreSQL) and columnar databases.

•       Understand the principles of columnar databases (such as ClickHouse, Doris), including underlying storage and query optimization mechanisms.

•       Strong development capabilities: proficient in Java, with experience in developing REST API services using the Spring Boot framework.

•       Excellent communication and collaboration skills, able to work closely with data analysis, product, and business teams.

•       Strong sense of responsibility, good stress resistance, problem-solving abilities, and proactive in driving work progress.

After six months, the workplace will be in Japan, requiring relocation and on-site work (the company provides visa support and related relocation assistance).

Bonus points if you also have:

1.  Practical experience with components such as Kafka, Zookeeper, HDFS, Hive, Airflow;

2.  Experience with recommendation systems, user profiling, personalized ranking, etc.;

3.  Background in predictive algorithms (such as time series forecasting, user behavior prediction);

4.  Familiarity with and practical experience in applying large language models (LLM, such as ChatGPT, Llama, etc.) in business or system scenarios (such as data augmentation, automatic summarization, intelligent Q&A);

5.  Experience participating in real-time data warehouse and indicator system construction projects;

6.  Experience in building big data platforms from 0 to 1 is preferred.

 ClickHouse 

•       Proficient in ClickHouse data modeling, query optimization, and resource management;

•       Practical experience in cluster operation and maintenance, fault diagnosis, and performance monitoring;

•       Familiar with the integration of ClickHouse with data middleware/stream processing/scheduling systems.

Flink 

•       Familiar with Flink's underlying mechanisms, such as checkpointing, state backend, and window mechanisms;

•       Able to independently write complex Flink real-time processing tasks and resolve performance issues during execution;

•       Experience in deploying and maintaining the stability of Flink clusters.