Data platform engineer for AI research

Описание

Join a highly technical AI research and engineering organisation building the data infrastructure behind next-generation AI models. As a Senior Data Platform Engineer, you’ll design and operate distributed data-processing systems capable of handling petabyte-scale workloads, ensuring research teams can access reliable, reproducible datasets and run complex pipelines efficiently. This role combines distributed systems engineering, workflow orchestration and performance optimisation, with a focus on scalability, fault tolerance and cost efficiency across large-scale data and machine learning infrastructure.

Задачи

• Design, build and operate distributed data-processing infrastructure for petabyte-scale workloads.
• Develop and optimise data pipelines, execution frameworks and workflow orchestration across diverse computational workloads, including GPU-accelerated stages.
• Improve system performance, throughput and resource utilisation while managing infrastructure costs.
• Build reliable, observable pipelines with robust failure recovery, restartability and monitoring.
• Implement dataset versioning, lineage and reproducibility to ensure research datasets can be reliably traced and recreated.
• Integrate data infrastructure with large-scale ML training workflows and establish engineering standards and best practices.

Требования

• PhD in Computer Science, Computer Engineering, Software Engineering or a related discipline.
• At least 3 years of relevant full-time experience building or operating distributed data-processing systems or ML data infrastructure.
• Strong experience with distributed processing frameworks such as Ray, Apache Spark or Flink, alongside workflow orchestration and large-scale pipeline execution.
• Solid understanding of distributed computing, performance profiling, throughput optimisation and fault tolerance.
• Experience with high-performance data formats such as Apache Arrow or Parquet, and dataset versioning, lineage or reproducibility.
• Familiarity with workload orchestration and scheduling tools such as Kubernetes or Slurm, and containerisation technologies such as Docker or Enroot.
• Strong software engineering skills and experience owning production systems where reliability, performance and cost matter.

Условия

Help build the infrastructure that enables cutting-edge AI research at scale. You’ll tackle challenging distributed-systems problems, influence core architectural decisions and work closely with researchers and engineers to make large-scale data and ML workflows more efficient, reliable and reproducible.