Data Engineering & Web Crawling

Job Description:

About xAI

xAI’s mission is to create AI systems that can accurately understand the universe and aid humanity in its pursuit of knowledge.

Our team is small, highly motivated, and focused on engineering excellence. This organization is for individuals who appreciate challenging themselves and thrive on curiosity. Engineers are encouraged to work across multiple areas of the company, and as a result, all engineers and researchers share the title "Member of Technical Staff."

We operate with a flat organizational structure. All employees are expected to be hands-on and to contribute directly to the company’s mission. Leadership is given to those who show initiative and consistently deliver excellence. Work ethic and strong prioritization skills are important.

All engineers and researchers are expected to have strong communication skills. They should be able to concisely and accurately share knowledge with their teammates.

xAI does not have recruiters. Every application is reviewed directly by a technical member of the team.

Tech Stack

Rust
Python
Spark

Focus

Building petabyte-scale, high-throughput data processing systems.
Managing workloads across large cloud compute clusters.
Pre-processing datasets for AI training.

Ideal Experience

Building bespoke data processing libraries from scratch.
Designing and implementing distributed systems in Rust.
Keeping up with state-of-the-art techniques for preparing AI training data.
Organizing and meticulously bookkeeping data across multiple clouds, of multiple modalities, and from many sources.