Senior Data Engineer for Common Crawl & AWS

We’re building a next-generation talent search engine that helps companies discover top freelancers—from marketing mavens and design virtuosos to tech wizards and business strategists. To power this, we aim to build an automated pipeline that can efficiently analyze Common Crawl datasets, extract personal websites of freelancers, and return structured data containing contact details (job title, full name, email, etc). If you’re an expert in data processing, AWS data tools, and SQL with a keen understanding of web-scale data extraction, we’d love to have you on board! Your Role As our Data Engineer, you’ll provide strategic guidance to our development team, ensuring we build the most efficient, scalable, and cost-effective pipeline for extracting freelancer data. You will: - Set us up using AWS Athena for querying & filtering Common Crawl indexes to find freelancer websites. - Help us structure AWS Lambda functions for retrieving website content from WARC/WET files. - Optimize S3 storage for organizing extracted data into structured CSVs. - Share insights on automation strategies to minimize manual intervention. - Help refine queries to return only the most relevant personal websites - Help us fine-tune querying strategies for narrowing down freelancer websites from Common Crawl. - Advise on data validation and deduplication to ensure high-quality results. - Provide recommendations on cost-efficient storage, retrieval, and processing. - Share best practices for automating workflows using AWS services or alternative big data tools. What You Bring SQL & Data Processing Expertise – Strong knowledge of writing and optimizing queries for large-scale datasets. AWS Data Tools Mastery – Hands-on experience with Athena, Glue, S3, Lambda, and EC2. Common Crawl Experience – Deep understanding of WARC files, Common Crawl indexing, and large-scale web data extraction. Big Data Strategy – Experience in Apache Spark, Presto, and AWS EMR

We’d like you to complete a practical task that reflects the core challenges of this role: https://docs.google.com/document/d/1zTKenJfwvZbcwIlGEP2ZLx5ozNO4BUbOArGGdW9NAPs/edit?usp=sharing