AI Research & Web-Scale Data Platforms
Academic, web-crawl, and cloud-hosted data platforms used for AI research and large-scale model training.
Common Crawl
A free, web-scale crawl archive updated monthly, commonly used as a base corpus for language model training and web-scale research.
language model pretraining / web-scale research
AWS Open Data Registry
Public datasets hosted on cloud storage with no transfer cost when accessed from compute in the same region, useful for large-scale processing pipelines.
large-scale processing / cloud-native pipelines
Awesome Public Datasets
A community-maintained, categorized list of public dataset links spanning agriculture, finance, biology, and dozens of other fields.
broad topic discovery / niche domain datasets