AI engineering glossary · LLM training alignment
What is Common Crawl?
Common Crawl: A non-profit's public archive of web pages gathered by crawling the internet regularly. It is a major raw source for pretraining text, but needs heavy filtering for quality, duplicates and unsafe content.
Common Crawl: an example
A pipeline extracts text from Common Crawl snapshots, filters by language, and removes near-duplicates.
Related terms
More LLM training alignment terms
- Compute-optimal training
- Constitutional AI
- Continued pretraining
- Critical batch size
- Data curation
- Data mixture
- Debate (AI safety)
- DeepSpeed
Where you learn it
Common Crawl is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.