AI engineering glossary · RAG vector search
What is ingestion pipeline?
Ingestion pipeline: The offline process that prepares source material for retrieval: loading files, extracting text, cleaning it, splitting into chunks, attaching metadata, computing embeddings and writing them to an index. Its design decides what the retriever can ever find, so errors here silently hurt every later answer.
Ingestion pipeline: an example
A nightly job loads new PDFs from cloud storage, parses them, splits them into 500-token chunks, embeds them and upserts them into a vector database.
Related terms
More RAG vector search terms
- Inverted index
- IVF
- Knowledge base
- Knowledge graph
- Late interaction
- Layout-aware parsing
- Learned sparse retrieval
- LLM-as-judge
Where you learn it
Ingestion pipeline is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.