AI engineering glossary · NLP embeddings
What is chunking?
Chunking (also called Shallow parsing): Grouping consecutive words into flat phrases such as noun phrases or verb phrases, without building a full sentence tree. It sits between part-of-speech tagging and full parsing and is often done with BIO tags.
Chunking: an example
"The old bridge" is grouped as one noun phrase (NP) in "The old bridge collapsed".
Chunking in Python
A short example you can run (Python 3.11). The output below is what it really prints.
def chunk(text, size=40, overlap=10):
out, start = [], 0
while start < len(text):
out.append(text[start:start + size])
start += size - overlap
return out
doc = "Refunds are allowed within 7 days. Certificates have a public verification ID."
for i, c in enumerate(chunk(doc)):
print(i, repr(c))Output
0 'Refunds are allowed within 7 days. Certi' 1 'ays. Certificates have a public verifica' 2 'c verification ID.'
Related terms
More NLP embeddings terms
- Co-occurrence matrix
- Code-mixing
- Collocation
- Conditional random field
- CoNLL-2003
- Constituency parsing
- Contextual embedding
- Continuous bag of words
Where you learn it
Chunking is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.