AI engineering glossary · NumPy pandas visualization
What is chunked reading?
Chunked reading (also called chunksize): Loading a large file in pieces of a fixed number of rows instead of all at once, so memory use stays bounded. In pandas, read_csv with a chunksize returns an iterator of DataFrames that you process and combine one at a time.
Chunked reading: an example
for chunk in pd.read_csv('logs.csv', chunksize=100000): totals += chunk['bytes'].sum() handles a file too big for RAM.
Related terms
More NumPy pandas visualization terms
- Colormap
- Colour-blind-safe palette
- Concatenate
- Copy-on-write
- crosstab
- Data-ink ratio
- DataFrame
- DatetimeIndex
Where you learn it
Chunked reading is part of the AI Engineering Program, which teaches it with notes, a hands-on lab and an exam. The first lesson is free.