concepts
Ideas worth remembering
Data Infrastructure
The standard parts data-intensive apps are assembled from: databases, caches, search indexes, stream processors, and batch processors — glued together by application code.
Data Lake & Lakehouse
A lake is a central store of raw files in any format (Parquet, text, images, video) for data scientists that SQL can't serve. A lakehouse layers warehouse-style tables and transactions on top of lake files.
Data Warehouse
A separate database holding a read-only copy of data from all company OLTP systems, reshaped for analysis — so analysts can query freely without endangering production.
Data-intensive Applications
An app is data-intensive when managing data is the hard part -storing large volumes, handling a torrent of changes, staying consistent through failures and concurrency -not raw computation.
ETL / ELT
ETL: extract data from OLTP sources, transform it into analysis-friendly shape, load into the warehouse. ELT swaps the last two load raw first, transform inside. Generalizes into data pipelines.
HTAP
Hybrid transactional/analytical processing: one engine promising OLTP and analytics without ETL. Often secretly two engines behind one interface; earns its keep when one app genuinely needs both (e.g., fraud detection).
OLAP (Online Analytical Processing)
Analytical access pattern: few heavyweight queries that scan enormous record counts and compute aggregates (counts, sums, averages) — ad hoc, exploratory, run by analysts with raw SQL or BI tools.
OLTP (Onliine Transaction Processing)
Operational access pattern: vast numbers of small, low latency point queries (a few records by key) plus inserts/updates/deletes, with a fixed query repertoire baked into app code.
Operational vs Analytical Systems
Operational systems are where data is born — services reading and updating data as users act. Analytical systems hold a read-only copy restructured for heavy analysis. The book's most-used divide.
System of Record
The authoritative version of some data (source of truth): new facts land here first, each represented exactly once, typically normalized. If another system disagrees, this on e is right by definition.