Data Engineering4 min read
Apache ParquetColumnar StorageParquet MetadataDremel RecordBig Data

Deep Dive into Apache Parquet

Apache Parquet is an open-source, column-oriented binary storage format designed for fast, space-efficient analytical query processing (OLAP). Originally inspired by Google's Dremel paper, Parquet is native to modern lakehouse execution engines (such as Apache Spark, Trino, Apache Arrow, DuckDB, and ClickHouse) due to its built-in support for column projection, predicate pushdown, and vectorized SIMD processing.

In this chapter, we explore Parquet's physical layout hierarchy, footer-first reading architecture, nested record shredding, page encoding/compression pipeline, query pruning indexes, and vectorization optimizations.

1. Physical Layout Hierarchy

Parquet organizes data hierarchically into horizontal partitions (Row Groups) and vertical slices (Column Chunks), down to individual physical pages.

Header Offset 0 • 4-byte Magic Number: 'PAR1' (File identifier) Row Group 1 (~128MB - 1GB) Horizontal Partition • Column Chunk A (Dictionary + Data Pages)    • Column Chunk B (Pages)    • Page Indexes Row Group 2 (~128MB - 1GB) Horizontal Partition • Column Chunk A (Dictionary + Data Pages)    • Column Chunk B (Pages)    • Page Indexes File Footer (Thrift FileMetaData) Metadata Directory • Schema Tree    • Row Group Offsets & Sizes    • Column Statistics (Min, Max, Null Count) File Trailer EOF — Read First • 4-byte Footer Length (L)    • 4-byte Magic: 'PAR1' (Reader seeks EOF - 4 bytes to begin)

Hierarchy Breakdown

  1. Row Group: A logical horizontal partition of data containing a fixed number of rows (typically 128 MB to 1 GB in size).
  2. Column Chunk: The contiguous physical storage of a single column's data within a specific Row Group.
  3. Page: The atomic unit of storage, compression, and encoding within a Column Chunk (typically 64 KB to 1 MB in size). Pages are read and decompressed as a single block.

Data Page (V1/V2): Contains actual encoded values along with definition and repetition levels. Dictionary Page: Stores unique dictionary values when dictionary encoding is active.

2. Footer-First Reading & Thrift Metadata

Unlike row-based formats (like CSV or Avro) that are parsed sequentially top-to-bottom, Parquet files are read bottom-up:

Step 1: Seek to EOF - 4 bytes ---> Verify "PAR1" magic bytes
Step 2: Read preceding 4 bytes --> Extract Footer Length (L)
Step 3: Seek back (L + 4) bytes -> Deserialize Thrift FileMetaData into memory

Thrift FileMetaData Contents

The footer contains all schema and structural metadata needed to plan I/O before fetching actual data pages: Schema: Column data types, logical types (e.g., TIMESTAMP(NANOS)), and nested structures. Row Group Metadata: Total byte size and row count. File offsets for every Column Chunk. Column Statistics: min_value, max_value, null_count, and distinct_count per Column Chunk.

3. Nested Representation: Dremel Record Shredding

Parquet handles arbitrary nested structures (structs, arrays, maps) without flattening by implementing Google Dremel's Definition Levels (D-levels) and Repetition Levels (R-levels).

For every value in a nested field, Parquet stores a triplet: (R-level, D-level, Value).

Definition Level (D): An integer specifying how many optional fields along the column's path are defined (non-NULL). This is used to reconstruct NULL states. Repetition Level (R): An integer specifying the exact depth level in the schema path at which a value repeats. This is used to reconstruct nested arrays/lists.

ℹ️ NOTE: Key Benefit: This encoding scheme allows readers to reconstruct nested records completely or project a deeply nested leaf field without reading neighboring fields in the struct.

4. Encoding & Compression Pipeline

Data inside Parquet pages undergoes a two-stage transformation pipeline to minimize disk footprint:

Raw Column Values ---> [ Encoding Engine ] ---> [ Bit-Packing / RLE ] ---> [ Block Compression ] ---> Disk

Primary Encoding Schemes

Dictionary Encoding: Replaces repeating values with integer keys referencing a lookup array stored in the Dictionary Page. Switches back to PLAIN encoding if cardinality exceeds threshold limits.
Run-Length Encoding (RLE) / Bit-Packing: Combines RLE for repeated sequences with bit-packing (using only the minimum bits required for the max integer value). Widely used for encoding R-levels, D-levels, and dictionary indexes.
Delta Encoding: Stores the initial value followed by consecutive differences (value[i] - value[i-1]). Highly effective for sorted sequences, timestamps, and monotonic IDs.
Delta Length Byte Array / Delta ByteArray: Special variants for variable-length strings that extract common prefixes/lengths to optimize memory density.

Page Compression

After encoding, individual pages are compressed independently using algorithms like Snappy (fast CPU-bound decompression), ZSTD (high compression ratio with fast decompression), or LZ4. Because compression occurs at the page level, readers only decompress pages relevant to the query predicates.

5. Indexing & Query Pruning Mechanisms

Parquet provides a three-tier pruning hierarchy to eliminate unnecessary disk I/O:

[ Query Predicate: WHERE age > 30 ]
               │
               v
+------------------------------+
| 1. Row Group Statistics      | ---> Skip entire Row Group if max_value <= 30
+------------------------------+
               │
               v
+------------------------------+
| 2. Column / Offset Indexes   | ---> Skip individual Data Pages within Column Chunk
+------------------------------+
               │
               v
+------------------------------+
| 3. Page-Level Bloom Filters  | ---> Evaluate point lookups (e.g., WHERE uuid = 'x')
+------------------------------+
  1. Row Group Pruning: The reader checks FileMetaData statistics (min_value, max_value) for target columns. If a Row Group's min/max bounds fall completely outside the predicate range, the entire Row Group offset range is skipped.
  2. Page Index (Parquet 2.0+):

ColumnIndex: Stores min/max boundaries for individual data pages within a Column Chunk. OffsetIndex: Contains exact byte offsets and sizes for each page. Allows engines to perform fine-grained page-level skipping without reading an entire Column Chunk.

  1. Bloom Filters: Optional split-block Bloom filters stored before the footer, enabling O(1) probabilistic presence checks for point lookups without scanning data pages.

Related Guides:
← PreviousOpen Table Formats vs Open File FormatsNext →Deep Dive into Apache ORC