Part 3: Modern Table Formats, File Formats & Storage Internals (Questions 110–149)

Curated from lakehouse architecture interviews at Netflix (Apache Iceberg), Databricks (Delta Lake), Uber (Apache Hudi), and Snowflake, this guide covers 40 modern table formats and storage internals interview questions. Below are the complete questions and detailed, human-written architectural answers, complete with schema trade-offs, internal mechanics, ASCII diagrams, and production code snippets.

Question 110: Compare Parquet, ORC, and Avro. When should each format be used?

Question 111: How is an Apache Parquet file structured internally?

A Parquet file is organized hierarchically:

• File: Contains one or more Row Groups.
• Row Group: A logical horizontal partitioning of data containing rows (typically 128MB to 512MB).
• Column Chunk: Within a Row Group, each column's data is stored contiguously in its own chunk.
• Pages: Column chunks are divided into Pages (Data Pages and Dictionary Pages, ~1MB each). Compression and encoding happen at the page level.
• Footer Metadata: Stored at the end of the file, containing schema, total row count, and min/max statistics for every column chunk in every row group. This enables query engines to skip irrelevant row groups without reading them (Statistics / Predicate Pushdown).

graph TD
    subgraph ParquetFile ["Parquet File Structure"]
        Header[File Header Magic Bytes] --> RG1[Row Group 1 - 128MB]
        RG1 --> RG2[Row Group 2 - 128MB]
        RG2 --> Footer[File Footer: Schema + Min/Max Column Stats]
        
        subgraph RGDetail ["Inside Row Group 1"]
            ColA[Column A Chunk: Pages 1..N]
            ColB[Column B Chunk: Pages 1..N]
        end
    end

Question 112: Why are Apache Iceberg, Delta Lake, and Apache Hudi replacing plain Parquet on S3?

Plain Parquet on S3 has severe limitations: object stores lack atomic directory renames, leading to corrupted reads during pipeline crashes; schema changes require rewrites; and listing millions of file prefixes causes S3 rate-limiting bottlenecks.

Iceberg / Delta / Hudi solve this by adding a transactional metadata layer:

• ACID Transactions: Multi-engine concurrent writes commit atomically via snapshot logs.
• Time Travel: Query previous snapshots of data at any historical point in time.
• In-Place Schema Evolution: Rename, drop, or reorder columns without rewriting underlying Parquet files.
• Partition Evolution: Change partitioning schemes (e.g., daily to hourly) without migrating historical data.
• File Pruning via Metadata: Evaluates queries against central metadata rather than executing slow S3 LIST operations.

Question 113: How does Apache Iceberg's metadata tree structure work?

Iceberg uses a hierarchical snapshot tree:

1. Catalog: Points to the current metadata.json file pointer.
2. Metadata File: Contains table schema, partition specs, and a log of snapshots.
3. Manifest List: Each snapshot points to a Manifest List file, which catalogs individual manifest files and tracks partition field ranges.
4. Manifest File: Lists the actual physical Parquet data files along with per-file record counts and lower/upper bounds for every column.

Why this matters: An analytical query reads only the metadata files to eliminate 99% of data files before touching object storage.

Question 114: What is Copy-on-Write (CoW) vs. Merge-on-Read (MoR) in Lakehouse formats?

• Copy-on-Write (CoW): When a row is updated or deleted, the entire Parquet file containing that row is rewritten into a new Parquet file. Advantage: Fast read performance (no join needed during reads). Disadvantage: High write amplification for streaming updates.
• Merge-on-Read (MoR): Updates are appended to lightweight row-based delta files (Avro) or positional delete files. Advantage: Blazing-fast writes and ingestion. Disadvantage: Reads require merging the base Parquet file with delta files at query time, incurring compute overhead until a background compaction job runs.

Question 115: What is Data Compaction in Apache Iceberg and Delta Lake?

Frequent streaming ingestion creates thousands of small files (100KB–1MB). Compaction is an asynchronous maintenance job that reads hundreds of small data files and rewrites them into optimal, balanced ~512MB columnar files while maintaining full ACID isolation.

Question 116: What is the architectural difference between a Data Lake, Data Warehouse, and Lakehouse?

• Data Warehouse (Snowflake, BigQuery): Proprietary storage format, compute coupled to proprietary storage, highly performant SQL, expensive, lacks direct support for unstructured data.
• Data Lake (S3 + Athena/Hive): Open formats on cheap cloud storage, scalable, but historically lacked ACID guarantees, schema enforcement, and indexing.
• Lakehouse (Iceberg/Delta on S3 + Spark/Trino): Combines the cheap, open storage of data lakes with the ACID transactions, governance, and indexing capabilities of data warehouses.

Question 117: What is Z-Ordering (Multidimensional Clustering)?

Linear sorting works well for a single column (e.g., sorting by date), but if you query by customer and region, the second column loses locality. Z-Ordering maps multidimensional data points onto a 1-dimensional space-filling curve (Z-curve) while preserving spatial locality. It allows query engines to prune data blocks efficiently across multiple query dimensions simultaneously.

Question 118: Compare Snappy, Gzip, and ZSTD compression for analytical workloads.

• Snappy: Extremely fast compression and decompression speeds with moderate compression ratio (~2x–3x). Default choice for Parquet and Spark shuffles where CPU efficiency is prioritized over raw disk space.
• Gzip: High compression ratio (~4x–6x) but CPU-heavy. Splitting Gzip files in older frameworks is difficult.
• Zstandard (ZSTD): Developed by Meta. Combines the compression ratio of Gzip with the decompression speed of Snappy. Rapidly becoming the new production default across modern data engineering platforms.

Question 119: How do table formats handle Schema Evolution without breaking queries?

Older systems identified columns by position (index 0, index 1). If you added a column in the middle, historical data corrupted. Modern formats (Iceberg/Delta) assign an immutable Field ID to every column. When a column is renamed, deleted, or reordered, the metadata updates the Field ID mapping. Older physical Parquet files retain their integrity and are mapped seamlessly at query time.

Question 120: How do you write an Iceberg Time Travel query, and how is it used in disaster recovery?

-- Query dataset as it existed at a specific historical timestamp:
SELECT * FROM prod_db.customer_snapshots
FOR SYSTEM_TIME AS OF '2026-09-01 12:00:00 UTC';

-- Query dataset by specific snapshot ID after a pipeline bug corrupts state:
SELECT * FROM prod_db.customer_snapshots
FOR SYSTEM_VERSION AS OF 849204918204928104;

Disaster Recovery: You can rollback the production table to a pre-corruption snapshot in milliseconds by issuing CALL rollback_to_snapshot('table', snapshot_id).

Question 121: What is a Bloom Filter and how does it prevent unnecessary file scans?

A Bloom Filter is a space-efficient, probabilistic data structure used to test set membership.

• It can return: "Definitely Not in Set" or "Possibly in Set". It never returns a false negative.
• Storage formats write Bloom filters for high-cardinality columns into file footers. If a query filters for user_id = 'x-123' and the Bloom filter returns "Definitely Not", the engine skips reading the entire file from S3.

Question 122: How do AWS S3 Request Limits impact large-scale data pipelines?

Amazon S3 enforces a limit of 3,500 PUT/POST/DELETE and 5,500 GET requests per second per prefix. If a Spark job with 1,000 tasks simultaneously reads or writes to a single prefix like s3://my-lake/data/, tasks will fail with 503 Slow Down errors.

Question 123: How does Partition Evolution work in Apache Iceberg?

In traditional Hive architectures, if you partitioned by /year/month/ and later decided to switch to /year/month/day/, you had to rewrite petabytes of existing data. In Iceberg, partition specs are versioned. Old data remains in the original directory layout, while new writes use the updated partition spec. Queries seamlessly span both layouts using metadata pruning without requiring manual data migration.

Question 124: What is Orphan File Removal and Snapshot Expiration in Lakehouse maintenance?

When files are overwritten or deleted in Iceberg/Delta, the old data files remain on S3 to support Time Travel queries. If unmanaged, storage costs grow unbounded. Teams must run periodic maintenance jobs:

• Expire Snapshots: Removes metadata references to snapshots older than N days (e.g., 7 days).
• Remove Orphan Files: Scans object storage for files that are not referenced in any active manifest (e.g., leftover files from failed worker jobs) and physically deletes them.

Question 125: Compare the physical file layout and footer metadata structure of Apache Parquet versus Apache ORC. How do their stripe/row-group structures differ?

Both Apache Parquet and Apache ORC are open-source columnar storage formats featuring compression, dictionary encoding, and data skipping, but their physical architectures differ in striping and indexing mechanisms:

Question 126: How does Apache Hudi handle copy-on-write (CoW) versus merge-on-read (MoR) storage types, and what are the specific workload trade-offs for each?

Apache Hudi provides two core table storage types to manage upserts, inserts, and deletes:

• Copy-on-Write (CoW): Every commit that updates or deletes a record reads the existing base Parquet file, merges the new row mutations in memory, and writes out a brand new version of the Parquet file.
  • Pros: Maximum read performance! Readers read pure columnar Parquet files with zero merge overhead.
  • Cons: High write amplification. Updating 10 rows in a 100MB Parquet file requires rewriting all 100MB to disk. Poor fit for high-frequency streaming updates.
• Merge-on-Read (MoR): Updates and deletes are appended immediately to lightweight row-based delta log files (stored in Apache Avro format) alongside the base Parquet file.
  • Pros: Blazing-fast write throughput and low ingestion latency.
  • Cons: Readers querying real-time views must merge base Parquet files with uncompacted Avro delta logs on the fly, consuming CPU and memory until background compaction jobs merge delta logs into new base files.

Question 127: Explain the internal structure of Apache Avro. Why is Avro preferred over Parquet for Kafka event streaming layers, and how does schema resolution work?

Apache Avro is a binary, row-oriented data serialization format designed specifically for distributed streaming and message buses:

• Internal Layout: Contains a JSON schema declaration embedded in the file header, followed by a sequence of data blocks separated by 16-byte random synchronization markers (Sync Markers). Data values are encoded compactly in binary using variable-length zigzag encoding for integers without field names or tag overhead.
• Why Avro Beats Parquet for Kafka:
  • Row-by-Row Appends: Kafka streams events one row at a time. Parquet requires buffering thousands of rows in RAM to assemble a 128MB Columnar Row Group before writing. Writing single events to Parquet creates fatal small-file fragmentation!
  • Serialization Speed: Avro serializes single records in sub-microsecond time with minimal CPU footprint.
• Schema Resolution: Avro allows readers and writers to use different schemas. If a field exists in the reader schema but not the writer schema, Avro fills in the reader's specified default value, enabling frictionless evolution.

Question 128: How do modern table formats (Iceberg, Delta Lake, Hudi) handle atomic commits and optimistic concurrency control (OCC) using object storage without a central locking daemon?

Traditional databases rely on a central lock manager in memory. Cloud object storage (S3) has no lock daemon, no flock(), and historically lacked atomic multi-file operations.

Optimistic Concurrency Control (OCC) Mechanics:

1. Optimistic Execution: Writers read the current snapshot version (Snapshot V1). They write new Parquet data files to S3 independently without acquiring any locks.
2. Atomic Commit Attempt (Compare-And-Swap): Writers attempt to commit a new metadata pointer referencing Snapshot V2:
   • In Delta Lake: Uses an atomic PutObject with If-None-Match: * condition or a DynamoDB commit lock (on AWS S3) to ensure only one writer successfully writes 000002.json.
   • In Apache Iceberg: Uses an atomic Compare-And-Swap (CAS) in the Catalog database (e.g., UPDATE table_metadata SET file = 'v2.metadata.json' WHERE file = 'v1.metadata.json').
3. Conflict Resolution: If Writer B commits V2 first, Writer A's CAS commit fails. Writer A inspects the commit log: if Writer B's commit modified files that do not overlap with Writer A's write, Writer A automatically rebases its commit onto V2 and retries the commit without re-writing data!

Question 129: What are the performance and memory implications of small file problems in columnar file formats (Parquet/ORC), and how do compaction utilities resolve them internally?

When streaming pipelines write data every minute, a table can accumulate 500,000 tiny Parquet files (e.g., 50KB to 200KB each).

• Performance & Memory Penalties:
  • Columnar Header Overhead: A 50KB Parquet file still contains schemas, metadata footers, dictionary pages, and block statistics. Metadata consumes up to 30% of total file size!
  • Query Engine Driver OOM: The query planner must fetch and deserialize 500,000 file footers over HTTP, exhausting driver memory and causing queries to sit in planning phase for 15 minutes before reading a single data row.
  • S3 Rate Limiting: 1,000 tasks requesting small files trigger S3 503 Slow Down limits.
• How Compaction Utilities Resolve It: Asynchronous compaction jobs (e.g., Iceberg rewrite_data_files) launch a distributed job that reads thousands of tiny files, groups them into target 512MB bin-packs, and rewrites them into balanced, contiguous columnar files, atomically swapping metadata pointers in a single transaction.

Question 130: How does Parquet's dictionary encoding work, and under what high-cardinality conditions does it degrade query performance or blow up memory?

In Parquet, when a column chunk is written, the writer attempts to store distinct values in a Dictionary Page, replacing occurrences in Data Pages with bit-packed integer indices:

The High-Cardinality Trap (Dictionary Fallback):

• Parquet writers configure a maximum dictionary memory size (typically 1MB per column chunk).
• If a column has massive cardinality (e.g., unique UUIDs, random session IDs, or high-precision floating timestamps), the dictionary page quickly exceeds the 1MB threshold.
• Fallback to Plain Encoding: When the threshold is reached, the Parquet writer abandons dictionary encoding for the rest of that column chunk and falls back to uncompressed PLAIN encoding.
• Consequences: The memory allocated to the aborted dictionary page is wasted, write throughput drops due to dynamic reallocation, file size balloons, and query engines can no longer use dictionary-based vectorized filtering.

Question 131: Explain how Apache Hudi implements record-level index lookups (e.g., Bloom filter index vs. Global level indexes) to optimize upserts and deletes.

When an upsert arrives, Hudi must answer one fundamental question: "Does this record key already exist in our petabyte-scale table, and if so, in which physical file group does it reside?"

• Bloom Filter Index (Default): Hudi embeds Bloom filters in the footer of every base Parquet file. For an incoming batch of keys, Hudi tests the keys against file-level Bloom filters in parallel. If a filter returns true, it confirms by checking record keys in candidate files. Constraint: Scoped per partition. If updates contain the partition key, search is fast; if partition path is unknown, all partitions must be searched.
• Global Bloom Filter Index: Enforces uniqueness across the entire table regardless of partition. If a record moves from partition 2026/01 to 2026/02, the global index detects the old location and emits a delete in the old partition.
• HBase / Metadata Table Record-Level Index: Uses an internal LSM-tree metadata table (or external HBase store) to maintain a direct mapping: record_key → (file_id, partition_path). Provides O(1) point-lookup upsert tagging for high-write-volume streaming pipelines.

Question 132: How do blooming filters and statistics stored within Parquet footers (min/max values, null counts) allow predicate pushdown to skip entire data blocks?

When an analytical engine (Spark, DuckDB, Trino) compiles a SQL query with a WHERE clause:

1. Read Footer Only: The query engine reads the last few kilobytes of the Parquet file (Footer Thrift metadata). The footer contains statistics for every column chunk in every Row Group.
2. Min/Max Range Evaluation: For query WHERE salary > 150000: The engine checks Row Group 1: min_value = 40000, max_value = 110000. Since 150000 > max_value, Row Group 1 cannot possibly contain matching rows and is skipped completely without fetching its byte ranges from S3.
3. Null Count Pruning: For query WHERE email IS NOT NULL: If null_count == row_count, the entire block is dropped.
4. Bloom Filter Evaluation: For point lookups: WHERE user_id = 'u-9842': If min/max ranges overlap, the engine queries the embedded column Bloom filter. If the Bloom filter returns false, the block is skipped, completely eliminating false-positive I/O scans.

Question 133: Contrast how Avro and Parquet handle schema evolution. What happens downstream if a field's data type is changed from integer to long in both formats?

Changing a column type from int (32-bit) to long (64-bit) behaves very differently between row and columnar formats:

• In Apache Avro: Avro natively supports Schema Resolution Type Promotion rules. When the reader encounters an int in an older file while using a reader schema specifying long, Avro automatically promotes the 32-bit integer to a 64-bit long in memory. The read succeeds seamlessly!
• In Plain Apache Parquet: Parquet stores data in physical encodings (e.g., INT32 vs INT64). If an engine attempts to read an older Parquet file containing INT32 using an execution schema expecting INT64, the low-level columnar reader will fail with a ColumnTypeException: Schema evolution not supported / cannot cast INT32 to INT64!
• How Modern Lakehouses (Iceberg) Fix Parquet: Apache Iceberg wraps Parquet with a physical type-promotion metadata layer that safely intercepts and widens primitive types (int → long, float → double) during reader vectorized execution.

Question 134: Why is CSV fundamentally broken for analytical big data workloads at scale, and what specific challenges arise regarding type safety, escaping, and splitability?

CSV remains common in business spreadsheets but is considered an architectural anti-pattern for large-scale distributed analytics:

• Zero Type Safety: CSV stores everything as raw text. Every single query engine must parse ASCII strings into binary integers/floats/dates on every single scan, burning massive CPU cycles.
• Splitability & Escaping Hell: In distributed systems, a 10GB file must be split into 128MB chunks across worker nodes. If a multi-line text field contains an escaped newline (" ") or delimiter (",") inside quotation marks, an arbitrary byte split inside that field causes the worker to misinterpret the remaining text, corrupting row boundaries across the entire cluster!
• No Column Pruning: To read 1 column out of 50 in a CSV, the engine must stream and parse every single character of all 50 columns from disk.
• No Zone Maps / Data Skipping: CSV contains zero metadata statistics, requiring a 100% full table scan for every query.

Question 135: How does Apache Hudi handle timeline management, and what is the exact lifecycle of an instant (requested, in-flight, completed)?

Apache Hudi maintains a centralized Timeline (stored in the .hoodie/ metadata folder) that tracks all actions performed on the table. Each action is represented as an Instant with an exact timestamp (e.g., 20260910010000).

The 3-State Instant Lifecycle:

stateDiagram-v2
    [*] --> Requested: Action Planned (e.g., Commit / Compaction)
    Requested --> Inflight: Transaction Begins Writing Files
    Inflight --> Completed: Atomic Commit & State Published
    Completed --> [*]

1. Requested: Marks that an action (e.g., commit, deltacommit, compaction, clean) has been scheduled by the planner.
2. In-flight: The action is actively executing. Data files or delta logs are being written to storage. Readers ignore in-flight instants to guarantee snapshot isolation.
3. Completed: The write finishes successfully, commit metadata is atomically published, and the action becomes visible to readers.

Crash Recovery: If a writer dies during execution, subsequent Hudi jobs inspect the timeline, identify dangling inflight instants with dead heartbeats, and automatically roll back their uncommitted data files.

Question 136: Explain the internal representation of nested and complex data types (structs, maps, lists) in Apache Parquet using the Dremel shredding algorithm (definition and repetition levels).

Storing flat columnar data is straightforward, but storing deeply nested structures (e.g., an array of objects containing maps of lists) in pure columnar form without flattening or losing structural relationships requires Google's Dremel Record Shredding Algorithm.

Dremel encodes nested hierarchies using two small integers per value:

• Definition Level (DL): Specifies how many optional ancestors in the field's path are actually defined (not null). If a value is null, DL indicates which level of the parent hierarchy became null.
• Repetition Level (RL): Specifies at what repeated field depth the value repeats.
  • RL = 0 indicates the start of a completely new root record.
  • RL > 0 indicates a continuation of an existing list or array at that specific nesting depth.

Why this is brilliant: Using DL and RL, Parquet stores deeply nested JSON/Avro records as pure, independent columnar byte streams, and can reconstruct the exact nested document tree in memory with zero structural ambiguity!

Question 137: How do storage layers implement time travel and snapshot isolation in table formats like Hudi, Delta Lake, and Iceberg using manifest lists and commit logs?

All modern open table formats achieve ACID Snapshot Isolation by making data files immutable and managing state through a hierarchical tree of pointer files:

graph TD
    Catalog[Iceberg Catalog Pointer] --> Snap2[Snapshot 2: metadata.json]
    Snap2 --> ML2[Manifest List 2]
    ML2 --> M1[Manifest A: Parquet Files 1..100]
    ML2 --> M2[Manifest B: Parquet Files 101..150]
    
    Snap1[Snapshot 1: Historical] --> ML1[Manifest List 1]
    ML1 --> M1

• When a query starts, it queries the catalog for the latest Snapshot ID (e.g., Snapshot 2).
• Snapshot 2 points to a frozen Manifest List, which points to immutable Manifest Files cataloging the exact physical Parquet file URIs.
• Even if a concurrent writer commits Snapshot 3 five seconds later, the active query continues scanning the manifest tree of Snapshot 2 without dirty reads.
• Time Travel: To query data as it existed last Friday, the engine simply reads the historical metadata file that was current last Friday!

Question 138: What are the internal mechanics of Bloom filter data skipping in ORC files compared to Parquet's statistics-based row group pruning?

While both formats store min/max stats, their implementation of Bloom filters differs:

• Apache ORC: Bloom filters were integrated into ORC from its inception. In ORC, Bloom filters can be configured on any column and are written directly into the Stripe Index Stream for every 10,000 rows (Row Group index). This provides granular, sub-stripe skipping: queries can skip reading individual 10,000-row chunks inside a stripe!
• Apache Parquet: Historically, Parquet relied strictly on min/max statistics in the Row Group footer. Bloom filters were added in the Parquet 2.x specification (PARQUET-41) as optional blocks stored after column chunks. Many legacy readers do not implement Parquet Bloom filter evaluation, whereas ORC engines (Hive/Trino) evaluate them natively by default.

Question 139: How does Apache Hudi's log compaction process work in Merge-on-Read tables, and how do readers handle reading uncompacted delta logs?

In a Hudi Merge-on-Read (MoR) table:

1. Ingestion: Incoming inserts write to base Parquet files; incoming updates and deletes write to append-only .log files (Avro).
2. Query Types:
   • Read-Optimized Queries: Scans strictly the base Parquet files. Blazing fast, but excludes recent uncompacted updates.
   • Snapshot Queries (Real-Time View): Scans the base Parquet file and loads the delta .log files. Readers maintain an in-memory hash map or priority queue on the record key to merge the latest delta updates over the base rows in real time.
3. Log Compaction: An asynchronous background compaction job runs on a separate schedule. It loads the base Parquet file and all delta log files, resolves merges, and writes a brand new, clean base Parquet file (e.g., version 2). The old log files are then scheduled for garbage collection.

Question 140: Explain how columnar storage formats handle null values physically. Do they consume storage space for nulls, and how are bitmaps used to track them?

In traditional row databases, nullable columns often reserve a fixed byte slot or add a null bitmap byte per row.

In columnar formats (Parquet, ORC, Arrow), null values consume zero data bytes in the actual column value stream!

• Definition Levels (Parquet): If a value is null, its definition level is set to 0 (or less than the required depth). The value itself is never written to the data page! Only the definition level integer is stored.
• Validity Bitmaps (Apache Arrow / ORC): Uses a contiguous bit-vector where each bit represents one row (1 = valid, 0 = null). 8 rows are tracked in a single byte. If bit 4 is 0, the engine skips reading from the value array entirely.
• RLE Compression: If a column contains 10,000 consecutive nulls, Run-Length Encoding compresses the definition levels into just 2 bytes: (0, 10000).

Question 141: Compare the serialization overhead and CPU cost of JSON, Avro, and Protobuf when streaming data into a data lakehouse.

Question 142: How do open table formats handle partition evolution without requiring a full rewrite of historical underlying files (e.g., changing partitioning from daily to hourly)?

In legacy Hive architectures, table partition paths were hardcoded into the directory hierarchy: s3://lake/orders/year=2026/month=09/day=10/. Changing to hourly partitioning required running a massive, dangerous ETL script to rewrite all historical petabytes into new subdirectories.

How Apache Iceberg Achieves In-Place Partition Evolution:

• Decoupled Partition Transforms: Iceberg tables store a list of versioned Partition Specs in metadata.json:
  • Spec 0 (Historical): day(event_time)
  • Spec 1 (Current): hour(event_time)
• Per-File Spec ID: Every Manifest file and Parquet file records the exact spec_id under which it was written. Old historical data remains completely untouched in daily layouts.
• Query Pruning Adaptation: When a user writes WHERE event_time BETWEEN '2026-09-10 10:00:00' AND '2026-09-10 11:00:00', Iceberg converts the filter dynamically: for Spec 1 files it filters by hour; for Spec 0 files it filters by the matching day, seamlessly reading both generations of files in a single unified query!

Question 143: What are the exact metadata bottlenecks when scanning a billion-file Parquet-based data lake, and how do partition projection and manifest caching solve them?

When a data lake scales to 1,000,000,000 files across millions of partition directories:

• The Bottleneck (Listing Latency): Query planners attempting to prune partitions using traditional Hive Metastore or S3 ListObjects must issue millions of HTTP calls. Query planning alone takes 45 minutes to 2 hours before any actual data is processed!
• Solution 1: AWS Athena Partition Projection: Instead of querying the metastore or listing S3, you configure mathematical projection patterns (e.g., projection.date.range = 2020-01-01,NOW). Athena calculates matching S3 directory prefixes purely in memory using arithmetic calculations, bypassing listing entirely.
• Solution 2: Apache Iceberg Manifest Caching: Manifest lists store precomputed partition boundaries and min/max statistics for each manifest file. Query engines (Trino/Spark) cache manifest files in local NVMe/RAM buffers, pruning 99.9% of manifests in memory in < 200 milliseconds.

Question 144: How does Apache Hudi implement clustering (layout optimization), and how does it balance write latency with read performance?

Frequent streaming ingestion creates unorganized, small files on disk. Hudi Clustering reorganizes data layout to optimize query read performance without slowing down live ingestion:

• Decoupled Execution: Clustering runs asynchronously via a background service, leaving foreground streaming ingestion unblocked.
• Scheduling & Execution:
  1. Hudi identifies small file groups or un-ordered files within partitions.
  2. It writes a clustering.requested instant to the timeline, reserving the file groups.
  3. Clustering reads the files, sorts them along high-cardinality query dimensions (e.g., Z-Ordering or linear sort on customer_id, region), and writes new optimized ~512MB columnar files.
• Read Optimization: Subsequent analytical queries filter on customer_id and skip 90% of file groups via min/max range pruning.

Question 145: Explain how predicate pushdown and partition pruning interact in a distributed query engine reading from a Parquet-backed data lakehouse.

Both techniques eliminate unnecessary data scans, but operate at completely different architectural layers:

graph TD
    Q["SELECT * FROM sales WHERE sale_date = '2026-09-10' AND amount > 500"]
    
    subgraph Level1 ["Level 1: Partition Pruning (Catalog Layer)"]
        Q --> P1[Prune Partitions: Keeps only /sale_date=2026-09-10/]
        P1 --> Skip1[Skips other 364 daily directories completely]
    end

    subgraph Level2 ["Level 2: Predicate Pushdown (File Format Layer)"]
        Skip1 --> Scan[Fetch matching Parquet files]
        Scan --> P2[Inspect Parquet Footers: Evaluate amount > 500]
        P2 --> Skip2[Skip Row Groups where max_amount <= 500]
    end

• Partition Pruning (Coarse-Grained): Occurs during initial query planning in the catalog. The engine eliminates entire partition prefixes (directories) matching partition keys (e.g., date).
• Predicate Pushdown (Fine-Grained): Occurs inside the worker reader when opening candidate Parquet files. Non-partition filter predicates (e.g., amount > 500) are pushed down to the Parquet reader to skip internal Row Groups and Data Pages based on column min/max statistics.

Question 146: How do ORC's Lightweight Encodings (direct encoding, dictionary encoding, bit packing, run-length encoding) function internally?

Apache ORC applies specialized, lightweight encodings adaptively per column stream before running general compression (ZSTD/Snappy):

• Integer Run-Length Encoding (RLE v2): Detects three distinct integer patterns:
  • Short Repeat: Contiguous identical values: [5, 5, 5, 5].
  • Direct: Random sequences encoded using bit-packing (storing values in the minimum number of bits required, e.g., 5-bit integers).
  • Delta: Monotonically increasing sequences (e.g., timestamps: 1000, 1002, 1004...) stored as base value plus uniform delta offsets, achieving 90% compression ratios.
• String Dictionary Encoding: Extracts distinct strings into an ordered dictionary. String values in data streams are replaced by integer dictionary offsets, which are then compressed using RLE v2.

Question 147: What is the impact of block size and compression codec selection (Snappy vs. Gzip vs. ZSTD) on Parquet write throughput, file size, and decompression CPU cost?

Question 148: How do transactional table formats handle garbage collection (vacuuming) of orphaned data files without deleting active snapshots being read by concurrent long-running queries?

When a table update occurs, older Parquet files become unreferenced in the newest snapshot. However, they cannot be deleted immediately because:

1. A long-running analytical query may have started 2 hours ago and is still reading that historical snapshot.
2. Data analysts require Time Travel to query data from yesterday.

Safe Vacuum / Garbage Collection Mechanics:

• Retention Horizon (Safety Grace Period): Commands like VACUUM table RETAIN 168 HOURS (7 days) enforce a strict safety threshold. Any snapshot committed within the last 7 days is protected.
• Manifest Reachability Graph: The garbage collector walks the metadata tree, collecting all file URIs referenced in any snapshot newer than the retention cutoff.
• Orphan Identification: The collector lists physical storage files and subtracts all referenced files. Only unreferenced files whose creation timestamp is older than the retention threshold are physically deleted via S3 DeleteObjects.

Question 149: Explain how column projection works at the storage engine level when an analytical query only selects 3 columns out of 200 in a wide Parquet file.

When a query executes SELECT id, total_amount, status FROM wide_table; against a 200-column, 1GB Parquet file:

1. Read Footer Metadata: The storage client issues an S3 HTTP Range request to read the file footer (e.g., Range: bytes=1048000000-1048576000).
2. Offset Identification: The footer Thrift metadata lists the exact byte offsets and byte lengths of every column chunk in every Row Group:
   • Column 'id': Offset 4096, Length 1024000
   • Column 'total_amount': Offset 54000000, Length 2048000
   • Column 'status': Offset 98000000, Length 512000
3. Targeted HTTP Range Requests: The client issues precisely three targeted byte-range requests fetching only the 3.5MB corresponding to those columns.
4. Result: 99.6% of the physical file (996.5MB) is never transferred over the network, saving bandwidth and saturating CPU caches.