Deep Dive into Apache ORC: High-Performance Columnar Storage

As big data analytics evolved beyond the limitations of text-based formats and row-oriented storage, the Hadoop ecosystem needed a storage format built specifically for blazing-fast analytical queries. Enter Apache ORC (Optimized Row Columnar).

In this technical deep dive, we explore ORC's internal architecture, advanced compression techniques, predicate pushdown capabilities, and how it compares to its main rival, Apache Parquet.

1. Architecture & File Layout

Unlike row-oriented formats (like Avro) that serialize data row by row, ORC is a column-oriented format. It organizes data into chunks called Stripes, and within each stripe, it stores data column by column. This makes it ideal for analytical workloads (OLAP) where queries aggregate specific columns across millions of rows.

The ORC File Structure

An ORC file is self-describing and contains a Postscript, a File Footer, and one or more Stripes.

• Stripes: The core data containers (typically 64MB to 256MB in size). Each stripe operates independently.
• File Footer: Contains a list of all stripes in the file, the file's schema, and aggregate statistics (min, max, sum, count) for every column across the entire file.
• Postscript: Located at the very end of the file. Readers read the Postscript first (by reading the last byte of the file to get its length), which tells them how to decompress and read the File Footer.

2. Binary Encoding & Compression

ORC achieves massive compression ratios—often 75% smaller than equivalent Parquet files and infinitely smaller than JSON—by deploying highly specialized encodings based on the column's data type before applying generic compression (like Zlib or Snappy).

Dictionary Encoding

For string columns with low cardinality (e.g., "State", "Country", or "Status"), ORC automatically builds a dictionary of unique values. Instead of storing the string "California" millions of times, it stores it once in the dictionary and replaces the row data with small integer IDs (e.g., 1).

Run-Length Encoding (RLE)

If a column has long runs of repeating values, ORC compresses them instantly. For example, if 100 consecutive rows have the value 7, instead of writing 7 one hundred times, ORC writes a token saying [Repeat value 7, 100 times].

Bit-Packing

For small integer values (like booleans or small IDs), ORC packs multiple values into a single byte, ensuring zero wasted bits.

3. Indexing & Predicate Pushdown

ORC's superpower is its ability to skip reading massive amounts of data that aren't relevant to a query. This is achieved through multi-level statistics and indexes.

Three Levels of Statistics

1. File Level: Contained in the File Footer. If a query asks for WHERE year = 2020 and the file-level max year is 2019, the query engine skips the entire file.
2. Stripe Level: Contained in the Stripe Footer. Allows the engine to skip entire 256MB chunks of data.
3. Row Group Level: Within a stripe, data is logically divided into Row Groups (default 10,000 rows). ORC maintains min/max stats and optional Bloom Filters for each Row Group.

[!TIP] Predicate Pushdown: By pushing SQL WHERE clauses down into the ORC reader, the engine only decompresses and reads the specific 10,000-row chunks that might contain matching data, vastly reducing CPU and disk I/O.

4. ORC vs. Parquet

While both are leading columnar formats, they have distinct historical roots and slight architectural differences.

The Verdict: If you are heavily invested in the Apache Hive ecosystem or flat, wide tables, ORC is exceptional. If you are using Apache Spark, nested JSON-like data structures, or modern Open Table Formats like Iceberg, Parquet is the industry standard.

5. Apache ORC Java Example

An Apache ORC example in Java demonstrating schema definition, writing (serialization), and reading (deserialization) data using the ORC Core API.

Maven Dependency

<dependency>
    <groupId>org.apache.orc</groupId>
    <artifactId>orc-core</artifactId>
    <version>1.9.2</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.3.6</version>
</dependency>

Complete Java Example

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector;
import org.apache.hadoop.hive.ql.exec.vector.LongColumnVector;
import org.apache.hadoop.hive.ql.exec.vector.VectorizedRowBatch;
import org.apache.orc.OrcFile;
import org.apache.orc.Reader;
import org.apache.orc.RecordReader;
import org.apache.orc.TypeDescription;
import org.apache.orc.Writer;

import java.io.IOException;

public class OrcExample {

    public static void main(String[] args) throws IOException {
        Configuration conf = new Configuration();
        Path filePath = new Path("users.orc");

        // 1. Define the Schema
        TypeDescription schema = TypeDescription.fromString("struct<user_id:int,name:string>");

        // 2. Write Data to ORC File
        try (Writer writer = OrcFile.createWriter(filePath,
                OrcFile.writerOptions(conf).setSchema(schema))) {
            
            // Create a vectorized row batch
            VectorizedRowBatch batch = schema.createRowBatch();
            LongColumnVector idVector = (LongColumnVector) batch.cols[0];
            BytesColumnVector nameVector = (BytesColumnVector) batch.cols[1];

            // Add rows
            for (int i = 0; i < 10; ++i) {
                int row = batch.size++;
                idVector.vector[row] = i;
                
                String name = "User_" + i;
                byte[] buffer = name.getBytes();
                nameVector.setRef(row, buffer, 0, buffer.length);
                
                // Flush the batch when full
                if (batch.size == batch.getMaxSize()) {
                    writer.addRowBatch(batch);
                    batch.reset();
                }
            }
            
            // Flush any remaining rows
            if (batch.size != 0) {
                writer.addRowBatch(batch);
                batch.reset();
            }
            System.out.println("Data written to " + filePath);
        }

        // 3. Read Data from ORC File
        try (Reader reader = OrcFile.createReader(filePath, OrcFile.readerOptions(conf));
             RecordReader rows = reader.rows()) {
             
            VectorizedRowBatch batch = reader.getSchema().createRowBatch();
            System.out.println("\nReading records from ORC file:");
            
            while (rows.nextBatch(batch)) {
                LongColumnVector idVector = (LongColumnVector) batch.cols[0];
                BytesColumnVector nameVector = (BytesColumnVector) batch.cols[1];
                
                for (int r = 0; r < batch.size; r++) {
                    int id = (int) idVector.vector[r];
                    String name = new String(nameVector.vector[r], nameVector.start[r], nameVector.length[r]);
                    System.out.println("User ID: " + id + ", Name: " + name);
                }
            }
        }
    }
}