GFS-Style Distributed File System Design Interview 2026

This walkthrough studies the workload-specific design published in the 2003 Google File System (GFS) paper, then updates its metadata availability and scale discussion with current HDFS architecture. It is not a universal distributed-filesystem recipe: large sequential files and append-heavy pipelines justify choices that are wrong for POSIX workloads, low-latency random writes, or billions of tiny objects.

This walkthrough assumes the 6-step system design framework and applies it at senior-plus depth. It is Part 16 of an extended system design series, closing Tier 5.

Table of Contents

1. The Problem
2. Step 1 - Clarify Requirements
3. Step 2 - Estimate Scale
4. Step 3 - API and Data Model
5. Step 4 - High-Level Design
6. Step 5 - Deep Dive: Master, Chunks, and Replication
7. Step 6 - Bottlenecks and Trade-offs
8. Reference Architecture
9. Common Mistakes in the Interview
10. Quick Reference
11. Frequently Asked Questions
12. Sources
13. Related Articles

The Problem

We are designing a GFS-style distributed file system: hierarchical paths, large files split across many storage machines, replica placement across failure domains, and high aggregate streaming throughput. The reference point is classic GFS, with HDFS as a related open-source architecture. Modern object stores and Google's successor systems may share metadata/data separation and chunking but should not be described as the same implementation.

The senior framing is that this design is shaped by the workload it serves: large files, mostly sequential reads, and append-oriented processing. It chooses weaker mutation semantics and coarse chunks to optimize aggregate throughput and metadata scale. Quantify the trade-offs without promising an “order of magnitude” that depends on hardware and comparison baseline.

Step 1 - Clarify Requirements

Functional requirements:

• Hierarchical namespace: create, read, write/append, delete, snapshot.
• Large files (GB to TB) with mostly sequential access.
• Atomic record append - the workhorse for log-style pipelines.
• Many concurrent writers to the same file via append.

Out of scope (name, then defer): full POSIX semantics, low-latency random small writes, millions of tiny files (a different system - a key-value store - is the right tool for those).

Non-functional requirements:

• Petabyte-scale capacity, growing without redesign.
• High aggregate throughput - many GB/s across the cluster.
• Durability through replication on commodity hardware that fails constantly.
• Sequential read and append performance prioritised over random IO.
• Single-datacenter in the canonical design; cross-cluster replication is a separate layer above.

The defining clarifying question: what workload? Large sequential reads, large appends, no small random writes. State this explicitly, because every design choice - chunk size, single master, the consistency model - is calibrated to that workload and would be wrong for a general-purpose filesystem.

Step 2 - Estimate Scale

These are hypothetical capacity inputs, far beyond the 2003 paper's published cluster, and should expose limits rather than prove one master will work.

Total capacity. Assume 100 PB of logical user data. Replication factor 3 requires 300 PB before checksums, metadata, free-space headroom, snapshots, and temporary recovery copies. At 10 TB usable per chunkserver that is at least 30,000 chunkservers, more after headroom and failure-domain reserve.

Chunk count. At 64 MB per chunk, 100 PB = 100 x 10^15 / 64 x 10^6 ≈ ~1.6 billion chunks of unique data, or ~4.8 billion chunk replicas.

Metadata. The GFS paper reported less than 64 bytes of metadata per 64 MB chunk for its implementation. Multiplying 1.6B by 64 bytes gives a ~102 GB lower-bound estimate for chunks alone. Namespace objects, file-to-chunk vectors, replica locations, indexes, allocator overhead, locks, snapshots, and operational headroom add substantially. At this extrapolated size, memory, checkpoint/replay time, heartbeat/inventory load, and metadata operations all need measurement; federation or partitioning may already be required.

Throughput. If one node sustains an assumed 100 MB/s for the tested mix, the arithmetic upper bound is large, but client skew, replication, rack links, checksums, disks, recovery, and metadata limits reduce it. Benchmark read/write/append tails and failure-mode throughput; do not label a linear extrapolation “realistic.”

The shape is enormous total bytes and large operations, but the metadata estimate is now a challenge to validate—not proof that one physical host is sufficient.

Step 3 - API and Data Model

The client interacts with two parties:

# Metadata - to the master
open(path)              -> fileHandle
lookup(fileHandle, off) -> [(chunkId, version, [chunkserver locations])]
create(path) / delete(path) / snapshot(path)
 
# Data - directly to chunkservers
read(chunkId, version, offset, length)   -> bytes
write(chunkId, version, offset, bytes)
recordAppend(chunkId, bytes)             -> offset  (atomic)

A subtle point: the master persists the namespace and file-to-chunks mapping durably (operation log + checkpoints), but chunk-to-location mapping is volatile and rebuilt from chunkserver heartbeats at startup. Locations are authoritative at the chunkservers; the master mirrors them. This is what keeps the metadata small enough to fit in memory and the master log small enough to replay quickly.

Step 4 - High-Level Design

flowchart TD
    Client([Client])
    Master[Active Metadata Master<br/>in-memory namespace]
    Standby[Standby Metadata Master<br/>tails committed edits]
    Journal[(Quorum Metadata Journal)]
    subgraph CS["Chunkservers - thousands of nodes"]
        C1[(Chunkserver)]
        C2[(Chunkserver)]
        C3[(Chunkserver)]
    end
    Client -->|metadata lookup| Master
    Master -->|commit namespace edits| Journal
    Journal --> Standby
    Client -->|read / write chunks directly| C1
    Client --> C2
    Client --> C3
    C1 -.heartbeat: chunks held.-> Master
    C2 -.heartbeat.-> Master
    C3 -.heartbeat.-> Master
    C1 -.inventory / heartbeat.-> Standby

Figure 1. One active metadata writer preserves simple namespace ordering while a standby follows quorum-committed edits and storage-node inventories. Clients obtain locations from metadata, then move bytes directly to storage nodes. Fencing prevents both metadata nodes from acting as active after a partition.

The architecture separates metadata from data. Clients cache location answers for a bounded period and talk to storage nodes for bytes; they return to metadata on cache miss, stale location, lease change, or new chunk allocation. The original GFS paper's shadow masters offered read-only access and could lag; they were not an automatic hot standby. A current HA variant can use one active plus standby metadata nodes, a quorum journal, storage reports to the standbys, client failover, and fencing—similar to current HDFS HA.

Step 5 - Deep Dive: Master, Chunks, and Replication

This is the core. Four ideas cooperate: the single master, fixed-size chunking, primary-driven replicated writes, and the failure-handling that keeps the cluster at full replication.

Part A - Why a single master

“Single master” should mean one logical metadata writer, not necessarily one process with no HA. Three observations make the baseline viable for its target workload:

• The active holds metadata only, but its real memory footprint and recovery time must be measured; the extrapolated 100+ GB estimate above is not automatically safe.
• Clients cache chunk locations after the first lookup, so steady-state read traffic does not touch the master. The master sees one query per file open, not per byte.
• Data flow bypasses the master entirely - reads and writes go straight from client to chunkservers.

The active metadata service is bounded by namespace operations, opens/location lookups, lease work, storage-node reports, recovery, and checkpoint costs rather than payload throughput. The simplification is one ordered namespace and lease authority. It is also a real ceiling: current HDFS Federation adds independent NameNodes/namespaces and block pools for horizontal namespace scale and isolation. Do not claim an undocumented Colossus topology as the automatic next step.

Failover requires more than replay. The standby catches up from a quorum journal, obtains the active role, and fences the former active before serving mutations. Cached locations can keep some reads working during metadata unavailability. Writes may continue only within valid leases and existing protocol state; new files/chunks, expired leases, and stale lookups require metadata. State the availability boundary instead of promising that all existing writes continue.

Part B - Chunking

Files are split into large configurable chunks/blocks. Classic GFS used 64 MB and 64-bit chunk handles; current HDFS documentation describes 128 MB as typical. The size is a workload calibration, not a timeless constant:

• Larger chunks reduce metadata per byte stored - 64 MB vs 4 KB means ~16,000x less metadata.
• Larger chunks also let one TCP connection do long sustained transfers, amortising overhead and saturating the disk.
• Smaller chunks would balance load on hot files better and waste less space on small files.

A small file does not necessarily consume a fully materialized 64/128 MB block on disk, but it still consumes namespace and chunk/block metadata and creates inefficient RPC/IO patterns. Huge small-file counts are therefore a metadata and operational problem. Aggregate only when access, lifecycle, and deletion semantics allow it; otherwise choose storage designed for many small objects.

Part C - Replication and the primary-lease write path

Each chunk is replicated to N chunkservers (typically 3), placed on distinct racks so a rack failure cannot lose any chunk. The master grants a lease on each chunk to one replica - the primary - which serialises writes. The lease has a TTL; the primary renews while alive, and a failed primary's lease expires so the master can grant a new one.

A write involves two flows that the design deliberately separates:

sequenceDiagram
    participant C as Client
    participant M as Master
    participant P as Primary Replica
    participant S1 as Secondary 1
    participant S2 as Secondary 2
 
    C->>M: lookup chunk -> primary + secondaries
    M-->>C: P, S1, S2 (current version)
    Note over C,S2: Data flow - pipelined
    C->>S1: push data
    S1->>S2: forward
    S2->>P: forward
    Note over C,P: Control flow - serialise the write
    C->>P: write (commit)
    P->>S1: apply at offset X (chosen by P)
    P->>S2: apply at offset X
    S1-->>P: ack
    S2-->>P: ack
    P-->>C: ack write complete

Figure 2. The client injects the payload once and replicas pipeline it while receiving, reducing client uplink work and overlapping transfers. The network still transports a copy along each replication hop. The lease-holding primary separately assigns mutation order; a failed secondary makes the attempt fail and may leave replicas temporarily different.

Data flow is pipelined: each replica can forward portions while receiving them, so the client sends the payload once and transfers overlap. Replication still consumes network capacity for every copy. Control flow happens separately: the client tells the primary to mutate, the primary chooses a serial order and instructs secondaries. If a secondary fails, the attempt returns failure even though some replicas may already contain the mutation; the client retries through the protocol rather than assuming an all-or-nothing transaction.

Part D - Atomic record append

The specialized GFS recordAppend primitive lets the client supply bytes while the primary chooses an offset. If the record does not fit in the current chunk, the primary pads the remainder and asks the client to retry on the next chunk. Concurrent appends are serialized by the primary.

After an ambiguous failure, the client retries and the same record can appear more than once. Failed attempts can also leave padding or inconsistent regions on replicas. Successful records are written atomically at least once, so applications attach IDs/checksums, skip padding/corrupt fragments, and deduplicate. Do not reduce this to a generic messaging slogan or claim that the append “definitely” happened when the client received failure.

Part E - Failure handling

The cluster runs on commodity hardware that fails continuously - this is not the exceptional case, it is the steady state.

• Chunkserver crash. After failure detection, the master marks affected chunks under-replicated and prioritizes copies onto placement-compliant targets. Recovery is rate-limited against foreground IO and is complete only when the desired healthy failure-domain placement is restored.
• Stale replica. In classic GFS, the master increases a chunk's version when granting a new lease, not on every write. A replica that missed the version change is excluded as stale and later garbage-collected. Versioning identifies history/lease generation; it does not repair arbitrary divergent bytes from a failed mutation.
• Bit rot. Chunkservers checksum each chunk; a checksum failure on a read returns an error to the client and reports the corruption to the master, which triggers re-replication from a clean copy.
• Metadata failover. A caught-up standby must win the active role and fence the old active before mutations resume. Quorum-journal availability and client failover determine the outage.
• Network partition. Lease expiry prevents the metadata service from safely granting overlapping primaries. Cached reads may succeed, but writes and stale-location handling follow explicit lease/version rules; do not let an isolated storage node invent authority.

Consistency model

GFS makes a deliberate trade. Namespace operations (creating, deleting, renaming files) are strongly consistent through the master. Chunk data has a weaker model:

• A successful serial write leaves the mutated region defined: consistent across replicas and matching the mutation.
• Successful concurrent overlapping writes may leave a region consistent but undefined: replicas agree, but the bytes may be fragments from multiple mutations.
• A failed mutation can leave a region inconsistent across replicas. Re-replication from one copy does not reconstruct the writers' intent; GFS applications use checksums, self-validating records, retries, and append semantics appropriate to the workload.
• A successful record append writes a defined record atomically at least once, while retries may duplicate records and failed attempts can leave padding/inconsistent surrounding regions.

This is far less than POSIX promises and far more than nothing - it is calibrated to the workload it serves.

Multi-cluster / multi-region

For this interview baseline, keep the metadata quorum and synchronous chunk-write path within a latency-bounded region/failure domain. Cross-region durability can use asynchronous replication or a higher-level copy service, which introduces explicit RPO, lag, conflict/ownership, and failover semantics. If the requirement demands one writable cross-region namespace, it is a materially different consensus and data-placement design—not a free extension of the GFS paper.

Evolution path

If building this system is justified, define immutable IDs, checksummed formats, metadata journal/checkpoints, fencing epochs, chunk generations, and client retry/idempotency rules before scale. Do not wait only for RAM exhaustion: namespace-operation rate, startup/replay time, storage reports, fault isolation, and small-file count can trigger federation first.

Observability

Track missing, corrupt, under-replicated, mis-placed, and over-replicated chunks separately; bytes at risk by failure domain; repair queue age/bytes/rate; time below desired redundancy; metadata RPC p99 and queueing; journal quorum latency; standby apply lag; checkpoint/replay duration; namespace objects and memory; heartbeat/inventory lag; capacity/skew; foreground versus repair bandwidth; checksum failures; stale generations; and client retry outcomes. Define repair SLOs by failure size and available headroom rather than demanding zero under-replication at all times.

Step 6 - Bottlenecks and Trade-offs

• Metadata memory, RPC rate, journal/checkpoint latency, inventory processing, and restart time can each be the control-plane ceiling; federate or partition before the operational limit is crossed.
• Re-replication bandwidth after a failure must complete fast enough to keep up with the next failure - the cluster is in a race.
• Hot files/chunks concentrate read load; increase placement selectively, cache immutable ranges, or split the access pattern while preserving version correctness.
• Small files create metadata and IO overhead; aggregate only when independent lifecycle/access semantics are not required.
• Replication cost is substantial. Erasure coding can reduce cold-data overhead at the price of encode/decode, repair fan-in, and small-write complexity.
• GFS mutation semantics are deliberate for this workload. A requirement for POSIX-compatible random writes means choosing or designing a different system, not dismissing the requirement.

Security is part of the control plane. Authenticate clients and storage nodes; authorize every namespace and chunk-location request; use short-lived scoped block/chunk access tokens so a leaked location is not perpetual authority; encrypt traffic and storage when required; isolate tenants and quotas; audit namespace mutations; and protect deletion/snapshot keys. Checksums detect accidental corruption, not malicious tampering or unauthorized reads.

Reference Architecture

The pattern this problem teaches, reusable beyond file systems:

One fenced logical metadata writer with standby/quorum-backed recovery, paired with many storage nodes holding checksummed chunks; clients cache versioned locations and move bytes directly, while the control plane manages leases, placement, repair, rebalancing, and garbage collection.

flowchart LR
    subgraph Meta["Metadata service - one active writer"]
        M[Active + Standby<br/>Quorum Journal + Fencing]
    end
    subgraph Data["Data plane - thousands of nodes"]
        N1[(Node)] --- N2[(Node)] --- N3[(Node)]
    end
    Client([Client]) -->|metadata, cached with version/lease| Meta
    Client -->|all data flow| Data
    Meta -.orchestrate replication.-> Data

Figure 3. The metadata/data split removes payload bandwidth from the active namespace writer, but it does not remove control-plane scaling or availability limits. Standby state, quorum-committed edits, fencing, bounded client caches, and storage inventories make failover and stale-location handling explicit.

The metadata/data split appears in HDFS and many storage/control-plane systems, but their consistency and HA protocols differ. Removing bytes from the metadata service helps; it does not guarantee linear scale, instant failover, or that all metadata fits forever.

Common Mistakes in the Interview

• Putting data through the master, defeating the whole reason a single master is acceptable.
• Assuming POSIX semantics and designing for fine-grained random writes that the workload does not need.
• Ignoring the small-file antipattern and proposing the design as a general-purpose filesystem.
• No story for chunkserver failure - re-replication and version numbers must be in the design, not bolted on.
• Treating the original shadow master as an automatic hot standby, or omitting quorum journal and fencing in a current HA design.
• Promising exactly-once record append instead of at-least-once with idempotent consumers.
• Saying chunk versions increment on every write; classic GFS increments the generation when granting a new lease.
• Claiming the pipeline uses only one payload of network bandwidth; it makes the client send once and overlaps replica transfers, but every replica still receives bytes.
• Calling all successful writes equivalent instead of distinguishing serial defined, concurrent consistent-but-undefined, failed inconsistent, and record-append semantics.
• Assuming re-replication repairs logical mutation conflicts rather than restoring copies from a selected healthy replica.

Quick Reference

Frequently Asked Questions

Why does a GFS-style file system use one master instead of many?

One logical writer simplifies ordered namespace mutations, leases, and placement while data bypasses it. For current HA, run active/standby metadata nodes over a quorum journal and fence the old active. The original GFS shadow masters were read-only and could lag; federation is a separate scaling mechanism.

Why are files broken into fixed-size chunks?

Large chunks reduce metadata per byte, amortize lookups, and suit streaming and parallel placement. GFS used 64 MB; current HDFS documentation describes 128 MB as typical and configurable. Small files do not necessarily reserve a full chunk on disk, but their namespace/block overhead still hurts at scale.

How does a write actually flow through chunkservers?

The client pipelines bytes through every replica, then asks the lease-holding primary to order the mutation. Pipelining overlaps transfers and makes the client inject data once, but the network still carries each replica copy. A secondary failure can leave a partially applied attempt and causes a retry.

What is an atomic record append?

The primary chooses the append offset. A successful GFS record append writes a defined record atomically at least once, but ambiguous retries can duplicate it and failed attempts can leave padding or inconsistent regions. Consumers validate records and deduplicate by an application ID.

How does the system recover from a chunkserver failure?

The metadata service detects loss, prioritizes under-replicated chunks, and copies verified replicas to placement-compliant targets. Repair takes capacity and time. Classic GFS increments a chunk generation when granting a new lease so replicas that missed it can be excluded as stale.

What consistency does a GFS-style file system provide?

Classic GFS distinguishes defined, consistent-but-undefined, and inconsistent regions. Serial successful writes are defined; concurrent successful writes may mix but agree across replicas; failed mutations may differ. Record append is atomic at least once. These are deliberately not general POSIX semantics.

Sources

• Google Research: The Google File System (2003)
• Apache Hadoop 3.5.0: HDFS Architecture
• Apache Hadoop 3.5.0: HDFS High Availability with Quorum Journal Manager
• Apache Hadoop 3.5.0: HDFS Federation
• Apache Hadoop 3.5.0: HDFS Erasure Coding

Related Articles

• System Design Interview Problems: A Senior's Roadmap - the full series index and pattern library.
• System Design Interview Guide: The 6-Step Framework - the method this walkthrough applies.
• Design Video Streaming - Part 10; the chunked-blob storage that sits a layer above this kind of file system.
• Design a Unique ID Generator - Part 13; the global chunk IDs that index every replica.
• Design a Notification Service - Part 3; at-least-once delivery with idempotent consumers, exactly the contract record append offers.

This is Part 16, the close of Tier 5 and of the extended system design series. The distributed file system makes a fitting end because it inverts almost every default the rest of the series accepted - a single master instead of partitioned ownership, deliberate weakening of consistency instead of strengthening, large sequential operations instead of small fast ones - and shows that the right architecture is always the one calibrated to the workload, not the one that ticks the most "modern distributed systems" boxes. Return to the series roadmap to revisit any pattern.

Frequently Asked Questions

Why does a GFS-style file system use one master instead of many?

One logical metadata writer simplifies namespace ordering, leases, and placement while clients move file bytes directly to storage nodes. It is still a throughput, memory, and availability boundary. The 2003 GFS paper used replicated logs plus shadow masters for read-only access, not an automatic hot-standby takeover. A current interview design should use an active/standby metadata service with a quorum journal and fencing, or federate independent namespaces when one service no longer scales.

Why are files broken into fixed-size chunks?

Large configurable chunks reduce metadata per byte, amortize lookups and connection setup, and enable parallel placement and streaming IO. The original GFS used 64 MB chunks; current HDFS documentation describes 128 MB as typical, not universal. A small file does not necessarily reserve a full chunk on disk, but it still creates namespace and block/chunk metadata, so huge small-file counts pressure the metadata service and create inefficient IO.

How does a write actually flow through chunkservers?

In the GFS write path, the client pipelines data through every replica, then sends a control request to the lease-holding primary. The primary assigns a serial order and asks secondaries to apply that order. Pipelining overlaps transfers and makes the client send the payload once, but the network still carries a copy to each replica. If any secondary fails, the client receives failure and retries; replicas can temporarily contain divergent mutation results.

What is an atomic record append?

The client supplies a record but not its offset; the primary selects an offset and orders the append on replicas. In GFS, a successful append creates a defined region containing the record, but retries after ambiguous or partial failures can create duplicates, padding, and inconsistent regions around successful records. Consumers therefore include record IDs, validate records, and deduplicate. This is a specialized GFS API, not a universal filesystem guarantee.

How does the system recover from a chunkserver failure?

After failure detection, the metadata service marks affected chunks under-replicated and schedules prioritized copies from healthy replicas to placement-compliant targets. Recovery consumes real disk and network capacity and is not instantaneous. In classic GFS, the master increments a chunk version when granting a new lease; a replica that missed that change is stale and is excluded and later garbage-collected. Checksums detect corruption, while repair copies from a verified replica.

What consistency does a GFS-style file system provide?

In classic GFS, successful serial writes leave a defined region; successful concurrent writes may be consistent but undefined, meaning replicas agree but the bytes may mix writes. A failed mutation can leave an inconsistent region and re-replication alone does not reconstruct intent. Record append is atomic at least once and can duplicate records. Namespace mutations are serialized by the metadata master. State these workload-specific semantics instead of claiming POSIX behavior.