juniorHadoop

What are the core components of the Hadoop ecosystem?

Updated Feb 20, 2026

Short answer

Hadoop has three core pieces: HDFS for distributed storage, YARN for cluster resource management and job scheduling, and MapReduce as the original processing engine. Around them sits an ecosystem — Hive for SQL, HBase for random access, Spark as a faster processing engine — and in practice most workloads have moved off MapReduce while HDFS and YARN remained useful for longer.

Deep explanation

HDFS splits files into large blocks — 128 MB by default — and replicates each three times across nodes.

  • The NameNode holds all metadata: the directory tree, and which DataNodes hold which blocks. It keeps this in memory, which caps the number of files a cluster can hold and makes small files genuinely problematic.
  • DataNodes store the blocks and report in with heartbeats.
  • The large block size is deliberate: HDFS is designed for streaming reads of huge files, not random access. It is write-once, append-only — you cannot modify a file in place.

The NameNode was a single point of failure until HA configurations added a standby with shared edit logs and automatic failover.

YARN separated resource management from processing, which is what let engines other than MapReduce run on a Hadoop cluster.

  • ResourceManager — allocates cluster resources globally.
  • NodeManager — one per node, launches and monitors containers.
  • ApplicationMaster — one per job, negotiates resources and coordinates its own tasks.

MapReduce processes in two phases with a shuffle between: map produces key-value pairs, the shuffle groups them by key across the network, reduce aggregates each group. Its defining characteristic — and its downfall — is that it writes intermediate results to disk between every stage. Reliable, but for an iterative algorithm running twenty passes, that is twenty round trips to disk.

The ecosystem around them: Hive (SQL over HDFS), HBase (a column-oriented store for random reads and writes), Pig (a dataflow language), Sqoop (relational import/export), Oozie (workflow scheduling), ZooKeeper (coordination), Spark (in-memory processing, typically 10–100× faster on iterative work).

Where it stands now. Cloud object storage — S3, GCS, ADLS — has largely displaced HDFS, since it decouples storage from compute and removes the NameNode's scaling limits. Kubernetes has displaced much of YARN. Spark displaced MapReduce almost entirely. What survives most strongly is the conceptual model: partition the data, move computation to it, tolerate node failure by re-executing tasks.

Real-world example

Why the small-files problem is a real operational constraint:

TypeScript
1 file  of 10 GB   ->  80 blocks  ->  ~80 metadata entries in the NameNode
100,000 files of 100 KB -> 100,000 blocks -> ~100,000 metadata entries

The second case stores far less data but consumes over a thousand times more NameNode memory, and each entry is roughly 150 bytes of heap. A cluster can exhaust NameNode memory long before it runs out of disk.

It also destroys processing performance: MapReduce assigns one map task per block, so 100,000 tiny files means 100,000 tasks whose startup overhead dwarfs their work. The standard fixes are compacting into sequence files or Avro/Parquet containers, or using HAR archives.

Common mistakes

  • - Storing many small files, which exhausts NameNode memory and creates one task per file regardless of how little data each holds.
  • - Treating HDFS as a general-purpose filesystem
  • it is write-once and optimised for streaming reads, not random access or in-place edits.
  • - Assuming HDFS replication is a backup — it protects against node failure, not against accidental deletion or corruption propagated to all replicas.
  • - Confusing YARN with MapReduce
  • YARN schedules resources for any engine, and MapReduce is only one tenant.
  • - Running iterative algorithms on MapReduce, where writing to disk between every stage dominates the runtime.
  • - Overlooking that the NameNode is a single point of failure unless HA is explicitly configured.

Follow-up questions

  • Why is the default HDFS block size so large?
  • What did YARN change about Hadoop?
  • Why is Spark faster than MapReduce for iterative work?
  • Why has cloud object storage displaced HDFS?

More Hadoop interview questions

View all →