Last updated on

MapReduce

πŸ“Š Big Data Processing Methods

Big data processing generally falls into two categories: Batch and Streaming. The core difference lies in how the data is received and processed.

πŸ“¦ Batch Processing

  • Definition: The entire dataset is available up front before processing begins.
  • Use Cases: Ideal for tasks where data is collected over a period and then processed all at once.
    • Example 1: Redacting Personal Identifiable Information (PII) from a massive, existing financial data set.
    • Example 2: Counting the occurrences of every word in an entire book or set of documents (Word Count).
  • Execution: Jobs can be run just once or on a set interval (e.g., daily, weekly).
  • Modern Tools: Typically done using tools like Apache Spark (which succeeded Hadoop).

🌊 Streaming Processing

  • Definition: Data is processed in real-time as it is continuously generated or received; the complete dataset is not available up front.
  • Use Cases: Handling real-time data flows.
    • Example: Processing individual credit card transactions as they happen (swiping a card) to perform a task like redacting an expiration date.
  • Input Source: The input for a streaming job is often a message queue.
  • Micro-Batching: A technique (supported by tools like Spark) that runs small batch jobs very quickly (e.g., every 30 seconds) to imitate real-time streaming.
  • Modern Tools: For true streaming, technologies like Apache Flink are often used.

πŸ—ΊοΈ MapReduce: A Batch Processing Model

MapReduce, introduced by Google in 2004, is a foundational programming model designed to process large datasets in parallel across a cluster of computers. Although now considered outdated and largely replaced by systems like Spark, its core concepts of parallel, distributed computation remain vital.

πŸ’» Architecture and Coordination

The MapReduce job is distributed across a cluster of machines:

  • Master/Leader: A designated machine responsible for coordinating the entire job, delegating work, and handling the splitting and shuffling of data.
  • Workers: Individual servers that perform the actual computation on a portion of the data. The input data is automatically split and distributed among these workers.

πŸͺœ The Three Core Steps

The MapReduce process breaks the computation into three main phases:

1. Map (User-Defined)

  • Function: This is the core, user-defined function where the initial transformation happens on each worker’s local subset of data.
  • Goal: To process the input data and generate intermediate key-value pairs.
  • Word Count Example: A worker receives a set of pages, and the map task outputs pairs of (word, 1) for every instance of a word on those pages.
    • Input: β€œThe cat sat on the mat.”
    • Output (Map): (The, 1), (cat, 1), (sat, 1), (on, 1), (the, 1), (mat, 1)

2. Shuffle (System-Managed)

  • Function: This is a middle step where the system groups all intermediate key-value pairs generated by the Map step and distributes them to the Reduce workers based on the key.
  • Goal: To ensure that all values associated with a single unique key are routed to the same worker for aggregation.
  • Word Count Example: All intermediate pairs for the key β€œthe” ((the, 1), (the, 1), (the, 1)) from all Map workers are sent to a single designated Reduce worker. Other keys, like β€œcat” and β€œdog,” might be sent to a different Reduce worker.

3. Reduce (User-Defined)

  • Function: This is the second user-defined function where the values for each key are aggregated. This is also called the aggregation step.
  • Goal: To combine or aggregate the list of values into a smaller, final output value for that key.
  • Word Count Example: The Reduce worker receives the list of all counts for the word β€œthe” (e.g., [1, 1, 1, 1, 1…]) and sums them up to get the final total count.
    • Input: (the, [1, 1, 1, 100, 7])
    • Output (Reduce/Final): (the, 110)
  • Output: The final results are then written to a specified destination, such as file or object storage.

πŸ”‘ Parallelism and Performance

The fundamental benefit of this model is parallel processing. By splitting the data and having multiple workers compute simultaneously, the total processing time can be significantly reduced.

  • Theory: With three workers of equal speed and equal data distribution, a job should theoretically run three times faster than on a single machine.

🚫 Limitations of MapReduce

While powerful, MapReduce is constrained:

  • Fixed Model: Computation is strictly limited to the Map and Reduce functions, which may not be suitable for highly complex or iterative algorithms.
  • Key-Value Dependence: The model fundamentally relies on data being expressible as key-value pairs.
  • Chaining Complexity: Complex workflows require chaining multiple MapReduce jobs, where the output of one becomes the input for the next, which can be inefficient.