Hadoop Mapreduce Interview Questions

The two main components of the MapReduce Job are the JobTracker and TaskTracker.

JobTracker - It is the master that creates and runs the job in the MapReduce. It runs on the name node and allocates the job to TaskTrackers. The JobTracker first receives the request from the client and talks to the name node to determine the location of the data. Then, it finds the best TaskTracker node to execute the tasks based on the data locality. JobTracker also monitors the individual TaskTrackers and submits the overall status of the job to the client.

TaskTracker - It is the slave that runs on the data node. TaskTracker runs the job sent by the JobTracker and reports the status of the task back to it. The TaskTracker will be assigned with the Mapper and Reducer tasks to execute the job.

The two main operations executed by the MapReduce framework is the map and reduce task.

In the map operation, the data is split and mapped into different nodes for processing.

In the reduce task, the processed data shuffled and reduced.

The Distributed cache in the Hadoop MapReduce framework is used to cache the files that are needed by the applications.

It is used to cache read-only text files, archives, jar files, etc. The cached file in the MapReduce distributed cache is available on each data node where the map/reduce tasks are running. Distributed cache provides many benefits such as storing complex data, eliminating a single point of failure, and ensuring data consistency.

Shuffling is the process of transferring the data. It transfers the data from the mappers to the reducers. The output data from the map is sent as input to the reducer. This process is necessary for the reducers or they would not have any input.

Sorting operation sorts the keys generated by the mapper. It is done to easily distinguish when a new reduce task should start. When a new key in the sorted input data is different from the previous, then a new reduce task starts.

Mapper is one of the functions in the MapReduce that is used to process the input data. It processes the data by creating several small chunks of it. The mapper function takes the input in the form of a key/value pairs and processes the data into several chunks. The input data is specified to the Mapper function by the InooutFormat that defines the location of the input data. The RecordReader objects present in the InputFormat are used to extract the key/value from the input source. Then, the Mapper processes the input and creates an intermediate output that is sent o the Reducer.

InputSplit in the MapReduce is used to represent the data logically that is used by the mapper process. So the number of InputSplits are equal to the number of map tasks. Every InputSplit has a storage location and the length of the InputSplits is measured in bytes. The important thing to note is that the InputSplit just references the data and it doesn’t actually contain the data. The Input format in the Hadoop is responsible for creating the InputSplits. The split size based on the size of the data in the MapReduce program can be user-defined.

As the name suggests, the reducer reduces the set of intermediate values. The intermediate values are reduced to the smaller set of values that share a key.

The Reducer has three phases. They are the shuffle, sort, and the reduce phase.

The shuffle and sort phase occur concurrently to get the input from the mapper phase and organize it. The reduce phase occurs after the shuffle. and sort to aggregate the key-value pairs. After it reduces the data, the output is written to the FileSystem.

There are many input formats available in the MapReduce such as the TextInputFormat, KeyValueTextInputFormat, FileInputFormat, etc. But the default InputFormat in the MapReduce is the TextInput Format. It treats each line of the input file as a separate record. This format is useful for unformatted data or line-based records like the log files.

JonConf in the Hadoop MapReduce is the primary interface used by the user to describe a map-reduce job to the Hadoop framework for execution. The job parameters set in the JobConf cannot be altered if they are marked final by the administrators. Also, some parameters in the JobConf are easy to set while some parameters are relatively more complex for the user to control finely. You can also define other advanced facets of the job such as comparators to be used, files to be put in the DistributedCache, etc in the JobConf.

Mapper is the first phase in the MapReduce model to solve the problem. In this phase, the data is distributed and computed on each node. Here, the concept of parallel processing is executed for fast processing.

The next and the final phase is the Reducer in the MapReduce model. It reduces the data from the mapper and stores the data into the file. It also performs tasks such as shuffle and sort.

The Identity Mapper is one of the pre-defined mapper class that can be used with any key/value pairs of data. It is a generic class and also the default mapper class provided by Hadoop. When no mapper class is specified in the MR Driver class, the Identity Mapper class is invoked automatically when a MapReduce job is assigned.

The ChainMapper is also one of the pre-defined mapper classes that allows using multiple mapper classes within a single Map task. All the mappers are run in a chain fashion, that is the output of the first mapper becomes the input of the second mapper and so on. The output of the last mapper class is written to the intermediate files.

The LazyOutputFormat is just a wrapper output format. It ensures that the output file is created only when the given partition of the data exists. It won’t create any empty files in the directories in the MapReduce job if there are no records for the specific partition.

The Counters in the Hadoop MapReduce provides a way to measure the number of operations that occur in the MapReduce job. It is really useful in diagnosing the problem and gathering statistics about the MapReduce job. Counters are defined either by the MapReduce framework or the applications. Two types of Counters are the Built-In counters and User-Defined counters. Each counter in the Hadoop has along for the value and is named by an Enum.

MapReduce is a programming model based on the Java programming language designed for distributed computing. Mapreduce framework. It is used to process parallel problems on a huge dataset across a number of distributed nodes. Using MapReduce, it is easy to scale data processing over multiple computing nodes. Many libraries have been written in different programming languages for the MapReduce programming model. But the popular open-source library is the Apache Hadoop

A Combiner or a semi-reducer is just an optional class in the MapReduce. It mainly summarizes the map output records with the same key. The combiner accepts the input from the map class. The output of the combiner will be sent over the network to the reducer task as the input. So the combiner works between the Map and Reduce class to reduce the amount of data transfer between them. For the combiner to work properly, the key-value type should be the same between the mapper and the reducer class. Also, the combiner doesn’t have a predefined interface as it implements the reduce() method from the interface of the reducer.

Job Tracker performs following activities in Hadoop:

• Client application presents jobs to the job tracker
• Job Tracker imparts to the Name mode to decide data area
• Near the data or with accessible openings Job Tracker finds Task Tracker nodes
• On choosing Task Tracker Nodes, it submits the work
• When a task fails, Job tracker notifies and chooses what to do then.
• Job Tracker observes the Task Tracker nodes

Web Distributed Authoring and Versioning (WebDAV) is an expansion of the Hypertext Transfer Protocol (HTTP) that enables customers to perform remote Web content composing tasks. The WebDAV protocol gives a structure to clients to make, change and move reports on a server, normally a web server or web share. On most working framework WebDAV shares can be mounted as file systems, so it is conceivable to get to HDFS as a standard file system by uncovering HDFS over WebDAV.

• Spark
• Hive
• Pig
• Hbase
• Oozie
• Sqoop

• FileInputFormat
• TextInputFormat
• KeyValueTextInputFormat

• Job’s input area in HDFS.
• Job’s output area in HDFS.
• Input and Output format.
• Classes that contain the map and decrease capacities.
• Lastly jar file for reducer, mapper and driver classes.