Progression from a user story to Hadoop installation and cluster set up

Progression from a user story to Hadoop installation and cluster set up

Yarn-yet another resorce manager settings related to containers minimum and maximum allocated memory explanation

 Yarn-yet another resorce manager settings related to containers minimum and maximum allocated memory explanation

-----
Yarn Resource manager ( The Yarn service Master component)

1) is the controller of the total  resource capacity of the cluster.
2) whatever the container is needed in the cluster it sets the minimum container size that is controlled by
yarn configuration property


yarn.scheduler.minimum-allocation-mb 1024(This value changes based on cluster ram capacity)

Description: The minimum allocation for every container request at the RM, in MBs. Memory requests lower than this won't take effect, and the specified value will get allocated at minimum.

and similarly Max container size


yarn.scheduler.maximum-allocation-mb  8192 (This value changes based on cluster ram capacity)

Description:The maximum allocation for every container request at the RM, in MBs. Memory requests higher than this won't take effect, and will get capped to this value


3) In same way the number of cores to assign for each job. 

 yarn.scheduler.minimum-allocation-vcores  1 (value)

Description:The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum 


yarn.scheduler.maximum-allocation-vcores 32(value)
Description:
The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get cancelled .

Node manager ( The yarn service worker component) 

So jobs on yarn cluster runs in individual containers which is allocated  by Node Manager which in turn gets permissions from Resource Manager.
So few configuration parameters of node manager those are important in context of jobs running in the containers.


yarn.nodemanager.resource.memory-mb  8192(value)

Amount of physical memory, in MB, that can be allocated for containers.


yarn.nodemanager.pmem-check-enabled  true(value)

Whether physical memory limits will be enforced for containers.


yarn.nodemanager.vmem-check-enabled   true(value)

Whether virtual memory limits will be enforced for containers.


yarn.nodemanager.resource.cpu-vcores   8(value)

Number of CPU cores that can be allocated for containers.

Note:
And jobs are monitored by AM(Application master) once the application is done execution the AM gets killed .

Lets take a example here( The value in real time changes based on cluster capaciy)
For a map reduce job according to the above settings the minimum container size is 1GB as defined in (yarn.scheduler.minimum-allocation-mb) and can be increased to 8 GB on the whole given in setting  yarn.nodemanager.resource.memory-mb

So if the AM has started a container for MR job then MR container size can be further controlled  by
mapreduce.map.memory.mb = 8Gb(for this scenario) and mapred.map.child.java.opts.

The value for mapreduce.map.memory.mb must be between  Yarn.scheduler.minimum-allocation-mb and 
 yarn.mapreduce.map.memory.mb(the high limit for mapreduce job set by mapreduce.map.memory)

So the boundary value for MR job container would be 1GB on lower end and 8gb on higher end.

Because any MR job is Hadoop mapper is a java process and each Java process has its own heap memory maximum allocation settings configured via mapred.map.child.java.opts and mapreduce.map.memory.mb is the upper memory limit that Hadoop allows to be allocated to a mapper, in megabytes. 

Thus, the Hadoop and the Java settings are related. The Hadoop setting is more of a resource enforcement/controlling one and the Java is more of a resource configuration one. The Java heap settings should be smaller than the Hadoop container memory limit because we need reserve memory for Java code .

IMP: In general the java heap size should be equal to 1/3 memory of the container size.

In hive:

While running any complex queries in hive the container sizes for MR can be tweaked to convert  shuffle joins to Map joins  . If you execute the visual Explain plan for any hive statement query, then it displays the query plan .
Based on query plan tweak the container sizes and view the Visual Explain again to see weather the job is converted from shuffle to map join.


The errors:
 For mapreduce.map.memory.mb If this limit is exceeded, Hadoop will kill the mapper with an error like this:
Current usage:  1GB of  1536 physical memory used; 970.1 MB of 1.0 GB virtual memory used. Killing container. (displays the container id)

If the mapper process runs out of heap memory given in mapreduce.map.java.opts in Hadoop 2+ , the mapper throws a java out of memory exceptions 

Error: java.lang.RuntimeException: java.lang.OutOfMemoryError 

Finally: The containers are launched on local node only when the following parameter value is true

yarn.nodemanager.disk-health-checker.min-healthy-disks 0.25(value) 
The minimum fraction of number of disks to be healthy for the nodemanager to launch new containers. This correspond to both yarn-nodemanager.local-dirs and yarn.nodemanager.log-dirs. i.e. If there are less number of healthy local-dirs (or log-dirs) available, then new containers will not be launched on this node
 
In a nutshell if the total cluster ram capacity is 400gb and if the TEZ container size is 10gb, the cluster environment can run 40 jobs at a time.