Skip to content

Configuring Hadoop on Ubuntu 20.04

    -Using Ubuntu 20.04 & Hadoop 3.3.0-

    *inspiration taken from https://www.digitalocean.com/community/tutorials/how-to-spin-up-a-hadoop-cluster-with-digitalocean-droplets but changes have been made to reflect updates to Ubuntu and Hadoop*

    Initial Configuration (on each node)

    Open a terminal on each node and run the following commands:

    sudo apt-get update && sudo apt-get -y dist-upgrade
sudo adduser hadoop
sudo usermod -aG sudo hadoop

    On each node, change the hostname to something unique. For my setup, I used “namenode” on the name node, and “datanode” on the worker node. Use the following command on each respective node to do this, replacing “<nodename>” with the name you choose:

    sudo hostnamectl set-hostname <nodename>

    Reboot each node using the following command:

    sudo reboot

    Login to the newly created “hadoop” user. Open a terminal and run the following command to edit the “hosts” file. Comment out the “localhost” entries by adding a preceding “#” and add the IP addresses and respective hostnames of your Hadoop nodes. See the picture below for an example.

    sudo nano /etc/hosts

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following commands:

    sudo apt-get -y install openjdk-8-jdk
sudo apt install openssh-server openssh-client -y
mkdir my-hadoop-install && cd my-hadoop-install
wget http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-3.3.0/hadoop-3.3.o.tar.gz
tar xvzf hadoop-3.3.0.tar.gz

    Hadoop Environment Configuration (on each node)

    Open a terminal on each node and run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/hadoop-env.sh

    Add the following lines anywhere in the file, and make sure they aren’t commented out (make sure there is no preceding “#”)

    export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export HDFS_NAMENODE_USER="hadoop"
export HDFS_DATANODE_USER="hadoop"
export HDFS_SECONDARYNAMENODE_USER="hadoop"
export YARN_RESOURCEMANAGER_USER="hadoop"
export YARN_NODEMANAGER_USER="hadoop"

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following commands:

    source ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/hadoop-env.sh
sudo mkdir -p /usr/local/hadoop/hdfs/data
sudo chown -R hadoop:hadoop /usr/local/hadoop/hdfs/data
nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/core-site.xml

    Between “” and “”, add the following, replacing “master-server-ip” with your own master/namenode server’s IP address. Do NOT replace it with each respective server’s own IP, as the DigitalOcean guide recommends.

    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master-server-ip:9000</value>
</property>

    Setup Passwordless SSH

    On the master/namenode server, run the following command, and then press “Enter” three times.

    ssh-keygen

    Run the following command and copy the entire output onto your clipboard.

    cat ~/.ssh/id_rsa.pub

    Run the following command on both the master/namenode server and any worker nodes, and paste in the output from the previous command.

    nano ~/.ssh/authorized_keys

    On the master/namenode server, run the following command:

    nano ~/.ssh/config

    Edit the file using the following format, replacing the underlined portions with your respective master/namenode and worker node IPs.

    Host hadoop-master-server-ip
    HostName hadoop-master-server-ip
    User hadoop
    IdentityFile ~/.ssh/id_rsa

Host hadoop-worker-01-server-ip
    HostName hadoop-worker-01-server-ip
    User hadoop
    IdentityFile ~/.ssh/id_rsa

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal.

    SSH into your worker node(s) from your master/namenode server, replacing “hadoop-worker-01-ip” with your respective IP(s).

    ssh hadoop@hadoop-worker-01-server-ip

    Reply to the prompt with “yes”, and then logout by typing the following (admittedly self-explanatory) command:

    logout

    Configure the Master Node

    On the master node, run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/hdfs-site.xml

    Between “<configuration>” and “</configuration>”, add the following:

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///usr/local/hadoop/hdfs/data</value>
</property>

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/mapred-site.xml

    Between “<configuration>” and “</configuration>”, add the following, replacing “hadoop-master-server-ip” with your master/namenode server’s IP address.

    <property>
        <name>mapreduce.jobtracker.address</name>
        <value>hadoop-master-server-ip:54311</value>
    </property>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
</property>

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/yarn-site.xml

    Between “<configuration>” and “</configuration>”, add the following, replacing “hadoop-master-server-ip” (underlined) with your master/namenode server’s IP address.

    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>hadoop-master-server-ip</value>
</property>

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/masters

    Type in your master/namenode server’s IP address.

    hadoop-master-server-ip

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal. Run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/workers

    Type in your worker node(s) IP address(es), one per line, below the “localhost” entry.

    localhost
hadoop-worker-01-server-ip
hadoop-worker-02-server-ip
hadoop-worker-03-server-ip

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal.

    Configure the Worker Node(s)

    On the worker node(s), run the following command:

    nano ~/my-hadoop-install/hadoop-3.3.0/etc/hadoop/hdfs-site.xml

    Between “<configuration>” and “</configuration>”, add the following.

    <property>
        <name>dfs.replication</name>
        <value>3</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///usr/local/hadoop/hdfs/data</value>
</property>

    Press “Ctrl + X”, “Y” and then “Enter” to save your changes and return to the terminal.

    Starting up Hadoop

    On the master/namenode server, run the following commands:

    cd ~/my-hadoop-install/hadoop-3.3.0/
sudo ./bin/hdfs namenode -format
sudo ./sbin/start-dfs.sh
./sbin/start-yarn.sh

    Verify Functionality

    On each of your nodes, run the following command to ensure Hadoop processes are running:

    jps

    On the master/namenode, visit the following URL in a web browser, replacing “hadoop-master-server-ip” with your own master/namenode server’s IP address:

    http://hadoop-master-server-ip:9870

    Click on “Datanodes” on the menu bar, and ensure that all of your worker nodes’ IP addresses are showing up on the web GUI.

    Tags: