Tuesday, November 8, 2016

Setting up Hadoop to run on Single Node in Ubuntu 15.04

This is tested on hadoop-2.7.3.

Improvement on Hadoop documentation : http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html

Step 1 

Make sure Java is installed

Installation instruction : http://suhothayan.blogspot.com/2010/02/how-to-set-javahome-in-ubuntu.html

Step 2

Install pre-requisites

$ sudo apt-get install ssh
$ sudo apt-get install rsync

Step 3

Setup Hadoop

$ gedit hadoop-2.7.3/etc/hadoop/core-site.xml

Add (replace {user-name} with system username, E.g "foo" for /home/foo/)

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
<name>hadoop.proxyuser.{user-name}.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.{user-name}.hosts</name>
        <value>*</value>
    </property>
</configuration>

$ gedit hadoop-2.7.3/etc/hadoop/hdfs-site.xml 

Add 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Step 4

Run

$ ssh localhost 

If it requested for password, run:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Try ssh localhost again.
If it still asks for password, run following and try again:

$ ssh-keygen -t rsa
#Press enter for each line
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod og-wx ~/.ssh/authorized_keys 

Step 5

Clean namenode

$ ./hadoop-2.7.3/bin/hdfs namenode -format

Step 6 * Not provided in Hadoop Documentation 

Replace ${JAVA_HOME} with hardcoded path in hadoop-env.sh

gedit hadoop-2.7.3/etc/hadoop/hadoop-env.sh

Edit the file as 

# The java implementation to use.
export JAVA_HOME={path}/jdk1.8.0_111

Step 7

Start Hadoop 

$ ./hadoop-2.7.3/sbin/start-all.sh

The Hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

Browse the web interface for the NameNode;

http://localhost:50070/

Step 8

Check processors running by running:

$ jps

Output: 

xxxxx NameNode
xxxxx ResourceManager
xxxxx DataNode
xxxxx NodeManager
xxxxx SecondaryNameNode

Step 9

Make HDFS directories for MapReduce jobs:

$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user
$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user/{user-name}


38 comments:

  1. Thanks for sharing the information very useful info about Hadoop and

    keep updating us, Please........

    ReplyDelete
  2. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have Keep up the good work you are doing here.Well, got a good knowledge.

    Hadoop Training in Chennai

    Dot Net Training in Chennai

    ReplyDelete
  3. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Hadoop Training in Chennai

    ReplyDelete
  4. I have seen a lot of blogs and Info. on other Blogs and Web sites But in this Hadoop Blog Information is useful very thanks for sharing it........

    ReplyDelete
  5. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  6. Before choosing a Job Oriented Training program it is important to evaluate your skills, interests, strength and weakness. Job Oriented Courses enable you to get a identity once you finish the same. Choose eNvent software Technology that suits you and make your career worthwhile.

    ReplyDelete
  7. I would suggest to take training from someone who is working in real time. Let me tell my story, I was working as a software engineer for a company. After 5 years it was very hard for me to move to other company as I my knowledge is very less. So I thought to change my platform to get new skills and new package in future. After continuous research I decided to take Hadoop training. So I googled on internet for best institute to learn Hadoop, shortlisted SV Soft Solutions institute and attended demo session, impressed to the trainer demo and joined. The course duration was 3 months. The trainer has great knowledge and he explained real time scenarios and taught real time project. I was able to clear my interview with great package. And finally moved to new company.
    You can also reach SV Soft Solutions http://www.svsoftsolutions.com,
    The trainer contact number is +1-845-915-8712, +91-9642373173

    ReplyDelete
  8. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

    Base SAS Training in Chennai

    MSBI Training in Chennai

    ReplyDelete
  9. Besant Technologies in Bangalore is the BEST AWS training institute in Bangalore which offers complete AWS training in Bangalore by well experienced professionals having more than 10+ years of IT experience. AWS Training in Bangalore |
    AWS Training in Chennai |

    ReplyDelete
  10. Setting up Hadoop to run on Single Node in Ubuntu 15.04, As per the specification I have tried in Windows and executed Well As I being a PMP Certified and I got my PMP Training in Mumbai, I was well experienced with Hadoop projects and also have certain knowledge.

    ReplyDelete
  11. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  12. Really very good article, If you really want to start good career in Hadoop training in Hyderabad check the social media reviews search for the good quality institutes for your future path.

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. 6-week summer course in Noida - 6 weeks The summer course plays a crucial role in shaping the career of young aspiring / informatics students. This training has been specifically introduced so that students can become familiar with current industrial culture and industrial needs. Webtrackker technology offers a 6-month training program for students / graduates that includes small and large projects.
    6-week summer course in Noida

    ReplyDelete
  15. A career in vehicle CAD (computer aided design) means you get to apply the cutting-edge computer era to help layout and draft automotive components. In this visual and symbol-based verbal exchange approach, you will be capable of tailor your understanding to the automotive area and to the profession within the car area which you want to go into into. This will ensure a protracted-term, high tech profession route that is now not only applicable today, but for the next day as nicely. The following kinds of individuals may be excellent candidates for a career in auto CAD:

    AutoCAD training in Noida

    AutoCAD training institute in Noida


    Best AutoCAD training institute in Noida


    Company address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Phone: 01204330760, 8802820025
    Email: info@webtrackker.com

    ReplyDelete
  16. Cloud Computing Training In Noida
    Webtrackker is IT based company in many countries. Webtrackker will provide you a real time projects based training on Cloud Computing. If you are looking for the Cloud computing training in Noida then you can join the webtrackker technology.
    Cloud Computing Training In Noida , Cloud Computing Training center In Noida , Cloud Computing Training institute In Noida ,

    Company Address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Email: info@webtrackker.com
    Website: www.webtrackker.com
    http://webtrackker.com/Cloud-Computing-Training-Institutes-In-Noida.php

    ReplyDelete
  17. Video editing course in Noida
    Video editing training institute in Noida- Webtrackker Technology is and IT Training institute providing the Video editing course in Noida, FCP, Final Cut Pro Training in Noida. For more call us- 8802820025.
    Video editing course in Noida
    best video editig course in noida
    best video edtitng institute in noida
    Company Address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Phone: 01204330760, 8802820025
    Email: info@webtrackker.com
    Website: http://webtrackker.com/Best-training-institute-Video-editing-FCP-course-in-Noida.php

    ReplyDelete
  18. hank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me 
    Click here:
    angularjs6 Training in Chennai
    Click here:
    angularjs Training in online

    ReplyDelete
  19. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Click here:
    Microsoft azure training in btm
    Click here:
    Microsoft azure training in rajajinagar

    ReplyDelete
  20. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.
    Devops training in sholinganallur

    ReplyDelete
  21. Very nice post. I always like and such a super contents of these post. Excellent and very cool idea and great content of different kinds of the valuable information's. Big Data Hadoop Training in Pune

    ReplyDelete
  22. Like different elements of India, numerous oil and spices usually cross into making food. This effects in substances getting caught to the partitions of the filter out.
    Visit here
    http://kitchenware.ml
    Best kitchen chimney installation and service
    Auto clean chimney sevice in Faridabad

    ReplyDelete
  23. A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.

    rpa training in electronic-city | rpa training in btm | rpa training in marathahalli | rpa training in pune

    ReplyDelete
  24. After reading this web site I am very satisfied simply because this site is providing comprehensive knowledge for you to audience. Thank you to the perform as well as discuss anything incredibly important in my opinion. We loose time waiting for your next article writing in addition to I beg one to get back to pay a visit to our website in
    java training in chennai | java training in USA

    ReplyDelete
  25. Thank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me
    Data Science course in Chennai | Data science course in bangalore
    Data science course in pune | Data science online course
    Data Science Interview questions and answers | Python course in Kalyan nagar

    ReplyDelete

  26. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    Selenium Interview Questions and Answers

    Best Selenium Training in Chennai | Selenium Training Institute in Chennai | Besant Technologies

    Selenium Training in Bangalore | Best Selenium Training in Bangalore

    Free Selenium Tutorial |Selenium Webdriver Tutorial |For Beginners

    ReplyDelete