Tuesday, November 8, 2016

Setting up Hadoop to run on Single Node in Ubuntu 15.04

This is tested on hadoop-2.7.3.

Improvement on Hadoop documentation : http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html

Step 1 

Make sure Java is installed

Installation instruction : http://suhothayan.blogspot.com/2010/02/how-to-set-javahome-in-ubuntu.html

Step 2

Install pre-requisites

$ sudo apt-get install ssh
$ sudo apt-get install rsync

Step 3

Setup Hadoop

$ gedit hadoop-2.7.3/etc/hadoop/core-site.xml

Add (replace {user-name} with system username, E.g "foo" for /home/foo/)

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
<name>hadoop.proxyuser.{user-name}.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.{user-name}.hosts</name>
        <value>*</value>
    </property>
</configuration>

$ gedit hadoop-2.7.3/etc/hadoop/hdfs-site.xml 

Add 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Step 4

Run

$ ssh localhost 

If it requested for password, run:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Try ssh localhost again.
If it still asks for password, run following and try again:

$ ssh-keygen -t rsa
#Press enter for each line
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod og-wx ~/.ssh/authorized_keys 

Step 5

Clean namenode

$ ./hadoop-2.7.3/bin/hdfs namenode -format

Step 6 * Not provided in Hadoop Documentation 

Replace ${JAVA_HOME} with hardcoded path in hadoop-env.sh

gedit hadoop-2.7.3/etc/hadoop/hadoop-env.sh

Edit the file as 

# The java implementation to use.
export JAVA_HOME={path}/jdk1.8.0_111

Step 7

Start Hadoop 

$ ./hadoop-2.7.3/sbin/start-all.sh

The Hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

Browse the web interface for the NameNode;

http://localhost:50070/

Step 8

Check processors running by running:

$ jps

Output: 

xxxxx NameNode
xxxxx ResourceManager
xxxxx DataNode
xxxxx NodeManager
xxxxx SecondaryNameNode

Step 9

Make HDFS directories for MapReduce jobs:

$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user
$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user/{user-name}


60 comments:

  1. Thanks for sharing the information very useful info about Hadoop and

    keep updating us, Please........

    ReplyDelete
  2. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have Keep up the good work you are doing here.Well, got a good knowledge.

    Hadoop Training in Chennai

    Dot Net Training in Chennai

    ReplyDelete
  3. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Hadoop Training in Chennai

    ReplyDelete
  4. I have seen a lot of blogs and Info. on other Blogs and Web sites But in this Hadoop Blog Information is useful very thanks for sharing it........

    ReplyDelete
  5. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  6. Before choosing a Job Oriented Training program it is important to evaluate your skills, interests, strength and weakness. Job Oriented Courses enable you to get a identity once you finish the same. Choose eNvent software Technology that suits you and make your career worthwhile.

    ReplyDelete
  7. I would suggest to take training from someone who is working in real time. Let me tell my story, I was working as a software engineer for a company. After 5 years it was very hard for me to move to other company as I my knowledge is very less. So I thought to change my platform to get new skills and new package in future. After continuous research I decided to take Hadoop training. So I googled on internet for best institute to learn Hadoop, shortlisted SV Soft Solutions institute and attended demo session, impressed to the trainer demo and joined. The course duration was 3 months. The trainer has great knowledge and he explained real time scenarios and taught real time project. I was able to clear my interview with great package. And finally moved to new company.
    You can also reach SV Soft Solutions http://www.svsoftsolutions.com,
    The trainer contact number is +1-845-915-8712, +91-9642373173

    ReplyDelete
  8. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

    Base SAS Training in Chennai

    MSBI Training in Chennai

    ReplyDelete
  9. Besant Technologies in Bangalore is the BEST AWS training institute in Bangalore which offers complete AWS training in Bangalore by well experienced professionals having more than 10+ years of IT experience. AWS Training in Bangalore |
    AWS Training in Chennai |

    ReplyDelete
  10. Setting up Hadoop to run on Single Node in Ubuntu 15.04, As per the specification I have tried in Windows and executed Well As I being a PMP Certified and I got my PMP Training in Mumbai, I was well experienced with Hadoop projects and also have certain knowledge.

    ReplyDelete
  11. Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  12. Really very good article, If you really want to start good career in Hadoop training in Hyderabad check the social media reviews search for the good quality institutes for your future path.

    ReplyDelete
  13. This comment has been removed by the author.

    ReplyDelete
  14. 6-week summer course in Noida - 6 weeks The summer course plays a crucial role in shaping the career of young aspiring / informatics students. This training has been specifically introduced so that students can become familiar with current industrial culture and industrial needs. Webtrackker technology offers a 6-month training program for students / graduates that includes small and large projects.
    6-week summer course in Noida

    ReplyDelete
  15. A career in vehicle CAD (computer aided design) means you get to apply the cutting-edge computer era to help layout and draft automotive components. In this visual and symbol-based verbal exchange approach, you will be capable of tailor your understanding to the automotive area and to the profession within the car area which you want to go into into. This will ensure a protracted-term, high tech profession route that is now not only applicable today, but for the next day as nicely. The following kinds of individuals may be excellent candidates for a career in auto CAD:

    AutoCAD training in Noida

    AutoCAD training institute in Noida


    Best AutoCAD training institute in Noida


    Company address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Phone: 01204330760, 8802820025
    Email: info@webtrackker.com

    ReplyDelete
  16. Cloud Computing Training In Noida
    Webtrackker is IT based company in many countries. Webtrackker will provide you a real time projects based training on Cloud Computing. If you are looking for the Cloud computing training in Noida then you can join the webtrackker technology.
    Cloud Computing Training In Noida , Cloud Computing Training center In Noida , Cloud Computing Training institute In Noida ,

    Company Address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Email: info@webtrackker.com
    Website: www.webtrackker.com
    http://webtrackker.com/Cloud-Computing-Training-Institutes-In-Noida.php

    ReplyDelete
  17. Video editing course in Noida
    Video editing training institute in Noida- Webtrackker Technology is and IT Training institute providing the Video editing course in Noida, FCP, Final Cut Pro Training in Noida. For more call us- 8802820025.
    Video editing course in Noida
    best video editig course in noida
    best video edtitng institute in noida
    Company Address:
    Webtrackker Technology
    C- 67, Sector- 63, Noida
    Phone: 01204330760, 8802820025
    Email: info@webtrackker.com
    Website: http://webtrackker.com/Best-training-institute-Video-editing-FCP-course-in-Noida.php

    ReplyDelete
  18. hank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me 
    Click here:
    angularjs6 Training in Chennai
    Click here:
    angularjs Training in online

    ReplyDelete
  19. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Click here:
    Microsoft azure training in btm
    Click here:
    Microsoft azure training in rajajinagar

    ReplyDelete
  20. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.
    Devops training in sholinganallur

    ReplyDelete
  21. Very nice post. I always like and such a super contents of these post. Excellent and very cool idea and great content of different kinds of the valuable information's. Big Data Hadoop Training in Pune

    ReplyDelete
  22. Like different elements of India, numerous oil and spices usually cross into making food. This effects in substances getting caught to the partitions of the filter out.
    Visit here
    http://kitchenware.ml
    Best kitchen chimney installation and service
    Auto clean chimney sevice in Faridabad

    ReplyDelete
  23. A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.

    rpa training in electronic-city | rpa training in btm | rpa training in marathahalli | rpa training in pune

    ReplyDelete
  24. After reading this web site I am very satisfied simply because this site is providing comprehensive knowledge for you to audience. Thank you to the perform as well as discuss anything incredibly important in my opinion. We loose time waiting for your next article writing in addition to I beg one to get back to pay a visit to our website in
    java training in chennai | java training in USA

    ReplyDelete
  25. Thank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me
    Data Science course in Chennai | Data science course in bangalore
    Data science course in pune | Data science online course
    Data Science Interview questions and answers | Python course in Kalyan nagar

    ReplyDelete

  26. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    Selenium Interview Questions and Answers

    Best Selenium Training in Chennai | Selenium Training Institute in Chennai | Besant Technologies

    Selenium Training in Bangalore | Best Selenium Training in Bangalore

    Free Selenium Tutorial |Selenium Webdriver Tutorial |For Beginners

    ReplyDelete
  27. Thank you for providing useful information and this is the best article blog for the students.learn Oracle Fusion Financials Online Training.

    Oracle Fusion Financials Online Training

    ReplyDelete
  28. Thank you for sharing such a valuable article with good information containing in this blog.learn Oracle Fusion Technical Online Training.

    Oracle Fusion Technical Online Training

    ReplyDelete
  29. Thanks for sharing valuable information in the article.students can make a good career by learning Oracle Fusion SCM Online Training.

    Oracle Fusion SCM Online Training

    ReplyDelete
  30. Thanks for providing such a great information in the blog and also very helpful to all.learn best Oracle Fusion HCM Online Training.

    Oracle Fusion HCM Online Training

    ReplyDelete
  31. Amazing Article ! I have bookmarked this article page as i received good information from this. All the best for the upcoming articles.

    TekSlate Online Trainings

    ReplyDelete
  32. I also do not understand how to write code. I'm studying this. For this site, I know a lot of code. And if anyone wants to write code.


    สูตรบาคาร่า

    ReplyDelete
  33. Thanks For sharing such a wonderful Blog on RPA. This blog contains so much of data about RPA that anyone who is searching for RPA, its really helpful for them to grab this data from your blog on RPA. Again thank you so much for your blog on RPA.
    Thanks and Regards,
    blue prism training in chennai
    Best blue prism training in chennai
    blue prism training cost in chennai

    ReplyDelete
  34. Great efforts put it to find the list of articles which is very useful to know, Definitely will share the same to other forums.Roles and reponsibilities of hadoop developer | hadoop developer skills Set | hadoop training course fees in chennai | Hadoop Training in Chennai Omr

    ReplyDelete
  35. Thank you for sharing such a nice and interesting blog with us. I have seen that all will say the same thing repeatedly. But in your blog, I had a chance to get some useful and unique information.
    Oracle Fusion Financials Online Training

    ReplyDelete
  36. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well.
    Oracle Fusion Financials Online Training

    ReplyDelete
  37. Very interesting blog Awesome post. your article is really informative and helpful for me and other bloggers too
    Oracle Fusion Financials Online Training

    ReplyDelete
  38. I simply wanted to thank you so much again. I am not sure the things that I might have gone through without the type of hints revealed by you regarding that situation.
    safety course in chennai

    ReplyDelete
  39. Thank you for your guide to with upgrade information about Hadoop
    Hadoop admin Online Training

    ReplyDelete
  40. Thank you for providing useful information and this is the best article blog for the students.learn Oracle Fusion Technical Online Training.

    Oracle Fusion Technical Online Training

    ReplyDelete
  41. Thank you for sharing such a valuable article with good information containing in this blog.learn Oracle Fusion SCM Online Training.

    Oracle Fusion SCM Online Training

    ReplyDelete
  42. Thanks for sharing valuable information in the article.students can make a good career by learning HR Training In Hyderabad

    HR Training In Hyderabad

    ReplyDelete
  43. Thanks for providing such a great information in the blog and also very helpful to all.learn best Core HR Training Hyderabad

    Core HR Training Hyderabad

    ReplyDelete
  44. Thank you for sharing such a valuable article with good information containing in this blog.

    Oracle Fusion HCM Online Training

    ReplyDelete