Monday, November 7, 2016

Setting up Hadoop to run on Single Node in Ubuntu 15.04

This is tested on hadoop-2.7.3.

Improvement on Hadoop documentation : http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html

Step 1 

Make sure Java is installed

Installation instruction : http://suhothayan.blogspot.com/2010/02/how-to-set-javahome-in-ubuntu.html

Step 2

Install pre-requisites

$ sudo apt-get install ssh
$ sudo apt-get install rsync

Step 3

Setup Hadoop

$ gedit hadoop-2.7.3/etc/hadoop/core-site.xml

Add (replace {user-name} with system username, E.g "foo" for /home/foo/)

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <property>
<name>hadoop.proxyuser.{user-name}.groups</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.{user-name}.hosts</name>
        <value>*</value>
    </property>
</configuration>

$ gedit hadoop-2.7.3/etc/hadoop/hdfs-site.xml 

Add 

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Step 4

Run

$ ssh localhost 

If it requested for password, run:

$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys

Try ssh localhost again.
If it still asks for password, run following and try again:

$ ssh-keygen -t rsa
#Press enter for each line
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ chmod og-wx ~/.ssh/authorized_keys 

Step 5

Clean namenode

$ ./hadoop-2.7.3/bin/hdfs namenode -format

Step 6 * Not provided in Hadoop Documentation 

Replace ${JAVA_HOME} with hardcoded path in hadoop-env.sh

gedit hadoop-2.7.3/etc/hadoop/hadoop-env.sh

Edit the file as 

# The java implementation to use.
export JAVA_HOME={path}/jdk1.8.0_111

Step 7

Start Hadoop 

$ ./hadoop-2.7.3/sbin/start-all.sh

The Hadoop daemon log output is written to the $HADOOP_LOG_DIR directory (defaults to $HADOOP_HOME/logs).

Browse the web interface for the NameNode;

http://localhost:50070/

Step 8

Check processors running by running:

$ jps

Output: 

xxxxx NameNode
xxxxx ResourceManager
xxxxx DataNode
xxxxx NodeManager
xxxxx SecondaryNameNode

Step 9

Make HDFS directories for MapReduce jobs:

$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user
$ ./hadoop-2.7.3/bin/hdfs dfs -mkdir /user/{user-name}


56 comments:

  1. Thanks for sharing the information very useful info about Hadoop and

    keep updating us, Please........

    ReplyDelete
    Replies
    1. Computer Repair Service Madurai IlamanurMadurai
      https://123just.com/ad/67/computer-repair-service-in-madurai

      Delete
  2. Great post! I am actually getting ready to across this information, is very helpful my friend. Also great blog here with all of the valuable information you have Keep up the good work you are doing here.Well, got a good knowledge.

    Hadoop Training in Chennai

    Dot Net Training in Chennai

    ReplyDelete
  3. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon.

    Base SAS Training in Chennai

    MSBI Training in Chennai

    ReplyDelete
  4. Besant Technologies in Bangalore is the BEST AWS training institute in Bangalore which offers complete AWS training in Bangalore by well experienced professionals having more than 10+ years of IT experience. AWS Training in Bangalore |
    AWS Training in Chennai |

    ReplyDelete
  5. Setting up Hadoop to run on Single Node in Ubuntu 15.04, As per the specification I have tried in Windows and executed Well As I being a PMP Certified and I got my PMP Training in Mumbai, I was well experienced with Hadoop projects and also have certain knowledge.

    ReplyDelete
  6. Really very good article, If you really want to start good career in Hadoop training in Hyderabad check the social media reviews search for the good quality institutes for your future path.

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. hank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me 
    Click here:
    angularjs6 Training in Chennai
    Click here:
    angularjs Training in online

    ReplyDelete
  9. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    Click here:
    Microsoft azure training in btm
    Click here:
    Microsoft azure training in rajajinagar

    ReplyDelete
  10. Very nice post. I always like and such a super contents of these post. Excellent and very cool idea and great content of different kinds of the valuable information's. Big Data Hadoop Training in Pune

    ReplyDelete
  11. After reading this web site I am very satisfied simply because this site is providing comprehensive knowledge for you to audience. Thank you to the perform as well as discuss anything incredibly important in my opinion. We loose time waiting for your next article writing in addition to I beg one to get back to pay a visit to our website in
    java training in chennai | java training in USA

    ReplyDelete
  12. Thank you for benefiting from time to focus on this kind of, I feel firmly about it and also really like comprehending far more with this particular subject matter. In case doable, when you get know-how, is it possible to thoughts modernizing your site together with far more details? It’s extremely useful to me
    Data Science course in Chennai | Data science course in bangalore
    Data science course in pune | Data science online course
    Data Science Interview questions and answers | Python course in Kalyan nagar

    ReplyDelete

  13. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    Selenium Interview Questions and Answers

    Best Selenium Training in Chennai | Selenium Training Institute in Chennai | Besant Technologies

    Selenium Training in Bangalore | Best Selenium Training in Bangalore

    Free Selenium Tutorial |Selenium Webdriver Tutorial |For Beginners

    ReplyDelete
  14. Thank you for sharing such a valuable article with good information containing in this blog.learn Oracle Fusion Technical Online Training.

    Oracle Fusion Technical Online Training

    ReplyDelete
  15. Amazing Article ! I have bookmarked this article page as i received good information from this. All the best for the upcoming articles.

    TekSlate Online Trainings

    ReplyDelete
  16. I also do not understand how to write code. I'm studying this. For this site, I know a lot of code. And if anyone wants to write code.


    สูตรบาคาร่า

    ReplyDelete
  17. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well.
    Oracle Fusion Financials Online Training

    ReplyDelete
  18. Thank you for sharing such a valuable article with good information containing in this blog.

    Oracle Fusion HCM Online Training

    ReplyDelete
  19. Great post.Thank u for sharing the nice information and good content.

    http://www.rstrainings.com/hadoop-online-training.html

    ReplyDelete
  20. Some us know all relating to the compelling medium you present powerful steps on this blog and therefore strongly encourage
    contribution from other ones on this subject while our own child is truly discovering a great deal.
    Have fun with the remaining portion of the year.
    Selenium training in bangalore
    Selenium training in Chennai
    Selenium training in Bangalore
    Selenium training in Pune
    Selenium Online training

    ReplyDelete
  21. https://classifieds.singaporeexpats.com/showproduct.php/product/308023

    ReplyDelete
  22. Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts
    Microsoft Azure online training
    Selenium online training
    Java online training
    Python online training
    uipath online training

    ReplyDelete
  23. Amazing web journal I visit this blog it's extremely marvelous. Interestingly, in this blog content composed plainly and reasonable. The substance of data is educational.

    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  24. Thanks For Sharing The Information The Information Shared Is Very Valuable Please Keep Updating

    Us Time Just Went On Reading The article Hadoop Online Course

    ReplyDelete
  25. Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this. Also Checkout:CEH Training In Hyderabad

    ReplyDelete
  26. Are you looking for Distance Learning Courses in India most of the students choose and apply, Talentedgenex there are many popular courses which attract the students for having distance education. For more info visit this site:- Distance learning courses in India ,

    ReplyDelete
  27. Talentedgenext Way of Online Learning, Distance Education, is an increasing number of becoming popular all over the world due as it has many benefits. For further details visit in this site:- Distance Education Website,

    ReplyDelete
  28. Bachelor of Business Administration is three year UG course, if you are searching for BBA Distance Education in India then you can contact us Talentedgenext. To know more, visit:
    BBA Distance Education ,

    ReplyDelete
  29. wow great blog!Thanks for giving this valuable information about hadoop and it is very helpful to beginers who are choosing this domain please keep blogging..
    Tableau Training In hyderabad
    Hadoop Training In Hyderabad

    ReplyDelete
  30. Really nice post. Provided a helpful information. I hope that you will post more updates like this

    AWS Online Training

    AI Training

    Big Data Training

    ReplyDelete
  31. Very useful information for the blog
    Yaaron Studios is one of the rapidly growing editing studios in Hyderabad. We are the best Video Editing services in Hyderabad. We provides best graphic works like logo reveals, corporate presentation Etc. And also we gives the best Outdoor/Indoor shoots and Ad Making services.
    video editors studio in hyderabad
    short film editors in hyderabad
    corporate video editing studio in hyderabad
    ad making company in hyderabad

    ReplyDelete


  32. Thanks for sharing such useful information.The content is high quality and results-oriented.Keep sharing more
    Best python certification course in Bangalore

    ReplyDelete
  33. Thank you for this post. Thats all I are able to say. You most absolutely have built this blog website into something speciel. You clearly know what you are working on, youve insured so many corners. Thanks

    Data Science Training in Hyderabad

    ReplyDelete
  34. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    oracle training in bangalore

    hadoop training in bangalore
    oracle training

    oracle online training

    oracle training in hyderabad

    hadoop training in chennai

    ReplyDelete
  35. Good Post! , it was so good to read and useful to improve my knowledge as an updated one, keep blogging. After seeing your article I want to say that also a well-written article with some very good information which is very useful for the readers....thanks for sharing it and do share more posts like this.

    Bigdata Hadoop Training in Gurgaon
    Spark Training in Gurgaon

    ReplyDelete
  36. Excellent blog!!! I got to know more useful information by reading your blog. Thanks for posting this blog.

    R Programming Training in Bangalore
    apache spark online course

    ReplyDelete
  37. Great and really helpful article! Adding to the conversation, providing more information, or expressing a new point of view...Nice information and updates.
    Best Hadoop Training Institutes

    ReplyDelete
  38. Very Nice Post really expalined good information and Please keep updating us..... Thanks





    ReplyDelete
  39. Howdy! I know this is somewhat off topic but I was wondering which blog platform are you using for this website?

    MLSU BA 1st Year Result
    MLSU BA 2nd Year Result
    MLSU BA 3rd Year Result

    ReplyDelete