Apache Spark Installation in 6 Steps

Posted on April 15, 2016
apache-spark-installation-6-steps

I have installed Apache Spark on Ubuntu 14.04. I have gone through many hardships to install this as the installation documentation is not good. Here I am documenting each and every step required to successfully install Apache Spark on Ubuntu.


#1. Run the following command to check if Java is Installed


java -version from terminal. If java is not installed, run the command sudo apt-get install default-jdk


#2. Run the following command to download scala


sudo wget www.scala-lang.org/files/archive/scala-2.11.7.deb


If you are running behind a proxy then run the following command to download scala


sudo wget -e use_proxy=yes -e http_proxy=myproxy.com:myPort www.scala-lang.org/files/archive/scala-2.11.7.deb


If you don't know your proxy settings, run the following command to check your proxy settings.


command set | grep -i proxy


Once you have downloaded scala, install it using the following command.


sudo dpkg -i scala-2.11.7.deb


Once scala is installed verify the installation using the following command


scala -version


#3. Download maven using the following command


sudo wget http://mirror.fibergrid.in/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz


If you are running behind a proxy, use the proxy settings as mentioned in point (2)


Once you have downloaded maven, extract the .tar.gz file using the following command


tar xvf apache-maven-3.3.9-bin.tar.gz


After the command is run, the folder apache-maven-3.3.9-bin will be created. Move this directory to /usr/local directory using the following command.


sudo mv apache-maven-3.3.9-bin /usr/local/


Put the binary in the PATH using the following command


export PATH=$PATH:/usr/local/apache-maven-3.3.9-bin/bin


#4. Download Apache Spark 1.6.1 using the following command


sudo wget http://a.mbbsindia.com/spark/spark-1.6.1/spark-1.6.1.tgz


Once you have downloaded Spark, extract the .tar.gz file using the following command


tar xvf spark-1.6.1.tgz


After the command is run, the folder spark-1.6.1 will be created. Move this directory to /usr/local directory using the following command.


sudo mv spark-1.6.1 /usr/local/


Put the binary in the PATH using the following command


export PATH=$PATH:/usr/local/spark-1.6.1/bin


#5. Go to the directory /usr/local/spark-1.6.1 using the following command


cd /usr/local/spark-1.6.1


Now, run the following command to install Apache Spark


mvn -DskipTests clean package


If you are running behind a proxy, then use the following command instead.


mvn -DproxySet=true -DproxyHost=myproxy.com -DproxyPort=myPort -DskipTests clean package


#6. After Apache Spark is installed, verify the installation by running the following command


spark-shell


You should see the following screen







Sharing is Caring!

GET FREE UPDATES


RECOMMENDED POSTS FOR YOU


profile image

Kaushik Baruah

Research Engineer @XRCI


ABOUT

My name is Kaushik Baruah and I am the chief blogger on this Blog and here I like to share my experience as software engineer and research engineer with my online readers. I will try to focus on career planning, latest emerging technologies and tutorials on various computer science subjects. You can follow me on Twitter, Facebook and Google+

GET FREE UPDATES

POPULAR POSTS

Copyright © 2016
About Us

My name is Kaushik Baruah and I am the chief blogger on this Blog and here I like to share my experience as software engineer and research engineer with my online readers. I will try to focus on career planning, latest emerging technologies and tutorials on various computer science subjects.

Get Free Updates