7/2/2023 0 Comments Spark vagrant for mac![]() Whats happening above? Each layer is being downloaded for the spark shell from the remote docker image repositories.Īfter a while, you’ll get dumped right into your scala shell where you can run spark commands: You should now see something like this.Į58fb78373de: Downloading 75.04 MB/136.6 MB 3m33s For example, networking went down during the process in my VM, and that gave me some cryptic errors… Also the “dig” command was not found (but that is solved above by yum installing of bind-utils. I had to run this a couple of times to get it to work. deploy/deploy.sh -i amplab/spark:0.8.0 -cĪnd we wait. # vi sudoers to allow “root” to do sudo operations. #Do this as sudo, edit sudoers if you have to. ![]() #The “dig” command is required for spark nameserver setup script # Start up docker before we try to deploy… # We will need this for spark nameserver script # the post also used -blogpost- branch, but that failed for me, so I just used master… just being explicit here. # The above blog has a branch for this spark docker recipe, so switch to it. So lets see if we can pull the bits in from …. Lets do something useful with it… LIKE SETTING UP A SPARK CLUSTER. Okay ! Now we’ve got a centos-6-vbox instance with a docker instance running. Once the box is spun up, run “vagrant ssh”, and become root - su (password is vagrant), and do vagrant]# yum install vagrant]# yum -y install docker-io vagrant]# yum install vagrant]# service docker start nfigure(VAGRANTFILE_API_VERSION) do |config| #Now, modify the configure section in Vagrantfile to look like Jays-MacBook-Pro:sandbox_docker Jpeerindex$ vagrant box listĪlright ! Now we’ve got a centos-6-vbox instance sitting around. You can use “vagran box list” to list all your base OS’s, and see if anyone fits the bill (if not, you can just add a box yourself using ‘vagrant box add’, from ). On my mac… So I need a base linux box to run LXC’s on. PART 1: Spin up a VM that has LXC support. – Finally: We will run the provisioner script which will download all the docker containers and create the spark containers for us. ![]() – Then, we will pull down the github repo containing docker utilities for setting up spark. – Then, we will install docker and do couple of minor things, like start the docker services. Actually, it is a “host” from docker’s perspective. – First, we will spin up a local CENTOS or FEDORA box using “vagrant up”. Vagrant is like the giving tree that keeps on giving even after I abandoned it □ … So here’s how this will work. So SPARK will run in hosted linux containers, which are created via docker, running on a CENTOS or FEDORA VM which is running as a GUEST inside of my Mac OS X box Im on a mac… how can I spin up linux containers on my Mac? The folks at berkeley’s amplab have been maintaining docker recipes.I doubt my laptop can handle that with full VM setups. I want to test larger clusters of 5-10 nodes.Anyways, all the cool folks are using docker nowadays.I love vagrant: but I don’t have the packer chops to create boxes easily (yet).So I might as well go all the way of the deep end and try docker… Normally I’d try this in vagrant, but I’ve already been cheating on vagrant lately with libvirt. Especially since I’m probably going to try to put gluster underneath it :), and so I will want the setup to be hackable. However, given the complexity of maintaining hadoop distros, I figured it would be overkill to spend all day reading the spark documentation. I’ve been wanting to create a reproducible, hackable spark environment for a while now. Using vagrant to spin up your docker instances gives you a completely reproducible way of rapidly spinning up software stacks. Docker, on the other hand, gives you a lightweight framework for running containerized, layered software stacks, by using linux containers. Vagrant lets you reproduce a pure linux box, anywhere, automatically, but it comes at the cost of provisioning a brand new VM, which takes a while.
0 Comments
Leave a Reply. |