Using whirr to setup ec2 cluster

2019-09-12 05:35发布

I managed to launch a cluster of 10 nodes on amazon ec2 using Whirr. Now I need to install R and Packages.

This is the command:

whirr run-script --script /home/cloudera/TutorialBreen/config/whirr-ec2/install-r+packages.sh --config /home/cloudera/TutorialBreen/config/whirr-ec2/hadoop-ec2.properties

Unfortunately I get an error because the link to the rmr-package in the .sh-file isn't live anymore. This is the original install-r+packeges.sh file:

sudo yum -y --enablerepo=epel install R R-devel

sudo R --no-save << EOF
install.packages(c('RJSONIO', 'itertools', 'digest', 'Rcpp', 'plyr'), repos="http://cran.revolutionanalytics.com", INSTALL_opts=c('--byte-compile') )
EOF

# if you always like to be up-to-date, you can install the latest version
# of rmr directly from RHadoop's github repository:
#
# branch=master
#
# wget --no-check-certificate https://github.com/RevolutionAnalytics/RHadoop/tarball/$branch -O - | tar zx
# mv RevolutionAnalytics-RHadoop* RHadoop
# sudo R CMD INSTALL --byte-compile RHadoop/rmr/pkg/

# but I'm usually not that adventurous:

wget --no-check-certificate https://github.com/downloads/RevolutionAnalytics/RHadoop/rmr_1.3.1.tar.gz
sudo R CMD INSTALL rmr_1.3.1.tar.gz

sudo su << EOF1 
cat >> /etc/profile <<EOF

export HADOOP_HOME=/usr/lib/hadoop

EOF
EOF1

I modified it and inserted the new link to rmr1.3.1: https://github.com/RevolutionAnalytics/RHadoop/tarball/rmr-1.3.1/RevolutionAnalytics-RHadoop-rmr-1.3.1-0-gfff2743.tar.gz

This is the new .sh-File:

sudo yum -y --enablerepo=epel install R R-devel

sudo R --no-save << EOF
install.packages(c('RJSONIO', 'itertools', 'digest', 'Rcpp', 'plyr'), repos="http://cran.revolutionanalytics.com", INSTALL_opts=c('--byte-compile') )
EOF

# if you always like to be up-to-date, you can install the latest version
# of rmr directly from RHadoop's github repository:
#
# branch=master
#
# wget --no-check-certificate https://github.com/RevolutionAnalytics/RHadoop/tarball/$branch -O - | tar zx
# mv RevolutionAnalytics-RHadoop* RHadoop
# sudo R CMD INSTALL --byte-compile RHadoop/rmr/pkg/

# but I'm usually not that adventurous:

wget --no-check-certificate https://github.com/RevolutionAnalytics/RHadoop/tarball/rmr-1.3.1/RevolutionAnalytics-RHadoop-rmr-1.3.1-0-gfff2743.tar.gz
sudo R CMD INSTALL RevolutionAnalytics-RHadoop-rmr-1.3.1-0-gfff2743.tar.gz

sudo su << EOF1 
cat >> /etc/profile <<EOF

export HADOOP_HOME=/usr/lib/hadoop

EOF
EOF1

Unfortunately it doesn't work either. I get the following error (after pages of output):

    --2012-11-05 10:28:02--  https://github.com/RevolutionAnalytics/RHadoop/tarball/rmr-1.3.1/RevolutionAnalytics-RHadoop-rmr-1.3.1-0-gfff2743.tar.gz
    Resolving github.com... 207.97.227.239
    Connecting to github.com|207.97.227.239|:443... connected.
    WARNING: cannot verify github.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV CA-1':
      Unable to locally verify the issuer's authority.
    HTTP request sent, awaiting response... 302 Found
    Location: https://nodeload.github.com/RevolutionAnalytics/RHadoop/legacy.tar.gz/rmr-1.3.1 [following]
    --2012-11-05 10:28:02--  https://nodeload.github.com/RevolutionAnalytics/RHadoop/legacy.tar.gz/rmr-1.3.1
    Resolving nodeload.github.com... 207.97.227.252
    Connecting to nodeload.github.com|207.97.227.252|:443... connected.
    WARNING: cannot verify nodeload.github.com's certificate, issued by `/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance CA-3':
      Unable to locally verify the issuer's authority.
    HTTP request sent, awaiting response... 200 OK
    Length: 15699365 (15M) [application/x-gzip]
    Saving to: `rmr-1.3.1.4'

100%[======================================>] 15,699,365  14.6M/s   in 1.0s    

2012-11-05 11:31:49 (14.6 MB/s) - `rmr-1.3.1.4' saved [15699365/15699365]

Warning: invalid package 'RevolutionAnalytics-RHadoop-rmr-1.3.1-0-gfff2743.tar.gz'
Error: ERROR: no packages specified

Does someone know what I need to change (maybe in the install-r+packeges.sh file)?

0条回答
登录 后发表回答