When you need an AWS command line toolset not provided by Ubuntu
packages, you can download the tools directly from Amazon and install
them locally.
In a previous article I provided instructions on how to install AWS
command line tools using Ubuntu packages. That method is
slightly easier to set up and easier to upgrade when Ubuntu releases
updates. However, the Ubuntu packages aren’t always up to date with
the latest from Amazon and there are not yet Ubuntu packages published
for every AWS command line tools you might want to use.
Unfortunately, Amazon does not have one single place where you can
download all the command line tools for the various services, nor are
all of the tools installed in the same way, nor do they all use the
same format for accessing the AWS credentials.
The following steps show how I install and configure the AWS command
line tools provided by Amazon when I don’t use the packages provided
by Ubuntu.
Prerequisites
Install required software packages:
sudo apt-get update
sudo apt-get install -y openjdk-6-jre ruby1.8-full rubygems
libxml2-utils libxml2-dev libxslt-dev
unzip cpanminus build-essential
sudo gem install uuidtools json httparty nokogiri
Create a directory where all AWS tools will be installed:
sudo mkdir -p /usr/local/aws
Now we’re ready to start downloading and installing all of the
individual software bundles that Amazon has released and made
available in scattered places on their web site and various S3
buckets.
Download and Install AWS Command Line Tools
These steps should be done from an empty temporary directory so you
can afterwards clean up all of the downloaded and unpacked files.
Note: Some of these download URLs always get the latest version and
some tools have different URLs every time a new version is released.
Click through on the tool link to find the latest [Download] URL.
EC2 API command line tools:
wget –quiet http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip -qq ec2-api-tools.zip
sudo rsync -a –no-o –no-g ec2-api-tools-*/ /usr/local/aws/ec2/
EC2 AMI command line tools:
wget –quiet http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip -qq ec2-ami-tools.zip
sudo rsync -a –no-o –no-g ec2-ami-tools-*/ /usr/local/aws/ec2/
IAM (Identity and Access Management) commmand line tools:
wget –quiet http://awsiammedia.s3.amazonaws.com/public/tools/cli/latest/IAMCli.zip
unzip -qq IAMCli.zip
sudo rsync -a –no-o –no-g IAMCli-*/ /usr/local/aws/iam/
RDS (Relational Database Service) command line tools:
wget –quiet http://s3.amazonaws.com/rds-downloads/RDSCli.zip
unzip -qq RDSCli.zip
sudo rsync -a –no-o –no-g RDSCli-*/ /usr/local/aws/rds/
ELB (Elastic Load Balancer) command line tools:
wget –quiet http://ec2-downloads.s3.amazonaws.com/ElasticLoadBalancing.zip
unzip -qq ElasticLoadBalancing.zip
sudo rsync -a –no-o –no-g ElasticLoadBalancing-*/ /usr/local/aws/elb/
AWS CloudFormation command line tools:
wget –quiet https://s3.amazonaws.com/cloudformation-cli/AWSCloudFormation-cli.zip
unzip -qq AWSCloudFormation-cli.zip
sudo rsync -a –no-o –no-g AWSCloudFormation-*/ /usr/local/aws/cfn/
Auto Scaling command line tools:
wget –quiet http://ec2-downloads.s3.amazonaws.com/AutoScaling-2011-01-01.zip
unzip -qq AutoScaling-*.zip
sudo rsync -a –no-o –no-g AutoScaling-*/ /usr/local/aws/as/
AWS Import/Export command line tools:
wget –quiet http://awsimportexport.s3.amazonaws.com/importexport-webservice-tool.zip
sudo mkdir /usr/local/aws/importexport
sudo unzip -qq importexport-webservice-tool.zip -d /usr/local/aws/importexport
CloudSearch command line tools:
wget –quiet http://s3.amazonaws.com/amazon-cloudsearch-data/cloud-search-tools-1.0.0.1-2012.03.05.tar.gz
tar xzf cloud-search-tools*.tar.gz
sudo rsync -a –no-o –no-g cloud-search-tools-*/ /usr/local/aws/cloudsearch/
CloudWatch command line tools:
wget –quiet http://ec2-downloads.s3.amazonaws.com/CloudWatch-2010-08-01.zip
unzip -qq CloudWatch-*.zip
sudo rsync -a –no-o –no-g CloudWatch-*/ /usr/local/aws/cloudwatch/
ElastiCache command line tools:
wget –quiet https://s3.amazonaws.com/elasticache-downloads/AmazonElastiCacheCli-2012-03-09-1.6.001.zip
unzip -qq AmazonElastiCacheCli-*.zip
sudo rsync -a –no-o –no-g AmazonElastiCacheCli-*/ /usr/local/aws/elasticache/
Elastic Beanstalk command line tools:
wget –quiet https://s3.amazonaws.com/elasticbeanstalk/cli/AWS-ElasticBeanstalk-CLI-2.1.zip
unzip -qq AWS-ElasticBeanstalk-CLI-*.zip
sudo rsync -a –no-o –no-g AWS-ElasticBeanstalk-CLI-*/ /usr/local/aws/elasticbeanstalk/
Elastic MapReduce command line tools:
wget –quiet http://elasticmapreduce.s3.amazonaws.com/elastic-mapreduce-ruby.zip
unzip -qq -d elastic-mapreduce-ruby elastic-mapreduce-ruby.zip
sudo rsync -a –no-o –no-g elastic-mapreduce-ruby/ /usr/local/aws/elasticmapreduce/
Simple Notification Serivice (SNS) command line tools:
wget –quiet http://sns-public-resources.s3.amazonaws.com/SimpleNotificationServiceCli-2010-03-31.zip
unzip -qq SimpleNotificationServiceCli-*.zip
sudo rsync -a –no-o –no-g SimpleNotificationServiceCli-*/ /usr/local/aws/sns/
sudo chmod 755 /usr/local/aws/sns/bin/*
Route 53 (DNS) command line tools:
sudo mkdir -p /usr/local/aws/route53/bin
for i in dnscurl.pl route53tobind.pl bindtoroute53.pl route53zone.pl; do
sudo wget –quiet –directory-prefix=/usr/local/aws/route53/bin
http://awsmedia.s3.amazonaws.com/catalog/attachments/$i
sudo chmod +x /usr/local/aws/route53/bin/$i
done
cpanm –sudo –notest –quiet Net::DNS::ZoneFile NetAddr::IP
Net::DNS Net::IP Digest::HMAC Digest::SHA1 Digest::MD5
CloudFront command line tool:
sudo wget –quiet –directory-prefix=/usr/local/aws/cloudfront/bin
http://d1nqj4pxyrfw2.cloudfront.net/cfcurl.pl
sudo chmod +x /usr/local/aws/cloudfront/bin/cfcurl.pl
S3 command line tools:
wget –quiet http://s3.amazonaws.com/doc/s3-example-code/s3-curl.zip
unzip -qq s3-curl.zip
sudo mkdir -p /usr/local/aws/s3/bin/
sudo rsync -a –no-o –no-g s3-curl/ /usr/local/aws/s3/bin/
sudo chmod 755 /usr/local/aws/s3/bin/s3curl.pl
AWS Data Pipeline command line tools:
wget –quiet https://s3.amazonaws.com/datapipeline-us-east-1/software/latest/DataPipelineCLI/datapipeline-cli.zip
unzip -qq datapipeline-cli.zip
sudo rsync -a –no-o –no-g datapipeline-cli/ /usr/local/aws/datapipeline/
Now that we have all of the software installed under /usr/local/aws we
need to set up the AWS credentials and point the tools to where they
can find everything.
Set up AWS Credentials and Envronment
Create a place to store the secret AWS credentials:
mkdir -m 0700 $HOME/.aws-default/
Copy your AWS X.509 certificate and private key to
this subdirectory. These files will have names that look something
like this:
$HOME/.aws-default/cert-7KX4CVWWQ52YM2SUCIGGHTPDNDZQMVEF.pem
$HOME/.aws-default/pk-7KX4CVWWQ52YM2SUCIGGHTPDNDZQMVEF.pem
Create the file $HOME/.aws-default/aws-credential-file.txt with your AWS
access key id and secret access key in the following
format:
AWSAccessKeyId=
AWSSecretKey=
Create the file $HOME/.aws-default/aws-credentials.json in the following
format:
{
“access-id”: “”,
“private-key”: “”,
“key-pair”: “”,
“key-pair-file”: “”,
“region”: “”,
“use-ssl”: “true”,
“log-uri”: “s3://yourbucket/datapipelinelogs”
}
Create the file $HOME/.aws-secrets in the following format:
%awsSecretAccessKeys = (
‘default’ => {
id => ”,
key => ”,
},
);
Create a symbolic link for s3curl to find its hardcoded config file
and secure the file permissions
ln -s $HOME/.aws-secrets $HOME/.s3curl
chmod 600 $HOME/.aws-default/* $HOME/.aws-secrets
Add the following lines to your $HOME/.bashrc file so that the AWS
command line tools know where to find themselves and the credentials.
We put the new directories in the front of the $PATH so that we run
these instead of any similar tools installed by Ubuntu packages.
export JAVA_HOME=/usr
export EC2_HOME=/usr/local/aws/ec2
export AWS_IAM_HOME=/usr/local/aws/iam
export AWS_RDS_HOME=/usr/local/aws/rds
export AWS_ELB_HOME=/usr/local/aws/elb
export AWS_CLOUDFORMATION_HOME=/usr/local/aws/cfn
export AWS_AUTO_SCALING_HOME=/usr/local/aws/as
export CS_HOME=/usr/local/aws/cloudsearch
export AWS_CLOUDWATCH_HOME=/usr/local/aws/cloudwatch
export AWS_ELASTICACHE_HOME=/usr/local/aws/elasticache
export AWS_SNS_HOME=/usr/local/aws/sns
export AWS_ROUTE53_HOME=/usr/local/aws/route53
export AWS_CLOUDFRONT_HOME=/usr/local/aws/cloudfront
for i in $EC2_HOME $AWS_IAM_HOME $AWS_RDS_HOME $AWS_ELB_HOME
$AWS_CLOUDFORMATION_HOME $AWS_AUTO_SCALING_HOME $CS_HOME
$AWS_CLOUDWATCH_HOME $AWS_ELASTICACHE_HOME $AWS_SNS_HOME
$AWS_ROUTE53_HOME $AWS_CLOUDFRONT_HOME /usr/local/aws/s3
do
PATH=$i/bin:$PATH
done
PATH=/usr/local/aws/elasticbeanstalk/eb/linux/python2.7:$PATH
PATH=/usr/local/aws/elasticmapreduce:$PATH
PATH=/usr/local/aws/datapipeline:$PATH
export EC2_PRIVATE_KEY=$(echo $HOME/.aws-default/pk-*.pem)
export EC2_CERT=$(echo $HOME/.aws-default/cert-*.pem)
export AWS_CREDENTIAL_FILE=$HOME/.aws-default/aws-credential-file.txt
export ELASTIC_MAPREDUCE_CREDENTIALS=$HOME/.aws-default/aws-credentials.json
export DATA_PIPELINE_CREDENTIALS=$HOME/.aws-default/aws-credentials.json
Set everything up in your current shell:
source $HOME/.bashrc
Test
Make sure that the command line tools are installed and have
credentials set up correctly. These commands should not return
errors:
ec2-describe-regions
ec2-ami-tools-version
iam-accountgetsummary
rds-describe-db-engine-versions
elb-describe-lb-policies
cfn-list-stacks
cs-describe-domain
mon-version
elasticache-describe-cache-clusters
eb –version
elastic-mapreduce –list –all
sns-list-topics
dnscurl.pl –keyname default https://route53.amazonaws.com/2010-10-01/hostedzone | xmllint –format –
cfcurl.pl –keyname default https://cloudfront.amazonaws.com/2008-06-30/distribution | xmllint –format –
s3curl.pl –id default http://s3.amazonaws.com/ | xmllint –format –
datapipeline –list-pipelines
Are you aware of any other command line tools provided by Amazon?
Let other readers know in the comments on this article.
[Update 2012-09-06: New URL for ElastiCache tools. Thanks iknewitalready]
[Upate 2012-12-21: Added AWS Data Pipeline command line tools. May break Elastic MapReduce due to Ruby version conflict.]
Original article:
http://alestic.com/2012/09/aws-command-line-tools
Source: Planet Ubuntu