even though aws-cli is not available by default in AWS Lambda
The AWS Lambda environments for each programming language (e.g.,
Python, Node, Java) already have the AWS client SDK packages
pre-installed for those languages. For example, the Python AWS Lambda
environment has boto3 available, which is ideal for connecting to and
using AWS services in your function.
This makes it easy to use AWS Lambda as the glue for AWS. A function
can be triggered by many different service events, and can respond by
reading from, storing to, and triggering other services in the AWS
ecosystem.
However, there are a few things that aws-cli currently does better
than the AWS SDKs alone. For example, the following command is an
efficient way to take the files in a local directory and recursively
update a website bucket, uploading (in parallel) files that have
changed, while setting important object attributes including MIME
types guessing:
The aws-cli software is not currently pre-installed in the AWS Lambda
environment, but we can fix that with a little effort.
Background
The key to solving this is to remember that aws-cli is available as a
Python package. Mitch Garnaat reminded me of this when I was lamenting
the lack of aws-cli in AWS Lambda, causing me to smack my virtual
forehead. Amazon has already taught us how to install most Python
packages, and we can apply the same process for aws-cli,
though a little extra work is required, because a command line program
is involved.
NodeJS/Java/Go developers: Don’t stop reading! We are using Python
to install aws-cli, true, but this is a command line program. Once the
command is installed in the AWS Lambda environment, you can invoke it
using the system command running functions in your respective
languages.
Steps
Here are the steps I followed to add aws-cli to my AWS Lambda
function. Adjust to suit your particular preferred way of building AWS
Lambda functions.
Create a temporary directory to work in, including paths for a
temporary virtualenv, and an output ZIP file:
Create the virtualenv and install the aws-cli Python package into it
using a subshell:
Copy the aws command file into the ZIP file, but adjust the first
(shabang) line so that it will run with the system python command in
the AWS Lambda environment, instead of assuming python is in the
virtualenv on our local system. This is the valuable nugget of
information buried deep in this article!
Copy the Python packages required for aws-cli into the ZIP file:
Copy in your AWS Lambda function, other packages, configuration, and
other files needed by the function code. These don’t need to be in
Python.
Upload the ZIP file to S3 (or directly to AWS Lambda) and clean up:
In your Lambda function, you can invoke aws-cli commands. For example,
in Python, you might use:
Note that you will need to specify the location of the aws command
with a leading "./" or you could add /var/task (cwd) to the $PATH
environment variable.
Example
This approach is used to add the aws-cli command to the AWS Lambda
function used by the AWS Git-backed Static Website
CloudFormation stack.
You can see the code that builds the AWS Lambda function ZIP file
here, including the installation of the aws-cli command:
https://github.com/alestic/aws-git-backed-static-website/blob/master/build-upload-aws-lambda-function
Notes
I would still love to see aws-cli pre-installed on all the AWS
Lambda environments. This simple change would remove quite a bit of
setup complexity and would even let me drop my AWS Lambda function
inline in the CloudFormation template. Eliminating the external
dependency and having everything in one file would be huge!
I had success building awscli on Ubuntu for use in AWS Lambda,
probably because all of the package requirements are pure
Python. This approach does not always work. It is recommended you
build packages on Amazon Linux so that they are compatible with the
AWS Lambda environment.
The pip install -t DIR approach did not work for aws-cli when I
tried it, which is why I went with virtualenv. Tips welcomed.
I am not an expert at virtualenv or Python, but I am persistent when
I want to figure out how to get things to work. The above approach
worked. I welcome improvements and suggestions from the experts.
Original article and comments: https://alestic.com/2016/11/aws-lambda-awscli/