This is a faithful narrative of my dealings deploying a python/flask backend with terraform and AWS’s Lambda.
I set up my python virtual environment with python3.11 and worked with the library psycopg2, a popular postgreSQL database adapter for Python, to interact (connect, perform SQL queries, database operations) with my postgreSQL database on elephantSQL.
When the backend was ready to be deployed, I used terraform to create an AWS Lambda function that would include the application’s code and a Lambda layer that would contain the application’s dependencies. The application was running without any problems locally and everything seemed to be going well with creating the Lambda and the Lambda layer until I went to test it. When I invoked the Lambda function via the AWS Console to test that everything was working I received an error message No module named 'psycopg2._psycopg': ModuleNotFoundError. I did some research to understand why even though pscopg2 was among the dependencies included in the Lambda layer the Lambda function was unable to find it. The error came down to an incompatibility in the machine’s instruction set that psycopg2 needed for local development versus deploying in AWS.
The psycopg2 library that I was running locally had an instruction set that is compatible with an arm64 architecture, which is what an Apple Silicon M1 chip needs, so this worked great in my laptop but aws expected the set of instructions to be in x86_64 architecture for compatibility with aws’s system. To check your machine’s instruction set go to your Python virtual environment and under lib/python3.9/site-packages you’ll find psycopg2. The file you are looking for ends in .soso for my local set up this file was called _psycopg.cpython-39-darwin.so
Looking up the error No module named 'psycopg2._psycopg': ModuleNotFoundErrorthe following thread https://www.reddit.com/r/aws/comments/1034r3h/psycopg2_not_working_with_lambda/ led to a stack overflow article https://stackoverflow.com/questions/71493439/unable-to-import-module-lambda-function-no-module-named-psycopg2-psycopg-aw that had a suggestion for how to address this challenge. One of the suggested solutions was to install the library aws-psycopg2. This suggestion led to checking out the following Github repo https://github.com/jkehler/awslambda-psycopg2. This repo is a “custom compiled psycopg2 C library for Python. Due to AWS Lambda missing the required PostgreSQL libraries in the AMI image, I needed to compile psycopg2 with the PostgreSQL libpq.so library statically linked libpq library instead of the default dynamic link.” They also provide instructions on how to compile this package from scratch. In their instructions they said “Just copy the psycopg2-3.6 directory into your AWS Lambda project and rename it to psycopg2 before creating your AWS Lambda zip package.” They included directories that were compatible with Python 3.6 to Python 3.9.
To be able to use the library I changed the version of Python I was using from 3.11 to 3.9. I made sure to also change the runtime for Python in terraform file to create a lambda function and a lambda layer. The key modification was to the bash build script. Instead of creating a zip file that included the local psycopg2 library in the dependencies, the instructions were modified to replace the local psycopg2 library with a version of the psycopg2 library that is compatible with the expected architecture, x86_64.
Before creating your bash script, make sure you clone the repo from https://github.com/jkehler/awslambda-psycopg2 into your local machine. Then add the following instructions in your script. Replace names with the correct file names where appropriate. Same with the file paths.
#!/bin/bash
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo "$SCRIPT_DIR"
# p="$1" # First argument "archive prefix directory name"
# d="$2" # Second argument "source directory name"
# z="$3" # Third argument "result ZIP archive name without .zip"
# [ -e "$p" -o -e "${z}.zip" ] && return "Prefix directory name: $p Or target zip file name: ${z}.zip exist in PWD, cannot continue"
tmp="$(mktemp)"
echo "Creating temporary folder: $tmp"
echo "Creating temporary 'es-flask-app/python' directory where all Python deps will be placed."
gtar cf "$tmp" -C "$SCRIPT_DIR/../.venv/lib/python3.9/site-packages" --transform "s|^|python/|" "."
gtar xf "$tmp"
## BEFORE YOU ZIP THIS FILE AND UPLOAD TO AWS
# When you are running the estilo-calico app locally, you'll need a version of psycopg2
# that is compatible with the arm64 architecture that Mac machines use (assuming you're on a Mac laptop).
#
# However, in the cloud, AWS needs a version of psycopg2 that is compatible with x86_64 architectures,
# (Intel chips have x86_64 instruction sets).
#
# So the problem is, we need to run one version of the app on a Mac, and we need to build and upload
# a different version to the cloud. So in our build script, we will have to override the pip installed
# psycopg binaries before we push to AWS.
echo "Removing psycopg2 binary and replacing with AWS-compatible psycopg2"
rm -r "$SCRIPT_DIR/../python/psycopg2"
cp -r ~/code/awslambda-psycopg2/psycopg2-3.9 "$SCRIPT_DIR/../python/psycopg2/"
echo "Zipping python folder to create lambda layer archive"
zip -rq "$SCRIPT_DIR/../estilo-calico-deps-layer.zip" "python"
# Cleanup
echo "Cleaning up - removing temporary folders"
rm "$tmp"
rm -r "python"
Here is a good summary of what the SCRIPT_DIR command is doing. From a very good description I found here https://stackoverflow.com/questions/39340169/dir-cd-dirname-bash-source0-pwd-how-does-that-work. “In summary, that command gets the script's source file pathname, strips it to just the path portion, cds to that path, then uses pwd to return the (effectively) full path of the script. This is assigned to DIR. After all of that, the context is unwound so you end up back in the directory you started at but with an environment variable DIR containing the script's path.” From the same answer this is helpful:
Bash maintains a number of variables including BASH_SOURCE which is an array of source file pathnames.
${} acts as a kind of quoting for variables.
$() acts as a kind of quoting for commands but they're run in their own context.
dirname gives you the path portion of the provided argument.
cd changes the current directory.
pwd gives the current path.
&& is a logical and but is used in this instance for its side effect of running commands one after another.
Create a temporary folder called tmp
Create a file tmp and change directory per -C flag. Include file source and destination with the --transform flag. Create a directory called python in your current directory.
gtar cf "$tmp" -C "$SCRIPT_DIR/../.venv/lib/python3.9/site-packages" --transform "s|^|python/|" "."
Extract contents of tmp file with gtar xf "$tmp"