Google App Engine Flexible with Datastore and Pyramid

It is possible to run a Pyramid application on Google App Engine. This tutorial is written "environment agnostic", meaning the commands here should work on Linux, macOS or Windows. This tutorial also assumes you've already installed and created a Pyramid application, and that you have a Google App Engine account.

Setup

First we'll need to set up a few things in App Engine. If you don't need Datastore access for your project or any other GCP service, you can skip the Credentials section.

Credentials

Navigate to App Engine's IAM And Admin section and click on Service Accounts in the left sidebar, then create a Service Account.

Once a service account is created, you will be given a .json key file. This will be used to allow your Pyramid application to communicate with GCP services. Move this file to your Pyramid project. A best practice here would be to make sure this file is listed in .gitignore so that it's not checked in with the rest of your code.

Now that we have a service account, we'll need to give it a couple of roles. Click IAM in the left sidebar of IAM And Admin. Find the service account you've just created and click the Edit button. Give this account the Cloud Datastore User role for read/write access. For read-only access, give it Cloud Datastore Viewer.

Project Files

Create the files with content as follows.

  1. requirements.txt

    1Pyramid
    2waitress
    3pyramid_debugtoolbar
    4pyramid_chameleon
    5google-cloud-ndb
    

    If you are not using Datastore, you can exclude google-cloud-ndb.

  2. dockerfile

     1FROM gcr.io/google-appengine/python
     2# Create a virtualenv for dependencies. This isolates these packages from
     3# system-level packages.
     4# Use -p python3 or -p python3.7 to select python version. Default is version 2.
     5RUN virtualenv /env -p python3
     6
     7# Setting these environment variables are the same as running
     8# source /env/bin/activate.
     9ENV VIRTUAL_ENV /env
    10ENV PATH /env/bin:$PATH
    11ENV PYTHONUNBUFFERED 0
    12
    13# Copy the application's requirements.txt and run pip to install all
    14# dependencies into the virtualenv.
    15ADD requirements.txt /app/requirements.txt
    16ADD my-gcp-key.json /app/my-gcp-key.json
    17ENV GOOGLE_APPLICATION_CREDENTIALS /app/my-gcp-key.json
    18RUN pip install -r /app/requirements.txt
    19
    20# Add the application source code.
    21ADD . /app
    22
    23# Run a WSGI server to serve the application. waitress must be declared as
    24# a dependency in requirements.txt.
    25RUN pip install -e .
    26
    27CMD pserve production.ini
    

    Replace my-gcp-key.json filename with the JSON file you were provided when you created the Service Account.

  3. datastore_tween.py

     1from my_project import datastore_client
     2
     3
     4class datastore_tween_factory(object):
     5    def __init__(self, handler, registry):
     6        self.handler = handler
     7        self.registry = registry
     8
     9    def __call__(self, request):
    10
    11        with datastore_client.context():
    12            response = self.handler(request)
    13
    14        return response
    
  4. app.yaml

     1runtime: custom
     2env: flex
     3service: default
     4runtime_config:
     5  python_version: 3.7
     6
     7manual_scaling:
     8  instances: 1
     9resources:
    10  cpu: 1
    11  memory_gb: 0.5
    12  disk_size_gb: 10
    

    For more details about app.yaml, see app.yaml Reference.

  5. __init__.py

    This file should already exist in your project at the root level as it would've been generated by Pyramid's cookiecutters. Add the following line within the main method's config context:

    config.add_tween('my_project.datastore_tween.datastore_tween_factory')
    

    This allows you to communicate with Datastore within every request.

  6. production.ini

    Your Pyramid application should already contain both a development.ini and a production.ini. For App Engine to communicate with your application, it will need to be listening on port 8080. Assuming you are using the Waitress WSGI server, modify the listen variable within the server:main block.

    listen = *:8080
    

Now let's assume you have the following model defined somewhere in your code that relates to a Datastore "kind":

 1from google.cloud import ndb
 2
 3
 4class Accounts(ndb.Model):
 5
 6    email = ndb.StringProperty()
 7    password = ndb.StringProperty()
 8
 9    def __init__(self, **kwds):
10        super(Accounts, self).__init__(**kwds)

You could then query this model within any handler/endpoint like so:

Accounts.query().filter(Accounts.email == user_email).get()

Running locally

Unlike App Engine's Standard environment, we're running Pyramid in a pretty typical fashion. You can run this locally on your machine using the same line in the dockerfile we created earlier as pserve development.ini, or you can run in a Docker container using the same dockerfile that Flexible will be using. No changes need to be made there. This is useful for debugging any issues you may run in to under Flexible, without needing to deploy to it.

Deploying

Using the Google Cloud SDK, deploying is pretty straightforward.

$ gcloud app deploy app.yaml --version my-version --project my-gcp-project

Replace my-version with some kind of identifier so you know what code is deployed. This can pretty much be anything.

Replace my-gcp-project with your App Engine application's ID.

Your Pyramid application is now live to the world! You can access it by navigating to your domain name, by "<applicationid>.appspot.com", or if you've specified a version outside of your default then it would be "<version-dot-applicationid>.appspot.com".