Google App Engine Flexible with Datastore and Pyramid

It is possible to run a Pyramid application on Google App Engine. This tutorial is written "environment agnostic", meaning the commands here should work on Linux, macOS or Windows. This tutorial also assumes you've already installed and created a Pyramid application, and that you have a Google App Engine account.

Setup

First we'll need to set up a few things in App Engine. If you don't need Datastore access for your project or any other GCP service, you can skip the Credentials section.

Credentials

Navigate to App Engine's IAM And Admin section and click on Service Accounts in the left sidebar, then create a Service Account.

Once a service account is created, you will be given a .json key file. This will be used to allow your Pyramid application to communicate with GCP services. Move this file to your Pyramid project. A best practice here would be to make sure this file is listed in .gitignore so that it's not checked in with the rest of your code.

Now that we have a service account, we'll need to give it a couple of roles. Click IAM in the left sidebar of IAM And Admin. Find the service account you've just created and click the Edit button. Give this account the Cloud Datastore User role for read/write access. For read-only access, give it Cloud Datastore Viewer.

Project Files

Create the files with content as follows.

1. requirements.txt

   Pyramid
   waitress
   pyramid_debugtoolbar
   pyramid_chameleon
   google-cloud-ndb
   

   If you are not using Datastore, you can exclude google-cloud-ndb.

2. dockerfile

   FROM gcr.io/google-appengine/python
   # Create a virtualenv for dependencies. This isolates these packages from
   # system-level packages.
   # Use -p python3 or -p python3.7 to select python version. Default is version 2.
   RUN virtualenv /env -p python3
   
   # Setting these environment variables are the same as running
   # source /env/bin/activate.
   ENV VIRTUAL_ENV /env
   ENV PATH /env/bin:$PATH
   ENV PYTHONUNBUFFERED 0
   
   # Copy the application's requirements.txt and run pip to install all
   # dependencies into the virtualenv.
   ADD requirements.txt /app/requirements.txt
   ADD my-gcp-key.json /app/my-gcp-key.json
   ENV GOOGLE_APPLICATION_CREDENTIALS /app/my-gcp-key.json
   RUN pip install -r /app/requirements.txt
   
   # Add the application source code.
   ADD . /app
   
   # Run a WSGI server to serve the application. waitress must be declared as
   # a dependency in requirements.txt.
   RUN pip install -e .
   
   CMD pserve production.ini
   

   Replace my-gcp-key.json filename with the JSON file you were provided when you created the Service Account.

3. datastore_tween.py

   from my_project import datastore_client
   
   
   class datastore_tween_factory(object):
       def __init__(self, handler, registry):
           self.handler = handler
           self.registry = registry
   
       def __call__(self, request):
   
           with datastore_client.context():
               response = self.handler(request)
   
           return response
   

4. app.yaml

   runtime: custom
   env: flex
   service: default
   runtime_config:
     python_version: 3.7
   
   manual_scaling:
     instances: 1
   resources:
     cpu: 1
     memory_gb: 0.5
     disk_size_gb: 10
   

   For more details about app.yaml, see app.yaml Reference.

5. __init__.py

   This file should already exist in your project at the root level as it would've been generated by Pyramid's cookiecutters. Add the following line within the main method's config context:

   config.add_tween('my_project.datastore_tween.datastore_tween_factory')
   

   This allows you to communicate with Datastore within every request.

6. production.ini

   Your Pyramid application should already contain both a development.ini and a production.ini. For App Engine to communicate with your application, it will need to be listening on port 8080. Assuming you are using the Waitress WSGI server, modify the listen variable within the server:main block.

   listen = *:8080
   

Now let's assume you have the following model defined somewhere in your code that relates to a Datastore "kind":

from google.cloud import ndb


class Accounts(ndb.Model):

    email = ndb.StringProperty()
    password = ndb.StringProperty()

    def __init__(self, **kwds):
        super(Accounts, self).__init__(**kwds)

You could then query this model within any handler/endpoint like so:

Accounts.query().filter(Accounts.email == user_email).get()

Running locally

Unlike App Engine's Standard environment, we're running Pyramid in a pretty typical fashion. You can run this locally on your machine using the same line in the dockerfile we created earlier as pserve development.ini, or you can run in a Docker container using the same dockerfile that Flexible will be using. No changes need to be made there. This is useful for debugging any issues you may run in to under Flexible, without needing to deploy to it.

Deploying

Using the Google Cloud SDK, deploying is pretty straightforward.

$ gcloud app deploy app.yaml --version my-version --project my-gcp-project

Replace my-version with some kind of identifier so you know what code is deployed. This can pretty much be anything.

Replace my-gcp-project with your App Engine application's ID.

Your Pyramid application is now live to the world! You can access it by navigating to your domain name, by "<applicationid>.appspot.com", or if you've specified a version outside of your default then it would be "<version-dot-applicationid>.appspot.com".