6: Storing Resources In ZODB

Store and retrieve resource tree containers and items in a database.

Background

We now have a resource tree that can go infinitely deep, adding items and subcontainers along the way. We obviously need a database, one that can support hierarchies. ZODB is a transaction-based Python database that supports transparent persistence. We will modify our application to work with the ZODB.

Along the way we will add the use of pyramid_tm, a system for adding transaction awareness to our code. With this we don't need to manually manage our transaction begin/commit cycles in our application code. Instead, transactions are setup transparently on request/response boundaries, outside our application code.

Objectives

  • Create a CRUD app that adds records to persistent storage.

  • Setup pyramid_tm and pyramid_zodbconn.

  • Make our "content" classes inherit from Persistent.

  • Set up a database connection string in our application.

  • Set up a root factory that serves the root from ZODB rather than from memory.

Steps

  1. We are going to use the previous step as our starting point:

    $ cd ..; cp -r addcontent zodb; cd zodb
    
  2. Introduce some new dependencies in zodb/setup.py:

     1from setuptools import setup
     2
     3requires = [
     4    'pyramid',
     5    'pyramid_jinja2',
     6    'ZODB3',
     7    'pyramid_zodbconn',
     8    'pyramid_tm',
     9    'pyramid_debugtoolbar'
    10]
    11
    12setup(name='tutorial',
    13      install_requires=requires,
    14      entry_points="""\
    15      [paste.app_factory]
    16      main = tutorial:main
    17      """,
    18)
    
  3. We can now install our project:

    $ $VENV/bin/python setup.py develop
    
  4. Modify our zodb/development.ini to include some configuration and give database connection parameters:

     1[app:main]
     2use = egg:tutorial
     3pyramid.reload_templates = true
     4pyramid.includes =
     5    pyramid_debugtoolbar
     6    pyramid_zodbconn
     7    pyramid_tm
     8zodbconn.uri = file://%(here)s/Data.fs?connection_cache_size=20000
     9
    10[server:main]
    11use = egg:pyramid#wsgiref
    12host = 0.0.0.0
    13port = 6543
    14
    15# Begin logging configuration
    16
    17[loggers]
    18keys = root, tutorial
    19
    20[logger_tutorial]
    21level = DEBUG
    22handlers =
    23qualname = tutorial
    24
    25[handlers]
    26keys = console
    27
    28[formatters]
    29keys = generic
    30
    31[logger_root]
    32level = INFO
    33handlers = console
    34
    35[handler_console]
    36class = StreamHandler
    37args = (sys.stderr,)
    38level = NOTSET
    39formatter = generic
    40
    41[formatter_generic]
    42format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
    43
    44# End logging configuration
    
  5. Our startup code in zodb/tutorial/__init__.py gets some bootstrapping changes:

     1from pyramid.config import Configurator
     2from pyramid_zodbconn import get_connection
     3
     4from .resources import bootstrap
     5
     6
     7def root_factory(request):
     8    conn = get_connection(request)
     9    return bootstrap(conn.root())
    10
    11def main(global_config, **settings):
    12    config = Configurator(settings=settings,
    13                          root_factory=root_factory)
    14    config.include('pyramid_jinja2')
    15    config.scan('.views')
    16    return config.make_wsgi_app()
    
  6. Our views in zodb/tutorial/views.py have modest changes in add_folder and add_content for how new instances are made and put into a container:

     1from random import randint
     2
     3from pyramid.httpexceptions import HTTPFound
     4from pyramid.location import lineage
     5from pyramid.view import view_config
     6
     7from .resources import (
     8    Root,
     9    Folder,
    10    Document
    11    )
    12
    13
    14class TutorialViews(object):
    15    def __init__(self, context, request):
    16        self.context = context
    17        self.request = request
    18        self.parents = reversed(list(lineage(context)))
    19
    20    @view_config(renderer='templates/root.jinja2',
    21                 context=Root)
    22    def root(self):
    23        page_title = 'Quick Tutorial: Root'
    24        return dict(page_title=page_title)
    25
    26    @view_config(renderer='templates/folder.jinja2',
    27                 context=Folder)
    28    def folder(self):
    29        page_title = 'Quick Tutorial: Folder'
    30        return dict(page_title=page_title)
    31
    32    @view_config(name='add_folder', context=Folder)
    33    def add_folder(self):
    34        # Make a new Folder
    35        title = self.request.POST['folder_title']
    36        name = str(randint(0, 999999))
    37        new_folder = Folder(title)
    38        new_folder.__name__ = name
    39        new_folder.__parent__ = self.context
    40        self.context[name] = new_folder
    41
    42        # Redirect to the new folder
    43        url = self.request.resource_url(new_folder)
    44        return HTTPFound(location=url)
    45
    46    @view_config(name='add_document', context=Folder)
    47    def add_document(self):
    48        # Make a new Document
    49        title = self.request.POST['document_title']
    50        name = str(randint(0, 999999))
    51        new_document = Document(title)
    52        new_document.__name__ = name
    53        new_document.__parent__ = self.context
    54        self.context[name] = new_document
    55
    56        # Redirect to the new document
    57        url = self.request.resource_url(new_document)
    58        return HTTPFound(location=url)
    59
    60    @view_config(renderer='templates/document.jinja2',
    61                 context=Document)
    62    def document(self):
    63        page_title = 'Quick Tutorial: Document'
    64        return dict(page_title=page_title)
    
  7. Make our resources persistent in zodb/tutorial/resources.py:

     1from persistent import Persistent
     2from persistent.mapping import PersistentMapping
     3import transaction
     4
     5
     6class Folder(PersistentMapping):
     7    def __init__(self, title):
     8        PersistentMapping.__init__(self)
     9        self.title = title
    10
    11
    12class Root(Folder):
    13    __name__ = None
    14    __parent__ = None
    15
    16
    17class Document(Persistent):
    18    def __init__(self, title):
    19        Persistent.__init__(self)
    20        self.title = title
    21
    22
    23def bootstrap(zodb_root):
    24    if not 'tutorial' in zodb_root:
    25        root = Root('My Site')
    26        zodb_root['tutorial'] = root
    27        transaction.commit()
    28    return zodb_root['tutorial']
    
  8. No changes to any templates!

  9. Run your Pyramid application with:

    $ $VENV/bin/pserve development.ini --reload
    
  10. Open http://localhost:6543/ in your browser.

Analysis

We install pyramid_zodbconn to handle database connections to ZODB. This pulls the ZODB3 package as well.

To enable pyramid_zodbconn:

  • We activate the package configuration using pyramid.includes.

  • We define a zodbconn.uri setting with the path to the Data.fs file.

In the root factory, instead of using our old root object, we now get a connection to the ZODB and create the object using that.

Our resources need a couple of small changes. Folders now inherit from persistent.PersistentMapping and document from persistent.Persistent. Note that Folder now needs to call super() on the __init__ method, or the mapping will not initialize properly.

On the bootstrap, note the use of transaction.commit() to commit the change. This is because on first startup, we want a root resource in place before continuing.

ZODB has many modes of deployment. For example, ZEO is a pure-Python object storage service across multiple processes and hosts. RelStorage lets you use a RDBMS for storage/retrieval of your Python pickles.

Extra Credit

  1. Create a view that deletes a document.

  2. Remove the configuration line that includes pyramid_tm. What happens when you restart the application? Are your changes persisted across restarts?

  3. What happens if you delete the files named Data.fs*?