6: Storing Resources In ZODB

Store and retrieve resource tree containers and items in a database.

Background

We now have a resource tree that can go infinitely deep, adding items and subcontainers along the way. We obviously need a database, one that can support hierarchies. ZODB is a transaction-based Python database that supports transparent persistence. We will modify our application to work with the ZODB.

Along the way we will add the use of pyramid_tm, a system for adding transaction awareness to our code. With this we don't need to manually manage our transaction begin/commit cycles in our application code. Instead, transactions are setup transparently on request/response boundaries, outside our application code.

Objectives

• Create a CRUD app that adds records to persistent storage.

• Setup pyramid_tm and pyramid_zodbconn.

• Make our "content" classes inherit from Persistent.

• Set up a database connection string in our application.

• Set up a root factory that serves the root from ZODB rather than from memory.

Steps

1. We are going to use the previous step as our starting point:

   $ cd ..; cp -r addcontent zodb; cd zodb
   

2. Introduce some new dependencies in zodb/setup.py:

   from setuptools import setup
   
   requires = [
       'pyramid',
       'pyramid_jinja2',
       'ZODB3',
       'pyramid_zodbconn',
       'pyramid_tm',
       'pyramid_debugtoolbar'
   ]
   
   setup(name='tutorial',
         install_requires=requires,
         entry_points="""\
         [paste.app_factory]
         main = tutorial:main
         """,
   )
   

3. We can now install our project:

   $ $VENV/bin/python setup.py develop
   

4. Modify our zodb/development.ini to include some configuration and give database connection parameters:

   [app:main]
   use = egg:tutorial
   pyramid.reload_templates = true
   pyramid.includes =
       pyramid_debugtoolbar
       pyramid_zodbconn
       pyramid_tm
   zodbconn.uri = file://%(here)s/Data.fs?connection_cache_size=20000
   
   [server:main]
   use = egg:pyramid#wsgiref
   host = 0.0.0.0
   port = 6543
   
   # Begin logging configuration
   
   [loggers]
   keys = root, tutorial
   
   [logger_tutorial]
   level = DEBUG
   handlers =
   qualname = tutorial
   
   [handlers]
   keys = console
   
   [formatters]
   keys = generic
   
   [logger_root]
   level = INFO
   handlers = console
   
   [handler_console]
   class = StreamHandler
   args = (sys.stderr,)
   level = NOTSET
   formatter = generic
   
   [formatter_generic]
   format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
   
   # End logging configuration
   

5. Our startup code in zodb/tutorial/__init__.py gets some bootstrapping changes:

   from pyramid.config import Configurator
   from pyramid_zodbconn import get_connection
   
   from .resources import bootstrap
   
   
   def root_factory(request):
       conn = get_connection(request)
       return bootstrap(conn.root())
   
   def main(global_config, **settings):
       config = Configurator(settings=settings,
                             root_factory=root_factory)
       config.include('pyramid_jinja2')
       config.scan('.views')
       return config.make_wsgi_app()
   

6. Our views in zodb/tutorial/views.py have modest changes in add_folder and add_content for how new instances are made and put into a container:

   from random import randint
   
   from pyramid.httpexceptions import HTTPFound
   from pyramid.location import lineage
   from pyramid.view import view_config
   
   from .resources import (
       Root,
       Folder,
       Document
       )
   
   
   class TutorialViews(object):
       def __init__(self, context, request):
           self.context = context
           self.request = request
           self.parents = reversed(list(lineage(context)))
   
       @view_config(renderer='templates/root.jinja2',
                    context=Root)
       def root(self):
           page_title = 'Quick Tutorial: Root'
           return dict(page_title=page_title)
   
       @view_config(renderer='templates/folder.jinja2',
                    context=Folder)
       def folder(self):
           page_title = 'Quick Tutorial: Folder'
           return dict(page_title=page_title)
   
       @view_config(name='add_folder', context=Folder)
       def add_folder(self):
           # Make a new Folder
           title = self.request.POST['folder_title']
           name = str(randint(0, 999999))
           new_folder = Folder(title)
           new_folder.__name__ = name
           new_folder.__parent__ = self.context
           self.context[name] = new_folder
   
           # Redirect to the new folder
           url = self.request.resource_url(new_folder)
           return HTTPFound(location=url)
   
       @view_config(name='add_document', context=Folder)
       def add_document(self):
           # Make a new Document
           title = self.request.POST['document_title']
           name = str(randint(0, 999999))
           new_document = Document(title)
           new_document.__name__ = name
           new_document.__parent__ = self.context
           self.context[name] = new_document
   
           # Redirect to the new document
           url = self.request.resource_url(new_document)
           return HTTPFound(location=url)
   
       @view_config(renderer='templates/document.jinja2',
                    context=Document)
       def document(self):
           page_title = 'Quick Tutorial: Document'
           return dict(page_title=page_title)
   

7. Make our resources persistent in zodb/tutorial/resources.py:

   from persistent import Persistent
   from persistent.mapping import PersistentMapping
   import transaction
   
   
   class Folder(PersistentMapping):
       def __init__(self, title):
           PersistentMapping.__init__(self)
           self.title = title
   
   
   class Root(Folder):
       __name__ = None
       __parent__ = None
   
   
   class Document(Persistent):
       def __init__(self, title):
           Persistent.__init__(self)
           self.title = title
   
   
   def bootstrap(zodb_root):
       if not 'tutorial' in zodb_root:
           root = Root('My Site')
           zodb_root['tutorial'] = root
           transaction.commit()
       return zodb_root['tutorial']
   

8. No changes to any templates!

9. Run your Pyramid application with:

   $ $VENV/bin/pserve development.ini --reload
   

10. Open http://localhost:6543/ in your browser.

Analysis

We install pyramid_zodbconn to handle database connections to ZODB. This pulls the ZODB3 package as well.

To enable pyramid_zodbconn:

• We activate the package configuration using pyramid.includes.

• We define a zodbconn.uri setting with the path to the Data.fs file.

In the root factory, instead of using our old root object, we now get a connection to the ZODB and create the object using that.

Our resources need a couple of small changes. Folders now inherit from persistent.PersistentMapping and document from persistent.Persistent. Note that Folder now needs to call super() on the __init__ method, or the mapping will not initialize properly.

On the bootstrap, note the use of transaction.commit() to commit the change. This is because on first startup, we want a root resource in place before continuing.

ZODB has many modes of deployment. For example, ZEO is a pure-Python object storage service across multiple processes and hosts. RelStorage lets you use a RDBMS for storage/retrieval of your Python pickles.

Extra Credit

1. Create a view that deletes a document.

2. Remove the configuration line that includes pyramid_tm. What happens when you restart the application? Are your changes persisted across restarts?

3. What happens if you delete the files named Data.fs*?