GRPUG Presentation on OIE

Last night I presented at the Grand Rapids Python Users Group about the OpenGroupware Coils project and specifically the workflow component: OpenGroupware Integration Engine.

Click to download presentation (PDF).

This presentation convers the basic terminology and concepts behind the OpenGroupware Integration Engine - how to think about process modelling with OIE.  For full details on how to use OIE, as always, see the WMOGAG document.

Note: the OIE section of the presentation is featured above. The full presentation, as given, with a few more slides and some general Python bits is also available here.

New Edition of WMOGAG Coils Available

A new edition of WMOGAG Coils [Whitemice Consulting's OpenGroupware Administrator's Guide] has been uploaded to SourceForge; download here.  This edition documents the new features available in recent production and RC releases include using the workflow scheduler, the new translateAction, and configuration of the services for SSH access to other hosts.
As always please report any comments, recommendations, or even typographic errors to either the Coil's SourceForge project or the Coil's maillist.

The Workflow Scheduler

With the 0.1.49 release [and currently available in the 0.1.49rc releases] OpenGroupware Coils exposes the workflow scheduler via WebDAV at the path "/dav/Workflow/Schedule".  This collection [folder] allows enumeration of schedule entries by WebDAV clients or via REST, as well as the creation and deletion of schedule entries. 

Each entry is represented by a JSON serialized resource.  These resources can be retrieved via GET, created via PUT, or removed via DELETE. For REST clients the resource "/var/Workflow/Schedule/.contents" will return a JSON encoded list of all the folder entries.  Via either RPC mechanism clients only see the schedule entries available to their own security context.

$ cat send-request.json
{"routeId":1158480,"priority":300,"repeat":1,"type":"interval","minutes":2}
$ curl -vvvv -u adam:******* -X PUT --upload-file send-request.json http://127.0.0.1:8080/dav/Workflow/Schedule/request
...
< Location: /dav/Workflow/Schedule/0f6bd804a4f14cf6a2807dad625e2bb1.json
< X-OpenGroupware-ScheduleEntryUUID: 0f6bd804a4f14cf6a2807dad625e2bb1 
...
Create a scheduler entry where an instance of the route with the objectId 1158480 will be created two minutes from the current time having a priority of 300.  The schedule will only occur once.

The required keys in the JSON data are: "routeId" and "type".  The keys "priority", "contextId", "xattrDict", "attachmentUUID", and "repeat" are optional.  If unspecified the priority for scheduled entries will be 200, they will repeat without limit according to their schedule, and have the context of the currently authenticated user.  A user may only create schedule entries with contexts that are available to them. Additional keys are required or optional based upon the value of "type" which describes the kind of scheduling entry to create: "simple", "interval", or "cron".

If the workflow requires an input message an attachment may be created via the AttachFS protocol and the UUID of that attachment then specified in the scheduling entry; at run time the content of the attachment will be read into the input message of the new process.  The attachment's UUID is sent in the  "attachmentUUID" attribute of the scheduling request.

In the future it will also be possible to specify a project document as an input message.  Project documents have the advantages over attachments that they have a security descriptor and can be modified in-place.

Using the "xattrDict" attribute a dictionary of key and values can be declared that will be applied to a new process as its XATTR values.

An "interval" schedule type must specify at least one of the following values: "weeks", "days", "hours", "minutes", "seconds".  The type of any of these values must be an integer.  Multiple values may be specified and their accumulative value will specify the frequency the process will be run.

A "simple" schedule requires on a "date" value; this specifies a specific time when the process will be created.  "simple" schedule entries do not repeat. 

The "cron" type schedule entry requires at least one of the following values: "year", "month", "day", "weekday", "hour", or "minute".  All these are crontab style pattern matching strings; both division and comma separated lists are supported.  Any of these values if not specified default to the string "*".

All date/time values must be in UTC and expressed in either "%Y%m%dT%H%M%S" or "%Y-%m-%d %H:%M:%S" format. In either format the seconds value is optional, and if provided is ignored.  Remember that the OIE scheduler is a best-effort scheduler - it will attempt to create the process within a minute of the time indicated by the schedule, but delay is possible.  Also the scheduler only creates the process and queues it for execution.  It is up to the workflow manager component when the process will actually be executed/performed.  A variety of factors can influence the timing of process's execution, including singletons, system load, and administrative holds.

A schedule entry retrieved from the server will have all possible fields filled in as well as having two additional read-only fields: "iterationsPerformed" and "iterationsRemaining".  "iterationsPerformed" records how many times the schedule entry has been executed and "iterationsRemaining" indicates how many addition additional iterations remain.  A value of -1 in "iterationsRemaining" means the schedule entry will continue indefinitely; this field will only have a real value if "repeat" was specified when the scheduler entry was created.  If the entry has a repeat limit the "iterationsRemaining" serves as a count down to the expiration of the entry, once the remaining iteration count reaches zero the schedule entry will be automatically deleted.

{"dayOfWeek": "3,5", "week": "*", "nextIteration": "2012-11-11T23:00:00", "month": "*", "second": "0", "iterationsRemaining": -1, "year": "*", "day": "*", "minute": "0", "attachmentUUID": null, "xattrDict": {"start": "$__MONTHSTART__;", "end": "$__TODAY__;"}, "contextId": 15211340, "UUID": "{c658993a-dcf1-410b-8003-5dc134feb564}", "hour": "23", "routeId": 70470079, "priority": 200, "iterationsPerformed": 7, "type": "cron"}
A complete schedule entry as retrieved from the server.  Each time this entry creates a process that process will have a "start" XATTR whose value represents the date for first of the current month and an "end" XATTR that represents the current date. 

For additional flexibility the OIE built-in labels are supported in the value of process XATTRs (specified in the schedule entry's "xattrDict" value).  The values will be substituted each time a new process is created from the schedule entry.  Using built-in labels is especially useful when creating schedules for workflow processes that require a date range.

Process XATTR's and label substitution is documented in the Coils edition of WMOGAG.

Snurtle: get-performance

The latest version of OpenGroupware Coils + Snurtle provides a new command "get-performance". get-performance will retrieve the performance counters recorded by the coils.adminstrator component concerning Logic layer operations.

Output is a single line for each Logic operation that includes its frequency, average duration, minimum and maximum duration, and the total system time that has been consumed performing that operation..

Aside: Snurtle's command completion only requires you to type enough of a command to make the command unique, so "get-perf" is plenty.

Snurtle has also recently added the commands lock, unlock, list-locks, and create-contact.  The built-in help covers the syntax for these as well as get-performance.

Makefile Deployment

In the latest commits to OpenGrouwpare we've added "Makefile deployment".   This allows for simple and rapid creation of a development instance.

So long as you have PostgreSQL and RabbitMQ available you can just

1. Check out the code
2. Tweak the first few lines of the Makefile to tell it your PostgreSQL database name, user name, etc...
3. If you already have a database then touch 0_database_complete.txt to indicate that the database has already been initialized.
4. To fire-up a working test instance just run "make run-master"
5. Point your WebDAV client at 127.0.0.1:8080/dav

That's it! The make file will create a Python virtualized container to handle dependencies, retrieve the required dependencies, install a bare minimum configuration, create a document root, and start the service.  You will have the administrative account with the password you specified in the first few lines of the make file.

You can run "make install-dependencies" if you want to update the Coils dependencies check  - just delete the 0_dependencies_installed.txt file which indicates that dependencies have already been isntalled.

Aside: You may also want to add some of the optional Python modules to your virtual environment such as PIL, informixdb, z3c.rml, procname, or YaJL.  These are not required by OpenGroupware Coils and are not automatically installed.

Simple steps to provision PostgreSQL & RabbitMQ

sudo sudo -u postgres createuser --password --no-superuser --no-created --no-createrole OGoDev
sudo sudo -u postgres createdb -E UTF-8 -O OGoDev OGoDev0
sudo /usr/sbin/rabbitmqctl add_user OGoDev {AMQPASSWORD}
sudo /usr/sbin/rabbitmqctl add_vhost OGoDev0
sudo /usr/sbin/rabbitmqctl set_permissions -p OGoDev0 OGoDev ".*" ".*" ".*"
Text1: Provision PostgreSQL and RabbitMQ

This will give you a PG and AMQ user "OGoDev" with the password specified, a database named "OGoDev0", and an AMQ virtual host named "OGoDev0".  These values are set in the top of the Makefile.

Creating An OpenGroupware Coils Development Instance

These instructions provide a simple recipe to create an OpenGroupware Coils development instance.  Previously creating an instance has admittedly been a bit tedious; but in the last couple of weeks an effort has been made to simplify deployment.

Aside: Always see the Administrator's Guide (WMOGAG) for complete documentation.  Ask questions on the coils-project mailing list.

Step#1) Provisioning dependencies

Both PostgreSQL and RabbitMQ must be provisioned on the host as well as the dependencies of the required Python modules.  The dependencies are easily met on either CentOS6 (with RPMForge enabled) and openSUSE 12.1.  Installing PostgreSQL and RabbitMQ should be performed using the standard methods.

CentOS6  rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
  rpm -Uvh
    http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
  yum install python python-ldap  python-devel python-setuptools gcc \
              make binutils libxml2-devel libxslt-devel libyaml \
              libyaml-devel postgresql-devel postgresql-libs cmake \
              gcc-c++ freetype-devel libpng-devel libjpeg-devel \
              libsmbclient-devel

openSUSE
  zypper in python python-ldap  python-devel python-Distutils2 gcc make \
            binutils libxml2-devel libxslt-devel libyaml libyaml-devel \
            postgresql-devel postgresql cmake gcc-c++ freetype2-devel \
           libpng14-devel libjpeg62-devel libsmbclient-devel
Text: These commands will install the required system prerequisites for the required Python modules.

A simple method for installing RabbitMQ on openSUSE 12.1 is described  in the "Idjit's Guide To Installing RabbitMQ on openSUSE 12.1"

Step#1.1) Installed a compatible version of the YAJL library.


The YAJL library provides SAX like stream-processing of JSON data.  This library is required by the ijson Python module. This module is not required by OpenGroupware Coils so this step can be skipped;  but usage of this module (and thus library) are strongly recommended

curl -o yajl-1.0.11.tar.gz http://gentoo.osuosl.org/distfiles/yajl-1.0.11.tar.gz

tar xzvf yajl-1.0.11.tar.gz

cd lloyd-yajl-f4baae0/

mkdir build

cd build

cmake ..

sudo make install

sudo /sbin/ldconfig

Text: Fetching, building, and installing the YAJL library.

Step#1.2) Creating PostgreSQL role & database.

A role must be created in the PostgreSQL engine; the recommendation is to name the role "OGoDev".  Remember the password you enter for this role!

sudo -u postgres createuser --password --no-superuser --no-created --no-createrole OGoDev

Text: Creating the "OGoDev" role.

Once a role has been created a database must be created.  The default name for the OpenGroupware database is typically "OGo" but in order to disambiguate the development database(s) from a production instance the recommendation is to name the database "OGoDev?".  I prefer to make ten databases [OGoDev0, OGoDev1, ... OGoDev9] so I have several to play with - this is especially useful for testing.

sudo -u postgres createdb -E UTF-8 -O OGoDev OGoDev0

Text: Creating a database named "OGoDev0" owned by the OGoDev role.

Step#1.3) Define RabbitMQ role & vhost(s).

The OpenGroupware Coils components also need a context with which to connect to the RabbitMQ service.  Keeping to the same role naming scheme you used with PostgreSQL is helpful for keeping this straight.  Just as with PostgreSQL I prefer to define ten roles and virtual hosts [OGoDev0, OGoDev1, .... OGoDev9] so that I can switch the instance between fresh instances.  The commands listed here will create a role with a password, create an eponymous virtual host, and then grant that role full permissions over its vhost.

sudo rabbitmqctl add_user OGoDev0 {AMQPASSWORD}
sudo rabbitmqctl add_vhost OGoDev0
sudo rabbitmqctl set_permissions -p OGoDev0 OGoDev0 ".*" ".*" ".*"

Text: Creating a role and virtual host in RabbitMQ.

A RabbitMQ virtual host is a means to allow multiple services to use the same message broker service while remaining isolated from each other/

Step#2) Create the container for the development install.

virtualenv OGo

cd OGo

echo -e '\nexport PYTHONPATH="$VIRTUAL_ENV/coils/src"\n' >> bin/activate

echo -e '\nexport OGO_SERVER_ROOT="$VIRTUAL_ENV/root"\n' >> bin/activate

Text: Creating a virtualenv instance for development.

The Python module virtualenv provides a convenient means for creating a container for performing Python development.  This [virtualenv] module should be installed in the host's global Python site-packages.

For this container it is also helpful to add the Coils code checkout to the Python path [via PYTHONPATH] and the establish a server root [via OGO_SERVER_ROOT] within the development container.  This customer server root keeps file created by the instance, including configuration, confined to the development container.

Aside: If OGO_SERVER_ROOT is not defined the server root will default to "/var/lib/opengroupware.org".

Step#2) Populate the container.

Now the container needs to be activated and the required Python modules installed.  The ". bin/activate" builds the environment for the container and should be performed from the root directory of the container whenever working with the development instance.

. bin/activate

pip install lxml==2.3.4

pip install psycopg2==2.4.5

pip install pytz

pip install sqlalchemy==0.7.4

pip install xlrd==0.7.7

pip install xlwt==0.7.4

pip install pysmbc==1.0.13

pip install procname==0.3

pip install PyYAML==3.10

pip install ijson==0.8.0

pip install apscheduler==2.0.3

pip install python-dateutil==2.1
pip install vobject==0.8.1c

pip install http://effbot.org/downloads/Imaging-1.1.7.tar.gz

Text: Activating the virtualenv container and installing Python modules.

Now we are ready to checkout the OpenGroupware Coils code into the container.

Read/Write

  hg clone ssh://{YOURUSERNAME}@hg.code.sf.net/p/coils/code coils

Read-Only

  hg clone http://hg.code.sf.net/p/coils/code coils

Text: Checking out the OpenGroupware Coils code base.

Step#3) Test the development container.

The "coils-dependency-check" tool tries to load all the modules used by OpenGroupware Coils and reports success or failure.  This verifies that the Python requirements for operating an OpenGroupware Coils instance have been met.

(OGo)awilliam@workstation:~/OGo> cd coils/src

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-dependency-check

OK: Module xlwt (XLS<2007 write support) available.

OK: Module sqlalchemy (Object Relational Modeling) available.

OK: Module coils.foundation.api.dateutil (Date & Time Arithmatic) available.

OK: Module pytz (Python Time Zone tables) available.

OK: Module xlrd (XLS<2007 read support) available.

OK: Module coils.foundation.api.vobject (vCard and vEvent parsing) available.

OK: Module lxml (SAX & DOM XML Processing) available.

OK: Module PIL (Python Imaging Library) available.

OK: Module psycopg2 (PostgreSQL RDBMS connectivity) available.

OK: Module base64 (Encode and decode Base64 data) available.

OK: Module coils.foundation.api.elementflow (Streaming XML Creation) available.

OK: Module coils.foundation.api.pypdf (Simple PDF Operations) available.

OK: Module yaml (YAML parser & serializer) available.

WARN: Module informixdb (Informix RDBMS connectivity) not available.

1 database connectivity modules found.

 * Make sure the RDBMS you intend to use for the SQLalchemy  *

 * ORM is installed and operational.                         *

1 package warnings found.

 * You are missing packages that extend the operation and    *

 * capacity of the OpenGroupware Coils service.  The service *

 * will provide core functionality but some features,        *

 * particularly in regard to OIE, may not be available. It   *

 * is recommended you install the appropriate packages.      *

Text: Run the coils-dependency-check tool to verify your Python installation.

The warning ("WARN") indicates that an optional module could not be loaded.  Warnings indicate that the service will operate but with potentially reduced functionality.  Any error ["ERROR"] means the service will fail to operate.

Step#4) Initialize the development instance.

Now that the environment is ready the Coils tools can be used to initialize and configure the instance.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-install  --user=awilliam --group=users --log=../../coils.log

Text: Initialize the server's installation; this create the required structure in the server's document root - in this case the value defined by $OGO_SERVER_ROOT.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --bootstrap

Initialized a new server defaults file.

Loaded configuration BLOB successfully.

Text: Initialize the instance's configuration with default values.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive LSConnectionDictionary --value "{'databaseName': 'OGoDev0', 'hostName': '127.0.0.1', 'password': '{SQLPASSWORD}', 'port': 5432, 'userName': 'OGoDev'}"

Text: Configure the connection to the PostgreSQL database.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive AMQConfigDictionary --value "{'hostname': '127.0.0.1', 'password': '{AMQPASSWORD}', 'port': 5672, 'username': 'OGoDev0', 'vhost': 'OGoDev0' }"

Text: Configure the connection to the RabbitMQ message broker.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-database --initdb --password={COILSADMINPASSWORD}

Text: Create the initial database schema and provision the administrative "ogo" account with the specified password.

Step#5) Test the instance.

A simple way to test the instance is to start just the HTTP component.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-service-http --asuser

Text: State the Coils HTTP component.

If this component is running you should be able to connect to and browse the WebDAV hierarchy at http://127.0.0.1:8080/dav (authenticate as the OpenGroupware Coils administrative account "ogo" and the COILSADMINPASSWORD you provided.  Browsing the hierarchy can be performed with Nautilus, cadaver, or any WebDAV client.  Use your systems break sequence [typically Ctrl-C] to stop the component.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-master-service --asuser

Text: Start the Coil's master service which will start and manage an instance of every available component.

The coils-master-service tool can be used in the same manner to start-up the full suite of service components providing HTTP and workflow services (include the TCP/9100 and SMTP listeners).

OIE XSLT Extension Methods

To facilitate the use of stylesheets as a component of workflow the OpenGroupware Integration Engine provides numerous functions callable as XSLT extensions. The extensions functions provide the ability to retrieve data from workflow messages into the style sheet as well as to access and manipulate sequences, exercise look-up tables, and search the server's database. These extension functions are available in the http://www.opengroupware.us/oie namespace.

<!-- Search to see if we can find a copy of that document-->
<xsl:variable name="documentid"
     select="oie:searchforobjectid(string('document'),
  'property.{http://example.com/financial}bankCode', string(bank_code),
  'property.{http://example.com/financial}documentType', 'PDF',
  'property.{http://example.com/financial}invoiceNumber', string(workorder),
  'property.{http://example.com/financial}invoiceDate', $invoicedate )"/>
<xsl:choose>  
  <xsl:when test="$documentid">
    <document_id isNull="false" dataType="integer"><xsl:value-of select="$documentid"/></document_id> 
  </xsl:when>
  <xsl:otherwise>  
    <document_id isNull="true" dataType="integer"/>    
  </xsl:otherwise>    
</xsl:choose>
Perform a search for the object id of the document matching the four specified conditions. In this example $invoicedate is a previously defined XSLT variable and bank_code a current element.

Currently implemented extension functions are:

• sequencevalue(scope, name) - Retrieve the value of named sequence. An exception will occur if the sequence does not exist.
• sequencereset(scope, name, value) - Set or reset the value of the named sequence to the specified value. This method will create a new sequence if a sequence by that name in the specified scope does not exist.
• sequenceincrement(scope, name, increment) - Increment the named sequence by the specified value. An exception will occur if the sequence does not exist.
• messagetext(label) - Retrieves the contents of the specified message within the current scope. If the messsage does not exist an exception will be raised by the underlying Logic operation. Care should be taken as to the encoding and type of data inserted into a style-sheet; no verification that the content of the message is suitable for inclusion in the style-sheet at the relevant point is performed.
• searchforobjectid(domain, key, value, key, value, ….) - Performs a search of the specified domain using the provided key and value pairs. There is no hard limit on the number of value pairs that may be provided.  The domain must be one of: “appointment”, “contact”, “document”, “enterprise”, “process”, “project”, “resource”,  or “task” to identify the type of object being searched for.  If the search uniquely identities one entity of the specified type the object id of that entity is returned; if the search identifies either no entities or multiple entities an empty string is returned.  The intention of this method is to allow for content from the source document to be used to identity objects from the OpenGroupware database.
• countobjects(domain, key, value, key, value, ….) - Performs a search of the specified domain using the provided key and value pairs. There is no hard limit on the number of value pairs that may be provided.  The domain must be one of: “appointment”, “contact”, “document”, “enterprise”, “process”, “project”, “resource”,  or “task” to identify the type of object being searched for.  The return value of the function is the number of objects that matched the search criteria up to 1,000. The search is limited to 1,000 results..
• tablelookup(name, value) - Lookup the specified value in the named table.
• reformatdate(value, format) - Reformats the StandardXML date or date-time value into the specified format. Since value is a string a length of 10 is assumed to represent a date input and a length of 19 to represent a date-time value.
• datetimetodate(value) - Converts a StandardXML date-time representation to a StandardXML date representation.
• stringtodate(value, format) - Reformats a date in the specified format to a StandardXML date representation.
• stringtodatetime(value, format) - Reformats a date time in the specified format to a StandardXML date-time representation.
• xattrvalue(name) - Returns the value of the named XATTR for the current process.  If no such XATTR is defined an empty string is returned.

All XSLT extensions are documented for the transform OIE action in the WMOGAG document. With the help of these methods XSLT transforms can be used to generate sophisticated and context-sensitive output documents.