OpenGroupware Coils Optional Dependencies

Installing OpenGroupware Coils via easy_install or pip will automatically install a variety of required dependencies.  There are however serveral optional dependencies.  The server will run without the availability of these modules but some features will be automatically disabled.  Installing the following modules will extend the capabilities of your OpenGroupware Coils instance.

• informixdb - Allows an Informix RDBMS to be used as an external data source.  This means you can define an Informix database via OIESQLDataSources and utilize it for SQL related workflow actions such as sqlInsertAction, sqlSelectAction, etc... 
• z3c.rml / zope.interface - Installing these modules will enable the conversions of XML to PDF.  Using XSLT transforms you can turn your data into RML, and then using the rmlToPDF workflow action convert the data into sophisticated PDF documents.
• pysmbc - Enables the OIE workflow engine to retrieve and upload files from or to a CIFS / SMB volume. 
• PIL - Enables the OSSF module for retrieving thumbnails of images stored in OpenGroupware projects or attachments. 
• paramiko - Presence of this module activates the SSH related OIE workflow actions such as sshExecuteCmmand. 
• python-ldap - The availability of the LDAP module enables both support for LDAP authentication (such as when using Active Directory, OpenLDAP, etc... as your user database) and support for querying an LDAP DSA from a workflow process using OIE's ldapSearchAction.

Once your OpenGroupware Coils instance is up and running you should evaluate which additional, optional, dependencies would be useful in your environment.

GRPUG Presentation on OIE

Last night I presented at the Grand Rapids Python Users Group about the OpenGroupware Coils project and specifically the workflow component: OpenGroupware Integration Engine.

Click to download presentation (PDF).

This presentation convers the basic terminology and concepts behind the OpenGroupware Integration Engine - how to think about process modelling with OIE.  For full details on how to use OIE, as always, see the WMOGAG document.

Note: the OIE section of the presentation is featured above. The full presentation, as given, with a few more slides and some general Python bits is also available here.

Creating An OpenGroupware Coils Development Instance

These instructions provide a simple recipe to create an OpenGroupware Coils development instance.  Previously creating an instance has admittedly been a bit tedious; but in the last couple of weeks an effort has been made to simplify deployment.

Aside: Always see the Administrator's Guide (WMOGAG) for complete documentation.  Ask questions on the coils-project mailing list.

Step#1) Provisioning dependencies

Both PostgreSQL and RabbitMQ must be provisioned on the host as well as the dependencies of the required Python modules.  The dependencies are easily met on either CentOS6 (with RPMForge enabled) and openSUSE 12.1.  Installing PostgreSQL and RabbitMQ should be performed using the standard methods.

CentOS6  rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
  rpm -Uvh
    http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
  yum install python python-ldap  python-devel python-setuptools gcc \
              make binutils libxml2-devel libxslt-devel libyaml \
              libyaml-devel postgresql-devel postgresql-libs cmake \
              gcc-c++ freetype-devel libpng-devel libjpeg-devel \
              libsmbclient-devel

openSUSE
  zypper in python python-ldap  python-devel python-Distutils2 gcc make \
            binutils libxml2-devel libxslt-devel libyaml libyaml-devel \
            postgresql-devel postgresql cmake gcc-c++ freetype2-devel \
           libpng14-devel libjpeg62-devel libsmbclient-devel
Text: These commands will install the required system prerequisites for the required Python modules.

A simple method for installing RabbitMQ on openSUSE 12.1 is described  in the "Idjit's Guide To Installing RabbitMQ on openSUSE 12.1"

Step#1.1) Installed a compatible version of the YAJL library.


The YAJL library provides SAX like stream-processing of JSON data.  This library is required by the ijson Python module. This module is not required by OpenGroupware Coils so this step can be skipped;  but usage of this module (and thus library) are strongly recommended

curl -o yajl-1.0.11.tar.gz http://gentoo.osuosl.org/distfiles/yajl-1.0.11.tar.gz

tar xzvf yajl-1.0.11.tar.gz

cd lloyd-yajl-f4baae0/

mkdir build

cd build

cmake ..

sudo make install

sudo /sbin/ldconfig

Text: Fetching, building, and installing the YAJL library.

Step#1.2) Creating PostgreSQL role & database.

A role must be created in the PostgreSQL engine; the recommendation is to name the role "OGoDev".  Remember the password you enter for this role!

sudo -u postgres createuser --password --no-superuser --no-created --no-createrole OGoDev

Text: Creating the "OGoDev" role.

Once a role has been created a database must be created.  The default name for the OpenGroupware database is typically "OGo" but in order to disambiguate the development database(s) from a production instance the recommendation is to name the database "OGoDev?".  I prefer to make ten databases [OGoDev0, OGoDev1, ... OGoDev9] so I have several to play with - this is especially useful for testing.

sudo -u postgres createdb -E UTF-8 -O OGoDev OGoDev0

Text: Creating a database named "OGoDev0" owned by the OGoDev role.

Step#1.3) Define RabbitMQ role & vhost(s).

The OpenGroupware Coils components also need a context with which to connect to the RabbitMQ service.  Keeping to the same role naming scheme you used with PostgreSQL is helpful for keeping this straight.  Just as with PostgreSQL I prefer to define ten roles and virtual hosts [OGoDev0, OGoDev1, .... OGoDev9] so that I can switch the instance between fresh instances.  The commands listed here will create a role with a password, create an eponymous virtual host, and then grant that role full permissions over its vhost.

sudo rabbitmqctl add_user OGoDev0 {AMQPASSWORD}
sudo rabbitmqctl add_vhost OGoDev0
sudo rabbitmqctl set_permissions -p OGoDev0 OGoDev0 ".*" ".*" ".*"

Text: Creating a role and virtual host in RabbitMQ.

A RabbitMQ virtual host is a means to allow multiple services to use the same message broker service while remaining isolated from each other/

Step#2) Create the container for the development install.

virtualenv OGo

cd OGo

echo -e '\nexport PYTHONPATH="$VIRTUAL_ENV/coils/src"\n' >> bin/activate

echo -e '\nexport OGO_SERVER_ROOT="$VIRTUAL_ENV/root"\n' >> bin/activate

Text: Creating a virtualenv instance for development.

The Python module virtualenv provides a convenient means for creating a container for performing Python development.  This [virtualenv] module should be installed in the host's global Python site-packages.

For this container it is also helpful to add the Coils code checkout to the Python path [via PYTHONPATH] and the establish a server root [via OGO_SERVER_ROOT] within the development container.  This customer server root keeps file created by the instance, including configuration, confined to the development container.

Aside: If OGO_SERVER_ROOT is not defined the server root will default to "/var/lib/opengroupware.org".

Step#2) Populate the container.

Now the container needs to be activated and the required Python modules installed.  The ". bin/activate" builds the environment for the container and should be performed from the root directory of the container whenever working with the development instance.

. bin/activate

pip install lxml==2.3.4

pip install psycopg2==2.4.5

pip install pytz

pip install sqlalchemy==0.7.4

pip install xlrd==0.7.7

pip install xlwt==0.7.4

pip install pysmbc==1.0.13

pip install procname==0.3

pip install PyYAML==3.10

pip install ijson==0.8.0

pip install apscheduler==2.0.3

pip install python-dateutil==2.1
pip install vobject==0.8.1c

pip install http://effbot.org/downloads/Imaging-1.1.7.tar.gz

Text: Activating the virtualenv container and installing Python modules.

Now we are ready to checkout the OpenGroupware Coils code into the container.

Read/Write

  hg clone ssh://{YOURUSERNAME}@hg.code.sf.net/p/coils/code coils

Read-Only

  hg clone http://hg.code.sf.net/p/coils/code coils

Text: Checking out the OpenGroupware Coils code base.

Step#3) Test the development container.

The "coils-dependency-check" tool tries to load all the modules used by OpenGroupware Coils and reports success or failure.  This verifies that the Python requirements for operating an OpenGroupware Coils instance have been met.

(OGo)awilliam@workstation:~/OGo> cd coils/src

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-dependency-check

OK: Module xlwt (XLS<2007 write support) available.

OK: Module sqlalchemy (Object Relational Modeling) available.

OK: Module coils.foundation.api.dateutil (Date & Time Arithmatic) available.

OK: Module pytz (Python Time Zone tables) available.

OK: Module xlrd (XLS<2007 read support) available.

OK: Module coils.foundation.api.vobject (vCard and vEvent parsing) available.

OK: Module lxml (SAX & DOM XML Processing) available.

OK: Module PIL (Python Imaging Library) available.

OK: Module psycopg2 (PostgreSQL RDBMS connectivity) available.

OK: Module base64 (Encode and decode Base64 data) available.

OK: Module coils.foundation.api.elementflow (Streaming XML Creation) available.

OK: Module coils.foundation.api.pypdf (Simple PDF Operations) available.

OK: Module yaml (YAML parser & serializer) available.

WARN: Module informixdb (Informix RDBMS connectivity) not available.

1 database connectivity modules found.

 * Make sure the RDBMS you intend to use for the SQLalchemy  *

 * ORM is installed and operational.                         *

1 package warnings found.

 * You are missing packages that extend the operation and    *

 * capacity of the OpenGroupware Coils service.  The service *

 * will provide core functionality but some features,        *

 * particularly in regard to OIE, may not be available. It   *

 * is recommended you install the appropriate packages.      *

Text: Run the coils-dependency-check tool to verify your Python installation.

The warning ("WARN") indicates that an optional module could not be loaded.  Warnings indicate that the service will operate but with potentially reduced functionality.  Any error ["ERROR"] means the service will fail to operate.

Step#4) Initialize the development instance.

Now that the environment is ready the Coils tools can be used to initialize and configure the instance.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-install  --user=awilliam --group=users --log=../../coils.log

Text: Initialize the server's installation; this create the required structure in the server's document root - in this case the value defined by $OGO_SERVER_ROOT.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --bootstrap

Initialized a new server defaults file.

Loaded configuration BLOB successfully.

Text: Initialize the instance's configuration with default values.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive LSConnectionDictionary --value "{'databaseName': 'OGoDev0', 'hostName': '127.0.0.1', 'password': '{SQLPASSWORD}', 'port': 5432, 'userName': 'OGoDev'}"

Text: Configure the connection to the PostgreSQL database.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive AMQConfigDictionary --value "{'hostname': '127.0.0.1', 'password': '{AMQPASSWORD}', 'port': 5672, 'username': 'OGoDev0', 'vhost': 'OGoDev0' }"

Text: Configure the connection to the RabbitMQ message broker.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-database --initdb --password={COILSADMINPASSWORD}

Text: Create the initial database schema and provision the administrative "ogo" account with the specified password.

Step#5) Test the instance.

A simple way to test the instance is to start just the HTTP component.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-service-http --asuser

Text: State the Coils HTTP component.

If this component is running you should be able to connect to and browse the WebDAV hierarchy at http://127.0.0.1:8080/dav (authenticate as the OpenGroupware Coils administrative account "ogo" and the COILSADMINPASSWORD you provided.  Browsing the hierarchy can be performed with Nautilus, cadaver, or any WebDAV client.  Use your systems break sequence [typically Ctrl-C] to stop the component.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-master-service --asuser

Text: Start the Coil's master service which will start and manage an instance of every available component.

The coils-master-service tool can be used in the same manner to start-up the full suite of service components providing HTTP and workflow services (include the TCP/9100 and SMTP listeners).