2013-12-17

OpenGrouwpare As A CUPS Backend

A script suitable for use as a back-end for the CUPS print server package is included with OpenGroupware Coils. This back-end allows for queues to be created that perform HTTP PUT requests to the server using configurable [per queue] paths.  Using OpenGroupware Coils as a CUPS back-end easily enables client applications - and any application - to submit documents to work-flows or to store documents in project folders; the client simply prints via a print queue using a Postscript print driver and the CUPS server converts the document to PDF and pushes it to the server using the specified path.  An example of this is a receipt printer - printing to a project document folder allows the document to be filed electronically;  event driven document processing can then turn that same PDF around and submit it to an actual print queue if necessary.

Installation
To install the CUPs backend copy the “cups-to-ogo” script to the appropriate back-end directory on the print server and modify the permissions to permit the script to be executed by the CUPs user [typically “lp” or “root”]. 
This script does not require OpenGroupware components or the OpenGroupware package to be installed on the print server; all Python requirements are satisfied by the Python standard library.
A tool such as “ps2pdf” for converting Postscript to PDF will also be required; the script assumes all content posted to the server is in PDF format. These tools are generally already installed on any CUPS instance as CUPS itself uses the ghostscript tool-chain for processing the content of jobs.
When the script is executed by the CUPS server it will log messages to the local syslog server under the LPR facilty.

Configuration
[global]
uri=http://coils.
example.com/
auth_mech=plain
ps2pdf=/usr/bin/ps2pdf
username=lp
password=
theverysupersecretpassword

Text 2: An example cups-to-ogo.ini configuration file.
The script will attempt to read it's configuration from an INI file located at “/etc/cups/cups-to-ogo.ini”. This location may be overridden by setting the environment variable OGOCUPSCONFIGINI to the full path of the required INI file; in order to be effective that environment variable must be present in the environment of the CUPS server itself – the environment of the server is inherited by all filters and back-ends executed by the CUPS instance.
The INI file has a single global section. This section specifies the URI of the OpenGroupware Coils server, the authentication information , and the path to the postscript to PDF conversion tool. 
Section Parameter Description
global
uri
The base URI to the OpenGroupware COILS server. This URI should end with a “/”.
global
auth_mech
The authentication mechanism to use when authenticating to the server; at this time only “plain” is supported.
global
ps2pdf
The path to the executable postscript to PDF conversion tool. This tool must read its input from standard-input and write the converted document to standard-output. The tool must also be executable within the context of the CUPS server instance.
global
username
If auth_mech is “plain” this specifies the user-name to use when performing BASIC authentication to the CUPS server.
global
password
If auth_mech is “plain” this specifies the password to use when performing BASIC authentication to the CUPS server.
Table 2: cups-to-ogo configuration parameters supported in the cups-to-ogo.ini file.
Once the script has been installed and configured it should be reported when listing the enabled CUPS back-ends using the “lpinfo -v” command on the print server. 

$ lpinfo -v
network ipp
network socket
direct scsi
network smb
network http
network lpd
network beh
network https
serial serial:/dev/ttyS0?baud=115200
serial serial:/dev/ttyS1?baud=115200

network cups-to-ogo

network socket://192.168.1.10

Text 3: Listing the available CUPS back-ends using "lpinfo -v"
Security
When defining a target to receive documents from the CUPS integration do not forget to provision security ACLs of the resource to make it accessible to the user which the back-end authenticates as. In order to create a document within a folder the CUPS context must be able to realize the folder's project and have been granted list and insert permissions on the target folder.
It is recommended that a site create a specific intranet account for use by the CUPS server for authenticating to the OpenGroupware instance; this facilitates change control and auditing. This special purpose account should be excluded from the all-intranet team to avoid the possibility of inheriting additional privileges. The CUPS server should never be provided with the OpenGroupware administrative credentials or any credentials holding an administrative or developer role. Authentication and access control related to queuing jobs to queues defined on the CUPS server is entirely an issue relating to the administration and configuration of the CUPS instance,.
Creating Queues

Queues for pushing documents to the OpenGroupware Coils server can created using either the CUPS web interface or the lpadmin command line tool.
The recommended CUPS print driver for all cups-to-ogo queues is “Generic PostScript Printer Foomatic/Postscript”. The postscript input job will be converted to PDF using the configured conversion tool.
Illustration 1: CUPS web interface for creating a new print queue, via "Add Printer".
The utility of the cups-to-ogo backend is exercised through the URL portion of the printer definition. The URI specified for the queue is appended to the server URI provided in the INI configuration file. The print job, after being converted to PDF format, will be uploaded to the resulting URL by an HTTP PUT operation. Utilizing the standard HTTP PUT operation allows documents to be submitted to work-flow routes, uploaded via AttachFS, or placed in WebDAV collections.
In order to facilitate uploads into collections, via either AttachFS or WebDAV, as well as to allow capturing of all source meta-data replacement labels may be placed in the queue's URI.  These strings have the form of “${label}” where label is the label name;  each label in the queue's URI will be replaced with the appropriate value when a job is processed.
For example, to submit a print job [converted to a PDF document] to a work-flow route named “MailMeExample” and include the user name of the user who submitted the job to the CUPS server a queue URI of “cups-to-ogo://dav/Workflow­/Routes/MailMeExample/input?username=${user}” would be used. The content of the print job would be the input message of the new process [instance of the work-flow route] and an XATTR named “username” would be created and attached to the process, the value of that XATTR would be the name of the username of the user who authenticated to the CUPS server. 
Label Description
user
The user name of the context which submitted the print job [and potentially authenticated to the CUPS server]. This value will be URL quoted when the URL is constructed as user names may contain punctuation and whitespace.
id
The job id assigned to the job by the CUPS server.
title
The title attribute of the print job. This value will be URL quoted when the URL is constructed as job titles may contain punctuation and whitespace.
guid
Generates a GUID string. This is intended for use-cases when a unique identifier is required within the PUT URI.

Text 4: Substitution labels supported by the cups-to-ogo backend.

Another common use for the CUPS backend is to auto-file jobs to project folders. This can be accomplished by setting the URI of the queue to an AttachFS target indicating the folder in which to file the document. Note that auto-filing, auto-printing, etc... are still effective – those services do no consider the origin of the document. In order to file a document into a project folder with an object id of 8,891,874 a queue should be created with a URI of “cups-to-ogo:// attachfs/8891874/${guid}.pdf?mode=file”. Note how this is a standard AttachFS URI; the file mode is specified since we desire to create a document rather than an attachment. A unique file name within the folder is required so the guid substitution label is used; from this URI a unique named document will be created with a “pdf” file extension.
cupsJobUser:
  attribute: cupsJobUser
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046

cupsJobTitle:

  attribute: cupsJobTitle
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046

cupsJobId:

  attribute: cupsJobId
  namespace: 57c7fc84-3cea-417d-af54-b659eb87a046
Text 5: Example for adding the three CUPS meta-data related properties to the server's object property alias map.
Three object-properties have been defined in the document management name-space intended to contain the CUPS job information. To expose these properties for use in AttachFS URIs add them to the property aliases map of the server. If the properties are exposed as the default aliases the additional meta-data can be recorded as object properties of the new document by extending the URI of the previous example to “cups-to-ogo://attachfs/8891874/${guid}.pdf?mode=file&pa.cupsJobId=${id}&pa.cupsJobUser=${user}&pa.cupsJobTitle=${title}”.
Debugging

When the script is executed by the CUPS server it will log messages to the local syslog server under the LPR facilty.
Provided the back-end attempts to submit a job to the OpenGroupware server – meaning the input for the filter was processed successfully – the result provided back to the CUPS server will depend upon the server's response to the HTTP PUT.
A success response, any of 200 [OK], 201 [Created], 202 [Accepted], 204 [No Content], or 301 [Moved Permanently] result in a status 0 [OK] to the CUPS server indicating the job was successfully processed and the server may proceed to the next job.
A server response of 401 [Authentication Required], 419 [Authentication Time Out], 511 [Network Authentication Required], 496 [No Certificate], or 495 [Certificate Error] results in a status of 2 [Authenticated Required2] indicating to the CUPS server than an authentication or identity related issue occurred. CUPS will place the job on hold and add the “cups-held-for-authentication” keyword to the “jobs-reasons” CUPS job attribute.
If the server response indicates a temporary error by responding with 423 [Locked (WebDAV)], 409 [Conflict], 408 [Request Timeout], or 429 [Too Many Requests (Back Off)] the CUPS status of 6 [Retry] will be returned. This status indicates the CUPS server may retry the job at a later time; the server may still proceed to processing the next job in the queue.
A response value of either 418 [Teapot] or 451 [Unavailable For Legal Reasons] will result in CUPS status 5 [Cancel]. The CUPS server responds to a status 5 by automatically canceling the current job and proceeding to the next job.
Any other server response, most notably a 404 [Not Found] or 500 [Server Error], will return status 1 [Failed] to the server. The server will respond to this status using the queues defined error-policy attribute. Generally speaking the CUPS default error policy is to change the queue to a paused state when encountering a status 1 [Failed] response; a paused queue will continue to accept jobs but will not attempt to submit them to the backend.

2013-04-09

Collection Creation

Many things are "collections";  projects are collections, document folders are collections, in OpenGrouwpare Coils even "Collections" are collections!  A frequent action is the creation of a new collection - and there are lots of ways to do it.  Most obviously a collection can be created interactively using a WebDAV client such as Nautilus (which accesses GVFS's WebDAV support), the Microsoft Windows WebDAV Redirector, Cyberduck, cadaver, WebDAV Nav Lite for Android or IOS, etc... 
All these WebDAV clients create collections by making an HTTP MKCOL requests.  With MKCOL requests it is all a matter of "where". If you create a collection in "/dav/Projects" you are creating a new project (which is a collection).  If you create a collection in a documents folder then you are creating a new sub-folder (which is a collection).  If you create a collection in "/dav/Contacts" then you are creating a new address book (which is a collection).  HTTP MKCOL operations are very simple - make a MKCOL request for the collection you need to exist.  And then it exists.  So if interactive clients are not your thing or you need to automate some collection creation you can do it from the command line using a standard HTTP utility such as curl:
curl -u $LOGNAME -X MKCOL \
  http://coils.example.com/dav/Projects/test123/Documents/george
Making a new subfolder named "george" in the document hierarchy of project "test123"

curl -u $LOGNAME -X MKCOL \
  http://coils.example.com/dav/Projects/newproject
Create a new project identified as "newproject".
If a project is deep within a project hierarchy a useful trick is that you can always navigate to a project from "/dav/Projects/number" if you know the number of the project - project numbers are unique, so this is always safe.  This simplifies the creation of document sub-folders, or sub-projects, in projects that are in a project hierarchy [it avoids the tedium of having to construct the entire hierarchy].  For example if I want to create a document folder named "stanley" in "project3" which is a child of "project2" which is itself a child of "project1" I would be dealing with a complete WebDAV path like "/dav/Projects/project1/Projects/project2/Projects/project3/Documents/stanley".  Oy!  That is just fine for navigating in a file-manager (where I can create bookmarks and shortcuts) but not much fun for the developer. The alternative is to ignore the hierarchy as presented to clients and use the project number shortcut.
curl -u $LOGNAME -X MKCOL http://coils.example.com/dav/Projects/top3-1/Documents/stanley
As project "project3" is a child project a WebDAV client will not see it at the top of the hierarchy - but it is still addressable there by making an explicit request.
Alternatively project, folder, and collection objects can be created by using the zOGI API via XML-RPC or JSON-RPC.  A call to zogi.putObject can create any of the aforementioned collection types [actually it can create just about any object type, collection or not].
import xmlrpclib
server = xmlrpclib.ServerProxy('http://coils.example.com/RPC2')
server.zogi.putObject( { 'entityName': 'Folder',
                                       'parentObjectid': 1234567
                                       'title': 'stanley' }
Create a new subfolder of the folder with objectId 1234567 named "stanley" via zOGI over XML-RPC.
zOGI over JSON-RPC is what is used by the snurtle client.  In the development version of snurtle it is now possible to select a project and then navigate around within its document hierarchy creating subfolders, deleting objects, etc...  And with the availability of the zOGI API snurtle allows you to view and modify the ACLs and properties of those objects as well (operations that are rarely supported by WebDAV clients).
adam>select-project --objectid=81203049
adam:Equipment:/>mkdir incoming
adam:Equipment:/>cd incoming
adam:Equipment:incoming>cd incoming
adam:Equipment:incoming>mkdir invoices
adam:Equipment:incoming>mkdir receipts
adam:Equipment:incoming>mkdir warranties
adam:Equipment:incoming>ls
  + 88930399   invoices
  + 88930419   receipts
  + 88930439   warranties
adam:C-Tabs:incoming>set-acl --objectid=88930399 --contextid=8999 --read --write
adam:C-Tabs:incoming>set-acl --objectid=88930419 --contextid=8999 --read --write
adam:C-Tabs:incoming>set-acl --objectid=88930439 --contextid=8999 --read --write
adam:C-Tabs:incoming>set-property --objectid=88930399 --namespace=http://www.opengroupware.us/smtp --attribute=collectMIMEType --value=application/pdf
adam:C-Tabs:incoming>set-property --objectid=88930419 --namespace=http://www.opengroupware.us/smtp --attribute=collectMIMEType -alue=application/pdf
adam:C-Tabs:incoming>set-property --objectid=88930439 --namespace=http://www.opengroupware.us/smtp --attribute=collectMIMEType --value=application/pdf
adam:C-Tabs:incoming>list-properties --objectid=88930399
{http://www.opengroupware.us/smtp}collectMIMEType = application/pdf
adam:C-Tabs:incoming>list-acls --objectid=88930399
  ACL (Contact, 8999) = (allowed, wr)
Select a project and create some sub-folders that can receive PDF documents via SMTP from document scanners.

2013-04-04

Starting & Stopping Process Invocation

When running the coils.workflow.manager is constantly trolling for queued processes that are ready to run.  On the other hand the System Administrator needs to perform some maintenance and needs the system quiescent.... what to do?  Use the coils-workflow-control tool.  This tool [command] can be used on any node in an OpenGroupware Coils cluster to change the state of the coils.workflow.manager component.
coils-workflow-control --disable
coils-workflow-control --enable
coils-workflow-control --scan
Options
  • --disable : Disables the coils.workflow.manager component's starting of queued processes.  When disabled no new processes will be started.  Processes may still be created and queued, but execution will not commence.  Currently running processes will not be stopped, but no further processes will start.  It is 'safe' to kill the engine once no processes are running and the manager is in a disabled state.  NOTE: Currently the coils.workflow.manager component will always start in an enabled state even if it was disabled when shutdown - but it always waits approximately three minutes after start-up before starting any processes.
  • --enable : Enables the coils.workflow.manager component's starting of queued processes.  This just undoes a --disable request.  Enabling an already enabled engine has no effect.
  • --scan : Request that the coils.workflow.manager component immediately check its process queue for available processes.  Normally there is no reason to use --scan as the component performs this check whenever a running process completes or fails, a new process is queued (via WebDAV for example) as well as at regular intervals.
If the Python module procname is installed on the workflow host processes will change their OS process name to contain the OpenGroupware Coils Process Id - with just this tool and regular sys-admin tools like "top" it is simple to monitor and control the execution of workflow processes.

2013-02-24

Mounting Via WebDAV From Windows

As of Windows XP the Microsoft Filesystem Redirector, by default, only supports DIGEST authentication.  To utilize a WebDAV server using BASIC authentication the support has to be enabled via a registry key.  The WebDAV client is configured via keys in the system hive at "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WebClient\Parameters".  Create a DWORD key named "UseBasicAuth" with a value of 0, 1, or 2.
  • 0 - Basic authentication disabled, this is equivalent to the default if the UseBasicAuth key is not  set.
  • 1 - Basic authentication is allowed for WebDAV provided that the connection is protected by SSL/TLS.
  • 2 - Basic authentication is allowed for WebDAV regardless if the connection is protected.
 A value of "1" is recommended for any type of production network;  for debugging and development purporses "2" is useful as unprotected connection can be watched with any packet capture tool such as WireShark.  Only allow unprotected connections if you are comfortable with them being seen by any third party [seeing includes both the username and password].  Attempting to mount a volume from a server using BASIC authentication that does not match the requirements set by "UseBasicAuth" will generally result in a "System Error 67".
Regardless of how user enters their credentials when prompted Windows will send a username qualified by the NetBIOS domain of either the current machine or the current domain.  If your OpenGroupware Coils server does not use an authentication scheme where qualified names are recognized you can set the StripAuthenticationDomain server default;  with that default enabled the server still strip the NetBIOS domain from the from the beginning of any login credentials before authentication is attempted.  However that operation also means that any usernames containing a "\" character will fail to authenticate.
coils-server-config --directive=StripAuthenticationDomain --value=YES
Mounting OpenGroupware Coils as drive X:
Typically if your attempts to authenticate fail with a Windows "System Error 5 has occurred" it related to sending qualified usernames to a server that expects unqualified usernames.
With BASIC authentication enabled and automatically unqualifying usernames it should be possible to mount your OpenGroupware Coils' WebDAV presentation on a Windows Workstation.
More technical information for either using the Microsoft Windows Filesystem Redirector or OpenGroupware Coils authentication options can be found at the Wiki of the SourceForge project.

2013-02-19

XLSUpload

How often have you looked at a spreadsheet, sighed, and said "Ugh, now I have to get this into a database."  Often this involves upload tools, and mapping columns, exporting to a CSV file... lots of tedious error-prone steps.  Now a quick and simple OIE workflow route can solve this problem! The combination of the new xmlToXMLAction action and the sqlInsertAction allows the creation a potent and generic workflow route - the ability to upload an XLS file directly into a database table!

xlsToXmlAction
The OIE xlsToXmlAction allows the transformation of any columnar XLS data-stream into a StandardXML data-stream.  Utilizing this action provides a very linear way for savvy users to import data into a given database. This action transforms the first sheet in the XLS workbook into StandardXML making the following assumptions:
  • The first row corresponds to field names
  • The structure of the sheet is columnar with each row corresponding to StandardXML row having fields label as found in the first row.
Data type detection of the values is automatic.  Remember that with XLS there are actually only three data types: float, string, and date-time. All numeric values in XLS are floating-point; Excel has no concept of integer values.  If a cell is none of these types the value of the cell will be taken as a NULL (this will include empty cells).
One caveat with this transformation is that the column headings [which become field names] must be valid XML element names.  In an effort to work around common column heading conventions any spaces, slashes, or hyphens in the heading values will be replaced with underscores;  spaces, etc... are not legal in XML element names.   This shouldn't be an issue for this usage of xlsToXMLAction as [since we are targeting an RDBMS table] it seems very unlikely you would have those characters in a field name.
<action name="actionActivity" id="000100" extensionAttributes="XLSUpload/000100">
<input property="TranslatedInput" formatter="StandardXML"/>
<output><source property="InputMessage"/></output>
<attributes xmlns="">
<extension name="activityName">xlsToXmlAction</extension>
<extension name="description"/>
</attributes>
</action>
Text 1: Example xlsToXmlAction stanza;  translates the InputMessage to TranslatedInput.    
sqlInsertAction
The sqlInsertAction does exactly what it's name implies - it inserts the contents of the StandardXML data-stream into an RDBMS table.  The target table can be specified either via an attribute of the ResultSet element of the data-stream itself or by the tableName parameter of the sqlInsertAction stanza in the workflow markup.  If the data had been transformed from the source format into StandardXML via a format description the data-stream would most likely already contain the name of the table to which the rows correspond.  But when transforming an arbitrary XLS document into StandardXML the table name in the the data-stream  will be "__undefined__", so it will have to be specified in
the action.  The most flexible way to achieve this is to use a process XATTR and reference if by label substitution.
<action name="actionActivity" id="000150" extensionAttributes="XLSUpload/000150">
  <output><source property="TranslatedInput"/></output>
  <attributes xmlns="">
    <extension name="activityName">sqlInsertAction</extension>
    <extension name="dataSource">myDSName</extension>
    <extension name="tableName">$__XATTR_TABLENAME__;</extension>
    <extension name="description"/>
  </attributes>
</action>
Text 2: Example sqlInsertAction that inserts the contents of TranslatedInput into the specified table of the named data source.
Route Creation
Grab the example markup from the wiki and use it to create a route in your OIE instance.  You should get a 301 response indicating that a collection has been representing the route.   Your target database need to be define in the OIE instance's OIESQLDataSources default.
curl -u adam -T markup.bpml http://coils.example.com/dav/Workflow/Routes
Invocation
Now we have a route we can use to upload an XLS document into an RDBMS table - provided the columns of the XLS file correspond to the column of
the target table.  Initiating the upload is as simple as a curl command:
curl -u adam -T myXLSFile.xls
"http://coils.example.com/dav/Workflow/Routes/XLSUpload?tableName=myTable"
The contents of the myXLSFile.xls will be transformed to StandardXML and inserted into the myTable table. It is still up to you to verify the data you are uploading has appropriate types and will not violate any constraints applied to the table; but you should get an informative error message if something fails.  This is a much more reliable mechanism to upload arbitrary data than exporting to CSV and all the subsequent steps.

2012-12-05

OpenGroupware Coils Optional Dependencies

Installing OpenGroupware Coils via easy_install or pip will automatically install a variety of required dependencies.  There are however serveral optional dependencies.  The server will run without the availability of these modules but some features will be automatically disabled.  Installing the following modules will extend the capabilities of your OpenGroupware Coils instance.
  • informixdb - Allows an Informix RDBMS to be used as an external data source.  This means you can define an Informix database via OIESQLDataSources and utilize it for SQL related workflow actions such as sqlInsertAction, sqlSelectAction, etc... 
  • z3c.rml / zope.interface - Installing these modules will enable the conversions of XML to PDF.  Using XSLT transforms you can turn your data into RML, and then using the rmlToPDF workflow action convert the data into sophisticated PDF documents.
  • pysmbc - Enables the OIE workflow engine to retrieve and upload files from or to a CIFS / SMB volume. 
  • PIL - Enables the OSSF module for retrieving thumbnails of images stored in OpenGroupware projects or attachments. 
  • paramiko - Presence of this module activates the SSH related OIE workflow actions such as sshExecuteCmmand
  • python-ldap - The availability of the LDAP module enables both support for LDAP authentication (such as when using Active Directory, OpenLDAP, etc... as your user database) and support for querying an LDAP DSA from a workflow process using OIE's ldapSearchAction.
Once your OpenGroupware Coils instance is up and running you should evaluate which additional, optional, dependencies would be useful in your environment.

2012-12-02

0.1.49rc7 & CardDAV-Sync

With the release of 0.1.49rc7 comes a collection of bugfixes and enhancements - most notably is the early arrival of service discovery support for the CardDAV-Sync Android application.  CardDAV-Sync, as the name implies, adds CardDAV support to an Android device. 
OpenGroupware Coils 0.1.49rc7 has been uploaded to PyPI and SourceForge.
In order to provide easy configuration CardDAV-Sync uses the well_known standard - so you can avoid entering tedious WebDAV paths to collections of contacts (aka address books).  The new well_known support in 01.49rc7 allows CardDAV-Sync to discover the location of the address books and present them clearly to the user.  So configuring a CardDAV-Sync client requires nothing more than the host name, user name, password, and selecting the address book you want on your device.  Of course you can create multiple accounts in order to have multiple address books.
Thanks to service discovery CardDAV-Sync configuration is simple.
This release also changes the WebDAV hierarchy for presenting contact collections to clients - this is necessitated by the new support for user-defined addressbooks.  Now all contact collections are available under "/dav/Contacts" and the previous contents of "/dav/Contacts" (all the server's contacts) are moved to "/dav/Contacts/All Contacts". 
For convenience the "All Contacts" addressbook is hidden from the CardDAV-Sync client (based on user agent detection) - as all contacts can be simply too large a collection for easy use on a small and resource constrained device such as a phone.
The address books "Favorites" and "Users" are fixed and cannot be renamed or deleted by the user - these contain favorite contacts and all the server's users, respectively.  The "Users" address book is read-only.  Creating a contact in the favorite address book will automatically make that contact a favorite.
New address books can be created via WebDAV by simply creating a new collection [folder] in the "/dav/Contacts" collection [folder].
The contents of /dav/Contacts viewed in GNOME's Nautilus file browser.  New address books can be created simply by creating a new folder.
Android 4.x users can use CardDAV-Sync alone, but users on previous versions of Android must all install the ContactEditor application as earlier Android versions did not permit the editing of contacts on third-party servers. 
Even user's of 4.x devices may want to look at DMFS' excellent CardDAV-Sync Pro and ContactEditor-Pro applications as they extend the capabilities of contact management on Android.  If you are using a device linked to a proprietary marketplace these applications should be available there, otherwise they can be acquired from the AndroidPIT.
Technical information regarding support forthe CardDAV-Sync client can be found at that client's wiki page of the OpenGroupware Coil's development wiki.

2012-11-20

GRPUG Presentation on OIE

Last night I presented at the Grand Rapids Python Users Group about the OpenGroupware Coils project and specifically the workflow component: OpenGroupware Integration Engine.
Click to download presentation (PDF).
This presentation convers the basic terminology and concepts behind the OpenGroupware Integration Engine - how to think about process modelling with OIE.  For full details on how to use OIE, as always, see the WMOGAG document.
Note: the OIE section of the presentation is featured above. The full presentation, as given, with a few more slides and some general Python bits is also available here.

2012-11-12

New Edition of WMOGAG Coils Available

A new edition of WMOGAG Coils [Whitemice Consulting's OpenGroupware Administrator's Guide] has been uploaded to SourceForge; download here.  This edition documents the new features available in recent production and RC releases include using the workflow scheduler, the new translateAction, and configuration of the services for SSH access to other hosts.
As always please report any comments, recommendations, or even typographic errors to either the Coil's SourceForge project or the Coil's maillist.

2012-11-09

The Workflow Scheduler

With the 0.1.49 release [and currently available in the 0.1.49rc releases] OpenGroupware Coils exposes the workflow scheduler via WebDAV at the path "/dav/Workflow/Schedule".  This collection [folder] allows enumeration of schedule entries by WebDAV clients or via REST, as well as the creation and deletion of schedule entries. 

Each entry is represented by a JSON serialized resource.  These resources can be retrieved via GET, created via PUT, or removed via DELETE. For REST clients the resource "/var/Workflow/Schedule/.contents" will return a JSON encoded list of all the folder entries.  Via either RPC mechanism clients only see the schedule entries available to their own security context.

$ cat send-request.json
{"routeId":1158480,"priority":300,"repeat":1,"type":"interval","minutes":2}
$ curl -vvvv -u adam:******* -X PUT --upload-file send-request.json http://127.0.0.1:8080/dav/Workflow/Schedule/request
...
< Location: /dav/Workflow/Schedule/0f6bd804a4f14cf6a2807dad625e2bb1.json
< X-OpenGroupware-ScheduleEntryUUID: 0f6bd804a4f14cf6a2807dad625e2bb1 

...
Create a scheduler entry where an instance of the route with the objectId 1158480 will be created two minutes from the current time having a priority of 300.  The schedule will only occur once.
The required keys in the JSON data are: "routeId" and "type".  The keys "priority", "contextId", "xattrDict", "attachmentUUID", and "repeat" are optional.  If unspecified the priority for scheduled entries will be 200, they will repeat without limit according to their schedule, and have the context of the currently authenticated user.  A user may only create schedule entries with contexts that are available to them. Additional keys are required or optional based upon the value of "type" which describes the kind of scheduling entry to create: "simple", "interval", or "cron".

If the workflow requires an input message an attachment may be created via the AttachFS protocol and the UUID of that attachment then specified in the scheduling entry; at run time the content of the attachment will be read into the input message of the new process.  The attachment's UUID is sent in the  "attachmentUUID" attribute of the scheduling request.
In the future it will also be possible to specify a project document as an input message.  Project documents have the advantages over attachments that they have a security descriptor and can be modified in-place.
Using the "xattrDict" attribute a dictionary of key and values can be declared that will be applied to a new process as its XATTR values.

An "interval" schedule type must specify at least one of the following values: "weeks", "days", "hours", "minutes", "seconds".  The type of any of these values must be an integer.  Multiple values may be specified and their accumulative value will specify the frequency the process will be run.

A "simple" schedule requires on a "date" value; this specifies a specific time when the process will be created.  "simple" schedule entries do not repeat. 

The "cron" type schedule entry requires at least one of the following values: "year", "month", "day", "weekday", "hour", or "minute".  All these are crontab style pattern matching strings; both division and comma separated lists are supported.  Any of these values if not specified default to the string "*".

All date/time values must be in UTC and expressed in either "%Y%m%dT%H%M%S" or "%Y-%m-%d %H:%M:%S" format. In either format the seconds value is optional, and if provided is ignored.  Remember that the OIE scheduler is a best-effort scheduler - it will attempt to create the process within a minute of the time indicated by the schedule, but delay is possible.  Also the scheduler only creates the process and queues it for execution.  It is up to the workflow manager component when the process will actually be executed/performed.  A variety of factors can influence the timing of process's execution, including singletons, system load, and administrative holds.

A schedule entry retrieved from the server will have all possible fields filled in as well as having two additional read-only fields: "iterationsPerformed" and "iterationsRemaining".  "iterationsPerformed" records how many times the schedule entry has been executed and "iterationsRemaining" indicates how many addition additional iterations remain.  A value of -1 in "iterationsRemaining" means the schedule entry will continue indefinitely; this field will only have a real value if "repeat" was specified when the scheduler entry was created.  If the entry has a repeat limit the "iterationsRemaining" serves as a count down to the expiration of the entry, once the remaining iteration count reaches zero the schedule entry will be automatically deleted.
{"dayOfWeek": "3,5", "week": "*", "nextIteration": "2012-11-11T23:00:00", "month": "*", "second": "0", "iterationsRemaining": -1, "year": "*", "day": "*", "minute": "0", "attachmentUUID": null, "xattrDict": {"start": "$__MONTHSTART__;", "end": "$__TODAY__;"}, "contextId": 15211340, "UUID": "{c658993a-dcf1-410b-8003-5dc134feb564}", "hour": "23", "routeId": 70470079, "priority": 200, "iterationsPerformed": 7, "type": "cron"}
A complete schedule entry as retrieved from the server.  Each time this entry creates a process that process will have a "start" XATTR whose value represents the date for first of the current month and an "end" XATTR that represents the current date. 
For additional flexibility the OIE built-in labels are supported in the value of process XATTRs (specified in the schedule entry's "xattrDict" value).  The values will be substituted each time a new process is created from the schedule entry.  Using built-in labels is especially useful when creating schedules for workflow processes that require a date range.
Process XATTR's and label substitution is documented in the Coils edition of WMOGAG.

2012-08-03

Snurtle: get-performance

The latest version of OpenGroupware Coils + Snurtle provides a new command "get-performance". get-performance will retrieve the performance counters recorded by the coils.adminstrator component concerning Logic layer operations.
Output is a single line for each Logic operation that includes its frequency, average duration, minimum and maximum duration, and the total system time that has been consumed performing that operation..
Aside: Snurtle's command completion only requires you to type enough of a command to make the command unique, so "get-perf" is plenty.
Snurtle has also recently added the commands lock, unlock, list-locks, and create-contact.  The built-in help covers the syntax for these as well as get-performance.


2012-08-02

Makefile Deployment

In the latest commits to OpenGrouwpare we've added "Makefile deployment".   This allows for simple and rapid creation of a development instance.

So long as you have PostgreSQL and RabbitMQ available you can just
  1. Check out the code
  2. Tweak the first few lines of the Makefile to tell it your PostgreSQL database name, user name, etc...
  3. If you already have a database then touch 0_database_complete.txt to indicate that the database has already been initialized.
  4. To fire-up a working test instance just run "make run-master"
  5. Point your WebDAV client at 127.0.0.1:8080/dav
That's it! The make file will create a Python virtualized container to handle dependencies, retrieve the required dependencies, install a bare minimum configuration, create a document root, and start the service.  You will have the administrative account with the password you specified in the first few lines of the make file.

You can run "make install-dependencies" if you want to update the Coils dependencies check  - just delete the 0_dependencies_installed.txt file which indicates that dependencies have already been isntalled.

Aside: You may also want to add some of the optional Python modules to your virtual environment such as PIL, informixdb, z3c.rml, procname, or YaJL.  These are not required by OpenGroupware Coils and are not automatically installed.

Simple steps to provision PostgreSQL & RabbitMQ
sudo sudo -u postgres createuser --password --no-superuser --no-created --no-createrole OGoDev
sudo sudo -u postgres createdb -E UTF-8 -O OGoDev OGoDev0
sudo /usr/sbin/rabbitmqctl add_user OGoDev {AMQPASSWORD}
sudo /usr/sbin/rabbitmqctl add_vhost OGoDev0
sudo /usr/sbin/rabbitmqctl set_permissions -p OGoDev0 OGoDev ".*" ".*" ".*"
Text1: Provision PostgreSQL and RabbitMQ
This will give you a PG and AMQ user "OGoDev" with the password specified, a database named "OGoDev0", and an AMQ virtual host named "OGoDev0".  These values are set in the top of the Makefile.

2012-05-24

Creating An OpenGroupware Coils Development Instance

These instructions provide a simple recipe to create an OpenGroupware Coils development instance.  Previously creating an instance has admittedly been a bit tedious; but in the last couple of weeks an effort has been made to simplify deployment.

Aside: Always see the Administrator's Guide (WMOGAG) for complete documentation.  Ask questions on the coils-project mailing list.


Step#1) Provisioning dependencies

Both PostgreSQL and RabbitMQ must be provisioned on the host as well as the dependencies of the required Python modules.  The dependencies are easily met on either CentOS6 (with RPMForge enabled) and openSUSE 12.1.  Installing PostgreSQL and RabbitMQ should be performed using the standard methods.
CentOS6  rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
  rpm -Uvh
    http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
  yum install python python-ldap  python-devel python-setuptools gcc \
              make binutils libxml2-devel libxslt-devel libyaml \
              libyaml-devel postgresql-devel postgresql-libs cmake \
              gcc-c++ freetype-devel libpng-devel libjpeg-devel \
              libsmbclient-devel

openSUSE
  zypper in python python-ldap  python-devel python-Distutils2 gcc make \
            binutils libxml2-devel libxslt-devel libyaml libyaml-devel \
            postgresql-devel postgresql cmake gcc-c++ freetype2-devel \
           libpng14-devel libjpeg62-devel libsmbclient-devel
Text: These commands will install the required system prerequisites for the required Python modules.
A simple method for installing RabbitMQ on openSUSE 12.1 is described  in the "Idjit's Guide To Installing RabbitMQ on openSUSE 12.1"

Step#1.1) Installed a compatible version of the YAJL library.


The YAJL library provides SAX like stream-processing of JSON data.  This library is required by the ijson Python module. This module is not required by OpenGroupware Coils so this step can be skipped;  but usage of this module (and thus library) are strongly recommended
curl -o yajl-1.0.11.tar.gz http://gentoo.osuosl.org/distfiles/yajl-1.0.11.tar.gz
tar xzvf yajl-1.0.11.tar.gz
cd lloyd-yajl-f4baae0/
mkdir build
cd build
cmake ..
sudo make install
sudo /sbin/ldconfig
Text: Fetching, building, and installing the YAJL library.


Step#1.2) Creating PostgreSQL role & database.

A role must be created in the PostgreSQL engine; the recommendation is to name the role "OGoDev".  Remember the password you enter for this role!
sudo -u postgres createuser --password --no-superuser --no-created --no-createrole OGoDev
Text: Creating the "OGoDev" role.


Once a role has been created a database must be created.  The default name for the OpenGroupware database is typically "OGo" but in order to disambiguate the development database(s) from a production instance the recommendation is to name the database "OGoDev?".  I prefer to make ten databases [OGoDev0, OGoDev1, ... OGoDev9] so I have several to play with - this is especially useful for testing.
sudo -u postgres createdb -E UTF-8 -O OGoDev OGoDev0
Text: Creating a database named "OGoDev0" owned by the OGoDev role.

Step#1.3) Define RabbitMQ role & vhost(s).

The OpenGroupware Coils components also need a context with which to connect to the RabbitMQ service.  Keeping to the same role naming scheme you used with PostgreSQL is helpful for keeping this straight.  Just as with PostgreSQL I prefer to define ten roles and virtual hosts [OGoDev0, OGoDev1, .... OGoDev9] so that I can switch the instance between fresh instances.  The commands listed here will create a role with a password, create an eponymous virtual host, and then grant that role full permissions over its vhost.
sudo rabbitmqctl add_user OGoDev0 {AMQPASSWORD}
sudo rabbitmqctl add_vhost OGoDev0
sudo rabbitmqctl set_permissions -p OGoDev0 OGoDev0 ".*" ".*" ".*"
Text: Creating a role and virtual host in RabbitMQ.
A RabbitMQ virtual host is a means to allow multiple services to use the same message broker service while remaining isolated from each other/

Step#2) Create the container for the development install.
virtualenv OGo
cd OGo
echo -e '\nexport PYTHONPATH="$VIRTUAL_ENV/coils/src"\n' >> bin/activate
echo -e '\nexport OGO_SERVER_ROOT="$VIRTUAL_ENV/root"\n' >> bin/activate
Text: Creating a virtualenv instance for development.
The Python module virtualenv provides a convenient means for creating a container for performing Python development.  This [virtualenv] module should be installed in the host's global Python site-packages.

For this container it is also helpful to add the Coils code checkout to the Python path [via PYTHONPATH] and the establish a server root [via OGO_SERVER_ROOT] within the development container.  This customer server root keeps file created by the instance, including configuration, confined to the development container.
Aside: If OGO_SERVER_ROOT is not defined the server root will default to "/var/lib/opengroupware.org".

Step#2) Populate the container.

Now the container needs to be activated and the required Python modules installed.  The ". bin/activate" builds the environment for the container and should be performed from the root directory of the container whenever working with the development instance.
. bin/activate
pip install lxml==2.3.4
pip install psycopg2==2.4.5
pip install pytz
pip install sqlalchemy==0.7.4
pip install xlrd==0.7.7
pip install xlwt==0.7.4
pip install pysmbc==1.0.13
pip install procname==0.3
pip install PyYAML==3.10
pip install ijson==0.8.0
pip install apscheduler==2.0.3
pip install python-dateutil==2.1
pip install vobject==0.8.1c
pip install http://effbot.org/downloads/Imaging-1.1.7.tar.gz
Text: Activating the virtualenv container and installing Python modules.

Now we are ready to checkout the OpenGroupware Coils code into the container.

Read/Write
  hg clone ssh://{YOURUSERNAME}@hg.code.sf.net/p/coils/code coils
Read-Only
  hg clone http://hg.code.sf.net/p/coils/code coils
Text: Checking out the OpenGroupware Coils code base.


Step#3) Test the development container.

The "coils-dependency-check" tool tries to load all the modules used by OpenGroupware Coils and reports success or failure.  This verifies that the Python requirements for operating an OpenGroupware Coils instance have been met.

(OGo)awilliam@workstation:~/OGo> cd coils/src
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-dependency-check
OK: Module xlwt (XLS<2007 write support) available.
OK: Module sqlalchemy (Object Relational Modeling) available.
OK: Module coils.foundation.api.dateutil (Date & Time Arithmatic) available.
OK: Module pytz (Python Time Zone tables) available.
OK: Module xlrd (XLS<2007 read support) available.
OK: Module coils.foundation.api.vobject (vCard and vEvent parsing) available.
OK: Module lxml (SAX & DOM XML Processing) available.
OK: Module PIL (Python Imaging Library) available.
OK: Module psycopg2 (PostgreSQL RDBMS connectivity) available.
OK: Module base64 (Encode and decode Base64 data) available.
OK: Module coils.foundation.api.elementflow (Streaming XML Creation) available.
OK: Module coils.foundation.api.pypdf (Simple PDF Operations) available.
OK: Module yaml (YAML parser & serializer) available.
WARN: Module informixdb (Informix RDBMS connectivity) not available.

1 database connectivity modules found.
 * Make sure the RDBMS you intend to use for the SQLalchemy  *
 * ORM is installed and operational.                         *

1 package warnings found.
 * You are missing packages that extend the operation and    *
 * capacity of the OpenGroupware Coils service.  The service *
 * will provide core functionality but some features,        *
 * particularly in regard to OIE, may not be available. It   *
 * is recommended you install the appropriate packages.      *
Text: Run the coils-dependency-check tool to verify your Python installation.
The warning ("WARN") indicates that an optional module could not be loaded.  Warnings indicate that the service will operate but with potentially reduced functionality.  Any error ["ERROR"] means the service will fail to operate.

Step#4) Initialize the development instance.

Now that the environment is ready the Coils tools can be used to initialize and configure the instance.
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-install  --user=awilliam --group=users --log=../../coils.log
Text: Initialize the server's installation; this create the required structure in the server's document root - in this case the value defined by $OGO_SERVER_ROOT.


(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --bootstrap
Initialized a new server defaults file.
Loaded configuration BLOB successfully.
Text: Initialize the instance's configuration with default values.
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive LSConnectionDictionary --value "{'databaseName': 'OGoDev0', 'hostName': '127.0.0.1', 'password': '{SQLPASSWORD}', 'port': 5432, 'userName': 'OGoDev'}"
Text: Configure the connection to the PostgreSQL database.

(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-server-config --directive AMQConfigDictionary --value "{'hostname': '127.0.0.1', 'password': '{AMQPASSWORD}', 'port': 5672, 'username': 'OGoDev0', 'vhost': 'OGoDev0' }"
Text: Configure the connection to the RabbitMQ message broker.
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-initialize-database --initdb --password={COILSADMINPASSWORD}
Text: Create the initial database schema and provision the administrative "ogo" account with the specified password.

Step#5) Test the instance.

A simple way to test the instance is to start just the HTTP component.
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-service-http --asuser
Text: State the Coils HTTP component.
If this component is running you should be able to connect to and browse the WebDAV hierarchy at http://127.0.0.1:8080/dav (authenticate as the OpenGroupware Coils administrative account "ogo" and the COILSADMINPASSWORD you provided.  Browsing the hierarchy can be performed with Nautilus, cadaver, or any WebDAV client.  Use your systems break sequence [typically Ctrl-C] to stop the component.
(OGo)awilliam@workstation:~/OGo/coils/src> tools/coils-master-service --asuser
Text: Start the Coil's master service which will start and manage an instance of every available component.
The coils-master-service tool can be used in the same manner to start-up the full suite of service components providing HTTP and workflow services (include the TCP/9100 and SMTP listeners).