Showing posts with label workflow. Show all posts
Showing posts with label workflow. Show all posts

2013-04-04

Starting & Stopping Process Invocation

When running the coils.workflow.manager is constantly trolling for queued processes that are ready to run.  On the other hand the System Administrator needs to perform some maintenance and needs the system quiescent.... what to do?  Use the coils-workflow-control tool.  This tool [command] can be used on any node in an OpenGroupware Coils cluster to change the state of the coils.workflow.manager component.
coils-workflow-control --disable
coils-workflow-control --enable
coils-workflow-control --scan
Options
  • --disable : Disables the coils.workflow.manager component's starting of queued processes.  When disabled no new processes will be started.  Processes may still be created and queued, but execution will not commence.  Currently running processes will not be stopped, but no further processes will start.  It is 'safe' to kill the engine once no processes are running and the manager is in a disabled state.  NOTE: Currently the coils.workflow.manager component will always start in an enabled state even if it was disabled when shutdown - but it always waits approximately three minutes after start-up before starting any processes.
  • --enable : Enables the coils.workflow.manager component's starting of queued processes.  This just undoes a --disable request.  Enabling an already enabled engine has no effect.
  • --scan : Request that the coils.workflow.manager component immediately check its process queue for available processes.  Normally there is no reason to use --scan as the component performs this check whenever a running process completes or fails, a new process is queued (via WebDAV for example) as well as at regular intervals.
If the Python module procname is installed on the workflow host processes will change their OS process name to contain the OpenGroupware Coils Process Id - with just this tool and regular sys-admin tools like "top" it is simple to monitor and control the execution of workflow processes.

2013-02-19

XLSUpload

How often have you looked at a spreadsheet, sighed, and said "Ugh, now I have to get this into a database."  Often this involves upload tools, and mapping columns, exporting to a CSV file... lots of tedious error-prone steps.  Now a quick and simple OIE workflow route can solve this problem! The combination of the new xmlToXMLAction action and the sqlInsertAction allows the creation a potent and generic workflow route - the ability to upload an XLS file directly into a database table!

xlsToXmlAction
The OIE xlsToXmlAction allows the transformation of any columnar XLS data-stream into a StandardXML data-stream.  Utilizing this action provides a very linear way for savvy users to import data into a given database. This action transforms the first sheet in the XLS workbook into StandardXML making the following assumptions:
  • The first row corresponds to field names
  • The structure of the sheet is columnar with each row corresponding to StandardXML row having fields label as found in the first row.
Data type detection of the values is automatic.  Remember that with XLS there are actually only three data types: float, string, and date-time. All numeric values in XLS are floating-point; Excel has no concept of integer values.  If a cell is none of these types the value of the cell will be taken as a NULL (this will include empty cells).
One caveat with this transformation is that the column headings [which become field names] must be valid XML element names.  In an effort to work around common column heading conventions any spaces, slashes, or hyphens in the heading values will be replaced with underscores;  spaces, etc... are not legal in XML element names.   This shouldn't be an issue for this usage of xlsToXMLAction as [since we are targeting an RDBMS table] it seems very unlikely you would have those characters in a field name.
<action name="actionActivity" id="000100" extensionAttributes="XLSUpload/000100">
<input property="TranslatedInput" formatter="StandardXML"/>
<output><source property="InputMessage"/></output>
<attributes xmlns="">
<extension name="activityName">xlsToXmlAction</extension>
<extension name="description"/>
</attributes>
</action>
Text 1: Example xlsToXmlAction stanza;  translates the InputMessage to TranslatedInput.    
sqlInsertAction
The sqlInsertAction does exactly what it's name implies - it inserts the contents of the StandardXML data-stream into an RDBMS table.  The target table can be specified either via an attribute of the ResultSet element of the data-stream itself or by the tableName parameter of the sqlInsertAction stanza in the workflow markup.  If the data had been transformed from the source format into StandardXML via a format description the data-stream would most likely already contain the name of the table to which the rows correspond.  But when transforming an arbitrary XLS document into StandardXML the table name in the the data-stream  will be "__undefined__", so it will have to be specified in
the action.  The most flexible way to achieve this is to use a process XATTR and reference if by label substitution.
<action name="actionActivity" id="000150" extensionAttributes="XLSUpload/000150">
  <output><source property="TranslatedInput"/></output>
  <attributes xmlns="">
    <extension name="activityName">sqlInsertAction</extension>
    <extension name="dataSource">myDSName</extension>
    <extension name="tableName">$__XATTR_TABLENAME__;</extension>
    <extension name="description"/>
  </attributes>
</action>
Text 2: Example sqlInsertAction that inserts the contents of TranslatedInput into the specified table of the named data source.
Route Creation
Grab the example markup from the wiki and use it to create a route in your OIE instance.  You should get a 301 response indicating that a collection has been representing the route.   Your target database need to be define in the OIE instance's OIESQLDataSources default.
curl -u adam -T markup.bpml http://coils.example.com/dav/Workflow/Routes
Invocation
Now we have a route we can use to upload an XLS document into an RDBMS table - provided the columns of the XLS file correspond to the column of
the target table.  Initiating the upload is as simple as a curl command:
curl -u adam -T myXLSFile.xls
"http://coils.example.com/dav/Workflow/Routes/XLSUpload?tableName=myTable"
The contents of the myXLSFile.xls will be transformed to StandardXML and inserted into the myTable table. It is still up to you to verify the data you are uploading has appropriate types and will not violate any constraints applied to the table; but you should get an informative error message if something fails.  This is a much more reliable mechanism to upload arbitrary data than exporting to CSV and all the subsequent steps.

2012-11-20

GRPUG Presentation on OIE

Last night I presented at the Grand Rapids Python Users Group about the OpenGroupware Coils project and specifically the workflow component: OpenGroupware Integration Engine.
Click to download presentation (PDF).
This presentation convers the basic terminology and concepts behind the OpenGroupware Integration Engine - how to think about process modelling with OIE.  For full details on how to use OIE, as always, see the WMOGAG document.
Note: the OIE section of the presentation is featured above. The full presentation, as given, with a few more slides and some general Python bits is also available here.

2012-11-09

The Workflow Scheduler

With the 0.1.49 release [and currently available in the 0.1.49rc releases] OpenGroupware Coils exposes the workflow scheduler via WebDAV at the path "/dav/Workflow/Schedule".  This collection [folder] allows enumeration of schedule entries by WebDAV clients or via REST, as well as the creation and deletion of schedule entries. 

Each entry is represented by a JSON serialized resource.  These resources can be retrieved via GET, created via PUT, or removed via DELETE. For REST clients the resource "/var/Workflow/Schedule/.contents" will return a JSON encoded list of all the folder entries.  Via either RPC mechanism clients only see the schedule entries available to their own security context.

$ cat send-request.json
{"routeId":1158480,"priority":300,"repeat":1,"type":"interval","minutes":2}
$ curl -vvvv -u adam:******* -X PUT --upload-file send-request.json http://127.0.0.1:8080/dav/Workflow/Schedule/request
...
< Location: /dav/Workflow/Schedule/0f6bd804a4f14cf6a2807dad625e2bb1.json
< X-OpenGroupware-ScheduleEntryUUID: 0f6bd804a4f14cf6a2807dad625e2bb1 

...
Create a scheduler entry where an instance of the route with the objectId 1158480 will be created two minutes from the current time having a priority of 300.  The schedule will only occur once.
The required keys in the JSON data are: "routeId" and "type".  The keys "priority", "contextId", "xattrDict", "attachmentUUID", and "repeat" are optional.  If unspecified the priority for scheduled entries will be 200, they will repeat without limit according to their schedule, and have the context of the currently authenticated user.  A user may only create schedule entries with contexts that are available to them. Additional keys are required or optional based upon the value of "type" which describes the kind of scheduling entry to create: "simple", "interval", or "cron".

If the workflow requires an input message an attachment may be created via the AttachFS protocol and the UUID of that attachment then specified in the scheduling entry; at run time the content of the attachment will be read into the input message of the new process.  The attachment's UUID is sent in the  "attachmentUUID" attribute of the scheduling request.
In the future it will also be possible to specify a project document as an input message.  Project documents have the advantages over attachments that they have a security descriptor and can be modified in-place.
Using the "xattrDict" attribute a dictionary of key and values can be declared that will be applied to a new process as its XATTR values.

An "interval" schedule type must specify at least one of the following values: "weeks", "days", "hours", "minutes", "seconds".  The type of any of these values must be an integer.  Multiple values may be specified and their accumulative value will specify the frequency the process will be run.

A "simple" schedule requires on a "date" value; this specifies a specific time when the process will be created.  "simple" schedule entries do not repeat. 

The "cron" type schedule entry requires at least one of the following values: "year", "month", "day", "weekday", "hour", or "minute".  All these are crontab style pattern matching strings; both division and comma separated lists are supported.  Any of these values if not specified default to the string "*".

All date/time values must be in UTC and expressed in either "%Y%m%dT%H%M%S" or "%Y-%m-%d %H:%M:%S" format. In either format the seconds value is optional, and if provided is ignored.  Remember that the OIE scheduler is a best-effort scheduler - it will attempt to create the process within a minute of the time indicated by the schedule, but delay is possible.  Also the scheduler only creates the process and queues it for execution.  It is up to the workflow manager component when the process will actually be executed/performed.  A variety of factors can influence the timing of process's execution, including singletons, system load, and administrative holds.

A schedule entry retrieved from the server will have all possible fields filled in as well as having two additional read-only fields: "iterationsPerformed" and "iterationsRemaining".  "iterationsPerformed" records how many times the schedule entry has been executed and "iterationsRemaining" indicates how many addition additional iterations remain.  A value of -1 in "iterationsRemaining" means the schedule entry will continue indefinitely; this field will only have a real value if "repeat" was specified when the scheduler entry was created.  If the entry has a repeat limit the "iterationsRemaining" serves as a count down to the expiration of the entry, once the remaining iteration count reaches zero the schedule entry will be automatically deleted.
{"dayOfWeek": "3,5", "week": "*", "nextIteration": "2012-11-11T23:00:00", "month": "*", "second": "0", "iterationsRemaining": -1, "year": "*", "day": "*", "minute": "0", "attachmentUUID": null, "xattrDict": {"start": "$__MONTHSTART__;", "end": "$__TODAY__;"}, "contextId": 15211340, "UUID": "{c658993a-dcf1-410b-8003-5dc134feb564}", "hour": "23", "routeId": 70470079, "priority": 200, "iterationsPerformed": 7, "type": "cron"}
A complete schedule entry as retrieved from the server.  Each time this entry creates a process that process will have a "start" XATTR whose value represents the date for first of the current month and an "end" XATTR that represents the current date. 
For additional flexibility the OIE built-in labels are supported in the value of process XATTRs (specified in the schedule entry's "xattrDict" value).  The values will be substituted each time a new process is created from the schedule entry.  Using built-in labels is especially useful when creating schedules for workflow processes that require a date range.
Process XATTR's and label substitution is documented in the Coils edition of WMOGAG.