Introducing Vista Search

The first version of the Vista Logic and Protocol bundles have been merged into the default branch.  Vista provides a fast full-text indexing method for content stored on the OpenGroupware Coils server.  The Vista component can currently index Contact, Document [meta-data], Enterprise, Note, and Task entities.  All the text fields of these entities are indexed including the values of object properties and company values. Utilizing the power of Vista search is performed by performing HTTP requests to the protocol exposed at "/vista" on the server. 

curl -v -u fred -o output 'http://coils.example.com/vista?term=detroit&term=steel&archived&type=enterprise'
Authenticate as user fred and search for enterprise entities, regardless of archived status, that contain the terms "detroit" and "steel".

The results of the HTTP request are JSON encoded Omphalos representations of the first 100 entities to match the specified criteria.  The default Omphalos detail level is 2056 [Comment + Company Values].  If an alternate detail level is desired this default can be overridden using the "detail" URL parameter.   The following URL parameters are supported:

• archived - Include entities in the search regardless of there archived status. If no specified archived entities will be excluded from the search results.
• detail - The Omphalos detail level to use when representing entities in the response.  It is important to recognize that specifying a high detail level will reduce performance.
• term - Specify a search term.  Any number of terms may be specified.
• type - Limit the searched entities by type.  If no type is specified all indexed entities are searched regardless of type.  Multiple type parameters may be specified.

For code local to the server searches can be performed using the "vista::search" Logic command:

results = ctx.run_command('vista::search', keywords = [ 'detroit', 'steel' ],
                                                                    entity_types = [ 'enterprise' ],
                                                                    include_archived = True)
Peform a Vista search via Logic for all enterprises, regardless of archived status, that contain the terms "detroit" and "steel".

The recently packaged tool coils-request-index can request the creation or update of an entities search vector.  Normally when an entity is modified a re-index is requested automatically [search vector generation happens in the background and is performed by the coils.vista.index component].  However, if large changes are made to the database or for the initial index generation the use of coils-request-index may expedite the process.

coils-request-index --contact --enterprises --notes --documents --tasks
Request an index/reindex of all the entities of the specified types.

coils-request-index --objectid=10100
Request an index/reindex the entity with objectId 10,100.

If the index is already current for the entity the vector generation request will be discarded.
This new feature does require a schema update to existing OpenGroupware databases.  This schema update will be required for version 0.1.45.

CREATE TABLE vista_vector (
  object_id  INT PRIMARY KEY,
  version    INT DEFAULT 0,
  edition    INT,
  entity     VARCHAR(25) NOT NULL,
  event_date DATE DEFAULT 'TODAY', 
  archived   BOOL DEFAULT FALSE,
  keywords   VARCHAR(128)[],
  vector     tsvector);
CREATE INDEX vista_idx_i0 ON vista_vector (entity);
CREATE INDEX vista_idx_i1 ON vista_vector (event_date);
CREATE INDEX vista_idx_i2 ON vista_vector USING gin(vector);

Vista search utilizes PostgreSQL's powerful tsearch text indexing module.  tsearch provides lexeme oriented indexing - so the server knows, for example, that "rats" and "rat" share the same stem.  Thanks to tsearch Vista searches are not only fast - they're clever!

Accessing Server Configuration (Defaults)

When developing Logic, either a Command or a Service component, one frequent need is to check the value of a server's configuration directive [a "default" in OpenGroupware speak].  Accessing server configuration is performed using an instance of the ServerDefaultsManager object.  The following code retrieves the value of the CoilsListenAddress address; and if no such default is defined it returns the value "127.0.0.1".

sd = ServerDefaultsManager()
HTTP_HOST = sd.string_for_default('CoilsListenAddress', '127.0.0.1')

The ServerDefaultsManager will cache the server's configuration - so if you are going to be checking a lot of defaults is better to keep the object around rather than repeatedly creating it. The ServerDefaultsManager provides the following methods for retrieving server defaults:

• bool_for_default(directive) - Values of boolean configuration values are stored as "YES" and "NO" strings.  Actually, any value that isn't "YES" is interpreted as False, which is also the default if no such directive is defined. The value returned by the method is a Python bool type.
• string_for_default(directive, default value) - Returns the value of the default as string or returns the specified default value if no such directive is defined.
• integer_for_default(directive, default value) - Returns the value as an integer, or returns the specified default value if no such directive is defined. An exception is raised if the value cannot be represented as an integer.
• default_as_dict(directive, default value) - Returns the value of the specified directive as a dictionary, or the specified default value if no such directive is defined.  An exception is raised if the value is not a dictionary.
• default_as_list(directive, default) - Returns the value as a list, or the specified default value if no such directive is defined.  An exception is raised if the value is not a list.

Regarding the actually loading of defaults the defaults manager will load from (or save to) one of two sources.  If the file ".server_defaults.pickle" exists in the document root of the server the defaults are loaded from (and saved to) that Python pickle file; otherwise the defaults are loaded from (and saved to) the OpenSTEP plist file at ".libFoundation/Defaults/NSGlobalDomain.plist".  Use of the OpenSTEP plist file facilitates parallel operation of OpenGroupware Coils with Legacy - both OpenGroupware Coils and OpenGroupware Legacy will operate using the shared configuration. 

One caveat to remember is that OpenSTEP plist files are always stored in the ISO8859-1 encoding.  This includes both the server defaults and user defaults.  Both OpenGroupware Coils and OpenGroupware Legacy always store user defaults in OpenSTEP plist format.  Facilities for parsing and writing OpenSTEP plist files are provided by the coils.foundation module.

If you are developing a remote component that does not have access to the server's document root your component can acquire a copy of the server's configuration by sending a "get_server_defaults" message to the coils.administrator component.  The payload of the response should contain the cluster's GUID [as the "GUID" key] and a copy of all the server defaults [in the "defaults" key).