Standard Flow

In the standard flow, the Access Layer initiates processing by sending an initial "Process Message" into the message bus queue that the initial process module is listening to. After processing & app intelligence produced a result the result is sent back to the Access Layer by the module that holds the final result using the SOAP service method storeProcessingResultsAndFinalizeJobs(...).

Notes:

• The Access Layer collects and stores all input from external sources like Harvesters before processing starts. While the information is managed in a transaction safe way it is written through to the CoreDB once the "Initial Process Message" was sent.

  This is important with regards to timing issues that would be more likely if source information would get updated after the complete processing finished.

• By calling the service method startJob(...) the Access Layer sends the "Initial Process Message" into the message bus. One of the processing modules needs to pick it up and start processing on it.
• The processing modules are responsible to update the progress or failure of a processed Job by using the SOAP service method updateJob(...) that allows to send information on the current progress or change the job state if applicable.
• As mentioned above, the last module in the chain should finalize a job by sending the process results back to the Access Layer. For the case that the processing failed and no result could be produced, the module that failed processing should update the referring Job to avoid having orphaned entries.
• Jobs are managed inside the CoreDB and long running Jobs are supported. Multiple Access Layer instances share the Job details among each other and are aware of all Jobs running inside one GRID site.

Non Standard Flows

Using this concept, non standard flows can be supported as well. The diagram above shows 2 possible entry points to insert pre-processed data:

• Importing Legacy Data: Existing data may be imported into the standard GRID work flow by assembling and sending "Process Messages" into the relevant "queue" inside the message bus. In the example above this is the queue that the "App Intelligence" processes are listening to, however it can be inserted in any other step.

  Note: By 1st July 2010, many legacy processes are still in use. The agent that is capable of translating legacy data into the "Process Messages" may also be used inside the normal processing work flow.

• Multisite Data Feed: "Process Messages" that went through a full processing cycle may not only be sent to one GRID site, instead they can also be multiplexed to multiple sites. This is possible because the Access Layer resolves conflicts automatically and runs de-duplication on the data that is sent against the interfaces. See Multisite Processing for more information on this topic.

Resources

• Initial Process Message: See Initial Process Message
• Process Result Message: See: Process Result Message
• XML Schema:
  • process-package-data-set.xsd
  • process-package-data-set-2.xsd
  • process-package-data-set-3.xsd

Initiate Processing

The following message shows the content that is sent by the access layer in order to initiate processing. The Access Layer sends this message against the "incoming" queue of the processing message bus. Once sent, the processing system uses this as input to initiate all further tasks.

Finalize Processing

The following message shows the content that is expected by the Access Layer to finish processing, on a single package. The Access Layer offers a SOAP service method called storeProcessingResultsAndFinalizeJobs(...) that excepts one or more of such messages in order to update the CoreDB using the contained process results.

Notes:

• This message contains the content of a single package like a single installer archive or any other file-container that the processing system can decode.
• When a package contains another package, a subsequent message of this type is required for every child package. Linkage between packages is realized by "packageMember" elements in the same way as with standard files.
• A single "Initial Process Message" may produce many "Process Result Messages" depending on the contents of the initial processed package. The processing system is responsible for the creation of multiple messages.

• A "packageMember" may contain detailed or basic information on the referenced file. Adding detailed information on a referenced package is a waste of space as the same information is also contained in the subsequent message describing the package.
• Every "Process Results Message" requires a unique Job-ID. In case of a single "Initial Process Message" produces many "Process Result Messages" they should be assigned with a "SubJob-ID". See the SOAP service method prepareSubJob(UUID parentJobId).
• Named elements inside the result message follow the naming rule as defined inside "Naming Elements".
• The element <packageFamily/> (including <vendor/>) is optional. In case of vendor or package family entries are missing inside the CoreDB, the Access Layer will implicitly create the missing entries based on the names specified inside the retrieved "Process Result Message".

Example Code (snippet from "...integrationtest.HarvestingFlowTest")

See Code Examples for more examples.

Example Code (Multiple Options)

Legacy Processing

Outlook on Processing without Legacy Support

Convert Legacy Results and continue with AppInt

Collect Final Results