Welcome to another edition of the Documaker Tech blog! Today we'll be showing how to connect MQSeries queues to Oracle Documaker Enterprise Edition (ODEE). Fair warning: this post will be acronym-intensive, and as such I will endeavor to present the full meaning of an acronym before using it. So let's dive in!
If you're not familiar with the concept of the a message queue a cursory internet search will turn up a wealth of information. You need to know that message queues are used to provide synchronous or asynchronous communication between two or more processes. Synchronous (sync) communication means that the sender will wait for the receiver to acknowledge receipt of the message (also known as a response), whereas asynchronous (async) means that the sender will not wait for a response. Async is also known as “fire-and-forget”, or FAF. It is also possible to have multiple senders and receivers using the same queue - that is, you might have multiple senders placing messages into a queue, and then multiple receivers retrieving messages from the queue. It is this capability that is used to provide scalability within ODEE.
Internal Queueing
Internally, ODEE uses queues to distribute work units among the workers in a factory Assembly Line. Queues are necessary to support distributed work and provide part of the backbone of the infrastructure that enables ODEE to scale to enterprise-level processing capacity - the other part of the backbone being the database. ODEE follows the factory model for document production: an Assembly Line represents a document production configuration, which is serviced by multiple workers to generate documents. The workers perform different tasks, and scale independently of one another to accommodate changing work loads. ODEE has a defined path that each document request will follow in order to complete assembly. This path is orchestrated by the Scheduler worker, which notifies each successive pool of workers when work is available. This notification is done using queues - here’s an example:
Document Generation transaction is enqueued from external application into the Request Queue.
The Receiver, an ODEE Worker, dequeues the transaction, and starts a Job within ODEE. Note that there could be multiple Receiver workers, and only one is needed to pick up the request to start the transaction.
The Scheduler, an ODEE Worker, monitors the system for new Jobs, and notes the new Job. The Scheduler enqueues a message for the Identifier worker pool.
All Identifier workers periodically check in to their request queue for new work - one of these instances will pick up the Job, and will mark it as in process.
The Identifier worker completes its task with this Job, and updates the system accordingly.
The Scheduler, ever watchful, notes that the Identifier phase of this Job has completed, and so notifies the next pool of workers that need to service the job.
This process, Scheduler->queue->worker->update repeats until the Job is completed. This entire process typically takes place in a second or two (or on decent hardware, sub-second!).
External Queuing
In addition to internal queues, ODEE uses queues externally as an integration point, enabling it to accept processing requests from other applications. In the default installation, these are JMS queues named ReceiverReq and ReceiverRes.
Queue Requirements
ODEE 12.4 and earlier Enterprise Edition releases use:
Java Message Service (JMS) queues to distribute workload among Assembly Line factory worker pools.
Java Application Server (JAS) such as like Oracle WebLogic Server (WLS) or IBM WebSphere Application Server ND (WAS).
JMS providers which implement the JMS 1.1 Specification; more precisely WLS 10.3.6 and WAS 7.0.0.27 ND.
During ODEE installation, the deployment scripts will create the necessary artifacts within the target JAS. For WLS, this means a JMS Server and associated module and sub deployments will be created and configured automatically. For WAS, this means the associated components will be created and configured on the WAS Service Integration Bus (SIB). The resulting software deployment is configured to utilize the chosen JAS queues.
Integration
During a recent implementation I was presented with a design decision: how to integrate ODEE with IBM WebSphere MQ (aka MQSeries), to extend interoperability to a customer’s application landscape that was already using MQSeries? ODEE can use MQSeries for its queuing infrastructure, provided the connectors have been configured to activate JMS capability within MQSeries. In this particular situation we wished to avoid placing the internal queuing infrastructure on MQSeries for a number of reasons (cost and proximity of the MQ host to the ODEE environment to name two) - so we chose a different approach: use the WLS JMS implementation for internal queuing and MQSeries for external queuing, and support an out-of-the-box configuration. Amazingly, this solution is already provided out of the box with Oracle WebLogic Server with some minimal configuration, which will connect the MQSeries queues with the external integration queues ReceiverReq and ReceiverRes. Let’s start with a few assumptions:
Physical MQSeries queues should already exist; we will use REQQ and RESQ as our example queues;
MQ Queue Manager name, host, and port are known (QMGRNAME, QHOST, and 1480 are our respective values in this example). Note that 1414 is the default, and we are using a non-default value;
Network paths from the WLS server to MQSeries server exist and are open;
WLS 10.3.6 will be used as the JAS for ODEE; and
You have sufficient rights to connect and create objects.
Activating MQSeries JMS
First, we need to create a JNDI tree that references and binds the MQSeries artifacts (e.g. Queues and connection factories). The JNDI tree can be file-based, LDAP-based, or JAS-based, depending on your needs. For the purposes of this post we’ll assume a file-based JNDI tree. MQSeries includes a tool called JMSAdmin tool, which is in the MQ_HOME/Java/bin (MQ_HOME is the installation directory of MQSeries). In order to run this tool, you will need to modify the JMSAdmin configuration file, which is called JMSAdmin.config. This file is located in the same directory as the tool itself, and you can edit the file with any text editor. Set the following values:
INITIAL_CONTEXT_FACTORY=com.sun.jndi.fscontext.RefFSContextFactory
PROVIDER_URL=file:/c:/mq_jndi
The directory specified in the PROVIDER_URL setting must be created before you attempt to start the JMSAdmin tool - otherwise, the tool will fail! Now you can run the tool by executing MQ_HOME/Java/bin/JMSAdmin.bat or MQ_HOME/Java/bin/JMSAdmin.sh. Note that the tool uses a proprietary command protocol which is documented here. In the tool, you will execute the following steps:
1. Define the references to the queues in the tool. Note: it is not required to use a different local name ( e.g. “MQRES” or "MQREQ") in fact it could be the same as the physical queue name.
InitCtx> Def q(MQREQ) queue(REQQ) qmgr(QMGRNAME) host(QHOST) port(1480)
InitCtx> Def q(MQRES) queue(RESQ) qmgr(QMGRNAME) host(QHOST) port(1480)
2. Define the reference to a queue connection factory in the tool:
InitCtx> Def qcf(MQQCF)
3. Display the context, inspect the output, and end.
InitCtx> dis ctx
Contents of InitCtx
.bindings java.io.File
a MQREQ com.ibm.mq.jms.MQQueue
a MQRES com.ibm.mq.jms.MQQueue
a MQQCF com.ibm.mq.jms.MQQueueConnectionFactory
4 Object(s)
0 Context(s)
4 Binding(s), 3 Administered
InitCtx> end
As I mentioned, It is possible to also create an LDAP-based or JAS-based JNDI tree, but we’ll explore that in an additional post. For now, let’s continue using the file-based JNDI tree.
Preliminary Setup
First, you’ll need to obtain some JAR files from your MQSeries installation and add them the ODEE domain. Locate the following files and copy them to MIDDLEWARE_HOME/user_projects/domains/idocumaker_domain/lib (where MIDDLEWARE_HOME is the WLS installation directory):
com.ibm.mq.commonservices.jar
com.ibm.mq.defaultconfig.jar
com.ibm.mq.headers.jar
com.ibm.mq.jar
com.ibm.mq.jms.Nojndi.jar
com.ibm.mqjms.jar
connector.jar
dhbcore.jar
fscontext.jar
jms.jar
jndi.jar
providerutil.jar
Once placed, you’ll need to restart the domain (e.g. ODEE WLS AdminServer).
Add MQSeries to WebLogic
Next, we’ll add our MQSeries configuration to WLS as a Foreign JMS Provider. Make sure WLS is running, and open a browser to the administration console (http://hostname:port/console). In the console, use the left-hand pane to navigate to Services ->Messaging->JMS Modules. Locate the installed JMS Module with ODEE, usually it’s called AL1Module, and click it. Click the New button and from the list of available options select Foreign Server, and then click Next. Given the Foreign Server a name (e.g. MQSERIES) then click Next, and accept the default targeting (to jms_server) then click Finish.
A. Click on the Foreign Server you just created. You will need to define some additional parameters to your Foreign Server:
JNDI Initial Context Factory. Set this to the same value we used in the JMSAdmin.config, that is, com.sun.jndi.fscontext.RefFSContextFactory.
JNDI Connection URL. Set this to the same value we used in the in the JMSAdmin.config, that is, file:/c:/mq_jndi.
Click Save.
B. Click the Destinations sub tab and on the next screen, click New. Here we will define the Foreign Destinations (recall we created these with the JMSAdmin tool), which requires three parameters:
Name. This is the internal name of the MQSeries queue, used only for display purposes. Set to MQREQ, to keep things simple.
Local JNDI Name. Set to MQREQ. Can be anything, as it is used locally and not on the MQSeries side, but I recommend using the same name as the next parameter.
Remote JNDI name. Must be set to the name of the queue defined in JMSAdmin, e.g. MQREQ.
Click Ok. Repeat the above step to create another Foreign Destination, this time for MQRES.
C. Click on the Connection Factories sub tab, and then click New. Enter the following settings to define the Foreign Connection Factory:
Name. This is the internal name of the MQSeries queue connection factory, used only for display purposes. Set to MQQCF, to keep things simple.
Local JNDI Name. Set to MQQCF. Can be anything, as it is used locally and not on the MQSeries side, but I recommend using the same name as the next parameter.
Remote JNDI name. Must be set to the name of the queue defined in JMSAdmin, e.g. MQQCF.
Click Ok.
At this point, you should be able to click on View JNDI Tree in the WebLogic console and see the two queues and queue connection factory listed. If not, this means that the Foreign JMS Server references could not be created - usually an indication that either the required MQSeries JAR files are not present in the ODEE domain, or the JNDI Connection URL is not accessible. Check your log files for additional information.
Bridging the Connection from MQ
At this point, we have added the MQSeries queues as foreign JMS queues to our ODEE domain in WLS. What remains is to bridge the default external integration queues ReceiverReq and ReceiverRes to the foreign queues. To do so, back in the WLS Console, click on Services->Messaging->Bridges. Click New. We are creating the bridge for messages coming from MQSeries - enter the following properties:
Name - this is for viewing purposes only; call it BRIDGEFROMMQ.
Selector - not required; leave blank.
Quality of Service - this determines how the bridge tracks messages and ensures they are delivered (e.g. in case of a possible missed delivery, it can resend the message). For this demonstration, choose Exactly Once.
Initial State - tick the Started box.
Click Next. Click New Destination. We are creating Source destination for our BRIDGEFROMMQ bridge, so we’ll need to define the source queue:
Name - this is for viewing purposes only; call it FROMMQ_SOURCE.
Adpater JNDI Name - select eis.jms.WLSConnectionFactoryJNDINoTX (note: if using XA, select the XA adapter name).
Adapter Classpath- leave blank.
Connection URL - leave blank.
Connection Factory JNDI Name - set to MQQCF.
Destination JNDI Name - set to MQREQ.
Click Ok. You should now see FROMMQ_SOURCE selected in the dropdown. Click Next. In the Messaging Provider selection, choose Other JMS Provider. Click Next. Click New Destination. We are creating Target destination for our FROMMQ bridge, so we’ll need to define the queue:
Name - this is for viewing purposes only; call it FROMMQ_TARGET.
Adpater JNDI Name - select eis.jms.WLSConnectionFactoryJNDINoTX (note: if using XA, select the XA adapter name).
Adapter Classpath- leave blank.
Connection URL - leave blank.
Connection Factory JNDI Name - set to jms.al1.qcf - This is the queue connection factory of the target destination, which is the ReceiverReq queue. The name I’ve chosen here is the default installation name.
Destination JNDI Name - set to jms.al1.receiverreq.
Click Ok. Choose FROMMQ_TARGET in the dropdown. Click Next. In the Messaging Provider selection, choose WebLogic Server 7.0 or Higher. Click Next. Choose jms_server as the target, click Next, then click Finish. We’re almost done!
Bridging the Connection to MQ
As you might have guessed, we’ve created the bridge from MQ to WLS, and now we need to create the bridge from WLS to MQ. In the WLS Console, click on Services->Messaging->Bridges. Click New. We are creating the bridge for messages coming from MQSeries - enter the following properties:
Name - this is for viewing purposes only; call it BRIDGETOMQ.
Selector - not required; leave blank.
Quality of Service - this determines how the bridge tracks messages and ensures they are delivered (e.g. in case of a possible missed delivery, it can resend the message). For this demonstration, choose Exactly Once.
Initial State - tick the Started box.
Click Next. Click New Destination. We are creating Source destination for our BRIDGETOMQ bridge, so we’ll need to define the source queue:
Name - this is for viewing purposes only; call it TOMQ_SOURCE.
Adpater JNDI Name - select eis.jms.WLSConnectionFactoryJNDINoTX (note: if using XA, select the XA adapter name).
Adapter Classpath- leave blank.
Connection URL - leave blank.
Connection Factory JNDI Name - set to jms.al1.qcf
Destination JNDI Name - set to jms.al1.receiverres
Click Ok. You should now see TOMQ_SOURCE selected in the dropdown. Click Next. In the Messaging Provider selection, choose WebLogic Server 7.0 or higher. Click Next. Click New Destination. We are creating Target destination for our TOMQ bridge, so we’ll need to define the queue:
Name - this is for viewing purposes only; call it TOMQ_TARGET.
Adpater JNDI Name - select eis.jms.WLSConnectionFactoryJNDINoTX (note: if using XA, select the XA adapter name).
Adapter Classpath- leave blank.
Connection URL - leave blank.
Connection Factory JNDI Name - set to MQQCF
Destination JNDI Name - set to MQRES
Click Ok. Choose FROMMQ_TARGET in the dropdown. Click Next. In the Messaging Provider selection, choose Other JMS Provider. Click Next. Choose jms_server as the target, click Next, then click Finish. That’s it! Make sure your changes have been activated, and requisite WLS server(s) restarted. To test, deposit a message in the MQREQ queue (it should take the same XML input in SOAP format as the doPublishFromImport web service method). Here’s an example - note where the input extract XML should be placed in Base-64 encoded format:
<soapenv:Envelope
xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:tns="oracle/documaker/schema/ws/publishing"
xmlns:pubcmn="oracle/documaker/schema/ws/publishing/common"
xmlns:v1="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1"
xmlns:cmn="oracle/documaker/schema/common"
xmlns:req="oracle/documaker/schema/ws/publishing/doPublishFromImport/v1/request">
<soapenv:Header/>
<soapenv:Body>
<tns:doPublishFromImportRequest>
<tns:doPublishFromImportRequestV1>
<pubcmn:timeoutMillis>90000</pubcmn:timeoutMillis>
<v1:JobRequest>
<req:Payload>
<req:Transaction>
<req:Data>
<cmn:Content>
<cmn:Binary>**replace with base-64 encoded extract data**</cmn:Binary>
</cmn:Content>
</req:Data>
</req:Transaction>
</req:Payload>
</v1:JobRequest>
<v1:ResponseProperties>
<!--cmn:ResponseType>Attachments</cmn:ResponseType-->
<cmn:ResponseType>JOB_ID</cmn:ResponseType>
</v1:ResponseProperties>
</tns:doPublishFromImportRequestV1>
</tns:doPublishFromImportRequest>
</soapenv:Body>
</soapenv:Envelope>
After a moment check the MQRES queue for a response message. You may uncomment the <cmn:ResponseType> node with the Attachments value if your system is configured to return PDF values. There can be additional configuration that is necessary depending on your specific system and requirements - consult with an ODEE and/or MQSerires subject matter expert and you’ll be on your way to integrated messaging in no time!