Building a job dispatcher in BPM -or- Iterating over collections in BPM

Recently I was working with a customer who wanted to create a kind of ‘job dispatcher’ in BPM – basically a process that could take an array of things as input and do something different to each thing, depending on what type of thing it is.

This seems like a fairly common pattern, so I thought I would make it generic and share it here.

This example is built on, and requires JDeveloper 11.1.1.5 and BPM 11.1.1.5 with patch 12413651 (also known as the ‘Feature Pack’) applied to both.  This will not work with earlier versions or without that patch applied.

Let’s start by creating a new BPM Application by selecting New from the Application menu.  Give your application a name, I called mine JobDispatcher, and select the BPM Application template as shown below.  Then click on Next.

Give your project a name, I used the same name, then click on Finish.

After a moment, the Create BPMN Process wizard will open.  Give your process a name.  I called mine DispatchJobs and choose the Asynchronous Service type as shown below.  Then click on Finish.

Before we fill out the details of the process, let’s create some data types we will need.  Select New from the File menu to open the New Gallery.  Click on the tab for All Technologies and selct the XML category on the left hand side, then XML Schema on the right hand side, and click on OK.

Enter a name for your schema, I called mine jobs.xsd.  Store it in the xsd directory inside your project directory, as shown below, and add a meaningful suffix to the Target Namespace, I added jobs.  It is a good idea to always customise the target namespace to prevent namespace clashes later on.  Click on OK.

When the editor opens, you want to set up your schema like the image below.  A fast way to do this is to switch to the source view using the tab at the bottom of the editor and paste in the XSD code (below).  You can then switch back to design view.

Here is the XSD code:

<?xml version="1.0" encoding="windows-1252" ?>
<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema"
            xmlns="http://www.example.org/jobs"
            targetNamespace="http://www.example.org/jobs"
            elementFormDefault="qualified">
  <xsd:element name="data" type="jobsType">
    <xsd:annotation>
      <xsd:documentation>
        A sample element
      </xsd:documentation>
    </xsd:annotation>
  </xsd:element>
  <xsd:complexType name="jobType">
    <xsd:sequence>
      <xsd:element name="jobType" type="xsd:string"/>
      <xsd:element name="otherData" type="xsd:string"/>
    </xsd:sequence>
  </xsd:complexType>
  <xsd:complexType name="jobsType">
    <xsd:sequence>
      <xsd:element name="job" type="jobType" maxOccurs="unbounded"/>
    </xsd:sequence>
  </xsd:complexType>
</xsd:schema>

This allows us to specify any number of jobs as the input to our process – we will see later how we use this data type to create the inputs for the process.  Notice that we have an arbitrary ‘otherData‘ attribute in our job.  In real life, you would obviously have something a bit more interesting.

Now let’s set up our process data objects.  Open the BPM Project Navigator from the tab or the View menu, then expand out your project and the Business Catalog as shown below.  Right click on the Business Catalog and select Module form the New menu.  Create a new Module named Data to hold your definitions.

Now right click on that Data folder that you just created and select Business Object from the New menu.

Create a new business object called CollectionOfJobs in the Data folder.  Select the checkbox next to Based on External Schema and then click on the ellipses (…) button find the schema.  Locate the JobsType that we defined in our XSD earlier and press OK and then OK again.

Create another business object called SingleJob of type JobType  in the Data folder using the same steps.

Now we want to create a data object in our process itself.  Open your process in the main editor window, or return to it.  Click somewhere in the process editor to make it active.  The Structure view should be visible in the lower left corner.  If it is not, you can open it from the View menu.  Right click on Process Data Objects – that’s Process not Project, be careful.  Then select New from the popup menu.

Create a new data object called theJobs with type <Component> as shown below.  Click on the litthe magnifying glass icon and select the CollectionOfJobs business object in the Data folder that we just created a moment ago.  Then click on OK.

We will use this data object to store the actual data inside the process instance.

Now, let’s get to work on the process.  Open the Component Palette by click on its tab, or from the View menu.  Expand out the Activities section and drag a Subprocess activity into your process model and hold it over the line between the Start and End activities.  Notice that the line turns blue when you hold another activity over the centre of it.  Drop the Subprocess activity when the line is blue.  This will add it in the middle of that line.  If yhou were to drop it when the line were not blue, it would just be placed on the editor canvas but not connected into the process.  Of course you can do this and add the arrows yourself if you prefer.

Make sure you get the Subprocess activity, not the Event Subprocess as these do different things.

Double click on the gray subprocess area, or right click on it and select Properties.  This will open the properties editor for the subprocess.  The subprocess activity is pretty versatile and it can be used in a variety of ways.  Here we want to kick off some uknown number of subprocesses based on the data type, in parallel, so we want to go to the Loop Characteristics tab and make the following selections.

First, set the Loop Characteristics to MultiInstance.  Next, set the Mode to Parallel.

In the Creation Type, select Collection.  This lets us iterate over a collection.  You could of course find the size of the collection and then use Cardinality and array indexes if you are that way inclined.

Side note:  The difference between cardinality and collection is a bit subtle.  Here are some Java examples that are conceptually equivalent:

List someData = getTheDataFromSomewhere();

// cardinality works like this
for (int i = 0; i <= someData.size(); i++) {
  doSomethingTo(someData[i]);
}

// collection works like this
for (Object data : someData) {
  doSomethingTo(data);
}

Now click on the little pencil icon for Loop Data Input and create an argument of type <Array> as shown below.  Select <Component> as the Element Type and then click on the little magnifying glass icon and select the SingleJob type in your Data folder.  This will create an argument which is an array of SingleJob‘s.

In the Expression field for the Loop Data Input click on the little calculator icon and then pick XSLT Expression and add theJobs.job from the lower left pane into the expression (or just go ahead and type it in).

Then repeat these steps to create the Loop Data Output as well.  When you are done you should see something like the following image:  Go ahead and reward yourself by clicking on OK 🙂

Now, stretch out your subprocess so we have a bit of room to put some more activities inside there.

Open up the Gateways section in the Component Palette and drag an Exclusive Gateway out into the process, drop it on the line between the Start1 and End1 activities in the subprocess.  Also drag out two Activity activites from the Activities section and drop them into the subproces as shown.  Call them JobA and JobB as shown below.

Go ahead and connect up your process as shown in the two diagrams below.  You can create the connections by right clicking on the first of the two activities you want to join and selecting Add default sequence flow or Add conditional sequence flow as appropriate.  Then click on the activity you want to connect it to.

Right click on each of the two Activity activities and select the option to Mark Node as Draft.  This will allow us to go ahead and deploy and execute our process before we actually define what these two will actually do.  That is fine for this example.  In real life, you probably want to have them call another process to handle the job, a different process for each type of job.  Look for our upcoming article on correlation for some details on how to do this 🙂 I will add a link here when I post it.

You should now see something like the following image.  Right click on the line leading into JobA and select Properties.  Go to the Properties tab and in the Expression field, enter the following expression, as shown below:

inputDataItem.jobType == "A"

Do the same for the line leading to JobB but change it to look for jobType B.

The third line (marked with the little cross at the start) is the unconditional path.  It will be followed if none of the conditions evaluate to true.

Finally, let’s add our input data for the process.  Right click on the Start activity at the beginning of the process (not the one in the subprocess) and select Properties.  Go to the Implementation tab and click on the green plus icon to add an argument.  You can leave the name as argument1.  Set the Type to <Component> and select your CollectionOfJobs component as shown below.  Then click on OK in the Create Argument window.

Click on the Data Associations link and add a line to copy the data in argument1 into our process data object, theJobs, as shown below.  Then click on OK and then OK again.

Now we are ready to deploy and test our job dispatcher.  Go back to the Application navigator.  You can open it from the View menu if you don’t see it.  Right click on your project root folder and select JobDispatcher… from the Deploy option in the popup menu, as shown below.

Go ahead and deploy it on your test BPM server.  If you don’t know how to do this, it has been covered many times in earlier posts, or you should be able to work it out, its not too hard.

Now log on to Enterprise Manager (again how to do this is covered in earlier posts) and navigate to your newly deployed composite inside soa-infra as shown below.  Open the composite and click on the Test button.

The test page will open.   Notice that you can specify how large you want the array to be.  Type in 3 and click on the little refresh icon highlighted below.

The page will refresh and you now have room to enter three jobs.  Enter some test data as shown below.  One should have jobType  set to A, one to B, and the other to something else.  The otherData can be anything you like.  When you have all of your test data ready, click on the Test Web Service button (top right).

This will start an instance of your process.  Click on the Launch Flow Trace button to see what happened 🙂  Note: make sure your browser popup blocker does not prevent the flow trace window from opening.

In the flow trace window, click on the DispatchJobs process as shown below.

Exapand out the audit trail to see what happened.  You can see here (look at the arrows) that the first job went through and was ‘processed’ by JobA as we expected, the second went to JobB and the third just went straight through and was not processed.  This is exactly what we expected.

So there you have a simple job dispatcher implemented in BPM.  Look for some follow up articles on correlation and scheduling which can be combined with this job scheduler to do some really interesting stuff.  Enjoy!