Oracle has just released Oracle SOA Suite and Oracle BPM Suite 126.96.36.199 (often referred to as ‘Patch Set 3,’) the second release that includes comprehensive support for both Business Process Modeling Notation (BPMN) and Business Process Execution Language (BPEL) for modeling and executing business processes.
Organisations who have been using Oracle SOA Suite (and BPEL) for several years now sometimes ask us what extra value Oracle BPM Suite adds to the already rich SOA platform they are used to. And process analysts and integration developers often ask about the relative strengths of BPEL and BPMN – which to use when, and how they complement each other.
It turns out there is a lot of extra value added by the addition of Oracle BPM Suite – which is basically a superset of Oracle SOA Suite, but at the same time, it fits seamlessly into an existing SOA Suite environment and uses the same development tools, deployment and build processes, management and monitoring infrastructure and the same programming model – Service Component Architecture (SCA).
Oracle BPM Suite sits right on top of the solid foundation provided by Oracle SOA Suite. Because of this is inherits significant integration capabilities. It really is a ‘best of both worlds’ – providing excellent feature sets and capabilities for both business and technical people working in the business process management space. The strong BPM capabilities really complement the SOA foundation. It’s hard to ‘do BPM’ well without SOA, and you could argue that SOA lacks a real purpose without BPM. A lot of people who have tried to justify an investment in SOA have found it very difficult to build a successful business case without tying SOA to business driven BPM initiatives.
Organisations with a significant investment in Oracle SOA Suite should see Oracle BPM Suite as an upgrade which provides additional value – and they wont need to retrain staff, replace existing infrastructure or migrate existing artifacts to realise the additional value added.
So, on the occasion of the second release of Oracle BPM Suite in the 11g release stream, let’s take a deep dive into the value it brings to the table and also look at how well it is integrated with Oracle SOA Suite.
The right tool for the right job
BPEL and BPMN are both ‘languages’ or ‘notations’ for describing and executing business processes. Both are open standards. Most business process engines will support one or the other of these languages. Oracle however has chosen to support both and treat them as equals. This means that you have the freedom to choose which language to use on a process by process basis. And you can freely mix and match, even within a single composite. (A composite is the deployment unit in an SCA environment.)
So why support both? Well it turns out that BPEL is really well suited to modeling some kinds of processes and BPMN is really well suited to modeling other kinds of processes. Of course there is a pretty significant overlap where either will do a great job.
There are different ways of looking at which language is more suited for various kinds of processes. Let’s look a two common approaches – these both provide high level guidance and are not meant to be exhaustive or mutually exclusive. Nor do they replace the need to do your own research and possibly a small ‘proof of concept’ modeling activity to validate which is right in your environment with your people and skills.
The ‘who is the audience’ approach
This approach looks at who is going to be doing the process modeling and whether models are going to be shared with ‘business’ people.
- If the process models are going to be shared with business people, e.g. process participants, process owners or sponsors, I would tend to use BPMN,
- If the people who are doing the process modeling are coming from a business background, e.g. process analysts or business analysts, I would tend to use BPMN,
- If they were coming from an IT background, e.g. developers or architects, I would tend to use BPEL,
- If the people who are going to be doing the modeling have extensive skills and experience in one language, I would probably be inclined to use that language, unless there was a good reason to introduce the other.
The ‘type of process’ approach
This approach uses a simple rule of thumb: If the process involves ‘people’ or ‘paper,’ I would lean towards BPMN. If it involves systems or applications integration, I would lean towards BPEL. That is a pretty high level and generic rule of thumb, so there are also some other things I would consider:
- Generally speaking, I would tend to use BPMN for higher level, more ‘business’-oriented processes and BPEL for lower level, more ‘system’-oriented processes,
- If the ‘process’ is really an ‘integration’ or a ‘service,’ I would tend to use BPEL.
Layers of business process
The natural result of both of these approaches tends to be a pattern where the higher level processes – the ones that business users interact with – are modeled in BPMN and these in turn call other processes that are also modeled in BPMN which in turn call ‘services’ that are implemented in BPEL. In fact, if you take a look at the Oracle Application Integration Architecture Reference Process Models, you will see that they follow this same pattern (with even higher level models in Value Added Chain diagrams.)
BPEL is a ‘structured’ language – much like Java is – that means it has ‘control structures’ like sequence (one activity follows another), decisions (called switches), looping (using a ‘while’ loop) and ‘scopes’ which set boundaries for exception handling. Exceptions are handled in a ‘try/catch’ style like many modern programming languages. A scope in BPEL can ‘throw’ and exception to it’s parent scope where it may be handled or ‘rethrown’ to a higher scope still.
As a result of this, BPEL feels very natural to people from a programming background. It has the same kind of logic and control structures that they are used to, and lets them think about problems the way they are accustomed to thinking.
BPMN on the other hand is a ‘directed graph.’ This means that it allows you to arbitrarily move around the process. We often find that real world business processes are able to be modeled directly using directed graphs, that is we don’t need to do a lot of analysis to work out how to structure the process in such a way as to make it ‘fit’ into the language.
Now of course there is a healthy overlap where many of the processes that you could model in BPEL could also be modeled in BPMN and vice versa. However, there are some processes that can be model very simply in BPMN which are quite difficult to model in BPEL. Take for example the following hypothetic ‘flight booking’ process. For whatever reason (probably the way the ‘legacy’ system works) there are only certain points where the customer can go back to an earlier step and, depending on where they are in the process, it is a different point they can return to.
This process can be modeled very simply in BPMN, as shown below, however it would be quite difficult to model in BPEL. It could be done, of course, but it would be necessary to sit down and work out the logic. We would probably need to introduce some kind of ‘state’ variables and use them as ‘guards’ in some large switch construct inside a while loop. It could be done, but we might lose a lot of the clarity that the BPMN model (below) has – that is, it might be harder for us to just look at the model and understand the process logic.
So this is an example of the type of process that is easier to model in BPMN due to its directed graph nature. Many Oracle SOA Suite (BPEL) users may have come across processes that required a bit of work to model in BPEL, so here is one benefit we can see of ‘upgrading’ to Oracle BPM Suite.
BPMN includes an ‘embedded sub-process’ activity that allows for looping, parallel execution and iterating over members of collections (like arrays.) The embedded sub-process runs in the same instance, so it does not incur the overhead of starting a new process instance.
Embedded sub-processes can be nested and you can choose to execute the iterations sequentially or in parallel. This allows for very elegant modeling of processes that involve looping through collections (and nested collections). The example below shows a BPMN process that processes a set of pathology test series in parallel, each of which may contain multiple individual tests which are processed sequentially, before consolidating the results for review and the possible repeating of some or all tests.
Often we have a part of a process that will take some time to execute but which may be cancelled during that time. For example, while fulfilling and order (picking, packing, shipping, etc.,) we may receive an order cancellation from the customer. BPMN includes a concept called a ‘boundary event’ which can be used to model this kind of situation.
The example below demonstrates such a process. The ‘Fulfill Order’ activity is actually an embedded sub-process (shown in its ‘minimised’ form to reduce clutter.) The sub-process has a ‘message boundary event’ attached to it. If the matching message is received at any time while the sub-process is still executing, the sub-process will be interrupted and the exception path (to ‘Cancel Order’) will be followed immediately.
Boundary events can also be attached to individual activities (not just sub-processes) and can handle messages, time-related events and catch errors.
In a BPMN process each activity must have exactly one default flow coming out of it (except for the ‘end’ event) but many can also have zero or more conditional flows. A conditional flow is one that will be flowed if and only if the condition attached to it evaluates to true.
Conditions may be expressed using a simple visual editor/expression language, in XPath or may even be a set of rules that are evaluated by the embedded rules engine. Conditional paths can also be named and documented. The name appears on the process model making it easy for non-technical users to understand the process model without needed to learn how to read conditions.
BPMN also provides a rich set of ‘gateways’ that allow for modeling of different kinds of decisions in a process. These include the ability to follow exactly one path, some paths, or all paths and then to join the paths back together when one or all or completed.
Import models from Visio and other tools
Many organisations have created some process documentation using Microsoft Visio and want to be able to reuse that investment. With the release of Oracle BPM Suite 188.8.131.52, Oracle has added the ability to import process diagrams from Visio, or other tools that can export in XPDL format, into BPM.
Many of these tools allow you to include multiple BPMN ‘pools’ on the same diagram. The import facility gives you the option of importing the pools as separate process models or combining them into a single process model. Multi-tabbed diagrams can also be imported, with each tab becoming a separate process model.
The import is tested with a variety of open source and commercial modeling tools which support XPDL export.
BPM includes a ‘business catalog’ which contains shared artifacts like services, data definitions, business exceptions, event definitions and rules. The business catalog promotes reuse and collaboration between integration developers and process modelers. It allows you to easily adopt a top-down (starting with the process flow) or bottom-up (starting with the services, data, interface definitions) approach to process modeling.
Both BPEL and BPMN have a ‘template’ mechanism which allows you to define a base process and a number of variations. These mechanisms work slightly differently but both provide a similar kind of capability. The template mechanism for BPMN is more geared to allow ‘business users’ to participate in the definition of variations.
The BPM Project Template mechanism allows business users to customise processes in the Process Composer (web-based modeling environment, more on this later) within certain constraints. The constraints help to promote communication, governance and control of process customisations.
Templates include selected components like human tasks, services, business objects and of course the process flow. Business/process analysts can reuse templates to create new processes or to modify existing processes, and can even deploy their customisations directly to the runtime environment without ever touching JDeveloper.
Of course, if you want to enable this capability, you should be careful to ensure that the customisations allowed will not require any additional integration developer work to implement them – that is, you should probably be using a ‘bottom up’ approach where the process analysts create the models from a set of well tested services and other components.
Oracle BPM Suite, specifically the ‘design time’ environment in JDeveloper (sometimes called ‘BPM Studio,’) adds the ability to simulate a process before actually implementing and deploying it.
Simulation is the use of a mathematical model to predict how the process will behave in terms of time, cost and resource utilisation. Comparison of simulations with different parameters allows us to make some informed decisions about the design of the process and things like appropriate staffing levels for human tasks which are involved in the process.
When we define a simulation, we provide various parameters including the ‘arrival rate’ (or ‘creation rate’) for new instances, the ‘service time’ for each activity, which resources are required to perform each activity, the capacity of each class of resource (e.g. how many people we have in that role,) the probability that each path out of a decision point will be followed, and so on. Generally, these parameters can be provided as a ‘scalar’ value or a statistical distribution with the appropriate parameters. For example, we may define the arrival rate as a normal distribution with mean 50 and standard deviation 3.
The simulations can be animated on screen (as shown below). The animation shows each packet of work moving through the process. It also makes queuing (bottlenecks) obvious by showing queues develop before activities as instances wait for service. Queues often occur when there are not enough resources available (free) to process the amount of work arriving. Simulation animation provides a simple and effective way to clearly demonstrate bottlenecks in a process to business people like sponsors. It also provides a convenient and simple way to demonstrate the impact of changing something in the process, e.g. adding some more resources or changing the order of some activities in the process.
All of the raw data produced by the simulation engine can be saved and exported for use in other analysis tools, like Excel for example, and to make charts and tables for documents like business cases. Simulation is another benefit of ‘upgrading’ to Oracle BPM Suite.
Business-friendly process modeling and discovery
Oracle BPM Suite includes a web-based process modeling capability called ‘Process Composer.’ Process Composer allows business users to easily access and review BPMN process models from a web browser without the need to install any special software. In addition to viewing the models, users with appropriate privileges are also able to change models and create new models.
Models can be easily sychronised between the web-based business user-friendly modeling environment and the design time tools used by integration developers who complete the implementation of the process models and prepare them for deployment.
‘Process discovery’ is a vitally important aspect of a successful BPM project. Often, people may assume that process discovery means detailed workflow modeling of a process. While the detailed workflow is important, it is just one part of process discovery – and not even the most essential one.
The most important things you need to understand about your processes during discovery are the key activities, milestones, responsibilities, resource requirements, problems affecting performance and the key goals and measures of the process.
Every time I have sat down with a group of business people and modeled a business process it has been abundantly apparent that the stakeholders do not have a common, agreed understanding of the process. Usually I find that people like senior management, executives and process owners have a better understanding of the goals and measures, and how the process interacts with other processes in other parts of the business. However, it is the process participants, the people who actually carry out the process on a day to day basis, who have a much better understanding of how things are actually done, often why they are done that way, and what the problems are.
In order for a BPM project to be successful, it is essential that you consult all the relevant stakeholders and that you drive towards a consensus. This is the essence of what we call ‘process discovery’ and the business-friendly web-based modeling capability provided in Process Composer is a key enabler of the clear communication necessary to make this a reality.
Remember that automating garbage just gives you automated garbage. The process models that are produced through discovery and consensus are not just documentation. They are the actual requirements that get handed over to integration developers. They represent well considered, agreed and tested (through simulation) requirements for a business process. Process discovery not only helps to get better quality requirements, but it also reduces rework (removing some of the dependency on interpretation) and improves communication across the board.
For many human-centric processes, conventional ‘work lists’ and BPMN diagrams are not the most intuitive way to present tasks and progress through the process to business users. The ‘work list’ metaphor can make it difficult for users to understand where they are in the overall end-to-end flow of the process. Without this context it can be difficult for them to give customers or constituents advice about the progress, next steps and expected completion time for the process.
To address this, Oracle created the notion of ‘guided business processes,’ in which process designers define milestones in the BPMN process model and users interact with the process through an alternative user interface called an ‘activity guide’ that tracks progress against those milestones.
The diagram above shows an example of an activity guide for a ‘new hire’ process. Activity guides can have quite rich user interfaces and provide a lot more context to the user.
Oracle BPM Suite provides the ability for business users to easily create a team space to facilitate collaboration around either a process or even a specific instance of a process. These ‘process spaces’ can be created with the click of button, in just a few moments, without any need for assistance from IT staff.
The self-service provisioned process spaces are built from templates which can be easily customized to suit your needs and give business users access to information about the process/instance and collaboration tools like presence awareness, instant messaging, email, shared document libraries, threaded discussion forums, lists and shared calendars.
Process spaces, like the example shown below, are a simple and cost effective way of facilitating collaboration amongst communities of interest or project teams. Because they are all stored on the central server, they are easy to manage, backup and search. And the environment will integrate easily with existing directories like LDAP and Active Directory.
Oracle BPM Suite includes additional support for automatically generated analytics and dashboards, above and beyond the ‘Monitor Express’ dashboards you may be familiar with. Business users can easily create these dashboards. They can add various charts to display information about the performance of processes they care about.
Business/process analysts can include‘business indicators’ in their process model. These can be used to count how often an activity occurs, take note of the value of some instance data, or measure the time between points in the process.
From the business indicators that process modelers include in their models, BPM will automatically create ‘process cubes’ which are star schemas containing ‘real time’ data about the process performance and support OLAP-style reporting and business intelligence using the defined dimensions and measures.
BPM provides a rich set of pre-defined ‘out of the box’ dashboards that can be automatically generated with just a couple of clicks. The diagram below highlights a business indicator on the process model and a dashboard. Additionally, you can easily use a more comprehensive and powerful business intelligence tool, like Oracle Business Intelligence or a third party tool, against the process cubes.
Of course, because BPMN processes are part of the SCA composite, just like BPEL processes, you can also send data out to Oracle Business Activity Monitoring if you have a need to monitor larger numbers of processes, include data from other sources as well, and/or create more complex dashboards for larger user communities. Pre-defined dashboards are also available in BAM. We will look at BAM in more detail later in this article.
BPMN processes are modeled in ‘swim lanes.’ Swim lanes represent participants’ roles in the process. They provide a clear visual representation of who carries out each activity in the process. The roles you define in your process model can then be easily mapped to users or groups in your corporate directory using either static or dynamic membership rules.
‘Business calendars’ can also be defined so that the process engine can understand when people in various roles will be unavailable due to holidays and operating hours. This allows expiration and escalation times specified on activities to be measure in ‘business hours’ rather than arbitrary ‘wall clock’ time which may produce incorrect results around holidays and weekends. It also allows for handling of process participants in different time zones or shift workers.
Built on a solid foundation
Oracle BPM Suite can be thought of as a layer on top of Oracle SOA Suite – it adds new capabilities, including those discussed already, but it also makes extensive use of the same core components that you would use when building a BPEL process. In fact, there is only actually one process engine which can run both BPEL and BPMN processes.
A key strength of Oracle BPM Suite is the extensive integration capabilities that it inherits from the very solid . Let’s take a tour through some of the other similarities to discover the depth of integration.
Oracle SOA Suite, and by extension Oracle BPM Suite, is based on the Service Component Architecture (SCA) standard which provides a language independent way of assembling ‘service components’ to create a ‘composite application.’ The composite is the unit which can be built, deployed, tested and managed. It is built using an assembly diagram like the one shown in the diagram below.
The ‘service components’ can have various implementation styles. They may be BPEL processes, BPMN processes, rules, mediators, human tasks and so on. You can also see references to external components on the right hand side of the composite diagram. These are the various services that are used (consumed) by this composite. They are often provided by JCA adapters or are web services. The lines (called ‘wires’) between the service components indicate usage not sequence.
Test suites can be defined at the composite level. Test suites are made up of test cases. Test cases can provide simulated inputs and check for the outcome of the composite’s processing. Services can be simulated if necessary, for example if they do not exist yet or if they do not have dedicated accounts or instances available for testing.
Within a composite you can freely mix and match processes that are modeled in BPMN and BPEL. Each can call the other as a sub-process or service.
BPEL and BPMN processes are both first-class citizens in a composite. Both can be exposed as a web service or using other binding styles, both can create and consume human tasks, both can call (consume) business rules, both can use JCA adapters to integrate with external systems, both have synchronous and asynchronous invocation styles.
Both are monitored and managed in exactly the same way. Both use the ‘execution context ID (ECID)’ for instance tracking. This allows you to view details of an instance of a composite and drill down through the instance to see all of the service components involved, regardless of implementation style. You can even drill right down to view the messages sent between them and variables updated in each activity in a process. The diagram below shows an example of drilling down into an instance of a composite and then into a service component in that composite that happens to be a BPMN process. You can see the green highlighting on the process model that tells us where the execution of the process instance is currently.
Both BPEL and BPMN processes can be secured and have logging and auditing policies applied to them using Oracle Web Services Manager, the component of Oracle SOA Suite that is responsible for policy-based management and security of composites.
Oracle Business Activity Monitoring (BAM) is a component of Oracle SOA Suite (and therefore Oracle BPM Suite also) that allows you to create comprehensive dashboards for reporting which are updated in ‘real time.’
BAM is different to the process analytics mentioned earlier in a few key aspects:
- BAM dashboards can take input from many sources, not just performance metrics attached to processes,
- They are automatically updated in ‘real time’ by a ‘push’-based update mechanism, i.e. the user does not need to ‘refresh’ them,
- They can show consolidated metrics across a number of processes, services or other data sources,
- You can define thresholds and alerts, and
- You can display data using time series.
Oracle SOA Suite also includes a business to business engine called Oracle B2B that supports many common B2B protocols like AS2, EDI and RosettaNet for example. It handles issues like authentication, guaranteed delivery and non-repudiation in the business to business messaging context. B2B integrations manifest as adapter references in a composite and can be wired to BPEL and BPMN processes equally.
JDeveloper provides more and less technical views of process diagrams for both BPEL and BPMN. The less technical view is called a ‘blueprint.’ These can be used to facilitate exchange of models with other process modeling tools.
If you are reading this in early 2011, just after the release of 184.108.40.206, then there are a couple of things that may currently be slightly easier to model in BPEL. If you have a process needs these kinds of capabilities, you might want to consider modeling it in BPEL.
The first is compensation. Compensation is the issuing of ‘reversal’ transactions to undo work that was previously done and committed. A business process can run for a long time (hours, days, even weeks) – far too long to hold a transaction open. BPEL has excellent support for compensation built in to the language and it is easy to model compensation in your processes. This also means that the process engine will know when it is running forwards through a process and when it is compensating.
It is of course possible to build compensation logic into a BPMN process, though the directed graph nature of BPMN can make compensation a little more complicated to define because there are potentially many more cases that you need to cater for. It is perhaps better to model those parts of your overall business process that may need to be compensated in BPEL and call those BPEL processes from your overall BPMN business level process.
Correlation is another consideration. Correlation is the ability for a process which calls a service asynchronously to identify the corresponding response (‘callback’) from that service. This is especially important in loops or parallel execution, or when many instances of a process will be running concurrently. BPEL provides native correlation set support in the language, which allows you to define the keys to use to identify the correct response. BPMN provides correlation through its support of WS-Addressing correlation.
Update: These considerations are moot now that the BPM 220.127.116.11 ‘Feature Pack’ has been released. See this post for more information.
A big thank you to Manoj Das, Robert Patrick, Dave Shaffer and Meera Srinivasan for their time and their most helpful input and suggestions.
Claudio Ivaldi created a presentation using information in this post, and it is available from here.