Blogroll

JBoss Fuse and JBoss BPM Suite Integrated

Last week Apache Camel v2.16 was released with over 600 issues resolved (bug fixes, updates and new features). Claus blogged about top 10 highlights already and here I want to briefly show you a use case about my tiny contribution in this release: camel-jbpm component.

This component is intended to make integration with jBPM (a Business Process Management Application) easier. It uses Remote Java API and basically does REST calls to jBPM to manage business processes (start, stop, abort), execute Business Rules, manage Human Tasks, etc. So rather than manually creating a REST client or using Remote API, you can use the Camel component. The remote API provides also HornetQ based integration, but I've dropped out that part from the Camel component as it didn't play nicely with OSGI.

In the sample use case (which is inspired by a real one) there is an integration application based on Camel that runs on JBoss Fuse which is supposed to do real time integration. And whenever an error occurs in the integration route, the error details are sent to jBPM for evaluation and review by human being if necessary. To demonstrate that I've created two separate projects:
  • A Camel project that runs on Fuse, handles any exception and sends the details to jBPM
  • A Business Process project that runs on jBPM, classifies the errors using Drools and Complex Event Processing and creates Human Tasks if the error is critical.

The Camel route below is self explanatory, there is a route that generates 10 exceptions with half a second delay each and an error handler that catches those exceptions and sends them to jBPM. Notice that it also populates CamelJBPMParameters header with the details about the error.

You can find the sample code on github with instructions about how to build and run it on Fuse.

The second part of the demo consist of a jBPM process that receives the errors and processes them. The BPMN2 diagram of the process is below:

The process have input parameters matching the values from CamelJBPMParameters header. In the first step it initializes a RequestQualifier object that is marked as non-critical. In the second step we use Drools and Complex Event Processing to qualify errors. The idea with CEP is that we want to mark an error as critical only if there were more than 5 errors in 10 seconds time window. Also to prevent marking multiple errors as critical and flooding the system with Human Tasks, we limit it to only 1 critical error at most in 10 minutes interval. All these conditions are written in Drools as following:

Once the request with the error details has been qualified (as critical or not), we use that information to do conditional routing. All non critical errors are logged and the business processes complete as there are no further action defined for them. We can see the full list of completed processes and error related information in the following screen shot:
On the other branch of the business process, if an error has been marked as critical, we do create a Human Task and the process stops at this point. This requires a user to assign the task and take some actions. Once the Human task is claimed and completed, then the business process continues and completes in the next step.
From the ten error messages generated in quick succession, only one is qualified as critical and generates Human Task and the remaining nine gets logged and completed as expected.
The business process project with instructions on how to deploy it to jBPM is on github too.

JBoss Fuse Deployment Considerations

Fuse is a framework for creating distributed applications and as such it has many moving parts. There are bundles, features, profiles, containers and various ways of combining those into the resulting deployment architecture. Here are some thoughts and guidelines based on the field experiences from a number of projects.

If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.

An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.

A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)

An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.

Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).

So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.

Next, let's have a look at different types of containers we have in a typical Fuse application:

  • Fabric Server - This is where the ZooKeeper server is running, together with the maven proxy, git server and Hawtio instance.
  • A-MQ Container - runs A-MQ server instance(s).
  • Worker/ESB container - this is where the business/integration logic is running. That is typically Camel and CXF based (our order profile would be deployed here).
  • Insight container - is where the Elastic Search server runs.
  • Fabric Gateway - used for discovery and load balancing of CXF or A-MQ endpoints.
One point that is worth clarifying is that the containers are generic, it is the profile that is deployed that gives a container a role. And in theory it is possible to deploy various combination of profiles from the above categories to the the same container but that is not a wise idea as these profiles are fundamental for the rest of the application and it is better to isolate and allocate enough resources to each of them rather than making them share the same JVM. Under a heavy load on the message broker you wouldn't want to have ZK server struggling for resources and jeopardize the health of the whole Fabric. On the other hand deploying business profiles (order and invoice profile for example) to the same container is perfectly fine if it makes sense from non functional requirements point of view. So try to put the above profiles/responsibilities/roles into separate containers and split/group your business profiles into containers based on the use case. More on this topic below.

So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.

Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!

The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.

The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:

  • Business criticality: highly critical services on dedicated containers and safest hosts
  • Load: services with high load on hosts with more resources
  • Coupling: services coupled together (direct in memory calls are more efficient that http or jms for service interaction) or service depending on same resources
  • Functionality: services belonging to the same business domain or applications layer
No matter what is the topology, keep in mind that it is not final and Fabric makes it really easy to move profiles across containers and containers accords hosts. So you have the tools to change and adapt when needed.

Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.

Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.

Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).

No matter whether you call it SOA or Microsservices, ESB or light integration layer, distributed applications have many moving parts and building one is a complex (and enjoyable) journey. The post here is quite high level and focusing on the happy paths only. For any real discussions, have a chat with Red Hat.

About Me