JBoss Fuse and JBoss BPM Suite Integrated

Last week Apache Camel v2.16 was released with over 600 issues resolved (bug fixes, updates and new features). Claus blogged about top 10 highlights already and here I want to briefly show you a use case about my tiny contribution in this release: camel-jbpm component.

This component is intended to make integration with jBPM (a Business Process Management Application) easier. It uses Remote Java API and basically does REST calls to jBPM to manage business processes (start, stop, abort), execute Business Rules, manage Human Tasks, etc. So rather than manually creating a REST client or using Remote API, you can use the Camel component. The remote API provides also HornetQ based integration, but I've dropped out that part from the Camel component as it didn't play nicely with OSGI.

In the sample use case (which is inspired by a real one) there is an integration application based on Camel that runs on JBoss Fuse which is supposed to do real time integration. And whenever an error occurs in the integration route, the error details are sent to jBPM for evaluation and review by human being if necessary. To demonstrate that I've created two separate projects:
  • A Camel project that runs on Fuse, handles any exception and sends the details to jBPM
  • A Business Process project that runs on jBPM, classifies the errors using Drools and Complex Event Processing and creates Human Tasks if the error is critical.

The Camel route below is self explanatory, there is a route that generates 10 exceptions with half a second delay each and an error handler that catches those exceptions and sends them to jBPM. Notice that it also populates CamelJBPMParameters header with the details about the error.

You can find the sample code on github with instructions about how to build and run it on Fuse.

The second part of the demo consist of a jBPM process that receives the errors and processes them. The BPMN2 diagram of the process is below:

The process have input parameters matching the values from CamelJBPMParameters header. In the first step it initializes a RequestQualifier object that is marked as non-critical. In the second step we use Drools and Complex Event Processing to qualify errors. The idea with CEP is that we want to mark an error as critical only if there were more than 5 errors in 10 seconds time window. Also to prevent marking multiple errors as critical and flooding the system with Human Tasks, we limit it to only 1 critical error at most in 10 minutes interval. All these conditions are written in Drools as following:

Once the request with the error details has been qualified (as critical or not), we use that information to do conditional routing. All non critical errors are logged and the business processes complete as there are no further action defined for them. We can see the full list of completed processes and error related information in the following screen shot:
On the other branch of the business process, if an error has been marked as critical, we do create a Human Task and the process stops at this point. This requires a user to assign the task and take some actions. Once the Human task is claimed and completed, then the business process continues and completes in the next step.
From the ten error messages generated in quick succession, only one is qualified as critical and generates Human Task and the remaining nine gets logged and completed as expected.
The business process project with instructions on how to deploy it to jBPM is on github too.

JBoss Fuse Deployment Considerations

Fuse is a framework for creating distributed applications and as such it has many moving parts. There are bundles, features, profiles, containers and various ways of combining those into the resulting deployment architecture. Here are some thoughts and guidelines based on the field experiences from a number of projects.

If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.

An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.

A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)

An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.

Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).

So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.

Next, let's have a look at different types of containers we have in a typical Fuse application:

  • Fabric Server - This is where the ZooKeeper server is running, together with the maven proxy, git server and Hawtio instance.
  • A-MQ Container - runs A-MQ server instance(s).
  • Worker/ESB container - this is where the business/integration logic is running. That is typically Camel and CXF based (our order profile would be deployed here).
  • Insight container - is where the Elastic Search server runs.
  • Fabric Gateway - used for discovery and load balancing of CXF or A-MQ endpoints.
One point that is worth clarifying is that the containers are generic, it is the profile that is deployed that gives a container a role. And in theory it is possible to deploy various combination of profiles from the above categories to the the same container but that is not a wise idea as these profiles are fundamental for the rest of the application and it is better to isolate and allocate enough resources to each of them rather than making them share the same JVM. Under a heavy load on the message broker you wouldn't want to have ZK server struggling for resources and jeopardize the health of the whole Fabric. On the other hand deploying business profiles (order and invoice profile for example) to the same container is perfectly fine if it makes sense from non functional requirements point of view. So try to put the above profiles/responsibilities/roles into separate containers and split/group your business profiles into containers based on the use case. More on this topic below.

So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.

Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!

The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.

The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:

  • Business criticality: highly critical services on dedicated containers and safest hosts
  • Load: services with high load on hosts with more resources
  • Coupling: services coupled together (direct in memory calls are more efficient that http or jms for service interaction) or service depending on same resources
  • Functionality: services belonging to the same business domain or applications layer
No matter what is the topology, keep in mind that it is not final and Fabric makes it really easy to move profiles across containers and containers accords hosts. So you have the tools to change and adapt when needed.

Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.

Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.

Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).

No matter whether you call it SOA or Microsservices, ESB or light integration layer, distributed applications have many moving parts and building one is a complex (and enjoyable) journey. The post here is quite high level and focusing on the happy paths only. For any real discussions, have a chat with Red Hat.

Apache Camel for Micro­service Architectures

I've been using microservice architectures before I knew they were called so. I used to work with pipeline applications made up of isolated modules that interact with each other through queues. Since then a number of (ex)ThoughtWorks gurus talked about microservices. First Fred George, then James Lewis and finally Martin Fowler blogged about microservices making it the next buzzword so every company wants to have few microservices. Nowadays there are #hashtags, endorsements, likes, trainings, even 2 day conference about it. The more I read and listen about microservice architectures, the more I realize how Apache Camel (and the accompanying projects around it) fits perfectly to this style of applications. In this post we will see how Apache Camel framework can help us create microservice style applications in Java without much hussle.

Microservices Characteristics
There is nothing new in microservices. Many applications have already been designed and implemented as such for a long time. Microservices is just a new term that describes a style of software systems that have certain characteristics and follow certain principles. It is an architectural style where an application or software system is composed of individual standalone services communicating using lightweight protocols in event based manner. The same way as TDD helps us to create decoupled single responsibility classes, microservices principles guide us to create simple applications at system level. Here we will not discuss the principles and characteristics of such architectures or argue whether it is a way of implementing SOA in practice or a totally new approach to application design, but rather look at the most common practices used for implementing microservices and how Apache Camel can helps us accomplish that in practice. There is not definitive list (yet) but if you read around or watch the videos posted above, you will notice that the following are quite common practices for creating microservices:

1. Small in size. The very fundamental principle of micro services says that each application is small in size and it only does one thing and does it well. It is debatable what is small or large, the number varies from 10 LOC to 1000 but form me I like the idea that it should be small enough to fit in your head. There are people with big heads, so even that is debatable ;) but I think as long as an application does one thing and does it well so that it is not considered a nanoservices, that is a good size.
Camel applications are inherently small in size. A camel context with couple of routes with error handling and helper beans is approximately 100 LOC. Thanks to Camel DSLs and URIs for abstracting endpoints, receiving an event either through HTTP or JMS, unmarshaling it, persisting and sending a response back is around 50 LOC. That is small enough to be tested end-to-end, rewritten and even thrown away without feel any remorse.

2. Having transaction boundaries. An application consisting of multiple microservices forms an eventually consistent system of systems where the state of the whole system is not known at any given time. This on its own creates a barrier for understanding and adopting microservices with teams who are not used to work with this kind of distributed applications. Even though the state of the whole system is not fixed, it is important to have transaction boundaries that define where a message currently belongs. 
Ensuring transactional behaviour across heteregenous systems is not an easy task, but Camel has great transactional capabilities. Camel has endpoints that can participate in transactions, transacted routes and error handlers, idempotent consumers and compensating actions, all of which help developers easily create services with transactional behavior.

3. Self monitoring. This is one of my favorite areas with microservices. Services should expose information that describes the state of various resources it depends on and the service itself. These are statistics such as average, min, max time to process a message, number of successful and failed messages, being able to track a message and so forth.
This is something you get OOTB with Camel without any effort. Each Camel application gathers JMX statistics by default for the whole application, individual routes, endpoints, etc. It will tell you how many messages have completed successfully, how many failed, where they failed, etc. This is not read only API, JMX allows also updating and tuning the application at run time, so based on these statistics, using the same API you can tune the application. Also the information can be accessed with tools such as jConsole, VisualVM, Hyperic HQ, exposed over HTTP using Jolokia or feed into a great web UI called hawtio.
If the functionality that is available OOTB doesn't fit your custom requirements, there multiple extension points such as the nagios, jmx, amazon cloudwatch and the new metrics components, or use Event Notifiers for custom events.
Logging in messaging applications is another challenge, but Camel's MDC logging combined with Throughput logger makes it easy to track individual messages or get aggregated statistics as part of the logging output.

5. Designed for failure - Each of the microservices can be down or unresponsive for some time but that should not bring the whole system down. Thus microservices should be fault tolerant and be able to recover when that is possible.
Camel has lots of helpful tools and patterns to cope with these scenarios too. Dead Letter Channel can make sure messages are not lost in case of failure, the retry policy can retry to send a message couple of times for certain error conditions using custom backoff method and collision avoidance. Patterns such as Load balancer which supports Circuit breaker, Failover and other policies, Throttler to make sure certain endpoints do not get overload, Detour, Sampler, are all needed in various failure scenarios. So why not use them rather than reinventing the wheel in each service.

6. Highly Configurable - It should be easy to configure the same application for high availability, scale it for reliability or throughput, or said another way: have different degrees of freedom through configuration.
When creating a Camel application using the DSLs, all we do is to define the message flow and configure various endpoints and other characteristics of the application. So Camel applications are highly configurable by design. When all the various options are externalized using properties component, it is possible to configure an application for different expectations and redeploy without touching the actual source code at all. Camel is so configurable that you can change an endpoint with another (for example replace HTTP endpoint with JMS) without changing the application code which we will cover next.

7. With smart endpoints.  Micro services favour RESTish protocols and lightweight messaging rather than Web Services.
Camel favors anything. It has HTTP support as no other framework. It has components for Asynchronous Http, GAE URL fetch service, Apache HTTP Client, Jetty, Netty, Servlet, Restlet, CXF and multiple data formats for serializing/deserializing messages. In addition the recent addition of Rest DSL makes REST a first class citizen in the Camel world and simply creating such services a lot. As for the queuing support, OOTB there are connectors for JMS, ActiveMQ, ZeroMQ, Amazon SQS, Amazon SNS, AMQP, Kestrel, Kafka, Stomp, you name it.

8. Testable. There is no common view on this characteristic. Some favor no testing at all and relying on business metrics. Some cannot afford to have bad business metrics at all. I like TDD and for me having the ability to test my business POJOs in isolation from the actual message flow, then test the flow separately by mocking some of the external endpoints is invaluable. Camel testing support can intercept and mock endpoints, simulate events, verify expectations with ease. Having a well tested microservice for the expected behavior is the only guarantee to have the whole system to work as expected.

9. Provisioned individually. The most important characteristics of microservices is that they run in isolation from other services most commonly as standalone Java applications. Camel can be embedded in Spring, OSGI or web containers. Camel can also run as a standalone Java application with embedded Jetty endpoints easily. But managing multiple processes, all running in isolation without a centralized tool is a hard job. This is what Fabric8 is made for. Fabric8 is developed by the same guys who developed Camel and supported by Red Hat JBoss. It is a poly Java application provisioning and management tool that can deploy and manage a variety of Java containers and standalone processes. To find out more about Fabric8, here is nice post by Christian Posta.

10. Language neutral. Having small and independently deployed applications allow developers to choose thebest suited language for the given task. Camel has XML, Java, Scala, Groovy and few other DSLs with similar syntax and capabilities .But if you don't want to you use Camel at all for a specific micro service, you can still use Fabric8 do deploy and manage applications written in other languages and run them as native processes.

In summary: Microservices are not strictly defined and that's the beauty. It is a lightweight style of implementing SOA that works. So is Apache Camel. It is not a full featured ESB, but it can be as part of JBoss Fuse. It is not a strictly defined specification driven project, but a lightweight tool that works and developers love it.

1. Micro-Service Architecture by Fred George (video)
2. Micro-Services - Java, the UNIX way by James Lewis
3. Microservices by Martin Fowler
4. ┬ÁServices by Peter Kriens
5. Micro Services the easy way with Fabric8 by James Strachan (with video)
6. Fabric8 by Red Hat
7. Meet Fabric8: An open­source integration platform by Christian Posta
8. Micro Services the easy way with Fabric8 by James Strachan

Clustered Idempotent Consumer Pattern with Infinispan

I've created a small project that shows how to use JBoss Infinispan with Apache Camel and the Idempotent Consumer Pattern to guarantee a message will not be processed twice in a clustered environment.
Imagine you have an application that has to scale out easily by deploying it on multiple containers. But the application has to process each unique request only once across the cluster.
The solution is simple: use Idempotent Consumer Pattern in Camel with a repository that can scale out easily. This is where Infinispan comes into play. Infinispan is extremely scalable, highly available key/value store and data grid. If you use InfinispanIdempotentRepository with an idempotent consumer, it will create an in-memory cache to store the requests, and the moment you start another instance of the application, the cache instances will sync and the idempotent consumers in all applications will not process existing requests any longer.
With this project (idempotent consumer demo5) you can start as many containers as you want, each container will start a rest endpoint on a new port starting from 8080 (http://localhost:8080/idempotent/KEY), and if you perform a GET request with a key, the subsequent requests with the same key to any other container will be rejected. Behind the scene Infinispan is replicating all the processed keys across the cluster of Camel applications and ensuring consistency.
The core of the application is the following route definition that finds a new free port number for each instance of the application:
Simple, isn't it.

Why bother contributing a Camel component?

Camel has a staggering number of connectors and that is thanks to you - the community. May be you also know a cool library that still doesn't have a Camel connector and wonder whether you should create a connector and contribute it to Apache Camel? Hopefully this article will give you the answer why you should do so.

The learning experience - There are great books about Apache Camel (mine is not bad either), but nothing can teach you better than banging your head while trying to write your first component. Understanding how Camel components works equals to understanding half of the project (the other half is the framework lifecycle and EIPs which are similar to components). So if you want to learn Camel and do more than than hello world, try writing a component.

A sense of accomplishment - the moment your code is pushed to Apache repository by a committer it becomes available to everyone and it is owned by the Apache foundation. But your name remains in the commit history forever, and no matter how much the code has changed, you can always claim: "Oh yeah, I've created that Camel component".

Becoming a Camel contributor - you get the right to add your name to the Camel team list. It is one of the most popular open source projects in the Java world and being part of something great is ... great. And who knows, may be this small contribution will spark a new passion and you will follow up with many other contributions and become a Camel committer in a short time.

Sharing can only bring you good - may be you are luckier than the average developer and you work for a company that let's you hack whatever you want for a day every fortnights. Or may be you are even luckier and you get paid for contributing to open source projects as my colleagues at Red Hat do (here is a list with Camel jobs at Red Hat for those interested). If you are neither of those groups, you can still work on something that is interesting to you and contribute to open source projects. All you need is couple of hours in a train or Starbucks.

Why a component? Writing a component is the easiest way to contribute something to Camel. It doesn't require deep Camel or EIP knowledge. You need good understanding of Camel components, good understand of the library you want to create connector for, and a mindset open for sharing.

This article is only the appetizer, if you liked it, the next article will explain how to create a Camel component.