Blogroll

Performance Tuning Ideas for Apache Camel

Every now and then, I get questions around optimising Camel applications with the argument that Camel is slow. Camel is just the glue connecting disparate systems, the routing engine is all in-memory, and it doesn’t require any persistent state. So 99% of the cases, performance issues are due to bottlenecks in other systems, or having the application design done without performance considerations. If that is the case, there isn’t much you can achieve by tuning Camel further, and you have to go back to the drawing board.

But sometimes, it might be worth squeezing few more milliseconds from your Camel routes. Tuning every application is very specific and dependent on the technology and the use case. Here are some ideas for tuning Camel based systems, which may apply for you (or not).

Endpoint Tuning


Endpoints in Camel are the integration points with other systems and the way they are configured will have a huge impact on the performance of the system. Understanding how different endpoints work and tuning those should be the one of the first places to start with. Here are few examples:
  • 
Messaging - If your Camel application is using messaging, the overall performance will be heavily dependent on the performance of the messaging system. There are too many factors to consider here, but main ones are:
  • Message broker - the network and disk speed, combined with the broker topology will shape the broker performance. To give you an idea, with ActiveMQ, a relational database based persistent store will perform around 50% of a file based store, and using network of brokers to scale horizontally will cost another 30% of performance. It is amazing how one configuration change in ActiveMQ can have huge impact on the messaging system and then the Camel application . There is a must read ActiveMQ tuning guide by Red Hat with lot's of details to consider and evaluate. Also a real life example from Chrisitan Posta showing how to speed up the broker 25x times in certain cases. Another recent article by Simon Green shows how to approach an ActiveMQ Tunning Adventure step by step.
  • Message client - if performance is a priority, there are also some hacks you can do on the ActiveMQ client side, such as: increasing TCP socketBufferSize and ioBufferSize, tuning the OpenWire protocol parameters, using message compression, batch acknowledgements with optimizeAcknowledge, asynchronous send with useAsyncSend, adjusting pre-fetch limit, etc. There are some nice slides again from Christina here and old but still very relevant video from Rob Davies about tuning ActiveMQ. All of these resources should give you enough ideas to experiment and improve the performance from messaging point of view.
  • 
Database writes - use batching whenever possible. You can use an aggregator to collect a number of entries before performing a batch operation to interact with the database (for example with the SQL component.
  • Working with templates
 - if you have to use a template component as part of the routing, try out the existing templating engines (FreeMarker, Velocity, SpringTeplate, Mustache, Chunk )  with a small test as the following one and measure which one performs better. There is a great presentation titled Performance optimisation for Camel by Christian Mueller with the source code supporting the findings (UPDATE: after this blog post was published, Christian created new slides with latest version of Camel 2.16.2 and Java 7/8, check out those too). From those measurements we can see that FreeMarker performs better than Velocity and SprintTemplates in general.
  • Using Web Services - whenever you have to use a web endpoint, the web container itself (has to be tuned separately. From Camel endpoint point of view, you can further optimise a little bit by skipping the unmarshalling if you don't need Java objects, and using asynchronous processing.
  • concurrentConsumers - there are a number of components (Seda, VM, JMS, RabbitMQ, Disruptor, AWS-SQS, etc) that support parallel consumption. Before using an endpoint, check the component documentation for thread pool or batch processing capabilities. To give you an idea, see how Amzon SQS processing can be improved through these options.

Data Type Choice

The type and the format of the data the is passing through Camel routes will also have performance implications. To demonstrate that let's see few examples.

  • Content based router, splitter, filter are examples of EIPs that perform some work based on the message content. And the type of the message affects the processing speed of these elements. Below is a chart from Christian Mueller's presentation, visualising how Content Based Router is performing with different kinds of messages:
    Content Based Routing for different data types
    Content Based Routing based on different data types
For example, if you have a large XML document in the Exchange, and based on it you perform content based routing, filtering, etc., that will affect the speed of the route. Instead you can extract some key information from the document and populate the Exchange headers for faster access and routing later.
  • Marshaling/Unmarshaling - similarly to the templating engines, different 
data format covenrtors perform differently. To see some metrics check again Christian's presentation, but also keep in mind that performance of the supported data formats may vary between different versions and platforms so measure it for your use case.
  • Streaming  - Camel streaming and stream caching are one of the underrated features that can be useful for dealing with large messages.
  • Claim check EIP - if the application logic allows it, consider using claim check pattern to improve performance and reduce resource consumption.

Multithreading

Camel offers multithreading support in a number of places. Using those can improve the application performance too.

  • Paralel processing EIPs - the following Camel EIP implementations support parallel processing - multicast, recipient list, splitter, delayer, wiretap, throttler, error handler. If you are going to enable parallel processing for those, it would be even better if you also provide a custom thread pool specifically tuned for your use case rather than relying on Camel's default thread pool profile.
  • Threads DSL
 construct - some Camel endpoints (such as the File consumer) are single threaded by design and cannot be parallelized at endpoint level. In case of File consumer, a single thread picks a file at a time and processes it through the route until it reaches the end of the route and then the consumer thread picks the next file. This is when Camel Threads construct can be useful. As visualised below, File consumer thread can pick a file and pass it to a thread from the Threads construct for further processing. Then the File consumer can pick another file without waiting for the previous Exchange to complete processing fully.
    Parallel File Consuming
    Parallel File Consuming
  • Seda component - Seda is another way to achieve parallelism in Camel. The Seda component has in-memory list to accumulate incoming messages from the producer and concurrentConsumers to process those incoming request in parallel by multiple threads.
  • Asynchronous Redelivery/Retry - if you are using an error handler with a redelivery policy as part of the routing process, you can configure it to be asynchronous and do the redeliveries in a separate thread. That will use a separate thread pool for the redelivery not block the main request processing thread while waiting. If you need long delayed redeliveries, it might be a better approach to use ActiveMQ broker redelivery (that is different from consumer redelivery BTW) where the redeliveries will be persisted on the message broker and not kept in Camel application memory. Another benefit of this mechanism is that the redeliveries will survive application restart and also play nicely when the application is clustered. I have described different retry patterns in Camel Design Patterns book.

Other Optimisations

There are few other tricks you can do to micro-tune Camel further. 
  • Logging configurations - hopefully you don't have to log every message and its content on the production environment. But if you have to, consider using some asynchronous logger. On a high throughput system, ane option would be to log statistics and aggregated metrics through Camel Throughput logger. Throughput logger allows logging aggregated statistics on fixed intervals or based on the number of processed messages rather than  per message bases. Another option would be to use the not so popular Camel Sampler EIP and log only sample messages every now and then.
  • Disable JMX - by default, Camel JMX instrumentation is enabled which creates a lot of MBeans. This allows monitoring and management of Camel runtime, but also has some performance hit and requires more resources. I still remember the time when I had to fully turn off JMX in Camel in order to run it with 512MB heap on a free AWS account. As a minimum, consider whether you need any JMX enabled at all, and if so whether to use RoutesOnly, Default or Extended JMX profiles. 
  • Message Histroy - Camel implements the Message History EIP and runs it by default. While on development environmnet, it might be useful to see every endpoint a message has been too, but on the produciton environment you might consider to disable this feature.
  • Original message - Every Camel route will make a copy of the original incoming message before any modifications to it. This pristine copy of the message is kept in case it is needed to be redelivered during error handling or with onCompletion construct. If you are not using these features, you can disable creating and storing the original state of every incoming message.
  • Other customisations -  Almost every feature in CamelContext can be customized. For example you can use lazyLoadTypeConverters for a faster application startup, or configure the shutdownStrategy for a quicker shutdown when there are inflight messages, or a use a custom UuidGenerator that performs faster, etc.

Application Design

All of the previous tunings are micro optimizations compared to the application design and architecture. If your application is not designed for horizontal scalability and performance, sooner or later the small tuning hacks will hit their limit. The chances are, what you are doing has been done previously, and instead of reinventing the wheel or coming up with some clever designs, learn from the experience of others and use well known patterns, principles and practises. Use principles from SOA, Microservices architectures, resiliency principles, messaging best practises, etc. Some of those patterns such as Parallel Pipelines, CQRS, Load Leveling, Circuit Breaker are covered in Camel Design Patterns book and do help to improve the overall application design.

JVM

There are many articles about tuning the JVM. Here I only want to mention the JVM configuration generation application by Red Hat which can generate for you JVM configurations based on the latest industry best practices. You can use it as long as you have a Red Hat account (which is free for developers anyway). Using the latest JVM and latest version of Camel (with its updated dependencies) is another way to improve application performance for free.


OS

You can squeeze the application only so much. In order to do proper high load processing, tuning the host system is a must too. To get an idea for the various OS level options, have a look at the following check list from the Jetty project.


In Conclusion

This article is here just to give you some ideas and show you the extend of the possible areas to consider when you have to improve the performance of a Camel application. Instead of looking for a magical recipe or go through a checklist, do small incremental changes supported by measurements until you get to a desired state. And rather than focusing on micro optimisations and hacks, have an holistic view of the system, get the design right, and start tuning from the host system, to JVM, CamelContext, routing elements, endpoints and the data itself.

Using well known patterns, principles and practises with focus on simple and scalable design is always a good start. Good luck.

Idempotent Consumer EIP Icon

While writing about the Idempontet Filter Patter in my recent Camel Design Patterns book, I wanted to visualise the Idempontet Consumer EIP on the Camel routes but couldn't find an icon for it. Inspired by existing Normalizer and Resequencer icons, I've created an Idempotent Consumer Icon.

If you look at the Normalizer pattern, it uses different shapes to emphasise that incoming messages are of different data formats. And the Resequencer has the same number of incoming and outgoing messages, but reordered.
Similarly, for the Idempotent Consumer I've used a circle and square to represent different incoming messages (that is not different data formats, but different business data). And to emphasise that it is a stateful pattern and can remember past messages, I've put the duplicate square messages apart from one another. Then from the icon we can see that, the pattern removes the duplicate square messages and allows only unique message to pass on.

Couple of days after that, I've read A Decade of Enterprise Integration Patterns: A Conversation with the Authors where Gregor said this:

Olaf: And what would you do differently now?
Gregor: I would make an icon for the Idempotent Receiver pattern, which describes a receiver that can process the same message multiple times without any harm. Somehow we seem to have missed that one. 

Not sure whether one day the icon will become part of the existing EIP icon set or not, but I've found it visually helpful to depict the idempotent behaviour. You can download a .png, visio and OmniGraffle stencil from here.

Camel Design Patterns eBook is Out

I've been involved with Apache Camel for many years now and apart from the occasional contributions, and blogging, I've used it in tens of projects over the years. That includes projects for large broadcasting companies, newspapers, mobile operators, oil companies, airlines, digital agencies, government organisations, you name it. One common theme across all these projects is that the development team loves Camel. Camel has always been a flexible and productive tool that gives the developers the edge over the changing requirements and short deadlines.

Having seen many successful Camel projects, I try to share my experiences through blogging, but this time decided to invest more time and create an ebook called Camel Design Patterns. It is not another Camel book documenting the framework itself and the individual Enterprise Integration Patterns, but rather a collection of SOA, Microservices, Messaging, Cloud, Resiliency patterns that I've used in Camel based solutions day by day. Its format is similar to a series of essays or blog posts with high level examples showing different techniques and Camel tips for designing and architecting modern Camel applications.

Table of Contents 

  • I Foundational Patterns
    • Edge Component Pattern
    • VETRO Pattern
    • CQRS Pattern
    • Canonical Data Model Pattern
    • Reusable Route Pattern (new)
    • Idempotent Filter Pattern
    • External Configuration Pattern
  • II Error Handling Patterns
  • III Deployment Patterns
    • Service Instance Pattern
    • Singleton Service Pattern
    • Parallel Pipeline Pattern
    • Load Leveling Pattern
    • Bulkhead Pattern
    • Service Consolidation Pattern
It is a live book, and depending on interest and feedback, I plan to add more chapters and use cases in the future. The book costs around aVenti Espresso Frappuccino and for now it is available only on leanpub. To get a feel about the content, have a look at the sample Data Integrity Pattern chapter.

I hope you find this ebook useful and looking forward to receiving your feedback.

JBoss Fuse and JBoss BPM Suite Integrated

Last week Apache Camel v2.16 was released with over 600 issues resolved (bug fixes, updates and new features). Claus blogged about top 10 highlights already and here I want to briefly show you a use case about my tiny contribution in this release: camel-jbpm component.

This component is intended to make integration with jBPM (a Business Process Management Application) easier. It uses Remote Java API and basically does REST calls to jBPM to manage business processes (start, stop, abort), execute Business Rules, manage Human Tasks, etc. So rather than manually creating a REST client or using Remote API, you can use the Camel component. The remote API provides also HornetQ based integration, but I've dropped out that part from the Camel component as it didn't play nicely with OSGI.

In the sample use case (which is inspired by a real one) there is an integration application based on Camel that runs on JBoss Fuse which is supposed to do real time integration. And whenever an error occurs in the integration route, the error details are sent to jBPM for evaluation and review by human being if necessary. To demonstrate that I've created two separate projects:
  • A Camel project that runs on Fuse, handles any exception and sends the details to jBPM
  • A Business Process project that runs on jBPM, classifies the errors using Drools and Complex Event Processing and creates Human Tasks if the error is critical.

The Camel route below is self explanatory, there is a route that generates 10 exceptions with half a second delay each and an error handler that catches those exceptions and sends them to jBPM. Notice that it also populates CamelJBPMParameters header with the details about the error.

You can find the sample code on github with instructions about how to build and run it on Fuse.

The second part of the demo consist of a jBPM process that receives the errors and processes them. The BPMN2 diagram of the process is below:

The process have input parameters matching the values from CamelJBPMParameters header. In the first step it initializes a RequestQualifier object that is marked as non-critical. In the second step we use Drools and Complex Event Processing to qualify errors. The idea with CEP is that we want to mark an error as critical only if there were more than 5 errors in 10 seconds time window. Also to prevent marking multiple errors as critical and flooding the system with Human Tasks, we limit it to only 1 critical error at most in 10 minutes interval. All these conditions are written in Drools as following:

Once the request with the error details has been qualified (as critical or not), we use that information to do conditional routing. All non critical errors are logged and the business processes complete as there are no further action defined for them. We can see the full list of completed processes and error related information in the following screen shot:
On the other branch of the business process, if an error has been marked as critical, we do create a Human Task and the process stops at this point. This requires a user to assign the task and take some actions. Once the Human task is claimed and completed, then the business process continues and completes in the next step.
From the ten error messages generated in quick succession, only one is qualified as critical and generates Human Task and the remaining nine gets logged and completed as expected.
The business process project with instructions on how to deploy it to jBPM is on github too.

JBoss Fuse Deployment Considerations

Fuse is a framework for creating distributed applications and as such it has many moving parts. There are bundles, features, profiles, containers and various ways of combining those into the resulting deployment architecture. Here are some thoughts and guidelines based on the field experiences from a number of projects.

If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.

An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.

A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)

An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.

Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).

So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.

Next, let's have a look at different types of containers we have in a typical Fuse application:

  • Fabric Server - This is where the ZooKeeper server is running, together with the maven proxy, git server and Hawtio instance.
  • A-MQ Container - runs A-MQ server instance(s).
  • Worker/ESB container - this is where the business/integration logic is running. That is typically Camel and CXF based (our order profile would be deployed here).
  • Insight container - is where the Elastic Search server runs.
  • Fabric Gateway - used for discovery and load balancing of CXF or A-MQ endpoints.
One point that is worth clarifying is that the containers are generic, it is the profile that is deployed that gives a container a role. And in theory it is possible to deploy various combination of profiles from the above categories to the the same container but that is not a wise idea as these profiles are fundamental for the rest of the application and it is better to isolate and allocate enough resources to each of them rather than making them share the same JVM. Under a heavy load on the message broker you wouldn't want to have ZK server struggling for resources and jeopardize the health of the whole Fabric. On the other hand deploying business profiles (order and invoice profile for example) to the same container is perfectly fine if it makes sense from non functional requirements point of view. So try to put the above profiles/responsibilities/roles into separate containers and split/group your business profiles into containers based on the use case. More on this topic below.

So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.

Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!

The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.

The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:

  • Business criticality: highly critical services on dedicated containers and safest hosts
  • Load: services with high load on hosts with more resources
  • Coupling: services coupled together (direct in memory calls are more efficient that http or jms for service interaction) or service depending on same resources
  • Functionality: services belonging to the same business domain or applications layer
No matter what is the topology, keep in mind that it is not final and Fabric makes it really easy to move profiles across containers and containers accords hosts. So you have the tools to change and adapt when needed.

Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.

Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.

Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).

No matter whether you call it SOA or Microsservices, ESB or light integration layer, distributed applications have many moving parts and building one is a complex (and enjoyable) journey. The post here is quite high level and focusing on the happy paths only. For any real discussions, have a chat with Red Hat.