Apache Camel for Micro­service Architectures

I've been using microservice architectures before I knew they were called so. I used to work with pipeline applications made up of isolated modules that interact with each other through queues. Since then a number of (ex)ThoughtWorks gurus talked about microservices. First Fred George, then James Lewis and finally Martin Fowler blogged about microservices making it the next buzzword so every company wants to have few microservices. Nowadays there are #hashtags, endorsements, likes, trainings, even 2 day conference about it. The more I read and listen about microservice architectures, the more I realize how Apache Camel (and the accompanying projects around it) fits perfectly to this style of applications. In this post we will see how Apache Camel framework can help us create microservice style applications in Java without much hussle.

Microservices Characteristics
There is nothing new in microservices. Many applications have already been designed and implemented as such for a long time. Microservices is just a new term that describes a style of software systems that have certain characteristics and follow certain principles. It is an architectural style where an application or software system is composed of individual standalone services communicating using lightweight protocols in event based manner. The same way as TDD helps us to create decoupled single responsibility classes, microservices principles guide us to create simple applications at system level. Here we will not discuss the principles and characteristics of such architectures or argue whether it is a way of implementing SOA in practice or a totally new approach to application design, but rather look at the most common practices used for implementing microservices and how Apache Camel can helps us accomplish that in practice. There is not definitive list (yet) but if you read around or watch the videos posted above, you will notice that the following are quite common practices for creating microservices:

1. Small in size. The very fundamental principle of micro services says that each application is small in size and it only does one thing and does it well. It is debatable what is small or large, the number varies from 10 LOC to 1000 but form me I like the idea that it should be small enough to fit in your head. There are people with big heads, so even that is debatable ;) but I think as long as an application does one thing and does it well so that it is not considered a nanoservices, that is a good size.
Camel applications are inherently small in size. A camel context with couple of routes with error handling and helper beans is approximately 100 LOC. Thanks to Camel DSLs and URIs for abstracting endpoints, receiving an event either through HTTP or JMS, unmarshaling it, persisting and sending a response back is around 50 LOC. That is small enough to be tested end-to-end, rewritten and even thrown away without feel any remorse.

2. Having transaction boundaries. An application consisting of multiple microservices forms an eventually consistent system of systems where the state of the whole system is not known at any given time. This on its own creates a barrier for understanding and adopting microservices with teams who are not used to work with this kind of distributed applications. Even though the state of the whole system is not fixed, it is important to have transaction boundaries that define where a message currently belongs. 
Ensuring transactional behaviour across heteregenous systems is not an easy task, but Camel has great transactional capabilities. Camel has endpoints that can participate in transactions, transacted routes and error handlers, idempotent consumers and compensating actions, all of which help developers easily create services with transactional behavior.

3. Self monitoring. This is one of my favorite areas with microservices. Services should expose information that describes the state of various resources it depends on and the service itself. These are statistics such as average, min, max time to process a message, number of successful and failed messages, being able to track a message and so forth.
This is something you get OOTB with Camel without any effort. Each Camel application gathers JMX statistics by default for the whole application, individual routes, endpoints, etc. It will tell you how many messages have completed successfully, how many failed, where they failed, etc. This is not read only API, JMX allows also updating and tuning the application at run time, so based on these statistics, using the same API you can tune the application. Also the information can be accessed with tools such as jConsole, VisualVM, Hyperic HQ, exposed over HTTP using Jolokia or feed into a great web UI called hawtio.
If the functionality that is available OOTB doesn't fit your custom requirements, there multiple extension points such as the nagios, jmx, amazon cloudwatch and the new metrics components, or use Event Notifiers for custom events.
Logging in messaging applications is another challenge, but Camel's MDC logging combined with Throughput logger makes it easy to track individual messages or get aggregated statistics as part of the logging output.

5. Designed for failure - Each of the microservices can be down or unresponsive for some time but that should not bring the whole system down. Thus microservices should be fault tolerant and be able to recover when that is possible.
Camel has lots of helpful tools and patterns to cope with these scenarios too. Dead Letter Channel can make sure messages are not lost in case of failure, the retry policy can retry to send a message couple of times for certain error conditions using custom backoff method and collision avoidance. Patterns such as Load balancer which supports Circuit breaker, Failover and other policies, Throttler to make sure certain endpoints do not get overload, Detour, Sampler, are all needed in various failure scenarios. So why not use them rather than reinventing the wheel in each service.

6. Highly Configurable - It should be easy to configure the same application for high availability, scale it for reliability or throughput, or said another way: have different degrees of freedom through configuration.
When creating a Camel application using the DSLs, all we do is to define the message flow and configure various endpoints and other characteristics of the application. So Camel applications are highly configurable by design. When all the various options are externalized using properties component, it is possible to configure an application for different expectations and redeploy without touching the actual source code at all. Camel is so configurable that you can change an endpoint with another (for example replace HTTP endpoint with JMS) without changing the application code which we will cover next.

7. With smart endpoints.  Micro services favour RESTish protocols and lightweight messaging rather than Web Services.
Camel favors anything. It has HTTP support as no other framework. It has components for Asynchronous Http, GAE URL fetch service, Apache HTTP Client, Jetty, Netty, Servlet, Restlet, CXF and multiple data formats for serializing/deserializing messages. In addition the recent addition of Rest DSL makes REST a first class citizen in the Camel world and simply creating such services a lot. As for the queuing support, OOTB there are connectors for JMS, ActiveMQ, ZeroMQ, Amazon SQS, Amazon SNS, AMQP, Kestrel, Kafka, Stomp, you name it.

8. Testable. There is no common view on this characteristic. Some favor no testing at all and relying on business metrics. Some cannot afford to have bad business metrics at all. I like TDD and for me having the ability to test my business POJOs in isolation from the actual message flow, then test the flow separately by mocking some of the external endpoints is invaluable. Camel testing support can intercept and mock endpoints, simulate events, verify expectations with ease. Having a well tested microservice for the expected behavior is the only guarantee to have the whole system to work as expected.

9. Provisioned individually. The most important characteristics of microservices is that they run in isolation from other services most commonly as standalone Java applications. Camel can be embedded in Spring, OSGI or web containers. Camel can also run as a standalone Java application with embedded Jetty endpoints easily. But managing multiple processes, all running in isolation without a centralized tool is a hard job. This is what Fabric8 is made for. Fabric8 is developed by the same guys who developed Camel and supported by Red Hat JBoss. It is a poly Java application provisioning and management tool that can deploy and manage a variety of Java containers and standalone processes. To find out more about Fabric8, here is nice post by Christian Posta.

10. Language neutral. Having small and independently deployed applications allow developers to choose thebest suited language for the given task. Camel has XML, Java, Scala, Groovy and few other DSLs with similar syntax and capabilities .But if you don't want to you use Camel at all for a specific micro service, you can still use Fabric8 do deploy and manage applications written in other languages and run them as native processes.

In summary: Microservices are not strictly defined and that's the beauty. It is a lightweight style of implementing SOA that works. So is Apache Camel. It is not a full featured ESB, but it can be as part of JBoss Fuse. It is not a strictly defined specification driven project, but a lightweight tool that works and developers love it.

1. Micro-Service Architecture by Fred George (video)
2. Micro-Services - Java, the UNIX way by James Lewis
3. Microservices by Martin Fowler
4. ┬ÁServices by Peter Kriens
5. Micro Services the easy way with Fabric8 by James Strachan (with video)
6. Fabric8 by Red Hat
7. Meet Fabric8: An open­source integration platform by Christian Posta
8. Micro Services the easy way with Fabric8 by James Strachan

Clustered Idempotent Consumer Pattern with Infinispan

I've created a small project that shows how to use JBoss Infinispan with Apache Camel and the Idempotent Consumer Pattern to guarantee a message will not be processed twice in a clustered environment.
Imagine you have an application that has to scale out easily by deploying it on multiple containers. But the application has to process each unique request only once across the cluster.
The solution is simple: use Idempotent Consumer Pattern in Camel with a repository that can scale out easily. This is where Infinispan comes into play. Infinispan is extremely scalable, highly available key/value store and data grid. If you use InfinispanIdempotentRepository with an idempotent consumer, it will create an in-memory cache to store the requests, and the moment you start another instance of the application, the cache instances will sync and the idempotent consumers in all applications will not process existing requests any longer.
With this project (idempotent consumer demo5) you can start as many containers as you want, each container will start a rest endpoint on a new port starting from 8080 (http://localhost:8080/idempotent/KEY), and if you perform a GET request with a key, the subsequent requests with the same key to any other container will be rejected. Behind the scene Infinispan is replicating all the processed keys across the cluster of Camel applications and ensuring consistency.
The core of the application is the following route definition that finds a new free port number for each instance of the application:
Simple, isn't it.

Why bother contributing a Camel component?

Camel has a staggering number of connectors and that is thanks to you - the community. May be you also know a cool library that still doesn't have a Camel connector and wonder whether you should create a connector and contribute it to Apache Camel? Hopefully this article will give you the answer why you should do so.

The learning experience - There are great books about Apache Camel (mine is not bad either), but nothing can teach you better than banging your head while trying to write your first component. Understanding how Camel components works equals to understanding half of the project (the other half is the framework lifecycle and EIPs which are similar to components). So if you want to learn Camel and do more than than hello world, try writing a component.

A sense of accomplishment - the moment your code is pushed to Apache repository by a committer it becomes available to everyone and it is owned by the Apache foundation. But your name remains in the commit history forever, and no matter how much the code has changed, you can always claim: "Oh yeah, I've created that Camel component".

Becoming a Camel contributor - you get the right to add your name to the Camel team list. It is one of the most popular open source projects in the Java world and being part of something great is ... great. And who knows, may be this small contribution will spark a new passion and you will follow up with many other contributions and become a Camel committer in a short time.

Sharing can only bring you good - may be you are luckier than the average developer and you work for a company that let's you hack whatever you want for a day every fortnights. Or may be you are even luckier and you get paid for contributing to open source projects as my colleagues at Red Hat do (here is a list with Camel jobs at Red Hat for those interested). If you are neither of those groups, you can still work on something that is interesting to you and contribute to open source projects. All you need is couple of hours in a train or Starbucks.

Why a component? Writing a component is the easiest way to contribute something to Camel. It doesn't require deep Camel or EIP knowledge. You need good understanding of Camel components, good understand of the library you want to create connector for, and a mindset open for sharing.

This article is only the appetizer, if you liked it, the next article will explain how to create a Camel component.

10 Good Reasons for using Apache Camel to create Microservices

Microservices is an architectural style for distributed applications that is taking over the developer community in high pace. There are tens of blog posts on the net every day, conferences and even books on the topic already.
To contribute to all the buzz around it I've written a post where I gave 10 good reasons why Apache Camel and Fabric8 are great tools for creating microservices applications. If you are interested in microservices topic, the article is published on

Circuit Breaker Pattern in Apache Camel

Camel is very often used in distributed environments for accessing remote resources. Remote services may fail for various reasons and periods. For services that are temporarily unavailable and recoverable after short period of time, a retry strategy may help. But some services can fail or hang for longer period of time making the calling application unresponsive and slow. A good strategy to prevent from cascading failures and exhaustion of critical resources is the Circuit Breaker pattern described by Michael Nygard in the Release It! book.
Circuit Breaker is a stateful pattern that wraps the failure-prone resource and monitors for errors. Initially the Circuit Breaker is in closed state and passes all calls to the wrapped resource. When the failures reaches a certain threshold, the circuit moves to open state where it returns error to the caller without actually calling the wrapped resource. This prevents from overloading the already failing resource. While at this state, we need a mechanism to detect whether the failures are over and start calling the protected resource. This is where the third state called half-open comes into play. This state is reached after a certain time following the last failure. At this state, the calls are passed through to the protected resource, but the result of the call is important. If the call is successful, it is assumed that the protected resource has recovered and the circuit is moved into closed state, and if the call fails, the timeout is reset, and the circuit is moved back to open state where all calls are rejected. Here is the state diagram of Circuit Breaker from Martin Fowler's post:

How Circuit Breaker is implemented in Camel?

Circuit Breaker is available in the latest 2.14 version of Camel as a Load balancer policy. Camel Load Balancer already has policies for Round Robin, Random, Failover, etc. and now also CircuiBreaker policy.
Here is an example load balancer that uses Circuit Breaker policy with threshold of 2 errors and halfOpenAfter timeout of 1 second. Notice also that this policy applies only to errors caused by MyCustomException
And here is the same example using Spring XML DSL:

A Docker Maven Plugin for Integration Testing

What is Docker?
Docker is the buzzword that is taking the DevOps world. If you don't know yet what is Docker, be warned, you will find yourself using it one way or another very soon. The rest of this post assumes some basic understanding of Docker, but if you are not familiar with it now, I'm sure you will come back later and read this.
Docker is ideal for integration testing, complex demos of distributed systems or even running production systems. It is an open source software container. You can imagine it as a very lightweight and ultra fast virtual machine.
An example
Inspired by the "Integration testing with Maven and Docker" article and using Docker Java API I've created a simple Docker Maven Plugin that can manage Docker containers. Given a Docker image, the plugin will create a container from it and start it as part of maven build process and stop and remove the container when the build process is finished. If the image is not available locally it will pull it down from the public Docker registry before creating a container from it.

The following integration test in Apache Camel is ignored because it requires a running Redis instance:

To make it pass, we can use docker-maven-plugin with a Redis image and 6379 port accessible for the test:

The plugin will start a Docker container at compile phase that has a running Redis instance and shut it down at post-integration-test phase.
This is a very simple example, but the plugin can support more advanced scenarios with multiple images configured differently and started/stopped at different phases. Enjoy.

Software engineering vs Software consulting

I've seen articles with this topic in the past but have to admit to have never read one. If you have, probably you already know which one is the better profession so stop reading this now. If you are still interested, this is my take on the topic.

Software engineering
Software engineering is great profession. I was engineer in companies in different sizes, from startups with 5 people to organizations with 5K stuff. In general the work load is usually predictable, products to work and roadmaps known well in advance. Technology may change, but usually the company and the team already have experience with specific languages and tools and uses those during the everyday job. There are quiet times where there is nothing to do and say in the standups and there are overwhelming times where you get back and wrist pain from typing code for hours. But overall engineering involves predictable work where you have to ship lot's of code regularly. And if you are lucky enough to work in a normal company, you might be doing agile development with all the rituals and practices, the company may send you to trainings and conferences and even have some days in a month to hack something based on your interest. What else to want form life, isn't it.

Software consulting
Consulting is tough. Depending on the nature of consulting, you might be at different customer every other week or month. Different customers from different industries, each of them with different habits and rituals. There are days and even weeks where you might not write one line of code, and there are days where you have to write code without any tests that goes directly to a production system. No dedicated time to learn new stuff or do hack whatever you want Fridays. Customers pay for getting things done in the way they think is best. Overall lets opportunity to have an impact on the organization or change its culture towards something better. The good side of all this can be summarized with "What doesn't kill you, makes you stronger". There is no training or conference that can teach you new things in such a short time than a real Customer.

Software craftsmanship
I enjoy both professions and try to get most out of them during the different stages of my short career so far. My drive is, regardless of your job title, be a software craftsman first. Being software craftsman is a state of mind, it is your attitude towards the everyday activities at work. Whatever you do, do it well and then be proud of it. Don't blindly follow any old habit or ritual that will make you write "WTF" code. Write well-crafted software that adds real value, and next time make it even better. To find out more about software craftsmanship, join a user group(like LSCC), read a book, and pass the craft on.

Master/Slave Failover for Camel Routes

One way to implement a master/slave failover pattern is to have a cluster of instances of an application where one instance (the master) is currently active and the other instances (the slaves) are on standby, ready to take over whenever the master fails. Some projects provide this kind of master/slave support out of the box:
Creating a failover deployment for Apache Karaf is straight forward: we start two or more Karaf instances and let them point to the same lock (file system or database). Then the first instance that starts gets the lock and becomes the master while the other instances will be waiting to get the lock before starting the bundles. In addition Karaf offers hot standby functionality where some bundles are started even in the slave instances and other bundles wait for to get the lock.

Apache ActiveMQ offers couple of ways for creating master/slave configurations but the simplest is to start two or more instances of ActiveMQ pointing to the same datasource(file or database) where the first broker gets the lock and becomes the master and the second and other brokers become slaves, waiting for the lock. Simple.

What about Camel? How can we have multiple routes (in one or separate containers) where one is the master (in running state) and the other routes are waiting to take over as soon as the master route stops ensuring high availability at route level? There are couple of components providing such a capability and all of the them rely on having some kind of centralized external system used as a lock.

1. Camel Quartz component has clustering support.
- If you are using quartz consumers, in clustered mode, you can have only one of the routes triggered at a time.
- Or if a quartz based CronScheduledRoutePolicy is used, in clustered mode, only one of the routes will be started/stopped.
Both of these options rely on having quartz to be configured with a datasource that is shared among all the routes in the cluster. This usage is not exactly master/slave but will have the same effect at the end.

2. Camel Zookeeper component offers a RoutePolicy that can start/stop routes in master/slave fashion. The first route that gets the lock will be started where the remaining routes will be waiting to get the lock. One advantage of this component is that it can be configured to have more than one master running.

3. Camel JGroups component also has master/slave capability using JGroupsFilters.

4. JBoss Fuse Master component is probably the easiest way to have master/slave setup in a Fuse environment. Internally it relies on Zookeeper's znode capability similarly to zookeeper component above.

5. This is not implemented yet but in theory it is possible to implement a RoutePolicy using ActiveMQ's exclusive consumers feature that provides a distributed lock. Do let me know if you implement this ;)

6. Use database as a lock. Christian Schneider has demonstrated how to have "Standby failover for Apache Camel routes" using a database here.

A Camel Demo for Amazon's Simple Worklfow Service

In a previous post I explained why AWS SWF service is good and announced the new Camel SWF component. Now the component documentation is ready and here is a simplistic fully working demo. It consist of three independent standalone Camel routes:
A workflow producer allows us to interact with a workflow. It can start a new workflow execution, query its state, send signals to a running workflow, or terminate and cancel it. In our demo, the WorkflowProducer starts a route that schedules 10 workflow executions where the each execution receives as an argument a number.
Once a workflow execution is scheduled, we need a process that will decide what are the next steps for it. In Camel it is done using a Workflow Consumer. A workflow consumer represents the workflow logic. When it is started, it will start polling workflow decision tasks and process them. In addition to processing decision tasks, a workflow consumer route, will also receive signals (send from a workflow producer) or state queries. The primary purpose of a workflow consumer is to schedule activity tasks for execution using activity producers. Actually activity tasks can be scheduled only from a thread started by a workflow consumer.
The logic in our demo decider is simple: if the incoming argument is greater than 5, we schedule a task for execution. Otherwise the workflow will complete as there are no other tasks to be executed. Notice that it also has branches for handing signal and state query events.

The final peace of our distributed (since it consists of three independent applications) workflow application is the ActivityConsumer that actually performs some calculations. It has the simplest possible implementation: increments the given argument and returns it.
All you need to do to run this demo is to create the appropriate workflow domain and add your key/secret to the route.

About Me