Circuit Breaker Pattern in Apache Camel

Camel is very often used in distributed environments for accessing remote resources. Remote services may fail for various reasons and periods. For services that are temporarily unavailable and recoverable after short period of time, a retry strategy may help. But some services can fail or hang for longer period of time making the calling application unresponsive and slow. A good strategy to prevent from cascading failures and exhaustion of critical resources is the Circuit Breaker pattern described by Michael Nygard in the Release It! book. 
Circuit Breaker is a stateful pattern that wraps the failure-prone resource and monitors for errors. Initially the Circuit Breaker is in closed state and passes all calls to the wrapped resource. When the failures reaches a certain threshold, the circuit moves to open state where it returns error to the caller without actually calling the wrapped resource. This prevents from overloading the already failing resource. While at this state, we need a mechanism to detect whether the failures are over and start calling the protected resource. This is where the third state called half-open comes into play. This state is reached after a certain time following the last failure. At this state, the calls are passed through to the protected resource, but the result of the call is important. If the call is successful, it is assumed that the protected resource has recovered and the circuit is moved into closed state, and if the call fails, the timeout is reset, and the circuit is moved back to open state where all calls are rejected. Here is the state diagram of Circuit Breaker from Martin Fowler's post:

How Circuit Breaker is implemented in Camel?

Circuit Breaker is available in the latest 2.14 version of Camel as a Load balancer policy. Camel Load Balancer already has policies for Round Robin, Random, Failover, etc. and now also CircuiBreaker policy.
Here is an example load balancer that uses Circuit Breaker policy with threshold of 2 errors and halfOpenAfter timeout of 1 second. Notice also that this policy applies only to errors caused by MyCustomException
And here is the same example using Spring XML DSL:

A Docker Maven Plugin for Integration Testing

What is Docker?

Docker is the buzzword that is taking the DevOps world. If you don't know yet what is Docker, be warned, you will find yourself using it one way or another very soon. The rest of this post assumes some basic understanding of Docker, but if you are not familiar with it now, I'm sure you will come back later and read this.
Docker is ideal for integration testing, complex demos of distributed systems or even running production systems. It is an open source software container. You can imagine it as a very lightweight and ultra fast virtual machine.

An example

Inspired by the "Integration testing with Maven and Docker" article and using Docker Java API I've created a simple Docker Maven Plugin that can manage Docker containers. Given a Docker image, the plugin will create a container from it and start it as part of maven build process and stop and remove the container when the build process is finished. If the image is not available locally it will pull it down from the public Docker registry before creating a container from it.

The following integration test in Apache Camel is ignored because it requires a running Redis instance:

To make it pass, we can use docker-maven-plugin with a Redis image and 6379 port accessible for the test:

The plugin will start a Docker container at compile phase that has a running Redis instance and shut it down at post-integration-test phase.
This is a very simple example, but the plugin can support more advanced scenarios with multiple images configured differently and started/stopped at different phases. Enjoy.

Software engineering vs Software consulting

I've seen articles with this topic in the past but have to admit to have never read one. If you have, probably you already know which one is the better profession so stop reading this now. If you are still interested, this is my take on the topic.

Software engineering

Software engineering is great profession. I was engineer in companies in different sizes, from startups with 5 people to organizations with 5K stuff. In general the work load is usually predictable, products to work and roadmaps known well in advance. Technology may change, but usually the company and the team already have experience with specific languages and tools and uses those during the everyday job. There are quiet times where there is nothing to do and say in the standups and there are overwhelming times where you get back and wrist pain from typing code for hours. But overall engineering involves predictable work where you have to ship lot's of code regularly. And if you are lucky enough to work in a normal company, you might be doing agile development with all the rituals and practices, the company may send you to trainings and conferences and even have some days in a month to hack something based on your interest. What else to want form life, isn't it.

Software consulting

Consulting is tough. Depending on the nature of consulting, you might be at different customer every other week or month. Different customers from different industries, each of them with different habits and rituals. There are days and even weeks where you might not write one line of code, and there are days where you have to write code without any tests that goes directly to a production system. No dedicated time to learn new stuff or do hack whatever you want Fridays. Customers pay for getting things done in the way they think is best. Overall lets opportunity to have an impact on the organization or change its culture towards something better. The good side of all this can be summarized with "What doesn't kill you, makes you stronger". There is no training or conference that can teach you new things in such a short time than a real Customer.

Software craftsmanship

I enjoy both professions and try to get most out of them during the different stages of my short career so far. My drive is, regardless of your job title, be a software craftsman first. Being software craftsman is a state of mind, it is your attitude towards the everyday activities at work. Whatever you do, do it well and then be proud of it. Don't blindly follow any old habit or ritual that will make you write "WTF" code. Write well-crafted software that adds real value, and next time make it even better. To find out more about software craftsmanship, join a user group(like LSCC), read a book, and pass the craft on.

Master/Slave Failover for Camel Routes

One way to implement a master/slave failover pattern is to have a cluster of instances of an application where one instance (the master) is currently active and the other instances (the slaves) are on standby, ready to take over whenever the master fails. Some projects provide this kind of master/slave support out of the box:
Creating a failover deployment for Apache Karaf is straight forward: we start two or more Karaf instances and let them point to the same lock (file system or database). Then the first instance that starts gets the lock and becomes the master while the other instances will be waiting to get the lock before starting the bundles. In addition Karaf offers hot standby functionality where some bundles are started even in the slave instances and other bundles wait for to get the lock.

Apache ActiveMQ offers couple of ways for creating master/slave configurations but the simplest is to start two or more instances of ActiveMQ pointing to the same datasource(file or database) where the first broker gets the lock and becomes the master and the second and other brokers become slaves, waiting for the lock. Simple.

What about Camel? How can we have multiple routes (in one or separate containers) where one is the master (in running state) and the other routes are waiting to take over as soon as the master route stops ensuring high availability at route level? There are couple of components providing such a capability and all of the them rely on having some kind of centralized external system used as a lock.

1. Camel Quartz component has clustering support.
- If you are using quartz consumers, in clustered mode, you can have only one of the routes triggered at a time.
- Or if a quartz based CronScheduledRoutePolicy is used, in clustered mode, only one of the routes will be started/stopped.
Both of these options rely on having quartz to be configured with a datasource that is shared among all the routes in the cluster. This usage is not exactly master/slave but will have the same effect at the end.

2. Camel Zookeeper component offers a RoutePolicy that can start/stop routes in master/slave fashion. The first route that gets the lock will be started where the remaining routes will be waiting to get the lock. One advantage of this component is that it can be configured to have more than one master running.

3. Camel JGroups component also has master/slave capability using JGroupsFilters.

4. JBoss Fuse Master component is probably the easiest way to have master/slave setup in a Fuse environment. Internally it relies on Zookeeper's znode capability similarly to zookeeper component above.

5. This is not implemented yet but in theory it is possible to implement a RoutePolicy using ActiveMQ's exclusive consumers feature that provides a distributed lock. Do let me know if you implement this ;)

6. Use database as a lock. Christian Schneider has demonstrated how to have "Standby failover for Apache Camel routes" using a database here.

A Camel Demo for Amazon's Simple Worklfow Service

In a previous post I explained why AWS SWF service is good and announced the new Camel SWF component. Now the component documentation is ready and here is a simplistic fully working demo. It consist of three independent standalone Camel routes:
A workflow producer allows us to interact with a workflow. It can start a new workflow execution, query its state, send signals to a running workflow, or terminate and cancel it. In our demo, the WorkflowProducer starts a route that schedules 10 workflow executions where the each execution receives as an argument a number.
Once a workflow execution is scheduled, we need a process that will decide what are the next steps for it. In Camel it is done using a Workflow Consumer. A workflow consumer represents the workflow logic. When it is started, it will start polling workflow decision tasks and process them. In addition to processing decision tasks, a workflow consumer route, will also receive signals (send from a workflow producer) or state queries. The primary purpose of a workflow consumer is to schedule activity tasks for execution using activity producers. Actually activity tasks can be scheduled only from a thread started by a workflow consumer.
The logic in our demo decider is simple: if the incoming argument is greater than 5, we schedule a task for execution. Otherwise the workflow will complete as there are no other tasks to be executed. Notice that it also has branches for handing signal and state query events.

The final peace of our distributed (since it consists of three independent applications) workflow application is the ActivityConsumer that actually performs some calculations. It has the simplest possible implementation: increments the given argument and returns it.
All you need to do to run this demo is to create the appropriate workflow domain and add your key/secret to the route.

Building Distributed Workflow Applications on Amazon with Camel

Pipeline with SNS-SQS

A workflow consist of independent tasks performed in particular sequence determined by dynamic conditions. Very often a workflow represents a business process, for example the order processing steps in a ecommerce store.
Amazon Web Services offer various tools for building distributed and scalable workflow applications. One approach for building such an application is to use topics and queues for connecting the distinct steps in the workflow process. Then we can use publish/subscribe,  competing consumers and other mechanisms to scale our application and soon even the simplest application takes a shape similar to this:

Each step of the pipeline is connected to the next one with a queue and each step performs some actions and takes decision what is the next step. In addition using SNS/SQS involves some other low level tasks:
- Serialize/deserialize the data
- Ensure consistency (FIFO order) for SQSmessages
- Make sure message size is not exceeded
- Invent some kind of auditing support
- Subscriber queues to topics, assign permissions
- Manage DLQs
At the end it works, but overcoming these technical challenges takes as much time as writing the actual code that delivers the business value.

Simple Workflow Service

SWF on the other hand offers a higher level API for writing distributed, asynchronous workflow applications. It automatically serializes/deserializes data, manages application state, offers auditability, guarantees strong consistency, supports multiple versions. Most importantly, it ensures that the workflow orchestration and business  logic execution are separated. Any typical SWF application has the following building blocks:

In SWF terms, a workflow is the actual template that describes the distinct steps a process should follow. And a workflow execution is one run of this template.
Starter - the process that can start, stop and interact with a workflow execution.
Decider - the process that orchestrates and decides what is the next step of a workflow exection.
Worker - a process that executes a tasks from a specific type.
SWF Console - provides full visibility and control of the execution.
An example workflow execution can go through the following steps: a starter starts a workflow execution, SWF receives it, asks the decider what is the next step, then based on the decision passes the task to an appropriate activity worker. Once the result from the activity worker is received SWF asks the decider again for the next step, and depending on the response may execute another worker or not. This flow continues till the decider replies that the workflow is completed. You can see how the decider orchestrate each of the steps of the workflow and the activity workers perform the individual tasks. All that is managed by SWF and auditable at any stage.

Why use Camel?

The amazon provided Java clients work by using annotations to generate proxy classes to access SWF services. The whole process of generating and using proxy classes combined with the dependency from the starter to the decider, and from the decider to the activity workers is not very joyful. And what can be better than using a Camel route for orchestration and another route for the actual activity worker? The result is a Camel SWF component that is in Camel master now. Camel-swf component has two types of endpoints: workflow and activity.
A workflow producer allows us to start, terminate, cancel, signal, get state or retrieve the whole execution history of a workflow execution. In our diagram it represents the starter. Here is an example of how to start a workflow execution:
A workflow consumer is the decider. It receives decision tasks from SWF service and either schedules activity tasks for execution or indicates that the workflow execution has completed. It is a stateless deterministic route that only job is to orchestrate tasks:
The activity endpoints allow us to interact with the activity tasks. An activity producer is used to schedule activity tasks, and it can be used only from a decider route (actually decider thread). It is because only a decider can schedule activity tasks. The last box in our diagram that we have to provide implementation is the activity worker, which can be created using an activity consumer. This endpoint will receive activity tasks from SWF, execute them and return the results back to SWF. This is the bit that actually performs the business logic:
So any SWF application consist of a starter(workflow producer) that starts the execution, a decider (worfklow consumer) that receives decision tasks and schedules activity tasks (using activity producer) and the activity workers (activity consumer) that performs the tasks. And the communication between these endpoints is asynchronous, consistent and managed by SWF service.
It is not the easiest component to use, but it pays off with a simple and scalable architecture.
PS: Thanks to my ex-manager S. Wheeler for letting me contribute this component back to the Camel community.

More Apache Camel Books

It is hard to write impartial book review when you are the author of the first one, technical reviewer of the second one and really like the third book, but I'll try my best with this post.

Recently I blogged about my Instant Apache Camel Message Routing book. It is a short book intented for new comers to Apache Camel, showing quickly, how to create messaging applications in Camel using Enterprise Integration Patters. In this book I tried to distile and put the most important bits and peices about Camel and most commonly used integration patters. So there is no fluff, no repetion, only 50 pages of Camel integration crush course with examples and diagrams. If you want to get a high level view of Camel and EIPs, without diving into the details, that's the book to read.

The other new Apache Camel book that is expected by the end of this year is called Apache Camel Developer's Cookbook by Scott Cranton and Jakub Korab. It is a lookup guide full of recipes for everything you might want to do with Camel. It provides around 500 pages of best practice tips for using Apache Camel and lots of examples. I enjoyed reviewing it and even learned some new useful tips. If you prefer learing with examples, this cookbook is a must have.

And there is of course the famous Camel in Action by Claus Ibsen and Jonathan Anstey. It is a book which I started learning Camel couple of years ago and I still read some of the chapters from time to time. If you want to learn the philisofy behind Camel and find out how Camel internals work this is the book. A must read before putting #Camel on your CV.

I cannot stop myself mentioning the Bible of Integrations, the Enterprise Integration Patterns book by Gregor Hohpe. This is not a Camel book, but it lays down the foundation of Enterprise Integration Patterns which Camel implements. The book's website is a great pattern reference with use cases and diagrams. It is a huge book (with more than 700 pages), but a good to have as a reference for any team. You might be surpised to find out that there is pattern for everything.

In short: if you want to discover what is Apache Camel and how to use EIPs without going into the  internals go for Camel Message Routing book. If you want to see lots of example recipes and tips, go for Camel Cookbook. And if you want to deep dive into Camel go for Camel in Action.

PS: When I created this post there were three Camel books, but by the time I decided to publish it there were four already. A really productive year for the Camel community. The fourt book in my list is called Instant Apache Camel Messaging System by Evgeniy Sharapov. It is another short book, intented for beginners that emphsasys on test driven approach for writing Camel applications. I haven't read the book, but Claus Ibsen did and posted a review here.
Choose a book, read it, and hack something.

How to do FIFO messaging with Amazon SQS

If you have used Amazon Web Services, you probably know Simple Queue Service(SQS) - it is a reliable, highly scalable hosted queue for storing messages. One of the main drawbacks of SQS is that it does not guarantee first-in, first-out (FIFO) access to messages and that's clearly stated in the Amazon documentation:

"Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it."

Here is a quick example how Camel Resequencer pattern can help you overcome this drawback in 2-3 lines of code. To setup our example scenario, let's first create a route that will populate our queue with 100 messages each containing the message number: Then create a consumer route, that will read from the queue and log each message:
To prove that Amazon doesn't guarantee FIFO ordering we will write a test: Don't be misled by the short size of the test, it starts both routes and verifies that all 100 messages are received in the right order.

The above test fails in most of the runs and proves that SQS doesn't support FIFO order(when the messages are sent too quickly). The idea in this example is that our producer will index the messages or provide some kind of sequencing information, so that the message consumer can interpret it and order the messages. To do that we simply send a number as message body, but in a real world application that can be a field in a JSON or XML message. Then to make the test pass and ensure that the messages are received in the same order as they were sent, all we have to do is to add the Resequencer pattern in our consumer route: The streaming based Resequencer will let the messages go without any delay as long as they are in the right order. If the messages are not in the right sequence, it can hold up to 100 messages for 1 second while waiting for the missing message. Depending on your message load, you should adjust these numbers to hold enough messages while waiting for the missing one, but not hold for too long and reduce the throughput. Another option would be to try the non-streaming batch based Resequencer, which always collects a number of messages before sorting and releasing them.
If you are new to Apache Camel and Enterprise Integration Patterns (like the Resequencer), have a look at my recently published "Instant Apache Camel Message Routing" book where similar patterns and how to use them in Camel are explained in a short and focused manner.
Or if you want to deep dive into the integration world, I recommend you start from "Enterprise Integration Patterns" and "Camel in Action" books.

An old pet project based on Sencha Ext JS and Apache OFBiz

Play with the DEMO using username: scrum and password: scrum

Couple of years ago while working on software project we used Pivotal Tracker as our project management tool. It was a great free SAAS when it suddenly changed its terms and conditions and became a paid one. As a developer with great enthusiasm I said to myself "I know a great UI library (with not so great license) - Sencha Ext JS and great backend project with not so great UI - Apache OFBiz, why not combine them and create something better and still free". After couple of months I created LazyPlanner and realized that it is full with free project management tools out there, so it never went live. I wish I had read Eric Ries's The Lean Startup book earlier and had not started my idea by coding it first.

Any way, now I found this old project on my computer and put it on github. It is a standard component for OFBiz which works just by putting it in hot-deploy folder. For this demo installation I put some data and created a project with couple of task lists (called sprints) and few tasks. It supports multiple projects, with multiple tasks lists and tasks... It will be also running on the cloud for couple of days, so play with it and if you find it appealing get the code and use it on your own risk.

Instant Apache Camel and Enterprise Integration Patterns Book

I'm excited to announce that my book "Instant Apache Camel Message Routing" is published and ready for reading!
With new APIs and technologies emerging every day, the need for integrating applications is greater than ever before. With the right tools, integrating applications is not hard. Apache Camel is the leading open source integration and message orchestration framework with a variety of connectors and numerous well-known integration pattern implementations.
Instant Apache Camel Message Routing will help you to get started with Camel and Enterprise Integration Patterns in matter of hours. The book, is a short, focused and practical guide to Apache Camel that provides a high level overview of the Camel architecture and message routing principles. It introduces a number of integration patterns, complete with diagrams, common use cases, and examples about how to use them. It also explains how to test and monitor Camel applications and cope with failure scenarios.

The book contains the following chapters:
Creating a Camel project (Simple)
Routing messages to different destinations (Simple)
Using components (Simple)
Connecting routes (Simple)
Removing unwanted messages (Simple)
Transforming messages (Intermediate)
Splitting a message into many (Intermediate)
Aggregating multiple messages into one (Intermediate)
Reorganizing messages (Intermediate)
Multicasting messages (Intermediate)
Error handling and monitoring (Advanced)
Testing the messaging applications (Advanced)

In summary: Instant Apache Camel Message Routing is an easy to read and focused book that contains only the essence of Apache Camel and Enterprise Integration Patterns. It is ideal for developers who want to get started with Camel and message routing quickly.