Blogroll

How to do FIFO messaging with Amazon SQS

If you have used Amazon Web Services, you probably know Simple Queue Service(SQS) - it is a reliable, highly scalable hosted queue for storing messages. One of the main drawbacks of SQS is that it does not guarantee first-in, first-out (FIFO) access to messages and that's clearly stated in the Amazon documentation:

"Amazon SQS does not guarantee FIFO access to messages in Amazon SQS queues, mainly because of the distributed nature of the Amazon SQS. If you require specific message ordering, you should design your application to handle it."

Here is a quick example how Camel Resequencer pattern can help you overcome this drawback in 2-3 lines of code. To setup our example scenario, let's first create a route that will populate our queue with 100 messages each containing the message number: Then create a consumer route, that will read from the queue and log each message:
To prove that Amazon doesn't guarantee FIFO ordering we will write a test: Don't be misled by the short size of the test, it starts both routes and verifies that all 100 messages are received in the right order.

The above test fails in most of the runs and proves that SQS doesn't support FIFO order(when the messages are sent too quickly). The idea in this example is that our producer will index the messages or provide some kind of sequencing information, so that the message consumer can interpret it and order the messages. To do that we simply send a number as message body, but in a real world application that can be a field in a JSON or XML message. Then to make the test pass and ensure that the messages are received in the same order as they were sent, all we have to do is to add the Resequencer pattern in our consumer route: The streaming based Resequencer will let the messages go without any delay as long as they are in the right order. If the messages are not in the right sequence, it can hold up to 100 messages for 1 second while waiting for the missing message. Depending on your message load, you should adjust these numbers to hold enough messages while waiting for the missing one, but not hold for too long and reduce the throughput. Another option would be to try the non-streaming batch based Resequencer, which always collects a number of messages before sorting and releasing them.
If you are new to Apache Camel and Enterprise Integration Patterns (like the Resequencer), have a look at my recently published "Instant Apache Camel Message Routing" book where similar patterns and how to use them in Camel are explained in a short and focused manner.
Or if you want to deep dive into the integration world, I recommend you start from "Enterprise Integration Patterns" and "Camel in Action" books.

An old pet project based on Sencha Ext JS and Apache OFBiz

Play with the DEMO using username: scrum and password: scrum

Couple of years ago while working on software project we used Pivotal Tracker as our project management tool. It was a great free SAAS when it suddenly changed its terms and conditions and became a paid one. As a developer with great enthusiasm I said to myself "I know a great UI library (with not so great license) - Sencha Ext JS and great backend project with not so great UI - Apache OFBiz, why not combine them and create something better and still free". After couple of months I created LazyPlanner and realized that it is full with free project management tools out there, so it never went live. I wish I had read Eric Ries's The Lean Startup book earlier and had not started my idea by coding it first.
Any way, now I found this old project on my computer and put it on github. It is a standard component for OFBiz which works just by putting it in hot-deploy folder. For this demo installation I put some data and created a project with couple of task lists (called sprints) and few tasks. It supports multiple projects, with multiple tasks lists and tasks... It will be also running on the cloud for couple of days, so play with it and if you find it appealing get the code and use it on your own risk.

Instant Apache Camel and Enterprise Integration Patterns Book

I'm excited to announce that my book "Instant Apache Camel Message Routing" is published and ready for reading!
With new APIs and technologies emerging every day, the need for integrating applications is greater than ever before. With the right tools, integrating applications is not hard. Apache Camel is the leading open source integration and message orchestration framework with a variety of connectors and numerous well-known integration pattern implementations.
Instant Apache Camel Message Routing will help you to get started with Camel and Enterprise Integration Patterns in matter of hours. The book, is a short, focused and practical guide to Apache Camel that provides a high level overview of the Camel architecture and message routing principles. It introduces a number of integration patterns, complete with diagrams, common use cases, and examples about how to use them. It also explains how to test and monitor Camel applications and cope with failure scenarios.
The book contains the following chapters:
Creating a Camel project (Simple)
Routing messages to different destinations (Simple)
Using components (Simple)
Connecting routes (Simple)
Removing unwanted messages (Simple)
Transforming messages (Intermediate)
Splitting a message into many (Intermediate)
Aggregating multiple messages into one (Intermediate)
Reorganizing messages (Intermediate)
Multicasting messages (Intermediate)
Error handling and monitoring (Advanced)
Testing the messaging applications (Advanced)


In summary: Instant Apache Camel Message Routing is an easy to read and focused book that contains only the essence of Apache Camel and Enterprise Integration Patterns. It is ideal for developers who want to get started with Camel and message routing quickly.

About Me