From Fragile to Antifragile Software

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)

One of my favourite books is Antifragile by Nassim Taleb where the author talks about things that gain from disorder. Nacim introduces the concept of antifragility which is similar to hormesis in biology or creative destruction (accessible version creative-destruction) in economics and analyses it characteristics in great details. If you find this topic interesting, there are also other authors who have examined the same phenomenon in different industries such as Gary Hamel, C. S. Holling, Jan Husdal. The concept of antifragile is the opposite of the fragile. A fragile thing such as a package of wine glasses is easily broken when dropped but an antifragile object would benefit from such stress. So rather than marking such a box with "Handle with Care", it would be labelled "Please Mishandle" and the wine would get better with each drop (would be awesome woulnd't it).

It didn't take long for the concept of antifragility to be used also for describing some of the software development principles and architectural styles. Some would say that SOLID prinsiples are antifragile, some would say that microservices are antifragile, and some would say software systems cannot be antifragile ever. This article is my take on the subject.

According to Taleb, fragility, robustness, resilience and antifragility are all very different. Fragility involves loss and penalisation from disorder. Robustness is enduring to stress with no harm nor gain. Resilience involves adapting to stress and staying the same. And antifragility involves gain and benefit from disorder. If we try to relate these concepts and their characteristics to software systems, one way to define them would be as the following.

Disorder 

Different systems are affected by different kind of disorder, such as stress, time, change, volatility, debt, etc. For software systems, the main disorder is the change. The business is in a constant changing environment and the software needs to adapt to the business needs quickly. That is implementing new requirements, changes to existing functionality, even creating new business opportunities through innovation. A software system has to change all the time, otherwise it is obsolete.
Apart from development time challenges, there are also runtime challenges for software systems too. Software systems are created and exists to add value by running in a production environment. And while doing so, they are under stress by end users and other systems. This is another kind of disorder, that software systems have to deal with.

Fragile 

This property describes systems that suffer when put under stress. Imagine a software project that is not easy to change at development time. For example if it is not easy to extend, modify and deploy to the production environment. Or a system that is not able to handle unexpected user inputs or external system failures and breaks easily. That's is a fragile system that is harmed by stress and penalised by change, a good example of fragile.

Robust

This is a system that can continue functioning in the presence of internal and external challenges without adaptation. Every system is robust up to a level. For example a bottle is robust until it reaches the level of breaking point. A software system can be made robust to handle unanticipated user input, or failures in external systems. For example handling NPE in Java, using try-catch-finally statements to handle unreliable invocations, having a thread pool to handle concurrent users, creating network connections using timeouts, are all examples of robustness for a software system. But a robust system doesn't adapt to a chaining environment and when the stress and change threshold is reached it would break. The qualities that define the robustness vary from system to system. An ATM for example needs to be robust and not fail in the middle of a transaction, whereas, a media streaming service can drop a frame or two, as long as it continues streaming under stress.

Resilient

A system is resilient when it can adapt to internal and external challenges by changing its method of operation. The key here is that the system is responding to stress by changing its internal behaviour rather than resisting stress with a predefined buffer.

• A typical example here is the circuit breaker pattern which changes its internal state to adapt to the external system behaviour to protect itself. 
• Another example would be using a retry mechanism with some backoff algorithm to handle transient failures in external systems.
• A different technique for creating resilient systems is through graceful degradation, both on the UI and the server side. Rendering UI based on the user agent capabilities, or failing fast on the server side and fallback to some default values are commonly used techniques to adapt to failures.
• Systems with self healing and autorepair capabilities are another example of resiliency. These systems are self-aware and can detect abnormalities and take corrective actions. For example Kubernetes/OpenShift will perform regular liveness checks for the running Docker containers and if they detect any anomalies they will restart the container and perform necessary backoffs until the application stabilises. This is another mechanism to cope with stress and improve application resilience.

Overview

Before looking at the next level of software evolution - the antifragile systems, let's visualise and summarise different kind of software system characteristics.

• A fragile system is difficult to modify, and cannot cope with a changing environment. Even if it provides some value when used in a stable non changing environment, when faced with further stress and change, it quickly turns into a liability. Many organisations have applications (mainframes for example) which are impossible to change, very expensive to maintain, but still running on high cost as they are very critical to the business.
• A robust system is the one that is implemented with certain buffers to handle change and stress. So when the stress level increases, it can withstand it for up to a level without losing its capabilities and still provide a good value. But a robust system does not adapt, and if the stress and change levels continue raising, such a system can also stops providing benefit and may turn into liability and run on loss.
• A resilient system can handle more stress and change as it is designed and implemented with stress in mind and adaptability features. Even if it is not benefiting from stress, it can survive lot's of different kind of stress and change and provide value up to a greater degree.
• An antifragile system is created with change in mind and it feeds from stress and change. It is much harder to create such a system (it is not a software system but a social-technical system) but once it is in place, it drives the business based on change, and even creates the change.

Antifragile

Many things in life are antifragile, such as the human body. When stressed at the right level, a muscle or bone wold come back stronger. But can a software system be antifragile? Certainly there are some tools, platforms, architectural styles, methodologies that can help create software with antifragile characteristics. Let's see some of the more popular ones.

Auto scaling feature allows applications to handle increasing load by creating more instances of the application. To achieve that, the software system have to able to measure and then react to change and stress. Some good examples here are AWS autoscaling of EC2 instances at infrastructure level, and OpenShift autoscaling of application containers for the application level. This is a feature that transitions applications from resiliency to antifragility since the software system is shifting resources from one part of the system into another to respond to stress.

Microservices. According to Taleb, at times of stress, the large is doomed to breaking. And that phenomen has been observed with mammals, corporations, administrations, etc. In software and large projects, this behaviour has been observed even more often. The bigger a software project is, the harder it becomes to change and react to stress. Microservices is an architecture styles that allows easier change by having autonomous services with well defined APIs - features that allow change. Russ Miles is a strong believer and proponent of Antifragile Software through Microservices (here is an intro video from him).

Chaos engineering is a technique to create antifragility by evolving systems to survive chaos. Rather than waiting for things to break at the worst possible time, the idea of chaos engineering is to proactively inject failures in order to be prepared when disaster strikes. Netflix’s Simian Army is a very good materialization of this technique, designed to generate failures and help isolate system’s weaknesses.

Continuous deployments to a production environment creates continuous partial system failures and forces organisations to react better to failures through redundancy, rolling upgrades, rollbacks, and avoiding single points of failure. Other techniques such as canary release, blue-green deployments, are used to reduce the risk of introducing new software into production environment. Some other methods such as A/B testing even allows experimenting with change and measuring its effect, in order to gain from change.

The human element

Antifragility is not an universal characteristics. Different systems are antigragile towards different kind of disorder. For example Chaos Monkey will make your system antifragile towards EC2 deaths, and autoscaller will make your system respond to specific type of load. But your systems will not be antifragile towards other kinds of stress. And if you look above at the different ways of introdusing antifragility into software systems, all of them are means for making the social-technical system antifragile (and not only the software system):

• Chaos engineering forces human feedback to the injected randomness and makes the system antifragile.
• Microservices alone do not make a software system antifragile. But microservices combined with appropriate organisational and team structure enables antifragility. If microservices are a way for architecting applications into autonomous structures, DevOps is a means for organizing teams into similar structures. You need both in order to benefit from them and gain from disorder.
• Continuous deployment pipeline is a tool that allows teams to react to stress faster, introduce or retract change faster and generally use the stress as the driver.
• Similarly, iterative development is not enough to benefit from changing environment. But iterative development with open and honest retrospective rituals is.

Innovate

If it takes few weeks to create a new developer environment, you can not react to change. If it takes three months to release a new feature, you can not react to change. If your ops team is watching the metrics dashboard and manually scaling applications up and down, you cannot embrace change. If the team is hardly catching up with the change, there is no way to gain from change. But once you put an appropriate organisational structure, the right tools and culture in place, then you can start gaining from change. Then you can afford having Friday Hackathons, then you can start exploring open source projects and start contributing to them, then you can start open sourcing your internal projects and benefit from a community, and generally be the change itself. And why not the Netflix or the Amazon of tomorrow.

Visualizing Integration Applications

(This post was originally published on Red Hat Middleware Blog. To read the original post, click here.)
Since I've changed role and started performing architect duties, I have to draw more boxes and arrows than write code. There are ways to fight that, like contributing to open source projects during sleepless nights, POCs, demos, but drawing boxes to express architectures and designs is still big part of it. This post is about visualising distributed messaging/SOA/microservices applications in agile (this term has lost its meaning, but cannot find a better one in this case) environments. What I like about the software industry in recent years is that the majority of organisations I've worked with, value the principles behind lean and agile software development methodologies. As long as it is practical, everyone strives to deliver working software (rather than documentation), deliver fast (rather than plan for a long time), eliminate waste, respond to change, etc. And there are management practises such as Scrum and Kanban, and Technical Practises from Extreme programming (XP) methodology such as unit testing, pair programing, and other practises such as CI, CD, DevOps to help implement the aforementioned principles. In this line of thinking, I decided to put together a summary of the design tools and diagrams I find useful in my day to day job while working with distributed Systems.

Issues with 4+1 View Model and death by UML

Every project kicks off with big ambitions, but there is never enough time to do things perfectly, and at the end we have to deliver whatever works. And that is a good thing, it is the way the environment helps us avoid gold plating, YAGNI, KISS, etc. so we do just enough and adapt to chance.

Looking back, I can say that most of the diagrams I've seen around are inspired by 4+1 view model of Philippe Kruchten which has Logical, Development, Process and Physical views.

4+1 View Model

I quite like the ideas and the motivation behind this framework: using separate views and perspectives to address specific set of constraints and targeting the different stakeholders. That is a great way of describing complex software architectures. But I have two issues with using this model for integration applications.

Diagram applicability

Typically these views are expressed through UML, and for each view, you have to use one or more UML diagrams. The fact that I have to use 15 types of UML diagrams to communicate and express a system architecture in an accessible way, defeats its purpose.

Death by UML

With such a complexity, the chances are that there are only one or two people in the whole organisation who has the tools to create, ability to understand and maintain these diagrams. And having hard to interpret, out of date diagrams is as useful as having out of date gibberish documentation. These diagrams are too complex and with limited value, and very quickly they turns into liability that you have to maintain rather than asset expressing the state of a constantly changing system.
Another big drawback is that the existing UML diagram types are primarily focused on describing object-oriented architectures rather than Pipes and Filters architectures. The essence of messaging applications is around interaction styles, routing, data flow rather than structure. Class, object, component, package, and other diagrams are of less value for describing a Pipes and Filters based processing flows. Behavioural UML diagrams such as activity and sequence get closer, but still cannot express easily concepts such filtering and content based routing which are fundamental part of integration applications.

View applicability 

Having different set of views for a system, to address different concerns is a great way of expressing intend. But the existing views of 4+1 model doesn't reflect the way we develop and deploy software nowadays. The idea that you have a logical view first, which then leads to development and process view, and those lead to physical view is not always the case. The systems development life cycle, is not following the (waterfall) sequence of requirement gathering, designing, implementing and maintaining.

Software Development Lifecycle

Instead other development methodologies such as agile, prototyping, synchronise and stabilise, spike and stabilise are used too. In addition to the process, the stakeholders are changing too. With practises such as DevOps, developers have to know about the final physical deployment model, operations have to know about the application processing flows too. Modern architectures such as microservices affect the views too. Knowing one microservice is in a plethora of a services is not very useful. Knowing too much about all the services is not practical either. Having the right abstraction level to have a system wide view with just enough details becomes vital.

Practical Visualisation for Integration Applications

There are some good tips for drawing in general. The closest thing that has been working for me is described by Simon Brown as C4 model. (You should also get a free copy of Simon's awesome The Art of Visualising Software Architecture book). In his model, Simon is talking about the importance of a common set of abstractions rather than common notation (such as UML) and then using simple set of diagrams for different level of abstractions: system context diagram, container diagram, component diagram and class diagram. I quite like this "Outside-In" approach, where you first have 10000 foot view and with each next level, going deeper with more detailed views.
C4 is also not an exact match for middleware/integration applications either, but it is getting closer. If we were to use C4 model, then system context diagram would be one box that says ESB (or middleware, MOM, microservices) with tens of arrows from north to south. Not very useful. Container diagram is quite close, but the term container is so overloaded (VM, application container, docker container) which makes it less useful for communication. Component and class diagrams are also not a good fit as Pipes and Filter architectures are focused around Enterprise Integration Patterns, rather than classes and packages.
So at the end, what is it that worked for me? It is the following 3 types of diagrams which abbreviate as SSD (not as cool as C4):  System Context Diagram, Service Design Diagram and Deployment Diagram.

System Context Diagram

The aim of this model is to show all the services (whether they are SOA, Microservices) with their inputs and outputs. Ideally having the external systems on the north, the services in the middle section, and internal services in the south. Or you could use both external and internal services on both side of the middleware layer as shown below. Also having the protocol (such as HTTP, JMS, file) on the arrows, with the data format (XML, JSON, CSV) gives useful context too, but it is not mandatory. If there are too many services, you can leave the protocol and the data format for the service level diagrams. I use the direction of the arrow to indicate which service is initiating the call rather than the data flow direction.

System Context Diagram

Having such a diagram gives a good overview of the scope of a distributed system. We can see all the services, the internal and external dependencies, the types of interaction (with protocol and data format), and the call initiator.

Service Design Diagram

The aim of this diagram is to show what is going on in each box representing a middleware service from the System Context Diagram. And the best diagram for this is to use EIP icons and connect those as message flows. A service may have a number of flows, support a number of protocols, implement real time and/or batch behaviour.

Service Design Diagram

At this level, we want to show all possible data flows implemented by a specific service, from any source to any destination.

Deployment Diagram

The previous two diagrams are the logical views of the system as a whole and each service separately. With the deployment diagram, we want to show where each service is going to be deployed. May be there will be multiple instances of the same service running on multiple hosts. May be some services will be active on one host, and passive on the other. May be there will be a load balancer fronting the services, etc.

Deployment Diagram

The deployment diagram is supposed to show how individual services and the system as a whole relates to the host systems (regardless whether that is physical or virtual).

What tools do I use?

The System Context and the Deployment Diagrams are composed only of boxes and arrows and do not require any specail tools. For the Service Design Diagram, you will need a tool that has the Enterprise Integration Pattern icons installed. So far, I have seen the following tools with EIP icon support:

• Mac OS: OmniGraffle with icons from graffletopia.This is what I've used to create all the diagrams for Camel Design Patterns book.
• Windows: Enterprise Architect by Sparx Systems with EIP icons by Harald Westphal. There are also MS Visio Stencils here.
• Web: LucidCharts which ships EIP icons by default. That is the easiest to use tool and accessible from everywhere. It is my favourite tool (with MS Visio import/export options for Windows users), and has free account plans to start with.
• Web: DrawIO another web tool with EIP icons. The beauty of this tool is that it forces you to use your own storage options for the diagrams, such as: google drive, dropbox, locally, etc. So you own the diagrams and keep them safe.
• Canva: not specifically for EIP diagrams, but in general this site has a nice templates and photos to use in various presentations. Check it out.

Other development tools that could be also used for creating EIP diagrams are:

• Red Hat JBoss Developer Studio
• EIP Designer by Sirius
• Talend Open Studio for ESB
• Hawtio - the Camel plugin can visualize running Camel routes as EIP diagrams.
  

System Context Diagram is useful to show the system wide scope and reach of the services, Service Design Diagram is good for describing what a service does, and Deployment Diagram is useful mapping all that into something physical. In IT, we can expand work and fill up all the available time with things to do. I'm sure given more time, we can invent 10 more useful views. But w/o the above three, I cannot imagine myself describing an integration application. As Antoine de Saint-Exupery put it long ago: "Perfection is finally attained not when there is no longer anything to add but when there is no longer anything to take away."

Create Resilient Camel applications with Hystrix DSL

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)
 
Apache Camel is a mature integration library (over 9 years old now) that implements all the patterns from Enterprise Integration Patterns book. But Camel is not only an EIP implementation library, it is a modern framework that constantly evolves, adds new patterns and adapts to the changes in the industry. Apart from tens of connectors added in each release, Camel also goes hand-in-hand with the new features provided by the new versions of Java language itself and other Java frameworks. With time some architectural styles such as SOA and ESB lose attraction and new architectural styles such as REST, Microservices get popular. To enable developers do integrations using these new trends, Camel responds by adding new DSLs such the REST DSL and new patterns such as the Circuit Breaker, and components such as Spring Boot. And that's not all and we are nowhere near done. With technologies such as Docker containers and Kubernetes, the IT industry is moving forward even faster now, and  Camel is evolving too in order to ease the developers as it always has been. To get an idea of kinda tools you would need to develop and run applications on Docker and Kubernetes, check out the Fabric8 project and specifically tools such as the Docker Maven plugin, Kubernetes CDI extension, Kubernetes Java client, Arquilian tests for Kubernetes, etc. Exciting times ahead with lot's of cool technologies, so let's have a look at one of those: Camel and the Hystrix based circuit breaker.

Two Circuit Breakers in Camel, which one to chose?

Two years ago, when I first read Release It from Michael Nygard, I couldn't stop myself implementing the Circuit Breaker pattern in Camel. Usually I drive my contributions by real customer needs, but Circuit Breaker pattern is so elegant, I had to do it. To implement it in a non-intrusive manner, I have added it as a Camel Load Balancer strategy. Here how simple it is:
The DSL above is self describing: if the number of MyCustomExceptions thrown by mock:result endpoint reaches the threshold number, the CircuitBreaker goes to open state and starts rejecting all requests. After 1000ms it moves to halfOpenAfter state and the result of the first request in this state will define its next state as closed or open. It is the simplest possible implementation of the CircuitBreker you can imagine, but still useful.

Since then, the Microseservices architecture has became more popular, and so is the Circuit Breaker Pattern and its java implementation Hystrix. At some point Raúl Kripalani started the Hystrix implementation in Camel and put all the ground work in place, but with time it lost momentum. Then seeing the same request again and again from different customers, I took the relay and continued the work and pushed to Camel a Hystrix component. Seeing the feedback from the community, it still didn't feel as elegant as it could be. Then Claus stepped in and made Hystrix part of the Camel DSL by turning it into an EIP (rather than component). So how does it look like to create a Hystrix based Circuit Breaker in Camel now?

In the example above, you can see only very few of the available options for a Circuit Breaker, to see all of them checkout the offical documents and try out the example application Claus put in place.

Based on this example you may think that Hystrix is part of Camel core, but it is not. Camel core is still this light and without dependencies to third party libraries. If you want to use the Hystrix based Circuit Breaker, you need to add camel-hystrix dependency to your dependencies as it is with any other non-core component and make it available at runtime.

Fail Fast, Fallback, Bulkhead, Timeout and more

The Hystrix library implements more than Circuit Breaker patter. It also does bulkheading, request caching, timeouts, request collapsing, etc. The implementation in Camel does not support request collapsing and caching as these are done using other patterns and components available in Camel already. Request collapsing in Camel can be done using Aggregator EIP and caching can be done using cache components such as Redis, Inifinspan, Hazelcast, etc. The Hystrix DSL in Camel offers around 30 configuration options supported by Hystrix, also exposes metrics over JMX and/or REST for the Hystrix Dashboard.

As a final note, don't foget that to create a true resilient application, you need more than Hystrix. Hystrix will do bulkheading at thread pool level, but that is not enough if you don't apply the same principle at process, host and phisical machine level. To create a create a resilient distributed system, you will need to use also Retry, Throttling, Timeout... and other good bractices some of which I have described in Camel Design Patterns book.

To get some hands on feeling of the new pattern, check the example and then start defending your Camel based Microservices with Hystrix.

Scalable Microservices through Messaging

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)

Microservices are everywhere nowadays, and so is the idea of using service choreography (instead of service orchestration) for microservices interactions. In this article I describe how to set up service choreography using ActiveMQ virtual topics, which also enables scalable event based service interactions.

Service Interaction Styles

There are two main types of service interaction: synchronous and asynchronous.
With synchronous interactions, the service consumer makes a request and blocks until the operation completes and a response is received. The HTTP protocol is a great example for a synchronous interaction. This type of interaction is usually associated with request/response interaction style and the HTTP protocol. (Of course, it also possible to do request/response with asynchronous requests or event messaging, via registering a callback for the result, but that is a less common use case).
With an asynchronous interaction style, the service consumer makes a request, but doesn’t wait for the operation to complete. As soon as the request is acknowledged as received, the service consumer moves on. This type of interaction allows publish/subscribe style of service communication — e.g. instead of a service consumer invoking an operation from another service, a producer raises an event and expects interested consumers to react.
Apart from these technical considerations, there is also another aspect to consider with service interactions: coupling and responsibility.
If service A has to interact with service B, is it the responsibility of service A to invoke service B (orchestration) or is it the responsibility of service B to subscribe for the right events (choreography)?
With service orchestration, there is a central entity (as the service A itself in our case), which has the knowledge of other services to be called. With the choreography approach, this responsibility is delegated to the individual services and they are responsible for subscribing for the “interesting” events.
To read more about this topic, checkout Chapter 4 from the excellent Building Microservices book. For the rest of this article, we will focus on doing service choreography using messaging.

Service Orchestration Through Messaging

Service orchestration in messaging is achieved through queues. A queue implements load balancer semantics using the competing consumer pattern and makes sure that there is only one consumer of a message.
Let’s say there is a “Customer Service” that has to interact with “Email Service”. The easiest way to implement this is to use a queue and let “Customer Service” send a message to “Email Queue”. If the “Customer Service” has to interact with “Loyalty Point Service”, again, “Customer Service” has to send another message — this time to “Loyalty Point Queue”. With this approach, it is the responsibility of “Customer Service” to know about “Loyalty Point Service” and “Email Service”, and subsequently send the right messages to the corresponding queues. In short, the whole interaction is orchestrated by “Customer Service”.
One advantage of using queues is that, it allows scaling of consumers easily. We could start multiple instances of “Loyalty Point Service” and “Email Service”, and the queues will do the load balancing among the consumers.

 

Service Choreography Through Messaging

With the service choreography approach, “Customer Service” doesn’t have any knowledge of “Loyalty Point Service” or “Email Service”. “Customer Service” simply emits an event to “Customer Topic”, and it is the responsibility of “Loyalty Point Service” and “Email Service” to know about the Customer event contract and subscribe to the right topic — the publish/subscribe semantics of the topics will ensure that that every event is distribute to both subscribers.

 

Scaling Service Choreography

Since topics implement publish/subscribe semantics rather than competing consumers, scaling the consumers becomes harder. If we (horizontally) scale “Loyalty Point Service” and start two instances, both instances of the service will receive the same event and there won’t be any benefit in scaling (unless the services are idempotent).

ActiveMQ Virtual Topics to the Rescue

So what we need is some kind of mixture between topic and queue semantics. We want the “Customer Service” to publish an event using publish/subscribe semantics so that all services receive the event, but we also want competing consumers, so that individual service instances can load balance events and scale.

There are a number of ways we could achieve that with Camel and ActiveMQ:

• The very obvious one that comes to (my) mind is to have a simple Camel route that is consuming events from “Customer Topic” and sends them to both “Loyalty Point Queue” and “Email Queue”. This is easy to implement, but every time there is a new service interested from the “Customer Service” events, we have to update the Camel routes. Also, if you run the Camel route on a separate process than the broker, there will be unnecessary networking overhead only to move messages from a topic to a set queues in the same message broker.

• An improvement to the above approach would be, to have Camel routes running in the ActiveMQ broker process using ActiveMQ Camel plugin. In that case, you still have to update the Camel route every time there is change to the subscribers, but the routing will happen in the broker process itself, so no networking overhead.

• And even a better solution would be to have the queues subscribed to the topic w/o any coding, but using a declarative approach using ActiveMQ virtual topics (hence the whole reason for writing this article).

Virtual topics are a declarative way of subscribing queues to a topic, by simply following a naming convention — all you have to do is define or use the default naming convention for your topic and queues.
For example, if we create a topic with a name matching VirtualTopic.> expression, such as: VirtualTopic.CustomerTopic, then have the “Loyalty Point Service” consume from Consumer.LoyaltyPoint.VirtualTopic.CustomerTopic queue, the message broker will forward every event from VirtualTopic.CustomerTopic topic to Consumer.LoyaltyPoint.VirtualTopic.CustomerTopic queue.
Then we could scale Loyalty Point Service by starting multiple service instances, all of which consume from Consumer.LoyaltyPoint.VirtualTopic.CustomerTopic queue.
Similarly, later we can create a queue for the Email Service by following the same naming convention:
Consumer.Email.VirtualTopic.CustomerTopic. This feature allows us to simply name our topics and queues in a specific way,  and have them subscribed without any coding.

Camel Desing Patterns

Final thoughts

This is only one of the many patterns I have described in my recently published Camel Design Patterns book. Camel is quite often used with ActiveMQ, and as such, you can find also some ActiveMQ patterns in the book too.
Another way to scale microservices using choreography can be achieved through event sourcing. You can find a nice blog post describing it here.

Performance Tuning Ideas for Apache Camel

Every now and then, I get questions around optimising Camel applications with the argument that Camel is slow. Camel is just the glue connecting disparate systems, the routing engine is all in-memory, and it doesn’t require any persistent state. So 99% of the cases, performance issues are due to bottlenecks in other systems, or having the application design done without performance considerations. If that is the case, there isn’t much you can achieve by tuning Camel further, and you have to go back to the drawing board.

But sometimes, it might be worth squeezing few more milliseconds from your Camel routes. Tuning every application is very specific and dependent on the technology and the use case. Here are some ideas for tuning Camel based systems, which may apply for you (or not).

Endpoint Tuning

Endpoints in Camel are the integration points with other systems and the way they are configured will have a huge impact on the performance of the system. Understanding how different endpoints work and tuning those should be the one of the first places to start with. Here are few examples:

• 
Messaging - If your Camel application is using messaging, the overall performance will be heavily dependent on the performance of the messaging system. There are too many factors to consider here, but main ones are:

• Message broker - the network and disk speed, combined with the broker topology will shape the broker performance. To give you an idea, with ActiveMQ, a relational database based persistent store will perform around 50% of a file based store, and using network of brokers to scale horizontally will cost another 30% of performance. It is amazing how one configuration change in ActiveMQ can have huge impact on the messaging system and then the Camel application . There is a must read ActiveMQ tuning guide by Red Hat with lot's of details to consider and evaluate. Also a real life example from Chrisitan Posta showing how to speed up the broker 25x times in certain cases. Another recent article by Simon Green shows how to approach an ActiveMQ Tunning Adventure step by step.

• Message client - if performance is a priority, there are also some hacks you can do on the ActiveMQ client side, such as: increasing TCP socketBufferSize and ioBufferSize, tuning the OpenWire protocol parameters, using message compression, batch acknowledgements with optimizeAcknowledge, asynchronous send with useAsyncSend, adjusting pre-fetch limit, etc. There are some nice slides again from Christina here and old but still very relevant video from Rob Davies about tuning ActiveMQ. All of these resources should give you enough ideas to experiment and improve the performance from messaging point of view.

• 
Database writes - use batching whenever possible. You can use an aggregator to collect a number of entries before performing a batch operation to interact with the database (for example with the SQL component.
  
• Working with templates
 - if you have to use a template component as part of the routing, try out the existing templating engines (FreeMarker, Velocity, SpringTeplate, Mustache, Chunk )  with a small test as the following one and measure which one performs better. There is a great presentation titled Performance optimisation for Camel by Christian Mueller with the source code supporting the findings (UPDATE: after this blog post was published, Christian created new slides with latest version of Camel 2.16.2 and Java 7/8, check out those too). From those measurements we can see that FreeMarker performs better than Velocity and SprintTemplates in general.
• Using Web Services - whenever you have to use a web endpoint, the web container itself (has to be tuned separately. From Camel endpoint point of view, you can further optimise a little bit by skipping the unmarshalling if you don't need Java objects, and using asynchronous processing.
• concurrentConsumers - there are a number of components (Seda, VM, JMS, RabbitMQ, Disruptor, AWS-SQS, etc) that support parallel consumption. Before using an endpoint, check the component documentation for thread pool or batch processing capabilities. To give you an idea, see how Amzon SQS processing can be improved through these options.

Data Type Choice

The type and the format of the data the is passing through Camel routes will also have performance implications. To demonstrate that let's see few examples.

• Content based router, splitter, filter are examples of EIPs that perform some work based on the message content. And the type of the message affects the processing speed of these elements. Below is a chart from Christian Mueller's presentation, visualising how Content Based Router is performing with different kinds of messages:

  

  Content Based Routing based on different data types

For example, if you have a large XML document in the Exchange, and based on it you perform content based routing, filtering, etc., that will affect the speed of the route. Instead you can extract some key information from the document and populate the Exchange headers for faster access and routing later.

• Marshaling/Unmarshaling - similarly to the templating engines, different 
data format covenrtors perform differently. To see some metrics check again Christian's presentation, but also keep in mind that performance of the supported data formats may vary between different versions and platforms so measure it for your use case.
• Streaming  - Camel streaming and stream caching are one of the underrated features that can be useful for dealing with large messages.
• Claim check EIP - if the application logic allows it, consider using claim check pattern to improve performance and reduce resource consumption.

Multithreading

Camel offers multithreading support in a number of places. Using those can improve the application performance too.

• Paralel processing EIPs - the following Camel EIP implementations support parallel processing - multicast, recipient list, splitter, delayer, wiretap, throttler, error handler. If you are going to enable parallel processing for those, it would be even better if you also provide a custom thread pool specifically tuned for your use case rather than relying on Camel's default thread pool profile.
• Threads DSL
 construct - some Camel endpoints (such as the File consumer) are single threaded by design and cannot be parallelized at endpoint level. In case of File consumer, a single thread picks a file at a time and processes it through the route until it reaches the end of the route and then the consumer thread picks the next file. This is when Camel Threads construct can be useful. As visualised below, File consumer thread can pick a file and pass it to a thread from the Threads construct for further processing. Then the File consumer can pick another file without waiting for the previous Exchange to complete processing fully.

  

  Parallel File Consuming

• Seda component - Seda is another way to achieve parallelism in Camel. The Seda component has in-memory list to accumulate incoming messages from the producer and concurrentConsumers to process those incoming request in parallel by multiple threads.
  
• Asynchronous Redelivery/Retry - if you are using an error handler with a redelivery policy as part of the routing process, you can configure it to be asynchronous and do the redeliveries in a separate thread. That will use a separate thread pool for the redelivery not block the main request processing thread while waiting. If you need long delayed redeliveries, it might be a better approach to use ActiveMQ broker redelivery (that is different from consumer redelivery BTW) where the redeliveries will be persisted on the message broker and not kept in Camel application memory. Another benefit of this mechanism is that the redeliveries will survive application restart and also play nicely when the application is clustered. I have described different retry patterns in Camel Design Patterns book.
  

Other Optimisations

There are few other tricks you can do to micro-tune Camel further. 

• Logging configurations - hopefully you don't have to log every message and its content on the production environment. But if you have to, consider using some asynchronous logger. On a high throughput system, ane option would be to log statistics and aggregated metrics through Camel Throughput logger. Throughput logger allows logging aggregated statistics on fixed intervals or based on the number of processed messages rather than  per message bases. Another option would be to use the not so popular Camel Sampler EIP and log only sample messages every now and then.
• Disable JMX - by default, Camel JMX instrumentation is enabled which creates a lot of MBeans. This allows monitoring and management of Camel runtime, but also has some performance hit and requires more resources. I still remember the time when I had to fully turn off JMX in Camel in order to run it with 512MB heap on a free AWS account. As a minimum, consider whether you need any JMX enabled at all, and if so whether to use RoutesOnly, Default or Extended JMX profiles. 

• Message Histroy - Camel implements the Message History EIP and runs it by default. While on development environmnet, it might be useful to see every endpoint a message has been too, but on the produciton environment you might consider to disable this feature.
• Original message - Every Camel route will make a copy of the original incoming message before any modifications to it. This pristine copy of the message is kept in case it is needed to be redelivered during error handling or with onCompletion construct. If you are not using these features, you can disable creating and storing the original state of every incoming message.
• Other customisations -  Almost every feature in CamelContext can be customized. For example you can use lazyLoadTypeConverters for a faster application startup, or configure the shutdownStrategy for a quicker shutdown when there are inflight messages, or a use a custom UuidGenerator that performs faster, etc.

Application Design

All of the previous tunings are micro optimizations compared to the application design and architecture. If your application is not designed for horizontal scalability and performance, sooner or later the small tuning hacks will hit their limit. The chances are, what you are doing has been done previously, and instead of reinventing the wheel or coming up with some clever designs, learn from the experience of others and use well known patterns, principles and practises. Use principles from SOA, Microservices architectures, resiliency principles, messaging best practises, etc. Some of those patterns such as Parallel Pipelines, CQRS, Load Leveling, Circuit Breaker are covered in Camel Design Patterns book and do help to improve the overall application design.

JVM

There are many articles about tuning the JVM. Here I only want to mention the JVM configuration generation application by Red Hat which can generate for you JVM configurations based on the latest industry best practices. You can use it as long as you have a Red Hat account (which is free for developers anyway). Using the latest JVM and latest version of Camel (with its updated dependencies) is another way to improve application performance for free.

OS

You can squeeze the application only so much. In order to do proper high load processing, tuning the host system is a must too. To get an idea for the various OS level options, have a look at the following check list from the Jetty project.

In Conclusion

This article is here just to give you some ideas and show you the extend of the possible areas to consider when you have to improve the performance of a Camel application. Instead of looking for a magical recipe or go through a checklist, do small incremental changes supported by measurements until you get to a desired state. And rather than focusing on micro optimisations and hacks, have an holistic view of the system, get the design right, and start tuning from the host system, to JVM, CamelContext, routing elements, endpoints and the data itself.

Using well known patterns, principles and practises with focus on simple and scalable design is always a good start. Good luck.

Idempotent Consumer EIP Icon

While writing about the Idempontet Filter Patter in my recent Camel Design Patterns book, I wanted to visualise the Idempontet Consumer EIP on the Camel routes but couldn't find an icon for it. Inspired by existing Normalizer and Resequencer icons, I've created an Idempotent Consumer Icon.

If you look at the Normalizer pattern, it uses different shapes to emphasise that incoming messages are of different data formats. And the Resequencer has the same number of incoming and outgoing messages, but reordered.
Similarly, for the Idempotent Consumer I've used a circle and square to represent different incoming messages (that is not different data formats, but different business data). And to emphasise that it is a stateful pattern and can remember past messages, I've put the duplicate square messages apart from one another. Then from the icon we can see that, the pattern removes the duplicate square messages and allows only unique message to pass on.

Couple of days after that, I've read A Decade of Enterprise Integration Patterns: A Conversation with the Authors where Gregor said this:

Olaf: And what would you do differently now?
Gregor: I would make an icon for the Idempotent Receiver pattern, which describes a receiver that can process the same message multiple times without any harm. Somehow we seem to have missed that one. 

Not sure whether one day the icon will become part of the existing EIP icon set or not, but I've found it visually helpful to depict the idempotent behaviour. You can download a .png, visio and OmniGraffle stencil from here.

Camel Design Patterns eBook is Out

I've been involved with Apache Camel for many years now and apart from the occasional contributions, and blogging, I've used it in tens of projects over the years. That includes projects for large broadcasting companies, newspapers, mobile operators, oil companies, airlines, digital agencies, government organisations, you name it. One common theme across all these projects is that the development team loves Camel. Camel has always been a flexible and productive tool that gives the developers the edge over the changing requirements and short deadlines.

Having seen many successful Camel projects, I try to share my experiences through blogging, but this time decided to invest more time and create an ebook called Camel Design Patterns. It is not another Camel book documenting the framework itself and the individual Enterprise Integration Patterns, but rather a collection of SOA, Microservices, Messaging, Cloud, Resiliency patterns that I've used in Camel based solutions day by day. Its format is similar to a series of essays or blog posts with high level examples showing different techniques and Camel tips for designing and architecting modern Camel applications.

Table of Contents 

• I Foundational Patterns
  • Edge Component Pattern
    
  • VETRO Pattern
  • CQRS Pattern
  • Canonical Data Model Pattern
  • Reusable Route Pattern (new)
  • Idempotent Filter Pattern
  • External Configuration Pattern
• II Error Handling Patterns
  • Data Integrity Pattern (PDF)
  • Retry Pattern
  • Throttling Pattern
  • Circuit Breaker Pattern
  • Saga Pattern
  • Error Channel Pattern
• III Deployment Patterns
  • Service Instance Pattern
  • Singleton Service Pattern
  • Parallel Pipeline Pattern
  • Load Leveling Pattern
  • Bulkhead Pattern
  • Service Consolidation Pattern

It is a live book, and depending on interest and feedback, I plan to add more chapters and use cases in the future. The book costs around aVenti Espresso Frappuccino and for now it is available only on leanpub. To get a feel about the content, have a look at the sample Data Integrity Pattern chapter.

I hope you find this ebook useful and looking forward to receiving your feedback.

JBoss Fuse and JBoss BPM Suite Integrated

Last week Apache Camel v2.16 was released with over 600 issues resolved (bug fixes, updates and new features). Claus blogged about top 10 highlights already and here I want to briefly show you a use case about my tiny contribution in this release: camel-jbpm component.

This component is intended to make integration with jBPM (a Business Process Management Application) easier. It uses Remote Java API and basically does REST calls to jBPM to manage business processes (start, stop, abort), execute Business Rules, manage Human Tasks, etc. So rather than manually creating a REST client or using Remote API, you can use the Camel component. The remote API provides also HornetQ based integration, but I've dropped out that part from the Camel component as it didn't play nicely with OSGI.

In the sample use case (which is inspired by a real one) there is an integration application based on Camel that runs on JBoss Fuse which is supposed to do real time integration. And whenever an error occurs in the integration route, the error details are sent to jBPM for evaluation and review by human being if necessary. To demonstrate that I've created two separate projects:

• A Camel project that runs on Fuse, handles any exception and sends the details to jBPM
• A Business Process project that runs on jBPM, classifies the errors using Drools and Complex Event Processing and creates Human Tasks if the error is critical.

The Camel route below is self explanatory, there is a route that generates 10 exceptions with half a second delay each and an error handler that catches those exceptions and sends them to jBPM. Notice that it also populates CamelJBPMParameters header with the details about the error.

You can find the sample code on github with instructions about how to build and run it on Fuse.

The second part of the demo consist of a jBPM process that receives the errors and processes them. The BPMN2 diagram of the process is below:

The process have input parameters matching the values from CamelJBPMParameters header. In the first step it initializes a RequestQualifier object that is marked as non-critical. In the second step we use Drools and Complex Event Processing to qualify errors. The idea with CEP is that we want to mark an error as critical only if there were more than 5 errors in 10 seconds time window. Also to prevent marking multiple errors as critical and flooding the system with Human Tasks, we limit it to only 1 critical error at most in 10 minutes interval. All these conditions are written in Drools as following:

Once the request with the error details has been qualified (as critical or not), we use that information to do conditional routing. All non critical errors are logged and the business processes complete as there are no further action defined for them. We can see the full list of completed processes and error related information in the following screen shot:

On the other branch of the business process, if an error has been marked as critical, we do create a Human Task and the process stops at this point. This requires a user to assign the task and take some actions. Once the Human task is claimed and completed, then the business process continues and completes in the next step.
From the ten error messages generated in quick succession, only one is qualified as critical and generates Human Task and the remaining nine gets logged and completed as expected.
The business process project with instructions on how to deploy it to jBPM is on github too.

JBoss Fuse Deployment Considerations

Fuse is a framework for creating distributed applications and as such it has many moving parts. There are bundles, features, profiles, containers and various ways of combining those into the resulting deployment architecture. Here are some thoughts and guidelines based on the field experiences from a number of projects.

If we start from bottom up, we have bundles, features and profiles as the main resulting artifacts from the development process.

An OSGI bundle corresponds to a maven module and it is basically a jar file with additional MANIFEST.MF that contains OSGI specific metadata. A developer doesn't have to do much other than declare the maven packaging type as a bundle and let maven-bundle-plugin generate MANIFEST.MF file at compile time.

A Karaf feature is the preferred way for declaring dependencies in Karaf. So rather than installing each bundle and its runtime dependencies one by one, features allow us to group declaratively a bunch of bundles that should be deployed together at run time. Think of features as a way to describe the minimal deployment unit that represents a business capability (kinda a miroservice descriptor ;)

An example may help here: let's say we have an order-domain bundle, order-spec bundle with a wsdl and xml schemas, order-routes bundle with few Camel routes, and order-persistence bundle. And order-feature would combine all these bundles and other dependencies (which in turn can be bundles or features) such as camel components and third party libraries. The point is that order-feature is the minimal deployment unit that provides business value and represents a runnable unit of code (with inputs and outputs). It is also possible to use features just to group a set of bundles that usually go together and provide technical capability like JTA/JPA feature. Having a business capability and all of its dependencies declared in one place makes it really easy to move that deployment unit from container to container for scalability and HA purposes. Deploying and running features to a container is achieved through profiles.

Fuse profiles consist of two main ingredients: binaries (which are bundles described by features) and configurations/properties (which are usually environment specific) needed to run the binaries. So I would create an order-profile (that maps 1 to 1 to order-feature) containing order-feature and set of property files (to configure order-persistence bundle with DB configs, order-routes bundle with port number of the order service, etc). So order-profile would contain the same binaries on all environments but different property files (specific for each environment - dev/test/int/prod).

So we have camel routes->bundles->features->profiles. The next question is how do we organize profiles into containers considering that in addition to the business profiles we create during development, there are also out-of-box technical profiles that a typical Fuse application uses. Let's clarify the term container first. A container is overloaded term, especially with the popularity of Docker containers, but when we say container in the Fuse world, it refers to Karaf container, that is a standalone Java process (imagine a microservice instance if you prefer). And the big magic done by Fuse Fabric is the ability to deploy and run code on Karaf containers by assigning profiles to a container. So profiles dictates what features and bundles are deployed on certain container giving a container a specific responsibility/flavor.

Next, let's have a look at different types of containers we have in a typical Fuse application:

• Fabric Server - This is where the ZooKeeper server is running, together with the maven proxy, git server and Hawtio instance.
• A-MQ Container - runs A-MQ server instance(s).
• Worker/ESB container - this is where the business/integration logic is running. That is typically Camel and CXF based (our order profile would be deployed here).
• Insight container - is where the Elastic Search server runs.
• Fabric Gateway - used for discovery and load balancing of CXF or A-MQ endpoints.

One point that is worth clarifying is that the containers are generic, it is the profile that is deployed that gives a container a role. And in theory it is possible to deploy various combination of profiles from the above categories to the the same container but that is not a wise idea as these profiles are fundamental for the rest of the application and it is better to isolate and allocate enough resources to each of them rather than making them share the same JVM. Under a heavy load on the message broker you wouldn't want to have ZK server struggling for resources and jeopardize the health of the whole Fabric. On the other hand deploying business profiles (order and invoice profile for example) to the same container is perfectly fine if it makes sense from non functional requirements point of view. So try to put the above profiles/responsibilities/roles into separate containers and split/group your business profiles into containers based on the use case. More on this topic below.

So we have our containers deployed with profiles which in turn assign specific responsibilities to the containers. How do we distribute these containers into available nodes/hosts(physical or VMs)? This is a very common question and the answer is as you may have guesses: it dependents. It dependents on the available resources and non functional requirements. The main consideration at this stage is to avoid single point of failure, enable horizontal scaling of the application based on the anticipated load and keeping in mind application maintainability too. All this sounds nice but doesn't say anything concrete. So here are few areas to think further before making a decision and connecting the boxes in the deployment diagram. Each of these different types of Karaf containers has specific resource and availability requirements and depending on the expected load and available resource you will end up choosing one or another way of deploying these profiles.

Fabric Servers require odd number of instances, with minim of 3 but preferably 5 or more. That is because ZK favors consistency over availability and it requires the majority if its instances be up and running. If that condition is not met, the rest of the ZK instances stop functioning, meaning the other containers will not have any of the Fabric services such as A-MQ discovery, CXF-discovery, master-component, provisioning available. To prevent that, it is recommended to isolate the Fabric Server containers into separate nodes with good connectivity between the nodes and not run other containers on the same host that may steal resources. These containers don't require too much resources, so probably 1-2 cores and 1-2GB of heap will be enough in most cases but they are essential for the rest of the Fabric!

The other fundamental part of a Fabric installation are the A-MQ containers. If the application is using messaging layer and the broker goes down, the rest of the system becomes useless, that's why you would usually have A-MQ either in Master-Slave or/and in Network of Brokers cluster. In most cases that will be Master-Slave setup, where the two broker containers are split in separate hosts with access to shared datastore. In such a setup you will have an active A-MQ instance on one host and passive A-MQ instance in the second host. As such the second host will not be utilized at all, and it is good idea to put another set of Master-Slave cluster in the opposite hosts to have better utilization of the resources. Also keep in mind the broker resource appetite (cpu/memory/io) is directly dependent on the message load, so treat your broker with the same respect as you would treat a database. If the broker is persistent, it will be IO intensive, if there is not enough memory the broker will flush to disk, if your consumers are slow you may run out of disk space, if there is not enough processing power, you may run out of heap space... Many ways to kill a broker, if it doesn't have the resources it needs.

The containers that require some more imagination in terms of deployment topology are the ESB/Worker containers. The way these containers are deployed is really dependent on how many business services are running, what are the dependencies between those services and what are the non functional requirements for each service. A service (usually consisting of few Camel routes) may be CPU intensive (complex calculation for example) or memory intensive (huge messages) or IO intensive (typically file copying) or a combination of those. A service might be a low latency real time service with auto scaling requirements, or business critical that should be highly available, or batch oriented with manual fail over, etc. So when deploying multiple business profiles into one container or multiple containers into one host consider the following options for each service:

• Business criticality: highly critical services on dedicated containers and safest hosts
• Load: services with high load on hosts with more resources
• Coupling: services coupled together (direct in memory calls are more efficient that http or jms for service interaction) or service depending on same resources
• Functionality: services belonging to the same business domain or applications layer

No matter what is the topology, keep in mind that it is not final and Fabric makes it really easy to move profiles across containers and containers accords hosts. So you have the tools to change and adapt when needed.

Monitoring containers with Fabric Insight profiles are new to Fuse and their role is primarily running Elastic Search servers that read and write logs to Lucene indexes. These containers will be primarily IO intensive and not critical for the rest of system, but keep in mind the available disk space and how much logs you would want to keep around. From business perspective it might be hard to justify dedicated hosts only for logging purposes, so you might end up sticking monitoring containers on Fabric Server hosts (makes sense as both types of containers have monitoring/management responsibilities) or to any host that have some additional capacity.

Fabric Gateway is a lightweight load balancing and discovery service for CXF and A-MQ endpoints that are managed by Fabric. As this profile has a distinct and crucial responsibility that enables consumers to discover service providers, it seems logical to have multiple instances of such containers located on the hosts with the service providers or any other hosts including locating them on the host where the consumer is.

Below is a diagram that illustrates a sample Fuse deployment topology following the above thoughts. Depending on your use case, you may have more or less Fabric servers (hopefully 5 rather than 1), may be only one A-MQ Master/Slave cluster, and more ESB containers. If you can get also connectivity from build server to the Fabric Server 1, that will enable automatic deployments (requires some scripting) and have full CI/CD cycles (also your Fabric servers will need access to maven repositories).

No matter whether you call it SOA or Microsservices, ESB or light integration layer, distributed applications have many moving parts and building one is a complex (and enjoyable) journey. The post here is quite high level and focusing on the happy paths only. For any real discussions, have a chat with Red Hat.

Apache Camel for Micro­service Architectures

I've been using microservice architectures before I knew they were called so. I used to work with pipeline applications made up of isolated modules that interact with each other through queues. Since then a number of (ex)ThoughtWorks gurus talked about microservices. First Fred George, then James Lewis and finally Martin Fowler blogged about microservices making it the next buzzword so every company wants to have few microservices. Nowadays there are #hashtags, endorsements, likes, trainings, even 2 day conference about it. The more I read and listen about microservice architectures, the more I realize how Apache Camel (and the accompanying projects around it) fits perfectly to this style of applications. In this post we will see how Apache Camel framework can help us create microservice style applications in Java without much hussle.

Microservices Characteristics

There is nothing new in microservices. Many applications have already been designed and implemented as such for a long time. Microservices is just a new term that describes a style of software systems that have certain characteristics and follow certain principles. It is an architectural style where an application or software system is composed of individual standalone services communicating using lightweight protocols in event based manner. The same way as TDD helps us to create decoupled single responsibility classes, microservices principles guide us to create simple applications at system level. Here we will not discuss the principles and characteristics of such architectures or argue whether it is a way of implementing SOA in practice or a totally new approach to application design, but rather look at the most common practices used for implementing microservices and how Apache Camel can helps us accomplish that in practice. There is not definitive list (yet) but if you read around or watch the videos posted above, you will notice that the following are quite common practices for creating microservices:

1. Small in size. The very fundamental principle of micro services says that each application is small in size and it only does one thing and does it well. It is debatable what is small or large, the number varies from 10 LOC to 1000 but form me I like the idea that it should be small enough to fit in your head. There are people with big heads, so even that is debatable ;) but I think as long as an application does one thing and does it well so that it is not considered a nanoservices, that is a good size.
Camel applications are inherently small in size. A camel context with couple of routes with error handling and helper beans is approximately 100 LOC. Thanks to Camel DSLs and URIs for abstracting endpoints, receiving an event either through HTTP or JMS, unmarshaling it, persisting and sending a response back is around 50 LOC. That is small enough to be tested end-to-end, rewritten and even thrown away without feel any remorse.

2. Having transaction boundaries. An application consisting of multiple microservices forms an eventually consistent system of systems where the state of the whole system is not known at any given time. This on its own creates a barrier for understanding and adopting microservices with teams who are not used to work with this kind of distributed applications. Even though the state of the whole system is not fixed, it is important to have transaction boundaries that define where a message currently belongs. 
Ensuring transactional behaviour across heteregenous systems is not an easy task, but Camel has great transactional capabilities. Camel has endpoints that can participate in transactions, transacted routes and error handlers, idempotent consumers and compensating actions, all of which help developers easily create services with transactional behavior.

3. Self monitoring. This is one of my favorite areas with microservices. Services should expose information that describes the state of various resources it depends on and the service itself. These are statistics such as average, min, max time to process a message, number of successful and failed messages, being able to track a message and so forth.
This is something you get OOTB with Camel without any effort. Each Camel application gathers JMX statistics by default for the whole application, individual routes, endpoints, etc. It will tell you how many messages have completed successfully, how many failed, where they failed, etc. This is not read only API, JMX allows also updating and tuning the application at run time, so based on these statistics, using the same API you can tune the application. Also the information can be accessed with tools such as jConsole, VisualVM, Hyperic HQ, exposed over HTTP using Jolokia or feed into a great web UI called hawtio.

If the functionality that is available OOTB doesn't fit your custom requirements, there multiple extension points such as the nagios, jmx, amazon cloudwatch and the new metrics components, or use Event Notifiers for custom events.
Logging in messaging applications is another challenge, but Camel's MDC logging combined with Throughput logger makes it easy to track individual messages or get aggregated statistics as part of the logging output.

5. Designed for failure - Each of the microservices can be down or unresponsive for some time but that should not bring the whole system down. Thus microservices should be fault tolerant and be able to recover when that is possible.
Camel has lots of helpful tools and patterns to cope with these scenarios too. Dead Letter Channel can make sure messages are not lost in case of failure, the retry policy can retry to send a message couple of times for certain error conditions using custom backoff method and collision avoidance. Patterns such as Load balancer which supports Circuit breaker, Failover and other policies, Throttler to make sure certain endpoints do not get overload, Detour, Sampler, are all needed in various failure scenarios. So why not use them rather than reinventing the wheel in each service.

6. Highly Configurable - It should be easy to configure the same application for high availability, scale it for reliability or throughput, or said another way: have different degrees of freedom through configuration.
When creating a Camel application using the DSLs, all we do is to define the message flow and configure various endpoints and other characteristics of the application. So Camel applications are highly configurable by design. When all the various options are externalized using properties component, it is possible to configure an application for different expectations and redeploy without touching the actual source code at all. Camel is so configurable that you can change an endpoint with another (for example replace HTTP endpoint with JMS) without changing the application code which we will cover next.

7. With smart endpoints.  Micro services favour RESTish protocols and lightweight messaging rather than Web Services.
Camel favors anything. It has HTTP support as no other framework. It has components for Asynchronous Http, GAE URL fetch service, Apache HTTP Client, Jetty, Netty, Servlet, Restlet, CXF and multiple data formats for serializing/deserializing messages. In addition the recent addition of Rest DSL makes REST a first class citizen in the Camel world and simply creating such services a lot. As for the queuing support, OOTB there are connectors for JMS, ActiveMQ, ZeroMQ, Amazon SQS, Amazon SNS, AMQP, Kestrel, Kafka, Stomp, you name it.

8. Testable. There is no common view on this characteristic. Some favor no testing at all and relying on business metrics. Some cannot afford to have bad business metrics at all. I like TDD and for me having the ability to test my business POJOs in isolation from the actual message flow, then test the flow separately by mocking some of the external endpoints is invaluable. Camel testing support can intercept and mock endpoints, simulate events, verify expectations with ease. Having a well tested microservice for the expected behavior is the only guarantee to have the whole system to work as expected.

9. Provisioned individually. The most important characteristics of microservices is that they run in isolation from other services most commonly as standalone Java applications. Camel can be embedded in Spring, OSGI or web containers. Camel can also run as a standalone Java application with embedded Jetty endpoints easily. But managing multiple processes, all running in isolation without a centralized tool is a hard job. This is what Fabric8 is made for. Fabric8 is developed by the same guys who developed Camel and supported by Red Hat JBoss. It is a poly Java application provisioning and management tool that can deploy and manage a variety of Java containers and standalone processes. To find out more about Fabric8, here is nice post by Christian Posta.

10. Language neutral. Having small and independently deployed applications allow developers to choose thebest suited language for the given task. Camel has XML, Java, Scala, Groovy and few other DSLs with similar syntax and capabilities .But if you don't want to you use Camel at all for a specific micro service, you can still use Fabric8 do deploy and manage applications written in other languages and run them as native processes.

In summary: Microservices are not strictly defined and that's the beauty. It is a lightweight style of implementing SOA that works. So is Apache Camel. It is not a full featured ESB, but it can be as part of JBoss Fuse. It is not a strictly defined specification driven project, but a lightweight tool that works and developers love it.

Referecences
1. Micro-Service Architecture by Fred George (video)
2. Micro-Services - Java, the UNIX way by James Lewis
3. Microservices by Martin Fowler
4. µServices by Peter Kriens
5. Micro Services the easy way with Fabric8 by James Strachan (with video)
6. Fabric8 by Red Hat
7. Meet Fabric8: An open­source integration platform by Christian Posta
8. Micro Services the easy way with Fabric8 by James Strachan