Blogroll

Which Camel DSL to Choose and Why?

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)

Apache Camel is a powerful integration library that provides mainly three things: lot’s of integration connectors + implementation of multiple integration patterns + a higher level Domain Specific Language abstraction to glue all together nicely. While the connectors and pattern choices are use case and feature driven and easy to make, choosing which Camel DSL to use might be a little hard to reason about. I hope this article will be able to guide you in you first Camel journey.

I work for Red Hat Consulting as an Integration architect and one of my primary goals is to help customers get the design and the architecture of their future systems as right as possible, and consequently get the best value out of Apache Camel. One of the common questions I get at the start of every new Camel based project is: “Which Camel DSL should we use? What are the pros and cons of each?

I have one good news and one more good news for you. First, by choosing to use Apache Camel you have already done the right choice and Camel will turn out to be a very useful toolkit in your arsenal of libraries for lot’s of future use cases and projects to come. And second, the DSL is just a technicality and it will not impact the success of your project and you can always change your mind later and even mix and match.

If you are a part of large company, with multiple independent two-pizza size teams that use Java language here and there, the chances are that some teams are already using Camel. Even in small companies, teams use Camel without being aware of each other as it is useful for all kind of tasks and a small enough library you to add to your pom.xml and use it without a permission from the technical design board. If that is the case, just talk to your colleagues and learn from first hand their experience with their DSL of choice.

If you need a more comprehensive comparison and a reason to choose one of the DSLs, below is a brain dump from multiple engineers developing developing Apache Camel and consultants using Camel at multiple customer projects across the globe. Pick the arguments that are valid in your context and make your choice.

Comparing Apache Camel's XML and Java DSLs
Comparing Apache Camel's XML and Java DSLs
If this table doesn’t give you the straight answer you were looking for, probably the answer is: it doesn’t matter. Camel has multiple DSLs, but there are good reasons for both Java and XML based DSLs to be equally popular. The more important takeaway from here is for developers to get used to think in terms of Pipes and Filters, learn the Enterprise Integration Patterns and their notations. Then using one of the Camel DSLs to express these patterns is a technicality without a technical consequence. Usually it is a team preference and culture based choice, such as “We are a hard-core Java shop and we hate XML” or “Can we do all through drag and drop?”.

All that said, the only advice I can suggest is strive for consistency. Avoid using different DSLs in the same service, even for different services in the same project. And if you can convince everybody in the company to use the same DSL even better.

Check out my Camel Design Patterns book for more Camel related topoics and follow me @bibryam for future blog posts.

Rapid SEMAT Application Development with Apache Isis

TL;DR This post talks about a SEMAT pet project I created using Apache Isis and deployed to OpenShift Online here http://semat.ofbizian.com

Apache Isis

As a Java developer who is working primarily on backend systems, I hate do not enjoy creating user interfaces and dealing with Javascript. Luckily, there are Java projects such as JSF (grrr), Apache Wicket, Vaadin that can help avoid Javascript altogether and still create functional user interfaces. But even with these projects, the developer has to think about and actively create the user interface from Java code. That is similar to writing your own SQL statements in the age of ORMs such as Hibernate - an activity we do only when OOTB ORM is not good enough for the use case. And that is exactly where Apache Isis fits in: given a domain model and mapping annotations, it generates the complete user interface at runtime. In a sense, Apache Isis is a OUIM (Object/User Interface Mapping) framework for Java.
There is much more to Apache Isis than only creating user interfaces, it is a full stack rapid application development framework focused on domain driven design. But rather than talking about it, let's see a complete application created with Apache Isis.

SEMAT Essence Kernel

To learn Apache Isis, I decided to implement the SEMAT model and deploy it to OpenShift Online asa  Docker container. Simply said, SEMAT (Software Engineering Method and Theory) Essence Kernel is a OMG Standard that helps define among other things a framework for describing the state of software projects from multiple perspectives (called alphas).


SEMAT Alpa States
The idea is, that every project can be described in a generic way using the following seven alphas:
Stakeholders, Opportunity, Requirements, Software System, Work, Team, Way-of-Working. And each Alpha can be in one or multiple states, for example, the Stakeholders can be: Recognized, Represented, Involved, In Agreement, Satisfied, etc. In addition, each state has certain items to be satisfied before an Alpha can be transitioned to that state.

Stakeholders Alpha's States
As you can see, this is a pretty simple domain model with a state machine logic behid it.

The Showcase Application

Enough said, to see how much Java I had to write for this application, check the dom module of the project on github. All of the other skeleton code is generated through a maven plugin and no user interface code is required. And here is a screenshot of the Project domain entity screen rendering:

Project view as Apache Wicket screen
In addition to generating a user interface, Apache Isis will generate also a REST API using the same domain model. How cool is that.
SEMAT REST API generated from domain model
And the beauty of all this is, that generating UI allows you quickly to iterate over the domain model, show it to the business owners to get feedback and contonue evolving the model.

Some of the SEMAT Application Features implemented/enabled

  • Multi tenancy
  • Manage multiple projects per tenant
  • Manage project Alpha states
  • Custom Essence Alpha state list per tenancy
  • Custom Essence Checklist items per tenancy
  • Alpha state spider/radar diagram
  • Automatic Apache Wicket based UI generation from domain model
  • Automatic REST API generation from the same domain model
  • Self Signup/Registration
  • Auditing user actions
  • Session logging
  • Internationalization
  • Breadcrumb trail
  • Bookmarks

Build and Run

Check the readme for full details, but you can build and run the application locally or on OpenShift to try it out.
mvn clean install
cd webapp
mvn jetty:run


or 

mvn clean install
docker build --rm -t bibryam/semat .
docker run -p 8080:8080 bibryam/semat
Then go to http://localhost:8080/ and login: user/user
 

Deploy to OpenShift

Once you have got an OpenShift running either locally or online, and have a oc client installed, then you can deploy the already build semat docker image with the following commands:
oc new-project semat
oc new-app bibryam/semat:latest -e CATALINA_OPTS=“-Xmx300m”
oc expose service semat
If you do not trust docker images build by others (you should not!) then you can build your own docker image as shown above with options 2 and 3 and push it your own docker registry and run the application from it:
oc new-app your_name/semat:latest -e CATALINA_OPTS=“-Xmx300m”
Alternatively, you could avoid installing and running docker all together, and have the source code and the docker image build on OpenShift. That is called OpenShift Source-to-Image approach. You can do this from OpenShift UI by using for example "Red Hat JBoss Web Server 3.1 Tomcat 8 1.0" template and pointing to the SEMAT github repo. Or use the template provided in the project itself:
oc create -f semat-openshift-template.json
oc process semat
Using source to image approach allows setting up github webhooks, have a Red Hat base image, have jolokia added, Java memory configurations done, etc.

Live demo on OpenShift

See try out the application, check live demo running on a OpenShift Online http://semat.ofbizian.com

In summary, if you have a domain model that changes often, and the agility in changing the domain logic is more important than how the user user interface looks like, check out Apache Isis. It is an incredible productive and fast business application development framework.
Follow me @bibryam for future blog posts on related topics.

Hexagonal Architecture as a Natural fit for Apache Camel

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)

There are architectures and patterns that look cool on a paper, and there are ones that are good in practice. Implementing the hexagonal architecture with Camel is both: cool to talk about, and a natural implementation outcome. I love going hexagonal with Camel because it is one of these combinations where the architecture and the tool come together naturally and many end up doing it without realising it. Let’s see why that is the case.

Why go Hexagonal?

Hexagonal architecture is originally described by Alistair Cockburn as an approach for dividing an application into inside and outside parts. Its intent is to move focus from multiple conceptual layers of an application to a distinction between the inside and outside parts of the application. The inside part represents the domain layer or the business logic, and the outside part consists of all the possible incoming or outgoing interaction points of the application. The same architecture is also known as Ports and Adapters as the the connection between the inside and the outside of the application is realized through ports and adapters. The word “port” is inspired by the operating systems ports where any application that conform to the protocol of a port can send or receive signals from an application. In a sense, a port represent a purposeful conversation. And the adapters represent the technology specific implementations of a port. Depending on the business benefits offered through the port, there might be multiple adapters that would like to expose the port using different technologies.
Hexagonal architecture visualized with Enterprise Integration Patterns
Hexagonal architecture visualized with Enterprise Integration Patterns
Notice that all ports and adapters are fundamentally similar at the architectural level, but Alistair acknowledges that the ports and adapters come up in two flavours: primary and secondary or driving and driven. For example, if there is a simple REST based service that reads and writes to a database, the REST side of the service would be the primary actor port and adapter as it initiates and drives the interactions. The port and adapter for writing to the database side would be the secondary and driven actor as it is not initiating any calls (assuming we are not using any data change capture listeners in which case this adapter would also be a primary one).
Briefly said, hexagonal architecture helps us avoid multi-layered architectures that are prone to end up being baklava architecture (anti-pattern). Instead it pushes us towards simplified separation of concerns, and onion-architecture, clean architecture, and similar.

Why is Camel Hexagonal in Nature?

Let’s look at the two extremes: a layered architecture manages the complexity of a large application by decomposing it and structuring into groups of subtasks of particular abstraction level called layer. Each layer has a specific role and responsibility within the application and changes made in one layer of the architecture usually don’t impact the components of other layers. In practice, this architecture splits an application into horizontal layers, and it is a very common approach for large monolithic web or ESB applications of the JEE world.
Layered architecture compared to Pipes and Filters pattern
Layered architecture compared to Pipes and Filters pattern
On the other extreme is Camel with its expressive DSL and route/flow abstractions. Based on Pipes and Filters pattern, Camel would divide a large processing task into a sequence of smaller independent processing steps (Filters) connected by channel (Pipes). There is no notion of layers that depend on each other, and in fact, because of its powerful DSL, a simple integration can be done in few lines and a single layer only. In practice, Camel routes split your application by use case and business flow into vertical flows rather than horizontal layers. And a typical Camel application is composed of multiple independently working Camel routes that collaborate for achieving the common business goals. As mentioned previously, when working with Camel, services created with it tend to end up as a single layer. Whereas this is fine for most of the simpler cases, applying the hexagonal architecture principles will help with creating better applications when working on large scale projects. What I mean by that is, split your Camel routes into two layers that represent the inside and the outside of the application. The inside of the application is represented by Camel routes that implement the business logic of your integration, and intended to be reused by multiple other routes and protocols. The outside of the application would be implemented by Camel routes that are the adapters in the hexagonal architecture i.e. routes that provide technology specific logic e.g. handle a specific protocol, error handling logic specific for the endpoint, transactionality and recovery actions specific for the endpoint as well.

How to Map Hexagonal Architecture to Camel?

Identify the inside of your application

Even the simplest services created with Camel have some kind of business logic. Usually that is a combination of transforming data, content-based routing, filtering, splitting, aggregating, etc. Very often, none of the out of the box enterprise integration patterns will be applicable and you will have to use your own custom Java bean as part of a Camel route. The awesome part is that Camel is completely non-intrusive and you can develop, test and use Java beans in Camel routes with absolute no dependency on the Camel APIs. Camel bean component will make sure that the bean method parameters are populated with the correct values and also take the return value and put it back into Camel routes. If you have identified the routes containing the elements mentioned above, typically these represent the inside of your application. These kind of routes should not contain logic that is technology and protocol specific. For example avoid using data that is directly populated by components, such as http headers, jms headers, and also error handling retry logic common for the HTTP protocol, compensating action logic, etc. Instead keep this inside Camel routes focused on the business logic only and isolated from outside Camel routes.

Isolate the inside from the outside

In the hexagonal architecture, the inside of the application is reached through ports that abstract conversations. Camel direct component is the perfect implementation of a port. It provides synchronous invocation, the same as a method call in Java. It is not technology and protocol specific, there is no specific data format or schema validation requirement and can be used to pass in and out any kind of data. Typically the preferred data format to pass is a POJO as it is the easiest and most flexible structure to manipulate in a Camel route. But if in your domain, the primary data format is XML, JSON, or anything else, you can keep to such a format as well. No strong rules to follow, but whatever works for you. The only thing that is fixed with direct component is that it is a synchronous interaction model and I think that is the correct one by default. If asynchronicity is required, rather than using SEDA component for a port, it would be better to implement asynchronous logic either as part of the outside route (if asynchronicity is required by an adapter) or in the inside route if it is part of the business logic. But don’t limit the port to asynchronous model only. A port represents a meaningful conversation in the context of a service. That in Camel is represented by direct component which is identified uniquely as a String value in the context of a JVM. One would think that direct component was implemented as a response for Alistairs port definition.

Keep outside out

So we have the business logic of our application implemented as Camel routes accessible only through direct component endpoints as ports. Such a setup allows testing, reusing, and exposing the business logic over multiple protocols to the outside world using other routes. The outside of the application contains any logic that is dependent on the endpoints. Nowadays most common of these are the messaging or file based for asynchronous interaction, and HTTP based for synchronous interaction. But it can also be any of the other over 200 connectors that are present in Camel. Keep in mind that the components you use for the outside, are not only dictating the interaction model, but also usually define the data format, the transaction semantics, the error handling logic, and potentially even other aspects of the applications. For example a SOAP endpoint will perform a schema validation, but consuming a JMS message will require an additional validation step. A transactional endpoint will perform rollback in the case of a failure, but a non-transactional endpoint will require a recovery action, an idempotent endpoint will allow retry, and a non-idempotent endpoint will not. I have described these kind of considerations and other related Camel use cases in more details in the Camel Design Patterns book. Putting it all together, a Camel based service that exposes some business functionality over SOAP and JMS is visualized below. The same business functionality is accessible through direct component for JMS and SOAP based routes. Also, on the right hand side, the same route is using an email notifications port and adapter for sending emails.

Ports and Adapters based Camel service
Ports and Adapters based Camel service
 Notice that, outside routes are not only on the consumer side, they are also on the producer side, i.e. routes that send messages to other systems (remember driving and driven ports/adapters). The intent of the outside routes is to represent the various adapters that should handle everything that is outside specific: protocol, data format, additional logic specific for the endpoint. In addition, an outside route should prepare the data in format that is expected by the port by populating expected headers and the message body. This would allow the same port to be reused by multiple adapter routes. This includes also test fixtures in Camel for unit testing Camel routes, and even the error handling code. The error handling constructs in Camel (no doTry, doCatch, doFinally, but the onException construct) is actually representing a port that is automatically called by the framework on different types of exceptional conditions. Such a concept doesn’t exist in the Java language, but in Camel it is a very commonly used execution path for unhappy scenarios. And treating the error handling flow as just another port in your application (even if it is not called by you but the framework on certain occasion), will help you to reuse it for common error handling across multiple Camel routes.

In Summary

There are no clear rules or guidelines on how to compose an application with Camel routes. Defining those at design time usually limits the creativity of developers at implementation time, and not having guidelines can be a recipe for a spaghetti architecture. In this line of thought, I think hexagonal architecture is sufficiently lightweight, doesn’t kill creativity and imagination during implementation by forcing specific structure. At the same time, it provides just enough guidance for structuring routes. And the best part is, it naturally fits the Camel programing model.
My suggestion would be start with VETRO pattern (Validate, Enrich, Transform, Route, Operate), then apply hexagonal architecture style (Edge Component Pattern as described in Camel Design Patterns book). This is a good starting point for structuring Camel routes for the happy paths. Then pay special attention to achieving data consistency with the various error handling and recovery patterns. And don’t forget, there are no best practices, but only good practices in a context. Focus on your context and Camel will be on your side.
Follow me @bibryam for future blog posts on related topics.

About Me