Blogroll

The next integration evolution - blockchain

Below is the conclusion from an article I wrote at TechCrunch. Checkout the full article here.

Enterprise integration has multiple nuances. Integration challenges within an organization, where all systems are controlled by one entity and participants have some degree of trust to each other, are mostly addressed by modern ESBs, BPMs and Microservices architectures. But when it comes to multi-party B2B integration, there are additional challenges. These systems are controlled by multiple organizations, have no visibility of the business processes and do not trust each other. In these scenarios, we see organizations experimenting with a new breed of blockchain-based technology that relies not only on sharing of the protocols and contracts but sharing of the end-to-end business processes and state.
Integration evolution stages

And this trend is aligned with the general direction integration has been evolving over the years: from sharing the very minimum protocols, to sharing and exposing more and more in the form of contracts, APIs and now business processes. This shared integration infrastructure enables new transparent integration models where the previously private business processes are now jointly owned, agreed, built, maintained and standardized using the open-source collaboration model. This can motivate organizations to share business processes and form networks to benefit further from joint innovation, standardization and deeper integration in general.

Open Source and Deforestation

Follow me on twitter for other posts in this space. If you prefer, read the same post on Medium.

Open source is like a forest

A forest is a complex ecosystem of plants, animals microorganisms, non-living material, all balanced delicately by nature. It requires the right geography, the right soil, the right amount of rain and sun, and decades to build a forest.

So is open source. An open source project is a delicate ecosystem of contributors, reviewer, users, supporting organizations, all balanced by a feeling of a community. It requires the right ideas at the right time, the right group of developers, the right technology, an enormous amount of dedication and passion, and years to build a project.

Forests are home for many species, the source of oxygen, clean water, and air, prevent floods block winds, source of wood when used in a sustainable manner. Forests offer endless benefits to many when consumed responsibly and without destroying it completely.

So is open source. Open source is the place where newbies learn to collaborate, communicate and code. It is the place where experienced innovate, standardize and distribute cutting edge software. It is the place where remote developers scratch their itch, get paid, and the result benefits everybody. The open source model provides the foundations of the digital infrastructure of modern human life.

Deforestation

In the late 1960s, the deforestation of the Amazon (not the company, but the rainforest) started at an enormous rate. Trees were cleared, lands were transformed. The delicately balanced rain forests were destroyed unrecoverably leading to varying degrees of loss of soil, erosion, landslides, climate change, and even change the patterns of weather. And some companies captured enormous value from destroying the commons of nature by cutting the trees for wood and fuel in an unsustainable manner.

Tragedy of the commons image by Wikipedia

Today, open source is the latest battleground that could have consequences similar to deforestation. It is a battleground of business models, a battleground of small against large, of consultancy and tool producers against large cloud service providers.

On one hand, there are the small companies employing open source developers to build and maintain open source projects and become de facto consultancy and tooling provider around that technology. The open source model helps attract customers for the small businesses and they contribute back their work in return. It is a win-win for free riders, paying customers, maintainers, and the open source projects continue getting wider adoption.

But enterprise customers don’t want consultancy or tooling, they want to use technology in a fast, scalable, reliable and secure way and focus their resources on the business domain instead. And this is what cloud services offer. If that premise is true, any widespread open source technology will eventually be wrapped as a cloud service, and companies will prefer to use it there rather than running themselves in-house with the help of consultancy or proprietary automation tools.

That leaves small companies benefiting from and contributing to open source, out of business. As Thomas Dinsmore said it on Twitter “It’s impossible to argue that software should be open AND the originators have the sole right to monetize. If you believe in the latter, the solution is commercial software. With a license key.”. Any company benefiting from the open source model is also accepting the risk of being eaten by a large cloud service provider eventually. These are the rules of the open source game, and they are fair.

That makes the future of these open source project unclear. When technology is offered as a cloud service, the smaller companies start protecting their investments by introducing additional licenses and moving away from being truly open source. We have seen that with Redis, MongoDB, Kafka, and others to follow eventually. That also means the small companies will have less incentive to develop the open source project beyond the open core elements and instead will focus more effort on their non-open source value adding competitive edges.

As for the cloud providers, they are not obliged to sustain open source by license, not forced by their business model either. If a project stagnates, loses contributors and users, a cloud provider could quicker than anybody jump into the next popular project and offer that as a service. That is likely to lead to a change of dynamics of contributors from small and mid-range companies to individuals (as these early indications around Redis) or new active on the open source arena cloud providers. This is a new reality, and we are yet to see how open source adapts to it.

A sustainable future

The mightiest corporations of our times capture enormous value by using the commons of the open source community, by wrapping the projects into services and offering them for the cost of hardware usage. While benefiting from open source unconditionally and without any expectations is perfectly fine according to the license agreements and the rules of capitalism, benefiting without contributing back a fair share, nor helping for sustainability, has the effects of deforestation of the ecosystem. In the absence of a sustainable model, the delicate balance of contributors and users can be easily broken, leading to confusion, fear, license changes, leading to multi-license projects, leading to discouraged contributors, leading to cautious users, changing the open source model as we all know it, irreversibly.

The good news is that the software industry is increasingly getting more educated on understanding how open source works, what it takes to produce, and sustain it. With recent reactions on social media, we can see that there is no legal, but moral expectation from the companies benefitting from open source most, to play their equal part in sustaining the same open source by being exemplary with their actions, and not only exploit the commons the other contributors are building and the whole society is relying upon.

Today, every household has something made of wood. We can keep it that way by sustaining our forests. Every business depends on something made of open source. We can keep it that way by sustaining our open source ecosystem.

A Java developer’s first impressions from Corda

Follow me on twitter for other posts in this space. If you prefer, read the same post on Medium.

Recently I had a chance to play a little bit with the open source permissioned JVM based blockchain platform Corda. I was surprised to discover how it blends blockchain ideas with the commodity middleware technology and creates a new brief of decentralized enterprise integration. Below are my first impressions from it along with an Apache Camel connector contribution.

What is Corda? 

Corda is a decentralized database and business process platform designed and built from the ground up for the implementation of legal agreements among identifiable parties. It is a DLT implementation heavily influenced by the Bitcoin's UTXO model and driven by the "enterprisy" requirements of the financial industry. Corda is written in Kotlin, runs on the JVM and uses many of the proven middleware technologies. As such, compared to other blockchain platforms, Corda offers a low-entry barrier for Java developers experienced with integration, messaging and business processes management.

Design Principles

  • Permissioned (instead of permissionless network such as Bitcoin, Ethereum, etc) - this is a no surprise as enterprise blockchain use cases are primarily focused around automating the business integration challenges among identifiable parties.
  • Point-to-point (instead of global transaction broadcasts as in Bitcoin and others) - this enables data to be shared only among the nodes that need-to-know it which also leads to improved privacy and scalability.
  • UTXO model similar to Bitcoin (instead of the account model of Ethereum) - which is the part that makes Corda a DLT/Blockchain rather than a distributed business process management platform.
  • Re-use (instead of building everything from scratch) - this is the favorite part of Corda for me. Reuse of the Java ecosystem, reuse of relational databases, reuse of the messaging systems, etc.
The combination of these design principles makes Corda a very unique DLT platform among its competitors. It has elements of Bitcoin UTXO model, Ethereum smart contracts capabilities, Fabric private channels and most importantly - it reuses and builds on top of the existing battle-tested middleware technologies whenever possible.

Main Concepts

  • A permissioned network made up of point-to-point communicating nodes.
  • A ledger where each node maintains its unique database, rather than a single store.
  • Notary nodes that prevent double spends and validate transactions.
  • Oracle services that only sign transactions if the included facts are true (slightly different to typical oracles).
  • State objects are immutable that represent on-ledger facts. The state is modified through transactions and stored on owning nodes only.
  • Contracts are deterministic JVM based functions that validate the transactions.
  • Transactions are candidate updates to the ledger and must be contractually valid and signed to be committed.
  • Flows encapsulate business processes and abstract all the networking, I/O, storage and concurrency. All smart contract activity occurs within the scope of flows which can be started through RPC calls or other flow calls. Flows do not run within sandbox as in the case of contracts but executed as regular Java code.
A Corda Flow that interacts with Node A, Node B, and the Notary Pool

In Ethereum, the concepts of smart contracts encapsulate both the business logic and the state into one. In Corda, state and contract objects are separate concepts: the state is persisted, and contracts are deterministic functions (meaning all transaction validations that are performed by the contracts on different nodes and should produce the same result).
In addition, Corda introduces the concept of a Flow (kind of a distributed orchestration engine), which in Ethereum world would be similar to contracts calling each other (kind of choreography). But Corda flows are not deployed into all nodes, they are not part of the shared state, but rather represent standard JVM code, specific to individual nodes.

Technology Stack

Driven by the "re-use" principle, Corda is reusing existing storage, messaging and Java solutions. While blockchain platforms such as Quorum take a permissionless PoW framework such as Ethereum, and make it "enterprisy" by replacing the consensus mechanism, removing gas payments, introducing private transactions, etc., Corda takes the opposite approach. Corda, takes existing middleware technologies and applies the Bitcoin concept of UTXO and creates a new class of software that can be described as "a distributed business process and state management system". Corda achieves that through the use of commodity technologies such as relational databases for storage, and messaging for state replication and distributed business process coordination.
 Main components of Corda
High level technology stack:
  • It builds with Gradle, requires Oracle JDK 8, runs on Docker (and Linux on production).
  • A Corda node is a flat classpath JVM application (no Spring Boot, App Server, or OSGI container required).
  • Storage: relational database - H2, PostgreSQL, SQL Server, OracleDB.
  • Object-Relational Mapping: JPA - JBoss Hibernate.
  • Messaging: AMQP based - Apache ActiveMQ /Artemis.
  • Metrics: Jolokia
  • Other: Quasar, Kryo, Shiro, Jackson, etc,

Apache Camel Integration with Corda

Driven by my background in enterprise integration and interest in the blockchain, recently I created and wrote about Apache Camel connector for Ethereum and Quorum. In the same spirit of exploring enterprise blockchains, I created an Apache Camel connector for Corda. The connector uses Corda-RPC library and provides Camel producer/consumer endpoints to interact with a Corda node. The component offers a consumer for signing up and receiving events from a Corda node, and producer to send commands to a node.

Apache Camel connector for Corda
 Here is the full set of supported operations:

Consumer: vaultTrack, vaultTrackBy, vaultTrackByCriteria, vaultTrackByWithPagingSpec, vaultTrackByWithSorting, stateMachinesFeed, networkMapFeed, networkMapFeed, stateMachineRecordedTransactionMappingFeed, startTrackedFlowDynamic.

Producer: currentNodeTime, getProtocolVersion, networkMapSnapshot, stateMachinesSnapshot, stateMachineRecordedTransactionMappingSnapshot, registeredFlows, clearNetworkMapCache, isFlowsDrainingModeEnabled, setFlowsDrainingModeEnabled, notaryIdentities, nodeInfo, addVaultTransactionNote, getVaultTransactionNotes, uploadAttachment, attachmentExists, openAttachment, queryAttachments, nodeInfoFromParty, notaryPartyFromX500Name, partiesFromName, partyFromKey, wellKnownPartyFromX500Name, wellKnownPartyFromAnonymous, startFlowDynamic, vaultQuery, vaultQueryBy, vaultQueryByCriteria, vaultQueryByWithPagingSpec, vaultQueryByWithSorting.

To find more about Camel, and how it can complement Corda solutions, read the Camel Ethereum connector article linked above.

Conclusion

Public permissionless blockchains are facing serious technical challenges in the form of scaling, governance, energy waste and non-technical challenges with speculation, regulation, general usefulness and applicability. They have the noble idea of decentralizing everything but are yet to prove that the technology and the economic models are capable of delivering that vision.

On the other hand, private permissioned blockchains such as Corda, Fabric, Quorum are immune to these technical challenges as they target use cases with a smaller number of identifiable parties in regulated markets. Their goal is to improve and automate the existing business models of the enterprise rather than trying to discover brand new economic models. In a sense, permissioned blockchains represent the next generation cross-organization business process and data integration systems.

In this space, Corda is not revolutionary, but rather an evolutionary platform built on top of a well-estabilished storage and middleware technology ecosystems. The blockchain technology still has to prove itself, and building on top of a proven technology is the first step.

My next stop on exploring enterprise blockchains will be Hyperledger Fabric. Take care.

Tipping Points in Open Source

Follow me on twitter for other posts in this space. A shorter version of this post was originally published on Opensource.com under CC BY-SA 4.0. If you prefer, read the same post on Medium.


Over the last two decades open source has been expanding into all aspects of technology: from software to hardware, from small disruptive startup technologies to large boring enterprise software, from open standards to open patents. In this short post, I will try to call out a three tendencies that I think are reaching the tipping point in open source.

Open for Non-Coders

For good or bad, as the name open "source" suggests, this model has been primarily focused around the source code. Regardless of the intent or the believes, if we look at the open source communities, they are primarily composed of developers who are working on the source code. If we look deeper, the tools used in open source projects such as source control systems, issue trackers, mailing list names, chat channel names, etc - they all assume that the developers are the center of the universe. And that has caused a big loss. It is a big loss not having creative people, designers, document writers, event organizers, community managers, lawyers, accountants, and many others as part of the open source communities. We don't have such individuals actively participating in the open source communities because we don't have the processes and tools for such an inclusion yet. We need and want non-code contributions, but we don't have the means to measure their value, nor ways to reward their efforts in return. Reward by recognition from peers, by the community, by the employers, or anybody in general. As a result, it has been a lose-lose for decades. And we can see the implications in many open source projects with ugly looking websites, amateur logos, badly written and formatted documentation, disorganized events, etc. All of that leading for the so called "open source companies" filling the gap when it is inherently an open source problem.

The good news is that we are getting various signals indicating it is reaching the tipping point and a change is on the way:
  • Linus Torvalds apologises for his "bad behavior". While this is not an action specifically focused around non-coders, it is a very symbolic act in making "open source" a non-hostile place for less-technical contributors.
  • CNCF introduced the non-code contributions guide. In addition to showing how many ways there are for contributing to open source projects as a non-coder, this also sets the baseline for such contributions that other open source projects and foundations will end up following to keep up.
  • More or less around at the same time, ASF has been working in the same direction. There have been long discussions and we will have some concrete output very soon (that is ASF soon).
And there is a little known secret. One thing that non-coders (and new to open source) do not realize yet is that the easiest way to be recognized and become part of an established open source project is by performing non-coding activities. Nowadays, with complex software stacks, and tough competition, there is a pretty high bar for entering a project as a committer. Performing non-coding activities is less popular and it opens a fast backdoor to the open source communities.

Macro Acquisitions

Open source may have started in the hacker community as a way of scratching developers' personal itch, but today, it is the model where innovation happens. It is the model that even worlds largest software companies are transitioning to in order to continue dominating. And why is the enterprise so interested in open source? The "Open Source in the Enterprise" fee e-book I came across recently lists a few good reasons:
  • Multiplying the company’s investment through contributions.
  • Benefiting from the most recent advances and avoid reinvent the wheel in-house.
  • Spreading knowledge of the software and its broader adoption.
  • Increasing the developer base and hiring pool.
  • Upgrading internal developer skills by learning from top coders in the field.
  • Building reputation - developers want to work for organizations they can boast about.
  • Recruiting and retaining - developers want to work on exciting projects that affect large groups of people.
  • Faster startup of new companies and projects through open source networking effect, etc.
Many of these and other benefits of open source are recognized by large organizations which leads to even more open source adoption through company acquisitions. Building an open source company takes many years of effort in the open. Hiring developers who are good and also willing to work in the open, building a community around a project, and a business model is delicate effort. Companies that can manage that are particularly attractive for investment and acquisition as they serve as a catalyst in turning the acquirer an open source company at scale.
Open source related large acquisions of 2018
 Above is a list of the biggest open source software companies acquired in the current year. The list of open source companies and the following acquisitions is getting bigger every other day and this trend is only getting stronger.

Micro OSS Fundings

In addition to the macro acquisitions of open source companies, there is also an increase in the decentralized micro funding of self sustaining open source projects. 
On one end of the spectrum, there are open source projects that are maintained primarily by intrinsically motivated developers. On the other end, large open source companies are hiring developers to work on open source projects driven by company road maps and strategies. That leaves in the middle a large portion of open source projects that are not exciting enough for the accidental contributors, nor in the radar of the enterprise open source companies. In the recent years there is an increase in platforms for funding and sustaining these open source projects through bug bounties, micro payments, recurring donations, funding, subscriptions, etc. These open source funding platforms (which I have listed at http://oss.fund) allow individuals and open source users to take the responsibility of the open source sustainability in their own hands and pay the open source maintainers directly. It is the same open source model, but applied for value transfer rather than code contributions.

Open source contributor funnels

 The diagram above show the three intensive channels for open source contributors:
  • Hobbyist contributing to open source projects because of intrinsic motivations rather than monetary value.
  • Regular, planned and centralized subsidization by companies with open source business model (open core, SaaS, support, services, etc), monetizing the open source projects directly.
  • Irregular, micro, decentralized subsidization by independent open source users through OSS funding platforms.
While the hobbyist and hackers started the open source movement, it got quickly turned into an enterprise monetization model. Now, having something for the remaining open source projects is welcome.

What Can Blockchain Projects Learn from Open Source?

I've been involved with open source over a decade now. I've been part of small projects with innovative ideas which grew into large projects with solid communities. I've also witnessed how dysfunctional communities can suck the energy of projects for years. All that thanks to the open source development and collaboration.
In recent times, I'm active on the blockchain space as well: reading, writing, and contributing to projects. And I came to the conclusion that blockchain projects are startups with open development and open business models. And to be successful, the first and foremost, blockchain startups have to learn how to build communities the open source way.

Open source code

One of the fundamental premises of blockchain is decentralization and giving control and data back to the user. Such decentralization cannot be achieved without transparency and openness. If the source code is closed, that is no different to the centralized closed systems of today. Without making the code open, there is no way to read and confirm that a system is doing what it is promising to do. There are projects that are trying to avoid it, but even they recognize that the code has to be open to a certain level at a minimum. For example, Hedera Hashgraph (which is technically not a blockchain project, but a similar class of software) has said the code cannot be freely distributed (forked), but it will be open for review. That proves our premise: blockchain projects, first and foremost are open source projects. Whether this can be classified as open source according to "The Open Source Initiative" is not in the scope of this article. The point is, if the source code is not readable/verifiable, there is no point in having something run on a non-trusted blockchain platform.

Open runtime

In addition to the source being open, what differentiates blockchain from non-blockchain open source projects is that fact that for the first the runtime is open as well. An open source project can be developed in the open, but then run and consumed as an open core, as a service, or as part of a closed system. Public blockchain (not looking into private ones here) are permissionless, anyone can join and leave a network, anyone can run a node or two. It represents a trustless and borderless runtime with open governance.

Open data

Another distinct aspect of blockchain is that blockchain projects in addition to the open source code, open runtime, also have open data. Anyone can fork the code (the client application), fork the data  (the blockchain history) and start a new network. That ultimately makes blockchain projects the most open software systems ever existed. Open code, open data, open runtime, open business model, ensure openness in multiple dimensions.

Open business model

Blockchain startups are a very unique mix of open source development, and open value capture models, all blended into one at source code level. While a non-blockchain based open source project is typically used for creating value through collaborative development and open adoption, capturing value happens through a separate business model. The business model can be thought in advance or defined later such as SaaS, open core, subscription, etc. With the blockchain projects, the business model is described in a white paper, and the token model capturing value is implemented in the source code in advance. All that makes blockchain projects a unique blend of value creation and instant capture and distribution.

Why be so open?

Most of the blockchain projects are aiming to become some kind of platform or a hub with open standards and protocols that will attract and be adopted by the developers and consumed by users subsequently. The primary way these platforms and protocols attract developers is not through technical superiority over non-blockchain technology, but by the unique decentralization, characteristics achieved through openness in multiple dimensions. These platforms have to be open in order to become more attractive than the existing closed systems which already have all the developers and users on them. Being open is not only a prerequisite for its transparency, but also for its distribution and adoption. That is especially valid for projects which are aimed to be consumed as a platform or protocol by developers rather than end users. Open source is the primary way for developers to explore, learn and start using a project.

Isn't "open" a weakness?

There was a time when being open source was considered a dangerous act as a competitor could copy and steal the code or the ideas. The recent times proved that being open source is the primary way for developer adoption, especially for developer-centric platforms, tools, and libraries. But as we have seen above, blockchain is also open runtime and open data as well. Which means anybody can fork the code and the data and start a parallel network. That makes a project vulnerable to even more kinds of splits/forks and value grab. And we have seen this happened many times with the forks of the most popular blockchain networks such as Bitcoin and Ethereum. Yet, these projects are performing better than projects which are looking for ways to prevent forking but also lack the ability to attract followers. That is because being open is actually a sign of strength. If a network is so open and has survived forks and attacks, it makes its community only stronger.
We can observe the act of being open not only in projects, but also people and organizations. Today, people and organization rush into sharing and showing off their knowledge through open source code, conference talks, blogging, tweeting, etc. The innovation is happening so fast in certain areas that by the time somebody can understand and copy an idea, the inventor of the idea will have created the next one. And being a copycat in a winner takes all markets has a negative networking effect on community growth. In the journey to conquer the closed and centralized systems, being open is the primary weapon.

Hype is different than a community

I've seen many times, how successful Initial Coin Offering (ICO) investors measure hype around a project for an early investment. Typically such a measure works only when the early investment is accompanied by an early exit. In practical terms that means identifying the most hyped ICO, and selling all tokens as soon as it hits an exchange. Measuring such a hype is done by simple statistics around Twitter followers, Facebook followers, Reddit subscribers, Telegram users, etc. These metrics have a little value for measuring a community strength for the following reasons:
  • Metrics are artificially inflated with fake accounts, paid followers, subscribers, etc;
  • The ICOs themselves run airdrops campaigns and distribute tokens for following, subscribing, joining, etc;
  • These are the wrong metrics for measuring a developer-centric community;
What I mean by the latter is that an open source project that is going to be used by developers (as a platform, protocol, whatever) should measure developer activity, rather than airdrop hunter activities. None of the actions mentioned above are building stickiness in a project community. In fact, all of these activities are purposefully skewing the community metrics using temporary incentives.

Community over market cap

The Apache Software Foundation (ASF) is one of the biggest and oldest software foundations, home of hundreds of popular open source projects. And there, we (I'm a member, committer, and PMC there) have a very fundamental belief that says: "Community over Code". As a software foundation, we are all about code, and wouldn't have a reason for existing without the code, but this slogan actually codifies how we do things, and how we go about decision making. ASF is first a home for communities rather than a repository for code. The code is the by-product of a good and healthy community. And we first try to grow healthy communities united around projects.

If we look for example how an ASF project measures its quarterly activity and progress, that is by the number of mailing list subscribers, emails sent, issues opened/closed, pull requests created/merged, software releases done, committers and PMCs voted for. The last one is a very important long term indicator for the health of a project measuring the ultimate level of commitment of community members to the success of the projects. If you look at these metrics, these are all about activities performed by technical people rather than temporarily incentivised airdrop hunters. These activities are harder to fake as they require doing something for the project (usually consuming brain power and time) rather than clicking a like/follow button which easier to outsource.

A blockchain project has a more complex ecosystem than an open source project alone. There are developers, but also miners (or their equivalent for running the network), investors, and eventually users. Measuring only the developer activity won't be indicative enough for the full ecosystem, but focusing on the right metrics would be a good start.
In a similar spirit to the ASF's "Community over Code", I think the cryptocurrencies would benefit from "Community over Market Cap". A healthy community is a far more important long-term measure than a temporary large market cap. The price of a token/coin and its market cap can be artificially manipulated or temporarily affected by a bear market. A strong and healthy community can hodl and survive ups and downs. An unhealthy community, without any stickiness to the project would fall apart anyway.

Building communities the blockchain way

Are there good examples of building stickiness and community around the new blockchain projects? I have seen a few projects that have recognized the importance of the community from the very beginning and approached their token sale completely uniquely. These projects aimed to familiarizing the prospective early investors with the project goals, white paper, mission and not only ask for money. There are definitely more examples, but the projects with unique token sale processes I have seen are the following.
  • DFINITY project had a registration process that cost close to 10$. Then they gave that money back in the form of a swag and a free t-shirt. But it was a good method to get rid of the people who are there only for the noise and not even willing to commit 10 bucks.
  • QuarkChain ICO process had quiz with 25 not very simple questions. In order to join the token sale, one had to be part of their telegram channel from early days + have a good score on the quiz + pass the lottery. While the lottery and telegram channel components were already present in other ICOs at the time, the quiz actually forced candidates to find the answers in a short time, and learn about the project (that led to a blackmarket of quiz answers, but it was a nice attempt the least).
  • One of the best executions of community building during ICO phase has been of Mainframe. Mainframe run three crowdgift campaigns:
  • Proof of Being - where tokens where literally physically dropped from the air in certain locations around the world. To get tokens, one had to get to the meetup, meet the team and grab some tokens.
  • Proof of Freedom - where participants had to answer the question why Mainframe mission mattered to them, and submit the answers in any form: tweet, blog post, audio, video, drawing, etc. I also took part in it by writing a blog post.
  • Proof of Heart - where participants were asked to donate Ether which then went to a few non-profit organizations.
We can see how Mainframe used three different methods (each with its pros and cons) to build stickiness, awareness and community around its project and even managed to raise money for non-profit organizations.
Blockchain projects are especially sensitive to Metcalfe's law and their value is directly proportional to the size of its community. A token not used by anybody is worth nothing. A platform without developers is a zombi platform. Building a community around the crypto project is as important as building the platform itself, if not more. While the crypto world knows how to raise money, the open source world knows how to build communities. They can learn something from each other.

Follow me on twitter for other posts in this space. A shorter version of this post was originally published on Opensource.com under CC BY-SA 4.0. If you prefer, read the same post on Medium.

The New Kingdom Builders

Today’s developers aren’t just kingmakers; thanks to blockchain, they’re building their own kingdoms.

 The New Kingmakers

"The New Kingmakers" by Stephen O'Grady is a great book explaining why the developers are the most important assets a business has. In it, Stephen explains how developers are shaping products in new ways and organizations that understand and embrace the value of this shift will be the most successful in the years to come. It shows how IT decision makers aren’t making the decisions any longer, but the developers are. They have the power to make or break businesses, whether by their experience, their talent, or their passion. The book also has quotes from the legendary CEOs (the Kings - if we keep the book analogy) quantifying developers:
  • Steve Jobs who believed that an elite talent was 25 times more valuable to Apple than an average alternative.
  • Facebook's Mark Zuckerberg saying that someone who is exceptional in their role is not just a little better than someone who is pretty good, they are 100 times better. 
  • For Bill Gates, the number is 10,000 times better, etc.
In summary, every business is a software business and the developers are the most important constituency in it. The developers can make a company great, or break it apart. Developers can help a company conquer the world and make new kings (hence - The New Kingmakers).

The New Kingdom Builders

The era of the kingmaker developers has not ended. The urge to hire the best developers continues. The quest for engaging with developers, getting the developer mind-share and acceptances continues. The open cloud, open standards, free developer tools, open patents, open source are just the latest tools in this endeavor. The fact that open source has become an ubiquitous indication of how important it is to be accepted by developers.

While I agree with the written above, I think we have come to a new era where developers can conquer the world by creating their own kingdoms. Today there is technology that allows great developers to challenge established kingdoms, to build new kingdoms, and become kings themselves. This technology is the blockchain.

Open source is a collaborative software development and distribution model that allows people with common interests produce something that no individual can create on their own. It allows best ideas to spread openly and implemented collectively. It allows great developers to express their creativity and mastery in a subject. But open source doesn't allow capturing value. Open source produces value, then a business model built separately on top of open source captures the value.

Since there is no verified photo of Satoshi,
I went with Vitalik's photo
While open source is a value creation model, blockchain is a value distribution and capturing model. Open source is a development time characteristic, whereas blockchain is the runtime characteristic of a software. Developers that can combine both (not only create but also capture the value with code) can create new kingdoms out of thin air. That is possible because the value capturing in blockchain based projects is embedded in its core, it is part of the code, rather than being a distinct model build on top of the code and allowing somebody else to monetize it. Open source unites mainly the techies in creating something new, but the blockchain model can unite investors through ICOs to support best ideas, it can unite miners (not investors, not users, but a new class of actors in this ecosystem) to run the network nodes, it can attract the final consumers who care about decentralization and transparency. Blockchain brings a closure to the end-to-end cycle of building and then running software in the open i.e. from idea stage into mass consumption.

To be precise, open source is not a prerequisite for capturing value, but it helps for its creation. Also there are other technologies similar to blockchain, such as tangle, hashgraph, etc which follow a similar distribution and value capture model. The common characteristic among all of them is the fact that value capturing and distribution is embedded in the technology itself rather than being a separate model.

Here are a few of the pioneer kingdom creators of our time:
  • Satoshi Nakamoto started wrote the Bitcoin whitepaper. After 10 years, his/her/their vision is worth over $100 billion. More importantly, that vision gave the spark for many more to follow.
  • Vitalik Buterin created the Ethereum project which is worth over $1 billion now.
  • Daniel Larimer created Steem project worth over $250 million and then created EOS that is close to $5 billion.
  • Charles Hoskinson created Cardano worth over $2 billion.
  • Jed McCaleb created Ripple and then Stellar projects, worth billions...
Clearly, the list above contains only the geniuses and visionaries of this new world. These are the Linus Torvalds of blockchain and they are the only handful. But there are many who follow them. There are many who use the platforms created by these geniuses and come up with new business models and experiment in creating their own kingdoms. Some will fail, and some will succeed. But clearly, there is a new path for developers to conquer the world and this time for real.

Follow me on twitter for other posts in this space. This post was originally published on Opensource.com under CC BY-SA 4.0. If you prefer, read the same post on Medium.

Cloud Native Container Design Principles

Creating a containerized application that behaves like a good cloud native citizen and can be automated effectively by a platform such as Kubernetes requires some discipline. See below what it is. (Alternatively, read the same post on Medium)

Software design principles

Principles exist in many areas of life, and they generally represent a fundamental truth or belief from which others are derived. In software, principles are rather abstract guidelines, which are supposed to be followed while designing software. There are fundamental principles for writing quality software such as KISS (Keep it simple, stupid), DRY (Don’t repeat yourself), YAGNI (You aren’t gonna need it), SoC (Separation of concerns), etc. Even if these principles do not specify concrete rules, they represent a language and common wisdom that many developers understand and refer to regularly.

There are also SOLID principles that were introduced by Robert C. Martin, which represent guidelines for writing better object-oriented software. It is a framework consisting of complementary principles that are generic and open for interpretation but still give enough direction for creating better object-oriented designs. The SOLID principles use object-oriented primitives and concepts such as classes, interfaces, and inheritance for reasoning about object-oriented designs. In a similar way, there also principles for designing cloud native applications in which the main primitive is the container image rather than a class. Following these principles will ensure that the resulting containers behave like a good cloud native citizen, allowing them to be scheduled, scaled, and monitored in an automated fashion.

SOLID principles for cloud native applications

Cloud native applications anticipate failure; they run and scale reliably even when their infrastructure experiences outages. To offer such capabilities, cloud native platforms like Kubernetes impose a set of contracts on applications. These contracts ensure that applications they run conform to certain constraints and allow the platform to automate application management. Nowadays, it is possible to put almost any application in a container and run it. But to create a containerized application that can be automated and orchestrated effectively by a cloud native platform such as Kubernetes requires additional efforts. The principles for creating containerized applications listed here use the container as the basic primitive and the container orchestration platforms as the target container runtime environment.

Principles of container-based application design
Below is a short summary of what each principle dictates. To read in more details, download the freely available white paper from here (no signup required).

Build time:

  • Single Concern: Each container addresses a single concern and does it well.
  • Self-Containment: A container relies only on the presence of the Linux kernel. Additional libraries are added when the container is built.
  • Image Immutability: Containerized applications are meant to be immutable, and once built are not expected to change between different environments.

Runtime:

  • High Observability: Every container must implement all necessary APIs to help the platform observe and manage the application in the best way possible.
  • Lifecycle Conformance: A container must have a way to read events coming from the platform and conform by reacting to those events.
  • Process Disposability: Containerized applications must be as ephemeral as possible and ready to be replaced by another container instance at any point in time.
  • Runtime Confinement: Every container must declare its resource requirements and restrict resource use to the requirements indicated.
The build time principles ensure that containers have the right granularity, consistency, and structure in place. The runtime principles dictate what functionalities must be implemented in order for containerized applications to possess cloud native function. Adhering these principles, we are more likely to create containerized applications that are better suited for automation in cloud native platforms such as Kubernetes. Check out the white paper for more details.

The Cathedral and the Bazaar: Moving from Barter to a Currency System

This post was originally published as "How blockchain can complement open source" on Opensource.com under CC BY-SA 4.0. If you prefer, you can also read the same post on Medium.

Open Won Over Closed

The Cathedral and The Bazaar is the classic open source story written 20 years ago by Eric Steven Raymond. In the story, Eric describes a new revolutionary software development model where complex software projects are built without (or with a very little) central management. This new model is open source. Eric's story compares two models:
  • The classic model (represented by the cathedral) where software is crafted by a small group of individuals in a closed and controlled environment through slow and stable releases.
  • And the new model (represented by the bazaar) where software is crafted in an open environment where individuals can participate freely, but still produce a stable and coherent system.
Some of the reasons for open source being so successful can be traced back to the founding principles described by Eric. Releasing early, releasing often, accepting the fact that many heads are inevitably better than one allows open source projects to tap into the world's pool of talent (and not many companies can match that using the closed source model).

Two decades after Eric's reflective analysis of the hacker community, we see open source becoming dominant. It is not any longer a model only for scratching a developer’s personal itch, but instead, the place where innovation happens. It is the model that even worlds largest software companies are transitioning to in order to continue dominating.

A Barter System

If we look closely at how the open source model works in practice, we realize that it is a closed system exclusive only to open source developers and techies. The only way to influence the direction of a project is by joining the open source community, understanding the written and the unwritten rules, learning how to contribute, the coding standards, etc, and doing it yourself. This is how the bazaar works and where the barter system analogy comes from. A barter system is a method of exchange of services and goods for other services and goods in return. In the bazaar - where the software is built, that means, in order to take something, you have to be also a producer yourself, and give something back in return. And that is, by exchanging your time and knowledge for getting something done. A bazaar is a place where open source developers interact with other open source developers and produce open source software, the open source way.

The barter system is a great step forward and an evolution from the state of self-sufficiency where everybody has to be a jack of all trades. The bazaar (open source model) using the barter system allows people with common interests and different skills to gather, collaborate and create something that no individual can create on their own. The barter system is simple and lacks complex problems of the modern monetary systems, but it also has some limitations to name a few:
  • Lack of divisibility - in the absence of a common medium of exchange, a large indivisible commodity/value cannot be exchanged for a smaller commodity/value. For example, even if you want to do a small change in an open source project, you may still have to go through a high entry barrier sometimes.
  • Storing value - if a project is important to your company, you may want to have a large investment/commitment in it. But since it is a barter system among open source developers, the only way to have a strong say is by employing many open source committers and that is not always possible.
  • Transferring value - if you have invested in a project (trained employees, hired open source developers) and want to move focus to another project, it is not possible to transfer expertise, reputation, influence quickly.
  • Temporal decoupling - the barter system does not provide a good mechanism for deferred or in advance commitments. In the open source world, that means a user cannot express its commitment/interest in a project in a measurable way in advance, or continuously for future periods.
We will see below what is the back door to the bazaar and how to address these limitations.

A Currency System

People are hanging at the bazaar for different reasons: some are there to learn, some are there to scratch a developer’s personal itch and some work for large software farms. And since the only way to have a say in the bazaar is by becoming part of the open source community and joining the barter system, in order to gain credibility in the open source world, many large software companies pay these developers in a monetary value and employ them. The latter represents the use of a currency system to influence the bazaar. Open source is not any longer for scratching the personal developer itch only. It also accounts for a significant part of the overall software production worldwide and there are many who want to have an influence.

Open source sets the guiding principles through which developers interact and build a coherent system in a distributed way. It dictates how a project is governed, software is built and the output distributed to users. It is an open consensus model for decentralized entities for building quality software together. But the open source model does not cover how open source is subsidized. Whether it is sponsored, directly or indirectly, through intrinsic or extrinsic motivators is irrelevant to the bazaar.
Centralized and decentralized ecosystems supporting open source

Currently, there is no equivalent of the decentralized open source development model for subsidization purpose. The majority of the open source subsidization is centralized and monopolized typically by one company which dominates a project by employing the majority of the open source developers of that project. And to be honest, this is currently the best case scenario which guarantees that the developers will be employed and the project will continue flourishing. While a company is working on its paid software or services whether that is SaaS subsodizes open source development indirectly.

There are also exceptions for the project monopoly scenario: for example, some of the Cloud Native Computing Foundation projects are developed by a large number of competing companies. Also, the Apache Software Foundation aims for their projects not to be dominated by a single vendor by encouraging diverse contributors, but most of the popular projects, in reality, are still single vendor projects...

What we are still missing is an open and decentralized model that works like the bazaar without a central coordination and ownership, where consumers (open source users) and producers (open source developers) interact with each other driven by market forces and open source value. In order to complement open source, such a model also has to be open and decentralized and this is why the blockchain technology would fit here best. This would create an complementary ecosystem that flows subsidies from users to developers, but without a centralized and monopolized entity (such as an open source company). There are already successful open source cryptocurrency projects such as Decred, Dash, Monero, Zcash, that use a similar decentralized funding model where a portion of the mining or donations subsidy is used for their own development.

Most of the existing platforms (blockchain or non-blockchain) that aim to subsidize open source development are targeting primarily bug bounties, small and piecemeal tasks. There are also a few focused on the funding of new open source projects. But there are not many that aim to provide mechanisms for sustaining continued development of open source projects. Basically, a system that would emulate the behavior of an open source service provider company, or open core, open source based SaaS product company: ensuring developers get continued and predictable incentives, and guiding the project development based on the priorities of the incentivizers, i.e. the users. Such a model would address the limitations of the barter system listed above:
  • Allow divisibility - if you want something small fixed, you can pay a small amount without paying the full premium of becoming an open source developer for a project.
  • Storing value - you can invest a large amount into a project and ensure its continued development and ensure your voice is heard.
  • Transferring value - at any point, you can stop investing in the project and move funds into other projects.
  • Temporal decoupling - allow regular recurring payments and subscriptions.
There would be also other benefits raising purely from the fact that such a blockchain based system is transparent and decentralized: to quantify a project's value/usefulness based on its users' commitment, decentralized roadmap governance, decentralized decision making, etc. While there still will be user who prefer to use the more centrally managed software, there will be others who prefer the more transparent and decentralized way of influencing projects. There is enough room for all parties.

Conclusion

On the one hand, we see large companies hiring open source developers, and acquiring open source startups and even foundational platforms (such as Microsoft buying Github). Many if not most long-running successful open source projects are centralised around a single vendor. The significance of open source, and its centralisation is a fact.

On the other hand, the challenges around sustaining open source software are becoming more apparent, and there are many investigating deeper this space and its foundational issues. There are a few projects with high visibility and a large number of contributors, but there are also many other still important projects but with not enough contributors and maintainers.

There are many efforts trying to address the challenges of open source through blockchain. These projects should improve the transparency, decentralization, subsidization, and establish a direct link between open source users and developers. This space is still very young but progressing fast, and with time, the bazaar is going to have a cryptocurrency system.

Given enough time, and adequate technology, decentralization is happening at many levels:
  • The Internet is a decentralised medium that unlocked world's potential for sharing and acquiring knowledge.
  • Open source is a decentralized collaboration model that unlocked the world's potential for innovation.
  • And similarly, blockchain can complement open source and become the decentralized open source subsidization model.
Follow me on twitter for other posts in this space.

The Rise of Non-Microservices Architectures

(This post was originally published on Red Hat Developers, the community to learn, code, and share faster. To read the original post, click here.)

This is a short summary of my recent experiences with customers implementing architectures similar to microservices but with different characteristics in the post-microservices world.

The microservices architectural style has been around close to five years now, and much has been said and written about it. Today, I see teams deciding not to follow strictly certain principles of the "pure" microservices architecture and breaking some of the "rules". Teams are now more informed about pros and cons of microservices and take context driven decisions respecting team experience, organizational boundaries and accepting the fact that not every company is Netflix. Below are some examples I see in my recent microservices gigs.

No premium in advance

Teams (composed of devs, ops, testers, business analysts, architects, etc) are becoming more and more aware of on the premium they have to pay for the privilege of going into pure microservices based architecture. A typical Java based microservice running on Kubernetes (the most popular microservices platform) will require: a git repository, maven module, a collection of tests (unit, integration, acceptance), APIs, maven artifacts, container images, configurations, secure configurations, build pipelines, design, documentation, etc. At runtime, it will require CPU, memory, disk, networking, metrics aggregation, log aggregation, database, endpoints, service mesh proxy side car, etc. Also a collection of Kubernetes objects: container, volume, configmap, secret, pod, service, replica set, deployment, etc. Navigating and managing tens or hundreds of these artifacts puts a serious burden on everybody in a team. No surprise that recently Thoughtworks announced they are not intending to put microservices architecture into adopting phase of their technology radar in a foreseeable future.

Raison d'être

Considering there is a price per service (not envisaging all the hidden premium), rather than the original "start small, start with few hundred lines of code", systems start as one service, as a mono repository as long as it belongs to one team. Then for every service, there is a clearly identified reason with benefits justifying its existence as an independent microservice. That is a mandatory existence check before carving a service as a standalone. Talking about reasons, below are a few very valid reasons.

Breaking the bounded context

While the most talked method for decomposition into microservices is decomposition by bounded context, in practice there are many more reasons for creating microservices: decomposing by maturity, decomposing by data-access pattern (read vs write), decomposition by data source (rather than partitioning a data source per microservice, create a microservice per data source), aggregation for a derived functionality (create an orchestrating service for a few other services), aggregation for client convenience (such as backend for frontend pattern), aggregation to aid system performance, etc.

Shared data sources

One of the fundamental principles of microservices is that every service has a separate data store. While in theory, this principles makes perfect sense, in practice, for brownfield projects, it is the hardest part about microservices, and not always worth the effort. That is especially true for integration projects where the data source is typically owned by a different team or company and cannot be partitioned to start with. It is still possible to benefit from having independent services sharing the same data store, by acknowledging its future constraints caused by the data source level coupling.

Less inflated expectations

The good news is that teams are now making more informed decisions rather than blindly trusting microservices conference slides. In regards to Gartner's Hype cycle, that means, after a couple of years of "Inflated Expectations", the microservices architecture is heading (down) towards "Trough of Disillusionment" stage where expectations are more aligned with the real benefits.
Hype cycle
From here on, the future is full of enlightenment and productivity. Unless another cycle starts (such as serverless) before we rip the benefits of this one.

Mutated microservices

Microservices favor event-driven interactions and choreography over orchestration to decrease service coupling. But at the same time we have seen projects like Cadence by Uber, Conductor by Netflix created specifically to orchestrate distributed long-running services as an alternative of the choreography approach.

Bernd Ruecker has done a very good review of using events, orchestration and workflow engines in distributed system and analysing their real against perceived benefits.

In a different post titled Microservices in a Post-Kubernetes Era, I also described what are the changes in the microservices architectural style driven purely by Kubernetes and the cloud native primitives.

There are also others who also wrote about non-microservices architectures such as self-contained systems, miniservices, goodbye-microservices that are better alternatives to pure microservices in certain contexts. These are all good reasons for breaking away from pure microservices principles whenever the context requires it.

Conclusion

The best architecture is the context-driven architecture where you take a well-understood architecture and adapt it your needs. You question every principle, every rule and are not afraid of breaking away from some of the prescriptive elements as long as you understand and accept the consequences. A good analogy is “The map is not the territory”. If the architecture is the map, the context is the territory. Let me know which microservices rules you are breaking and it works for you even better. Be brave.

From Agile to Serverless

If you prefer, read the same post on Medium

Looking back. And forth.

The microservices architecture was born as a technological answer for the iterative Agile development methodology. At the early days of microservices, many companies were doing a form of Agile (XP, Scrum, Kanban, or a broken mixture of these) during development, but the existing software architectures didn't allow an incremental way of design and deployment. As a result, features were developed in fortnight iterations but deployed every six to twelve months iteration. Microservices came as a panacea in the right time, promising to address all these challenges. Architects and developers strangled the monoliths into tens of services which enabled them to touch and change different parts of the system without breaking the rest (in theory).

Microservices on its own put light into the existing challenges of distributed systems and created new ones as well. Creating tens of new services didn't mean they are ready to deploy into production and use. The process of releasing and handing them over to Ops teams had to be improved. While some fell for the extreme of "You Build It, You Run It", others joined the DevOps movement. DevOps meant, better CI/CD pipelines, better interaction between Devs and Ops and everything it takes. But a practice without the enabling tools wasn't a leap, and burning large VMs per service didn't last for a very long. That led to containers with the Docker format which took over the IT industry overnight. Containers came as a technical solution to the pain of microservices architecture and DevOps practice. With containers, applications could be packaged and run in a format that the Devs and Ops would both understand and use. Even at the very early days, it was clear that managing tens or hundreds of containers will require automation, and Kubernetes came from the heavens and swept all the competition with a swing.

Now, we do Agile development and use microservices architectures. We have one-click build and deployment pipelines with a unified application format. We practice DevOps and manage hundreds of microservices at scale, do blue-green and canary releases with Kubernetes. This was supposed to be the time of long-lasting technological peace where we can focus on the business problems and solve them. But it was not.

From Agile to Serverless and Beyond

With Kubernetes, Devs can develop and Ops can deploy in a self-service manner. But there are still only one or two company-wide Kubernetes clusters that occasionally run out of resources. The cluster is now the center of the universe. And it is even more critical than the monolith was before as it runs the build server, and the git server, and the maven repository, the website, MongoDB, and a few other things. So a promise for a serverless platform came from the clouds. It came even before the containers reached into production. With serverless, every service would have its own cluster, a cluster where resources are limited only by the limit of the credit cards. It is a platform to which you tightly couple, and bet the existence of your company for the price of going faster. So it started all over again.

About Me