Thursday, February 26, 2015

The Hyper-Converged Integration Platform

Frankly, I find it amazing that it has taken so long for the concept of convergence of integration to become a topic of discussion. In fact, it’s mind-boggling to me that almost all of the manifestations of integration functionality appeared on the scene as islands, with delineation only just now beginning to blur. ETL tools do ETL; EAI tools do transactions; SOA does web services; DV tools do data virtualization, and so on.

Stone Bond’s Enterprise Enabler® came on the scene ten years ago or so, as a platform with metadata shared across all “patterns,” rendering the classic integration patterns  essentially moot. If someone stepped into data integration, contemplating it as a general problem to be solved, they might identify these various patterns, but they would also quickly see that they are not mutually exclusive. There is clearly more overlap in the demands across these patterns than an observer of the evolution of data integration tools would support.
The providers of integration tools were much too hasty in solving the problem: not considering anything beyond the particular integration style at hand. It’s reminiscent of the custom programmed applications that are designed for a specific customer. Eventually it dawns on someone that this solves a problem for a large set of businesses. What happens? This nice, clean solution gets bells, whistles, and tweaks for the second customer and... Voila! It becomes a (usually lousy) “Product” that requires months of customization to implement. Now, think about how different the Product or the integration tools  would have been, had the initial design taken into consideration the superset of potential users and uses.

 
Click to Enlarge Picture
 
Whether you are physically moving data, packaging integrations as web services, or generating virtual data models for MDM or for querying, there are some critical key elements that are necessary to have at the core of the integration platform.
Let’s go back to the idea of hyper-converged integration platform. It is only possible if the overall design takes into account the essence of shared functionality across the characteristics that will, or may, be needed in every pattern. Even if you don’t know what the patterns will be, you do know that the platform should always be able to, for example,

  • Access data and write data to any kind of endpoint - live
  • Federate and align that data across multiple sources, whether they are the same or totally different
  • Align and  transform to also ensure the data makes sense to the receiving endpoint, whether physical or virtual
  • Apply business logic
  • Validate and filter data
  • Manage various modes of security
  • Apply workflow, error handling, and notification
  • Package the integration in many different ways
  • Scale up and out
  • Reuse as much configuration as possible

A hyper-converged integration platform has all of these capabilities, and as a single platform, all of the objects configured are reusable and available for more universal value. For example, an ETL that brings five data sources together and posts to a destination (no staging), can also be reused as a Data Virtualization model for live querying on demand.
Whatever mental picture you have of integration toolsets,  try thinking instead about an Enterprise Nervous System, with data flowing freely throughout the company exactly how and when it is needed.

Enterprise Enabler is a hyper-converged Integration platform, perhaps because the overall design came about from years of contemplating the essentials of integration as a whole. It’s easier to start out with a universal consolidated solution than to back into it from ten different, fully developed tools.
An integrated set of tools is highly unlikely to become a Converged Integration Platform, and will forego the powerful agility and elimination of tech debt that Enterprise Enabler can bring.