Agile Integration Software Rescues the Dead Side of Bimodal IT Architecture

I’m sure you’ve heard of the recent high profile discussions about how to bring much-needed innovation to mature companies that are carrying the comforting ballast of old fashioned infrastructure.  That infrastructure is the greatest impediment to agility and innovation (unless you count the people and culture that go along with it). I first heard about “bimodal” at a local Gartner program a few weeks ago and found the concept both thrilling and disturbing.

The idea of bimodel divorces the reliable and stable back-office (Mode 1 “Core”) from all that is innovative (Mode 2 “Innovation”). This means that innovation is explicitly separate, with new, presumably agile, infrastructure to create new lines of business for generating new revenue streams, and to provide more contemporary modes of interacting with consumers and employees.

I’m a little concerned that a bi-modal declaration promotes an easy way out for Mode 1 laggards. Their management no longer have to worry about modernizing or even interacting with Mode 2. In approaching the problem this way, we are continuing to be enablers of the infrastructure and its management who are addicted and afraid to even try to wean from their brittle, ancient technologies and methodologies.  I suspect that part of the reason that we got to this point is their continued abdication of decisions and recommendations to vendors and consultants on the dead side of bimodal. With advancement generally limited to creative marketing and re-messaging the same 20-year-old technologies and ideas, the name-brand consultants in enterprise IT nominally grab the buzz but only deliver it on the periphery.

I definitely agree that relying on the Mode 1 (reliable, stable) IT is highly unlikely to bring  significant innovation, and I also believe that the best way to get real innovation underway is to completely separate it out, with different people, skill sets, management, and objectives. But if we imagine how this will play out, there is likely to be a complete bifurcation where the innovative side never is able to leverage the back-office functions. They will inevitably invent their own (less reliable and less stable) versions of back-office. What happens then? Mode 1 eventually dies on the vine? We regress to pre-1980 basics?  Business in general accepts worse performance on the backend functions?

(You didn't really try clicking that button, did you?)

Makes more sense to contact Stone Bond Technologies.

It’s probably obvious that my take is that BOTH modes should advance aggressively, ‘though I do believe the innovative side should be unencumbered by the Old World. Mode 1 management should take this as a gauntlet to push hard to replace their integration infrastructure with Agile Integration Software, such as Stone Bond’s Enterprise Enabler®, which is a proven enterprise-ready framework that boosts agility, interacts with both Mode 1 and Mode 2 applications and data, and offers up to 90% reduction in time to production along with a huge reduction in tech debt. That is what is needed for companies to survive and enjoy a competitive advantage in the coming years.

You Mode 1 people do have a choice. You can continue as is and sit by waiting for your inevitable demise, or you can be the hero that solidly bridges Mode 2 and Mode 1. We are seeing this successfully implemented by forward-looking CIOs. So, find a leader and press GO!

The Hyper-Converged Integration Platform

Frankly, I find it amazing that it has taken so long for the concept of convergence of integration to become a topic of discussion. In fact, it’s mind-boggling to me that almost all of the manifestations of integration functionality appeared on the scene as islands, with delineation only just now beginning to blur. ETL tools do ETL; EAI tools do transactions; SOA does web services; DV tools do data virtualization, and so on.

Stone Bond’s Enterprise Enabler® came on the scene ten years ago or so, as a platform with metadata shared across all “patterns,” rendering the classic integration patterns  essentially moot. If someone stepped into data integration, contemplating it as a general problem to be solved, they might identify these various patterns, but they would also quickly see that they are not mutually exclusive. There is clearly more overlap in the demands across these patterns than an observer of the evolution of data integration tools would support.

The providers of integration tools were much too hasty in solving the problem: not considering anything beyond the particular integration style at hand. It’s reminiscent of the custom programmed applications that are designed for a specific customer. Eventually it dawns on someone that this solves a problem for a large set of businesses. What happens? This nice, clean solution gets bells, whistles, and tweaks for the second customer and... Voila! It becomes a (usually lousy) “Product” that requires months of customization to implement. Now, think about how different the Product or the integration tools  would have been, had the initial design taken into consideration the superset of potential users and uses.

 

Click to Enlarge Picture

 

Whether you are physically moving data, packaging integrations as web services, or generating virtual data models for MDM or for querying, there are some critical key elements that are necessary to have at the core of the integration platform.

Let’s go back to the idea of hyper-converged integration platform. It is only possible if the overall design takes into account the essence of shared functionality across the characteristics that will, or may, be needed in every pattern. Even if you don’t know what the patterns will be, you do know that the platform should always be able to, for example,

• Access data and write data to any kind of endpoint - live
• Federate and align that data across multiple sources, whether they are the same or totally different
• Align and  transform to also ensure the data makes sense to the receiving endpoint, whether physical or virtual
• Apply business logic
• Validate and filter data
• Manage various modes of security
• Apply workflow, error handling, and notification
• Package the integration in many different ways
• Scale up and out
• Reuse as much configuration as possible

A hyper-converged integration platform has all of these capabilities, and as a single platform, all of the objects configured are reusable and available for more universal value. For example, an ETL that brings five data sources together and posts to a destination (no staging), can also be reused as a Data Virtualization model for live querying on demand.

Whatever mental picture you have of integration toolsets,  try thinking instead about an Enterprise Nervous System, with data flowing freely throughout the company exactly how and when it is needed.

Enterprise Enabler is a hyper-converged Integration platform, perhaps because the overall design came about from years of contemplating the essentials of integration as a whole. It’s easier to start out with a universal consolidated solution than to back into it from ten different, fully developed tools.

An integrated set of tools is highly unlikely to become a Converged Integration Platform, and will forego the powerful agility and elimination of tech debt that Enterprise Enabler can bring.

The Power and Potential of Data Virtualization

Let’s be blunt here. If you are not seriously planning to launch Data Virtualization initiatives in the coming months, you are likely to lag in supporting your company’s ability to remain competitive. You simply cannot sustain the historic infrastructure costs of all the ballast along with the time to implement and maintain integration middleware as it has been for years (decades, really). Data Virtualization is clearly a game changer.

There is a whole spectrum of Data Virtualization (DV).  I see a significant amount of confusion about data virtualization once you get past the basic concept that data from multiple sources can be combined and made available as if it were a single source. Most people are extremely narrow in their view of the scope, value and uses of DV.  

• DV can be about creating a virtual enterprise data model for querying
• DV can be about creating virtual MDM definitions
• DV can be about simplifying the layers of services for any application
• DV can be about eliminating the majority of staging data bases in your organization
• DV can be about defining exactly the domain required by an application or portal
• DV can be about significantly improving your time to value for any integration you need
• DV can be about simplifying most any IT project
• DV can be about Business Analytics and Business Intelligence
• DV can be about Big Data

Data Virtualization is poised to drive a fundamental shift in the way IT departments and solution providers address the classic messy challenges of data integration and data availability. Think in terms of shedding the layers and layers of legacy accommodation that have been necessary simply because it has been “impossible” to align disparate data.

A word of caution:  if your Data Virtualization platform requires your data to be in a specific format (e.g., relational or XML) in order to include it in the data virtualization, then you are defying the concept altogether. You are having to do things like move your operational data from instruments to relational databases, and you have to put your SAP data into some form that the DV platform can deal with.  That’s a far cry from accessing live data!

Enterprise Enabler® deals with all types of sources live directly from the source. And, the concepts of DV are also applied to other patterns, for example, eliminating staging for ETL.

Cache as Cache Can

Caching is one of those afterthoughts, when you know you have a great solution, but you start wondering about performance. Since caching is about moving data at varying speeds, it is (or should be) an inherent feature and responsibility of any integration solution. You will find that a truly Agile Integration Software, such as Enterprise Enabler, makes it easy to configure a wide range of models of caching, and to adjust as your requirements change.

Agile Integration covers everything from ETL through near real time Bi-directional Data Virtualization (DV), all with federation at the core, so caching can be implemented anywhere, end-to-end, in the data flow cycle.

The Continuum of Caching

According to Wikipedia, cache is a “component that transparently stores data so that future requests for that data can be served faster.” I think of it as being any data store, however static or ephemeral, however Big or small, and whether the cached data is exactly in the source form, perhaps to be federated on the way out, or federated already as the endpoint needs or the Master Data form, ready to go on to its destination, or somewhere else in the flow of the data. The specific subset of data to be cached should be optimized to ensure the greatest efficiency, minimal size, and highest reusability. The transparency comes in because, in the big scheme of things, the destination, the consumer, or the workflow steps never need to know the data is not all coming live from the original sources.

This is where data federation and Data Virtualization add to the flexibility of caching. Agile Data Virtualization supports cache as one of the sources, so there could be DV involved to create the cache, whether in-memory, on disk or in a database, and then that cache can be used as one source in a federation that is delivered either on-demand or event-triggered.

Today, most people talk about cache as being refreshed as opposed to accumulating a history, however with all the options that can be configured, this is actually a  realistic and sometimes useful consideration. You can see that the possible combinations are many, clearly enough that one must be careful not to get tangled up, and not to lose sight of the original objectives of caching! 

One could easily argue that caching is more like ETL than like Data Virtualization, however DV often requires caching more than other integration patterns, since the uses generally expect rapid, “live” data, without latency. When the rubber meets the road, in many situations, caching is the only way to ensure that a DV solution with many users does not bring the source applications “to their knees.” This is why Agile Integration Software, which combines all the integration patterns, solves Data Virtualization problems better than pure DV platforms.

What do you need to determine before you configure caching?

·         Which data to cache

·         Why you selected caching this particular data

·         Where to cache – memory, disk, database, etc

·         How often to refresh – schedule, event, as soon as available

·         Where in its path to cache – directly from source, partially processed, before or after federation, endpoint ready, as part of a Master Data definition

·         When to release from cache- as soon as read, as soon as a particular set of consumers have read

·         Is the cache subject to bi-directional data flow

When should you plan to Cache?

First of all, keep in mind that if you don’t identify your caching needs up front, with Agile Integration Software, you can easily add it as your traffic grows and the parameters get to point where it’s needed.  Particularly when you are using Data Virtualization, and are hitting backend source systems live at each request, you should take a close look at the needs and best approaches to caching. You should consider caching in situations where:

·         You are concerned that too much traffic hitting mission critical or any sources could adversely impact the performance of those systems.

·         You are concerned about the response times for end users.

·         You need to have the same value throughout a process where you might be accessing it multiple times

What to Cache?

·         Data that doesn’t need to be real-time

·         Data that you want to ensure the same snapshot is used for different things

·         Data that changes so slowly that having it real-time doesn’t matter. You could refresh the cache once an hour or day or month, even.

Agile Caching

Agile Integration Software offers a wide range of options for caching, with ease of configuring even complex caching patterns without custom programming. With the ability to select full data sets, specific fields,  mixed in-memory and on-disk caching, and all combinations, including conditional full workflow-driven caches,  great architecting doesn’t have to be constrained by what is practical to implement.

I Hate Data

I Hate Data. Loathing is the Mother of Invention.

I hate data. I’ve always hated data. As a programmer right out of school, I worked in “technical programming” as opposed to “business programming.”  Business was about accounting mostly, and technical about, well, technical stuff. One would think that technical would involve data and numbers while business would be about less precise things. Nevertheless, the fact was that it was the business side that worried about data and about uploading huge amounts for backup every night. Therein, of course, lies my initial dilemma. It happened that I was fortunate enough to focus on my passion, which was computer graphics. This was back in the days when we were figuring out how to make circles look round and to get rid of the “jaggies.”  At the time, I had little respect, if not outright disdain for business, but then, I was much younger and living in my 3-d graphics world.  We on the technical side focused on what you could program computers to do. I wrote code that made every graphics device I could find sing and dance, and the input devices like mice and early tablets and joysticks, too.  Imagine the thrill of making one of the first joysticks move cursers and move through 3D spaces rendered on 2D screens! Compare that to the dubious activity of staring at tons of numbers on countless reports. Those guys debugged things by studying numerical data, while I had the joy of detecting bugs with the screen looking like a war zone of odd shapes, colors, and flashes, or simply by crashing the computer altogether. Who wouldn’t choose the latter?

Sometime later, after my phases of programming robots and working with pattern recognition algorithms, I began working with refinery modeling and programs to perform mathematical optimization (LPs). These programs combined data from the refinery operations, from laboratories, inventories, planning systems and such, along with current economic information.  Lots of data was involved, but the huge impediment was getting the data from ten different sources in a manner that aligned all of it meaningfully. Often the end users would manually enter some of the data and they would pretend that running a very precise optimizer would give just as good results with some of last month’s data. Wrong! Why couldn’t the data just work smoothly in the background?

I hate data. It’s hard to deal with. Too many problems. That’s not what I want to focus on – there are much more interesting things to think about. That’s why Enterprise Enabler® just had to come along – so that I wouldn’t have to deal with all the idiosyncrasies of disparate data. It was very selfish of me. Let a computer handle all that craziness. Hide everything behind the scenes and automate everything that has to be done more than a couple of times.  But then Enterprise Enabler unexpectedly swept me into “business” and all kinds of things I never imagined. I have to be careful, now that the headaches of data are managed, I might start liking it. I can’t admit it, but I’m starting to think data may be what it’s all about. Big Data, little data, virtual, bi-virtual, octy-virtual, and numberical, too.

Do you have the guts to be a hero? Take the Agile Integration Plunge.

Come on now. This day and age, you business leaders are still beholden to your IT organization. You are the cleverest business person you know. You have successfully negotiated the biggest acquisition in the history of your company. Besides that, you are on the leading edge with all the latest innovations in video, phone, and personal computer and tablet technology. You even rigged up a sensor to notify you when the bird feeder in your yard is empty. How can you not be frustrated that you just can’t seem to feel as confident about your IT infrastructure?

There are a number of reasons why some corporate IT tools and infrastructure have lagged generations behind the advances of consumer technologies, but that’s for another blog. The important message here is that finally, the next generation integration platforms have matured and are ready to turn the ship. Change has been incredibly difficult, and large companies, in particular, have been unable to respond quickly to opportunities because IT could not keep up.  We have reached a point where those that adopt agile integration software will have a clear competitive advantage. We are seeing that transformation take place, and not at the speed of classic IT. At the speed of change.

I heard at a recent Gartner conference, the keynote speaker announcing that with the new imperatives of agility and supporting the “Nexus” of data integration demands, the Big Players will be DOA (“Dead on Arrival”). You must take charge and get on board with the next generation, even though you are faced with resistance from people whose IT knowledge may intimidate you.

How can you tell if it’s time to take up agility?

1.       Not everyone is working off of the same numbers

2.       Long wait times to get access to new data that you need

3.       Manpower costs for building integration are exorbitant

4.       More than 40% of the costs of new projects are for integration and data access

5.       Data you get is not up-to-the-minute

6.       You have business processes that are highly manpower-intensive

7.       Your partners and customers are not getting information as fast and in the forms they would like

8.       You may be moving forward with Cloud, Big Data, others, but the rest of the IT team can’t keep up with the demands of everyday project work

One tell-tale sign is that the data warehouse is the center of the universe, but is less than agile. Data warehouse is good for historic data, but not for real-time or close to live data. It’s taking an extra step in the storage process that requires the data to be staged when it’s needed.

How can you get the ship turning? Institute just a couple of new guidelines:

“All new integration must:

1. Leverage Agile Integration Software

2. Use Virtual Data Federation wherever data needs to be combined across more than one data source

3. Use Data Virtualization for all “on-demand” integration (proactively asking for data when you need it so you can get live data from the sources as opposed to getting stale data)

You will be met with resistance from all sides, but you will gain strong supporters quickly after the first jack-rabbit projects come in ahead of time and below budget. There are plenty of causes of resistance:

•          The Big Vendors are always a safe choice
•          Fear of change; stubbornness; laziness about learning new things
•          There is a relationship between how much time it takes for consultants to do something and the amount of money they earn
•         Anything new is up for more scrutiny than going with the tried and slow

And you can certainly augment the list in the context of your business.

The promises of Agile Integration Software like Enterprise Enabler are real and are being realized in many companies. The technology is definitely Enterprise-Ready, and not to be relegated to small projects with small potential benefits.

Do yourself and your company a favor. Take the plunge. Stare down fear with the guts that got you here in the first place.

Agile Big Data is Coming

Agility with Big Data?

Shiny but not Agile, perhaps destined to never be Agile.

Just look at ETL. Never, never agile.

So Clunky. After all these years, why is it so Clunky? So Clunky it’s fragile.

Big Data follows in the footsteps.

Big feet. Big hype, Big opportunity

for discoveries otherwise unfathomed.

Great programmers performing great feats,

Coding, coding new horizons… Open Source assist.

No hope for Agility.

Where are the trailblazers of Agility?

Ah, yes!

They’ve conquered ETL, EAI.

They’ve conquered Data Virtualization.

Agile Big Data is coming.

Are great programmers actually an impediment to Agility in complex software problems?  I‘m not talking about Agile development, but rather Agile solutions. Great programmers like to be on the leading edge of the latest new shiny trend. I remember the rush of programmers to work on the Y2K-driven Enterprise Application Integration and ERP rage. Now we’re all over Big Data.

It takes time for the market, business, and technologists to evolve the thinking about what the Shiny thing really is for, what it means, and what the technology requirements are. So, who jumps in first? The really good programmers who want to blaze the trails and solve the problems first-hand. Unfortunately, the early players can’t benefit from the evolution and maturing of the Shiny and what the requirements will really be. By the time it’s clear what it takes to genericize the problem, the Great programmers have moved on to the next trend, and are certainly not going to step back and solve the problem again in a tool or platform that hides all the repetitive work behind the scenes. 

Product companies then step in to harvest the rest of the hype curve. Leveraging their Big Name, they make plenty of revenues on hard-coded, specialized solutions where custom coding is accepted as the only way to solve the problem. In my view, a timeless benchmark for Agility is the minimization of custom code. The more coding involved in generating a solution, the farther away that solution is from Agile.

The growing love affair with Open Source code is a huge step backward for the cause of Agility. Programmers love coding. I, for one, love coding, too. But I HATE having to code essentially the same thing over and over, with just a couple of tweaks difference each time. This is exactly what integration is all about: lots of small (very important) tweaks to the same code over and over.

That is why Enterprise Enabler exists. That is why we hide all the technical details behind the scenes of our agile integration software. Any time a programmer has to do essentially the same thing more than a handful of times, it is automated with tweaks configurable in a UI. Programmers should be programming exciting, innovative new things, not laboring over repetitive, boring scripts and code changes, and maintenance.

We’re applying the same philosophy to automate Big Data integration and analysis.