The Dirty "Little" Secrets of Legacy Integration

The more I learn about integration products on the market, the more astounded I become.  Fortune XXX companies buy politically “safe” products from IBM, Informatica, SAP, and Oracle and launch into tens of millions of dollars’ worth of services to implement them. Egad!   They’d be better off with holes in their collective heads!

Remember the childrens’ story, The Emperor’s New Clothes? Isn’t it time for someone to come out and tell the real story?

Shouldn’t an enterprise grade integration solution simplify data integration instead of creating Rube Goldberg projects?  Does it really make sense to have to send personnel to intense training classes that take months?

Nine things that astound me about other enterprise grade integration products, along with how Enterprise Enabler makes it easier, ultimately reducing the time-to-value by 60% to 80%.

1.     Robust transformation engines are mostly non-existent. This means that anything beyond the simplest relationships and formulas must be hand coded. That’s a huge impediment to fast time-to-value. Enterprise Enabler has a powerful transformation engine that captures mapping, validation, federation, and business rules as metadata instructions through a single GUI without requiring programming. 

2.      Transformation engines cannot interact with the endpoints in their native state. This means there has to be a complete integration to get each source’s data into a compatible format before transformation. Enterprise Enabler's transformation engine receives data from multiple sources in their native formats, combines (federates) them live, and delivers results in any form, or by query.  

3.       Data federation is not applied to ETL, DV, or other modes directly from the sources. Each source is handled individually, and posted to a staging area in memory or database. Enterprise Enabler brings the data together logically "on-the-fly" without storing it in memory or anywhere, and passes it through the transformation engine to the destination or the virtual model. Sometimes for performance select data is cached and refreshed as required. 

  

4.       Many, if not most, endpoints are accessed via some standard like ODBC as opposed to using their native mode. This means that it is not possible to leverage special features of the native application, and negates the possibility of being a strong player in IoT. Enterprise Enabler accesses each source in its native format, enabling the execution to leverage various features specific to the endpoints at execution. Because of the robust proprietary endpoint connectors, called AppComms, Enterprise Enabler easily incorporates legacy ERPs with electronic instrumentation in a single integration step.

5.   Data Virtualization does not support “Write-Back” to the sources. (probably because of #4) Enterprise Enabler supports authorized end-user aware CRUD (Create, Read, Update, and Delete) write-back to source or sources when data is changed or entered from the calling application or dashboard. 

6.     Implementing an integration solution is a matter of working with a number of mostly stand-alone, disconnected tools each of which imposes rules for interaction. Enterprise Enabler is a single platform where everything is configured, tested, deployed, and monitored, with design-time validation, and embedded C# and VB code editors and compilers for outlier situations. A developer of DBA never needs to leave the Integrated Development Environment. 

7.      Various data integration modes (e.g., ETL, EAI, DV, SOA) are delivered by separate tools and do not offer reusability across those modes. With Enterprise Enabler, all modes, including complex integration patterns are configured within the same tool, leveraging and re-using metadata across modes. This means that an enterprise master virtual model can be easily re-purposed, with all the same logic, as an ETL. 

8.     Further, Enterprise Enabler has a data workflow engine, that serves as a composite application builder, with full visibility to the active state and process variables defined throughout the execution. 

9.   Finally, Enterprise Enabler's Integration Integrity Manager monitors endpopints for schema changes at integration touchpoints. When found, IIM traverses the metadata to determine the impact and notifies the owners of those objects.  

In short, none of the legacy integration platforms can hold up to the demands for agility that are essential for maintaining competitive advantage.

Value In The Integrated Metadata Stack

If you're using or looking at Agile Integration Software (AIS), the chances are you are discovering that there's metadata for everything that's not tied down (and even for those that are). Think about the conceptual epitome of integration. There have been various analogies over time, conjuring up a brain with information flowing (ENS - Enterprise Nervous System), or the flow and pervasiveness of water, and more recently we hear about the fabric. A few years ago I coined the term "synchronapse" to represent the idea of information flowing intelligently, like synapses firing anywhere as needed. Of course, that never took off - new words are fun, but an uphill battle.

I like the fabric metaphor. Good word: the fabric of nations, the geologic structure of a roc; something that represents the essence and the underlying structure; maintaining integrity but flexibly, so that if one point on the fabric moves, the fabric shifts to accommodate that change.

The only way to capture and control the fluid movement of the fabric and be able to ensure that the enterprise can quickly respond to internal and external changes, is to describe everything that can change with metadata. That's a cornerstone philosophy of AIS. Whether the fabric needs to adjust for planned business initiatives or unforeseen external events, the supporting integration infrastructure is adjusted via metadata changes.

Notwithstanding security controls, the full metadata stack must be available to any object or process in the environment, so that conditions at one point on the fabric can affect change in another. That is at best very difficult if each component of your integration stack has its own independent set of metadata. With AIS, as you build your integration with GUI tools, the various layers of metadata and the inter-relationships across the layers is being captured and managed automatically.

What's the value of an integrated metadata stack?

• Reusability of metadata across the stack
• Example: a for-purpose data selection from a source (e.g., customer demographics) can be reused as needed for any map. Also rules and formulas are reusable, along with processes and many other objects.
• At run-time, any business rule can take action based on current values of any metadata
• Example: a different transformation map can be executed depending on customer ID
• Any layer can incorporate other metadata by reference
• Example: an enterprise master data model can reference all the metadata that is needed to bi-directionally access and federate the appropriate sources

This is definitely one of the cool things about Agile Integration Software, possible because it's an IDE, all under one roof.

How's Your Tech Debt?

The term "tech debt" has been around for a number of years, and resurfaces periodically. The early usage was a way to talk about the cost of short-sighted programming, bugs, and badly architected solutions. The metaphor gave programmers a financial analogy for contemplating the value of good programming practices, and for non-programmers to appreciate the need for plenty of time to do development well. If you don't do it well the first time, you will need to go back and patch it up, creating more potential break points and incurring increasing tech debt.

Clearly, this is a micro-view of tech debt. If you zoom out a bit, you start seeing debt in not just the quality of the starting point, but also in the ability to keep up with change over time. Eventually, even the best programming needs to be retired and replaced. Stepping back, you start looking beyond programming, to the quality of the product's design and architecture, and the infrastructure and hardware it is tied to.

Tech debt comes from:

• Poor programming practices followed by patches
• Not keeping your assets maintained
• Isolating from the rest of the world
• Not keeping up with universal trends

Over ten years ago,Y2K forced the greatest surge of eliminating technical debt, of course at a huge financial and operational cost. Now, that "new" software and infrastructure is nearing fifteen years in place, and it is incurring its own technical debt for its own reasons. In particular, the EAI that emerged during that Y2K time period was badly needed, but may have been an afterthought for ERPs, hastily architected, and starting out with a tech debt burden.

What about the next fifteen years? You've mastered SOA, but will there be something new that sends it the way of EAI? We're just getting off the ground with SaaS, but maybe in that time the pendulum will begin to swing back to new takes on old trends.

If you look at the dimensions that are generating tech debt, you'll see:

• Quality
• Time
• Trends

Quality is clear; time inevitably invites new requirements, new infrastructures, as the old age out, forcing change on the other old-timer components; trends are a bit more insidious.

What do you do when the tech debt increases at an unsustainable rate? At that point of no return, you replace it. That point comes earlier if you have not kept up with basic investments in maintenance. It's inevitable that you will need to reformulate your strategy and tactics and get a fresh start.

How is your tech debt doing? Is it getting more expensive to keep the status quo than it would to replace or stepwise replace what you have? One of the key technologies you need to look at that can give immediate benefit in managing your tech debt is Agile Integration Software.

SaaS Vendors Take Note: You Can Operate in the Cloud but Not in a Vacuum

This week I watched a webinar that proposed that more responsibility of integration with SaaS applications needs to be carried by the software provider. Analyst Dana Gardner of Interarbor discussed this topic in the webinar and on his blog http://tinyurl.com/4fgyw5j. Workday recently announced its cloud-based integration services as part of its SaaS ERP offering, stepping up to the plate to provide tools that will ensure that customers can integrate to their own software. They very well may be setting a trend that other ISVs will need to keep up with.

It is clear that, while the convenience and specialization of cloud-based solutions opens a new opportunity for businesses to trim down their infrastructure and the associated maintenance costs and effort, it doesn't eliminate the need for integration. Instead, it calls for a new paradigm for data integration.

Unfortunately, the heavy middleware in enterprises today will have trouble keeping up with the increasing business demands for agility. Change is tough when no one wants to touch the integration for fear of breaking something.

With the growing dependence on cloud-based software, your customers need a new generation of integration that can streamline data flows as their business processes move freely across legacy, clouds, and collaboration portals. If you are a SaaS vendor you need to seriously think about how you can rise to the occasion for your customers. The better you do it, the happier your customers will be.

Workday addressed the challenge by buying an integration software company! That's not only an expensive way to go, but it means that now, in addition to domain expertise and software development teams for their ERP, they must also maintain expertise and developers in the rapidly-changing integration space. That's probably not a prudent business approach for most SaaS vendors.

The alternative is to embed, rebrand, and/or offer Agile Integration Software, such as Enterprise Enabler® in your offerings www.enterpriseenabler.com. That way you get all the benefits without the headaches. I don't know if Workday's integration fits the Agile Integration Software model (see http://tinyurl.com/3yv85bw), but with AIS, even a SaaS software vendor is able to offer integration across cloud apps, and also incorporate on-premise backend legacy systems as well as pass data to and from your customer's SharePoint installations, on-premise or hosted.

Web Services and APIs Just aren't Enough

We've had a number of questions lately about how AppComm technology is different or better than Web services or APIs (Application Programming Interfaces). That's probably because we have been working on packaging Enterprise Enabler® various ways for low end, not-so-demanding point solutions. For example, if you want to access Salesforce.com from a WSS application, you could use the Salesforce.com AppComm with or without another AppComm, and in a few minutes configure a bi-directional ADO.net driver. When you want to get data from Salesforce.com, you just read from and write to the ADO.net object.

The AppComm technology assures simplicity beyond any API or Web service connectivity by leveraging, behind the scenes, all the knowledge of the APIs as well as any necessary coding, scripting and deciphering. The data is simply provided for selection and mapping, eliminating the hard-coding and difficult maintenance necessary when programming to APIs and Web services directly.

Well, that's the whole point! The fundamental philosophy behind Agile Integration Software http://agileintegrationsoftware.blogspot.com/2010/05/characteristics-of-agile-integration.html design hinges on encapsulating absolutely the most intelligence possible behind the scenes so that the tough, complex, or tedious work is only done once. If I have to study the API for SAP's RFCs (Remote Function Calls) to build connectivity components, why not design the connectivity as an AppComm, which means that if I do it well once, no one will ever have to read the API specs again!!! (sorry, for the triple exclamation points - my mother would not have approved, with her proper and perfect usage of the English language, but in this case, I think they are well deserved). Not only do you never have to study the API, but you don't have to understand the prerequisites, assumptions, and in what order all the calls must be made.

Off the shelf, AppComms encapsulate all that knowledge and programming in a flexible and reusable manner, so you can get going quickly. It discovers all the schema information specifically for the particular instance at hand, and allows you to select the data of interest. You can then configure the mapping without programming, which you would need to do without an AppComm.

The truth is, contrary to coerced popular belief, the fact that there exist Web services for accessing an application's data does not mean they are easy to use. And since you are programming instead of configuring, if there's a change to your application's data schema, you have to go back into the code and make changes in order to access new fields, or make sure that your integration doesn't crash because expected data is missing. AppComms know when the schema changes, and alert you to reconfigure.

There are clearly time and cost savings when utilizing the AppComm technology compared to accessing the raw Web services. When you combine the AppComm with its Agile Integration platform, you have the synergy for coordinating data access from multiple applications simultaneously, aligning and transforming the information on the fly and passing it to the destinations without staging.

AppComms:Removing the Splints from the Octopus

AIS (Agile Integration Software) conjures up the image of information flowing freely wherever it needs to be whenever it needs to be there, aligning and merging with other information as needed by each endpoint person or application. The information is not yesterday's wilting snapshot of the world, or dusty data that has struggled and morphed through five different formats along the way. Think clean, precise, and now.

I contend that the ultimate culprit that works against AIS is the classic "Adapter," along with its cohort "standards." Now please understand that they are not equally culpable, and perhaps neither is, in the perfect world. Unfortunately, as far as I can tell, we haven't reached Utopia yet. An integration model that adopts a specific standard simply pushes the responsibility of dealing with the tough issues outside its world for someone else to handle. That's why Adapters exist; they must get data from whatever/wherever/however it is into a specific destination's requirements ("standard"). The Adapter has no visibility or responsibility to adjust to run time situations. Its job is simply to get predefined data from one application or format to another.

AppComms have a different responsibility; that is as an "application specialist" that can interact with the endpoint application or database at run time as it is instructed by the integration coordinator. Each record or block of data from that endpoint that is needed somewhere in the integrated environment is provided by the AppComm on request, and similarly, data is posted to the destination as requested. AppComms know how to auto-discover the schema for that application or data store, whatever "schema" means for that application; they know how to perform all CRUD (Create, Read, Update, Delete) operations as appropriate (or not) for the specific endpoint application; and they are responsible for generating the reusable metadata defining desired data across scenarios. The specialization of an AppComm is not for the specific instance of an application, but reusable across all instance of that system. For example, a Salesforce.com AppComm will be able to work with any configuration of Salesforce, as opposed to assuming a specific schema.

The elimination of the classic Adapter is like taking splints off an octopus. The integration coordinator can orchestrate a huge range of concurrent information flows performing all kinds of tasks. Multiple live AppComms can be coordinated to access data in an order and timing to resolve cross-application virtual relationships at run time. Imagine eliminating all those staging databases! Adapters simply can not possibly offer the same fluidity and flexibility for integration as AppComms.

From Dashboard to Control Center with AIS

What's wrong with dashboards today? Nothing really, but stretch the limits by adding a few more dimensions of reality and they can fly! Maybe that's the difference between a car's dashboard and the dashboard in the cockpit of an airplane. One just displays the situation and the other allows you to interact and control the airplane.

Executive and management dashboards have become pervasive and invaluable contributors to many businesses' decision making by providing access to a broad range of key corporate and specialized data in easily assimilated visual screens. Today's dashboards are mostly like cars. If you're lucky, you get real time data displayed, along with relevant analysis so that you can make operational or strategic decisions. When you make decisions in a car, you react by letting up on the gas pedal or stopping at the gas station. With business dashboards, you make decisions and then have to go to another system to update data, or send an email or otherwise convey or initiate execution of those decisions. Why can't you act on your decisions from the dashboard? Why do dashboard have to be "read-only?"

With data coming from a range of sources, the prevailing approach to accessing the data from multiple backends is for the dashboard vendor to tell the customer that it’s in their court. (Just get all the data in one database so they can read from it and provide awesome analysis and visualization. I wish I had a dollar for every sale that was lost on that alone! )

Even if it were easy to do, that staging database is the inherent problem with dashboards that limits their usefulness. Whether the vendor develops and populates it or the customer does, there is a specs effort, a design effort, and ongoing implementation effort. Populating it requires a huge programming effort. But the biggest problem is that you simply cannot display current, live data, and you cannot really even think about the data being updated in the dashboard and sent back instantly to the backend systems.

Instead, if you use an Agile Integration Software to implement the dashboard integration, you can automatically have secure, bi-directional data flows. You can eliminate all of the overhead associated with the database design, implementation, and maintenance. That means a rapid dashboard implementation project, and the ability to turn the dashboard into a control center where decisions can not only be made, but can be executed. Maybe you can't even call it a dashboard anymore!

ISVs are rebranding and packaging AIS with their products or embedding into their products using an Agile Integration Software ADO.Net driver. See Stone Bond's Enterprise Enabler

Coercing the Mind Shift to Comprehend the Paradigm Shift

The mind certainly is stubborn. Something about re-directing those synapses to another mindset is astoundingly challenging. Just like the synapses connect "sphere" and "bounce" to the concept of "ball," "enterprise integration" is only recognizable as "integration" if it looks like a heavy, time-consuming, difficult, expensive endeavor. So what if a cubical ball could actually bounce even better than a spherical one? Even if I see it, I can't make the "connection." Well, I don't know much about synapses, but I sure have had lots of experience trying to initiate the plasticity to reformulate the way people think about integration!

Recently we presented a bit about our technology to a handful of people who are extremely well-informed about the integration space. We planned to hit a few points that we are pretty sure can't be done with other products. We zeroed in on one specific point we thought would amaze anyone who really knows the current state of integration and what is possible and what is not. This is a capability that is a natural fall-out of our new paradigm: we showed how to create virtual relationships across totally different systems even faster than you would define a relationship across tables in a single database.

In ten seconds we showed creating a virtual relationship between Microsoft Dynamics CRM and a SQL database. In that ten seconds, all the run time components were generated behind the scenes, so we ran it. This scenario happened to be sending data to an out-of-box SharePoint 2010 web part; it could have gone to any other application, dashboard, web service, or pumped to a messaging system .

We showed how an end user, from a single SharePoint web part (screen), can see live data from multiple applications, aligned and transformed as appropriate for their usage. As if that weren't enough, we showed updating a couple of fields in SharePoint, and the backend systems were immediately updated!

No staging of the data, no copy made at all, anywhere. The data was accessed, aligned, displayed, changed, and updated back in the backend systems. Our product, Enterprise Enabler, eliminates a HUGE amount of time, manpower, and cost by eliminating the need for a staging database any time you need data merged from two or more disparate systems. The systems could be SAP, Salesforce, XML, spreadsheets… it doesn't matter. So, no one needs to design a database model that is everything to everybody. No one needs to build the database, and no one needs to maintain it.

A response came from our audience that many enterprise integration platforms can do this! I would love to do a "side-by-side" bake-off with the cited competitors, Websphere or Tibco, any day. I am extremely confident that we would probably have to go home before their years of work are done. If they even can do it.

In the end I have to write the response off to the discipline's persistent twenty-plus year track record that has forced consistent synapses dancing around in the same circle, convincing everyone that there is just no other way integration can possibly be done. If Websphere, Tibco, Informatica, and others can't do it, we'll just pretend they can!

Is "Agile Integration" an Oxymoron?

Most people would agree that "Agile Integration" is an oxymoron. The heavy footprint, heavy customization, and huge amounts of interdependence across an integrated enterprise absolutely negate the possibility of Agility! But we discuss Agile Enterprises as if you could instantly have an intelligent SOA implementation that detects and adjusts to business changes, maybe even anticipating the evolution of the business. What are we talking about, anyway? All the SOA initiatives I've heard about are isolated subsets of business solutions that impose the web services environment (which takes lots of design, development, and definitely anticipation of everything you will want to do in the future). The "agility" offered is limited to a specific, known set of options.

If we want to reflect the Agility that business wants and needs in this age of merger, acquisition, and increasing regulation, we need to be able to almost instantly support the underlying infrastructure implications. Buy a company and you buy a new ERP that manages and records that business. What can you do to absorb the new business? Switch them to your SAP system? Even if that would make your company Agile, you'll be frozen in place for minimum a year or two working to get them up. That's the only way many people, companies, and consultants can think about to achieve that objective. If you buy lots of companies and perhaps sell them again, you MUST be able to very quickly get the key information aligned and the systems sharing appropriate information.

And how can you ever get the latest live information to dashboards instantly, when it comes from a multitude of sources and it needs to be transformed and aligned in order to make sense? Pulling it together via a staging database simply doesn't cut it.

Agility is essential, but it's simply not achievable with the classic approaches to integration.

Agility will only come with a complete mind shift that reinvents the essence of integration. Stone Bond Technologies thinks about integration totally differently with its Enterprise Enabler system. By automating every step along the way and generating the run-time components behind the scenes, the time to implement and maintain an integration is reduced by up to 90%. I know, "90% - you are either crazy or lying!" Well, hear me out a bit. We really do have an off-the shelf product that is the product of 100+ man years of development and is built on a new paradigm. I'll discuss the fundamentals of it in the coming entries.