Agile Integration for the Nexus

Last month I attended the Gartner Application Architecture Development & Integration Summit. The Keynote and theme throughout the conference was the Nexus of forces, Social, Cloud, Mobile, and Information (Big Data). A crucial supporting theme was the necessity for Agile Integration Technologies. The big integration players, they say, will be “DOA” (dead on arrival) as the importance of agility spills over from business to technology in a bigger way than ever.

Agile integration is all about communicating across all kinds of endpoints in all kinds of formats with all kinds of integration patterns, and all adjusted to make sense together. It is about getting it in place quickly and it’s about integration that adapts to the changing environment, devices, and fast pace business changes. The archaic picture of integration- making data move from one place to another- simply won’t do the trick.

Most integration products focus only on Social Media OR On-Premise OR Cloud OR mobile OR Big Data. They tend to be based on a model of SOA OR ETL, OR Data Virtualization, OR Messaging Bus. This means that you need to have multiple integration products, each handling its own specialty.  Even with the big integration players, who would blatantly use AND instead of OR, you are jumping from one environment to another, since their claim to breadth comes largely from acquisition. Their “AND” consists of the camouflage duct tape and baling wire utilities provided.  Suppose you have need for an integration pattern that combines ETL, Cloud, Mobile, and Data Virtualization?  It’s really unfathomable that you could tie integration from multiple platforms together in a way that anyone could call “Agile.”

What does agile integration really mean to your business and your tech team?  It means being able to support agile decisions and an agile enterprise, with significantly less resources and at a lower cost.  

Here are some requirements to look for to assess the agility of an integration product, and examples:

1)      Agility requires:  Reusability

       Reusable connections, business logic, schemas… most everything.

2)      Agility requires:  Rapid time-to-value

       Generate integration definitions, test them, and deploy quickly to begin getting immediate business benefit                                                                                                                                                

3)      Agility requires:  Instantly adapting to changing requirements

       Add a new source, change the business rules, move some data to cache, and anything else in the configuration must be easily changed in minutes. Deploy to cloud when needed, or as Mobile accessible integration

4)      Agility requires:  Scaling up or down quickly and easily

       Fast-growing businesses need scaling up for growth or acquisition, and scaling down as needed for distributed use

5)      Agility requires:  Connectivity to Everything

       Single platform connects and reuses integration definitions for electronic instruments, enterprise ERPs, cloud apps, spreadsheets, relational databases, b2b standards, etc.

6)      Agility requires:  Simple construction of Complex Integration Patterns

       Add caching, data synchronization, or ETL to an on-demand data virtualization with write-back.

7)      Agility requires:  Deploying integration logic multiple ways

       One Button Deployment and Hosting of Data Virtualization as web service, REST, ODBC, JDBC, OData, Ado.net connection string, SharePoint BDC/BCS etc.

8)      Agility requires:  Single product; Single platform.

       No jumping in and out of multiple environments and trying to make them work together

9)      Agility requires:  Completely metadata driven; Single Metadata Stack

       Means external programming is reduced to nothing; heavy reusability; visibility of the state across the whole integration at any time

10)   Agility requires:  Configured, not coded.

        Enough said.

Agility means:  Minimal Tech Debt. All of these points greatly reduce tech debt by virtue of agility. It just doesn’t get any better than that.

Stone Bond’s Enterprise Enabler® is THE Agile Integration Software.

The Good Ghosts of Data Virtualization

I’m a datum, tra-la, tra-la, just humming along, happier than most of my colleagues, now that Data Virtualization has come to my life.  I’m what they call a Master data, or rather a source of record, which has been quite a challenge, since I used to get cloned over and over, morphed, and turned upside down. That took lots of time and seemed to be necessary because they couldn't quite get to ME, the original: unvarnished, and clean as a whistle. My colleagues still deal with this and are copied to staging databases all the time, leaving many places where versions reside. There is a constant complex of synchronization and updates across all the databases. Just imagine all the time to build all the comings and goings! Often it’s not even clear what the original, real value is or where it all began. Of course, this is done through such a gallimaufry of tools and custom coding that the information they convey is old by the time it is used. And everyone knows the greater the gallimaufry rating, the higher the tech debt accumulation. Bad stuff.

With this new Data Virtualization approach, I get to stay right where I am, and whenever called upon, I send a fresh virtual ME instantly. I say “virtual” because there’s no copy made of me, only a ghost of me is transmitted, aligned with other data, usually to a browser or to feed some analytics algorithm. Here’s where my psychiatrist has to get involved, because I have this existential dilemma. Does the ghost that’s passed forward actually exist? Does a datum exist if it’s passed virtually? Well, at least it can’t be stolen like copies of data can. I stay up nights worrying about my colleagues who have to be physically copied, or moved altogether, sometimes getting a little beat up, to some cloud in order to be used by a SaaS application. I never have to move because my ghost is passed directly to the cloud when it’s needed, never taking residence there. And one of the really cool things is that if a user of the browser decides that my value needs to be updated, they change it in the browser and send the ghost back to me with the new value, assuming they have permissions to do that.

Sometimes when I’m needed virtually, there are too many calls for ghosts, and the phantom of the opera sings one note too high and crashes my host software system. Fortunately, the masters at Stone Bond, with Enterprise Enabler®, not only have the speediest development environment on the planet, they also have a way of caching my ghost along with others in memory since I really don’t change that often. In the background, the cache gathers fresh ghosts whenever needed. When my call comes in, my ghost flies from the cache of ghosts and finishes the journey, combined with some other ghosts coming live from other systems. The phantom just hums along and the host system lives happily ever after. 

The Synergy of SharePoint 2013 and Data Virtualization

It’s a powerful combination: Data Virtualization and SharePoint. SharePoint 2013, the most recent release, supports broader-than-ever opportunities for data integration and data-centric implementations. SharePoint has evolved to include its availability as a service in the cloud and a hybrid architecture, along with its fully on premise model.

The availability of these alternate architectures is a compelling argument for adopting SharePoint for many types of applications beyond the historical content management uses. The rich Business Connectivity Services (BCS) provide a range of features that are data-centric, elevating SharePoint to another level as a platform for applications, dashboards, and BI tools.

As implementations mature, though, certain requirements are coming to the forefront. In particular, the reality of most applications is that data needs are not limited to a single data source, or even a single data source type, which is a limitation of not only SharePoint, but of the third-party vendors that provide connectivity. The potential of the built-in powerful BCS is not fully leveraged, and the end result is either limited scope of applications or the need to build a staging database either on premise or in the cloud, the latter of which brings additional security concerns for many customers.

The ideal use of SharePoint leverages BCS to access data live from the original sources, validates the data, presents it to the end user, or application for action or editing, and pushes changes back to the original sources without ever saving it anywhere. End user security ensures that end users only see or edit data to which they have access.

This is what Data Virtualization (DV) brings to the table, and Stone Bond’s Enterprise Enabler is a full-scope integration platform, with data federation at its core. The single Integrated Development Environment (IDE) for development, testing, deployment and monitoring, is built on and fully extensible with Microsoft .Net. Within a few minutes, one can configure and generate data virtualization in all of the data access modes supported by SharePoint, OData,SOAP, REST, ODBC, ADO.Net, and Enterprise Enabler's Custom Connector.

Convergence is, at Best, Asymptotic

No, not asymptomatic. Asymptotic. “Convergence” is a term we hear these days in IT. The convergence of Data Integration, in particular, is the one I care about. In the analysts’ vernacular, converged integration seems to mean a product, company, or platform that handles all modes of data integration – ETL, EAI, ESB, DV et al.

By definition, “convergence” means a coming together, which clearly implies the parts started from other places and came voluntarily or were coerced into coming together. Just looking at history, companies like IBM, Oracle, and Informatica absorbed outside companies and products to nominally have a product suite with all the modes. Here’s how it works: Need ETL? Buy Ascential, rename it, give it a massage with hot towels, then say, “Voilà! Voilà! Voilà!” and your product now covers ETL, too! Or, take a Data Virtualization product, write a bunch of code, and again, “Voilà!” and your product covers ETL, too.

With all due respect to the analysts, the word convergence may describe the reality of most companies incorporating more and more integration modes, but keep in mind though, that in mathematics convergence means getting closer and closer but never quite getting there: asymptotic [translation: Close, but no cigar!].

I am certain that the analysts do not mean Convergence in the mathematical sense. The term is quite useful for establishing a powerful vision of dramatically reduced time-to-value, clean architectures, flexible integration patterns, and highly streamlined change management and maintenance over time.

If you read my last blog http://tinyurl.com/lmwtzth you probably recognize that there is a significant difference with Enterprise Enabler®. It was designed from the ground up with a powerful core that handles all the common functionality of the range of modes, with implicit data federation across disparate sources (databases, electronic instruments, spreadsheets, data warehouses, ERP systems, cloud services and so on). 

 That core is the common root of all data integration modes, a bit like the trunk of a tree that has any number of branches and leaves, instead of trying to converge a bunch of branches by stuffing them all in an opaque vase and pretending like they have a single trunk. That spells trouble, and doesn't even come close to asymptotic. Probably not asymptomatic, either. 

On-Demand vs. Event-Driven. Who Cares?

Let’s say I’m an integration. (Don't laugh!) What is this “integration”  that I actually am? Well, I have been configured for a specific purpose or set of uses, and I know exactly how to access each source, not just the class of source, like SAP, web service, Oracle database, or Excel spreadsheet, for example, but I know where each resides, what its security access is, and what subset of information is of interest for this purpose. I also know what data to filter out from each, what validation needs to be done as I grab the data, and how to make whatever multiple sources are involved meaningful together.

I cross-reference key fields, change units of measure, align formats, and apply functions to aggregate, analyze, or otherwise manipulate the data so it makes sense for this set of uses. And if I’m passed a query along the way, I definitely know how to deal with providing a subset from the domain I’m designed for.   I’m pretty smart, don’t you think? Not to mention self-centered. I’m an Active Integration from Enterprise Enabler® that has been configured in fifteen minutes.

What do I care how I get initiated, or by whom? Certainly not me! I’m just standing by the phone waiting for a call. In the end, whatever terminology you use, I’m the core of “Convergence” of integration modes. Just tell me, and I’ll do the heavy-lifting of an ETL, physically moving data from several heres to a there, or serve data up virtually as a data set through a web service. Put me on a schedule or maybe have me do my integration whenever the ice cream truck comes. (That’s my favorite kind of event to respond to!).  So you see that even if it’s a user refreshing a browser page, or querying a specific subset of data, it’s all the same to me.

Sometimes I inadvertently find myself in the middle of a brawl of my counterparts from other worlds, who only know one way of operating. You know, ETL, EAI, SOA, or Data Virtualization. They think they’re the best at what they do, which is always only one of those. They have to contort themselves to the point of pain in order to play in more than one arena.  I, on the other hand, am so flexible I can do any of the above and touch my toes (with and ice cream cone in my hand) without grunting. If it starts to drip, I just cache it somewhere. And by the way, I'll challenge my counterparts to their specialty any day!

It’s true, though, that sometimes I have my own internal existential contemplation, wondering if I really exist. All my counterparts call me names, saying I’m nothing but a bunch of metadata.  L  Nevertheless, I am the Master that can be used anywhere in any mode, maintaining standardization and agility at the same time. It’s a bit like yoga.

What is Data Virtualization?

“Virtualization” is everywhere but nowhere. The term is virtually ubiquitous. The first use I remember when computers got into the picture was “virtual reality,” when we computationally rendered 3D worlds in 2D, complete with lighting models and all that. The good thing is that all those complicated algorithms are now encapsulated for cool things and used by beginner gamers. In those days, we had to actually calculate every pixel ourselves. But I digress.

First, let’s clarify that “virtualizing data” means putting in the cloud or elsewhere in order to eliminate some of the hassles of its existence and maintenance. That has nothing to do with data virtualization, which is a term that I believe is still evolving.

Data Virtualization, according to Rick van der Lans, who literally wrote the book, is “the technology that offers data consumers a unified, abstracted, and encapsulated view for querying and manipulating data stored in a heterogeneous set of data stores.**”  

As the discipline matures, he is expanding his view, as in his new white paper, Creating an AgileData Integration Platform Using Data Virtualization. Definitely recommended reading. 

The “unified, abstracted, and encapsulated view,” from his original definition is the core concept, in my opinion, of data virtualization. In other words, there is a mechanism to bring together, or “federate,” virtually, data from many sources in a way that is useful. This means that the data is federated without creating a physical or cached staging database, but is aligned, transformed, and made available for use. So, for example, you may have a SharePoint BCS application that needs data from SAP, Oracle, and Salesforce.com. Data virtualization will provide a mechanism to merge all of the data into the form necessary for the end user’s interaction in SharePoint. The data is federated “on the fly” and delivered virtually to a web page, on-demand upon refresh of the screen. Think about the security of the backend data that has been accessed…it never actually moves from its original source! Data virtualization also includes writeback to the sources (with end user security, but that’s for another blog) so that an end user can, for example correct his phone number or address, sending it as an update directly to the backend source. )See more examples at http://tinyurl.com/a3wkffc)

You can see that this description expands the definition to include any sources, not just data stores, although the focus of most data virtualization products is BI, in which case, that limitation it makes sense. The BI view of using data virtualization usually is with respect to federating relational databases for the sole purpose of querying. The tools that were designed assuming that constraint have some difficulty accommodating the expanding definition. 

In addition to evolving from federating data stores to federating any kinds of disparate sources, data virtualization is shedding the concept of “on-demand” only. Now federated data is not just available by web services, ado.net, ODBC, JDBC, etc, but for any type of data integration, such as ETL, EAI, etc. 

In fact, it is the “data virtualization” concept of federation that becomes the kingpin for “Convergence,” as Gartner is wont to say, of all integration modalities in a single toolset, sharing metadata and business rules across all. 

**Rick F. van der Lans, Data virtualization for Business Intelligence Systems, Morgan Kauyfmann, 2012

The Domain, It’s Plain, Is Mainly Under Strain

Data Virtualization is a technology for ETL, EAI, as well as Business Intelligence

I’m enjoying talking with enterprise architects about Data Virtualization (DV) in their companies, how they are fitting it in their overall architecture and what their expectations are. What I’m seeing is the strong tendency toward thinking of DV as a tool for Business Intelligence and less as the basis for “convergence” of all silos of integration. The latter is the topic of conversations I had not too long ago with several Gartner analysts.

Typically data virtualization is considered only for “on-demand” uses. This makes sense if your fundamental premise is that DV is essentially the same having a database, which is never pro-active about delivering data. That means that you must apply technology on top of it to provide queries and act on them. Why not apply the same federation to other forms of data integration? On-demand or delivered physically to an endpoint, the approach eliminates staging databases and other overhead.

Enterprise Data Virtualization Domains

A few of the enterprise architects I have spoken with recently are very clear that agile data federation is one of the main reasons data virtualization is compelling. However, it is difficult – arguably impossible- to define an enterprise data model that is “everything to everybody.” DV brings the opportunity to think “agile,” and I just don’t remember any “single universal” approach to anything in IT that has ever succeeded for more than a few cool success stories about a handful of companies.

Just because a data model is virtual, that definitely does not mean that it is not brittle. In the beginning, you may have a clean model, but the natural progression of additional and changing requirements will eventually yield the traditional issues of staging databases and data warehouses and lead to instability because of the intense level of interdependency across the model. That “extra mile” of custom programming required to make the data exactly relevant to each end use also carries risk and overhead to maintain.

Smaller domains for Data Virtualization tend to increase agility

Consider instead defining reusable rapidly-built, for-purpose federations that include appropriate data validation and conversion to exactly the requirements of the end usages, and that deliver to the endpoint either physically or on-demand as a service. From that perspective, it is not necessary to always define the universe as a single model, and it is not necessary to use different products and development approaches whether you want to use the same data federation to move data (ETL), to handle operational transactions (EAI) with a surrounding orchestration, or on-demand as a web service or other calling mode.

Perhaps “reusable rapidly-built, for-purpose” seems a bit oxymoronic. Think instead of the concept of Master Data, where each definition is constructed in a way that is reusable and is sanctioned for a specific context. One would never have a Master Data definition that is a complete enterprise data model. Instead, there would be useful subsets of that, such as Customer, which with Data Virtualization, is packaged with all the sources, transformation, alignment, and business rules needed to access it. That Customer can be called upon for any use; it can be incorporated in data workflows, ETL, on-demand, uploading to a database or application, dashboard, or queried for reporting.

Bite-sized chunks are always more palatable. While it’s possible to define a virtual enterprise model, queries against that model will almost always yield information that needs further modification to make sense with the objectives and in the context of the calling program. What does that mean? It means that additional data manipulation, even complex transform will most often be required. So, from the perspectives of agility, of ease of use and understanding, and consideration of execution risk, enterprise models should be undertaken with discretion.

For-purpose mapping and data conversion, as with Stone Bond's Enterprise Enabler®, ensures that the usage receives the data in exactly the form required, with the most appropriate performance and latency.