Tech Debt Out-of-the-Box ... "And all the Ills of Integration-kind were Unleashed"

We often talk about having cool capabilities “out of the box,” which is a good thing. That means that you don’t have to do anything but a quick install and you can start using the feature. That is, unless you are talking about Legacy Integration Software (LIS), in which case, when you first “opened the box,” it began spewing Tech Debt before anything else happened. All the ills of integration-kind were unleashed. Years later, you are still prisoner to your Pandora’s Box.

You launch a new project using Legacy Integration Software. First open Pandora’s Box:
        1. Install new instance and all related tools
        2. Apply 64 patches; you can implement the work-arounds in a few months when you start actually developing the integrations.
        3. Send team to a few weeks of Legacy Integration University
        4. Better hire a few consultants, too.
Tech Debt abounds already!

Below is an actual post on a recent Integration Consortium’s LinkedIn Group discussion. The topic has to do with updating customer communications to a standard XML format as opposed to legacy file ftps. The Legacy Integration Software limits the options. Adding XML to the mix means that the LIS needs additional work.

**************
"We already have in house Informatica footprint. So we plan to use that for generating all outbound files. Here is the concern.
We have two options for generating these files.
1) DB -> Informatica -> Standard XML -> XSLT -> Custom File
2) DB -> Informatica -> Custom File
First option provides the benefit of standardization/canonical information model, long term migration path of custom files to standardized xml, and less development since only one Informatica process is required and all custom format are through XSLT.
Problem we see with this approach is the potential performance issue; outbound XML file size in some cases is more than 1GB due to XML tags while the corresponding custom file is a 50MB or so. Second issue is an additional hop that makes support/troubleshooting activities a little harder i.e. where/why a file generation process failed."
*******************

No one should have to think about these things. Agile Integration Software (AIS) like Stone Bond’s Enterprise Enabler would require only a single process for this solution. The differences in the mapping and destination format required would be handled by passing the customer ID, which determines either which map to run or passes variables directly into the transformation engine at run-time to modify the actions. The same process can alternatively step through a standard XML, although the value of doing that escapes me. Performance would not be an issue, and troubleshooting through the streamlined solution is simplified. Stone Bond customers implement such B2B transactions using DBAs as opposed to specially-skilled programmers.

Why, in the twenty-first century, do you have to jump through hoops to get data wherever you want it whenever you want it? Here we are, musing over the “leading edge” Big Data hype and allocating millions of dollars for pilot projects next year, when we can’t even get clean, quick, agile data exchange with our customers and business partners. Does that make sense? I don’t think so. Isn’t it time to embrace twenty-first century technology and start eliminating the Tech Debt you have accumulated instead of continuing on a path that parallels the national debt?

Agile Integration is easy to try out. You do owe it to your shareholders.

2 Keys to Identifying Agile Integration Software

For years I’ve been talking about Agile Integration Software (AIS) as a class of products that enable very flexible and agile data flows across the enterprise, eliminating the clunky, expensive, time consuming infrastructure of the past. This paradigm is agnostic to integration patterns and shares metadata for all uses, such as Data Virtualization (DV), ETL, EAI, SOA, etc.

Over these years, I have come to understand that AIS is not a class of products. As it turns out, Enterprise Enabler® is the only product that has all of the characteristics necessary to fulfill the AIS vision in scope and flexibility. So much for a product-agnostic concept-oriented blog! Many of the features can be found in other products, but somehow there is always at least one critical feature missing that negates the possibility of agility for the solution.

As I think about what constitutes agility in this space, a few things come to mind that are imperatives in such a platform. The two most telling indicators lead the list:

1. The product must have a transformation engine that aligns and transforms data a) from multiple disparate sources at once and b) in their native formats. Without this, complex integrations get little assistance from the platform itself, but are accomplished by extensive custom coding. A streamlined, high-performance data virtualization solution is impossible. Writing back to the sources becomes cumbersome at best, and live, real-time end user interaction with the endpoint simply cannot be effective. Older one-to-one transformation engines do not satisfy the needs; XSLT transformation engines also do not meet the two criteria, because all data must be converted to XML before it is transformed, and the XML output must then be converted to the destination format. Each of those conversions: to and from XML are effectively additional full transformations that generally must be accomplished with custom coding.
2.  There must be a single Integrated Development Environment (IDE) that crosses the entire scope of functionality for all integration patterns. This IDE must incorporate the run-time engines in order to be able to design, develop, test, deploy, and monitor in the same environment. Leaving the environment for anything dramatically reduces the speed of implementation and of change, which is essential for agility, time to value, and minimizing tech debt.

Some things are only possible with AIS

The most recent feature that has been touted by all is in the Data Virtualization space, where data from multiple sources is brought together, cleaned, aligned and transformed without actually moving, staging, or copying the data anywhere, and then delivering it virtually as a “view” to an application or dashboard on demand, again without ever moving it. The most powerful functionality that simply cannot be done successfully with other technologies is data virtualization with write-back to the sources. Enterprise Enabler automatically generates data virtualization, including the ability to write back securely to the sources. These integrations are published for consumption in multiple formats for consumption, such as web services, ADO.Net driver, SharePoint External List, and others.

                          (Click picture to see full chart)

Bottom line value of Enterprise Enabler (AKA Agile Integration Software)

• Time to Value is reduced by up to 90%
• Tech debt, the cost of maintenance and change over time is similarly reduced
• No need for expensive, specialized skill sets particular to a specific tool
• Streamlined architecture inherently enables high performance
• Lower security risks: with Data Virtualization data remains securely in the sources of record, without copies being made
• With write-back to sources in Data Virtualization patterns, dashboards become interactive consoles instead of simply reporting tools
• Without copies being made, a multitude of application specific databases become unnecessary, and synchronization activity is reduced
• Latency issues are removed since applications and end users have always the most current (“live”) information.
• Maintenance and change over time is no longer an overwhelming problem.

 We have observed that while Enterprise Enabler automatically generates bidirectional services, competing product companies tend to proliferate bi-directional lip service.

The Virtual Cycle: Bi-Directional Data Virtualization

With all the buzz about data virtualization (DV), it surprises me how little bi-directional data virtualization is discussed.  Without the ability to write back to sources, the use of DV is limited to Business Intelligence, and other reporting.  Of course, it’s hugely important for that, but when you add write-back to the sources, you are opening up a whole new world of possibilities for a new dimension of interaction with data. 

Suddenly all those dashboards become consoles where business operations can be performed, with end users not just viewing data, but correcting it, updating it, and taking action on decisions. Any application can leverage bi-directional DV to access federated data and to write back to the sources without having to know where it came from, treating it as a single entity. This capability goes a long way to reducing the time to value of many IT initiatives beyond reporting and analytics.

For those skeptics who are not already familiar with bi-directional data virtualization, the first questions are typically, “How do you handle the security to make sure users only write back when they have permission to do so?” and “What happens if  there’s a failure writing back to one of multiple sources?” 

The short answers are that end user security is handled for full CRUD capabilities using SSS or other models, and transaction rollback is managed using two phase commit or other modes.

Now, we can move on to the cool things you can do with this.  Suppose your training is frustrating and time consuming for new employees to learn how to navigate and use multiple systems that are necessary for them to handle their responsibilities. They need to log in to SAP, then the CRM, and then a scheduling system, plus a spreadsheet, all just to perform one task. You could build a browser based app that presents the relevant data from all these systems in one screen, aligned and meaningfully presented. This is the standard data virtualization, which is essentially a reporting tool. Now, turn on write capability for appropriate fields, and voila! That browser screen is a full-service, role-based application, interacting directly with backend systems and data stores just as if you were logged in to all of them. This, my friends, is the virtuous virtual cycle of bi-directional data virtualization.

Using an Agile Integration Software like Enterprise EnablerÒ you can leverage all the federated data services not just for BI, but also for Business Operations. These light weight rapidly deployed nuggets enable this third generation of data virtualization to make agile business a reality.

Agile Integration: Foundation for All That Hype

By definition, Agile Integration Software (AIS) is charged with accommodating all data sources, standards, etc. and moving data agilely throughout the enterprise, adjusting to changes over time. As it does, it captures information about all of the participating endpoints and modalities of data movement. The more an AIS like Enterprise EnablerÒ is used, the more it learns and the more metadata it maintains about the information that is involved in the company’s activities.  Perhaps it’s time to think about AIS as a central core of actionable metadata and a useful engine that can be leveraged for use in new initiatives as they are defined, as opposed to continuing to use AIS to adapt to whatever the new initiative independently demands.  

Consider Master Data Management (MDM) for example. Like all other hype waves, the need is there, and the initiative is fueled by the analysts in concert with a surge of new tech companies as well as the Big Players.  Your company decides it better get on the bandwagon; MDM is imperative to maintaining competitive advantage.  One of the top project architects is charged with establishing MDM, so he studies, attends symposia, and learns what it’s all about.

Next comes a technology selection phase, looking at the emerging companies, but mostly looking at the Big Guys, since that’s always safe, and besides the architect already knows the “Rep” really well. It seems the Big Guy has just bought one of the up-and-coming MDM companies, so they are off to the races.  After a couple of years, the architects begin contemplating questions like, “How does this MDM solution relate to the last decade’s SOA path?”   Of course, at this point it’s obvious that SOA is pretty closely related, or could be, had the two initiatives not each been addressed in its own blissful vacuum.  How does the MDM metadata relate to SOA, and how does it relate to the metadata that your multiple integration platforms require?  Some, with wishful thinking, fall for the Big Guys’ claims of interoperability across all the products of the companies they bought.  In the Quadrant or Wave, they cover all the requisite features, but Alas, poor Yorick! Those mighty features are compartmentalized and each discipline is a separate product with separate underpinnings unable to work cleanly together.  

Let’s look at this from a different perspective. With Agile Integration Software comes a complete flexible integrated metadata stack for use and reusability across all the historic and forward-looking integration schema and models.  Instead of integration adapting to the stand-alone fragmented hype solutions, leverage the power of an existing AIS platform  that brings together the disciplines of Data and Application Integration, Application Development, as well as all the special initiatives of SOA, MDM, B2B, Middleware, Virtualization, Federation, Cloud, Change Management and Big Data, all leveraging the AIS integrated metadata stack.

That means eliminating lots of steps to accomplish any task.  With visibility via the metadata stack, a Master Data definition can be combined with all the associated sanctioned sources and all the related business rules and security. Auto-generated bi-directional web services can handle security and rollback to federated sources. SOA is just another mechanism to make an integration or process available for consumption. Different Data Masters and integrations are chosen at run time based on the current state of any other activity throughout the system.

And finally, AIS monitors for change throughout the metadata stack, validating against the actual endpoints and determining the potential impact and remediating and reconciling as necessary across all those hype wave initiatives. This means stability with agility in your IT infrastructure.

9 Questions to Help Uncover Your Big Data Requirements

The whole concept of Big Data projects can be overwhelming, 'though the promise is compelling. Whether you are analyzing Social Media Data or digging through corporate data,  it's not just about processing huge amounts of data. Just like any other new technology project, it is easy get caught up in the vortex of the hype and lose the bigger picture of what’s involved.  You don't want to find out after the swirling starts that you may be swimming in unwelcome growing tech debt. If you understand the type of functions your solution will need to handle, you will be better equipped to select the most appropriate tools to solve it in ways that incur the least tech debt.

 Here is a quick methodology that will help you develop your own perspective on the Big Data opportunity at hand. Of course, you do need to understand what you really want to accomplish with your BD project, but let's assume you already know the objectives. This will help reveal how complex it will be to handle the data capture, manipulation, and analysis and put the Big Data part of it into perspective of the overall project.

Print this out. Cut out the nine Big Data game cards. Now put them all on a flat surface, turn off your ipod, close the door, and consider each one carefully. Pick "blue" or "red" for each, whichever best describes the data you will be dealing with. Set aside any that you really want to answer "both" or "purple."

 The Big Data Game

As you handle and shuffle the cards, you will see some interdependencies across the cards, and perhaps you start lining them up in the order of processing. If you have the inclination to throw one out completely, set it aside to think about again.

 Now, when you're done, if your answers are a loud and clear "Blue!" on every front, you have the most straight-forward Big Data situation - one that is just about Big Data without the noise of most realities that magnify the project dramatically. Does your table look like this, with all the blues marked?

"Blue!"

Most likely not.  Hopefully you have identified lots of ancillary tasks that will be necessary and that make this look like a data integration project as much as a Big Data project.  You will have to deal with other issues like:

·         Data security

·         Data transformation

·         Data federation

·         Data cleansing

·         Data capture

·         Data migration

·         Data updates

·         Data latency

These are all known problems, with solutions, of sorts. All of these requirements incur additional steps, and are often solved via staging of the data.  More than likely, with this exercise, you are contemplating that you will either need to have multiple staging of the Big Data (3 times Big Data is Big Big Big Data).  This is a huge driver for your company to adopt agile integration software (AIS), an imperative to such projects. Complementing Hadoop, AIS handles federation, inline cleansing and analytics, transformation and other processing without multiple steps along the way. Its transformation engine works directly across multiple sources, orchestrating and merging in their native modes as opposed to requiring intermediate conversion to XML, as XSLT engines do. Secure write-back to sources offers more degrees of freedom to the way you can think about Big Data problems.

 Enterprise Enabler® represents a new paradigm of integration, tremendously streamlining the creation of a Big Data processing environment, eliminating separate steps along the way. Enterprise Enabler is a leading edge federation and virtualization technology, combining EAI, ETL, ESB, and data orchestration to keep up with a constantly changing Big Data environment.

You've Got Big Data! Use it in the Cloud… Virtualized!

I get why Big Data is capitalized. It's even getting to the point where it could rate all caps, "BIG DATA." The part I don't get is why this is such a new big deal. Big Data has been around for a long time. Just think about the huge bodies of data that scientists have been gathering and analyzing just fine over the last who-knows-how many years.  The explosion of social media certainly brings a new dimension on rapidly growing, potentially valuable data, and poses the challenge of making it as valuable as the Big Data you already have.

The most valuable information in your company is probably not social media, but rather your corporate data. What about all that Big Data lurking in your corporate systems, data warehouses and across your multiple divisions? Your corporate Big Data is not like the scientific nor the Social Media Big Data:

o It's not in one place like the scientific data or the Social Media Cloud.
o It's across different organizations and geographies
o It looks different each place

I think of the Big Data focus as being on how to make it useful, not just for business analytics, but more importantly, to make it actionable to your elastic and agile enterprise. This means your solutions needs to be something that can be configured in very short time frames, which means eliminating custom coding. 

In order to be effective at leveraging your Big Data assets, integrating Big Data sources must be treated with streamlined transformation, federation, and virtualization, in an extremely unified architecture. This is necessary to provide the high performance requirement and is achieved by eliminating the classic steps through multiple components for extraction, transformation, staging to federate, and transforming again to align to the destination. It also requires the ability to address complex data manipulation with inline quality checks and error management.

o Access data from multiple disparate systems
o Leverage premises based and cloud based data virtually together in your own cloud
o Federate as data is being accessed from multiple sources, lookup tables
o Virtualize your data. Make it consumable by applications on premise or in the cloud live, virtually. This means that there is never a copy made of your data. (Of course, if you should want to actually send data, that's fine, too!)

Now, about that cloud…

Leveraging the cloud brings a host of opportunities not usually found within the walls of a corporate enterprise. The cloud brings an elasticity to try new architectures, explore new applications and meld your legacy data with Social and New Media for breakthrough business tools such as Predictive Analytics.

The trouble with the cloud is that we are concerned about maintaining the security of our data.  We feel this new technology should bring a tectonic shift in the way we think about integration to the cloud and in the value it can bring, without paying the generally accepted price.

To the best of my knowledge, there is only one product on the market that actually can do this: Stone Bond Technologies' Enterprise Enabler. With Enterprise Enabler you have the solution for Federating, Virtualizing and Leveraging your Big Data in the cloud.

o Your Data remains your data. Your data is never physically copied or placed to the cloud or anywhere along the way. It is accessed, transformed, aligned and virtualized in a single execution
o The data integration is end-user aware. The end user of the cloud application will see only the data he has authority to see.
o Your data is immediately actionable. Users of the cloud applications can update and add data on their screens, and the data is passed back to the source for updating, provided the user has authorization to do that.

A use-case example is a financial institution using this technology to supplement the Salesforce.com data with on-premise information that needs to be seen by the loan personnel alongside the cloud data.

Inspiring New Patterns for Integration

Integration is an age-old problem that doesn't have much  opportunity to get people excited.  It's the same old problem, even if you have new ways to solve it. Over the years, we have tended to just keep massaging old "patterns" pretending like they are new  configurations of moving data around and stuffing it into databases only to take it out again. It takes totally new concepts for the underlying integration architecture to give birth to new patterns that can renovate solution concepts, and more importantly, can enable new business uses that have not been possible before.

Data federation is now finally coming of age enough to have at least a small number of products that can deliver live virtual data federated across disparate systems.

[[Wait..."Live virtual data?" If it doesn't exist, can it ever be dead? Or even stale? If it's live can it be virtual? 

Definition: Virtual

1. Existing in the mind, especially as a product of the imagination.

2. Computer Science Created, simulated, or carried on by means of a computer or computer network: virtual conversations in a chatroom

And I like the origins, from old English and Latin words  that mean effective, excellence, virtue.]]

But I digress!  And maybe it's a good thing, because we tend to throw all these terms around with great authority, completely confusing those who just want to understand what this new breed of software does.  To add to the confusion, "virtualization" is a pervasive term in hardware/operating system jargon, meaning essentially a stand-alone computer that is simulated on another, bigger computer or in the cloud. Now that I have said that, forget it, because that has nothing to do with virtualization with respect to integration.

 For the purpose of this discussion, let's say that in integration speak, "virtual" means that there is never a copy made of the data, and that it never moves anywhere. A virtual view of data allows you to look at data on a dashboard, web page, or the like, without storing it anywhere. When the screen is refreshed, it’s gone. A "federated" virtual view is a virtual view that is consolidated and aligned data from multiple sources. "Live" means that the data comes directly from the sources without staging it in any data store or virtual database along the way. "Bi-directional federation and virtualization" means that the virtual federated data can be interacted with. For example, an end user can update or correct the data on the screen, and it is sent back to the sources as updates, still without staging en route either direction.

With this new technology,  federated data no longer must be staged in order to transform and align it from multiple sources and send it to an ephemeral display endpoint, which presents the live data to the end user ("virtually").  This does wonders for Business Intelligence and Business Analytics.

But going beyond BI, there is one product that enables these displays to take end user's  changes and updates  and pass then securely back to the sources. This is what turns dashboards into control centers. http://tinyurl.com/bqfnzw9

Imagine, though, what a mind shift this requires, and the uncertainty and fear, even, of have a portal like SharePoint, for example, that can effectively become the a single window into all the applications a business user needs. He is not just looking at analytics and drilling down for detail input.

With the ability to interact with backend systems, the business user is no longer just a dead-end in the data flow,  absorbing, and absorbing, and only interacting with the business intelligence software to see more data to absorb. Instead, with bi-directional federation and virtualization, as he sees data that needs to be updated, and draws conclusions from the information, he  can enter his updates as if he were working directly with the backend source systems.

Imagine having a portal that presents you with exactly the information you need to perform your business role. This may include information from SAP, Salesforce,  and, let's say, a scheduling system. From that one screen on your portal, you may update a customer's address, look up prices, and place an order. All of the relevant information will be sent back into the appropriate backend systems automatically. You don't have to learn how to navigate in three separate systems, and you don’t have to figure out how the information is required to be handled differently with each system. Instead, all the work happens behind the scenes, and you only work with easy-to-use aliases for the virtual consolidated data.

  

While this new paradigm inspires new patterns for integration, it also brings pessimists, nay-sayers and even realists who find worrisome points to ponder.   For example, if you have everyone in the company hitting the backend systems directly, could this bring those systems to their knees? See my white paper, Inspiring New Patterns for Integration about a complex pattern that rationalizes and reduces the hits to the backend systems.

With this new technology, there's a whole new world of ideas and opportunities for streamlining integration implementations, which is what has been consistently the largest investment and risk with virtually every IT project since the beginning of time (as we know it).  

SAP Best Practices Streamlined by Agile Integration

With the augmentation of Business Object Data Integrator with ETL and the Rapid Marts, the orderliness of SAP reaches further out toward legacy sources.  SAP's ECC Master Data Structures, SAP AIO BP ("SAP All-in-One Best Practices" ), among other things, define and document IDOC structures for master data elements.

These tools work reasonably well as long as you are dealing with single source to the master definition.  Unfortunately, what lies beyond SAP in your environment is anybody's guess. If  more than half of these work for you "out-of-the-box,"  you must live a charmed life!   More than likely, your legacy sources impose a huge need for custom programming to align multiple data sources and transform them to meet the Best Practices requirements.  The variability and unknowns of your environment  probably will never be able to be addressed easily by SAP's tools, and yet that necessary custom programming consumes a good chunk of your budget and timeline.

Typical integration certification is for one source to SAP. Both the source and the destination have pre-defined schemas, which is virtually never true in real life. Again, with this, we see that this type of adapter seldom works without custom programming adjustments.

Whatever approach you are planning on using for your migration of legacy data to SAP,  you are most certainly facing a formidable, time consuming, and costly exercise.  SAP has greatly improved their tools and best practices over the years, and highly trained and experienced SAP consultants, architects, and programmers have contributed to streamlining the task of data migration. 

This is where Agile Integration Software (AIS), with its light-of-foot cross-application federation, can make a huge difference to your overall  success in meeting your objectives, and in ensuring that as changes occur, they can be accommodated quickly and easily.  That point where data migration becomes a nightmare is where AIS like Enterprise Enabler are the best solution.  The earlier it is introduced into the plan, the better, as some project steps may simply no longer be necessary.

Given the magnitude of a migration from Legacy systems to SAP, and of the ongoing integration required, it has to be a worthwhile effort to establish what the value could be to your project. With a high probability of a minimum of 50% reduction in manpower and elapsed time to implement the integrations, and a rapid, easy path to demonstrating this, why take the risk of NOT investigating?

You owe it to your shareholders, and to yourself to seriously consider AIS!

Agile Master Data Services

The existence of Agile Integration Software gives rise to solid footing for a new model that is emerging for Master Data Management.

In the context of Agile Master Data Management, a Data Master is considered to be not just a schema with all the standard ancillary information about field definitions, units of measure, formats, etc., but it is actually a series of automatically generated services that make the master data accessible live from the multiple root sources that are federated and transformed as required to fulfill the requirements of the Master definition. So, once a Master Data is defined, you can use it in many ways without having to know anything about where the data comes from or whether the official sources behind it changed yesterday or will change tomorrow.

Since the bottom line is about accessing or moving the correct data, we turn the MDM exercise upside-down. The classic approach of defining data models for the whole organization is a daunting task, smiled upon, I am sure, by those Systems Integrators who have a history of making lots of money attacking tasks that are verging on the impossible. Can you really know what the data needs to be from the perspective of every potential use? As always... the Planning group sees the world differently from the Operations group who sees it differently from Finance.

Instead of focusing on an unwieldy data model, consider the approach of a having a growing body of for-purpose schemas, most of which have been defined because the data is required for a project. So what you need, rather than a big data model, is a mechanism to manage this metadata in reasonable chunks, identifying, tagging, and "sanctioning" some as Master Data, and then making them available for re-use in multiple ways with appropriate security.

With Agile Integration Software such as Stone Bond Technologies' Enterprise Enabler, you already have defined reusable virtual schemas , packaged with all the sources of record, federation, transformation, validation, and sometimes the data workflow relevant to general use of the data. With a single click, you deploy one of these nuggets in any number of ways, such as a web service (SOAP, JSON, REST) or as an ADO.net object, or as a SharePoint external list. These particular modes of delivery are virtual, on-demand, and they are not just for reading data, but contain all the information for full CRUD (Create, Read, Update, Delete). Instructions for transaction rollback are included as well as end-user awareness for securing data access to individual permissions. ETL with federation, data workflow, and notifications are processes that work with or on specific data sets and can also be packaged for reuse.

Agile MDM leverages those nuggets by "sanctioning" the ones that have general usability as Master Data Services. Then, where there are gaps, new virtual schemas can be built.

Since the generation of these nuggets, rich with all the layers of information, is a matter of minutes, their proliferation could be a challenge at some point. The System Integrators need to reinvent their Best Practices for a more agile set of processes, to ensure success for these Agile MDM projects and avoid the proverbial herding of cats. In many ways, Agile MDM parallels the growth of social media. It's extremely easy to generate complex integrations, so there could easily be many for each project. The key for MDM is to start to manage the situation at the right point, late enough to leverage the body of for-purpose metadata, but early enough to guide re-usability of Master Data before there are too many to review.

Data Virtualization: No Need to Look Beyond Your Own Back Yard

How is this possible? I did a few Google searches looking for "data virtualization" and "SharePoint" together. SharePoint 2007 and 2010 were designed with the concept of data virtualization built in. How is it possible that only one of the players in this new-hype of data virtualization and federation shows up on the radar as feeding SharePoint? After all, SharePoint is already found in most medium-to-large businesses, so there's no need to look for a separate dashboard or application development platform. You probably have access to it in your company right now!

Out-of-the box, SharePoint has the capabilities to interact with data virtually. I'm not talking about the old SharePoint lists, which you can write to, store data in and change; I'm talking about the BCS External Lists, which were designed with rich features for data virtualization. Why Microsoft doesn’t promote this is beyond me. Conceptually and technically, SharePoint could become the only interface end users in your company need ; they can access data from any Saas, or on-premise application, or any data from electronic instruments in one place. Federated data aligned from multiple systems can be presented in SharePoint out-of-box web parts with live refreshes from all of the sources. SharePoint even has the capability to enforce end-user specific security at the data field level for all CRUD (Create, Read, Update, Delete) functions, using SSS (formerly SSO) or claims authorization approved live at the endpoint application.

Probably one reason that these features are not often leveraged has to do with the scary-looking requirements for the format of the metadata you have to provide and import into the Business Data Catalog. It is, actually, pretty complex, with the need to generate a big hairy XML file and provide a series of specialized web services to use it. But that's where the product companies that specialize in data federation and virtualization should all be stepping handily into the picture. Are those companies oblivious to the ubiquitous state of Microsoft's SharePoint? Stone Bond Technologies, a leader in enterprise data federation and virtualization automatically generates the metadata for SharePoint as well as auto-generating and deploying the web services that execute the bi-directional (if desired) interaction with the backend applications live, federating and transforming across multiple sources.

This approach does not require a team of ten architects, twenty programmers, and long development/implementation cycles. Maybe that's one of the disconnects: the architects and developers can't even imagine such projects taking less than a day. Just think - they could actually take a vacation once in a while.

IT and OT - Shall the Twain Never Meet?

Yesterday I learned a new term in a conversation with Gartner: “Operations Technology," or “OT.” Operations Technology, they tell us, is separate from “IT,” which has the business technology focus. Having frustratingly observed this gap in the past, I found it particularly interesting to see that finally it is being recognized, although the recognition could instead work against closing the gap, forcing categorization one way or the other.

When I first graduated in computer science and began working, I worked at Shell Oil Company, along with the other dinosaurs. There were two totally separate groups, “business computing” and “technical computing.” The business side handled the mundane COBOL programs for accounting and such as well as the infamous midnight data uploads. We on the technical side focused on all the fun stuff that impacted the actual core business of an integrated oil company. I made sure I never admitted to having any knowledge of COBOL, lest I be drawn to the dark side. My focus and passion for computer graphics (in the days when you had to write to the pixel on/off level) led me to develop graphics for 3D reservoir simulation, engineering design, and manufacturing plant simulation.

As I moved on in my career, I observed from both sides the lack of mutual respect for the activities and technology requirements on the other side of the divide. I also discovered that operations couldn’t make the best decisions without a view of the market and the impact of those decisions on the overall well-being of the company. I think the hype wave of re-engineering in the late 80’s recognized this, but the IT department usually did not touch anything on the operations side, and did not want to deal with systems that include process control, complex mathematical optimizers and simulators. So the re-engineering efforts, moving on to BPM, pretended that there was nothing beyond the head office.

It was on this OT side that I began to contemplate how to put in the hands of the engineers a tool that would allow them to get the data they needed for their various models without programming. After all, they were not programmers, and it was impossible to get the programmers to leave what the engineers called the “Ivory Tower” to help out. (That turned out to be the origin of Enterprise EnablerÒ, an Agile Integration platform that naturally bridges the gap between IT and OT).

Apart from some ad hoc custom programming to feed the likes of SAP, there has been little done to bridge the gap. In my humble opinion,this is largely due to two factors. First, the Big Consultants rarely had the depth of knowledge to delve into this type of industry-specific systems and requirements. Second, the Big Integration products were designed as either ETL or EAI, neither of which alone could handle the needs of operations. Of course, integration solutions that bridge IT and OT would likely be funded by both, and there aren’t many people who are in the position to buy that understand enough technology or care enough to make it happen.

It’s much safer to get excited about Big Data and figure out some way that BD can be important. Forget closing the gap between IT and OT. That sounds too hard. But if your competitors decide to bridge the gap, you will definitely lose the edge!

MDM's Unsustainable Tech Debt

One of the nice side-effects of Agile Integration Software is the ability to get useful master data easily and quickly. For most companies, an enterprise MDM project takes years to achieve value, and with the huge effort required to maintain each step, tech debt can skyrocket. Before the project is halfway done, changes and new data and sources have already impacted the viability of the outcome. For an MDM project to bring the promised inherent integrity that offers value, the tech debt simply cannot be ignored. Every single change anywhere in the MDM supply chain must be accommodated immediately and correctly throughout the interdependent process and data networks that constitute MDM.

Only stagnant companies don't change! And only a handful of data sets in any company will remain stable enough to last through the MDM implementation lifecycle. The bottom line is that with the current approach to MDM and the speed with which data is proliferating, MDM, is a self-contradictory concept, and we are likely to see long term initiatives slowly and expensively committing suicide.

MDM - long time-to-value:
Consider the components of the cost of implementing MDM.

•  To start with, you can count on a costly (high six or seven figures) software purchase, likely requiring multiple products.
•  A team of consultants with a range of expertise to implement the multi-year project.
•  Internal resources to manage, guide, and work with the consultants

Consider the components of value of an MDM implementation.

•  Assurance that everyone is using the same data for decisions
•  Quality and correctness of data

Detracting from the validity, therefore the value of MDM

•  Continuously changing sources and formats of data. A heavy solution will have difficulty responding immediately to these changes, leaving gaps in the validity of the information.
•  Latency of data availability due to staging data in a data warehouse or other database. With the current speed of business, users and decision makers need data that is as fresh as possible.

MDM - the building tech debt:

Is MDM even sustainable in its current manifestations? With the complexity of an implementation, the accumulation of tech debt begins as soon as the first Master Data is defined. Every step along the implementation path is fraught with instability.

•  Defining each master data schema that can be everything to everyone who needs the data
•  Determining the most correct sources for each component of the master
•  Determining the criteria for correctness of the data
•  Determining the optimal refresh time
•  Designing a database or data warehouse entry appropriate to the master
•  Implementing the integration necessary to populate the master data store ...or,
•  Defining and implementing ways users can access and aggregate the data components directly from the sourcesThis doesn't include all the discovery and such that the experts and company team must perform. Clearly this constitutes a large, time-consuming effort, that generally is nowhere near agile or responsive to changes in the company, systems, and requirements.

The MDM project is like an armadillo walking down the hill with its tech-debt snowball rolling behind, growing bigger at every step, threatening to consume MDM. Remember that the snowball started accumulating before the first master data was ever used. It is conceivable that the speed of growing tech debt means that the time to value is infinite (never happens)!

Here is another opportunity for rescue by the paradigm of Agile Integration Software.


More on tech debt:
http://agileintegrationsoftware.blogspot.com/2011/04/hows-your-tech-debt.html
http://agileintegrationsoftware.blogspot.com/2011/05/lean-and-mean-beats-sloth.html