MDM - Making it Actionable/Transactional as you Define it

How useful is your MDM, really? Does it just sit there in a repository, waiting for your MDM team to update it?  One of the common criticisms of MDM projects is the magnitude of the project and the low ROI.  More than likely, you are in the middle of a project with great expectations of value.

When most people think of metadata, the scope is limited. It's a schema that defines  a virtual data set, for example. And a Data Master may include meaningful keywords and tags for identification. It may include a cross-reference in a lookup table. And maybe it includes definitions of what each element means and what unit of measure it is in. Then what? Then you have to add references to where the data ought to come from. But then what? You've spent quite a lot of  resources  defining this. Are you any better off than with  the ancient "Corporate Dictionary?"  How do you actually use it?

The most common ways to implement Master Data definitions are indicative of Big Projects:

1.       Define a data warehouse to store the data in, so that it is accessible in the form defined in the Data Master. Once the data warehouse is designed,  corresponding integration must be built to populate it from the appropriate sources, aggregating and transforming as needed, as often as necessary for minimal latency.

2.       Write web services to access the data from the sources and make them available as  Master Data sets.

When I talk about metadata, I think in terms of representing not only the data schemas but also the metadata that describes  where the data is,  what part of it is relevant, how it aligns with other data of interest, how you or the real or virtual destination (master)  needs to see it, and  how it must be converted, or mapped, to be meaningful to the destination.  Then there are the events that trigger data flows, and all the surrounding logic notifications, security, and a host of other things.  If you can capture all of this information as metadata, in reusable, separable  "layers,"  you will have a highly flexible and "actionable"  collection of metadata.

If you define a metadata  Master, say,  "Customer,"  for use corporate-wide, you will have several different sources that are in play to ensure that the various parts of the virtual "Customer" definition has the best information from the most appropriate sources.  Part may come from your ERP, part from Salesforce.com, and another part from an Oracle database. Does your Master definition encapsulate everything you need to use the data? That is, can your metadata be pumped onto a message bus?  Can it be  packaged as a web service?  As an ADO.net object? As a SharePoint external content type?  Does it incorporate the capabilities to perform CRUD (Create, Read, Update and Delete) operations at the endpoints? If one of the sources schemas changes, do you have to do anything to accommodate it? Do you even  need to know a source changed?

If I'm a programmer, I want to leverage the corporate Master Data for my programs and the users of my programs.  I can look up the data definitions, sources, etc., and use them, but that still requires a lot of work.  When the Master Data includes a full set of metadata, then all I have to do is invoke the web service or External Content Type in SharePoint, or  ADO.net and so on.  I simply select the Master I need and indicate how I want to use it.  I don't need to know what the various sources even are, and if the source changes, I won't need to make any changes, since the metadata will reflect what it needs to. And I can pass that selection process on tot the end user of my application or dashboard.

The diagram above shows the scope of metadata captured for MDM by Agile Integration Software. The metadata is generated from a GUI and has an atomic structure so that a change to any metadata can be made without impacting  the whole hierarchy of metadata. Using this type of metadata infrastructure, changes are absorbed without creating waves. Data is accessed directly from the original source , eliminating the need for a costly data warehouse to resolve virtual relationships across sources.

Atomic Architectures for Flexibility and Best Time-to-Value

Big Data definitely doesn't scare me as much as Big Projects. The good thing is that Agile Integration and cloud solutions, along with the pervasive viral nature of Social Media are fueling a shift away from Big Projects and toward incremental atomic approaches with highly reduced time to value.


Historically, Big Projects have been the only way to solve IT problems for Big companies. I've watched "generations" of IT management fall for the "next, next Big Project" promoted by BiG hardware companies, Big Systems Integrators, Big-time analysts. After all, who are you going to trust to set the direction for your Big company? The Big waves always are very well sold, and for the newbies, there is an air of doing something really new and really Big. Of course there's also Big money involved, enough to keep the economy healthy, maybe. As soon as one Big wave of Big Projects are several years in progress, the next Big begins to emerge and put the last one out of business before most are completed. Many stall, are pared way down to the only working prototype, or are abandoned altogether to be replaced with a fresh new Big approach.

Big Projects started long ago, but in the last twenty or so years they have included:

○ Defining a single corporate database, planning for all the applications to share that same db

○ Corporate Dictionary - standardizing the data names and documenting the source
○ ERP - A single comprehensive application means that you don't have to rewrite all the apps to use that db
○ EAI - to address the reality that the above two Big Projects can't be realized
○ Business Process Re-engineering (BPR) - Shifting focus from data to processes- Big Consulting Projects with no need to know much of anything about technology
○ Change Management - because radical BPR created lots of employee issues and confusion
○ Data Warehouse (DW) - in spite of the intentions, smaller projects and best of breed applications were more successful than Big Projects, and businesses came to rely heavily on those systems. Data Warehouses were supposed to bring all the data together for reporting.
○Business Intelligence (BI) - analyze the data in the DW.
○ MDM - the modern Big Project for a corporate dictionary.

There were others, of course, but you get the idea. Finally wedges are putting crevices in the Big Project and opening it for solutions that are more atomic and less global. Some of the wedges are being driven by:

○ SOA
○ SaaS
○ Agile Integration Software (AIS)
○ Social Media
○ The economy and the imperative for improved time-to-value on projects

These factors open the floodgates for a bifurcation of approaches to enterprise technologies. As Mark Twain said, "If there's a fork in the road, take it." Traditionally Big technologies, like BI and ERP, are now offered as a cloud based service and for single users without the overhead of Big.

These wedges are all eroding the cornerstone of Monolithic solutions. For example, SOA is inherently atomic, with an enterprise solution being a collection of SOAP objects. While the initial SOA initiatives were envisioned as enterprise-wide, in the end even the prototype projects were Big, long, and difficult. If the technology were not built on reusable components, the ongoing work that continues to be done would likely have been abandoned. Similarly, while data warehouses continue to be expanded for Business Intelligence, we are seeing a huge number of BI tools coming on the market for specific use or end user-centric implementation. Cloud computing is also whittling away at Big Projects, with significant cost and time reductions as well as shorter time to value.

One of the interesting things is that this split is creating an environment where emerging technology waves now may have two completely different interpretations, one the old Big approach and the other a more agile and atomic approach.

Take data federation and virtualization, for example. The Big approach is to define a complete (or at least really Big) virtual enterprise data model for federation that acts like a staging database would, and then to implement the integration across and through the virtual staging model. Of course, at some point, it's necessary to define those integrations based on what the end result datasets or use happen to be for the consumers.

The new fork in the road (which I would take) requires no data model, virtual or not. An Agile Integration Software addresses federation and virtualization in an atomic manner, with the end use the initial driving force. Entities that describe , for example, "customer" are defined, the source of record for each piece of the Customer data is identified, and metadata is auto-generated and packaged to grab the data from the sources, federated it "on the fly" and deliver it to the calling program, end user or data workflow on demand or in an event-driven manner. An atomic approach to MDM naturally follows.

Hooray for the fork in the road!

Query Optimization across Apples and Oranges

I just recently realized that the problem of federated query optimization that my colleagues and I think about is a completely different problem from the one that has been so well addressed by academics and big database vendors. Even the more contemporary players in the federation and virtualization world don’t extend this concept across disparate sources, and they focus only on run-time speed, but not agility.

 

Those approaches simply do not address the reality that is brought to the forefront now that we have integration solutions that federate everything from web services, spreadsheets, medical instruments, social media, and many other sources, including relational databases in a single "query." The fundamental value of Agile Integration Software (AIS) is violated by the inherent constraints posed by the query optimization tools on the market.


       •        What good to us is a query optimizer that assumes all of the
              data sources are relational databases?

       •        And adding XML to the mix just doesn't "cut the mustard!

       •     What if, in order to use these tools, I have to construct a
              universal data model that includes all of the data that could
              possibly be in play? (The clunky antithesis of agility!)

       •     Do I have to anticipate every data query I might want to
             optimize?

       •    What if there is a lot of transformation that needs to be
             performed along the way to make the data meaningful
             across the sources?


For "pull" integration, where a user's browser interaction or a calling program triggers and specifies the data to be accessed, a SQL query is a universally comfortable way to access information. For a live query in virtual federation, that needs to be interpreted by the federating software into whatever the endpoints understand. The data flowing in from multiple connections needs to be synchronized as the query is being fulfilled from the disparate systems. A "push" integration typically is usually better known, with at least the sources pinned down ahead of time, and often with the exact data being sent each time.

 

In our world, performance is a different problem from typical query optimization on or across relational databases. In complex cross-application joins, the critical path is often more related to the i/o speed of one of the applications or the frequency of disbursement of data, or some other macro factor. The join and access order logic, for example, can be tuned to accommodate the highest resource consumer.

 

So you can see that our problem is not the same one. When people ask us about query optimization, we are sometimes talking apples and oranges!

 

 

 

The Illusion of Pre-Built Adapters

Why do people continue to fall for the idea of "pre-built" adapters? I guess that's pretty obvious. Anything you really want to believe in, you can. Unfortunately, it doesn’t follow that believing in something makes it so.

Dick: Ok, guys, have you figured out how we're going to get this Salesforce/SAP integration done in time for me to meet the VP's deadline?

Harry: I've been online all week studying the possibilities. I saw Adapters from three companies that look really good.

Dick: Come on, we've been down this road before.

Harry: Right, but things have changed! The latest Adapters work immediately off the shelf! Let me show you the videos on the one that looks like it has the most customers… (beep .. "Hello - Welcome to Something SOA Great's web site. I am about to show you the latest thing since…")

 Harry and Dick watch, enthralled. Tom stands behind them with a frown, rolling his eyes.

Dick: If I hadn't seen it, I wouldn't believe it.

 Tom: Hmm. I've seen it and I don’t believe it.

 Harry: Don’t be obstructionist. You just saw that SSG's Adapter automatically connected to both Salesforce and SAP. All the mapping is already built in, so we don't have to even know what the data fields are. You know what that means - we don’t have to deal with those know-it-all data analysts.

 Dick: We could just download it and be off to the races to make the deadline with time to spare. 

Tom: And what if we need to use custom field in Salesforce?

 Harry: Didn't you see that they have 10,000 Adapters in their library? And fifty different versions of this one, so we can look for the closest fit. Then we can tweak it just a little bit to fit what we need. They said they have tools for that.

 Dick: Let's do it!
 

Tom: I need a vacation. Have fun.

So Harry downloaded the Adapter to his desktop.

Harry: Here we go! I'll install here and get it up and running.

 Adapter: /very faint chuckle/

 Harry doesn’t hear. He’s reading the on-screen instructions.

Harry: OK, I'm connecting to SAP

 Two weeks later

Harry: Now I'm connecting to Salesforce…

 Two weeks later

Harry: I think I'm going crazy. I keep hearing this noise that's getting louder every day. But I digress. Here we go - let me try running this beast.

 Adapter: BANG! CRASH ! HA! HA! HA! /hysterical laughter that can be heard all the way to the VP's office/
 
Tom is back from a month’s vacation overseas; He runs to Harry’s cube to see what's going on.

Tom: AARGH! What's going on here? .. Oh, no! The Adapter is squirting SAP data out the port all over the desk!

Dick: /loudly/ Not again! Everyone to their stations! Call 911! Call the auditors! Call OSHA!


Tom: Unplug something before someone drowns in this big pile of SAP Data.

 As the VP arrives at the scene, a cloud forms near the ceiling, creeping out to the hallway. A final Guffaw from Adapter, and the light mist of Salesforce data turns into a terrible storm

------------ End of Same Story, 23rd time around ---------------

What is it that we all want so very badly from Adapters?

• Off-the-shelf solution
• Effortless integration between two endpoints
• No need to program complex mapping and business rules
• No need to know the technical aspects of connecting with either endpoint
• No need to have domain or business knowledge in either endpoint application.
• No need for a data analysts to be involved
• A perfect fit with both endpoints

What makes that impossible?

• Almost every implementation of an endpoint is customized or changes over time
• Your selection of source data is different from what is in the adapter
• Your other endpoint also has been customized
• Your business rules don’t match what's there already

What do you have to do to accommodate?

• Write code to be able to feed the data to the adapter the way it expects to see it ( a full integration in itself!)
• Write code to adjust the manipulation and fit to the customized endpoint
• Open up the adapter, if possible, and add code to modify the business and mapping rules. 

 What do pre-built adapters offer?

• Working at most once off the shelf
• Good experience in re-working code
• Opportunity to practice emotion control
• Incentive to find an alternative.

The alternative:

Connectivity must be designed in such a way that the re-usable parts are solid, and reusable for every instance of a source or destination. Decoupling the business rules from the technical business rules and the connectivity improves reusability. This is the model used by agile integration software. AppComms Removing Splints from Octopus



     

Harnessing Social Media's Big Data

With all the latest hadoopla, there are a lot of people wondering what Big Data means to them. There's a sea of data being generated constantly from Facebook, LinkedIn, and Twitter, and the value of mining and analyzing that body of information is easy to imagine. You can find out all kinds of things that are relevant to your business decisions as well as information that can be turned into stellar marketing initiatives.

Social media easily trumps structured data and documents from the hype perspective. Since its arrival has been relatively recent, we don't really have the same internalized model to extrapolate from in order to conceptualize its meaning and treatment as Big Data. Data from RFIDs and medical instruments is also growing at an exponentially increasing rate, and also offers tremendous basis for completely new innovative solutions.

Forrester's Brian Hopkins, in his informative and interesting "Big Opportunities in Big Data"  discusses the areas and issues of Big Data that are at various stages of commercial readiness. It seems that we still don’t have all the bases covered.

One of the interesting aspects of Big Data, particularly the Big Data that is being captured via social media or instruments like RFIDs is that as soon as it's captured, it becomes history. The reason I'm focused on this type of data is that a huge body of historic information may not be particularly useful to many businesses. By its very nature, the value to businesses is mostly immediate. In the bigger picture of science and statistics, of course, or for fortune 500 companies, it can be, but it is possible to capitalize on the rapidly changing trends of your customer base and the mob mentality displayed before the opportunity eludes you. With the speed of change we are experiencing today, by the time you can get practical results from any Big Data project, you will have missed opportunities to react and reap today's value.

Yesterday I was discussing federation of Big Data with Dana Gardner, principal analyst at Interarbor Solutions, who noted, "Large amounts of data need to be mined, sure, but there are gems in the fresh data from the right applications at the right time that also spell business gold. The needle may be in a hay stack, or it may be inside two or more applications, where the value of the data is only accessible in the context of the integration activity."

Most people interested in Big Data focus on capturing and mining huge bodies of social media data, or in the case of RFIDs, having a complete picture of all of them at one time. There are plenty of uses of this kind of data that are much more practical, in some cases more useful, and certainly do not incur huge projects. For social media, you can leverage the great search capabilities of Facebook, Twitter, and LinkedIn. With RFIDs, just focus on the subset you care about, meld it with a rule, action, response, and you're off to the races long before your Fortune 999 competitors can get started. If you have been dreaming up great ideas about the value that social media's Big Data can bring to your business, let's consider an interim, easy answer to the somewhat premature heavy-duty Big Data approach.

Rather than get the data and then ask the questions from it in the traditional data warehouse tradition, figure out the questions and resultant actions first. Then capture exactly the data you need going forward. Anything in the past is social history. Grab the data as you see it and react immediately, or turn on selective capture for data for a month or so, and analyze the data as it arrives, or trends as they happen, take action, and "dispose of" the data. Keep in mind that the Big Guys in IT thrive on Big, whether it be databases, hardware, or global corporate projects. Do you need those? Can you wait years to get results from Big Data? Think about getting a head start on competitors with a 5 figure investment and your own creativity.

Value In The Integrated Metadata Stack

If you're using or looking at Agile Integration Software (AIS), the chances are you are discovering that there's metadata for everything that's not tied down (and even for those that are). Think about the conceptual epitome of integration. There have been various analogies over time, conjuring up a brain with information flowing (ENS - Enterprise Nervous System), or the flow and pervasiveness of water, and more recently we hear about the fabric. A few years ago I coined the term "synchronapse" to represent the idea of information flowing intelligently, like synapses firing anywhere as needed. Of course, that never took off - new words are fun, but an uphill battle.

I like the fabric metaphor. Good word: the fabric of nations, the geologic structure of a roc; something that represents the essence and the underlying structure; maintaining integrity but flexibly, so that if one point on the fabric moves, the fabric shifts to accommodate that change.

The only way to capture and control the fluid movement of the fabric and be able to ensure that the enterprise can quickly respond to internal and external changes, is to describe everything that can change with metadata. That's a cornerstone philosophy of AIS. Whether the fabric needs to adjust for planned business initiatives or unforeseen external events, the supporting integration infrastructure is adjusted via metadata changes.

Notwithstanding security controls, the full metadata stack must be available to any object or process in the environment, so that conditions at one point on the fabric can affect change in another. That is at best very difficult if each component of your integration stack has its own independent set of metadata. With AIS, as you build your integration with GUI tools, the various layers of metadata and the inter-relationships across the layers is being captured and managed automatically.

What's the value of an integrated metadata stack?

• Reusability of metadata across the stack
• Example: a for-purpose data selection from a source (e.g., customer demographics) can be reused as needed for any map. Also rules and formulas are reusable, along with processes and many other objects.
• At run-time, any business rule can take action based on current values of any metadata
• Example: a different transformation map can be executed depending on customer ID
• Any layer can incorporate other metadata by reference
• Example: an enterprise master data model can reference all the metadata that is needed to bi-directionally access and federate the appropriate sources

This is definitely one of the cool things about Agile Integration Software, possible because it's an IDE, all under one roof.

The Soft Side of Tech Debt

The lean and mean beats the sloth. Sure, some rabbits are a "flash in the pan," but eventually the turtle will lose. As I recall, in that fable the rabbit was fast but lazy and not so smart. You can't count on that being the case with your competitors that have less tech debt than your company. Just look at the big Dotcom successes. They solved problems that hadn't been solved before with completely new approaches and carried no tech debt. Now the problems they solved so well have become shared technology demands for the old "bricks and mortar" companies, implicitly increasing their tech debt, and whittling away at their competitive advantage.

Tech debt refers to the ever-increasing overhead and cumbersome nature that technology infrastructure brings to your company. Old programs that have been patched over and over, ancient hardware, and ever changing trends over time contribute to the tech deficit http://tinyurl.com/3tnyjxf .  Moving toward new trends always complicates your infrastructure unless you can make the 100% shift. Without a complete shift, the left-over ballast limits your ability to leverage new trends.

There is a soft underbelly of tech debt that can be equally debilitating to your company's competitive advantage, and that is the collective aspects of the IT department and services that prevent you from being able to address and keep up with the demands from the business side. There's a backlog of projects, too few people, and not enough of the right skills on the IT team. The focus is on high-profile, new trend projects that presumably would alleviate some of the older creaking infrastructure.

"All I need is five data points for my dashboard every week. How can it be that I'm looking at six months before I get it?" or, "I'm just building a little SharePoint application and I need a couple of pieces of information to include." Business just cannot comprehend why it is so difficult. Then they discover a "back-door" way to get the data themselves via a Rube Goldberg contraption that downloads, puts it in a spreadsheet, tosses it around with formulas and macros, and "Voila! Voila!" there's a palatable concoction to feed their needs. And so is born Shadow IT. The good thing is that the business person stops asking for things, and the bad thing is the surge of new, secret tech debt lurking in every department, where you least expect it.

Apart from subscribing to special purpose SaaS applications, or buying an in-house piece of software, the majority of Shadow IT centers around data access and integration. With continuous increases in empowerment of non-IT employees with tools such as SharePoint and others, it behooves you to start looking at ways to control the spike in tech debt by incorporating an agile tool for integration. Lean Integration methodologies are sensible and may reduce the rate of accumulation of long term tech debt with regard to existing tech debt-ridden infrastructure. Adding an inherently lean Agile Integration Software to your mix means that you will not only be able to respond quickly to many of the backlogged requests, but do so with less specialized IT skills, and ultimately, if your stars are aligned, to turn around the trend of tech debt.