9 Questions to Help Uncover Your Big Data Requirements

The whole concept of Big Data projects can be overwhelming, 'though the promise is compelling. Whether you are analyzing Social Media Data or digging through corporate data,  it's not just about processing huge amounts of data. Just like any other new technology project, it is easy get caught up in the vortex of the hype and lose the bigger picture of what’s involved.  You don't want to find out after the swirling starts that you may be swimming in unwelcome growing tech debt. If you understand the type of functions your solution will need to handle, you will be better equipped to select the most appropriate tools to solve it in ways that incur the least tech debt.

 Here is a quick methodology that will help you develop your own perspective on the Big Data opportunity at hand. Of course, you do need to understand what you really want to accomplish with your BD project, but let's assume you already know the objectives. This will help reveal how complex it will be to handle the data capture, manipulation, and analysis and put the Big Data part of it into perspective of the overall project.

Print this out. Cut out the nine Big Data game cards. Now put them all on a flat surface, turn off your ipod, close the door, and consider each one carefully. Pick "blue" or "red" for each, whichever best describes the data you will be dealing with. Set aside any that you really want to answer "both" or "purple."

 The Big Data Game

As you handle and shuffle the cards, you will see some interdependencies across the cards, and perhaps you start lining them up in the order of processing. If you have the inclination to throw one out completely, set it aside to think about again.

 Now, when you're done, if your answers are a loud and clear "Blue!" on every front, you have the most straight-forward Big Data situation - one that is just about Big Data without the noise of most realities that magnify the project dramatically. Does your table look like this, with all the blues marked?

"Blue!"

Most likely not.  Hopefully you have identified lots of ancillary tasks that will be necessary and that make this look like a data integration project as much as a Big Data project.  You will have to deal with other issues like:

·         Data security

·         Data transformation

·         Data federation

·         Data cleansing

·         Data capture

·         Data migration

·         Data updates

·         Data latency

These are all known problems, with solutions, of sorts. All of these requirements incur additional steps, and are often solved via staging of the data.  More than likely, with this exercise, you are contemplating that you will either need to have multiple staging of the Big Data (3 times Big Data is Big Big Big Data).  This is a huge driver for your company to adopt agile integration software (AIS), an imperative to such projects. Complementing Hadoop, AIS handles federation, inline cleansing and analytics, transformation and other processing without multiple steps along the way. Its transformation engine works directly across multiple sources, orchestrating and merging in their native modes as opposed to requiring intermediate conversion to XML, as XSLT engines do. Secure write-back to sources offers more degrees of freedom to the way you can think about Big Data problems.

 Enterprise Enabler® represents a new paradigm of integration, tremendously streamlining the creation of a Big Data processing environment, eliminating separate steps along the way. Enterprise Enabler is a leading edge federation and virtualization technology, combining EAI, ETL, ESB, and data orchestration to keep up with a constantly changing Big Data environment.

MDM's Unsustainable Tech Debt

One of the nice side-effects of Agile Integration Software is the ability to get useful master data easily and quickly. For most companies, an enterprise MDM project takes years to achieve value, and with the huge effort required to maintain each step, tech debt can skyrocket. Before the project is halfway done, changes and new data and sources have already impacted the viability of the outcome. For an MDM project to bring the promised inherent integrity that offers value, the tech debt simply cannot be ignored. Every single change anywhere in the MDM supply chain must be accommodated immediately and correctly throughout the interdependent process and data networks that constitute MDM.

Only stagnant companies don't change! And only a handful of data sets in any company will remain stable enough to last through the MDM implementation lifecycle. The bottom line is that with the current approach to MDM and the speed with which data is proliferating, MDM, is a self-contradictory concept, and we are likely to see long term initiatives slowly and expensively committing suicide.

MDM - long time-to-value:
Consider the components of the cost of implementing MDM.

•  To start with, you can count on a costly (high six or seven figures) software purchase, likely requiring multiple products.
•  A team of consultants with a range of expertise to implement the multi-year project.
•  Internal resources to manage, guide, and work with the consultants

Consider the components of value of an MDM implementation.

•  Assurance that everyone is using the same data for decisions
•  Quality and correctness of data

Detracting from the validity, therefore the value of MDM

•  Continuously changing sources and formats of data. A heavy solution will have difficulty responding immediately to these changes, leaving gaps in the validity of the information.
•  Latency of data availability due to staging data in a data warehouse or other database. With the current speed of business, users and decision makers need data that is as fresh as possible.

MDM - the building tech debt:

Is MDM even sustainable in its current manifestations? With the complexity of an implementation, the accumulation of tech debt begins as soon as the first Master Data is defined. Every step along the implementation path is fraught with instability.

•  Defining each master data schema that can be everything to everyone who needs the data
•  Determining the most correct sources for each component of the master
•  Determining the criteria for correctness of the data
•  Determining the optimal refresh time
•  Designing a database or data warehouse entry appropriate to the master
•  Implementing the integration necessary to populate the master data store ...or,
•  Defining and implementing ways users can access and aggregate the data components directly from the sourcesThis doesn't include all the discovery and such that the experts and company team must perform. Clearly this constitutes a large, time-consuming effort, that generally is nowhere near agile or responsive to changes in the company, systems, and requirements.

The MDM project is like an armadillo walking down the hill with its tech-debt snowball rolling behind, growing bigger at every step, threatening to consume MDM. Remember that the snowball started accumulating before the first master data was ever used. It is conceivable that the speed of growing tech debt means that the time to value is infinite (never happens)!

Here is another opportunity for rescue by the paradigm of Agile Integration Software.


More on tech debt:
http://agileintegrationsoftware.blogspot.com/2011/04/hows-your-tech-debt.html
http://agileintegrationsoftware.blogspot.com/2011/05/lean-and-mean-beats-sloth.html

The Soft Side of Tech Debt

The lean and mean beats the sloth. Sure, some rabbits are a "flash in the pan," but eventually the turtle will lose. As I recall, in that fable the rabbit was fast but lazy and not so smart. You can't count on that being the case with your competitors that have less tech debt than your company. Just look at the big Dotcom successes. They solved problems that hadn't been solved before with completely new approaches and carried no tech debt. Now the problems they solved so well have become shared technology demands for the old "bricks and mortar" companies, implicitly increasing their tech debt, and whittling away at their competitive advantage.

Tech debt refers to the ever-increasing overhead and cumbersome nature that technology infrastructure brings to your company. Old programs that have been patched over and over, ancient hardware, and ever changing trends over time contribute to the tech deficit http://tinyurl.com/3tnyjxf .  Moving toward new trends always complicates your infrastructure unless you can make the 100% shift. Without a complete shift, the left-over ballast limits your ability to leverage new trends.

There is a soft underbelly of tech debt that can be equally debilitating to your company's competitive advantage, and that is the collective aspects of the IT department and services that prevent you from being able to address and keep up with the demands from the business side. There's a backlog of projects, too few people, and not enough of the right skills on the IT team. The focus is on high-profile, new trend projects that presumably would alleviate some of the older creaking infrastructure.

"All I need is five data points for my dashboard every week. How can it be that I'm looking at six months before I get it?" or, "I'm just building a little SharePoint application and I need a couple of pieces of information to include." Business just cannot comprehend why it is so difficult. Then they discover a "back-door" way to get the data themselves via a Rube Goldberg contraption that downloads, puts it in a spreadsheet, tosses it around with formulas and macros, and "Voila! Voila!" there's a palatable concoction to feed their needs. And so is born Shadow IT. The good thing is that the business person stops asking for things, and the bad thing is the surge of new, secret tech debt lurking in every department, where you least expect it.

Apart from subscribing to special purpose SaaS applications, or buying an in-house piece of software, the majority of Shadow IT centers around data access and integration. With continuous increases in empowerment of non-IT employees with tools such as SharePoint and others, it behooves you to start looking at ways to control the spike in tech debt by incorporating an agile tool for integration. Lean Integration methodologies are sensible and may reduce the rate of accumulation of long term tech debt with regard to existing tech debt-ridden infrastructure. Adding an inherently lean Agile Integration Software to your mix means that you will not only be able to respond quickly to many of the backlogged requests, but do so with less specialized IT skills, and ultimately, if your stars are aligned, to turn around the trend of tech debt.

How's Your Tech Debt?

The term "tech debt" has been around for a number of years, and resurfaces periodically. The early usage was a way to talk about the cost of short-sighted programming, bugs, and badly architected solutions. The metaphor gave programmers a financial analogy for contemplating the value of good programming practices, and for non-programmers to appreciate the need for plenty of time to do development well. If you don't do it well the first time, you will need to go back and patch it up, creating more potential break points and incurring increasing tech debt.

Clearly, this is a micro-view of tech debt. If you zoom out a bit, you start seeing debt in not just the quality of the starting point, but also in the ability to keep up with change over time. Eventually, even the best programming needs to be retired and replaced. Stepping back, you start looking beyond programming, to the quality of the product's design and architecture, and the infrastructure and hardware it is tied to.

Tech debt comes from:

• Poor programming practices followed by patches
• Not keeping your assets maintained
• Isolating from the rest of the world
• Not keeping up with universal trends

Over ten years ago,Y2K forced the greatest surge of eliminating technical debt, of course at a huge financial and operational cost. Now, that "new" software and infrastructure is nearing fifteen years in place, and it is incurring its own technical debt for its own reasons. In particular, the EAI that emerged during that Y2K time period was badly needed, but may have been an afterthought for ERPs, hastily architected, and starting out with a tech debt burden.

What about the next fifteen years? You've mastered SOA, but will there be something new that sends it the way of EAI? We're just getting off the ground with SaaS, but maybe in that time the pendulum will begin to swing back to new takes on old trends.

If you look at the dimensions that are generating tech debt, you'll see:

• Quality
• Time
• Trends

Quality is clear; time inevitably invites new requirements, new infrastructures, as the old age out, forcing change on the other old-timer components; trends are a bit more insidious.

What do you do when the tech debt increases at an unsustainable rate? At that point of no return, you replace it. That point comes earlier if you have not kept up with basic investments in maintenance. It's inevitable that you will need to reformulate your strategy and tactics and get a fresh start.

How is your tech debt doing? Is it getting more expensive to keep the status quo than it would to replace or stepwise replace what you have? One of the key technologies you need to look at that can give immediate benefit in managing your tech debt is Agile Integration Software.