Agile Integration Software

Data Quality and your Enabled Enterprise

2011-11-21T14:48:00.001-08:00

As a good example of Agile Integration Software, Enterprise Enabler's data quality features and capabilities serve a representative discussion. (http://www.enterpriseenabler.com/) In the context of data integration, I tend to think of data cleansing and profiling in two separate categories, "batch" and “in transit," or "real time."

Batch - Often this is performed as a first-step-project to an integration implementation to ensure that any existing data that is being used is as correct as possible. The context of correctness is generally defined by the source for which it exists. When the source is an existing data warehouse, the correctness is usually considered with respect to a pre-defined master data definition.

In-Transit or Real-Time - Once the integration is in place, new data is being generated and flows through the organization and systems via the agile integration framework. This data must be validated as soon as it appears in play, as well as when it is passed to its destination, since the definition of "correctness" is ultimately determined by the target use.

With Agile Integration, the philosophy is to focus on the data required for the purpose of the project at hand. While cleansing/validating an entire database or data warehouse full of data may be important, the chances are that it is not important for any particular integration project. Addressing the subset needed means a more efficient project and faster time-to-value.

Pre-validating existing data

Using the inherent capabilities of Enterprise Enabler to discover data schemas and objects, one can simply "point" the appropriate AppComm (application communicator) to a database or application that is to become a source to the integration, and the schema or services available are presented. Select the tables, fields, objects, etc. of interest, and grab a sample or the full set of data. In a configured process, the data can be cleaned, validated and standardized using pre-built rules, external tools, or special logic for each unit of data, by field, by record, or by other cross-section. Rules for logging, notifications, and mediation are configured as part of the process. With this approach, you are focused specifically on the data that will be used for the subsequent integration, and a staging database is not required. Once this process is configured, it can be triggered to automatically run as desired to ensure ongoing monitoring and validation of new data. The results can be fed to a BI tool or spreadsheet for statistical analysis on the data quality ("profiling").

With the AppComm approach, combined with the ability to easily create virtual relationships across disparate sources, cross validation ("matching") across systems or merging data to enrich it, becomes a reasonable exercise, without having to design and build a consolidated staging database. Of course, if the situation still requires a staging database, there's no more efficient way to populate it than Enterprise Enabler.

After you have completed this step, the chances are that the new data that will be captured from here forward needs to be cleansed, too. This can be done "real-time" as it is being acquired from the source and passed to the federation and transformation steps of an integration.

Validating data on-the-fly

As is the nature of Agile Integration, Enterprise Enabler offers multiple places where data cleansing, validation, and remediation can be managed within the flow of data through an integration. Some amount of detection of erroneous data is done as a natural part the data acquisition by the intelligent AppComm technology. Driven by metadata definitions, AppComms check not only for valid data (type, format, etc.), but also for the expected schema. Additionally,

o Validation/cleansing rules, pre-built processes or 3rd party tools can be dropped in or invoked for detection and mediation at various points in execution:

· As soon as the data has been acquired

· As it is being transformed and merged with other sources

· After it has been transformed

· By the destination's Appcomm before/as the data is being posted (plus transaction rollback and assurance in the case of multiple destinations)

· Anywhere in the data workflow process surrounding the transformations

o Enterprise Master System ensures that the data comes from the correct source when an end user invokes a particular piece of information.

o Since Enterprise Enabler's user interface ("Designer Studio") is tied directly to a copy of the run-time engines, as you design an integration, you can do a trial run from the studio and see a sample of the data for inspection to get an idea of the quality of data you are dealing with.

Still don’t trust your data?

Sometimes there are situations where validation rules just won't cut it. Example: setting hard minimum and maximum values for something coming from a physical processing plant. You may be able to determine a reasonable range, but only with the knowledge of what happened yesterday will you be able to determine that a "way out of whack" set of numbers are actually due to a disruption at some part of the plant yesterday. Enterprise Enabler has a preview/analysis feature that holds the result data (post transformation and process) just before it is posted to the destination, in a virtual store, only to be released and posted after review and approval by an authorized human being. That person can do quick tests on ranges, averages, etc. as a gut feel reality check and then fix it if necessary before releasing the data set.

And for those of you who care about data governance

Only an AIS is a single end-to-end integration solution. This means that security can be maintained throughout the integration infrastructure. Developers and Data Analysts log in with the permissions of their role and group, and anything they build or change is logged with who-what-when stamp. Every object in Enterprise Enabler is locked down in such a way, preventing intentional or accidental diversion or modification of data and their flows through the enterprise.

And what about bad data in your ERP?

My apologies, but I just can't help saying to the ERP vendors, "shame on you" for not taking the responsibility to ensure that the data captured and generated by your system is completely correct. How could you let that happen? People trusted you! Ok. Ok.. I'll stop short of calling for an "Occupy ERP" movement.

Alltogether..

With all of the various angles on Data Quality, it’s clear that Agile Integration inherently brings a range of capabilities that are simply not possible with other DQ products. Whether you are looking to correct existing data or ensure the quality of new data as it is created, the fact that the data quality is handled as a natural aspect of integration means a more efficient overall solution.

Big Data Quality

2011-11-17T15:09:00.000-08:00

Big Data means big data quality issues, right? Well, of course, right. Big data means more data that can be bad or go bad one way or another. Big, bad data could have big bad consequences. But just think about some of the ways Big Data may have be in better shape than others.

Big Data

is usually captured automatically, without manual intervention
often has been gathered over many years, so that the framework for capture and validation at the source has improved and been "debugged" over time. Various standards may also play a role in the data capture and ultimate quality. Examples might be weather related data and GIS data.
is often used in ways where analytics and conclusions improve with data volume and errors in individual data become less important. Data quality is essential for Business Intelligence (BI), but from some perspectives, and some aspects of data quality, DQ may move into the background.

Big Data from Social Media has some additional considerations.

Capture mechanisms are well known. Facebook, emails, Twitter, etc.
We know that the quality of information from these is highly questionable - that's the nature, and the beauty of the beast.
We also know that they are well structured. For example an email has a very easily determined structure: there is the header, the body, attachments, etc. The content of the unstructured data (body, attachments) can be searched for relevant information and key words. Bad data might be a corrupted attachment or garbled text in the body, but other than that, errors are, almost by definition, not really bad data.
What do you/we want from social Media’s Big Data? Mostly the trends of the masses. If you clean it up that very exercise could corrupt the data.

Senile data forgets its source and loses relevance and accuracy

There is an altogether different situation with many of the nouveau trendy Corporate Big Data projects. In this case, big data is likely to be consolidated data coming from a number of sources, including those suffering from data senility. Senile data has been through the wringer, moved from residence to residence, been "cleansed" and perhaps never saw the light. A data warehouse usually is populated with data from a huge number of sources, and fallible humans have pored through it, run human-defined cleansing and validation algorithms, and then subjected it to manually-programmed integration code. It is incumbent upon the mining and analysis functions to accommodate assumptions about data quality.

So, as you can see, data quality and cleansing becomes an altogether different problem for Big Data.

Mainframe nearly to the cloud…

2011-10-31T05:42:00.000-07:00

Most people know that Salesforce.com is one of the first and certainly most successful SaaS (Software-as-a-Service) applications on the market. One good thing is that Salesforce stores all the data in the cloud and manages it, eliminating the need for their customers to have the skills and the hardware, software, and maintenance costs to keep it on-premise. That good thing is also the biggest downside of SaaS: the concern that the data is stored in the cloud. Unfortunately, companies worry about having their data stored off-premise with very little control over its management, security, and perhaps even accessibility.

Nevertheless, Salesforce.com has a huge customer base and offers business functionality important to every business I can think of. While business sectors like financial institutions and healthcare could easily make valuable use of the functionality of Salesforce and other cloud apps, the risk and regulatory restrictions make storing their data in the cloud impossible.

These institutions simply cannot make copies of their data or move it to the cloud. The data that is inherent to the functionality of Salesforce may not necessarily be the concern, but often it must be presented to users side-by-side with ancillary data that must come from the company's backend, on-premise systems.

But all is not lost! Agile Integration Software (AIS) naturally solves this problem by creating federated views from multiple sources and making them available to any application, complete with end user access authentication. Here is the crux of the solution with Salesforce as the example:

1. Salesforce.com offers the capability of modifying the screens, so anyone who is conversant in doing that can modify a screen to populate the data from an external source. One option would be to configure it to call a web service when the screen is presented or refreshed.

2. Within a few minutes, an Agile Integration Software, such as Stone Bond's Enterprise Enabler Virtuoso, can be configured, generating metadata that virtualizes and aligns backend data with Salesforce data, and packages it as a web service compliant with Salesforce. Optionally, this would be a bi-directional (Read/write) connection.

3. When an end user brings up the Salesforce page, Salesforce calls the web service, and Enterprise Enabler Virtuoso accesses the on-premise data live, aligns it with the relevant Salesforce data, and sends it to Salesforce screen. With the bi-directional option, data can be entered or corrected on the screen to automatically update not only the Salesforce data, but also the on-premise data, assuming the end user has proper permissions to write back to those systems.

Companies have spent millions of dollars over the last few years trying to do this, and with the Agile Integration Software as the basis, Enterprise Enabler Virtuoso was configured in three weeks to incorporate this Salesforce connectivity. Now it is available off-the-shelf so that anyone can implement it in a few minutes or at most a day.

The diagrams below depict the data residence and flow where on-premise data is required in a Salesforce.com implementation. The first is the common solution where a copy of the on-premise data is made and resides on the Salesforce cloud. I don’t need to tell you the overhead and pervasive concern with doing this. The second shows the on-premise remaining on-premise, where it belongs, and AIS accessing, federating, and delivering a data view virtually to the Salesforce page.

http://tiny.cc/id5h0

MDM - Making It Actionable/Transactional as You Define It

2011-10-11T07:31:00.001-07:00

How useful is your MDM… really? Does it just sit there in a repository, waiting for your MDM team to update it?

One of the common criticisms of MDM projects is the magnitude of the project and the low ROI. More than likely, you are in the middle of a project with great expectations of value.

Metadata and MDM

When most people think of metadata, the scope is limited. It's a schema that defines a virtual data set, for example. It may include a cross-reference in a lookup table. And maybe it includes definitions of what each element means and what unit of measure it is in.

Then what?

Then you have to add references to where the data ought to come from.

But then what?

You've spent quite a lot of resources defining this. Are you any better off than with the ancient
"Corporate Dictionary?" How do you actually use it?

The most common ways to implement Master Data definitions are indicative of Big Projects:

1) Define a data warehouse to store the data in, so that it is accessible in the form defined in the Data Master. Once the data warehouse is designed, corresponding integration must be built to populate it from the appropriate sources, aggregating and transforming as needed, as often as necessary for minimal latency.

2) Write web services to access the data from the sources and make them available as Master Data sets.

When I talk about metadata, I think in terms of representing not only the data schemas but also the metadata that describes where the data is, what part of it is relevant, how it aligns with other data of interest, how you or the real or virtual destination (master) needs to see it, and how it must be converted, or mapped, to be meaningful to the destination.

Then there are the events that trigger data flows, and all the surrounding logic notifications, security, and a host of other things. If you can capture all of this information as metadata, in reusable, separable "layers," you will have a highly flexible and "actionable" collection of metadata.

If you define a metadata Master, say, "Customer," for use corporate-wide, you will have several different sources that are in play to ensure that the various parts of the virtual "Customer" definition has the best information from the most appropriate sources. Part may come from your ERP, part from Salesforce.com, and another part from an Oracle database.

· Does your Master definition encapsulate everything you

need to use the data?

· Can your metadata be pumped onto a message bus?

· Can it be packaged as a web service?

· As an ADO.net object?

· As a SharePoint external content type?

· Does it incorporate the capabilities to perform CRUD

(Create, Read, Update and Delete) operations at the

endpoints?

· If one of the sources schemas changes, do you have to do

anything to accommodate it?

· Do you even need to know a source changed?

If I'm a programmer, I want to leverage the corporate Master Data for my programs and the users of my programs. I can look up the data definitions, sources, etc., and use them, but that still requires a lot of work. When the Master Data includes a full set of metadata, then all I have to do is invoke the web service or External Content Type in SharePoint, or ADO.net and so on. I simply select the Master I need and indicate how I want to use it. I don't need to know what the various sources even are, and if the source changes, I won't need to make any changes, since the metadata will reflect what it needs to. And I can pass that selection process on tot the end user of my application or dashboard.

The diagram above shows the scope of metadata captured for MDM by Agile Integration Software. The metadata is generated from a GUI and has an atomic structure so that a change to any metadata can be made without impacting the whole hierarchy of metadata. Using this type of metadata infrastructure, changes are absorbed without creating waves. Data is accessed directly from the original source, eliminating the need for a costly data warehouse to resolve virtual relationships across sources.

Atomic Architectures for Flexibility and Best Time-to-Value

2011-09-26T15:34:00.000-07:00

Big Data definitely doesn't scare me as much as Big Projects. The good thing is that Agile Integration and cloud solutions, along with the pervasive viral nature of Social Media are fueling a shift away from Big Projects and toward incremental atomic approaches with highly reduced time to value.

Historically, Big Projects have been the only way to solve IT problems for Big companies. I've watched "generations" of IT management fall for the "next, next Big Project" promoted by BiG hardware companies, Big Systems Integrators, Big-time analysts. After all, who are you going to trust to set the direction for your Big company? The Big waves always are very well sold, and for the newbies, there is an air of doing something really new and really Big. Of course there's also Big money involved, enough to keep the economy healthy, maybe. As soon as one Big wave of Big Projects are several years in progress, the next Big begins to emerge and put the last one out of business before most are completed. Many stall, are pared way down to the only working prototype, or are abandoned altogether to be replaced with a fresh new Big approach.

Big Projects started long ago, but in the last twenty or so years they have included:

○ Defining a single corporate database, planning for all the applications to share that same db

○ Corporate Dictionary - standardizing the data names and documenting the source
○ ERP - A single comprehensive application means that you don't have to rewrite all the apps to use that db
○ EAI - to address the reality that the above two Big Projects can't be realized
○ Business Process Re-engineering (BPR) - Shifting focus from data to processes- Big Consulting Projects with no need to know much of anything about technology
○ Change Management - because radical BPR created lots of employee issues and confusion
○ Data Warehouse (DW) - in spite of the intentions, smaller projects and best of breed applications were more successful than Big Projects, and businesses came to rely heavily on those systems. Data Warehouses were supposed to bring all the data together for reporting.
○Business Intelligence (BI) - analyze the data in the DW.
○ MDM - the modern Big Project for a corporate dictionary.

There were others, of course, but you get the idea. Finally wedges are putting crevices in the Big Project and opening it for solutions that are more atomic and less global. Some of the wedges are being driven by:

○ SOA
○ SaaS
○ Agile Integration Software (AIS)
○ Social Media
○ The economy and the imperative for improved time-to-value on projects

These factors open the floodgates for a bifurcation of approaches to enterprise technologies. As Mark Twain said, "If there's a fork in the road, take it." Traditionally Big technologies, like BI and ERP, are now offered as a cloud based service and for single users without the overhead of Big.

These wedges are all eroding the cornerstone of Monolithic solutions. For example, SOA is inherently atomic, with an enterprise solution being a collection of SOAP objects. While the initial SOA initiatives were envisioned as enterprise-wide, in the end even the prototype projects were Big, long, and difficult. If the technology were not built on reusable components, the ongoing work that continues to be done would likely have been abandoned. Similarly, while data warehouses continue to be expanded for Business Intelligence, we are seeing a huge number of BI tools coming on the market for specific use or end user-centric implementation. Cloud computing is also whittling away at Big Projects, with significant cost and time reductions as well as shorter time to value.

One of the interesting things is that this split is creating an environment where emerging technology waves now may have two completely different interpretations, one the old Big approach and the other a more agile and atomic approach.

Take data federation and virtualization, for example. The Big approach is to define a complete (or at least really Big) virtual enterprise data model for federation that acts like a staging database would, and then to implement the integration across and through the virtual staging model. Of course, at some point, it's necessary to define those integrations based on what the end result datasets or use happen to be for the consumers.

The new fork in the road (which I would take) requires no data model, virtual or not. An Agile Integration Software addresses federation and virtualization in an atomic manner, with the end use the initial driving force. Entities that describe , for example, "customer" are defined, the source of record for each piece of the Customer data is identified, and metadata is auto-generated and packaged to grab the data from the sources, federated it "on the fly" and deliver it to the calling program, end user or data workflow on demand or in an event-driven manner. An atomic approach to MDM naturally follows.

Hooray for the fork in the road!

Query Optimization across Apples and Oranges

2011-08-23T15:23:00.000-07:00

I just recently realized that the problem of federated query optimization that my colleagues and I think about is a completely different problem from the one that has been so well addressed by academics and big database vendors. Even the more contemporary players in the federation and virtualization world don’t extend this concept across disparate sources, and they focus only on run-time speed, but not agility.

Those approaches simply do not address the reality that is brought to the forefront now that we have integration solutions that federate everything from web services, spreadsheets, medical instruments, social media, and many other sources, including relational databases in a single "query." The fundamental value of Agile Integration Software (AIS) is violated by the inherent constraints posed by the query optimization tools on the market.

       •        What good to us is a query optimizer that assumes all of the
              data sources are relational databases?

       •        And adding XML to the mix just doesn't "cut the mustard!

       •     What if, in order to use these tools, I have to construct a
             universal data model that includes all of the data that could
              possibly be in play? (The clunky antithesis of agility!)

      •     Do I have to anticipate every data query I might want to
             optimize?

       •    What if there is a lot of transformation that needs to be
             performed along the way to make the data meaningful
             across the sources?

For "pull" integration, where a user's browser interaction or a calling program triggers and specifies the data to be accessed, a SQL query is a universally comfortable way to access information. For a live query in virtual federation, that needs to be interpreted by the federating software into whatever the endpoints understand. The data flowing in from multiple connections needs to be synchronized as the query is being fulfilled from the disparate systems. A "push" integration typically is usually better known, with at least the sources pinned down ahead of time, and often with the exact data being sent each time.

In our world, performance is a different problem from typical query optimization on or across relational databases. In complex cross-application joins, the critical path is often more related to the i/o speed of one of the applications or the frequency of disbursement of data, or some other macro factor. The join and access order logic, for example, can be tuned to accommodate the highest resource consumer.

So you can see that our problem is not the same one. When people ask us about query optimization, we are sometimes talking apples and oranges!

The Illusion of Pre-Built Adapters

2011-08-05T16:08:00.000-07:00

Why do people continue to fall for the idea of "pre-built" adapters? I guess that's pretty obvious. Anything you really want to believe in, you can. Unfortunately, it doesn’t follow that believing in something makes it so.

Dick: Ok, guys, have you figured out how we're going to get this Salesforce/SAP integration done in time for me to meet the VP's deadline?

Harry: I've been online all week studying the possibilities. I saw Adapters from three companies that look really good.

Dick: Come on, we've been down this road before.

Harry: Right, but things have changed! The latest Adapters work immediately off the shelf! Let me show you the videos on the one that looks like it has the most customers… (beep .. "Hello - Welcome to Something SOA Great's web site. I am about to show you the latest thing since…")

Harry and Dick watch, enthralled. Tom stands behind them with a frown, rolling his eyes.

Dick: If I hadn't seen it, I wouldn't believe it.

Tom: Hmm. I've seen it and I don’t believe it.

Harry: Don’t be obstructionist. You just saw that SSG's Adapter automatically connected to both Salesforce and SAP. All the mapping is already built in, so we don't have to even know what the data fields are. You know what that means - we don’t have to deal with those know-it-all data analysts.

Dick: We could just download it and be off to the races to make the deadline with time to spare.

Tom: And what if we need to use custom field in Salesforce?

Harry: Didn't you see that they have 10,000 Adapters in their library? And fifty different versions of this one, so we can look for the closest fit. Then we can tweak it just a little bit to fit what we need. They said they have tools for that.

Dick: Let's do it!

Tom: I need a vacation. Have fun.

So Harry downloaded the Adapter to his desktop.

Harry: Here we go! I'll install here and get it up and running.

Adapter: /very faint chuckle/

Harry doesn’t hear. He’s reading the on-screen instructions.

Harry: OK, I'm connecting to SAP

Two weeks later

Harry: Now I'm connecting to Salesforce…

Two weeks later

Harry: I think I'm going crazy. I keep hearing this noise that's getting louder every day. But I digress. Here we go - let me try running this beast.

Adapter: BANG! CRASH ! HA! HA! HA! /hysterical laughter that can be heard all the way to the VP's office/

Tom is back from a month’s vacation overseas; He runs to Harry’s cube to see what's going on.

Tom: AARGH! What's going on here? .. Oh, no! The Adapter is squirting SAP data out the port all over the desk!

Dick: /loudly/ Not again! Everyone to their stations! Call 911! Call the auditors! Call OSHA!

Tom: Unplug something before someone drowns in this big pile of SAP Data.

As the VP arrives at the scene, a cloud forms near the ceiling, creeping out to the hallway. A final Guffaw from Adapter, and the light mist of Salesforce data turns into a terrible storm

------------ End of Same Story, 23rd time around ---------------

What is it that we all want so very badly from Adapters?

Off-the-shelf solution
Effortless integration between two endpoints
No need to program complex mapping and business rules
No need to know the technical aspects of connecting with either endpoint
No need to have domain or business knowledge in either endpoint application.
No need for a data analysts to be involved
A perfect fit with both endpoints

What makes that impossible?

Almost every implementation of an endpoint is customized or changes over time
Your selection of source data is different from what is in the adapter
Your other endpoint also has been customized
Your business rules don’t match what's there already

What do you have to do to accommodate?

Write code to be able to feed the data to the adapter the way it expects to see it ( a full integration in itself!)
Write code to adjust the manipulation and fit to the customized endpoint
Open up the adapter, if possible, and add code to modify the business and mapping rules.

What do pre-built adapters offer?

Working at most once off the shelf
Good experience in re-working code
Opportunity to practice emotion control
Incentive to find an alternative.

The alternative:

Connectivity must be designed in such a way that the re-usable parts are solid, and reusable for every instance of a source or destination. Decoupling the business rules from the technical business rules and the connectivity improves reusability. This is the model used by agile integration software. AppComms Removing Splints from Octopus

Harnessing Social Media's Big Data

2011-06-24T10:34:00.000-07:00

With all the latest hadoopla, there are a lot of people wondering what Big Data means to them. There's a sea of data being generated constantly from Facebook, LinkedIn, and Twitter, and the value of mining and analyzing that body of information is easy to imagine. You can find out all kinds of things that are relevant to your business decisions as well as information that can be turned into stellar marketing initiatives.

Social media easily trumps structured data and documents from the hype perspective. Since its arrival has been relatively recent, we don't really have the same internalized model to extrapolate from in order to conceptualize its meaning and treatment as Big Data. Data from RFIDs and medical instruments is also growing at an exponentially increasing rate, and also offers tremendous basis for completely new innovative solutions.

Forrester's Brian Hopkins, in his informative and interesting "Big Opportunities in Big Data" discusses the areas and issues of Big Data that are at various stages of commercial readiness. It seems that we still don’t have all the bases covered.

One of the interesting aspects of Big Data, particularly the Big Data that is being captured via social media or instruments like RFIDs is that as soon as it's captured, it becomes history. The reason I'm focused on this type of data is that a huge body of historic information may not be particularly useful to many businesses. By its very nature, the value to businesses is mostly immediate. In the bigger picture of science and statistics, of course, or for fortune 500 companies, it can be, but it is possible to capitalize on the rapidly changing trends of your customer base and the mob mentality displayed before the opportunity eludes you. With the speed of change we are experiencing today, by the time you can get practical results from any Big Data project, you will have missed opportunities to react and reap today's value.

Yesterday I was discussing federation of Big Data with Dana Gardner, principal analyst at Interarbor Solutions, who noted, "Large amounts of data need to be mined, sure, but there are gems in the fresh data from the right applications at the right time that also spell business gold. The needle may be in a hay stack, or it may be inside two or more applications, where the value of the data is only accessible in the context of the integration activity."

Most people interested in Big Data focus on capturing and mining huge bodies of social media data, or in the case of RFIDs, having a complete picture of all of them at one time. There are plenty of uses of this kind of data that are much more practical, in some cases more useful, and certainly do not incur huge projects. For social media, you can leverage the great search capabilities of Facebook, Twitter, and LinkedIn. With RFIDs, just focus on the subset you care about, meld it with a rule, action, response, and you're off to the races long before your Fortune 999 competitors can get started. If you have been dreaming up great ideas about the value that social media's Big Data can bring to your business, let's consider an interim, easy answer to the somewhat premature heavy-duty Big Data approach.

Rather than get the data and then ask the questions from it in the traditional data warehouse tradition, figure out the questions and resultant actions first. Then capture exactly the data you need going forward. Anything in the past is social history. Grab the data as you see it and react immediately, or turn on selective capture for data for a month or so, and analyze the data as it arrives, or trends as they happen, take action, and "dispose of" the data. Keep in mind that the Big Guys in IT thrive on Big, whether it be databases, hardware, or global corporate projects. Do you need those? Can you wait years to get results from Big Data? Think about getting a head start on competitors with a 5 figure investment and your own creativity.

Value In The Integrated Metadata Stack

2011-05-27T11:25:00.000-07:00

If you're using or looking at Agile Integration Software (AIS), the chances are you are discovering that there's metadata for everything that's not tied down (and even for those that are). Think about the conceptual epitome of integration. There have been various analogies over time, conjuring up a brain with information flowing (ENS - Enterprise Nervous System), or the flow and pervasiveness of water, and more recently we hear about the fabric. A few years ago I coined the term "synchronapse" to represent the idea of information flowing intelligently, like synapses firing anywhere as needed. Of course, that never took off - new words are fun, but an uphill battle.

I like the fabric metaphor. Good word: the fabric of nations, the geologic structure of a roc; something that represents the essence and the underlying structure; maintaining integrity but flexibly, so that if one point on the fabric moves, the fabric shifts to accommodate that change.

The only way to capture and control the fluid movement of the fabric and be able to ensure that the enterprise can quickly respond to internal and external changes, is to describe everything that can change with metadata. That's a cornerstone philosophy of AIS. Whether the fabric needs to adjust for planned business initiatives or unforeseen external events, the supporting integration infrastructure is adjusted via metadata changes.

Notwithstanding security controls, the full metadata stack must be available to any object or process in the environment, so that conditions at one point on the fabric can affect change in another. That is at best very difficult if each component of your integration stack has its own independent set of metadata. With AIS, as you build your integration with GUI tools, the various layers of metadata and the inter-relationships across the layers is being captured and managed automatically.

What's the value of an integrated metadata stack?

Reusability of metadata across the stack

Example: a for-purpose data selection from a source (e.g., customer demographics) can be reused as needed for any map. Also rules and formulas are reusable, along with processes and many other objects.

At run-time, any business rule can take action based on current values of any metadata

Example: a different transformation map can be executed depending on customer ID

Any layer can incorporate other metadata by reference

Example: an enterprise master data model can reference all the metadata that is needed to bi-directionally access and federate the appropriate sources

This is definitely one of the cool things about Agile Integration Software, possible because it's an IDE, all under one roof.

The Soft Side of Tech Debt

2011-05-04T04:19:00.000-07:00

The lean and mean beats the sloth. Sure, some rabbits are a "flash in the pan," but eventually the turtle will lose. As I recall, in that fable the rabbit was fast but lazy and not so smart. You can't count on that being the case with your competitors that have less tech debt than your company. Just look at the big Dotcom successes. They solved problems that hadn't been solved before with completely new approaches and carried no tech debt. Now the problems they solved so well have become shared technology demands for the old "bricks and mortar" companies, implicitly increasing their tech debt, and whittling away at their competitive advantage.

Tech debt refers to the ever-increasing overhead and cumbersome nature that technology infrastructure brings to your company. Old programs that have been patched over and over, ancient hardware, and ever changing trends over time contribute to the tech deficit http://tinyurl.com/3tnyjxf . Moving toward new trends always complicates your infrastructure unless you can make the 100% shift. Without a complete shift, the left-over ballast limits your ability to leverage new trends.

There is a soft underbelly of tech debt that can be equally debilitating to your company's competitive advantage, and that is the collective aspects of the IT department and services that prevent you from being able to address and keep up with the demands from the business side. There's a backlog of projects, too few people, and not enough of the right skills on the IT team. The focus is on high-profile, new trend projects that presumably would alleviate some of the older creaking infrastructure.

"All I need is five data points for my dashboard every week. How can it be that I'm looking at six months before I get it?" or, "I'm just building a little SharePoint application and I need a couple of pieces of information to include." Business just cannot comprehend why it is so difficult. Then they discover a "back-door" way to get the data themselves via a Rube Goldberg contraption that downloads, puts it in a spreadsheet, tosses it around with formulas and macros, and "Voila! Voila!" there's a palatable concoction to feed their needs. And so is born Shadow IT. The good thing is that the business person stops asking for things, and the bad thing is the surge of new, secret tech debt lurking in every department, where you least expect it.

Apart from subscribing to special purpose SaaS applications, or buying an in-house piece of software, the majority of Shadow IT centers around data access and integration. With continuous increases in empowerment of non-IT employees with tools such as SharePoint and others, it behooves you to start looking at ways to control the spike in tech debt by incorporating an agile tool for integration. Lean Integration methodologies are sensible and may reduce the rate of accumulation of long term tech debt with regard to existing tech debt-ridden infrastructure. Adding an inherently lean Agile Integration Software to your mix means that you will not only be able to respond quickly to many of the backlogged requests, but do so with less specialized IT skills, and ultimately, if your stars are aligned, to turn around the trend of tech debt.

How's Your Tech Debt?

2011-04-27T06:30:00.000-07:00

The term "tech debt" has been around for a number of years, and resurfaces periodically. The early usage was a way to talk about the cost of short-sighted programming, bugs, and badly architected solutions. The metaphor gave programmers a financial analogy for contemplating the value of good programming practices, and for non-programmers to appreciate the need for plenty of time to do development well. If you don't do it well the first time, you will need to go back and patch it up, creating more potential break points and incurring increasing tech debt.

Clearly, this is a micro-view of tech debt. If you zoom out a bit, you start seeing debt in not just the quality of the starting point, but also in the ability to keep up with change over time. Eventually, even the best programming needs to be retired and replaced. Stepping back, you start looking beyond programming, to the quality of the product's design and architecture, and the infrastructure and hardware it is tied to.

Tech debt comes from:

Poor programming practices followed by patches
Not keeping your assets maintained
Isolating from the rest of the world
Not keeping up with universal trends

Over ten years ago,Y2K forced the greatest surge of eliminating technical debt, of course at a huge financial and operational cost. Now, that "new" software and infrastructure is nearing fifteen years in place, and it is incurring its own technical debt for its own reasons. In particular, the EAI that emerged during that Y2K time period was badly needed, but may have been an afterthought for ERPs, hastily architected, and starting out with a tech debt burden.

What about the next fifteen years? You've mastered SOA, but will there be something new that sends it the way of EAI? We're just getting off the ground with SaaS, but maybe in that time the pendulum will begin to swing back to new takes on old trends.

If you look at the dimensions that are generating tech debt, you'll see:

Quality
Time
Trends

Quality is clear; time inevitably invites new requirements, new infrastructures, as the old age out, forcing change on the other old-timer components; trends are a bit more insidious.

What do you do when the tech debt increases at an unsustainable rate? At that point of no return, you replace it. That point comes earlier if you have not kept up with basic investments in maintenance. It's inevitable that you will need to reformulate your strategy and tactics and get a fresh start.

How is your tech debt doing? Is it getting more expensive to keep the status quo than it would to replace or stepwise replace what you have? One of the key technologies you need to look at that can give immediate benefit in managing your tech debt is Agile Integration Software.

SaaS Vendors Take Note: You Can Operate in the Cloud but Not in a Vacuum

2011-03-31T16:09:00.000-07:00

This week I watched a webinar that proposed that more responsibility of integration with SaaS applications needs to be carried by the software provider. Analyst Dana Gardner of Interarbor discussed this topic in the webinar and on his blog http://tinyurl.com/4fgyw5j. Workday recently announced its cloud-based integration services as part of its SaaS ERP offering, stepping up to the plate to provide tools that will ensure that customers can integrate to their own software. They very well may be setting a trend that other ISVs will need to keep up with.

It is clear that, while the convenience and specialization of cloud-based solutions opens a new opportunity for businesses to trim down their infrastructure and the associated maintenance costs and effort, it doesn't eliminate the need for integration. Instead, it calls for a new paradigm for data integration.

Unfortunately, the heavy middleware in enterprises today will have trouble keeping up with the increasing business demands for agility. Change is tough when no one wants to touch the integration for fear of breaking something.

With the growing dependence on cloud-based software, your customers need a new generation of integration that can streamline data flows as their business processes move freely across legacy, clouds, and collaboration portals. If you are a SaaS vendor you need to seriously think about how you can rise to the occasion for your customers. The better you do it, the happier your customers will be.

Workday addressed the challenge by buying an integration software company! That's not only an expensive way to go, but it means that now, in addition to domain expertise and software development teams for their ERP, they must also maintain expertise and developers in the rapidly-changing integration space. That's probably not a prudent business approach for most SaaS vendors.

The alternative is to embed, rebrand, and/or offer Agile Integration Software, such as Enterprise Enabler® in your offerings www.enterpriseenabler.com. That way you get all the benefits without the headaches. I don't know if Workday's integration fits the Agile Integration Software model (see http://tinyurl.com/3yv85bw), but with AIS, even a SaaS software vendor is able to offer integration across cloud apps, and also incorporate on-premise backend legacy systems as well as pass data to and from your customer's SharePoint installations, on-premise or hosted.

Crossing the Chasm Between Consumer and Business Technology

2011-03-14T04:34:00.000-07:00

How can it be that consumer technology manages always to deliver on ease of use, compatibility, and basic human appeal? Why do we have to deal with ugly, difficult, clunky software in the business world? How is it that there are standards that have been pervasively accepted and implemented for cell phones and all manner of electronic gadgets? Don't you ever wonder why it is that consumer technology hits the spot and continues to do so?

A huge contributor to the difference is the nature of the products themselves. Consumer electronics, phones, and even software are basically "throw-away." They are low cost and are expected to be completely replaced every couple of years, so consumers are forgiving of bugs and don't even think about upgrades. They simply get a new one. Maintaining the history of the world part 2 or part 1 is not the responsibility of these devices and software. The closest they come is the need to accommodate a few years of contacts' phone numbers and addresses. And even that.. only the business users of these things care.

For consumer software, the imperatives of interoperability with existing and old technology are very minimal, limited to internet and communications standards, which are discreet and maturing. The consumer market is huge, and with the constant replacement by customers, it's a path of enviable "recurring revenue" streams. Besides that, marketing to consumers is much more intuitive and pervasive than targeting the specific individuals who might want/need the business solution and also have the authority and the budget to buy it. With business systems, there's no walking by a store, seeing a flashing display, and buying on impulse.

For business software product companies, it's a whole different world: more complexity, expectations for longevity, necessarily higher price tags, and requirements of "upward compatibility" for new releases. There is always the imperative of having to work in tandem with everything else that's in place or being invented by competitors.

So our challenge is to figure out how to close the gap and move business technology closer to consumer technology. Of course, it's SaaS that offers the greatest potential to shake up the software product market over the next few years, but I think that Agile Integration will come into the mix in a big way, dramatically simplifying the underlying corporate IT infrastructure.

The Power and Flexibility of Data Federation and Virtualization Together

2011-02-28T10:29:00.000-08:00

Finally, the world seems ready to take data federation and virtualization seriously. The power of each is, and has historically been, severely limited by the absence of the other in the same product. Using data virtualization means that you can switch out the underlying data and where it comes from without changing the actual integration layer, but rather by just redefining the metadata. That definitely adds flexibility to a company’s underlying infrastructure. Federating data means that you can bring data from multiple sources, combine them into a single data set or view, and let users and applications consume it without having to stage it. That eliminates the overhead of defining, implementing, and maintaining a data store. Combining the two, along with a robust transformation engine, in a single execution, adds orders of magnitude improvement in the agility of the overall IT infrastructure.

It seems to me that this is a very good time for IT decision makers to start aggressively implementing these combined capabilities throughout their organizations. Data warehouses have come of age, maybe have even reached "a certain age" status, and are carrying the ballast of years of data, maintenance, and accommodation of change. Streamlining and speeding up can only done by more computing power and better access algorithms and tools: new forklifts. Meanwhile, to remain competitive, the company must be able to rapidly deliver information quickly and appropriately wherever it is needed.

It's essential that the core IT department management and corporate executives not leave the flexibility to the "Shadow IT" that comes of desperation from the business and operations people who cannot wait and deal with the front lines of delivering the company's activity and daily business decisions.

With Agile Integration Software, the IT department could quickly deliver flexible, maintainable layer that interacts with everything that is already in place, but that steps up to the plate with agility. Please check out my white paper on Data Federation and Virtualization. http://tinyurl.com/6a8zbxv

Federated Data for SharePoint 2010 using AIS

2011-01-31T15:24:00.000-08:00

If you're not switching to SharePoint 2010, you are missing a huge opportunity. The new features actually position it so that it can become the ONLY application that end users need to log into. All kinds of useful SharePoint applications can be easily built to bring data from multiple backend systems together, aligned in a single window, for user viewing and updating.

You probably don't think of SharePoint as an MDM product, either, but maybe you don't really need the multi-million-dollar, huge upkeep, of the MDM products from the (other) Big Guys.

You may know that SharePoint 2010 advanced the concept and usability of virtual data definitions from the earlier Microsoft Office SharePoint Services (MOSS) to include full CRUD (Create, Read, Update, Delete) and other features like SSO (security). This "external entity" catalog, called BDC, or Business Data Catalog, functions as an MDM of sorts that contains the metadata describing corporate business entities along with the metadata describing how to get the data and where it comes from. When a user opens a window that is bound to one of the entities, s/he can interact with the live data without knowing or caring where it came from. The interaction is with virtual data accessed live from backend systems.

SharePoint Designer is a nice tool for building the metadata when it comes from a limited range of single sources. With some amount of programming, you can extend its use a little. Unfortunately, without the ability to update an external entity's metadata definition, SharePoint Designer must create a whole new definition, which then needs to be switched out for all the apps that use it.

SharePoint 2010 really flies, though, when you combine it with an Agile Integration Software (AIS) like Enterprise Enabler http://www.enterpriseenabler.com/. Agile integration brings the ability to build complex data mappings, from multiple sources (without programming) and generate the desired external entities in SharePoint. Federating data on the fly from multiple sources is powerful.

Now, suppose the backend systems changes. Maybe your entity combines data from SAP and Dynamics CRM. If you switch from Dynamics CRM to Salesforce, an AIS can apply the changes behind the scenes to connect the same external entity definition in SharePoint without any disruption.

There are a couple of things you might be concerned about, like security and transaction rollback on updates to multiple back ends. SharePoint supports SSO, so AIS will pick up the end user security from SharePoint and pass it to the backend systems. If the user does not have permission to read from a backend system, s/he will not see the data. If the user does not have write access to one of the backends, the full transaction will be rolled back. This is generally the only time data is actually persisted in an AIS.

This should definitely give you something to think about. You could use all the savings for a company trip to Colorado, or Hawaii, or anywhere else. Or maybe give a fat raise to everyone in IT.

----------------------------------
Video of building a SharePoint entity in AIS http://tinyurl.com/5vxce55

Agile Integration Serving End Users Directly

2010-12-28T15:13:00.000-08:00

Social media is beyond ubiquitous, putting the power of communication and availability of information in the hands of everyone individually and collectively. It has moved from being a thrill for the eager to being a fact of life even for the reluctant. Social sharing of unstructured data is proliferating, and its management is the subject of much conversation, posting, tweeting, and blogging.

But what does this mean and how does it relate to individuals who need structured data? Isn't it time for an end user to be able to access data without calling the boss, who calls the IT department, who calls the consultant, who plans the project, that gets a team, that does an analysis, that includes everything but the kitchen sink, that has a garbage disposer, that's the only thing that works, with which the project finally gets euthanized?

It's time to think about what this means to data integration: lean, mean, or otherwise. These days end users are empowered by SharePoint and BI tools and such to determine for themselves what information they need in order to do their job and make decisions. If an end user can sign in to SAP and to Salesforce and to JD Edwards, why can't they use their SharePoint portal or a dashboard to access the same data from those applications? Of course, one of the key reasons is the issues around security. Historically integration has been behind the scenes, working with no visibility to the actual end user. ETL, of course, has none, and EAI is also not end-user aware. Dashboards are designed for specific users who have permission to see the data displayed. Although the dashboard app may have its own login for security, the integration behind the scenes does what it's told without knowing anything about the user.

Agile Integration Software http://tinyurl.com/3yv85bw supports SSS and other layers of security so that an end user of SharePoint, for example, can see live data with updates passing back to the applications, only if the user has privileges to do so. That same security is available out of box in AIS for any vendor software, BPM, or BI that is designed to read and write data to an ADO.Net entity via an ADO.Net driver, or alternatively by web service calls.

For example, Enterprise Enabler® http://www.enterpriseenabler.com/ is an AIS that can be configured in minutes to merge and align data from multiple applications and generate the entity definition and custom connector for SharePoint (BDC) or generate the entity for the ADO.Net driver that is called from WSS or any other ADO.Net Application. If preferred, it can generate a web service just as easily. The SSS end user security is honored in all these models.

So, in response to the expectations of the "me and everybody else now" surge, integration needs to step up to the plate.

Two Laws of War, IT, and Human Nature

2010-12-21T15:09:00.000-08:00

There are two things that I have considered to be fundamental rules of life from as far back as I can remember. In my early twenties, it dawned on me that I actually thought of them with greater conviction than I ever did any of the laws of physics I learned. I decided that these rules held equivalent un-challengeable level of "truth" as laws of physics. Of course I did, in fact challenge every "truth" handed out in my first physics class. How in the world could everyone in the class just sit there and believe that there's some Force pressing up on the floor to keep us from falling through? Because the textbook said so? Hmm. I grew up in Virginia, and everyone knew that the State published the history books, and it was in Virginia's best interest to publish that every President except Lincoln was from Virginia! So, how can you believe there's an equal and opposite arrow somewhere that you can't see?

Now for my two laws, in the order of discovery.

Pamela's Law of Set Points. This may not be a very good name for it, and since I never named it before this moment, I reserve the right to change the name some time. The law is something like this: More things than you can imagine try to obey a set point. The set point may gradually shift over a long time, especially in response fundamental change due to extraordinary events, but in general nature gravitates to that set point. I remember thinking about this law and human nature when I was "into" change management and a person's or institution's capacity for change ("resilience"). While each person or organization has a different level of tolerance, that maximum is essentially constant. The same can be applied to a person's good humor. I believe there's a set point for each of us that is fulfilled by our reactions and attitudes no matter what is actually happening in the real world outside us. Worriers manage to find a certain level of worry, easily filling the void with a new subject of worry as another gets resolved. I'd love to be one of those eternally happy people that have a set point way on the happy side. I guess it's kind of the "glass half full" conundrum, which somehow leads to:

Pamela's Law of Constants. Again a name of the moment. I grew up in the Washington, D.C. area and from the time I was six or so, through high school, I ritualistically read the front page of the Washington Post every single morning (then I read the comics if my siblings were done). The most obvious conclusion to me was that there was always a war somewhere, often in some tiny country I never heard of before. As one subsided (from the headlines) another popped up somewhere else, like the cartoons where the guy with the big hammer pounds down one bump and it pops up somewhere else. That's when I began formulating the theory that unrelated happenings like this really were fulfilling some constant amount of world-wide war. Over the years, I discovered that this same law applies in many, many areas. I could elaborate on the law of constants in the IT world, but that's for another day. You can see, though that if there are constants to be balanced by seemingly unrelated events, that means the events really are related… my crude contribution to the evolution of Chaos theory.

Five Ways to Get Immediate ROI from Agile Integration Software

2010-12-13T07:33:00.000-08:00

One of the great things about Agile Integration Software (AIS) is that you can install it and start reaping the benefits immediately. You don't have to perform weeks or months of analysis to figure out whether it is a good idea or not. You don't have to cut a million dollar check. Just jump in! Since it's quick to install, easy to learn, and completely non-invasive, I can't imagine any significant risk to just doing it, with the common exception of internal politics. But, if you keep it under the radar, and dance around the political push-back, in short order you can be the hero! While the consultants are still estimating their huge project, you can "just do it" in a fraction of the time and resources. The key is that you don't have to change your whole company's philosophy about middleware. Think about AIS as an adjunct to what's already there, and just solve a couple of important challenges quickly.

Don't know what AIS is? Take a look at this: http://tinyurl.com/3yv85bw

Here are a few ideas that can prove the value of AIS and bring real value to your company before the rest of the team wakes up.

ONE. Get all manner of data into SharePoint. Empower your SharePoint developers and users with secure access to data from all your backend systems. That means that the people that need to see and interact with data that comes from SAP, Salesforce, and Excel, or a custom application, for example, can have that without even declaring a project. Line of business (LOB) applications built in SharePoint invariably need data from multiple systems, and it all needs to be aligned somehow to make sense together. Try AIS for this. Build a couple of interfaces to SharePoint External Lists - BDC (business data catalog) or to SharePoint Lists and see how it goes. The data is secure, and you can even use the same interfaces to handle write-backs to those systems. Here's a video showing how you would do this. http://tinyurl.com/3xkclgj

TWO. Rescue a project that is way behind schedule. Chances are that the part of the project that's lagging behind is the integration with other systems. Don't displace whatever technology is in being used, but try a rapid parallel path with Agile Integration. Skip the staging database and get the data live from the systems align it on the fly and send it where it needs to go. This is what AIS is designed to do really well. If you find that the easiest way to get started is to go with a staging database that's in place, then use AIS to get the missing data into the repository quickly. Or build the integration and expose it as a web service in two clicks.. Voila! Then sit back and reap the benefits of success (and be a hero).

THREE. Fast-track your BI dashboard project. Let's say you have a dashboard project that has lots of promise. Maybe you just bought a very cool analytics software package that will allow you to understand your business and the market better so you can maintain your competitive advantage. The vendor delivered the software and gave a two-day training on how to configure and use the cool graphs and such. But now they've gone home, and it's up to you to figure out how to get all the data you need into the tool. The more data that's available, the greater the value of this investment. Big problem. Big project, fraught with all the perils of any integration project. Before you spend a year creating the database, see if the tool is just making a call that could be tied to an AIS that could grab the necessary data, align it, and pass it live to the dashboard. That's definitely faster and a much better solution. Otherwise, at least reduce your population of the staging database by 70% by using AIS.

FOUR. Close your books in half the time. If your company is like most, there are multiple ERP or financial systems as well as inventory systems and even spreadsheets involved in the process of consolidating all of the information required to close the books every month. With a changing environment, new entities must be accommodated or eliminated from the process on a regular basis. Using AIS you can automate the capture and alignment of the information from the multiple sources, but even better than that, you can update the integration in a matter of minutes to reflect the changes in your business from month to month or quarter -to-quarter.

FIVE. Improve the productivity of your IT team. Empower your Data Analysts to configure integration without programming. Free up your programmers from doing repetitive integration activities. AIS can eliminate a tremendous amount of activity that simply does not need to be done anymore. You owe it to your company to take advantage of the value of AIS. Start with new data flows that need to be done rather than initiating a lifelong switchover project.

BPM's Nemesis : Clunky Data Integration

2010-11-02T13:13:00.000-07:00

Back in the '90's, I jumped right in there with all the other consultants and IT operatives when Business Process Reengineering came along. Since I had been heavily involved in computer graphics before that, I even wrote a crude business process designer so we could abandon the yellow stickies on the walls and halls as well as all the tedious documentation (and re-documentation). BPR evolved to BP'X', and by the time mainstream BPM software came along, I was working on developing software to handle the data flows across all the systems involved in what I called at the time, "Decision Support Chains."

I lost touch for a while with what BPM in general was doing and realized later that most of the products probably actually began with the design and documenting of the processes. I opined that they were started contemporaneously with my similar effort, and were abandoned in favor of Y2K gold. Once that was over, maybe they picked up the old code, dusted it off, and went on from there. Most started with design and evolved to handle complex as well as practical of business process management, but the necessary and tough behind-the-scenes integration has been generally assumed to be outside the discipline. There still appears to be a missing layer or two.

With the great tools now available to design, model, optimize, and automate human workflow, it seems that the toughest part of a full implementation of new processes, or automating old ones, is the ability to get the data where it needs to be when it needs to be there. The necessary overhead and inherent inflexibility of data integration are very often so prohibitive that they kill projects before they even get started.

Agile Integration Software (AIS) working with BPM provides the complete "stack" of necessary functionality. AIS fills in with rapidly configured, adaptable data integration, along with the data logic layer that interacts with the BPM layer, triggering data flows as needed exactly when the process and participating decision support applications require it.

All the overhead of designing and maintaining staging databases goes away along with the resultant data latency. The cost of BPM implementation is greatly reduced, and the solution is able to respond as quickly "under the covers" to business change as it is on the BPM layer!

When BPM inherits all of the features and assets of AIS, BPM become agile, too.

ABPM - Agile Business Process Management? I like it!

Earth Huggers and Underground Sculpture

2010-10-27T08:38:00.000-07:00

I need to take a side trip from integration software!

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk http://www.patmonk.com/ told me about his Earth Huggers sculptures last night. These are 16G stainless steel, each 32 by 48 inches. You can see from the pictures that they are embedded in the ground and make inviting "ground pieces." Ground pieces? Well, if you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above, you may see a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and create beautiful sculptures that will have to be unearthed in order to to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. (I have the fish, but don't remember what's inside. I think there's a picture somewhere, though.) Another was a life-sized carved wood woman that he took horizontal cross-sections from and re-assembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like an elongated teardrop. The pointy end is in the ground with a steel plate around it. He claims it's the switch that he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

Web Services and APIs Just aren't Enough

2010-10-20T13:04:00.000-07:00

We've had a number of questions lately about how AppComm technology is different or better than Web services or APIs (Application Programming Interfaces). That's probably because we have been working on packaging Enterprise Enabler® various ways for low end, not-so-demanding point solutions. For example, if you want to access Salesforce.com from a WSS application, you could use the Salesforce.com AppComm with or without another AppComm, and in a few minutes configure a bi-directional ADO.net driver. When you want to get data from Salesforce.com, you just read from and write to the ADO.net object.

The AppComm technology assures simplicity beyond any API or Web service connectivity by leveraging, behind the scenes, all the knowledge of the APIs as well as any necessary coding, scripting and deciphering. The data is simply provided for selection and mapping, eliminating the hard-coding and difficult maintenance necessary when programming to APIs and Web services directly.

Well, that's the whole point! The fundamental philosophy behind Agile Integration Software http://agileintegrationsoftware.blogspot.com/2010/05/characteristics-of-agile-integration.html design hinges on encapsulating absolutely the most intelligence possible behind the scenes so that the tough, complex, or tedious work is only done once. If I have to study the API for SAP's RFCs (Remote Function Calls) to build connectivity components, why not design the connectivity as an AppComm, which means that if I do it well once, no one will ever have to read the API specs again!!! (sorry, for the triple exclamation points - my mother would not have approved, with her proper and perfect usage of the English language, but in this case, I think they are well deserved). Not only do you never have to study the API, but you don't have to understand the prerequisites, assumptions, and in what order all the calls must be made.

Off the shelf, AppComms encapsulate all that knowledge and programming in a flexible and reusable manner, so you can get going quickly. It discovers all the schema information specifically for the particular instance at hand, and allows you to select the data of interest. You can then configure the mapping without programming, which you would need to do without an AppComm.

The truth is, contrary to coerced popular belief, the fact that there exist Web services for accessing an application's data does not mean they are easy to use. And since you are programming instead of configuring, if there's a change to your application's data schema, you have to go back into the code and make changes in order to access new fields, or make sure that your integration doesn't crash because expected data is missing. AppComms know when the schema changes, and alert you to reconfigure.

There are clearly time and cost savings when utilizing the AppComm technology compared to accessing the raw Web services. When you combine the AppComm with its Agile Integration platform, you have the synergy for coordinating data access from multiple applications simultaneously, aligning and transforming the information on the fly and passing it to the destinations without staging.

AppComms:Removing the Splints from the Octopus

2010-09-20T06:47:00.000-07:00

AIS (Agile Integration Software) conjures up the image of information flowing freely wherever it needs to be whenever it needs to be there, aligning and merging with other information as needed by each endpoint person or application. The information is not yesterday's wilting snapshot of the world, or dusty data that has struggled and morphed through five different formats along the way. Think clean, precise, and now.

I contend that the ultimate culprit that works against AIS is the classic "Adapter," along with its cohort "standards." Now please understand that they are not equally culpable, and perhaps neither is, in the perfect world. Unfortunately, as far as I can tell, we haven't reached Utopia yet. An integration model that adopts a specific standard simply pushes the responsibility of dealing with the tough issues outside its world for someone else to handle. That's why Adapters exist; they must get data from whatever/wherever/however it is into a specific destination's requirements ("standard"). The Adapter has no visibility or responsibility to adjust to run time situations. Its job is simply to get predefined data from one application or format to another.

AppComms have a different responsibility; that is as an "application specialist" that can interact with the endpoint application or database at run time as it is instructed by the integration coordinator. Each record or block of data from that endpoint that is needed somewhere in the integrated environment is provided by the AppComm on request, and similarly, data is posted to the destination as requested. AppComms know how to auto-discover the schema for that application or data store, whatever "schema" means for that application; they know how to perform all CRUD (Create, Read, Update, Delete) operations as appropriate (or not) for the specific endpoint application; and they are responsible for generating the reusable metadata defining desired data across scenarios. The specialization of an AppComm is not for the specific instance of an application, but reusable across all instance of that system. For example, a Salesforce.com AppComm will be able to work with any configuration of Salesforce, as opposed to assuming a specific schema.

The elimination of the classic Adapter is like taking splints off an octopus. The integration coordinator can orchestrate a huge range of concurrent information flows performing all kinds of tasks. Multiple live AppComms can be coordinated to access data in an order and timing to resolve cross-application virtual relationships at run time. Imagine eliminating all those staging databases! Adapters simply can not possibly offer the same fluidity and flexibility for integration as AppComms.

Value of Embedding Agile Integration Software (AIS) in Software Products

2010-08-20T04:35:00.000-07:00

Is there a software product on the planet that doesn't need to integrate with other data and applications? Well, OK, I downloaded a cool iPhone app that acts as a carpenter's level all by itself. But I can't think of a standalone business application that doesn't need to connect to anything else.

While they all may appear to be the same to a casual observer, in my book there are four types of software product companies:

The 1st Type:

Those that use code bases as a starting point for custom programming services projects. These really are poseurs, pretending to have products, enticing customers, and consuming huge amounts of service dollars. Integration with other systems is the majority of their gravy. There's no point in discussing the value of AIS to this first category; their business model is based on hourly services, so reducing the time to integrate by 60-90 percent would be a curse.

The 2nd Type:

Those with off-the-shelf products that leave the integration to the customer.

These products usually operate against a database that the customer is responsible to populate and maintain.
These companies lose sales because the customer does not want to undertake the time and cost of developing and maintaining the necessary integration.

Value of Agile Integration Software:

Eliminate the imperative of a staging database and take advantage of read using live data and write-backs to customers' existing applications.
Increase the percentage of sales closed by greatly reducing the integration effort.

The 3rd Type:

Those that have off-the-shelf products but still count on their integration services as a welcome and important revenue stream.

Value of using Agile Integration Software for your projects:

Reduce the risk associated with the inevitable unknowns of integration projects.
Fixed bid projects can be offered with lower risk and high margins.
Where staging databases are used, the product can enter a new realm of interactivity.
Competitive advantage with expanded capabilities and lower total cost of implementation.

The 4th Type:

Those that have off-the-shelf products that embed or package with an agile integration software (AIS). These companies are already reaping the benefits of agile integration and are becoming the leaders in their domains.

Benefits these companies are getting from Agile Integration Software:

Reduced overall cost to implement their products.
You can focus more on your own product and domain expertise, and not have to invent other ways to integrate with customer's systems.
Reduce the risks associated with integration.
Fixed bid projects can be offered with lower risk and high margins.
Eliminate the overhead of staging databases.
Re-brand to your own name so there is no need to also sell a third party integration tool.
Expand the scope and value of their products by enabling information/data sharing back to customers' systems.

Now that Agile Integration Software really does exist, you owe it to your shareholders to look into the bottom line benefits that it can bring to your business. We are seeing rapidly growing adoption of this paradigm, and we are seeing these companies reap the advantage. Maybe they'll make the sale you can't because of the painful integration your implementation requires.

From Dashboard to Control Center with AIS

2010-07-28T10:24:00.000-07:00

What's wrong with dashboards today? Nothing really, but stretch the limits by adding a few more dimensions of reality and they can fly! Maybe that's the difference between a car's dashboard and the dashboard in the cockpit of an airplane. One just displays the situation and the other allows you to interact and control the airplane.

Executive and management dashboards have become pervasive and invaluable contributors to many businesses' decision making by providing access to a broad range of key corporate and specialized data in easily assimilated visual screens. Today's dashboards are mostly like cars. If you're lucky, you get real time data displayed, along with relevant analysis so that you can make operational or strategic decisions. When you make decisions in a car, you react by letting up on the gas pedal or stopping at the gas station. With business dashboards, you make decisions and then have to go to another system to update data, or send an email or otherwise convey or initiate execution of those decisions. Why can't you act on your decisions from the dashboard? Why do dashboard have to be "read-only?"

With data coming from a range of sources, the prevailing approach to accessing the data from multiple backends is for the dashboard vendor to tell the customer that it’s in their court. (Just get all the data in one database so they can read from it and provide awesome analysis and visualization. I wish I had a dollar for every sale that was lost on that alone! )

Even if it were easy to do, that staging database is the inherent problem with dashboards that limits their usefulness. Whether the vendor develops and populates it or the customer does, there is a specs effort, a design effort, and ongoing implementation effort. Populating it requires a huge programming effort. But the biggest problem is that you simply cannot display current, live data, and you cannot really even think about the data being updated in the dashboard and sent back instantly to the backend systems.

Instead, if you use an Agile Integration Software to implement the dashboard integration, you can automatically have secure, bi-directional data flows. You can eliminate all of the overhead associated with the database design, implementation, and maintenance. That means a rapid dashboard implementation project, and the ability to turn the dashboard into a control center where decisions can not only be made, but can be executed. Maybe you can't even call it a dashboard anymore!

ISVs are rebranding and packaging AIS with their products or embedding into their products using an Agile Integration Software ADO.Net driver. See Stone Bond's Enterprise Enabler

Agile Integration for SharePoint : Our Dev Kitchen Experience

2010-06-29T10:23:00.000-07:00

They used to call it the "Death Kitchen," but now they use the kinder and gentler "Dev Kitchen." That was our first introduction in person to the Microsoft SharePoint 2010 development team, back in January 2009. We had been working for about a year with them, preparing our agile integration product, Enterprise Enabler, to connect tightly with the new version of SharePoint. We tied our cool technology to their cool technology via their next generation of Business Data Catalog (BDC), which incorporated full CRUD (Create, Read Update, and Delete) capabilities. Before Dev Kitchen, we had worked with notes and samples about BDC from their team, to auto-generate the cryptic and complicated BDC XML file, including services that we auto-generated to initiate and leverage our powerful integration capabilities.

By the time January came, we had validated some of our work, but hadn't really been able to test it much. The whole Dev Kitchen experience was quite interesting and exciting for our team, as we had little idea what to expect. We were one of less than twenty companies invited to participate that week. Some were product companies, some were customers, but we were the only one doing this kind of tight connectivity. Each company had its own separate and secure room; only our team knew the access code to open our door. There was an air of excitement and secrecy, but mostly everyone was hard at work testing against a very early version of "O14."

One of the most amazing things was working side-by-side with the developers and visionaries of the Business Connectivity Services (BCS). If we ran into a snag, within a couple of minutes the specialist for that particular feature of BCS was in the room with us tracing the source of the issue and clarifying to us or making instant fixes if they needed to. Our work spanned the full range of BCS features, so we met and worked with quite a number of the team members. What an exciting, vibrant, enthusiastic group of people!

We had, as I recall, two and a half days to finish our work. On the last day we all gathered in a large room and took turns presenting to everyone the things we had been working on. We had ten minutes to do it. When Mike Guillory, our head SharePoint technologist spoke, the room was packed with participants and Microsoft developers, architects, and management. In ten minutes we showed building from ground up a mashup of data live from Salesforce.com, merging and aligning it with data from a SQL Server database, auto-generating the BDC , and configuring an out-of box web part for the data. We opened the window in SharePoint, and Voila! The data appeared from both sources. We edited a couple of fields, and showed that back in Salesforce and the database, the fields were updated!

Spontaneous standing ovation from the Microsoft team! Later that evening when we were invited to show a little more detail to part of the team, someone commented that what he was seeing was what he had dreamed about his whole career. We later found out he was a very high level Microsoft director. All in all, it was a rewarding experience, and since then, we have tied up all the loose ends with the pre-alpha, alpha, beta, and now production versions, and have added security with SSS and transaction assurance and rollback. The Microsoft team advised on the architecture, recommending the "Custom Connector," which resides on the SharePoint server for tightest possible security and best performance possible.

Now, even the TAP program has ended, and we're all in full production. Thanks and congratulations to architect Mohammed Nazeeruddin and his team, who met with us every week to ensure the best fit for our embeddable integration.

B2B : Brain-to-Brain

2010-06-21T14:11:00.000-07:00

My generation was not born with cell phones. While on vacation for a few days , I tried to disconnect from the digital world with only mild success. Not a good time to be reading Born Digital: Understanding the First Generation of Digital Natives, by John Palfrey http://www.amazon.com/Born-Digital-Understanding-Generation-Natives/dp/0465005152, if I didn't want to think about this stuff. When I was growing up, telephones were ultimately connected by oodles of wires all over the planet. We never even thought of wireless telephones.

We did, though, talk a lot about ESP (extra-sensory perception, for those of you too young to remember), and we debated about whether psychics were communicating, or whether they cheated. I wonder if ESP even seems ridiculous to a young person who has always known cell phones. You don't have to even be in the same room to hear what the other person is saying. No wires. I bet it's not such a stretch to believe that you could have direct brain-to-brain communications. After all, knowing what they know now, why not? Think about it - we wouldn't even need to talk, but just think about talking. Read just a little about all the new research on brain placticity, and you have to wonder if brain-to-brain communications is just a matter of practice.

Then you have to wonder what brain-to-brain (B2B) means in the integration space. We'll be transforming everything from Venus to Mars and back again.

Agile Integration Software Means New Best Practices

2010-05-24T13:46:00.000-07:00

One of the most exciting benefits of Agile Integration Software is the ability to dramatically streamline the analysis and other processes around designing and implementing integration solutions. If RAD (Rapid Application Development) is fundamental to the essence of the software, then it is natural that a new genre of best practices will emerge leveraging the agility of the software.

Enterprise Enabler® has "fuzzied" the lines between the classic steps of analysis, design, development, testing, and deployment. I'd like to share a couple of ways that it is changing the way teams develop integrated solutions.

Analysis Workshops replace Specs. "What?!!!," you say. "That's heresy! No self-respecting IT department would eliminate writing specs!" Ok, so hear me out. Everyone knows that the most time consuming, messiest, most unpleasant part of the whole project is defining the specs, especially the mapping specs. Well, not just defining them, but documenting them, running them by everyone for approval, re-working them, running them by everyone for approval (and so on), and even worse, keeping them up to date over time. Wouldn't you love for that whole exercise to be history, and yet to be able to get the exact current definition any time? With Enterprise Enabler, the specs are synonymous with the solution.

Here we go. You're in a conference room (or web meeting) with three other people, Tom, Dick, and Jane. You are the project manager or an analyst, or even a programmer. You're on the keyboard with Enterprise Enabler's user interface projected on the screen. With you, remotely or in the room, are the three business analysts that really know their systems' data and how it is used. Think how many questions can be resolved in a very short time if these key people can discuss what they require while you build it!

Pass the doughnuts around while we define the big picture of the integration we are solving. Jane is the expert on the destination application, "D," so in my book, she's the boss. After all, the destination's the only the application that cares about the data in an integration. Tom owns source "A" and Dick owns source "B." You pull up the template builder to define the metadata for the appropriate data from "A." Tom points you to the instance of the system that you will be working with and you show him the schema as he indicates what is of interest from his app. Then you do the same with Dick, and then with Jane for the destination. She has indicated what needs to be populated. You go to the mapper and everyone can see on the left all the fields from both sources and on the right, the fields to be filled. First Dick and Tom help you define the cross-application relationship by identifying the appropriate key fields, while you drag and drop them to create the virtual relationship. Now you continue to work with these experts to build and test the data mapping en situ. Tom discovers he left out a couple of tables he need, so you bring up the template and add the tables, and you're back in business. You run the map through and let Tom, Dick, and Jane discuss the nuances of whichever data are not obvious until it is correct. You're done, maybe even before the doughnuts are finished. And the programmed mapping will always match exactly what you see on the screen. When Jane changes her mind or Dick's application has updates that change the schema, you simply go in an reflect it in a couple of minutes.

Integration will become a set of well defined and productive workshops with all the right people, wherever they are. You'll only reminisce about the old days and how difficult it was to get the specs from each expert, document them, and go from one to the other, changing and updating, and then not being able to find the specs document to be able to keep it up to date. Remember when the programmer would discover a problem with the specs and would fix the code, but the change never got back to the specs? Ah... for these headaches to be history!

Coercing the Mind Shift to Comprehend the Paradigm Shift

2010-05-21T12:59:00.000-07:00

The mind certainly is stubborn. Something about re-directing those synapses to another mindset is astoundingly challenging. Just like the synapses connect "sphere" and "bounce" to the concept of "ball," "enterprise integration" is only recognizable as "integration" if it looks like a heavy, time-consuming, difficult, expensive endeavor. So what if a cubical ball could actually bounce even better than a spherical one? Even if I see it, I can't make the "connection." Well, I don't know much about synapses, but I sure have had lots of experience trying to initiate the plasticity to reformulate the way people think about integration!

Recently we presented a bit about our technology to a handful of people who are extremely well-informed about the integration space. We planned to hit a few points that we are pretty sure can't be done with other products. We zeroed in on one specific point we thought would amaze anyone who really knows the current state of integration and what is possible and what is not. This is a capability that is a natural fall-out of our new paradigm: we showed how to create virtual relationships across totally different systems even faster than you would define a relationship across tables in a single database.

In ten seconds we showed creating a virtual relationship between Microsoft Dynamics CRM and a SQL database. In that ten seconds, all the run time components were generated behind the scenes, so we ran it. This scenario happened to be sending data to an out-of-box SharePoint 2010 web part; it could have gone to any other application, dashboard, web service, or pumped to a messaging system .

We showed how an end user, from a single SharePoint web part (screen), can see live data from multiple applications, aligned and transformed as appropriate for their usage. As if that weren't enough, we showed updating a couple of fields in SharePoint, and the backend systems were immediately updated!

No staging of the data, no copy made at all, anywhere. The data was accessed, aligned, displayed, changed, and updated back in the backend systems. Our product, Enterprise Enabler, eliminates a HUGE amount of time, manpower, and cost by eliminating the need for a staging database any time you need data merged from two or more disparate systems. The systems could be SAP, Salesforce, XML, spreadsheets… it doesn't matter. So, no one needs to design a database model that is everything to everybody. No one needs to build the database, and no one needs to maintain it.

A response came from our audience that many enterprise integration platforms can do this! I would love to do a "side-by-side" bake-off with the cited competitors, Websphere or Tibco, any day. I am extremely confident that we would probably have to go home before their years of work are done. If they even can do it.

In the end I have to write the response off to the discipline's persistent twenty-plus year track record that has forced consistent synapses dancing around in the same circle, convincing everyone that there is just no other way integration can possibly be done. If Websphere, Tibco, Informatica, and others can't do it, we'll just pretend they can!

Characteristics of Agile Integration Software

2010-05-19T15:03:00.000-07:00

One of my objectives in starting this blog is to see what I can do to overcome the pervasive mindset, with respect to integration in general, that conjures up a heavy, limited framework that can only actually work with a huge amount of custom coding. It is only by stepping outside that box that we can imagine the next generation of integration software.

What should one look for in an integration product that indicates that it will actually be able to generate an agile environment? Here are a few of my thoughts.

First, it must be "off the shelf" or "out of box" or downloadable, whatever term you like. That means that you can install it in just a few minutes. If it takes a week or months to install, you are definitely not going to have an agile environment!

It must require little or no training to get started building connectivity. A data analyst should be able to design and build data mapping and transfers. This means the people who know what the result really needs to be can actually do the implementation.

You must be able to design, develop, test, and deploy from within a single environment. No separate effort to generate and assemble a run-time components.

It must support accessing, transforming, and aligning data from multiple sources live without staging in a database or interim store. Real time data access and availability is a critical aspect of an agile enterprise. Think what work it takes and what value is lost when the data must be staged before it is transferred to wherever it is going.

It must be end-to-end metadata driven.

If there is any programming required, it should be doable from within the same environment, and the code saved and versioned as part of the metadata.

The transformation engine should be able to operate on data in any application's native format as opposed to having to step through a central view such as XML. How can you possibly get high performance when your integration must run through multiple steps.

If you are looking for agility at an Enterprise level, the product must be scalable across clusters and extensible so that it becomes an environment tailored with all the special requirement you have that are needed over and over.

I guess that's plenty for now. Over time, I imagine I'll discuss most of these points in more detail, and how they are present in our Enterprise Enabler agile integration software.

Is "Agile Integration" an Oxymoron?

2010-05-17T12:15:00.000-07:00

Most people would agree that "Agile Integration" is an oxymoron. The heavy footprint, heavy customization, and huge amounts of interdependence across an integrated enterprise absolutely negate the possibility of Agility! But we discuss Agile Enterprises as if you could instantly have an intelligent SOA implementation that detects and adjusts to business changes, maybe even anticipating the evolution of the business. What are we talking about, anyway? All the SOA initiatives I've heard about are isolated subsets of business solutions that impose the web services environment (which takes lots of design, development, and definitely anticipation of everything you will want to do in the future). The "agility" offered is limited to a specific, known set of options.

If we want to reflect the Agility that business wants and needs in this age of merger, acquisition, and increasing regulation, we need to be able to almost instantly support the underlying infrastructure implications. Buy a company and you buy a new ERP that manages and records that business. What can you do to absorb the new business? Switch them to your SAP system? Even if that would make your company Agile, you'll be frozen in place for minimum a year or two working to get them up. That's the only way many people, companies, and consultants can think about to achieve that objective. If you buy lots of companies and perhaps sell them again, you MUST be able to very quickly get the key information aligned and the systems sharing appropriate information.

And how can you ever get the latest live information to dashboards instantly, when it comes from a multitude of sources and it needs to be transformed and aligned in order to make sense? Pulling it together via a staging database simply doesn't cut it.

Agility is essential, but it's simply not achievable with the classic approaches to integration.

Agility will only come with a complete mind shift that reinvents the essence of integration. Stone Bond Technologies thinks about integration totally differently with its Enterprise Enabler system. By automating every step along the way and generating the run-time components behind the scenes, the time to implement and maintain an integration is reduced by up to 90%. I know, "90% - you are either crazy or lying!" Well, hear me out a bit. We really do have an off-the shelf product that is the product of 100+ man years of development and is built on a new paradigm. I'll discuss the fundamentals of it in the coming entries.