BPM's Nemesis : Clunky Data Integration

Back in the '90's, I jumped right in there with all the other consultants and IT operatives when Business Process Reengineering came along. Since I had been heavily involved in computer graphics before that, I even wrote a crude business process designer so we could abandon the yellow stickies on the walls and halls as well as all the tedious documentation (and re-documentation). BPR evolved to BP'X', and by the time mainstream BPM software came along, I was working on developing software to handle the data flows across all the systems involved in what I called at the time, "Decision Support Chains."

I lost touch for a while with what BPM in general was doing and realized later that most of the products probably actually began with the design and documenting of the processes. I opined that they were started contemporaneously with my similar effort, and were abandoned in favor of Y2K gold. Once that was over, maybe they picked up the old code, dusted it off, and went on from there. Most started with design and evolved to handle complex as well as practical of business process management, but the necessary and tough behind-the-scenes integration has been generally assumed to be outside the discipline. There still appears to be a missing layer or two.

With the great tools now available to design, model, optimize, and automate human workflow, it seems that the toughest part of a full implementation of new processes, or automating old ones, is the ability to get the data where it needs to be when it needs to be there. The necessary overhead and inherent inflexibility of data integration are very often so prohibitive that they kill projects before they even get started.

Agile Integration Software (AIS) working with BPM provides the complete "stack" of necessary functionality. AIS fills in with rapidly configured, adaptable data integration, along with the data logic layer that interacts with the BPM layer, triggering data flows as needed exactly when the process and participating decision support applications require it.

All the overhead of designing and maintaining staging databases goes away along with the resultant data latency. The cost of BPM implementation is greatly reduced, and the solution is able to respond as quickly "under the covers" to business change as it is on the BPM layer!

When BPM inherits all of the features and assets of AIS, BPM become agile, too.

ABPM - Agile Business Process Management? I like it!

Earth Huggers and Underground Sculpture

I need to take a side trip from integration software!

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk http://www.patmonk.com/ told me about his Earth Huggers sculptures last night. These are 16G stainless steel, each 32 by 48 inches. You can see from the pictures that they are embedded in the ground and make inviting "ground pieces." Ground pieces? Well, if you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above, you may see a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and create beautiful sculptures that will have to be unearthed in order to to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. (I have the fish, but don't remember what's inside. I think there's a picture somewhere, though.) Another was a life-sized carved wood woman that he took horizontal cross-sections from and re-assembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like an elongated teardrop. The pointy end is in the ground with a steel plate around it. He claims it's the switch that he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk www.patmonk.com told me about his Earth Huggers sculptures last night. You can see from the pictures that these are embedded in the ground and make charming "ground pieces." If you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

From what I can tell, the earth loves the hugs and, who knows? Maybe they will help keep the earth Green, or at least happy.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above right now, he has a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and have beautiful sculptures that will have to be unearthed to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. Another was a life-sized carved wood woman that he took horizontal cross-sections from and reassembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like a teardrop. The pointy end is in the ground. He claims it's the a switch the he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

Web Services and APIs Just aren't Enough

We've had a number of questions lately about how AppComm technology is different or better than Web services or APIs (Application Programming Interfaces). That's probably because we have been working on packaging Enterprise Enabler® various ways for low end, not-so-demanding point solutions. For example, if you want to access Salesforce.com from a WSS application, you could use the Salesforce.com AppComm with or without another AppComm, and in a few minutes configure a bi-directional ADO.net driver. When you want to get data from Salesforce.com, you just read from and write to the ADO.net object.

The AppComm technology assures simplicity beyond any API or Web service connectivity by leveraging, behind the scenes, all the knowledge of the APIs as well as any necessary coding, scripting and deciphering. The data is simply provided for selection and mapping, eliminating the hard-coding and difficult maintenance necessary when programming to APIs and Web services directly.

Well, that's the whole point! The fundamental philosophy behind Agile Integration Software http://agileintegrationsoftware.blogspot.com/2010/05/characteristics-of-agile-integration.html design hinges on encapsulating absolutely the most intelligence possible behind the scenes so that the tough, complex, or tedious work is only done once. If I have to study the API for SAP's RFCs (Remote Function Calls) to build connectivity components, why not design the connectivity as an AppComm, which means that if I do it well once, no one will ever have to read the API specs again!!! (sorry, for the triple exclamation points - my mother would not have approved, with her proper and perfect usage of the English language, but in this case, I think they are well deserved). Not only do you never have to study the API, but you don't have to understand the prerequisites, assumptions, and in what order all the calls must be made.

Off the shelf, AppComms encapsulate all that knowledge and programming in a flexible and reusable manner, so you can get going quickly. It discovers all the schema information specifically for the particular instance at hand, and allows you to select the data of interest. You can then configure the mapping without programming, which you would need to do without an AppComm.

The truth is, contrary to coerced popular belief, the fact that there exist Web services for accessing an application's data does not mean they are easy to use. And since you are programming instead of configuring, if there's a change to your application's data schema, you have to go back into the code and make changes in order to access new fields, or make sure that your integration doesn't crash because expected data is missing. AppComms know when the schema changes, and alert you to reconfigure.

There are clearly time and cost savings when utilizing the AppComm technology compared to accessing the raw Web services. When you combine the AppComm with its Agile Integration platform, you have the synergy for coordinating data access from multiple applications simultaneously, aligning and transforming the information on the fly and passing it to the destinations without staging.

AppComms:Removing the Splints from the Octopus

AIS (Agile Integration Software) conjures up the image of information flowing freely wherever it needs to be whenever it needs to be there, aligning and merging with other information as needed by each endpoint person or application. The information is not yesterday's wilting snapshot of the world, or dusty data that has struggled and morphed through five different formats along the way. Think clean, precise, and now.

I contend that the ultimate culprit that works against AIS is the classic "Adapter," along with its cohort "standards." Now please understand that they are not equally culpable, and perhaps neither is, in the perfect world. Unfortunately, as far as I can tell, we haven't reached Utopia yet. An integration model that adopts a specific standard simply pushes the responsibility of dealing with the tough issues outside its world for someone else to handle. That's why Adapters exist; they must get data from whatever/wherever/however it is into a specific destination's requirements ("standard"). The Adapter has no visibility or responsibility to adjust to run time situations. Its job is simply to get predefined data from one application or format to another.

AppComms have a different responsibility; that is as an "application specialist" that can interact with the endpoint application or database at run time as it is instructed by the integration coordinator. Each record or block of data from that endpoint that is needed somewhere in the integrated environment is provided by the AppComm on request, and similarly, data is posted to the destination as requested. AppComms know how to auto-discover the schema for that application or data store, whatever "schema" means for that application; they know how to perform all CRUD (Create, Read, Update, Delete) operations as appropriate (or not) for the specific endpoint application; and they are responsible for generating the reusable metadata defining desired data across scenarios. The specialization of an AppComm is not for the specific instance of an application, but reusable across all instance of that system. For example, a Salesforce.com AppComm will be able to work with any configuration of Salesforce, as opposed to assuming a specific schema.

The elimination of the classic Adapter is like taking splints off an octopus. The integration coordinator can orchestrate a huge range of concurrent information flows performing all kinds of tasks. Multiple live AppComms can be coordinated to access data in an order and timing to resolve cross-application virtual relationships at run time. Imagine eliminating all those staging databases! Adapters simply can not possibly offer the same fluidity and flexibility for integration as AppComms.

Value of Embedding Agile Integration Software (AIS) in Software Products

Is there a software product on the planet that doesn't need to integrate with other data and applications? Well, OK, I downloaded a cool iPhone app that acts as a carpenter's level all by itself. But I can't think of a standalone business application that doesn't need to connect to anything else.

While they all may appear to be the same to a casual observer, in my book there are four types of software product companies:

The 1st Type:

Those that use code bases as a starting point for custom programming services projects. These really are poseurs, pretending to have products, enticing customers, and consuming huge amounts of service dollars. Integration with other systems is the majority of their gravy. There's no point in discussing the value of AIS to this first category; their business model is based on hourly services, so reducing the time to integrate by 60-90 percent would be a curse.

The 2nd Type:

Those with off-the-shelf products that leave the integration to the customer.

• These products usually operate against a database that the customer is responsible to populate and maintain.
• These companies lose sales because the customer does not want to undertake the time and cost of developing and maintaining the necessary integration.

Value of Agile Integration Software:

• Eliminate the imperative of a staging database and take advantage of read using live data and write-backs to customers' existing applications.
• Increase the percentage of sales closed by greatly reducing the integration effort.

The 3rd Type:

Those that have off-the-shelf products but still count on their integration services as a welcome and important revenue stream.

Value of using Agile Integration Software for your projects:

• Reduce the risk associated with the inevitable unknowns of integration projects.
• Fixed bid projects can be offered with lower risk and high margins.
• Where staging databases are used, the product can enter a new realm of interactivity.
• Competitive advantage with expanded capabilities and lower total cost of implementation.

The 4th Type:

Those that have off-the-shelf products that embed or package with an agile integration software (AIS). These companies are already reaping the benefits of agile integration and are becoming the leaders in their domains.

Benefits these companies are getting from Agile Integration Software:

• Reduced overall cost to implement their products.
• You can focus more on your own product and domain expertise, and not have to invent other ways to integrate with customer's systems.
• Reduce the risks associated with integration.
• Fixed bid projects can be offered with lower risk and high margins.
• Eliminate the overhead of staging databases.
• Re-brand to your own name so there is no need to also sell a third party integration tool.
• Expand the scope and value of their products by enabling information/data sharing back to customers' systems.

Now that Agile Integration Software really does exist, you owe it to your shareholders to look into the bottom line benefits that it can bring to your business. We are seeing rapidly growing adoption of this paradigm, and we are seeing these companies reap the advantage. Maybe they'll make the sale you can't because of the painful integration your implementation requires.

From Dashboard to Control Center with AIS

What's wrong with dashboards today? Nothing really, but stretch the limits by adding a few more dimensions of reality and they can fly! Maybe that's the difference between a car's dashboard and the dashboard in the cockpit of an airplane. One just displays the situation and the other allows you to interact and control the airplane.

Executive and management dashboards have become pervasive and invaluable contributors to many businesses' decision making by providing access to a broad range of key corporate and specialized data in easily assimilated visual screens. Today's dashboards are mostly like cars. If you're lucky, you get real time data displayed, along with relevant analysis so that you can make operational or strategic decisions. When you make decisions in a car, you react by letting up on the gas pedal or stopping at the gas station. With business dashboards, you make decisions and then have to go to another system to update data, or send an email or otherwise convey or initiate execution of those decisions. Why can't you act on your decisions from the dashboard? Why do dashboard have to be "read-only?"

With data coming from a range of sources, the prevailing approach to accessing the data from multiple backends is for the dashboard vendor to tell the customer that it’s in their court. (Just get all the data in one database so they can read from it and provide awesome analysis and visualization. I wish I had a dollar for every sale that was lost on that alone! )

Even if it were easy to do, that staging database is the inherent problem with dashboards that limits their usefulness. Whether the vendor develops and populates it or the customer does, there is a specs effort, a design effort, and ongoing implementation effort. Populating it requires a huge programming effort. But the biggest problem is that you simply cannot display current, live data, and you cannot really even think about the data being updated in the dashboard and sent back instantly to the backend systems.

Instead, if you use an Agile Integration Software to implement the dashboard integration, you can automatically have secure, bi-directional data flows. You can eliminate all of the overhead associated with the database design, implementation, and maintenance. That means a rapid dashboard implementation project, and the ability to turn the dashboard into a control center where decisions can not only be made, but can be executed. Maybe you can't even call it a dashboard anymore!

ISVs are rebranding and packaging AIS with their products or embedding into their products using an Agile Integration Software ADO.Net driver. See Stone Bond's Enterprise Enabler