Agile Integration Serving End Users Directly

Social media is beyond ubiquitous, putting the power of communication and availability of information in the hands of everyone individually and collectively. It has moved from being a thrill for the eager to being a fact of life even for the reluctant. Social sharing of unstructured data is proliferating, and its management is the subject of much conversation, posting, tweeting, and blogging.

But what does this mean and how does it relate to individuals who need structured data? Isn't it time for an end user to be able to access data without calling the boss, who calls the IT department, who calls the consultant, who plans the project, that gets a team, that does an analysis, that includes everything but the kitchen sink, that has a garbage disposer, that's the only thing that works, with which the project finally gets euthanized?

It's time to think about what this means to data integration: lean, mean, or otherwise. These days end users are empowered by SharePoint and BI tools and such to determine for themselves what information they need in order to do their job and make decisions. If an end user can sign in to SAP and to Salesforce and to JD Edwards, why can't they use their SharePoint portal or a dashboard to access the same data from those applications? Of course, one of the key reasons is the issues around security. Historically integration has been behind the scenes, working with no visibility to the actual end user. ETL, of course, has none, and EAI is also not end-user aware. Dashboards are designed for specific users who have permission to see the data displayed. Although the dashboard app may have its own login for security, the integration behind the scenes does what it's told without knowing anything about the user.

Agile Integration Software http://tinyurl.com/3yv85bw supports SSS and other layers of security so that an end user of SharePoint, for example, can see live data with updates passing back to the applications, only if the user has privileges to do so. That same security is available out of box in AIS for any vendor software, BPM, or BI that is designed to read and write data to an ADO.Net entity via an ADO.Net driver, or alternatively by web service calls.

For example, Enterprise Enabler® http://www.enterpriseenabler.com/ is an AIS that can be configured in minutes to merge and align data from multiple applications and generate the entity definition and custom connector for SharePoint (BDC) or generate the entity for the ADO.Net driver that is called from WSS or any other ADO.Net Application. If preferred, it can generate a web service just as easily. The SSS end user security is honored in all these models.

So, in response to the expectations of the "me and everybody else now" surge, integration needs to step up to the plate.

Two Laws of War, IT, and Human Nature

There are two things that I have considered to be fundamental rules of life from as far back as I can remember. In my early twenties, it dawned on me that I actually thought of them with greater conviction than I ever did any of the laws of physics I learned. I decided that these rules held equivalent un-challengeable level of "truth" as laws of physics. Of course I did, in fact challenge every "truth" handed out in my first physics class. How in the world could everyone in the class just sit there and believe that there's some Force pressing up on the floor to keep us from falling through? Because the textbook said so? Hmm. I grew up in Virginia, and everyone knew that the State published the history books, and it was in Virginia's best interest to publish that every President except Lincoln was from Virginia! So, how can you believe there's an equal and opposite arrow somewhere that you can't see?

Now for my two laws, in the order of discovery.

Pamela's Law of Set Points. This may not be a very good name for it, and since I never named it before this moment, I reserve the right to change the name some time. The law is something like this: More things than you can imagine try to obey a set point. The set point may gradually shift over a long time, especially in response fundamental change due to extraordinary events, but in general nature gravitates to that set point. I remember thinking about this law and human nature when I was "into" change management and a person's or institution's capacity for change ("resilience"). While each person or organization has a different level of tolerance, that maximum is essentially constant. The same can be applied to a person's good humor. I believe there's a set point for each of us that is fulfilled by our reactions and attitudes no matter what is actually happening in the real world outside us. Worriers manage to find a certain level of worry, easily filling the void with a new subject of worry as another gets resolved. I'd love to be one of those eternally happy people that have a set point way on the happy side. I guess it's kind of the "glass half full" conundrum, which somehow leads to:

Pamela's Law of Constants. Again a name of the moment. I grew up in the Washington, D.C. area and from the time I was six or so, through high school, I ritualistically read the front page of the Washington Post every single morning (then I read the comics if my siblings were done). The most obvious conclusion to me was that there was always a war somewhere, often in some tiny country I never heard of before. As one subsided (from the headlines) another popped up somewhere else, like the cartoons where the guy with the big hammer pounds down one bump and it pops up somewhere else. That's when I began formulating the theory that unrelated happenings like this really were fulfilling some constant amount of world-wide war. Over the years, I discovered that this same law applies in many, many areas. I could elaborate on the law of constants in the IT world, but that's for another day. You can see, though that if there are constants to be balanced by seemingly unrelated events, that means the events really are related… my crude contribution to the evolution of Chaos theory.

Five Ways to Get Immediate ROI from Agile Integration Software

One of the great things about Agile Integration Software (AIS) is that you can install it and start reaping the benefits immediately.  You don't have to change your whole company's philosophy about middleware. Think about AIS as an adjunct to what's already there, and just solve a couple of important challenges quickly.

Don't know what AIS is? Take a look at this: http://tinyurl.com/3yv85bw

Here are a few ideas that can prove the value of AIS and bring real value to your company before the rest of the team wakes up.

ONE. Get all manner of data into SharePoint. Empower your SharePoint developers and users with secure access to data from all your backend systems. That means that the people that need to see and interact with data that comes from SAP, Salesforce, and Excel, or a custom application, for example, can have that without even declaring a project. Line of business (LOB) applications built in SharePoint invariably need data from multiple systems, and it all needs to be aligned somehow to make sense together. Try AIS for this. Build a couple of interfaces to SharePoint External Lists - BDC (business data catalog) or to SharePoint Lists and see how it goes. The data is secure, and you can even use the same interfaces to handle write-backs to those systems.

TWO. Rescue a project that is way behind schedule. Chances are that the part of the project that's lagging behind is the integration with other systems. Don't displace whatever technology is in being used, but try a rapid parallel path with Agile Integration. Skip the staging database and get the data live from the systems align it on the fly and send it where it needs to go. This is what AIS is designed to do really well. If you find that the easiest way to get started is to go with a staging database that's in place, then use AIS to get the missing data into the repository quickly. Or build the integration and expose it as a web service in two clicks.. Voila! Then sit back and reap the benefits of success (and be a hero).

THREE. Fast-track your BI dashboard project. Let's say you have a dashboard project that has lots of promise. Maybe you just bought a very cool analytics software package that will allow you to understand your business and the market better so you can maintain your competitive advantage. The vendor delivered the software and gave a two-day training on how to configure and use the cool graphs and such. But now they've gone home, and it's up to you to figure out how to get all the data you need into the tool. The more data that's available, the greater the value of this investment. Big problem. Big project, fraught with all the perils of any integration project. Before you spend a year creating the database, see if the tool is just making a call that could be tied to an AIS that could grab the necessary data, align it, and pass it live to the dashboard. That's definitely faster and a much better solution. Otherwise, at least reduce your population of the staging database by 70% by using AIS.

FOUR. Close your books in half the time. If your company is like most, there are multiple ERP or financial systems as well as inventory systems and even spreadsheets involved in the process of consolidating all of the information required to close the books every month. With a changing environment, new entities must be accommodated or eliminated from the process on a regular basis. Using AIS you can automate the capture and alignment of the information from the multiple sources, but even better than that, you can update the integration in a matter of minutes to reflect the changes in your business from month to month or quarter -to-quarter.

FIVE. Improve the productivity of your IT team. Empower your Data Analysts to configure integration without programming. Free up your programmers from doing repetitive integration activities. AIS can eliminate a tremendous amount of activity that simply does not need to be done anymore. You owe it to your company to take advantage of the value of AIS. Start with new data flows that need to be done rather than initiating a lifelong switchover project.

BPM's Nemesis : Clunky Data Integration

Back in the '90's, I jumped right in there with all the other consultants and IT operatives when Business Process Reengineering came along. Since I had been heavily involved in computer graphics before that, I even wrote a crude business process designer so we could abandon the yellow stickies on the walls and halls as well as all the tedious documentation (and re-documentation). BPR evolved to BP'X', and by the time mainstream BPM software came along, I was working on developing software to handle the data flows across all the systems involved in what I called at the time, "Decision Support Chains."

I lost touch for a while with what BPM in general was doing and realized later that most of the products probably actually began with the design and documenting of the processes. I opined that they were started contemporaneously with my similar effort, and were abandoned in favor of Y2K gold. Once that was over, maybe they picked up the old code, dusted it off, and went on from there. Most started with design and evolved to handle complex as well as practical of business process management, but the necessary and tough behind-the-scenes integration has been generally assumed to be outside the discipline. There still appears to be a missing layer or two.

With the great tools now available to design, model, optimize, and automate human workflow, it seems that the toughest part of a full implementation of new processes, or automating old ones, is the ability to get the data where it needs to be when it needs to be there. The necessary overhead and inherent inflexibility of data integration are very often so prohibitive that they kill projects before they even get started.

Agile Integration Software (AIS) working with BPM provides the complete "stack" of necessary functionality. AIS fills in with rapidly configured, adaptable data integration, along with the data logic layer that interacts with the BPM layer, triggering data flows as needed exactly when the process and participating decision support applications require it.

All the overhead of designing and maintaining staging databases goes away along with the resultant data latency. The cost of BPM implementation is greatly reduced, and the solution is able to respond as quickly "under the covers" to business change as it is on the BPM layer!

When BPM inherits all of the features and assets of AIS, BPM become agile, too.

ABPM - Agile Business Process Management? I like it!

Earth Huggers and Underground Sculpture

I need to take a side trip from integration software!

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk http://www.patmonk.com/ told me about his Earth Huggers sculptures last night. These are 16G stainless steel, each 32 by 48 inches. You can see from the pictures that they are embedded in the ground and make inviting "ground pieces." Ground pieces? Well, if you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above, you may see a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and create beautiful sculptures that will have to be unearthed in order to to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. (I have the fish, but don't remember what's inside. I think there's a picture somewhere, though.) Another was a life-sized carved wood woman that he took horizontal cross-sections from and re-assembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like an elongated teardrop. The pointy end is in the ground with a steel plate around it. He claims it's the switch that he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk www.patmonk.com told me about his Earth Huggers sculptures last night. You can see from the pictures that these are embedded in the ground and make charming "ground pieces." If you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

From what I can tell, the earth loves the hugs and, who knows? Maybe they will help keep the earth Green, or at least happy.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above right now, he has a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and have beautiful sculptures that will have to be unearthed to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. Another was a life-sized carved wood woman that he took horizontal cross-sections from and reassembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like a teardrop. The pointy end is in the ground. He claims it's the a switch the he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

Web Services and APIs Just aren't Enough

We've had a number of questions lately about how AppComm technology is different or better than Web services or APIs (Application Programming Interfaces). That's probably because we have been working on packaging Enterprise Enabler® various ways for low end, not-so-demanding point solutions. For example, if you want to access Salesforce.com from a WSS application, you could use the Salesforce.com AppComm with or without another AppComm, and in a few minutes configure a bi-directional ADO.net driver. When you want to get data from Salesforce.com, you just read from and write to the ADO.net object.

The AppComm technology assures simplicity beyond any API or Web service connectivity by leveraging, behind the scenes, all the knowledge of the APIs as well as any necessary coding, scripting and deciphering. The data is simply provided for selection and mapping, eliminating the hard-coding and difficult maintenance necessary when programming to APIs and Web services directly.

Well, that's the whole point! The fundamental philosophy behind Agile Integration Software http://agileintegrationsoftware.blogspot.com/2010/05/characteristics-of-agile-integration.html design hinges on encapsulating absolutely the most intelligence possible behind the scenes so that the tough, complex, or tedious work is only done once. If I have to study the API for SAP's RFCs (Remote Function Calls) to build connectivity components, why not design the connectivity as an AppComm, which means that if I do it well once, no one will ever have to read the API specs again!!! (sorry, for the triple exclamation points - my mother would not have approved, with her proper and perfect usage of the English language, but in this case, I think they are well deserved). Not only do you never have to study the API, but you don't have to understand the prerequisites, assumptions, and in what order all the calls must be made.

Off the shelf, AppComms encapsulate all that knowledge and programming in a flexible and reusable manner, so you can get going quickly. It discovers all the schema information specifically for the particular instance at hand, and allows you to select the data of interest. You can then configure the mapping without programming, which you would need to do without an AppComm.

The truth is, contrary to coerced popular belief, the fact that there exist Web services for accessing an application's data does not mean they are easy to use. And since you are programming instead of configuring, if there's a change to your application's data schema, you have to go back into the code and make changes in order to access new fields, or make sure that your integration doesn't crash because expected data is missing. AppComms know when the schema changes, and alert you to reconfigure.

There are clearly time and cost savings when utilizing the AppComm technology compared to accessing the raw Web services. When you combine the AppComm with its Agile Integration platform, you have the synergy for coordinating data access from multiple applications simultaneously, aligning and transforming the information on the fly and passing it to the destinations without staging.