Agile Integration Serving End Users Directly

Social media is beyond ubiquitous, putting the power of communication and availability of information in the hands of everyone individually and collectively. It has moved from being a thrill for the eager to being a fact of life even for the reluctant. Social sharing of unstructured data is proliferating, and its management is the subject of much conversation, posting, tweeting, and blogging.

But what does this mean and how does it relate to individuals who need structured data? Isn't it time for an end user to be able to access data without calling the boss, who calls the IT department, who calls the consultant, who plans the project, that gets a team, that does an analysis, that includes everything but the kitchen sink, that has a garbage disposer, that's the only thing that works, with which the project finally gets euthanized?

It's time to think about what this means to data integration: lean, mean, or otherwise. These days end users are empowered by SharePoint and BI tools and such to determine for themselves what information they need in order to do their job and make decisions. If an end user can sign in to SAP and to Salesforce and to JD Edwards, why can't they use their SharePoint portal or a dashboard to access the same data from those applications? Of course, one of the key reasons is the issues around security. Historically integration has been behind the scenes, working with no visibility to the actual end user. ETL, of course, has none, and EAI is also not end-user aware. Dashboards are designed for specific users who have permission to see the data displayed. Although the dashboard app may have its own login for security, the integration behind the scenes does what it's told without knowing anything about the user.

Agile Integration Software http://tinyurl.com/3yv85bw supports SSS and other layers of security so that an end user of SharePoint, for example, can see live data with updates passing back to the applications, only if the user has privileges to do so. That same security is available out of box in AIS for any vendor software, BPM, or BI that is designed to read and write data to an ADO.Net entity via an ADO.Net driver, or alternatively by web service calls.

For example, Enterprise Enabler® http://www.enterpriseenabler.com/ is an AIS that can be configured in minutes to merge and align data from multiple applications and generate the entity definition and custom connector for SharePoint (BDC) or generate the entity for the ADO.Net driver that is called from WSS or any other ADO.Net Application. If preferred, it can generate a web service just as easily. The SSS end user security is honored in all these models.

So, in response to the expectations of the "me and everybody else now" surge, integration needs to step up to the plate.

Two Laws of War, IT, and Human Nature

There are two things that I have considered to be fundamental rules of life from as far back as I can remember. In my early twenties, it dawned on me that I actually thought of them with greater conviction than I ever did any of the laws of physics I learned. I decided that these rules held equivalent un-challengeable level of "truth" as laws of physics. Of course I did, in fact challenge every "truth" handed out in my first physics class. How in the world could everyone in the class just sit there and believe that there's some Force pressing up on the floor to keep us from falling through? Because the textbook said so? Hmm. I grew up in Virginia, and everyone knew that the State published the history books, and it was in Virginia's best interest to publish that every President except Lincoln was from Virginia! So, how can you believe there's an equal and opposite arrow somewhere that you can't see?

Now for my two laws, in the order of discovery.

Pamela's Law of Set Points. This may not be a very good name for it, and since I never named it before this moment, I reserve the right to change the name some time. The law is something like this: More things than you can imagine try to obey a set point. The set point may gradually shift over a long time, especially in response fundamental change due to extraordinary events, but in general nature gravitates to that set point. I remember thinking about this law and human nature when I was "into" change management and a person's or institution's capacity for change ("resilience"). While each person or organization has a different level of tolerance, that maximum is essentially constant. The same can be applied to a person's good humor. I believe there's a set point for each of us that is fulfilled by our reactions and attitudes no matter what is actually happening in the real world outside us. Worriers manage to find a certain level of worry, easily filling the void with a new subject of worry as another gets resolved. I'd love to be one of those eternally happy people that have a set point way on the happy side. I guess it's kind of the "glass half full" conundrum, which somehow leads to:

Pamela's Law of Constants. Again a name of the moment. I grew up in the Washington, D.C. area and from the time I was six or so, through high school, I ritualistically read the front page of the Washington Post every single morning (then I read the comics if my siblings were done). The most obvious conclusion to me was that there was always a war somewhere, often in some tiny country I never heard of before. As one subsided (from the headlines) another popped up somewhere else, like the cartoons where the guy with the big hammer pounds down one bump and it pops up somewhere else. That's when I began formulating the theory that unrelated happenings like this really were fulfilling some constant amount of world-wide war. Over the years, I discovered that this same law applies in many, many areas. I could elaborate on the law of constants in the IT world, but that's for another day. You can see, though that if there are constants to be balanced by seemingly unrelated events, that means the events really are related… my crude contribution to the evolution of Chaos theory.

Five Ways to Get Immediate ROI from Agile Integration Software

One of the great things about Agile Integration Software (AIS) is that you can install it and start reaping the benefits immediately.  You don't have to change your whole company's philosophy about middleware. Think about AIS as an adjunct to what's already there, and just solve a couple of important challenges quickly.

Don't know what AIS is? Take a look at this: http://tinyurl.com/3yv85bw

Here are a few ideas that can prove the value of AIS and bring real value to your company before the rest of the team wakes up.

ONE. Get all manner of data into SharePoint. Empower your SharePoint developers and users with secure access to data from all your backend systems. That means that the people that need to see and interact with data that comes from SAP, Salesforce, and Excel, or a custom application, for example, can have that without even declaring a project. Line of business (LOB) applications built in SharePoint invariably need data from multiple systems, and it all needs to be aligned somehow to make sense together. Try AIS for this. Build a couple of interfaces to SharePoint External Lists - BDC (business data catalog) or to SharePoint Lists and see how it goes. The data is secure, and you can even use the same interfaces to handle write-backs to those systems.

TWO. Rescue a project that is way behind schedule. Chances are that the part of the project that's lagging behind is the integration with other systems. Don't displace whatever technology is in being used, but try a rapid parallel path with Agile Integration. Skip the staging database and get the data live from the systems align it on the fly and send it where it needs to go. This is what AIS is designed to do really well. If you find that the easiest way to get started is to go with a staging database that's in place, then use AIS to get the missing data into the repository quickly. Or build the integration and expose it as a web service in two clicks.. Voila! Then sit back and reap the benefits of success (and be a hero).

THREE. Fast-track your BI dashboard project. Let's say you have a dashboard project that has lots of promise. Maybe you just bought a very cool analytics software package that will allow you to understand your business and the market better so you can maintain your competitive advantage. The vendor delivered the software and gave a two-day training on how to configure and use the cool graphs and such. But now they've gone home, and it's up to you to figure out how to get all the data you need into the tool. The more data that's available, the greater the value of this investment. Big problem. Big project, fraught with all the perils of any integration project. Before you spend a year creating the database, see if the tool is just making a call that could be tied to an AIS that could grab the necessary data, align it, and pass it live to the dashboard. That's definitely faster and a much better solution. Otherwise, at least reduce your population of the staging database by 70% by using AIS.

FOUR. Close your books in half the time. If your company is like most, there are multiple ERP or financial systems as well as inventory systems and even spreadsheets involved in the process of consolidating all of the information required to close the books every month. With a changing environment, new entities must be accommodated or eliminated from the process on a regular basis. Using AIS you can automate the capture and alignment of the information from the multiple sources, but even better than that, you can update the integration in a matter of minutes to reflect the changes in your business from month to month or quarter -to-quarter.

FIVE. Improve the productivity of your IT team. Empower your Data Analysts to configure integration without programming. Free up your programmers from doing repetitive integration activities. AIS can eliminate a tremendous amount of activity that simply does not need to be done anymore. You owe it to your company to take advantage of the value of AIS. Start with new data flows that need to be done rather than initiating a lifelong switchover project.

BPM's Nemesis : Clunky Data Integration

Back in the '90's, I jumped right in there with all the other consultants and IT operatives when Business Process Reengineering came along. Since I had been heavily involved in computer graphics before that, I even wrote a crude business process designer so we could abandon the yellow stickies on the walls and halls as well as all the tedious documentation (and re-documentation). BPR evolved to BP'X', and by the time mainstream BPM software came along, I was working on developing software to handle the data flows across all the systems involved in what I called at the time, "Decision Support Chains."

I lost touch for a while with what BPM in general was doing and realized later that most of the products probably actually began with the design and documenting of the processes. I opined that they were started contemporaneously with my similar effort, and were abandoned in favor of Y2K gold. Once that was over, maybe they picked up the old code, dusted it off, and went on from there. Most started with design and evolved to handle complex as well as practical of business process management, but the necessary and tough behind-the-scenes integration has been generally assumed to be outside the discipline. There still appears to be a missing layer or two.

With the great tools now available to design, model, optimize, and automate human workflow, it seems that the toughest part of a full implementation of new processes, or automating old ones, is the ability to get the data where it needs to be when it needs to be there. The necessary overhead and inherent inflexibility of data integration are very often so prohibitive that they kill projects before they even get started.

Agile Integration Software (AIS) working with BPM provides the complete "stack" of necessary functionality. AIS fills in with rapidly configured, adaptable data integration, along with the data logic layer that interacts with the BPM layer, triggering data flows as needed exactly when the process and participating decision support applications require it.

All the overhead of designing and maintaining staging databases goes away along with the resultant data latency. The cost of BPM implementation is greatly reduced, and the solution is able to respond as quickly "under the covers" to business change as it is on the BPM layer!

When BPM inherits all of the features and assets of AIS, BPM become agile, too.

ABPM - Agile Business Process Management? I like it!

Earth Huggers and Underground Sculpture

I need to take a side trip from integration software!

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk http://www.patmonk.com/ told me about his Earth Huggers sculptures last night. These are 16G stainless steel, each 32 by 48 inches. You can see from the pictures that they are embedded in the ground and make inviting "ground pieces." Ground pieces? Well, if you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above, you may see a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and create beautiful sculptures that will have to be unearthed in order to to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. (I have the fish, but don't remember what's inside. I think there's a picture somewhere, though.) Another was a life-sized carved wood woman that he took horizontal cross-sections from and re-assembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like an elongated teardrop. The pointy end is in the ground with a steel plate around it. He claims it's the switch that he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

It's really hard to keep up with my favorite physicist-turned-artist. Actually he's always been an artist, and once a physicist, there's no turning back. Pat Monk www.patmonk.com told me about his Earth Huggers sculptures last night. You can see from the pictures that these are embedded in the ground and make charming "ground pieces." If you talk about "wall pieces," why not talk about ground pieces, although no one I know ever does.

From what I can tell, the earth loves the hugs and, who knows? Maybe they will help keep the earth Green, or at least happy.

Pat went on to tell me about his newest phase, which he calls "Underground Sculpture." If you check the link above right now, he has a picture of one called "Double Mushroom" under construction. He told me that he's nearly done now, and that he inverted it and will bury it a couple of feet, so just the mushrooms will be above ground. I'm a little worried that he will go off the deep end and have beautiful sculptures that will have to be unearthed to see them. Sounds crazy, but I know this man.

A few years ago he had an "unsculpture" phase. He very creatively disposed of some of his older sculptures by deconstructing them in all sorts of ways. He hid one inside a big cast concrete fish. Another was a life-sized carved wood woman that he took horizontal cross-sections from and reassembled as a table top, which he covered with glass. Actually that one was very nice.

One of my favorites of Pat's sculptures is a stainless steel piece about 18 inches long, shaped like a teardrop. The pointy end is in the ground. He claims it's the a switch the he throws to get the earth on track each vernal and autumnal equinox.

Pat's my father, so I believe everything he tells me.

Web Services and APIs Just aren't Enough

We've had a number of questions lately about how AppComm technology is different or better than Web services or APIs (Application Programming Interfaces). That's probably because we have been working on packaging Enterprise Enabler® various ways for low end, not-so-demanding point solutions. For example, if you want to access Salesforce.com from a WSS application, you could use the Salesforce.com AppComm with or without another AppComm, and in a few minutes configure a bi-directional ADO.net driver. When you want to get data from Salesforce.com, you just read from and write to the ADO.net object.

The AppComm technology assures simplicity beyond any API or Web service connectivity by leveraging, behind the scenes, all the knowledge of the APIs as well as any necessary coding, scripting and deciphering. The data is simply provided for selection and mapping, eliminating the hard-coding and difficult maintenance necessary when programming to APIs and Web services directly.

Well, that's the whole point! The fundamental philosophy behind Agile Integration Software http://agileintegrationsoftware.blogspot.com/2010/05/characteristics-of-agile-integration.html design hinges on encapsulating absolutely the most intelligence possible behind the scenes so that the tough, complex, or tedious work is only done once. If I have to study the API for SAP's RFCs (Remote Function Calls) to build connectivity components, why not design the connectivity as an AppComm, which means that if I do it well once, no one will ever have to read the API specs again!!! (sorry, for the triple exclamation points - my mother would not have approved, with her proper and perfect usage of the English language, but in this case, I think they are well deserved). Not only do you never have to study the API, but you don't have to understand the prerequisites, assumptions, and in what order all the calls must be made.

Off the shelf, AppComms encapsulate all that knowledge and programming in a flexible and reusable manner, so you can get going quickly. It discovers all the schema information specifically for the particular instance at hand, and allows you to select the data of interest. You can then configure the mapping without programming, which you would need to do without an AppComm.

The truth is, contrary to coerced popular belief, the fact that there exist Web services for accessing an application's data does not mean they are easy to use. And since you are programming instead of configuring, if there's a change to your application's data schema, you have to go back into the code and make changes in order to access new fields, or make sure that your integration doesn't crash because expected data is missing. AppComms know when the schema changes, and alert you to reconfigure.

There are clearly time and cost savings when utilizing the AppComm technology compared to accessing the raw Web services. When you combine the AppComm with its Agile Integration platform, you have the synergy for coordinating data access from multiple applications simultaneously, aligning and transforming the information on the fly and passing it to the destinations without staging.

AppComms:Removing the Splints from the Octopus

AIS (Agile Integration Software) conjures up the image of information flowing freely wherever it needs to be whenever it needs to be there, aligning and merging with other information as needed by each endpoint person or application. The information is not yesterday's wilting snapshot of the world, or dusty data that has struggled and morphed through five different formats along the way. Think clean, precise, and now.

I contend that the ultimate culprit that works against AIS is the classic "Adapter," along with its cohort "standards." Now please understand that they are not equally culpable, and perhaps neither is, in the perfect world. Unfortunately, as far as I can tell, we haven't reached Utopia yet. An integration model that adopts a specific standard simply pushes the responsibility of dealing with the tough issues outside its world for someone else to handle. That's why Adapters exist; they must get data from whatever/wherever/however it is into a specific destination's requirements ("standard"). The Adapter has no visibility or responsibility to adjust to run time situations. Its job is simply to get predefined data from one application or format to another.

AppComms have a different responsibility; that is as an "application specialist" that can interact with the endpoint application or database at run time as it is instructed by the integration coordinator. Each record or block of data from that endpoint that is needed somewhere in the integrated environment is provided by the AppComm on request, and similarly, data is posted to the destination as requested. AppComms know how to auto-discover the schema for that application or data store, whatever "schema" means for that application; they know how to perform all CRUD (Create, Read, Update, Delete) operations as appropriate (or not) for the specific endpoint application; and they are responsible for generating the reusable metadata defining desired data across scenarios. The specialization of an AppComm is not for the specific instance of an application, but reusable across all instance of that system. For example, a Salesforce.com AppComm will be able to work with any configuration of Salesforce, as opposed to assuming a specific schema.

The elimination of the classic Adapter is like taking splints off an octopus. The integration coordinator can orchestrate a huge range of concurrent information flows performing all kinds of tasks. Multiple live AppComms can be coordinated to access data in an order and timing to resolve cross-application virtual relationships at run time. Imagine eliminating all those staging databases! Adapters simply can not possibly offer the same fluidity and flexibility for integration as AppComms.

Value of Embedding Agile Integration Software (AIS) in Software Products

Is there a software product on the planet that doesn't need to integrate with other data and applications? Well, OK, I downloaded a cool iPhone app that acts as a carpenter's level all by itself. But I can't think of a standalone business application that doesn't need to connect to anything else.

While they all may appear to be the same to a casual observer, in my book there are four types of software product companies:

The 1st Type:

Those that use code bases as a starting point for custom programming services projects. These really are poseurs, pretending to have products, enticing customers, and consuming huge amounts of service dollars. Integration with other systems is the majority of their gravy. There's no point in discussing the value of AIS to this first category; their business model is based on hourly services, so reducing the time to integrate by 60-90 percent would be a curse.

The 2nd Type:

Those with off-the-shelf products that leave the integration to the customer.

• These products usually operate against a database that the customer is responsible to populate and maintain.
• These companies lose sales because the customer does not want to undertake the time and cost of developing and maintaining the necessary integration.

Value of Agile Integration Software:

• Eliminate the imperative of a staging database and take advantage of read using live data and write-backs to customers' existing applications.
• Increase the percentage of sales closed by greatly reducing the integration effort.

The 3rd Type:

Those that have off-the-shelf products but still count on their integration services as a welcome and important revenue stream.

Value of using Agile Integration Software for your projects:

• Reduce the risk associated with the inevitable unknowns of integration projects.
• Fixed bid projects can be offered with lower risk and high margins.
• Where staging databases are used, the product can enter a new realm of interactivity.
• Competitive advantage with expanded capabilities and lower total cost of implementation.

The 4th Type:

Those that have off-the-shelf products that embed or package with an agile integration software (AIS). These companies are already reaping the benefits of agile integration and are becoming the leaders in their domains.

Benefits these companies are getting from Agile Integration Software:

• Reduced overall cost to implement their products.
• You can focus more on your own product and domain expertise, and not have to invent other ways to integrate with customer's systems.
• Reduce the risks associated with integration.
• Fixed bid projects can be offered with lower risk and high margins.
• Eliminate the overhead of staging databases.
• Re-brand to your own name so there is no need to also sell a third party integration tool.
• Expand the scope and value of their products by enabling information/data sharing back to customers' systems.

Now that Agile Integration Software really does exist, you owe it to your shareholders to look into the bottom line benefits that it can bring to your business. We are seeing rapidly growing adoption of this paradigm, and we are seeing these companies reap the advantage. Maybe they'll make the sale you can't because of the painful integration your implementation requires.

From Dashboard to Control Center with AIS

What's wrong with dashboards today? Nothing really, but stretch the limits by adding a few more dimensions of reality and they can fly! Maybe that's the difference between a car's dashboard and the dashboard in the cockpit of an airplane. One just displays the situation and the other allows you to interact and control the airplane.

Executive and management dashboards have become pervasive and invaluable contributors to many businesses' decision making by providing access to a broad range of key corporate and specialized data in easily assimilated visual screens. Today's dashboards are mostly like cars. If you're lucky, you get real time data displayed, along with relevant analysis so that you can make operational or strategic decisions. When you make decisions in a car, you react by letting up on the gas pedal or stopping at the gas station. With business dashboards, you make decisions and then have to go to another system to update data, or send an email or otherwise convey or initiate execution of those decisions. Why can't you act on your decisions from the dashboard? Why do dashboard have to be "read-only?"

With data coming from a range of sources, the prevailing approach to accessing the data from multiple backends is for the dashboard vendor to tell the customer that it’s in their court. (Just get all the data in one database so they can read from it and provide awesome analysis and visualization. I wish I had a dollar for every sale that was lost on that alone! )

Even if it were easy to do, that staging database is the inherent problem with dashboards that limits their usefulness. Whether the vendor develops and populates it or the customer does, there is a specs effort, a design effort, and ongoing implementation effort. Populating it requires a huge programming effort. But the biggest problem is that you simply cannot display current, live data, and you cannot really even think about the data being updated in the dashboard and sent back instantly to the backend systems.

Instead, if you use an Agile Integration Software to implement the dashboard integration, you can automatically have secure, bi-directional data flows. You can eliminate all of the overhead associated with the database design, implementation, and maintenance. That means a rapid dashboard implementation project, and the ability to turn the dashboard into a control center where decisions can not only be made, but can be executed. Maybe you can't even call it a dashboard anymore!

ISVs are rebranding and packaging AIS with their products or embedding into their products using an Agile Integration Software ADO.Net driver. See Stone Bond's Enterprise Enabler

Agile Integration for SharePoint : Our Dev Kitchen Experience

They used to call it the "Death Kitchen," but now they use the kinder and gentler "Dev Kitchen." That was our first introduction in person to the Microsoft SharePoint 2010 development team, back in January 2009. We had been working for about a year with them, preparing our agile integration product, Enterprise Enabler, to connect tightly with the new version of SharePoint. We tied our cool technology to their cool technology via their next generation of Business Data Catalog (BDC), which incorporated full CRUD (Create, Read Update, and Delete) capabilities. Before Dev Kitchen, we had worked with notes and samples about BDC from their team, to auto-generate the cryptic and complicated BDC XML file, including services that we auto-generated to initiate and leverage our powerful integration capabilities.

By the time January came, we had validated some of our work, but hadn't really been able to test it much. The whole Dev Kitchen experience was quite interesting and exciting for our team, as we had little idea what to expect. We were one of less than twenty companies invited to participate that week. Some were product companies, some were customers, but we were the only one doing this kind of tight connectivity. Each company had its own separate and secure room; only our team knew the access code to open our door. There was an air of excitement and secrecy, but mostly everyone was hard at work testing against a very early version of "O14."

One of the most amazing things was working side-by-side with the developers and visionaries of the Business Connectivity Services (BCS). If we ran into a snag, within a couple of minutes the specialist for that particular feature of BCS was in the room with us tracing the source of the issue and clarifying to us or making instant fixes if they needed to. Our work spanned the full range of BCS features, so we met and worked with quite a number of the team members. What an exciting, vibrant, enthusiastic group of people!

We had, as I recall, two and a half days to finish our work. On the last day we all gathered in a large room and took turns presenting to everyone the things we had been working on. We had ten minutes to do it. When Mike Guillory, our head SharePoint technologist spoke, the room was packed with participants and Microsoft developers, architects, and management. In ten minutes we showed building from ground up a mashup of data live from Salesforce.com, merging and aligning it with data from a SQL Server database, auto-generating the BDC , and configuring an out-of box web part for the data. We opened the window in SharePoint, and Voila! The data appeared from both sources. We edited a couple of fields, and showed that back in Salesforce and the database, the fields were updated!

Spontaneous standing ovation from the Microsoft team! Later that evening when we were invited to show a little more detail to part of the team, someone commented that what he was seeing was what he had dreamed about his whole career. We later found out he was a very high level Microsoft director. All in all, it was a rewarding experience, and since then, we have tied up all the loose ends with the pre-alpha, alpha, beta, and now production versions, and have added security with SSS and transaction assurance and rollback. The Microsoft team advised on the architecture, recommending the "Custom Connector," which resides on the SharePoint server for tightest possible security and best performance possible.

Now, even the TAP program has ended, and we're all in full production. Thanks and congratulations to architect Mohammed Nazeeruddin and his team, who met with us every week to ensure the best fit for our embeddable integration.

B2B : Brain-to-Brain

My generation was not born with cell phones. While on vacation for a few days , I tried to disconnect from the digital world with only mild success. Not a good time to be reading Born Digital: Understanding the First Generation of Digital Natives, by John Palfrey http://www.amazon.com/Born-Digital-Understanding-Generation-Natives/dp/0465005152, if I didn't want to think about this stuff. When I was growing up, telephones were ultimately connected by oodles of wires all over the planet. We never even thought of wireless telephones.

We did, though, talk a lot about ESP (extra-sensory perception, for those of you too young to remember), and we debated about whether psychics were communicating, or whether they cheated. I wonder if ESP even seems ridiculous to a young person who has always known cell phones. You don't have to even be in the same room to hear what the other person is saying. No wires. I bet it's not such a stretch to believe that you could have direct brain-to-brain communications. After all, knowing what they know now, why not? Think about it - we wouldn't even need to talk, but just think about talking. Read just a little about all the new research on brain placticity, and you have to wonder if brain-to-brain communications is just a matter of practice.

Then you have to wonder what brain-to-brain (B2B) means in the integration space. We'll be transforming everything from Venus to Mars and back again.

Agile Integration Software Means New Best Practices

One of the most exciting benefits of Agile Integration Software is the ability to dramatically streamline the analysis and other processes around designing and implementing integration solutions. If RAD (Rapid Application Development) is fundamental to the essence of the software, then it is natural that a new genre of best practices will emerge leveraging the agility of the software.

Enterprise Enabler® has "fuzzied" the lines between the classic steps of analysis, design, development, testing, and deployment. I'd like to share a couple of ways that it is changing the way teams develop integrated solutions.

Analysis Workshops replace Specs. "What?!!!," you say. "That's heresy! No self-respecting IT department would eliminate writing specs!" Ok, so hear me out. Everyone knows that the most time consuming, messiest, most unpleasant part of the whole project is defining the specs, especially the mapping specs. Well, not just defining them, but documenting them, running them by everyone for approval, re-working them, running them by everyone for approval (and so on), and even worse, keeping them up to date over time. Wouldn't you love for that whole exercise to be history, and yet to be able to get the exact current definition any time? With Enterprise Enabler, the specs are synonymous with the solution.

Here we go. You're in a conference room (or web meeting) with three other people, Tom, Dick, and Jane. You are the project manager or an analyst, or even a programmer. You're on the keyboard with Enterprise Enabler's user interface projected on the screen. With you, remotely or in the room, are the three business analysts that really know their systems' data and how it is used. Think how many questions can be resolved in a very short time if these key people can discuss what they require while you build it!

Pass the doughnuts around while we define the big picture of the integration we are solving. Jane is the expert on the destination application, "D," so in my book, she's the boss. After all, the destination's the only the application that cares about the data in an integration. Tom owns source "A" and Dick owns source "B." You pull up the template builder to define the metadata for the appropriate data from "A." Tom points you to the instance of the system that you will be working with and you show him the schema as he indicates what is of interest from his app. Then you do the same with Dick, and then with Jane for the destination. She has indicated what needs to be populated. You go to the mapper and everyone can see on the left all the fields from both sources and on the right, the fields to be filled. First Dick and Tom help you define the cross-application relationship by identifying the appropriate key fields, while you drag and drop them to create the virtual relationship. Now you continue to work with these experts to build and test the data mapping en situ. Tom discovers he left out a couple of tables he need, so you bring up the template and add the tables, and you're back in business. You run the map through and let Tom, Dick, and Jane discuss the nuances of whichever data are not obvious until it is correct. You're done, maybe even before the doughnuts are finished. And the programmed mapping will always match exactly what you see on the screen. When Jane changes her mind or Dick's application has updates that change the schema, you simply go in an reflect it in a couple of minutes.

Integration will become a set of well defined and productive workshops with all the right people, wherever they are. You'll only reminisce about the old days and how difficult it was to get the specs from each expert, document them, and go from one to the other, changing and updating, and then not being able to find the specs document to be able to keep it up to date. Remember when the programmer would discover a problem with the specs and would fix the code, but the change never got back to the specs? Ah... for these headaches to be history!

Coercing the Mind Shift to Comprehend the Paradigm Shift

The mind certainly is stubborn. Something about re-directing those synapses to another mindset is astoundingly challenging. Just like the synapses connect "sphere" and "bounce" to the concept of "ball," "enterprise integration" is only recognizable as "integration" if it looks like a heavy, time-consuming, difficult, expensive endeavor. So what if a cubical ball could actually bounce even better than a spherical one? Even if I see it, I can't make the "connection." Well, I don't know much about synapses, but I sure have had lots of experience trying to initiate the plasticity to reformulate the way people think about integration!

Recently we presented a bit about our technology to a handful of people who are extremely well-informed about the integration space. We planned to hit a few points that we are pretty sure can't be done with other products. We zeroed in on one specific point we thought would amaze anyone who really knows the current state of integration and what is possible and what is not. This is a capability that is a natural fall-out of our new paradigm: we showed how to create virtual relationships across totally different systems even faster than you would define a relationship across tables in a single database.

In ten seconds we showed creating a virtual relationship between Microsoft Dynamics CRM and a SQL database. In that ten seconds, all the run time components were generated behind the scenes, so we ran it. This scenario happened to be sending data to an out-of-box SharePoint 2010 web part; it could have gone to any other application, dashboard, web service, or pumped to a messaging system .

We showed how an end user, from a single SharePoint web part (screen), can see live data from multiple applications, aligned and transformed as appropriate for their usage. As if that weren't enough, we showed updating a couple of fields in SharePoint, and the backend systems were immediately updated!

No staging of the data, no copy made at all, anywhere. The data was accessed, aligned, displayed, changed, and updated back in the backend systems. Our product, Enterprise Enabler, eliminates a HUGE amount of time, manpower, and cost by eliminating the need for a staging database any time you need data merged from two or more disparate systems. The systems could be SAP, Salesforce, XML, spreadsheets… it doesn't matter. So, no one needs to design a database model that is everything to everybody. No one needs to build the database, and no one needs to maintain it.

A response came from our audience that many enterprise integration platforms can do this! I would love to do a "side-by-side" bake-off with the cited competitors, Websphere or Tibco, any day. I am extremely confident that we would probably have to go home before their years of work are done. If they even can do it.

In the end I have to write the response off to the discipline's persistent twenty-plus year track record that has forced consistent synapses dancing around in the same circle, convincing everyone that there is just no other way integration can possibly be done. If Websphere, Tibco, Informatica, and others can't do it, we'll just pretend they can!

Characteristics of Agile Integration Software

One of my objectives in starting this blog is to see what I can do to overcome the pervasive mindset, with respect to integration in general, that conjures up a heavy, limited framework that can only actually work with a huge amount of custom coding. It is only by stepping outside that box that we can imagine the next generation of integration software.

What should one look for in an integration product that indicates that it will actually be able to generate an agile environment? Here are a few of my thoughts.

• First, it must be "off the shelf" or "out of box" or downloadable, whatever term you like. That means that you can install it in just a few minutes. If it takes a week or months to install, you are definitely not going to have an agile environment!

• It must require little or no training to get started building connectivity. A data analyst should be able to design and build data mapping and transfers. This means the people who know what the result really needs to be can actually do the implementation.

• You must be able to design, develop, test, and deploy from within a single environment. No separate effort to generate and assemble a run-time components.

• It must support accessing, transforming, and aligning data from multiple sources live without staging in a database or interim store. Real time data access and availability is a critical aspect of an agile enterprise. Think what work it takes and what value is lost when the data must be staged before it is transferred to wherever it is going.

• It must be end-to-end metadata driven.

• If there is any programming required, it should be doable from within the same environment, and the code saved and versioned as part of the metadata.

• The transformation engine should be able to operate on data in any application's native format as opposed to having to step through a central view such as XML. How can you possibly get high performance when your integration must run through multiple steps.

• If you are looking for agility at an Enterprise level, the product must be scalable across clusters and extensible so that it becomes an environment tailored with all the special requirement you have that are needed over and over.
  
  I guess that's plenty for now. Over time, I imagine I'll discuss most of these points in more detail, and how they are present in our Enterprise Enabler agile integration software.
  

Is "Agile Integration" an Oxymoron?

Most people would agree that "Agile Integration" is an oxymoron. The heavy footprint, heavy customization, and huge amounts of interdependence across an integrated enterprise absolutely negate the possibility of Agility! But we discuss Agile Enterprises as if you could instantly have an intelligent SOA implementation that detects and adjusts to business changes, maybe even anticipating the evolution of the business. What are we talking about, anyway? All the SOA initiatives I've heard about are isolated subsets of business solutions that impose the web services environment (which takes lots of design, development, and definitely anticipation of everything you will want to do in the future). The "agility" offered is limited to a specific, known set of options.

If we want to reflect the Agility that business wants and needs in this age of merger, acquisition, and increasing regulation, we need to be able to almost instantly support the underlying infrastructure implications. Buy a company and you buy a new ERP that manages and records that business. What can you do to absorb the new business? Switch them to your SAP system? Even if that would make your company Agile, you'll be frozen in place for minimum a year or two working to get them up. That's the only way many people, companies, and consultants can think about to achieve that objective. If you buy lots of companies and perhaps sell them again, you MUST be able to very quickly get the key information aligned and the systems sharing appropriate information.

And how can you ever get the latest live information to dashboards instantly, when it comes from a multitude of sources and it needs to be transformed and aligned in order to make sense? Pulling it together via a staging database simply doesn't cut it.

Agility is essential, but it's simply not achievable with the classic approaches to integration.

Agility will only come with a complete mind shift that reinvents the essence of integration. Stone Bond Technologies thinks about integration totally differently with its Enterprise Enabler system. By automating every step along the way and generating the run-time components behind the scenes, the time to implement and maintain an integration is reduced by up to 90%. I know, "90% - you are either crazy or lying!" Well, hear me out a bit. We really do have an off-the shelf product that is the product of 100+ man years of development and is built on a new paradigm. I'll discuss the fundamentals of it in the coming entries.