Agile Big Data is Coming

Agility with Big Data?

Shiny but not Agile, perhaps destined to never be Agile.

Just look at ETL. Never, never agile.

So Clunky. After all these years, why is it so Clunky? So Clunky it’s fragile.

Big Data follows in the footsteps.

Big feet. Big hype, Big opportunity

for discoveries otherwise unfathomed.

Great programmers performing great feats,

Coding, coding new horizons… Open Source assist.

No hope for Agility.

Where are the trailblazers of Agility?

Ah, yes!

They’ve conquered ETL, EAI.

They’ve conquered Data Virtualization.

Agile Big Data is coming.

Are great programmers actually an impediment to Agility in complex software problems?  I‘m not talking about Agile development, but rather Agile solutions. Great programmers like to be on the leading edge of the latest new shiny trend. I remember the rush of programmers to work on the Y2K-driven Enterprise Application Integration and ERP rage. Now we’re all over Big Data.

It takes time for the market, business, and technologists to evolve the thinking about what the Shiny thing really is for, what it means, and what the technology requirements are. So, who jumps in first? The really good programmers who want to blaze the trails and solve the problems first-hand. Unfortunately, the early players can’t benefit from the evolution and maturing of the Shiny and what the requirements will really be. By the time it’s clear what it takes to genericize the problem, the Great programmers have moved on to the next trend, and are certainly not going to step back and solve the problem again in a tool or platform that hides all the repetitive work behind the scenes. 

Product companies then step in to harvest the rest of the hype curve. Leveraging their Big Name, they make plenty of revenues on hard-coded, specialized solutions where custom coding is accepted as the only way to solve the problem. In my view, a timeless benchmark for Agility is the minimization of custom code. The more coding involved in generating a solution, the farther away that solution is from Agile.

The growing love affair with Open Source code is a huge step backward for the cause of Agility. Programmers love coding. I, for one, love coding, too. But I HATE having to code essentially the same thing over and over, with just a couple of tweaks difference each time. This is exactly what integration is all about: lots of small (very important) tweaks to the same code over and over.

That is why Enterprise Enabler exists. That is why we hide all the technical details behind the scenes of our agile integration software. Any time a programmer has to do essentially the same thing more than a handful of times, it is automated with tweaks configurable in a UI. Programmers should be programming exciting, innovative new things, not laboring over repetitive, boring scripts and code changes, and maintenance.

We’re applying the same philosophy to automate Big Data integration and analysis.