With all the latest hadoopla, there are a lot of people wondering what Big Data means to them. There's a sea of data being generated constantly from Facebook, LinkedIn, and Twitter, and the value of mining and analyzing that body of information is easy to imagine. You can find out all kinds of things that are relevant to your business decisions as well as information that can be turned into stellar marketing initiatives.
Social media easily trumps structured data and documents from the hype perspective. Since its arrival has been relatively recent, we don't really have the same internalized model to extrapolate from in order to conceptualize its meaning and treatment as Big Data. Data from RFIDs and medical instruments is also growing at an exponentially increasing rate, and also offers tremendous basis for completely new innovative solutions.
Forrester's Brian Hopkins, in his informative and interesting "Big Opportunities in Big Data" discusses the areas and issues of Big Data that are at various stages of commercial readiness. It seems that we still don’t have all the bases covered.
One of the interesting aspects of Big Data, particularly the Big Data that is being captured via social media or instruments like RFIDs is that as soon as it's captured, it becomes history. The reason I'm focused on this type of data is that a huge body of historic information may not be particularly useful to many businesses. By its very nature, the value to businesses is mostly immediate. In the bigger picture of science and statistics, of course, or for fortune 500 companies, it can be, but it is possible to capitalize on the rapidly changing trends of your customer base and the mob mentality displayed before the opportunity eludes you. With the speed of change we are experiencing today, by the time you can get practical results from any Big Data project, you will have missed opportunities to react and reap today's value.
Yesterday I was discussing federation of Big Data with Dana Gardner, principal analyst at Interarbor Solutions, who noted, "Large amounts of data need to be mined, sure, but there are gems in the fresh data from the right applications at the right time that also spell business gold. The needle may be in a hay stack, or it may be inside two or more applications, where the value of the data is only accessible in the context of the integration activity."
Most people interested in Big Data focus on capturing and mining huge bodies of social media data, or in the case of RFIDs, having a complete picture of all of them at one time. There are plenty of uses of this kind of data that are much more practical, in some cases more useful, and certainly do not incur huge projects. For social media, you can leverage the great search capabilities of Facebook, Twitter, and LinkedIn. With RFIDs, just focus on the subset you care about, meld it with a rule, action, response, and you're off to the races long before your Fortune 999 competitors can get started. If you have been dreaming up great ideas about the value that social media's Big Data can bring to your business, let's consider an interim, easy answer to the somewhat premature heavy-duty Big Data approach.
Rather than get the data and then ask the questions from it in the traditional data warehouse tradition, figure out the questions and resultant actions first. Then capture exactly the data you need going forward. Anything in the past is social history. Grab the data as you see it and react immediately, or turn on selective capture for data for a month or so, and analyze the data as it arrives, or trends as they happen, take action, and "dispose of" the data. Keep in mind that the Big Guys in IT thrive on Big, whether it be databases, hardware, or global corporate projects. Do you need those? Can you wait years to get results from Big Data? Think about getting a head start on competitors with a 5 figure investment and your own creativity.
