Certainly there are valid uses for collecting data ad-infinitum but the most common reason I hear is the notion 'why not?' Data transfer and storage capacities have become exceedingly affordable to the point that the cost of a few additional terabytes is worth the gamble that the data stored will eventually be useful. In fact one of the uses often mentioned for analyzing big data is to discover patterns and answers we haven't yet developed questions for.

I think we should be thinking about big data in 2 distinct categories. Category 1 can be seen as a collection of data that is part of a process, has known values (even if they don't consist of structured data in tables and fields) and can provide business value within a known period of time. Category 2 consists of data that is a byproduct of the environment. This can be working documents, social media streams, video clips, and anything else that is not directly related to the process of getting specific tasks completed.

Category 1 should be no mystery to anyone involved in supply chain activities. What's different now from just a few years ago is that this data can be retained far longer than it has previously been kept. The benefit is that in addition to its defined and current purpose, the datastor can be mined for patterns and facts over a longer period of time. And predictive analysis over longer periods using more data can be more accurate, thus delivering some measure of ROI against the additional expense of storing and analyzing the data.

Category 2 is the area in which we step off the cliff of human experience and into science fiction. This doesn't mean large data sets can't produce stunning insights. But what we need to be careful of is too heavy a reliance on what the Magic 8 Ball tells us. Overall, we have collectively been finding our way through the murk of business and human interaction relatively successfully for a few millennia. With insights that big data provides we may be able to improve our success rate by a few percent - or maybe a few fractions of a percent. But like all technologies turn out - they are not the single answer to doing everything better. And the fact that it is not humanly possible to cross-check the insights that big data provides means we can be trusting the black box to be a better guide than are our own instincts and experiences.

I suggest using big data and its byproducts to help with things we know, and questioning amazing guidance produced by processes that are beyond comprehension against our own intuition and historical successes (and failures).