— The interesting topic isn’t big data but rather, just data —
The term ‘big data’ is thrown around an awful lot lately to describe a set of problems and solutions. Often though, the term is used in connection with an interesting set of trending issues that actually have little to do with the size of a particular data set.
These are:
- The increasing availability of data
- The emergence of software tools to process this data in contextually useful ways
- The increasing availability of tools to share this data and the resulting democratization of the data
It seems that the interesting topic isn’t big data but rather, just data. I’m making this suggestion not to be nitpicky or even controversial (though I’ve been know at times to be both). To be sure, there are specific cases in which the sheer size of the data poses interesting challenges in processing or storage but I’m concerned that the focus on the size of the data is a distraction.
Generally, the trends are around data becoming available to organizations with the promise that it can help them be more productive and more efficient in their operations. Datasets might be huge but they might not. The interesting challenge is in handling all the new data that’s becoming available, in a manner that benefits the organization.
At Scope 5 for example, the datasets that we handle aren’t huge – we might be tracking a few dozen data points, sampled monthly, across a few hundred facilities. That’s on the order of 100,000 data points per year, a far cry from the billions of data points typically associated with big data.
The challenges that we’re seeing are not in the technical processing – that’s easy. We see the challenges falling into two general categories – one is an organizational behavior challenge, the other is in imparting to our customers domain specific knowledge (in Scope 5’s case, sustainability knowledge) related to using the data. My impression is that most of the organizations that are part of the new data trends are grappling with a similar set of challenges.
Our leading customers have the notion of data tracking and analysis deeply embedded but many organizations are only just realizing the potential benefits of putting data to use. We’re finding that these organizations benefit from handholding or consulting through the process. Let’s take a look at the two areas in which we’ve started to provide additional functionality and consulting support.
Organizational Behavior Around Data
For many organizations, the flood of data is new. Nobody has been assigned the task of looking at the data and acting on it. They’re asking questions such as: Whose job is it to identify the relevant data sources. Whose job is it to analyze the data? What is the goal in analyzing the data? How are those individuals empowered to act on their findings? We often help customers decide which data sources should be automated and which are better done manually. We help them to assign responsibilities and establish a workflow.
Domain Specific Opportunities
The next set of challenges is domain specific. For example, in the case of a customer that has many facilities, they can use data to identify over-performing facilities and to apply lessons learned from these to under-performing facilities. Here again, we serve in a consulting role, recommending to our customers that they plot energy, carbon and cost across multiple facilities, normalized to square footage or units produced (or whatever normalizing factors make sense for them).
In other cases we may help customers select libraries of greenhouse gas conversion factors to use or we may help them decide which metrics should be tracked at a fine grain and which can be tracked more coarsely. We may work with customers to help them use the data to present their sustainability story more clearly.
Other Opportunities
That’s just the data analysis part. When you begin to look at the data you have and how it can be sorted, a whole world of unrealized transparency benefits emerge. As just one example, our customers have found that making data more available to rank and file employees can help develop and retain talent through a whole new level of engagement and empowerment.
Conclusion
It’s not the size of the data that our customers are grappling with – it’s how to use the data. Helping customers navigate through this conundrum, both with our tool and through our consulting, has shown us just how quickly rewarding our collaborations can be.