Best Practices Big Data

What benefits can you expect, and what best practices can make your BIG DATA project a succes?

The real challenge isn’t the technology. It’s shifting our organisations to address the opportunities that Big Data creates: Become a “Data Driven Enterprise” by efficiently mining internal and external data.

1. Consistency

Identify the strategic objective and information need:

Identify preferred data integration method

Big Data consumes a lot of technical infrastructure, storage, bandwidth, CPU, etc. And it generates highly variable workloads as it does so. You need lots of infrastructure at some times, very little at others.Fortunately, the Cloud is made for this. The challenge isn’t technical so much as it’s one of finding a reliable cloud vendor, and of getting the economic model right. Just don’t underestimate how challenging that can be in the current, rather opaque market for cloud services.

Implement an easy-to-use Big Data application that integrates with preferred desktop applications.

This creates several challenges: you need to get up several learning curves at once, integrate many tools with your existing application stack, and build a stable operating environment out of these disparate pieces.

Only select a tool if there is an identified business need.

Include the tru user in the selection process

Collecting and Organising the Right Data:

Most organisational data is highly fragmented.The web team has a bunch of logs. Sales owns some of the customer data.

This creates challenges at several levels: syntactic (defining common formats), semantic (agreeing definitions) and political (negotiating ownership and responsibilities).

It also creates data quality problems as no-one’s responsible for the complete picture, so no-one ensures that data is correct, consistent and up to date.

Big Data needs to face all these challenges head on. (As data warehousing did before it. But Big Data has the added complications of semi-structured
data and rapidly changing data definitions.)

Turning Data into Information and Insights:

Quality and Valuation
You can only do this effectively if you can ascribe clear value to the outcomes, otherwise you have no way to prioritise activity across your portfolio of experiments and investments.

Yet few organisations are able to put clear valuations on their current data, let alone on the fuzzy web that Big Data exposes.

2. Communication

Communicating Information and Insights:

Voice of the Customer
Identify all involved stakeholders and make sure they are involved in all steps of the big data project

Good Big Data teams will be very tolerant of “failure”. (If 50% of your experiments don’t fail, then you’re probably not testing the boundaries).
Right now, many Big Data projects are merely playing with the data, exploring the tools and shifting data around within its silos. If we could build some stable, cross-functional teams and focus them on business-led experimentation, then we’d probably begin to find real value in the data we have stashed away. And along the way, we’d start to break down some of the silos that have grown around our data.

You need a deep stack of skills to do Big Data. As well as business specialists (to ask the right questions) and technologists (to tame the infrastructure and applications), you need “data scientists”.

These are the people who understand the statistical algorithms, can drive the visualisation tools, etc. They’re not easy to find. And once you’ve found them, you need to integrate them with the rest of your team, build appropriate reward and reporting structures, and so on.

3. Responsibility

Turning information into actionable knowledge:

Big Data projects operate on a different cycle to traditional ones.It’s not so much “plan then do” as “experiment, learn and evolve”. It requires a mindset that’s attuned to research as much as delivery, yet which is able to temper research with business objectives.

Create culture where Big Data management becomes a priority in the daily processes.