Why 55% of Big Data projects fail – and what IT can do about it

Big Data is quickly becoming a big deal for companies and their IT departments. However, a recent survey says that more than half of Big Data projects never even get off the ground.

Most businesses find that planning difficulties and organizational problems prevent them from finishing projects, according to a recent Big Data survey from Infochimps

Most businesses are finding that Big Data will be essential for staying competitive in the long term. When asked to list their top IT priorities for the next few years, only 6% didn’t include Big Data in their top ten.

Unfortunately, planning issues and organizational problems often stand in the way of companies taking advantage of Big Data initiatives. In fact, more than half (55%) of Big Data projects never even get completed, according to the survey.

These were the three main obstacles that stood in the way:

Lack of cooperation among departments

According to many survey respondents, the IT team — the ones who will actually implement the Big Data technology — often aren’t consulted until the project has already begun.

Among the 300 IT pros surveyed, 39% said a lack of cooperation between IT and the business was a major obstacle to impeding a Big Data project.

To fix those problems, IT departments can do their part by staying in close contact with business units to keep up-to-date on their tech needs. And when a project begins, it’s important to appoint a project manager with the right communication skills needed to be an effective bridge between IT and the business.

Poor planning

In addition to poor communication, organizations often underestimate how much time and effort the project will require, and therefore, they don’t allocate enough resources.

Two of the top obstacles blamed for project failures were:

  • Failure to recognize the scope of the project (cited by 58% of respondents), and
  • Technical stumbling blocks (41%).

As those answers show, one of the reasons IT needs to be involved in Big Data projects early on is to offer their expertise regarding what resources will be needed.

Lack of staff experienced in Big Data

Big Data is a recent hot trend, and that means that there are more companies are looking for people with the right skills and experience than there are people with that experience. In fact, most companies (80%) say they’re having a hard time finding the right expertise for their projects. In addition, 73% say understanding the technology required is difficult.

What can IT departments do to find the right talent? One option for attracting new hires is to work closely with educational institutions offering Big Data programs. However, some experts say companies are better off giving current staffers new training to help the gain Big Data skills.