THE concept of “big data” is one that, whilst not new, is starting to appear more frequently in business conversations.

Big data is a loose term that covers large data sets usually too big for the average tools to manage, certainly far bigger than any Excel spreadsheet can handle. The challenges of bid data include capturing, storing and analysing the data in an efficient manner to deliver efficient insights.

Big data sets would include the search database of Google, the tweets from Twitter or the Search of Extra Terrestrial Ingelligence project, SETI. Companies like Amazon’s AWS and Google’s Big Query are making the ability to collect, store and analyse data more available to businesses.

Data is all around us. From census data to mapping train delays, this data is being collated and used by individuals, companies and governments to make better decisions. Much of this data, and those decisions will never be seen by you, but may affect you.

The train timetables are created from data on traveller volume, time and locations. In France, data from each train is being mapped to identify the key delay points and times every week, which in turn will help improve services by identifying and rectifying the services.

Some of this data is now being made accessible to us all. If you stand at a Cambridge bus stop, you can text a special number to see when the next bus will reach you. The text message will return data from the bus itself, which will identify its location and time to your bus stop. A simple text will tell you what you need to know; when the bus is coming. This simple text is part of a concept called, ‘The Internet of Things’ - look at our feature box for a wider explanation.

With the advent of mobile communications and social networks, the amount of data available is greater than ever. This, combined with the availability of easy data interrogation tools and data displays means data, and the learning we can achieve can be affordable. Apps like Foursquare will show you how many people have checked into a location, who they are and where else they have checked in. The real time data from Foursquare will also show you when a place is busy. Obvious examples include Liverpool Street Station at rush hour, but when a bar is busy with checkins, it will encourage others to visit the place.

Social Media is also creating vast sets of data that is being used. Google, a huge user of data to create the search indexes, is now using ‘social signals’ including +1’s to redefine the search results you see. Brands are also using tools to monitor mentions of their products across social networks, helping them improve their products and their marketing.

The wealth and importance of data is spawning a dedicated industry and plenty of websites to give you access to data. The Guardian Data store is a perfect example of how data is being made accesible. In this case, The Guardian shares the data and allows others to help interogate and visuluase the data to achieve insight.

Which brings me to a crucial point about data. Any business can access millions of rows of data and do nothing with it. Finding value in data requires interrogation to deliver insight. The companies that can create or harness that insight will have a strong future.

The wealth of data now available, and the value that can be achieved from it, has lead one commentator to ask, “is data the new oil?”

: : Jamie Riddell is chief executive of Suffolk-based Digital Tomorrow Today, a future-looking digital marketing consultancy. He publishes a Directors’ Briefing to monthly premium subscribers covering the latest digital trends and how businesses can harness them today.