The Human Face of Big Data

Big Data recently has been a pretty hot technology area. The Human Face of Big Data project is a globally crowdsourced media project focusing on humanity's new ability to collect, analyze, triangulate and visualize vast amounts of data in real time. Rick Smolan of the "Day in the Life" book series fame is the person behind this project. Here is a small video:

The data collection will happen through the downloadable mobile app for both iOS and Android (available September 25th), so anyone with a smartphone will be able to participate. The data will be collected from the users and their trackable activities in real time between September 25th and October 2nd and will be analyzed by data scientists to draw some interesting insights in an invitation only event in New York.

In November the data collection and analysis activity will involve connecting students around the world in grades 6-12 and their teachers sharing opinions and learnings. Also a large format book "The Human Face of Big Data" will be released on November 20th, its a collection of pictures illustrating the topic.

Big Data - Tools

To manage and use the large amount of data there has been a great amount of innovation.

There are various storage devices that are out there targeted towards this market like EMC Isilon, IBM SoNAS and HP X9000. Netapp recently announced E-Series range of storage devices.

Storage software includes relational databases like Teradata which is row-based. There are column oriented databases that stores content by column rather than row. They are advantageous for fetching data for large number of rows but only a small subset of columns. Examples include Oracle Exadata, EMC Greenplum and HP Vertica. One of the new trends for big data is move away from traditional RDBMS software.There have been a new class of "NoSQL" (Not Only SQL) alternatives that does not require fixed schemas and are highly distributed and horizontally scalable. Examples of a few such databases are Cassandra, CouchDB and MongoDB.

To analyze and process the big datasets, the processing is not carried on a single powerful computer but a network of commodity servers clustered together. The most popular framework in this realm is Hadoop. Inspired from Google's MapReduce and Google File System (GFS), it was originally developed by Yahoo but now is open source managed by Apache. Hadoop utilizes a highly scalable and distributed file system HDFS to store data and MapReduce to rapidly process data in parallel on cluster of nodes. In addition to HDFS and MapReduce there are other useful sub-projects associated with Hadoop like HBase, Hive and Pig. Hive is a data warehouse infrastructure that provides data summarization and ad hoc querying similar to SQL. Pig is a high-level data-flow language and execution framework for parallel computation.

Hadoop has been a framework of choice for many big data implementations in various companies including Yahoo, Facebook, EBay and Twitter. There is a healthy ecosystem of commercial vendors that provide tools and services based on Hadoop platform like Cloudera, Datameer and Karmasphere.

As we can see the list of tools for big data is itself big and growing.

Big Data - Introduction

The term "Big Data" is used for referring to datasets that are typically in the order of petabytes, exabytes and even larger. The data could be either structured or unstructured. In simple terms any data that has a definite structure and could be stored easily in relational databases could be considered structured. Unstructured data does not have an identifiable structure, examples are email, images, word documents, phone conversations, etc.

With the advent of cheap digital storage and advancements in computational power we are seeing an explosion in the growth of digital data. It is estimated that 90% of the data in the world today has been created in the past two years. According to IDC, in 2011 more than 1.2 zettabytes of information will be created and stored and by 2020 will grow to 44 times that of 2009.

The big question is what does it mean for us?

Should this scare us? Governments and companies acting like big brothers with their ability to collect and retain personal information about us and our behaviors through records of our searches, chats, health records and any other form of interactions that can be digitally tracked.

Shelving the privacy concerns for now let us see how it can benefit the businesses and consumers. Businesses can analyze this data to gain efficiencies within their organization and strategic advantage over their competitors. Consumers will greatly benefit from the innovations that will affect some core sectors like health care where effective use of data will help doctors make more informed decisions in treating patients. Real time traffic and weather data from Mobile phones and GPS devices will help avoid traffic congestions.

What opportunities does it bring for engineers and scientists? Big Data is still mostly a buzz-word and adopted by larger companies - internet giants like Google, Yahoo, Facebook; retailers like Walmart; financial services companies. So there are lot of businesses that in coming years will be looking for talent to manage and use big data and associated services.

In the upcoming blog I will talk about some of the popular tools associated with Big Data.