Enterprise Features

Top Sources Of Worldwide Data Growth

by Paul Rudo on 17/04/12 at 7:50 pm

It’s no surprise that data is growing exponentially. After all, Moore’s law has been steadily increasing the speed and power with which organizations and users can produce, copy, share and store information.

And within the past few years, many experts estimate that the amount of data we produce annually has exceeded the worldwide hardware capacity to store it. And to make things worse, the gap between storage and data production seems to also be growing exponentially.

Most of this exponential data growth can be traced back to a few key sources.

Financial Sector

Finance is a data-driven industry, where accuracy and resiliency are absolutely critical to survival. And the financial market itself is made up of extremely large, resource-intensive firms which consume and process massive volumes of data.

It’s not uncommon for a bank or a stock market to process billions of transactions per month. And for regulatory reasons, these firms need to store historical customer data which goes back for several years.

Aside from this transactional database data, you also need to consider the massive amounts of regular file data which is generated by thousands of employees as part of daily operations.

Within this category, you could also classify utilities and governments, which are very similar in the way they work with data and the massive transaction volumes which they must process.

Media and Design

Although the next category of firms produces less data on a per-company basis, they make up a large contributor as a group.

Video production companies, broadcasters, and other media companies produce high volumes of video data which must be stored for use and archived for retention.

Within this category, you’ll also find graphic design and photography firms, which generate large volumes of giant image files with great speed and ease.

Finally, you have other types of design firms, such as architecture or engineering, which generate highly detailed project files.

Manufacturing

Although most of us wouldn’t really think about this, the manufacturing industry is highly automated and this automation generates massive amounts of data which must be stored. Lean, JIT and Six Sigma manufacturing processes are highly data-driven. And the statistical data collected through process and discrete manufacturing is constantly analyzed to find opportunities for improving quality and optimizing margins.

Within this category, you could also add the massive data-driven distribution infrastructure which serves as the backbone of the retail industry.

Research And Scientific

It’s often been said that companies are born in garages, but industries are born in laboratories. Today’s most important scientific research projects rely heavily on deep analysis of massive data pools. For example, the Large Hadron Collider generates 700 Megabytes of data per second, and the bioinformatics field is struggling to keep up with the petabytes of data being produces by their research.

Social Media And Internet

Major web properties like Google, YouTube, Facebook, Yahoo and Amazon.com are amongst the world’s largest consumers of storage hardware.

• Currently, Facebook has over 800,000 users, all generating data and uploading media which must be stored on Facebook’s servers.
• Google currently caches every page on the internet, which is estimated at 55 Billion publicly accessible pages.
• Every day, 8 years worth of video are uploaded to YouTube.

And as users interact with this data, they generate automated data trails and log files which are stored for analysis so that the companies can extract business value and provide better service.

Machine-To-Machine and Automated Tracking

In the future, we’ll start to see the biggest area of data producting coming from completely automated sources. Our cell phones are already generating geographic information about our daily activities through their GPS functionality.

And new internet-enabled consumer products are being released every day. Soon, everything from hydro meters, to refrigerators, to MP3 players, to automobiles will be able to communicate with your computer or a remote server.

Also, society is filled with sensors and recording devices which collect data about us every day. Turnstiles, cash registers, security cameras, and other common everyday hardware will be collecting statistical data and warehousing it in a database.

Medical Industry

Electronic medical imaging is growing due to its ability to help doctors save lives, its ability to reduce operational costs for healthcare facilities, and because of regulatory requirements such as HIPAA.

And when it comes to medical imaging, high-resolution video is the format of choice. This means that the medical industry will continue to grow as a leading source of big data production.

Are there any other sectors which you feel should be on this list of major contributors to the big data trend? Leave your comments below and add to the discussion!