How Big Data Spawned The Birth Of Next-Generation Non-Relational Databases

10 years ago, Google had a problem. They were constantly spidering, indexing and caching every page on an Internet which was growing at an exponential rate. In addition to this, they were expected to perform sophisticated calculations, sorts and searches on this massive and ever-growing body of data.

Traditional relational data bases are designed to perform complex transactions using structured data. But there are scaling limits. As a data store moves into the petabyte range, typical operations begin to take much longer and eat up lots of resources. Also, relational databases are impractical for making sense of messy, unstructured data from multiple data sources.

A new approach was needed.

Google had attempted to deal with this problem by abandoning relational databases altogether, and developing an entirely new methodology which they called “BigTable”. The idea was to structure all of their data in a very simple way, and then break up this information across many nodes.

With the old-school methodology, all of the database processing was centralized. But with the new Non-Relational or “MapReduce” methodologies, processing is distributed against many “nodes” which each have their own storage and processing capabilities.

This approach to managing large data stores is ideal for applications that don’t require absolutely consistent information. For example, small inaccuracies in a mapping program or a weather report might be acceptable. For applications which are well-suited to this new breed of data management, these minor inconsistencies are more than off-set by the boost in performance.

On the other hand, minor inaccuracies in a financial reporting system would not be acceptable at all. So these types of applications will continue to be the domain of relational databases.

Another benefit of these non-relational databases is that they allow large problems to be broken up into smaller pieces which can be attacked simultaneously by thousands of nodes. This means that complex statistics and data analysis can be performed very quickly on extremely large volumes of data. This has spawned the birth of new industries in the areas of machine learning, artificial intelligence, face recognition and other types of data manipulation which would have been impossible or extremely slow using traditional methodologies.

Non-relational databases are the hottest new thing in tech. This is the same technology that powers sites like Yahoo, Facebook, Twitter and many of the hottest new start-ups. We’re also seeing a spike in demand for programmers with experience in NoSQL, Hadoop, Cassandra and many of the technologies which power this space.

Non-Relational Databases are more than just a new tool. It’s going to be a fundamental shift in the way we work with information.

Software AG

Mar 27th, 2012

As you mention, the needs of a business would dictate if they could handle minor inaccuracies. It all comes down to what business goals you are working to achieve and where you can become more efficient (and what you might have to sacrifice to do so).