One of the great things about working in IT is that the industry is constantly in a state of change.
In recent years, we’ve seen the software market evolve through a series of incarnations. First, software was a product. Then, software was a service. And now, we’re entering a whole new era which promises some interesting possibilities.
Today, we’re seeing the emergence of companies – like Facebook, LinkedIn, Google, Amazon, etc – who have built thriving businesses through the collection of data. This new trend – often called “big data” – is powered by the emergence of extremely cheap data storage and new non-relational database technologies which are optimized for extremely fast analysis across very large data volumes.
Tomorrow’s big software vendors will offer data as a service. They will store, collect, and combine large amounts of data, and analyze this data in order to extract value for the customer.
This has important implications for people working in tech.
Yesterday’s software was developed by coders with training in PHP, C++ or Ruby on rails or other programming languages. Although it takes years of experience to become a “great” programmer, most hobbyists can become proficient in a programming language in a matter of months without any formal schooling.
In other words, the barriers to entry for becoming software developer were fairly low.
But as the software industry moves more towards a “data-as-a-service” approach, developers will demand a whole new skill set from their programmers.
Tomorrow’s “big data” databases will consist of massive volumes of unstructured data. Information on this scale can’t be sorted or classified by hand. Working with this kind of information will involve advanced computer science concepts such as machine learning and artificial intelligence.
Analyzing and using this data will also require a deep understanding of mathematical concepts like statistics, calculus and analysis.
The skills of someone with a science background will be extremely helpful since the software developers will create value by extracting useable information from their data.
Currently, relational databases are being used as a one-size-fits-all tool for every analysis, search and reporting application online. But in the future, traditional relational databases will become the exclusive domain of transactions and financial reporting. Instead, technologies like NoSQL, MapReduce and Hadoop will handle most of the heavy lifting for applications which require high-performance processing and analysis of large data volumes.
Data isn’t just being collected from users. It’s also being collected automatically through the explosive growth of web-enabled hardware and mobile devices. And when combined and analyzed, this provides interesting new business opportunities.
- A downtown restaurant can use mobile device GPS data from social networks to discover where their best customers come from, and target local advertising to those areas.
- Shopping sites could create highly personal mailing lists which are crafted to provide maximum value to the recipient. This not only improves sales, but it also reduces spam complaints and unsubscribe rates.
- Stock traders could analyze market activity to take advantage of new opportunities within milliseconds of being detected. (This kind of high-speed trading is already happening)
- Video gaming companies could use statistical data to match online players with others of similar skill level. This helps keep the game challenging, and make it more enjoyable for everyone.
- Doctors could use massive virus and illness databases to create extremely personalized doses which minimize side-effects and maximize results. This kind of technology is already being used to predict flu epidemics, and has saved countless lives.
But dealing with large amounts of unstructured data can be very tricky. For example, when someone is searching for a blazer, how do you know if they want a car or a piece of clothing?
On a smaller scale, it’s a simple matter to manually create such exceptions. But when dealing on a massive scale, algorithms must be able to spot these problems and correct them automatically. It’s simply not practical or economical for a human to do this by hand.
In the future, the actual mechanics of computer programming will become much simpler. Instead, programmers will require a high-level understanding of the mathematical concepts which govern the massive, unstructured data storage which these programs are meant to leverage. These are skills that go far beyond simple coding.
How do you think IT careers will be affected by the new wave of data-driven online services? Leave a comment below.