The field of big data is one that has seen explosive growth in recent years. Companies all over the globe have integrated big data and analytics technologies into virtually every aspect of their operations, from managing sales pipelines to streamlining production practices and boosting productivity up and down the supply chain. It’s a business revolution that is reaching into every industry in about every way imaginable. It has also created a demand for data skills that has far outpaced the ability of the labor market to provide them.
The skyrocketing demand for big data skills has led to a severe shortfall of qualified candidates for data science-related positions all over the globe. The talent shortage is so pronounced, that industry observers have fallen all over themselves to shout about it from every available mountaintop. The prevailing consensus is that the big data skills gap is continuing to widen and that employers haven’t seen the worst of it yet.
The problem with current big data labor forecasts is that they’re probably wrong. Most of the current industry observers are ignoring some mitigating factors and early signs that the forces of supply and demand in the labor market are moving back towards alignment. Here’s what they aren’t considering.
A.I. Continues to Advance
For years, the media has been awash in stories designed to frighten the average worker into believing that artificial intelligence is going to replace them at work. They’ve focused on workers on factory floors and back-offices, who conduct the mundane and repetitive tasks that make modern businesses function. Well, the reality is that A.I. isn’t only coming for the rank and file of the average business; it’s reaching far higher up the chain much faster than anyone anticipated. A.I. is already beginning to reach into the work of data scientists, which should put downward pressure on demand in the industry in the coming years.
Education Catching Up
The education industry hasn’t exactly been sitting idly by while data science has risen to become one of the hottest employment fields on earth. On the contrary, educational institutions all over the world have been creating programs designed to produce a whole new generation of big data professionals. It’s also important to note that the industry seems to be targeting their efforts toward where the big data market is going, rather than where it has been. They’ve recognized that the average data science job now requires applicants to hold a master of data science degree rather than a Ph.D., which had been the norm previously. To align with that shift, the number of master’s level data science programs has increased in almost every corner of the world.
Standardization Increasing
In big data, as in any new technology field, the earliest adopters serve as the pioneers that blaze a trail for others to follow. In the beginning, there were no off-the-shelf analytics and visualization tools available, and industry pioneers had to invent whole new systems to suit their specific needs. Today, vendors like IBM, Oracle, and SAP have come to market with products that allow companies to standardize their big data operations in ways that weren’t available a few short years ago. The result is that companies now require far less specialized skills in their big data workforce, and further consolidation of major platforms should keep that trend going for many years into the future.
A Narrowing Gap
None of the above factors are going to close the big data skills gap in the near-term, but their very existence should blunt the dire warnings about labor shortages emanating from most of the industry. The reality is that the big data labor market is going to stabilize itself over time, as it has in every other sector in the industrialized era. Like nature, labor markets abhor a vacuum, and it won’t persist for anywhere near as long as most analysts insist. That’s not to say that challenges won’t remain for the big data industry, but if history has taught us anything, it is that labor inexorably flows to where the money is, and you’d be hard-pressed to find a more valuable 21st-century industry than big data.