Tarush Aggarwal is one of the world’s leading experts in helping organisations leverage data for exponential growth. He holds a degree in Computer Engineering from Carnegie Mellon University, and he became the first Data Engineer on the analytics team at Salesforce. Tarush is Founder of the 5X company, which supports entrepreneurs in scaling their businesses.
In this post, Tarush reflects on how the evolving data landscape has transformed the structure of data teams, making them more compact and efficient. This change has also prompted the emergence of a new kind of data professional: enter the data generalist.
In the evolving data landscape, driven by ai and advanced data vendors, data teams are expected to become more compact, prioritising efficiency over team size.In a landmark blog post from 2017, Maxime Beauchemin talked about the “Rise of the Data Engineer.” This marked the moment when data engineering became a recognised job title, with Facebook leading the way in 2012. But let’s take a step back and see how we got here.
A decade ago, we saw the rise of cloud-based super apps like Facebook, Yelp, and Foursquare. These apps needed to handle vast amounts of data in the cloud, and we didn’t have the right tools and infrastructure to make the most of this data. Things like data pipelines (ways to move data), data storage, modelling (making data usable), business intelligence (BI), and other tools were either not around or just getting started. This deficiency in data management tools led to the need for specialisation in the field.
A new breed of professionals, known as “data engineers,” emerged in response to this growing challenge. They were responsible for creating tools and infrastructure to handle and optimise data in the cloud, filling the crucial gap.
Over the past decade, as the data landscape has evolved, we’ve witnessed the emergence and transformation of various roles within the field to meet the changing demands of the data ecosystem.
But here’s another reason we ended up with so many roles: the data world got really complicated. Each part of the data puzzle had its own set of tools. This complexity meant companies needed bigger teams to manage it all, translating to larger spends and management challenges.
This trend was fuelled by buzzwords like ‘data is the new oil’ and cheered on by investors and pundits. As a result, companies often boasted about the size of their data teams.
However, 18 months ago, things took a turn…
It’s time to refocus on the true purpose of these teams: driving business growth. It’s no longer about building large teams or using trendy tools; it’s about generating ROI for the business. The era of unchecked spending on data teams is behind us. Today, our goal is efficiency, which is going to be achieved by solving for both the platform and people:
1. Reducing the overhead of managing multiple vendors: As the data landscape becomes increasingly complex, “Fully Managed” is the newest category on the block. It eliminates the need for managing multiple vendors, handling vendor discovery, POCs, negotiation, procurement, integration, and maintenance.
There are over 500 vendors in 30 different data categories. The analogy we use is all these vendors are selling car parts. Imagine walking into a Honda and instead of selling you a Civic, they sold you an engine and you had to build your own car. Businesses spend a lot of time figuring out which vendors to use and how to integrate them. This results in large, cumbersome data teams to manage these vendors, leading to rising costs and complexity.
The fully managed category will give you all of the advantages of an end-to-end platform with the flexibility of best-of-breed vendors. Depending on your industry, use case, size and budget you will be able to build a tailored platform for your business.
By 2025, I anticipate that 30% of data teams will migrate to a fully managed solution.
2. Creating a lean team that moves fast: As we start to consolidate tooling, we have an equally or arguably bigger opportunity to reduce team size; “Do more with less” is becoming a theme. People costs represent a substantial portion of a company’s expenses, making this shift significant. Moreover, tooling has become more mature, and many current data roles would not justify dedicated titles moving forward. As a result, we’re witnessing the rise of a new archetype – the “data generalist” – who can operate around different areas of the data stack. Here are a few examples of consolidation in titles in light of this shift:
Sure, we will still need specialised talent, especially at larger companies, and will still have enough workflows for each role, but in general, smaller to medium-sized companies would need their data team members to have generalist skills as well. Over time, there is a possibility that the next generation will focus on an all-rounded approach to data instead of specialising in data science, data engineering, or other skill sets. Existing specialists will also need to retrain to gather more rounded expertise.
Specialists who don’t retrain run the risk of getting left behind. For example, a number of years ago, Microsoft Stack specialists worried their skills had limited use outside of Microsoft as the industry. Just like everything else, highly skilled specialists will continue to excel, but an increasing percentage may struggle to adapt to the job market, which starts to favour generalists in an environment with a shrinking number of highly specialised roles.
The data landscape has evolved significantly. We’ve moved from having big teams to focusing on efficiency. This shift has given rise to data generalists who can handle various tasks, making traditional roles less necessary. Larger teams may still specialise, but the skills gap is widening, prompting specialists to explore other paths.
Adaptability is key. Expect lean, agile, and highly effective data teams building on top of fully managed data platforms.
Over time, there is a possibility that the next generation will focus on an all-rounded approach to data instead of specialising in data science, data engineering, or other skill sets.
