My story of becoming an MLOps tech lead
Before I go into my personal story, I will go through some historical details that made my career path possible.
The emergence of the machine learning engineer
MLOps popularity has grown significantly over the last few years, this can be seen using Google Trends and exploring the number of searches for "MLOps". Interest in MLOps is not surprising: it came from an earlier boom in the data science field that started around 2015.
Many large corporate organisations started investing in data science around that time. Unfortunately, many of the early data science initiatives ended up failing and costing companies a fortune. It turned out, bringing machine learning models to production is much harder than just building a model: data scientists did not have the knowledge of integrating their models with the rest of the company infrastructure, and teams that did have the knowledge, did not know much about machine learning. A lot of organisational changes were required to make it work, and a new profession has emerged: machine learning engineer.
I think, there are two main groups of people that ended up becoming machine learning engineers: a data scientist who learned about software development or a software developer who learned about machine learning. I belong to the first group, and the whole story about the emergence of MLOps feels very personal to me.
From data analyst to an MLOps tech lead
Back in 2013, I had a Bachelor's and Master's degree in Economics, which was statistics & econometrics heavy, I knew R quite well (probably the most popular language for data analytics back then) and wanted to do something with data. Data science positions did not even exist back then, this is how I ended up being a data analyst in a large telecom company in the Netherlands. I was busy with running ETL pipelines, building churn/acquisition/cross-sell models, making business reports, and trying to automate it all via CRON scheduling on some machines in the office. I can say, I always had a passion for automation and standardization.
In 2016, my title officially changed to "data scientist", I moved to another department with a bunch of enthusiastic people that wanted to change the way data science was done within the organization. We introduced version control, CI/CD pipelines, and orchestration to all data science departments and even built the first-ever MLOps platform before it even became a thing. We even reiterated it multiple times with different tool stacks. Since then I believe MLOps is about certain principles and ways of working and not about the tools (see our article on Minimum set for MLOps). As a team, built multiple impactful models and had to figure out ourselves how to deploy an API, how to make it secure, and figure out networking setup. It was a lot of fun! I realized I became a machine learning engineer and that I enjoyed it more than being a data scientist.
After a lot of changes in my personal life (having 2 kids, going through a depression) I was up for a new challenge. I moved to a larger and more complex organization to lead the MLOps initiative there. I quickly found that it was much harder to build an MLOps platform when you need to align with many different teams that do not always want to work with you and have their own views. Nevertheless, we were able to successfully accomplish MLOps transformation. The main learning I had is that you need to start small, build it in a reusable way, show the value you create (extra revenue from faster time to market, operational cost reduction), and talk about it as much as possible.
My future in MLOps
Well, I can’t believe it has been almost 10 years already, and yet in some ways, it feels like the journey has only just begun. The field is still young and moving quickly. Thanks to AI and LLM a new revolution is on the horizon: LLMOps. I believe our work as ML/MLOps engineers will be transformed with the new LLM revolution:
We will spend less time writing the code, but spend more time on architectural setup and integration
Many ML engineers will be doing more data engineering in the future because not more but better data will be leading us to the next tech revolution.
Companies that actually benefit from using LLMs for customer-facing applications will need to figure out the way of deploying them, and it will require more communication and better integration with business teams.
The most complex part of any technological change is not technology itself but people: the way they communicate and work together, and soft skills will become more important than ever before.
Looking forward to the bright future!