top of page
red_stars_BACKGROUND 2.png

How to stop data firefighting… and start data engineering

Updated: Apr 23


Illustrative banner with the headline 'How to stop data firefighting and start data engineering.' It features two animated firefighters using a hose to extinguish a large flame that is wrapped around a computer monitor displaying a fire icon, symbolising emergency issues in data management. The background is a deep blue with a grid and dot matrix pattern.

Data engineering has evolved rapidly, particularly since AI has become such a hot topic in the past few years. The big focus in AI and Machine Learning requires businesses to do a much better job of investing in their systems and tooling to ingest and process data and ensure that it’s accessible and of a high quality.


The evolving role of the data engineer


In my view, the role of the data engineer is really evolving to become a data platform engineer. Meaning, data engineering isn’t just about writing python scripts and pushing data from one system to another, but a much more strategic role.


Besides creating the infrastructure, tools and frameworks to support data systems, a data platform engineering team should be building a robust DataOps practice. That might involve implementing CI/CD data pipelines, setting up alerts to be proactively notified on quality and cost issues, and making things more automated and easier for data consumers to self-serve. Ultimately, it’s about moving from a reactive role to one that enables others in the business to perform better.


From reactive to proactive


It’s not easy to shift from a firefighting role to a more proactive one. Data is an incredibly complex field. With the pace of advancements in the field, the number of tools, technologies and skills that data engineers are expected to know is mind-boggling, and it’s only going to grow exponentially. Many data teams are often under-resourced and don’t have enough time to get everything done. And when there’s a bug that breaks the system - the data consumers in your business WILL let you know. And they’d want it fixed ASAP.


For whatever reason – because ETL pipelines are complex, because your data systems have a lot of interdependencies, because there’s still tons of technical debt still to be refactored  it will never be the case that you’d eliminate firefighting completely. 


But there are some core principles to apply that can help uplift data engineering teams to engage in more strategic, value-adding activities. 


In this blog series, I’ll cover the following topics:


 I’ll offer practical advice and tips and tricks and share how adopting these simple but effective practices can enhance a data team’s capabilities and overall value to the business. As data engineers, a strong DataOps and DevOps culture helps everything ‘gel together’ and positions us to do more innovative work that helps drive the business forward.


Data teams, it's time to start creating more stable data platforms and end the vicious cycle of constant firefighting. Let's get started!




Comments


bottom of page