In April 2024, Cloud Shuttle conducted a survey of data professionals in Australia and New Zealand. We were interested in exploring the common challenges that data professionals face in data platform engineering and management.
The job category breakdown of the respondents were as follows: Data managers (36.4%), Data engineers (31.8%), Data architects (9.1%), Data scientists/analysts (9.1%) and other (e.g. consultant).
The majority of respondents were located in Australia (84%) with a minority in New Zealand (16%). A wide range of company sizes were represented in the survey, from small startups to large enterprises, highlighting the widespread nature of many of these challenges.
Notable trends
Survey respondents listed their biggest challenges (in order) as: data governance and compliance, data quality, data observability and tooling and data integration.
Data observability and tooling
The lack of centralised platforms for monitoring data events was raised as a common problem. Respondents wrestled with data silos and quality detection issues, resulting in reduced trust in the data outputs and an ultimate slowing in productivity due to the time lost navigating and managing complex, redundant data workflows.
Data governance and compliance
Data governance frameworks remained a stumbling block, with many respondents mentioning that their governance was not up to scratch. There were often inconsistent standards and unclear data ownership. Poor documentation and communication made the loss of institutional knowledge a much higher risk for businesses.
Data Integration
Many teams struggle with the ingestion and acquisition of data across disparate systems, hindered by the sheer number of tools available in the market, integration challenges, legacy systems and growing technical debt.
Data quality
These challenges have a common result: less-than-ideal data quality issues and data that is less useful as a strategic business asset. If the data quality is poor, it’s more difficult to drive actionable insights and establish a solid foundation to inform business decisions.
Conclusion
While the sample size was small (<100), it highlights shared obstacles faced by data professionals in the region. These challenges aren’t isolated ones and could be a good indicator of broader trends affecting data management across different industries and organisational sizes. Identifying and understanding these commonalities is the first step towards developing more effective strategies and solutions to uplift our data practices.
At Cloud Shuttle, we are familiar with the common data challenges organisations of various sizes faced. And we’re committed to partnering with our clients to overcome these issues. If you're experiencing any of these issues or looking for ways to leverage your data more effectively, get in touch today!