12:35 | 13:15
Keywords defining the session:
Takeaway points of the session:
- What DataOps is and how to apply it in a iterative manner.
- How to change towards a culture based on automation that supports multidisciplinary teams
At urbanDataAnalytics we deal with data arranged in three dimensions: geospatial, temporal and taxonomic. We have weekly data updates. We have customers that demand complex answers within hours. All of this requires a streamlined and scalable culture if you want to stay sane.
After struggling for months, we’ve found that DataOps is the perfect fit for us. It borrows the automation spirit of DevOps applied to data, but it’s much more than that.In this talk we’ll share our lessons learned during 2 years of constant evolution and improvements in our software, data processes and culture.
Data arranged in these three axes (geospatial, temporal and taxonomic), together with fresh data snapshots every week, different roles working in the same team, and demanding customers, make the challenge specially fun.
From the very beginning, we started mangling with plain CSVs, but soon we realized that proper tools, efficient ETLsandwell chosen formats is not enough. We needed to create a culture about how to operate the data and automate the processes.
Months ago, we discovered the term “DataOps”, and we started the development of a new data processing platform based on its principles and spirit.
In this talk we’ll share about how DataOps changed our approach of creating data analytics products, and how our platform has become the cornerstone to support and reinforce the self-service data culture we needed.