In this episode Andrew and Danny talk about modern data warehouses – what they are, how they differ from old-fashioned data warehouses, and why you might want one.
What is a modern data warehouse?
- A modern data warehouse is a database and analytics technology that helps you process big data using all the scale of the cloud.
- We are talking about products such as Azure Synapse Analytics and Snowflake.
How do they differ from older generations of data warehouses?
- Older generations of data warehouse were built on relational database technology, and therefore constrained by the performance of individual machines.
- A next generation introduced MPP – Massively Parallel Processing – which gave scale to data warehouses.
- Modern data warehouses separate compute from storage. Compute is the expensive part, storage is cheap (relatively). You can scale compute up and down as demand requires, whilst benefitting from massive-scale cloud storage.
- Modern cloud data warehouses are as much a system for managing cloud resources as they are a single product.
Why you might want a modern data warehouse?
- Increasingly, we see companies adopting SaaS products for major line of business systems.
- This is great for many businesses – lower operating and support costs, functionality develops over time, can get industry-specific solutions that fit with their businesses.
- Downside is that this creates data silos – isolated pockets of data within the SaaS apps.
- You need to own your own data, don’t rent it. Build a data warehouse and ingest the data from your SaaS apps so that you have a single view of the data your organisation relies on.
- This single view of data allows you to combine and crunch datasets together, which is simply not possible when your data lives in separate SaaS apps.
Watch the video here:
Listen to the audio here:
In this episode of the podcast we are again at the Data Science South Coast meetup. This time I’m presenting an Introduction to Data Lakes.
In this post I’m describing the work we did to build a Big Data platform for one of our Financial Services clients – a major
In this episode of the podcast we introduce data strategy and transforming your business with the information you have and the questions you need to