Novatia is an ICT consultancy that has specialised in the UK education space since being founded in 2003. Its Questa product is a data aggregation tool that schools use to report on data such as attendance registers, pupil behaviour and exam performance. Questa’s reporting features help multiacademy trusts and individual schools to understand trends in absence, lateness and disciplinary incidents. They also cater for deeper analysis, such as linking whether lateness could be correlated to the uptake of school meals. Novatia’s product has been well received by the schools using it.
What was the problem?
While Novatia produced an excellent reporting solution for the schools it served, the backend setup of Questa made it difficult to cope with the interest generated by the positive feedback from early users.
The client’s requirement was to scale its operation 100-fold, from serving only 100 schools up to supplying 10,000 schools.
This simply wasn’t possible with the existing backend architecture, and so we were engaged to help re-engineer Questa’s data platform to meet future demand.
What did we do?
To address the principal problem of the Questa backend being overloaded, we knew we would need to redesign the database and the data loading system.
Our work on this project focused on 3 main areas:
- Designing: we designed a new data platform running in Microsoft Azure, built around Data Factory and Azure SQL, as well as new security measures to keep all data safe.
- Mentoring: we worked with the Novatia development team to demonstrate new technologies and instruct them in best practice, to improve future development.
- Troubleshooting: we assessed and fixed technical issues as they arose, helping the Novatia team to simplify ongoing support and maintenance.
What was the result?
We delivered a cloud-based platform capable of handling the 100× scale of data requested by the client. This new platform comes with
efficiency savings, meaning a reduction in Novatia’s operating costs per pupil.
Not only can the new system cope with more data, but also that data becomes faster for schools to access.
The wider data platform engineering created 4 more positive outcomes:
- Storing the data in a relational database (as opposed to the JSON files the product originally used) has reduced data storage volumes by over 90%.
- Memory usage has already been reduced by over 50%, with more improvements expected as we tune performance.
- Power BI is able to navigate the relationships in the database, which has made report-building easier and accelerated report development.
- Using an ETL process (extract, transform & load) has ensured the data is cleaned and filtered before being loaded into the dashboard, improving data quality.