How UK Curriculum Digital Centre was built
A record of what this project is, how it came together, and why it works the way it does — kept here on the site itself, not only in build notes, so it's always easy to explain to someone new or pick back up later.
Project timeline
Source collection
A first harvesting pass gathered official curriculum documents from GOV.UK, the Department for Education, the Standards and Testing Agency, Ofqual and the devolved nations' education bodies — the raw material for everything that follows.
Building the database
The harvested documents were imported, de-duplicated, dated, and tagged with subject, key stage and licence information (Open Government Licence vs. exam-board copyright). Every document was then extracted and split into searchable passages. The one deliberate gap: 2020–2021 exam papers don't exist in the collection because those exams were cancelled during the pandemic — there was nothing to collect.
- 1,924 documents finalised and released as the first production dataset
- 9,198 searchable passages (later corrected to 9,800 — see next entry)
- 1,464 learning objectives and 641 real past exam questions imported alongside
Embeddings and hybrid search
A search engine was built on top of the database: every passage was converted into an AI embedding (OpenAI) so the site can find relevant material by meaning, not just keyword matching, combined with traditional keyword search for the best of both. Two data-quality issues were found and fixed during this work — a handful of passages were too large for the search model to process, and a database export step had silently dropped the unique ID from 817 passages, making them invisible to search. Both were diagnosed here, fixed at the source, and verified before going live.
- 9,800 passages, all successfully searchable — zero known gaps
- An internal dashboard added so the owner can rebuild the search index safely, without needing a developer
Plain-English descriptions, the full site, and going live
The biggest stage of work: every document in the catalogue was given a short, plain-English description — written by AI, grounded only in the document's actual text, never invented — so a parent or teacher can understand what a document says without reading the whole thing. 565 descriptions were carried over from an earlier, smaller dataset; the remaining 1,359 were generated in a single batch run by the site owner directly from the admin dashboard.
- 100% of the 1,924 documents now have a plain-English description.
- The whole public site was switched over to run on this new database — the older version was retired from public use but kept safely archived, never deleted, in case anything ever needs to be checked against it.
- A reported broken link led to a wider fix: around 3% of documents had an internal filing reference instead of a real web link. The site now always shows the best available link — the real source if one exists, the site's own copy of the document if it holds one, and never a dead link.
- The site was reorganised from a single long page into separate pages — Home, Catalogue, and Ask a Question — each with its own address, plus a dedicated page for every individual document showing its description, related past exam questions, related learning objectives, and other documents like it.
- The site's visual design was updated to match the rest of the iFairy family of projects (dark background, light-blue accents), and the home page was rewritten to clearly explain, in plain terms, what this project — UK Curriculum Digital Centre — actually is.