This is the story of a real project: a secondary school came to Lycore wanting to replace their 40-page student handbook with a RAG chatbot student handbook. What they asked for was a chatbot. What they actually needed – and what we built – was somewhat different. This case study covers what the school initially wanted, what we discovered when we dug into the real problem, the technical architecture we chose, the compromises we made, and what we would do differently if we built it again.
What the School Actually Wanted
The initial brief was clear in its ambition and underspecified in its constraints. The school’s head of digital wanted students to be able to ask questions about school policies, timetables, events, and procedures and get instant answers – without emailing the admin office or downloading a PDF that nobody reads. The 40-page handbook covered everything from uniform policy to exam procedures to safeguarding contacts to canteen menus. The head of digital had seen demos of RAG chatbots and concluded that this technology would solve their problem. In the first meeting, they described their vision as ‘ChatGPT but for our school handbook’.
The three things they wanted most clearly were: instant answers to common questions (reducing admin office email volume), a friendly interface that students would actually use, and something that could be updated without technical help when policies changed. The three things they had not thought through were: who would be responsible for keeping the chatbot content accurate, how they would handle questions the chatbot could not answer, and what they would do about safeguarding – the possibility that a student might disclose a safeguarding concern through the chatbot interface.
The Discovery Phase: What We Found
Before writing a line of code, we spent two weeks doing discovery. We interviewed the admin office team (who currently handled the email enquiries the chatbot was meant to replace), three form tutors, the head of sixth form, and the school’s data protection lead. We also reviewed the handbook itself. The discovery findings changed the project significantly. The admin office told us that about 60% of their enquiry email volume was questions that were answered in the handbook – but that students did not use the handbook because it was difficult to navigate, not because they could not find it. The data protection lead raised an immediate concern about safeguarding: any channel where students could type free-form text needed to be treated as a potential disclosure channel, with safeguarding referral protocols built in. The form tutors pointed out that the handbook was out of date in several areas – the canteen menu had changed, the uniform policy had been updated, and two of the safeguarding contact names were wrong. A RAG chatbot built on the existing handbook would give students incorrect information with confident-sounding AI delivery, which was worse than the handbook itself.
What We Actually Built: RAG Chatbot Architecture: RAG chatbot student handbook
After discovery, we rebuilt the brief around three requirements that the school had not articulated but that discovery had revealed: the content needed to be maintainable by non-technical staff, the chatbot needed safeguarding guardrails, and the system needed to handle uncertainty gracefully rather than confidently answering questions it could not reliably answer.
Content Management for the RAG Chatbot Student Handbook
The decision to not use the existing 40-page PDF as the RAG source document was the most important technical decision of the project. Instead, we built a structured content management system where school staff could maintain the knowledge base as discrete, versioned articles – each covering a single topic (uniform policy, exam procedures, library hours, canteen menu) with a title, content body, last reviewed date, and a responsible owner. The content management interface was built as a simple Django admin customisation with a rich text editor. Staff could create, edit, and review articles without technical assistance. When an article was updated, the RAG embedding pipeline automatically reprocessed the article and updated the vector index within five minutes. The structured article format – rather than a single large PDF – produced significantly better retrieval quality than chunking the original handbook, because each article was a coherent unit that answered a specific type of question without the irrelevant context from adjacent sections that PDF chunking produces.
Safeguarding Guardrails: The Hard Problem
The safeguarding requirement was the most technically and ethically complex aspect of the project. Any channel where students can type free-form text to a school system must be treated as a potential disclosure channel under safeguarding policies.
Detecting and Handling Safeguarding Disclosures in the RAG Chatbot
We implemented a two-layer approach to safeguarding detection. The first layer was a keyword and phrase classifier that ran before the RAG query, flagging messages containing language associated with safeguarding concerns (self-harm, abuse, neglect, exploitation, mental health crisis). The second layer was an LLM-based classifier that ran on every message regardless of keyword matches – given the message, classify whether it contains or suggests a safeguarding concern. When either layer flagged a message, the system response followed a strict script: acknowledge the student, provide the school’s safeguarding contact information prominently, encourage them to speak to a trusted adult, and log the message (without the response) for immediate review by the designated safeguarding lead. Critically, the system never attempted to respond to the substance of a safeguarding disclosure or to provide pastoral support – it routed to a human immediately. We also added a persistent, non-dismissible link to safeguarding contacts in the chat interface footer on every screen, so the information was always visible regardless of what the student was asking about.
Handling Uncertainty: When the RAG Chatbot Should Say It Does Not Know
The school’s initial vision of a chatbot that answered every question confidently was the second thing we pushed back on. A RAG chatbot that generates a plausible-sounding answer to a question that is not in its knowledge base is actively harmful in a school context – a student who receives incorrect information about an exam deadline or a medical emergency procedure has been made worse off than if they had been told to check with the admin office. We implemented a confidence threshold on retrieval: if the top retrieved article had a similarity score below a defined threshold, the response template was ‘I do not have reliable information about that. Please contact the admin office at [email] or speak to your form tutor.’ This deflection was not a failure mode – it was a design choice that the school explicitly approved after we explained the alternative. In production, approximately 15% of queries were deflected to the admin office in the first month, and that number gradually fell to 8% as the content team added articles covering the topics students were asking about but that were not in the initial content set.
What We Would Do Differently
Every project produces lessons. The RAG chatbot student handbook project produced several that are worth sharing for teams planning similar builds.
Start with the Content Audit, Not the Technology
We spent two weeks in discovery before touching the technology, and it was the most valuable two weeks of the project. If we had built first and discovered second, we would have built a RAG system on a PDF containing incorrect safeguarding contact information. The content audit – verifying that every piece of information in the knowledge base was current and accurate – should be completed before any embedding pipeline is built. For any organisation considering a RAG chatbot on existing documentation, the first question to ask is: do we know this content is accurate? If the answer is anything other than a definitive yes, fix the content first.
The Interface Needed More Work Than the RAG Pipeline
We spent approximately equal time on the RAG pipeline (embedding, retrieval, prompt engineering, safeguarding detection) and on the chat interface (mobile-first design, accessibility, integration with the school’s website). In retrospect, we underinvested in the interface. The RAG pipeline worked well from early testing – retrieval quality was good and the LLM responses were accurate and appropriately hedged. The interface took longer to get right because the student population was more diverse in digital confidence than we had assumed, and the initial chat interface design made some students hesitant to use it. A more extensive user testing phase with actual students before launch would have caught the UX issues that we addressed post-launch through iterative improvements.
The RAG Chatbot Student Handbook: Content Ownership Matters More Than Technology
The single biggest determinant of whether the chatbot continued to be useful six months after launch was whether the school maintained a clear content ownership model – specific staff members responsible for specific articles, with a review schedule. Schools where content ownership was clear and review schedules were followed maintained chatbot quality over time. The technology to build a RAG chatbot is not the hard part. Maintaining accurate, current content in any knowledge base is the ongoing operational challenge that determines long-term value.
Technical Implementation Summary
For teams planning similar projects, here is the technical stack and key implementation details from the build.
RAG Chatbot Stack for the Student Handbook
Backend: Django with a custom CMS for article management, Celery for embedding pipeline, PostgreSQL with pgvector for vector storage. Embedding model: sentence-transformers/all-MiniLM-L6-v2 running on CPU inference – fast enough for the content volume (approximately 80 articles) and no GPU infrastructure required. LLM: Claude Haiku for response generation – cost-efficient for the query volume and fast enough for interactive use. Safeguarding classifier: a fine-tuned BERT classifier for the keyword-sensitive first layer, and an LLM call to Claude Haiku for the second layer assessment. Frontend: a React chat widget embedded in the school’s existing website via a script tag. The entire backend ran on a single AWS EC2 t3.medium instance, costing approximately GBP 28 per month in infrastructure. LLM API costs at the school’s query volume (approximately 300 queries per school day) ran to approximately GBP 15-20 per month. The total operational cost of the system was under GBP 50 per month, which the school found significantly more acceptable than the commercial chatbot platforms they had evaluated at GBP 200-500 per month.
RAG Chatbot for Student Handbook: Pros and Cons
Pros
- Significant admin query reduction – the school’s admin office email volume for handbook-related queries fell by approximately 55% in the first three months after launch, which was the primary success metric.
- Non-technical content management – structured article CMS allowed school staff to update content without technical assistance, solving the handbook maintenance problem that the original PDF suffered from.
- Low operational cost – the custom build cost less per month than commercial chatbot alternatives, with no per-user pricing that would have made it expensive as the student population grew.
- Appropriate uncertainty handling – the confidence threshold deflection ensured students were never given confident incorrect answers, maintaining trust in the system.
Cons
- Content maintenance dependency – the value of the chatbot is entirely dependent on the ongoing maintenance of the knowledge base. Schools without clear content ownership processes saw quality decay within three to four months of launch.
- Safeguarding complexity – the safeguarding requirement added significant design and testing overhead that would not exist for a commercial or less sensitive context. Schools must take this seriously and not treat it as a minor edge case.
- Limited to handbook scope – the chatbot was deliberately scoped to handbook topics. Students frequently asked questions outside the scope (timetable lookups, homework help, teacher contact details) that required separate systems to address.
Frequently Asked Questions: RAG Chatbot for Student Handbooks
Is a RAG chatbot appropriate for a school environment?
A RAG chatbot is appropriate for a school environment when it is scoped to informational queries about school policies and procedures, implemented with robust safeguarding guardrails, and maintained with accurate, current content. It is not appropriate as a pastoral support tool, a replacement for teacher-student communication, or a channel for sensitive disclosures. The school context adds requirements that a typical business RAG chatbot does not need: safeguarding detection and referral, content accuracy obligations (incorrect information about exam procedures or safeguarding contacts has real consequences), accessibility for students with varying digital confidence, and GDPR compliance for under-18 users. These are manageable requirements but they must be addressed explicitly, not assumed to be covered by standard chatbot implementation patterns.
How do you keep a RAG chatbot knowledge base accurate over time?
Keeping a RAG chatbot knowledge base accurate requires treating it as a living document system with the same governance as any other official school publication. Assign named owners to each article or topic area – a member of the admin team owns general procedures, the examinations officer owns exam-related articles, the head of sixth form owns sixth form-specific content. Build article review reminders into the CMS: each article has a review date (typically annually, or sooner for time-sensitive content like canteen menus), and owners receive automated reminder emails when their articles are due for review. Track which articles are most frequently retrieved and which queries are being deflected – high-deflection query patterns indicate knowledge gaps that need new articles. Conduct a full content audit at the start of each academic year before the chatbot is in use by new cohorts. The technology maintains itself; the content requires human attention on an ongoing basis.
What are the GDPR implications of a school chatbot?
A school chatbot that processes student queries is processing personal data under GDPR, with additional obligations under the UK GDPR because the users are likely under 18. Key obligations: the school must have a lawful basis for processing student query data – legitimate interests (providing a school information service) is the most appropriate basis, documented in the record of processing activities. Chat logs that contain student queries must have a defined retention period (we recommended 30 days for normal queries, longer for safeguarding-flagged messages) and must be deleted automatically after the retention period. The chatbot privacy notice must be accessible from the interface and must explain what data is collected, why, and how long it is retained, in language appropriate for the age group. Data minimisation: do not collect student names or account identifiers in the chatbot if anonymous use is technically feasible – the school’s chatbot was deployed without student login, meaning queries were anonymous by design. If student login is required to personalise the experience, the additional data processing must be disclosed and justified.
How do you measure whether the RAG chatbot is working?
Measuring RAG chatbot effectiveness in a school context requires metrics at three levels. Usage metrics: daily active users, queries per user, sessions per day, and the ratio of returning users (students who come back indicate the chatbot is providing value). Quality metrics: retrieval success rate (percentage of queries answered from the knowledge base versus deflected), user feedback (a simple thumbs up/down on each response), and deflection pattern analysis (which topics are most frequently deflected, indicating content gaps). Outcome metrics: the primary outcome metric for this project was admin office email volume reduction. Track the inbound enquiry volume to the admin office before and after launch, categorised by topic, to isolate the handbook-related queries that the chatbot was designed to replace. In our case, a 55% reduction in handbook-related email in the first three months was the headline metric that the school used to justify the investment. Secondary outcome metrics included student survey feedback on information accessibility and admin staff time freed from repetitive queries.
Conclusion
The RAG chatbot student handbook project succeeded because we invested two weeks in discovery before building anything, pushed back on the initial brief where our findings justified it, and designed the system around the real requirements (safeguarding, content maintenance, appropriate uncertainty handling) rather than the initial brief (build a ChatGPT for the handbook). The technology was straightforward. The value was in understanding what the school actually needed, which was different from what they initially asked for – and building that.
Building an AI chatbot on institutional documentation and want a development team that will challenge the brief where necessary and build what actually solves the problem? At Lycore, we build RAG systems, knowledge base chatbots, and AI-powered information tools for education, professional services, and enterprise clients across the UK – with a discovery process that surfaces the real requirements before the first line of code is written. Talk to our team about your chatbot or RAG project.
