10 Questions to Ask an AI Provider Before Starting a Pilot
The demonstration is polished. Before your committee agrees to a pilot, use these questions to test the provider's answers, evidence and readiness for a real school setting.
In this playbook12 parts
- Prepare for the session
- 01 What school problem will this pilot solve, for whom and how will we know?
- 02 How will the product fit our curriculum, languages, learner needs and local context?
- 03 Where does teacher judgement remain in the workflow?
- 04 What data will the service use, where will it go and will any of it train or improve models?
- 05 Who can access what, and how are accounts and information secured?
- 06 What happens when the AI is wrong, biased, harmful or inappropriate?
- 07 What will teachers, leaders and technical staff have to do in everyday use?
- 08 What training, onboarding and ongoing support are included?
- 09 What evidence supports each claim, and how will this pilot test it here?
- 10 What will it cost to run, change, scale and leave, and what happens after a successful pilot?
- Completion check
Ask every provider the same ten questions. Request evidence for every material answer. If a safeguarding, data or governance gap remains unresolved, pause rather than allowing a strong feature list or attractive price to cancel it out.
This playbook helps an international-school committee run a focused provider meeting. It is not legal or procurement advice. Apply the law, policies, safeguarding duties and approval routes that govern your setting.
What school problem will this pilot solve, for whom and how will we know?
The provider can name the intended users, one clearly defined use, what sits outside the pilot and which signs or measures will show whether it is useful.
Without a clear purpose, the committee cannot judge whether the product, data, workflow or evidence fits the problem. A busy pilot can still teach the school very little.
The DfE's guidance on generative AI in education recommends defining the intended use and checking that its benefits outweigh the risks.
- Ask to see
- A short written description of the pilot use, a worked example, the starting point and measures, and the limits of what the product is intended to do.
- Pause if
- The answer returns to a feature tour, promises broad transformation, or treats logins and positive comments as proof that learning, workload or operations improved.
- Write down
- The intended users, the one-sentence pilot purpose, exclusions, baseline, measures and person who will review them.
How will the product fit our curriculum, languages, learner needs and local context?
The provider names the curricula, subjects, age ranges, languages and accessibility needs it supports, explains the limits and lets your academic team test realistic material.
Built for education does not show that a product fits an international school's curriculum, language mix, learner needs or expectations for teacher review.
UNICEF's EdTech for Good Framework 2.0 asks whether a tool is appropriate for its setting and accessible to different learners.
- Ask to see
- Curriculum mapping, examples using school-supplied material, localisation notes, an accessibility statement and a record of known limitations.
- Pause if
- Curriculum aligned is not tied to a named curriculum, the provider will not test your material, or staff are expected to correct extensive errors without that work appearing in the pilot plan.
- Write down
- The contexts supported, the gaps, the examples your school will test and who decides whether the fit is sufficient.
Where does teacher judgement remain in the workflow?
The provider can show where a teacher reviews, changes, approves or stops an AI-supported action, and who can see what happened.
Saying that a human is involved is not enough. The control must appear at the point where an output could affect teaching, feedback, access, support or assessment.
UNESCO's human-centred guidance for generative AI places human agency at the centre of decisions about educational use.
- Ask to see
- A demonstration of the real workflow, a clear list of who is responsible, the permission settings, admin guidance and an example activity record.
- Pause if
- Human in the loop cannot be demonstrated, important actions proceed without meaningful review, or staff can see an output but cannot correct or stop it.
- Write down
- The review, correction, approval and stop points, the responsible role at each point and the route for exceptions.
What data will the service use, where will it go and will any of it train or improve models?
The provider can walk you through what information the pilot uses, why it needs it, where it goes, who can see it, how long it stays and whether it is used to train or improve a model.
Your school needs that full picture before it can judge the privacy and safeguarding risks of the proposed pilot.
The ICO's AI and data protection risk toolkit helps organisations examine personal-data flows, responsibilities and risks across an AI system.
- Ask to see
- A simple data-flow diagram, the current privacy notice and contract terms covering other companies involved, storage locations, transfers, retention, deletion and model training.
- Pause if
- The provider cannot explain the flow in plain language, optional collection is on by default, or the provider cannot give your school the controls it needs.
- Write down
- What information is used, why, where it goes, who receives it, how long it is kept, how it is deleted, whether it is used for training and what your school can control.
Who can access what, and how are accounts and information secured?
The provider can show that access matches school roles, administrators can change it and important activity is recorded. It can also explain how accounts are protected and what happens if something goes wrong.
A pilot often involves different responsibilities for teachers, learners, leaders, support staff and provider personnel. One access level for everyone creates avoidable risk.
The DfE's cyber security standard for schools and colleges sets expectations for secure accounts, access controls and incident preparation.
- Ask to see
- A list of what each role can see and do, how accounts are protected, what administrators can control, an example activity log, recent security checks and who to contact if something goes wrong.
- Pause if
- Shared accounts, broad default access, old or unclear security evidence, or no named route for reporting and managing an incident.
- Write down
- Each role, what it can access, who administers it, how accounts are protected, what activity is recorded, the security evidence reviewed and the incident route.
What happens when the AI is wrong, biased, harmful or inappropriate?
The provider explains known limits, how it tests and reduces risks, what users and staff can report, who receives an alert and when the school can restrict or stop use.
AI output can sound confident while being inaccurate, unsuitable or unsafe. Learner-facing uses need clear prevention, monitoring and human response.
The DfE's generative AI product safety standards cover testing, filtering, monitoring, reporting and responses to harmful output.
- Ask to see
- The current risk assessment, safety-test summary, filtering and moderation coverage, age and role settings, incident workflow, reporting example and recent product-change log.
- Pause if
- Guarantees of perfect accuracy or safety, no realistic testing across relevant languages or media, no route to alert the school, or responsibility is pushed back to the user without practical controls.
- Write down
- The known risks, controls, monitoring, report route, provider response, school owner and conditions that trigger an immediate pause.
What will teachers, leaders and technical staff have to do in everyday use?
The provider can show the normal workflow, preparation, checking and administration, including what happens when normal use goes wrong and who is responsible.
A demonstration can hide the correction, setup and support work that determines whether the pilot is manageable for staff.
The EEF's school implementation guidance focuses attention on how a new approach changes people's day-to-day work in schools.
- Ask to see
- A worked example using school material, a simple list of who does what, the setup plan, administrator instructions, time assumptions and the support route when normal use fails.
- Pause if
- Manual work is described as simple but not shown, staff gaps are left for the school to solve later, or the pilot depends on one enthusiastic colleague being permanently available.
- Write down
- The workflow, tasks by role, estimated preparation and review, provider duties, exceptions and unresolved capacity gaps.
What training, onboarding and ongoing support are included?
The provider sets out what each role needs before use, who delivers it, how staff get help later and how the school learns about product changes that affect risk or practice.
A sales demonstration does not prepare teachers, administrators, safeguarding staff and leaders for their different responsibilities during a pilot.
TeachAI's practical guides for school users include role-specific preparation, AI literacy training and ongoing review.
- Ask to see
- The training plan and materials, delivery responsibilities, support channels, response expectations, escalation route, refresher offer and product-change communication process.
- Pause if
- The demonstration is presented as training, support ends at launch, the school must create all guidance, or response expectations remain verbal.
- Write down
- Training by role, delivery owner and date, support route, response expectation, escalation contact and change-notice process.
What evidence supports each claim, and how will this pilot test it here?
Each learning, workload, operational or cost claim is matched to evidence that measures that outcome, with the method, setting, users, duration and limits made clear.
A case study, usage figure or satisfaction survey may be useful, but it does not prove every result a provider may promise.
The EEF's guide to using research evidence helps school teams question what evidence can show, where it is limited and whether it applies locally.
- Ask to see
- The original study or evaluation, how it was carried out, the relevant sample and context, its limitations, an explanation of how usage measures are defined, case-study details and a proposed pilot measurement plan tied to your starting point.
- Pause if
- Testimonials or logins are used as proof of learning or workload reduction, the method is unavailable, unfavourable findings are omitted, or the provider guarantees the same outcome in your school.
- Write down
- Each claim, the evidence supplied, its limits, the local measure, baseline, evidence owner and review date.
What will it cost to run, change, scale and leave, and what happens after a successful pilot?
The provider can show the full cost and responsibilities for the pilot, renewal, change, expansion and exit, including data export and deletion and the conditions for stopping, changing or expanding the use.
Licence price alone does not show staff time, integrations, variable usage, optional features, support, renewal or lock-in. A successful pilot still needs another decision before wider use.
The DfE's guide to procuring education technology recommends clarifying responsibilities, data portability and exit arrangements before signing.
- Ask to see
- Itemised pricing, usage limits, implementation and integration scope, support terms, renewal and change clauses, export format, deletion process and the proposed post-pilot plan.
- Pause if
- Important charges appear only later, variable costs are opaque, the school cannot export its information, deletion cannot be confirmed, or wider rollout is promised without named responsibilities and conditions.
- Write down
- Full cost, usage assumptions, contract change points, export and deletion, post-pilot support and the stop, change or expand decision criteria.
