Randomised trial evaluation of the In:tuition programme

This summary reports the findings from two cluster-randomised trials of Drinkaware’s school-based In:tuition life skills and alcohol education intervention: one trial of the programme for 10-11 year olds in primary schools, and another for 12-13 year olds in secondary schools. The trials have been carried out by the National Foundation for Educational Research (NFER), funded and overseen by Alcohol Research UK using a grant provided by Drinkaware.

Although there has been a downward trend in consumption of alcohol amongst 11-15 year olds in England over recent years (Fuller, 2014), two-fifths (39 per cent) of young people in this age group have still drunk a whole alcoholic drink at least once; nine per cent of those have done so in the last week and five per cent usually drink alcohol once a week. The Government’s Department for Education (DfE) recognises that effective drug and alcohol education is essential to tackling the problem of drug and alcohol misuse. Reviews of school-based alcohol misuse prevention programmes, including those undertaken by Foxcroft and Tsertsvadze (2011), Cairns et al. (2011) and Martin et al. (2013) have concluded that the evidence base for effective alcohol education programmes has been mixed.

The In:tuition intervention

The development of the intervention was informed by evidence-based life skills programmes such as ‘Unplugged’ (see Faggiano et al., 2010). However, there are differences between the programmes and the measures of impact that mean the results are not directly comparable. Alcohol is the main theme of In:tuition, but its focus is wider and includes, for example, attitudes and behaviour, the influence of peers, communication skills and assertive behaviour. The main aim of the programme is to delay the age of first alcoholic drink.

The primary and secondary programmes each consist of 12 lessons (a period of 40 minutes is recommended for each lesson). Home learning tasks are included to encourage discussion with parents/carers. To access the programme, a user needs to register on the website. A helpdesk is available via the site. Lesson content is different for primary and secondary schools and, therefore, two separate trials have been implemented. A feasibility study was carried out in 34 schools across the UK (Barksfield and Hull, 2012). The trials were commissioned following this feasibility study.

Aims of the trials

The overarching aim of both trials was to evaluate the process and impact of implementing In:tuition in schools. The primary outcomes (those of greatest importance) and secondary outcomes (possible additional effects of the programme) were linked to the main aims of the intervention and are illustrated below

Schools were randomly allocated to receive the In:tuition intervention or to a ‘business-as-usual’ control. The trials compared any change over time between groups by carrying out a self-report questionnaire survey of students at two time points across two school years: before the intervention was implemented (June-September 2013) and after (June-July 2014).

The definitive analysis for the trial was an ‘intention-to-treat’ analysis, which includes all students who completed a questionnaire at both time points, regardless of the extent of In:tuition delivery (all intervention schools are treated the same). This represents how delivery might occur in a real-world scenario. To assess any differential impact by pupil characteristics, we conducted sub-group analysis in relation to primary outcomes using interaction terms. In addition, ‘on-treatment’ analysis was carried out, which accounts for differences in programme delivery.

A process evaluation, involving interviews with staff and pupils in nine case-study schools and a teacher survey, was carried out to explore programme implementation.

Sampling, recruitment and randomisation

For the primary school trial the aim was to recruit 70 schools in England containing Years 5 and 6. In total, 79 primary schools were recruited (after randomisation, 40 intervention schools and 39 control schools).

For the secondary school trial the aim was to recruit 80 schools in England containing Years 7 and 8. In total, 55 secondary schools were recruited (after randomisation, 28 intervention schools and 27 control schools).

Initially, schools were randomly selected from a sampling frame of 33 local authorities (LAs). It was then necessary to draw two top-up samples of new LAs, leading to a total of 55 LAs and 808 primary schools and 1513 secondary schools to achieve the obtained response. Schools that did not agree to participate were asked for feedback on the reasons why, although the majority said they were ‘unable to help’ and did not give specific reasons relating to concerns about the trial or intervention. Difficulties with recruitment could have been due to In:tuition’s focus on life-skills and alcohol education, which are likely to be given less of a priority in schools compared with ‘core’ subjects, such as mathematics, English or science (due to timetable and curriculum pressures).

Schools in the control group were prevented from registering on the site during the course of the trial, but were given access at the end.

Results from the primary school trial

Of the 79 primary schools randomised (into 40 intervention schools and 39 control schools), there was some attrition before follow-up. A total of 24 intervention schools and 31 control schools completed both baseline and follow-up surveys. This yielded data from 723 intervention and 1019 control pupils for the final analysis. Comparisons of their characteristics suggested that the final intervention and control groups of primary pupils can be regarded as equivalent.

Of the 40 primary schools randomised into the intervention group, only 15 were known (based on information provided) to have delivered at least some of the intervention lessons. Little information was received about why some schools had agreed to participate and then either officially withdrew or went on to not deliver lessons. The issues of attrition and programme fidelity are likely to have an impact on the measurable effects of the intervention.

There is no evidence that the In:tuition programme improves the primary outcome of resistance skills (confidence to manage peer pressure in 10 and 11 year-olds). No significant differences were evident between the intervention and control groups and there was no effect on any sub-groups.

There was an indication of an effect of the intervention on the knowledge of primary pupils. The results show that, on average, those in the intervention group had slightly better knowledge about alcohol and its effects than those in the control group, although the results were not statistically significant at a p=<0.05 level (p=0.07).

There was no significant difference between the groups in terms of their self-reported decision-making skills. Furthermore pupils in the intervention group were found to be no more likely to understand social norms relating to the proportion of young people who drink alcohol (that most people will not have a whole alcoholic drink before they are age 16) than those in the control group

Results from the secondary school trial

Of the 55 schools randomised (into 28 intervention schools and 27 control schools), there was attrition prior to follow-up, particularly for the intervention group. A total of 12 intervention schools and 15 control schools completed both baseline and follow-up surveys. This yielded data from 586 intervention and 814 control pupils for the final analysis. A comparison of their baseline characteristics yielded imbalance in two variables, neither of which were significant predictors of the primary outcome.

Of the 28 secondary schools randomised into the intervention group, only five were known to have delivered at least some of the intervention lessons; only two delivered all or most of the lessons. Again, little information was received about why some schools had agreed to participate and then either officially withdrew or went on to not deliver lessons. As for the primary school trial, the issues of attrition and programme fidelity are likely to have an impact on the measurable effects of the intervention; these issues were most prevalent among secondary schools.

There was no evidence that participation in the In:tuition programme had an impact on the primary outcome relating to the proportion of students overall aged 12-13 who were drinking frequently. However, males were more likely and females were less likely to be frequent drinkers compared to their counterparts in the control group at follow up. Note that this analysis is exploratory and there is insufficient evidence to assert this was a genuine effect of the intervention.

Although the proportion of pupils that had had an alcoholic drink by the time of the follow up survey was slightly higher in the intervention group than the control group, this difference was not statistically significant. Nor was there any significant difference between the intervention and control groups in relation to the other secondary outcomes.

Results from the process evaluation

Overall, most case-study teachers reported a range of perceived impacts on pupils, including increased knowledge and awareness, development of strategies and skills and modified behaviour. But, they felt that they would have achieved the same impact using existing provision.

Teachers were positive about the content and teaching approaches but adapted the programme to take into account the time available, the needs/context of the school, content covered in other lessons and pupils of different abilities in their class. Suggestions for improvements included:

• reducing the duration and content of the programme
• providing different formats to deliver the resources such as slides and more visual resources
• greater differentiation of content for pupils with lower levels of literacy
• pupils would like more opportunities for discussion and more pupil led activities.

Limitations of the trials

There were a number of limitations to these trials which could have had an impact on results:

• the level of measurement attrition, particularly in secondary schools
• the nature of the data obtained to conduct analysis on programme fidelity
• the programme materials being made available to schools later than expected, which could have impacted on fidelity
• the involvement of NFER staff in reminding schools to register for the intervention, which would not happen in the real world
• schools included in the process evaluation were those willing to take part (and thus are likely to be those most engaged with the programme).

To conclude, recruitment difficulties were faced, which was followed by school attrition (particularly in secondary schools) which could have had an impact on results. There was little evidence of positive – or indeed negative – impact of the programme. Pupils in schools which delivered the intervention did no better – or worse – than those in schools doing their normal Personal, Social, and Health Education (PSHE) curriculum. Evidence from the fidelity analysis and process evaluation indicates that not all teachers in the intervention group delivered the programme in its entirety – indeed some did not deliver it at all. Moreover, the extent to which In:tuition was delivered in schools did not impact on this result. Although teachers saw some value in the individual elements of the programme, for the programme to have a positive impact in its entirety it needs fundamental revision.

Barksfield, J. and Hull, T. (2012). In:tuition: Process Evaluation and Feasibility Study. London: Alcohol Research UK [online]. Available:https://www.drinkaware.co.uk/media/96379/intuition%20process%20evaluation%20final%20report%20sept%202012.pdf[27 October, 2014].

Cairns, G., Purves, R., Bryce, S., McKell, J., Gordon, R. and Angus, K. (2011). Investigating the Effectiveness of Education in Relation to Alcohol: a Systematic Investigation of Critical Elements for Optimum Effectiveness of Promising Approaches and Delivery Methods in School and Family Linked Alcohol Education. London: Alcohol Research UK [online]. Available: http://alcoholresearchuk.org/downloads/finalReports/FinalReport_0083.pdf [27 October, 2014].

Faggiano, F., Vigna-Taglianti, F., Burkhart, G., Bohrn, K., Cuomo, L., Gregori, D., Panella, M., Scatigna, M., Siliquini, R., Varona, L., van der Kreeft, P., Vassara, M., Wiborg, G. and Galanti, M.R. (2010). ‘The effectiveness of a school-based substance abuse prevention program: 18-month follow-up of the EU-Dap cluster randomized controlled trial’, Drug and Alcohol Dependence, 108, 56–64.

Fuller, E, Hawkins, V. (Eds) (2014). Smoking, Drinking and Drug Use Among Young People in England in 2013. Leeds: Health and Social Care Information Centre [online]. Available: www.hscic.gov.uk/catalogue/PUB14579/smok-drin-drug-youn-peop-eng-2013-rep.pdf [27 October, 2014].

Foxcroft, D.R. and Tsertsvadze, A. (2011). ‘Universal school-based prevention programs for alcohol misuse in young people (review)’, Cochrane Database of Systematic Reviews, 5 [online]. Available: http://onlinelibrary.wiley.com/doi/10.1002/14651858.CD009113/abstract;jsessionid=0459097D1D9E6F135C9FBD8DF60386C8.f02t03 [27 October, 2014].

Martin, K., Nelson, J. and Lynch, S. (2013). Effectiveness of School-based Life-skills and Alcohol Education Programmes: a Review of the Literature. Slough: NFER [online]. Available: http://www.nfer.ac.uk/publications/AETT01/AETT01.pdf [27 October, 2014].