Note: This report was funded and/or written by our predecessor organisation, the Alcohol Education Research Council (AERC).
5 July 2006
Researchers:
Tober, G., Clyne, W., Finnegan, O., Farrin, A. and Russell, I. in collaboration with the UKATT Research Team
Key findings
The scale was able accurately to detect components of each of the treatments and to discriminate between them, to demonstrate that treatment was delivered as planned and that characteristics of the other treatment were either missing or were delivered infrequently. The summary ratings for each treatment are indicative of the ability of the scale to discriminate the two treatments. All treatment-specific items in the UKATT-PRS showed significant differences between the two treatments indicating that the scale is able both to discriminate the treatments generally, and to detect the delivery of all specific components of the content and style of the two treatments that are included in the scale.
Measurement of agreement between the three independent raters for individual items was relatively high and agreement was comparable for MET and SBNT sessions. Agreement between independent raters was equally high for frequency and quality ratings. This suggests that the scale is a reliable measure of components of the two treatments.
Introduction
The United Kingdom Alcohol Treatment Trial (UKATT) research team have developed a Process Rating Scale (PRS) to measure the delivery of Motivational Enhancement Therapy (MET) and Social Behaviour and Network Therapy (SBNT) for the treatment of alcohol dependence and misuse in the UK context. This method is unique in its ability to rate the delivery of SBNT, and to compare this treatment with another treatment of proven effectiveness.
Social Behaviour and Network Therapy aims to identify and work with a social network supportive of positive change in eight structured sessions including network based communication and coping skills, relapse prevention and alternative activities to drinking. The purpose of Motivational Enhancement Therapy is to bring about an internally motivated decision to change drinking behaviour.
The purpose of the present study was to develop and validate a manual based method of rating treatment fidelity, based upon frequency and quality of the delivery of treatment components, treatment manual adherence, therapeutic style and discriminability between treatments. The UK Alcohol Treatment Trial Process Rating Scale (UKATT-PRS) is a 26 point rating scale designed to be time-efficient, to allow all sessions to be rated by the same rater, to be readily adaptable to use with a range of therapeutic approaches to substance misuse treatment and appropriate for use in a UK context.
For each item, a definition, a description of the characteristics of high and low ratings for frequency and quality and examples of therapist dialogue illustrating these were provided in manual form. General guidance on differentiating the frequency and quality of therapist behaviours, on avoiding common pitfalls relating to possible rater bias and on the method for note-taking during the session was also included.
Video recordings of two psychosocial treatments of alcohol misuse and dependence delivered in the UK Alcohol Treatment Trial (UKATT) were available for the purpose of validating the rating scale. One video per client (where available) was sampled for process rating. The sample of over 400 video tapes was stratified by treatment (MET, SBNT), session number (1-3 for MET, 1-8 for SBNT) and centre (South Wales, Leeds, West Midlands). Replacement sampling was used when a video was subsequently found to be un rateable, in order to retain the balance between treatments, session numbers and centres. A target of 50 randomly selected videos was set for double rating and of these 25 would be triple rated by a further two independent raters. This method ensured inclusion of ratings throughout the entire treatment phase of the trial and ensured balance by treatment, session number and centre.
Implications
The requirement to monitor the delivery of psychological and social treatment underlies clinical governance of routine clinical practice, supervision and psychotherapy research. It is therefore necessary to have a reliable and valid method to quantify and assess the quality of treatment delivery. Treatment integrity or fidelity checks provide the means to examine the extent to which treatments are delivered and the quality of such delivery.
Interpretation of the findings in both effectiveness and efficacy trials in psychotherapy research requires measurement of treatment implementation. Variations in competence can be identified and potential treatment effects more accurately attributed. For example, where there are no treatment fidelity checks, treatment effects could be wrongly attributed to the treatments themselves rather than a difference in therapist competence. Equally the potential emergence of a treatment effect may be masked by variations in the extent and quality of the delivery of treatment.
The UKATI-PRS is a valid and reliable method of rating the delivery of two psychosocial treatments for alcohol problems and dependence. It is likely to be able to be adapted for the purpose of rating the delivery of other psycho-social treatments applying the same principles used in its development. It can therefore form the basis of measuring performance and treatment fidelity in clinical trials, in treatment audit and in routine supervision of practice.
