Developing “Mystery Shopper” capacity for alcohol services

Note: This report was funded and/or written by our predecessor organisation, Alcohol Research UK (ARUK).

Mystery shopping is the technique whereby people are paid to go into shops and services to determine the quality of the response customers receive. The aim of this project was to review the feasibility of mystery shopping in a local alcohol treatment system.

Finding a service was the most challenging stage of the project. The assumption had been that commissioners would be interested in this opportunity to review services. However, the first two contacted rejected it because they were about to re-tender services or were concerned about the impact it might have on the relationship with services. Eventually Enfield expressed a willingness to be engaged. Enfield has a single Tier 2 /3 alcohol service provided by Compass in Edmonton and the service itself was very happy to be involved. It is possible to speculate that commissioners may be more nervous about these processes than the services.

The biggest constraints on the content of the shopping were ethical concerns. Compass was very happy for shoppers to comment on issues such as: service availability, appointment systems, reception practices and information provided; however, two boundaries were set:

• Shoppers could not participate in group programmes because this would jeopardise other clients’ confidentiality.
• Ongoing counselling or other one to one interventions after an initial assessment were felt to be a bad use of staff time in a service which had enough clients to deal with already.

This limited the shopping to telephone and initial assessment contacts.

In addition, it was agreed that once the shopper had completed the interaction with the service, Alcohol Concern would immediately tell the service that they had been visited by a mystery shopper. This would mean that no further service time was used on the intervention and any records could be deleted.

Alcohol Concern approached Foundation 66’s service user committee and five people readily agreed to be part of the process. All of them saw it as a way of giving back something in their recovery process and felt it would be helpful and interesting rather than threatening. The one major concern was that no permanent record would be created of their visit in the service’s files. This had already been agreed with Compass.

Alcohol Concern developed a one day training course for the mystery shoppers. The course itself is set out in Appendix 1 of the full report and the accompanying PowerPoint slides are also available. The training focused on giving them the confidence to undertake the role using their own experience to present a convincing persona, explaining the information required and enabling them to report back in an appropriate manner.

The core of the training was role play in which the participants practised playing a mystery shopper. They key was to find a role which was as close to their own persona as possible. As recovering drinkers they could not go in smelling of drink or pretending they were very unwell when obviously they were neither. Instead the course focused on finding a role which fitted their persona. One, for example, chose to be someone in recovery who felt threatened by relapse. Another was a family member seeking help. A mystery shopper agreement was established for the participants to sign and this is set out in Appendix 2 of the full report.

Compass’s manager had agreed the process and staff were aware that it was going on. All staff were provided with an information sheet which is in Appendix 3 of the full report. Compass staff were then told that the shopping would occur over a four month period in autumn 2011. The long time period was to ensure that staff began to forget about the process and not spend all their time trying to spot the mystery shopper.

Three phone calls and four visits were made as part of the process. Each of these was written up and the findings shared with the Compass Manager and, at the end of the process, with Compass staff. Once the shopper had completed the interaction with the service, Alcohol Concern immediately told the service that they were a mystery shopper. The mystery shoppers were asked to report:

• factual information about the experience of being in a service; and
• the shoppers’ feelings about what happened to them with an explanation as to what actions or experiences made them feel that way.

They were not to make a judgement on the service only describe what happened. The shoppers either wrote a report themselves or worked with the Alcohol Concern lead to draw up a report for discussion with the service and the commissioner. In the end the most challenging problem to overcome was where the shoppers were to say they came from and who their GP was given that none of them were locals.

None of the shoppers were identified during the interviews. One suspected she might have been but the staff interviews suggested that this was not the case. Indeed the Compass staff reported identifying someone as a mystery shopper who turned out to be a genuine client!

Importantly shoppers were told that if challenged they were to deny it and if the process became impossible were to simply make their excuses and leave. This situation did not occur.

After the mystery shopping each shopper was offered a face to face support meeting. Two of them took the opportunity. Neither felt that the process had been particularly stressful and there is no evidence that the shopping jeopardised recovery in any way. All the shoppers felt nervous before the process but stated that they had enjoyed the experience in retrospect.

The staff at Compass were offered a debrief session at the end of the process. The comments were not always consistent but the key points to emerge were:

• The staff were aware that the process was taking place.
• Initial concerns focused on client confidentiality especially with regard to groupwork.
• They were simply told the process was happening and would have appreciated a meeting about it.
• They felt that as it was part and parcel of what they do on a daily basis they did not think that much about it and being very busy soon forgot about it.
• There was occasional speculation about who was and who was not a mystery shopper.
• They feel that mystery shopping is crucial.
• They felt it was good to get a neutral view of what the service does.
• It kept them on their toes in assessment.
• If the shopper moved beyond assessment it would detract from genuine client work.
• Groupwork mystery shopping would be hard for the shopper and unfair on the other clients. They would not want people to be pretending in groups.

This project was a success but not an unqualified success. Challenges and problems were experienced; however, this learning and the framework Alcohol Concern has developed can provide guidance for other partnerships who are interested in undertaking mystery shopping.

It is possible to undertake mystery shopping. The staff who were being shopped felt it was a beneficial, indeed crucial, process. The shoppers found the process relatively simple. Former service users can play the role of mystery shoppers with support and training. It is interesting that the greatest anxiety seemed to be on the part of commissioners.

The material in the full report provides a structure which other partnerships can use to develop their own processes.