Thursday, 22 November 2018 15:37

What can we learn from our mystery shopper experiments

Written by

Mystery shopper 1

Like many others, we recall being first astounded and then fascinated by Chris Durban’s two ‘mystery shopper’ experiments, on which she reported in detail at the 2011 ITI (Institute of Translation and Interpreting) Conference in Birmingham. We decided that we would love to replicate the experiment in the Netherlands, where we live and work. And so we did, adding a number of new elements at the same time. If there is one thing we learnt from our mystery shopper experiment for the 2015 Dutch National Translation Conference, it was that standards in the translation industry in the Netherlands are just as low as they proved to be in France – indeed perhaps even lower.

First of all, standards proved to be low in terms of marketing, where our mystery shopper – a Dutch marketing expert – found it much harder than he had expected to choose a translation agency as they all looked the same. He encountered bland, lookalike websites, meaningless slogans, stock photos and massive and inexplicable price differences. Plus banners brandishing cheap, cheap, cheap, with the occasional ‘quick and cheap’ thrown in for good measure. Standards were also low in customer relations, with shoddy, poorly phrased quotes, a frequent lack of interest in the demands of the job on offer, and minimal customer briefing and debriefing.

Perhaps more shockingly, standards were also low in translation quality, with the translations sold to our mystery shopper proving to be clunky and full of poor collocations, unfortunate phrasing, mis-punctuations, spelling mistakes, typing and formatting errors and – perhaps most damning of all – not even vaguely fit for purpose. This was despite the many extravagant claims made by the suppliers themselves on their websites about ‘working exclusively with native-speaker subject specialists’, producing ‘creative translations’, employing ‘experienced revisors’ and ‘specialist translators from all over the world’ and of course adopting ‘the appropriate tone of voice’.

In assessing the five translations sourced by our mystery shopper, we decided first to ask a panel of three external assessors for their general opinions. Our panel members were less than impressed: three of the translations were dismissed out of hand as being completely unusable, and the other two received mixed assessments.

Perhaps the only morsel of relief for the conference-goers was that the two not-quite-as-bad-as-the-rest translations were also the most expensive ones.


We then performed our own detailed examination, marking glaring errors in red, stylistic problems in yellow and nice ideas in green. We allotted points as follows: red = minus 3; yellow = minus 1 and green = plus 2. We were also interested in seeing whether there was any correlation with the cost of the translations in question, so Table A also shows their prices.

Table A Quality and price of five translations sourced in the Netherlands

As there was not enough time at the conference for a wide-ranging discussion of the implications of the experiment, we organized a follow-up meeting for colleagues from the profession at which they could take a closer look at the translations and debate our findings.

Their analysis of the translations generated two fascinating findings: firstly, the professional translators were broadly in agreement with the original panel of experts. They too felt that the same three translations were worthless and that the remaining two were a mix of good points and alarming weaknesses. None of the translations earned a pass mark from everyone.

Secondly, it was interesting to see that, once the professionals actually embarked on the translation of a short passage from the source text used for the experiment, they soon started asking all manner of questions. Not so much about technical terms as about the target audience and the purpose of the translation. Things like: what exactly does the company want to offer their foreign customers (the source text was a marketing brochure for a corporate events organizer)? Are they planning to organize activities abroad or do they want to invite their foreign customers to come over to the Netherlands? And, which particular foreign markets do they want to break into? Hardly trivia these, but vitally important questions if the translation is to make sense and be effective.

Curiously, not a single one of our original translation suppliers came up with any questions along these lines. True, one of them asked (before starting on the translation) whether our mystery shopper had a preference for either US or UK spelling, and another asked him whether he had any ‘special requirements’. But none of them stopped to think.

Yet stopping to think is exactly what is going to distinguish us human translators from our (ever smarter) mechanical competitors in the future. It was staggering to see that none of our five original suppliers asked for any sort of briefing. None of them called the mystery shopper with questions before delivering their translation. This was despite the fact that we actually asked our mystery shopper to add an extra (poorly formatted) paragraph containing two unnecessary technical details, and various repetitions of points that had already been made. None of our translation providers queried these. Everything was translated simply ‘as is’, including (in most cases) the wonky formatting.

And none of them reported back afterwards to explain what they had done. True, two of them sent a perfunctory ‘How was it for you?’ email, but there was no meaningful engagement at any stage. Indeed, the preferred business model would appear to be: customer chucks a text over the fence; translator translates it as literally as possible and chucks it back over the fence. Quick. Cheap. No questions asked.

We were aware, however, that we were working with a pretty small sample and could hardly claim to have applied much in the way of academic rigour to our experiment. The least we could do was to see whether the situation in the UK differed from that in the Netherlands. So we recently performed a mini-repeat of the experiment for the 2017 ITI Conference in Cardiff. Same text, same mystery shopper, same panel of experts.

What did we find? Well, first of all, it was well-nigh impossible to track down a UK-based translation provider. Despite using a variety of internet search techniques, we consistently found ourselves staring at a hit list consisting largely of global translation agencies and the odd Dutch agency dressed up as a UK operator. Even more surprisingly, individual translators were almost entirely hidden from view: even when our search phrase contained the word ‘translator’, the hits still consisted of agencies.

In the end, we managed to select five potential suppliers for our mystery shopper to contact. Two of them failed to respond to our mystery shopper’s request for a quote. This left us with two ostensibly UK-based agencies and one freelance translator. We say ‘ostensibly’ because one of the agencies proved to be Irish and the other may well have been a one-person eastern-European operation, albeit purporting to be based in a specific English town...

Table B Quality and price of three translations sourced in the UK

As expected, we encountered the same variety in terms of quality and pricing, except that the range was wider: one of the translations was quite simply appalling (‘Why did you not have a ‘woeful’ category?’ one of the assessors asked), while another was actually much better than the others, including the five we had previously sourced in the Netherlands. The findings are shown in Table B. (Good news for freelance translators: the best translation was that supplied by the freelance translator!)

So, what can we conclude from these two experiments?

1. Four of the eight translations we bought on the open market were complete and utter rubbish.
2. Three were substandard; just one was ‘natural-sounding’, showing ‘good idiomatic choices’ and ‘OK as a first draft’.
3. There was no meaningful customer communication.
4. Agency websites tend to be bland, impersonal lookalikes. They all claim to deliver high-quality products, very often both cheaply and quickly.
5. It’s hard to find individual, professional translators.

If we want to be treated as professionals, we need to act as professionals. That means, for example:

• Doing more than just supplying a passable literal translation, and hence delivering a far more professional product than our clients can get from any mechanical source.
• Communicating with our clients: asking them questions, finding out what they want and telling them what we’ve done.
• Making sure that our clients can find us on the internet and encouraging our professional associations to advertise.

Tony Parr and Marcel Lemmens are professional translators and translator trainers based in the Netherlands. They both have extensive experience as translators (freelance and in-house) and as teachers of translation, principally at the National College of Translation in Maastricht. They are the authors of Handboek voor de Vertaler Nederlands-Engels. Operating under the name of Teamwork, they organize short courses, workshops and conferences for language professionals in the Netherlands.



Read 7304 times Last modified on Saturday, 21 August 2021 22:57

Other blog articles