Turing test transcripts reveal how chatbot 'Eugene' duped the judges

Transcripts which show how a chatbot became the first computer to pass the renowned Turing test are revealed for the first time in a newly published paper in the Journal of Experimental & Theoretical Artificial Intelligence.

A year after Eugene Goostman – an ‘artificial conversational entity’ posing as a 13 year-old Ukrainian boy – fooled enough judges into incorrectly identifying it during the test, researchers from Coventry University claim that idiosyncrasy, inquisitiveness and the odd grammatical error were behind the bot’s success.

Professor Kevin Warwick and Dr Huma Shah, the paper’s authors, examined 10 of the transcripts in which judges were duped by Eugene, and considered how the deception was achieved.

The researchers conclude that one ‘ploy’ by the chatbot was to attempt to take control of the conversation and steer it by answering a question with a question or even changing the subject as real people often do. Such personality traits – combined with the occasional spelling error – “add human credibility”.

In one spiky exchange, the judge – who does not know whether they are talking with a computer or a human – tells Eugene they are an actress and writer. The conversation proceeds as follows:

EUGENE: Well, I’ll read something of your books, if they ain’t too tedious (which is highly possible:-)
JUDGE: grammar… aren’t too tedious! A bit rude of you…
EUGENE: Calm down, please.
JUDGE: i assure you i am very calm… are you?
EUGENE: Don’t even bother me with the fact that you are very calm…

Another transcript sees a different judge take a firmer line of questioning:

JUDGE: what is your gender
EUGENE: I’m a young boy, if you care to know. And not very ugly, by the way!
JUDGE: what are you aware of at this moment
EUGENE: I would rather not talk about it if you don’t mind. Could you tell me what are you? I mean your profession.
JUDGE: how old are you
EUGENE: I’m a little boy 13 years old.
JUDGE: what are your qualifications
EUGENE: My qualifications? I’m not gonna tell you. Oooh. Anything else?

And in a further repartee with a different judge, the conversation unfolds as follows:

JUDGE: Hello
EUGENE: Hello, I’m really glad to have the chance to chat with you! My guinea pig Bill sends his regards too!
JUDGE: Is Bill a male or a female?
EUGENE: Ask Bill personally, please.
JUDGE: Well I’d rather talk to you. What is your name?
EUGENE: Call me Eugene. I am glad to talk to you!
JUDGE: My name is Jane and I am female. How about you? What’s your gender?
EUGENE: I’m a male. A “guy” I’d say.
JUDGE: Pleased to meet you Eugene. What’s the weather like where you are?
EUGENE: Let’s get on with our conversation!
JUDGE: Don’t you like talking about the weather?
EUGENE: All these talks about weather is a waste of time.
JUDGE: What would you like to discuss?
EUGENE: I don’t know… Better tell me more about yourself!

The excerpts are from the 10 out of 30 (33%) conversations with Eugene which resulted in the judges not being able to identify the chatbot as a machine (30% is the barrier which needs to be broken to pass the Turing test).

Professor Kevin Warwick, deputy vice-chancellor for research at Coventry University, said:

Wherever you stand on the Turing tests, what is clear from last year’s results and our new examination of the transcripts is that there are plenty of emerging discussion points to fuel the philosophical debate around artificial intelligence and how humans communicate.

It’s fascinating to read through Eugene’s remarks during the exchanges. Evidently the judges were deceived by the bot’s propensity for light-hearted, idiosyncratic responses to questions and for humanlike errors. Ultimately, when it leads to the judges failing to tell the difference between a computer and a human in this way then we’re gaining a significant insight and indication into the capabilities of today’s machine conversation systems. And it’s quite impressive.

The paper, entitled ‘Can machines think? A report on Turing test experiments at the Royal Society’ can be accessed here. 

For further information, please contact Mark Farnan, press officer, Coventry University, on +44 (0)24 7765 8245 or email mark.farnan@coventry.ac.uk.