The Loebner Prize is the first formal competition that implements the famous Turing Test, which asks "can a machine think?" Turing suggests that if the artificial intelligence agent can fool humans that it is actually human, the machine is thinking. If a machine is found to pass the test, it will win the Loebner Prize gold medal and 100,000 USD. Since machines are pretty far off from this goal, to stimulate interest, the Loebner prize awards a bronze medal every year along with 3000 USD to the agent "closest" to passing the Turing Test.
This year's winner is Elbot by Artificial Solutions, which you can chat with online. I am rather surprised by the fact that it makes grammatical sentences while incorporating semi-relevant information in its responses. I am going to guess that someone spent a really long time writing up rules for responses. Elbot's responses are a big improvement (depending on how you look at it) over ELIZA, which was made in 1966.
A lot of things are worth doing for money. For example, I would probably take a job doing data entry eight hours a day if I was getting paid 300,000 USD a year. Current contestants in the Loebner prize can only practically expect to win 3,000 USD, which is far less than the amount of work put into a lot of these bots. Therefore, in my opinion, there almost always has to be some kind of exterior funding to the production of these bots. There is no way any self-respecting AI researcher would spend his own time developing a chatbot like the ones in this year's competition.
The Loebner prize (and perhaps the Turing Test), at best, is a poorly defined fitness function that a population of chatbot developers are converging towards. Instead of using actual learning or reasoning, the developers of these hacks use shortcuts such as a massive amount of predefined rules, reorganizing input text, etc. The Loebner prize has created a local maxima that is difficult to get out of in the grand scheme of developing truly intelligent machines.
I think the main reason for so much hatred directed at the Loebner prize comes out of the fact that Turing's monumental article Computing Machinery and Intelligence has been twisted into some sideshow contest. Turing is one of the founders of computer science and AI, and many agree that the Loebner prize has very little to do with Turing and his vision. In my opinion, like many things like this, people should not read too literally into statements written by people in the naive past.
Something to realize, though, is that these chatbots do have some purpose in this world. Why else are they getting funding by businesses? They could help with automated telephone calls (not sure if this is a good thing or not), automated online technical support, seemingly intelligent conversations with non-player-characters in video games, etc. There is demand for applications in these domains and many people aren't willing to wait around for someone to create Strong AI. The Loebner prize's downfall is this rather small domain in AI has an unfortunate association with the founding ideas set forth through Turing, that many people respect.
There is one major reason I think the Loebner prize is hurtful to the AI community. Unfortunately, from first glance, chatbots may be the most interesting thing to ever come out of AI. The media and uninformed bloggers just can't help to over-sensationalize this contest. For example, the article titled "Loebner Prize Winner Announced; Is He Human?". NO. It is NOT human. What the hell is wrong with you? AI in general, and even more so chatbots are not anything close to actual intelligence. Do you consider a huge database of response rules intelligent? The article "University of Reading to host artificial intelligence battle royale" is more subtle in its small contribution to the distortion of the public perception of AI. The Loebner prize is NOT an AI contest. It is a chatbot contest. Would you say an ice sculpture contest is an "artistic battle royale"? NO. This makes people seem to think that myself and other AI researchers are sitting around working on chatbots. I guess this is a problem throughout the media with all domains, not only this one. Regardless, every time I read some article about how the Loebner prize might produce the next HAL9000 and may be the dawn of the real-world Matrix, I throw up a little bit in the back of my throat.
Oh, and please stop with the HAL9000 eye image that is usually placed along with these articles.
Aside from those mentioned here, these are the problems I see with the Loebner prize...
ReplyDelete1. The questions asked to the computer are sometimes things like; "Do you like to swim?" The computer, even if truly intelligent, does not swim and is forced to lie in order to fool the human judge.
2. As the computer "intelligence" becomes better, the odds of a computer being selected as human approaches 50/50. If the computer's intelligence were evenly matched with that of a human's, the judge would essentially be flipping a quarter to decide who is real and who is a computer. It then becomes luck of the draw.