Todai Robot Project

English examination questions and artificial intelligence

---

As both Japanese and English are foreign languages for a robot, it is not particularly difficult for a robot to analyze text just because it is English. Then as a challenge for artificial intelligence, what kind of difficulties would there be in answering English examination questions?

Some of the representative themes are shown below. Compared with other subjects, English examinations contain more questions with varying difficulties, from very easy to very difficult, as problems of artificial intelligence. The key feature is that what is difficult for humans differ from what is difficult for a robot in many cases. As we advance the analysis process to solve examination questions, it becomes more evident how difficult it is (for a robot) to do what a human is doing naturally.

Questions regarding pronunciation and accented syllables appear to be answered if dictionaries are available. Especially, questions that do not require any consideration of the context in the sentence (e.g. some words are shown and choose the one that has same pronunciation) can be answered with certainty if dictionaries are consulted. Regarding the questions of pronunciation and accented syllables for the National Center Test for University Admissions, we have developed a system which refers to the dictionaries, and we have achieved almost 100 percent of the questions being answered correctly.

As the technology of machine translations has made a rapid advance in recent years, translation from English to Japanese of the exam questions in theory, should be able to be answered by machine translation.　However, when actually applying the machine translations to examination questions, the result is not as good as expected. One reason is that the accuracy of translation is presently sufficient for ordinary use, however, is not at a high enough level for answering examination questions correctly. Thus, the accuracy of machine translation needs further improvement.

Another problem is that because current translation technology is not designed to consider the word sense and context, it cannot answer examination questions effectively. The question of translation in examinations requires disambiguation based on the context in many cases. However, as machine translation in the present level of development disregards this case, it cannot translate well in the first place. We are pursuing research on the methods to perform proper translation while considering the context, and we aim to develop machine translation technique with such a high level of precision to be able to answer examination questions correctly.

Although the approaches for solving the two cases mentioned above can be successful to some degree, actually, there are not many questions like these. Many of the actual questions are asking for text reading comprehension or recognition of conversation, as in the example below.

Choose the most appropriate one for            of the following conversation.
Zack： It's already ten. We'd better be going when Bob comes back from the restroom. Shall we split the bill equally?
Koji： I'd rather not do that. I ate and drank a lot more than you two. I think I should pay more.
Zack：           
Koji： That sounds fair.
① Calm down. You don't have to get so excited.
② How about asking for a discount?
③ I wish I'd brought the coupon from the magazine.
④ Should we ask for separate checks?
 (2009 Academic Year Main Examination: English)

It is trivial to find the answer if we have common sense about restaurants (of course the answer is choice 4).  However, why is no.4 the correct answer? A logical explanation is not possible for this question, and it can only be said “Because it is natural”.

This problem does not require special reading comprehension capability, but it is asking whether the text written in English is recognized correctly.  “Recognized correctly” here means that it is understood in the same way as the mother language, and it also includes “knowing common sense of the society”. Though it is testing the comprehension on the premise of what every human would understand, it is a difficult problem for a computer to solve, as a computer cannot recognize “what every human would understand”.

Question analysis up until now, English questions are often relying on human common sense like this, an effective method to answer these types of questions has not been found yet. Questions of common sense are known to be very difficult in the research of artificial intelligence up until now, and how this problem can be avoided has been one of the key points in a lot of current research. In this project, we study a scheme for recognition of common sense questions though within the limited scope of problems asked in examinations. We advance the analysis in regards to developing mechanisms and operations that are capable of answering examination questions that rely on understanding the meaning and common sense.

The English questions of the National Center Test for University Admissions use many figures and illustrations. It is a simple figure that any human can recognize; however, it is very difficult for computers to recognize them.  A lot of research has been done on the image recognition, however, most of that research is focusing on photographs and studies of recognizing figures and illustrations is very limited. A reason for this could be that recognizing deformed images needs common sense and an effective approach has not been found so far. For humans, recognizing figures and illustrations is not an important point when answering the examination questions, but that is why it is very difficult for computers, and very natural for humans.