

8 months ago admin Comments Off on Today in bullsh*t AI PR: Computers learn to read as well as humans (no)

Nice tech, but shame on Microsoft, Alibaba’s spinners

Journos fell over themselves to breathlessly report that, for instance, “ROBOTS CAN NOW READ BETTER THAN HUMANS, PUTTING MILLIONS OF JOBS AT RISK.” Real headline.




Clearly, Redmond has learned from last year’s Ms Pac-Man overreach.


SQuAD is arranged like this: you have chunks of text taken from Wikipedia, then a set of five questions and answers for every text chunk. For example, from a page on southern California, a text chunk about the region’s surroundings has the question “what is the name of the border to the south?” and the correct answer “the Mexico–United States border.”



But read the description of SQuAD more closely. It consists of “questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage.”

Alarm bells should be ringing inside your head right now. The answer to every question is explicitly contained in the text. It’s not so much reading comprehension as text extraction. There is no real understanding of the prose by the machines; it’s a case of enhanced pattern matching. Human beings are smarter than this.

Let’s go back to the southern California example. The question is…

…and the answer is “the Mexico–United States border.” But look at the associated text chunk. We’ve highlighted the relevant part by making it italic, not that, as a human reader, you need it:

The AI just has to work out that it needs to locate the words relevant to “border” and “south,” and pull out “Mexico–United States border.” Cool. Nice bit of programming. Hats off, really. But a breakthrough in machine intelligence? No. way.


Yoav Goldberg, a senior lecturer teaching computer science at Bar Ilan University in Israel, pointed out that the conditions in which humans are tested against the dataset, to set the goal line for the AI software, isn’t great. That part is carried out by people who sign up to Amazon’s Mechanical Turk, a service that pays serfs peanuts to complete jobs such as identifying objects in photos or videos, or transcribing audio to collect data.




