A Human-Machine Collaboration Framework for the Development of Schemas
Nicos Isaak
The Winograd Schema Challenge (WSC), a seemingly well-thought-out test for
machine intelligence, has been proposed to shed light on developing systems
that exhibit human behavior. Since its introduction, it aimed to pivot the
focus of the AI community from the technology to the science of AI. While
common and trivial for humans, studies show that it is still challenging for
machines, especially when they have to deal with novel schemas, that is,
well-designed sentences that require the resolving of definite pronouns. As
researchers have become increasingly interested in the challenge itself, this
presumably necessitates the availability of an extensive collection of Winograd
schemas, which goes beyond what human experts can reasonably develop
themselves, especially after proposed ways of utilizing them as novel forms of
CAPTCHAs.
  To address this necessity, we propose a novel framework that explicitly
focuses on how humans and machines can collaborate as teammates to design novel
schemas from scratch. This is being accomplished by combining two recent
studies from the literature: i) Winventor, a machine-driven approach for the
development of large amounts of Winograd schemas, albeit not of high quality,
and ii) WinoFlexi, an online crowdsourcing system that allows crowd workers to
develop a limited number of schemas often of similar quality to that of
experts. Our proposal crafts a new road map toward developing a novel
collaborative platform that amplifies human and machine intelligence by
combining their complementary strengths.