Consider toy riddles (sorry if contrived) like:

Passage:

“The weather each day can be snowy, sunny or cloudy. If it’s cloudy or sunny, airplanes will fly. If it’s snowy, airplanes don’t fly, and the post won’t arrive. If airplanes fly, the post will arrive. Today the post arrived.”

Question:

“What is the weather?”

This can also be phrased as an entailment problem. The needed information is pretty straightforwardly contained in the text.

Needless to say, reading comprehension and entailment models at https://demo.allennlp.org get this wrong. The ability to represent and utilize declarative knowledge seems essential for moving past fancy pattern matching. Does anyone know of toy “bAbI” style datasets for this? You could similarly generate such riddles pretty easily. Wondering if such skills can be more readily compositionally applied to tougher problems, as bAbI hasn’t exactly achieved that.

I’ve seen this: Declarative Question Answering over Knowledge Bases containing Natural Language Text with Answer Set Programming

But those questions are considerably more complex, some requiring external knowledge not contained in the text.