But it was very interesting to see the 'next response' candidates for the two sample chats in Table 1 (p3 of the PDF). In particular : it was alarming to see how much their Deep Learning response selection mechanism had to chose not so much the best response out of a selection of decent responses, but more the most acceptable response out of a selection of mostly horrible ones.
While we're in this topic: Does anyone know of existing open source implementation (or at least a good starting point should I start myself) of chatbot that can read textual input (e.g. FAQ, handbook) and automatically use it to answer chat?
I've used WIT.ai and API.ai (now Dialogflow), and they both require you to give a bunch of example sentences (e.g. "Yes", "Okay", "Sure"), assign them an intent ("YES"), and use that intent in your custom code (if intent is YES then...). I found this to be tedious and limiting.
I believe the Microsoft Bot Framework will consume a standard FAQ and answer questions from it, but you'll have to confirm. I saw a demo doing just that last year, but I'm unsure how much extra work there was.
I tryied something like that- to generate a sort of stack overflow chatbot from a tech support chat my OS-Comunity keeps up.
The problems usually are not in training the thing, the problem comes with filtering the data.
There is usually a lot of meta- and memes in the chatlogs, and lots of answers come as a web link. Which is not always helpfull. Imagine a user is asked to pastebin a log. The neural net must expand this pastebins into the chat.
Its very easy for a neural net to get lost in these logs or deduce the wrong thing from them.