DialogFlow Testing Tutorial

This guide should be used in conjunction with Florian’s GitHub Wiki post and Florian’s Chatbots Magazine article. It assumes you have basic understanding of the command line, that you’re running Windows, have an existing English Dialogflow chatbot, but are otherwise clueless on how to advance. Sound like you? Great, let’s get started.

Installation

For installation, you should adhere to the instructions in the above links, but here are the highlights for your convenience:

1. Download and install node.js and git
2. From the command line type npm install -g botium-cli
   • Running into errors? Try npm install -g windows-build-tools, try installing Python 2, or run command prompt as administrator.
3. Read through the rest of the installation section of the article, especially making sure that your botium.json file is set up correctly.
4. When you're done, wrap up with npm install -g mochawesome

Creating Test Files

Go ahead and run botium-cli import dialogflow-conversations, making sure that you execute it from a clean folder in which you’re okay with a bunch of text files appearing there shortly.

Hopefully you saw a flash of text in the console, and files have appeared in the folder of your choosing. Good? Great! Now come creating test cases. It’s most straightforward if you do it manually, but I have some tips for semi-automating this process. But first:

Background on the botium file format

For each intent your bot had, you will see three files generated after running that import command.

1. The *.convos.txt file: the heart of the test case
   • Open one and you'll see a title line, a #me line, an input line, a #bot line, and an output line. These details will matter if you will be typing out your test cases manually.
2. The *_input.txt file: if the input line in the convos file refers to a filename, it is referring to this file
3. The *_output_0.txt file: if the output line refers to a filename, it is referring to this file.

Now, this guide assumes that all conversations with your bot end with one exchange. I realize that some bots support more complex conversations (or in Dialogflow language, have "context"). If this applies to you, that means you should add more lines of #me and #bot within a single convos file (albeit, you'll have to do it manually as my automation techniques do not support this case).

For an example and a more extensive explanation of the botium file format see this Wiki page.

---

Ok, essentially you have three options for how you'd like to create test inputs:

1. type them directly into your convos files,
2. collect them through outside sources, or
3. generate variants for the inputs that your bot already supports.

I hope I've done a good enough job introducing the convos format to make (1) self-explanatory, but I can always add more instructions to anything upon request. For now though, allow me to jump to (2).

Test Cases through Outside Means

So if you're reading this section, you should have a file somewhere that has a list of inputs (or "utterances", as botium calls them) you'd like to test your bot with. Ideally, they should be in a .txt file with the inputs separated one-per-line. If not, reformat it now, or else refer to (1).

Feel free to use/adapt this script to generate new convos files using your .txt file (did you install Python?). To use it, move the file a new directory (so it can generate new convos files) and run it. Notice that the script did not generate any input or output files, but that should be okay! The only exception would be if a single intent in your bot has more than one programmed response. Otherwise, it's just a matter of going through these new files and pasting in the correct response under the #bot tag.

Automated Variant Generation

This option is fully automated, but relies on an API that is limited to the simplest of sentence restructuring techniques. Still, it is valuable to see how your bot fares to minor rewordings of your inputs. Here's the code. But not too fast—sign up for a trial at api.bitext.com, go into your settings, and replace the User Token! For this script, place it in the same directory as the convos files that you imported in the beginning, and let it run. It will automatically modify your input files for testing, but don't worry; if something goes wrong you can always re-import your files from the command prompt.