Integration with Google Dialogflow

Blueworx Voice Response can connect to an Google Dialogflow using CCXML.

Using Google Dialogflow with BVR

Integrating a BVR CCXML application with Google Dialogflow is achieved using a dialogstart with the src field specifying a path to the Google Dialogflow project's JSON credentials. Alternatively, the application can get the Dialogflow definition from VIRTUAL_ASSISTANT_DIALOGFLOW_ES, VIRTUAL_ASSISTANT_DIALOGFLOW_CX or VIRTUAL_ASSISTANT_NU_WORKFORCE call features by setting the src to "use_call_features"

The application MAY use external TTS and STT engines if desired by attaching external TTS and STT call features to the application. If there are no TTS or STT call features specified or external_tts/external_stt parameters are set to false, Dialogflow will use its own internal TTS and STT capabilities. See here for details: BAM Command Line Utility Call Features Panel

Along with the Google Dialogflow JSON credentials path to be used for the src parameter, the following CCXML variables may optionally be set up for use with the dialogstart:

The following optional parameters may be supplied

external_tts Determines whether to attempt to use external TTS call features. If set to false, Dialogflow's built in TTS capabilities will always be used. If set to true and no matching external TTS call features are attached to the application, Dialogflow's built in TTS capabilities will be used. If set to true and matching external TTS call features are assigned to the application, they will be used.
external_stt Determines whether to attempt to use external STT call features. If set to false, Dialogflow's built in STT capabilities will always be used. If set to true and no matching external STT call features are attached to the application, Dialogflow's built in STT capabilities will be used. If set to true and matching external STT call features are assigned to the application, they will be used.
input_detection_mode Sets the method BVR uses to determine whether speech has started (which affects when prompts are barged in to and when the max speech timer starts). Valid options are NOISE, FIRST_INTERIM_RESULT and FIRST_FINAL_RESULT. NOISE is where BVR considers input to have started if initial_speech_duration milliseconds of noise has been detected. FIRST_INTERIM_RESULT is where BVR considers input to have started when an interim result is returned from the STT engine. FIRST_FINAL_RESULT is where BVR considers input to have started when a final result is returned from the STT engine. NOISE is faster, but can be triggered by unexpected non-speech sounds. The default is NOISE.
initial_speech_duration Determines the amount of speech (in milliseconds) that is considered enough noise to be indicate the start of input. Used when input detection mode is set to NOISE.
no_input_timeout Sets the no input timeout (in milliseconds). If not specified, the no input timeout defaults to 30 seconds. The no input timer is started after any text has finished rendering (regardless of the bargein parameter). If the no input is triggered, BVR will send a message to Dialogflow with empty input text and sets the custom context variable bw_no_inputto true
max_speech_duration Sets the maximum amount of speech to detect. The timer starts when initial speech has been detected. If the timer fires before a final result comes back from the STT engine, the STT is stopped and any result is sent to Dialogflow.
stt_confidence_threshold Sets the confidence threshold to use when getting an STT response. If the STT result is below the confidence threshold, BVR will send a message to Dialogflow with empty input text and sets the custom context variable bw_no_match to true
initial_input_text If specified, BVR will send the text value of this variable to Dialogflow when it first starts the dialog, which will behave as if the words were spoken to start with. If this is not specified, Dialogflow will await voice input.

Additional more complex parameters can be supplied in the hints

context An object containing Context objects to set the initial input context for Dialogflow. As these are not arbitrary variables, they must be in a specific format. See "Setting initial input contexts"
payload An object containing a custom initial payload to send to Dialogflow. See "Setting the initial payload"
stt_parms An object containing key/value pairs to send to the STT engine.
tts_parms An object containing key/value pairs to send to the TTS engine. See "Setting the TTS parameters"

For more on how to configure STT parameters, please see Dialogflow Virtual Assistant Specifics

Once these variables have been setup, the Google Dialogflow dialog can be started with a dialogstart like the following:

<dialogstart src="dialogflow_credentials_json" type="'application/google-dialogflow'" connectionid="evt.connectionid" parameters="list of parameter variable names" hints="hints_obj"/>
            

Or alternatively, to use VIRTUAL_ASSISTANT_DIALOGFLOW_ES, VIRTUAL_ASSISTANT_DIALOGFLOW_CX, or VIRTUAL_ASSISTANT_NU_WORKFORCE Call Feature definitions:

<dialogstart src="'use_call_features'" type="'application/google-dialogflow'" connectionid="evt.connectionid" parameters="list of parameter variable names" hints="hints_obj"/>
            

The provided sample application has an example of defining and using those variables.

The Google Dialogflow dialog finishes when Dialogflow returns a JSON payload containing the bw_action variable, and it is set to an action that ends the dialog, such as "end_dialog" or "transfer"

Setting initial input contexts

It is possible to set the initial Context(s) when starting a Dialogflow session. This is an Object containing the Context name as the key, and an Object containing parameters relating to the context as the body

The only mandatory parameter for the context object to contain is the lifespanCount, however it is also possible to attach parameters to the input context if desired

            
<!-- Create the hints object -->
<var name="dialogflow_hints" expr="new Object()"/>
<!-- Create the context map that will contain all input contexts we want to send to Dialogflow -->
<var name="dialogflow_context_map" expr="new Object()"/>

<!-- Create a context -->
<var name="my_input_context" expr="new Object()"/>
<assign name="my_input_context.lifespanCount" expr="'2'"/>
<!-- Optionally, you can add parameters to the context -->
<var name="my_input_context_parms" expr="new Object()"/>
<assign name="my_input_context_parms.custom_parameter" expr="'custom_parm_value'"/>
<assign name="my_input_context.parameters" expr="my_input_context_parms"/>

<!-- Add the context to the contexts map -->
<assign name="dialogflow_context_map.input_context_name" expr="my_input_context"/>
         
<!-- Set these custom context variables to the hints object under the name dialogflow_custom_context_vars -->
<assign name="dialogflow_hints.context" expr="dialogflow_context_map"/>

<!-- Now supply these hints when starting the dialog -->
<dialogstart src="dialogflow_service_url" type="'application/google-dialogflow'" connectionid="evt.connectionid" hints="dialogflow_hints"/> 
         

Setting TTS parameters

Just like with any VoiceXML started by CCXML, you can override your Text To Speech parameters for a specific VoiceXML session - see here for details.

Here's a dialogflow specific example:

            
<!-- Create the hints object -->
<var name="dialogflow_hints" expr="new Object()"/>
<!-- Create the tts_params object that will contain all TTS params -->
<var name="dialogflow_tts_params_obj" expr="new Object()"/>
<!-- Create the google object that will contain all Google TTS specific params -->
<var name="dialogflow_google_obj" expr="new Object()"/>
<!-- Setup some custom google variables -->
<assign name="dialogflow_google_obj.my_variable" expr="'some value'"/>
<assign name="dialogflow_google_obj.my_other_variable" expr="'some other value'"/>
<assign name="dialogflow_google_obj.my_number_variable" expr="9"/>
<assign name="dialogflow_google_obj.my_boolean_variable" expr="true"/>
<!-- Set the Google TTS parameters to the TTS object -->
<assign name="dialogflow_tts_params_obj.google" expr="dialogflow_google_obj"/>
<!-- Set the TTS params object to the hints object -->
<assign name="dialogflow_hints.google" expr="dialogflow_tts_params_obj"/>

<!-- Now supply these hints when starting the dialog -->
<dialogstart src="dialogflow_service_url" type="'application/google-dialogflow'" connectionid="evt.connectionid" hints="dialogflow_hints"/> 
         

Setting the initial payload

An initial JSON payload is sent on the first message sent to Dialogflow

BVR sets some variables in a payload by default. These can be overridden in custom payload variables. The defaults are:

Name Sent on Description
bw_DNIS First message The "DNIS" or called number (the "number", or user-part of the SIP URI that was used to call BVR)
bw_ANI First message The ANI or calling number (the "number", or user-part of the SIP URI that called into BVR)
bw_ccx_session_id First message The CCXML session ID that is being used to connect to Watson Assistant
bw_confidence Every message The confidence value returned from the STT engine
bw_no_input Every message A boolean value. If set to true this indicates that no input was received
bw_no_match Every message A boolean value. If set to true this indicates that the input received by the STT engine was below the confidence threshold (set by bw_stt_confidence_threshold)

It is also possible to include custom variables in the initial payload. These can be supplied as an object called "payload" in supplied hints when starting the dialog. Note that these will be merged with the variables supplied by default.

            
<!-- Create the hints object -->
<var name="dialogflow_hints" expr="new Object()"/>
<!-- Create the payload object that will contain all custom payload variables we want to send to Dialogflow -->
<var name="dialogflow_payload_obj" expr="new Object()"/>
<!-- Setup some custom payload variables -->
<assign name="dialogflow_payload_obj.my_variable" expr="'some value'"/>
<assign name="dialogflow_payload_obj.my_other_variable" expr="'some other value'"/>
<assign name="dialogflow_payload_obj.my_number_variable" expr="9"/>
<assign name="dialogflow_payload_obj.my_boolean_variable" expr="true"/>
<!-- Set these custom payload variables to the hints object under the name dialogflow_custom_payload_vars -->
<assign name="dialogflow_hints.payload" expr="dialogflow_payload_obj"/>

<!-- Now supply these hints when starting the dialog -->
<dialogstart src="dialogflow_service_url" type="'application/google-dialogflow'" connectionid="evt.connectionid" hints="dialogflow_hints"/> 
         

Using custom payloads to control BVR

It is possible to trigger certain actions on BVR by setting a custom payload response in your Dialogflow project. This response must be in JSON format and can be set in the "Responses" section of the intent in the Dialogflow console. Note that if multiple payloads are supplied, BVR will merge these internally and treat them as a single payload, so ensure that multiple payloads don't contain conflicting bw_* variables.

The custom payload variables

bw_action Trigger action within CCXML - either end dialog or a transfer
bw_transfer_target Transfer target to use if the bw_action is transfer
bw_bargein Sets bargein for this one prompt. After the prompts, bargein will be set back to the default set by the bargein parameter when the dialog started.
bw_no_input_timeout Sets no input timeout in milliseconds. After the prompts, the no input timeout will be set back to the default set by the no_input_timeout parameter when the dialog started
bw_max_speech_duration Sets the max speech duration in milliseconds. After the prompts, the max speech duration will be set back to the default set by the max_speech_duration parameter when the dialog started
bw_stt_confidence_threshold Sets the confidence threshold in milliseconds. After the prompts, the confidence threshold will be set back to the default set by the stt_confidence_threshold parameter when the dialog started. Note that this has no effect if Dialogflow's built-in STT is used, only if a separate STT call feature is used
bw_dtmf_config Sets whether BVR should perform DTMF detection for this set of prompts. After the prompts this will default back to off. This parameter is a JSON object of additional parameters explained below

The bw_action payload variable

end_dialog Ends the current Dialogflow dialog, triggering a CCXML dialog.exit transition
transfer Triggers the dialog.transfer CCXML transition and ends the dialog. It is possible to set a transfer target by setting the session variable "bw_transfer_target". Further information on transferring in CCXML can be found on the CCXML Call Transfer and Bridging page

Example of the custom payload format

            
{
   "bw_action": "end_dialog"
} 
         

DTMF detection and Dialogflow

To enable DTMF detection the bw_dtmf_config custom payload must be set to a JSON object with the following parameters. Once the set of prompts has finished the DTMF detection will be turned off. If any DTMFs are detected that match the conditions specified in the bw_dtmf_config this is reported to DialogFlow as a DTMF input.

inter_digit_timeout Time to wait between each DTMF digit before a result is considered. The default if not specified is 1 second
min_digits Minimum number of digits to be collected before a match is considered. The default if not specified is 1 digit
max_digits Maximum number of allowed digits before a no match is reported. The default if not specified is 16 digits
term_char Option termination character to stop DTMF detection. The default if not specified is no termination character
term_timeout Optional timeout waiting for a termination character. The default if not specified is 0 seconds

Example of enabling DTMF detection in a custom payload format


{
  "bw_dtmf_config": {
    "term_char": "#",
    "term_timeout": "1s",
    "max_digits": 4,
    "min_digits": 2,
    "inter_digit_timeout": "5s"
  },
  "bw_no_input_timeout": 60000
} 
         

Dialogflow Exit Values

When a Dialogflow dialog has finished, some values will be set to the "dialogflow" property on the dialog.exit event values evt.values.dialogflow. Within this object are the following values:

user_defined A map containing user defined values that were set in the last Dialogflow payload

Any variable names within these structures with a space character will have that space automatically replaced with an underscore '_' character

If the exit values contain any arrays, they will be presented as a map which contains a "count" property to determine the number of items in the array. Each item in the array will be a property of the map where the key is the index (starting from 0)

As an example, if the user_defined data from Dialogflow contains an array called "custom data" with 2 entries, there will be the following values:

evt.values.dialogflow.user_defined.custom_data.count The number of items in the array - in this case, the value for this will be 2
evt.values.dialogflow.user_defined.custom_data.0 The first item in the array
evt.values.dialogflow.user_defined.custom_data.1 The second item in the array

It is possible to loop through an array in CCXML. Here is an example of some Javascript code that can be added to a CCXML document in the <script> tag and invoked by calling printArray(arrayObject). This script relies on the existence of the printObject function

function printArray(toLog) {
   var output = "";

   if (toLog == null) {
      return "Cannot turn null object into an array";
   }
   var count = toLog.count;
   if (count == null) {
      return "This object is not an array";
   }
   //Total now has the last index value for the list of attributes in toPrint
   output += "Array has " + count + " items\n";
   var i;
   for (i = 0; i &lt; count; i++) {
      output += "\nItem " + i + ": " + toLog[i];
   }
   return output;
} 

Sample Applications

We have provided sample applications that describe the integration and can be used to connect to Google Dialogflow. The sample applications are installed on the BVR server in the following location:

/opt/blueworx/vr/sample/GoogleDialogflow/GoogleDialogflowSample.ccxml 

Debugging Google Dialogflow applications

Dialogflow applications have a single trace component: G_DIALOGFLOW