Overview

ClientV2 contains folowing features:

  1. Chat window - output.text
  2. Timeout - output.timeout and output.timeout_text
  3. Sound - output.sound and output.sound_onFinished
  4. Text-to-speech - output.tts and output.tts_onFinished
  5. Talking head - output.talking_head and output.talking_head_onFinished
  6. Paper head
  7. GUI - output.graphics and output.url
  8. Buttons - response options / suggestions / generic options

Chat window

Default output method. Displays contents of output.text. Arbitrary html is inserted into the page. This allows simple text formatting by common html means.

Buttons

This example will output a text prompt with two buttons like this:

Buttons in chat window

Clicking a button in this example simply outputs the button's content. Make sure to have proper intents and dialog nodes in place to handle buttons' messages.

Simple response:

<p>Choose one of the following options or type an arbitrary response:</p><button onclick='converse(\$(this).text()); \$(this).parent().find("button").prop("disabled", true);'>I choose the first option.</button> <button onclick='converse(\$(this).text()); \$(this).parent().find("button").prop("disabled", true);'>I choose the second option.</button>

Advenced response:

"output": {
    "text": "<p>Choose one of the following options or type an arbitrary response:</p><button onclick='converse(\\$(this).text()); \\$(this).parent().find(\"button\").prop(\"disabled\", true);'>I choose the first option.</button> <button onclick='converse(\\$(this).text()); \\$(this).parent().find(\"button\").prop(\"disabled\", true);'>I choose the second option.</button>"
}

Timeout

In every response, one can define a message to be shown to the user, if he does not respond during specified period of time. We call it the timeout message.

"output": {
    "timeout": number,     // [miliseconds] of timeout
    "timeout_text": string // input to send to the conversation service, if user does not react before timeout period
},

Sound

Plays the sound file provided. It is possible to hook action at the end of playback.

"output": {
    "sound": string // URL to file
    "sound_onFinished"?: {"output"?:{"text"?:string, "sound"?:string, ...}, "input"?:string, "intents"?:Array<string>, "entities"?:Array<string>}
}

The sound_onFinished object can contain following fields:

Text to speech

Usage:

"output": {
    // message to be said out loud. Can contain xml commands.
    "tts": string,
    // optional action to perform right after the is speech finished
    "tts_onFinished"?: {"output"?:{"text"?:string, "sound"?:string, ...}, "input"?:string, "intents"?:Array<string>, "entities"?:Array<string>}
    "voice": string
},

tts_onFinished event

The action can contain following fields:

voice

Voice field can override prefered_tts configuration for single tts output if the selected voice belongs to another tts service. Available values are:

For more up-to date list of bluemix TTS voices refer to it's documentation.

Configuration

{
    "prefered_tts": "talking_head" | "bluemix_tts" | "none",
    "tts_interruption_reaction": "replace" | "queue", // default: "replace"
    "bluemix_tts":{
        "username": string,
        "password": string,
        "voice": string
    }
}

Prefered TTS

There can be more TTS engines available at once, (talking head)[#talking-head] and bluemix TTS service. When one of them is not available the other one will take it's place automaticaly. Default priority order is:

  1. talking_head
  2. bluemix_tts none can be used to forbid any speech synthesis.

Interruption policy

Determines what to do, when user acts before syntesized speech is finished speaking. replace - cut off current speech and immediately start reading new response queue - finish speaking, then follow with new response

Bluemix TTS

Bluemix speech synthesis service integration. Usage:

"output": {
    // message to be said out loud. This field is used for speech generation if present.
    "bluemix_tts": string,
    // alternatively takes general TTS output field.
    "tts": string,
    // bluemix service specific speech completition event, takes the same value as tts_onFinished
    "bluemix_tts_onFinished"?: {"output"?:{"text"?:string, "sound"?:string, ...}, "input"?:string, "intents"?:Array<string>, "entities"?:Array<string>}
}

if neither bluemix_tts nor tts output field is present system strips markup from talking_head field. If not even talking_head field is present output.text is used.

Talking head

Usage:

"output": {
    "talking_head": string     // message to be said out loud. Can contain xml commands.
},

Talking head.is a standalone application rendering 3D model of the human head. The talking head is capable of talking (local TTS) with lips synchronized. It is able to express emotions and gestures. It is controlled remotely over TCP socket, using a simple xml syntax commands. It can be connected to clientv2 and controlled from dialog. Talking head should be running before starting clientv2 and connection to it is created at start and maintained through execution of clientv2. If disconnected for any reason, clientv2 has to be restarted.

Browser side configuration in config tab

There are no talking head specific configuration keys, but the following two general tts settings affect talking head's behavior:

{
    "prefered_tts": "talking_head" | "bluemix_tts" | "none",
    "tts_interruption_reaction": "replace" | "queue"
}

See text to speech section for more detail.

Configuration in config.json:

"talking_head": {
    "enable":"true",
    "use_output_text":"true",
    "uri":"localhost",
    "port":"7222"
}

Taking head needs to be enabled and its uri and port set. Talking head can run on a different machine then clientv2, but it must be accessible from clientv2. Typically, a dedicated (local) copy of clientv2 is run along talking head. If use_output_text is true, the talking head renderes outut.text messages, but only if output.talking_head is not provided. We expect three modes of use:

Update policy

ClientV2 simulates update policies for output.talking_head the way they work for output.text.

onFinished event

The talking_head_onFinished field specifies an action to be performed when the head ends it's speech.

"output": {
    "talking_head": string
    "talking_head_onFinished"?: {"output"?:{"text"?:string, "sound"?:string, ...}, "input"?:string, "intents"?:Array<string>, "entities"?:Array<string>}
}

The action can contain following fields:

Paper head

Paper head is a simple demonstration platform of the hardware actuators connected to the dialog runtime using clientv2. It is able to turn the head (stepper motor), lit the eyes and mouth (on/off pins connected to LEDs) and move the head up and down by a servo. The paper head is controlled by Arduino micro running firmanta and connected to clientv2 using Johnny Five library.

Paper head is completely operated through service requests. See service requests for more detail.

GUI

ClientV2 provides channels for displaying graphics from runtime dialog nodes.

Default graphics

When $defaultGraphics context variable is set, it's content is presented the same way output.graphics is with every response, which doesn't specify, what should be presented to the user.

Example

Dialog node

{
	"go_to": null,
	"output": {
		"text": "<? $weather[$location][$day].narrative ?>",
		"graphics": "<iframe height='245' src='http://forecast.io/embed/\\#lat=<?$weather[$location][$day].latitude?>&lon=<?$weather[$location][$day].longitude?>&name=<?$location?>&units=si&color=\\#325c80' />"
	},
	"conditions": "$weather != null && $weather[$location] != null && $weather[$location][$day] != null",
	"dialog_node": "node_8_1467637658358",
	"parent": null,
	"previous_sibling": null
}

Result

weather example