Marcus Guttenplan | Google Assistant Powered Ping Pong Scoreboard

back end dev, front end dev, creative technology

Google Assistant Powered Ping Pong Scoreboard

Building voice and UI experiences with Google Assistant

VUI

With Alexa and Assistant everywhere, designing Voice Interfaces is all the rage. For many people, using voice to control things is more intuitive than using a screen, and unlocking the capabilities of digital tools with messy hands in the kitchen presents interesting design challenges. After working with Google Assistant for Discount Tire, the power that comes from bridging voice control and screen content cannot be ignored.

My friend Tim Kim recently made an offhand joke about how nice it would be to have something to keep score of his ping pong tournaments at work, and it sounded too good to pass up.

The basics of building an Assistant app that works on Google Home, Android, and through the Assistant applications for iOS are:

An "Actions" project
Dialogflow conversation elements
A Google Cloud Function back end
A Firebase Project

Setup

First, create a Google Cloud Platform account, walk through the onboarding flow, and get access to some great products and $300 worth of credits. Once that's complete, create a new project, and give it a few seconds to complete. This project will contain all of the resources we'll need, and is a good logical way to group and track things (a lot hard on AWS).

Next, navigate to the Actions Console, and import the new project to the dashboard here. Once this is imported, choose any category to move on to the setup screen. Use the sidebar on the left to navigate.

Choose an "invokation" name that users can ask to invoke the app. For a ping pong scorekeeper, I'm going to choose "Table Keeper." Continue down the sidebar to "Actions," select "Get Started" and "Build a Custom Intent." This will direct you to login to the DialogFlow Console, where we will scaffold out out conversation.

"Intents" are specific elements of a conversation -- "What time is it?", "Where am I?", "Why am I here?" -- that a user may ask. Thinking in voice interfaces is difficult, and I struggle with this all the time.

For a ping pong game, the basic intents may be something like:

Stop game
Start game
Get Score
Add Point to Player X
Remove Point from Player X

Scaffolding

The basic idea is that this will consist of DialogFlow and Google Assistant, a Cloud Function to fulfill the conversation, and a front end HTML file to display scores. Firebase will handle the deployment of the function and front end, and the rest can be configured and built online.

The basic project structure will look like this:

.
├── functions
└── public

Install the firebase tools for deploying:

npm install -g firebase-tools

Init and select functions and hosting:

firebase init

Select Language
ESLint: N
Install dependencies: Y
Public dir: public
Configure as SPA: N

Edit firebase.json:

{
  "hosting": {
    "public": "public",
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ],
    "headers": [
      {
        "source": "**",
        "headers": [
          {
            "key": "Cache-Control",
            "value": "no-cache,no-store,must-revalidate"
          },
          {
            "key": "Access-Control-Allow-Origin",
            "value": "*"
          },
          {
            "key": "Access-Control-Expose-Headers",
            "value": "ETag"
          }
        ]
      }
    ]
  }
}

Install the software requirements:

cd functions && npm install --save firebase-functions actions-on-google dotenv firebase-admin

// functions/index.js
const functions = require('firebase-functions');
const {
  dialogflow,
  HtmlResponse
} = require('actions-on-google');

const firebaseConfig = JSON.parse(process.env.FIREBASE_CONFIG);

const app = dialogflow({
  debug: true
});

// TODO: Write your code here.

exports.fulfillment = functions.https.onRequest(app);

firebase deploy --project {PROJECT_ID}

Save the url for the deployed function. Mine looks like: https://us-central1-<project-id>.cloudfunctions.net/fulfillment

Conversation

On the DialogFlow Dashboard, start to add intents. The Default Welcome Intent and Default Fallback Intent will cover hellos and goodbyes.

To handle DialogFlow understanding what a "player" is, we need to create a new entity called Player: Screenshot of DialogFlow Entities

With a Player now defined, we need to create the intent for users to add points. Add a new intent, and add training phrases that users may say to add points: Screenshot of DialogFlow Intent

Ignore events, contexts, actions, and responses for now, and finish off by clicking Enable webhook fulfillment under Fulfillment.

Screenshot of DialogFlow Fulfillment

Set up Fulfillment: In the sidebar, select fulfillment, and use the URL for the cloud function (https://us-central1-<project-id>.cloudfunctions.net/fulfillment)

Update Cloud Function: Add the handlers for specific intents:

// Intents
app.intent('Add Point', (conv) => {
    console.log('Add Point')
    conv.ask("Adding Point!");
});

app.intent('Get Score', (conv) => {
    console.log('Get Score')
    conv.ask("Current Score is 0-0");
});

app.intent('New game', (conv) => {
    console.log('New Game')
    conv.ask("Starting a New Game");
});



// Fallbacks
app.catch((conv, error) => {
    conv.ask(`Oops! I'm having some issues. Can you please try again?`);
    console.log(error)
});

app.fallback((conv) => {
    conv.ask(`Oops! I missed that. Can you please try again?`);
});

Scaffolding the Fulfillment

We need to get and handle scores in the cloud function. Google Assistant can store data within the conv conversation context, using data like conv.data. We can access parameters from the user in the callback function.

Something like this:

app.intent('Add Point', (conv, params) => {
    console.log('Add Point')
    console.log(params)

    var player = params.Player

    if (player == 'Player 1') {
        conv.data.score.player_1 += 1
    } else if (player == 'Player 2') {
        conv.data.score.player_2 += 1
    }

    console.log("SCORE", conv.data.score)
    conv.ask(`Got it! Point for ${player}`);
});

app.intent('Get Score', (conv) => {
    console.log('Get Score')

    conv.ask(`Player 1 currently has ${conv.data.score.player_1}, Player 2 currently has ${conv.data.score.player_2}`);
});

app.intent('New game', (conv) => {
    console.log('New Game')

    conv.data.score = {
        player_1: 0,
        player_2: 0
    }

    conv.ask("Starting a New Game!");
});

The conv.data object looks like this:

conv.data.score = {
    player_1: 0,
    player_2: 0
}

This will work in the simulator, as long as you start a new game.

Scaffolding the Interactive Canvas

Build out the folder structure that will serve the HTML from Firebase. In the public directory, create a css and a js folder; and add a main.css and a main.js to the respective folders.

For the HTML, we just need some score cards:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <title>Rock, Paper, Scissors!</title>
    <link rel="shortcut icon" type="image/x-icon" href="data:image/x-icon;," />
    <link rel="stylesheet" href="css/main.css" />
    <script src="https://www.gstatic.com/assistant/interactivecanvas/api/interactive_canvas.min.js"></script>
  </head>
  <body>
    <div class="container">
      <div id="welcome">
          <div class="flip-card" id="player-1">
              0
          </div>
          <div class="flip-card" id="player-2">
              0
          </div>
      </div>
    </div>
    <script src="js/main.js"></script>
  </body>
</html>

And some basic CSS:

html {
  display: flex;
  height: 100%;
}

body {
  display: flex;
  flex: 1;
  margin: 0;
  background-color: white;
  flex-direction: column;
  justify-content: center;
  align-items: center;
}

div.container {
  width: 100%;
  text-align: center;
}

#welcome {
  display: flex;
}

.flip-card {
    flex: 1;
    display: flex;
    width: 50%;
    border: 3px solid #eee;
    font-size: 50vw;
    font-family: sans-serif;
    color: #eee;
    justify-content: center;
    align-items: center;
    height: 100vh;
}

#player-1 {
    background-color: #008efb;
}

#player-2 {
    background-color: #ff4e4e;
}

And the js that will listen for interactivecanvas events from Assistant:

'use strict';

interactiveCanvas.ready({
    onUpdate(data) {
        if (data.scene === 'score') {
            var player_1 = document.querySelector('#player-1');
            var player_2 = document.querySelector('#player-2');

            player_1.innerHTML = data.score.player_1;
            player_2.innerHTML = data.score.player_2
        }
    }
});

Assistant Interactive Canvas

Update the fulfillment cloud function, adding HtmlResponses that we will use to populate the screen:

app.intent('Add Point', (conv, params) => {
    console.log('Add Point')
    console.log(params)

    const score = conv.data.score

    var player = params.Player

    if (!conv.data.score) {
        conv.data.score = {
            player_1: 0,
            player_2: 0
        }
    }

    if (player == 'Player 1') {
        conv.data.score.player_1 += 1
    } else if (player == 'Player 2') {
        conv.data.score.player_2 += 1
    }

    console.log("SCORE", conv.data.score)

    conv.ask(`Got it! Point for ${player}`);
    conv.ask(new HtmlResponse({
    data: {
      scene: 'score',
      score: score
    }
  }));
});

So that the whole file will look like this:

// functions/index.js
const functions = require('firebase-functions');
const {
    dialogflow,
    HtmlResponse,
} = require('actions-on-google');

const firebaseConfig = JSON.parse(process.env.FIREBASE_CONFIG);

const app = dialogflow({
    debug: true
});

// Intents
app.intent('Default Welcome Intent', (conv) => {
    conv.ask("Howdy Howdy")
    conv.ask(new HtmlResponse({
        url: `https://${firebaseConfig.projectId}.firebaseapp.com/`
    }));
});

app.intent('Add Point', (conv, params) => {
    console.log('Add Point')
    console.log(params)

    const score = conv.data.score

    var player = params.Player

    if (!conv.data.score) {
        conv.data.score = {
            player_1: 0,
            player_2: 0
        }
    }

    if (player == 'Player 1') {
        conv.data.score.player_1 += 1
    } else if (player == 'Player 2') {
        conv.data.score.player_2 += 1
    }

    console.log("SCORE", conv.data.score)

    conv.ask(`Got it! Point for ${player}`);
    conv.ask(new HtmlResponse({
    data: {
      scene: 'score',
      score: score
    }
  }));
});

app.intent('Get Score', (conv) => {
    console.log('Get Score')

    conv.ask(`Player 1 currently has ${conv.data.score.player_1}, Player 2 currently has ${conv.data.score.player_2}`);

    conv.ask(new HtmlResponse({
        url: `https://${firebaseConfig.projectId}.firebaseapp.com/`
    }));
});

app.intent('New game', (conv) => {
    console.log('New Game')

    conv.data.score = {
        player_1: 0,
        player_2: 0
    }

    conv.ask("Starting a New Game!");
    conv.ask(new HtmlResponse({
        url: `https://${firebaseConfig.projectId}.firebaseapp.com/`
    }));
});


// Fallbacks
app.catch((conv, error) => {
    conv.ask(`Oops! I'm having some issues. Can you please try again?`);
    conv.ask(new HtmlResponse({
        url: `https://${firebaseConfig.projectId}.firebaseapp.com/`
    }));
    console.log(error)
});

app.fallback((conv) => {
    conv.ask(`Oops! I missed that. Can you please try again?`);
    conv.ask(new HtmlResponse({
        url: `https://${firebaseConfig.projectId}.firebaseapp.com/`
    }));
});

exports.fulfillment = functions.https.onRequest(app);