Install Google Assistant In Gnome Shell

Updated on 19th June 2019. Note the latest Assistant SDK renders the Gnome Shell extension broken. Google Assistant is still able to run from starting via the terminal.

Cloud-based voice recognition assistants have grown massively over the past few years with Apples Siri, Amazons Alexa, Microsoft’s Cortana and the open source Mycroft pushing the expansion. Voice Assistants, however, do not seem to have easy access towards the Linux desktop but are possible using software development kits released by the producers.

In this tutorial, we will show you how to set up Google Assistant on ubuntu 18.04 and control it with a Gnome shell extension.

What you will need;

  • A PC with Ubuntu 18.04+ installed and Gnome Shell Extensions Installed (Details Below)
  • A Google Account
  • An Android Phone to allow personal responses
  • A working microphone.
  • A working speaker system.

Many people, me included have struggled with getting the Gnome shell extension to work. This is down to conflicting advice from both Google and the developer. Not intentionally but it’s all down to where the python modules are placed. Google recommends a virtual environment (Sensible) but the Extension can’t seem to interact with it. Google also gives information about installing for python 2 & 3, which can get a bit confusing but anyway:

The complete steps to install:

Check that ALSA is configured correctly to play and record

First, we need to check the speakers are configured correctly. On a fresh install or unless you have some custom ALSA configuration. This should just work as intended. To test the speakers open a terminal and enter:

speaker-test -t wav

you can cancel the test with ctrl+c. If all is successful you should hear the speakers identifying both the front left and front right speakers.

Test recording

Now we need to record a short audio clip with some talking to playback by entering:

arecord --format=S16_LE --duration=5 --rate=16000 --file-type=raw out.raw

You can play the audio clip recorded and check it’s working by entering:

aplay --format=S16_LE --rate=16000 out.raw

If sound levels need changing this can be done with ALSA mixer

alsamixer

Configure a developer account:

Go to https://console.actions.google.com/, and log in to your google account.

Click on Add/Import Project

ss1

Enter a name for your project. i.e. linuxassistant, gsassistant etc

Choose your country from the list underneath.

On the next page click on device registration near the bottom of the page.

click on the gear icon and select project settings.

make a note of your project ID, this is needed later for testing.

open a new browser window and go to https://console.developers.google.com/apis/api/embeddedassistant.googleapis.com/overview

ss5

Search for google assistant API select and click Enable

ss6

We now need to create the Oauth credentials page. Click on the credentials option on the left-hand menu and then the configure consent screen button on the right-hand side.

ss7

Now enter your email address and a product name and click save.

ss8

Now open a new window or tab go to https://myaccount.google.com/activitycontrols , and toggle all switches to on.

Switch Back to the actions console https://console.actions.google.com/ and click on Device Registration.

ss9

Click on Register Model and enter a product name and manufacturer name. Google also requires you to enter a device type so select speaker. Note down the Device Model Id as this is needed for testing. Clicking Register Model will enable you to download a clients_secrets.json file to register your PC with your account. On the third option to specify traits, you can press skip as none are needed.

ss10

Optional

If you ever need to download the client_secret.json again simply go back to Register Device, select the entry and click on the 3 dots menu to give an option to download the Oauth2 file.

ss11

Install the Google Assistant SDK

open a terminal and install the dependencies:

sudo apt-get install python3-dev python3-pip
sudo apt-get install portaudio19-dev libffi-dev libssl-dev libmpg123-dev python3-openssl

Now we need to get the Google SDK python libraries:

pip3 install --user google-assistant-library
pip3 install --user google-assistant-sdk[samples]
pip3 install --user google-auth-oauthlib[tool]

Authorise our Google Assistant

In a terminal enter the following, be careful though as it’s very dependant on how it’s formatted.

./.local/bin/google-oauthlib-tool --scope https://www.googleapis.com/auth/assistant-sdk-prototype \
      --scope https://www.googleapis.com/auth/gcm \
      --save --headless --client-secrets /path/to/client_secret_client-id.json

Make sure you give the full path to the client_secret.json (Quick tip, select the file in a file manager, right click to select rename and copy and paste then add .json to the end, this helps against typos!)

ss12

Right click on the link produced in the terminal and a new browser window will open, log in to your google account and copy and paste the authorisation code back into the terminal.

Testing the Assistant

Run the following code with the Project ID and Device ID saved from setting up a Developer account:

./.local/bin/google-assistant-demo --credentials /home/$USER/.config/google-oauthlib-tool/credentials.json --device_model_id my_device_model --project_id my_project_id 

when loaded say “ok google” and ask a question, if Google recognises the speech it will give an appropriate response.

You may need to turn up the microphone and speaker volumes using the sliders in the Gnome Shell user menu, and the alsa mixer mentioned earlier. If speech responses from Google are still quiet, say “OK Google set the volume to 10” to boost the internal assistant volume level. If all is successful then the device will start and register as shown below.

Close the terminal window (or ctrl+c) to end the assistant when done.

Installing the Gnome Shell Extension (NOT CURRENTLY WORKING – Consider adding the google-assistant demo command above to a startup script. See below)

Please see our quick tip about enabling gnome shell extensions, if not already enabled on your system.

Go to https://extensions.gnome.org/extension/1441/gassistant/ in your browser and toggle the on/off slider to enable the extension. Click on Install and wait for it to load onto the Gnome Shell panel. Right click and select ‘Restart Google Assistant’. If the ZZZ icon is showing it means the extension has found the Assistant installation and is waiting to be triggered with the hot word (OK Google).

screenshot_1441_Xv2ZfMq

The gnome extension will give the option to enable, restart or turn off the service (for those paranoid that Google is always listening) it will also give an audible beep when google assistant is invoked to show it is ready for voice input.

Make the Assistant automatically start (alternative to the gnome-shell extension)

Open a text editor and enter the following bash script making sure you change it to your assistant details.

#!/bin/bash

./.local/bin/google-assistant-demo --credentials /home/$USER/.config/google-oauthlib-tool/credentials.json --device_model_id my_device_model --project_id my_project_id

Save the file as googleassistant.sh in your home directory. Then set the file as executable either with right clicking properties>permissions in the file manager or the following terminal command:

chmod u+x googleassistant.sh

Now add it to the list of startup applications by finding the startup applications app in your app launcher. Open and select the add button and fill out the Name and location to the googleassistant.file

Google Assistant should now automatically start whenever the PC is turned on and logged in. To stop the assistant open a terminal and enter

killall google-assistant-demo

Personalise with Android

You can control some features of the Assistant through an android phone (connected to the same account)
go to settings>google> search> Settings the device should now show up in the Devices list, enabling the options gives it access to personal accounts. It’s also possible to give it a location for things like weather, travel updates and to set up news providers.

Expanding your Assistant

Google assistant SDK gives access to nearly all the features of Google assistant on Android, but most notably missing is access to music services. It is possible by setting up local servers solutions exposed to the internet and constructing actions to expand on the skills available through the SDK, but it’s not a simple method to achieve. It is possible to control a Linux PC using skills such as the TriggerCMD service.

Do you use voice assistants? Do you have some of them controlling some amazing devices or functions if so we would love to hear about them in the comments?

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

%d bloggers like this: