Creating Assistants with DeepPavlov Dream. Part 1. Composing Distribution from Existing Components

2023-07-17 12:39 Products Updates Community Tutorials

Our goal at DeepPavlov is to enable anyone to build their very own assistant. We offer a range of components that can be easily utilized as they are or customized according to your needs to build a personalized assistant. In this tutorial, we will guide you through the process of creating a simple and light-weight bot that specializes in discussing movies and answering factoid questions. Our approach will only utilize the existing Dream components, without introducing any new ones.

Dream architecture

Let’s take a closer look at the parts that make up Dream before we start developing our bot. As you can see in the figure below, our pipeline architecture consists of the following components :

Annotators perform NLU preprocessing on an utterance;
Skill Selector chooses the relevant skills that can provide the next bot’s response;
Skills are in charge of generating candidate responses;
Candidate Annotators perform NLU postprocessing on the candidate responses;
Response Selector selects the most suitable response from the generated candidates;
Response Annotators carry out post-processing of the bot’s response;
Dialogue State stores all the information about the current dialog including all annotations and meta-data.

What do we need?

Now that we are familiar with the Dream architecture, let’s choose the components we need for our future bot. The complete list of available components can be found here. For our bot, we will need the following components:

Annotators:

Sentseg – allows us to handle long and complex user’s utterances by splitting them into sentences and recovering punctuation if it is missing;
Combined Classification light-weight — annotator for topic classification, dialog acts classification, sentiment, toxicity, emotion, factoid classification;
Entity Detection — extracts entities and their types from utterances;
Entity Linking — finds Wikidata Knowledge Graph’s entity identifiers for the entities detected with Entity Detection;
Wiki Parser — extracts Wikidata triplets for the entities detected with Entity Linking.

Skills:

Movie Skill — conducts scripted conversations related to movies;
Factoid QA — answers factoid questions using Knowledge Graph’s annotations;
Program Y Skill — an AIML-based skill that responds to some common questions and requests using pre-written answers;
Dummy Skill — a skill with multiple non-toxic candidate fallback responses in case other skills fail to provide a response.

Candidate Annotators:

Combined Classification light-weight — annotator for topic classification, dialog acts classification, sentiment, toxicity, emotion, factoid classification.

Let’s create our bot!

Install DeepPavlov Dreamtools:

pip install git+https://github.com/deeppavlov/deeppavlov_dreamtools.git

Clone Dream repository:

git clone https://github.com/deeppavlov/dream.git

Go to cloned repository:

cd dream

Create your own distribution with above discussed components: dreamtools new dist movies_and_qa \

dreamtools new dist movies_and_qa --display-name "New Movies and QA Assistant" 
--author deepypavlova@email.org 
--description "Distribution for discussing movies and answering questions" 
--overwrite 
--annotators components/1Q9QXih1U2zhCpVm9zxdsA.yml 
--annotators components/dflkgjdlkh342r9ndf.yml 
--annotators components/tgzaSQggV7wgMprOmF1Ww.yml 
--annotators components/M1sE6hOm20EGBWBdr0vIOw.yml 
--annotators components/O4FVnkAwjay1mL1FbuRGWw.yml 
--skills components/4yA8wZWOEnafRfz6Po9nvA.yml 
--skills components/qx0j5QHAzog0b39nRnuA.yml 
--skills components/ckUclxqUplyzwmnYyixEw.yml 
--skills components/uYkoK0vRp4bbIg9akI1yw.yml 
--response-annotators components/dflkgjdlkh342r9ndf.yml 
--response-annotators components/05PqJXVd7gV7DqslN5z3A.yml 
--last-chance-service components/70NLr5qqOow5.yml 
--timeout-service components/x8rLTpIWct4P.yml 
--response-annotator-selectors components/LXrJDIf43gwNmPMNXG5Eg.yml 
--skill-selectors components/xSwFvtAUdvtQosvzpb7oMg.yml 
--response-selectors components/KX4drAocVa5APcivWHeBNQ.yml

Go to asisstant_dists/movies_and_qa and create proxy.yml, and paste the following code:

services:

 sentseg:
   command: ["nginx", "-g", "daemon off;"]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8011
     - PORT=8011
  entity-detection:
   command: [ "nginx", "-g", "daemon off;" ]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8103
     - PORT=8103

 entity-linking:
   command: [ "nginx", "-g", "daemon off;" ]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8075
     - PORT=8075

 wiki-parser:
   command: [ "nginx", "-g", "daemon off;" ]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8077
     - PORT=8077

 dff-movie-skill:
   command: ["nginx", "-g", "daemon off;"]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8023
     - PORT=8023

 factoid-qa:
   command: ["nginx", "-g", "daemon off;"]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8071
     - PORT=8071

 dff-program-y-skill:
   command: ["nginx", "-g", "daemon off;"]
   build:
     context: dp/proxy/
     dockerfile: Dockerfile
   environment:
     - PROXY_PASS=proxy.deeppavlov.ai:8008
     - PORT=8008

version: '3.7'

Build your distribution:

docker-compose -f docker-compose.yml -f 
assistant_dists/movies_and_qa/docker-compose.override.yml -f 
assistant_dists/movies_and_qa/proxy.yml up --build

*Please note that in the command, we also utilize the assistant_dists/movies_and_qa/proxy.yml configuration. This configuration enables you to conserve your local resources by employing proxied copies of certain services hosted by DeepPavlov.

In a separate terminal tab, run:

docker-compose exec agent python -m deeppavlov_agent.run 
agent.debug=false agent.channel=cmd 
agent.pipeline_config=assistant_dists/movies_and_qa/pipeline_conf.json

Enter your username and have a chat with Dream!

Useful links

Kseniia Petukhova