• re:Invent 2018 Session/Slides
• Introduction
• Getting Started
  • Prerequisites
  • Back End Setup
  • Interacting with Chatbots
  • Interacting with other AWS AI Services
• Building, Deploying and Publishing with the Amplify CLI
• Back End Setup, Back End and Front End Building, Deploying and Publishing with the Amplify Console
• Clean Up

This is a Starter React Progressive Web Application (PWA) that uses AWS AppSync to implement offline and real-time capabilities in a chat application with AI/ML features such as image recognition, text-to-speech, language translation, sentiment analysis as well as conversational chatbots developed as part of the re:Invent session Bridging the Gap Between Real Time/Offline and AI/ML Capabilities in Modern Serverless Apps. In the chat app, users can search for users and messages, have conversations with other users, upload images and exchange messages. The application demonstrates GraphQL Mutations, Queries and Subscriptions with AWS AppSync integrating with other AWS Services:

• Amazon Cognito for user management as well as AuthN/Z
• Amazon DynamoDB with 4x NoSQL Data Sources (Users, Messages, Conversations, ConvoLink)
• Amazon Elasticsearch Data Source for full text search on messages and users
• AWS Lambda as a Serverless Data Source connecting to AI Services
• Amazon Comprehend for sentiment and entity analysis as well as language detection
• Amazon Rekognition for object, scene and celebrity detection on images
• Amazon Lex for conversational chatbots
• Amazon Polly for text-to-speech on messages
• Amazon Translate for language translation
• Amazon S3 for Media Storage

You can use this for learning purposes or adapt either the application or the GraphQL Schema to meet your needs.

• AWS Account with appropriate permissions to create the related resources
• NodeJS with NPM
• AWS CLI with output configured as JSON (pip install awscli --upgrade --user)
• AWS Amplify CLI configured for a region where AWS AppSync and all other services in use are available (npm install -g @aws-amplify/cli)
• AWS SAM CLI (pip install --user aws-sam-cli)
• Create React App (npm install -g create-react-app)
• Install JQ
• If using Windows, you'll need the Windows Subsystem for Linux (WSL)

Note: This solution uses Amazon Lex. The service is only supported in us-east-1, us-west-2 and eu-west-1. We recommending launching this entire solution in one of these regions.

1. First, clone this repository and navigate to the created folder:

   git clone https://github.com/aws-samples/aws-appsync-chat-starter-react.git
   cd aws-appsync-chat-starter-react

2. Install the required modules:

   npm install

3. Init the directory as an amplify Javascript app using the React framework:

   amplify init

   Set the region we are deploying resources to:

   export AWS_REGION=$(jq -r '.providers.awscloudformation.Region' amplify/#current-cloud-backend/amplify-meta.json)
   echo $AWS_REGION

   Make sure ALL services are supported in this region or else you'll get errors in the next steps.

4. Add an Amazon Cognito User Pool auth resource. Use the default configuration.

   amplify add auth

5. Add an AppSync GraphQL API with Amazon Cognito User Pool for the API Authentication. Follow the default options. When prompted with "Do you have an annotated GraphQL schema?", select "YES" and provide the schema file path backend/schema.graphql

   amplify add api

6. Add S3 Private Storage for Content to the project with the default options. Select private read/write access for Auth users only:

   amplify add storage

7. Now it's time to provision your cloud resources based on the local setup and configured features. When asked to generate code, answer "NO" as it would overwrite the current custom files in the src/graphql folder.

   amplify push

   Wait for the provisioning to complete. Once done, a src/aws-exports.js file with the resources information is created.

At this point you have an usable serverless chat application with no AI features. The next steps are only needed to deploy and configure the integration with services that provide image recognition, text-to-speech, language translation, sentiment analysis as well as conversational chatbots. From here you can skip to step 13 if there's no interest to setup the AI integration.

8. Look up the S3 bucket name created for user storage:

   export USER_FILES_BUCKET=$(sed -n 's/.*"aws_user_files_s3_bucket": "\(.*\)".*/\1/p' src/aws-exports.js)
   echo $USER_FILES_BUCKET

9. Retrieve the API ID of your AppSync GraphQL endpoint

   export GRAPHQL_API_ID=$(jq -r '.api[(.api | keys)[0]].output.GraphQLAPIIdOutput' ./amplify/#current-cloud-backend/amplify-meta.json)
   echo $GRAPHQL_API_ID

10. Retrieve the project's deployment bucket and stackname . It will be used for packaging and deployment with SAM

    export DEPLOYMENT_BUCKET_NAME=$(jq -r '.providers.awscloudformation.DeploymentBucketName' ./amplify/#current-cloud-backend/amplify-meta.json)
    export STACK_NAME=$(jq -r '.providers.awscloudformation.StackName' ./amplify/#current-cloud-backend/amplify-meta.json)
    echo $DEPLOYMENT_BUCKET_NAME
    echo $STACK_NAME

11. Now we need to deploy 3 Lambda functions (one for AppSync and two for Lex) and configure the AppSync Resolvers to use Lambda accordingly. First, we install the npm dependencies for each lambda function. We then package and deploy the changes with SAM.

    cd ./backend/chuckbot-lambda; npm install; cd ../..
    cd ./backend/moviebot-lambda; npm install; cd ../..
    sam package --template-file ./backend/deploy.yaml --s3-bucket $DEPLOYMENT_BUCKET_NAME --output-template-file packaged.yaml
    export STACK_NAME_AIML="$STACK_NAME-extra-aiml"
    sam deploy --template-file ./packaged.yaml --stack-name $STACK_NAME_AIML --capabilities CAPABILITY_IAM --parameter-overrides appSyncAPI=$GRAPHQL_API_ID s3Bucket=$USER_FILES_BUCKET --region $AWS_REGION

    Wait for the stack to finish deploying then retrieve the functions' ARN

    export CHUCKBOT_FUNCTION_ARN=$(aws cloudformation describe-stacks --stack-name  $STACK_NAME_AIML --query "Stacks[0].Outputs" --region $AWS_REGION --output json | jq -r '.[] | select(.OutputKey == "ChuckBotFunction") | .OutputValue')
    export MOVIEBOT_FUNCTION_ARN=$(aws cloudformation describe-stacks --stack-name  $STACK_NAME_AIML --query "Stacks[0].Outputs" --region $AWS_REGION --output json | jq -r '.[] | select(.OutputKey == "MovieBotFunction") | .OutputValue')
    echo $CHUCKBOT_FUNCTION_ARN
    echo $MOVIEBOT_FUNCTION_ARN

12. Let's set up Lex. We will create 2 chatbots: ChuckBot and MovieBot. Execute the following commands to add permissions so Lex can invoke the chatbot related functions:

    aws lambda add-permission --statement-id Lex --function-name $CHUCKBOT_FUNCTION_ARN --action lambda:\* --principal lex.amazonaws.com --region $AWS_REGION
    aws lambda add-permission --statement-id Lex --function-name $MOVIEBOT_FUNCTION_ARN --action lambda:\* --principal lex.amazonaws.com --region $AWS_REGION

    Update the bots intents with the Lambda ARN:

    jq '.fulfillmentActivity.codeHook.uri = $arn' --arg arn $CHUCKBOT_FUNCTION_ARN backend/ChuckBot/intent.json -M > tmp.txt ; cp tmp.txt backend/ChuckBot/intent.json; rm tmp.txt
    jq '.fulfillmentActivity.codeHook.uri = $arn' --arg arn $MOVIEBOT_FUNCTION_ARN backend/MovieBot/intent.json -M > tmp.txt ; cp tmp.txt backend/MovieBot/intent.json; rm tmp.txt

    And, deploy the slot types, intents and bots:

    aws lex-models put-slot-type --cli-input-json file://backend/ChuckBot/slot-type.json --region $AWS_REGION
    aws lex-models put-intent --cli-input-json file://backend/ChuckBot/intent.json --region $AWS_REGION
    aws lex-models put-bot --cli-input-json file://backend/ChuckBot/bot.json --region $AWS_REGION
    aws lex-models put-slot-type --cli-input-json file://backend/MovieBot/slot-type.json --region $AWS_REGION
    aws lex-models put-intent --cli-input-json file://backend/MovieBot/intent.json --region $AWS_REGION
    aws lex-models put-bot --cli-input-json file://backend/MovieBot/bot.json --region $AWS_REGION

13. Finally, execute the following command to install your project package dependencies and run the application locally:

    amplify serve

14. Access your ChatQLv2 app at http://localhost:3000. Sign up at least 2 different users, authenticate with each user to get them registered in the backend Users table, then search for new users to start a conversation and test real-time/offline messaging as well as other features using different devices or browsers.

The chatbots retrieve information online via API calls from Lambda to The Movie Database (TMDb) (MovieBot, which is based on this chatbot sample) and chucknorris.io (ChuckBot)

1. In order to initiate or respond to a chatbot conversation, you need to start the message with either @chuckbot or @moviebot to trigger or respond to the specific bot, for example:

   • @chuckbot Give me a Chuck Norris fact
   • @moviebot Tell me about a movie

2. Each subsequent response needs to start with the bot handle (@chuckbot or @moviebot) so the app can detect the message is directed to Lex and not to the other user in the same conversation. Both users will be able to view Lex chatbot responses in real-time powered by GraphQL subscriptions.

3. Alternatively you can start a chatbot conversation from the message drop-down menu:

   • Just selecting ChuckBot will display options for further interaction
   • Send a message with a nothing but a movie name and selecting MovieBot subsequently will retrieve the details about the movie

1. Click or select uploaded images to trigger Amazon Rekognition object, scene and celebrity detection.
2. From the drop-down menu, select LISTEN -> TEXT TO SPEECH to trigger Amazon Polly and listen to messages in different voices based on the message automatically detected source language (supported languages: English, Mandarin, Portuguese, French and Spanish).
3. To perform entity and sentiment analysis on messages via Amazon Comprehend, select ANALYZE -> SENTIMENT from the drop-down menu.
4. To translate the message select the desired language under TRANSLATE in the drop-down menu (supported languages: English, Mandarin, Portuguese, French and Spanish). In the translation pane, click on the microphone icon to listen to the translated message.

1. Execute amplify add hosting from the project's root folder and follow the prompts to create an S3 bucket (DEV) and/or a CloudFront distribution (PROD).

2. Build, deploy, upload and publish the application with a single command:

   amplify publish

3. If you are deploying a CloudFront distribution, be mindful it needs to be replicated across all points of presence globally and it might take up to 15 minutes to do so.

4. Access your public ChatQL application using the S3 Website Endpoint URL or the CloudFront URL returned by the amplify publish command. Share the link with friends, sign up some users, and start creating conversations, uploading images, translating, executing text-to-speech in different languages, performing sentiment analysis and exchanging messages. Be mindful PWAs require SSL, in order to test PWA functionality access the CloudFront URL (HTTPS) from a mobile device and add the site to the mobile home screen.

(More info here)

1. Fork this repository into your own GitHub account and clone it
2. Repeat Steps 3 to 6 from the Back End Setup in the previous section. Do not perform step 7 (amplify push).
3. Commit the changes to your forked repository. A new folder amplify will be commited with the project details.
4. Connect your repository to the Amplify Console as per the instructions here, making sure the name of the branch in your repository matches the name of the environment configured on amplify init (i.e. master). When prompted with "We detected a backend created with the Amplify Framework. Would you like Amplify Console to deploy these resources with your frontend?", select "YES" and provide or create an IAM role with appropriate permissions to build the backend resources
5. Wait for the build, deployment and verification steps

At this point you have an usable serverless chat application with no AI features. The next steps are only needed to deploy and configure the integration with services that provide image recognition, text-to-speech, language translation, sentiment analysis as well as conversational chatbots. From here you can skip to step 8 if there's no interest to setup the AI integration.

7. Now perform steps 7 to 12 from the Back End Setup
8. Access your app from the hosted site generated by the Amplify Console(https://master.xxxxxxxx.amplifyapp.com)

To clean up the project, you can simply delete the bots, delete the stack created by the SAM CLI:

aws lex-models delete-bot --name `jq -r .name backend/ChuckBot/bot.json` --region $AWS_REGION
aws lex-models delete-bot --name `jq -r .name backend/MovieBot/bot.json` --region $AWS_REGION
aws lex-models delete-intent --name `jq -r .name backend/ChuckBot/intent.json` --region $AWS_REGION
aws lex-models delete-intent --name `jq -r .name backend/MovieBot/intent.json` --region $AWS_REGION
aws lex-models delete-slot-type --name `jq -r .name backend/ChuckBot/slot-type.json` --region $AWS_REGION
aws lex-models delete-slot-type --name `jq -r .name backend/MovieBot/slot-type.json` --region $AWS_REGION

aws cloudformation delete-stack --stack-name $STACK_NAME_AIML --region $AWS_REGION

and use:

amplify delete

to delete the resources created by the Amplify CLI.