How to Add Language Translation and Text to Speech Audio Using AWS Amplify

Outside of tech and software development, a passion of mine for the past five years has been learning French. I have used many different tools along my language journey. From Rosetta Stone to Duolingo and then over to Memrise. Each tool was another step along my journey to becoming fluent.

But, each of these tools only really helped me improve my vocabulary and memory. Both are critical to learning a language. But as I progressed and spoke more French at home I learned that these tools weren’t focusing on what I needed.

As I have progressed in my French, I have found that the piece I need the practice in is verb conjugations. Learning the different verbs and memorizing tense construction. Tools like Duolingo and Memrise largely focus on vocabulary. They spend very little time on when to use each tense of a verb or how that tense gets structured.

So, being the developer that I am, I decided to make a tool to help me with my verb conjugations in their various tenses. If you want to jump ahead and see the app, you can check out my french verb conjugator.

Combining two passions

It’s always fun when you have many passions and they can converge on one project. Learning French is my biggest non-technical passion. Teaching folks how to leverage Amazon Web Services to build interesting ideas within AWS is my biggest technical passion.

So it’s only natural that I built French Verb Practice entirely serverless within AWS.

For this project, I wanted to take AWS Amplify through its paces. Why? You might ask. The answer is rather simple, Amplify is a framework within the AWS ecosystem that I wanted to get familiar with. I believe that one of the best ways to learn AWS is to roll up your sleeves and start building something with it. In fact, I created an entire learn AWS by using it course that does exactly that.

That said, this post isn’t going to run through how I created the entire project using AWS Amplify. There are many other posts out there that cover getting started with Amplify. But, I will touch on the basic concept behind Amplify. But I want to focus on how I used it to add two key features, language translation, and text to speech, to my language learning app.

French Verb Practice Introduction

The core concept for French Verb Practice is to provide a place to practice the various conjugations of french verbs. As such, there are only two components in the entire React application. There is the Start component which is the landing page you see up above. Then there is the Verb component where you actually practice a conjugation, see below.

The idea is that a user comes here, they see the verb exprimer and they need to conjugate it in the first person present tense. When they have it worked out, they enter exprime in the input and get the correct answer.

But as a fellow language learner, I always like to see the translation of a verb as well as hear it spoken. That is where the power and simplicity of AWS Amplify shines.

Leveraging AWS Amplify

I’m not going to cover everything in terms of how to get set up with AWS Amplify. But, the basic concept is that Amplify gives you an interface for deploying and managing AWS resources. It abstracts away the need to know all the details of the underlying services.

As an example, when I wanted to add a GraphQL API to my project it was a few simple steps on the command line.

$ amplify api add
? Please select from one of the below mentioned services: GraphQL
? Provide API name: kyle
? Choose the default authorization type for the API API key
? Enter a description for the API key:
? After how many days from now the API key should expire (1-365): 7
? Do you want to configure advanced settings for the GraphQL API No, I am done.
? Do you have an annotated GraphQL schema? No
? Do you want a guided schema creation? No
? Provide a custom type name MyType
Creating a base schema for you...

The following types do not have '@auth' enabled. Consider using @auth with @model
	 - MyType
Learn more about @auth here: https://aws-amplify.github.io/docs/cli-toolchain/graphql#auth


GraphQL schema compiled successfully.

Some next steps:
"amplify push" will build all your local backend resources and provision it in the cloud
"amplify publish" will build all your local backend and frontend resources (if you have hosting category added) and provision it in the cloud

With one command line call, amplify api add and a deployment, amplify push, I have provisioned a GraphQL API inside of my AWS account. Underneath the hood its using AWS AppSync and DynamoDB, but you don’t actually need to know that fact. You can just start implementing your GraphQL API and application.

That’s the power of AWS Amplify in my eyes. It lowers the barrier to entry to AWS by providing a quality tool that allows you to focus on building your applications without thinking about the underlying services.

Every backend feature currently built into French Verb Practice is leveraging AWS Amplify. The storage is backed by DynamoDB. The API is GraphQL using AWS AppSync. Authentication is handled with Cognito.

Adding language translation with AWS Amplify

If you recall from the screenshot earlier, the user sees the English translation right below the French version. This little feature is powered by AWS Amplify as well!

This comes from the predictions category provided by Amplify. You can add that category to your own application via the following steps:

$ amplify predictions add
? Please select from one of the categories below Convert
? What would you like to convert? Translate text into a different language
? Provide a friendly name for your resource translateTexta54aee3f
? What is the source language? French
? What is the target language? English
? Who should have access? Auth and Guest users
Successfully added resource translateTexta54aee3f locally

The steps above tell Amplify to provision the AWS services behind the scenes to support converting French text into English in our application.

Now how do we actually convert the text in my Verb component into English? It’s a lot easier than you might think.

import Predictions, {AmazonAIPredictionsProvider} from '@aws-amplify/predictions'

Amplify.addPluggable(new AmazonAIPredictionsProvider())
Amplify.configure(awsmobile)

function Verb(props: any) {
  const loadTranslation = async (infinitiveVerb: string) => {
    let result = await Predictions.convert({
      translateText: {
        source: {
          text: infinitiveVerb,
        },
      },
    })

    setTranslation(result.text)
  }
}

💥 With less than ten lines of code, this function will translate the French infinitiveVerb into English. It does so by leveraging the AmazonAIPredictionsProvider supplied by the @aws-amplify/predictions package. Once we configure the provider all we have to do is call Predictions.convert with the appropriate arguments 🎉

But wait, what about the feature where a user can hear the French verb spoken? That’s handled with AWS Amplify as well.

Adding text to speech with AWS Amplify

Similarly to what we did above for translating the text, we need to add a prediction for text to speech.

$ amplify predictions add
? Please select from one of the categories below Convert
? What would you like to convert? Generate speech audio from text
? Provide a friendly name for your resource speechGenerator7f0c7e00
? What is the source language? French
? Select a speaker Mathieu - Male
? Who should have access? Auth and Guest users
Successfully added resource speechGenerator7f0c7e00 locally

Notice that this time we selected the Generate speech audio from text in question two. We can then configure the source language and even which voice we would like.

To add the text to speech feature into French Verb Practices we can append onto our code from above another function called playTextInFrench.

import Predictions, {AmazonAIPredictionsProvider} from '@aws-amplify/predictions'

Amplify.addPluggable(new AmazonAIPredictionsProvider())
Amplify.configure(awsmobile)

function Verb(props: any) {
  const loadTranslation = async (infinitiveVerb: string) => {
    let result = await Predictions.convert({
      translateText: {
        source: {
          text: infinitiveVerb,
        },
      },
    })

    setTranslation(result.text)
  }

  const playTextInFrench = async () => {
    let result = await Predictions.convert({
      textToSpeech: {
        source: {
          text: (verb as VerbForm).infinitive,
        },
      },
    })

    var audio = new Audio()
    audio.src = result.speech.url
    audio.play()
  }
}

🇫🇷 Now we can hear the French verb spoken! In playTextInFrench we are using the same Predictions provider and convert function. Only this time we specify that we want to convert the text to audio by setting the textToSpeech property. What we get back is the url of the speech that we can then playback via the browser Audio object.

Just like that with two functions and less than twenty lines of code we have text translation and text to speech. This shows the power of the tooling AWS Amplify is providing. We don’t need to know which AWS services are being leveraged to do this, we can just focus on building our app.

Conclusion

As Amazon Web Services continues to grow the need for solid tooling continues to be critical. Great tooling leads to better and simpler ways for leveraging the platform. Those that are experts in AWS may not need interfaces like Amplify is providing. That’s OK, use what works for you.

Speaking from my own experience I can see well versed AWS folks outgrowing a tool like Amplify. Often times once your application is built and running, you want to optimize it for your use cases. This is where knowing what the underlying services are and how to leverage them is important.

But if your just beginning to venture into AWS and want to stay focused on building your application, Amplify is a hard thing to turn down. It may not be perfect. But it will allow you to do the most important thing when your building an application, build features.