Agora 4.x + Mini Game ASR Integration (Android)

SudMGP provides interactive mini-games like "Pictionary", "Guess the Word", and "Number Bomb" that support voice interactions, enhancing playability and social aspects. The integration steps are straightforward, and this document outlines the steps to integrate the ASR feature of the SudMGP SDK.

I. Background

Mini-games can have voice interaction capabilities, and the App only needs to obtain specific format PCM data from Agora RTC and pass it to the SudMGP SDK in a specified way. Taking hello-sud-plus-android as an example, the source code can be found at: https://github.com/SudTechnology/hello-sud-plus-android hello-sud-plus-android encapsulates the integration of the SudMGP SDK with SudMGPWrapper. We recommend clients to integrate the SDK using SudMGPWrapper.

II. Integration Steps

SDK GitHub link: https://github.com/SudTechnology/sud-mgp-android Please use the latest version, and you can integrate using Maven. For example, using version V1.3.2.1154:

• 1. Integrate SudMGP SDK: Standard version: implementation 'tech.sud.mgp:SudMGP:1.3.2.1154' Lite version: implementation 'tech.sud.mgp:SudMGP-lite:1.3.2.1154'
• 1. Integrate Speech Recognition SudASR SDK: implementation 'tech.sud.mgp:SudASR:1.3.2.1154' Explanation:
• SudASR is an extension library that the new SDK runtime will look for to implement ASR multilingual recognition capabilities.
• Library download link: https://github.com/SudTechnology/sud-mgp-android/releases
• Demo link: https://github.com/SudTechnology/hello-sud-plus-android

III. Starting ASR in the Mini Game

When the mini-game enters the speech recognition scene, the ASR capability will automatically start. At this point, it will send the MG_COMMON_GAME_ASR state to the App with isOpen == true, as seen in: SudFSMMGListener.onGameMGCommonGameASR

IV. App Starts Listening to RTC Audio Streams

Once the App receives the MG_COMMON_GAME_ASR state with isOpen == true, call the Agora interface RtcEngine.registerAudioFrameObserver(iAudioFrameObserver) and implement the IAudioFrameObserver interface object to start Agora local PCM data collection.

1. Calling RtcEngine.registerAudioFrameObserver:

   @Override
   public void startPCMCapture() {
    AsyncCallWrapper.sharedInstance().executeInSerial(new Runnable() {
        @Override
        public void run() {
            RtcEngine engine = getEngine();
            if (engine != null) {
                engine.registerAudioFrameObserver(iAudioFrameObserver);
            }
        }
    });
   }
   

2. Implementing the IAudioFrameObserver interface object: (Provided code snippet)

getRecordAudioParams() callback method sets the PCM data format. The onRecordAudioFrame() callback method returns the local PCM data captured by RTC, with further processing explained in the next section.

1. Passing the RTC Captured PCM Data to the SDK: The onRecordAudioFrame() callback method returns local PCM data slices, and the following method is used to pass the PCM data to the SDK:

   public void onCapturedAudioData(AudioPCMData audioPCMData) {
    sudFSTAPPDecorator.pushAudio(audioPCMData.data, audioPCMData.dataLength);
   }
   

   The pushAudio interface can be called in a working thread. Agora's audio slices default to 10ms, but the length can be adjusted for better results when passed to pushAudio.

V. App Stops Listening to RTC Audio Streams

When the mini-game exits the speech recognition scene due to a hit or timeout, it will send a state notification to the App to stop capturing PCM data. Once the App receives the MG_COMMON_GAME_ASR state with isOpen == false, call the Agora interface RtcEngine.registerAudioFrameObserver(null) to stop local PCM data collection by Agora.

VI. Playing Games with ASR Only

When playing games with ASR only, the App only needs to handle the MG_COMMON_GAME_ASR state to enable/disable local PCM data collection. There is no need to send mg_common_key_word_to_hit to the game as in text-based hits.

Seven, Text-Based Hits for Playing Mini Games

In mini-games with speech recognition scenarios, text input can often be used simultaneously for hits. The game informs the App about the hit scene by sending the mg_common_key_word_to_hit state. The App receives this notification through the callback interface:

SudFSMMGListener.onGameMGCommonKeyWordToHit(ISudFSMStateHandle handle, SudMGPMGState.MGCommonKeyWordToHit model)

Text-based hit scenarios in mini-games can be categorized into two types:

1. For games where the App holds the keyword, such as "Pictionary" and "Guess the Word", where model.word is not empty, the App needs to locally determine if the hit is successful. After determining the hit, the App notifies the game through the method sudFSTAPPDecorator.notifyAPPCommonSelfTextHitState.
2. For games where the App does not hold the keyword, such as "Number Bomb", where model.word is empty, the App needs to send the text to the game each time for the game to determine if there is a hit.

   public void sendMsgCompleted(String msg) {
    if (msg == null || msg.isEmpty()) {
        return;
    }
    // Number Bomb
    if (sudFSMMGDecorator.isHitBomb() && HSTextUtils.isInteger(msg)) {
        sudFSTAPPDecorator.notifyAPPCommonSelfTextHitState(false, null, msg, null, null, null);
        return;
    }
    String keyword = gameKeywordLiveData.getValue();
    if (keyword == null || keyword.isEmpty()) {
        return;
    }
    // Pictionary, check if the keyword is hit, using contains check. The integration partner can use equals or other custom conditions based on their business requirements.
    if (msg.contains(keyword)) {
        sudFSTAPPDecorator.notifyAPPCommonSelfTextHitState(true, keyword, msg, null, null, null);
        gameKeywordLiveData.setValue(null);
    }
   }