Integrating ASR with SudMGP SDK in Small Games (iOS)

SudMGP provides interactive small games such as "Draw and Guess," "Speak and Guess," and "Number Bomb," supporting players to interact using voice commands to enhance playability and social attributes. The integration steps for incorporating ASR functionality into the SudMGP SDK are straightforward. This article outlines the steps for integrating ASR functionality into small games using the SudMGP SDK.

1. Background

Small games can be equipped with voice interaction capabilities. Apps need to obtain specific PCM data formats from the Agora RTC and pass them into the SudMGP SDK in a specified manner. Taking "hello-sud-plus-ios" as an example, the source code is available at: https://github.com/SudTechnology/hello-sud-plus-ios. "hello-sud-plus-ios" encapsulates the SudMGP SDK with SudMGPWrapper, and we recommend clients to integrate the SDK using SudMGPWrapper.

2. Integration Steps

SDK GitHub repository: https://github.com/SudTechnology/sud-mgp-ios. Please use the latest version available. Integration can be done via CocoaPods. Here's an example using version 1.3.4.1290:

1. Integrate the SudMGP SDK:

   • Standard version:

     pod 'SudMGPWrapper', '~> 1.3.4.1'
     

   • Lite version:

     pod 'SudMGPWrapper_Lite', '~> 1.3.4.1'
     

2. Integrate the Speech Recognition Library:

   pod 'MicrosoftCognitiveServicesSpeech-iOS', '1.23.0'
   

3. Starting ASR in Small Games

When a small game enters the ASR scene, ASR capability is automatically initiated. At this point, the game sends the MG_COMMON_GAME_ASR status to the app with isOpen set to true. Refer to the SudFSMMGListener protocol:

- (void)onGameMGCommonGameASR:(nonnull id <ISudFSMStateHandle>)handle model:(MGCommonGameASRModel *)model;

4. Starting RTC Audio Stream Monitoring in the App

Upon receiving the MG_COMMON_GAME_ASR status with isOpen == YES, the app invokes the AgoraRtcEngineKit method:

- (BOOL)setAudioFrameDelegate:(id<AgoraAudioFrameDelegate> _Nullable)delegate;

and implements the AgoraAudioFrameDelegate protocol to start capturing PCM data on the Agora side.

1. Calling [AgoraRtcEngineKit setAudioFrameDelegate:]: ```objc /// Start capturing original audio

2. (void)startPCMCapture { [[AsyncCallWrapper sharedInstance] addOperationWithBlock:^{

     AgoraRtcEngineKit *engine = [self getEngine];
     if (engine != nil) {
         /* Enable PCM data acquisition */
         [engine setAudioFrameDelegate:self];
     }
   

   }]; } ```

3. AgoraAudioFrameDelegate protocol: ```objc

   pragma mark - AgoraAudioFrameDelegate

4. (BOOL)onRecordAudioFrame:(AgoraAudioFrame _Nonnull)frame channelId:(NSString _Nonnull)channelId { [HSThreadUtils runOnUiThread:^{

     if (self.mISudAudioEventListener != nil && [self.mISudAudioEventListener respondsToSelector:@selector(onCapturedPCMData:)]) {
         NSUInteger length = frame.samplesPerChannel * frame.channels * frame.bytesPerSample;
         NSData *pcmData = [[NSData alloc] initWithBytes:frame.buffer length:length];
         [self.mISudAudioEventListener onCapturedPCMData:pcmData];
     }
   

   }]; return YES; }

// Other delegate methods...

3. Passing the RTC-captured PCM data to the SDK:
```objc
/**
 * Audio stream data
 */
- (void)onCapturedPCMData:(NSData *)data {
    [self.sudFSTAPPDecorator pushAudio:data];
}

5. Stopping RTC Audio Stream Monitoring in the App

When the small game exits the ASR scene due to hitting or timing out, it sends a notification to the app to stop capturing PCM data. Upon receiving MG_COMMON_GAME_ASR status with isOpen == NO, the app calls the AgoraRtcEngineKit method [AgoraRtcEngineKit setAudioDataFrame:nil] to close PCM data collection on the Agora side.

/// Stop capturing original audio
- (void)stopPCMCapture {
    [[AsyncCallWrapper sharedInstance] addOperationWithBlock:^{
        AgoraRtcEngineKit *engine = [self getEngine];
        if (engine != nil) {
            /* Disable PCM data acquisition */
            [engine setAudioDataFrame:nil];
        }
    }];
}

6. Using ASR Only to Play Games

When playing games using ASR only, the app only needs to handle the MG_COMMON_GAME_ASR status, responsible for starting/stopping PCM data collection on the local side. It does not need to send mg_common_key_word_to_hit to the game like in text hit scenarios.

7. Text Input Hit Scene for Playing Small Games

Small games with speech recognition scenes usually support concurrent use of text input for hits. The game notifies the app of the hit scene start via the mg_common_key_word_to_hit status. The app receives this notification through the SudFSMMGListener protocol:

- (void)onGameMGCommonKeyWordToHit:(nonnull id <ISudFSMStateHandle>)handle model:(MGCommonKeyWrodToHitModel *)model;

Text hit scenes in small games are divided into two categories:

1. Games where the App holds the keywords: For example, "Draw and Guess," "Speak and Guess." When model.word is not empty, the App needs to locally determine if there's a hit. After determining the hit, the App notifies the game via the interface method [SudFSTAPPDecorator notifyAppComonDrawTextHit].

2. Games where the App doesn't hold the keywords: For example, "Number Bomb." When model.word is empty, the App needs to send the text to the game each time for hit determination by the game.

- (void)handleGameKeywordHitting:(NSString *)content {
    // Number Bomb
    if (self.sudFSMMGDecorator.isHitBomb) {
        if ([self isPureInt:content]) {
            /// Keyword hit
            [self.sudFSTAPPDecorator notifyAppComonDrawTextHit:false keyWord:@"" text:content];
        }
        return;
    }

    // Draw and Guess
    if (self.sudFSMMGDecorator.keyWordHiting == YES && [content isEqualToString:self.sudFSMMGDecorator.drawKeyWord]) {
        /// Keyword hit
        [self.sudFSTAPPDecorator notifyAppComonDrawTextHit:true keyWord:self.sudFSMMGDecorator.drawKeyWord text:self.sudFSMMGDecorator.drawKeyWord];
    }
}