Large Model Game Interaction Access (Web)

In games that support AI interaction, users can interact with large model AI players through voice or text. The premise for interaction is that there must be at least one real person in the game, and then the large model AI player can be added through the corresponding interface. Subsequent interactions can be conducted through the ISudAiAgent-related interfaces.

ISudAiAgent Interface

interface ISudAiAgent {

  /**
   * The incoming audio slice should be PCM data obtained from RTC.
   * PCM data format must be:
   *   Sample rate: 16000
   *   Bit depth: 16
   *   Channel: MONO
   * PCM data length can be adjusted based on desired performance:
   *   Longer length: higher accuracy but more delay.
   *   Shorter length: lower latency but reduced accuracy.
   */
  pushAudio(data: Buffer, dataLength: number): void;

  /** Send text content */
  sendText(text: string): void;

  /**
   * Pause voice recognition — temporarily stop calling pushAudio.
   * When called, any pending recognition results will be finalized.
   */
  pauseAudio(): void;

  /**
   * Stop voice recognition completely.
   */
  stopAudio(): void;

  /** Set callback listener */
  setISudListenerAiAgent(listener: ISudListenerAiAgent): void;
}

I. Getting the ISudAiAgent

After the game is loaded, obtain the ISudAiAgent through the getAiAgent method of the ISudFSTAPP interface. In the demo, you can retrieve ISudAiAgent via iSudFSTAPP after the onGameStarted callback. Once you have the ISudAiAgent instance, you can use it to interact with the large-model AI player through voice or text, and set up a listener to receive the AI player's response messages.

Example:

onGameStarted() {
  // Retrieve it after onGameStarted is triggered to ensure it is not null.
  const ai = nsdk.iSudFSTAPP?.getAiAgent()
  console.log('[ ai ] >', ai)
  if (ai) {
    setAiAgent(ai!)
  }
}

II. Adding an AI Player

Interface documentation: Common State - notifyStateChange

52. Add AI players powered by large models
app_common_game_add_big_scale_model_ai_players

When there is at least one real player in the game, the client can use this protocol to send a message to the game and add an AI player.

// Example in Demo/QuickStart: adding an AI player

const aiPlayer = {
  userId: 'userid', // Player ID
  avatar: 'url', // Avatar URL
  name: `xxxname`, // Name
  gender: 'male', // Gender (male or female)
  aiId: '1' // Random AI personality (currently supports 1–370)
}
const aiPlayers: IAiModel = {
  aiPlayers: [aiPlayer],
  isReady: 1 // Whether the AI should auto-ready after joining (1 = yes, 0 = no, default is 1)
}

SudSDK && SudSDK.sudFSTAPPDecorator.notifyAPPCommon(
  'app_common_game_add_big_scale_model_ai_players',
  JSON.stringify(aiPlayers)
)

III. Text Interaction

The app can send text messages to the AI player. Use the sendText method from ISudAiAgent.

public interface ISudAiAgent {
    ...
    /** Send text content */
    sendText(text: string): void;
    ...
}

const sendText = (text: string) => {
  if (aiAgent && text) {
    aiAgent.sendText(text)
  }
}

IV. Voice Interaction

The app can also send voice messages to the AI player. Use the pushAudio method to send audio data. pauseAudio temporarily pauses interaction, and stopAudio stops it completely.

public interface ISudAiAgent {
    ...
    /**
     * Send voice data (optional implementation)
     * The audio slice must be PCM data from RTC or recording.
     * PCM format:
     *   Sample rate: 16000
     *   Bit depth: 16
     *   Channel: MONO
     * Longer audio chunks give higher accuracy but more delay, shorter chunks have lower delay but less accuracy.
     */
     pushAudio(data: Buffer, dataLength: number): void

    /** Pause recognition */
    pauseAudio(): void

    /** Stop recognition */
    stopAudio(): void
    ...
}

aiAgent?.pushAudio(pcm as any, length)

V. Receiving AI Player Messages

When the app sends text or voice to the AI player, the AI may reply with one or more messages. Use the setISudListenerAiAgent method to listen for AI messages.

public interface ISudAiAgent {
    ...
    /** Set listener */
    setISudListenerAiAgent(listener: ISudListenerAiAgent);
    ...
}

public interface ISudListenerAiAgent {
    /** AI chat message callback */
    onRoomChatMessage(json: string): void;
}

Example JSON:

{
    "uid": "26", 
    "content": "I’m a Gemini, what about you?", 
    "audioData": "Base64-encoded audio data"
}

Demo Example:

onGameStarted() {
  const ai = nsdk.iSudFSTAPP?.getAiAgent()
  console.log('[ ai ] >', ai)
  if (ai) {
    setAiAgent(ai!)
    ai.setISudListenerAiAgent({
      // Receive AI messages
      onRoomChatMessage(data) {
        if (data) {
          const parseData = JSON.parse(data)
          console.log('[ parseData uid] >', parseData.uid, '[ parseData content] >', parseData.content)

          setUserAudioPlayState(parseData.uid, { state: 1, uid: parseData.uid })
          // Simulate audio playback
          player.pushSrc({
            src: base64ToBlobUrl(`data:audio/aac;base64,${parseData.audioData}`),
            data: parseData
          })
          const list = aiUserContentList
          list.push(parseData)
          setAiUserContentList([...list])
        }
      }
    })
  }
},

When onRoomChatMessage is triggered, you can display the AI’s text or play its voice depending on your product design.

VI. Voice Wave Control

When the app receives the following game message, it indicates that the app can control the visual representation of the player's voice wave in the game. See documentation: Common State - Game

69. Notify app to start pushing microphone state
mg_common_game_player_mic_state

After receiving this instruction, the app can control it using: Common State - notifyStateChange

51. App sends player mic state to the game
app_common_game_player_mic_state

This is a toggle instruction — send once to start or stop.

onGameCustomerStateChange(handle, state, data) {
  switch (state) {
    case 'mg_common_game_player_mic_state': {
      console.log('[ Can start pushing mic state ] >', data)
      setIsGamePlayerMicStateOk(true)
      break
    }
  }
}

const value = {
  "uid": "user id", // Player ID
  "state": 0   // 0: not speaking, 1: speaking
}
// Control player’s voice wave effect
SudSDK && SudSDK.sudFSTAPPDecorator.notifyAPPCommon(
  'app_common_game_player_mic_state',
  JSON.stringify(value)
)

Web