> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orgo.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Audio WebSocket

> Stream the desktop speaker over WebSocket as low-latency PCM frames.

Stream real-time audio from a computer's virtual speaker over WebSocket. Audio is captured from PulseAudio and delivered as raw PCM data in 20 ms frames - low-latency enough for live monitoring.

## Connection URL

```
wss://www.orgo.ai/desktops/{computer_id}/ws/audio?token={password}
```

### Authentication

The audio WebSocket requires the computer's password as the `token` query parameter. Retrieve it from the [Get VNC Password](/api-reference/computers/vnc-password) endpoint before connecting.

Connections without a valid token are rejected with close code `4401`.

### Query Parameters

<ParamField query="token" type="string" required>
  Computer password. Retrieve via the [Get VNC Password](/api-reference/computers/vnc-password) endpoint.
</ParamField>

<ParamField query="sample_rate" type="number" default="24000">
  Audio sample rate in Hz.
</ParamField>

<ParamField query="channels" type="number" default="1">
  Number of audio channels. `1` for mono, `2` for stereo.
</ParamField>

## Audio Format

| Property       | Value                                     |
| -------------- | ----------------------------------------- |
| Encoding       | `s16le` (signed 16-bit little-endian PCM) |
| Sample rate    | 24,000 Hz                                 |
| Channels       | 1 (mono)                                  |
| Frame duration | 20 ms                                     |
| Frame size     | 960 bytes                                 |
| Bitrate        | \~48 KB/s                                 |

Raw PCM - no codec, no container format. Each binary WebSocket frame contains one or more audio frames ready for playback.

## Protocol

### Connection Flow

1. Client connects with `?token=` parameter
2. Server validates the token
3. Server sends a JSON text frame confirming the audio configuration:
   ```json theme={null}
   { "type": "started", "sample_rate": 24000, "channels": 1 }
   ```
4. Server continuously sends **binary frames** containing raw PCM audio data
5. Client decodes and plays via Web Audio API or any PCM-capable player

### Client → Server Messages

<AccordionGroup>
  <Accordion title="stop" icon="stop">
    Stop audio capture and close the connection.

    ```json theme={null}
    {
      "type": "stop"
    }
    ```
  </Accordion>

  <Accordion title="ping" icon="heart-pulse">
    Send a heartbeat ping to keep the connection alive.

    ```json theme={null}
    {
      "type": "ping"
    }
    ```
  </Accordion>
</AccordionGroup>

### Server → Client Messages

<AccordionGroup>
  <Accordion title="started" icon="play">
    Sent immediately after connection. Confirms the audio stream configuration.

    ```json theme={null}
    {
      "type": "started",
      "sample_rate": 24000,
      "channels": 1
    }
    ```

    <ResponseField name="type" type="string">
      Always `"started"`.
    </ResponseField>

    <ResponseField name="sample_rate" type="number">
      Audio sample rate in Hz.
    </ResponseField>

    <ResponseField name="channels" type="number">
      Number of audio channels.
    </ResponseField>
  </Accordion>

  <Accordion title="Binary frames" icon="waveform-lines">
    Raw PCM audio data. Each binary frame contains signed 16-bit little-endian samples.

    At the default 24 kHz mono, each 20 ms frame is 960 bytes (480 samples × 2 bytes per sample).
  </Accordion>

  <Accordion title="pong" icon="heart-pulse">
    Response to a ping message.

    ```json theme={null}
    {
      "type": "pong"
    }
    ```
  </Accordion>

  <Accordion title="error" icon="circle-exclamation">
    Error starting or maintaining the audio stream.

    ```json theme={null}
    {
      "type": "error",
      "message": "Failed to start audio capture"
    }
    ```

    <ResponseField name="type" type="string">
      Always `"error"`.
    </ResponseField>

    <ResponseField name="message" type="string">
      Human-readable error description.
    </ResponseField>
  </Accordion>
</AccordionGroup>

## Examples

<CodeGroup>
  ```javascript JavaScript (Web Audio API) theme={null}
  const computerId = 'a3bb189e-8bf9-3888-9912-ace4e6543002';
  const apiKey = process.env.ORGO_API_KEY;

  // Step 1: Get the computer password
  const res = await fetch(
    `https://www.orgo.ai/api/computers/${computerId}/vnc-password`,
    { headers: { 'Authorization': `Bearer ${apiKey}` } }
  );
  const { password } = await res.json();

  // Step 2: Connect to audio stream
  const ws = new WebSocket(
    `wss://www.orgo.ai/desktops/${computerId}/ws/audio?token=${password}`
  );
  ws.binaryType = 'arraybuffer';

  // Step 3: Set up Web Audio API playback
  const ctx = new AudioContext({ sampleRate: 24000 });
  let nextPlayTime = 0;

  ws.onmessage = (event) => {
    // Text frames are JSON control messages
    if (typeof event.data === 'string') {
      const msg = JSON.parse(event.data);
      if (msg.type === 'started') {
        console.log(`Audio stream: ${msg.sample_rate}Hz, ${msg.channels}ch`);
      }
      return;
    }

    // Binary frames are raw PCM audio
    const pcm = new Int16Array(event.data);
    const floats = new Float32Array(pcm.length);
    for (let i = 0; i < pcm.length; i++) {
      floats[i] = pcm[i] / 32768;
    }

    const buffer = ctx.createBuffer(1, floats.length, 24000);
    buffer.getChannelData(0).set(floats);

    const source = ctx.createBufferSource();
    source.buffer = buffer;
    source.connect(ctx.destination);

    // Schedule with drift correction
    const now = ctx.currentTime;
    if (nextPlayTime < now || nextPlayTime > now + 0.15) {
      nextPlayTime = now + 0.02;
    }
    source.start(nextPlayTime);
    nextPlayTime += buffer.duration;
  };
  ```

  ```python Python theme={null}
  import asyncio
  import websockets
  import json
  import struct
  import requests

  async def stream_audio(computer_id: str, api_key: str):
      # Step 1: Get the computer password
      res = requests.get(
          f"https://www.orgo.ai/api/computers/{computer_id}/vnc-password",
          headers={"Authorization": f"Bearer {api_key}"}
      )
      password = res.json()["password"]

      # Step 2: Connect to audio stream
      url = f"wss://www.orgo.ai/desktops/{computer_id}/ws/audio?token={password}"

      async with websockets.connect(url) as ws:
          async for message in ws:
              if isinstance(message, str):
                  data = json.loads(message)
                  if data["type"] == "started":
                      print(f"Audio: {data['sample_rate']}Hz, {data['channels']}ch")
                  elif data["type"] == "error":
                      print(f"Error: {data['message']}")
                      break
              else:
                  # Binary frame - raw s16le PCM
                  samples = struct.unpack(f"<{len(message)//2}h", message)
                  # Process samples (save to file, play back, analyze, etc.)
                  print(f"Received {len(samples)} audio samples")

  computer_id = "a3bb189e-8bf9-3888-9912-ace4e6543002"
  api_key = "sk_live_..."
  asyncio.run(stream_audio(computer_id, api_key))
  ```

  ```typescript TypeScript theme={null}
  interface AudioConfig {
    type: 'started';
    sample_rate: number;
    channels: number;
  }

  class AudioStream {
    private ws!: WebSocket;
    private ctx!: AudioContext;
    private nextPlayTime = 0;

    static async connect(computerId: string, apiKey: string) {
      const stream = new AudioStream();

      // Get the computer password
      const res = await fetch(
        `https://www.orgo.ai/api/computers/${computerId}/vnc-password`,
        { headers: { 'Authorization': `Bearer ${apiKey}` } }
      );
      const { password } = await res.json();

      // Connect
      stream.ctx = new AudioContext({ sampleRate: 24000 });
      stream.ws = new WebSocket(
        `wss://www.orgo.ai/desktops/${computerId}/ws/audio?token=${password}`
      );
      stream.ws.binaryType = 'arraybuffer';

      stream.ws.onmessage = (event) => {
        if (typeof event.data === 'string') {
          const msg = JSON.parse(event.data);
          if (msg.type === 'started') {
            console.log(`Audio: ${msg.sample_rate}Hz, ${msg.channels}ch`);
          }
          return;
        }
        stream.playPCM(event.data);
      };

      return stream;
    }

    private playPCM(data: ArrayBuffer) {
      const pcm = new Int16Array(data);
      const floats = new Float32Array(pcm.length);
      for (let i = 0; i < pcm.length; i++) {
        floats[i] = pcm[i] / 32768;
      }

      const buffer = this.ctx.createBuffer(1, floats.length, 24000);
      buffer.getChannelData(0).set(floats);

      const source = this.ctx.createBufferSource();
      source.buffer = buffer;
      source.connect(this.ctx.destination);

      const now = this.ctx.currentTime;
      if (this.nextPlayTime < now || this.nextPlayTime > now + 0.15) {
        this.nextPlayTime = now + 0.02;
      }
      source.start(this.nextPlayTime);
      this.nextPlayTime += buffer.duration;
    }

    stop() {
      this.ws.send(JSON.stringify({ type: 'stop' }));
      this.ws.close();
      this.ctx.close();
    }
  }

  // Usage
  const audio = await AudioStream.connect(
    'a3bb189e-8bf9-3888-9912-ace4e6543002',
    'sk_live_...'
  );

  // Later: stop streaming
  audio.stop();
  ```
</CodeGroup>

## Playback Tips

<CardGroup cols={2}>
  <Card title="Browser Autoplay" icon="browser">
    Browsers require a user gesture (click, keypress) before `AudioContext` can play. Create the context inside a click handler, or call `ctx.resume()` after a user interaction.
  </Card>

  <Card title="Drift Correction" icon="clock">
    Schedule buffers slightly ahead of real time and reset when drift exceeds \~150 ms. This prevents gaps and keeps latency below 50 ms.
  </Card>

  <Card title="Heartbeat" icon="heart-pulse">
    Send periodic `ping` messages (every 30 seconds) to keep the connection alive and detect disconnections early.
  </Card>

  <Card title="Sample Rate" icon="waveform-lines">
    The default 24 kHz mono is optimized for voice and system sounds. Pass `sample_rate=48000` for higher fidelity if needed.
  </Card>
</CardGroup>

<Note>
  Audio streaming requires the computer to be running. If the computer is stopped, the WebSocket connection will be rejected. Audio is captured from the VM's virtual PulseAudio speaker - any sound the desktop produces (browser media, system alerts, application audio) is streamed.
</Note>

## Close codes

| Code   | Meaning                                              |
| ------ | ---------------------------------------------------- |
| `1000` | Normal closure (client sent `stop` or disconnected). |
| `1001` | Server shutting down or computer stopping.           |
| `4401` | Missing or invalid `token`.                          |
| `4404` | Computer not found or not running.                   |
| `4500` | Failed to start audio capture.                       |