Multi-Speaker Separation

​

Create a Task

import requests

API_KEY = "your_api_key"
HEADERS = {"Content-Type": "application/json", "x-api-key": API_KEY}

response = requests.post(
    "https://api.audioshake.ai/tasks",
    headers=HEADERS,
    json={
        "assetId": "your_asset_id",
        "targets": [
            {"model": "multi_voice", "formats": ["wav"], "residual": True}
        ]
    }
)

task_id = response.json()["id"]
print(f"Task created: {task_id}")

const API_KEY = "your_api_key";
const headers = { "Content-Type": "application/json", "x-api-key": API_KEY };

const createRes = await fetch("https://api.audioshake.ai/tasks", {
  method: "POST",
  headers,
  body: JSON.stringify({
    assetId: "your_asset_id",
    targets: [
      { model: "multi_voice", formats: ["wav"], residual: true }
    ]
  })
});

const { id: taskId } = await createRes.json();
console.log(`Task created: ${taskId}`);

curl -X POST "https://api.audioshake.ai/tasks" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $AUDIOSHAKE_API_KEY" \
  -d '{
    "assetId": "your_asset_id",
    "targets": [
      { "model": "multi_voice", "formats": ["wav"], "residual": true }
    ]
  }'

{ "model": "multi_voice", "formats": ["wav"], "numSpeakers": 2 }

​

Outputs

​

The results file

{
  "results": {
    "diarization": {
      "segments": [
        { "speaker": "SPEAKER_01", "start": 0.333, "end": 4.633 },
        { "speaker": "SPEAKER_02", "start": 4.832, "end": 12.672 },
        { "speaker": "SPEAKER_01", "start": 11.921, "end": 19.861 }
      ],
      "num_speakers": 2,
      "scores": {
        "scores_per_frame": {
          "assignment_confidence": { "resolution": 0.02, "score": [0.798, 0.982, "..."] },
          "separation_confidence": { "resolution": 0.02, "score": [0.998, 0.998, "..."] }
        },
        "scores": {
          "assignment_confidence": 0.887,
          "separation_confidence": 0.786
        }
      }
    }
  }
}

• segments — one entry per speech segment, with speaker (matching the speaker_XX audio outputs) and start/end in seconds. The timeline is overlap-aware: segments from different speakers can overlap, as SPEAKER_01 and SPEAKER_02 do between 11.921s and 12.672s above. Nothing gets dropped when people talk at once.
• num_speakers — number of distinct speakers detected.
• scores.scores_per_frame.<metric> — per-frame confidence values in [0, 1]. resolution is the frame duration in seconds (0.02 = 20 ms); multiply a frame index by it to get the timestamp.
• scores.scores.<metric> — whole-file confidence values, averaged with weighting by speech activity so silence can’t inflate them.

​

Confidence scores

• separation_confidence — how confident the model is that it correctly pulled apart overlapping speech. Most informative where speakers actually overlap; on single-speaker frames it saturates near 1.0 unless there is bleed between tracks.
• assignment_confidence — how confident the model is that speech is attributed to the right speaker. Treat this as the diarization-quality signal.

• Only separation_confidence dips — the labels are right, but the audio in that region likely carries bleed from another voice. Still fine for talk-time analytics; exclude it from per-speaker ASR, dubbing, or voice training data.
• Only assignment_confidence dips — the audio is clean, but it may be attributed to the wrong speaker. Fine for identity-independent uses like cleanup; risky for anything routed by speaker.
• Both dip together — usually overlapping speech the model couldn’t untangle. Drop the region, or route it to review.
• Both high — trust the region. Note that silence also scores high, so check the region against segments before treating a high score as confirmed clean speech.

​

Use cases

• Transcription and captioning — transcribe each speaker’s track independently so overlapping speech is never assigned to the wrong person or dropped. The diarization timeline is already aligned to the tracks, with no second pass to reconcile.
• Training data for speech AI — turn real multi-party recordings into speaker-isolated tracks with turn boundaries and quality scores attached, and filter a corpus by confidence instead of by ear.
• Post-production — get a track per person in unscripted TV, documentaries, and interviews, and edit one voice without touching another. The residual track keeps room tone and effects available.
• Voice and media pipelines — feed clean per-speaker audio into translation, dubbing, or analysis, on the phone-quality and far-field recordings those pipelines actually receive.