Instrument Stem Separation

Once the SDK is initialized, you can start using AudioShake functionalities in your app. The SDK zip package includes useful example demo apps for iOS and MacOS which you can build and run to see how to implement the various use cases.

1. File Processing

Once you've initialized the AudioShake SDK, you can begin leveraging its features to process audio files. One common task is to separate audio tracks into stems using the SourceSeparationOperation class. This operation allows you to specify the source audio file URL, the destination URL for the processed files, the compute device to be used (such as neural accelerator), and the stem configuration defining which stems to extract.

import AudioShake

// Example function to perform source separation
func performSourceSeparation(sourceFileURL: URL) {
  // Define a temporary directory for storing processed files
  let tempDir = FileManager.default.temporaryDirectory
  
  // Configure the stem separation (e.g., Drums, Vocals, Bass, Other)
  let stemConfiguration: StemConfiguration = .drumsVocalsBassOther
  
  // Initialize the source separation operation
  let separationOperation = 
          SourceSeparationOperation(sourceFileURL: sourceFileURL,
                                  destinationURL: tempDir,
                                    computeDevice: .neuralAccelerator,
                                stemConfiguration: stemConfiguration)
  
  // Perform the source separation asynchronously
  separationOperation.completionBlock {
    if let stemFilesDirectory = separationOperation.stemFilesDirectory {
          print("Audio separation completed successfully.")
          // Handle the separated stems
      } else {
          print("Error occurred during audio separation")
          // Handle the error. Check logs for SDK errors
      }
  }
}

In the above code snippet, we define a function performSourceSeparation that takes the URL of the source audio file as input. We specify a temporary directory to store the processed files and configure the stem separation using the StemConfiguration enum, in this case, drumsVocalsBassOther which separates the audio into drums, vocals, bass, and other stems.

Next, we initialize the SourceSeparationOperation with the specified parameters, including the source file URL, destination directory, compute device (here, we've chosen the neural accelerator for faster processing), and the stem configuration.

Finally, we start the source separation operation asynchronously. Upon completion, the completion handler is called and if the separation completed successfully then the stemFilesDirectory

2. File Playback

After processing audio files with AudioShake, you may want to playback the separated stems to provide users with an immersive audio experience. The SourceSeparationTask class allows you to perform real-time playback of audio stems, making it easy to integrate into your app's audio playback functionality.

import AudioShake
import AVFoundation

let audioEngine: AudioPlayerEngine?
 
// Example function for playing back separated audio stems
func playSeparatedAudioStems(input: URL) {
  // Initialize an AudioPlayerEngine 
  // which uses the source separation task
  audioEngine = AudioPlayerEngine(fileURL: input)
  audioEngine?.start()
}

In the above example, we define a function playSeparatedAudioStems that takes the input audio file URL. In the function we initialize the AudioPlayerEnginer class which internally uses SourceSeparationTask to pass the separated output data to the buffers for playback.

3. Live Input

In addition to processing pre-recorded audio files, AudioShake allows you to capture live audio input in real-time, enabling dynamic audio manipulation and separation. By leveraging the AVAudioEngine class and AudioShake's capabilities, you can create immersive audio experiences that respond to user input or external audio sources.

import AudioShake
import AVFoundation

// Example function for capturing and processing live audio input
func captureAndProcessLiveAudio() {
  // Configure AudioShake settings
  let stemConfiguration: StemConfiguration = .drumsVocalsBassOther
  
  // Initialize the AudioShake engine
  // Initialize an MicrophoneEngine 
  // which uses the source separation task
  audioEngine = MicrophoneEngine()
  audioEngine?.start()
}

In the above example, we define a function captureAndProcessLiveAudio that sets up AudioShake to process live audio input. We configure AudioShake with the desired stem configuration (e.g., drums, vocals, bass, other).

In the MicrophoneEngine code we initialize an AVAudioEngine instance and retrieve its input node. We set a preferred buffer size for audio processing and install a tap on the input node using the installTap method. Inside the tap block, we push each audio buffer to the audio engine for processing.

Finally, we start the AVAudioEngine to begin capturing and processing live audio input.

Then we spawn another thread that will accept the input audio from the audio engine and then pass it to a SourceSeparationTask which will do the separation. We also create and pass in an QueueOutput that will keep track of how much data is available to be played out. We can pull frames from this queue output and schedule them for playback in our output nodes.