More AVAudioPlayerNode with Swift and CompletionHandlers

Continuing on the subject of my previous posts looking at the new AVFoundation Audio classes in OSX10.10/iOS8 with Swift I finally found the error and a relatively obvious one at that. It wasn't closures specifically, but reading about them extended my knowledge and helped finding the cause of the problem.

I had the type-alias for the AVAudioNodeCompletionHandler all wrong. Not sure where I got that definition from, but my newness to the terminology of the error report put me off the scent. Xcode was quite clearly saying what the problem was:

Taking out atTime and options brings the function down to the simpler case of:

func scheduleBuffer(_ buffer: AVAudioPCMBuffer!,
  completionHandler completionHandler: AVAudioNodeCompletionHandler!)

What the error message is saying (when I finally understood it) was that the completion handler Tuple  - e.g. parameter types were not correct - the correct one is () and I had used (AVAudioPCMBuffer!, AVAudioTime!). It helped reading about closures to understand this, although that wasn't the cause of the problem. It does help understanding the syntax and concepts of Swift in a good deal more detail though.

The type alias of AVAudioNodeCompletionHandler is far much simpler, and for completeness is described below:

typealias AVAudioNodeCompletionHandler = @objc_block () -> Void

Putting this into the code (again in a Playground this is too slow), you get something like this:

fun handler() -> Void

{

    // do some audio work

}

player.scheduleBuffer(buffer,atTime:nil, options:nil, completionHandler: handler)

Or, now with my new found understanding of closures, like this:

player.scheduleBuffer(buffer,atTime:nil, options:nil,

    completionHandler: { () -> Void in

        // do some audio work

    })

Trying this again with the completion handler trick works nicely this time, but still annoyingly beats, so there is some other effect here that isn't working

//

//  main.swift

//  Audio

//

//  Created by hondrou on 11/09/2014.

//  Copyright (c) 2014 hondrou. All rights reserved.

//

import Foundation

import AVFoundation

let twopi:Float = 2.0 * 3.14159

var freq:Float = 440.00

var sampleRate:Float = 44100.00

var engine = AVAudioEngine()

var player:AVAudioPlayerNode = AVAudioPlayerNode()

var mixer = engine.mainMixerNode

var length = 4000

var buffer = AVAudioPCMBuffer(PCMFormat: player.outputFormatForBus(0),frameCapacity:AVAudioFrameCount(length))

buffer.frameLength = AVAudioFrameCount(length)

engine.attachNode(player)

engine.connect(player,to:mixer,format:mixer.outputFormatForBus(0))

var error:NSErrorPointer = nil

engine.startAndReturnError(error)

var j:Int=0;

func handler() -> Void

{

    for (var i=0; i<length; i++)

    {

        var val:Float = 5.0 * sin(Float(j)*twopi*freq/sampleRate)

        buffer.floatChannelData.memory[i] = val

        j++

    }

    

    player.scheduleBuffer(buffer,atTime:nil,options:.InterruptsAtLoop,completionHandler:handler)

}

handler()

player.play()

while (true)

{

    NSThread.sleepForTimeInterval(2)

    

    freq += 10

}

Hmph.... more still to get sorted


Now, just in case you wondered (I did) where I cooked up the handler function parameters, it was all down to mixing up two function type aliases that I'd been looking at. The previous incorrect handler function for the completionHandler below:

func handler(buffer:AVAudioPCMBuffer!,time:AVAudioTime!) -> Void

{

}

Is completely the proper type of function if you are installing an audio node tap block:

typealias AVAudioNodeTapBlock = (AVAudioPCMBuffer!, AVAudioTime!) -> Void

Which I'd also been thinking about at the time (and we will most likely be coming to next in our investigations).

For completeness, this is used in the AudioNode function installTapOnBus below:

func installTapOnBus(_ bus: AVAudioNodeBus,
          bufferSize bufferSize: AVAudioFrameCount,
              format format: AVAudioFormat!,
               block tapBlock: AVAudioNodeTapBlock!)

AVFoundation Audio with Swift using AVAudioPlayerNode

Having been blocked using AudioUnit callbacks in Swift in my previous exploration, I decided to take a different direction and had another look at the WWDC video, presentation and transcript given for the new AVFoundation changes for Audio. Unfortunately it's described in terms of Objective-C, but the one of the interesting points is around the description of using AVAudioPlayerNode and the scheduleBuffer function.

func scheduleBuffer(_ buffer: AVAudioPCMBuffer!,
             atTime when: AVAudioTime!,
            options options: AVAudioPlayerNodeBufferOptions,
  completionHandler completionHandler: AVAudioNodeCompletionHandler!)

My first thoughts were, great, look there's a callback to indicate that the buffer has played out which can then be called and re-filled, it's in Swift, which means we can workaround the previous callback problems. So, I knocked up another playground to test this.

The scheduleBuffer call allows for options to be set to Loops, Interrupts, InterruptsAtLoop or nil. Check out the WWDC material which explains this with some diagrams and an ADSR sort of example.

Taking baby steps, I thought I'd basically fill-up a buffer with a simple Sine wave and then play that out as a continuous loop to get started. The buffer needs to be an AVAudioPCMBuffer. If you take a look at Bob Burns' post on Gene de Lisa's blog, he's trying something similar. My code looks like this:

import Cocoa

import AVFoundation

let twopi:Float = 2.0 * 3.14159

var freq:Float = 440.00

var sampleRate:Float = 44100.00

var engine = AVAudioEngine()

var player:AVAudioPlayerNode = AVAudioPlayerNode()

var mixer = engine.mainMixerNode

var buffer = AVAudioPCMBuffer(PCMFormat: player.outputFormatForBus(0),frameCapacity:100)

var length = 100

buffer.frameLength = AVAudioFrameCount(length)

// fill up the buffer with some samples

for (var i=0; i<length; i++)

{

    var val:Float = 10.0 * sin(Float(i)*twopi*freq/sampleRate)

    buffer.floatChannelData.memory[i] = val

}

engine.attachNode(player)

engine.connect(player,to:mixer,format:mixer.outputFormatForBus(0))

var error:NSErrorPointer = nil

engine.startAndReturnError(error)

player.scheduleBuffer(buffer,atTime:nil,options:.Loops,completionHandler:nil)

player.play()

// keep playground running

import XCPlayground

XCPSetExecutionShouldContinueIndefinitely(continueIndefinitely:true)

Fantastic, I got some audio out, seemed like a tone, but was getting some glitchy audio effects I expect due to the buffer not smoothly containing a single cycle. After trying this and googling a bit I found Thomas Royal had also tried something similar. At least I'm getting some sound out now.

So, taking this further I thought, rather than making the Sine fit a cycle I could simply set the completionHandler callback and get an indication of when to play the next buffer chunk and I'd be away generating what I liked. [Just as a note, my assumption was that options could be set to nil or InterruptsAtLoop and effectively we'd be creating audio double-buffering so that samples could be created during the buffer playout and there would be no wait from getting the completion handler to setting the next buffer].

The empty completion handler looks like this:

func handler(buffer:AVAudioPCMBuffer!,time:AVAudioTime!) -> Void

{

}

I then tried setting as follows:

player.scheduleBuffer(buffer,atTime:nil,options:.InterruptsAtLoop,completionHandler:handler)

And got this 'helpful' error:



Hmmm. I tried taking this out of the Playground, I tried a number of different ideas. None worked. Damn! I googled a lot on this and completion handlers generally and didn't get any results. Shame.

That avenue blocked (hopefully for now), undeterred I thought I'd give this another go. Changing approach again, I thought, well, if I'm not getting a callback, maybe I just create a thread and stuff buffers into the player, I could get cleverer later on and use the atTime parameters (assuming that would work) and put the buffers in given some consideration for timing. Indeed doing this might be a nice way to ensure that the timing alignment of various players were synched. But I'm getting ahead of myself now.

The revised fragment looks like this:

let queue = NSOperationQueue()

queue.addOperationWithBlock({

    

    var j:Int=0;

    

    while(true)

    {

        for (var i=0; i<length; i++)

        {

            var val:Float = 5.0 * sin(Float(j)*twopi*freq/sampleRate)

            buffer.floatChannelData.memory[i] = val
            j++

        }

        

        player.scheduleBuffer(buffer,atTime:nil,options:.InterruptsAtLoop,completionHandler:nil)

        let thread = NSThread.currentThread()

        

        NSThread.sleepForTimeInterval(0.1)

    }

})

This proved to be problematic in the playground as it tried to show filling the loop each cycle, which took longer than the playback, so I first tried to move this part of the code to a Framework to import (unsuccessfully, something I'll come back to later as it's going to be key to being able to use Playgrounds effectively) and then just into a normal Console application:

//

//  main.swift

//  Audio

//

//  Created by hondrou on 11/09/2014.

//  Copyright (c) 2014 hondrou. All rights reserved.

//

import Foundation

import AVFoundation

let twopi:Float = 2.0 * 3.14159

var freq:Float = 440.00

var sampleRate:Float = 44100.00

var engine = AVAudioEngine()

var player:AVAudioPlayerNode = AVAudioPlayerNode()

var mixer = engine.mainMixerNode

var length = 4000

var buffer = AVAudioPCMBuffer(PCMFormat: player.outputFormatForBus(0),frameCapacity:AVAudioFrameCount(length))

buffer.frameLength = AVAudioFrameCount(length)

engine.attachNode(player)

engine.connect(player,to:mixer,format:mixer.outputFormatForBus(0))

var error:NSErrorPointer = nil

engine.startAndReturnError(error)

let queue = NSOperationQueue()

queue.addOperationWithBlock({

    

    var j:Int=0;

    while(true)

    {

        for (var i=0; i<length; i++)

        {

            var val:Float = 5.0 * sin(Float(j)*twopi*freq/sampleRate)

            buffer.floatChannelData.memory[i] = val
            j++

        }

        

        player.scheduleBuffer(buffer,atTime:nil,options:.InterruptsAtLoop,completionHandler:nil)

        let thread = NSThread.currentThread()

        

        NSThread.sleepForTimeInterval(0.1)

    }

})

player.play()

while (true)

{

    NSThread.sleepForTimeInterval(1)

    

    //freq += 10

}

Rather than the keep-alive for the playground at the end I'm keeping the main thread alive with a simple loop (which I'll use later to adjust the frequency to check that this is not just playing a single tone).

This played back and ok, I got audio, but those funny glitches were still there. So I played around with the sleep loop interval and the size of the buffer with varying results, but none of them nice, then decided to go to bed! Stumped and not too happy about it.

Hmmmm, not all lost yet as I have some other ideas, but I'm away for the next few days on a biz trip so will have to try this later on. If anyone has any good comments/suggestions before then I'd be most grateful. I'm hoping that Swift should be man-enough for the job. C# certainly can cope with this kind of relatively simple synthesis and it's running in the CLR.



Update 
doh! that'll teach me for late night coding. I finally found the problem with the completion handler and have just posted another blog entry