Showing posts with label livecoding. Show all posts
Showing posts with label livecoding. Show all posts

Wednesday, 10 September 2014

Lunchtime thoughts - LiveCoding, Ephemeral Code and Engineering

I've recently been swimming around in the new pool of Swift like many others and am particularly enjoying the Playgrounds concept which is allowing me to explore as I learn and develop and keep a set of code scratchpads to one side. I was mulling this over in the car this morning and recollected the article by Alex McLean on Transient and Ephemeral code that I read with interest over the summer which resonated.

In the modern way that new thinking spreads in the internet age, I'd come to Alex's post by way of getting a FB update from the band 65 days of static, who I'd first heard of by way of the Comfort Conspiracy podcast which referenced the LiveCoding tool Gibber, from which I'd explored live-coding languages and found Tidal and Alex's fantastic blog on his various explorations making music with text.

Seeing coding as a performance art was a bit of a mind-blowing revelation to me. Having an engineering background and being used to pair-programming and code demonstrations this was quite a different way of seeing code being used as an artistic tool, just for the joy of it, it's purpose being the 'shape' or effect it generated at that time rather than to craft something that had a longer term goal and necessarily repeated use.

Which had me thinking this morning about the differences of engineering and arts approaches. I'll put my cards on the table as an artist now and say that I'm a printmaker. However, monoprinting never really resonated with me and I always gravitated towards wood/lino-cut and etching which has a strong process element and on the engineering side of fine-art. This again, is about creating something with longevity and practicing/honing skills and the end result into something tangible and permanent.

Where my mind drifted was towards performance art - video, animation, dance and obviously music, where the training was effectively directed to having a set of patterns and expertise that could be called upon at an instant in time to achieve a controlled or experimental result. Isn't this where LiveCoding really challenges and starts to come into its own?

Coding, due to the syntax and range of effort required on a time-basis to achieve a certain outcome result has always seemed (to me) too slow to make a real performance without it being pre-canned and choreographed like a keynote speech. Then it occured to me that isn't that what all performance is to a certain extent? We learn scales and chord patterns repeatedly to get good at being able to reproduce them not so much to conjure them creatively live. Even jazz is building a skilled repertoire that allows the creativity to work within the trained/learnt/explored boundaries. And Tidal is inherently designed to explore patterns. 

So, where does this leave us with code? My thinking returned back to the experiments in the playground. I'd seen from Mike Hodnick (kindohm)'s explorations with Tidal in his 365 Tidal Patterns learning and seeing text as a musical tool. Tidal and Gibber, which led me here, and many other custom languages are being driven by the relative ease with which it is now possible to make our own 'text-musical' instruments aka compilers or interpreters and the relative ease with which many concepts are being exposed as high-level APIs or frameworks which can be exploited (similar to the possibilities that Mashups initially simulated through the easy combination of many open, widely available APIs). I'll be interested to see if the open-source release of Rosyln by Microsoft creates any new musical languages, since like a traditional musician this is the syntax (or fingering) that I am most familiar with.

I've found that this has changed my behaviour and thinking more radically than I initially thought. Firstly, I'm sold now on text being an input device for a musical instrument and that live-coding to generate music is going to be an increasing phenomena as a performance art form. Second, I've noticed that I'm much more likely to kick-off some code to solve a problem and use it as a throw-away scratchpad. I still don't delete it, but it's more like one of my many scribbed moleskines and notebooks that contain half captured and explored ideas for code, creations, art and ideas. I've got a littered directory that is building up with results that are going no further than having achieved the results at the time. It's similar to all those spreadsheets that I call-up to do a set of complex calculations rather than using a calculator and then close without saving. I'm finding myself much more used to writing some code.

Which gets me on to the final point, which called to mind an article in the Guardian earlier this year on the need for a new revolution to teaching maths in schools and the approach now used in Estonia for a computer based approach to teaching maths. Seeing how my own children approach education and problems, maybe we need to increasingly consider ourselves that some of these text input tools that we're using have a far wider set of application and we should be using them more ephemerally for achieving our daily tasks - coding television programs anyone, rather than using an expensive, custom video editor??? I'm certain it's being done already.... which is why it's all the more laudable to see the work of the BBC once again supporting education for coding in schools (again, like Estonia)....

Lunchtime over



Tuesday, 9 September 2014

LiveCoding with Swift Audio continued...

After getting stuck trying out Swift Playgrounds using the new AVFoundation classes because I was on Mavericks the temptation was too much at the weekend and I created a new partition, downloaded the latest Yosemite build and got started. Jamie Bullock's Live coding audio with Swift playgrounds worked like a dream and I was up and running with some playground code making sounds. Great.

Googling a bit further I came across Gene de Lisa's excellent post trailblazing Swift and Core Audio and his note on using Swift with AVFoundation to play audio and generate midi notes from a sound bank. Good, this all worked nicely.

So, as this was going so smoothly I decided to jump straight in at the deep end and see how far the new AVFoundation classes and Swift bindings could be pushed... undeterred by Thomas Royal's notes on the current limitations of Audio Unit implementations with Swift as his blocker had seemed to be resolved in XCode6 Build7 I thought I'd see how far I could get following Matt Gallagher's wonderfully simple introduction to AudioUnits building an iOS tone generator. I thought I'd see how far this could go in a playground and with the basis of generating a tone the whole of audio synthesis would be open to me.

It's always fun to have something real and a little challenging as a goal to learning something new like Swift so I stumbled through reading the unfamiliar syntax and how to make the various cross-language bindings. All reasonably straightforward and answers were pretty forthcoming with some googling. Then, I got stuck. It seems the final step of being able to give a Swift function as a callback from C/C++ code is currently blocked by Apple. I followed this up on a number of different APIs and it looks like everyone is running into the same problem and the Apple Dev forums are saying this is currently not available and the alternatives are to make a C++/Objective-C/Closure trampoline [maybe I'll tackle that later] or another bit of kludgy glue. Hmmm, not nice! It's a little bit frustrating as Apple have done a good job of getting the typealias of the AURenderCallback nicely imported into AVFoundation, so let's just hope this is a matter of time and it will be resolved soon.

I'll detail where I got to and the code so far so I can share this in case it gets unblocked soon (it's in a playground, so not written for beauty, but to kick the tyres of what is possible):

import Cocoa
import AVFoundation


func RenderTone(inRefCon:UnsafeMutablePointer<Void>,
        ioActionFlags:UnsafeMutablePointer<AudioUnitRenderActionFlags>,
        inTimeStamp:UnsafePointer<AudioTimeStamp>,
        inBusNumber:UInt32,
        inNumberFrames:UInt32,
        ioData:UnsafeMutablePointer<AudioBufferList>) -> (OSStatus)
{
    // no-code as could not get this far!
    return 0;
}


var acd:AudioComponentDescription = AudioComponentDescription(
    componentType: OSType(kAudioUnitType_Output),
    componentSubType: OSType(kAudioUnitSubType_DefaultOutput),
    componentManufacturer: OSType(kAudioUnitManufacturer_Apple),
    componentFlags: 0,
    componentFlagsMask: 0)

var ao:AudioComponent = AudioComponentFindNext(nil, &acd)

var err:OSStatus

var aci:AudioComponentInstance = nil

err = AudioComponentInstanceNew(ao, &aci)

var aci2:UnsafeMutablePointer<AudioComponentInstance> = UnsafeMutablePointer<AudioComponentInstance>(aci)


var callback:AURenderCallbackStruct = AURenderCallbackStruct(RenderTone,nil)

It's the last line where things get stuck. What is happening is that the Swift function (RenderTone) cannot be matched to the typealias of AURenderCallback which is defined as a CFunctionPointer:

typealias AURenderCallback = CFunctionPointer<(
            (UnsafeMutablePointer<Void>,
             UnsafeMutablePointer<AudioUnitRenderActionFlags>,
             UnsafePointer<AudioTimeStamp>,
             UInt32,
             UInt32,

             UnsafeMutablePointer<AudioBufferList>)->OSStatus)

Having got all this way, I then found this post on the Apple Developer forums..... which kind of summarises that I'm not the only one to have got here only to bash my head against a wall :-(

However, as another post indicates, this might not be forever as it does seem that Apple may not be completely against putting this in eventually and the devs could just have a long list to get Swift, Yosemite and iOS8 out of the door... I know how that feels!

(Note, the reason for the slight differences in the type alias definition in the message and what I have stated above is that a number of these bindings changed in Xcode6 Build4 - ConstUnsafePointer to UnsafePointer and UnsafePointer to UnsafeMutablePointer and UnsafePointer<()> to UnsafeMutablePointer<Void>)


Friday, 5 September 2014

LiveCoding and Swift Audio

So, with everything else going on over the summer I finally got a chance to take a look at Swift Playgrounds and have a look at what kind of LiveCoding opportunities it presented. I was spurred on by Jamie Bullock's Live coding audio with Swift playgrounds  posting which has been well linked. Unfortunately I ran straight into the same problem that Kevin Irlen posted and did a bit of digging to find the problem which was easily identified by googling the functions and checking on the Apple Dev site - looking at the AVAudioMixerNode it clearly indicates that AVFoundation is for iOS8 or here for Mac OSX 10.10 (Yosemite). I've added this comment onto the post for others.

As my home machine is used for lots of other things I'm a bit reluctant to put a dev build of Yosemite on so will have to wait a little longer before playing with this audio example playground. Shame.


Update - the temptation was finally too much and I decided to give this a go on Yosemite....