Monday, 15 August 2016

AWS Lambda - more than ImageMagick and configuring API Gateway

We've looked at getting setup a basic Lambda function that returns a scaled version of a file held on S3. Not very exciting or dynamic. It's great that ImageMagick is available as a package for Lambda and this means that some pretty basic Lambdas can be setup just using the inline code page on AWS to get things kicked off. For anything more ImageMagick doesn't quite give us enough. This also pushes us on to the next step of using Lambdas for a bit more functionality.

We're going to take another baby step and look at providing a slightly improved function to take a cutout tile from the image used prevsiously. Imagine we have an image that is composed of a tile of 5 rows and 10 columns of images and give some consideration to a little border region between each. What we want to do is return one of those tiles in the function call.

ImageMagick no longer has enough functionality to help, so we're going to use GraphicsMagick to provide the additional functionality. The code example is shown below:


var im = require("imagemagick");
var fs = require("fs");
var gm = require('gm').subClass({
    imageMagick: true
});
var os = require('os');

var AWS = require('aws-sdk');
AWS.config.region = 'eu-west-1';

exports.handler = function(event, context) {
    
    var params = {Bucket: 'bucket', Key: 'image.png'};
    var s3 = new AWS.S3();
                
    console.log(params);


    s3.getObject(params, function(err, data) {
        if (err) 
            console.log("ERROR", err, err.stack); // an error occurred
        else     
            console.log("SUCCESS", data);         // successful response
        
            var resizedFile = os.tmpdir() + '/' + Math.round(Date.now() * Math.random() * 10000);
        
            console.log("filename", resizedFile); 
            
            
            gm(data.Body).size(function(err, size) {
                
                console.log("Size", size); 
                
                var num = 10;

                var x = size.width / num;
                var y = size.height / 5;
            
                var col = 0;
                var row = 0;
            
                
                gm(data.Body)
                    .crop(x-14, y-14,
                            (col * x)+7,(row * y)+7)
                    .write(resizedFile, function(err) {
                    
                        console.log("resized", resizedFile); 
                        
                        if (err) {
                            console.log("Error", err); 
                            context.fail(err);
                        }
                        else {
                            var resultImgBase64 = new Buffer(fs.readFileSync(resizedFile)).toString('base64');
                            context.succeed({ "img" : resultImgBase64 });
                        }
                        
                    });
                });
                
        });
    
};

In this example we're still not sending in any parameters and we're looking to get back the (0,0) referenced tile in the top-left corner. Using the below image, this would be the blue circle:


If you put this Lambda code directly into the inline editor and try to call it you'll get an error as it cannot find gm, GraphicsMagick. To add it in unfortunately we need to start packaging up our Lambda. So, first step, you need to download the gm package, which is easily done with npm as follows:


npm install gm

If you do this in the root of your project you'll get a folder called nod_modules containing a bunch of files. Now, create a zip file along with your Node,js function above. The contents inside should look like this:





Now in your Lambda you will need to select your Zip file, upload it (by clicking Save - yep, a bit unintuitive) and you'll need to configure the Configuration tab of your Lambda. The Handler will need to be set to call the Node.js filename and the function that you export, so in this case the filename is called 'getImage' and the handler exported has been called 'handler'.







Great. Now things should run smoothly and you'll get back a tile (hopefully the round circle) from the image. To make this a bit more useful it would be good to now pass in the row + column information to be able to choose the tile, so let's do that.

Adjusting the Lambda code we can get parameter information in via the 'event' information passed into the handler function. Let's add a line to log this to the console.


exports.handler = function(event, context) {
    
    console.log('Received event:', JSON.stringify(event, null, 2));

Now configure your test event from the Lambda console - Actions/Configure Test Event to something like this:














And run a test execution, you should get something like this in the log (Click on the Monitoring tab in Lambda and scroll down):












Great, we're calling this from within AWS as a test and getting back the row & column information. It's now simple enough to get these values to set the row & column variables. For the sake of simplicity I'll leave out error handling in case events are not present and if the values are not correct, that's standard code work.


     var col = event.col;
     var row = event.row;

Run again and take a look at the base64 string in the JSON return result 'img'.

So this is getting the parameter information from within AWS but we want to specify it from our simple HTML page, so we need to configure API Gateway to pass the parameters in the HTTP GET method in the traditional kind of 'getImage?row=1&col=1' kind of formulation.

Navigate to the Lambda Triggers tab and click on the 'GET' method which will lead you to the API Gateway configuration.  Choose your API, click on Resources and choose your function to give you this view:














Now click on te Integration Request and scroll down and open out the Body Mapping Templates:


Add in a new Content-Type and paste in the template above which maps the input parameters to the event object.

This should now be ready to go, so a simple test in a browser first and then tweak your HTML code to have something like this:


    var response = $http.get('https://xxxxxx.execute-api.eu-west-1.amazonaws.com/prod/getImage2?row=2&col=2');

It's simple exercise now to play around with your JavaScript in the HTML to select the tile you want to get.

AWS Lambda - Getting Started

Since seeing the applications of Lambda at AWS Media London and the huge momentum behind it at AWS Summit London 2016 I thought it was time to get some of the basics down here as a jump-off reference along with some helpful links.

Getting started with Lambda is pretty easy and the benefits of 'serverless' infrastructure are quite self-evident (and if not are well described in numerous blogs and articles). AWS is the obvious starting reference place for kicking off:
There are a number of other tutorials of how to link up with other AWS products like S3, Kinesis and DynamoDB. I'm going to focus on using Node.js in using Lambda for now. There's a great online course available as well for getting started with Lambda offered by Udemy.

Beyond Hello World, most of the starting examples are derivatives of a thumbnail creation service using S3 from the AWS docs. The typical process is to drop a file onto S3, trigger a Lambda, process the image to scale it using ImageMagick and put the resulting file into a new S3 bucket. 


This is great to start with and there are a lot of good walk through tutorials explaining additional details and jumping off points. I wanted to collect together a couple of those references and take a look at some other ideas around this example and offer some suggestions for smoothing some of the details around getting started.

Following on pretty much directly from the AWS documentation, this tutorial steps smoothly through the same example. This article takes the same example and describes packaging and deployment in more detail using CloudFormation and the AWS CLI (command line interface). It's not quite time to delve into that detail yet, but it's a good reference to come back to later.

Well, this gets us started. Now what I wanted to explore is how to move beyond dropping files into S3 and creating new S3 files and to start using them with a simple web-page. 

AWS Lambda - Image Generator via API Gateway

Following on from the AWS Lambda - Getting Started post I wanted to continue exploring some of the starting blocks. In this article I'm going to look at continuing using AWS Lambda for working with images but instead stepping away from the well-trodden step of initiating the Lambda by dropping a file into an S3 bucket.

What I'd like to look at is calling a Lambda service via REST API and getting an image back. Now, you can think of any number of applications here, we're just going to look at some of the technology bits.

First step in moving away from initiating the Lambda from S3 is that we need some way to trigger the Lambda from a web-service call. AWS provides this via their API Gateway product and there is an accompanying tutorial in the reference Lambda documentation. API Gateway is pretty neat and relatively easy to configure if you want to send some JSON and return the result also in JSON. We're not going to explore that any further here. Now, say we want to create a service that just calls a Lambda and returns an image. Should be just the kind of thing for Lambda, right?

Now, this is where things get interesting. There are a number of blogs and articles talking about this with varying approaches trying to get API Gateway to somehow return an image/xxx MIME type. None of them seemed to work for me at the current time. This one is instructive on some failed approaches and a stackoverflow discussion worth reading as well.

So, it depends how you might want to use this. In my simple example case, which we'll step onto next I wanted to call from a basic web-page and get an image back from a Lambda rather than an S3 hosted file. It's probably fighting with all AWS is designed to do, but let's go with it for now.

For this example, the alternative seems to be to return the image (binary data) as Base64 encoded data in the JSON response. While a little more inefficient as we need to encode, decode and transport using more bytes it should be friendly with API Gateway and as we'll see in the next step also works pretty easily with simple HTML as well.

Getting started, this is how things turn out. I'll explain what's going on below:


var im = require("imagemagick");
var fs = require("fs");
var os = require('os');

var AWS = require('aws-sdk');
AWS.config.region = 'eu-west-1';

exports.handler = function(event, context) {
    
        var params = {Bucket: 'mybucket', Key: 'myimage.png'};
        var s3 = new AWS.S3();
        
        console.log("bucket", params);

        s3.getObject(params, function(err, data) {
            if (err) 
                console.log("ERROR", err, err.stack); // an error occurred
            else     
                console.log("SUCCESS", data);         // successful response
            
                var resizedFile = os.tmpdir() + '/' + Math.round(Date.now() * Math.random() * 10000);
            
                im.resize({
                    srcData: data.Body,
                    dstPath: resizedFile,
                    width: 100,
                    height: 100,
                    progressive: false
                }, function(err, stdout, stderr){
                    
                    console.log("resised", resizedFile); 
                    
                    if (err) {
                        context.fail(err);
                    }
                    else {
                        var resultImgBase64 = new Buffer(fs.readFileSync(resizedFile)).toString('base64');
                        context.succeed({ "img" : resultImgBase64 });
                    }
                });
                
        });

        // allow callback to complete 
};

This simple Lambda is using the AWS API in node and setting up the parameters to be able to read a file in a bucket. First thing to remember is that this file needs to be visible to Lambda for it to work, so either make this a publically available file (you need to set this for each file on S3) or set the roles and permissions via IAM for S3 and your Lambda appropriately.

The function then uses ImageMagick, which is kindly provided by AWS as an available package to do the resize. We're using the 'os' to enable the creation of a local file within the Lambda context. This file is then encoded in a buffer to Base64 and returned. None of this is hugely efficient, but it does the job.

This can all be dropped in to AWS Lambda as inline code and simply tested by calling the resulting URL given in the API Gateway setup via a browser something like this:


https://xxxxxxx.execute-api.eu-west-1.amazonaws.com/prod/getImage

Which if everything is working ok, should give a response something like this:


{"img":"iVBORw0KGgoAAAANSUhEUgAAAIkAAACJCAMAAAAv+uv7AAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAAAYBQTFRF////1eDpw9LgoLjNepy8Z4+yc5i4vc7dz9vm5+3zscXWQXGcUYCrU4WwToa4Uo3BVJDHUo3DU4WyUoKtq8DT4env2+TsmbPLW4atTHumUoa2W5vVUYe3Tn2oUX6nka3Hydfj8/b5m7XMTXqjUH+qToe6VpPKWJbOT4i7U4OvRXSegaLAT32mU4m6WJfQWprTUYvAjqzG7fL2+fv8WZjSUoq8TXymqL7SUH2nUX+pXoivVIi3V4SrVIGrVIKstMfZoLnOVYe1hKTCcpe4VYm5Vo2/ZI2yXIeuVo/EVZLIWYWtV5XMU4OtYYuxb5W3V5HGapG1nbfOWIu6VY7CeJu7wNDfVYaxVoy8ka7JVYi2rsPVUH6oTHqjSnijU4i5orrPbZO1U4/FVoKqt8naU4a0UYi5nrfNYYqvR3ahUIa4pbzQUIq9iafDSHehUoCqU4Cpa5K1WIStWoauV4WuxdXiWIeypb3RuszcVou6VI3AmbTLV4m3f6G/VoOtVYSuq8HUY1MLMAAAAAFiS0dEAIgFHUgAAAAJb0ZGcwAAAAcAAAAHAIhuELEAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAJdnBBZwAABegAAAL0AM6p+5YAAANMSURBVHja7ZoJXxJREMBXZREEBFy5fIqiHB7JKuCRiqZlZnSZSqlZppYn3ff51ds1K82i3ZlZrH7z/wL7/70383bezJMkhmEYhmEYhmEYhjFIVXWNTT7EXlPrOB0LZ53d5fbUe32H+OsbFJfcGKiwRjAUjjT5xQl8zS3R1raKacTaO+KJpPgNqc6u7h5nZTzO9KZFWdS+SH/Gao+s5pETf2Zg0FqX7NCwIQ99XQbOtscsE3GOjKaNeejkeseCFok48knjHgfBOz5hicg5xeDGHNmiyaksuUfs/AWzHjrTF6kTOjBzCSIixGye9h/gvFyAiWiBe4VSJXbVZKweU8nTbVD22nW4iJZCY2QHy9Q0RkT7L94gyqC5mzgRIeYXSEQWi1gRIW7dJhDJDKp4E6EQJNDCPIGISIbRIlVuChEhlpaxJmHESXKU9DAyfyYmaUS02qkfJZIdNlGQlEddQR21/QNUIkL47ZgluWO6JCnDKqKyrW0gFBF378FN1u5TmhTWwSLOCKWIEO4qqMnGJq3Jg4dQk54+WpPZLahJNEVrIraBJVNmlVhEeKphJtUeapOdXZjJ7g61yd4+zMTmpTaBhqzsozZJRWEmVKXJD3IlmAl5Egvh+tdNOsiqpP9nTdYL1CJqN8zk78liux//7eNA71+hBLWJ1wYzoa1idRIhmEkwTm3SUAszob1j6ESgly/qkIWmjiQ1NtOadLZCTagDpasNakJcF2AaF0OjlCaYvgVNj+0bccyQhTJ7cK025wrdoiDbj3StHFQjRyJsb6mPsDPt5SUaE2TDT4fmTEk/xs8zHAqFyegQWgQ/U9Ehmqs8Qc2ZdAozNLOm7AgyVNJPqeZvmWeom486TjdLDz7HVG/FRTIRSXrxEn7qL70iFJGk1+PADcoV50hFtA0aA/Wrk3nKrflKbO2NeZG3I5a8E3pXNBm36vsP9E8tDlj8aKpw8n36bI2HRsAeN9w/SHbJ4IGBEYKyYsgl6Qm3Wemh4zDgktoMA/viZl1ammfLLEfCHd2ohIdOoHFre9P7i0xS/U2RcMjS+DhBbMNWUnY6fd/L3Jxvr14p2esq8tDwZxy7rXKH65CSvF/jsOj4YBiGYRiGYRiGYRiGOSW+AE9BVtXctxSRAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDE2LTA4LTE1VDEwOjA3OjM1KzAwOjAwX7Lc1AAAACV0RVh0ZGF0ZTptb2RpZnkAMjAxNi0wOC0xNVQxMDowNzozNSswMDowMC7vZGgAAAAASUVORK5CYII="}

There's a useful github repo for someone trying the same idea with more involved code.

Next step we'll get this called from a basic web page.

AWS Lambda - Using Image function in a basic web-page

Following on from the previous posting, we're now going to try to use our basic Lambda function which provides a simple HTTP GET method with no parameters and returns JSON with an image in base64 encoding. We're going to make a simple web-page to call the function and display the image.

I'm going to use AngularJS as it just means we don't need to fight with basic HTML and scripting. If you're not familiar, then this or this should give a quick Hello World intro just for what is needed in this example.

First of all we need to think how to get a Base64 chunk of data to show as an image. Thankfully most browsers now support inline images with data URLs. More details can be found here. The format looks like this:

<img src="data:image/png;base64,iVBORw0KGgoAAAANS..." />


Which all seems quite easy. There are some discussions on stackoverflow of where to go from here to link up to JavaScript.

Taking our basic Angular app it's quite a simple matter to add in a button that pushes the base64 data we had previously into a text box and also into the image data:

<!DOCTYPE html>
<html ng-app="TestApp">
<head>
 <script type="text/javascript"
  src="http://ajax.googleapis.com/ajax/libs/angularjs/1.0.7/angular.min.js"></script>

</head>
<body>
 <div ng-controller="TestController">
        
        <h1>base64: {{data}}</h1>
    
        <img height="100" width="100" data-ng-src="data:image/png;base64,{{data}}" data-err-src="empty.png"/>

        <button ng-click="click()">GetImage</button>
 </div>
    
 <script>
        angular.module('TestApp', [])
            .controller('TestController', function($scope) {
            $scope.data = "empty";
            
            $scope.click = function() {
                $scope.data = "iVBORw0KGgoAAAANSUhEUgAAAIkAAACJCAMAAAAv+uv7AAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAgY0hSTQAAeiYAAICEAAD6AAAAgOgAAHUwAADqYAAAOpgAABdwnLpRPAAAAYBQTFRF////1eDpw9LgoLjNepy8Z4+yc5i4vc7dz9vm5+3zscXWQXGcUYCrU4WwToa4Uo3BVJDHUo3DU4WyUoKtq8DT4env2+TsmbPLW4atTHumUoa2W5vVUYe3Tn2oUX6nka3Hydfj8/b5m7XMTXqjUH+qToe6VpPKWJbOT4i7U4OvRXSegaLAT32mU4m6WJfQWprTUYvAjqzG7fL2+fv8WZjSUoq8TXymqL7SUH2nUX+pXoivVIi3V4SrVIGrVIKstMfZoLnOVYe1hKTCcpe4VYm5Vo2/ZI2yXIeuVo/EVZLIWYWtV5XMU4OtYYuxb5W3V5HGapG1nbfOWIu6VY7CeJu7wNDfVYaxVoy8ka7JVYi2rsPVUH6oTHqjSnijU4i5orrPbZO1U4/FVoKqt8naU4a0UYi5nrfNYYqvR3ahUIa4pbzQUIq9iafDSHehUoCqU4Cpa5K1WIStWoauV4WuxdXiWIeypb3RuszcVou6VI3AmbTLV4m3f6G/VoOtVYSuq8HUY1MLMAAAAAFiS0dEAIgFHUgAAAAJb0ZGcwAAAAcAAAAHAIhuELEAAAAJcEhZcwAADsMAAA7DAcdvqGQAAAAJdnBBZwAABegAAAL0AM6p+5YAAANMSURBVHja7ZoJXxJREMBXZREEBFy5fIqiHB7JKuCRiqZlZnSZSqlZppYn3ff51ds1K82i3ZlZrH7z/wL7/70383bezJMkhmEYhmEYhmEYhjFIVXWNTT7EXlPrOB0LZ53d5fbUe32H+OsbFJfcGKiwRjAUjjT5xQl8zS3R1raKacTaO+KJpPgNqc6u7h5nZTzO9KZFWdS+SH/Gao+s5pETf2Zg0FqX7NCwIQ99XQbOtscsE3GOjKaNeejkeseCFok48knjHgfBOz5hicg5xeDGHNmiyaksuUfs/AWzHjrTF6kTOjBzCSIixGye9h/gvFyAiWiBe4VSJXbVZKweU8nTbVD22nW4iJZCY2QHy9Q0RkT7L94gyqC5mzgRIeYXSEQWi1gRIW7dJhDJDKp4E6EQJNDCPIGISIbRIlVuChEhlpaxJmHESXKU9DAyfyYmaUS02qkfJZIdNlGQlEddQR21/QNUIkL47ZgluWO6JCnDKqKyrW0gFBF378FN1u5TmhTWwSLOCKWIEO4qqMnGJq3Jg4dQk54+WpPZLahJNEVrIraBJVNmlVhEeKphJtUeapOdXZjJ7g61yd4+zMTmpTaBhqzsozZJRWEmVKXJD3IlmAl5Egvh+tdNOsiqpP9nTdYL1CJqN8zk78liux//7eNA71+hBLWJ1wYzoa1idRIhmEkwTm3SUAszob1j6ESgly/qkIWmjiQ1NtOadLZCTagDpasNakJcF2AaF0OjlCaYvgVNj+0bccyQhTJ7cK025wrdoiDbj3StHFQjRyJsb6mPsDPt5SUaE2TDT4fmTEk/xs8zHAqFyegQWgQ/U9Ehmqs8Qc2ZdAozNLOm7AgyVNJPqeZvmWeom486TjdLDz7HVG/FRTIRSXrxEn7qL70iFJGk1+PADcoV50hFtA0aA/Wrk3nKrflKbO2NeZG3I5a8E3pXNBm36vsP9E8tDlj8aKpw8n36bI2HRsAeN9w/SHbJ4IGBEYKyYsgl6Qm3Wemh4zDgktoMA/viZl1ammfLLEfCHd2ohIdOoHFre9P7i0xS/U2RcMjS+DhBbMNWUnY6fd/L3Jxvr14p2esq8tDwZxy7rXKH65CSvF/jsOj4YBiGYRiGYRiGYRiGOSW+AE9BVtXctxSRAAAAJXRFWHRkYXRlOmNyZWF0ZQAyMDE2LTA4LTE1VDEwOjA3OjM1KzAwOjAwX7Lc1AAAACV0RVh0ZGF0ZTptb2RpZnkAMjAxNi0wOC0xNVQxMDowNzozNSswMDowMC7vZGgAAAAASUVORK5CYII=";
            };
        });
     </script>
</body>
</html>

Now to get it calling the Lambda service, the $scope.click function can simply be changed to make an http request and put the data into the image. Hey presto!


$scope.click = function() {
    
    var response = $http.get('https://xxxxxxx.execute-api.eu-west-1.amazonaws.com/prod/getImage2');

    response.success(function(data, status, headers, config) {

        $scope.data = data.img;
    });

    response.error(function(data, status, headers, config) {
        alert("Error.");
    });
};

Now, if this isn't working for you, and almost part of the point of this post in itself it means that you have some API Gateway config to get sorted. I must admit first time I went through this there was a bit of head bashing and using fiddler to check that everything was ok.

If everything worked ok up to the previous post and you're getting the base64 data coming back from a browser call then it's sure to be some fun with JavaScript. This is most likely that you have not got CORS setup for your API Gateway binding. Biggest clue here is that you only have a GET method defined in API Gateway and there is no OPTIONS method. AWS makes this pretty easy, so follow the online instructions to configure for CORS.

If you're still having problems and I was bashing my head at this for a while it seems (for me at least) that the default CORS settings from AWS were not working and I needed to manually add in X-Requested-With to the Access-Control-All-Headers  configuration. Remember you'll need to deploy the API again for the changes to take effect. Thanks to this stackoverflow discussion and this one as well for the help.

The other problem I bumped into on this first time round and you'll not have it on the example posted is that if you use the function context.fail(error); in your Lambda to return an error, this will give an http 403 response, not a 200 response, and this is not handled in the basic API Gateway bindings so looks like an error that the service is unavailable. Very confusing!

Thursday, 11 August 2016

65DOS No Man's Sky


Well finally the wait is over. Returning from holiday I was happy to find a package containing the album of tunes by 65DOS inspired by creating the generative soundtrack to the endless universe game No Man's Sky.



Listening to it on the commute it's living up to expectations and continues the journey on from Wild Light. There's been quite a bit of press before (Guardian) and on release Metro and Digital Trends on the approach and challenges to making music this way.

And here's Debutante, the track that kicked off the collaboration:




Thursday, 28 July 2016

Queues.io

I've been reading quite a bit about queuing technology recently as the concepts of event-based processing, enterprise design patterns and scaling backend cloud and integration technologies seem to be using common patterns to build solutions. Apart from the most well recognised ones such as Amazon SQS and RabbitMQ there are quite a lot of other projects to look at and build upon. I stumbled across the excellent queues.io site on my journey which provides an excellent aggregation of the various options and a great jumping off point to investigate further.

Sunday, 26 July 2015

WOMAD 2015: Hannah Peel

Absolutely loved this set.... beautiful voice and wonderfully acoustic music box:

Tainted Love @ WOMAD



Blue Monday