Robin Ricard

Generative Adversarial Networks Part 2 - Implementation with Keras 2.0

Wed, 05 Apr 2017 18:00:00 +0000

⌛ This article is quite old! This article dates back from when I was studying for my Master of Science in Machine Learning at Georgia Tech. Since then, I recentered my career away from the subject. While the subject here is interesting, I am clearly not an authoritative source on the subject and you should keep that in mind when looking at this article…

In this part, we’ll consider a very simple problem (but you can take and adapt this infrastructure to a more complex problem such as images just by changing the sample data function and the models). In our real sampled data, we’ll generate random sinusoid curves and we’ll try to make our GAN generate correct sinusoidal curves.

All of the code required to run the GAN is in this tutorial. All of the required imports are located at the end.

Getting the “real” data

We just need to randomly generate curves, this is pretty simple to do. You can readapt this function to sample from a dataset if you want to take images for instance.

Here is the data sampling function:

def sample_data(n_samples=10000, x_vals=np.arange(0, 5, .1), max_offset=100, mul_range=[1, 2]):
    vectors = []
    for i in range(n_samples):
        offset = np.random.random() * max_offset
        mul = mul_range[0] + np.random.random() * (mul_range[1] - mul_range[0])
        vectors.append(
            np.sin(offset + x_vals * mul) / 2 + .5
        )
    return np.array(vectors)

ax = pd.DataFrame(np.transpose(sample_data(5))).plot()

Create the models

As we saw in the first part, we need to create a model that we will train to generate images from some noise and a model supposed to detect real data from generated data.

Generative model

We create our generative model using simple dense layers activated by tanh. This model will take noise and try to generate sinusoidals from it. This model will not be trained directly but trained via the GAN.

def get_generative(G_in, dense_dim=200, out_dim=50, lr=1e-3):
    x = Dense(dense_dim)(G_in)
    x = Activation('tanh')(x)
    G_out = Dense(out_dim, activation='tanh')(x)
    G = Model(G_in, G_out)
    opt = SGD(lr=lr)
    G.compile(loss='binary_crossentropy', optimizer=opt)
    return G, G_out

G_in = Input(shape=[10])
G, G_out = get_generative(G_in)
G.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 10)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 200)               2200      
_________________________________________________________________
activation_1 (Activation)    (None, 200)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 50)                10050     
=================================================================
Total params: 12,250.0
Trainable params: 12,250.0
Non-trainable params: 0.0
_________________________________________________________________

Discriminative model

Similarly, we create our discriminative model that will define if the curve is real or outputted by the generative model. This one, though will be trained directly.

def get_discriminative(D_in, lr=1e-3, drate=.25, n_channels=50, conv_sz=5, leak=.2):
    x = Reshape((-1, 1))(D_in)
    x = Conv1D(n_channels, conv_sz, activation='relu')(x)
    x = Dropout(drate)(x)
    x = Flatten()(x)
    x = Dense(n_channels)(x)
    D_out = Dense(2, activation='sigmoid')(x)
    D = Model(D_in, D_out)
    dopt = Adam(lr=lr)
    D.compile(loss='binary_crossentropy', optimizer=dopt)
    return D, D_out

D_in = Input(shape=[50])
D, D_out = get_discriminative(D_in)
D.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         (None, 50)                0         
_________________________________________________________________
reshape_1 (Reshape)          (None, 50, 1)             0         
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 46, 50)            300       
_________________________________________________________________
dropout_1 (Dropout)          (None, 46, 50)            0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 2300)              0         
_________________________________________________________________
dense_3 (Dense)              (None, 50)                115050    
_________________________________________________________________
dense_4 (Dense)              (None, 2)                 102       
=================================================================
Total params: 115,452.0
Trainable params: 115,452.0
Non-trainable params: 0.0
_________________________________________________________________

Chained model: GAN

Finally, we chain the two models into a GAN that will serve to train the generator while we freeze the discriminator.

In order to freeze the weights of a given model, we create this freezing function that we will apply on the discriminative model each time we train the GAN, in order to train the generative model.

def set_trainability(model, trainable=False):
    model.trainable = trainable
    for layer in model.layers:
        layer.trainable = trainable
        
def make_gan(GAN_in, G, D):
    set_trainability(D, False)
    x = G(GAN_in)
    GAN_out = D(x)
    GAN = Model(GAN_in, GAN_out)
    GAN.compile(loss='binary_crossentropy', optimizer=G.optimizer)
    return GAN, GAN_out

GAN_in = Input([10])
GAN, GAN_out = make_gan(GAN_in, G, D)
GAN.summary()

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         (None, 10)                0         
_________________________________________________________________
model_1 (Model)              (None, 50)                12250     
_________________________________________________________________
model_2 (Model)              (None, 2)                 115452    
=================================================================
Total params: 127,702.0
Trainable params: 12,250.0
Non-trainable params: 115,452.0
_________________________________________________________________

Training

Now we did setup our models, we can train the models by alterning training the discriminator and the chained models.

Pre-training

Let’s now generate some fake and real data and pre-train the discriminator before starting the gan. This also let us check if our compiled models correclty runs on our real and noisy input.

def sample_data_and_gen(G, noise_dim=10, n_samples=10000):
    XT = sample_data(n_samples=n_samples)
    XN_noise = np.random.uniform(0, 1, size=[n_samples, noise_dim])
    XN = G.predict(XN_noise)
    X = np.concatenate((XT, XN))
    y = np.zeros((2*n_samples, 2))
    y[:n_samples, 1] = 1
    y[n_samples:, 0] = 1
    return X, y

def pretrain(G, D, noise_dim=10, n_samples=10000, batch_size=32):
    X, y = sample_data_and_gen(G, n_samples=n_samples, noise_dim=noise_dim)
    set_trainability(D, True)
    D.fit(X, y, epochs=1, batch_size=batch_size)

pretrain(G, D)

Epoch 1/1
20000/20000 [==============================] - 3s - loss: 0.0078     

Alternating training steps

We can now train our GAN by alternating the training of the Discriminator and the training of the chained GAN model with Discriminator’s weights freezed.

def sample_noise(G, noise_dim=10, n_samples=10000):
    X = np.random.uniform(0, 1, size=[n_samples, noise_dim])
    y = np.zeros((n_samples, 2))
    y[:, 1] = 1
    return X, y

def train(GAN, G, D, epochs=500, n_samples=10000, noise_dim=10, batch_size=32, verbose=False, v_freq=50):
    d_loss = []
    g_loss = []
    e_range = range(epochs)
    if verbose:
        e_range = tqdm(e_range)
    for epoch in e_range:
        X, y = sample_data_and_gen(G, n_samples=n_samples, noise_dim=noise_dim)
        set_trainability(D, True)
        d_loss.append(D.train_on_batch(X, y))
        
        X, y = sample_noise(G, n_samples=n_samples, noise_dim=noise_dim)
        set_trainability(D, False)
        g_loss.append(GAN.train_on_batch(X, y))
        if verbose and (epoch + 1) % v_freq == 0:
            print("Epoch #{}: Generative Loss: {}, Discriminative Loss: {}".format(epoch + 1, g_loss[-1], d_loss[-1]))
    return d_loss, g_loss

d_loss, g_loss = train(GAN, G, D, verbose=True)

Epoch #50: Generative Loss: 4.795285224914551, Discriminative Loss: 0.49175772070884705
Epoch #100: Generative Loss: 3.5881073474884033, Discriminative Loss: 0.05777553841471672
Epoch #150: Generative Loss: 3.3537204265594482, Discriminative Loss: 0.23116469383239746
Epoch #200: Generative Loss: 3.221881151199341, Discriminative Loss: 0.11503896117210388
Epoch #250: Generative Loss: 3.9224398136138916, Discriminative Loss: 0.06002074107527733
Epoch #300: Generative Loss: 3.3472070693969727, Discriminative Loss: 0.08401770144701004
Epoch #350: Generative Loss: 3.591559648513794, Discriminative Loss: 0.04007931426167488
Epoch #400: Generative Loss: 3.796259880065918, Discriminative Loss: 0.03941405192017555
Epoch #450: Generative Loss: 4.634820938110352, Discriminative Loss: 0.022133274003863335
Epoch #500: Generative Loss: 3.2044131755828857, Discriminative Loss: 0.05611502751708031

ax = pd.DataFrame(
    {
        'Generative Loss': g_loss,
        'Discriminative Loss': d_loss,
    }
).plot(title='Training loss', logy=True)
ax.set_xlabel("Epochs")
ax.set_ylabel("Loss")

What is interesting here is that both sides are playing with each other, each learning how to get better than the other. We see that during some phases, either the generator or the discriminator is lowering its loss relative to a loss gain on the other side.

Watching the loss of the models does not seem to have a huge importance to quantify the progress in quality of the models. We actually don’t really want to have models converging to zero too fast, otherwise, that means that they managed to trick each other. However the amount of times each model got better relatively to the other may be an interesting metric. to watch.

Results

We can now vizualize some generated sinusoidal curves from our generator:

N_VIEWED_SAMPLES = 2
data_and_gen, _ = sample_data_and_gen(G, n_samples=N_VIEWED_SAMPLES)
pd.DataFrame(np.transpose(data_and_gen[N_VIEWED_SAMPLES:])).plot()

This is not perfect, however, we see that if we smooth a bit the curve using a rolling mean, we approach a more precise sinusoidal shape.

N_VIEWED_SAMPLES = 2
data_and_gen, _ = sample_data_and_gen(G, n_samples=N_VIEWED_SAMPLES)
pd.DataFrame(np.transpose(data_and_gen[N_VIEWED_SAMPLES:])).rolling(5).mean()[5:].plot()

Obviously with more training and more tuning, we could get better results. A greater amount of epochs usually prooves to be efficient on a GAN, but it would be long to run in a simple tutorial like this one.

Finally, we see that the generator does not takes many risks and will more or less output the same curve for different noise input. This is to be expected, our very simple model will prefer refining something it already knows to do instead of trying something new. This is a risk with GANs that can be mitigated with more varied data but will require more training overall (and a good GPU if you want this training to finish one day…).

Conclusion

Creating GANs on keras is not a really hard task technically since all you have to do is create those 3 models and define how to do a training step with them but depending on the task you may want to achieve, more or less tuning and computations will be required.

GANs are overall very powerful but may be very hard to tune correctly especially since measuring how well they do is hard. The loss is not necessarily a good indicator and for now, the most reliable way to check if their output makes sense is to put the output in front of an actual person.

References

MNIST Generative Adversarial Model in Keras [Code]

Imports

%matplotlib inline
import os
import random
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from tqdm import tqdm_notebook as tqdm
from keras.models import Model
from keras.layers import Input, Reshape
from keras.layers.core import Dense, Activation, Dropout, Flatten
from keras.layers.normalization import BatchNormalization
from keras.layers.convolutional import UpSampling1D, Conv1D
from keras.layers.advanced_activations import LeakyReLU
from keras.optimizers import Adam, SGD
from keras.callbacks import TensorBoard

Generative Adversarial Networks Part 1 - Understanding GANs

Wed, 05 Apr 2017 18:00:00 +0000

⌛ This article is quite old! This article dates back from when I was studying for my Master of Science in Machine Learning at Georgia Tech. Since then, I recentered my career away from the subject. While the subject here is interesting, I am clearly not an authoritative source on the subject and you should keep that in mind when looking at this article…

I don’t talk much about machine learning on this blog in general, having pretty much focused on web-related software engineering lately, but I do study this field at Georgia Tech. Recently, a professor of mine (Dr. Andrew Gardner) made us focus on GANs, a kind of model that I’d like to present to you today. I assume you have a basic understanding of neural networks and know a bit about Tensorflow and Keras. If you’ve only played with something like MNIST, I think this tutorial will be accessible enough for you to understand. Now that you’ve been warned, let’s get to it!

Generative Adversarial Networks or GANs is a recent unsupervised technique (introduced in 2014 by Ian Goodfellow et al.) to train neural networks to generate plausible data using a zero sum game. Hopefully, this is not as complicated as it sounds.

In this blog post I’m going to explain how GANs work in general and which applications they commonly have and in a next blogpost, I’m going to show you how we can use Keras to train such model.

Why GANs ?

Generating data can be used to perform various tasks. The first thing that comes to mind is to be able to augment any dataset used for any kind of task with plausible additional data: a popular use of GANs is to feed them the imagenet dataset and make it generate plausible pictures that do not exist in the original dataset. From a human standpoint, those images may be a bit irrealistic but they may still make sense.

Sample of original data	Some generated images

An example of generated images presented by the OpenAI team.

By looking at the generated output, we can see that the GAN managed to represent a certain understanding of our world and is able, from this understanding, to create new images that could fit in our own vision of the world.

GANs can also be used for many other interesting tasks that may seem unrelated. For instance, GANs are able to upscale an image in a plausible way, inferring details that do not exist in the original. GANs are also able to infer the next frames in videos in a realistic way. But my favorite application may be this one where they managed to transform simplistic drawings into photorealistic sceneries. Finally, GANs seems to be researched extensively by OpenAI with the goal of being able to generate seemlingly real situations for Reinforcment Learning experiments.

Model Overview

The overall idea behind GANs is that you have two models playing with each other. As Ian Goodfellow describes it: one is a counterfeiter trying to produce seemingly real data while the other is a cop trying to determine what the fake counterfeit data is while trying to not raise false positives on real data.

The Generative Model (aka The counterfeiter), by taking any random input will try to generate some real things (curves, images, sounds, text, …). It has no idea of what is the real data, it wll only try to adjust from the feedback of the other model.
The Discriminative Model (aka The cop), will take as its dataset input a bunch of generated data from the other model and actual real data we want the other model to learn to fake. The output results of this model will serve for the backpropagation of the other model.

An overview of GANs: the results of the discriminator model will serve to retrain the generator during the backpropagation.

How to train a GAN?

How do we train this efficiently though? We’ll see in the next part how to actually train the model with Keras but in general, training a GAN is very simple: for each epoch, you only need to do a two-step training.

1. Train the discriminator	2. Train the chained GAN

Train the discriminator Sample some images from your dataset and some noise that you will pipe through the generator model. Then use this data to train the Discriminator to recognize Generator data from Real data.
Train the generator via the chained models In this part, you will generate sample data and try to push the chained generator and discriminator to tell you that it is real data. However, we don’t want to alter the weights in the discriminator during this step. That’s why we’ll need to freeze the training of the weights in the discriminator.

What are their limits? How do they compare with similar models?

Let’s start with the main issue with GANs, it is particularly hard to quantify how good their output is: first thing, they run unsupervised so theorically the loss would be a good metric to watch, however, since GANs are in fact two models fighting with each other, they will both try to have the lowest loss while augmenting the other model loss. They also are exposed to the risk of generating very similar results for different noise input. Keep in mind that there is very good research that has been made on this subject to mitigate those issues but that goes way out of the scope of this introductory tutorial. If you are interested in learning more about this, I advise you to follow the work made recently by the University of Montréal and OpenAI (where the inventors of GANs are primarily working). I especially recommend Improved techniques for training gans.

GANs are very good for their generative abilities but are also used in tasks where other concurrent models are very strong such as latent modeling (semi-supervized approach, this time we would not feed noise into the GAN) in which Variational Autoencoders (VAEs) are already very good at.

VAEs are good at reducing dimensions into interpretable dimensions while GANs, as said before are harder to evaluate. However, GANs will, compared to VAEs, generate less “blurry” output.

There are also a lot of other models that I will not cover here since they are more specialized than GANs in specific fields (such as pixelCNN for images) that may work better. That’s why that, depending on the task, it may be interesting to combine GANs with an another model (Adversarial VAEs for instance) to get the most out of a specific situation.

Conclusion

I’ll conclude this first part by saying that GANs are a good fundamental start and, while not perfect, they can be used and combined to create very interesting models. The research goes fast and I would advise you to keep an eye out if you are interested in that. I really think there is still a lot to be done with GANs that we did not really imagine just yet! I’ll stop there since I’m also interested in showing you how to create your own GANs in the next part.

References

🔁 Apollo GraphQL Dev Blog: An enhanced GraphQL developer experience with TypeScript

Thu, 08 Dec 2016 21:00:00 +0000

⌛ This article is quite old! This article dates back from when I was still studying computer science in university. I was back then quite enthusiastic about a lot of things and this article reflects that well! It may be out of date with my current way of thinking and working as I discovered different constraints in software development. Just keep that in mind and enjoy!

This story was posted on Apollo GraphQL’s Dev Blog.

ES6 on AWS Lambda

Sun, 29 Nov 2015 21:00:00 +0000

⌛ This article is quite old! This article dates back from when I was still studying computer science in university. I was back then quite enthusiastic about a lot of things and this article reflects that well! It may be out of date with my current way of thinking and working as I discovered different constraints in software development. Just keep that in mind and enjoy!

AWS Lambda is a very cool and simple service able to run “cloud functions” very fast and cost-effectively. It is particularly interesting if you want to process events that spawn into your AWS infrastructure or scaffold a small API very fast.

Originally, Lambda was based on Node.js. Now lambda covers more languages with Python 2.7 and Java 8. Unfortunately, Lambda doesn’t seem to have a really up-to-date Node.js engine preventing us from using Node’s ES6 capabilities.

With Babel and Webpack, we’ll be able to have all of the ES6 features running on Lambda.

TL;DR; Check out the example repository to run lambdas with ES6.

Setup webpack & babel

Initialize and dependencies

First thing we need to do is setup a basic npm package.json. It’s as simple as running npm init.

Then, we’ll need to import webpack, babel and its presets and plugins.

npm i --save-dev webpack babel-core babel-loader json-loader \
  babel-preset-es2015 babel-plugin-transform-flow-strip-types \
  babel-plugin-syntax-flow

Note that in this example I import the webpack’s json-loader to allow requires on JSON files. I also import the Babel flowtype plugins to allow the use of flow type-checking annotations.

Configure webpack

Configuring webpack is trickiest part of this tutorial but once explained, there’s no magic needed, don’t worry!

Let’s start with a basic webpack.config.js:

// webpack.config.js

var path = require("path");

module.exports = {

};

Module Loaders

I usually start my webpack config with the module loaders:

module: {
  loaders: [
    {
      test: /\.js$/,
      exclude: /node_modules/,
      loader: 'babel',
      query: {
        presets: ['es2015'],
        plugins: ['syntax-flow', 'transform-flow-strip-types']
      }
    },
    {
      test: /\.json$/,
      loader: 'json'
    }
  ]
}

Ok, if you don’t know webpack, nothing really complicated here: each js file imported will be transformed through babel with the es2015 preset while stripping the flow annotations. The json files will be imported like Node would do it.

Output

Then, we can define how and where we want webpack to output my files:

// webpack.config.js

var path = require("path");

module.exports = {
  output: {
    path: path.join(__dirname, "dist"),
    library: "[name]",
    libraryTarget: "commonjs2",
    filename: "[name].js"
  },
  target: "node",
  module: { loaders: [ /* ... */ ] }
};

Here are some changes between the usual output you configure when you do a web project and what we intend to do here:

We’ll have multiple output files (one by lambda), so we need to use the [name] placeholder in our config.
We’ll need to expose those files as correct node modules, that’s why library and libraryTarget are essential.
We also need webpack to ignore native node deps such as fs or http, that’s why we use the node target.

Don’t forget to add the dist/ directory in your .gitignore!

Entry points

Since we’ll have an undefined number of lambdas, we want to list a directory to use all of the files in this dir as entry points.

To that end, we need to read the directory synchronously and transform the result to something that looks like this: {lambda1: "./lambdas/lambda1.js", lambda2: "./lambdas/lambda2.js"}.

// webpack.config.js

var path = require("path");
var fs = require("fs");

module.exports = {
  entry: entry: fs.readdirSync(path.join(__dirname, "./lambdas"))
         .filter(filename => /\.js$/.test(filename))
         .map(filename => {
            var entry = {};
            entry[filename.replace(".js", "")] = path.join(
             __dirname,
             "./lambdas/",
             filename
           );
           return entry;
         })
         .reduce((finalObject, entry) => Object.assign(finalObject, entry), {}),
  output: { /* ... */ },
  target: "node",
  module: { loaders: [ /* ... */ ] }
};

Note that the precedent snippet uses arrow functions & Object.assign() so you’ll need to use a fairly recent node version to develop locally.

Try the compilation

Now, we can test a simple lambda function locally to test if webpack does its job correctly.

You can now try the compilation with a simple lambda: hello.js

// lambdas/hello.js

/* @flow */

type HelloOptions = {
  name: string
};

export function hello(options: HelloOptions, context: any): void {
  context.succeed(`Hello ${options.name || "world"}!`);
}

As you can see, this file uses ES6 & flow heavily.

We’ll now be able to start a small script to ensure if we can execute our lambda: try-hello.js

// bin/try-hello.js

var helloModule = require("../dist/hello.js");

var fakeLambdaContext = {
  succeed: function succeed(results) {
    console.log(results);
    process.exit(0);
  }
};

helloModule.hello({name: "bob"}, fakeLambdaContext);

You can now run webpack and try to run the script:

./node_modules/.bin/webpack
node ./bin/try-hello.js

Try on Lambda

You can also use the code generated in dist/ directly on Lambda.

Copy the contents of dist/hello.js and paste it in a new lambda function.

Don’t forget to use the Node.js runtime and to point your handler on index.hello instead of index.handler.

Now you can test it!

Note that you can’t rely on any compiled dependency. We may see how to solve this in an another post.

Better local development

Now that we have everything in place to compile our lambdas, we’ll try to make things much easier for ourselves.

Continuous webpack compilation

Each webpack run takes a lot of time. Fortunately, webpack becomes quite fast once watching your project.

To get an easy shortcut to the webpack watcher, I advise you to add a script in your package.json.

"scripts": {
  "start": "webpack --watch",
  "test": "exit 1"
},

That way you just need to start it with npm start.

Fast-exec script

Instead of creating a file to inject our options and context inside of the imported Lambda, I created a small script that can be ran from your CLI.

You can place it in bin/run.js:

// bin/run.js

const args = process.argv;
const requiredLambda = args[2].split(".");
const lambdaModule = require(`../dist/${requiredLambda[0]}.js`);
const lambda = lambdaModule[requiredLambda[1]];
const options = JSON.parse(args[3]);

function succeed(result) {
  console.log(result);
  process.exit(0);
}

function fail(error) {
  console.error(error);
  process.exit(1);
}

const context = {
  succeed: succeed,
  fail: fail,
  done: (err, res) => err ? fail(err) : succeed(res),
  getRemainingTimeInMillis: () => Infinity,
  functionName: "fakeLambda",
  functionVersion: "0",
  invokedFunctionArn: "arn:aws:lambda:fake-region:fake-acc:function:fakeLambda",
  memoryLimitInMB: Infinity,
  awsRequestId: "fakeRequest",
  logGroupName: "fakeGroup",
  logStreamName: "fakeStream",
  identity: null,
  clientContext: null
};

lambda(options, context);

Note again that you’ll need a recent Node.js engine for local development.

This script can be executed like this:

# node bin/run.js $lambdaFile.$lambdaFunction $lambdaOptions
node bin/run.js hello.hello '{"name": "cli"}'

Testing & Reusability

What’s cool about lambdas is that they’re really easy to test. It’s just inputs and outputs. Of course testing a lambda needs to setup a context.

Hopefully, we already kind of did that with the fast-exec script. All we need is to refactor a little bit to have everything usable in mocha. More than that, we could create a Promise wrapper to transform our lambdas into reusable Promises in other projects.

Install mocha

We need to install mocha: npm i --save-dev mocha.

Mocha will need to register itself to Babel. Just insert the right script into the package.json.

{
  "scripts": {
    // ...
    "test": "mocha --compilers js:babel-core/register --recursive tests/"
  }
}

At this point we’ll need a .babelrc to load our presets and plugins. This rc could be used by webpack to load its configuration.

// .babelrc
{
  "presets": ["es2015"],
  "plugins": ["syntax-flow", "transform-flow-strip-types"]
}

To use the same rc in webpack, just change the query in the module loader:

{
  test: /\.js$/,
  exclude: /node_modules/,
  loader: 'babel',
  query: JSON.parse(
    fs.readFileSync(path.join(__dirname, ".babelrc"), {encoding: "utf8"})
  )
},

Promise-wrapper

Amazon made it easy to start with Lambda but it is actually not really standard so the lambda you can create here are not really reusable elsewhere.

Since we are going to test them later, it may be interesting to use a standard asynchronous API: the Promise.

We only need to reuse the fake context created earlier and put it in a promisifying function.

// lib/lambda-promisifier.js

/* @flow */

import type {LambdaContext} from "./lambda-types.js";

export function lambdaPromisifier(
  lambda: (options: any, context: LambdaContext) => void
): (options: any) => Promise<any> {
  return (options: any) =>
    new Promise((resolve, reject) =>
      lambda(options, {
        succeed: resolve,
        fail: reject,
        done: (err, res) => err ? reject(err) : resolve(res),
        getRemainingTimeInMillis: () => Infinity,
        functionName: "fakeLambda",
        functionVersion: "0",
        invokedFunctionArn: "arn:aws:lambda:fake-region:fake-acc:function:fakeLambda",
        memoryLimitInMB: Infinity,
        awsRequestId: "fakeRequest",
        logGroupName: "fakeGroup",
        logStreamName: "fakeStream",
        identity: null,
        clientContext: null
      })
    );
};

If you use flow, you will find the LambdaContext type here.

Tests

Finally, we can now easily test our lambda with mocha:

// tests/hello.js

import assert from "assert";

import {lambdaPromisifier} from "../lib/lambda-promisifier.js"
import {hello} from "../lambdas/hello.js"

const promisifiedHello = lambdaPromisifier(hello);

describe("hello lambda", function() {
  it("should greet the world by default", function(done) {
    promisifiedHello({})
    .then(res => {
      assert.equal(res, "Hello world!")
    })
    .then(() => done(), done);
  });

  it("should greet someone when precised", function(done) {
    promisifiedHello({name: "someone"})
    .then(res => {
      assert.equal(res, "Hello someone!")
    })
    .then(() => done(), done);
  });
});

You can now fire up mocha via npm: npm t.

An another example: HTTP call with fetch

We’ll see here how to do something really simple but that requires using external modules: getting a webpage somewhere in the internet with the fetch standard.

For that, we’ll need node-fetch first: npm i --save node-fetch

Now we can implement the lambda really easily:

// lambdas/fetch.js

/* @flow */

import fetch from "node-fetch";

import type {LambdaContext} from "../lib/lambda-types.js";

type FetchOptions = {
  url: string,
  method?: string,
  headers?: {[key: string]: string},
  body?: string
};

export function handler({
  url,
  method,
  headers,
  body
}: FetchOptions, context: LambdaContext): void {
  fetch(url, {
    method: method || "GET",
    headers: headers || {},
    body: body
  })
  .then(res => res.text())
  .then(context.succeed, context.fail);
}

And a small test:

// tests/fetch.js

/* @flow */

import assert from "assert";

import {lambdaPromisifier} from "../lib/lambda-promisifier.js"
import {handler} from "../lambdas/fetch.js"

const promisifiedFetch = lambdaPromisifier(handler);

describe("fetch lambda", function() {
  it("should be able to make an http call", function(done) {
    promisifiedFetch({url: "http://www.rricard.me"})
    .then(res => {
      assert(/Posts/.test(res));
    })
    .then(() => done(), done);
  });
});

You can test it on AWS now, you’ll see that the webpack output gets quite large but it does take all of the dependencies in account at once.

Conclusion

Making ES6 and Lambda work together is not that hard but having proper tooling to develop your ES6 lambdas easily require some knowledge.

I hope that this small tutorial helped you develop lambdas faster and/or understand better the webpack compilation process.

Later I’ll see if I can give you good tooling to export lambdas with compiled dependencies and how to deploy them more easily directly on AWS (without the copy/paste process).

Discuss on HN

Isomorphism, React and client platforms

Wed, 28 Jan 2015 18:34:00 +0000

⌛ This article is quite old! This article dates back from when I was still studying computer science in university. I was back then quite enthusiastic about a lot of things and this article reflects that well! It may be out of date with my current way of thinking and working as I discovered different constraints in software development. Just keep that in mind and enjoy!

This article could feel like I’m just writing on a trend. That’s not true, I was thinking about it for a while now and the React Native Announcement today just gave me a good excuse to write this !

There is already a batch of good posts about Isomorphism out there (such as this one from Airbnb or this one from a Mozillian where he explains how he implemented his isomorphic blog) so I won’t spend much time explaining it. An isomorphic app uses the same code to render views on the server and the client. React makes it easier and better by being able to just change what was updated on the client side without re-computing the whole DOM. It is mostly costless to add the rendering on the server-side, React playing really well with Node. This is a good way to make mobile web apps start faster on bad cellular networks for instance.

But as far as we know, “mobile web apps” still have a pretty bad UX in general. The future is promising but we’re not quite there yet unfortunately ! That’s why Facebook started working on React Native (for iOS and Android and I hope they will be releasing it soon). As they said the whole thing is not about “Write once, run anywhere”, it’s about “Learn once, write anywhere”. That doesn’t mean that we can’t share things between our modules.

But let’s stop quoting other articles and let’s explain what I have in mind.

The isomorphic app, generalized

Here are some boxes and some arrows:

So, what’s new in this ?

I’m a big fan of loosely coupled modules and that’s what I want to develop here. A classic isomorphic app would only take the Frontend Server and client parts to render the DOM. The frontend server would directly query the database.

One thing that companies like Twitter are doing for a long time now is to separate frontend server code and the “logic”, their backend. So I added this to my schema. You can see now that the Frontend server can be a generic brick binding the data from the API to the React views. That’s exactly what I want to work on soon !

So what’s shared at this point ? What’s reusable ?

The React components between the frontend server and the client
The simple utility functions: pure, that can be shared by the backend and the frontend

Wait… It’s not loosely coupled anymore !

It’s still loosely coupled ! We just reuse some code but each module keeps it’s concern and what it has to do:

The backend provides an API (and that’s even better if your backend is split in small modules interacting with each other)
The frontend server just binds views and data (but for that, it needs to get access to the components !)
The frontend client does the exact same thing but is able to handle user input at the same time

Now, what about the Mobile Client ?

That’s the interesting part, but we don’t know yet how it will be precisely implemented since Facebook has not released React Native for now. All we know is that it will bring the ability to write Native apps with JS and React. That’s great news ! Writing apps in JS on the native platform is not new. But JS alone doesn’t make it easier to write applications. React does. That’s why it’s a great announcement.

That doesn’t mean we’ll be able to share the same components between the platforms but we can share the same logic between clients this time. (ie. the Flux pattern and how to interact with the backend…)

I don’t know if this structure is the best way to do things for now, it has to be tested but it just feels natural to me.

The implementation

How can we do that ? That’s an interesting point !

We need a common language between all modules.

The natural response to this would be to use Node.js on every server obviously.

First of all, it’s not mandatory to share code between the backend and the frontend so the backend could be written in whatever language you like (you don’t have to change it at all !).

Here’s what I would do and on which I’m starting to work on:

The API backend would be running on the JVM written in Clojure.
The Frontend server could be running on Node.js with the code written in Clojurescript using om. (Why not ? I’m working on a ring-like API on top of core.async)
The Client frontend would be written in Clojurescript too and use om.
Same thing on mobile: cljs+om ! Imagine having REPLs, figwheel and everything cool about clojurescript development while working on a native mobile app ! To achieve that we may need a little bit of work but that’s the exciting part, isn’t it ?
When no side effects are needed, some functions can be shared between cljs and clj with cljx.

With this stack, everything could be written in the same language, keep the performance of the JVM, keep the isomorphism with the Node runtime and share code between all the frontends while having different UI implementations.

Most of the required things to make it work are already there ! All we need is a little bit of magic glue and it’s done !

I’ll keep you updated on my research on this. If I find out it’s not efficient or stupid, I’ll tell you, don’t worry ! I’ll also show you some code if I have something cool working !

If what I just wrote feels a bit fuzzy or if I made a major spelling mistake (that happens quite often), please open an issue or a PR for this blog.

Discussion on Hacker News

Getting to know Clojure

Wed, 31 Dec 2014 16:51:00 +0000

⌛ This article is quite old! This article dates back from when I was still studying computer science in university. I was back then quite enthusiastic about a lot of things and this article reflects that well! It may be out of date with my current way of thinking and working as I discovered different constraints in software development. Just keep that in mind and enjoy!

Hi hello ! First of all let me introduce all of this: this is a blog, yeah. This is exactly my third one, the other ones are cut off the Internet for the following reason: I was ashamed of them. Didn’t talked that much about things that I like but stuff I thought it was good to have in a blog. As a result, it was not good material.

So this one is my third and last take at a blog. If this one fail, I’m not tring again ! So I’m only going to speak about what I want to speak about !

So I want to speak about clojure today ! 2009 was my year of PHP (a dark year). 2010 was the year of the Python enlightenment. 2011 I discovered Rails and then Ruby. 2012 was the year of the webdev with the killer couple: Rails & jQuery. 2013 was the year of JS as a go-to everything language with Node.js. 2014 was the year of doubts, I tried many things such as functionnal programming in JS or Python, golang, struggling with Java (had to: Android) and many other stuff. Always feeling not good about a few points when experimenting and toying with those (except Java where everything was quite wrong). I knew clojure was really loved by it’s community and was quite curious about this. But didn’t had much time to learn it. So yeah, I decided it: 2015 would be the year of clojure for me !

TL;DR Clojure is awesome, you should definitely take a few days (5/6 days) to learn it. That’s enough because it’s awesomely minimal too !

The path to enlightenment

First things first, I won’t explain again why clojure is awesome. Search on Google or HN, there are many articles saying that clojure is awesome and I personally never saw anyone complaining about it online. I don’t see often languages that people don’t complain about. One reason could be that LISPs never made it to the mainstream but I’m quite sure that there are really few points on which clojure can be criticized on (except from being a LISP, which in my opinion is a good thing).

The only thing I will say here is: Clojure is insanely simple to learn ! Not easy, simple. And I would like to introduce to you how you can learn it too. And get to like it !

Day 1: Struggle

Clojure is not something you know right there, out of the box. It’s not like picking up another imperative/OO language and start a project just like that. Nope you’ll need to get used to it. I never said it is easy but you’ll see the reward later. Everything becomes effectively easy then.

First of all, and it is essential, you must get the right tooling before even starting: the lein repl is ok but you better get the repl into your text editor. Often clojurists use emacs. Don’t go away ! I personally don’t like using emacs. So like other clojurists out there I use vim-fireplace but if you’re more of an IDE person, just use counterclockwise or cursive. As long as you can fire repl commands from your editor, it’s okay !

At this point I recommend you to start with a brief tutorial even if you don’t understand everything there. It’s better than diving directly into a book. I recommend the excellent Clojure Doc’s Introduction for that.

Now you can rest or do something else ! Let your brain go past the first confusing things you encountered.

Day 2: Understand

Ok, now you must be curious. Manipulating immutable data must feel weird for now. That’s ok, you’ll get used to it. At this point you should print a cheat sheet, that’ll help later !

Right now you don’t really know what clojure is all about. That’s why you should directly try to do fun stuff with it instead of diving into macros or other stuff.

Just try to mount a simple ring server that writes back the path you inputted in your browser. After that, try some routing with compojure. Without really knowing it, you already made an highly concurrent webserver. Then you can easily play with some templating engines or try to connect to an SQL database with jdbc. One other thing to do is to try the java interop by creating a frame and drawing inside it from the repl. Take a look at the toolbox to know what the community already did for you !

I won’t give you more clues, you have to learn how to find things by yourself but don’t worry there’s a lot of material available online.

Days 3-4: Start a project

At this point you should start writing something, a project. Cool or not, already existing or not… It doesn’t matter ! Do whatever you want even if it’s reinventing the wheel ! I personally implemented a basic Paxos consensus system (I like networking stuff !). But do whatever you like !

It’s ok if you don’t know everything. You’ll figure out later that if you can do something with basic clojure, you better stick with it ! Only use complex stuff when needed ! (In case of interop or in desperate need for a state). But try to stick with the most basic clojure you know when possible (the more you rely on pure functions, the best it is).

Last thing here: write your project by using intensively tests and the repl. They are your best tools. If you don’t, clojure will feel painful to you !

Days 5-6: Continue your project and start reading a book

Everything’s in the title ! I’m just there and I feel great right now, I learn new things but I know how to do many things, and it only took me a small week. Reading books and continuing to learn is important. I may have understood 90% of the language, the essentials, but the last 10% may take much time (as they are not essential in every project).

If you want a word of advice, The Joy of Clojure is a great book to work with. It won’t spend much time learning you the basics of clojure (you already did !) and will bring you directly to the new parts.

Have fun !

Really, if you start not liking clojure after the rough first days, you might not be in the right place and that’s ok. It must feel fun, because it is. It must feel predictable, because it is. And at the end you’ll start doing less and less mistakes, the repl being your strongest ally there !

I hope I motivated you learning clojure ! If not, try golang like everyone else ! (Without kidding, golang must be an awesome language, it just never worked really well with me.).

Anyway, be prepared to hear more about clojure soon I hope !