/home/jpettit coder, sys admin, overall geek
Showing posts tagged with node.js

in Getting Started, node.js, promises, callbacks

Coming from Python, Node.js quickly blew my mind with the whole concept of asychronous stuff. I don't think I actually understood managing things asynchronously for a few months after I started writing Javascript (lets be honest, I probably still don't completely understand it). While the wound is still fresh, I figured I would take some time to write a basic guide for navigating callbacks and promises in Node.js.

Why

If you've come from something like Python or other popular languages threading has likely been something that was either done automatically by a module or something you've had complete control over. In Node threading doesn't really exist, you have one thread so good luck and have fun!

With >1 threads you can do several things (perhaps things that are even blocking) at the same time without impacting the application as a whole (that is, other requests can still be served). With a single thread, a blocking action will block the application as a whole - if you make a synchronous call other requests will need to wait for that task to finish. This is clearly undesirable for things like web applications. What good is an application that can only serve one request at a time? Node addresses this with callbacks, and more recently promises.

Callbacks

A callback is a function that is executed after a task is completed. It usually looks something like this:

asynchronousFunction(argument, function(response) {
	// When the function is finished doing whatever it is doing, this stuff is executed
    // The callback takes any return values as arguments, so if asynchronousFunction returned 1+1, response would be 2
    console.log("asynchronousFunction is done with the response", response);
});

You can also use a callback that is defined elsewhere like this:

asynchronousFunction(argument, callbackFunction);

function callbackFunction(response) {
	// This is executed when the function is done doing its stuff
    console.log("asynchronousFunction is done!");
}

In general, callbacks will first take an error argument and then other arguments that are returned by the function that is utilizing the callback. You can test to see if the error object exists after each call to see if the call was successful:

asynchronousFunction(argument, function(error, response) {
	if (error) { 
    	console.log("Something broke!");
        return; 
    }
    // If this is executed there was no error!
});

The problem with callbacks is they are a little messy. Consider a chain of asynchronous functions that need to run one after another:

aFunction(argument1, function(response) {
	// This runs after aFunction is done
	bFunction(argument2, function(response2) {
    	// This runs after bFunction is done
        cFunction(argument3, function(response3) {
        	// This runs after cFunction is done
        });
    });
});

Not too pretty right? It gets even uglier if you want to deal with errors properly at every step of the chain:

aFunction(argument1, function(error, response) {
	// This runs after aFunction is done
    if (error) { return error; }
	bFunction(argument2, function(error, response2) {
    	// This runs after bFunction is done
        if (error) { return error; }
        cFunction(argument3, function(error, response3) {
        	if (error) { return error; }
        	// This runs after cFunction is done
        });
    });
});

Fortunately promises help make this much neater, easier to read and easier to control.

Promises

A promise is essentially an abstraction of a callback that uses simple to understand syntax and structures. A promise is just that, a promise to give something in the future. For example, say asynchronousFunction from above makes a call to an API requesting all of the virtual machines on an account. That process isn't instant, so it returns a promise which is a placeholder for the list of all the virtual machines on an account. A promise can be either resolved or rejected. A resolved promise is one that fulfilled its promise so to speak and actually has the data you were promised. A rejected promise happens if an error is encountered.

So to rewrite the asynchronousFunction from above using promises you would do something like this:

asynchronousFunction(argument).then(function(response) {
	// Executed after function is done
});

The flow works just like it states, do this function, then do this. You can also pass around promises pretty easily:

function aFunction() {
	return asynchronousFunction(argument);
}

aFunction.then(function(response) {
	// Executed after asynchronousFunction is done
});

Error management is a breeze:

asynchronousFunction(argument).then(function(response) {
	// This is executed if the promise is resolved
}).catch(function(error) {
	// This is executed if the promise is rejected
});

Promises really shine when you have to link a bunch of stuff together. Recall how callbacks looked with multiple functions and then take a look at the syntax used by promises:

aFunction(argument).then(function(response) {
	// Executed after aFunction is done
	return bFunction(argument2);
}).then(function(response2) {
	// Executed after bFunction is done
	return cFunction(argument3);
}).then(function(response3) {
	// Executed after cFunction is done
    return "All done";
});

Much cleaner and easier to read in my opinion. Errors can also propagate up the promise chain fairly easily as long as all promises in the chain are returned properly. A single catch statement for the entire chain will catch any errors thrown while processing those promises:

aFunction(argument).then(function(response) {
	// Executed after aFunction is done
	return bFunction(argument2);
}).then(function(response2) {
	throw new Error("foobar"); 
	// Executed after bFunction is done
	return cFunction(argument3);
}).then(function(response3) {
	// Executed after cFunction is done
    return "All done";
}).catch(function(error) {
	// Caught error from bFunction and stops executing
});

Bluebird is (in my opinion) the best library to use for this sort of thing. You can do tons of really cool stuff with it, including converting things that require callbacks to promises using promisifyAll:

var libraryThatUsesCallbacks = require("callbackLibrary");
var Promise = require("bluebird");

Promise.promisifyAll(libraryThatUsesCallbacks);

// libraryThatUsesCallbacks can now be used like so

libraryThatUsesCallbacks.aFunctionAsync(argument).then(function(response) {
	// I'm a promise now! 
});

// Bluebird adds new functions that basically map to the old function name + async, so bFunction would become bFunctionAsync

This can generally be done for all functions that use the callback(error, arguments) pattern though Bluebird does support other arrangements as well. Bluebird supports a wide array of other things, I would recommend taking a look at the API documentation.

Conclusion

Given the single threaded nature of Node applications learning how to write asynchronous code, and more importantly, think asynchronously, is necessary to becoming a successful Node developer. This isn't an easy task, especially if you are not used to thinking about things in this way, it took me a long time to fully grasp the concepts and properly use asynchronous functions so don't worry if it takes a bit. Hopefully this post will help you learn the basics if you are looking to develop Node applications. Please feel free to leave any questions in the comments.

in libvirt, node, node.js, node-libvirt, promises

After adding a couple things to Node-Libvirt one of the maintainers pointed out my original article on Node-Libvirt referenced the old callback style of using the library so I wanted to take a moment to write up a new guide for using Node-Libvirt with promises rather than callbacks. Since my first post offers a fair amount of detail regarding Node-Libvirt in general, I'll limit this one to showing syntax only.

Hypervisor

First lets take a look at connecting to the hypervisor using promises. Of the changes, this is the most prominent (in my opinion anyway).

Connection

var libvirt = require('libvirt');
var hypervisorObject = libvirt.hypervisor;

var hypervisor = new hypervisorObject('qemu:///system');

hypervisor.connect(function() { 
    // The hypervisor object is now connected. From here you can do just about any call you want...
    // Like hypervisor.lookupDomainById(1);
});

You'll probably want to wrap the code you need to access the hypervisor with in the hypervisor.connect() function to ensure the connection has been made before you attempt to communicate with the hypervisor.

Everything Else

It isn't really worth noting each and every function I mentioned originally, but I will give you a couple examples for how things look using promises.

The basic idea is you add Async to the name of the function you want to call, and then use then instead of a callback. For example:

hypervisor.helloWorld(function(callback) {
	// Old style
});

hypervisor.helloWorldAsync(arguments).then(function(return_value) {
	// New style
});

For real example take a look at a call to getNodeInfo:

hypervisor.getNodeInfoAsync().then(function(nodeStats) {
	// nodeStats now has the return value from getNodeInfo
});

Domains

Domains use pretty much the same syntax shown above but I'll run through a few examples:

// Lets make a new domain
hypervisor.defineAsync(domainXML).then(function(domain) {
	// Use domain to access this domain now
    
    // Now lets restart that domain
    domain.startAsync().then(function() {
    	// Domain has been restarted
    });
});

// Lets lookup a domain with the id 1
hypervisor.lookupDomainByIdAsync(1).then(function(domain) {
	// Use the domain object to do other stuff
});

Would you like to know more?

From here, everything looks almost identical to the syntax you see above. Just append Async to the name of the function and use then instead of a callback and you will be good to go.

If there are any specific calls you are interested in seeing please don't hesitate to leave a comment. Most of these calls do have examples in the Node-Libvirt tests. If you are planning on using Node-Libvirt consider following me on Twitter as I intend to document the full Node-Libvirt API in the future.

Thanks for reading!

in Getting Started, libvirt, node, node.js, domain, hypervisor, node-libvirt

Recently I've been doing a fair amount of work with the libvirt API, specifically using the Node.js bindings found here. The bindings are great and work as expected but I was rather disappointed in the lack of documentation.

The authors reference the tests for the project as documentation, but this can be a bit of a pain to read through especially if you are new to the language. Given this, I'm going to start documenting some of the functions I use most often.

This post assumes you are using the latest version of node-libvirt (>=0.1.3). Please note this is important because earlier versions of this library were not asynchronous - code snippets in this post will not work for earlier versions.

Hypervisor

Before you can do anything else, you must communicate with the hypervisor its self.

Connection

Communication with the hypervisor can be achieved by instantiating a new hypervisor object:

var libvirt = require('libvirt');
var hypervisor_object = new libvirt.Hypervisor(CONNECT_URI);

Connect URI will vary depending on your specific hypervisor (KVM, Xen, etc.) and how you are connecting exactly. For KVM this would be qemu:///system.

Next, you need to actually connect to the hypervisor:

hypervisor_object.connect(function(err) {
	if (!err) {
		console.log("I'm connected!"); 
    }
});

Statistics

Most projects will want to know a bit about the hypervisor, you can gather all sorts of information but I have found the virNodeInfo call the most useful. This provides you with the model of the CPU, memory allocated to the host, number of CPUs, speed of CPUs, nodes, sockets, cores and threads per core. This call is made using the hypervisor object:

hypervisor_object.getNodeInfo(function(err, info) {
	console.log(info); 
});

All remaining hypervisor functions are called in a similar fashion with a callback. Other stats functions of interest include:

  • listDefinedDomains - Returns an array of defined, but not active domain IDs
  • listActiveDomains - Returns an array of active domain names.
  • getNumberOfDefinedDomains - Returns an integer representing the total number of defined but not active domains.
  • getNumberOfActiveDomains - Returns an integer representing the total number of active domains.
  • getNodeFreeMemory - Returns an integer representing the amount of free memory on the host.

From here you can use the hypervisor object to do other things.

Domains

Next, lets get to the meat of the library and discuss domain handling.

Create a Domain

Creating a domain is done using the hypervisor object we created before. You'll need to magic up some XML in the appropriate format to make this work. Once you have that being generated and accessible from Node you can use the following call to make a domain:

hypervisor_object.createDomain(domain_xml, function(err, domain) {
	// Domain is created if no error
});

This will create a non-persistent domain which basically means if the domain is destroyed (completely stopped) it ceases to exist. This is okay if you have all the information you need to reconstruct the domain XML elsewhere (like in a database), if you don't you might want to look into using persistent domains. These can be created using the define function instead of the createDomain wrapper:

hypervisor_object.define(domain_xml, function(err, domain) {
	// Persistent domain created
});

Actions

There are a number of actions available for a domain including:

  • reset
  • reboot
  • suspend
  • resume
  • shutdown
  • start

All of these are called the same way using the domain object. You can get the domain object by accessing domain in the create domain example or you can look a domain up using lookupDomainById or lookupDomainByName, both are covered later.

domain.ACTION(function(err) {
	// If no error action successful
});

Substitute an action from the list above for ACTION.

Get

Getting a domain can be done using either lookupDomainById or lookupDomainByName. Realistically, unless you have a unique application lookupDomainByName will be better used because the domain ID is not static and there is no reliable way of predicting it, the name however is static and should be unique.

hypervisor_object.lookupDomainByName(DOMAIN_NAME_HERE, function(err, domain) {
	// You've got your domain! 
});

Should you need to look up the domain by ID you call that function the same way as shown above.

Destroy

Some people get confused by the nomenclature used by libvirt regarding destroying a domain. In the context of libvirt, destroy does not mean delete in the traditional sense, it essentially means stop the domain now (force it to stop if it cannot be stopped otherwise). This is like pulling the plug on the machine so data loss is a possibility. There is an exception however, if the domain is not persistent destroy will actually remove the domain from libvirt. This is done like the other actions mentioned above:

domain.destroy(function(err) {
	// I'm stopped!
});

If you have yourself a persistent domain and you want to actually make libvirt forget about it completely you would need to use the undefine function:

domain.undefine(function(err) {
	// Poof, I'm gone!
});

Would you like to know more?

I originally started writing this guide with the intention of documenting all the things, but in truth it would be a waste of time. All calls are made using the same basic format:

the_object.WHAT_I_WANT(function(err, RESPONSE_IF_THERE_IS_ONE) {
});

If you find anything particularly challenging regarding hypervisor or domain functions please let me know - I'm happy to help!

While a bit annoying, the tests for node-libvirt are useful if you want to see the context of these calls. You can find the tests for domains here and hypervisors here.

If there is any interest I'd be happy to continue covering the basic calls for other libvirt areas - please let me know if you are interested in that at all.

Thanks for reading!

in Getting Started, development, bookshelf.js, node, node.js, guide

I recently started a project where I was tasked to use Bookshelf.js to interact with the database. The library its self is pretty handy, it is built on top of Knex.js, a great query builder that works with Postgres, MySQL, and SQLite, but my major gripe was an overall lack of examples in the documentation. That being said, I wanted to document what I learned for folks using Bookshelf.js in the future (and for myself when I inevitably forget how it was done).

Installing Bookshelf.js & Knex.js

Installation requires Bookshelf.js, Knex.js as well as the corresponding package for the database server you are using (MySQL, Postgres, SQLite):

npm install knex --save
npm install bookshelf --save

Next, install the package for your database server:

// Pick one
npm install pg --save // Postgres
npm install mysql --save // MySQL
npm install mariasql --save // MariaDB
npm install sqlite3 --save // SQLite

Connecting with Database

Once Establishing a connection with the database is fairly simple, pass in a JSON object of the necessary options:

var knex = require('knex')({
	client: '{pg|mysql|sqlite3}',
    connection: {
    	host: 'host' // IP or domain name
        user: 'user' // DB username
        password: 'password' // DB password
        database: 'database' // DB name
        charset: 'utf8' // Or your preferred charset
    }
});

If you are using SQLite, add the filename directive to the connection directives, this should be the path to your database on the filesystem.

Next, bring Bookshelf into the loop:

var bookshelf = require('bookshelf')(knex);

With that, you're ready to get started!

Models

The first thing you'll want to do is extend Model so you can interact with tables in your database. This is done like so:

var model = bookshelf.Model.extend({
	tableName: "nameOfTable"
});

You can now use the variable model to interact with this table.

A quick side note, you can append functions to the model delcaration if need be as shown in this example from bookshelfjs.org:


var checkit  = require('checkit');
var Promise  = require('bluebird');
var bcrypt   = Promise.promisifyAll(require('bcrypt'));

var Customer = bookshelf.Model.extend({

  initialize: function() {
    this.on('saving', this.validateSave);
  },

  validateSave: function() {
    return checkit(rules).run(this.attributes);
  },

  account: function() {
    return this.belongsTo(Account);
  },

}, {

  login: Promise.method(function(email, password) {
    if (!email || !password) throw new Error('Email and password are both required');
    return new this({email: email.toLowerCase().trim()}).fetch({require: true}).tap(function(customer) {
      return bcrypt.compareAsync(customer.get('password'), password);
    });
  })

});

Customer.login(email, password)
  .then(function(customer) {
    res.json(customer.omit('password'));
  }).catch(Customer.NotFoundError, function() {
    res.json(400, {error: email + ' not found'});
  }).catch(function(err) {
    console.error(err);
  });

Creating a New Record

Next, lets create a record - I'll continue using the variable model as declared in the previous section:

new model({
	'column1': 'columnValue',
    'column2': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
	console.log(newRow.id); // Returns ID of new row
	// Code executed after row successfully created
}).catch(function(err) {
	// Handle errors
});

You can also use forge instead of the new notation:

model.forge({
	'column1': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
	console.log(newRow.id); // Returns ID of new row
}).catch(function(err) {
	// Handle errors
});

Bookshelf has built in promisification so be sure to take advantage of that feature in your code!

Update Record(s)

Updating records looks pretty similar to creating a new record except you'll also need to pass save() a JSON object that contains the updates you want to make:

new model({
	// The query will match these parameters
    'id': 1
    // Will return row with ID 1
}).save({
	// These updates will be made
    'name': 'Joe'
    // Record's name will be updated to 'Joe'
}).function(updatedModel) { ... }).catch(function(err) { ... });

Fetch Record(s)

Fetching records also looks pretty similar to the previous operations except you will need to use fetch():

new model({
	// Query params 
}).fetch().function(fetchedModel) {
	// Do stuff with fetchedModel 
}).catch(function(err) { ... });

Delete Record

Last but not least, you can delete a record using just about everything used in the fetch example while adding the destroy() function:

new model({
	// Query params
}).fetch().then(function(fetchedModel) {
	fetchedModel.destroy(); 
}).catch(function(err) { ... });

You can avoid requiring the then() handling if your query includes the id of the row:

new model({
	'id': 1
}).fetch().destroy();

Conclusion

This is as far as I have gotten with Bookshelf.js - I do intend to continue writing about this library as I learn more myself. I hope this helps somebody else getting started with Bookshelf.js! If you are interested in reading the official documentation you can find that here.