/home/jpettit coder, sys admin, overall geek

in Getting Started, libvirt, node, node.js, domain, hypervisor, node-libvirt

Recently I've been doing a fair amount of work with the libvirt API, specifically using the Node.js bindings found here. The bindings are great and work as expected but I was rather disappointed in the lack of documentation.

The authors reference the tests for the project as documentation, but this can be a bit of a pain to read through especially if you are new to the language. Given this, I'm going to start documenting some of the functions I use most often.

This post assumes you are using the latest version of node-libvirt (>=0.1.3). Please note this is important because earlier versions of this library were not asynchronous - code snippets in this post will not work for earlier versions.

Hypervisor

Before you can do anything else, you must communicate with the hypervisor its self.

Connection

Communication with the hypervisor can be achieved by instantiating a new hypervisor object:

var libvirt = require('libvirt');
var hypervisor_object = new libvirt.Hypervisor(CONNECT_URI);

Connect URI will vary depending on your specific hypervisor (KVM, Xen, etc.) and how you are connecting exactly. For KVM this would be qemu:///system.

Next, you need to actually connect to the hypervisor:

hypervisor_object.connect(function(err) {
	if (!err) {
		console.log("I'm connected!"); 
    }
});

Statistics

Most projects will want to know a bit about the hypervisor, you can gather all sorts of information but I have found the virNodeInfo call the most useful. This provides you with the model of the CPU, memory allocated to the host, number of CPUs, speed of CPUs, nodes, sockets, cores and threads per core. This call is made using the hypervisor object:

hypervisor_object.getNodeInfo(function(err, info) {
	console.log(info); 
});

All remaining hypervisor functions are called in a similar fashion with a callback. Other stats functions of interest include:

  • listDefinedDomains - Returns an array of defined, but not active domain IDs
  • listActiveDomains - Returns an array of active domain names.
  • getNumberOfDefinedDomains - Returns an integer representing the total number of defined but not active domains.
  • getNumberOfActiveDomains - Returns an integer representing the total number of active domains.
  • getNodeFreeMemory - Returns an integer representing the amount of free memory on the host.

From here you can use the hypervisor object to do other things.

Domains

Next, lets get to the meat of the library and discuss domain handling.

Create a Domain

Creating a domain is done using the hypervisor object we created before. You'll need to magic up some XML in the appropriate format to make this work. Once you have that being generated and accessible from Node you can use the following call to make a domain:

hypervisor_object.createDomain(domain_xml, function(err, domain) {
	// Domain is created if no error
});

This will create a non-persistent domain which basically means if the domain is destroyed (completely stopped) it ceases to exist. This is okay if you have all the information you need to reconstruct the domain XML elsewhere (like in a database), if you don't you might want to look into using persistent domains. These can be created using the define function instead of the createDomain wrapper:

hypervisor_object.define(domain_xml, function(err, domain) {
	// Persistent domain created
});

Actions

There are a number of actions available for a domain including:

  • reset
  • reboot
  • suspend
  • resume
  • shutdown
  • start

All of these are called the same way using the domain object. You can get the domain object by accessing domain in the create domain example or you can look a domain up using lookupDomainById or lookupDomainByName, both are covered later.

domain.ACTION(function(err) {
	// If no error action successful
});

Substitute an action from the list above for ACTION.

Get

Getting a domain can be done using either lookupDomainById or lookupDomainByName. Realistically, unless you have a unique application lookupDomainByName will be better used because the domain ID is not static and there is no reliable way of predicting it, the name however is static and should be unique.

hypervisor_object.lookupDomainByName(DOMAIN_NAME_HERE, function(err, domain) {
	// You've got your domain! 
});

Should you need to look up the domain by ID you call that function the same way as shown above.

Destroy

Some people get confused by the nomenclature used by libvirt regarding destroying a domain. In the context of libvirt, destroy does not mean delete in the traditional sense, it essentially means stop the domain now (force it to stop if it cannot be stopped otherwise). This is like pulling the plug on the machine so data loss is a possibility. There is an exception however, if the domain is not persistent destroy will actually remove the domain from libvirt. This is done like the other actions mentioned above:

domain.destroy(function(err) {
	// I'm stopped!
});

If you have yourself a persistent domain and you want to actually make libvirt forget about it completely you would need to use the undefine function:

domain.undefine(function(err) {
	// Poof, I'm gone!
});

Would you like to know more?

I originally started writing this guide with the intention of documenting all the things, but in truth it would be a waste of time. All calls are made using the same basic format:

the_object.WHAT_I_WANT(function(err, RESPONSE_IF_THERE_IS_ONE) {
});

If you find anything particularly challenging regarding hypervisor or domain functions please let me know - I'm happy to help!

While a bit annoying, the tests for node-libvirt are useful if you want to see the context of these calls. You can find the tests for domains here and hypervisors here.

If there is any interest I'd be happy to continue covering the basic calls for other libvirt areas - please let me know if you are interested in that at all.

Thanks for reading!

in project, spacepanel

SpacePanel started as a learning experience primarily - I wanted to learn the in's and out's of virtualization using libvirt. Given the nature of the project, that basic goal was quickly met. However, during the early stages of development SpacePanel received a fair amount of attention from colleagues and general folks across the internet, this inspired me to continue developing SpacePanel. I've never had one of my projects receive any significant amount of attention so I was honestly surprised when SpacePanel was received this way.

Eventually SpacePanel grew to the point I needed help. Lev Lazinskiy, my good friend and former colleague at Linode joined the team to help move development further. The project continued to grow and we eventually reached a point where we simply needed more resources to make SpacePanel bigger, better and hopefully a success. We started an Indiegogo campaign shortly after. Unfortunately, despite lots of hits and encouraging comments we fell significantly short of our fundraising goal. This put Lev and myself in an uncomfortable situation of not knowing what to do next.

Shortly after this we were contacted by the founders of Virtkick. After a bit of discussion with the founders it was decided that instead of trying to go at it alone (and with extremely limited resources) with SpacePanel, I would join their efforts and work on Virtkick exclusively. This started on a trial basis, but recently it was decided that I would continue doing this on a full time basis.

With this being said, it is with a heavy heart that I am officially ending the development of SpacePanel. If you are interested in a control panel like SpacePanel I would highly recommend taking a look at Virtkick. Virtkick is exactly what I eventually wanted SpacePanel to be and I am confident you will enjoy using it.

To those who generously donated to the SpacePanel project, free free to reach out to me at social@spacepanel.io if you would like to reclaim your donation.

In closing, I'd like to run through a few numbers (I like numbers and statistics) concerning SpacePanel:

  • 111 total stars on Github
  • 221 commits spanning a 5 month timeline
  • 58 issues total
  • 30 closed issues
  • 3 milestones met, 1 pending
  • 11 pull requests
  • 2 non-author contributors - KeiroD and deanperry
  • 2 authors
  • ~30 downloads
  • ~70,000 lines of code
  • Countless hours of development time

It's been a fun ride - thank you for those who have helped along the way.

in Getting Started, development, bookshelf.js, node, node.js, guide

I recently started a project where I was tasked to use Bookshelf.js to interact with the database. The library its self is pretty handy, it is built on top of Knex.js, a great query builder that works with Postgres, MySQL, and SQLite, but my major gripe was an overall lack of examples in the documentation. That being said, I wanted to document what I learned for folks using Bookshelf.js in the future (and for myself when I inevitably forget how it was done).

Installing Bookshelf.js & Knex.js

Installation requires Bookshelf.js, Knex.js as well as the corresponding package for the database server you are using (MySQL, Postgres, SQLite):

npm install knex --save
npm install bookshelf --save

Next, install the package for your database server:

// Pick one
npm install pg --save // Postgres
npm install mysql --save // MySQL
npm install mariasql --save // MariaDB
npm install sqlite3 --save // SQLite

Connecting with Database

Once Establishing a connection with the database is fairly simple, pass in a JSON object of the necessary options:

var knex = require('knex')({
	client: '{pg|mysql|sqlite3}',
    connection: {
    	host: 'host' // IP or domain name
        user: 'user' // DB username
        password: 'password' // DB password
        database: 'database' // DB name
        charset: 'utf8' // Or your preferred charset
    }
});

If you are using SQLite, add the filename directive to the connection directives, this should be the path to your database on the filesystem.

Next, bring Bookshelf into the loop:

var bookshelf = require('bookshelf')(knex);

With that, you're ready to get started!

Models

The first thing you'll want to do is extend Model so you can interact with tables in your database. This is done like so:

var model = bookshelf.Model.extend({
	tableName: "nameOfTable"
});

You can now use the variable model to interact with this table.

A quick side note, you can append functions to the model delcaration if need be as shown in this example from bookshelfjs.org:


var checkit  = require('checkit');
var Promise  = require('bluebird');
var bcrypt   = Promise.promisifyAll(require('bcrypt'));

var Customer = bookshelf.Model.extend({

  initialize: function() {
    this.on('saving', this.validateSave);
  },

  validateSave: function() {
    return checkit(rules).run(this.attributes);
  },

  account: function() {
    return this.belongsTo(Account);
  },

}, {

  login: Promise.method(function(email, password) {
    if (!email || !password) throw new Error('Email and password are both required');
    return new this({email: email.toLowerCase().trim()}).fetch({require: true}).tap(function(customer) {
      return bcrypt.compareAsync(customer.get('password'), password);
    });
  })

});

Customer.login(email, password)
  .then(function(customer) {
    res.json(customer.omit('password'));
  }).catch(Customer.NotFoundError, function() {
    res.json(400, {error: email + ' not found'});
  }).catch(function(err) {
    console.error(err);
  });

Creating a New Record

Next, lets create a record - I'll continue using the variable model as declared in the previous section:

new model({
	'column1': 'columnValue',
    'column2': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
	console.log(newRow.id); // Returns ID of new row
	// Code executed after row successfully created
}).catch(function(err) {
	// Handle errors
});

You can also use forge instead of the new notation:

model.forge({
	'column1': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
	console.log(newRow.id); // Returns ID of new row
}).catch(function(err) {
	// Handle errors
});

Bookshelf has built in promisification so be sure to take advantage of that feature in your code!

Update Record(s)

Updating records looks pretty similar to creating a new record except you'll also need to pass save() a JSON object that contains the updates you want to make:

new model({
	// The query will match these parameters
    'id': 1
    // Will return row with ID 1
}).save({
	// These updates will be made
    'name': 'Joe'
    // Record's name will be updated to 'Joe'
}).function(updatedModel) { ... }).catch(function(err) { ... });

Fetch Record(s)

Fetching records also looks pretty similar to the previous operations except you will need to use fetch():

new model({
	// Query params 
}).fetch().function(fetchedModel) {
	// Do stuff with fetchedModel 
}).catch(function(err) { ... });

Delete Record

Last but not least, you can delete a record using just about everything used in the fetch example while adding the destroy() function:

new model({
	// Query params
}).fetch().then(function(fetchedModel) {
	fetchedModel.destroy(); 
}).catch(function(err) { ... });

You can avoid requiring the then() handling if your query includes the id of the row:

new model({
	'id': 1
}).fetch().destroy();

Conclusion

This is as far as I have gotten with Bookshelf.js - I do intend to continue writing about this library as I learn more myself. I hope this helps somebody else getting started with Bookshelf.js! If you are interested in reading the official documentation you can find that here.

in 3d printing, da vinci

I finally decided to take the plunge and jump on the 3D printing bandwagon. My new 3D printer was delivered Thursday last week (03/19). I settled on the Da Vinci 2.0 Duo after a fair amount of reading. The primary thing that sold me was a) the price and b) the dual extrusion heads.

3D Printer 1

This particular printer was priced at $650, it comes with two extruders so you can mix it up and print in multiple colors, a heated build platform (it seems like this is included on most 3D printers now though) and a couple spools of ABS filament. I was very impressed, the printer required very little set up, by that I mean I took it out of the box, plugged it in and started printing. XYZprinting, the manufacturers of the Da Vinci, supposedly calibrate everything in the factory and my experience seems to confirm that.

3D Printer 2

The picture above is one of the demos that come built in to the printer. Granted, I am not an expert in 3D printing, but I think it came out fairly well given I had to make no adjustments.

I of course spent most of the following days printing everything I could think of including figurines, a cup, a soap dish, and a headphone holder:

Headphone holder / charmander figurine

Sweet cup

All of these were printed at the second highest detail setting and frankly I think they look fine at that setting but I was curious what the highest setting would look like so I printed this dinaosaur figurine:

Dinosaur figurine

It is probably tough to see it because of my potato image quality, but the lines in the dinosaur figurine are much less pronounced, however this print also took subsantially longer (to the tune of 5-6 hours) more than prints at the medium detail setting. I'm not sure if this is to be expected with all 3D printers or if it is this model in particular, but I feel like the quicker build time is a worthwhile trade off for slightly poorer quality.

All in, this particular printer model seems to be a great option for those looking to get into 3D printing. I am not yet experienced enough in printing to really have any real qualms, however I'm sure there were trade offs somewhere to make the printer so affordable. The only gripe I have so far is a bit of trouble with some replacement filament I purchased but the company's support seems responsive so I'll give them a chance to help with that before bashing them too much.

The one thing that surprised me the most about purchasing the printer is the sort of sudden realization that you can print just about anything. The popular go to here is the whole 3D printing a gun, though I am not convinced that is 100% conceivable unless you have a ridiculous printer, but there are tons of practical things you can print like my soap dish, or headphone holder. Most of my prints have been spontaneous because its cool prints, but I can definitely see this sort of changing things in the future for me and, well, the world!

In short:

3D print all the things

in arduino, project, zigbee, smartthings, electronics

Since I purchased my SmartThings hub a few months ago I've been all about adding new sensors, lights and other smart things to my apartment.

Most recently I have been experimenting with the GE Link Light Bulbs. Overall I love them, but the one thing I (and my fiancée) have found a bit annoying is having to use a smartphone to control them. For the most part, this isn't necessary as I have a number of rules to automatically turn on the lights when need be, but there is always the odd situation where I want to turn the lights on/off manually.

Thinking of a better way to turn on these lights the old fashioned way (without having to install Zigbee light switches, we rent) turned me on to this latest electronics project.

SmartThings has a pretty robust API and I've tinkered around with controlling the lights from a web browser. The process basically involves a simple GET request to a predefined URL with a token, the ID of the switch I want to manipulate and what I want to do.

This lead me to develop physical buttons that hook into an Arduino to send the GET request when I press a button:

SmartThings Button

The circuit its self is pretty simple, each button is wired into a 150 Ohm resistor to ground, the 5V rail, and a separate line to the Arduino inputs. The Arduino has an ethernet shield to communicate with the greater internet.

SmartThings Button 2

Each button controls a different light (I have three bulbs total currently), the red and green switches will eventually be used to turn all the lights on or off but this hasn't been implemented yet.

Code-wise I've made use of the Arduino ethernet library to handle all of the networking and the HTTP request directly:

#include <Dhcp.h>
#include <Dns.h>
#include <Ethernet.h>
#include <EthernetClient.h>
#include <EthernetServer.h>
#include <EthernetUdp.h>
#include <util.h>
#include <SPI.h>

int BUTTON_DESK_LOWER= 7;
int BUTTON_DESK_UPPER = 8;
int BUTTON_SLIDING = 9;

char server[] = "graph.api.smartthings.com";

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

IPAddress ip(192,168,0,174);

EthernetClient client;

void setup()
{
  Serial.begin(9600);
   while (!Serial) {
    ;
  }
  
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    Ethernet.begin(mac, ip);
  }
 
  delay(1000);
  
  pinMode(BUTTON_DESK_LOWER,INPUT);
  pinMode(BUTTON_DESK_UPPER,INPUT);
  pinMode(BUTTON_SLIDING,INPUT);
}

void loop()
{
  if(digitalRead(BUTTON_DESK_LOWER) == HIGH)
  {
    if (client.connect(server, 80)) 
    {
      Serial.println("connected");
      client.println("GET /api/smartapps/installations/REPLACE HTTP/1.1");
      client.println("Host: graph.api.smartthings.com");
      client.println("User-Agent: Mozilla/5.0");
      client.println("Connection: close");
      client.println();
      
      delay(5000);
 
      client.stop();
     } 
    else 
    {
      Serial.println("connection failed");
    }
  }
  if(digitalRead(BUTTON_DESK_UPPER) == HIGH)
  {
    if (client.connect(server, 80)) 
    {
      Serial.println("connected");
      client.println("GET /api/smartapps/installations/REPLACE HTTP/1.1");
      client.println("Host: graph.api.smartthings.com");
      client.println("User-Agent: Mozilla/5.0");
      client.println("Connection: close");
      client.println();
      
      delay(5000);
 
      client.stop();
     } 
    else 
    {
      Serial.println("connection failed");
    }
  }
  if(digitalRead(BUTTON_SLIDING) == HIGH)
  {
    if (client.connect(server, 80)) 
    {
      Serial.println("connected");
      client.println("GET /api/smartapps/installations/REPLACE HTTP/1.1");
      client.println("Host: graph.api.smartthings.com");
      client.println("User-Agent: Mozilla/5.0");
      client.println("Connection: close");
      client.println();
      
      delay(5000);
 
      client.stop();
     } 
    else 
    {
      Serial.println("connection failed");
    }
  }
}

It is still a bit (see: extremely) rough around the edges but it works! I'm still working on making it look a bit better and finding a way to mount it in some sort of useful position.

If you are interested in reusing the code you are welcome to - just replace the REPLACE strings with the token, switch ID and action with your own and change the button inputs to the inputs you have chosen. The MAC address I used was the default one from the example code (my shield did not have a MAC on it) so you'll probably want to change that and the IP assignment depending on your environment. You can find the repository on Github.