Irakli Gozalishvili

isolate.js via AST analysis

2013-01-23T00:00:00-08:00

In the Add-on SDK we have a problem that is both annoying and confusing for a lot of our users. The problem is not SDK-specific though and maybe interesting for anyone dealing with JS in concurrent execution contexts.

Problem overview

The Add-on SDK was designed to be compatible with browser architecture where chrome and content may be in a separate isolated processes. This imposes a lot of limitations on how add-on code can interact with a page / tab content, and the lack of concurrency constructs in Javascript make this problem a lot more irritating. In the SDK we ended up implementing an API that closely resembles web workers. The main add-on code can execute content scripts in a page context that acts like a worker and interaction between add-on and content scripts happen through a message passing (for more details please take a look at content script documention)

We quickly discovered that not a lot of people are comfortable with message passing APIs, but even putting that aside it’s really annoying to be creating separate content script files for a few lines of code (most of the add-ons have content scripts that consist of few lines of code, unless accompanied with external js libraries like jQuery):

#!env/javascript
var pageMod = require("sdk/page-mod")

var mod = new pageMod.PageMod({
  include: ["*.co.uk"],
  contentScriptFile: require("self").data.url("./my.js")
}))

As an option we also allow passing content scripts in form of JS string, which is far from ideal:

#!env/javascript
var pageMod = require("page-mod")

pageMod.add(new pageMod.PageMod({
  include: ["*.co.uk"],
  contentScript: "document.body.innerHTML = " +
             	 "'<h1>this page has been eaten</h1>'"
}))

What you would actully want is to just write a function that will be executed in the context of the content. Unfortunately it’s not just matter of serialising function and then evaluating it in the context, since functions can access bindings from the outer scopes.

Idea

If we could prove that function does not refers to anything but arguments passed to it or definitions with in it, it would be pretty safe to transplant it into completely different execution context:

#!env/javascript
var pageMod = require("sdk/page-mod")

var mod = new pageMod.PageMod({
  include: ["*.co.uk"],
  contentScript: isolate(function() {
    document.body.innerHTML = "<h1>this page has been eaten</h1>"
}))

As a matter of fact such isolate function can be implemented relatively easy, all it has to do is serialise given function, parse it, perform static analyzes to verify no outer references are made and return some wrapping of source back. If given function does have prohibited references then throw TypeError back.

Code

With a great projects like Esprima, Acorn parsing JS is no-brainer. Also for SDK we could / should just use Spidermonkey parser API, but since all of them produce de facto standad AST format we can swap parsers as we pleased. So the only remaining chunk of work was static analysis. I made few micro-libraries solving very specific problems.

interset

While playing with AST I quickly discovered the need for binary operations for logical sets. So I end up writing interset small library for doing exactly that:

#!env/javascript
var union = require("interset/union")

union([1, 2], [2, 3], [3, 4])
// => [1, 2, 3, 4]

var intersection = require("interset/intersection")
intersection([1, 2], [2, 3])
// => [2]

var difference = require("interset/difference")
difference([1, 2, 3], [1], [1, 4], [3])
// => [2]

episcope

ECMAScript scope analyzer. This Library provides set of functions that perform analyzes on the nodes of the AST in the de facto AST format. All the API function take AST nodes denoting a lexical scope and performed static analyzes at the given scope level (Note: examples use esprima but library does not really cares about it and expects AST format implemented by all popular JS parsers):

#!env/javascript
var esprima = require("esprima")
var references = require("episcope/references")

var ast = esprima.parse("console.log('>>>', error)")
references(ast)
// =>  [{ type: "Identifier", name: "console" }, { type: "Identifier", name: "error" }]

var bindings = require("episcope/bindings")
var ast = esprima.parse("function foo(a, b) { var c = a + b; return c * c }")
ast.body[0].id
// => { type: 'Identifier', name: 'foo' }
bindings(ast.body[0])
// =>  [ { type: 'Identifier', name: 'a' },
//       { type: 'Identifier', name: 'b' },
//       { type: 'Identifier', name: 'c' } ]

var scopes = require("episcope/scopes")
var ast = esprima.parse(String(function root() {
  function nested() { /***/ }
  try { /***/ } catch(error) { /***/ }
}))
scopes(ast.body[0])
// => [
//  { 
//    type: 'FunctionDeclaration',
//    id: { type: 'Identifier', name: 'nested' },
//    // ...
//  },
//  {
//    type: 'CatchClause',
//    param: { type: 'Identifier', name: 'error' },
//    body: { type: 'BlockStatement', body: [] }
//  }
//]

isoscope

ECMAScript function isolation analyzer. This is very alpha, but intention is to provide API for performing static analyzes on the JS AST nodes denoting a function definition / declaration and perform analyzes to gather info about their isolation:

#!env/javascript
var esprima = require("esprima")
var enclosed = require("isoscope/enclosed")

// Parse some code
var form = esprima.parse(String(function fn(a, b) {
  console.log(String(a) + b)
}))

// Get a function form we'll be analyzing
var fn = form.body[0]
fn.id
// => { type: "Identifier", name: "fn" }

// Get names of enclosed references
enclosed(fn)
// => [ "console", "String" ]

Now the only thing left for isolate to do is ensure that enclosed references are legit globals or built-ins and that nasty things like with and eval are not used.

If this experiment works out in practice that would allow us to explore a lot more to make new and brave concurrent world a lot more accessible and pleasant to use for our users.

All the libraries mentioned are open source and work both in SDK and Node, probably in browsers too. So feel free to play with them, break them and maybe even contribute fixes!

interactivate

2012-11-12T00:00:00-08:00

I have made a little live-coding experiment that is primarily inspired by Mathematica notepad, Bret Victor’s Inventing on Principle and has being pioneered by othe cool projects like light table.

I do write JS & HTML every single day and I can’t even imagine doing that without a REPL. Although scratchpad made me realized how I hate loop part of REPL! Another problem is an amount of time one needs to waste, while being inspired, to get a glimpse of an idea, live on a screen. Creating HTML, JS files wiring them with each other, page refreshes, etc… I’m sure we can do better than that, here is a little demo of what I think it should be like:

You can give it a try yourself or check out the code on github.

This expreiment was done in a couple of hours, so it’s far from being stable. It has bunch of known bugs than I need to fix, but I still hope it was worth sharing.

Making this would have being impossible without awesome open source projects like codemirror & browserify who’m I wish to give a good credit.

Finally I really hope to see Scratchpad picking up some of this soon!

Recent changes in SDK

2012-10-29T00:00:00-07:00

There have being some significant changes to the Add-on SDK lately, which is a step forward to landing and shipping SDK in Firefox. In this post I’ll try to cover what the changes are.

Probably the most important detail is that SDK no longer uses packages. This is an improvement for two reasons:

once SDK is in Firefox we won’t be able to perform complicated module search at compile time and even more so at runtime.
We also see it as great improvement to current ambiguous behavior of require. Now SDK is just a library of modules. Once it’s in Firefox, all addons will be able to load any SDK module without needing to convert completely to an SDK-style project.
We also hope to empower other Mozilla projects with CommonJS so that new APIs & features can be authored in de facto standard CommonJS module format.

Going forward, the idiomatic way to require an SDK modules now is: require("sdk/panel"). Low level and usually less stable APIs will have longer require paths: require("sdk/window/utils").

To ensure that addons built using the old layout are not affected by this change we have analyzed all the module requirements of all AMO hosted addons and created a mapping that maps module requirements used to a new style. CFX - SDK’s CLI tool uses this mapping to allow backwards compatible behavior, while we have no plans on breaking backwards compatibility in a visible future, we still would like to encourage authoring new add-ons in idiamatic style this will make your code a lot more portable!

This new layout is currently on the Master branch and is scheduled to be released on December 11 as version 1.12.

(feedback :clojurescript)

2012-09-16T00:00:00-07:00

Lately I have being very entusiastic about clojurescript, but I keep running into small annoyances that I just can not get over with. I think it may be useful to provide a feedback form a user who’s primary language for the past decade was JS, which brings me to a folloing point:

1. Enable github issues

At the moment clojure discussion group is the only place for providing feedback. Group is very active and it’s clojure (not clojurescript) discussion group so most of the discussions are not really relevant to me. I find keeping up with all the noise there hard, and would personally have created bunch of issues reports already if I could have submitted them on github.

2. Accessing properties with computed names

I find API for accessing properties with computed names awkward:

#!env/scheme
(aget js/require.extensions ".cljs")

At least because documentation of get function, matches better my intent then aget’s. Also I find use case common enough, that I think a syntax sugar would be more appropriate.

3. Setting properties with computed names

API for setting properties with computed names is even clunkier:

#!env/scheme
(set! (aget js/require.extensions ".cljs") compile)

I think extending set! special form would make a lot more sense:

#!env/scheme
(set! js/require.extensions ".cljs" compile)

4. Regural expressions

I really enjoy how simple regular expressions translate to JS:

#!env/scheme
#"[^a-z\s]" ;; => /[^a-z\s]/

Although I could not find any way to create regular expressions with advance search flags, other then by using RegExp constructor:

#!env/scheme
(js/RegExp. "\\w+\\s" "g")        ;; => /\w+\s/g
(js/RegExp. "^\\-\\-\\w*$" "m")   ;; => /^\-\-\w*$/m
(js/RegExp. "foo" "i")            ;; => /foo/i

I wish literal syntax had support for search flags, maybe even via clojure tags.

5. Try / catch

In clojure and inherently clojurscript catch clause of the try special form takes error type, error name and expressions:

#!env/scheme
;; for Clojurescript use js/Object as type
(try
   (/ 1 0)
   (catch js/Object e
       (.log js/console e)))

I do understad that it makes sence on JVM, but it definitely does not on JS. I really wish that error type was either optional or was removed entirely in clojurescript.

7. Less dependency on JVM

While I fully understand rational behind dependency on JVM, it’s still painful for someone coming from JS who’s not comfortable with configuring class paths. Tools like leiningen also spend more time on booting JVM then on performing actual tasks. I think clojurscript could be significally more successful among JS developers if they could strat witout any dependencies (even if they will have to deal with them later once they’re more comfortable).

Overall I’m pretty happy with clojurscript, it’s pretty young project and I hope it will only get better and more accessible over time!

nodeconf 2012

2012-07-05T00:00:00-07:00

I had a great pleasure of attending nodeconf that took place in Portland. Conference far exceeded all my expectations. This post has no chance to capture all of the interesting things that happened there, but I still would like to make an effort and share some of the highlights.

First off I would love to thank Mikael Rogers and everyone who helped him to set a whole new standard of how conferences should be held! Theatrical feel was through the whole conference: venue at Imago theatre, talks that flowed into each other according to well crafted scenario. Remarkably single track (or single thread if you will :) model was used, which once again proved to be a perfect fit for event(ed) JS. Another novelty was in a “save questions for the bar” rule that worked out amazingly well.

Day I

Both nodejs and conference was started by a Ryah Dahl, who shared with us a little bit of a background story on how & why node was started. Then very first contributors took a scene to talk about very early days of project, when it was very unstable, but still very fun to hack on. That flowed into talks about how node was maturing and making it’s way into production at Yammer and elsewhere. Was interesting to learn why companies like Microsoft invest into nodejs and how it became first class citizen on windows. That required core changes to an underline platform layer resulting into libuv. There were several interesting talks about libuv, specifically showing how node like servers can be written in C, how much work is still ahead to be done, and how libuv go life of it’s own. Today it is used by other new projects, including Mozilla’s own rust language, by Luvit - node for a Lua language. I was completely blown away by Fedor Indutny’s crazy simple language candor with dedicated VM and JIT compiler he wrote. Unfortunately I did not get a chance to chat with him as he disappeared immediately after his talk.

At the end of the day we all headed to a bar. It turned out there is a nodejs powered karaoke bar in portland, that even has a public REST API. That is also where I was rocking out with others until almost loosing a voice (My apologies to everyone who had to bare with that).

Day II

Node is all about streaming and piping data from one stream to another. That’s what it’s good at and that’s what everyone writing node programs should embrace! We got some interesting talks on that topic. Although there was a talk that demoed that streams are not a magic dust that makes things fast, sometimes buffering is a better approach. Most important take away was that everyone should benchmark to make reasonable decisions based on numbers, not instincts (I feel people often tend to jump to conclusions without any mesures). It was also interesting to learn how streams and other node concepts had being used to drive innovation into browsers.

Another section of talks covered scaling node. That involved talks about tools for benchmarking, profiling, memory leak hunting and real life experience from Voxer.

Final section of talks was about hardware that completely blew away everybody’s minds! All kinds of crazy stuff, that I won’t even attempt to describe, just make sure to watch videos once they’re realized! Here is small teaser of Rick Waldron’s amazing node powered robot to give you an idea.

After party had an epic live performance from the band assembled of a fellow community members! You guys really rocked, I still don’t understand how did you manage to put this amazing performance without much practice.

Socializing

It was pleasure to finally meet friends from internet. I’m also super happy I got a chance to share some of my ideas how I think node streams can be made even better, I’m encouraged and plan to put more effort into this. Also, glad I got some folks excited about my recent obsession lispyscript! Finally thanks for encouraging me to submit a talk for upcoming jsconf.eu, I’m motivated and plan on doing it this weekend!

Speciall thanks to Mozilla for letting me be part of this amazing event!

Write logic, not mechanics

2012-04-26T00:00:00-07:00

It strikes me that developers in JS community tend to choose patterns for solving recurring problems over abstractions.

If we are not busy with semicolon debates, we argue about widely used “callback pattern” for dealing with asynchronous API. Many have learned / invented ways to avoid “pyramids of doom”, but I believe they miss the point: pyramids are not the issue, it’s an indication that we have one.

In order to describe what I consider to be a real issues, I have to move back a little first:

Function

In mathematics, a function is a relation between a set of inputs and a set of potential outputs with the property that each input is related to exactly one output.

Black box

In science and engineering, a “black box” abstraction is used to model systems as set of components which can be viewed solely in terms of its input, output and transfer characteristics, without any knowledge of its internal workings. This components are opaque (black) boxes.

Bigger boxes can be created out of smaller ones just by describing a data flow with in them (connecting inputs and outputs):

This allows one to reduce details as necessary and change internal implementation of any box without affecting other parts of the system as long as transfer characteristics remain same.

Functions in JS

In programing functions are modeled around the same concepts, even though we messed up an input sets by adding implicit parts that may change over time. In JS function input set consists of:

Given arguments.
Pseudo-variable this.
Scope bindings.

Problems

1. No output

Since functions in JS are first class, they can be part of both input and output sets. Most of the asynchronous APIs take advantage of this fact and require special callback function argument for continuation passing:

#!/bin/javascript
require('fs').open(path, 'r+', function(error, fd) {
  // ...
})

As you can see such functions no longer have have any useful output, which means that they can’t be used for building systems as black boxes. Such functions don’t return values that can be passed over to other boxes.

2. Error handling on each step

I have heard many times people criticizing how in Java exceptions are caught, wrapped and re-thrown again. This reminds me of following:

#!/bin/javascript
function readJSON(path, callback) {
  fs.readFile(path, function(error, content) {
    if (error) callback(error);
    else callback(null, JSON.parse(content));
  })
}

Basically error propagation in “callback” style APIs is done manually. Note, that in some cases you may want to try catch actual function body as well.

3. Polygamy

It easy to end up with two types of functions: synchronous and asynchronous. While it’s possible to make sync function async it’s not the case other way round. This usually means that if one the functions had being converted to be asynchronous all of it’s users will have to be converted as well:

#!/bin/javascript
function readJSON(path) {
  var data = fs.readFileSync(path)
  return JSON.parse(String(data))
}

You can’t simply switch to readFile if becomes necessary.

4. Progress tracking

Finally if your function depends on multiple asynchronous inputs then you will have to manually track each.

#!/bin/javascript
function makeView(templateURI, dataURI, callback) {
  var template, data, pending = 2
  readURI(templateURI, function(error, content) {
    if (error) return callback(error)
    template = content
    if (!--pending) Mustache.render(template, data)
  })
  readURI(dataURI, function(error, content) {
    if (error) return callback(error)
    data = content
    if (!--pending) Mustache.render(template, data)
  })
}

Also note that this code assumes that readURI will call a callback only once, which is not guaranteed.

Describe logic not mechanics

Now consider our last example. Most of the code there is for handling mechanics rather than describing a logic, which feels absolutely very wrong. As a matter of fact actual logic can be expressed as:

#!/bin/javascript
function makeView(templateURI, dataURI) {
  var template = readURI(templateURI)
  var data = readURI(dataURI)
  return Mustache.render(template, data)
}

So, would not it be better to abstract timing out of logic when it’s not necessary rather than keep solving all this issues in each and every function ? As a mater of fact solution has being there for ages in a form of promises, but for some reason people and web standards tend to use callbacks instead. Maybe because they feel complicated, but that does not necessary has to be the case:

#!/bin/javascript
// Everyone knows how to write a function:
function sum(a, b) { return a + b }
console.log(sum(1, 2))  // => 3

// Working with promises should not require nothing more
// than marker telling that function can accept input in
// form of promises
sum = promised(sum)
console.log = promised(console.log)

// Will continue to work with plain old values
console.log(sum(1, 2))  // => 3

// Will also accept promises as arguments
var deferred = defer()          // make promise
var a = deferred.promise
var b = sum(a, 1)
var c = sum(b, 5)
console.log(c)                  // eventually prints => 17
deferred.resolve(11)            // fulfill promise

We should not be handling and propagating exceptions manually in each function, we should only handle them when we plan to recover:

#!/bin/javascript
// Utility funciton that throws exceptions
var raise = promised(function(_) { throw Error(_) })

var a = raise('Boom !')   // Now we got an exception
var b = sum(a, 2)         // Now it has propagated to b
var c = sum(b, 12)        // Now it has propagaed to c

// Finally if when ready we handle exception in computation
c.then(null, console.error) // => Error: Boom !

If we just want to group multiple values into one there is an Array for that no need to track progress of each eventual value if we just care about a group!

#!/bin/javascript
Array = promised(Array)
var results = Array(readAsync(a), readAsync(b))
console.log(sum.apply(sum, rusults))

Noticed a pattern ? We just write a logic, and if it needs to handle asynchronous input we wrap it into promised(logic).

Whats really important here is that such functions can be used to build systems in black boxes, since they do have input and output. Demonstration of that is an example from above (assuming that readURI returns promise):

#!/bin/javascript
function makeView(templateURI, dataURI) {
  var template = readURI(templateURI)
  var data = readURI(dataURI)
  // Assuming Mustache.render = promised(Mustache.render)
  return Mustache.render(template, data)
}

Implementing such a solution takes about 100 lines of code (ignoring comments), and that’s more or less what any other control flow library costs anyway. I wish all of you to have more time to concentrate on logic of your program instead of mechanics & small promise library may be a good first step!

protocol based polymorphism

2012-03-21T00:00:00-07:00

Not a long time ago I learned about clojure’s polymorphism constructs and protocols. I was so inspired by a porwer and flexibility of protocol based polymorphism that I decide to prototyped it for JS. In this post I will try to give you a taste of protocols and maybe even motivate you give them a try.

Rationale

In programing we usually write and consume various abstractions. Typically in OOP languages abstractions are defined via (class / object) interfaces and have a nasty expression problems. Imagine that you have A and B sets of abstractions and sets of implementations of those abstractions. A should be able to work with B’s abstractions, and vice versa without modifications of the original code. While it may not sound as a problem at first, it usually is in practice. Sometimes A can’t use B, either because they were not designed to work with each other as they were written by a different authors or because one is newer than the other. Either way such cases require code changes, which may be difficult because code is old, or complicated or has a license restrictions and there could be millions of other reasons. Any code hits these issue in some form and it’s just matter of time. When that happens we’re left only with a few possible solutions:

Feature detection

Typically this is a code that is written not in terms of abstractions, but entities, that do runtime branching by “feature detection”. Which may be a type (if (value instanceof Type)) or shape (if (value && typeof(value.length) === 'number')) based. This not only makes code harder to read & reason about, but it also closed. In other words every new abstraction will require rewrite of those entities, in order to accumulate more conditions and branches.

Wrappers

Typically this means that entities implementing abstraction A need to be wrapped by a “glue code” implementing abstraction from B and vice versa, if consumption is bidirectional. Unfortunately this introduces lot’s of incidental complexity as wrappers ruin identity & don’t compose (every new abstraction requires wrappers for all existing ones and vice versa). Finally problem and required changes grow progressively with a number of abstractions used.

Monkey patching

Typically this means that implementation of A abstraction is patched with a “glue code” implementing support for B abstraction. This still introduces complexity by ruining namespacing (different abstractions may have conflicting names). Again problem gets worth with an amount of code. Also in some languages this is not even possible.

Note: For more details I would recommend watching a “A quick overview of clojure protocols” by Stuart Halloway.

Protocols

In Clojure polymorphism is achieved using protocols. They provide a powerful way for decoupling abstraction interface definition from an actual implementation per type, without risks of interference with other libraries. Protocols allow to add polymorphic behavior to things that already exist without changing them. I’ll go into more details on protocols, but for code examples I will use my JS prototype implementation instead of clojure code.

There are several motivations for JS protocol library:

Provide a high-performance, dynamic polymorphism construct as an alternative to an existing object inheritance that does not provides any mechanics for dealing with name conflicts.
Provide the best parts of interfaces:
- Specification only, no implementation
- Single type can implement multiple protocols
Allow independent extension of types, protocols and implementations of protocols on types, by different parties.

Define protocol

A protocol is a named set of functions and their signatures defined by calling protocol function:

/*jshint asi:true */
// module: ./event-protocol

var protocol = require('protocol/core').protocol

// Defining a protocol for working with an event listeners / emitters.
module.exports = protocol({
  // Function on takes event `target` object implementing
  // `Event` protocol as first argument, event `type` string
  // as second argument and `listener` function as a third
  // argument. Optionally forth boolean argument can be
  // specified to use a capture. Function allows registration
  // of event `listeners` on the event `target` for the given
  // event `type`.
  on: [ protocol, String, Function, [ Boolean ] ],

  // Function allows registration of single shot event `listener`
  // on the event `target` of the given event `type`.
  once: [ protocol, 'type', 'listener', [ 'capture=false' ] ],

  // Unregisters event `listener` of the given `type` from the given
  // event `target` (implementing this protocol) with a given `capture`
  // face. Optional `capture` argument falls back to `false`.
  off: [ protocol, 'type', 'listener', [ 'capture=false'] ],

  // Emits given `event` for the listeners of the given event `type`
  // of the given event `target` (implementing this protocol) with a given
  // `capture` face. Optional `capture` argument falls back to `false`.
  emit: [ protocol, 'type', 'event', [ 'capture=false' ] ]
})

No implementations are provided
Code above returns a set of polymorphic functions and a protocol object
Resulting functions dispatch on the argument with an index denoted in a signature via special protocol element.
Other array elements of the signature represent interface definition, and does not yet carry any special meaning. (You could use functions to highlight types or strings to highlight names or come up with something more creative).
Protocol implementations can be provided at any time from any scope that has access to defined protocol.

Implement protocol

Defined protocols can be implemented per type bases. Since everything in JS is an Object protocol implementation for Object type can be though as a default, since all values will automatically support protocol via that implementation:

/*jshint asi:true */
// module: ./event-object

var Event = require('./event-protocol'), on = Event.on

// Weak registry of listener maps associated
// to event targets.
var map = WeakMap()

// Returns listeners of the given event `target`
// for the given `type` with a given `capture` face.
function getListeners(target, type, capture) {
  // If there is no listeners map associated with
  // this target then create one.
  if (!map.has(target)) map.set(target, Object.create(null))

  var listeners = map.get(target)
  // prefix event type with a capture face flag.
  var address = (capture ? '!' : '-') + type
  // If there is no listeners array for the given type & capture
  // face than create one and return.
  return listeners[address] || (listeners[address] = [])
}

Event(Object, {
  on: function(target, type, listener, capture) {
    var listeners = getListeners(target, type, capture)
    // Add listener if not registered yet.
    if (!~listeners.indexOf(listener)) listeners.push(listener)
  },
  once: function(target, type, listener, capture) {
    on(target, type, listener, capture)
    on(target, type, function cleanup() {
      off(target, type, listener, capture)
    }, capture)
  },
  off: function(target, type, listener, capture) {
    var listeners = getListeners(target, type, capture)
    var index = listeners.indexOf(listener)
    // Remove listener if registered.
    if (~index) listeners.splice(index, 1)
  },
  emit: function(target, type, event, capture) {
    var listeners = getListeners(target, type, capture).slice()
    // TODO: Exception handling
    while (listeners.length) listeners.shift().call(target, event)
  }
})

Note: Implementing protocol for Object type is not a requirement.

Extend existing types

Existing types (prototypes or constructors / classes) may be extended to implement certain protocol by providing type specific implementation. For example our protocol functions would work with node.js’s EventEmitter objects, but in a funny way. Listeners registered by a standard API won’t be called when emitting events with protocol function and vice versa. To fix that we can implement our protocol for the EventEmitter type:

/*jshint asi:true */
// module: ./event-emitter

var EventProtocol = require('./event-protocol')
var EventEmitter = require('events').EventEmitter

EventProtocol(EventEmitter, {
  on: function(target, type, listener, capture) {
    target.on(type, listener)
  },
  once: function(target, type, listener, capture) {
    target.once(type, listener)
  },
  off: function(target, type, listener, capture) {
    target.removeListener(target, type)
  },
  emit: function(target, type, event, capture) {
    target.emit(type, event)
  }
})

Now this is cool, we managed to add support for our event abstraction to a type that was not designed to work with it without changing a single line of code. But this is just a tip of the iceberg, we could implement this protocol for more types, let’s try to do it for DOM elements:

/*jshint asi:true latedef: true */
// module: ./event-dom

var Event = require('./event-protocol')

Event(Element, {
  on: function(target, type, listener, capture) {
    target.addEventListener(type, listener, capture)
  },
  off: function(target, type, listener, capture) {
    target.removeListener(type, listener, capture)
  },
  emit: function(target, type, option, capture) {
    // Note: This is simplified implementation for demo purposes.
    var document = target.ownerDocument
    var event = document.createEvent('UIEvents')
    event.initUIEvent(type, option.bubbles, option.cancellable,
                      document.defaultView, 1)
    event.data = option.data
    target.dispatchEvent(event)
  }
})

Think of all the different JS frameworks (Backbone, YUI, Three.js, InfoVis, Raphaël, Moo Tools, …) that have their own flavored API for working with events, you could easily extend them to support our event protocol and make their abstractions interchangeable through the rest of the codebase (that makes use of protocols) without original code changes.

Multiple protocols

All the examples above showed how support for a given protocol may be added to a different types, but it’s not only that, any type may be extended to implement multiple protocols with absolutely no risks of naming conflicts. Here is pretty dummy, but still an example illustrating this point:

/*jshint asi:true latedef: true */
// module: ./installable

// Protocol for working with installable application components.
var Installable = protocol({
  // Installs given `component` implementing this protocol. Takes optional
  // configuration options.
  install: [ protocol, [ 'options:Object' ] ],
  // Uninstall given `component` implementing this protocol.
  uninstall: [ protocol ],
  // Activate given `component` implementing this protocol.
  on: [ protocol ],
  // Disable given `component` implementing this protocol.
  off: [ protocol ]
})

Installable(Object, {
  install: function(component, options) {
    // Implementation details...
  },
  uninstall: function(component, options) {
    // Implementation details...
  },
  on: function(component) {
    component.enabled = true
  },
  off: function(component) {
    component.enabled = false
  }
})

module.exports = Installable

Note: That even though both Event and Installable protocols define functions on and off. Also Object implements both still protocols, but there no conflicts arise and functions defined by both protocols can be used without any issues!

Summary

I hope you find this interesting & I’m looking forward to your feedback. All the code examples from this post can be found in the project repository. At the moment library is tested and can be used on node.js & browser, also there are no reasons why it would not work in other JS environments.

I personally think that protocols are much better feet for a JS language than redundant classes and I really wish ES.next was considering them instead!

(clojurescripting :intro)

2012-03-17T00:00:00-07:00

This is a first in a series of posts I’m planning to write about my clojurescript learning experience. If you have not heard of clojurescript yet, it’s a lispy programing language and a flavor of pretty popular clojure that compile targets JS.

You can find a many impressive demos and blog posts about clojurescript, but most of them assume you’re coming from clojure. Even though I read a book and have watched all the clojure screencasts, I never managed to dive into it. Now with clojurescript it’s even more tempting, so I decided to start over again and document every iteration.

Objectives

1. Reduce dependencies to minimum

Most of the JS developers are spoiled by using a language that has a brilliant runtime, it requires absolutely nothing to get started. Also, nature of JS (everything usually comes over the wire) makes us pretty picky when it comes to introducing dependencies. No matter if it’s a library or tool we’d like to avoid it, unless absolutely necessary. So in order to be comfortable hacking with clojurescript my first objective is to reduce dependencies I’ll use to an absolute minimum.

2. Write “Hello world”

Another objective is of course “hello world” application, which in this case will mean writing a clojurescript program that writes “hello world” into an html document.

Start a package

First of all we need to create a project / package. Assuming you know and love npm first thing you will consider doing is finding an equivalent for a given language, which turned out to be a leiningen! There are multiple ways one can install it. I used brew install leiningen since homebrew is a package manager of my choice. Once leiningen is installed, lein new clojurescripting can be run to generate a blank package. Generated project.clj file is a package descriptor containing package metadata, equivalent of package.json in JS. After some tweaking I ended up with something like this:

(defproject clojurescripting "1.0.0-SNAPSHOT"
  :description "Learning clojurescript"
  :url "http://documentup.com/gozala/clojurescripting/"
  :license { :name "MIT"
             :url "http://jeditoolkit.com/LICENSE" })

Write some code

The most basic thing I could think of was an alert dialog with “Hello World” message in it. Note that it requires calling JS function out from the clojurescript, which turned out to be trivial:

(ns clojurescripting.core)

(js/alert "Hello World!")

Compile to JS

Now that the code is there, we need to compile it to JS. Clojurescript quick start document describes how to do that manually, but luckily there is a lein-cljsbuild plugin for leiningen which can be used to automates this process. To do that you need to configure project.clj accordingly:

(defproject clojurescripting "1.0.0-SNAPSHOT"
  :description "Learning clojurescript"
  :url "http://documentup.com/gozala/clojurescripting/"
  :license { :name "MIT"
             :url "http://jeditoolkit.com/LICENSE" }
  :plugins [[lein-cljsbuild "0.1.2"]]
  :cljsbuild { :builds [{ :source-path "src"
                          :compiler { :output-to "lib/app.js"
                                      :optimizations :whitespace
                                      :pretty-print true }}]})

Once project is configured we need to tell leiningen to install all the dependencies by running lein deps. The best way to compile cljs to JS is by running lein cljsbuild auto which will watch source files and automatically recompile to JS on changes. This way ergonomics of refresh-driven development is preserved when working in clojurescript.

Unfortunately I had to straggle for some time before I figured out why JS file was not generated. The problem is that lein new generates core.clj which is a clojure file not a clojurescript one, so one needs to make sure to rename it to core.cljs instead.

Play

In order to play with a result I needed an html page, so I’ve created most basic index.html in the root of the project directory:

<body></body>
<script src=./lib/app.js></script>

As you would expect opening it in a browser displayed alert dialog. But since I wanted to write more clojurescript code I have decided to advance my example just a little bit so it writes “Hello World” into document body:

(ns clojurescripting.core)

(defn set-html
  "Sets `.innerHTML` of the given tagert element to the give `html`"
  [target & html]
  (set! target.innerHTML (apply str html)))

(defn set-text
  "Sets `.textContent` of the given `tagret`  to the given `text`"
  [target & text]
  (set! target.textContent (apply str text)))

;; Ineject "Hello world!" into document body.
(set-html document.body
          "<div style='background: black; color: white;'>"
          "<p>Hello world!</p>"
          "</div>")

Just a refresh and changes are applied! In fact, clojurescript has a better alternative than page refresh but that’s topic of the next post!

Summary

Overall I’m pretty happy with amount of tooling I had to use in order to write this basic clojurescript powered page. I also really liked leiningen’s plugin system and I hope npm will get something similar at some point in a future. Working with pure JS and DOM on the page is seamless and ergonomics of refresh driven development is preserved! On the flip side, I found leiningen to be a painfully slow (as it’s written in clojure and runs on JVM). Hopefully they will speed it up by switch to clojurescript on node.js or clojure on bare metal sometime in a future.

Code used in this post can be found under my clojurescripting repository on github.

namespaces

2012-03-15T00:00:00-07:00

I have previously blogged about inability of haveing properties with limited, controlled accessibility & described a hack I prototyped to overcome this limitation. Unfortunately I have failed to explain why one would need such properties, so I plan to explain it in this post. In addition, we already have a WeakMaps in spidermonkey (and also in V8) that made it possible to implement sharable privates in a proper way, you’ll find more details about how we did it in Add-on SDK also in this post.

Why do we need privates

Sometimes program is written to work with third party, potentially malicious, code which may use variety of attack vectors in order to escalate privileges and do something harmful. This is exact scenario we have in add-on SDK, where we wrap sudo powered browser internals into higher level APIs with reduced capabilities, such that by looking at add-on’s module graph we’re able to say what it’s capable of doing. For example if add-on only requires notifications module we know that at most it can spam user with notifications. Now if we’ve just stored sudo powered components used to implement these API under pseudo private (_ prefixed) properties we would’ve had no way of saying what it’s capable of. There would be no guarantee that add-on code won’t use those pseudo privates to wipe users hard drive (accidentally or intentionally).

This is not the only scenario where one would need to have controlled access to the implementation details. Any JS library may be used in an environment where it’s exposed to code that wishes to override it’s behavior by monkey-patching it’s pseudo privates, which is absolutely fine as long as, there is no other code that tries to do the same in a conflicting way. It’s not to say that we should be paranoid about it, it’s just there may be things that are not meant to be exposed in order to guarantee desired behavior.

Controlled access via namespaces

In Add-on SDK we have a namespace module, that may be used to create a namespace functions. These functions are used to access any objects namespaced sub-objects where properties, that are not part of public interface, may be saved:

let { ns } = require('namespace')
let foo = ns()

foo(myObject).secret = secret

Now only parties that have access to both foo and myObject are capable of seeing a secret. This approach allows us to create groups of internal properties that may be shared with other components of the program by giving access to the namespace functions. Also note that not only we can use foo namespace with multiple objects, but we also can use multiple namespaces with the same object:

let bar = ns()

bar(myObject).baz = secret.baz

This way we can create namespace per role that objects play in the program and associate properties to that objects in groups based on the roles. As a side effect we also eliminated naming conflicts with in the role object plays as properties are defined in different namespaces. This is very powerful, as we can define event emitter and many other APIs that may work with a same object safely no matter what’s their prototypes or own properties look like and no matter how many roles they play at the same time in program.

Under the hood

Implementation under the hood is very trivial. In nutshell namespace functions are sugared wrappers around WeakMap instances, each holding a reference to a WeakMap instance, which are used to map objects to an associated “namespaced sub-objects”, where namespaced properties are stored. Also, since objects are used as keys associated “namespaced sub-objects” can be claimed by garbage collector as soon as objects are collected. As an early adopters we had to face a platform bug requiring an ugly workaround, but now it’s fixed and it’s just matter of time when it ships!

Future

While this is a good enough solution that we plan to migrate all the existing SDK code to, it still has limitations. For example properties are instance specific, or in other words namespaced properties of ancestors (objects in the prototype chain) are not inherited and have to be explicitly accessed, which may feel bit awkward:

let decedent = Object.create(myObject);
'secret' in foo(decedent) // => false

In a future we may have even better solution via private names, implementation for spidermonkey is in progress so we’re looking forward!

JS Guards

2011-07-03T00:00:00-07:00

I’ve written micro library, to solve a very common problem in JS code today. Most likely following example looks familiar:

function Constructor(options) {
  options = options || {}
  if (typeof options.url !== 'string')
    throw new TypeError('url is required!')

  // Finally, your logic here!
}

First of all, code is wrong. If falsy value is passed to a function it will misbehave. Secondly, boilerplate code makes it less readable. It takes three lines to validate just one property (very likely to have more) of the given data structure. Finally, validation code does not even belongs there, as it has nothing to do with an actual logic of a function.

What we actually need is an annotation enforcing certain constraints. We don’t have annotations in JS, but we have functions that are omnipotent! Just about everything can be solved everything using functions.

function ConstructorOptions(options) {
  if (options === null || options === undefined)
    options = {}
  if (typeof options !== 'object')
    throw new TypeError('Options must an object!')

  if (typeof options.url !== 'string')
    throw new TypeError('url is required!')
}

function Constructor(options) {
  options = ConstructorOptions(options)

  // Your logic here!
}

This example is much better, because reader of Constructor function can concentrate on function logic, without being distracted by a validation boilerplate. In addition, options’ validation code can be independently unit tested and in some cases reused.

JS guards, is a small library that takes this idea to the next level. It provides declarative and recomposable way to define data type & data structure validations.

var guards = require('https://raw.github.com/Gozala/guards/v0.3.0/guards.js')

// Define a { x, y } data structure, where x and y fallback to 0.
var Point = guards.Schema({
  x: guards.Number(0),
  y: guards.Number(0)
})

// Define any type of guard as a function
function color(value) {
  // If validates just return value
  if (typeof value === "number" && value <= 255 && value >= 0) return value
  // If not throw TypeError
  throw new TypeError("Color is a number between 0 and 255")
}

// Define a [0-255, 0-255, 0-255] data structure guard.
var RGB = guards.Tuple([ color, color, color ])

// Compose data structure out of existing guards.
var Segment = guards.Schema({
  start: Point,
  end: Point,
  color: RGB,
})

Segment({ end: { y: 23 }, color: [17, 255, 0] })
// { start: { x: 0, y: 0 }, end: { x: 0, y: 23 }, color: [ 17, 255, 0 ] }
Segment({ start: 0, end: { y: 23 }, color: [17, 255, 0] })
// TypeError: Object expected instead of number `0`
Segment({ color: [ 10, 40, '30' ]})
// TypeError: Color is a number between 0 and 255

You can use this library in browsers, jetpack and nodejs. Also, you can try it out right now via interactive example on JSFiddle. For more details and examples check out library documentation. Finally, there is a guards proposal for EcmaScript and chances are we’ll get a syntax sugar some day (It will take a while though as it’s not even in Harmony).

As always I’m more than happy to hear your feedback!