Irakli Gozalishvili

Write logic, not mechanics

2012-04-26T00:00:00-07:00

It strikes me that developers in JS community tend to choose patterns for solving recurring problems over abstractions.

If we are not busy with semicolon debates, we argue about widely used “callback pattern” for dealing with asynchronous API. Many have learned / invented ways to avoid “pyramids of doom”, but I believe they miss the point: pyramids are not the issue, it’s an indication that we have one.

In order to describe what I consider to be a real issues, I have to move back a little first:

Function

In mathematics, a function is a relation between a set of inputs and a set of potential outputs with the property that each input is related to exactly one output.

Black box

In science and engineering, a “black box” abstraction is used to model systems as set of components which can be viewed solely in terms of its input, output and transfer characteristics, without any knowledge of its internal workings. This components are opaque (black) boxes.

Bigger boxes can be created out of smaller ones just by describing a data flow with in them (connecting inputs and outputs):

This allows one to reduce details as necessary and change internal implementation of any box without affecting other parts of the system as long as transfer characteristics remain same.

Functions in JS

In programing functions are modeled around the same concepts, even though we messed up an input sets by adding implicit parts that may change over time. In JS function input set consists of:

Given arguments.
Pseudo-variable this.
Scope bindings.

Problems

1. No output

Since functions in JS are first class, they can be part of both input and output sets. Most of the asynchronous APIs take advantage of this fact and require special callback function argument for continuation passing:

#!/bin/javascript
require('fs').open(path, 'r+', function(error, fd) {
  // ...
})

As you can see such functions no longer have have any useful output, which means that they can’t be used for building systems as black boxes. Such functions don’t return values that can be passed over to other boxes.

2. Error handling on each step

I have heard many times people criticizing how in Java exceptions are caught, wrapped and re-thrown again. This reminds me of following:

#!/bin/javascript
function readJSON(path, callback) {
  fs.readFile(path, function(error, content) {
    if (error) callback(error);
    else callback(null, JSON.parse(content));
  })
}

Basically error propagation in “callback” style APIs is done manually. Note, that in some cases you may want to try catch actual function body as well.

3. Polygamy

It easy to end up with two types of functions: synchronous and asynchronous. While it’s possible to make sync function async it’s not the case other way round. This usually means that if one the functions had being converted to be asynchronous all of it’s users will have to be converted as well:

#!/bin/javascript
function readJSON(path) {
  var data = fs.readFileSync(path)
  return JSON.parse(String(data))
}

You can’t simply switch to readFile if becomes necessary.

4. Progress tracking

Finally if your function depends on multiple asynchronous inputs then you will have to manually track each.

#!/bin/javascript
function makeView(templateURI, dataURI, callback) {
  var template, data, pending = 2
  readURI(templateURI, function(error, content) {
    if (error) return callback(error)
    template = content
    if (!--pending) Mustache.render(template, data)
  })
  readURI(dataURI, function(error, content) {
    if (error) return callback(error)
    data = content
    if (!--pending) Mustache.render(template, data)
  })
}

Also note that this code assumes that readURI will call a callback only once, which is not guaranteed.

Describe logic not mechanics

Now consider our last example. Most of the code there is for handling mechanics rather than describing a logic, which feels absolutely very wrong. As a matter of fact actual logic can be expressed as:

#!/bin/javascript
function makeView(templateURI, dataURI) {
  var template = readURI(templateURI)
  var data = readURI(dataURI)
  return Mustache.render(template, data)
}

So, would not it be better to abstract timing out of logic when it’s not necessary rather than keep solving all this issues in each and every function ? As a mater of fact solution has being there for ages in a form of promises, but for some reason people and web standards tend to use callbacks instead. Maybe because they feel complicated, but that does not necessary has to be the case:

#!/bin/javascript
// Everyone knows how to write a function:
function sum(a, b) { return a + b }
console.log(sum(1, 2))  // => 3

// Working with promises should not require nothing more
// than marker telling that function can accept input in
// form of promises
sum = promised(sum)
console.log = promised(console.log)

// Will continue to work with plain old values
console.log(sum(1, 2))  // => 3

// Will also accept promises as arguments
var deferred = defer()          // make promise
var a = deferred.promise
var b = sum(a, 1)
var c = sum(b, 5)
console.log(c)                  // eventually prints => 17
deferred.resolve(11)            // fulfill promise

We should not be handling and propagating exceptions manually in each function, we should only handle them when we plan to recover:

#!/bin/javascript
// Utility funciton that throws exceptions
var raise = promised(function(_) { throw Error(_) })

var a = raise('Boom !')   // Now we got an exception
var b = sum(a, 2)         // Now it has propagated to b
var c = sum(b, 12)        // Now it has propagaed to c

// Finally if when ready we handle exception in computation
c.then(null, console.error) // => Error: Boom !

If we just want to group multiple values into one there is an Array for that no need to track progress of each eventual value if we just care about a group!

#!/bin/javascript
Array = promised(Array)
var results = Array(readAsync(a), readAsync(b))
console.log(sum.apply(sum, rusults))

Noticed a pattern ? We just write a logic, and if it needs to handle asynchronous input we wrap it into promised(logic).

Whats really important here is that such functions can be used to build systems in black boxes, since they do have input and output. Demonstration of that is an example from above (assuming that readURI returns promise):

#!/bin/javascript
function makeView(templateURI, dataURI) {
  var template = readURI(templateURI)
  var data = readURI(dataURI)
  // Assuming Mustache.render = promised(Mustache.render)
  return Mustache.render(template, data)
}

Implementing such a solution takes about 100 lines of code (ignoring comments), and that’s more or less what any other control flow library costs anyway. I wish all of you to have more time to concentrate on logic of your program instead of mechanics & small promise library may be a good first step!

protocol based polymorphism

2012-03-21T00:00:00-07:00

Not a long time ago I learned about clojure’s polymorphism constructs and protocols. I was so inspired by a porwer and flexibility of protocol based polymorphism that I decide to prototyped it for JS. In this post I will try to give you a taste of protocols and maybe even motivate you give them a try.

Rationale

In programing we usually write and consume various abstractions. Typically in OOP languages abstractions are defined via (class / object) interfaces and have a nasty expression problems. Imagine that you have A and B sets of abstractions and sets of implementations of those abstractions. A should be able to work with B’s abstractions, and vice versa without modifications of the original code. While it may not sound as a problem at first, it usually is in practice. Sometimes A can’t use B, either because they were not designed to work with each other as they were written by a different authors or because one is newer than the other. Either way such cases require code changes, which may be difficult because code is old, or complicated or has a license restrictions and there could be millions of other reasons. Any code hits these issue in some form and it’s just matter of time. When that happens we’re left only with a few possible solutions:

Feature detection

Typically this is a code that is written not in terms of abstractions, but entities, that do runtime branching by “feature detection”. Which may be a type (if (value instanceof Type)) or shape (if (value && typeof(value.length) === 'number')) based. This not only makes code harder to read & reason about, but it also closed. In other words every new abstraction will require rewrite of those entities, in order to accumulate more conditions and branches.

Wrappers

Typically this means that entities implementing abstraction A need to be wrapped by a “glue code” implementing abstraction from B and vice versa, if consumption is bidirectional. Unfortunately this introduces lot’s of incidental complexity as wrappers ruin identity & don’t compose (every new abstraction requires wrappers for all existing ones and vice versa). Finally problem and required changes grow progressively with a number of abstractions used.

Monkey patching

Typically this means that implementation of A abstraction is patched with a “glue code” implementing support for B abstraction. This still introduces complexity by ruining namespacing (different abstractions may have conflicting names). Again problem gets worth with an amount of code. Also in some languages this is not even possible.

Note: For more details I would recommend watching a “A quick overview of clojure protocols” by Stuart Halloway.

Protocols

In Clojure polymorphism is achieved using protocols. They provide a powerful way for decoupling abstraction interface definition from an actual implementation per type, without risks of interference with other libraries. Protocols allow to add polymorphic behavior to things that already exist without changing them. I’ll go into more details on protocols, but for code examples I will use my JS prototype implementation instead of clojure code.

There are several motivations for JS protocol library:

Provide a high-performance, dynamic polymorphism construct as an alternative to an existing object inheritance that does not provides any mechanics for dealing with name conflicts.
Provide the best parts of interfaces:
- Specification only, no implementation
- Single type can implement multiple protocols
Allow independent extension of types, protocols and implementations of protocols on types, by different parties.

Define protocol

A protocol is a named set of functions and their signatures defined by calling protocol function:

/*jshint asi:true */
// module: ./event-protocol

var protocol = require('protocol/core').protocol

// Defining a protocol for working with an event listeners / emitters.
module.exports = protocol({
  // Function on takes event `target` object implementing
  // `Event` protocol as first argument, event `type` string
  // as second argument and `listener` function as a third
  // argument. Optionally forth boolean argument can be
  // specified to use a capture. Function allows registration
  // of event `listeners` on the event `target` for the given
  // event `type`.
  on: [ protocol, String, Function, [ Boolean ] ],

  // Function allows registration of single shot event `listener`
  // on the event `target` of the given event `type`.
  once: [ protocol, 'type', 'listener', [ 'capture=false' ] ],

  // Unregisters event `listener` of the given `type` from the given
  // event `target` (implementing this protocol) with a given `capture`
  // face. Optional `capture` argument falls back to `false`.
  off: [ protocol, 'type', 'listener', [ 'capture=false'] ],

  // Emits given `event` for the listeners of the given event `type`
  // of the given event `target` (implementing this protocol) with a given
  // `capture` face. Optional `capture` argument falls back to `false`.
  emit: [ protocol, 'type', 'event', [ 'capture=false' ] ]
})

No implementations are provided
Code above returns a set of polymorphic functions and a protocol object
Resulting functions dispatch on the argument with an index denoted in a signature via special protocol element.
Other array elements of the signature represent interface definition, and does not yet carry any special meaning. (You could use functions to highlight types or strings to highlight names or come up with something more creative).
Protocol implementations can be provided at any time from any scope that has access to defined protocol.

Implement protocol

Defined protocols can be implemented per type bases. Since everything in JS is an Object protocol implementation for Object type can be though as a default, since all values will automatically support protocol via that implementation:

/*jshint asi:true */
// module: ./event-object

var Event = require('./event-protocol'), on = Event.on

// Weak registry of listener maps associated
// to event targets.
var map = WeakMap()

// Returns listeners of the given event `target`
// for the given `type` with a given `capture` face.
function getListeners(target, type, capture) {
  // If there is no listeners map associated with
  // this target then create one.
  if (!map.has(target)) map.set(target, Object.create(null))

  var listeners = map.get(target)
  // prefix event type with a capture face flag.
  var address = (capture ? '!' : '-') + type
  // If there is no listeners array for the given type & capture
  // face than create one and return.
  return listeners[address] || (listeners[address] = [])
}

Event(Object, {
  on: function(target, type, listener, capture) {
    var listeners = getListeners(target, type, capture)
    // Add listener if not registered yet.
    if (!~listeners.indexOf(listener)) listeners.push(listener)
  },
  once: function(target, type, listener, capture) {
    on(target, type, listener, capture)
    on(target, type, function cleanup() {
      off(target, type, listener, capture)
    }, capture)
  },
  off: function(target, type, listener, capture) {
    var listeners = getListeners(target, type, capture)
    var index = listeners.indexOf(listener)
    // Remove listener if registered.
    if (~index) listeners.splice(index, 1)
  },
  emit: function(target, type, event, capture) {
    var listeners = getListeners(target, type, capture).slice()
    // TODO: Exception handling
    while (listeners.length) listeners.shift().call(target, event)
  }
})

Note: Implementing protocol for Object type is not a requirement.

Extend existing types

Existing types (prototypes or constructors / classes) may be extended to implement certain protocol by providing type specific implementation. For example our protocol functions would work with node.js’s EventEmitter objects, but in a funny way. Listeners registered by a standard API won’t be called when emitting events with protocol function and vice versa. To fix that we can implement our protocol for the EventEmitter type:

/*jshint asi:true */
// module: ./event-emitter

var EventProtocol = require('./event-protocol')
var EventEmitter = require('events').EventEmitter

EventProtocol(EventEmitter, {
  on: function(target, type, listener, capture) {
    target.on(type, listener)
  },
  once: function(target, type, listener, capture) {
    target.once(type, listener)
  },
  off: function(target, type, listener, capture) {
    target.removeListener(target, type)
  },
  emit: function(target, type, event, capture) {
    target.emit(type, event)
  }
})

Now this is cool, we managed to add support for our event abstraction to a type that was not designed to work with it without changing a single line of code. But this is just a tip of the iceberg, we could implement this protocol for more types, let’s try to do it for DOM elements:

/*jshint asi:true latedef: true */
// module: ./event-dom

var Event = require('./event-protocol')

Event(Element, {
  on: function(target, type, listener, capture) {
    target.addEventListener(type, listener, capture)
  },
  off: function(target, type, listener, capture) {
    target.removeListener(type, listener, capture)
  },
  emit: function(target, type, option, capture) {
    // Note: This is simplified implementation for demo purposes.
    var document = target.ownerDocument
    var event = document.createEvent('UIEvents')
    event.initUIEvent(type, option.bubbles, option.cancellable,
                      document.defaultView, 1)
    event.data = option.data
    target.dispatchEvent(event)
  }
})

Think of all the different JS frameworks (Backbone, YUI, Three.js, InfoVis, Raphaël, Moo Tools, …) that have their own flavored API for working with events, you could easily extend them to support our event protocol and make their abstractions interchangeable through the rest of the codebase (that makes use of protocols) without original code changes.

Multiple protocols

All the examples above showed how support for a given protocol may be added to a different types, but it’s not only that, any type may be extended to implement multiple protocols with absolutely no risks of naming conflicts. Here is pretty dummy, but still an example illustrating this point:

/*jshint asi:true latedef: true */
// module: ./installable

// Protocol for working with installable application components.
var Installable = protocol({
  // Installs given `component` implementing this protocol. Takes optional
  // configuration options.
  install: [ protocol, [ 'options:Object' ] ],
  // Uninstall given `component` implementing this protocol.
  uninstall: [ protocol ],
  // Activate given `component` implementing this protocol.
  on: [ protocol ],
  // Disable given `component` implementing this protocol.
  off: [ protocol ]
})

Installable(Object, {
  install: function(component, options) {
    // Implementation details...
  },
  uninstall: function(component, options) {
    // Implementation details...
  },
  on: function(component) {
    component.enabled = true
  },
  off: function(component) {
    component.enabled = false
  }
})

module.exports = Installable

Note: That even though both Event and Installable protocols define functions on and off. Also Object implements both still protocols, but there no conflicts arise and functions defined by both protocols can be used without any issues!

Summary

I hope you find this interesting & I’m looking forward to your feedback. All the code examples from this post can be found in the project repository. At the moment library is tested and can be used on node.js & browser, also there are no reasons why it would not work in other JS environments.

I personally think that protocols are much better feet for a JS language than redundant classes and I really wish ES.next was considering them instead!

(clojurescripting :intro)

2012-03-17T00:00:00-07:00

This is a first in a series of posts I’m planning to write about my clojurescript learning experience. If you have not heard of clojurescript yet, it’s a lispy programing language and a flavor of pretty popular clojure that compile targets JS.

You can find a many impressive demos and blog posts about clojurescript, but most of them assume you’re coming from clojure. Even though I read a book and have watched all the clojure screencasts, I never managed to dive into it. Now with clojurescript it’s even more tempting, so I decided to start over again and document every iteration.

Objectives

1. Reduce dependencies to minimum

Most of the JS developers are spoiled by using a language that has a brilliant runtime, it requires absolutely nothing to get started. Also, nature of JS (everything usually comes over the wire) makes us pretty picky when it comes to introducing dependencies. No matter if it’s a library or tool we’d like to avoid it, unless absolutely necessary. So in order to be comfortable hacking with clojurescript my first objective is to reduce dependencies I’ll use to an absolute minimum.

2. Write “Hello world”

Another objective is of course “hello world” application, which in this case will mean writing a clojurescript program that writes “hello world” into an html document.

Start a package

First of all we need to create a project / package. Assuming you know and love npm first thing you will consider doing is finding an equivalent for a given language, which turned out to be a leiningen! There are multiple ways one can install it. I used brew install leiningen since homebrew is a package manager of my choice. Once leiningen is installed, lein new clojurescripting can be run to generate a blank package. Generated project.clj file is a package descriptor containing package metadata, equivalent of package.json in JS. After some tweaking I ended up with something like this:

(defproject clojurescripting "1.0.0-SNAPSHOT"
  :description "Learning clojurescript"
  :url "http://documentup.com/gozala/clojurescripting/"
  :license { :name "MIT"
             :url "http://jeditoolkit.com/LICENSE" })

Write some code

The most basic thing I could think of was an alert dialog with “Hello World” message in it. Note that it requires calling JS function out from the clojurescript, which turned out to be trivial:

(ns clojurescripting.core)

(js/alert "Hello World!")

Compile to JS

Now that the code is there, we need to compile it to JS. Clojurescript quick start document describes how to do that manually, but luckily there is a lein-cljsbuild plugin for leiningen which can be used to automates this process. To do that you need to configure project.clj accordingly:

(defproject clojurescripting "1.0.0-SNAPSHOT"
  :description "Learning clojurescript"
  :url "http://documentup.com/gozala/clojurescripting/"
  :license { :name "MIT"
             :url "http://jeditoolkit.com/LICENSE" }
  :plugins [[lein-cljsbuild "0.1.2"]]
  :cljsbuild { :builds [{ :source-path "src"
                          :compiler { :output-to "lib/app.js"
                                      :optimizations :whitespace
                                      :pretty-print true }}]})

Once project is configured we need to tell leiningen to install all the dependencies by running lein deps. The best way to compile cljs to JS is by running lein cljsbuild auto which will watch source files and automatically recompile to JS on changes. This way ergonomics of refresh-driven development is preserved when working in clojurescript.

Unfortunately I had to straggle for some time before I figured out why JS file was not generated. The problem is that lein new generates core.clj which is a clojure file not a clojurescript one, so one needs to make sure to rename it to core.cljs instead.

Play

In order to play with a result I needed an html page, so I’ve created most basic index.html in the root of the project directory:

<body></body>
<script src=./lib/app.js></script>

As you would expect opening it in a browser displayed alert dialog. But since I wanted to write more clojurescript code I have decided to advance my example just a little bit so it writes “Hello World” into document body:

(ns clojurescripting.core)

(defn set-html
  "Sets `.innerHTML` of the given tagert element to the give `html`"
  [target & html]
  (set! target.innerHTML (apply str html)))

(defn set-text
  "Sets `.textContent` of the given `tagret`  to the given `text`"
  [target & text]
  (set! target.textContent (apply str text)))

;; Ineject "Hello world!" into document body.
(set-html document.body
          "<div style='background: black; color: white;'>"
          "<p>Hello world!</p>"
          "</div>")

Just a refresh and changes are applied! In fact, clojurescript has a better alternative than page refresh but that’s topic of the next post!

Summary

Overall I’m pretty happy with amount of tooling I had to use in order to write this basic clojurescript powered page. I also really liked leiningen’s plugin system and I hope npm will get something similar at some point in a future. Working with pure JS and DOM on the page is seamless and ergonomics of refresh driven development is preserved! On the flip side, I found leiningen to be a painfully slow (as it’s written in clojure and runs on JVM). Hopefully they will speed it up by switch to clojurescript on node.js or clojure on bare metal sometime in a future.

Code used in this post can be found under my clojurescripting repository on github.

namespaces

2012-03-15T00:00:00-07:00

I have previously blogged about inability of haveing properties with limited, controlled accessibility & described a hack I prototyped to overcome this limitation. Unfortunately I have failed to explain why one would need such properties, so I plan to explain it in this post. In addition, we already have a WeakMaps in spidermonkey (and also in V8) that made it possible to implement sharable privates in a proper way, you’ll find more details about how we did it in Add-on SDK also in this post.

Why do we need privates

Sometimes program is written to work with third party, potentially malicious, code which may use variety of attack vectors in order to escalate privileges and do something harmful. This is exact scenario we have in add-on SDK, where we wrap sudo powered browser internals into higher level APIs with reduced capabilities, such that by looking at add-on’s module graph we’re able to say what it’s capable of doing. For example if add-on only requires notifications module we know that at most it can spam user with notifications. Now if we’ve just stored sudo powered components used to implement these API under pseudo private (_ prefixed) properties we would’ve had no way of saying what it’s capable of. There would be no guarantee that add-on code won’t use those pseudo privates to wipe users hard drive (accidentally or intentionally).

This is not the only scenario where one would need to have controlled access to the implementation details. Any JS library may be used in an environment where it’s exposed to code that wishes to override it’s behavior by monkey-patching it’s pseudo privates, which is absolutely fine as long as, there is no other code that tries to do the same in a conflicting way. It’s not to say that we should be paranoid about it, it’s just there may be things that are not meant to be exposed in order to guarantee desired behavior.

Controlled access via namespaces

In Add-on SDK we have a namespace module, that may be used to create a namespace functions. These functions are used to access any objects namespaced sub-objects where properties, that are not part of public interface, may be saved:

let { ns } = require('namespace')
let foo = ns()

foo(myObject).secret = secret

Now only parties that have access to both foo and myObject are capable of seeing a secret. This approach allows us to create groups of internal properties that may be shared with other components of the program by giving access to the namespace functions. Also note that not only we can use foo namespace with multiple objects, but we also can use multiple namespaces with the same object:

let bar = ns()

bar(myObject).baz = secret.baz

This way we can create namespace per role that objects play in the program and associate properties to that objects in groups based on the roles. As a side effect we also eliminated naming conflicts with in the role object plays as properties are defined in different namespaces. This is very powerful, as we can define event emitter and many other APIs that may work with a same object safely no matter what’s their prototypes or own properties look like and no matter how many roles they play at the same time in program.

Under the hood

Implementation under the hood is very trivial. In nutshell namespace functions are sugared wrappers around WeakMap instances, each holding a reference to a WeakMap instance, which are used to map objects to an associated “namespaced sub-objects”, where namespaced properties are stored. Also, since objects are used as keys associated “namespaced sub-objects” can be claimed by garbage collector as soon as objects are collected. As an early adopters we had to face a platform bug requiring an ugly workaround, but now it’s fixed and it’s just matter of time when it ships!

Future

While this is a good enough solution that we plan to migrate all the existing SDK code to, it still has limitations. For example properties are instance specific, or in other words namespaced properties of ancestors (objects in the prototype chain) are not inherited and have to be explicitly accessed, which may feel bit awkward:

let decedent = Object.create(myObject);
'secret' in foo(decedent) // => false

In a future we may have even better solution via private names, implementation for spidermonkey is in progress so we’re looking forward!

JS Guards

2011-07-03T00:00:00-07:00

I’ve written micro library, to solve a very common problem in JS code today. Most likely following example looks familiar:

function Constructor(options) {
  options = options || {}
  if (typeof options.url !== 'string')
    throw new TypeError('url is required!')

  // Finally, your logic here!
}

First of all, code is wrong. If falsy value is passed to a function it will misbehave. Secondly, boilerplate code makes it less readable. It takes three lines to validate just one property (very likely to have more) of the given data structure. Finally, validation code does not even belongs there, as it has nothing to do with an actual logic of a function.

What we actually need is an annotation enforcing certain constraints. We don’t have annotations in JS, but we have functions that are omnipotent! Just about everything can be solved everything using functions.

function ConstructorOptions(options) {
  if (options === null || options === undefined)
    options = {}
  if (typeof options !== 'object')
    throw new TypeError('Options must an object!')

  if (typeof options.url !== 'string')
    throw new TypeError('url is required!')
}

function Constructor(options) {
  options = ConstructorOptions(options)

  // Your logic here!
}

This example is much better, because reader of Constructor function can concentrate on function logic, without being distracted by a validation boilerplate. In addition, options’ validation code can be independently unit tested and in some cases reused.

JS guards, is a small library that takes this idea to the next level. It provides declarative and recomposable way to define data type & data structure validations.

var guards = require('https://raw.github.com/Gozala/guards/v0.3.0/guards.js')

// Define a { x, y } data structure, where x and y fallback to 0.
var Point = guards.Schema({
  x: guards.Number(0),
  y: guards.Number(0)
})

// Define any type of guard as a function
function color(value) {
  // If validates just return value
  if (typeof value === "number" && value <= 255 && value >= 0) return value
  // If not throw TypeError
  throw new TypeError("Color is a number between 0 and 255")
}

// Define a [0-255, 0-255, 0-255] data structure guard.
var RGB = guards.Tuple([ color, color, color ])

// Compose data structure out of existing guards.
var Segment = guards.Schema({
  start: Point,
  end: Point,
  color: RGB,
})

Segment({ end: { y: 23 }, color: [17, 255, 0] })
// { start: { x: 0, y: 0 }, end: { x: 0, y: 23 }, color: [ 17, 255, 0 ] }
Segment({ start: 0, end: { y: 23 }, color: [17, 255, 0] })
// TypeError: Object expected instead of number `0`
Segment({ color: [ 10, 40, '30' ]})
// TypeError: Color is a number between 0 and 255

You can use this library in browsers, jetpack and nodejs. Also, you can try it out right now via interactive example on JSFiddle. For more details and examples check out library documentation. Finally, there is a guards proposal for EcmaScript and chances are we’ll get a syntax sugar some day (It will take a while though as it’s not even in Harmony).

As always I’m more than happy to hear your feedback!

Packageless modules

2011-06-16T00:00:00-07:00

Packages

Every time I write code in javasciprt for jetpack or nodejs I question myself: “Do we really need packages ?”. Even with amazing package manager like npm it still feels wrong to me for a few reasons:

1. Size and scope

Writing code for the web taught us keeping line numbers tight, to a bare minimum. It could be a platform limitation, but more importantly it’s a best practice that has multiple advantages:

You never overgeneralize by trying to solve each and every problem.
You reduce a surface for bugs, less lines => less bugs.
You build reusable building blocks, with a focus on one thing only. Also, such blocks are easy to mix and match.
Faster to download, parse, execute.

Most of the modern js libraries / frameworks consist of one file, solve one problem and do it well. MicroJS started a great job of promoting this trend by building a nice spot for discovering fantastic micro frameworks and libraries!

Package definition, on the other hand, implies wrapping collection of things into a single form. So no matter how great package manager is, packages will tend to grow in size attempting to solve bigger set of tasks.

2. Explicitness

Looking at source should be enough to understand what program does, what it depends on, etc.. Unfortunately this is not the case with packages as it requires looking at the package descriptors and understanding non trivial module search logic. To put it other way

bar = require('foo/bar')

expression depends on too many things: requirer module, package descriptor, existence of various files in various places. In jetpack module search logic ended up being very complex. NodeJS has a better story, but still there more than two axis involved. It should be simple & straight forward.

3. Simplicity

When I find some useful & well maintained module on the web, I just want to be able to use it straight away, without downloading and bundling with my code, or even worth creating a package out of it publishing into some centralized registry in order to finally be able to require it! And when I want to alter some functionality in third party module I don’t want to go through change / rename / publish routine, I just want to fork on github and require!

4. Interoperability

Packages are not for browsers, because they have dependencies that have their own dependencies etc.. All this dependencies are expressed in a separate package.json files and it’s impossible to interpret require without reading package descriptor or knowledge of a requirer and I won’t even go into details of search paths and dependency duplications. Only visible solution for using packages in browsers requires build step on each file change or a server that does this at runtime. It’s a big compromise to make even though we do builds before deployments.

5. Far from harmony

No packages are in harmony, just a simple modules with compile-time resolution and linking. Hopefully very soon, I heard that by end of this year, we will have a first class modules in JavaScript and that will be the ultimate way to distribute building blocks with built-in support from js engines, so closer we stick to the model cheaper will be migration.

Alternative

Web is awesome, it’s cross platform, fully distributed, yet connected via links / URLs.

I think we should be building ecosystem for building web, with a same spirit: cross-platform interoperable modules, that are fully distributed across the web, still connected via URLs. Each doing one thing only, but doing it well. No need for complicated module search logic or central authority for distribution! I’m very happy that, that’s exactly how modules are thought to work in Harmony.

Solution

In my spare time I have prototyped a tool called graphquire that makes it possible to write and use such modules, that will work across different platforms including: jetpack, nodejs and browsers. Unfortunately different platforms limitations made it impossible to require modules with an actual URLs, but it very close:

// Relative
require('./foo/bar')
require('./bla.js')
require('../baz')

// Absolute
require('http!foo.org/bar')
require('https!bla.org/baz.js)

You might have some constraints:

Long require statements:

Can be solved in many ways, starting from URL shortening services, finished with a private module registries, similar to MicroJS.
Updating require statements more than 1 place:

Not really an issue with right set of tools like replace for example, also editors and even graphquire might get build-in support for this. In any case that’s something we will have to face in harmony.
Content under URL can change:

This is true, but same can happen in package registries. We just need to make sure to require modules from URLs that are guaranteed to have the same content: modules under version tags, registries, own servers.

Please let me know what you think!

Addons in multi process future

2011-06-04T00:00:00-07:00

The most frequently asked questions on jetpack mailing list concern content scripts. It’s not obvious for users why these scripts are powerless, why modules don’t have access to the content’s dom and why they need to communicate between them by message passing. The short answer is: Mozilla platform is moving towards a model were separate processes are used for chrome, content and add-ons. Therefore SDK modules were designed with Electrolysis in mind, to guarantee painless transition to a new model. Much more details about this may be found in the documentation of content scripts.

Even though we won’t ship 1.0 with support for Electrolysis, some of us (Atul, Alex and me) have explored some ideas, but until few month ago I was still unsure we were heading a right path.

Several month ago I was reading Peter Michaux’s blog post about MVC Architecture for JavaScript Applications from which I’d like to quote following description of “Real MVC”:

“In a nutshell the classic MVC architecture works like this. There is a model that is at the heart of the whole thing. If the model changes, it notifies its observers that a change occurred. The view is the stuff you can see and the view observes the model. When the view is notified that the model has changed, the view changes its appearance. The user can interact with the view (e.g. clicking stuff) but the view doesn’t know what to do. So the view tells the controller what the user did and assumes the controller knows what to do. The controller appropriately changes the model. And around and around it goes.”

Even though I’ve been writing apps with MVC architecture for a while, it never occurred to me to look at jetpack in MVC.

Light side

Ironically, most of Add-on SDK APIs provide access to the things that are basically models, associated with a different views of the browser UI (like windows, tabs, widgets). Also, low level modules can be seen as controllers of two types:

Model observers that propagate changes to the associated views.
View observers that update associated models.

Such perspective also makes whole multi-process architecture switch way more simple. All we need to do is to move models to the other process and make sure that changes on views and models are synchronized via controllers.

Dark side

Unfortunately big part of the code base was not written with such MVC perspective, or in other words we don’t have a strict separation between models and outlined controller types.

Technical side

This part of the blog is probably too technical to enjoy, unless you plan to hack jetpack itself, so feel free to skip :)

Overall, everything boils down to few key components:

Pipe for delivering messages across processes, something like port on our worker objects.
Models that are serializeable and fully reconstructible via JSON. This is important as models are going to live in add-on processes. Model synchronization logic can be encapsulated in one function that just emits events on the port.
Controllers observing views, or in other words, event listeners that extract and serialize interesting data from events and emits to the pipe so that models can react.
Controllers observing model changes, via listeners on the pipe and updating views.

Outlined strong separation of roles is a good practice anyway, so I tried to stick to it to make transition to E10S easier.

Models that can be serialized / recreated: hotkey
Controllers that emit events on view (Browser UI) changes: tabs/observer, keyboard/observer, windows/observer
I don’t have examples of controllers that update views on model changes.

Finally here is a link to “in progress” implementation of models that I hope to get into jetpack in a future.

Yet another take on inheritance

2011-04-23T00:00:00-07:00

Every now and then you need to use inheritance in JavaScript. When you do, you suffer, because there’s just too many ways to do that and regardless of your choice, syntax noise will make your eyes bleed! Well you may consider saving your eyes by using a library, but this is not going to save you as you’ll be drown in an ocean of endless choices!

Choice is good, but when it’s in manageable quantities. Anyway only reasonable solution is: Yet another new library that does exact same thing in a slightly different fashion.

That’s a short story of how extendables micro-library started. It is super minimalistic. Function Extendable is only thing it exports, which is just like built-in Object. Only difference is extend own property that behaves similar to backbone’s Model.extend.

Extendable.extend may be called with an object containing properties that will be defined on the resulting constructor’s prototype. Which by the way, will inherit from the target function’s (function who’s extend was called) prototype. In addition all own properties of a target will be copied to the resulting constructor.

That’s actually all of what this library does! In contrast to backbone, library is built with ES5 in mind, which makes it aware of new goodies: non-enumerable, non-writable and non-configurable properties. This also means that it requires ES5 engine to run. But don’t get broken hearted yet, as you still can use it on IE6 (Please note that you’ll burn in hell if you do!) and friends, with a help of another micro-lib es5-shim.

Library is known to work in browsers via AMD loaders, in jetpack and in nodejs. Likely it’s going to work on any other CommonJS compliant platform as well. Finally here is a jsfiddle and source gist showing it off with a few examples:

/* vim:set ts=2 sw=2 sts=2 expandtab */
/*jshint asi: true undef: true es5: true node: true devel: true
         forin: true latedef: false supernew: true */
/*global define: true */


define('demo', function(require, exports, module, undefined) {

'use strict';

var Extendable = require("https://github.com/Gozala/extendables/raw/v0.1.2/lib/extendables.js").Extendable

var Base = Extendable.extend({
  inherited: function inherited() {
    return "inherited property"
  },
  overridden: function overridden() {
    return "property to override"
  },
  // No supers by default, use prototype and be proud, but if you really want
  // super get one!
  _super: function _super() {
    return Object.getPrototypeOf(Object.getPrototypeOf(this))
  }
})
// Adding static method.
Base.implement = function implement(source) {
  // Going through each argument to copy properties from each source.
  Array.prototype.forEach.call(arguments, function(source) {
    // Going through each own property of the source to copy it.
    Object.getOwnPropertyNames(source).forEach(function(key) {
      // If property is already owned then skip it.
      if (Object.prototype.hasOwnProperty.call(this.prototype, key)) return null
      // Otherwise define property.
      Object.defineProperty(this.prototype, key,
                            Object.getOwnPropertyDescriptor(source, key))
    }, this)
  }, this)
}

var b1 = new Base
console.log(b1 instanceof Base)              // -> true
console.log(b1 instanceof Extendable)        // -> true
console.log(b1.inherited())                  // -> "inherited property"

var b2 = Base()                             // -> Works same as without `new`
console.log(b2 instanceof Base)             // -> true
console.log(b2 instanceof Extendable)       // -> true
console.log(b2.inherited())                 // -> "inherited property"


var Decedent = Base.extend({
  constructor: function Decedent(options) {
    this.name = options.name;
  },
  overridden: function override() {
    // I'd rather copied `overridden` with a diff name overriddenBase for
    // example or used `Base.prototype.overridden.call(this)`
    // But this works as well :)
    return "No longer " + this._super().overridden.call(this)
  },
  // overriddenBase: Base.prototype.overridden
})
Decedent.implement({
  bye: function bye() {
    return "Buy my dear " + this.name
  }
})

var d1 = new Decedent({ name: "friend" })
console.log(d1 instanceof Decedent)       // -> true
console.log(d1 instanceof Base)           // -> true
console.log(d1 instanceof Extendable)     // -> true
console.log(d1.inherited())               // -> "inherited property"
console.log(d1.overridden())              // -> No longer a property to override
console.log(d1.bye())                     // -> "Bye my dear friend"

})
require.main("demo")

Finally, if you are a coffee fan like me, you’ll love the fact that regular coffee class syntax may be used to extend Extendable and it’s decedents. Also all coffee classes that extend Extendable can be extended further from javascript.

Shareable private properties

2011-04-11T00:00:00-07:00

In JavaScript it is not possible to create properties that have limited or controlled accessibility. It is possible to create non-enumerable and non-writable properties, but still they can be discovered and accessed. Usually so called “closure capturing” pattern is used to encapsulate such properties in lexical scope:

Unfortunately given approach does not works very well with OO inheritance:

Code readability degrades and some code duplication becomes necessary.
(See firs lines of Base and Decedent constructors).
Prototypes do not contain any properties and there for each instance gets it’s own fresh copy of them.
(Not very good for performance).
Own and inherited methods access different copies of private properties and there for changes do not propagate.
(boom method of Decedent illustrates the issue).
Access to a private properties can not be shared with a limited group.

I believe because of these reasons ”-” prefix was established as a “de facto” standard for private property names. Unfortunately it’s just a coding style that does not implies any property access restrictions.

In Jetpack we’ve explored different techniques to overcome issues mentioned above, but property access sharing to the limited groups of consumers remained painful and unobvious.

Recently I got an interesting idea inspired by private names proposal for ECMAScript (I recommend to reading Allen Wirfs blog post explaining details of the proposal). Here is an illustration of the idea with a same example:

Please note that this solves all the issues outlined. Also access to the privates can be shared with any group, just by giving them a KEY. After tinkering with this idea for some time, I’m happy to announce birth of namespace micro-library. Below you can see same example, but this time coded as a commonjs module and using mentioned library:

Library not only wraps outlined idea into a nice API, but also allows provides a way to use multiple namespaces with the same object.

Finally here is a jsfiddle link containing this example if you wan to play with it. Of course comments / feedback / ideas is more then welcome!

Evolving VS Adjusting

2010-10-20T00:00:00-07:00

We often have to make the choice between adjusting to conditions or paying a price for improving them. Usually it is safer to adjust, but still we have to be very careful we don’t miss an opportunity to improve. Probably the best way to do that is to compare the price and improvements.

Asynchronous Module Definition is one of the hottest topics on the CommonJS mailing list this days. Unfortunately my impression is that most of the participants of these debates forget to measure price VS improvement and this can be harmful.

I believe there are two sides:

People that believe that CommonJS modules are not well suited for in-browser usage (suggesting an alternative, “browser friendly” syntax).
People who believe that CommonJS modules are perfectly usable within browser without any changes.

I’m one among few others who uses CommonJS modules both on the server and in the browser already. With that in mind it’s not easy to convince me, that change in well established standard is absolutely necessary.

Issues with current modules

There were many arguments why current modules are not suited for in-browser usage, but unfortunately I did not manage to convince enough people to make a list of them, so I will summarise them as: “Non of the existing module loading techniques provides perfect debugging experience across the different browsers without involvement of a server component or a building step.”

Different view on these issues

Debugging story is not that bad actually (much better than it used to be not a long time ago), but it’s true, rarely, but still strange issues occur. In any case those are bugs in debuggers not with a modules. Only way to fix bugs is by report them. I’m pretty confident that firebug team is going to make their best, to make debugging experience as good as possible. I also hope that it’s true for a web inspector.
Dismissing a server as a show stopper is also inadequate IMO, since there are few other things like Offline Application Caching that would still depend on it and won’t work without. And in the end it just matter of running one simple command before starting development. For instance to hack on jetpack you need to run source bin/activate and I believe it was show stopper for anyone.

Issues with proposed changes

Proposed changes are not perfect either:

Some people will argue against this statement, but there is nothing in common between proposed Asynchronous Module Definition and existing module specs other then the CommonJS label. This will be a source of confusion and community fragmentation.

Below you can find a simple module being rewritten according to two different specs:

Asynchronous definition:

CommonJS module:
Asynchronous definition seem to encourage handwritten boilerplate but what is puzzling here is, why would any programmer prefer writing a boilerplate code by hand over writing a program that would do that for him. Yes it’s nice to refresh browser and be ready to go without any build step, but nothing prevents us form writing programs that support this workflow.
A most important issue is that the new proposal suggests to adjust to the unpleasant situation browser are in now, and what is even worse, it brings this mess to the server. Adjustment to broken tooling is only going to delay innovation! I truly believe that if we had had such a big adoption of CommonJS modules a few years ago, we would have perfect debugging story today (well maybe not in all browsers :). I also think that wide adoption of CommonJS modules already triggered some innovation and partially that is the reason why we might get native modules in harmony. And you know what ? Once we’ll get them, no magic will happen, as it did not happened when Web Workers arrived. It will take some time to get a decent tooling support. Again it should not stop innovation by trying to adjust new amazing platforms to the mess we are at with browsers.

Evolving

I think that the only sane thing to do now is to bring the innovation that happens in server side javascript back to the browser without compromising anything. With that in mind I’m planning to put a lot of effort in teleport, a tool that I wrote some time ago. It allowed few projects to share CommonJS modules across server and client. I don’t believe there is any perfect solution to all problems, but each problem has a solution and I hope that teleport will be a set of solutions addressing CommonJS module loading problems. The list of tasks I’ll be working on, is probably the best way to get a better idea what is it all about. Feel free to fork me, suggest new ideas or just comment here.