Coding backwards - a design technique

The worst design formula I ever encountered was by courtesy of a group of interns, ten years ago. When given the problem statement, they proceeded to convert the top level business entity names into classes and the verbs into their methods. Needless to say, what came out of the exercise was an abomination: it looked beautifully elegant as a class diagram, but lacking any consideration of how the thing might work at implementation level, it (and the code it inspired) had to be thrown out with prejudice.

I have seen many such formulas over the intervening decade. They all had one thing in common: they were formulas. In that they substitute the hard work, i.e. the thinking, with a series of steps based on some theory (or worse yet, based on a collection of  buzzwords masquerading as a theory). I myself have tried to come up with a few over the years. Not because I dislike thinking, but as I have come to realize, because I hate chaos.

A design technique, in my opinion, should not try to reduce the amount of thinking required. It should instead try to reduce the amount of chaos in the process so that the developer can do his thinking methodically. Here I outline a process that I have been using for the past four years. It has worked equally well with C++ (my mother tongue of ten years), PHP and especially JavaScript (my recently adopted second language).


Before I found Python, I was a Perl afficionado; and those who know the temperament of that language would probably already know its remarkable resistance to attempts at structure. You may start with the best process, standards (and intentions) in mind and three months into the project, Perl has a tendency to suddenly jump out of the repository and proclaim "Ha ha, look at me I'm a mess!". So it didn't quite work there. The Pythonian way on the other hand, intrinsically encourages some of the things I'm trying to achieve with the technique below, so I found that my Python code often came out looking better (and being more maintainable) in general even without the benefit of this technique.

Enough preamble now. Let us have a look. Compared to most people I know, I develop backwards: while most people write classes, modules and functions and then use them, I use them in code before I write them.

I recently introduced IndexedDB to a hybrid JavaScript application. In plain vanilla JavaScript, IndexedDB usage looks a bit verbose:

	// referece: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
	var dbname = "companydb";
	var dbversion = 1;
	var db = null;
	// open the database, do first time initializations if necessary
	function openidb(callback) {
	var request = window.indexedDB.open(dbname, dbversion);
	request.onsuccess = function(e) {
	console.log('database opened successfully');
	db = request.result;
	callback(); // all db ops have to happen after onsuccess
	};
	request.onerror = function(e) {
	console.error('failed to open database: ' + e.target.error.message);
	};
	request.onupgradeneeded = function(e) {
	console.info('previous version of database schema detected. upgrading...');
	db = e.target.result;
	var employees = db.createObjectStore('employees', { keyPath: 'empid' });
	employees.createIndex('employee id', 'empid', { unique: true });
	employees.createIndex('department', 'deptid');
	employees.createIndex('full name', 'name');
	var departments = db.createObjectStore('departments', { keyPath: 'deptid' });
	departments.createIndex('department id', 'deptid', { unique: true });
	departments.transaction.oncomplete = function(e) {
	var tx = db.transaction(['departments'], 'readwrite');
	tx.objectStore('departments').add({ deptid: 1, name: 'Accounting' });
	tx.objectStore('departments').add({ deptid: 2, name: 'Engineering' });
	console.info('...database schema upgrade done.');
	};
	};
	}
	// write some data
	function testidbwrite(callback) {
	console.log('writing data...');
	var emp1 = { empid: 1, name: 'John Doe', deptid: 1 };
	var emp2 = { empid: 2, name: 'Jeremy Doe', deptid: 2 };
	var tx = db.transaction('employees', 'readwrite');
	tx.oncomplete = function(e) {
	console.log('...writing data complete');
	callback();
	};
	tx.onerror = function(e) {
	console.error('failed to write data: ' + e.target.error.message);
	};
	tx.objectStore('employees').add(emp1);
	tx.objectStore('employees').add(emp2); // each returns a request to which
	// you can attach onerror/onsuccess, but i'm only catching the full
	// transaction level errors and completion
	}
	// read written data
	function testidbread() {
	console.log('reading data...');
	var request = db.transaction('employees').objectStore('employees').get(1);
	request.onsuccess = function(e) {
	console.log('... and the data: ');
	console.log(request.result);
	};
	request.onerror = function(e) {
	console.error('failed to read data: ' + e.target.error.message);
	};
	}
	// since this is a test, deleting any older versions of the database
	// and starting froms scratch
	var request = window.indexedDB.deleteDatabase(dbname);
	request.onsuccess = test;
	request.onerror = test;
	function test() {
	openidb(function() {
	testidbwrite(function() {
	testidbread();
	});
	});
	}

view raw indexeddb.js hosted with ❤ by GitHub

I need to wrap this for a number of reasons. First, I don't allow boilerplate code where I work -- a library or a module may not force its user to do something it can do itself. Usage needs to be a lot more terse. Second, it has to fit into our existing application's event model.

Here's how I start: I assume the existence of a hypothetical module that meets all my requirements, and I start using it:

	// connectivity: ITERATION 1
	var database = new Database('companydb');
	database.connect(function(db) {
	if (db.error) {
	// handle error
	}
	// continue db ops
	});

view raw con1.js hosted with ❤ by GitHub

So we're assuming that the database already exists with the right schema? Can't do that with IndexedDB. The only time you get to meddle with the schema -- whether it is for the first time or during an application upgrade -- is when the onupgradeneeded event is triggered during a database open. So we need to know the schema at open/connect time.

	// connectivity: ITERATION 2
	var database = new Database(configs);
	database.connect(function(db) {...});

view raw con2.js hosted with ❤ by GitHub

That looks better. Now I'm going to pull out the configs in to a separate JSON file because schemas are better specified declaratively than constructed procedurally.

	"configs: ITERATION 1":
	{
	"database": "companydb",
	"version": 3,
	"stores": {
	"departments": {
	"keyPath": "deptid",
	"indexes": {
	"departments-deptid": { "keyPath": "deptid", "unique": "true" }
	}
	},
	"employees": {
	"keyPath": "empid",
	"indexes": {
	"employees-empid": { "keyPath": "empid", "unique": "true" },
	"employees-deptid": { "keyPath": "deptid" },
	"employees-name": { "keyPath": "name" }
	}
	}
	}
	}

view raw con3.json hosted with ❤ by GitHub

Still we're missing the scripts or schema deltas for upgrading the database from the two previous versions. I'm not going to handle that problem in it's entirety right now, but here's one way I see it happening:

	"configs: ITERATION 2":
	{
	"database": "companydb",
	"versions": {
	"1": {
	"stores": {
	"departments": {
	"keyPath": "deptid",
	"indexes": {
	"departments-deptid": {
	"keyPath": "deptid",
	"unique": "true"
	}
	}
	}
	}
	},
	"2": {
	"stores": {
	"departments": {
	...
	},
	"employees": {
	...
	}
	}
	}
	}
	}

view raw con4.json hosted with ❤ by GitHub

So there's a snapshot of each version's schema in the database config file? Not going to be maintainable over the long run. Conversely, you could just have delta configs that gives you the differences since the previous version and then at the end have a full schema for the latest version. I'm leaning towards that latter solution. Otherwise the upgrade script will need to do diff's on the schemas on its own.

	"configs: ITERATION 3":
	{
	"database": "companydb",
	"version": 3,
	"stores": {
	"departments": {
	"keyPath": "deptid",
	"indexes": {
	"departments-deptid": { "keyPath": "deptid", "unique": "true" }
	}
	},
	"employees": {
	"keyPath": "empid",
	"indexes": {
	"employees-empid": { "keyPath": "empid", "unique": "true" },
	"employees-deptid": { "keyPath": "deptid" },
	"employees-name": { "keyPath": "name" }
	}
	}
	},
	"upgrades": {
	"1": "schema/upgrades/version1.js",
	"2": "schema/upgrades/version2.js",
	"3": "schema/upgrades/version3.js"
	}
	}

view raw con5.json hosted with ❤ by GitHub

The onupgradeneeded handler will have to load the upgrade files in the correct order and run the synchronously. For example: 0 -> 3: no need for a script, create the current schema. 2 -> 3: fetch version3.js and execute it. 1 -> 3: execute version2.js and version3.js in order. Upgrades are done procedurally because there may be more to them than just schema changes. Migration scripts in particular will need to look at existing data.

	// connectivity: ITERATION 3
	var database = new Database(JSON.parse(dbconfigs)); // assuming we're loading it directly from file
	database.connect(...);

view raw con6.js hosted with ❤ by GitHub

Now, how do I start writing data? I tried this first:

	// writing: ITERATION 1
	database.connect(onconnect);
	function onconnect(db) {
	if (db.error) {
	// handle
	return;
	}
	var data = { empid: 3, name: 'William Tell', deptid: 1 };
	var store = database.getStore('employees');
	store.add(data, function(tx) {
	if (tx.error) {
	// handle
	return;
	}
	});
	}

view raw write1.js hosted with ❤ by GitHub

Notice I've converted IndexedDB's result.callback = handler syntax to a more familiar operation(callback). If you come from a multi-threaded programming background, the former will look as if it has a race condition (in reality it doesn't because the JavaScript engine will execute function scope before handling asynchronous callbacks). I've also simplified the error communications with function(result) { if (result.error) ... } where the result depends on your operation.

What I'm not happy with: that database.getStore('storename') looks superfluous. IndexedDB's transaction model is well suited for transactions spanning multiple stores, but for most day-to-day operations, I'm going to make the following simplification:

	// writing: ITERATION 2
	var data = { empid: 3, name: 'William Tell', deptid: 1 };
	database.add('employees', data, function(tx) {
	if (tx.error) ...
	});

view raw write2.js hosted with ❤ by GitHub

So now the datastore is an argument in the add() call. Looks good enough to start.

But most operations in our app won't go like that. We need transactions. Here's how I'd like to see those handled:

	// transactions: ITERATION 1
	var tx = database.transaction();
	for (var i in hires)
	tx.add('employees', hires[i]);
	for (var i in fires)
	tx.remove('employees', fires[i].empid);
	tx.execute(function(tx) {
	if (tx.error) {
	}
	});

view raw write3.js hosted with ❤ by GitHub

I've assumed a remove(storename, key) function here. We can easily implement that in IndexedDB. Notice how we don't repeatedly specify store names anymore. Our module's transaction object will have to collect an array of all referenced stores and use that when creating the IndexedDB transaction. So that's an advantage of passing the store name into the operation. Here Transaction.add() will only add the operation to a list. It will only be executed via Database.add() during Transaction.execute().

One iteration seems good enough for that bit, but we could also do it this way:

	// transactions: ITERATION 2
	database.transaction([
	database.add('departments', marketing),
	database.add('employees', mktVP),
	database.add('employees', mktManager),
	database.add('employees', mktExecutive),
	]).execute(function(tx) {
	if (tx.error) {
	}
	});

view raw write4.js hosted with ❤ by GitHub

Looks more terse, but implementation might be more complicated. We will need the Database to know the difference between calling add() normally (dispatches the request immediately) and calling add() within a transaction (collects and dispatches only when execute() is called). Plus it doesn't allow you to add items to the transaction within a loop, at least not directly. So for the first version of this module, I'm going to stick with the syntax from ITERATION 1.

Once I've gone through a similar process for the other operations I need (update and query, in this case), I'm ready to implement. If I've done everything right, the above snippets should just work once I included the completed module.