I really like this essay entitled “On the Criteria To Be Used in Decomposing Systems into Modules” from Communications of the ACM, Vol. 15, No. 12, December 1972 pp. 1053 – 1058 Copyright © 1972, Association for Computing Machinery Inc.

It obviously deals with the criteria that should be used when modularizing a system, especially one with more than 10,000 LOC (which is not very much at all). The essay concludes that of the two major ways in which modularity can be thought of — dividing a system based on procedural tasks in the problem statement, and dividing it based on information hiding — the second is almost invariably the way to go.

What does it mean to divide a system based on information hiding? What they mean is that you must consider the components of the system with respect to their responsibilities. Each component should have only one responsibility. That way, changes to the mechanisms therein will not affect other components, and that is crux of modularity. Each module is then like a service that can be called upon at any time during the life of the program. If we use the first method of modularizing, on the other hand, this does make the code easier to understand inasmuch as it is divided based on logical steps, but how does its modularity really speak to the needs of a changing system?

When working with large code bases that use multi-tiered architectures and a dissing array of technologies — say, for example, web applications that have a DB layer, stored SQL procedures, a data layer in Java using JDBC, application layers written in any of Java, C#, Python, Perl (not that I know Perl), interfaces heavily reliant on JavaScript and other front-end technologies, passing information through message queues, marshaling XML thereof, load balancers between the layers in the architecture, and all that jazz, and on top of that, managing all of this distributed across servers, across sites, across continents, all while keeping things cheap, fast and green — when working with this, there’s a tremendous amount of effort involved in bootstrapping the systems. I mean getting the database going, and all the processes started and talking to each other.

And invariably there’s someone who wants to do this in an unexpected environment — say a Windows machine and they want to use SQLite when all of this was designed for Linux using an Oracle DBMS. Well, as much as we’d like to make things scalable and portable, this often poses a problem. So the poor sap who’s doing this for the first time is now stuck with writing the documentation for the next guy.

Anyone who’s done something like this knows all too well that this kind of thing is no simple task; you need to read a plethora of documentation for the individual systems, try, try, and try again until things start working. It can be a tremendously tedious task that often becomes repetitive due to trial and error and, by the end, everything is a huge garble in your head and you have no idea how you got to be where you are.

So how do you document your procedures? You can’t document while your working; things are too repetitive, you’d be changing the document far too often which results in frustration, wasted time and is severely error-prone. You certainly can’t document after the fact; I don’t care if you’re Jesus, you’re just not going to remember that at 2:19 on Friday morning you added environment-variable `ding-bat` and removed `/gimme-a-break` from your CLASSPATH.

So what do you do?

So I just got back an assignment for my Database System Technologies course which was focused on the PostgreSQL buffer management system and, although I expected a D+ or C due to database configuration issues, I actually got an A. I think the reason is that, although I had incorrect answers, I indicated that they were indeed wrong and I explained what the correct answers should approximately be, thereby demonstrating my understanding of the material and grasp of concepts. And lo and behold, an A.

So this is extremely troubling… read this Slashdot Post. The gist of it is that a Wikipedia article contained some faulty information which was propagated to reputable media outlets. When the accuracy of the article was questioned, those same (reputable) media were cited in defense; ergo the cyclic inconsistency.

This particular incident was minor, but this sort of stuff happens at the professional-journal level as well, where faulty data via poor experiments or just bad practices is published in scientific journals and is then propagated elsewhere. As rigorous as we try to be in the sciences, and even the maths, we’re still far from a sublime system.

Here’s an awesome set of visualizations of inter-journal citations across several fields of study. http://well-formed.eigenfactor.org/. Stolen from Greg Wilson’s blog.

So I emailed the A&S department at Harvard asking some questions regarding admission requirements and scholarship opportunities. And here’s the response summary: even for a degree in Arabic grammar, I need advanced reading knowledge of French and German, Harvard offers no funding for masters students internally.

I’ve been seriously considering a one-year graduate degree (Masters) at Harvard in the Graduate School of Arts & Science, specifically in the Near Eastern Languages and Civilizations program of study, focusing on Arabic and Islamic Studies. I’d be writing my dissertation in liturgical-Arabic grammar and would take courses accordingly. I’m thinking about the 2010 enrolment session (September of that year).

But that would mean leaving home for a year; my wife. Also, the total fee comes to around $50,000 USD!

Based on my understanding, here’s a very high level description of how part of the buffer management system of the storage layer in PostgreSQL (version 4.7.13) works. Oh and by the way, the comments in PostgreSQL, as I’ve seen them, are superb!

So there’s this thing called a `BufferTag` struct which holds enough data to tell the system where the page (i.e. the data) lives (think inodes). Now all these `BufferTags` we’re using right now live in a giant hash table where the system can do lookups, inserts, deletes and pass on the data in the page to higher levels of the database.

Good, now `BufferTag`s have a wrapper called a `BufferDesc` struct which holds metadata, like the buffer ID, the number of references to it, whether it’s dirty, and of course a `BufferTag`. The `BufferDesc` is basically what gets passed around.

Now there’s this doubly linked circular list of `BufferDesc` that represents buffers not in the hash table (not in use).

So let’s say we have a database going and I’m using a page which is already in a buffer in the hash table. If I no longer need the page, I decrease the ref count on the buffer. And if the ref count reaches zero? Then I remove the `BufferTags` from the hash table and move the corresponding `BufferDesc` to the linked list of free buffers.

Okay, now if I want a page and it’s already in the hash table, then I just increase the buffer’s ref count. And if it’s not in the hash? Well then I grab it out of the free buffers linked list. And if it’s not there either (it got flushed to disk)? Then I grab a buffer from the front of the free list, writing out the data if it’s dirty, and put my new page in there.

If you notice, the order in which we put buffers into the linked list is crucial; it determines which buffer is at the front of the list and hence which gets evicted if I can’t find my buffer anywhere.

Details:

For buffers, there’s a local ref count and a global one; the global one is just on or off while the local one is actually the true counter. This eliminates the need to keep acquiring a lock every time you need to change the ref count… cool.

When `BufferTag` go into the hash table. they’re first wrapped in a `BufferLookupEnt` struct.

Okay wait, the circular linked list is actually just an array that can be indexed using a buffer’s ID. But this is an array of `BufferDesc` structs and each one has a link to a previous and next. So it’s both a simple array that can be indexed, and a doubly linked circular list… cool, eh? So don’t let it confuse you! It’s two data structures in one. So something could be taken out of the link list but it could still be in the array (and in fact this happens). The array contains all the buffers, whereas the underlying linked list shrinks and grows.

For one of my Near East courses, I’m writing a critique of one of the chapters in a book of advanced classical Arabic grammar. I consider myself – and I hope it never turns into conceit – to be well versed in the field and I’ve accordingly done much original and non-original research resulting in essays and unpublished textbooks (which I’ve lost over the years). But I’ve never critiqued someone else’s work before.

Anyways, in doing this critique and in doing original research, I discovered how important it is to be able to think critically. That is, to analyze and objectively synthesize conclusions from a work. I actually think that the University of Toronto has prepared me well in this respect… although, if it hasn’t, I wouldn’t really be smart enough to tell, would I?

Anyways, even though the university has done a good job of this, I think critical thinking is at the core of education and universities should concentrate their efforts on this and those corrupt, pigeon-holing, government-regulated, corrupt pieces of shit we call schools should be geared towards it as well.

• or can name several cooking techniques
• where tea and coffee come from
• the largest mountain on land, on Earth, and in the solar system
• and can make the simple connection between Yoko Ono and the Beatles
• of the Hawthorne Effect