Perceptron = Goo + Blob = Glob

I'm working with Audrey Tang (of Pugs and Perl6 fame) on cleaning up the design of The Goo. One problem we identified is funky terminology which can act as a barrier to new developers.

So today during a design discussion with Audrey I decided to say goodbye to the term 'perceptron' for the more descriptive term 'blob' - but perceptrons are slightly more complicated than blobs - so Audrey suggested 'glob' as in Goo + Blob = Glob. In Perl parlance a glob is a generic handle on a thing - which fits what I'm trying to model. What's more:

Glob: 1. a drop or globule of liquid. 2. a usually rounded quantity or lump of some plastic or moldable substance

So that decided it. A Perceptron is now known as a Glob™.

Gedit - The Goo's Editor

I've finally found a text editor that does syntax highlighting, word completion and starts quickly.

Gedit with the help of the word completion plugin does what I'm looking for.

1 Medal = 500 Karma++

After every 500 points of karma++ you now receive a medal!

I scored my first medal a couple of weeks ago. Now it's really starting to feel like playing a game - which is all part of the plan.

I'm feeling progress being made --- which is one of the preconditions to feeling "flow".

Sensations Snookered? - The Careenium to the Rescue

So I'm modelling conscious thoughts with 'perceptrons' and lower level thoughts with 'sensations' - which are fired automatically by 'receptors'. But at what point do sensations fire a perceptron?

If you touch something hot the sensation quickly gets upgraded to a perceptron - Ouch! Some receptors have a deservedly high priority - but what about more subtle sensations that need to coagulate and mingle before finally firing? - hmmm .... tricky.

Douglas Hofstadter conjures an amazing metaphor in his book, "I am a Strange Loop" to describe this process. He likens the human brain to a frictionless snooker table where tiny balls, called sims (small interacting marbles) constantly career into each other. Some sims bounce off each other, while others may stick together and form larger 'simmballs'. He suggests that consciousess emerges from the simms amd simmballs crashing around what he coins the 'careenium'.

I like the idea! So does The Goo need a careenium? Well currently I'm programming a 'See' Sensation to find filenames in emails, comments, web pages etc. Take this for example: Login.pm. Most of you will recognise this as a filename. The "." visual sensation, the mixed case, and the filename suffix (.pm) all combine to give this impression. But is that enough to fire a perceptron? Are there enough simms (i.e., sensations) for a simmball?

If you're a Perl programmer you will "see" a Perl module thanks to the ".pm" suffix and your mental model of what it is to be a Perl module. But this still may not be enough for a perceptron to fire. If Login.pm appears in an email from your boss, however, he may be referring to the Login.pm module that you wrote and suddenly there is a mental collision between your environment - the display of Login.pm in pixels on your screen and your internal simmball for Login.pm. Pop! And a perceptron is born -> Login.pm.

The firing process, in this example, is a combination of your current context, your memory (the trail) and the sensory forces in your environment (communicated via receptors). There's lots to do so back to implementing but for version one I'll settle for a very simplistic careenium.

Sensations need Receptors

I've decided I need to model "sensations" as well as "perceptrons" but the problem now is what picks up a sensation? Here again psychology offers some helpful terminology: receptor. Just like the receptors in the back of your retina - some sensors fire automatically in response to a stimulus. These low level sensors are important for classifying low level sensations (e.g., ___ ).

So practically speaking, in The Goo a Receptor generates Sensations which in combination with other Sensations may be picked up by a Sensor that may in turn "fire" a Perceptron (e.g., I see a red line).

Snipping The Gordian Knot

I recently painted myself into a corner with the design rule "everything must be a perceptron". The problem is that just like with human perception, events need to bubble up from their source until they reach a threshold and "fire". A perceptron is a perceptual event that has fired - but how do I model the preconditions for a perceptron to fire? I need something smaller than a perceptron.

Borrowing from psychology again:

Sensation is the first stage in the biochemical and neurologic events that begins with the impinging upon the receptor cells of a sensory organ, which then leads to a perception.

Sensational!

So instead of tying the design in knots trying to model perceptrons with perceptrons I've snipped the proverbial Gordian Knot and now lesser perceptual events are simply sensations that can chain sensors together to make a perceptron.

Do, Delegate, Defer + Divide = Smoother Workflow?

The todo subsystem is at the heart of The Goo. One of our ex-employees recently wrote:

"The job is great however I thought being involved in such a large organization there would be better procedures/communication in place! I miss the goo - esp the tutorials. You guys are certainly on the right track there."

But there's more to come! After my YAPC::EU presentation, Farley Balasuriya gave me some great feedback and said he'd recently seen similar ideas in 'Getting Things Done' by David Allen. I've since read the book and Allen makes some important suggestions - if the task takes less than 2 minutes Do It now - otherwise Delegate it to someone else or Defer it.

So in the pursuit of atomic todos I would add one more D - Divide It up. If the task is too big it can be a flow-stopper so it should be divided into smaller todos. I'm currently designing how todos will appear on your mental horizon - and the fours Ds will help you to wade through them.

giz - sticky editing

I'm going to release a small part of The Goo to CPAN before the London Perl Workshop in December.

giz is an $EDITOR wrapper that:

• works with your existing editor (e.g., vi, emacs, nano, pico, kwrite etc)
• helps remember where your files are located - so you can afford to forget
• injects a special comment into your Perl code to help with keyword completion
• can be called from a url (giz://filename=your_filename)
  

The Goo powered by Perl

I recently presented about how The Goo is powered by Perl at the YAPC::EU Conference in Vienna. Here are the slides:



Thanks to those who were there for the feedback and encouragement.

Group Karma++

It's a great feeling being in a state of flow. Sometimes you can get into a state of flow with a group of people, a kind of 'group flow'. This is where the team is working well and everyone is excitedly making progress while the technology is supporting their communication and ability to stay productive. I've been watching developments in the Perl6 Pugs project and the IRC log shows moments when the group hit productivity gold and achieved group flow.

One of the preconditions for flow is feeling progress being made. In the Perl6 pugs IRC log this is achieved by giving each other karma++. As you make commits to the pugs project you get a small pat on the back in the form of another point of karma++ - the more contributions you make the more karma++ you collect. You can see progress being made as the karma++ points tally up in the log.

I've been designing the scoring system for The Goo and I think karma++ is a really important part of feeling progress being made. But to help achieve flow we also need small mentally manageable tasks - atomic todos. So to help the system converge at smaller task sizes a user receives karma++ when they create a task - this should motivate them to create todos, and they receive karma++ when they complete a task too. When you view a person's context you see what channels they are tuned into and how much karma++ they have.

Sensors vs Censors

I'm reading Marvin Minsky's, "The Emotion Machine" and he makes a strong case for the role of "censors" in expert cognition.

Experts can arrive at the correct answer quicker. But how? Minsky suggests it's because the expert brain blocks unhelpful information and is not slowed down by useless processing. He refers to these automatic blocking agents as "censors". Censors, he suggests, free the expert brain to think about the important stuff. An expert learns from the mistakes of the past and censors help to avoid them in the future.

So far The Goo has been sucking in perceptrons via positive "sensors" but there is also a role for "censors" too. Spamassassin, for example, does a great job of filtering emails before they become email perceptrons. Spamassassin acts as a blocking censor.

But actively blocking events before they become perceptrons solves only part of the problem. There are still too many perceptrons coming in to stay in a state of "flow". They need to be channelled in light of your current mental context to prevent needless context switching.

So channelling perceptrons is the next step ...

Atomic ToDos

I'm having fun designing how to make the ToDo part of The Goo flow.

Flow happens when you can context switch smoothly in the face of a wave of challenging incoming tasks. The Goo needs to make it easy for users to resolve a task quickly, reward them in some way and move onto the next one. But there is a fine line between being challenged and overpowered by a task. Some tasks, for example, stop us in our tracks because the set up time is too expensive.

A ToDo task that is too unwieldy will break the state of flow. You might get stressed even thinking about where to start on a task. Stress is the flow killer!

Generally at Trexy and Turbo10 we try to break all our projects down into mentally manageable ToDos - which means we minimise stress while still making headway. No ToDo is too small and we actively avoid "projects" and "deadlines".

The trick is to break down a complex project into atomic ToDos --- indivisible, non-blocking, time slices of attention that get the job done.

Hmmm. How can I force the system to converge at atomic ToDos?

One way is to set a time limit per ToDo. How about a 'billable unit' to borrow a solicitor's term? Let's say five minutes.

Now some tasks will take less than five minutes (e.g., increase font size of heading to 20px) while others may take longer (e.g., write blog entry) but the time limit will generally encourage a smaller task size.

It also means that a user could acquire extra time over the course of the day if they 'beat the clock'. This helps to 'keep score' which is an important part of maintaining flow in computer games. User's should be rewarded in some way for beating the clock.

OK. So this is the next step for The Goo's ToDo management - atomic, 5 minute ToDos - and a cumulative scoring system based on time. Let's see if this helps find the elusive flow ...

Working with "Flow"

Game designers know implicitly that "flow" is the mental state they want their players to be in: happy, productive and engrossed in their game.

Wouldn't it be good if Microsoft Office could achieve the same effect?

Well my virus-ridden XP machine is currently showing the blue screen of death - so for me this is an entirely rhetorical question - but I hope you get my gist. If I can get into a state of flow while playing Tetris why doesn't office software try and do the same?

The Goo is some of the way there but to be honest it can't reliably get me into a state of flow. What can The Goo learn from game design? Here are some features of flow mentioned in a presentation by Daniel Johnson from the University of Queensland:

• Tasks that offer a chance of completion
• Tasks that facilitate concentration
• The task has clear goals
• The task provides immediate feedback (the need for a score or energy bar)
• The task leads to deep but effortless involvement (removes awareness of everyday life)
• A task that facilitates a sense of control over one's actions
• Concern for the self disappears during the task, yet paradoxically the sense of self emerges as stronger when the flow task is completed
• The sense of duration of time is altered

OK. In light of this it's time for the rudimentary user interface to go through another evolution ...

Bashing away at The Goo

A goal of The Goo is to help stick Things together and for a long time I've been wanting to add all my shell commands into The Goo. Thanks to some quick help from London.pm I can now capture all my bash shell history with the following ~/.bashrc setting:



export PROMPT_COMMAND='history 1 | /home/search/goo/lib/Goo/Sensor/ShellHistory.pm'

This inserts a perceptron into the database each time a command is executed and any 'things' in the argument list are automatically marked up. It also means the context is preserved between web based actions and shell actions.

Too Many Perceptrons?

In the last month over 70,000 perceptrons have flowed into The Goo.

I know that sounds like a lot but it doesn't feel too bad. It's not like your email inbox is suddenly inundated with 70,000 messages. The perceptrons are spread more or less evenly through time, and they kind of wash over you, for want of a better description.

Seeing wave after wave of perceptrons makes you feel slightly out of control but in a good way. All the perceptrons are flying in but once they go past it's reassuring to know they are frozen in the trail. Like events in a computer game some things can just fly by whilst others require special attention. The trail acts like a mental crutch collecting all the perceptrons.

Meanwhile, between waves of perceptrons, I've been bringing my cognitive science knowledge back up to date. Things have moved on since I studied 'cog sci' at the University of Queensland in the early 90's and I've really enjoyed reading Doug Hofstadter's, "I am a Strange Loop".

I'm halfway through the book and can't find too much I disagree with so far. If indeed our consciousness is, as Hofstadter suggests, a kind of "strange loop" then as a computer scientist it's very tempting to think of ways to optimise the loop!

All in all I still feel like I'm on the right trail.

Split Screen - Prior Art

Just found some prior art for a split screen - Vannevar Bush's Memex, 1945. I must be on the right trail!

New Interface - Balancing Perceptrons and Actitrons

Here is a screenshot of the really rudimentary interface that handles "perceptrons" (on the left) and "actitrons" (on the right). This tries to retain the 50/50, Ying/Yang, relationship between what you see and what you do when you're in a state of flow. The interface works with console based browsers and graphical browsers.

There is much more to be done on the interface but the essential 50/50 nature of the interplay between perceptrons and actitrons will hopefully get the user into a state of "flow" quicker.

Perceptrons - Feel the Flow

Email perceptrons are raining into The Goo. Great!

But this is only part of the problem solved. What about context switching and preventing buffer blowout?

I can only help improve context switching if The Goo is aware of the context I'm in. Added to this the context itself must be mentally manageable. Ideally I only want 7+/-2 things to juggle at a time.

Have you ever lost track of time in an "immersive" computer game? Computer game designers want you to feel "flow" as you context switch smoothly around the game. It's not a bad state to be in: your brain is working well, you're feeling good and you're getting stuff done. It feels smooth. It would be great if The Goo could help me get into, and stay in, a state of "flow".

The next phase is to capture the context I'm in by taking a snapshot of the perceptrons and actitrons in my buffer. See the thin green lines on top of the stick figure above. The perceptrons that get through will be appended to the "tail" of the trail as time goes on.

First Sensor - Goo::Sensor::MailFolder

I'm spending two hours a day implementing the next part of The Goo: perceptrons and actitrons.

I've decided on the layout of the interface and remarkably it works in the "links" console-based browser - no AJAX here!

Today I wrote my first "sensor" that captures perceptrons from email folders. Implementation options included:

• scraping user email in webmin
• fetchmail handler
• .procmail forwarding

But each of these introduce another prerequisite and configuration complexity. CPAN came to the rescue with two handy modules: Mail::Box and Schedule::Cron.

Currently a sensor POSTS perceptrons to a Goo::Server where they are recorded in a database. Each Goo::Sensor has a scan() method for searching its environment.

The next sensors to come are: tasks, chat and RSS...

I'll post a screen shot soon!

Modelling Perceptrons and Actirons

I use a hybrid database modelling methodology of ORM and E-R and over the weekend I started to model what a Perceptron and an Actitron is and how they relate to the Trail.

Here are some of the points that came out of it:

• A "perceptron" is a unit of perception that is picked up by a "sensor".
• A sensor is a software device for capturing perceptrons.
• The sensor must send where the perceptron was captured, when and who the sensor operates for.
  
• The perceptron may be observed but is not seen unless it goes via someone's short term memory buffer (7 +/- 2).
• A sensor passes perceptrons to a user or group of users or another sensor.
• An "actitron" is a unit of action which may be performed by a person or a sensor.
• An action performed by a person is always "seen" - but this may not be the case for a sensor.
• Over time a user's trail is filled with seen perceptrons and seen actitrons.
• Actitrons can happen on Things.
• A Thing is an object in your perceptual environment.

So what does this mean? Well by creating sensors your potential perceptual environment can be expanded (i.e., see more). Your own actions are remembered and can then be temporally associated with other Things which means you can make lighter context switches as time goes by (i.e., juggle more). It means you can also add future events into your trail by putting something in front of a sensor at the right time.

Actitrons + Perceptrons = Ying + Yang

I started writing a SciFi novel where in the future there are two types of things: Perceptrons and Actitrons. A perceptron is a unit of perception that somebody perceives (apologies to AI guru, Marvin Minsky for pinching his term) and actitrons are units of action.

My SciFi novel is on the back burner but I think these ideas could be useful in designing the next part of The Goo. When we work we perceive things and then perform actions as a result. A lot of time is spent moving from perceiving to acting and going back again.

Hopefully this diagram will help explain:

Seventh Heaven? Not quite.

At the start of the week every member of the team receives a "Pending Things Report" which shows the top ten pending Things. The problem is all the Things are set to care factor: 7. The care factor is not discriminating between what's not so important and what's really important. But the real problem is no one cares about the care factor - what is care factor 7 anyway?

So I'm looking at a different way of ordering Things without the need for maintaining a care factor.

Email is Br0keN

I started using email in 1990 and over the last 16 years I've progressively received more and more emails while sending less and less. In the last six months though, I've felt a change. The Russian "spam bots" are winning - exponentially so.

I now receive way more spam than real emails and I feel we're heading towards a tipping point - spamassassin does a valiant job of applying a Holtzam-esque shield but the slow, and crafty spam still gets through! I think we're converging on the "Spam Singularity".

The demise of email may not be such a bad thing --- I think there's more that can be done with a controlled hypertext system. A worthwhile message, for example, should always have a "what's next". I wonder what's next for email?

Drilling Down

I was talking to my dentist, of all people, about the CareOMeter. It turns out my dentist is full of ideas about workflow processes. His big idea for improving work processes is to capture the "what's next". He suggested that everything has a "what's next" even if it means "place this in the archive". He punctuates his points by spinning the dental drill so I wasn't in a position to argue - but I like the idea!

I didn't expect to feel this sort of pain whilst in the dentist chair.

Goo Location Creep

I've been wrestling with the problem of defining the file locations where The Goo will look to stick things together. I first implemented a monolithic config file, but although this gives fine-grained control over where The Goo should look I often find myself stopping and adding new locations as I move into new areas of the filesystem - where possible I want to avoid this feeling of going backwards.

The idea is The Goo should faithfully follow behind, seamlessly helping you to stick stuff together. The solution is for The Goo to incrementally discover the locations you're interested in and remember them. So the more you traverse your filesystem the more Things become "associated".

Help!! Information Overload - An Illustration

Subversive Heresy

I've tried various source control systems: clearcase, subversion and svk. Received software development wisdom says you *must* use a source control system. Why? Well here are some of the purported advantages:

• you can rewind the tape
• you can hack simultaneously
  
• you can branch
• you can browse (e.g., SVN::Web)
• you can blame

But in practice I rarely find myself rewinding the tape - my editor does that, and in the worst case scenario I can get the original file from production or staging. The retrieval cost of 'rewinding the tape' from the repository to a specific version is often higher than coding afresh (which version was that again?).

Simultaneous hacking on the same file in a huge code base happens infrequently but when it does resolving clashing files is a pain. The "blame game" is also pretty pointless. Mentally managing commits adds an unnecessary cognitive tax and takes time too. I don't know about you but I get a feeling of "repository drag"?

Ideally the software scaffold should not slow you down - but speed you up! I wonder if there is a better way? Fast browsing, no more commit overhead, clash prevention, automatic change logging and easy history traversal ...

Measuring Future Events

It is a goal of The Goo to help you stick Things together - now - but in the future too.

The ability to reason about the future depends on reasoning clearly about events that might, or might not happen, and their relative importance.

A recent O'Reilly book, "Mind Performance Hacks" suggests an interesting way of measuring the importance of something in the future. Paraphrasing loosely from the book it combines two Likert scales (1-7):

Likelihood of event happening:

Never                   Likely                          Certain
1         2         3         4         5         6         7

Importance/Impact of event

Not Important           Somewhat Important              Very Important
1         2         3         4         5         6         7

By combining the two scales you get a value out of 49 - and by doubling it a value roughly out of 100!

This simple heuristic gives you a measure of relative future importance. Why worry about something if it is never going to happen? On the other hand something may seem relatively unimportant yet it is certain to occur.

Stay tuned for how future events will find their way into the Care-o-Meter via this simple measure.

New Action Dispatcher - Targeting Things

The Goo has a new action dispatcher that does type-based dispatch. The action handler is chosen based on the type of Thing and it works from the web and the shell:

http://goo.localhost/r?arg1=tn7rc
shell> goo r tn7rc
shell> goo resolve tn7rc
shell> goo p HelloWorld.pm
http://goo.localhost/profile?arg1=HelloWorld.pm

But there are at least two other environments where I want to do actions on Things: email and while editing. Previously I mentioned the idea of ThereDOCs (e.g., p>>) - a way of embedding removable commands into text that help you context switch. Although there is a nice symmetry between Perl's HereDOCs <<>> - the ">>" characters are overloaded in the context of email (e.g., >> forwarding quotes) and Perl6 (i.e., hyperoperator).

So the new ThereDOCs (i.e., embedded action character sequence) will look like this: ~> and <~. The dispatcher must analyse the target of the action for 'Things' and based on type do the dispatch.

# imaging you're editing some code
use Hello::World; <~t # you want to jump to the tests for this module
use Template::Simple; <~p # you want to profile this module

# use Uri Guttman's sublime Template::Simple
Template::Simple->new->render("complicated.tpl", $tokens); <~p # jump to the profile of complicated.tpl

My::Logger::write("/tmp/look.log", $message); <~t # jump to the tail of look.log

When it comes to email I should be able to resolve a task by forwarding an embedded action to goo@somedomain.com. For example,

To: nigel@turbo10.com
CC: goo@somedomain.com
Subject: resolve~> tn5rc

Because all actions go via the Dispatcher it is also a natural place for remembering the trail of actions.

Mnemonic Thing IDs

I've just moved to mnemonic Thing IDs. Thing ID 15466, for example, has now become, tn6rc.

Why? Well mnemonics are one way we can do cognitive compression/chunking. But also because ...

• The ID itself carries meaning (e.g., tn6rc = Trexy task given to Nigel with care factor 6 Restart Cluster).
• Automatically finding Things IDs in emails, web pages, databases, logs etc. based purely on numbers can be ambiguous.
• Prefixing numeric Thing IDs with a hash # causes the Thing ID to be commented out in the bash shell and can also appear as anchors in URLs - making automatic Thing finding trickier.
  

Mnemonic IDs look more at home in a 70's airline booking system but I think they're due a comeback - TMTOWTDI - but that's also the point.

Test -> Code -> Test -> Code

How can I make this cycle quicker?

Well I just added a Perl5Tester to The Goo which jumps straight to a test for a module if one exists or generates a stub test file and then jumps to it. It sounds simple but this saves lots of time. Phew. No more directory changes between modules and tests.

Completing the Completer

I've been feeling the pain lately. My wrists are starting to hurt from typing too much. How can I type less while coding?

For the past week I've been considering changing editors. Currently I use nano because the key bindings are similar to the bash shell, it supports Perl syntax highlighting, and it works anywhere. But there is a crucial feature it lacks - the ability to do string completion in a file - I need this to save on keystrokes, short term memory and context switching.

So I've gone on a mission searching for an editor that fits my criteria: Perl syntax highlighting, in-file text completion, cross-platform, ideally works in a terminal, ability to customise key bindings, with a minimal learning curve. Here are just some of the editors I tried: mp, vim, vi, glimmer, gobby, emacs, kate, kwrite, nano, jpico, joe, jed, nedit, jedit, bluefish and others.

The winner is kwrite and an honorable mention goes to mp (minimum profit) for many reasons including this quote from the mp man page, "it helps you abandon vi, emacs and other six-legged freaks definitely."

The downside of kwrite is it won't work on a remote terminal and annoyingly flashes on start up - but apart from that it does the trick: in-file string completion.

Ok. So where does The Goo come in? Well it helps stick Things together by injecting a special comment into the file you're about to edit that is laden with Perl reserved words and context-sensitive method names. The string completion feature in your editor (e.g., vi, emacs, kwrite) sees the comment and picks up an expanded lexicon of keywords to use in string completion.

When you exit the editor the special comment is removed. Great! Less keystrokes, less context switching (no more looking up method names), and it works across different editors. Check it out:

Focussing the Email Filter

For the last week all my incoming email passed through a Goo email filter (i.e., milter).

The filter scans each incoming message for Things (e.g., HelloWorld.pm, 1212.idea, 233.pain etc) and adds a hyperlink to each Thing. The idea is I want to minimise the overhead of context switching from email to performing a task.

My first reaction was surprise at the number of Things found in my email - there are Things everywhere! Even things you don't expect. I received an email about London.pm, for example, and The Goo marked up this email with a link to a non-existent Perl module called London.pm - OOps. Generally though it works well.

I receive a constant barrage of email from "robots" on the Turbo10 and Trexy clusters. One robot, "Ernie the Exception Checker" sends details of exceptions via email. The Goo filter now automatically adds hyperlinks to the underlying programs so I can context switch quickly to the offending line of code, squash the bug and get back again.

Chicago Perl Conference

I'm doing a quick presentation on The Goo at YAPC::NA Perl Conference and here are the slides [PDF].

Critiquing your Code

The Perl::Critic module can critique your code based on Damian Conway's, "Perl Best Practices". This function is now available in The Goo too. Check out the screen shot:

Tackling Set Up Time

A big part of peforming a task is the time it takes to "set up". A painter, for example, must prepare the canvas, mix the paints, position the subject etc.

Sometimes the barrier to completing a task is not the task itself but the set up time required to do the task. Hmmm ... an improved task management system needs to take on this pain. Ideally the task description should minimise the set up time required for a task by including as much "scaffolding" and "materials" as possible. It should also help you clean up afterwards.

Hmmm ... this is definitely one for The Goo.

"Horizon of Care"

We need to log future events into The Goo "When is that conference coming up? When does registration close?"

A future event is a temporal thing and The Goo already has temporal events covered in the Trail. The Trail records the actions you perform on artefacts in your environment as "events" and as time goes by a trail of events forms.

So why don't we add future events into the Trail?

-----|---|--|--|--|--||--|-----|-----> [ The Zone ] ---------------------|------------|---->

This way you can traverse your temporal "Horizon of Care" by moving backwards and forward along the Trail - what you cared about in the past - what you are likely to care about in the future - what you want to care about now.

Polymorphic Queues - Feel the Buzz

We need to try to track the marketing "buzz" around Trexy.com and The Goo came to the rescue.

Thanks to the polymorphic queues of Things in The Goo we could easily add a new queue: Buzz. This means any marketing buzz can be forwarded to The Goo where it is recorded in a queue along with a "care factor".

Getting Sticky

To enable members of the team to email The Goo I needed to create an email filter to parse incoming emails. The email parser is called via .procmailrc and is implemented with the help of the Email::Filter Perl module.

But there's much more to be done. Why not apply an email filter across all emails in the organisation that helps to stick Things together? The idea is for a special email filter that analyses the content of an email and associates it with other Things by embedding links. The recipient of the email can quickly jump to, or perform actions on, these associated Things.

Emailing The Goo

Members of the Trexy team can now directly email The Goo! :-)

This means a really light context switch for those members of the team who spend most of their time working with email - all they need to do is carbon copy The Goo and a ticket/task/Thing is automatically lodged into the CareOMeter.

Action Time

I've just finished implementing a new action handling table for TheGoo at Trexy. It enables me to bind an action to a handler for that action.

For example: [P] -> Profile -> PerlProfiler.pm.

This means typing "goo -p HelloWorld.pm" on the command-line will cause the PerlProfiler.pm to execute on HelloWorld.pm. But it works in the Web environment too:

http://localhost/goo.cgi?action=p&filename=HelloWorld.pm

There is a new internal dispatch mechanism that maps an action to an action handler based on the "type of Thing". All this means that I can start to add functions back into The Goo quickly and the system can expand to handle lots of different Things (e.g., Javascript, HTML, SQL, Python, Ruby, Perl6 etc).

Goo Wobble

On the weekend I wondered if re-implementing the IDE-like features of The Goo was really a good idea? Was I wasting my time? Reinventing the wheel? Am I nuts?

So I downloaded "Eclipse", the IDE of the moment, and installed the Perl EPIC plugin. The raison d'etre of The Goo is to help programmers stick Things together, and there is one feature I simply can't do without: the ability to traverse the structural associations in your program. For example, I often want to traverse the inheritance hierarchy - what's in the superclass? When I clicked on the "Outline View" in Eclipse it took me no further than the program itself. I'm sure the class browser is more impressive for Java programs but for Perl programs I came to a deadend fast. :-(

I don't know about you but my programs hardly ever work in isolation - they always involve other Things. When I write a program I often have to context switch between these Things (e.g., program -> output, program -> file, program -> database, program -> log, program -> shell, program -> other programs etc). The Goo should help you to make all these context switches - fast.

So I closed Eclipse, noted the stream of uncaught Java exceptions in my console, and jumped back into the bosom of The Goo. No more wobbles.

The Goo at Trexy.com

We are now starting to heavily rely on The Goo at Trexy.com. Lots of tasks, ideas, bugs, etc. have been filed into the Care[O]Meter. I can now see at a glance what's happening across all team members and what I need to do too. We are more productive, communication overhead is lower, and nothing is "falling off the radar".

I haven't used the "I feel like ..." functionality as much as I had hoped but this is because I have too many tasks with a high care factor! Maybe when the pace slows down feelings can come back into the equation? But will it ever? The harsh reality is, some tasks just need doing, whether you feel like it or not. Hmmm. I still want to design a human-friendly, task management system and emotions matter - more design coming soon.

The "strategic" and "tactical" functions of The Goo are working well but we have lost "operational" functions while migrating from the console-based Goo to the web/console-based Goo. I'm really missing the ability to view my current trail, traverse backlinks, automate Perl module making and test making. What am I going to do about this? Check out the Goo related tasks in my CareOMeter:

Sparky powering The Goo?

Sparky is a lean, mean webserving machine I developed for Turbo10 and Trexy. I found configuring and performance tuning Apache + mod_perl so painful I ended up writing my own webserver!

Sparky is doing a great job of serving millions of hits per day for Turbo10 and Trexy. It's faster than mod_perl, consumes much less memory, has few prerequisites and is written in pure Perl. Last week I added POST request handling so Sparky now serves The Goo too.

The next phase in the development of Sparky is to fuse its currently separate Request and Response objects into a Context object (ala Catalyst). But I'm not sure how far I should go? Should I emulate the Catalyst dispatch mechanism too? Will this retrofit easily with Uwe Voelker's Catalyst port of The Goo?

I don't want to make yet another Catalyst (e.g., Jifty) but Sparky can't help but benefit from distilling the lessons learnt from these frameworks and others (CGI::Application, CGI::Prototype etc). Hmmm. No matter what happens I need to improve Sparky for Turbo10 and Trexy and a new Context object with a simple dispatch mechanism is definitely on the cards.

The Goo on the other hand will have its own application-specific dispatch mechanism no matter what underlying webserver is used. Given a 'Thing' in a URL (e.g., goo.cgi?thing=HelloWorld.pm&action=p), we need to look up its type (e.g., Perl 5 Thing), the action to be performed (e.g., p -> profile) and dispatch to the appropriate handler (e.g., Goo::Thing::pm::Perl5Profiler).

Each Goo user will also need to run a local "Goo Server" to access local files (i.e., a local file finder) and a light weight standalone server like Sparky is required here. Hmmm. Stay tuned ...

Finding Files and Doing Things

I've integrated the file finder with the Care-O-Meter. This means I can jump to a file quickly. Here is the latest screenshot of the Care-O-Meter (see the file finder on the right):

This means I need a separate form for updating the locations of Things in the file system:

Cool. Now I don't have to bother about remembering where something is located - no more tiresome directory traversals - up, down, up, up, down. The Goo automatically looks in these locations when it sticks Things together.

Goo URLs? Yes, please.

Hypertext is the original system for traversing associations between Things.

In a previous blog entry (Goo URLs? No Thanks) I mentioned that I didn't want to encumber The Goo with having to address Things using http://really_long_unambiguous_urls/that/are/a/pain/to/type. But The Goo sticks Things together by association and hypertext offers the path of least resistance!

For example, ideally I want to browse my Perl code in hypertext and then jump straight to an editor and then jump back again as quickly as possible (to minimise context switching). A recent patch by Jonas Fonseca to the Elinks browser unfortunately didn't quite do what I needed but he suggested an even better solution: Goo URLs!

To enable your own protocol in Elinks all you need to do is add this to ~/.elinks/elinks.conf:

set protocol.user.goo.unix = "/home/search/dev/goo %u"
set protocol.user.goo.unix-xwin = "/home/search/dev/goo %u"

I now have Goo URLs! This means if I want to edit a given method I can use a URL like this:

goo://?method=someMethodDeepInTheModule&filename=/tmp/MyModule.pm

Clicking on this URL launches an external text editor (e.g., nano, vi, emacs etc), starting at the appropriate line and means I can edit the file in place and jump back out again - quick!

It turns out the same method works in Firefox too. If you search on Trexy you will find my search trails for "firefox registered protocol". But here is a summary of what I found. To add a new protocol to Firefox do the following:

1. Type "about:config" in the location bar.
   
2. Right click, select New --> String.
3. Name of string should be: "network.protocol-handler.app.goo"
4. Value should be the path to your executable: "/home/search/dev/goo.sh"

This activates goo:// URLs in Firefox. I use KDE on Linux so I need to launch a window to wrap the application with a shell script (e.g., /home/search/dev/goo.sh):

#!/bin/bash
# launch the editor in a new window
/usr/bin/konsole -e /home/search/dev/goo "$1"

This means the Goo can be fully integrated with a graphical browser (Firefox) and a console based browser (Elinks) too.

There is now no technical impasse to traversing from high level tasks and ideas all the way down to editing, compiling and testing a small Perl script buried in the file system. Cool! ;-)

Care-O-Meter Feedback

The team has been using the Care-O-Meter for a week and thanks to their feedback I've made quite a few changes:

• private emotions - how you feel about doing a job/task is now kept private
• free text searching - you can search all Things for keywords
• you can choose to notify a specific email address when the status of a Thing changes

More changes are on the way!

More New Goo ...

I haven't posted a blog entry recently because my Care-O-Meter has filled up with lots of Things with a high care factor - sadly Goo blog entries are being pushed down. But today I feel like doing one!

Goo developments continue. I've written a file finder for quickly finding Things in GooSpace. Here is a screen shot.

The motivations for this are:

• reduce context switching - I need to jump from the Care-O-Meter to the underlying files as quickly as possible
• reduce the need to navigate hierarchical directories - no more up, down, up, up, down and all the context switching that entails - just jump to the file I want!
  
• simplify the .goo configuration files - we now no longer need to store locations per Thing
• quickly resolve filename clashes that are found in multiple locations
• gracefully handle mapping Perl::Module::Names to the underlying file
  

Each user has a list of directory locations in which to look for file Things and I've made a design decision to store this in the database instead of flat configuration files. Goo actions will also be stored in a MySQL database along with each user's trail.

In the next week I hope to hook up with elinks so I can find a file and then edit it in place and then jump back to the Care-O-Meter.

An Emotional Goo Debut

After spending two weeks reviewing "tracking", "task" and "bug" management systems I decided to stick with The Goo. Why?

The other task management systems were too harsh. Is that task priority 87 or 88? What's more none of them tuned in to how people really use them. I don't know about you, but I always end up scouring over the task/bug/project/todo list looking for what I feel like doing next.

So why not create a task management system that lets you tell it what you feel like doing?

Enter the Care-O-Meter. It shows a ranked list of Things you care about that you can assign a care factor to and search by what you feel like doing next. Thanks to Elinks I can access it via the console and the rest of the team at Trexy.com and Turbo10.com can access it from their graphical browsers too!

This is an alpha version so it's not 'industrial strength' like RT. Then again it is designed to be really malleable (i.e., squishy) and thanks to RT's brilliant idea of polymorphic "tickets", it lets you care about lots of different "things" (i.e., bugs, tasks, ideas, pain, projects etc).

Anyway why not see for yourself? Check out the new tour.

Uwe Voelker has even offered to turn it into a Catalyst web application! Stay tuned ...

Email Meets Web Meets Console

The Goo is all about sticking Things together. There is one big queue of Things I deal with that feels largely disconnected from the rest of my work yet is very important: incoming email.

Important events come in via email (e.g., server down, database replication failed, friend is getting married, have a baby etc) but also unimportant events (e.g., email spam).

I want The Goo to help file events into my Care-O-Meter, manage my event horizon and generally help stick Things together automatically so I don't have too.

So why not pass this queue of incoming email via a Goo-powered filter first? The email filter can scan incoming emails for other Things and associate them with existing Things. But perception, and association is only part of the problem. The Goo needs to take actions too. Actions include adding hyperlinks to all Things found in the email, inserting an event high up into my Care-O-Meter (e.g., friend is getting married) or performing an automatic action (e.g., restart server), or injecting a special Goo-signature into the email that contains URL-based actions to minimise context switching.

Now that would help stick Things together. ;-)

Web Meets Console

Since breaking up with RT I've been ruminating on a way to unify the web with the command-line features and profiling functions of The Goo. Can I find a web-based system for handling tasks/people/events/bugs/ideas/pain etc that can be easily unified with the rest of The Goo? I need the web-based interface for the rest of my team - this is a must have - but I don't want to go to the bother of creating the interface twice.

Ideally I want to blaze a trail from the higher order Things (e.g., pain: bugs are bad -> task: reporting system is failing -> bug: cronjob fails at the start of the month) to the lower order Things (line 618 CronJobManager.pm) and back again while staying in the one 'environment'. I don't want to do a heavy context switch from browser to command line. I also want to record *everything* in a trail.

The great news is I've found a solution! Elinks. A console-based browser derived from Lynx. It renders forms, tables and colours, but most importantly it executes local CGI scripts. This means we can call local Goo actions based on links in the interface. There was one slight hiccup with calling external editors like vi and nano but one of the project maintainers has coded a patch in the latest unstable release.

Here are some screenshots of this new hybrid console/web Goo in action:

• console version of the Care-O-Meter
• web version of the Care-O-Meter
• console version of Edit a Thing
• web version of Edit a Thing

I'm really excited about the future of The Goo now it has two environments to thrive in.

The Glass is Not Big Enough

If ever I get asked, "is the glass 'half full' of 'half empty'?", my standard answer is, "the glass is not big enough!" ;-)

Perception is about the frame of reference we use to see things but also the quantity of stuff we can stuff into the frame! My brain is not big enough to perceive all incoming events at once! ... so a queue forms ... and then events drop off my perceptual horizon. Ouch.

So ideally I want to perceive more, but not have to deal with all those unruly things bustling in the queue and I also want to make sure important things don't fall below the event horizon. But perception without action is pretty useless. Why queue Things up, if, when they get to the top of the queue nothing happens? You might as well have not perceived it in the first place.

I need an omnipotent helper, scanning the far event horizon, managing my incoming event queues so I can have greater perceptual breadth while at the same time preserve my attention for the really important stuff and delegate simple actions to my omnipotent helper.

Hmmm. Sounds like a lot ... but if I can solve this pain it will help me "stick Things together" better - which means this solution needs to fall within the ambit of The Goo. Stay tuned for more ideas to help solve this pain ...

Sorry RT - We're Just Not Meant To Be

This time last week I got very excited about the Request Tracker (RT) being the perfect match for The Goo's higher level requirements: email alerts, tasks, bugs, web interface, command line interface etc. It sounded so right! :-)

But the relationship got off to a rocky start when it took two days to install! When we finally got it installed it was a mixture of relief and excitement. Then we started to try RT out. For three days we tried to get to know RT better but the web interface felt clunky and lacked clear workflows. We didn't even bother to look under the covers - it just wasn't right.

So, sadly this marks the end of our short, but intense, relationship with RT. At least we have a better idea of what we're looking for!

Tracking a Request for a new Care[O]Meter

Our install of Request Tracker (RT) at Trexy.com and Turbo10.com is complete and we're busily making lots of new task and bug tickets. The web interface works OK but over the next week I need to explore the rt command line interface and then create action handlers for a new Thing (i.e., rt.goo). This should give me the ability to handle multiple tasks and bug queues via The Goo and most importantly to traverse the scaffold between problem and solution and juggle things while not getting overloaded. Phew!! I've managed to duck writing a task / bug management system as part of The Goo. Which I suppose is the point:

The Goo is a mechanism for sticking Things together not implementing the underlying Things - it just tries to implement the association layer.

My new Care[O]Meter will need to talk to RT, but tasks and bugs are not the only Things I care about while coding. What about ideas? pain? projects? feelings? people? Why not have multiple queues? After all this is what happens in real life. My desk, for example, acts like a kind of queue. I use the "messy desk" filing system - more important stuff gravitates to the top. I'm considering filing Things like ideas, pain etc, into RT request queues. But it all depends on how good RT is at handling permissions and multiple queues etc - the next week will tell.

If it works well, I will no longer have to implement the care factor as a DatabaseThing and I can store it all in RT.

Yum Yum a Thing Installer

Installing Things should be really easy. So here's the plan:

shell> goo -p look.pdf
No Thing found for "pdf". Do you want to check the Thing Registry at TheGoo.org? [Y/n]
Found "pdf.goo" Thing:

Thing: pdf.goo
Author: Marcel Holan (mh@petamem.com)
Web: http://petamem.com/pdf-thing.html
Description: Handle adobe stuff. Trust me it works!
Version: 8.9

Do you want to install this "pdf" Thing? [Y/n]
Downloading pdf Thing ... done.
Unpacking ... done.
Installing "pdf.goo" done.
Continue with "goo -p look.pdf"? [Y/n]
shell>

The Goo is getting Squished

With all the progress that has been made on the Thing Registry it's time to turn my attention to the core Goo that has been released to CPAN.

The core Goo needs to be as lean and mean as possible. The modules that thrive in the CPAN environment are simple, useful and functionally elegant. This is not how I'd describe the current CPAN distribution (i.e., 0.0.9), more like "complex" and "unwieldy". So it's time to start code cutting.

The CPAN distribution needs to act as a minimal bootstrap. The rest of the Things can be dynamically loaded from the Thing Registry at TheGoo.org. This dramatically reduces the size of the CPAN distribution, which means less pain for me as a maintainer and greater simplicity for CPAN users. It also means I can release/update individual Things without making changes to the core CPAN distribution so I can incrementally stick Things back into The Goo and other contributors can too. Because my only conduit to add Things back into The Goo is via the Thing Registry it will exercise this new part of the system too.

I've removed over half the modules in the current distribution and there's more to come. It sounds harsh but if I want The Goo to thrive it needs to adapt to its environment.

Where the Trail Leads ...

Now that Trexy.com has been released to the world. I can go into more detail about what I mean by a "Trail". I've just added this page to TheGoo.org that brings together some of my previous blog entries from blog.Trexy.com.

Registering Things

So here's the plan. I've decided that Thing contribution and distribution needs to be as simple as possible. The other big decision is that Things can be written in any language - not just the mighty Perl. Hopefully we will see action handlers written in Python, Ruby, Java, C, bash etc.

This means we need a way of handling Thing contribution and distribution for non-Perl Things. CPAN is perfect for Perl distribution but does not handle other Things easily.

So, I've decided that the core of The Goo will still be released to CPAN, but all the other addon/plugin/extension Things are registered and distributed via TheGoo.org. This means TheGoo.org is the nexus of all Things.

So last night I hacked together:

• Packaging Things - a quick HowTo on what you need to start
• Register your Thing - a simple Thing registration page
• Registered Things a page listing all user-contributed Things

I want to pay tribute to Jamie Cameron and his brilliant Webmin system for inspiring me here.

Mailing List Madness

Three members of the Goo crew worked at various times to try and install a mailing list. Something like this should be really easy but each package proved problematic in one way or another. The good news is after much persistence the mailing list is here!

The mailing list is for anyone interested in The Goo. And if you're reading this that includes you! So what are you waiting for? Join now.

Keeping DRY - "Request Tracker (RT)" to the Rescue

I implemented a very simplistic task/bug tracking system while developing the two search engines: Trexy and Turbo10. It worked really well because I could stay inside The Goo and jump from task (problem) to code (solution) and back again.

But Trexy and Turbo10 are growing fast, and our task system is finding it hard to survive in this new environment: more people, dependencies, locations, tasks and bugs etc. We could spend precious coding time and effort mutating our existing task management system so it works well in its new environment ... or we could drop in an entirely new system that is more mature and can handle our enlarged requirements. We need a system that has already weathered the demanding requirements of an enterprise task/ticket system and that can stick together with The Goo.

We need a console-based "Goo" interface (for coders) and a web interface (for business developers). I think I've found what I'm looking for, the Request Tracker (RT) from bestpractical.com will save us having to repeat all this work, so we stay DRY and the RSS interface will hopefully provide the pipeline to and from The Goo.

Universal Action Handlers

You can specify the actions you want to perform on a Thing in its .goo configuration file. For example the pm.goo file is for Perl module Things. Here is an excerpt from pm.goo:

# show a profile of a Perl module
[P]rofile Goo::Thing::pm::Profiler.pm

The left hand column is the action (i.e., [P]rofile) and the right hand column specifies the action handler (i.e., Goo::Thing::pm::Profiler.pm). I want The Goo to work with lots of different Things but also lots of different action handlers. Including action handlers written in another language (e.g., Python, Ruby, Java etc). So this has got to work too:

# show a profile of a Python module
[P]rofile Profiler.py

So I'm taking the opportunity to move the foundation stones of the Goo while I still can. Then I can set it on its way.

Typeless Translator - Naturally Selected?

Here is an example of what I mean by Darwinian software development.

Last year I felt some pain: "I'm a crap typist. I need to type faster and more accurately."

So I came up with an idea: "Imagine if you could write code in abbreviated form like, SMS txt mesgs."

And a task: "Create a Perl module to help me type less: TypelessTranslator.pm."

The TypelessTranslator works as a post-edit filter. It scans my edited code for abbreviated code and expands it. For example, "fe m $member (@a)" expands to "foreach my $member (@array)". In a similar way that you, or your mobile phone, expands SMS messages. The idea is to "type less" but "read more".

Sadly, from a Darwinian software development point of view, this one is tettering on the edge of falling back into the primordial soup. It turns out to be quite tricky to implement and has an annoying habit of expanding into the wrong words. But it is an idea worth having as it may mutate into something truly useful.

Darwinian Software Development

One of my old bosses scared the hell out of me. She described the first release of the software we were working on as like a shiny new car rolling out of a garage that runs perfectly first time. Needless to say the project failed and it completely dot-bombed. Why? She was managing the software development yet didn't understand the fundamental nature of the beast.

Software must be malleable. If we're looking for metaphors I'd say it has more in common with genetic engineering than mechanical engineering. Software needs to survive in harsh environments and must adapt quickly. The adaptation process in nature is cruel but effective: survival of the fittest.

If our software is going to thrive in its environment we need lots of ideas (mutations) to be generated but only the best survive - until the environment changes again - and the process repeats.

Help is on the Way

Uwe Voelker paid TheGoo.org a visit yesterday after having a difficult time installing The Goo at CPAN. The "difficult time installing" is the bad news. The Goo is supposed to save you time, not waste it!

The really good news is that Uwe has offered to lend a helping hand! ;-)

He has already sent in some great patches and we're sending him the keys to the subversion repository.

Now is a great time to join the project. Email nige@thegoo.org if you're interested!

A moment to PAUSE?

"Release early release often" is an extreme programming mantra. But we don't want to be too "extreme". Our seventh version in as many days is now on its way to the Perl Author's Upload Server (PAUSE) at CPAN.

The release rate will calm down soon but we're still fixing install bugs so please be patient. The ALPHA version needs to work straight out of the box.

Injecting Events into the Trail

The Goo Trail records the actions or events that you do to Things in your environment. But what about incoming events that arrive from your environment? Server down. High load on cluster. Girlfriend on phone.

I spend about half my time making Things happen and the other half responding to "events".

Ideally I don't want to be buffeted by incoming events: "Chill out everybody! Form an orderly queue and please take note of my Care[O]Meter." Some Things deservedly fall below my threshold of "care" (e.g., spam email) while others deservedly displace everything else I was doing (e.g., server has gone down).

I want the Care[O]Meter to act as a lens on incoming events. Focusing my mind's eye on the important stuff while maximising my productivity and minimising the number of context switches I need to make during the day as I process useless events (e.g., spam email).

The Care Factor

The Care[O]Meter is one of the more unusual ideas behind The Goo - it is tied to the idea of the Emotional Programmer. A purely dry software construction memory isn't fun. I want a Memex with feelings.

That said, I have reservations about including it in the ALPHA release. One technical downside is it relies on a MySQL database which is a hefty prerequisite. A conceptual downside is it relies on the 'Care Factor'. We all care about different things while programming and it is presumptuous of me to expect that my fellow programmer will care about the same Things.

Hmmm. This is an important subsystem which I definitely want to share and I think advances the prior art. So does it stay or does it go?

I've decided it has got to stay but it means I'm going to model the "Care Factor" really *simply*. Here's how: any DatabaseThing that has a "care_factor" column gets juggled by the Care[O]Meter. No care_factor means it will never appear on the horizon of your Care[O]Meter. It also means you get to set the care_factor which means I don't have to worry about making you care about the same Things I do.

OK. That's decided then.

The "Associator"

The Goo just needs to provide a simple mechanism to stick Things together. As to what Things it can stick together, it should be as agnostic as possible (apart from the mighty Perl). It should provide an easy way for users to plug in their own Things and action handlers. Ideally action handlers could be written in any language, not just Perl.

A Thing is really just a "handle" on something in your environment: a file, a module, a script, database entity, a url, a log etc. A Thing is an artefact in your environment that you need to perform actions on. At the moment we have three types of Things: FileThing, DatabaseThing and a Thing. This covers an awesome number of artefacts but we also need to deal with Things at the application layer so here are a few more: URLThing, EmailThing and ApplicationThing.

The mechanism to stick Things together needs to firstly capture the trail of actions as we context switch between Things in our mental working zone. The TrailManager looks after recording the temporal associations between Things. But we also want to traverse the structural associations between Things too. It is the job of the "Associator" to distill and traverse the relationships between a Thing, its environment, and its creator.

Coding the Installer - Next Steps

We are working hard to get the install process running smoothly. It's not up to scratch yet but it will be soon. Please be patient if you are trying to install The Goo - this is an Alpha release. And if you can't wait, why not join in the coding? Email me at nige@thegoo.org for the keys to the repository.

Here is what the install process should do:

• create a per user ~/.goo directory
• create an SQLite database to store your trail. Found here: ~/,goo/goo-trail.db
• create a Things subdirectory ~/.goo/things to store all your .goo config files

And that's it for the moment. Each user has ultimate control over their Goo installation, including their Goo trail.

Subversion Repository is Here

You can now browse the code online.

Now is a great time to get involved with the project. We need all hands on deck to get to a BETA release. If you're interested in contributing, email me (nige@thegoo.org) and I'll send you the keys to the repository.

A mailing list is coming soon ...

The Goo Released

It was great to finally let The Goo out of the bag at the London Perl Workshop!

The audience seemed to really tune in to what I was saying and I hope my central idea of developing a Memex-inspired software construction tool got across. A Lisbon-based Perl Monger gave me some great feedback afterwards and he is very interested in coding on The Goo.

The Subversion repository is coming soon ... in the meantime you can download The Goo. Please note this is an ALPHA release - so expect bugs.

Alpha Release is Here

Thanks to the great help of the "expert programmer from PetaMem" we have a tar file ready for CPAN that you can download now.

I'm looking forward to sharing The Goo with everyone at the London Perl Workshop today. This is an alpha release but it also means it's a great time to get involved with the project. We are setting up a subversion repository and mailing list so stay tuned.

Some Thing for my Bro

My brother Andrew had his birthday on the weekend and I was trying to think of some "Thing" to get him ...

The best present we received as boys was a computer called a "Dick Smith Wizard". Imagine a Commodore 64 meets an Atari. My Uncle, who himself was a computer programmer, advised my mother to forget about the Atari and get us a *real* computer. The Wizard had only 16K of RAM, which by today's standards makes it more like a calculator. Nonetheless, the cartridge that spent the most time plugged in was not "Tank Attack" or "Crazy Chicken" but the "Basic" programming cartridge. At 7 and 8 years old the two of us were a little programming pair: extreme, underaged programming. I would do the 'vision thing' and Andrew would patiently explain two dimensional arrays to me. We had loads of fun and learnt lots too.

So in the same spirit of fun I'm going to make him a Thing for his birthday. His lingua franca is Python so that means:

> goo -m py.goo

London Perl Workshop Talk Submitted

I just submitted my slides for the London Perl Workshop this Saturday. It contains lots of screenshots of The Goo in action so the file size is pretty huge.

Click here to download the presentation:

• http://thegoo.org/thegoo-lpw-2005.odp (6632031 bytes)
• http://thegoo.org/thegoo-lpw-2005.pdf (21853598 bytes)

Perl6 Thangs

I've been avidly reading about the development of Perl6 and especially the awesome code cutting powers of Autrijus and the team at pugscode.org.

I'm also really keen to start using Perl6. So much so I want to make a new Thing for Perl6.

This would normally be easy. Just create a .goo configuration file Perl6 Things:

goo -m pugs

But the suffix of Perl6 Things is not "pugs" it's ".pm"! Bugger ... I've already got a Perl5 config file for ".pm" Things "pm.goo".

But this will happen from time to time and The Goo needs to be flexible enough to deal with it. Fortunately the solution is straightforward - we just need a slightly more sophisticated set of action handlers in pm.goo that can inspect the #!/shebang/line to determine what to do next.

I'd like to demonstrate The Goo working with Perl5 and Perl6 at the workshop at the end of the week.

Living Things

We need to model the team of people working in The Goo. So that they can email reports when bugs get fixed and tasks get done. I thought for a moment about defining people in the goo.goo config file but that puts too much onus on the config file parser, it doesn't extend easily and isn't as fun as making a "person" DatabaseThing.

This is how we will model people who support your activity in The Goo (i.e., other programmers, testers etc).

Care[O]Meter is back on!

Things are now starting to move fast. The Care[O]Meter is back working. I can now see what's happening and jump quickly to the next Thing. My programming pair can see it too.

Making Bugs

That was quick. Phew. Great I can make bugs again!

Going into The Zone

I'm in deep at the moment! I'm trying to get The Goo into a format ready for CPAN. This means lots of changes! ... Luckily I'm using The Goo to make The Goo.

Because it's a simple text based system I can use it directly on the staging machine at TheGoo.org. But not all the features are working and I'm missing them. Like: Bugs!!! - there's lots but no where I can quickly log them, Tasks!! Lots of these too but sharing them is slow. No [Z]one yet so I keep dropping balls :-(. The Care[O]Meter is also busted - so I have to manage what to do next.

I'm working in tandem with "the expert Perl programmer from Petamem" and it would be great if we could see the scaffold for this beast together.

Ok ... back to the [Z]one ...

The Goo @ London Perl Workshop

Cool! ;-)

I'm confirmed to speak at the London Perl Workshop.

> goo -m feeling
> Select a feeling? [1] Happy [2] Frustrated [3] Angry [4] Tired
> [1] Happy!!!

Now if I can just get my login on CPAN working! ... we may even have an alpha release ready in time.

Planning the Alpha Release

Here are the minimum features I'd like to include in the alpha release later in the month:

• Things - we need to include a few example Things (e.g., Perl modules, configuration files, log files, tasks, bugs)
• Goo Trail - this is a crucial subsystem
• The Zone - show your current working space (the tail of the Trail)
  
• ThereDocs - jump quickly between Things
• BackLinkManager - show all the backlinks that point to a given Thing
  
• Care-O-Meter - a very simple Care-O-Meter (no emotions just yet ;-))
  

Working with PetaMem will really help structuring the project for release. A lot of the functions I use need to be removed because these "Things" are specific to my environment. For example, the two search engines I have developed with the help of The Goo, Turbo10.com and Trexy.com use a homegrown templating system and many of the Things in the environment relate to this. These Things won't transfer well to another environment and it doesn't make sense to bundle them with The Goo.

The main purpose of the alpha release is to share the ideas behind The Goo and to demonstrate a proof of concept - it really is an alpha release. We need to distribute a minimal version which will hopefully act as a framework in which others can plug in their own Things. I'm interested to hear comments and suggestions too ...

<<HEREDOCS to THEREDOCS>>

Perl has a great feature for embedding a document straight into your program: HEREDOCS. The programmer can declare, "here is a document" and clearly delimit it from the rest of the program. The syntax for HEREDOCs starts with a <<MARKER and ends with a MARKER on its own line.

Sometimes while programming you don't want to include a document, but instead jump to it. I'm constantly needing to jump to another Thing and this context switching can often be time consuming. Too many context switches can lead to "buffer blowout" (a kind of mental coredump).

"I want to jump to 'there'" as quickly as possible. "There" is often a template, test file, command line or another program - but importantly it could be any "Thing".

While editing Things in The Goo you can use THEREDOCs to jump from one Thing to another. A THEREDOC is an action followed by two arrow characters (e.g., p>>). The letter corresponds to the action you want to carry out on a Thing.

I was about to launch into a written description of THEREDOCs but the best way to explain it is to show you. Slide show coming soon ...

Synaptic Silliness

OK. So I'm going to implement a few more Things: pain, ideas, projects, people and events (more detail on these Things coming soon).

These Things will be stored in a database. Now, I could do it the traditional relational way ... many-many relationships between projects and people etc. But maintaining the relationships between entities in a database adds an extra implementation tax to the system that is paid regularly in the form of constraint checking, coding for update anomalies, transaction checking, locking and the need to maintain many-many relationship tables (i.e., referential integrity).

Hmmm ... I want an easy life. I'm also keen to try something new.

I've decided to associate Things in the database a bit like our brain does. Cognitively speaking, the more a memory engram fires around the synaptic gaps of our brain the more indelible the synaptic trail becomes (emotions help too). Hmmm ... maybe the only associations that really count in the database are the ones that 'fire'.

So I've decided to associate Things in the database by processing the trail. An "Associator" will analyse the trail for associations between Things. You can think of this as a many-many table for all the entities in your database.

The associations table records the associations between Things so you can afford to forget them.

Goo URLs? No Thanks.

The Goo works by using a filename as a kind of tiny URL for files in your workspace. The Goo maps this filename to a location. It can be tricky if we have a file with the same name and different locations - which one do we choose?

Hmmm ... I want to keep everything really light and I think it would be a drag if The Goo required you to use formal URIs: goo://my_thing_in_unambiguous_location/thing.thing). It should just work. Generally the user should not need to worry about location - The Goo should stick to a Thing wherever it is. But what about filename clashes?

The principle of DRY (Don't Repeat Yourself) helps here. If there is a name clash, do you need both files? Is one incorrectly named? If it is the same file aren't you repeating yourself? If an ambiguity arises The Goo can ask the user to choose a file and optionally delete one.

Computer science loves hierarchy - just think of inheritance, file systems, URIs etc. Personally I'm sick of traversing hierarchies to get to what I need to do. Up, up, down, down, down, no not there, up, up, down. Crazy! Ideally I just want to associate one Thing to another - this means overlaying the hierarchical file systems we're stuck with at the moment. So rather than working in a strict hierachical environment my ideal is more like a blob of associated Things - an association layer - coloured green. ;-)

Decisions, Decisions ...

I'm trying to decide what is the best way to distribute The Goo.

But ideally I'd like users to contribute their own Things. So we need a system for contribution and distribution.

We could go with a simple tar.gz file, which would simplify distribution, but what about contribution? What's a good system for user contributed Perl code? CPAN.

The Goo Tour

Here is a tour which helps to explain what The Goo does: http://thegoo.org/goo-tour.pdf.

Changes @ TheGoo.org

Yesterday I improved the look and feel of TheGoo.org. I don't want to blow the "buffer" of first time visitors, so the main page has just three links: about, download and blog.

The about page tries to gently introduce The Goo before encouraging visitors to take a tour. The tour is a simple slide show of annotated screenshots. A second slide show will demonstrate how to make your own "Things". After all, There's More Than One Way To Do It (TMTOWTDI) and I want Goo users to make their own Things. It would be great if users could contribute extra Things to The Goo.

The yet-to-be-designed download page will provide simple install instructions and a guide on what to do next!

If someone wants to read about the background of The Goo the blog is the place to look.

Structural Association vs Temporal Association

Some Things are connected structurally. For example, an image that is embedded in a Word document is structurally associated with the Word document. Other types of associations are more subtle. Where did the image come from? Was it cropped, resized from the original? There may only be a temporal association between the image in the Word document and the original. But it may be important to traverse this temporal association later. The Goo Trail is attempting to capture the temporal associations between Things in your working environment so you can afford to forget.

The Goo Trail

At the core of The Goo is a "trail". A trail is the path you leave behind when you traverse Things in your programming environment. The idea of a trail is inspired by the "Memex" (a device conceived in 1945 by Vannevar Bush). This was also the inspiration behind Trexy a trailblazing search engine.

Every action you take while using The Goo is stored in a trail. But why is this useful?
The Goo trail:

• stores temporal associations between Things.
• helps answer the question, what else was I working on when I last accessed this Thing?
• provides a working context of Things that you are currently juggling. Your current mental working 'zone' is the tail of the trail.
• preserves your programming memory for later retrieval by yourself or others
  

The Goo Prerequisites

At the moment The Goo relies on a few prerequisites to work:

• MySQL database (for sharing database Things with others)
• SQLite (for client-side GooTrail recording)
• Linux

and the following Perl modules:

• Text::FormatTable
• DBI::PurePerl
• DBD::DBM
• DBD::ExampleP
• DBD::File
• DBD::NullP
• DBD::SQLite
• DBD::Sponge
• DBD::mysql
• DBD::mysql::GetInfo
  
• DBI
• Term::ReadKey
• Term::Complete
• Text::FormatTable
• Term::ANSIColor

The Goo also uses PerlTidy to automatically tidy your Perl code. A syntax highlighting text editor should also be present (e.g., vi, nano, emacs etc).

Help packing up The Goo (CC the Blog)

Hi,

Thanks for accepting the challenge of packing up The Goo for an alpha release!

Here's what to expect:

The Goo is a system for sticking Things together in your programming environment.

The Goo assumes that all "Things" have a unique filename suffix. Some of the Things that The Goo can stick together in the alpha release will include: perl modules (.pm). perl scripts (.pl), perl tests (.tpm), text templates (.tpl), configuration files (.conf), log files (.log) etc. There are also database Things and these too are named with a suffix. For example, "121.bug" referers to bugid 121 in the bug table.

Each type of Thing has a corresponding Goo configuration file based on its filename suffix. For example, the perl modules (.pm) config file is called "pm.goo" and the perl script (.pl) config file is called "pl.goo". The Goo itself has a top level config file: "goo.goo" and all config files are found in the same directory: ./goo/things/goo. Here is an example Goo config file.

Each config file contains a set of actions that can be performed on the Thing. For example, some of the actions that can be performed on a perl module (.pm) include:

title = Perl5 Module
filename = pm.goo
suffix = pm
description = Perl modules
[M]ake = ModuleMaker
[E]dit = ProgramEditor
[P]rofile = ProgramProfiler
[D]elete = ProgramDeleter

Any letter included in square brackets becomes a command line option. For example, the command "goo -e Example.pm" (i.e., [E]) will invoke ProgramEditor.pm to edit the module, Example.pm.

To install this first version of The Goo unpack the tar file and edit the .goo configuration files in /goo/things/ to point to the file locations in your system. This will give you a starting point to do the code review. Type goo -p to profile a Thing. This should get you started. Good luck ...

Nige

Packing up The Goo

I've submitted a talk to the London Perl Workshop on The Goo. I won't know if it's been accepted for another week or so but either way it's a good excuse to start publishing the code behind The Goo and getting it ready to be shared.

This is an early alpha version - so expect bugs.

An expert Perl programmer from PetaMem is going to help me pack it up and ready it for very early alpha release. There's lots to be done. I'm going to share some of our correspondence on the blog to help provide some background to this very first release.

Care-O-Meter Version 2

Well version 1 of the Care-O-Meter worked and gave me a glimpse into how my productivity can skyrocket. It has forced me to stay on the case and focus on what I *really* need to be doing. This is helping but changes are needed already. The first problem is there is too much information coming through. The Care-O-Meter is listing too many incoming tasks and bugs. The number one design objective of The Goo is to help manage cognitive load:

Seven Bugs + Seven Tasks = Buffer Blowout

So this is going to change.

But then comes the next questions: how many tasks? bugs? which bug is more worthy? why? The importance rank helps, but ultimately I want to choose to do what I 'feel' like doing. The Care-O-Meter should act as a guide not a task master. This is where the 'emotional' part comes in. What do you *feel* like doing next? Stay tuned for Care-O-Meter version 2 ...

The Care-O-Meter - Version 1

The Care-O-Meter is a ranked queue of Things I care about while coding. Have you found yourself finishing a batch of work and then thinking "what should I do next?" The Care-O-Meter answers this question. For version 1 it shows a ranked list of tasks and bugs. You only see the top three most important tasks and the top four bugs - never more than seven total Things at a time. This means when you finish a task you can quickly context switch to the next task - without slowing down to work out what to do next. The task and bug descriptions are scanned for "Things" so they are automatically associated with the software. This means you can jump quickly to the Thing(s) that the task (or bug) relates too.

But tasks and bugs are just two examples of the Things a programmer needs to care about .... stay tuned for more Things handled by the Care-O-Meter ....

The Emotional Programmer

How's that for an oxymoron?

But if I'm trying to model how the programmer copes with context switching and cognitive load - shouldn't I consider other mental activities? Like emotional stuff too? During the process of programming I can get happy, sad, frustrated and even excited - just to name a few emotions. Contrary to the stereotype, programmers aren't code cutting automatons without any heart - we've got feelings too! ;-)

Perl programmers, for example, have even been known to write 'poetic' programs. So what about the emotional side of programming? I haven't heard of an IDE that helps the 'emotional' programmer - I wonder if The Goo can help me out here? Why not.

Injecting emotional artefacts into the 'trail' can help recall. That's how I 'felt' when I was last dealing with that part of the system. It will also help ordering artefacts in the "Care-o-Meter".

"The Zone"

The zone is your working mental buffer. It's what you're currently juggling. At most we can juggle 7 plus or minus 2 things at a time. The Goo tries to make coding easier and quicker by throttling your mental buffer. It does this by keeping you in the productive "zone" while minimising the cost of context switching. It maintains a trail of all the actions you do. The Zone is the 'tail' of this trail.

Software Scar Tissue

I was going on about The Goo with some friends over a few pints the other night.

One of the guys from the Micro$oft side of the force described how he often has to chop out large parts of his system as his software evolves. All large software projects have a kind of software scar tissue that builds up over time.

The Goo is a kind of mental scaffold for your software and I drew a metaphor with a building scaffold earlier. One of the functions of a building scaffold I forget to mention is that they often include rubbish slides for quickly throwing out building rubbish. The Goo also needs to help remove software scar tissue (rubbish) as your system evolves.

Implementing Things

Ok. Lets get into implementation details.

Things represent all the stuff you stick together as a programmer: modules, scripts, html files, database queries, log files, configuration files, templates, test files etc. To enable The Goo to help you stick these Things together it makes a number of assumptions:

• all Things have a filename suffix (e.g., "pm" for Perl modules, "pl" for Perl scripts, "sql" for SQL queries etc.)
• each type of Thing has a global "goo" configuration file (e.g., pm.goo, pl.goo, sql.goo etc)
• the goo configuration file includes a title, description and a set of locations (e.g., CGI scripts are foung/home/web/cgi-bin/)
• the goo config file also includes all the actions you can perform on a Thing (e.g., Profile, Clone, Delete, Jump etc.)
  

Here is the contents of the config file, goo.goo:

title = Goo Configuration
filename = goo.goo
description = Meta details about Goo configuration files
locations = /home/search/goo/things/goo
[M]ake = GooMaker
[E]dit = Editor
[P]rofiler = ConfigProfiler
[D]eleter = FileDeleter
[C]lone = Cloner
E[X]it = Exiter

More implementation details to follow ....

The Goo Developments

There's been lots!

I missed out on presenting at London.pm as I had a wedding to attend in France but developments on The Goo have continued apace.

I want The Goo to help maximise my coding memory. To solve this problem The Goo now records a history of the actions (i.e., trails) I perform while interacting with "Things" in Goo-space. This means whenever I return to a Thing in the future I can see what other Things I was dealing with at the time. A "sliding window" of "Things" then describes the coding context. This means I can get all the balls back into the air quickly by retrieving the coding context.

Ok. That's it for the "vision" for the moment - now it's time to go into how I've implemented it. Stay tuned for implementation details in coming blogs ...

London.pm Presentation

I'm making a presentation about The Goo early next month to London.pm the London Perl Mongers. I want to share some of the ideas behind The Goo while at the same time not coming across like "this is the one true way" of doing software development. The Goo is supposed to intimately model the process of sticking things together in software - but this process is different for everybody. There's More Than One Way To Do It (TMTOWTDI). I just hope some of the themes strike a chord with the audience and they are inspired to think about making their own Goo-like system. So to this end I'm going to present a smorgasboard of problems and one possible solution for each. People can then pick and choose the ideas that may help them.

Inheriting Code

Have you ever inherited badly written code from someone else? It's a nightmare isn't it? Even well written code can be hard to digest - mentally speaking. You often need to spend days studying it before you can add just one extra line. Why did they do it that way? Where did they get that from? Why are they bothering to do that? What's that all about?

Part of the problem is you need time to build your own mental model of the software before you can deal with it. The software documents the solution but you need to understand the problem space too. Wouldn't it be good if the cost of bootstrapping a programmer's mental model could be lowered? To enable a new programmer to start juggling quicker.

To achieve this the problem space needs to be intimately tied to the solution space - stuck together: Goo-style. A new programmer inherits the existing software solution but it's really the problem space they need to know to start making a difference. So we need to intimately stick the code and the problem together. The programmer can then context switch quickly between "why it's a problem" and "how we solved it". Every line of code and module needs to relate to the problem it's trying to solve. For that matter every artefact (Thing) in The Goo should be associated to its underlying raison d'tre.

I see the association as: Pain -> Problem -> People -> Solution. A new programmer must freely traverse this scaffold so they can start sticking their own artefacts together - quick!

Metalwork Meets Computer Science

One of the few tangible things I left school with was a spatula made in metal work. It was one of the most useful assignments I was given. I learnt something, plus I had something to fry eggs with in the future! :-)

I once lectured in "Programming on the Internet" at the University of Technology, Sydney. Each semester 300+ students would spend 40 hours or more working on assignments for the course. That's 12,000 hours or around 500 person days! I was determined that all this time would not be wasted so I tried to come up with assignments that were fun, practical and meaningful. I largely succeeded, but I could have set an even better assignment.

Programming your own programming tools is one of the best lessons that I could have taught a young programmer. It's a bit like saving money, the earlier you do it the more compounding interest is gained. Setting a "mini Goo" as an assignment is really useful - not only do the students introspect on their own process of programming but they get to keep something that will be useful in the future.

Code Cutting Chefs

Have you seen those TV chefs with their own selection of special knives that work just the way they like it? Or maybe you've been to a hairdresser who has their own special set of scissors? Painters with their own mix of paints? A surfboarder with the wax applied in just the right way? You get the idea ...

What about computer programmers? What about your set of knives? Do you have just one tool (e.g., Eclipse) or a whole set (e.g., vi, emacs, perl, sed, awk etc)? Some of these tools help our keystrokes and ability to edit files - but this is just the start. Programming is about a lot more than editing files: feeling the pain, analysing, modelling, designing, testing, optimising, monitoring, refactoring, maintaining, learning, sharing, remembering, sifting, sorting, inventing, cloning, creating, making, deleting, profiling etc.

You need a tool that is intimately tuned into your own process of programming - all the way from pain, problem to solution and back again. Not just the coding bit - I mean the lot. The tool needs to grow with you. You need to constantly sharpen it - introspecting on your own process of programming (what's going through your head?) and automating as much as possible. This is what I'm trying to do with The Goo. Because it's a tool I'm creating for myself I don't expect everyone to like it. There's more than one way to do it (TMTOWTDI). Your set of knives will be necessarily different but I hope I inspire you to start sharpening .... :-)

Scaffolding

Scaffolding is crucial to building construction. A scaffold does the following:

• enables builders to access parts of the building
• interlocks with other scaffolding easily
  
• acts as a staging area for further construction
• can act as a conduit for more materials
  
• holds the building up
• protects builders
• protects passers by

The Goo needs to use scaffolding too. But unlike building scaffolding it doesn't have to be taken down after the event. We've got oodles of disk space now. We can afford to employ extra artefacts in the process of software construction. Why stop at unit tests and documentation?

Remember the "software requirements specification"? It hung around a project like a bad smell - most people ignored it. If it wasn't wrong in the first place it soon went out of date as the software inevitably changed. Software scaffolding, like the software itself, must be malleable. Wouldn't it be better if the scaffolding emanated directly from the requirements? Even better again ... if the requirements were properly specified in the first place, so that the underlying "pain" is identified first, then the problem, people and solution. Even better again ... what if we could traverse the trails of association between all these things (pain, problem, people, solution) so that there was a shared software construction memory?

Contract First Programming

Extreme Programming (XP) is a methodology for improving programming productivity and I've borrowed lots of useful ideas from it. One such idea is "test first programming", however, I've slightly modified it to "contract first programming". Test first programming means writing tests first and then progressively coding a program against these tests until they all pass. The tests act like a building scaffold holding up your program while you build it. But unlike a building scaffold you don't have to take it all down when you finish! The test becomes part of a test suite which can help guarantee the quality of your software into the future.

But you need a plan even before you erect a scaffold. You need a scaffold for the scaffold.

This plan is the "contract" between what you're about to create and the world it is about to enter. If you can distill the contract of a program before you write it you can automatically generate large parts of the test program and the real program upfront. You can automatically scaffold both the test and the program.

But this is just beginning. When you think about it most "things" have a contract - an essential point of being - so why not create scaffolds for these too? The Goo is about sticking artefacts together but these artefacts have to be born into the world first! That's why all Goo artefacts have an accompanying "Maker" that helps distill the contract of a "Thing" and then automatically generates a scaffold for it. In the case of a computer program the Maker distills the contract of the program you're about to write and generates a test program and the program itself.

Of course Things can contain other Things. So we will need a cascading chain of Makers to erect scaffolding for all the sub-Things as well.

All "Things" in The Goo have an accompanying "Maker". A maker distills the contract of a Thing and generates a scaffold for it.

Accountable Code

I often lose the plot about why a bit of code was written. The business context or the problem domain is important to remember especially when you want to refactor something fast. Why did you decide to do it like that in the first place? Have you often changed something only to discover that it was right the first time? You need a way to quickly get all the balls back into the air fast. What were the business needs behind this coding decision? Here The Goo helps by sticking together: The Pain -> The Problem -> Your Code. So you can link back in time to why you did something (or someone else in your team for that matter).

D.R.Y. I.C.E.

The principle of Don't Repeat Yourself (D.R.Y.) helps when it comes to managing complexity. It means that there's only one place to look for the canonical version of something. That's the good part. The downside of D.R.Y. is that you need to connect all these "things" together and the number of "things" keeps growing. So how do you get the benefits of D.R.Y. without the pain of dealing with lots of little things? You need an Interlinked Coding Environment (I.C.E) where you can navigate and back track quickly among the artefacts of your system. Where is that template? database? table? class? test? config file? All these things need to be interlinked in your coding environment - stuck together not just by the program but by the programmer. This is where the Goo comes in. It provides a way of navigating the relationships between all "things" in your mental coding zone.

Augmentation

Some of the ideas behind The Goo are inspired by Vannevar Bush's Memex. In 1945 Bush dreamt of a device called a Memex an "enlarged intimate supplement to his memory." Like the Memex, The Goo is designed to help augment mental power. It's just a tool on one hand, but on the other, it represents an extension of my inner thought processes. Because it is an intimate extension of own process of programming it may not suit how you think, but I hope some of these ideas will inspire you to build your own Goo.

Context Switching

The Goo is about helping you mentally juggle "things". Context switching is what happens when your phone rings while you're working on something else. Suddenly you drop all the mental balls you were juggling. This is a fairly big context switch. However smaller context switches happen all the time: a new email arrives (what do they want?), detach a file (what format is it?), where should I put it?, this folder? that folder? Opps. Virus warning? Close. Close. Close. Where was I? Each time you make a switch your mental buffer fills up. Too many switches and you start to become overloaded, frustrated and begin to drop the ball.

Sometimes you know exactly what you want to achieve but lots of little context switches get in the way. I need to print that file? Where is it? What format is it again? Is the printer on? Hang on there's another email. What was I doing again? Wouldn't it be good if you didn't have to change context? You could jump to the next thing and jump back quickly and easily. To achieve this "The Goo" helps you manage "Things" by automatically associating them. You can traverse the associations between "Things" quickly and then easily jump back to your starting context. The Goo maintains a manageable buffer of associated things that acts as a sliding window of "Things" you're dealing with. The Goo is a lens that focuses on the associations between things in your natural workflow.

Cognitive Chunking

Miller's 7 plus or minus 2 rule is the limit on the number of "things" we can juggle at any one time. A cognitive scientist might calls these things "chunks". When we remember phone numbers we often break up the problem into "chunks" to make it easier: the country code (44) the locality (207) and the rest (450 0221). But cognitive chunks can apply to any "thing" - not just numbers. Cognitive chunking is an important strategy for managing short term memory. Mnemonics are a great example of chunking in action (e.g., FAQ, WYSIWYG etc). By compressing bigger ideas into atomic chunks we achieve cognitive compression and greater throughput! ;-)

The Goo must cognitively compress the artefacts (i.e., "things") you work with so they form manageable cognitive chunks. These chunks must be atomic in the context in which they appear - they shouldn't break apart while you're juggling them!

As a result of this, all artefacts (i.e., Things) in "The Goo" must have a "profile" which is a mentally digestable chunk.

Saving Mental Bandwidth

We only have so much mental buffer space in which to juggle "things". At a certain point we get overloaded. Like a juggler with one ball too many we end up dropping things - a mental core dump. Miller's classic psychological study showed that our short term memory can cope with 7 plus or minus 2 things at a time. Maximising the throughput of this mental buffer is one of the design goals of The Goo. We need to swap things in and out quickly while not mentally overloading overselves. Many programming IDEs focus on the program as the only artefact that matters but while you're programming you often have to deal with other things: templates, files, SQL queries, databases, log files, web browsers, emails, real life etc. Everytime you switch contexts more buffer space is required. The Goo is designed to streamline context switching between "Things".

Goo Design Decisions - Care-o-Meter?

More design ideas keep coming for where The Goo should go. I've decided to extend it to non-programming artefacts as well - emails, directions, internet searches (i.e., trails), events, contacts etc. Ideally I want to scaffold a whole project by cascading artefact creation from pain -> problem -> project -> people.

I've also decided to reverse the cognitive polarity of my email inbox. Do you get 'buffeted' by your inbox? Incoming emails are a huge source of distraction and information overload. I find myself losing the plot under an avalanche of emails. But some of these emails are important for "projects" and involve the "people" working or helping on them. The Goo should maintain a "Care-o-Meter" where incoming emails are filtered depending on what you care about - what's giving you the most pain lately? Rather than your email blurring the focus of your work, your work should act as a lens on your email - filtering and focusing on the important stuff when you need it.

Pain means Gain

All problems worth solving have some associated "pain". Inventors will create something to "scratch their itch" and it is often said that "necessity is the mother of invention". The more pain you identify the more inventive ideas you can create. When it comes to invention pain is not a dirty word: pain means gain.

The initial pain I had was "code overload". I'd spent the previous four years writing two search engines: turbo10.com and trexy.com. But there was now just too much code!!! 100,000+ lines of code and growing. I could no longer hold it all in my brain. I was rewriting code that I'd already written because I couldn't find the old code in my morass of modules.

So to scratch this itch I needed to erect scaffolding separate from the code that allowed me to see it all in one place and traverse the interlinkages. Which modules call this module? The system I devised is similar to JavaDoc (a hypertext class browser) but also acts as a Content Management System (CMS). I gave a presentation to London Perl Mongers and YAPC::Europe 2004 called HELP!!! Code Overload.

The system worked well but it turned out to be only the beginning. I started to tune into more of the pain that I had while programming. I soon found lots of sources of pain. How else could I write code to help me code? This led me on the trail to the rest of the ideas behind The Goo ...

What is "The Goo"?

The Goo helps programmers stick things together. This is a pretty lame definition but it will do for the moment. In the next few blog entries I'll be exploring some of the design themes behind "The Goo". I'll come back at some point and come up with a more useful definition - so watch this space! In the meantime I want to explore the design ideas behind The Goo. Here they are:

• The Pain
• Saving your mental bandwidth
• Cognitive Chunking
• Augmentation
• Context Switching
• DRY ICE
• Contract First Programming
• Scaffolding
• Code Cutting Chefs
• Coding Trails
• Metal work meets Computer Science
• Sticking Together Things and Thangs
• <<Heredocs and Theredocs>>
• Typeless Translation SMS Style
• DIY Toolbox
• Much more to come ...
  

All this so I can code better, faster, easier and hopefully it will inspire ideas in you too!

Welcome to the Blog

Follow me on the trail towards programming nirvana ...