A book every programmer should read

The Pragmatic Programmer: From Journeyman to Master. Every programmer I hired was required to read this book. It’s a great resource for people managing programmers too. The book is entirely language agnostic, and focuses on how programmers do (or should) think and work. Most programming books do an adequate job of covering syntax, but few actually address the process of programming. My collegiate programming classes typically expected me to already know the syntax and focused more on the theory of the system, but they too offered little advice on how to actually program. This book, more than any other I’ve found, fills those gaps.

This book was so unlike most technical books, the authors went on to start their own publishing house for programming books. (I was fortunate to be a technical reviewer for one of them.) All of their books that I’ve tried thus far have been of high quality.

Another excellent (but less essential) read for programmers is Head First Design Patterns. The book’s example are in Java, but are easily adapted to other languages.

A book every technical manager should read

Peopleware: Productive Projects and Teams, by Tom DeMarco and Timothy Lister. It covers everything from productivity to working conditions and incentives, and explains why programmers are different from “regular” folk. The advice here can be easily applied to all technical and knowledge workers. Highly recommended. Slack, also by DeMarco (which I think is a more recent rendering of some of the same concepts) is on my “to read” list—when I finally get around to finishing it, I’ll comment on it as well.

Two books every manager should read

Influencer, by Kerry Patterson. This book changed the way I think about organizational issues. It begins with the incredible premise that social problems can be addressed by leveraging a single behavior, then delivers example after monumental example demonstrating—with step-by step instructions—how true it can be. From eliminating a parasite in remote Africa, to quelling an AIDS epidemic, to more local and mundane organizational or personal problems, Influencer details in easy language why so many transformation efforts fail, and what to do about it.

Good to Great, by Jim Collins. Despite criticisms I’ve heard from a couple of academic researchers calling it “junk science,” this book is used in many highly ranked MBA programs, and made rounds at several organizations I’ve worked with, including Utah state government. Collins examines the differences between companies that were extraordinarily successful year after year (as measured by stock price) and those that hovered near the industry average. It is an interesting look at many factors that shape the success of organizations.

More worthwhile books

Blink, by Malcolm Galdwell. The book is an easy to read and incredibly fascinating exploration of how we think and, despite our best intentions, are affected by biases. I’m surprised at how often anecdotes and insight from this book were relevant during my MBA courses. If you like Blink and want a more academic and in-depth treatment, I’ve found Judgement in Managerial Decision Making to be similarly facinating.

Once an Eagle, by Anton Myrer. A fictional account of two soldiers spanning most the greatest wars of the 20th century. Through the characters, the author explores the ethics of leadership. I’ve heard it rumored (but don’t know for certain) that this book is required reading at some of the U.S. military academies. An excellent—although lengthy—read.

Others I’ve found worthwhile include:

• Judgement in Managerial Decision Making, by Max Bazerman and Don Moore
• The Goal: A Process of Ongoing Improvement, by Eliyahu Goldratt and Jeff Cox
• The Five Dysfunctions of a Team, by Patrick Lencioni
• Leadership and Self-Deception, by The Arbinger Institute

Is there something else that should be on the list?

The article itself didn’t do anything for me, but the way John described the link connected some synapses for me. (I wish I could recover what it was—something about exposing government APIs to journalists.)

Utah does pretty well with e-government, but one thing continues to bug me: our election contributions reporting system. The key point is this: How much more powerful could it be if it exposed the data in a published RESTful XML format, suitable for use in mashups?

Untapped potential aside, I think the current system is flawed for the following reasons:

• Contributions aren’t tagged with the date their entered (only the date of the contribution). I’ve read of more than one instance where someone suspects a report was amended after the fact, but that information (if collected) isn’t exposed.
• Multiple pages of contributions/expenditures reports don’t have unique links. (e.g. each “Next” link goes to the same URL. Some magic is happening with cookies to enforce state.)
• Multiple pages of reports aren’t numbered.
• First/Next/Previous/Last links always appear, even for single page reports.
• PAC/PIC searches are broken. (Try searching for statements that “contain” some nonsense word. You’ll get the same results as if you’d searched for your own name.)
• Some expenditures are marked with asterisks, but there’s no explanation of what that means.

Let me put it this way: there are some pretty serious usability issues on the citizen side. Without having to go through the rigamarole of trying to remember the password I need only once a year (and it can only be set by contacting the Lt. Gov’s office directly—and no area code is provided when the phone number is first listed) let me just say that using the system as a candidate isn’t a walk in the park either.

Let’s be honest—it’s a simple store data/show data application. It shouldn’t be to hard to do well, if one doesn’t pick the lowest bidder. …

I happen to like Litebox, although it requires a bit of manual setup: moving specific files to the web server. Don’t worry about the setup instructions on the Litebox site. We automate the file interactions below.

Getting Ready

The following requires OS X 10.4, and uses Automator.

Download the following supporting files, and put them on your web server. I put my CSS files in “/css” and my javascript files in “/js”. It’s up to you.

• TagLoader.js — Inside this file, you’ll need to change the two hard-coded file paths to match the paths to the js and css folders on your server. Change the following lines to match your setup:  BASE_JS: '/js/min/',
  BASE_CSS: '/css/',
• litebox-1.0.js
• prototype.lite.js
• moo.fx.js
• lightbox.css

Add the following to your site/theme CSS file. (Feel free to modify this CSS as you prefer. These values suited me.)ul.thumbs {
  text-align:center;
  list-style-type: none;
}
ul.thumbs li {
  display:inline;
  padding: 0;
  margin: 0 5px;
  list-style-type: none;
}

Build the following Automator action (or just download the complete version)

1. Finder : Get Selected Finder Items
2. Preview : Create Thumbnail Images — I left the sized at 128. Leave the suffix as “_tn”; it’s important later on.
3. Automator : (For some reason, WordPress is mangling the following code—it’s not showing the escaped quotes properly.)

   JS_FOLDER="/js/"
   IMG_FOLDER="/images/"
   LB_TAG=$(date "+%Y%m%d-%H%M%S")
   echo "<script type=\"text/javascript\" src=\"${JS_FOLDER}TagLoader.js\"></script>
   <script type=\"text/javascript\"> //<![CDATA[ 
   TagLoader.load('litebox');//
   //]]></script>
   <ul class=\"thumbs\">"
   for tn in "$@"
   do
     #Get img base name (i.e. w/o path)
     tn=${tn##*/}
     #Guess at original image name
     img=${tn%_tn.*}.${tn##*.}
   
     echo "<li><a href=\"${IMG_FOLDER}${img}\" rel=\"lightbox[${LB_TAG}]\"><img src=\"${IMG_FOLDER}${tn}\" alt=\"\" /></a></li>"
   done
   
   echo "</ul>"

4. System : Copy Text To Clipboard
5. (optional) TextEdit : New Text File — Pop down the options. Check “Show action when run” and “Show entire action”

Within the Automator action you just built (and, of course, saved appropriately), you’ll need to modify some code to match your server file paths. Change JS_FOLDER and IMG_FOLDER (the top two lines of the Bash script) to match the file paths.JS_FOLDER="/js/"
IMG_FOLDER="/images/"

Save your Automator action to “~/Library/Workflows/Applications/Finder/”

You now have an Automator action that takes selected image files, automatically makes thumbnails, generates the HTML snippet needed to load Litebox and display the thumbnails. The HTML is copied to the clipboard for easy pasting, Nifty.

If you want to dig further into Automator, you get bonus points for using the “Finder : Get Folder Contents” action instead of having to select individual files. Extra bonus points for integrating the Upload to FTP Automator Action. (Hint: You might have to add it twice: once before the thumbnails are created, and once after.)

Using our new tool

To use the Automator action, select the files you wish to insert/thumbnail/etc. Right-click, pick “Automator”, and from the pop-out menu, pick your new action. It’ll take a moment to run (depending on how many images you selected), and prompt to save the generated html file when it’s completed.

FTP the image files and newly created thumbnails to your web server. Paste the generated HTML into your blog. You’re done!

One bug

One bug I discovered when building this, is that my script assumes the original image file has the same file extension as the thumbnail. If the initial image is a png, Automator will output the thumbnail as a jpg, so you might have to do some find/replace.

The output

I ran my Automator action on three images, and followed the process above. This is the output (I had to manually fix the png file extensions, per my bug, above). Click the images to see Litebox in action!

Introducing Subversion (SVN) Hooks

Subversion allows for custom scripts to be called when it performs certain actions on a repository. Browse to your repository. The server-side one, not your working copy. In /path/to/svn/repository_name/hooks you will see a list of sample scripts for the five actions subversion recognizes.

start-commit

Run before the commit transaction is created. May be used to stop the transaction and return error messages to clients.

pre-commit

Run when the transaction is complete, but before it is committed. It can be used to require comments to include a bug tracker ticket number, or enforce fine-grained access to parts of the repository. It can cause the transaction to rollback and return error messages to clients.

post-commit

Run after the transaction is committed, and a new revision is created. Often used to send notification emails or trigger repository backups. Errors from this hook are ignored Subversion comes with some scripts for common actions. Read the manual for more details.

pre-revprop-change

Revision properties such as log messages aren’t versioned. If you were to go back and change a log message, for example, the old one would be lost. This hook, and its post- counterpart, offer a chance to reject or log the change.

post-revprop-change

Run after revision properties are changed. There are some security restrictions built in, be sure to read the manual.

Well, that sounds simple. All we have to do is write a script that will log into our bug tracking system and insert a comment on the appropriate ticket. That sounds easy enough, but let’s not make things too complicated. With a few minor customizations, Mantis will do most of the work for us.

Customizing Mantis

The manual for Mantis has two relevant pages. One, titled “Source Control Integration”, provides a list of customizable settings, but offers no directions on where these settings are most appropriately accessed. The other, “CVS integration”, is not terribly helpful either, but it does provide another piece of our puzzle—it tells us there is a command line PHP script included with Mantis that accepts a version control (e.g. SVN, CVS) log message via stdin, and adds comments to any tickets as appropriate.

First things first. Create a new user in Mantis. This is the account the script will use. I called mine “svn”. Make sure the user has permission to add comments and resolve tickets. Don’t forget to add it to your project(s).

Open up /path/to/mantis/config_inc.php. If it doesn’t exist, create it. (If it doesn’t exist, you didn’t follow the installation directions.) This file is used to override default settings. It’s not distributed as part of Mantis, so when you upgrade, it won’t be overwritten. This is where you should have your database connection information, and where you would override the default colors, status/severity options, or any number of configurable settings.

Add the following lines to /path/to/mantis/config_inc.php:

$g_source_control_account = 'svn';
$g_source_control_regexp = '/\b(?:bug|issue)\s*[#]{0,1}(\d+)\b/i';

What does it all mean? (Some credit is due Digitalpeer for helping me through this. I found it just as I was getting the integration working. It would have saved some time.)

$g_source_control_account

This is the user we created in Mantis.

$g_source_control_regexp

This is the regular expression Mantis will match against the log message. This is an optional setting; the default is /\bissue [#]{0,1}(\d+)\b/i.

Using a custom RegExp

You’ll notice I defined a regular expression different from the default. I wanted to not only match the string “issue #1337″, but to be a bit more liberal, matching the phrase “bug #1337″ too. For those of you savy enough to craft your own expressions, the first capturable group must be the bug number, or the script won’t work. (This is hard-coded into Mantis.) Hence the non-capturing group (?:bug|issue).

Automatically closing tickets

These custom settings lay the groundwork for Mantis to automatically add comments to tickets based on SVN log messages. There are additional settings we can use to get Mantis to resolve tickets automatically too, based on a different regular expression. To enable this behavior, add the following lines to /path/to/mantis/config_inc.php:

$g_source_control_set_status_to = RESOLVED;
$g_source_control_set_resolution_to = FIXED;
$g_source_control_fixed_regexp = '/\bfix(?:ed|es)\s+(?:bug|issue)?\s*[#]{0,1}(\d+)\b/i';

$g_source_control_set_status_to

The status to set the ticket to. Use CLOSED if you prefer.

$g_source_control_set_resolution_to

The resolution attached to the ticket. There are constants defined for each built-in resolution. You may use a different one, but FIXED makes the most sense.

$g_source_control_fixed_regexp

This is an undocumented setting. The default is to be equal to $g_source_control_regexp, but I only wanted certain tickets to be automatically closed, so I used a different regular expression. My expression matches “fixes issue #1337″, “fix bug #1337″, or simply “fixed #1337″. (Plus several iterations on those words.) The ticket number must be the first captured group. Altering the expression to also accept the words “solved” and “solves” is left as an exercise for the reader.

Testing Mantis

Before we go further, let’s make sure we implemented the Mantis settings properly. Run the following from the command line (alter the paths, of course)
%> /path/to/php /path/to/mantis/core/checkin.php <<< "My web monkey fixed issue #1337 by accident."

If you’ve configured Mantis correctly, there should be a new comment on ticket #1337.

Connecting the pieces

Now that we have Mantis set to accept log messages from our version control system, we need to connect Subversion to it. Remember the hooks from the beginning of this article? We’re going to write one. I used Bash script. (Be nice, I’m still learning Bash.)

Save the following file to /path/to/svn/hooks/post-commit (no file extension):

#!/bin/bash
 
REPOS="$1"
REV="$2"
 
log=$(svnlook log -r $REV $REPOS)
 
/usr/bin/php -q /var/bugtracker/core/checkin.php <<< "$log"

What is svnlook?

The program svnlook is distributed as part of Subversion. Given a revision and repository path as parameters, it will return various properties of the revision, such as the author or which files were changed. In this case we’re asking only for the log message.

Permissions

Give the file permission to executable by your repository. This will be slightly different for each setup, but something similar to the following should do the trick:

%> cd /path/to/svn/hooks
%> chmod 770 post-commit
%> chgrp svn post-commit

More on post-commit

The post-commit hook takes two parameters: the repository path and the revision number. If you needed to run post-commit manually (testing your script!), it would look something like this for revision 746:

%> /path/to/svn/hooks/post-commit /path/to/svn/repository_name 746

SVN and Mantis in perfect harmony

There you have it. That’s how you integrate Subversion with Mantis. If you have questions, feel free to leave a comment.

CAPTCHA (an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart”) has been on my mind ever since Phil Windley suggested a graphical CAPTCHA would make a good web service. I thought there might be those willing to pay to use it. Well, it’s been done.

There is a need for this type of test. Yahoo! and Hotmail use a CAPTCHA to stave off spammers when a user requests an email account. I suspect the most common use on other sites is an attempt block automated comment spam in blogs.

CAPTCHA excludes legitimate users

As the W3C points out graphical CAPTCHAs are a significant barrier to low-vision and blind users. Those with learning disabilities, such as dyslexia, may also be adversely affected. As visual CAPTCHAs become more sophisticated, busy, patterned background becomes more of an issue for color-blind users.

The U.S. Census Bureau estimated that in 1997 about 7.7 million Americans had difficulty seeing the words and letters in an ordinary newspaper. The American Foundation for the blind reported about 5 in 1,000 Americans are legally blind, and gives a low estimate of 1.5 million visually impaired computer users. That’s a fairly significant potential market to ignore.

Requiring users to interpret a visual CAPTCHA may lead to legal challenges. Earlier this month, the National Federation for the Blind filed suit against Target, claiming target.com discriminates by not being accessible to visually impaired users.

Audio CAPTCHA

Some companies are experimenting with audio CAPTCHAs, spelling out random letters with random noise in the background. However, aural disabilities are more common than visual ones, so the approach isn’t really more accessible. Speech recognition software is more advanced than character recognition, so the purported purpose of differentiating between humans and computers is not filled anyway.

CAPTCHA is broken

Several projects to crack common visual CAPTCHA algorithms, particularly The CAPTCHA Project (by the Carnegie Mellon School of Computer Science), the UC Berkeley Computer Vision Group, and Sam Hocevar’s PWNtcha, have had good success. Howard Yeend demonstrated a vulnerability in several public algorithms where he could reuse a solution several thousand times after manually solving it once.

Social engineering is often easier than fancy programming. The first widely recognized social engineering solution was “borrowing” CAPTCHAs from target sites and showing them at entry points to porn sites. Visitors to porn sites would solve the CAPTCHAs, allowing spammers to get essentially free labor. Amazon’s Mechanical Turk (tagline: “Artificial Artificial Intelligence”), which gives micro-payments for simple tasks is an example of another way CAPTCHAs could be defeated. Even at a few cents per image, the cost may still be too high for spammers, but it is a demonstration that the process can be outsourced. After all, the world is flat.

What is the underlying purpose?

The real reason for CAPTCHA is to screen undesirables. For low traffic sites, it means preventing automated access. This can be accomplished in a relatively simple way: add a single required question to the comment submit form. Something like “What color was George Washington’s white horse?” or “Enter the fourth word in this sentence.” This is enough to make the form non-standard, thus unusable by generic bots. Bypassing this added security would be very easy for spammers, the advantage is the relative obscurity of most blogs. To target multiple blogs, a spammer would need to address each one individually; individual attention is unlikely, so I suggest this method is the easiest for bloggers with a knowledge of web programming, and is as accessible as a comment form without a CAPTCHA.

Major sites like Yahoo! and Google have a bigger problem. After all, they are targets both because of the value of their services, and their size. When it first launched Gmail, Google limited accounts to those who had been invited by other active users. Initially there was a good bit of commotion in the tech community as gmail.com addresses became a sign of prestige. The invitation system allows Google to track which users may be abusing the service, and which users invited the abusers. Google has gone a step further, and now allows potential users to have an invitation code sent to their mobile phones. The number of accounts requested per phone number can be tracked. The potential gain from a limited handful of throw-away email accounts, and the cost of mobile phones (even disposable ones) is enough to deter spammers, because less troublesome alternatives exist.

If you look at Google’s account request page, you’ll see a CAPTCHA there. Google responsibly offers a way for users with disabilities to bypass the CAPTCHA, although it involves human-to-human interaction (and quite a bit more time) to complete—a costly alternative.

Real solutions

Several solutions to the problems with CAPTCHA have been proposed and debated. Most have major cost or accessibility problems.

It would seem the only good solution is some sort of federated identity system, which is really just offloading the trouble of user validation to someone else.

After several searches, I discovered IE doesn’t allow the name attribute to be changed after the element is created—and it can’t be set in a DOM compatible way during creation.

Bennett McElwee suggested a solution in his blog that is nicely cross-browser; anything other than IE throws an exception and gets created properly. (I suspect modifying the parent node’s innerHTML would work as well.)

function createElement(type, name) {
   var element = null;

   try {
      // First try the IE way; if this fails then use the standard way
      element = document.createElement('<'+type+' name="'+name+'">');
   } catch (e) {
      // Probably failed because we're not running on IE
   }
   if (!element) {
      element = document.createElement(type);
      element.name = name;
   }
   return element;
}

And your type attribute too

Another part of the project requires we transform checkboxes to radio buttons and hidden fields. This could be accomplished through a page reload, but it’s overkill for such a small change. Once again, in truly DOM-compliant browsers, this requires only a couple of lines of code:

element.setAttribute("type", "radio");

The MSDN reference for the type attribute says:

As of Microsoft Internet Explorer 5, the type property is read/write-once, but only when an input element is created with the createElement method and before it is added to the document.

QuirksMode has a bug report for this, complete with test page and workaround submitted by Stijn Peeters. Stijn admits the workaround needs a little bit of cleanup.

Essentially, his solution is to always remove the element, and recreate a modified one. (See the bug above!) Here’s my solution:

try {
   element.setAttribute("type", "radio");
} catch (e) {
   var newElement = null;
   var tempStr = element.getAttribute("name");
   try {
      newElement = document.createElement("<input type=\"" +typeStr+ "\" name=\"" +tempStr+ "\">");
   } catch (e) {}
   if (!newElement) {
      newElement = document.createElement("input");
      newElement.setAttribute("type", "radio");
      newElement.setAttribute("name", tempStr);
   }
   if (tempStr = element.getAttribute("value")) {
      newElement.setAttribute("value", tempStr);
   }
   element.parentNode.replaceChild(newElement, element);
}

% defaults write com.apple.Safari IncludeDebugMenu 1

The application has a “skin” feature allowing non-technical users to customize colors, font, and a handful of other things easily accomplished with CSS. Rather than edit more than twenty other pages to add an extra <link /> tag, we chose to add the following line to the top of our primary CSS file:

@import("custom.css") screen;

Notice how we defined the media type for the imported CSS file as part of the import rule, as suggested by the CSS2 specification.

A little more background: using a little bit of Apache magic courtesy of mod_rewrite in the httpd.conf, requests for the file custom.css are redirected to custom.php, in the same folder.

<Directory "/www/css">
   RewriteEngine on
   RewriteRule ^custom\.css$ custom.php [nc]
</Directory>

It didn’t work. Firefox’s Javascript Console was pointed directly to the problem: “The stylesheet http://localhost/css/custom.css was not loaded because its MIME type, ‘text/html’, is not ‘text/css’.” (Also, custom.css did not show up in Firefox’s built-in DOM Inspector.) The error was easily fixed, by adding the following line to custom.php:

header("Content-type: text/css");

The styles then loaded properly in Firefox (and Safari), but not in Internet Explorer. Browsing directly to custom.css worked fine in Firefox, but IE behaved oddly. It first acted as if it was about to download the file (oddly, the default action for the working CSS file), but then popped up a dialog saying the URL could not be found.

I then examined the HTTP headers, and compared them with a working file. (It’s off-topic for this post, but I stumbled across this while Googling for solutions. If you’re a geek and want to see some fun headers in the wild, visit Fun With HTTP Headers.)

Headers unique to main.css (working):

Last-Modified: Mon, 09 Jan 2006 22:28:54 GMT
ETag: "ce134-ed8-43c2e3a6"
Accept-Ranges: bytes
Content-Length: 3800
X-Pad: avoid browser bug

Headers unique to custom.css (not working):

X-Powered-By: PHP/5.0.4
Set-Cookie: PHPSESSID=uah38ktr3mgq3td5j6qmcdgjf3; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

The cookie is generated automatically by PHP because of the need to use session information as part of the customization process. But, this is a CSS file, so I want it to be cached by default. Adding the following line before session_start() is called hid the no-cache headers, and things began to work in IE almost as they should.

session_cache_limiter("none");

I said, “Almost.” After tweaking the headers, I could now visit the file directly in IE without the “URL not found” error. (Meaning, it attempted to download it, rather than display it, but it was a step in the right direction.) However, it would still not apply the styles from the imported CSS file. It was difficult to debug as there were no error messages, and IE’s DOM tree is not consistent with W3C standards. The final piece—and I could find no documentation anywhere for this—was to strip the media type from the import statement (at the top of this post), making it:

@import("custom.css");

It sounds easy now that I write it, but this series of bugs cost several hours of work. I hope this post will save someone else similar trouble. (If you see something I missed, or know of a faster course we could have taken to solve this problem, feel free to add your comments!)