compXnonsense

U.S. Courts PACER: An Accessible Open Source API

2012-01-23T18:50:00.000-08:00

Get Access to All Information on the U.S. Courts Docketing System

Anyone who has tried to look up a court case on a government website has run into the Public Access to Court Electronic Records system, or as everyone calls it: PACER. I have developed and just released a new API, that gives programmers access all public information on the U.S. Federal Courts docketing system.

Features include:

1. Search for cases by party name, docket number, and filing date.

2. Retrieve the names of parties to a case, their attorneys, and law firms.

3. Download the entire docket of a particular case.

4. Download pdfs of individual filings and their attachments.

5. Keep track of costs of each PACER transaction.

Right now, there are hooks into all Federal District Courts, most Appeals Courts, most Bankruptcy Courts and also the I.T.C. I am not aware of any other service or API which offers something similar for the I.T.C.

This project does not make PACER free. It still costs $0.08 per page (which can add up quickly). Although the API works perfectly as stand-alone python, it can plug into Django (or any other python framework) very easily. There are also hooks (and some meager documentation) to make it work on google app engine.

Also note that this project is released under the AGPL, a relatively restrictive license.

The project can be found:

https://gitorious.org/pacer-us-courts-api

2011-09-27T19:08:00.001-07:00

Studying hard? Check out my friend's flashcard app: www.kleio.info

2011-09-14T22:09:00.001-07:00

Found a mirror for android source despite kernel.org (and android.git.kernel.org) being down: http://ping.fm/MXbba

2011-09-09T13:17:00.001-07:00

First time a hacking incident directly effected my productivity... kernel.org is down, no more android source.

What is s.nsdsvc.com?

2011-07-21T09:11:00.000-07:00

I have been getting a bunch of hits from http://s.nsdsvc.com/App/DddWrapper.swf?c=3. The hits seem like real people, not robots. But there's nothing on that website. Their whois information is private. If you go to that page you just get a blank flash script (is it blank?). Some random post mentions it and it seems to be related to adobe, but that may be just because it's a ".swf". Anyone have any better ideas?

My guess is that it's a script for bouncing people from one site to another, but I really have no idea.

Trend Micro is Recording Your Every Mouse-Click

2011-04-14T11:29:00.001-07:00

Many of us use some sort of anti-virus protection. One of the more popular products is Trend Micro. After a bit of investigating, I have concluded that once you install a Trend Micro anti-virus product, their software records every link that you click on the internet and sends that information back to Trend Micro servers.

Recommendation: Trend Micro is Spyware. Uninstall it immediately.

Investigation - Odd Website Behavior:

I run a small website. I am kind of obsessed at making sure my site is operating properly and I monitor the logs closely. I started noticing some unusual activity.

To access a large part of my site, you have to be logged in. However, I started noticing requests from users who were not logged in; it appeared they were clicking on links that required them to be logged in. For example, if someone wants to search the database on my site, they must first log in. I saw quite a bit of activity of people who were not logged in trying to search my database. A normal user would never normally do this because in order to get to the search page, you must first log in. When a user tries to search without logging in, the search fails and returns an error message. (Note: It could also happen if the user logs in, keeps their browser open until their session expires and does a searche. However, this is relatively uncommon and was not happening with the frequency I was seeing).

A Possible Connection to Trend Micro

The strange activity got kept on happening over and over again and it got me suspicious. I checked the IP addresses of the users that did these types of searches. I found that several different IPs contributed to the vast majority of these types of errors. I did a reverse DNS on the IP address and I found that every single one of them comes from a Trend Micro server.

I thought that was strange and I wondered why or how Trend Micro was clicking around on my site. Then I noticed that every single one of the queries that came from Trend Micro, was done by a real logged-in person earlier. Sometimes a user would do it days earlier, sometimes hours. But there was no truly unique request coming from Trend Micro.

I then noticed that my site was getting requests to the administrator area. These areas are unexposed to the open internet and should only be accessed by me. I had an "ah ha" moment when I saw that Trend Micro is the installed anti-virus program on my work computer, where I occasionally view the administrator pages.

I have to conclude that Trend Micro is recording the requests from every user that has installed its software. The software must record the user’s internet requests and send them to Trend Micro. Then Trend Micro re-runs them again, probably to do some security analysis.

I have no idea what else Trend Micro is recording. The only thing I know for certain is that they record what you click on the internet. However, it is certainly possible, and based on Trend Micro's past behavior it is probable, that they are recording your every move on your computer.

An Explanation?

I searched the Trend Micro website and I could not find anything that actually discloses what they are doing. My guess however is that it is involved with the Trend Micro Smart Protection Network. The website says that it "[l]everag[es] cloud computing across Trend Micro's security solutions and services." However it does not say that it is tracking everyone's click and movement on the internet.

If Trend Micro is doing what it appears they are doing, it is a huge affront to your privacy and they may be violating the law. Users of any software should know what the software is doing and have an opportunity to turn it off. Trend Micro has not been upfront with their reprehensible behavior.

IP Addresses

Below are some of the IP addresses where I noticed this traffic. All of them are registered to Trend Micro. The 150.70.x.x IPs are registered in Japan and the 216.104.x.x IPs are registered in Cupertino, California.

150.70.172.107
150.70.64.195
150.70.75.27
216.104.15.130
216.104.15.138
216.104.15.142
216.104.15.134

There are probably more IPs than the ones listed above, these are only a snapshot from a couple of days. I am considering blacklisting them from my site.

Lincoln Lawyer: Just a big law-school hypo?

2011-03-21T07:35:00.000-07:00

I just saw Lincoln Lawyer this weekend. It seemed like a good movie but I was not really able to enjoy it. The entire time it brought me back to my professional responsibility class, which I'm taking this semester. Throughout the two hour movie I could not stop analyzing the dozen or so legal ethics issues that popped up. Some of the issues are pretty clear cut: In the first five minutes, the lead protagonist intentionally delays his client's case because his client hadn't paid him. But other issues including attorney client privilege and conflicts are harder and the entire time my mind was racing through model rules of professional responsibility.

The movie is filled with plenty cliches and cheesy scenes (getting paid in cash by a motorcycle gang), but it also has some good ones too ("There is no client as scary as an innocent man.").

Overall a pretty good movie. But if you're a law student watching it, try to relax and enjoy rather than treating it like a law school exam.

Addicted to Logs

2011-02-14T08:09:00.000-08:00

I released my website just a week or two ago. Since then, I've probably checked and rechecked the server logs a hundred times a day. I'm totally addicted. I now check them more than email. When I'm away from my computer I get nervous thinking about the activity I'm missing.

What I love the most is sitting there refreshing the logs watching a user's every move. I see what pages they click on and how long they stay on a page. I try to imagine them sitting there, reading the different blurbs and thinking about where to click next. When they inevitably navigate away from my site I wonder in agony why they would leave. Was the price too high? Does the site look too unprofessional or do they think it's a scam? Or is there some random browser bug on their system that caused it to be displayed improperly? I may never know. But I always blame myself.

But I continue hitting refresh. The unrelenting self-doubt wont stop me from eavesdropping on the next person who comes on the site. I'll change some text here tweak the layout there. I keep wanting to slash and burn whole areas of the site but I'm also too scared to touch it because it works sort-of.

I think I'll just stay here and keep watching.

New Website

2011-02-10T13:46:00.000-08:00

It can be difficult to keep your programming teeth sharp while studying in law school, but I've done just that. I'm happy to say that after learning a host of technologies (see previous post), my new website is largely complete.

The result is Resumé Launchpad. I got the idea for the website while applying to law firms during the end of my first year of law school. It's a tedious process. There are widely published databases containing the firm names, human resources representative names, and email addresses. So the process normally involves building a generic cover letter, copying and pasting the appropriate information into the cover letter, and blasting the email off.

When you first start, you try to customize the cover letter to each firm. It wouldn't be uncommon for me to spend a good twenty to forty minutes per cover letter. But after doing a dozen of these and receiving only form rejection letters, you begin to realize that quantity is more important that quality. Then you literally just modify the name of the firm and recipient and send the email blindly hoping for the best.

I ran into problems. I tried tracking everything in excel, but on several occasions I sent the my application to the same person twice. Worse than that, sometimes I would copy & paste incorrectly and get the firm's name, the recipient's name, or their gender wrong. After sending nearly a hundred cover letters like this, my programming instincts got the better of me and I decided to automate the process.

So I put together a database of legal employers from widely available sources (all legit). Then I wrote a python program which was kind of like a mail-merge on steroids; it could send hundreds of customized emails to various recipients stored in a local database.

Finally, the day of reckoning came. I put together my cover letter, resumé, and other attachments and let my program do its thing. I sent about three hundred emails in a couple of minutes. What would have taken me hours and days took only seconds.

It was successful. I got a number of interviews at some very prestigious firms. At the time, I didn't see it as a product, but more as a tool to make life easier for me. The idea to build a general-purpose web interface to my program started percolating.

The rest is programming history. I learned how to put together a modern complex site and two weeks ago I launched it to the world.

So if you're an attorney, law student, paralegal, or legal secretary and you're looking for legal work, check out www.resumelaunchpad.com.

Websites....

2010-11-20T15:32:00.001-08:00

I consider myself a pretty competent programmer. I can pick up a new language / programming environment in a day and hammer out a cool prototype within three or four days. So when I had an idea for a website, I jumped right in.

Let me start of at the end... the website works! It's in private beta right now and I hope to release it to all in a month.

But it's been a long hard slog; far more work than I imagined. The last website I built was circa 2001. It was really advanced for its time: simple javascript and flash animations! Now, in this Web 2.0 world, "simple" anything won't cut it. It's not just that javascript has changed and flash has been dropped. Today, the list of technologies to choose from to build a site is staggering.

But even if you don't want to learn everything, say you just want to learn enough to put something decent together; well you're still in for a steep learning curve. Here's the list of mandatory technologies I've had to either know or learn to jump into the Web 2.0 world:

Advanced Javascript - learning object oriented programming the javascript way is definitely something new. It isn't bad, there's a lot to like about javascript, just different.
HTML - Remember the days when tags like "center" and "table" were your friend? Forget it, you're just using div and span now.
CSS - I hate CSS. It isn't a programming language at all. It's fine for colors and sizes. But once you use it to do layout, your headaches will begin. It's an art more than a science and nothing feels deterministic:
Do you use the "top" or "margin-top" attribute? Who knows, try both.
Why do we have both inline and inline-block? Because we can.
Unfortunately, CSS is the only thing out there that does what you want. You just have to swallow it.
python - Admittedly, I was already pretty well versed in python (i luv python). If you don't want python, you're going to have to learn PHP or one of the other web-programming languages.

And those are just the languages you'll need to learn. The fun really begins when you include all the libraries:

jQuery - Very cool, I must admit. But for the uninitiated, it's totally different from anything you've done before.

jQuery Plugins - There's a jQuery plugin for everything. Great. Unfortunately, its a new API you have to learn to do anything.

Django - The web-backend. Cool technology but pretty massive and you have to learn quite a bit of it to do anything. Oh, did I mention that to use Django you have to learn its own special html-markup language too? It can easily take a day of tutorials to get familiar with the system. I've been working with it for a month now and I've only started becoming comfortable.

Google App Engine - It isn't necessary to learn google app engine. But they're a ridiculously cheap web-hosting solution... actually, for most cases it's totally free. But they do things so totally different, that there is a nice learning curve. To start with, there is no file system. Enjoy.

And perhaps I forgot the biggest pain: compatibility between different browsers. Just when you think you put the final touches on your beautiful site, you check it out on a different browser and it looks like crap. I'm not a Microsoft-hater, but this experience has definitely left a sour taste in my mouth for IE and I haven't even tried the mobile browsers.

Moral of the story is that making a website is pretty complicated involving nearly a dozen technologies. But websites are the future (ahem, present) and it feels pretty gratifying being able to put a real-life site online.

More Employed U.S. Engineers!

2010-11-20T15:02:00.000-08:00

I saw this article about China's new UAV drones, some of which are superior in some ways than America's. Most patriots probably have a knot in their stomach, concerned that China just beat us in one more thing.

Don't fret! Rejoice!

This is the beginning of a new arms race! No U.S. chief of staff will let some other country have something that flies faster, harder, or longer than our overly expensive aircraft. They're going to have to make something better. That in turn, means more big expensive projects and more employed engineers. Looking forward to it!

Bloom Energy Hype

2010-02-26T11:01:00.000-08:00

There has been quite a bit of hub-ub about Bloom energy. Probably overly hyped. People have been making SOFC fuel cells for years. I don't know why he thinks he can make them so cheap.

If you want to see Blooms patents, everything is online:

http://tinyurl.com/BloomEnergyPatent

Posner off his Rocker

2009-11-02T18:59:00.000-08:00

I usually enjoy reading the Posner-Becker blog for its great analysis of current events. But a recent post has really amazed me in its stupidity. The article is generally ok, but Posner makes two extremely ridiculous arguments.

First, Posner rants about how American's health care system is a result of "a large 'underclass' (corresponding to the residents of our inner cities) that is poor and has a very high murder rate and high infant mortality and a high incidence of AIDS and other diseases." As a federal judge, you would think that Posner would support his argument with some facts. As it turns out less than a quarter of the United State's poor live in cities. Most live in rural areas. Look a bit harder before blaming America's health problems on the urban areas.

Posner's next argument is that Europeans expect worse medical care than Americans because they are more "fatalistic." Posner has no idea what he is talking about and is totally speculating. Well I thought I would add my own counter-speculation... American's are far more religious than Europeans and are much more likely to accept their "fate" and refuse further medical procedures than Europeans. Because Europeans are less devout they are are more likely to do a rational analysis of their medical situation. Also, Europeans are on average more educated and are probably more informed of their medical options. Of course my facts (at least pertaining to the second paragraph) are just as unsubstantiated as Posners.

Iqbal Strikes Orly Tatz!

2009-09-19T11:07:00.000-07:00

Last May the Supreme Court of the United States issued its decision Ashcroft v. Iqbal. The case concerned Iqbal who was the unwitting victim of the post-9/11 round-up of suspected terrorists. Even though there was little evidence that Iqbal was a terrorist, the government used a variety harsh interrogation tactics on him and kept him in jail for many months. Iqbal sued the government and the case went to the supreme court. The court threw out Iqbal's case. What was special in this case is that they threw out the case at a very early stage of the litigation, at the pleading stage. Previously, it was very easy to get passed the pleading stage.

Now, the holding in Ashcroft v. Iqbal has now been used against Orly Tatz, who claimed that Barack Obama was not born in the United States and therefore is ineligible to be President. Citing Iqbal, the court said that "To state a claim upon which relief may be granted, Plaintiff must allege sufficient facts to state a claim to relief that is plausible on its face."

It will be interesting to see if this holds up on appeal.

Is the ABA biased or are republicans?

2009-03-30T12:32:00.001-07:00

A recent newstory by the New York Times questions the ABA's neutrality on advising the president on the qualifiactions of federal judges. The story brings to light evidence that the ABA is more likely to favor judges nominated by democratic presidents than republican presidents. It then goes on to make the claim that the ABA favors liberal judges over conservative judges.

The New York Time's logic is flawed and they ommit several other considerations which may also account for the disparity. Below are several other reasons why the ABA may favor judges by democratic presidents rather than republican presidents.

Republicans are more likely to nominate right-wing judges than democrats are to pick left-wing judges

Republicans have been in power longer than democrats and they may feel that it may be easier to push more ideological candidates through Congress than democrats. Furthermore, there is something about the liberal-tone which is more concilatory. Democrats are therefore more likely to pick centrist judges than extreme ones.

Ideologically liberal judges are, on average, more qualified than ideologically conservative judges.

Law professors tend to be liberal. It is clear that they have a strong influence on their students. There are probably more liberal lawyers than conservative ones. Therefore there is probably a larger candidate pool to choose from for democratic judges, and thus better qualified judges.

The ideals of the ABA are better matched to liberals.

The ABA highly value equal protection and compassion, ideals that are much more associated with liberal values than conservative ones.

The New York Times article jumps over all of these other factors. The reporting reeks of bias.

Conficker Day is Coming

2009-03-30T07:36:00.000-07:00

On April 1, 2009 the Conficker virus will activate and start unleashing its malware on the internet. The virus is already running on over ten million computers, silently awaiting orders. No one knows what those new orders will be, but it can potentially be devastating. Some have hypothesized that Conficker could launch a national cyber-attack or even take down the entire internet.

Up until today, Conficker has been undetectable by anti-malware software and it seems unlikely that a mainstream fix will be available before April 1st. The virus is incredibly powerful: it uses multiple attacks to infect a computer, a genious encryption system to protect itself, and a p2p network to spread its malicous code. Read more about this amazing worm here, and hope that on April 1st the internet keeps on running.

I had the right idea....

2009-03-06T10:06:00.000-08:00

In a very strange coincidence, Carnegie Mellon released a report last month that basically did the same analysis that I did in my previous post (with significantly more robust analysis). They basically came up with the same conclusion... that the Volt won't really save you money. Without too much explanation, they took it further doubting that the Volt will significantly reduce emissions too. Read the paper. When you're done, read GM's response.

How the Chevy Volt doesn't save you money

2009-02-17T22:20:00.000-08:00

Everyone discusses the MPG of cars and for good enough reason. If you're interested in reducing our dependence on foreign oil then you want to reduce the number of gallons you use. However the MPG metric breaks down when you try to compare normal gasoline cars with electric vehicles (like the forthcoming Chevy Volt). A much more relevant metric is Miles per Dollar rather than Miles per Gallon.

So lets work out the Miles per Dollar for a standard car and also for the Chevy Volt. First we have to make some assumptions about gasoline cars. Lets say gas costs $1.95 a gallon. Lets also assume that your average sedan gets about 27 miles per gallon.

Now we have to make some assumptions about the cost of electricity and figure out how much the Volt uses. According to wikipedia, the Volt uses only electricity for its first 40 miles. To run those first 40 miles, it uses up 8kWH of electricity. Depending on where you live, electricity in the US costs as little as 6 cents/kWH in places like Washington and Idaho and up to 17 cents/kWH in New York and California. After the first 40 miles, the Volt operates like a normal car running at 50MPG.

I'll omit the math, but we get the following result:
Normal Sedan MP$: 13.8 miles per dollar
Chevy Volt first 40 miles MP$ (cheap electricty): 83 miles per dollar
Chevy Volt first 40 miles MP$ (expensive electricty): 29 miles per dollar
Chevy Volt after 40 miles MP$: 26 miles per dollar

So depending on where you live and how much you drive, the Volt will be anywhere from 100% to 500% cheaper to run compared to an average Sedan.

That sounds a whole lot cheaper, right! Wrong! To see why, we have to take the calculation a step further.

Suppose you drive 50 miles round trip to work every day and you live in a state with average gas prices, like Texas. In your sedan the daily commute should cost you about $3.61. But in a Volt it will only cost $1.19. Thats a savings of $2.42 every day. Now say you're a workaholic and you make trips like this 7 days a week for 52 weeks a year. Thats a yearly mileage of 18,200 miles, which is pretty high for most people. Driving a Volt you'll get a savings of $881.28 per year! But lets assume that GM engineers are even better than we expect and instead of saving $881.28 a year, we actually save $1000 a year.

Still with me? Good.

Now here comes the big question: much more do you think the Volt costs compared to a similar gasoline car? Lets compare it to a Chevy Malibu which gets about 27MPG and costs about $22,000. The final sticker price on the Volt is not set yet, but everyone says GM wants to get it down to around $32,000 (thats including all of the tax breaks). But lets give GM some credit (even though they're already getting enough from our government!) and assume that in a couple of years they will be able to bring it down further to $28,000.

So a Volt will certainly cost at least $6,000 more than a Malibu. Suppose that instead of putting that six grand into your car, you have the option of putting it into savings earning 5% interest. Therefore, in order for the Chevy Volt to win out over the Malibu, you will have to drive the Volt for ten years!

Thats right. If you want to make back your initial investment in a Volt, you have to drive the car for ten years!

Unfortunatly, GM says that the Volt battery is only expected to last ten years.

There are plenty of reasons to buy a Volt, but saving money isn't one of them.

Wireless Protocol for a Toothbrush

2009-01-01T14:42:00.000-08:00

The most interesting gift I received this holiday season was a toothbrush. You've probably seen advanced toothbrushes before but this one really takes the cake. I'm talking about OralB's Triumph "smart" toothbrush. Like most advanced toothbrushes, the Triumph has a powerful mechanical head and also a timer that vibrates when you brush for long enough. However, the Triumph goes further by adding a wireless communication link with a base station. The base station displays your elapsed brushing time, the current brushing mode, a low battery warning, and tells you if you're brushing too hard. There are a few modifiable options and if someone else sticks on their own toothbrush head, it loads up their saved options. I can't go into all the features here... the manual is ten pages long. This may be toothbrush technology overkill... but who cares, its really cool!

The the real kicker here is the wireless technology. I wonder what wireless protocol OralB uses. The communication is fairly simple and the data rate is minuscule so I would bet that it is just an in-house protocol. But if it uses a WiFi connection, it would be fun to be able to log into your toothbrush's embedded web server and upload toothbrush mods. You could create your own brushing modes and customize their vibration pattern. By modulating the vibration pattern you could even make the toothbrush emit simple tones. Using a more complex pattern you could make your brush play songs while you brush.

Of course any wireless protocol does raise security concerns. A hacker could break into the toothbrush, and modify it to give you a sub-par brushing experience. A hostile country could attack our toothbrushing infrastructure to increase our national rate of cavities or gingivitus. OralB should know that any time you open up your device to the airwaves you have to protect it from malicious users.

...And I'm only half-joking about all of this.

New SugarSync Features in the Pipeline?

2008-12-17T22:57:00.001-08:00

One of my favorite software tools of all time, SugarSync, seems to be ready to shake things up again. I already wrote about this company and their killer backup and sync software. They have since released their iPhone and blackberry client but just recently released a new windows mobile client. They have also announced upcoming Symbian and (long awaited) Linux support too.

But now things are getting really interesting. Now it seems that they are getting ready to sync up with social networks. I haven't seen anything from SugarSync yet, but I can presume that once your data is synced with SugarSync, the new features will allow you to easily (or maybe even automatically) send your data to facebook, myspace, youtube, and the like. Thus, all of your "stuff" can be streamed out all over the internet even though its only uploaded once. It seems that they're pitching this for media publishers who upload their crap to a dozen different websites every time they make a small change. But it would be nice for consumers too.

Additionally, on their second, somewhat hidden website, they have a modified logo with the image of a TV being synced. It seems that Sugarsync has plans to invade the living room. I'm not sure how this would work, but I would bet its either an XBox Live application, or something similar to the NetFlix appliance. Its a great idea. You could go to your friends house, turn on his XBox and listen to all the music thats sitting on your home computer... even if your computer is turned off.

Its clear that SugarSync wants to be on anything with a display to broaden its user-base to the max. I'm all for it. The software is easy to use, works great, and I can't wait to see what they have got next. SugarSync is the begining of true cloud computing.

Has Obama infected the tech-world?

2008-12-13T10:22:00.000-08:00

It seemed like just a couple of months ago, the only thing the tech-world was interested in was adding new facebook gadgets and finding another way to make an old application "social" (See flock, wesabe, mint, and probably hundreds more). But in the last few months I've seen some change in the blogosphere and I'm wondering if its here to stay. Some big players (see Lessig) are starting to pay attention to the "stuff that matters." Even Obama's site change.gov, with its web 2.0 and social feel, bridges the do-goodery world with the tech one.

I can't tell if this is the beginning of a trend or just a handful of people caught up with Obamaitis. I hope its the former, but for it to have any impact and produce anything impressive, it has got to keep its pace up for at least six months.

I'm gonna start looking out for this more now.

Copyright Violations as bad as Child Pornography?

2008-11-16T05:39:00.001-08:00

I found this legal factoid in a washington post article a bit surprising.... "Web hosting providers are generally not liable for illegal activity carried out on their networks, except in cases involving copyright violations and child pornography." Hmmm... why are copyright violations being held to the same standard as child pornography? I was about to get myself into a big huff believing that recording and movie industry lobbyists snuck this in without anyone realizing.

But then I did a bit of research and realized that the quote is actually innacurate. Web hosting providers can get themselves into legal trouble for a variety of reasons other than copyright violations and child pornography including: trademark violations, inciting or abating terrorism, and defamation (with limits). Additionally, ISPs are almost never liable unless they are aware of the activity on their network. So if the ISP is ignorant of the activity, then they're probably safe.

Future of the Web?

2008-04-03T02:12:00.000-07:00

When the internet was first invented in the 70s it was used to connect military personal and emergency workers in a time of crisis. It was purely utilitarian: used to convey facts and events from one point to another.

However as time went on, and researches in academia recognized the internet's utility, they started using it to share research between colleagues. Initially it was largely used to send data from one point to another but later became a place where ideas could be shared.

In the mid to late 80s the internet evolved further. No longer was it limited to well funded researchers in big universities. Any technologically apt person with a couple hundred dollars could get on the internet and talk to users across the world. This is where the social web really began: the old BBSs' and newsgroups of the early internet. Instead of communicating purely in ideas, jokes and stories were being told. People became "friends" online. This concept continued throughout the 90s. AOL helped bring chartrooms to the masses and all of a sudden millions of people were meeting online.

However for years the internet was really limited to the computer literate. Many didn't understand why or how to use computers. The internet was mostly a domain for nerds and businesses who knew how to gain an edge. It took another ten years, a whole generation, for everyone to catch on. In the late nineties we saw the beginning of the information revolution. Even if many did not know how to use the internet, nearly everyone realized its utility.

Now ten years later so much is happening its harder to pin down exactly where we are or where we're going. Nearly all of the major websites today have some social component to them. Most grandparents know how to get on the internet to read and write email. Anyone living on one side of the world can get immediate and personal access to others living on the other side.

The notion of the internet cloud has emerged. Instead of a clear-cut way to send information from one point to another, we have something as complex and nuanced as our own society. We submit information which can be read, commented, tagged, or even altered by anyone, but mostly by our own friends and connections. Instead of interacting with anonymous people across the world we have now started interacting with our own friends over the inernet.

I have a couple of predictions. As more and more people start jumping on the web and interact socially on it, we'll start to see the technology content and "nerd" sites lose market share to more common interests sites. A big milestone will be when Huffington Post beats TechCrunch as the #1 blog on the internet. Keep updated here.

However the web will go much further than blogs and social networks. All sorts of new technology is bringing locality onto the web. Instead of connecting people across the world, I think we'll start seeing the web connect people next door. Craigslist is a good example but I think we'll start seeing much more sophisticated. You can already peer into other people's houses with amazing detail. Its only a matter of time before all of your neighbors are indexed, tagged, and shared.

I'm not quite sure what the implications for the new local internet will be. Obviously there are privacy issues. There are plenty of avenues for spammers and stalkers alike to take advantage of the technology. But I think there is much good that come out of it too. It will connect and organize local communities. It will bring these communities closer together and make them feel more like home.

Honestly I'm pretty excited about whats coming out on the internet. The dot-com bubble didn't burst, it just deflated for a couple years. But now its back, albeit a bit less obnoxious and we're going to see loads of new changes.

The Backup Problem Is Solved

2008-03-31T13:01:00.000-07:00

Like just about everyone else I hate doing backups. My main computer is a laptop so I use external hard-drives to backup everything. Jesus are external hard-drives finicky! I've gone through three broken drives in two years. It sucks up my time and my money.

Well no more. I've found SugarSync.

SugarSync is honestly one of the coolest Web-2.0 technologies I've seen yet. It solves the backup problem with as few headaches as possible. There is no need to buy extra drives, DVD-RWs, or backup appliances. Everything is stored on the internet.

One downside is that SugarSync is kinda pricy. It charges a yearly subscription fee which costs about the same price as buying a hard drive every year. However I bet the price will come down and besides, there are loads of extra perks.

Because everything is stored on the internet you can access your documents from anywhere. It also uses Amazon's redundant backup system so there is zero chance you'll lose any data. It can also sync data from multiple computers/laptops/phones. It runs quietly in the background and and uses less system resources than winamp. But best of all, its brain-dead easy while being extremely powerful.

I'm normally a skeptic when it comes to new technology, however take my word for it... this is good, real good. Buy the product, or if you can, some stock in the company.

Lockheed: Who are you working for?

2007-07-29T19:21:00.001-07:00

I have always been amused with Lockheed's corporate slogan: "We never forget who we're working for". Who the hell are they working for? My first guess would be their shareholders. The companies stock price is prominently displayed on the front page of their corp orate web site. They are a for-profit company so they must eventually answer to their shareholders.

But we all know thats not the image that Lockheed wants to project. They want an altruistic image. They want to show us lay-people that they aren't a giant defense contractor which spends millions of dollars a year to buy our government. They want us to think that they are really here to help us out.

But their slogan doesn't make clear who they are "working for". Are they working for the soldiers? the American people? the pentagon? the president? This is just smoke-and-mirrors marketing at its best.

On the Lockheed website the slogan is displayed underneath their logo, but it isn't mentioned anywhere else. Their "Vision and Value" statement seems to put their customer (the pentagon) above all else.

I don't have a problem with companies which are motivated by their customer satisfaction, stock price, or plain old greed. If they want to serve the interests of the pentagon over the will of the American people then thats a worthy purpose, and they should explain it. However I do have a problem with idiotic slogans which are only intended to obfuscate the companies real motivation.

Update: I'm not the only one who has a problem with this slogan.

WebOS Farce

2007-07-23T20:38:00.000-07:00

I've been playing around with a handful of WebOS's as of late. In the last six months they have sprung up like hot cakes. These web-gadgets are wonderfully cool and wonderfully useless. The value of having an internet desktop on top of your own desktop seems slight. I could only see it being useful for someone paranoid about privacy or who jumps around between dozens of computers. But even then, there are better tools for doing either job. And if you're paranoid about privacy you're hardly going to put all of your trust in some quirky WebOS company. Especially given their history.

I also don't understand why nearly all WebOS designers decide to implement their "OS" in their own proprietary AJAX/flash system. Every WebOS has to write their own text editor, calculator, web browser, file management system, etc. It seems very inefficient to rewrite all of those programs. Not to mention that the quality of the apps are much much worse then their desktop counterparts.

Instead of writing a whole new operating system with a closed set of apps, it makes much more sense to provide a web-interface to a real computer or virtual machine. The reason why Windows, Mac, and Linux are popular is because its easy to write software for them. If any WebOS ever takes off they are going to have to be able to run lots of software.

One implementation may be a WebOS sever which simply finds a user's virtual machine image and loads it up. It then launches a browser plugin like VNC or remote desktop. You then get access to a whole computer, not some mushy half-ass web-app.

All of these pieces already exist. All you need to do is hook them up. You could use the open source implementation of remote desktop, the open source virtual machine software, and then either find or write some kind of user and security management front-end. I'm kinda amazed no one has done it yet and called it a WebOS.

Finishing Up Our Release Cycle

2007-07-21T14:02:00.000-07:00

At my company we just finished development of our major product and are finally shipping the new version. I have to say that I'm pretty happy that the release cycle is beginning to come to a close. There is nothing more boring or soul-sucking then fixing bug after boring bug.

As would be expected from just about any software project, our project shipped about four months later than our originally intended target. I'm pretty lucky to work in a field which isn't extremely time critical. The four month slip wasn't met with huge disdain. Our product was almost three years in the making so four months wasn't the end of the world.

Honestly, even though we had many problems, I think our release went/is going pretty well. Early in the dev cycle we could have and should have written more validations, we should have tried harder to create a more detailed spec of what we wanted the product to do. However overall I'm pretty pleased with the result.

So when you read those crazy statistics about how 30% of projects fail and 50% have serious problems, I'm happy to say our project fell in the 20% success category.

I'm not a resource

2007-07-19T20:05:00.000-07:00

Has your manager ever referred to you as a "resource"? Its pretty demeaning. I'm an engineer, software developer, person, and/or asshole, but I'm not a fucking resource.

Why Functional Programming Languages Fail

2007-07-13T22:23:00.000-07:00

I've recently been dabbling in Haskel. Its a purely functional programming language and I have to say that its pretty cool. I've also messed around with SMLNJ in college and enjoyed it.

The proponents of functional programming languages always say that they are easier to program in, more concise, and are less prone to errors. The detractors always say that they're slow. In response to that proponents of functional programming languages say they aren't that much slower and that most people don't care about performance anyway.

So why isn't everyone programming in a functional language?

My theory is that the proponents of functional languages have their heads up their academic asses. Everyone writing interesting and marketable code cares about performance. Why do you think Intel and AMD make so much money selling the latest and greatest silicon? Could you imagine implementing OpenGL, a JVM, a JIT, or a graphics library in a functional programming language? It would be unbearably slow.

Some argue that if you try hard enough, you can get Haskel to run just as fast as C. They are probably right. But I would bet that its easier to write really fast C code than it is to write really fast Haskel code.

Also with the advent of quality static analysis (which in a couple years I believe will be the default) many of the benefits of a functional programming language can also be found in C/C++.

For everyone else who isn't writing interesting or marketable code, there are languages like Python and Visual Basic. People who write in this straitforward and simple languages prefer very simple languges. Its much easier to understand 'if' and 'for' statements (and also structs and classes) than it is to understand foldr, monads, and polymorphism.

Today we're not programming in Haskel because most of us care about performance, and those who don't prefer easy scripting languages.

Open Source Bashing

2007-07-13T18:46:00.000-07:00

Now I'm not an open source zealot or anything, but it pisses me off when the government comes in with their preconceived notions about what open source "is" and why its bad. Check out this report just issued by the FCC. Especially the sentence:

A system that is wholly dependent on open source elements will have
a high burden to demonstrate that it is sufficiently secure to warrant
authorization as a software defined radio.

I think that any software going into an SDR system should have a high security burden, open source or not. To just assume that open source code is crap is a bad move on the governments fault. I wouldn't be surprised if someone at Cisco was all buddy-buddy with some guy at the FCC and got this report issued.

Back In Time Debuggers

2007-06-02T12:53:00.000-07:00

In the near future as processing power gets cheaper and developing software gets harder Back in Time debuggers will become much more prominent. Back in time debuggers allow you step a program backwards just as you would forwards.

The debugging benefits to going back in time are huge. Normally when finding a problem in computer code you run the program over and over again tracking down where the bug first occurred. Sometimes running the program can take several minutes. Sometimes the bug does not reproduce when you re-run the program. With a back in time debugger these problems largely go away. All you have to do is get the bug to reproduce once, and then step backwards to where it first occurred.

The idea of backwards debugging has been around for a long time. However due to recent improvements in hardware and growing complexity in software their time to shine is really coming.
Next, I'll try to come up with a list of the major players in both the academic world and industry.

The Five Software Development Tools You Wish You Had

2007-04-07T20:23:00.000-07:00

Last week I went to the embedded systems conference (ESC) in San Josa, CA. This is my second time going and last time I was unimpressed. This time however there were five technologies which really shook my cookie.

All of the products come from different vendors so there is no chance that we will get all of them in a single bundle any time soon. Also, they all cost big $$$ so don't expect them to be GPL'd either.

5) WindRiver's Sensor Points - This is a pretty cool technology which is like an automatic update API for arbitrary code. It allows you to send and run arbitrary code to any system running VxWorks. Thus if you find a bug after your product has been released, you can send a patch to the system while its still running. There are some security concerns here which I never got an answer to, but I'm going to assume (hope) that WindRiver has got those worked out.

4) Virtutech Hardware Simulators - These are some pretty insane simulators. Once you define your hardware model they run the hardware on your host as a Virtual Machine. In many cases it can run even faster than real-time. But thats not all, you can turn on checkpointing in the simulator and record the state of the simulator over time. They you can actually run the simulator backwards and forwards, and even send the checkpoints over the internet so others can see view its state as well.
Sounds great right? The catch is creating the hardware model. Making new models is both technically challenging and can take months. They have a number of models already created but if you have your own funky hardware you're probably going to have to do a lot from scratch. Virtuatech makes their money by creating the model for you... which can be expensive.

3) QNX's Multi-Core Analysis Tool
Multi-core is taking off and managing all of those cores is not going to be trivial. I took a look at this tool and I was pretty impressed. You can see what each of your cores is doing at any given moment. It tracks which programs are running and what types of system calls and interrupts are occurring on each core in a nice easy to use graphical interface.

2) Coverity Static Analysis (and others)
This product has got enough of its own publicity so I'm not even going to go further here. However I will add that it is really cool, and everyone should be using some sort of static analysis (if they can afford it).

1) Green Hills Software and Lauterbach's Back-in-Time Debuggers
These technologies are really cool... you have to see em to believe em. They record the state of your system at every point in time and lets you go back in time and check out memory, registers and more. This would be helpful for tracking down hard bugs and extremely useful just in general.

The catch is that you need special hardware to run this stuff though green hills has some workaround technology called TraceEdge.

WindRiver and RTLinux .... My take on it

2007-03-26T00:36:00.000-07:00

I know the news is about two months old, but I don't like to rush in to commentary...

WindRiver's purchase of RTLinux is the first smart move that WindRiver has made in years. Before their new acquisition, WindRiver only sold their legacy OS VxWorks and a six-month old version of linux kernel they downloaded from kernel.org.

Let me just rip into VxWorks for a bit, cause I have some well needed venting to do.....

Technologically, VxWorks lags behind every other major embedded operating system. Its amazing that the most popular OS in the industry is also the worst. One of their more recent innovations, memory protected processes, has been in most embedded operating systems for years.

Also their ARINC-653 partitioning is pathetic. They implement ARINC by running a master VxWorks OS which schedules slave VxWorks OS's which each runs its own ARINC partition. So if you want three ARINC partitions, you will need to run four copies of VxWorks on your hardware. Pretty much all other competitors can sanely implement ARINC with just one OS.

The main argument for buying VxWorks, is that your code is locked into it and you have no other choice.

... End anti VxWorks Rant

So even though VxWorks continues to make WindRiver a lot of dough, its not going to get them anywhere interesting.

On the other hand, their purchase of RTLinux is interesting. They have a real chance of bringing enterprise software developers into the embedded world. Most embedded products only have a small amount of hardware-related or realtime code. Everything else is pretty similar to desktop development. If RTLinux can get every-day software engineers to work on an embedded systems, it should become pretty popular.

Buying RTLinux brings WindRiver's technology to the forefront and I hope it moves our industry forward. However I doubt Windriver will make any money off of it. Whenever you sell Linux products, you also have to give them away for free. Why would anyone buy RTLinux when they can download it for free?

.NET Microframework

2007-03-26T00:20:00.000-07:00

So Michael Hall and the rest of the .NET nerds are making a big rucus over their .net microframework. They promise to let all of the C# junkies start writing code for the next generation ipods and cell phones. Most C# junkies don't know what malloc is (let alone an interrupt) so I am skeptical. But if Microsoft can get non-embedded software engineers into embedded technology they will make the big bucks.

The problem with the microframework is that it does not have any real-time support and only runs on ARM. There are very few BSPs (I think only one!). This API is a good start but what is really needed will be more of a hybrid.

What you really want is to write all of your real-time code on a true embedded OS, and the high-level non-critical apps in a managed language like C# or Java. Microsoft has not pulled out a solution like that yet, but others have.

AONIX has a real-time java VM which runs on top an embedded OS. It doesn't look like its catching on yet, but I do believe that the technology is the way of the future (give it five years).

Until Microsoft comes out with a solution that plays nice with real-time systems, all they are offering is an API with a huge memory footprint (half a Meg!), horrible threading support, no partitioning, and crappy performance. Good work Microsoft nerds.

Ripping Apart Microsoft Visual C++: Performance

2007-03-06T10:24:00.000-08:00

I said I'd do it and I never lie. Lets tear apart Visual Studio. For now I'll start with performance, not my favorite topic, but I'll get to the debugging features later. Each one of these entries deserves its own bog post, but who cares, you probably just think this is flame anyway (I think its thoughtful analysis!).

Performance
1) Starting Up (and Stability)
2) Loading DLL debug modules
3) Running a program

Performance
1) Starting Up
Normally I don't really care about program start up time. I use Visual Studio extensivly, once I start it up, I should never to start it up again. For me its a constant-order performance hit which should only effect me when I restart my computer. Most people restart the computer once a day, I do it much less. Waiting an extra ten seconds a day isn't a big deal.
However Visual Studio crashes. It crashes a lot. I find that halting in the wrong place, or stopping debugging at a key moment will cause it to crash. This happens about 4 to 5 times a day. Therefore 4 to 5 times a day I find myself restarting Visual Studio, loading my project, and pulling everything back into RAM. Altogether its probably a 45 second ordeal which is over three minutes a day. Not a biggie, but definitely an annoyance.

2) Loading DLL debug information
Why does visual studio have to load all of the debug information about a DLL the instant its loaded into program memory? It doesn't take long to load each DLL, probably about a quarter second. However most modern win32 apps link in about 5 to 15 DLLs (complicated programs much more). Most of the DLLs are loaded up at the beginning of program execution so there is a visible delay waiting for them when your program starts. That means I wait two to three seconds every time I want to run a program, even if its a simple console app. If I run the console app alone it completes in a fraction of a second.

This is only a real problem with the CLR. I'm guessing that its a problem with JIT caching or something, but I'm not too sure.

3) Program Performance
A good debugger should be as passive as possible. However running a program within visual studio is significantly slower then running it standalone. I'm not too sure why but I'm guessing that my item above plays a large roal. It could also be that Visual Studio installs all sorts of exception handlers and instrumentation to get things working. With managed CLR Visual Studio may be doing much much more. But its all really speculation.

Blogging on weekends

2007-03-04T20:08:00.000-08:00

Why is it that there are never any good blog posts on weekends? Its almost as if people are not in front of computers or god-forbid.... outdoors.

Getting a new boss

2007-03-03T18:44:00.000-08:00

My boss is quitting my company. He wants to move back with family and have kids and stuff. Good for him, but crappy for me. The guy who is supposed to replace has some very big shoes to fill.

My old boss was excellent. He cared about and believed in the product we're making, he supported my colleagues and I to the best of his ability. He let me have very flexible work hours and was easy about taking vacation days. Most importantly he was extremely enthusiastic about our work and it was contagious.

The guy who is going to take his place isn't bad. I know him and he's a nice guy. But he isn't in the same league as my former boss. Additionally he is about to start managing a huge code-base that he knows nothing about. My former boss wrote/architected a good amount of our code and now this other guy is supposed to come in and take it over.

It'll be interesting to see how it plays out.

Crash the Visual Studio Compiler....

2007-03-02T19:23:00.000-08:00

Wanna know a real easy way to crash the visual studio compiler? Compile the following c++ file using Microsoft's CLR:

_w64 a;

For some reason the Visual Studio compiler really doesn't like _w64. Its pretty funny.

Java and its insecurities

2007-02-28T12:13:00.000-08:00

Why are there so many Java cheerleaders out there telling us that its so much faster than C/C++? I feel like for every one article complaining about how Java is too slow, there are dozens that spring up explaining the the opposite?

I think there is a real Java inferiority complex going on.

Finding bugs that don't want to be found

2007-02-27T20:19:00.000-08:00

I'm a big complainer when it comes to modern IDEs and debuggers. They mostly suck. I've got a good example of how they suck.

For the past week I've been working on a bug which only reproduces about 1 out of 500 times I run my validation test. It takes about 30 seconds to run the test, so even if I script the validation, it still takes minutes to reproduce the problem. Obviously there is some crazy race condition going on.

So where would you start? At this point nearly every debugger in the world is useless. You cannot sit there and run your program 500 times, it would take forever. And even if you did manage to reproduce it in the debugger, if you step over you precious bug you're screwed.

Well I started off by adding assertions... everywhere. Assertions are kind of like printf's except they only do the printing on a condition and when they get hit, they normally exit the program. One benefit of assertions over printfs is that you can keep them in your code. They also serve as documentation for other programmers who comes along.

I was able to get my assertion to be hit. But it wasn't enough. In order to figure out what was wrong I needed to know the state of some data structures when the assertion was hit. So I needed to use printfs... there was no other way to do it. I started dumping lots and lots of data.

After I inserted my logging code I tried reproducing the bug again. Every run I would dump about 5MB of logging data... for 200 runs thats 1GB. After about 250 runs I hit my assertion. I looked through the logging output and found my bug... I was locking a data structure just a couple of lines too late. I moved the lock up a couple of lines and the bug vanished. It was a two-line fix to very hard problem.

In a perfect world finding this bug could have been much easier. It would be really nice if when the assertion was hit, a debugger would automatically be launched on that program. Once the debugger was launched I could view all of the data structures that got corrupted. In this particular case that wouldn't have been enough. I also needed the execution history of how the data structure got corrupt. However that too would not have been enough because there was another process which was writing to the data structure too. I also needed to know know how that process was changing the data structure....

To solve this problem I not only needed to know what code I ran, or the data structures when that code was run, I also needed to know how those data structures changed over time. Thats where the difficulty of debugging comes in, figuring out when and how data structures change.

Can visual studio help you with this? How about gdb or eclipse? No, no and no. They all suck at debugging and not a single one would help you get closer to finding the problem. I need something else...

How Idiots Develop Software...

2007-02-23T20:41:00.000-08:00

I hate all of the modern software development tools. Nearly all of the IDE's out there focus on writing code, even though about 90% of your time is spent debugging code. I think the motivation behind this is that IDE developers have no idea how to make a debugger, so they help us with writing code so we can litter our code with printf's.

I'm sick of using printfs. Why is it that developers still use them even though its a 30 year old technique? Every time you want to find something out about how your code is working, you have to find the printf insertion point, recompile your entire project (which could take minutes), and then re-run the program until your printf gets hit. And what happens when the printf doesn't get hit, or it doesn't spit out enough information? Well then you have to start all over again. If you repeat this process enough, pretty soon your code is littered with printf's and the debugging output becomes a nightmare to analyze.

I got more gripes with modern debuggers....

One of the biggest advances in the new IDEs are error highlighting. Don't get me wrong, error highlighting is cool. But how much of your day is spent tracking down hard to find syntax errors. For me its about 20 to 30 seconds a day. So thank you error highlighting! You saved me 2 hours last year!

Seriously is error highlighting the best that you can do? I've spent days, weeks, months (years??), tracking down impossible to find memory leaks and corruptions, race conditions, and bugs in complicated control flows. I spend a negligible time amount of time tracking down compile-time bugs.

Today IDEs either provide no UI making it difficult to browse and write code (ahem gdb), or there's so much UI that it waters down all of the features (ahem eclipse). Visual Studio is better than both, but I could still spend a week explaining why it sucks. None of them make it easy to browse code without clicking a half-dozen menus. None of them make it easy to track race conditions. Some of them help you with memory corruptions (valgrind) but are a bitch to setup. None of them really help you understand how your code works.... displaying the class hierarchy is a nice visual touch, but is pretty useless when trying to track down bugs.

I've got some ideas on how to make these debuggers much much better. Maybe later on I'll try to explain my new ideas techniques and formally analyze why each IDE sucks, but for now I'll just bitch.

Why do software tools still suck?

2007-02-21T18:08:00.000-08:00

Why is it that software tools have only inched along in the last 15 years? Fifteen years ago we were using similar debuggers and text editors. Fifteen years ago the software development cycle looks strikingly different to todays. So where is the innovation in software development?

Well there has been a good amount of innovation in software development, it just hasn't been in the tools. In the past fifteen years we've seen the rise of managed code like Java and Microsoft's CLR. We've seen software modeling tools like UML. We've also seen a rise of powerful scripting languages starting with VBScript and now Python, PHP, and the rest. More recently we've starting seeing static analysis tools like Coverity and Klocwork.

Of all of the technologies listed above, the only one which could accurately be described as a tool is static analysis (which I'll get to later). Everything else is a language or meta-language. So the real innovation in the past 15 years has been languages and the extra features that they support. The motivation for language innovation has been the travesty which is C/C++. Of the thousands of languages in existence C/C++ is probably the most hated [see here here here here ...] .

In the newer languages they've tried to build on all of the good things that were in C/C++ and strip out all of the crappy parts. Some advantages of the newer languages are easier memory management, syntactic sugar, and the supplied libraries.

Without a question the real advance in these languages are their libraries. Nearly every single new language that comes out also comes with a huge set of powerful libraries that make application development much much easier. Nuff said about that.

Complexity may be managed a bit better in the newer languages by pushing programmers to use more object oriented design. However, object oriented spaghetti code is often more unreadable then a flat design simply because OO programming gives the impression of modularity.

the newer memory management models are also not fool-proof. In many languages you can still corrupt memory in a similar way that you can in C/C++, however you get a programmer friendly exception instead of a crash. An end user doesn't know or care about the difference between an uncaught exception and a crash.

In some languages there are built-in types to handle multi-threaded constructs. The only real advance in multi-threading has been the use of moniters. I have yet to meet anyone who uses moniters in practice and they are really just syntactic sugar, not a real advance in programming. But most importantly when you have a multi-threaded application you still get the same race conditions and dead locks that you got in C/C++.

It isn't enough to focus on languages. If we really want to advance software development to the next level and create more reliabile and feature-packed software faster, then we need better tools. The main focus here should be handling complexity, error diagnosis (aka debugging), and multi-threading.

I'm going to stress multi-threading because if the semiconductor industry continues on its current track in a couple years we're not going to be able to write single-threaded applications. Processors will come with many cores and we're going to have to learn to take advantage of them whether we like it or not.

So do I have suggestions on how to get us out of this rut? Of course I do....

Its the tools stupid

2007-02-20T10:09:00.000-08:00

Think about how we were developing software fifteen years ago. For the mathematically handicapped thats 1992. At that time Visual C++ 1.0 was hot off the press and you most likely didn't use it. You were probably using ed, vi, emacs, or some other cult-following editor. All of the editors had syntax highlighting. The modern debuggers had features such as watchpoints, single stepping, and variable inspection. However you probably never used any of these features. You would be considered state-of-the-art if you used breakpoints. Back then the king of the debugging world was printf.

Now lets look at today. Compilers today are much better. They have fewer bugs and are much more standardized. The biggest breakthroughs in editors have been automatic code-completion, integrated documentation, and error highlighting. The biggest breakthroughs in debuggers have been conditional/scriptable breakpoints (which no one uses) and a little bit better GUI design, and more recently multi-core debugging. It is shocking, but from empirical evidence the most common debugging technique in the world is still printf!!

Why is it that the tools have only inched along? Thats coming up next, but I'll tell you now that our modern tools are still not good enough. The fact that printf is still so widely used is a key indicator that we haven't changed much at all in the past 15 years.... more on that later.

Why Specifying Software Never Happens

2007-02-19T17:10:00.000-08:00

In my last post I asked why is it so hard to specify a computer program before you start writing code? I mentioned how architects can draw blueprints for an entire building without laying a brick, and how scientists can design nuclear weapons without testing them. However computer programmers have huge difficulty grasping what needs to be done before they do it.

There are several reasons why programmers cannot specify their work early on in the process, however most of the reasons are hand-wavy and are part of the same old story we've been hearing for years. So here are the hand-wavy (but important) explanations first, then we'll get on to the real meat of the issue.

Poor Internal Communication
This is obvious but is something that consultants love to point out so they can charge $300 an hour. The main problem here is that if a developer/customer cannot properly communicate what he wants, then he'll get something else altogether.
Every organization has communication problems. Ideally we should wire all of our brains together so everyone knows what everyone else is thinking every instant. Until that happens we'll over pay consultants to tell us how to "refactor the enterprise". Besides, there is nothing specific to computer programming here.
Lack of precedent
Generally whenever a team creates a computer program it is piece of software that does something that no other piece of software has done before. If there was another program which did the same thing, then there would be little reason to reinvent something that already works. Everyone has heard of the not invented here syndrome, but in practice, things get reused if they meet quality standards, are not prohibitively expensive, and have the appropriate software license. Sure there are me too products out there, but most of those products were developed because they didn't meet one of the three above requirements.
Whenever anything is created without much precedent, it isn't easy to figure out how it should work. When making a building architects already understand all of the basic mechanics of what is physically possible. Scientists already understand the reactions necessary for a nuclear explosion. However if a computer programmer is writing a brand-new piece of code, there is no one telling him how it will work best. Of course as developers we always try to make use of precedent by reusing libraries and sharing code. However at some point you're going to be writing some code that no one has written before.
Complexity
This is everyone's favorite excuse for failing projects, but really its horse shit. It like a kid complaining "I can't do my homework cause its too hard." Software development is hard. Complexity is a bad excuse why projects fail not a reason. Complexity will provide a good segway into my next and final reason why systems cannot be specified earlier in the development process.

My so-called hand-wavy explanations don't explain the full problem. Communication issues can be solved with proper communication and management. Lack of precendent and complexity are complaints more than reasons that projects fail.

I believe that there is one major reason why software engineers do not and cannot specify their products. It is the same reason that the Waterfall Model is broken and also the reason why most software projects end in failure. The reason will be the main focus of this blog and I'll tell you now that it isn't agile programming (though I have nothing against agile)...

The Waterfall Method and Process

2007-02-19T12:37:00.000-08:00

Most computer programmers have had to deal with 'process' at some point or another. To some, process is an organized way to get abstract ideas into code. For others process is an unnecessary bureaucratic slowdown. Many see it as a way for managers to keep themselves busy and avoid real management. Regardless of what your general feelings towards process are, there is probably no process that is more nonsensical to computer programmers than the Waterfall Model.

The Waterfall Model was one of the first computer programming processes that really caught on. It dictates several stages of development. I'm not going to give a primer on the Waterfall Model, Wikipedia can do a better job than I. However the most important feature of the waterfall process, and the reason it is doomed to fail, is that you must know everything about what you want your computer program is to do before you start writing code. In nearly all applications of computer programming, it is impossible to know what a computer program can do that early on in development.

Development is a fluid and iterative process. You try something, it works well, and you expand on it. You try something, it doesn't work, and then you try something else. I have never worked on a project where we knew everything up front. There was always some experimentation and there was always some backtracking to get things working properly.

However we must ask the question, why cant we know everything upfront? Architects can generate all of the blue-prints needed to build a sky-scraper without laying a single brick. Today nuclear engineers can design nuclear warheads without even testing them. So what makes computer programming so different? Why is it that computer programmers are so special that we cannot predict what the outcome of our projects will be?