Black Hole Projects

August 1, 2011 4 Comments

OK, so I may have reset my blog, but there were some interesting posts that probably shouldn’t disappear totally down the memory hole. This is one of them, which I am rescuing from back in 2004 because of it’s continuing relevance. It seems that six months can’t go by without something I hear making me think of this. Edited from the original for clarity and to bring it up-to-date.

Many, many years ago, Steve Maine took the opportunity to reminisce about a project at Microsoft that was being worked on while he was an intern. He says:

[ When I was an intern… ] there was this mythical project codenamed “Netdocs”, and it was a black hole into which entire teams disappeared. I had several intern friends who got transferred to the Netdocs team and were never heard from again. Everyone knew that Netdocs was huge and that there were a ton of people working on it, but nobody had any idea what the project actually did.

I also knew a few people who invested quite a few years of their lives into “Netdocs” and it got me thinking about the phenomenon of “black hole projects” at Microsoft (and elsewhere, I’ll wager). There was one I was very close to very early in my career that I managed to avoid, many others that I just watched from afar, and one or two that I got dragged into despite my best intentions. I can’t really talk about most of them since most never saw the light of day, but it did get me thinking about the peculiarly immutable traits of a black hole project. They seem to be:

  • They must have absurdly grandiose goals. Something like “fundamentally reimagine the way that people work with computers.” Nobody, including the people who originate the goals, has a clear idea what the goals actually mean.
  • They must involve throwing out some large existing codebase and rewriting everything from scratch, “the right way, this time.”
  • They must have completely unrealistic deadlines. Often this is because they believe that they can rewrite the original codebase in much, much less time than it took to write that codebase in the first place.
  • They must have completely unrealistic beliefs about compatibility. Usually this takes the form of believing you can rewrite a huge codebase and preserve all of its little quirks without a massive amount of extra effort.
  • They are always “six months” from from major deadlines that never seem to arrive. Or, if they do arrive, another milestone is added on to the end of the project to compensate.
  • They must consume huge amounts of resources, sucking the lifeblood out of one or more established products that make significant amounts of money or have significant market share.
  • They must take over any group that does anything that relates to their absurdly broad goals, especially if that group is small, focused, has modest goals, and actually has a hope of shipping in a reasonable timeframe.
  • They must be prominently featured as demos in public settings such as company meetings, all-hands, conferences, etc. to the point where people groan “Oh, god, not another demo of this thing. When is it ever going to ship?”
  • They usually are prominently talked up publicly by high level executives for years before dying a quiet death.
  • They usually involve “componentizing” some monolithic application or system. This means that not only are you rewriting a huge amount of code, you’re also splitting it up across one or more teams that have to all seamlessly work together.
  • As a result of the previous point, they also usually involve absolutely massive integration problems as different teams try madly to get their components working with each other.
  • They usually involve rewriting the application or system on top of brand-new technology that has not been proven at a large scale yet. As such, they get to flush out all the scalability problems with the new technology.
  • They are usually led by one or more Captain Ahabs, madly pursuing the white whale with absolute conviction, while the deckhands stand around saying “Gee, that whale looks awfully big. I’m not sure we can really take him down.”
  • Finally, 90% of the time, they must fail and die a flaming death, taking down other products with it (or at least severely damaging them). If they do ship, they must have taken at least 4-5 years to ship and be at least 2 years overdue.

It’s kind of frightening how easy it is to come up with this list – it all kind of just poured out. Looking back over 19 years at Microsoft, I’m also kind of frightened at how many projects this describes. Including some projects that are ongoing at the moment…

You should also follow me on Twitter here.

Filed under Microsoft, Programming Tagged with , ,

Seven Rules for Beginning Programmers

May 16, 2011 59 Comments

A little while ago Phil Wadler posted “VS Naipaul’s Rules for Beginners,” listing the famous author’s seven rules for beginning writers. Upon reading them it occurred to me that, with a little adaptation, they could equally apply to beginning programmers. So, with apologies to Mr. Naipaul, here are my “Rules for Beginners:”

  1. Do not write long procedures. A procedure should not have more than ten or twelve lines.
  2. Each procedure should have a clear purpose. It should not overlap in purpose with the procedures that went before or come after. A good program is a series of clear, non-overlapping procedures.
  3. Do not use fancy language features. If you’re using something more than variable declarations, procedure calls, control flow statements and arithmetic operators, there is something wrong. The use of simple language features compels you to think about what you are writing. Even difficult algorithms can be broken down into simple language features.
  4. Never use language features whose meaning you are not sure of. If you break this rule you should look for other work.
  5. The beginner should avoid using copy and paste, except when copying code from one program they have written to a new one they are writing. Use as few files as possible.
  6. Avoid the abstract. Always go for the concrete. [Ed. note: This one applies unchanged.]
  7. Every day, for six months at least, practice programming in this way. Short statements; short, clear, concrete procedures. It may be awkward, but it’s training you in the use of a programming language. It may even be getting rid of the bad programming language habits you picked up at the university. You may go beyond these rules after you have thoroughly understood and mastered them.

You should also follow me on Twitter here.

Filed under Programming Tagged with ,

Why every language needs a language specification…

April 19, 2011 2 Comments

One of the things that I discovered when I started working on SQL Server is that T-SQL, like VB prior to .NET, has no language specification. This continues to mystify me—how language teams get away without having a language specification for so long. I should probably back up for a moment and explain what I mean when I say “language specification.” I mean a document that:

  • Is public.
  • Is kept up to date.
  • Describes the entire language (syntax, semantics, type system, etc.).
  • Is produced by the team that produces the language itself.

An initial spec that was allowed to lapse doesn’t count, for obvious reasons. Books written by people outside of the team/company don’t count because there’s no way someone who doesn’t live and breath the language every single day can possibly hope to shoot for completeness. Documents that just describe syntax or sort-of describe the language (SQL Server’s Books Online falls into this category) don’t count because often what you need is a holistic description of the language, and for that you need, well, the whole language described. And documents that aren’t public don’t count because what good is a document that no one reads? (Not that many people read language specifications anyway, unless the author’s name happens to be Hejlsberg or Gosling.)

I think the reason that language specifications often don’t get written is that they are a huge amount of work and an incredible pain in the ass. Writing the Visual Basic Language Specification was no mean feat and consumed a whole lot of time that I could have productively spent elsewhere. But I do think that every programming language needs one, for a variety of reasons:

  1. Languages are not algorithms. Although many programs of some sort or another could use a good specifications written down, the reality is that many don’t need one because the code is the specification. For example, I can imagine that there might be some specification written down about how the Excel recalc engine works, but it’s probably not that useful because, well, you could just go look at that code itself if you want to know how the recalc engine works. Most pieces of code can be more easily understood by tracing through the code than by reading a description of them. Languages, however, are a special case—they are an emergent property of the compiler, and although there are various algorithms that contribute to a language (say, the binding rules), much of the most important parts of the language arise from the interplay of the various algorithms. Thus, just having the code often is not really enough to understand what a language is or does.
  2. Specifications keep you honest. And honesty is important when it comes to programming languages. You can cheat like hell when it comes to user interfaces, and most of the time you can get away with it—extraneous menu items or options or notifications or other junk may clutter the UI up a bit but by and large you can deal with it, and you can always “clean up” a UI without too much trouble. You can cheat less with libraries and APIs, but even there you can always come up with a new version of the interface or library and gradually move people. With a language, however, it’s nearly impossible to fix something once it’s in the language. SQL Server’s own deprecation process takes three major releases, which means that a mistake you make in the language will take well over a decade to get rectified, if at all. Having a central language specification helps the team monitor what’s really going in the language and helps keep the team honest about what they’re doing.
  3. Having to explain things forces you to actually try and understand what your language does. I was continually amazed when writing the Visual Basic language specification how superficially I actually understood some features until I tried to write them down and explain them. Features that had been extensively discussed in design meetings and were already prototyped, even. In this way, it’s somewhat analogous to coding—how often I think I understand the solution to a problem until I sit down to write the code and realize how foolish I have been to think I really understood the problem!
  4. You need some kind of institutional memory. It’s all well and good to rely on the one guy who knows everything about everything in the language, but what happens when they retire/quit/move on/get hit by a bus? Now all you’re left with is a mass of language rules and often no idea why things were done one particular way or another. This is not an uncommon problem in general with programming, but it’s even more acute with language design. When I was a young, naïve language designer, there were a number of times when I looked at existing languages and the choices they made and thought to myself, “Boy, is that a dumb design decision.” Then, a couple of years down the road, having ignored some of the wisdom of those who came before me and having run into a brick wall at full speed, I would think, “Oh, I see, that’s why they did that…”

Anyway, I’m not planning on just complaining about this and not doing anything about it. Stay tuned…

You should also follow me on Twitter here.

Filed under Language Design, TSQL Tagged with , ,

“I Didn’t Quit.”

April 4, 2011 2 Comments

I ran across this great blog entry today called “What’s The Most Difficult CEO Skill? Managing Your Own Psychology”. I don’t expect to ever, ever, ever be a CEO, but I think a lot of the lessons that Ben Horowitz talks about apply to just about any leadership position you might ever find yourself in, from team lead to Little League Coach. Thing like needing to learn to handle stress, needing to take the responsibility seriously but not too seriously, needing to cope with loneliness, etc. But the thing that caught my eye was the end of the entry:

Whenever I meet a successful CEO, I ask them how they did it. Mediocre CEOs point to their brilliant strategic moves or their intuitive business sense or a variety of other self-congratulatory explanations. The great CEOs tend to be remarkably consistent in their answers. They all say: “I didn’t quit.”

In my experience, this one phrase—“I didn’t quit.”—is the dividing line between success and failure. A few of the successes that I’ve had in my life (such as they are) can be attributed to dumb luck or raw talent. But most, and certainly the most satisfying ones, were primarily attributable to the fact that I didn’t give up even though I often might have wanted to.

The hard part is that it’s often impossible to know whether you’re on the right path or not. Sure things may turn out to be dead ends. Things that people say cannot be done may turn out to be golden opportunities. The best thing you can do is be clear on what’s important to you, follow your gut, and… don’t quit.

You should also follow me on Twitter here.

Filed under Philosophy Tagged with ,

External Requests Versus Internal Requirements

March 29, 2011 1 Comment

A commenter observed that a side-effect of my blog reset is that now any answers on Stack Overflow that pointed to my blog are broken. This is an unfortunate situation that I didn’t consider when I decided to reset my blog, and it did give me some pangs of regret when I looked at the list of answers that referred to me. It also got me thinking about the inevitable conflict in life between external requests (i.e. things other people want from me) and internal requirements (i.e. things that I want for myself).

From the Internet’s perspective, it would be happy if every piece of information that ever appears on the web would: a) stay on the web forever, and b) stay in exactly the same place forever. This makes total sense from the collective perspective—I’ve been amazed by the number of times that I’ve gone to research some obscure thing (such as how to unbind a DOS executable from a DOS 4/GW extender… don’t ask me why) and found some answer from back in the dark ages (like, 1996). Information disappearing, from a public utility standpoint, is a Very Bad Thing because you never know how useful that information might be some day. In college I was always amazed at the variety of interesting historical information that could be deduced from everyday things like private letters, diaries, commercial correspondence, etc. The true enemy of the scholar indeed is people who throw away… well, anything, really.

But from my perspective, it’s a lot of dead weight. I mean, a lot of the stuff I dumped off my blog was written five or six years ago. Some of it was wrong and a lot of it was irrelevant when looked at through the lens of later events. Some of it reminded me of things that I’d rather forget all about. And some of it was, frankly, embarrassing. One could say, “Well, then, just don’t look at it!” And, most of the time, I didn’t. But, you know, it was still there, taunting me from the archive list on the right hand side of the blog.

Some changes in my personal life recently motivated me to go through a bunch of boxes that had been sitting in storage for years. They were mostly full of stuff from my high school and college years—old papers, letters, other random stuff. I’d been holding on to a lot of it because, well, it was my stuff. But in looking through it, I realized how much of a burden most it had become. Like the blog, I didn’t look at the boxes very often, but there they always were, taking up space, having to be moved around, making me keep track of them. So I decided to go through and throw away anything that didn’t have a strong, tangible, positive, personal meaning to me. I’d say I threw away about 90% of what I had stored in those boxes, tons of stuff I no longer even remembered anything about. And the wonderful thing? Once I was over the initial trauma, I felt a lot better, freer, and lighter. It’s amazing what wonders getting rid of old stuff can do for you, even if it does make my theoretical future biographer’s job harder.

So my apologies to the Internet: I realize that losing my miniscule contributions to global knowledge might make life a little more difficult, and I’m sorry about that. But I have to say: I feel a whole lot better letting go of that stuff. I’m sure it’s going to cost me some (or even a lot) of visitors, but it seems like a small price to pay. At least, for me.

You should also follow me on Twitter here.

Filed under Meta Tagged with ,

Landed…

March 25, 2011 3 Comments

As I warned last week, I’ve moved my blog entirely over to WordPress now. Please let me know if you encounter any difficulties.

Filed under Meta Tagged with ,

Hitting the Big Red Switch

March 18, 2011 5 Comments

As you’ve probably noticed (and I’ve commented on from time to time) my blogging output over the past several years has fallen off precipitously, to the point where there has basically been nothing written on the blog at all in the past six months. I spent a while debating whether it was just time to go ahead and shut Panopticon Central down but in the end I decided that it’s likely that I’m going to have a few more things to say as I start to really settle in to the SQL Server team.

One thing I did decide I wanted to do was get out of the blog hosting business entirely. Those of you who have been exceptionally loyal readers may remember that this blog started off with a hand-built engine running on a server in my basement and has steadily moved in the direction of requiring less and less of my time to administer it and keep it running: first to using standard software (.TEXT and then SubText), then to a hosting provider (Server Intellect). To be honest, though, even administering a standard blog package on a hosting provider is more than I want to be doing. So after a lot of thinking I’ve decided to move wholesale to WordPress in the hopes that I can forever be done with such fun and exciting tasks as upgrading my blogging engine database.

That leaves the question, though, as to what to do with the large amount of previous stuff on the blog. Reading through the archives, most posts seemed to fall into the following categories, in order of frequency:

  1. Inside baseball discussions of the design/shipping process for previous versions of Visual Basic.
  2. Highly technical discussions of design decisions for features shipped in previous versions of Visual Basic.
  3. Random moment-in-time personal commentary.
  4. Highly confused (and often very inaccurate) discussions of now-cancelled or re-directed projects.

After some serious consideration, I’ve decided that rather than trying to bring all the past posts and comments over to WordPress, I’m going to wipe the slate clean and start over with a fresh, clean blog. It was surprisingly difficult to decide to toss out eight years of accumulated blogging detritus, but as I started to look through some of my old posts, I was convinced it was time. There just wasn’t much there that seemed particularly important to save for posterity. OK, maybe posts that fall into category #2 above might provide some value for people, but even a lot of that information is quietly starting to rot or become irrelevant. And, to be honest, I’d rather just expunge some of the things I talked about regarding future plans for Visual Basic (most of which haven’t turned out to be true) or Oslo/"M" (ditto), to the degree that’s possible in today’s Internet.

If you’re seeing this in a RSS reader, then you’ve probably got a narrow window of opportunity to grab what you want off the old blog, as the first step of the move is going to be redirecting my RSS feed to my WordPress blog but not moving anything else. After that’s been given a little while to settle, I’ll move the entire domain over and the old blog will be decommissioned. I will dump all of the old blog into a database, so if you find yourself missing some pearl of my wisdom, let me know and I can probably dig it up for you. There might also be some select gems that I might republish, but we’ll just have to see.

So thanks for still being a reader, and look for newer and fresher content soon! And you should also follow me on Twitter here.

[Ed note—in case you’re too young to remember it, the “big red switch” was the power switch on the back of the original IBM PC. With only floppy drives, turning off the power meant resetting the machine entirely and starting fresh.]

Filed under Meta Tagged with ,

Follow

Get every new post delivered to your Inbox.