‘gress

Sucky day. Three crashed dev systems (you know it’s trouble when you have to use another computer as a power supply for the one that got screwed up so badly that it wouldn’t take an OS), a bunch of blue-screens, and about a day’s additional work for every hour that went by.

Then, around 4:00 in the afternoon, the clouds parted, the heavens rolled back, angelic beings buzzed by the window (okay, hyperbole, I don’t actually rate a window office), and the thing I’d been working on for a week and a half came undeniably to life. Well, it crashed when something didn’t clean up on termination, but before that it ran solidly for like thirty minutes and that’s better than being a zombie box that an hour earlier had been on +5V life support from a comrade.

And you wonder why engineers are odd ducks. And why soldering — and knowing the difference between positive and neh, nugga … um, the other one — is still a valuable skill for software people to have.

Some days are not progress, nor are they regress.  I’d have to call them “‘gress.”

Productivity

Interesting musing on programmer productivity:

Some will quickly point out that programming quality, like interest payments, compounds over time, and small differences in day-to-day productivity may turn into order-of-magnitude differences in delivery time…

link

One thing I’ve observed about “superstars” is the near-inhuman attention to detail that these people have.  I mean, analogous to compilers noticing that you forgot a semicolon, but at a higher level (e.g., noticing that a source-code rearrangement would help storage and performance issues, or getting all the doc-comments right, all the time, or alphabetizing variable declarations).

 

Hardware Engineers are from Planet Gonzo

Hardware engineers are . . . different.

Okay, I’ve done my share of soldering stuff together, then applying power in a burst of enthusiasm and seeing expensive smoke go curling up because I’d been a little hasty. (Best show: A poor little 555 timer that had been socketed backwards; it gave a “pop!” as it blew the top of its plastic case off, which shot across the room and bonked my office mate in the back). Frankly, the out-of-pocket expense of my hardware mistakes are what drove me to prefer writing software over screwing chips together. Also, if you think about it, you can do a lot more damage with software.

But a lot of my interactions with hardware engineers follow the three steps: (A) agreeing on a set of registers and functionality, (B) waiting around for the hardware (occasionally checking in to see that what we’d agreed on hasn’t changed), and (C) finally getting the stupid thing and having to figure out what the heck it that Joe Hardware Guy actually designed.

“Uh, Joe? This thing doesn’t have any interrupts.”

“Well, I thought you could poll it for completion.”

“No, we were all agreed on interrupt mask registers, resets and enables, all that stuff, right?”

“Well, it’s all finished, so I’ll have to go tell my boss you need interrupts now and the schedule is going to slip and turning the chip again is going to cost…”

And now you know the real reason that hardware guys all hang out together in a lab, surrounded by hot soldering irons, heavy desks, and many, many sharp tools, it’s because when the software guys scream and leap, the hardware team can mount a defense.

“Christ, it’s the OS guys again. They must have found out what the memory system latency really was.  Joe, see if you can make it to the parts drawers; I’ll cover you.”

“Right!”

“Fibgy! Get that smoke generator working again!”

“Yessir!”

“Everybody, I want your scopes on Auto, and prepare to fix high impedance probes. Now MOVE!”

I’ve only seen one hardware lab that didn’t have a back way out. The bastards, the poor, unlucky bastards had to raise the white flag and give us an error status register after all.

Phew

Phew. I finally got some time and polished off a couple things that were on the burner (sorry about the delay, it took about a month for the old Atari ST stuff to even come close to being readable. I hate pressure to write).

Posts should be smaller and more bite-sized for a while.  Again: Phew.

William Gibson A-B-C

A is for AI, lurking in cyberspace
B is for Biz, the kind that’s best face-to-face

C for the Clinic Tanks, where non-payers go
D is for Dermatrodes, Deals and Dough

E for the Enzymes a-slosh in your head
F is for Flatliner, not really dead

G is for Gaijin, no way to blend in
H the Hosaka you take for a spin

I for Immortal (well, probably not)
J is for Johnny, who’s still very hot

K is for Ketamine, Kirin and Kink
L is for LED, Laser and Link

M for a microchip stuck in your bum
N is a Non-lethal (technically) gun

O stands for Override, which we do a lot
P is for payment, that we haven’t got

Q is for Quickly, Quiet and Quine
R is for Razor-girl, Radar and Rhine

S for Sabotage we do for a hobby
T for the TNT down in the lobby

U, the Ultrascan we get when we’re able
V for the Vat-grown stuff on our table

W for Worms, not the kind in the dirt
X is in Anxious, which keeps us alert

Y, the Yakuza would have us believe, is just
Z, a Zaibatsu, and no den of thick thieves

(Future titles probably include: “The Hack in the Cat,” and “Oh! The Places You’ll Be Routed!”)

The Atari ST, Part 2

My memory of the ST project is scattered and biased, but here’s how I remember it, starting in July of 1984.  There are a lot of details missing; for the most part I remember the bits that I worked on, but the other folks on the project almost certainly remember different or conflicting stuff.  Forgive my omissions in advance; it’s been a while.

July: Tramiels buy Atari and consolidate the people that they don’t lay off into a single building. The “ST” plan is bandied about, but nobody knows a whole lot.

August: The ST hardware becomes clearer. We evaluate other OSes, etc.

September: Work starts in Monterey, near the Digital Research campus.

October: Work. We get (rented) houses in Monterey.

November: More work. We barely see those houses.

December: Much more work. The ST boots TOS for the first time.

January: CES (with STs running CP/M-68K). Decision made to move to new file system (GEMDOS).

February: 16K boot ROMs written (a couple-week side effort).

March: Even more work. Two weeks to crunch TOS to fit into 192K.

April: ROMs actually work (do you know how long it takes to burn 192K of ROM, not to mention UV-erasing older chips?)
May: ROM TOS 1.0 shipped.  Phew!

Well, and there are some details.

– – – –

The other engineer was screaming at me: “If you’ll be patient with me, I’ll be patient with you!”

He stomped away to his office. Not his real office, which he would have been delighted to stomp back to. His real office was in Digital Research nirvana, in the buildings that we were forbidden to visit. Instead he had to be happy with stomping off to his crappy shared office in the building where he was temporarily relocated, where he had to work with . . . us. The pushy Atari guys. The hicks from Silly Valley who just wanted stuff to work.

We’d been in the satellite building next to the DRI campus for several months. Early prototype hardware was still maybe a month away (though we didn’t know that then). The pressure to get something working was intense, and the truth about what we had purchased from DRI was becoming clear: The software wasn’t technically sweet, and getting it running on the Atari hardware wasn’t a matter of just doing a port because large parts of GEM simply weren’t finished.

Theoretically you can write very portable code in C. As long as you stick to certain rules and have a reasonable degree of paranoia you have a good chance of your code running on different platforms, with minimal effort. The best way to do this is to have your code running, from day one, on several different machines using several different compilers. DRI had not done this with everything, and the resulting issues ranged from simple errors that were caught by the compiler and were easily fixed, to deeper issues that required design work if the code was going to run on anything other than an 8086. It was slow, frustrating, and there was a lot of friction. And some yelling.

That’s a dismal picture of the DRI / Atari relationship. Very often, things worked great. Some of the DRI guys were hilarious and fun to work with. But we all had our faults. If the worst of the DRI engineers thought they were programming gods who could code no wrong, then the worst of the Atari engineers were pushy bastards who just wanted stuff to work. Sometimes things clicked, but sometimes we were like scorpions in a bottle.

I forget what the “patience” argument was about. The other engineer was right, or I was right, or maybe we were both horribly wrong. An hour later we apologized to each other and sat down and just got the job done. But a couple of years later, long after we had returned to Atari’s Sunnyvale office, we still remembered that argument in lunchtime banter.

– – – –

The Atari side of the ST software team roughly broke down into six small groups:

Graphics. Two or three guys took the DRI-specified graphics layer and wrote font renderers, blits, line-drawing and other primitives. In my opinion the graphics guys were having the most fun, and since they were video game programmers (well, technically speaking, ex video game programmers now) they were running architectural rings around the rather pedestrian graphics abstractions that DRI had decided on. The graphics primitives on the ST had interesting extensions, and GEM only used a subset of what was available.

Porting Getting GEM working. Two or three more of our people were helping get GEM onto the 68000. This wasn’t just “compile, debug, rinse, repeat” deal, since GEM wasn’t really finished. These guys worked really closely with DRI engineers every day, and they were probably the most frustrated of us all.

BIOS (drivers) and OS (two guys, including me). Straight-forward systems bringup stuff. Lots of work, but mostly ho-hum.

Infrastructure: Build wrangling, source management, odd-and-ends, and pithy observations about the nature of humanity as it pertained to its use of computers, and in particular, nasty habits in source code. We didn’t use source control. I’m not sure we even used diff. Mostly we had several directories-full of source files that were compiled by a guy who knew how to do it. Rustic, but it worked.

Applications. Well, application, we had a guy working on porting the DRI Basic. It was pretty much a disaster, though the work eventually did get done. I have vague memories of an engineer hired by the Tramiels who didn’t do a very good job — I think he ran into a bunch of portability minefields, got discouraged and beat up on by management (we hadn’t truly grokked the unpolished and unported state of a lot of the DR software yet). He wound up quitting or being fired, and I can’t remember which.

Morale (in the form of a very friendly german shepherd doggie, who was capable of playing frisbee far beyond human endurance; very useful for destressing).

– – – –

Before the ST hardware started to work, we had to use existing 68000-based systems for cross development. The graphics guys had Apple Lisas that were running CP/M-68K; the Lisas had nice bitmap displays which we used as “practice” STs. The disks on these machines took forever to come back after a crash (tens of minutes). For some reason the boot code on these machines had been written to display a bitmap of a fish. You’d hear a mutter or curse from down the hall (crash), then the creaky footsteps of someone walking around, cooling their heels and waiting for their “God damned” Lisa Profile drives to boot, then a triumphant yell “CarpDOS!” and typing sounds.

The BIOS/OS guys had some Motorola VME-10 workstations that were (ahem) “Unix Ready!” (the boxes they came in said so, in large, proud letters) but instead we had them running CP/M-68K, and I’m sure they felt sad inside; I know I did. The VME-10 systems were very flaky; my own system died and needed repairs three times in six months. (A year and a half later, Gary Tramiel, the son who was heading the financial arm of Atari, asked us if we were still using the VME systems. By then we had moved all our development over to the ST itself, and the VMEs were gathering dust in a corner. “Hell no,” I said. “Fine,” said Gary, “Then we won’t pay the repair bill.” A good lesson in the Tramiel school of start-up economics).

– – – –

DRI had us housed in an old TV studio building (KMST, if you care) that was about a hundred yards from the rest of their buildings. The building was cold and creaky, and when it rained (which, during the fall and winter we were there, was a lot) the steps got pretty slippery.

Typical workday: Get up and do the usual stuff (coffee!) in our rented house in Carmel, two blocks from the beach. Usually we could hear the ocean, and sometimes I’d get quick beach fix in, if it wasn’t raining. Drive five miles to DRI, go up the creaky steps without breaking my neck. Make awful instututional-style coffee from the horrid little mylar bags of Columbian bridge-sweepings (put two in, just to make sure the coffee isn’t totally crappy).

Go into the office. Huh, the VME/10 won’t boot again. Flip power switch on and off for a while until it finally works (leave it on the rest of the day).

Make a backup right now because (a) it’s a nice, warm fuzzy feeling, especially when you can’t trust your stupid workstation to even fucking turn on, and (b) was the one I did last night at 1am really any good?

Flail about at drivers. Trace through file system code that mostly works, but sometimes doesn’t. Wish for working hardware. Try to decode the latest spec from the hardware guys. Stare at the ceiling (“That doesn’t make any sense.”) Stare at the wall (“That can’t possibly work.”) Write some more code anyway.

– – – –

There was a bug that had been causing all kinds of grief; some kind of simple botch. I’d spent half of the previous day working out exactly what was going on, and it turned out to be in some DRI code. I groaned. Not that guy again.

The DRI engineer responsible for that part of the system was notoriously arrogant. I tried to explain the problem to him, down to the offending line of code, and he was objecting all the way. But later I overheard him saying to his office mate, “Hey, I found a bug in my code that could explain that weird problem.”

This just drove us batshit.

I took a doggie break. One of the Atari engineers had a wonderful german shepard named Divot. You could take Divot outside with a frisbee and she’d play fetch until one of you dropped from exhaustion (and she could fetch for hours). It’s hard to get worked up about a screwed-up OS when someone is utterly dependent on you for the next frisbee toss.

“He’s a bozo, Divot.”

“Woof!”

“So what if he came from HugeCorp and did systems programming on machines so big he couldn’t lift them; he’s a graduate of the Arrogant Jerk Academy and he doesn’t know how to interact with humans.”

“Woof, woof!” [Speaking of interacting, throw the stupid thing already, okay?]

I went back inside. Whereupon: Much more programming, a late-night run for chinese food of dubious quality, and work, work, work. One big happy family, hatching an operating system out of thin air and ego and fear. Oh yeah.

– – – –

CP/M-68K was an “Operating System” that had its roots in the 70s. About ten years earlier Gary Kildall had worked on some DEC PDP-11 systems, liked them, and had been inspired to write a small OS for the very early 8080-based microcomputers. For years CP/M had been a defacto standard. Gary had started a company called Inter-Galactic Digital Research to further develop and market it. MSDOS had only been out for a couple of years, and DRI (renamed — sensibly losing the Intergalatic bit so that people, especially conversative suit-types, would take them more seriously) was vying for market share with a port of CP/M to the 8086, the CPU of the IBM-PC.

CP/M-68K was a port the 68000, and was the OS that the Tramiels had contracted for.

CP/M (in any of its variants) didn’t really do a whole lot. There was a simple flat file system. There was some character-at-a-time console output (useless on a computer with a graphical interface). And CP/M could load and programs. That was about it. (By modern standards it was missing: A heirarchical file system with directories, networking, memory management, processes and process scheduling, a notion of time, synchronization and locking primitives, a driver architecture, graphics, fonts, character sets . . . you get the idea).

GEM was was bolted on top of this primitive base. Since the underlying OS didn’t support more than one task, GEM had a lot of its own stuff to enable things like “desk accessories” that could run concurrently with (say) a word processor. It was pretty clunky.

None of us liked CP/M-68K. So when we heard that someone at DRI had been doing something much better, even though it was still unfinished, we unofficially jumped at it. GEMDOS started as a skunkworks project by a DRI engineer who had a reputation for being a loose canon. GEMDOS had a heirarchical file system that was compatible with MSDOS; it had a few other improvements, but this was the biggie. But in December 1984 GEMDOS was still being written.

The STs that went to the CES show were running CP/M-68K. In late January, after a bunch of hand-wringing, Leonard Tramiel made the decision to go with GEMDOS. We’d had it substantially working for several weeks, and it looked like it was going to be fine. Notably we did not have any hard disks to try it out on, so all of our testing was done on floppy disk based systems — this would come back to hit us hard later.

– – – –

It was pretty clear that TOS was going to be late. But we had the boot code working fine, so we spent a few weeks doing a small 16K loader ROM. All it did was paint some pretty graphics, load a sector from floppy disk and run it. We sent the boot ROM images out without actually knowning if they’d boot an OS, but they worked fine.

Around the time the boot ROMs were sent off, the software team was feeling pretty blue. Things were taking much longer than we had expected; there were lots of bugs to fix, there were missing features, there were features that would never make it into the product, and it was pretty clear that the Mac had us outclassed. Also, most of us were feeling pretty burned-out.

Jack Tramiel called a meeting. We didn’t often meet with him, and it was a big deal. He started by saying, “I hear you are unhappy.” Think of a deep, authoritarian voice, a lot like Darth Vader, and the same attitude, pretty much.
Sorry, Jack, things aren’t going all that hot. We tried to look humble, but we probably just came across as tired.
“I don’t understand why you are unhappy,” he rumbled. “You should be very happy; I am paying your salary. I am the one who is unhappy. The software is late. Why is it so late?”

Young and idealistic, I piped up: “You know, I don’t think we’re in this for the money. I think we just want to ship the best computer we can –”

Jack shut me down. “Then you won’t mind if I cut your salary in half?”

I got the message. He didn’t even have to use the Force.
– – – –

We got busy again and shipped the first ROM-based systems a month or two later. My memory of this has really faded, but a few things stick:

TOS wasn’t going to fit even in the 192K of ROM. It was well over 200K (210? 220?) and still climbing. So for two weeks everyone dropped what they were doing and started removing code. It’s amazing how much stuff you can toss out if you really try. Our linker didn’t do dead-code stripping, but even if it had that wouldn’t have shown us the fat pieces of common code, the pathetic reimplementations of strlen and strcpy that were everywhere, and the useless crap and horrible layering that could be replaced with a few simple lines of code.

[I’ve since found that removing code is a great way to improve an existing system; not only do you get rid of a lot of bugs, but the result is usually easier to understand, and often runs faster. Have a large, unwieldy project that takes forever to build and you have trouble making changes to? Wade in and start deleting. Become a ruthless of constructive destruction; if you accidentally nuke something critical, just resurrect it from the project depot. Software is great!]

A little while after the first TOS ROMs shipped, Leonard Tramiel arranged a celebratory dinner for the engineers and managed to get Jack to come as well. About halfway through the meal (which was at a wonderful Chinese place called Fung Lum, in Campbell), Leonard started relating the story of how he and John Feagans had arrived at the Atari Coin-op building to interview people and see who they wanted to keep.

“Then this voice called out over the intercom –”

— oh, shit. One thing you need to know about Jack is that when he was twelve years old, he was in the concentration camp at Auschwitz. I’ve seen the tattoo. That he survived being there pretty much defined him, as far as most people were concerned. And —

“– and the voice called out, ‘Imperial storm-troopers have entered the base!'”.

Jack hadn’t seen Star Wars, not ever, and didn’t get the reference. And to him, the phrase “Storm Trooper” has a completely different meaning. It took a little while for Leonard to convince Jack that it was really a funny thing, no, honestly, really it was a joke, okay? And I’m not sure that Jack really understood. But in the end he gave a little laugh; everyone else seemed to enjoy the story.
I kept my job.

I hung on for another couple of years before going to Apple. There were some nasty bugs in GEMDOS that were never really fixed (you can download the sources — I did, a number of years ago, and found the same set of unsatisfactory fixes that I’d come up with, but that I’m not sure ever shipped). I took an ST with me, but I didn’t ever do much with it. I don’t keep in touch much with the people on the ST team; some light email, but that’s about it.

The ST community did really awesome things; some actual decent multi-tasking operating systems, a ton of music-related software and so on. It’s neat having had a part in helping all of that happen. I also know what I’d like to be able to do a second time around on a project like the ST. I’ve got this little list . . . .

Thoughts while rebooting

I’m using my business laptop, on a plane, on battery. I passed through security with flying colors, remembering to put my pocket change and tooth fillings in a separate ziplock bag before submitting to the metal detector and the steely-eyed stares bored cud-chewing of the TSA goons, but what didn’t show up on their X-ray machines was that my laptop is possessed by demons from the depths of Hell. I look out the window at corn and soybean fields seven miles below. There’s nothing else to do. I can’t even put the damned laptop to sleep now.

You see, I’ve just made the mistake of installing a small piece of software, written by a company with its heels dug well into the 1990s, the bad times when everything you did to a Windows machine involved a reboot. As near as I can tell, this fine package (a C# parser generator whose name I will not disclose) dropped some help files, some sample code and a simple executable into the Program Files folder, then had a panic attack and prompted with a “Reboot now?” dialog and a helpful countdown timer. I was a dozen keystrokes ahead and the installer saw me type [Enter], and quicker than snot out of a toddler with a week-old head cold the laptop was shutting down and I was screaming inside.

I don’t know when booting a computer became an event of epic proportions. I’ve helped write several OSes that were interactive and doing honest work in the amount of time it takes a present day BIOS to figure out that there’s a disk available to boot from. Once today’s systems are past the initial power-on hazing ritual, booting the OS is like reenacting the construction of the pyramids. Civilizations rise and fall as legions of threads and helper processes move massive heaps of code into place. But like all construction work I’ve seen, most of the workers involved aren’t doing a damned useful thing. They stand around jawing to each other, and if the boss shows up they start aimlessly moving things around, looking busy. The foreman: “Don’ worry, we’ll boot dis thing and still leave ya wid 20% of the battery. That’ll be plenty for ya. No? You maybe want it to be 15%? Guido, getta load of dis guy! He likes his battery. Haw haw haw.”

Finally I’m able to log in, and things do not look happy. The CPU is still pegged, the laptop’s fans are running full tilt, and the hard disk sounds like a set of maracas tied to an amphetamine-popping epileptic. The stewardess stops by with a sandwichette and offers dilute coffee. “No thanks,” I say. I don’t add: I’m battling the forces of outer darkness. I’m not stupid. Instead I say, “Can you please tell the Captain that we have a situation here in 27-C?”

Instantly she looks professional. “What, a laptop? Condition seven?”

“Yes. And getting worse.” Indeed, the laptop is becoming alarmingly warm.

“Yes sir!” She hustles off to the cockpit, and a minute later I’m escorted by one of the flight crew to the spare seat in First Class, the one right next to the priest who is always available on business-class flights. He’s that anonymous age that men of the cloth have when they’re halfway between looking too young to have any experience and too old to understand what a boot sector is.

He closes the curtain. “What do you have here, my son?”

“Out of control WBEM scanning and virus detection, some SNMP oids that have gone recursive, and the usual IT-installed crapware.”

“Ahhh. Just a moment.” He sends a high-voltage prayer further up into the stratosphere. It is apparently answered immediately, because the laptop is surrounded by a momentary blue glow and its hard disk stops chattering. “An easy one, son.”

“Thank you, Father.”

“Not like last week. Some benighted Ubuntu user tried to invoke M-X ghost-of-stallman on approach to SFO.”

“But, wait a minute, Stallman isn’t dead.”

“No, but every Emacs session for fifty miles thought he was, and went into mourning. There’s a reason we don’t let people use customizable interfaces under 10,000 feet. The only thing that can handle that,” he smiles and shakes a tiny vial on a chain around his neck. “Water from Lourdes. Do you need anything else?” He gets out his card reader. “The church has a special on access violations this week.”

“No, thank you again, Father. Is Corporate Amex okay?”

“Fine, son.”

– – – –

Back in my seat and in a comfortable editor session, I reflect on my past years’ experience with Vista. I run it on several different machines, and mostly what I’ve found is that the ones that are heavy with IT crapware are pretty miserable, but the ones that I’ve set up without work pretty well. On the IT-ridden machines I regularly have to swab out twenty megabyte log files, logs from things that I didn’t even know were running on the machine, and when I find something like “ArScnr38” running I have no idea if it’s spyware or something that an IT monkey stuck on my laptop to scan my Excel spreadsheets.

It’s hilarious when four different scanners are fighting for disk access. No wonder our drives are dying after like a year in service. I don’t work late, so I can only imagine what the buildings sound like at 3AM when Windows Update goes into its happy dance and reboots every single workstation.

“Shhhh… wait for it.”

“What, Dad? I’m sleepy.”

“Any second now…”

clikclickClickClikCLIKCLICKCCLLIICCKK-CLICK-***KA-CHUNGGGKGKGKG!!!!!***

“Wow! Do the lights flicker like that in every time zone?”

“That was nothing. Wait until they all ask the DHCP servers for an address!”

As I was saying, at home, the worst I’ve have to deal with is the crapware from minor league players, like the miserable bloatware that Creative and Nero spam to the desktop, or the unasked-for media players that assign themselves lord, master and gatekeeper of my music collection (how many of these damned things do we need, anyway?).

The IT philosophy of bloat appears to be: “Screw the user, we own the machines, and if they can’t get work done with them then they can’t do any damage. More scanners! And loggers! And Java-based enterprisey things with fucked-up XML configuration schemas! If there’s CPU or disk space left we’re not doing our jobs; we have to pay for that call center expansion somehow!”

The consumer-ware philosophy of bloat is equally uncaring and fear-based, but with a twist:

“Figby, we need to ship a media center with our mouse driver.”

“Yes. Oh yes. You’re the marketing guy. You’re my hero. You are the font of wisdom, the center of –”

“Yes, I am. Now, it has to look like the media thingy we did last month, except that we have reports of users being able to recognize the controls behind all the chrome and widgets, people were actually able to find their music and play it, and the MPAA called and said they didn’t like that, so we’ve brought in a new user interface design consultant to help us out.”

“Oh, yes, that’s simply great!”

“Mongo, this is Figby. Figby will show you where the keyboard is. No, no, Mongo, that’s the radiator –”

“Mongo like type.”

“He’s all yours, Figby. Done by tomorrow, right? By the way, Mongo is trained not to let anyone leave until the bug database shows zero unresolved bugs.”

“Mongo WANT *TYPE*!”

And my battery is completely done for.