In the "multimedia gulch" of San Francisco, Internet businesses have sprung to life thriving on a dense grid of fiber-optics, cheap capital, and smart high-tech workers. I recently toured one of the "server farms" located there, one of the data pumps that drive the net. I found a surreal world.
Located in an oddly transforming neighborhood where scattered homeless lay about the dilapidated warehouses is an architecturally unimpressive but gleaming techno-office space where packs of nerds move in and out while pouring down Starbucks, talking about software, and being generally oblivious to the blight around them. The business I was visiting, whose identity shall remain nameless (we'll call it My-B2B-E-Portal-Server-Farm.com) is a place where the virtual bits of the Internet manifest themselves as real structures and is, as such, one of the few places where one can "see" the net. Being the pathetic technology voyeur that I am, my friend who worked there knew I would be excited about the tour, and I was!
As I left my vehicle and imagined the broken window and stereo-less hole that I would likely find upon my return, I noticed a man sifting through the trash. He was wearing a black t-shirt emblazoned with the logo of the very company I was visiting. In fact, to be more exact, he was wearing six of them and holding a dozen more as he had obviously found a discarded box of sad techno-marketing tees, their usefulness as swag either over-estimated, faded, or, most likely, wastefully rounded up to the nearest 100. The man headed off, seemingly happy with his new nerd wardrobe.
In front of the building my eye was drawn to a set of tall transparent cylinders, something like gigantic drinking glasses 12 ft tall and 10 feet in diameter guarding the building. Large and odd enough to pass as corporate modern art, they stood empty and soulless, somehow lacking any artistic merit whatsoever. Upon closer inspection, the floor of each cylinder was actually a metal grate as if they were not colossal drinking glasses, but rather gigantic coffee filters. The grates appeared to cover ventilation shafts, although the Plexiglas prevented detailed inspection. Oddly, each cylinder was filled with trash, mostly of the flying-though-the-breeze type such as newsprint, Styrofoam cups, and french-fry containers.
"How odd this is", I thought to myself as a man walked up and asked for change.
"Not now," thinks I, "can't you see, I'm contemplating modern art!" I stared, shifted vantage places, and stared again. The transparent monoliths remained completely opaque in their purpose. Another man pushed two grocery carts filled with possessions and built a tent between them. I continued to stare, squint, and blink at the mystery.
At last to my dull wit their intent became clear; the goal of these garbage-filled Plexiglas rings was to prevent the, shall we say, differently-housed from sleeping over the warm ventilation shafts.
"How heartless," I mused to myself as I heartlessly ignored another solicitation for change. Heartless, yes, but a thermodynamic imperative no doubt. If the vents were impeded by the paper and plastic insulating material of the address-impaired, their efficiency at dispersing heat from the hot-air-generating businesses within would be, well, challenged. I further supposed that the garbage filling the tubes was an unintended side-effect - a bug in the design (albeit, an incredibly predictable one.) As there were no doors to provide access into the cylinders, one supposes that a ladder-enabled janitor is paid to climb over and periodically sweep. Thousands had been spent on the installation, thousands were being spent on the upkeep, thousands of BTUs were venting to the atmosphere. It all seemed like a failure of basic microeconomics - demand and excess supply for heat in the very same place and yet somehow it is more politically efficient to keep the homeless cold. Why there couldn't be a small shelter which dissipated the heat and simultaneously warmed the cold remained unclear. Politics, economics, and thermodynamics, three more subjects I guess I'll never understand.
I entered into a lobby that was a vast cavern of polished surfaces and open mezzanines bridged by oddly placed escalators and filled with deafening echoes - as if a spelunker had been hired as the architect. I explored up and down the escalators searching for the right floor, my destination tantalizingly visible above but without obvious access. I discovered an elevator that went only down and escalators to nowhere. I was tempted to just climb the mezzanine walls, and stood in frustration illuminating my goal three floors above with my key-chain laser as if I could just ride up the beam. Eventually I located a secret passage with the correct lift. "Card access required"; no problem, I just loaded in with the other white nerdly-looking ("tastefulness-challenged") guys like myself, and rode up - one considerate security violator even held the door open for me!
The elevator passengers were all wearing marketing-oriented black techno tees, the ubiquitous shirt of the elite programmer class. One of them was wearing the very same shirt that the homeless man outside had happily discovered and now proudly sported en masse. How excellent, in what other culture would two men on opposite ends of the income spectrum be found wearing the same clothes? If it weren't for the conversation regarding the ups and downs of various dynamic load-balancing routers, one might not have been able to distinguish this pack of nerds from an assembly gathered in front of a Seven Eleven at midnight.
I momentarily found myself at a receptionist's desk. My trivial ruse could have easily been continued past her by clinging like a pup to the pack, but I decide to stop like a good little dog and sign in; besides, I didn't know where my friend's cube was so it seemed easier to simply ask.
The receptionist was beautifully dressed, a trim black skirt and stylish blouse. Her hair coiled and piled on her head like a French noodle dish obviously required hours of preparation time; her face and manicured hands radiated their time and expense. In short, compared to the employees walking past her whose combined yearly compensation was more than should would make in a lifetime, she was the epitome of style and class - at least in appearance.
"John Smith Please," I asked.
I could tell immediately that this was going to be a tedious interaction as I had interrupted her highly focused concentration on a celebrity web-page and I regretted not sticking with the pack.
"Does he work here?" she inquired, not even bothering to look me in the eye.
"Yes." (No, I just like asking for strangers.)
She was skeptical; after a few moments of trace-like meditation she asked, "Are you here to see him?"
"Yes." (No, I'm conducting a survey of people with this name, come on.)
She stared past me as if there was something very interesting going on behind my back. I glanced around and saw nothing. Maybe she was waiting for a laser beam to appear out of nowhere again.
"Hmm, OK, sign the log please." and I did using my fake name "Puck Pandyco" (my lie being an automatic response - part of a one-man privacy campaign to stop the epidemic recording of unnecessary information, such as any purchase Radio Shack, hence the name.)
She stared at her screen which displayed the internal home page of the company's intranet. It had a company logo (what, the employees aren't sure of who they work for?) and no useful information on it other than a link to "Phone List". She stared at it for about 15 seconds and finally mobilized her arm for a click. She entered "J" and "S" into the search fields (apparently bright enough to note this abbreviation trick during training and lazy enough to use it) and managed to hit Enter despite the elaborately decorated inch-long fingernails that impeded accurate typing. The scrollbar indicated that the resulting list was about four pages; the first page ended at "Simmons". She unhesitatingly asked: "Are you sure he works here? He must be new."
"Try scrolling down to his name," I suggested, pointing at the scroll bar.
"No, I don't think he works here."
"He's worked here for about six months, just hit Page Down."
"Hmm... he has?"
"Yes, I'm sure if you'll just scroll down by clicking on this you will..." but before I could begin to explain the ingenious concept of an ordered alphabet, she had found a hard-copy version of the phone list and called his voice mail.
"A mister Puck...Uh... Pandyco to see you." (Oops, probably shouldn't have used my fake name.) She told me to sit down so that she could return to uninterrupted web-surfing; apparently Yahoo's Saturday weather forecast was very important to her job and needed immediate attention.
After a minute or two of contemplating my navel, I requested to use the phone so that I might call my friend's cell phone.
"I'm sorry, you can't use the phone." Ah yes, thinks I, with access to the phone I might be able to just walk right into this place and steal sensitive information, we can't allow that.
"How about you call him then." I asked.
I had her trapped; she glanced around again for stray laser beams and within a few minutes agreed to the call. Millions of dollars of sophisticated digital cellular radio technology thus deployed, my friend was located on the other side of the wall - I could have just spoken in a loud voice; oh well, ubiquitous technology creates ubiquitous waste.
The server tour gathered among the cube walls which were covered in nerd white-board graffiti - UML diagrams, code snippets, stock quotes, explicatives regarding Microsoft.
The inflationary Bay Area real-estate economy laid sprawled out around us. Almost every cube was doubled-up, two nerds per cube (with solitary managers, actually 1.8 n.p.c. I'd estimate). Flush with investor cash, every doubled cube was complete with two flat-panel LCD screens (~$3000 a piece, doing the same work as a $400 monitor) and Herman Miller chairs (~$700 a piece holding the same shapeless nerd asses that an $80 chair would hold). Loads of cash can buy you loads of toys, but it can buy you neither space nor attention in San Francisco as land and customers are the two commodities in short supply in dot com city.
We began the tour of the server infrastructure. I was informed that this server hosted one of the most heavily trafficked sites on the net - about 4 million page views a day. My friend and I calculated the hits per second as we walked (about 46 per second, maybe twice or triple that at peak). That's a lot, although by computer standards it's pretty trivial, at 2,000 bytes of text per page, a mere 100,000 bytes per second or so. A common desktop computer could handle that in its sleep and still play a decent game of Quake. But graphics add a lot and networks are orders of magnitude slower than the computers that run them - this was no trivial web page, and it was expected to be up 24/7 without interruptions. This requires industrial strength engineering, and my nerd excitement swelled.
The servers were downstairs, halfway underground - almost metaphorically halfway between the realities of the street-level homeless (the silhouettes of their bodies visible pressed against the upper windows) and the subterranean virtual world of fiber-optic cables and digital content.
To enter the server room required a key, a pass-code, and a "biometric scan" of the hand. Security cameras watched our every move. "There's not one square foot unmonitored here" our guide told us as he passed his hand over the scanning contraption which sent out sweeping laser beams and beeped satisfyingly. I noted that the ceiling tiles appeared to allow entry over the wall, good thing there's a $5,000 biometric scanner here to prevent that. As I entered, I pointed my pocket laser at the scanner in order that it might get a chance to meet one of its own; it again beeped satisfyingly.
The entryway displayed a prominent raised-letter, brushed stainless steel logo illuminated by halogens on quite expensive-looking bendable shafts. This is a place where no unguided persons should ever be, so why put an expensive logo here? Maybe it is to remind the employees who hadn't read the intranet homepage who they worked for - they worked for a company with too much cash.
The server room was a vast array of locker-like closets arrayed in deep perspective, each big enough for a man to stand in. They were flat black metal and very borg-like, just the kind of thing you might imagine such a place should include. On the other hand, the floors and walls were covered in fun swishy colors detracting from the digital death-star theme. However, this thematic flaw was more than compensated for by the overhead industrial wire guides traversing the ceiling in all directions carrying neatly organized bundles of fiber. Big Fat Data Pipes, the very beasts we came to the zoo to see. Data was downloading, bits were banging! This was a major artery of the net.
But something was very wrong with the scene. Where was the humming? It was deafeningly quiet; where was the fan noise, the disk-drives whirring, the smelly heat of thousands of hardworking microchips? Where was the "big hummer" as my father used to call the mainframes on which he "system analyzed" - i.e. toiled - at IBM? (Once my mother took a tour of his office and asked if a noisy machine in the corner of the server-room was "the big hummer that you're always talking about?" No, he replied, that was "the air conditioner.")
Indeed, the only things in the room making any noise (or heat for that matter) were the gigantic air conditioning systems along the walls. (Exhaust upstairs, bypassing cold homeless.) The computers should be cranking out the heat, the fans should be whirring up a storm! It should be a digital hurricane in here, what gives?
A few more steps and the answer was clear. The hundreds of racks were empty, awaiting customers. The hollowness made clear by the shimmering aliasing visible as the meshed front and back locker walls interfered with one another creating a moire pattern and a ghostly emptiness.
"Where is everything?"
Apparently they only have one customer so far. Apparently the timeline of bringing in the DS3s from the telecom companies is months overdue. Apparently there's been some trouble with the installation. Apparently they're planning and hoping for more. One sympathizes, engineer to engineer.
That's OK, I suggest, if there aren't any customers the place can be easily converted into a cryogenic facility - the lockers are just the right size for a body, and there's plenty of refrigeration here, just turn down the temperature. Surely there's going to be a big market in the Bay Area for that when all these over-monetized nerds start kicking over, and if the dot com stocks keep tumbling, this may be sooner than expected.
My brilliant business plans noted silently, we shuffled into the far corner. There, locked in a cage like monkeys, lived the beasts we had come to see: servers.
They hummed satisfyingly.
To the untrained eye, not much to look at; similar but less interesting than a audiophile's stereo cabinet: uniform boxes stacked with protruding cables, a few switches and power supplies, no blinky lights, no graphic equalizers wiggling up and down to a network heartbeat, no scrolling status panels. Nothing really, a common burglar probably wouldn't even bother to steal this stuff were it to be laying around in a house into which he had broken, regarding it as some-kind of uninteresting junk. But to the trained eye, it said something different. To me it said one thing, and it said it very, very loudly. It said: "OOGLES OF INVESTOR MONEY".
There were millions of dollars of hardware trapped in the cage. The box in the corner, a top-end Cisco router, quarter of a million right there. Six boxes next to that, eight CPU servers with a six gigs of RAM each. Hot-swappable RAID arrays, redundant power supplies, big cool switches. Some joker had put an old-fashioned Frankenstein knife switch in there just for good measure.
Our guide described the redundancy in detail. Two of these connected to four of those. Each gizmo fault-detected a pair of wizwangs. The widgets supported the foozles and the foozles monitored the widgets. Four separate fibers ran into the building from four different directions. Everything was fault tolerant out the wazoo. It was an engineering marvel, there was "no single point of failure," our guide said proudly.
Well, other than the fact that this building is sitting in one of the most seismically active parts of the world, I pointed out.
Yes, he admitted, there was no backup until the site on the East Coast went up.
"When will we have that?" someone else asked.
"We have no current plans."
That's OK, I mused, according to Yahoo weather no earthquakes are expected until at least Saturday.
The tour dispersed, we left the server room and returned to the offices. Past the facade of security (guards playing Gameboys), past the clean indoor putting green, past the tiny nine square-foot Zen rock garden gurgling its happy little gurgle, past the unscratched pristine indoor merry-go-round (stopping briefly to test the reality of the Coriolis force, of course) we wandered around the fantasy of a techno-worker paradise. A fantasy, of course, because these toys all sound cool in theory but nobody really uses them - everyone is far too busy fixing bugs and drinking coffee after all. Why, I thought, couldn't this excess budget have built a playground outside where the children of the neighborhood could have used it while the nerds satisfyingly stared at it as they do now? At least Rockefeller's generation built lasting structures as testaments to their obscene wealth and loose cash, this generation of entrepreneurs just buys fancy office furniture and toys, destined to be returned and resold from discount office furniture stores. (Someday, I suppose, a smart furniture store will open located inside the same building as the techno-offices and have a permanently installed crane on the roof which will rotate into place and repossess the cube furniture at a moment's notice of investor fall out.)
The correct buzzwordy techno-term for the business I toured is: "a server co-location facility." I couldn't help but think as I left the building, past a woman huddled under the shelter of the ATM machine, past the homeless-made tent, past the garbage-filled cylinders dutifully keeping the homeless cold, that maybe some of those millions would have been better spent on a "homeless co-location facility". Maybe some of that expensive real-estate filled with empty black lockers ought to be filled with apartments. Maybe there's paradoxically such a thing as too much capital and not enough cash. Maybe there's something sick about a society that chooses brushed stainless steel logos and $3000 LCD monitors over keeping everyone warm and healthy. But fortunately those thoughts passed quickly and I went home to surf the net.
Cisco's stock is up three and a half, and the weather tomorrow will be a lot like today... gimme some more content and content accessories quick, I'm getting impatient.