Technology

Between 1998 and early 2000, analog dialup started to slowly slip as ADSL and cable services were deployed. There was no doubt that faster, full-time connections were needed for both homes and businesses. After all, these were the heady days of the dotcom bubble. Telecom companies and cable providers were just not deploying services quickly enough. It's no conspiracy theory that telecom companies didn't want to provide the service because it threatened to undermine an existing business, and that cable companies had quite a bit of network upgrading to perform before wide deployment could even occur. Furthermore, both ADSL and cable were -- for the most part -- the purview of the incumbent telecom and cable companies. Internet Service Providers, try as they may [1][2], didn't have a chance in these markets.

The dotcom crash, 9/11/01, and the telecom scandals didn't help things. What did occur following those disruptions was the popularization of Wi-Fi (IEEE 802.11). Also, the value of the Internet barely skipped a beat. E-commerce, digital music, voice-over-IP and video were almost entirely grass-roots movements, driving the necessity for bandwidth ever higher. Podcasting, video blogging, and television distribution through Apple's iTunes are happening today -- not at some distant future time. The ADSL and cable bottlenecks are the dilemma that continue to constrict progress.

Without touching on the distinction between theoretical and real bandwidth, let's continue the relative network speeds graph from Part I of this post.

Although both ADSL and cable services have begun to deliver faster speeds, they're both dwarfed by the network performances most of us experience every day in our home and office environments. Furthermore, they are not technically different from the ADSL and cable services provided in 1998 -- they're just much closer to their maximum potential. To put my point into greater relief, the following graph presents a comparison between the connection speeds of my actual networks as I write the post.

The server where my blog exists is in a data center next to Seattle's Space Needle. It connects to the physical Internet via a full-duplex, 100 megabit per second Ethernet port. My office has both an 802.11g Wi-Fi access point (half-duplex, 54 megabits per second) and a full-duplex, 100 megabit per second Ethernet network. The fastest Internet connection I could have installed at my office -- without paying several thousand dollars per month -- was 7 megabit per second (down) by 1 megabit per second (up) ADSL. Most people don't even have that fast of a connection.

With the growing popularity of gigabit per second Ethernet and the new 802.11n Wi-Fi specification, the disparity will only grow.

In summary, new Internet applications such as podcasting, music and video distribution are continuing to necessitate higher network speeds. Whereas local area networks based on Ethernet, fiber and Wi-Fi regularly operate at speeds between 100 and 1,000 megabits per second, telecom and cable companies have reached the end of their bandwidth potential using the existing copper and coax infrastructure. A move to ADSL2+ or Hybrid-Fiber-Coax will not be enough to address this ever-increasing disparity. Part III of this post will explore possible futures.

In the early days of commercial Internet access -- when I ran a regional Internet Service Provider in Washington State -- the primary service was analog public switched telephone network (PSTN) connections at 14,400 bits per second, 28,800 bits per second, and 33,600 bits per second. The 57,600 bit per second analog modems, introduced in 1996, exhausted the bandwidth capability of the traditional voice network. In fact, it was the first time in which a large proportion of the connections could not reach the maximum performance rating.

During this "analog age", there was also a great battle waged between providers that sold a metered service and providers that sold a flat (a.k.a., "unlimited") service. The main limitation for both types of providers was that they didn't achieve a 1:1 ratio of PSTN ports to customers -- doing so would have been both cost prohibitive and, essentially, impossible. Contrary to what you might imagine, however, the challenge in deciding which camp a provider should belong was not related to determining an appropriate port-to-customer ratio through a distribution analysis of past call frequency and duration. This would have been totally inadequate in light of the dramatic change in usage patterns occurring month after month. Instead, the providers selling an "unlimited" service were making a losing bet.

It didn't take a Nostradamus to predict the future, as the Internet shifted from a primarily text environment (E-mail, Telnet, Gopher, FTP, et al.) to one that included graphics and multimedia (HTTP, RTSP, etc.). Call frequency and duration skyrocketed. The new features not only demanded more bandwidth, but also increased the number of reasons people might want to connect. There was a rapidly growing desire for high-speed, full-time connections to the Internet, and the public switch telephone network was not a good fit.

The analog voice network, however, was so pervasively installed that "technologies" even started to appear for bonding multiple PSTN lines. These connections were almost always assumed to be full-time as well. Although bonding analog connections never really took hold as a viable practice, it's important to realize how much momentum is generally behind squeezing capability from a ubiquitous network. This brings us to ISDN.

We began selling ISDN connections in 1995. The same copper wire pairs used for delivering PSTN circuits can, in most cases, deliver an ISDN circuit. An ISDN Basic Rate Interface (BRI) delivers up to two 64,000 bit per second digital channels. Most ISDN equipment can bond the two channels, for a maximum of 128,000 bits per second. ISDN, however, operates on the same "switched network" paradigm as PSTN. As a technology for delivering high-speed, full-time Internet service, it was nothing more than a flash in the pan. Asymmetric Digital Subscriber Line (ADSL) and cable Internet service overtook ISDN by the end of 1998. Both ADSL and cable service provided faster, full-time connections to the Internet.

During this same period, it was popular for businesses to purchase dedicated Internet connections at rates between 56,000 (DS0) and 1,544,000 (DS1) bits per second. Although these full-time, digital circuits used the same sort of copper wire pairs (or pairs of pairs, as is the case with DS1) that analog voice circuits used, they were separate from the public switched telephone network. The very maximum Internet connection speed that most businesses could financially justify between 1995 and 1998 would have been the DS1, costing between $1,000 and $2,000 per month. Only the largest organizations and Internet Service Providers purchased circuits larger than a DS1, and I only mention them here for comparative purposes.

In summary, the early growth of the Internet saw a rapid shift from part-time connections based on the analog PSTN to full-time connections based on the much faster DSL and cable services. The demand for faster, full-time Internet connections resulted from the growing benefit derived from its use. Part II of this post looks at the period from 1998 to the present.

This year marks the beginning of my two-year appointment by Greg Nickels, Mayor of the City of Seattle, to the Citizens' Telecommunications and Technology Advisory Board (CTTAB). The board advises the Mayor's office and the City Council on technology and communications issues facing Seattle. Although my background includes a variety of technology competencies, my primary contribution will likely be with the committee consulting on Seattle's Broadband Initiative. Watch for the new FTTx (Fiber To The Home and Fiber To The Premises) category on this blog to follow developments over the next couple years.

Melissa -- now the proud owner of an 8GB iPod Nano (Product)^RED -- turned me on to the best workout mixes I've ever heard. DJ Steveboy mixes hour-long tracks of 130 to 180 beat-per-minute ear candy. It's the sort of music that makes you want to throw up your arms and dance while you run. Not even the music from Rocky touches the sort of nerve that PODRUNNER tracks tickle. Take a dose for your next run.

You can get it through iTunes:

In an age where progress is heavily constrained by the limits of the first mile networks (i.e., the twisted-pair copper from phone companies, and coax from cable companies that connect homes and business to core networks), it's extremely important that anyone approaching the issue of addressing that problem understand the weight that the late-1990's long-haul fiber bubble places on Fiber To The Home (FTTH) or Fiber To The Premesis (FTTP) initiatives (FTTx collectively). Fiber is, without question, the technology that will be deployed in place of twisted-pair and coax networks. The journey to this inevitable FTTx future is, however, obstructed by many countervailing interests and historical baggage. Om Malik's book, Broadbandits: Inside the $750 Billion Telecom Heist, documents how the telecommunications industry stumbled in a race to cash in on the build-out of long-haul fiber networks. It's my opinion that the market and public skepticism resulting from the telecom scandals have, and will, retard the oh-so-necessary deployment of fiber infrastructure in the first mile. In fact, it's this "hangover" that appears to be keeping the telecom industry from playing a leadership role in FTTx. Read the book.

Over the past week I've finished quite a few projects at the data center. These were the sorts of projects that took months to coordinate (as evidenced by my March 5th, 2006 post about being "halfway there"). I wish I could claim that all of these projects were proactive; that they were borne out of my impeccable engineering competence. Sadly, I have to admit that some of them were provoked by necessity.

Nathan Rolander sheds some light on the fundamental issue in his December 2005 masters thesis, "An Approach For The Robust Design of Air Cooled Data Center Server Cabinets", when he states:

"A lifecycle mismatch is present in data center operation. This is because data centers receive new high powered [sic] servers every 2 to 3 years, whereas the center infrastructure is only upgraded on the order of every 25 years. This means that the center must be reconfigured to handle the increased heat load quite frequently, and after a few iterations of the process the center is required to dissipate far greater loads than initially intended."

Since heat production in a data center is intimately tied to power consumption (every Watt of power used produces approximately 3.4 BTUs), the lifecycle mismatch should be understood to encompass both cooling and power. This is definitely true in our case, and is a problem exacerbated by what amounts to "absentee landlords" at most data center facilities. I don't wish to place all the blame on data center operators, but they really should be taking a position of leadership in this situation. The power outages on July 24th and July 28th at the Garland Building in Los Angeles, California (knocking out DreamHost and MySpace), and the power outage on July 30th at the Fisher Plaza building in Seattle, Washington (knocking out LiveJournal and us [Geckowerx]), are - at a minimum - ominous.

As background, each cabinet (28" W x 36" D x 84" H) at our data center is equipped with one 20 Amp primary and one 20 Amp secondary AC power circuit. The total power available to a cabinet should be understood as 20 Amps - not 40 Amps. (The secondary circuit is intended for redundancy purposes.) The cabinets are cooled using the vertical flow design, where chilled air enters the cabinet through an opening in the base and exits through a fan (rated at 500 CFM) in the top of the cabinet. The doors and sides of the cabinets are not perforated, whereas they would be perforated in a horizontal flow design.

It's important to note that - unlike power - cooling is an issue of degrees. (Yes, I'm sorry. That's a pun.) Data center equipment can operate within a fairly permissive temperature range (e.g., 41F to 104F, in the case of our servers); however, power is usually present or not present - there's very little leniency for "partial power". For this reason (and because heat production and power consumption are so closely correlated), adequate cooling tends to receive attention only after power shortages occur. The attention given our cooling and power adequacy happened, coincidentally, in parallel.

Following an upgrade to several servers, I become concerned with what appeared (anecdotally) to be a pooling of very hot air in the top of a cabinet. Not wanting to rely on anecdote, I ordered and installed AVTECH's TemPageR 4E with one sensor at the base of the cabinet and one sensor at the top (both at the rear). The empirical evidence confirmed the anecdote. The cabinet was experiencing a differential temperature of more than 20F (which is considered the maximum differential you would want in an enclosure of this sort).

After some trial and error, I settled on Delphi's Enclosure Blower (rated at 250 CFM) and sealed any empty rack spaces with blanks. Although this did not exactly resolve the pooling of hot air in the top of the cabinet, it did ensure that each device - despite its location in the cabinet - received a supply of chilled air. (Note: Since most of the equipment has internal temperature sensors that can be queried, I know whether or not each device is operating within its specified temperature range.) Adding two more temperature sensors, this time to the front of the cabinet, confirmed that the Delphi Enclosure Blower was effective. In fact, the temperature at the top, front of the cabinet is about 5F lower than its bottom, front. This is clearly the result of keeping the bottom temperature sensor outside of the blower's direct airflow, whereas the top temperature sensor cannot avoid the chilled air delivered by the blower.

The hotspot in the top of the cabinet still exists, but has been reduced by several degrees. Its potential for damage has also been reduced somewhat. Unfortunately, the 500 CFM exhaust fan simply isn't powerful enough to eliminate the hotspot in the top of the cabinet. My calculations indicate that a fan closer to 900 CFM would be necessary. Also, the delivery of chilled air should be increased to match this higher flow rate. A possible alternative is to implement a hybrid of the vertical and horizontal flow designs by adding a perforated back door to the cabinet, while leaving the front door unperforated. Due to the depth (which may constrain airflow) of several devices in the cabinet, a hybrid approach might be the better option.

"What about power," you ask? It was during some of the server upgrades responsible for the additional heat that I tripped the primary 20 Amp circuit during a boot cycle. It's not a pretty sight to watch a cabinet full of equipment lose power in an instant. Running everything with journaled filesystems and really good backups meant that I didn't need to pop the cyanide pill that all systems engineers and some administrators (the smart ones) carry with them in case of a really catastrophic failure of their own doing. Nonetheless, it was a serious slap across the face. My power requirement calculations were obviously off, even accounting for the inrush demand of booting.

In all honesty, I hadn't revised my power requirements to include the upgraded equipment. (There's a reason why 30% of system downtime is attributed to human error, and this was just such an illustration. Note to self: the additional heat should have been a clue that revising the power requirements was necessary.) Past estimates had indicated a fair capacity margin for a full cabinet (e.g., nominal use of approximately 12.7 Amps out of the 20 Amp circuit). The revised estimates freaked me out. The nominal use estimate was at 24.1 Amps. I had to power down equipment just to bring things back into a safe operational range. Not powering down equipment would have risked throwing a circuit breaker while physically away from the data center.

Two tasks were necessitated by the revised power estimates. First, we needed to start collecting empirical power usage data (similar to what we had already started doing for the temperature). Power estimates are just that: estimates. Real-time data is the experiment that can validate or invalidate your estimates (a.k.a., hypothesis).

Second, we needed larger circuits or additional cabinets.

Following an unhelpful conversation with the data center owner (InterNAP), I opted to order a pair of the APC AP7830 smart PDUs (rated for 20 Amps). At least I could begin gathering data even if they weren't able to provide circuits larger than 20 Amps. A subsequent conversation, involving a different representative, yielded a better result. InterNAP could, in fact, upgrade our 20 Amp circuits to 30 Amp circuits. Of course, this would come at a 41% increase in our per-cabinet, monthly fees (not including our bandwidth charges - which are billed separately). We had no choice. The power upgrade was essential. I immediately ordered a pair of the APC AP7832 smart PDUs (rated for 30 Amps) for the cabinet in question. I'll reserve the AP7830s for another cabinet that has lower power requirements.

The planned power upgrade and installation of the smart PDUs gave me an opportunity to test another power enhancement for the cabinet. Even though much of the data center equipment contains redundant power supplies, certain items (such as high-density servers) are notorious for their lack of power redundancy. A potential solution, which I'd had no prior experience with, is the "dual input PDU". The dual input PDU accepts power from two sources, and switches between them in less than one cycle if power is lost or restored on the primary source. Although dual input PDUs are no replacement for redundant power supplies, they do eliminate a single point of failure that exists when they aren't present. They also allow for complete dependence on a primary circuit, while failing all equipment over to the secondary circuit only when the primary fails. I was a little skeptical at first, but the engineers at Pulizzi were incredibly knowledgeable and informative. I purchased three of the EMF/RFI filtered TPC2234s for the cabinet getting the 30 Amp circuit upgrade. They've been tested under heavy load since installation, and operate exactly as advertised.

These projects have resulted in a cabinet with four points of real-time temperature monitoring and two 30 Amp circuits served through smart PDUs in a truly redundant configuration (thanks to the addition of dual input PDUs for all single power supply devices). Unfortunately, the knowledge gained through real-time temperature and power monitoring indicates that approximately half of a traditional 40U cabinet must remain empty to accommodate the cooling and power demands of the latest data center equipment. I will be curious to see how the necessary accommodations influence costing models that were based on larger unit densities of the newer equipment.