PYTHON · NUMPY

Where a switch spends its joules

An optical network draws under 3% of a supercomputer's power. Swap it for a packet-switched network and training slows by 10%. Before the new switches draw a single watt, how much more energy does a training run take?

Recover the share of training time a network governs from two published slowdowns, and show that its energy is decided by the seconds it saves every other chip rather than by its own share of the power.

01Challenge

Google's TPU v4 supercomputer joins 4,096 chips through optical circuit switches: mirrors that steer light from fibre to fibre without ever turning it back into electricity. Its authors put the entire optical fabric at under 3% of the machine's power, and report that a hybrid network with Infiniband switches in its place would slow a training job by as little as about 10%. Decide which of those two numbers the energy bill follows.

Predict firstAn optical network draws under 3% of a supercomputer's power. Swap it for a packet-switched network and training slows by 10%. Before the new switches draw a single watt, how much more energy does a training run take?

02Model

The last lesson's two numbers could not be multiplied. These two cannot even be compared until you see that they are in different currencies. A share of the power is counted in watts. An energy bill is watts multiplied by seconds, and a network is the one part of a computer whose real product is seconds: the seconds every other chip does not spend waiting for data. So the question to ask of a switch is how much time it saves everything else, and what everything else draws while it waits. The TPU v4 paper answers that from its own sentences. The whole optical fabric, optics modules, fibre and switches together, is under 3% of the machine's power. Replacing its 48 optical switches would take 568 Infiniband switches, 11.83 times as many boxes, and in the authors' simulation that network runs an all-reduce 1.8x to 2.4x slower and slows a whole training job by as little as about 10%. The share of the job that was communication appears in neither sentence, and together they pin it, because a job slows by exactly its communication share times how much the communication stretched. Then the seconds get a price, and the price is larger than the fabric's entire share of the power. The same arithmetic holds in any machine whose parts stay powered while they wait, and a robot balancing on two legs is such a machine.

03Guided practice

1. Write down what stretches. On the optical machine a job spends a fraction f of its time communicating and 1 - f computing. Swap the network and only the communication stretches, by a factor s, so the job now takes (1 - f) + f * s of its old time. 2. Invert it. The paper gives the whole job's slowdown S and the communication's slowdown s. Solve (1 - f) + f * s = S for f. Check yourself: communication twice as slow and the job 10% slower means communication was 10% of the time. 3. Read a range as a range. The all-reduce slowdown is 1.8x to 2.4x, so f is a range too. The smaller stretch gives the larger fraction, because a smaller stretch has to act on more of the run to cost the same 10%. 4. Price the seconds. The added time is time the chips spend waiting. The paper measures a chip at 90 W idle and 170 W on average under production load, so a waiting second costs between 90/170 of an average second and all of one. The bench assumes the rest of the machine waits the way the chip does, and says so.

04Feedback

PASS WHEN You recovered a fraction that two published sentences imply between them, and then priced a network in the currency its power share is not counted in: the seconds every other chip spends waiting.

If your run fails, check:

Upside down gives a fraction above 1. The answer is (job_slowdown - 1) / (comm_slowdown - 1), and the check case must give 0.10.
Only the communication stretches. The computing share 1 - f keeps its length, so the whole job becomes (1 - f) + f * s, not f * s.
Define comm_fraction(job_slowdown, comm_slowdown) returning the fraction of the original run spent communicating.

05Retrieve & space

The simulator behind the 1.8x to 2.4x leaves out protocol processing on the host CPU, which the authors say can be significant. Which way does that move the price of the added seconds, and why does it leave the fraction you recovered where it is?
The optical fabric is quoted at under 3% of the power. Say exactly what that number bounds, and name the energy term it cannot bound by construction.
Communication was 7.14% to 12.50% of the run for an all-reduce-bound job. Why does the smaller stretch, 1.8x, give the larger fraction?

06Mastery & project

You can take a component quoted by its share of a machine's power, recover how much of the machine's time it governs from published slowdowns, and say which of the two an energy bill follows.

Pick one shared part of a machine you know, a bus, a network link, a scheduler or a sensor hub, and write down two numbers: its share of the power, and the share of everything else's time it decides. Price the second in watts multiplied by seconds. If it comes out larger than the first, the part's energy case has been argued in the wrong currency.

GO FURTHER · THE LIVE PLAYGROUND

The TPU v4 paper ↗ · Jouppi, Kurian, Li, Ma, Nagarajan, Nai, Patil, Subramanian, Swing, Towles, Young, Xiang Zhou, Zongwei Zhou and Patterson, ISCA 2023. The optical fabric's power share is in Section 2.10, the Infiniband comparison in Section 7.3, and measured chip power in Table 4.
Bounding what you cannot measure ↗ · The same inversion, solving a published ratio for the term it leaves unstated.
The fence a fall buys ↗ · A machine that cannot idle while it waits: a balancing robot's safe state is actively held.

← Energy at one length, accuracy at another Back to course →

PYTHON · NUMPY

Where a switch spends its joules

Two sentences from one paper, one fraction that appears in neither, and a price for the seconds a network saves.

You read

the arrays and values already in scope

You change

the code you write in each cell

Fixed

the dataset and the checks that grade you