Lumentum's optical circuit switch for AI data centres

Part 3: Data Centre Switching

The resurgence of optical circuit switches for use in data centres is gaining momentum, driven by artificial intelligence (AI) workloads that require scalable connectivity.

Peter Roorda

Lumentum is one of several companies that showcased an optical circuit switch at the OFC event in San Francisco in March.

Lumentum's R300 switch connects optically the 300 input ports to any of the 300 output ports. The optical circuit switch uses micro-electro-mechanical systems (MEMS), tiny mirrors that move electrostatically, to direct light from an input port to one of the 300 output ports.

The R300 addresses the network needs of AI data centres, helping link large numbers of AI accelerator chips such as graphics processor units (GPUs).

"We've been talking to all the hyperscalers in North America and China," says Peter Roorda, general manager of the switching business unit at Lumentum. "The interest is pretty broad for the applications of interconnecting GPUs and AI clusters; that's the exciting one."

Optical circuit switches

In a large-scale data centre, two or three tiers of electrical switch platforms link the many servers' processors.

The number of tiers needed depends on the overall processor count. The same applies to the back-end network used for AI workloads.

These tiers of electrical switches are arranged in what is referred to as a Clos or "Fat Tree" architecture.

Tiers of electrical switches arranged in a Clos architecture. Source: Lumentum

Google presented a paper in 2022 revealing that it had been using an internally developed MEMS-based optical circuit switch for several years. Google used its optical circuit switches to replace all the top-tier ‘spine' layer electrical switches across its data centres, resulting in significant cost and power savings.

Google subsequently revealed a second use for its switches to directly connect between racks of its tensor processor unit (TPU) accelerator chips. Google can move workloads across thousands of TPUs in a cluster, efficiently using its hardware and bypassing a rack when a fault arises.

Google's revelation rejuvenated interest in optical switch technology, and at OFC, Lumentum showed its first R300 optical switch product in operation.

Unlike packet switches, which use silicon to process data at the packet level, an optical circuit switch sets up a fixed, point-to-point optical connection, akin to a telephone switchboard, for the duration of a session.

The optical switch is ideal for scenarios where large, sustained data flows are required, such as in AI training clusters.

How optical circuit switches (blue boxes) are used in a data centre. Source: Lumentum

Merits 

The optical circuit switch's benefits include cost and power savings and improved latency.

Optical-based switch ports are data-rate independent. They can support 400 gigabit, 800 gigabit, and soon 1.6-terabit links without requiring an upgrade.

"Now, it's not apples to apples; the optical circuit switch is not a packet switch," says Roorda. "It's just a dumb circuit switch, so there must be control plane software to manage it."

However, the cost, power, space savings, and port transparency incentives suffice for the hyperscalers to invest in the technology.

The MEMS-based R300

Lumentum has a 20-year history using MEMS. It first used the technology in its wavelength-selective switches used in telecom networks before the company adopted liquid crystal on silicon (LCOS) technology.

"We have 150,000 MEMS-based wavelength selective switches in the field," says Roorda. "This gives us a lot of confidence about their reliability."

MEMS-based switches are renowned for their manufacturing complexity, and Lumentum has experience in MEMS.

"This is a key claim as users are worried about the mechanical aspect of MEMS' reliability," says Michael Frankel, an analyst at LightCounting Market Research, which published an April report covering Ethernet, Infiniband and optical switches in cloud data centres. "Having a reliable volume manufacturer is critical."

In its system implementation, Google revealed that it uses bi-directional transceivers in conjunction with the OCS.

"Using bi-directional ports is clever because you get to double the ports out of your optical circuit switch for the same money, "says Mike DeMerchant, Lumentum's senior director of product line management, optical circuit switch. "But then you need customised, non-standard transceivers."

A bi-directional design complicates the control plane management software because bi-directional transponders effectively create two sets of connections. "The two sets of transceivers can only talk in a limited fashion between each other, so you have to manage that additional control plane complexity," says DeMerchant.

Lumentum enters the market with a 300x300 radix switch. Some customers have asked about a 1,000x1,000 port switch. From a connectivity perspective, bigger is better, says Roorda. "But bigger is also harder; if there is a problem with that switch, the consequences of a failure—the blast radius—are larger too," he says.

Mike DeMerchant

Lumentum says there are requests for smaller optical circuit switches and expects to offer a portfolio of different-sized products in the next two years.

The R300 switch is cited as having a 3dB insertion loss, but Roorda says the typical performance is close to 1.5dB at the start of life. "And 3dB is good enough for using a standard off-the-shelf -FR4 or a -DR4 or -DR8 optical module [with the switch]," says Roorda.

A 400G QSFP-DD FR4 module uses four wavelengths on a single-mode fibre and has a reach of 2km, whereas a DR4 or DR8 uses a single wavelength on each fibre and has 4 or 8 single-mode fibre outputs, respectively, with a reach of 500m.

An FR4 interface is ideal with an optical circuit switch since multiple wavelengths are on a single fibre and can be routed through one port. However, many operators use DR4 and DR8 interfaces and are exploring using such transceivers. "More ports would be consumed, diluting the cost-benefit, but the power savings would still be significant," says Roorda.

Additionally, in some applications, individually routing and recombining the separate ‘rails’ of DR4 or DR8 offer greater networking granularity. Here, the optical circuit switch still provides value, he says.

One issue with an optical circuit switch compared to an electrical-based one is that the optics go through both optical ports before reaching the destination transceiver, adding an extra 3dB loss. By contrast, for an electrical switch, the signal is regenerated optically by the pluggable transceiver at the output port.

LightCounting's Frankel also highlights the switch's loss numbers. "Lumentum's claim of a low loss - under 2dB - and a low back reflection (some 60dB) are potential differentiators," he says. "It is also a broadband design – capable of operating across the O-, C- and L-bands: O-band for data centre and C+L for telecom."

Software and Hyperscaler Control

Lumentum is controlling the switch using the open-source SONiC [Software for Open Networking in the Cloud] network operating system (NOS), based on Linux. The hyperscalers will add the higher-level control plane management software using their proprietary software.

"It's the basic control features for the optics, so we're not looking to get into the higher control plane," says DeMerchant.

Challenges and Scalability

Designing a 300x300 optical circuit switch is complicated. "It's a lot of mirrors," says Roorda. "You've got to align them, so it is a complicated, free-space, optical design."

Reliability and scalable manufacturing are hurdles. "The ability to build these things at scale is the big challenge," says Roorda. Lumentum argues that its stable MEMS design results in a reliable, simpler, and less costly switch.

Lumentum envisions data centres evolving to use a hybrid switching architecture, blending optical circuit switches with Ethernet switches.

Roorda compares it to how telecom networks transitioned to using reconfigurable optical add-drop multiplexers (ROADMs).

"It'll be hybridised with packet switches because you need to sort the packets sometimes," says Roorda.

Future developments may include multi-wavelength switching and telecom applications for optical circuit switches. "For sure, it is something that people are talking about," he adds.

Lumentum says its R300 will be generally available in the second half of this year.