Understanding ZeRO vs Pipeline Parallelism: A Systems-Level Perspective

Introduction

Training large AI models is no longer just a question of better architectures, it is a systems problem. Teams routinely encounter GPUs that sit idle despite abundant compute, or run out of memory long before compute is saturated. At this scale, performance is dictated by how well you orchestrate memory, computation, and communication across devices.

Two techniques dominate this space: Pipeline Parallelism and ZeRO Parallelism (Zero Redundancy Optimizer). They are often presented as interchangeable solutions for large models, but in practice, they solve different bottlenecks and behave very differently at runtime.

This article provides a concise, systems-level comparison of these approaches and clarifies when each should be used.

Core Idea: Computation vs Memory

Distributed training addresses two key problems:

• Distributing computation
• Managing memory efficiently

Pipeline Parallelism focuses on computation distribution by splitting the model into stages across GPUs. Each GPU executes a different portion of the model.

ZeRO Parallelism focuses on memory optimization by partitioning model states such as parameters, gradients, and optimizer states across GPUs, while keeping computation logically identical on each device.

In simple terms:

• Pipeline Parallelism divides the work
• ZeRO Parallelism divides the storage

Pipeline Parallelism

Pipeline Parallelism divides a model into sequential stages, each assigned to a different GPU. Data flows through these stages like an assembly line.

To improve efficiency, batches are split into micro-batches, allowing multiple GPUs to work simultaneously.

Key Characteristics

• Each GPU holds a subset of layers
• Computation is distributed across devices
• Activations are passed between GPUs
• Memory usage per GPU is reduced

Pipeline Bubbles and Scheduling

A key challenge in pipeline parallelism is pipeline bubbles, where some GPUs remain idle due to sequential dependencies.

To reduce this inefficiency, scheduling strategies are used:

• GPipe: All forward passes followed by backward passes
• 1F1B (One Forward One Backward): Alternates forward and backward passes for better utilization

The 1F1B strategy significantly improves GPU utilization and reduces idle time.

ZeRO Parallelism

ZeRO Parallelism is designed to eliminate memory redundancy in distributed training. Instead of replicating the full model on every GPU, it partitions model states across devices.

ZeRO Stages

• Stage 1: Partition optimizer states
• Stage 2: Partition optimizer states and gradients
• Stage 3: Partition parameters, gradients, and optimizer states

Execution in ZeRO

Forward Pass

• GPUs hold only parameter shards
• Parameters are gathered dynamically using AllGather
• Computation is performed
• Parameters are released after use

Backward Pass

• Gradients are computed locally
• ReduceScatter distributes gradients across GPUs
• Each GPU updates only its own shard

This approach maintains the same computation while reducing memory usage significantly.

Communication Differences

The primary difference lies in communication behavior:

Pipeline Parallelism

• Point-to-point communication
• Activations passed between stages
• Lower communication frequency

ZeRO Parallelism

• Collective communication
• Uses AllGather and ReduceScatter
• Higher bandwidth dependency

When to Use Each

Pipeline Parallelism is suitable when:

• The model is structurally too large for one GPU
• Interconnect bandwidth is limited (e.g., PCIe)
• Communication overhead must be minimized

ZeRO Parallelism is suitable when:

• Memory is the primary constraint
• Model architecture should remain unchanged
• Moderate communication bandwidth is available

Final Perspective

Pipeline Parallelism is often misunderstood as purely a memory solution. While it reduces memory usage, its primary role is to distribute computation across GPUs.

ZeRO Parallelism, in contrast, does not change how computation happens. Instead, it optimizes where model states are stored and how they are accessed.