Migrating 10TB Over the Weekend: Will Your Link Handle It?

You need to migrate 10TB of data to a new location by Monday morning. Can your network handle it? Let's do the math and figure out what's actually possible.

Bandwidth Calculator Calculate data transfer times for your migration

The Basic Formula

Time (seconds) = Data Size (bits) / Bandwidth (bits per second)

Remember: there are 8 bits in a byte. Storage is measured in bytes (TB, GB), but bandwidth is usually measured in bits per second (Mbps, Gbps).

10TB Transfer Times by Bandwidth

Converting 10TB to bits: 10 × 1024 × 1024 × 1024 × 1024 × 8 = 87.96 trillion bits

Bandwidth	Theoretical Time	Realistic Time*
100 Mbps	10.2 days	12-15 days
500 Mbps	2.0 days	2.5-3 days
1 Gbps	24.4 hours	30-36 hours
2.5 Gbps	9.8 hours	12-15 hours
10 Gbps	2.4 hours	3-4 hours

*Realistic times account for protocol overhead, disk I/O limitations, and network inefficiencies.

Why Theoretical Doesn't Match Reality

Several factors reduce actual throughput:

1. Protocol Overhead

TCP/IP adds headers to every packet. Typical overhead:

• Ethernet: 38 bytes per frame (minimum)
• IP: 20 bytes per packet
• TCP: 20 bytes per packet
• SMB/NFS/CIFS: Additional protocol overhead

Expect 5-10% loss from protocol overhead alone.

2. Disk I/O Bottlenecks

Your network might be fast, but can your storage keep up?

Storage Type	Sequential Read	Sequential Write
Spinning HDD	150-200 MB/s	150-180 MB/s
SATA SSD	500-550 MB/s	450-520 MB/s
NVMe SSD	3,000-7,000 MB/s	2,000-5,000 MB/s
RAID Array (8×HDD)	800-1,200 MB/s	600-900 MB/s

A single spinning HDD at 150 MB/s (1.2 Gbps) can't saturate a 10 Gbps link. You'd need multiple disks or SSDs.

3. Small File Penalty

Millions of small files transfer much slower than a few large files:

• 1 million × 10KB files: Might transfer at 20-50 MB/s
• 10 × 1GB files: Might transfer at 200+ MB/s

Each file requires metadata operations, open/close syscalls, and protocol negotiations.

4. Network Latency and Distance

TCP throughput is limited by latency:

Max throughput ≈ Window Size / Round Trip Time

A 64KB window with 100ms RTT caps at about 5 Mbps. For high-bandwidth, high-latency links, you need:

• Increased TCP window sizes
• Multiple parallel streams
• WAN optimization appliances
• Transfer protocols designed for high latency (like GridFTP or Aspera)

A Realistic Weekend Migration

Let's plan the 10TB migration with a 1 Gbps dedicated link and 20ms latency:

1. Calculate theoretical time:

   10TB × 8 bits/byte / 1 Gbps = 24.4 hours

2. Account for overhead (10%):

   24.4 × 1.10 = 26.8 hours

3. Check disk speed: Source is RAID array (900 MB/s = 7.2 Gbps) - not the bottleneck
4. Check destination: Single SSD (500 MB/s = 4 Gbps) - this is the bottleneck!
5. Recalculate with bottleneck:

   10TB / 500 MB/s = 20,000 seconds = 5.5 hours... wait, that's better?

Actually, 500 MB/s destination can absorb 1 Gbps (125 MB/s) network throughput easily. The network is the bottleneck at 125 MB/s effective.

10TB / 125 MB/s = 80,000 seconds = 22.2 hours

Add 20% buffer for small files and retries: ~27 hours

The Weekend Window

A typical weekend migration window:

• Friday 6 PM to Monday 6 AM: 60 hours
• Saturday 6 AM to Sunday 6 PM: 36 hours

With a 27-hour transfer time and 60-hour window, you have margin for:

• Initial sync (maybe 20 hours)
• Verification and checksums (2-4 hours)
• Final delta sync (1-4 hours)
• Testing and validation (2-4 hours)
• Rollback buffer (remaining time)

Faster Options

If the math doesn't work, consider:

Parallel Transfers

Run multiple rsync/robocopy processes on different subdirectories. This helps with small files and can better saturate the link.

Compression

If data is compressible (logs, text, uncompressed images), enable compression:

• 50% compression = 2x effective speed
• Costs CPU on both ends
• Already-compressed files (ZIP, JPEG, video) won't benefit

Pre-Staging

Start copying non-changing data early:

• Old archives and historical data
• Read-only reference files
• Application binaries

Only the delta needs to transfer during the maintenance window.

Physical Shipping

For very large datasets, physical shipping wins:

"Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway." - Andrew Tanenbaum

10TB on NVMe drives, overnight shipped: effectively ~100 Gbps

Quick Reference: Transfer Time Calculator

Hours = (Data in TB × 2.22) / Link speed in Gbps

Examples:
10TB on 1 Gbps  = 10 × 2.22 / 1 = 22 hours
10TB on 10 Gbps = 10 × 2.22 / 10 = 2.2 hours
50TB on 1 Gbps  = 50 × 2.22 / 1 = 111 hours

Add 20-50% for real-world conditions.