We benchmarked our own 4 vCPU cloud VM in public: NVMe-backed Ceph, raw data included

Most VPS benchmark posts show one big IOPS number, from one run, on a machine you cannot rent. We did it the other way. We wrote an open-source benchmark suite, ran it three times on an ordinary BookMyHost cloud VM, and put every raw fio, sysbench and iperf3 file on GitHub. You can rerun every number here on your own server, or on ours.

This post leads with what our storage is good at: throughput under parallel load and consistency over time. It also shows plainly where it is slower than a single local NVMe drive, and why we built it that way.

The setup

Every number below is the median of three separate runs. The run-to-run table is in the raw data.

Headline results

Throughput under parallel load

With four jobs at queue depth 32, the VM read 80,088 random 4k blocks per second and wrote 19,839. That is the load a busy database, a CI runner or a container host generates: many requests in flight at once. Ceph spreads them over every NVMe drive in the pool, so parallel IO scales across drives instead of queuing behind one. Sequential 1 MiB reads reached 4,026 MB/s and writes 1,822 MB/s, which matters for backups, restores and media.

Writes cost more than reads on erasure-coded storage. Each 4k write becomes three data chunks plus one parity chunk on four different hosts, so random write IOPS are roughly a quarter of random reads. We think that is the honest price of not losing data when a drive dies.

Consistency: the noisy-neighbour probe

A benchmark that runs for 15 seconds can hit a quiet moment. The suite ends with a 60 second probe that records CPU steal every second and a 4k QD1 read latency sample every 10 seconds.

• CPU steal: 0.00 % on average, 0.00 % at the worst second. Above about 2 % sustained, a VM's CPU numbers stop meaning much. We were nowhere near it.
• Latency spread: the QD1 read p99 across the probe samples stayed within 1.09x (p99 between 1,761 and 1,958 us). A spread above 2x usually means a neighbour is hammering the same disks.
• Run to run: across the three full runs, the QD32 random read result moved 77,403 to 82,186.

CPU thread scaling

sysbench prime search scored 514.9 events/s on one thread and 2,051 on four, a 3.97x scaling on four vCPUs. Scaling well below the vCPU count hints that the "cores" share physical threads with someone else under load. Memory bandwidth for a single thread was 25,190 MiB/s read and 22,423 MiB/s write.

Network: iperf3 between two VMs

We ran an iperf3 server on a second VM in the same zone and pushed 4 parallel TCP streams for 10 s in each direction: 191.1 Mbit/s up and 190.9 Mbit/s down, with 0.33 ms average ping between them. This is VM to VM inside one zone, not internet throughput. It sits right at the 200 Mbit/s per-VM throttle that applied to this test offering (CloudStack's vm.network.throttling.rate of 200 Mbit/s, because the test offering sets no rate of its own), so it measures the shaping limit, not the fabric.

Where local NVMe is faster, and why we accept it

Here are the numbers a local-NVMe VPS will beat us on. We are publishing them anyway.

A local NVMe drive answers a single 4k read in well under 100 microseconds. On our storage, the same read crosses the network to the host that holds the object, and a durable write waits until enough chunks have landed on enough hosts. That round trip is why a single-threaded app that does one IO at a time (QD1), or a database that fsyncs every commit, sees about a millisecond or more per operation instead of a fraction of one.

What you get for that latency:

• A dead drive or a dead host does not take your VM down or lose your data. The EC 3+1 pool rebuilds from the surviving chunks.
• Live migration. We can move your running VM off a host for maintenance or a hardware fault without a reboot, because the disk is not tied to that host.
• Throughput that grows with the cluster, as the QD32 numbers show, instead of the ceiling of one drive.

If your workload is a single-threaded, fsync-heavy database on one VM and you handle replication yourself, a local-NVMe host will be faster per commit. For most web apps, APIs, CI and container workloads, which run many IOs in parallel, the parallel numbers above are the ones you feel, and you keep your data when hardware fails.

Methodology

• fio with --direct=1 (bypasses the guest page cache), ioengine=libaio, time-based 60 s jobs on a 4 GiB file pre-filled with real data, so thin-provisioned zero blocks cannot inflate reads.
• Tests: 4k random read and write at QD1 and QD32 x 4 jobs, 70/30 mixed, 64k and 1M sequential at QD16 x 2 jobs, a dedicated QD1 latency test with lat_percentiles, and a 4k write with fsync=1.
• sysbench CPU (--cpu-max-prime=20000, 1 and 4 threads) and memory (1 MiB blocks, 1 thread), openssl AES-256-GCM, a pure-Python loop.
• iperf3, 4 streams, 10 s each way, to a second VM in the same zone. Ping to 1.1.1.1 and HTTPS TTFB to public sites.
• A 60 s noisy-neighbour probe: steal % every second, fio QD1 sample every 10 s.
• Three separate runs on a fresh VM with nothing else running; every published number is the median across runs (report.py median).
• The VMs were temporary and destroyed after the run.

Full details, including VM caveats like steal, ballooning, burst credits and thin provisioning: docs/methodology.md.

Reproduce it

sudo apt-get install -y fio sysbench iperf3 jq
git clone https://github.com/geetmangal/bmh-nvme-bench && cd bmh-nvme-bench
# optional: start "iperf3 -s" on a second machine in the same region
for i in 1 2 3; do
  ./bench.sh --profile standard --no-prompt --label my-vps-4c8g --iperf-server <server-ip> --target $HOME/bench-tmp
done
python3 report.py median results/my-vps-4c8g-* --out results/my-vps-median
python3 report.py compare data/2026-10-25-bmh-noida-standard/median results/my-vps-median --labels BookMyHost,MyVPS

• Repo (MIT): https://github.com/geetmangal/bmh-nvme-bench
• Raw data for this post: https://github.com/geetmangal/bmh-nvme-bench/tree/main/data/2026-10-25-bmh-noida-standard

How other providers compare

Placeholder, to be filled from the competitor trial runs (see docs/comparison-runbook.md in the repo). Same standard profile, Ubuntu 24.04, 4 vCPU / 8 GB, region nearest India, three runs, median, raw data published.

FAQ

How many IOPS does a BookMyHost 4 vCPU / 8 GB VM get? 80,088 4k random read and 19,839 4k random write IOPS at QD32 x 4 jobs, median of three runs on 25 Oct 2026.

Is the disk local NVMe? No. It is Ceph RBD on an erasure-coded 3+1 pool over Micron 7400 NVMe drives: network block storage that survives a drive or host failure and allows live migration.

Why are QD1 and fsync latency higher than on some VPS hosts? Each IO crosses the network, and each durable write lands on several hosts before it is acknowledged. That is the cost of redundancy; parallel throughput is where the design pays back.

Can I run the same benchmark? Yes: the suite is MIT licensed, and the commands are above.

Did you test other providers? Not yet. Those runs are planned with the same rules, and this post will be updated with their raw data.

Repository · Raw benchmark data