Article URL: https://tracewayapp.com/blog/sqlite-vs-duckdb Comments URL: https://news.ycombinator.com/item?id=49097730 Points: 65 # Comments: 36

Same $16.49/month server, same Traceway binary, two embedded databases. DuckDB writes 4x to 15x faster than SQLite, serves dashboards at 100x the row count, and stores a billion metric points in 10.8 GB. Full numbers and methodology inside. I spent six days working on and running a benchmark of observability data on DuckDB. I've done this with SQLite in the last blog post and I really wanted to see how DuckDB compares on a cheap CCX13 Hetzner instance. The way I've done the measurements is by implementing DuckDB as a valid storage engine for Traceway and then running its benchmarking suite. The benchmarks show that DuckDB's columnar engine is able to query 100x more data points without any major backend changes. The write throughput is also 3x to 15x higher. The result is that you can now self-host the full OTel stack that can handle a large data volume on a pretty small server. Keep reading to find out how I've done the measurements! A read cliff, throughout this series, is the largest table size at which real dashboard pages still load: the median of three endpoint probes at or under 5 seconds, none timing out. It earns the name because of what sits one 10x step past it: queries don't slow down, they stop coming back. Each signal's read cliff sits exactly 100x further out on DuckDB than on SQLite, at equal or better latency, on identical hardware. I checked that symmetry against the raw JSON twice before I believed it. The box also ingested a billion metric points in under an hour into 10.8 GB of disk and stayed up, a scale the SQLite benchmark never got within 100x of. Logs, the signal post 2 said to keep off SQLite entirely, are DuckDB's biggest win. Post 2 ended promising a ClickHouse comparison, and that post is still coming. But every time I sat down to build it, the same question got in front of me: before reaching for a client-server OLAP database, with its own container, its own memory appetite, its own failure modes, how far does an embedded one go? Traceway ships a DuckDB telemetry backend as an opt-in build, same single binary, same deployment story as the SQLite build, different storage engine under the telemetry tables. If you self-host on one cheap box, these two builds are the actual decision in front of you, and I couldn't find anyone who had published numbers for it. So I ran post 2's entire methodology against the DuckDB build and put the results side by side. Here's the plan. Methodology first, including what changed in the harness since post 2 and one fix in the backend itself that these numbers depend on. Then writes, all three signals against their SQLite baselines. Then reads, same shape. Then the billion-row run, what queries cost past the cliffs, the fleet math for both builds, and which build I would actually run. The setup is post 2's, so I'll keep it short: a system under test (SUT) running the Traceway binary and a separate load generator on a private link, both in Hetzner's Nuremberg datacenter, OTLP, the OpenTelemetry wire protocol, over gzipped protobuf, uniform-random data in bounded ranges. Traceway built from commit 14b4aa6e, Ubuntu 24.04, retention off during the bench. Two scenarios per signal. The throughput ramp: batch-size ramp at fixed rate, then a request-rate ramp at the winning batch (this time with a finer rate ladder, 1 through 25 req/sec, because the interesting cliffs turned out to sit between post 2's coarser steps), then a small-batch high-rate ramp shaped like an SDK fleet. A step passes at under 5% errors and at least 70% of target rate achieved. The read-probe: fill the table to 1M, 10M, 100M, 1B, then 5B rows, and at each level load three real dashboard endpoints per signal, with a level passing when the median is at or under 5 s and nothing hits the 6 s timeout. The 5-second bar is generous on purpose and post 2's discussion of that still applies unchanged.