Running DuckDB?

DuckDB queries slow or extensions refusing to load? We fix it.

DuckDB fits on a laptop and still runs analytics queries over gigabytes of Parquet without asking permission. It shows up in notebooks, in CI pipelines, inside Lambda functions and as the embedded engine behind commercial products. When a query that was fast yesterday crawls today, a httpfs extension refuses to load on the production image, or a vector-type cast breaks a Python pipeline, we work through the PRAGMA output, the catalog and the storage layer until it is fast and boring again.

  • Since 2015Fixing technology
  • FreeEstimates
  • US teamReal engineers

duckdb-query.log

still slow

What it is

DuckDB at a glance

DuckDB is an in-process analytical database designed by CWI Amsterdam and now developed by DuckDB Labs. It is often described as the SQLite for analytics: a single binary with no server process, a rich SQL dialect, and a column-oriented execution engine optimized for OLAP workloads over Parquet, CSV, JSON and Arrow.

By 2026 DuckDB downloads cross 25 million a month across pip, npm, Rust, Go, C, R, Java and the native CLI. It is the engine inside many modern data tools, the default local engine for notebook analytics, and increasingly the way CI pipelines validate data without paying for a warehouse. Its extension system brings geospatial, JSON path, httpfs, iceberg and full-text search into the same binary.

Type
Embeddable OLAP database
First released
2019
License
MIT, permissive
Maintained by
DuckDB Labs and CWI Amsterdam
Typically used for
Local analytics on Parquet and CSV, notebook queries, data CI, embedded analytics inside applications, lakehouse table access

The honest version

DuckDB pros and cons

No technology is all upside. This is where DuckDB earns its place and where it tends to cause trouble.

Where DuckDB is strong

  • In-process, no server to runYou import duckdb from your language of choice and get an analytical engine. No ports, no service, no connection pool, no credentials. For notebooks and scripts this changes the shape of the whole workflow.
  • Column store with vectorized executionThe engine reads columnar data and processes it in vectors, which makes OLAP queries orders of magnitude faster than running the same SQL in SQLite or Postgres on the same hardware.
  • First-class Parquet, CSV, JSON and ArrowSELECT from s3://bucket/data.parquet works out of the box with the httpfs extension, no ETL step required. The Arrow integration means zero-copy between DuckDB and Polars, pandas or pyarrow.
  • Rich SQL dialectWindow functions, qualify, exclude, pivot, list and struct types, lateral joins, recursive CTEs and user-defined functions make it a serious analytical SQL engine rather than a toy.
  • Portable binaryThe same database works on macOS, Linux and Windows, in Python, R, Node, Rust and Go, and the storage format is stable. A file produced on a laptop opens the same in a Lambda.
  • Fast enough to replace a warehouse for many jobsFor gigabyte to low-terabyte analytical workloads on a single machine, DuckDB is often faster than a remote Snowflake or BigQuery query, with no idle cost and no network round trip.

Where DuckDB bites

  • Single-node by designDuckDB is embarrassingly fast on one machine and does not try to be a cluster. Workloads that outgrow a big VM need a different engine, or MotherDuck for the hybrid cloud answer.
  • Concurrency model is read-many, write-oneMultiple readers are fine, but writers take a file lock. A pipeline that assumes a Postgres-style concurrent write model will fight DuckDB immediately.
  • Extensions have load-time frictionhttpfs, spatial, iceberg and others are not in the base binary and need to be installed and loaded. In sandboxed environments like Lambda or restricted CI runners, that step is a common source of surprises.
  • Type system edges can biteImplicit casts between VARCHAR and numeric types, list versus array semantics, and timestamp timezone handling all have sharp edges when moving data between pandas, Arrow and DuckDB.
  • Storage format is evolvingNative .duckdb files have had format changes between major versions that required a conversion step. The upgrade is well documented; the surprise is still a real event.
  • Memory limit requires thoughtThe engine will happily spill to disk, but the default memory limit may be too generous on a shared box or too stingy for the query you want. Tuning is manual.

Sound familiar?

Common DuckDB problems we fix

The DuckDB problems that bring people to us, and what you have once each one is fixed.

Extension load failed: httpfs is not installed

S3 or HTTP reads fail in production

The httpfs extension is present in local development but was never installed in the deployed image, or the autoload path is blocked. Reads from s3:// and https:// fail before the query runs.

Extensions baked into the image

IO Error: Could not set lock on file

Two writers fight for the same database

A pipeline assumed Postgres-style concurrency and started a second writer. DuckDB took the file lock and the second process blocks or errors depending on mode.

Writes serialized or routed

Out of Memory Error: could not allocate 2 GB

A query blows past the memory budget

A group by over a very wide table with high-cardinality keys, or a join that materialized more than the configured memory limit allowed. Spilling or a lower parallelism is usually the fix.

Query fits the memory budget

Conversion Error: Could not convert string 'NaN'

Parquet or CSV types crash the pipeline

A CSV with mixed numeric and NaN values parsed as DOUBLE, or a Parquet file whose schema has drifted between batches. Explicit casts or a safe_cast helper close the gap.

Type conversions explicit and safe

HTTP Error: HTTP GET error (403)

S3 Parquet scan returns 403 or 404

A bucket policy that forbids anonymous access, an expired AWS credential, or a path that references a region the client is not configured for. The query never reaches the engine.

S3 credentials and regions aligned

Binder Error: No function matches the given name

Function signatures changed between versions

A query that worked on DuckDB 0.9 uses a function that was renamed or whose argument order changed in 1.x. The error names the function but not the version gap.

Queries pinned to a DuckDB version

memory limit reached, spilling to disk

Slow query because everything spills

A memory limit too low for the query shape, forcing every intermediate to disk and multiplying the runtime. Raising the limit or restructuring the query gets it back in RAM.

Hot path runs in memory

TransactionContext Error: cannot commit

A long transaction aborts and loses work

An error during a long COPY or INSERT INTO ... SELECT aborted the transaction, and the next commit refuses because the context is poisoned. The pipeline has to reset cleanly before retrying.

Transactions short and retryable

What we do

DuckDB help, from extension loading to warehouse replacement

Call us for one slow Parquet scan, or hand us the whole analytics stack.

Rescue
Extensions that refuse to load, S3 reads returning 403, or queries that have gotten mysteriously slow. We diagnose from EXPLAIN ANALYZE and the storage layer first.
Performance tuning
EXPLAIN ANALYZE output read against the storage layout, Parquet row group sizing, predicate pushdown and memory limit adjustments against your real workload.
Pipeline integration
Python, Rust, Node or Go pipelines that call DuckDB cleanly, with Arrow zero-copy where it helps and explicit type handling where it matters.
Warehouse offload
Workloads that do not need a Snowflake or BigQuery query credit, moved to DuckDB in a Lambda or an EC2 box, often for a fraction of the cost.
MotherDuck migration
Projects that have outgrown single-node DuckDB extended to MotherDuck for shared cloud storage without rewriting the SQL.
Managed Tech
A standing review of query cost, extension health, DuckDB version upgrades and the health of the Parquet or lakehouse layout the queries run against.

How it works

From first message to fixed

  1. Look Free estimate

    Tell us what's going wrong. We look at it, then tell you plainly what's wrong and what it will cost.

  2. Fix

    We fix the cause, not the symptom. If something is down, getting it back up comes first.

  3. Harden

    Backups, monitoring, updates and access, so the same problem can't quietly come back.

  4. Hand back or stay on

    You get a plain-language write-up. Or we stay on for hosting, upkeep and new work.

Often found together

Running DuckDB alongside something else?

We work on the whole stack, so the problem can be wherever it actually is.

Questions

DuckDB questions

Something else? Call+1 737 727 7327

When should we pick DuckDB over Postgres or SQLite?

DuckDB wins when the workload is analytical: large scans, aggregates, window functions, and read-mostly access to Parquet or CSV. SQLite is still the answer for transactional small-record workloads on one machine, and Postgres is still the answer when many services need concurrent writes. We often use DuckDB alongside one of them rather than in place of it.

Can DuckDB really replace a warehouse?

For workloads in the gigabyte to low-terabyte range that run on a schedule rather than a fleet of ad hoc users, yes. Many pipelines that pay a warehouse per query would run cheaper on DuckDB inside a Lambda or a scheduled container. For interactive analytics across a company, you still want a warehouse or MotherDuck.

Why do our DuckDB queries slow down over time?

Usually either the Parquet files have grown and no longer have useful row group statistics, or an extension did not autoload and a scan fell back to a slower path, or the memory limit is too low and intermediates are spilling. EXPLAIN ANALYZE tells the story quickly.

How do we deploy DuckDB with extensions on Lambda?

Bake the extension binaries into the deployment package, set the extension directory to a path the runtime can read, and pre-load rather than autoload at query time. The cold start cost is a one-time hit rather than a per-query one.

DuckDB or MotherDuck for a small team?

Start with DuckDB on the devs' machines and in CI. Move to MotherDuck when you need shared datasets across the team, cloud storage, or queries that span local and hosted data. The SQL is identical, so the migration is mostly a connection string change.

How much does DuckDB help cost?

The first conversation is free. Send a sample of the query plan or the error, and we send a price before any work is booked.

Free estimate

DuckDB giving you trouble?

Tell us what you're running and what's going wrong. A person reads it and comes back with an estimate.

  • No charge to find out what it will take
  • If it's urgent, say so

Replies come from a person. By sending, you authorize us to email you back. No marketing.