Skill of the Month – DuckDB

What is DuckDB? 

DuckDB is an open-source, in-process analytical database designed to run directly inside your application – no server, no setup, no separate infrastructure to manage. It’s built for OLAP (online analytical processing) workloads, meaning it’s optimised for querying and analysing large datasets rather than handling high volumes of transactional writes. Think of it as SQLite, but purpose-built for analytics.

What makes it particularly compelling is that it runs entirely in-process, which means you can embed it inside a Python script, a data pipeline, or an application and run complex analytical queries against local files – CSV, Parquet, JSON – without spinning up a database server. It’s become a go-to tool for data engineers and analysts who need fast, flexible querying without the overhead of a full data warehouse.

What are some things to know about DuckDB?

• Columnar storage, serious speed – DuckDB uses a columnar execution engine, which means it processes analytical queries dramatically faster than row-based databases like PostgreSQL or SQLite when working with large datasets. Aggregations, filters, and scans across millions of rows are where it genuinely shines.

• Runs anywhere, no server required – because it’s in-process, DuckDB runs inside your Python environment, your notebook, your CLI, or your application with a single import. There’s no database server to configure, maintain, or scale – which makes it genuinely frictionless to adopt.

• Reads files directly, including remote ones – DuckDB can query Parquet, CSV, and JSON files directly, including those stored in S3 or other cloud storage, without loading them into a database first. For data engineers working with file-based pipelines, that’s a significant time saver.

Why learn DuckDB?

The data tooling landscape has been shifting away from heavy, always-on infrastructure towards leaner, more composable tools – and DuckDB sits right at the centre of that shift. It’s become widely adopted in the data engineering and analytics communities, and integrates cleanly with Python, dbt, Pandas, and Arrow, which means it slots naturally into stacks that are already common in data teams.

For engineers and analysts, it’s a fast skill to pick up with immediate practical value. The ability to run SQL analytics locally at speed – without standing up a warehouse – is useful in a surprisingly wide range of contexts, from exploratory analysis to production pipelines. As the modern data stack continues to evolve, DuckDB is well positioned to remain a relevant and frequently reached-for tool.

Use Cases for DuckDB

• Local exploratory data analysis on large CSV or Parquet files without a database server

• Lightweight ETL and data transformation pipelines as an alternative to spinning up a full warehouse

• Embedded analytics inside Python applications and data science notebooks

• Querying files stored in S3 or cloud storage directly with SQL

• Replacing Pandas for heavy aggregation and filtering workloads where performance matters

• Powering analytical features in applications without adding database infrastructure

Topic of the Month

The Case for Lightweight Analytics Infrastructure

For years, the assumption in data engineering was that serious analytical workloads required serious infrastructure – a cloud data warehouse, a managed cluster, a team to run it. That assumption made sense when datasets were large, teams were bigger, and the cost of standing up infrastructure was just part of the job. But the tooling landscape has shifted considerably, and DuckDB is one of the clearest examples of what’s changed.

The appeal isn’t just that DuckDB is fast – though it is. It’s that it removes an entire category of infrastructure decision from the process. When you can query a hundred-million-row Parquet file in seconds from inside a Python script, the question of whether you need a data warehouse for a given task becomes a genuine one rather than a foregone conclusion. For smaller teams, early-stage data platforms, and ad hoc analytical work, that flexibility is genuinely valuable.

The broader trend DuckDB represents is worth paying attention to. The modern data stack is becoming more modular, more local-first where appropriate, and more oriented around composable tools that do one thing well. Engineers who understand not just how to use these tools but when to reach for them – and when not to – are increasingly the ones adding the most value in data teams. DuckDB is a practical and well-timed skill to have in that context.

For our newest jobs, please visit our Jobs Page!