DuckDB makes things simple

In many analytics workflows, data does not come from a single table or database. Instead, it is often generated and stored as multiple data files such as .txt or .csv, typically split by system, process, or time window.

A common approach is to load each file separately and then combine the data using SQL JOINs or staging tables. However, when files share the same structure and do not represent relational data, JOINs are not required.

DuckDB provides a simple and efficient way to read multiple data files together and treat them as a single dataset.

A Common Example: Log Files

Consider a system that generates application logs as text files:

app_logs_server1.txt

app_logs_server2.txt

Each file contains logs with the same structure:

timestamp,log_level,message

2024-01-01 10:00:00,INFO,Service started

The files represent logs from different servers but follow the same schema. There is no relationship between rows across files — each log entry is independent.

The goal is to analyze logs collectively, not to match records between files.

Reading Multiple Text Files Together

DuckDB allows multiple text files to be read in a single query.

SELECT *

FROM read_csv([

 'app_logs_server1.txt',

 'app_logs_server2.txt'

]);

When executed:

DuckDB reads each file independently

Rows from all files are appended together

The result behaves like one unified dataset

This is functionally equivalent to UNION ALL, without manually writing multiple queries.

Why JOINs Are Not Needed in This Case

JOINs are designed to:

match related records

enforce relationships using keys

In this example:

log entries do not depend on each other

there are no keys to match

the schema is identical across files

Using JOINs would not add value and would unnecessarily complicate the pipeline.

Handling Schema Changes

Over time, log formats may evolve. DuckDB supports column-based alignment to safely handle such changes.

SELECT *

FROM read_csv(

 ['file1.txt', 'file2.txt'],

 union_by_name = true

);

This approach:

matches columns by name

fills missing columns with NULL

prevents ingestion failures due to schema drift

Benefits of This Approach

Simpler SQL – no JOIN or staging logic

Cleaner pipelines – reduced transformation layers

Better performance – no join processing overhead

Easier maintenance – new files can be added without query changes

When This Approach Is Appropriate

This pattern works well when:

files represent the same type of data

records are independent

the objective is analysis, monitoring, or reporting

When JOINs Are Still Required

JOINs should still be used when:

relationships exist between datasets

records must be matched using keys

relational or dimensional analysis is required

Conclusion

DuckDB enables direct and efficient analysis of multiple text files without unnecessary relational logic. By reading files together instead of joining them, data workflows become simpler, more maintainable, and easier to scale.

Choosing the correct approach based on the nature of the data leads to clearer logic and better-performing analytics systems.