How would you process a forty gigabyte CSV that will not fit in memory?

Stream it. Open the file and iterate line by line, or use csv.reader over the file object, so only one row is resident at a time, and push each record through generator stages rather than building intermediate lists. Batch writes to the destination in chunks of a few thousand. If the work is CPU bound, split by byte ranges across processes.

Nothing gets loaded whole. A file object is already an iterator over lines, so I read row by row with csv.reader and chain generator stages for parsing, filtering, and transforming, which keeps peak memory at roughly one record regardless of file size. Writes get batched, typically a few thousand rows per insert, because per row round trips to Postgres dominate everything else. Since forty gigabytes takes a while, I make it restartable: track the byte offset or the last processed key in a small state file, log progress every hundred thousand rows, and make the write idempotent with an upsert so a rerun cannot duplicate. If parsing turns out to be the bottleneck rather than I/O, I split the file by byte offsets and hand ranges to a process pool, taking care to align on line boundaries. Honestly, though, my first suggestion is often to skip Python for the load entirely and use the database bulk copy, then do the transformation in SQL, because that is usually an order of magnitude faster.