You need a nightly job to read twenty million rows, transform them and write results. How do you write it so it does not exhaust memory?

Stream rather than materialize: read with a scrollable cursor or keyset paging, set a fetch size so the driver does not buffer everything, and process in chunks with a commit per chunk. If you use JPA, clear the persistence context between chunks or use a stateless session, otherwise every entity stays in the first level cache. Batch the writes with JDBC batching, and make the job restartable from its last committed chunk.

The core rule is that no step holds twenty million of anything. I read with keyset paging on the primary key, or a scrollable cursor with an explicit fetch size, and it is worth knowing that some drivers ignore that and buffer the entire result set unless you also run inside a transaction with autocommit off, which people discover the hard way when the heap fills before any processing starts. Then I process in chunks of a few thousand, write with JDBC batching so it is one round trip per chunk rather than per row, and commit per chunk. With JPA the extra trap is the persistence context: every entity read stays managed, so memory grows and flushes get slower as dirty checking walks more objects. I clear it every chunk or use a stateless session. Restartability matters as much as memory, so the job records the last committed key, which means a failure at row eighteen million resumes rather than starting again. And I throttle it, because a batch job that saturates the database at two in the morning still hits whoever is awake.