What problem do table formats like Iceberg and Delta Lake solve that plain Parquet on object storage does not?

Plain Parquet files in a bucket give you no transactions, so a reader can see a half written directory and a failed job leaves partial data. Table formats add a metadata layer with atomic commits and snapshot isolation, so writes appear all at once. They also bring schema evolution, row level updates and deletes, time travel to previous snapshots, and file pruning through statistics rather than directory listing.

With plain Parquet the table is really just a directory convention, so there is no atomicity. If a job dies halfway, readers see partial data, and there is no way to update or delete a single row without rewriting whole partitions by hand. Table formats fix that with a metadata layer. A commit swaps a pointer to a new snapshot atomically, so readers either see the old state or the new one, never a mixture, and you get time travel almost for free because old snapshots still exist. That last part saved us once when a bad transform overwrote a mart and we recovered by querying the previous snapshot rather than restoring from backups. The other thing I like about Iceberg specifically is hidden partitioning, where the table tracks the partition transform itself, so an analyst filtering on a timestamp gets pruning without knowing the layout. The cost is maintenance: you need scheduled compaction of small files and snapshot expiry, or metadata grows without bound.