How do you know whether a design you shipped actually worked?

Agree the measure before you design, not after launch. Pick a primary behavioral metric tied to the user's goal (task completion, time to first value, error rate), a guardrail metric that catches damage elsewhere, and a qualitative check. Instrument it before release, set a review date, and be willing to call it a failure. A design with no success measure can only be admired.

I will not start a project now without agreeing what would count as it working. Usually that is one behavioral metric tied to the user's goal, one guardrail, and something qualitative. On a scheduling feature the primary was the share of bookings completed without a support contact, the guardrail was double bookings, because the fastest way to make booking quick is to make it careless, and the qualitative check was a handful of sessions two weeks in. The guardrail earned its keep: completion went up nicely and double bookings ticked up too, so we added a confirmation step that cost us a little speed and fixed the real problem. The other thing I insist on is checking the events exist before we ship, because the worst outcome is launching, waiting a month, and finding out nothing was tracked. And I put the review on the calendar four weeks out, with the PM. If it did not work I would rather say so and iterate than let everyone quietly move on to the next thing.