This is done by using the same /leave flow as you would use if you had
already accepted the invite and wanted to leave.