If somebody accidentally clicked on a upvote or downvote, how can they fix that, e.g. fully remove the vote?

It looks like there is no way to remove a vote after the buttons where pressed. Is that on purpose? A better idea might be maybe to allow to change the vote for 1-2 hours until it got fixed.