You can change the vote for five minutes. After that, the vote can only be removed after the voted-upon question or answer has been edited. This is by design.
If you accidentally voted, just click the button again to remove the vote, again within the five minute limit.