You can change the vote for five minutes. After that, the vote can only be removed after the voted-upon question or answer has been edited. This is [by design][1]. 

If you accidentally voted, just click the button again to remove the vote, again within the five minute limit. 


  [1]: https://meta.stackexchange.com/questions/5212/what-are-the-limits-on-how-i-can-cast-change-and-retract-votes