You can change the vote for five minutes. After that, the vote can only be removed after the voted-upon question or answer has been edited. This is [by design][1]. 


  [1]: http://meta.stackexchange.com/questions/5212/what-are-the-limits-on-how-i-can-cast-change-and-retract-votes