Why is reputation offered up as bounty taken away even if the bounty is never awarded? Even after offering a bounty, I received only one answer to my question. In fact, the "answer" mostly just asked more questions, and really should be a comment. If I didn't really get any help from posting the question, why should I lose rep?
Partly to prevent gaming the system.
Also consider that one of the features of the bounty is that your question is featured. You could look at this from the perspective that you are paying to have your question be advertised like a sponsored link. Just because you didn't get a good answer doesn't mean your question wasn't listed in the featured tab.
This is covered on the SO blog, in detail. Note the reference to non-refundable rep.
The reputation is taken when the bounty is started, regardless of whether it actually gets awarded. So that reputation is "gone" as soon as the bounty starts.