I wouldn't consider downvoting *only* because the post doesn't respond but I do downvote if the lack of response means the question is poor and a response could have saved it. In such cases I'll also often cast a close vote.