I downvote questions that are poorly written, while making allowances for the fact that English may not be the poster's primary language (which is usually pretty obvious). Some question can most generously be classed as idiotic, while others simply aren't a real question or are written in gibberish.
Rather than downvote questions that don't belong on the site, I prefer to vote to either close or move the question to a more appropriate site.
I normally downvote answers that I know to be technically incorrect, invalid or simply don't address the question that was asked. Less often I will downvote answers that are poorly worded or written in a style that is inappropriate for this site, such as those which are intended to insult or inflame, although the latter are generally flagged for moderator attention instead.
The reason for downvoting is to try and discourage poor questions and answers, just as upvoting is intended to improve the standard and reward those who do so.