Since the idea seems to have been well-received, I've turned this feature on. Feel free to log out and verify that it works...
So how will we know if this has had any positive effects? Here's some baseline data on questions from [new users][1]:
week Questions Bad Good PctNewUserQuestionsBad
------------- --------- --- ---- ----------------------
2013-07-22 364 147 217 40.38
2013-07-29 382 179 203 46.86
2013-08-05 352 163 189 46.31
2013-08-12 361 166 195 45.98
2013-08-19 400 196 204 49
2013-08-26 432 178 254 41.2
2013-09-02 453 189 264 41.72
2013-09-09 389 171 218 43.96
2013-09-16 410 189 221 46.1
2013-09-23 386 159 227 41.19
![New user question quality graph][2]
...Where again, a "bad" question is one that was closed, deleted (by someone other than the author) or down-voted below 0. If we can get the % to drop significantly without seriously hurting the number of *good* questions, we'll call this a success.
### Three weeks later
At this point, it's possible to look at data from two weeks following the change to see if there are any noticeable differences. Sadly, there's no dramatic reduction in low-quality questions from new users - but on the bright side, there isn't any reduction in decent questions either:
week Questions Bad Good PctNewUserQuestionsBad
------------- --------- --- ---- ----------------------
2013-09-30 366 171 195 46.72
2013-10-07 365 156 209 42.74
2013-10-14 330 128 202 38.79
That covers the first three weeks after the change (strictly-speaking, the first two days in the first week were prior to the change). Note that the total number of questions asked by new users takes an immediate hit, and... So does the number of good questions. That's *not* good. By week three, the situation appears to reverse itself somewhat, but there's still a noticeable drop in the # of questions across the board.
![New user question quality graph (after the change)][3]
Overall, I'm fairly underwhelmed by the effects of this; the only reasonable conclusion I can draw from the results thus far is that it appears to drive away some small % of new users - some of whom would otherwise ask bad questions, others who might ask decent ones. As a sort of blind "hurdle" for new folks, it works - but as a way to increase quality, it's kind of a bust.
For now, I'll leave with this graph of OT closures over the past few weeks:
![Questions closed as off-topic, by week and by reason][4]
---
### Addendum: bad questions broken down by author reputation
This deserves a closer look at some point in the future, but I want to drop a note in here to address Iain's concern that any approach targeting only [new users][5] - that is, folks who've yet to earn at least 10 reputation points OR a network association bonus - is missing the mark.
Using the same criteria for "bad" as above, I divided all of the bad questions asked during the past 90 days into buckets according to the reputation of the author *at the time they were asked* (questions are bucketed by rounding author rep down to the nearest multiple of 100):
Reputation rounded down to the nearest 100 BadQuestions
------------------------------------------ ------------
0 2630
100 926
200 70
300 45
400 26
500 11
600 15
700 19
800 6
900 5
1000 9
1100 10
1200 2
1300 3
1400 2
1500 3
1800 1
1900 2
2200 1
2400 5
2500 5
2800 1
3100 1
3500 1
4000 1
4100 1
4300 1
31800 1
More interestingly, only 415 "bad" questions were asked during this time period by folks whose only reputation came from the association bonus, compared to 2630 questions from folks who AT THE TIME THEY POSTED THE QUESTION would've been classified as [new users][6].
Whether or not this is particular change is a useful strategy, I'm at least confident that we are targeting the right group of users.
[1]: https://serverfault.com/help/privileges/new-user
[2]: https://i.sstatic.net/HfApg.png
[3]: https://i.sstatic.net/9KaW3.png
[4]: https://i.sstatic.net/afOq8.png "Questions closed as off-topic, by week and by reason"
[5]: https://serverfault.com/help/privileges/new-user
[6]: https://serverfault.com/help/privileges/new-user