Return to Answer

###Three weeks later

###Addendum: bad questions broken down by author reputation

Three weeks later

Addendum: bad questions broken down by author reputation

So how will we know if this has had any positive effects? Here's some baseline data on questions from new users:

This deserves a closer look at some point in the future, but I want to drop a note in here to address Iain's concern that any approach targeting only new users - that is, folks who've yet to earn at least 10 reputation points OR a network association bonus - is missing the mark.

More interestingly, only 415 "bad" questions were asked during this time period by folks whose only reputation came from the association bonus, compared to 2630 questions from folks who AT THE TIME THEY POSTED THE QUESTION would've been classified as new users.

So how will we know if this has had any positive effects? Here's some baseline data on questions from new users:

This deserves a closer look at some point in the future, but I want to drop a note in here to address Iain's concern that any approach targeting only new users - that is, folks who've yet to earn at least 10 reputation points OR a network association bonus - is missing the mark.

More interestingly, only 415 "bad" questions were asked during this time period by folks whose only reputation came from the association bonus, compared to 2630 questions from folks who AT THE TIME THEY POSTED THE QUESTION would've been classified as new users.

Don't let me forget to come back in a couple weeks and run this query again...

###Three weeks later

At this point, it's possible to look at data from two weeks following the change to see if there are any noticeable differences. Sadly, there's no dramatic reduction in low-quality questions from new users - but on the bright side, there isn't any reduction in decent questions either:

week          Questions Bad Good PctNewUserQuestionsBad 
------------- --------- --- ---- ---------------------- 
2013-09-30    366       171 195  46.72       
2013-10-07    365       156 209  42.74       
2013-10-14    330       128 202  38.79

That covers the first three weeks after the change (strictly-speaking, the first two days in the first week were prior to the change). Note that the total number of questions asked by new users takes an immediate hit, and... So does the number of good questions. That's not good. By week three, the situation appears to reverse itself somewhat, but there's still a noticeable drop in the # of questions across the board.

Overall, I'm fairly underwhelmed by the effects of this; the only reasonable conclusion I can draw from the results thus far is that it appears to drive away some small % of new users - some of whom would otherwise ask bad questions, others who might ask decent ones. As a sort of blind "hurdle" for new folks, it works - but as a way to increase quality, it's kind of a bust.

For now, I'll leave with this graph of OT closures over the past few weeks: