Should folks have to click through an interstitial page to ask questions on Server Fault?

Question

A few weeks back, Jay and I had a nice conversation with voretaq7 and Chopper3 where we discussed and debated the various items raised by the community here in http://meta.serverfault.com/questions/5744/questions-to-ask-the-se-staff-regarding-the-future-of-serverfault. Both of these fine gentlemen were extremely helpful in helping us to understand some of the problems faced by the folks here, and we came up with a few different ideas for improvement.

One of the biggest problems facing this site are the number of poorly asked and straight-up off-topic questions asked here on a daily basis. While y'all are pretty good about closing and migrating these, it's demoralizing to see so many of them flooding in - not to mention discouraging for the folks asking them. We all liked the idea presented in Question #7:

Question #7 - What could be done to ensure that new users can't ignore notices regarding the site's objectives and our expectations for new users? Could we force new users to agree that they've read these before asking their first N questions, maybe even answers? Perhaps even warn them that they may see some negativity if they ignore the requirements?

When Stack Overflow was facing a similar problem a few years back, a very similar solution was implemented:

Every new Stack Overflow user with <= 10 reputation is now presented with a mandatory “How To Ask” page that they must click through before asking their first question. The text on this page is a heavily edited subset of Google’s excellent Tips for Getting Help.

This is how that page would look on Server Fault if it was enabled today. How do you force people to read something like this? Well, you can't - but if they don't read it carefully, they're gonna miss the bit that tells them they have to click that checkbox in order for the "proceed" link to do anything - that is, anything other than reload the advice page and start them back over at the top. There's also a handy search field above the fold, for folks more inclined to attack a problem by typing into any empty entryfield than to search.

But... Right now it doesn't do much to directly attack the problem of folks completely misunderstanding what sort of questions this site is for. Sure, there's a link to that, but that link is on the Ask page itself - if you're not inclined to click through already, you're probably not going to see it. So how about we modify the introduction on that page a bit, make it directly address the most common point of confusion:

To improve the chances of your question staying open and getting an answer, make sure that it is about professional systems administration. If your question concerns personal equipment, try asking on Super User instead. Here are some additional tips:

That text would appear in this context:

To be clear, this is in no way a panacea; it certainly didn't eliminate low-quality questions on Stack Overflow, and I don't expect it will here either. But every little bit helps, and in particular it is always useful to catch misguided but well-intentioned people sooner rather than later. Rather than just adding another short hurdle for them to leap over, this offers at least a chance of redirecting their efforts in a more constructive direction.

I think it's worth a try. What about you?

Answer 1

Drawing on my knowledge of rapid-growth startups, and so on, my favourite phrase is "Try it, if it doesn't work, we'll try something else".

Seems like a good fit for this scenario.

Answer 2

This wasn't already enabled? Why the hell not? Turn it on now!

Answer 3

Seems sensible - We may want to wordsmith the language a little more (A much-shortened version of How can I ask better questions on Server Fault?) but I'm in favor of enabling this now and refining the language later (we all know nobody reads the manual anyway :-)

Answer 4

Since the idea seems to have been well-received, I've turned this feature on. Feel free to log out and verify that it works...

So how will we know if this has had any positive effects? Here's some baseline data on questions from new users:

week          Questions Bad Good PctNewUserQuestionsBad 
------------- --------- --- ---- ---------------------- 
2013-07-22    364       147 217  40.38                  
2013-07-29    382       179 203  46.86                  
2013-08-05    352       163 189  46.31                  
2013-08-12    361       166 195  45.98                  
2013-08-19    400       196 204  49                     
2013-08-26    432       178 254  41.2                   
2013-09-02    453       189 264  41.72                  
2013-09-09    389       171 218  43.96                  
2013-09-16    410       189 221  46.1                   
2013-09-23    386       159 227  41.19                  

...Where again, a "bad" question is one that was closed, deleted (by someone other than the author) or down-voted below 0. If we can get the % to drop significantly without seriously hurting the number of good questions, we'll call this a success.

Three weeks later

At this point, it's possible to look at data from two weeks following the change to see if there are any noticeable differences. Sadly, there's no dramatic reduction in low-quality questions from new users - but on the bright side, there isn't any reduction in decent questions either:

week          Questions Bad Good PctNewUserQuestionsBad 
------------- --------- --- ---- ---------------------- 
2013-09-30    366       171 195  46.72       
2013-10-07    365       156 209  42.74       
2013-10-14    330       128 202  38.79

That covers the first three weeks after the change (strictly-speaking, the first two days in the first week were prior to the change). Note that the total number of questions asked by new users takes an immediate hit, and... So does the number of good questions. That's not good. By week three, the situation appears to reverse itself somewhat, but there's still a noticeable drop in the # of questions across the board.

Overall, I'm fairly underwhelmed by the effects of this; the only reasonable conclusion I can draw from the results thus far is that it appears to drive away some small % of new users - some of whom would otherwise ask bad questions, others who might ask decent ones. As a sort of blind "hurdle" for new folks, it works - but as a way to increase quality, it's kind of a bust.

For now, I'll leave with this graph of OT closures over the past few weeks:

---

Addendum: bad questions broken down by author reputation

This deserves a closer look at some point in the future, but I want to drop a note in here to address Iain's concern that any approach targeting only new users - that is, folks who've yet to earn at least 10 reputation points OR a network association bonus - is missing the mark.

Using the same criteria for "bad" as above, I divided all of the bad questions asked during the past 90 days into buckets according to the reputation of the author at the time they were asked (questions are bucketed by rounding author rep down to the nearest multiple of 100):

Reputation rounded down to the nearest 100 BadQuestions 
------------------------------------------ ------------ 
0                                          2630         
100                                        926          
200                                        70           
300                                        45           
400                                        26           
500                                        11           
600                                        15           
700                                        19           
800                                        6            
900                                        5            
1000                                       9            
1100                                       10           
1200                                       2            
1300                                       3            
1400                                       2            
1500                                       3            
1800                                       1            
1900                                       2            
2200                                       1            
2400                                       5            
2500                                       5            
2800                                       1            
3100                                       1            
3500                                       1            
4000                                       1            
4100                                       1            
4300                                       1            
31800                                      1            

More interestingly, only 415 "bad" questions were asked during this time period by folks whose only reputation came from the association bonus, compared to 2630 questions from folks who AT THE TIME THEY POSTED THE QUESTION would've been classified as new users.

Whether or not this is particular change is a useful strategy, I'm at least confident that we are targeting the right group of users.

Answer 5

I think it is a good idea to implement it but I'm just throwing this out there as a possible suggestion.

Knowing that folks skim and not really read something that long, let alone click the links embedded on that page, I think that this part:

We’d love to help you, but the reality is that not every question gets answered. To improve your chances, here are some tips:

Should be modified to something like:

We’d love to help you, but the reality is that not every question is a good fit for our site. To help ensure your question will NOT get closed, here are some tips:

and should somehow stand out a little more.

I just think the issue overall isn't that a question is formatted poorly and won't get answered, but rather that the first time user is asking questions that call for immediate downvotes and close reasons.