18

Hi All,

I was reading a Jeff Attwood post on ServerFault here (quite a long one, but not obscenely long) and saw that the last part of the question got cut off:

Cut off text http://www.41085.org/SF-1.jpg

When viewing the revision history, the full text is intact:

Full text intact http://www.41085.org/SF-2.jpg

And no, the text had not been deleted at a later date...

This appears to happen on both Questions and Answers.

1
  • I'm re-opening this specifically to focus on the cleanup aspect: correcting any posts out there that might still be truncated. @Jeff: please feel free to reverse that if it breaks or shows up again on some internal tracking tool.
    – Joel Coehoorn
    Sep 4, 2009 at 16:14

5 Answers 5

13

I think I know what that is. I used a SQL server .NET CLR regex to un-nofollow internal links in posts, and I suspect it has a maximum length based on the SQL data type it is cast to.

Hopefully that was the rare post that was

  1. extra long

  2. contained internal site links

otherwise, there may be others like it out there..

The revision history will contain the actual post in any case, so doing an edit no-op (inserting a space) will restore the truncated content. Not an easy thing to query for, though, because the rendered HTML (in the root post), and the Markdown (in revision history) will always be different lengths.

7
  • 2
    I found a total of 9 answers of mine with their HTML truncated (see my answer below). Would you agree this may likely affect other answers from other dedicated users? If yes, would that be worth of a blog post explaining the situation (and how the users can check from themselves if they are affected, see my answer), or would you say the number is marginal and that nobody cares?
    – VonC
    Aug 24, 2009 at 13:37
  • Also, do you have a more precise idea of the "maximum length based on the SQL data type"? I found empirically 4500 or 5000 for the HTML length of "Body". To have a more precise number would help, in order to request the smallest amount of answers as possible, for the user to check.
    – VonC
    Aug 24, 2009 at 13:40
  • 3
    Just a quick update (in case you missed my previous comments): I have completed my answer: 250 potentially truncated answers are concerned (for StackOverflow), including a very interesting answer of yours about abstraction. Do you you have a status about what you feel adequate to do in term of information for the users affected by this issue?
    – VonC
    Aug 25, 2009 at 7:46
  • final update: 171 users may be affected by this "truncated answers" issue (see my answer below). Is there an internal fix in plan for this, or will the users be notified (for them to edit and fix their message)?
    – VonC
    Aug 27, 2009 at 7:58
  • 1
    Just a "ping", to check if you have a status on the "truncated answers" issue (details are in my answer below).
    – VonC
    Aug 31, 2009 at 7:39
  • Good thing you released the data to the public frequently (and I assume it was all backed up before you ran the script anyway).
    – dlamblin
    Sep 18, 2009 at 2:05
  • As far as I can tell, this hasn't been fixed, as the post here: stackoverflow.com/questions/51094#52640 is getting cut off.
    – notJim
    Mar 17, 2010 at 21:44
7

For those who want to check their lengthy answers, you can do so at statoverflow, with a request similar to:

SELECT ParentID, LENGTH(Body) AS length, Body, LastEditDate FROM posts WHERE PostTypeId = 2 AND OwnerUserId = 6309 AND LENGTH(Body) > 5000 AND Body LIKE '%http://stackoverflow.com%' ORDER BY length DESC LIMIT 200

Replace '6309' by your userid, and 5000 by "the proper limit"

CAVEAT: statoverflow is based on the July dump, not the August public dump, meaning there is a all month of answers, also potentially affected (because edited before August 7th). All the following numbers (of answers truncated or users affected) can be actually higher.

Since my 2 initial truncated answers reported in this (duplicate) MSO question, I did found similar problems (i.e. "answers truncated") with:

They all:

  • contained a link to another stackoverflow question
  • had an HTML body greater than 5000 characters

If I consider this SO answer with only 4480 characters (in the HTML Body), it has not been truncated (even though it does contain a internal link to stackoverflow).
So the limit to consider may be 5000 characters.

Warning: The previous SQL request may miss some other truncated answers which contain internal stackoverflow links, but for which the HTML rendering truncates said link (the string "http://stackoverflow.com" would not be present in "Body")

So you should also check any post of yours with more than 5000 characters, without "stackoverflow.com"

SELECT ParentID, LENGTH(Body) AS length, Body, LastEditDate FROM posts WHERE PostTypeId = 2 AND OwnerUserId = 6309 AND LENGTH(Body) > 5000 AND Body NOT LIKE '%http://stackoverflow.com%' ORDER BY length DESC LIMIT 200

How many potential answers are affected?

SELECT COUNT(*) FROM posts WHERE PostTypeId = 2 AND LENGTH(Body) > 5000 AND Body LIKE '%http://stackoverflow.com%'

There are 252 "potentially truncated" answers in StackOverflow alone.
You can list them with:

SELECT ParentID, LENGTH(Body) AS length, Body, LastEditDate FROM posts WHERE PostTypeId = 2 AND LENGTH(Body) > 5000 AND Body LIKE '%http://stackoverflow.com%' ORDER BY length DESC LIMIT 300

Most of them (not all, because they have recently been edited) are truncated:


Note: the following SQL request seems to indicate 171 users potentially affected by this "truncated message" issue (which includes, again potentially, more than 200 answers)

 SELECT OwnerUserId,COUNT(*) AS n FROM posts WHERE PostTypeId = 2 AND LENGTH(Body) > 5000 AND Body LIKE '%http://stackoverflow.com%' GROUP BY OwnerUserId ORDER BY n DESC
6
  • +1 Well done. The character limit seems to be touchy though (the original bugged post was truncated after ~3200 characters).
    – Ian Elliott
    Aug 25, 2009 at 6:49
  • @Ian: the length of the truncated post varies a lot: 2800, 3000, 3200... but not post originally under 5000 chars seems to have been affected. So my limit for SQL request is 5000.
    – VonC
    Aug 25, 2009 at 7:14
  • 2
    @Ian: by the way, a huge thank you for statoverflow: it is a very, very useful tool to have!
    – VonC
    Aug 25, 2009 at 7:15
  • Hmm... Try using CHAR_Length() rather than LENGHT() to get a more-consistent character count, and limit the query to posts with the lasteditdate < '2009-08-07'
    – Joel Coehoorn
    Sep 4, 2009 at 15:14
  • lasteditdate < '2009-08-07' ? Not with statoverflow: it is still based on the July dump :(. I will try this week end with the last September dump.
    – VonC
    Sep 4, 2009 at 19:53
  • 1
    @VonC: Thanks for the statoverflow query! It helped me find and fix a few of my answers that were affected by this problem.
    – gnovice
    Sep 10, 2009 at 13:50
4

I've been playing with this today, and I found three things:

  • Even if a link is trunctated, the "body" column shows the encoded html and therefor the href="..." part is still included. So I don't think you can have an internal link without the http://stackoverflow.com showing up in the "body" column somewhere. It should be safe to leave that in.
  • The LENGTH() function returns the number of bytes, not the number of characters. I haven't checked it thoroughly yet, but I suspect that using CHAR_LENGTH() will have more consistent results as far as that goes.
  • The bug only effects posts that were last edited on or before August 7th. Anything after then should be okay.

Adding in all that new information, I put together this query:

SELECT 
    ParentID, CHAR_LENGTH(Body) AS length, Body, LastEditDate, CreationDate 
FROM posts 
WHERE PostTypeId = 2 AND CHAR_LENGTH(Body) > 4500 AND Body LIKE '%http://stackoverflow.com%'
AND LastEditDate < '2009-08-08'  AND CreationDate  < cast('2009-08-08' as datetime)
 ORDER BY length DESC LIMIT 200

At this point, that has us down to about 74 records, at least one of which was already corrected. That's small enough we could clean it all up by hand, if I could get another set of eyes on this to check my assumptions and query.

Update
Well that's one big assumption down: statoverflow doesn't have the august dump yet :/

Update 2
I also realized that query only covered answers. I now have the september dataset loaded on my server at home, and I ran an updated query there that should cover both questions and answers. It's still just 96 posts (mixed questions and answers, only question ID shown). That's still small enough to work by hand, if anyone want to take a few here and there. I'm marking this post community wiki so that after someone fixes a few posts they can mark them off the list:

368858
380819
429995
78756
1046016
788084
242996
717725
841231
470942
683273
1154571
553264
1106741
375881
279860
891681
450835
669619
243387
402283
945288
474584
1059574
784417
1078412
823138
1167060
1001569
1181533
47394
379371
219827
359763
146271
181050
814627
657751
189156
688325
387783
904353
749852
922402
1149611
271686
815443
489425
323346
1041266
432154
1214621
696378
631852
535385
368750
543995
1173912
796246
415035
961601
1151658
582412
764710
2

It happened to an answer of mine, and I found this post when I came here to report it. The good news is that editing the post and adding a single space was enough to make it come back to life.

The post was: https://stackoverflow.com/questions/1156572/evaluating-mathematical-expressions-using-lua/1157717#1157717 in case anyone wants to know.

1

I found this on one of my questions, too, but the work-around is pretty simple.

I'm posting now because this seems like something that we should be able to test for and fix: just use the dump to run a query for all posts where the length > N and the last edit is before Aug 7.

1
  • @Joel It seems there may be a little more than one or two answers truncated: see my answer above.
    – VonC
    Aug 26, 2009 at 20:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .