[Informational] What the hell was all that?!?

Status
Not open for further replies.

Dave

Staff member
Seems we had a DB crash. Our search index kept crashing and the db was repairing it over and over. So I contacted the host and we attempted to fix it, but it just wasn't happening. So we reverted back to the backup, truncated the table and am reindexing the search to rebuild the db table.

So we're back and all is well.
 
Oh, so that's what was going on early this morning PDT; when it kept telling me that it couldn't post my responses due to a server error, but kept posting them anyway?
 

Dave

Staff member
Oh, so that's what was going on early this morning PDT; when it kept telling me that it couldn't post my responses due to a server error, but kept posting them anyway?
Yup. The search index was corrupted and was being rebuilt over and over until the server said, "Fuck it!" and just crashed instead. I had a hard time getting into it to repair and when I tried it hung on me. I had to have the host kill the process. Once that was done I emptied the table so that the db was working, rolled back the files to last night's backup, and we were up. Search will be wonky until we finish rebuilding the cache, but it's nearly done.[DOUBLEPOST=1364485591][/DOUBLEPOST]Good morning!
 

Dave

Staff member
Oh God... Where did we go?

Were we dead?
As close as we've ever been! Worst case scenario is we would have lost everything from last night until this morning. We have daily backups of the DB and site so we will never lose more than 24 hours at the most.
 
Might not be a good idea to take anything down, not with the largest DDoS attack in 'net history going on as we speak.
 

Dave

Staff member
Might not be a good idea to take anything down, not with the largest DDoS attack in 'net history going on as we speak.
I seriously thought that's what happened, but then I started getting the SQL errors while still being able to hit the server. So it wasn't a connectivity issue. When I went to the database directly, the table was hung and not moving. Which is a problem. Or was. Hopefully was.
 
The ONE time it's calm at work and I have time to visit HF during my lunch hour, and I get "we're upgrading, check back later" for the whole time. Pfuh!
Also, bunch of meanies giving Dave the 10 different thingies on his post so I can't like it without disturbing the one-of-each order...*twitch*
 
Status
Not open for further replies.
Top