Tweet
I'm very sorry to report that the old forum - 9 years of progress galleries, press citations, grief postings, and Q&A - was destroyed by GoDaddy's incompetence and my naiveté.
As background, note that the entire contents of a forum like this are stored in a single database file. The database is comprised of over 100 related tables. So for example in a POSTS table it might include each posting's date and details of the post, but instead of the user's name it just includes the user's ID number, and the software looks up the user's name, avatar, and details from a separate USERS table to construct the posting as we view it.
Also of note, I have always made it a priority to ensure the forum would survive even if I was no longer around. I kept the hosting and domain registry paid up for 5 years into the future and had some users set up with administrator privilege so if I got hit by a bus they would have time to figure out what would keep the forum community alive.
So in August 2015 I got a message from GoDaddy saying they would soon move my forum hosting account to a more modern server, and that likely I would be unaffected unless my account hosts secondary domains or subdomains (it doesn't). They said certain applications might need a pointer to the new host. I didn't think anything of it.
On August 27th the forum stopped working. A couple days prior we had been getting sporadic "vBulletin Database error" messages (like many times over the years) but after a day of the forum not snapping back to life I assumed our forum database finally got corrupted somehow. I called GoDaddy and asked them to restore a backed up copy of the database from the 26th when it was working.
That didn't work, so the following Monday morning I contacted the forum software experts at vBulletin. The guy who helps me is in the UK, so every question and answer cycle takes a couple days. The vBulletin guy said he was able to verify that our config file pointed correctly to the new server, and that he could see that our database was incomplete. It's supposed to have tables with names starting from A through W, and our alphabetized list ended in the Ps. We lacked PICS, POSTS, USERS, etc, so no use could be made of the file on our server.
So I contacted another GoDaddy support person and explained about the missing tables and we decided to revert back another whole week to make sure we got a database file that was complete. It didn't work but I couldn't prove why until I talked again to vBulletin.
So several days later I was talking to the third or fourth GoDaddy person who finally seemed to know something. This guy said my database file was about twice as large as my hosting plan allows (Who knew? Must have been in the fine print.). If it stayed like that I would have eventually gotten a notice warning me to pare down the file or buy a different plan. I asked for a quote on the next plan. Twice what I was paying would have still been a bargain to me.
What else the knowledgeable guy said was that there was probably never anything wrong with our database file, but the new servers were designed to only accept databases up to 1 gigabyte. Once that much was pushed into the server the rest would be ignored. He said what we had seen to-date was a predictable result of the too-large database and that the prior help people should have known that.
Most importantly, the knowledgeable guy said that if we had just left the database alone it could have been accessed with a proper tweaking of a config file, but since we UN-mounted the too-large database file there was no way to push a file that large back into the server. I said that the 1 gigabyte limit simply had to be a choice they had made, and that they could surely over-ride it in order to force my file back in. I explained that I had no way to prune the database file size EXCEPT by using the forum software's online pruning tools. Those let you purge users and their pics if users haven't logged in since a certain data for example. He agreed to put in a ticket with his higher-ups to get my database re-mounted so I could reduce it.
GoDaddy left that ticket open for a week. When they closed it the forum still didn't have a full database file. It's like they did nothing. I don't know.
BTW, on September 10th I got an email confirming the move to the new server was complete, which belies what the "knowledgeable" guy said about the timing of the switch to the new server.
I called back to beg GoDaddy to just mail me the earliest raw database file on a DVD-ROM. I got some run-around and then they finally declined to do that.
Anyway, each time GoDaddy tried something I had to go back to the vBulletin guy to verify that the present file was unworkable. I was also at this time investigating ways to get the large database file into a tool that could edit it smaller locally in case that could help. We transmitted the whole huge thing to vBulletin several ways in order for them to test it.
Around September 25th I finally got an off-shift GoDaddy support person who seemed eager to take our cause to the highest levels and not rest until we were running.
All this time I had been personally missing my forum and suffering for the inconvenience I was surely causing the members who must also be missing it, but I knew at least backups existed at our host GoDaddy, and OK, if it meant reverting back a whole month or whatever there was always going to be some way to at least get back to where we were at some point earlier in the summer. I confess it's not like I was hounding people 24/7 on this. At times - due to family needs and business and various reasons - a day or two would pass before I pursued a resolution to the most recent confounder.
So in the first week of October is when I finally got word from vBulletin that none of the 4 different existing versions of the huge database file was usable. I also found out from a GoDaddy person that
BACKUPS ARE DONE ONLY DAILY AND ARE HELD FOR ONLY 30 DAYS.
What we needed was ANY full database file from before August 27th, and now there was zero chance we would ever see it. GoDaddy ate our forum.
Completely apart from my astonishment at how GoDaddy had failed to protect data that was part of these active help tickets, and my astonishment that they don't keep monthly or yearly backups (Who knew? Must have been in the fine print) I was left speechless at how GoDaddy never did know the real manner - technically - in which they had robbed us.
As I see it now, they NEVER had the whole database installed on the new server and just failed to compare the prior installation to the new one before switching off the old server.
I just can't put into words the sleepless anguish I felt as this reality set in. I liken it to realizing that you backed over your child in the driveway because a trivial dashboard rear-obstacle-sensor light was burned out. Knowing that I could have somehow avoided this by being more knowledgeable and hands-on is still just eating me. But I will say it is not/was never about keeping local backups of the forum. Nobody transmits a 2 gigabyte file regularly when they know their host is making backups. Indeed GoDaddy charged $80 each time I wanted to get at the older versions of the file. It's not something they expect people to do.
So now I've fired GoDaddy and contracted with vBulletin directly. They are hosting the forum and they are the real experts who know how to protect forum database content. Our premium plan includes them keeping us up-to-date with the latest most-hacker-proof version of the forum system.
But a few settings and options we enjoyed in version 3 are just not part of version 5, and I would actually consider going back to version 4 and a self-hosted plan to get those features back, once our current contract expires. I am assured that no content would be lost.
So at this point I can only apologize to you for being reckless with your forum and trusting GoDaddy with it. I am very sorry to the folks who worked so hard to build our beautiful dearly departed Press Room which was a true research resource. I am very sorry to our members who were kind enough to share their progress gallery with the community to inspire men who had not yet committed to help themselves by restoring. In some cases the photos were stored only online for in-home privacy reasons and that just kills me. I'm sorry to all the folks who made the forum better in so many ways, by answering questions, by posting about their grief, by helping us debug forum quirks, by running special interest groups.
We're already on the way to being great again, and it starts with what you contribute right now.
Thanks.
As background, note that the entire contents of a forum like this are stored in a single database file. The database is comprised of over 100 related tables. So for example in a POSTS table it might include each posting's date and details of the post, but instead of the user's name it just includes the user's ID number, and the software looks up the user's name, avatar, and details from a separate USERS table to construct the posting as we view it.
Also of note, I have always made it a priority to ensure the forum would survive even if I was no longer around. I kept the hosting and domain registry paid up for 5 years into the future and had some users set up with administrator privilege so if I got hit by a bus they would have time to figure out what would keep the forum community alive.
So in August 2015 I got a message from GoDaddy saying they would soon move my forum hosting account to a more modern server, and that likely I would be unaffected unless my account hosts secondary domains or subdomains (it doesn't). They said certain applications might need a pointer to the new host. I didn't think anything of it.
On August 27th the forum stopped working. A couple days prior we had been getting sporadic "vBulletin Database error" messages (like many times over the years) but after a day of the forum not snapping back to life I assumed our forum database finally got corrupted somehow. I called GoDaddy and asked them to restore a backed up copy of the database from the 26th when it was working.
That didn't work, so the following Monday morning I contacted the forum software experts at vBulletin. The guy who helps me is in the UK, so every question and answer cycle takes a couple days. The vBulletin guy said he was able to verify that our config file pointed correctly to the new server, and that he could see that our database was incomplete. It's supposed to have tables with names starting from A through W, and our alphabetized list ended in the Ps. We lacked PICS, POSTS, USERS, etc, so no use could be made of the file on our server.
So I contacted another GoDaddy support person and explained about the missing tables and we decided to revert back another whole week to make sure we got a database file that was complete. It didn't work but I couldn't prove why until I talked again to vBulletin.
So several days later I was talking to the third or fourth GoDaddy person who finally seemed to know something. This guy said my database file was about twice as large as my hosting plan allows (Who knew? Must have been in the fine print.). If it stayed like that I would have eventually gotten a notice warning me to pare down the file or buy a different plan. I asked for a quote on the next plan. Twice what I was paying would have still been a bargain to me.
What else the knowledgeable guy said was that there was probably never anything wrong with our database file, but the new servers were designed to only accept databases up to 1 gigabyte. Once that much was pushed into the server the rest would be ignored. He said what we had seen to-date was a predictable result of the too-large database and that the prior help people should have known that.
Most importantly, the knowledgeable guy said that if we had just left the database alone it could have been accessed with a proper tweaking of a config file, but since we UN-mounted the too-large database file there was no way to push a file that large back into the server. I said that the 1 gigabyte limit simply had to be a choice they had made, and that they could surely over-ride it in order to force my file back in. I explained that I had no way to prune the database file size EXCEPT by using the forum software's online pruning tools. Those let you purge users and their pics if users haven't logged in since a certain data for example. He agreed to put in a ticket with his higher-ups to get my database re-mounted so I could reduce it.
GoDaddy left that ticket open for a week. When they closed it the forum still didn't have a full database file. It's like they did nothing. I don't know.
BTW, on September 10th I got an email confirming the move to the new server was complete, which belies what the "knowledgeable" guy said about the timing of the switch to the new server.
I called back to beg GoDaddy to just mail me the earliest raw database file on a DVD-ROM. I got some run-around and then they finally declined to do that.
Anyway, each time GoDaddy tried something I had to go back to the vBulletin guy to verify that the present file was unworkable. I was also at this time investigating ways to get the large database file into a tool that could edit it smaller locally in case that could help. We transmitted the whole huge thing to vBulletin several ways in order for them to test it.
Around September 25th I finally got an off-shift GoDaddy support person who seemed eager to take our cause to the highest levels and not rest until we were running.
All this time I had been personally missing my forum and suffering for the inconvenience I was surely causing the members who must also be missing it, but I knew at least backups existed at our host GoDaddy, and OK, if it meant reverting back a whole month or whatever there was always going to be some way to at least get back to where we were at some point earlier in the summer. I confess it's not like I was hounding people 24/7 on this. At times - due to family needs and business and various reasons - a day or two would pass before I pursued a resolution to the most recent confounder.
So in the first week of October is when I finally got word from vBulletin that none of the 4 different existing versions of the huge database file was usable. I also found out from a GoDaddy person that
BACKUPS ARE DONE ONLY DAILY AND ARE HELD FOR ONLY 30 DAYS.
What we needed was ANY full database file from before August 27th, and now there was zero chance we would ever see it. GoDaddy ate our forum.
Completely apart from my astonishment at how GoDaddy had failed to protect data that was part of these active help tickets, and my astonishment that they don't keep monthly or yearly backups (Who knew? Must have been in the fine print) I was left speechless at how GoDaddy never did know the real manner - technically - in which they had robbed us.
As I see it now, they NEVER had the whole database installed on the new server and just failed to compare the prior installation to the new one before switching off the old server.
I just can't put into words the sleepless anguish I felt as this reality set in. I liken it to realizing that you backed over your child in the driveway because a trivial dashboard rear-obstacle-sensor light was burned out. Knowing that I could have somehow avoided this by being more knowledgeable and hands-on is still just eating me. But I will say it is not/was never about keeping local backups of the forum. Nobody transmits a 2 gigabyte file regularly when they know their host is making backups. Indeed GoDaddy charged $80 each time I wanted to get at the older versions of the file. It's not something they expect people to do.
So now I've fired GoDaddy and contracted with vBulletin directly. They are hosting the forum and they are the real experts who know how to protect forum database content. Our premium plan includes them keeping us up-to-date with the latest most-hacker-proof version of the forum system.
But a few settings and options we enjoyed in version 3 are just not part of version 5, and I would actually consider going back to version 4 and a self-hosted plan to get those features back, once our current contract expires. I am assured that no content would be lost.
So at this point I can only apologize to you for being reckless with your forum and trusting GoDaddy with it. I am very sorry to the folks who worked so hard to build our beautiful dearly departed Press Room which was a true research resource. I am very sorry to our members who were kind enough to share their progress gallery with the community to inspire men who had not yet committed to help themselves by restoring. In some cases the photos were stored only online for in-home privacy reasons and that just kills me. I'm sorry to all the folks who made the forum better in so many ways, by answering questions, by posting about their grief, by helping us debug forum quirks, by running special interest groups.
We're already on the way to being great again, and it starts with what you contribute right now.
Thanks.
Comment