Updates - Tuesday, February 9th, 2010 5:55:31pm
February 2, 2:09 PM - Mysql Server is Back
Posted by: tony
That was a hard one.
We finally found and fixed why connections were being rejected.
The mysql server is back up and running!
February 2, 12:05 PM - Still Working
Posted by: tony
The Mysql issue is more complex that initially anticipated. We are still working on resolving the problem. Connections are not going through.
February 2, 9:19 AM - Network Outtage and cPanel "Improvement"
Posted by: tony
There was a major network outtage involving a blown core router and a misconfiguration last night. It prevented traffic to the servers from getting there until around 1:30am.
At that point, cPanel ran it's nightly update and added some "improvements" to the systems. These "improvements" messed up access to your home folders (they were safe, just not visible) and are still affecting access to your mysql databases (we are working on it actively).
I will update you as soon as the mysql access has been restored.
Tony
January 17, 10:43 AM - Email Authentication Failure
Posted by: tony
This is one of the reasons I have a growing gripe list with cPanel.
After the nightly security check, cPanel restarted a bunch of services (normal), but one was broken (the pop3/imap process). For whatever reason, the monitor that checks pop3/imap was enabled but not running. Further, no alerts were sent out.
This prevented email from being picked up (deliveries were fine).
I had to manually give it a kick and alls well.
January 15, 1:08 PM - Possible Database Access Issues
Posted by: tony
The server that houses the cwpro databases is beginning the long, slow decline and needs to be retired. This is not bad - we have replacement gear ready and am starting the move over of data.
However, we did notice that some sites stopped connecting - it appears some of the username/passwords have been corrupted (this is an ongoing issue, just magnified).
*IF* you have a script that was working and now complains about not being able to connect, just send a quick email to cwpro@crosswinds.net - in it just say you have a connection issue and include the url of page that is now not working (forum/shopping cart/etc).
With that information we can quickly get you reconnected.
Thanks!
January 1, 10:35 AM - Happy New Year!
Posted by: tony
Happy New Year to everyone!
We would like to thank every one of you that has chosen to use Crosswinds Hosting. Your support means the world to us!
We have some exciting plans for 2010 to make Crosswinds even better.
Best wishes to each and every one of you from us here at Crosswinds!
December 30, 8:58 AM - Horde Fixed
Posted by: tony
The harder of the 2, Horde, is now up!
December 30, 8:44 AM - Roundcube is Fixed
Posted by: tony
The easier of the 2, Roundcube, has been fixed!
December 30, 8:41 AM - Horde and Roundcube
Posted by: tony
You know, for a supposedly production quality system, cPanel likes to destroy itself.
On a standard update, it did not do the right thing - thus connections to the database for horde and roundcube were broken. Broken to the point where I have to reinstall from scratch...
Makes me pull out my hair, but I will fix this and let you know when it's ready!
December 29, 9:16 AM - Fantastico Fixed
Posted by: tony
Got email from Fantastico this morning and had to reinstall everything.
Once that was done, Fantastico was up and working once again!
December 28, 10:50 PM - Other Errors
Posted by: tony
As part of the steps to fix Fantastico, an upgrade to cPanel was requested. A couple errors have appeared (such as the error on the left for Cannot find GDBM). These are normal on a forced upgrade as this - do not panic!
December 28, 10:38 PM - Fantastico Issue
Posted by: tony
There is an issue with Fantastico that is causing an error message to appear like this:
"Fatal error: Call to undefined function: XXXXXX() in /tmp/some_file_name on line YYYY"
This is a problem with the updates from the creators of Fantastico and they are aware of it. At this point we have no other information and the ETA is as soon as they can - they have to recall some staff that are enjoying the holidays.
As soon as we know more, we will update you.
December 20, 12:35 PM - It's Back
Posted by: tony
Finally nailed the last of the DDOS attackers and we are back up and running!
December 20, 11:39 AM - Attack
Posted by: tony
One pro server has been hit by a DDOS that we are fending off. It caused all sorts of weirdness - email not working, web pages not appearing, etc, then the system was pretty much unusable.
We have blocked most and are in the process of bringing the system back to a usable state.
Watch here for details!
September 24, 11:31 AM - Partial Network Outage
Posted by: tony
One of the major feeds into the NOC was cut - literally. A backhoe at a construction site near one of the internet peering points (a place where lots of backbones connect to share traffic) cut a fiber trunk.
This is a major internet cable with approximately 1000 strands of fiber optic cable. They noticed it almost immediately and a crew was on site repairing in under an hour. The link reconnecting us was brought up just prior to 9am EST.
I wasn't affected, but all of Europe, Africa and a good chunk of the US was affected - which depended on which provider you used.
Due to business issues (and not techical) traffic on the affected providers was not re-routed to other networks. It's silly and, in my opinion, completely defeats the purpose of these connections if businesses won't do business with each other to ensure traffic can be redirected.
The servers and everything were up the entire time and my access was not affected - it wasn't until our NOC was informed at approximately 8am that we knew what had happened. They did know there was a traffic reduction on one of their feeds, but no details.
We apologize for those affected, even though it was beyond our control.
-- Tony
September 23, 7:07 PM - Emergency Maintenance Complete!
Posted by: tony
It is finished!
The data has been kept safe, the drive replaced, an additional drive added. The file system is up and running well, all services have been restored.
I do apologize for the extended downtime.
I could not risk any of your data and had to bring the system back in as safe and careful a manner as possible.
Thank you for bearing with us and I hope to not do this for a VERY long time!
Thank you for your patience and understanding.
-- Tony
September 23, 6:44 PM - FSCK Underway
Posted by: tony
System is up and the data partition is fsck'ing (filesystem check).
Since the partition is big, this does take a while. Keep checking.
-- Tony
September 23, 6:33 PM - Emergency Maintenance Update - Second Drive Swap
Posted by: tony
The first drive swap and data mirror sync finished successfully.
This ensures your data is duplicated and safe.
The second drive swap is for another level of safety in case one of the drives fail - it needs to be integrated into the system.
This can be done after the system is up.
Once the cables are are reseated, the system is coming up. I then need to fsck (filesystem check) the data partitions and then we will be live!
Stay tuned!
-- Tony
September 23, 5:11 PM - Emergency Maintenance Update - Rebuild Phase 1 Restarted
Posted by: tony
You will see some life as the web site is displaying the error page. We allow this up so that people will be pointed here.
The first replacement is being resynched. Knock on wood or anything else to fend off Murphy!
Once this drive is synced, we are doing 1 more drive swap but the sync for it will take place with the system online.
Will keep you updated!
-- Tony
September 23, 4:55 PM - New Drives Arrived
Posted by: tony
You might have seen a blip of uptime - we were conducting a test prior to the drive swapout.
The drives are there and ready. We'll be replacing it in approximately 10 minutes, bringing the system back up and looking at resynching the mirrors.
Powered by Coranto
|
