On 2011-07-16, one of our several CVS servers suffered a catastrophic multiple disk failure, resulting in full loss of the server. We have multiple backups of the data, and we believe only a very small time window exists for when data may have been lost (minutes). Due to the amount of data involved, and the complexities of the CVS offering, this is delaying our ability to bring the service back online for the impacted project set. This recovery is being further hampered by a disk failure on the host we’re migrating data to. Once we are confident that service will be stable, the disk failures resolved, and normal routine operations are functional, we will restore service. Our current best guess is that service will be restored on 2011-07-21. Our primary concern is ensuring data safety, followed by service availability.
Moving forward, we have a renewed commitment to the CVS service offering. We plan on a multi-pronged strategy to help protect from bitrot and improve CVS availability and uptime:
1. We plan on re-implementing the CVS service offering using current best practices
2. We plan on discouraging new projects from using the CVS service and directing them to something that is more current
3. We plan on helping projects interested in migrating to a more current SCM platform through documentation and direct assistance as needed
The re-implementation of the CVS offering will require additional future CVS downtime. That downtime is not anticipated to last more than a day per project during migration to the new platform. We do not at present have an estimate on when the new service will be deployed, but will provide ongoing updates as the plan is hashed out further.
Again, thanks for your continued patience and we apologize for any inconvenience caused.
Sincerely,
SF.net Service Operations Group