On 7/16, Slashdot Media sites (including Slashdot and SourceForge) experienced a storage fault. Work has continued 24×7 on service restoration. Updates have been provided as each key service component was restored. We’ve provided one large update summarizing our infrastructure and service restoration status, and are providing a second large update with this post.
- Slashdotmedia.com – online
- Slashdot.org – online
- Slashdot Engineering infrastructure – online
- Slashdot Media’s WordPress sites – online
- SourceForge Engineering infrastructure – online
- Slashdot Media operations infrastructure – online
- SourceForge databases – online
- SourceForge download service – online
- SourceForge Directory services (project summary page, download pages, search, front page, directory) – online
- SourceForge Developer Services – partially restored (see detailed status below)
- SourceForge site’s Developer pages backed by Apache Allura (tickets, wikis, forums) – online
- SourceForge Mailing List services (email, web archives, archiving) – online
- SourceForge Project Web service – offline, filesystem checks complete, 22 project letters restored to date (all except jkms), data validation and per-letter service resumption pending, ETA 7/22 for restored letters, remaining four to follow pending restore.
- SourceForge User Web service – offline, filesystem checks complete, 23 user letters restored to date (all except bhl), data validation pending, ETA 7/23, service resumption planned when all letters ready
- SourceForge File Upload service – offline, filesystem checks complete, cryptographic summing in-progress, data preparation in-progress. Filesystem checks complete. Cryptographic sums of files on disk at 75% completion with expected summing completion on 7/23. Data preparation in-progress and at 10% completion, ETA to follow (to be re-estimated when we allocate increased I/O to the data prep tasks on 7/23).
- SourceForge Allura Git service – offline, filesystem checks complete, all project data restored, data validation (repository presence check 100%, repository data presence check 100%, ‘git fsck’ of 10% representative from non-empty repositories 100%). Git validation was aided by its feature set. Final data validation pending and ETA 7/22 for resumption of service.
- SourceForge Allura Mercurial (Hg) service – offline, filesystem checks complete, all project data restored. Data validation (repository presence check, repository data presence check, and repository validation to occur and ETA 7/23 for service resumption.
- SourceForge Allura Subversion (SVN) service – offline, filesystem checks complete, data restoration at 50%. Restoration priority after Git and Hg services. ETA TBD, Future update will provide ETA.
- SourceForge non-Allura SCM platforms and CVS service – offline, filesystems checks and data restoration have not commenced. Priority given to modern SCMs which include internal data validation mechanisms; and those repositories fully backed by Apache Allura. Service restoration ETA TBD.
Engagement with our storage platform vendor will continue and post-mortem activity is anticipated after data restoration is completed. The team continues split operation between data restoration and service restoration as to expedite return to full service. Knowledge capture has been continuous throughout this outage and will drive continuous improvement.
We intend to continue our existing communications approach — incremental updates will be provided on individual service restoration, and large updates (like this one) will be provided with additional metrics and technical details as work progresses.
Work continues 24×7 on restoration of SourceForge file upload, SCM, and project web services.
Thank you for your continued support and patience.