Archive | Site Status RSS for this section

SourceForge Infrastructure and Service Restoration update for 7/31

On 7/16, Slashdot Media sites (including Slashdot and SourceForge) experienced a storage fault. Work has continued 24×7 on service restoration. Updates have been provided as each key service component was restored. We’ve provided four prior updates (7/18, 7/22, 7/24, 7/28) summarizing our infrastructure and service restoration status. This is our fifth and final large update for this incident.

As of 7/31, all Slashdot Media sites and services have been restored.

Activity since our 7/28 full update:

  • SourceForge file upload capability was restored to service on 7/31, matching our announced ETA.
  • SourceForge CVS service, non-Allura-backed Bazaar (bzr) SCM service, and interactive shell service were restored on 7/31, ahead of our announced ETA.

SourceForge staff will continue to monitor and respond to the support ticket queue.  If you are in need of support, please submit a ticket at: https://sourceforge.net/p/forge/site-support/new/

Root cause analysis (RCA) for this incident has been formed.  Slashdot Media teams will continue to meet next week to complete preparation of recommendations and post-mortem documentation.  Additional information to be provided to the community via this blog.

Thank you for your continued support and patience.

SourceForge Infrastructure and Service Restoration update for 7/28

On 7/16, Slashdot Media sites (including Slashdot and SourceForge) experienced a storage fault. Work has continued 24×7 on service restoration. Updates have been provided as each key service component was restored. We’ve provided two prior updates (7/18, 7/22, 7/24) summarizing our infrastructure and service restoration status. This is our fourth large update.

The format for this update has changed. Since we are well-past the 50% mark on service restoration, we will be providing updates only on service outages mitigated since 7/24 and ETA detail on outstanding service outages.

All services except SourceForge Developer Services were fully restored on or before 7/24. Services are online except those listed here as outstanding. For full service listing, see our 7/24 update.

Recently restored

  • Project Web service for k* projects is back online.
  • Allura-backed Subversion service is online
  • Classic (non-Allura) Git service is online.
  • Classic (non-Allura) Subversion (SVN) service is online.
  • Classic (non-Allura) Mercurial (Hg) service is online.

Outstanding

  • File upload service data has been reconstructed and is pending final copying, ETA for service restoration is end of day 7/31.
  • Classic (non-Allura) Bzr service is pending data validation. ETA for service restoration is end of day 8/3. Dataset is undergoing analysis, particularly to identify previously-migrated repositories.
  • CVS service data is pending validation, and infrastructure is being brought back online. ETA for service restoration is end of day 8/3. Data analysis is in-progress, to be followed by restore.  Validation of CVS data requires a greater degree of manual validation than other SCMs.
  • Interactive shell service is offline pending availability of all other service data. This service will be the last to come online. ETA for service restoration is end of day 8/3.

Additional notes

  • Targeted communications were sent to projects using Allura-backed Subversion service where we were able to identify commits occurred between time of backup and time of incident.  These projects were supplied commit metadata (committer, date, commit message) to aid in re-capture of these changes.
  • Post-mortem activity is anticipated after data restoration is completed.
  • Scheduled (and pre-announced) downtime of Developer Services occurred on 7/28 to support maintenance on our NFS servers. This downtime was completed successfully and ahead of schedule.
  • One additional Ceph-backed database is being migrated to the recently-provisioned SSD-backed database cluster.
  • Additional storage has been onboarded to support service restoration activities. In some cases we currently have three copies of production data to maintain during restoration.
  • Users on “Classic” non-Allura-backed SCM services should anticipate an upcoming pre-announced migration to Allura-backed service (which was restored first).

Work continues 24×7. Thank you for your continued support and patience.

SourceForge Classic Hg and Git service online

This is a service-specific update to our most recent full update.

SourceForge-hosted non-Allura-backed Hg and Git repositories have been restored to service as of 7/28. In addition to other checks, a representative sample of data has been tested and a functional test performed. Please contact the SourceForge Support team (https://sourceforge.net/support/) if you have any questions or concerns regarding your Hg repository.

Please note that all Hg and Git repositories (both Allura-backed and non-Allura-backed) are now online.

Subversion repository gap notifications by email

This is a service-specific update to our most recent full update.

SourceForge Subversion service restoration was completed yesterday (7/25).  Additional repository analysis was completed today (7/26) to identify projects whose latest commits occurred between the backup restore point and the storage incident.  To identify these repositories, we compared the number of the highest revision on disk to the highest revision seen by our repository tracking code (which stores metadata to a database).

Through this process we identified 497 Subversion repositories which require project action to capture previously-committed data which was not included in the latest backup.  Emails are currently being sent to the developers and administrators for these projects (delivery in the next ~hour), including a recent committer, timestamp and commit message log to aid in remediating repository gaps.

If you have any follow-up questions or concerns, please contact the SourceForge Support team by submitting a ticket at: https://sourceforge.net/p/forge/site-support/new/

Work continues 24×7 on restoration of file upload service, CVS, and non-Allura SCM repositories.  Restore of CVS and non-Allura SCM repository data is in-progress.  File upload service data has completed summing and mount reconstruction is pending; 6 TB of data has been reconstructed in the past 8 hours.

SourceForge Subversion (SVN) service online

This is a service-specific update to our most recent full update.

SourceForge-hosted Allura-backed Subversion (SVN) repositories have been restored to service as of 7/25. In addition to other checks, a representative sample of data has been tested using ‘svnadmin verify’ and a functional test performed (commit and/or checkout via http, ro svn, rw svn+ssh, etc.). Please contact the SourceForge Support team (https://sourceforge.net/support/) if you have any questions or concerns regarding your SVN repository.

Please note that SVN repositories not previously migrated to the Allura platform remain offline at this time and will be restored separately. See the above full update for information on restoration of other SCM platforms.

Work continues 24×7 on restoration of file upload service, CVS, and non-Allura SCM repositories.