Archive | Site Status RSS for this section

SourceForge Strengthens Anti-Spam Controls

We care about content quality and have recently stepped-up our efforts to analyze Terms of Use agreement abuse patterns.  One problem area we’re focused on is pure spam projects registered and then reused to sell shoes, herbal products, and home goods; or left empty for future reuse.  Automated controls are key at our scale.  As most of this abuse occurs for newly-registered users, but we do not want to impede legitimate project registration or established devs, starting today we will perform phone-based verification the first time a user account is used to register a project for hosting at SourceForge.

Some of the finer points of our implementation, focused on achieving results while keeping the smallest possible data footprint:

  • Phone-based verification is performed using a reputable third-party provider (Nexmo).
  • We store a one-way hashed (SHA1) copy of phone numbers in our database, allowing us to identify repeat offenders using multiple accounts.
  • We do not store clear phone numbers in our database — numbers are used for verification only at time of first project registration.
  • Nexmo maintains transaction logs containing phone numbers, available to us for diagnosis of PIN code delivery problems.
  • Verification PIN codes are transmitted by SMS or voice and are good for five minutes.

We have baseline registration metrics and will evaluate the effectiveness of this control over the next few days.  We’ll keep an eye out for issues during this rollout — feel free to contact us via Twitter @sfnet_ops or via ticket at https://sourceforge.net/p/forge/site-support/new/

Thanks for your continued support!

SourceForge Infrastructure and Service Restoration update for 7/31

On 7/16, Slashdot Media sites (including Slashdot and SourceForge) experienced a storage fault. Work has continued 24×7 on service restoration. Updates have been provided as each key service component was restored. We’ve provided four prior updates (7/18, 7/22, 7/24, 7/28) summarizing our infrastructure and service restoration status. This is our fifth and final large update for this incident.

As of 7/31, all Slashdot Media sites and services have been restored.

Activity since our 7/28 full update:

  • SourceForge file upload capability was restored to service on 7/31, matching our announced ETA.
  • SourceForge CVS service, non-Allura-backed Bazaar (bzr) SCM service, and interactive shell service were restored on 7/31, ahead of our announced ETA.

SourceForge staff will continue to monitor and respond to the support ticket queue.  If you are in need of support, please submit a ticket at: https://sourceforge.net/p/forge/site-support/new/

Root cause analysis (RCA) for this incident has been formed.  Slashdot Media teams will continue to meet next week to complete preparation of recommendations and post-mortem documentation.  Additional information to be provided to the community via this blog.

Thank you for your continued support and patience.

SourceForge Infrastructure and Service Restoration update for 7/28

On 7/16, Slashdot Media sites (including Slashdot and SourceForge) experienced a storage fault. Work has continued 24×7 on service restoration. Updates have been provided as each key service component was restored. We’ve provided two prior updates (7/18, 7/22, 7/24) summarizing our infrastructure and service restoration status. This is our fourth large update.

The format for this update has changed. Since we are well-past the 50% mark on service restoration, we will be providing updates only on service outages mitigated since 7/24 and ETA detail on outstanding service outages.

All services except SourceForge Developer Services were fully restored on or before 7/24. Services are online except those listed here as outstanding. For full service listing, see our 7/24 update.

Recently restored

  • Project Web service for k* projects is back online.
  • Allura-backed Subversion service is online
  • Classic (non-Allura) Git service is online.
  • Classic (non-Allura) Subversion (SVN) service is online.
  • Classic (non-Allura) Mercurial (Hg) service is online.

Outstanding

  • File upload service data has been reconstructed and is pending final copying, ETA for service restoration is end of day 7/31.
  • Classic (non-Allura) Bzr service is pending data validation. ETA for service restoration is end of day 8/3. Dataset is undergoing analysis, particularly to identify previously-migrated repositories.
  • CVS service data is pending validation, and infrastructure is being brought back online. ETA for service restoration is end of day 8/3. Data analysis is in-progress, to be followed by restore.  Validation of CVS data requires a greater degree of manual validation than other SCMs.
  • Interactive shell service is offline pending availability of all other service data. This service will be the last to come online. ETA for service restoration is end of day 8/3.

Additional notes

  • Targeted communications were sent to projects using Allura-backed Subversion service where we were able to identify commits occurred between time of backup and time of incident.  These projects were supplied commit metadata (committer, date, commit message) to aid in re-capture of these changes.
  • Post-mortem activity is anticipated after data restoration is completed.
  • Scheduled (and pre-announced) downtime of Developer Services occurred on 7/28 to support maintenance on our NFS servers. This downtime was completed successfully and ahead of schedule.
  • One additional Ceph-backed database is being migrated to the recently-provisioned SSD-backed database cluster.
  • Additional storage has been onboarded to support service restoration activities. In some cases we currently have three copies of production data to maintain during restoration.
  • Users on “Classic” non-Allura-backed SCM services should anticipate an upcoming pre-announced migration to Allura-backed service (which was restored first).

Work continues 24×7. Thank you for your continued support and patience.

SourceForge Classic Hg and Git service online

This is a service-specific update to our most recent full update.

SourceForge-hosted non-Allura-backed Hg and Git repositories have been restored to service as of 7/28. In addition to other checks, a representative sample of data has been tested and a functional test performed. Please contact the SourceForge Support team (https://sourceforge.net/support/) if you have any questions or concerns regarding your Hg repository.

Please note that all Hg and Git repositories (both Allura-backed and non-Allura-backed) are now online.

Subversion repository gap notifications by email

This is a service-specific update to our most recent full update.

SourceForge Subversion service restoration was completed yesterday (7/25).  Additional repository analysis was completed today (7/26) to identify projects whose latest commits occurred between the backup restore point and the storage incident.  To identify these repositories, we compared the number of the highest revision on disk to the highest revision seen by our repository tracking code (which stores metadata to a database).

Through this process we identified 497 Subversion repositories which require project action to capture previously-committed data which was not included in the latest backup.  Emails are currently being sent to the developers and administrators for these projects (delivery in the next ~hour), including a recent committer, timestamp and commit message log to aid in remediating repository gaps.

If you have any follow-up questions or concerns, please contact the SourceForge Support team by submitting a ticket at: https://sourceforge.net/p/forge/site-support/new/

Work continues 24×7 on restoration of file upload service, CVS, and non-Allura SCM repositories.  Restore of CVS and non-Allura SCM repository data is in-progress.  File upload service data has completed summing and mount reconstruction is pending; 6 TB of data has been reconstructed in the past 8 hours.