Archive | January, 2011

Sourceforge Attack: Full Report

As we’ve previously announced, SourceForge.net has been the target of a directed attack. We have completed the first round of analysis, and have a much more solid picture of what happened, the extent of the impact, our plan to reduce future risk of attack. We’re still working hard on fixing things, but we wanted to share what we know with the community.

We discovered the attack on Wednesday, and have been working hard to get things back in order since then. While several boxes were compromised we believe we caught things before the attack escalated beyond its first stages.

Our early assessment of which services and hosts were impacted, and the choice to disable CVS, ishell, file uploads, and project web updates appears to have prevented any further escalation of the attack or any data corruption activities.

We expect to continue work on validating data through the weekend, and begin restoring services early next week. There is a lot of data to be validated and these tests will take some time to run.  We’ll provide more timeline information as we have more information.

We recognize that we could get services back online faster if we cut corners on data validtion. We know downtime causes serious inconveniences for some of you. But given the negative consequences of corrupted data, we feel it’s vital to take the time to validate everything that could potentially have been touched.

Attack Description

The general course of the attack was pretty standard. There was a root privilege escalation on one of our platforms which permitted exposure of credentials that were then used to access machines with externally-facing SSH. Our network partitioning prevented escalation to other zones of our network.

This is the point where we found the attack, locked down servers, and began work on analysis and response.

Immediate Response

Our first action response included many of the standard steps:

* analysis of the attack and log files on the compromised servers
* methodically checking all other services and servers for exploits
* further network lockdown and updating of server credentials

Service shutdown

Once we knew the attack was present, we locked down the impacted hosts, so that we could reduce the risk of escalation, from those servers to other hosts, and prevent possible data gathering activities.

This strategy resulted in service downtime for:

* CVS Hosting
* ViewVC
* New Release upload capability
* ProjectWeb/shell

Password invalidation

Our analysis uncovered (among other things) a hacked SSH daemon, which was modified to do password capture. We don’t have reason to the attacker was successful in collecting passwords. But, the presence of this daemon and server level access to one-way hashed, and encrypted, password data led us to take the precautionary measure of invalidating all SourceForge user account passwords. Users have been asked to recover account access by email.

Data Validation

It’s better to be safe than sorry, so we’ve decided to perform a comprehensive validation of project data from file releases, to SCM commits. We will compare data agains pre-attack backups, and will identify changed and added. We will review that data, and will will also refer anything suspicious to individual project teams for further assessment as needed.

The validation work is a precaution, because while we don’t have evidence of any data tampering, we’d much prefer to burn a bunch of CPU cycles verifying everything than to discover later that some extra special trickery lead to some undetected badness.

Service Restoration

Now that most of the analysis is done, we’ve started the next stage of our efforts, which includes the obvious work of restoring compromised boxes from bare metal, and implementing a number of new controls to reduce likelihood of future attack.

We will of course also be updating the credentials which reside on these hosts and performed quite a few steps to further lock down access to these machines.

We are in process of bringing services back one by one, as data validation is completed, and we get the newly configured hosts online. We expect that data validation will progress through the weekend, and we’ll really start getting swinging on service restoration early next week.

File Release Services

Many folks have suggested that the most likely motivation for an attack against sourceforge would be to corrupt project releases.

We’ve found no evidence of this, but are taking extrodinary care to make sure that we don’t somehow distribute corrupted release files.

We are performing validation of data against stored hashes, backups, and additional data copies.

We expect to restore these services first, as soon as data validation is completed.

Project Web

One attack vector that impacts our services directly is the shared project web space. So, let’s talk about that in a bit more detail.

Sourceforge.net has been around a long time, and security decisions made a decade ago are now being reassessed. In most cases past decisions were made around the general principle that we trust open source developers to work together, play nice, and generally do the right thing. Services were rolled out based on widespread trust for the developer community. And that philosophy served us well.

But in the years since then, we’ve evolved from hundreds of sf.net users to millions, and in many cases it’s time to re-asses the balance between widespread trust and security. Project Web is a prime example of this, and we’ve been working at a deliberate pace to isolate project web space, and have begun rolling out the new “secure project web” service to many of our projects.

This new secure project web includes a new security model that moves us away from shared hosting while preserving the scalability we need for mass hosting.

Because of this attack we’ll be accelerating the rollout of Secure Project Web services as part of the process of bringing the project web service back online. This will allow us to provide both improved functionality, and better secruity.

CVS

CVS service is one of SourceForge.net’s oldest services and, due to limitations in CVS itself, cannot readily live on our scalable network storage solution. Validation of this data is going to require several days and we anticipate that this service will be restored sometime in the later part of week.

We are also considering the end-of-life of the CVS service and hope to have user support in migrating CVS users to Subversion in coming months. Subversion generally provides parity to CVS commands, and many of our users have made this transition successfully in the past.

From SVN, projects can move to Git if desired.

Looking forward

We are very much committed to the ongoing process of improving our security, and we will continue making behind the scenes improvements to our infrastructure on a regular basis. This isn’t a one time event, it’s a process, and we’re going to stay fully engaged over the long term.

I’d like to end with a more personal note, I’ve been working with our Ops team a lot this week, and I think we can all say that the patience and support that we’ve received from the community has been the best part of a very bad week.

Thanks again for all the support and encouragement.

SourceForge.net passwords reset

We recently experienced a directed attack on SourceForge infrastructure (http://sourceforge.net/blog/sourceforge-net-attack/) and so we are resetting all passwords in the sf.net database — just in case.

Our investigation uncovered evidence of password sniffing attempts. We have no evidence to suggest that the sniffing attempt was completed successfully.   But, what we definitely don’t want is to find out in 2 months that passwords were compromised and we didn’t take action.
So, we’ve invalidated all sourceforge.net account passwords, and to access the site again, everyone will need to go through the email recovery process and choose a shiny new password:

We have received a lot of support and sympathy  from our community, and I know our ops team is immensely grateful for all of it.  Thanks again for your  patience with us as we work to respond to this attack.  We’ll be working through the weekend to get things back to normal as quickly as possible.

Sourceforge.net attack

Yesterday our vigilant operations guys detected a targeted attack against some of our developer infrastructure.  The attack resulted in an exploit of several SourceForge.net servers, and we have proactively shut down a handful of developer centric services to safeguard data and protect the majority of our services.

Our immediate priorities are to prevent further exposure and ensure data integrity.  We’ve had all hands on deck working on identifying the exploit vector or vectors, eliminating them, and are now focusing on verifying data integrity and restoring the impacted services.

The problem was initially discovered on the servers that host CVS but our analysis indicates that several other machines were involved, and while we believe we’ve determined the extent of the attack, we are verifying all of our other services and data.

As a short term response, we’ve taken down the following services to prevent any possible escalation:

* CVS Hosting
* ViewVC (web based code browsing)
* New Release upload capability
* Interactive Shell services

    Once the immediate response to this attack is over, we will be providing a much more detailed account of what’s happened, and what specific actions we are taking to prevent further exploits.

    Flexible Backups and File Syncs with Create Synchronicity

    create-synchronicity-logo-128x128

    There are a lot of ways you can backup and sync your files, but when I came across this interesting project, I had to learn more. If you’re looking for a solution and nothing seems to meet your needs, you should pay attention and add Create Synchronicity to your list of backup software to try. Featured on sites like Lifehacker, Cybernet News, and many others, this is one piece of software to watch. I had the pleasure of speaking with Clément Pit–Claudel of Create Software, the man behind the project.

    In line with its ideal of combining speed,
    simplicity and power,
    setting up a backup in Create Synchronicity
    is a matter of 5 clicks, from launching
    the program to running a backup,
    configuration included.
    Can you tell me about the project and how it works?

    Create Synchronicity is a backup and synchronization utility, mainly used to keep data safe and up-to-date. It can be used for many things, from backing up family pictures to syncing documents or music, or even to do server backups to NAS. The tool is extremely user-friendly and lightweight (about 160 KB), and yet features a wealth of both classic and unique options, such as a powerful scheduler with catching-up support, precise folder selection, file exclusion by type (or even using regular expressions), full previews, command-line options, and extensive logging.

    In line with its ideal of combining speed, simplicity and power, setting up a backup in Create Synchronicity is a matter of 5 clicks, from launching the program to running a backup, configuration included. And of course, a full manual is available, along with a tutorial.

    How did the project get started?

    The project started about a year and a half ago, mostly out of frustration. At the time, I was searching for a fast and visually-appealing synchronization program to keep my MP3 player in sync with my musical library, which would let me select precisely which folder to sync, and which file types to include. I tried plenty of apps, but I always ended up missing something.
    And there was more: every time someone would ask me for a straightforward, free backup application, they’d say that the program I advised was difficult to understand, and they’d buy a costly app. So I decided to design my own backup/synchronization solution, with simplicity and ease-of-use in mind.

    This streamlining goal has persisted with successive versions, although many new features were included. For example, Create Synchronicity can now understand such paths as “My USB”\Documents, and automagically figure out which drive letter to use. Later, I improved the range of available command line switches, because the CEO of a medium-sized business wanted to use Create Synchronicity for enterprise backups; and now, it has a lot of advanced settings, along with an expert mode for highly-specific server administration needs.

    How long has the team been together?

    I do all the core development, including designing and programming the application, releasing updates, programming the website, and answering support requests, alone. On the internationalization front, however, translators have progressively joined in, so that the program is now available in 25 languages. There were early adopters (SourceForge users!), and the translation to German was the very first contribution to the project by the community — all in all, there are people that I’ve been working together for a year and a half now.

    Create Synchronicity Screenshot

    Create Synchronicity Screenshot

    What roadblocks and obstacles have you had to overcome and how did you do that

    I’d say the hardest thing for this project is reaching new users. Backup/Synchronization are competitive markets, with well-known commercial brands, which dedicate high amounts of money to advertising. I took the decision to never do so, and therefore simply rely on the community to spread the word, and on an awesome hosting/services provider for all distribution and communication needs, namely SourceForge.

    What piece of advice would you give small open source projects just starting out?

    My best advice would be: know your users, and know your goals! Know who you’re creating for, even if it’s a broad range of people (for Create Synchronicity, it ranges from tech-aware families to IT experts and sysadmins), know what you want to create (for Create Synchronicity, a fast, simple yet customizable, reliable backup application), and try to gather as many comments as you can; you’ll often find that users have lots of creative ideas, interesting suggestions, and nice remarks. Also, be patient: if your project is good, it will eventually develop and gain some popularity; but sure, it can take some time to take off.

    Also, don’t hesitate to ask for donations; as rare as they can be for small, newly-created projects, they are still a great boost to your motivation when they start coming in. I, for example, have just launched an original donation campaign for new year 2011: I’m collecting money to invite my girlfriend to a nice restaurant, and hoping that people will chip in =).

    What are your plans for the future?

    In the short term, try to gather a stronger community, and collect patches, and possibly new translations, along with feature ideas and suggestions. In the longer term, I’m planning on a port to Linux and Mac Os using Mono, to bring Linux users an easy, intuitive backup solution, thus contributing to making Linux more easily accessible to new users.

    I want to thank Clément for taking the time to answer my questions about his project, and I look forward to seeing what else he can accomplish! This is a great piece of software and I encourage you to give it a look.

    Download Create Synchronicity: https://sourceforge.net/projects/synchronicity/
    Contribute to Create Synchronicity: http://synchronicity.sourceforge.net/contribute.html
    Donate to Create Synchronicity: Through PayPal or Through Flattr

    My Top 5 Favorite Open Source Happenings in 2010

    It’s a new year, and as we look to 2011 as the year that open source kicks butt, we should take a few moments to reflect back on 2010. After all, that’s what you’re supposed to do in January, right? Look back, then look ahead, then resolve to be better.

    So during my retrospective look, I realized that sitting among the rubble of the open source landscape in 2010, there were a few gems that stood out. These are in no particular order, and I think they all represent what the future of open source is truly about: community, giving back, driving imagination, challenging assumptions, and not accepting the status quo or the mandates of others. In short, it’s about freedom, innovation, and collaboration.

    While some of these may not have been the biggest open source news stories, they are my personal favorites, and your results may vary.

    1) Adafruit’s Open Kinect Bounty
    In case you didn’t hear about this, Adafruit offered a $3000 prize to anyone who could write a fully documented, open source licensed, multiplatform driver for Microsoft’s Kinect device.

    adafruit2

    We all like a good challenge, and when Adafruit announced this contest, imagination and inspiration sprung up everywhere. Ultimately, that’s what we want, right? To work on a cool project that inspires us. There’s nothing better. And for those jokers who think that people are solely motivated by money, think again. What did the winner, Hector Martin, spend his $3000 bounty on?

    “Hector has decided to invest this bounty into hacking tools and devices for a group of people he works with closely (e.g. iPhone Dev Team members, Wii hacker team Team Twiizers, and a few others). They don’t have much expendable income to buy tools and devices to hack, and sometimes this hobby can be a bit expensive, this will be a good investment that will allow them to hack more and newer devices.”


    Not hookers and blow or steaks and strippers, folks, but more hardware for his team, so they can continue doing what they love. That’s all open source devs really want. To be able to do what they love.

    Kudos to Adafruit for inspiring developers, for not being satisfied with the mandate from corporations, and for giving back to the community and the EFF. And kudos to Hector Martin for being a kickass developer.

    2) The Open Source Hydrogen Car

    A car that runs on electricity. Powered by hydrogen fuel cells. And most importantly, whose design is being open sourced so that anyone can contribute. Can it be?!

    riversimple_urban_car_56_1

    Apparently, it can! A company called Riversimple is behind the venture that sticks its nose up at traditional auto manufacturers and the auto industry as a whole. They have created an open source group called the 40 Fires Foundation that oversees the design and development of the car. Riversimple decided to open source the design for two reasons: “Firstly, it’s the right thing to do,” as quoted from their website. And secondly, they say they are a small company “trying to establish ourselves in the biggest industry in the world.” By open sourcing the development, they hope to:

    * speed up development times
    * produce more robust, reliable products
    * drive the adoption of common standards
    * drive down component costs

    Lofty goals for a small group of innovators, and I applaud their efforts. One more thing to note is that the 40 Fires Foundation lists the car as their “first project,” which tells me that they have other things in the incubator. I can’t *wait* to see where they are headed.

    (Kinda bummed about having to wait until 2012 to test drive one of these babies, but I suppose I’ll live.)

    To me, this car represents the power of open source in other things besides just software apps. It’s a mindset. A philosophy of innovation and collaboration. May they help pave the way for open sourcing in other industries.

    3) The LibreOffice split from OpenOffice, and creation of The Document Foundation

    In a move that was met with resounding support from the open source world, a group of core developers on the OpenOffice software suite (controlled by Oracle) picked up and started their own gig under the umbrella of The Document Foundation.

    lo_startcenter_small

    As such, LibreOffice was born. I can only say two words about this: Twisted Sister. No sir, they are indeed not gonna take it any more. (Okay, so maybe that’s a bit dramatic. Forgive me.) From the official announcement about the group:

    “We believe that the Foundation is a key step for the evolution of the free office suite, as it liberates the development of the code and the evolution of the project from the constraints represented by the commercial interests of a single company. Free software advocates around the world have the extraordinary opportunity of joining the group of founding members today, to write a completely new chapter in the history of FLOSS”.


    What this represents to me is the essence of open source: the power to take control over something and make it truly your own, if you don’t like where a project is headed, or if you want it to do something it currently doesn’t do. Power, freedom, and choice. And not having to accept the status quo, or corporate mandates.

    4) ASF taking over Google Wave

    google-wave-01

    When the Google Wave project was killed by Google, and subsequently proposed in the Apache Software Foundation incubator, there was an “overwhelmingly positive” seal of approval from the ASF members. While Google scares me on many levels, I’ll say that it’s great to see so many of the original devs assigned to the project volunteering to continue their work through the open sourced version. I look forward to seeing how the ASF community molds and shapes this project into something awesome.

    To me, this represents how community can pull together and fix something that is broken. It’s collaboration at its finest, and I really look forward to seeing what the Apache Wave has in store for us all.

    5) Ghana’s Open Source Resource Centre

    ghana_osresource_centre

    While the Free Software and Open Source Foundation for Africa (FOSSFA) has been around since 2003, and several open source initiatives have come out of Africa since, this year marks the opening of the shiny new Open Source Resource Centre as a part of the Ghana India Kofi Annan Centre of Excellence in Information, Communication and Technology (ICT). The goals are simple, according to Dorothy Gordon, Director of the Centre. She was recently quoted as saying that the Centre “would help governments, businesses, non-governmental agencies, institutions, departments and citizens in general with the needed technical support in training, development, research and deployment of open source solutions.” They are calling on Africa’s youth to create software that will “help solve the continent’s problems.”

    This is exciting to me because it continues to give hope to the future of open source in Africa, and what it can mean to its citizens (and frankly, the rest of us). Besides Ubuntu, take Ushahidi, for example. A social crowdsourcing platform that enables people to combine information for visual and interactive mapping, Ushahidi was a collaboration between Kenyan journalists and developers back in 2008 as “a way to map reports of violence in Kenya after the post-election fallout.” This year, Ushaidi was used to help the victims of the Haiti quake, map the Gulf oil spill, and to help the citizens of Washington, D.C. during a blizzard that crippled the city. From their site, “We build tools for democratizing information, increasing transparency and lowering the barriers for individuals to share their stories. We’re a disruptive organization that is willing to take risks in the pursuit of changing the traditional way that information flows.” It doesn’t get much more awesome than that, and makes me feel like high fiving somebody.

    The fact that Ghana is opening a physical facility *dedicated to open source* while we struggle to simply introduce it into our schools and government, speaks volumes. I can’t wait to see Africa become serious players in the open source arena.

    So there you have it. My top 5 open source favorites of 2010. What are some of yours?