As IT organizations adopt practices like Agile and DevOps, experimentation must become a core competency to help them mitigate risks and improve impact when launching new products or features. Through experimentation, marketing and product development teams can have the ability to iterate quickly on the user experience by testing every idea, learning quickly from the data, and then adjusting the next release to suit real-world user behavior.
One company that leverages the power of experimentation to build better products faster and cost-efficiently is Optimizely. As the world’s leading experimentation platform, Optimizely empowers marketing and product teams to test, learn, and roll out winning digital experiences, every time.
SourceForge had the chance to speak with Robin Pam, the Director of Product Marketing at Optimizely, to discuss the benefits of experimentation platform for today’s software development teams. Pam also highlights their recently released Optimizely Full Stack 2.0 with Feature Management and shares how Optimizely supports better customer experiences through experimentation.
Q: First and foremost, can you please tell us a bit about Optimizely? When was the company established and who are your current clients?
Robin Pam, the Director of Product Marketing at Optimizely
A: Optimizely was established in 2010 by Dan Siroker and Pete Koomen. As Director of Analytics for the Obama 2008 campaign, Dan’s job was to use data to help the campaign make better decisions. He ran hundreds of A/B tests that directly generated tens of millions in incremental donations for the campaign, however, at the time, A/B testing was a painful process available only to sophisticated users. They started Optimizely to enable companies to experiment quickly and easily on any website.
Now, eight years later, we are the world’s best digital experience software platform for marketing and product development teams. Our experimentation platform enables any company to test all decisions about digital products, commerce, and campaigns in order to ensure those decisions are based on data, evidence, and facts — not gut instinct and guesswork.
Our customers include digital leaders such as Slack, Atlassian, and eBay, venerable brands like IBM, the New York Times, and Fox, and 26 of companies in the Fortune 100.
Q: What exactly is an experimentation platform? And how can this solution empower app development teams?
A: Leading digital businesses like Amazon, Facebook, Google, and Netflix pioneered the practice of experimentation in software development so that they could eliminate guesswork from decisions about how to design and build digital products. Google famously tested 40 shades of blue, back before most companies had adopted the practice, and at this point, each of these companies runs thousands of experiments each year.
At a basic level, experimentation platforms like Optimizely enable developers and product managers to run A/B tests, and continuously iterate on the customer experience. In an A/B test, you can implement at least 2 different versions of a feature or idea, release both of them to customers in a production environment at the same time, and measure whether there are statistically significant differences in key metrics as a result of the changes you made. The other key capabilities of experimentation platforms are the ability to target or personalize experiences for different types of visitors, and the ability to use feature flags to control the release of features to production, whether to slowly roll out features across all visitors, or to turn features on and off selectively, independent of a code deploy.
Development teams in particular benefit from experimentation in two ways:
- Feature flags can help teams avoid the nightmare scenario of launching a new feature to everyone, only to find out once it hits production that customers hate it, and it hurts key business metrics like conversion rate or revenue.
- Experiments can help developers prove the impact of the features they build on business metrics, bringing them closer to the customer experience and breaking down the silos between business teams and application development teams.
Q: As the world’s leading experimentation platform, how can Optimizely help executives and product and development teams deliver better customer experiences through experimentation? Or in what ways can Optimizely help organizations create and run bold experiments that enable them to make data-driven decisions?
A: One of our customers recently told me that before they started experimenting, executives often asked low-level employees to implement specific features, creative, or copy on the website, because it seemed “right”. This is commonly referred to as “HIPPO” syndrome, for “highest-paid person’s opinion”. Now that they are experimenting, members of the product and development team can say: “Great idea, let’s test it.” With data from real customers, they can then make an informed decision, and know whether the executive’s gut instinct is the right move for the business.
In this way, experimentation really levels hierarchies within companies and across teams, providing a way to de-risk any idea, no matter how big or small. It gives teams the tools to have a conversation grounded in real user behavior and data, rather than opinions. And, it doesn’t hurt that experimentation provides a massive return on investment (ROI). Our customers see increases in conversion rates, engagement, and revenue that amount to 5-10x ROI, just from the ability to continuously iterate on experiences.
Optimizely customer HP created a culture of experimentation as a way to ensure that teams throughout their organization had the independence they needed to run experiments that make their operations, products, and end customer experiences better – pulling ideas from the tens of thousands of HP employees. Also, by challenging executives to embrace failure and empowering employees to work on their own mini-projects to experiment, they’re shifting their culture and changing the way they approach the toughest tech problems, resulting in 500 campaigns, which has driven an incremental $21 million in revenue with Optimizely.
Q: Please tell us more about Optimizely X Full Stack. How can your solution drive engaging digital experiences across every device and every customer touchpoint?
Optimizely’s big innovation in 2010 was our visual editor, which enabled anyone to swap out components like copy and images on the front end of a website through the magic of client-side Javascript. Over the last 18 months, Optimizely has gone through a huge transformation. We developed SDKs for server-side testing (Optimizely Full Stack) to add to our client-side testing product, which opened up an entirely new buyer for us. Full Stack is our developer solution, built for developers to architect experiments directly in code, whether server-side, client-side, or in mobile apps and connected devices.
Full Stack came out of our experience working with product and development teams at some of our most sophisticated customers. These teams wanted to be able to experiment on complex elements of the customer experience, like algorithms, pricing, voice and chat apps, business models, and entire onboarding or checkout flows. They also cared deeply about application performance and configuring Optimizely to work with their existing infrastructure.
Now, we have customers using Full Stack to test and iterate on everything from the ordering experience on in-store restaurant kiosks to Alexa skills and chatbots.
Q: Last April, you released Optimizely X Full Stack 2.0 with Feature Management. Can you please tell us a bit more about Optimizely Feature Management. What challenges or issues does it seek to address?
A: The old way of developing software–design a feature, build it, launch it to production, then pray that it works the way you expect–is full of risk. Things go wrong that you can’t predict, whether its bugs that break the user experience, systems that don’t scale, or user outrage at a surprising new feature.
Today’s leading companies are using techniques like agile and DevOps to deliver software on time and with high quality. Feature flags give teams who are moving quickly control over the release of new features so that customer-facing functionality can be turned on for customers gradually independent of a code deploy. Feature management provides a kind of control center for your product — you can turn features on and off for different groups of users, slowly ramp up traffic to a given feature, and monitor for unforeseen impacts.
A big challenge we see is that many development teams launch features without a plan to measure how well that feature is performing against key business goals, or even whether it’s better than the experience that existed before. That’s where Optimizely really stands out.
Our expertise is in the digital experience optimization space and helping teams scale experimentation programs across some of the world’s largest digital businesses. So we’ve taken the approach of natively integrating feature management and experimentation in our platform. We believe that these two practices:
– minimizing launch risk with feature flags and maximizing business impact with product experimentation
– go hand in hand, and are each an essential part of the product development lifecycle.
Our goal is to enable development teams to bring control and, more importantly, rigorous decision making based on end-user behavior to every step of the product development process.
Q: What are the key capabilities of Optimizely Feature Management? What makes it unique from other similar solutions?
A: Feature management is deeply integrated into our best-in-class experimentation platform to provide the tools teams needed to implement, build, measure, and learn feedback loops in product development. We enable customers to manage the entire lifecycle of a feature, from development to launch, from to iteration and growth:
- It starts with creating a flag to limit the blast radius of any new feature;
- Then, you can run a Feature Test to ensure that the new feature is driving the business value you expect, prior to launching;
- With rollouts, you can release a feature slowly across the user base to monitor for unforeseen impact; and
- With A/B tests, teams can continue to iterate on a new feature post-launch, to ensure they’re always providing the best customer experience.
Two key differentiators for Optimizely here are:
- Experimentation first. Our experimentation platform is second to none, from Stats Engine, our advanced statistical methodology, which ensures that any decision is backed by accurate and reliable data, to Program Management, which enables large teams to collaborate on ideation, managing the experiment lifecycle, and sharing knowledge from a test and learn program, and our visual editor for rapid experimentation on the front end of websites.
- Focus on customer experience. We integrate feature management and experimentation in one easy workflow to ensure that teams can move quickly and that developers, product managers, and analysts can all collaborate effectively on decisions about the digital experience.
Q: Looking into the future, do you see any emerging trends, methods, features, or technologies that will impact the customer experience optimization market?
A: Other than experimentation and feature flags, we think three factors may have an impact on the customer experience optimization market:
- The increasing proliferation of digital touchpoints. Many of our customers are starting to design voice and chat interfaces, omnichannel experiences that cross the boundaries of the digital and physical worlds, and focus more on mobile. Digital is invading every aspect of our lives, and with it, companies will need to continue to move faster, experiment more, and become more agile to keep up.
- The rise of product management. In Silicon Valley, the product manager role has been well established for decades, but in many other industries, digital product management is just beginning to mature as a discipline. In the past, marketing teams dictated to back-office IT what was needed for a good experience. Now, product managers, who sit at the intersection of business and IT, are taking on more and more accountability for the business outcomes of digital experiences. There is a new ecosystem of tools emerging to arm product teams—PMs, analysts, developers, and designers—with the insight and control they need to make fast, informed decisions about the customer experience.
- Focus on performance. Performance is top of mind for our largest customers for two reasons. First, the number of people using mobile phones as their primary way of accessing the Internet continues to grow. Second, speed is paramount for digital business—slow sites can equal lost conversions and revenue. As a result, I think you’ll see more companies adopting server-side methods for optimization and personalization that have zero impact on performance, and therefore putting more control over the customer experience in the hands of developers and product managers.
Q: What can customers expect from Optimizely? Are there any new product updates or offerings that you are brewing?
A: We’re very focused on continuing to build out our platform to enable our customers to scale digital experience optimization across teams and touchpoints. Our customers like IBM, the Gap, and the New York Times are constantly pushing the boundaries of digital experience optimization and keeping us busy building out new personalization capabilities, even better developer tools, automation and machine learning, program management enhancements, and a more robust data platform.
Additionally, our new, enhanced Data Export is being updated with enterprise-grade capabilities to address the needs of the modern experimentation team. The enhanced Data Export will enable customers to perform deeper and more flexible analysis on the metrics that matter most to their business by enriching their data warehouse with Optimizely experimentation data. This capability will be available in the coming quarters.
About Optimizely
Optimizely is the world’s leader in digital experience optimization, allowing product development teams to deliver continuous experimentation and personalization across mobile apps, websites, and all connected devices. Recognized as one of the “Best Places To Work In The Bay Area” by the San Francisco Business Times from 2014 to 2018, Optimizely empowers organizations to develop and deploy bold experiments that enable them to make smart, data-driven business decisions. Headquartered in San Francisco, Optimizely also has satellite offices in Austin and New York and other cities across the world, including Amsterdam, Munich, London, Cologne, and Sydney.