The world is becoming more interconnected than ever before. The Internet of Things–a sophisticated array of networked physical devices equipped with sensors and processors–has created a whole range of opportunities and has disrupted the way we live. It provides the ability to capture, measure, and monitor the occurrence of any activity, otherwise known as instrumentation, to enable streams of data and information to be quickly turned into relevant and real-time decisions, something that would have been unimaginable two decades ago.
Instrumentation is a reality not just in the physical world, but most especially in the world of IT where companies desire to get greater visibility and control over all the system and software components that drive their business processes. The need to instrument anything and everything that can be instrumented can be attributed to the desire of organizations to become more data-driven, applying data insights to detect patterns and uncover potential business opportunities and threats.
Sourceforge recently spoke with Mark Herring, the CMO of InfluxData, to discuss the Age of Instrumentation and how the InfluxData Time Series Platform enables today’s organizations to compete and succeed in this era.
Q: Please tell us a bit about InfluxData. When was the company established and what challenges does it seek to address?
InfluxData CMO Mark Herring
A: InfluxData was founded in 2013 under the name Errplane, to provide a SaaS offering for DevOps monitoring with a strong focus on machine learning. The company was founded by Paul Dix, who previously helped build software for startups and large organizations, such as Microsoft, Google, McAfee, Thomson Reuters, and Air Force Space Command, as well as headed up one of the largest machine learning meetup groups in New York City.
Paul and the team realized as they started to build the solution that there was a fundamental problem with the architecture around any datastore they tried to use. MySQL, Cassandra, and MongoDB all were designed to solve some very different problems, and there was no modern architecture that could handle the volume of instrumented information they wanted to store.
As a result, the company pivoted from a SaaS offering and focused on this new problem – how to create a performant and purpose-built Time Series Platform that could handle the new volume of metrics and events. As a result, this modern platform is designed to handle metrics (such as server stats that happen at regular time intervals) and events (such as exceptions, sensor readings that happen at irregular time intervals), address millions of writes per second, provide real-time analytics of this streaming data, be very easy to use, and to be optimized for time series data and time series functions. It focuses on developer happiness – those who are building the next generation of applications to derive better business insights, deliver data-driven real-time actions, and have a consolidated single view of all instrumented infrastructure – from applications to microservices, and from systems to sensors.
Today the company has more than 135,000 active installs and more than 420 paying customers since launching its Time Series Platform in late 2016. From its roots serving developers, the company has also rapidly built its developer community and customer base across industries – including manufacturing, financial services, energy, and telecommunications.
Q: Can you please tell our readers more about the Age of Instrumentation?
A: We are witnessing the instrumentation of many surfaces in the material world – streets, cars, factories, power grids, ice caps, satellites, clothing, phones, microwaves, milk containers, planets, human bodies, and more.
Everything has, or has the potential to have, a sensor. In parallel, we are seeing the instrumentation of the virtual world, with trends such as microservices, containerization, elastic storage and software-defined networking all pushing increasing volumes of metrics and events of status information, even down to the smallest of software components.
Companies have the opportunity to capture and analyze all of these new data streams to find greater business insight and interesting business moments that will enable them to make real-time decisions deriving massive competitive advantage. What we’re witnessing, and what the times demand, is a paradigmatic shift in how we approach our data infrastructure and how we approach building, monitoring, controlling and managing this instrumentation for competitive advantage.
Q: The idea that everything in the world can now be “instrumented” is both terrifying and amazing. Organizations can now collect, correlate, and analyze vast amounts of data, and then use that data to make well-informed decisions – but there’s also a lot to be worried about including privacy and security. How can organizations reap the benefits of instrumentation with less risk of harm?
A: Users need and want real-time immediate access to all data. Instrumentation helps organizations identify patterns, predict the future, control systems, and get the insights necessary to stay ahead of the curve. But to reap the benefits, organizations need to pick the right tool for the job. Solutions exist that make data available and queryable as soon as it is written, since business decisions demand immediate results.
For example, a power company might use instrumentation to re-orient each wind turbine to achieve 5 percent greater yield in energy production, or a retailer can understand its customer journey and increase in-store sales by 20 percent by relevant cross and upsell promotions. Organizations using systems not real-time ready or able to handle massive streams of instrumented data are not poised to succeed in the modern world or benefit from instrumentation.
Naturally with great changes in technology there are the laws of unintended consequences to contend with. With all this instrumentation, the needs for privacy and security are important. Some of these are socio-economic issues that only laws can help enforce (such as the GDPR). We think a platform that can monitor and instrument all these interactions will be key to ensuring compliance.
Q: What do organizations need to fully transition into the Age of Instrumentation?
A: Organizations can take advantage of all this instrumented data for real-time decisions and business benefit by leveraging platforms able to support the gathering and analysis of these new workloads of more data points and data sources, needing more monitoring and controls. The Age of Instrumentation is causing a paradigmatic shift in how we approach our data infrastructure and how we approach building, monitoring, controlling, and managing systems.
In the Age of Instrumentation, sensor, server, application, and human event driven data is all time series in nature. To gain insight and act, users will have to evaluate and ask questions about this data which are based on time frames and ranges. Time is no longer something added to data; it is constitutive of the data. There’s no time to write time series routines as they arise because time is already here.
In InfluxData’s case, we deliver a modern Time Series Platform built specifically for instrumented metrics, events, and other time-based data. Whether the data comes from humans, sensors, or machines, InfluxData empowers developers to build next-generation monitoring, analytics, and IoT applications faster, easier, and to scale delivering real business value quickly.
Q: Tell us more about the InfluxData Time Series Platform. How does it stand out over other similar solutions in the market?
A: InfluxData’s Time Series Platform is the only platform built from the ground up to support organizations looking to best leverage the wealth of sensor data delivered from devices and sensors to provide an advantage for their customers. Sensors deliver time stamped or time series data to measure change over time, requiring a time series database. The data is streamed relentlessly from sensors, and sensors capture and emit data in real time.
InfluxData’s approach to build the platform from the ground up is unique. Unlike other vendors who are feverishly adding timestamps and time functions to their data platforms, InfluxData has focused on delivering a modern platform that provides these three tenets:
- Designed for Real Time – The modern world is mercilessly real-time. Users need and want real-time immediate access to all data. They need to identify patterns, predict the future, control systems, and get the insights necessary to stay ahead of the curve. Data should be available and queryable as soon as it is written. Business decisions demand immediate results.
- Biased for Action – Basic monitoring is too passive. You can’t manage what you don’t monitor, but advances in machine learning and analytics make automation and self-regulating actions a reality. A modern system must be able to trigger actions, perform automated control functions, be self-regulating, and provide the basis for performing actions based on predictive trends.
- Cloud Scale – The world demands systems that are available 24x7x365 and can automatically scale up and down depending on demand. They must be able to be deployed across different infrastructures without undue complexity. They need to make optimal use of resources, for instance keeping only what is needed in memory, compress data on disk when necessary, and move less relevant data to cold storage for later analysis.
For organizations to fully leverage and gain efficiencies in this Age of Instrumentation, they need to understand the behavior of their devices and sensors in real time, have the ability to iterate solutions to address any behavior issues of the instrumented devices, and continue to innovate solutions at lightning speed. InfluxData’s Time Series Platform is purpose-built to deliver unique advantages for customers around data storage, data aggregation, streaming analytics and visualization – all critical areas around instrumentation. InfluxData enables companies to always have a direct view into the state of each of their instrumented devices and sensors so changes can be made real-time to keep services running efficiently.
Q: What types of clients do you currently work with? How is InfluxData helping these businesses compete and succeed in the Age of Instrumentation?
A: InfluxData has many customers leveraging its time series database to succeed in today’s Age of Instrumentation. For example, customer BBOXX uses InfluxData to provide a monitoring and control plane for its IoT platform that has resulted in $2.4 million in savings, an offset of 40,000 tons of CO2, and most importantly more than 63,000 school-aged children being able to study comfortably without straining their eyes or destroying their lungs with Kerosene fires.
Houghton Mifflin Harcourt leverages the real-time visibility, time to value, and control that InfluxData provides to better align performance with the fiduciary aspects of infrastructure operation. The resource ID’s (captured by AWS) are used as tags in InfluxData to perform data drilldowns to discover all the different ways that the company’s products are consuming AWS services, and also to view cost changes to identify deviations from what they expect versus what they know.
In another case, Spiio uses its sensor-based Spiio Cloud Platform – featuring wireless systems, smart irrigation, and plant data analytics – to give clients a full view of green wall installations anytime, anywhere. Using Spiio’s real-time analytics powered by InfluxData, clients can understand their plants’ condition, share insights across their organization, and make data-driven decisions to boost maintenance efficiency and improve green wall design.
Also, with InfluxData NewVoiceMedia gains real-time visibility into their service’s metrics and events to meet callers’ expectations for seamless cloud communication and bridge the gap between telephony and data. They use InfluxData in conjunction with New Relic for insight on their entire stack, powered by the millisecond resolution metrics they need to achieve the response times and analyze future performance. InfluxData also gathers NewVoiceMedia’s telemetry data directly from their code to determine how new features are being used, what the performance looks like versus the old, and whether the new features can scale as expected.
In addition, a large multinational company is using InfluxData to deal with renewable energy distribution challenges with the goal of reducing our country’s dependence on fossil fuels and making use of renewable energy that will impact generations to come.
Finally, Coupa Software uses InfluxData to derive operational metrics of its spend management platform, and unify and streamline its enhanced monitoring framework. InfluxData helped it achieve a consistent track record of delivering close to 100 percent uptime SLA across major product releases and product module offerings, as well as solves Coupa’s data accessibility, aggregation, and retention challenges. Operational metrics have created a custom alerting framework, which has become the foundation to the path of building a system that is self-healing and can provide predictive analytics key to accurate forecasting.
Q: The InfluxData Time Series Platform is based on an open source core. Why is this approach important to your company?
A: Open source is not about license – it is about sharing ideas and information, it is about participating and collaborating on solutions and ideas, and it is about community where the whole is greater than the sum of its parts. It is about full transparency where nothing is hidden. This is the new reality of the market we are in. Developers are the new influencers and the new buyers, even though they don’t own the budget.
Developers live in the open source world, they consume open source code, they collaborate with each other and they can escape “marketing hype” with product reality. Being in open source allows InfluxData to stand out because our technology is great, the developers say it is great, and no amount of marketing budget can buy developer passion. This passion translates into usage and usage is the lifeblood of our company.
Q: Looking ahead, what emerging trends do you think will impact the Age of Instrumentation? How is InfluxData addressing these?
A: We see numerous trends that have a great impact on the Age of Instrumentation, and vice versa. Without instrumentation, analysis, machine learning/artificial intelligence and automated decisions and actions, these trends could not be possible.
There’s a vast movement toward autonomy happening in many industries, including manufacturing, automotive, healthcare, information technology, and others. Perhaps the best known examples today are autonomous vehicles, or self-driving cars. They are quickly becoming a reality, with companies such as Tesla, Uber, Waymo and Apple striving to bring fully autonomous vehicles to the open road. Meanwhile, partially autonomous cars are all around us, with features such as automatic emergency braking and hands-off self-parking.
Autonomous systems are important in other industries as well. In the information technology space, the big trend is toward containers and virtualization, where servers and microservices are spun up and down as needed to support varying workloads. In manufacturing, “Industry 4.0” is the new industrial revolution that utilizes automation and data exchange of cyber-physical systems to create a smart factory that includes machines that can make decisions for themselves.
For these types of systems, instrumentation is the first step toward autonomy. Measuring and understanding what is going on with thousands or millions of individual elements is vital. The next step is to observe and determine if we are measuring the right kinds of data, and then make sense of that data. Then we need to learn from the observations and try to take some actions from them. In the learning process, we might want to teach the system what to do, but after a while, we want it to learn for itself, so machine learning is very important. For example, we would want the vehicle to learn how much pressure to apply to the brakes if the road surface is slick and the tires are under-inflated.
Of course, all of this data needs to be ingested quickly so that it can be analyzed, and actions can be taken while they still matter. The trend toward the Age of Autonomy is upon us in many industries, and it’s only possible because the Age of Instrumentation has arrived as well.
About InfluxData
InfluxData, the creator of InfluxDB, delivers a modern Open Source Platform built from the ground up for analyzing metrics and events (time series data) for DevOps and IoT applications. Whether the data comes from humans, sensors, or machines, InfluxData empowers developers to build next-generation monitoring, analytics, and IoT applications faster, easier, and to scale delivering real business value quickly. Based in San Francisco, InfluxData’s more than 420 customers include Cisco, eBay, IBM and Siemens.