Open Source Observability, Meet Data Transformation: Grafana 7.0 promises to connect, unify and visualize all your data

Grafana Labs, makers of the popular open source observation platform Grafana, announced the general availability of Grafana 7.0. This comes just months after Grafana Labs scored $ 24 million in Series A funding to double the open source strategy and build what it links to the world’s first open and composable observation platform.

Today’s release provides enhancements to simplify the development of custom plugins and increase the effect, speed and flexibility of visualization. Open source Grafana is among the world’s most popular dashboard solutions and boasts nearly 600,000 active installations and millions of dashboards used across the globe.

Grafana Labs’ plans appear to be well underway. ZDNet joined with CEO Raj Dutt to discuss Grafana 7.0 and the way forward.

There is a plugin to it: Using your programming language selection, enter all the data anywhere

Like many of us at this point, Dutt does business from home. The difference is, it sounds like Dutt’s business is going well, and Dutt seemed to have a genuine enthusiasm for it. After seeing what Grafana 7.0 brings to the table, and hearing how Grafana Labs has been rushing, we think the enthusiasm may be justified.

Dutt noted that Grafana 7.0 is an accumulation of efforts beginning after 6.0, spanning 18,000 commitments and 3,699 pull requests from 362 contributors worldwide. In addition, there are hundreds of company, commercial and plugin data sources and thousands of sample / startup dashboards that support users’ needs both on-site and in the cloud.

This is the first thing that stands out about Grafana 7.0. Like anyone open source product, is due to a good part of Grafana’s success open source community and the contributions the community makes. Grafana Labs is very aware of this and therefore they emphasized the simplification of the plugin development framework for Grafana 7.0.


What Grafana Labs means with compatibility is the ability to integrate data from a variety of data sources. Image: Grafana Labs

Grafana 7.0 includes new plugins as a result of partnerships across Google (Stackdriver / Cloud Monitor), Microsoft (Azure Monitor) and Amazon (Cloudwatch), plus log support with open source Loki and track input from Zipkin and Hunter. It’s “what” and it’s interesting in itself as it adds more data sources to Grafana’s “Big Tent” open source ecosystem.

Equally interesting is the “how”. Dutt put it quite in perspective by saying that what used to take 1,000 lines of code to evolve can now be done with 100 lines of code. It’s an order of magnitude smaller, so we wondered how Grafana 7.0 achieved it. While Grafana has so far been focused on time series data, the new version adopts a broader overview.

Grafana 7.0 features new component libraries, tools, data structures and a completely rebuilt common and unified data and plugin framework based on Apache Arrow. The consistent data structure brought down Apache arrow reduces the effort required to develop plugins. Arrow is a language-agnostic software framework for developing data analysis programs that process column data.

Dutt noted just as important from a developer-friendliness point of view that Grafana’s execution environment for plugins is programming language agnostic. Users can write their plugins in any programming language and Grafana executes them. It sounds like a custom virtual machine and it hardly gets any more developer friendly than that.

There is a data transformation for it: Process and shape your data the way you want it

The other notable feature of Grafana 7.0 is its data transformation capabilities. Grafana plugins can ingest data from various sources, but as Dutt said, until now, if the data you consumed was not in a format that worked for you, you were a bit lucky. Grafana 7.0 changes that by introducing the ability to process and transform data.

Grafana already offered the opportunity to visualize captured data, but now users can also apply data processing rules to transform the data before visualizing it. Dutt said that a shared set of shared data features that were previously duplicated as custom functions in different locations are now part of Grafana’s data processing pipeline, something that all data sources and panels can benefit from.

To achieve this, Grafana introduced a query processing and transformation language and execution environment, which is a pretty impressive feat. In a way, this sounds like one streaming intake and processing of data within Grafana, so we wondered if that is what it is based on. Although Dutt acknowledged the similarity, I have noticed that this is not based on a streaming data platform.

digital transformation.jpg

Grafana 7.0 offers new data entry and processing features as well as GUI enhancements

Queries are automatically generated behind the scenes while users work in one GUI to map and transform their data. Transformations include renaming, summarizing, combining and performing calculations from different panels, placing time series labels in columns, reusing and refining query results across other panels, and more.

This is especially useful for data sources that do not have their own data processing capabilities, such as Logs. At execution time, queries run against incoming data and results appear on Grafana’s dashboards. Users can view, export and perform simple transformations on the underlying source data.

Users can also drill into details about execution queries for faster troubleshooting. Dutt said that at a later date, the plan is to allow power users to intervene behind the scenes and write their queries directly or optimize auto-generated queries.

Concluding with the highlights of the new and remarkable in Grafana 7.0, a new tracking viewer to complement existing support for metrics and logs allows users to trace the path to a single request through a distributed system. Some more usability improvements have been introduced along with the ability to search, discover and secure dashboards for Grafana’s Enterprise version.

Growth, in the cloud and on the ground

Speaking of business, Dutt also touched the business side of Grafana Labs’ path since funding for Series A. The business has grown from 80 to 140 people, and the effects of the closure are minimal, Dutt said, because it’s always been a remote company.

Revenue has grown over 50 percent, Dutt continued to add. What’s interesting about the broader scheme of things is where this growth comes from. According to Dutt, there is a 50/50 split between Grafana’s Enterprise version running on site and Grafana Cloud. This is interesting dynamics, especially considering Grafana Labs’ cloud-first strategy, and overall trend favoring sky-first, also.

Dutt went into this and explained that Grafana Cloud has brought many more customers, but Grafana Enterprise is where the big customers are. The explanation offered by Dutt is that, in addition to the usual reasons that can keep organizations away from the cloud, such as compliance and security, there is another often overlooked factor: data gravity.

These days, we typically use the term data gravity to refer to the fact that as more data moves to the cloud, data storage and processing software also follows. But it can also work the other way around, Dutt pointed out. For large organizations with lots of local data, it makes much less sense to send everything to some cloud.

Installing a version of on-site data storage and processing software makes things a lot easier if most of your data resides there. Except that on-site Dutt was quick to clarify could actually mean your own AWS or Azure or Google Cloud storage. Anything that is not part of your software provider’s offer, basically. Not exactly what we typically relate to using the term on the spot, but overall, the idea makes some sense.

Cloud or onsite, however, Grafana Labs seems to be on a growth path.

Source link