Dremio, supports of Apache arrow, which is focused on brochure for querying data tools for BI tools, has introduced a new Amazon Web Services (AWS) version of the product. This new version is tailor made for AWS Simple Storage Service (S3) data lakes and found in AWS Marketplaceand introduces two new features that the company calls resilient engines and parallel projects.
Parallel projects provide a multi-tenant-like approach to implementations of the Dremio platform. As the name suggests, they allow implementations per. Project, all under a single client account. Elastic motors meanwhile allow multiple implementations before a project based on user-specified templates that defines the number of nodes in the “engine” (which is really a cluster) and a defined period of inactivity, after which it automatically shuts down. Furthermore, since Dremio’s “reflections” (a pre-aggregation structure based on Apache parquet and Apache Arrow) continue with the S3, they can speed up queries, even for engines that have stopped and restarted.
The goal of Elastic Engines is to provide high performance and optimized billing of cloud resources through the definition of both high-scale and more modest engine profiles, based on the expected user role and associated workload requirements. Certain roles can be assigned to engines defined by a large number of nodes, perform their query work quickly by parallelizing the workload and then shutting down. Roles associated with smaller, ad hoc workloads can be assigned to smaller engines that can be run longer, but with an effective smaller number of billable Amazon Elastic Compute Cluster (EC2) resources.
As a market product, Dremio AWS Edition is not a software like a service (SaaS) offering. But with a multi-tenant, multi-cluster approach and a simple portal experience to implement, Dremio intentionally creates a SaaS-like experience. Furthermore, with the granularity provided by Elastic Engines, the company can get its customers close to a pay-for-what-you billing model.
Dremio’s evolving orientation to cloud models, first with regard to cloud lakes and now with multi-tenancy, ease of use and more granular cloud billing models, aligns with trends in the broader data and analytics space. Customers have moved to cloud object storage for their data lakes and even data warehouses, and want platforms that are either serverless or reliant on volatile data processing resources to enable usage-based billing.
Also read: Data Lake Engines for AWS and Azure Releases
For independent companies to compete with services offered by cloud providers themselves, it is necessary to turn to cloud delivery and billing models. Dremio has done what it needed to do here, and will likely do it again for other cloud platforms. Beyond that, for example, making things more automated by defining and allocating engines implicitly will add even more value.