Elastic Cloud customers can now ingest data more simply, quickly, and securely, and the latest updates to the core Elastic Stack provide users with new tools for maximizing performance and exploring their data. The 7.15 release of Elastic Cloud brings new integrations with Google Cloud that allow customers to ingest Google Cloud services data directly […]
Elastic Cloud customers can now ingest data more simply, quickly, and securely, and the latest updates to the core Elastic Stack provide users with new tools for maximizing performance and exploring their data.
The 7.15 release of Elastic Cloud brings new integrations with Google Cloud that allow customers to ingest Google Cloud services data directly into their Elastic Cloud deployments and take advantage of additional network security with Google Cloud Private Service Connect. Plus, the Elastic Stack brings enhancements to Elasticsearch and Kibana including improved data transfer, better resiliency, and more flexible data ingest and analysis.
Ready to roll up your sleeves and get started? We have the links you need:
Introducing the first ever native Google Cloud data source integration for Elastic Cloud — Google Cloud Dataflow. This integration allows customers to ship Pub/Sub, Big Query, and Cloud Storage data directly into Elastic Cloud deployments without having to set up an extra intermediary data shipper, utilizing Google Cloud’s native serverless extract, transform, load (ETL) service. Customers benefit from simplified data architecture and increased speed when ingesting data into Elastic Cloud. Read our blog post on the Pub/Sub integration to learn more.
We’re excited to announce that support for Google Private Service Connect is now generally available. Google Private Service Connect provides private connectivity from Google Cloud virtual private cloud (VPC) to Elastic Cloud deployments. The traffic between Google Cloud and Elastic Cloud deployments on Google Cloud travels only within the Google Cloud network, utilizing Private Service Connect endpoints and ensuring that customer data stays off the Internet. Read the blog post to learn more.
Soon, customers will be able to leverage Amazon Web Services (AWS) ARM-based Graviton2 virtual machines (VMs) for Elastic Cloud deployments running on AWS. VMs running on Graviton2 hardware provide up to 40% better price performance compared to previous generation x86-based instances.
Since the inception of Elasticsearch, we’ve been on a mission to be the best and fastest search engine around. To further this mission, we’ve lowered the costs of storing and searching data, improved cluster resiliency and search performance, lowered memory heap usage, improved storage efficiency, and introduced faster aggregations in multiple Elasticsearch releases. In this release, we not only improve data resiliency but also reduce data transfer traffic — a change designed specifically to lower our customers’ Elastic Cloud bills.
By compressing specific inter-node traffic and using snapshot storage to shortcut relocating shards between nodes, we have reduced the amount of network traffic that traverses across the cluster, resulting in a reduction in Data Transfer and Storage (DTS) cost. This change will be most prominent for Elastic Cloud customers with heavy indexing or data migration between tiers.
The best decisions are always data driven. Three new experimental APIs in 7.15 give you the tools to help analyze how you are using Elasticsearch usage and ultimately drive improved performance.
The field usage API helps you decide how to index a field based on usage statistics. For example, if a field is used frequently, it should be created with schema on write or at ingest time by using a mapping. If the field is used infrequently, consider defining it at query time with runtime fields. Changing a text field with an
inverted_index.term_frequencies of zero and low
match_only_text (added in 7.14) can save around 10% of disk.
With the index disk usage API you can see how much disk space is consumed by every field in your index. Knowing what fields take up disk, you can decide which indexing option or field type is best. For example,
match_only_text may be better than
text for certain fields where scoring and positional information is not important. Or, use runtime fields to create a
keyword at query time for flexibility and saving space.
Finally, the vector tiles API provides a huge performance and scalability improvement when searching geo_points and geo_shapes drawn to a map (through use of vector tiles). Offloading these calculations to the local GPU significantly improves performance while also lowering costs by reducing network traffic both within the cluster and to the client.
Elastic 7.15 continues to evolve the implementation of runtime fields in Elasticsearch and Kibana.
In Elasticsearch, composite runtime fields enable users to streamline field creation using one Painless script to emit multiple fields, with added efficiencies for field management. Use patterns like grok or dissect to emit multiple fields using one script instead of creating and maintaining multiple scripts. Using existing grok patterns also makes it faster to create new runtime fields and reduces the time and complexity of creating and maintaining regex expressions. This development makes it easier and more intuitive for users to ingest new data like custom logs. See more on runtime fields in Kibana 7.15 below.
Combined with the introduction of composite fields for Elasticsearch (above), a new preview pane in the runtime fields editor in Kibana 7.15 makes it even easier to create fields on the fly. The preview pane empowers users to test and preview new fields before creating them — for example, by evaluating a new script against documents to check accuracy in Index Patterns, Discover, or Lens. In addition, pinning specific fields in the preview pane simplifies script creation. This enhancement also includes better error handling for the editor, all to help streamline the field creation process and allow users to create runtime fields more quickly. More developments for runtime fields are on the horizon as we continue to make previously ingested data easier to parse from Kibana.
Existing Elastic Cloud customers can access many of these features directly from the Elastic Cloud console. If you’re new to Elastic Cloud, take a look at our Quick Start guides (bite-sized training videos to get you started quickly) or our free fundamentals training courses. You can always get started for free with a free 14-day trial of Elastic Cloud. Or download the self-managed version of the Elastic Stack for free.
Read about these capabilities and more in the 7.15 release notes (Elasticsearch, Kibana, Elastic Cloud, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes), and other Elastic 7.15 highlights in the Elastic 7.15 announcement post.
The release and timing of any features or functionality described in this post remain at Elastic’s sole discretion. Any features or functionality not currently available may not be delivered on time or at all.