Skip to content

Latest commit

聽

History

History
795 lines (601 loc) 路 50.5 KB

opensearch-release-notes-2.13.0.md

File metadata and controls

795 lines (601 loc) 路 50.5 KB
Raw

OpenSearch and OpenSearch Dashboards 2.13.0 Release Notes

RELEASE HIGHLIGHTS

OpenSearch 2.13 includes several new features designed to help you build AI-powered applications, along with upgrades to help you access and analyze your operational data, and new ways to support resiliency for your OpenSearch clusters.

NEW FEATURES

  • The OpenSearch flow framework is generally available, allowing you to automate the configuration of search and ingest pipeline resources required by advanced search features like semantic, multimodal, and conversational search.
  • AI connectors receive several enhancements, including predefined templates that let you automate setup for machine learning models that are integrated through OpenSearch connectors to APIs such as those from OpenAI, Amazon Bedrock, and Cohere. New settings for AI connectors allow users to configure timeouts and automatically deploy models.
  • The OpenSearch Assistant Toolkit is now generally available, allowing developers to build interactive, AI-powered assistant experiences in OpenSearch that let users query their operational and security log data using natural language.
  • The agent framework is now generally available, allowing you to automate machine learning tasks using agents and tools.
  • OpenSearch now supports guardrails for large language models (LLMs). The agent framework adds support for user-defined regex rules that can be used to filter inappropriate text generation that could be produced by integrated LLMs.
  • You can now index quantized vectors with FAISS engine based k-NN indexes. Instead of storing vectors that require 4 bytes per dimension, you can compress the dimensions down to 2 bytes, which can reduce memory requirements and improve query latency.
  • You can now post-filter hybrid search results, allowing you to apply aggregations to the results to support use cases such as faceting.
  • By default, OpenSearch executes upsert and get operations by executing a term query on document id to determine if the document is present in the index. With this release, users can now control whether fuzzy_set should be enabled to optimize document ID lookups in index or search calls using the Bloom filter data structure, with potential for throughput improvements of up to 30%.
  • This release adds I/O-based admission control, a proactive mechanism designed to support cluster resiliency by protecting against spikes or increases in capacity if a cluster鈥檚 I/O usage breaches a defined threshold.
  • New cross-cluster monitors are added to the alerting plugin, allowing you to manage alerts across clusters. This release also introduces the option to set up a cluster that is dedicated to alerting, separating alerting resources from indexing and querying workloads.

RELEASE DETAILS

[OpenSearch and OpenSearch Dashboards 2.13.0](https://opensearch.org/versions/opensearch-2-3-0.html) includes the following features, enhancements, bug fixes, infrastructure, documentation, maintenance and refactoring updates.

FEATURES

OpenSearch Flow Framework

  • Added create ingest pipeline step (#558)
  • Added create search pipeline step (#569)
  • Added create index step (#574)
  • Added default use cases (#583)

OpenSearch ML Commons

  • Hidden agent (#2204)
  • Auto deployment for remote models (#2206)
  • Support question answering model (#2208)
  • Guardrails for remote model input and output (#2209)

OpenSearch Neural Search

  • Implement document chunking processor with fixed token length and delimiter algorithm (#607)
  • Enabled support for applying default modelId in neural sparse query (#614

OpenSearch Reporting

  • xlsx support (#940)

OpenSearch Security Analytics

  • Findings api enhancements (#914) (#795)
  • Get all findings as part of findings API enhancement (#803)
  • Support object fields in aggregation based sigma rules (#789)
  • Add latest sigma rules (#942)

OpenSearch Skills

  • Fix SearchAnomalyDetectorsTool indices param bug
  • Fix detector state params in SearchAnomalyDetectorsTool
  • Update ppl tool claude model prompts to use <ppl> tags
  • Add parameter validation for PPL tool

Dashboards Observability

  • Add integration installation to data sources flyout (#1568)
  • Integrations: Update delete modal to support custom verify prompt (#1567)
  • Data sources bug fixes and UI improvements (#1565)
  • Allow browsing integrations in Flyout from Data Sources page (#1562)
  • Add auto-suggestions for skipping index definition and export types (#1552)
  • Data Sources component Improvements and bug fixes (#1551)
  • Adding datasource status and filter for hive tables (#1549)
  • Implement redirection to explorer within data sources (#1548)
  • Add actual integration queries to table (#1544)
  • Remove modal for discover redirection (#1543)
  • Acceleration Actions Implementation (#1540)
  • Updating UI for create acceleration flyout (#1532)
  • Add conditional installation for S3 integrations (#1528)
  • Add datasource field in accelerations cache (#1525)
  • Acceleration components' data implementation (#1521)
  • Add Retrieval from Catalog Cache (#1517)
  • Export observability start interface (#1515)
  • Expose create acceleration flyout, update acceleration docs link (#1513)
  • Bump plugin version to 2.13.0 (#1506)
  • Catalog cache and Session update for async queries (#1500)
  • Add flyout pages to associated objects table (#1496)
  • Remove index store region and index store URI for data connection panel (#1490)
  • Accelerations Tab and Flyout Skeletons (#1489)
  • Associated objects searchbar filters (#1474)
  • Data sources associated objects tab (#1470)

OpenSearch Dashboards Reporting

  • Add xlsx support(#275)

OpenSearch Dashboards Search Relevance

  • Feature: Add Option to Expand Document Source (#350) (#378)

OpenSearch Query Workbench

  • Updates create accelerations flyout usage and side tree (#285)

OpenSearch Security Analytics Dashboards

  • Added spinner to better indicate that rules are loading (#905)
  • Rule editor enhancements (#924)
  • Fetch all findings and alerts for the detectors when displaying in the tables (#942)

ENHANCEMENTS

OpenSearch Alerting

  • Enhance per bucket, and per document monitor notification message ctx. (#1450) (#1477)
  • Findings API Enhancements changes and integ tests fix (#1464) (#1474)
  • Feature findings enhancement (#1427) (#1457)
  • Add distributed locking to jobs in alerting (#1403) (#1458)

OpenSearch Custom Codecs

  • Remove unneccessary ArrayUtil.grow in Zstd compression modes. (#121)

OpenSearch Common Utils

  • Add queryFieldNames field in Doc Level Queries (#582) (#597)

Features

  • Fix findings API enhancemnts (#611) (#617)
  • Feature findings enhancemnt (#596) (#606)

OpenSearch Flow Framework

  • Substitute REST path or body parameters in Workflow Steps (#525)
  • Added an optional workflow_step param to the get workflow steps API (#538)
  • Add created, updated, and provisioned timestamps to saved template (#551)
  • Enable Flow Framework by default (#553)
  • Adding new exception type for workflow step failures (#577)

OpenSearch ML Commons

  • Adding connector http timeout in the connector level (#1835)
  • Enable auto redeploy for hidden model (#2102)
  • Add verification to rate limiter number field (#2113)
  • Asymmetric embeddings (#2123)
  • Set the number of ml system index primary shards to 1 (#2137)
  • Prevent exposing internal ip when an agent gets an internal OpenSearch exception (#2154)
  • Change the index update settings to make it only contain dynamic settings (#2156)
  • Add remote predict thread pool (#2207)
  • Add local inference enabling/disabling setting (#2232)
  • Add request level parameters for system_prompt and user_instructions (#2236)
  • Add support for Cohere and other chat model input/outputs in the RAG pipeline (#2238)

OpenSearch Neural Search

  • Adding aggregations in hybrid query (#630)
  • Support for post filter in hybrid query (#633)

OpenSearch Security

  • Admin role for Query insights plugin (#4022)
  • Add query assistant role and new ml system indices (#4143)
  • Redact sensitive configuration values when retrieving security configuration (#4028)
  • v2.12 update roles.yml with new API for experimental alerting plugin feature (#4035)
  • Add deprecate message that TLSv1 and TLSv1.1 support will be removed in the next major version (#4083)
  • Log password requirement details in demo environment (#4082)
  • Redact sensitive URL parameters from audit logging (#4070)
  • Fix unconsumed parameter exception when authenticating with jwtUrlParameter (#4065)
  • Regenerates root-ca, kirk and esnode certificates to address already expired root ca certificate (#4066)
  • Add exclude_roles configuration parameter to LDAP authorization backend (#4043)
  • Refactor and update existing ml roles (#4157)

OpenSearch k-NN

  • Optize Faiss Query With Filters: Reduce iteration and memory for id filter #1402
  • Detect AVX2 Dynamically on the System #1502
  • Validate zero vector when using cosine metric #1501
  • Persist model definition in model metadata [#1527] (opensearch-project/k-NN#1527)
  • Added Inner Product Space type support for Lucene Engine #1551
  • Add Range Validation for Faiss SQFP16 #1493
  • SQFP16 Range Validation for Faiss IVF Models #1557

SQL

OpenSearch Security Dashboards Plugin

  • Clear the contents of opensearch_dashboards prior to putting settings (#1781)
  • Add loose flag to OSD bootstrap (#1789)
  • Hide tenant when disabled in the account nav button popover (#1792)
  • Use start-opensearch and setup-opensearch-dashboards actions (#1808)
  • Fix cookie expiry issues from IDP/JWT auth methods, disables keepalive for JWT/IDP (#1806)
  • Copy tenant with Short URL (#1812)
  • Add toast handling for purge cache action (#1827)

BUG FIXES

OpenSearch Alerting

  • Add an exists check to document level monitor queries (#1425) (#1456)
  • Optimize sequence number calculation and reduce search requests in doc level monitor execution (#1445)
  • Clean up doc level queries on dry run (#1430)
  • Optimize to fetch only fields relevant to doc level queries in doc level monitor instead of entire _source for each doc #1441
  • Add jvm aware setting and max num docs settings for batching docs for percolate queries #1435
  • Fix for MapperException[the [enabled] parameter can't be updated for the object mapping [metadata.source_to_query_index_mapping] (#1432) (#1434)
  • Adding tracking_total_hits in search query for findings (#1487)

OpenSearch Flow Framework

  • Fixing create index and use case input parsing bugs (#600)

OpenSearch Geospatial

  • Adjusted dependency versions to address CVEs (#635)

OpenSearch Index Management

  • Fix Typo in Alias Message (#1137)

OpenSearch ML Commons

  • Fix error code when executing agent (#2120)
  • Fix npe when executing agent with empty parameter (#2145)
  • Fix delete model cache on macOS causing model deploy fail with model (#2180)
  • Adding BWC for connector config field (#2184)
  • Fix onnx dep (#2198)
  • Update the response code to 404 when deleting a memory (#2212)
  • Fix model enable flag not loading (#2221)
  • Updating ml_connector schema version (#2228)
  • Fix json error (#2234)
  • Update remote model auto deploy tests in predict runner (#2237)

OpenSearch Neural Search

  • Fix typo for sparse encoding processor factory(#600)
  • Add non-null check for queryBuilder in NeuralQueryEnricherProcessor (#619)
  • Fix runtime exceptions in hybrid query for case when sub-query scorer return TwoPhase iterator that is incompatible with DISI iterator (#624)

OpenSearch Notifications

  • Adding hostname support for notifications deny list (#858) (#860)

OpenSearch Security Analytics

  • Fix get mappings view API incorrectly returning ecs path (#867) (#905) (#866)
  • Add an "exists" check for "not" condition in sigma rules (#852) (#897)
  • Fix duplicate ecs mappings which returns incorrect log index field in mapping view API (#786) (#788) (#898)
  • ArrayIndexOutOfBoundsException for inconsistent detector index behavior (#843) (#858)
  • Fail the flow when detector type is missing in the log types index (#845) (#857)
  • Remove blocking calls and change threat intel feed flow to event driven (#871) (#876)
  • Fixes OCSF integ test (#918)
  • Pass rule field names in doc level queries during monitor/creation. Remove blocking actionGet() calls (#873)
  • Add search request timeouts for correlations workflows (#893) (#901) (#879])

OpenSearch k-NN

  • Disable sdc table for HNSWPQ read-only indices #1518
  • Switch SpaceType.INNERPRODUCT's vector similarity function to MAXIMUM_INNER_PRODUCT #1532
  • Add patch to fix arm segfault in nmslib during ingestion #1541
  • Share ivfpq-l2 table allocations across indices on load #1558

SQL

Dashboards Observability

  • Update integrations to allow custom checkpoint locations (#1501)
  • (query assist) show error toasts if summary is disabled (#1480)
  • Fixing style overriding issue in dashboards core vizBuilder (#1451)
  • Fix jaeger spans key names for filtering (#1428)

Dashboards Assistant

  • Integrate chatbot with sidecar service. (#164)
  • Update view trace button from suggestions to message action bar. (#158)
  • Fetch root agent id before executing the agent. (#165)
  • Encode id if it is used in request path. (#174)

OpenSearch Alerting Dashboards Plugin

  • Fix fetching of channels for composite monitors (#820)
  • Fix error to disblay monitors with disable/enable state (#869)

OpenSearch Dashboards Reporting

  • Reporting is missing from the navigation menu (#295)
  • Allow negative signs in duration(#284)
  • Fix condition to determine if the date field's value is an array (#299)
  • Fix menu cut off (#315)

OpenSearch Query Workbench

  • Refactor async calls and minor bug fixes (#274)
  • Add empty tree state for SQL sidebar (#292)
  • Catch no database found errors from cache manager (#296)

OpenSearch Security Analytics Dashboards

  • Added more mime types for yaml file (#909)
  • Load log type from log source if present (#894)
  • Update actions menu after start/stop detector action for the selected detector (#895)
  • Loading spinner added; fixed copied popup (#904)
  • Fixed view surrounding logs for aliases (#906)
  • Fetching channel types using API; updated type imports (#919)
  • Fixed UI for setting alert severity (#920)
  • Do not filter timestamp field from required mappings when at least one rule is selected (#925)
  • Fixed create button staying in submit state on review config (#926)
  • Don't show index-pattern creation form once created (#927)
  • Fixed logic to get all alerts for a detector (#965)

INFRASTRUCTURE

OpenSearch Anomaly Detection

  • Fixed lucene url (#1158)

OpenSearch ML Commons

  • Add integration tests for the RAG pipeline covering OpenAI and Bedrock (#2213)

OpenSearch k-NN

  • Manually install zlib for win CI #1513
  • Update k-NN build artifact script to enable SIMD on ARM for Faiss #1543

SQL

Dashboards Assistant

  • Add new workflow to verify binary install. (#159)

Dashboards Observability

  • Add single version flag during bootstrap to fix version conflicts (#1460)

OpenSearch Dashboards Reporting

  • Add single version flag during bootstrap to fix version conflicts(#303)
  • Add workflow to verify binary installation is successful (#313)

OpenSearch Dashboards Search Relevance

  • Add workflow to verify binary installation (#373) (#375)

OpenSearch Dashboards Visualizations

  • Add workflow to verify binary installation (#347)
  • [CI/CD] Add single version flag during bootstrap to fix version conflicts (#341)

OpenSearch ML Commonss Dashboards

  • Add new workflow to verify binary install. (#306)

OpenSearch Query Workbench

  • Add single version flag during bootstrap to fix version conflicts (#269)
  • Add OSD react as required plugin (#278)

OpenSearch Security Analytics Dashboards

  • Updated cypress version to match core OSD (#907)

OpenSearch Build

  • Fix default config permissions for RPM and Debian Packages (#3815)
  • Allow automatically restarting of OpenSearch and OpenSearch-Dashboards Service after upgrading the Debian package (#4530)

DOCUMENTATION

OpenSearch Alerting

  • Added 2.13 release notes (#1483)

OpenSearch Common Utils

  • Added 2.13.0.0 release notes (#622)

OpenSearch Index Management

  • Version 2.13 Release Notes Draft (#1142)

OpenSearch ML Commons

  • Add Cohere Chat blueprint with RAG (#1991)
  • Add tutorial for semantic search with byte quantized vector and Cohere embedding model (#2127)
  • Add tutorial for rerank pipeline with Cohere rerank model (#2134)
  • Add tutorial for chatbot with rag (#2141)
  • Add tutorial for building your own chatbot (#2144)
  • Add tutorial for CFN template integration (#2161)
  • Fix cohere chat blueprint (#2167)
  • Add demo notebook for creating connector (#2192)
  • Enhance connector helper notebook to support 2.9 (#2202)

OpenSearch Notifications

  • Add 2.13.0 release notes (#878)

OpenSearch Security Analytics

  • Added 2.13.0 release notes (#945)

OpenSearch Alerting Dashboards Plugin

  • Added 2.13 release notes. (#904)

OpenSearch Dashboards Maps

  • Update data layer source name #588

OpenSearch Dashboards Notifications

  • Added 2.13.0 release notes. (#173)

OpenSearch Index Management Dashboards Plugin

  • Version 2.13 Release Notes Draft (#1012)

OpenSearch Security Analytics Dashboards

  • Added release notes for 2.13.0 (#959)
  • Updated release notes for 2.13.0 (#968)

MAINTENANCE

OpenSearch Alerting

  • Bump up the version to 2.13 (#1460)

OpenSearch Anomaly Detection

  • Increment version to 2.13.0-SNAPSHOT (#1156)

OpenSearch Asynchronous Search

  • Increment version to 2.13.0 (#544)

OpenSearch Common Utils

  • Increment version to 2.13.0-SNAPSHOT (#591)

OpenSearch Index Management

  • Increment version to 2.13.0-SNAPSHOT. (#1096)
  • Upgrade com.github.seancfoley:ipaddress to mitigate CVE-2023-50570 (#1126)
  • Updates demo certs and admin keystore (#1116)

OpenSearch Job Scheduler

  • Increment version to 2.13.0 (#575)
  • Spotless plugin remove apply false (#590) (#591)
  • dependabot: bump com.google.googlejavaformat:google-java-format (#596) (#597)
  • dependabot: bump com.netflix.nebula.ospackage from 11.8.0 to 11.8.1 (#593) (#598)
  • dependabot: bump org.slf4j:slf4j-api from 2.0.11 to 2.0.12 (#587) (#599)

OpenSearch ML Commons

  • Updates sample cert and admin keystore (#2143)
  • Bump common-compress package to fix CVE (#2186)
  • Suppress removal AccessController in java.security has been deprecated and marked for removal (#2195)

OpenSearch Notifications

  • Bump up the version to 2.13 (#873)

OpenSearch Observability

  • Increment version to 2.13.0-SNAPSHOT (#1796)

OpenSearch Security

  • Add exlusion for logback-core to resolve CVE-2023-6378 (#4050)
  • Bump com.netflix.nebula.ospackage from 11.7.0 to 11.8.1 (#4041, #4075)
  • Bump Wandalen/wretry.action from 1.3.0 to 1.4.10 (#4042, #4092, #4108, #4135)
  • Bump spring_version from 5.3.31 to 5.3.33 (#4058, #4131)
  • Bump org.scala-lang:scala-library from 2.13.12 to 2.13.13 (#4076)
  • Bump com.google.googlejavaformat:google-java-format from 1.19.1 to 1.21.0 (#4078, #4110)
  • Bump ch.qos.logback:logback-classic from 1.2.13 to 1.5.3 (#4091, #4111)
  • Bump com.fasterxml.woodstox:woodstox-core from 6.6.0 to 6.6.1 (#4093)
  • Bump kafka_version from 3.5.1 to 3.7.0 (#4095)
  • Bump jakarta.xml.bind:jakarta.xml.bind-api from 4.0.1 to 4.0.2 (#4109)
  • Bump org.apache.zookeeper:zookeeper from 3.9.1. to 3.9.2 (#4130)
  • Bump org.awaitility:awaitility from 4.2.0 to 4.2.1 (#4133)
  • Bump com.google.errorprone:error_prone_annotations from 2.25.0 to 2.26.1 (#4132)

OpenSearch Skills

  • Update mockito monorepo to v5.10.0 (#128) (#197)
  • Update dependency org.apache.commons:commons-lang3 to v3.14.0 (#47)
  • Update dependency org.apache.commons:commons-text to v1.11.0 (#62)
  • Update plugin io.freefair.lombok to v8.6 (#245) (#249)
  • Update plugin de.undercouch.download to v5.6.0 (#239) (#250)
  • Update plugin com.diffplug.spotless to v6.25.0 (#127) (#252)
  • Update dependency org.json:json to v20240205 (#246) (#251)

OpenSearch Security Analytics

  • Increment to 2.13. (#913)
  • Add goyamegh as a maintainer (#868) (#899)

OpenSearch k-NN

  • Bump faiss lib commit to 32f0e8cf92cd2275b60364517bb1cce67aa29a55 #1443
  • Fix FieldInfo Parameters Mismatch #1490
  • Upgrade faiss to 12b92e9 #1509

SQL

Dashboards Observability

  • Copy Updates: Integration Flows -> Integration Resources (#1555)
  • Update UI styles for query assist (#1523)
  • Move create acceleration flyout from workbench to datasources (#1508)
  • Minor integration name updates (#1505)
  • Update integration format for better handling of multiple asset types (#1502)
  • Update names and descriptions for integrations (#1499)
  • (query assist) get agent id through config API (#1482)
  • Changed Explorer Data Grid useage of timestamp (#1479)
  • Flint bug fix explorer failure (#1476)
  • Fixing Flaky Panels Test (#1463)
  • Remove hardcoded width for generate ppl button (#1447)
  • Upgrade plotly to v2 (#1432)

OpenSearch Alerting Dashboards Plugin

  • Increment version to 2.13.0.0 (#892)

OpenSearch Anomaly Detection Dashboards

  • Increment version to 2.13.0.0 (#695)

OpenSearch Dashboards Notifications

  • Update snapshot. (#168)

OpenSearch Dashboards Visualizations

  • Upgrade plotly to v2 (#336)
  • Increment version to 2.13.0.0 (#348)

OpenSearch Index Management Dashboards Plugin

  • Increment version to 2.13.0.0. (#1000)
  • CVE fixes for CVE-2022-38900, CVE-2021-3807, CVE-2021-43138, CVE-2022-25858, CVE-2022-37603, CVE-2023-26136, CVE-2022-37599, CVE-2022-37601 (#1008)

OpenSearch ML Commonss Dashboards

  • Increment version to 2.13.0.0 (#307)

OpenSearch Query Workbench

  • Add workflow to verify binary install (#272)

OpenSearch Security Analytics Dashboards

  • Added riysaxen-amzn as a maintainer (#898)

REFACTORING

OpenSearch Flow Framework

  • Moved workflow-steps.json to Enum (#523)
  • Refactored logging for consistency (#524)

OpenSearch ML Commons

  • Refactor memory logs (#2121)
  • Parse tool input to map (#2131)

OpenSearch Security Analytics

  • Refactor invocation of Action listeners in correlations (#880) (#900) (#879])

SQL