Opensearch Vector Search

Amazon OpenSearch vector search expert knowledge base. Comprehensive guidance on vector search configuration, cluster tuning, quantization, cost optimization...

Security Scan

The skill's declared requirements, optional AWS credentials, and included scripts align with its stated purpose (OpenSearch vector search guidance, pricing, and optional read-only cluster analysis), but exercise caution before running the provided scripts and before giving any credentials.

Purpose & Capability

The name/description (OpenSearch vector search KB + live analysis + pricing) matches the included artifacts: reference docs, a cluster analyzer script, and an AWS pricing script. Declared tools (python3, boto3, opensearch-py) and optional AWS env vars are proportional to the pricing and analysis features.

Instruction Scope

SKILL.md explicitly warns not to execute reference examples automatically and states the scripts are read-only and run only with explicit user consent. That scope is appropriate, but running the analyzer requires the user to provide cluster credentials — that grants the script read access to cluster state and metadata, so users should only provide least-privilege credentials and inspect the script before running. The SKILL.md's instructions are reasonably scoped but rely on operator discipline.

Install Mechanism

This is instruction-only (no install spec). There are two Python scripts included but no automatic installation or remote download steps. No high-risk download URLs or archive extraction were specified in the manifest.

Credentials

No required environment variables are forced; AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are declared optional and explicitly documented as only necessary for running the pricing script. No unrelated credentials, config paths, or excessive env requirements are requested.

Persistence & Privilege

The skill does not request persistent presence (always:false) and does not claim to modify other skill configs. It requires explicit user action to run scripts that may open network connections.

Assessment

This skill appears coherent with its stated purpose, but take these precautions before running anything or providing credentials: - Inspect the two scripts (scripts/analyze_cluster.py and scripts/get_opensearch_pricing.py) yourself or run them in an isolated environment to confirm they only perform read-only API calls and send data only to expected endpoints (your OpenSearch cluster and AWS Pricing API). - When asked to provide OpenSearch credentials, supply a least-privilege account (read-only, limited indices) rather than an admin user. Prefer short-lived credentials if possible. - For AWS pricing, use an IAM user/role with minimal permissions required for the Pricing API (or run the script locally under your own AWS profile) and rotate any credentials after use. - Do not allow the agent to automatically execute example curl/PUT/POST snippets from the references; treat those as documentation only. - Review network activity (outbound connections) when running the scripts to ensure no unexpected endpoints receive data. If you want additional assurance, paste the full contents of the two scripts here (or review them locally) and I can point out any lines that attempt to write, modify indices, or contact unexpected hosts. If you cannot review the code, consider running it on a disposable machine or container with no credentials first.

OpenSearch Vector Search Expert

GitHub: norrishuang/opensearch-vector-search-skill — Issues, PRs, and new reference contributions are welcome!

Safety Notes

• Pricing script (scripts/get_opensearch_pricing.py): Makes outbound HTTPS requests to the AWS Pricing API (pricing.us-east-1.amazonaws.com). Requires boto3 and valid AWS credentials. The script is read-only (fetches public pricing data) and does not modify any AWS resources. Only run it when the user explicitly requests cost estimation.
• Reference examples: Code snippets in references/ contain example API calls to localhost:9200 (standard OpenSearch endpoint). These are documentation examples only — do NOT execute them automatically. Present them to the user as configuration references.
• Cluster analyzer (scripts/analyze_cluster.py): Connects to a user-provided OpenSearch cluster and performs read-only analysis. It NEVER creates, modifies, or deletes any indices or data. Only run it when the user explicitly provides cluster credentials (URL + username/password).

Knowledge Base Structure

Read the corresponding reference file based on the question type:

Question Type	Reference File	Keywords
Vector search, k-NN, HNSW, disk mode	references/vector-search.md	vector, knn, hnsw, warmup, disk mode, on_disk
Quantization techniques	references/quantization-techniques.md	quantization, compression, binary, byte, fp16, product quantization
Cost optimization, instance sizing, memory calc	references/cost-optimization.md	cost, pricing, instance, memory calculation, cluster sizing, budget
Cluster tuning, JVM, thread pools	references/cluster-tuning.md	JVM, heap, thread pool, node role, shard allocation
Performance benchmarks, dataset sizing	references/performance-benchmarks.md	benchmark, QPS, latency, recall, dataset size
Indexing strategies, mapping	references/indexing-strategies.md	index, mapping, shard, replica, lifecycle
Query optimization	references/query-optimization.md	query, filter, aggregation, cache, pagination
Optimized instances (OR1/OR2/OM2/OI2)	references/optimized-instances.md	optimized, OR1, OR2, OM2, OI2, S3 durability, indexing throughput
Live cluster analysis	scripts/analyze_cluster.py	analyze cluster, connect, diagnose, review config, health check

Core Workflows

1. Answering Vector Search Configuration Questions

1. Read references/vector-search.md
2. Recommend in-memory mode or disk mode based on user scenario (latency requirements, data scale, QPS)
3. Provide specific mapping JSON configuration
4. Recommend FAISS engine + cosine similarity + 7/8 series instances

2. Capacity Planning & Instance Sizing (Most Common Scenario)

After user provides vector count and dimensions:

1. Read references/cost-optimization.md for memory calculation formulas and examples
2. Calculate using the standard HNSW memory formula (source: AWS official blog):

   Unquantized (float32):
     Memory = 1.1 × (4 × d + 8 × m) × num_vectors × (replicas + 1) bytes
   
   Quantized (FAISS engine, compressed vectors in memory):
     FP16 (2x):    Memory = 1.1 × (2 × d + 8 × m) × num_vectors × (replicas + 1)
     Byte (4x):    Memory = 1.1 × (1 × d + 8 × m) × num_vectors × (replicas + 1)
     Binary 4-bit: Memory = 1.1 × (d/2 + 8 × m) × num_vectors × (replicas + 1)
     Binary 2-bit: Memory = 1.1 × (d/4 + 8 × m) × num_vectors × (replicas + 1)
     Binary 1-bit: Memory = 1.1 × (d/8 + 8 × m) × num_vectors × (replicas + 1)
   
   Where: d=vector dimensions, m=HNSW connections (default 16), num_vectors=total vector count
   

3. Apply OpenSearch node memory allocation rules:

   JVM Heap = min(node_memory × 50%, 32GB)
   Remaining memory = node_memory - JVM Heap
   KNN available memory = remaining × 75%  (with knn.memory.circuit_breaker.limit=70%, ~35% of node memory)
   

4. Select instance type, ensuring total cluster KNN available memory > vector index memory requirement
5. Run pricing script for real-time pricing (see below)

3. Cost Estimation (with Real-Time Pricing)

When user needs cost estimation:

1. Complete capacity planning above
2. Run pricing script for real-time prices:

   python3 scripts/get_opensearch_pricing.py --region <region> --instance-type <type>
   

3. Calculate monthly cost:

   Instance cost = unit_price × node_count × (1 + replica_count)
   EBS cost = capacity(GB) × $0.08 + additional IOPS charges
   Total cost = Instance cost + EBS cost
   

4. Compare cost differences across quantization options

4. Live Cluster Analysis (When User Provides Cluster Credentials)

When the user provides an OpenSearch cluster URL and credentials, use the cluster analyzer to connect and review their vector search configuration. This is read-only — never modify the cluster.

Prerequisites: User must explicitly provide:

• Cluster URL (e.g., https://my-cluster.us-east-1.es.amazonaws.com)
• Username and password (basic auth), OR --no-auth for clusters without authentication

Workflow:

1. Ask for credentials if not provided: URL, username, password
2. Run cluster overview to get health, nodes, and k-NN index list:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action cluster-overview -f pretty
   

3. Analyze specific index if user specifies one, or pick the most important k-NN index:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action index-detail --index <index_name> -f pretty
   

4. Analyze shard distribution for the target index:

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action shard-analysis --index <index_name> -f pretty
   

5. Run all analyses at once (for a comprehensive report):

   python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> --action all --index <index_name> -f pretty
   

6. Interpret the JSON output and present findings to the user:
   • Cluster health status and node resource utilization
   • Vector field configurations (engine, dimensions, HNSW params, quantization)
   • Memory estimates vs actual cluster capacity
   • Auto-generated recommendations (from the script)
7. Provide actionable advice based on findings:
   • Suggest better engine/quantization if needed (provide example mapping JSON)
   • Suggest instance resizing if memory is over/under-provisioned
   • Suggest shard rebalancing if distribution is uneven
   • NEVER execute write operations — only provide example configurations for the user to apply

Cluster Analyzer Script Reference:

Usage:
  python3 scripts/analyze_cluster.py --url <url> -u <user> -p <pass> [options]

Actions:
  --action cluster-overview   Cluster health, nodes, k-NN stats, and all k-NN index summary (default)
  --action index-detail       Deep dive into a specific index's vector config + memory estimates
  --action shard-analysis     Shard distribution and sizing for a specific index
  --action all                Run all analyses

Options:
  --index <name>     Target a specific index (required for index-detail and shard-analysis)
  --no-auth          Connect without authentication
  --verify-ssl       Verify SSL certificates (default: skip)
  --format pretty    Human-readable JSON output

Output: JSON with these top-level keys:
  - cluster_overview: health, version, nodes (memory/CPU/JVM), knn_stats
  - knn_indices: list of all k-NN enabled indices with vector field summaries
  - index_detail/index_details: vector field configs, memory estimates, search stats
  - shard_analysis/shard_analyses: shard distribution across nodes
  - recommendations: auto-generated optimization suggestions with severity levels

Safety constraints for live cluster analysis:

• The script is strictly read-only (uses only GET/CAT APIs)
• NEVER create, update, or delete indices on the user's cluster
• NEVER change cluster settings or mappings
• Only provide example JSON configurations for the user to review and apply themselves
• If the user asks to apply changes, provide the exact API calls/JSON but let the user execute them

Pricing Script Usage

# Query all instance prices for a region
python3 scripts/get_opensearch_pricing.py --region us-east-1

# Query specific instance type (no .search suffix needed)
python3 scripts/get_opensearch_pricing.py --region us-east-1 --instance-type r7g.xlarge

# JSON format output (for calculations)
python3 scripts/get_opensearch_pricing.py --region us-east-1 --instance-type r7g.xlarge --format json

Output fields: instance_type, vcpu, memory_gib, price_per_hour_usd, price_per_month_usd, network

Recommended Defaults

Always recommend these defaults unless user has specific requirements:

• Engine: FAISS
• Similarity: cosine
• Instance family (Gen 7+ only, never recommend older generations):
  • Vector search (k-NN): r7g/r8g/r8gd (memory-optimized, lowest search latency; r8g Graviton4 ~30% faster than r7g)
  • Indexing-heavy + vector: OR2 (optimized, S3 durability, good memory-to-price ratio)
  • Indexing-heavy (no vector): OM2 (highest indexing throughput, 15% faster than OR1)
  • Large dataset with NVMe: OI2 (storage-optimized, no EBS needed)
  • Do NOT recommend: r6g, r5, m5, c5, i3, or any older instance families
• HNSW parameters: ef_construction=512, m=16
• Quantization preference: Byte (4x) for production, Binary (32x) for aggressive cost optimization
• Disk mode threshold: Consider when data > 50M vectors and 100-200ms latency is acceptable

Instance Selection Decision Tree

Is this primarily a vector search (k-NN) workload?
├─ YES → r7g/r8g/r8gd (best search latency, standard EBS; prefer r8g for Graviton4)
│        └─ Need S3 durability? → OR2 (accept 10s refresh interval tradeoff)
├─ Mixed (logs + vectors) → OR2 for log nodes, r7g/r8g for vector nodes
└─ NO (logs/observability/analytics)
   ├─ Write-heavy → OM2 (highest ingest throughput)
   ├─ Balanced → OR2 (good all-around with S3 durability)
   └─ Need NVMe IOPS → OI2

Response Template

Organize cost/sizing answers in this structure:

1. Requirements confirmation: Vector count, dimensions, QPS, latency requirements
2. Memory calculation: Raw size → quantized size → required KNN memory
3. Cluster configuration: Instance type × count, shards, replicas
4. Cost estimation: Instance cost + EBS cost = monthly total
5. Optimization suggestions: Quantization comparison, Reserved Instance discounts