{"skill":{"slug":"elasticsearch-skill","displayName":"Elasticsearch","summary":"Interact with Elasticsearch and Kibana via REST API using curl. Use when querying, indexing, managing indices, checking cluster health, writing aggregations,...","description":"---\nname: elasticsearch\ndescription: >\n  Interact with Elasticsearch and Kibana via REST API using curl. Use when querying, indexing,\n  managing indices, checking cluster health, writing aggregations, deploying dashboards, or\n  troubleshooting Elasticsearch. Requires cluster URL and API key. Covers: search (Query DSL),\n  CRUD operations, index management, mappings, aggregations, cluster health, ILM, ES|QL,\n  Kibana API (dashboards, data views, saved objects), OpenTelemetry data patterns, and common\n  troubleshooting patterns.\n---\n\n# Elasticsearch\n\nAll Elasticsearch interaction is via REST API using `curl`. No SDK or client library required.\n\n## Authentication\n\nEvery request needs the cluster URL and an API key:\n\n```bash\n# Set these for your session (or export in .env / shell profile)\nES_URL=\"https://your-cluster.es.cloud.elastic.co:443\"\nES_API_KEY=\"your-base64-api-key\"\n\n# All requests follow this pattern:\ncurl -s \"${ES_URL%/}/<endpoint>\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '<json-body>'\n```\n\n**API key format:** Base64-encoded `id:api_key` string. Pass as-is in the `Authorization: ApiKey` header.\n\nIf the user provides a URL and key, export them as `ES_URL` and `ES_API_KEY` before running commands.\n\n**Important — variable expansion in curl:**\n- Always use `$(printenv ES_API_KEY)` instead of `$ES_API_KEY` in curl headers. The `$ES_API_KEY` variable may not expand correctly in the shell, resulting in empty `Authorization` headers and 401 errors.\n- Always use `${ES_URL%/}` to strip any trailing slash from the URL, preventing double-slash path issues (e.g., `//_cluster/health`).\n\n## Quick Health Check\n\n```bash\n# Cluster health (green/yellow/red) — NOT available on serverless\ncurl -s \"${ES_URL%/}/_cluster/health\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Node stats summary — NOT available on serverless\ncurl -s \"${ES_URL%/}/_cat/nodes?v&h=name,heap.percent,ram.percent,cpu,load_1m,disk.used_percent\"  \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n\n# Index overview (works on both serverless and traditional)\ncurl -s \"${ES_URL%/}/_cat/indices?v&s=store.size:desc&h=index,health,status,docs.count,store.size\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n```\n\n**Serverless Elasticsearch:** If you get `api_not_available_exception` errors, the cluster is running in serverless mode. The following APIs are **not available** in serverless:\n- `_cluster/health`, `_cluster/settings`, `_cluster/allocation/explain`, `_cluster/pending_tasks`\n- `_cat/nodes`, `_cat/shards`\n- `_nodes/hot_threads`, `_nodes/stats`\n- ILM APIs (`_ilm/*`)\n\nUse `_cat/indices` and `_search` APIs as the starting point instead — these work everywhere.\n\n## Search (Query DSL)\n\n```bash\n# Simple match query\ncurl -s \"${ES_URL%/}/my-index/_search\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"query\": { \"match\": { \"message\": \"error timeout\" } },\n    \"size\": 10\n  }' | jq .\n\n# Bool query (must + filter + must_not)\ncurl -s \"${ES_URL%/}/my-index/_search\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"query\": {\n      \"bool\": {\n        \"must\": [ { \"match\": { \"message\": \"error\" } } ],\n        \"filter\": [ { \"range\": { \"@timestamp\": { \"gte\": \"now-1h\" } } } ],\n        \"must_not\": [ { \"term\": { \"level\": \"debug\" } } ]\n      }\n    },\n    \"size\": 20,\n    \"sort\": [ { \"@timestamp\": { \"order\": \"desc\" } } ]\n  }' | jq .\n```\n\nFor full Query DSL reference (term, terms, range, wildcard, regexp, nested, exists, multi_match, etc.), see [references/query-dsl.md](references/query-dsl.md).\n\n## Index & Document Operations\n\n```bash\n# Create index with mappings\ncurl -s -X PUT \"${ES_URL%/}/my-index\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"settings\": { \"number_of_shards\": 1, \"number_of_replicas\": 1 },\n    \"mappings\": {\n      \"properties\": {\n        \"message\":    { \"type\": \"text\" },\n        \"@timestamp\": { \"type\": \"date\" },\n        \"level\":      { \"type\": \"keyword\" },\n        \"count\":      { \"type\": \"integer\" }\n      }\n    }\n  }'\n\n# Index a document (auto-generate ID)\ncurl -s -X POST \"${ES_URL%/}/my-index/_doc\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"message\": \"hello world\", \"@timestamp\": \"2026-01-31T12:00:00Z\", \"level\": \"info\" }'\n\n# Index with specific ID\ncurl -s -X PUT \"${ES_URL%/}/my-index/_doc/doc-123\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"message\": \"specific doc\", \"level\": \"warn\" }'\n\n# Get document\ncurl -s \"${ES_URL%/}/my-index/_doc/doc-123\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Update document (partial)\ncurl -s -X POST \"${ES_URL%/}/my-index/_update/doc-123\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"doc\": { \"level\": \"error\" } }'\n\n# Delete document\ncurl -s -X DELETE \"${ES_URL%/}/my-index/_doc/doc-123\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n\n# Bulk operations (newline-delimited JSON)\ncurl -s -X POST \"${ES_URL%/}/_bulk\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/x-ndjson\" \\\n  --data-binary @- << 'EOF'\n{\"index\":{\"_index\":\"my-index\"}}\n{\"message\":\"bulk doc 1\",\"level\":\"info\",\"@timestamp\":\"2026-01-31T12:00:00Z\"}\n{\"index\":{\"_index\":\"my-index\"}}\n{\"message\":\"bulk doc 2\",\"level\":\"warn\",\"@timestamp\":\"2026-01-31T12:01:00Z\"}\nEOF\n```\n\n## Aggregations\n\n```bash\n# Terms aggregation (top values)\ncurl -s \"${ES_URL%/}/my-index/_search?size=0\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"aggs\": {\n      \"levels\": { \"terms\": { \"field\": \"level\", \"size\": 10 } }\n    }\n  }' | jq '.aggregations'\n\n# Date histogram + nested metric\ncurl -s \"${ES_URL%/}/my-index/_search?size=0\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"query\": { \"range\": { \"@timestamp\": { \"gte\": \"now-24h\" } } },\n    \"aggs\": {\n      \"over_time\": {\n        \"date_histogram\": { \"field\": \"@timestamp\", \"fixed_interval\": \"1h\" },\n        \"aggs\": {\n          \"avg_count\": { \"avg\": { \"field\": \"count\" } }\n        }\n      }\n    }\n  }' | jq '.aggregations'\n```\n\nFor more aggregation types (cardinality, percentiles, composite, filters, significant_terms, etc.), see [references/aggregations.md](references/aggregations.md).\n\n## Mappings & Index Management\n\n```bash\n# Get mapping\ncurl -s \"${ES_URL%/}/my-index/_mapping\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Add field to existing mapping (mappings are additive — you can't change existing field types)\ncurl -s -X PUT \"${ES_URL%/}/my-index/_mapping\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"properties\": { \"new_field\": { \"type\": \"keyword\" } } }'\n\n# Reindex (change mappings, rename index, etc.)\ncurl -s -X POST \"${ES_URL%/}/_reindex\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"source\": { \"index\": \"old-index\" },\n    \"dest\":   { \"index\": \"new-index\" }\n  }'\n\n# Delete index\ncurl -s -X DELETE \"${ES_URL%/}/my-index\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n\n# Index aliases\ncurl -s -X POST \"${ES_URL%/}/_aliases\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"actions\": [\n      { \"add\": { \"index\": \"my-index-v2\", \"alias\": \"my-index\" } },\n      { \"remove\": { \"index\": \"my-index-v1\", \"alias\": \"my-index\" } }\n    ]\n  }'\n\n# Index templates (for time-series / rollover patterns)\ncurl -s -X PUT \"${ES_URL%/}/_index_template/my-template\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"index_patterns\": [\"logs-*\"],\n    \"template\": {\n      \"settings\": { \"number_of_shards\": 1 },\n      \"mappings\": {\n        \"properties\": {\n          \"message\":    { \"type\": \"text\" },\n          \"@timestamp\": { \"type\": \"date\" }\n        }\n      }\n    }\n  }'\n```\n\n## Cluster & Troubleshooting\n\n> **Note:** Most APIs in this section are **not available on serverless** Elasticsearch. They only work on self-managed or traditional Elastic Cloud deployments.\n\n```bash\n# Allocation explanation (why is a shard unassigned?) — NOT serverless\ncurl -s \"${ES_URL%/}/_cluster/allocation/explain\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{ \"index\": \"my-index\", \"shard\": 0, \"primary\": true }' | jq .\n\n# Pending tasks\ncurl -s \"${ES_URL%/}/_cluster/pending_tasks\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Hot threads (performance debugging)\ncurl -s \"${ES_URL%/}/_nodes/hot_threads\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n\n# Shard allocation\ncurl -s \"${ES_URL%/}/_cat/shards?v&s=store:desc&h=index,shard,prirep,state,docs,store,node\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\"\n\n# Task management (long-running operations)\ncurl -s \"${ES_URL%/}/_tasks?actions=*search&detailed\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Cluster settings (persistent + transient)\ncurl -s \"${ES_URL%/}/_cluster/settings?include_defaults=false\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n```\n\nFor Kibana API operations (dashboards, data views, saved objects, alerting rules), see [references/kibana-api.md](references/kibana-api.md).\n\n## Data Streams & ILM\n\n> **Note:** ILM APIs (`_ilm/*`) are **not available on serverless**. Data stream listing works on both.\n\n```bash\n# List data streams\ncurl -s \"${ES_URL%/}/_data_stream\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n\n# Create ILM policy\ncurl -s -X PUT \"${ES_URL%/}/_ilm/policy/my-policy\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"policy\": {\n      \"phases\": {\n        \"hot\":    { \"actions\": { \"rollover\": { \"max_age\": \"7d\", \"max_size\": \"50gb\" } } },\n        \"warm\":   { \"min_age\": \"30d\", \"actions\": { \"shrink\": { \"number_of_shards\": 1 } } },\n        \"delete\": { \"min_age\": \"90d\", \"actions\": { \"delete\": {} } }\n      }\n    }\n  }'\n\n# Check ILM status for an index\ncurl -s \"${ES_URL%/}/my-index/_ilm/explain\" -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" | jq .\n```\n\n## ES|QL (Elasticsearch Query Language)\n\nFor Elasticsearch 8.11+, ES|QL offers a pipe-based query syntax:\n\n```bash\ncurl -s -X POST \"${ES_URL%/}/_query\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"query\": \"FROM logs-* | WHERE level == \\\"error\\\" | STATS count = COUNT(*) BY service.name | SORT count DESC | LIMIT 10\"\n  }' | jq .\n```\n\nFor querying OpenTelemetry data (OTEL logs, traces, metrics, correlation patterns), see [references/otel-data.md](references/otel-data.md).\n\n## Ingest Pipelines\n\n```bash\n# Create pipeline\ncurl -s -X PUT \"${ES_URL%/}/_ingest/pipeline/my-pipeline\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"processors\": [\n      { \"grok\": { \"field\": \"message\", \"patterns\": [\"%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:msg}\"] } },\n      { \"date\": { \"field\": \"timestamp\", \"formats\": [\"ISO8601\"] } },\n      { \"remove\": { \"field\": \"timestamp\" } }\n    ]\n  }'\n\n# Test pipeline\ncurl -s -X POST \"${ES_URL%/}/_ingest/pipeline/my-pipeline/_simulate\" \\\n  -H \"Authorization: ApiKey $(printenv ES_API_KEY)\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"docs\": [\n      { \"_source\": { \"message\": \"2026-01-31T12:00:00Z ERROR something broke\" } }\n    ]\n  }' | jq .\n```\n\n## Tips\n\n- **Always use `jq`** to format JSON output — Elasticsearch responses are verbose.\n- **`?size=0`** on search requests when you only want aggregations (skip hits).\n- **`_cat` APIs** (`_cat/indices`, `_cat/shards`, `_cat/nodes`) give human-readable tabular output — add `?v` for headers, `?format=json` for JSON.\n- **Scroll/PIT for large exports** — don't use `from`/`size` beyond 10,000 hits. Use search_after + PIT instead.\n- **Field types matter** — `keyword` for exact match/aggs, `text` for full-text search. Check mappings before querying.\n- **Date math in index names** — `logs-{now/d}` resolves to today's date. Useful for time-based indices.\n","topics":["Rest Api","Health"],"tags":{"latest":"0.1.0"},"stats":{"comments":0,"downloads":364,"installsAllTime":14,"installsCurrent":0,"stars":0,"versions":1},"createdAt":1771593610567,"updatedAt":1778991876652},"latestVersion":{"version":"0.1.0","createdAt":1771593610567,"changelog":"Initial release — provides command-line examples for interacting with Elasticsearch and Kibana via REST API using curl.\n\n- Covers authentication, cluster health checks, searching (Query DSL), CRUD and bulk document operations, mappings, indexing, aggregations, and index management.\n- Includes guidance for both traditional and serverless Elasticsearch environments, with notes on unsupported endpoints.\n- Demonstrates common troubleshooting patterns and explains critical variable handling for curl authentication.\n- Reference sections link to additional Query DSL and aggregation commands.","license":null},"metadata":null,"owner":{"handle":"davidgeorgehope","userId":"s17chhxqggdvjs75set5hjtr6h885zdj","displayName":"David Hope","image":"https://avatars.githubusercontent.com/u/30470909?v=4"},"moderation":{"isSuspicious":false,"isMalwareBlocked":false,"verdict":"clean","reasonCodes":["review.llm_review"],"summary":"Review: review.llm_review","engineVersion":"v2.4.24","updatedAt":1779943046521}}