LangSmith CLI

Query and analyze LangSmith traces with natural language or structured commands. Use when the user asks about LangSmith runs, trace failures, latency, cost,...

Security Scan

The skill's code, instructions, and required credential (LANGSMITH_API_KEY) are consistent with its stated purpose of querying LangSmith traces; nothing in the package indicates unrelated privileges or hidden exfiltration.

Purpose & Capability

Name/description match the behavior: the script calls LangSmith endpoints and requires a single LANGSMITH_API_KEY. No extraneous services, binaries, or credentials are requested.

Instruction Scope

SKILL.md and the script restrict network I/O to the LangSmith API and print fetched trace data to stdout for local analysis. Note: printing traces to stdout is necessary for 'ask' usage but means sensitive trace contents will be visible to the agent and any destinations the agent is configured to use.

Install Mechanism

No install spec; the skill is instruction-only with an included Python script. Nothing is downloaded at install time and no archives or external installers are used.

Credentials

Only LANGSMITH_API_KEY is required and is the declared primary credential. The key is used solely as an HTTP header to api.smith.langchain.com; no other secrets or unrelated env vars are requested.

Persistence & Privilege

The skill is not always-enabled and does not request system-wide persistence or modify other skills. It can be invoked autonomously by the agent (platform default) but the package itself does not escalate privileges.

Assessment

This skill appears to do exactly what it claims: fetch LangSmith traces and print summaries. Before installing, verify you intend the agent to see raw trace inputs/outputs (these may contain secrets or PII). Confirm you trust the environment's agent configuration so that printed traces are not forwarded to external LLMs you don't control. Review scripts/langsmith.py yourself (it's included) and limit the LANGSMITH_API_KEY scope or rotate the key if you are unsure. If you want to be extra cautious, run the script locally from a terminal rather than allowing autonomous agent runs, and test it on a non-production project first.

LangSmith CLI Skill

CLI: scripts/langsmith.py. Requires LANGSMITH_API_KEY in env (or ~/.zshrc).

No second API key needed — the ask command fetches and formats traces as structured context for your agent to analyze. No trace data is sent to any third-party LLM.

Commands

Tier 0 — Ask (agent Q&A over traces)

python3 scripts/langsmith.py ask "<question>" --project <name> [--since 24h] [--limit 50]

Fetches recent runs and prints them as structured JSON context. Your agent reads the output and answers the question — no external LLM calls, no data leaving your machine beyond the LangSmith API.

Examples:

• ask "why is my chain slow this week" --project my-project
• ask "what do failing runs have in common" --project my-project --since 7d
• ask "did the system prompt change on Friday affect output quality" --project my-project

Tier 1 — Situational Awareness

python3 scripts/langsmith.py runs <project> [--since 2h] [--status error|success] [--limit 20]
python3 scripts/langsmith.py cost <project> [--since 7d]       # token spend by chain/node
python3 scripts/langsmith.py latency <project> [--since 24h]   # p50/p95/p99 per run name

Tier 2 — Before/After Comparisons

python3 scripts/langsmith.py diff <project> --before <ISO_date> --after <ISO_date>
python3 scripts/langsmith.py prompt-diff <run_id_a> <run_id_b>

diff compares avg latency, error rate, cost, output length across two time windows. prompt-diff shows side-by-side system prompts + outputs for two specific runs.

Tier 3 — Deep Analysis (stubs, expand as needed)

python3 scripts/langsmith.py cluster-failures <project> [--since 7d]
python3 scripts/langsmith.py replay <run_id>

Auth Setup

export LANGSMITH_API_KEY=<your-key>
# or add to ~/.zshrc

Test with: python3 scripts/langsmith.py runs <project> --limit 3

Security & Data Flow

This skill makes outbound network requests only to api.smith.langchain.com (the LangSmith API). That's it.

• LANGSMITH_API_KEY — sent as an HTTP header to api.smith.langchain.com only. Never logged or stored.
• Trace data — fetched from LangSmith and printed to stdout for your agent to read. No trace data is sent to any third-party LLM or external service.
• No second API key required — the ask command outputs structured trace context for your existing agent to analyze, rather than making its own LLM calls.
• No telemetry — the script collects no usage data.

The script is ~300 lines of pure Python with no obfuscation. Audit it at scripts/langsmith.py.

API Reference

See references/langsmith-api.md for endpoint details and run object schema.