Multi-source retrieval with confidence scoring - web, academic, and Tavily in one unified API

Intelligent search for agents. Multi-source retrieval with confidence scoring - web, academic, and Tavily in one unified API.

0xjordansg-yolo@0xjordansg-yolo

Install

openclaw skills install @0xjordansg-yolo/openclaw-aisa-search-website-academic-tavily-serp-exa

OpenClaw Search 🔍

Intelligent search for autonomous agents. Powered by AIsa.

One API key. Multi-source retrieval. Confidence-scored answers.

Inspired by AIsa Verity - A next-generation search agent with trust-scored answers.

🔥 What Can You Do?

Research Assistant

text

"Search for the latest papers on transformer architectures from 2024-2025"

Market Research

text

"Find all web articles about AI startup funding in Q4 2025"

Competitive Analysis

text

"Search for reviews and comparisons of RAG frameworks"

News Aggregation

text

"Get the latest news about quantum computing breakthroughs"

Deep Dive Research

text

"Smart search combining web and academic sources on 'autonomous agents'"

Quick Start

bash

export AISA_API_KEY="your-key"

🏗️ Architecture: Multi-Stage Orchestration

OpenClaw Search employs a Two-Phase Retrieval Strategy for comprehensive results:

Phase 1: Discovery (Parallel Retrieval)

Query 4 distinct search streams simultaneously:

Scholar: Deep academic retrieval
Web: Structured web search
Smart: Intelligent mixed-mode search
Tavily: External validation signal

Phase 2: Reasoning (Meta-Analysis)

Use AIsa Explain to perform meta-analysis on search results, generating:

Confidence scores (0-100)
Source agreement analysis
Synthesized answers

text

┌─────────────────────────────────────────────────────────────┐
│                      User Query                              │
└─────────────────────────────────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
        ┌─────────┐     ┌─────────┐     ┌─────────┐
        │ Scholar │     │   Web   │     │  Smart  │
        └─────────┘     └─────────┘     └─────────┘
              │               │               │
              └───────────────┼───────────────┘
                              ▼
                    ┌─────────────────┐
                    │  AIsa Explain   │
                    │ (Meta-Analysis) │
                    └─────────────────┘
                              │
                              ▼
                    ┌─────────────────┐
                    │ Confidence Score│
                    │  + Synthesis    │
                    └─────────────────┘

Core Capabilities

Web Search

bash

# Basic web search
curl -X POST "https://api.aisa.one/apis/v1/scholar/search/web?query=AI+frameworks&max_num_results=10" \
  -H "Authorization: Bearer $AISA_API_KEY"

# Full text search (with page content)
curl -X POST "https://api.aisa.one/apis/v1/search/full?query=latest+AI+news&max_num_results=10" \
  -H "Authorization: Bearer $AISA_API_KEY"

Academic/Scholar Search

bash

# Search academic papers
curl -X POST "https://api.aisa.one/apis/v1/scholar/search/scholar?query=transformer+models&max_num_results=10" \
  -H "Authorization: Bearer $AISA_API_KEY"

# With year filter
curl -X POST "https://api.aisa.one/apis/v1/scholar/search/scholar?query=LLM&max_num_results=10&as_ylo=2024&as_yhi=2025" \
  -H "Authorization: Bearer $AISA_API_KEY"

Smart Search (Web + Academic Combined)

bash

# Intelligent hybrid search
curl -X POST "https://api.aisa.one/apis/v1/scholar/search/smart?query=machine+learning+optimization&max_num_results=10" \
  -H "Authorization: Bearer $AISA_API_KEY"

Tavily Integration (Advanced)

bash

# Tavily search
curl -X POST "https://api.aisa.one/apis/v1/tavily/search" \
  -H "Authorization: Bearer $AISA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"latest AI developments"}'

# Extract content from URLs
curl -X POST "https://api.aisa.one/apis/v1/tavily/extract" \
  -H "Authorization: Bearer $AISA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/article"]}'

# Crawl web pages
curl -X POST "https://api.aisa.one/apis/v1/tavily/crawl" \
  -H "Authorization: Bearer $AISA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","max_depth":2}'

# Site map
curl -X POST "https://api.aisa.one/apis/v1/tavily/map" \
  -H "Authorization: Bearer $AISA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

Explain Search Results (Meta-Analysis)

bash

# Generate explanations with confidence scoring
curl -X POST "https://api.aisa.one/apis/v1/scholar/explain" \
  -H "Authorization: Bearer $AISA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"results":[...],"language":"en","format":"summary"}'

📊 Confidence Scoring Engine

Unlike standard RAG systems, OpenClaw Search evaluates credibility and consensus:

Scoring Rubric

Factor	Weight	Description
Source Quality	40%	Academic > Smart/Web > External
Agreement Analysis	35%	Cross-source consensus checking
Recency	15%	Newer sources weighted higher
Relevance	10%	Query-result semantic match

Score Interpretation

Score	Confidence Level	Meaning
90-100	Very High	Strong consensus across academic and web sources
70-89	High	Good agreement, reliable sources
50-69	Medium	Mixed signals, verify independently
30-49	Low	Conflicting sources, use caution
0-29	Very Low	Insufficient or contradictory data

Python Client

bash

# Web search
python3 {baseDir}/scripts/search_client.py web --query "latest AI news" --count 10

# Academic search
python3 {baseDir}/scripts/search_client.py scholar --query "transformer architecture" --count 10
python3 {baseDir}/scripts/search_client.py scholar --query "LLM" --year-from 2024 --year-to 2025

# Smart search (web + academic)
python3 {baseDir}/scripts/search_client.py smart --query "autonomous agents" --count 10

# Full text search
python3 {baseDir}/scripts/search_client.py full --query "AI startup funding"

# Tavily operations
python3 {baseDir}/scripts/search_client.py tavily-search --query "AI developments"
python3 {baseDir}/scripts/search_client.py tavily-extract --urls "https://example.com/article"

# Multi-source search with confidence scoring
python3 {baseDir}/scripts/search_client.py verity --query "Is quantum computing ready for enterprise?"

API Endpoints Reference

Endpoint	Method	Description
`/scholar/search/web`	POST	Web search with structured results
`/scholar/search/scholar`	POST	Academic paper search
`/scholar/search/smart`	POST	Intelligent hybrid search
`/scholar/explain`	POST	Generate result explanations
`/search/full`	POST	Full text search with content
`/search/smart`	POST	Smart web search
`/tavily/search`	POST	Tavily search integration
`/tavily/extract`	POST	Extract content from URLs
`/tavily/crawl`	POST	Crawl web pages
`/tavily/map`	POST	Generate site maps

Search Parameters

Parameter	Type	Description
query	string	Search query (required)
max_num_results	integer	Max results (1-100, default 10)
as_ylo	integer	Year lower bound (scholar only)
as_yhi	integer	Year upper bound (scholar only)

🚀 Building a Verity-Style Agent

Want to build your own confidence-scored search agent? Here's the pattern:

1. Parallel Discovery

python

import asyncio

async def discover(query):
    """Phase 1: Parallel retrieval from multiple sources."""
    tasks = [
        search_scholar(query),
        search_web(query),
        search_smart(query),
        search_tavily(query)
    ]
    results = await asyncio.gather(*tasks)
    return {
        "scholar": results[0],
        "web": results[1],
        "smart": results[2],
        "tavily": results[3]
    }

2. Confidence Scoring

python

def score_confidence(results):
    """Calculate deterministic confidence score."""
    score = 0
    
    # Source quality (40%)
    if results["scholar"]:
        score += 40 * len(results["scholar"]) / 10
    
    # Agreement analysis (35%)
    claims = extract_claims(results)
    agreement = analyze_agreement(claims)
    score += 35 * agreement
    
    # Recency (15%)
    recency = calculate_recency(results)
    score += 15 * recency
    
    # Relevance (10%)
    relevance = calculate_relevance(results, query)
    score += 10 * relevance
    
    return min(100, score)

3. Synthesis

python

async def synthesize(query, results, score):
    """Generate final answer with citations."""
    explanation = await explain_results(results)
    return {
        "answer": explanation["summary"],
        "confidence": score,
        "sources": explanation["citations"],
        "claims": explanation["claims"]
    }

For a complete implementation, see AIsa Verity.

Pricing

API	Cost
Web search	~$0.001
Scholar search	~$0.002
Smart search	~$0.002
Tavily search	~$0.002
Explain	~$0.003

Every response includes usage.cost and usage.credits_remaining.

Get Started

Sign up at aisa.one
Get your API key
Add credits (pay-as-you-go)
Set environment variable: export AISA_API_KEY="your-key"

Full API Reference

See API Reference for complete endpoint documentation.

Resources

AIsa Verity - Reference implementation of confidence-scored search agent
AIsa Documentation - Complete API documentation

Multi-source retrieval with confidence scoring - web, academic, and Tavily in one unified API

Install

OpenClaw Search 🔍

🔥 What Can You Do?

Research Assistant

Market Research

Competitive Analysis

News Aggregation

Deep Dive Research

Quick Start

🏗️ Architecture: Multi-Stage Orchestration

Phase 1: Discovery (Parallel Retrieval)

Phase 2: Reasoning (Meta-Analysis)

Core Capabilities

Web Search

Academic/Scholar Search

Smart Search (Web + Academic Combined)

Tavily Integration (Advanced)

Explain Search Results (Meta-Analysis)

📊 Confidence Scoring Engine

Scoring Rubric

Score Interpretation

Python Client

API Endpoints Reference

Search Parameters

🚀 Building a Verity-Style Agent

1. Parallel Discovery

2. Confidence Scoring

3. Synthesis

Pricing

Get Started

Full API Reference

Resources

Related skills