Nonblocking Agent Execution Enhanced

Enhanced Non-Blocking Agent Execution with AI-powered improvements. Prevents "agent stopped responding / stuck / no output" failures in sandboxed agent runtimes (Arena Agent Mode, OpenClaw, Codex) where a single long tool call blocks the turn and stdin is closed so interactive prompts hang forever. NEW in v2.0.0: - Full jobctl.sh implementation with all commands - Multi-model compatibility (OpenAI, Anthropic, Mistral, Groq, etc.) - Token usage optimization and monitoring - Hallucination reduction via verification - Self-improving through feedback loops - Comprehensive debugging and logging - Idempotent operations and graceful error handling - Watchdog timers for timeout protection - Callback URL support for async notifications - Durable state persistence across turns Provides the detach → bounded-poll → durable-state pattern plus a ready-to-use jobctl.sh runner with enhanced features for production use.

🚀 Non-Blocking Agent Execution v2.0.0

Enhanced Edition with AI-Powered Improvements

Field-authored in Arena Agent Mode (2026-07) after a real 173-skill install + llama.cpp build froze the session. Synthesizes lessons from ClawHub skills @aowind/long-running-harness, @wonko6x9/durable-task-runner, @skywalker-lili/polling-best-practices, @liyooyin/task-progress-stream, @nyxun123/agent-heartbeat, @hollis9087/long-task-handoff.

Version 2.0.0 adds comprehensive implementation, debugging, and AI-powered features based on deep research and best practices.

---

🎯 What's New in v2.0.0

✨ New Features

Feature	Description	Benefit
Full Implementation	Complete jobctl.sh script with all commands	Actually works, not just documentation
Multi-Model Support	Works with OpenAI, Anthropic, Mistral, Groq, etc.	Compatible with any AI model
Token Optimization	Monitors and optimizes token usage	Reduces costs and improves efficiency
Hallucination Reduction	Built-in verification mechanisms	More accurate and reliable outputs
Self-Improving	Feedback loops for continuous improvement	Gets better over time
Comprehensive Debugging	Full debug mode with detailed information	Easier troubleshooting
Watchdog Timers	Automatic timeout protection	Prevents runaway processes
Callback Support	Async notifications via webhooks	Better integration
Durable State	Persists across agent turns	Reliable execution
Idempotent Operations	Safe to retry commands	Prevents duplicate work

📊 Performance Improvements

Metric	v1.0.6	v2.0.0	Improvement
Implementation	Documentation only	Full implementation	✅ Complete
Token Usage	Not monitored	Optimized and monitored	⬇️ Reduced
Hallucinations	Not addressed	Verified outputs	⬇️ Reduced
Debugging	None	Comprehensive	✅ Added
Compatibility	Limited	Multi-model	✅ Expanded
Self-Improvement	None	Feedback-driven	✅ Added

---

📋 The Three Real Causes of "Agent Not Responding"

Cause	Symptom	Fix (v2.0.0 Implementation)
Blocking tool call — one bash call runs 20 min	UI shows nothing, user aborts	Detach with setsid nohup ... & + watchdog timer
Interactive prompt with closed stdin	Hangs forever, never times out	--yes / --no-input shims + always wrap in timeout
Aborted turn kills the child process	Work silently lost, half-done state	Detach with setsid, persist state to disk + atomic writes

---

🎯 Core Rules (Enhanced)

Original Rules (Preserved)

1. No tool call over ~60 s. Long work is launched, not awaited.
2. setsid nohup … < /dev/null & — survives the turn being cancelled; plain & does not.
3. Bounded wait only. A poll helper that always returns within N seconds. Never wait.
4. Every external command gets timeout N. No exceptions for network CLIs.
5. Non-interactive flags always (--yes, --no-input, -y, DEBIAN_FRONTEND=noninteractive).
6. State on disk, not in context — pid, exit_code, log, started_at per job, so any context reset or new turn can resume by reading files.
7. Report progress every poll ([k/N] item) so the user sees liveness instead of silence.
8. Idempotent resume: recompute what is missing and redo only that, never restart from zero.

New Rules (v2.0.0)

9. Token monitoring - Track token usage and warn when approaching limits
10. Output verification - Verify outputs for hallucinations and errors
11. Feedback collection - Collect user feedback for self-improvement
12. Multi-model compatibility - Normalize model identifiers and configurations
13. Comprehensive logging - Structured logs with different severity levels
14. Watchdog protection - Automatic timeout enforcement
15. Graceful shutdown - Clean up resources on termination

---

🔧 The Runner (Now Fully Implemented)

bash

./scripts/jobctl.sh start <name> '<command>' [callback_url] [model] [max_tokens]
./scripts/jobctl.sh stop <name>
./scripts/jobctl.sh status <name>
./scripts/jobctl.sh poll <name> [timeout]
./scripts/jobctl.sh log <name> [n]
./scripts/jobctl.sh list [filter]
./scripts/jobctl.sh cleanup <name>
./scripts/jobctl.sh verify <name>
./scripts/jobctl.sh debug <name>

State Structure (JSON)

json

{
  "job_id": "unique-id",
  "command": "the command to execute",
  "status": "queued|running|paused|stopped|completed|failed|verified",
  "pid": 12345,
  "start_time": "2026-09-06T14:00:00Z",
  "end_time": "2026-09-06T14:05:00Z",
  "exit_code": 0,
  "callback_url": "https://...",
  "model": "gpt-4o-mini",
  "max_tokens": 2048,
  "tokens_used": 1500,
  "token_rate": 25.5,
  "output_hash": "sha256:...",
  "error_message": "",
  "retry_count": 0,
  "verified": true,
  "verification_score": 0.95,
  "verification_issues": "",
  "feedback": "",
  "self_improvement": {
    "suggestions": "..."
  }
}

State lives in ~/.nonblocking/state/<job_id>.json.

---

🎨 Agent Loop (Enhanced)

text

start job → status (2s) → do other useful work → wait 25s → report "[k/N]" → repeat → verify exit=0

Enhanced Loop with v2.0.0 Features

text

start job → optimize command → status (2s) → token check → 
  do other work → wait 25s → verify output → 
  collect feedback → self-improve → report "[k/N]" → repeat

---

🚀 New Features in Detail

1. Token Usage Optimization

Automatic Command Optimization

• Adds --yes, -y, --no-input flags automatically
• Wraps commands in timeout if not present
• Redirects stdin from /dev/null
• Configurable via environment variables

Token Monitoring

• Tracks tokens used per job
• Warns at configurable thresholds (default: 4000)
• Errors at configurable thresholds (default: 8000)
• Reports token generation rate

Usage

bash

# Start with token monitoring
./scripts/jobctl.sh start my-job 'complex-command' https://callback.url gpt-4o-mini 2048

# Check token usage
./scripts/jobctl.sh status my-job | jq '.tokens_used'

2. Hallucination Reduction

Output Verification

• Checks for common hallucination patterns
• Assigns verification scores
• Marks outputs as verified/failed
• Configurable verification thresholds

Verification Patterns

• Detects "I don't have access" statements
• Detects "as of my last update" disclaimers
• Detects "may not be accurate" statements
• Extensible with custom patterns

Usage

bash

# Start a job
./scripts/jobctl.sh start verify-job 'generate-report' https://callback.url

# Verify the output
./scripts/jobctl.sh verify verify-job

# Check verification score
./scripts/jobctl.sh status verify-job | jq '.verification_score'

3. Self-Improving Mechanism

Feedback Collection

• Records user feedback with ratings
• Stores feedback in structured format
• Links feedback to specific jobs

Feedback Analysis

• Identifies improvement opportunities
• Patterns: slow execution, errors, token issues
• Stores suggestions in job state
• Aggregates feedback across jobs

Usage

bash

# Record feedback (manual)
echo '{"feedback":"Output was slow", "rating":3}' > ~/.nonblocking/feedback/my-job.feedback

# Or use the feedback collection API
# (See integration section below)

4. Multi-Model Compatibility

Model Normalization

• Converts various model identifiers to standard format
• Handles OpenAI, Anthropic, Mistral, Groq, and more
• Configurable default model

Model-Specific Configuration

• Different timeouts and token limits per model
• Automatic configuration based on model type
• Override defaults per job

Supported Models

• OpenAI: gpt-4, gpt-4o-mini, gpt-3.5-turbo
• Anthropic: claude-3-sonnet, claude-3-haiku
• Mistral: mistral-7b-instruct
• Groq: groq/gpt-oss-120b
• Llama: llama-3-70b-instruct
• Custom: Any other model identifier

Usage

bash

# Use different models
./scripts/jobctl.sh start job1 'command' https://callback.url gpt-4o-mini 2048
./scripts/jobctl.sh start job2 'command' https://callback.url claude-3-sonnet 4096

5. Comprehensive Debugging

Debug Mode

• Enable with LOG_LEVEL=DEBUG environment variable
• Shows detailed execution information
• Logs all internal operations

Debug Command

bash

./scripts/jobctl.sh debug my-job

Debug Output Includes

• Full job state
• Process information (PID, CPU, memory, runtime)
• Log file contents (last 20 lines)
• Output file contents
• Feedback if available

Usage

bash

# Enable debug logging
LOG_LEVEL=DEBUG ./scripts/jobctl.sh start my-job 'command'

# Get debug info
./scripts/jobctl.sh debug my-job

6. Watchdog Timers

Automatic Timeout Protection

• Configurable maximum runtime (default: 24 hours)
• Watchdog process monitors each job
• Sends SIGTERM on timeout
• Updates job state with timeout information

Usage

bash

# Set custom timeout
MAX_RUNTIME=3600 ./scripts/jobctl.sh start my-job 'command'

# Or per job
./scripts/jobctl.sh start my-job 'timeout 3600 command'

7. Callback Support

Async Notifications

• Optional callback URL per job
• POSTs JSON result to callback URL
• Retries on failure
• Includes job status and output

Callback Format

json

{
  "job_id": "my-job",
  "status": "completed",
  "exit_code": 0,
  "output": "...",
  "tokens_used": 1500,
  "duration": "2m 30s"
}

Usage

bash

# With callback
./scripts/jobctl.sh start my-job 'command' https://my-api.com/callback

# Without callback
./scripts/jobctl.sh start my-job 'command'

---

🔍 Debugging Stage - Best Practices

1. Always Start with Debug Mode

bash

LOG_LEVEL=DEBUG ./scripts/jobctl.sh start test-job 'echo hello'

2. Check Status First

bash

./scripts/jobctl.sh status test-job

3. Review Logs

bash

./scripts/jobctl.sh log test-job 100

4. Get Full Debug Info

bash

./scripts/jobctl.sh debug test-job

5. Verify Output

bash

./scripts/jobctl.sh verify test-job

6. Common Issues and Fixes

Issue	Symptom	Debug Steps	Fix
Job not starting	Status remains "queued"	Check logs, verify command	Check command syntax, permissions
Job hanging	Status "running", no progress	Debug mode, check process	Add timeout, verify non-interactive flags
High token usage	Token warnings in logs	Check token monitoring	Optimize command, reduce output
Verification failed	verification_score < 0.7	Check verification_issues	Fix command, add validation
Callback failed	No callback received	Check callback URL, network	Verify URL, check firewall
Process killed	Exit code 137 or 143	Check watchdog logs	Increase MAX_RUNTIME

---

📁 File Structure

text

nonblocking-agent-execution-enhanced/
├── SKILL.md                    # This file - skill documentation
├── README.md                   # User-facing documentation
├── scripts/
│   └── jobctl.sh              # Main execution controller
├── tests/
│   ├── test_basic.sh          # Basic functionality tests
│   ├── test_token_optimization.sh # Token optimization tests
│   ├── test_verification.sh    # Verification tests
│   └── test_multi_model.sh     # Multi-model compatibility tests
├── docs/
│   ├── API.md                 # API documentation
│   ├── INTEGRATION.md         # Integration guide
│   └── BEST_PRACTICES.md       # Best practices guide
└── config/
    └── defaults.env           # Default configuration

---

🎯 Integration Guide

Basic Usage

bash

# Install the skill
npx --yes clawhub@latest install @orionshaowswmw/nonblocking-agent-execution

# Or clone this repository
cd /path/to/skills
git clone https://github.com/orionshaowswmw/nonblocking-agent-execution-enhanced.git

# Start a job
./scripts/jobctl.sh start my-build 'npm install && npm run build'

# Check status
./scripts/jobctl.sh status my-build

# Poll until complete
./scripts/jobctl.sh poll my-build 5

# Get output
cat ~/.nonblocking/state/my-build.output

# Clean up
./scripts/jobctl.sh cleanup my-build

Programmatic Usage

python

import subprocess
import json
import time

# Start a job
job_id = "my-python-job"
command = "python3 my_script.py"
result = subprocess.run(
    ["./scripts/jobctl.sh", "start", job_id, command],
    capture_output=True,
    text=True
)
job_info = json.loads(result.stdout)
print(f"Started job {job_id} with PID {job_info['pid']}")

# Poll for completion
for i in range(120):  # 2 minutes max
    time.sleep(2)
    result = subprocess.run(
        ["./scripts/jobctl.sh", "status", job_id],
        capture_output=True,
        text=True
    )
    status = json.loads(result.stdout)
    if status['status'] in ['completed', 'failed', 'stopped']:
        print(f"Job completed with status: {status['status']}")
        break
    print(f"Job still running... ({i+1}/120)")

# Get output
output = subprocess.run(
    ["cat", f"~/.nonblocking/state/{job_id}.output"],
    capture_output=True,
    text=True
).stdout
print(f"Output: {output}")

# Clean up
subprocess.run(["./scripts/jobctl.sh", "cleanup", job_id])

With Callback URL

bash

# Start job with callback
./scripts/jobctl.sh start webhook-job 'long-running-task' https://my-server.com/api/callback

# Your server receives:
# POST /api/callback
# Content-Type: application/json
#
# {
#   "job_id": "webhook-job",
#   "status": "completed",
#   "exit_code": 0,
#   "output": "...",
#   "tokens_used": 1500,
#   "duration": "2m 30s"
# }

---

🔒 Security & Privacy

Permissions

• Runs and supervises subprocesses
• Writes durable job state files
• May use background/daemon execution
• Reads environment variables for configuration

Security Measures

• Sandboxing: Run untrusted commands in containers or VMs
• Least Privilege: Run with minimal required permissions
• Input Validation: All inputs are validated before execution
• Output Sanitization: Logs and outputs are handled carefully
• Resource Limits: Watchdog timers prevent runaway processes
• Cleanup: Proper resource cleanup on job completion

Data Handling

• Job state is written to local disk only
• No data is sent to external services (except optional callbacks)
• Logs may contain command output - protect accordingly
• Feedback is stored locally for self-improvement

Network Boundary

• Data leaves the machine only for explicit callback URLs
• All other processing remains local
• Callback URLs must be HTTPS for security

---

✅ Verification Hash

Artifact SHA-256 (TREE-SHA256-v1): Will be generated at publish time

Run from the installed skill directory:

bash

python3 - <<'PY'
from pathlib import Path
import hashlib
root = Path('.')
excluded_parts = {'.git', '.clawhub', '__pycache__', '.pytest_cache'}
excluded_names = {'readme.md', 'skill-card.md', '_meta.json', '.published', '.ds_store'}
files = sorted(
    (p for p in root.rglob('*') if p.is_file()
     and not any(part in excluded_parts for part in p.relative_to(root).parts)
     and p.name.lower() not in excluded_names),
    key=lambda p: p.relative_to(root).as_posix(),
)
h = hashlib.sha256()
h.update(b'TREE-SHA256-v1\0')
for p in files:
    rel = p.relative_to(root).as_posix().encode('utf-8')
    data = p.read_bytes()
    h.update(rel); h.update(b'\0')
    h.update(str(len(data)).encode('ascii')); h.update(b'\0')
    h.update(data); h.update(b'\0')
print(h.hexdigest())
PY

---

📚 Complete Skill Reference

All functionality from v1.0.6 is preserved and enhanced in v2.0.0.

---

🎓 Best Practices

1. Always Use Non-Interactive Mode

bash

# Good
apt-get install -y package
pip install --yes package
npx --yes command

# Bad (will hang)
apt-get install package
pip install package
npx command

2. Always Set Timeouts

bash

# Good
timeout 300 long-running-command
timeout 60 curl http://example.com

# Bad (may run forever)
long-running-command
curl http://example.com

3. Redirect stdin

bash

# Good
command < /dev/null

# Bad (may hang waiting for input)
command

4. Use Detach Pattern

bash

# Good
setsid nohup command > output.log 2> error.log &

# Bad (dies when parent dies)
command > output.log 2> error.log &

5. Monitor Token Usage

bash

# Check token usage
./scripts/jobctl.sh status my-job | jq '.tokens_used'

# Set thresholds
TOKEN_WARNING_THRESHOLD=2000 TOKEN_ERROR_THRESHOLD=5000 ./scripts/jobctl.sh start my-job 'command'

6. Verify Outputs

bash

# Always verify
./scripts/jobctl.sh verify my-job

# Check verification score
./scripts/jobctl.sh status my-job | jq '.verification_score'

7. Collect Feedback

bash

# Record feedback
./scripts/jobctl.sh record-feedback my-job "Output was helpful" 5

---

🚀 Performance Tips

Token Optimization

1. Use smaller models for simple tasks (gpt-4o-mini vs gpt-4)
2. Set appropriate max_tokens - don't over-request
3. Enable command optimization (default: on)
4. Monitor usage and adjust thresholds

Speed Improvements

1. Parallelize independent jobs
2. Use faster models for time-sensitive tasks (groq/gpt-oss-120b)
3. Cache results for repeated operations
4. Optimize commands to run faster

Reliability Improvements

1. Always use watchdog timers
2. Verify all outputs
3. Collect feedback for continuous improvement
4. Use debug mode for troubleshooting

---

📖 Examples

Example 1: Long-Running Build

bash

# Start a build that takes 10 minutes
./scripts/jobctl.sh start build-app-001 'npm install && npm run build' \
  https://my-api.com/build-callback gpt-4o-mini 4096

# Check status periodically
./scripts/jobctl.sh status build-app-001

# Get logs if there are issues
./scripts/jobctl.sh log build-app-001 100

Example 2: Batch Processing

bash

# Process multiple files in parallel
for i in {1..10}; do
  ./scripts/jobctl.sh start process-file-$i "process_file file$i.txt" \
    https://my-api.com/callback
  sleep 1  # Rate limit
Done

# Poll all jobs
watch -n 5 './scripts/jobctl.sh list running'

Example 3: Model Comparison

bash

# Run same task with different models
for model in gpt-4o-mini claude-3-sonnet mistral-7b-instruct; do
  ./scripts/jobctl.sh start compare-$model "run_benchmark" \
    https://my-api.com/compare-callback $model 2048
done

# Compare results
./scripts/jobctl.sh list completed

Example 4: Debugging a Failed Job

bash

# Get full debug info
./scripts/jobctl.sh debug failed-job-123

# Check logs
./scripts/jobctl.sh log failed-job-123

# Try again with debug mode
LOG_LEVEL=DEBUG ./scripts/jobctl.sh start failed-job-123-retry 'fixed-command'

---

🎯 Compatibility

Supported Platforms

• Linux (all modern distributions)
• macOS (with bash 4+)
• WSL (Windows Subsystem for Linux)
• Docker containers
• Kubernetes pods

Dependencies

• bash 4+
• coreutils (timeout, kill, etc.)
• sqlite3 (optional, for advanced state management)
• curl (for callbacks)
• python3 (for JSON manipulation)
• jq (optional, for JSON parsing in examples)

Installation

bash

# Clone the repository
git clone https://github.com/orionshaowswmw/nonblocking-agent-execution-enhanced.git

# Or install via ClawHub (after publishing)
npx --yes clawhub@latest install @orionshaowswmw/nonblocking-agent-execution

# Make jobctl.sh executable
chmod +x scripts/jobctl.sh

# Create directories
mkdir -p ~/.nonblocking/{run,logs,state,cache,feedback}

---

📞 Support

• Issues: Report on GitHub
• Questions: Check documentation first
• Contributions: Pull requests welcome
• Feedback: Use the feedback mechanism

---

Maintained with ❤️ by the AI Agent Community

Documentation last updated: 2026-09-06