Scrask

When the user sends a screenshot via Telegram, parse it using Gemini (fast, default) with automatic Claude fallback when confidence is low. Saves results to...

Scrask Bot

Overview

This skill activates when the user sends a screenshot via Telegram. It uses vision AI to extract actionable information from the image, then:

• High confidence (≥ 0.75): Saves immediately and replies with a brief confirmation.
• Low confidence (< 0.75): Shows a structured preview in Telegram and asks for confirmation before saving.

Provider behaviour (auto mode, default):

Step	What happens
1	Gemini 2.0 Flash parses the screenshot (fast, cheap)
2	If any item confidence < 0.60, Claude Opus reruns the parse
3	Whichever provider scores higher average confidence wins
4	Output includes provider, fallback_triggered, and confidence delta

Set vision_provider to "claude" or "gemini" to lock a specific provider.

Output destinations (AI-decided by content type):

Detected type	Destination
Event (has date+time, venue, or invite link)	Google Calendar
Reminder (deadline, due date, personal action)	Google Tasks (with due date)
Task (no date, pure action item)	Google Tasks (no due date)

---

Trigger Conditions

Activate when:

1. The user sends a message in Telegram that contains an image attachment
2. The image appears to be a screenshot — not a photo of a person, place, or physical object
3. No other skill has already claimed the image

Do not activate for:

• Photos of people, places, food, scenery
• Screenshots of code, errors, or UI bugs (leave for other skills)
• Images the user explicitly asks to edit, describe, or analyze for another purpose

---

Step-by-Step Instructions

Step 1: Acknowledge Immediately

Reply in Telegram right away so the user knows the skill is working:

"📸 Got it — analyzing your screenshot..."

Do not make the user wait silently.

---

Step 2: Run the Parser

python3 ~/.openclaw/skills/scrask-bot/scripts/scrask_bot.py \
  --image-path "<path-to-temp-image>" \
  --provider "$CONFIG_VISION_PROVIDER" \
  --timezone "$CONFIG_TIMEZONE" \
  --google-credentials "$GOOGLE_CREDENTIALS"

The script auto-resolves the API key from ANTHROPIC_API_KEY or GEMINI_API_KEY
depending on the provider — no need to pass it explicitly.

The script returns a JSON object with:

• success — whether parsing worked
• no_actionable_content — true if nothing found
• results[] — one entry per detected item, each with confidence, type, destination, needs_confirmation, action_taken
• telegram_reply — the pre-formatted message to send back to the user

---

Step 3: Handle the Output

If no_actionable_content is true: Reply: "🤷 I couldn't find any event, reminder, or task info in that screenshot. Could you describe what you'd like to add?"

If success is true: Send the telegram_reply value directly back to the user in Telegram. The script has already:

• Saved high-confidence items silently
• Formatted confirmation prompts for low-confidence items

Do not rephrase or reformat the telegram_reply — send it as-is.

---

Step 4: Handle Confirmation Responses

If the script returned items with needs_confirmation: true, wait for the user's reply.

"yes" or "save" or "add": Re-run the script for that specific item with confirmed=true, or use the calendar_create / tasks_create tools directly with the extracted fields.

"edit": Ask what to change, update the relevant field, then save.

"skip" or "no": Reply: "Got it, skipped ✓"

---

Step 5: Confirm Saves

For items saved silently (high confidence), the telegram_reply from the script already contains the confirmation message. Examples of what the user will see:

• 📅 Added to Calendar: **Team Standup** — 2026-03-01 at 09:00
• 🔔 Added to Tasks: **Pay electricity bill** (due 2026-02-28)
• ✅ Added to Tasks: **Review PR for Arjun**

---

Edge Cases

Scenario	Behavior
Screenshot is in Hindi, Tamil, or another language	Extract and translate silently; save title in English
Recurring event ("every Monday")	Set RRULE on the calendar event; mention it in the reply
Date has already passed	Flag in the reply: "⚠️ This date has already passed (Feb 10). Save anyway?"
Multiple items in one screenshot	Process each independently; confirm per item if needed
Screenshot of someone's calendar	Detect already_in_calendar_hint; reply: "Looks like this event is already in your calendar 🗓️"
Google API auth failure	Reply with the specific error and suggest re-checking GOOGLE_CREDENTIALS
Zoom/Meet link found	Add to Calendar as both location and description

---

Configuration

{
  "skills": {
    "entries": {
      "scrask-bot": {
        "enabled": true,
        "env": {
          "GEMINI_API_KEY": "AIza-your-gemini-key",
          "ANTHROPIC_API_KEY": "sk-ant-your-key-here",
          "GOOGLE_CREDENTIALS": "/home/user/.openclaw/google-creds.json"
        },
        "config": {
          "vision_provider": "auto",
          "fallback_threshold": 0.60,
          "timezone": "Asia/Kolkata",
          "confidence_threshold": 0.75,
          "reminder_minutes_before": 30
        }
      }
    }
  }
}

---

Permissions Required

• image:read — to access the screenshot from Telegram
• network:outbound — to call Anthropic API and Google APIs
• telegram:reply — to send confirmation messages back to the user