Telegram Import

Incrementally import Telegram messages from SQLite to LanceDB with Qwen3-Embedding-4B vectors, supporting checkpoint resume and filtering empty messages.

Security Scan

The skill's code and instructions are consistent with its stated purpose (incrementally importing Telegram rows from a local SQLite DB into a local LanceDB using a local embedding service); there are no external credentials or remote endpoints, but there are practical issues to be aware of (pickle-based checkpointing, hardcoded Windows paths, and no install guidance for Python dependencies).

Purpose & Capability

The name/description (Telegram -> LanceDB with Qwen3 embeddings) matches the code and SKILL.md. The script reads a local SQLite DB, generates embeddings from a local LM Studio endpoint, and writes to a local LanceDB dataset. No unrelated credentials, remote APIs, or unrelated binaries are requested.

Instruction Scope

Instructions and the script operate on local files and the local LM Studio endpoint (127.0.0.1). They reference explicit Windows paths (D:\...) and the checkpoint file. The script reads the SQLite DB and writes to LanceDB and a checkpoint; it does not contact external domains beyond localhost. Note: the SKILL.md and script assume specific absolute paths which may not exist on a user's machine and will cause failures if paths are different.

Install Mechanism

No install spec is provided (instruction-only plus an included script). The script imports third-party Python packages (requests, numpy, pyarrow, lance). The absence of an install spec is not incoherent, but users must manually ensure these dependencies are installed; missing guidance could lead to runtime errors.

Credentials

No environment variables, credentials, or external tokens are required. The only network call is to http://127.0.0.1:1234 (local LM Studio), which is proportional to producing embeddings.

Persistence & Privilege

The skill does not request always:true, does not modify other skills, and only writes its own checkpoint and the LanceDB dataset at the configured paths. It does perform persistent writes to the filesystem (checkpoint and dataset) as expected for an importer.

Assessment

This skill is coherent with its stated purpose, but review these before running: 1) Paths: the script hardcodes Windows paths (e.g., D:\chat\telegram_messages.db and D:\edata.lance). Update them if your files live elsewhere or run in an environment that matches these paths. 2) Dependencies: install the required Python packages (requests, numpy, pyarrow, lance) in a controlled virtual environment. 3) Local embedding service: the embedding endpoint is http://127.0.0.1:1234 — ensure LM Studio (or another compatible service) is running locally and you trust it. 4) Checkpoint safety: the script uses pickle.load to read the checkpoint file; unpickling untrusted files can execute arbitrary code. Only reuse checkpoints you created/trust; if you expect untrusted inputs, replace pickle with a safe format (JSON) or validate the checkpoint file before loading. 5) Backups & permissions: the script will read your Telegram DB and write into the LanceDB directory; back up data and run with appropriate filesystem permissions. If you want higher assurance, inspect and test the script in an isolated VM or container before running on sensitive data.

telegram-import Skill

将 Telegram 消息从 SQLite 数据库增量导入 LanceDB 向量库。

数据源

• DB: D:\chat\telegram_messages.db
• 表: telegram_messages
• 字段: id, group_name, group_id, message_id, date, sender_id, sender_name, message, matched_keywords, is_reply, media_type, has_6_digit_number, created_time

目标存储

• LanceDB: D:\edata.lance
• Checkpoint: D:\edata.lance\temp\telegram.ckpt（pickle 格式，断点续传）

Schema

字段	类型	说明
id	string	telegram:{group_id}:{message_id}
text	string	消息文本
vector	float[2560]	Qwen3-Embedding-4B 向量
category	string	固定 telegram
scope	string	固定 shared
importance	float	固定 0.5
timestamp	int	消息时间戳（Unix）
metadata	string	JSON：group_name, group_id, message_id, sender_id, sender_name, matched_keywords, is_reply, media_type, has_6_digit_number

Embedding

• LM Studio: http://127.0.0.1:1234/v1/embeddings
• Model: text-embedding-qwen3-embedding-4b
• 维度: 2560

执行命令

python.exe "C:\Users\admin\.openclaw\workspace\skills\telegram-import\scripts\chunk_db.py" telegram

增量逻辑

1. 启动时加载 telegram.ckpt 恢复 seen_keys 和 last_idx
2. 从 last_idx + 1 继续扫描 DB
3. 跳过 seen_keys 中已存在的 key
4. 过滤空消息（message 为 NULL 或空字符串）
5. 每 500 行保存一次 checkpoint