Memory Deduplication

Automatically identify, merge, and clean duplicate or outdated entries in MEMORY.md to maintain concise, organized, and up-to-date information.

Security Scan

The skill's code and instructions align with its stated purpose (deduplicating and merging MEMORY.md); it reads and rewrites MEMORY.md, makes backups, and contains no network or unexpected credential access.

Purpose & Capability

The name/description (Memory Deduplication) match the contained script and SKILL.md: the tool parses MEMORY.md, finds similar sections, merges/removes them, and writes results. No unrelated binaries, credentials, or external services are requested.

Instruction Scope

SKILL.md instructs running the provided Node script and optionally scheduling a cron job. The runtime instructions and script operate only on MEMORY.md in the workspace and create backups; they do not read network endpoints or other system credentials. Note: the script overwrites MEMORY.md (with backups by default) — use --dry-run to preview changes and review diffs before accepting.

Install Mechanism

No install spec or external downloads; this is an instruction-only/embedded script. The package files are local and there are no archives or remote install URLs, so install-side risk is low.

Credentials

The registry metadata declares no required env vars, which is consistent. The code does read an optional OPENCLAW_WORKSPACE environment variable to determine the workspace path; this env var is not declared in requires.env (it's optional). This is not malicious but worth noting because setting OPENCLAW_WORKSPACE changes which MEMORY.md is edited.

Persistence & Privilege

The skill is not always-enabled and does not request elevated platform privileges. It writes to files within the workspace (MEMORY.md and a memory/ backup folder) which is expected for its purpose. It does not modify other skills or global agent configuration.

Assessment

This skill appears coherent with its purpose and contains no network or secret-exfiltration behavior. Before installing/running: (1) run node dedup.mjs --dry-run to preview changes; (2) verify the workspace target (set OPENCLAW_WORKSPACE explicitly if needed) so it edits the intended MEMORY.md; (3) review the backup folder after a real run and confirm diffs; (4) consider running first in an isolated agent/session or on a copy of your repository. If you want stricter guarantees, inspect the script locally (dedup.mjs is small and readable) or run it inside a sandboxed environment.

Memory Deduplication — 记忆去重与合并

保持 MEMORY.md 清洁，避免信息冗余

问题

随着时间推移，MEMORY.md 会出现：

• 重复信息 — 同一事件多次记录
• 过时信息 — 已完成的任务仍标记为进行中
• 冗余描述 — 同一项目多处描述
• 碎片化 — 相关信息散落各处

解决方案

1. 自动去重

识别并合并相似内容：

原始:
- AgentAwaken 网站开发中
- AgentAwaken 项目进行中
- AgentAwaken 待部署

合并后:
- AgentAwaken 网站: 开发中，待部署

2. 过时信息清理

原始:
- [P0] NeuroBoost v5.0 发布待重试

更新后:
- [P0] NeuroBoost v5.0 ✅ 已发布 (2026-02-26)

3. 信息聚合

原始:
### AgentAwaken
- 代码: /root/.openclaw/workspace/agentawaken
### AgentAwaken 域名
- agentawaken.xyz 待绑定
### AgentAwaken 部署
- 需要 Vercel

合并后:
### [P0] AgentAwaken 网站
- 代码: /root/.openclaw/workspace/agentawaken
- 域名: agentawaken.xyz (待绑定)
- 部署: Vercel (待配置)

实现

相似度计算

function similarity(text1, text2) {
  // Jaccard 相似度
  const words1 = new Set(text1.toLowerCase().split(/\s+/));
  const words2 = new Set(text2.toLowerCase().split(/\s+/));
  const intersection = new Set([...words1].filter(x => words2.has(x)));
  const union = new Set([...words1, ...words2]);
  return intersection.size / union.size;
}

去重规则

1. 相似度 >0.8 — 完全重复，删除
2. 相似度 0.5-0.8 — 部分重复，合并
3. 相似度 <0.5 — 不同内容，保留

合并策略

• 保留最新时间戳
• 合并所有唯一信息
• 保留最高优先级标记

使用

# 运行去重
node skills/memory-dedup/dedup.mjs

# 预览（不修改文件）
node skills/memory-dedup/dedup.mjs --dry-run

# 备份后去重
node skills/memory-dedup/dedup.mjs --backup

输出示例

=== Memory Deduplication Report ===

📊 统计:
- 原始条目: 87
- 重复条目: 12
- 合并条目: 5
- 删除条目: 7
- 最终条目: 68

🔍 发现的重复:
1. "AgentAwaken 网站开发" (3 次)
   → 合并为 1 条
2. "NeuroBoost v5.0 发布" (2 次)
   → 保留最新版本

✅ MEMORY.md 已优化
💾 备份保存到: memory/MEMORY-backup-2026-03-01.md

定期执行

# 每周日凌晨 2 点自动去重
openclaw cron add --name "memory-dedup-weekly" \
  --cron "0 2 * * 0" --tz "Asia/Shanghai" \
  --session isolated --agent main \
  --message "运行记忆去重，清理 MEMORY.md 冗余信息"

安全措施

1. 自动备份 — 去重前备份原文件
2. 人工审核 — 生成 diff 供审核
3. 可回滚 — 保留最近 10 次备份
4. 白名单 — 某些关键信息不去重

效果

• 文件大小减少 30-50%
• 检索速度提升 2-3 倍
• 信息密度提升 40%
• 维护成本降低 60%