Lowyat Forum Research

Research any topic by searching Lowyat forum (forum.lowyat.net), scraping relevant threads into Excel, and analyzing the results. Use this skill whenever the...

Security Scan

The skill's code and runtime instructions match its stated purpose (searching and scraping Lowyat forum threads into Excel and analyzing them); there are no hidden endpoints, extra credentials, or unexplained installs, but you should still consider terms-of-service and privacy implications before scraping.

Purpose & Capability

Name/description match the provided files and instructions. The included datascraping.py and SKILL.md implement exactly: search (via WebSearch guidance), scrape forum.lowyat.net pages, and save results to .xlsx. Required packages are proportional to the task (requests, BeautifulSoup, openpyxl, tqdm).

Instruction Scope

SKILL.md stays within scope: it instructs how to find threads, run the included scraper, and analyze the resulting Excel files. It does not request unrelated file reads, credentials, or remote endpoints beyond forum.lowyat.net. It explicitly recommends polite scraping behavior (delays, limited parallelism).

Install Mechanism

There is no automated install spec; SKILL.md asks the user to pip-install common Python libraries. No downloads from untrusted URLs or archive extraction are present. The code is included in the package (datascraping.py).

Credentials

The skill requests no environment variables or credentials. All external network activity is limited to the target forum pages. The pyproject specifies Python >=3.12 which is reasonable but not enforced by SKILL.md; this mismatch is minor and not a security risk.

Persistence & Privilege

Skill is not marked always:true and does not modify other skills or agent-wide configs. It runs as a user-invoked tool and does not request elevated system presence.

Assessment

This skill appears to do what it says (scrape Lowyat threads into Excel), but before installing or running it consider: 1) Terms of Service & robots.txt — scraping may violate Lowyat's terms or local laws; verify permission. 2) Privacy — scraped data includes public usernames and timestamps; handle, store, and share it responsibly and anonymize if needed. 3) Load on the site — follow the documented delays and avoid bulk parallel scraping to prevent denial-of-service. 4) Environment setup — ensure you run the pip installs in a virtual environment and that your Python version meets pyproject (>=3.12) if you encounter issues. 5) Source verification — SKILL.md links to a GitHub repo; if provenance matters, inspect that repository for updates or additional code. If you need the agent to run this autonomously, be aware autonomous invocation plus data collection increases risk, so restrict usage accordingly.

Lowyat Forum Research Tool

End-to-end research pipeline: Search → Scrape → Analyze

Workflow

Step 1: Understand the user's research topic

• Ask clarifying questions if needed (e.g. what specifically they want to learn)
• Break the topic into 3-5 search keyword variations

Step 2: Search for relevant threads

• Use WebSearch with site:forum.lowyat.net <keywords> to find threads
• Use allowed_domains: ["forum.lowyat.net"] to filter results
• Run multiple searches in parallel with different keyword angles
• Present the most relevant threads to the user with titles and URLs
• Let the user pick which threads to scrape, or recommend the best ones

Step 3: Scrape the selected threads

• The scraper script (datascraping.py) should be in the project root
• Install Python dependencies:

pip install requests beautifulsoup4 html5lib openpyxl tqdm

Or if you have uv installed:

uv sync

• Run the scraper for each thread:

python datascraping.py <TOPIC_URL>

• IMPORTANT: Do NOT include /all or /+N suffixes in the URL — just use the base topic URL (e.g. https://forum.lowyat.net/topic/5411252)
• To scrape multiple threads, run them sequentially (one at a time) to be respectful to the server. Only run up to 3 in parallel if the user explicitly asks for speed, using & and wait
• Output: <topic_id>.xlsx files with columns: Name, Date, Comment

Step 4: Analyze the scraped data

• Read the scraped .xlsx files to understand the forum discussions
• Synthesize findings across all threads into a structured summary
• Organize insights by the user's research questions
• Include: consensus opinions, brand recommendations, price ranges, warnings, and specific user experiences
• Cite which thread/user said what when relevant

Scraper Details

• Forum uses 20 posts per page, paginated via /+N URL suffix
• Scraper auto-detects total pages and crawls all of them
• Random 0.5–2s delay between page requests
• Saves incrementally after each page — safe to interrupt
• If .xlsx already exists, it resumes by appending

Tips for good searches

• Use brand names: site:forum.lowyat.net Toto toilet
• Use Malay keywords too: site:forum.lowyat.net kipas exhaust tandas
• Add "recommendation" or "review": site:forum.lowyat.net water heater recommendation
• Search by location: site:forum.lowyat.net bathroom shop KL Selangor
• Try year filters for recency: site:forum.lowyat.net smart toilet 2024 2025

Example usage

User: "I want to research mechanical keyboards on Lowyat"

1. Search with variations: mechanical keyboard recommendation, cherry mx switch, keychron Malaysia, custom keyboard
2. Present top threads to user
3. Scrape selected threads in parallel
4. Read the xlsx files and provide analysis: popular brands, price ranges, where to buy, common complaints

Links

• GitHub: github.com/superoo7/lowyat-forum-research
• ClawHub: clawhub.ai/superoo7/lowyat-forum-research

Disclaimer

Scraped data contains publicly available usernames, dates, and comments from forum.lowyat.net. This tool is intended for personal research purposes only. Users are responsible for how they store, share, and use the scraped data in compliance with applicable privacy laws and Lowyat forum's terms of service.