Agent Sam Lite

A local, offline Windows desktop agent that controls your PC from a single Telegram chat. A small on-device model decides what to do; a fact-only screen parser tells it what is on screen; a clicker does the physical action. Three jobs, kept strictly separate.

v0.1.0 — first release. Three Python files. One perception engine. No cloud. No GPU required.

---

Table of Contents

• What it is
• The core idea
• Full flow (one diagram)
• Highlights
• Architecture
• How routing works
• How the GUI loop stays safe
• Built-in skills
• Telegram commands
• Quickstart
• Configuration
• Performance notes
• Known limits
• Project layout
• Credits

---

What it is

Agent Sam Lite is a Telegram-controlled assistant for a Windows machine. You send plain language; the agent either answers, runs a pre-built tool, manipulates the live screen, or — when no tool exists — writes, tests, and saves a new tool for next time. Everything runs locally on a 4B-parameter model served by KoboldCpp, with screen understanding provided by the SAM ScreenParser (RapidOCR + Windows UI Automation + a curated UI dataset).

It is built for a small model on purpose. Instead of asking a 4B model to plan long sequences, the system asks it only to dispatch: pick a route, pick a skill, or pick a single on-screen element by id. The hard work (OCR, accessibility reads, pixel math, clicking) is done by deterministic code the model never has to reason about.

---

The core idea

The model is the dispatcher. The skills are the workers. The parser is the eyes. The clicker is the hands. None of them are allowed to do each other's job.

This separation is what makes a small local model reliable enough to drive a real machine:

• The router makes one cheap call and never sees the screen.
• The parser reports only verified facts (text, OS control type, dataset-matched type). It never guesses a role and never invents a coordinate.
• The model receives element ids and text — never pixels — so it physically cannot hallucinate a coordinate.
• The clicker only ever clicks an id it was handed, resolving it to pixels from the same capture. A bad id clicks nothing.

---

Full flow (one diagram)

graph TD
    U["User message via Telegram"] --> R{"Semantic Router<br/>one LLM call, no screen"}

    R -->|chat| CH["Direct text reply"]
    R -->|ask| AS["Ask a clarifying question"]
    R -->|memory| ME["Save fact to user_info.json"]
    R -->|command| CM["Run bot command<br/>todo / timer / send / winsearch / screen / info"]
    R -->|skill| SK["Skill executor (skills.py)<br/>ppt / word / files / calculator / youtube / browser"]
    R -->|create_skill| CR["Skill Creator<br/>generate, test, save a new tool"]
    R -->|gui| P
    SK -->|loop skills, e.g. paint| P

    subgraph LOOP["GUI Task Loop — parser first, the model never sees pixels"]
        P["SAM ScreenParser<br/>RapidOCR + UI Automation + dataset"]
        P --> ST["semantic table: id + text + type"]
        P --> CT["coordinate table: id + bounds + center"]
        ST --> AI["AI Planner<br/>emits a target_id only"]
        AI --> CK["Clicker / Executor<br/>resolves id to pixels, then acts"]
        CT --> CK
        CK --> VF{"Screen changed as expected?"}
        VF -->|yes, more to do| P
        VF -->|no or stuck| AI
        VF -->|goal met| DN["Task complete"]
    end

Loading

Reading the diagram top to bottom: every message hits the router exactly once. Chat, questions, memory, and commands resolve without touching the screen. Skills run as plain functions. Only the gui route (and loop-style skills like Paint) enters the iterative loop, where the parser produces two id-aligned tables, the model names an id, the clicker acts, and the screen is re-read until the goal is met or the agent gives up.

---

Highlights

• Parser-first vision. The screen is read as structured facts, not as an image. The model gets ids and text; coordinates stay in the executor. This is the safety property the whole design rests on.
• One skill per topic. ppt handles create, edit, convert, and read in one place, so the router can never pick the wrong sub-tool.
• Self-growing toolbox. When no skill matches and the task is data/API/math, the Skill Creator writes a Python tool, runs it in a sandbox, and — only if it passes — saves it permanently under data/custom_skills/.
• NLP skill browser. /readymade finds the right tool with fuzzy matching (keyword + sequence similarity + token overlap) and confirms before acting.
• Iterative GUI loop with verification. Every GUI step re-captures the screen and checks that something changed; ids are snapshot-bound and never reused across captures.
• Telegram-native file browser. /send walks drives and folders with inline buttons and pagination, using short numeric ids to stay under Telegram's callback-data limit.
• Honest unknowns. Elements the parser cannot identify carry no type and no confidence score; a top-level guide tells the model to treat them as static text unless context says otherwise.
• Fully offline. No API keys, no telemetry, no GPU. Runs on a standard laptop.

---

Architecture

The entire agent is three Python modules plus a small data directory. Files are intentionally few and large.

File	Role
main.py	Router, LLM client, prompts, GUI loop, clicker, Telegram bot, commands, file browser, scheduler, context and task managers.
parser_bridge.py	Bridge to SAM ScreenParser. Runs one capture and returns the semantic table (for the model) and the coordinate table (for the clicker).
skills.py	Embedded skill registry, NLP matcher, executor, Skill Creator, and all built-in skills in one file.
element_dataset.json	Curated map of UI text to element type, used for exact and fuzzy matching.

Runtime data lives under data/: todo.json, user_info.json, logs/agent.log, and auto-created skills under data/custom_skills/.

---

How routing works

The router is the only component that reads free text without the screen. It returns exactly one JSON object with a route field.

Route	When it fires	Screen used?
chat	Greetings, jokes, questions, casual talk	No
skill	A built-in tool can do it without the screen	No
gui	Anything that must click, type, open, select, draw on screen	Yes (loop)
create_skill	No tool exists; task is API / math / data (not GUI)	No
ask	The request is ambiguous	No
memory	The user states a personal fact	No
command	A bot action (todo, timer, send, winsearch, screen)	No

The router decides whether the screen is needed, never how to see it. When the route is gui, the parser is always the eyes.

---

How the GUI loop stays safe

Each iteration of the loop does four things in order: parse, plan, act, verify.

1. Parse. SAM ScreenParser sweeps the screen once and emits two lists that share the same ids. The semantic list (id, text, optional type, optional control type) goes to the model. The coordinate list (id, bounds, center) goes only to the clicker.
2. Plan. The model reads the semantic list plus window title, app kind, screen state, and cursor position, then emits one action such as {"target_id": 12}, {"target_id": 30, "input": "hello"}, or {"target_id": 1, "action": "double_click"} for a desktop icon.
3. Act. The clicker looks up the id in the coordinate list and clicks its center. If the id is absent, the clicker refuses — a wrong reference can never hit the wrong place.
4. Verify. The screen is captured again. If nothing changed, the click missed and the agent retries once; if the goal is met, it stops.

Two rules carry most of the weight: a list item outside any window (app kind desktop) needs a double click; a list or sidebar entry inside an app needs a single click. And ids are valid for one capture only — the model is told to match elements by window + type + text when it needs to remember something across frames.

---

Built-in skills

Skill id	Operations	Notes
ppt	create, edit, to_pdf, read	Uses python-pptx; PDF via PowerPoint COM
word	create, edit, to_pdf, read	Uses python-docx; PDF via Word COM
files	organize, rename, search, list, info	Organize sorts by extension into folders
calculator	calculate, convert	Sandboxed eval; unit and temperature conversion
youtube	search, play, play_url	Opens results; play selects the first hit
browser	open_url, search, navigate	11 search engines
paint	draw, edit	Opens MS Paint, then hands off to the GUI loop
gui_shell	execute	Runs a generated Python script in a subprocess (fallback)

Custom skills created at runtime are appended to this registry automatically and persist on disk.

---

Telegram commands

Commands are grouped by purpose. /cancel is registered last so it never shadows another handler.

Group	Command	Effect
General	/start, /help	Show this command reference
General	/status	Busy or idle
Tasks	/timer <min> <task>	Run the task after N minutes
Tasks	/winsearch <query>	Native Windows search with category buttons
Tasks	/screen	Send a full-quality PNG (as a document, no recompression)
Files	/send	Interactive drive / folder / file browser (up to 5 files)
Skills	/readymade	Browse all skills by category
Skills	/readymade <query>	Fuzzy-search skills with confidence scores
Todo	/todo	List items
Todo	/todo add <text>	Add an item
Todo	/todo done <id>	Mark done
Todo	/todo delete <id>	Remove an item
Memory	/info	Show stored user facts
Memory	/info add <key> <value>	Store a fact
Memory	/info delete <key>	Remove a fact
Control	/cancel	Stop the running task immediately

Any other text is routed by the model (chat, skill, gui, create_skill, ask, or memory).

---

Quickstart

1. Create and activate a Python 3.12 virtual environment, then install dependencies.

   py -3.12 -m venv .venv
   .\.venv\Scripts\Activate.ps1
   pip install -r requirements.txt

2. Verify the perception stack and the Telegram library load together.

   python -c "from rapidocr_onnxruntime import RapidOCR; import uiautomation; from rapidfuzz import fuzz; import telegram; print('ALL OK')"

3. Edit start_kobold.bat and set the three paths to your KoboldCpp binary and model files.

4. Fill the two required values in .env (everything else has defaults).

   TELEGRAM_BOT_TOKEN=your_token_here
   TELEGRAM_AUTHORIZED_USER_ID=your_numeric_id_here

5. Run the bot.

   .\run_telegram.bat
   # or, directly:
   python main.py --telegram

The first run auto-starts KoboldCpp if no LLM is reachable on http://localhost:5001, waits for the API, then enters Telegram polling. A blank cursor in the terminal after Starting Agent Sam Lite... is normal — the bot is alive and waiting for messages.

Useful flags: --check (test the LLM endpoint), --setup / --download (start or fetch KoboldCpp and the model).

---

Configuration

Only TELEGRAM_BOT_TOKEN and TELEGRAM_AUTHORIZED_USER_ID are required. The full set, with defaults, is in .env.example.

Full .env reference

# --- Telegram ---
TELEGRAM_BOT_TOKEN=
TELEGRAM_AUTHORIZED_USER_ID=

# --- LLM / KoboldCpp ---
LLM_BASE_URL=http://localhost:5001/v1
LLM_MODEL=Qwen3.5-4B
LLM_THINKING=false
LLM_REASONING_EFFORT=none
LLM_TEMPERATURE=0.1
LLM_TIMEOUT=240
LLM_MAX_TOKENS=2048

# --- KoboldCpp paths (leave empty; start_kobold.bat handles launch) ---
KOBOLDCPP_EXE_PATH=
KOBOLDCPP_MODEL_PATH=
KOBOLDCPP_MMPROJ_PATH=

# --- KoboldCpp options ---
KOBOLDCPP_THREADS=8
KOBOLDCPP_JINJA=true
KOBOLDCPP_JINJA_THINK=false
AUTO_DOWNLOAD_LLM=false

# --- Execution ---
ENABLE_TELEGRAM=true
MAX_LOOP_ITERATIONS=20
MAX_LOOP_TASK_ITERATIONS=10
MAX_SKILL_EXECUTION_TIMEOUT=120
MAX_GUI_CODE_TIMEOUT=60

# --- Vision (SAM ScreenParser is the default; no extra config needed) ---
SCREEN_COMPRESS_SCALE=1.0
SCREEN_JPEG_QUALITY=95

Prompt design notes

The prompts are written for a 4B model: short sentences, one instruction per line, explicit JSON shapes, and DO / DO NOT lists instead of long paragraphs. Placeholders use $VAR (replaced with .replace), never {var}, so JSON braces in the templates cannot collide with Python formatting. Thinking and extended reasoning are disabled on purpose — a 4B model over-thinks and burns its token budget; deterministic dispatch is faster and more reliable here.

---

Performance notes

• The parser does not run on chat or skills. It runs only inside the gui route and loop-style skills. A hello message triggers one router call and nothing else; you can confirm this by the absence of any Parser: N elements log line.
• The dominant latency on chat is the LLM backend, not the GUI. The bundled koboldcpp.exe is the CPU build. If you have a GPU, switch to koboldcpp_cuda.exe (NVIDIA) or koboldcpp_vulkan.exe (any GPU) in start_kobold.bat and add --gpulayers 99 to push the model to VRAM; router replies then drop from seconds to well under one.
• GUI tasks are intentionally re-parsed every step. SAM ScreenParser runs on CPU and takes roughly 7–13 seconds per 1080p frame by design; a 5-step GUI task is therefore 5 parses plus 5 short model calls. This is the cost of fresh, honest ids each capture. Lower MAX_LOOP_ITERATIONS to cap long tasks.
• Slash commands skip the router entirely and are effectively instant.

---

Known limits

• RapidOCR sees text-like regions only; a control with neither text nor an accessibility name is invisible to the parser.
• Electron and Chromium apps (VS Code, Brave, Discord) report PaneControl for almost everything, so type assignment there depends on dataset coverage; unmatched text is left without a type by design.
• ListItemControl is used for both sidebar rows and desktop icons; the model disambiguates using the app kind, applying the double-click rule on the desktop.
• The UI dataset is English-only and not exhaustive.
• Perception ids are snapshot-bound; cross-frame memory must go through matching by window + type + text.
• Tested on a single monitor; multi-monitor needs all_screens=True plus per-monitor DPI handling.

---

Project layout

agent-sam-lite/
│
├── .env.example                # Config template (committed to git)
├── .env                        # Your personal config (git-ignored)
├── .gitignore                  # Excludes llm/, data/, .env, __pycache__
├── README.md                   # Project overview + single flow diagram
├── SETUP.md                    # User-facing installation & LLM download guide
├── requirements.txt            # Python dependencies
├── element_dataset.json        # UI element type dataset for SAM parser
│
├── install.bat                 # Installs pip packages, creates dirs, offers LLM download
├── download_llm.bat            # Fetches koboldcpp.exe + model + mmproj into llm/
├── start_kobold.bat            # Launches KoboldCpp with wait loop (no --launch)
├── run_telegram.bat            # Starts the Telegram bot
│
├── main.py                     # Router, LLM client, GUI loop, clicker, Telegram bot,
│                               # commands, file browser, scheduler, context/task managers
├── parser_bridge.py            # Bridge to SAM ScreenParser (OCR + UIA + dataset matcher)
├── skills.py                   # Skill registry, NLP matcher, executor, creator, all skills
│
├── llm/                        # Engine + model (git-ignored, populated by download_llm.bat)
│   ├── koboldcpp.exe
│   ├── Qwen3.5-4B-UD-Q4_K_XL.gguf
│   └── mmproj-BF16.gguf        # Optional; unused by default
│
└── data/                       # Runtime data (git-ignored, auto-created at startup)
    ├── todo.json
    ├── user_info.json
    ├── logs/
    │   └── agent.log
    └── custom_skills/          # Auto-created skills saved here
        ├── *.py
        └── *.json

---

Credits

• SAM ScreenParser — fact-only perception engine (RapidOCR + Windows UI Automation + RapidFuzz dataset matching).
• RapidOCR / PaddleOCR — PaddlePaddle / Breezedeus (Apache 2.0).
• ONNX Runtime — Microsoft (MIT).
• OpenCV — OpenCV Team (Apache 2.0).
• RapidFuzz — maxbachmann (MIT).
• UIAutomation — yinkaisheng (MIT), wrapping the Windows SDK accessibility APIs.
• KoboldCpp — LostRuins (MIT).
• Qwen3.5 — Alibaba / unsloth GGUF quants.
• python-telegram-bot — the python-telegram-bot contributors (LGPL).

Released under the MIT License.