Compare commits
16 Commits
59c16ea393
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
238e57e4db | ||
|
|
50f03d8939 | ||
|
|
61d35dd851 | ||
| 45fd1d5874 | |||
| b90727fc11 | |||
| 356ae9367c | |||
|
|
cb16e8688f | ||
|
|
3c1b268074 | ||
|
|
b2eb4fbfa3 | ||
|
|
0b4c2bbb15 | ||
|
|
cfd1876359 | ||
|
|
1d1454e1b9 | ||
|
|
cb2c92452e | ||
|
|
156c6db2f8 | ||
|
|
f9a863e271 | ||
|
|
8d58436f05 |
8
.gitattributes
vendored
Normal file
8
.gitattributes
vendored
Normal file
@@ -0,0 +1,8 @@
|
||||
# raw/ 是 hash 釘選的 provenance 帳本(AGENTS.md §1.4 / §9 / §14):
|
||||
# git 一律不得正規化其換行,否則 checkout 後磁碟 bytes 會與 manifest 的
|
||||
# SHA-256 不符,讓 --verify 對帳失準(本專案 core.autocrlf=true)。
|
||||
# 轉換器已在寫入時以 write_bytes 固定為登記的那份 bytes;此處把同樣的
|
||||
# 保證延伸到 git 的 checkin/checkout。
|
||||
raw/originals/** -text
|
||||
raw/converted/** -text
|
||||
raw/manifest.json -text
|
||||
29
.gitea/workflows/claude.yml
Normal file
29
.gitea/workflows/claude.yml
Normal file
@@ -0,0 +1,29 @@
|
||||
name: Claude Assistant
|
||||
on:
|
||||
issue_comment:
|
||||
types: [created]
|
||||
pull_request_review_comment:
|
||||
types: [created]
|
||||
issues:
|
||||
types: [opened, assigned, labeled]
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
jobs:
|
||||
claude-response:
|
||||
runs-on: yellow-zeabur-runner
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- uses: markwylde/claude-code-gitea-action@v1.0.20
|
||||
with:
|
||||
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||
gitea_token: ${{ secrets.USER_TOKEN }}
|
||||
claude_git_name: Claude
|
||||
claude_git_email: claude@anthropic.com
|
||||
path_to_claude_code_executable: /usr/local/bin/claude
|
||||
claude_args: |
|
||||
--append-system-prompt "請一律使用繁體中文回覆,包含 issue/PR 留言、commit message 說明文字、todo list 項目。程式碼註解與變數命名維持英文慣例即可。"
|
||||
env:
|
||||
GITEA_SERVER_URL: https://yellow-git.zeabur.app
|
||||
31
.gitea/workflows/lint.yaml
Normal file
31
.gitea/workflows/lint.yaml
Normal file
@@ -0,0 +1,31 @@
|
||||
# kb-lint 定時健檢(Phase 4 草稿,部署到 Gitea 時啟用)
|
||||
#
|
||||
# 部署前提(ASSUMPTION:部署環境未定,以下依常見 self-hosted 配置假設):
|
||||
# - self-hosted runner 與 Ollama 同機或內網可達(必要時以 OLLAMA_BASE_URL 覆寫,
|
||||
# 這是唯一允許的設定覆寫來源,AGENTS.md §1.2)
|
||||
# - runner 具 Python 3.11+;文件內容全程不離開內網(AGENTS.md §1.1)
|
||||
# - 報告以 artifact 上傳供人工下載審核;lint 絕不修改 repo 內容
|
||||
name: kb-lint
|
||||
on:
|
||||
schedule:
|
||||
- cron: "0 21 * * 0" # UTC 週日 21:00 = 台北時間週一 05:00
|
||||
workflow_dispatch: {}
|
||||
|
||||
jobs:
|
||||
lint:
|
||||
runs-on: self-hosted
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Set up venv
|
||||
run: |
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt
|
||||
- name: Run lint
|
||||
# 結束碼 1 = 有待人工核准項目 → workflow 標紅作為通知信號
|
||||
run: .venv/bin/python tools/lint.py --output reports
|
||||
- name: Upload report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: lint-report
|
||||
path: reports/
|
||||
0
.github/agents/.gitkeep
vendored
0
.github/agents/.gitkeep
vendored
33
.github/agents/kb-ingest.agent.md
vendored
Normal file
33
.github/agents/kb-ingest.agent.md
vendored
Normal file
@@ -0,0 +1,33 @@
|
||||
---
|
||||
name: kb-ingest
|
||||
description: 攝入文件到 QA 知識庫:本地轉換 → 本地 Ollama 攝入 → 開 PR。agent 只編排,不處理文件內容。
|
||||
tools: ['runCommands']
|
||||
---
|
||||
<!-- ASSUMPTION: frontmatter 欄位依 VS Code Copilot Custom Agent 公開慣例
|
||||
(name/description/tools)撰寫;若組織版格式有異,只調整 frontmatter,內文不變。 -->
|
||||
|
||||
你是 QA 知識庫的攝入編排者。規範正本:`AGENTS.md`(尤其 §1 硬性約束、§6 Ingest 工作流)。
|
||||
|
||||
## 成本紀律(不可違反)
|
||||
|
||||
你是**編排者不是處理者**:文件內容的 LLM 處理一律由本地 Ollama 腳本完成。
|
||||
**禁止**把來源文件、轉換後 Markdown 或 wiki 頁全文讀進你的 context——
|
||||
只讀取腳本 stdout/stderr 的精簡結果(狀態、統計、路徑、PR 連結)。
|
||||
|
||||
## 工作流
|
||||
|
||||
輸入:一個或多個檔案路徑或網頁 URL。
|
||||
|
||||
1. 轉換並登記 manifest:
|
||||
`.venv\Scripts\python tools\convert\convert.py <路徑或URL...>`
|
||||
- 從 stdout 取得 `raw/converted/...` 路徑;`needs_ocr` 表示掃描件已登記待人工,不要嘗試 OCR。
|
||||
2. 攝入(本地 Ollama 處理、自動建分支與 commit):
|
||||
`.venv\Scripts\python tools\ingest.py <轉換後路徑...>`
|
||||
3. 回報使用者:成功/失敗份數、變更頁面清單(stdout 已含)、PR 建立網址。
|
||||
攝入失敗時引用 stderr 的錯誤摘要(一兩行即可)。
|
||||
|
||||
## 禁止事項
|
||||
|
||||
- 不直接編輯 `wiki/`、`index.md`、`log.md`、`raw/`(一律經由腳本)。
|
||||
- 不 commit 或 push 到 main(HITL 閘門,AGENTS.md §1.5)。
|
||||
- 不呼叫任何雲端 API 處理文件內容(AGENTS.md §1.1)。
|
||||
31
.github/agents/kb-lint.agent.md
vendored
Normal file
31
.github/agents/kb-lint.agent.md
vendored
Normal file
@@ -0,0 +1,31 @@
|
||||
---
|
||||
name: kb-lint
|
||||
description: 執行 QA 知識庫健檢並摘要報告重點,列出待人工核准項目。agent 只編排,不處理頁面內容。
|
||||
tools: ['runCommands']
|
||||
---
|
||||
<!-- ASSUMPTION: frontmatter 欄位依 VS Code Copilot Custom Agent 公開慣例撰寫。 -->
|
||||
|
||||
你是 QA 知識庫的健檢編排者。規範正本:`AGENTS.md`(尤其 §7 Lint 工作流)。
|
||||
|
||||
## 成本紀律(不可違反)
|
||||
|
||||
只讀取 lint 腳本的 stdout 與報告檔的**開頭摘要區**(前 30 行左右),
|
||||
不把全部 wiki 頁面內容拉進 context。
|
||||
|
||||
## 工作流
|
||||
|
||||
1. 執行健檢(LLM 比對由本地 Ollama 完成):
|
||||
`.venv\Scripts\python tools\lint.py --output reports`
|
||||
- 結束碼 0 = 無發現;1 = 有待人工核准項目。
|
||||
- stdout 會印出報告路徑。
|
||||
2. 讀取報告開頭的統計摘要,回報使用者:
|
||||
- 各類發現數量(schema / stale / coverage / orphan / duplicate / contradiction)。
|
||||
- 需要人工核准的重點項目(最多列 10 項,附頁面路徑)。
|
||||
- 完整報告路徑。
|
||||
|
||||
## 禁止事項
|
||||
|
||||
- 絕不刪除或修改任何 wiki 頁、raw 檔(lint 本身也不會,AGENTS.md §7)。
|
||||
- 不自行「修復」發現的問題——一律留給人工核准後處理。
|
||||
- 若同類問題重複出現,建議使用者修改 AGENTS.md schema 或 lint 規則
|
||||
(change the ruler, not the output,AGENTS.md §12)。
|
||||
25
.github/agents/kb-query.agent.md
vendored
Normal file
25
.github/agents/kb-query.agent.md
vendored
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: kb-query
|
||||
description: 查詢 QA 知識庫:BM25 檢索候選頁 → 只讀命中頁 → 綜合回答並附 source_ref。
|
||||
tools: ['runCommands', 'search']
|
||||
---
|
||||
<!-- ASSUMPTION: frontmatter 欄位依 VS Code Copilot Custom Agent 公開慣例撰寫。 -->
|
||||
|
||||
你是 QA 知識庫的查詢助手。規範正本:`AGENTS.md`(尤其 §8 Query 工作流)。
|
||||
|
||||
## 工作流
|
||||
|
||||
1. 以使用者問題的關鍵詞檢索(繁中或英文皆可):
|
||||
`.venv\Scripts\python tools\search.py "<關鍵詞>" -k 10`
|
||||
- 回傳 JSON:候選頁的 path / title / description / tags / score。
|
||||
2. 只開啟**命中的少數頁面**(最多 10 頁,通常前 3–5 頁已足夠)讀取內文。
|
||||
3. 綜合回答使用者問題:
|
||||
- **必附 source_ref**(各頁 frontmatter 的 `sources` 欄位,格式 `路徑#hash前8碼`)。
|
||||
- 若各頁說法矛盾,兩種說法並陳並指出出處。
|
||||
- 候選頁都不相關時,直說知識庫沒有涵蓋,不要腦補。
|
||||
|
||||
## 禁止事項
|
||||
|
||||
- 禁止全庫掃描:不遍歷 `wiki/` 全部頁面、不讀 `raw/` 原始文件全文(AGENTS.md §8)。
|
||||
- 不修改任何檔案——查詢是唯讀操作。
|
||||
- 回答只根據 wiki 頁內容;wiki 沒有的知識明說沒有。
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
@@ -2,3 +2,4 @@
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.pytest_cache/
|
||||
reports/
|
||||
|
||||
13
.markdownlint.jsonc
Normal file
13
.markdownlint.jsonc
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
// 本專案文件以繁體中文為主,表格一律按「終端顯示寬度」對齊(CJK 字元佔 2 欄),
|
||||
// 這是人類在編輯器裡讀得順的排法。MD060 以「字元位置」比對管線符號,對 CJK
|
||||
// 無感——兩者不可能同時滿足。選擇保留顯示寬度對齊,關閉此規則。
|
||||
// 其餘規則維持預設(含 MD040:程式碼區塊必須標語言)。
|
||||
"MD060": false,
|
||||
|
||||
// MD013:本設定檔一存在,就會取代 VS Code 擴充內建的預設集(該預設集關閉
|
||||
// MD013),使行長檢查被動啟用——這會讓既有檔案(尤其 CJK 表格與逐字保存的
|
||||
// bootstrap prompt)冒出大量新警告。維持專案原本的行為:關閉。
|
||||
// 若日後想強制 80 字元散文慣例,改為 { "tables": false } 並處理既有檔案。
|
||||
"MD013": false
|
||||
}
|
||||
62
AGENTS.md
62
AGENTS.md
@@ -26,7 +26,10 @@
|
||||
EmbeddingGemma(Google)或 snowflake-arctic-embed(Snowflake)。
|
||||
4. **raw 層不可變**:`raw/` 內的文件只讀不改。原始檔與轉換後 Markdown 皆登記
|
||||
SHA-256 至 `raw/manifest.json`(結構見 §9),轉換檔條目必須回指原始檔 hash。
|
||||
wiki 頁面引用來源必須帶 `source_ref`(格式見 §3.3)。
|
||||
wiki 頁面引用來源必須帶 `source_ref`(格式見 §3.3)。登記的 hash 必須等於檔案在
|
||||
磁碟上的實際 bytes(UTF-8,換行不轉換——轉換器一律以 write_bytes 寫入所登記的
|
||||
那份 bytes,不受平台 CRLF 影響;git 亦以 `.gitattributes` 對 `raw/**` 關閉換行
|
||||
正規化,避免 checkout 重新引入 CRLF);完整性以 §14 `--verify` 稽核。
|
||||
5. **HITL 閘門**:所有 wiki 層的變更以 git branch + PR 形式提交,經人工審核後才
|
||||
合併至 main。攝入腳本永遠不直接 commit 到 main。lint 絕不擅自刪檔,一律標記
|
||||
待人工核准。
|
||||
@@ -39,7 +42,7 @@
|
||||
|
||||
## 2. 目錄結構
|
||||
|
||||
```
|
||||
```text
|
||||
pp-qa-knowledge/
|
||||
├── AGENTS.md # 本檔:schema 正本
|
||||
├── CLAUDE.md # 指標檔 → AGENTS.md
|
||||
@@ -93,7 +96,7 @@ status: draft # draft(攝入產出)| reviewed(人工審
|
||||
|
||||
### 3.3 source_ref 格式
|
||||
|
||||
```
|
||||
```text
|
||||
<repo 相對路徑>#<sha256 前 8 碼>
|
||||
```
|
||||
|
||||
@@ -193,13 +196,13 @@ wiki 頁檔名用英文 kebab-case slug(如 `wiki/entities/payment-gateway.md`
|
||||
|
||||
**index.md** — 目錄式導航,依頁面類型分節,每頁一行:
|
||||
|
||||
```
|
||||
```markdown
|
||||
- [標題](wiki/entities/payment-gateway.md) — 一句摘要 | #payment #gateway
|
||||
```
|
||||
|
||||
**log.md** — append-only 操作日誌,新條目追加於檔案末尾,不修改既有條目:
|
||||
|
||||
```
|
||||
```markdown
|
||||
## [YYYY-MM-DD] <op> | <title>
|
||||
```
|
||||
|
||||
@@ -272,6 +275,13 @@ output**)。單筆修正只治標;規則修正才治本。
|
||||
(global lock、O(n²) 掃描、naive heuristic),註解要寫明天花板與升級路徑。
|
||||
- 非平凡邏輯要留下**一個**可執行的檢查——邏輯壞掉就會失敗的最小東西
|
||||
(assert 式自檢或一個小測試檔;不用框架、不用 fixtures)。
|
||||
- **測資照現實建,不是照「能過」建**:來源檔名用中文+空格(真實 QA 文件就長這樣)、
|
||||
語料要有跨頁共用的核心詞。便利值會變成盲點的形狀——ASCII 檔名讓 `git status`
|
||||
的路徑轉義 bug 溜過,互不重疊的語料讓 BM25 的 idf 退化 bug 溜過。
|
||||
新增檢查時先問:**這組測資和真實輸入差在哪?差異處就是沒被測到的地方。**
|
||||
- **斷言要驗「結果可用」,不是驗「字串存在」**:`assert "。單筆修正只治標;規則修正才治本。
|
||||
**有效的跡象**:diff 裡不必要的變更變少、因過度複雜而重寫的次數變少、
|
||||
釐清問題發生在動手**之前**而非犯錯之後、刻意的捷徑是可見的(`ponytail:`)
|
||||
而非沉默的。
|
||||
|
||||
## 14. raw 完整性與修復(對帳)
|
||||
|
||||
raw 層以 manifest 的 SHA-256 為信任根(§1.4、§9)。檔案被**手動刪除**或**就地改動**時,
|
||||
下列為正規稽核與修復流程。核心原則:manifest 是 provenance **真相帳本**、不是 cache——
|
||||
hash 的用途是讓刪除**可復原、可驗證**,故先**復原檔案**,而非改帳本去遷就殘缺的磁碟。
|
||||
|
||||
### 14.1 稽核
|
||||
|
||||
```text
|
||||
python tools/convert/convert.py --verify
|
||||
```
|
||||
|
||||
純唯讀(不改任何檔,含 manifest),re-hash 全部登記檔並掃描 `raw/`,回報三類意外、
|
||||
有不一致以退出碼 1 表示(可當 CI/排程閘門):
|
||||
|
||||
- **missing** — 帳本有登記,磁碟上不見了(手動刪除)。
|
||||
- **mismatch** — 檔案還在但 sha256 與登記值不符(raw 被就地改動,違反 §1.4)。
|
||||
- **unregistered** — `raw/originals`/`raw/converted` 下有檔卻不在帳本。
|
||||
`converted/assets/*`(圖片)由 markdown 連結追蹤而非帳本,故略過;`.gitkeep` 亦略過。
|
||||
|
||||
### 14.2 修復決策(可衍生 vs 信任根)
|
||||
|
||||
<!-- ASSUMPTION: 本決策樹為推論的正規流程,原始需求只定義 raw 不可變與 hash 登記;
|
||||
依「可衍生(converted)優先還原、信任根(original)不可再生」的性質分流。 -->
|
||||
|
||||
1. **先還原,不改帳本**(多數「誤刪」到此為止):`raw/` 進版控,`git restore <path>`
|
||||
取回精確 bytes(或從備份),再跑 §14.1 確認 hash == 登記值。帳本零改動。
|
||||
2. **converted 救不回** → 從 original 重轉。注意兩個陷阱:(a) `convert.py` 以 **original
|
||||
的 hash** 去重,original 條目還在會直接 skip、**不會**因 converted 不見而重生,需先
|
||||
自帳本移除該 converted 條目;(b) 函式庫版本變動可能使重轉 bytes 不同 → converted
|
||||
hash 變 → 連累所有指向它的 wiki `source_ref`。故**能還原就別重轉**;重轉屬實質變更,
|
||||
連同 source_ref 一起走 PR。
|
||||
3. **original 救不回** → 不可逆的 provenance 損失,**不得靜默刪條目**(會連鎖 orphan 掉
|
||||
對應 converted、再斷所有 source_ref);據實記錄該檔遺失並上報人工(HITL,§5)。
|
||||
|
||||
### 14.3 動 manifest 的紀律
|
||||
|
||||
- 任何帳本變更走 **branch + PR**(§5)——manifest 是 provenance 帳本,改它需人工審核。
|
||||
- `log.md` 追加一筆 `edit` 條目(§10),說明對了什麼、為什麼。
|
||||
- 刪任何 `converted` 條目前,先確認沒有 wiki `source_ref` 引用它(否則 §7 的 schema
|
||||
檢查會抓到斷鏈);有引用就改為還原檔案,或同 PR 一併更新引用。
|
||||
|
||||
227
README.md
Normal file
227
README.md
Normal file
@@ -0,0 +1,227 @@
|
||||
# README — QA 知識庫使用手冊
|
||||
|
||||
金融電子支付 QA 部門知識庫,採 **Karpathy LLM Wiki 模式**:知識在文件**攝入時**由本地
|
||||
LLM 編譯成 wiki 頁,查詢時只讀已編譯頁面(**非 RAG**,檢索用 BM25)。
|
||||
|
||||
> **本檔是操作手冊(怎麼裝、怎麼跑)。** 所有 schema、工作流細節、硬性約束與行為紀律的
|
||||
> **唯一正本**是 [AGENTS.md](AGENTS.md)——動手前請先讀它。本檔只做操作說明,不複製其內容。
|
||||
|
||||
---
|
||||
|
||||
## 1. 環境需求
|
||||
|
||||
| 項目 | 需求 |
|
||||
|------|------|
|
||||
| Python | 3.11+ |
|
||||
| 本地 Ollama | 執行於 `http://localhost:11434`(見 [config/models.yaml](config/models.yaml)) |
|
||||
| Ollama 模型 | 已 `ollama pull` 且與 `config/models.yaml` 的 tag 一致的**合規本地模型** |
|
||||
| Git remote(選用) | 攝入自動開 PR 用;無 remote 時分支保留於本地 |
|
||||
|
||||
**資料落地是硬性約束([AGENTS.md](AGENTS.md) §1.1)**:所有文件內容的 LLM 處理一律走本地
|
||||
Ollama,**禁止**任何雲端 API、雲端轉換服務,及任何 `:cloud` 後綴模型。供應鏈限制(§1.3):
|
||||
禁用中國關聯模型(bge / Qwen / GLM / MiniMax / Jina 等)。
|
||||
|
||||
---
|
||||
|
||||
## 2. 安裝
|
||||
|
||||
Windows PowerShell(本專案主要環境):
|
||||
|
||||
```powershell
|
||||
python -m venv .venv
|
||||
.venv\Scripts\pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Linux / Gitea runner:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements.txt
|
||||
```
|
||||
|
||||
接著確認 Ollama 已拉好 `config/models.yaml` 內指定的模型:
|
||||
|
||||
```powershell
|
||||
ollama list # 對照 tasks.ingest.model / tasks.lint.model / tasks.fallback.model
|
||||
```
|
||||
|
||||
**換模型只改 [config/models.yaml](config/models.yaml),不動任何程式碼**(§1.2)。設定缺欄位時程式直接
|
||||
報錯,不使用隱含預設值。唯一允許的覆寫來源是環境變數 `OLLAMA_BASE_URL`。
|
||||
|
||||
> 下文指令以 Windows 的 `.venv\Scripts\python` 為例;Linux 換成 `.venv/bin/python`。
|
||||
|
||||
---
|
||||
|
||||
## 3. 四大工作流
|
||||
|
||||
資料流:**原始檔 → `convert` → `raw/converted/` Markdown → `ingest` → `wiki/` 頁面 →
|
||||
`search` / MCP 查詢**;`lint` 定期健檢整庫。
|
||||
|
||||
### 3.1 轉換 `convert` — 原始檔轉 Markdown
|
||||
|
||||
把 docx / xlsx / pdf / pptx / html 或網頁 URL 轉成 Markdown,並登記 SHA-256 至
|
||||
`raw/manifest.json`(`raw/` 只讀不可變,§1.4)。
|
||||
|
||||
```powershell
|
||||
# 本機檔案(可多個)
|
||||
.venv\Scripts\python tools\convert\convert.py 報告.docx 測項.xlsx 規格.pdf
|
||||
|
||||
# 網頁快照(保存完整 HTML,因來源可能消失)
|
||||
.venv\Scripts\python tools\convert\convert.py https://example.com/guideline
|
||||
|
||||
# 只看計畫、不寫檔
|
||||
.venv\Scripts\python tools\convert\convert.py 報告.docx --dry-run
|
||||
```
|
||||
|
||||
- 支援:`.docx .xlsx .pdf .pptx .html/.htm` 與 `http(s)://` URL。
|
||||
- hash 相同的檔會自動 `skip`(去重)。
|
||||
- 掃描件(PDF 無文字層)標記 `needs_ocr` 並跳過,**不擅自 OCR**,留待人工決定。
|
||||
- 單檔失敗不中斷批次,最後彙總並以結束碼 1 回報。
|
||||
|
||||
### 3.2 攝入 `ingest` — 編譯成 wiki 頁
|
||||
|
||||
讀 `raw/converted/` 的 Markdown,呼叫本地 Ollama 產出 / 更新 wiki 頁,重建
|
||||
`index.md`、追加 `log.md`,並**開新 git 分支 + PR**(HITL 閘門,§1.5——永不直接 commit 到 main)。
|
||||
|
||||
```powershell
|
||||
# 攝入指定來源
|
||||
.venv\Scripts\python tools\ingest.py raw\converted\xxx.md
|
||||
|
||||
# 攝入所有「尚無 summary 頁引用」的來源
|
||||
.venv\Scripts\python tools\ingest.py --all-pending
|
||||
|
||||
# 只列計畫(分支名、模型、分段數),不呼叫 LLM、不寫檔
|
||||
.venv\Scripts\python tools\ingest.py --all-pending --dry-run
|
||||
|
||||
# 已攝入過(hash 相同)也強制重跑
|
||||
.venv\Scripts\python tools\ingest.py raw\converted\xxx.md --force
|
||||
```
|
||||
|
||||
**前置條件**:須在 `main` 分支、且工作區除 `raw/` 外無未提交變更,否則會報錯中止。
|
||||
|
||||
> **`convert` 的產出請保持未提交,不要另開分支 commit。** 除 `raw/` 外無未提交變更的條件
|
||||
> 刻意把 `raw/` 排除在外——正確流程是在 `main` 上 `convert`、確認 `raw/converted/*.md` 內容
|
||||
> 無誤後直接 `ingest`,由 ingest 把來源與編譯出的 wiki 頁**一起** commit 進同一個分支。這樣
|
||||
> PR 的 diff 同時含來源與產出,審核者才能對照 LLM 有無編譯錯誤。若先開分支 commit 了轉換檔,
|
||||
> 切回 `main` 時那些檔案會隨分支離開工作區;此時請先 `git merge` 該分支回 `main` 再攝入
|
||||
> (`raw/` 不受 §1.5 HITL 閘門管轄,該條只規範 wiki 層變更)。
|
||||
|
||||
執行後:變更落在 `ingest/<時間戳>-<slug>` 分支。有 remote 時自動 push 並印出 PR 建立網址;
|
||||
無 remote / 離線時分支保留本地,請手動 push 後開 PR。**經人工審核合併後才進 main。**
|
||||
|
||||
### 3.3 健檢 `lint` — 全庫品質檢查
|
||||
|
||||
檢查 schema 完整性、過期主張、覆蓋缺口、孤兒頁,並用 LLM 偵測重複 / 矛盾頁。
|
||||
**絕不修改或刪除任何檔案**,一律產出報告列為「待人工核准」(§1.5、§7)。
|
||||
|
||||
```powershell
|
||||
# 完整健檢,報告寫入 reports/
|
||||
.venv\Scripts\python tools\lint.py --output reports
|
||||
|
||||
# 跳過 LLM 重複/矛盾比對(較快,純 schema/結構檢查)
|
||||
.venv\Scripts\python tools\lint.py --no-llm
|
||||
```
|
||||
|
||||
- 報告路徑印在 stdout,例如 `reports/lint-YYYYMMDD-HHMMSS.md`。
|
||||
- **結束碼**:`0` = 無發現;`1` = 有待人工核准項目(供 CI 判斷 / 標紅通知)。
|
||||
- 排程:[.gitea/workflows/lint.yaml](.gitea/workflows/lint.yaml)(每週一台北時間 05:00,草稿,部署 Gitea 時啟用)。
|
||||
|
||||
> **`source_ref 不在 manifest` 的成因分類**:lint 只報「對不上」,不分辨成因。跑
|
||||
> [tools/diag_refs.py](tools/diag_refs.py) 把每個對不上的 source_ref 分成五類(hash 不符 /
|
||||
> 路徑字串岔開 / 檔名 hash 後綴岔開 / 檔在磁碟未登記 / 完全無對應),各附修法——
|
||||
> 「補帳本」(§14.3)與「修規則」(§12)的處置相反,先分類再動手。純唯讀不改任何檔,
|
||||
> 結束碼同 lint(`0` 全部解得開 / `1` 有對不上),可當搬機或改 manifest 後的閘門。
|
||||
>
|
||||
> ```powershell
|
||||
> .venv\Scripts\python tools\diag_refs.py
|
||||
> ```
|
||||
|
||||
### 3.4 查詢 `search` — BM25 檢索
|
||||
|
||||
```powershell
|
||||
.venv\Scripts\python tools\search.py "AML 測試準則" -k 10
|
||||
```
|
||||
|
||||
輸出 JSON 陣列(`path / title / description / tags / score`),供人工或 agent 定位候選頁。
|
||||
`-k` 上限 10(查詢工作流 §8:先讀 `index.md` 定位、最多 10 頁、只讀命中頁、回答必附
|
||||
`source_ref`、禁止全庫掃描)。
|
||||
|
||||
---
|
||||
|
||||
## 4. MCP server — 供內部 AI agent / 團隊查詢
|
||||
|
||||
[mcp/server.py](mcp/server.py) 是 FastMCP 薄殼,只暴露兩個唯讀 tool:`search_wiki`(BM25 檢索)與
|
||||
`read_page`(讀單頁全文,依 `models.yaml` 的 token 上限截斷)。只讀已編譯 wiki 頁,不觸碰
|
||||
`raw/` 層、不做全庫掃描。
|
||||
|
||||
```powershell
|
||||
# stdio 啟動
|
||||
.venv\Scripts\python mcp\server.py
|
||||
```
|
||||
|
||||
測試時可用環境變數 `PP_QA_ROOT` 指定專案根目錄。`read_page` 有路徑跳脫防護,只接受
|
||||
`wiki/` 下的 `.md`。
|
||||
|
||||
---
|
||||
|
||||
## 5. Copilot Custom Agents(雙棧)
|
||||
|
||||
同一套 [AGENTS.md](AGENTS.md) 規範同時服務 Claude Code 與 GitHub Copilot(§11 雙棧規則)。
|
||||
[.github/agents/](.github/agents/) 提供三個 Copilot 編排 agent——它們**只編排、不處理文件內容**
|
||||
(成本紀律 §1.6:LLM 處理一律 shell out 給本地腳本,agent 只讀精簡結果):
|
||||
|
||||
| Agent | 職責 |
|
||||
|-------|------|
|
||||
| [kb-ingest](.github/agents/kb-ingest.agent.md) | 轉換 → 攝入 → 開 PR |
|
||||
| [kb-lint](.github/agents/kb-lint.agent.md) | 執行健檢並摘要報告重點 |
|
||||
| [kb-query](.github/agents/kb-query.agent.md) | 檢索 → 只讀命中頁 → 附 source_ref 回答 |
|
||||
|
||||
---
|
||||
|
||||
## 6. 目錄速查
|
||||
|
||||
```text
|
||||
config/models.yaml # Ollama 端點與模型 tag(唯一設定來源)
|
||||
raw/originals/ # 原始檔,只讀
|
||||
raw/converted/ # 轉換後 Markdown,攝入的實際輸入,只讀
|
||||
raw/manifest.json # SHA-256 登記 + 原始檔↔轉換檔對應
|
||||
wiki/summaries/ # 每份來源一頁摘要
|
||||
wiki/entities/ # 實體頁(系統/模組/API/法規/專案)
|
||||
wiki/concepts/ # 概念頁(跨來源綜合)
|
||||
index.md # 目錄式導航(由 ingest 自動重建)
|
||||
log.md # append-only 操作日誌
|
||||
tools/ # convert/ ingest.py lint.py diag_refs.py search.py kb.py(共用模組)
|
||||
mcp/server.py # MCP 薄殼
|
||||
```
|
||||
|
||||
完整結構與各檔用途見 [AGENTS.md](AGENTS.md) §2。
|
||||
|
||||
---
|
||||
|
||||
## 7. 疑難排解
|
||||
|
||||
| 訊息 / 現象 | 原因與處置 |
|
||||
|-------------|-----------|
|
||||
| `config/models.yaml 缺欄位` | 設定不完整;補齊欄位,不要用預設值繞過(§1.2)。 |
|
||||
| `tasks.X.model 為 :cloud 模型` | 違反資料落地;改用本地模型 tag(§1.1)。 |
|
||||
| `須在 main 分支執行` | 攝入前先切回 `main`。 |
|
||||
| `工作區有 raw/ 以外的未提交變更` | 先 commit / stash 非 `raw/` 的變更再攝入。 |
|
||||
| `... 不在 manifest 的 converted 條目中` | 該來源尚未轉換;先跑 `convert.py`。 |
|
||||
| lint 報 `source_ref 不在 manifest` | 跑 `tools/diag_refs.py` 分類成因:帳本掉條目→補登走 PR(§14.3);路徑字串岔開→修規則(§12),別逐頁改。 |
|
||||
| `LLM 輸出驗證失敗(含 fallback)` | 本地模型連主模型 + fallback 都無法產出合規 JSON;檢查 Ollama 是否在線、模型是否拉好。 |
|
||||
| `push 失敗(無 remote 或離線)` | 正常;分支已保留本地,手動 push 後開 PR。 |
|
||||
| ingest 跑很久、看不到進度/想中斷 | 已逐來源 `[i/N]` +逐分段回報進度。Ctrl-C 安全:main 不受影響,中斷時會印回復指令(`git checkout main && git stash -u && git branch -D <branch>`),續跑 `--all-pending` 自動接續。 |
|
||||
|
||||
---
|
||||
|
||||
## 8. 開發者自檢
|
||||
|
||||
各工具附 assert 式自檢腳本(免框架、免 fixtures,§13.4):
|
||||
|
||||
```powershell
|
||||
.venv\Scripts\python tools\convert\selfcheck.py
|
||||
.venv\Scripts\python tools\selfcheck_ingest.py
|
||||
.venv\Scripts\python tools\selfcheck_lint.py
|
||||
.venv\Scripts\python tools\selfcheck_diag_refs.py
|
||||
.venv\Scripts\python tools\selfcheck_search.py
|
||||
```
|
||||
@@ -20,6 +20,13 @@ tasks: # 依任務指定模型,全部可調
|
||||
max_retries: 3
|
||||
fallback:
|
||||
model: "hermes3:8b" # 主模型格式驗證連續失敗時切換
|
||||
# vision(convert --vision 的掃描/圖片型 PDF fallback 才會用到):預設不啟用。
|
||||
# 要啟用時取消下方註解並填入「合規本地 vision 模型」——須自行確認非中國關聯、
|
||||
# 非 :cloud(§1.1/§1.3)。留成註解可避免 load_config 對未設定值報錯。
|
||||
# vision:
|
||||
# model: "CHANGE_ME" # 例:Google Gemma 視覺版 / IBM Granite Vision 等合規本地模型
|
||||
# temperature: 0.1
|
||||
# max_retries: 2
|
||||
|
||||
limits:
|
||||
mcp_response_max_tokens: 2000
|
||||
|
||||
56
mcp/server.py
Normal file
56
mcp/server.py
Normal file
@@ -0,0 +1,56 @@
|
||||
"""FastMCP 薄殼(AGENTS.md §8):search_wiki 與 read_page 兩個 tool。
|
||||
查詢時只讀已編譯的 wiki 頁——不觸碰 raw 層、不做全庫掃描。
|
||||
|
||||
啟動(stdio):.venv/Scripts/python mcp/server.py
|
||||
測試時可用環境變數 PP_QA_ROOT 指定專案根目錄。
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
ROOT = pathlib.Path(os.environ.get("PP_QA_ROOT",
|
||||
pathlib.Path(__file__).resolve().parents[1]))
|
||||
sys.path.insert(0, str(ROOT / "tools"))
|
||||
import kb
|
||||
import search as searchmod
|
||||
from fastmcp import FastMCP
|
||||
|
||||
mcp = FastMCP("pp-qa-knowledge")
|
||||
|
||||
|
||||
def _truncate(body, max_tokens):
|
||||
# ponytail: token 估算用「1 token ≈ 1.5 字元」的 naive heuristic
|
||||
# (繁中約 1 字/token、英文約 4 字元/token 的折衷);
|
||||
# 升級路徑:以實際模型 tokenizer 校正。
|
||||
limit = int(max_tokens * 1.5)
|
||||
return (body, False) if len(body) <= limit else (body[:limit], True)
|
||||
|
||||
|
||||
def search_wiki(query: str, top_k: int = 10) -> list:
|
||||
"""BM25 檢索 wiki 頁,回傳 index 條目(path/title/description/tags)+ 相關度 score。
|
||||
query 用繁體中文或英文關鍵詞。"""
|
||||
return searchmod.search(ROOT, query, max(1, min(int(top_k), 10))) # 上限 10(AGENTS.md §8)
|
||||
|
||||
|
||||
def read_page(path: str) -> dict:
|
||||
"""讀取單一 wiki 頁全文:frontmatter + 內文(依 models.yaml token 上限截斷)
|
||||
+ source_refs。path 例:wiki/entities/payment-gateway.md"""
|
||||
cfg = kb.load_config(ROOT)
|
||||
p = (ROOT / path).resolve()
|
||||
wiki = (ROOT / "wiki").resolve()
|
||||
# 信任邊界:擋路徑跳脫,只允許 wiki/ 下的 .md
|
||||
if not p.is_relative_to(wiki) or p.suffix != ".md":
|
||||
raise ValueError("path 須指向 wiki/ 下的 .md 頁面")
|
||||
if not p.is_file():
|
||||
raise FileNotFoundError(f"頁面不存在:{path}")
|
||||
meta, body = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||
body, truncated = _truncate(body, cfg["limits"]["mcp_response_max_tokens"])
|
||||
return {"frontmatter": meta, "body": body, "truncated": truncated,
|
||||
"source_refs": [str(s) for s in meta.get("sources") or []]}
|
||||
|
||||
|
||||
mcp.tool(search_wiki)
|
||||
mcp.tool(read_page)
|
||||
|
||||
if __name__ == "__main__":
|
||||
mcp.run()
|
||||
11
requirements.txt
Normal file
11
requirements.txt
Normal file
@@ -0,0 +1,11 @@
|
||||
# 全部為本地程式庫,無任何雲端轉換服務(AGENTS.md §1.1)
|
||||
pyyaml # config/models.yaml
|
||||
python-docx # Word → Markdown
|
||||
openpyxl # Excel → Markdown
|
||||
pymupdf # PDF → Markdown
|
||||
python-pptx # PowerPoint → Markdown
|
||||
trafilatura # 網頁快照 → Markdown 正文萃取
|
||||
rank-bm25 # tools/search.py BM25(第一版不用向量庫)
|
||||
fastmcp # mcp/server.py 薄殼
|
||||
markdown-it-py # 自檢用:以 CommonMark 解析驗證產出的圖片連結真的能 render
|
||||
# (原為 fastmcp 的傳遞依賴,明確宣告以免上游調整後自檢失效)
|
||||
325
tools/convert/convert.py
Normal file
325
tools/convert/convert.py
Normal file
@@ -0,0 +1,325 @@
|
||||
"""統一轉換入口:依副檔名/URL 分派轉換器,登記 SHA-256 至 raw/manifest.json。
|
||||
|
||||
流程(AGENTS.md §6.1、§9):
|
||||
原始檔存入 raw/originals/ 並登記 hash → 轉換 → Markdown 存入 raw/converted/
|
||||
並登記 hash 與原始檔對應 → 之後攝入管線只讀 converted 層。
|
||||
|
||||
用法:python tools/convert/convert.py <檔案路徑或URL>... [--vision] [--dry-run] [--root DIR]
|
||||
支援 .docx/.xlsx/.pdf/.pptx/.html 與 URL;舊版 .doc/.xls/.ppt 經 LibreOffice 升版;
|
||||
--vision 讓掃描/圖片型 PDF 走本地 vision 模型轉錄(見 from_vision.py)。
|
||||
--verify 對帳模式:re-hash 全部登記檔、掃出未登記檔,純唯讀不改檔,供刪除/竄改後稽核。
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import hashlib
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import traceback
|
||||
import urllib.parse
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import from_docx
|
||||
import from_legacy
|
||||
import from_pdf
|
||||
import from_pptx
|
||||
import from_web
|
||||
import from_xlsx
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
CONVERTERS = {
|
||||
".docx": (from_docx.convert, "from_docx", "docx"),
|
||||
".xlsx": (from_xlsx.convert, "from_xlsx", "xlsx"),
|
||||
".pdf": (from_pdf.convert, "from_pdf", "pdf"),
|
||||
".pptx": (from_pptx.convert, "from_pptx", "pptx"),
|
||||
}
|
||||
# 舊版 OLE 二進位 → 現代格式(經 LibreOffice 升版後走上表對應轉換器)。
|
||||
# <!-- ASSUMPTION: media_type 記真實舊格式(doc/xls/ppt),擴充 §9 的列舉;
|
||||
# converter 記為 "from_docx+libreoffice" 等,保留兩段式 provenance。 -->
|
||||
LEGACY_MAP = {".doc": ".docx", ".xls": ".xlsx", ".ppt": ".pptx"}
|
||||
|
||||
|
||||
def _now():
|
||||
return datetime.datetime.now().astimezone().isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def _slug(s):
|
||||
s = re.sub(r"[^a-z0-9一-鿿]+", "-", s.lower()).strip("-")
|
||||
return s or "page"
|
||||
|
||||
|
||||
def load_manifest(root):
|
||||
p = root / "raw" / "manifest.json"
|
||||
m = json.loads(p.read_text(encoding="utf-8"))
|
||||
if m.get("version") != 1 or "files" not in m:
|
||||
raise SystemExit(f"manifest 結構不符:{p}")
|
||||
return m
|
||||
|
||||
|
||||
def save_manifest(root, m):
|
||||
p = root / "raw" / "manifest.json"
|
||||
tmp = p.with_suffix(".json.tmp")
|
||||
tmp.write_text(json.dumps(m, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
tmp.replace(p) # 原子替換,避免寫壞 manifest
|
||||
|
||||
|
||||
def _find_by_sha(m, sha):
|
||||
for path, e in m["files"].items():
|
||||
if e["kind"] == "original" and e["sha256"] == sha:
|
||||
return path
|
||||
return None
|
||||
|
||||
|
||||
def _dest_name(dir_, name, sha):
|
||||
"""既有同名檔且內容不同時,附 hash 前 8 碼避免覆蓋(raw 不可變)。"""
|
||||
p = dir_ / name
|
||||
if p.exists() and hashlib.sha256(p.read_bytes()).hexdigest() != sha:
|
||||
stem, ext = pathlib.Path(name).stem, pathlib.Path(name).suffix
|
||||
return f"{stem}-{sha[:8]}{ext}"
|
||||
return name
|
||||
|
||||
|
||||
def _register_pair(m, root, orig_rel, orig_entry, md_rel, md_text, converter, dry):
|
||||
if not md_text.strip():
|
||||
# 空轉換結果幾乎都是失敗(如純圖片 docx);登記前示警,避免空頁靜默入庫
|
||||
print(f"warning: {orig_rel} 轉換結果為空白,請人工檢查來源", file=sys.stderr)
|
||||
if not dry:
|
||||
md_path = root / md_rel
|
||||
# write_bytes(非 write_text):登記的 sha256 算在 md_text.encode() 上,
|
||||
# 而 write_text 在 Windows 會把 \n 轉 \r\n,使磁碟 bytes 與登記 hash 不符,
|
||||
# 令日後「還原後 re-hash 比對」與 --verify 失準。寫入即登記的那份 bytes。
|
||||
md_path.write_bytes(md_text.encode("utf-8"))
|
||||
m["files"][orig_rel] = orig_entry
|
||||
m["files"][md_rel] = {
|
||||
"kind": "converted",
|
||||
"sha256": hashlib.sha256(md_text.encode("utf-8")).hexdigest(),
|
||||
"original_path": orig_rel,
|
||||
"original_sha256": orig_entry["sha256"],
|
||||
"converter": converter,
|
||||
"converted_at": _now(),
|
||||
}
|
||||
print(f"converted: {orig_rel} -> {md_rel}" + (" [dry-run]" if dry else ""))
|
||||
|
||||
|
||||
def _vision_pdf(root, pdf_path, assets, conv_name):
|
||||
"""--vision fallback(AGENTS.md §3 稽核原則):needs_ocr 的 PDF 逐頁 render →
|
||||
本地 vision 模型轉錄。回傳 (markdown, status, converter)。
|
||||
|
||||
成功 → ('...', 'ok', 'from_vision');vision 執行期失敗 → 降級回 needs_ocr(不遺失檔案)。
|
||||
設定錯誤(未設 vision / :cloud)由 from_vision 拋 SystemExit,向上傳播不靜默降級。
|
||||
"""
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) # tools/ 供 import kb
|
||||
import kb
|
||||
import to_image
|
||||
import from_vision
|
||||
cfg = kb.load_config(root)
|
||||
md_dir = root / "raw" / "converted"
|
||||
try:
|
||||
images = to_image.render_pdf_pages(pdf_path, assets)
|
||||
texts, model = from_vision.transcribe(cfg, images)
|
||||
except SystemExit:
|
||||
raise
|
||||
except Exception as e:
|
||||
print(f"warning: {pathlib.Path(pdf_path).name} vision 轉錄失敗,降級為 needs_ocr:"
|
||||
f"{type(e).__name__}: {e}", file=sys.stderr)
|
||||
return "", "needs_ocr", conv_name
|
||||
blocks = []
|
||||
for i, (img, text) in enumerate(zip(images, texts), 1):
|
||||
rel = img.relative_to(md_dir).as_posix()
|
||||
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||
blocks.append(f"<!-- page {i} (vision: {model}) -->\n\n\n\n{text}")
|
||||
return "\n\n".join(blocks), "ok", "from_vision"
|
||||
|
||||
|
||||
def verify(root, m):
|
||||
"""對帳模式:re-hash 每個 manifest 條目對應的檔案,並掃出未登記的 raw 檔。
|
||||
純唯讀——不改任何檔案(含 manifest),只回報。有不一致以 SystemExit(1) 表示,
|
||||
可供 CI/排程當閘門用。
|
||||
|
||||
偵測三類意外:
|
||||
missing —— 帳本有登記,磁碟上檔案不見了(手動刪除)。
|
||||
mismatch —— 檔案還在但 sha256 與登記值不符(raw 層被就地改動,違反 §1.4)。
|
||||
unregistered —— raw/originals 或 raw/converted 下有檔卻不在帳本;
|
||||
converted/assets/* 由 markdown 連結而非帳本追蹤,故略過,.gitkeep 亦略過。
|
||||
"""
|
||||
missing, mismatch = [], []
|
||||
for rel, e in m["files"].items():
|
||||
p = root / rel
|
||||
if not p.is_file():
|
||||
missing.append(rel)
|
||||
elif hashlib.sha256(p.read_bytes()).hexdigest() != e["sha256"]:
|
||||
mismatch.append(rel)
|
||||
|
||||
unregistered = []
|
||||
for sub in ("originals", "converted"):
|
||||
base = root / "raw" / sub
|
||||
if not base.is_dir():
|
||||
continue
|
||||
for p in base.rglob("*"):
|
||||
if not p.is_file() or p.name == ".gitkeep":
|
||||
continue
|
||||
if "assets" in p.relative_to(base).parts: # 由 markdown 連結,不入帳本
|
||||
continue
|
||||
rel = p.relative_to(root).as_posix()
|
||||
if rel not in m["files"]:
|
||||
unregistered.append(rel)
|
||||
|
||||
for rel in missing:
|
||||
print(f"missing: {rel}(帳本有登記,磁碟上不見了)", file=sys.stderr)
|
||||
for rel in mismatch:
|
||||
print(f"mismatch: {rel}(sha256 與登記值不符,raw 層被就地改動)", file=sys.stderr)
|
||||
for rel in sorted(unregistered):
|
||||
print(f"unregistered: {rel}(磁碟上有檔,帳本未登記)", file=sys.stderr)
|
||||
|
||||
if missing or mismatch or unregistered:
|
||||
print(f"\nverify: {len(m['files'])} 筆登記 → "
|
||||
f"{len(missing)} missing / {len(mismatch)} mismatch / "
|
||||
f"{len(unregistered)} unregistered(純唯讀,未改任何檔)", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
print(f"verify: {len(m['files'])} 筆登記全部對得上,raw/ 無未登記檔案")
|
||||
|
||||
|
||||
def process_local(root, m, path, dry, vision=False):
|
||||
src = pathlib.Path(path).resolve()
|
||||
if not src.is_file():
|
||||
raise FileNotFoundError(src)
|
||||
ext = src.suffix.lower()
|
||||
|
||||
sha = hashlib.sha256(src.read_bytes()).hexdigest()
|
||||
if _find_by_sha(m, sha):
|
||||
print(f"skip: {src.name} 已登記(hash 相同)")
|
||||
return
|
||||
|
||||
# raw/originals 一律保存「真正的原始檔」(舊格式也是),維持 provenance(§1.4)
|
||||
originals = root / "raw" / "originals"
|
||||
dest = src if src.parent == originals.resolve() else originals / _dest_name(originals, src.name, sha)
|
||||
orig_rel = f"raw/originals/{dest.name}"
|
||||
|
||||
tmpdir = None
|
||||
try:
|
||||
if ext in LEGACY_MAP:
|
||||
if dry:
|
||||
print(f"legacy: {src.name} 需 LibreOffice 升版為 {LEGACY_MAP[ext]}(dry-run 不執行)")
|
||||
return
|
||||
tmpdir = tempfile.mkdtemp(prefix="ppqa-legacy-")
|
||||
upgraded = from_legacy.upgrade(src, tmpdir) # 找不到 soffice / 升版失敗 → RuntimeError
|
||||
conv, base_name, _ = CONVERTERS[upgraded.suffix.lower()]
|
||||
conv_name, media, conv_src = f"{base_name}+libreoffice", ext.lstrip("."), upgraded
|
||||
elif ext in (".html", ".htm"):
|
||||
conv = (lambda p, assets_dir=None, md_dir=None: from_web.extract(
|
||||
pathlib.Path(p).read_text(encoding="utf-8", errors="replace")))
|
||||
conv_name, media, conv_src = "from_web", "html", src
|
||||
elif ext in CONVERTERS:
|
||||
conv, conv_name, media = CONVERTERS[ext]
|
||||
conv_src = src
|
||||
else:
|
||||
raise ValueError(f"不支援的格式:{src.name}")
|
||||
|
||||
if not dry and dest is not src and not dest.exists():
|
||||
dest.write_bytes(src.read_bytes())
|
||||
orig_entry = {"kind": "original", "sha256": sha, "media_type": media,
|
||||
"added_at": _now(), "status": "converted"}
|
||||
|
||||
md_name = f"{dest.stem}-{sha[:8]}.md"
|
||||
assets = None if dry else root / "raw" / "converted" / "assets" / f"{dest.stem}-{sha[:8]}"
|
||||
md_text, status = conv(str(conv_src), assets_dir=assets,
|
||||
md_dir=root / "raw" / "converted")
|
||||
if status == "needs_ocr":
|
||||
if vision and not dry:
|
||||
md_text, status, conv_name = _vision_pdf(root, conv_src, assets, conv_name)
|
||||
if status == "needs_ocr":
|
||||
orig_entry["status"] = "needs_ocr"
|
||||
if not dry:
|
||||
m["files"][orig_rel] = orig_entry
|
||||
print(f"needs_ocr: {orig_rel} 無文字層,已登記並跳過"
|
||||
+ ("(--vision 轉錄未成功)" if vision and not dry else "(不擅自 OCR)")
|
||||
+ (" [dry-run]" if dry else ""))
|
||||
return
|
||||
_register_pair(m, root, orig_rel, orig_entry, f"raw/converted/{md_name}",
|
||||
md_text, conv_name, dry)
|
||||
finally:
|
||||
if tmpdir:
|
||||
shutil.rmtree(tmpdir, ignore_errors=True)
|
||||
|
||||
|
||||
def process_url(root, m, url, dry):
|
||||
html = from_web.fetch(url)
|
||||
sha = hashlib.sha256(html.encode("utf-8")).hexdigest()
|
||||
if _find_by_sha(m, sha):
|
||||
print(f"skip: {url} 已登記(快照 hash 相同)")
|
||||
return
|
||||
u = urllib.parse.urlparse(url)
|
||||
slug = _slug(f"{u.netloc}{u.path}")[:80]
|
||||
originals = root / "raw" / "originals"
|
||||
name = _dest_name(originals, f"{slug}.html", sha)
|
||||
orig_rel = f"raw/originals/{name}"
|
||||
orig_entry = {"kind": "original", "sha256": sha, "media_type": "html",
|
||||
"added_at": _now(), "status": "converted",
|
||||
"source_url": url, "fetched_at": _now()}
|
||||
if not dry:
|
||||
# write_bytes:同 §_register_pair,登記 hash 算在 html.encode() 上,
|
||||
# 避免 Windows 換行轉換讓快照 bytes 與登記 hash 不符(完整快照,來源可能消失)
|
||||
(originals / name).write_bytes(html.encode("utf-8"))
|
||||
md_text, status = from_web.extract(html)
|
||||
if status != "ok":
|
||||
orig_entry["status"] = "pending"
|
||||
if not dry:
|
||||
m["files"][orig_rel] = orig_entry
|
||||
print(f"pending: {url} 萃取不到正文,僅保存快照" + (" [dry-run]" if dry else ""))
|
||||
return
|
||||
md_name = f"{pathlib.Path(name).stem}-{sha[:8]}.md"
|
||||
_register_pair(m, root, orig_rel, orig_entry, f"raw/converted/{md_name}",
|
||||
md_text, "from_web", dry)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("inputs", nargs="*", help="檔案路徑或 http(s) URL")
|
||||
ap.add_argument("--verify", action="store_true",
|
||||
help="對帳模式:re-hash 全部登記檔、掃出未登記檔,純唯讀不改檔;"
|
||||
"有不一致以退出碼 1 表示(不吃 inputs)")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
ap.add_argument("--vision", action="store_true",
|
||||
help="掃描/圖片型 PDF(needs_ocr)改走本地 vision 模型轉錄,"
|
||||
"而非跳過(需 config/models.yaml 設定 tasks.vision)")
|
||||
ap.add_argument("--traceback", action="store_true",
|
||||
help="轉換失敗時額外印出完整堆疊,供定位除錯")
|
||||
ap.add_argument("--root", default=str(ROOT), help="專案根目錄(測試用)")
|
||||
a = ap.parse_args(argv)
|
||||
root = pathlib.Path(a.root).resolve()
|
||||
m = load_manifest(root)
|
||||
if a.verify:
|
||||
verify(root, m)
|
||||
return
|
||||
if not a.inputs:
|
||||
ap.error("需指定至少一個檔案/URL,或改用 --verify 對帳")
|
||||
failed = []
|
||||
for item in a.inputs:
|
||||
try:
|
||||
if item.startswith(("http://", "https://")):
|
||||
process_url(root, m, item, a.dry_run)
|
||||
else:
|
||||
process_local(root, m, item, a.dry_run, vision=a.vision)
|
||||
except Exception as e: # 單檔失敗不中斷批次,最後彙總報錯
|
||||
failed.append((item, e))
|
||||
# 帶例外型別:只印訊息字串常無法判斷成因(如 KeyError 只印 key)
|
||||
print(f"error: {item}: {type(e).__name__}: {e}", file=sys.stderr)
|
||||
if a.traceback:
|
||||
traceback.print_exc()
|
||||
if not a.dry_run:
|
||||
save_manifest(root, m)
|
||||
if failed:
|
||||
# 尾端彙總,避免失敗行淹沒在大批次輸出中而漏看
|
||||
print(f"\n{len(failed)} 個項目轉換失敗:", file=sys.stderr)
|
||||
for item, e in failed:
|
||||
print(f" - {item}: {type(e).__name__}: {e}", file=sys.stderr)
|
||||
if not a.traceback:
|
||||
print("(加 --traceback 可印完整堆疊定位)", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
98
tools/convert/from_docx.py
Normal file
98
tools/convert/from_docx.py
Normal file
@@ -0,0 +1,98 @@
|
||||
"""Word (.docx) → Markdown。保留標題層級與表格結構,圖片抽出為附檔並留佔位引用。
|
||||
|
||||
可獨立執行:python tools/convert/from_docx.py <file.docx> [-o out.md] [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
import re
|
||||
|
||||
import docx
|
||||
from docx.oxml.ns import qn
|
||||
from docx.table import Table
|
||||
from docx.text.paragraph import Paragraph
|
||||
|
||||
|
||||
def _heading_level(par):
|
||||
# ponytail: 只認 "Heading N" / "標題 N" 樣式名;其他語系的 Word 樣式名不涵蓋,
|
||||
# 升級路徑:改讀 w:outlineLvl。
|
||||
m = re.match(r"(?:Heading|標題)\s*(\d)", par.style.name or "")
|
||||
return int(m.group(1)) if m else None
|
||||
|
||||
|
||||
def _table_md(tbl):
|
||||
rows = [
|
||||
[c.text.strip().replace("\n", " ").replace("|", "\\|") for c in row.cells]
|
||||
for row in tbl.rows
|
||||
]
|
||||
if not rows:
|
||||
return ""
|
||||
width = max(len(r) for r in rows)
|
||||
rows = [r + [""] * (width - len(r)) for r in rows]
|
||||
out = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width]
|
||||
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def _par_md(par, doc, assets_dir, md_dir, counter):
|
||||
parts = []
|
||||
for run in par.runs:
|
||||
for blip in run._element.findall(".//" + qn("a:blip")):
|
||||
rid = blip.get(qn("r:embed"))
|
||||
if not rid:
|
||||
continue
|
||||
part = doc.part.related_parts[rid]
|
||||
ext = pathlib.Path(part.partname).suffix or ".png"
|
||||
counter[0] += 1
|
||||
name = f"img{counter[0]}{ext}"
|
||||
if assets_dir is not None:
|
||||
assets_dir.mkdir(parents=True, exist_ok=True)
|
||||
(assets_dir / name).write_bytes(part.blob)
|
||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||
else:
|
||||
rel = name # dry-run:僅佔位
|
||||
# 連結目標以 <> 包住:來源檔名可能含空格或不平衡括號,兩者都會讓
|
||||
# Markdown 連結失效(圖片存在卻整行退化成純文字)
|
||||
parts.append(f"")
|
||||
parts.append(run.text)
|
||||
text = "".join(parts).strip()
|
||||
lvl = _heading_level(par)
|
||||
if lvl and text:
|
||||
return "#" * lvl + " " + text
|
||||
return text
|
||||
|
||||
|
||||
def convert(src, assets_dir=None, md_dir=None):
|
||||
"""回傳 (markdown, status)。status 恆為 'ok'(docx 必有文字層)。"""
|
||||
src = pathlib.Path(src)
|
||||
md_dir = pathlib.Path(md_dir) if md_dir else src.parent
|
||||
doc = docx.Document(str(src))
|
||||
counter = [0]
|
||||
blocks = []
|
||||
for child in doc.element.body.iterchildren():
|
||||
if child.tag == qn("w:p"):
|
||||
blocks.append(_par_md(Paragraph(child, doc), doc, assets_dir, md_dir, counter))
|
||||
elif child.tag == qn("w:tbl"):
|
||||
blocks.append(_table_md(Table(child, doc)))
|
||||
md = "\n\n".join(b for b in blocks if b)
|
||||
return md, "ok"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
if a.dry_run:
|
||||
md, status = convert(a.src)
|
||||
print(f"[dry-run] {a.src}: status={status}, {len(md)} chars, "
|
||||
f"{md.count(chr(10)) + 1} lines")
|
||||
return
|
||||
out = pathlib.Path(a.output) if a.output else pathlib.Path(a.src).with_suffix(".md")
|
||||
md, _ = convert(a.src, assets_dir=out.parent / f"{out.stem}-assets", md_dir=out.parent)
|
||||
out.write_text(md, encoding="utf-8")
|
||||
print(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
91
tools/convert/from_legacy.py
Normal file
91
tools/convert/from_legacy.py
Normal file
@@ -0,0 +1,91 @@
|
||||
"""舊版 OLE 二進位格式(.doc/.xls/.ppt)→ 現代格式(.docx/.xlsx/.pptx)升版。
|
||||
|
||||
以 LibreOffice headless 在本機升版(deterministic、內容不外流,AGENTS.md §1.1),
|
||||
升版後由 convert.py 回頭走既有 from_docx / from_xlsx / from_pptx 轉換器。
|
||||
升版產物是暫時中介檔,不進 raw/;raw/originals 仍保存真正的舊格式原始檔(§1.4)。
|
||||
|
||||
可獨立執行:python tools/convert/from_legacy.py <file.doc> [--out DIR]
|
||||
"""
|
||||
import argparse
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
LEGACY_TARGET = {".doc": "docx", ".xls": "xlsx", ".ppt": "pptx"}
|
||||
|
||||
# 常見 Windows 安裝路徑;其他平台靠 PATH(soffice/libreoffice)或 SOFFICE_BIN 覆寫。
|
||||
_WIN_CANDIDATES = (
|
||||
r"C:\Program Files\LibreOffice\program\soffice.exe",
|
||||
r"C:\Program Files (x86)\LibreOffice\program\soffice.exe",
|
||||
)
|
||||
|
||||
|
||||
def find_soffice():
|
||||
"""定位 LibreOffice 執行檔。SOFFICE_BIN 環境變數優先(唯一允許的覆寫來源)。"""
|
||||
env = os.environ.get("SOFFICE_BIN")
|
||||
if env and pathlib.Path(env).is_file():
|
||||
return env
|
||||
for c in _WIN_CANDIDATES:
|
||||
if pathlib.Path(c).is_file():
|
||||
return c
|
||||
return shutil.which("soffice") or shutil.which("libreoffice")
|
||||
|
||||
|
||||
def upgrade(src, out_dir, timeout=180):
|
||||
"""升版 src 至 out_dir,回傳升版後檔案路徑。
|
||||
|
||||
找不到 LibreOffice、格式不支援、或升版未產出檔案時拋 RuntimeError(由
|
||||
convert.py 逐檔捕捉並回報,不中斷批次)。
|
||||
"""
|
||||
src = pathlib.Path(src)
|
||||
ext = src.suffix.lower()
|
||||
if ext not in LEGACY_TARGET:
|
||||
raise RuntimeError(f"非舊版格式:{src.name}")
|
||||
soffice = find_soffice()
|
||||
if not soffice:
|
||||
raise RuntimeError(
|
||||
"找不到 LibreOffice(soffice);.doc/.xls/.ppt 升版需要它。"
|
||||
"請安裝 LibreOffice,或以 SOFFICE_BIN 環境變數指定執行檔路徑。")
|
||||
|
||||
out_dir = pathlib.Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
target = LEGACY_TARGET[ext]
|
||||
# 每次呼叫用獨立 user profile,避免 headless 並行/殘留 profile 造成的鎖定失敗
|
||||
# (LibreOffice headless 的已知痛點,§13.3:真實環境不是規格書上的理想值)。
|
||||
profile = out_dir / ".lo-profile"
|
||||
try:
|
||||
subprocess.run(
|
||||
[soffice, f"-env:UserInstallation=file:///{profile.as_posix().lstrip('/')}",
|
||||
"--headless", "--convert-to", target, "--outdir", str(out_dir), str(src)],
|
||||
check=True, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.CalledProcessError as e:
|
||||
raise RuntimeError(f"LibreOffice 升版失敗(exit {e.returncode}):{e.stderr.strip()}")
|
||||
except subprocess.TimeoutExpired:
|
||||
raise RuntimeError(f"LibreOffice 升版逾時(>{timeout}s):{src.name}")
|
||||
finally:
|
||||
shutil.rmtree(profile, ignore_errors=True)
|
||||
|
||||
out = out_dir / f"{src.stem}.{target}"
|
||||
if not out.is_file():
|
||||
raise RuntimeError(f"LibreOffice 未產出預期檔案:{out.name}")
|
||||
return out
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("--out", help="升版產物輸出目錄(預設暫存目錄)")
|
||||
a = ap.parse_args(argv)
|
||||
out_dir = pathlib.Path(a.out) if a.out else pathlib.Path(tempfile.mkdtemp(prefix="ppqa-legacy-"))
|
||||
try:
|
||||
print(upgrade(a.src, out_dir))
|
||||
except RuntimeError as e:
|
||||
print(f"error: {e}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
71
tools/convert/from_pdf.py
Normal file
71
tools/convert/from_pdf.py
Normal file
@@ -0,0 +1,71 @@
|
||||
"""PDF → Markdown。無文字層(掃描件)回報 needs_ocr 並跳過,不擅自呼叫 OCR。
|
||||
|
||||
可獨立執行:python tools/convert/from_pdf.py <file.pdf> [-o out.md] [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
|
||||
import fitz # pymupdf
|
||||
|
||||
# ponytail: 掃描件偵測用「平均每頁字元數 < 25」的 naive heuristic,
|
||||
# 混合型 PDF(部分頁掃描)會整份判為有文字層;升級路徑:逐頁判定 + 逐頁標記。
|
||||
_MIN_CHARS_PER_PAGE = 25
|
||||
|
||||
|
||||
def convert(src, assets_dir=None, md_dir=None):
|
||||
"""回傳 (markdown, status)。status ∈ {'ok', 'needs_ocr'}。
|
||||
|
||||
ponytail: 表格輸出為純文字行(不重建表格結構),升級路徑:page.find_tables()。
|
||||
"""
|
||||
src = pathlib.Path(src)
|
||||
md_dir = pathlib.Path(md_dir) if md_dir else src.parent
|
||||
doc = fitz.open(str(src))
|
||||
pages = [page.get_text("text").strip() for page in doc]
|
||||
total = sum(len(p) for p in pages)
|
||||
if total < _MIN_CHARS_PER_PAGE * max(len(pages), 1):
|
||||
doc.close()
|
||||
return "", "needs_ocr"
|
||||
|
||||
counter = 0
|
||||
blocks = []
|
||||
for i, page in enumerate(doc):
|
||||
blocks.append(f"<!-- page {i + 1} -->\n\n{pages[i]}")
|
||||
if assets_dir is None:
|
||||
continue
|
||||
for img in page.get_images(full=True):
|
||||
xref = img[0]
|
||||
pix = fitz.Pixmap(doc, xref)
|
||||
if pix.n > 4: # CMYK 等 → 轉 RGB
|
||||
pix = fitz.Pixmap(fitz.csRGB, pix)
|
||||
counter += 1
|
||||
name = f"img{counter}.png"
|
||||
assets_dir.mkdir(parents=True, exist_ok=True)
|
||||
pix.save(str(assets_dir / name))
|
||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||
blocks.append(f"")
|
||||
doc.close()
|
||||
return "\n\n".join(blocks), "ok"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
if a.dry_run:
|
||||
md, status = convert(a.src)
|
||||
print(f"[dry-run] {a.src}: status={status}, {len(md)} chars")
|
||||
return
|
||||
out = pathlib.Path(a.output) if a.output else pathlib.Path(a.src).with_suffix(".md")
|
||||
md, status = convert(a.src, assets_dir=out.parent / f"{out.stem}-assets", md_dir=out.parent)
|
||||
if status == "needs_ocr":
|
||||
print(f"needs_ocr: {a.src} 無文字層,已跳過(不擅自 OCR)")
|
||||
raise SystemExit(2)
|
||||
out.write_text(md, encoding="utf-8")
|
||||
print(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
75
tools/convert/from_pptx.py
Normal file
75
tools/convert/from_pptx.py
Normal file
@@ -0,0 +1,75 @@
|
||||
"""PowerPoint (.pptx) → Markdown。每張投影片一節,含表格、圖片佔位與講者備註。
|
||||
|
||||
可獨立執行:python tools/convert/from_pptx.py <file.pptx> [-o out.md] [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
|
||||
from pptx import Presentation
|
||||
from pptx.enum.shapes import MSO_SHAPE_TYPE
|
||||
|
||||
|
||||
def _table_md(tbl):
|
||||
rows = [
|
||||
[c.text.strip().replace("|", "\\|").replace("\n", " ") for c in row.cells]
|
||||
for row in tbl.rows
|
||||
]
|
||||
if not rows:
|
||||
return ""
|
||||
width = len(rows[0])
|
||||
out = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width]
|
||||
out += ["| " + " | ".join(r) + " |" for r in rows[1:]]
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def convert(src, assets_dir=None, md_dir=None):
|
||||
"""回傳 (markdown, status)。status 恆為 'ok'。"""
|
||||
src = pathlib.Path(src)
|
||||
md_dir = pathlib.Path(md_dir) if md_dir else src.parent
|
||||
prs = Presentation(str(src))
|
||||
counter = 0
|
||||
sections = []
|
||||
for i, slide in enumerate(prs.slides, 1):
|
||||
blocks = [f"## Slide {i}"]
|
||||
for shape in slide.shapes:
|
||||
if shape.has_text_frame and shape.text_frame.text.strip():
|
||||
blocks.append(shape.text_frame.text.strip())
|
||||
elif shape.has_table:
|
||||
blocks.append(_table_md(shape.table))
|
||||
elif shape.shape_type == MSO_SHAPE_TYPE.PICTURE:
|
||||
counter += 1
|
||||
name = f"img{counter}.{shape.image.ext}"
|
||||
if assets_dir is not None:
|
||||
assets_dir.mkdir(parents=True, exist_ok=True)
|
||||
(assets_dir / name).write_bytes(shape.image.blob)
|
||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||
else:
|
||||
rel = name
|
||||
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||
blocks.append(f"")
|
||||
if slide.has_notes_slide:
|
||||
notes = slide.notes_slide.notes_text_frame.text.strip()
|
||||
if notes:
|
||||
blocks.append(f"### 講者備註\n\n{notes}")
|
||||
sections.append("\n\n".join(blocks))
|
||||
return "\n\n".join(sections), "ok"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
if a.dry_run:
|
||||
md, status = convert(a.src)
|
||||
print(f"[dry-run] {a.src}: status={status}, {len(md)} chars")
|
||||
return
|
||||
out = pathlib.Path(a.output) if a.output else pathlib.Path(a.src).with_suffix(".md")
|
||||
md, _ = convert(a.src, assets_dir=out.parent / f"{out.stem}-assets", md_dir=out.parent)
|
||||
out.write_text(md, encoding="utf-8")
|
||||
print(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
61
tools/convert/from_vision.py
Normal file
61
tools/convert/from_vision.py
Normal file
@@ -0,0 +1,61 @@
|
||||
"""影像 → Markdown 轉錄(本地 vision 模型,AGENTS.md §1.1 資料落地)。
|
||||
|
||||
用於 vision fallback:文字抽取失敗的頁面 render 成影像後,逐頁交本地 vision
|
||||
模型忠實轉錄為 Markdown。模型設定讀 config/models.yaml 的 tasks.vision(§1.2),
|
||||
未設定即報錯、不使用預設值;:cloud 模型一律拒絕(影像不得離開本機)。
|
||||
"""
|
||||
import base64
|
||||
import pathlib
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) # tools/ 供 import kb
|
||||
import kb
|
||||
|
||||
VISION_PROMPT = (
|
||||
"你是文件轉錄引擎。忠實轉錄此頁影像中的所有文字與表格內容為 Markdown:"
|
||||
"表格用 Markdown table,保留標題階層。只輸出轉錄內容本身,不要加任何說明或臆測。"
|
||||
"影像中沒有可辨識內容時,只回覆「(本頁無可辨識文字)」。")
|
||||
|
||||
|
||||
def _require_vision(cfg):
|
||||
"""取得並驗證 vision 模型設定。缺設定 / :cloud 皆報錯(信任邊界,§1.1/§1.2)。"""
|
||||
spec = (cfg.get("tasks") or {}).get("vision")
|
||||
if not isinstance(spec, dict) or not spec.get("model") or spec["model"] == "CHANGE_ME":
|
||||
raise SystemExit(
|
||||
"models.yaml 未設定 tasks.vision.model;--vision 需要合規本地 vision 模型"
|
||||
"(請自行確認非中國關聯、非 :cloud,AGENTS.md §1.1/§1.3)。")
|
||||
if str(spec["model"]).endswith(":cloud"):
|
||||
raise SystemExit("tasks.vision.model 為 :cloud 模型,禁止(文件影像不得離開本機,§1.1)。")
|
||||
return spec
|
||||
|
||||
|
||||
def _b64(path):
|
||||
return base64.b64encode(pathlib.Path(path).read_bytes()).decode("ascii")
|
||||
|
||||
|
||||
def transcribe(cfg, image_paths):
|
||||
"""逐頁轉錄影像,回傳 (每頁 markdown 列表, 實際使用的 model)。
|
||||
|
||||
單頁重試 max_retries 次仍失敗即拋 RuntimeError(vision 無文字 fallback 可用——
|
||||
fallback 模型非 vision,不切換)。
|
||||
"""
|
||||
spec = _require_vision(cfg)
|
||||
model = spec["model"]
|
||||
retries = int(spec.get("max_retries", 2))
|
||||
temperature = float(spec.get("temperature", 0.1))
|
||||
pages = []
|
||||
for idx, img in enumerate(image_paths, 1):
|
||||
b64 = _b64(img)
|
||||
last = None
|
||||
for _ in range(max(1, retries)):
|
||||
try:
|
||||
text = kb.ollama_chat(cfg, model, VISION_PROMPT, temperature, images=[b64])
|
||||
if text and text.strip():
|
||||
pages.append(text.strip())
|
||||
break
|
||||
last = "空回應"
|
||||
except Exception as e:
|
||||
last = f"{type(e).__name__}: {e}"
|
||||
else:
|
||||
raise RuntimeError(f"vision 轉錄失敗(第 {idx} 頁,model={model}):{last}")
|
||||
return pages, model
|
||||
53
tools/convert/from_web.py
Normal file
53
tools/convert/from_web.py
Normal file
@@ -0,0 +1,53 @@
|
||||
"""網頁 URL → 完整 HTML 快照 + Markdown 正文萃取(trafilatura,全程本地處理)。
|
||||
|
||||
可獨立執行:python tools/convert/from_web.py <url> [-o out.md] [--snapshot out.html] [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
|
||||
import trafilatura
|
||||
|
||||
|
||||
def fetch(url):
|
||||
"""抓取 URL,回傳完整 HTML 字串。失敗時拋 RuntimeError。"""
|
||||
html = trafilatura.fetch_url(url)
|
||||
if not html:
|
||||
raise RuntimeError(f"抓取失敗:{url}")
|
||||
return html
|
||||
|
||||
|
||||
def extract(html):
|
||||
"""HTML → Markdown 正文。回傳 (markdown, status)。萃取不到正文時 status='empty'。"""
|
||||
md = trafilatura.extract(
|
||||
html, output_format="markdown", include_tables=True, include_links=True
|
||||
)
|
||||
if not md or not md.strip():
|
||||
return "", "empty"
|
||||
return md.strip(), "ok"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("url")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("--snapshot", help="HTML 快照輸出路徑")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
html = fetch(a.url)
|
||||
md, status = extract(html)
|
||||
if a.dry_run:
|
||||
print(f"[dry-run] {a.url}: status={status}, snapshot {len(html)} chars, "
|
||||
f"markdown {len(md)} chars")
|
||||
return
|
||||
if status != "ok":
|
||||
print(f"empty: {a.url} 萃取不到正文")
|
||||
raise SystemExit(2)
|
||||
if a.snapshot:
|
||||
pathlib.Path(a.snapshot).write_text(html, encoding="utf-8")
|
||||
out = pathlib.Path(a.output) if a.output else pathlib.Path("page.md")
|
||||
out.write_text(md, encoding="utf-8")
|
||||
print(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
56
tools/convert/from_xlsx.py
Normal file
56
tools/convert/from_xlsx.py
Normal file
@@ -0,0 +1,56 @@
|
||||
"""Excel (.xlsx) → Markdown。每個工作表一節,內容輸出為 Markdown 表格。
|
||||
|
||||
可獨立執行:python tools/convert/from_xlsx.py <file.xlsx> [-o out.md] [--dry-run]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
|
||||
import openpyxl
|
||||
|
||||
|
||||
def _cell(v):
|
||||
return "" if v is None else str(v).replace("|", "\\|").replace("\n", " ").strip()
|
||||
|
||||
|
||||
def convert(src, assets_dir=None, md_dir=None):
|
||||
"""回傳 (markdown, status)。status 恆為 'ok'。
|
||||
|
||||
ponytail: merged cell 只有左上角有值(openpyxl read_only 行為),不展開;
|
||||
升級路徑:非 read_only 模式讀 merged_cells.ranges 補值。
|
||||
"""
|
||||
wb = openpyxl.load_workbook(str(src), data_only=True, read_only=True)
|
||||
sections = []
|
||||
for ws in wb.worksheets:
|
||||
rows = [[_cell(v) for v in row] for row in ws.iter_rows(values_only=True)]
|
||||
rows = [r for r in rows if any(r)] # 去除全空列
|
||||
body = f"## 工作表:{ws.title}\n\n"
|
||||
if not rows:
|
||||
body += "(空白工作表)"
|
||||
else:
|
||||
width = max(len(r) for r in rows)
|
||||
rows = [r + [""] * (width - len(r)) for r in rows]
|
||||
lines = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * width]
|
||||
lines += ["| " + " | ".join(r) + " |" for r in rows[1:]]
|
||||
body += "\n".join(lines)
|
||||
sections.append(body)
|
||||
wb.close()
|
||||
return "\n\n".join(sections), "ok"
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("-o", "--output")
|
||||
ap.add_argument("--dry-run", action="store_true")
|
||||
a = ap.parse_args(argv)
|
||||
md, status = convert(a.src)
|
||||
if a.dry_run:
|
||||
print(f"[dry-run] {a.src}: status={status}, {len(md)} chars")
|
||||
return
|
||||
out = pathlib.Path(a.output) if a.output else pathlib.Path(a.src).with_suffix(".md")
|
||||
out.write_text(md, encoding="utf-8")
|
||||
print(out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
315
tools/convert/selfcheck.py
Normal file
315
tools/convert/selfcheck.py
Normal file
@@ -0,0 +1,315 @@
|
||||
"""Phase 2 自檢:程式化產生各格式最小測試檔,驗證轉換器與 convert.py 端到端。
|
||||
|
||||
執行:.venv/Scripts/python tools/convert/selfcheck.py
|
||||
邏輯壞掉時 assert 會失敗(AGENTS.md §13.4:一個可執行的檢查,不用框架)。
|
||||
"""
|
||||
import contextlib
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1])) # tools/ 供 import kb
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent)) # tools/convert
|
||||
import convert
|
||||
import from_docx
|
||||
import from_pdf
|
||||
import from_pptx
|
||||
import from_vision
|
||||
import from_web
|
||||
import from_xlsx
|
||||
import kb
|
||||
import to_image
|
||||
|
||||
|
||||
# 測資檔名一律用「中文 + 空格」——真實來源檔就長這樣,而 git status --porcelain
|
||||
# 對這兩者都會加引號轉義。ASCII 檔名的測資曾讓 ingest 的 preflight bug 溜過(§12)。
|
||||
DOCX, XLSX, PDF, SCAN = ("壓測 報告(1).docx", "測試案例 清單.xlsx",
|
||||
"規格 說明.pdf", "掃描件(2 期.pdf")
|
||||
PPTX, HTML, TXT, DOC = ("結案 簡報.pptx", "指引 快照.html",
|
||||
"不支援 筆記.txt", "舊版 報告.doc")
|
||||
|
||||
|
||||
def assert_links_ok(md, base):
|
||||
"""每個圖片連結都要真的 render 成 <img>、且目標檔存在。
|
||||
|
||||
只斷言 "![" 在不在會讓損毀連結矇混過關——含空格/不平衡括號的檔名會產出
|
||||
語法有效但解析失敗的連結,圖檔明明在磁碟上卻整行退化成純文字(§13.4)。
|
||||
"""
|
||||
import re
|
||||
import urllib.parse
|
||||
from markdown_it import MarkdownIt
|
||||
|
||||
want = md.count("![")
|
||||
html = MarkdownIt("commonmark").render(md)
|
||||
got = html.count("<img")
|
||||
assert got == want, f"{want} 個圖片連結只有 {got} 個 render 成功:\n{md[:300]}"
|
||||
for m in re.finditer(r'<img src="([^"]+)"', html):
|
||||
p = base / urllib.parse.unquote(m.group(1))
|
||||
assert p.is_file(), f"連結目標不存在:{p}"
|
||||
return want
|
||||
|
||||
|
||||
def make_fixtures(d):
|
||||
import fitz
|
||||
png = d / "img.png" # 內嵌圖片用:docx/pdf/pptx 三條抽圖路徑都要走到
|
||||
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 8, 8))
|
||||
pix.clear_with(128)
|
||||
pix.save(str(png))
|
||||
|
||||
import docx as docxlib
|
||||
doc = docxlib.Document()
|
||||
doc.add_heading("支付閘道測試報告", level=1)
|
||||
doc.add_paragraph("逾時缺陷於壓力測試重現。")
|
||||
t = doc.add_table(rows=2, cols=2)
|
||||
t.rows[0].cells[0].text = "案例"
|
||||
t.rows[0].cells[1].text = "結果"
|
||||
t.rows[1].cells[0].text = "TC-001"
|
||||
t.rows[1].cells[1].text = "FAIL"
|
||||
doc.add_picture(str(png))
|
||||
doc.save(d / DOCX)
|
||||
|
||||
import openpyxl
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.title = "測項"
|
||||
ws.append(["編號", "說明"])
|
||||
ws.append(["TC-001", "逾時 | 重試"])
|
||||
wb.save(d / XLSX)
|
||||
|
||||
pdf = fitz.open()
|
||||
page = pdf.new_page()
|
||||
page.insert_text((72, 72), "Payment gateway timeout defect reproduced under load test.")
|
||||
page.insert_image(fitz.Rect(72, 100, 122, 150), filename=str(png))
|
||||
pdf.save(d / PDF)
|
||||
pdf.close()
|
||||
scanned = fitz.open()
|
||||
scanned.new_page() # 無文字層 → 應判 needs_ocr
|
||||
scanned.save(d / SCAN)
|
||||
scanned.close()
|
||||
|
||||
from pptx import Presentation
|
||||
from pptx.util import Inches
|
||||
prs = Presentation()
|
||||
slide = prs.slides.add_slide(prs.slide_layouts[1])
|
||||
slide.shapes.title.text = "結案報告"
|
||||
slide.shapes.add_picture(str(png), Inches(1), Inches(3))
|
||||
slide.notes_slide.notes_text_frame.text = "備註:法遵項目全數通過"
|
||||
prs.save(d / PPTX)
|
||||
|
||||
(d / HTML).write_text(
|
||||
"<html><body><article><h1>AML 測試指引</h1>"
|
||||
+ "<p>可疑交易監控測試需涵蓋大額交易與分散交易兩種樣態。</p>" * 8
|
||||
+ "</article></body></html>", encoding="utf-8")
|
||||
|
||||
|
||||
def main():
|
||||
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-selfcheck-"))
|
||||
try:
|
||||
fx = tmp / "fx"
|
||||
fx.mkdir()
|
||||
make_fixtures(fx)
|
||||
|
||||
md, st = from_docx.convert(fx / DOCX)
|
||||
assert st == "ok" and "# 支付閘道測試報告" in md and "| TC-001 | FAIL |" in md, md
|
||||
md, st = from_xlsx.convert(fx / XLSX)
|
||||
assert st == "ok" and "## 工作表:測項" in md and "逾時 \\| 重試" in md, md
|
||||
md, st = from_pdf.convert(fx / PDF)
|
||||
assert st == "ok" and "<!-- page 1 -->" in md and "timeout defect" in md, md
|
||||
_, st = from_pdf.convert(fx / SCAN)
|
||||
assert st == "needs_ocr", st
|
||||
md, st = from_pptx.convert(fx / PPTX)
|
||||
assert st == "ok" and "## Slide 1" in md and "法遵項目全數通過" in md, md
|
||||
md, st = from_web.extract((fx / HTML).read_text(encoding="utf-8"))
|
||||
assert st == "ok" and "可疑交易監控" in md, md
|
||||
print("PASS: 5 個轉換器 + needs_ocr 偵測")
|
||||
|
||||
# convert.py 端到端(沙盒 root)
|
||||
root = tmp / "root"
|
||||
for sub in ("raw/originals", "raw/converted"):
|
||||
(root / sub).mkdir(parents=True)
|
||||
(root / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||
|
||||
convert.main([str(fx / DOCX), str(fx / XLSX), str(fx / PDF), str(fx / SCAN),
|
||||
str(fx / PPTX), str(fx / HTML), "--root", str(root)])
|
||||
m = json.loads((root / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||
origs = {k: v for k, v in m["files"].items() if v["kind"] == "original"}
|
||||
convs = {k: v for k, v in m["files"].items() if v["kind"] == "converted"}
|
||||
assert len(origs) == 6 and len(convs) == 5, m # SCAN 為 needs_ocr,無 converted
|
||||
assert origs[f"raw/originals/{SCAN}"]["status"] == "needs_ocr"
|
||||
for k, v in convs.items():
|
||||
assert (root / k).is_file(), k
|
||||
assert m["files"][v["original_path"]]["sha256"] == v["original_sha256"]
|
||||
assert (root / f"raw/originals/{DOCX}").is_file()
|
||||
print("PASS: convert.py 端到端 + manifest 對應")
|
||||
|
||||
# 圖片連結必須真的能 render——DOCX/PDF/PPTX 三條抽圖路徑都要驗到,
|
||||
# 且測資檔名刻意含空格與不平衡括號(最容易讓連結失效的形狀)
|
||||
linked = sum(assert_links_ok((root / k).read_text(encoding="utf-8"),
|
||||
root / "raw/converted") for k in convs)
|
||||
assert linked >= 3, f"應至少驗到 docx/pdf/pptx 各一個圖片連結,實際 {linked}"
|
||||
print(f"PASS: 圖片連結可 render 且目標存在({linked} 個,檔名含空格/括號)")
|
||||
|
||||
# 冪等:同檔再跑一次 → skip,manifest 不變
|
||||
before = (root / "raw/manifest.json").read_text(encoding="utf-8")
|
||||
convert.main([str(fx / DOCX), "--root", str(root)])
|
||||
assert (root / "raw/manifest.json").read_text(encoding="utf-8") == before
|
||||
print("PASS: 冪等(同 hash 跳過)")
|
||||
|
||||
# --verify 對帳(純唯讀):clean 通過、三類意外都抓到、且完全不改 manifest。
|
||||
# root 已有 docx/pdf/pptx 抽出的 assets(不入帳本)——clean 案例即證明 assets 不被誤報。
|
||||
def run_verify(rt):
|
||||
mm = json.loads((rt / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||
err, code = io.StringIO(), 0
|
||||
with contextlib.redirect_stderr(err), contextlib.redirect_stdout(io.StringIO()):
|
||||
try:
|
||||
convert.verify(rt, mm)
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
return code, err.getvalue()
|
||||
|
||||
before = (root / "raw/manifest.json").read_text(encoding="utf-8")
|
||||
code, _ = run_verify(root)
|
||||
assert code == 0, "clean 帳本應通過對帳(含未入帳本的 assets)"
|
||||
|
||||
rroot = tmp / "root_del" # missing:刪掉一個 converted .md
|
||||
shutil.copytree(root, rroot)
|
||||
gone = next(p for p in (rroot / "raw/converted").glob("*.md"))
|
||||
gone.unlink()
|
||||
code, log = run_verify(rroot)
|
||||
assert code == 1 and "missing" in log and gone.name in log, log
|
||||
|
||||
rroot = tmp / "root_mut" # mismatch:就地改動一個 original 的 bytes
|
||||
shutil.copytree(root, rroot)
|
||||
tampered = next(p for p in (rroot / "raw/originals").iterdir() if p.is_file())
|
||||
tampered.write_bytes(tampered.read_bytes() + b"tampered")
|
||||
code, log = run_verify(rroot)
|
||||
assert code == 1 and "mismatch" in log, log
|
||||
|
||||
rroot = tmp / "root_unreg" # unregistered:raw/originals 塞入未登記檔
|
||||
shutil.copytree(root, rroot)
|
||||
(rroot / "raw/originals/未登記 檔.docx").write_bytes(b"stray")
|
||||
code, log = run_verify(rroot)
|
||||
assert code == 1 and "unregistered" in log and "未登記 檔.docx" in log, log
|
||||
|
||||
assert (root / "raw/manifest.json").read_text(encoding="utf-8") == before # 全程唯讀
|
||||
print("PASS: --verify 對帳(clean/missing/mismatch/unregistered,純唯讀)")
|
||||
|
||||
# dry-run:全新沙盒不落地
|
||||
root2 = tmp / "root2"
|
||||
for sub in ("raw/originals", "raw/converted"):
|
||||
(root2 / sub).mkdir(parents=True)
|
||||
(root2 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||
convert.main([str(fx / DOCX), "--dry-run", "--root", str(root2)])
|
||||
m2 = json.loads((root2 / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||
assert m2["files"] == {} and not list((root2 / "raw/originals").iterdir())
|
||||
print("PASS: --dry-run 不落地")
|
||||
|
||||
# 失敗回報:壞輸入不中斷批次,錯誤帶例外型別 + 尾端彙總,好檔仍轉換
|
||||
root3 = tmp / "root3"
|
||||
for sub in ("raw/originals", "raw/converted"):
|
||||
(root3 / sub).mkdir(parents=True)
|
||||
(root3 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||
(fx / TXT).write_text("不支援的格式", encoding="utf-8") # → ValueError
|
||||
err = io.StringIO()
|
||||
code = None
|
||||
with contextlib.redirect_stderr(err):
|
||||
try:
|
||||
convert.main([str(fx / DOCX), str(fx / TXT),
|
||||
str(fx / "不存在 檔案.docx"), # 不存在 → FileNotFoundError
|
||||
"--root", str(root3)])
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
log = err.getvalue()
|
||||
assert code == 1, code
|
||||
assert "ValueError" in log and "FileNotFoundError" in log, log # 錯誤帶例外型別
|
||||
assert "2 個項目轉換失敗" in log, log # 尾端彙總
|
||||
m3 = json.loads((root3 / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||
assert (root3 / f"raw/originals/{DOCX}").is_file() # 好檔仍轉換並落地
|
||||
assert any(v["kind"] == "converted" for v in m3["files"].values()), m3
|
||||
print("PASS: 失敗回報(例外型別 + 尾端彙總,批次不中斷)")
|
||||
|
||||
# B: 舊格式路由 —— .doc 應走 legacy(LibreOffice),而非被當「不支援格式」拒收
|
||||
root4 = tmp / "root4"
|
||||
for sub in ("raw/originals", "raw/converted"):
|
||||
(root4 / sub).mkdir(parents=True)
|
||||
(root4 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||
(fx / DOC).write_bytes(b"\xd0\xcf\x11\xe0legacy-stub") # 非有效 .doc,僅測路由/錯誤
|
||||
err = io.StringIO()
|
||||
code = None
|
||||
with contextlib.redirect_stderr(err):
|
||||
try:
|
||||
convert.main([str(fx / DOC), "--root", str(root4)])
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
log = err.getvalue()
|
||||
assert code == 1 and DOC in log, log
|
||||
assert "不支援的格式" not in log and "RuntimeError" in log, log # 已路由 legacy,乾淨報錯
|
||||
print("PASS: 舊格式路由至 legacy(soffice 缺失/升版失敗皆乾淨報錯)")
|
||||
|
||||
# A: to_image render(deterministic,真跑)
|
||||
imgs = to_image.render_pdf_pages(fx / PDF, tmp / "imgs")
|
||||
assert imgs and all(p.is_file() and p.stat().st_size > 0 for p in imgs), imgs
|
||||
print(f"PASS: to_image render({len(imgs)} 頁 PNG)")
|
||||
|
||||
# A: from_vision 轉錄邏輯(注入假 Ollama,不需真模型)+ 合規防護
|
||||
vis_cfg = {"tasks": {"vision": {"model": "fake-vision", "temperature": 0.1,
|
||||
"max_retries": 1}}}
|
||||
orig_chat = kb.ollama_chat
|
||||
calls = {}
|
||||
|
||||
def fake_chat(cfg, model, prompt, temperature, json_format=False, images=None):
|
||||
calls["images"] = images # 記錄 base64 是否傳入
|
||||
return "可疑交易監控:大額 12,000 元"
|
||||
|
||||
kb.ollama_chat = fake_chat
|
||||
try:
|
||||
texts, model = from_vision.transcribe(vis_cfg, imgs[:1])
|
||||
finally:
|
||||
kb.ollama_chat = orig_chat
|
||||
assert model == "fake-vision" and texts and "可疑交易監控" in texts[0], texts
|
||||
assert calls.get("images"), "vision 呼叫必須帶影像"
|
||||
for bad in ({"tasks": {}}, {"tasks": {"vision": {"model": "foo:cloud"}}}):
|
||||
try:
|
||||
from_vision._require_vision(bad)
|
||||
assert False, "應拒絕未設定 / :cloud"
|
||||
except SystemExit:
|
||||
pass
|
||||
print("PASS: from_vision 轉錄 + :cloud/未設定防護")
|
||||
|
||||
# A: convert.py --vision 端到端(沙盒 config + 注入假 vision)
|
||||
root_v = tmp / "root_v"
|
||||
(root_v / "config").mkdir(parents=True)
|
||||
for sub in ("raw/originals", "raw/converted"):
|
||||
(root_v / sub).mkdir(parents=True)
|
||||
(root_v / "config/models.yaml").write_text(
|
||||
'ollama:\n base_url: "http://localhost:11434"\n timeout_seconds: 300\n'
|
||||
'tasks:\n ingest: {model: "x", temperature: 0.2, max_retries: 1}\n'
|
||||
' lint: {model: "x", temperature: 0.1, max_retries: 1}\n'
|
||||
' fallback: {model: "x"}\n'
|
||||
' vision: {model: "fake-vision", temperature: 0.1, max_retries: 1}\n'
|
||||
'limits:\n mcp_response_max_tokens: 2000\n ingest_chunk_max_chars: 12000\n',
|
||||
encoding="utf-8")
|
||||
(root_v / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||
kb.ollama_chat = fake_chat
|
||||
try:
|
||||
convert.main([str(fx / SCAN), "--vision", "--root", str(root_v)])
|
||||
finally:
|
||||
kb.ollama_chat = orig_chat
|
||||
mv = json.loads((root_v / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||
conv_rel = [k for k, v in mv["files"].items() if v["kind"] == "converted"]
|
||||
assert len(conv_rel) == 1 and mv["files"][conv_rel[0]]["converter"] == "from_vision", mv
|
||||
md_v = (root_v / conv_rel[0]).read_text(encoding="utf-8")
|
||||
assert "可疑交易監控" in md_v and "(vision" in md_v and "![page 1]" in md_v, md_v
|
||||
assert_links_ok(md_v, root_v / "raw/converted") # vision 頁的連結同樣要能 render
|
||||
print("PASS: convert.py --vision 端到端(needs_ocr → 本地 vision 轉錄)")
|
||||
|
||||
print("ALL PASS")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
48
tools/convert/to_image.py
Normal file
48
tools/convert/to_image.py
Normal file
@@ -0,0 +1,48 @@
|
||||
"""PDF 頁 → PNG 影像(pymupdf,deterministic、全程本地)。
|
||||
|
||||
供 vision fallback 使用:文字抽取失敗(needs_ocr)的掃描/圖片型 PDF 逐頁 render
|
||||
成影像後,交給 from_vision.py 以本地 vision 模型判讀。render 本身不含任何 LLM,
|
||||
維持「deterministic 工具做結構、LLM 只做語意」的分工(AGENTS.md §3 稽核原則)。
|
||||
|
||||
可獨立執行:python tools/convert/to_image.py <file.pdf> --out DIR [--dpi 150]
|
||||
"""
|
||||
import argparse
|
||||
import pathlib
|
||||
|
||||
import fitz # pymupdf
|
||||
|
||||
# ponytail: 預設 150 DPI 是「可讀性 vs 影像大小/推論成本」的折衷;掃描件文字偏小時
|
||||
# 可調高。升級路徑:依頁面尺寸/內容自適應 DPI。
|
||||
DEFAULT_DPI = 150
|
||||
|
||||
|
||||
def render_pdf_pages(src, out_dir, dpi=DEFAULT_DPI, prefix="page"):
|
||||
"""將 src 每頁 render 成 PNG,回傳影像路徑列表(依頁序)。"""
|
||||
src = pathlib.Path(src)
|
||||
out_dir = pathlib.Path(out_dir)
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
paths = []
|
||||
doc = fitz.open(str(src))
|
||||
try:
|
||||
for i, page in enumerate(doc, 1):
|
||||
pix = page.get_pixmap(dpi=dpi)
|
||||
p = out_dir / f"{prefix}{i}.png"
|
||||
pix.save(str(p))
|
||||
paths.append(p)
|
||||
finally:
|
||||
doc.close()
|
||||
return paths
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("src")
|
||||
ap.add_argument("--out", required=True, help="影像輸出目錄")
|
||||
ap.add_argument("--dpi", type=int, default=DEFAULT_DPI)
|
||||
a = ap.parse_args(argv)
|
||||
paths = render_pdf_pages(a.src, a.out, a.dpi)
|
||||
print(f"{len(paths)} 頁 → {a.out}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
134
tools/diag_refs.py
Normal file
134
tools/diag_refs.py
Normal file
@@ -0,0 +1,134 @@
|
||||
"""source_ref ↔ manifest 對帳診斷(AGENTS.md §7 schema / §14 對帳的輔助工具)。
|
||||
|
||||
lint.py 對「source_ref 不在 manifest」只報一句,無法區分成因;本工具把每個
|
||||
對不上的 source_ref 分類成可據以修復的型別(見 CAT_TITLE / FIX),讓「帳本掉
|
||||
條目」「路徑字串岔開」「檔名 hash 後綴岔開」「檔真的遺失」彼此不再混為一談。
|
||||
|
||||
純唯讀——不改任何檔(含 manifest),有對不上以結束碼 1 表示(可當搬機/改
|
||||
manifest 後的閘門)。分類邏輯與 lint 一致:以最後一個 '#' 切出 path 與 sha8。
|
||||
|
||||
用法:python tools/diag_refs.py [--root DIR]
|
||||
結束碼:0 = 全部解得開,1 = 有對不上。
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
|
||||
CAT_TITLE = {
|
||||
"hash-mismatch": "Hash 不符(帳本有此路徑,但 sha256 與 ref 不同)",
|
||||
"normalize": "路徑字串岔開(正規化後與帳本 key 相同:斜線/前綴/大小寫)",
|
||||
"stem": "檔名 hash 後綴岔開(帳本有同檔名幹、hash 後綴不同的 key)",
|
||||
"unregistered": "檔在磁碟、帳本無此條目",
|
||||
"absent": "完全無對應(帳本、磁碟、檔名幹皆無)",
|
||||
}
|
||||
FIX = {
|
||||
"hash-mismatch": "raw 被就地改動或重轉(違反 §1.4);還原檔案,或連同 source_ref 走 PR(§14.2)。",
|
||||
"normalize": "修產生岔開字串的那一個環節(路徑正規化),別逐頁改 source_ref(§12 change the ruler)。",
|
||||
"stem": "source_ref 指到不同 hash 後綴的檔名;核對正確檔名,別改帳本去遷就(§14.2)。",
|
||||
"unregistered": "帳本掉了條目、檔還在;走 branch+PR 補登 manifest(§14.3),別動 wiki 頁。",
|
||||
"absent": "source_ref 可能攝入時寫歪,或原始檔真的遺失;上報 HITL 據實記錄(§14.2)。",
|
||||
}
|
||||
# 分類輸出順序(愈可機械修復的排愈前)
|
||||
ORDER = ["hash-mismatch", "normalize", "stem", "unregistered", "absent"]
|
||||
|
||||
|
||||
def _norm(path):
|
||||
"""正規化路徑字串以偵測「同一檔、不同寫法」:反斜線→斜線、去開頭 ./、轉小寫。"""
|
||||
s = path.replace("\\", "/")
|
||||
if s.startswith("./"):
|
||||
s = s[2:]
|
||||
return s.lower()
|
||||
|
||||
|
||||
def _stem(path):
|
||||
"""取檔名幹:去目錄、去 .md、去結尾的 -<8 碼 hex>(convert 的檔名 hash 後綴)。"""
|
||||
name = path.replace("\\", "/").rsplit("/", 1)[-1]
|
||||
name = re.sub(r"\.md$", "", name, flags=re.I)
|
||||
return re.sub(r"-[0-9a-f]{8}$", "", name, flags=re.I).lower()
|
||||
|
||||
|
||||
def classify_ref(source_ref, files, exists):
|
||||
"""把單一 source_ref 分類。files:manifest['files'](path→entry);
|
||||
exists(path)→bool 判磁碟是否有該檔。回傳 (category, detail)。
|
||||
category 為 "resolved" 或 CAT_TITLE 的任一鍵。"""
|
||||
path, _, sha8 = str(source_ref).rpartition("#")
|
||||
entry = files.get(path)
|
||||
if entry is not None:
|
||||
if entry["sha256"].startswith(sha8):
|
||||
return "resolved", path
|
||||
return "hash-mismatch", f"帳本 sha256={entry['sha256'][:8]} ≠ ref #{sha8}"
|
||||
norm = _norm(path)
|
||||
for k in files:
|
||||
if _norm(k) == norm:
|
||||
return "normalize", f"帳本 key = '{k}'"
|
||||
stem = _stem(path)
|
||||
for k, e in files.items():
|
||||
if e.get("kind") == "converted" and _stem(k) == stem:
|
||||
return "stem", f"帳本 key = '{k}'"
|
||||
if exists(path):
|
||||
return "unregistered", f"磁碟有 '{path}',帳本無此 key"
|
||||
return "absent", f"'{path}' 帳本無、磁碟無、無同名幹 key"
|
||||
|
||||
|
||||
def collect(root, files, exists):
|
||||
"""掃全庫 wiki 頁,回傳 (findings, n_refs)。findings:category→[(page, ref, detail)]。"""
|
||||
findings = {c: [] for c in ORDER}
|
||||
n_refs = 0
|
||||
for p in kb.iter_pages(root):
|
||||
rel = p.relative_to(root).as_posix()
|
||||
try:
|
||||
meta, _ = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||
except Exception as e:
|
||||
findings["absent"].append((rel, "(無法解析頁面)", str(e)))
|
||||
continue
|
||||
for s in meta.get("sources") or []:
|
||||
n_refs += 1
|
||||
cat, detail = classify_ref(s, files, exists)
|
||||
if cat != "resolved":
|
||||
findings[cat].append((rel, str(s), detail))
|
||||
return findings, n_refs
|
||||
|
||||
|
||||
def format_report(findings, n_refs):
|
||||
total = sum(len(v) for v in findings.values())
|
||||
lines = [f"檢查 source_ref:{n_refs},對不上:{total}", ""]
|
||||
if total == 0:
|
||||
lines.append("全部解得開(source_ref 與 manifest 一致)。")
|
||||
return "\n".join(lines)
|
||||
for cat in ORDER:
|
||||
items = findings[cat]
|
||||
if not items:
|
||||
continue
|
||||
lines.append(f"## {CAT_TITLE[cat]}({len(items)})")
|
||||
lines.append(f"→ 修法:{FIX[cat]}")
|
||||
for page, ref, detail in items:
|
||||
lines.append(f" - [{page}] {ref}")
|
||||
lines.append(f" {detail}")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--root", default=str(kb.ROOT))
|
||||
a = ap.parse_args(argv)
|
||||
root = pathlib.Path(a.root).resolve()
|
||||
mpath = root / "raw" / "manifest.json"
|
||||
try:
|
||||
files = json.loads(mpath.read_text(encoding="utf-8"))["files"]
|
||||
except (OSError, KeyError, json.JSONDecodeError) as e:
|
||||
raise SystemExit(f"讀不到 / 解析不了 manifest:{mpath}({e})")
|
||||
|
||||
findings, n_refs = collect(root, files, lambda p: (root / p).is_file())
|
||||
print(format_report(findings, n_refs))
|
||||
if sum(len(v) for v in findings.values()):
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
326
tools/ingest.py
Normal file
326
tools/ingest.py
Normal file
@@ -0,0 +1,326 @@
|
||||
"""攝入管線(AGENTS.md §6):讀 raw/converted 的 Markdown,呼叫本地 Ollama
|
||||
產出/更新 wiki 頁,重建 index.md、追加 log.md,並以 git branch 準備 PR。
|
||||
|
||||
用法:
|
||||
python tools/ingest.py raw/converted/xxx.md [...] # 攝入指定來源
|
||||
python tools/ingest.py --all-pending # 攝入所有尚無 summary 的來源
|
||||
共用選項:[--force] [--dry-run] [--root DIR]
|
||||
|
||||
注意:--all-pending 以「目前分支」的 wiki 為準;尚未合併的 ingest PR 不會被看到。
|
||||
攝入腳本永遠不直接 commit 到 main(AGENTS.md §1.5)。
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import json
|
||||
import pathlib
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
|
||||
SUMMARY_PROMPT = """你是金融電子支付 QA 部門知識庫的攝入引擎。閱讀以下來源文件,只輸出一個 JSON 物件,不得有任何其他文字。
|
||||
|
||||
規則:
|
||||
- 文字內容用繁體中文;slug 用英文小寫 kebab-case。
|
||||
- description 為一句話摘要;tags 為 2 至 6 個英文小寫 kebab-case 標籤。
|
||||
- summary_markdown:文件重點摘要(目的、關鍵事實、結論、教訓)。不要逐步驟複製測試案例內容——單檔可 grep 的細節不進 wiki。
|
||||
- knowledge_items:值得建頁或更新的項目。entity=系統/模組/API/法規條目/專案;concept=跨來源的缺陷模式/測試策略/法遵準則。沒有就給空陣列。
|
||||
- facts_markdown:本文件中關於該項目的事實要點(markdown bullet),供合併至該項目頁面。
|
||||
|
||||
JSON 格式:
|
||||
{{"title": "...", "description": "...", "slug": "...", "tags": ["..."],
|
||||
"summary_markdown": "...",
|
||||
"knowledge_items": [{{"type": "entity 或 concept", "slug": "...", "title": "...",
|
||||
"description": "...", "tags": ["..."], "facts_markdown": "..."}}]}}
|
||||
|
||||
檔名:{filename}
|
||||
文件內容:
|
||||
<<<
|
||||
{content}
|
||||
>>>"""
|
||||
|
||||
CHUNK_PROMPT = """以下是一份長文件的第 {i}/{n} 段。請以繁體中文摘要此段重點(markdown bullet),只輸出 JSON:{{"summary_markdown": "..."}}
|
||||
|
||||
<<<
|
||||
{content}
|
||||
>>>"""
|
||||
|
||||
MERGE_PROMPT = """你是知識庫維護引擎。以下是既有 wiki 頁面內文,以及來自新來源的事實要點。
|
||||
請把新事實整合進頁面(就地編輯:合併、去重;若與既有內容矛盾,兩種說法並陳並標註「⚠ 待查證」)。
|
||||
只輸出 JSON:{{"updated_markdown": "...", "updated_description": "..."}}
|
||||
規則:繁體中文;保留原有結構與仍然有效的內容。
|
||||
|
||||
頁面標題:{title}
|
||||
既有內文:
|
||||
<<<
|
||||
{body}
|
||||
>>>
|
||||
新事實(來源 {source_ref}):
|
||||
<<<
|
||||
{facts}
|
||||
>>>"""
|
||||
|
||||
|
||||
def _s(o, k):
|
||||
return isinstance(o.get(k), str) and o[k].strip()
|
||||
|
||||
|
||||
def _tags_ok(o):
|
||||
return isinstance(o.get("tags"), list) and o["tags"] and all(isinstance(t, str) for t in o["tags"])
|
||||
|
||||
|
||||
def validate_summary(obj):
|
||||
for k in ("title", "description", "slug", "summary_markdown"):
|
||||
if not _s(obj, k):
|
||||
return f"欄位 {k} 缺漏或非字串"
|
||||
if not _tags_ok(obj):
|
||||
return "tags 須為非空字串列表"
|
||||
if not isinstance(obj.get("knowledge_items"), list):
|
||||
return "knowledge_items 須為列表"
|
||||
for it in obj["knowledge_items"]:
|
||||
if not isinstance(it, dict) or it.get("type") not in ("entity", "concept"):
|
||||
return "knowledge_item 須為物件且 type 為 entity|concept"
|
||||
for k in ("slug", "title", "description", "facts_markdown"):
|
||||
if not _s(it, k):
|
||||
return f"knowledge_item 欄位 {k} 缺漏或非字串"
|
||||
if not _tags_ok(it):
|
||||
return "knowledge_item.tags 須為非空字串列表"
|
||||
return None
|
||||
|
||||
|
||||
def validate_chunk(obj):
|
||||
return None if _s(obj, "summary_markdown") else "欄位 summary_markdown 缺漏"
|
||||
|
||||
|
||||
def validate_merge(obj):
|
||||
for k in ("updated_markdown", "updated_description"):
|
||||
if not _s(obj, k):
|
||||
return f"欄位 {k} 缺漏或非字串"
|
||||
return None
|
||||
|
||||
|
||||
def split_chunks(text, limit):
|
||||
"""依段落邊界切塊,每塊不超過 limit 字元(單一超長段落則硬切)。"""
|
||||
chunks, cur = [], ""
|
||||
for para in text.split("\n\n"):
|
||||
while len(para) > limit: # 單段超長:硬切
|
||||
chunks.append(para[:limit])
|
||||
para = para[limit:]
|
||||
if len(cur) + len(para) + 2 > limit and cur:
|
||||
chunks.append(cur)
|
||||
cur = para
|
||||
else:
|
||||
cur = f"{cur}\n\n{para}" if cur else para
|
||||
if cur:
|
||||
chunks.append(cur)
|
||||
return chunks
|
||||
|
||||
|
||||
def summarize(cfg, filename, text):
|
||||
limit = cfg["limits"]["ingest_chunk_max_chars"]
|
||||
if len(text) > limit:
|
||||
parts = split_chunks(text, limit)
|
||||
partials = []
|
||||
for i, part in enumerate(parts, 1):
|
||||
# 長文件的分段迴圈是最容易「長時間靜默」的地方(issue #2):逐段回報進度
|
||||
print(f" 分段 {i}/{len(parts)} …", flush=True)
|
||||
obj, _ = kb.llm_json(cfg, "ingest",
|
||||
CHUNK_PROMPT.format(i=i, n=len(parts), content=part),
|
||||
validate_chunk)
|
||||
partials.append(f"### 分段 {i} 摘要\n{obj['summary_markdown']}")
|
||||
text = "(以下為長文件的分段摘要,請據此綜合)\n\n" + "\n\n".join(partials)
|
||||
return kb.llm_json(cfg, "ingest",
|
||||
SUMMARY_PROMPT.format(filename=filename, content=text),
|
||||
validate_summary)
|
||||
|
||||
|
||||
def _unique_path(d, slug, sha8):
|
||||
p = d / f"{slug}.md"
|
||||
return p if not p.exists() else d / f"{slug}-{sha8}.md"
|
||||
|
||||
|
||||
def write_summary_page(root, obj, source_ref, sha8, today):
|
||||
d = root / kb.PAGE_DIRS["summary"]
|
||||
p = _unique_path(d, kb.safe_slug(obj["slug"], f"doc-{sha8}"), sha8)
|
||||
meta = {"type": "summary", "title": obj["title"], "description": obj["description"],
|
||||
"tags": obj["tags"], "timestamp": today, "sources": [source_ref],
|
||||
"status": "draft"}
|
||||
p.write_text(kb.dump_page(meta, obj["summary_markdown"]), encoding="utf-8")
|
||||
return p
|
||||
|
||||
|
||||
def upsert_item(root, cfg, item, source_ref, today, changed):
|
||||
slug = kb.safe_slug(item["slug"], f"item-{source_ref[-8:]}")
|
||||
existing = None
|
||||
for t in ("entity", "concept"): # 既有頁優先(就地編輯),不管本次 LLM 判的型別
|
||||
p = root / kb.PAGE_DIRS[t] / f"{slug}.md"
|
||||
if p.exists():
|
||||
existing = p
|
||||
break
|
||||
if existing:
|
||||
meta, body = kb.parse_page(existing.read_text(encoding="utf-8"))
|
||||
obj, _ = kb.llm_json(cfg, "ingest",
|
||||
MERGE_PROMPT.format(title=meta["title"], body=body,
|
||||
source_ref=source_ref,
|
||||
facts=item["facts_markdown"]),
|
||||
validate_merge)
|
||||
meta["description"] = obj["updated_description"]
|
||||
meta["tags"] = sorted(set(meta["tags"]) | set(item["tags"]))
|
||||
meta["timestamp"] = today
|
||||
meta["status"] = "draft" # 內容變動須重新人審
|
||||
if source_ref not in meta["sources"]:
|
||||
meta["sources"].append(source_ref)
|
||||
existing.write_text(kb.dump_page(meta, obj["updated_markdown"]), encoding="utf-8")
|
||||
msg = f"更新 {existing.relative_to(root).as_posix()}"
|
||||
else:
|
||||
p = root / kb.PAGE_DIRS[item["type"]] / f"{slug}.md"
|
||||
meta = {"type": item["type"], "title": item["title"],
|
||||
"description": item["description"], "tags": item["tags"],
|
||||
"timestamp": today, "sources": [source_ref], "status": "draft"}
|
||||
p.write_text(kb.dump_page(meta, item["facts_markdown"]), encoding="utf-8")
|
||||
msg = f"新增 {p.relative_to(root).as_posix()}"
|
||||
changed.append(msg)
|
||||
print(f" ↳ {msg}", flush=True)
|
||||
|
||||
|
||||
# ---------- git ----------
|
||||
|
||||
def run_git(root, *args, capture=False):
|
||||
return subprocess.run(["git", *args], cwd=root, check=True, text=True,
|
||||
encoding="utf-8", capture_output=capture)
|
||||
|
||||
|
||||
def git_preflight(root):
|
||||
branch = run_git(root, "branch", "--show-current", capture=True).stdout.strip()
|
||||
if branch != "main":
|
||||
raise SystemExit(f"須在 main 分支執行(目前:{branch})")
|
||||
# git 預設 core.quotepath=true,含非 ASCII(中文檔名)或空格的路徑會被引號包住並轉義,
|
||||
# 故比對前先剝除開頭引號,否則 convert 產出的中文檔名會被誤判為 raw/ 以外的變更。
|
||||
dirty = [l for l in run_git(root, "status", "--porcelain", capture=True)
|
||||
.stdout.splitlines() if l.strip() and not l[3:].lstrip('"').startswith("raw/")]
|
||||
if dirty:
|
||||
raise SystemExit("工作區有 raw/ 以外的未提交變更,請先處理:\n" + "\n".join(dirty))
|
||||
|
||||
|
||||
def referenced_sha8s(root):
|
||||
refs = set()
|
||||
for p in kb.iter_pages(root):
|
||||
meta, _ = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||
for s in meta.get("sources", []):
|
||||
refs.add(str(s).rsplit("#", 1)[-1])
|
||||
return refs
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("paths", nargs="*", help="raw/converted 下的 Markdown 路徑")
|
||||
ap.add_argument("--all-pending", action="store_true")
|
||||
ap.add_argument("--force", action="store_true", help="已攝入過(hash 相同)也重跑")
|
||||
ap.add_argument("--dry-run", action="store_true", help="只列計畫,不呼叫 LLM、不寫檔")
|
||||
ap.add_argument("--root", default=str(kb.ROOT))
|
||||
a = ap.parse_args(argv)
|
||||
root = pathlib.Path(a.root).resolve()
|
||||
cfg = kb.load_config(root)
|
||||
manifest = json.loads((root / "raw" / "manifest.json").read_text(encoding="utf-8"))
|
||||
conv = {k: v for k, v in manifest["files"].items() if v["kind"] == "converted"}
|
||||
|
||||
if a.paths:
|
||||
targets = []
|
||||
for p in a.paths:
|
||||
rel = pathlib.Path(p)
|
||||
rel = (rel.relative_to(root) if rel.is_absolute() else rel).as_posix()
|
||||
if rel not in conv:
|
||||
raise SystemExit(f"{rel} 不在 manifest 的 converted 條目中,請先跑 convert.py")
|
||||
targets.append(rel)
|
||||
elif a.all_pending:
|
||||
done = referenced_sha8s(root)
|
||||
targets = [k for k, v in conv.items() if v["sha256"][:8] not in done]
|
||||
else:
|
||||
ap.error("請指定來源路徑或 --all-pending")
|
||||
if not a.force:
|
||||
done = referenced_sha8s(root)
|
||||
skipped = [t for t in targets if conv[t]["sha256"][:8] in done]
|
||||
targets = [t for t in targets if conv[t]["sha256"][:8] not in done]
|
||||
for t in skipped:
|
||||
print(f"skip: {t} 已有 summary 引用(--force 可重跑)", flush=True)
|
||||
if not targets:
|
||||
print("沒有待攝入的來源。")
|
||||
return
|
||||
|
||||
today = datetime.date.today().isoformat()
|
||||
stamp = datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
|
||||
branch = f"ingest/{stamp}-{kb.safe_slug(pathlib.Path(targets[0]).stem, 'batch')[:30]}"
|
||||
|
||||
if a.dry_run:
|
||||
limit = cfg["limits"]["ingest_chunk_max_chars"]
|
||||
print(f"[dry-run] 分支:{branch},模型:{cfg['tasks']['ingest']['model']}")
|
||||
for t in targets:
|
||||
n = len((root / t).read_text(encoding="utf-8"))
|
||||
print(f"[dry-run] {t}: {n} chars,{max(1, -(-n // limit))} 段")
|
||||
return
|
||||
|
||||
git_preflight(root)
|
||||
run_git(root, "checkout", "-q", "-b", branch)
|
||||
ok, failed, changed = [], [], []
|
||||
try:
|
||||
for i, t in enumerate(targets, 1):
|
||||
try:
|
||||
print(f"[{i}/{len(targets)}] 攝入 {t} …", flush=True)
|
||||
text = (root / t).read_text(encoding="utf-8")
|
||||
sha8 = conv[t]["sha256"][:8]
|
||||
source_ref = f"{t}#{sha8}"
|
||||
obj, model = summarize(cfg, pathlib.Path(conv[t]["original_path"]).name, text)
|
||||
p = write_summary_page(root, obj, source_ref, sha8, today)
|
||||
msg = f"新增 {p.relative_to(root).as_posix()}"
|
||||
changed.append(msg)
|
||||
print(f" ↳ {msg}", flush=True)
|
||||
for item in obj["knowledge_items"]:
|
||||
upsert_item(root, cfg, item, source_ref, today, changed)
|
||||
ok.append((t, model))
|
||||
print(f" ✓ {t} 完成(model={model})", flush=True)
|
||||
except Exception as e:
|
||||
failed.append((t, e))
|
||||
print(f" ✗ {t}: {e}", file=sys.stderr, flush=True)
|
||||
if not ok:
|
||||
run_git(root, "checkout", "-q", "main")
|
||||
run_git(root, "branch", "-q", "-D", branch)
|
||||
raise SystemExit("全部來源攝入失敗,已還原至 main。")
|
||||
kb.rebuild_index(root)
|
||||
kb.append_log(root, "ingest", f"攝入 {len(ok)} 份來源(branch {branch})",
|
||||
changed + [f"失敗:{t}({e})" for t, e in failed])
|
||||
run_git(root, "add", "wiki", "index.md", "log.md", "raw")
|
||||
run_git(root, "commit", "-q", "-m",
|
||||
f"ingest: {len(ok)} 份來源\n\n" + "\n".join(f"- {c}" for c in changed))
|
||||
try:
|
||||
run_git(root, "push", "-q", "-u", "origin", branch)
|
||||
remote = run_git(root, "remote", "get-url", "origin",
|
||||
capture=True).stdout.strip().removesuffix(".git")
|
||||
print(f"PR 建立網址:{remote}/compare/main...{branch}")
|
||||
except subprocess.CalledProcessError:
|
||||
print(f"push 失敗(無 remote 或離線)。分支 {branch} 保留於本地,請手動 push 後開 PR。")
|
||||
run_git(root, "checkout", "-q", "main")
|
||||
print(f"完成:{len(ok)} 成功、{len(failed)} 失敗。變更在分支 {branch},經 PR 審核後合併。")
|
||||
except SystemExit:
|
||||
raise
|
||||
except KeyboardInterrupt:
|
||||
# KeyboardInterrupt 不是 Exception 子類,會略過下方善後——故獨立攔截。
|
||||
# 中斷時尚未 commit(commit 在迴圈之後),main 必然未受影響;給一條可照做的回復路徑。
|
||||
print(f"\n已中斷(Ctrl-C)。main 未受影響——本次半成品都在分支 {branch}、尚未 commit。\n"
|
||||
f"回到乾淨的 main:\n"
|
||||
f" git checkout main && git stash -u && git branch -D {branch}\n"
|
||||
f"(git stash -u 收起分支上未提交的半成品含未追蹤頁面;確定不要再 git stash drop)\n"
|
||||
f"續跑 python tools/ingest.py --all-pending 會自動從尚未攝入的來源接續。",
|
||||
file=sys.stderr, flush=True)
|
||||
raise SystemExit(130)
|
||||
except Exception:
|
||||
print(f"攝入中斷。目前在分支 {branch},工作區可能有未提交變更,"
|
||||
"請人工檢查(不自動清除以免遺失資料)。", file=sys.stderr)
|
||||
raise
|
||||
if failed:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
192
tools/kb.py
Normal file
192
tools/kb.py
Normal file
@@ -0,0 +1,192 @@
|
||||
"""共用模組:config 載入驗證、本地 Ollama 呼叫(重試 + fallback)、
|
||||
wiki 頁 frontmatter 解析/寫出、index.md 重建、log.md 追加。
|
||||
|
||||
供 ingest.py / lint.py / search.py / mcp/server.py 使用。
|
||||
"""
|
||||
import datetime
|
||||
import json
|
||||
import pathlib
|
||||
import os
|
||||
import re
|
||||
import urllib.request
|
||||
|
||||
import yaml
|
||||
|
||||
ROOT = pathlib.Path(__file__).resolve().parents[1]
|
||||
PAGE_DIRS = {"summary": "wiki/summaries", "entity": "wiki/entities", "concept": "wiki/concepts"}
|
||||
PAGE_TYPES = set(PAGE_DIRS)
|
||||
STATUSES = {"draft", "reviewed", "stale"}
|
||||
REQUIRED_FM = ("type", "title", "description", "tags", "timestamp", "sources", "status")
|
||||
|
||||
|
||||
# ---------- config ----------
|
||||
|
||||
def load_config(root=ROOT):
|
||||
"""讀取 config/models.yaml。缺欄位報錯,不使用隱含預設值(AGENTS.md §1.2)。"""
|
||||
path = root / "config" / "models.yaml"
|
||||
cfg = yaml.safe_load(path.read_text(encoding="utf-8"))
|
||||
try:
|
||||
cfg["ollama"]["base_url"]
|
||||
cfg["ollama"]["timeout_seconds"]
|
||||
for task in ("ingest", "lint"):
|
||||
for key in ("model", "temperature", "max_retries"):
|
||||
cfg["tasks"][task][key]
|
||||
cfg["tasks"]["fallback"]["model"]
|
||||
cfg["limits"]["mcp_response_max_tokens"]
|
||||
cfg["limits"]["ingest_chunk_max_chars"]
|
||||
except (KeyError, TypeError) as e:
|
||||
raise SystemExit(f"config/models.yaml 缺欄位:{e}")
|
||||
for task, spec in cfg["tasks"].items():
|
||||
if spec["model"] == "CHANGE_ME":
|
||||
raise SystemExit(f"config/models.yaml:tasks.{task}.model 尚未設定")
|
||||
if spec["model"].endswith(":cloud"):
|
||||
raise SystemExit(f"tasks.{task}.model 為 :cloud 模型,違反資料落地(AGENTS.md §1.1)")
|
||||
# 環境變數是唯一允許的覆寫來源(AGENTS.md §1.2)
|
||||
cfg["ollama"]["base_url"] = os.environ.get("OLLAMA_BASE_URL", cfg["ollama"]["base_url"])
|
||||
return cfg
|
||||
|
||||
|
||||
# ---------- Ollama ----------
|
||||
|
||||
def ollama_chat(cfg, model, prompt, temperature, json_format=False, images=None):
|
||||
"""單次呼叫本地 Ollama /api/chat(stdlib urllib,不需額外依賴)。
|
||||
|
||||
images:base64 影像字串列表(供 vision 模型判讀);附在 user 訊息上。
|
||||
"""
|
||||
msg = {"role": "user", "content": prompt}
|
||||
if images:
|
||||
msg["images"] = list(images)
|
||||
body = {
|
||||
"model": model,
|
||||
"messages": [msg],
|
||||
"stream": False,
|
||||
"options": {"temperature": temperature},
|
||||
}
|
||||
if json_format:
|
||||
body["format"] = "json"
|
||||
req = urllib.request.Request(
|
||||
cfg["ollama"]["base_url"].rstrip("/") + "/api/chat",
|
||||
data=json.dumps(body).encode("utf-8"),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=cfg["ollama"]["timeout_seconds"]) as r:
|
||||
return json.loads(r.read())["message"]["content"]
|
||||
|
||||
|
||||
def llm_json(cfg, task, prompt, validate):
|
||||
"""要求 JSON 輸出的呼叫:主模型重試 max_retries 次,仍失敗切 fallback 模型。
|
||||
|
||||
本地模型輸出不穩定是預期情況(AGENTS.md 前提),故每次失敗把錯誤附回
|
||||
prompt 再試。validate(obj) 回傳錯誤訊息字串或 None。
|
||||
回傳 (obj, 實際使用的 model)。全部失敗拋 RuntimeError。
|
||||
"""
|
||||
spec = cfg["tasks"][task]
|
||||
attempts = []
|
||||
for model in (spec["model"], cfg["tasks"]["fallback"]["model"]):
|
||||
hint = ""
|
||||
for _ in range(spec["max_retries"]):
|
||||
try:
|
||||
raw = ollama_chat(cfg, model, prompt + hint, spec["temperature"],
|
||||
json_format=True)
|
||||
obj = json.loads(raw)
|
||||
err = validate(obj)
|
||||
if err is None:
|
||||
return obj, model
|
||||
except Exception as e:
|
||||
err = f"{type(e).__name__}: {e}"
|
||||
attempts.append(f"{model}: {err}")
|
||||
hint = f"\n\n注意:上次輸出無效({err})。請只輸出符合規格的 JSON,不得有其他文字。"
|
||||
raise RuntimeError("LLM 輸出驗證失敗(含 fallback):" + " | ".join(attempts[-4:]))
|
||||
|
||||
|
||||
# ---------- wiki 頁 ----------
|
||||
|
||||
def parse_page(text):
|
||||
"""回傳 (frontmatter dict, body)。格式不符拋 ValueError。"""
|
||||
m = re.match(r"^---\r?\n(.*?)\r?\n---\r?\n(.*)$", text, re.S)
|
||||
if not m:
|
||||
raise ValueError("缺少 YAML frontmatter")
|
||||
meta = yaml.safe_load(m.group(1))
|
||||
if not isinstance(meta, dict):
|
||||
raise ValueError("frontmatter 不是 mapping")
|
||||
return meta, m.group(2).strip()
|
||||
|
||||
|
||||
def dump_page(meta, body):
|
||||
fm = yaml.safe_dump(meta, allow_unicode=True, sort_keys=False)
|
||||
return f"---\n{fm}---\n\n{body.strip()}\n"
|
||||
|
||||
|
||||
def validate_meta(meta):
|
||||
"""依 AGENTS.md §3.2 檢查 frontmatter,回傳錯誤訊息列表。"""
|
||||
errs = [f"缺欄位 {k}" for k in REQUIRED_FM if k not in meta]
|
||||
if errs:
|
||||
return errs
|
||||
if meta["type"] not in PAGE_TYPES:
|
||||
errs.append(f"type 不合法:{meta['type']}")
|
||||
if meta["status"] not in STATUSES:
|
||||
errs.append(f"status 不合法:{meta['status']}")
|
||||
if not isinstance(meta["tags"], list) or not meta["tags"]:
|
||||
errs.append("tags 須為非空列表")
|
||||
if not isinstance(meta["sources"], list) or not meta["sources"]:
|
||||
errs.append("sources 須為非空列表")
|
||||
for s in meta.get("sources") or []:
|
||||
if not re.match(r"^raw/(converted|originals)/.+#[0-9a-f]{8}$", str(s)):
|
||||
errs.append(f"source_ref 格式不符:{s}")
|
||||
if not re.match(r"^\d{4}-\d{2}-\d{2}$", str(meta["timestamp"])):
|
||||
errs.append(f"timestamp 須為 YYYY-MM-DD:{meta['timestamp']}")
|
||||
return errs
|
||||
|
||||
|
||||
def safe_slug(s, fallback):
|
||||
s = re.sub(r"[^a-z0-9]+", "-", str(s).lower()).strip("-")[:60]
|
||||
return s or fallback
|
||||
|
||||
|
||||
def iter_pages(root=ROOT):
|
||||
for sub in PAGE_DIRS.values():
|
||||
d = root / sub
|
||||
if d.is_dir():
|
||||
yield from sorted(d.glob("*.md"))
|
||||
|
||||
|
||||
# ---------- index.md / log.md ----------
|
||||
|
||||
def rebuild_index(root=ROOT):
|
||||
"""由全部 wiki 頁 frontmatter 決定性重建 index.md(AGENTS.md §10)。"""
|
||||
groups = {"summary": [], "entity": [], "concept": []}
|
||||
for p in iter_pages(root):
|
||||
meta, _ = parse_page(p.read_text(encoding="utf-8"))
|
||||
rel = p.relative_to(root).as_posix()
|
||||
tags = " ".join(f"#{t}" for t in meta["tags"])
|
||||
groups[meta["type"]].append(
|
||||
(str(meta["title"]), f"- [{meta['title']}]({rel}) — {meta['description']} | {tags}"))
|
||||
def section(key):
|
||||
lines = [line for _, line in sorted(groups[key])]
|
||||
return "\n".join(lines) if lines else "(尚無頁面)"
|
||||
text = f"""# 知識庫索引
|
||||
|
||||
<!-- 目錄式導航(AGENTS.md §10):每頁一行,由 ingest 管線自動重建,格式:
|
||||
- [標題](路徑) — 一句摘要 | #tag1 #tag2
|
||||
查詢工作流一律先讀本檔定位候選頁(最多 10 頁)。 -->
|
||||
|
||||
## 摘要頁(summaries)
|
||||
|
||||
{section('summary')}
|
||||
|
||||
## 實體頁(entities)
|
||||
|
||||
{section('entity')}
|
||||
|
||||
## 概念頁(concepts)
|
||||
|
||||
{section('concept')}
|
||||
"""
|
||||
(root / "index.md").write_text(text, encoding="utf-8")
|
||||
|
||||
|
||||
def append_log(root, op, title, lines):
|
||||
entry = f"\n## [{datetime.date.today().isoformat()}] {op} | {title}\n\n"
|
||||
entry += "".join(f"- {l}\n" for l in lines)
|
||||
with open(root / "log.md", "a", encoding="utf-8") as f:
|
||||
f.write(entry)
|
||||
211
tools/lint.py
Normal file
211
tools/lint.py
Normal file
@@ -0,0 +1,211 @@
|
||||
"""wiki 健檢(AGENTS.md §7):schema 完整性、過期主張、覆蓋缺口、孤兒頁、
|
||||
重複/矛盾頁偵測(LLM)。產出 markdown 報告;絕不修改或刪除任何檔案,
|
||||
所有項目一律標記待人工核准。
|
||||
|
||||
用法:python tools/lint.py [--output reports] [--no-llm] [--root DIR]
|
||||
結束碼:0 = 無發現,1 = 有發現(供 CI 判斷)。
|
||||
"""
|
||||
import argparse
|
||||
import datetime
|
||||
import difflib
|
||||
import itertools
|
||||
import json
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
|
||||
# ponytail: 過期門檻寫死 180 天(naive heuristic);若各類頁面需要不同節奏,
|
||||
# 升級路徑:依 type/tags 分級門檻。
|
||||
STALE_DAYS = 180
|
||||
# ponytail: LLM 逐對比較上限 15 對,避免 O(n²) 呼叫爆量;升級路徑:以 embedding
|
||||
# 預篩(Phase 6 之後評估 EmbeddingGemma / snowflake-arctic-embed)。
|
||||
MAX_LLM_PAIRS = 15
|
||||
SIM_THRESHOLD = 0.6
|
||||
|
||||
PAIR_PROMPT = """判斷以下兩個 QA 知識庫 wiki 頁是否「重複」(描述同一事物,應合併)或「矛盾」(對同一事實給出不相容的主張)。只輸出 JSON:
|
||||
{{"relation": "duplicate" 或 "contradiction" 或 "none", "explanation": "一句話理由(繁體中文)"}}
|
||||
|
||||
頁 A:{a_title} — {a_desc}
|
||||
內文節錄:
|
||||
<<<
|
||||
{a_body}
|
||||
>>>
|
||||
|
||||
頁 B:{b_title} — {b_desc}
|
||||
內文節錄:
|
||||
<<<
|
||||
{b_body}
|
||||
>>>"""
|
||||
|
||||
|
||||
def validate_pair(obj):
|
||||
if obj.get("relation") not in ("duplicate", "contradiction", "none"):
|
||||
return "relation 須為 duplicate|contradiction|none"
|
||||
if not isinstance(obj.get("explanation"), str):
|
||||
return "explanation 須為字串"
|
||||
return None
|
||||
|
||||
|
||||
def load_pages(root, findings):
|
||||
pages = []
|
||||
for p in kb.iter_pages(root):
|
||||
rel = p.relative_to(root).as_posix()
|
||||
try:
|
||||
meta, body = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||
pages.append((rel, meta, body))
|
||||
except Exception as e:
|
||||
findings.append(("schema", rel, f"無法解析:{e}"))
|
||||
return pages
|
||||
|
||||
|
||||
def check_schema(pages, manifest, findings):
|
||||
for rel, meta, _ in pages:
|
||||
for err in kb.validate_meta(meta):
|
||||
findings.append(("schema", rel, err))
|
||||
for s in meta.get("sources") or []:
|
||||
path, _, sha8 = str(s).rpartition("#")
|
||||
entry = manifest["files"].get(path)
|
||||
if entry is None:
|
||||
findings.append(("schema", rel, f"source_ref 不在 manifest:{s}"))
|
||||
elif not entry["sha256"].startswith(sha8):
|
||||
findings.append(("schema", rel, f"source_ref hash 與 manifest 不符:{s}"))
|
||||
|
||||
|
||||
def check_stale(pages, findings):
|
||||
today = datetime.date.today()
|
||||
for rel, meta, _ in pages:
|
||||
try:
|
||||
ts = datetime.date.fromisoformat(str(meta.get("timestamp")))
|
||||
except ValueError:
|
||||
continue # schema 檢查已報
|
||||
age = (today - ts).days
|
||||
if meta.get("status") == "stale":
|
||||
findings.append(("stale", rel, "已標記 stale,待人工複審或更新"))
|
||||
elif age > STALE_DAYS:
|
||||
findings.append(("stale", rel, f"最後更新 {age} 天前,建議人工複審後標記 stale"))
|
||||
|
||||
|
||||
def check_coverage(pages, manifest, findings):
|
||||
referenced = {str(s).rsplit("#", 1)[-1]
|
||||
for _, meta, _ in pages for s in meta.get("sources") or []}
|
||||
for path, e in manifest["files"].items():
|
||||
if e["kind"] == "converted" and e["sha256"][:8] not in referenced:
|
||||
findings.append(("coverage", path, "已轉換但沒有任何 wiki 頁引用(攝入缺口)"))
|
||||
if e["kind"] == "original" and e.get("status") == "needs_ocr":
|
||||
findings.append(("coverage", path, "needs_ocr:掃描件待人工決定 OCR 方案"))
|
||||
|
||||
|
||||
def check_orphans(root, pages, findings):
|
||||
# ponytail: 以「檔名出現在任何連結目標中」判斷被連結,不解析相對路徑;
|
||||
# 升級路徑:正規化解析所有 markdown 連結。
|
||||
link_targets = ""
|
||||
idx = root / "index.md"
|
||||
if idx.is_file():
|
||||
link_targets += idx.read_text(encoding="utf-8")
|
||||
for _, _, body in pages:
|
||||
link_targets += body
|
||||
linked = set(re.findall(r"\(([^)]+\.md)\)", link_targets))
|
||||
linked_names = {pathlib.PurePosixPath(t).name for t in linked}
|
||||
for rel, _, _ in pages:
|
||||
if pathlib.PurePosixPath(rel).name not in linked_names:
|
||||
findings.append(("orphan", rel, "不被 index.md 或任何其他頁連結"))
|
||||
|
||||
|
||||
def candidate_pairs(pages):
|
||||
cands = {}
|
||||
for (ra, ma, ba), (rb, mb, bb) in itertools.combinations(pages, 2):
|
||||
sim = difflib.SequenceMatcher(
|
||||
None, f"{ma['title']} {ma['description']}",
|
||||
f"{mb['title']} {mb['description']}").ratio()
|
||||
shared_tags = len(set(ma.get("tags") or []) & set(mb.get("tags") or []))
|
||||
if sim >= SIM_THRESHOLD or shared_tags >= 2:
|
||||
cands[(ra, rb)] = (sim, (ma, ba), (mb, bb))
|
||||
ranked = sorted(cands.items(), key=lambda kv: -kv[1][0])
|
||||
return ranked[:MAX_LLM_PAIRS]
|
||||
|
||||
|
||||
def check_pairs_llm(cfg, pages, findings):
|
||||
for (ra, rb), (sim, (ma, ba), (mb, bb)) in candidate_pairs(pages):
|
||||
try:
|
||||
obj, _ = kb.llm_json(cfg, "lint", PAIR_PROMPT.format(
|
||||
a_title=ma["title"], a_desc=ma["description"], a_body=ba[:1500],
|
||||
b_title=mb["title"], b_desc=mb["description"], b_body=bb[:1500]),
|
||||
validate_pair)
|
||||
except RuntimeError as e:
|
||||
findings.append(("llm-pair", f"{ra} ↔ {rb}", f"LLM 比對失敗:{e}"))
|
||||
continue
|
||||
if obj["relation"] == "duplicate":
|
||||
findings.append(("duplicate", f"{ra} ↔ {rb}", f"疑似重複:{obj['explanation']}"))
|
||||
elif obj["relation"] == "contradiction":
|
||||
findings.append(("contradiction", f"{ra} ↔ {rb}", f"疑似矛盾:{obj['explanation']}"))
|
||||
|
||||
|
||||
SECTION_TITLES = {
|
||||
"schema": "Schema 完整性", "stale": "過期主張(staleness)",
|
||||
"coverage": "覆蓋缺口", "orphan": "孤兒頁",
|
||||
"duplicate": "重複頁(LLM 判定)", "contradiction": "矛盾頁(LLM 判定)",
|
||||
"llm-pair": "LLM 比對失敗(需重跑)",
|
||||
}
|
||||
|
||||
|
||||
def write_report(out_dir, findings, n_pages, llm_note):
|
||||
now = datetime.datetime.now()
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / f"lint-{now.strftime('%Y%m%d-%H%M%S')}.md"
|
||||
counts = {}
|
||||
for check, _, _ in findings:
|
||||
counts[check] = counts.get(check, 0) + 1
|
||||
lines = [f"# Lint 報告 — {now.strftime('%Y-%m-%d %H:%M')}", "",
|
||||
f"- 檢查頁面數:{n_pages}",
|
||||
f"- 發現項目:{len(findings)}(" + "、".join(
|
||||
f"{SECTION_TITLES[k]} {v}" for k, v in counts.items()) + ")"
|
||||
if findings else "- 發現項目:0",
|
||||
f"- LLM 比對:{llm_note}", "",
|
||||
"> lint 絕不修改或刪除任何檔案;以下所有項目皆**待人工核准**後才處置。", ""]
|
||||
for check in SECTION_TITLES:
|
||||
items = [(w, msg) for c, w, msg in findings if c == check]
|
||||
if not items:
|
||||
continue
|
||||
lines += [f"## {SECTION_TITLES[check]}", ""]
|
||||
lines += [f"- [ ] `{w}` — {msg}" for w, msg in items]
|
||||
lines.append("")
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("--output", default="reports")
|
||||
ap.add_argument("--no-llm", action="store_true", help="跳過重複/矛盾的 LLM 比對")
|
||||
ap.add_argument("--root", default=str(kb.ROOT))
|
||||
a = ap.parse_args(argv)
|
||||
root = pathlib.Path(a.root).resolve()
|
||||
cfg = kb.load_config(root)
|
||||
manifest = json.loads((root / "raw" / "manifest.json").read_text(encoding="utf-8"))
|
||||
|
||||
findings = []
|
||||
pages = load_pages(root, findings)
|
||||
check_schema(pages, manifest, findings)
|
||||
check_stale(pages, findings)
|
||||
check_coverage(pages, manifest, findings)
|
||||
check_orphans(root, pages, findings)
|
||||
if a.no_llm:
|
||||
llm_note = "已略過(--no-llm)"
|
||||
else:
|
||||
llm_note = f"model={cfg['tasks']['lint']['model']},上限 {MAX_LLM_PAIRS} 對"
|
||||
check_pairs_llm(cfg, pages, findings)
|
||||
|
||||
out = pathlib.Path(a.output)
|
||||
report = write_report(out if out.is_absolute() else root / out,
|
||||
findings, len(pages), llm_note)
|
||||
print(f"報告:{report}")
|
||||
print(f"發現 {len(findings)} 項" if findings else "無發現")
|
||||
if findings:
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
84
tools/search.py
Normal file
84
tools/search.py
Normal file
@@ -0,0 +1,84 @@
|
||||
"""wiki BM25 全文檢索(第一版不用向量庫,AGENTS.md §0)。
|
||||
|
||||
用法:python tools/search.py "查詢詞" [-k 10] [--root DIR]
|
||||
輸出:JSON 陣列(path、title、description、tags、score),供 agent 與 MCP 使用。
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
from rank_bm25 import BM25Okapi
|
||||
|
||||
|
||||
def tokenize(text):
|
||||
"""英數字詞 + CJK bigram。
|
||||
|
||||
ponytail: bigram 斷詞無語意理解(同義詞不匹配),天花板是字面重疊;
|
||||
升級路徑:Phase 6 後評估 EmbeddingGemma / snowflake-arctic-embed 向量檢索。
|
||||
"""
|
||||
text = text.lower()
|
||||
tokens = re.findall(r"[a-z0-9]+", text)
|
||||
for run in re.findall(r"[一-鿿]+", text):
|
||||
tokens += [run] if len(run) == 1 else [run[i:i + 2] for i in range(len(run) - 1)]
|
||||
return tokens
|
||||
|
||||
|
||||
class _BM25(BM25Okapi):
|
||||
"""改用 Lucene 式 idf:log(1 + (N-df+0.5)/(df+0.5)),恆為正。
|
||||
|
||||
BM25Okapi 原式在 df >= N/2 時 idf <= 0(rank_bm25 對負 idf 的替代值
|
||||
epsilon × average_idf 在 average_idf 為負時同樣是負的),與 search() 的
|
||||
`s > 0` 過濾相乘,會讓「每頁都提到的核心詞」查詢全數落空——知識庫愈小、
|
||||
詞愈核心愈嚴重。恆正 idf 讓常見詞只是權重低,而不是被整批濾掉。
|
||||
"""
|
||||
|
||||
def _calc_idf(self, nd):
|
||||
for word, freq in nd.items():
|
||||
self.idf[word] = math.log(1 + (self.corpus_size - freq + 0.5) / (freq + 0.5))
|
||||
|
||||
|
||||
def build_corpus(root):
|
||||
docs = []
|
||||
for p in kb.iter_pages(root):
|
||||
try:
|
||||
meta, body = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||
except ValueError:
|
||||
continue # 壞頁由 lint 報,檢索直接略過
|
||||
text = (f"{meta['title']} " * 3 + f"{meta['description']} " * 2
|
||||
+ " ".join(meta.get("tags") or []) + " " + body)
|
||||
docs.append({"path": p.relative_to(root).as_posix(), "title": str(meta["title"]),
|
||||
"description": str(meta["description"]),
|
||||
"tags": [str(t) for t in meta.get("tags") or []],
|
||||
"tokens": tokenize(text)})
|
||||
return docs
|
||||
|
||||
|
||||
def search(root, query, k=10):
|
||||
docs = build_corpus(pathlib.Path(root))
|
||||
if not docs:
|
||||
return []
|
||||
bm = _BM25([d["tokens"] for d in docs])
|
||||
scores = bm.get_scores(tokenize(query))
|
||||
ranked = sorted(zip(docs, scores), key=lambda x: -x[1])[:k]
|
||||
return [{"path": d["path"], "title": d["title"], "description": d["description"],
|
||||
"tags": d["tags"], "score": round(float(s), 4)}
|
||||
for d, s in ranked if s > 0]
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
ap = argparse.ArgumentParser(description=__doc__)
|
||||
ap.add_argument("query")
|
||||
ap.add_argument("-k", type=int, default=10, help="回傳筆數上限(查詢工作流上限 10)")
|
||||
ap.add_argument("--root", default=str(kb.ROOT))
|
||||
a = ap.parse_args(argv)
|
||||
hits = search(a.root, a.query, max(1, min(a.k, 10)))
|
||||
print(json.dumps(hits, ensure_ascii=False, indent=2))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
114
tools/selfcheck_diag_refs.py
Normal file
114
tools/selfcheck_diag_refs.py
Normal file
@@ -0,0 +1,114 @@
|
||||
"""Phase 4 自檢:沙盒植入五種 source_ref 岔開型別(hash 不符、路徑正規化、
|
||||
檔名 hash 後綴岔開、檔在磁碟未登記、完全無對應)+一個正常解得開的 ref,
|
||||
驗證 diag_refs 逐一分類正確、報告涵蓋各型別、結束碼 1、且不修改任何檔案。
|
||||
另驗全部解得開時結束碼 0。
|
||||
|
||||
執行:.venv/Scripts/python tools/selfcheck_diag_refs.py
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import redirect_stdout
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
import diag_refs
|
||||
|
||||
SHA_A = "abcd1234" + "0" * 56
|
||||
SHA_B = "beef5678" + "0" * 56
|
||||
# 帳本 key 一律 posix 正斜線+中文/連字號檔名(真實 convert 產出的形狀)
|
||||
K_OK = "raw/converted/www-pluspay-faq-06b02fd9.md"
|
||||
K_STEM = "raw/converted/www-pluspay-terms-11112222.md" # 同幹、不同 hash 後綴
|
||||
|
||||
FILES = {
|
||||
K_OK: {"kind": "converted", "sha256": SHA_A, "original_path": "raw/originals/faq.html",
|
||||
"original_sha256": SHA_A, "converter": "from_web", "converted_at": "2026-07-01T00:00:00"},
|
||||
K_STEM: {"kind": "converted", "sha256": SHA_B, "original_path": "raw/originals/terms.html",
|
||||
"original_sha256": SHA_B, "converter": "from_web", "converted_at": "2026-07-01T00:00:00"},
|
||||
}
|
||||
|
||||
# ref → 預期分類。unregistered 的檔會實際落地;absent 的不落地。
|
||||
CASES = {
|
||||
"resolved": f"{K_OK}#abcd1234",
|
||||
"hash-mismatch": f"{K_OK}#deadbeef",
|
||||
"normalize": r"raw\converted\www-pluspay-faq-06b02fd9.md#abcd1234", # 反斜線
|
||||
"stem": "raw/converted/www-pluspay-terms-99998888.md#cccccccc", # 同幹、異後綴、非 key
|
||||
"unregistered": "raw/converted/www-pluspay-instructions-77776666.md#77777777",
|
||||
"absent": "raw/converted/www-pluspay-ghost-00001111.md#cafebabe",
|
||||
}
|
||||
|
||||
|
||||
def unit():
|
||||
"""classify_ref 純函式逐型別驗證。"""
|
||||
on_disk = {"raw/converted/www-pluspay-instructions-77776666.md"}
|
||||
exists = lambda p: p in on_disk
|
||||
for expect, ref in CASES.items():
|
||||
cat, detail = diag_refs.classify_ref(ref, FILES, exists)
|
||||
assert cat == expect, f"{ref!r} 應分類為 {expect},實得 {cat}({detail})"
|
||||
print("PASS: classify_ref 六型別(含 resolved)分類正確")
|
||||
|
||||
|
||||
def build(root):
|
||||
for sub in ("raw/converted", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||
(root / sub).mkdir(parents=True)
|
||||
(root / "raw/manifest.json").write_text(
|
||||
json.dumps({"version": 1, "files": FILES}), encoding="utf-8")
|
||||
# unregistered 的檔要真的存在於磁碟,才會被判為「檔在、帳本無」而非 absent
|
||||
(root / "raw/converted/www-pluspay-instructions-77776666.md").write_text(
|
||||
"# 未登記\n", encoding="utf-8")
|
||||
today = "2026-07-01"
|
||||
for i, (name, ref) in enumerate(CASES.items()):
|
||||
meta = {"type": "entity", "title": f"頁{name}", "description": "測試頁",
|
||||
"tags": ["t"], "timestamp": today, "sources": [ref], "status": "draft"}
|
||||
(root / f"wiki/entities/p{i}.md").write_text(kb.dump_page(meta, "內文。"), encoding="utf-8")
|
||||
|
||||
|
||||
def run(argv):
|
||||
buf = io.StringIO()
|
||||
code = 0
|
||||
try:
|
||||
with redirect_stdout(buf):
|
||||
diag_refs.main(argv)
|
||||
except SystemExit as e:
|
||||
code = e.code
|
||||
return code, buf.getvalue()
|
||||
|
||||
|
||||
def main():
|
||||
unit()
|
||||
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-diag-"))
|
||||
try:
|
||||
root = tmp / "repo"
|
||||
build(root)
|
||||
before = {p: p.read_bytes() for p in root.rglob("*")
|
||||
if p.is_file()}
|
||||
|
||||
code, out = run(["--root", str(root)])
|
||||
assert code == 1, f"有岔開應以結束碼 1,實得 {code}\n{out}"
|
||||
for cat in ("hash-mismatch", "normalize", "stem", "unregistered", "absent"):
|
||||
assert diag_refs.CAT_TITLE[cat].split("(")[0] in out, f"報告缺型別 {cat}\n{out}"
|
||||
assert "對不上:5" in out, out
|
||||
after = {p: p.read_bytes() for p in root.rglob("*") if p.is_file()}
|
||||
assert before == after, "diag_refs 修改了檔案!"
|
||||
print("PASS: 整合——五型別全報、結束碼 1、零寫檔")
|
||||
|
||||
# 全部解得開 → 結束碼 0
|
||||
good = {"type": "entity", "title": "好頁", "description": "d", "tags": ["t"],
|
||||
"timestamp": "2026-07-01", "sources": [f"{K_OK}#abcd1234"], "status": "draft"}
|
||||
for p in (root / "wiki/entities").glob("*.md"):
|
||||
p.unlink()
|
||||
(root / "wiki/entities/ok.md").write_text(kb.dump_page(good, "內文。"), encoding="utf-8")
|
||||
code, out = run(["--root", str(root)])
|
||||
assert code == 0, f"全部解得開應以結束碼 0,實得 {code}\n{out}"
|
||||
assert "全部解得開" in out, out
|
||||
print("PASS: 全部解得開時結束碼 0")
|
||||
print("ALL PASS")
|
||||
finally:
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
203
tools/selfcheck_ingest.py
Normal file
203
tools/selfcheck_ingest.py
Normal file
@@ -0,0 +1,203 @@
|
||||
"""Phase 3 自檢:以假 LLM + 沙盒 git repo 驗證攝入管線的完整 plumbing——
|
||||
分支建立、頁面產出、frontmatter 合規、index/log 更新、冪等跳過、就地編輯合併,
|
||||
以及 kb.llm_json 的重試與 fallback 切換。不需要 Ollama 在線。
|
||||
|
||||
執行:.venv/Scripts/python tools/selfcheck_ingest.py
|
||||
"""
|
||||
import hashlib
|
||||
import io
|
||||
import pathlib
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
import ingest
|
||||
|
||||
CFG = """ollama: {base_url: "http://localhost:11434", timeout_seconds: 5}
|
||||
tasks:
|
||||
ingest: {model: "fake:test", temperature: 0.2, max_retries: 2}
|
||||
lint: {model: "fake:test", temperature: 0.1, max_retries: 2}
|
||||
fallback: {model: "fake:fb"}
|
||||
limits: {mcp_response_max_tokens: 2000, ingest_chunk_max_chars: 200}
|
||||
"""
|
||||
|
||||
|
||||
def fake_llm(cfg, task, prompt, validate):
|
||||
if prompt.startswith("以下是一份長文件的第"):
|
||||
obj = {"summary_markdown": "- 分段重點"}
|
||||
elif '"updated_markdown"' in prompt:
|
||||
obj = {"updated_markdown": "- 壓測逾時\n- 新來源事實",
|
||||
"updated_description": "支付閘道模組(已更新)"}
|
||||
else:
|
||||
obj = {"title": "支付閘道壓測報告", "description": "壓測發現逾時缺陷",
|
||||
"slug": "gateway-load-test", "tags": ["load-test", "gateway"],
|
||||
"summary_markdown": "- 逾時缺陷於壓測重現",
|
||||
"knowledge_items": [{"type": "entity", "slug": "payment-gateway",
|
||||
"title": "支付閘道", "description": "支付閘道模組",
|
||||
"tags": ["gateway"], "facts_markdown": "- 壓測逾時"}]}
|
||||
err = validate(obj) # 假輸出也要通過真 validator,確保契約一致
|
||||
assert err is None, err
|
||||
return obj, "fake:test"
|
||||
|
||||
|
||||
def git(root, *args):
|
||||
return subprocess.run(["git", *args], cwd=root, check=True, text=True,
|
||||
encoding="utf-8", capture_output=True).stdout.strip()
|
||||
|
||||
|
||||
def add_doc(root, name, text):
|
||||
p = root / "raw" / "converted" / name
|
||||
p.write_text(text, encoding="utf-8")
|
||||
sha = hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
import json
|
||||
mp = root / "raw" / "manifest.json"
|
||||
m = json.loads(mp.read_text(encoding="utf-8"))
|
||||
m["files"][f"raw/converted/{name}"] = {
|
||||
"kind": "converted", "sha256": sha, "original_path": f"raw/originals/{name}",
|
||||
"original_sha256": sha, "converter": "from_docx", "converted_at": "2026-07-14T00:00:00"}
|
||||
mp.write_text(json.dumps(m, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return sha
|
||||
|
||||
|
||||
def main():
|
||||
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-ingest-"))
|
||||
real_llm, real_chat = kb.llm_json, kb.ollama_chat
|
||||
try:
|
||||
root = tmp / "repo"
|
||||
for sub in ("config", "raw/originals", "raw/converted",
|
||||
"wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||
(root / sub).mkdir(parents=True)
|
||||
(root / "config" / "models.yaml").write_text(CFG, encoding="utf-8")
|
||||
(root / "raw" / "manifest.json").write_text('{"version": 1, "files": {}}',
|
||||
encoding="utf-8")
|
||||
(root / "index.md").write_text("# 知識庫索引\n", encoding="utf-8")
|
||||
(root / "log.md").write_text("# 操作日誌\n", encoding="utf-8")
|
||||
git(root, "init", "-q", "-b", "main")
|
||||
git(root, "config", "user.name", "selfcheck")
|
||||
git(root, "config", "user.email", "selfcheck@local")
|
||||
text = "支付閘道於壓力測試下出現逾時缺陷,交易峰值時重試機制未生效。\n\n" * 12
|
||||
add_doc(root, "doc1.md", text) # > 200 chars → 走分段路徑
|
||||
git(root, "add", "-A")
|
||||
git(root, "commit", "-q", "-m", "init")
|
||||
|
||||
# 1) 攝入:建分支、產頁、commit、回到 main(並驗進度輸出可見,issue #2)
|
||||
kb.llm_json = fake_llm
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
ingest.main(["raw/converted/doc1.md", "--root", str(root)])
|
||||
out = buf.getvalue()
|
||||
for marker in ("[1/1] 攝入", "分段 1/", "↳ 新增", "✓ ", "完成"):
|
||||
assert marker in out, f"進度輸出缺少 {marker!r}\n{out}"
|
||||
assert git(root, "branch", "--show-current") == "main"
|
||||
branch = git(root, "branch", "--list", "ingest/*").strip("* ").strip()
|
||||
assert branch, "未建立 ingest 分支"
|
||||
assert not git(root, "status", "--porcelain"), "工作區不乾淨"
|
||||
git(root, "checkout", "-q", branch)
|
||||
summary = list((root / "wiki/summaries").glob("*.md"))
|
||||
entity = root / "wiki/entities/payment-gateway.md"
|
||||
assert len(summary) == 1 and entity.is_file()
|
||||
meta, _ = kb.parse_page(entity.read_text(encoding="utf-8"))
|
||||
assert kb.validate_meta(meta) == [], kb.validate_meta(meta)
|
||||
assert "payment-gateway" in (root / "index.md").read_text(encoding="utf-8")
|
||||
assert "] ingest |" in (root / "log.md").read_text(encoding="utf-8")
|
||||
print("PASS: 攝入 → 分支 + summary/entity 頁 + index/log")
|
||||
|
||||
# 2) 模擬 PR 合併後:冪等跳過;新來源觸發既有頁就地編輯
|
||||
git(root, "checkout", "-q", "main")
|
||||
git(root, "merge", "-q", branch)
|
||||
ingest.main(["raw/converted/doc1.md", "--root", str(root)]) # 應 skip
|
||||
assert git(root, "branch", "--show-current") == "main"
|
||||
add_doc(root, "doc2.md", "第二份文件:支付閘道逾時已修復並通過回歸測試。")
|
||||
git(root, "add", "-A")
|
||||
git(root, "commit", "-q", "-m", "doc2")
|
||||
ingest.main(["raw/converted/doc2.md", "--root", str(root)])
|
||||
b2 = [b.strip("* ").strip() for b in
|
||||
git(root, "branch", "--list", "ingest/*").splitlines() if branch not in b]
|
||||
git(root, "checkout", "-q", b2[0])
|
||||
meta, body = kb.parse_page(entity.read_text(encoding="utf-8"))
|
||||
assert len(meta["sources"]) == 2 and "已更新" in meta["description"], meta
|
||||
assert "新來源事實" in body
|
||||
git(root, "checkout", "-q", "main")
|
||||
print("PASS: 冪等跳過 + 既有 entity 頁就地編輯(sources 追加)")
|
||||
|
||||
# 3) preflight:raw/ 下未提交的 convert 產出(含中文檔名)不得被誤判為髒污
|
||||
(root / "raw" / "converted" / "測試報告.md").write_text("中文", encoding="utf-8")
|
||||
ingest.git_preflight(root) # 不應拋錯
|
||||
stray = root / "wiki" / "concepts" / "stray.md"
|
||||
stray.write_text("x", encoding="utf-8")
|
||||
try:
|
||||
ingest.git_preflight(root)
|
||||
raise AssertionError("raw/ 以外的未提交變更應中止")
|
||||
except SystemExit:
|
||||
pass
|
||||
stray.unlink()
|
||||
print("PASS: preflight 放行 raw/(含中文檔名)、擋下 raw/ 以外的未提交變更")
|
||||
|
||||
# 4) kb.llm_json:重試後切 fallback;全失敗拋錯
|
||||
kb.llm_json = real_llm
|
||||
cfg = kb.load_config(root)
|
||||
calls = {"n": 0}
|
||||
def flaky(cfg_, model, prompt, temperature, json_format=False):
|
||||
calls["n"] += 1
|
||||
return "not-json" if calls["n"] < 3 else '{"summary_markdown": "ok"}'
|
||||
kb.ollama_chat = flaky
|
||||
obj, model = kb.llm_json(cfg, "ingest", "p", ingest.validate_chunk)
|
||||
assert obj["summary_markdown"] == "ok" and model == "fake:fb" and calls["n"] == 3
|
||||
kb.ollama_chat = lambda *a, **k: "junk"
|
||||
try:
|
||||
kb.llm_json(cfg, "ingest", "p", ingest.validate_chunk)
|
||||
raise AssertionError("應拋 RuntimeError")
|
||||
except RuntimeError as e:
|
||||
assert "fallback" in str(e)
|
||||
print("PASS: 重試 + fallback 切換 + 全失敗報錯")
|
||||
|
||||
# 5) Ctrl-C 安全(issue #2):中斷時 main 不受影響、以 130 收場,
|
||||
# 且文件宣稱的回復指令真的能回到乾淨 main。第 2 份來源開始摘要時中斷,
|
||||
# 此時第 1 份的頁面已寫入(未追蹤、未 commit)——正是使用者最怕的半成品狀態。
|
||||
add_doc(root, "docA.md", "來源A:待攝入。")
|
||||
add_doc(root, "docB.md", "來源B:待攝入。")
|
||||
git(root, "add", "-A")
|
||||
git(root, "commit", "-q", "-m", "docA/docB")
|
||||
head_before = git(root, "rev-parse", "main") # 攝入不得動到的 main 狀態
|
||||
state = {"n": 0}
|
||||
def boom(cfg, task, prompt, validate):
|
||||
state["n"] += 1
|
||||
if state["n"] >= 2: # 第 2 份來源的 summarize 呼叫
|
||||
raise KeyboardInterrupt
|
||||
return ({"title": "來源A", "description": "d", "slug": "fresh-a",
|
||||
"tags": ["x"], "summary_markdown": "- a", "knowledge_items": []},
|
||||
"fake:test")
|
||||
kb.llm_json = boom
|
||||
err = io.StringIO()
|
||||
try:
|
||||
with redirect_stdout(io.StringIO()), redirect_stderr(err):
|
||||
ingest.main(["raw/converted/docA.md", "raw/converted/docB.md",
|
||||
"--root", str(root)])
|
||||
raise AssertionError("KeyboardInterrupt 應轉為 SystemExit(130)")
|
||||
except SystemExit as e:
|
||||
assert e.code == 130, e.code
|
||||
msg = err.getvalue()
|
||||
assert "git checkout main" in msg and "main 未受影響" in msg, msg
|
||||
assert git(root, "rev-parse", "main") == head_before, "main 被動到了!"
|
||||
cur = git(root, "branch", "--show-current")
|
||||
assert cur.startswith("ingest/"), f"中斷後應停在 ingest 分支,實得 {cur!r}"
|
||||
assert git(root, "status", "--porcelain"), "應有未提交的半成品"
|
||||
# 照文件指令回復,斷言真的回到乾淨 main(驗「結果可用」,非「字串存在」,§13.4)
|
||||
git(root, "checkout", "main")
|
||||
git(root, "stash", "-u")
|
||||
git(root, "branch", "-D", cur)
|
||||
assert git(root, "branch", "--show-current") == "main"
|
||||
assert not git(root, "status", "--porcelain"), "回復後工作區仍不乾淨"
|
||||
print("PASS: Ctrl-C → main 未受影響、130 收場、回復指令實測可回乾淨 main")
|
||||
print("ALL PASS")
|
||||
finally:
|
||||
kb.llm_json, kb.ollama_chat = real_llm, real_chat
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
119
tools/selfcheck_lint.py
Normal file
119
tools/selfcheck_lint.py
Normal file
@@ -0,0 +1,119 @@
|
||||
"""Phase 4 自檢:沙盒植入各類違規(schema 缺欄位、壞 source_ref、過期頁、
|
||||
攝入缺口、needs_ocr、孤兒頁、相似頁對),驗證 lint 全部抓到、產出報告、
|
||||
且不修改任何檔案。LLM 比對以假 LLM 驗證矛盾偵測路徑。
|
||||
|
||||
執行:.venv/Scripts/python tools/selfcheck_lint.py
|
||||
"""
|
||||
import json
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
import lint
|
||||
|
||||
CFG = """ollama: {base_url: "http://localhost:11434", timeout_seconds: 5}
|
||||
tasks:
|
||||
ingest: {model: "fake:test", temperature: 0.2, max_retries: 2}
|
||||
lint: {model: "fake:test", temperature: 0.1, max_retries: 2}
|
||||
fallback: {model: "fake:fb"}
|
||||
limits: {mcp_response_max_tokens: 2000, ingest_chunk_max_chars: 12000}
|
||||
"""
|
||||
SHA_A = "abcd1234" + "0" * 56
|
||||
SHA_B = "beef5678" + "0" * 56
|
||||
SRC_OK = "raw/converted/好來源 報告.md"
|
||||
SRC_UNCOVERED = "raw/converted/未覆蓋 來源.md"
|
||||
|
||||
|
||||
def page(root, rel, meta, body="內文。"):
|
||||
p = root / rel
|
||||
p.write_text(kb.dump_page(meta, body), encoding="utf-8")
|
||||
|
||||
|
||||
def build(root):
|
||||
import datetime
|
||||
today = datetime.date.today().isoformat()
|
||||
for sub in ("config", "raw/converted", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||
(root / sub).mkdir(parents=True)
|
||||
(root / "config/models.yaml").write_text(CFG, encoding="utf-8")
|
||||
# 來源檔名一律「中文 + 空格」——真實 QA 文件就長這樣(§12:測資要照現實建,不是照能過建)
|
||||
(root / "raw/manifest.json").write_text(json.dumps({"version": 1, "files": {
|
||||
SRC_OK: {"kind": "converted", "sha256": SHA_A,
|
||||
"original_path": "raw/originals/好來源 報告.docx",
|
||||
"original_sha256": SHA_A, "converter": "from_docx",
|
||||
"converted_at": "2026-07-01T00:00:00"},
|
||||
SRC_UNCOVERED: {"kind": "converted", "sha256": SHA_B,
|
||||
"original_path": "raw/originals/未覆蓋 來源.docx",
|
||||
"original_sha256": SHA_B, "converter": "from_docx",
|
||||
"converted_at": "2026-07-01T00:00:00"},
|
||||
"raw/originals/掃描件 無文字層.pdf": {"kind": "original", "sha256": SHA_B,
|
||||
"media_type": "pdf",
|
||||
"added_at": "2026-07-01T00:00:00",
|
||||
"status": "needs_ocr"},
|
||||
}}), encoding="utf-8")
|
||||
ref = f"{SRC_OK}#{SHA_A[:8]}"
|
||||
good = {"type": "entity", "title": "支付閘道", "description": "支付閘道模組",
|
||||
"tags": ["gateway"], "timestamp": today, "sources": [ref], "status": "draft"}
|
||||
page(root, "wiki/entities/good.md", good)
|
||||
bad = {"type": "entity", "title": "壞頁", "description": "缺 tags",
|
||||
"timestamp": today, "sources": ["raw/converted/ghost.md#deadbeef"],
|
||||
"status": "draft", "type": "entity"}
|
||||
page(root, "wiki/entities/bad-schema.md", bad)
|
||||
old = dict(good, title="舊概念", type="concept", timestamp="2024-01-01")
|
||||
page(root, "wiki/concepts/old.md", old)
|
||||
dup_a = dict(good, title="支付閘道逾時缺陷", tags=["gateway", "timeout"],
|
||||
description="逾時缺陷模式")
|
||||
dup_b = dict(good, title="支付閘道逾時缺陷模式", tags=["gateway", "timeout"],
|
||||
description="逾時的缺陷模式")
|
||||
page(root, "wiki/entities/dup-a.md", dup_a, "重試機制在峰值失效。")
|
||||
page(root, "wiki/entities/dup-b.md", dup_b, "重試機制在峰值一律生效。")
|
||||
(root / "index.md").write_text(
|
||||
"- [支付閘道](wiki/entities/good.md)\n- [舊概念](wiki/concepts/old.md)\n"
|
||||
"- [A](wiki/entities/dup-a.md)\n- [B](wiki/entities/dup-b.md)\n", encoding="utf-8")
|
||||
|
||||
|
||||
def run_lint(args):
|
||||
try:
|
||||
lint.main(args)
|
||||
return 0
|
||||
except SystemExit as e:
|
||||
return e.code
|
||||
|
||||
|
||||
def main():
|
||||
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-lint-"))
|
||||
real = kb.llm_json
|
||||
try:
|
||||
root = tmp / "repo"
|
||||
build(root)
|
||||
before = {p: p.read_bytes() for p in root.rglob("*.md")}
|
||||
|
||||
code = run_lint(["--no-llm", "--root", str(root)])
|
||||
assert code == 1, code
|
||||
report = next((root / "reports").glob("lint-*.md")).read_text(encoding="utf-8")
|
||||
for expected in ("缺欄位 tags", "source_ref 不在 manifest",
|
||||
"old.md", "建議人工複審", "未覆蓋 來源.md", "攝入缺口",
|
||||
"needs_ocr", "bad-schema.md` — 不被 index.md"):
|
||||
assert expected in report, f"報告缺少:{expected}\n{report}"
|
||||
assert "good.md" not in report
|
||||
after = {p: p.read_bytes() for p in root.rglob("*.md") if "reports" not in str(p)}
|
||||
assert all(before[p] == after[p] for p in after), "lint 修改了檔案!"
|
||||
print("PASS: schema/stale/coverage/orphan 全抓到,報告產出,檔案零修改")
|
||||
|
||||
kb.llm_json = lambda cfg, task, prompt, validate: (
|
||||
{"relation": "contradiction", "explanation": "重試機制生效與否說法相反"}, "fake:test")
|
||||
code = run_lint(["--root", str(root)])
|
||||
assert code == 1
|
||||
report = sorted((root / "reports").glob("lint-*.md"))[-1].read_text(encoding="utf-8")
|
||||
assert "疑似矛盾" in report and "dup-a.md ↔ wiki/entities/dup-b.md" in report, report
|
||||
print("PASS: LLM 矛盾頁偵測(Phase 6 成功標準的機制驗證)")
|
||||
print("ALL PASS")
|
||||
finally:
|
||||
kb.llm_json = real
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
100
tools/selfcheck_search.py
Normal file
100
tools/selfcheck_search.py
Normal file
@@ -0,0 +1,100 @@
|
||||
"""Phase 5 自檢:BM25 檢索排序、MCP search_wiki / read_page、
|
||||
路徑跳脫防護(信任邊界)、token 上限截斷。
|
||||
|
||||
執行:.venv/Scripts/python tools/selfcheck_search.py
|
||||
"""
|
||||
import os
|
||||
import pathlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import kb
|
||||
import search
|
||||
|
||||
CFG = """ollama: {base_url: "http://localhost:11434", timeout_seconds: 5}
|
||||
tasks:
|
||||
ingest: {model: "fake:test", temperature: 0.2, max_retries: 2}
|
||||
lint: {model: "fake:test", temperature: 0.1, max_retries: 2}
|
||||
fallback: {model: "fake:fb"}
|
||||
limits: {mcp_response_max_tokens: 100, ingest_chunk_max_chars: 12000}
|
||||
"""
|
||||
|
||||
|
||||
def page(root, rel, title, desc, tags, body):
|
||||
meta = {"type": {"summaries": "summary", "entities": "entity",
|
||||
"concepts": "concept"}[rel.split("/")[1]],
|
||||
"title": title, "description": desc, "tags": tags,
|
||||
"timestamp": "2026-07-14", "sources": ["raw/converted/x.md#abcd1234"],
|
||||
"status": "draft"}
|
||||
(root / rel).write_text(kb.dump_page(meta, body), encoding="utf-8")
|
||||
|
||||
|
||||
def main():
|
||||
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-search-"))
|
||||
try:
|
||||
root = tmp / "repo"
|
||||
for sub in ("config", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||
(root / sub).mkdir(parents=True)
|
||||
(root / "config/models.yaml").write_text(CFG, encoding="utf-8")
|
||||
page(root, "wiki/entities/payment-gateway.md", "支付閘道", "支付閘道模組",
|
||||
["gateway"], "壓力測試時出現逾時缺陷,重試機制未生效。")
|
||||
page(root, "wiki/concepts/aml-testing.md", "AML 測試準則", "可疑交易監控測試綜合",
|
||||
["aml", "compliance"], "大額交易與分散交易樣態的監控測試要求。")
|
||||
page(root, "wiki/summaries/long-doc.md", "長文件摘要", "截斷測試用",
|
||||
["test"], "逾時。" * 500)
|
||||
# 與 payment-gateway 共用核心詞「支付閘道」→ df=2, N=4,正是 BM25Okapi
|
||||
# 原式 idf 歸零的配置;沒有這頁就測不出核心詞落空的 bug
|
||||
page(root, "wiki/summaries/gateway-report.md", "支付閘道版本沿革", "支付閘道改版紀錄",
|
||||
["gateway"], "支付閘道於 2026 年改版,新增分期付款。")
|
||||
|
||||
hits = search.search(root, "支付閘道 逾時")
|
||||
assert hits and hits[0]["path"] == "wiki/entities/payment-gateway.md", hits
|
||||
assert all(h["score"] > 0 for h in hits)
|
||||
hits2 = search.search(root, "可疑交易監控")
|
||||
assert hits2[0]["path"] == "wiki/concepts/aml-testing.md", hits2
|
||||
assert search.search(root, "zzz-nonexistent-term") == []
|
||||
print("PASS: BM25 檢索排序(繁中 bigram)")
|
||||
|
||||
# 核心詞(出現在多頁)不得因 idf <= 0 被整批濾掉
|
||||
core = search.search(root, "支付閘道")
|
||||
paths = {h["path"] for h in core}
|
||||
assert {"wiki/entities/payment-gateway.md",
|
||||
"wiki/summaries/gateway-report.md"} <= paths, core
|
||||
assert all(h["score"] > 0 for h in core), core
|
||||
print("PASS: 核心詞跨頁共用仍可檢索(idf 恆正)")
|
||||
|
||||
os.environ["PP_QA_ROOT"] = str(root)
|
||||
import importlib
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parents[1] / "mcp"))
|
||||
server = importlib.import_module("server")
|
||||
res = server.search_wiki("支付閘道 逾時", top_k=99) # top_k 超限 → 收斂為 10
|
||||
assert res[0]["path"] == "wiki/entities/payment-gateway.md"
|
||||
pg = server.read_page("wiki/entities/payment-gateway.md")
|
||||
assert pg["frontmatter"]["title"] == "支付閘道" and not pg["truncated"]
|
||||
assert pg["source_refs"] == ["raw/converted/x.md#abcd1234"]
|
||||
long = server.read_page("wiki/summaries/long-doc.md")
|
||||
assert long["truncated"] and len(long["body"]) <= 150 and long["source_refs"]
|
||||
print("PASS: MCP search_wiki + read_page(含 token 截斷、source_refs 保留)")
|
||||
|
||||
for bad in ("../AGENTS.md", "wiki/../config/models.yaml", "wiki/entities/x.txt"):
|
||||
try:
|
||||
server.read_page(bad)
|
||||
raise AssertionError(f"應拒絕:{bad}")
|
||||
except (ValueError, FileNotFoundError) as e:
|
||||
assert isinstance(e, ValueError), f"{bad} 應為 ValueError"
|
||||
try:
|
||||
server.read_page("wiki/entities/no-such.md")
|
||||
raise AssertionError("應拋 FileNotFoundError")
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
print("PASS: 路徑跳脫防護(信任邊界)")
|
||||
print("ALL PASS")
|
||||
finally:
|
||||
os.environ.pop("PP_QA_ROOT", None)
|
||||
shutil.rmtree(tmp, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user