Compare commits
10 Commits
cfd1876359
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
238e57e4db | ||
|
|
50f03d8939 | ||
|
|
61d35dd851 | ||
| 45fd1d5874 | |||
| b90727fc11 | |||
| 356ae9367c | |||
|
|
cb16e8688f | ||
|
|
3c1b268074 | ||
|
|
b2eb4fbfa3 | ||
|
|
0b4c2bbb15 |
8
.gitattributes
vendored
Normal file
8
.gitattributes
vendored
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
# raw/ 是 hash 釘選的 provenance 帳本(AGENTS.md §1.4 / §9 / §14):
|
||||||
|
# git 一律不得正規化其換行,否則 checkout 後磁碟 bytes 會與 manifest 的
|
||||||
|
# SHA-256 不符,讓 --verify 對帳失準(本專案 core.autocrlf=true)。
|
||||||
|
# 轉換器已在寫入時以 write_bytes 固定為登記的那份 bytes;此處把同樣的
|
||||||
|
# 保證延伸到 git 的 checkin/checkout。
|
||||||
|
raw/originals/** -text
|
||||||
|
raw/converted/** -text
|
||||||
|
raw/manifest.json -text
|
||||||
29
.gitea/workflows/claude.yml
Normal file
29
.gitea/workflows/claude.yml
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
name: Claude Assistant
|
||||||
|
on:
|
||||||
|
issue_comment:
|
||||||
|
types: [created]
|
||||||
|
pull_request_review_comment:
|
||||||
|
types: [created]
|
||||||
|
issues:
|
||||||
|
types: [opened, assigned, labeled]
|
||||||
|
pull_request_review:
|
||||||
|
types: [submitted]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
claude-response:
|
||||||
|
runs-on: yellow-zeabur-runner
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
with:
|
||||||
|
fetch-depth: 0
|
||||||
|
- uses: markwylde/claude-code-gitea-action@v1.0.20
|
||||||
|
with:
|
||||||
|
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
|
||||||
|
gitea_token: ${{ secrets.USER_TOKEN }}
|
||||||
|
claude_git_name: Claude
|
||||||
|
claude_git_email: claude@anthropic.com
|
||||||
|
path_to_claude_code_executable: /usr/local/bin/claude
|
||||||
|
claude_args: |
|
||||||
|
--append-system-prompt "請一律使用繁體中文回覆,包含 issue/PR 留言、commit message 說明文字、todo list 項目。程式碼註解與變數命名維持英文慣例即可。"
|
||||||
|
env:
|
||||||
|
GITEA_SERVER_URL: https://yellow-git.zeabur.app
|
||||||
13
.markdownlint.jsonc
Normal file
13
.markdownlint.jsonc
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
{
|
||||||
|
// 本專案文件以繁體中文為主,表格一律按「終端顯示寬度」對齊(CJK 字元佔 2 欄),
|
||||||
|
// 這是人類在編輯器裡讀得順的排法。MD060 以「字元位置」比對管線符號,對 CJK
|
||||||
|
// 無感——兩者不可能同時滿足。選擇保留顯示寬度對齊,關閉此規則。
|
||||||
|
// 其餘規則維持預設(含 MD040:程式碼區塊必須標語言)。
|
||||||
|
"MD060": false,
|
||||||
|
|
||||||
|
// MD013:本設定檔一存在,就會取代 VS Code 擴充內建的預設集(該預設集關閉
|
||||||
|
// MD013),使行長檢查被動啟用——這會讓既有檔案(尤其 CJK 表格與逐字保存的
|
||||||
|
// bootstrap prompt)冒出大量新警告。維持專案原本的行為:關閉。
|
||||||
|
// 若日後想強制 80 字元散文慣例,改為 { "tables": false } 並處理既有檔案。
|
||||||
|
"MD013": false
|
||||||
|
}
|
||||||
62
AGENTS.md
62
AGENTS.md
@@ -26,7 +26,10 @@
|
|||||||
EmbeddingGemma(Google)或 snowflake-arctic-embed(Snowflake)。
|
EmbeddingGemma(Google)或 snowflake-arctic-embed(Snowflake)。
|
||||||
4. **raw 層不可變**:`raw/` 內的文件只讀不改。原始檔與轉換後 Markdown 皆登記
|
4. **raw 層不可變**:`raw/` 內的文件只讀不改。原始檔與轉換後 Markdown 皆登記
|
||||||
SHA-256 至 `raw/manifest.json`(結構見 §9),轉換檔條目必須回指原始檔 hash。
|
SHA-256 至 `raw/manifest.json`(結構見 §9),轉換檔條目必須回指原始檔 hash。
|
||||||
wiki 頁面引用來源必須帶 `source_ref`(格式見 §3.3)。
|
wiki 頁面引用來源必須帶 `source_ref`(格式見 §3.3)。登記的 hash 必須等於檔案在
|
||||||
|
磁碟上的實際 bytes(UTF-8,換行不轉換——轉換器一律以 write_bytes 寫入所登記的
|
||||||
|
那份 bytes,不受平台 CRLF 影響;git 亦以 `.gitattributes` 對 `raw/**` 關閉換行
|
||||||
|
正規化,避免 checkout 重新引入 CRLF);完整性以 §14 `--verify` 稽核。
|
||||||
5. **HITL 閘門**:所有 wiki 層的變更以 git branch + PR 形式提交,經人工審核後才
|
5. **HITL 閘門**:所有 wiki 層的變更以 git branch + PR 形式提交,經人工審核後才
|
||||||
合併至 main。攝入腳本永遠不直接 commit 到 main。lint 絕不擅自刪檔,一律標記
|
合併至 main。攝入腳本永遠不直接 commit 到 main。lint 絕不擅自刪檔,一律標記
|
||||||
待人工核准。
|
待人工核准。
|
||||||
@@ -39,7 +42,7 @@
|
|||||||
|
|
||||||
## 2. 目錄結構
|
## 2. 目錄結構
|
||||||
|
|
||||||
```
|
```text
|
||||||
pp-qa-knowledge/
|
pp-qa-knowledge/
|
||||||
├── AGENTS.md # 本檔:schema 正本
|
├── AGENTS.md # 本檔:schema 正本
|
||||||
├── CLAUDE.md # 指標檔 → AGENTS.md
|
├── CLAUDE.md # 指標檔 → AGENTS.md
|
||||||
@@ -93,7 +96,7 @@ status: draft # draft(攝入產出)| reviewed(人工審
|
|||||||
|
|
||||||
### 3.3 source_ref 格式
|
### 3.3 source_ref 格式
|
||||||
|
|
||||||
```
|
```text
|
||||||
<repo 相對路徑>#<sha256 前 8 碼>
|
<repo 相對路徑>#<sha256 前 8 碼>
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -193,13 +196,13 @@ wiki 頁檔名用英文 kebab-case slug(如 `wiki/entities/payment-gateway.md`
|
|||||||
|
|
||||||
**index.md** — 目錄式導航,依頁面類型分節,每頁一行:
|
**index.md** — 目錄式導航,依頁面類型分節,每頁一行:
|
||||||
|
|
||||||
```
|
```markdown
|
||||||
- [標題](wiki/entities/payment-gateway.md) — 一句摘要 | #payment #gateway
|
- [標題](wiki/entities/payment-gateway.md) — 一句摘要 | #payment #gateway
|
||||||
```
|
```
|
||||||
|
|
||||||
**log.md** — append-only 操作日誌,新條目追加於檔案末尾,不修改既有條目:
|
**log.md** — append-only 操作日誌,新條目追加於檔案末尾,不修改既有條目:
|
||||||
|
|
||||||
```
|
```markdown
|
||||||
## [YYYY-MM-DD] <op> | <title>
|
## [YYYY-MM-DD] <op> | <title>
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -272,6 +275,13 @@ output**)。單筆修正只治標;規則修正才治本。
|
|||||||
(global lock、O(n²) 掃描、naive heuristic),註解要寫明天花板與升級路徑。
|
(global lock、O(n²) 掃描、naive heuristic),註解要寫明天花板與升級路徑。
|
||||||
- 非平凡邏輯要留下**一個**可執行的檢查——邏輯壞掉就會失敗的最小東西
|
- 非平凡邏輯要留下**一個**可執行的檢查——邏輯壞掉就會失敗的最小東西
|
||||||
(assert 式自檢或一個小測試檔;不用框架、不用 fixtures)。
|
(assert 式自檢或一個小測試檔;不用框架、不用 fixtures)。
|
||||||
|
- **測資照現實建,不是照「能過」建**:來源檔名用中文+空格(真實 QA 文件就長這樣)、
|
||||||
|
語料要有跨頁共用的核心詞。便利值會變成盲點的形狀——ASCII 檔名讓 `git status`
|
||||||
|
的路徑轉義 bug 溜過,互不重疊的語料讓 BM25 的 idf 退化 bug 溜過。
|
||||||
|
新增檢查時先問:**這組測資和真實輸入差在哪?差異處就是沒被測到的地方。**
|
||||||
|
- **斷言要驗「結果可用」,不是驗「字串存在」**:`assert "。單筆修正只治標;規則修正才治本。
|
|||||||
**有效的跡象**:diff 裡不必要的變更變少、因過度複雜而重寫的次數變少、
|
**有效的跡象**:diff 裡不必要的變更變少、因過度複雜而重寫的次數變少、
|
||||||
釐清問題發生在動手**之前**而非犯錯之後、刻意的捷徑是可見的(`ponytail:`)
|
釐清問題發生在動手**之前**而非犯錯之後、刻意的捷徑是可見的(`ponytail:`)
|
||||||
而非沉默的。
|
而非沉默的。
|
||||||
|
|
||||||
|
## 14. raw 完整性與修復(對帳)
|
||||||
|
|
||||||
|
raw 層以 manifest 的 SHA-256 為信任根(§1.4、§9)。檔案被**手動刪除**或**就地改動**時,
|
||||||
|
下列為正規稽核與修復流程。核心原則:manifest 是 provenance **真相帳本**、不是 cache——
|
||||||
|
hash 的用途是讓刪除**可復原、可驗證**,故先**復原檔案**,而非改帳本去遷就殘缺的磁碟。
|
||||||
|
|
||||||
|
### 14.1 稽核
|
||||||
|
|
||||||
|
```text
|
||||||
|
python tools/convert/convert.py --verify
|
||||||
|
```
|
||||||
|
|
||||||
|
純唯讀(不改任何檔,含 manifest),re-hash 全部登記檔並掃描 `raw/`,回報三類意外、
|
||||||
|
有不一致以退出碼 1 表示(可當 CI/排程閘門):
|
||||||
|
|
||||||
|
- **missing** — 帳本有登記,磁碟上不見了(手動刪除)。
|
||||||
|
- **mismatch** — 檔案還在但 sha256 與登記值不符(raw 被就地改動,違反 §1.4)。
|
||||||
|
- **unregistered** — `raw/originals`/`raw/converted` 下有檔卻不在帳本。
|
||||||
|
`converted/assets/*`(圖片)由 markdown 連結追蹤而非帳本,故略過;`.gitkeep` 亦略過。
|
||||||
|
|
||||||
|
### 14.2 修復決策(可衍生 vs 信任根)
|
||||||
|
|
||||||
|
<!-- ASSUMPTION: 本決策樹為推論的正規流程,原始需求只定義 raw 不可變與 hash 登記;
|
||||||
|
依「可衍生(converted)優先還原、信任根(original)不可再生」的性質分流。 -->
|
||||||
|
|
||||||
|
1. **先還原,不改帳本**(多數「誤刪」到此為止):`raw/` 進版控,`git restore <path>`
|
||||||
|
取回精確 bytes(或從備份),再跑 §14.1 確認 hash == 登記值。帳本零改動。
|
||||||
|
2. **converted 救不回** → 從 original 重轉。注意兩個陷阱:(a) `convert.py` 以 **original
|
||||||
|
的 hash** 去重,original 條目還在會直接 skip、**不會**因 converted 不見而重生,需先
|
||||||
|
自帳本移除該 converted 條目;(b) 函式庫版本變動可能使重轉 bytes 不同 → converted
|
||||||
|
hash 變 → 連累所有指向它的 wiki `source_ref`。故**能還原就別重轉**;重轉屬實質變更,
|
||||||
|
連同 source_ref 一起走 PR。
|
||||||
|
3. **original 救不回** → 不可逆的 provenance 損失,**不得靜默刪條目**(會連鎖 orphan 掉
|
||||||
|
對應 converted、再斷所有 source_ref);據實記錄該檔遺失並上報人工(HITL,§5)。
|
||||||
|
|
||||||
|
### 14.3 動 manifest 的紀律
|
||||||
|
|
||||||
|
- 任何帳本變更走 **branch + PR**(§5)——manifest 是 provenance 帳本,改它需人工審核。
|
||||||
|
- `log.md` 追加一筆 `edit` 條目(§10),說明對了什麼、為什麼。
|
||||||
|
- 刪任何 `converted` 條目前,先確認沒有 wiki `source_ref` 引用它(否則 §7 的 schema
|
||||||
|
檢查會抓到斷鏈);有引用就改為還原檔案,或同 PR 一併更新引用。
|
||||||
|
|||||||
24
README.md
24
README.md
@@ -99,6 +99,13 @@ ollama list # 對照 tasks.ingest.model / tasks.lint.model / tasks.fallba
|
|||||||
|
|
||||||
**前置條件**:須在 `main` 分支、且工作區除 `raw/` 外無未提交變更,否則會報錯中止。
|
**前置條件**:須在 `main` 分支、且工作區除 `raw/` 外無未提交變更,否則會報錯中止。
|
||||||
|
|
||||||
|
> **`convert` 的產出請保持未提交,不要另開分支 commit。** 除 `raw/` 外無未提交變更的條件
|
||||||
|
> 刻意把 `raw/` 排除在外——正確流程是在 `main` 上 `convert`、確認 `raw/converted/*.md` 內容
|
||||||
|
> 無誤後直接 `ingest`,由 ingest 把來源與編譯出的 wiki 頁**一起** commit 進同一個分支。這樣
|
||||||
|
> PR 的 diff 同時含來源與產出,審核者才能對照 LLM 有無編譯錯誤。若先開分支 commit 了轉換檔,
|
||||||
|
> 切回 `main` 時那些檔案會隨分支離開工作區;此時請先 `git merge` 該分支回 `main` 再攝入
|
||||||
|
> (`raw/` 不受 §1.5 HITL 閘門管轄,該條只規範 wiki 層變更)。
|
||||||
|
|
||||||
執行後:變更落在 `ingest/<時間戳>-<slug>` 分支。有 remote 時自動 push 並印出 PR 建立網址;
|
執行後:變更落在 `ingest/<時間戳>-<slug>` 分支。有 remote 時自動 push 並印出 PR 建立網址;
|
||||||
無 remote / 離線時分支保留本地,請手動 push 後開 PR。**經人工審核合併後才進 main。**
|
無 remote / 離線時分支保留本地,請手動 push 後開 PR。**經人工審核合併後才進 main。**
|
||||||
|
|
||||||
@@ -119,6 +126,16 @@ ollama list # 對照 tasks.ingest.model / tasks.lint.model / tasks.fallba
|
|||||||
- **結束碼**:`0` = 無發現;`1` = 有待人工核准項目(供 CI 判斷 / 標紅通知)。
|
- **結束碼**:`0` = 無發現;`1` = 有待人工核准項目(供 CI 判斷 / 標紅通知)。
|
||||||
- 排程:[.gitea/workflows/lint.yaml](.gitea/workflows/lint.yaml)(每週一台北時間 05:00,草稿,部署 Gitea 時啟用)。
|
- 排程:[.gitea/workflows/lint.yaml](.gitea/workflows/lint.yaml)(每週一台北時間 05:00,草稿,部署 Gitea 時啟用)。
|
||||||
|
|
||||||
|
> **`source_ref 不在 manifest` 的成因分類**:lint 只報「對不上」,不分辨成因。跑
|
||||||
|
> [tools/diag_refs.py](tools/diag_refs.py) 把每個對不上的 source_ref 分成五類(hash 不符 /
|
||||||
|
> 路徑字串岔開 / 檔名 hash 後綴岔開 / 檔在磁碟未登記 / 完全無對應),各附修法——
|
||||||
|
> 「補帳本」(§14.3)與「修規則」(§12)的處置相反,先分類再動手。純唯讀不改任何檔,
|
||||||
|
> 結束碼同 lint(`0` 全部解得開 / `1` 有對不上),可當搬機或改 manifest 後的閘門。
|
||||||
|
>
|
||||||
|
> ```powershell
|
||||||
|
> .venv\Scripts\python tools\diag_refs.py
|
||||||
|
> ```
|
||||||
|
|
||||||
### 3.4 查詢 `search` — BM25 檢索
|
### 3.4 查詢 `search` — BM25 檢索
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
@@ -163,7 +180,7 @@ ollama list # 對照 tasks.ingest.model / tasks.lint.model / tasks.fallba
|
|||||||
|
|
||||||
## 6. 目錄速查
|
## 6. 目錄速查
|
||||||
|
|
||||||
```
|
```text
|
||||||
config/models.yaml # Ollama 端點與模型 tag(唯一設定來源)
|
config/models.yaml # Ollama 端點與模型 tag(唯一設定來源)
|
||||||
raw/originals/ # 原始檔,只讀
|
raw/originals/ # 原始檔,只讀
|
||||||
raw/converted/ # 轉換後 Markdown,攝入的實際輸入,只讀
|
raw/converted/ # 轉換後 Markdown,攝入的實際輸入,只讀
|
||||||
@@ -173,7 +190,7 @@ wiki/entities/ # 實體頁(系統/模組/API/法規/專案)
|
|||||||
wiki/concepts/ # 概念頁(跨來源綜合)
|
wiki/concepts/ # 概念頁(跨來源綜合)
|
||||||
index.md # 目錄式導航(由 ingest 自動重建)
|
index.md # 目錄式導航(由 ingest 自動重建)
|
||||||
log.md # append-only 操作日誌
|
log.md # append-only 操作日誌
|
||||||
tools/ # convert/ ingest.py lint.py search.py kb.py(共用模組)
|
tools/ # convert/ ingest.py lint.py diag_refs.py search.py kb.py(共用模組)
|
||||||
mcp/server.py # MCP 薄殼
|
mcp/server.py # MCP 薄殼
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -190,8 +207,10 @@ mcp/server.py # MCP 薄殼
|
|||||||
| `須在 main 分支執行` | 攝入前先切回 `main`。 |
|
| `須在 main 分支執行` | 攝入前先切回 `main`。 |
|
||||||
| `工作區有 raw/ 以外的未提交變更` | 先 commit / stash 非 `raw/` 的變更再攝入。 |
|
| `工作區有 raw/ 以外的未提交變更` | 先 commit / stash 非 `raw/` 的變更再攝入。 |
|
||||||
| `... 不在 manifest 的 converted 條目中` | 該來源尚未轉換;先跑 `convert.py`。 |
|
| `... 不在 manifest 的 converted 條目中` | 該來源尚未轉換;先跑 `convert.py`。 |
|
||||||
|
| lint 報 `source_ref 不在 manifest` | 跑 `tools/diag_refs.py` 分類成因:帳本掉條目→補登走 PR(§14.3);路徑字串岔開→修規則(§12),別逐頁改。 |
|
||||||
| `LLM 輸出驗證失敗(含 fallback)` | 本地模型連主模型 + fallback 都無法產出合規 JSON;檢查 Ollama 是否在線、模型是否拉好。 |
|
| `LLM 輸出驗證失敗(含 fallback)` | 本地模型連主模型 + fallback 都無法產出合規 JSON;檢查 Ollama 是否在線、模型是否拉好。 |
|
||||||
| `push 失敗(無 remote 或離線)` | 正常;分支已保留本地,手動 push 後開 PR。 |
|
| `push 失敗(無 remote 或離線)` | 正常;分支已保留本地,手動 push 後開 PR。 |
|
||||||
|
| ingest 跑很久、看不到進度/想中斷 | 已逐來源 `[i/N]` +逐分段回報進度。Ctrl-C 安全:main 不受影響,中斷時會印回復指令(`git checkout main && git stash -u && git branch -D <branch>`),續跑 `--all-pending` 自動接續。 |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -203,5 +222,6 @@ mcp/server.py # MCP 薄殼
|
|||||||
.venv\Scripts\python tools\convert\selfcheck.py
|
.venv\Scripts\python tools\convert\selfcheck.py
|
||||||
.venv\Scripts\python tools\selfcheck_ingest.py
|
.venv\Scripts\python tools\selfcheck_ingest.py
|
||||||
.venv\Scripts\python tools\selfcheck_lint.py
|
.venv\Scripts\python tools\selfcheck_lint.py
|
||||||
|
.venv\Scripts\python tools\selfcheck_diag_refs.py
|
||||||
.venv\Scripts\python tools\selfcheck_search.py
|
.venv\Scripts\python tools\selfcheck_search.py
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -7,3 +7,5 @@ python-pptx # PowerPoint → Markdown
|
|||||||
trafilatura # 網頁快照 → Markdown 正文萃取
|
trafilatura # 網頁快照 → Markdown 正文萃取
|
||||||
rank-bm25 # tools/search.py BM25(第一版不用向量庫)
|
rank-bm25 # tools/search.py BM25(第一版不用向量庫)
|
||||||
fastmcp # mcp/server.py 薄殼
|
fastmcp # mcp/server.py 薄殼
|
||||||
|
markdown-it-py # 自檢用:以 CommonMark 解析驗證產出的圖片連結真的能 render
|
||||||
|
# (原為 fastmcp 的傳遞依賴,明確宣告以免上游調整後自檢失效)
|
||||||
|
|||||||
@@ -7,6 +7,7 @@
|
|||||||
用法:python tools/convert/convert.py <檔案路徑或URL>... [--vision] [--dry-run] [--root DIR]
|
用法:python tools/convert/convert.py <檔案路徑或URL>... [--vision] [--dry-run] [--root DIR]
|
||||||
支援 .docx/.xlsx/.pdf/.pptx/.html 與 URL;舊版 .doc/.xls/.ppt 經 LibreOffice 升版;
|
支援 .docx/.xlsx/.pdf/.pptx/.html 與 URL;舊版 .doc/.xls/.ppt 經 LibreOffice 升版;
|
||||||
--vision 讓掃描/圖片型 PDF 走本地 vision 模型轉錄(見 from_vision.py)。
|
--vision 讓掃描/圖片型 PDF 走本地 vision 模型轉錄(見 from_vision.py)。
|
||||||
|
--verify 對帳模式:re-hash 全部登記檔、掃出未登記檔,純唯讀不改檔,供刪除/竄改後稽核。
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import datetime
|
import datetime
|
||||||
@@ -87,7 +88,10 @@ def _register_pair(m, root, orig_rel, orig_entry, md_rel, md_text, converter, dr
|
|||||||
print(f"warning: {orig_rel} 轉換結果為空白,請人工檢查來源", file=sys.stderr)
|
print(f"warning: {orig_rel} 轉換結果為空白,請人工檢查來源", file=sys.stderr)
|
||||||
if not dry:
|
if not dry:
|
||||||
md_path = root / md_rel
|
md_path = root / md_rel
|
||||||
md_path.write_text(md_text, encoding="utf-8")
|
# write_bytes(非 write_text):登記的 sha256 算在 md_text.encode() 上,
|
||||||
|
# 而 write_text 在 Windows 會把 \n 轉 \r\n,使磁碟 bytes 與登記 hash 不符,
|
||||||
|
# 令日後「還原後 re-hash 比對」與 --verify 失準。寫入即登記的那份 bytes。
|
||||||
|
md_path.write_bytes(md_text.encode("utf-8"))
|
||||||
m["files"][orig_rel] = orig_entry
|
m["files"][orig_rel] = orig_entry
|
||||||
m["files"][md_rel] = {
|
m["files"][md_rel] = {
|
||||||
"kind": "converted",
|
"kind": "converted",
|
||||||
@@ -125,10 +129,59 @@ def _vision_pdf(root, pdf_path, assets, conv_name):
|
|||||||
blocks = []
|
blocks = []
|
||||||
for i, (img, text) in enumerate(zip(images, texts), 1):
|
for i, (img, text) in enumerate(zip(images, texts), 1):
|
||||||
rel = img.relative_to(md_dir).as_posix()
|
rel = img.relative_to(md_dir).as_posix()
|
||||||
blocks.append(f"<!-- page {i} (vision: {model}) -->\n\n\n\n{text}")
|
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||||
|
blocks.append(f"<!-- page {i} (vision: {model}) -->\n\n\n\n{text}")
|
||||||
return "\n\n".join(blocks), "ok", "from_vision"
|
return "\n\n".join(blocks), "ok", "from_vision"
|
||||||
|
|
||||||
|
|
||||||
|
def verify(root, m):
|
||||||
|
"""對帳模式:re-hash 每個 manifest 條目對應的檔案,並掃出未登記的 raw 檔。
|
||||||
|
純唯讀——不改任何檔案(含 manifest),只回報。有不一致以 SystemExit(1) 表示,
|
||||||
|
可供 CI/排程當閘門用。
|
||||||
|
|
||||||
|
偵測三類意外:
|
||||||
|
missing —— 帳本有登記,磁碟上檔案不見了(手動刪除)。
|
||||||
|
mismatch —— 檔案還在但 sha256 與登記值不符(raw 層被就地改動,違反 §1.4)。
|
||||||
|
unregistered —— raw/originals 或 raw/converted 下有檔卻不在帳本;
|
||||||
|
converted/assets/* 由 markdown 連結而非帳本追蹤,故略過,.gitkeep 亦略過。
|
||||||
|
"""
|
||||||
|
missing, mismatch = [], []
|
||||||
|
for rel, e in m["files"].items():
|
||||||
|
p = root / rel
|
||||||
|
if not p.is_file():
|
||||||
|
missing.append(rel)
|
||||||
|
elif hashlib.sha256(p.read_bytes()).hexdigest() != e["sha256"]:
|
||||||
|
mismatch.append(rel)
|
||||||
|
|
||||||
|
unregistered = []
|
||||||
|
for sub in ("originals", "converted"):
|
||||||
|
base = root / "raw" / sub
|
||||||
|
if not base.is_dir():
|
||||||
|
continue
|
||||||
|
for p in base.rglob("*"):
|
||||||
|
if not p.is_file() or p.name == ".gitkeep":
|
||||||
|
continue
|
||||||
|
if "assets" in p.relative_to(base).parts: # 由 markdown 連結,不入帳本
|
||||||
|
continue
|
||||||
|
rel = p.relative_to(root).as_posix()
|
||||||
|
if rel not in m["files"]:
|
||||||
|
unregistered.append(rel)
|
||||||
|
|
||||||
|
for rel in missing:
|
||||||
|
print(f"missing: {rel}(帳本有登記,磁碟上不見了)", file=sys.stderr)
|
||||||
|
for rel in mismatch:
|
||||||
|
print(f"mismatch: {rel}(sha256 與登記值不符,raw 層被就地改動)", file=sys.stderr)
|
||||||
|
for rel in sorted(unregistered):
|
||||||
|
print(f"unregistered: {rel}(磁碟上有檔,帳本未登記)", file=sys.stderr)
|
||||||
|
|
||||||
|
if missing or mismatch or unregistered:
|
||||||
|
print(f"\nverify: {len(m['files'])} 筆登記 → "
|
||||||
|
f"{len(missing)} missing / {len(mismatch)} mismatch / "
|
||||||
|
f"{len(unregistered)} unregistered(純唯讀,未改任何檔)", file=sys.stderr)
|
||||||
|
raise SystemExit(1)
|
||||||
|
print(f"verify: {len(m['files'])} 筆登記全部對得上,raw/ 無未登記檔案")
|
||||||
|
|
||||||
|
|
||||||
def process_local(root, m, path, dry, vision=False):
|
def process_local(root, m, path, dry, vision=False):
|
||||||
src = pathlib.Path(path).resolve()
|
src = pathlib.Path(path).resolve()
|
||||||
if not src.is_file():
|
if not src.is_file():
|
||||||
@@ -207,7 +260,9 @@ def process_url(root, m, url, dry):
|
|||||||
"added_at": _now(), "status": "converted",
|
"added_at": _now(), "status": "converted",
|
||||||
"source_url": url, "fetched_at": _now()}
|
"source_url": url, "fetched_at": _now()}
|
||||||
if not dry:
|
if not dry:
|
||||||
(originals / name).write_text(html, encoding="utf-8") # 完整快照(來源可能消失)
|
# write_bytes:同 §_register_pair,登記 hash 算在 html.encode() 上,
|
||||||
|
# 避免 Windows 換行轉換讓快照 bytes 與登記 hash 不符(完整快照,來源可能消失)
|
||||||
|
(originals / name).write_bytes(html.encode("utf-8"))
|
||||||
md_text, status = from_web.extract(html)
|
md_text, status = from_web.extract(html)
|
||||||
if status != "ok":
|
if status != "ok":
|
||||||
orig_entry["status"] = "pending"
|
orig_entry["status"] = "pending"
|
||||||
@@ -222,7 +277,10 @@ def process_url(root, m, url, dry):
|
|||||||
|
|
||||||
def main(argv=None):
|
def main(argv=None):
|
||||||
ap = argparse.ArgumentParser(description=__doc__)
|
ap = argparse.ArgumentParser(description=__doc__)
|
||||||
ap.add_argument("inputs", nargs="+", help="檔案路徑或 http(s) URL")
|
ap.add_argument("inputs", nargs="*", help="檔案路徑或 http(s) URL")
|
||||||
|
ap.add_argument("--verify", action="store_true",
|
||||||
|
help="對帳模式:re-hash 全部登記檔、掃出未登記檔,純唯讀不改檔;"
|
||||||
|
"有不一致以退出碼 1 表示(不吃 inputs)")
|
||||||
ap.add_argument("--dry-run", action="store_true")
|
ap.add_argument("--dry-run", action="store_true")
|
||||||
ap.add_argument("--vision", action="store_true",
|
ap.add_argument("--vision", action="store_true",
|
||||||
help="掃描/圖片型 PDF(needs_ocr)改走本地 vision 模型轉錄,"
|
help="掃描/圖片型 PDF(needs_ocr)改走本地 vision 模型轉錄,"
|
||||||
@@ -233,6 +291,11 @@ def main(argv=None):
|
|||||||
a = ap.parse_args(argv)
|
a = ap.parse_args(argv)
|
||||||
root = pathlib.Path(a.root).resolve()
|
root = pathlib.Path(a.root).resolve()
|
||||||
m = load_manifest(root)
|
m = load_manifest(root)
|
||||||
|
if a.verify:
|
||||||
|
verify(root, m)
|
||||||
|
return
|
||||||
|
if not a.inputs:
|
||||||
|
ap.error("需指定至少一個檔案/URL,或改用 --verify 對帳")
|
||||||
failed = []
|
failed = []
|
||||||
for item in a.inputs:
|
for item in a.inputs:
|
||||||
try:
|
try:
|
||||||
|
|||||||
@@ -50,7 +50,9 @@ def _par_md(par, doc, assets_dir, md_dir, counter):
|
|||||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||||
else:
|
else:
|
||||||
rel = name # dry-run:僅佔位
|
rel = name # dry-run:僅佔位
|
||||||
parts.append(f"")
|
# 連結目標以 <> 包住:來源檔名可能含空格或不平衡括號,兩者都會讓
|
||||||
|
# Markdown 連結失效(圖片存在卻整行退化成純文字)
|
||||||
|
parts.append(f"")
|
||||||
parts.append(run.text)
|
parts.append(run.text)
|
||||||
text = "".join(parts).strip()
|
text = "".join(parts).strip()
|
||||||
lvl = _heading_level(par)
|
lvl = _heading_level(par)
|
||||||
|
|||||||
@@ -42,7 +42,8 @@ def convert(src, assets_dir=None, md_dir=None):
|
|||||||
assets_dir.mkdir(parents=True, exist_ok=True)
|
assets_dir.mkdir(parents=True, exist_ok=True)
|
||||||
pix.save(str(assets_dir / name))
|
pix.save(str(assets_dir / name))
|
||||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||||
blocks.append(f"")
|
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||||
|
blocks.append(f"")
|
||||||
doc.close()
|
doc.close()
|
||||||
return "\n\n".join(blocks), "ok"
|
return "\n\n".join(blocks), "ok"
|
||||||
|
|
||||||
|
|||||||
@@ -45,7 +45,8 @@ def convert(src, assets_dir=None, md_dir=None):
|
|||||||
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
rel = (assets_dir / name).relative_to(md_dir).as_posix()
|
||||||
else:
|
else:
|
||||||
rel = name
|
rel = name
|
||||||
blocks.append(f"")
|
# <> 包住:檔名含空格或不平衡括號時,未包住的連結會失效
|
||||||
|
blocks.append(f"")
|
||||||
if slide.has_notes_slide:
|
if slide.has_notes_slide:
|
||||||
notes = slide.notes_slide.notes_text_frame.text.strip()
|
notes = slide.notes_slide.notes_text_frame.text.strip()
|
||||||
if notes:
|
if notes:
|
||||||
|
|||||||
@@ -24,7 +24,41 @@ import kb
|
|||||||
import to_image
|
import to_image
|
||||||
|
|
||||||
|
|
||||||
|
# 測資檔名一律用「中文 + 空格」——真實來源檔就長這樣,而 git status --porcelain
|
||||||
|
# 對這兩者都會加引號轉義。ASCII 檔名的測資曾讓 ingest 的 preflight bug 溜過(§12)。
|
||||||
|
DOCX, XLSX, PDF, SCAN = ("壓測 報告(1).docx", "測試案例 清單.xlsx",
|
||||||
|
"規格 說明.pdf", "掃描件(2 期.pdf")
|
||||||
|
PPTX, HTML, TXT, DOC = ("結案 簡報.pptx", "指引 快照.html",
|
||||||
|
"不支援 筆記.txt", "舊版 報告.doc")
|
||||||
|
|
||||||
|
|
||||||
|
def assert_links_ok(md, base):
|
||||||
|
"""每個圖片連結都要真的 render 成 <img>、且目標檔存在。
|
||||||
|
|
||||||
|
只斷言 "![" 在不在會讓損毀連結矇混過關——含空格/不平衡括號的檔名會產出
|
||||||
|
語法有效但解析失敗的連結,圖檔明明在磁碟上卻整行退化成純文字(§13.4)。
|
||||||
|
"""
|
||||||
|
import re
|
||||||
|
import urllib.parse
|
||||||
|
from markdown_it import MarkdownIt
|
||||||
|
|
||||||
|
want = md.count("![")
|
||||||
|
html = MarkdownIt("commonmark").render(md)
|
||||||
|
got = html.count("<img")
|
||||||
|
assert got == want, f"{want} 個圖片連結只有 {got} 個 render 成功:\n{md[:300]}"
|
||||||
|
for m in re.finditer(r'<img src="([^"]+)"', html):
|
||||||
|
p = base / urllib.parse.unquote(m.group(1))
|
||||||
|
assert p.is_file(), f"連結目標不存在:{p}"
|
||||||
|
return want
|
||||||
|
|
||||||
|
|
||||||
def make_fixtures(d):
|
def make_fixtures(d):
|
||||||
|
import fitz
|
||||||
|
png = d / "img.png" # 內嵌圖片用:docx/pdf/pptx 三條抽圖路徑都要走到
|
||||||
|
pix = fitz.Pixmap(fitz.csRGB, fitz.IRect(0, 0, 8, 8))
|
||||||
|
pix.clear_with(128)
|
||||||
|
pix.save(str(png))
|
||||||
|
|
||||||
import docx as docxlib
|
import docx as docxlib
|
||||||
doc = docxlib.Document()
|
doc = docxlib.Document()
|
||||||
doc.add_heading("支付閘道測試報告", level=1)
|
doc.add_heading("支付閘道測試報告", level=1)
|
||||||
@@ -34,7 +68,8 @@ def make_fixtures(d):
|
|||||||
t.rows[0].cells[1].text = "結果"
|
t.rows[0].cells[1].text = "結果"
|
||||||
t.rows[1].cells[0].text = "TC-001"
|
t.rows[1].cells[0].text = "TC-001"
|
||||||
t.rows[1].cells[1].text = "FAIL"
|
t.rows[1].cells[1].text = "FAIL"
|
||||||
doc.save(d / "report.docx")
|
doc.add_picture(str(png))
|
||||||
|
doc.save(d / DOCX)
|
||||||
|
|
||||||
import openpyxl
|
import openpyxl
|
||||||
wb = openpyxl.Workbook()
|
wb = openpyxl.Workbook()
|
||||||
@@ -42,27 +77,29 @@ def make_fixtures(d):
|
|||||||
ws.title = "測項"
|
ws.title = "測項"
|
||||||
ws.append(["編號", "說明"])
|
ws.append(["編號", "說明"])
|
||||||
ws.append(["TC-001", "逾時 | 重試"])
|
ws.append(["TC-001", "逾時 | 重試"])
|
||||||
wb.save(d / "cases.xlsx")
|
wb.save(d / XLSX)
|
||||||
|
|
||||||
import fitz
|
|
||||||
pdf = fitz.open()
|
pdf = fitz.open()
|
||||||
page = pdf.new_page()
|
page = pdf.new_page()
|
||||||
page.insert_text((72, 72), "Payment gateway timeout defect reproduced under load test.")
|
page.insert_text((72, 72), "Payment gateway timeout defect reproduced under load test.")
|
||||||
pdf.save(d / "text.pdf")
|
page.insert_image(fitz.Rect(72, 100, 122, 150), filename=str(png))
|
||||||
|
pdf.save(d / PDF)
|
||||||
pdf.close()
|
pdf.close()
|
||||||
scanned = fitz.open()
|
scanned = fitz.open()
|
||||||
scanned.new_page() # 無文字層 → 應判 needs_ocr
|
scanned.new_page() # 無文字層 → 應判 needs_ocr
|
||||||
scanned.save(d / "scanned.pdf")
|
scanned.save(d / SCAN)
|
||||||
scanned.close()
|
scanned.close()
|
||||||
|
|
||||||
from pptx import Presentation
|
from pptx import Presentation
|
||||||
|
from pptx.util import Inches
|
||||||
prs = Presentation()
|
prs = Presentation()
|
||||||
slide = prs.slides.add_slide(prs.slide_layouts[1])
|
slide = prs.slides.add_slide(prs.slide_layouts[1])
|
||||||
slide.shapes.title.text = "結案報告"
|
slide.shapes.title.text = "結案報告"
|
||||||
|
slide.shapes.add_picture(str(png), Inches(1), Inches(3))
|
||||||
slide.notes_slide.notes_text_frame.text = "備註:法遵項目全數通過"
|
slide.notes_slide.notes_text_frame.text = "備註:法遵項目全數通過"
|
||||||
prs.save(d / "closing.pptx")
|
prs.save(d / PPTX)
|
||||||
|
|
||||||
(d / "page.html").write_text(
|
(d / HTML).write_text(
|
||||||
"<html><body><article><h1>AML 測試指引</h1>"
|
"<html><body><article><h1>AML 測試指引</h1>"
|
||||||
+ "<p>可疑交易監控測試需涵蓋大額交易與分散交易兩種樣態。</p>" * 8
|
+ "<p>可疑交易監控測試需涵蓋大額交易與分散交易兩種樣態。</p>" * 8
|
||||||
+ "</article></body></html>", encoding="utf-8")
|
+ "</article></body></html>", encoding="utf-8")
|
||||||
@@ -75,17 +112,17 @@ def main():
|
|||||||
fx.mkdir()
|
fx.mkdir()
|
||||||
make_fixtures(fx)
|
make_fixtures(fx)
|
||||||
|
|
||||||
md, st = from_docx.convert(fx / "report.docx")
|
md, st = from_docx.convert(fx / DOCX)
|
||||||
assert st == "ok" and "# 支付閘道測試報告" in md and "| TC-001 | FAIL |" in md, md
|
assert st == "ok" and "# 支付閘道測試報告" in md and "| TC-001 | FAIL |" in md, md
|
||||||
md, st = from_xlsx.convert(fx / "cases.xlsx")
|
md, st = from_xlsx.convert(fx / XLSX)
|
||||||
assert st == "ok" and "## 工作表:測項" in md and "逾時 \\| 重試" in md, md
|
assert st == "ok" and "## 工作表:測項" in md and "逾時 \\| 重試" in md, md
|
||||||
md, st = from_pdf.convert(fx / "text.pdf")
|
md, st = from_pdf.convert(fx / PDF)
|
||||||
assert st == "ok" and "<!-- page 1 -->" in md and "timeout defect" in md, md
|
assert st == "ok" and "<!-- page 1 -->" in md and "timeout defect" in md, md
|
||||||
_, st = from_pdf.convert(fx / "scanned.pdf")
|
_, st = from_pdf.convert(fx / SCAN)
|
||||||
assert st == "needs_ocr", st
|
assert st == "needs_ocr", st
|
||||||
md, st = from_pptx.convert(fx / "closing.pptx")
|
md, st = from_pptx.convert(fx / PPTX)
|
||||||
assert st == "ok" and "## Slide 1" in md and "法遵項目全數通過" in md, md
|
assert st == "ok" and "## Slide 1" in md and "法遵項目全數通過" in md, md
|
||||||
md, st = from_web.extract((fx / "page.html").read_text(encoding="utf-8"))
|
md, st = from_web.extract((fx / HTML).read_text(encoding="utf-8"))
|
||||||
assert st == "ok" and "可疑交易監控" in md, md
|
assert st == "ok" and "可疑交易監控" in md, md
|
||||||
print("PASS: 5 個轉換器 + needs_ocr 偵測")
|
print("PASS: 5 個轉換器 + needs_ocr 偵測")
|
||||||
|
|
||||||
@@ -95,31 +132,77 @@ def main():
|
|||||||
(root / sub).mkdir(parents=True)
|
(root / sub).mkdir(parents=True)
|
||||||
(root / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
(root / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||||
|
|
||||||
convert.main([str(fx / "report.docx"), str(fx / "scanned.pdf"),
|
convert.main([str(fx / DOCX), str(fx / XLSX), str(fx / PDF), str(fx / SCAN),
|
||||||
str(fx / "page.html"), "--root", str(root)])
|
str(fx / PPTX), str(fx / HTML), "--root", str(root)])
|
||||||
m = json.loads((root / "raw/manifest.json").read_text(encoding="utf-8"))
|
m = json.loads((root / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||||
origs = {k: v for k, v in m["files"].items() if v["kind"] == "original"}
|
origs = {k: v for k, v in m["files"].items() if v["kind"] == "original"}
|
||||||
convs = {k: v for k, v in m["files"].items() if v["kind"] == "converted"}
|
convs = {k: v for k, v in m["files"].items() if v["kind"] == "converted"}
|
||||||
assert len(origs) == 3 and len(convs) == 2, m
|
assert len(origs) == 6 and len(convs) == 5, m # SCAN 為 needs_ocr,無 converted
|
||||||
assert origs["raw/originals/scanned.pdf"]["status"] == "needs_ocr"
|
assert origs[f"raw/originals/{SCAN}"]["status"] == "needs_ocr"
|
||||||
for k, v in convs.items():
|
for k, v in convs.items():
|
||||||
assert (root / k).is_file(), k
|
assert (root / k).is_file(), k
|
||||||
assert m["files"][v["original_path"]]["sha256"] == v["original_sha256"]
|
assert m["files"][v["original_path"]]["sha256"] == v["original_sha256"]
|
||||||
assert (root / "raw/originals/report.docx").is_file()
|
assert (root / f"raw/originals/{DOCX}").is_file()
|
||||||
print("PASS: convert.py 端到端 + manifest 對應")
|
print("PASS: convert.py 端到端 + manifest 對應")
|
||||||
|
|
||||||
|
# 圖片連結必須真的能 render——DOCX/PDF/PPTX 三條抽圖路徑都要驗到,
|
||||||
|
# 且測資檔名刻意含空格與不平衡括號(最容易讓連結失效的形狀)
|
||||||
|
linked = sum(assert_links_ok((root / k).read_text(encoding="utf-8"),
|
||||||
|
root / "raw/converted") for k in convs)
|
||||||
|
assert linked >= 3, f"應至少驗到 docx/pdf/pptx 各一個圖片連結,實際 {linked}"
|
||||||
|
print(f"PASS: 圖片連結可 render 且目標存在({linked} 個,檔名含空格/括號)")
|
||||||
|
|
||||||
# 冪等:同檔再跑一次 → skip,manifest 不變
|
# 冪等:同檔再跑一次 → skip,manifest 不變
|
||||||
before = (root / "raw/manifest.json").read_text(encoding="utf-8")
|
before = (root / "raw/manifest.json").read_text(encoding="utf-8")
|
||||||
convert.main([str(fx / "report.docx"), "--root", str(root)])
|
convert.main([str(fx / DOCX), "--root", str(root)])
|
||||||
assert (root / "raw/manifest.json").read_text(encoding="utf-8") == before
|
assert (root / "raw/manifest.json").read_text(encoding="utf-8") == before
|
||||||
print("PASS: 冪等(同 hash 跳過)")
|
print("PASS: 冪等(同 hash 跳過)")
|
||||||
|
|
||||||
|
# --verify 對帳(純唯讀):clean 通過、三類意外都抓到、且完全不改 manifest。
|
||||||
|
# root 已有 docx/pdf/pptx 抽出的 assets(不入帳本)——clean 案例即證明 assets 不被誤報。
|
||||||
|
def run_verify(rt):
|
||||||
|
mm = json.loads((rt / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||||
|
err, code = io.StringIO(), 0
|
||||||
|
with contextlib.redirect_stderr(err), contextlib.redirect_stdout(io.StringIO()):
|
||||||
|
try:
|
||||||
|
convert.verify(rt, mm)
|
||||||
|
except SystemExit as e:
|
||||||
|
code = e.code
|
||||||
|
return code, err.getvalue()
|
||||||
|
|
||||||
|
before = (root / "raw/manifest.json").read_text(encoding="utf-8")
|
||||||
|
code, _ = run_verify(root)
|
||||||
|
assert code == 0, "clean 帳本應通過對帳(含未入帳本的 assets)"
|
||||||
|
|
||||||
|
rroot = tmp / "root_del" # missing:刪掉一個 converted .md
|
||||||
|
shutil.copytree(root, rroot)
|
||||||
|
gone = next(p for p in (rroot / "raw/converted").glob("*.md"))
|
||||||
|
gone.unlink()
|
||||||
|
code, log = run_verify(rroot)
|
||||||
|
assert code == 1 and "missing" in log and gone.name in log, log
|
||||||
|
|
||||||
|
rroot = tmp / "root_mut" # mismatch:就地改動一個 original 的 bytes
|
||||||
|
shutil.copytree(root, rroot)
|
||||||
|
tampered = next(p for p in (rroot / "raw/originals").iterdir() if p.is_file())
|
||||||
|
tampered.write_bytes(tampered.read_bytes() + b"tampered")
|
||||||
|
code, log = run_verify(rroot)
|
||||||
|
assert code == 1 and "mismatch" in log, log
|
||||||
|
|
||||||
|
rroot = tmp / "root_unreg" # unregistered:raw/originals 塞入未登記檔
|
||||||
|
shutil.copytree(root, rroot)
|
||||||
|
(rroot / "raw/originals/未登記 檔.docx").write_bytes(b"stray")
|
||||||
|
code, log = run_verify(rroot)
|
||||||
|
assert code == 1 and "unregistered" in log and "未登記 檔.docx" in log, log
|
||||||
|
|
||||||
|
assert (root / "raw/manifest.json").read_text(encoding="utf-8") == before # 全程唯讀
|
||||||
|
print("PASS: --verify 對帳(clean/missing/mismatch/unregistered,純唯讀)")
|
||||||
|
|
||||||
# dry-run:全新沙盒不落地
|
# dry-run:全新沙盒不落地
|
||||||
root2 = tmp / "root2"
|
root2 = tmp / "root2"
|
||||||
for sub in ("raw/originals", "raw/converted"):
|
for sub in ("raw/originals", "raw/converted"):
|
||||||
(root2 / sub).mkdir(parents=True)
|
(root2 / sub).mkdir(parents=True)
|
||||||
(root2 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
(root2 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||||
convert.main([str(fx / "report.docx"), "--dry-run", "--root", str(root2)])
|
convert.main([str(fx / DOCX), "--dry-run", "--root", str(root2)])
|
||||||
m2 = json.loads((root2 / "raw/manifest.json").read_text(encoding="utf-8"))
|
m2 = json.loads((root2 / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||||
assert m2["files"] == {} and not list((root2 / "raw/originals").iterdir())
|
assert m2["files"] == {} and not list((root2 / "raw/originals").iterdir())
|
||||||
print("PASS: --dry-run 不落地")
|
print("PASS: --dry-run 不落地")
|
||||||
@@ -129,13 +212,13 @@ def main():
|
|||||||
for sub in ("raw/originals", "raw/converted"):
|
for sub in ("raw/originals", "raw/converted"):
|
||||||
(root3 / sub).mkdir(parents=True)
|
(root3 / sub).mkdir(parents=True)
|
||||||
(root3 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
(root3 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||||
(fx / "notes.txt").write_text("不支援的格式", encoding="utf-8") # → ValueError
|
(fx / TXT).write_text("不支援的格式", encoding="utf-8") # → ValueError
|
||||||
err = io.StringIO()
|
err = io.StringIO()
|
||||||
code = None
|
code = None
|
||||||
with contextlib.redirect_stderr(err):
|
with contextlib.redirect_stderr(err):
|
||||||
try:
|
try:
|
||||||
convert.main([str(fx / "report.docx"), str(fx / "notes.txt"),
|
convert.main([str(fx / DOCX), str(fx / TXT),
|
||||||
str(fx / "missing.docx"), # 不存在 → FileNotFoundError
|
str(fx / "不存在 檔案.docx"), # 不存在 → FileNotFoundError
|
||||||
"--root", str(root3)])
|
"--root", str(root3)])
|
||||||
except SystemExit as e:
|
except SystemExit as e:
|
||||||
code = e.code
|
code = e.code
|
||||||
@@ -144,7 +227,7 @@ def main():
|
|||||||
assert "ValueError" in log and "FileNotFoundError" in log, log # 錯誤帶例外型別
|
assert "ValueError" in log and "FileNotFoundError" in log, log # 錯誤帶例外型別
|
||||||
assert "2 個項目轉換失敗" in log, log # 尾端彙總
|
assert "2 個項目轉換失敗" in log, log # 尾端彙總
|
||||||
m3 = json.loads((root3 / "raw/manifest.json").read_text(encoding="utf-8"))
|
m3 = json.loads((root3 / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||||
assert (root3 / "raw/originals/report.docx").is_file() # 好檔仍轉換並落地
|
assert (root3 / f"raw/originals/{DOCX}").is_file() # 好檔仍轉換並落地
|
||||||
assert any(v["kind"] == "converted" for v in m3["files"].values()), m3
|
assert any(v["kind"] == "converted" for v in m3["files"].values()), m3
|
||||||
print("PASS: 失敗回報(例外型別 + 尾端彙總,批次不中斷)")
|
print("PASS: 失敗回報(例外型別 + 尾端彙總,批次不中斷)")
|
||||||
|
|
||||||
@@ -153,21 +236,21 @@ def main():
|
|||||||
for sub in ("raw/originals", "raw/converted"):
|
for sub in ("raw/originals", "raw/converted"):
|
||||||
(root4 / sub).mkdir(parents=True)
|
(root4 / sub).mkdir(parents=True)
|
||||||
(root4 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
(root4 / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||||
(fx / "old.doc").write_bytes(b"\xd0\xcf\x11\xe0legacy-stub") # 非有效 .doc,僅測路由/錯誤
|
(fx / DOC).write_bytes(b"\xd0\xcf\x11\xe0legacy-stub") # 非有效 .doc,僅測路由/錯誤
|
||||||
err = io.StringIO()
|
err = io.StringIO()
|
||||||
code = None
|
code = None
|
||||||
with contextlib.redirect_stderr(err):
|
with contextlib.redirect_stderr(err):
|
||||||
try:
|
try:
|
||||||
convert.main([str(fx / "old.doc"), "--root", str(root4)])
|
convert.main([str(fx / DOC), "--root", str(root4)])
|
||||||
except SystemExit as e:
|
except SystemExit as e:
|
||||||
code = e.code
|
code = e.code
|
||||||
log = err.getvalue()
|
log = err.getvalue()
|
||||||
assert code == 1 and "old.doc" in log, log
|
assert code == 1 and DOC in log, log
|
||||||
assert "不支援的格式" not in log and "RuntimeError" in log, log # 已路由 legacy,乾淨報錯
|
assert "不支援的格式" not in log and "RuntimeError" in log, log # 已路由 legacy,乾淨報錯
|
||||||
print("PASS: 舊格式路由至 legacy(soffice 缺失/升版失敗皆乾淨報錯)")
|
print("PASS: 舊格式路由至 legacy(soffice 缺失/升版失敗皆乾淨報錯)")
|
||||||
|
|
||||||
# A: to_image render(deterministic,真跑)
|
# A: to_image render(deterministic,真跑)
|
||||||
imgs = to_image.render_pdf_pages(fx / "text.pdf", tmp / "imgs")
|
imgs = to_image.render_pdf_pages(fx / PDF, tmp / "imgs")
|
||||||
assert imgs and all(p.is_file() and p.stat().st_size > 0 for p in imgs), imgs
|
assert imgs and all(p.is_file() and p.stat().st_size > 0 for p in imgs), imgs
|
||||||
print(f"PASS: to_image render({len(imgs)} 頁 PNG)")
|
print(f"PASS: to_image render({len(imgs)} 頁 PNG)")
|
||||||
|
|
||||||
@@ -212,7 +295,7 @@ def main():
|
|||||||
(root_v / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
(root_v / "raw/manifest.json").write_text('{"version": 1, "files": {}}', encoding="utf-8")
|
||||||
kb.ollama_chat = fake_chat
|
kb.ollama_chat = fake_chat
|
||||||
try:
|
try:
|
||||||
convert.main([str(fx / "scanned.pdf"), "--vision", "--root", str(root_v)])
|
convert.main([str(fx / SCAN), "--vision", "--root", str(root_v)])
|
||||||
finally:
|
finally:
|
||||||
kb.ollama_chat = orig_chat
|
kb.ollama_chat = orig_chat
|
||||||
mv = json.loads((root_v / "raw/manifest.json").read_text(encoding="utf-8"))
|
mv = json.loads((root_v / "raw/manifest.json").read_text(encoding="utf-8"))
|
||||||
@@ -220,6 +303,7 @@ def main():
|
|||||||
assert len(conv_rel) == 1 and mv["files"][conv_rel[0]]["converter"] == "from_vision", mv
|
assert len(conv_rel) == 1 and mv["files"][conv_rel[0]]["converter"] == "from_vision", mv
|
||||||
md_v = (root_v / conv_rel[0]).read_text(encoding="utf-8")
|
md_v = (root_v / conv_rel[0]).read_text(encoding="utf-8")
|
||||||
assert "可疑交易監控" in md_v and "(vision" in md_v and "![page 1]" in md_v, md_v
|
assert "可疑交易監控" in md_v and "(vision" in md_v and "![page 1]" in md_v, md_v
|
||||||
|
assert_links_ok(md_v, root_v / "raw/converted") # vision 頁的連結同樣要能 render
|
||||||
print("PASS: convert.py --vision 端到端(needs_ocr → 本地 vision 轉錄)")
|
print("PASS: convert.py --vision 端到端(needs_ocr → 本地 vision 轉錄)")
|
||||||
|
|
||||||
print("ALL PASS")
|
print("ALL PASS")
|
||||||
|
|||||||
134
tools/diag_refs.py
Normal file
134
tools/diag_refs.py
Normal file
@@ -0,0 +1,134 @@
|
|||||||
|
"""source_ref ↔ manifest 對帳診斷(AGENTS.md §7 schema / §14 對帳的輔助工具)。
|
||||||
|
|
||||||
|
lint.py 對「source_ref 不在 manifest」只報一句,無法區分成因;本工具把每個
|
||||||
|
對不上的 source_ref 分類成可據以修復的型別(見 CAT_TITLE / FIX),讓「帳本掉
|
||||||
|
條目」「路徑字串岔開」「檔名 hash 後綴岔開」「檔真的遺失」彼此不再混為一談。
|
||||||
|
|
||||||
|
純唯讀——不改任何檔(含 manifest),有對不上以結束碼 1 表示(可當搬機/改
|
||||||
|
manifest 後的閘門)。分類邏輯與 lint 一致:以最後一個 '#' 切出 path 與 sha8。
|
||||||
|
|
||||||
|
用法:python tools/diag_refs.py [--root DIR]
|
||||||
|
結束碼:0 = 全部解得開,1 = 有對不上。
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||||
|
import kb
|
||||||
|
|
||||||
|
CAT_TITLE = {
|
||||||
|
"hash-mismatch": "Hash 不符(帳本有此路徑,但 sha256 與 ref 不同)",
|
||||||
|
"normalize": "路徑字串岔開(正規化後與帳本 key 相同:斜線/前綴/大小寫)",
|
||||||
|
"stem": "檔名 hash 後綴岔開(帳本有同檔名幹、hash 後綴不同的 key)",
|
||||||
|
"unregistered": "檔在磁碟、帳本無此條目",
|
||||||
|
"absent": "完全無對應(帳本、磁碟、檔名幹皆無)",
|
||||||
|
}
|
||||||
|
FIX = {
|
||||||
|
"hash-mismatch": "raw 被就地改動或重轉(違反 §1.4);還原檔案,或連同 source_ref 走 PR(§14.2)。",
|
||||||
|
"normalize": "修產生岔開字串的那一個環節(路徑正規化),別逐頁改 source_ref(§12 change the ruler)。",
|
||||||
|
"stem": "source_ref 指到不同 hash 後綴的檔名;核對正確檔名,別改帳本去遷就(§14.2)。",
|
||||||
|
"unregistered": "帳本掉了條目、檔還在;走 branch+PR 補登 manifest(§14.3),別動 wiki 頁。",
|
||||||
|
"absent": "source_ref 可能攝入時寫歪,或原始檔真的遺失;上報 HITL 據實記錄(§14.2)。",
|
||||||
|
}
|
||||||
|
# 分類輸出順序(愈可機械修復的排愈前)
|
||||||
|
ORDER = ["hash-mismatch", "normalize", "stem", "unregistered", "absent"]
|
||||||
|
|
||||||
|
|
||||||
|
def _norm(path):
|
||||||
|
"""正規化路徑字串以偵測「同一檔、不同寫法」:反斜線→斜線、去開頭 ./、轉小寫。"""
|
||||||
|
s = path.replace("\\", "/")
|
||||||
|
if s.startswith("./"):
|
||||||
|
s = s[2:]
|
||||||
|
return s.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def _stem(path):
|
||||||
|
"""取檔名幹:去目錄、去 .md、去結尾的 -<8 碼 hex>(convert 的檔名 hash 後綴)。"""
|
||||||
|
name = path.replace("\\", "/").rsplit("/", 1)[-1]
|
||||||
|
name = re.sub(r"\.md$", "", name, flags=re.I)
|
||||||
|
return re.sub(r"-[0-9a-f]{8}$", "", name, flags=re.I).lower()
|
||||||
|
|
||||||
|
|
||||||
|
def classify_ref(source_ref, files, exists):
|
||||||
|
"""把單一 source_ref 分類。files:manifest['files'](path→entry);
|
||||||
|
exists(path)→bool 判磁碟是否有該檔。回傳 (category, detail)。
|
||||||
|
category 為 "resolved" 或 CAT_TITLE 的任一鍵。"""
|
||||||
|
path, _, sha8 = str(source_ref).rpartition("#")
|
||||||
|
entry = files.get(path)
|
||||||
|
if entry is not None:
|
||||||
|
if entry["sha256"].startswith(sha8):
|
||||||
|
return "resolved", path
|
||||||
|
return "hash-mismatch", f"帳本 sha256={entry['sha256'][:8]} ≠ ref #{sha8}"
|
||||||
|
norm = _norm(path)
|
||||||
|
for k in files:
|
||||||
|
if _norm(k) == norm:
|
||||||
|
return "normalize", f"帳本 key = '{k}'"
|
||||||
|
stem = _stem(path)
|
||||||
|
for k, e in files.items():
|
||||||
|
if e.get("kind") == "converted" and _stem(k) == stem:
|
||||||
|
return "stem", f"帳本 key = '{k}'"
|
||||||
|
if exists(path):
|
||||||
|
return "unregistered", f"磁碟有 '{path}',帳本無此 key"
|
||||||
|
return "absent", f"'{path}' 帳本無、磁碟無、無同名幹 key"
|
||||||
|
|
||||||
|
|
||||||
|
def collect(root, files, exists):
|
||||||
|
"""掃全庫 wiki 頁,回傳 (findings, n_refs)。findings:category→[(page, ref, detail)]。"""
|
||||||
|
findings = {c: [] for c in ORDER}
|
||||||
|
n_refs = 0
|
||||||
|
for p in kb.iter_pages(root):
|
||||||
|
rel = p.relative_to(root).as_posix()
|
||||||
|
try:
|
||||||
|
meta, _ = kb.parse_page(p.read_text(encoding="utf-8"))
|
||||||
|
except Exception as e:
|
||||||
|
findings["absent"].append((rel, "(無法解析頁面)", str(e)))
|
||||||
|
continue
|
||||||
|
for s in meta.get("sources") or []:
|
||||||
|
n_refs += 1
|
||||||
|
cat, detail = classify_ref(s, files, exists)
|
||||||
|
if cat != "resolved":
|
||||||
|
findings[cat].append((rel, str(s), detail))
|
||||||
|
return findings, n_refs
|
||||||
|
|
||||||
|
|
||||||
|
def format_report(findings, n_refs):
|
||||||
|
total = sum(len(v) for v in findings.values())
|
||||||
|
lines = [f"檢查 source_ref:{n_refs},對不上:{total}", ""]
|
||||||
|
if total == 0:
|
||||||
|
lines.append("全部解得開(source_ref 與 manifest 一致)。")
|
||||||
|
return "\n".join(lines)
|
||||||
|
for cat in ORDER:
|
||||||
|
items = findings[cat]
|
||||||
|
if not items:
|
||||||
|
continue
|
||||||
|
lines.append(f"## {CAT_TITLE[cat]}({len(items)})")
|
||||||
|
lines.append(f"→ 修法:{FIX[cat]}")
|
||||||
|
for page, ref, detail in items:
|
||||||
|
lines.append(f" - [{page}] {ref}")
|
||||||
|
lines.append(f" {detail}")
|
||||||
|
lines.append("")
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv=None):
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__)
|
||||||
|
ap.add_argument("--root", default=str(kb.ROOT))
|
||||||
|
a = ap.parse_args(argv)
|
||||||
|
root = pathlib.Path(a.root).resolve()
|
||||||
|
mpath = root / "raw" / "manifest.json"
|
||||||
|
try:
|
||||||
|
files = json.loads(mpath.read_text(encoding="utf-8"))["files"]
|
||||||
|
except (OSError, KeyError, json.JSONDecodeError) as e:
|
||||||
|
raise SystemExit(f"讀不到 / 解析不了 manifest:{mpath}({e})")
|
||||||
|
|
||||||
|
findings, n_refs = collect(root, files, lambda p: (root / p).is_file())
|
||||||
|
print(format_report(findings, n_refs))
|
||||||
|
if sum(len(v) for v in findings.values()):
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -123,6 +123,8 @@ def summarize(cfg, filename, text):
|
|||||||
parts = split_chunks(text, limit)
|
parts = split_chunks(text, limit)
|
||||||
partials = []
|
partials = []
|
||||||
for i, part in enumerate(parts, 1):
|
for i, part in enumerate(parts, 1):
|
||||||
|
# 長文件的分段迴圈是最容易「長時間靜默」的地方(issue #2):逐段回報進度
|
||||||
|
print(f" 分段 {i}/{len(parts)} …", flush=True)
|
||||||
obj, _ = kb.llm_json(cfg, "ingest",
|
obj, _ = kb.llm_json(cfg, "ingest",
|
||||||
CHUNK_PROMPT.format(i=i, n=len(parts), content=part),
|
CHUNK_PROMPT.format(i=i, n=len(parts), content=part),
|
||||||
validate_chunk)
|
validate_chunk)
|
||||||
@@ -170,14 +172,16 @@ def upsert_item(root, cfg, item, source_ref, today, changed):
|
|||||||
if source_ref not in meta["sources"]:
|
if source_ref not in meta["sources"]:
|
||||||
meta["sources"].append(source_ref)
|
meta["sources"].append(source_ref)
|
||||||
existing.write_text(kb.dump_page(meta, obj["updated_markdown"]), encoding="utf-8")
|
existing.write_text(kb.dump_page(meta, obj["updated_markdown"]), encoding="utf-8")
|
||||||
changed.append(f"更新 {existing.relative_to(root).as_posix()}")
|
msg = f"更新 {existing.relative_to(root).as_posix()}"
|
||||||
else:
|
else:
|
||||||
p = root / kb.PAGE_DIRS[item["type"]] / f"{slug}.md"
|
p = root / kb.PAGE_DIRS[item["type"]] / f"{slug}.md"
|
||||||
meta = {"type": item["type"], "title": item["title"],
|
meta = {"type": item["type"], "title": item["title"],
|
||||||
"description": item["description"], "tags": item["tags"],
|
"description": item["description"], "tags": item["tags"],
|
||||||
"timestamp": today, "sources": [source_ref], "status": "draft"}
|
"timestamp": today, "sources": [source_ref], "status": "draft"}
|
||||||
p.write_text(kb.dump_page(meta, item["facts_markdown"]), encoding="utf-8")
|
p.write_text(kb.dump_page(meta, item["facts_markdown"]), encoding="utf-8")
|
||||||
changed.append(f"新增 {p.relative_to(root).as_posix()}")
|
msg = f"新增 {p.relative_to(root).as_posix()}"
|
||||||
|
changed.append(msg)
|
||||||
|
print(f" ↳ {msg}", flush=True)
|
||||||
|
|
||||||
|
|
||||||
# ---------- git ----------
|
# ---------- git ----------
|
||||||
@@ -191,8 +195,10 @@ def git_preflight(root):
|
|||||||
branch = run_git(root, "branch", "--show-current", capture=True).stdout.strip()
|
branch = run_git(root, "branch", "--show-current", capture=True).stdout.strip()
|
||||||
if branch != "main":
|
if branch != "main":
|
||||||
raise SystemExit(f"須在 main 分支執行(目前:{branch})")
|
raise SystemExit(f"須在 main 分支執行(目前:{branch})")
|
||||||
|
# git 預設 core.quotepath=true,含非 ASCII(中文檔名)或空格的路徑會被引號包住並轉義,
|
||||||
|
# 故比對前先剝除開頭引號,否則 convert 產出的中文檔名會被誤判為 raw/ 以外的變更。
|
||||||
dirty = [l for l in run_git(root, "status", "--porcelain", capture=True)
|
dirty = [l for l in run_git(root, "status", "--porcelain", capture=True)
|
||||||
.stdout.splitlines() if l.strip() and not l[3:].startswith("raw/")]
|
.stdout.splitlines() if l.strip() and not l[3:].lstrip('"').startswith("raw/")]
|
||||||
if dirty:
|
if dirty:
|
||||||
raise SystemExit("工作區有 raw/ 以外的未提交變更,請先處理:\n" + "\n".join(dirty))
|
raise SystemExit("工作區有 raw/ 以外的未提交變更,請先處理:\n" + "\n".join(dirty))
|
||||||
|
|
||||||
@@ -238,7 +244,7 @@ def main(argv=None):
|
|||||||
skipped = [t for t in targets if conv[t]["sha256"][:8] in done]
|
skipped = [t for t in targets if conv[t]["sha256"][:8] in done]
|
||||||
targets = [t for t in targets if conv[t]["sha256"][:8] not in done]
|
targets = [t for t in targets if conv[t]["sha256"][:8] not in done]
|
||||||
for t in skipped:
|
for t in skipped:
|
||||||
print(f"skip: {t} 已有 summary 引用(--force 可重跑)")
|
print(f"skip: {t} 已有 summary 引用(--force 可重跑)", flush=True)
|
||||||
if not targets:
|
if not targets:
|
||||||
print("沒有待攝入的來源。")
|
print("沒有待攝入的來源。")
|
||||||
return
|
return
|
||||||
@@ -259,21 +265,24 @@ def main(argv=None):
|
|||||||
run_git(root, "checkout", "-q", "-b", branch)
|
run_git(root, "checkout", "-q", "-b", branch)
|
||||||
ok, failed, changed = [], [], []
|
ok, failed, changed = [], [], []
|
||||||
try:
|
try:
|
||||||
for t in targets:
|
for i, t in enumerate(targets, 1):
|
||||||
try:
|
try:
|
||||||
|
print(f"[{i}/{len(targets)}] 攝入 {t} …", flush=True)
|
||||||
text = (root / t).read_text(encoding="utf-8")
|
text = (root / t).read_text(encoding="utf-8")
|
||||||
sha8 = conv[t]["sha256"][:8]
|
sha8 = conv[t]["sha256"][:8]
|
||||||
source_ref = f"{t}#{sha8}"
|
source_ref = f"{t}#{sha8}"
|
||||||
obj, model = summarize(cfg, pathlib.Path(conv[t]["original_path"]).name, text)
|
obj, model = summarize(cfg, pathlib.Path(conv[t]["original_path"]).name, text)
|
||||||
p = write_summary_page(root, obj, source_ref, sha8, today)
|
p = write_summary_page(root, obj, source_ref, sha8, today)
|
||||||
changed.append(f"新增 {p.relative_to(root).as_posix()}")
|
msg = f"新增 {p.relative_to(root).as_posix()}"
|
||||||
|
changed.append(msg)
|
||||||
|
print(f" ↳ {msg}", flush=True)
|
||||||
for item in obj["knowledge_items"]:
|
for item in obj["knowledge_items"]:
|
||||||
upsert_item(root, cfg, item, source_ref, today, changed)
|
upsert_item(root, cfg, item, source_ref, today, changed)
|
||||||
ok.append((t, model))
|
ok.append((t, model))
|
||||||
print(f"ingested: {t}(model={model})")
|
print(f" ✓ {t} 完成(model={model})", flush=True)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
failed.append((t, e))
|
failed.append((t, e))
|
||||||
print(f"error: {t}: {e}", file=sys.stderr)
|
print(f" ✗ {t}: {e}", file=sys.stderr, flush=True)
|
||||||
if not ok:
|
if not ok:
|
||||||
run_git(root, "checkout", "-q", "main")
|
run_git(root, "checkout", "-q", "main")
|
||||||
run_git(root, "branch", "-q", "-D", branch)
|
run_git(root, "branch", "-q", "-D", branch)
|
||||||
@@ -295,6 +304,16 @@ def main(argv=None):
|
|||||||
print(f"完成:{len(ok)} 成功、{len(failed)} 失敗。變更在分支 {branch},經 PR 審核後合併。")
|
print(f"完成:{len(ok)} 成功、{len(failed)} 失敗。變更在分支 {branch},經 PR 審核後合併。")
|
||||||
except SystemExit:
|
except SystemExit:
|
||||||
raise
|
raise
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
# KeyboardInterrupt 不是 Exception 子類,會略過下方善後——故獨立攔截。
|
||||||
|
# 中斷時尚未 commit(commit 在迴圈之後),main 必然未受影響;給一條可照做的回復路徑。
|
||||||
|
print(f"\n已中斷(Ctrl-C)。main 未受影響——本次半成品都在分支 {branch}、尚未 commit。\n"
|
||||||
|
f"回到乾淨的 main:\n"
|
||||||
|
f" git checkout main && git stash -u && git branch -D {branch}\n"
|
||||||
|
f"(git stash -u 收起分支上未提交的半成品含未追蹤頁面;確定不要再 git stash drop)\n"
|
||||||
|
f"續跑 python tools/ingest.py --all-pending 會自動從尚未攝入的來源接續。",
|
||||||
|
file=sys.stderr, flush=True)
|
||||||
|
raise SystemExit(130)
|
||||||
except Exception:
|
except Exception:
|
||||||
print(f"攝入中斷。目前在分支 {branch},工作區可能有未提交變更,"
|
print(f"攝入中斷。目前在分支 {branch},工作區可能有未提交變更,"
|
||||||
"請人工檢查(不自動清除以免遺失資料)。", file=sys.stderr)
|
"請人工檢查(不自動清除以免遺失資料)。", file=sys.stderr)
|
||||||
|
|||||||
@@ -5,6 +5,7 @@
|
|||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
|
import math
|
||||||
import pathlib
|
import pathlib
|
||||||
import re
|
import re
|
||||||
import sys
|
import sys
|
||||||
@@ -27,6 +28,20 @@ def tokenize(text):
|
|||||||
return tokens
|
return tokens
|
||||||
|
|
||||||
|
|
||||||
|
class _BM25(BM25Okapi):
|
||||||
|
"""改用 Lucene 式 idf:log(1 + (N-df+0.5)/(df+0.5)),恆為正。
|
||||||
|
|
||||||
|
BM25Okapi 原式在 df >= N/2 時 idf <= 0(rank_bm25 對負 idf 的替代值
|
||||||
|
epsilon × average_idf 在 average_idf 為負時同樣是負的),與 search() 的
|
||||||
|
`s > 0` 過濾相乘,會讓「每頁都提到的核心詞」查詢全數落空——知識庫愈小、
|
||||||
|
詞愈核心愈嚴重。恆正 idf 讓常見詞只是權重低,而不是被整批濾掉。
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _calc_idf(self, nd):
|
||||||
|
for word, freq in nd.items():
|
||||||
|
self.idf[word] = math.log(1 + (self.corpus_size - freq + 0.5) / (freq + 0.5))
|
||||||
|
|
||||||
|
|
||||||
def build_corpus(root):
|
def build_corpus(root):
|
||||||
docs = []
|
docs = []
|
||||||
for p in kb.iter_pages(root):
|
for p in kb.iter_pages(root):
|
||||||
@@ -47,7 +62,7 @@ def search(root, query, k=10):
|
|||||||
docs = build_corpus(pathlib.Path(root))
|
docs = build_corpus(pathlib.Path(root))
|
||||||
if not docs:
|
if not docs:
|
||||||
return []
|
return []
|
||||||
bm = BM25Okapi([d["tokens"] for d in docs])
|
bm = _BM25([d["tokens"] for d in docs])
|
||||||
scores = bm.get_scores(tokenize(query))
|
scores = bm.get_scores(tokenize(query))
|
||||||
ranked = sorted(zip(docs, scores), key=lambda x: -x[1])[:k]
|
ranked = sorted(zip(docs, scores), key=lambda x: -x[1])[:k]
|
||||||
return [{"path": d["path"], "title": d["title"], "description": d["description"],
|
return [{"path": d["path"], "title": d["title"], "description": d["description"],
|
||||||
|
|||||||
114
tools/selfcheck_diag_refs.py
Normal file
114
tools/selfcheck_diag_refs.py
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
"""Phase 4 自檢:沙盒植入五種 source_ref 岔開型別(hash 不符、路徑正規化、
|
||||||
|
檔名 hash 後綴岔開、檔在磁碟未登記、完全無對應)+一個正常解得開的 ref,
|
||||||
|
驗證 diag_refs 逐一分類正確、報告涵蓋各型別、結束碼 1、且不修改任何檔案。
|
||||||
|
另驗全部解得開時結束碼 0。
|
||||||
|
|
||||||
|
執行:.venv/Scripts/python tools/selfcheck_diag_refs.py
|
||||||
|
"""
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
import pathlib
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from contextlib import redirect_stdout
|
||||||
|
|
||||||
|
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||||
|
import kb
|
||||||
|
import diag_refs
|
||||||
|
|
||||||
|
SHA_A = "abcd1234" + "0" * 56
|
||||||
|
SHA_B = "beef5678" + "0" * 56
|
||||||
|
# 帳本 key 一律 posix 正斜線+中文/連字號檔名(真實 convert 產出的形狀)
|
||||||
|
K_OK = "raw/converted/www-pluspay-faq-06b02fd9.md"
|
||||||
|
K_STEM = "raw/converted/www-pluspay-terms-11112222.md" # 同幹、不同 hash 後綴
|
||||||
|
|
||||||
|
FILES = {
|
||||||
|
K_OK: {"kind": "converted", "sha256": SHA_A, "original_path": "raw/originals/faq.html",
|
||||||
|
"original_sha256": SHA_A, "converter": "from_web", "converted_at": "2026-07-01T00:00:00"},
|
||||||
|
K_STEM: {"kind": "converted", "sha256": SHA_B, "original_path": "raw/originals/terms.html",
|
||||||
|
"original_sha256": SHA_B, "converter": "from_web", "converted_at": "2026-07-01T00:00:00"},
|
||||||
|
}
|
||||||
|
|
||||||
|
# ref → 預期分類。unregistered 的檔會實際落地;absent 的不落地。
|
||||||
|
CASES = {
|
||||||
|
"resolved": f"{K_OK}#abcd1234",
|
||||||
|
"hash-mismatch": f"{K_OK}#deadbeef",
|
||||||
|
"normalize": r"raw\converted\www-pluspay-faq-06b02fd9.md#abcd1234", # 反斜線
|
||||||
|
"stem": "raw/converted/www-pluspay-terms-99998888.md#cccccccc", # 同幹、異後綴、非 key
|
||||||
|
"unregistered": "raw/converted/www-pluspay-instructions-77776666.md#77777777",
|
||||||
|
"absent": "raw/converted/www-pluspay-ghost-00001111.md#cafebabe",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def unit():
|
||||||
|
"""classify_ref 純函式逐型別驗證。"""
|
||||||
|
on_disk = {"raw/converted/www-pluspay-instructions-77776666.md"}
|
||||||
|
exists = lambda p: p in on_disk
|
||||||
|
for expect, ref in CASES.items():
|
||||||
|
cat, detail = diag_refs.classify_ref(ref, FILES, exists)
|
||||||
|
assert cat == expect, f"{ref!r} 應分類為 {expect},實得 {cat}({detail})"
|
||||||
|
print("PASS: classify_ref 六型別(含 resolved)分類正確")
|
||||||
|
|
||||||
|
|
||||||
|
def build(root):
|
||||||
|
for sub in ("raw/converted", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||||
|
(root / sub).mkdir(parents=True)
|
||||||
|
(root / "raw/manifest.json").write_text(
|
||||||
|
json.dumps({"version": 1, "files": FILES}), encoding="utf-8")
|
||||||
|
# unregistered 的檔要真的存在於磁碟,才會被判為「檔在、帳本無」而非 absent
|
||||||
|
(root / "raw/converted/www-pluspay-instructions-77776666.md").write_text(
|
||||||
|
"# 未登記\n", encoding="utf-8")
|
||||||
|
today = "2026-07-01"
|
||||||
|
for i, (name, ref) in enumerate(CASES.items()):
|
||||||
|
meta = {"type": "entity", "title": f"頁{name}", "description": "測試頁",
|
||||||
|
"tags": ["t"], "timestamp": today, "sources": [ref], "status": "draft"}
|
||||||
|
(root / f"wiki/entities/p{i}.md").write_text(kb.dump_page(meta, "內文。"), encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def run(argv):
|
||||||
|
buf = io.StringIO()
|
||||||
|
code = 0
|
||||||
|
try:
|
||||||
|
with redirect_stdout(buf):
|
||||||
|
diag_refs.main(argv)
|
||||||
|
except SystemExit as e:
|
||||||
|
code = e.code
|
||||||
|
return code, buf.getvalue()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
unit()
|
||||||
|
tmp = pathlib.Path(tempfile.mkdtemp(prefix="ppqa-diag-"))
|
||||||
|
try:
|
||||||
|
root = tmp / "repo"
|
||||||
|
build(root)
|
||||||
|
before = {p: p.read_bytes() for p in root.rglob("*")
|
||||||
|
if p.is_file()}
|
||||||
|
|
||||||
|
code, out = run(["--root", str(root)])
|
||||||
|
assert code == 1, f"有岔開應以結束碼 1,實得 {code}\n{out}"
|
||||||
|
for cat in ("hash-mismatch", "normalize", "stem", "unregistered", "absent"):
|
||||||
|
assert diag_refs.CAT_TITLE[cat].split("(")[0] in out, f"報告缺型別 {cat}\n{out}"
|
||||||
|
assert "對不上:5" in out, out
|
||||||
|
after = {p: p.read_bytes() for p in root.rglob("*") if p.is_file()}
|
||||||
|
assert before == after, "diag_refs 修改了檔案!"
|
||||||
|
print("PASS: 整合——五型別全報、結束碼 1、零寫檔")
|
||||||
|
|
||||||
|
# 全部解得開 → 結束碼 0
|
||||||
|
good = {"type": "entity", "title": "好頁", "description": "d", "tags": ["t"],
|
||||||
|
"timestamp": "2026-07-01", "sources": [f"{K_OK}#abcd1234"], "status": "draft"}
|
||||||
|
for p in (root / "wiki/entities").glob("*.md"):
|
||||||
|
p.unlink()
|
||||||
|
(root / "wiki/entities/ok.md").write_text(kb.dump_page(good, "內文。"), encoding="utf-8")
|
||||||
|
code, out = run(["--root", str(root)])
|
||||||
|
assert code == 0, f"全部解得開應以結束碼 0,實得 {code}\n{out}"
|
||||||
|
assert "全部解得開" in out, out
|
||||||
|
print("PASS: 全部解得開時結束碼 0")
|
||||||
|
print("ALL PASS")
|
||||||
|
finally:
|
||||||
|
shutil.rmtree(tmp, ignore_errors=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -5,11 +5,13 @@
|
|||||||
執行:.venv/Scripts/python tools/selfcheck_ingest.py
|
執行:.venv/Scripts/python tools/selfcheck_ingest.py
|
||||||
"""
|
"""
|
||||||
import hashlib
|
import hashlib
|
||||||
|
import io
|
||||||
import pathlib
|
import pathlib
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
from contextlib import redirect_stderr, redirect_stdout
|
||||||
|
|
||||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||||
import kb
|
import kb
|
||||||
@@ -82,9 +84,14 @@ def main():
|
|||||||
git(root, "add", "-A")
|
git(root, "add", "-A")
|
||||||
git(root, "commit", "-q", "-m", "init")
|
git(root, "commit", "-q", "-m", "init")
|
||||||
|
|
||||||
# 1) 攝入:建分支、產頁、commit、回到 main
|
# 1) 攝入:建分支、產頁、commit、回到 main(並驗進度輸出可見,issue #2)
|
||||||
kb.llm_json = fake_llm
|
kb.llm_json = fake_llm
|
||||||
|
buf = io.StringIO()
|
||||||
|
with redirect_stdout(buf):
|
||||||
ingest.main(["raw/converted/doc1.md", "--root", str(root)])
|
ingest.main(["raw/converted/doc1.md", "--root", str(root)])
|
||||||
|
out = buf.getvalue()
|
||||||
|
for marker in ("[1/1] 攝入", "分段 1/", "↳ 新增", "✓ ", "完成"):
|
||||||
|
assert marker in out, f"進度輸出缺少 {marker!r}\n{out}"
|
||||||
assert git(root, "branch", "--show-current") == "main"
|
assert git(root, "branch", "--show-current") == "main"
|
||||||
branch = git(root, "branch", "--list", "ingest/*").strip("* ").strip()
|
branch = git(root, "branch", "--list", "ingest/*").strip("* ").strip()
|
||||||
assert branch, "未建立 ingest 分支"
|
assert branch, "未建立 ingest 分支"
|
||||||
@@ -117,7 +124,20 @@ def main():
|
|||||||
git(root, "checkout", "-q", "main")
|
git(root, "checkout", "-q", "main")
|
||||||
print("PASS: 冪等跳過 + 既有 entity 頁就地編輯(sources 追加)")
|
print("PASS: 冪等跳過 + 既有 entity 頁就地編輯(sources 追加)")
|
||||||
|
|
||||||
# 3) kb.llm_json:重試後切 fallback;全失敗拋錯
|
# 3) preflight:raw/ 下未提交的 convert 產出(含中文檔名)不得被誤判為髒污
|
||||||
|
(root / "raw" / "converted" / "測試報告.md").write_text("中文", encoding="utf-8")
|
||||||
|
ingest.git_preflight(root) # 不應拋錯
|
||||||
|
stray = root / "wiki" / "concepts" / "stray.md"
|
||||||
|
stray.write_text("x", encoding="utf-8")
|
||||||
|
try:
|
||||||
|
ingest.git_preflight(root)
|
||||||
|
raise AssertionError("raw/ 以外的未提交變更應中止")
|
||||||
|
except SystemExit:
|
||||||
|
pass
|
||||||
|
stray.unlink()
|
||||||
|
print("PASS: preflight 放行 raw/(含中文檔名)、擋下 raw/ 以外的未提交變更")
|
||||||
|
|
||||||
|
# 4) kb.llm_json:重試後切 fallback;全失敗拋錯
|
||||||
kb.llm_json = real_llm
|
kb.llm_json = real_llm
|
||||||
cfg = kb.load_config(root)
|
cfg = kb.load_config(root)
|
||||||
calls = {"n": 0}
|
calls = {"n": 0}
|
||||||
@@ -134,6 +154,45 @@ def main():
|
|||||||
except RuntimeError as e:
|
except RuntimeError as e:
|
||||||
assert "fallback" in str(e)
|
assert "fallback" in str(e)
|
||||||
print("PASS: 重試 + fallback 切換 + 全失敗報錯")
|
print("PASS: 重試 + fallback 切換 + 全失敗報錯")
|
||||||
|
|
||||||
|
# 5) Ctrl-C 安全(issue #2):中斷時 main 不受影響、以 130 收場,
|
||||||
|
# 且文件宣稱的回復指令真的能回到乾淨 main。第 2 份來源開始摘要時中斷,
|
||||||
|
# 此時第 1 份的頁面已寫入(未追蹤、未 commit)——正是使用者最怕的半成品狀態。
|
||||||
|
add_doc(root, "docA.md", "來源A:待攝入。")
|
||||||
|
add_doc(root, "docB.md", "來源B:待攝入。")
|
||||||
|
git(root, "add", "-A")
|
||||||
|
git(root, "commit", "-q", "-m", "docA/docB")
|
||||||
|
head_before = git(root, "rev-parse", "main") # 攝入不得動到的 main 狀態
|
||||||
|
state = {"n": 0}
|
||||||
|
def boom(cfg, task, prompt, validate):
|
||||||
|
state["n"] += 1
|
||||||
|
if state["n"] >= 2: # 第 2 份來源的 summarize 呼叫
|
||||||
|
raise KeyboardInterrupt
|
||||||
|
return ({"title": "來源A", "description": "d", "slug": "fresh-a",
|
||||||
|
"tags": ["x"], "summary_markdown": "- a", "knowledge_items": []},
|
||||||
|
"fake:test")
|
||||||
|
kb.llm_json = boom
|
||||||
|
err = io.StringIO()
|
||||||
|
try:
|
||||||
|
with redirect_stdout(io.StringIO()), redirect_stderr(err):
|
||||||
|
ingest.main(["raw/converted/docA.md", "raw/converted/docB.md",
|
||||||
|
"--root", str(root)])
|
||||||
|
raise AssertionError("KeyboardInterrupt 應轉為 SystemExit(130)")
|
||||||
|
except SystemExit as e:
|
||||||
|
assert e.code == 130, e.code
|
||||||
|
msg = err.getvalue()
|
||||||
|
assert "git checkout main" in msg and "main 未受影響" in msg, msg
|
||||||
|
assert git(root, "rev-parse", "main") == head_before, "main 被動到了!"
|
||||||
|
cur = git(root, "branch", "--show-current")
|
||||||
|
assert cur.startswith("ingest/"), f"中斷後應停在 ingest 分支,實得 {cur!r}"
|
||||||
|
assert git(root, "status", "--porcelain"), "應有未提交的半成品"
|
||||||
|
# 照文件指令回復,斷言真的回到乾淨 main(驗「結果可用」,非「字串存在」,§13.4)
|
||||||
|
git(root, "checkout", "main")
|
||||||
|
git(root, "stash", "-u")
|
||||||
|
git(root, "branch", "-D", cur)
|
||||||
|
assert git(root, "branch", "--show-current") == "main"
|
||||||
|
assert not git(root, "status", "--porcelain"), "回復後工作區仍不乾淨"
|
||||||
|
print("PASS: Ctrl-C → main 未受影響、130 收場、回復指令實測可回乾淨 main")
|
||||||
print("ALL PASS")
|
print("ALL PASS")
|
||||||
finally:
|
finally:
|
||||||
kb.llm_json, kb.ollama_chat = real_llm, real_chat
|
kb.llm_json, kb.ollama_chat = real_llm, real_chat
|
||||||
|
|||||||
@@ -23,6 +23,8 @@ limits: {mcp_response_max_tokens: 2000, ingest_chunk_max_chars: 12000}
|
|||||||
"""
|
"""
|
||||||
SHA_A = "abcd1234" + "0" * 56
|
SHA_A = "abcd1234" + "0" * 56
|
||||||
SHA_B = "beef5678" + "0" * 56
|
SHA_B = "beef5678" + "0" * 56
|
||||||
|
SRC_OK = "raw/converted/好來源 報告.md"
|
||||||
|
SRC_UNCOVERED = "raw/converted/未覆蓋 來源.md"
|
||||||
|
|
||||||
|
|
||||||
def page(root, rel, meta, body="內文。"):
|
def page(root, rel, meta, body="內文。"):
|
||||||
@@ -36,20 +38,22 @@ def build(root):
|
|||||||
for sub in ("config", "raw/converted", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
for sub in ("config", "raw/converted", "wiki/summaries", "wiki/entities", "wiki/concepts"):
|
||||||
(root / sub).mkdir(parents=True)
|
(root / sub).mkdir(parents=True)
|
||||||
(root / "config/models.yaml").write_text(CFG, encoding="utf-8")
|
(root / "config/models.yaml").write_text(CFG, encoding="utf-8")
|
||||||
|
# 來源檔名一律「中文 + 空格」——真實 QA 文件就長這樣(§12:測資要照現實建,不是照能過建)
|
||||||
(root / "raw/manifest.json").write_text(json.dumps({"version": 1, "files": {
|
(root / "raw/manifest.json").write_text(json.dumps({"version": 1, "files": {
|
||||||
"raw/converted/good-src.md": {"kind": "converted", "sha256": SHA_A,
|
SRC_OK: {"kind": "converted", "sha256": SHA_A,
|
||||||
"original_path": "raw/originals/g.docx",
|
"original_path": "raw/originals/好來源 報告.docx",
|
||||||
"original_sha256": SHA_A, "converter": "from_docx",
|
"original_sha256": SHA_A, "converter": "from_docx",
|
||||||
"converted_at": "2026-07-01T00:00:00"},
|
"converted_at": "2026-07-01T00:00:00"},
|
||||||
"raw/converted/uncovered.md": {"kind": "converted", "sha256": SHA_B,
|
SRC_UNCOVERED: {"kind": "converted", "sha256": SHA_B,
|
||||||
"original_path": "raw/originals/u.docx",
|
"original_path": "raw/originals/未覆蓋 來源.docx",
|
||||||
"original_sha256": SHA_B, "converter": "from_docx",
|
"original_sha256": SHA_B, "converter": "from_docx",
|
||||||
"converted_at": "2026-07-01T00:00:00"},
|
"converted_at": "2026-07-01T00:00:00"},
|
||||||
"raw/originals/scan.pdf": {"kind": "original", "sha256": SHA_B,
|
"raw/originals/掃描件 無文字層.pdf": {"kind": "original", "sha256": SHA_B,
|
||||||
"media_type": "pdf", "added_at": "2026-07-01T00:00:00",
|
"media_type": "pdf",
|
||||||
|
"added_at": "2026-07-01T00:00:00",
|
||||||
"status": "needs_ocr"},
|
"status": "needs_ocr"},
|
||||||
}}), encoding="utf-8")
|
}}), encoding="utf-8")
|
||||||
ref = f"raw/converted/good-src.md#{SHA_A[:8]}"
|
ref = f"{SRC_OK}#{SHA_A[:8]}"
|
||||||
good = {"type": "entity", "title": "支付閘道", "description": "支付閘道模組",
|
good = {"type": "entity", "title": "支付閘道", "description": "支付閘道模組",
|
||||||
"tags": ["gateway"], "timestamp": today, "sources": [ref], "status": "draft"}
|
"tags": ["gateway"], "timestamp": today, "sources": [ref], "status": "draft"}
|
||||||
page(root, "wiki/entities/good.md", good)
|
page(root, "wiki/entities/good.md", good)
|
||||||
@@ -90,7 +94,7 @@ def main():
|
|||||||
assert code == 1, code
|
assert code == 1, code
|
||||||
report = next((root / "reports").glob("lint-*.md")).read_text(encoding="utf-8")
|
report = next((root / "reports").glob("lint-*.md")).read_text(encoding="utf-8")
|
||||||
for expected in ("缺欄位 tags", "source_ref 不在 manifest",
|
for expected in ("缺欄位 tags", "source_ref 不在 manifest",
|
||||||
"old.md", "建議人工複審", "uncovered.md", "攝入缺口",
|
"old.md", "建議人工複審", "未覆蓋 來源.md", "攝入缺口",
|
||||||
"needs_ocr", "bad-schema.md` — 不被 index.md"):
|
"needs_ocr", "bad-schema.md` — 不被 index.md"):
|
||||||
assert expected in report, f"報告缺少:{expected}\n{report}"
|
assert expected in report, f"報告缺少:{expected}\n{report}"
|
||||||
assert "good.md" not in report
|
assert "good.md" not in report
|
||||||
|
|||||||
@@ -44,6 +44,10 @@ def main():
|
|||||||
["aml", "compliance"], "大額交易與分散交易樣態的監控測試要求。")
|
["aml", "compliance"], "大額交易與分散交易樣態的監控測試要求。")
|
||||||
page(root, "wiki/summaries/long-doc.md", "長文件摘要", "截斷測試用",
|
page(root, "wiki/summaries/long-doc.md", "長文件摘要", "截斷測試用",
|
||||||
["test"], "逾時。" * 500)
|
["test"], "逾時。" * 500)
|
||||||
|
# 與 payment-gateway 共用核心詞「支付閘道」→ df=2, N=4,正是 BM25Okapi
|
||||||
|
# 原式 idf 歸零的配置;沒有這頁就測不出核心詞落空的 bug
|
||||||
|
page(root, "wiki/summaries/gateway-report.md", "支付閘道版本沿革", "支付閘道改版紀錄",
|
||||||
|
["gateway"], "支付閘道於 2026 年改版,新增分期付款。")
|
||||||
|
|
||||||
hits = search.search(root, "支付閘道 逾時")
|
hits = search.search(root, "支付閘道 逾時")
|
||||||
assert hits and hits[0]["path"] == "wiki/entities/payment-gateway.md", hits
|
assert hits and hits[0]["path"] == "wiki/entities/payment-gateway.md", hits
|
||||||
@@ -53,6 +57,14 @@ def main():
|
|||||||
assert search.search(root, "zzz-nonexistent-term") == []
|
assert search.search(root, "zzz-nonexistent-term") == []
|
||||||
print("PASS: BM25 檢索排序(繁中 bigram)")
|
print("PASS: BM25 檢索排序(繁中 bigram)")
|
||||||
|
|
||||||
|
# 核心詞(出現在多頁)不得因 idf <= 0 被整批濾掉
|
||||||
|
core = search.search(root, "支付閘道")
|
||||||
|
paths = {h["path"] for h in core}
|
||||||
|
assert {"wiki/entities/payment-gateway.md",
|
||||||
|
"wiki/summaries/gateway-report.md"} <= paths, core
|
||||||
|
assert all(h["score"] > 0 for h in core), core
|
||||||
|
print("PASS: 核心詞跨頁共用仍可檢索(idf 恆正)")
|
||||||
|
|
||||||
os.environ["PP_QA_ROOT"] = str(root)
|
os.environ["PP_QA_ROOT"] = str(root)
|
||||||
import importlib
|
import importlib
|
||||||
sys.path.insert(0, str(pathlib.Path(__file__).parents[1] / "mcp"))
|
sys.path.insert(0, str(pathlib.Path(__file__).parents[1] / "mcp"))
|
||||||
|
|||||||
Reference in New Issue
Block a user