---
license: cc-by-sa-4.0
language:
- zh
- en
tags:
- image-classification
- multi-label
- chinese-painting
- clip
- onnx
base_model: Mitsua/mitsua-japanese-clip-vit-b-16
pipeline_tag: image-classification
---

# 看图识象 Kantu Shixiang v1

**看图识象**是一个小型「看图认物象」模型：给一张照片或一幅画，认出其中的山水花鸟人物等 **99 类诗词常见物象**，判断**画种**（照片／油画／版画／素描／水彩／国画），并指出物象大致落在**整图／左上／右上／左下／右下**哪一格。全部训练材料来自公有领域、开放授权或明示授权的来源，不含任何网络抓取或 AI 生成的图文。

*Kantu Shixiang* ("look at the picture, recognise the imagery") is a small multi-label tagger for the **99 kinds of imagery common in classical Chinese poetry** (mountains, rivers, boats, egrets, herd boys, thatched cottages…), with a 6-way **medium** classifier (photo / oil / print / drawing / watercolour / Chinese ink painting) and a coarse **position** (whole / four quadrants). Every piece of training material is public-domain or openly licensed; nothing is web-scraped or AI-generated.

## 结构 Architecture

| 部件 Part | 来源 Source | 参数 Params | 文件 File |
|---|---|---|---|
| 图像编码器 Image encoder (ViT-B/16 + projection) | [Mitsua Japanese CLIP ViT-B/16](https://huggingface.co/Mitsua/mitsua-japanese-clip-vit-b-16), ELAN MITSUA Project / Abstract Engine, CC BY-SA 4.0, **未改动 unmodified** | 86.4 M | `encoder.safetensors` (fp16, 173 MB) |
| 物象分类头 Imagery head (99 classes, multi-instance over 5 views) | 本项目训练 trained here | 0.97 M | `head.pt` (1.9 MB) |
| 画种分类头 Medium head (6 classes) | 本项目训练 trained here | 4.6 K | in `head.pt` |
| 零样本文字向量 Zero-shot text vectors (19 words, used for 塔 and as a second opinion) | Mitsua text encoder, precomputed | – | `zs_text.npz` |
| 门槛表 Thresholds (photo / painting) | 人眼校准 human-calibrated | – | `thresholds.json`, `paint_thresholds.json` |
| 零样本补标规则 Zero-shot rules (thresholds, which words may be added, minimum head score) | 人眼抽查定 set by human spot-check | – | `zs_rules.json` |

一张图切成 5 个视角（整图拉伸到 224×224，再把 448×448 的整图四等分），各自过编码器得到 5 个 768 维特征；分类头给每个视角打 99 类分数，取最大值作整图分数，最大值所在视角即位置。

Each image is encoded as 5 views (the whole image at 224×224, plus the four quadrants of a 448×448 resize). The head scores all 99 classes per view; the per-class maximum is the image score and the arg-max view is the position.

## 支持的物象 Supported imagery (99)

| 组 Group | 类 Classes |
|---|---|
| 人物 People (12) | 人 童子 仕女 僧 农夫 渔翁 牧童 骑者 行人 读书 饮酒 茶 |
| 山水 Landscape (22) | 山 峰 崖 谷 江 湖 海 溪* 瀑 浪 岸 沙 石 原 田 路 云 雾 烟 雨 雪 虹 |
| 天象 Sky (4) | 日 夕阳 夜 星* |
| 草木 Plants (11) | 树 松 柳 枫 竹 花 荷 草 叶 果 篱 |
| 飞禽 Birds & others (13) | 鸟 鹭 雁 鸭 鸡 鸽 鸦 天鹅 鹰 鹦鹉 孔雀 蝶 鱼 |
| 走兽 Animals (9) | 马 牛 羊 犬 猫 鹿 驴 虎 猿 |
| 建筑 Buildings (12) | 屋 茅屋 楼 亭 寺 塔† 城 村 门 窗 墙 桥 |
| 器物 Objects (10) | 舟船 帆 桨 车 风车 灯 钟 琴 伞 酒 |
| 泉* | 泉 |
| 不发布为可用 Not usable (2) | 月 火 |

`*` 低可信 low confidence（抽查准确率 55–75%）；`†` 塔 只用零样本第二意见，不用分类头 (zero-shot only)。舟与船合并输出为「舟船」。月、火两类分数不可信，门槛表里标为不可用，默认不输出。

零样本第二意见 Zero-shot second opinion：19 个词另算 Mitsua 零样本分数，过门槛可补出标签。分类头没判「有」、只靠零样本单独补出时，要求分类头把握不低于 0.1（屋、书、塔免检）；树不由零样本单独补出（人眼抽查：零样本常把花草丛当成树）。When a tag comes from the zero-shot score alone, the head's own score must be at least 0.1 (house, book and pagoda exempt); trees are never added by zero-shot alone. 这些规则全部写在 `zs_rules.json` 里，只用 ONNX 自己写推理时照表实现即可。All of these rules are published as data in `zs_rules.json`, so an ONNX-only implementation can follow them.

Another 35 poetic words (梅 兰 菊 牡丹 鹤 鸥 燕 樵夫 隐士…) are in the lexicon but had fewer than 300 photo examples and are **not** in this version.

## 用法 Usage

```bash
pip install torch safetensors transformers pillow numpy
python3 shixiang.py your_picture.jpg
```

```python
from PIL import Image
from shixiang import Shixiang
m = Shixiang('.')                      # 目录里有 encoder.safetensors / head.pt / *.json / zs_text.npz
r = m.tag(Image.open('pic.jpg'))       # painting=None → 按画种自动选门槛
print(r['kind'], r['tags'])
# 国画 [{'word': '人', 'score': 0.91, 'where': '左下', 'src': 'head', 'confidence': '正常'}, ...]
```

### ONNX

`shixiang_fp16.onnx`（175 MB）输入 `views` float32 `[B,5,3,224,224]`（像素 /255 后减 0.5 除 0.5），输出 `scores [B,99]`（sigmoid）、`view [B,99]`、`kind [B,6]`（softmax）、`zs [B,19]`、`feat [B,768]`。门槛与合并规则见 `shixiang.py` 的 `decide()`。单张 5 视角在 4 线程 CPU 上约 0.4–0.5 秒，手机可跑。

`shixiang_fp16.onnx` (175 MB): input `views` float32 `[B,5,3,224,224]` normalised with mean 0.5 / std 0.5; outputs `scores`, `view`, `kind`, `zs`, `feat`. Apply the thresholds and merge rules from `decide()` in `shixiang.py`. About 0.4–0.5 s per image on 4 CPU threads; phone-feasible. fp16 agrees with fp32 on 386/400 tag sets (15 tag differences out of 518). An int8 build is not shipped: dynamic quantisation lost too much (60 tag differences out of 518).

## 训练材料与来源 Training data & lineage

- **图像编码器**：Mitsua Japanese CLIP ViT-B/16 原样使用，其训练材料见该模型卡（opt-in 授权图、开放授权图、公有领域图；不含 AI 生成物）。The encoder is used as released; see its model card for its data.
- **物象与画种分类头**：PD12M 第 005–046 包里能按 sha1 对上 Wikimedia Commons 原图的公有领域图；标签**只用人写文字**（Commons 文件名与分类，CC BY-SA 4.0），不用 PD12M 自带的 AI 生成说明。有人写标注的 2,071,107 张进入训练，其中 445,952 张含至少一个物象正例；5% 按 id 哈希留作验证。词表到标签的对应规则（英文正则，含专名与歧义排除）见 `lexicon_v22.json`（134 词，含 35 个未进本版的词）。
- **画种分类头**另加 25,037 幅公有领域／CC BY 4.0 中国画（台北故宫、大都会、克利夫兰、芝加哥艺术博物馆、史密森尼等开放资源）作「国画」正例，20% 留作验证。
- **门槛**：照片门槛由验证集 F1 起步，再按两轮人眼抽查放宽／收紧；画上门槛用 369 幅人工画框的画校准（准确率 ≥ 0.7 下取召回最高），只在画种判为国画时启用（`paint_thresholds.json` 的 `_kinds`）；西洋油画、版画、素描仍用照片门槛，因为校准画中 364 幅是国画，画上门槛在西洋画上未经验证、实测偏松。Painting thresholds apply only when the medium is predicted as Chinese painting; other media use the photo thresholds.
- `training_index.parquet`（145 MB）列出全部训练图的 PD12M id、Commons sha1、文件名、画种与物象标签，供核查。

The heads were trained on Mitsua CLIP features of public-domain images from PD12M parts 005–046 that could be matched back to Wikimedia Commons by sha1. Labels come **only from human-written text** (Commons file names and categories, CC BY-SA 4.0); PD12M's synthetic captions were not used. 2,071,107 labelled images were used (445,952 with at least one positive imagery label). The medium head additionally saw 25,037 public-domain Chinese paintings from museum open-access collections. `training_index.parquet` lists every training image for audit.

## 评测 Evaluation

**照片，人眼抽查 Human spot-check on photos**（每类随机抽 20 张模型判为「有」的照片，由人判对错；80 类共 1,484 张）：**93.9% 准确** (1,394 / 1,484). 16 个弱类改规则重训后复查，从 67.6% 提到 85.6%。逐类数字在 `thresholds.json` 的 `human_check`。

**画，与零样本模型对照 Paintings, vs. zero-shot CLIP-type models**（标准答案＝369 幅人工画框的公有领域画，8 个画框类；每类分数取类内各词最高；AP）：

| 类 Category | 正例 | Mitsua CLIP 零样本 | SigLIP 2 Giant 零样本（中文提示） | SigLIP 2 Giant 零样本（英文提示） | **看图识象** |
|---|---|---|---|---|---|
| 人物 | 130 | 0.584 | 0.581 | 0.775 | **0.798** |
| 屋舍楼阁 | 92 | 0.633 | **0.741** | 0.681 | 0.645 |
| 花 | 50 | 0.602 | 0.113 | 0.685 | **0.842** |
| 走兽 | 41 | 0.619 | 0.826 | **0.851** | 0.765 |
| 舟船 | 32 | 0.420 | 0.771 | **0.849** | 0.528 |
| 鸟 | 22 | 0.548 | 0.562 | 0.536 | **0.656** |
| 月 | 16 | 0.362 | 0.036 | 0.335 | **0.432** |
| 桥 | 14 | 0.063 | **0.396** | 0.364 | 0.368 |
| **mAP** | | 0.479 | 0.503 | **0.635** | 0.629 |

SigLIP 2 Giant 为 giant-opt-patch16-384，系列中最大的一档。同一套 369 幅画上的其他尺寸 Other SigLIP 2 sizes on the same paintings（mAP）：

| 模型 Model | 英文提示 English | 中文提示 Chinese | 整机体积 Download |
|---|---|---|---|
| SigLIP 2 Base/16-224 | 0.602 | 0.453 | 1.50 GB |
| SigLIP 2 So400m/14-384 | 0.648 | 0.453 | 4.54 GB |
| SigLIP 2 Giant-opt/16-384 | 0.635 | 0.503 | 7.49 GB |
| **看图识象** | **0.629** | **0.629** | **0.17 GB** |

看图识象以同一个 86M 图像编码器，在中国画上与系列最大的 SigLIP 2 Giant 英文提示结果基本持平：低 0.006，按图重抽样 5,000 次的 95% 置信区间为 −0.064 到 +0.055，统计上分不出高下；而 Giant 的体积是本模型的 44 倍。中文提示下，SigLIP 2 各档都明显低于看图识象。 分项看，走兽、舟船 Giant 更好，人物、花、鸟、月看图识象更好。分类头有经人眼校准的门槛，零样本模型没有。SigLIP 2 的训练材料是网络抓取的 WebLI，不符合本项目的来源要求，这里只作能力对照。

With the same 86M-parameter image tower, Kantu Shixiang is roughly on par with the largest SigLIP 2 model, Giant, with English prompts on Chinese paintings: 0.006 lower, with a 95% bootstrap interval (5,000 resamples over images) of −0.064 to +0.055, so the two cannot be told apart statistically, while Giant is 44 times larger. With Chinese prompts every SigLIP 2 size falls clearly behind. Per category, Giant is better on animals and boats; Kantu Shixiang is better on people, flowers, birds and the moon. The heads also ship human-calibrated thresholds, which zero-shot models do not. SigLIP 2 is trained on web-scraped WebLI and is listed only as a capability reference.

**体量对照 Size comparison**（两个口径：视觉塔／整机。数字按 Hugging Face 上的权重文件核过；Two framings: vision tower only vs. the whole download. Counted from the safetensors files on Hugging Face）：

| 型号 Model | 视觉塔 Vision tower | 文字塔 Text tower | 整机 Whole | 权重文件 Weights |
|---|---|---|---|---|
| SigLIP 2 Base | 93M | 282M | 375M | 1.50 GB (fp32) |
| SigLIP 2 Large | 316M | 566M | 882M | 3.53 GB (fp32) |
| SigLIP 2 So400m | 428M | 708M | 1,135M | 4.54 GB (fp32) |
| SigLIP 2 Giant | 1,163M | 708M | 1,871M | 7.49 GB (fp32) |
| Mitsua CLIP ViT-B/16（底座 base） | 86M | 135M | 221M | 0.88 GB (fp32) |
| **看图识象 v1 运行件 runtime** | 86M | 不需要 not needed | **87M** | **0.17 GB (fp16)** |

看图识象运行时不需要文字塔，整机只有 SigLIP 2 Base 的四分之一；视觉塔与 Base 同级。SigLIP 2 文字塔大是因为用了 Gemma 的 25.6 万词多语言词表。准确率主表对照的是四档里最大的 Giant。

Kantu Shixiang needs no text tower at inference, so the whole runtime is a quarter of SigLIP 2 Base; its vision tower is the same class as Base. The main accuracy table compares against Giant, the largest of the four.

**画种 Medium**（留出验证）：国画召回 96.8%；照片 94.6%、油画 90.1%、版画 89.4%、素描 73.9%、水彩约 55%（样本少）。

## 已知弱项 Known limitations

- 月、火不可用；星、溪、泉低可信。Moon and fire are unusable; stars, brooks and springs are low-confidence.
- 画上召回偏低：舟船 31%、桥 14%、月 13%（准确率 ≥ 0.7 时）。Recall on paintings is low for boats, bridges and the moon.
- 词表限于山水花鸟人物，不认现代物品、文字、人脸身份。Vocabulary is limited to classical landscape / flora / fauna / figure imagery.
- 训练图以 Commons 照片为主，画上的表现靠 25,037 幅画的画种补强与门槛校准，物象头本身没见过画。
- 英文标签歧义曾造成误学（falcon＝火箭、spring＝春季、cat.＝编目号等），v1 已按人眼抽查逐条排除，仍可能有漏网。

## 许可与署名 License & attribution

整包按 **CC BY-SA 4.0** 发布（图像编码器为 CC BY-SA 4.0，整体须同许可）。使用时请署名：

> 看图识象 Kantu Shixiang v1 (SVOIC, CC BY-SA 4.0), built on Mitsua Japanese CLIP ViT-B/16 by ELAN MITSUA Project / Abstract Engine (CC BY-SA 4.0). Labels from Wikimedia Commons contributors (CC BY-SA 4.0). Images from PD12M (public domain) and museum open-access collections.

This package is released under **CC BY-SA 4.0**. The image encoder is the unmodified Mitsua Japanese CLIP ViT-B/16 (CC BY-SA 4.0); its own attribution files (Commons, Localized Narratives, STAIR, museum lists) are at its repository. Our additions (heads, thresholds, lexicon, training index) are CC BY-SA 4.0 as well. The model is provided as is; recognition can be wrong, and it is not suitable for any use requiring high accuracy.

本模型属 SVOIC「开放文化授权」系列：训练材料来源可查、可追溯、可核对。