Re-embed the knowledge base when the embedding model changes
Vectors from two different embedding models are not comparable — Titan v2 has 1024 dimensions where nomic-embed-text has 768 — and _cosine_similarity zips them, so a mismatch produced quietly meaningless scores rather than an error. Nothing forced a re-embed either: sync_local_kb only looked at the content hash, so switching models left every cached vector in place. local_knowledge_docs now records the model each vector came from, and a row is re-embedded when its content hash OR its model differs. Saving a new LOCAL_KB_EMBEDDING_MODEL from the settings panel starts a background pass over every file; the knowledge base panel also has a button for it. While that pass runs, the per-request sync steps aside so no file is embedded twice. The panel now shows what the embedding model is doing: model name, vector size, how many files are embedded with the current model, how many are still on an older one, a progress bar while a pass runs, and the last error. A "接続テスト" button embeds one short string so a broken or unreachable model can be told apart from one that is merely slow to warm up. Also read TAVILY_API_KEY from os.environ at call time. It was imported as a config constant, which is evaluated before load_into_env() runs, so a key saved from the settings panel was stored but never used — not even after a restart. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -104,7 +104,7 @@ class LocalKnowledgeDoc(Base):
|
||||
"""backend/knowledge_base/ 配下のMarkdownファイルとその埋め込みベクトル。
|
||||
|
||||
話題に応じて関連ファイルをシステムプロンプトに注入するために使う。
|
||||
ファイル内容のハッシュが変わった時だけ埋め込みを再計算する。
|
||||
ファイル内容のハッシュか埋め込みモデルが変わった時だけ埋め込みを再計算する。
|
||||
"""
|
||||
__tablename__ = "local_knowledge_docs"
|
||||
|
||||
@@ -113,6 +113,10 @@ class LocalKnowledgeDoc(Base):
|
||||
title: Mapped[str] = mapped_column(String)
|
||||
content_hash: Mapped[str] = mapped_column(String)
|
||||
embedding_json: Mapped[str] = mapped_column(Text)
|
||||
# この埋め込みを計算したモデル名。モデルを切り替えた行を見分けて埋め直すために持つ
|
||||
# (次元数の違うベクトル同士を比較しても意味のないスコアしか出ないため)。
|
||||
# このカラムより前に作られた行は NULL で、モデル不明として埋め直し対象になる。
|
||||
embedding_model: Mapped[str | None] = mapped_column(String, nullable=True)
|
||||
updated_at: Mapped[datetime] = mapped_column(UtcDateTime, default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user