新增CMS+RAG+AI知识库模块/向量数据库检索增强生成及人工智能对话
This commit is contained in:
132
modules/cms-ai/README.md
Normal file
132
modules/cms-ai/README.md
Normal file
@@ -0,0 +1,132 @@
|
||||
|
||||
## 技术交流
|
||||
|
||||
* 官方网站:<https://jeesite.com>
|
||||
* 使用文档:<https://jeesite.com/docs>
|
||||
* 问题反馈:<http://jeesite.net> [【新手必读】](https://gitee.com/thinkgem/jeesite5/issues/I18ARR)
|
||||
* 需求收集:<https://gitee.com/thinkgem/jeesite5/issues/new>
|
||||
* 联系我们:<http://s.jeesite.com>
|
||||
* 关注微信公众号,了解最新动态:
|
||||
|
||||
<p style="padding-left:40px">
|
||||
<img alt="JeeSite微信公众号" src="https://jeesite.com/assets/images/mp.png" width="200">
|
||||
</p>
|
||||
|
||||
* QQ 群:`127515876`、`209330483`、`223507718`、`709534275`、`730390092`、`1373527`、`183903863(外包)`
|
||||
* 微信群:如果二维码过期,请尝试刷新图片,或者添加客服微信 jeesitex 邀请您进群
|
||||
|
||||
<p style="padding-left:40px"><a href="https://jeesite.com/assets/images/wxg_cur.png" target="_blank">
|
||||
<img alt="JeeSite微信群" src="https://jeesite.com/assets/images/wxg_cur.png" width="200"/></a>
|
||||
</p>
|
||||
|
||||
* 源码仓库地址:
|
||||
[Gitee](https://gitee.com/thinkgem/jeesite5)、
|
||||
[GitCode](https://gitcode.com/thinkgem/jeesite5)、
|
||||
[GitHub](https://github.com/thinkgem/jeesite5)
|
||||
* 分离版源码仓库地址:
|
||||
[Gitee](https://gitee.com/thinkgem/jeesite-vue)、
|
||||
[GitCode](https://gitcode.com/thinkgem/jeesite-vue)、
|
||||
[GitHub](https://github.com/thinkgem/jeesite-vue)
|
||||
* 源码合集仓库地址:
|
||||
[GVP](https://gitee.com/thinkgem/jeesite/tree/v5.springboot3)、
|
||||
[G-Star](https://gitcode.com/thinkgem/jeesite/overview?ref=v5.springboot3)、
|
||||
[GitHub](https://github.com/thinkgem/jeesite/tree/v5.springboot3)
|
||||
|
||||
## 模块简介
|
||||
|
||||
本模块基于 Spring AI 和 JeeSite 内容管理系统(CMS)并结合了检索增强生成(Retrieval-Augmented Generation, RAG)技术
|
||||
和先进的人工智能算法(AI),打造了一个强大的企业级知识管理和智能对话平台。该模块专为企业设计,旨在通过高效的知识获取和精准的对话能力,
|
||||
提升企业的信息管理效率和员工的工作效能。
|
||||
|
||||
检索增强生成 RAG 技术使系统能够自动从海量的企业文档中检索最相关的信息,并将其融入到生成的回答中,确保每一次查询都
|
||||
能获得最新且准确的结果。这种检索与生成相结合的方式,不仅提高了信息检索的准确性,还增强了回答的上下文关联性,
|
||||
特别适合处理复杂的企业知识库。
|
||||
|
||||
此外该模块,支持云上大模型和本地部署的大模型,如:DeepSeek、通义千问,理论上支持所有 OpenAPI 标准接口的 AI 提供商。
|
||||
并能无缝集成多种嵌入式 AI 模型的向量数据库,如 PGVector、Elasticsearch、Milvus 等,实现高效的数据存储、检索及分析。
|
||||
无论是大规模数据集还是高度专业化的领域知识,JeeSite CMS + RAG + AI 都能提供定制化解决方案,满足企业多样化的业务需求和技术要求。
|
||||
企业可以轻松管理和访问复杂的信息资源,促进内部知识共享和创新,从而在竞争激烈的市场环境中保持领先地位。
|
||||
|
||||
优势:本模块结构清晰,代码简洁易懂,不管是正式项目、或是学习 AI 技术、都能轻松应对读懂源代码。
|
||||
|
||||
## AI 模型配置
|
||||
|
||||
支持的 AI 模型列表:<https://docs.spring.io/spring-ai/reference/1.0/api/index.html>
|
||||
|
||||
* 线上模型:理论上支持所有 [OpenAPI](https://help.aliyun.com/zh/model-studio/developer-reference/use-qwen-by-calling-api) 标准接口的 AI 提供商。
|
||||
|
||||
* 本地模型:使用 [Ollama](https://ollama.com) 安装方法,本文不多赘述,网上有很多安装资料。
|
||||
|
||||
* 模型类型包括:聊天对话模型和嵌入式向量库模型,需注意 dimensions 维度参数,要和模型要求的匹配。
|
||||
|
||||
具体配置项详见 `jeesite-cms-ai.yml` 文件,有注释。
|
||||
|
||||
## 向量数据库配置
|
||||
|
||||
支持的向量库列表:<https://docs.spring.io/spring-ai/reference/1.0/api/vectordbs.html>
|
||||
|
||||
* PGVector
|
||||
* Elasticsearch
|
||||
* Milvus
|
||||
* ...
|
||||
|
||||
具体配置项详见 `jeesite-cms-ai.yml` 文件,有注释。
|
||||
|
||||
### PGVector 建表语句
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION IF NOT EXISTS vector;
|
||||
CREATE EXTENSION IF NOT EXISTS hstore;
|
||||
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
|
||||
|
||||
-- 使用 all-minilm 模型时创建
|
||||
DROP TABLE IF EXISTS vector_store_384;
|
||||
CREATE TABLE IF NOT EXISTS vector_store_384 (
|
||||
id varchar(64) DEFAULT uuid_generate_v4() PRIMARY KEY,
|
||||
content text,
|
||||
metadata json,
|
||||
embedding vector(384)
|
||||
);
|
||||
CREATE INDEX ON vector_store_384 USING HNSW (embedding vector_cosine_ops);
|
||||
|
||||
-- 使用 nomic-embed-text 模型时创建
|
||||
DROP TABLE IF EXISTS vector_store_786;
|
||||
CREATE TABLE IF NOT EXISTS vector_store_786 (
|
||||
id varchar(64) DEFAULT uuid_generate_v4() PRIMARY KEY,
|
||||
content text,
|
||||
metadata json,
|
||||
embedding vector(768)
|
||||
);
|
||||
CREATE INDEX ON vector_store_786 USING HNSW (embedding vector_cosine_ops);
|
||||
|
||||
-- 使用 bge-m3 模型时创建
|
||||
DROP TABLE IF EXISTS vector_store_1024;
|
||||
CREATE TABLE IF NOT EXISTS vector_store_1024 (
|
||||
id varchar(64) DEFAULT uuid_generate_v4() PRIMARY KEY,
|
||||
content text,
|
||||
metadata json,
|
||||
embedding vector(1024)
|
||||
);
|
||||
CREATE INDEX ON vector_store_1024 USING HNSW (embedding vector_cosine_ops);
|
||||
```
|
||||
|
||||
## 授权协议声明
|
||||
|
||||
1. 基于 Apache License Version 2.0 协议发布,可用于商业项目,但必须遵守以下补充条款。
|
||||
2. 不得将本软件应用于危害国家安全、荣誉和利益的行为,不能以任何形式用于非法为目的的行为。
|
||||
3. 在使用本软件时,由于它集成了众多第三方开源软件,请共同遵守这些开源软件的使用许可条款规定。
|
||||
4. 在延伸的代码中(修改和有源代码衍生的代码中)需要带有原来代码中的协议、版权声明和其他原作者
|
||||
规定需要包含的说明(请尊重原作者的著作权,不要删除或修改文件中的`Copyright`和`@author`信息)
|
||||
更不要,全局替换源代码中的 jeesite 或 ThinkGem 等字样,否则你将违反本协议条款承担责任。
|
||||
5. 您若套用本软件的一些代码或功能参考,请保留源文件中的版权和作者,需要在您的软件介绍明显位置
|
||||
说明出处,举例:本软件基于 JeeSite 快速开发平台,并附带链接:http://jeesite.com
|
||||
6. 任何基于本软件而产生的一切法律纠纷和责任,均于我司无关。
|
||||
7. 如果你对本软件有改进,希望可以贡献给我们,共同进步。
|
||||
8. 本项目已申请软件著作权,请尊重开源,感谢阅读。
|
||||
9. 无用户数限制,无在线人数限制,放心使用。
|
||||
|
||||
## 技术支持与服务
|
||||
|
||||
* 本软件免费,我们也提供了相应的收费服务,因为:
|
||||
* 没有资金的支撑就很难得到发展,特别是一个好的产品,如果 JeeSite 帮助了您,请为我们点赞。支持我们,您可以获得更多回馈,我们会把公益事业做的更好,开放更多资源,回报社区和社会。请给我们一些动力吧,在此非常感谢已支持我们的朋友!
|
||||
* **联系我们**:请访问技术支持与服务页面:<http://s.jeesite.com>
|
||||
22
modules/cms-ai/bin/deploy.bat
Normal file
22
modules/cms-ai/bin/deploy.bat
Normal file
@@ -0,0 +1,22 @@
|
||||
@echo off
|
||||
rem /**
|
||||
rem * Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
rem * No deletion without permission, or be held responsible to law.
|
||||
rem *
|
||||
rem * Author: ThinkGem@163.com
|
||||
rem */
|
||||
echo.
|
||||
echo [<5B><>Ϣ] <20><><EFBFBD>̵<F0B9A4B3>Maven<65><6E><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>
|
||||
echo.
|
||||
|
||||
%~d0
|
||||
cd %~dp0
|
||||
|
||||
call mvn -v
|
||||
echo.
|
||||
|
||||
cd ..
|
||||
call mvn clean deploy -Dmaven.test.skip=true -Pdeploy
|
||||
|
||||
cd bin
|
||||
pause
|
||||
18
modules/cms-ai/bin/deploy.sh
Normal file
18
modules/cms-ai/bin/deploy.sh
Normal file
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
# /**
|
||||
# * Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
# * No deletion without permission, or be held responsible to law.
|
||||
# *
|
||||
# * Author: ThinkGem@163.com
|
||||
# */
|
||||
echo ""
|
||||
echo "[信息] 部署工程到Maven服务器。"
|
||||
echo ""
|
||||
|
||||
mvn -v
|
||||
echo ""
|
||||
|
||||
cd ..
|
||||
mvn clean deploy -Dmaven.test.skip=true -Pdeploy
|
||||
|
||||
cd bin
|
||||
22
modules/cms-ai/bin/package.bat
Normal file
22
modules/cms-ai/bin/package.bat
Normal file
@@ -0,0 +1,22 @@
|
||||
@echo off
|
||||
rem /**
|
||||
rem * Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
rem * No deletion without permission, or be held responsible to law.
|
||||
rem *
|
||||
rem * Author: ThinkGem@163.com
|
||||
rem */
|
||||
echo.
|
||||
echo [<5B><>Ϣ] <20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>װ<EFBFBD><D7B0><EFBFBD>̣<EFBFBD><CCA3><EFBFBD><EFBFBD><EFBFBD>jar<61><72><EFBFBD>ļ<EFBFBD><C4BC><EFBFBD>
|
||||
echo.
|
||||
|
||||
%~d0
|
||||
cd %~dp0
|
||||
|
||||
call mvn -v
|
||||
echo.
|
||||
|
||||
cd ..
|
||||
call mvn clean install -Dmaven.test.skip=true -Ppackage
|
||||
|
||||
cd bin
|
||||
pause
|
||||
18
modules/cms-ai/bin/package.sh
Normal file
18
modules/cms-ai/bin/package.sh
Normal file
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
# /**
|
||||
# * Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
# * No deletion without permission, or be held responsible to law.
|
||||
# *
|
||||
# * Author: ThinkGem@163.com
|
||||
# */
|
||||
echo ""
|
||||
echo "[信息] 打包安装工程,生成jar包文件。"
|
||||
echo ""
|
||||
|
||||
mvn -v
|
||||
echo ""
|
||||
|
||||
cd ..
|
||||
mvn clean install -Dmaven.test.skip=true -Ppackage
|
||||
|
||||
cd bin
|
||||
3590
modules/cms-ai/db/cms-ai.erm
Normal file
3590
modules/cms-ai/db/cms-ai.erm
Normal file
File diff suppressed because it is too large
Load Diff
117
modules/cms-ai/pom.xml
Normal file
117
modules/cms-ai/pom.xml
Normal file
@@ -0,0 +1,117 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
|
||||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
|
||||
<modelVersion>4.0.0</modelVersion>
|
||||
|
||||
<parent>
|
||||
<groupId>com.jeesite</groupId>
|
||||
<artifactId>jeesite-parent</artifactId>
|
||||
<version>5.11.0.springboot3-SNAPSHOT</version>
|
||||
<relativePath>../../parent/pom.xml</relativePath>
|
||||
</parent>
|
||||
|
||||
<artifactId>jeesite-module-cms-ai</artifactId>
|
||||
<packaging>jar</packaging>
|
||||
|
||||
<name>JeeSite Module CMS+RAG+AI 向量数据库及人工智能对话知识库</name>
|
||||
<url>http://jeesite.com</url>
|
||||
<inceptionYear>2013-Now</inceptionYear>
|
||||
|
||||
<properties>
|
||||
|
||||
<spring-ai.version>1.0.0-M6</spring-ai.version>
|
||||
|
||||
</properties>
|
||||
|
||||
<dependencies>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.jeesite</groupId>
|
||||
<artifactId>jeesite-module-core</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
<groupId>com.jeesite</groupId>
|
||||
<artifactId>jeesite-module-cms</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
|
||||
<!-- 云上大模型 -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-openai-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- 本地大模型 -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-ollama-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- PG 向量数据库 -->
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-pgvector-store-spring-boot-starter</artifactId>
|
||||
</dependency>
|
||||
|
||||
<!-- ES 向量数据库
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-elasticsearch-store-spring-boot-starter</artifactId>
|
||||
</dependency> -->
|
||||
|
||||
<!-- Milvus 向量数据库
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-milvus-store-spring-boot-starter</artifactId>
|
||||
<exclusions>
|
||||
<exclusion>
|
||||
<artifactId>slf4j-reload4j</artifactId>
|
||||
<groupId>org.slf4j</groupId>
|
||||
</exclusion>
|
||||
</exclusions>
|
||||
</dependency>
|
||||
<dependency>
|
||||
<groupId>io.netty</groupId>
|
||||
<artifactId>netty-resolver-dns-native-macos</artifactId>
|
||||
<classifier>osx-aarch_64</classifier>
|
||||
</dependency> -->
|
||||
|
||||
<!-- HTML 转 Markdown -->
|
||||
<dependency>
|
||||
<groupId>com.vladsch.flexmark</groupId>
|
||||
<artifactId>flexmark-html2md-converter</artifactId>
|
||||
<version>0.64.8</version>
|
||||
</dependency>
|
||||
|
||||
</dependencies>
|
||||
|
||||
<dependencyManagement>
|
||||
<dependencies>
|
||||
<dependency>
|
||||
<groupId>org.springframework.ai</groupId>
|
||||
<artifactId>spring-ai-bom</artifactId>
|
||||
<version>${spring-ai.version}</version>
|
||||
<type>pom</type>
|
||||
<scope>import</scope>
|
||||
</dependency>
|
||||
</dependencies>
|
||||
</dependencyManagement>
|
||||
|
||||
<developers>
|
||||
<developer>
|
||||
<id>thinkgem</id>
|
||||
<name>WangZhen</name>
|
||||
<email>thinkgem at 163.com</email>
|
||||
<roles><role>Project lead</role></roles>
|
||||
<timezone>+8</timezone>
|
||||
</developer>
|
||||
</developers>
|
||||
|
||||
<organization>
|
||||
<name>JeeSite</name>
|
||||
<url>http://jeesite.com</url>
|
||||
</organization>
|
||||
|
||||
</project>
|
||||
@@ -0,0 +1,54 @@
|
||||
/**
|
||||
* Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
* No deletion without permission, or be held responsible to law.
|
||||
*/
|
||||
package com.jeesite.modules.cms.ai.config;
|
||||
|
||||
import com.jeesite.common.datasource.DataSourceHolder;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
|
||||
import org.springframework.context.annotation.Bean;
|
||||
import org.springframework.context.annotation.Configuration;
|
||||
import org.springframework.context.annotation.Primary;
|
||||
import org.springframework.jdbc.core.JdbcTemplate;
|
||||
|
||||
import javax.sql.DataSource;
|
||||
import java.sql.SQLException;
|
||||
|
||||
/**
|
||||
* AI 聊天配置类
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Configuration
|
||||
public class CmsAiChatConfig {
|
||||
|
||||
/**
|
||||
* PG向量库数据源
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Bean
|
||||
@Primary
|
||||
@ConditionalOnProperty(name = "jdbc.ds_pgvector.type")
|
||||
public JdbcTemplate pgVectorStoreJdbcTemplate() throws SQLException {
|
||||
DataSource dataSource = DataSourceHolder.getRoutingDataSource()
|
||||
.createDataSource("ds_pgvector");
|
||||
return new JdbcTemplate(dataSource);
|
||||
}
|
||||
|
||||
/**
|
||||
* 聊天对话客户端
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Bean
|
||||
public ChatClient chatClient(ChatClient.Builder builder) {
|
||||
return builder
|
||||
.defaultSystem("你是我的知识库AI助手,请帮我解答我提出的相关问题。")
|
||||
.build();
|
||||
}
|
||||
|
||||
// @Bean
|
||||
// public BatchingStrategy batchingStrategy() {
|
||||
// return new TokenCountBatchingStrategy(EncodingType.CL100K_BASE, Integer.MAX_VALUE, 0.1);
|
||||
// }
|
||||
|
||||
}
|
||||
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
* No deletion without permission, or be held responsible to law.
|
||||
*/
|
||||
package com.jeesite.modules.cms.ai.service;
|
||||
|
||||
import com.jeesite.common.collect.ListUtils;
|
||||
import com.jeesite.common.collect.MapUtils;
|
||||
import com.jeesite.common.lang.StringUtils;
|
||||
import com.jeesite.common.lang.TimeUtils;
|
||||
import com.jeesite.common.utils.PageUtils;
|
||||
import com.jeesite.modules.cms.entity.Article;
|
||||
import com.jeesite.modules.cms.service.ArticleVectorStore;
|
||||
import com.jeesite.modules.cms.utils.CmsUtils;
|
||||
import com.vladsch.flexmark.html2md.converter.FlexmarkHtmlConverter;
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
import org.springframework.ai.document.Document;
|
||||
import org.springframework.ai.transformer.splitter.TokenTextSplitter;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.ai.vectorstore.filter.FilterExpressionBuilder;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* CMS 文章向量库存储
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Service
|
||||
public class ArticleVectorStoreImpl implements ArticleVectorStore {
|
||||
|
||||
protected Logger logger = LoggerFactory.getLogger(getClass());
|
||||
|
||||
@Autowired
|
||||
private VectorStore vectorStore;
|
||||
|
||||
/**
|
||||
* 保存文章到向量库
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Override
|
||||
public void save(Article article) {
|
||||
Map<String, Object> metadata = MapUtils.newHashMap();
|
||||
metadata.put("id", article.getId());
|
||||
metadata.put("siteCode", article.getCategory().getSite().getSiteCode());
|
||||
metadata.put("categoryCode", article.getCategory().getCategoryCode());
|
||||
metadata.put("categoryName", article.getCategory().getCategoryName());
|
||||
metadata.put("title", article.getTitle());
|
||||
metadata.put("href", article.getHref());
|
||||
metadata.put("keywords", article.getKeywords());
|
||||
metadata.put("description", article.getDescription());
|
||||
metadata.put("url", article.getUrl());
|
||||
metadata.put("status", article.getStatus());
|
||||
metadata.put("createBy", article.getCreateBy());
|
||||
metadata.put("createDate", article.getCreateDate());
|
||||
metadata.put("updateBy", article.getUpdateBy());
|
||||
metadata.put("updateDate", article.getUpdateDate());
|
||||
String content = article.getTitle() + ", " + article.getKeywords() + ", "
|
||||
+ article.getDescription() + ", " + StringUtils.toMobileHtml(
|
||||
article.getArticleData().getContent());
|
||||
String markdown = FlexmarkHtmlConverter.builder().build().convert(content);
|
||||
List<Document> documents = List.of(new Document(article.getId(), markdown, metadata));
|
||||
List<Document> splitDocuments = new TokenTextSplitter().apply(documents);
|
||||
this.delete(article); // 删除原数据
|
||||
ListUtils.pageList(splitDocuments, 64, params -> {
|
||||
vectorStore.add((List<Document>)params[0]); // 增加新数据
|
||||
return null;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除向量库文章
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Override
|
||||
public void delete(Article article) {
|
||||
if (StringUtils.isNotBlank(article.getId())) {
|
||||
vectorStore.delete(new FilterExpressionBuilder().eq("id", article.getId()).build());
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 重建向量库文章
|
||||
* @author ThinkGem
|
||||
*/
|
||||
public String rebuild(Article article) {
|
||||
logger.debug("开始重建向量库。 siteCode: {}, categoryCode: {}",
|
||||
article.getCategory().getSite().getSiteCode(),
|
||||
article.getCategory().getCategoryCode());
|
||||
long start = System.currentTimeMillis();
|
||||
try{
|
||||
article.setIsQueryArticleData(true); // 查询文章内容
|
||||
PageUtils.findList(article, null, e -> {
|
||||
List<Article> list = CmsUtils.getArticleService().findList((Article) e);
|
||||
if (!list.isEmpty()) {
|
||||
list.forEach(this::save);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
});
|
||||
}catch(Exception ex){
|
||||
logger.error("重建向量库失败", ex);
|
||||
return "重建向量库失败:" + ex.getMessage();
|
||||
}
|
||||
String message = "重建向量库完成! 用时" + TimeUtils.formatTime(System.currentTimeMillis() - start) + "。";
|
||||
logger.debug(message);
|
||||
return message;
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
/**
|
||||
* Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
* No deletion without permission, or be held responsible to law.
|
||||
*/
|
||||
package com.jeesite.modules.cms.ai.service;
|
||||
|
||||
import com.jeesite.common.cache.CacheUtils;
|
||||
import com.jeesite.common.collect.ListUtils;
|
||||
import org.springframework.ai.chat.memory.ChatMemory;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.stereotype.Service;
|
||||
|
||||
import java.util.List;
|
||||
|
||||
/**
|
||||
* AI 对话消息存储
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Service
|
||||
public class CacheChatMemory implements ChatMemory {
|
||||
|
||||
private static final String CMS_CHAT_MSG_CACHE = "cmsChatMsgCache";
|
||||
|
||||
@Override
|
||||
public void add(String conversationId, List<Message> messages) {
|
||||
List<Message> conversationHistory = CacheUtils.get(CMS_CHAT_MSG_CACHE, conversationId);
|
||||
if (conversationHistory == null) {
|
||||
conversationHistory = ListUtils.newArrayList();
|
||||
}
|
||||
conversationHistory.addAll(messages);
|
||||
CacheUtils.put(CMS_CHAT_MSG_CACHE, conversationId, conversationHistory);
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Message> get(String conversationId, int lastN) {
|
||||
List<Message> all = CacheUtils.get(CMS_CHAT_MSG_CACHE, conversationId);
|
||||
return all != null ? all.stream().skip(Math.max(0, all.size() - lastN)).toList() : List.of();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void clear(String conversationId) {
|
||||
CacheUtils.remove(CMS_CHAT_MSG_CACHE, conversationId);
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
/**
|
||||
* Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
* No deletion without permission, or be held responsible to law.
|
||||
*/
|
||||
package com.jeesite.modules.cms.ai.service;
|
||||
|
||||
import com.jeesite.common.cache.CacheUtils;
|
||||
import com.jeesite.common.collect.MapUtils;
|
||||
import com.jeesite.common.idgen.IdGen;
|
||||
import com.jeesite.common.lang.DateUtils;
|
||||
import com.jeesite.common.lang.StringUtils;
|
||||
import com.jeesite.common.service.BaseService;
|
||||
import com.jeesite.modules.sys.utils.UserUtils;
|
||||
import org.springframework.ai.chat.client.ChatClient;
|
||||
import org.springframework.ai.chat.client.advisor.MessageChatMemoryAdvisor;
|
||||
import org.springframework.ai.chat.client.advisor.QuestionAnswerAdvisor;
|
||||
import org.springframework.ai.chat.memory.ChatMemory;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.ai.chat.messages.UserMessage;
|
||||
import org.springframework.ai.chat.model.ChatResponse;
|
||||
import org.springframework.ai.vectorstore.SearchRequest;
|
||||
import org.springframework.ai.vectorstore.VectorStore;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.stereotype.Service;
|
||||
import reactor.core.publisher.Flux;
|
||||
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* AI 聊天服务类
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@Service
|
||||
public class CmsAiChatService extends BaseService {
|
||||
|
||||
private static final String CMS_CHAT_CACHE = "cmsChatCache";
|
||||
|
||||
@Autowired
|
||||
private ChatClient chatClient;
|
||||
@Autowired
|
||||
private ChatMemory chatMemory;
|
||||
@Autowired
|
||||
private VectorStore vectorStore;
|
||||
|
||||
/**
|
||||
* 获取聊天对话消息
|
||||
* @author ThinkGem
|
||||
*/
|
||||
public List<Message> getChatMessage(String conversationId) {
|
||||
return chatMemory.get(conversationId, 100);
|
||||
}
|
||||
|
||||
private static String getChatCacheKey() {
|
||||
String key = UserUtils.getUser().getId();
|
||||
if (StringUtils.isBlank(key)) {
|
||||
key = UserUtils.getSession().getId().toString();
|
||||
}
|
||||
return key;
|
||||
}
|
||||
|
||||
public Map<String, Map<String, Object>> getChatCacheMap() {
|
||||
Map<String, Map<String, Object>> cache = CacheUtils.get(CMS_CHAT_CACHE, getChatCacheKey());
|
||||
if (cache == null) {
|
||||
cache = MapUtils.newHashMap();
|
||||
}
|
||||
return cache;
|
||||
}
|
||||
|
||||
/**
|
||||
* 新建或更新聊天对话
|
||||
* @author ThinkGem
|
||||
*/
|
||||
public Map<String, Object> saveChatConversation(String conversationId, String title) {
|
||||
if (StringUtils.isBlank(conversationId)) {
|
||||
conversationId = IdGen.nextId();
|
||||
}
|
||||
if (StringUtils.isBlank(title)) {
|
||||
title = "新对话 " + DateUtils.getTime();
|
||||
}
|
||||
Map<String, Object> map = MapUtils.newHashMap();
|
||||
map.put("id", conversationId);
|
||||
map.put("title", title);
|
||||
Map<String, Map<String, Object>> cache = getChatCacheMap();
|
||||
cache.put(conversationId, map);
|
||||
CacheUtils.put(CMS_CHAT_CACHE, getChatCacheKey(), cache);
|
||||
return map;
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除聊天对话
|
||||
* @author ThinkGem
|
||||
*/
|
||||
public void deleteChatConversation(String conversationId) {
|
||||
Map<String, Map<String, Object>> cache = getChatCacheMap();
|
||||
cache.remove(conversationId);
|
||||
CacheUtils.put(CMS_CHAT_CACHE, getChatCacheKey(), cache);
|
||||
chatMemory.clear(conversationId);
|
||||
}
|
||||
|
||||
/**
|
||||
* 聊天对话,流输出
|
||||
* @author ThinkGem
|
||||
*/
|
||||
public Flux<ChatResponse> chatStream(String conversationId, String message) {
|
||||
return chatClient.prompt()
|
||||
.messages(new UserMessage(message))
|
||||
.advisors(
|
||||
new MessageChatMemoryAdvisor(chatMemory, conversationId, 1024),
|
||||
new QuestionAnswerAdvisor(vectorStore, SearchRequest.builder().similarityThreshold(0.6F).topK(6).build()))
|
||||
.stream()
|
||||
.chatResponse();
|
||||
}
|
||||
|
||||
}
|
||||
@@ -0,0 +1,84 @@
|
||||
/**
|
||||
* Copyright (c) 2013-Now http://jeesite.com All rights reserved.
|
||||
* No deletion without permission, or be held responsible to law.
|
||||
*/
|
||||
package com.jeesite.modules.cms.ai.web;
|
||||
|
||||
import com.jeesite.common.config.Global;
|
||||
import com.jeesite.common.web.BaseController;
|
||||
import com.jeesite.modules.cms.ai.service.CmsAiChatService;
|
||||
import org.springframework.ai.chat.messages.Message;
|
||||
import org.springframework.ai.chat.model.ChatResponse;
|
||||
import org.springframework.beans.factory.annotation.Autowired;
|
||||
import org.springframework.http.MediaType;
|
||||
import org.springframework.web.bind.annotation.RequestMapping;
|
||||
import org.springframework.web.bind.annotation.RestController;
|
||||
import reactor.core.publisher.Flux;
|
||||
|
||||
import java.util.Collection;
|
||||
import java.util.Comparator;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
import java.util.stream.Collectors;
|
||||
|
||||
/**
|
||||
* AI 聊天控制器类
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RestController
|
||||
@RequestMapping("${adminPath}/cms/chat")
|
||||
public class CmsAiChatController extends BaseController {
|
||||
|
||||
@Autowired
|
||||
private CmsAiChatService cmsAiChatService;
|
||||
|
||||
/**
|
||||
* 获取聊天对话消息
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RequestMapping("/message")
|
||||
public List<Message> message(String id) {
|
||||
return cmsAiChatService.getChatMessage(id);
|
||||
}
|
||||
|
||||
/**
|
||||
* 聊天对话列表
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RequestMapping("/list")
|
||||
public Collection<Map<String, Object>> list() {
|
||||
return cmsAiChatService.getChatCacheMap().values().stream()
|
||||
.sorted(Comparator.comparing(map -> (String) map.get("id"),
|
||||
Comparator.reverseOrder())).collect(Collectors.toList());
|
||||
}
|
||||
|
||||
/**
|
||||
* 新建或更新聊天对话
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RequestMapping("/save")
|
||||
public String save(String id, String title) {
|
||||
Map<String, Object> map = cmsAiChatService.saveChatConversation(id, title);
|
||||
return renderResult(Global.TRUE, "保存成功", map);
|
||||
}
|
||||
|
||||
/**
|
||||
* 删除聊天对话
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RequestMapping("/delete")
|
||||
public String delete(String id) {
|
||||
cmsAiChatService.deleteChatConversation(id);
|
||||
return renderResult(Global.TRUE, "删除成功", id);
|
||||
}
|
||||
|
||||
/**
|
||||
* 聊天对话,流输出
|
||||
* @author ThinkGem
|
||||
*/
|
||||
@RequestMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
|
||||
public Flux<ChatResponse> stream(String id, String message) {
|
||||
return cmsAiChatService.chatStream(id, message);
|
||||
}
|
||||
|
||||
}
|
||||
12
modules/cms-ai/src/main/resources/application-assistant.yml
Normal file
12
modules/cms-ai/src/main/resources/application-assistant.yml
Normal file
@@ -0,0 +1,12 @@
|
||||
|
||||
## 重要提示(Tip):
|
||||
|
||||
## 请勿在该配置文件中添加其它任何配置(添加也不会生效)。
|
||||
## 该文件,仅仅是为了让 jeesite-cms-ai.yml 文件,
|
||||
## 在 IDEA 中有一个自动完成及帮助提示,并无其它用意。
|
||||
## 参数配置请在 jeesite-cms-ai.yml 文件中添加。
|
||||
|
||||
spring:
|
||||
config:
|
||||
import:
|
||||
- classpath:config/jeesite-cms-ai.yml
|
||||
125
modules/cms-ai/src/main/resources/config/jeesite-cms-ai.yml
Normal file
125
modules/cms-ai/src/main/resources/config/jeesite-cms-ai.yml
Normal file
@@ -0,0 +1,125 @@
|
||||
# 温馨提示:不建议直接修改此文件,为了平台升级方便,建议将需要修改的参数值,复制到application.yml里进行覆盖该参数值。
|
||||
|
||||
spring:
|
||||
ai:
|
||||
|
||||
# 云上大模型(使用该模型,请开启 enabled 参数)
|
||||
openai:
|
||||
base-url: https://api.siliconflow.cn
|
||||
api-key: ${SFLOW_APP_KEY}
|
||||
#base-url: https://ai.gitee.com
|
||||
#api-key: ${GITEE_APP_KEY}
|
||||
# 聊天对话模型
|
||||
chat:
|
||||
enabled: true
|
||||
options:
|
||||
model: deepseek-ai/DeepSeek-R1-Distill-Qwen-7B
|
||||
#model: DeepSeek-R1-Distill-Qwen-14B
|
||||
max-tokens: 1024
|
||||
temperature: 0.6
|
||||
top-p: 0.7
|
||||
frequency-penalty: 0
|
||||
logprobs: true
|
||||
# 向量库知识库模型(注意:不同的模型维度不同)
|
||||
embedding:
|
||||
enabled: true
|
||||
options:
|
||||
model: BAAI/bge-m3
|
||||
#model: bge-large-zh-v1.5
|
||||
dimensions: 512
|
||||
|
||||
# 本地大模型配置(使用该模型,请开启 enabled 参数)
|
||||
ollama:
|
||||
base-url: http://localhost:11434
|
||||
# 聊天对话模型
|
||||
chat:
|
||||
enabled: false
|
||||
options:
|
||||
#model: qwen2.5
|
||||
model: deepseek-r1:7b
|
||||
max-tokens: 1024
|
||||
temperature: 0.6
|
||||
top-p: 0.7
|
||||
frequency-penalty: 0
|
||||
# 向量库知识库模型(注意:不同的模型维度不同)
|
||||
embedding:
|
||||
enabled: false
|
||||
# 维度 dimensions 设置为 384
|
||||
#model: all-minilm:33m
|
||||
# 维度 dimensions 设置为 768
|
||||
#model: nomic-embed-text
|
||||
# 维度 dimensions 设置为 1024
|
||||
model: bge-m3
|
||||
|
||||
# 向量数据库配置
|
||||
vectorstore:
|
||||
|
||||
# Postgresql 向量数据库(PG 连接配置,见下文,需要手动建表)
|
||||
pgvector:
|
||||
initialize-schema: false
|
||||
id-type: TEXT
|
||||
index-type: HNSW
|
||||
distance-type: COSINE_DISTANCE
|
||||
#table-name: vector_store_384
|
||||
#dimensions: 384
|
||||
#table-name: vector_store_786
|
||||
#dimensions: 768
|
||||
table-name: vector_store_1024
|
||||
dimensions: 1024
|
||||
batching-strategy: TOKEN_COUNT
|
||||
max-document-batch-size: 10000
|
||||
|
||||
# # ES 向量数据库(ES 连接配置,见下文)
|
||||
# elasticsearch:
|
||||
# initialize-schema: true
|
||||
# index-name: vector-index
|
||||
# dimensions: 1024
|
||||
# similarity: cosine
|
||||
# batching-strategy: TOKEN_COUNT
|
||||
|
||||
# # Milvus 向量数据库(字符串长度不超过65535)
|
||||
# milvus:
|
||||
# initialize-schema: true
|
||||
# client:
|
||||
# host: "localhost"
|
||||
# port: 19530
|
||||
# username: "root"
|
||||
# password: "milvus"
|
||||
# database-name: "default2"
|
||||
# collection-name: "vector_store2"
|
||||
# embedding-dimension: 384
|
||||
# index-type: HNSW
|
||||
# metric-type: COSINE
|
||||
|
||||
# ========= Postgresql 向量数据库数据源 =========
|
||||
|
||||
jdbc:
|
||||
ds_pgvector:
|
||||
type: postgresql
|
||||
driver: org.postgresql.Driver
|
||||
url: jdbc:postgresql://127.0.0.1:5433/jeesite-ai
|
||||
username: postgres
|
||||
password: postgres
|
||||
testSql: SELECT 1
|
||||
|
||||
# ========= ES 向量数据库连接配置 =========
|
||||
|
||||
spring.elasticsearch:
|
||||
enabled: true
|
||||
socket-timeout: 120s
|
||||
connection-timeout: 120s
|
||||
uris: http://127.0.0.1:9200
|
||||
username: elastic
|
||||
password: elastic
|
||||
|
||||
# 对话消息存缓存,可自定义存数据库
|
||||
j2cache:
|
||||
caffeine:
|
||||
region:
|
||||
# 对话消息的超期时间,默认 30天,根据需要可以设置更久。
|
||||
cmsChatCache: 100000, 30d
|
||||
cmsChatMsgCache: 100000, 30d
|
||||
|
||||
#logging:
|
||||
# level:
|
||||
# org.springframework: debug
|
||||
@@ -0,0 +1 @@
|
||||
5.11.0
|
||||
@@ -16,6 +16,7 @@
|
||||
<module>core</module>
|
||||
<module>app</module>
|
||||
<module>cms</module>
|
||||
<module>cms-ai</module>
|
||||
<module>static</module>
|
||||
<module>test</module>
|
||||
</modules>
|
||||
|
||||
@@ -66,6 +66,13 @@
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency>
|
||||
|
||||
<!-- 内容管理 AI + RAG 模块
|
||||
<dependency>
|
||||
<groupId>com.jeesite</groupId>
|
||||
<artifactId>jeesite-module-cms-ai</artifactId>
|
||||
<version>${project.parent.version}</version>
|
||||
</dependency> -->
|
||||
|
||||
<!-- 内容管理-页面静态化(标准版)
|
||||
<dependency>
|
||||
<groupId>com.jeesite</groupId>
|
||||
|
||||
Reference in New Issue
Block a user