Merge old pr (#235)

* 新增openai tts (#137)

On branch openai_tts
Changes to be committed:
	modified:   config.yaml
	new file:   core/providers/tts/openai_tts.py

Co-authored-by: 欣南科技 <huangrongzhuang@xin-nan.com>

* Update config.yaml (#201)

* 修改了coze的实现方式,使用了v3接口,修复了只能对话一次的bug

* 使用mem0ai api实现记忆功能

* Fix docker space (#179)

* update:调试docker编译缓存

* update:调试docker编译

* update:调试docker编译容量

* update:调试docker编译

---------

Co-authored-by: hrz <1710360675@qq.com>

* TTS处理增加index信息，完善TTS出错时的处理 (#177)

* TTS处理增加index信息，完善TTS出错时的处理

* 第一句语音index处理，歌曲播放index处理

* update:去掉了无用变量，修复continuebug

---------

Co-authored-by: hrz <1710360675@qq.com>

* update:更新demo (#180)

Co-authored-by: hrz <1710360675@qq.com>

* 优化chat中异步调用方式

* 记忆增加时间，以便大模型判断先后关系

* 记忆对时间戳排序，便于梳理前后关系

* add: 不跟踪music下的文件的更改 (#187)

* Mem0ai (#199)

* update:增加mem0ai包依赖

* update:文档增加mem0ai的描述

---------

Co-authored-by: hrz <1710360675@qq.com>

* Update config.yaml

* Update config.yaml

* Update config.yaml

* Update config.yaml

---------

Co-authored-by: strix214 <a1425699467@gmail.com>
Co-authored-by: 玄凤科技 <eric230308@gmail.com>
Co-authored-by: HonestQiao <honestqiao@163.com>
Co-authored-by: 欣南科技 <huangrongzhuang@xin-nan.com>
Co-authored-by: hrz <1710360675@qq.com>
Co-authored-by: TOM88812 <ttangxs0808@163.com>
Co-authored-by: 香草味的纳西妲 <151599587+NyaOH-Nahida@users.noreply.github.com>

* update:添加忽略文件

* feat: 增加fastgpt适配：支持think、variables参数 (#213)

* 修改了coze的实现方式,使用了v3接口,修复了只能对话一次的bug

* 使用mem0ai api实现记忆功能

* Fix docker space (#179)

* update:调试docker编译缓存

* update:调试docker编译

* update:调试docker编译容量

* update:调试docker编译

---------

Co-authored-by: hrz <1710360675@qq.com>

* TTS处理增加index信息，完善TTS出错时的处理 (#177)

* TTS处理增加index信息，完善TTS出错时的处理

* 第一句语音index处理，歌曲播放index处理

* update:去掉了无用变量，修复continuebug

---------

Co-authored-by: hrz <1710360675@qq.com>

* update:更新demo (#180)

Co-authored-by: hrz <1710360675@qq.com>

* 优化chat中异步调用方式

* 记忆增加时间，以便大模型判断先后关系

* 记忆对时间戳排序，便于梳理前后关系

* add: 不跟踪music下的文件的更改 (#187)

* Mem0ai (#199)

* update:增加mem0ai包依赖

* update:文档增加mem0ai的描述

---------

Co-authored-by: hrz <1710360675@qq.com>

* feat: 增加fastgpt适配：支持think、variables参数

---------

Co-authored-by: strix214 <a1425699467@gmail.com>
Co-authored-by: 玄凤科技 <eric230308@gmail.com>
Co-authored-by: HonestQiao <honestqiao@163.com>
Co-authored-by: 欣南科技 <huangrongzhuang@xin-nan.com>
Co-authored-by: hrz <1710360675@qq.com>
Co-authored-by: TOM88812 <ttangxs0808@163.com>
Co-authored-by: 香草味的纳西妲 <151599587+NyaOH-Nahida@users.noreply.github.com>
Co-authored-by: pt <ptsghs@163.com>

* update：合并最新代码

* update:合并群友pr

* update:openaitts 测试通过

* update:ACGNTTS测试完成

---------

Co-authored-by: Chris <119588753+Chris-websketch@users.noreply.github.com>
Co-authored-by: ikun441 <3497689533@qq.com>
Co-authored-by: strix214 <a1425699467@gmail.com>
Co-authored-by: 玄凤科技 <eric230308@gmail.com>
Co-authored-by: HonestQiao <honestqiao@163.com>
Co-authored-by: hrz <1710360675@qq.com>
Co-authored-by: TOM88812 <ttangxs0808@163.com>
Co-authored-by: 香草味的纳西妲 <151599587+NyaOH-Nahida@users.noreply.github.com>
Co-authored-by: ptisnoob <49125305+ptisnoob@users.noreply.github.com>
Co-authored-by: pt <ptsghs@163.com>

main/xiaozhi-server/config.yaml

@@ -158,6 +158,15 @@ LLM:
     base_url: http://homeassistant.local:8123
     agent_id: conversation.chatgpt
     api_key: 你的home assistant api访问令牌
+  FastgptLLM:
+    # 定义LLM API类型
+    type: fastgpt
+    # 如果使用fastgpt，配置文件里prompt(提示词)是无效的，需要在fastgpt控制台设置提示词
+    base_url: https://host/api/v1
+    api_key: fastgpt-xxx
+    variables:
+      k: "v"
+      k2: "v2"
 TTS:
   # 当前支持的type为edge、doubao，可自行适配
   EdgeTTS:
@@ -316,6 +325,36 @@ TTS:
     voice: "zh_female_wanwanxiaohe_moon_bigtts"
     output_file: tmp/
     access_token: "你的302API密钥"
+  ACGNTTS:
+    #在线网址：https://acgn.ttson.cn/
+    #token购买：www.ttson.cn
+    #开发相关疑问请提交至3497689533@qq.com
+    #角色id获取地址：ctrl+f快速检索角色——网站管理者不允许发布,可询问网站管理者：1069379506
+    #各参数意义见开发文档：https://www.yuque.com/alexuh/skmti9/wm6taqislegb02gd?singleDoc#
+    type: ttson
+    token: your_token
+    voice_id: 1695
+    speed_factor: 1
+    pitch_factor: 0
+    volume_change_dB: 0
+    to_lang: ZH
+    url: https://u95167-bd74-2aef8085.westx.seetacloud.com:8443/flashsummary/tts?token=
+    format: mp3
+    output_file: tmp/
+    emotion: 1
+  OpenAITTS:
+    # openai官方文本转语音服务，可支持全球大多数语种
+    type: openai
+    api_key: 你的openai api key
+    # 国内需要使用代理
+    api_url: https://api.openai.com/v1/audio/speech
+    # 可选tts-1或tts-1-hd，tts-1速度更快tts-1-hd质量更好
+    model: tts-1
+    # 演讲者，可选alloy, echo, fable, onyx, nova, shimmer
+    voice: onyx
+    # 语速范围0.25-4.0
+    speed: 1
+    output_file: tmp/
 # 模块测试配置
 module_test:
   test_sentences:  # 自定义测试语句

main/xiaozhi-server/core/providers/llm/fastgpt/fastgpt.py

@@ -0,0 +1,65 @@
+import json
+from config.logger import setup_logging
+import requests
+from core.providers.llm.base import LLMProviderBase
+
+TAG = __name__
+logger = setup_logging()
+
+
+class LLMProvider(LLMProviderBase):
+    def __init__(self, config):
+        self.api_key = config["api_key"]
+        self.base_url = config.get("base_url")
+        self.detail = config.get("detail", False)
+        self.variables = config.get("variables", {})
+
+    def response(self, session_id, dialogue):
+        try:
+            # 取最后一条用户消息
+            last_msg = next(m for m in reversed(dialogue) if m["role"] == "user")
+
+            # 发起流式请求
+            with requests.post(
+                    f"{self.base_url}/chat/completions",
+                    headers={"Authorization": f"Bearer {self.api_key}"},
+                    json={
+                        "stream": True,
+                        "chatId": session_id,
+                        "detail": self.detail,
+                        "variables": self.variables,
+                        "messages": [
+                            {
+                                "role": "user",
+                                "content": last_msg["content"]
+                            }
+                        ]
+                    },
+                    stream=True
+            ) as r:
+                for line in r.iter_lines():
+                    if line:
+                        try:
+                            if line.startswith(b'data: '):
+                                if line[6:].decode('utf-8') == '[DONE]':
+                                    break
+
+                                data = json.loads(line[6:])
+                                if 'choices' in data and len(data['choices']) > 0:
+                                    delta = data['choices'][0].get('delta', {})
+                                    if delta and 'content' in delta and delta['content'] is not None:
+                                        content = delta['content']
+                                        if '<think>' in content:
+                                            continue
+                                        if '</think>' in content:
+                                            continue
+                                        yield content
+
+                        except json.JSONDecodeError as e:
+                            continue
+                        except Exception as e:
+                            continue
+
+        except Exception as e:
+            logger.bind(tag=TAG).error(f"Error in response generation: {e}")
+            yield "【服务响应异常】"

main/xiaozhi-server/core/providers/llm/gemini/gemini.py

@@ -1,25 +1,24 @@
-from config.logger import setup_logging
 import google.generativeai as genai
+from core.utils.util import check_model_key
 from core.providers.llm.base import LLMProviderBase
 
-TAG = __name__
-logger = setup_logging()
 
 class LLMProvider(LLMProviderBase):
     def __init__(self, config):
         """初始化Gemini LLM Provider"""
-        self.model_name = config.get("model_name", "gemini-1.5-pro") 
+        self.model_name = config.get("model_name", "gemini-1.5-pro")
         self.api_key = config.get("api_key")
-        
+
-        if not self.api_key or "你" in self.api_key:
+        have_key = check_model_key("LLM", self.api_key)
-            logger.bind(tag=TAG).error("你还没配置Gemini LLM的密钥，请在配置文件中配置密钥，否则无法正常工作")
+
+        if not have_key:
             return
-            
+
         try:
             # 初始化Gemini客户端
             genai.configure(api_key=self.api_key)
             self.model = genai.GenerativeModel(self.model_name)
-            
+
             # 设置生成参数
             self.generation_config = {
                 "temperature": 0.7,
@@ -55,7 +54,7 @@ class LLMProvider(LLMProviderBase):
 
             # 创建新的聊天会话
             chat = self.model.start_chat(history=chat_history)
-            
+
             # 发送消息并获取流式响应
             response = chat.send_message(
                 current_msg,
@@ -71,7 +70,7 @@ class LLMProvider(LLMProviderBase):
         except Exception as e:
             error_msg = str(e)
             logger.bind(tag=TAG).error(f"Gemini响应生成错误: {error_msg}")
-            
+
             # 针对不同错误返回友好提示
             if "Rate limit" in error_msg:
                 yield "【Gemini服务请求太频繁,请稍后再试】"

main/xiaozhi-server/core/providers/llm/openai/openai.py

@@ -1,10 +1,7 @@
-from config.logger import setup_logging
 import openai
+from core.utils.util import check_model_key
 from core.providers.llm.base import LLMProviderBase
 
-TAG = __name__
-logger = setup_logging()
-
 
 class LLMProvider(LLMProviderBase):
     def __init__(self, config):
@@ -14,8 +11,7 @@ class LLMProvider(LLMProviderBase):
             self.base_url = config.get("base_url")
         else:
             self.base_url = config.get("url")
-        if "你" in self.api_key:
+        check_model_key("LLM", self.api_key)
-            logger.bind(tag=TAG).error("你还没配置LLM的密钥，请在配置文件中配置密钥，否则无法正常工作")
         self.client = openai.OpenAI(api_key=self.api_key, base_url=self.base_url)
 
     def response(self, session_id, dialogue):

main/xiaozhi-server/core/providers/memory/mem0ai/mem0ai.py

@@ -1,5 +1,6 @@
 from ..base import MemoryProviderBase, logger
 from mem0 import MemoryClient
+from core.utils.util import check_model_key
 
 TAG = __name__
 
@@ -8,8 +9,8 @@ class MemoryProvider(MemoryProviderBase):
         super().__init__(config)
         self.api_key = config.get("api_key", "")
         self.api_version = config.get("api_version", "v1.1")
-        if len(self.api_key) == 0 or "你" in self.api_key:
+        have_key = check_model_key("Mem0ai", self.api_key)
-            logger.bind(tag=TAG).error("你还没配置Mem0ai的密钥，请在配置文件中配置密钥，否则无法提供记忆服务")
+        if not have_key :
             self.use_mem0 = False
             return
         else:

main/xiaozhi-server/core/providers/tts/doubao.py

@@ -4,6 +4,7 @@ import json
 import base64
 import requests
 from datetime import datetime
+from core.utils.util import check_model_key
 from core.providers.tts.base import TTSProviderBase
 
 
@@ -17,6 +18,7 @@ class TTSProvider(TTSProviderBase):
         self.api_url = config.get("api_url")
         self.authorization = config.get("authorization")
         self.header = {"Authorization": f"{self.authorization}{self.access_token}"}
+        check_model_key("TTS", self.access_token)
 
     def generate_filename(self, extension=".wav"):
         return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")

main/xiaozhi-server/core/providers/tts/fishspeech.py

@@ -1,4 +1,3 @@
-
 import base64
 import os
 import uuid
@@ -9,6 +8,7 @@ from pydantic import BaseModel, Field, conint, model_validator
 from typing_extensions import Annotated
 from datetime import datetime
 from typing import Literal
+from core.utils.util import check_model_key
 from core.providers.tts.base import TTSProviderBase
 from config.logger import setup_logging
 
@@ -24,7 +24,7 @@ class ServeReferenceAudio(BaseModel):
     def decode_audio(cls, values):
         audio = values.get("audio")
         if (
-            isinstance(audio, str) and len(audio) > 255
+                isinstance(audio, str) and len(audio) > 255
         ):  # Check if audio is a string (Base64)
             try:
                 values["audio"] = base64.b64decode(audio)
@@ -36,6 +36,7 @@ class ServeReferenceAudio(BaseModel):
     def __repr__(self) -> str:
         return f"ServeReferenceAudio(text={self.text!r}, audio_size={len(self.audio)})"
 
+
 class ServeTTSRequest(BaseModel):
     text: str
     chunk_length: Annotated[int, conint(ge=100, le=300, strict=True)] = 200
@@ -70,6 +71,7 @@ def audio_to_bytes(file_path):
         wav = wav_file.read()
     return wav
 
+
 def read_ref_text(ref_text):
     path = Path(ref_text)
     if path.exists() and path.is_file():
@@ -77,31 +79,32 @@ def read_ref_text(ref_text):
             return file.read()
     return ref_text
 
+
 class TTSProvider(TTSProviderBase):
 
     def __init__(self, config, delete_audio_file):
         super().__init__(config, delete_audio_file)
 
         self.reference_id = config.get("reference_id")
-        self.reference_audio = config.get("reference_audio",[]) 
+        self.reference_audio = config.get("reference_audio", [])
-        self.reference_text = config.get("reference_text",[])
+        self.reference_text = config.get("reference_text", [])
-        self.format =  config.get("format","wav")
+        self.format = config.get("format", "wav")
-        self.channels = config.get("channels",1)
+        self.channels = config.get("channels", 1)
-        self.rate = config.get("rate",44100)
+        self.rate = config.get("rate", 44100)
-        self.api_key = config.get("api_key","YOUR_API_KEY")
+        self.api_key = config.get("api_key", "YOUR_API_KEY")
-        if "你" in self.api_key:
+        have_key = check_model_key("FishSpeech TTS", self.api_key)
-            logger.bind(tag=TAG).error("你还没配置FishSpeech TTS的密钥，请在配置文件中配置密钥，否则无法正常工作")
+        if not have_key:
             return
-        self.normalize = config.get("normalize",True)
+        self.normalize = config.get("normalize", True)
-        self.max_new_tokens = config.get("max_new_tokens",1024)
+        self.max_new_tokens = config.get("max_new_tokens", 1024)
-        self.chunk_length = config.get("chunk_length",200)
+        self.chunk_length = config.get("chunk_length", 200)
-        self.top_p = config.get("top_p",0.7)
+        self.top_p = config.get("top_p", 0.7)
-        self.repetition_penalty = config.get("repetition_penalty",1.2)
+        self.repetition_penalty = config.get("repetition_penalty", 1.2)
-        self.temperature = config.get("temperature",0.7)
+        self.temperature = config.get("temperature", 0.7)
-        self.streaming = config.get("streaming",False)
+        self.streaming = config.get("streaming", False)
-        self.use_memory_cache = config.get("use_memory_cache","on")
+        self.use_memory_cache = config.get("use_memory_cache", "on")
         self.seed = config.get("seed")
-        self.api_url =   config.get("api_url","http://127.0.0.1:8080/v1/tts")
+        self.api_url = config.get("api_url", "http://127.0.0.1:8080/v1/tts")
 
     def generate_filename(self, extension=".wav"):
         return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
@@ -148,11 +151,9 @@ class TTSProvider(TTSProviderBase):
 
             with open(output_file, "wb") as audio_file:
                 audio_file.write(audio_content)
-                
+
 
 
         else:
             print(f"Request failed with status code {response.status_code}")
             print(response.json())
-
-

main/xiaozhi-server/core/providers/tts/openai.py

@@ -0,0 +1,40 @@
+import os
+import uuid
+import requests
+from datetime import datetime
+from core.utils.util import check_model_key
+from core.providers.tts.base import TTSProviderBase
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+        self.api_key = config.get("api_key")
+        self.api_url = config.get("api_url", "https://api.openai.com/v1/audio/speech")
+        self.model = config.get("model", "tts-1")
+        self.voice = config.get("voice", "alloy")
+        self.response_format = "wav"
+        self.speed = config.get("speed", 1.0)
+        self.output_file = config.get("output_file", "tmp/")
+        check_model_key("TTS", self.api_key)
+
+    def generate_filename(self, extension=".wav"):
+        return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
+
+    async def text_to_speak(self, text, output_file):
+        headers = {
+            "Authorization": f"Bearer {self.api_key}",
+            "Content-Type": "application/json"
+        }
+        data = {
+            "model": self.model,
+            "input": text,
+            "voice": self.voice,
+            "response_format": "wav",
+            "speed": self.speed
+        }
+        response = requests.post(self.api_url, json=data, headers=headers)
+        if response.status_code == 200:
+            with open(output_file, "wb") as audio_file:
+                audio_file.write(response.content)
+        else:
+            raise Exception(f"OpenAI TTS请求失败: {response.status_code} - {response.text}")

main/xiaozhi-server/core/providers/tts/ttson.py

@@ -0,0 +1,64 @@
+import os
+import uuid
+import json
+import requests
+import shutil
+from datetime import datetime
+from core.providers.tts.base import TTSProviderBase
+
+
+class TTSProvider(TTSProviderBase):
+    def __init__(self, config, delete_audio_file):
+        super().__init__(config, delete_audio_file)
+        self.url = config.get("url", "https://u95167-bd74-2aef8085.westx.seetacloud.com:8443/flashsummary/tts?token=")
+        self.voice_id = config.get("voice_id", 1695)
+        self.token = config.get("token")
+        self.to_lang = config.get("to_lang")
+        self.volume_change_dB = config.get("volume_change_dB", 0)
+        self.speed_factor = config.get("speed_factor", 1)
+        self.stream = config.get("stream", False)
+        self.output_file = config.get("output_file")
+        self.pitch_factor = config.get("pitch_factor", 0)
+        self.format = config.get("format", "mp3")
+        self.emotion = config.get("emotion", 1)
+        self.header = {
+            "Content-Type": "application/json"
+        }
+
+    def generate_filename(self, extension=".mp3"):
+        return os.path.join(self.output_file, f"tts-{datetime.now().date()}@{uuid.uuid4().hex}{extension}")
+
+    async def text_to_speak(self, text, output_file):
+        url = f'{self.url}{self.token}'
+        result = "firefly"
+        payload = json.dumps({
+            "to_lang": self.to_lang,
+            "text": text,
+            "emotion": self.emotion,
+            "format": self.format,
+            "volume_change_dB": self.volume_change_dB,
+            "voice_id": self.voice_id,
+            "pitch_factor": self.pitch_factor,
+            "speed_factor": self.speed_factor,
+            "token": self.token
+        })
+
+        resp = requests.request("POST", url, data=payload)
+        if resp.status_code != 200:
+            return None
+        resp_json = resp.json()
+        try:
+            result = resp_json['url'] + ':' + str(
+                resp_json[
+                    'port']) + '/flashsummary/retrieveFileData?stream=True&token=' + self.token + '&voice_audio_path=' + \
+                     resp_json['voice_path']
+        except Exception as e:
+            print("error:", e)
+
+        audio_content = requests.get(result)
+        with open(output_file, "wb") as f:
+            f.write(audio_content.content)
+            return True
+        voice_path = resp_json.get("voice_path")
+        des_path = output_file
+        shutil.move(voice_path, des_path)

main/xiaozhi-server/core/utils/util.py

@@ -1,9 +1,9 @@
 import os
-import re
 import json
 import yaml
 import socket
 import subprocess
+import logging
 
 
 def get_project_dir():
@@ -87,35 +87,13 @@ def remove_punctuation_and_length(text):
         return 0, ""
     return len(result), result
 
-
+def check_model_key(modelType, modelKey):
-def check_password(password):
+    if "你" in modelKey:
-    """
+        logging.error("你还没配置" + modelType + "的密钥，请在配置文件中配置密钥，否则无法正常工作")
-    检查密码是否满足以下条件：
-    1. 密码长度大于八位。
-    2. 密码包含英文和数字。
-    3. 密码不能包含“xiaozhi”字符。
-
-    :param password: 要检查的密码
-    :return: 如果密码满足条件，则返回True；否则返回False。
-    """
-    # 检查密码长度
-    if len(password) < 8:
         return False
-
-    # 检查是否包含英文字符和数字
-    if not re.search(r'[A-Za-z]', password) or not re.search(r'[0-9]', password):
-        return False
-
-    # 检查是否包含“xiaozhi”字符
-    if "xiaozhi" in password:
-        return False
-
-    if "1234" in password:
-        return False
-
-    # 如果满足所有条件，则返回True
     return True
 
+
 def check_ffmpeg_installed():
     ffmpeg_installed = False
     try:
@@ -140,4 +118,4 @@ def check_ffmpeg_installed():
         error_msg += "\n建议您：\n"
         error_msg += "1、按照项目的安装文档，正确进入conda环境\n"
         error_msg += "2、查阅安装文档，如何在conda环境中安装ffmpeg\n"
-        raise ValueError(error_msg)
+        raise ValueError(error_msg)