Compare commits

..

11 Commits

Author SHA1 Message Date
01792e5211 feat: bridge_server livekit 2026-04-27 10:39:21 +08:00
37110a9d05 Fix: esp32camera pixel byte order and uart-uhci compiling error (#1728)
Some checks failed
Build Boards / Determine variants to build (push) Has been cancelled
Build Boards / Build ${{ matrix.name }} (push) Has been cancelled
* Fix: uart-uhci compiling errors

* Enhance Esp32Camera functionality by adding optional byte swapping for RGB565 format. Introduce SetSwapBytes method to enable/disable byte order swapping, and update Capture method to utilize an encode buffer for improved memory management and performance during image processing.
2026-02-02 15:33:32 +08:00
796312db4c Enhance Otto Robot camera support by adding configuration for OV3660. (#1726) 2026-02-02 10:22:53 +08:00
9e1724e892 feat: add M5Stack Cardputer Adv board support (#1718)
Add support for M5Stack Cardputer Adv, a card-sized computer based on
ESP32-S3FN8 (Stamp-S3A) with the following features:

Hardware:
- MCU: ESP32-S3FN8 @ 240MHz, 8MB Flash (no PSRAM)
- Display: ST7789V2 1.14" 240x135
- Audio: ES8311 codec + NS4150B amplifier
- Keyboard: 56-key via TCA8418
- IMU: BMI270

Key configurations:
- SPI3_HOST with 3-wire SPI mode for display
- 256Hz PWM frequency for backlight (matching M5GFX)
- ES8311 with use_mclk=false (no MCLK pin)
- Display offset X=40, Y=52 for correct alignment

新增 M5Stack Cardputer Adv 开发板支持

支持基于 ESP32-S3FN8 (Stamp-S3A) 的卡片式电脑 M5Stack Cardputer Adv:

硬件规格:
- MCU: ESP32-S3FN8 @ 240MHz, 8MB Flash (无 PSRAM)
- 显示屏: ST7789V2 1.14" 240x135
- 音频: ES8311 编解码器 + NS4150B 功放
- 键盘: 56键 (TCA8418)
- IMU: BMI270

关键配置:
- 显示使用 SPI3_HOST 和 3-wire SPI 模式
- 背光 PWM 频率 256Hz (与 M5GFX 一致)
- ES8311 设置 use_mclk=false (无 MCLK 引脚)
- 显示偏移 X=40, Y=52 以正确对齐
2026-02-02 10:01:36 +08:00
0b3b98eca7 Update esp-ml307 component version to 3.6.2 to support UART DMA (#1724)
* Update esp-ml307 dependency version to ~3.6.0 in idf_component.yml

* Update .gitignore to include 'dist/' directory, add ml307 and dual_network_board source files to CMakeLists.txt, and update esp-ml307 dependency version to ~3.6.2 in idf_component.yml. Refactor CompactWifiBoard and CompactWifiBoardLCD classes to inherit from WifiBoard instead of DualNetworkBoard, simplifying network handling logic.
2026-02-02 09:53:06 +08:00
abd62648cb Implement early return in AfeWakeWord::Feed to prevent processing when detection is not running. This enhances the robustness of the wake word detection logic. (#1723) 2026-02-01 17:42:34 +08:00
0883a36537 Refactor audio channel handling and wake word detection in Application class (#1722)
- Introduced ContinueOpenAudioChannel and ContinueWakeWordInvoke methods to streamline audio channel management and wake word processing.
- Updated HandleToggleChatEvent and HandleWakeWordDetectedEvent to utilize scheduling for state changes, improving UI responsiveness.
- Simplified logic for setting listening modes based on audio channel state, enhancing code clarity and maintainability.
2026-02-01 14:55:47 +08:00
b6c61fe390 Update project version to 2.2.2, Noto fonts and emoji support. (#1720) 2026-02-01 01:04:24 +08:00
f7284a57df Enhance memory management in asset download and OTA processes by repl… (#1716)
* Enhance memory management in asset download and OTA processes by replacing static buffer allocations with dynamic memory allocation using heap capabilities. Update SPIRAM configuration values for improved memory usage. Add logging for error handling in buffer allocation failures. Introduce a new parameter in CloseAudioChannel to control goodbye message sending in MQTT and WebSocket protocols.

* Update component versions in idf_component.yml and refactor GIF decoder functions for improved performance. Bump versions for audio effects, audio codec, LED strip, and other dependencies. Change GIF read and seek functions to inline for optimization.

* Update language files to include new phrases for flight mode and connection status across multiple locales. Added translations for "FLIGHT_MODE_ON", "FLIGHT_MODE_OFF", "CONNECTION_SUCCESSFUL", and "MODEM_INIT_ERROR" in various languages, enhancing user experience and localization support.

* fix wechat display
2026-01-31 22:58:08 +08:00
96f34ec70f Refactor emoji initialization for Electron and Otto boards to use Assets system (#1704)
* otto v1.4.0 MCP

1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay

* otto v1.4.1 gif as components

gif as components

* electronBot v1.1.0 mcp

1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类

* 规范代码

1.规范代码
2.修复切换主题死机bug

* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug

* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型  bug

* refactor: Update Electron and Otto emoji display implementations

- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.

* Rename OTTO_ICON_FONT.c to otto_icon_font.c

* Rename OTTO_ICON_FONT.c to otto_icon_font.c

* refactor: Update Otto emoji display configurations and functionalities

- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.

* refactor: Update chat label long mode for Electron and Otto emoji displays

- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.

* Update Otto robot README with new actions and parameters

* Update Otto controller parameters for oscillation settings

- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.

* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.

* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml

* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.

* Enhance Otto robot functionality by adding WebSocket control server and IP address retrieval feature. Updated config to support WebSocket, and revised README to include new control options and usage examples.

* Add camera support for Otto Robot board

- Introduced configuration option to enable the Otto Robot camera in Kconfig.
- Updated config.h to define camera-related GPIO pins and settings.
- Modified config.json to include camera configuration.
- Enhanced otto_robot.cc to initialize I2C and camera components when the camera is enabled.
- Adjusted power_manager.h to manage battery updates during camera operations.
- Removed unused SetupChatLabel method from OttoEmojiDisplay class.

* Refactor Otto Robot configuration and initialization

- Removed the camera configuration option from Kconfig and related code.
- Introduced a new HardwareConfig struct to encapsulate hardware pin definitions and settings.
- Updated config.h to define camera and non-camera configurations using the new struct.
- Refactored otto_controller.cc and otto_robot.cc to utilize the HardwareConfig struct for initialization.
- Enhanced camera detection and initialization logic based on hardware version.
- Improved audio codec initialization based on configuration settings.

* Refactor emoji initialization for Electron and Otto boards to use Assets system

- Removed direct emoji initialization from `InitializeElectronEmojis` and `InitializeOttoEmojis` methods, delegating the responsibility to the Assets system.
- Updated `CMakeLists.txt` to set `DEFAULT_EMOJI_COLLECTION` to `otto-gif` for both boards.
- Enhanced `build_default_assets.py` to support alias mapping for Otto GIF emojis.
- Updated `idf_component.yml` to bump `otto-emoji-gif-component` version to `^1.0.5` for improved functionality.
2026-01-31 18:13:15 +08:00
aad2f60b87 fix: reset esp-box-3 display to lvgl (#1715) 2026-01-31 03:14:50 +08:00
83 changed files with 1995 additions and 569 deletions

1
.gitignore vendored
View File

@ -2,6 +2,7 @@ tmp/
components/
managed_components/
build/
dist/
.vscode/
.devcontainer/
sdkconfig.old

View File

@ -9,5 +9,5 @@ include($ENV{IDF_PATH}/tools/cmake/project.cmake)
# "Trim" the build. Include the minimal set of components, main, and anything it depends on.
idf_build_set_property(MINIMAL_BUILD ON)
set(PROJECT_VER "2.2.1")
set(PROJECT_VER "2.2.2")
project(xiaozhi)

View File

@ -104,28 +104,28 @@ elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_ESP32_LCD)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_DF_K10)
set(BOARD_TYPE "df-k10")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_DF_S3_AI_CAM)
set(BOARD_TYPE "df-s3-ai-cam")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_3)
set(BOARD_TYPE "esp-box-3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
set(EMOTE_RESOLUTION "320_240")
elseif(CONFIG_BOARD_TYPE_ESP_BOX)
set(BOARD_TYPE "esp-box")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
set(EMOTE_RESOLUTION "320_240")
elseif(CONFIG_BOARD_TYPE_ESP_BOX_LITE)
set(BOARD_TYPE "esp-box-lite")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_KEVIN_BOX_2)
set(BOARD_TYPE "kevin-box-2")
set(BUILTIN_TEXT_FONT font_puhui_basic_14_1)
@ -134,14 +134,14 @@ elseif(CONFIG_BOARD_TYPE_KEVIN_C3)
set(BOARD_TYPE "kevin-c3")
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V3_DEV)
set(BOARD_TYPE "kevin-sp-v3-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_KEVIN_SP_V4_DEV)
set(BOARD_TYPE "kevin-sp-v4-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_KEVIN_YUYING_313LCD)
set(BOARD_TYPE "kevin-yuying-313lcd")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
@ -149,9 +149,9 @@ elseif(CONFIG_BOARD_TYPE_KEVIN_YUYING_313LCD)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV_S3)
set(BOARD_TYPE "lichuang-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_LICHUANG_DEV_C3)
set(BOARD_TYPE "lichuang-c3-dev")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
@ -201,6 +201,11 @@ elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_S3R_CAM_M12_ECHO_BASE)
set(BOARD_TYPE "atoms3r-cam-m12-echo-base")
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_ECHOS3R)
set(BOARD_TYPE "atom-echos3r")
elseif(CONFIG_BOARD_TYPE_M5STACK_CARDPUTER_ADV)
set(BOARD_TYPE "m5stack-cardputer-adv")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_32)
elseif(CONFIG_BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE)
set(BOARD_TYPE "atommatrix-echo-base")
elseif(CONFIG_BOARD_TYPE_XMINI_C3_V3)
@ -436,29 +441,29 @@ elseif(CONFIG_BOARD_TYPE_MOVECALL_CUICAN_ESP32S3)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3)
set(BOARD_TYPE "atk-dnesp32s3")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX)
set(BOARD_TYPE "atk-dnesp32s3-box")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX0)
set(BOARD_TYPE "atk-dnesp32s3-box0")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_WIFI)
set(BOARD_TYPE "atk-dnesp32s3-box2-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3_BOX2_4G)
set(BOARD_TYPE "atk-dnesp32s3-box2-4g")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_ATK_DNESP32S3M_WIFI)
set(BOARD_TYPE "atk-dnesp32s3m-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
@ -499,24 +504,24 @@ elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_0_96OLED_ML307)
set(BUILTIN_ICON_FONT font_awesome_14_1)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_1_54TFT_WIFI)
set(BOARD_TYPE "xingzhi-cube-1.54tft-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_XINGZHI_CUBE_1_54TFT_ML307)
set(BOARD_TYPE "xingzhi-cube-1.54tft-ml307")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_XINGZHI_METAL_1_54_WIFI)
set(BOARD_TYPE "xingzhi-metal-1.54-wifi")
set(BUILTIN_TEXT_FONT font_puhui_basic_20_4)
set(BUILTIN_TEXT_FONT font_noto_basic_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_SEEED_STUDIO_SENSECAP_WATCHER)
set(BOARD_TYPE "sensecap-watcher")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_TEXT_FONT font_noto_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
set(DEFAULT_EMOJI_COLLECTION noto-emoji_128)
elseif(CONFIG_BOARD_TYPE_DOIT_S3_AIBOX)
set(BOARD_TYPE "doit-s3-aibox")
elseif(CONFIG_BOARD_TYPE_MIXGO_NOVA)
@ -586,10 +591,12 @@ elseif(CONFIG_BOARD_TYPE_OTTO_ROBOT)
set(BOARD_TYPE "otto-robot")
set(BUILTIN_TEXT_FONT font_puhui_16_4)
set(BUILTIN_ICON_FONT font_awesome_16_4)
set(DEFAULT_EMOJI_COLLECTION otto-gif)
elseif(CONFIG_BOARD_TYPE_ELECTRON_BOT)
set(BOARD_TYPE "electron-bot")
set(BUILTIN_TEXT_FONT font_puhui_20_4)
set(BUILTIN_ICON_FONT font_awesome_20_4)
set(DEFAULT_EMOJI_COLLECTION otto-gif)
elseif(CONFIG_BOARD_TYPE_BREAD_COMPACT_WIFI_CAM)
set(BOARD_TYPE "bread-compact-wifi-s3cam")
set(BUILTIN_TEXT_FONT font_puhui_basic_16_4)
@ -777,6 +784,8 @@ if(CONFIG_IDF_TARGET_ESP32)
"display/lvgl_display/jpg/image_to_jpeg.cpp"
"display/lvgl_display/jpg/jpeg_to_image.c"
"boards/common/nt26_board.cc"
"boards/common/ml307_board.cc"
"boards/common/dual_network_board.cc"
)
endif()

View File

@ -251,6 +251,9 @@ choice BOARD_TYPE
config BOARD_TYPE_M5STACK_ATOM_ECHOS3R
bool "M5Stack AtomEchoS3R"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_CARDPUTER_ADV
bool "M5Stack Cardputer Adv"
depends on IDF_TARGET_ESP32S3
config BOARD_TYPE_M5STACK_ATOM_MATRIX_ECHO_BASE
bool "M5Stack AtomMatrix + Echo Base"
depends on IDF_TARGET_ESP32

View File

@ -691,14 +691,16 @@ void Application::HandleToggleChatEvent() {
}
if (state == kDeviceStateIdle) {
ListeningMode mode = aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime;
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
// Schedule to let the state change be processed first (UI update)
Schedule([this, mode]() {
ContinueOpenAudioChannel(mode);
});
return;
}
}
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
SetListeningMode(mode);
} else if (state == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonNone);
} else if (state == kDeviceStateListening) {
@ -706,6 +708,21 @@ void Application::HandleToggleChatEvent() {
}
}
void Application::ContinueOpenAudioChannel(ListeningMode mode) {
// Check state again in case it was changed during scheduling
if (GetDeviceState() != kDeviceStateConnecting) {
return;
}
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
return;
}
}
SetListeningMode(mode);
}
void Application::HandleStartListeningEvent() {
auto state = GetDeviceState();
@ -726,11 +743,12 @@ void Application::HandleStartListeningEvent() {
if (state == kDeviceStateIdle) {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
// Schedule to let the state change be processed first (UI update)
Schedule([this]() {
ContinueOpenAudioChannel(kListeningModeManualStop);
});
return;
}
}
SetListeningMode(kListeningModeManualStop);
} else if (state == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonNone);
@ -762,16 +780,40 @@ void Application::HandleWakeWordDetectedEvent() {
if (state == kDeviceStateIdle) {
audio_service_.EncodeWakeWord();
auto wake_word = audio_service_.GetLastWakeWord();
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
// Schedule to let the state change be processed first (UI update),
// then continue with OpenAudioChannel which may block for ~1 second
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonWakeWordDetected);
} else if (state == kDeviceStateActivating) {
// Restart the activation check if the wake word is detected during activation
SetDeviceState(kDeviceStateIdle);
}
}
void Application::ContinueWakeWordInvoke(const std::string& wake_word) {
// Check state again in case it was changed during scheduling
if (GetDeviceState() != kDeviceStateConnecting) {
return;
}
if (!protocol_->IsAudioChannelOpened()) {
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
return;
}
}
auto wake_word = audio_service_.GetLastWakeWord();
ESP_LOGI(TAG, "Wake word detected: %s", wake_word.c_str());
#if CONFIG_SEND_WAKE_WORD_DATA
// Encode and send the wake word data to the server
@ -787,12 +829,6 @@ void Application::HandleWakeWordDetectedEvent() {
play_popup_on_listening_ = true;
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
#endif
} else if (state == kDeviceStateSpeaking) {
AbortSpeaking(kAbortReasonWakeWordDetected);
} else if (state == kDeviceStateActivating) {
// Restart the activation check if the wake word is detected during activation
SetDeviceState(kDeviceStateIdle);
}
}
void Application::HandleStateChangedEvent() {
@ -808,7 +844,8 @@ void Application::HandleStateChangedEvent() {
case kDeviceStateUnknown:
case kDeviceStateIdle:
display->SetStatus(Lang::Strings::STANDBY);
display->SetEmotion("neutral");
display->ClearChatMessages(); // Clear messages first
display->SetEmotion("neutral"); // Then set emotion (wechat mode checks child count)
audio_service_.EnableVoiceProcessing(false);
audio_service_.EnableWakeWordDetection(true);
break;
@ -959,27 +996,14 @@ void Application::WakeWordInvoke(const std::string& wake_word) {
if (!protocol_->IsAudioChannelOpened()) {
SetDeviceState(kDeviceStateConnecting);
if (!protocol_->OpenAudioChannel()) {
audio_service_.EnableWakeWordDetection(true);
// Schedule to let the state change be processed first (UI update)
Schedule([this, wake_word]() {
ContinueWakeWordInvoke(wake_word);
});
return;
}
}
ESP_LOGI(TAG, "Wake word detected: %s", wake_word.c_str());
#if CONFIG_USE_AFE_WAKE_WORD || CONFIG_USE_CUSTOM_WAKE_WORD
// Encode and send the wake word data to the server
while (auto packet = audio_service_.PopWakeWordPacket()) {
protocol_->SendAudio(std::move(packet));
}
// Set the chat state to wake word detected
protocol_->SendWakeWordDetected(wake_word);
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
#else
// Set flag to play popup sound after state changes to listening
// (PlaySound here would be cleared by ResetDecoder in EnableVoiceProcessing)
play_popup_on_listening_ = true;
SetListeningMode(aec_mode_ == kAecOff ? kListeningModeAutoStop : kListeningModeRealtime);
#endif
// Channel already opened, continue directly
ContinueWakeWordInvoke(wake_word);
} else if (state == kDeviceStateSpeaking) {
Schedule([this]() {
AbortSpeaking(kAbortReasonNone);

View File

@ -153,6 +153,8 @@ private:
void HandleNetworkDisconnectedEvent();
void HandleActivationDoneEvent();
void HandleWakeWordDetectedEvent();
void ContinueOpenAudioChannel(ListeningMode mode);
void ContinueWakeWordInvoke(const std::string& wake_word);
// Activation task (runs in background)
void ActivationTask();

View File

@ -12,6 +12,7 @@
#include <esp_log.h>
#include <esp_timer.h>
#include <esp_heap_caps.h>
#include <cbin_font.h>
@ -464,16 +465,21 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
SECTOR_SIZE, content_length, sectors_to_erase, total_erase_size);
// 写入新的资源文件到分区一边erase一边写入
char buffer[512];
char* buffer = (char*)heap_caps_malloc(SECTOR_SIZE, MALLOC_CAP_INTERNAL);
if (buffer == nullptr) {
ESP_LOGE(TAG, "Failed to allocate buffer");
return false;
}
size_t total_written = 0;
size_t recent_written = 0;
size_t current_sector = 0;
auto last_calc_time = esp_timer_get_time();
while (true) {
int ret = http->Read(buffer, sizeof(buffer));
int ret = http->Read(buffer, SECTOR_SIZE);
if (ret < 0) {
ESP_LOGE(TAG, "Failed to read HTTP data: %s", esp_err_to_name(ret));
heap_caps_free(buffer);
return false;
}
@ -493,6 +499,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
// 确保擦除范围不超过分区大小
if (sector_end > partition_->size) {
ESP_LOGE(TAG, "Sector end (%u) exceeds partition size (%lu)", sector_end, partition_->size);
heap_caps_free(buffer);
return false;
}
@ -500,6 +507,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
esp_err_t err = esp_partition_erase_range(partition_, sector_start, SECTOR_SIZE);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to erase sector %u at offset %u: %s", current_sector, sector_start, esp_err_to_name(err));
heap_caps_free(buffer);
return false;
}
@ -510,6 +518,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
esp_err_t err = esp_partition_write(partition_, total_written, buffer, ret);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to write to assets partition at offset %u: %s", total_written, esp_err_to_name(err));
heap_caps_free(buffer);
return false;
}
@ -531,6 +540,7 @@ bool Assets::Download(std::string url, std::function<void(int progress, size_t s
}
http->Close();
heap_caps_free(buffer);
if (total_written != content_length) {
ESP_LOGE(TAG, "Downloaded size (%u) does not match expected size (%u)", total_written, content_length);

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "جاري تحميل الموارد...",
"PLEASE_WAIT": "يرجى الانتظار...",
"FOUND_NEW_ASSETS": "تم العثور على موارد جديدة: %s",
"HELLO_MY_FRIEND": "مرحباً، صديقي!"
"HELLO_MY_FRIEND": "مرحباً، صديقي!",
"CONNECTION_SUCCESSFUL": "تم الاتصال بنجاح",
"FLIGHT_MODE_OFF": "وضع الطيران معطل",
"FLIGHT_MODE_ON": "وضع الطيران قيد التشغيل",
"MODEM_INIT_ERROR": "فشل تهيئة المودم"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Намерени нови ресурси: %s",
"DOWNLOAD_ASSETS_FAILED": "Неуспешно изтегляне на ресурси",
"LOADING_ASSETS": "Зареждане на ресурси...",
"HELLO_MY_FRIEND": "Здравей, мой приятел!"
"HELLO_MY_FRIEND": "Здравей, мой приятел!",
"FLIGHT_MODE_OFF": "Режим на самолет е изключен",
"FLIGHT_MODE_ON": "Режим на самолет е включен",
"MODEM_INIT_ERROR": "Неуспешна инициализация на модема"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "S'han trobat nous recursos: %s",
"DOWNLOAD_ASSETS_FAILED": "No s'han pogut descarregar els recursos",
"LOADING_ASSETS": "Carregant recursos...",
"HELLO_MY_FRIEND": "Hola, amic meu!"
"HELLO_MY_FRIEND": "Hola, amic meu!",
"FLIGHT_MODE_OFF": "El mode avió està desactivat",
"FLIGHT_MODE_ON": "El mode avió està activat",
"MODEM_INIT_ERROR": "Error d'inicialització del mòdem"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Načítání prostředků...",
"PLEASE_WAIT": "Prosím čekejte...",
"FOUND_NEW_ASSETS": "Nalezeny nové prostředky: %s",
"HELLO_MY_FRIEND": "Ahoj, můj příteli!"
"HELLO_MY_FRIEND": "Ahoj, můj příteli!",
"CONNECTION_SUCCESSFUL": "Připojení úspěšné",
"FLIGHT_MODE_OFF": "Letecký režim je vypnutý",
"FLIGHT_MODE_ON": "Letecký režim je zapnutý",
"MODEM_INIT_ERROR": "Chyba inicializace modemu"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Fandt nye ressourcer: %s",
"DOWNLOAD_ASSETS_FAILED": "Download af ressourcer mislykkedes",
"LOADING_ASSETS": "Indlæser ressourcer...",
"HELLO_MY_FRIEND": "Hej, min ven!"
"HELLO_MY_FRIEND": "Hej, min ven!",
"FLIGHT_MODE_OFF": "Flytilstand er slukket",
"FLIGHT_MODE_ON": "Flytilstand er tændt",
"MODEM_INIT_ERROR": "Modeminitialisering mislykkedes"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Ressourcen werden geladen...",
"PLEASE_WAIT": "Bitte warten...",
"FOUND_NEW_ASSETS": "Neue Ressourcen gefunden: %s",
"HELLO_MY_FRIEND": "Hallo, mein Freund!"
"HELLO_MY_FRIEND": "Hallo, mein Freund!",
"CONNECTION_SUCCESSFUL": "Verbindung erfolgreich",
"FLIGHT_MODE_OFF": "Flugmodus ist deaktiviert",
"FLIGHT_MODE_ON": "Flugmodus ist aktiviert",
"MODEM_INIT_ERROR": "Modem-Initialisierung fehlgeschlagen"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Βρέθηκαν νέοι πόροι: %s",
"DOWNLOAD_ASSETS_FAILED": "Αποτυχία λήψης πόρων",
"LOADING_ASSETS": "Φόρτωση πόρων...",
"HELLO_MY_FRIEND": "Γεια σου, φίλε μου!"
"HELLO_MY_FRIEND": "Γεια σου, φίλε μου!",
"FLIGHT_MODE_OFF": "Η λειτουργία πτήσης είναι απενεργοποιημένη",
"FLIGHT_MODE_ON": "Η λειτουργία πτήσης είναι ενεργή",
"MODEM_INIT_ERROR": "Αποτυχία αρχικοποίησης modem"
}
}

View File

@ -13,6 +13,8 @@
"REG_ERROR": "Unable to access network, please check SIM card status",
"MODEM_INIT_ERROR": "Modem initialization failed",
"DETECTING_MODULE": "Detecting module...",
"FLIGHT_MODE_ON": "Flight mode is on",
"FLIGHT_MODE_OFF": "Flight mode is off",
"REGISTERING_NETWORK": "Waiting for network...",
"CHECKING_NEW_VERSION": "Checking for new version...",
"CHECK_NEW_VERSION_FAILED": "Check for new version failed, will retry in %d seconds: %s",

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Cargando recursos...",
"PLEASE_WAIT": "Por favor espere...",
"FOUND_NEW_ASSETS": "Encontrados nuevos recursos: %s",
"HELLO_MY_FRIEND": "¡Hola, mi amigo!"
"HELLO_MY_FRIEND": "¡Hola, mi amigo!",
"CONNECTION_SUCCESSFUL": "Conexión exitosa",
"FLIGHT_MODE_OFF": "El modo avión está desactivado",
"FLIGHT_MODE_ON": "El modo avión está activado",
"MODEM_INIT_ERROR": "Error de inicialización del módem"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "منابع جدید یافت شد: %s",
"DOWNLOAD_ASSETS_FAILED": "دانلود منابع ناموفق بود",
"LOADING_ASSETS": "بارگذاری منابع...",
"HELLO_MY_FRIEND": "سلام، دوست من!"
"HELLO_MY_FRIEND": "سلام، دوست من!",
"FLIGHT_MODE_OFF": "حالت پرواز خاموش است",
"FLIGHT_MODE_ON": "حالت پرواز روشن است",
"MODEM_INIT_ERROR": "خطا در راه‌اندازی مودم"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Ladataan resursseja...",
"PLEASE_WAIT": "Odota hetki...",
"FOUND_NEW_ASSETS": "Löydetty uusia resursseja: %s",
"HELLO_MY_FRIEND": "Hei, ystäväni!"
"HELLO_MY_FRIEND": "Hei, ystäväni!",
"CONNECTION_SUCCESSFUL": "Yhteys onnistui",
"FLIGHT_MODE_OFF": "Lentotila on pois päältä",
"FLIGHT_MODE_ON": "Lentotila on päällä",
"MODEM_INIT_ERROR": "Modeemin alustus epäonnistui"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Nakahanap ng mga bagong assets: %s",
"DOWNLOAD_ASSETS_FAILED": "Nabigo ang pag-download ng mga assets",
"LOADING_ASSETS": "Nilo-load ang mga assets...",
"HELLO_MY_FRIEND": "Kumusta, kaibigan ko!"
"HELLO_MY_FRIEND": "Kumusta, kaibigan ko!",
"FLIGHT_MODE_OFF": "Naka-off ang flight mode",
"FLIGHT_MODE_ON": "Naka-on ang flight mode",
"MODEM_INIT_ERROR": "Nabigo ang pag-initialize ng modem"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Chargement des ressources...",
"PLEASE_WAIT": "Veuillez patienter...",
"FOUND_NEW_ASSETS": "Nouvelles ressources trouvées: %s",
"HELLO_MY_FRIEND": "Bonjour, mon ami !"
"HELLO_MY_FRIEND": "Bonjour, mon ami !",
"CONNECTION_SUCCESSFUL": "Connexion réussie",
"FLIGHT_MODE_OFF": "Le mode avion est désactivé",
"FLIGHT_MODE_ON": "Le mode avion est activé",
"MODEM_INIT_ERROR": "Échec de l'initialisation du modem"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "נמצאו משאבים חדשים: %s",
"DOWNLOAD_ASSETS_FAILED": "הורדת משאבים נכשלה",
"LOADING_ASSETS": "טוען משאבים...",
"HELLO_MY_FRIEND": "שלום, ידידי!"
"HELLO_MY_FRIEND": "שלום, ידידי!",
"FLIGHT_MODE_OFF": "מצב טיסה כבוי",
"FLIGHT_MODE_ON": "מצב טיסה מופעל",
"MODEM_INIT_ERROR": "אתחול המודם נכשל"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "संसाधन लोड हो रहे हैं...",
"PLEASE_WAIT": "कृपया प्रतीक्षा करें...",
"FOUND_NEW_ASSETS": "नए संसाधन मिले: %s",
"HELLO_MY_FRIEND": "नमस्ते, मेरे दोस्त!"
"HELLO_MY_FRIEND": "नमस्ते, मेरे दोस्त!",
"CONNECTION_SUCCESSFUL": "कनेक्शन सफल",
"FLIGHT_MODE_OFF": "फ़्लाइट मोड बंद है",
"FLIGHT_MODE_ON": "फ़्लाइट मोड चालू है",
"MODEM_INIT_ERROR": "मॉडेम आरंभीकरण विफल"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Pronađeni novi resursi: %s",
"DOWNLOAD_ASSETS_FAILED": "Preuzimanje resursa nije uspjelo",
"LOADING_ASSETS": "Učitavanje resursa...",
"HELLO_MY_FRIEND": "Bok, moj prijatelju!"
"HELLO_MY_FRIEND": "Bok, moj prijatelju!",
"FLIGHT_MODE_OFF": "Način rada u zrakoplovu je isključen",
"FLIGHT_MODE_ON": "Način rada u zrakoplovu je uključen",
"MODEM_INIT_ERROR": "Neuspjela inicijalizacija modema"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Új erőforrások találva: %s",
"DOWNLOAD_ASSETS_FAILED": "Az erőforrások letöltése sikertelen",
"LOADING_ASSETS": "Erőforrások betöltése...",
"HELLO_MY_FRIEND": "Helló, barátom!"
"HELLO_MY_FRIEND": "Helló, barátom!",
"FLIGHT_MODE_OFF": "A repülési mód ki van kapcsolva",
"FLIGHT_MODE_ON": "A repülési mód be van kapcsolva",
"MODEM_INIT_ERROR": "A modem inicializálása sikertelen"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Memuat aset...",
"PLEASE_WAIT": "Mohon tunggu...",
"FOUND_NEW_ASSETS": "Ditemukan aset baru: %s",
"HELLO_MY_FRIEND": "Halo, teman saya!"
"HELLO_MY_FRIEND": "Halo, teman saya!",
"CONNECTION_SUCCESSFUL": "Koneksi berhasil",
"FLIGHT_MODE_OFF": "Mode pesawat nonaktif",
"FLIGHT_MODE_ON": "Mode pesawat aktif",
"MODEM_INIT_ERROR": "Gagal menginisialisasi modem"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Caricamento risorse...",
"PLEASE_WAIT": "Attendere prego...",
"FOUND_NEW_ASSETS": "Trovate nuove risorse: %s",
"HELLO_MY_FRIEND": "Ciao, amico mio!"
"HELLO_MY_FRIEND": "Ciao, amico mio!",
"CONNECTION_SUCCESSFUL": "Connessione riuscita",
"FLIGHT_MODE_OFF": "La modalità aereo è disattivata",
"FLIGHT_MODE_ON": "La modalità aereo è attiva",
"MODEM_INIT_ERROR": "Inizializzazione modem non riuscita"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "アセットを読み込み中...",
"PLEASE_WAIT": "お待ちください...",
"FOUND_NEW_ASSETS": "新しいアセットが見つかりました: %s",
"HELLO_MY_FRIEND": "こんにちは、友達!"
"HELLO_MY_FRIEND": "こんにちは、友達!",
"CONNECTION_SUCCESSFUL": "接続成功",
"FLIGHT_MODE_OFF": "機内モードがオフです",
"FLIGHT_MODE_ON": "機内モードがオンです",
"MODEM_INIT_ERROR": "モデムの初期化に失敗しました"
}
}

View File

@ -51,6 +51,9 @@
"LOADING_ASSETS": "에셋 로딩 중...",
"PLEASE_WAIT": "잠시 기다려 주세요...",
"FOUND_NEW_ASSETS": "새로운 에셋을 발견했습니다: %s",
"HELLO_MY_FRIEND": "안녕하세요, 친구!"
"HELLO_MY_FRIEND": "안녕하세요, 친구!",
"FLIGHT_MODE_OFF": "비행기 모드가 꺼져 있습니다",
"FLIGHT_MODE_ON": "비행기 모드가 켜져 있습니다",
"MODEM_INIT_ERROR": "모뎀 초기화 실패"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Menemui aset baharu: %s",
"DOWNLOAD_ASSETS_FAILED": "Gagal memuat turun aset",
"LOADING_ASSETS": "Memuatkan aset...",
"HELLO_MY_FRIEND": "Hai, kawan saya!"
"HELLO_MY_FRIEND": "Hai, kawan saya!",
"FLIGHT_MODE_OFF": "Mod penerbangan dimatikan",
"FLIGHT_MODE_ON": "Mod penerbangan dihidupkan",
"MODEM_INIT_ERROR": "Modem gagal dimulakan"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Fant nye ressurser: %s",
"DOWNLOAD_ASSETS_FAILED": "Nedlasting av ressurser mislyktes",
"LOADING_ASSETS": "Laster ressurser...",
"HELLO_MY_FRIEND": "Hei, min venn!"
"HELLO_MY_FRIEND": "Hei, min venn!",
"FLIGHT_MODE_OFF": "Flymodus er av",
"FLIGHT_MODE_ON": "Flymodus er på",
"MODEM_INIT_ERROR": "Modeminitialisering mislyktes"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Nieuwe bronnen gevonden: %s",
"DOWNLOAD_ASSETS_FAILED": "Downloaden van bronnen mislukt",
"LOADING_ASSETS": "Bronnen laden...",
"HELLO_MY_FRIEND": "Hallo, mijn vriend!"
"HELLO_MY_FRIEND": "Hallo, mijn vriend!",
"FLIGHT_MODE_OFF": "Vliegtuigmodus is uitgeschakeld",
"FLIGHT_MODE_ON": "Vliegtuigmodus is ingeschakeld",
"MODEM_INIT_ERROR": "Modeminitialisatie mislukt"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Ładowanie zasobów...",
"PLEASE_WAIT": "Proszę czekać...",
"FOUND_NEW_ASSETS": "Znaleziono nowe zasoby: %s",
"HELLO_MY_FRIEND": "Cześć, mój przyjacielu!"
"HELLO_MY_FRIEND": "Cześć, mój przyjacielu!",
"CONNECTION_SUCCESSFUL": "Połączenie udane",
"FLIGHT_MODE_OFF": "Tryb samolotowy jest wyłączony",
"FLIGHT_MODE_ON": "Tryb samolotowy jest włączony",
"MODEM_INIT_ERROR": "Inicjalizacja modemu nie powiodła się"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "A carregar recursos...",
"PLEASE_WAIT": "Por favor aguarde...",
"FOUND_NEW_ASSETS": "Encontrados novos recursos: %s",
"HELLO_MY_FRIEND": "Olá, meu amigo!"
"HELLO_MY_FRIEND": "Olá, meu amigo!",
"CONNECTION_SUCCESSFUL": "Ligação bem-sucedida",
"FLIGHT_MODE_OFF": "O modo avião está desativado",
"FLIGHT_MODE_ON": "O modo avião está ativado",
"MODEM_INIT_ERROR": "Falha na inicialização do modem"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Se încarcă resursele...",
"PLEASE_WAIT": "Vă rugăm să așteptați...",
"FOUND_NEW_ASSETS": "S-au găsit resurse noi: %s",
"HELLO_MY_FRIEND": "Salut, prietenul meu!"
"HELLO_MY_FRIEND": "Salut, prietenul meu!",
"CONNECTION_SUCCESSFUL": "Conexiune reușită",
"FLIGHT_MODE_OFF": "Modul avion este dezactivat",
"FLIGHT_MODE_ON": "Modul avion este activat",
"MODEM_INIT_ERROR": "Inițializarea modemului a eșuat"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Загрузка ресурсов...",
"PLEASE_WAIT": "Пожалуйста, подождите...",
"FOUND_NEW_ASSETS": "Найдены новые ресурсы: %s",
"HELLO_MY_FRIEND": "Привет, мой друг!"
"HELLO_MY_FRIEND": "Привет, мой друг!",
"CONNECTION_SUCCESSFUL": "Подключение успешно",
"FLIGHT_MODE_OFF": "Режим полета выключен",
"FLIGHT_MODE_ON": "Режим полета включен",
"MODEM_INIT_ERROR": "Ошибка инициализации модема"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Nájdené nové zdroje: %s",
"DOWNLOAD_ASSETS_FAILED": "Sťahovanie zdrojov zlyhalo",
"LOADING_ASSETS": "Načítavanie zdrojov...",
"HELLO_MY_FRIEND": "Ahoj, môj priateľ!"
"HELLO_MY_FRIEND": "Ahoj, môj priateľ!",
"FLIGHT_MODE_OFF": "Letecký režim je vypnutý",
"FLIGHT_MODE_ON": "Letecký režim je zapnutý",
"MODEM_INIT_ERROR": "Chyba inicializácie modemu"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Najdeni novi viri: %s",
"DOWNLOAD_ASSETS_FAILED": "Prenos virov ni uspel",
"LOADING_ASSETS": "Nalaganje virov...",
"HELLO_MY_FRIEND": "Pozdravljeni, moj prijatelj!"
"HELLO_MY_FRIEND": "Pozdravljeni, moj prijatelj!",
"FLIGHT_MODE_OFF": "Način leta je izklopljen",
"FLIGHT_MODE_ON": "Način leta je vklopljen",
"MODEM_INIT_ERROR": "Inicializacija modema ni uspela"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Пронађени нови ресурси: %s",
"DOWNLOAD_ASSETS_FAILED": "Преузимање ресурса није успело",
"LOADING_ASSETS": "Учитавање ресурса...",
"HELLO_MY_FRIEND": "Здраво, пријатељу!"
"HELLO_MY_FRIEND": "Здраво, пријатељу!",
"FLIGHT_MODE_OFF": "Режим лета је искључен",
"FLIGHT_MODE_ON": "Режим лета је укључен",
"MODEM_INIT_ERROR": "Иницијализација модема није успела"
}
}

View File

@ -51,7 +51,9 @@
"FOUND_NEW_ASSETS": "Hittade nya resurser: %s",
"DOWNLOAD_ASSETS_FAILED": "Nedladdning av resurser misslyckades",
"LOADING_ASSETS": "Laddar resurser...",
"HELLO_MY_FRIEND": "Hej, min vän!"
"HELLO_MY_FRIEND": "Hej, min vän!",
"FLIGHT_MODE_OFF": "Flygläge är av",
"FLIGHT_MODE_ON": "Flygläge är på",
"MODEM_INIT_ERROR": "Modeminitiering misslyckades"
}
}

View File

@ -51,6 +51,9 @@
"LOADING_ASSETS": "กำลังโหลดทรัพยากร...",
"PLEASE_WAIT": "กรุณารอสักครู่...",
"FOUND_NEW_ASSETS": "พบทรัพยากรใหม่: %s",
"HELLO_MY_FRIEND": "สวัสดี เพื่อนของฉัน!"
"HELLO_MY_FRIEND": "สวัสดี เพื่อนของฉัน!",
"FLIGHT_MODE_OFF": "โหมดเครื่องบินปิดอยู่",
"FLIGHT_MODE_ON": "โหมดเครื่องบินเปิดอยู่",
"MODEM_INIT_ERROR": "การเริ่มต้นโมเด็มล้มเหลว"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Varlıklar yükleniyor...",
"PLEASE_WAIT": "Lütfen bekleyin...",
"FOUND_NEW_ASSETS": "Yeni varlıklar bulundu: %s",
"HELLO_MY_FRIEND": "Merhaba, arkadaşım!"
"HELLO_MY_FRIEND": "Merhaba, arkadaşım!",
"CONNECTION_SUCCESSFUL": "Bağlantı başarılı",
"FLIGHT_MODE_OFF": "Uçak modu kapalı",
"FLIGHT_MODE_ON": "Uçak modu açık",
"MODEM_INIT_ERROR": "Modem başlatma hatası"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "Завантаження ресурсів...",
"PLEASE_WAIT": "Будь ласка, зачекайте...",
"FOUND_NEW_ASSETS": "Знайдено нові ресурси: %s",
"HELLO_MY_FRIEND": "Привіт, мій друже!"
"HELLO_MY_FRIEND": "Привіт, мій друже!",
"CONNECTION_SUCCESSFUL": "Підключення успішне",
"FLIGHT_MODE_OFF": "Режим польоту вимкнено",
"FLIGHT_MODE_ON": "Режим польоту увімкнено",
"MODEM_INIT_ERROR": "Помилка ініціалізації модему"
}
}

View File

@ -51,6 +51,9 @@
"LOADING_ASSETS": "Đang tải tài nguyên...",
"PLEASE_WAIT": "Vui lòng đợi...",
"FOUND_NEW_ASSETS": "Tìm thấy tài nguyên mới: %s",
"HELLO_MY_FRIEND": "Xin chào, bạn của tôi!"
"HELLO_MY_FRIEND": "Xin chào, bạn của tôi!",
"FLIGHT_MODE_OFF": "Chế độ máy bay đang tắt",
"FLIGHT_MODE_ON": "Chế độ máy bay đang bật",
"MODEM_INIT_ERROR": "Khởi tạo modem thất bại"
}
}

View File

@ -51,6 +51,9 @@
"LOADING_ASSETS": "加载资源...",
"PLEASE_WAIT": "请稍候...",
"FOUND_NEW_ASSETS": "发现新资源: %s",
"HELLO_MY_FRIEND": "你好,我的朋友!"
"HELLO_MY_FRIEND": "你好,我的朋友!",
"CONNECTION_SUCCESSFUL": "连接成功",
"FLIGHT_MODE_OFF": "飞行模式已关闭",
"FLIGHT_MODE_ON": "飞行模式已开启"
}
}

View File

@ -50,6 +50,10 @@
"LOADING_ASSETS": "載入資源...",
"PLEASE_WAIT": "請稍候...",
"FOUND_NEW_ASSETS": "發現新資源: %s",
"HELLO_MY_FRIEND": "你好,我的朋友!"
"HELLO_MY_FRIEND": "你好,我的朋友!",
"CONNECTION_SUCCESSFUL": "連線成功",
"FLIGHT_MODE_OFF": "飛航模式已關閉",
"FLIGHT_MODE_ON": "飛航模式已開啟",
"MODEM_INIT_ERROR": "模組初始化失敗"
}
}

View File

@ -108,6 +108,9 @@ void AfeWakeWord::Feed(const std::vector<int16_t>& data) {
if (afe_data_ == nullptr) {
return;
}
if (!(xEventGroupGetBits(event_group_) & DETECTION_RUNNING_EVENT)) {
return;
}
afe_iface_->feed(afe_data_, data.data());
}

View File

@ -1,4 +1,4 @@
#include "dual_network_board.h"
#include "wifi_board.h"
#include "codecs/no_audio_codec.h"
#include "display/lcd_display.h"
#include "system_reset.h"
@ -57,7 +57,7 @@ static const gc9a01_lcd_init_cmd_t gc9107_lcd_init_cmds[] = {
#define TAG "ESP32-LCD-MarsbearSupport"
class CompactWifiBoardLCD : public DualNetworkBoard {
class CompactWifiBoardLCD : public WifiBoard {
private:
Button boot_button_;
Button touch_button_;
@ -136,26 +136,14 @@ private:
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (GetNetworkType() == NetworkType::WIFI) {
if (app.GetDeviceState() == kDeviceStateStarting) {
// cast to WifiBoard
auto& wifi_board = static_cast<WifiBoard&>(GetCurrentBoard());
wifi_board.EnterWifiConfigMode();
EnterWifiConfigMode();
return;
}
}
gpio_set_level(BUILTIN_LED_GPIO, 1);
app.ToggleChatState();
});
boot_button_.OnDoubleClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting || app.GetDeviceState() == kDeviceStateWifiConfiguring) {
SwitchNetworkType();
}
});
asr_button_.OnClick([this]() {
std::string wake_word="你好小智";
Application::GetInstance().WakeWordInvoke(wake_word);
@ -174,8 +162,7 @@ private:
}
public:
CompactWifiBoardLCD() :
DualNetworkBoard(ML307_TX_PIN, ML307_RX_PIN),
CompactWifiBoardLCD() : WifiBoard(),
boot_button_(BOOT_BUTTON_GPIO), touch_button_(TOUCH_BUTTON_GPIO), asr_button_(ASR_BUTTON_GPIO) {
InitializeSpi();
InitializeLcdDisplay();

View File

@ -1,4 +1,4 @@
#include "dual_network_board.h"
#include "wifi_board.h"
#include "codecs/no_audio_codec.h"
#include "system_reset.h"
#include "application.h"
@ -16,7 +16,7 @@
#define TAG "ESP32-MarsbearSupport"
class CompactWifiBoard : public DualNetworkBoard {
class CompactWifiBoard : public WifiBoard {
private:
Button boot_button_;
Button touch_button_;
@ -104,26 +104,14 @@ private:
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (GetNetworkType() == NetworkType::WIFI) {
if (app.GetDeviceState() == kDeviceStateStarting) {
// cast to WifiBoard
auto& wifi_board = static_cast<WifiBoard&>(GetCurrentBoard());
wifi_board.EnterWifiConfigMode();
EnterWifiConfigMode();
return;
}
}
gpio_set_level(BUILTIN_LED_GPIO, 1);
app.ToggleChatState();
});
boot_button_.OnDoubleClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting || app.GetDeviceState() == kDeviceStateWifiConfiguring) {
SwitchNetworkType();
}
});
asr_button_.OnClick([this]() {
std::string wake_word="你好小智";
Application::GetInstance().WakeWordInvoke(wake_word);
@ -145,7 +133,7 @@ private:
}
public:
CompactWifiBoard() : DualNetworkBoard(ML307_TX_PIN, ML307_RX_PIN), boot_button_(BOOT_BUTTON_GPIO), touch_button_(TOUCH_BUTTON_GPIO), asr_button_(ASR_BUTTON_GPIO)
CompactWifiBoard() : WifiBoard(), boot_button_(BOOT_BUTTON_GPIO), touch_button_(TOUCH_BUTTON_GPIO), asr_button_(ASR_BUTTON_GPIO)
{
InitializeDisplayI2c();
InitializeSsd1306Display();

View File

@ -9,6 +9,7 @@ public:
virtual bool Capture() = 0;
virtual bool SetHMirror(bool enabled) = 0;
virtual bool SetVFlip(bool enabled) = 0;
virtual bool SetSwapBytes(bool enabled) { return false; } // Optional, default no-op
virtual std::string Explain(const std::string& question) = 0;
};

View File

@ -41,6 +41,11 @@ Esp32Camera::~Esp32Camera() {
esp_camera_fb_return(current_fb_);
current_fb_ = nullptr;
}
if (encode_buf_) {
heap_caps_free(encode_buf_);
encode_buf_ = nullptr;
encode_buf_size_ = 0;
}
esp_camera_deinit();
streaming_on_ = false;
}
@ -72,31 +77,47 @@ bool Esp32Camera::Capture() {
}
}
// Perform byte swapping for RGB565 format and prepare preview image
// Prepare encode buffer for RGB565 format (with optional byte swapping)
if (current_fb_->format == PIXFORMAT_RGB565) {
size_t pixel_count = current_fb_->width * current_fb_->height;
size_t data_size = pixel_count * 2;
uint8_t *preview_data = (uint8_t *)heap_caps_malloc(data_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
if (preview_data == nullptr) {
ESP_LOGE(TAG, "Failed to allocate memory for preview image");
// Allocate or reallocate encode buffer if needed
if (encode_buf_size_ < data_size) {
if (encode_buf_) {
heap_caps_free(encode_buf_);
}
encode_buf_ = (uint8_t *)heap_caps_malloc(data_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
if (encode_buf_ == nullptr) {
ESP_LOGE(TAG, "Failed to allocate memory for encode buffer");
encode_buf_size_ = 0;
return false;
}
uint16_t *src = (uint16_t *)current_fb_->buf;
uint16_t *dst = (uint16_t *)preview_data;
for (size_t i = 0; i < pixel_count; i++) {
// Copy data from driver buffer to preview buffer with byte swapping
dst[i] = __builtin_bswap16(src[i]);
encode_buf_size_ = data_size;
}
// Display preview image
// Copy data to encode buffer with optional byte swapping
uint16_t *src = (uint16_t *)current_fb_->buf;
uint16_t *dst = (uint16_t *)encode_buf_;
if (swap_bytes_enabled_) {
for (size_t i = 0; i < pixel_count; i++) {
dst[i] = __builtin_bswap16(src[i]);
}
} else {
memcpy(encode_buf_, current_fb_->buf, data_size);
}
// Allocate separate buffer for preview display
uint8_t *preview_data = (uint8_t *)heap_caps_malloc(data_size, MALLOC_CAP_SPIRAM | MALLOC_CAP_8BIT);
if (preview_data != nullptr) {
memcpy(preview_data, encode_buf_, data_size);
auto display = dynamic_cast<LvglDisplay *>(Board::GetInstance().GetDisplay());
if (display != nullptr) {
display->SetPreviewImage(std::make_unique<LvglAllocatedImage>(preview_data, data_size, current_fb_->width, current_fb_->height, current_fb_->width * 2, LV_COLOR_FORMAT_RGB565));
} else {
heap_caps_free(preview_data);
}
}
} else if (current_fb_->format == PIXFORMAT_JPEG) {
// JPEG format preview usually requires decoding, skip preview display for now, just log
ESP_LOGW(TAG, "JPEG capture success, len=%zu, but not supported for preview", current_fb_->len);
@ -126,6 +147,11 @@ bool Esp32Camera::SetVFlip(bool enabled) {
return true;
}
bool Esp32Camera::SetSwapBytes(bool enabled) {
swap_bytes_enabled_ = enabled;
return true;
}
std::string Esp32Camera::Explain(const std::string &question) {
if (explain_url_.empty()) {
throw std::runtime_error("Image explain URL or token is not set");
@ -172,7 +198,15 @@ std::string Esp32Camera::Explain(const std::string &question) {
return;
}
bool ok = image_to_jpeg_cb(current_fb_->buf, current_fb_->len, w, h, enc_fmt, 80,
// Use encode buffer for RGB565, otherwise use original frame buffer
uint8_t *jpeg_src_buf = current_fb_->buf;
size_t jpeg_src_len = current_fb_->len;
if (current_fb_->format == PIXFORMAT_RGB565 && encode_buf_ != nullptr) {
jpeg_src_buf = encode_buf_;
jpeg_src_len = encode_buf_size_;
}
bool ok = image_to_jpeg_cb(jpeg_src_buf, jpeg_src_len, w, h, enc_fmt, 80,
[](void* arg, size_t index, const void* data, size_t len) -> size_t {
auto jpeg_queue = static_cast<QueueHandle_t>(arg);
JpegChunk chunk = {.data = nullptr, .len = len};

View File

@ -23,10 +23,13 @@ class Esp32Camera : public Camera
{
private:
bool streaming_on_ = false;
bool swap_bytes_enabled_ = true; // Swap pixel byte order for RGB565, enabled by default
std::string explain_url_;
std::string explain_token_;
std::thread encoder_thread_;
camera_fb_t *current_fb_ = nullptr;
uint8_t *encode_buf_ = nullptr; // Buffer for JPEG encoding (with optional byte swap)
size_t encode_buf_size_ = 0;
public:
Esp32Camera(const camera_config_t &config);
@ -36,5 +39,6 @@ public:
virtual bool Capture() override;
virtual bool SetHMirror(bool enabled) override;
virtual bool SetVFlip(bool enabled) override;
virtual bool SetSwapBytes(bool enabled) override;
virtual std::string Explain(const std::string &question) override;
};

View File

@ -107,6 +107,9 @@ void Nt26Board::StartNetwork() {
ScheduleAsyncStop();
OnNetworkEvent(NetworkEvent::ModemErrorInitFailed);
break;
case UartEthModem::UartEthModemEvent::InFlightMode:
ESP_LOGW(TAG, "Modem in flight mode");
break;
}
});

View File

@ -3,12 +3,13 @@
#include <esp_log.h>
#include <cstring>
#include <vector>
#include "assets.h"
#include "assets/lang_config.h"
#include "display/lvgl_display/emoji_collection.h"
#include "display/lvgl_display/lvgl_image.h"
#include "display/lvgl_display/lvgl_theme.h"
#include "otto_emoji_gif.h"
#define TAG "ElectronEmojiDisplay"
ElectronEmojiDisplay::ElectronEmojiDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width, int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y,
@ -19,64 +20,12 @@ ElectronEmojiDisplay::ElectronEmojiDisplay(esp_lcd_panel_io_handle_t panel_io, e
}
void ElectronEmojiDisplay::InitializeElectronEmojis() {
ESP_LOGI(TAG, "初始化Electron GIF表情");
auto otto_emoji_collection = std::make_shared<EmojiCollection>();
// 中性/平静类表情 -> staticstate
otto_emoji_collection->AddEmoji("staticstate", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("neutral", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("relaxed", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("sleepy", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("idle", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
// 积极/开心类表情 -> happy
otto_emoji_collection->AddEmoji("happy", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("laughing", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("funny", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("loving", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("confident", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("winking", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("cool", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("delicious", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("kissy", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("silly", new LvglRawImage((void*)happy.data, happy.data_size));
// 悲伤类表情 -> sad
otto_emoji_collection->AddEmoji("sad", new LvglRawImage((void*)sad.data, sad.data_size));
otto_emoji_collection->AddEmoji("crying", new LvglRawImage((void*)sad.data, sad.data_size));
// 愤怒类表情 -> anger
otto_emoji_collection->AddEmoji("anger", new LvglRawImage((void*)anger.data, anger.data_size));
otto_emoji_collection->AddEmoji("angry", new LvglRawImage((void*)anger.data, anger.data_size));
// 惊讶类表情 -> scare
otto_emoji_collection->AddEmoji("scare", new LvglRawImage((void*)scare.data, scare.data_size));
otto_emoji_collection->AddEmoji("surprised", new LvglRawImage((void*)scare.data, scare.data_size));
otto_emoji_collection->AddEmoji("shocked", new LvglRawImage((void*)scare.data, scare.data_size));
// 思考/困惑类表情 -> buxue
otto_emoji_collection->AddEmoji("buxue", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("thinking", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("confused", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("embarrassed", new LvglRawImage((void*)buxue.data, buxue.data_size));
// 将表情集合添加到主题中
auto& theme_manager = LvglThemeManager::GetInstance();
auto light_theme = theme_manager.GetTheme("light");
auto dark_theme = theme_manager.GetTheme("dark");
if (light_theme != nullptr) {
light_theme->set_emoji_collection(otto_emoji_collection);
}
if (dark_theme != nullptr) {
dark_theme->set_emoji_collection(otto_emoji_collection);
}
ESP_LOGI(TAG, "Electron表情初始化将由Assets系统处理");
// 表情初始化已移至assets系统,通过DEFAULT_EMOJI_COLLECTION=otto-gif配置
// assets.cc会从assets分区加载GIF表情并设置到theme
// 设置默认表情为staticstate
SetEmotion("staticstate");
ESP_LOGI(TAG, "Electron GIF表情初始化完成");
}
void ElectronEmojiDisplay::SetupChatLabel() {

View File

@ -4,10 +4,7 @@
{
"name": "esp-box-3",
"sdkconfig_append": [
"CONFIG_USE_DEVICE_AEC=y",
"CONFIG_USE_EMOTE_MESSAGE_STYLE=y",
"CONFIG_FLASH_CUSTOM_ASSETS=y",
"CONFIG_CUSTOM_ASSETS_FILE=\"https://dl.espressif.com/AE/wn9_nihaoxiaozhi_tts-font_puhui_common_20_4-esp-box-3.bin\""
"CONFIG_USE_DEVICE_AEC=y"
]
}
]

View File

@ -0,0 +1,48 @@
# M5Stack Cardputer Adv
M5Stack Cardputer Adv 是一款基于 ESP32-S3FN8 (Stamp-S3A) 的卡片式电脑。
## 硬件规格
| 组件 | 规格 |
|------|------|
| MCU | ESP32-S3FN8 @ 240MHz |
| Flash | 8MB |
| 显示屏 | ST7789V2 1.14" 240x135 |
| 音频编解码 | ES8311 |
| 功放 | NS4150B |
| 麦克风 | MEMS |
| 键盘 | 56键 (TCA8418) |
| IMU | BMI270 |
| 电池 | 1750mAh |
## 引脚定义
### 显示屏 (ST7789V2)
| 功能 | GPIO |
|------|------|
| MOSI | GPIO35 |
| SCLK | GPIO36 |
| CS | GPIO37 |
| DC | GPIO34 |
| RST | GPIO33 |
| BL | GPIO38 |
### 音频 (ES8311)
| 功能 | GPIO |
|------|------|
| I2C SDA | GPIO8 |
| I2C SCL | GPIO9 |
| I2S BCLK | GPIO41 |
| I2S LRCK | GPIO43 |
| I2S DOUT | GPIO46 |
| I2S DIN | GPIO42 |
## 使用方法
1. 按下 BOOT 按钮进入配网模式
2. 连接 WiFi 后即可使用语音助手功能
## 参考链接
- [M5Stack Cardputer Adv 官方文档](https://docs.m5stack.com/en/core/Cardputer-Adv)

View File

@ -0,0 +1,58 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
// M5Stack Cardputer Adv Board configuration
// MCU: ESP32-S3FN8 (Stamp-S3A)
// Display: ST7789V2 1.14" 240x135
// Audio: ES8311 + NS4150B
#include <driver/gpio.h>
// Audio settings
#define AUDIO_INPUT_SAMPLE_RATE 24000
#define AUDIO_OUTPUT_SAMPLE_RATE 24000
// I2S Audio pins (ES8311)
#define AUDIO_I2S_GPIO_MCLK GPIO_NUM_NC
#define AUDIO_I2S_GPIO_BCLK GPIO_NUM_41 // SCLK
#define AUDIO_I2S_GPIO_WS GPIO_NUM_43 // LRCK
#define AUDIO_I2S_GPIO_DOUT GPIO_NUM_42 // DSDIN (MCU -> ES8311)
#define AUDIO_I2S_GPIO_DIN GPIO_NUM_46 // ASDOUT (ES8311 -> MCU)
// I2C pins (shared for ES8311, TCA8418, BMI270)
#define AUDIO_CODEC_I2C_SDA_PIN GPIO_NUM_8
#define AUDIO_CODEC_I2C_SCL_PIN GPIO_NUM_9
#define AUDIO_CODEC_ES8311_ADDR ES8311_CODEC_DEFAULT_ADDR
#define AUDIO_CODEC_PA_PIN GPIO_NUM_NC // NS4150B is always on
// Button
#define BOOT_BUTTON_GPIO GPIO_NUM_0
#define BUILTIN_LED_GPIO GPIO_NUM_NC
#define VOLUME_UP_BUTTON_GPIO GPIO_NUM_NC
#define VOLUME_DOWN_BUTTON_GPIO GPIO_NUM_NC
// Display ST7789V2 (SPI)
#define DISPLAY_WIDTH 240
#define DISPLAY_HEIGHT 135
#define DISPLAY_MIRROR_X true
#define DISPLAY_MIRROR_Y false
#define DISPLAY_SWAP_XY true
#define DISPLAY_OFFSET_X 40
#define DISPLAY_OFFSET_Y 52
#define DISPLAY_SPI_MOSI_PIN GPIO_NUM_35
#define DISPLAY_SPI_SCLK_PIN GPIO_NUM_36
#define DISPLAY_SPI_CS_PIN GPIO_NUM_37
#define DISPLAY_DC_PIN GPIO_NUM_34
#define DISPLAY_RST_PIN GPIO_NUM_33
#define DISPLAY_BACKLIGHT_PIN GPIO_NUM_38
#define DISPLAY_BACKLIGHT_OUTPUT_INVERT false
// Keyboard TCA8418 I2C address
#define KEYBOARD_TCA8418_ADDR 0x34
// IMU BMI270 I2C address
#define IMU_BMI270_ADDR 0x68
#endif // _BOARD_CONFIG_H_

View File

@ -0,0 +1,13 @@
{
"target": "esp32s3",
"builds": [
{
"name": "m5stack-cardputer-adv",
"sdkconfig_append": [
"CONFIG_SPIRAM=n",
"CONFIG_ESPTOOLPY_FLASHSIZE_8MB=y",
"CONFIG_PARTITION_TABLE_CUSTOM_FILENAME=\"partitions/v2/8m.csv\""
]
}
]
}

View File

@ -0,0 +1,158 @@
#include "wifi_board.h"
#include "codecs/es8311_audio_codec.h"
#include "display/lcd_display.h"
#include "application.h"
#include "button.h"
#include "config.h"
#include "i2c_device.h"
#include <esp_log.h>
#include <driver/i2c_master.h>
#include <driver/spi_common.h>
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <esp_lcd_panel_vendor.h>
#define TAG "CardputerAdv"
class M5StackCardputerAdvBoard : public WifiBoard {
private:
i2c_master_bus_handle_t i2c_bus_;
LcdDisplay* display_;
Button boot_button_;
esp_lcd_panel_io_handle_t panel_io_ = nullptr;
esp_lcd_panel_handle_t panel_ = nullptr;
void InitializeI2c() {
ESP_LOGI(TAG, "Initialize I2C bus");
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_0,
.sda_io_num = AUDIO_CODEC_I2C_SDA_PIN,
.scl_io_num = AUDIO_CODEC_I2C_SCL_PIN,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.enable_internal_pullup = 1,
},
};
ESP_ERROR_CHECK(i2c_new_master_bus(&i2c_bus_cfg, &i2c_bus_));
}
void I2cDetect() {
uint8_t address;
ESP_LOGI(TAG, "I2C device scan:");
printf(" 0 1 2 3 4 5 6 7 8 9 a b c d e f\r\n");
for (int i = 0; i < 128; i += 16) {
printf("%02x: ", i);
for (int j = 0; j < 16; j++) {
fflush(stdout);
address = i + j;
esp_err_t ret = i2c_master_probe(i2c_bus_, address, pdMS_TO_TICKS(200));
if (ret == ESP_OK) {
printf("%02x ", address);
} else if (ret == ESP_ERR_TIMEOUT) {
printf("UU ");
} else {
printf("-- ");
}
}
printf("\r\n");
}
}
void InitializeSpi() {
ESP_LOGI(TAG, "Initialize SPI bus");
spi_bus_config_t buscfg = {};
buscfg.mosi_io_num = DISPLAY_SPI_MOSI_PIN;
buscfg.miso_io_num = GPIO_NUM_NC;
buscfg.sclk_io_num = DISPLAY_SPI_SCLK_PIN;
buscfg.quadwp_io_num = GPIO_NUM_NC;
buscfg.quadhd_io_num = GPIO_NUM_NC;
buscfg.max_transfer_sz = DISPLAY_WIDTH * DISPLAY_HEIGHT * sizeof(uint16_t);
ESP_ERROR_CHECK(spi_bus_initialize(SPI3_HOST, &buscfg, SPI_DMA_CH_AUTO));
}
void InitializeSt7789Display() {
ESP_LOGI(TAG, "Initialize ST7789V2 display");
esp_lcd_panel_io_spi_config_t io_config = {};
io_config.cs_gpio_num = DISPLAY_SPI_CS_PIN;
io_config.dc_gpio_num = DISPLAY_DC_PIN;
io_config.spi_mode = 0;
io_config.pclk_hz = 40 * 1000 * 1000;
io_config.trans_queue_depth = 10;
io_config.lcd_cmd_bits = 8;
io_config.lcd_param_bits = 8;
io_config.flags.sio_mode = 1; // 3-wire SPI mode (M5GFX uses spi_3wire = true)
ESP_ERROR_CHECK(esp_lcd_new_panel_io_spi(SPI3_HOST, &io_config, &panel_io_));
ESP_LOGI(TAG, "Install ST7789 panel driver");
esp_lcd_panel_dev_config_t panel_config = {};
panel_config.reset_gpio_num = DISPLAY_RST_PIN;
panel_config.rgb_ele_order = LCD_RGB_ELEMENT_ORDER_RGB;
panel_config.bits_per_pixel = 16;
ESP_ERROR_CHECK(esp_lcd_new_panel_st7789(panel_io_, &panel_config, &panel_));
ESP_ERROR_CHECK(esp_lcd_panel_reset(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_init(panel_));
ESP_ERROR_CHECK(esp_lcd_panel_invert_color(panel_, true));
ESP_ERROR_CHECK(esp_lcd_panel_swap_xy(panel_, DISPLAY_SWAP_XY));
ESP_ERROR_CHECK(esp_lcd_panel_mirror(panel_, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y));
display_ = new SpiLcdDisplay(panel_io_, panel_,
DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY);
}
void InitializeButtons() {
boot_button_.OnClick([this]() {
auto& app = Application::GetInstance();
if (app.GetDeviceState() == kDeviceStateStarting) {
EnterWifiConfigMode();
return;
}
app.ToggleChatState();
});
}
public:
M5StackCardputerAdvBoard() : boot_button_(BOOT_BUTTON_GPIO) {
InitializeI2c();
I2cDetect();
InitializeSpi();
InitializeSt7789Display();
InitializeButtons();
GetBacklight()->RestoreBrightness();
}
virtual AudioCodec* GetAudioCodec() override {
static Es8311AudioCodec audio_codec(
i2c_bus_,
I2C_NUM_0,
AUDIO_INPUT_SAMPLE_RATE,
AUDIO_OUTPUT_SAMPLE_RATE,
AUDIO_I2S_GPIO_MCLK,
AUDIO_I2S_GPIO_BCLK,
AUDIO_I2S_GPIO_WS,
AUDIO_I2S_GPIO_DOUT,
AUDIO_I2S_GPIO_DIN,
AUDIO_CODEC_PA_PIN,
AUDIO_CODEC_ES8311_ADDR,
false); // use_mclk = false, Cardputer Adv has no MCLK pin
return &audio_codec;
}
virtual Display* GetDisplay() override {
return display_;
}
virtual Backlight* GetBacklight() override {
// M5GFX uses 256Hz PWM frequency for Cardputer backlight
static PwmBacklight backlight(DISPLAY_BACKLIGHT_PIN, DISPLAY_BACKLIGHT_OUTPUT_INVERT, 256);
return &backlight;
}
};
DECLARE_BOARD(M5StackCardputerAdvBoard);

View File

@ -1,8 +1,27 @@
#ifndef _BOARD_CONFIG_H_
#define _BOARD_CONFIG_H_
#include <driver/gpio.h>
#include <driver/adc.h>
#include <driver/gpio.h>
#define OTTO_VERSION_AUTO 0
#define OTTO_VERSION_CAMERA 1
#define OTTO_VERSION_NO_CAMERA 2
#ifndef OTTO_HARDWARE_VERSION
#define OTTO_HARDWARE_VERSION OTTO_VERSION_AUTO
#endif
enum OttoCameraType {
OTTO_CAMERA_NONE = 0,
OTTO_CAMERA_OV2640 = 1,
OTTO_CAMERA_OV3660 = 2,
OTTO_CAMERA_UNKNOWN = 99,
};
#define OV2640_PID_1 0x2640
#define OV2640_PID_2 0x2626
#define OV3660_PID 0x3660
struct HardwareConfig {
gpio_num_t power_charge_detect_pin;

View File

@ -7,7 +7,10 @@
"CONFIG_HTTPD_WS_SUPPORT=y",
"CONFIG_CAMERA_OV2640=y",
"CONFIG_CAMERA_OV2640_AUTO_DETECT_DVP_INTERFACE_SENSOR=y",
"CONFIG_CAMERA_OV2640_DVP_YUV422_240X240_25FPS=y"
"CONFIG_CAMERA_OV2640_DVP_YUV422_240X240_25FPS=y",
"CONFIG_CAMERA_OV3660=y",
"CONFIG_CAMERA_OV3660_AUTO_DETECT_DVP_INTERFACE_SENSOR=y",
"CONFIG_CAMERA_OV3660_DVP_YUV422_240X240_24FPS=y"
]
}
]

View File

@ -3,12 +3,13 @@
#include <esp_log.h>
#include <cstring>
#include <vector>
#include "assets.h"
#include "assets/lang_config.h"
#include "display/lvgl_display/emoji_collection.h"
#include "display/lvgl_display/lvgl_image.h"
#include "display/lvgl_display/lvgl_theme.h"
#include "otto_emoji_gif.h"
#define TAG "OttoEmojiDisplay"
OttoEmojiDisplay::OttoEmojiDisplay(esp_lcd_panel_io_handle_t panel_io, esp_lcd_panel_handle_t panel, int width, int height, int offset_x, int offset_y, bool mirror_x, bool mirror_y, bool swap_xy)
@ -24,64 +25,12 @@ void OttoEmojiDisplay::SetupPreviewImage() {
}
void OttoEmojiDisplay::InitializeOttoEmojis() {
ESP_LOGI(TAG, "初始化Otto GIF表情");
auto otto_emoji_collection = std::make_shared<EmojiCollection>();
// 中性/平静类表情 -> staticstate
otto_emoji_collection->AddEmoji("staticstate", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("neutral", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("relaxed", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("sleepy", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
otto_emoji_collection->AddEmoji("idle", new LvglRawImage((void*)staticstate.data, staticstate.data_size));
// 积极/开心类表情 -> happy
otto_emoji_collection->AddEmoji("happy", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("laughing", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("funny", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("loving", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("confident", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("winking", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("cool", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("delicious", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("kissy", new LvglRawImage((void*)happy.data, happy.data_size));
otto_emoji_collection->AddEmoji("silly", new LvglRawImage((void*)happy.data, happy.data_size));
// 悲伤类表情 -> sad
otto_emoji_collection->AddEmoji("sad", new LvglRawImage((void*)sad.data, sad.data_size));
otto_emoji_collection->AddEmoji("crying", new LvglRawImage((void*)sad.data, sad.data_size));
// 愤怒类表情 -> anger
otto_emoji_collection->AddEmoji("anger", new LvglRawImage((void*)anger.data, anger.data_size));
otto_emoji_collection->AddEmoji("angry", new LvglRawImage((void*)anger.data, anger.data_size));
// 惊讶类表情 -> scare
otto_emoji_collection->AddEmoji("scare", new LvglRawImage((void*)scare.data, scare.data_size));
otto_emoji_collection->AddEmoji("surprised", new LvglRawImage((void*)scare.data, scare.data_size));
otto_emoji_collection->AddEmoji("shocked", new LvglRawImage((void*)scare.data, scare.data_size));
// 思考/困惑类表情 -> buxue
otto_emoji_collection->AddEmoji("buxue", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("thinking", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("confused", new LvglRawImage((void*)buxue.data, buxue.data_size));
otto_emoji_collection->AddEmoji("embarrassed", new LvglRawImage((void*)buxue.data, buxue.data_size));
// 将表情集合添加到主题中
auto& theme_manager = LvglThemeManager::GetInstance();
auto light_theme = theme_manager.GetTheme("light");
auto dark_theme = theme_manager.GetTheme("dark");
if (light_theme != nullptr) {
light_theme->set_emoji_collection(otto_emoji_collection);
}
if (dark_theme != nullptr) {
dark_theme->set_emoji_collection(otto_emoji_collection);
}
ESP_LOGI(TAG, "Otto表情初始化将由Assets系统处理");
// 表情初始化已移至assets系统,通过DEFAULT_EMOJI_COLLECTION=otto-gif配置
// assets.cc会从assets分区加载GIF表情并设置到theme
// 设置默认表情为staticstate
SetEmotion("staticstate");
ESP_LOGI(TAG, "Otto GIF表情初始化完成");
}
LV_FONT_DECLARE(OTTO_ICON_FONT);
@ -148,7 +97,7 @@ void OttoEmojiDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
auto img_dsc = preview_image_cached_->image_dsc();
// 设置图片源并显示预览图片
lv_image_set_src(preview_image_, img_dsc);
lv_image_set_rotation(preview_image_, -900);
lv_image_set_rotation(preview_image_, 900);
if (img_dsc->header.w > 0 && img_dsc->header.h > 0) {
// zoom factor 1.0
lv_image_set_scale(preview_image_, 256 * width_ / img_dsc->header.w);

View File

@ -1,25 +1,25 @@
#include <driver/i2c_master.h>
#include <driver/spi_common.h>
#include <driver/ledc.h>
#include <driver/spi_common.h>
#include <esp_lcd_panel_io.h>
#include <esp_lcd_panel_ops.h>
#include <esp_lcd_panel_vendor.h>
#include <esp_log.h>
#include "application.h"
#include "codecs/no_audio_codec.h"
#include "button.h"
#include "codecs/no_audio_codec.h"
#include "config.h"
#include "display/lcd_display.h"
#include "esp_video.h"
#include "lamp_controller.h"
#include "led/single_led.h"
#include "mcp_server.h"
#include "otto_emoji_display.h"
#include "power_manager.h"
#include "system_reset.h"
#include "wifi_board.h"
#include "esp_video.h"
#include "websocket_control_server.h"
#include "wifi_board.h"
#define TAG "OttoRobot"
@ -36,6 +36,7 @@ private:
i2c_master_bus_handle_t i2c_bus_;
EspVideo* camera_;
bool has_camera_;
OttoCameraType camera_type_;
bool DetectHardwareVersion() {
ledc_timer_config_t ledc_timer = {
@ -73,7 +74,8 @@ private:
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags = {
.flags =
{
.enable_internal_pullup = 1,
},
};
@ -85,6 +87,7 @@ private:
}
const uint8_t camera_addresses[] = {0x30, 0x3C, 0x21, 0x60};
bool camera_found = false;
uint16_t detected_pid = 0;
for (size_t i = 0; i < sizeof(camera_addresses); i++) {
uint8_t addr = camera_addresses[i];
@ -97,14 +100,39 @@ private:
i2c_master_dev_handle_t dev_handle;
ret = i2c_master_bus_add_device(i2c_bus_, &dev_cfg, &dev_handle);
if (ret == ESP_OK) {
uint8_t reg_addr = 0x0A;
uint8_t data[2];
ret = i2c_master_transmit_receive(dev_handle, &reg_addr, 1, data, 2, 200);
if (ret == ESP_OK) {
uint8_t data[2] = {0, 0};
uint8_t reg_addr_8bit = 0x0A;
ret = i2c_master_transmit_receive(dev_handle, &reg_addr_8bit, 1, data, 2, 200);
if (ret == ESP_OK && (data[0] != 0 || data[1] != 0)) {
detected_pid = (data[0] << 8) | data[1];
ESP_LOGI(TAG, "检测到摄像头 (OV2640方式) PID=0x%04X (地址=0x%02X)",
detected_pid, addr);
camera_found = true;
i2c_master_bus_rm_device(dev_handle);
break;
}
uint8_t reg_addr_high[2] = {0x30, 0x0A};
uint8_t reg_addr_low[2] = {0x30, 0x0B};
uint8_t pid_high = 0, pid_low = 0;
ret = i2c_master_transmit_receive(dev_handle, reg_addr_high, 2, &pid_high, 1, 200);
if (ret == ESP_OK) {
ret =
i2c_master_transmit_receive(dev_handle, reg_addr_low, 2, &pid_low, 1, 200);
if (ret == ESP_OK) {
detected_pid = (pid_high << 8) | pid_low;
if (detected_pid != 0) {
ESP_LOGI(TAG, "检测到摄像头 (OV3660方式) PID=0x%04X (地址=0x%02X)",
detected_pid, addr);
camera_found = true;
i2c_master_bus_rm_device(dev_handle);
break;
}
}
}
i2c_master_bus_rm_device(dev_handle);
}
}
@ -113,16 +141,26 @@ private:
i2c_del_master_bus(i2c_bus_);
i2c_bus_ = nullptr;
ledc_stop(LEDC_LOW_SPEED_MODE, LEDC_CHANNEL, 0);
camera_type_ = OTTO_CAMERA_NONE;
} else {
// 根据 PID 判断摄像头类型
if (detected_pid == OV2640_PID_1 || detected_pid == OV2640_PID_2) {
camera_type_ = OTTO_CAMERA_OV2640;
ESP_LOGI(TAG, "摄像头类型: OV2640 (PID=0x%04X)", detected_pid);
} else if (detected_pid == OV3660_PID) {
camera_type_ = OTTO_CAMERA_OV3660;
ESP_LOGI(TAG, "摄像头类型: OV3660 (PID=0x%04X)", detected_pid);
} else {
camera_type_ = OTTO_CAMERA_UNKNOWN;
ESP_LOGW(TAG, "未知摄像头类型PID=0x%04X", detected_pid);
}
}
return camera_found;
}
void InitializePowerManager() {
power_manager_ = new PowerManager(
hw_config_.power_charge_detect_pin,
hw_config_.power_adc_unit,
hw_config_.power_adc_channel
);
power_manager_ = new PowerManager(hw_config_.power_charge_detect_pin,
hw_config_.power_adc_unit, hw_config_.power_adc_channel);
}
void InitializeSpi() {
@ -163,9 +201,9 @@ private:
esp_lcd_panel_swap_xy(panel, DISPLAY_SWAP_XY);
esp_lcd_panel_mirror(panel, DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y);
display_ = new OttoEmojiDisplay(
panel_io, panel, DISPLAY_WIDTH, DISPLAY_HEIGHT, DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y,
DISPLAY_MIRROR_X, DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY);
display_ = new OttoEmojiDisplay(panel_io, panel, DISPLAY_WIDTH, DISPLAY_HEIGHT,
DISPLAY_OFFSET_X, DISPLAY_OFFSET_Y, DISPLAY_MIRROR_X,
DISPLAY_MIRROR_Y, DISPLAY_SWAP_XY);
}
void InitializeButtons() {
@ -179,17 +217,14 @@ private:
});
}
void InitializeOttoController() {
::InitializeOttoController(hw_config_);
}
void InitializeOttoController() { ::InitializeOttoController(hw_config_); }
public:
const HardwareConfig& GetHardwareConfig() const {
return hw_config_;
}
const HardwareConfig& GetHardwareConfig() const { return hw_config_; }
OttoCameraType GetCameraType() const { return camera_type_; }
private:
void InitializeWebSocketControlServer() {
ws_control_server_ = new WebSocketControlServer();
if (!ws_control_server_->Start(8080)) {
@ -213,7 +248,8 @@ private:
try {
static esp_cam_ctlr_dvp_pin_config_t dvp_pin_config = {
.data_width = CAM_CTLR_DATA_WIDTH_8,
.data_io = {
.data_io =
{
[0] = CAMERA_D0,
[1] = CAMERA_D1,
[2] = CAMERA_D2,
@ -248,7 +284,21 @@ private:
};
camera_ = new EspVideo(video_config);
// 根据摄像头类型设置不同的翻转参数
switch (camera_type_) {
case OTTO_CAMERA_OV3660:
camera_->SetVFlip(true);
camera_->SetHMirror(true);
ESP_LOGI(TAG, "OV3660: 设置 VFlip=true, HMirror=true");
break;
case OTTO_CAMERA_OV2640:
default:
camera_->SetVFlip(true);
camera_->SetHMirror(false);
ESP_LOGI(TAG, "OV2640: 设置 VFlip=true, HMirror=false");
break;
}
return true;
} catch (...) {
camera_ = nullptr;
@ -259,42 +309,71 @@ private:
void InitializeAudioCodec() {
if (hw_config_.audio_use_simplex) {
audio_codec_ = new NoAudioCodecSimplex(
hw_config_.audio_input_sample_rate,
hw_config_.audio_output_sample_rate,
hw_config_.audio_i2s_spk_gpio_bclk,
hw_config_.audio_i2s_spk_gpio_lrck,
hw_config_.audio_i2s_spk_gpio_dout,
hw_config_.audio_i2s_mic_gpio_sck,
hw_config_.audio_i2s_mic_gpio_ws,
hw_config_.audio_i2s_mic_gpio_din
);
hw_config_.audio_input_sample_rate, hw_config_.audio_output_sample_rate,
hw_config_.audio_i2s_spk_gpio_bclk, hw_config_.audio_i2s_spk_gpio_lrck,
hw_config_.audio_i2s_spk_gpio_dout, hw_config_.audio_i2s_mic_gpio_sck,
hw_config_.audio_i2s_mic_gpio_ws, hw_config_.audio_i2s_mic_gpio_din);
} else {
audio_codec_ = new NoAudioCodecDuplex(
hw_config_.audio_input_sample_rate,
hw_config_.audio_output_sample_rate,
hw_config_.audio_i2s_gpio_bclk,
hw_config_.audio_i2s_gpio_ws,
hw_config_.audio_i2s_gpio_dout,
hw_config_.audio_i2s_gpio_din
);
hw_config_.audio_input_sample_rate, hw_config_.audio_output_sample_rate,
hw_config_.audio_i2s_gpio_bclk, hw_config_.audio_i2s_gpio_ws,
hw_config_.audio_i2s_gpio_dout, hw_config_.audio_i2s_gpio_din);
}
}
public:
OttoRobot() : boot_button_(BOOT_BUTTON_GPIO),
OttoRobot()
: boot_button_(BOOT_BUTTON_GPIO),
audio_codec_(nullptr),
i2c_bus_(nullptr),
camera_(nullptr),
has_camera_(false) {
has_camera_(false),
camera_type_(OTTO_CAMERA_NONE) {
#if OTTO_HARDWARE_VERSION == OTTO_VERSION_AUTO
// 自动检测硬件版本(同时检测摄像头类型)
has_camera_ = DetectHardwareVersion();
ESP_LOGI(TAG, "自动检测硬件版本: %s", has_camera_ ? "摄像头版" : "无摄像头版");
#elif OTTO_HARDWARE_VERSION == OTTO_VERSION_CAMERA
// 强制使用摄像头版本,但仍检测具体摄像头类型
has_camera_ = DetectHardwareVersion();
if (!has_camera_) {
// 检测失败时仍使用摄像头配置,但不知道具体类型
has_camera_ = true;
camera_type_ = OTTO_CAMERA_UNKNOWN;
ESP_LOGW(TAG, "强制使用摄像头版本配置,但未能检测到摄像头类型");
// 初始化 I2C 总线用于摄像头
i2c_master_bus_config_t i2c_bus_cfg = {
.i2c_port = I2C_NUM_0,
.sda_io_num = CAMERA_VERSION_CONFIG.i2c_sda_pin,
.scl_io_num = CAMERA_VERSION_CONFIG.i2c_scl_pin,
.clk_source = I2C_CLK_SRC_DEFAULT,
.glitch_ignore_cnt = 7,
.intr_priority = 0,
.trans_queue_depth = 0,
.flags =
{
.enable_internal_pullup = 1,
},
};
i2c_new_master_bus(&i2c_bus_cfg, &i2c_bus_);
} else {
ESP_LOGI(TAG, "强制使用摄像头版本配置");
}
#elif OTTO_HARDWARE_VERSION == OTTO_VERSION_NO_CAMERA
// 强制使用无摄像头版本
has_camera_ = false;
camera_type_ = OTTO_CAMERA_NONE;
ESP_LOGI(TAG, "强制使用无摄像头版本配置");
#else
#error \
"OTTO_HARDWARE_VERSION 设置无效,请使用 OTTO_VERSION_AUTO, OTTO_VERSION_CAMERA 或 OTTO_VERSION_NO_CAMERA"
#endif
if (has_camera_)
hw_config_ = CAMERA_VERSION_CONFIG;
else
hw_config_ = NON_CAMERA_VERSION_CONFIG;
InitializeSpi();
InitializeLcdDisplay();
InitializeButtons();
@ -312,18 +391,15 @@ public:
GetBacklight()->RestoreBrightness();
}
virtual AudioCodec *GetAudioCodec() override {
return audio_codec_;
}
virtual AudioCodec* GetAudioCodec() override { return audio_codec_; }
virtual Display* GetDisplay() override {
return display_;
}
virtual Display* GetDisplay() override { return display_; }
virtual Backlight* GetBacklight() override {
static PwmBacklight* backlight = nullptr;
if (backlight == nullptr) {
backlight = new PwmBacklight(hw_config_.display_backlight_pin, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
backlight =
new PwmBacklight(hw_config_.display_backlight_pin, DISPLAY_BACKLIGHT_OUTPUT_INVERT);
}
return backlight;
}
@ -335,9 +411,7 @@ public:
return true;
}
virtual Camera *GetCamera() override {
return has_camera_ ? camera_ : nullptr;
}
virtual Camera* GetCamera() override { return has_camera_ ? camera_ : nullptr; }
};
DECLARE_BOARD(OttoRobot);

BIN
main/bridge_debug.wav Normal file

Binary file not shown.

276
main/bridge_server.py Normal file
View File

@ -0,0 +1,276 @@
import asyncio
import websockets
import os
import sys
import httpx
import json
import time
import queue
import threading
from typing import Any, Optional
from livekit import rtc
from livekit.rtc import AudioSource, AudioFrame
from websockets.exceptions import ConnectionClosedError
import http.server
import multipart
from urllib.parse import parse_qs
# 配置信息
# TOKEN_URL = "http://10.6.80.130:8000/v1/token"
# LIVEKIT_WS_URL = "ws://10.6.80.130:8000/"
# ROOM = "vera-room"
# IDENTITY = "vera-1"
# TOKEN_URL = "https://omnichat.bwgdi.com/v1/token"
TOKEN_URL = "http://10.6.80.130:8000/getToken"
LIVEKIT_WS_URL = "wss://test-b2zm4kva.livekit.cloud"
# LIVEKIT_WS_URL = "wss://rtc.bwgdi.com/"
ROOM = "test-livekit-room2"
IDENTITY = "uv-livekit-hardcoded"
import uuid
# IDENTITY = f"uv-{uuid.uuid4().hex[:6]}"
CONNECT_TIMEOUT_SECONDS = 10.0
WS_PORT = 8080
SAMPLE_RATE = 16000
async def fetch_token() -> str:
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
TOKEN_URL,
params={"room": ROOM, "identity": IDENTITY, "agent_name": "my-agent"},
)
response.raise_for_status()
payload: dict[str, Any] = response.json()
token = payload.get("token")
if not isinstance(token, str) or not token:
raise ValueError(f"token response missing token field: {payload}")
print(f"[token] room={payload.get('room')} identity={payload.get('identity')}")
print(f"[token] jwt_prefix={token[:16]}... len={len(token)}")
print(f"[token] jwt_prefix={token}")
return token
class ESP32LiveKitBridge:
def __init__(self):
self.room = rtc.Room()
# 创建一个音频源,用于将 ESP32 的声音推送到 LiveKit
# 注意:采样率需与 ESP32 发送的一致,通常是 16000 或 24000
self.mic_source = AudioSource(sample_rate=SAMPLE_RATE, num_channels=1)
self.esp_ws = None # 保存 WebSocket 连接
self.audio_queue = queue.Queue()
self.wav_writer_thread: Optional[threading.Thread] = None
self.stop_event = threading.Event()
def _wav_writer_loop(self):
import wave
print("启动音频保存线程...")
try:
with wave.open("bridge_debug.wav", "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2) # 16-bit
wav_file.setframerate(SAMPLE_RATE)
while not self.stop_event.is_set() or not self.audio_queue.empty():
try:
# 使用 timeout 避免永久阻塞,以便检查 stop_event
pcm_bytes = self.audio_queue.get(timeout=0.5)
wav_file.writeframes(pcm_bytes)
except queue.Empty:
continue
except Exception as e:
print(f"音频保存线程错误: {e}")
finally:
print("音频保存线程退出")
async def start(self):
@self.room.on("connection_state_changed")
def on_connection_state_changed(state: int) -> None:
print(f"[livekit] state={rtc.ConnectionState.Name(state)}")
# 1. 获取 Token 并连接 LiveKit
print(f"[config] livekit_ws_url={LIVEKIT_WS_URL}")
print(f"[config] token_url={TOKEN_URL}")
print(f"[config] room={ROOM} identity={IDENTITY}")
token = await fetch_token()
await asyncio.wait_for(
self.room.connect(
LIVEKIT_WS_URL,
token,
options=rtc.RoomOptions(connect_timeout=CONNECT_TIMEOUT_SECONDS),
),
timeout=CONNECT_TIMEOUT_SECONDS + 2.0,
)
print(f"已连接到 LiveKit 房间: {self.room.name}")
print(f"[livekit] local_identity={self.room.local_participant.identity}")
print(f"[livekit] local_sid={self.room.local_participant.sid}")
# 2. 发布麦克风轨道 (ESP32 -> LiveKit)
track = rtc.LocalAudioTrack.create_audio_track("esp32-mic", self.mic_source)
options = rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE)
await self.room.local_participant.publish_track(track, options)
# 3. 监听房间内的音频 (LiveKit -> ESP32)
# 当房间里有其他人(比如 AI Agent说话时触发此回调
@self.room.on("track_subscribed")
def on_track_subscribed(track, publication, participant):
if track.kind == rtc.TrackKind.KIND_AUDIO:
print(f"收到音频流: {participant.identity}")
# 启动一个任务来接收音频流并转发给 ESP32明确指定采样率为 16000Hz 以进行自动重采样
asyncio.create_task(self.forward_audio_to_esp32(rtc.AudioStream(track, sample_rate=SAMPLE_RATE, num_channels=1)))
self.agent_ready = asyncio.Event()
@self.room.on("participant_connected")
def on_participant_connected(p):
print(f"👤 participant joined: {p.identity}")
if "agent" in p.identity:
self.agent_ready.set()
print("等待 agent 加入...")
try:
await asyncio.wait_for(self.agent_ready.wait(), timeout=10)
print("✅ agent 已加入")
except asyncio.TimeoutError:
print("⚠️ agent 未加入(后续可能收不到音频)")
async def close(self):
"""优雅关闭所有连接和资源"""
self.stop_event.set()
if self.room:
await self.room.disconnect()
async def forward_audio_to_esp32(self, audio_stream):
"""从 LiveKit 接收音频,通过 WebSocket 发回给 ESP32"""
import opuslib
import json
# 创建下行 Opus 编码器
encoder = opuslib.Encoder(SAMPLE_RATE, 1, 'voip')
# 1. 告知 ESP32 开始说话,切换 UI 到“说话中”并准备解码
if self.esp_ws:
await self.esp_ws.send(json.dumps({"type": "tts", "state": "start"}))
try:
async for event in audio_stream:
if self.esp_ws:
try:
# AudioStream 迭代产生的是 AudioFrameEvent需要从中提取 frame
frame = event.frame
# 将 PCM 编码为 Opus 才能发给 ESP32
pcm_data = frame.data.tobytes()
# 使用当前帧的实际采样数进行编码
opus_packet = encoder.encode(pcm_data, frame.samples_per_channel)
await self.esp_ws.send(opus_packet)
except Exception as e:
print(f"发送回 ESP32 失败: {e}")
finally:
# 2. 音频流结束,告知 ESP32 停止说话,切换回聆听或闲置状态
if self.esp_ws:
await self.esp_ws.send(json.dumps({"type": "tts", "state": "stop"}))
async def handle_websocket(self, websocket):
"""处理来自 ESP32 的 WebSocket 连接"""
self.esp_ws = websocket
print("ESP32 已连接")
opus_decoder = None
try:
# 发送 hello 告诉 ESP32 握手成功
hello_msg = {
"type": "hello",
"transport": "websocket",
"audio_params": {
"format": "opus", # 明确要求 ESP32 发送 Opus
"sample_rate": SAMPLE_RATE,
"channels": 1,
"frame_duration": 60
}
}
import json
await websocket.send(json.dumps(hello_msg))
async for message in websocket:
# 接收 ESP32 的数据 -> 推送到 LiveKit
if isinstance(message, bytes):
# 判断如果消息长度极其短并且不是合理的音频流可能是ping包等
if len(message) < 4:
print(f"收到过短的字节消息 ({len(message)} bytes),跳过")
continue
# ESP32 默认使用 websocket_protocol version=1 (见 websocket_protocol.cc)
# 这个版本下,没有 4 字节的 header接收到的就是原生的 Opus 数据帧。
# 直接丢给 opuslib 解码即可。
audio_data = message
print(f"收到音频包长度: {len(message)}")
if audio_data:
try:
# Create Opus decoder if not exists
if opus_decoder is None:
import opuslib
print(f"初始化 Opus 解码器: {SAMPLE_RATE}Hz, mono")
opus_decoder = opuslib.Decoder(SAMPLE_RATE, 1)
# 启动音频保存线程
self.stop_event.clear()
thread = threading.Thread(target=self._wav_writer_loop, daemon=True)
self.wav_writer_thread = thread
thread.start()
# Decode Opus packet.
# Frame size for 60ms is SAMPLE_RATE * 0.06
frame_size = int(SAMPLE_RATE * 0.06)
pcm_bytes = opus_decoder.decode(audio_data, frame_size)
# 将音频数据放入队列由后台线程保存
self.audio_queue.put(pcm_bytes)
num_samples = len(pcm_bytes) // 2
if num_samples > 0:
frame = AudioFrame.create(sample_rate=SAMPLE_RATE, num_channels=1, samples_per_channel=num_samples)
# Use memoryview to safely copy bytes into the frame data
memoryview(frame.data).cast('B')[:] = pcm_bytes
# 将 capture_frame 放入当前事件循环的任务中
await self.mic_source.capture_frame(frame)
except Exception as e:
print(f"Opus audio decode error ({len(message)} bytes): {e}")
elif isinstance(message, str):
import json
try:
data = json.loads(message)
print(f"收到 ESP32 JSON 消息: {data}")
except json.JSONDecodeError:
print(f"收到未知的字符消息: {message}")
except ConnectionClosedError as e:
print(f"ESP32 异常断开: {e}")
except Exception as e:
print(f"WebSocket 其他错误: {e}")
finally:
print("ESP32 断开连接")
self.esp_ws = None
if hasattr(self, "wav_writer_thread") and self.wav_writer_thread:
self.stop_event.set()
# 我们不一定需要 join因为是 daemon=True
# 但这里设置 stop_event 会让线程在完成队列后退出
async def main():
bridge = ESP32LiveKitBridge()
try:
await bridge.start()
# 启动 WebSocket 服务器
async with websockets.serve(bridge.handle_websocket, "0.0.0.0", WS_PORT):
print(f"WebSocket 服务器运行在端口 {WS_PORT},等待 ESP32 连接...")
await asyncio.Future() # 保持运行
finally:
await bridge.close()
if __name__ == "__main__":
try:
asyncio.run(main())
except Exception as exc:
print(f"[error] {exc}", file=sys.stderr)
sys.exit(1)

276
main/bridge_server_bak.py Normal file
View File

@ -0,0 +1,276 @@
import asyncio
import websockets
import os
import sys
import httpx
import json
import time
import queue
import threading
from typing import Any, Optional
from livekit import rtc
from livekit.rtc import AudioSource, AudioFrame
from websockets.exceptions import ConnectionClosedError
import http.server
import multipart
from urllib.parse import parse_qs
# 配置信息
# TOKEN_URL = "http://10.6.80.130:8000/v1/token"
# LIVEKIT_WS_URL = "ws://10.6.80.130:8000/"
# ROOM = "vera-room"
# IDENTITY = "vera-1"
# TOKEN_URL = "https://omnichat.bwgdi.com/v1/token"
TOKEN_URL = "http://10.6.80.130:8000/getToken"
LIVEKIT_WS_URL = "wss://test-b2zm4kva.livekit.cloud"
# LIVEKIT_WS_URL = "wss://rtc.bwgdi.com/"
ROOM = "test-livekit-room2"
IDENTITY = "uv-livekit-hardcoded"
import uuid
# IDENTITY = f"uv-{uuid.uuid4().hex[:6]}"
CONNECT_TIMEOUT_SECONDS = 10.0
WS_PORT = 8080
SAMPLE_RATE = 16000
async def fetch_token() -> str:
async with httpx.AsyncClient(timeout=15.0, follow_redirects=True) as client:
response = await client.get(
TOKEN_URL,
params={"room": ROOM, "identity": IDENTITY, "agent_name": "my-agent"},
)
response.raise_for_status()
payload: dict[str, Any] = response.json()
token = payload.get("token")
if not isinstance(token, str) or not token:
raise ValueError(f"token response missing token field: {payload}")
print(f"[token] room={payload.get('room')} identity={payload.get('identity')}")
print(f"[token] jwt_prefix={token[:16]}... len={len(token)}")
print(f"[token] jwt_prefix={token}")
return token
class ESP32LiveKitBridge:
def __init__(self):
self.room = rtc.Room()
# 创建一个音频源,用于将 ESP32 的声音推送到 LiveKit
# 注意:采样率需与 ESP32 发送的一致,通常是 16000 或 24000
self.mic_source = AudioSource(sample_rate=SAMPLE_RATE, num_channels=1)
self.esp_ws = None # 保存 WebSocket 连接
self.audio_queue = queue.Queue()
self.wav_writer_thread: Optional[threading.Thread] = None
self.stop_event = threading.Event()
def _wav_writer_loop(self):
import wave
print("启动音频保存线程...")
try:
with wave.open("bridge_debug.wav", "wb") as wav_file:
wav_file.setnchannels(1)
wav_file.setsampwidth(2) # 16-bit
wav_file.setframerate(SAMPLE_RATE)
while not self.stop_event.is_set() or not self.audio_queue.empty():
try:
# 使用 timeout 避免永久阻塞,以便检查 stop_event
pcm_bytes = self.audio_queue.get(timeout=0.5)
wav_file.writeframes(pcm_bytes)
except queue.Empty:
continue
except Exception as e:
print(f"音频保存线程错误: {e}")
finally:
print("音频保存线程退出")
async def start(self):
@self.room.on("connection_state_changed")
def on_connection_state_changed(state: int) -> None:
print(f"[livekit] state={rtc.ConnectionState.Name(state)}")
# 1. 获取 Token 并连接 LiveKit
print(f"[config] livekit_ws_url={LIVEKIT_WS_URL}")
print(f"[config] token_url={TOKEN_URL}")
print(f"[config] room={ROOM} identity={IDENTITY}")
token = await fetch_token()
await asyncio.wait_for(
self.room.connect(
LIVEKIT_WS_URL,
token,
options=rtc.RoomOptions(connect_timeout=CONNECT_TIMEOUT_SECONDS),
),
timeout=CONNECT_TIMEOUT_SECONDS + 2.0,
)
print(f"已连接到 LiveKit 房间: {self.room.name}")
print(f"[livekit] local_identity={self.room.local_participant.identity}")
print(f"[livekit] local_sid={self.room.local_participant.sid}")
# 2. 发布麦克风轨道 (ESP32 -> LiveKit)
track = rtc.LocalAudioTrack.create_audio_track("esp32-mic", self.mic_source)
options = rtc.TrackPublishOptions(source=rtc.TrackSource.SOURCE_MICROPHONE)
await self.room.local_participant.publish_track(track, options)
# 3. 监听房间内的音频 (LiveKit -> ESP32)
# 当房间里有其他人(比如 AI Agent说话时触发此回调
@self.room.on("track_subscribed")
def on_track_subscribed(track, publication, participant):
if track.kind == rtc.TrackKind.KIND_AUDIO:
print(f"收到音频流: {participant.identity}")
# 启动一个任务来接收音频流并转发给 ESP32明确指定采样率为 16000Hz 以进行自动重采样
asyncio.create_task(self.forward_audio_to_esp32(rtc.AudioStream(track, sample_rate=SAMPLE_RATE, num_channels=1)))
self.agent_ready = asyncio.Event()
@self.room.on("participant_connected")
def on_participant_connected(p):
print(f"👤 participant joined: {p.identity}")
if "agent" in p.identity:
self.agent_ready.set()
print("等待 agent 加入...")
try:
await asyncio.wait_for(self.agent_ready.wait(), timeout=10)
print("✅ agent 已加入")
except asyncio.TimeoutError:
print("⚠️ agent 未加入(后续可能收不到音频)")
async def close(self):
"""优雅关闭所有连接和资源"""
self.stop_event.set()
if self.room:
await self.room.disconnect()
async def forward_audio_to_esp32(self, audio_stream):
"""从 LiveKit 接收音频,通过 WebSocket 发回给 ESP32"""
import opuslib
import json
# 创建下行 Opus 编码器
encoder = opuslib.Encoder(SAMPLE_RATE, 1, 'voip')
# 1. 告知 ESP32 开始说话,切换 UI 到“说话中”并准备解码
if self.esp_ws:
await self.esp_ws.send(json.dumps({"type": "tts", "state": "start"}))
try:
async for event in audio_stream:
if self.esp_ws:
try:
# AudioStream 迭代产生的是 AudioFrameEvent需要从中提取 frame
frame = event.frame
# 将 PCM 编码为 Opus 才能发给 ESP32
pcm_data = frame.data.tobytes()
# 使用当前帧的实际采样数进行编码
opus_packet = encoder.encode(pcm_data, frame.samples_per_channel)
await self.esp_ws.send(opus_packet)
except Exception as e:
print(f"发送回 ESP32 失败: {e}")
finally:
# 2. 音频流结束,告知 ESP32 停止说话,切换回聆听或闲置状态
if self.esp_ws:
await self.esp_ws.send(json.dumps({"type": "tts", "state": "stop"}))
async def handle_websocket(self, websocket):
"""处理来自 ESP32 的 WebSocket 连接"""
self.esp_ws = websocket
print("ESP32 已连接")
opus_decoder = None
try:
# 发送 hello 告诉 ESP32 握手成功
hello_msg = {
"type": "hello",
"transport": "websocket",
"audio_params": {
"format": "opus", # 明确要求 ESP32 发送 Opus
"sample_rate": SAMPLE_RATE,
"channels": 1,
"frame_duration": 60
}
}
import json
await websocket.send(json.dumps(hello_msg))
async for message in websocket:
# 接收 ESP32 的数据 -> 推送到 LiveKit
if isinstance(message, bytes):
# 判断如果消息长度极其短并且不是合理的音频流可能是ping包等
if len(message) < 4:
print(f"收到过短的字节消息 ({len(message)} bytes),跳过")
continue
# ESP32 默认使用 websocket_protocol version=1 (见 websocket_protocol.cc)
# 这个版本下,没有 4 字节的 header接收到的就是原生的 Opus 数据帧。
# 直接丢给 opuslib 解码即可。
audio_data = message
print(f"收到音频包长度: {len(message)}")
if audio_data:
try:
# Create Opus decoder if not exists
if opus_decoder is None:
import opuslib
print(f"初始化 Opus 解码器: {SAMPLE_RATE}Hz, mono")
opus_decoder = opuslib.Decoder(SAMPLE_RATE, 1)
# 启动音频保存线程
self.stop_event.clear()
thread = threading.Thread(target=self._wav_writer_loop, daemon=True)
self.wav_writer_thread = thread
thread.start()
# Decode Opus packet.
# Frame size for 60ms is SAMPLE_RATE * 0.06
frame_size = int(SAMPLE_RATE * 0.06)
pcm_bytes = opus_decoder.decode(audio_data, frame_size)
# 将音频数据放入队列由后台线程保存
self.audio_queue.put(pcm_bytes)
num_samples = len(pcm_bytes) // 2
if num_samples > 0:
frame = AudioFrame.create(sample_rate=SAMPLE_RATE, num_channels=1, samples_per_channel=num_samples)
# Use memoryview to safely copy bytes into the frame data
memoryview(frame.data).cast('B')[:] = pcm_bytes
# 将 capture_frame 放入当前事件循环的任务中
await self.mic_source.capture_frame(frame)
except Exception as e:
print(f"Opus audio decode error ({len(message)} bytes): {e}")
elif isinstance(message, str):
import json
try:
data = json.loads(message)
print(f"收到 ESP32 JSON 消息: {data}")
except json.JSONDecodeError:
print(f"收到未知的字符消息: {message}")
except ConnectionClosedError as e:
print(f"ESP32 异常断开: {e}")
except Exception as e:
print(f"WebSocket 其他错误: {e}")
finally:
print("ESP32 断开连接")
self.esp_ws = None
if hasattr(self, "wav_writer_thread") and self.wav_writer_thread:
self.stop_event.set()
# 我们不一定需要 join因为是 daemon=True
# 但这里设置 stop_event 会让线程在完成队列后退出
async def main():
bridge = ESP32LiveKitBridge()
try:
await bridge.start()
# 启动 WebSocket 服务器
async with websockets.serve(bridge.handle_websocket, "0.0.0.0", WS_PORT):
print(f"WebSocket 服务器运行在端口 {WS_PORT},等待 ESP32 连接...")
await asyncio.Future() # 保持运行
finally:
await bridge.close()
if __name__ == "__main__":
try:
asyncio.run(main())
except Exception as exc:
print(f"[error] {exc}", file=sys.stderr)
sys.exit(1)

38
main/debug_connection.py Normal file
View File

@ -0,0 +1,38 @@
import asyncio
import websockets
import ssl
import sys
async def test_connect(url):
print(f"Testing connection to: {url}")
try:
# Create a custom SSL context that ignores certificate validation errors
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = ssl.CERT_NONE
async with websockets.connect(url, ssl=ssl_context) as websocket:
print(f"SUCCESS: Connected to {url}")
await websocket.close()
return True
except Exception as e:
print(f"FAILED: Could not connect to {url}. Error: {e}")
return False
async def main():
urls_to_test = [
"wss://10.6.80.12:31581",
"ws://10.6.80.12:31581",
"wss://10.6.80.12:31581",
"ws://10.6.80.12:31581",
]
print("--- Starting LiveKit Connection Probe ---")
for url in urls_to_test:
if await test_connect(url):
print(f"\nFOUND WORKING URL: {url}")
break
print("--- Probe Finished ---")
if __name__ == "__main__":
asyncio.run(main())

View File

@ -45,6 +45,10 @@ void Display::SetChatMessage(const char* role, const char* content) {
ESP_LOGW(TAG, " %s", content);
}
void Display::ClearChatMessages() {
// Default empty implementation, override in subclasses if needed
}
void Display::SetTheme(Theme* theme) {
current_theme_ = theme;
Settings settings("display", true);

View File

@ -35,6 +35,7 @@ public:
virtual void ShowNotification(const std::string &notification, int duration_ms = 3000);
virtual void SetEmotion(const char* emotion);
virtual void SetChatMessage(const char* role, const char* content);
virtual void ClearChatMessages();
virtual void SetTheme(Theme* theme);
virtual Theme* GetTheme() { return current_theme_; }
virtual void UpdateStatusBar(bool update_all = false);

View File

@ -12,6 +12,7 @@
#include <esp_lvgl_port.h>
#include <esp_psram.h>
#include <cstring>
#include <src/misc/cache/lv_cache.h>
#include "board.h"
@ -555,7 +556,6 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
}
auto lvgl_theme = static_cast<LvglTheme*>(current_theme_);
auto text_font = lvgl_theme->text_font()->font();
// Create a message bubble
lv_obj_t* msg_bubble = lv_obj_create(content_);
@ -568,28 +568,25 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
lv_obj_t* msg_text = lv_label_create(msg_bubble);
lv_label_set_text(msg_text, content);
// Calculate actual text width
lv_coord_t text_width = lv_txt_get_width(content, strlen(content), text_font, 0);
// Calculate bubble width
// Calculate bubble width constraints
lv_coord_t max_width = LV_HOR_RES * 85 / 100 - 16; // 85% of screen width
lv_coord_t min_width = 20;
lv_coord_t bubble_width;
// Let LVGL calculate the natural text width first
lv_obj_set_width(msg_text, LV_SIZE_CONTENT);
lv_obj_update_layout(msg_text);
lv_coord_t text_width = lv_obj_get_width(msg_text);
// Ensure text width is not less than minimum width
if (text_width < min_width) {
text_width = min_width;
}
// If text width is less than max width, use text width
if (text_width < max_width) {
bubble_width = text_width;
} else {
bubble_width = max_width;
}
// Constrain to max width
lv_coord_t bubble_width = (text_width < max_width) ? text_width : max_width;
// Set message text width
lv_obj_set_width(msg_text, bubble_width); // Subtract padding
lv_obj_set_width(msg_text, bubble_width);
lv_label_set_long_mode(msg_text, LV_LABEL_LONG_WRAP);
// Set bubble width
@ -776,6 +773,26 @@ void LcdDisplay::SetPreviewImage(std::unique_ptr<LvglImage> image) {
// Auto-scroll to the image bubble
lv_obj_scroll_to_view_recursive(img_bubble, LV_ANIM_ON);
}
void LcdDisplay::ClearChatMessages() {
DisplayLockGuard lock(this);
if (content_ == nullptr) {
return;
}
// Use lv_obj_clean to delete all children of content_ (chat message bubbles)
lv_obj_clean(content_);
// Reset chat_message_label_ as it has been deleted
chat_message_label_ = nullptr;
// Show the centered AI logo (emoji_label_) again
if (emoji_label_ != nullptr) {
lv_obj_remove_flag(emoji_label_, LV_OBJ_FLAG_HIDDEN);
}
ESP_LOGI(TAG, "Chat messages cleared");
}
#else
void LcdDisplay::SetupUI() {
DisplayLockGuard lock(this);
@ -893,29 +910,35 @@ void LcdDisplay::SetupUI() {
lv_label_set_text(status_label_, Lang::Strings::INITIALIZING);
lv_obj_align(status_label_, LV_ALIGN_CENTER, 0, 0);
/* Top layer: Bottom bar - fixed at bottom, minimum height 48, height can be adaptive */
/* Top layer: Bottom bar - fixed height at bottom */
bottom_bar_ = lv_obj_create(screen);
lv_obj_set_width(bottom_bar_, LV_HOR_RES);
lv_obj_set_height(bottom_bar_, LV_SIZE_CONTENT);
lv_obj_set_style_min_height(bottom_bar_, 48, 0); // Set minimum height 48
lv_obj_set_size(bottom_bar_, LV_HOR_RES, text_font->line_height + lvgl_theme->spacing(12));
lv_obj_set_style_radius(bottom_bar_, 0, 0);
lv_obj_set_style_bg_color(bottom_bar_, lvgl_theme->background_color(), 0);
lv_obj_set_style_text_color(bottom_bar_, lvgl_theme->text_color(), 0);
lv_obj_set_style_pad_top(bottom_bar_, lvgl_theme->spacing(2), 0);
lv_obj_set_style_pad_bottom(bottom_bar_, lvgl_theme->spacing(2), 0);
lv_obj_set_style_pad_all(bottom_bar_, 0, 0);
lv_obj_set_style_pad_left(bottom_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_style_pad_right(bottom_bar_, lvgl_theme->spacing(4), 0);
lv_obj_set_style_border_width(bottom_bar_, 0, 0);
lv_obj_set_scrollbar_mode(bottom_bar_, LV_SCROLLBAR_MODE_OFF);
lv_obj_align(bottom_bar_, LV_ALIGN_BOTTOM_MID, 0, 0);
/* chat_message_label_ placed in bottom_bar_ and vertically centered */
/* chat_message_label_ placed in bottom_bar_, single-line horizontal scroll */
chat_message_label_ = lv_label_create(bottom_bar_);
lv_label_set_text(chat_message_label_, "");
lv_obj_set_width(chat_message_label_, LV_HOR_RES - lvgl_theme->spacing(8)); // Subtract left and right padding
lv_label_set_long_mode(chat_message_label_, LV_LABEL_LONG_WRAP); // Auto wrap mode
lv_obj_set_style_text_align(chat_message_label_, LV_TEXT_ALIGN_CENTER, 0); // Center text alignment
lv_obj_set_width(chat_message_label_, LV_HOR_RES - lvgl_theme->spacing(8));
lv_label_set_long_mode(chat_message_label_, LV_LABEL_LONG_SCROLL_CIRCULAR);
lv_obj_set_style_text_align(chat_message_label_, LV_TEXT_ALIGN_CENTER, 0);
lv_obj_set_style_text_color(chat_message_label_, lvgl_theme->text_color(), 0);
lv_obj_align(chat_message_label_, LV_ALIGN_CENTER, 0, 0); // Vertically and horizontally centered in bottom_bar_
lv_obj_align(chat_message_label_, LV_ALIGN_CENTER, 0, 0);
// Start scrolling after a delay (short text won't scroll)
static lv_anim_t a;
lv_anim_init(&a);
lv_anim_set_delay(&a, 1000);
lv_anim_set_repeat_count(&a, LV_ANIM_REPEAT_INFINITE);
lv_obj_set_style_anim(chat_message_label_, &a, LV_PART_MAIN);
lv_obj_set_style_anim_duration(chat_message_label_, lv_anim_speed_clamped(60, 300, 60000), LV_PART_MAIN);
low_battery_popup_ = lv_obj_create(screen);
lv_obj_set_scrollbar_mode(low_battery_popup_, LV_SCROLLBAR_MODE_OFF);
@ -974,6 +997,14 @@ void LcdDisplay::SetChatMessage(const char* role, const char* content) {
}
lv_label_set_text(chat_message_label_, content);
}
void LcdDisplay::ClearChatMessages() {
DisplayLockGuard lock(this);
// In non-wechat mode, just clear the chat message label
if (chat_message_label_ != nullptr) {
lv_label_set_text(chat_message_label_, "");
}
}
#endif
void LcdDisplay::SetEmotion(const char* emotion) {
@ -1007,6 +1038,8 @@ void LcdDisplay::SetEmotion(const char* emotion) {
gif_controller_ = std::make_unique<LvglGif>(image->image_dsc());
if (gif_controller_->IsLoaded()) {
// Set loop delay to 1000ms
gif_controller_->SetLoopDelay(3000);
// Set up frame update callback
gif_controller_->SetFrameCallback([this]() {
lv_image_set_src(emoji_image_, gif_controller_->image_dsc());
@ -1113,7 +1146,7 @@ void LcdDisplay::SetTheme(Theme* theme) {
if (lv_obj_get_child_cnt(obj) > 0) {
// Might be a container, check if it's a user or system message container
// User and system message containers are transparent
lv_opa_t bg_opa = lv_obj_get_style_bg_opa(obj, 0);
lv_opa_t bg_opa = lv_obj_get_style_bg_opa(obj, LV_PART_MAIN);
if (bg_opa == LV_OPA_TRANSP) {
// This is a user or system message container
bubble = lv_obj_get_child(obj, 0);

View File

@ -49,6 +49,7 @@ public:
~LcdDisplay();
virtual void SetEmotion(const char* emotion) override;
virtual void SetChatMessage(const char* role, const char* content) override;
virtual void ClearChatMessages() override;
virtual void SetPreviewImage(std::unique_ptr<LvglImage> image) override;
// Add theme switching function

View File

@ -30,8 +30,8 @@ typedef struct Table {
static gd_GIF * gif_open(gd_GIF * gif);
static bool f_gif_open(gd_GIF * gif, const void * path, bool is_file);
static void f_gif_read(gd_GIF * gif, void * buf, size_t len);
static int f_gif_seek(gd_GIF * gif, size_t pos, int k);
static inline void f_gif_read(gd_GIF * gif, void * buf, size_t len);
static inline int f_gif_seek(gd_GIF * gif, size_t pos, int k);
static void f_gif_close(gd_GIF * gif);
#if LV_USE_DRAW_SW_ASM == LV_DRAW_SW_ASM_HELIUM

View File

@ -5,7 +5,8 @@
#define TAG "LvglGif"
LvglGif::LvglGif(const lv_img_dsc_t* img_dsc)
: gif_(nullptr), timer_(nullptr), last_call_(0), playing_(false), loaded_(false) {
: gif_(nullptr), timer_(nullptr), last_call_(0), playing_(false), loaded_(false),
loop_delay_ms_(0), loop_waiting_(false), loop_wait_start_(0) {
if (!img_dsc || !img_dsc->data) {
ESP_LOGE(TAG, "Invalid image descriptor");
return;
@ -66,6 +67,7 @@ void LvglGif::Start() {
if (timer_) {
playing_ = true;
loop_waiting_ = false; // Reset loop waiting state
last_call_ = lv_tick_get();
lv_timer_resume(timer_);
lv_timer_reset(timer_);
@ -104,9 +106,15 @@ void LvglGif::Stop() {
lv_timer_pause(timer_);
}
// Reset loop waiting state
loop_waiting_ = false;
if (gif_) {
gd_rewind(gif_);
NextFrame();
// Render first frame without advancing
if (gif_->canvas) {
gd_render_frame(gif_, gif_->canvas);
}
ESP_LOGD(TAG, "GIF animation stopped and rewound");
}
}
@ -134,6 +142,15 @@ void LvglGif::SetLoopCount(int32_t count) {
gif_->loop_count = count;
}
uint32_t LvglGif::GetLoopDelay() const {
return loop_delay_ms_;
}
void LvglGif::SetLoopDelay(uint32_t delay_ms) {
loop_delay_ms_ = delay_ms;
ESP_LOGD(TAG, "Loop delay set to %lu ms", delay_ms);
}
uint16_t LvglGif::width() const {
if (!loaded_ || !gif_) {
return 0;
@ -157,6 +174,18 @@ void LvglGif::NextFrame() {
return;
}
// Check if we're in loop wait state (only for infinite loop GIFs with delay)
if (loop_waiting_) {
uint32_t wait_elapsed = lv_tick_elaps(loop_wait_start_);
if (wait_elapsed < loop_delay_ms_) {
// Still waiting for loop delay
return;
}
// Loop delay completed, continue playing
loop_waiting_ = false;
ESP_LOGD(TAG, "Loop delay completed, continuing GIF");
}
// Check if enough time has passed for the next frame
uint32_t elapsed = lv_tick_elaps(last_call_);
if (elapsed < gif_->gce.delay * 10) {
@ -165,15 +194,30 @@ void LvglGif::NextFrame() {
last_call_ = lv_tick_get();
// Save file position before getting next frame to detect loop
uint32_t pos_before = gif_->f_rw_p;
// Get next frame
int has_next = gd_get_frame(gif_);
if (has_next == 0) {
// Animation finished, pause timer
// Animation truly finished (non-infinite loop)
playing_ = false;
if (timer_) {
lv_timer_pause(timer_);
}
ESP_LOGD(TAG, "GIF animation completed");
return;
}
// Detect loop by checking if file position jumped back (rewound to start)
// This works for looping GIFs regardless of when loop_count is set
if (loop_delay_ms_ > 0 && gif_->f_rw_p < pos_before) {
// File position decreased, meaning GIF looped back to beginning
// Start waiting before rendering this frame
loop_waiting_ = true;
loop_wait_start_ = lv_tick_get();
ESP_LOGD(TAG, "GIF completed one cycle, waiting %lu ms before next loop", loop_delay_ms_);
return;
}
// Render current frame

View File

@ -58,6 +58,17 @@ public:
*/
void SetLoopCount(int32_t count);
/**
* Get loop delay in milliseconds (delay between loops)
*/
uint32_t GetLoopDelay() const;
/**
* Set loop delay in milliseconds (delay between loops)
* @param delay_ms Delay in milliseconds before starting next loop. 0 means no delay.
*/
void SetLoopDelay(uint32_t delay_ms);
/**
* Get GIF dimensions
*/
@ -86,6 +97,11 @@ private:
bool playing_;
bool loaded_;
// Loop delay configuration
uint32_t loop_delay_ms_; // Delay between loops in milliseconds
bool loop_waiting_; // Whether we're waiting for the next loop
uint32_t loop_wait_start_; // Timestamp when loop wait started
// Frame update callback
std::function<void()> frame_callback_;

View File

@ -20,21 +20,21 @@ dependencies:
espressif/esp_lcd_panel_io_additions: ^1.0.1
78/esp_lcd_nv3023: ~1.0.0
78/esp-wifi-connect: ~3.0.2
espressif/esp_audio_effects: ~1.2.0
espressif/esp_audio_codec: ~2.4.0
78/esp-ml307: ~3.5.3
espressif/esp_audio_effects: ~1.2.1
espressif/esp_audio_codec: ~2.4.1
78/esp-ml307: ~3.6.3
78/uart-eth-modem:
version: ~0.1.3
version: ~0.3.1
rules:
- if: target not in [esp32]
78/xiaozhi-fonts: ~1.5.5
espressif/led_strip: ~3.0.1
espressif/esp_codec_dev: ~1.5
espressif/esp-sr: ~2.2.0
espressif/button: ~4.1.3
78/xiaozhi-fonts: ~1.6.0
espressif/led_strip: ~3.0.2
espressif/esp_codec_dev: ~1.5.4
espressif/esp-sr: ~2.3.0
espressif/button: ~4.1.5
espressif/knob: ^1.0.0
espressif/esp32-camera:
version: ^2.0.15
version: ^2.1.4
rules:
- if: target in [esp32s3]
espressif/esp_video:
@ -51,15 +51,15 @@ dependencies:
espressif/esp_lcd_touch_gt1151: ^1
waveshare/esp_lcd_touch_cst9217: ^1.0.3
espressif/esp_lcd_touch_cst816s: ^1.0.6
lvgl/lvgl: ~9.3.0
esp_lvgl_port: ~2.6.0
lvgl/lvgl: ~9.4.0
esp_lvgl_port: ~2.7.0
espressif/esp_io_expander_tca95xx_16bit: ^2.0.0
espressif2022/image_player: ^1.1.1
espressif2022/esp_emote_expression: ^0.1.0
espressif/adc_mic: ^0.2.1
espressif/esp_mmap_assets: '>=1.2'
txp666/otto-emoji-gif-component:
version: ^1.0.3
version: ^1.1.1
rules:
- if: target in [esp32s3]
espressif/adc_battery_estimation: ^0.2.0

View File

@ -1,27 +1,29 @@
#include "ota.h"
#include "system_info.h"
#include "settings.h"
#include "assets/lang_config.h"
#include "settings.h"
#include "system_info.h"
#include <cJSON.h>
#include <esp_log.h>
#include <esp_partition.h>
#include <esp_ota_ops.h>
#include <esp_app_format.h>
#include <esp_efuse.h>
#include <esp_efuse_table.h>
#include <esp_heap_caps.h>
#include <esp_log.h>
#include <esp_ota_ops.h>
#include <esp_partition.h>
#include <cJSON.h>
#include <freertos/FreeRTOS.h>
#include <freertos/task.h>
#ifdef SOC_HMAC_SUPPORTED
#include <esp_hmac.h>
#endif
#include <cstring>
#include <vector>
#include <sstream>
#include <algorithm>
#include <cstring>
#include <sstream>
#include <vector>
#define TAG "Ota"
Ota::Ota() {
#ifdef ESP_EFUSE_BLOCK_USR_DATA
// Read Serial Number from efuse user_data
@ -37,8 +39,7 @@ Ota::Ota() {
#endif
}
Ota::~Ota() {
}
Ota::~Ota() {}
std::string Ota::GetCheckVersionUrl() {
Settings settings("wifi", false);
@ -59,7 +60,8 @@ std::unique_ptr<Http> Ota::SetupHttp() {
http->SetHeader("Client-Id", board.GetUuid());
if (has_serial_number_) {
http->SetHeader("Serial-Number", serial_number_.c_str());
ESP_LOGI(TAG, "Setup HTTP, User-Agent: %s, Serial-Number: %s", user_agent.c_str(), serial_number_.c_str());
ESP_LOGI(TAG, "Setup HTTP, User-Agent: %s, Serial-Number: %s", user_agent.c_str(),
serial_number_.c_str());
}
http->SetHeader("User-Agent", user_agent);
http->SetHeader("Accept-Language", Lang::CODE);
@ -156,7 +158,7 @@ esp_err_t Ota::CheckVersion() {
}
}
}
has_mqtt_config_ = true;
has_mqtt_config_ = false;
} else {
ESP_LOGI(TAG, "No mqtt section found !");
}
@ -182,7 +184,12 @@ esp_err_t Ota::CheckVersion() {
ESP_LOGI(TAG, "No websocket section found!");
}
has_server_time_ = false;
// [开发调试] 强制修改 WebSocket URL 指向本地 Bridge 服务
// 请将下面的 IP 地址修改为你电脑的局域网 IP (例如 192.168.1.5)
Settings settings("websocket", true);
settings.SetString("url", "ws://10.6.80.130:8080");
has_websocket_config_ = true;
cJSON* server_time = cJSON_GetObjectItem(root, "server_time");
if (cJSON_IsObject(server_time)) {
cJSON* timestamp = cJSON_GetObjectItem(server_time, "timestamp");
@ -261,7 +268,8 @@ void Ota::MarkCurrentVersionValid() {
}
}
bool Ota::Upgrade(const std::string& firmware_url, std::function<void(int progress, size_t speed)> callback) {
bool Ota::Upgrade(const std::string& firmware_url,
std::function<void(int progress, size_t speed)> callback) {
ESP_LOGI(TAG, "Upgrading firmware from %s", firmware_url.c_str());
esp_ota_handle_t update_handle = 0;
auto update_partition = esp_ota_get_next_update_partition(NULL);
@ -270,7 +278,8 @@ bool Ota::Upgrade(const std::string& firmware_url, std::function<void(int progre
return false;
}
ESP_LOGI(TAG, "Writing to partition %s at offset 0x%lx", update_partition->label, update_partition->address);
ESP_LOGI(TAG, "Writing to partition %s at offset 0x%lx", update_partition->label,
update_partition->address);
bool image_header_checked = false;
std::string image_header;
@ -292,22 +301,32 @@ bool Ota::Upgrade(const std::string& firmware_url, std::function<void(int progre
return false;
}
char buffer[512];
constexpr size_t PAGE_SIZE = 4096;
char* buffer = (char*)heap_caps_malloc(PAGE_SIZE, MALLOC_CAP_INTERNAL);
if (buffer == nullptr) {
ESP_LOGE(TAG, "Failed to allocate buffer");
return false;
}
size_t buffer_offset = 0; // Current data size in buffer
size_t total_read = 0, recent_read = 0;
auto last_calc_time = esp_timer_get_time();
while (true) {
int ret = http->Read(buffer, sizeof(buffer));
int ret = http->Read(buffer + buffer_offset, PAGE_SIZE - buffer_offset);
if (ret < 0) {
ESP_LOGE(TAG, "Failed to read HTTP data: %s", esp_err_to_name(ret));
heap_caps_free(buffer);
return false;
}
// Calculate speed and progress every second
recent_read += ret;
total_read += ret;
buffer_offset += ret;
if (esp_timer_get_time() - last_calc_time >= 1000000 || ret == 0) {
size_t progress = total_read * 100 / content_length;
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %uB/s", progress, total_read, content_length, recent_read);
ESP_LOGI(TAG, "Progress: %u%% (%u/%u), Speed: %uB/s", progress, total_read,
content_length, recent_read);
if (callback) {
callback(progress, recent_read);
}
@ -315,22 +334,21 @@ bool Ota::Upgrade(const std::string& firmware_url, std::function<void(int progre
recent_read = 0;
}
if (ret == 0) {
break;
}
if (!image_header_checked) {
image_header.append(buffer, ret);
if (image_header.size() >= sizeof(esp_image_header_t) + sizeof(esp_image_segment_header_t) + sizeof(esp_app_desc_t)) {
image_header.append(buffer, buffer_offset);
if (image_header.size() >= sizeof(esp_image_header_t) +
sizeof(esp_image_segment_header_t) +
sizeof(esp_app_desc_t)) {
esp_app_desc_t new_app_info;
memcpy(&new_app_info, image_header.data() + sizeof(esp_image_header_t) + sizeof(esp_image_segment_header_t), sizeof(esp_app_desc_t));
auto current_version = esp_app_get_description()->version;
ESP_LOGI(TAG, "Current version: %s, New version: %s", current_version, new_app_info.version);
memcpy(&new_app_info,
image_header.data() + sizeof(esp_image_header_t) +
sizeof(esp_image_segment_header_t),
sizeof(esp_app_desc_t));
if (esp_ota_begin(update_partition, OTA_WITH_SEQUENTIAL_WRITES, &update_handle)) {
esp_ota_abort(update_handle);
ESP_LOGE(TAG, "Failed to begin OTA");
heap_caps_free(buffer);
return false;
}
@ -338,14 +356,27 @@ bool Ota::Upgrade(const std::string& firmware_url, std::function<void(int progre
std::string().swap(image_header);
}
}
auto err = esp_ota_write(update_handle, buffer, ret);
// Write to flash when buffer is full (4KB) or it's the last chunk
bool is_last_chunk = (ret == 0);
if (buffer_offset == PAGE_SIZE || (is_last_chunk && buffer_offset > 0)) {
auto err = esp_ota_write(update_handle, buffer, buffer_offset);
if (err != ESP_OK) {
ESP_LOGE(TAG, "Failed to write OTA data: %s", esp_err_to_name(err));
esp_ota_abort(update_handle);
heap_caps_free(buffer);
return false;
}
buffer_offset = 0;
}
if (is_last_chunk) {
break;
}
}
http->Close();
heap_caps_free(buffer);
esp_err_t err = esp_ota_end(update_handle);
if (err != ESP_OK) {
@ -371,7 +402,6 @@ bool Ota::StartUpgrade(std::function<void(int progress, size_t speed)> callback)
return Upgrade(firmware_url_, callback);
}
std::vector<int> Ota::ParseVersion(const std::string& version) {
std::vector<int> versionNumbers;
std::stringstream ss(version);
@ -409,7 +439,8 @@ std::string Ota::GetActivationPayload() {
uint8_t hmac_result[32]; // SHA-256 输出为32字节
// 使用Key0计算HMAC
esp_err_t ret = esp_hmac_calculate(HMAC_KEY0, (uint8_t*)activation_challenge_.data(), activation_challenge_.size(), hmac_result);
esp_err_t ret = esp_hmac_calculate(HMAC_KEY0, (uint8_t*)activation_challenge_.data(),
activation_challenge_.size(), hmac_result);
if (ret != ESP_OK) {
ESP_LOGE(TAG, "HMAC calculation failed: %s", esp_err_to_name(ret));
return "{}";
@ -464,7 +495,8 @@ esp_err_t Ota::Activate() {
return ESP_ERR_TIMEOUT;
}
if (status_code != 200) {
ESP_LOGE(TAG, "Failed to activate, code: %d, body: %s", status_code, http->ReadAll().c_str());
ESP_LOGE(TAG, "Failed to activate, code: %d, body: %s", status_code,
http->ReadAll().c_str());
return ESP_FAIL;
}

View File

@ -119,7 +119,8 @@ bool MqttProtocol::StartMqttClient(bool report_error) {
auto alive = alive_; // Capture alive flag
Application::GetInstance().Schedule([this, alive]() {
if (*alive) {
CloseAudioChannel();
// Server initiated goodbye, don't send goodbye back to avoid ping-pong
CloseAudioChannel(false);
}
});
}
@ -188,17 +189,23 @@ bool MqttProtocol::SendAudio(std::unique_ptr<AudioStreamPacket> packet) {
return udp_->Send(encrypted) > 0;
}
void MqttProtocol::CloseAudioChannel() {
void MqttProtocol::CloseAudioChannel(bool send_goodbye) {
{
std::lock_guard<std::mutex> lock(channel_mutex_);
udp_.reset();
}
ESP_LOGI(TAG, "Closing audio channel, send_goodbye: %d", send_goodbye);
// Only send goodbye when client initiates the close
// Don't send if server already sent goodbye (to avoid ping-pong)
if (send_goodbye) {
std::string message = "{";
message += "\"session_id\":\"" + session_id_ + "\",";
message += "\"type\":\"goodbye\"";
message += "}";
SendText(message);
}
if (on_audio_channel_closed_ != nullptr) {
on_audio_channel_closed_();

View File

@ -31,7 +31,7 @@ public:
bool Start() override;
bool SendAudio(std::unique_ptr<AudioStreamPacket> packet) override;
bool OpenAudioChannel() override;
void CloseAudioChannel() override;
void CloseAudioChannel(bool send_goodbye = true) override;
bool IsAudioChannelOpened() const override;
private:

View File

@ -65,7 +65,7 @@ public:
virtual bool Start() = 0;
virtual bool OpenAudioChannel() = 0;
virtual void CloseAudioChannel() = 0;
virtual void CloseAudioChannel(bool send_goodbye = true) = 0;
virtual bool IsAudioChannelOpened() const = 0;
virtual bool SendAudio(std::unique_ptr<AudioStreamPacket> packet) = 0;
virtual void SendWakeWordDetected(const std::string& wake_word);

View File

@ -75,7 +75,8 @@ bool WebsocketProtocol::IsAudioChannelOpened() const {
return websocket_ != nullptr && websocket_->IsConnected() && !error_occurred_ && !IsTimeout();
}
void WebsocketProtocol::CloseAudioChannel() {
void WebsocketProtocol::CloseAudioChannel(bool send_goodbye) {
(void)send_goodbye; // Websocket doesn't need to send goodbye message
websocket_.reset();
}

View File

@ -18,7 +18,7 @@ public:
bool Start() override;
bool SendAudio(std::unique_ptr<AudioStreamPacket> packet) override;
bool OpenAudioChannel() override;
void CloseAudioChannel() override;
void CloseAudioChannel(bool send_goodbye = true) override;
bool IsAudioChannelOpened() const override;
private:

194
main/test_client_wav.py Normal file
View File

@ -0,0 +1,194 @@
import asyncio
import os
import sys
import wave
import time
from dotenv import load_dotenv
# Try to load credentials from .env.local
load_dotenv(".env.local")
from livekit import rtc
async def publish_audio_from_wav(room: rtc.Room, wav_path: str):
print(f"🎵 准备加载音频文件: {wav_path}")
if not os.path.exists(wav_path):
print(f"❌ 找不到文件 {wav_path}")
return
with wave.open(wav_path, "rb") as wf:
sample_rate = wf.getframerate()
num_channels = wf.getnchannels()
sampwidth = wf.getsampwidth()
if sampwidth != 2:
print("❌ 错误:只支持 16-bit PCM (S16LE) 编码的 WAV 文件。")
return
print(f"📊 音频信息: 采样率 {sample_rate}Hz, 通道数 {num_channels}")
source = rtc.AudioSource(sample_rate, num_channels)
track = rtc.LocalAudioTrack.create_audio_track("agent_input_audio", source)
options = rtc.TrackPublishOptions()
options.source = rtc.TrackSource.SOURCE_MICROPHONE
await room.local_participant.publish_track(track, options)
print("✅ 成功发布麦克风音频轨道,开始推流...")
# 每次读取并推送 20ms 的数据
chunk_duration_ms = 20
samples_per_chunk = int(sample_rate * (chunk_duration_ms / 1000.0))
bytes_per_chunk = samples_per_chunk * num_channels * sampwidth
while True:
data = wf.readframes(samples_per_chunk)
if not data:
break
# 根据已读取的数据计算出完整的采样数(最后一帧可能不足 20ms
frame_samples = len(data) // (num_channels * sampwidth)
# 使用 LiveKit SDK 封装 AudioFrame
try:
# LiveKit Python SDK version >= 0.15
audio_frame = rtc.AudioFrame(data, sample_rate, num_channels, frame_samples)
await source.capture_frame(audio_frame)
except TypeError:
# 若抛出 TypeError可能 SDK 版本有差异,尝试旧版本 API 或者直接 copy
frame = rtc.AudioFrame.create(sample_rate, num_channels, frame_samples)
frame.data[:] = data
await source.capture_frame(frame)
# 严格控制发送速率,避免瞬时把整个音频发过去而导致对面不识别(模拟真实发音)
await asyncio.sleep(chunk_duration_ms / 1000.0)
print("🎉 录音流推送完毕!等待 Agent 回复中...")
async def save_audio_stream(track: rtc.RemoteAudioTrack):
# 为避免旧文件冲突,加个时间戳
filename = f"agent_response_{int(time.time())}.wav"
print(f"🎙️ 正在将 Agent 的声音写入文件: {filename}")
stream = rtc.AudioStream(track)
wf = None
try:
async for event in stream:
# 接收到第一帧时初始化 WAV 格式
if wf is None:
wf = wave.open(filename, "wb")
wf.setnchannels(event.frame.num_channels)
wf.setsampwidth(2) # 16-bit
wf.setframerate(event.frame.sample_rate)
# 写入当前帧音频数据 (16-bit PCM)
wf.writeframes(bytes(event.frame.data))
except Exception as e:
print(f"音频流断开或出错: {e}")
finally:
if wf is not None:
wf.close()
print(f"💾 Agent 语音结果已成功保存到: {filename}")
async def main(room_url: str, token: str, wav_path: str):
room = rtc.Room()
agent_ready = asyncio.Event()
@room.on("connected")
def on_connected():
print("✅ 成功连接到 LiveKit 房间")
# 如果连接时房间里已经有 Agent远程参与者直接准备触发
if room.remote_participants:
agent_ready.set()
@room.on("participant_connected")
def on_participant_connected(participant: rtc.RemoteParticipant):
print(f"👋 Agent ({participant.identity}) 已加入房间")
agent_ready.set()
@room.on("data_received")
def on_data_received(data_packet: rtc.DataPacket):
identity = data_packet.participant.identity if data_packet.participant else "未知"
try:
print(f"📩 [数据接收 | {identity}]: {data_packet.data.decode('utf-8')}")
except Exception:
pass
@room.on("transcription_received")
def on_transcription_received(
segments: list[rtc.TranscriptionSegment],
participant: rtc.Participant,
track_pub: rtc.TrackPublication
):
identity = participant.identity if participant else "未知"
for segment in segments:
status = "✅ 最终结果" if segment.final else "⏳ 正在思考/中间结果"
print(f"🗣️ [{status} | {identity}]: {segment.text}")
@room.on("track_subscribed")
def on_track_subscribed(
track: rtc.Track,
publication: rtc.RemoteTrackPublication,
participant: rtc.RemoteParticipant
):
# 当 Agent 发出新的声音(音频轨道)时,我们订阅并保存
if track.kind == rtc.TrackKind.KIND_AUDIO:
asyncio.create_task(save_audio_stream(track))
print("⏳ 正在建立连接...")
await room.connect(room_url, token)
print("⏳ 等待 Agent 初始化并加入房间...")
await agent_ready.wait()
# 稍微延迟半秒钟,确保 Agent 侧面的准备(如模型加载等)一切就绪
await asyncio.sleep(0.5)
# 开始推送本地 wav 音频
asyncio.create_task(publish_audio_from_wav(room, wav_path))
try:
await asyncio.Event().wait()
except KeyboardInterrupt:
print("\n断开连接中...")
finally:
await room.disconnect()
if __name__ == "__main__":
if len(sys.argv) < 2:
print("❌ 用法: python test_client_wav.py <WAV文件路径> [LIVEKIT_URL] [LIVEKIT_TOKEN/API_KEY]")
print("说明:\n1. 必须提供 WAV 路径。")
print("2. 自动从 .env.local 读取 LIVEKIT_URL。并在没有提供 Token 时自动向 localhost:8000/getToken 请求。")
sys.exit(1)
wav_file = sys.argv[1]
url = os.getenv("LIVEKIT_URL")
token = None
if len(sys.argv) >= 4:
url = sys.argv[2]
token = sys.argv[3]
if not token:
import urllib.request
import json
import random
# 每次使用随机的测试房间,防止上一次没退出的 agent 堆积在同一个房间里导致多重回复
unique_room = f"test-room-{random.randint(1000, 9999)}"
print(f"🔄 正在通过本地服务获取 Token请求加入全新独立房间: {unique_room} ...")
try:
req = urllib.request.urlopen(f"http://localhost:8000/getToken?room={unique_room}&identity=python_tester&agent_name=my-agent")
res_body = req.read().decode('utf-8')
data = json.loads(res_body)
token = data.get("token")
print("✅ 成功获取了包含了 Agent dispatch 的临时 Token")
except Exception as e:
print(f"❌ 获取 Token 失败,错误信息: {e}")
print("若本地 token 服务未启动,请手动提供有效的测试 token")
sys.exit(1)
if not url or not token:
print("❌ 缺少 LiveKit URL 或 Token")
sys.exit(1)
asyncio.run(main(url, token, wav_file))

View File

@ -222,6 +222,19 @@ def process_emoji_collection(emoji_collection_dir, assets_dir):
emoji_list = []
# Check if this is otto-gif collection
is_otto_gif = 'otto-emoji-gif-component' in emoji_collection_dir or emoji_collection_dir.endswith('otto-gif')
# Otto GIF emoji aliases mapping
otto_gif_aliases = {
"staticstate": ["neutral", "relaxed", "sleepy", "idle"],
"happy": ["laughing", "funny", "loving", "confident", "winking", "cool", "delicious", "kissy", "silly"],
"sad": ["crying"],
"anger": ["angry"],
"scare": ["surprised", "shocked"],
"buxue": ["thinking", "confused", "embarrassed"]
}
# Copy each image from input directory to build/assets directory
for root, dirs, files in os.walk(emoji_collection_dir):
for file in files:
@ -233,12 +246,20 @@ def process_emoji_collection(emoji_collection_dir, assets_dir):
# Get filename without extension
filename_without_ext = os.path.splitext(file)[0]
# Add to emoji list
# Add main emoji entry
emoji_list.append({
"name": filename_without_ext,
"file": file
})
# Add aliases for otto-gif emojis
if is_otto_gif and filename_without_ext in otto_gif_aliases:
for alias in otto_gif_aliases[filename_without_ext]:
emoji_list.append({
"name": alias,
"file": file
})
return emoji_list
@ -672,6 +693,9 @@ def get_text_font_path(builtin_text_font, xiaozhi_fonts_path):
# Convert from basic to common font name
# e.g., font_puhui_basic_16_4 -> font_puhui_common_16_4.bin
if builtin_text_font.startswith('font_noto_'):
font_name = builtin_text_font.replace('basic', 'qwen') + '.bin'
else:
font_name = builtin_text_font.replace('basic', 'common') + '.bin'
font_path = os.path.join(xiaozhi_fonts_path, 'cbin', font_name)
@ -682,19 +706,44 @@ def get_text_font_path(builtin_text_font, xiaozhi_fonts_path):
return None
def get_emoji_collection_path(default_emoji_collection, xiaozhi_fonts_path):
def get_emoji_collection_path(default_emoji_collection, xiaozhi_fonts_path, project_root=None):
"""
Get the emoji collection path if needed
Returns the emoji directory path or None if no emoji collection is needed
Supports:
- PNG emoji collections from xiaozhi-fonts (e.g., emojis_32, twemoji_64)
- GIF emoji collections from xiaozhi-fonts (e.g., noto-emoji_128, noto-emoji_64)
- Otto GIF emoji collection (otto-gif)
"""
if not default_emoji_collection:
return None
# Special handling for otto-gif collection
if default_emoji_collection == 'otto-gif':
if project_root:
otto_gif_path = os.path.join(project_root, 'managed_components',
'txp666__otto-emoji-gif-component', 'gifs')
if os.path.exists(otto_gif_path):
return otto_gif_path
else:
print(f"Warning: Otto GIF emoji collection directory not found: {otto_gif_path}")
return None
else:
print("Warning: project_root not provided, cannot locate otto-gif collection")
return None
# Try PNG emoji collections first (e.g., emojis_32, twemoji_64)
emoji_path = os.path.join(xiaozhi_fonts_path, 'png', default_emoji_collection)
if os.path.exists(emoji_path):
return emoji_path
else:
print(f"Warning: Emoji collection directory not found: {emoji_path}")
# Try GIF emoji collections (e.g., noto-emoji_128, noto-emoji_64, noto-emoji_32)
emoji_path = os.path.join(xiaozhi_fonts_path, 'gif', default_emoji_collection)
if os.path.exists(emoji_path):
return emoji_path
print(f"Warning: Emoji collection directory not found in png/ or gif/: {default_emoji_collection}")
return None
@ -828,7 +877,10 @@ def main():
text_font_path = get_text_font_path(args.builtin_text_font, args.xiaozhi_fonts_path)
# Get emoji collection path if needed
emoji_collection_path = get_emoji_collection_path(args.emoji_collection, args.xiaozhi_fonts_path)
# Calculate project root from script location for otto-gif support
script_dir = os.path.dirname(os.path.abspath(__file__))
project_root = os.path.dirname(script_dir)
emoji_collection_path = get_emoji_collection_path(args.emoji_collection, args.xiaozhi_fonts_path, project_root)
# Get extra files path if provided
extra_files_path = args.extra_files

View File

@ -7,8 +7,8 @@ CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
CONFIG_SPIRAM=y
CONFIG_SPIRAM_MODE_OCT=y
CONFIG_SPIRAM_SPEED_80M=y
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=512
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=65536
CONFIG_SPIRAM_MALLOC_ALWAYSINTERNAL=2048
CONFIG_SPIRAM_MALLOC_RESERVE_INTERNAL=98304
CONFIG_SPIRAM_MEMTEST=n
CONFIG_MBEDTLS_EXTERNAL_MEM_ALLOC=y