An Unreal Engine plugin that helps you integrate artificial intelligence and machine learning technologies.
-brightgreen.svg){kind=icon}
Latest News | Documentation | Download | M4U Remoting(Android App) | Speech Model Packages
Demo Projects: Full Demo (Windows) | Android Demo
MediaPipe4U provides a suite of components and tools that let you quickly apply artificial intelligence (AI) and machine learning (ML) technologies in Unreal Engine projects. You can immediately integrate these solutions into your UE projects and customize them to your needs. They include motion capture, facial expression capture, text-to-speech (TTS), automatic speech recognition (ASR), and more. All features are real-time, offline, low-latency, and easy to use.
- [new] π Added Unreal Engine 5.7 and 5.8 support
- [new] π Added
VideoPlayerImageSource, which supports opening video files from the Android gallery for motion and expression capture - [fix] Bug fixes: #243, #262
- [fix] Fixed a crash when stopping motion capture
- [new] Added Unreal Engine 5.6 support.
- [fix] Incorrect negative-angle handling when using NvAR to capture head rotation. #122
- [fix] Packaging with UE 5.3 or earlier may fail. #237
- [new] π The free edition can now package all features, including speech and expression capture.
- [new] π Ollama support: integrate Ollama for large language model inference, supporting many LLMs such as DeepSeek, LLaMA, Phi, Qwen, and QWQ.
- [new] π Dialogue component
LLMSpeechChatRuntime: integrates LLM, TTS, and ASR, making it easy to implement chatbots in Blueprints. - [new] π Added new TTS support: Kokoro, Melo.
- [new] π Added new ASR support: FunASR (Chinese/English hotword edition), FireRedASR (Chinese/English/dialects), MoonShine (English), SenseVoice (multilingual: Chinese/English/Japanese/Korean/Cantonese).
- [new] π Added a Transformer-based TTS model: F5-TTS, with zero-shot voice cloning capabilities (DirectML/CUDA inference; AMD and Nvidia GPUs supported)
- [new] π Voice wake-up: small-model inference-based voice wake-up, supporting custom wake words to activate ASR and standalone voice-command wake words.
- [new] π₯ Updated Google MediaPipe to the latest version.
- [new] π₯ Added Unreal Engine 5.5 support.
- [new] π Added the NvAR pose capture algorithm, allowing switching between MediaPipe and Nvidia Maxine algorithms.
- [new] π Open-sourced MediaPipe4U Remoting (an Android facial-capture app for MediaPipe4U).
- [new] π«
Custom Mediapipe Connector(C++): a custom connector that can completely replacegoogle mediapipeto implement custom logic (#195, #204). - [new] π«
Custom Mediapipe Feature(C++): a custom feature that can partially replacegoogle mediapipeto implement custom logic (#195, #204). - [new] π Added the Android Demo project (Gitlab)
- [improve]:footprints: Upgraded the Demo project to UE5.5 and added
Fake Demo, a C++ extension example that reads local files instead of MediaPipe. - [improve]:footprints: Added voice wake-up and speaker-selection examples to the Demo project
- [improve]:footprints: Added LLM examples to the Demo project
- [improve]:footprints: Added voice chatbot examples to the Demo project
- [improve]:footprints: The Demo project now supports packaging
- [break change] π₯ Motion capture has been split into a standalone
MediaPipe4UMotionplugin. This may break existing Blueprints; update your Blueprints after upgrading. - [break change] π₯ Due to changes in the license-file format, existing licenses will no longer work. You can obtain free licenses here going forward.
Speech package downloads have moved to huggingface.
Only the Unreal Engine 5.5 version is currently available; other versions will be packaged and released subsequently ...
- The new Google Holistic Task API does not support GPU, so Android uses CPU inference; as before, Windows also uses CPU inference.
- Starting with Unreal Engine 5.4, the built-in
OpenCVplugin no longer includes precompiled libraries (DLL files). On the first launch of the UE Editor, because M4U depends on theOpenCVplugin, the UE Editor will attempt to download theOpenCVsource code and compile it on your computer. This may take a long time and make it appear that the engine is stuck at 75% loading. Please wait patiently and check the logs in the Saved directory under the project root to confirm that the process has completed. Users in China may also need a proxy to access the Internet. You can also follow the steps in #166 to solve this manually.
For more updates, see the update log:
πAll features are pure C++, with no Python or external programs required.
- Motion Capture
- Body motion capture
- Finger motion capture
- Translation capture
- 3D model driving
- Real-time driving
- Supports standard cameras
- ControlRig support
- Expression Capture
- Facial expressions
- Arkit Blendshape compatible (52 expressions)
- Live Link compatible
- Real-time driving
- Supports standard cameras
- Multiple Capture Sources
- Standard cameras
- Video files
- Images
- Streaming media (RTMP/SMTP)
- Android devices (phones and tablets) (M4U Remoting)
- Large Language Models (LLM)
- Ollama support
- Text-to-Speech (TTS)
- Offline synthesis
- Real-time synthesis
- 3D model lip-sync
- Multi-model support (Browse models)
- Automatic Speech Recognition (ASR)
- Offline recognition
- Real-time recognition
- Multi-model support (Browse models)
- Voice wake-up (similar to
Xiao Ai Tong XueandXiao Du Xiao Du)
- Animation Export
- BVH data export
- Plugin Only
- No external programs required
- Everything is done in Unreal Engine
-
ζ₯ζ¬θͺ (η³γ訳γγγΎγγγγγγγ―εγͺγGoogle翻訳γ§γ)
-
νκ΅μ΄ (μ£μ‘ν©λλ€, μ΄κ²μ λ¨μ§ κ΅¬κΈ λ²μμ λλ€)
UnrealEngine 5.1 - 5.8
To learn what was included in each release, visit the Update Log page
Because the plugin uses precompiled binaries and includes a large number of C++ linker symbols and debug symbols, it occupies 5G of disk space after extraction (mostly UE-generated code in Intermediate)
You do not need to worry about disk usage. This is only disk usage during development; after packaging the project, the plugin uses approximately 200M of disk space (mainly GStreamer dynamic libraries)
M4U now supports Android and Windows (Linux support is coming soon), but platform support differs slightly.
| Plugin (Module) | Windows | Android | Linux |
|---|---|---|---|
| MediaPipe4U | βοΈ | βοΈ | Coming Soon |
| MediaPipe4ULiveLink | βοΈ | βοΈ | Coming Soon |
| GStreamer | βοΈ | β | Coming Soon |
| MediaPipe4UGStreamer | βοΈ | β | Coming Soon |
| MediaPipe4UBVH | βοΈ | βοΈ | Coming Soon |
| MediaPipe4USpeech | βοΈ | β | Coming Soon |
License files are periodically published in the Discussions, and are also automatically included in new plugin packages.
About M4U Remoting
This is an open-source Android application that can send expression data to the MediaPipe4U Windows application.
With it, you can use an Android mobile device for expression capture and display expressions on Unreal Engine characters.
Clone or download the source code for the Demo projects from the following GitHub repositories:
Windows Demo: https://gitlab.com/endink/mediapipe4u-demoAndroid Demo: https://gitlab.com/endink/mediapipe4u-android-demo
Use a Git client to obtain the Demo projects (you must install git and git lfs) :
Windows Demo is a full-featured demo. If you are learning how to use
MediaPipe4U, the Windows Demo is recommended.
Windows Demo:
git lfs clone https://gitlab.com/endink/mediapipe4u-demo.gitAndroid Demo:
git lfs clone https://gitlab.com/endink/mediapipe4u-android-demo.gitThe Demo projects do not include the plugin. Download the plugin yourself and place it in the project's Plugins directory.
English Video Tutorial (YouTube)
Chinese Video Tutorial (bilibili)
If you encounter a problem, first check the FAQ, which collects issues encountered by many users and may include yours. If you cannot find it in the FAQ, submit an issue on GitHub first; direct messages may cause me to forget the problem.
Because the Windows version of MediaPipe does not support GPU inference, Windows relies on the CPU for human-pose prediction (see MediaPipe for details):
Test Environment
CPU: AMD 3600 CPU
Memory: 32GB
GPU: 1660s (this GPU does not support ray tracing)
Performance:
Frame rate (fps): 18-24
CPU utilization: 20% (the test DEMO contains only the MetahumanLight scene and VRM model)
Users in China who cannot view the video can visit the Bilibili video
We thank the following open-source projects and frameworks for their contributions, which have had an important impact on the development of M4U:
- M4U uses MediaPipe for motion capture and face capture.
- M4U uses NVIDIA Maxine AR SDK for advanced face tracking and capture.
- M4U uses PaddleSpeech for text-to-speech (TTS) synthesis.
- M4U uses FunASR for automatic speech recognition (ASR).
- M4U uses whisper.cpp as an ASR solution.
- M4U uses Sherpa Onnx to enhance ASR capabilities.
- M4U uses F5-TTS-ONNX to export F5-TTS models.
- M4U uses GStreamer for video processing and real-time streaming-media decoding.
- M4U uses code from PowerIK to implement inverse kinematics (IK) and ground adaptation.
- M4U uses concepts from Kalidokit in the motion-capture domain.
- M4U uses code from wongfei to enhance interoperability between GStreamer and MediaPipe.
We extend our sincere thanks to the developers and contributors of these projects. Their innovation and open-source contributions have greatly advanced M4U.