AI Pals Engine · published
Yesterday, Google gifted us Gemma 4 in multiple variants and experimental support has been added to the Steam publictest version and other editions since then (see inapp Recommended Models).
Model
Parameter Size
Context Window
Gemma 4 E2B
2.3B effective (5.1B w/ embeddings)
128K
Gemma 4 E4B
4.5B effective (8B w/ embeddings)
128K
Gemma 4 31B
31B dense model
256K
Gemma 4 26B A4B
Mixture-of-experts (4B active / 26B total)
256K
Since the Early Access release, a lot has changed and many improvements have been made. Now is a good time to take a step back and look at where things stand.
Are there any features previously mentioned that you feel are still missing (aside from those outlined in What’s on the Horizon for AI Pals Engine)? Or are there bugs you’re still encountering? Your feedback is incredibly valuable in helping shape the next steps.
Hopefully, you’ve already noticed the progress across this and other editions of the app... and that your feedback is making a real difference!
For this devlog, I want to focus on a new feature that has been requested since day one: Voice Activity Mode, as well as take a look at the current state of the Workshop and the Bundle concept.
An Image Pal Bundle Test (available to you via the Workshop). The images are randomly replaced via Behavior (SetPalAvatarByTagAction using tag AImage) to achieve e.g. a different Pal per Session.
Using the microphone without needing to press a button has been one of the most common requests.
I generally agree with the Discord that multimodal LLMs with fully integrated Audio are the future and hopefully the multimodal trend continues. Gemma 4's addition of native audio support (the small E models) is another reason to experiment further with implementing native audio support.
But until we fully get there, I wanted to improve the current mic experience with something that will remain essential even in a multimodal future:
Automatic speech detection
In other words, this is a mic auto mode. Once Voice Activity Mode is enabled, you can simply start speaking - no button presses required - and the system handles the rest.
In its current implementation, a lightweight model continuously analyzes the audio input to detect when someone is speaking.
Detected speech is buffered and once you stop talking, it gets sent to the Speech-to-Text model. From there, the result is passed to the AI model and the response is returned as text or TTS.
There is still some noticeable delay, but especially with CUDA enabled and a performant model, it's already quite responsive. At the moment, a few basic voice commands are supported, along with an optional wake word.
Let me know how it performs in your testing!
You already have access to it as it evolves, but the Workshop isn't officially released yet. There is also an EULA related bug that blocked new submissions, but that has now been fixed in the publictest version.
The main reason for holding back the full release is simple: I want to spend a bit more time refining the system and to understand which tags are actually useful.
Recently, improvements have been made to the UI (Current support: Pal Images and Chars). I'd love your input here: which asset category should be supported next?
Bundles are essentially collections of related Chars and Assets. For example:
The same Character across different scenarios
A Pal Image set with multiple emotions or visual variations
Specialized assets like stylized button image sets for action buttons
Bundles can also be shared via the Workshop, making it easier to distribute sets of content.
As shown in the initial GIF, there are also specialized behavior nodes that allow you to:
Retrieve a random Char or Asset from a bundle
Filter results based on tags
For example, you can dynamically switch expressions based on emotion (see Widget Example Advanced 1 – Lightweight Expression) or load a different Character scenario each time a widget is reset.
This opens up a lot of interesting possibilities. Imagine pulling in external data, like retrieving the current weather and automatically selecting a matching Pal Image.
The Workshop has the potential to go far beyond just Char and current asset types. In the future, it could expand to include:
Full Widgets
Themes
Small, game-like widgets
Abilities
Tools
I'm especially interested in experimenting with more stateful, game-inspired widgets, systems that follow rulesets, similar to tabletop experiences like DnD. It would be wonderful if, in the future, we could share even small minigames. First things first, though: let's start by integrating more standard assets.
Right now the backlog looks like this (excluding personal things I want):
Chatformat Auto Mode and Overhaul (Long Term)
Long Context Performance Improvements (+KV Quant support)
More of the following: Help pages, Internationalization, Fixes, QoL Improvements
Summarized from smaller updates deployed since the last Devlog:
✨ New Features
Plugins: Added Voice Activity Mode and Editable Voice Commands
Gemma 4 Support (V0.9.1.1)
🔧 Improvements
Compatibility of the different AI Pals Engine Editions
System Tray: Changed the style to a Dark Theme
TTS: Added subtle fadein/fadeout
Memory: MD File Support (V0.9.1.0)
Settings - Audio: Added TTS Volume Slider
Settings - Audio: Added Widget Volume Slider
Settings - Audio: Added Microphone Id
Widget: Choosing a Char or Text Model now autocloses the sidepanel (+ added key shortcuts)
Chars and Assets: Added Bundle Filter
Workshop: Added Search Text
Workshop: Added Favorite Button
Workshop: Added Pagination
Workshop: Improved and fixed Sort and Filter Options
Help: Added Multimodal Model/Plugins Help, modified Get Started and improved inapp tooltips
🐞 Fixes
Workshop Submit stuck on EULA check (V0.9.1.1)
Widget Dialog Cancel+Ok button are now more consistent
Widget Alt+F4 now properly Unsets the Active Space