3D Models, Milkdrop & layered Deck support, blazingly fast! Version 1.0 soon!
Multi-layered decks!
The back-end was long ready for multi-layered decks, logos already had them, and they are finally here for all decks now! This allows you to play two videos on top of each other, and have them blend, or have a video blend with a Milkdrop animation, or a 3D model in front of a video, etc.
When a deck is multi-layer, a split thumbnail is shown on the tab. Decks can be set to 2 layers using a dropdown at the top-right of the deck. Visuals on the foreground layer (bottom, imagine in 3D perspective, looking up at the layers horizon) have a Blend Mode option now. You can access this by clicking the little gears icon, and then selecting one or multiple visuals. There are many Blend Modes. Alpha uses the media's alpha channel to put it on top; good for 3D models, and images with an alpha channel. Multiply multiplies the color values for each pixel, and gives a really cool effect for most content. And Subtract, subtracts the color values where they overlap, leading to a fancy picture-negatives effect.
Blend Mode changes take effect immediately. And for those who like to tweak things until they look exactly right, there is a blend power slider too, to adjust the strength of the selected Blend Mode.
Now that we have multi-layered decks, we've also added 2 extra free sample decks, combining video with our 2 new supported file formats. If you want to know how a certain result is achieved, just click the small gears icon, and select a visual from the decks foreground / bottom layer.
Support for 3D Models (glb)
A feature I'd been planning for a long time, is support for 3D models! It's one of the reasons we use Unity!
Now it is here! We support the GLB and glTF (.glb, and .gltf) file format pair. The former being a packaged version of the latter. This is a widely used, modern, and open standard, that provides meshes, materials, textures, skeletal rigging, and animations all in one place, and supports live model loading. Most 3D software can export to it, and conversion tools exist as well.
When we import models, they are automatically analyzed for height and width, and positioned, so they fit on screen, and are on an imaginary dance-floor. If they have built in animations, these are played randomly, switching to another on the beat. And we've even added a real dance-floor, in the form of stages that can be selected from the settings side-panel (gears icon) when a 3D model visual is selected. At the moment we have 5 different color presets and prop variations of a night club setting, making good use of the power of Unity.
In addition, 8 play modes can be selected for 3D models. From a calm spin, to random turns on the beat. The spin burst is a nice one too, which gives the model a rotational push on each beat, which then naturally slows down until the next beat.
And four of the play modes are our own dance moves! They work for almost all humanoid 3D models with a skeleton! The main standards for 3D model skeletons we have covered (Mixamo, Blender / Rigify, 3ds Max Biped / Character Studio, VRM / VRoid, Daz, and plain name rigs). Corrections to make them match our dance-moves are only carried out when one of the dance play modes is selected.
Support for Milkdrop (Winamp Visualizer)
Decades ago my first experiments with audio reactive computer generated visuals were with the Winamp Visualizer plugin, using Milkdrop files. That ecosystem has continued, and 10.000+ of them are freely available online now. And there's several awesome Open Source implementations of libraries that can play them. We use libprojectM 4 to allow you to play them in AI VJ now!
Turns out many of these, even the older ones, combine really well with video visuals! (with the Multiply Blend Mode). One of our new sample decks shows off a large number of these! Stay tuned to more of them appearing on the Workshop, or go look for them & experiment with them yourself!
Record shorts and videos straight from AI VJ
By popular request, (and for my own use too), it is now possible to record shorts for use on social media such as TikTok, Snap Chat, and Youtube Shorts. They'll be 64 beats long (about 30 seconds) for better audio looping, be portrait mode (1080 x 1920), and fill the bottom and top of the screen with mirrored reflections of the recorded middle 1080 x 1080 square. So they look pretty nifty on mobile!
Making landscape, full HD recordings is also possible! All of these recordings have a small AI VJ watermark. This to help spread the word about AI VJ. And as an added bonus it makes the content compliant with new EU regulation about marking all AI content. If you want this watermark removed, you can use Spout and OBE, or other recording software to make your own recordings the old way.
Twice as fast! Adaptive quality and performance tweaks
As smooth visuals are so much better than jittery ones, in this update we've done a thorough review of the performance of AI VJ, and anything that interrupts it. After the video playback engine, relying on ffmpeg, custom written in c, most gains were made by tweaking Unity's High Definition Render Pipeline to the max. And having different settings for the various stages of processing.
A friend of mine (thanks Morris!) who works as a device driver programmer for Nvidia even went through the code with a fine-toothed comb, and came up with several more optimizations, especially around 3D model rendering. In all AI VJ is now twice as fast as it was only 3 months ago!
Besides these tweaks, AI VJ now also adapts itself to the power of your setup. Rendering is capped at the refresh rate of your projector, or PC screen if no projector is present (absolute cap at 120 fps). And if it detects that it goes below 30fps for a certain media type (3D Models, Milkdrops, and the Fractal Shader), it automatically reduces the quality of that type up to twice, with re-checks every 10 minutes. This should ensure smooth performance even on older hardware.
Near full test coverage
AI VJ is now covered by over 650 automated test, with each of these testing several cases. This was about 200 before. We have near full coverage now.
Most of this update has thus been thoroughly tested, even in live performances for several weeks, and proven stable.
Still, with the size of the update, some new bugs are to be expected. Report them please!
Version 1.0 to be launched!
After three years of development, at the end of October, on the 28th to be precise, AI VJ will come out of Early Access, and version 1.0 will be launched!
It will be a smaller update than this one, but it will be an important symbolic milestone.
Not only have we now implemented most of the core features that we think a proper 1.0 version should include, but even more so, AI VJ also has reached a level of stability and maturity that warrants a full version number!
The upcoming version 1.0 does not mean that it is finished though. We have many more updates and improvements planned.
Bugs Fixed
Several bugs were fixed, and more small improvements made:
- the info screen shows a visual's path (end first, whole path as tooltip), a 3D model's clips, rig and triangles, and a preset's shaders and complexity. For milkdrops it also shows the version
- the keyboard/edge scroll stops on a pixels-per-second threshold, not pixels per frame, avoiding it from being locked on fast hardware
- a manual play (item, bank, deck) preloads its item for a beat, then swaps on the next real beat. Faster than before!
- launching with -fps tracks frame per second stats in memory and prints the summary on exit
- small DJ logo positions - added 2 options: one centered logo at the top or bottom. Nice for shorts!
- a Workshop create that lost a preview previously ignored the previews that still came back
- a watchdog for thumbnail encoding and other processes, should not fail anymore if one fails
- a given-up task reports to its requester, and a post-handling throw can no longer kill the processor
- FFmpeg video wrapper decodes native error strings as UTF-8 so non-ASCII paths survive
- corrupt settings files are quarantined as .corrupt like the autosave, no longer deleted
- a load recovers an interrupted save left as <path>.tmp - complete temps are swapped in
- saves write to a temp file and swap atomically (mid-write failure no longer loses the old save)
- fixed a bug causing audio detection for new installs to be skipped
If you reported (other) bugs, we're keeping a list and working through them! Please keep reports coming! We won't know if you don't tell us (best place to do so is in our Discord)
Join us on Discord!
Feature requests & bug reports are welcome. Either here or on our Discord!