HeyGen AI software program is a ground-breaking AI know-how, bringing video avatars and translations/dubbing to the typical prosumer manufacturing. For those who want a speaking head video for advertising, coaching and how-to movies, then that is the place to begin! Apart from all the opposite options, choices, templates for producing content material from pre-made avatars and pictures, the true juice with this software program is the flexibility to clone your self from a video & audio clip is actually unimaginable.
As I’ve been saying in all my AI Instruments updates: AI Software know-how is advancing at such a fee that we’re measuring it by days now, not even weeks, months or years. The builders at HeyGen are an ideal instance of such fast growth, that I’ve needed to change this product evaluation a number of occasions up to now month; as a result of both their developments in know-how, method to creating these avatars, and their pricing construction of all the things has modified nearly every day.
It actually took off once I noticed a video posted on LinkedIn from the CEO of HeyGen teasing the capabilities of their new “Avatar Lite” beta, which I promptly acquired on board and utilized to begin testing – and I acquired this response in e mail the identical day. The remaining is a really, very quick historical past!
Whereas I’ll define a number of options from this AI Software, the largest focus for me is on the Video Avatars – which have been evolving quickly as I acknowledged above. As an illustration, I made this video a couple of week in the past and it’s already outdated within the options, high quality and naming of the assorted instruments. The “Avatar Lite” used to take 3-5 enterprise days to generate a usable video avatar as proven (principally with hands-on techs refining the method), however now could be at the moment automated to generate an “Prompt Avatar” in mere minutes! You may also now “Finetune” your Prompt Avatar (for an extra payment of $59/mo – which we’ll focus on later on this article) and comes again to you in underneath 24 hrs.
Right here’s a very AI-generated video displaying the method – together with the voice translations characteristic talked about under from solely a pair weeks in the past:
Video Productions from Templates
Relying in your talent stage and necessities, there are lots of methods to begin producing content material in HeyGen’s studio. You may choose one in every of dozens of starter templates you may modify straight within the portal and even change your supplied avatars from their library. The consumer interface is absolutely easy and simple to navigate and make changes.
There are dozens of pre-loaded avatars and voices accessible to select from.
You may also simply make a video with an avatar on a inexperienced background and composite them straight in your NLE of alternative. This instance was from a generic avatar and voice generated in HeyGen after which composited in After Results for an instance social media quick video.
Animated Faces from Images & Photos
This was step one in discovering how a lot enjoyable this software program may very well be. I found it a couple of months in the past and performed round with varied images and pictures rendered from Midjourney. The method has modified just a little since then, however the high quality has improved a fantastic deal.
In addition they have an choice to generate AI characters with a textual content description straight inside HeyGen’s interface. It got here up with some fascinating outcomes however word that solely the faces/heads get animated and never the complete torso once you generate avatars this manner.
It’s a fast course of – merely add your picture and apply an AI voice (or cloned voice out of your ElevenLabs API) after which create a video along with your textual content enter. Simply add your picture or rendered picture to begin (ensuring the face is full seen and central to yoru picture).
Right here’s a couple of examples from my headshot picture and a pair Midjourney photos:
Take a look at the instance under the place I’m establishing the inexperienced display screen studio and our studio model “Leana” complains. That was carried out from an iPhone picture in HeyGen utilizing this similar course of.
Video Avatars & Voice Cloning
That is the place we break up off from the remainder of the pack – and what acquired me enthusiastic about utilizing HeyGen for normal advertising and educational functions at my day gig at a biotech firm. It actually has generated a number of curiosity with our product advertising people.
Step one is to be sure to have a superb video and audio recording to work from. You may simply put up a tripod and shoot your self or your topic in an off-the-cuff or enterprise setting with a gradual background and clear audio to your submission. You shouldn’t transfer round or make sudden gestures or facial expressions and let the video run for a full uninterrupted 2-5 minutes for one of the best cloning outcomes.
Once you create your avatar, you must submit a video authorization (from the topic straight) for safety functions. This retains the positioning secure from nefarious actions.
On this first video I generated from my dwelling studio workplace was a baseline to construct my different experiments on:
For extra flexibility in my avatars, I set-up the inexperienced display screen studio to shoot extra checks of myself, studying the identical 2-1/2 minute script from a teleprompter for my comparisons. Organising the greenscreen after a couple of years because the first shutdown for Covid took awhile to dial all the things in, so our model “Leana” acquired a bit impatient standing there all day. (additionally animated with HeyGen) 😉
The method is absolutely easy and I don’t want to stipulate all of the steps right here as a result of it’s simple to observe their directions from the website online and so they have a number of video tutorials on the positioning they’ve created. You should use both a prerecorded voice audio file or TTS utilizing a built-in voice or choose a clone you’ve generated. I’ve downloaded a number of from ElevenLabs to generate lots of my take a look at movies however now choose utilizing the built-in third-party API to generate straight within HeyGen and I can entry no matter ElevenLabs voices I’ve in my account by means of the voice supervisor.
So for this instance, I utilized one AI generated VO audio from ElevenLabs.com to create three completely different variations of the identical script to see how they in contrast – or differed from one another. Remember the fact that these three avatars aren’t simply dressed otherwise, they have been sourced from three separate movies that I shot on the inexperienced display screen at separate occasions. Making use of the prerecorded AI voice from ElevenLabs assured the Avatars would sync correctly. I couldn’t get this similar outcome had I run the ElevenLabs API to generate the VO on the fly repeatedly as there can be variations within the voices.
On this instance, I ran the identical script and composite in Premiere simply exchanging the inexperienced display screen composites from After Results in the identical sequence.
Prompt Avatar vs “Finetuned”
The Prompt Avatars you get along with your plan are ample for many functions (you should buy extra if wanted), however the Finetuned Avatars do have higher mouth and lip sync efficiency, as seen in my testing.
On this instance video, I used ElevenLabs to supply the audio monitor which I uploaded to HeyGen once I created the video avatars, in order that they have the very same audio monitor for true side-by-side comparability. Discover the accuracy of the lip sync is improved on the Finetuned model on the best.
Higher but – I’ve discovered that utilizing the ElevenLabs API hyperlink straight within HeyGen, I get a lot better lip syncing and mouth actions on BOTH the Prompt and Finetuned avatars.
That is solely the start… watch this tech intently within the coming months!
Translations & Dubbing
There are two methods of producing translations in HeyGen. One is to enter translated textual content into the video avatar producer and choose a multilingual voice out of your ElevenLabs API and create a clear video avatar from there.
The opposite technique helps you to add any video clip with a topic operating a minimum of 30 seconds dealing with the digital camera and it’ll generate a brand new video for you with a clone of the actor’s voice and lip syncing capabilities mechanically in a couple of minutes. Right here’s their rundown on the method in video kind from the HeyGen web site:
I’ve examined a number of video clips and the outcomes have been wonderful! Take a look at the intro video on the high of the article to see extra examples I’ve created.
Right here’s an instance clip that I created from a scene from Pulp Fiction with Christopher Walken and translated into Spanish and French. You may see the place this may very well be actually useful for video dubbing and regionalizations sooner or later.
Execs & Cons
Whereas I’ve been a significant fanboy the previous month or so over these new options and capabilities, I’d be remiss to not level out some issues that I hope get resolved or up to date in future variations of the HeyGen software program instruments – and pay constructions.
The instruments are evolving shortly – to the purpose that I feel most of this evaluation will probably be out of date by year-end. And with that, probably positioned to be purchased up by an even bigger model or one other spherical of financing encourages the builders to make a leap towards world domination. (solely barely kidding) 😉
I want to see the flexibility to regulate the Prompt Avatars extra with gestures, facial features, and so on. When the voices get extra energetic, the faces ought to replicate that as properly. Mabe simply an “exaggeration/enhancement” slider or one thing.
The speaking images might use extra management as properly – like the best way the puppet device works in After Results, the place you may outline the factors that transfer or a minimum of outline the boundaries of the top/hair so the entire head strikes – not simply the face.
And pricing appears to be in all places at the moment – however that may be because of the adjustments in product choices as they develop. As an illustration, the $99/yr for a voice clone that I really feel is sub-par to what you may generate in ElevenLabs. (which I’m actually grateful for the appliance of the ElevenLabs API which produces one of the best of each worlds in a single simple step). The month-to-month payment for the bottom service is truthful, particularly when 3 Prompt Avatars are included with the $59 Creator bundle. The “Finetuned” choice is an extra $59/mo for EACH AVATAR you improve this feature for. Which means when you improve all three Prompt Avatars you create, that’s an extra $150 mo simply to proceed to make use of them. I suppose when you don’t want them any extra, simply cancel the improve plan for each, however I’m probably not seeing that a lot worth within the little little bit of distinction that the “Finetuning” gives at this level for many clients – however professionals will justify the extra value to get a greater stage outcome.
