Table of Contents Show
Archive notice: This article was originally published on March 1, 2019. Links and embedded videos are preserved as part of the historical record.
Since every late night show needs a sidekick of stature, we built a digital and bad-tempered avatar and animated him with the Rokoko Smartsuit. And it went like this …
Chapter 1 – The challenge
Our story takes place in late summer 2018 and begins with a phone call from Sascha.
“Would you like to produce a late night show?” he asked.
Sascha is a creative producer for Optimist, among others. We’ve known each other for a long time.
We said: “Sure, tell me about it.”
Sascha called as a producer on behalf of the Reeperbahn Festival.
Sascha: “There’s no concept, just a rough idea. We have six weeks from now and the budget is small.”
We: “Uhh … ?”
Sascha can be very persuasive.
And when do you ever get the opportunity to paint an almost white canvas? So then a late night show – new territory for us too. The idea of giving the presenter a spiteful and bad-tempered sidekick came up during the initial brainstorming sessions. Ray Cokes was still being discussed for this in the first concepts. Older readers may still remember him as a lively MTV presenter. But the programme’s target group is 14 to 25-year-olds. That’s what you call a target group conflict. So we crossed Ray Cokes out again and suggested a digital alternative.
The task was clear: in addition to the CI for the show, a digital avatar was to be created that would repeatedly interrupt the show and take over parts of the moderation.
After initial drafts, we finally worked with illustrator and artist Jim Avignon on the layout of our character, whose style fits the target group perfectly. Once the look was finalised, we were able to start production.
Chapter 2 – Monsters
Good stories need drama. And what results in time pressure plus a lack of resources? That’s right. Drama. Or stress. And that’s what 3D artist Julian Gummich had to do first, modelling and rigging a working character from Jim Avignon’s sketches within three days.
Meanwhile, we were already starting to carry out initial tests with the Rokoko smartsuit and were able to determine where the hurdles lay, what we had to pay attention to when modelling and how we could use the suit most efficiently. It quickly became clear that a lot was possible. But some things don’t work either.
The Smartsuit works according to a simple principle: it collects the wearer’s movement data via a total of nineteen sensors distributed at central points on the body. A transmitter on the back transmits the data to the computer via Wi-Fi, where it is recorded or processed live. The advantage: there is no need for time-consuming tracking of individual points and movements can be quickly transferred to digital models. For a price of 2,495 US dollars, this is quite an achievement.
Smartsuit Studio
The data is recorded by the included Smartsuit Studio software, which has since undergone a comprehensive update and is now called Rokoko Studio.
As we relied on Cinema 4D for our project, we did not have a plug-in available to display or record the data directly in C4D. But it was no problem to use the data in C4D via the FBX export from Smartsuit Studio.
In any case, our biggest problem was the short production time. We were only able to test the suit for a few hours while modelling and rigging our character. As we were also working on the concept and casting the actor at the same time, we had to reckon with the fact that in the end we only had 2 to 3 days between motion capture and finalisation. After these initial tests, we immediately realised that we would never be able to produce more than a few minutes of animation cleanly and without glitches. However, we were asked for around fifteen minutes.
And we were only working with two artists on the project. There was simply no time available for a sophisticated workflow with retargeting and data clean-up. “It’s not a glitch – it’s a feature” – we were aware of this problem from the outset, so we focussed on making everything a little rough and edgy during the design process.
Theoretically, it is even possible to control the avatar live with the suit via the Unreal Engine or Unity. However, we deliberately decided against this workflow for this production because we simply didn’t have the time to test the process in a meaningful way. Both technically and dramaturgically. Despite all the technical challenges, we naturally wanted to produce a good show first and foremost. So while we were bringing the avatar to life, we were also fine-tuning his character. So: How does he act? What makes him tick? We wanted a grumpy sourpuss who stumbles into our show, badmouths everything and really just wants to go home. His name? Monster.
And the more we gave Monster a character and a role, the more we realised: Monster has to be able to talk. But the smartsuit isn’t able to track the face, let alone the mouth movements.
Lead artist and motion director Felix Koziol finally had the brilliant idea. Using Adobe Character Animator and Cinema 4D, we built a simple workflow that allowed us to apply numerous voice takes as lip sync to monsters as quickly as possible. Using Adobe’s Character Animator, we analysed the takes and mapped them to different shades of grey. We then exported just a small greyscale film, which provided us with the necessary values in Cinema 4D using the Shader Effector. With a simple Xpresso
Circuit, we then mapped the generated values to the different mouth poses.
And then we were delighted to have found such a well-functioning workaround that we were able to animate the required takes practically automatically.
Chapter 3 – the production
The third act of our drama could now begin – the actual production.
During a casting session, we initially decided in favour of Jan David Rönfeldt as the narrator. His enthusiasm and perfect English suited our character perfectly and gave Monster the necessary twist. But the movement was still missing.
And we were able to convince the professional dancer and musician Booz to jump into the smartsuit for us. It was clear from the outset that we needed someone who has expression and knows how to move and dance. Precisely because the character was limited in his possibilities, we wanted to set a clear focus here.
We then recorded the individual takes, movement sequences and scenes in our studio. The included Smartsuit Studio software helps us set up the suit. The Danish manufacturer offers comprehensive tutorials for inexperienced users, which are unfortunately sometimes somewhat confusingly organised. It therefore takes some time to find the right tutorial.
However, once you have understood the basic functions, you can get started very quickly and quickly achieve your first results.
Step one is to calibrate the suit. To do this, the actor moves into the so-called zero pose. The software recognises the position of the sensors and places a simple model in the virtual space. The suit and avatar are now connected and movements are transmitted in real time. It all looks great at first.
But if you look at the details, it quickly becomes clear that the actor’s poses are transmitted too inaccurately. For example, when the actor grabs his head, the avatar’s hand is still about 10 cm away from his head. In other places, different parts of the body collide.
In this case, Rokoko suggests measuring the performer by hand to teach the software how far apart the individual sensors are in reality. And indeed, the poses work much better after that. However, we still couldn’t get the hands and knees to touch – Rokoko needs to improve this. Even the friendly and fast email support was ultimately unable to help us here.
Another realisation: the suit must fit the wearer well. If it sags on the body because the clothing size does not fit the performer, the recorded data cannot really be used.
Another point of criticism is that the software on our Macs was generally too unstable and crashed at least fifteen times during our production day – too often.
This is because the suit has to be recalibrated after every restart. Although this never takes long, it disrupts the flow for the director and actors every time.
The software was more stable on Windows systems, but you can demand stability everywhere. Hopefully Rokoko has got these problems under control with the comprehensive update.
We quickly found a workaround for the sometimes imprecise transfer of movement data and soon knew which poses looked good. In addition, we had already decided in advance to make a virtue of necessity and incorporate the collision of body parts into the design concept.
Glitches were therefore okay. We later deconstructed the backgrounds to match this, so that the overall picture was coherent.
Chapter 4: The animation
Four days before recording, we finally had the audio takes and motion capture together. All in all, we had to have a good 15 minutes, including rendering, ready for recording. As we were aware of this beforehand, we had developed a workflow that gave us the option in C4D to simply load the motion capture into the project as XRef. Add to this the greyscales from Adobe’s Character Animator and after a short sync either 30 seconds or 5 minutes were ready. Now the animations just had to be staged appropriately. In our first layouts, we experimented with backgrounds and elements also illustrated by Jim Avignon and quickly realised that the look was too cutesy. We felt too reminiscent of Asterix – a direction we wanted to avoid, as this style did not fit in with the overall concept.
In our search for alternatives, we were lucky enough to meet Marc Broyer, a Hamburg photographer and art director who, with his unusual view of the harbour city, had exactly the images we had in mind and was fortunately willing to support us. This provided the perfect backdrop for our Campfire Show, which is closely associated with Hamburg as an event during the Reeperbahn Festival.
We mapped Marc’s photos to simple geometry in C4D using camera mapping, which could then be animated. It was important that a stage was always created for Monster, as we still didn’t know which movements we had to use there. In order to be able to react flexibly here, we built scenes with different movement radii. The centrepiece was always the campfire, which ultimately gave the show its title. We were then simply allowed to play around it, and so the barge Hedi could also be pulled into the depths by an octopus in the harbour basin.
Not a machine took care of the missing sound design for us, giving the scenes even more life, so that we were actually finished with the animations the day before the Campfire Show was recorded. Precision landing.
Chapter 5: The future
At the end of our story, we look to the future. The Smartsuit definitely has great potential in character animation. With other means, it would have been impossible for us to animate anywhere near as many scenes in the length we needed for the Campfire Show. But you can also see that the product has not yet been fully developed. In any case, the weak points are the transmission of data, the unstable software and the sometimes imprecise positioning of the sensors in relation to each other. Nevertheless, we are already looking forward to the next use. Because thanks to the Smartsuit, we can now think in completely new directions in terms of design.
