How do you direct performance inside closed generative video systems when you don’t have model access?

Not prompt hacking.
Not emotional adjectives.
Not “make it intense.”

Direction.

Generative video systems are improving at a breathtaking pace.
Which paradoxically makes them harder to write about.

New model releases outpace critical reflection. Bugs vanish between versions. Artifacts disappear without being acknowledged. Workarounds become “features.” And consensus around best practices forms quietly in Discord servers long before anyone formally documents it.

Yet that is precisely why documentation matters.

If we only write once the systems stabilize, we erase the most interesting part — the transition itself. The glitches, the workarounds, the unintended aesthetic artifacts. The moments when creators collectively reverse-engineer emergent behavior before the companies fix it.

The history of generative film is not just the polished output.
It’s the evolving relationship between creators and unstable tools.

Lighting is cinematic.
Camera motion feels intentional.
Visual style holds together across shots.

But performance — the thing that actually makes a scene alive — is still unstable.

Characters drift.
Emotions overshoot.
Dialogue slides between actors.
Continuity collapses between takes.

And most creators are working inside closed commercial systems. No model access. No fine-tuning. No control beneath the prompt box.

So the question becomes:

How do you direct performance when you cannot direct the model?

Over six months, I’ve explored this through structured experimentation: generating conversational scenes, refining prompt structure, repeating shots under controlled conditions, and testing whether principles from film acting pedagogy translate into closed generative systems.

What I found:

You cannot command performance.
You can only shape the conditions under which it might emerge.

Reframing the Problem

Most prompts name emotions.

“Angrily.”
“Embarrassed.”
“Sheepish smile.”
“Devastated.”

And the systems deliver something legible — but often exaggerated. The performances read clearly in isolation but become difficult to edit together across shots.

Traditional acting pedagogy almost never directs emotion directly.

Instead, directors specify:

• The circumstances

• The stakes

• The objective

• The observable action

Given who you are, in this situation, what do you do?

When prompts were reframed around circumstance and observable action rather than affect labels, something changed.

Performance did not become controllable.

They became narrower.

More grounded.
More editable.
More consistent across takes.

The Core Insight: Probabilistic Direction

Closed generative systems are not deterministic tools.

You are not instructing an actor.

You are not controlling a puppet.

You are shaping a probability field.

Prompts do not specify outcomes.
They constrain behavioral possibility.

Creative control happens in three places:

1. Prompt discipline

2. External identity scaffolding (persistent visual anchors like distinctive clothing or accessories)

Editorial selection across multiple generations

You are not directing a single take.

You are directing across takes.

But unlike human actors, generative systems do not “lock” performance between iterations. Even when scene structure, camera framing, and prompt language remain constant, performance elements fluctuate.

Observed Failure Modes

Figure 1. Two takes generated from identical prompts. While composition and scene geometry remain stable, tonal intensity, micro-expression, and eye-line direction shift between outputs, demonstrating probabilistic performance drift.

Across repeated scene generations, three patterns consistently appeared:

1. Performance Drift
Characters shift tone, intensity, gaze direction, or micro-expression between takes, even under identical prompts.

This drift is not pure noise; it appears statistically bounded yet semantically meaningful. The system preserves structural constraints but re-samples performative nuance.

2. Dialogue Reassignment
Lines and reactions migrate between characters.

3. Over-Signaling
Emotion-labeled prompts produce theatrical, exaggerated performances that do not cut well.

In many cases, the scene was visually beautiful — but unusable due to performance instability.

What Improved Results

When prompts emphasized:

• Physical actions instead of emotions

• Behavioral descriptions instead of affect labels

• Situation before reaction

the system produced performances that were:

• Less theatrical

• More grounded

• Easier to assemble across shots

Not predictable.
But narrower in range.

That difference matters.

Why This Matters Beyond Experimentation

This isn’t just about better prompts.

It’s about redefining creative control in AI systems.

In closed generative video tools, directing becomes:

• Constraining behavior probabilistically

• Designing continuity scaffolds

• Selecting from probabilistic outputs

• Editing as authorship

The model interprets.

The creator selects.

Direction happens in the selection.

Where This Is Going

I’m particularly interested in how these directing principles can translate into structured, reproducible workflows for AI filmmaking education across different learning environments.

The goal is not to teach students how to “prompt better.”

It’s to teach them how to direct.

There is a difference.

Over the coming months, I’ll be expanding these experiments, interviewing filmmakers working at the edge of AI performance, and exploring how circumstance-driven prompting might become part of a teachable creative framework.

Because visual fidelity is no longer the bottleneck.

Performance is.

Three Pieces - referenced

Here:

…but then there was something uncanny valley about the lips movements and the eyes and so I bailed on the testing for other animation methods.

Last week I tried the Unreal 5.5 Audio2Face Lip Sync on a metahuman - here was the result:

I thought the voice over had improved alot but the Metahuman still looked a bit fake - like 90s Machinima (a bit Half Life 2).

But then over the last few days I have been experimenting with upgrading the skin shaders and textures on the Metahumans using the 3D scan Store identities. Here is what I came up with:

This is the raw footage for and ad that I am working on for a new project. I channeled my Dads voice for the voice over. It seems like you can puppeteer a lot of nuances in a performance. I am super happy with the result and feel inspired to go back to some of my earlier animation attempts and redo some earlier animations with this new method. Who knows it may end up being the look and style I have been searching for all these years. State tuned!

“Got a letter in the mail, go to Iraq or go to jail

And it won’t be long, ‘till I get up back home.”

• Army marching song

When I was growing up everyone had a hero. Mine was the world famous “escape artist” Harry Houdini, because he could escape from anything that he got tangled up in: locked safes, underwater contraptions, jails. I wanted to be – I needed to be – someone who could always find their way out of any situation (more about that primal fear coming up, but first: 2005). Every Thursday afternoon I leave work early to beat the traffic. On my way into town I check the mailbox because, with American troops in short supply, the Pentagon is searching for soldiers who still owe time on their service contracts to ship out to Iraq.

As a member of the Inactive Ready Reserve I’m uneasy on these weekly trips so I hum army songs to distract myself from thinking about what life would be like if I were to get the “battle call”. Of all the disastrous things that could happen to me overseas,nothing terrifies me more than my nightmare of being treated as an American POW in a prison like Abu Ghraib.

When I joined the Army I was given a Geneva Conventions ID card. If I was ever captured I was to show the card in order to guarantee my basic human rights, but after seeing the home movies of the Iraqi prisoners it’s clear nobody follows the rules, that the Geneva card is meaningless. I’m not nearly so afraid of driving over a land mine, being shot or getting hit by an RPG, as I am petrified of being treated like an American POW.

As I was telling you, I used to be deathly claustrophobic. The thought of being confined or blindfolded paralyzed me. As a kid I tried every home remedy to rid myself of this asphyxiation phobia: I used to crawl inside my sleeping bag the wrong way, under my comforter, under my bed and try and see how long it would take me to escape; when all the other kids were playing with their hot wheels in the sand pile I was creating elaborate labyrinths full of deadly traps for ants, I would grab the biggest Carpenter ants that I could find and drop them into my mazes to see how long it would take them to escape. If they happened to take a wrong turn they would end up getting 1) fried by a magnifying glass that I set up; 2) set adrift in the lake on a piece of wood, or, 3) end up in a pit where I would make them fight other ants by grabbing there heads and forcing their pincers to lock on to each other. In the winter I would dig snow tombs and throw dogs and kids in snow suits inside and – that’s enough, you get the point. I realize now that these sadistic childhood games (the torture – let’s say it) I played with those ants was a pathetic attempt to overcome the terror and powerlessness I was feeling then. That’s how I know, now, what drove those young US prison guards to be tricked and seduced into playing out the spymaster’s game at Abu Ghraib, where, they believed, you were either the victim or the torturer, one or the other, zero/sum, me or them, me or the ants.

I know all about it now – terror and torture – because I was lucky back then at ten years-old when I read the biography of Harry Houdini and thought that, at last, I had found the perfect role model. His ability to escape from everything was empowering. I no longer had to play God with the ants to overcome my feelings of helplessness. It was all a trick but it gave me an ideal to hang onto, if I could just figure it all out.

The Houdini biography had pictures in the middle of the book that showed the master of illusion tied up in various contraptions. Thinking that if I learned some of his techniques I would be able to escape the panic of my phobia and be cured, I went into the basement and with a chair, a sleeping bag, a garden hose and some rope, I asked my brother and his friend Tommy to put me into something that I wouldn’t be able to get out of. They sat me in the chair, tied my hands behind my back and stuffed the sleeping bag over my head, then wrapped the hose around the sleeping bag. Tommy kicked the chair over and they turned out the lights and split. I was trapped! We were living in Puerto Rico at the time and had just had the basement “hurricane proofed” – so nobody could hear my screaming. About an hour later my Dad came into the basement looking for his toolbox. He cut me loose. Another failure and my terror dogged and haunted me through High School and after – until I joined the Army.

I used to be a high school stud: Now I’m crawlin’ in the mud

A few months ago the wife and kids of an army buddy, currently in Iraq, came to stay with me for a few weeks. His tour kept getting extended; first he was coming home for Christmas, then it was Easter, then maybe for Tyler’s (his son’s) birthday. He’s yet to come home.

My friend’s wife, Sue, got sick of the wait and took her kids to Disney Land for a vacation, but it soon became clear that a big part of her decision to come to L.A. was so that her boys could spend time around a male, someone whom they associated with their Dad. So, at night Sue sent me downstairs to tuck the boys in. This was a difficult mission. A week before the family came to visit I had wiped a bunch of photos that my friend had sent me off my hard drive. I knew the kids would want to play Doom on my computer – Christ Almighty! – seeing what their Dad was really doing in Iraq. (He had e-mailed me a picture of himself smoking a Cuban cigar – they had found Saddam’s private stash – with a big grin on his face, machine gun in hand, standing beside a hooded Iraqi prisoner.) I walked down the steps, to tuck in his sons, trying to come up with a way to put a happy spin on the world of their missing Dad – and there were the boys: a pair of seven-year old and fifteen-year old eyes watching my every move.

“Why is my Dad there and not you?”

“When’s my Dad coming home?”

“Is he going to die?”

I used to date the high school queen; Now I date my M-16

I keep trying to figure out what possessed me to join the military in the first place. I guess I was drawn to the idea of being part of a group, belonging to something, hiding in plain sight. I wanted to be a part of what soldiers feel towards each other. When I got to boot camp I figured out that war was all a game, and mastering it became the greatest escape of my life. I no longer had to be stuck being the same person everyday. I could see that it was all like a play with the Drill Sergeant as an acting coach. I would wake up in the morning on my cot, wiggle my toes, and there I was the same old me. Then Drill Sergeant Ballard would walk in, wearing mirrored Aviator sun glasses to look like The Terminator, screaming, “Get the hell out of bed and into the pit”. As I did push ups in the mud pit day after day, living out my Drill’s torture fantasies, I would think about how great it was to escape from myself: no more unpaid bills, dead end desk jobs, family obligations. I was free! I was an American soldier and it was the greatest acting role I had ever been cast to play. And I got to play it over and over again.

That’s what I thought. My private life at the time was, to paraphrase the army’s slogan, “a disaster of one.” All of us who fell for the trap had what we thought were reasons. But the reasons behind the reasons are another story about power and men, and finally about being so goddamn young that the military looks like a home when it’s really a cage.

But, then, in boot camp someone leaked to Drill Sergeant Ballard that I was claustrophobic.

“I went to SERES school, Private, I know just what you need.”

That’s “Survival, Evasion, Resistance and Escape School”— the one where they treat you like both a German and a Japanese POW for a week. A place that reminds me of the dreaded Room 101 from George Orwell’s 1984 – a torture room where one’s worst fears are made real. During your incarceration they try and break you down by using a variety of torture techniques ranging from sleep deprivation, forced standing, choking in water, solitary confinement, humiliation, etc. – and it’s the “etcetera” that has me sweating.

Drill Sergeant Ballard’s “claustrophobia remedy” involved stuffing the entire platoon “nut to butt” into a medic transport truck, the kind that locks from the outside. He then proceeded to drive the truck around and around in circles. When he got tired of that he stopped the truck, climbed on top of it and starting jumping up and down on the roof. I lay between two privates in the fetal position with my eyes closed. Mortified. I don’t know what the hell we were supposed to gain from his guerilla psychotherapy exercise but it, too, failed to cure my affliction.

Standin’ tall and lookin’ good; I should be in Hollywood

After basic I was sent to train as an army propagandist. I learned how to design “battlefield advertising” leaflets that we drop from planes; how to gather intelligence from POW’s, and how to use a loud speaker. All through training I was looking forward to the day I would get deployed and do my job for real. This was during peace-keeping time, when Bosnia and Kosovo were the deployments. Our job “in country” was to flex our military muscle to keep the peace, educate kids about land mines, and fix roads. It was very Peace Corpesque but with more perks; a signing bonus, student loan repayment and the GI Bill.

But now I know that the Army is a two-headed beast: Nation Builder/Death Machine; the high school kids scraped up to feed the monster only to find out the truth too late…

What I really did when I signed up was to put my Social Security number into a lottery system. The winners are the ones who use the army for the benefits, do their time, and get out without permanent mental or bodily harm. The losers are those of us whose numbers are called, who ship out and are killed or permanently wounded for something that they don’t even know that they don’t believe in. The grunts and the dog faces are there on the killing ground like the ants were, back then, trapped in my maze – they may think that God and the Flag are over them, but it’s only fear that fills up the sky.

I pull into a free parking space at “Mailboxes Ect.”. I ‘m still humming “I used to drive a Cavailer, now I’m humping all this gear… And it won’t be long”…

2024-12-05

I wrote this essay during my stint in LA back in 2004, where the struggle was glamorous—or at least that’s what I told myself. It somehow made its way into Now Magazine on May 26, 2005. Later, animators Paul Beck and Jason Archer—yes, the Austin legends behind Waking Life and A Scanner Darkly—graced the project with their brilliance, likely out of curiosity or a lapse in judgment.

The film went on to win Best Animation at the Cinéfest Sudbury International Film Festival and Best Short Film at the Rincón International Film Festival in Puerto Rico (where I spent my childhood). It also screened at the Tiburon International Film Festival in Marin County, the backdrop of my 90s coming-of-age saga.

I’m posting this as a sneak peek of what’s to come in my book, Riding the Rogue Wave. Consider it the appetizer to the main course of chaos and self-discovery.

Initially, I thought I’d use the tapes as background material for Rogue Wave, the book I’m working on. But a new idea struck: why not finally remix and release the project? The catch? Those tapes weren’t just a creative endeavor—they were also my safety net. Back then, when things felt chaotic, the camera became my shield. It gave me a purpose, a reason to say, “I’m making a documentary,” and keep moving forward.

In the 90s, the indie film dream was alive and well. Everyone in Hollywood aspired to be the next Tarantino, hoping their quirky, low-budget project would catapult them into stardom. I was no different, chasing that elusive Swingers or El Mariachi moment.

Fast forward to today, and the biggest hurdle to releasing this footage is consent. Some of the situations I shot were, let’s say, less than straightforward, and I’m not confident all the subjects would be on board with signing release forms. My workaround? Innovating with animation and sound.

I developed a workflow where I alter the pitch of the audio to anonymize voices and layer in sound effects. For the visuals, I’m recreating scenes with animated sequences using MetaHumans and a toon shader in Unreal Engine 5.5. This process leverages Mixamo’s vast library of animations—thanks to the Mixamo Conversion Tool by Terribilis—and saves a ton of time. Unreal’s new Voice-to-Lipsync feature has been a game-changer, finally delivering believable lip sync without veering into uncanny valley territory.

Here’s how it all comes together: I “shot” MetaHumans on a virtual greenscreen, masked out the green, and overlaid them on backgrounds taken from the original 90s footage. I didn’t need to anonymize myself since I fully consent to appear.

The results? A surprisingly solid hybrid style. With MetaHumans, Mixamo animations, and assets from the Unreal FAB store—props, vehicles, and more—I’ve created a blend of the old and the new. The lip sync looks decent, and the animations feel smooth, making this approach a viable way to breathe life into this decades-old project.

That said, there are still kinks to iron out. I noticed ghosting in some shots, where animations bleed through the background, and I need to split the audio tracks to avoid the AI doubling up on lines spoken by both characters. For now, I’ve tested just a minute of footage, keeping things manageable to avoid getting too deep before refining the process.

Stay tuned for V2! This is shaping up to be something different. It was a fun day!

4o