Why a walk to the beach needed a 3D scene

The beach is far below the apartment window and right behind the door. To make that walk from four generated views, we rebuilt the exterior in Blender and used it to prepare consistent stills for the video model.

This is a Rift City apartment, following the one we showed in Cribs. The door opens onto the beach visible through the window. It is the kind of door I would love to have at home.

In Muse, our node canvas at SplayFlo, each Generate Video node takes a start still and an end still. We needed three clips: the turn toward the closed door, the door opening, and the walk outside.

Each clip ends on the still that starts the next. That connects the clips at their boundaries; the model still has to invent the movement between them.

Where the images broke#

The original views were generated separately. A bookshelf changed angle between room views. Outside, the nearest stone post looked close to the door from inside but farther down the path from outside. As the camera moved, objects warped or appeared where they did not belong.

We revised the prompts too, but the images themselves needed repair: their paving, posts, and doorway had to describe a place the camera could move through.

For the turn, the camera rotated about one fixed point. We built one wider picture of the room and sampled both views from it, so the window, furniture, and doorway followed the same rotation. The coffee table and a small plant needed artwork repairs before the views agreed.

The walk needed depth#

Walking changes the camera's position. Nearby posts move across the view faster than the distant coast, and surfaces hidden behind the door frame become visible. Aligning two flat pictures could not account for all of that.

The reconstruction started from the illustrated views. Python scripts fitted the camera and placed the walkway, threshold, posts, and rails in a shared coordinate system.

Each post had a fixed footing; each rail met the same two posts. The far city stayed a painted background. The nearby posts and rocks had geometry, with artwork attached to their surfaces.

We used the scene to render the exterior from the two camera positions. The inside view was composited into the existing doorway, keeping the apartment around it. The outside view became the walk's end still. Those images went to the video model; the scene and its camera path stayed outside Muse.

Geometry was not enough#

Some versions connected geometrically but had smeared or blurry artwork. The city occupied only about 490 × 350 pixels in the earlier source painting. Rendering a larger image did not recover missing windows, roofs, or foliage.

We generated more detailed paintings for the coast and individual surfaces, then attached them to the existing scene.

The paving needed a fixed network of joints as well as stone texture. A grain overlay left the old blurry shapes underneath, so those interiors were replaced while the joint layout stayed consistent between views.

The last repair was smaller: the nearest post was smooth yellow stone while its neighbors had a mottled texture. We reused a neighboring post's artwork on it. The geometry stayed unchanged, and the final image correction was confined to that post.

For a different image problem, see how we label multiple reference pictures for an image model.

Muse generated the motion#

The repaired stills became the inputs to three Generate Video nodes on MiniMax H3 Max Turbo at 768p: 3 s for the turn, 3 s for the opening, and 5 s for the walk.

Merge Video joined the clips, and FILM interpolated the result from 24 to 120 fps, using the same process as the Cribs loop. The reel's edit then eased the stop at the door and shortened the pause before walking.

The shot worked, but preparing the stills took enough work that I probably would not use this method again.