Hello lucid

World models are getting more interesting by the week, but actually trying them is still harder than it should be.

Not only are most people GPU-poor, myself included, but every model tends to come with its own conditioning logic and inference quirks. Some of that is annoying to wire into a real-time loop, and some of it is just brittle enough that even codegen tools often get it wrong. As a result, many interesting world model releases go under the radar.

I went looking for an open-source library that made this easier and couldn't find one, so I started building one: Lucid Runtime.

Lucid is a tiny real-time library for world models. Lucid separates your model from real-time transport and interaction plumbing. Porting a model stays slim instead of turning into yet another rewrite of session handling, controls, and output plumbing.

Lucid has a declarative API:

from lucid import LucidModel, LucidSession, SessionContext, hold, input, publish
 
 
class MySession(LucidSession["MyModel"]):
    def __init__(self, model: "MyModel", ctx: SessionContext) -> None:
        super().__init__(model, ctx)
 
    @input(description="Seed the initial frame", paused=True)
    def set_seed_frame(self, image: InputFile = image_input(size=(WAYPOINT_FRAME_WIDTH, WAYPOINT_FRAME_HEIGHT))) -> None:
 
    @input(description="Update the prompt", paused=True)
    def set_prompt(self, prompt: str) -> None:
        self.prompt = prompt
 
    @input(binding=hold(keys=("KeyW",)))
    def move_fwd(self, pressed: bool) -> None:
        self.active_inputs.add("fwd") if pressed else self.active_inputs.discard("fwd")
 
    async def run(self) -> None:
        await engine.start_session(
            self.prompt,
            seed_frame=self._pending_seed_frame,
        )
        while self.ctx.running:
            frame = self.model.render_frame(self.prompt, frozenset(self.active_inputs))
            await self.ctx.publish("main_video", frame)
 
 
class MyModel(LucidModel):
    name = "my-model"
    session_cls = MySession
    outputs = (publish.video(name="main_video", width=640, height=360),)
 
    def create_session(self, ctx: SessionContext) -> MySession:
        return MySession(self, ctx)

Actions that condition the model are exposed with the @input decorator. Outputs are declared in a similar way with publish.video(name="main_video"), and the same pattern applies to audio or other data tracks.

Your average clanker should be able to one-shot a model port.

The library is still in a very early phase, and I'm still feeling out the right API. Once I'm happy I want to start porting different open-source world models on top of it.

It's early days, but if this sounds interesting I'd love feedback. The code is on GitHub. I'll keep posting updates as the runtime evolves.

Lucid dreams!