How I Pick Up Where a Dead Runtime Left Off

Introduction

A. Work that outlives the machine doing it

My laptop's GPU has 4 GB of video memory, or VRAM — the memory a model must fit into to run on the GPU. Generating images with SDXL, an open image-generation model, needs far more than that, so the generating runs on a Google Colab runtime — a virtual machine Google lends for the length of a session — with a T4 GPU and about 15 GB of VRAM. The laptop prepares the work and reviews the results; the runtime does the part that needs the GPU.

The work is long. A single project can mean a hundred images or more, each taking the better part of a minute and a half once the quality passes are counted, plus several minutes at the start of every session just to download and load an eleven-gigabyte model. A run measured in hours is normal.

A Colab runtime is not a machine I own. It is lent for a session, and the session ends. Some of the ways it can end are preventable — a runtime left idle is shut down, and that can be stopped. Two are not: every session has a maximum length, and the free GPU comes with a daily quota. When either runs out, the runtime is gone, along with everything in its memory and on its disk.

The challenge is to pick up where a dead runtime left off — to carry hours of GPU work across runtimes that are certain to end, without losing what was finished and without doing it twice.

The word that shapes everything below is certain. This is not error handling for an unlikely event. The runtime will end, probably in the middle of something, and the next one will start with nothing. Designing for that changes which parts of the system are the main path and which are the exceptions: surviving a restart is not a recovery feature added at the end, it is the ordinary way the work proceeds.

Easy to state, awkward to build. The difficulty is not in saving results — every result is written to shared storage the moment it exists. It is in everything a new runtime needs in order to know what those results mean: which are complete, which are missing, which were never needed, and whether it should start working at all.

B. What a new runtime starts with

It helps to list what survives the end of a runtime, because it is very little.

Nothing in memory survives. Every variable, every queue, every flag saying what was running — gone. Nothing on the runtime's disk survives either, which includes every model downloaded during the session. What survives is the shared storage outside the runtime, where images and the records describing them are written as they are produced, and the interface on the laptop, which never stopped running. That split decides where every piece of state has to live: anything the next runtime will need must already be in storage, because there is no moment of warning in which to put it there.

Less obviously, the new runtime is not quite the same machine as the old one. Its libraries are installed fresh when the first cell runs, from whatever versions happen to be current that day. Code that behaved one way on Saturday can behave differently on Monday without a single line of it changing.

So a new runtime has to rebuild its whole picture of the work from the outside, on a machine that may differ from the last one, and without spending the GPU on anything before I have decided it should. Those three requirements, more than the saving of results, are where the obstacles came from.

C. Eight obstacles in the way

Some deaths can be prevented, and some cannot. A runtime that looks idle gets shut down, and a background process does not count as activity. That one can be prevented. Session limits and quotas cannot, so they have to be designed around rather than fought.

Some deaths leave no message. A runtime that runs out of system memory is not told so. The process is simply killed, and the log stops mid-line with no error at all.

Status cannot live in memory. Whatever the last runtime knew about which items were done disappeared with it. The next runtime has to find out again, from evidence it can trust.

The order of writes decides what a death leaves behind. Saving one result means writing an image and a record that points to it. Dying between those two writes is expected, and one order leaves harmless debris while the other leaves permanent damage.

"What is left to do" needs exactly one definition. Left alone, the progress bar, the queue, and the status map each form their own idea of the remaining work, and they can disagree by a factor of twenty.

Each new runtime is a slightly different machine. The same notebook that runs on Saturday can freeze on Monday, from a change nobody made.

Starting over costs gigabytes. Every new runtime downloads its models again. Some of that is the unavoidable price of a borrowed machine; some of it turned out to be waste.

A new runtime must not spend the GPU on its own. The easiest way to resume is to resume automatically, as soon as the runtime connects. That was deliberately not done.

The order moves from the death itself, through what the next runtime has to reconstruct, to what it has to be protected against. The first two are about how runtimes end; the next three about recovering an accurate picture of the work; the last three about starting again on a machine that is new every time.

Method

A. Preventing the deaths that can be prevented

Here is the answer to the first obstacle. Colab shuts down a runtime it considers idle, and it judges idleness narrowly: activity in a browser tab of my own interface does not count, and neither does a background thread. The first solution was a dedicated cell, run last, that printed a status line every minute:

KEEPALIVE_INTERVAL_S = 60      # one beat per minute
KEEPALIVE_CLEAR_EVERY = 60     # clear the output roughly every hour

It worked, and it was the wrong shape. The worker itself ran in a background thread, started by an earlier cell that finished within seconds, and the keep-alive cell kept the runtime alive while doing no work of its own. Worse, a background thread dies silently when the cell that started it ends.

The later design removed the keep-alive cell entirely by making the worker's own loop the thing that blocks:

def run_worker():
    # blocks on purpose: a daemon thread dies silently when its cell ends,
    # and Colab only counts a blocking cell as "running"
    try:
        while True:
            _wake.wait(timeout=5)
            ...

A blocking loop is its own keep-alive, and it has a second benefit: stopping the cell interrupts the loop directly, so cleanup code gets to run instead of the thread vanishing.

What remains after that is the part that cannot be prevented. Session limits and the daily GPU quota end runtimes on their own schedule, and in practice they are the only things that do. Everything in the rest of this article exists because of those two.

B. Making a silent death readable

This answers the second obstacle. A runtime can die in the middle of loading the model, with the log stopping just after a routine warning from the library and just before the line announcing the model is ready — and no traceback at all.

No traceback means no exception: the process is killed from outside. Between those two log lines there is only one operation, moving the model onto the GPU. During that copy the weights exist in system memory and in video memory at the same time, and the free tier has about 12.7 GB of system memory. Without measurements, that cause is an inference, not an observation.

So the first half of the answer is not a fix. It is making the cause visible whenever it happens, with measurements taken from sources that do not mislead:

def ram_available_gb() -> float:
    # MemAvailable, not MemFree: MemFree ignores reclaimable cache
    for line in Path("/proc/meminfo").read_text().splitlines():
        if line.startswith("MemAvailable:"):
            return int(line.split()[1]) / 1024 / 1024
    return -1.0

Video memory is read from nvidia-smi rather than from the framework's own allocator, which only sees its own allocations. Both are logged before and after every step that would otherwise be dark.

The second half reduces the peak, which is what gets a process killed, rather than the total. A reference to the image encoder is released once the pipeline has it, instead of holding about 2.5 GB for the rest of the function. And the model moves to the GPU one component at a time, with memory reclaimed between each:

for nama in ("text_encoder", "text_encoder_2", "vae", "unet",
             "image_encoder"):
    komponen = getattr(pipe, nama, None)
    if komponen is None or not hasattr(komponen, "to"):
        continue
    komponen.to("cuda")
    gc.collect()

The alternative that sounded best was rejected on reading its documentation. Automatic device placement in the library was marked experimental, supported only one strategy for pipelines, and conflicted with the explicit .to() calls the code already made. Adopting it would trade a risk that is understood for one that is not.

C. Rebuilding status from the evidence

This answers the third obstacle. A new runtime does not ask what the last one was doing. It rebuilds the entire status map from what exists in storage, every time a project is opened:

def item_done(name, rec, subdir):
    img = rec.get("image")
    return bool(img) and img in _images.get(name, {}).get(subdir, set())

An item counts as done only when two independent facts agree: its record points to an image, and that image actually exists. Either one alone is wrong in a way that has happened. Checking only for files would count a stray image as finishing an item whose record says nothing was generated. Checking only records would count an item as finished after its image had been deleted.

Knowing which files exist has to be cheap, because it happens on every start. Checking each item individually against the storage meant 2,131 separate requests. Two listings of the image folders give the same answer:

def image_index(name):
    def names(prefix):
        return {o["key"].rsplit("/", 1)[-1] for o in s3_list(prefix + "/")}
    return {
        "images/objects": names(pkey(name, "images/objects")),
        "images/jobs": names(pkey(name, "images/jobs")),
    }

The effect is that a runtime killed at any moment leaves nothing that the next one needs to be told. The truth is in storage; memory only ever held a copy of it.

D. Writing in the order that survives

This answers the fourth obstacle. Saving a result takes two writes: the image file, and the record that says which image belongs to this item. The runtime can die between them. It will, eventually.

rec["image"] = save_image(name, subdir, prefix, img)   # the image first
write_json()                                           # then the record
st[key] = "done"                                       # only after both

With the image written first, a death in between leaves an image with no record pointing to it — one wasted file, harmless, and never shown. With the record written first, a death in between leaves a record pointing to an image that does not exist. By the two-condition rule above that item would be caught; without that rule it would look finished forever while showing nothing.

The status is marked done only after the record is safely written, for the same reason. And images are never overwritten: each is saved under a new versioned name, so writing again after a crash cannot damage a file that a record already points to.

E. One definition of what remains

This answers the fifth obstacle. Without one definition, the most basic question after a restart — how much is left — gets three different answers, as a measurement on a real project shows:

# "remaining work", three ways, on the same project
progress bar denominator    2131   (every shot, every object state)
queue built by the worker    353   (252 of them referenced by nothing)
actually needed              101

A progress bar that counts everything that exists, including hundreds of shots not prepared yet, can never reach the end. A queue that counts every object state with a prompt includes states no prepared shot uses — hours of GPU time spent on images nobody would look at.

The answer is a single function defining what needs to be generated, used by both the queue and the progress calculation, so the two can no longer disagree. One distinction was kept on purpose. The status map is not narrowed to that definition. It records what is observably true of the world — does this image exist — while the work set records what is intended. Merging them would make an unused image that had already been generated look pending, and it would be generated again for nothing.

F. A machine that is new every time

This answers the sixth obstacle. A notebook that runs correctly on a Saturday can, on Monday, with the repository unchanged, stop immediately after the first image's progress bar reaches its end — no new line, no error, the cell still marked as running.

The difference is the runtime. A new one installs its libraries fresh, so a library that fails to import on Saturday may import successfully on Monday. That switches on an optional quality feature, which downloads a 600 MB scoring model the first time it is needed — without a time limit, and while holding the lock that gives one task exclusive use of the GPU. If the download hangs, it holds the GPU hostage while it does. Saturday works only because a library fails to install.

Two decisions answer it. Optional models are loaded before taking the GPU lock, so a slow download delays one item without freezing the rest:

warm_quality_models(setting)
with GPU_LOCK:
    load_model(setting.get("model") or MODEL_ID)

And every such load runs with a limit, announcing itself before it starts:

print(...)   # announce the load first, so a hang has an address
th = threading.Thread(target=_run, daemon=True)
th.start()
th.join(timeout)
if th.is_alive():
    ...      # give up waiting, fall back to working without it

The rule behind both is that an optional quality step is allowed to reduce quality and nothing else. It may never stop generation. A timeout that falls back to "without this feature" enforces that. Printing before the download begins means that if something does hang, the log says where.

G. The cost of starting from nothing

This answers the seventh obstacle. Each new runtime downloads roughly ten gigabytes of model weights before it can generate anything. Caching them in the shared storage was tried and abandoned: at the 2–4 MB/s the storage achieved, uploading the weights took tens of minutes more than downloading them again from their public source. The re-download is accepted as the price of a borrowed machine.

Accepting a cost is not the same as ignoring it. A download pattern with a wildcard would fetch the image encoder's folder, which holds the same weights in two formats — 2.53 GB twice in every session. So the download names each file:

IP_ADAPTER_PATTERNS = [
    f"sdxl_models/{IP_ADAPTER_WEIGHT}",
    # named one by one, not "models/image_encoder/*":
    # that folder holds the same weights in two formats
    "models/image_encoder/config.json",
    "models/image_encoder/model.safetensors",
]

A download that happens once is easy to leave inefficient. A download that happens at the start of every session, on a clock that is already running down, is worth reading line by line.

H. Waiting to be told

This answers the eighth obstacle. When a new runtime connects, it loads the model, announces that it is online, and then waits. It does not look at the queue and start. The GPU is used only after I press Start in the interface:

with _qlock:
    name = _active["name"]   # None until Start is pressed in the interface
    key = None
    if name is not None:
        ...

Another of my projects, a PDF converter, does the opposite: on connecting it scans for pending work and begins immediately. For that project it is the right choice. For this one it was rejected, for four reasons. A runtime that resumes on its own can spend the daily GPU quota on a hundred items before I notice. Between sessions I often want to correct prompts before anything else is generated. There may be more than one project, and the worker cannot know which one I mean. And none of that waiting is expensive, because the model stays loaded while the worker waits — releasing it and loading it again would cost minutes each time.

Resuming, in other words, is a decision rather than a reflex. The runtime's job is to be ready; whether to spend it is mine.

Results and Discussion

A. What the eight decisions add up to

A session can now end at any point without losing a finished result or leaving a broken one, and the next session picks up the remaining work — the right remaining work — once I tell it to.

B. Death on a schedule

The common thread is a change of assumption. Most software treats its own death as an exception, and puts the handling of it at the edges. A borrowed runtime reverses that: ending is the one event that is guaranteed, so the ordinary path has to be built around it.

That shows up in unglamorous places. It is why status is rebuilt from storage rather than remembered, why the image is written before its record, why there is one definition of remaining work instead of three, and why a new runtime waits instead of charging ahead. None of these is a recovery mechanism. Each is simply the way the normal path works once it assumes it will be interrupted.

The second thread is that a new runtime is new in more ways than its empty memory. It installs different library versions, downloads its models from scratch, and can die in ways that say nothing. Making each of those visible — memory logged before risky steps, a message printed before every download, a timeout on anything optional — turns a mysterious stop into one that can be explained the next morning.

The general version: when a machine's end is certain, stop designing recovery and start designing the normal path so that it has nothing to recover from.

C. What this does not cover

One item in progress when a runtime dies is lost and done again from the beginning. With items taking well under two minutes, finer-grained checkpoints were not worth the complexity; for work where one item takes hours, they would be.

This is also a single worker. Two runtimes working on the same project at the same time would need a way to claim items, so that both did not generate the same one. The design assumes that never happens, and for one person with one borrowed GPU, it does not.

Conclusion

A 4 GB GPU sent hours of image generation to a Colab runtime, and every Colab runtime ends — by session limit or by daily quota, whichever comes first. Idle shutdowns could be prevented, and are, by making the worker's own loop the thing that keeps the runtime busy. The rest could only be planned for.

Planning for it meant treating each new runtime as a machine that knows nothing and may not even be the same machine. It rebuilds its status from storage, trusting an item as done only when record and file agree. It relies on results having been written in an order that a crash cannot corrupt. It uses one definition of what remains. It loads optional models outside the GPU lock and with a time limit, and it waits to be told before spending the quota.

What carries over is the change of assumption. When the end of a machine is certain, recovery stops being a feature and becomes the design of the ordinary path.

Terms

runtime

A Colab session: a temporary virtual machine, with its own memory, disk, and GPU, that exists only until the session ends.

keep-alive

Anything that keeps a runtime from being judged idle and shut down. Here, a cell whose code blocks for as long as the work continues.

OOM

Out of memory. When system memory runs out, the operating system kills a process outright, so no error is raised inside the program.

peak memory

The highest amount of memory in use at any single moment. A process can be killed at the peak even when its average use is well within limits.

GPU lock

A lock ensuring only one task uses the GPU at a time. Anything slow done while holding it stalls every other task waiting for the GPU.

work set

The single definition of which items still need to be generated, shared by the queue and the progress display so the two cannot disagree.