How I Decide Where Each Piece of GPU Work Belongs

Introduction

A. A small GPU, a borrowed one, and someone else's

My laptop has an NVIDIA GPU with 4 GB of video memory. Video memory, or VRAM, is where a model has to sit while the GPU runs it: the weights, the working buffers, and whatever the current input needs. If the total does not fit, the model does not run slower — it does not run at all, or it runs on the CPU at a pace that makes it pointless. Four gigabytes is enough for a lot of everyday work and not enough for most of the models I wanted to use.

So for a while, the answer to every heavy job was the same: send it to Google Colab. A Colab runtime is a virtual machine lent for the length of a session, and the one I use comes with a T4 GPU and roughly 15 GB of VRAM. Four projects ended up built around that borrowed machine. One turns written chapters into narrated slideshow videos: it generates an image for every moment of the story, a voice for every line of narration, and assembles them into a video. One converts PDF documents into Markdown. One transcribes recordings into subtitles. One edits images. Each of them should have been an ordinary local program, and each was made to run on Colab instead because 4 GB could not hold its models.

Running on a borrowed machine has consequences that reach well beyond the GPU itself. The runtime ends when the session ends, so anything it holds has to be stored somewhere both machines can reach. It cannot be called directly from my laptop, so commands have to travel through a message broker. It can die in the middle of a job, so the work has to be able to resume where it stopped. Each of those needs became machinery: shared object storage, a broker with its own identity and access rules, resume logic, contracts between two codebases that never share a process. The machinery worked. It also had to be maintained, and every piece of it existed only because the work was somewhere else.

There is a third place work can go, and it only became a serious option later: a hosted service. Instead of running a model at all, the work is sent to someone else's model through an API, and the result comes back. No GPU of mine is involved, borrowed or otherwise. What is given up is control over the method: the model is theirs, and so are its behaviour and its limits.

The challenge is to decide where each piece of GPU work belongs — the laptop with its 4 GB, a borrowed Colab runtime, or a hosted service — on the basis of criteria rather than habit, and to let the answer change when the criteria do.

Easy to state, awkward to act on. The habit is strong because it once was right: when nothing fits locally, everything goes to Colab, and every later project copies the shape of the first. Breaking the habit means asking the question again for every piece of work, accepting that one project can end up in three places, and being willing to take apart machinery that took weeks to build once the reason for it is gone.

B. Three places, and what each one demands

The three places are not interchangeable, and it helps to name what each one demands before comparing them. The laptop demands nothing extra: files are on the disk, programs call each other directly, and a failure shows up on the screen in front of me. Its only limit is the one that started all of this — 4096 MiB of VRAM, and whatever the laptop's CPU can do when the GPU is not an option.

Colab offers a GPU almost four times larger, and demands the full set of machinery described above: storage that both sides can reach, a channel for commands, resilience against a runtime that disappears, and an interface that lives outside the runtime so it does not disappear too. It also comes with limits I do not set: session length, daily GPU quota, and rules about what a runtime may be used for.

A hosted service demands the least machinery of all — a credential and a request — and takes away the most control. The quality of the output, the settings that are exposed, and how the model behaves on awkward input all belong to the provider.

None of these is the right place for everything. The useful unit of decision turned out to be much smaller than a project. The slideshow project is the clearest example: its video rendering, its images, and its narration now live in three different places, while the PDF converter and the transcriber stay whole on Colab. That split, and the reasons behind each part of it, is what this article is about.

C. Seven obstacles in the way

"Too heavy" is a feeling, and the criterion has to be a number. Deciding by impression sends everything to the bigger machine. The laptop's limit is 4096 MiB, and a model that needs a little more than that does not fit at all.

The decision belongs to each piece of work, not to each project. Deciding per project, the obvious alternative, kept video rendering on Colab for a long time even though rendering never needed Colab's GPU. It was there only because the rest of the project was.

Some work leaves for reliability, not for memory. The self-hosted narration model fit on Colab and still could not be trusted. Tuning it harder, the alternative I tried first, only moved its failures around.

Some work leaves because the method changes. Image generation did not move to a bigger GPU. It stopped being done with a self-hosted model at all, and that change removed a whole category of Colab concepts from the project.

Work that comes home must not assume the GPU is there. Hard-coding the GPU encoder because this laptop has one would make the program fail on any machine where it does not work, including this one on a bad day.

Coming home should delete machinery. Leaving the storage, the broker, and the resume logic in place "just in case" is the easy alternative. It keeps paying for distance that no longer exists.

What stays remote can still be stripped. The transcriber cannot come home, but not every use of it needs the full surrounding machinery. Keeping one full-featured version for every job is the alternative that was rejected.

The order moves from the criterion to its consequences. First the number, then the unit it is applied to. Then two kinds of departure that have nothing to do with the number, which is where the habit is most misleading. Then what it takes for work to come home properly — without assuming a GPU, and without keeping the machinery. Last, the case where the work cannot come home and the useful question is how much of the surrounding machinery it still needs.

Method

A. A number instead of a feeling

Here is the answer to the first obstacle. The limit is not an estimate. The driver reports it directly:

$ nvidia-smi --query-gpu=name,memory.total --format=csv,noheader
NVIDIA GeForce RTX 3050 Ti Laptop GPU, 4096 MiB

Everything else is compared against that line. The speech-to-text model I use for transcription needs around 5 GB of VRAM for the model size that produces usable subtitles. That is only a little more than 4096 MiB, and "a little more" is exactly the case where intuition misleads: it sounds close enough to try. It is not. Once the weights and working buffers do not fit, loading fails with an out-of-memory error, and the only local options are a smaller model with worse transcripts or the CPU with unusable speed. So transcription stays on Colab, by measurement rather than by default.

The same measurement also shapes how the transcriber behaves on Colab. It reads VRAM usage through nvidia-smi rather than through the framework, because the inference engine keeps its own memory arena that the framework cannot see. When a model does run out of memory, it steps down one model size instead of retrying the same one — retrying an allocation that just failed tends to fail the same way.

The PDF converter stays on Colab for the same reason: its layout and OCR models — OCR being the recognition of text in scanned page images — need a large GPU, and the laptop's is not one. Neither of these is a judgement about how heavy the work feels. Both are comparisons against a number that anyone can read off the machine.

B. Splitting a project into pieces

I stopped placing whole projects and started placing pieces. The slideshow project used to have two workers on Colab, one for images and one for narration, running at the same time against the same project. The image worker also rendered the final video, taking each shot's image for the length of its audio and assembling them into one file with FFmpeg, the common command-line program for converting and combining audio and video. That rendering had never needed Colab's GPU. It ran there only because the images and the project data were there, and because the project as a whole had been placed on Colab.

Asking the question per piece instead gives a different answer for each:

Only two rows are decided by the number from the previous section. The other three are decided by something else, which is the whole point of splitting: a project-level decision would have averaged those reasons into one answer, and that answer would have been wrong for most of the rows. The next two obstacles are about the rows where memory was not the reason at all.

C. Leaving for reliability, not for memory

I moved narration off Colab even though it fit there. Narration was generated on Colab by F5-TTS, a self-hosted text-to-speech model. It fit comfortably in the T4's memory. The problem was that it did not reliably say what it was given. The ends of words were clipped: an Indonesian word such as selangkah came out without its final sound.

The first fix was padding — adding a few spaces after each line so the model would have room to finish. Raising the padding to four spaces only turned selangkah into selangka, still missing its last letter, and produced new clipping somewhere else. The padding was not removing the failure; it was moving it. That matched the cause: this model has no duration predictor. It estimates how long a line should take from the length of the text, and the approach is known to work poorly for short lines. No setting fixes a model that does not know how long to speak.

So I stopped chasing it. The immediate step was a manual shortcut: copy the narration, generate it elsewhere, and upload the audio file back into the project for that shot. The images were already working, and audio could not be allowed to become the single thing blocking a finished video. The lasting step was to move narration to a hosted text-to-speech service, called once per shot:

response = client.synthesize_speech(
    input=sdk.SynthesisInput(text=text),
    voice=sdk.VoiceSelectionParams(language_code=config["language_code"], name=config["voice"]),
    audio_config=sdk.AudioConfig(
        audio_encoding=sdk.AudioEncoding.OGG_OPUS,
        speaking_rate=speech_rate_multiplier(config["speech_rate"]),
    ),
)

Nothing in that call depends on VRAM. The decision to move narration was never about memory: the work already fit on Colab and left anyway, because a model that fits but cannot be trusted is not a place the work can stay.

D. Leaving because the method changed

Images left as well, and memory was not the reason. Images were generated on Colab with SDXL, an open image model that runs well on a T4. It also left, for three reasons. The consistency I needed from one image to the next — the same character, the same objects, the same look across a chapter — was not something I managed to get out of it. Online generative models produced far better images. And I wanted each shot reviewed individually by an operator before it was accepted, rather than generated in a large batch and sorted afterwards.

Images are now produced through an online generative model, driven by an AI agent through a skill that also handles the per-shot review. Changing the method changed what the project records. A beat — one moment of narration and the image shown during it — now stores the brief written for that moment and the image that was accepted for it:

{
  "beat_id": "b0",
  "time_range": { "start": 0.0, "end": 4.7 },
  "visual_brief": "Indonesian production brief for this exact moment.",
  "image": {
    "path": "images/beats/chapter_1_s0_h0_b0.png",
    "provider": "codex-interactive"
  }
}

The brief is a short scene description for a multimodal model, not a list of SDXL tags. The provider field records where the image came from. The project's contract is explicit about what no longer exists: storyboard folders, object keys in shared storage, broker topics, SDXL prompts, and image-generation queues are all gone as concepts. Every one of them had been there to feed a model on a remote GPU. When the method changed, they had nothing left to feed.

E. Coming home without assuming a GPU

I made the renderer test the GPU encoder instead of assuming it. Video rendering now runs on the laptop, and the laptop has an NVIDIA GPU with a hardware video encoder, NVENC. It would be natural to write the renderer for NVENC and be done. But a renderer that requires NVENC fails on any machine without one — and, less obviously, on a machine where NVENC is listed but does not work, which happens after a driver update or with a mismatched FFmpeg build.

So the encoder setting defaults to auto, and auto tests the encoder instead of trusting a list:

def _nvenc_works() -> bool:
    # a build that lists NVENC is not enough: encode one real frame
    probe = ["ffmpeg", "-f", "lavfi", "-i", "color=c=black:s=1280x720:r=30",
             "-frames:v", "1", "-c:v", "h264_nvenc", "-f", "null", "-"]
    ...

if preference != "cpu" and _nvenc_works():
    # use h264_nvenc; otherwise fall back to libx264 on the CPU

The probe encodes a single frame at a normal size, because NVENC rejects very small synthetic frames and a tiny probe would report a false failure. The result is cached, so the check happens once. If it fails, rendering uses libx264 on the CPU. The render is slower, but it does not fail. Work that comes home should be able to live on a machine that is weaker than the one it was written on.

F. Coming home should delete machinery

I deleted the old machinery instead of keeping it. Once the images and narration no longer came from Colab, and rendering ran on the laptop, the slideshow project had nothing left on the borrowed machine. The shared storage, the broker, the resume logic, and the contracts between the interface and the workers had all existed to bridge a distance that was gone. The change that moved the project home was measured mostly in what it removed:

$ git show --stat <migration commit> | tail -1
 67 files changed, 1508 insertions(+), 9975 deletions(-)

Almost seven lines deleted for every line added. Where the project data used to be spread across object keys in a bucket, it is now one folder on the local disk:

/data/studio/projects/<project-slug>/
├── setting.json
├── ori/                  # source chapters and glossary
├── audio/jobs/           # shot audio
├── jobs/<chapter>.json   # narrative structure and beat records
├── images/beats/         # accepted beat images
└── video/                # rendered chapter videos

Keeping the old machinery "in case the work goes back" was tempting, because it had taken real effort to build and it worked. But machinery that is kept without a reason still has to be understood, updated, and secured, and it quietly shapes how new features are written: every new feature would have been built around a broker it did not need. Infrastructure built to overcome a limitation should be loose enough to drop when the limitation goes away.

G. What stays remote can still be stripped

For transcription, I changed what surrounds the work rather than where it runs. Transcription cannot come home, as the first section showed. The full transcriber has an interface on the laptop, shared storage for media and transcripts, and a broker for commands — the right shape for a steady stream of work. For one or two quick recordings, or when the interface or storage cannot be used, all of that is more machinery than the job deserves.

So there is a headless variant: it still runs on Colab, but without an interface, without shared storage, and without a broker. Recordings are dragged into a folder in the runtime, a numbered list is chosen from inside the notebook, and each transcript is written next to its recording. No status is stored anywhere; it is derived from whether the subtitle file exists:

def scan_media(root: Path):
    out = []
    for p in sorted(root.rglob("*")):
        if not p.is_file() or p.suffix.lower() not in MEDIA_EXT:
            continue
        if transcript_path(p).exists():  # an existing .srt means done
            continue
        out.append(p)
    return out

That makes the subtitle file a completion marker, so it is written once, whole, only when the transcript is finished. A half-written file would make the next scan skip that recording forever. A recording that fails leaves no file behind, and is simply offered again the next time. The transcription core itself was carried over from the full version rather than rewritten, so both variants transcribe the same way; only the surroundings differ.

This is a different kind of decision from the others. It does not move the work anywhere. It changes how much is wrapped around the work in the place where it already runs.

Results and Discussion

A. What the seven decisions add up to

The slideshow project now runs entirely from the laptop: project data in one local folder, rendering with FFmpeg on whichever encoder actually works, and images and narration requested from hosted services. The PDF converter and the transcriber remain on Colab, and the transcriber has a headless variant for small jobs. No piece of work is where it is out of habit.

B. Two questions, not one

At the start, the decision looked like one question: where does this run? Working through the seven obstacles showed that it is two. The first is where the work runs — laptop, borrowed runtime, or hosted service. The second is how much machinery surrounds it in that place — an interface, shared storage, a broker, resume logic, or none of these.

The two can be set independently, and the best answers came from setting them separately. Rendering moved home and lost all its machinery at once. The transcriber stayed exactly where it was and still dropped most of its machinery for small jobs. Narration and images left my GPUs entirely, which removed the machinery that fed them without any of it having to move.

Behind both questions is the same observation: the reason a piece of work is somewhere is often not the reason it was put there. Rendering was on Colab because the project was. Narration was on Colab because it fit, not because it worked. Separating the reasons, one piece at a time, was most of the work.

C. What this does not cover

These placements are not permanent. The criteria move: models get lighter, GPUs get replaced, needs change. The answers in the table above describe the current state of those criteria. The method behind them — measure what the work needs, compare it with what the machine has, and check whether memory is even the deciding reason — is what should outlast them.

Hosted services bring a dependency that this article does not resolve. When narration or images come from someone else's model, the quality, the available settings, and the behaviour on unusual input are theirs to change. The per-shot review keeps a person, or an agent acting for one, between the model and the finished video, but it does not make the provider's decisions mine.

The PDF converter and the transcriber still depend on Colab, with every limit that comes with it: session length, daily GPU quota, and rules about how a runtime may be used. The machinery that makes them workable is still needed for them, and still has to be maintained.

Conclusion

A 4 GB GPU sent every heavy job to Colab, and for a while that single rule shaped every project. Asking the question again, one piece at a time, gave five different answers. Rendering came home because it never needed the big GPU. Narration left for reliability, after padding only moved its failures. Images left because the method changed, and took a whole category of Colab concepts with them. The PDF converter and the transcriber stayed, because a number said they had to.

Coming home was done without assuming a GPU, with an encoder chosen by testing rather than by listing, and it was measured mostly in deletions. Where work had to stay remote, it could still be stripped of machinery it did not need.

What carries over is to treat placement as a decision per piece of work, grounded in something that can be measured — and to build the machinery for distance so that it can be taken apart once the distance is gone.

Terms

VRAM

The memory on a GPU. A model has to fit in it, together with its working buffers, to run on that GPU.

runtime

The virtual machine Colab lends for one session, including its GPU. It ends when the session ends.

hosted service

A model run by someone else and reached through an API. The work is sent as a request, and the result comes back.

duration predictor

The part of a text-to-speech model that decides how long each piece of speech should last. Without it, the length has to be estimated from the text.

NVENC

The hardware video encoder on NVIDIA GPUs. FFmpeg can use it through encoders such as h264_nvenc.

libx264

A software H.264 encoder that runs on the CPU. Slower than NVENC, but available everywhere FFmpeg is.

beat

One moment of narration in the slideshow, together with the image shown while it is spoken.

headless

Running without any interface. Work is chosen from the notebook itself, and results are read from where they are written.

completion marker

A file whose existence means a piece of work is finished. It must only be written once the work is complete.