Deploy a GPT: Hugging Face, Gradio, a browser

Intro

0:00 Hello and welcome back to episode five, where we're going to deploy our model: we put it on Hugging Face, host it in a web app on a VPS that serves it, and run it in the browser. Let's begin with a quick look at where we're going to be by the end of the episode.

0:15 In a web page, we ask our fine-tuned model for a story with a dialogue, and the words cat, cake and jump. The page downloaded the model from Hugging Face, and the browser runs it, in this tab, with no server of ours doing the work.

0:29 On the website of the series, you can find more information about the project. The code is open source on GitHub, tagged at each step of each episode, so that you can follow along, and the docs get a page on sharing and serving the model today. The home page has a new section too, where you can ask the model for a story, which runs in your browser, like the page we're going to build at the end of the episode. And each episode has a page here as well, with its chapters and its code.

Last time

0:58 Now, before we get to today's work, let's quickly go over what we did last time. Our model writes a token at a time, and it ran the whole story through its blocks for each new token. As attention is causal, the keys and values of the earlier tokens don't change, so we kept them in a cache, a set per block, and ran the new token alone. The logits, the stories and the evaluation's numbers came out as before, and on my laptop's CPU, the model wrote about four times faster. On the GPU, one story got less of a speed-up, as with a model this small, most of each step goes to launching operations. And eight stories at once got about three point five times faster.

Three places to run it

1:39 So far, our model is a checkpoint, which loads with the code in our repo and with no other code. Today, we're going to deploy it, to the Hugging Face Hub and to a VPS, and run it in the browser. First, we push it to the Hugging Face Hub, where models are shared and versioned, in a format that the transformers library loads without our code, with a version per episode. Then we're going to write a web app for it with Gradio, and run it on my machine, and in Docker on a server, the one from my MMO series. Last, the browser, with Transformers.js, where the model runs on the visitor's machine. All right, let's get started with what our model has in common with GPT-2.

Our model is a GPT-2

2:19 OpenAI's GPT-2 is on the Hub, and curl fetches its config dot json, which jq cuts down to the fields we compare with ours. It has twelve blocks of twelve heads, an embedding size of seven hundred and sixty-eight, a context window of a thousand and twenty-four tokens, and a vocabulary of about fifty thousand. Apart from the sizes, it's the architecture we built in episode one: learned positions, a layer norm before each sublayer, GELU, and a head tied to the token embeddings. Its activation is GELU new, which approximates GELU with tanh, while we use the formula with the error function, and GPT-2's code in transformers has that too, as gelu. So we don't need to publish our own model code, as we can save our weights under GPT-2's names, and anyone with transformers can load them as a GPT-2.

3:06 The embeddings keep their shapes, under the names w t e and w p e, for word and position. GPT-2 computes the queries, keys and values with a single layer, three times as wide, called c attn, so ours are stacked into it, one after the other. And GPT-2 builds its linear layers with a class called Conv1D, which stores its weights as inputs by outputs, where n n linear stores them as outputs by inputs, so we transpose each weight matrix as we convert it. The layer norms map across one to one, and the head comes along with w t e, as both models tie them.

The Hugging Face format

3:44 We write hub dot py, to save a checkpoint in the Hugging Face format. It imports argparse and os, and torch, and from tokenizers, Hugging Face's library for fast tokenizers, it imports the tokenizer class, decoders, pre-tokenizers, and the word piece model, which we're going to set up as a word-level tokenizer like ours. From transformers, the auto classes, which load a model or a tokenizer from a folder or the Hub, and read its class from the config files, GPT-2's config and model with its head, and the fast tokenizer class. Then our checkpoint loader, the data folder, and our tokenizer's tokens: end of text, padding, unknown, and the list of the three. To GPT-2 builds GPT-2's config from ours: the vocabulary, the positions, the embedding size, the blocks and the heads. We ask for gelu, as our GELU is the one with erf, where GPT-2's default approximates it with tanh. The dropouts are zero, as this model is for writing stories, not training, and the beginning and the end of a text are our end-of-text token, with padding as the pad token. Ours is our state dict, the model's tensors by name, which is what a checkpoint holds. GPT-2's is a plain dictionary that we fill in name by name, and we start it with the token and position embeddings, as w t e and w p e, and the final norm, as l n f. Then, for each block, b is our prefix and h is GPT-2's, such as blocks dot zero for our first block. Cat stacks our three weights into one, one under the other, along the outputs, and dot t transposes the result, as GPT-2 projects q, k and v with one layer, three times as wide. The biases are stacked too, and need no transpose, as each is a single row of numbers. The output projection and the two MLP layers are transposed as well, in a loop over pairs of GPT-2's name and ours, as GPT-2's layers store their weights as inputs by outputs, the transpose of n n linear's. The two norms map across as they are, as l n one and l n two. Then we build an empty GPT-2 from the config, and load our tensors into it, with strict false, as there's one tensor we didn't give it. The head is tied to w t e, as ours is to embed tokens, so loading w t e sets the head too. The assert checks that the head is all that's missing, and that none of our names went unused, which would mean a typo in a name. To H F tokenizer writes our vocabulary in the tokenizers library's format. Vocab maps each token to its id, and the tokenizer is a word piece model. Its vocabulary has no pieces for the middle of a word, the ones written with two hash signs in front, so a word is in it whole, or it's unknown, like ours. Word piece's pieces would split an unknown word into smaller parts, but we have none. The pre-tokenizer splits the text into words before the model looks them up, and whitespace, despite its name, splits on runs of word characters or of punctuation, the regex our tokenizer uses. The decoder joins tokens back into text, and we add our three tokens, so that end of text is a single token. Then the fast tokenizer wraps it for transformers, with each token's role. Check loads the folder back with the auto classes, the way anyone else would, and compares it with our model and tokenizer. It opens the held-out instructions from episode three, reads their first two hundred thousand characters, encodes them with our tokenizer, and prints whether the Hugging Face tokenizer gives the token ids we do, and on how many tokens. Then it runs both models on the first window and prints the largest difference in the logits. Main makes an argument parser, with a line on what the script does, which dash dash help prints, along with the arguments it takes. Those are a checkpoint, and a folder to save to. It loads the model and our tokenizer from the checkpoint, on the CPU, as renaming tensors needs no GPU, and puts the model in evaluation mode. Then it converts the model and the tokenizer, and saves both with save pretrained, transformers' way of writing a model to a folder. It writes the weights as safetensors, a format that holds tensors and no code, where torch save writes a pickle, which can run code as it loads. We print what's in the folder, and check it. Transformers joins the requirements, as Hugging Face's library for loading and running models, and git ignores the h f folder, like the checkpoints.

8:08 Let's save our fine-tuned model in the Hugging Face format. Two, greater than, dev null hides the warnings transformers prints on the way, which aren't about anything we need to act on. The folder has the config, the generation settings, the weights as safetensors, and the tokenizer's files. Loaded back, the Hugging Face tokenizer gives our token ids, on about forty-six thousand tokens of held-out instructions, and the logits differ by about one point five times ten to the minus five, which is float rounding, as with the cache last episode.

ONNX, and 8 bits for the browser

8:42 The browser can't run PyTorch, so we're going to export the model to ONNX, a format for models that runtimes in other languages can run. One of them, Microsoft's ONNX Runtime, has a version for the browser. Hub dot py imports NumPy, ONNX Runtime, and its function for quantisation, along with optimum's exporter, from Hugging Face, and PyTorch's functional, for the loss. Check now returns the token ids, as we're going to measure the loss on them. Export ONNX writes to an onnx folder inside the model's folder. Main export traces the model as it runs, and writes the operations it ran, with the weights, to model dot onnx. The task is text generation with past: the model takes the cached keys and values as inputs, and returns them with the new ones, so the browser gets a KV cache too. Optimum also writes the config files into the onnx folder. The originals stay at the top, so we go through the folder and remove the copies of the configs, anything that isn't an ONNX file. Then quantize dynamic writes a second file, model quantized dot onnx, with the weights as 8-bit integers, a quarter of the size, for a browser to download. Each 8-bit weight is a step on a scale between the smallest and the largest weight of its matrix, and the activations are quantised as the model runs, which is the dynamic part. To see what that costs, onnx loss runs a file with ONNX Runtime on a window of tokens. The inputs are the ids, a mask of ones, as there's no padding, and the positions. For each block, the cache's keys and values come in empty, with zero tokens along the third dimension, as there are no tokens before these. Run returns the logits, and cross entropy gives the loss against the next tokens. In main, after the check, we export, take a window of the held-out ids, and print the loss of our PyTorch model on it, then the size and the loss of each ONNX file, so we can see what the 8-bit version costs us in loss. Accelerate, ONNX Runtime and optimum join the requirements, where accelerate is what optimum uses to keep the tied head and the embeddings as a single weight in the file.

10:49 Running it again, the export takes a few seconds. The ONNX model is about fifty-eight point six megabytes, the size of our checkpoint, and its loss matches PyTorch's, at three point three zero five. The 8-bit model is about fifteen megabytes, a quarter of the size, and its loss is three point three zero eight, up by a few thousandths. The folder now has the onnx folder, with the two models in it.

On the Hugging Face Hub

11:13 A model on the Hub comes with a model card, a page on what it is, how to use it, and what it can't do. Model card dot md is ours, as a template. The block at the top is metadata for the Hub: the licence, the language, the library that loads it, the task, the datasets and some tags, so it can be found. Then what mvGPT is, with dollar tag and dollar message for each version, the architecture, how to use it from Python and from the browser, the versions, and the limitations, along with the tokenizer's spacing when it decodes. In hub dot py, we import template, Python's simplest way to fill in dollar names in a text, and HF API, the Hub's client. Push takes the folder, the repo, the tag and the message. It reads the template from next to hub dot py, and fills it in with the repo, the tag and the message. Then it writes it as the folder's README, as the README of a model repo is its model card, the page the Hub shows for it. HF API uses the token from logging in with the h f command. Create repo makes the model repo, and exist ok means it's fine if it's there already. A repo on the Hub is a git repo, so upload folder sends the folder as a commit, with our message, and returns it. Then create tag puts a tag on that commit, with the message, and the comment over it says why: a tag names this commit, so revision equal to the tag loads it, whatever is pushed after it. Main takes dash dash push with a repo, a tag and a message, and if there's a repo, it pushes the folder once the checks are done. Without dash dash push, the script doesn't upload anything, so we can convert and check a model as often as we like.

12:52 I've logged in with h f auth login, with a token from my account's settings, and h f auth who am I says who I am. First, the pretrained model from episode two, to the repo nenuadrian slash mvGPT, tagged e p zero zero two pretrain, with a message saying what it is. It goes through the checks and the export, then uploads the weights, the tokenizer, the two ONNX files and the card, and prints the commit's address and the tag. Then the fine-tuned model from episode three, to that repo, tagged e p zero zero three s f t. The upload is a second commit on main, which replaces the files of the first, while the first tag still points at the pretrained model.

13:33 The model's page on the Hub is its model card, with the tags from its metadata at the top, and it shows the latest commit, the fine-tuned model. The code to use it is three lines of transformers, or a few of JavaScript. Further down are the versions, with their tags, and the limitations. Files and versions shows what we pushed: the config, the weights, the tokenizer, and the onnx folder, with each file's last commit. The history has our two commits, after the one the Hub made when it created the repo. And the branch menu lists main, and our two tags, so anyone can load the model from either episode, by its tag.

A web app with Gradio

14:13 Next, a web app, so that anyone can try the model without writing code. We're going to use Gradio, a Python library from Hugging Face for interfaces to models: we lay out inputs and outputs in Python, and it serves them as a web page, and calls our function when someone clicks. Streamlit is another option, which is more suited to dashboards, while Gradio is built around a model's inputs and outputs, with text that streams in as it's written. App dot py imports os, Gradio as g r, torch, transformers' auto classes, and detokenize from our tokenizer. The repo comes from an environment variable, with ours as the default. Models maps the labels the app shows to the tags on the Hub they load, and features lists the six features TinyStories-Instruct asks for. Load takes a revision, a tag on the Hub, and downloads that version of the tokenizer and the model. Each version is downloaded the first time it's asked for, and kept in memory after that, in loaded. Instruction builds the prompt from the form, with a line for the features, if any are checked, then the words and the summary, if they're filled in. It puts the fields in the order sft dot py gives them, ending with story and a colon, where the story starts. Write is what runs when someone clicks, and it loads the version, and encodes the instruction. Then we write the story a token at a time, with transformers' GPT-2, up to two hundred tokens, or fewer if the instruction is long, as the model has two hundred and fifty-six positions. Transformers' GPT-2 has a KV cache like ours: the cache goes in as past key values, and comes back with x's keys and values added. The first step runs the whole instruction, and the rest, the new token alone. Then the temperature and top k, as in our generate, and multinomial samples the next token. When the model samples end of text, the story ends there. Yielding the story so far makes Gradio show it a token at a time, and detokenize joins our tokens with our spacing, where the Hugging Face tokenizer's decoder would put spaces around the punctuation. The theme starts from base, Gradio's plainest theme, with indigo and slate as its colours, and it uses Space Grotesk, the series' font, with JetBrains Mono for code. A Gradio theme has a value for each setting in light mode, and another for dark mode, whose name ends in dark. So we copy each dark value over its light one, and the comment above it says what that gives us: an app that's dark, whatever the visitor's system prefers, as each colour takes its dark-mode value. Then come the series' near-black slate, for the page, the blocks and the inputs, and its periwinkle for the button, with dark text on it, set in both modes. Blocks lays the page out, where each with statement opens a part of the page, and what's inside it goes in that part. There's a title in markdown, with a link to the series, then a row with two columns. The form is on the left: a radio button for the model, checkboxes for the features, text boxes for the words and the summary, with an example in each, and a row with two sliders, for the temperature and top k, with the defaults we've used so far. Then the button, and on the right, a text box for the story. Clicking the button calls write with the form's values, and streams the story into the story box. When we run the file, launch serves the app on port seventy-eight sixty, Gradio's usual one, with our theme. In Docker, the app has to listen on all interfaces for a published port to reach it, so the host comes from an environment variable, with localhost as the default. And Gradio joins the requirements, next to transformers and torch.

17:51 We run it with python app dot py, and open it on port seventy-eight sixty. We ask the fine-tuned model for a dialogue, with the words dog, ball and happy, and the story streams in a token at a time, with the words we asked for. The first click downloaded the model from the Hub, and the next ones use the copy in memory. Picking the pretrained model loads the version tagged for episode two, which carries on from the instruction without starting a story, as it did in episode three, before fine-tuning.

On a server, with Docker

18:21 To run the app on a server, we put it in a Docker image, so it runs the way it does on my machine, with its Python and its libraries. The Dockerfile starts from Python three point twelve, in its slim image, and works in slash app. Then we install PyTorch with pip, from PyTorch's own index, and it's PyTorch's build for the CPU, as the server has no GPU, and the CUDA build is gigabytes bigger. Then the app's requirements, the app and the tokenizer, and the host set to all interfaces. Expose notes the port the app listens on, and the command runs the app. The app's requirements are Gradio and transformers, pinned to the versions we ran it with. The docker ignore file lets the build see the app and what it imports, and no other file, so a build on my machine doesn't send Docker the data and the checkpoints, which come to gigabytes. Deploy dot sh, like the script in my MMO series, takes the server, as a user at an address. Set dash e u o pipefail stops the script at the first command that fails. Git archive packs the committed code, and the pipe sends it over SSH, where the old copy is removed and tar unpacks the new one. Above it, we sum that up: the committed files of HEAD, as a tar stream, unpacked into mvgpt in the home folder on the server. Removing the old folder first means a file we've deleted from the repo doesn't stay behind on the server. And as git archive packs what's committed, an uncommitted change on my machine doesn't reach the server either. Then, over SSH, we build the image there, and swap the running container for one from it. Docker rm dash f removes the old container, if there is one, and docker run starts the new one in the background, named mvgpt. Restart unless stopped starts it again when the server reboots, dash p publishes the port, and the volume keeps the models downloaded from the Hub across deploys, so a new deploy doesn't download them again.

20:17 For the server, we're going to use the VPS I rented from DreamHost for my MMO series, with two virtual CPUs and four gigabytes of memory, which our game's server shares with this app. It came with Docker installed, so our image runs there as it does on my machine.

20:33 Let's deploy it with the script, to the VPS's address. The script sends the files and builds the image on the server. The first build took a couple of minutes, mostly to download PyTorch, and Docker keeps each step of a build, so a deploy with the requirements unchanged takes a few seconds, like this one. Docker build dash q prints the image's id alone, and docker run prints the new container's. Docker p s on the server shows the container up, with port seventy-eight sixty published on all its addresses. Docker publishes ports past ufw, the server's firewall, which we saw in the MMO series, so the app is public as soon as it runs.

21:11 On the server, the app is at the VPS's address, on port seventy-eight sixty. We ask for a story with a moral value, and the words rabbit, carrot and share, and the server downloads the model from the Hub on the first story, and then writes it a token at a time on its two CPUs.

In the browser, with Transformers.js

21:30 The Transformer Explainer, from Polo Chau's group at Georgia Tech, runs GPT-2 small in a browser tab. Its model is exported from nanoGPT's PyTorch code to ONNX, and runs with ONNX Runtime, which is how we're going to run ours. It draws each step of a pass, from the embeddings, through the queries, keys and values of each head, to the probabilities of the next token. Clicking generate writes the next word, and draws the pass that wrote it, through the steps we built in episode one.

21:59 Last, the browser, with Transformers.js, Hugging Face's library for running models in JavaScript. Underneath, ONNX Runtime runs the model with WebAssembly, or on the GPU with WebGPU. We write a single page, index dot html, with its style in style dot css, in the series' colours. The page starts like any page, with its character set, and a viewport line, so it fits a phone's screen. It links Space Grotesk and JetBrains Mono from Google Fonts, and our style. The body has a title, and a line on what the page is: a GPT that runs in your browser, which downloads once, about fifteen megabytes, when you first ask for a story. Then the form, with a fieldset for the features, which the script fills in, inputs for the words and the summary, with an example in each, a button, which sends the form, and a status line, and last, a paragraph for the story. The script is a module, so it can import other modules, and it imports pipeline and text streamer from Transformers.js, from the jsDelivr CDN, pinned to a version, so there's no package to install or build. Model is our repo on the Hub, and features are the six features. We keep the form, the status line and the story, found by their ids, and for each feature, we add a checkbox with a label to the fieldset. Detokenize joins the tokens with spaces, and then three regular expressions put each punctuation mark against the word before it, close up the apostrophes, as in don't, which our tokenizer splits into three tokens, and put an opening quote against the word after it. Instruction builds the prompt from the form, the way the app does. It collects the features whose boxes are checked, and adds a line for them, then a line for the words, trimmed of the spaces around them, and one for the summary, when they aren't empty. The fields go in the order sft dot py gives them, ending where the story starts. Writer starts as null, and it's made on the first story, so opening the page downloads nothing. Load makes it with pipeline, which is how Transformers.js gets a model and its tokenizer ready for a task, here text generation, from our repo, at the fine-tuned model's tag. Dtype q8 picks the 8-bit ONNX file, model quantized, and the progress callback shows how much of it has downloaded, as a percentage on the status line, for the ONNX file, which is most of the download. Write loads the model if it's the first story, and keeps the story's tokens. The text streamer gets each token as it's written, skipping the prompt. Its callback for the whole text is empty, as we handle each token in the token callback. Each new token is decoded on its own, and the story so far is joined with detokenize. The model has two hundred and fifty-six positions, for the instruction and the story together, so we write two hundred tokens, or what's left. The writer writes with the sampling settings we use elsewhere, and at the end, the status line shows how many tokens it wrote, and how long it took. Prevent default stops the browser from loading a new page, as it does when a form is sent, and submitting the form writes a story, with the button disabled while it does, and an error, such as a failed download, goes on the status line. The page needs no build step, so the web folder can be served as it is, by any web server.

25:09 Any web server can send the page, such as python dash m http dot server, as the server sends the files and the browser runs the model. We ask for a dialogue, with the words dog, ball and happy, and the page downloads the 8-bit model from the Hub, about fifteen megabytes, which the browser keeps for the next visit, and writes the story in the tab, in well under a second, with WebAssembly on my laptop's CPU.

What a production setup adds

25:34 Our app runs on a server, but a production deployment would add a few things. It would have a domain and HTTPS, as our app is served over plain HTTP, on an address and a port. It would have accounts or rate limits, as anyone can ask ours for stories, and Gradio's queue writes one story at a time. A large model would need GPUs, and a server such as vLLM, which batches many conversations into each step, and manages the KV cache we saw last episode. And it would be monitored, as we did with Grafana in my MMO series. For a model of about fifteen million parameters, two CPUs, or the visitor's browser, are plenty.

26:15 The docs get a page on sharing and serving the model, with the names GPT-2 uses for our weights, the tokenizer, the ONNX files and their losses, pushing to the Hub with a tag, the app, the deploy, and the page for the browser, and their home page links to it. And the README has the commands to save the model and run the app.

Wrap-up

26:34 So, today we deployed our model, to the Hugging Face Hub and to a VPS that hosts it and serves it. It has GPT-2's architecture, so we saved it under GPT-2's names, with its vocabulary as a tokenizer dot json, and transformers loads it with our token ids and our logits. We exported it to ONNX, and to 8 bits, a quarter of the size, and pushed the pretrained and the fine-tuned models to the Hugging Face Hub, each with a tag. A Gradio app loads either version from the Hub, on my machine and in Docker on a server, deployed over SSH. And a single page runs the model in the browser, with Transformers.js. A later episode that changes the model can push a new tag, and the app and the page can load it.

27:19 Let me know in the comments where you'd like to run a model like ours, and which parts of deploying it you'd like us to go deeper on. And if you'd like to follow along, subscribe to the channel. I hope you find this useful, and thanks for watching.