Thursday, November 09, 2023
Jam Copilot
Tim Ng
@tymothy6
Introduction
This post is an overview of my thought process when designing a widget to use ChatGPT in FigJam. Inspired by Google Visual Blocks and eventually Figma's own JamBot, I aimed to build a visual documentation tool for ChatGPT that leveraged whiteboards. As a primer, you should know that I'm a big fan of Notion as a second brain and task tracker (Kanban boards and databases are kind of my jam ๐), but a feature that I've always wanted is to organize and draw connections between my notes. I think this is an interface that complements large language models particularly well. When working with sequential prompts (think in the style of LangChain), lengthy prompt-response chains can be especially difficult to navigate.
I'm sure if you've used ChatGPT for any involved task, you'll know how time-consuming it can be to scroll through a long conversation without being able to intuitively reference the most important responses. With this in mind, one of my key requirements for this project was to strike a balance between flexible usage and structured queries like article summaries, idea generation, translation, or coding tasks.
My solution is a simple wrapper that takes free-form input in FigJam, sends it to the OpenAI API for processing, and returns responses to the user. The process is repeated as many times as deemed necessary. All the inputs and responses are kept together on a single pane of glass (or canvas, to be pedantic). Plug in this simple user flow into the multiplayer aspect of Figma and you can envision collaborative ChatGPT usage and discussion. Pretty cool, huh? Widgets are an ideal solution since users interact with them on the canvas like any other native object, and the entry point for any user flow always starts on the widget itself. Widgets are inherently collaborative since every user in the same file sees the same instance. Finally, there's no limit to the number of widget instances in a file, meaning that prompts and responses can be chained together indefinitely. Perfect!
You can access the full codebase for this project on GitHub.
Getting started
I had used Figma Plugins and experimented with some widgets in the past, but I wasn't familiar with the well-documented Widget API that uses TypeScript and JSX. If you've written React code, you will immediately notice the similarity. There are some constraints to keep in mind and I will discuss each in more detail when we come across them. On the UI side, you should know that there are a limited set of React-like hooks and front-end components, so don't expect complete freedom when it comes to styling. Any custom shapes or icons will need to be defined as SVG elements and you won't be able to import custom wrappers. To interact with and create native elements in a Figma file, we will need to use the Plugin API. Every Figma plugin or widget needs to define a manifest.json file that describes the plugin. This file is automatically created for you when you bootstrap your widget development in Figma. However, we need to specify that the widget is designed for FigJam by modifying the editorType key. This will be important to access the specific methods in the Plugin API we will be using.
Great, now we know the tools we need to use to build our ChatGPT widget. Let's get started with the process.
Nodes in Figma
At the start of this project, the first prototype I wanted to build was a simple interaction with my widget reading input stored in a sticky note and returning a response. You might imagine that the barebones flow goes something like this:
User opens the widget from the drawer.
User creates a new sticky and writes a prompt.
User initiates the call for a response from ChatGPT using an interaction handler.
Widget prints the response in a new sticky. This is important for users to keep track of their usage on the canvas without having to interact with the widget. The sticky can be edited and used for subsequent prompts.
What are the requirements for this user flow? First, we need a way to fetch the contents (text) of a sticky based on whether or not its selected. We then need to send the text in a request body to the OpenAI API and obtain a response (completion). Finally, we need to extract the completion and a method to generate a new sticky. On the UI side, we need an interactive element such as a button to initiate the OpenAI API call. We should also provide feedback to the user when prompts are ready to be sent. To do this, we can display the pending API call in a textbox. You might also consider styling the API call button differently depending on whether a prompt has been detected.
Basic overview of the UI for a ChatGPT widget in FigJam
Reading sticky notes
As I alluded to earlier, the Figma Plugin API provides all the methods that we need to retrieve sticky text, send it to our backend to handle the OpenAI API call, and print the completion. We can access almost any part of a file through the figma object and use functions to view, create, and update its contents. To detect when a sticky has been selected by users, we can use the on callback and the selectionchange event that listens for (you guessed it) selection changes. By checking the type property of the nodes in a file, we can specify that we are listening for sticky note selections (STICKY) only. This logic applies to multiple stickies as well. We can use a forEach loop to process each selected sticky as needed. Because we need to manage side effects like interacting with the Plugin API, registering, and cleaning up the event listener, we will use the useEffect hook from the Widget API (identical to React's useEffect hook).
Okay, this logic satisfies the requirements to process one or more prompts, at least on the surface. But how can we make this process even more intuitive? Let's allow users to prime the API call by linking stickies with the widget using FigJam connectors. Connectors are used to connect FigJam components to indicate relationships and possess directionality by default. That means that they are the ideal element to link prompt-response chains!
The code in the useEffect hook now monitors connectors that are created between stickies and the widget, using the widgetId value obtained from useWidgetNodeId to identify the instance. When a connector is created, we process the sticky note by extracting the text, updating a list of processed stickies, setting a pending API call state, and saving the fill colour so that our response sticky is colour-matched.
useSyncedState
processedStickies is an array that keeps track of the unique sticky IDs that have been processed. accumulatedStickyTexts is an array that stores the text contents of each processed sticky. stickyIdToIndexMap is a mapping from sticky ID to their indices in the accumulatedStickyTexts array. This lets us update the text of an existing sticky in the array without searching through the array each time. Lastly, pendingApiCall holds the aggregated text that has been concatenated into a single string. In the processSticky function, the ID of a sticky is first checked to see if it exists in stickyIdToIndexMap. If it does, the corresponding text in accumulatedStickyTexts is updated.
Fetching the OpenAI response
Now we're ready to implement the star of the show; fetching the actual ChatGPT response! The handleJamClick function triggers an API call using the content from pendingApiCall. We set up some default parameters for the systemPrompt, userMessage, and the number of completions (choices). If you aren't familiar with the ChatGPT API, I suggest you review the excellent documentation here. Of course, all of these parameters can be adjusted as per specific user requirements. For example, you might want to instruct ChatGPT to work with multiple prompts and return more than one response. In the next section, we'll discuss how you might use the choices array to create multiple stickies, each with their own completion.
You should know that I'm using a Vercel serverless function to handle the call on the backend. This is because Figma widgets don't natively support environment variables to store secrets like API keys. The function is a simple proxy for making POST requests to the OpenAI chat completions endpoint. The request includes headers for content type, authorization, and the JSON stringified version of the request body sent by the widget. This lets the widget interact with the OpenAI API without handling API keys or managing CORS issues. Make sure you update the allowedDomains key of the widget's manifest.json file if you choose to use a similar approach to handle data fetching.
Displaying the ChatGPT response to the user
You might have noticed that I defined a handleApiResponse function in the click handler. This is where we will handle the logic to display the response in a new sticky. To better indicate the interaction flow, we'll also use the Plugin API to create a new connector that complements the original one drawn from the prompt to the widget. We'll check that the completion exists before attempting to extract the chat completion (message) from the choices array of the data object. We then call createSticky with the completion, which creates a single sticky note and connector. There's also logic to handle a response that contains code snippets, creating a formatted code block component in FigJam using the createCodeBlock method. Lastly, you'll notice that I've included a condition to check if a selected function "Ideate" is true. This function invokes an alternative method createMultipleStickies with the entire data object as an argument to create a sticky for each response when multiple completions are fetched. I won't delve into the details of that logic but I invite you to check out the code base for this project if I've piqued your interest.
Let's go over the createSticky logic in more detail. Given the chat completion as a string, the function first calls figma.createSticky() to create a new sticky. The stickyFill style is applied to the newly created sticky to ensure that it matches the prompt. The function loads the default FigJam font (Inter Medium) using figma.loadFontAsync(defaultFont) and sets this font for the sticky text. Keep in mind that this is a prerequisite to render the completion text in the sticky correctly. The contents of the sticky are set using the passed content string. To position the sticky relative to the widget, we use the widget instance (widgetId) as a reference. Lastly, we call figma.createConnector() to link the new sticky to the widget.
How do we parse ChatGPT responses and render native code components in FigJam? This behaviour is a little bit more involved than our previous handler. We need to pass information about the programming language for syntax highlighting to be formatted properly. First, we use a regular expression to extract the language and the code from the context of the chat completion. This expression looks for text enclosed in triple backtick code block syntax and isolates the language and code respectively. Next, we need to load a code block font to satisfy the Plugin API. We call figma.createCodeBlock() to create a new FigJam code block component. We define isCodeLanguage as a function to check whether the extracted language is compatible with syntax highlighting in FigJam. Finally, we position the code block relative to the widget instance and create a connector to link the block with the widget.
Fine-tuning the widget
We've discussed the basic building blocks to satisfy basic requirements for ChatGPT usage using native FigJam components (stickies, connectors, code blocks). Obviously, there are many directions that we could focus on in future iterations. One feature of Figma's JamBot that I enjoy using are the structured queries where ChatGPT is trained to act a specific way. For example, the "Rabbit hole" option provides six distinct but related ideas to your prompt. You might define logic for different widget functions using cases to modify the system prompt in the API fetch function. In my codebase, selectedFunction is a handler that parses the user selection from the UI and declares the relevant case. Please keep in mind that different language models (ChatGPT vs. GPT-4) will respond quite differently to these prompts and a fair amount of fine-tuning is necessary to achieve optimal results.
Example of how you might handle additional inputs to modify the widget API call
You might also customize user prompts by providing additional inputs, for example by including a text area in the widget UI. In the example below, additionalInput is a state variable mapped to an <Input> component from the Widget API. In my code, additionalInput is appended to the pendingApiCall state before the OpenAI API is called.
Final words
Building out this ChatGPT widget for FigJam was a great experience as a novice developer. Having to consider different design decisions on the front-end, all while working within the specific constraints of widget development was challenging but rewarding. I enjoyed learning about the detailed codebase in the Plugin API and can see myself working on more complex plugins and widgets in the future. If you're thinking about developing a specific Figma plugin or widget, I strongly urge you to get started today ๐. You only know what you build yourself!
Liked my writing?
Buy me a coffee! โ๏ธRecommended posts
July 22, 2025
TNHow do policy and design intersect?

Exploring how design in government shifts between delivery, adaptation, and transformation, using a typology of design-policy relationships to make sense of transitions
July 20, 2025
TNOn 1 year of public service

A year-in-review on designing in government, exploring the tensions, responsibilities, and quiet growth that comes with practicing care and rigour