TN
Use ChatGPT visually in FigJam using stickies and connectors

Thursday, November 09, 2023

8 min read

Jam Copilot

TN

Tim Ng

@tymothy6

FigJam
Figma Plugin API
Figma Widget API
TypeScript
ChatGPT
Generative AI

Introduction

This post is an overview of my thought process when designing a widget to use ChatGPT in FigJam. Inspired by Google Visual Blocks and eventually Figma's own JamBot, I aimed to build a visual documentation tool for ChatGPT that leveraged whiteboards. As a primer, you should know that I'm a big fan of Notion as a second brain and task tracker (Kanban boards and databases are kind of my jam ๐Ÿ˜), but a feature that I've always wanted is to organize and draw connections between my notes. I think this is an interface that complements large language models particularly well. When working with sequential prompts (think in the style of LangChain), lengthy prompt-response chains can be especially difficult to navigate.

I'm sure if you've used ChatGPT for any involved task, you'll know how time-consuming it can be to scroll through a long conversation without being able to intuitively reference the most important responses. With this in mind, one of my key requirements for this project was to strike a balance between flexible usage and structured queries like article summaries, idea generation, translation, or coding tasks.

My solution is a simple wrapper that takes free-form input in FigJam, sends it to the OpenAI API for processing, and returns responses to the user. The process is repeated as many times as deemed necessary. All the inputs and responses are kept together on a single pane of glass (or canvas, to be pedantic). Plug in this simple user flow into the multiplayer aspect of Figma and you can envision collaborative ChatGPT usage and discussion. Pretty cool, huh? Widgets are an ideal solution since users interact with them on the canvas like any other native object, and the entry point for any user flow always starts on the widget itself. Widgets are inherently collaborative since every user in the same file sees the same instance. Finally, there's no limit to the number of widget instances in a file, meaning that prompts and responses can be chained together indefinitely. Perfect!

You can access the full codebase for this project on GitHub.

Getting started

I had used Figma Plugins and experimented with some widgets in the past, but I wasn't familiar with the well-documented Widget API that uses TypeScript and JSX. If you've written React code, you will immediately notice the similarity. There are some constraints to keep in mind and I will discuss each in more detail when we come across them. On the UI side, you should know that there are a limited set of React-like hooks and front-end components, so don't expect complete freedom when it comes to styling. Any custom shapes or icons will need to be defined as SVG elements and you won't be able to import custom wrappers. To interact with and create native elements in a Figma file, we will need to use the Plugin API. Every Figma plugin or widget needs to define a manifest.json file that describes the plugin. This file is automatically created for you when you bootstrap your widget development in Figma. However, we need to specify that the widget is designed for FigJam by modifying the editorType key. This will be important to access the specific methods in the Plugin API we will be using.

manifest
json
1{
2  "name": "Copilot",
3  "id": "1283872076476557239",
4  "api": "1.0.0",
5  "main": "dist/code.js",
6  "capabilities": [],
7  "enableProposedApi": false,
8  "editorType": [
9    "figjam"
10  ],
11  "containsWidget": true,
12  "widgetApi": "1.0.0",
13  "ui": "ui.html",
14  "networkAccess": {
15    "allowedDomains": [
16      "https://vercel-tymothy6.vercel.app/api/openai"
17    ]
18  }
19}

Great, now we know the tools we need to use to build our ChatGPT widget. Let's get started with the process.

At the start of this project, the first prototype I wanted to build was a simple interaction with my widget reading input stored in a sticky note and returning a response. You might imagine that the barebones flow goes something like this:

  1. User opens the widget from the drawer.

  2. User creates a new sticky and writes a prompt.

  3. User initiates the call for a response from ChatGPT using an interaction handler.

  4. Widget prints the response in a new sticky. This is important for users to keep track of their usage on the canvas without having to interact with the widget. The sticky can be edited and used for subsequent prompts.

What are the requirements for this user flow? First, we need a way to fetch the contents (text) of a sticky based on whether or not its selected. We then need to send the text in a request body to the OpenAI API and obtain a response (completion). Finally, we need to extract the completion and a method to generate a new sticky. On the UI side, we need an interactive element such as a button to initiate the OpenAI API call. We should also provide feedback to the user when prompts are ready to be sent. To do this, we can display the pending API call in a textbox. You might also consider styling the API call button differently depending on whether a prompt has been detected.

Basic overview of the UI for a ChatGPT widget in FigJam

Reading sticky notes

As I alluded to earlier, the Figma Plugin API provides all the methods that we need to retrieve sticky text, send it to our backend to handle the OpenAI API call, and print the completion. We can access almost any part of a file through the figma object and use functions to view, create, and update its contents. To detect when a sticky has been selected by users, we can use the on callback and the selectionchange event that listens for (you guessed it) selection changes. By checking the type property of the nodes in a file, we can specify that we are listening for sticky note selections (STICKY) only. This logic applies to multiple stickies as well. We can use a forEach loop to process each selected sticky as needed. Because we need to manage side effects like interacting with the Plugin API, registering, and cleaning up the event listener, we will use the useEffect hook from the Widget API (identical to React's useEffect hook).

handleStickySelection
jsx
1const { widget } = figma
2const { useEffect } = widget
3
4useEffect(() => {
5  const handleStickySelection = () => {
6    // Get the current selection
7    const currentSelection = figma.currentPage.selection;
8
9    // Filter the selection to include only sticky notes
10    const selectedStickies = currentSelection.filter(node => node.type === "STICKY");
11
12    // Process the selected sticky notes
13    selectedStickies.forEach(sticky => {
14      // Extract the text and make our API call
15    });
16  };
17
18  // Add the selection change listener
19  figma.on('selectionchange', handleStickySelection);
20
21  // Remove the event listener on cleanup
22  return () => {
23    figma.off('selectionchange', handleStickySelection);
24  };
25}, []);
26

Okay, this logic satisfies the requirements to process one or more prompts, at least on the surface. But how can we make this process even more intuitive? Let's allow users to prime the API call by linking stickies with the widget using FigJam connectors. Connectors are used to connect FigJam components to indicate relationships and possess directionality by default. That means that they are the ideal element to link prompt-response chains!

documentChangeListener
jsx
1const widgetId = useWidgetNodeId();
2
3useEffect(() => {
4
5  const documentChangeListener = (event: any) => {
6      // Check that an instance of the widget exists
7      if (!widgetId) {
8        console.error("Widget ID not found");
9        return;
10      }
11
12      for (const change of event.documentChanges) {
13        console.log('Document change:', change);
14
15       // Check for newly created connectors
16        if (change.type === "CREATE" && change.node.type === "CONNECTOR") {
17          console.log('New connector created:', change.node); // check that the connector is created
18
19          const connector = change.node as ConnectorNode;
20
21          let startNode: StickyNode | undefined;
22          let endNode: WidgetNode | undefined;
23
24          if ('endpointNodeId' in connector.connectorStart) {
25            const node = figma.getNodeById(connector.connectorStart.endpointNodeId);
26            if (node && node.type === "STICKY") {
27              startNode = node as StickyNode;
28            }
29          }
30
31          if ('endpointNodeId' in connector.connectorEnd) {
32            const node = figma.getNodeById(connector.connectorEnd.endpointNodeId);
33            if (node && node.id === widgetId) {
34              endNode = node as WidgetNode;
35            }
36          }
37
38          if (!startNode || !endNode) continue;
39
40          const isStickyConnectedToWidget = startNode !== undefined && endNode !== undefined && startNode.type === "STICKY" && endNode.id === widgetId;
41
42          if (isStickyConnectedToWidget) {
43            const defStartNode = startNode as StickyNode;
44            // First check if the sticky has been processed before ..
45            if (!processedStickies.includes(defStartNode.id)) {
46              processSticky(defStartNode);
47              setProcessedStickies(prevStickies => [...prevStickies, defStartNode.id]);
48
49            if (Array.isArray(defStartNode.fills)) {
50              setStickyFill(defStartNode.fills); // update the sticky fill
51            } else {
52              setStickyFill(null);
53            }
54          }
55        }
56      }
57
58    figma.on('documentchange', documentChangeListener);
59    }));
60
61    return () => {
62      figma.off('documentchange', documentChangeListener);
63    };
64  }, [])

The code in the useEffect hook now monitors connectors that are created between stickies and the widget, using the widgetId value obtained from useWidgetNodeId to identify the instance. When a connector is created, we process the sticky note by extracting the text, updating a list of processed stickies, setting a pending API call state, and saving the fill colour so that our response sticky is colour-matched.

processedStickies is an array that keeps track of the unique sticky IDs that have been processed. accumulatedStickyTexts is an array that stores the text contents of each processed sticky. stickyIdToIndexMap is a mapping from sticky ID to their indices in the accumulatedStickyTexts array. This lets us update the text of an existing sticky in the array without searching through the array each time. Lastly, pendingApiCall holds the aggregated text that has been concatenated into a single string. In the processSticky function, the ID of a sticky is first checked to see if it exists in stickyIdToIndexMap. If it does, the corresponding text in accumulatedStickyTexts is updated.

processSticky
jsx
1const { useSyncedState } = widget;
2
3// useSyncedState expects a key and default value
4const [stickyIdToIndexMap, setStickyIdToIndexMap] = useSyncedState<Record<string, number>>("stickyIdToIndexMap", {});
5const [processedStickies, setProcessedStickies] = useSyncedState<string[]>("processedStickies", []);
6const [pendingApiCall, setPendingApiCall] = useSyncedState<string | null>("pendingApiCall", null); 
7const [accumulatedStickyTexts, setAccumulatedStickyTexts] = useSyncedState<string[]>("accumulatedStickyTexts", []);
8const [stickyFill, setStickyFill] = useSyncedState<ReadonlyArray<Paint> | null>("stickyFill", null)
9
10// ... In the useEffect hook
11
12const processSticky = (sticky: StickyNode) => {
13      const newStickyText = sticky.text.characters;
14      let newAccumulatedTexts;
15
16      if (stickyIdToIndexMap[sticky.id] !== undefined) {
17        // update existing sticky text
18        newAccumulatedTexts = [...accumulatedStickyTexts];
19        newAccumulatedTexts[stickyIdToIndexMap[sticky.id]] = newStickyText;
20      } else {
21        // handle new stickies
22        newAccumulatedTexts = [...accumulatedStickyTexts, newStickyText];
23        setStickyIdToIndexMap(prevMap => ({
24          ...prevMap,
25          [sticky.id]: newAccumulatedTexts.length - 1
26        }));
27      }
28
29      setAccumulatedStickyTexts(newAccumulatedTexts);
30      const aggregatedText = newAccumulatedTexts.join('\n');
31      console.log('Aggregated sticky text:', aggregatedText);
32      setPendingApiCall(aggregatedText);
33      console.log('Pending API call:', pendingApiCall);
34    };
35

Fetching the OpenAI response

Now we're ready to implement the star of the show; fetching the actual ChatGPT response! The handleJamClick function triggers an API call using the content from pendingApiCall. We set up some default parameters for the systemPrompt, userMessage, and the number of completions (choices). If you aren't familiar with the ChatGPT API, I suggest you review the excellent documentation here. Of course, all of these parameters can be adjusted as per specific user requirements. For example, you might want to instruct ChatGPT to work with multiple prompts and return more than one response. In the next section, we'll discuss how you might use the choices array to create multiple stickies, each with their own completion.

handleJamClick
jsx
1const handleJamClick = async () => { 
2    console.log("handleJamClick triggered"); 
3
4    if(pendingApiCall) {
5      console.log("Making API call with:", pendingApiCall);
6
7      let systemPrompt = "You are a helpful assistant."; 
8      let userMessage = pendingApiCall; 
9      let choices = 1;
10
11      try {
12        const response = await fetch('https://vercel-tymothy6.vercel.app/api/openai', {
13          method: 'POST',
14          headers: {
15            'Content-Type': 'application/json',
16          },
17          body: JSON.stringify({
18            messages: [
19              { role: "system", content: systemPrompt },
20              { role: "user", content: userMessage }
21            ],
22            model: "gpt-3.5-turbo",
23            n: choices // the number of responses to return in the message.choices array
24          })
25        });
26
27        if (response.ok) {
28          const data = await response.json();
29          console.log('Received API response:', data);
30          handleApiResponse(data); // logic to handle the response
31          setAccumulatedStickyTexts([]); // reset the input array
32          setStickyFill(null); // clear the stored sticky colour
33        } else {
34          console.error("Error making API call:", await response.text());
35        }
36      } catch (error) {
37        console.error("Error making API call:", (error as Error).message);
38      }
39    }
40  }

You should know that I'm using a Vercel serverless function to handle the call on the backend. This is because Figma widgets don't natively support environment variables to store secrets like API keys. The function is a simple proxy for making POST requests to the OpenAI chat completions endpoint. The request includes headers for content type, authorization, and the JSON stringified version of the request body sent by the widget. This lets the widget interact with the OpenAI API without handling API keys or managing CORS issues. Make sure you update the allowedDomains key of the widget's manifest.json file if you choose to use a similar approach to handle data fetching.

vercel
javascript
1module.exports = async (req, res) => {
2  const fetch = (await import('node-fetch')).default;
3  const body = req.body;
4
5  res.setHeader('Access-Control-Allow-Origin', '*'); // this allows any domain to access this route
6  res.setHeader('Access-Control-Allow-Methods', 'GET, POST'); 
7  res.setHeader('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept, Authorization'); 
8
9  try {
10    const response = await fetch('https://api.openai.com/v1/chat/completions', {
11      method: 'POST',
12      headers: {
13        "Content-Type": "application/json",
14        "Authorization": `Bearer ${process.env.OPENAI_API_KEY}`
15      },
16      body: JSON.stringify(body)
17    });
18
19    const data = await response.json();
20    console.log('Outgoing response from OpenAI:', {
21      status: response.status,
22      statusText: response.statusText,
23      body: data
24    });
25    res.status(200).json(data);
26  } catch (error) {
27    res.status(500).json({ error: 'Failed fetching data from OpenAI' });
28  }
29};

Displaying the ChatGPT response to the user

You might have noticed that I defined a handleApiResponse function in the click handler. This is where we will handle the logic to display the response in a new sticky. To better indicate the interaction flow, we'll also use the Plugin API to create a new connector that complements the original one drawn from the prompt to the widget. We'll check that the completion exists before attempting to extract the chat completion (message) from the choices array of the data object. We then call createSticky with the completion, which creates a single sticky note and connector. There's also logic to handle a response that contains code snippets, creating a formatted code block component in FigJam using the createCodeBlock method. Lastly, you'll notice that I've included a condition to check if a selected function "Ideate" is true. This function invokes an alternative method createMultipleStickies with the entire data object as an argument to create a sticky for each response when multiple completions are fetched. I won't delve into the details of that logic but I invite you to check out the code base for this project if I've piqued your interest.

handleApiResponse
jsx
1function handleApiResponse(data: any) {
2    if (selectedFunction === "Ideate") {
3      createMultipleStickies(data);
4    } else {
5      if (data && data.choices && data.choices.length > 0 && data.choices[0].message) {
6        const completionText = data.choices[0].message.content.trim();
7
8        if (selectedFunction === "Code") {
9          createCodeBlock(completionText);
10        } else {
11        createSticky(completionText);
12        }
13      } else {
14        console.error("Error handling API response:", data);
15      }
16    }
17  }

Let's go over the createSticky logic in more detail. Given the chat completion as a string, the function first calls figma.createSticky() to create a new sticky. The stickyFill style is applied to the newly created sticky to ensure that it matches the prompt. The function loads the default FigJam font (Inter Medium) using figma.loadFontAsync(defaultFont) and sets this font for the sticky text. Keep in mind that this is a prerequisite to render the completion text in the sticky correctly. The contents of the sticky are set using the passed content string. To position the sticky relative to the widget, we use the widget instance (widgetId) as a reference. Lastly, we call figma.createConnector() to link the new sticky to the widget.

createSticky
jsx
1async function createSticky(content: string) {
2    try {
3    // Create a new sticky note using the widget position as reference
4    const newSticky = figma.createSticky();
5    if (stickyFill !== null) {
6      newSticky.fills = stickyFill;
7    }
8    // Load the font before setting characters
9    const defaultFont: FontName = { family: "Inter", style: "Medium" };
10    await figma.loadFontAsync(defaultFont);
11    newSticky.text.fontName = defaultFont;
12
13    newSticky.text.characters = content || '';
14
15    if (!widgetId) {
16      console.error("Widget ID not found");
17      return;
18    }
19    const widgetNode = figma.getNodeById(widgetId) as WidgetNode;
20
21    if (widgetNode) {
22      newSticky.x = widgetNode.x + widgetNode.width + 100;
23      newSticky.y = widgetNode.y + (widgetNode.height / 2) - (newSticky.height / 2);
24    } else {
25      console.error("Widget node not found.");
26      return;
27    }
28    // Create a connector between the widget and the new sticky
29    const connector = figma.createConnector();
30    connector.connectorStart = {
31      endpointNodeId: widgetId,
32      magnet: 'AUTO'
33    };
34    connector.connectorEnd = {
35      endpointNodeId: newSticky.id,
36      magnet: 'AUTO'
37    };
38  } catch (error) {
39    console.error("Error creating new sticky:", (error as Error).message);
40  }
41}

How do we parse ChatGPT responses and render native code components in FigJam? This behaviour is a little bit more involved than our previous handler. We need to pass information about the programming language for syntax highlighting to be formatted properly. First, we use a regular expression to extract the language and the code from the context of the chat completion. This expression looks for text enclosed in triple backtick code block syntax and isolates the language and code respectively. Next, we need to load a code block font to satisfy the Plugin API. We call figma.createCodeBlock() to create a new FigJam code block component. We define isCodeLanguage as a function to check whether the extracted language is compatible with syntax highlighting in FigJam. Finally, we position the code block relative to the widget instance and create a connector to link the block with the widget.

createCodeBlock
jsx
1 async function createCodeBlock(content: string) {
2    try {
3    const languageRegEx = /```(.*?)\n([\s\S]*?)```/g;
4    const languageMatch = languageRegEx.exec(content); // note that languageMatch[0] returns the entire matched string 
5
6    if (!languageMatch) {
7      console.error("Error parsing code block. Please try again with a supported language.");
8      return;
9    }
10
11    const defaultFont: FontName = { family: "Source Code Pro", style: "Medium" };
12    await figma.loadFontAsync(defaultFont);
13
14    // Match the language to the codeLanguage prop
15    if (languageMatch) {
16      const language = languageMatch[1].trim().toUpperCase(); // first match is the language right after the first triple backticks
17      const code = languageMatch[2].trim(); // second match is the enclosed code
18      // Create a new Code block using the widget position as reference
19      const newCode = figma.createCodeBlock();
20      newCode.code = code;
21
22      if (isCodeLanguage(language)) {
23        newCode.codeLanguage = language;
24      } else {
25        newCode.codeLanguage = 'PLAINTEXT';
26      }
27
28      if (!widgetId) {
29        console.error("Widget ID not found");
30        return;
31      }
32      const widgetNode = figma.getNodeById(widgetId) as WidgetNode;
33
34      if (widgetNode) {
35      newCode.x = widgetNode.x + widgetNode.width + 100;
36      newCode.y = widgetNode.y + (widgetNode.height / 2) - (newCode.height / 2);
37      } else {
38        console.error("Widget node not found.");
39        return;
40      }
41
42      // Create a connector between the widget and the code block
43      const connector = figma.createConnector();
44      connector.connectorStart = {
45        endpointNodeId: widgetId,
46        magnet: 'AUTO'
47      };
48      connector.connectorEnd = {
49        endpointNodeId: newCode.id,
50        magnet: 'AUTO'
51      };
52    }
53  } catch (error) {
54    console.error("Error creating code block:", (error as Error).message);
55  }
56}
isCodeResponse
jsx
1// All supported languages in FigJam
2type CodeLanguageValue = 'TYPESCRIPT' | 'CPP' | 'RUBY' | 'CSS' | 'JAVASCRIPT' | 'HTML' | 'JSON' | 'GRAPHQL' | 'PYTHON' | 'GO' | 'SQL' | 'SWIFT' | 'KOTLIN' | 'RUST' | 'BASH' | 'PLAINTEXT' | 'DART';
3
4// Check that language is supported, if not, default to plain text in createCodeBlock()
5  function isCodeLanguage(lang: string): lang is CodeLanguageValue {
6    const validLanguages: CodeLanguageValue[] = ['TYPESCRIPT', 'CPP', 'RUBY', 'CSS', 'JAVASCRIPT', 'HTML', 'JSON', 'GRAPHQL', 'PYTHON', 'GO', 'SQL', 'SWIFT', 'KOTLIN', 'RUST', 'BASH', 'PLAINTEXT', 'DART'];
7    return validLanguages.includes(lang as CodeLanguageValue);
8  }

Fine-tuning the widget

We've discussed the basic building blocks to satisfy basic requirements for ChatGPT usage using native FigJam components (stickies, connectors, code blocks). Obviously, there are many directions that we could focus on in future iterations. One feature of Figma's JamBot that I enjoy using are the structured queries where ChatGPT is trained to act a specific way. For example, the "Rabbit hole" option provides six distinct but related ideas to your prompt. You might define logic for different widget functions using cases to modify the system prompt in the API fetch function. In my codebase, selectedFunction is a handler that parses the user selection from the UI and declares the relevant case. Please keep in mind that different language models (ChatGPT vs. GPT-4) will respond quite differently to these prompts and a fair amount of fine-tuning is necessary to achieve optimal results.

switch-selected-function
jsx
1// In the API fetch handler
2
3switch (selectedFunction) {
4        case "Ideate":
5          systemPrompt = "You are a helpful assistant for brainstorming ideas. You will provide concise answers of one sentence or less. Given the following, provide one related idea:";
6          choices = 4;
7          break;
8        case "Teach me":
9          systemPrompt = "You are a helpful teacher. Explain the following in simple terms:";
10          break;
11        case "Rabbit hole":
12          systemPrompt = "You are a helpful assistant. You will provide concise answers of one sentence or less. You are going down a rabbit hole. Provide one example, idea, statistic, fact, or insight based on the following:";
13          choices = 4;
14          break;
15        case "Summarize":
16          systemPrompt = "You are a helpful assistant. Summarize the messages provided into a concise description.";
17          break;
18        case "Rewrite":
19          systemPrompt = "You are a helpful assistant. Rewrite the message provided according to the specified requirements.";
20          userMessage = pendingApiCall + (additionalInput || '');
21          break;
22        case "Code":
23          systemPrompt = "You are a programming assistant. Provide concise code in the requested language to implement the given task. Be sure to wrap your code in triple backticks with the specified language (```language) to format it as a code block.";
24          userMessage = pendingApiCall + (additionalInput || '');
25          break;
26        default:
27          break;
28}
Example of how you might handle additional inputs to modify the widget API call

You might also customize user prompts by providing additional inputs, for example by including a text area in the widget UI. In the example below, additionalInput is a state variable mapped to an <Input> component from the Widget API. In my code, additionalInput is appended to the pendingApiCall state before the OpenAI API is called.

additional-input
jsx
1function Widget () {
2const [additionalInput, setAdditionalInput] = useSyncedState<string | null>("additionalInput", null); 
3
4// ... 
5
6return (
7// UI code starts here...
8
9  <Input
10        value={additionalInput}
11        placeholder={inputPlaceholder}
12        onTextEditEnd={(e) => {setAdditionalInput(e.characters);}}
13        fontSize={16}
14        fill={'#FFFFFF'}
15        placeholderProps={{
16          opacity: 0.8,
17        }}
18        inputFrameProps={{
19          fill: "#020202",
20          stroke: "#D5D5D5",
21          cornerRadius: 8,
22          padding: 8,
23        }}
24        inputBehavior="wrap" // typing 'Enter' blurs and triggers     onTextEditEnd, the height of the input frame will auto-resize
25        />
26
27  // more UI code here..
28  )
29}
30

Final words

Building out this ChatGPT widget for FigJam was a great experience as a novice developer. Having to consider different design decisions on the front-end, all while working within the specific constraints of widget development was challenging but rewarding. I enjoyed learning about the detailed codebase in the Plugin API and can see myself working on more complex plugins and widgets in the future. If you're thinking about developing a specific Figma plugin or widget, I strongly urge you to get started today ๐Ÿ˜Š. You only know what you build yourself!

Recommended posts