Firefox's browser engine, Gecko, added support for Web Serial in release 151. This opens up a fun new path for hardware-to-web experiments and experiences. To try it out, I built Page Playground: a device featuring a variety of novel input methods that allows me to point at elements on a web page and alter their colors, layout, text properties, and transforms in real time. It reveals the potential for powerful and intuitive user experiences integrating hardware with the web.

Idea

As seen in the video, the Page Playground can alter various properties of elements on a webpage. It creates a new and fun way to interact with a page, but you can imagine a lot of professional use cases for such a tool, too (like web development, style tweaking, etc).

Assembly

Adafruit stocks a great set of essentially plug-n-play parts, so getting this off the ground didn't take much time at all! I decided to use the MACROPAD RP2040 as the base of the device functionality, since it has a nice screen, rotary encoder, and 12 keys with RGB LEDs. This was plenty to work with already, but I also added:

• two more rotary encoders,
• an IMU (BNO085),
• a color/proximity sensor (APDS-9960),
• a joystick, and
• a slide potentiometer.

All of these provide fun and different ways to interact with the webpage.

The joystick and potentiometer are simple analog devices, so I needed to get another board to access those over I2C (because the MACROPAD doesn't expose anything other than I2C over a STEMMA QT port). Conveniently, my order was large enough that Adafruit provided a KB2040 for free :)

MACROPAD

The MACROPAD was very easy to assemble; I just had to add the key switches and caps, and a rotary encoder top (I didn't realize the bare bones kit didn't come with one, so I came up short one top, whoops).

Soldering

Next, I soldered all components that needed it (well, almost all of them: I took a second pass to add a few more things once my minimum feature set was complete). This was mostly attaching headers, which went quickly. The encoders also needed to be soldered to the I2C boards.

Wiring

Next, I wired everything on a small breadboard. This was pretty simple, because all the devices were I2C, so I just needed to chain them all together with the same 4 wires. Adafruit was unfortunately sold out of the STEMMA QT cables, but it was just as well, because putting everything on a breadboard made it easier to assemble as a prototype later.

Note that I didn't yet attach the analog components and the KB2040. I could get my first prototype to work without those extra inputs, so I omitted them.

Because the STEMMA QT cables were sold out, I had to get a STEMMA QT to STEMMA cable and splice in wires to join to the breadboard. The threaded wires in the STEMMA QT connector are tiny, so I opted for twist wire connectors to reduce the possibility of them breaking.

After placing the components on the breadboard, I was happy to plug in the MACROPAD and see everything was lit up, with the I2C addresses enumerated on the MACROPAD screen.

Programming

I don't have any exciting pictures of programming, but I will outline how the code works and how I developed it.

If curious, you can find the full source code on GitHub.

Messaging

I chose to use MessagePack serialization because there was a readily-available library for Arduino (C++), and the same developer has a nice packetizing library using COBS that can prepend a byte to indicate the message type (a feature I used). The serial connection is fairly reliable, but the use of COBS to frame packets gave me flexibility to e.g. add extra data to packets without worrying about any other code handling them.

I didn't want to use text-based serialization like JSON because I wanted the modifications to be as fast and responsive as possible. The overhead of extra parsing and extra bytes over the interface may have been limiting. Yes, JSON may have worked (I didn't try, after all), but it was simple enough to just use MessagePack from the start.

Browser

On the browser side, I inject a GreaseMonkey script into every page using the TamperMonkey add-on. I found TamperMonkey a bit easier to use than the GreaseMonkey add-on, as the latter doesn't automatically prompt to install a script when you visit a .user.js URL, and to develop I needed to reload the script often.

The script is built using bun. This was useful because it:

• made it trivial to add dependencies (like uCOBS and msgpack-javascript),
• allowed me to use TypeScript (which caught many issues at compile/write time), and
• made it easy to bundle static resources.

The script adds a button in the bottom-right corner of the screen. When clicked, it opens a Web Serial connection (prompting the user to choose a device).

Note: In Firefox 151, Web Serial access also involves Firefox’s site-permission flow before the usual device chooser appears, which enforces explicit user consent to connect to a device.

Once connected, the script registers a mousemove event handler. As the mouse moves to new elements in the DOM, various style properties of the element are sent in a message to the device. When those properties are modified by the device, it sends back messages indicating the changes. The script applies those changes to the current element. In this way, the script is fairly simple.

For the effects, the images weren't readily accessible from network URIs, so I had to bundle images as base64 in the script (I can't e.g. access them as file URIs, either). The Kit GIF in particular (even when downsized and optimized) made the script magnitudes larger: it ended up around 1MB total.

Device

I usually use cross-compile toolchains and device programmers directly, but I opted for the Arduino IDE because the Adafruit guides all give (quite good) instructions around it; it was the path of least resistance! I was able to add most libraries I needed using the built-in library manager. It has been quite a while since I used the Arduino IDE, and the library manager is a welcome addition.

One issue I immediately encountered was that my laptop was auto-suspending the USB bus when there was no TX/RX for about 3 seconds. This would prevent the serial connection from working. It wouldn't disconnect, but it wouldn't be able to send from device to browser until the browser sent something out. It took a while to debug this and come to the conclusion that autosuspend was to blame. I first disabled autosuspend for the entire system. When that worked, I looked closer at the TinyUSB functions (this is the USB software stack used in the RP2040), and found some functions to wake it back up. Now, before writing to the serial connection, I call:

if (tud_suspended()) tud_remote_wakeup();

Thankfully, I found all the Adafruit libraries Just Worked without any fiddling! It was very fast to get up and running. I set up peripherals and started programming the MACROPAD. I had already decided on how it was going to function: the rotary encoder would change the tool "page", and each page would have related properties.

Global behavior

The device accepts two messages:

• a reset message to set the state back to initial state when first connecting to a new page, and
• a style info message to give information about the selected element.

Each page simply displays/updates the relevant information in the style info, though when changes are made there are individual messages to affect each change (rather than sending a big message with all the changes, and relying on the browser code to check for a delta or set everything).

The MACROPAD encoder button can also be clicked to lock the selected element (not shown in the video), which prevents it from changing as the mouse changes. It is indicated by inverted colors in the element name on the OLED screen, and can be unlocked by clicking again.

Colors

The color page allows setting the foreground and background color of an element. At one point I imagined this could be extended to alter the color palette of the entire page, but I didn't have time to implement that.

You can press and hold any of the keys to set the color using the two other rotary encoders (they alter HSL components). The color of the NeoPixel under the key reflects the associated color. You can also use the APDS-9960 color sensor to read a color, though I found that it really needs a bright input. It took a few iterations to refine the behavior, but it didn't take long to have something working. And dynamically changing the colors of elements on the screen was really cool! I was glad to see that everything was responsive and essentially instantaneous.

Transform

I worked on the transform page next, because it offered a very cool effect: you could move the device and have an element on the page move in a similar way! This uses the BNO085 IMU. I found the orientation data to be very accurate, so getting orientation to work wasn't too hard. Integrating the acceleration to movement was more tricky (as it always is), and I brought in some external Quaternion code to make transforming the acceleration vector simple. There's still some drift in the position, but I deemed it Good Enough.

Later on, I added an analog joystick and mapped the analog values to transforms, too (limited to XY).

Layout

The layout tool was a little more involved, mainly because there were many values to show. There was enough room on the screen to display a little box model layout, showing margins, padding, and width/height. As you click the related keys on the MACROPAD (in a 3x3 grid), you can select the left/right/top/bottom (or all of those) sides, as well as width and height. Then, the rotary encoder can be used to adjust values (pressing down to adjust padding rather than margin).

I also added a feature to set a value using the APDS-9960 proximity sensor. The sensor doesn't have a long range, but it was still cool to set values in that way.

Text

The text page was one of the last I did, only because it was somewhat straightforward. I programmed the buttons to change to common font families, and each rotary encoder changes font size and line height.

Effects

The effects on the device were trivial to implement: clicking a button sends a message with the numbered effect to perform.

Finishing Touches

More Inputs

After I had all of the above working, I really wanted to use a slide potentiometer to enable (and potentially scale) the use of sensors for the relevant pages, and I thought it'd be cool to get a joystick in there, too. This required wiring in the KB2040 and setting it up for I2C, because there were no analog inputs on the MACROPAD. I was out of space on my breadboard (once the joystick was mounted there), so I soldered the wires directly to the KB2040 (the headers I had attached earlier sort of got in the way, actually).

I added the analog inputs, and programmed the KB2040 to act as an I2C peripheral that forwards the analog values (this was nearly trivial, and worked the first time!). It only took a few adjustments to use these new inputs in the MACROPAD code. The bulk of the change (which was still small) was using the joystick values on the transform page. I forgot to take a picture after connecting the new wires, but you can see most of it in the enclosure picture.

Enclosure

I picked up a plastic 6"x4" photo box as an enclosure. I used a drill and a utility knife to cut holes in the plastic for the hardware. It isn't the prettiest thing, but it keeps everything together as one unit and offers some modest protection.

Conclusion

I had a lot of fun experimenting with Web Serial in Firefox. It made the browsing experience feel less like something shown to me, and more like something I can sculpt to my liking, reminding me of some of my earliest experiences with the Internet. It also excites me for what cool, interesting, and useful physical devices we may see in the future. Being able to use specialty hardware without installing special drivers or local software is liberating. This prototype was put together in a short period of time without much optimization, and even that was responsive enough to make changes on the page feel immediate and interactive. I can imagine many use cases for this class of device alone: design tweaking, accessibility development, live debugging, or just plain fun and unique experiences on the web. I can't wait to see what others come up with!