[balenaOS] On-screen keyboard

[benfrancis]

As a user of WebThings Shell on balenaOS I would like an on-screen keyboard to automatically appear when I focus a text field and automatically disappear when the text field loses focus, so that I can use the keyboard to input text when I need it and the keyboard gets out of my way when I don't need it.

[benfrancis]

I've managed to get WebThings Shell to run on balenaOS, using a fork of Balena's xserver block, which runs X11 inside a Docker container (which to be honest I didn't know was possible), using the matchbox window manager.

The problem I have (which is a solved problem on Ubuntu Core) is how to get an on-screen keyboard to appear whenever the user focuses a text box on a web page, and to make it to disappear on blur.

I tried installing matchbox-keyboard on the xserver Docker image, which supposedly supports GTK embedding, but that isn't baked in by default. You can manually start the keyboard on the command line, and kill the process to close it, but that's about it.

I thought we could potentially try to use a JavaScript-based keyboard like KioskBoard or electron-virtual-keyboard and use Electron APIs to inject JavaScript into all web pages to try to spot text boxes and attach event listeners to them, but that seems like it would be very brittle and prone to errors.

electron-virtual-keyboard

electron-virtual-keyboard has to be instantiated (using JQuery) for every element that it's used with, so every text input gets its own keyboard. That doesn't seem like it's going to work...

KioskBoard

KioskBoard can be instantiated with data attributes on an HTML element then KioskBoard.run('.js-kioskboard-input', { ... } )

Alternatively the keyboard can be instantiated with KioskBoard.init({...}) and then run on an element with run('.my-custom-class'). Using this approach it can be applied to all text inputs with that class, and seems to appear and disappear on focus and blur.

Haven't been able to get it to work with the BrowserWindow component yet.

Question: For webviews, what if a text input is added to a page at runtime? (Very likely). Would we have to use a MutationObserver?

[benfrancis]

My friend Sam (@samwwwblack) looked into this and discovered that other people have had issues getting on-screen keyboards to work with an Electron-based application https://stackoverflow.com/questions/67327242/control-visibility-of-gnome-3-on-screen-keyboard-in-electron-app

He spent some time trying to get this to work and reported back...

squeakboard

I've tried to set up squeekboard that's in the Ubuntu Frame thing you were using before, but trying to get Wayland/compositor/Electron to play nicely in containers is [challenging]

(I won't say what he actually said 😂).

squeekboard does seem to work once it's running; https://forums.raspberrypi.com/viewtopic.php?t=390053 but that fault (ie the keyboard is behind the fullscreen window) persists if I run squeekboard in a fedora VM to avoid the container pain.

Other X11 OSKs

There are some other X11 OSK (onboard, svkbd, xvkbd) but these all fail the auto show on input - onboard works if the app is GTK, but Electron inputs don't fire the required signal to notify the OSK, even though Electron uses GTK (?!).
The only method I can see is to fire out the required signal manually, as noted in the SO post, or to do as you suggest and include a node OSK in the shell app.