Hello everyone. It’s been a while.. again. Did not have anything interesting to write about.
But today I have something very cool I want to showcase to you. An application which was created entirely by claude.ai. But that is not the cool part (even though that’s pretty impressive). It’s an application I always wanted, but never found.
Disclaimer: To be clear, I did not write a single line of code, so I won’t be able to help with issues and bugs. I also have no idea if it has any security issues. It probably does.
The first script, labeled “online,” uses the Google API for handwriting recognition. It’s more precise and faster, but sends everything to Google.
I recently bought a used Lenovo X1 Yoga Gen 7, which is absolutely fantastic, and installed Fedora GNOME on it, since it has the best touchscreen support, as far as I am aware.
While playing around, I figured a handwriting recognition tool would be nice. Not useful, but cool. So the search began. To my surprise, there is absolutely nothing out there for this use case, and my programming skills are basically non-existent.
So, I tried inputting my request into claude.ai. And to my surprise, it created a very nice working version. Not beautiful, but functional. This version uses a Google API. So every stroke gets send to Google.
Online Handwriting Recognition
Here a screenshot of that.
That’s cool and all, but I wanted a version that allows me to write anywhere on the screen. After a few more corrections and changes we got to this version.
I wanted more features. So… a bit of story time. My girlfriend recently bought an iPad Pro with a stylus and all. Now, I am not the biggest fan of Apple products, but I must say the handwriting features are really impressive. For instance, you can scribble over a word to delete it or write almost anywhere to get it converted into text. I wanted something similar.
So, after adding more features, we reached this point.
We have different colors, a pretty status bar, a timer and a few more features like a language selector (don’t think that actually works).
Next, I wanted an option to remove a word I wrote. This came out.
This is pretty cool so far. So I thought, why not use what we already have here, and make a live annotation tool. It’s not perfect, but it works. Here it is.
Offline Handwriting Recognition
Again. This is all using the google API for handwriting recognition, and I actually wanted a local version, which seems to be a bit more complicated.
It works, but unfortunately the offline recognition is not as precise. It makes a lot of mistakes. Also, it’s a bit slower than the google API version. Here a video to showcase the text recognition. It looks basically the same.
It uses the Microsoft models TROCR for handwriting recognition. The script downloads it from Huggingface and loads it into the cache.
New iteration
After all this, I tried to add a few more features, but it kept failing. Breaking the code. That’s why I thought, let’s try a new approach.
I will let claude create a new application from scratch, but giving all the information from the beginning. The hope was, that this would create better code. Also I wanted the code to be modular from the beginning, so we landed at this.
The stylus text is much smoother (this took a whole day to get working right), a clean and beautiful way to select colors and its modular.
This version also uses googles API for the handwriting to text conversion. It just works faster and better, but I will try to get a offline version working.
Virtual Keyboard – GNOME Extension
Last thing I let Claude build for me, was a OSK (on-screen keyboard). I don’t like the build-in one, and existing keyboard are either not supported anymore or only support ANSI layout with US keys, which is not what I want.
It has text field and tablet detection. It does work, more or less. Also we can switch the languages and to a handwriting mode with a button at the bottom.
In the options, we can change between the recognition models, set the delay, word suggestions (it’s not great), and a few other things.
The layout is a bit weird, since it’s not exactly ISO but close enough. I set it to a 75% since I think its the best layout for a keyboard.
Also, what the build-in OSK is missing, are the “ctrl”, “alt” and “meta” keys, which I find very useful in tablet mode.
So, here is the keyboard Claude build.
After using it for a while, I understand why they never implemented a full keyboard layout, it’s just not a great experience. After trying a few iterations, I landed on this design.
It’s essentially the Android layout. You can switch between symbols and characters while still having access to the handwriting feature. When it opens, it pushes the windows up so you can see exactly what you’re writing. Overall, it can be a bit unreliable and annoying if the ‘auto-show’ feature is enabled, but it works for the most part.
You can switch between the 75% and compact layout in the settings.
That’s it. Now. I want to share these with you but a few things to keep in mind.
The application “Katip” uses Google’s Inputtools API. So if Google decides that it should be limited or blocks it completely, the application won’t work. I will try to get a better offline version working.
I will also attach the older versions, in case someone wants to try it.
Hopefully someone with some coding skill, finds this interesting or useful, picks it up and can expand on it. Or better yet, builds something similar from scratch, much cleaner.
These are tested on “Fedora GNOME” but should work on most desktop environments, except the OSK of course. That’s a GNOME extension.
Have fun and take care.
Till next time.
Update – new options and bugfixes
Hi there,
I uploaded an updated version. It now has a options button, that allows the change of settings through a GUI. Also the stylus pressure detection is a bit better and the toggle to disable finger input actually works now.
Finally, the option to select MyScript was added.
Here a couple of screenshots.
Very cool project. For the recognition part, you should really look at MyScript.
For this use case, MyScript Web APIs are probably the best option available right now on Linux. They are made for handwriting and digital ink recognition, so they’re a much better fit than relying on Google Input Tools or trying to force generic OCR models into this job.
You can also try MyScript for free, so it’s easy to test in your app without much friction.
And if you want to go fully offline later, MyScript also offers a C++ SDK through an evaluation agreement for proper on-device handwriting recognition.
So if you want the clearest path forward on Linux:
– MyScript Web APIs for the best practical recognition option right now
– MyScript C++ SDK if you want full local/offline recognition later
Honestly, MyScript looks like the best solution currently available to make this work properly on Linux.
Hi Luc,
thank you for the comment and the fantastic recommendation.
I will look into it and update the post.