Linux-Fu: Automation for Chrome and the Desktop By Matching Screenshots

Linux-Fu: Automation for Chrome and the Desktop By Matching Screenshots

I will be the first to admit it. This is almost not — at least not specifically — a Linux article. The subject? An automation tool for Chrome or Firefox. But before you hit the back button, hear me out. Sure, this Chrome plugin started out as a tool to automatically test web pages and automate repetitive tasks in the browser. However, it can extend that power to all programs on your computer. So, in theory, you can use it to graphically build macros that can interact with desktop applications in surprisingly sophisticated ways. In theory, anyway; there are a few problems.


The program has a few different names. Most documentation says UI Vision RPA, although there are some references to Kantu, which appears to be an older name. RPA is an acronym for Robotic Process Automation, which is an industry buzz word.


Let’s take it for a spin and see what it’s all about.

Going a Step Beyond Macro Recording


I tested is with Chrome, it works very well on both Windows and Linux (and, presumably MacOS, but I didn’t test that). The desktop mode, however, took a bit of finagling. I eventually worked out the problems — sort of — and they may or may not be dealbreakers, depending on your setup.


UI Vision RPA is quite clever. If you are just automating a web page, you can record a macro. When you interact with the browser during recording, the program can see what you are doing by monitoring the domain object model (DOM) in the browser. Then it is able to find the same things and repeat your actions during playback.


There are many ways it can identify e ..

Support the originator by clicking the read the rest link below.