Re: OCR: Interacting with items from recognized UI

Luke Robinett <blindgroupsluke@...>

Hi Kara,

Thanks for your reply. I think what I wasn’t clear on is that hitting enter or space in the OCR viewer actually generates a left click on that item, something I was able to confirm. That clears up a lot of my confusion.
I think you might be right that the likely culprit is simply that the label isn’t actually part of the UI control so isn’t doing anything when I click it. Until we get more sophisticated GUI recognition tools in NVDA, my best bet is probably just to have my wife line the mouse cursor up with the controls I need so I can capture some golden cursor hotspots for future use.

Thanks again to everybody who replied,

On Jan 11, 2021, at 2:08 AM, Kara Goldfinch <kara.louise18@...> wrote:

Hi Luke,

You can move the mouse to where the virtual focus is by pressing NVDA+numpad slash, then you can press numpad slash to click it.

I've also run into the  same issue as you where enter sometimes doesn't work, so here's a couple of things I also try.

After moving the mouse to the thing you want to  click, press shift+numpad slash twice with a short pause in-between. This locks and unlocks the left  mouse button. This could help if the app expects the button to be held down longer than usual.

Another thing to bear in mind is that sometimes the text that OCR recognises might just be the control's label, and the actual control could be above, below or to the side of it. I've noticed this in Omnissphere for example. The only way round this is to randomly click around the label with golden cursor, possibly listening for a different colour/brightness level depending on how you have it set up. I usually get tired of this pretty quickly, however.

I hope this helps any.


On 10/01/2021 01:02, Luke Robinett wrote:



For inaccessible UIs I’ll often run NVDA OCR on the UI and then press enter or space bar on the UI controls I find in the recognized text such as tabs, menus and buttons. I find this only works in some programs and other times seems to be ignored. I believe the issue in some cases is that some UIs only respond when an actual mouse is clicked on a control. Is there a way to have the mouse cursor move to the UI element corresponding with its label in the OCR viewer and produce a click on that element? If not, is this something I could submit on GitHub as a feature request? It would make the already helpful OCR feature even more useful.





Join to automatically receive all group messages.