So one thing I enjoy about VoiceOver on my iPhone is it has gotten really good at using UI to make otherwise inaccessible UI elements available to interact with. More than just simple OCR, it can ascertain the layout and make educated guesses about controls like buttons and tabs, greatly expanding the usability of apps that otherwise would be partially or totally inaccessible.

Is there any chance NVDA will eventually reach that level of sophistication? I know there are third party add-ons that attempt to bridge that gap for specific types of apps, for example the great Sibiac add-on which helps make certain music production apps and plugins accessible with NVDA, but it would be great to see these capabilities broadened and rolled into the core functionality of the product.

 

Thanks,

Luke

 

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

You can do Machine learning with python, so It may not be imposible to
implement AI.

On 12/1/2021 8:29 am, Rob Hudson wrote:

Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

Yeah all I know is that it's a scripting language. Didn't know how deep it got into machine learning fuzzy logic and AI>

----- Original Message -----
From: "Jaffar Sidek" <jaffar.sidek10@...>
To: nvda@nvda.groups.io
Date: Tue, 12 Jan 2021 08:55:24 +0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

You can do Machine learning with python, so It may not be imposible to
implement AI.

On 12/1/2021 8:29 am, Rob Hudson wrote:

Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

So one thing I enjoy about VoiceOver on my iPhone is it has gotten really good at using UI to make otherwise inaccessible UI elements available to interact with. More than just simple OCR, it can ascertain the layout and make educated guesses about controls like buttons and tabs, greatly expanding the usability of apps that otherwise would be partially or totally inaccessible.

Is there any chance NVDA will eventually reach that level of sophistication? I know there are third party add-ons that attempt to bridge that gap for specific types of apps, for example the great Sibiac add-on which helps make certain music production apps and plugins accessible with NVDA, but it would be great to see these capabilities broadened and rolled into the core functionality of the product.

Thanks,

Luke

On Jan 11, 2021, at 4:30 PM, Rob Hudson <rob_hudson3182@...> wrote:

Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

On Jan 11, 2021, at 5:01 PM, Rob Hudson <rob_hudson3182@...> wrote:

Yeah all I know is that it's a scripting language. Didn't know how deep it got into machine learning fuzzy logic and AI>

----- Original Message -----
From: "Jaffar Sidek" <jaffar.sidek10@...>
To: nvda@nvda.groups.io
Date: Tue, 12 Jan 2021 08:55:24 +0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

You can do Machine learning with python, so It may not be imposible to
implement AI.

On 12/1/2021 8:29 am, Rob Hudson wrote:
Is that capability even available in python.

----- Original Message -----
From: "Supanut Leepaisomboon" <supanut2000@...>
To: nvda@nvda.groups.io
Date: Mon, 11 Jan 2021 16:18:00 -0800
Subject: Re: [nvda] Will NVDA eventually use AI for better GUI recognition?

That is actually a great idea, I can imagine this capability being used to make PC games more accessible with NVDA.

On Jan 12, 2021, at 1:32 AM, Sean Murphy <mhysnm1964@...> wrote:



AI is not that smart as it is all dependent on the data used to train it. Thus all object recognition AI has someone who has applied a specific description to icons, a part of an image, etc. An example where this breaks is if the developer does not use standard icons to indicate navigation like backwards, forward, okay, etc. The user will get some description from the Object recognition and might not be meaningful due to the description provided. The term used in other countries for that description also might not be known:

Lets say an icon is used to indicate to the user that this is the delete button. The AI Object OCR detects this as a trash can. Outside the USA the term trash can is not used. So the user might not understand this is the delete button. The delete button is tied to some important bit of information. They press it to try and work out what it does. Oops, that information is gone. The object recognition then has to know what country your device is using and then use that’s countries term for a rubbish bin (Australian term for trash can). This again assumes this icon is a delete button. There is no guarantee the designer of that app will use internationally standard icons. Yes, there is an organisation that defines all the icons, emoticons, etc.

 

Outside simple standardised icons. Object recognition has a long way to go and requires far more research. I do have the hope it will be able to handle complex images like organisation graphs, etc.

 

Note, the above is just an example demonstrating AI is not a silver bullet. It does help, but I have seen situations on Apple where it has caused more problems than it fixes. Especially if the button, link, etc already has an accessible name which VoiceOver reads out. The Apple solution still gives you the object recognition icon which makes things to verbose. Before it is introduced into any screen reader. Real careful thinking needs to be taken into consideration how it is adopted. As I can see it becoming more of an annoyance than a solution.

 

 

I am not saying it should not be explored to see how it will help. One area is AI dynamically changing the UI of a program to make it easier for a keyboard user. This is one thing that people are already doing work on.

 

Sean

 

 

From: nvda@nvda.groups.io <nvda@nvda.groups.io> On Behalf Of Luke Robinett
Sent: Tuesday, January 12, 2021 10:49 AM
To: nvda@nvda.groups.io
Subject: [nvda] Will NVDA eventually use AI for better GUI recognition?

 

So one thing I enjoy about VoiceOver on my iPhone is it has gotten really good at using UI to make otherwise inaccessible UI elements available to interact with. More than just simple OCR, it can ascertain the layout and make educated guesses about controls like buttons and tabs, greatly expanding the usability of apps that otherwise would be partially or totally inaccessible.

Is there any chance NVDA will eventually reach that level of sophistication? I know there are third party add-ons that attempt to bridge that gap for specific types of apps, for example the great Sibiac add-on which helps make certain music production apps and plugins accessible with NVDA, but it would be great to see these capabilities broadened and rolled into the core functionality of the product.

 

Thanks,

Luke

 

Sean,

I’m not fully on board with your assessment. Machine learning isn’t a static thing; by its very definition, you feed it content in a particular domain area and it gets better and better at what it does. Just because something is still in its infancy doesn’t mean it isn’t worth pursuing. If that were the attitude, we wouldn’t have screen readers in the first place. Besides, much like current OCR capabilities, this wouldn’t be a replacement for how NVDA normally works – just another supplement. and like I said, it’s already being implemented quite successfully in voiceover, so the argument that the tech just isn’t reliable enough yet kind of goes out the window. That doesn’t mean it’s perfect, but when has anything about accessibility ever been perfect? LOL.

On Jan 12, 2021, at 1:32 AM, Sean Murphy <mhysnm1964@...> wrote:



AI is not that smart as it is all dependent on the data used to train it. Thus all object recognition AI has someone who has applied a specific description to icons, a part of an image, etc. An example where this breaks is if the developer does not use standard icons to indicate navigation like backwards, forward, okay, etc. The user will get some description from the Object recognition and might not be meaningful due to the description provided. The term used in other countries for that description also might not be known:

Lets say an icon is used to indicate to the user that this is the delete button. The AI Object OCR detects this as a trash can. Outside the USA the term trash can is not used. So the user might not understand this is the delete button. The delete button is tied to some important bit of information. They press it to try and work out what it does. Oops, that information is gone. The object recognition then has to know what country your device is using and then use that’s countries term for a rubbish bin (Australian term for trash can). This again assumes this icon is a delete button. There is no guarantee the designer of that app will use internationally standard icons. Yes, there is an organisation that defines all the icons, emoticons, etc.

 

Outside simple standardised icons. Object recognition has a long way to go and requires far more research. I do have the hope it will be able to handle complex images like organisation graphs, etc.

 

Note, the above is just an example demonstrating AI is not a silver bullet. It does help, but I have seen situations on Apple where it has caused more problems than it fixes. Especially if the button, link, etc already has an accessible name which VoiceOver reads out. The Apple solution still gives you the object recognition icon which makes things to verbose. Before it is introduced into any screen reader. Real careful thinking needs to be taken into consideration how it is adopted. As I can see it becoming more of an annoyance than a solution.

 

 

I am not saying it should not be explored to see how it will help. One area is AI dynamically changing the UI of a program to make it easier for a keyboard user. This is one thing that people are already doing work on.

 

Sean

 

 

From: nvda@nvda.groups.io <nvda@nvda.groups.io> On Behalf Of Luke Robinett
Sent: Tuesday, January 12, 2021 10:49 AM
To: nvda@nvda.groups.io
Subject: [nvda] Will NVDA eventually use AI for better GUI recognition?

 

So one thing I enjoy about VoiceOver on my iPhone is it has gotten really good at using UI to make otherwise inaccessible UI elements available to interact with. More than just simple OCR, it can ascertain the layout and make educated guesses about controls like buttons and tabs, greatly expanding the usability of apps that otherwise would be partially or totally inaccessible.

Is there any chance NVDA will eventually reach that level of sophistication? I know there are third party add-ons that attempt to bridge that gap for specific types of apps, for example the great Sibiac add-on which helps make certain music production apps and plugins accessible with NVDA, but it would be great to see these capabilities broadened and rolled into the core functionality of the product.

 

Thanks,

Luke

 

On Jan 12, 2021, at 4:22 PM, Pranav Lal <pranav.lal@...> wrote:

Hi all,

As others have mentioned, the problem is the model. We need someone to compile data on what user interfaces look like, probably at a control level and then determine how to interact with it.

Suppose there is a non-standard edit box, NVDA would need to know that there is an edit box and then would also need to interact with it. It may not get the relevant events from it so it is hard to say how good such a feature would be but this is very possible to do. Does anyone have a database of pictures of windows controls?


Pranav