|
=Hi all, I understand that this is quite a strange (or more of a geeky) subject line, but I believe that this is the one of those last opportunities where I can pass on whatever I know to the next group of power users and developers before I pass out from graduate school. While what I’m going to write for a while (in the “Explainer” series) is geared more towards NVDA developers and add-on authors, I think it would be helpful to post some notes to the users list so at least some folks can understand how a screen reader and related technologies work behind the scenes (there are other reasons, some related to academics, others related to ongoing screen reader development, some related to mentoring). The below post is a result of what I might only describe as a “celebratory week” and my wish to document most (if not everything) of what happened to bring a bug fix that took more than four years to get it right, as it will also talk about upcoming changes to NVDA (2022.4 to be exact), the subject being accessible drag and drop (discussed at lengths months ago). In April 2022, a user asked the question, “how can I drag and drop items?” Among many responses (see the group archive for details), the mouse-based drag and drop method was discussed, as well as commands provided by NVDA to manipulate the mouse in doing so. But two things were not discussed then )or if discussed, receive little attention_: how do controls expose drag and drop capability, and can screen readers inform users about drag and drop operation? These I will answer below. Before discussing accessible drag and drop, it is important to understand how graphical user interface (GUI) controls are laid out on screen, as well as visual and nonvisual interaction paradigms. At a high level, GUI elements are organized into a hierarchy (or a tree, if you will). At the top of the hierarchy is the shell, or sometimes called “desktop”. On top (or below) the shell element are top-level windows for an app that houses all the controls that an app will be showing you. Below each top-level window are smaller windows, housing elements that will be shown on the screen most of the time (this is called “foreground window”). Inside these smaller windows are another set of windows, inside are even smaller windows, and eventually, we end up with the actual control being interacted with (button, checkbox, edit field, web document, and so on). This is the foundation for NVDA’s object navigation principles – when you move between objects (NVDA+Numpad arrow keys/NVDA+Shift+arrow keys), you are effectively moving between windows at a higher or lower level. A useful analogy is street addresses. Suppose you order a computer from a retailer, and whoever is responsible for delivery (perhaps Amazon, drones, or whatever or whoever) needs to locate where you live. The street address you provide to the retailer will first include the country of residence, followed by regions (states, provinces, and what not), itself divided into smaller local regions (cities, counties, etc.), in turn moving down to the street grid, and finally, the street address. Another analogy, albeit visual at first, is focus of a camera – you can zoom in or out of a specific thing the camera is pointing at, and sometimes users move their camera to focus on a different thing. This is how GUI elements are internally organized: shell window housing top-level application windows, in turn housing smaller windows, toolbars, emu bar and what not, in turn leading users to a specific control (I and so many others have discussed why this is important in NVDA’s object navigation principles many times, so I’ll skip it for now). Another thing to keep in mind is visual and nonvisual interaction paradigms. Our eye (visual senses) is perhaps the first thing we use when looking and interacting with a person, a thing, a screen control, or an idea. As GUI’s use visual senses to help people perform tasks with computers (nowadays tablets and even augmented reality (AR) tools), a paradigm best suited for visual manipulation of screen controls is the dominant interaction paradigm. These include mice, touchscreens, digitizers, eye control, and our thoughts. One way to interact with and manipulate elements on screen is drag and drop. In short, the item to be dragged is first selected (technically called “grabbed”) by using whatever input tool we use (moving system focus using the keyboard, holding down the mouse, long press of touchscreens, focusing a bit longer on the item with our eyes, etc.), then it is dragged across the screen to the desired location (called a “drop target”). For applications of this concept, see April 2022 archive of NVDA users list. Even if we hold down the mouse and drag something, many things happen in the background, some of which are accessibility related if conditions are right. First, the app informs accessibility API’s that an item is grabbed. Second, accessibility API’s inform anyone listening that a drag and drop operation is in progress (this differs between API’s as explained below). Third, while dragging and dropping, accessibility API’s may inform users about where the item is going, sometimes helped by the host app as to where things are going (not all apps do this). Lastly, when a drop operation happens, the app informs folks that drag and drop is complete (or canceled in some cases), which in turn allows accessibility API’s to inform users that drag and drop is complete. Because at least two accessibility API’s are in use by Windows based assistive technologies, the above process differs between API’s: - Microsoft Active Accessibility (MSAA/IAccessible): an attribute change informs screen readers and other assistive technologies that a drag and drop operation is in progress.
- UI Automation (UIA): up to six events and six properties are defined just for accessible drag and drop. When drag starts, the item in question sets “is grabbed” UIA property to True, and this gets picked up by drag start UIA event and announced to screen readers. While dragging items, UIA uses two properties to communicate where the item is going, and in some cases, the control that supports dropping the grabbed item also raises a series of events and property changes to inform screen readers and other technologies about drag and drop progress. I won’t go into details here as this forum is meant for users (I can go into details if asked privately or on development-oriented forums).
A few days ago, a suggestion was posted on NVDA’s GitHub repository about letting the screen reader announce dragging (is grabbed) state, specifically when performing drag and drop from the keyboard. In order to do so, NVDA must recognize the “is grabbed” state, then announce the progress of this operation. The first part is now part of NVDA 2022.4 (alpha builds at the moment), and a follow-up pull request was posted to bring the second part (drag and drop progress announcement) to NVDA. For example, if you rearrange Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will be told that an item is being dragged (grabbed), and it can announce what happens when you drop the item somewhere. You can rearrange tiles in Start menu either from the mouse or the keyboard (Alt+Shift+arrow keys). Tip: if you are using a specific add-on, NVDA can announce the drag and drop state and progress (will explain why I’m saying this tip in a follow-up post). This raises two questions: - Why is it that NVDA will announce only part of the progress when I use the keyboard to drag and drop items? Unlike mouse-based drag and drop where you must hold the mouse down, keyboard commands will force the item to be dragged to somewhere in one sweep (may things happen but that’s what I think is the best way to explain it).
- How come I cannot hear drag and drop announcement when dragging things in some places? For three reasons: first, the item being dragged must support drag and drop operation and properties in a way that can be made accessible (sometimes, the drop target control must support accessible drag and drop operations properly). Second, not all accessibility API’s provide accessible drag and drop announcements and events. Third, even if accessible drag and drop is possible, assistive technologies such as screen readers must support it. The first reason is why NVDA (and for that matter, Narrator) may not announce progress of rearranging taskbar icons in Windows 11 if done from the keyboard. The second is the reason why accessible drag and drop was added in Windows 8 via UI Automation. The third reason is why Narrator can announce accessible drag and drop progress and events when NVDA could not… until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work required to support accessible drag and drop. Cheers, Joseph
|
|
|
Brian's Mail list account
I have to say I have never really managed to get drag and drop to be understandable. In the main along with many other windows things, there is often more than one way to do stuff, and cut and past often can be done instead and be a damn site less complicated. Of course I can see that for a sighted person drag and drop is intuitive and obvious, but its harder with a screenreader as you do not have the overview that you do with a working pair of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
toggle quoted message
Show quoted text
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from the perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject line,
but I believe that this is the one of those last opportunities where I can
pass on whatever I know to the next group of power users and developers
before I pass out from graduate school. While what I'm going to write for a
while (in the "Explainer" series) is geared more towards NVDA developers and
add-on authors, I think it would be helpful to post some notes to the users
list so at least some folks can understand how a screen reader and related
technologies work behind the scenes (there are other reasons, some related
to academics, others related to ongoing screen reader development, some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than four
years to get it right, as it will also talk about upcoming changes to NVDA
(2022.4 to be exact), the subject being accessible drag and drop (discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop items?"
Among many responses (see the group archive for details), the mouse-based
drag and drop method was discussed, as well as commands provided by NVDA to
manipulate the mouse in doing so. But two things were not discussed then )or
if discussed, receive little attention_: how do controls expose drag and
drop capability, and can screen readers inform users about drag and drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to understand
how graphical user interface (GUI) controls are laid out on screen, as well
as visual and nonvisual interaction paradigms. At a high level, GUI elements
are organized into a hierarchy (or a tree, if you will). At the top of the
hierarchy is the shell, or sometimes called "desktop". On top (or below) the
shell element are top-level windows for an app that houses all the controls
that an app will be showing you. Below each top-level window are smaller
windows, housing elements that will be shown on the screen most of the time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and eventually, we
end up with the actual control being interacted with (button, checkbox, edit
field, web document, and so on). This is the foundation for NVDA's object
navigation principles - when you move between objects (NVDA+Numpad arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between windows at a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer from a
retailer, and whoever is responsible for delivery (perhaps Amazon, drones,
or whatever or whoever) needs to locate where you live. The street address
you provide to the retailer will first include the country of residence,
followed by regions (states, provinces, and what not), itself divided into
smaller local regions (cities, counties, etc.), in turn moving down to the
street grid, and finally, the street address. Another analogy, albeit visual
at first, is focus of a camera - you can zoom in or out of a specific thing
the camera is pointing at, and sometimes users move their camera to focus on
a different thing. This is how GUI elements are internally organized: shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a specific
control (I and so many others have discussed why this is important in NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking and
interacting with a person, a thing, a screen control, or an idea. As GUI's
use visual senses to help people perform tasks with computers (nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited for
visual manipulation of screen controls is the dominant interaction paradigm.
These include mice, touchscreens, digitizers, eye control, and our thoughts.
One way to interact with and manipulate elements on screen is drag and drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus using
the keyboard, holding down the mouse, long press of touchscreens, focusing a
bit longer on the item with our eyes, etc.), then it is dragged across the
screen to the desired location (called a "drop target"). For applications of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen in the
background, some of which are accessibility related if conditions are right.
First, the app informs accessibility API's that an item is grabbed. Second,
accessibility API's inform anyone listening that a drag and drop operation
is in progress (this differs between API's as explained below). Third, while
dragging and dropping, accessibility API's may inform users about where the
item is going, sometimes helped by the host app as to where things are going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some cases),
which in turn allows accessibility API's to inform users that drag and drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in question
sets "is grabbed" UIA property to True, and this gets picked up by drag
start UIA event and announced to screen readers. While dragging items, UIA
uses two properties to communicate where the item is going, and in some
cases, the control that supports dropping the grabbed item also raises a
series of events and property changes to inform screen readers and other
technologies about drag and drop progress. I won't go into details here as
this forum is meant for users (I can go into details if asked privately or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository about
letting the screen reader announce dragging (is grabbed) state, specifically
when performing drag and drop from the keyboard. In order to do so, NVDA
must recognize the "is grabbed" state, then announce the progress of this
operation. The first part is now part of NVDA 2022.4 (alpha builds at the
moment), and a follow-up pull request was posted to bring the second part
(drag and drop progress announcement) to NVDA. For example, if you rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag and drop
state and progress (will explain why I'm saying this tip in a follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and drop
where you must hold the mouse down, keyboard commands will force the item to
be dragged to somewhere in one sweep (may things happen but that's what I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being dragged must
support drag and drop operation and properties in a way that can be made
accessible (sometimes, the drop target control must support accessible drag
and drop operations properly). Second, not all accessibility API's provide
accessible drag and drop announcements and events. Third, even if accessible
drag and drop is possible, assistive technologies such as screen readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in Windows
11 if done from the keyboard. The second is the reason why accessible drag
and drop was added in Windows 8 via UI Automation. The third reason is why
Narrator can announce accessible drag and drop progress and events when NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work required
to support accessible drag and drop.
Cheers,
Joseph
|
|
|
i am somehow able to drag and drop elements in system tray using nvda's facility and its add on. But, its not working in my bank's learning course where, where we have to match write statements with correct words. On 04/09/2022, Brian's Mail list account via groups.io <bglists@...> wrote: I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things, there is
often more than one way to do stuff, and cut and past often can be done
instead and be a damn site less complicated. Of course I can see that for a
sighted person drag and drop is intuitive and obvious, but its harder with a
screenreader as you do not have the overview that you do with a working pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I can
pass on whatever I know to the next group of power users and developers
before I pass out from graduate school. While what I'm going to write for a
while (in the "Explainer" series) is geared more towards NVDA developers
and
add-on authors, I think it would be helpful to post some notes to the users
list so at least some folks can understand how a screen reader and related
technologies work behind the scenes (there are other reasons, some related
to academics, others related to ongoing screen reader development, some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than four
years to get it right, as it will also talk about upcoming changes to NVDA
(2022.4 to be exact), the subject being accessible drag and drop (discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop items?"
Among many responses (see the group archive for details), the mouse-based
drag and drop method was discussed, as well as commands provided by NVDA to
manipulate the mouse in doing so. But two things were not discussed then
)or
if discussed, receive little attention_: how do controls expose drag and
drop capability, and can screen readers inform users about drag and drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to understand
how graphical user interface (GUI) controls are laid out on screen, as well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of the
hierarchy is the shell, or sometimes called "desktop". On top (or below)
the
shell element are top-level windows for an app that houses all the controls
that an app will be showing you. Below each top-level window are smaller
windows, housing elements that will be shown on the screen most of the time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and eventually, we
end up with the actual control being interacted with (button, checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's object
navigation principles - when you move between objects (NVDA+Numpad arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between windows at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer from a
retailer, and whoever is responsible for delivery (perhaps Amazon, drones,
or whatever or whoever) needs to locate where you live. The street address
you provide to the retailer will first include the country of residence,
followed by regions (states, provinces, and what not), itself divided into
smaller local regions (cities, counties, etc.), in turn moving down to the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific thing
the camera is pointing at, and sometimes users move their camera to focus
on
a different thing. This is how GUI elements are internally organized: shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking and
interacting with a person, a thing, a screen control, or an idea. As GUI's
use visual senses to help people perform tasks with computers (nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus using
the keyboard, holding down the mouse, long press of touchscreens, focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across the
screen to the desired location (called a "drop target"). For applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed. Second,
accessibility API's inform anyone listening that a drag and drop operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about where the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some cases),
which in turn allows accessibility API's to inform users that drag and drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in question
sets "is grabbed" UIA property to True, and this gets picked up by drag
start UIA event and announced to screen readers. While dragging items, UIA
uses two properties to communicate where the item is going, and in some
cases, the control that supports dropping the grabbed item also raises a
series of events and property changes to inform screen readers and other
technologies about drag and drop progress. I won't go into details here as
this forum is meant for users (I can go into details if asked privately or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so, NVDA
must recognize the "is grabbed" state, then announce the progress of this
operation. The first part is now part of NVDA 2022.4 (alpha builds at the
moment), and a follow-up pull request was posted to bring the second part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag and
drop
state and progress (will explain why I'm saying this tip in a follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and drop
where you must hold the mouse down, keyboard commands will force the item
to
be dragged to somewhere in one sweep (may things happen but that's what I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being dragged
must
support drag and drop operation and properties in a way that can be made
accessible (sometimes, the drop target control must support accessible drag
and drop operations properly). Second, not all accessibility API's provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in Windows
11 if done from the keyboard. The second is the reason why accessible drag
and drop was added in Windows 8 via UI Automation. The third reason is why
Narrator can announce accessible drag and drop progress and events when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
-- -- -- nothing is difficult unless you make it appear so. r. aravind, manager Department of sales bank of baroda specialised mortgage store, Chennai. mobile no: +91 9940369593, email id : aravind_069@..., aravind.andhrabank@.... aravind.rajendran@....
|
|
|
Are you on an Internet site or somewhere else where browse mode is being used? I don't think drag and drop can work for a blind user while in browse mode.
Gene
toggle quoted message
Show quoted text
On 9/4/2022 6:59 AM, Aravind R wrote: i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
On 04/09/2022, Brian's Mail list account via groups.io
<bglists@...> wrote:
I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things, there is
often more than one way to do stuff, and cut and past often can be done
instead and be a damn site less complicated. Of course I can see that for a
sighted person drag and drop is intuitive and obvious, but its harder with a
screenreader as you do not have the overview that you do with a working pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I can
pass on whatever I know to the next group of power users and developers
before I pass out from graduate school. While what I'm going to write for a
while (in the "Explainer" series) is geared more towards NVDA developers
and
add-on authors, I think it would be helpful to post some notes to the users
list so at least some folks can understand how a screen reader and related
technologies work behind the scenes (there are other reasons, some related
to academics, others related to ongoing screen reader development, some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than four
years to get it right, as it will also talk about upcoming changes to NVDA
(2022.4 to be exact), the subject being accessible drag and drop (discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop items?"
Among many responses (see the group archive for details), the mouse-based
drag and drop method was discussed, as well as commands provided by NVDA to
manipulate the mouse in doing so. But two things were not discussed then
)or
if discussed, receive little attention_: how do controls expose drag and
drop capability, and can screen readers inform users about drag and drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to understand
how graphical user interface (GUI) controls are laid out on screen, as well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of the
hierarchy is the shell, or sometimes called "desktop". On top (or below)
the
shell element are top-level windows for an app that houses all the controls
that an app will be showing you. Below each top-level window are smaller
windows, housing elements that will be shown on the screen most of the time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and eventually, we
end up with the actual control being interacted with (button, checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's object
navigation principles - when you move between objects (NVDA+Numpad arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between windows at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer from a
retailer, and whoever is responsible for delivery (perhaps Amazon, drones,
or whatever or whoever) needs to locate where you live. The street address
you provide to the retailer will first include the country of residence,
followed by regions (states, provinces, and what not), itself divided into
smaller local regions (cities, counties, etc.), in turn moving down to the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific thing
the camera is pointing at, and sometimes users move their camera to focus
on
a different thing. This is how GUI elements are internally organized: shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking and
interacting with a person, a thing, a screen control, or an idea. As GUI's
use visual senses to help people perform tasks with computers (nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus using
the keyboard, holding down the mouse, long press of touchscreens, focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across the
screen to the desired location (called a "drop target"). For applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed. Second,
accessibility API's inform anyone listening that a drag and drop operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about where the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some cases),
which in turn allows accessibility API's to inform users that drag and drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in question
sets "is grabbed" UIA property to True, and this gets picked up by drag
start UIA event and announced to screen readers. While dragging items, UIA
uses two properties to communicate where the item is going, and in some
cases, the control that supports dropping the grabbed item also raises a
series of events and property changes to inform screen readers and other
technologies about drag and drop progress. I won't go into details here as
this forum is meant for users (I can go into details if asked privately or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so, NVDA
must recognize the "is grabbed" state, then announce the progress of this
operation. The first part is now part of NVDA 2022.4 (alpha builds at the
moment), and a follow-up pull request was posted to bring the second part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag and
drop
state and progress (will explain why I'm saying this tip in a follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and drop
where you must hold the mouse down, keyboard commands will force the item
to
be dragged to somewhere in one sweep (may things happen but that's what I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being dragged
must
support drag and drop operation and properties in a way that can be made
accessible (sometimes, the drop target control must support accessible drag
and drop operations properly). Second, not all accessibility API's provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in Windows
11 if done from the keyboard. The second is the reason why accessible drag
and drop was added in Windows 8 via UI Automation. The third reason is why
Narrator can announce accessible drag and drop progress and events when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
|
|
|
yes. i was in brouse mode while using office's learning course where
we have to drag and drop objects.
in system the buttons are able to be dragged and dropped by us.
toggle quoted message
Show quoted text
On 04/09/2022, Gene <gsasner@...> wrote: Are you on an Internet site or somewhere else where browse mode is being
used? I don't think drag and drop can work for a blind user while in
browse mode.
Gene
On 9/4/2022 6:59 AM, Aravind R wrote:
i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
On 04/09/2022, Brian's Mail list account via groups.io
<bglists@...> wrote:
I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things, there
is
often more than one way to do stuff, and cut and past often can be done
instead and be a damn site less complicated. Of course I can see that for
a
sighted person drag and drop is intuitive and obvious, but its harder
with a
screenreader as you do not have the overview that you do with a working
pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from
the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I
can
pass on whatever I know to the next group of power users and developers
before I pass out from graduate school. While what I'm going to write for
a
while (in the "Explainer" series) is geared more towards NVDA developers
and
add-on authors, I think it would be helpful to post some notes to the
users
list so at least some folks can understand how a screen reader and
related
technologies work behind the scenes (there are other reasons, some
related
to academics, others related to ongoing screen reader development, some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than four
years to get it right, as it will also talk about upcoming changes to
NVDA
(2022.4 to be exact), the subject being accessible drag and drop
(discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop
items?"
Among many responses (see the group archive for details), the
mouse-based
drag and drop method was discussed, as well as commands provided by NVDA
to
manipulate the mouse in doing so. But two things were not discussed then
)or
if discussed, receive little attention_: how do controls expose drag and
drop capability, and can screen readers inform users about drag and drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to
understand
how graphical user interface (GUI) controls are laid out on screen, as
well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of
the
hierarchy is the shell, or sometimes called "desktop". On top (or below)
the
shell element are top-level windows for an app that houses all the
controls
that an app will be showing you. Below each top-level window are smaller
windows, housing elements that will be shown on the screen most of the
time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and eventually,
we
end up with the actual control being interacted with (button, checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's
object
navigation principles - when you move between objects (NVDA+Numpad arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between windows
at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer from
a
retailer, and whoever is responsible for delivery (perhaps Amazon,
drones,
or whatever or whoever) needs to locate where you live. The street
address
you provide to the retailer will first include the country of residence,
followed by regions (states, provinces, and what not), itself divided
into
smaller local regions (cities, counties, etc.), in turn moving down to
the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific
thing
the camera is pointing at, and sometimes users move their camera to
focus
on
a different thing. This is how GUI elements are internally organized:
shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking
and
interacting with a person, a thing, a screen control, or an idea. As
GUI's
use visual senses to help people perform tasks with computers (nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited
for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus
using
the keyboard, holding down the mouse, long press of touchscreens,
focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across
the
screen to the desired location (called a "drop target"). For
applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed.
Second,
accessibility API's inform anyone listening that a drag and drop
operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about where
the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some
cases),
which in turn allows accessibility API's to inform users that drag and
drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a
drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in
question
sets "is grabbed" UIA property to True, and this gets picked up by drag
start UIA event and announced to screen readers. While dragging items,
UIA
uses two properties to communicate where the item is going, and in some
cases, the control that supports dropping the grabbed item also raises a
series of events and property changes to inform screen readers and other
technologies about drag and drop progress. I won't go into details here
as
this forum is meant for users (I can go into details if asked privately
or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository
about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so, NVDA
must recognize the "is grabbed" state, then announce the progress of
this
operation. The first part is now part of NVDA 2022.4 (alpha builds at
the
moment), and a follow-up pull request was posted to bring the second
part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will
be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in
Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag and
drop
state and progress (will explain why I'm saying this tip in a follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and
drop
where you must hold the mouse down, keyboard commands will force the
item
to
be dragged to somewhere in one sweep (may things happen but that's what
I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being dragged
must
support drag and drop operation and properties in a way that can be made
accessible (sometimes, the drop target control must support accessible
drag
and drop operations properly). Second, not all accessibility API's
provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in
Windows
11 if done from the keyboard. The second is the reason why accessible
drag
and drop was added in Windows 8 via UI Automation. The third reason is
why
Narrator can announce accessible drag and drop progress and events when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
-- --
--
nothing is difficult unless you make it appear so.
r. aravind,
manager
Department of sales
bank of baroda specialised mortgage store, Chennai.
mobile no: +91 9940369593,
email id : aravind_069@..., aravind.andhrabank@....
aravind.rajendran@....
|
|
|
I don't know what you mean by in system.
Gene
toggle quoted message
Show quoted text
On 9/4/2022 7:20 AM, Aravind R wrote: yes. i was in brouse mode while using office's learning course where
we have to drag and drop objects.
in system the buttons are able to be dragged and dropped by us.
On 04/09/2022, Gene <gsasner@...> wrote:
Are you on an Internet site or somewhere else where browse mode is being
used? I don't think drag and drop can work for a blind user while in
browse mode.
Gene
On 9/4/2022 6:59 AM, Aravind R wrote:
i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
On 04/09/2022, Brian's Mail list account via groups.io
<bglists@...> wrote:
I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things, there
is
often more than one way to do stuff, and cut and past often can be done
instead and be a damn site less complicated. Of course I can see that for
a
sighted person drag and drop is intuitive and obvious, but its harder
with a
screenreader as you do not have the overview that you do with a working
pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from
the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I
can
pass on whatever I know to the next group of power users and developers
before I pass out from graduate school. While what I'm going to write for
a
while (in the "Explainer" series) is geared more towards NVDA developers
and
add-on authors, I think it would be helpful to post some notes to the
users
list so at least some folks can understand how a screen reader and
related
technologies work behind the scenes (there are other reasons, some
related
to academics, others related to ongoing screen reader development, some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than four
years to get it right, as it will also talk about upcoming changes to
NVDA
(2022.4 to be exact), the subject being accessible drag and drop
(discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop
items?"
Among many responses (see the group archive for details), the
mouse-based
drag and drop method was discussed, as well as commands provided by NVDA
to
manipulate the mouse in doing so. But two things were not discussed then
)or
if discussed, receive little attention_: how do controls expose drag and
drop capability, and can screen readers inform users about drag and drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to
understand
how graphical user interface (GUI) controls are laid out on screen, as
well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of
the
hierarchy is the shell, or sometimes called "desktop". On top (or below)
the
shell element are top-level windows for an app that houses all the
controls
that an app will be showing you. Below each top-level window are smaller
windows, housing elements that will be shown on the screen most of the
time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and eventually,
we
end up with the actual control being interacted with (button, checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's
object
navigation principles - when you move between objects (NVDA+Numpad arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between windows
at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer from
a
retailer, and whoever is responsible for delivery (perhaps Amazon,
drones,
or whatever or whoever) needs to locate where you live. The street
address
you provide to the retailer will first include the country of residence,
followed by regions (states, provinces, and what not), itself divided
into
smaller local regions (cities, counties, etc.), in turn moving down to
the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific
thing
the camera is pointing at, and sometimes users move their camera to
focus
on
a different thing. This is how GUI elements are internally organized:
shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking
and
interacting with a person, a thing, a screen control, or an idea. As
GUI's
use visual senses to help people perform tasks with computers (nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited
for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus
using
the keyboard, holding down the mouse, long press of touchscreens,
focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across
the
screen to the desired location (called a "drop target"). For
applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed.
Second,
accessibility API's inform anyone listening that a drag and drop
operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about where
the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some
cases),
which in turn allows accessibility API's to inform users that drag and
drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a
drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in
question
sets "is grabbed" UIA property to True, and this gets picked up by drag
start UIA event and announced to screen readers. While dragging items,
UIA
uses two properties to communicate where the item is going, and in some
cases, the control that supports dropping the grabbed item also raises a
series of events and property changes to inform screen readers and other
technologies about drag and drop progress. I won't go into details here
as
this forum is meant for users (I can go into details if asked privately
or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository
about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so, NVDA
must recognize the "is grabbed" state, then announce the progress of
this
operation. The first part is now part of NVDA 2022.4 (alpha builds at
the
moment), and a follow-up pull request was posted to bring the second
part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA will
be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in
Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag and
drop
state and progress (will explain why I'm saying this tip in a follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and
drop
where you must hold the mouse down, keyboard commands will force the
item
to
be dragged to somewhere in one sweep (may things happen but that's what
I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being dragged
must
support drag and drop operation and properties in a way that can be made
accessible (sometimes, the drop target control must support accessible
drag
and drop operations properly). Second, not all accessibility API's
provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in
Windows
11 if done from the keyboard. The second is the reason why accessible
drag
and drop was added in Windows 8 via UI Automation. The third reason is
why
Narrator can announce accessible drag and drop progress and events when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
|
|
|
system means not in brouser sir. regular button like what we see in
system tray or objects inside computers apart from websites.
toggle quoted message
Show quoted text
On 04/09/2022, Gene <gsasner@...> wrote: I don't know what you mean by in system.
Gene
On 9/4/2022 7:20 AM, Aravind R wrote:
yes. i was in brouse mode while using office's learning course where
we have to drag and drop objects.
in system the buttons are able to be dragged and dropped by us.
On 04/09/2022, Gene <gsasner@...> wrote:
Are you on an Internet site or somewhere else where browse mode is being
used? I don't think drag and drop can work for a blind user while in
browse mode.
Gene
On 9/4/2022 6:59 AM, Aravind R wrote:
i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
On 04/09/2022, Brian's Mail list account via groups.io
<bglists@...> wrote:
I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things,
there
is
often more than one way to do stuff, and cut and past often can be
done
instead and be a damn site less complicated. Of course I can see that
for
a
sighted person drag and drop is intuitive and obvious, but its harder
with a
screenreader as you do not have the overview that you do with a
working
pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from
the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I
can
pass on whatever I know to the next group of power users and
developers
before I pass out from graduate school. While what I'm going to write
for
a
while (in the "Explainer" series) is geared more towards NVDA
developers
and
add-on authors, I think it would be helpful to post some notes to the
users
list so at least some folks can understand how a screen reader and
related
technologies work behind the scenes (there are other reasons, some
related
to academics, others related to ongoing screen reader development,
some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than
four
years to get it right, as it will also talk about upcoming changes to
NVDA
(2022.4 to be exact), the subject being accessible drag and drop
(discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop
items?"
Among many responses (see the group archive for details), the
mouse-based
drag and drop method was discussed, as well as commands provided by
NVDA
to
manipulate the mouse in doing so. But two things were not discussed
then
)or
if discussed, receive little attention_: how do controls expose drag
and
drop capability, and can screen readers inform users about drag and
drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to
understand
how graphical user interface (GUI) controls are laid out on screen, as
well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of
the
hierarchy is the shell, or sometimes called "desktop". On top (or
below)
the
shell element are top-level windows for an app that houses all the
controls
that an app will be showing you. Below each top-level window are
smaller
windows, housing elements that will be shown on the screen most of the
time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and
eventually,
we
end up with the actual control being interacted with (button,
checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's
object
navigation principles - when you move between objects (NVDA+Numpad
arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between
windows
at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer
from
a
retailer, and whoever is responsible for delivery (perhaps Amazon,
drones,
or whatever or whoever) needs to locate where you live. The street
address
you provide to the retailer will first include the country of
residence,
followed by regions (states, provinces, and what not), itself divided
into
smaller local regions (cities, counties, etc.), in turn moving down to
the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific
thing
the camera is pointing at, and sometimes users move their camera to
focus
on
a different thing. This is how GUI elements are internally organized:
shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking
and
interacting with a person, a thing, a screen control, or an idea. As
GUI's
use visual senses to help people perform tasks with computers
(nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited
for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus
using
the keyboard, holding down the mouse, long press of touchscreens,
focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across
the
screen to the desired location (called a "drop target"). For
applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen
in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed.
Second,
accessibility API's inform anyone listening that a drag and drop
operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about
where
the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some
cases),
which in turn allows accessibility API's to inform users that drag and
drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a
drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in
question
sets "is grabbed" UIA property to True, and this gets picked up by
drag
start UIA event and announced to screen readers. While dragging items,
UIA
uses two properties to communicate where the item is going, and in
some
cases, the control that supports dropping the grabbed item also raises
a
series of events and property changes to inform screen readers and
other
technologies about drag and drop progress. I won't go into details
here
as
this forum is meant for users (I can go into details if asked
privately
or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository
about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so,
NVDA
must recognize the "is grabbed" state, then announce the progress of
this
operation. The first part is now part of NVDA 2022.4 (alpha builds at
the
moment), and a follow-up pull request was posted to bring the second
part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA
will
be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in
Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag
and
drop
state and progress (will explain why I'm saying this tip in a
follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and
drop
where you must hold the mouse down, keyboard commands will force the
item
to
be dragged to somewhere in one sweep (may things happen but that's
what
I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being
dragged
must
support drag and drop operation and properties in a way that can be
made
accessible (sometimes, the drop target control must support accessible
drag
and drop operations properly). Second, not all accessibility API's
provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen
readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in
Windows
11 if done from the keyboard. The second is the reason why accessible
drag
and drop was added in Windows 8 via UI Automation. The third reason is
why
Narrator can announce accessible drag and drop progress and events
when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
-- --
--
nothing is difficult unless you make it appear so.
r. aravind,
manager
Department of sales
bank of baroda specialised mortgage store, Chennai.
mobile no: +91 9940369593,
email id : aravind_069@..., aravind.andhrabank@....
aravind.rajendran@....
|
|
|
On Sun, Sep 4, 2022 at 07:59 AM, Aravind R wrote:
i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
- I have discussed this in the past, and in the context of saying that I have never gotten emulated drag and drop to work reliably (regardless of screen reader) when the drag and drop is not constrained to only one direction - horizontal or vertical. Dragging and dropping items in the taskbar or system tray is constrained - they can only be moved to the left or right (if the taskbar is in its default horizontal arrangement at the bottom of the screen) or up and down (if you've docked it vertically at one side of the screen or another). There is only one degree of freedom in movement. I've sometimes gotten that to work with emulated drag and drop, but still find that it's easier to use real drag and drop with a mouse (yes, even for a blind individual) with post-drop checking to determine whether the desired arrangement has been made. But when drag and drop is across two dimensions, and things like "drag a word to its definition" activities require dragging and dropping across two dimensions, both left/right and up/down involved at the same time. For myself, I've never had any luck with dragging and dropping via screen reader emulation in those situations. --
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
On Sun, Sep 4, 2022 at 07:59 AM, Aravind R wrote:
my bank's learning course where, where we have to match write statements with correct words.
- By the way, I always encourage folks who are encountering drag and drop activities like this one to report them as an accessibility issue, as they invariably become one. Something like this can be easily changed to an easier format like typing the number associated with the item in column A into the edit box for the correct item to match it in column B, or similar. Even dropdown boxes in the "to be matched to" column can be used to constrain responses only to the numbers/letters/whatever of the items in the "to be matched from" collection if that's desired. Drag and drop in this context is nothing more than the "intuitive" extension of drawing a line between items in two columns on paper, but even that can be done differently on paper for matching. --
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
Hi all,
To add to what Brian is saying: accessible drag and drop is possible if the underlying control supports it and exposes needed info to screen readers and assistive technologies, and users understand what to do in each situation (using keyboard commands, mouse clicks, etc.); that was what I'm trying to get across in my explainer. The example of a bank site where there is difficulty with drag and drop has multiple dimensions, not just availability of gestures for doing accessible drag and drop:
- Interaction medium: how are users supposed to drag and drop things on that website? Is it via mouse, or does keyboard-based alternatives exist?
- Exposing information: okay, we know that drag and drop is needed to interact with certain controls as described. But how is this information exposed not to humans, but to assistive technology applications such as screen readers? Even if this is exposed in a way that can be understood by NVDA, how is this communicated to users in the end? This question may raise points about Accessible Rich Internet Applications (ARIA) and its details (and politics surrounding it), something I won't go into (first, I'm not really an expert in ARIA, and second, parts of it is quite advanced).
About the term "system": let's just say that system may not be the right word for apps other than web browsers. I understand where this might be coming from (web centric interaction with the world). I think a better term to describe something other than "web" is "apps". I think we might as well dedicate a different thread to terminology so this thread can focus on accessible drag and drop.
Cheers,
Joseph
|
|
|
Joseph, Bless you for taking the time to explain all this. I find it quite interesting. That being said, and absolutely nothing that follows is meant to denigrate you and your work, the very nature of unconstrained drag and drop makes it an accessibility barrier. Even if everything could be done by keyboard and announced using that method would so divorce the activity from "how it's supposed to work" and likely involve so much extra verbiage in announcing drop zones that I just don't find it worth it. There are other ways to implement what is sometimes done by a visual drag and drop that can eliminate the visual element yet keep the matching element. I've actually noticed a distinct drop-off in the use of what I will call "emulated line-drawing matching drag and drop" over the course of years, and I always presumed part of this was because of reports regarding the accessibility issues it engenders. We're not talking something like replacing drag and drop in File Explorer which, by it's very nature, can be emulated via selection, cut/copy, and paste in a way that directly emulates the GUI visual equivalent but without the need for the visual elements. "Line matching" drag and drop has visual elements at its very core of operation, and should be discouraged if accessibility is wanted. --
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
Hi,
Visual drawings/drag/drop: yep, this is one of those cases that cannot be resolved by existing assistive technologies I know of, more so as the control does not expose needed info for screen reader users relying on mouse interaction, nor they might not do so despite feedback due to the nture of these apps and controls (unless a strong pressure is applied, which can yield good and bad results; anyone who've been into politics of assistive technology industry may recognize what I'm talking about, which is beyond the scope of this list).
For others: I think the case Brian provides is a good example of constraints and limitations of screen reading principles and technologies. Screen readers are not really "jack of all trades" - they live in an environment with possibilities and constraints; they must operate within the constraints imposed by the operating system, other applications, and user preferences. After all, a screen reader is an executable (or a set of executables) with many components inside, all designed to work together to inform users about screen content as much as allowed by the operating system, apps folks are using, and how much info users wnat it to announce. Our current discussion of accessible drag and drop, along with the recognition that this can be accessibility barrier, demonstrates both sides of the same coin: possibilities and verbal/nonverbal feedback on one side, situation, constraint, and barriers on the other side. Only after recognizing the need to balance between possibility and constraint can you begin to appreciate how difficult it is to design a solution for screen reader users, including, surprisingly enough, resolving things from the screen reader itself (such is my case at the moment as I balance between possibility and constraint while writing a pull request or two on drag and drop announcement for UIA controls).
I thought I discussed it a while back, but if I haven't: once drag and drop discussion is over, we might as well have a conversation (for the first time in years I think) about what goes inside a developer's head while writing screen reader pull requests, and perhaps give you a really small glimpse into politics involved in it (hopefully in a good way; a good case study is how NVDA came to support Windows Terminal and follow-up discussion between NVDA contributors and Microsoft engineers on text output).
Cheers,
Joseph
|
|
|
If you aren't in browse mode, you aren't, but the word system doesn't mean that. For clarity, saying when not in browse mode,, or using an application and not in browse mode would avoid confusion.
What you are saying about when drag and drop can be used supports my observation. I have never heard of a blind user successfully using drag and drop when in browse mode or the Virtual PC Cursor, a different name for the same thing.
I don't think drag and drop works there because you aren't dealing with the actual page, you are dealing with the page as reformatted in a buffer for screen-reader use.
I hope a way can be developed for it to work there but it may not be possible.
Gene
Gene
toggle quoted message
Show quoted text
On 9/4/2022 9:09 AM, Aravind R wrote: system means not in brouser sir. regular button like what we see in
system tray or objects inside computers apart from websites.
On 04/09/2022, Gene <gsasner@...> wrote:
I don't know what you mean by in system.
Gene
On 9/4/2022 7:20 AM, Aravind R wrote:
yes. i was in brouse mode while using office's learning course where
we have to drag and drop objects.
in system the buttons are able to be dragged and dropped by us.
On 04/09/2022, Gene <gsasner@...> wrote:
Are you on an Internet site or somewhere else where browse mode is being
used? I don't think drag and drop can work for a blind user while in
browse mode.
Gene
On 9/4/2022 6:59 AM, Aravind R wrote:
i am somehow able to drag and drop elements in system tray using
nvda's facility and its add on. But, its not working in my bank's
learning course where, where we have to match write statements with
correct words.
On 04/09/2022, Brian's Mail list account via groups.io
<bglists@...> wrote:
I have to say I have never really managed to get drag and drop to be
understandable. In the main along with many other windows things,
there
is
often more than one way to do stuff, and cut and past often can be
done
instead and be a damn site less complicated. Of course I can see that
for
a
sighted person drag and drop is intuitive and obvious, but its harder
with a
screenreader as you do not have the overview that you do with a
working
pair
of eyes.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 03, 2022 8:41 PM
Subject: [nvda] Explainer: accessible drag and drop notifications from
the
perspective of accessibility API's
=Hi all,
I understand that this is quite a strange (or more of a geeky) subject
line,
but I believe that this is the one of those last opportunities where I
can
pass on whatever I know to the next group of power users and
developers
before I pass out from graduate school. While what I'm going to write
for
a
while (in the "Explainer" series) is geared more towards NVDA
developers
and
add-on authors, I think it would be helpful to post some notes to the
users
list so at least some folks can understand how a screen reader and
related
technologies work behind the scenes (there are other reasons, some
related
to academics, others related to ongoing screen reader development,
some
related to mentoring). The below post is a result of what I might only
describe as a "celebratory week" and my wish to document most (if not
everything) of what happened to bring a bug fix that took more than
four
years to get it right, as it will also talk about upcoming changes to
NVDA
(2022.4 to be exact), the subject being accessible drag and drop
(discussed
at lengths months ago).
In April 2022, a user asked the question, "how can I drag and drop
items?"
Among many responses (see the group archive for details), the
mouse-based
drag and drop method was discussed, as well as commands provided by
NVDA
to
manipulate the mouse in doing so. But two things were not discussed
then
)or
if discussed, receive little attention_: how do controls expose drag
and
drop capability, and can screen readers inform users about drag and
drop
operation? These I will answer below.
Before discussing accessible drag and drop, it is important to
understand
how graphical user interface (GUI) controls are laid out on screen, as
well
as visual and nonvisual interaction paradigms. At a high level, GUI
elements
are organized into a hierarchy (or a tree, if you will). At the top of
the
hierarchy is the shell, or sometimes called "desktop". On top (or
below)
the
shell element are top-level windows for an app that houses all the
controls
that an app will be showing you. Below each top-level window are
smaller
windows, housing elements that will be shown on the screen most of the
time
(this is called "foreground window"). Inside these smaller windows are
another set of windows, inside are even smaller windows, and
eventually,
we
end up with the actual control being interacted with (button,
checkbox,
edit
field, web document, and so on). This is the foundation for NVDA's
object
navigation principles - when you move between objects (NVDA+Numpad
arrow
keys/NVDA+Shift+arrow keys), you are effectively moving between
windows
at
a
higher or lower level.
A useful analogy is street addresses. Suppose you order a computer
from
a
retailer, and whoever is responsible for delivery (perhaps Amazon,
drones,
or whatever or whoever) needs to locate where you live. The street
address
you provide to the retailer will first include the country of
residence,
followed by regions (states, provinces, and what not), itself divided
into
smaller local regions (cities, counties, etc.), in turn moving down to
the
street grid, and finally, the street address. Another analogy, albeit
visual
at first, is focus of a camera - you can zoom in or out of a specific
thing
the camera is pointing at, and sometimes users move their camera to
focus
on
a different thing. This is how GUI elements are internally organized:
shell
window housing top-level application windows, in turn housing smaller
windows, toolbars, emu bar and what not, in turn leading users to a
specific
control (I and so many others have discussed why this is important in
NVDA's
object navigation principles many times, so I'll skip it for now).
Another thing to keep in mind is visual and nonvisual interaction
paradigms.
Our eye (visual senses) is perhaps the first thing we use when looking
and
interacting with a person, a thing, a screen control, or an idea. As
GUI's
use visual senses to help people perform tasks with computers
(nowadays
tablets and even augmented reality (AR) tools), a paradigm best suited
for
visual manipulation of screen controls is the dominant interaction
paradigm.
These include mice, touchscreens, digitizers, eye control, and our
thoughts.
One way to interact with and manipulate elements on screen is drag and
drop.
In short, the item to be dragged is first selected (technically called
"grabbed") by using whatever input tool we use (moving system focus
using
the keyboard, holding down the mouse, long press of touchscreens,
focusing
a
bit longer on the item with our eyes, etc.), then it is dragged across
the
screen to the desired location (called a "drop target"). For
applications
of
this concept, see April 2022 archive of NVDA users list.
Even if we hold down the mouse and drag something, many things happen
in
the
background, some of which are accessibility related if conditions are
right.
First, the app informs accessibility API's that an item is grabbed.
Second,
accessibility API's inform anyone listening that a drag and drop
operation
is in progress (this differs between API's as explained below). Third,
while
dragging and dropping, accessibility API's may inform users about
where
the
item is going, sometimes helped by the host app as to where things are
going
(not all apps do this). Lastly, when a drop operation happens, the app
informs folks that drag and drop is complete (or canceled in some
cases),
which in turn allows accessibility API's to inform users that drag and
drop
is complete.
Because at least two accessibility API's are in use by Windows based
assistive technologies, the above process differs between API's:
* Microsoft Active Accessibility (MSAA/IAccessible): an attribute
change informs screen readers and other assistive technologies that a
drag
and drop operation is in progress.
* UI Automation (UIA): up to six events and six properties are defined
just for accessible drag and drop. When drag starts, the item in
question
sets "is grabbed" UIA property to True, and this gets picked up by
drag
start UIA event and announced to screen readers. While dragging items,
UIA
uses two properties to communicate where the item is going, and in
some
cases, the control that supports dropping the grabbed item also raises
a
series of events and property changes to inform screen readers and
other
technologies about drag and drop progress. I won't go into details
here
as
this forum is meant for users (I can go into details if asked
privately
or
on development-oriented forums).
A few days ago, a suggestion was posted on NVDA's GitHub repository
about
letting the screen reader announce dragging (is grabbed) state,
specifically
when performing drag and drop from the keyboard. In order to do so,
NVDA
must recognize the "is grabbed" state, then announce the progress of
this
operation. The first part is now part of NVDA 2022.4 (alpha builds at
the
moment), and a follow-up pull request was posted to bring the second
part
(drag and drop progress announcement) to NVDA. For example, if you
rearrange
Start menu tiles in Windows 10 or pinned items in Windows 11, NVDA
will
be
told that an item is being dragged (grabbed), and it can announce what
happens when you drop the item somewhere. You can rearrange tiles in
Start
menu either from the mouse or the keyboard (Alt+Shift+arrow keys).
Tip: if you are using a specific add-on, NVDA can announce the drag
and
drop
state and progress (will explain why I'm saying this tip in a
follow-up
post).
This raises two questions:
1. Why is it that NVDA will announce only part of the progress when I
use the keyboard to drag and drop items? Unlike mouse-based drag and
drop
where you must hold the mouse down, keyboard commands will force the
item
to
be dragged to somewhere in one sweep (may things happen but that's
what
I
think is the best way to explain it).
2. How come I cannot hear drag and drop announcement when dragging
things in some places? For three reasons: first, the item being
dragged
must
support drag and drop operation and properties in a way that can be
made
accessible (sometimes, the drop target control must support accessible
drag
and drop operations properly). Second, not all accessibility API's
provide
accessible drag and drop announcements and events. Third, even if
accessible
drag and drop is possible, assistive technologies such as screen
readers
must support it. The first reason is why NVDA (and for that matter,
Narrator) may not announce progress of rearranging taskbar icons in
Windows
11 if done from the keyboard. The second is the reason why accessible
drag
and drop was added in Windows 8 via UI Automation. The third reason is
why
Narrator can announce accessible drag and drop progress and events
when
NVDA
could not. until now (or rather, until 2018, improved a year later).
At least I hope this gave you an overview of behind the scenes work
required
to support accessible drag and drop.
Cheers,
Joseph
|
|
|
Joseph, In addition, accessibility software, and screen readers in particular, are all work-arounds for the sensory modality that is lacking and that they replace with another. You once said something to me that I still use to this day because it set off a light bulb in my head with regard to screen readers and web pages: Sighted users see webpages (and the world, really) as a gestalt and instantly (and subconsciously, for the most part) are applying filters so that only the visually salient items in that moment, for what you're intending to do, are what you see (literally, as far as where visual attention is directed). The best analogy to that filtering I can give is if anyone has ever moved in next to a major thoroughfare or highway or had a train track nearby where they had never experienced this before. Initially, the sound drives you crazy and you react to it. Over time, and not a lot of it, you start to filter out that sound from being attended to because you know it's not a threat and not salient to whatever it is you're trying to do at the moment. It becomes "less than background noise" and is not even heard, in any realistic sense, for all practical intents and purposes. A screen reader cannot do this because it has no idea what its user's intention is as far as what's being accessed and cannot impose the kinds of filters the human brain routinely does. There are also some things that are utterly sui generis to the sensory modality that processes them. Those here who are formerly sighted can appreciate just how impossible it is to communicate anything meaningful about what color actually is, how it is experienced, to those who have never been able to directly experience it. It's a complete abstraction with no visceral meaning if you've never been able to process that modality. It's akin to trying to tell someone who is completely deaf (and always has been) what a flute sounds like. There are no words that can ever adequately capture what any sensory modality does for the external stimuli it processes. Picture description software, while great at giving the overall sense of the picture, does not even come close to communicating all of the visual detail that is present in a picture. It couldn't. The old saw, "A picture is worth a thousand words," is an understatement. This is why I "hate" what I've called line-drawing matching drag and drop [and it often does involve a line being drawn when you do the match, just as you'd be drawing one on a paper version of the same task]. The visual modality is at the core, not the periphery, of that activity. Trying to capture where you are in 2D space and what's on one end of the match (what was grabbed) versus the drop zone you're over is just hellishly difficult when it comes to using words and where you don't necessarily have any idea of where the drop zones even begin and end. This is just not a technique that anyone who has the slightest concern about creating an accessible activity should even consider using, for even one second. --
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
Hi,
Ah yes, I remember... Essentially, information blackout, or rather, intentional blackout and intentional foci (plural of "focus"). In some ways, I'm rediscovering the "academic" side of this thread this semester (critical theory; won't go into details here).
For folks new to the forum: the last few messages between I and Brian could be best described as defining the term "information blackout" or "information tuning" - a concept where intentionally or not, computer users focus on a specific information, the immediate context being how screen reader users (or for that matter, assistive tech users) are (sometimes forced to) focus on a specific part of the whole information stream (screen content) and think that's all there is to it. I use the term "blackout" intentionally - it might be possible that a more important piece of information is present but not interacted with or intentionally (or unintentionally) hidden or obscured. You can call it "filtering," but filtering implies that you have an awareness of other pieces of screen content present and you are willing to dive into specifics - I am talking about the tendency of screen reader users to think whatever they are focused on is so important to notice that a much important message has appeared somewhere. For sighted folks (which is the mainstream, something I think we must admit even though folks may wish to deny it), the information is apparent as they can scan the screen content; not easily so for blind people as they must rely on alternatives. In application to the current context (drag and drop), there are places where accessible (keyboard-based and other medium) drag and drop is possible with constraints as it is really up to the control in question, software developers, and mindset of screen reader vendors to "restore or reformat information". And as Brian stated, there are simply places where there will be "perpetual information blackout" i.e. remain inaccessible and unusable by specific audiences, or a "starry night" where there is faint hope or a wish unless the larger public (app developers, to start with) realize the effect of such a blackout and offer alternatives (after all, the term "assistive technology" implies use of alternatives to access something while taking constraints into account).
To sum up and to bring up something I implied in my first post in this thread (as a follow-up): part of the reason for posting an explainer on accessible drag and drop has to do with Windows App Essentials add-on. Back when Windows 10 first came out (2015), an NVDA user asked if it is possible to announce text that Narrator says when rearranging Start menu tiles. I didn't have an answer back then, but after looking up UIA events, I learned that it raises events when things are dragged on screen. I added detection routines for drag events in April 2018 as part of Windows App Essentials, but it will not be until late 2019 that I added drop target events support to let NVDA announce dragged item and its new position albeit by faking a focus event. Then in 2021 I posted a pull request to NVDA project about drag and drop, but a discussion arose regarding drop target announcement and if focus event was a suitable experience for users (in short, I brought add-on contents to NVDA Core itself). Then graduate school happened and I gave up on the initial pull request until about a month ago when I heard reports about ability to rearrange Taskbar icons in Windows 11 and how inaccessible this was for keyboard users. Then two weeks ago, Quinn (the amazing producer of NVDA YouTube videos) asked on Windows App essentials GitHub repo if I can work on drag and drop announcement for NVDA after informing me that it works after installing the add-on. After doing some digging into UI Automation documentation from Microsoft, I realized that I've been using the wrong solution for the last three years, and after some experimentation, finally found the solution that's more stable and resolves the question posed by an anonymous NVDA user all those years ago. This happened last week, hence my statement about "celebratory week," and I'm happy to report that this stable solution will be part of Windows App Essentials 22.10 (late September to early October) and parts are incorporated into NVDA as of 2022.4 alpha (I'm working on remaining pieces and posted pull requests last week); of course, if you are using Windows App Essentials development builds, the solution is already there (NVDA will indeed announce drag and drop effects (progress) but if and only if this is done from UIA controls and the control supports it (hence the constraint).
Hope this answers lots of questions (and a huge credit and thank you to this anonymous NVDA user and Quinn for making me realize that I and my source code were wrong for all these years).
Cheers,
Joseph
|
|
|
On Sun, Sep 4, 2022 at 01:03 PM, Joseph Lee wrote:
For sighted folks (which is the mainstream, something I think we must admit even though folks may wish to deny it), the information is apparent as they can scan the screen content; not easily so for blind people as they must rely on alternatives.
- Which is a fact that you ignore or deny at your own peril. Windows (and all GUIs) are directly dependent on possession of functional vision if they are to be used as "natively intended," which is via the visual modality. The G, graphical, notes that as the first word. When you are any minority (in some dimension, mine is being a gay man) you have to acknowledge that you are just that. You live in "the {insert majoritarian adjective as appropriate here} world," and you're going to have to conform yourself to aspects of that whether you'd prefer to or not and to advocate for yourself throughout your life. And that's ignoring intentional discrimination. There are things that are "just different" that have nothing whatsoever to do with any intentional or unintentional attempt to discriminate against. Some of these can never be bridged in any meaningful way, either, whether one wished to do so or not. Graphical User Interfaces are designed, from their inception, with vision being the primary sensory modality on which they rely to communicate information to the user and for the user to interact with them. As soon as you remove vision from the equation, you have removed a lot of information that cannot be efficiently communicated in words, and sometimes removed all information that cannot be meaningfully communicated in words to someone who's never had vision. As an example of that, it makes no sense for me to ever tell a blind user to, "Hit the red X button at the upper right of the window." They cannot see where the window frame is (and the window may not be maximized) and if they've always been blind "red" as a qualifier is not of any real use to discriminate upon. I'd say, "Hit ALT + F4 to close the window," instead, as that's got all the context that's necessary and can be communicated directly, not relying on anything spacial or visual. Color coding is used, heavily, to instantly communicate things about classes, states, and the like to someone who can see and we take it all in in one fell visual swoop of the screen. That has no equivalent if you don't have vision as it can't be communicated or absorbed in the same way, or something even close to it. You hear those things and their classes, states, etc., as you encounter them while navigating. That's really, really not the same, and it can't be fixed nor should it be fixed, as it serves "the world at large" quite well, and that's what it needs to do. Even if one could take this aspect out, what would be the justification for doing so? Making things less immediately accessible for someone with vision does not make them more accessible for someone who is blind in a very great many cases. No matter how good accessibility for a given thing might be, you still will never have 100% of the information that can be, and is, communicated without speaking by certain visual elements. --
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
Brian's Mail list account
Yes I was reading a sighted instruction form a cd creator the other day which suggested that you had an explorer window and the app window open side by side and dragged files across effectively program boundaries. Of course that was not possible but cut and paste did work. I do often feel that when folk write instructions for things, a few seconds suggesting alternative methods might save a lot of head scratching.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
toggle quoted message
Show quoted text
----- Original Message ----- From: "Brian Vogel" <britechguy@...> To: <nvda@nvda.groups.io> Sent: Sunday, September 04, 2022 4:02 PM Subject: Re: [nvda] Explainer: accessible drag and drop notifications from the perspective of accessibility API's On Sun, Sep 4, 2022 at 07:59 AM, Aravind R wrote:
my bank's learning course where, where we have to match write statements
with correct words.
- By the way, I always encourage folks who are encountering drag and drop activities like this one to report them as an accessibility issue, as they invariably become one. Something like this can be easily changed to an easier format like typing the number associated with the item in column A into the edit box for the correct item to match it in column B, or similar. Even dropdown boxes in the "to be matched to" column can be used to constrain responses only to the numbers/letters/whatever of the items in the "to be matched from" collection if that's desired. Drag and drop in this context is nothing more than the "intuitive" extension of drawing a line between items in two columns on paper, but even that can be done differently on paper for matching. -- Brian - Windows 10, 64-Bit, Version 21H2, Build 19044 It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection. ~ Irving Babbitt
|
|
|
Brian's Mail list account
Yes I'd agree with all that. One particular council, a year or so back put in a dragand drop of a red pin to indicate on a map where a fly tip existed for clearing up.
Quite obviously, the person writing the page had never considered a blind user. Instead the site admin put in one of these overlays that claim to be everything to everyone with a disability, which resulted on hardly any of the rest of the site working either. This shows a complete misunderstanding of the issues. In the end we visually impaired users convinced the admin to ditch the overlay and simply add a basic multi input combo box for the nearest street address to be typed in. Problem solved.
However this is straying a bit off nvda here, and judging by seeing what screenreaders do, there will never be a solution to this problem until the makers of software and web content learn about accessibility even at a rudimentary level. Learn from Windows and make provision for other ways of doing the same thing.
Brian
--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.
toggle quoted message
Show quoted text
----- Original Message -----
From: "Brian Vogel" <britechguy@...>
To: <nvda@nvda.groups.io>
Sent: Sunday, September 04, 2022 5:18 PM
Subject: Re: [nvda] Explainer: accessible drag and drop notifications from the perspective of accessibility API's
Joseph,
In addition, accessibility software, and screen readers in particular, are all work-arounds for the sensory modality that is lacking and that they replace with another.
You once said something to me that I still use to this day because it set off a light bulb in my head with regard to screen readers and web pages: Sighted users see webpages (and the world, really) as a gestalt and instantly (and subconsciously, for the most part) are applying filters so that only the visually salient items in that moment, for what you're intending to do, are what you see (literally, as far as where visual attention is directed). The best analogy to that filtering I can give is if anyone has ever moved in next to a major thoroughfare or highway or had a train track nearby where they had never experienced this before. Initially, the sound drives you crazy and you react to it. Over time, and not a lot of it, you start to filter out that sound from being attended to because you know it's not a threat and not salient to whatever it is you're trying to do at the moment. It becomes "less than background noise" and is not even heard, in any realistic sense, for all practical intents and purposes. A screen reader cannot do this because it has no idea what its user's intention is as far as what's being accessed and cannot impose the kinds of filters the human brain routinely does.
There are also some things that are utterly sui generis to the sensory modality that processes them. Those here who are formerly sighted can appreciate just how impossible it is to communicate anything meaningful about what color actually is, how it is experienced, to those who have never been able to directly experience it. It's a complete abstraction with no visceral meaning if you've never been able to process that modality. It's akin to trying to tell someone who is completely deaf (and always has been) what a flute sounds like. There are no words that can ever adequately capture what any sensory modality does for the external stimuli it processes. Picture description software, while great at giving the overall sense of the picture, does not even come close to communicating all of the visual detail that is present in a picture. It couldn't. The old saw, "A picture is worth a thousand words," is an understatement.
This is why I "hate" what I've called line-drawing matching drag and drop [and it often does involve a line being drawn when you do the match, just as you'd be drawing one on a paper version of the same task]. The visual modality is at the core, not the periphery, of that activity. Trying to capture where you are in 2D space and what's on one end of the match (what was grabbed) versus the drop zone you're over is just hellishly difficult when it comes to using words and where you don't necessarily have any idea of where the drop zones even begin and end. This is just not a technique that anyone who has the slightest concern about creating an accessible activity should even consider using, for even one second.
--
Brian - Windows 10, 64-Bit, Version 21H2, Build 19044
It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.
~ Irving Babbitt
|
|
|
