The Inside Story of NVDA: what a screen reader is and is not, possibilities and constraints of screen readers #NVDA_Internals


 
Edited

Hi all,

Before we actually get into talking about NVDA components, it is important to think about what a screen reader is and is not, as well as overall concepts (and the big picture) behind possibilities and constraints of screen readers. We also need to go over accessibility in relation to screen reading. Only then the rest of the Inside Story posts will make sense because the story begins and ends with defining the reality, possibilities, and constraints of screen reading technology (for anyone wishing to submit code contributions to NVDA project, you need to think about the overall social and cultural reality NVDA and its users are facing).

First, let’s talk about what a screen reader is not. A screen reader is not an operating system, nor the user interface for an operating system. It is not a “jack of all trades” productivity tool, nor the only way for blind people to use computers (although screen readers get lots of attention because they are one of the most familiar tools the society will see). A screen reader is not your accessibility advocate, nor designed to bring disability justice to everyone. Most importantly, a screen reader is not the million-dollar answer to everything in life, blindness, and accessibility. Shocking? I assume so (for most of us).

The truth is, I sometimes feel that a screen reader is one or more of the “nots” I listed. Folks on this forum encounter and live with screen readers 24 hours a day, 7 days a week, 365 (or 366) days a year. And screen readers like NVDA are gaining more and more mainstream attention (do a Google search for the terms “accessibility” and “screen readers” and one of the results is an article from The Verge published not so long ago on the subject of screen reader history; the NVDA forum had an extensive talk about it a while back). We use screen readers in many places: schools, companies, accessibility testing, software development, or even as an example of progress of accessibility.

So what exactly is a screen reader? Among many Google searches, the common theme is that it is a program that helps blind people use computers by reading screen content. More specifically, a screen reader is a program that reads content the user is interacting with (or not). Sometimes the content is accessible and usable (both terms are important), while others are not, requiring tips and tricks to make them screen reader and user friendly. I will come back to what I just said in a moment.

In a more technical definition, a screen reader is an information processor that gathers, interprets, and presents information displayed on screen and provide ways to let blind users interact with the computer-based task at hand. Screen readers such as NVDA use facilities provided by the operating system (Microsoft Windows, in this case) and apps to gather information on the screen (and sometimes off-screen). Screen readers have rules and expectations about what the gathered information is and should be, and uses sophisticated rules to interpret what it has “seen” i.e. gathered with help from the operating system, the ap in question, and other ways. Based on information gathered and subsequently interpreted, screen readers use components such as text-to-speech (TTS), braille, and other output mechanisms to present screen content. I will address exactly which components are part of NVDA in the next Inside Story.

To illustrate the overall workings of a screen reader at the highest level (or not so high level), let us say that you open Notepad and type the letter “H”. On screen, the letter “H” is shown, and NVDA says “H” if speak typed characters is on (NVDA+number row 2). If a braille display is connected, it will show the letter “H” in braille (in Unified English Braille, it is dots 6 and then 125, or in this case, it could be dots 56, 6, then 125). But how can NVDA accomplish so much magic? Here’s how:

  1. User types the letter “H”.
  2. Windows realizes that something happened from the keyboard, so it tries to interpret what happened.
  3. Windows sees that a character was entered and sees where the system focus is.
  4. Windows sees that Notepad, a text editor is in use, so it displays the letter “H” on the screen.
  5. At the same time, a helper called accessibility API notices this event and sees that a character was entered.
  6. The accessibility API then tells whoever is listening (NVDA, in this case) that an input event occurred.
  7. In turn, Notepad (app) realizes that an accessibility API is running, so it says to the accessibility API, “please raise a value change event so the screen reader can announce it to the user.”
  8. In turn, the accessibility API raises value change event, which is then recognized by NVDA.
  9. NVDA knows that a value change event has occurred, so it tries to find out what has changed, and eventually sees that a new character was entered.
  10. NVDA then uses the configured speech synthesizer to inform the user that the letter “H” has been entered. This does not happen if the user says to NVDA, “don’t tell me typed characters.”

The steps listed above should provide just enough information to demonstrate the idea that a screen reader is, in essence, a sophisticated information processor: gathers, interprets, and presents information.

Going back to what I said above about accessible and inaccessible (and usable and unusable) content: what I outlined above may suggest that everything is accessible if things work out between the operating system, apps, and screen readers. This ignores the fact that screen readers are, believe it or not, workarounds to the current social and cultural conditions of computing, disability, accessibility, and usability. Remember one of the “nots” of screen readers: they are not accessibility advocates for you. Why? Think about the term “assistive technology”. What does it mean in practice? It means that computers, tablets, smartphones, and gadgets we live with are not designed with disability in mind, and screen readers came along to “fill” the gap for inaccessible and unusable computing. The history of screen readers is filled with slogans such as “equal access to technology”, “making things more productive”, “helping blind people get jobs” and others (the story of screen readers goes back decades, believe it or not).

The term “assistive technology”, at a first glance, is a positive definition for folks on this forum and elsewhere: tools to help you succeed in using computers to perform tasks. But on the flip side, it shows that, despite progress such as accessibility standards and novel approaches to provide “technological social justice” (disability emojis, for example), the world is still, for a lack of better word, unconcerned (or not educated enough or not fully aware of, perhaps) toward blind people. Screen readers exist precisely because they demonstrate the lack of consideration for the disabled when designing digital technologies, and as we will see in subsequent Inside Story of NVDA series, people like Mick Curran and others came up with workarounds upon workarounds to demonstrate the continued need for advocacy.

My statement that screen readers are workarounds should ring a bell for some of you. Not just because your life experiences are filled with accessibility advocacy, but also because it touches on one of my own mantras about accessibility and usability: mindset matters. Fixing inaccessible applications so it can become screen reader user friendly is just a micro-level solution. The steps I listed to demonstrate parts of NVDA internals came after years of advocacy by blind people, informing Microsoft that they need to do better (people who lived in the 1990’s should remember what I’m talking about). Accessibility standards and API’s are next level up in solving computing issues for screen reader users (by doing so, people and organizations writing standards are acknowledging the continued issues faced by disabled people thanks to larger social and cultural issues at hand). The fundamental issue, and the reason that NVDA is not the million-dollar answer to everything in life for screen reader users, is the perpetuation of ignorance by both sides of the coin: ignorance by the public (mainstream) that accessibility and usability matters in software design, and ignorance by screen reader users and disability advocacy organizations that we are a minority and must advocate continuously.

Putting all into context of NVDA, just because the screen reader is free and open-source does not mean equal access to technology is here at last. When you use NVDA or contribute code to the project, you are doing three things at once: shows dedication to the project, acknowledges the progress made in screen reading, and understands the effects of social and cultural attitude toward disability. The last one is the reality of screen reading as it stands in 2022: even if COVID-19 pandemic made us realize how screen readers are important for us, it also brought challenges such as inaccessible and unusable videoconferencing systems, unreadable online documents, and the notion that technology can solve world’s problems (it won’t, I think). When looking at NVDA from the big picture of accessibility and usability, it opens up possibilities and constraints. Possibilities because the code is out there so people can study and research it, and constraints as the same source code demonstrates the larger social and cultural issues faced by blind people. This is perhaps the biggest lesson I want readers to understand as we meet NVDA internals: screen readers such as NVDA represent the reality, possibilities, and constraints of people needing to use alternatives due to social and cultural attitudes. And throughout Insider Story series, I will highlight all three of them as much as possible.

Remember this: screen readers are not productivity tools, the solution to life’s problems, technological social justice, nor can advocate for users. As sophisticated information processors, screen readers represent the reality, possibilities, and constraints of disability in the form of technology. NVDA both shows the progress and waypoint toward accessibility and usability, and in extension, more need for disability advocacy. I want all of you to understand this, otherwise the rest of The Insider Story of NVDA will not make sense – not only I will take you on a journey on NVDA internals, but also help you contemplate a lot (for anyone wishing to contribute code to NVDA project, you must have the mindset that you are contributing to both the possibilities and constraints of accessibility and disability).

Next: NVDA screen reader components and/or any feature you would like me to cover (comments and topic suggestions are welcome).

Cheers,

Joseph


 

On Sun, Sep 18, 2022 at 05:10 PM, Joseph Lee wrote:
But on the flip side, it shows that, despite progress such as accessibility standards and novel approaches to provide “technological social justice” (disability emojis, for example), the world is still, for a lack of better word, hostile toward blind people.
-
Actually, I have to take issue with this, as there are better words.  Hostility implies intent on one side, and that intent is to injure, hurt, etc.  I'd say the better phrase is, "fundamentally unconcerned with the needs of," rather than, "hostile toward."

I don't think "the world at large" is actively, or even passively (for the most part), hostile toward those who are blind. They just don't think about them at all.  Accusing any individual or group of hostility will put up certain obstacles that need not occur if that accusation is avoided.  Given the size of the blind community as part of the whole of humanity, it's somewhat easy to see how this is a somewhat natural side effect.  It's a very small minority among humanity as a whole.

While the end result is what could accurately be called a "hostile digital environment," where hostile means, "very unfavorable to life or growth."  But that hostile digital environment is not, absolutely not, the result of active hostility toward those who are blind.  Not being considered due to relative invisibility (which is what's going on, still, with the general public) is not the same as being attacked, assailed, or similar.  And that's the reason that advocacy remains critical, still.

But in the time I've been in computing (starting in the early 1980s) when accesibility was not even really a concept yet, and certainly not taught, to where we are now and the path things are on, is a tectonic shift.  And I credit all of that shift to advocacy from the blind community for the blind community combined with the willingness of certain key players in the sighted world to pay attention. Accessibility principles are now taught at the university level which is where "the future of computing" has always had its roots.  It's "baked in" far more often these days rather than being very clumsily retrofitted. 

What the "general public" thinks about this is not a central issue here and it's completely unrealistic to believe that the general public as a collective whole will ever pay any significant attention to accessibility in computing, mostly because they have nothing to do with implementing it and have no need for it themselves.  I think that far too often people fail to take what is called Hanlon's Razor to heart:  Never attribute to malice that which is adequately explained by stupidity.
I would, however, substitute "ignorance" for stupidity.  And considering that circles straight back to why I can't and don't consider the broader world to be actively hostile toward the blind.  They just don't consider you all at all, for the most part.  The probability of that changing to any great extent among the general public, ever, is quite small.  Catching "the right ears" when advocating is key.

As the regular in these parts who's fully sighted I am definitely "the odd man out," in the NVDA community.  The fact that I have proficiency with NVDA (and JAWS, to a lesser extent these days) and other assistive technology makes me the same in the sighted community.  While I don't claim to speak for all sighted people, I believe my perspectives on computing and accessibility cannot be dismissed, as they have not been formed hastily nor in an information vacuum on all sides.
--

Brian - Windows 10, 64-Bit, Version 21H2, Build 19044  

It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.

       ~ Irving Babbitt


 

Hi,

Ah, thanks for correcting me (edited the original post).

Cheers,

Joseph


Gene
 

I am not making this comment because I think the essay should be changed.  But I think a slogan like equal access to technology, which sounds good and which you encounter often in terms of advocacy, is misleading and somewhat meaningless.  What is equal access?  You can't have equal access as a blind user because you are not accessing the technology in the same way and sight provides more information and faster when dealing with computer information if it is visual.  Obviously, I'm not talking about streaming something that is only audio content.  But a sighted person can look at a screen and find something much more quickly some of the time than a blind person can.  If a blind person is already familiar with an interface or knows enough what he/she is looking for, the person may find content as quickly or perhaps faster than a sighted person but there are many times when this is not the case and the blind person finds what is being looked for less efficiently.

I'm sure that, in an unfamiliar dialog, a sighted person can skim what is there and find what they are looking for, if they have an idea what they are looking for because of what they already know about how a certain type of program or dialog works, than a blind person who tabs and listens to field after field.  In a known and familiar dialog, the blind person, through use of shortcuts may do something as fast or faster than a sighted person.

And that brings up something else screen-readers are not.  They are not ways to enable you to use a program without putting in the time and work to learn enough about the interface to use it efficiently. 

Gene

On 9/18/2022 4:10 PM, Joseph Lee wrote:

[Edited Message Follows]
[Reason: Terminology]

Hi all,

Before we actually get into talking about NVDA components, it is important to think about what a screen reader is and is not, as well as overall concepts (and the big picture) behind possibilities and constraints of screen readers. We also need to go over accessibility in relation to screen reading. Only then the rest of the Inside Story posts will make sense because the story begins and ends with defining the reality, possibilities, and constraints of screen reading technology (for anyone wishing to submit code contributions to NVDA project, you need to think about the overall social and cultural reality NVDA and its users are facing).

First, let’s talk about what a screen reader is not. A screen reader is not an operating system, nor the user interface for an operating system. It is not a “jack of all trades” productivity tool, nor the only way for blind people to use computers (although screen readers get lots of attention because they are one of the most familiar tools the society will see). A screen reader is not your accessibility advocate, nor designed to bring disability justice to everyone. Most importantly, a screen reader is not the million-dollar answer to everything in life, blindness, and accessibility. Shocking? I assume so (for most of us).

The truth is, I sometimes feel that a screen reader is one or more of the “nots” I listed. Folks on this forum encounter and live with screen readers 24 hours a day, 7 days a week, 365 (or 366) days a year. And screen readers like NVDA are gaining more and more mainstream attention (do a Google search for the terms “accessibility” and “screen readers” and one of the results is an article from The Verge published not so long ago on the subject of screen reader history; the NVDA forum had an extensive talk about it a while back). We use screen readers in many places: schools, companies, accessibility testing, software development, or even as an example of progress of accessibility.

So what exactly is a screen reader? Among many Google searches, the common theme is that it is a program that helps blind people use computers by reading screen content. More specifically, a screen reader is a program that reads content the user is interacting with (or not). Sometimes the content is accessible and usable (both terms are important), while others are not, requiring tips and tricks to make them screen reader and user friendly. I will come back to what I just said in a moment.

In a more technical definition, a screen reader is an information processor that gathers, interprets, and presents information displayed on screen and provide ways to let blind users interact with the computer-based task at hand. Screen readers such as NVDA use facilities provided by the operating system (Microsoft Windows, in this case) and apps to gather information on the screen (and sometimes off-screen). Screen readers have rules and expectations about what the gathered information is and should be, and uses sophisticated rules to interpret what it has “seen” i.e. gathered with help from the operating system, the ap in question, and other ways. Based on information gathered and subsequently interpreted, screen readers use components such as text-to-speech (TTS), braille, and other output mechanisms to present screen content. I will address exactly which components are part of NVDA in the next Inside Story.

To illustrate the overall workings of a screen reader at the highest level (or not so high level), let us say that you open Notepad and type the letter “H”. On screen, the letter “H” is shown, and NVDA says “H” if speak typed characters is on (NVDA+number row 2). If a braille display is connected, it will show the letter “H” in braille (in Unified English Braille, it is dots 6 and then 125, or in this case, it could be dots 56, 6, then 125). But how can NVDA accomplish so much magic? Here’s how:

  1. User types the letter “H”.
  2. Windows realizes that something happened from the keyboard, so it tries to interpret what happened.
  3. Windows sees that a character was entered and sees where the system focus is.
  4. Windows sees that Notepad, a text editor is in use, so it displays the letter “H” on the screen.
  5. At the same time, a helper called accessibility API notices this event and sees that a character was entered.
  6. The accessibility API then tells whoever is listening (NVDA, in this case) that an input event occurred.
  7. In turn, Notepad (app) realizes that an accessibility API is running, so it says to the accessibility API, “please raise a value change event so the screen reader can announce it to the user.”
  8. In turn, the accessibility API raises value change event, which is then recognized by NVDA.
  9. NVDA knows that a value change event has occurred, so it tries to find out what has changed, and eventually sees that a new character was entered.
  10. NVDA then uses the configured speech synthesizer to inform the user that the letter “H” has been entered. This does not happen if the user says to NVDA, “don’t tell me typed characters.”

The steps listed above should provide just enough information to demonstrate the idea that a screen reader is, in essence, a sophisticated information processor: gathers, interprets, and presents information.

Going back to what I said above about accessible and inaccessible (and usable and unusable) content: what I outlined above may suggest that everything is accessible if things work out between the operating system, apps, and screen readers. This ignores the fact that screen readers are, believe it or not, workarounds to the current social and cultural conditions of computing, disability, accessibility, and usability. Remember one of the “nots” of screen readers: they are not accessibility advocates for you. Why? Think about the term “assistive technology”. What does it mean in practice? It means that computers, tablets, smartphones, and gadgets we live with are not designed with disability in mind, and screen readers came along to “fill” the gap for inaccessible and unusable computing. The history of screen readers is filled with slogans such as “equal access to technology”, “making things more productive”, “helping blind people get jobs” and others (the story of screen readers goes back decades, believe it or not).

The term “assistive technology”, at a first glance, is a positive definition for folks on this forum and elsewhere: tools to help you succeed in using computers to perform tasks. But on the flip side, it shows that, despite progress such as accessibility standards and novel approaches to provide “technological social justice” (disability emojis, for example), the world is still, for a lack of better word, unconcerned (or not educated enough or not fully aware of, perhaps) toward blind people. Screen readers exist precisely because they demonstrate the lack of consideration for the disabled when designing digital technologies, and as we will see in subsequent Inside Story of NVDA series, people like Mick Curran and others came up with workarounds upon workarounds to demonstrate the continued need for advocacy.

My statement that screen readers are workarounds should ring a bell for some of you. Not just because your life experiences are filled with accessibility advocacy, but also because it touches on one of my own mantras about accessibility and usability: mindset matters. Fixing inaccessible applications so it can become screen reader user friendly is just a micro-level solution. The steps I listed to demonstrate parts of NVDA internals came after years of advocacy by blind people, informing Microsoft that they need to do better (people who lived in the 1990’s should remember what I’m talking about). Accessibility standards and API’s are next level up in solving computing issues for screen reader users (by doing so, people and organizations writing standards are acknowledging the continued issues faced by disabled people thanks to larger social and cultural issues at hand). The fundamental issue, and the reason that NVDA is not the million-dollar answer to everything in life for screen reader users, is the perpetuation of ignorance by both sides of the coin: ignorance by the public (mainstream) that accessibility and usability matters in software design, and ignorance by screen reader users and disability advocacy organizations that we are a minority and must advocate continuously.

Putting all into context of NVDA, just because the screen reader is free and open-source does not mean equal access to technology is here at last. When you use NVDA or contribute code to the project, you are doing three things at once: shows dedication to the project, acknowledges the progress made in screen reading, and understands the effects of social and cultural attitude toward disability. The last one is the reality of screen reading as it stands in 2022: even if COVID-19 pandemic made us realize how screen readers are important for us, it also brought challenges such as inaccessible and unusable videoconferencing systems, unreadable online documents, and the notion that technology can solve world’s problems (it won’t, I think). When looking at NVDA from the big picture of accessibility and usability, it opens up possibilities and constraints. Possibilities because the code is out there so people can study and research it, and constraints as the same source code demonstrates the larger social and cultural issues faced by blind people. This is perhaps the biggest lesson I want readers to understand as we meet NVDA internals: screen readers such as NVDA represent the reality, possibilities, and constraints of people needing to use alternatives due to social and cultural attitudes. And throughout Insider Story series, I will highlight all three of them as much as possible.

Remember this: screen readers are not productivity tools, the solution to life’s problems, technological social justice, nor can advocate for users. As sophisticated information processors, screen readers represent the reality, possibilities, and constraints of disability in the form of technology. NVDA both shows the progress and waypoint toward accessibility and usability, and in extension, more need for disability advocacy. I want all of you to understand this, otherwise the rest of The Insider Story of NVDA will not make sense – not only I will take you on a journey on NVDA internals, but also help you contemplate a lot (for anyone wishing to contribute code to NVDA project, you must have the mindset that you are contributing to both the possibilities and constraints of accessibility and disability).

Next: NVDA screen reader components and/or any feature you would like me to cover (comments and topic suggestions are welcome).

Cheers,

Joseph



 

On Sun, Sep 18, 2022 at 08:26 PM, Gene wrote:
You can't have equal access as a blind user because you are not accessing the technology in the same way and sight provides more information and faster when dealing with computer information if it is visual.
-
Gene,

You and I are in close to absolute agreement about this.  But, and it's an important but, it's "as equal as can currently be achieved" when it comes to written content.  

If you want equal to be "equal to having vision," then that will never be achieved.  I've said it many times, and have no hesitation about doing so:  Screen readers are workarounds that substitute one sensory modality, audition/hearing, for another vision.  There is no way to do this without something being "lost in translation" even if the only thing (and it's usually not the only thing) is speed.  The same would be true in reverse.  And there are things, like picture description, that will never, ever, ever, come close to being able to transmit the information an image contains, in its totality, that can be and is transmitted by seeing it.  Just as you cannot say that describing something as "a flute playing," or "a piano playing," carries the same information, whether to a deaf or hearing person, as actually hearing it does.  There are things that are sui generis to each of our sensory modalities that defy any conversion.  So, that's another reason there can never be "perfection of equality."

The above being said, the equality of access to written material is really about as close as you're ever going to get to equal.  Slogans/tag lines are not meant to be perfect expressions that capture every nuance of an idea (or the limitations it entails, either).
--

Brian - Windows 10, 64-Bit, Version 21H2, Build 19044  

It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.

       ~ Irving Babbitt


Brian's Mail list account
 

I think the biggest loss and the one most program and web site developers cannot get their heads around is the over view issue. Yes you can list headers, lists links buttons and interactive areas, but the mental pictures we get are different to what the sighted will have.
I find it very frustrating to be told you just need to go right from where you are into the side bar, when a keyboard its impossible to achieve this in one movement unless you start emulating the mouse and manually shifting the focus, a task quite hard to do without accidentally triggering something else.
Brian

--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.

----- Original Message -----
From: "Brian Vogel" <britechguy@...>
To: <nvda@nvda.groups.io>
Sent: Monday, September 19, 2022 3:27 AM
Subject: Re: [nvda] The Inside Story of NVDA: what a screen reader is and is not, possibilities and constraints of screen readers #NVDA_Internals


On Sun, Sep 18, 2022 at 08:26 PM, Gene wrote:


You can't have equal access as a blind user because you are not accessing
the technology in the same way and sight provides more information and
faster when dealing with computer information if it is visual.
-
Gene,

You and I are in close to absolute agreement about this. But, and it's an important but, it's "as equal as can currently be achieved" when it comes to written content.

If you want equal to be "equal to having vision," then that will never be achieved. I've said it many times, and have no hesitation about doing so: Screen readers are workarounds that substitute one sensory modality, audition/hearing, for another vision. There is no way to do this without something being "lost in translation" even if the only thing (and it's usually not the only thing) is speed. The same would be true in reverse. And there are things, like picture description, that will never, ever, ever, come close to being able to transmit the information an image contains, in its totality, that can be and is transmitted by seeing it. Just as you cannot say that describing something as "a flute playing," or "a piano playing," carries the same information, whether to a deaf or hearing person, as actually hearing it does. There are things that are sui generis to each of our sensory modalities that defy any conversion. So, that's another reason there can never be "perfection of equality."

The above being said, the equality of access to written material is really about as close as you're ever going to get to equal. Slogans/tag lines are not meant to be perfect expressions that capture every nuance of an idea (or the limitations it entails, either).
--

Brian - Windows 10, 64-Bit, Version 21H2, Build 19044

It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.

~ Irving Babbitt


Çağrı Artan
 

Hi Joseph,
I am typing this message using a translation app. Because I am not knowledgeable enough to convey my views in spoken language. Please excuse words and phrases that may be perceived differently.
Your valuable message, which I believe will increase the possibility of finding solutions to the problems we are experiencing, shows what can be done when we take action to be a part of the solution and when ways of cooperation are opened to solve our own problems, it contains very accurate determinations.
Your message will contribute significantly to the rights-based studies I support in my country and to the digital accessibility business processes that I will be involved in as a consultant.
It is good that you and valuable minds like you are in the world. I feel happy and lucky to be able to use the technology that allows us to meet you.
 


 

On Mon, Sep 19, 2022 at 03:07 AM, Brian's Mail list account wrote:
I think the biggest loss and the one most program and web site developers cannot get their heads around is the over view issue. Yes you can list headers, lists links buttons and interactive areas, but the mental pictures we get are different to what the sighted will have.
-
And this is a problem that has no solution.  At least if you consider the problem to be, "the mental pictures we get are different to what the sighted will have."

If you have never been able to see, even the phrase "mental pictures" is misleading, as a picture/image is literally visually based.  If you've never been able to see you cannot form a mental picture.  You can, and do form a mental conception, framework, understanding, or any number of other accurate words related to how material is arranged to be worked with, but none of those is accurately termed "a picture."

Even I, as a sighted person, would not claim to have any sort of mental picture were I blindfolded and presented with a webpage, for example.  I have quite a bit of knowledge about what's on it based on what the screen reader tells me, but I have no idea where it is relative to anything else or what it literally looks like.  And I certainly don't have any idea of what the thing as a whole looks like or contains.

Joseph Lee coined the term "information blackout" (which I love and have alluded to, repeatedly) to describe the difference in the situation between what a screen reader user has immediate access to versus what a sighted person has access to when using a visually-based medium.  And I can assure you that most everything related to the web has a huge visually-based component that we who are sighted take in all at one time, as a gestalt, and filter out what's irrelevant to us at the moment beneath the level of even being conscious we've done so.  You can't have that, either, with a screen reader as the technology has no way to divine the intent of its user and it cannot make decisions about the salience of what's on a given page.

Even if AI is introduced at some point that does a very good job of dismissing with the clutter, initially anyway, you will still not be able to deal with "the whole" at one time as there's no way to do the translation from a fully laid-out page (web or otherwise) into verbal terms that can present it all at once.  That's a "lost in translation" situation where no solution exists because one sensory modality is being substituted for another.  Certain things are specific to vision, just as they are to hearing, taste, smell, and touch.  You just can't do a direct conversion from one to the other without "data loss."
--

Brian - Windows 10, 64-Bit, Version 21H2, Build 19044  

It is well to open one's mind but only as a preliminary to closing it . . . for the supreme act of judgment and selection.

       ~ Irving Babbitt


Shawn
 

Perhaps a better word than equal would be optimized. I.E. the best possible access within the limits we have to work within.

Shawn Klein

On 9/18/2022 7:26 PM, Gene wrote:
I am not making this comment because I think the essay should be changed.  But I think a slogan like equal access to technology, which sounds good and which you encounter often in terms of advocacy, is misleading and somewhat meaningless.  What is equal access?  You can't have equal access as a blind user because you are not accessing the technology in the same way and sight provides more information and faster when dealing with computer information if it is visual.  Obviously, I'm not talking about streaming something that is only audio content.  But a sighted person can look at a screen and find something much more quickly some of the time than a blind person can.  If a blind person is already familiar with an interface or knows enough what he/she is looking for, the person may find content as quickly or perhaps faster than a sighted person but there are many times when this is not the case and the blind person finds what is being looked for less efficiently.

I'm sure that, in an unfamiliar dialog, a sighted person can skim what is there and find what they are looking for, if they have an idea what they are looking for because of what they already know about how a certain type of program or dialog works, than a blind person who tabs and listens to field after field.  In a known and familiar dialog, the blind person, through use of shortcuts may do something as fast or faster than a sighted person.

And that brings up something else screen-readers are not.  They are not ways to enable you to use a program without putting in the time and work to learn enough about the interface to use it efficiently. 

Gene
On 9/18/2022 4:10 PM, Joseph Lee wrote:

[Edited Message Follows]
[Reason: Terminology]

Hi all,

Before we actually get into talking about NVDA components, it is important to think about what a screen reader is and is not, as well as overall concepts (and the big picture) behind possibilities and constraints of screen readers. We also need to go over accessibility in relation to screen reading. Only then the rest of the Inside Story posts will make sense because the story begins and ends with defining the reality, possibilities, and constraints of screen reading technology (for anyone wishing to submit code contributions to NVDA project, you need to think about the overall social and cultural reality NVDA and its users are facing).

First, let’s talk about what a screen reader is not. A screen reader is not an operating system, nor the user interface for an operating system. It is not a “jack of all trades” productivity tool, nor the only way for blind people to use computers (although screen readers get lots of attention because they are one of the most familiar tools the society will see). A screen reader is not your accessibility advocate, nor designed to bring disability justice to everyone. Most importantly, a screen reader is not the million-dollar answer to everything in life, blindness, and accessibility. Shocking? I assume so (for most of us).

The truth is, I sometimes feel that a screen reader is one or more of the “nots” I listed. Folks on this forum encounter and live with screen readers 24 hours a day, 7 days a week, 365 (or 366) days a year. And screen readers like NVDA are gaining more and more mainstream attention (do a Google search for the terms “accessibility” and “screen readers” and one of the results is an article from The Verge published not so long ago on the subject of screen reader history; the NVDA forum had an extensive talk about it a while back). We use screen readers in many places: schools, companies, accessibility testing, software development, or even as an example of progress of accessibility.

So what exactly is a screen reader? Among many Google searches, the common theme is that it is a program that helps blind people use computers by reading screen content. More specifically, a screen reader is a program that reads content the user is interacting with (or not). Sometimes the content is accessible and usable (both terms are important), while others are not, requiring tips and tricks to make them screen reader and user friendly. I will come back to what I just said in a moment.

In a more technical definition, a screen reader is an information processor that gathers, interprets, and presents information displayed on screen and provide ways to let blind users interact with the computer-based task at hand. Screen readers such as NVDA use facilities provided by the operating system (Microsoft Windows, in this case) and apps to gather information on the screen (and sometimes off-screen). Screen readers have rules and expectations about what the gathered information is and should be, and uses sophisticated rules to interpret what it has “seen” i.e. gathered with help from the operating system, the ap in question, and other ways. Based on information gathered and subsequently interpreted, screen readers use components such as text-to-speech (TTS), braille, and other output mechanisms to present screen content. I will address exactly which components are part of NVDA in the next Inside Story.

To illustrate the overall workings of a screen reader at the highest level (or not so high level), let us say that you open Notepad and type the letter “H”. On screen, the letter “H” is shown, and NVDA says “H” if speak typed characters is on (NVDA+number row 2). If a braille display is connected, it will show the letter “H” in braille (in Unified English Braille, it is dots 6 and then 125, or in this case, it could be dots 56, 6, then 125). But how can NVDA accomplish so much magic? Here’s how:

  1. User types the letter “H”.
  2. Windows realizes that something happened from the keyboard, so it tries to interpret what happened.
  3. Windows sees that a character was entered and sees where the system focus is.
  4. Windows sees that Notepad, a text editor is in use, so it displays the letter “H” on the screen.
  5. At the same time, a helper called accessibility API notices this event and sees that a character was entered.
  6. The accessibility API then tells whoever is listening (NVDA, in this case) that an input event occurred.
  7. In turn, Notepad (app) realizes that an accessibility API is running, so it says to the accessibility API, “please raise a value change event so the screen reader can announce it to the user.”
  8. In turn, the accessibility API raises value change event, which is then recognized by NVDA.
  9. NVDA knows that a value change event has occurred, so it tries to find out what has changed, and eventually sees that a new character was entered.
  10. NVDA then uses the configured speech synthesizer to inform the user that the letter “H” has been entered. This does not happen if the user says to NVDA, “don’t tell me typed characters.”

The steps listed above should provide just enough information to demonstrate the idea that a screen reader is, in essence, a sophisticated information processor: gathers, interprets, and presents information.

Going back to what I said above about accessible and inaccessible (and usable and unusable) content: what I outlined above may suggest that everything is accessible if things work out between the operating system, apps, and screen readers. This ignores the fact that screen readers are, believe it or not, workarounds to the current social and cultural conditions of computing, disability, accessibility, and usability. Remember one of the “nots” of screen readers: they are not accessibility advocates for you. Why? Think about the term “assistive technology”. What does it mean in practice? It means that computers, tablets, smartphones, and gadgets we live with are not designed with disability in mind, and screen readers came along to “fill” the gap for inaccessible and unusable computing. The history of screen readers is filled with slogans such as “equal access to technology”, “making things more productive”, “helping blind people get jobs” and others (the story of screen readers goes back decades, believe it or not).

The term “assistive technology”, at a first glance, is a positive definition for folks on this forum and elsewhere: tools to help you succeed in using computers to perform tasks. But on the flip side, it shows that, despite progress such as accessibility standards and novel approaches to provide “technological social justice” (disability emojis, for example), the world is still, for a lack of better word, unconcerned (or not educated enough or not fully aware of, perhaps) toward blind people. Screen readers exist precisely because they demonstrate the lack of consideration for the disabled when designing digital technologies, and as we will see in subsequent Inside Story of NVDA series, people like Mick Curran and others came up with workarounds upon workarounds to demonstrate the continued need for advocacy.

My statement that screen readers are workarounds should ring a bell for some of you. Not just because your life experiences are filled with accessibility advocacy, but also because it touches on one of my own mantras about accessibility and usability: mindset matters. Fixing inaccessible applications so it can become screen reader user friendly is just a micro-level solution. The steps I listed to demonstrate parts of NVDA internals came after years of advocacy by blind people, informing Microsoft that they need to do better (people who lived in the 1990’s should remember what I’m talking about). Accessibility standards and API’s are next level up in solving computing issues for screen reader users (by doing so, people and organizations writing standards are acknowledging the continued issues faced by disabled people thanks to larger social and cultural issues at hand). The fundamental issue, and the reason that NVDA is not the million-dollar answer to everything in life for screen reader users, is the perpetuation of ignorance by both sides of the coin: ignorance by the public (mainstream) that accessibility and usability matters in software design, and ignorance by screen reader users and disability advocacy organizations that we are a minority and must advocate continuously.

Putting all into context of NVDA, just because the screen reader is free and open-source does not mean equal access to technology is here at last. When you use NVDA or contribute code to the project, you are doing three things at once: shows dedication to the project, acknowledges the progress made in screen reading, and understands the effects of social and cultural attitude toward disability. The last one is the reality of screen reading as it stands in 2022: even if COVID-19 pandemic made us realize how screen readers are important for us, it also brought challenges such as inaccessible and unusable videoconferencing systems, unreadable online documents, and the notion that technology can solve world’s problems (it won’t, I think). When looking at NVDA from the big picture of accessibility and usability, it opens up possibilities and constraints. Possibilities because the code is out there so people can study and research it, and constraints as the same source code demonstrates the larger social and cultural issues faced by blind people. This is perhaps the biggest lesson I want readers to understand as we meet NVDA internals: screen readers such as NVDA represent the reality, possibilities, and constraints of people needing to use alternatives due to social and cultural attitudes. And throughout Insider Story series, I will highlight all three of them as much as possible.

Remember this: screen readers are not productivity tools, the solution to life’s problems, technological social justice, nor can advocate for users. As sophisticated information processors, screen readers represent the reality, possibilities, and constraints of disability in the form of technology. NVDA both shows the progress and waypoint toward accessibility and usability, and in extension, more need for disability advocacy. I want all of you to understand this, otherwise the rest of The Insider Story of NVDA will not make sense – not only I will take you on a journey on NVDA internals, but also help you contemplate a lot (for anyone wishing to contribute code to NVDA project, you must have the mindset that you are contributing to both the possibilities and constraints of accessibility and disability).

Next: NVDA screen reader components and/or any feature you would like me to cover (comments and topic suggestions are welcome).

Cheers,

Joseph