The Inside Story of NVDA: introduction and overall goals and mindset #NVDA_Internals


 

Hi all,

First, thank you to Brian V for giving me permission to do something I’ve been dreaming about for the last few years: giving you a tour of NVDA screen reader internals. For the last few years, I wished I could take some time to tell you how a screen reader works from the inside, as well as add a much needed body of knowledge to screen reader research.

When you do a forum archive search for terms such as “internals” and “development”, you will come across posts from yours truly and others talking about writing a series of articles on screen reader internals. To quote a few:

And we also had messages that became something outside of NVDA but nevertheless touched parts of NVDA such as:

nvda@nvda.groups.io | Article on Screen Reader History (including NVDA)

 

I’ll return to the history of NVDA in a future post, this time using artifacts that are not really screen reader related but important to explain choices made over the years.

The reason for writing The Inside Story of NVDA series is to explain how a screen reader works, specifically how NVDA works behind the scenes. Part of this series stem from the fact that NVDA, despite being an open-source screen reader, is not documented well at the source code level. This is perhaps one of the biggest reasons for difficulty experienced by people wishing to contribute code to make the screen reader better and more valuable. Better because of the notion of competition and needing to support different scenarios, more valuable because of perceived notion that more features mean more value and power (I can tell you right now that this is false; you’ll notice that I can and will sometimes become philosophical; as Brian V and others have observed (including in a thread on accessible drag and drop, screen readers are filled with workarounds for workarounds for workarounds for inaccessible computing; I’m not joking). At least that’s the reality of screen reader development, and I hope this series sheds some light on that reality, first by talking about internals at a high level, then take you on a journey of how such and such feature actually comes to life (I cannot go into line-by-line commentary on NVDA source code as this is not a development list, but I’ll do my best to at least illustrate what’s going on in a way that I hope users can appreciate).

The second reason for posting NVDA internals is to serve as a healing process, both for me and for the entire NVDA community. It wasn’t until taking graduate school classes that I realized that I was so focused on NVDA development to a point where I felt burnout. This manifested this year when I announced my month-long break as I needed some time to heal and reorganize my thoughts. Lack of documentation in NVDA source code did contribute to my stress, but what made it worse was my realization that users don’t know a lot about how their favorite screen reader works behind the scenes which is understandable. Making matters urgent was the fact that I am a graduate student and am standing on a crossroad, knowing that my involvement with NVDA community is coming to a close, and I felt this is the golden opportunity to pass on what I know about NVDA and screen reading in general to the next group of community members before I move on. Since writing seems to be one of the ways to get off stress and heal, I felt writing about NVDA internals will help me heal, and also to lift some of the veil off NVDA screen reader for the community so people can better understand what’s going on and suggest changes for the better.

The third reason for writing the Inside Story series is to teach you the art and science of technical communication. I’m personally glad that we have people such as Quinn who are the embodiment of technical communication: using multimedia presentations, storytelling, and other innovative ways to discuss technical concepts, including screen reading in a way that can be understood by many. Technical communication (or technical writing in general) is an artistic and scientific process. Artistic because the author needs to look at who the audience is and come up with ways to explain difficult concepts in a friendly way. Scientific because one must show rigor in research and understanding (including feedback from the audience), otherwise explanations fall apart. I hope that others can contribute their internals posts to enrich the community while practicing the art and science of technical communication.

The fourth reason for the Inside Story series is to leave future researchers with something to think about. Studying assistive technology at the source code level and understanding its internals was a dream for many. It wasn’t until recently that people began to appreciate the research value of software source code. But because it takes time to understand source code (especially for folks unfamiliar with the primary audience for which the source code is designed (computer users)), more so for assistive tech source code, I figured this is the perfect time to bring the researcher in me to life and leave something behind for future researchers to consider. I figured that a story or two about NVDA internals may provide some hints to researchers on where to look next.

Finally, I dedicate Insider Story series to countless people who have helped me in my journey as a member of the NVDA community. Special thank you to NV Access folks who have been my mentors and coworkers on the journey toward equal access to technology. I hope the upcoming Inside Story series can serve as a way to showcase my learning for the last ten years.

As The Inside Story of NVDA will talk about NVDA screen reader internals from user’s point of view, I’ll do my best to minimize use of jargon (or if I must, I’ll define what they are along the way). Even if storytelling is the primary mode of delivery, I expect that readers have some knowledge of computers, and a special bonus to people familiar with accessibility concepts and technology surrounding it such as accessibility standards and API’s, familiarity with programming, and willingness to dive a bit deeper into technical side of things. My primary audience member is an NVDA user who have been using the screen reader for some time (at least one year) and is curious about how features work, both at the high-level (how it appears to you) and slightly lower level (quite closer to the source code level) but not up to the point of coding the screen reader itself. Sometimes I will show you parts of source code that is essential for inner workings of a feature with explanations afterwards. And since NVDA is continuously being developed, what I say versus what is actually added to source code may differ. Lastly, this is not a development list, so the level of explanation is not a line by line commentary, but more towards a story or two of how things come to life (for an example, see my explanation on how browse mode is loaded).

The first story in Inside Story series will deal with high-level overview of components of a screen reader. This is in response to posts like the one in August where we had a discussion about math expression pronunciation where it was observed that it ultimately came down to speech synthesizers, not with the screen reader itself. I hope the upcoming post will clarify (once and for all) that screen readers are not text-to-speech engines and vice versa. As always, contact me directly if you have things you would like me to cover throughout the Inside Story series.

Thanks.

Cheers,

Joseph


Brian's Mail list account
 

One thing I would say here is that with source code if you want to make any sense of it at all, you need to have all punctuation on as you read it, as the syntax of code is very strict most of the time these days. Back in my days of being young and foolish, Basic had line numbers and all kinds of fixed parameters, but gradually as more was needed from code, especially when was a stepping stone toward compiling machine language native code, the concepts and types of data have expanded to suit the environment.
Brian

--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.

----- Original Message -----
From: "Joseph Lee" <joseph.lee22590@...>
To: <nvda@nvda.groups.io>
Sent: Saturday, September 17, 2022 10:09 PM
Subject: [nvda] The Inside Story of NVDA: introduction and overall goals and mindset #nvdaint


Hi all,

First, thank you to Brian V for giving me permission to do something I've
been dreaming about for the last few years: giving you a tour of NVDA screen
reader internals. For the last few years, I wished I could take some time to
tell you how a screen reader works from the inside, as well as add a much
needed body of knowledge to screen reader research.

When you do a forum archive search for terms such as "internals" and
"development", you will come across posts from yours truly and others
talking about writing a series of articles on screen reader internals. To
quote a few:

* Proposal: a series of posts/articles on NVDA internals (groups.io)
<https://nvda.groups.io/g/nvda/message/91943?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3A
Created%2C%2Cinternals%2C20%2C2%2C0%2C88497918>
* Re: Please update Enhanced Phonetic Reading App (groups.io)
<https://nvda.groups.io/g/nvda/message/97563?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3A
Created%2C%2Cinternals%2C20%2C2%2C0%2C92512544>
* Re: control names (groups.io)
<https://nvda.groups.io/g/nvda/message/92420?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3A
Created%2C%2Cinternals%2C20%2C2%2C0%2C88777986>

And we also had messages that became something outside of NVDA but
nevertheless touched parts of NVDA such as:

* nvda@nvda.groups.io | Tutorial On Using NVDA Single Letter
Navigation to navigate faster than even a sighted person
<https://nvda.groups.io/g/nvda/topic/87700584#90855>
*

nvda@nvda.groups.io | Article on Screen Reader History (including NVDA)
<https://nvda.groups.io/g/nvda/topic/92394151#97806>



I'll return to the history of NVDA in a future post, this time using
artifacts that are not really screen reader related but important to explain
choices made over the years.

The reason for writing The Inside Story of NVDA series is to explain how a
screen reader works, specifically how NVDA works behind the scenes. Part of
this series stem from the fact that NVDA, despite being an open-source
screen reader, is not documented well at the source code level. This is
perhaps one of the biggest reasons for difficulty experienced by people
wishing to contribute code to make the screen reader better and more
valuable. Better because of the notion of competition and needing to support
different scenarios, more valuable because of perceived notion that more
features mean more value and power (I can tell you right now that this is
false; you'll notice that I can and will sometimes become philosophical; as
Brian V and others have observed (including in a thread on accessible drag
and drop <https://nvda.groups.io/g/nvda/topic/93446323#99221> , screen
readers are filled with workarounds for workarounds for workarounds for
inaccessible computing; I'm not joking). At least that's the reality of
screen reader development, and I hope this series sheds some light on that
reality, first by talking about internals at a high level, then take you on
a journey of how such and such feature actually comes to life (I cannot go
into line-by-line commentary on NVDA source code as this is not a
development list, but I'll do my best to at least illustrate what's going on
in a way that I hope users can appreciate).

The second reason for posting NVDA internals is to serve as a healing
process, both for me and for the entire NVDA community. It wasn't until
taking graduate school classes that I realized that I was so focused on NVDA
development to a point where I felt burnout. This manifested this year when
I announced my month-long break
<https://nvda.groups.io/g/nvda/topic/91379776#96670> as I needed some time
to heal and reorganize my thoughts. Lack of documentation in NVDA source
code did contribute to my stress, but what made it worse was my realization
that users don't know a lot about how their favorite screen reader works
behind the scenes which is understandable. Making matters urgent was the
fact that I am a graduate student and am standing on a crossroad, knowing
that my involvement with NVDA community is coming to a close, and I felt
this is the golden opportunity to pass on what I know about NVDA and screen
reading in general to the next group of community members before I move on.
Since writing seems to be one of the ways to get off stress and heal, I felt
writing about NVDA internals will help me heal, and also to lift some of the
veil off NVDA screen reader for the community so people can better
understand what's going on and suggest changes for the better.

The third reason for writing the Inside Story series is to teach you the art
and science of technical communication. I'm personally glad that we have
people such as Quinn who are the embodiment of technical communication:
using multimedia presentations, storytelling, and other innovative ways to
discuss technical concepts, including screen reading in a way that can be
understood by many. Technical communication (or technical writing in
general) is an artistic and scientific process. Artistic because the author
needs to look at who the audience is and come up with ways to explain
difficult concepts in a friendly way. Scientific because one must show rigor
in research and understanding (including feedback from the audience),
otherwise explanations fall apart. I hope that others can contribute their
internals posts to enrich the community while practicing the art and science
of technical communication.

The fourth reason for the Inside Story series is to leave future researchers
with something to think about. Studying assistive technology at the source
code level and understanding its internals was a dream for many. It wasn't
until recently that people began to appreciate the research value of
software source code. But because it takes time to understand source code
(especially for folks unfamiliar with the primary audience for which the
source code is designed (computer users)), more so for assistive tech source
code, I figured this is the perfect time to bring the researcher in me to
life and leave something behind for future researchers to consider. I
figured that a story or two about NVDA internals may provide some hints to
researchers on where to look next.

Finally, I dedicate Insider Story series to countless people who have helped
me in my journey as a member of the NVDA community. Special thank you to NV
Access folks who have been my mentors and coworkers on the journey toward
equal access to technology. I hope the upcoming Inside Story series can
serve as a way to showcase my learning for the last ten years.

As The Inside Story of NVDA will talk about NVDA screen reader internals
from user's point of view, I'll do my best to minimize use of jargon (or if
I must, I'll define what they are along the way). Even if storytelling is
the primary mode of delivery, I expect that readers have some knowledge of
computers, and a special bonus to people familiar with accessibility
concepts and technology surrounding it such as accessibility standards and
API's, familiarity with programming, and willingness to dive a bit deeper
into technical side of things. My primary audience member is an NVDA user
who have been using the screen reader for some time (at least one year) and
is curious about how features work, both at the high-level (how it appears
to you) and slightly lower level (quite closer to the source code level) but
not up to the point of coding the screen reader itself. Sometimes I will
show you parts of source code that is essential for inner workings of a
feature with explanations afterwards. And since NVDA is continuously being
developed, what I say versus what is actually added to source code may
differ. Lastly, this is not a development list, so the level of explanation
is not a line by line commentary, but more towards a story or two of how
things come to life (for an example, see my explanation on how browse mode
is loaded
<https://nvda.groups.io/g/nvda/message/97305?p=%2C%2C%2C20%2C0%2C0%2C0%3A%3A
Created%2C%2CNVDA+says%2C20%2C2%2C20%2C92274149> ).

The first story in Inside Story series will deal with high-level overview of
components of a screen reader. This is in response to posts like the one in
August where we had a discussion about math expression pronunciation
<https://nvda.groups.io/g/nvda/topic/93127550#98750> where it was observed
that it ultimately came down to speech synthesizers, not with the screen
reader itself. I hope the upcoming post will clarify (once and for all) that
screen readers are not text-to-speech engines and vice versa. As always,
contact me directly if you have things you would like me to cover throughout
the Inside Story series.

Thanks.

Cheers,

Joseph


Pranav Lal
 

Joseph,

 

You may want to put these stories on the web perhaps on a wiki on the nvaccess github repository. E-mail works too but tends to get lost.

 

Pranav


Russell James
 

Hi Joseph,

This sounds very interesting, I've always wondered what was under the covers of NVDA.

Is there architectural/design documentation that can provide a framework on which we can place the stories?

Thank you

Russ


On Thu, Sep 22, 2022 at 9:52 PM Pranav Lal <pranav@...> wrote:

Joseph,

 

You may want to put these stories on the web perhaps on a wiki on the nvaccess github repository. E-mail works too but tends to get lost.

 

Pranav


 

Hi,

To answer a few questions:

  • Blog or a wiki: perhaps in the future.
  • Architectural document: there is a technical design overview document that's part of NVDA source code's dev docs folder (use to be on the NVDA project wiki collection but moved to source code directly in a recent release). The next Inside Story will be mostly based on that document as I'll be talking about NVDA components with a few source code examples.

Cheers,

Joseph