Re: A few thoughts: Web Aim survey, quantity versus quality, feeling burnt out and tutorials
toggle quoted messageShow quoted text
Thank you Joseph and merry Christmas to you too.
From: firstname.lastname@example.org [mailto:email@example.com] On Behalf Of Joseph Lee
Sent: Saturday, December 23, 2017 12:26 PM
Subject: [nvda] A few thoughts: Web Aim survey, quantity versus quality, feeling burnt out and tutorials
Dear NVDA community,
As I read messages on recent discussions, I realized just how much enthusiasm and concern people have over NVDA and its future. At the same time, it became clear to me that I and other developers and community elders need a day off and just listen to you all, as listening allows us to think about what others are saying and plan things accordingly.
But first, a humble opinion about surveys and other points:
First, when calls for the seventh Web AIM survey went out, I told people to not just do it to “increase” market share. I specifically told screen reader companies to not coerce users to do it, but let people take it out of their own willingness. This advice was to avoid a fiasco that happened with Web AIM 6 where AI Squared (now part of VFO) staff told Window-Eyes users to fill out the survey in mass numbers, which became a small controversy within the screen reading world, and to me, making Web AIM results no longer credible.
As some folks pointed out, Web AIM numbers depend on how many people fill it out and where they come from (and this is true of any surveys where word of mouth drives participation). The results also depend on demographics and other factors such as choices given, how the questions are worded, and overall objective. If one or more data points seems to be dominant, they can be either skewed or outliers, with the more extreme cases being termed “outliers” and they affect how the results are explicated (interpreted). Even skewed data, such as what I can perceive from some surveys including recent Web AIM iterations can affect statistical calculations to a point where it raises genuine questions about bias, credibility, and others (after all, success of statistics, particularly inferential statistics, depends on a representative sample or a close equivalent that allows researchers to approximate the real world, which is prone to errors if not done correctly such as misinterpretation, bad outliers, only some groups participating, not looking at things more deeply and what not).
One important thing to note is that Web AIM is a representative survey, thus the result in front of me could reflect reality. However, due to recent controversy, possible type I and II errors (false-positive and false-negative, respectively) and because of outliers and skewed data and participation, it does not truly reflect actual data, which is a point some folks here are trying to say and I concur with. My explication of Web AIM 7 is that, in some parts of the world, JAWS for Windows is more popular. However, given the fact that not all geographical regions are represented, I’d counter by saying that this is not a true representative sample that includes every continent, and if it did, the story would be different and will reflect reality a bit better (not a lot because there are other ways of skewing data such as filling it out on behalf of an organization, robotic fillers and so on). Coupled with the fact that Web AIM went through a major controversy recently that damaged its credibility somewhat, I would dare not trust Web AIM results again.
This leads to my second point: quantity versus quality. If NV Access went straight for quantity alone, they could have implemented all possible feature requests in hopes of boosting market share. The reality in front of us says otherwise: not all feature requests are here. Numerous factors contribute to this problem:
Most of these point to quality, not quantity alone. In summary, quantity is important, but quality is just as important as how many people download NVDA 2017.4 between Christmas and New Year.
Lastly, in regards to organization internals, I’d like to address something I really wanted to say for the past few weeks: sometimes, I felt burnt out. My initial response to your enthusiasm over my audio tutorials was that I’ll ask for justifications for producing an updated version, seeing that there are countless free videos and tutorials out there. This was partly because I truly felt burnt out with academics, speech and debate competitions and what not (especially after a debate regarding a potential feature held not long ago), at one point telling myself that I’ll retire from the NVDA community sooner than later and feeling as though I carried important burdens on my shoulders. But you didn’t see that justification post; instead, I posted links to where you can download the 2018 version of my audio tutorial series. In effect, I’ve given up my Christmas holidays for this community, knowing that I needed a time to listen to you all and do something about it. All this was possible because of a simple act of listening and thinking about what the community means to me and what my work means to everyone. I’m committed to finishing Welcome to NVDA 2018 series before NVDA 2018.1 ships, with several addenda coming after that, all because of support from this community and outsiders. And I promise again: The Welcome to NVDA 2018 series was, and will remain, free for all. All I ask of you in return is donate to a cause that makes equal access to technology possible, especially during this holiday season and beyond.
I’m sure for many of you, my musings above are a bit hard to digest. Now you know why I don’t trust Web AIM survey results, quality is just as important as quantity, and read a confession from a community leader on his inner feelings. But there are two more things you need to know, something all of us needs to think about:
Community leaders won’t stay with you forever. In early 2017, I sensed that a long-time NVDA developer would leave this community for something better. Only I and others didn’t know until summer that it would be Jamie moving onto Mozilla Foundation.
I also felt, back in early 2017, that my active time with the NVDA community is slowly drawing to a close. I don’t know when it’ll happen, but I’ve been laying foundations for the next generation of developers and enthusiasts to take the lead. This is one of the reasons for setting up the devlearning subgroup, because I felt it is time for me and other leaders to teach NVDA internals and other concepts to the next group of community leaders and developers so they can bring NVDA to the next level and do more amazing things than I and others did (in my case, for the past five years).
Lastly, I sense a time when this community will face a sharp divide to a point where people will start questioning the merits of this community. I only told a select few earlier because it wasn’t right for me to disclose it early and for them to prepare a plan. The screen shade debate is, in fact, a sort of a preview of what is to come. One of the fundamental questions you will face at that time will be whether you still have your first love for NVDA, and whether you still have your original reasons for joining this community. The survival of this community at that time will depend on your ability to unite to face a difficult situation, even if that means facing possible splits. One thing you should NOT do at that time: ignoring new users and outside critics, because they are influential opinion leaders and are key stakeholders in NVDA’s future. One thing you SHOULD do though: listen to others and think critically.
Hope this makes sense.
Merry (early) Christmas,