In most cases, most of us will only have the default dictionary, with occasional use of a voice dictionary, but you can, very rarely, be using the temporary dictionary for testing purposes.

I have never thought about it before, but since I know that NVDA dictionary processing always goes through all dictionary entries, passing a replaced string on down the line through the remaining dictionary entries for possible subsequent replacement, the order in which the matching occurs can be critical to figuring out your final result.

I would presume there must be some standard order, like default, followed by voice, followed by temporary in all cases, but the standard ordering could be with any one of those dictionaries in the first, second, and third positions.

Does anyone know what that processing order is?
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881

Brian Vogel wrote:

I have never thought about it before, but since I know that NVDA dictionary processing always goes through all dictionary
entries, passing a replaced string on down the line through the remaining dictionary entries for possible subsequent replacement, the order in which the
matching occurs can be critical to figuring out your final result.I would presume there must be some standard order, like default, followed by voice,
followed by temporary in all cases, but the standard ordering could be with any one of those dictionaries in the first, second, and third positions.Does
anyone know what that processing order is?

That's a bloody good question, and really should be specified in the manual. It isn't.

In fact, the manual doesn't even describe the cascade of rules effect at all, which is a gross oversight.

I feel a pull request coming on.

Luke

On Thu, Jan 12, 2023 at 11:38 PM, Luke Davis wrote:

In fact, the manual doesn't even describe the cascade of rules effect at all, which is a gross oversight.

-
That design itself is a gross oversight, in my opinion.  Dictionary processing should drop out when a match and replacement occurs.  That replacement should not be subject to subsequent "match and replace" processing through the remainder of the dictionaries.  That way lies madness in a very great many cases, particularly if regular expression matching is being used (or even "anywhere" matching).

I believe I opened an issue about this several years ago, after Quentin posted a simple example of the madness that resulted from that "drop through" additional matching and replacing being applied repeatedly.  I think of it as semi-recursion, and it's insane.

I'm certain I posted about this, but am not in the mood at the moment to do an archive search to try to find it.  I need to do that so I can get a permanent link I can use, but not now.
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881

On Jan 12, Brian Vogel wrote:

Does anyone know what that processing order is?

Temporary, voice, default, built-in.

Luke

Found the github issue I opened, which appears to have been acknowledged as a bug based on the tag that's on it.  But it's been open since 2020:

Dictionary processing, as currently implemented, does not stop once the first match has been made and iterative processing produces wildly unexpected results

I actually used Quentin's example in this bug report.  I bumped it last fall [October 2022] and Quentin bumped my bump saying users still wanted a fix.  Nothing since then.
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881

On Fri, Jan 13, 2023 at 12:03 AM, Luke Davis wrote:

Temporary, voice, default, built-in.

-
Thanks.

Not that I doubt you, but how did you determine this (this quickly)?  I never even think about the built-in.
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881

Luke Davis wrote:

On Jan 12, Brian Vogel wrote:

Does anyone know what that processing order is?

Temporary, voice, default, built-in.

I sent that prematurely.

The above is the order in which they are applied, according to a note in the dictionary handler module.

Thus, rules in temporary are applied first.
Then the voice rules are applied, and so on.

So if you have a rule that replaces something in temporary, the result of that replacement may be changed by a rule in voice or built-in.
I'd have to think about it some more, but that seems upside-down to me, except for the case of built-in, which makes sense to apply as a final step.

That's aside from the cascading recursion within each dictionary itself.
If they are going to have this Bonkers setup, there should be re-order (move up, move down) buttons in the dictionary dialogs.

I'm trying to figure out if this recursion is even intended behavior; that module has a lot of old code in it, and almost none of it is documented properly.

Luke

On Fri, Jan 13, 2023 at 12:32 AM, Luke Davis wrote:

That's aside from the cascading recursion within each dictionary itself.
If they are going to have this Bonkers setup, there should be re-order (move up, move down) buttons in the dictionary dialogs.

-
I think I said the same thing, but even I think that is not sufficient.

These dictionaries have as their central purpose making NVDA say what the user wants it to say, and I don't know of a single user, including myself, who would ever think to themselves:  If it matches this first, substitute this, then run that through the rest of the dictionaries where there could be a subsequent second, third, fourth, and so on match and substitute.  I'd say that doing this makes the potential for an absolute mess way more likely, and it's not hard to force messes to happen, if you know about this, but most users don't know (and really shouldn't know) about this.

Dictionary processing should work on a match, substitute, drop out basis.  It's what makes the most sense and what most people implicitly expect, even if they can't express it in the way codes do.

Switch/case statement type processing with an exit at each and every branch after the processing takes place.
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881

This hasn't happened to me more than a small number of times, but I have had the experience where I couldn't change something using the default dictionary and I could using the voice dictionary.  With the order given, it doesn't sound as though that should be caused by this bug, but am I wrong and could it?  Though I was pleased I found a work around, I didn't like using the voice dictionary for a change I wanted to be universal.

Also, since built-in is the last, it sounds to me as though there are times you can't change something if it is already in the built-in dictionary, which shouldn't be the case.  If I want to use something else, NVDA shouldn't force what I place in other dictionaries to be ignored.

Gene

Luke Davis wrote:

So if you have a rule that replaces something in temporary, the result of that replacement may be changed by a rule in voice or built-in.

The note says:
dictTypes = ("temp", "voice", "default", "builtin") # ordered by their priority E.G. voice specific speech dictionary is processed before the default

Testing this by using the same replacement text in all three editable dictionaries, indicates that temp entries are indeed the ones that reach the synth, even if there are identical entries in voice or default.

That is as one would expect.

However, any replacement is recursive--the text created by the temp dictionary can further be altered by an entry in the default dictionary, set to key off of that text.

Not to mention the other recursion, of also being modifiable by further (or earlier?) entries in the temp dictionary itself.

How does Jaws handle all of this?

Luke

Hi,

Luke provided a high-level overview, so if you want that one, read that one. This post wil go into what gets done at the source code i.e. slightly lower level:

Speech dictionary handler module (speechDictHandler) went through many changes in the beginning (around 2008 and 2009). It was originally a single Python file but was split into multiple files in late 2017 as part ofa a change to support versioned voice dictionaries (even I can't get my head around that at the moment).

Internally, speech dictionaries are lists, each entry representing a speech dictionary dialog entry. Each entry records the original text pattern, its replacement, and replacement rule: anywhere, whole word, regular expression. While NVDA is running, it will look up four lists to determine if a pattern and its replacement exists; in fact, as far as NVDA's speech dictionary processing algorithm is concerned, speech entries are a collection of regular expressions.

When each speech dictionary is loaded, NVDA will process entries by reading the actual dictionary file content. Each line in the dictionary file is a tab-separated values of original text pattern, its replacement, case sensitivity (0 = off, 1 = on), and entry type (0 = anywhere, 1 = regular expression, 2 = whole word). After parsing each entry, NVDA will precompile regular expressions for the replacement text, used later during actual speech dictionary processing.

When a text is given via speechDictHandler.processText function, NVDA (or rather, Python) will loop through lists of entries found in the temporary dictionary (stored in RAM), voice specific dictionary, default dictionary, and a built-in dictionary appropriately named builtin.dic containing three entries designed to process specific cases with uppercase characters and numbers, in that order. For each dictionary list, Python will loop through the entire list, keeping invalid entries as it goes. For each entry, Python will attempt to match the text given with the replacement, substituting parts of speech text with replacements, if any. Back in process text function, as patterns are looked up, text will be updated if matches are found, and eventually, the processed text will be returned. Process text function is called from speech modules whenever text is encountered.

In reality, process text function's job is not to return processed speech dictionary text early - it is designed to "mold" text based on patterns and replacements supplied by dictionary lists. Here's how it actually works:

1. NVDA encounters text.
2. Before speaking the encountered text, NVDA will call speechDictHandler.processText, passing the just acquired text.
3. In process text function, the text will be fed to the temporary dictionary to see if a pattern is found, in turn fed to replacement regular expression patterns, if any.
4. Regardless of results (parts of text replaced by patterns or not), the text is updated to hold whatever text found through temporary dictionary.
5. Steps 3 and 4 are repeated, this time the updated text is fed to the voice dictionary, default dictionary, and the built-in dictionary, with the text being molded (rather, mutated) based on additional patterns found in other dictionaries.
6. The resulting text with zero or more replacements is then returned to speech modules for further processing (next stop: character processing, this time to deal with punctuation and symbol pronunciation).

For example, suppose the word "text" is to be replaced by test", and the text is "text processing can be complicated if the text contains hard to understand replacement text patterns." Depending on pattern look up:

1. No pattern: original text is spoken.
2. Found in one dictionary: all occurrences of the text "text" wil be replaced by "test" throughout the entire text (result: test processing can be complicated if the test contains hard to understand replacement test patterns).
3. Found in multiple dictionaries: text will be updated based on whatever pattern is specified at that moment.

To make matters complicated, suppose the word "text" is replaced by:

1. Default dictionary: testy
2. Voice dictionary: textt (extra t)
3. Temporary dictionary: textpattern

It then becomes:

1. Original text: text processing can be complicated if the text contains hard to understand replacement text patterns
2. Temporary dictionary: textpattern processing can be complicated if the textpattern contains hard to understand replacement textpattern patterns
3. Voice dictionary: texttpattern processing can be complicated if the texttpattern contains hard to understand replacement texttpattern patterns
4. Default dictionary: testytpattern processing can be complicated if the testytpattern contains hard to understand replacement testytpattern patterns

Entry 4 is what is eventually returned.

Cheers,

Joseph

The more I think about this, the more I understand the reasoning behind dictionary cascading.

Presumably the built-in rules, are those that NV Access uses to correct pronunciation issues and similar, so they should always be the last pass, and should always be applied, even to replacements.

For the other three, it's currently the case that temporary rule outputs are modified by voice rules; and temporary and voice rule outputs are modified by default rules.

Voice rules are intended to be those rules that the user has found necessary to modify the particular synth/voice to sound as the user wishes.
Effectively, it's the same as built-in rules, but for specific voices.

In theory those might include something like saying someone's name correctly. So you might have a voice rule that changes the pronunciation of "Rhea" to "Ray" for eSpeak, which says it wrong.

So I can understand the argument for always applying those, even to replacements from other dictionaries. But I don't understand why default comes *after* voice rules in the current processing order.

Based on the way it works, I suspect that the idea was *supposed* to be, that you can have a default entry that effects all voices, and a more generalized voice-specific entry that effects only one voice.
But if that's true, voice rules should process the output of default rules, not the other way around.

I think when replacements happen, it is considered that these would be new words, and might need fine-tuning by built-in or voice entries again, or maybe even by default entries.

Consider that you use a replacement word in a dictionary entry, that Microsoft Dave always says wrong, but all your other voices say correctly.

Let's try a ridiculous example.

Let's say that Microsoft Dave, always says "cups" as "Cuban Pesos".

And let's say that your job involves editing a database of recipes, where a number of cups is generally written like "1C" or "3.5C".
Well that's hard to distinguish from "1.5se", which for some reason you also run into all the time.

So you create a regular expression entry in the default dictionary, that transforms any number of "C", into that number of " cups".

If you read the phrase "2C", it is read as "2 cups".
All well and good, except on Microsoft Dave, where dozens of times per day, you now have to hear:
"2 Cuban Pesos"

If we have recursion of dictionary processing, you could have a voice dictionary entry for Microsoft Dave, which converts "cups" to, for example, "cuups", or something else that actually pronounces close enough to correctly.

If we don't have dictionary recursion, how do you solve this problem?

I think you would need one entry that says "cuups" everywhere, or you would need an individual voice dictionary entry for every voice you use, telling them each how to individually pronounce "2C".

Obviously this is a goofy example, but you might be able to imagine larger real-world cases where this could be a serious problem.

To solve problems like this, the re-processing of the result of dictionary replacements seems reasonable.

I still question the reason for running default *after* voice though.

Luke

I can certainly see a usage case for it to work this way, but I don't think this particular method should be the default.

I'll grab an updated source tree (mine is several releases behind), and take a look at fixing this behavior.  It should be a relatively simple fix, assuming all is written well, it's either a matter of breaking out of a loop, or if it isn't coded that way, then by setting a flag, and having the rest of the dictionaries skip processing if the flag is set.

Honestly, I'm not all that familiar with the NVDA codebase, but I'd guess this fix won't take more than a few minutes to implement.  Kind of tough, especially for the bug having been around as long as it has.

No, I disagree.  I believe reversing the order in which the dictionary entries are processed would solve this problem.

Also, if something is in the temporary dictionary, it's because the user isn't liking something (perhaps only during use of program number 2265, but all other times, nobody cares what that says, because it's correct.  Therefore, when program 2265 is running, the user can load a temporary dictionary to say the word properly, and the problem is solved, but not if the default dictionary changes it again.

And, besides, the order of processing should be from the most rigid to the least, and built-in is about as rigid as it gets. That fix would take a bit more time, but I doubt a change like that would be approved.

The end user is the one that ultimately should control the way something is pronounced.  That means that if something is in the built-in dictionary, it should be allowed to be replaced by the default, then the voice, then the temporary, because if something's in the temporary dictionary, it's because it's failed to be correctly spoken in the first place, or the user wouldn't have it there, therefore, they should be processed in reverse order of now if the replacements are going to flow down through all of the dictionaries.  Alternatively, keep the order as it is now, but abort processing when it's found in any dictionary.  That way, the user still maintains the ultimate control over how things get pronounced.

Nobody should have to fight the screen reader just to get things pronounced correctly.  I can't tell you how many hours of productivity I've lost during programming sessions, just because the synthesizer said something that wasn't there.

I know most folks don't agree with me, but I'm very strongly of the opinion that it's the screen reader's job to read the screen. It's my job to interpret what those items mean.  I understand why it's done the way it is, but I completely disagree with it.

Hi,

A more elegant solution is using a chained map where a list of dictionaries will be employed to look up words (this will result in fundamental rewrite to speech dictionary processing facility). The biggest benefit is short-circuit - the winning dictionary will be the one with the entry defined, with the biggest downside being that, for each dictionary, whatever comes in later will replace older entries (that's how dictionaries work). Effectively, a chained map approach changes speech dictionary processor from a mutator to a retriever.

A less drastic solution is offering a setting in NVDA's speech settings panel to change dictionary priority. The advantage is that users can change speech dictionary processing order, with the downside being that it becomes a bit harder to troubleshoot speech processing issues. If implemented, this will require informing speech dictionary handler to respond to configuration profile changes and change processing order accordingly. At the GUI side, there is a handy control called "rearrange list" but the user experience for this control is, let's just say, not very user-friendly at the moment (rearrange list comes from wxWidgets/wxPython).

Cheers,

Joseph

Hi,

The other purpose of temporary dictionary is to experiment with something odd before committing it to the appropriate dictionary. Also, keep in mind that NVDA will process the replaced text further down the chain, and it is perfectly possible to put a filter somewhere (filter is an extension point where the caller can filter text based on specific rules, mostly used in speech processing) to customize the resulting spoken text.

Cheers,

Joseph

That may be the purpose in the minds of the designers but there is at least one more very useful purpose.

there may be changes you want to make in one context.  The Nation magazine recently changed its web site so that blind users hear distracting and unnecessary speech.  When you move from article to article to see what is being offered, you hear speech such as title, before the title, description before the teaser is read, and name when you are on the author name line.

This is so distracting and annoying that, before using the site, I use the temporary dictionary to have none of these words spoken.  Then, when I'm finished, I quit NVDA and run it again to remove the entries.

This description may or may not influence any changes in the order dictionaries are processed, though I've read the thread, I haven't paid a lot of attention since the first number of entries, but I'm offering this description for those who haven't thought of using it in this way who might want to and in case it makes any difference in how dictionaries may be treated if changes are made in the order in which they process entries.

Gene

Then we get the question, where should I alter things to catch al whatever it is that I want?
I have used the dictionaries to catch the Word Alexa in its many forms and replace it by Lady A, so as not to set mine off when I'm reading documents it it in. I keep on finding cases where it is still saying it and after a while you tend to lose the will to live having forgotten how you did it and why its not working!

Brian

--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.

Also, how do you avoid endless recursions?
It sounds like you could accidentally create a loop.
Brian

--
bglists@...
Sent via blueyonder.(Virgin media)
Please address personal E-mail to:-
briang1@..., putting 'Brian Gaff'
in the display name field.

On Fri, Jan 13, 2023 at 01:31 AM, Luke Davis wrote:

How does Jaws handle all of this?

-
I honestly don't know.  But JAWS does not have Regex matching, either.

This has been a great, and enlightening, discussion.  I will never, however, change my opinion that the "remaining cascade" method currently used is in any way, shape, or form a good idea.  I've been on "the front end" of the issues it creates on multiple occasions, and no one, and I do mean no one, expects them and it's most often a nightmare of "dictionary picking" to figure out what in the hell just happened.

I'll gladly take any problems that come from, "I've found a match, I've done a replacement, I'm done - now," processing.
--

Brian - Virginia, USA - Windows 11 Pro, 64-Bit, Version 22H2, Build 22621; Office 2016, Version 16.0.15726.20188, 32-bit

It is much easier to be critical than to be correct.

       ~ Benjamin Disraeli, 1804-1881