Add-on Updater notice: introducing Project Meteor to refactor add-on download and installation steps


 
Edited

Hi all,

Have a safe and healthy Independence Day (to folks in United States celebrating it).

The following will affect everyone using Add-on Updater add-on at some point as it amounts to rewriting significant parts of this add-on:

Of all the add-ons I have published so far, Add-on Updater is very special. Not only it is a proof of concept for an NVDA feature request, it is a necessary tool to keep add-ons up to date. While I treat Windows App Essentials with care, I take pride in seeing Add-on Updater used by thousands of users around the world.

This year marks the fourth anniversary of Add-on Updater. Released in August 2018, Add-on Updater’s core mission is to check for, download, and install add-on updates. In the course of bringing this to life, I have borrowed heavily from NVDA screen reader itself, as evidenced by ad-on download progress dialog which resembles the experience you go through when checking for NVDA updates. This is intentional as Add-on Updater was born out of a need to give the NVDA community two things: a tool to keep add-ons up to date, and a way to test NVDA issue 3208 in the form of an add-on (as folks pointed out recently, some add-on features do end up becoming part of NVDA, the most notable being screen curtain feature).

But as Add-on Updater became popular, the design decisions made in 2018 became roadblocks to adding more exciting features and increased maintenance burden. For example, due to the way add-on download facility is written, it is impossible to test the actual download internals without editing the associated progress dialog. This is more so when updating multiple add-ons, commonly seen when authors release add-ons compatible with latest NVDA release. Same goes with the actual ad-on update progress interface – it is impossible to test just the internals of add-on update mechanism without doing something to update progress dialog. More importantly, because of the way update check mechanism is structured inside the add-on, it is impossible to let Add-on Updater users obtain updates not hosted on community add-ons website, nor is prepared to handle different ways to obtain updates from community add-ons website (not here; ask add-ons list please ).

Enter Project Meteor, an effort to fundamentally restructure Add-on Updater internals. The core features of the ad-on will not change (Add-on Updater will still check for, download, and update add-ons). At a high level, Project Meteor will separate download and installation routines from the accompanying graphical user interface; in other words, it will become possible to check for, download, and update add-ons in the background without having to open NVDA menu/Tools/Check for add-on updates dialog, useful if you just want NVDA to update add-ons in the background and prompt you to restart. In addition, Project Meteor will lay the foundation for a feature some of us may have only dreamed about: accessing other add-on update sources from Add-on Updater (in order to fully realize this feature, NVDA 2022.1 or later is a must, will explain why later), which may take a while to implement. Most importantly, with Project Meteor, it will become possible to test internal add-on update routines without having to edit the associated user interface code.

The good news is that users of Add-on Updater 22.07 are already using parts of Project Meteor. Bug fixes from Add-on Updater 22.07 such as centering add-on download progress dialog were implemented while planning Project Meteor. But the latest Add-on Updater release (22.07) does not contain the core Project Meteor work – that’s coming later and parts of it are now available in the form of a try build (see below). As of time of this post, Project Meteor core is complete – add-on download and installation routines are now truly separated from progress dialogs. The foundation for checking add-on updates from different sources is ongoing.

For people wishing to test Project Meteor and provide feedback can do so with the following try build:

https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220704.nvda-addon

 

IMPORTANT NOTES:

  1. This is work in progress. Try builds are not stable add-on versions and should not be used in production environments! Please do provide feedback if you have suggestions/bugs/concerns/other comments (private message please).
  2. Before installing the linked try build, tell NVDA to not update Add-on Updater (NVDA menu/Preferences/Settings, Add-on Updater, check “Add-on Updater” from “do not update add-ons” list). This is because Add-on Updater will try to “upgrade” to 22.07 when in fact this is a downgrade.
  3. When you check for add-on updates, you will notice two things: the name of the dialog will be called “Project Meteor add-on update check”, and add-on updates list will report installed add-on versions as “0.0”. These are intentional – dialog title will let you know that you are testing a try build, and the strange version text is for compatibility with other parts of Add-on Updater. I plan to resolve these in a future try build once Project Meteor user interface matures.
  4. When downloading and installing updates (particularly when updating multiple add-ons), you will notice that instead of showing download and update progress dialogs after downloading each add-on, NVDA will show you an overall download progress dialog followed by an update dialog while installing ALL add-ons at once. This is perhaps the biggest thing you will notice in Project Meteor and is an evidence of separation of add-on download and install internals from the user interface; indeed this is the first goal of Project Meteor, and I’m happy to share the progress made so far.
  5. Speaking of downloading add-on updates, you will hear beeps similar to progress bar output. Due to the way add-on download routine works, it is impossible to show you progress bar output for individual add-on downloads – the new download dialog shows you the overall progress of add-on downloads. To compensate for it, progress beeps will be heard while individual add-ons are downloading.
  6. Add-on download and install interface changes will apply even if NVDA presents a toast message (Windows 10 and 11) about add-on updates.
  7. When installing add-on updates, NVDA may not present an error dialog even if NVDA finds errors during installation (among these are compatibility error). I will add the appropriate interface for it in a future try build.
    1. Folks reading the add-on source code will notice additional things, including a new Python module. See Project Meteor notices on NVDA Add-ons list (archive) for details on what they are.

 

I expect to release a few more try builds to showcase Project Meteor progress and to incorporate your feedback. If things go well, Project Meteor will be part of add-on Updater 22.08 (which might be the last version compatible with NVDA 2021.3).

 

Thanks.

Cheers,

Joseph


 

Hi all,

A major bug was discovered with July 4th try build resulting in inability to download add-on updates under some circumstances. This is now resolved in July 5th try build: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220705.nvda-addon

Cheers,

Joseph


 

Hi all,

I'm delighted to announce the immediate release of two try builds with fixes and a feature folks have only dreamed of:

These builds are identical apart from one major thing: where add-on update records are coming from. Yes, you read it right: if you are using the second try build, Add-on Updater will gather update information from Spanish community add-ons catalog, sometimes known as nvda-addons.org. While Add-on Updater will default to using community add-ons website (first try build), it is now possible to obtain add-on update data from other sources through a concept called "update check protocol" - a collection of tasks Add-on Updater can perform to check for updates from a variety of places. The reason for these two try builds is to test this feature thoroughly with different sources and because there is no setting in Add-on Updater settings panel to configure which source Add-on Updater should use. I do plan to add such an interface soon. Also, if you are using NVDA 2022.1 or later, you will be able to specify which source to use from the command line (coming soon).

Please remember that try builds are not really meant for production environments, but I'm giving you permission to test today's builds in stable environments for testing purposes. I hope to release a stable version of Add-on Updater with these exciting changes in August at the earliest.

Cheers,

Joseph


Clement Chou
 

This is awesome! I'm wondering though if there's going to be a way to
just have it pull from all sources to check? Or is that too much?

On 7/8/22, Joseph Lee <joseph.lee22590@...> wrote:
Hi all,

I'm delighted to announce the immediate release of two try builds with fixes
and a feature folks have only dreamed of:

* First try build:
https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220708.nvda-addon
* Second try build:
https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteorNVDAEs20220708.nvda-addon

These builds are identical apart from one major thing: where add-on update
records are coming from. Yes, you read it right: if you are using the second
try build, Add-on Updater will gather update information from Spanish
community add-ons catalog, sometimes known as nvda-addons.org. While Add-on
Updater will default to using community add-ons website (first try build),
it is now possible to obtain add-on update data from other sources through a
concept called "update check protocol" - a collection of tasks Add-on
Updater can perform to check for updates from a variety of places. The
reason for these two try builds is to test this feature thoroughly with
different sources and because there is no setting in Add-on Updater settings
panel to configure which source Add-on Updater should use. I do plan to add
such an interface soon. Also, if you are using NVDA 2022.1 or later, you
will be able to specify which source to use from the command line (coming
soon).

Please remember that try builds are not really meant for production
environments, but I'm giving you permission to test today's builds in stable
environments for testing purposes. I hope to release a stable version of
Add-on Updater with these exciting changes in August at the earliest.

Cheers,

Joseph






 

Hi,

In theory, yes. There are several downsides, however:

  • Varying update metadata formats: different add-on update sources return update metadata in different formats. For example, the metadata format used by Spanish community catalog is different than that of community add-ons website. At least this is mitigated by telling Add-on Updater to standardize around a common format independent of any source and telling update check protocols to return things in a standard way.
  • Latency: because websites are hosted around the world, it takes time for NVDA (specifically, Python) to connect to different sources and obtain what it needs. NVDA will actually appear to freeze when opening websites using Python, which is not the experience users want; fortunately, developers wrote workarounds such as using multiple threads to keep NVDA responsive even when a connection is being made to a remote location.
  • Code complexity: although I do my best to write clean code as much as possible, when dealing with things such as accessing multiple locations at once, it becomes important to think about how complex this task can become. Indeed, coordinating multiple threads (one thread per add-on source) can introduce unforeseen bugs such as freezes and unresponsive user interface.
  • Metadata conflicts: suppose you connect to two add-on sources at once, and both sources return information for the same add-on. One source says whatever version you have is the latest, while the second source may offer a newer version but incompatible with whatever NVDA version you have. Now you connect to a third source, which offers an older version of the same add-on which is compatible with the version of NVDA you have. For humans, the answer is to stay on the current version, but for computers, it becomes a question of which strategies they can learn from programmers. This is made complicated when we throw in compatibility information to the mix.
  • Future of add-ons repositories: without going deep into details, NV Access is researching (or did research) a way to host add-on metadata in a central location. Once that becomes a thing, the add-ons community will be advising folks to submit to the new central database.

Hope this explains a lot.

Cheers,

Joseph


 
Edited

Hi all,

I'm delighted to announce the immediate release of Add-on Updater July 11th try build: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220711.nvda-addon

This try build packs many goodies, including two new Add-on Updater settings and features some of you have only dreamed of. With this build, Project Meteor is complete - all I need to do is refine it based on feedback (speaking of feedback, feel free to use this try build in production environments; note that this is not a stable build yet).

Changes:

  • Various internal code changes.
  • Update add-ons in the background: yes, you read it right. If you are using Windows 10 and 11, you can now ask Add-on Updater to check for, download, and install add-on updates all on its own (in the background). A new checkbox named "Update add-ons in the background" is available to configure this feature. If you are an iPhone or Android phone user, this is similar to automatic app updates. Note that not all add-ons are optimized for it - some add-ons will present an installation dialog when updating them, and I have asked other add-on authors about what to do. The other change is that when you do review add-on updates, a simple list view will be presented instead of a list of checkboxes to tell you that add-ons were updated, and when you close add-on updates dialog, you will be prompted to restart NVDA. In case you are wondering if NVDA will tell you about automatic updates, yes it will via toast notifications.
  • Update source: building from July 8th try build, a new combo box named "Add-on update source" was added to Add-on Updater settings panel to let you select which source Add-on Updater should use when fetching add-on update information (I'm using Spanish community add-ons catalog to test this feature). For most users, community add-ons website (default source) should be sufficient unless you are using add-ons not listed there; the add-ons community members do welcome new add-ons to international community add-ons website hosted by NV Access and the NVDA community.

I expect to release one last try build in Project Meteor series later this week with bug fixes and refinements based on community feedback. If things go well, all the features I have described in this thread will become part of Add-on Updater 22.08.

Thanks.

Cheers,

Joseph


 

Hi all,

Add-on Updater Project Meteor July 12th build is now available: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220712.nvda-addon

I expect this will be the second to last try build in Project Meteor series. This try build includes four changes - two are visible, two I won't talk about on this forum (if you are curious, contact me privately please).

Changes:

  • Command-line options: Add-on Updater can now take in additional command-line options if available. This feature requires an installed copy of NVDA 2022.1 or later. In fact, in order to take advantage of this feature and other updates, an upcoming release of Add-on Updater (scheduled for later this year) will require NVDA 2022.2 or later.
  • Switching add-on update sources: if you do decide to change add-on update sources from Add-on Updater settings panel, NVDA will ask if you really wish to do it. If you say "yes", the new source will be used, otherwise the old source will be used. Note that if you say "no", Add-on Updater settings panel will not reopen (this is actually a bug in NVDA).
  • Experimental features: shhh, not here.

IMPORTANT: if you do install this try build and happen to tell Add-on Updater to check for Add-on Updater updates, you will be asked to "upgrade" (actually downgrade) to 22.07. I plan to fix this in the next (and final) try build (later this week).

Cheers,

Jseph


 

Hi all,

I'm delighted to announce the last try build in Project Meteor series: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220714.nvda-addon

This build includes mostly bug fixes. Please review the changelog and notes before installing this try build.

Changelog:

  • Updated translations for some languages.
  • Internal code tweaks.
  • NVDA will no longer present add-on update toast messages if no updates are available (traced to a logic error).
  • Refinements to restart NVDA dialog.
  • On Windows Server 2008 R2/2012/2012 R2/2016/2019/2022 systems, NVDA will no longer enable automatic add-on update checks; you must now check for add-on updates manually on server systems. Consequently, automatic add-on update check checkbox is removed from Add-on Updater settings panel if running on Windows Server.
  • The ability to configure add-on update notification (dialog or toast) and background updates is now restricted to Windows 10 and later provided that NVDA is actually installed (not a portable copy) due to technical limitations. This also fixes an issue where if using portable copy of NVDA, you may find that NVDA tools menu will say "review add-on updates" without explanation.
  • You can now enable update checks for Add-on Updater. Note that until the next stable version of Add-on Updater is released, no updates will be shown.

Additional notes:

  • Although this is a try build, do try using it on production environments and provide feedback so I can refine the add-on.
  • Regarding disabling automatic add-on update checks, this change applies to Windows Server systems for now. This change will apply to Windows 7, 8, and 8.1 in January 2023. Manual update checks will remain as part of "nightlight mode" I mentioned in June; there is a second part of nightlight mode which will require some work to implement: disabling updating to development releases on older Windows releases altogether. I plan to not restrict users of older Windows releases from seeking updates from different sources. As I mentioned last month, I advise folks to upgrade to supported Windows releases now (ideal), as soon as possible (strongly recommended), or before January 2023 (last resort). I will talk more about nightlight mode in an upcoming post.

Cheers,

Joseph


 

Hi all,

Major change to nightlight plan: after reading feedback from the community, I decided to scrap a major part of nightlight mode: on Windows 7, 8, and 8.1, Add-on Updater will not touch automatic update check setting. Proceeding with the original plan can create confusion for users and does not align with precedence. An updated try build is now available to apply this change: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220714A.nvda-addon

Cheers,

Joseph


 

Hi all,

A significant change is now being tested in Project Meteor in the form of a new try build: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220715.nvda-addon

The biggest change (to be included in upcoming Add-on Updater stable release) is that disabled add-ons will not be updated by default. Depending on background add-on updates setting:

  • Background updates on: NVDA will just ignore disabled add-ons altogether.
  • Background updates off: disabled add-ons will be unchecked in add-on updates dialog (the one you get after checking for add-on updates). If you somehow check disabled add-ons and select "update add-ons," NVDA will ask if you wish to update disabled add-ons as updating them will enable them after restarting NVDA. If you say "no", NVDA will return you to ad-on updates dialog where you can uncheck disabled add-ons.

Cheers,

Joseph


 

Well I'd like a feature where you can choose to update addons from either the nvda addon database or thee author spaciffic addons.

Or find a way for something to automatically pole updated releases pages for authors periodically to get updates.

Some updates like tony's addons have not got latest updates to 1.14 and while you can request them manually added to the addon files database surely we need more sources.

Now I realise the potential for misuse but there are somethings that are not even in the main database.

Some others have made other sources and tools but there really should be other ways to select other databases and other mirrors and users.

I mean they do it in linux several sources at once.


Not sure how security would work though.


On 16/07/2022 4:34 am, Joseph Lee wrote:

Hi all,

A significant change is now being tested in Project Meteor in the form of a new try build: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220715.nvda-addon

The biggest change (to be included in upcoming Add-on Updater stable release) is that disabled add-ons will not be updated by default. Depending on background add-on updates setting:

  • Background updates on: NVDA will just ignore disabled add-ons altogether.
  • Background updates off: disabled add-ons will be unchecked in add-on updates dialog (the one you get after checking for add-on updates). If you somehow check disabled add-ons and select "update add-ons," NVDA will ask if you wish to update disabled add-ons as updating them will enable them after restarting NVDA. If you say "no", NVDA will return you to ad-on updates dialog where you can uncheck disabled add-ons.

Cheers,

Joseph


 

Hi,

As I wrote to Clement a while back, this can cause confusion for both users and Ad-on Updater regarding version and compatibility conflicts.

Cheers,

Joseph


Ravindran V.S.
 

Agree on this.

Add-ons installed from beside the official pages should also have a feature to be updated through the Add-on updater.

 Thanks,

Ravi.

V.S.Ravindran.

Excuses leads to failure!””

 

From: nvda@nvda.groups.io <nvda@nvda.groups.io> On Behalf Of Shaun Everiss
Sent: Saturday, July 16, 2022 1:46 AM
To: nvda@nvda.groups.io
Subject: Re: [nvda] Add-on Updater notice: introducing Project Meteor to refactor add-on download and installation steps

 

Well I'd like a feature where you can choose to update addons from either the nvda addon database or thee author spaciffic addons.

Or find a way for something to automatically pole updated releases pages for authors periodically to get updates.

Some updates like tony's addons have not got latest updates to 1.14 and while you can request them manually added to the addon files database surely we need more sources.

Now I realise the potential for misuse but there are somethings that are not even in the main database.

Some others have made other sources and tools but there really should be other ways to select other databases and other mirrors and users.

I mean they do it in linux several sources at once.

 

Not sure how security would work though.

 

On 16/07/2022 4:34 am, Joseph Lee wrote:

Hi all,

A significant change is now being tested in Project Meteor in the form of a new try build: https://github.com/josephsl/addonUpdater/releases/download/22.07/addonUpdater-tryProjectMeteor20220715.nvda-addon

The biggest change (to be included in upcoming Add-on Updater stable release) is that disabled add-ons will not be updated by default. Depending on background add-on updates setting:

  • Background updates on: NVDA will just ignore disabled add-ons altogether.
  • Background updates off: disabled add-ons will be unchecked in add-on updates dialog (the one you get after checking for add-on updates). If you somehow check disabled add-ons and select "update add-ons," NVDA will ask if you wish to update disabled add-ons as updating them will enable them after restarting NVDA. If you say "no", NVDA will return you to ad-on updates dialog where you can uncheck disabled add-ons.

Cheers,

Joseph