#20 Back up, compare and restore the settings an IDE writes

[Thomas Mueller]

An IDE started for a test writes to the same settings as the user's IDE of that version: window positions, desktops, the list of recent files, the settings of experts. So a test has to back them up first and put them back afterwards. How to do that right is currently knowledge in people's notes, and it is easy to get wrong:

• The IDE's registry key is not all of it. From Delphi 2005 on, desktops and layouts are files in %APPDATA%.
• reg export and reg import are not a faithful round trip. The import silently drops a string value that contains line breaks (Delphi 7 keeps a cached web page in Direct\Headlines), and a restore by delete and import then loses it without any error.
• Recreated values come back in a different order, so a comparison of two exports reports a difference where there is none.

Proposal, three options of the client in the style of --install:

GxInspect --backup=<version> <dir> [--include=<key|folder> ...]
GxInspect --compare=<dir>
GxInspect --restore=<dir>

• --backup saves the IDE's registry key for that version (the client knows its name already, --key included) and its folder in %APPDATA%. --include adds more, e.g. HKCU\Software\GExperts; the client cannot ask a plugin which keys it uses, since the IDE is not running then.
• Its own format rather than reg export: every value with its type and raw bytes, so multi-line strings, REG_MULTI_SZ and binary values come back exactly, and recreating the values in the stored order keeps the order too.
• --compare says what changed since the backup, value by value and file by file, which is worth having on its own: it shows what a test session changed.
• --restore puts everything back and is refused while any IDE of that version is running, because that IDE would write its own state over the restored one when it closes.

A possible second step, to be measured first: Delphi's -r<key> start option makes an IDE use a registry key of its own, so a test IDE could run on a copy (--clone=<version> <key>) and never touch the user's settings. Open: whether -r also separates the %APPDATA% folder, whether it behaves the same from Delphi 7 to 13, and whether the settings of experts follow it.

Until this exists, AGENTS.md tells agents to export the keys by hand and compare afterwards.

Related: #19 (MCP mode).

[Thomas Mueller]

Measured -r<key> (started with -rGxRTest -np -ns) in Delphi 7, 2007 and 13, 32 and 64 bit. This answers the open questions of the second step.

Registry key. The key name takes the place of the product part:

• Delphi 13: HKCU\Software\Embarcadero\<key>\37.0
• Delphi 2007: HKCU\Software\Borland\<key>\5.0
• Delphi 7: HKCU\Software\Borland\<key>\7.0

The 64-bit Delphi 13 IDE uses the same <key>\37.0 as the 32-bit one, with its x64 subkeys. The user's own key and the other keys under Software\Embarcadero stayed byte-identical.

A new key is not a copy. The IDE fills it from the installer defaults in HKLM. It has no experts, no third-party packages and no GxInspect server, and all options are at their defaults. Delphi 7 also copies its licence values into it, so an export of such a key must not be passed around.

The first start with a new key asks questions:

• Delphi 13 asks "File Association" first. OK would change the user's file associations, so answer Cancel. Then comes a first-start setup window (Next, Get started!).
• Delphi 7 offers to replace the system-wide Just-in-Time debugger. Answer No, and leave "Don't ask me this again" unticked.
• Delphi 2007 asked nothing.

%APPDATA% follows the key. Delphi 13 wrote Roaming\Embarcadero\<key>\37.0 and Local\Embarcadero\<key>\37.0. Delphi 7 and 2007 wrote no files in that short test.

The settings of experts:

• Settings that an expert keeps under the IDE's key follow -r. That covers GExperts' <key>\37.0\GExperts-1.4 and GxInspect's <key>\37.0\GxInspect.
• Files do not. GExperts keeps its settings files in a folder of its own (ConfigPath, by default %APPDATA%\GExperts\<IDE name>), and that folder is shared with the user's IDE.

--packages, --install and --uninstall already take --key=<key>.

Clone test. Steps, for Delphi 13:

1. Copy the user's key with reg copy HKCU\Software\Embarcadero\BDS\37.0 HKCU\Software\Embarcadero\GxRTest\37.0 /s /f. The copy is identical apart from the key name. Multi-line values survive, unlike with reg import.
2. Start Delphi 13 with -rGxRTest. It asked nothing. GExperts and the server loaded, and VERSION answered IDE: GxRTest 37.0.
3. Quit with QUIT DISCARD. The session had changed only the copy, and the user's key was unchanged.

Two gaps remain for a full clone:

• The %APPDATA% folder. Desktops (*.dst, bds.dsk) live there, and copying the key does not copy them. The clone created fresh default layouts, and its desktop setting fell back to "Startup Layout". --clone therefore has to copy Roaming\Embarcadero\BDS\37.0 as well.
• GExperts' settings files. The copied key still points GExperts' ConfigPath and CachingPath (GExperts-1.4\Misc) at the user's folders. On quit, GExperts rewrote MacroLibrary.xml there. The content was the same, but it was a write to the user's file. --clone could copy those folders too and point the two values at the copies. This is not tested yet.

With those two, --clone=<version> <key> would give a test IDE that never touches the user's settings. Removing it afterwards means deleting the key and the two %APPDATA% folders.

[Thomas Mueller]

Where the code should live. Backing up, restoring and cloning an IDE's settings is specific to each Delphi version, and so is registering a package. That knowledge belongs in the server package of that version, not in the client. The same goes for a plugin's own data, such as GExperts' settings files, which GExperts knows best. But all of this has to happen while the IDE is not running, so there is no server to ask.

The client can load the server package itself. Measured with a small test program, built with Delphi 13 and without runtime packages. For each package it sets the Delphi version's bin folder as the DLL directory, then calls LoadLibrary, the package's Initialize export, Finalize and FreeLibrary. It worked for these packages, without an exception and without any window:

• Delphi 7: designide70, GxInspectServer70
• Delphi 2007: designide100, GxInspectServer100
• Delphi 13: rtl370, designide370, GxInspectServer370
• Delphi 13, 64-bit test program: designide370, GxInspectServer370_64

So a client built with one Delphi version can run the code of every version's package. The package runs in its own runtime library, so only plain strings and integers can cross the interface; no Delphi strings or objects.

r5967 makes that possible. The server used to start in the package's initialization section, so the test program got a listening server too. It now starts in Register, which only the IDE calls. The IDE calls Register again when the package is unchecked and checked again under Component, Install Packages; measured in Delphi 7, 2007 and 13 (32 and 64 bit).

Plan. The package exports plain-named functions for the version-specific work, and the client loads the package of the chosen version and calls them. The client keeps what is not version-specific:

• finding the package next to itself;
• refusing while an IDE of that version is running;
• the confirmation dialog of --install.

A good first step is to move the existing --packages, --install and --uninstall code (src/GxInspect_Packages.pas) into the package, because it exists and is tested, so moving it proves the interface. Backup, compare, restore and clone then follow on the same interface.

Open before writing code:

1. The interface. Delphi 7's strings are 8-bit, so the parameters need PAnsiChar there or PWideChar throughout. Errors and listings have to come back somehow, e.g. in a caller-supplied buffer or through a callback.
2. 32 vs 64 bit. The client is a 32-bit program and cannot load GxInspectServer370_64.bpl. The 32- and 64-bit Delphi 13 IDEs share one registry key, so the 32-bit package could do the work for both if it knows the x64 subkeys. Otherwise the client also needs a 64-bit build.
3. GExperts' settings files. These are not covered by the IDE's key. Either GExperts' DLL gets an export for this, which is untested: loading the DLL outside the IDE runs all of its initialization sections. Or GExperts notices at its first start that it runs on a copied key, and moves its settings folder itself.
4. The backup format described above, which would then live in the package.

[Thomas Mueller]

Correction to the previous comment: the Delphi 2007 line of the measurement loaded GxInspectServer100.bpl, which is the package for Delphi 2006, not the one for 2007 (GxInspectServer110.bpl); both use the same rtl100 and designide100. Measured again: GxInspectServer110 with Delphi 2007's bin and GxInspectServer100 with Delphi 2006's bin both load, initialize and finalize cleanly, without starting a server. The conclusion stands.

[Thomas Mueller]

First step done in r5971. --packages, --install and --uninstall now leave the work that depends on the Delphi version to the server package of that version. The package's new unit ide/GxInspect_Settings.pas exports five functions; their contract is common/GxInspect_SettingsIntf.pas, with PWideChar and Integer only, and the answer comes back as name=value lines. The client keeps the version table, the IDE's install directory (which it needs to load the package at all), the check for a running IDE and the confirmation dialogs. The 32-bit package also does the work for the 64-bit RAD Studio 13, whose settings it shares.

Compared with the client before, --packages gives byte-identical output and the dialogs are the same. Two things changed: a version whose package is missing shows ? with a note, and --uninstall is refused for an IDE that is no longer installed.

Found along the way:

• The client must not unload a package it has loaded. Unloading took about half a second each, so --packages took 11 s instead of 1.3 s. Worse, once Delphi 7's own designide70.bpl had been unloaded, loading Delphi 2005's designide90.bpl ended the process on the spot. All 22 packages loaded side by side are fine, so the client keeps them loaded until it exits.
• Load errors must not show a dialog. When a package the server package needs is missing, LoadLibrary fails with error 126. When the package is there but lacks an export it needs, e.g. from a different build, Windows shows "Entry Point Not Found" and waits for a click. A script cannot close that box, not even after the process has ended. The client therefore loads with SEM_FAILCRITICALERRORS, gets error 127 instead, and reports it in a note.
• Every Delphi bin folder is usually on the PATH, so a dependency missing from the IDE's own bin is still found there. Package names carry the version, so that is the same version from another folder.

GExperts' settings files. These are not in the IDE's key, so a clone needs GExperts' help. Two ways remain:

• GExperts fixes itself up at the clone's first start. It notices that it runs on a copied key, e.g. through a marker the clone gets or a stored key it wrote its settings under, and then copies its folders and repoints ConfigPath/CachingPath. This would also help with an IDE started with -r by hand.
• An export in GExperts' DLL that the client calls while cloning. This follows the same pattern as the server package.

The idea of a stored recipe is dropped: its only reason was that plugin code seemed impossible to run without the IDE.

Loading GExperts' DLL outside the IDE is harmless (measured). The Register-GExperts scripts already use the DLL outside the IDE through rundll32 ...,InstallGExperts. Loading GExpertsRS130.dll the way the client loads a package, keeping it loaded for 8 seconds without calling anything, and unloading it changed nothing:

• the IDE's registry key including GExperts-1.4, and HKCU\Software\GExperts, are byte-identical;
• GExperts' roaming folder has the same checksums;
• its local cache has the same sizes and times;
• no window appeared.

GExperts creates its configuration only on first use. So only an export that uses it would create or migrate the GExperts-1.4 key, and outside the IDE it would do so under the default key. A clone export must therefore get the clone's key as a parameter.

Open for a plugin export:

1. Parameters: the clone's key and the target folders.
2. How GxInspect finds such exports without knowing the plugins. Proposal: go through the DLLs registered under the clone's Experts key and read each file's export table first. Only a DLL that has the function is loaded, so other experts are never run outside the IDE.
3. The 64-bit IDE. Its Experts x64 key lists 64-bit DLLs, which the 32-bit client cannot load. For GExperts that does not matter, because the 32- and 64-bit IDEs share one configuration, but in general a plugin would have to name a 32-bit DLL that acts for its 64-bit one.

[Thomas Mueller]

The clone ("second step") is done, in r5980 to r5985:

GxInspect --clone=<version> --key=<key>
"<IDE>\bin\bds.exe" -r<key>
GxInspect --remove-clone=<version> --key=<key>

What --clone copies:

• The IDE's registry key: Software\<vendor>\BDS\<version> to Software\<vendor>\<key>\<version>, with RegCopyTree. Every value keeps its type and raw data, so multi-line strings survive, which reg import drops.
• The IDE's folders: <vendor>\BDS\<version> in %APPDATA% and %LOCALAPPDATA%, which hold the desktops.
• Plugin files: it asks every plugin that can do so to give the clone files of its own. That is a DLL under the key's Experts that exports GxInspectPluginClone (contract in common/GxInspect_SettingsIntf.pas). The client reads each DLL's export table from the file, so no other expert is loaded outside the IDE. A plugin gets GxInspect\Clones\<vendor>\<key>\<version>\<plugin> in both %APPDATA% and %LOCALAPPDATA%.
• GExperts copies its data folder there. It points ConfigPath, CachingPath and every other setting that named a file in its old folders at the copies.

Notes in the output name what the clone still shares with the normal IDE:

• the files of plugins that can't take part, e.g. CnWizards, IDEFixPack or DDevExtensions;
• a file a plugin's settings name outside its own folders, such as a capitalization file kept in a folder of the user's own;
• plugins registered for the 64-bit IDE only. The 32-bit client can't load those, unless a 32-bit DLL in the same folder takes part for them.

Safety checks:

• --clone is refused if the key exists already or one of its folders isn't empty. If any part fails, it removes what it did.
• --remove-clone deletes only a key that --clone made, which carries the value ClonedFrom under <key>\GxInspect.
• --remove-clone is refused while an IDE runs with that key. It reads the command lines of the running IDEs of that version. Where a command line can't be read (Windows before 8.1), that IDE counts as running with the key.
• --remove-clone asks the plugins which folders they use for the clone, then deletes the clone's folders, then the key. A folder outside the clone's own is left alone and named in the output.

Measured with Delphi 13 (32 bit) and 2007:

• After a clone and a QUIT DISCARD, the user's registry key, the IDE's folder and GExperts' folder were byte-identical to backups taken before.
• The IDE on the clone wrote its desktop and GExperts' MacroLibrary.xml into the clone's folders only.
• Cloning and removal were also tested with Delphi 7 and 2010. Those two share the version key 7.0, which is why the clone folders have a vendor level.

GExperts at the first start in a copied key (r5985): when GExperts runs in a key other than the IDE's own for the first time, it says which folders and files its settings there share with other IDEs.

• That is its data folder if it is the one of the IDE's own key, and every data file outside it.
• This covers a key copied by hand with reg copy and started with -r, which shares the whole data folder.
• It says so once per key and records the key in Misc\SettingsKey. --clone writes that value as well, so a clone doesn't get the message.
• The box doesn't wait for its OK, because the IDE still loads packages after its main window is up. A modal box held that up, the GxInspect server included, until it was closed.

Still open from the proposal: --backup, --compare and --restore.

AGENTS.md now recommends a clone over backing up and restoring the settings by hand.