Can inactive WordPress plugins still load scripts?

Many hypotheses could be made, but the most probable ones are that the theme or a third plugin loads the script if a specific option saved in the database has a certain value. If the plugin is installed but not active that hypothetical option still exists in the database, on the contrary, when you delete the plugin, it will delete that option.

Another hypothesis is that you think the plugin is deactivated, but a third plugin filters the option "active_plugins" in a mu-plugin, and on the frontend that plugin is active, even if in the page of plugins you see it disabled.

Another hypothesis is the script is loaded by a caching plugin, and it was a coincidence that when you deleted the plugin the cache was also deleted.

You can do also other suppositions, but without having more details you can only guess.

The answer is already in your question:

Curiously, the forms work when the plugin is installed and INACTIVE, but they do NOT work when the inactive plugin is DELETED.

Note: The full form code snippets are being included in the page (not shortcodes or anything else referring to the plugin).

The plugin itself can't interfere with your site when it's deactivated, but that doesn't mean its files aren't accessible. If you're directly referencing JS and CSS files in the plugin from your code snippets then that bypasses activation/deactivation. None of the PHP will run however.