So if you are following those examples and using the OpenZeppelin SDK on-chain, yes those contracts created by the factory are upgradable. However, they are currently not (easily) upgradable via the CLI, because, as you correctly identified, the CLI does not have any way of knowing about them.
Unfortunately we don’t currently have an easy way to take care of this, but what you would need to do is keep track of the contracts created by your factory (either with another contract, or emit events when your contracts have been created which are caught by an off chain service) and then those contracts you can upgrade manually via solidity code.
If you wanted to make your life easier and didn’t mind first making it harder , THEN easier, you can poke around the
dev-<<some number here>>.json file which is created to track your project on the network, and you can add the information related to the new upgradable files created by the factory so that you can manage it via the CLI.
Otherwise, though, this sounds like a really interesting feature, but it doesn’t really scale I think for using in the CLI. If users created thousands of contracts with the CLI, it would be very, very difficult to to manage to upgrade them individually via the CLI.