Upgrades fall into the category of trusted powers, such as the ability to make changes to rates, pause transfers, make operator transfers/burns.
What reduces the likelihood of a malicious action by an entity with trusted powers is the risk to their reputation and potential for consequences of the traditional legal system (civil and/or criminal).
Though this assumes that there are identifiable person(s), that the traditional legal system is accessible to the people impacted and it can reach the person(s) performing the malicious act and that the person(s) reputation has value (ideally of some significant value compared with the value stored in the contract).
Whenever we create special powers in contracts we need to consider under what circumstances we allow them to be used and how to prevent the powers being used maliciously (both by the entity with that power or someone gaining access to that power).
We should also design how we can move to a circumstance where we revoke those powers or transition to decentralized governance which control those special powers.
This might be time based or value based.
Looking specifically at upgrades, we should make contracts upgradeable where it is the best way to deal with bugs or add functionality. Which depends on the use case.