When a contract is upgradeable, whomever controls the upgrade can change functionality in that contract. In terms of a token, this could include changing or freezing balances of any token holders, or changing the total supply/minting additional tokens.
The community has to place trust in whomever has the control.
This is why it is a good idea to decide ahead of time the circumstances under which an upgrade would occur and discuss this with your community, including documenting this.
Upgrades governance could be via a multisig. Some of the signatories could be token holders not part of the project. Some of the signatories could be from the wider community. The upgrade could be controlled by a DAO.
OpenZeppelin Truffle Upgrades shows transferring control of upgrades to a Gnosis Safe.