How do I deploy an upgradeable token on Remix?

I understand how to do it with Truffle, but how do I do it on Remix? Do I deploy the proxy instead of the contract?

You should ask the specific Remix developers on how to do this.

I highly suggest using truffle or hardhat. Having a robust dev environment is vital to quickly testing.

+1 If you want to start working with upgradeability you should become familiar with Truffle or Hardhat and use OpenZeppelin Upgrades Plugins as they will save you from making bad mistakes.