I understand how to do it with Truffle, but how do I do it on Remix? Do I deploy the proxy instead of the contract?
You should ask the specific Remix developers on how to do this. https://remix-project.org/
I highly suggest using truffle or hardhat. Having a robust dev environment is vital to quickly testing.
+1 If you want to start working with upgradeability you should become familiar with Truffle or Hardhat and use OpenZeppelin Upgrades Plugins as they will save you from making bad mistakes.