Hi @EvilJordan,
Welcome to the community 
All questions are good. Please ask all the questions that you need. It improves the knowledge base of the community and can be used to improve the documentation too.
Upgrades allow us to change the contract code, while preserving the state, balance, and address.
The proxy is a simple contract that just delegates all calls to an implementation contract. A delegate call is similar to a regular call, except that all code is executed in the context of the caller, not of the callee. Any reads or writes to the contract storage will read or write from the proxy’s own storage.
This allows us to decouple a contract’s state and code: the proxy holds the state, while the logic (implementation) contract provides the code. And it also allows us to change the code by just having the proxy delegate to a different implementation contract.
In your example the Proxy points to logic Contract A and when upgraded the Proxy points to logic Contract B but the state is held with the Proxy contract.
I suggest reading the How Upgrades Work section of the Learn guides.
I would also encourage you to try it out yourself following the Upgrading a Contract using the CLI section of the Learn guides.
One thing to note, is that if you are using OpenZeppelin Contracts with upgradeable contracts you need to use @openzeppelin/contracts-ethereum-package, which is an official fork of OpenZeppelin Contracts that has been modified to use initializers instead of constructors. (see: https://docs.openzeppelin.com/upgrades/2.6/writing-upgradeable#use-upgradeable-packages)
If you are building an ERC721, I would suggest looking at the upcoming release of OpenZeppelin Contracts 3.0. The latest release candidate is: OpenZeppelin Contracts v3.0 final release candidate This has improvements for ERC721 such as automatic token URI.