Hi @Sam0000,
Welcome to the community 
See the documentation: https://docs.openzeppelin.com/learn/upgrading-smart-contracts#testing
Essentially we can create standard unit tests to test our implementation contracts and we can create higher level tests to check interaction via the proxy, including when we upgrade to check that state is maintained.
For an example of tests see: OpenZeppelin Upgrades: Step by Step Tutorial for Truffle