Hi @zaryab2000,
Great question. Thanks for posting here. 
I recommend creating unit tests (aim for 100% coverage) for your implementation contracts and then a selection of higher level tests to cover the use of the proxy. See the Learn guides for more details: https://docs.openzeppelin.com/learn/upgrading-smart-contracts#testing
I have an example of unit tests for the implementation contract and some higher level tests via the proxy in the step by step tutorials:
- OpenZeppelin Upgrades: Step by Step Tutorial for Truffle
- OpenZeppelin Upgrades: Step by Step Tutorial for Buidler
You can use Truffle or Hardhat (formerly Buidler), or OpenZeppelin Test Environment.
If you are using web3 then I suggest looking at OpenZeppelin Test Helpers (https://docs.openzeppelin.com/test-helpers/0.5/).
Please note we will be creating an Upgrades Plugin for Hardhat soon, until then you can use Buidler or Truffle.