To get started please follow the tutorial over at OpenZeppelin Upgrades: Step by Step Tutorial for Hardhat
It’s a great guide to get you a dev environment set up and ready for action. After setting up your dev environment and getting things working, then you can try to deploy custom contracts.