EnumerableSet doesn't have storage, but we still offer an "upgradeable" version so you can be sure that you're using a contract that is safe for upgrades.
Thanks @frangio! 've dived deeper into how arrays are stored in solidity too and so can see how the EnumerableSet pattern is upgrade safe so long as the usual caveats for upgradeable contracts are applied.