Dividend mechanism in ERC20Votes

[TheMNTN]

I am working on a token that is designed to inherit the ERC20Votes contract. In my design, the token holders elect representatives through the delegate function. Now I want to add a functionality where a dividend can be claim by the voters of a delegate for the work he has done in my platform. To achieve that I need the balance of each voter at the time the work was submitted. However I see that ERC20Votes allows me to retrieve only the past votes of the delegates and not the past balances of the users. Self delegation to track the balances also does not seem to be a solution since it would not be possible for each user to delegate to a representative. So now I am stuck not knowing what the best approach. What would be the best/safest approach to implement this mechanism.