Yes, you will lose privileges. 

See https://serverfault.com/help/bounty

> If your new reputation brings you below the requirement for any
> privileges, you will lose access to those privileges.