What might a utilitarian say to someone who thinks this is too high a sacrifice?

One famous worry about utilitarianism is that it demands that we regard our own set of desires, ends, and our own happiness, as just one among a great many others whose lives we might impact. Accordingly, our own desires, ends, etc. bear very little weight when determining what the greatest happiness of the greatest number is, and thus what our moral responsibility is. Think of a situation or area of life in which this might be true, and our concern for our own well-being and happiness has to take a back seat to the concern for the well-being and happiness of the greatest number. What might a utilitarian say to someone who thinks this is too high a sacrifice? Would this be a plausible response? Be sure to back up your answer with references to the resources, and respond to your peers by considering what someone who disagrees with them might say