Intellectually, I realize this is a mistake. I'm consequentialist (at least I think I want to be, maybe), and when I look at Haidt's Julie/Mark-style situations, I typically answer as a consequentialist would. When making most moral decisions in my life, I try my best to avoid using disgust to make my decisions. And yet, disgust creeps its way into my moral landscape.
To understand why this happens, I first need to distinguish between myself (Me1), who acts in ways that I act, and the person (Me2) that I wish I could be, who acts in ways that I wish I would act. I don't currently live life to the fullest; I don't fully enjoy the pleasures of life that I could. I don't help others in need as much as I should. I don't do the proper amount of exercise, nor eat the diet that would best suit my body. There is so very much I would change, if I could get around my akrasia.
Yet it's not enough to look at who I am (Me1) and who I want to be (Me2). There's also who I want to want to be (Me3) -- this is the person that I wish I could want to be. If I'm going to take ethics seriously, then I should want to follow the demands that seem so obvious to me. I should want to help others as effectively as is sustainable and marketable for me.
As an example, I (Me1) spend a modest amount of time each week working directly with effective altruism organizations. I wish that I (Me2) would instead reorganize my life so that I could devote more time to EA stuff. But, deep down, I feel as though I (Me3) should instead wish to do as much as I possibly could do to help the EA cause.
And yet, this rundown of desires and meta-desires is not enough to fully describe how I feel about this, because there is an even deeper version: Me4, who feels disgust at the idea of dramatically changing Me1 to Me3. From Me4's point of view, it would be almost like committing suicide if I actually followed through with changing myself from Me1 to Me3. Me4 represents how I feel about how I would want to want to change myself. Despite my attempts to avoid using disgust in moral considerations as I look as my desires through Me2 and Me3, I nevertheless end up acknowledging disgust when looking at my meta-meta-desires.
When a change is small enough, I'm okay with it. I get up each day only slightly feeling as though I may have died the previous night. But when I think back to the Eric of ten years ago, I cringe. I am so very, very different from that person. I think of him as an ancestor, not as me. When I think about the future Eric ten years' hence, I do not want him to be so different. I want him to share my values! Don't get me wrong: I don't mind extrapolating my volition. There are surely facts in the world that would cause me to do things differently were I to truly understand them. If future Eric makes different decisions because he's learned more true facts, then I count that Eric as myself -- it's just that he's in a better position to know what to do.
But if his values are as different to mine as mine are to the past Eric of a decade ago, then he is not just me in a different time period. No, then he is more like my offspring. A person close to me in many ways, but who is decidedly not me.
This feeling of hidden discontinuity comes about from feelings of disgust at the idea of suicide. I don't want to die. I don't want future Eric to hold such disparate values. The entire reason I hold the values I do is because those are the values that I would instantiate into infinity. I would make them the rule in extended space, in the far reaches of the future, etc. Of course, I don't mean to so universalize wrong principles, so I'm not saying that the vocalization of my values is what is sacrosanct. Rather it is the ideas behind them, of equity, fairness, etc., that I dare not express too succinctly in words. To the extent that I am mistaken in true facts about the world, the way that we would express those values might change -- but the values themselves
To the extent that I should change my mind in the future, there is no reason not to also make that shift today, so there is no reason to suspect that future Eric should have reason to change value systems that I would not also consider reasons to change my current value system. While I am in favor of changing one's mind in the face of new facts, I am not inclined to change my value system unless there is good reason to do so (moral trade might make it better for my values if I vector-average with others' values (even acausal trade could make this happen); learning the basis of why I have a value might cause me to hold that value less or more strongly; etc.).
I do not want to die. I do not want future Eric to not be me. I do not really want Me3 to actualize. Yes, I'd like to self-improve to Me2. But Me3 is too far, even though Me3 is the better man.
It is the disgust of Me4 at the idea of committing suicide that causes me to not wish to change beyond my current desire to move from Me1 to Me2.
Death occurs much more often in a single life than most people realize. I want to be a better me, but Death is the enemy. I must become better without committing suicide.