It depends. For reference, I run my system in "Minimize display mode changes" and I'm happy with that setting. It takes about 8-10 seconds for my video chain to restore a picture after a mode change, and I don't like that delay.
Assuming that you have an 18Gbps HDMI 2.0 connection to your display, the thing that's most likely to cause visible issues by having "minimize" on is the conversion of frame rate from 24fps to 60fps. Because 24 doesn't divide into 60 evenly, the process of conversion results in some frames being repeated more often than others, which causes a visible artifact called "judder". Many people don't notice this, but others do, especially on certain types of scenes (e.g. camera pans across a landscape).
However, since most modern high-end displays and projectors use a native frame rate of either 120 or 240fps, if you had 24fps content, the division is an even multiple and you don't get judder. So that's why sending 24fps to the display natively can be a good option. However, many displays include a technology to detect that the 60fps signal they're receiving is really a converted 24fps signal. They then undo the conversion to recover the original 24fps signal, and then re-convert to the native 120 or 240 display rate. If this is done well, then you get exactly as good of a result as you would have had sending 24fps in the first place. This is what I do, and my Sony 995ES does a perfect job. And so that's why I'm happy with the Minimize setting. Unfortunately, everybody calls this setting something different. I think Sony calls it "True Cinema".