The impermanence of the outer world seems more solidly founded. Five billion years hence, I'm pretty sure that this novelty shot glass next to me is not going to exist in any sort of recognisable novelty shot glass form. Nothing in this room will functionally persist as long as you only admit my Use Perspective as the only relevant lens of observation. The matter and energy will both still exist, but they won't exist in the configuration which I am accustomed to. And that, apparently, is supposed to fill me with a sense of existential dread. But it doesn't - at all - and this is the weakness of the conclusions that Buddhism draws from an impermanence theory of the external world. It supposes that I cannot hold in my mind at the same time both an appreciation and attachment to an object or a person as they stand in front of me right now AND a recognition that my use of a particular configuration of matter and energy at the moment doesn't determine how it will exist for all time. Buddhism's approach to use-based impermanence attempts to force us into a false binarism where we must either be the slaves of attachment or the cold observers of transience, and that only one of these offers us a way out of suffering. Compelled by the forced logic of its myopic perspective on self-analysis that we saw above, it opts for the latter, and presents that choice as an inevitable philosophical conclusion.
There certainly are those who see Buddhism as a means of escaping this vale of tears, but this criticism could equally apply to nihilism. Or Stoicism. (Not to panta rheism, though, which dissolves all such conceptual conundrums in its endlessly flowing waters.)