"My mind goes to bad places"
Here's the reason most people give when pressed. The trouble with quiet isn't the quiet. It's what the mind does in it. Left alone it goes to the meeting, the unanswered email, the mother, the thing that happened in 2019, and none of that is a good place to spend a walk.
Matthew Killingsworth and Daniel Gilbert, in 2010, used a phone app to ask thousands of people, at random moments through the day, what they were doing, what they were thinking about and how they felt. They found that people's minds were on something other than what they were doing almost half the time, and that when the mind was wandering, people were, on average, less happy than when it wasn't, whatever they were doing. A wandering mind, the title said, is an unhappy mind. The earlier book on the phone offers this as honest counter-evidence, and so do I. The drift often isn't pleasant. The bad places are real.
But look at what's in the bad places. The meeting. The email. The mother. Those aren't random unpleasantness. They're the day, and the week, arriving to be dealt with. The mind goes there because those are the things that need attention, and quiet is the first time all day that attention has been available. What the fill does isn't remove them. It postpones them. They're still there at three in the morning, when the audiobook has stopped and there's nothing left to postpone with, which is why three in the morning is where a lot of filled lives do their thinking.
There's a difference between the drift and the going over that an earlier book in this series calls a replay, and I'd ask you to notice which one you're avoiding. A replay is the same scene, again, with no new information. The drift is where the meeting gets filed, the email gets a first sentence, the worry about your mother becomes a plan to ring her. That's not a bad place. It's the day being sorted, and it feels unpleasant in the way that sorting does, and the fill has been standing between you and the sorting for long enough that the pile has grown.
Killingsworth and Gilbert's participants were unhappier while wandering, and I don't doubt it. What the study can't tell you is what those minutes were doing for them. Chapter 18 will suggest that some of what they were doing was work, and that a person who never lets their mind wander has a mind full of unfinished business that's found a slot at three in the morning.