The only time I truly stopped actively collecting was a few years while in the Navy. I only stayed for 6 years active duty, and the flame started back up before I got out. It was the time I was in school, away from home for the first time, finding new adventures and ultimately punching holes in the water (only in the Pacific for me @mm1sub). I am sure we have spoken about this, but it has been years. I was a STS3(SS) when I got out, made 2nd class in the reserves before separating completely in 2000.
I think about walking away, but only because of the fact that my kids don't seem to have any passion for anything like this. I don't know if they ever will. Both dabbled very lightly at my pushing and never kept an interest, even when both were passionate about basketball (just not the cards). As I get older and the stuff piles higher, I just can't help but think what am I going to do with it all or what happens if I die untimely or w/o warning. I don't want to burden my family with a couple rooms full of collectibles to deal with.
That said, if my kids liked cards, then I would probably never think twice about stopping and just plan to leave it all to them to figure out what to do with. Assuming I still have a few decades left, I probably should start selling off in the near future, as it is going to take forever if I want to maximize my returns. I never collected for the perceived fiscal gains, but I have wrapped up a lot of $ into this after 40+ years and it is money that I can squeeze back out, at least partially. Some stuff I have purchased has definitely appreciated, other stuff has tanked. It all washes out in the end hopefully.