To Sand or not to Sand

I just finished up the first hand-sanding using a sanding screen and my drywall pole sander. This sanding is 120 grit. I also did a little with my hand held drywall sander where I wanted to use more pressure and work smaller areas than I can do with the pole sander.
At this point I'm debating how far I will actually go with the sanding. The last floor I did I never went over 100 grit. The bulletin boards on the minwax site recommend going up to 180 grit with the sanding. I don't think going any higher with the grits will help as far as any of the swirls from the big sander or with any of the other marks on the floor at this point...And as far as the overall look of the finish goes, I think the last floor I did where I stopped at the 100 grit and did no hand sanding, looks really good. I don't see it getting any better if I go up 2 more grits. And the 120 sanding was done by hand with the grain of the wood to smooth out any marks left from the machine sanding. 150 grit sanding wont remove marks left by the 20 grit sanding...Each subsequent grit will only remove the marks left by the previous grit. So any swirl marks left on the floor at this point would require back-tracking a few grits and working my way back up in that area.



I did an extra-good shop-vac job before I took these pix. Before I stain or poly I'm going to have to do the super-extra-good shop-vacing including vacuuming all of the walls in the room because they are completely covered with a thin layer of dust.
But...Here's the floor as it looks right now.


So at some point I need to figure out where the mentor's statement, "Enough is enough and enough is too much" comes into play and I should just move on to the stain. The house is almost 100 years old, the last sand and re-finish job was done poorly (there are major divots around the edges of the room from the last guy not doing a good job with a drum sander). So at some point I need to decide it's not going to get any better than it is now if I keep going and just move on to the next step. The anal retentive perfectionist in me has a hard time with this.