Exactly. The mania over "TV market size" is, in and of itself, so not relevant.
As you point out, what matters is how many sets will tune in to watch Teams A and B play football - no matter where the teams are from, or where the sets are located.
Otherwise, Rice would be a heavily desired addition to the SEC, and Rutgers to the B10. How dumb would that be? (Oops).
Some more detailed data as it pertains to Delaney's Folly:
Total viewership for all nine of Rutgers' conference games in 2021:
6.16 Million. Under 700,000 per game.
Nearly 80% of that for their games against Michigan, OSU, and Penn State. When they were not playing PSU, Michigan, or OSU - an average of about 300,000 (About the same number who watched Buffalo take on Coastal Carolina in a Noon start on ESPN2. Maybe the Big Ten should have added Buffalo, to get that NYC market locked down?)
No one, in NYC, or anywhere else, turns on the TV to watch Rutgers.
FWIW: Rutgers' "expansion partner", Maryland, a very non-premiere program, in a smaller and more geographically competitive market, drew about 70% more eyeballs per game than Rutgers.
Penn State, FWIW, drew just under 39 million (and had only about 10% of their typical viewership when they played Rutgers)
Ohio State drew just under 52 million (and only had about 20% of their typical viewership when they played Rutgers)
Michigan drew just over 50 million (and only had about 45% of their typical viewership when they played Rutgers)
Rutgers not only doesn't provide any viewers, they significantly decrease viewership by dragging down the desirable programs when they have to schedule Rutgers.
It is what it is.
College football TV ratings, 2021 edition - Sports Media Watch
,