It should be written underneath the game mode how many players are playing waiting for that one, rather than how many players are matchmaking h overall.
Some maps should only be available for certain game modes, free for all dosent work that well on small maps.
1v1s and 2v2s could be make into separate game modes, for quick matches. The matchmaking for these should only select from small maps, but those who want 1v1s and 2v2s on larger maps could still do it in lobbies.
Also 5v5 game modes could be made and select from larger maps like the castle one, that still feel empty with 8 players.