One loaf of bread attracts one vagrant. BUT... The game does not match all vagrants with all loaves of bread at the same time. Here's how it works:
From time to time* the game checks for 1 idle vagrant on camps and 1 loaf of bread not being sought. If both are available there's a match: that vagrant starts running for that loaf of bread, which won't attract any other vagrant. Very similar to how free farm fields attract farmers idling at the town center, one match at a time. And unike tools and weapons on shops, that attracts every subject inside their range of influence.
* Around 4 - 6 times a day, I'm not sure about this frequency.
That said, there is no restriction to how many vagrants are running at the same time. If the camps are too far, the next check-match will probably happen before the 1st vagrant reachs the bakery, which will cause two vagrants to be running at the same time.
And a last thing to note is that portals in the middle of this process can mess things really hard. Vagrants normally pass by portals running, but they would stop if they find a greed. And if they stop, they forget they were seeking for bread and they would wander around back to the camp.
Now this is the behavior in New Lands for sure, and probably in Two Crowns which I haven't played as much to be sure.