At least three, sometime more, different groups "hire" the referees working at 60 acres on any given weekend. While we absolutely cooperate with each other, if you remove a referee from working for one group to another, their insurance may not be applicable (and yes, referees get injured too) - AND you may make it impossible for them to get paid.
Most referees really want to help out - but they also know colleagues that have needed insurance coverage and been denied because of trading around assignments. Some referees, especially the youth, are hoping to earn spending money - and if they take your game and weren't assigned to it, they may not get paid anything.
So, if you ask a ref to move, and they say no - please respect that.