Keiron Thomas struck the ball home with a sweet low drive in the 33rd minute.
Striker Jake Jackson received his marching orders 12 minutes earlier after appearing to kick out at Jordan Cullinane-Liburd; the latter player also picking up a booking in the process.
Up until that point, Gloucester were the better side and went close through Zack Kotwica, while Brandon saved a Joe Hanks free-kick on the stroke of half time.