If you want to swap the buttons for levers the gate becomes as simple as an XOR connected to an RS NOR latch. There are numerous versions of both, each with different dimensions and output directions.
Here's the setup I used. If you tell me some dimensions I'll toy around with some different layouts, too.
EDIT: Holy shit. My screenshots align perfectly. That was not intentional.