So this is what I've got. Please excuse to crudeness of the drawing
The reason I mention RS Latch is because you will likely need a method to prevent 1 2 3 and 4 from losing its settings after you change S, which means you will need an RS Latch to save the settings. The problem with that approach is that you will likely need a reset switch to reset the current side that the switch is set to.
Other than that, if it's currently possible to create small flip flops then it will better to use a flip flop for each output.
Either way, it's best to have buttons rather than switches for input 1 and 2.