The yellow one is like a tube split length-wise. It snaps onto the shaft it pivots around. The split is a bit smaller than the shaft, otherwise it'd fall off. There's no way it could be pushed into the housing, the only way is out. Straight out. It might help if you can squirt some silicon spray in there to make it slipperier.
On the bottom side of the black turn signal plastic piece is a phillips screw that threads into the flat steel of the switch. It's counter sunk and the head is flush with the surface. It's black as well, so take a close look.
I don't think I've seen one fastened any other way, but I haven't seen em all, either, so yours could be pressed on. If there's no screw, that'd be my guess, in which case, pull it off!
Both of them need to come off to remove the guts from the face. Looking at your photo again, it's possible the toggle switch could stay and allow enough room for you to solder.