Being able push the seal in by hand is not right .
I'm wondering if at some point in the bikes past, the area where the seal is installed, was damaged during a seal removal, and then 'dressed' out with something like a wheel cylinder hone, so now the hole is slightly oversize.
When you use the silicone sealer, make sure the area is free from gear lube, and let the sealant sit for a day or two to fully cure before getting any oil on it.
I don't know if this would be a good idea or not, but would wrapping a few thicknesses of teflon tape around the seal before installation, might make it a tighter fit in the hole ?