Mine came off real hard the other day, sort of snapped lose. I don't know about the stainless vs. aluminum idea.
My bike was weaving because the swing arm had some free play. I re-torqued the swing arm bearing pins and lock nuts, made sure there was no play, then the bike stopped weaving and handles fine now. The bearings felt OK by feel but next time I have the swing arm out to lube splines I'll replace them. There has to be something wrong in there.
To check, put the bike on the center stand with the rear wheel in the air. Grab the rear wheel at the back and try to move it from side to side; if you feel any play, see if the swing arm is moving. If not and you do feel play it's probably the rear wheel bearings. I don't know what symptoms that would cause because I've not experienced it.
Good luck,