A short time after I got my 82, the starter stopped working when i pressed the statrer. I could hear a click under the tank when i pushed the starter button, so I knew something was at least connecting - somewhere. I pulled off the tank and found the relay that was making the click and it was the starter relay. I managede to remove the plastic cover from the relay and saw that the electromagnetically controlled arm that pulled the relay points together was working, but current wasn't flowing through the points. If I gave the arm a push to force the points together, it started up fine. I tried bending the arm a bit for closer contact and even filing the point faces like I used to do on my old VW, and it worked sporatically. About half the time it would work and the other half I would have to pull the tank, pop the relay cover, push the relay, start the bike, replace the cover, replace the tank and reconnect the fuel line, and all was fine. I got really good at it. I finally replaced the relay and have had no problems ever since.
Perhaps your relay is getting on and it's getting a weak pull on its electromagnet and as your bike starts, it shakes the points apart before you really get going. I'd recommend you try opening up your relay and pushing the contact arm and see if that does it for you. Then replace the relay. It is a relatively inexpensive fix and a remove-and-replace job. (My favorite kind) I think my relay was 26 years old when I replaced it, so I can't really complain.

Good luck with yours!
John McC