Yes, it should move .
Lube it up with a light oil, then put a #2 Phillips screwdriver in the hole and work it loose .
Don't beat on it, it's cast aluminum, won't take too much abuse .
Also, when you reinstall it, the gap in serrated end, goes on the brake shaft with the dimple in the shaft in the gap on the arm .