A harder wood such as mahogany can be used - you may need to cut the layers thinner to get them installed and I'd definitely recommend that you don't skip the one screw up forward into the stem. Damage to the rails that expose the wood to wet can cause the wood to expand hard enough to pop off the hull. I've seen this once, with mahogany, but they'd skipped that reinforcing screw. I believe that when the rail expands (due to unrepaired damage that exposes the wood to water), that it separates at the very end (f'w'd) first ... the screw makes the bond stronger, so the crack never starts. It's the only required piece of metal left in the boat (per instructions).
UHMW is fine, but it expands and contracts a lot in the lengthwise direction. You have to use a tight screw schedule, say not more than 6" apart (especially in the curvy sections), else it will 'work' the screws until the loosen ... which can let water get to wood. Rubrails are tricky. The easiest thing is to just fiberglass as you did and then repair as soon as damage occurs ... taping off the rubrail and repainting is pretty easy.