What should after I varnish my wood?

Sounds like you need to get some very fine grit sandpaper (220 or even higher, depending on how rough the rough sections are) to smooth any imperfections. Before applying any more polyurethane, be sure to use a tack cloth on any sanded areas. Any dust left behind from sanding will become embedded in the poly and feel rough. Good luck.