You don't necessarily have to be physically fit. Once you become proficient, it doesn't take much effort to hang glide.
You're suspended in the harness by your centre-of-gravity on your body so in calm conditions, you barely have to even hold onto the bar. This allows you to relax and enjoy the flight even more.
As long as you are a coordinated person, fly the right-sized equipment for you and in flying conditions suitable for your level of experience most fitness levels are suitable.