My Budgie Is Afraid Of Hands Or Rather Fingers

Greetings, everyone
My family has a budgie for 4 years, since he was a baby and my father hand-fed him.
We have him free in our house, although he has a cage for sleeping, resting and eating, but it has no door and he can fly free in the house.
He loves people, he especially loves to hear us talking.
But he is afraid of hands, or rather fingers, I think. He can sit in our shoulder, back or head without problem, and he crawls up the arm if you offer him the hand with the palm downwards, but he step back if the palm is upwards. If you try to touch him, he tries to bite or flee.

We always thought it was some kind of trauma or fear, so we won't bother to touch him.
He don't have any problem of eating whatever is in our hands when we are eating and he's around, but don't you dare to raise a finger or he bites it.
Also, he love a lot to speak to toes nails, so it's some hand problem.

Is there any way to teach him to not be scared of fingers? I would really love to pet and hold him

Thank you very much