No manner of handwashing will prevent this. Skin sloughs off pretty constantly and over-washing will spur overproduction of sebum. Some people produce more than others.

But it is a gaming controller and there’s some likelihood of food being involved in the area, so you’re not entirely wrong.

And a change of diet may help too.

And change of environmental controls.

And change of lifestyle.