Should I be with my father for long periods of time each week if he is an atheist?

I see my father usually once a week but for about an hour or more at a time. I have had negative effects to my physical health from being with him. His advice was erroneous on health, and I became underweight. There is a holy scripture, which says: "He that walketh with wise men shall be wise: but a companion of fools shall be destroyed" (Proverbs 13:20). I almost literally was destroyed. At least, I was severely declining in health.