Asakta Betorah?

The Gemara in Yoma 36b discusses the three types of people that will try to excuse themselves from hitaskut Be'torah, the poor man, the rich man, and the rasha. The Gemara says that the retort to the poor man will be have you been poorer than Hillel etc. therefore Hillel obligates all poor people. The same goes for the rich man for you are no richer than R' Elazar Ben Charsom and the Gemara goes on to describe his great wealth. Both Hillel and R' Elazar ben Charsom could have made the excuse that they were so busy just getting by, Hillel working hard just to make ends meet and R' Elazar Ben Charsom just administrating to his affairs  that they simply would have had very little time to be Osek in Torah. However, the Gemarah says that both of them spent much time studying Torah, R' Elazar so much so that his servants did not recognize him and he did not know of all his wealth.

What seems odd is that the Rasha seems a little incongruous. The Gemara says that they will ask the rasha why werent you osek in Torah and if he says "Osuk Beyitzri Hayisi" they will ask him were you more osuk than Yosef who had the beautiful wife of Potifar do everything in her power to seduce him and he did not allow it to happen. The simple question is what does this have to do with being osek in Torah. Furthermore what does "Asuk Beyitzri Hayisi" have to do with being Osek in Torah? To keep things congruous and simple it must be that the "Rasha" is the person that has this great Yetzer Harah for he is beautiful or has some other great yetzer hara that for some reason makes it very hard for him to focus on learning Torah and to fight that yetzer harah he  feels that he needs to learn musar or occupy himself with some other occupation to keep his mind off his yetzer harah. As a direct response to that person the Torah shows us that Yosef was able to overcome his Yetzer harah through sheer willpower and not by sidetracking himself with other occupations that would preclude him from learning Torah