He's shown that he has a sex drive after being exposed to the Happiness Punch by Nami (and when Bentram disguised himself as Nami and did the same thing). Heck, Luffy himself even admitted to Boa that the only reason her powers didn't work on him is because she had pissed him off too much to think lewd thoughts.
Thats exactly what I was about to say too and they are all pretty valid reasons. Especially when it comes to Luffy and his attraction to woman. Along with the point you brought up about Boas power having no effect on Luffy. He was simply too annoyed to even have those kinda thoughts about her.