Yes IB has limits... when dealing with attacks that aren't of a similar nature to something that's "one-and-done" like a punch would be.
And the IT never lost to anyone with good luck (??), it just got straight up overpowered in the 2/4 times it was used (the first time it was used against Fiamma in WW3, the second time against High Priest in NT12(?), the third time against Aiwass in NT18(?), and the fourth time in NT22 where it took on the appearance of fish eggs, or at least that's what I believe that to be, it's up in the air as you can probably guess

).
Saying Touma's fighting skills aren't all that great is a bit disingenuous, but w/e.
And Touma doesn't actually get powered up by IB, he's just straight up that physically capable.
Needless to say, he's simply gotten stronger over the course of the series even if he didn't necessarily go through something similar to a typical shonen training arc.
Like, Popeye still punches him into bloody chunks but this ain't it.