Lol. Realized I can actually remove the hitbox by having a custom hitbox with 0 polygons.
It is improved the performance by 10 FPS and allowed me to put 1000 more objects on the screen while maintaining 30 FPS on my dev PC. But it is till way below the results of some other engines, wondering how much the calculation to update the hitbox cost even if I have an empty array of polygons. 