Did you order the events so it checks for collisions and then iterate over the results as a subevent?
Another thing that causes a slowdown is effects. Do you have any in the scene or on the objects?
You can also check out this thread on performance improvements. It may help with what your issue.