The rulebook states a seven-point penalty will be awarded to the opponent if a player uses any object to measure gaps or distances, but the incident was missed by match referee Terry Camilleri.
"The referee is unaware, Ronnie is too but for me, that is a foul. You are not allowed to do that," BBC Sport pundit Ken Doherty said.
Five-time champion O'Sullivan took the frame to lead 3-2, but finished the session 5-3 behind, with the pair continuing on Wednesday at 14:30 BST.
Available to UK users only.