Rangers, who dominated the game for long periods, were leading with six minutes to go through a penalty from Adel Taarabt.
But as the game moved into stoppage time, Rangers failed to deal with Jamal Campbell-Ryce's corner and on-loan Tottenham defender Caulker crashed a header into the net.
Despite bossing the first half, Rangers somehow found themselves behind just after the interval as Clint Hill's mistake allowed Brett Pitman to convert a rare City chance.
However, Neil Warnock's side, who had lost three of their last five matches having previously gone 19 games unbeaten to set the early pace, hit straight back when Taarabt set up Alejandro Faurlin.
City looked like holding out for a point until the 84th minute when referee Fred Graham deemed Elliott had held back Hulse and Taarabt did the rest from the penalty spot. But in a dramatic late twist Caulker rose highest at the far post to head in Campbell-Ryce's cross to salvage a point.
Source: PA
Source: PA