Yep that sucks, take it from me, a group needs at the very LEAST 1:50 - 2 min on that last floor. After over 50 wins, I've never killed him below that.
You'll need more if he decides to be a jerk spam the curse move.
@Nausi 4 good DD can destroy it in less than 20 sec... Less with 2hr. We killed it many times with 1min mark popped in the same time as we teleported and we still managed to lot for treasures and leave with Exit option.