I would disagree with ceci_v, either makes sense (at least in normal British English). It took me longer than it should have to realize why it was quiet It took me longer than it should to realize why it was quiet I would say the first sounds more formal, but the second sounds more regular/spoken. However, here's an example of it in a book that doesn't use "should have" - https://books.google.co.uk/books?id=w-iTAwAAQBAJ&pg=PA12&lpg=PA12&dq=%22It+took+me+longer+than+it+should%22&source=bl&ots=1U2QDVKoiY&sig=xq6FtMw8JxsT4a6DLVSJqVaPWO4&hl=en&sa=X&ei=gfvFVPb1NKWE7gaDloD4Bg&ved=0CCAQ6AEwADgK#v=onepage&q=%22It%20took%20me%20longer%20than%20it%20should%22&f=false

"I should have done" is like you did something but regret it "I should do" is that you want to do sonething in the future Just removing the "have" in the sentence is not correct, like "I should done" Hope this aswers your question! Which example did you have in mind?