Ok, first off there are not enough people in this damn game for it to be an HOUR long. We don't want to sit around for an entire hour waiting on someone to contest that isn't going to show up in the first place. I suggest two ideas.
1. Lessen the timer of lvl 15-30 castle sieges to 30 minutes
Lessen lvl 40+ castles to 40 minutes
2. If no attack has been attempted during a siege within 20 minutes, then you have a way of ending it early. I think this is very fair as well. We have had too many times where we sit there for nothing to ever happen. BORING.