I actually never cared about the library getting destroyed. I think the castle works a lot better as a home, and besides, the library is still kinda there as a decoration. Best of both worlds.