Duke was rarely invited to slumber parties