Shortly after birth, Drew and Garfield went their separate ways in showbiz