Your Meme Says No, But Your History Says Yes