Does the sign of correlation matter in feature selection?

If I understand correctly, the correlation between features and the target can be used to quantify whether those features are relevant to keep, hence the ritual of plotting the correlation matrix as a key step in data exploration.

However, does the sign of the correlation matter when it comes to feature selection? Isn't the only thing that matters the strength of the correlation (or anti-correlation)?