BYO has a pretty good, brief article on what yeast nutrient provides for your beer, as well as whether or not it's necessary. To summarize for you, wort by itself is pretty rich in nutrient, and may not need yeast nutrient, especially if you re-pitch yeast from a previous batch (proper amounts of slurry, of course). The one nutrient that is not present in wort, but is in yeast nutrient is zinc, which is necessary for cell growth, in small amounts, between 0.10–0.15 ppm. In ye' olden days when brewers used copper for brewing, the copper leached sufficient amounts of zinc into their brew to allow for yeast to ferment healthily.
According to the author of the post, a lack of zinc results in a prolonged lag phase in fermentation (that time in fermentation between when you pitch your yeast, and the krausen forms), with potential for stuck fermentations.
The question of yeast starters is a slightly separate conversation, as it has little to do with nutrient generation. Yeast starters aren't made to provide nutrition for yeast, they are done to increase the cell count of your yeast so that you are pitching a sufficient quantity of new, healthy, stress-free yeast. Since a starter involves fermentation, it will require zinc for healthy cell growth, so yeast nutrient would be a necessary addition for your starter, as it would be for your primary fermentation.
In short, yes, starters serve a purpose. Yes, yeast nutrient (specifically the zinc component) serve a purpose. Yes, there may be consequences to not using it in your beer, but no, you may be able to get away with not using it, just don't be alarmed if your fermentation is a little slow to kick off, and doesn't dry out as much as you might expect if you don't use it. I would recommend using it in both your starter, and your wort.