Why do my target labels need to begin at 0 for sparse categorical cross entropy to work?

I'm following a guide here to implement image segmentation in Keras.

One thing I'm confused about are these lines:

# Ground truth labels are 1, 2, 3. Subtract one to make them 0, 1, 2:
y[j] -= 1

The ground truth targets are .png files with either 1,2 or 3 in a particular pixel position to indicate the following:

Pixel Annotations: 1: Foreground 2:Background 3: Not classified

When I remove this -1, my sparse_categorical_crossentropy values come out as nan during training.

Epoch 1/15
 25/196 [==...........................] - ETA: 27s - loss: nan - accuracy: 0.0348 - sparse_categorical_crossentropy: nan  

Why is this the case? If the possible integer values are 1, 2, 3, why would I need to alter them to begin at 0 to be correctly used?

If I include the -1, the training looks correct:

Epoch 1/15
196/196 [==============================] - 331s 2s/step - loss: 2.0959 - accuracy: 0.6280 - sparse_categorical_crossentropy: 2.0959 - val_loss: 1.9682 - val_accuracy: 0.5749 - val_sparse_categorical_crossentropy: 1.9682