unit-testing Machine Learning models

I have been asked to unit-test my machine learning model(not the code that made the model). Since we wouldn't actually know what predictions models make, how to carry out the unit-testing to check the model's predictions against? How is this done?

EDIT 1:

The machine learning model I have is trained on tabular data of patients. let's take an example of cancer prediction(I am not allowed to disclose the actual one, but this example is very close). It takes multiple reading from various tests as inputs and outputs how close or how risky a patient is to get cancer.

EDIT 2:

Is there any way, like testing for range of value for every set of inputs (or) adversarial inputs(inputs that are sure model will fail on) or extreme input cases. What ate the best practices for this?