Unable to use pip package obtained from building Tensorflow 2.3 from source

I've managed to build Tensorflow 2.3 from source, following these instructions: https://towardsdatascience.com/how-to-compile-tensorflow-2-3-with-cuda-11-1-8cbecffcb8d3

But, when I install obtained pip package in new conda environment, and import tensorflow, I get following error: Could not load dynamic library 'libcudart.so.11.1'; dlerror: libcudart.so.11.1: cannot open shared object file: No such file or directory

I've managed to use GPU support with CUDA 11.1 for Tensorflow 2.5 nightly, without creating soft links between libs (I get Successfully opened dynamic library libcudart.so.11.0 message).

Any help appreciated.