I was wondering if spaCy supports multi-GPU via mpi4py? I am currently using spaCy's nlp.pipe for Named Entity Recognition on a high-performance-computing cluster that supports the MPI protocol and has many GPUs. It says here that I would need to specify the GPU to use with cupy, but with PyMPI, I am not sure if the following will work (should I import spacy after calling cupy device?): from mpi4py import MPI import cupy comm = MPI.COMM_WORLD rank = comm.Get_rank() if …

I am new to data science I need to create code to find speedup compared with the number of processes while using a k-nearest neighbor. which (k=1,2,3,4,5,6,7). this process should be after downloading some datasets. it is preferred to use python. What is the appropriate code in python for that?

As a part of my research in Deep Learning, I have to frequently train models which require a lot of computing power. As such, I use my university's HPC environment to submit my jobs and to train my models. However, I run into one major issue - MONITORING THE TRAINING PERFORMANCE & METRICS! I generally build my models with Keras, and it is convenient to check the console from time to time to get to know about the model training/performance. …

I'm currently working on applying data science to High Performance Computing cluster, by analyzing the log files generated and trying to see if there is a pattern that leads to a system failure(specifically STALE FILE HANDLEs for now in GPFS file system). I am categorizing the log files and clustering based on their instances per time interval. Since some messages are more predominant over the others in any given time frame than the others, i don’t want the clustering to …