We want to start working with AWS Sagemaker.
I understand that i can open Jupiter notebook and work like it was in my computer.
but i know pandas working on single node.
when i working for example on my machine i have 64gb memory and that is the limit for pandas because its not parallel but AWS is parallel so how pandas work with thatTopic sagemaker aws pandas machine-learning
Category Data Science
In my opinion, your question is composed of two parts :
When you open Sagemaker studio, you are by default working on a ml.t3.medium instance, a very small (and cheap instance). The reason is that this local machine is not designed to run big processing jobs, but to make some data exploration on small data, manage sagemaker, run notebooks etc...
To run a big job on Sagemaker, you will need to setup a processing job that you will run on a distant instance. You will only pay for the running time you have made on this distant instance. You can run pandas / sklearn / pyspark jobs.
More about processing jobs on Sagemaker : https://docs.aws.amazon.com/sagemaker/latest/dg/processing-job.html
The list of distant instances available in sagemaker : https://aws.amazon.com/fr/sagemaker/pricing/
Pandas can't run in parallel. It is not related to AWS Sagemaker. If you want some pandas alternatives in parallel, you can refer to :
Spark processing jobs : https://docs.aws.amazon.com/sagemaker/latest/dg/use-spark-processing-container.html
Dask ("pandas in parallel") running on AWS Sagemaker : https://aws.amazon.com/fr/blogs/machine-learning/machine-learning-on-distributed-dask-using-amazon-sagemaker-and-aws-fargate/
Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.