I am currently playing around with different CNN and LSTM model architectures for my multivariate time series classification problem. I can achieve validation accuracy of better than 50 %. I would like to lock down an exact architecture at some stage instead of experimenting endlessly. In order to decide this, I want to also tune my hyperparameters. Question: How do I balance the need to experiment with different models, such as standalone CNN and CNN with LSTM against hyperparameter tuning? …

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's DeepAR deals internally with seasonality, but does the same thing stands for trend? Let's say I have multiple timeseries, where some of them have positive, and some have negative trend. Should I remove trend and then use DeepAR prediction, or should I just ignore it and …

We have followed the following steps: Trained 5 TensorFlow models in local machine using 5 different training sets. Saved those in .h5 format. Converted those into tar.gz (Model1.tar.gz,...Model5.tar.gz) and uploaded it in the S3 bucket. Successfully deployed a single model in an endpoint using the following code: from sagemaker.tensorflow import TensorFlowModel sagemaker_model = TensorFlowModel(model_data = tarS3Path + 'model{}.tar.gz'.format(1), role = role, framework_version='1.13', sagemaker_session = sagemaker_session) predictor = sagemaker_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge') predictor.predict(data.values[:,0:]) The output was: {'predictions': [[153.55], [79.8196], [45.2843]]} Now the problem …

Hello Guys i have a question We want to start working with AWS Sagemaker. I understand that i can open Jupiter notebook and work like it was in my computer. but i know pandas working on single node. when i working for example on my machine i have 64gb memory and that is the limit for pandas because its not parallel but AWS is parallel so how pandas work with that

I ran a complete AWS SageMaker Autopilot experiment. I now want to generate batch forecasts using this model but I get the error: "No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config". I'm using this tutorial as reference. Here's my SageMaker Studio notebook Python code. import sagemaker from sagemaker import get_execution_role import boto3 import os from time import gmtime, strftime, sleep session = sagemaker.Session() bucket = sagemaker.Session().default_bucket() prefix …

I have deployed the background removal model( Pytorch- pre-trained u2net) in aws using lambda and EFS file system and APIgetway. I have stored my model in efs and loading to the lambda. the model is around 170MB. The API getaway response time is around 32 seconds. is it any way to speed up the response time?

I've been working on a simple computer vision API with a few endpoints for grabbing useful information from ebay images. The API lives in a docker container that looks for the h5 files, downloads them if they aren't there, spins everything up and starts the server whenever the container is built. One of the endpoints is a transfer trained VGG-16 classifier that divides images into buckets for further analysis and human review. This endpoint is exhibiting some odd behavior. When …

I recently started a new position as a data scientist at an E-commerce company. The company is founded about 4-5 years ago and is new to many data-related areas. Specifically, I'm their first data science employee. So I have to take care of both data analysis tasks as well as bringing new technologies to the company. They have used Elastic Search (and Kibana) to have reporting dashboards on their daily purchases and user's interactions on their e-commerce website. They also …

So I have trained a ML Model and I need to use it on AWS to make predictions on newly fed in data on a periodic daily basis, and output this data in the form of an attached email. However I am fairly new to AWS and am not sure, I don’t know if I should be using SageMaker, EC2 or Lambda for this purpose. Can someone point me in the right way? I have written a jupyter notebook for …

I am working concurrently with multiple very large datasets (10s-100s of GBs)). I signed up for Colab Pro+ thinking it is the best option. However, I face a significant bottleneck in getting data into Colab. My options all seem very bad: Downloading from AWS (where the data is located) - very slow. Uploading data to Google Drive and mounting Drive using below code. This is also surprisingly very slow. from google.colab import drive drive.mount('/content/drive') Paying for a persistent server. Something …

Ran my CNN on a SageMaker notebook and it started training, but I had to restart the kernel due to AWS disconnecting. However when I tried to rerun my code, I received an OOM error, and it never started training again. I tried: Restarting the kernel Restarted the AWS machine But the error still persisted. I find this strange due to the fact it ran before. ResourceExhaustedError: OOM when allocating tensor with shape[262145,25600] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator …

I am playing with AP to build a recommendation system and have a question that I am unable to find answers anywhere. My interactions dataset and items dataset contain several items. But I want the recommendations only on a subset of products. While this can be handled using some postprocessing steps, can I just provide the data for the products of interest in the items dataset and get the desired result without any performance implications? Any advice will be appreciated. …

So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe, and I'm failing greatly. The JSON file is incredibly complex. Here's an example of it based on the documentation: { "source": "Amazon SageMaker is a cloud machine-learning platform that was launched in November 2017. SageMaker enables developers to create, train, and deploy machine-learning (ML) …

I am trying to implement a docker file for Amazon Sagemaker Container,in initial step i am following this link https://towardsdatascience.com/brewing-up-custom-ml-models-on-aws-sagemaker-e09b64627722 In above link's section "Creating Your Own Docker Container" last command of docker image is COPY xgboost /opt/program I don't have any idea what xgboost file here is for this? Due to this my docker build is failing , please see below image of docker and its built Docker Image FROM ubuntu:latest MAINTAINER Amazon AI <[email protected]> RUN apt-get -y update …

Especially when considering GCP, the analytics offer from Google is quite interesting. Why would you go with Databricks? GCP has also great integration between tools as well as great support for ML/AI, etc.

It looks like there are different routes to deploying an ML model on SageMaker. You can: pre-train a model, create a deployment archive, then deploy create an estimator, train the model on SageMaker with a script, then deploy My question is: are there benefits of taking the second approach? To me, it seems like writing a training script would require a bit of trial and error and perhaps some extra work to package it all up neatly. Why not just …

I'm trying to read files from S3, using boto3, pandas, anaconda, but I have the following error: ImportError: Pandas requires version '0.3.0' or newer of 's3fs' (version '0.1.6' currently installed). How can I update the s3fs version? This is my code: import boto3 import pandas as pd s3 = boto3.resource('s3') bucket= s3.Bucket('bucketname') files = list(bucket.objects.all()) files objects = bucket.objects.filter(Prefix='bucketname/') objects = bucket.objects.filter(Prefix="Teste/") file_list = [] for obj in objects: df = pd.read_csv(f's3://bucketname/{obj.key}') file_list.append(df) final_df = pd.concat(file_list) print (final_df.head(4))

I fine-tuned an Inception V3 model provided in AWS SageMaker to detect COVID-19 Rapid Test Results (see the image below for an example). I provided about 20 pictures of negative and about 20 pictures of positive tests for the training. All pictures were taken with slightly changing angles and positions. However, when testing the fined-tuned model, the recognition did not work at all. Is the deviation between both image classes is too small (only changing red bars). Is there any …

I followed the instructions from this article about creating a code-free machine learning pipeline. I already had a working pipeline offline using the same data in TPOT (autoML). I uploaded my data to AWS, to try their autoML thing. I did the exact steps that were described in the article and uploaded my _train and _test csv files, both with a column named 'target' that contains the target value. The following error message was returned as a failure reason: AlgorithmError: …

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training. Most the the pipeline is on Lambda, but Lambda is known to have time limits on how long a job can be run (~15mins). Thus for the longer running jobs like batch training of ML models we will need(?) to access something like EC2 instances. For example a lambda function could be invoked and then kick off an EC2 instance …