Performance bug - slow DB query

Since 2017-09-26 I have terrible problems with performance on my website. Loading single post or any other page takes even 20s. I've been debugging it it found out, that the problems are DB queries. For example this one is executing almost with every page load and takes about 36s: SELECT t.*, tt.*, tr.object_id FROM az2_terms AS t INNER JOIN az2_term_taxonomy AS tt ON t.term_id = tt.term_id INNER JOIN az2_term_relationships AS tr ON tr.term_taxonomy_id = tt.term_taxonomy_id WHERE tt.taxonomy IN ('znacka', 'model') …
Category: Web

LSTM evaluation metric MAE explanation

I have a hard time understanding the LSTM model performance as I summarize my model as follow: X_train.shape (120, 7, 11) y_train.shape (120,) X_test.shape (16, 7, 11) y_test.shape (16,) model = keras.Sequential() model.add(keras.layers.LSTM(100, input_shape=(X_train.shape[1], X_train.shape[2]), return_sequences = True)) model.add(keras.layers.Dropout(rate = 0.2)) model.add(keras.layers.LSTM(20)) model.add(keras.layers.Dropout(rate = 0.2)) model.add(keras.layers.Dense(1)) model.compile(loss='mean_squared_error', optimizer=keras.optimizers.Adam(0.001), metrics = ['mae']) history = model.fit( X_train, y_train, epochs=60, batch_size=5, verbose= 0, validation_split = 0.1, shuffle=False ) Based on the below plots, both MSE and MAE decrease in the training process and …
Category: Data Science

Difference Between Performance Scores

I need some help to understand the meaning between these different scores. Currently, I am doing the classification problems using machine learning, and I have obtained the results for the classification as shown in the image below. To obtain the results like in the image I use the code: from sklearn.metrics import classification_report, confusion_matrix, accuracy_score print(confusion_matrix(y_test,y_pred)) print(accuracy_score(y_test,y_pred)) print(classification_report(y_test, y_pred)) Then I also try to use the code below to get the recall, precision and f1-score print(precision_score(y_test,y_pred)) print(recall_score(y_test,y_pred)) print(f1_score(y_test,y_pred)) Results : …
Category: Data Science

WP_Options ID high

I have a site where the options_id field in the wp_options table has grown very, very large. A month and a half after the installation, the auto_increment value is now at over 10 million. While there are a number of plugins installed (such as ACF and W3 Total Cache) they do not seem to be the cause. They have been installed on others sites together with the same theme framework without any issues. I have a local dev environment of …
Category: Web

How to remove /page/2/ from home page?

First of all: I don't understand the meaning or the action that http://myurl/page/2 does? I checked the Net tab in Firebug and saw that my page takes more than 3 seconds to load /page/2... but I don't need pagination at my front page at all. I don't get where it comes from either. My question: how do I disable / remove this and show it only when pagination is needed? This is the query I am using to get posts: …
Category: Web

Speed up / Optimize Admin Edit Post Page on Large Websites (50-150k posts, 50 custom fields per post)

I'm working with a set of large Wordpress installs with 50-150k posts, each with 40-50 custom fields. The installs are very basic, with limited plugins (less than 5), and a default Wordpress theme (i.e. twenty twelve, etc.). The sites are on 1-Core, 2GB RAM, SSD VPS servers. These are all private installs, with no public front-end or web traffic. Automatic updater, auto-update core, and revisions are all disabled. I have no featured images or other media on the edit post …
Category: Web

How to optimize Pandas DataFrame reorganizing?

I have a DataFrame which looks like this: date person value 2022-05-01 A 5 2022-05-01 B 4 2022-05-02 A 5 2022-05-02 B 9 I want to convert it to that form: date person A person B 2022-05-01 5 4 2022-05-02 5 9 I have implemented that code to colve the task: raw_data = ... # data in the original form people = raw_data["person"].unique() dates = raw_data["date"].unique() new_data = pd.DataFrame(columns=people, index=dates) for person in people: for date in dates: new_data.loc[date, person] …
Category: Data Science

Inference speed of ReLU networks

I'm fairly new in the topic, and I was wondering whether some of you can point to existing works in which the inference of deep neural networks with ReLU activation functions is tested on GPUs as a function of the number of hyperparameters. Just to have a rough idea on how fast those networks can give an answer back for, e.g., approximation/regression purposes.
Category: Data Science

Transfer learning: Poor performance with last layer replaced

I am using a transfer learning approach. For this I followed the tensorflow for poets tutorial. I use a pre-trained InceptionV3 architecture trained on the Imagenet dataset. The last layer and the softmax classification is replaced and retrained, using a new set of 7 classes. Data Per class I have around 4.000 - 5.000 images. I tried multiple training parameters with an AdamOptimizer. The labels are noisy, about 15-20% of the labels are incorrect. The images show products of a …
Category: Data Science

How to run list comprehensions on GPU?

Is there a way to run complex list comprehensions like the following on GPU? [[x[index] if x[index]>len(x) else x[index]-1 for x in slice] if (len(slice)==1) else slice for slice,index in zip(slices,indices)] To what degree is it Possible? Do I have to convert it to some kind of numpy comprehension (if so what part is speciffically possible/necessary) The goal is performance optimization on large datalists/arrays.
Category: Data Science

Detect change in performance of new rasa model

I have a rasa nlu model running to detect intents and entities. Whenever a new model is loaded by rasa after training, I want to know how good or bad it is from previous model. What I am planning is , whenever a model is loaded first time, I can test it against some gold standard data and save the output metrics to compare with previous saved metrics. Questions: Is there any event which is loaded first and one time …
Category: Data Science

Defer Code in Widgets - Page Speed

How to UP Page Speed With Widget Defer? Is there a way to defer a widget in the footer? I have an external API in a footer widget which is slowing down my page. It is not needed until the page is loaded. The caching plugin I use (W3 Total Cache) gives me the option to defer other scripts, but not scripts directly coded into the widget. What is the best way to manually defer custom code API that is …
Category: Web

Timing sequence in MapReduce

I'm running a test on MapReduce algorithm in different environments, like Hadoop and MongoDB, and using different types of data. What are the different methods or techniques to find out the execution time of a query. If I'm inserting a huge amount of data, consider it to be 2-3GB, what are the methods to find out the time for the process to be completed.
Category: Data Science

Faster Data Transfer?

I'm doing a project which involves a small dataset (4GB). I'm trying to upload it to Paperspace to do some analysis but it's taking an absurd amount of time. Using Gradient, it would upload maybe 40MB in 10min then stop uploading. This was done using the "upload" button in the jupyter notebook. Using a P400 hourly computer I'm transferring at 800MBPS over SSH (well, through Nautilus), but it's still taking hours to transfer this small dataset. What am I doing …
Topic: performance
Category: Data Science

WP Optimization: Removing Orphaned wp_options (especially the autoload ones)

Is there no mechanism which purges old options from the DB? I tried a few plugins in the past and I found out that there are a few pretty long option strings which are set to autoload but IMHO not used at all - as I'm not using those plugins anymore. So I guess it makes sense to remove those options completely - but I'm not 100% sure whether I miss some implications? Am I mistaken, or this the unfortunate …
Category: Web

How to preload images through Code Snippets wp_head for specific pages?

Circumstances: Website Structure My Wordpress website is used as presentation and portfolio for my graphic design services. Visual impact is important for me, so most of my pages contain big background hero image as first impresions and/or entrance animations. Languages Is relevant to share that all of the pages have two versions: one for english and one for spanish. It matters because I'm trying to preload specific images for both versions of a page (example: Contact page in english and …
Category: Web

Relating ROC curves with class statistics

I have three neural net models that I am running on the same dataset (of 7 classes) and calculate their class performance and also ROC curves. The firs tmodel is a 4-layer model with 8 neurons in each layer, the second one is a 3-layer network of 32 nodes each, and the last one is a two-layer network of 64 nodes each. The class statistics and ROC curves for each network is shown below: 4x8 Network: precision recall f1-score support …
Category: Data Science

bad regression performance on imbalanced dataset

My current dataset has a shape of 5300 rows by 160 columns with a numeric target variable range=[641, 3001]. That’s no big dataset, but should in general be enough for decent regression quality. The columns are features from different consecutive process steps. The project goal is to predict the numerical variable, with the satisfactory object to be very precise in the area up too 1200, which are 115 rows (2,1%). For target variables above 1200 the precision can be lower …
Category: Data Science

KNN efficient implementation

The KNN algorithm is very handy and particularly suited to some of my problems, but I can't find any resources on how to implement it in production. As a comparative example, when I use a neural network, I already have at my disposal high-level tools allowing me to apply the neural network to examples (either library allowing me to smartly exploit the hardware of my devices when I want to do embedded, or infrastructures allowing me to use my neural …
Category: Data Science

Speeding Up Bulk Post Creation - wp_insert_post & update_post_meta

I am creating posts (variable products) using (wp_insert_post) function. An Example, I have 9 colors, 9 sizes. Which makes 9×9 = 81 total variations for 1 product, which is 81 times below function to be executed. function create_product_variation( $product_id, $variation_data){ $product = wc_get_product($product_id); $variation_post = array( 'post_title' => $product->get_title(), 'post_name' => 'product-'.$product_id.'-variation', 'post_status' => 'publish', 'post_parent' => $product_id, 'post_type' => 'product_variation', 'guid' => $product->get_permalink() ); // Creating the product variation $variation_id = wp_insert_post( $variation_post ); // Get an instance of …
Category: Web

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.