I have a set of efficiency values based on 2 dependent variables (speed and torque). These numbers are provided from the manufacturer and I want to fit the efficiency values to a set of personal test data points (e.g. the singular data point shown in the bottom left of the following image). This image shows my problem (where the efficiency values have been randomized to retain confidentiality): Is there a way to use the RPM (x-axis on data set) and …
Ok. What is wrong with you code! I am trying to calculate transition probabilities for each leg. The code works for small array but for the actual dataset I got memory error. I have 64 g version python and maximized the memory usage so i believe need help to code efficiently. import numpy as np # sequence with 3 states -> 0, 1, 2 arr = [0, 1, 0, 0, 0, 2, 2, 1, 1, 1, 0, 0, 0, 0, …
I have some problems for computing derivative for sum of squares error in backprop neural network. For example, we have a neural network as in picture. For drawing simplicity, i've dropped the sample indexes. Сonventions: x - data_set input. W - is a weigth matrix. v - vector of product: W*x. F - activation function vector. y - vector of activated data D - vector of answers e - error signal lower index is a variable(NxN) - dimenstionality higher [index] …
The input to a convolutional layer of a neural network is an image of size $128\times 128\times 3.$ $ 40$ convolutional filters of size $5\times 5$ are applied to it. Would you get an output? If no then why not? If yes, what will be the output size?
The Tensorflow documentation here says that: tf.linalg.pinv is ''analogous to numpy.linalg.pinv. It differs only in default value of rcond''. However, tf.linalg.pinv requires the matrix to be float type while np.linalg.pinv could be used with complex matrices. I was wondering why they would only create it for float types and if there is a straightforward way to modify tf.linalg.pinv to be used with complex matrices.
I'm creating a recommendation system of football players based on stats of each player (number of passes, crosses, shots, tackles, etc ...) and I have already tried with a Content based recommender. Even though the results are ok, I want to try to apply a collaborative filtering. What I want to try is to recommend a player to a specific team and so far what i have thought is to create an implicit feedback system where i create an abstract …
Can you please tell me how to build an autoencoder with a single matrix(4,4) with integer numbers? I want to build an autoencoder for the below-mentioned data. I don't know whether I should convert the decimal numbers to binary first using one-hot encoding or a neural network will recognize integer numbers. e.g, input data = array([[ 4, 3, 8, 6], [ 1, 1, 2, 2], [24, 18, 32, 24], [ 6, 6, 8, 8]]) autoencoder(data) output data= array([[ 4, 3, …
Hello this might be a stupid question but i need some help indexing a Matlab matrix consisting of several sub-matrices. for k = 1:tf-1 r(k) = rand(1)/4; u(k+1) = 0.5; x1(k+1) = A(1,1)*x1(k) + A(1,2)*x2(k) + B(1,1)*u(k); x2(k+1) = A(2,1)*x1(k) + A(2,2)*x2(k) + B(2,1)*u(k); x = [x1(k) x2(k)]'; y(k) = C*x + r(k); P_prior(k+1) = A*P(k)*A.' + Q; K(k+1) = P_prior(k+1)*C.'/(C*P_prior(k+1)*C.' + R); xhat(k+1) = x(k+1) + K(k+1)*(y(k) - C*x(k+1)); P(k+1) = (eye(size(1,1)) - K(k+1)*C)*P_prior(k+1); end For example i want …
I have a data set where each record is a json document with a label, and an array of signals. The signals will vary for each record: { "label":"bad", "id": "0009", "signals":["high_debt_ratio", "no_job"] }, { "label":"good", "id": "0002", "signals":["high_debt_ratio", "great_credit", "no_id_match"] }, { "label":"good", "id": "0003", "signals":["low_debt_ratio", "great_credit"] }, { "label":"bad", "id": "0001", "signals":["high_risk_loc", high_debt_ratio", "no_job", "no_id_match"] } I want to convert this to a matrices that looks like this: id label high_risk_loc high_debt_ratio no_job great_credit no_id_match low_debt_ratio 0009 bad …
As a clarifier, I want to implement cross-correlation, but the machine learning literature keeps referring to it as convolution so I will stick with it. I am trying to implement image convolution using linear algebra. After looking around on the internet and thinking about it, I could come up with two possible solutions for that. The first one: Create an appropriate Toeplitz-like matrix out of the kernel as it is described here. The second one: Instead of the filter, modify …
I am currently studying the Elements of Statistical Learning book. The following equation is in page 120. It calculates the Hessian matrix for the log-likelihood function as follows \begin{equation} \dfrac{\partial^2 \ell(\beta)}{\partial\beta\partial\beta^T} = -\sum_{i=1}^{N}{x_ix_i^Tp(x_i;\beta)(1-p(x_i;\beta))} \end{equation} But is the following calculation it is only calculating $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i^2}$ terms. But Hessian matrix should also contain $\dfrac{\partial^2\ell(\beta)}{\partial\beta_i\partial\beta_j}$ where $i\neq j$. Please explain the reason for missing out these terms.
I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to achieve this?
Currently I am working on improving a stage on a data processing pipeline. The source data has a large number of fields and is getting normalized into a simpler entity. This entails that in many cases a destination field value may be copied from arbitrary input fields, according to the context. My idea was to regress a binary output sources-destinations matrix that associates the possible source fields to the possible source destinations. I was wondering: is this a problem that …
for better understanding, I would like to represent the calculations in a neural network with one hidden layer and one output layer as a matrix calculation. The hidden layer has 3072 neurons, the output layer 768. Specifically, we are talking about the Feed Forward Neural Network, which is used in BERT's Transformer architecture. There, a vector 1x768 is fed into the FFNN and the target is an output vector with 1x768 as well. The calculation would be as follows : …
I have downstream gradient for every $sample$ (each row for every $x_i$) $$ \begin{bmatrix} 0.0062123 & -0.00360166 & -0.00479891 \\ -0.01928449 & 0.01240768 & 0.01493274 \\ -0.01751177 & 0.01140975 & 0.01469825 \\ 0.0074906 & -0.00531709 & -0.00637952 \end{bmatrix} $$ And I have my inputs (my local gradient) $$ \begin{bmatrix} 0 & 0 \\ 0 & 1 \\ 1 & 0 \\ 1 & 1 \\ \end{bmatrix} $$ I want to calculate downgrade gradient and for this what I do is …
Just for a fun exercise, I am trying to invert a matrix, say size 28x28 (or even 5x5) with a neural network. The way I approached this (quite naively) is as follows: I built a fully convolutional network with some 8 layers and ReLU activations (not sure if this is the right choice) I put in an input $X$ and get an output $Y$. Here $Y=NN(X)$ where $X$ and $Y$ are both of same dimension, say $n\times n$. Assume NN …
I want to compute a similarity comparison for each entry in a dataset to every other entry that is labeled as class 1 (excluding the current entry if it has a label of 1). So, consider a matrix of training data that has columns for ID and class/label, and then a bunch of data columns. ID Label var1 var2 var3 ... varN 1 1 0.26 0.44 0.2 0.11 2 0 0.13 0.34 0.14 0.21 3 1 0.22 0.34 0.45 0.57 …
I'm writing my thesis and am trying to calculate a similarity matrix of houses. I currently have a dataset of 500,000 houses that I need to calculate the similarity between. I.e. I need to calculate 500,000 * 500,000 cells, and R cannot handle that, I am getting the error: Error: cannot allocate vector of size 2664.8 Gb Does anyone know a smart solution to this problem?
So I have a dataset containing different ID's and the time the data was created. ID Date 0 123123 2021-03-24 12:43:13.494000+00:00 1 123412 2021-03-24 12:43:13.494000+00:00 2 123123 2021-03-24 12:43:15.935000+00:00 3 234234 2021-03-24 12:43:15.935000+00:00 4 432424 2021-03-24 12:43:13.494000+00:00 The goal should be to validate that there is at least one data row for every id for every given day. What I did so far is converting the timestamps to dates like this: 0 2021-03-24 1 2021-03-24 2 2021-03-24 3 2021-03-24 4 …