Identifying patterns in tabular data

Question

Identifying patterns in tabular data

Bernhard

2022年3月21日 23:03

I have a set of tables containing some thousand entries and some tenths of columns from machine status values of production. The entries are of mixed types like string, float, or timestamp. Each table is pre-labeled with a certain failure mode (e.g. valve setting jump, the problem with inlet A, etc.). This could be due to a jump in the mean values in some columns or a special correlation between several columns. This is what I refer to as a pattern. I would like to generate a machine learning-based classification model for this data set that can recognize these patterns. This means that I need to feed in a table of tables as data points. Up to now I only came across either tabular data with individual data points (i.e. integer, float or strings) or images, which are kinds of tables but have a fixed ordering of rows and columns. Any idea on how to tackle this problem?

Topic pattern-recognition data-table

Category Data Science

Bernhard · Accepted Answer · 2021年6月21日 05:07

thanks Nicolas for your hint. I doesn't work, though. I tried to create a minimal example as I can't share the original data:

from sklearn import datasets
from sklearn.model_selection import train_test_split
from random import randint

#generate 100 datasets with arbitrary values
dataset_x = []
labels = []
for i in range(100):
    x_values, _ = datasets.make_sparse_uncorrelated(n_features=20,n_samples=150)
    dataset_x.append(x_values)
    #label each dataset
    if randint(0,10) < 5:
        labels.append('failure')
    else:
        labels.append('no failure')

#split into train and test
x_train, x_test, y_train, y_test = train_test_split(dataset_x,labels,train_size=.8)  

#here is where the magic should happen
from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(max_depth=5, random_state=0)
clf.fit(x_train,y_train)

as expected, this fails (ValueError: Found array with dim 3. Estimator expected <= 2.).

Nicolas Martin · Accepted Answer · 2021年6月17日 15:00

1

Nicolas Martin answered at 2021年6月17日 15:00

Can you share a sample of the data?

Otherwise, Random Forest Classifier could be a good solution to your problem... https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html

Identifying patterns in tabular data

About