Open source Anomaly Detection in Python

There is still an active and developed version of Skyline, just in case someone lands here and is interested.

Skyline (documentation)

I am the current maintainer of the project and it is now a lot more advanced than the original Etsy version, in terms of performance, UI, better handling of seasonality and has the added functionalities of an anomalies database, calculating correlations and the ability to fingerprint and learn not anomalous patterns.

I am currently on same stage like you. I am finding best option for anomaly detection, doing some research.

What I have found is I think best matches your need and is better compare to what you have seen. i.e., TwitterAnomalyDetection, SkyLine.

I have found better is Numenta's NAB (Numenta Anomaly Benchmark). It also have a very good community support and for you plus point is its open source & developed in Python. You can add your algorithm in it.

In case of algorithm, I found LOF, or CBLOF are good option.

So, check it out once. It may help you out.

If you found better option, please share.

I recently developed a toolbox: Python Outlier Detection toolbox (PyOD). See GitHub.

It is designed for identifying outlying objects in data with both unsupervised and supervised approaches. PyOD is featured for:

• Unified APIs, detailed documentation, and interactive examples across various algorithms.
• Advanced models, including Neural Networks/Deep Learning and Outlier Ensembles.
• Optimized performance with JIT and parallelization when possible, using numba and joblib. Compatible with both Python 2 & 3 (scikit-learn compatible as well).

Here are some important links:

• Github

• PyPI

• Documentation

• Interactive Jupyter Notebooks

If you use PyOD in a scientific publication, we would appreciate citations to the following paper

@article{zhao2019pyod,
  title={PyOD: A Python Toolbox for Scalable Outlier Detection},
  author={Zhao, Yue and Nasrullah, Zain and Li, Zheng},
  journal={arXiv preprint arXiv:1901.01588},
  year={2019},
  url={https://arxiv.org/abs/1901.01588}
}

It is currently under review at JMLR (machine learning open-source software track). See preprint.

Quick Introduction

PyOD toolkit consists of three major groups of functionalities: (i) outlier detection algorithms; (ii) outlier ensemble frameworks and (iii) outlier detection utility functions.

Individual Detection Algorithms:

• PCA: Principal Component Analysis (the sum of weighted projected distances to the eigenvector hyperplanes)
• MCD: Minimum Covariance Determinant (use the mahalanobis distances as the outlier scores)
• OCSVM: One-Class Support Vector Machines
• LOF: Local Outlier Factor
• CBLOF: Clustering-Based Local Outlier Factor
• LOCI: LOCI: Fast outlier detection using the local correlation integral
• HBOS: Histogram-based Outlier Score
• kNN: k Nearest Neighbors (use the distance to the kth nearest neighbor as the - **outlier score
• AvgKNN: Average kNN (use the average distance to k nearest neighbors as the outlier score)
• MedKNN: Median kNN (use the median distance to k nearest neighbors as the outlier score)
• ABOD: Angle-Based Outlier Detection
• FastABOD: Fast Angle-Based Outlier Detection using approximation
• SOS: Stochastic Outlier Selection
• IForest: Isolation Forest
• Feature Bagging
• LSCP: LSCP: Locally Selective Combination of Parallel Outlier Ensembles
• XGBOD: Extreme Boosting Based Outlier Detection (Supervised)
• AutoEncoder: Fully connected AutoEncoder (use reconstruction error as the outlier score)
• SO_GAAL: Single-Objective Generative Adversarial Active Learning
• MO_GAAL: Multiple-Objective Generative Adversarial Active Learning

Outlier Detector/Scores Combination Frameworks:

• Feature Bagging
• LSCP: LSCP: Locally Selective Combination of Parallel Outlier Ensembles
• Average: Simple combination by averaging the scores
• Weighted Average: Simple combination by averaging the scores with detector weights
• Maximization: Simple combination by taking the maximum scores
• AOM: Average of Maximum
• MOA: Maximization of Average

Utility Functions for Outlier Detection:

1. score_to_lable(): convert raw outlier scores to binary labels
2. precision_n_scores(): one of the popular evaluation metrics for outlier mining (precision @ rank n)
3. generate_data(): generate pseudo data for outlier detection experiment
4. wpearsonr(): weighted pearson is useful in pseudo ground truth generation

Comparison of all implemented models are made available below: (Figure, Code, Jupyter Notebooks):

If you are interested, please check Github for more information.

Since you have multivariate time series, I would go for a LSTM-RNN implementation that models the dynamics of your system based on training data, which are usually semi-supervised (only normal class included). This means that you train your model to learn what is "normal". During testing, you test both normal and anomalous conditions to see how well the model tells them apart.

An advantage of neural networks is that they "learn" the cross-correlations between input signals by themselves; you do not need to explore them manually. LSTM-RNNs, in particular, are an ideal choice when it comes to time series modelling simply because of their ability to keep memory of previous inputs, similar to a state space model in Control Theory (if you see the analogy).

In Python, it is almost trivial to implement an LSTM-RNN using Keras API (on top of Tensorflow backend). This network learns to estimate the signal(s) of interest given an arbitrary number of inputs, which you thereafter compare with the actual measured value. If there is "big" deviation, you got an anomaly (given that the model is accurate enough)!

I assume the feature you use to detect abnormality is one row of data in a logfile. If so, Sklearn is your good friend and you can use it as a blackbox. Check the tutorial of one-class SVM and Novelty detection.

However, in case that your feature is an entire logfile, you need to first summarize it to some feature of same dimension, and then apply Novealty detection.

h2o has an anomaly detection module and traditionally the code is available in R.However beyond version 3 it has similar module available in python as well,and since h2o is open source it might fit your bill.

You can see an working example over here

import sys
sys.path.insert(1,"../../../")
import h2o

def anomaly(ip, port):
    h2o.init(ip, port)

    print "Deep Learning Anomaly Detection MNIST"

    train = h2o.import_frame(h2o.locate("bigdata/laptop/mnist/train.csv.gz"))
    test = h2o.import_frame(h2o.locate("bigdata/laptop/mnist/test.csv.gz"))

    predictors = range(0,784)
    resp = 784

    # unsupervised -> drop the response column (digit: 0-9)
    train = train[predictors]
    test = test[predictors]

    # 1) LEARN WHAT'S NORMAL
    # train unsupervised Deep Learning autoencoder model on train_hex
    ae_model = h2o.deeplearning(x=train[predictors], training_frame=train, activation="Tanh", autoencoder=True,
                                hidden=[50], l1=1e-5, ignore_const_cols=False, epochs=1)

    # 2) DETECT OUTLIERS
    # anomaly app computes the per-row reconstruction error for the test data set
    # (passing it through the autoencoder model and computing mean square error (MSE) for each row)
    test_rec_error = ae_model.anomaly(test)

    # 3) VISUALIZE OUTLIERS
    # Let's look at the test set points with low/median/high reconstruction errors.
    # We will now visualize the original test set points and their reconstructions obtained
    # by propagating them through the narrow neural net.

    # Convert the test data into its autoencoded representation (pass through narrow neural net)
    test_recon = ae_model.predict(test)

    # In python, the visualization could be done with tools like numpy/matplotlib or numpy/PIL

if __name__ == '__main__':
    h2o.run_test(sys.argv, anomaly)

Anomaly Detection or Event Detection can be done in different ways:

Basic Way

Derivative! If the deviation of your signal from its past & future is high you most probably have an event. This can be extracted by finding large zero crossings in derivative of the signal.

Statistical Way

Mean of anything is its usual, basic behavior. if something deviates from mean it means that it's an event. Please note that mean in time-series is not that trivial and is not a constant but changing according to changes in time-series so you need to see the "moving average" instead of average. It looks like this:

The Moving Average code can be found here. In signal processing terminology you are applying a "Low-Pass" filter by applying the moving average.

You can follow the code bellow:

MOV = movingaverage(TimeSEries,5).tolist()
STD = np.std(MOV)
events= []
ind = []
for ii in range(len(TimeSEries)):
    if TimeSEries[ii] > MOV[ii]+STD:
        events.append(TimeSEries[ii])

Probabilistic Way

They are more sophisticated specially for people new to Machine Learning. Kalman Filter is a great idea to find the anomalies. Simpler probabilistic approaches using "Maximum-Likelihood Estimation" also work well but my suggestion is to stay with moving average idea. It works in practice very well.

I hope I could help :) Good Luck!