how to calculate evolution change in percentage in a dataset compared to itself

scenario: lets say I have a table for number of customers visiting a shop per day. now I want to calculate that how much change has happened in terms of number of visitors throughout the past 30 days and come up with a single value that tells me the change rate (in percentage). but I do not want to compare the changes of current month with the values of the previous month(s) instead I want to compare the current month …
Category: Data Science

How to select good inputs and fitness function to achive good results with NEAT for Icy Tower bot

I'm trying to make a bot to the famous "Icy Tower" game. I rebuilt the game using pygame and I'm trying to build the bot using Python-NEAT. Every generation a population of 70 characters tries to jump to the next platform and increase their fitness. right now the fitness is the number of platforms they jumped on, each platform gives +10. The problem I'm facing is that the bot isn't learning good enough after 1000 generations the best score was …
Category: Data Science

Transitioning from "furthest" to "most reliable" in fitness function

I am trying to train an evolutionary algorithm to take 100 steps perfectly. I figure that in the early game I want to let each AI run once, then select the ones that went the furthest, but in the late game when they start to reach step 100 (no other distinction can be made between them, as step 100 is victory) I want to breed for reaching the end reliably in a series of attempts. What is a good way …
Category: Data Science

Models for Long-Term Time-Series Forecasting and Pattern Recognition

I'm trying to find a solution for long-term electricity hourly prices forecasting. Explaining simply, I have some data from 2018 - 2021 containing Demand, Renewable Generation, Hydropower Generation, System Losses, Energy Exchange, and Electricity Prices for each Energy Submarket (all this in hourly format). What I want to answer is: is possible to create a model to predict the prices for long-term or mid-term forecasting that identify the correlation on the Data? For example, if I increase the Renewable Generation …
Category: Data Science

Best way to optimize problem with additively separable fitness function?

I am using a genetic algorithm to maximize a few hundred thousand real-valued variables. Each of the variables, $x_i$, has its own independent boundary condition. The fitness function uses each of these variables to compute another value and returns the sum of everything: $$fitness = g(x_1) + g(x_2) + g(x_3) \ + \ ...$$ This is taking incredibly long in python. In this situation, what do I gain by maximizing all values at the same time, i.e. using the genetic …
Category: Data Science

avoiding premature convergence with neural networks (EA's)

I am currently writing a program that would be able to play snake on an 25*25 grid. It works by optimizing a set of weights of 300 different solutions (each solution would be a different neural network) with the aid of an evolutionary strategy, thus by random mutation and parent selection. I have decided not to apply crossover on the parents pool due to the black box problem concerning MLPs and other neural networks. My population size is 300 and …
Category: Data Science

Paper about AE-CNN is unclear. Deriving layers of dense blocks?

I am implementing the algorithm called Automatically Evolving CNN (AE-CNN). Some things aren't specified which makes it a bit hard to understand what the paper actually means to say. In the chapter 3.2 Encoding Strategy it says: ...Note that the number of convolutional layers in a DB is known because it can be derived by the spatial sizes of input and output as well as k. ... By DB it means a single dense block from the DenseNet algorithm. K …
Category: Data Science

Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that it can find the minimum of a function in a reasonable time. The parameter optimization takes a very long time, and it only has to be done once after which I will just use the best combination and just run those …
Category: Data Science

Creating a generic mathematical formula using a genetic algorithm

Assuming all of the following; I have 4 known numbers, all within a 0-400 range, like this: Variable1 Variable2 Variable3 Variable4 0-400 0-400 0-400 0-400 I know that there is a mathematical relationship between the numbers. I would like to use a genetic algorithm (computer code) to estimate/approximiate Variable2 and Variable3 based on Variable1 and Variable4. Also, importantly, assume that there are many input samples and that each sample will differ slightly. Thus, a genetic algorithm optimization of "a mathematical …
Category: Data Science

Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) implementation with Tensorflow Sequential model

After coming across this article about evolution strategies http://blog.otoro.net/2017/10/29/visual-evolution-strategies/, it seems clear that the Covariance-Matrix Adaptation Evolution Strategy (CMA-ES) has the potential to be incredibly effective. I'd like to try to create a similar algorithm to be used with a Tensorflow Keras Sequential model. I've only seen simple examples, but here's how I think it could be done with a Sequential model: Build the model as if backpropagation were being used and split the data into training and test sets …
Category: Data Science

Mutation on demand in genetic algorithms

genetic algorithm usually use a "mutation rate" to control the rate of chromosome mutation. Most of the researchers at researchgate recommend to keep this rate low in order to converge quickly, to be able to find local optima and not to make the optimization a random walk. However, I see one major problem with keeping a low mutation rate. If the breeding at one point doesn't result into "new" children/chromosomes, the algorithm will run very inefficiently. Assume that all individuals …
Category: Data Science

Is there any advantage in using Particle Swarm Optimization for clustering than K-Means?

I have read some paper about using particle swarm optimization. It doesn't look give much different result than K-Means. I tried to use PSO for clustering but the result is pretty much the same with K-Means with some drawbacks like longer execution time and have a lot of different result caused by the random factor.
Category: Data Science

What makes a problem good for an evolutionary strategy vs a genetic algorithm vs particle swarm optimization?

I understand that evolutionary strategies (ES), genetic algorithms (GA), and particle swarm optimization (PSO) are all algorithms used to solve optimization types of problems, but what might make an optimization problem better to choose one of those techniques over the other? For example, I read in an article that genetic algorithms are typically preferred for combinatorial problems, but other than that one sentence or so, I haven't been able to find any source that explains when you might choose one …
Category: Data Science

What algorithm could be used to fuzzy merge multiple datasets?

Problem Description I have several tables that are related but do not share any unique key. I've come across this problem several times with customer data in separate source systems that needs to be compared together. Lets say my data is multiple tables, Table A through Z. There may be columns where I'm 100% certain on a match. For example table A and B have the column tax ID which is a certain match joining A to B. Both A …
Category: Data Science

What is new population in genetic algorithm?

Here is my (mis?)understanding of genetic algorithm: Create n individuals. This is initial population Calculate fitness of each individual in this population for i in range(n): select two individuals randomly with replacement from population with probability of selection of each individual proportional to its fitness value cross over the two selected individuals to create two offsprings introduce mutation Calculate fitness value for each offspring Now you have an initial population of n and n offsprings. What comes now? Is my …
Category: Data Science

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.