Tool for test/train automation

I need to test different datasets as well as different algorithm implementations. The current workflow looks like:

• Perform feature extraction from train set
• Train classifier on this features
• Feed this classifier to production code
• Run production code on a test set, feeding samples one by one
• Take results of production code and convert to the same format as test dataset
• Show statistics

How to automate this as much as possible? I currently use a bunch of bash/python scripts. I've looked through sklearn pipelines, but not sure they would give any benefit except for running functions in a line.