Batch-driven or event-driven ETL

I am trying to come up with a data pipeline architecture. The data I deal with is event logging for labs requested, failed, succeeded etc with timestamps and some customer info for several different customers. Eventually I want that data being dumped into a dashboard, for both external and internal use. What's the best way to approach it: event-driven or batch driven ETL? We don't care much for real-time processing, and the data is rather small.

Topic data-engineering etl

Category Data Science


If real-time ETL processing is not important, then use batch-driven processing. It is easier for you to monitor since you know the pipeline schedule and you save hardware by scheduling it to more quiet times.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.