Last week we had an issue with the logs being shipped to our central Elasticsearch cluster.
We noticed that the amount of traffic being sent was bigger than the amount of data being saved.
In order to fix it, we found two configuration issues that we had to deal with:
1. We found an issue in one of our plugins that kept crashing since it couldn't save the position of the logs it always started pulling them from the beginning of the file.
2. We have added a fingerprint plugin that now ensures that there can't be any duplication of data in the cluster.