Elasticsearch and Duplications

May 16, 2018
Contact Us
Weekly Shorts are topics we discuss in our weekly remote meeting related to recent work we have done with our customers
Elasticsearch and Duplications

Last week we had an issue with the logs being shipped to our central Elasticsearch cluster.

We noticed that the amount of traffic being sent was bigger than the amount of data being saved.

In order to fix it, we found two configuration issues that we had to deal with:

1. We found an issue in one of our plugins that kept crashing since it couldn't save the position of the logs it always started pulling them from the beginning of the file.

2. We have added a fingerprint plugin that now ensures that there can't be any duplication of data in the cluster.

Elasticsearch and Duplications
Josh Dvir
Senior Software / Operations Architect
Josh has 13 years experience as a software hands-on technologist, manager, and entrepreneur with experience building numerous web applications, varying from small websites to large, high scale, web applications with millions of users. With extensive knowledge in full stack dev, big data, Cloud, Linux operations, architecture and can work across any technology and industry to help introduce better DevOps practices and tools to help you win. In his free time, he enjoys resolving technical challenges and creating solutions that just work. He deserves a lot of credit for his biggest strength is that you can always rely on him and he'll always deliver. He is passionate to release software to the world - making simple solutions to complicated problems.