Slack logs5/31/2023 ![]() ![]() Suman mentioned they found out that running Elasticsearch on Kubernetes is quite painful. He notes that this is in contrast to pre-provisioned systems such as Elasticsearch.īeing cloud-native is another central criterion. What Slack has done to solve this problem is to build KalDB to automatically scale by the logging volume. As he says, that is a ton of log messages that your system has no control over. He gives the common case of log storms, a spike in logging volume like you might see during peak times (think first day of the year when everybody logs back in) or under a massive event such as outage or an infrastructure migration. In our conversation, Suman is eager to point out that KalDB is designed to handle scalable logging volume. Elasticsearch challenges and Slack’s KalDB approach I also see organizations moving to developing directly on top of Lucene to gain better performance and reduced cost, such as Yelp engineering that built Nrtsearch to replace Elasticsearch for their needs. ![]() Both Apache Solr and ElasticSearch are based on the Lucene library, and Solr certainly provides features that Elasticsearch lacks. Many assume that the shortcomings of Elasticsearch are indicative of problems with the Lucene library, but this is not the case. Lucene actually is a very good storage engine and the architecture we had for LogLens works fairly well even today.” In my opinion as well, the base Apache Lucene library is undervalued. In his words, “Most systems assume that Elasticsearch is expensive because Lucene is expensive, that’s actually not quite true. According to Suman, Lucene is a more efficient architecture than most give credit for. Apache Lucene is an open-source Java library for indexing and searching of textual data. At its roots, Slack’s KalDB, like Elasticsearch and Solr, is a Lucene-based indexing system.
0 Comments
Leave a Reply. |