The ELK stack, Elasticsearch, Logstash, and Kibana, is the default answer to “we need to search our logs.” It is open source, it is flexible, and it works. It also has a well-earned reputation for getting expensive and operationally heavy as log volume grows. This post explains why Elasticsearch cost and scaling problems appear, why they are structural rather than a tuning mistake, and what a lossless, storage-native alternative changes.
Why Elasticsearch gets expensive
Elasticsearch is an inverted-index search engine. To make logs searchable, it indexes them, and that index is not free:
- Index overhead. The index can rival or exceed the size of the raw data, so you store your logs more than once.
- Hot compute. Fast search wants data on hot nodes with CPU and memory, which is the pricey tier. Retention on hot storage gets costly fast.
- Cluster operations. Sharding, rebalancing, JVM heap tuning, and node failures are ongoing work. The human cost of running ELK at scale is real and easy to underestimate.
- Retention pressure. Keeping months of logs searchable means either a large hot tier or a tiered setup you have to design and babysit.
The usual response is to shorten retention or drop noisy logs, which is the same fidelity-for-cost trade we cover in sampling vs. lossless. You save money by keeping less, and you find the gap during the next investigation.
The structural issue: index-first design
The root cause is that the index-first model optimizes for low-latency search over recent data at the expense of storage efficiency and long retention. That is a great fit for some workloads and a painful one for high-volume observability and long compliance retention, where most data is written, rarely read, and must be kept for a long time. You end up paying hot-tier prices to store cold data.
A storage-native, lossless alternative
Sasquatch Labs takes a different shape. Instead of building and storing a large index, it compresses logs, metrics, and traces losslessly by an order of magnitude or more and stores the compressed data in your own object storage. Search does not require a hot index tier: the Snowman query engine reads the compressed data in place and speaks LogQL, SPL, TraceQL, and PromQL. The economics change because you are storing compressed data on cheap object storage instead of an inflated index on expensive hot nodes, and because there is no cluster to operate.
- No index bloat. Compressed data instead of a multiplied index footprint.
- Cheap retention. Long windows live on object storage, not hot search nodes. See log retention costs.
- Less to operate. No shard rebalancing, heap tuning, or 3 a.m. cluster firefights.
- In your cloud. Data stays in your environment, no egress. See BYOC observability.
The ELK stack earned its popularity, but if your Elasticsearch bill and operational burden are climbing with log volume, the fix is not more tuning, it is a model that does not store an expensive index of data you rarely read. See the alternative at sasquatchlabs.io.