Index Lifecycle and Shard Allocation Optimization in Large- Scale Elasticsearch Clusters: A Performance–Cost Trade-Off Analysis
DOI:
https://doi.org/10.15662/3qh1b663Keywords:
Elasticsearch, Index Lifecycle Management, Shard Allocation, Distributed Search Systems, Indexing ThroughputAbstract
Modern search solutions rely on Elasticsearch clusters to scale to lots of data with a low latency of the search. Shard under sizing and ineffective index lifecycle policy may however augment latency and infrastructure cost. In this paper, a quantitative analysis of the allocation of shards and lifecycle management is introduced in large- scale clusters. The experiment results indicate that the medium size of a shard (35 GB) decreased the mean query latency to 118 ms, as compared to 142 ms (15 GB) and 156 ms (60 GB). The optimized configuration had an indexing throughput of 20, 200 docs/sec. Lifecycle tiering (hot-warm-cold) cut the storage overhead by
1.80 to 1.45 and cut the total cost each month down to $19,200, which is a savings of 20%. Equal shard distribution also cut down on the latency by 134 ms to 121 ms and shard variance was also reduced by 64 percent. The results indicate a combination of appropriate sizing of shard, lifecycle policy that is tier-sensitive, and a balanced distribution can achieve better performance and a significant reduction in the cost of infrastructure.
References
[1] Li, Y. (2019). Shard replication and resource efficiency in distributed search systems. Journal of Distributed Systems, 12(3), 45-62.
[2] Aldailamy, A. (2018). Performance evaluation of Solr, Terrier, and Katta distributed search platforms. International Conference on Information Retrieval, 234-248.
[3] Moses, J. (2022). A taxonomy of big data indexing techniques: Structures, algorithms, and trade- offs. ACM Computing Surveys, 54(5), 1-38.
[4] Yichuan, Z. (2020). Reliable and cost-efficient storage architectures for large-scale data systems. IEEE Transactions on Cloud Computing, 8(2), 412-427.
[5] Shaik, R. (2018). Leveraging Elasticsearch shards for scalable duplicate removal in big data pipelines. Big Data Research, 11, 78-89.
[6] Ahmed, R., & Boutaba, R. (2011). Distributed search techniques: Architectures and algorithms. Computer Networks, 55(10), 2341-2359.
[7] Mackenzie, J. (2022). Anytime ranking algorithms for dynamic shard selection. ACM SIGIR Conference, 567-578.
[8] Liu, H., Chen, X., & Wang, L. (2021). Online migration strategies for cost optimization in cloud storage systems. IEEE Transactions on Parallel and Distributed Systems, 32(7), 1654-1668.
[9] Zhao, Y., Liu, J., & Zhang, M. (2017). Efficient metadata indexing for billion-scale object storage. USENIX Conference on File and Storage Technologies, 145-158.
[10] Wang, S., Li, D., & Zhou, X. (2016). SSD-accelerated computing architectures for high-performance search engines. ACM Transactions on Storage, 12(4), 1-29.
[11] Kulkarni, A., & Callan, J. (2015). Selective search: Efficient resource allocation in distributed information retrieval. Information Retrieval Journal, 18(6), 527-549.
[12] Baeza-Yates, R., Gionis, A., Junqueira, F., Murdock, V., Plachouras, V., & Silvestri, F. (2008). Design trade-offs for search engine caching. ACM Transactions on the Web, 2(4), 1-28.
[13] Lin, S., Zeinalipour-Yazti, D., Kalogeraki, V., Gunopulos, D., & Najjar, W. A. (2006). Efficient indexing data structures for flash-based sensor devices. ACM Transactions on Storage, 2(4), 468-503.
[14] Ganesan, P., Garcia-Molina, H., & Widom, J. (2005). Exploiting hierarchical domain structure to compute similarity. ACM Transactions on Information Systems, 21(1), 64-93.
[15] Datta, S., Patel, R., & Kumar, V. (2021). Quorum-based replication and reconstruction codes for distributed storage. IEEE Transactions on Dependable and Secure Computing, 18(3), 1245-1260.
[16] Chikhaoui, B., Wang, S., & Pigot, H. (2021). Data placement optimization in distributed storage systems: A survey. ACM Computing Surveys, 53(6), 1-35.
[17] Ha, K., Park, J., & Lee, S. (2021). Machine learning approaches for hot data identification in tiered storage systems. IEEE Access, 9, 45678-45692.
[18] Manchana, P. (2020). Optimizing batch workloads in Elasticsearch clusters. International Journal of Database Management Systems, 12(2), 23-38.
[19] Park, S., Kim, H., & Choi, Y. (2019). Design principles for cold storage in large-scale archival systems. ACM Transactions on Storage, 15(3), 1-26.
[20] Li, X., Zhang, Y., & Chen, W. (2018). Resource utilization analysis in shard replication for distributed databases. Journal of Parallel and Distributed Computing, 121, 89-103.
[21] di Vimercati, S. D. C., Foresti, S., Jajodia, S., Paraboschi, S., & Samarati, P. (2018). Enhancing data confidentiality through strategic data swapping. IEEE Transactions on Dependable and Secure Computing, 15(4), 667-681.
[22] Niu, Q., Dinan, J., Lu, Q., & Sadayappan, P. (2018). A comprehensive survey of hybrid storage systems: Architectures and optimization techniques. ACM Computing Surveys, 51(3), 1-33.
[23] Sri, K., Reddy, M., & Kumar, P. (2017). Data mining and analytics using Elasticsearch: A practical approach. International Journal of Advanced Computer Science and Applications, 8(5), 234-241.
[24] Duckham, M., Nittel, S., & Worboys, M. (2011). Decentralized spatial computing: Foundations of geosensor networks. Spatial Cognition & Computation, 5(2-3), 171-188.
[25] Wei, L., Zhao, H., & Liu, Q. (2020). Mathematical modeling for optimal shard number determination in Elasticsearch. Performance Evaluation, 139, 102-118.
[26] Aldailamy, A., Abdallah, M., & Fung, B. C. M. (2018). Solr vs Terrier: A comparative performance analysis of open-source search platforms. Information Processing & Management, 54(6), 1158-1177.
[27] Berglund, E. (2014). Shard selection strategies in Elasticsearch distributed search. Master's Thesis, KTH Royal Institute of Technology.
[28] Junqueira, F., Bhagwan, R., Hevia, A., Marzullo, K., & Voelker, G. M. (2012). Surviving slashdot: Reactive index replication for peer-to-peer search. IEEE Transactions on Parallel and Distributed Systems, 23(8), 1481-1492.
[29] Kulkarni, A., & Callan, J. (2010). Document allocation policies for selective searching of distributed indexes. ACM CIKM Conference, 449-458.
[30] Qian, G. (2008). Adaptive indexing for content-based search in P2P systems. Data & Knowledge Engineering, 67(3), 381-398.
