Vipul Pathak

In an elastic cluster, there are many entities that makes up a functioning cluster. Nodes, Primary Shards and Replicas are the ones frequently getting referred in documentation and articles.

The engine of elastic search, is an open source library written in Java called Lucene. This library is capable of indexing and performing full text search.

So what shards have to do with Lucene?

Ok, it’s time to take an example 🙂

Consider I have 72 million files of an average size 15 KB each. To store all these documents uncompressed for searchability, we will require about 1 TB of storage space. One option is to have a gigantic disk/SSD in a computer and store all these documents in one machine. However, there is always a risk that entire data will vanish, if this single computer fails. Also, instead of 72 million files, if I had 500 million files of 100 KB each, then this wouldn’t be any option.

Another option is to store all these documents on a group of computers. This group of computers is generally referred as a Cluster.

Nodes

In elastic search terms, a cluster consists of physical (or virtualized) computers that can do actual work. Every computer is one such node. Each node may have a specific role assigned to it. But in this example, we are only concerned about nodes that can store data and do real search work, (known as Data nodes).

So to store these 72 million files on this group on computer, I could divide these files in multiple parts, e.g. 18 million files on each computer, so on 4 computers all my 72 millions files can be stored.

I also need searchability in the stored files, so I created a Lucene Index on each computer to store 18 million files present on that computer. This Lucene index read each of my file and store them against a unique ID. I can retrieve any file any time not only using it’s ID but I can also retrieve multiple files using text search queries.

But still what are Shards?

In elastic search terms, every Lucene Index present on each of the computer is a Shard. A shard is a fully functional Lucene index, containing a set of data, i.e. Shard containing 18 million files out of the total of 72 million files. A node may contain one or more such Lucene Indices storing different sets of data. For example, one possibility is that I may store two sets of data on one computer, e.g. 1-18 million files in one index and 37-54 million files on second index, and both indices on the same computer.

But what if one out of these four computers fail?

This is of course possible. When that happens, we will loose 25% of out data. To mitigate risk of such failure, we should store copies of these partial indices on multiple computers. For example, set 1 of these files (18 million files) is stored in the Lucene index on computer 1 but we should also keep a copy of set 1 on computer 3. Similarly, set 3 is originally indexed on computer 3 but we should keep a copy of set 3 on computer 2 also. So if any computer fails, we can find a copy of data stored by that computer on a second computer on the network.

Is that what a Replica is?

Absolutely.

When computers (nodes) are working together for indexing purpose, Elastic Search do not call partial indices, Index. An index on an elastic cluster is a group of all partial indexes (shards).

In elastic search terms, what we were referring to as “Set 1 of originally stored files” is a Primary Shard 1 (Let’s call it P1). A copy of primary shard 1, is the Replica Shard- R1. Both primary and replica are Shards, except that order of indexing new documents is primary first and replica next.

Elastic search replicates the shards for failover and high availability and so it never keeps the replica and primary of the same set on the same node. Therefore, P1 and R1 can’t be stored on the same node. Similarly, P3 and R3 will be stored on different nodes.